2024-08-17T16:49:42.323 INFO:root:teuthology version: 1.2.0 2024-08-17T16:49:42.350 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-17T16:49:42.458 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-08-17_03:08:15-upgrade-main-distro-default-smithi/7858332 branch: main description: upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} email: ceph-qa@ceph.com first-half-sequence: - cephadm.shell: env: - sha1 mon.a: - ceph config set mgr mgr/cephadm/daemon_cache_timeout 60 - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps - echo wait for minority of mons to upgrade - while ! ceph mon versions | grep $sha1 && ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error; do sleep 2 ; done - ceph orch ps - ceph orch upgrade pause - sleep 60 - ceph orch upgrade resume - echo wait for majority of mons to upgrade - 'while ! ceph mon versions | grep $sha1 | egrep '': [23]'' && ceph orch upgrade status | jq ''.in_progress'' | grep true && ! ceph orch upgrade status | jq ''.message'' | grep Error; do sleep 2 ; done' - ceph orch ps - ceph orch upgrade pause - sleep 60 - ceph orch upgrade resume - echo wait for all mons to upgrade - 'while ! ceph mon versions | grep $sha1 | grep '': 3'' && ceph orch upgrade status | jq ''.in_progress'' | grep true && ! ceph orch upgrade status | jq ''.message'' | grep Error; do sleep 2 ; done' - ceph orch ps - ceph orch upgrade pause - sleep 60 - ceph orch upgrade resume - echo wait for half of osds to upgrade - 'while ! ceph osd versions | grep $sha1 | egrep '': [45678]'' && ceph orch upgrade status | jq ''.in_progress'' | grep true && ! ceph orch upgrade status | jq ''.message'' | grep Error; do sleep 2 ; done' - ceph orch upgrade pause - ceph orch ps - ceph orch ps - ceph versions first-half-tasks: - workunit: branch: quincy clients: client.0: - rbd/import_export.sh env: RBD_CREATE_ARGS: --new-format - print: '**** done rbd/import_export.sh 5-workload' first_in_suite: false job_id: '7858332' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'run basic import/export cli tests for rbd ' - desc: 'run randomized correctness test for rados operations generate write load with rados bench ' - desc: 'run basic cls tests for rbd ' - desc: 'run basic import/export cli tests for rbd ' - desc: 'librbd C and C++ api tests ' - desc: 'randomized correctness test for rados operations on a replicated pool, using only reads, writes, and deletes ' - desc: 'randomized correctness test for rados operations on a replicated pool with snapshot operations ' - desc: 'run randomized correctness test for rados operations generate write load with rados bench ' name: teuthology-2024-08-17_03:08:15-upgrade-main-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true create_rbd_pool: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - OSD_DOWN - mons down - mon down - MON_DOWN - out of quorum - PG_DEGRADED - Reduced data availability - Degraded data redundancy - OSDMAP_FLAGS - PG_AVAILABILITY sha1: cd8bdbeb9fbdab5733aa3aaadc042c965473e453 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: cd8bdbeb9fbdab5733aa3aaadc042c965473e453 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 workunit: branch: main sha1: 91f88e4d132ac08dc53355fa3d9717a53236a3a9 owner: scheduled_teuthology@teuthology priority: 850 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b second-half-sequence: sequential: - cephadm.shell: env: - sha1 mon.a: - ceph orch upgrade resume - sleep 60 - echo wait for upgrade to complete - while ceph orch upgrade status | jq '.in_progress' | grep true ; do ceph orch ps ; ceph versions ; sleep 30 ; done - echo upgrade complete - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 second-half-tasks: - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - print: '**** done end radosbench.yaml' seed: 17 sha1: cd8bdbeb9fbdab5733aa3aaadc042c965473e453 sleep_before_teardown: 0 stress-tasks: - thrashosds: aggressive_pg_num_changes: false chance_force_recovery: 0 chance_pgnum_grow: 1 chance_pgpnum_fix: 1 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 chance_thrash_pg_upmap_items: 0 disable_objectstore_tool_tests: true timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - print: '**** done end radosbench.yaml' - workunit: branch: quincy clients: client.0: - cls/test_cls_rbd.sh - print: '**** done cls/test_cls_rbd.sh 5-workload' - workunit: branch: quincy clients: client.0: - rbd/import_export.sh env: RBD_CREATE_ARGS: --new-format - print: '**** done rbd/import_export.sh 5-workload' - workunit: branch: quincy clients: client.0: - rbd/test_librbd.sh env: RBD_FEATURES: '61' - print: '**** done rbd/test_librbd.sh 7-workload' - full_sequential: - rados: clients: - client.0 objects: 500 op_weights: delete: 10 read: 45 write: 45 ops: 4000 write_append_excl: false - print: '**** done rados/readwrite 5-workload' - full_sequential: - rados: clients: - client.0 objects: 50 op_weights: delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 write_append_excl: false - print: '**** done rados/snaps-few-objects 5-workload' subset: 11/32 suite: upgrade suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_91f88e4d132ac08dc53355fa3d9717a53236a3a9/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 91f88e4d132ac08dc53355fa3d9717a53236a3a9 targets: smithi003.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCH6i447adJ6wn/imii8B2s80ixVL24yHaKkMUr4mwIX2ndRavpGrWppgUf0JtwEKBZF41FluZIRM2t7a+jyz4E= smithi053.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNkFjevriXiITaWp6AvbtkriGEgU+eEd3LGNRvmenBZ7GQXooU5AzURmei/145irrOKI1HxRAkDWqKCsLzcmavs= tasks: - pexec: all: - sudo dnf install nvmetcli nvme-cli -y - install: branch: quincy exclude_packages: - ceph-volume - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy - cephadm.shell: mon.a: - ceph fs volume create foo - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph.healthy: null - print: '**** upgrading first half of cluster, with stress ****' - parallel: - first-half-tasks - first-half-sequence - print: '**** done upgrading first half of cluster ****' - ceph.healthy: null - print: '**** applying stress + thrashing to mixed-version cluster ****' - parallel: - stress-tasks - ceph.healthy: null - print: '**** finishing upgrade ****' - parallel: - second-half-tasks - second-half-sequence - ceph.healthy: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 67db276914d29505c9707d04f3a3b8fb9d8aee4b timestamp: 2024-08-17_03:08:15 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-08-17T16:49:42.458 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_91f88e4d132ac08dc53355fa3d9717a53236a3a9/qa; will attempt to use it 2024-08-17T16:49:42.459 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_91f88e4d132ac08dc53355fa3d9717a53236a3a9/qa/tasks 2024-08-17T16:49:42.459 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-08-17T16:49:42.461 INFO:teuthology.task.internal:Checking packages... 2024-08-17T16:49:42.481 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'cd8bdbeb9fbdab5733aa3aaadc042c965473e453' 2024-08-17T16:49:42.481 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-17T16:49:42.481 INFO:teuthology.packaging:ref: None 2024-08-17T16:49:42.482 INFO:teuthology.packaging:tag: None 2024-08-17T16:49:42.482 INFO:teuthology.packaging:branch: main 2024-08-17T16:49:42.482 INFO:teuthology.packaging:sha1: cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T16:49:42.482 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-08-17T16:49:42.713 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-5842.g4887e20f 2024-08-17T16:49:42.714 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-08-17T16:49:42.721 INFO:teuthology.task.internal:no buildpackages task found 2024-08-17T16:49:42.722 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-08-17T16:49:42.744 INFO:teuthology.task.internal:Saving configuration 2024-08-17T16:49:42.767 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-08-17T16:49:42.774 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-08-17T16:49:42.809 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-08-17_03:08:15-upgrade-main-distro-default-smithi/7858332', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-17 16:44:59.697492', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCH6i447adJ6wn/imii8B2s80ixVL24yHaKkMUr4mwIX2ndRavpGrWppgUf0JtwEKBZF41FluZIRM2t7a+jyz4E='} 2024-08-17T16:49:42.840 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-08-17_03:08:15-upgrade-main-distro-default-smithi/7858332', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-17 16:44:59.698929', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNkFjevriXiITaWp6AvbtkriGEgU+eEd3LGNRvmenBZ7GQXooU5AzURmei/145irrOKI1HxRAkDWqKCsLzcmavs='} 2024-08-17T16:49:42.840 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-08-17T16:49:42.847 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-08-17T16:49:42.847 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-08-17T16:49:42.847 INFO:teuthology.run_tasks:Running task console_log... 2024-08-17T16:49:42.917 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1ee4977ac0>, signals=[15]) 2024-08-17T16:49:42.917 INFO:teuthology.run_tasks:Running task internal.connect... 2024-08-17T16:49:42.924 INFO:teuthology.task.internal:Opening connections... 2024-08-17T16:49:42.924 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi003.front.sepia.ceph.com 2024-08-17T16:49:42.925 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-17T16:49:43.012 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi053.front.sepia.ceph.com 2024-08-17T16:49:43.013 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-17T16:49:43.103 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-08-17T16:49:43.111 DEBUG:teuthology.orchestra.run.smithi003:> uname -m 2024-08-17T16:49:43.130 INFO:teuthology.orchestra.run.smithi003.stdout:x86_64 2024-08-17T16:49:43.130 DEBUG:teuthology.orchestra.run.smithi003:> cat /etc/os-release 2024-08-17T16:49:43.187 INFO:teuthology.orchestra.run.smithi003.stdout:NAME="CentOS Stream" 2024-08-17T16:49:43.187 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION="9" 2024-08-17T16:49:43.188 INFO:teuthology.orchestra.run.smithi003.stdout:ID="centos" 2024-08-17T16:49:43.188 INFO:teuthology.orchestra.run.smithi003.stdout:ID_LIKE="rhel fedora" 2024-08-17T16:49:43.188 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION_ID="9" 2024-08-17T16:49:43.188 INFO:teuthology.orchestra.run.smithi003.stdout:PLATFORM_ID="platform:el9" 2024-08-17T16:49:43.188 INFO:teuthology.orchestra.run.smithi003.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-17T16:49:43.188 INFO:teuthology.orchestra.run.smithi003.stdout:ANSI_COLOR="0;31" 2024-08-17T16:49:43.188 INFO:teuthology.orchestra.run.smithi003.stdout:LOGO="fedora-logo-icon" 2024-08-17T16:49:43.188 INFO:teuthology.orchestra.run.smithi003.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-17T16:49:43.188 INFO:teuthology.orchestra.run.smithi003.stdout:HOME_URL="https://centos.org/" 2024-08-17T16:49:43.188 INFO:teuthology.orchestra.run.smithi003.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-08-17T16:49:43.188 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-17T16:49:43.188 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-17T16:49:43.189 INFO:teuthology.lock.ops:Updating smithi003.front.sepia.ceph.com on lock server 2024-08-17T16:49:43.222 DEBUG:teuthology.orchestra.run.smithi053:> uname -m 2024-08-17T16:49:43.240 INFO:teuthology.orchestra.run.smithi053.stdout:x86_64 2024-08-17T16:49:43.241 DEBUG:teuthology.orchestra.run.smithi053:> cat /etc/os-release 2024-08-17T16:49:43.299 INFO:teuthology.orchestra.run.smithi053.stdout:NAME="CentOS Stream" 2024-08-17T16:49:43.299 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION="9" 2024-08-17T16:49:43.299 INFO:teuthology.orchestra.run.smithi053.stdout:ID="centos" 2024-08-17T16:49:43.299 INFO:teuthology.orchestra.run.smithi053.stdout:ID_LIKE="rhel fedora" 2024-08-17T16:49:43.299 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION_ID="9" 2024-08-17T16:49:43.299 INFO:teuthology.orchestra.run.smithi053.stdout:PLATFORM_ID="platform:el9" 2024-08-17T16:49:43.299 INFO:teuthology.orchestra.run.smithi053.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-17T16:49:43.299 INFO:teuthology.orchestra.run.smithi053.stdout:ANSI_COLOR="0;31" 2024-08-17T16:49:43.299 INFO:teuthology.orchestra.run.smithi053.stdout:LOGO="fedora-logo-icon" 2024-08-17T16:49:43.300 INFO:teuthology.orchestra.run.smithi053.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-17T16:49:43.300 INFO:teuthology.orchestra.run.smithi053.stdout:HOME_URL="https://centos.org/" 2024-08-17T16:49:43.300 INFO:teuthology.orchestra.run.smithi053.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-08-17T16:49:43.300 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-17T16:49:43.300 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-17T16:49:43.300 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server 2024-08-17T16:49:43.331 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-08-17T16:49:43.339 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-08-17T16:49:43.345 INFO:teuthology.task.internal:Checking for old test directory... 2024-08-17T16:49:43.345 DEBUG:teuthology.orchestra.run.smithi003:> test '!' -e /home/ubuntu/cephtest 2024-08-17T16:49:43.348 DEBUG:teuthology.orchestra.run.smithi053:> test '!' -e /home/ubuntu/cephtest 2024-08-17T16:49:43.364 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-08-17T16:49:43.370 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-08-17T16:49:43.370 DEBUG:teuthology.orchestra.run.smithi003:> test -z $(ls -A /var/lib/ceph) 2024-08-17T16:49:43.405 DEBUG:teuthology.orchestra.run.smithi053:> test -z $(ls -A /var/lib/ceph) 2024-08-17T16:49:43.428 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-08-17T16:49:43.502 INFO:teuthology.run_tasks:Running task kernel... 2024-08-17T16:49:43.515 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-08-17T16:49:43.515 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-08-17T16:49:43.516 DEBUG:teuthology.orchestra.run.smithi003:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-17T16:49:43.516 DEBUG:teuthology.orchestra.run.smithi053:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-17T16:49:43.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T16:49:43.533 DEBUG:teuthology.orchestra.run.smithi053:> uname -r 2024-08-17T16:49:43.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T16:49:43.534 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2024-08-17T16:49:43.589 INFO:teuthology.orchestra.run.smithi053.stdout:5.14.0-437.el9.x86_64 2024-08-17T16:49:43.590 INFO:teuthology.task.kernel:Running kernel on smithi053: 5.14.0-437.el9.x86_64 2024-08-17T16:49:43.590 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum install -y kernel 2024-08-17T16:49:43.591 INFO:teuthology.orchestra.run.smithi003.stdout:5.14.0-437.el9.x86_64 2024-08-17T16:49:43.591 INFO:teuthology.task.kernel:Running kernel on smithi003: 5.14.0-437.el9.x86_64 2024-08-17T16:49:43.591 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum install -y kernel 2024-08-17T16:49:45.818 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.2 MB 00:00 2024-08-17T16:49:45.885 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.2 MB 00:00 2024-08-17T16:49:47.761 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - AppStream 25 MB/s | 20 MB 00:00 2024-08-17T16:49:48.686 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - AppStream 12 MB/s | 20 MB 00:01 2024-08-17T16:49:52.312 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - CRB 9.8 MB/s | 6.6 MB 00:00 2024-08-17T16:49:53.237 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - CRB 9.7 MB/s | 6.6 MB 00:00 2024-08-17T16:49:54.696 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - Extras packages 17 kB/s | 18 kB 00:01 2024-08-17T16:49:54.721 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - Extras packages 103 kB/s | 18 kB 00:00 2024-08-17T16:49:56.208 INFO:teuthology.orchestra.run.smithi053.stdout:Extra Packages for Enterprise Linux 16 MB/s | 22 MB 00:01 2024-08-17T16:49:56.250 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 16 MB/s | 22 MB 00:01 2024-08-17T16:50:01.814 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-08-17T16:50:01.964 INFO:teuthology.orchestra.run.smithi053.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-08-17T16:50:03.904 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-17T16:50:03.904 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-08-17T16:50:04.002 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-08-17T16:50:04.014 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:50:04.014 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2024-08-17T16:50:04.014 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:50:04.014 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-08-17T16:50:04.014 INFO:teuthology.orchestra.run.smithi003.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-17T16:50:04.014 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-08-17T16:50:04.015 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-core x86_64 5.14.0-480.el9 baseos 17 M 2024-08-17T16:50:04.015 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules x86_64 5.14.0-480.el9 baseos 36 M 2024-08-17T16:50:04.015 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-core x86_64 5.14.0-480.el9 baseos 30 M 2024-08-17T16:50:04.015 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:50:04.015 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-08-17T16:50:04.015 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:50:04.015 INFO:teuthology.orchestra.run.smithi003.stdout:Install 4 Packages 2024-08-17T16:50:04.015 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:50:04.016 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 84 M 2024-08-17T16:50:04.016 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 126 M 2024-08-17T16:50:04.016 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-08-17T16:50:04.028 INFO:teuthology.orchestra.run.smithi053.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-17T16:50:04.028 INFO:teuthology.orchestra.run.smithi053.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-08-17T16:50:04.128 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-08-17T16:50:04.133 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:50:04.133 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-08-17T16:50:04.133 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:50:04.133 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-08-17T16:50:04.133 INFO:teuthology.orchestra.run.smithi053.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-17T16:50:04.133 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-08-17T16:50:04.133 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-core x86_64 5.14.0-480.el9 baseos 17 M 2024-08-17T16:50:04.133 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules x86_64 5.14.0-480.el9 baseos 36 M 2024-08-17T16:50:04.133 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules-core x86_64 5.14.0-480.el9 baseos 30 M 2024-08-17T16:50:04.133 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:50:04.133 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-08-17T16:50:04.134 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:50:04.134 INFO:teuthology.orchestra.run.smithi053.stdout:Install 4 Packages 2024-08-17T16:50:04.134 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:50:04.134 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 84 M 2024-08-17T16:50:04.134 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 126 M 2024-08-17T16:50:04.134 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-08-17T16:50:04.640 INFO:teuthology.orchestra.run.smithi053.stdout:(1/4): kernel-5.14.0-480.el9.x86_64.rpm 6.5 MB/s | 1.5 MB 00:00 2024-08-17T16:50:05.032 INFO:teuthology.orchestra.run.smithi053.stdout:(2/4): kernel-core-5.14.0-480.el9.x86_64.rpm 27 MB/s | 17 MB 00:00 2024-08-17T16:50:05.082 INFO:teuthology.orchestra.run.smithi003.stdout:(1/4): kernel-5.14.0-480.el9.x86_64.rpm 1.5 MB/s | 1.5 MB 00:01 2024-08-17T16:50:05.524 INFO:teuthology.orchestra.run.smithi053.stdout:(3/4): kernel-modules-5.14.0-480.el9.x86_64.rpm 32 MB/s | 36 MB 00:01 2024-08-17T16:50:05.799 INFO:teuthology.orchestra.run.smithi053.stdout:(4/4): kernel-modules-core-5.14.0-480.el9.x86_6 26 MB/s | 30 MB 00:01 2024-08-17T16:50:05.799 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-08-17T16:50:05.799 INFO:teuthology.orchestra.run.smithi053.stdout:Total 51 MB/s | 84 MB 00:01 2024-08-17T16:50:06.565 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-08-17T16:50:06.976 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-08-17T16:50:06.976 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-08-17T16:50:07.949 INFO:teuthology.orchestra.run.smithi003.stdout:(2/4): kernel-core-5.14.0-480.el9.x86_64.rpm 4.4 MB/s | 17 MB 00:03 2024-08-17T16:50:08.041 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-08-17T16:50:08.042 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-08-17T16:50:10.642 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-08-17T16:50:11.290 INFO:teuthology.orchestra.run.smithi003.stdout:(3/4): kernel-modules-5.14.0-480.el9.x86_64.rpm 5.0 MB/s | 36 MB 00:07 2024-08-17T16:50:11.350 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-modules-core-5.14.0-480.el9.x86_64 1/4 2024-08-17T16:50:11.441 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-17T16:50:14.032 INFO:teuthology.orchestra.run.smithi003.stdout:(4/4): kernel-modules-core-5.14.0-480.el9.x86_6 3.3 MB/s | 30 MB 00:08 2024-08-17T16:50:14.032 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-08-17T16:50:14.032 INFO:teuthology.orchestra.run.smithi003.stdout:Total 8.4 MB/s | 84 MB 00:10 2024-08-17T16:50:14.256 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-17T16:50:14.395 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-17T16:50:14.796 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-08-17T16:50:15.209 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-08-17T16:50:15.209 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-08-17T16:50:16.230 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-08-17T16:50:16.230 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-08-17T16:50:18.815 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-08-17T16:50:19.527 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-modules-core-5.14.0-480.el9.x86_64 1/4 2024-08-17T16:50:19.603 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-17T16:50:19.657 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-17T16:50:19.663 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-5.14.0-480.el9.x86_64 4/4 2024-08-17T16:50:22.478 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-17T16:50:22.618 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-17T16:50:24.890 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-17T16:50:27.836 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-17T16:50:27.928 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-5.14.0-480.el9.x86_64 4/4 2024-08-17T16:50:33.155 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-17T16:50:56.609 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 4/4 2024-08-17T16:50:56.651 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 4/4 2024-08-17T16:50:57.016 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 4/4 2024-08-17T16:50:57.016 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/4 2024-08-17T16:50:57.016 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-17T16:50:57.016 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-17T16:50:57.768 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-17T16:50:57.768 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:50:57.768 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-08-17T16:50:57.768 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-5.14.0-480.el9.x86_64 2024-08-17T16:50:57.768 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-core-5.14.0-480.el9.x86_64 2024-08-17T16:50:57.768 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules-5.14.0-480.el9.x86_64 2024-08-17T16:50:57.768 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules-core-5.14.0-480.el9.x86_64 2024-08-17T16:50:57.768 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:50:57.768 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-08-17T16:50:58.139 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q kernel | sort -rV | head -n 1 2024-08-17T16:50:58.222 INFO:teuthology.orchestra.run.smithi053.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-17T16:50:58.223 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-17T16:50:58.224 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-480.el9.x86_64 2024-08-17T16:50:58.224 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-08-17T16:50:58.224 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-08-17T16:50:58.224 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-08-17T16:50:58.224 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi053.front.sepia.ceph.com, path=None, version=distro) 2024-08-17T16:50:58.224 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q kernel | sort -rV | head -n 1 2024-08-17T16:50:58.282 INFO:teuthology.orchestra.run.smithi053.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-17T16:50:58.282 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-17T16:50:58.282 DEBUG:teuthology.orchestra.run.smithi053:> sudo rpm -qi grub2-tools 2024-08-17T16:50:58.321 INFO:teuthology.orchestra.run.smithi053.stdout:Name : grub2-tools 2024-08-17T16:50:58.321 INFO:teuthology.orchestra.run.smithi053.stdout:Epoch : 1 2024-08-17T16:50:58.321 INFO:teuthology.orchestra.run.smithi053.stdout:Version : 2.06 2024-08-17T16:50:58.321 INFO:teuthology.orchestra.run.smithi053.stdout:Release : 68.el9 2024-08-17T16:50:58.321 INFO:teuthology.orchestra.run.smithi053.stdout:Architecture: x86_64 2024-08-17T16:50:58.321 INFO:teuthology.orchestra.run.smithi053.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-08-17T16:50:58.321 INFO:teuthology.orchestra.run.smithi053.stdout:Group : Unspecified 2024-08-17T16:50:58.321 INFO:teuthology.orchestra.run.smithi053.stdout:Size : 8264254 2024-08-17T16:50:58.322 INFO:teuthology.orchestra.run.smithi053.stdout:License : GPLv3+ 2024-08-17T16:50:58.322 INFO:teuthology.orchestra.run.smithi053.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-08-17T16:50:58.322 INFO:teuthology.orchestra.run.smithi053.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-08-17T16:50:58.322 INFO:teuthology.orchestra.run.smithi053.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-08-17T16:50:58.322 INFO:teuthology.orchestra.run.smithi053.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-08-17T16:50:58.322 INFO:teuthology.orchestra.run.smithi053.stdout:Packager : builder@centos.org 2024-08-17T16:50:58.322 INFO:teuthology.orchestra.run.smithi053.stdout:Vendor : CentOS 2024-08-17T16:50:58.322 INFO:teuthology.orchestra.run.smithi053.stdout:URL : http://www.gnu.org/software/grub/ 2024-08-17T16:50:58.322 INFO:teuthology.orchestra.run.smithi053.stdout:Summary : Support tools for GRUB. 2024-08-17T16:50:58.322 INFO:teuthology.orchestra.run.smithi053.stdout:Description : 2024-08-17T16:50:58.322 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:50:58.322 INFO:teuthology.orchestra.run.smithi053.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-08-17T16:50:58.322 INFO:teuthology.orchestra.run.smithi053.stdout:customizable bootloader with modular architecture. It supports a rich 2024-08-17T16:50:58.323 INFO:teuthology.orchestra.run.smithi053.stdout:variety of kernel formats, file systems, computer architectures and 2024-08-17T16:50:58.323 INFO:teuthology.orchestra.run.smithi053.stdout:hardware devices. 2024-08-17T16:50:58.323 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:50:58.323 INFO:teuthology.orchestra.run.smithi053.stdout:This subpackage provides tools for support of all platforms. 2024-08-17T16:50:58.324 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-08-17T16:50:58.324 INFO:teuthology.task.kernel:Updating grub on smithi053 to boot 5.14.0-480.el9.x86_64 2024-08-17T16:50:58.324 DEBUG:teuthology.orchestra.run.smithi053:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-08-17T16:50:58.883 INFO:teuthology.orchestra.run.smithi053.stderr:Generating grub configuration file ... 2024-08-17T16:51:00.268 INFO:teuthology.orchestra.run.smithi053.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-08-17T16:51:00.305 INFO:teuthology.orchestra.run.smithi053.stderr:done 2024-08-17T16:51:00.307 DEBUG:teuthology.orchestra.run.smithi053:> mktemp 2024-08-17T16:51:00.322 INFO:teuthology.orchestra.run.smithi053.stdout:/tmp/tmp.bT2dJYXNXq 2024-08-17T16:51:00.322 DEBUG:teuthology.orchestra.run.smithi053:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.bT2dJYXNXq 2024-08-17T16:51:00.397 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0666 /tmp/tmp.bT2dJYXNXq 2024-08-17T16:51:00.532 DEBUG:teuthology.orchestra.remote:smithi053:/tmp/tmp.bT2dJYXNXq is 6KB 2024-08-17T16:51:00.580 DEBUG:teuthology.orchestra.run.smithi053:> rm -fr /tmp/tmp.bT2dJYXNXq 2024-08-17T16:51:00.595 DEBUG:teuthology.orchestra.run.smithi053:> sudo /bin/ls /boot/loader/entries || true 2024-08-17T16:51:00.665 INFO:teuthology.orchestra.run.smithi053.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-08-17T16:51:00.665 INFO:teuthology.orchestra.run.smithi053.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-08-17T16:51:00.665 INFO:teuthology.orchestra.run.smithi053.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-08-17T16:51:00.665 INFO:teuthology.orchestra.run.smithi053.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64.conf 2024-08-17T16:51:00.667 DEBUG:teuthology.orchestra.run.smithi053:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64 2024-08-17T16:51:00.819 DEBUG:teuthology.orchestra.run.smithi053:> sudo shutdown -r now 2024-08-17T16:51:04.321 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 4/4 2024-08-17T16:51:04.364 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 4/4 2024-08-17T16:51:04.702 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 4/4 2024-08-17T16:51:04.702 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/4 2024-08-17T16:51:04.702 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-17T16:51:04.702 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-17T16:51:05.504 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-17T16:51:05.504 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:51:05.504 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-08-17T16:51:05.504 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-5.14.0-480.el9.x86_64 2024-08-17T16:51:05.504 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-core-5.14.0-480.el9.x86_64 2024-08-17T16:51:05.504 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-5.14.0-480.el9.x86_64 2024-08-17T16:51:05.504 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-core-5.14.0-480.el9.x86_64 2024-08-17T16:51:05.504 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:51:05.504 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-08-17T16:51:05.878 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q kernel | sort -rV | head -n 1 2024-08-17T16:51:05.962 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-17T16:51:05.962 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-17T16:51:05.962 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-480.el9.x86_64 2024-08-17T16:51:05.962 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-08-17T16:51:05.962 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-08-17T16:51:05.962 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-08-17T16:51:05.962 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi003.front.sepia.ceph.com, path=None, version=distro) 2024-08-17T16:51:05.963 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q kernel | sort -rV | head -n 1 2024-08-17T16:51:06.021 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-17T16:51:06.021 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-17T16:51:06.022 DEBUG:teuthology.orchestra.run.smithi003:> sudo rpm -qi grub2-tools 2024-08-17T16:51:06.064 INFO:teuthology.orchestra.run.smithi003.stdout:Name : grub2-tools 2024-08-17T16:51:06.064 INFO:teuthology.orchestra.run.smithi003.stdout:Epoch : 1 2024-08-17T16:51:06.064 INFO:teuthology.orchestra.run.smithi003.stdout:Version : 2.06 2024-08-17T16:51:06.064 INFO:teuthology.orchestra.run.smithi003.stdout:Release : 68.el9 2024-08-17T16:51:06.064 INFO:teuthology.orchestra.run.smithi003.stdout:Architecture: x86_64 2024-08-17T16:51:06.064 INFO:teuthology.orchestra.run.smithi003.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-08-17T16:51:06.064 INFO:teuthology.orchestra.run.smithi003.stdout:Group : Unspecified 2024-08-17T16:51:06.064 INFO:teuthology.orchestra.run.smithi003.stdout:Size : 8264254 2024-08-17T16:51:06.064 INFO:teuthology.orchestra.run.smithi003.stdout:License : GPLv3+ 2024-08-17T16:51:06.064 INFO:teuthology.orchestra.run.smithi003.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-08-17T16:51:06.064 INFO:teuthology.orchestra.run.smithi003.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-08-17T16:51:06.065 INFO:teuthology.orchestra.run.smithi003.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-08-17T16:51:06.065 INFO:teuthology.orchestra.run.smithi003.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-08-17T16:51:06.065 INFO:teuthology.orchestra.run.smithi003.stdout:Packager : builder@centos.org 2024-08-17T16:51:06.065 INFO:teuthology.orchestra.run.smithi003.stdout:Vendor : CentOS 2024-08-17T16:51:06.065 INFO:teuthology.orchestra.run.smithi003.stdout:URL : http://www.gnu.org/software/grub/ 2024-08-17T16:51:06.065 INFO:teuthology.orchestra.run.smithi003.stdout:Summary : Support tools for GRUB. 2024-08-17T16:51:06.065 INFO:teuthology.orchestra.run.smithi003.stdout:Description : 2024-08-17T16:51:06.065 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:51:06.065 INFO:teuthology.orchestra.run.smithi003.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-08-17T16:51:06.065 INFO:teuthology.orchestra.run.smithi003.stdout:customizable bootloader with modular architecture. It supports a rich 2024-08-17T16:51:06.065 INFO:teuthology.orchestra.run.smithi003.stdout:variety of kernel formats, file systems, computer architectures and 2024-08-17T16:51:06.065 INFO:teuthology.orchestra.run.smithi003.stdout:hardware devices. 2024-08-17T16:51:06.065 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:51:06.066 INFO:teuthology.orchestra.run.smithi003.stdout:This subpackage provides tools for support of all platforms. 2024-08-17T16:51:06.066 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-08-17T16:51:06.067 INFO:teuthology.task.kernel:Updating grub on smithi003 to boot 5.14.0-480.el9.x86_64 2024-08-17T16:51:06.067 DEBUG:teuthology.orchestra.run.smithi003:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-08-17T16:51:06.789 INFO:teuthology.orchestra.run.smithi003.stderr:Generating grub configuration file ... 2024-08-17T16:51:08.208 INFO:teuthology.orchestra.run.smithi003.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-08-17T16:51:08.226 INFO:teuthology.orchestra.run.smithi003.stderr:done 2024-08-17T16:51:08.228 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2024-08-17T16:51:08.285 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.70O2bWqZbP 2024-08-17T16:51:08.285 DEBUG:teuthology.orchestra.run.smithi003:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.70O2bWqZbP 2024-08-17T16:51:08.351 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.70O2bWqZbP 2024-08-17T16:51:08.494 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.70O2bWqZbP is 6KB 2024-08-17T16:51:08.545 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.70O2bWqZbP 2024-08-17T16:51:08.559 DEBUG:teuthology.orchestra.run.smithi003:> sudo /bin/ls /boot/loader/entries || true 2024-08-17T16:51:08.622 INFO:teuthology.orchestra.run.smithi003.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-08-17T16:51:08.622 INFO:teuthology.orchestra.run.smithi003.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-08-17T16:51:08.622 INFO:teuthology.orchestra.run.smithi003.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-08-17T16:51:08.622 INFO:teuthology.orchestra.run.smithi003.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64.conf 2024-08-17T16:51:08.623 DEBUG:teuthology.orchestra.run.smithi003:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64 2024-08-17T16:51:08.774 DEBUG:teuthology.orchestra.run.smithi003:> sudo shutdown -r now 2024-08-17T16:51:30.842 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-08-17T16:51:30.843 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-08-17T16:51:30.844 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-17T16:51:38.780 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-08-17T16:51:38.780 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-08-17T16:51:38.781 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-17T16:51:49.343 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-08-17T16:51:57.279 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.3 2024-08-17T16:51:58.345 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-08-17T16:51:58.346 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-17T16:52:06.286 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-08-17T16:52:06.287 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-17T16:52:16.735 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-08-17T16:52:28.746 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-08-17T16:52:28.748 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-17T16:52:29.273 DEBUG:teuthology.orchestra.run.smithi053:> true 2024-08-17T16:52:29.686 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-08-17T16:52:29.686 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-480.el9.x86_64"... 2024-08-17T16:52:29.687 DEBUG:teuthology.orchestra.run.smithi053:> uname -r 2024-08-17T16:52:29.702 INFO:teuthology.orchestra.run.smithi053.stdout:5.14.0-480.el9.x86_64 2024-08-17T16:52:29.702 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-480.el9.x86_64 vs 5.14.0-480.el9.x86_64 2024-08-17T16:52:29.702 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-08-17T16:52:29.702 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-08-17T16:52:30.703 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-08-17T16:52:30.703 DEBUG:teuthology.orchestra.run.smithi053:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-17T16:52:30.771 INFO:teuthology.orchestra.run.smithi053.stdout:ttyS1 2024-08-17T16:52:30.806 DEBUG:teuthology.parallel:result is None 2024-08-17T16:53:06.298 DEBUG:teuthology.orchestra.remote:timed out 2024-08-17T16:53:18.298 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-08-17T16:53:18.300 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-17T16:53:18.634 DEBUG:teuthology.orchestra.run.smithi003:> true 2024-08-17T16:53:19.002 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-08-17T16:53:19.002 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-480.el9.x86_64"... 2024-08-17T16:53:19.003 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2024-08-17T16:53:19.019 INFO:teuthology.orchestra.run.smithi003.stdout:5.14.0-480.el9.x86_64 2024-08-17T16:53:19.020 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-480.el9.x86_64 vs 5.14.0-480.el9.x86_64 2024-08-17T16:53:19.020 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-08-17T16:53:19.020 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-08-17T16:53:20.021 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-08-17T16:53:20.021 DEBUG:teuthology.orchestra.run.smithi003:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-17T16:53:20.102 INFO:teuthology.orchestra.run.smithi003.stdout:ttyS1 2024-08-17T16:53:20.138 DEBUG:teuthology.parallel:result is None 2024-08-17T16:53:20.138 INFO:teuthology.run_tasks:Running task internal.base... 2024-08-17T16:53:20.145 INFO:teuthology.task.internal:Creating test directory... 2024-08-17T16:53:20.145 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-17T16:53:20.148 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-17T16:53:20.170 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-08-17T16:53:20.177 INFO:teuthology.run_tasks:Running task internal.archive... 2024-08-17T16:53:20.186 INFO:teuthology.task.internal:Creating archive directory... 2024-08-17T16:53:20.186 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-17T16:53:20.208 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-17T16:53:20.249 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-08-17T16:53:20.256 INFO:teuthology.task.internal:Enabling coredump saving... 2024-08-17T16:53:20.257 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-17T16:53:20.278 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-17T16:53:20.310 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-17T16:53:20.323 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-17T16:53:20.323 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-17T16:53:20.335 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-17T16:53:20.337 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-08-17T16:53:20.343 INFO:teuthology.task.internal:Configuring sudo... 2024-08-17T16:53:20.343 DEBUG:teuthology.orchestra.run.smithi003:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-17T16:53:20.371 DEBUG:teuthology.orchestra.run.smithi053:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-17T16:53:20.405 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-08-17T16:53:20.416 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-08-17T16:53:20.416 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-17T16:53:20.438 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-17T16:53:20.463 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-17T16:53:20.545 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-17T16:53:20.593 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-17T16:53:20.681 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-17T16:53:20.720 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-08-17T16:53:20.721 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-17T16:53:20.789 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-17T16:53:20.842 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-17T16:53:20.884 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-17T16:53:20.969 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-17T16:53:21.013 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-08-17T16:53:21.013 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-17T16:53:21.081 DEBUG:teuthology.orchestra.run.smithi003:> sudo service rsyslog restart 2024-08-17T16:53:21.084 DEBUG:teuthology.orchestra.run.smithi053:> sudo service rsyslog restart 2024-08-17T16:53:21.131 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-17T16:53:21.169 INFO:teuthology.orchestra.run.smithi053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-17T16:53:21.565 INFO:teuthology.run_tasks:Running task internal.timer... 2024-08-17T16:53:21.574 INFO:teuthology.task.internal:Starting timer... 2024-08-17T16:53:21.574 INFO:teuthology.run_tasks:Running task pcp... 2024-08-17T16:53:21.588 INFO:teuthology.run_tasks:Running task selinux... 2024-08-17T16:53:21.601 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-08-17T16:53:21.601 DEBUG:teuthology.orchestra.run.smithi003:> sudo service auditd rotate 2024-08-17T16:53:21.663 INFO:teuthology.orchestra.run.smithi003.stdout:Rotating logs: 2024-08-17T16:53:21.666 DEBUG:teuthology.orchestra.run.smithi053:> sudo service auditd rotate 2024-08-17T16:53:21.724 INFO:teuthology.orchestra.run.smithi053.stdout:Rotating logs: 2024-08-17T16:53:21.726 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-08-17T16:53:21.726 DEBUG:teuthology.orchestra.run.smithi003:> /usr/sbin/getenforce 2024-08-17T16:53:21.758 INFO:teuthology.orchestra.run.smithi003.stdout:Permissive 2024-08-17T16:53:21.758 DEBUG:teuthology.orchestra.run.smithi053:> /usr/sbin/getenforce 2024-08-17T16:53:21.794 INFO:teuthology.orchestra.run.smithi053.stdout:Permissive 2024-08-17T16:53:21.794 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi003.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi053.front.sepia.ceph.com': 'permissive'} 2024-08-17T16:53:21.795 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-17T16:53:21.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T16:53:21.827 DEBUG:teuthology.orchestra.run.smithi053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-17T16:53:21.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T16:53:21.864 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-08-17T16:53:21.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo /usr/sbin/setenforce permissive 2024-08-17T16:53:21.895 DEBUG:teuthology.orchestra.run.smithi053:> sudo /usr/sbin/setenforce permissive 2024-08-17T16:53:21.933 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-08-17T16:53:21.944 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-08-17T16:53:21.948 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-08-17T16:53:22.069 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-08-17T16:53:22.080 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-08-17T16:53:22.126 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi003.front.sepia.ceph.com,smithi053.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-08-17T16:58:01.567 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi003.front.sepia.ceph.com'), Remote(name='ubuntu@smithi053.front.sepia.ceph.com')] 2024-08-17T16:58:01.569 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-08-17T16:58:01.571 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-17T16:58:01.657 DEBUG:teuthology.orchestra.run.smithi003:> true 2024-08-17T16:58:01.747 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-08-17T16:58:01.747 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-08-17T16:58:01.748 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-17T16:58:01.829 DEBUG:teuthology.orchestra.run.smithi053:> true 2024-08-17T16:58:01.907 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-08-17T16:58:01.908 INFO:teuthology.run_tasks:Running task clock... 2024-08-17T16:58:01.920 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-08-17T16:58:01.920 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-17T16:58:01.920 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-17T16:58:01.923 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-17T16:58:01.923 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-17T16:58:01.950 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-17T16:58:01.965 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-17T16:58:01.979 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-17T16:58:01.990 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: ntpd: command not found 2024-08-17T16:58:01.995 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-17T16:58:02.001 INFO:teuthology.orchestra.run.smithi003.stdout:506 Cannot talk to daemon 2024-08-17T16:58:02.015 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-17T16:58:02.021 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: ntpd: command not found 2024-08-17T16:58:02.030 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-17T16:58:02.033 INFO:teuthology.orchestra.run.smithi053.stdout:506 Cannot talk to daemon 2024-08-17T16:58:02.048 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-17T16:58:02.062 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-17T16:58:02.096 INFO:teuthology.orchestra.run.smithi003.stderr:bash: line 1: ntpq: command not found 2024-08-17T16:58:02.100 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-17T16:58:02.100 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2024-08-17T16:58:02.100 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-17T16:58:02.100 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-17T16:58:02.100 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-17T16:58:02.100 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-17T16:58:02.119 INFO:teuthology.orchestra.run.smithi053.stderr:bash: line 1: ntpq: command not found 2024-08-17T16:58:02.122 INFO:teuthology.orchestra.run.smithi053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-17T16:58:02.123 INFO:teuthology.orchestra.run.smithi053.stdout:=============================================================================== 2024-08-17T16:58:02.123 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-17T16:58:02.123 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-17T16:58:02.123 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-17T16:58:02.123 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-17T16:58:02.124 INFO:teuthology.run_tasks:Running task pexec... 2024-08-17T16:58:02.133 INFO:teuthology.task.pexec:Executing custom commands... 2024-08-17T16:58:02.134 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi003.front.sepia.ceph.com 2024-08-17T16:58:02.134 DEBUG:teuthology.orchestra.run.smithi003:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-08-17T16:58:02.134 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi053.front.sepia.ceph.com 2024-08-17T16:58:02.134 DEBUG:teuthology.orchestra.run.smithi053:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-08-17T16:58:02.737 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:08 ago on Sat 17 Aug 2024 04:56:54 PM UTC. 2024-08-17T16:58:02.759 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:08 ago on Sat 17 Aug 2024 04:56:54 PM UTC. 2024-08-17T16:58:02.824 INFO:teuthology.orchestra.run.smithi003.stdout:Package nvme-cli-2.6-5.el9.x86_64 is already installed. 2024-08-17T16:58:02.845 INFO:teuthology.orchestra.run.smithi053.stdout:Package nvme-cli-2.6-5.el9.x86_64 is already installed. 2024-08-17T16:58:02.867 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-08-17T16:58:02.880 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:58:02.881 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2024-08-17T16:58:02.881 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:58:02.881 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-08-17T16:58:02.881 INFO:teuthology.orchestra.run.smithi003.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-08-17T16:58:02.881 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrading: 2024-08-17T16:58:02.881 INFO:teuthology.orchestra.run.smithi003.stdout: libnvme x86_64 1.9-2.el9 baseos 106 k 2024-08-17T16:58:02.881 INFO:teuthology.orchestra.run.smithi003.stdout: nvme-cli x86_64 2.9.1-3.el9 baseos 903 k 2024-08-17T16:58:02.881 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-08-17T16:58:02.881 INFO:teuthology.orchestra.run.smithi003.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-08-17T16:58:02.881 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-08-17T16:58:02.881 INFO:teuthology.orchestra.run.smithi003.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-08-17T16:58:02.881 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:58:02.881 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-08-17T16:58:02.881 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:58:02.882 INFO:teuthology.orchestra.run.smithi003.stdout:Install 4 Packages 2024-08-17T16:58:02.882 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrade 2 Packages 2024-08-17T16:58:02.882 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:58:02.882 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 2.0 M 2024-08-17T16:58:02.882 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-08-17T16:58:02.889 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-08-17T16:58:02.894 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:58:02.894 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-08-17T16:58:02.894 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:58:02.894 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-08-17T16:58:02.894 INFO:teuthology.orchestra.run.smithi053.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-08-17T16:58:02.895 INFO:teuthology.orchestra.run.smithi053.stdout:Upgrading: 2024-08-17T16:58:02.895 INFO:teuthology.orchestra.run.smithi053.stdout: libnvme x86_64 1.9-2.el9 baseos 106 k 2024-08-17T16:58:02.895 INFO:teuthology.orchestra.run.smithi053.stdout: nvme-cli x86_64 2.9.1-3.el9 baseos 903 k 2024-08-17T16:58:02.895 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-08-17T16:58:02.895 INFO:teuthology.orchestra.run.smithi053.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-08-17T16:58:02.895 INFO:teuthology.orchestra.run.smithi053.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-08-17T16:58:02.895 INFO:teuthology.orchestra.run.smithi053.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-08-17T16:58:02.895 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:58:02.895 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-08-17T16:58:02.895 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:58:02.895 INFO:teuthology.orchestra.run.smithi053.stdout:Install 4 Packages 2024-08-17T16:58:02.896 INFO:teuthology.orchestra.run.smithi053.stdout:Upgrade 2 Packages 2024-08-17T16:58:02.896 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:58:02.896 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 2.0 M 2024-08-17T16:58:02.896 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-08-17T16:58:03.117 INFO:teuthology.orchestra.run.smithi003.stdout:(1/6): nvmetcli-0.7-3.el9.noarch.rpm 376 kB/s | 44 kB 00:00 2024-08-17T16:58:03.134 INFO:teuthology.orchestra.run.smithi003.stdout:(2/6): python3-configshell-1.1.30-1.el9.noarch. 531 kB/s | 72 kB 00:00 2024-08-17T16:58:03.159 INFO:teuthology.orchestra.run.smithi003.stdout:(3/6): python3-kmod-0.9-32.el9.x86_64.rpm 526 kB/s | 84 kB 00:00 2024-08-17T16:58:03.202 INFO:teuthology.orchestra.run.smithi003.stdout:(4/6): libnvme-1.9-2.el9.x86_64.rpm 1.5 MB/s | 106 kB 00:00 2024-08-17T16:58:03.285 INFO:teuthology.orchestra.run.smithi003.stdout:(5/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 4.9 MB/s | 837 kB 00:00 2024-08-17T16:58:03.319 INFO:teuthology.orchestra.run.smithi003.stdout:(6/6): nvme-cli-2.9.1-3.el9.x86_64.rpm 5.5 MB/s | 903 kB 00:00 2024-08-17T16:58:03.319 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-08-17T16:58:03.319 INFO:teuthology.orchestra.run.smithi003.stdout:Total 4.6 MB/s | 2.0 MB 00:00 2024-08-17T16:58:03.395 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-08-17T16:58:03.411 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-08-17T16:58:03.411 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-08-17T16:58:03.615 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-08-17T16:58:03.615 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-08-17T16:58:03.644 INFO:teuthology.orchestra.run.smithi053.stdout:(1/6): nvmetcli-0.7-3.el9.noarch.rpm 72 kB/s | 44 kB 00:00 2024-08-17T16:58:03.703 INFO:teuthology.orchestra.run.smithi053.stdout:(2/6): python3-configshell-1.1.30-1.el9.noarch. 106 kB/s | 72 kB 00:00 2024-08-17T16:58:03.761 INFO:teuthology.orchestra.run.smithi053.stdout:(3/6): python3-kmod-0.9-32.el9.x86_64.rpm 115 kB/s | 84 kB 00:00 2024-08-17T16:58:03.978 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-08-17T16:58:04.188 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : libnvme-1.9-2.el9.x86_64 1/8 2024-08-17T16:58:04.250 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 2/8 2024-08-17T16:58:04.284 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 3/8 2024-08-17T16:58:04.317 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 4/8 2024-08-17T16:58:04.341 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : nvmetcli-0.7-3.el9.noarch 5/8 2024-08-17T16:58:04.386 INFO:teuthology.orchestra.run.smithi053.stdout:(4/6): libnvme-1.9-2.el9.x86_64.rpm 155 kB/s | 106 kB 00:00 2024-08-17T16:58:04.553 INFO:teuthology.orchestra.run.smithi053.stdout:(5/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 922 kB/s | 837 kB 00:00 2024-08-17T16:58:04.648 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 5/8 2024-08-17T16:58:04.670 INFO:teuthology.orchestra.run.smithi053.stdout:(6/6): nvme-cli-2.9.1-3.el9.x86_64.rpm 995 kB/s | 903 kB 00:00 2024-08-17T16:58:04.670 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-08-17T16:58:04.670 INFO:teuthology.orchestra.run.smithi053.stdout:Total 1.1 MB/s | 2.0 MB 00:01 2024-08-17T16:58:04.719 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : nvme-cli-2.9.1-3.el9.x86_64 6/8 2024-08-17T16:58:04.753 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-08-17T16:58:04.771 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-08-17T16:58:04.771 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-08-17T16:58:05.006 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-08-17T16:58:05.006 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-08-17T16:58:05.052 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: nvme-cli-2.9.1-3.el9.x86_64 6/8 2024-08-17T16:58:05.091 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : nvme-cli-2.6-5.el9.x86_64 7/8 2024-08-17T16:58:05.179 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : libnvme-1.6-1.el9.x86_64 8/8 2024-08-17T16:58:05.399 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-08-17T16:58:05.621 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : libnvme-1.9-2.el9.x86_64 1/8 2024-08-17T16:58:05.676 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 2/8 2024-08-17T16:58:05.710 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 3/8 2024-08-17T16:58:05.743 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 4/8 2024-08-17T16:58:05.767 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : nvmetcli-0.7-3.el9.noarch 5/8 2024-08-17T16:58:06.037 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 5/8 2024-08-17T16:58:06.062 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libnvme-1.6-1.el9.x86_64 8/8 2024-08-17T16:58:06.062 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 1/8 2024-08-17T16:58:06.062 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 2/8 2024-08-17T16:58:06.063 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 3/8 2024-08-17T16:58:06.063 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 4/8 2024-08-17T16:58:06.063 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libnvme-1.9-2.el9.x86_64 5/8 2024-08-17T16:58:06.063 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libnvme-1.6-1.el9.x86_64 6/8 2024-08-17T16:58:06.063 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : nvme-cli-2.9.1-3.el9.x86_64 7/8 2024-08-17T16:58:06.112 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : nvme-cli-2.9.1-3.el9.x86_64 6/8 2024-08-17T16:58:06.413 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : nvme-cli-2.6-5.el9.x86_64 8/8 2024-08-17T16:58:06.413 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:58:06.413 INFO:teuthology.orchestra.run.smithi003.stdout:Upgraded: 2024-08-17T16:58:06.413 INFO:teuthology.orchestra.run.smithi003.stdout: libnvme-1.9-2.el9.x86_64 nvme-cli-2.9.1-3.el9.x86_64 2024-08-17T16:58:06.413 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-08-17T16:58:06.413 INFO:teuthology.orchestra.run.smithi003.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-08-17T16:58:06.413 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-08-17T16:58:06.413 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:58:06.414 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-08-17T16:58:06.439 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: nvme-cli-2.9.1-3.el9.x86_64 6/8 2024-08-17T16:58:06.476 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : nvme-cli-2.6-5.el9.x86_64 7/8 2024-08-17T16:58:06.569 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : libnvme-1.6-1.el9.x86_64 8/8 2024-08-17T16:58:06.745 DEBUG:teuthology.parallel:result is None 2024-08-17T16:58:07.426 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libnvme-1.6-1.el9.x86_64 8/8 2024-08-17T16:58:07.426 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 1/8 2024-08-17T16:58:07.426 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 2/8 2024-08-17T16:58:07.426 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 3/8 2024-08-17T16:58:07.426 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 4/8 2024-08-17T16:58:07.426 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libnvme-1.9-2.el9.x86_64 5/8 2024-08-17T16:58:07.426 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libnvme-1.6-1.el9.x86_64 6/8 2024-08-17T16:58:07.426 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : nvme-cli-2.9.1-3.el9.x86_64 7/8 2024-08-17T16:58:07.769 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : nvme-cli-2.6-5.el9.x86_64 8/8 2024-08-17T16:58:07.769 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:58:07.769 INFO:teuthology.orchestra.run.smithi053.stdout:Upgraded: 2024-08-17T16:58:07.769 INFO:teuthology.orchestra.run.smithi053.stdout: libnvme-1.9-2.el9.x86_64 nvme-cli-2.9.1-3.el9.x86_64 2024-08-17T16:58:07.769 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-08-17T16:58:07.769 INFO:teuthology.orchestra.run.smithi053.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-08-17T16:58:07.769 INFO:teuthology.orchestra.run.smithi053.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-08-17T16:58:07.769 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:58:07.769 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-08-17T16:58:08.093 DEBUG:teuthology.parallel:result is None 2024-08-17T16:58:08.093 INFO:teuthology.run_tasks:Running task install... 2024-08-17T16:58:08.103 DEBUG:teuthology.task.install:project ceph 2024-08-17T16:58:08.103 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'cd8bdbeb9fbdab5733aa3aaadc042c965473e453'}} 2024-08-17T16:58:08.103 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': 'cd8bdbeb9fbdab5733aa3aaadc042c965473e453'} 2024-08-17T16:58:08.103 INFO:teuthology.task.install:Using flavor: default 2024-08-17T16:58:08.109 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-08-17T16:58:08.109 INFO:teuthology.task.install:extra packages: [] 2024-08-17T16:58:08.109 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'cd8bdbeb9fbdab5733aa3aaadc042c965473e453', 'tag': None, 'wait_for_package': False} 2024-08-17T16:58:08.110 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-17T16:58:08.110 INFO:teuthology.packaging:ref: None 2024-08-17T16:58:08.110 INFO:teuthology.packaging:tag: None 2024-08-17T16:58:08.110 INFO:teuthology.packaging:branch: quincy 2024-08-17T16:58:08.110 INFO:teuthology.packaging:sha1: cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T16:58:08.113 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-08-17T16:58:08.115 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'cd8bdbeb9fbdab5733aa3aaadc042c965473e453', 'tag': None, 'wait_for_package': False} 2024-08-17T16:58:08.116 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-17T16:58:08.116 INFO:teuthology.packaging:ref: None 2024-08-17T16:58:08.116 INFO:teuthology.packaging:tag: None 2024-08-17T16:58:08.116 INFO:teuthology.packaging:branch: quincy 2024-08-17T16:58:08.116 INFO:teuthology.packaging:sha1: cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T16:58:08.116 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-08-17T16:58:08.346 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/f2c39d4d5abc084f975be26cc0141421e55727d8/centos/9/flavors/default/ 2024-08-17T16:58:08.346 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1404.gf2c39d4d 2024-08-17T16:58:08.388 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/f2c39d4d5abc084f975be26cc0141421e55727d8/centos/9/flavors/default/ 2024-08-17T16:58:08.389 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1404.gf2c39d4d 2024-08-17T16:58:08.466 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/quincy/f2c39d4d5abc084f975be26cc0141421e55727d8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/f2c39d4d5abc084f975be26cc0141421e55727d8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/f2c39d4d5abc084f975be26cc0141421e55727d8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-17T16:58:08.466 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-08-17T16:58:08.466 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-17T16:58:08.501 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-08-17T16:58:08.501 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-17T16:58:08.501 INFO:teuthology.packaging:ref: None 2024-08-17T16:58:08.501 INFO:teuthology.packaging:tag: None 2024-08-17T16:58:08.501 INFO:teuthology.packaging:branch: quincy 2024-08-17T16:58:08.501 INFO:teuthology.packaging:sha1: cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T16:58:08.502 DEBUG:teuthology.orchestra.run.smithi003:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-17T16:58:08.506 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/quincy/f2c39d4d5abc084f975be26cc0141421e55727d8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/f2c39d4d5abc084f975be26cc0141421e55727d8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/f2c39d4d5abc084f975be26cc0141421e55727d8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-17T16:58:08.506 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-08-17T16:58:08.506 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-17T16:58:08.554 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-08-17T16:58:08.554 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-17T16:58:08.555 INFO:teuthology.packaging:ref: None 2024-08-17T16:58:08.555 INFO:teuthology.packaging:tag: None 2024-08-17T16:58:08.555 INFO:teuthology.packaging:branch: quincy 2024-08-17T16:58:08.555 INFO:teuthology.packaging:sha1: cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T16:58:08.555 DEBUG:teuthology.orchestra.run.smithi053:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-17T16:58:08.572 DEBUG:teuthology.orchestra.run.smithi003:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-17T16:58:08.623 DEBUG:teuthology.orchestra.run.smithi053:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-17T16:58:08.657 DEBUG:teuthology.orchestra.run.smithi003:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-17T16:58:08.701 DEBUG:teuthology.orchestra.run.smithi053:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-17T16:58:08.725 INFO:teuthology.orchestra.run.smithi003.stdout:check_obsoletes = 1 2024-08-17T16:58:08.727 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum clean all 2024-08-17T16:58:08.766 INFO:teuthology.orchestra.run.smithi053.stdout:check_obsoletes = 1 2024-08-17T16:58:08.768 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum clean all 2024-08-17T16:58:08.973 INFO:teuthology.orchestra.run.smithi003.stdout:45 files removed 2024-08-17T16:58:09.001 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-radosgw 2024-08-17T16:58:09.006 INFO:teuthology.orchestra.run.smithi053.stdout:45 files removed 2024-08-17T16:58:09.034 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-radosgw 2024-08-17T16:58:09.610 INFO:teuthology.orchestra.run.smithi003.stdout:ceph packages for x86_64 244 kB/s | 77 kB 00:00 2024-08-17T16:58:09.654 INFO:teuthology.orchestra.run.smithi053.stdout:ceph packages for x86_64 236 kB/s | 77 kB 00:00 2024-08-17T16:58:09.953 INFO:teuthology.orchestra.run.smithi003.stdout:ceph noarch packages 37 kB/s | 11 kB 00:00 2024-08-17T16:58:10.005 INFO:teuthology.orchestra.run.smithi053.stdout:ceph noarch packages 36 kB/s | 11 kB 00:00 2024-08-17T16:58:10.295 INFO:teuthology.orchestra.run.smithi003.stdout:ceph source packages 6.0 kB/s | 1.8 kB 00:00 2024-08-17T16:58:10.347 INFO:teuthology.orchestra.run.smithi053.stdout:ceph source packages 6.1 kB/s | 1.8 kB 00:00 2024-08-17T16:58:11.012 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-08-17T16:58:11.672 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - BaseOS 6.4 MB/s | 8.2 MB 00:01 2024-08-17T16:58:13.297 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - AppStream 17 MB/s | 20 MB 00:01 2024-08-17T16:58:13.556 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - AppStream 27 MB/s | 20 MB 00:00 2024-08-17T16:58:18.191 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - CRB 9.7 MB/s | 6.6 MB 00:00 2024-08-17T16:58:19.005 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.6 MB 00:01 2024-08-17T16:58:19.725 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - Extras packages 87 kB/s | 18 kB 00:00 2024-08-17T16:58:20.598 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - Extras packages 63 kB/s | 18 kB 00:00 2024-08-17T16:58:20.704 INFO:teuthology.orchestra.run.smithi053.stdout:Extra Packages for Enterprise Linux 26 MB/s | 22 MB 00:00 2024-08-17T16:58:21.677 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 24 MB/s | 22 MB 00:00 2024-08-17T16:58:26.369 INFO:teuthology.orchestra.run.smithi053.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-08-17T16:58:27.291 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 45 kB/s | 1.7 kB 00:00 2024-08-17T16:58:28.791 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-08-17T16:58:28.793 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:58:28.793 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-08-17T16:58:28.793 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:58:28.793 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-08-17T16:58:28.793 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-radosgw x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 14 M 2024-08-17T16:58:28.793 INFO:teuthology.orchestra.run.smithi053.stdout:Upgrading: 2024-08-17T16:58:28.793 INFO:teuthology.orchestra.run.smithi053.stdout: librados2 x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 3.1 M 2024-08-17T16:58:28.793 INFO:teuthology.orchestra.run.smithi053.stdout: librbd1 x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 2.9 M 2024-08-17T16:58:28.793 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-08-17T16:58:28.794 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-base x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 5.0 M 2024-08-17T16:58:28.794 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-common x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 22 M 2024-08-17T16:58:28.794 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-selinux x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 25 k 2024-08-17T16:58:28.794 INFO:teuthology.orchestra.run.smithi053.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-17T16:58:28.794 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs2 x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 637 k 2024-08-17T16:58:28.794 INFO:teuthology.orchestra.run.smithi053.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-17T16:58:28.794 INFO:teuthology.orchestra.run.smithi053.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-17T16:58:28.794 INFO:teuthology.orchestra.run.smithi053.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-17T16:58:28.794 INFO:teuthology.orchestra.run.smithi053.stdout: libradosstriper1 x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 457 k 2024-08-17T16:58:28.794 INFO:teuthology.orchestra.run.smithi053.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-17T16:58:28.794 INFO:teuthology.orchestra.run.smithi053.stdout: librgw2 x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 4.9 M 2024-08-17T16:58:28.794 INFO:teuthology.orchestra.run.smithi053.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-17T16:58:28.794 INFO:teuthology.orchestra.run.smithi053.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-17T16:58:28.795 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-argparse x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 45 k 2024-08-17T16:58:28.795 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-common x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 120 k 2024-08-17T16:58:28.795 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cephfs x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 182 k 2024-08-17T16:58:28.795 INFO:teuthology.orchestra.run.smithi053.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-17T16:58:28.795 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rados x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 368 k 2024-08-17T16:58:28.795 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rbd x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 343 k 2024-08-17T16:58:28.795 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rgw x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 116 k 2024-08-17T16:58:28.795 INFO:teuthology.orchestra.run.smithi053.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-17T16:58:28.795 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:58:28.795 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-08-17T16:58:28.795 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:58:28.795 INFO:teuthology.orchestra.run.smithi053.stdout:Install 22 Packages 2024-08-17T16:58:28.796 INFO:teuthology.orchestra.run.smithi053.stdout:Upgrade 2 Packages 2024-08-17T16:58:28.796 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:58:28.796 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 58 M 2024-08-17T16:58:28.796 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-08-17T16:58:29.578 INFO:teuthology.orchestra.run.smithi053.stdout:(1/24): ceph-base-17.2.7-1404.gf2c39d4d.el9.x86 8.3 MB/s | 5.0 MB 00:00 2024-08-17T16:58:29.636 INFO:teuthology.orchestra.run.smithi053.stdout:(2/24): ceph-selinux-17.2.7-1404.gf2c39d4d.el9. 431 kB/s | 25 kB 00:00 2024-08-17T16:58:29.673 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-08-17T16:58:29.675 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:58:29.675 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-08-17T16:58:29.675 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:58:29.675 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-08-17T16:58:29.675 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 14 M 2024-08-17T16:58:29.675 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrading: 2024-08-17T16:58:29.675 INFO:teuthology.orchestra.run.smithi003.stdout: librados2 x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 3.1 M 2024-08-17T16:58:29.675 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1 x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 2.9 M 2024-08-17T16:58:29.675 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-08-17T16:58:29.675 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 5.0 M 2024-08-17T16:58:29.675 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 22 M 2024-08-17T16:58:29.675 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 25 k 2024-08-17T16:58:29.676 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-17T16:58:29.676 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2 x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 637 k 2024-08-17T16:58:29.676 INFO:teuthology.orchestra.run.smithi003.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-17T16:58:29.676 INFO:teuthology.orchestra.run.smithi003.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-17T16:58:29.676 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-17T16:58:29.676 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1 x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 457 k 2024-08-17T16:58:29.676 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-17T16:58:29.676 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2 x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 4.9 M 2024-08-17T16:58:29.676 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-17T16:58:29.676 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-17T16:58:29.676 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 45 k 2024-08-17T16:58:29.676 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 120 k 2024-08-17T16:58:29.676 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 182 k 2024-08-17T16:58:29.676 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-17T16:58:29.677 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 368 k 2024-08-17T16:58:29.677 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 343 k 2024-08-17T16:58:29.677 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 116 k 2024-08-17T16:58:29.677 INFO:teuthology.orchestra.run.smithi003.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-17T16:58:29.677 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:58:29.677 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-08-17T16:58:29.677 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:58:29.677 INFO:teuthology.orchestra.run.smithi003.stdout:Install 22 Packages 2024-08-17T16:58:29.677 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrade 2 Packages 2024-08-17T16:58:29.677 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:58:29.678 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 58 M 2024-08-17T16:58:29.678 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-08-17T16:58:29.728 INFO:teuthology.orchestra.run.smithi053.stdout:(3/24): libcephfs2-17.2.7-1404.gf2c39d4d.el9.x8 6.8 MB/s | 637 kB 00:00 2024-08-17T16:58:29.804 INFO:teuthology.orchestra.run.smithi053.stdout:(4/24): libradosstriper1-17.2.7-1404.gf2c39d4d. 6.0 MB/s | 457 kB 00:00 2024-08-17T16:58:30.096 INFO:teuthology.orchestra.run.smithi053.stdout:(5/24): ceph-radosgw-17.2.7-1404.gf2c39d4d.el9. 13 MB/s | 14 MB 00:01 2024-08-17T16:58:30.137 INFO:teuthology.orchestra.run.smithi053.stdout:(6/24): python3-ceph-argparse-17.2.7-1404.gf2c3 1.1 MB/s | 45 kB 00:00 2024-08-17T16:58:30.213 INFO:teuthology.orchestra.run.smithi053.stdout:(7/24): librgw2-17.2.7-1404.gf2c39d4d.el9.x86_6 12 MB/s | 4.9 MB 00:00 2024-08-17T16:58:30.238 INFO:teuthology.orchestra.run.smithi053.stdout:(8/24): python3-ceph-common-17.2.7-1404.gf2c39d 1.2 MB/s | 120 kB 00:00 2024-08-17T16:58:30.263 INFO:teuthology.orchestra.run.smithi053.stdout:(9/24): python3-cephfs-17.2.7-1404.gf2c39d4d.el 3.5 MB/s | 182 kB 00:00 2024-08-17T16:58:30.289 INFO:teuthology.orchestra.run.smithi053.stdout:(10/24): python3-rados-17.2.7-1404.gf2c39d4d.el 7.1 MB/s | 368 kB 00:00 2024-08-17T16:58:30.314 INFO:teuthology.orchestra.run.smithi053.stdout:(11/24): python3-rbd-17.2.7-1404.gf2c39d4d.el9. 6.7 MB/s | 343 kB 00:00 2024-08-17T16:58:30.339 INFO:teuthology.orchestra.run.smithi053.stdout:(12/24): python3-rgw-17.2.7-1404.gf2c39d4d.el9. 2.3 MB/s | 116 kB 00:00 2024-08-17T16:58:30.556 INFO:teuthology.orchestra.run.smithi053.stdout:(13/24): ceph-common-17.2.7-1404.gf2c39d4d.el9. 14 MB/s | 22 MB 00:01 2024-08-17T16:58:30.759 INFO:teuthology.orchestra.run.smithi003.stdout:(1/24): ceph-base-17.2.7-1404.gf2c39d4d.el9.x86 8.5 MB/s | 5.0 MB 00:00 2024-08-17T16:58:30.809 INFO:teuthology.orchestra.run.smithi003.stdout:(2/24): ceph-selinux-17.2.7-1404.gf2c39d4d.el9. 504 kB/s | 25 kB 00:00 2024-08-17T16:58:30.815 INFO:teuthology.orchestra.run.smithi053.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 95 kB/s | 45 kB 00:00 2024-08-17T16:58:30.893 INFO:teuthology.orchestra.run.smithi003.stdout:(3/24): libcephfs2-17.2.7-1404.gf2c39d4d.el9.x8 7.5 MB/s | 637 kB 00:00 2024-08-17T16:58:30.968 INFO:teuthology.orchestra.run.smithi003.stdout:(4/24): libradosstriper1-17.2.7-1404.gf2c39d4d. 6.0 MB/s | 457 kB 00:00 2024-08-17T16:58:30.981 INFO:teuthology.orchestra.run.smithi053.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 240 kB/s | 160 kB 00:00 2024-08-17T16:58:31.082 INFO:teuthology.orchestra.run.smithi053.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 422 kB/s | 42 kB 00:00 2024-08-17T16:58:31.107 INFO:teuthology.orchestra.run.smithi053.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.0 MB/s | 292 kB 00:00 2024-08-17T16:58:31.216 INFO:teuthology.orchestra.run.smithi053.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 613 kB/s | 49 kB 00:00 2024-08-17T16:58:31.242 INFO:teuthology.orchestra.run.smithi053.stdout:(19/24): librdkafka-1.6.1-102.el9.x86_64.rpm 967 kB/s | 662 kB 00:00 2024-08-17T16:58:31.275 INFO:teuthology.orchestra.run.smithi053.stdout:(20/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.6 MB/s | 309 kB 00:00 2024-08-17T16:58:31.276 INFO:teuthology.orchestra.run.smithi003.stdout:(5/24): ceph-radosgw-17.2.7-1404.gf2c39d4d.el9. 13 MB/s | 14 MB 00:01 2024-08-17T16:58:31.326 INFO:teuthology.orchestra.run.smithi053.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 619 kB/s | 67 kB 00:00 2024-08-17T16:58:31.327 INFO:teuthology.orchestra.run.smithi003.stdout:(6/24): python3-ceph-argparse-17.2.7-1404.gf2c3 906 kB/s | 45 kB 00:00 2024-08-17T16:58:31.384 INFO:teuthology.orchestra.run.smithi053.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-08-17T16:58:31.427 INFO:teuthology.orchestra.run.smithi003.stdout:(7/24): librgw2-17.2.7-1404.gf2c39d4d.el9.x86_6 11 MB/s | 4.9 MB 00:00 2024-08-17T16:58:31.444 INFO:teuthology.orchestra.run.smithi003.stdout:(8/24): python3-ceph-common-17.2.7-1404.gf2c39d 1.0 MB/s | 120 kB 00:00 2024-08-17T16:58:31.543 INFO:teuthology.orchestra.run.smithi053.stdout:(23/24): librbd1-17.2.7-1404.gf2c39d4d.el9.x86_ 14 MB/s | 2.9 MB 00:00 2024-08-17T16:58:31.609 INFO:teuthology.orchestra.run.smithi053.stdout:(24/24): librados2-17.2.7-1404.gf2c39d4d.el9.x8 10 MB/s | 3.1 MB 00:00 2024-08-17T16:58:31.614 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-08-17T16:58:31.614 INFO:teuthology.orchestra.run.smithi053.stdout:Total 21 MB/s | 58 MB 00:02 2024-08-17T16:58:31.661 INFO:teuthology.orchestra.run.smithi003.stdout:(9/24): ceph-common-17.2.7-1404.gf2c39d4d.el9.x 15 MB/s | 22 MB 00:01 2024-08-17T16:58:31.675 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-08-17T16:58:31.686 INFO:teuthology.orchestra.run.smithi003.stdout:(10/24): python3-cephfs-17.2.7-1404.gf2c39d4d.e 702 kB/s | 182 kB 00:00 2024-08-17T16:58:31.711 INFO:teuthology.orchestra.run.smithi003.stdout:(11/24): python3-rados-17.2.7-1404.gf2c39d4d.el 1.3 MB/s | 368 kB 00:00 2024-08-17T16:58:31.712 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-08-17T16:58:31.712 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-08-17T16:58:31.737 INFO:teuthology.orchestra.run.smithi003.stdout:(12/24): python3-rbd-17.2.7-1404.gf2c39d4d.el9. 4.5 MB/s | 343 kB 00:00 2024-08-17T16:58:31.762 INFO:teuthology.orchestra.run.smithi003.stdout:(13/24): python3-rgw-17.2.7-1404.gf2c39d4d.el9. 1.5 MB/s | 116 kB 00:00 2024-08-17T16:58:31.879 INFO:teuthology.orchestra.run.smithi003.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 958 kB/s | 160 kB 00:00 2024-08-17T16:58:31.904 INFO:teuthology.orchestra.run.smithi003.stdout:(15/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 271 kB/s | 45 kB 00:00 2024-08-17T16:58:31.954 INFO:teuthology.orchestra.run.smithi003.stdout:(16/24): librdkafka-1.6.1-102.el9.x86_64.rpm 3.4 MB/s | 662 kB 00:00 2024-08-17T16:58:31.980 INFO:teuthology.orchestra.run.smithi003.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-08-17T16:58:32.030 INFO:teuthology.orchestra.run.smithi003.stdout:(18/24): python3-prettytable-0.7.2-27.el9.noarc 337 kB/s | 42 kB 00:00 2024-08-17T16:58:32.122 INFO:teuthology.orchestra.run.smithi003.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 428 kB/s | 49 kB 00:00 2024-08-17T16:58:32.147 INFO:teuthology.orchestra.run.smithi003.stdout:(20/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.6 MB/s | 309 kB 00:00 2024-08-17T16:58:32.172 INFO:teuthology.orchestra.run.smithi003.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 602 kB/s | 67 kB 00:00 2024-08-17T16:58:32.198 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-08-17T16:58:32.199 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-08-17T16:58:32.231 INFO:teuthology.orchestra.run.smithi003.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-08-17T16:58:32.373 INFO:teuthology.orchestra.run.smithi003.stdout:(23/24): librbd1-17.2.7-1404.gf2c39d4d.el9.x86_ 15 MB/s | 2.9 MB 00:00 2024-08-17T16:58:32.432 INFO:teuthology.orchestra.run.smithi003.stdout:(24/24): librados2-17.2.7-1404.gf2c39d4d.el9.x8 11 MB/s | 3.1 MB 00:00 2024-08-17T16:58:32.436 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-08-17T16:58:32.437 INFO:teuthology.orchestra.run.smithi003.stdout:Total 21 MB/s | 58 MB 00:02 2024-08-17T16:58:32.498 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-08-17T16:58:32.536 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-08-17T16:58:32.536 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-08-17T16:58:33.007 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-08-17T16:58:33.007 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-08-17T16:58:33.401 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-08-17T16:58:33.769 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-08-17T16:58:33.870 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : librados2-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2/26 2024-08-17T16:58:33.975 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados2-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2/26 2024-08-17T16:58:34.060 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rados-2:17.2.7-1404.gf2c39d4d.el9.x86_64 3/26 2024-08-17T16:58:34.088 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephfs2-2:17.2.7-1404.gf2c39d4d.el9.x86_64 4/26 2024-08-17T16:58:34.170 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephfs2-2:17.2.7-1404.gf2c39d4d.el9.x86_64 4/26 2024-08-17T16:58:34.203 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-08-17T16:58:34.216 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-08-17T16:58:34.251 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-08-17T16:58:34.620 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-08-17T16:58:34.652 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librados2-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2/26 2024-08-17T16:58:34.727 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-08-17T16:58:34.757 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librgw2-2:17.2.7-1404.gf2c39d4d.el9.x86_64 8/26 2024-08-17T16:58:34.758 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2/26 2024-08-17T16:58:34.804 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librgw2-2:17.2.7-1404.gf2c39d4d.el9.x86_64 8/26 2024-08-17T16:58:34.833 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-08-17T16:58:34.851 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rados-2:17.2.7-1404.gf2c39d4d.el9.x86_64 3/26 2024-08-17T16:58:34.879 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ceph-argparse-2:17.2.7-1404.gf2c39d4d.el9. 10/26 2024-08-17T16:58:34.879 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs2-2:17.2.7-1404.gf2c39d4d.el9.x86_64 4/26 2024-08-17T16:58:34.915 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cephfs-2:17.2.7-1404.gf2c39d4d.el9.x86_64 11/26 2024-08-17T16:58:34.969 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephfs2-2:17.2.7-1404.gf2c39d4d.el9.x86_64 4/26 2024-08-17T16:58:34.985 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rgw-2:17.2.7-1404.gf2c39d4d.el9.x86_64 12/26 2024-08-17T16:58:35.010 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libradosstriper1-2:17.2.7-1404.gf2c39d4d.el9.x86_6 13/26 2024-08-17T16:58:35.014 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-08-17T16:58:35.042 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-08-17T16:58:35.077 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1404.gf2c39d4d.el9.x86_6 13/26 2024-08-17T16:58:35.135 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-08-17T16:58:35.170 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-08-17T16:58:35.202 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-08-17T16:58:35.505 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-08-17T16:58:35.513 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-08-17T16:58:35.532 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librgw2-2:17.2.7-1404.gf2c39d4d.el9.x86_64 8/26 2024-08-17T16:58:35.556 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : librbd1-2:17.2.7-1404.gf2c39d4d.el9.x86_64 18/26 2024-08-17T16:58:35.611 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librgw2-2:17.2.7-1404.gf2c39d4d.el9.x86_64 8/26 2024-08-17T16:58:35.635 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librbd1-2:17.2.7-1404.gf2c39d4d.el9.x86_64 18/26 2024-08-17T16:58:35.640 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-08-17T16:58:35.686 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-argparse-2:17.2.7-1404.gf2c39d4d.el9. 10/26 2024-08-17T16:58:35.706 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rbd-2:17.2.7-1404.gf2c39d4d.el9.x86_64 19/26 2024-08-17T16:58:35.722 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cephfs-2:17.2.7-1404.gf2c39d4d.el9.x86_64 11/26 2024-08-17T16:58:35.733 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ceph-common-2:17.2.7-1404.gf2c39d4d.el9.x8 20/26 2024-08-17T16:58:35.792 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rgw-2:17.2.7-1404.gf2c39d4d.el9.x86_64 12/26 2024-08-17T16:58:35.818 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libradosstriper1-2:17.2.7-1404.gf2c39d4d.el9.x86_6 13/26 2024-08-17T16:58:35.884 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1404.gf2c39d4d.el9.x86_6 13/26 2024-08-17T16:58:35.925 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-08-17T16:58:35.960 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-08-17T16:58:35.993 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-08-17T16:58:36.310 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-08-17T16:58:36.347 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librbd1-2:17.2.7-1404.gf2c39d4d.el9.x86_64 18/26 2024-08-17T16:58:36.425 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-2:17.2.7-1404.gf2c39d4d.el9.x86_64 18/26 2024-08-17T16:58:36.497 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rbd-2:17.2.7-1404.gf2c39d4d.el9.x86_64 19/26 2024-08-17T16:58:36.524 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-common-2:17.2.7-1404.gf2c39d4d.el9.x8 20/26 2024-08-17T16:58:38.253 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-common-2:17.2.7-1404.gf2c39d4d.el9.x86_64 21/26 2024-08-17T16:58:38.301 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-common-2:17.2.7-1404.gf2c39d4d.el9.x86_64 21/26 2024-08-17T16:58:38.869 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-common-2:17.2.7-1404.gf2c39d4d.el9.x86_64 21/26 2024-08-17T16:58:38.905 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-base-2:17.2.7-1404.gf2c39d4d.el9.x86_64 22/26 2024-08-17T16:58:39.000 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-base-2:17.2.7-1404.gf2c39d4d.el9.x86_64 22/26 2024-08-17T16:58:39.000 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-17T16:58:39.000 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-17T16:58:39.001 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:58:39.031 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-selinux-2:17.2.7-1404.gf2c39d4d.el9.x86_64 23/26 2024-08-17T16:58:39.033 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:17.2.7-1404.gf2c39d4d.el9.x86_64 21/26 2024-08-17T16:58:39.072 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-common-2:17.2.7-1404.gf2c39d4d.el9.x86_64 21/26 2024-08-17T16:58:39.629 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:17.2.7-1404.gf2c39d4d.el9.x86_64 21/26 2024-08-17T16:58:39.667 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-base-2:17.2.7-1404.gf2c39d4d.el9.x86_64 22/26 2024-08-17T16:58:39.781 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-base-2:17.2.7-1404.gf2c39d4d.el9.x86_64 22/26 2024-08-17T16:58:39.782 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-17T16:58:39.782 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-17T16:58:39.782 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:58:39.810 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-selinux-2:17.2.7-1404.gf2c39d4d.el9.x86_64 23/26 2024-08-17T16:58:49.138 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1404.gf2c39d4d.el9.x86_64 23/26 2024-08-17T16:58:49.138 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /sys 2024-08-17T16:58:49.138 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /proc 2024-08-17T16:58:49.138 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /mnt 2024-08-17T16:58:49.138 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /var/tmp 2024-08-17T16:58:49.139 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /home 2024-08-17T16:58:49.139 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /root 2024-08-17T16:58:49.139 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /tmp 2024-08-17T16:58:49.139 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:58:49.783 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1404.gf2c39d4d.el9.x86_64 23/26 2024-08-17T16:58:49.784 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /sys 2024-08-17T16:58:49.784 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /proc 2024-08-17T16:58:49.784 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /mnt 2024-08-17T16:58:49.784 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /var/tmp 2024-08-17T16:58:49.784 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /home 2024-08-17T16:58:49.784 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /root 2024-08-17T16:58:49.784 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /tmp 2024-08-17T16:58:49.784 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:58:50.509 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-radosgw-2:17.2.7-1404.gf2c39d4d.el9.x86_64 24/26 2024-08-17T16:58:50.589 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1404.gf2c39d4d.el9.x86_64 24/26 2024-08-17T16:58:50.589 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-17T16:58:50.589 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-17T16:58:50.589 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-17T16:58:50.589 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-17T16:58:50.590 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:58:50.591 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-08-17T16:58:50.679 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-08-17T16:58:50.679 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-08-17T16:58:51.137 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-radosgw-2:17.2.7-1404.gf2c39d4d.el9.x86_64 24/26 2024-08-17T16:58:51.216 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1404.gf2c39d4d.el9.x86_64 24/26 2024-08-17T16:58:51.217 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-17T16:58:51.217 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-17T16:58:51.217 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-17T16:58:51.217 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-17T16:58:51.217 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:58:51.219 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-08-17T16:58:51.306 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-08-17T16:58:51.306 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-08-17T16:58:51.973 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-08-17T16:58:51.973 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-base-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/26 2024-08-17T16:58:51.973 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-common-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2/26 2024-08-17T16:58:51.973 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-radosgw-2:17.2.7-1404.gf2c39d4d.el9.x86_64 3/26 2024-08-17T16:58:51.973 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-selinux-2:17.2.7-1404.gf2c39d4d.el9.x86_64 4/26 2024-08-17T16:58:51.973 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephfs2-2:17.2.7-1404.gf2c39d4d.el9.x86_64 5/26 2024-08-17T16:58:51.973 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libradosstriper1-2:17.2.7-1404.gf2c39d4d.el9.x86_6 6/26 2024-08-17T16:58:51.973 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librgw2-2:17.2.7-1404.gf2c39d4d.el9.x86_64 7/26 2024-08-17T16:58:51.973 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1404.gf2c39d4d.el9. 8/26 2024-08-17T16:58:51.973 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ceph-common-2:17.2.7-1404.gf2c39d4d.el9.x8 9/26 2024-08-17T16:58:51.973 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cephfs-2:17.2.7-1404.gf2c39d4d.el9.x86_64 10/26 2024-08-17T16:58:51.974 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rados-2:17.2.7-1404.gf2c39d4d.el9.x86_64 11/26 2024-08-17T16:58:51.974 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rbd-2:17.2.7-1404.gf2c39d4d.el9.x86_64 12/26 2024-08-17T16:58:51.974 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rgw-2:17.2.7-1404.gf2c39d4d.el9.x86_64 13/26 2024-08-17T16:58:51.974 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-08-17T16:58:51.974 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-08-17T16:58:51.974 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-08-17T16:58:51.974 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-08-17T16:58:51.974 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-08-17T16:58:51.974 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-08-17T16:58:51.974 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-08-17T16:58:51.974 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-08-17T16:58:51.974 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-08-17T16:58:51.975 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados2-2:17.2.7-1404.gf2c39d4d.el9.x86_64 23/26 2024-08-17T16:58:51.975 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-08-17T16:58:51.975 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librbd1-2:17.2.7-1404.gf2c39d4d.el9.x86_64 25/26 2024-08-17T16:58:52.533 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-08-17T16:58:52.533 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-base-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/26 2024-08-17T16:58:52.533 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-common-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2/26 2024-08-17T16:58:52.533 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-radosgw-2:17.2.7-1404.gf2c39d4d.el9.x86_64 3/26 2024-08-17T16:58:52.533 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-selinux-2:17.2.7-1404.gf2c39d4d.el9.x86_64 4/26 2024-08-17T16:58:52.533 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs2-2:17.2.7-1404.gf2c39d4d.el9.x86_64 5/26 2024-08-17T16:58:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libradosstriper1-2:17.2.7-1404.gf2c39d4d.el9.x86_6 6/26 2024-08-17T16:58:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librgw2-2:17.2.7-1404.gf2c39d4d.el9.x86_64 7/26 2024-08-17T16:58:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1404.gf2c39d4d.el9. 8/26 2024-08-17T16:58:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-common-2:17.2.7-1404.gf2c39d4d.el9.x8 9/26 2024-08-17T16:58:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cephfs-2:17.2.7-1404.gf2c39d4d.el9.x86_64 10/26 2024-08-17T16:58:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rados-2:17.2.7-1404.gf2c39d4d.el9.x86_64 11/26 2024-08-17T16:58:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rbd-2:17.2.7-1404.gf2c39d4d.el9.x86_64 12/26 2024-08-17T16:58:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rgw-2:17.2.7-1404.gf2c39d4d.el9.x86_64 13/26 2024-08-17T16:58:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-08-17T16:58:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-08-17T16:58:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-08-17T16:58:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-08-17T16:58:52.534 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-08-17T16:58:52.535 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-08-17T16:58:52.535 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-08-17T16:58:52.535 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-08-17T16:58:52.535 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-08-17T16:58:52.535 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-2:17.2.7-1404.gf2c39d4d.el9.x86_64 23/26 2024-08-17T16:58:52.535 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-08-17T16:58:52.535 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-2:17.2.7-1404.gf2c39d4d.el9.x86_64 25/26 2024-08-17T16:58:52.674 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-08-17T16:58:52.674 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:58:52.674 INFO:teuthology.orchestra.run.smithi053.stdout:Upgraded: 2024-08-17T16:58:52.674 INFO:teuthology.orchestra.run.smithi053.stdout: librados2-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:58:52.674 INFO:teuthology.orchestra.run.smithi053.stdout: librbd1-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:58:52.674 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-08-17T16:58:52.675 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-base-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:58:52.675 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-common-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:58:52.675 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-radosgw-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:58:52.675 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-selinux-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:58:52.675 INFO:teuthology.orchestra.run.smithi053.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-17T16:58:52.675 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs2-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:58:52.675 INFO:teuthology.orchestra.run.smithi053.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-17T16:58:52.675 INFO:teuthology.orchestra.run.smithi053.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-17T16:58:52.675 INFO:teuthology.orchestra.run.smithi053.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-17T16:58:52.675 INFO:teuthology.orchestra.run.smithi053.stdout: libradosstriper1-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:58:52.675 INFO:teuthology.orchestra.run.smithi053.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-17T16:58:52.675 INFO:teuthology.orchestra.run.smithi053.stdout: librgw2-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:58:52.676 INFO:teuthology.orchestra.run.smithi053.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-17T16:58:52.676 INFO:teuthology.orchestra.run.smithi053.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-17T16:58:52.676 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-argparse-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:58:52.676 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-common-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:58:52.676 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cephfs-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:58:52.676 INFO:teuthology.orchestra.run.smithi053.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-17T16:58:52.676 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rados-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:58:52.676 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rbd-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:58:52.676 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rgw-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:58:52.676 INFO:teuthology.orchestra.run.smithi053.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-17T16:58:52.676 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:58:52.676 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-08-17T16:58:53.004 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-test 2024-08-17T16:58:53.282 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-08-17T16:58:53.282 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:58:53.283 INFO:teuthology.orchestra.run.smithi003.stdout:Upgraded: 2024-08-17T16:58:53.283 INFO:teuthology.orchestra.run.smithi003.stdout: librados2-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:58:53.283 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:58:53.283 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-08-17T16:58:53.283 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:58:53.283 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:58:53.283 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:58:53.283 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:58:53.283 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-17T16:58:53.283 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:58:53.283 INFO:teuthology.orchestra.run.smithi003.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-17T16:58:53.283 INFO:teuthology.orchestra.run.smithi003.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-17T16:58:53.283 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-17T16:58:53.283 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:58:53.284 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-17T16:58:53.284 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:58:53.284 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-17T16:58:53.284 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-17T16:58:53.284 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:58:53.284 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:58:53.284 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:58:53.284 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-17T16:58:53.284 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:58:53.284 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:58:53.284 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:58:53.284 INFO:teuthology.orchestra.run.smithi003.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-17T16:58:53.284 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:58:53.284 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-08-17T16:58:53.578 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:00:27 ago on Sat 17 Aug 2024 04:58:26 PM UTC. 2024-08-17T16:58:53.674 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-test 2024-08-17T16:58:53.709 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-08-17T16:58:53.710 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:58:53.710 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-08-17T16:58:53.710 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:58:53.710 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-08-17T16:58:53.710 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-test x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 48 M 2024-08-17T16:58:53.710 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-08-17T16:58:53.710 INFO:teuthology.orchestra.run.smithi053.stdout: libcephsqlite x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 162 k 2024-08-17T16:58:53.710 INFO:teuthology.orchestra.run.smithi053.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-17T16:58:53.710 INFO:teuthology.orchestra.run.smithi053.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-17T16:58:53.711 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:58:53.711 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-08-17T16:58:53.711 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:58:53.711 INFO:teuthology.orchestra.run.smithi053.stdout:Install 4 Packages 2024-08-17T16:58:53.711 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:58:53.711 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 48 M 2024-08-17T16:58:53.711 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 193 M 2024-08-17T16:58:53.712 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-08-17T16:58:54.108 INFO:teuthology.orchestra.run.smithi053.stdout:(1/4): libcephsqlite-17.2.7-1404.gf2c39d4d.el9. 608 kB/s | 162 kB 00:00 2024-08-17T16:58:54.243 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:27 ago on Sat 17 Aug 2024 04:58:27 PM UTC. 2024-08-17T16:58:54.341 INFO:teuthology.orchestra.run.smithi053.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 607 kB/s | 304 kB 00:00 2024-08-17T16:58:54.374 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-08-17T16:58:54.375 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:58:54.375 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-08-17T16:58:54.375 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:58:54.375 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-08-17T16:58:54.375 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 48 M 2024-08-17T16:58:54.375 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-08-17T16:58:54.375 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 162 k 2024-08-17T16:58:54.375 INFO:teuthology.orchestra.run.smithi003.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-17T16:58:54.375 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-17T16:58:54.376 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:58:54.376 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-08-17T16:58:54.376 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:58:54.376 INFO:teuthology.orchestra.run.smithi003.stdout:Install 4 Packages 2024-08-17T16:58:54.376 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:58:54.376 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 48 M 2024-08-17T16:58:54.376 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 193 M 2024-08-17T16:58:54.376 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-08-17T16:58:54.450 INFO:teuthology.orchestra.run.smithi053.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 186 kB/s | 64 kB 00:00 2024-08-17T16:58:54.760 INFO:teuthology.orchestra.run.smithi003.stdout:(1/4): libcephsqlite-17.2.7-1404.gf2c39d4d.el9. 473 kB/s | 162 kB 00:00 2024-08-17T16:58:55.035 INFO:teuthology.orchestra.run.smithi003.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 491 kB/s | 304 kB 00:00 2024-08-17T16:58:55.102 INFO:teuthology.orchestra.run.smithi003.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 186 kB/s | 64 kB 00:00 2024-08-17T16:58:55.400 INFO:teuthology.orchestra.run.smithi053.stdout:(4/4): ceph-test-17.2.7-1404.gf2c39d4d.el9.x86_ 31 MB/s | 48 MB 00:01 2024-08-17T16:58:55.403 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-08-17T16:58:55.403 INFO:teuthology.orchestra.run.smithi053.stdout:Total 29 MB/s | 48 MB 00:01 2024-08-17T16:58:55.431 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-08-17T16:58:55.452 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-08-17T16:58:55.452 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-08-17T16:58:55.837 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-08-17T16:58:55.838 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-08-17T16:58:56.372 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-08-17T16:58:56.453 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-08-17T16:58:56.495 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-08-17T16:58:56.519 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephsqlite-2:17.2.7-1404.gf2c39d4d.el9.x86_64 3/4 2024-08-17T16:58:56.994 INFO:teuthology.orchestra.run.smithi003.stdout:(4/4): ceph-test-17.2.7-1404.gf2c39d4d.el9.x86_ 19 MB/s | 48 MB 00:02 2024-08-17T16:58:56.996 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-08-17T16:58:56.996 INFO:teuthology.orchestra.run.smithi003.stdout:Total 18 MB/s | 48 MB 00:02 2024-08-17T16:58:57.024 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-08-17T16:58:57.046 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-08-17T16:58:57.047 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-08-17T16:58:57.432 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-08-17T16:58:57.433 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-08-17T16:58:57.973 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-08-17T16:58:58.045 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-08-17T16:58:58.089 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-08-17T16:58:58.112 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephsqlite-2:17.2.7-1404.gf2c39d4d.el9.x86_64 3/4 2024-08-17T16:59:01.178 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1404.gf2c39d4d.el9.x86_64 3/4 2024-08-17T16:59:01.257 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-test-2:17.2.7-1404.gf2c39d4d.el9.x86_64 4/4 2024-08-17T16:59:02.780 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1404.gf2c39d4d.el9.x86_64 3/4 2024-08-17T16:59:02.856 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-test-2:17.2.7-1404.gf2c39d4d.el9.x86_64 4/4 2024-08-17T16:59:03.577 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-test-2:17.2.7-1404.gf2c39d4d.el9.x86_64 4/4 2024-08-17T16:59:03.577 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-test-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/4 2024-08-17T16:59:03.577 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephsqlite-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2/4 2024-08-17T16:59:03.577 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-08-17T16:59:03.877 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-08-17T16:59:03.877 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:03.877 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-08-17T16:59:03.877 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-test-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:59:03.877 INFO:teuthology.orchestra.run.smithi053.stdout: libcephsqlite-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:59:03.878 INFO:teuthology.orchestra.run.smithi053.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-17T16:59:03.878 INFO:teuthology.orchestra.run.smithi053.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-17T16:59:03.878 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:03.878 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-08-17T16:59:04.129 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph 2024-08-17T16:59:04.695 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:00:38 ago on Sat 17 Aug 2024 04:58:26 PM UTC. 2024-08-17T16:59:04.832 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-08-17T16:59:04.835 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:59:04.835 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-08-17T16:59:04.835 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:59:04.835 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-08-17T16:59:04.835 INFO:teuthology.orchestra.run.smithi053.stdout: ceph x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 6.5 k 2024-08-17T16:59:04.836 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-08-17T16:59:04.836 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mds x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 2.1 M 2024-08-17T16:59:04.836 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 1.4 M 2024-08-17T16:59:04.836 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1404.gf2c39d4d.el9 ceph-noarch 240 k 2024-08-17T16:59:04.836 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mon x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 3.9 M 2024-08-17T16:59:04.836 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-osd x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 15 M 2024-08-17T16:59:04.836 INFO:teuthology.orchestra.run.smithi053.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-17T16:59:04.836 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-17T16:59:04.836 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-17T16:59:04.837 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-17T16:59:04.837 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-17T16:59:04.837 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-17T16:59:04.837 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-17T16:59:04.837 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-collections 2024-08-17T16:59:04.837 INFO:teuthology.orchestra.run.smithi053.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-08-17T16:59:04.837 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-17T16:59:04.837 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-17T16:59:04.837 INFO:teuthology.orchestra.run.smithi053.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-17T16:59:04.837 INFO:teuthology.orchestra.run.smithi053.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-17T16:59:04.837 INFO:teuthology.orchestra.run.smithi053.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-17T16:59:04.837 INFO:teuthology.orchestra.run.smithi053.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-17T16:59:04.838 INFO:teuthology.orchestra.run.smithi053.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-17T16:59:04.838 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-17T16:59:04.838 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-17T16:59:04.838 INFO:teuthology.orchestra.run.smithi053.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-17T16:59:04.838 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-17T16:59:04.838 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-17T16:59:04.838 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-17T16:59:04.838 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-17T16:59:04.838 INFO:teuthology.orchestra.run.smithi053.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-17T16:59:04.838 INFO:teuthology.orchestra.run.smithi053.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-17T16:59:04.838 INFO:teuthology.orchestra.run.smithi053.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-17T16:59:04.838 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-17T16:59:04.838 INFO:teuthology.orchestra.run.smithi053.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-17T16:59:04.839 INFO:teuthology.orchestra.run.smithi053.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-17T16:59:04.839 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:04.839 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-08-17T16:59:04.839 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:59:04.839 INFO:teuthology.orchestra.run.smithi053.stdout:Install 34 Packages 2024-08-17T16:59:04.839 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:04.839 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 27 M 2024-08-17T16:59:04.840 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 96 M 2024-08-17T16:59:04.840 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-08-17T16:59:05.071 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-test-2:17.2.7-1404.gf2c39d4d.el9.x86_64 4/4 2024-08-17T16:59:05.071 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-test-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/4 2024-08-17T16:59:05.071 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephsqlite-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2/4 2024-08-17T16:59:05.071 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-08-17T16:59:05.367 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-08-17T16:59:05.368 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:05.368 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-08-17T16:59:05.368 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:59:05.368 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:59:05.368 INFO:teuthology.orchestra.run.smithi003.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-17T16:59:05.368 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-17T16:59:05.368 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:05.368 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-08-17T16:59:05.394 INFO:teuthology.orchestra.run.smithi053.stdout:(1/34): ceph-17.2.7-1404.gf2c39d4d.el9.x86_64.r 40 kB/s | 6.5 kB 00:00 2024-08-17T16:59:05.611 INFO:teuthology.orchestra.run.smithi053.stdout:(2/34): ceph-mds-17.2.7-1404.gf2c39d4d.el9.x86_ 5.4 MB/s | 2.1 MB 00:00 2024-08-17T16:59:05.645 INFO:teuthology.orchestra.run.smithi053.stdout:(3/34): ceph-mgr-17.2.7-1404.gf2c39d4d.el9.x86_ 3.4 MB/s | 1.4 MB 00:00 2024-08-17T16:59:05.660 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph 2024-08-17T16:59:05.737 INFO:teuthology.orchestra.run.smithi053.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1404.gf2c3 2.6 MB/s | 240 kB 00:00 2024-08-17T16:59:05.796 INFO:teuthology.orchestra.run.smithi053.stdout:(5/34): ceph-mon-17.2.7-1404.gf2c39d4d.el9.x86_ 9.8 MB/s | 3.9 MB 00:00 2024-08-17T16:59:05.963 INFO:teuthology.orchestra.run.smithi053.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-08-17T16:59:06.121 INFO:teuthology.orchestra.run.smithi053.stdout:(7/34): ceph-osd-17.2.7-1404.gf2c39d4d.el9.x86_ 30 MB/s | 15 MB 00:00 2024-08-17T16:59:06.146 INFO:teuthology.orchestra.run.smithi053.stdout:(8/34): python3-ply-3.11-14.el9.noarch.rpm 581 kB/s | 106 kB 00:00 2024-08-17T16:59:06.196 INFO:teuthology.orchestra.run.smithi053.stdout:(9/34): python3-requests-2.25.1-8.el9.noarch.rp 2.5 MB/s | 125 kB 00:00 2024-08-17T16:59:06.227 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:39 ago on Sat 17 Aug 2024 04:58:27 PM UTC. 2024-08-17T16:59:06.255 INFO:teuthology.orchestra.run.smithi053.stdout:(10/34): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-08-17T16:59:06.297 INFO:teuthology.orchestra.run.smithi053.stdout:(11/34): python3-cryptography-36.0.1-4.el9.x86_ 2.4 MB/s | 1.2 MB 00:00 2024-08-17T16:59:06.322 INFO:teuthology.orchestra.run.smithi053.stdout:(12/34): python3-pycparser-2.20-6.el9.noarch.rp 674 kB/s | 135 kB 00:00 2024-08-17T16:59:06.364 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-08-17T16:59:06.366 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:59:06.366 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-08-17T16:59:06.366 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:59:06.366 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-08-17T16:59:06.366 INFO:teuthology.orchestra.run.smithi003.stdout: ceph x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 6.5 k 2024-08-17T16:59:06.367 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-08-17T16:59:06.367 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 2.1 M 2024-08-17T16:59:06.367 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 1.4 M 2024-08-17T16:59:06.367 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1404.gf2c39d4d.el9 ceph-noarch 240 k 2024-08-17T16:59:06.367 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 3.9 M 2024-08-17T16:59:06.367 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 15 M 2024-08-17T16:59:06.367 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-17T16:59:06.367 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-17T16:59:06.367 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-17T16:59:06.367 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-17T16:59:06.367 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-17T16:59:06.367 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-17T16:59:06.367 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-17T16:59:06.367 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-collections 2024-08-17T16:59:06.367 INFO:teuthology.orchestra.run.smithi003.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-08-17T16:59:06.368 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-17T16:59:06.368 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-17T16:59:06.368 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-17T16:59:06.368 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-17T16:59:06.368 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-17T16:59:06.368 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-17T16:59:06.368 INFO:teuthology.orchestra.run.smithi003.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-17T16:59:06.368 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-17T16:59:06.368 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-17T16:59:06.368 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-17T16:59:06.368 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-17T16:59:06.368 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-17T16:59:06.368 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-17T16:59:06.369 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-17T16:59:06.369 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-17T16:59:06.369 INFO:teuthology.orchestra.run.smithi003.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-17T16:59:06.369 INFO:teuthology.orchestra.run.smithi003.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-17T16:59:06.369 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-17T16:59:06.369 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-17T16:59:06.369 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-17T16:59:06.369 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:06.369 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-08-17T16:59:06.369 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:59:06.369 INFO:teuthology.orchestra.run.smithi003.stdout:Install 34 Packages 2024-08-17T16:59:06.369 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:06.370 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 27 M 2024-08-17T16:59:06.370 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 96 M 2024-08-17T16:59:06.370 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-08-17T16:59:06.406 INFO:teuthology.orchestra.run.smithi053.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 321 kB/s | 35 kB 00:00 2024-08-17T16:59:06.431 INFO:teuthology.orchestra.run.smithi053.stdout:(14/34): python3-mako-1.1.4-6.el9.noarch.rpm 980 kB/s | 172 kB 00:00 2024-08-17T16:59:06.457 INFO:teuthology.orchestra.run.smithi053.stdout:(15/34): python3-packaging-20.9-5.el9.noarch.rp 575 kB/s | 77 kB 00:00 2024-08-17T16:59:06.507 INFO:teuthology.orchestra.run.smithi053.stdout:(16/34): python3-pytz-2021.1-5.el9.noarch.rpm 506 kB/s | 51 kB 00:00 2024-08-17T16:59:06.532 INFO:teuthology.orchestra.run.smithi053.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 414 kB/s | 42 kB 00:00 2024-08-17T16:59:06.616 INFO:teuthology.orchestra.run.smithi053.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 274 kB/s | 43 kB 00:00 2024-08-17T16:59:06.641 INFO:teuthology.orchestra.run.smithi053.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 479 kB/s | 11 kB 00:00 2024-08-17T16:59:06.667 INFO:teuthology.orchestra.run.smithi053.stdout:(20/34): python3-cherrypy-18.6.1-2.el9.noarch.r 2.7 MB/s | 358 kB 00:00 2024-08-17T16:59:06.692 INFO:teuthology.orchestra.run.smithi053.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 349 kB/s | 18 kB 00:00 2024-08-17T16:59:06.717 INFO:teuthology.orchestra.run.smithi053.stdout:(22/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 945 kB/s | 172 kB 00:00 2024-08-17T16:59:06.742 INFO:teuthology.orchestra.run.smithi053.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-08-17T16:59:06.767 INFO:teuthology.orchestra.run.smithi053.stdout:(24/34): python3-jaraco-collections-3.0.0-8.el9 307 kB/s | 23 kB 00:00 2024-08-17T16:59:06.792 INFO:teuthology.orchestra.run.smithi053.stdout:(25/34): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-08-17T16:59:06.818 INFO:teuthology.orchestra.run.smithi053.stdout:(26/34): python3-logutils-0.3.5-21.el9.noarch.r 614 kB/s | 46 kB 00:00 2024-08-17T16:59:06.843 INFO:teuthology.orchestra.run.smithi053.stdout:(27/34): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-08-17T16:59:06.868 INFO:teuthology.orchestra.run.smithi053.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-08-17T16:59:06.894 INFO:teuthology.orchestra.run.smithi053.stdout:(29/34): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-08-17T16:59:06.919 INFO:teuthology.orchestra.run.smithi053.stdout:(30/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-08-17T16:59:06.928 INFO:teuthology.orchestra.run.smithi003.stdout:(1/34): ceph-17.2.7-1404.gf2c39d4d.el9.x86_64.r 39 kB/s | 6.5 kB 00:00 2024-08-17T16:59:06.944 INFO:teuthology.orchestra.run.smithi053.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-08-17T16:59:06.970 INFO:teuthology.orchestra.run.smithi053.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 800 kB/s | 20 kB 00:00 2024-08-17T16:59:06.995 INFO:teuthology.orchestra.run.smithi053.stdout:(33/34): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-08-17T16:59:07.028 INFO:teuthology.orchestra.run.smithi053.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-08-17T16:59:07.033 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-08-17T16:59:07.033 INFO:teuthology.orchestra.run.smithi053.stdout:Total 12 MB/s | 27 MB 00:02 2024-08-17T16:59:07.129 INFO:teuthology.orchestra.run.smithi003.stdout:(2/34): ceph-mds-17.2.7-1404.gf2c39d4d.el9.x86_ 5.6 MB/s | 2.1 MB 00:00 2024-08-17T16:59:07.162 INFO:teuthology.orchestra.run.smithi003.stdout:(3/34): ceph-mgr-17.2.7-1404.gf2c39d4d.el9.x86_ 3.5 MB/s | 1.4 MB 00:00 2024-08-17T16:59:07.163 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-08-17T16:59:07.203 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-08-17T16:59:07.203 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-08-17T16:59:07.246 INFO:teuthology.orchestra.run.smithi003.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1404.gf2c3 2.8 MB/s | 240 kB 00:00 2024-08-17T16:59:07.363 INFO:teuthology.orchestra.run.smithi003.stdout:(5/34): ceph-mon-17.2.7-1404.gf2c39d4d.el9.x86_ 9.0 MB/s | 3.9 MB 00:00 2024-08-17T16:59:07.438 INFO:teuthology.orchestra.run.smithi003.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-08-17T16:59:07.577 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-08-17T16:59:07.577 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-08-17T16:59:07.613 INFO:teuthology.orchestra.run.smithi003.stdout:(7/34): ceph-osd-17.2.7-1404.gf2c39d4d.el9.x86_ 32 MB/s | 15 MB 00:00 2024-08-17T16:59:07.630 INFO:teuthology.orchestra.run.smithi003.stdout:(8/34): python3-ply-3.11-14.el9.noarch.rpm 555 kB/s | 106 kB 00:00 2024-08-17T16:59:07.722 INFO:teuthology.orchestra.run.smithi003.stdout:(9/34): python3-requests-2.25.1-8.el9.noarch.rp 1.3 MB/s | 125 kB 00:00 2024-08-17T16:59:07.764 INFO:teuthology.orchestra.run.smithi003.stdout:(10/34): python3-pycparser-2.20-6.el9.noarch.rp 900 kB/s | 135 kB 00:00 2024-08-17T16:59:07.814 INFO:teuthology.orchestra.run.smithi003.stdout:(11/34): python3-cryptography-36.0.1-4.el9.x86_ 2.7 MB/s | 1.2 MB 00:00 2024-08-17T16:59:07.840 INFO:teuthology.orchestra.run.smithi003.stdout:(12/34): python3-urllib3-1.26.5-6.el9.noarch.rp 1.8 MB/s | 215 kB 00:00 2024-08-17T16:59:07.932 INFO:teuthology.orchestra.run.smithi003.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 296 kB/s | 35 kB 00:00 2024-08-17T16:59:07.957 INFO:teuthology.orchestra.run.smithi003.stdout:(14/34): python3-mako-1.1.4-6.el9.noarch.rpm 894 kB/s | 172 kB 00:00 2024-08-17T16:59:08.007 INFO:teuthology.orchestra.run.smithi003.stdout:(15/34): python3-packaging-20.9-5.el9.noarch.rp 461 kB/s | 77 kB 00:00 2024-08-17T16:59:08.032 INFO:teuthology.orchestra.run.smithi003.stdout:(16/34): python3-pytz-2021.1-5.el9.noarch.rpm 506 kB/s | 51 kB 00:00 2024-08-17T16:59:08.058 INFO:teuthology.orchestra.run.smithi003.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 414 kB/s | 42 kB 00:00 2024-08-17T16:59:08.158 INFO:teuthology.orchestra.run.smithi003.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 289 kB/s | 43 kB 00:00 2024-08-17T16:59:08.183 INFO:teuthology.orchestra.run.smithi003.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 472 kB/s | 11 kB 00:00 2024-08-17T16:59:08.208 INFO:teuthology.orchestra.run.smithi003.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 977 kB/s | 172 kB 00:00 2024-08-17T16:59:08.225 INFO:teuthology.orchestra.run.smithi003.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 421 kB/s | 18 kB 00:00 2024-08-17T16:59:08.242 INFO:teuthology.orchestra.run.smithi003.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 692 kB/s | 23 kB 00:00 2024-08-17T16:59:08.259 INFO:teuthology.orchestra.run.smithi003.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 579 kB/s | 19 kB 00:00 2024-08-17T16:59:08.277 INFO:teuthology.orchestra.run.smithi003.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 582 kB/s | 20 kB 00:00 2024-08-17T16:59:08.293 INFO:teuthology.orchestra.run.smithi003.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 1.3 MB/s | 46 kB 00:00 2024-08-17T16:59:08.319 INFO:teuthology.orchestra.run.smithi003.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-08-17T16:59:08.344 INFO:teuthology.orchestra.run.smithi003.stdout:(27/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-08-17T16:59:08.369 INFO:teuthology.orchestra.run.smithi003.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-08-17T16:59:08.394 INFO:teuthology.orchestra.run.smithi003.stdout:(29/34): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-08-17T16:59:08.411 INFO:teuthology.orchestra.run.smithi003.stdout:(30/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-08-17T16:59:08.436 INFO:teuthology.orchestra.run.smithi003.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-08-17T16:59:08.462 INFO:teuthology.orchestra.run.smithi003.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 799 kB/s | 20 kB 00:00 2024-08-17T16:59:08.487 INFO:teuthology.orchestra.run.smithi003.stdout:(33/34): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-08-17T16:59:08.512 INFO:teuthology.orchestra.run.smithi003.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-08-17T16:59:08.516 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-08-17T16:59:08.516 INFO:teuthology.orchestra.run.smithi003.stdout:Total 13 MB/s | 27 MB 00:02 2024-08-17T16:59:08.643 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-08-17T16:59:08.685 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-08-17T16:59:08.685 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-08-17T16:59:08.771 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-08-17T16:59:08.831 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-08-17T16:59:08.864 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-08-17T16:59:08.897 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-08-17T16:59:09.010 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-08-17T16:59:09.049 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-08-17T16:59:09.049 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-08-17T16:59:09.102 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-08-17T16:59:09.158 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/34 2024-08-17T16:59:09.194 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-08-17T16:59:09.223 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-08-17T16:59:09.257 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-08-17T16:59:09.290 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-08-17T16:59:09.332 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-08-17T16:59:09.375 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-08-17T16:59:09.409 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-08-17T16:59:09.458 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-08-17T16:59:09.498 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-08-17T16:59:09.584 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-08-17T16:59:09.736 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-08-17T16:59:09.819 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-08-17T16:59:09.877 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-08-17T16:59:09.915 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-08-17T16:59:09.990 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-08-17T16:59:10.054 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-08-17T16:59:10.242 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-08-17T16:59:10.259 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-08-17T16:59:10.298 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-08-17T16:59:10.310 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-08-17T16:59:10.331 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-08-17T16:59:10.364 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-08-17T16:59:10.391 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-08-17T16:59:10.462 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-08-17T16:59:10.491 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-08-17T16:59:10.526 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-08-17T16:59:10.554 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-08-17T16:59:10.609 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/34 2024-08-17T16:59:10.643 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-08-17T16:59:10.647 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-08-17T16:59:10.673 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-08-17T16:59:10.707 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-08-17T16:59:10.741 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-08-17T16:59:10.782 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-08-17T16:59:10.817 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-08-17T16:59:10.820 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1404.gf2c39d4d.el9. 29/34 2024-08-17T16:59:10.844 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-2:17.2.7-1404.gf2c39d4d.el9.x86_64 30/34 2024-08-17T16:59:10.851 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-08-17T16:59:10.871 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1404.gf2c39d4d.el9.x86_64 30/34 2024-08-17T16:59:10.872 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-17T16:59:10.872 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-17T16:59:10.872 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-17T16:59:10.872 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-17T16:59:10.872 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:10.901 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-08-17T16:59:10.940 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-08-17T16:59:11.050 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-08-17T16:59:11.203 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-08-17T16:59:11.293 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-08-17T16:59:11.344 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-08-17T16:59:11.381 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-08-17T16:59:11.440 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-08-17T16:59:11.503 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-08-17T16:59:11.714 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-08-17T16:59:11.760 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-08-17T16:59:11.841 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-08-17T16:59:11.932 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-08-17T16:59:11.976 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-08-17T16:59:12.101 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-08-17T16:59:12.258 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1404.gf2c39d4d.el9. 29/34 2024-08-17T16:59:12.286 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-2:17.2.7-1404.gf2c39d4d.el9.x86_64 30/34 2024-08-17T16:59:12.287 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-osd-2:17.2.7-1404.gf2c39d4d.el9.x86_64 31/34 2024-08-17T16:59:12.313 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1404.gf2c39d4d.el9.x86_64 30/34 2024-08-17T16:59:12.313 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-17T16:59:12.313 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-17T16:59:12.313 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-17T16:59:12.313 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-17T16:59:12.313 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:12.318 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-osd-2:17.2.7-1404.gf2c39d4d.el9.x86_64 31/34 2024-08-17T16:59:12.318 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-17T16:59:12.318 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-17T16:59:12.318 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-17T16:59:12.318 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-17T16:59:12.318 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:12.725 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mon-2:17.2.7-1404.gf2c39d4d.el9.x86_64 32/34 2024-08-17T16:59:12.751 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mon-2:17.2.7-1404.gf2c39d4d.el9.x86_64 32/34 2024-08-17T16:59:12.751 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-17T16:59:12.751 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-17T16:59:12.752 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-17T16:59:12.752 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-17T16:59:12.752 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:12.964 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mds-2:17.2.7-1404.gf2c39d4d.el9.x86_64 33/34 2024-08-17T16:59:12.990 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mds-2:17.2.7-1404.gf2c39d4d.el9.x86_64 33/34 2024-08-17T16:59:12.990 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-17T16:59:12.990 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-17T16:59:12.990 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-17T16:59:12.990 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-17T16:59:12.990 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:13.047 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-2:17.2.7-1404.gf2c39d4d.el9.x86_64 34/34 2024-08-17T16:59:13.720 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-osd-2:17.2.7-1404.gf2c39d4d.el9.x86_64 31/34 2024-08-17T16:59:13.751 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-osd-2:17.2.7-1404.gf2c39d4d.el9.x86_64 31/34 2024-08-17T16:59:13.751 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-17T16:59:13.751 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-17T16:59:13.751 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-17T16:59:13.751 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-17T16:59:13.752 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:14.138 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mon-2:17.2.7-1404.gf2c39d4d.el9.x86_64 32/34 2024-08-17T16:59:14.164 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mon-2:17.2.7-1404.gf2c39d4d.el9.x86_64 32/34 2024-08-17T16:59:14.164 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-17T16:59:14.164 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-17T16:59:14.164 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-17T16:59:14.164 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-17T16:59:14.164 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:14.372 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mds-2:17.2.7-1404.gf2c39d4d.el9.x86_64 33/34 2024-08-17T16:59:14.397 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mds-2:17.2.7-1404.gf2c39d4d.el9.x86_64 33/34 2024-08-17T16:59:14.397 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-17T16:59:14.397 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-17T16:59:14.397 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-17T16:59:14.397 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-17T16:59:14.397 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:14.455 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-2:17.2.7-1404.gf2c39d4d.el9.x86_64 34/34 2024-08-17T16:59:14.745 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-2:17.2.7-1404.gf2c39d4d.el9.x86_64 34/34 2024-08-17T16:59:14.745 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/34 2024-08-17T16:59:14.745 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mds-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2/34 2024-08-17T16:59:14.745 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-2:17.2.7-1404.gf2c39d4d.el9.x86_64 3/34 2024-08-17T16:59:14.746 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mon-2:17.2.7-1404.gf2c39d4d.el9.x86_64 4/34 2024-08-17T16:59:14.746 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-osd-2:17.2.7-1404.gf2c39d4d.el9.x86_64 5/34 2024-08-17T16:59:14.746 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1404.gf2c39d4d.el9. 6/34 2024-08-17T16:59:14.746 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-08-17T16:59:14.746 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-08-17T16:59:14.746 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-08-17T16:59:14.746 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-08-17T16:59:14.746 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-08-17T16:59:14.746 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-08-17T16:59:14.746 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-08-17T16:59:14.746 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-08-17T16:59:14.746 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-08-17T16:59:14.746 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-08-17T16:59:14.747 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-08-17T16:59:14.747 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-08-17T16:59:14.747 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-08-17T16:59:14.747 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-08-17T16:59:14.747 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-08-17T16:59:14.747 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-08-17T16:59:14.747 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-08-17T16:59:14.747 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-08-17T16:59:14.747 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-08-17T16:59:14.747 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-08-17T16:59:14.747 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-08-17T16:59:14.747 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-08-17T16:59:14.747 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-08-17T16:59:14.747 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-08-17T16:59:14.748 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-08-17T16:59:14.748 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 32/34 2024-08-17T16:59:14.748 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-08-17T16:59:15.732 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-08-17T16:59:15.732 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:15.732 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-08-17T16:59:15.732 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:59:15.732 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mds-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:59:15.732 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:59:15.732 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-modules-core-2:17.2.7-1404.gf2c39d4d.el9.noarch 2024-08-17T16:59:15.732 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mon-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:59:15.733 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-osd-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:59:15.733 INFO:teuthology.orchestra.run.smithi053.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-17T16:59:15.733 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-17T16:59:15.733 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-17T16:59:15.733 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-17T16:59:15.733 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-17T16:59:15.733 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-17T16:59:15.733 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-17T16:59:15.733 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-17T16:59:15.733 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-17T16:59:15.733 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-17T16:59:15.733 INFO:teuthology.orchestra.run.smithi053.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-17T16:59:15.733 INFO:teuthology.orchestra.run.smithi053.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-17T16:59:15.734 INFO:teuthology.orchestra.run.smithi053.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-17T16:59:15.734 INFO:teuthology.orchestra.run.smithi053.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-17T16:59:15.734 INFO:teuthology.orchestra.run.smithi053.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-17T16:59:15.734 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-17T16:59:15.734 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ply-3.11-14.el9.noarch 2024-08-17T16:59:15.734 INFO:teuthology.orchestra.run.smithi053.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-17T16:59:15.734 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-17T16:59:15.734 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-17T16:59:15.734 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-17T16:59:15.734 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-17T16:59:15.734 INFO:teuthology.orchestra.run.smithi053.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-17T16:59:15.734 INFO:teuthology.orchestra.run.smithi053.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-17T16:59:15.735 INFO:teuthology.orchestra.run.smithi053.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-17T16:59:15.735 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-17T16:59:15.735 INFO:teuthology.orchestra.run.smithi053.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-17T16:59:15.735 INFO:teuthology.orchestra.run.smithi053.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-17T16:59:15.735 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:15.735 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-08-17T16:59:16.044 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-2:17.2.7-1404.gf2c39d4d.el9.x86_64 34/34 2024-08-17T16:59:16.044 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/34 2024-08-17T16:59:16.044 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mds-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2/34 2024-08-17T16:59:16.044 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-2:17.2.7-1404.gf2c39d4d.el9.x86_64 3/34 2024-08-17T16:59:16.045 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mon-2:17.2.7-1404.gf2c39d4d.el9.x86_64 4/34 2024-08-17T16:59:16.045 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-osd-2:17.2.7-1404.gf2c39d4d.el9.x86_64 5/34 2024-08-17T16:59:16.045 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1404.gf2c39d4d.el9. 6/34 2024-08-17T16:59:16.045 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-08-17T16:59:16.045 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-08-17T16:59:16.045 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-08-17T16:59:16.045 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-08-17T16:59:16.045 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-08-17T16:59:16.045 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-08-17T16:59:16.045 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-08-17T16:59:16.045 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-08-17T16:59:16.045 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-08-17T16:59:16.045 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-08-17T16:59:16.045 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-08-17T16:59:16.046 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-08-17T16:59:16.046 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-08-17T16:59:16.046 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-08-17T16:59:16.046 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-08-17T16:59:16.046 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-08-17T16:59:16.046 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-08-17T16:59:16.046 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-08-17T16:59:16.046 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-08-17T16:59:16.046 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-08-17T16:59:16.046 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-08-17T16:59:16.047 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-08-17T16:59:16.047 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-08-17T16:59:16.047 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-08-17T16:59:16.047 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-08-17T16:59:16.047 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 32/34 2024-08-17T16:59:16.047 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-08-17T16:59:16.053 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-base 2024-08-17T16:59:16.636 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:00:50 ago on Sat 17 Aug 2024 04:58:26 PM UTC. 2024-08-17T16:59:16.724 INFO:teuthology.orchestra.run.smithi053.stdout:Package ceph-base-2:17.2.7-1404.gf2c39d4d.el9.x86_64 is already installed. 2024-08-17T16:59:16.770 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-08-17T16:59:16.771 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-08-17T16:59:16.772 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-08-17T16:59:16.839 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install cephadm 2024-08-17T16:59:16.896 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-08-17T16:59:16.896 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:16.896 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-08-17T16:59:16.896 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:59:16.896 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:59:16.896 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:59:16.896 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core-2:17.2.7-1404.gf2c39d4d.el9.noarch 2024-08-17T16:59:16.896 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:59:16.896 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:59:16.897 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-17T16:59:16.897 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-17T16:59:16.897 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-17T16:59:16.897 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-17T16:59:16.897 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-17T16:59:16.897 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-17T16:59:16.897 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-17T16:59:16.897 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-17T16:59:16.897 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-17T16:59:16.897 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-17T16:59:16.897 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-17T16:59:16.897 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-17T16:59:16.898 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-17T16:59:16.898 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-17T16:59:16.898 INFO:teuthology.orchestra.run.smithi003.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-17T16:59:16.898 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-17T16:59:16.898 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ply-3.11-14.el9.noarch 2024-08-17T16:59:16.898 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-17T16:59:16.898 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-17T16:59:16.898 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-17T16:59:16.898 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-17T16:59:16.898 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-17T16:59:16.898 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-17T16:59:16.898 INFO:teuthology.orchestra.run.smithi003.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-17T16:59:16.898 INFO:teuthology.orchestra.run.smithi003.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-17T16:59:16.899 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-17T16:59:16.899 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-17T16:59:16.899 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-17T16:59:16.899 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:16.899 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-08-17T16:59:17.298 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-base 2024-08-17T16:59:17.418 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:00:51 ago on Sat 17 Aug 2024 04:58:26 PM UTC. 2024-08-17T16:59:17.551 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-08-17T16:59:17.552 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:59:17.552 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-08-17T16:59:17.553 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:59:17.553 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-08-17T16:59:17.553 INFO:teuthology.orchestra.run.smithi053.stdout: cephadm noarch 2:17.2.7-1404.gf2c39d4d.el9 ceph-noarch 83 k 2024-08-17T16:59:17.553 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:17.553 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-08-17T16:59:17.553 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:59:17.553 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-08-17T16:59:17.553 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:17.553 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 83 k 2024-08-17T16:59:17.553 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 375 k 2024-08-17T16:59:17.553 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-08-17T16:59:17.772 INFO:teuthology.orchestra.run.smithi053.stdout:cephadm-17.2.7-1404.gf2c39d4d.el9.noarch.rpm 379 kB/s | 83 kB 00:00 2024-08-17T16:59:17.772 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-08-17T16:59:17.773 INFO:teuthology.orchestra.run.smithi053.stdout:Total 376 kB/s | 83 kB 00:00 2024-08-17T16:59:17.773 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-08-17T16:59:17.778 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-08-17T16:59:17.778 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-08-17T16:59:17.780 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-08-17T16:59:17.781 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-08-17T16:59:17.877 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:50 ago on Sat 17 Aug 2024 04:58:27 PM UTC. 2024-08-17T16:59:17.899 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-08-17T16:59:17.965 INFO:teuthology.orchestra.run.smithi003.stdout:Package ceph-base-2:17.2.7-1404.gf2c39d4d.el9.x86_64 is already installed. 2024-08-17T16:59:18.010 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-08-17T16:59:18.011 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-08-17T16:59:18.011 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-08-17T16:59:18.097 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install cephadm 2024-08-17T16:59:18.351 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: cephadm-2:17.2.7-1404.gf2c39d4d.el9.noarch 1/1 2024-08-17T16:59:18.465 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cephadm-2:17.2.7-1404.gf2c39d4d.el9.noarch 1/1 2024-08-17T16:59:18.670 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:51 ago on Sat 17 Aug 2024 04:58:27 PM UTC. 2024-08-17T16:59:18.793 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: cephadm-2:17.2.7-1404.gf2c39d4d.el9.noarch 1/1 2024-08-17T16:59:18.805 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-08-17T16:59:18.805 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:59:18.805 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-08-17T16:59:18.805 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:59:18.806 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-08-17T16:59:18.806 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm noarch 2:17.2.7-1404.gf2c39d4d.el9 ceph-noarch 83 k 2024-08-17T16:59:18.806 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:18.806 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-08-17T16:59:18.806 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:59:18.806 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-08-17T16:59:18.806 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:18.806 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 83 k 2024-08-17T16:59:18.807 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 375 k 2024-08-17T16:59:18.807 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-08-17T16:59:19.030 INFO:teuthology.orchestra.run.smithi003.stdout:cephadm-17.2.7-1404.gf2c39d4d.el9.noarch.rpm 371 kB/s | 83 kB 00:00 2024-08-17T16:59:19.030 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-08-17T16:59:19.030 INFO:teuthology.orchestra.run.smithi003.stdout:Total 368 kB/s | 83 kB 00:00 2024-08-17T16:59:19.030 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-08-17T16:59:19.036 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-08-17T16:59:19.036 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-08-17T16:59:19.038 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-08-17T16:59:19.038 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-08-17T16:59:19.060 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cephadm-2:17.2.7-1404.gf2c39d4d.el9.noarch 1/1 2024-08-17T16:59:19.060 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:19.060 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-08-17T16:59:19.060 INFO:teuthology.orchestra.run.smithi053.stdout: cephadm-2:17.2.7-1404.gf2c39d4d.el9.noarch 2024-08-17T16:59:19.060 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:19.060 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-08-17T16:59:19.150 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-08-17T16:59:19.238 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-immutable-object-cache 2024-08-17T16:59:19.579 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephadm-2:17.2.7-1404.gf2c39d4d.el9.noarch 1/1 2024-08-17T16:59:19.676 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cephadm-2:17.2.7-1404.gf2c39d4d.el9.noarch 1/1 2024-08-17T16:59:19.821 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:00:53 ago on Sat 17 Aug 2024 04:58:26 PM UTC. 2024-08-17T16:59:19.955 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-08-17T16:59:19.956 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:59:19.956 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-08-17T16:59:19.956 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:59:19.956 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-08-17T16:59:19.956 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 137 k 2024-08-17T16:59:19.956 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:19.956 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-08-17T16:59:19.956 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:59:19.956 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-08-17T16:59:19.956 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:19.956 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 137 k 2024-08-17T16:59:19.956 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 379 k 2024-08-17T16:59:19.957 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-08-17T16:59:20.023 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephadm-2:17.2.7-1404.gf2c39d4d.el9.noarch 1/1 2024-08-17T16:59:20.189 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-immutable-object-cache-17.2.7-1404.gf2c39d 592 kB/s | 137 kB 00:00 2024-08-17T16:59:20.189 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-08-17T16:59:20.189 INFO:teuthology.orchestra.run.smithi053.stdout:Total 589 kB/s | 137 kB 00:00 2024-08-17T16:59:20.190 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-08-17T16:59:20.198 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-08-17T16:59:20.198 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-08-17T16:59:20.258 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-08-17T16:59:20.258 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-08-17T16:59:20.283 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cephadm-2:17.2.7-1404.gf2c39d4d.el9.noarch 1/1 2024-08-17T16:59:20.283 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:20.283 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-08-17T16:59:20.283 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm-2:17.2.7-1404.gf2c39d4d.el9.noarch 2024-08-17T16:59:20.283 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:20.283 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-08-17T16:59:20.426 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-08-17T16:59:20.489 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1404.gf2c39d4d. 1/1 2024-08-17T16:59:20.498 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-immutable-object-cache 2024-08-17T16:59:20.515 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1404.gf2c39d4d. 1/1 2024-08-17T16:59:20.515 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-17T16:59:20.515 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-17T16:59:20.515 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:21.073 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:54 ago on Sat 17 Aug 2024 04:58:27 PM UTC. 2024-08-17T16:59:21.205 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-08-17T16:59:21.206 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:59:21.206 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-08-17T16:59:21.206 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:59:21.206 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-08-17T16:59:21.206 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 137 k 2024-08-17T16:59:21.206 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:21.206 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-08-17T16:59:21.207 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:59:21.207 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-08-17T16:59:21.207 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:21.207 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 137 k 2024-08-17T16:59:21.207 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 379 k 2024-08-17T16:59:21.207 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-08-17T16:59:21.409 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-immutable-object-cache-17.2.7-1404.gf2c39d 681 kB/s | 137 kB 00:00 2024-08-17T16:59:21.410 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-08-17T16:59:21.410 INFO:teuthology.orchestra.run.smithi003.stdout:Total 674 kB/s | 137 kB 00:00 2024-08-17T16:59:21.410 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-08-17T16:59:21.419 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1404.gf2c39d4d. 1/1 2024-08-17T16:59:21.419 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:21.419 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-08-17T16:59:21.419 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-immutable-object-cache-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:59:21.419 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:21.419 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-08-17T16:59:21.419 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-08-17T16:59:21.419 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-08-17T16:59:21.479 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-08-17T16:59:21.479 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-08-17T16:59:21.648 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr 2024-08-17T16:59:21.659 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-08-17T16:59:21.727 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1404.gf2c39d4d. 1/1 2024-08-17T16:59:21.753 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1404.gf2c39d4d. 1/1 2024-08-17T16:59:21.753 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-17T16:59:21.753 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-17T16:59:21.753 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:22.231 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:00:56 ago on Sat 17 Aug 2024 04:58:26 PM UTC. 2024-08-17T16:59:22.321 INFO:teuthology.orchestra.run.smithi053.stdout:Package ceph-mgr-2:17.2.7-1404.gf2c39d4d.el9.x86_64 is already installed. 2024-08-17T16:59:22.364 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-08-17T16:59:22.365 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-08-17T16:59:22.365 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-08-17T16:59:22.430 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr-dashboard 2024-08-17T16:59:22.677 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1404.gf2c39d4d. 1/1 2024-08-17T16:59:22.677 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:22.677 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-08-17T16:59:22.678 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:59:22.678 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:22.678 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-08-17T16:59:22.891 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr 2024-08-17T16:59:22.999 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:00:56 ago on Sat 17 Aug 2024 04:58:26 PM UTC. 2024-08-17T16:59:23.132 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-08-17T16:59:23.133 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:59:23.133 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-08-17T16:59:23.133 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:59:23.134 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-08-17T16:59:23.134 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1404.gf2c39d4d.el9 ceph-noarch 3.4 M 2024-08-17T16:59:23.134 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-08-17T16:59:23.134 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1404.gf2c39d4d.el9 ceph-noarch 24 k 2024-08-17T16:59:23.134 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1404.gf2c39d4d.el9 ceph-noarch 15 k 2024-08-17T16:59:23.134 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-08-17T16:59:23.134 INFO:teuthology.orchestra.run.smithi053.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-17T16:59:23.134 INFO:teuthology.orchestra.run.smithi053.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-17T16:59:23.134 INFO:teuthology.orchestra.run.smithi053.stdout:Installing weak dependencies: 2024-08-17T16:59:23.134 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-08-17T16:59:23.134 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:23.134 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-08-17T16:59:23.134 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:59:23.134 INFO:teuthology.orchestra.run.smithi053.stdout:Install 7 Packages 2024-08-17T16:59:23.135 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:23.135 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 3.7 M 2024-08-17T16:59:23.135 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 80 M 2024-08-17T16:59:23.135 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-08-17T16:59:23.481 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:56 ago on Sat 17 Aug 2024 04:58:27 PM UTC. 2024-08-17T16:59:23.509 INFO:teuthology.orchestra.run.smithi053.stdout:(1/7): ceph-prometheus-alerts-17.2.7-1404.gf2c3 73 kB/s | 15 kB 00:00 2024-08-17T16:59:23.534 INFO:teuthology.orchestra.run.smithi053.stdout:(2/7): ceph-grafana-dashboards-17.2.7-1404.gf2c 108 kB/s | 24 kB 00:00 2024-08-17T16:59:23.568 INFO:teuthology.orchestra.run.smithi003.stdout:Package ceph-mgr-2:17.2.7-1404.gf2c39d4d.el9.x86_64 is already installed. 2024-08-17T16:59:23.612 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-08-17T16:59:23.613 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-08-17T16:59:23.613 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-08-17T16:59:23.618 INFO:teuthology.orchestra.run.smithi053.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 83 kB/s | 9.0 kB 00:00 2024-08-17T16:59:23.643 INFO:teuthology.orchestra.run.smithi053.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-08-17T16:59:23.668 INFO:teuthology.orchestra.run.smithi053.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 305 kB/s | 41 kB 00:00 2024-08-17T16:59:23.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-dashboard 2024-08-17T16:59:23.694 INFO:teuthology.orchestra.run.smithi053.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-08-17T16:59:23.802 INFO:teuthology.orchestra.run.smithi053.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1404.gf2c39d4d 6.8 MB/s | 3.4 MB 00:00 2024-08-17T16:59:23.806 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-08-17T16:59:23.806 INFO:teuthology.orchestra.run.smithi053.stdout:Total 5.4 MB/s | 3.7 MB 00:00 2024-08-17T16:59:23.806 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-08-17T16:59:23.815 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-08-17T16:59:23.815 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-08-17T16:59:23.883 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-08-17T16:59:23.883 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-08-17T16:59:24.189 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-08-17T16:59:24.245 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-08-17T16:59:24.252 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:57 ago on Sat 17 Aug 2024 04:58:27 PM UTC. 2024-08-17T16:59:24.280 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-08-17T16:59:24.336 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-08-17T16:59:24.374 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-08-17T16:59:24.386 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-08-17T16:59:24.387 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:59:24.387 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-08-17T16:59:24.387 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:59:24.387 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-08-17T16:59:24.387 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1404.gf2c39d4d.el9 ceph-noarch 3.4 M 2024-08-17T16:59:24.387 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-08-17T16:59:24.387 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1404.gf2c39d4d.el9 ceph-noarch 24 k 2024-08-17T16:59:24.387 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1404.gf2c39d4d.el9 ceph-noarch 15 k 2024-08-17T16:59:24.387 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-08-17T16:59:24.388 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-17T16:59:24.388 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-17T16:59:24.388 INFO:teuthology.orchestra.run.smithi003.stdout:Installing weak dependencies: 2024-08-17T16:59:24.388 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-08-17T16:59:24.388 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:24.388 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-08-17T16:59:24.388 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:59:24.388 INFO:teuthology.orchestra.run.smithi003.stdout:Install 7 Packages 2024-08-17T16:59:24.388 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:24.389 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 3.7 M 2024-08-17T16:59:24.389 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 80 M 2024-08-17T16:59:24.389 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-08-17T16:59:24.403 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1404.gf2c39d4d.el9.n 5/7 2024-08-17T16:59:24.671 INFO:teuthology.orchestra.run.smithi003.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1404.gf2c 121 kB/s | 24 kB 00:00 2024-08-17T16:59:24.696 INFO:teuthology.orchestra.run.smithi003.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1404.gf2c3 65 kB/s | 15 kB 00:00 2024-08-17T16:59:24.788 INFO:teuthology.orchestra.run.smithi003.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 77 kB/s | 9.0 kB 00:00 2024-08-17T16:59:24.813 INFO:teuthology.orchestra.run.smithi003.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-08-17T16:59:24.838 INFO:teuthology.orchestra.run.smithi003.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 287 kB/s | 41 kB 00:00 2024-08-17T16:59:24.864 INFO:teuthology.orchestra.run.smithi003.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-08-17T16:59:24.947 INFO:teuthology.orchestra.run.smithi003.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1404.gf2c39d4d 7.0 MB/s | 3.4 MB 00:00 2024-08-17T16:59:24.950 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-08-17T16:59:24.951 INFO:teuthology.orchestra.run.smithi003.stdout:Total 6.5 MB/s | 3.7 MB 00:00 2024-08-17T16:59:24.951 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-08-17T16:59:24.960 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-08-17T16:59:24.960 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-08-17T16:59:25.027 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-08-17T16:59:25.028 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-08-17T16:59:25.357 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-08-17T16:59:25.415 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-08-17T16:59:25.450 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-08-17T16:59:25.461 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1404.gf2c39d4d.el9. 6/7 2024-08-17T16:59:25.498 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1404.gf2c39d4d.el9.noarc 7/7 2024-08-17T16:59:25.514 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-08-17T16:59:25.544 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-08-17T16:59:25.573 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1404.gf2c39d4d.el9.n 5/7 2024-08-17T16:59:26.518 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1404.gf2c39d4d.el9.noarc 7/7 2024-08-17T16:59:26.518 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1404.gf2c39d4d.el9. 1/7 2024-08-17T16:59:26.518 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1404.gf2c39d4d.el9.noarc 2/7 2024-08-17T16:59:26.518 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1404.gf2c39d4d.el9.n 3/7 2024-08-17T16:59:26.518 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-08-17T16:59:26.518 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-08-17T16:59:26.518 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-08-17T16:59:26.625 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1404.gf2c39d4d.el9. 6/7 2024-08-17T16:59:26.652 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1404.gf2c39d4d.el9.noarc 7/7 2024-08-17T16:59:26.801 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-08-17T16:59:26.801 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:26.801 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-08-17T16:59:26.801 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-grafana-dashboards-2:17.2.7-1404.gf2c39d4d.el9.noarch 2024-08-17T16:59:26.801 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-dashboard-2:17.2.7-1404.gf2c39d4d.el9.noarch 2024-08-17T16:59:26.801 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-prometheus-alerts-2:17.2.7-1404.gf2c39d4d.el9.noarch 2024-08-17T16:59:26.801 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-08-17T16:59:26.801 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-08-17T16:59:26.801 INFO:teuthology.orchestra.run.smithi053.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-17T16:59:26.801 INFO:teuthology.orchestra.run.smithi053.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-17T16:59:26.802 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:26.802 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-08-17T16:59:27.015 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr-diskprediction-local 2024-08-17T16:59:27.586 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:01 ago on Sat 17 Aug 2024 04:58:26 PM UTC. 2024-08-17T16:59:27.673 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1404.gf2c39d4d.el9.noarc 7/7 2024-08-17T16:59:27.674 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1404.gf2c39d4d.el9. 1/7 2024-08-17T16:59:27.674 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1404.gf2c39d4d.el9.noarc 2/7 2024-08-17T16:59:27.674 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1404.gf2c39d4d.el9.n 3/7 2024-08-17T16:59:27.674 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-08-17T16:59:27.674 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-08-17T16:59:27.674 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-08-17T16:59:27.719 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-08-17T16:59:27.721 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================= 2024-08-17T16:59:27.721 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-08-17T16:59:27.721 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================= 2024-08-17T16:59:27.721 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-08-17T16:59:27.721 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1404.gf2c39d4d.el9 ceph-noarch 7.4 M 2024-08-17T16:59:27.721 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-08-17T16:59:27.721 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-17T16:59:27.721 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-17T16:59:27.721 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-17T16:59:27.721 INFO:teuthology.orchestra.run.smithi053.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-08-17T16:59:27.721 INFO:teuthology.orchestra.run.smithi053.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-08-17T16:59:27.722 INFO:teuthology.orchestra.run.smithi053.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-08-17T16:59:27.722 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-08-17T16:59:27.722 INFO:teuthology.orchestra.run.smithi053.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-08-17T16:59:27.722 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-17T16:59:27.722 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-17T16:59:27.722 INFO:teuthology.orchestra.run.smithi053.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-17T16:59:27.722 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:27.722 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-08-17T16:59:27.722 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================= 2024-08-17T16:59:27.722 INFO:teuthology.orchestra.run.smithi053.stdout:Install 12 Packages 2024-08-17T16:59:27.722 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:27.723 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 38 M 2024-08-17T16:59:27.723 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 209 M 2024-08-17T16:59:27.723 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-08-17T16:59:27.957 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-08-17T16:59:27.957 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:27.957 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-08-17T16:59:27.957 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards-2:17.2.7-1404.gf2c39d4d.el9.noarch 2024-08-17T16:59:27.957 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard-2:17.2.7-1404.gf2c39d4d.el9.noarch 2024-08-17T16:59:27.957 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts-2:17.2.7-1404.gf2c39d4d.el9.noarch 2024-08-17T16:59:27.958 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-08-17T16:59:27.958 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-08-17T16:59:27.958 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-17T16:59:27.958 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-17T16:59:27.958 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:27.958 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-08-17T16:59:28.211 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-diskprediction-local 2024-08-17T16:59:28.418 INFO:teuthology.orchestra.run.smithi053.stdout:(1/12): ceph-mgr-diskprediction-local-17.2.7-14 15 MB/s | 7.4 MB 00:00 2024-08-17T16:59:28.527 INFO:teuthology.orchestra.run.smithi053.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 302 kB/s | 33 kB 00:00 2024-08-17T16:59:28.744 INFO:teuthology.orchestra.run.smithi053.stdout:(3/12): libquadmath-11.4.1-3.el9.x86_64.rpm 239 kB/s | 195 kB 00:00 2024-08-17T16:59:28.780 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:01 ago on Sat 17 Aug 2024 04:58:27 PM UTC. 2024-08-17T16:59:28.860 INFO:teuthology.orchestra.run.smithi053.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 157 kB/s | 18 kB 00:00 2024-08-17T16:59:28.913 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-08-17T16:59:28.914 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================= 2024-08-17T16:59:28.914 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-08-17T16:59:28.914 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================= 2024-08-17T16:59:28.914 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-08-17T16:59:28.914 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1404.gf2c39d4d.el9 ceph-noarch 7.4 M 2024-08-17T16:59:28.915 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-08-17T16:59:28.915 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-17T16:59:28.915 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-17T16:59:28.915 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-17T16:59:28.915 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-08-17T16:59:28.915 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-08-17T16:59:28.915 INFO:teuthology.orchestra.run.smithi003.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-08-17T16:59:28.915 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-08-17T16:59:28.915 INFO:teuthology.orchestra.run.smithi003.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-08-17T16:59:28.915 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-17T16:59:28.915 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-17T16:59:28.915 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-17T16:59:28.916 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:28.916 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-08-17T16:59:28.916 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================= 2024-08-17T16:59:28.916 INFO:teuthology.orchestra.run.smithi003.stdout:Install 12 Packages 2024-08-17T16:59:28.916 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:28.916 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 38 M 2024-08-17T16:59:28.916 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 209 M 2024-08-17T16:59:28.916 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-08-17T16:59:28.944 INFO:teuthology.orchestra.run.smithi053.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 422 kB/s | 35 kB 00:00 2024-08-17T16:59:28.994 INFO:teuthology.orchestra.run.smithi053.stdout:(6/12): libgfortran-11.4.1-3.el9.x86_64.rpm 752 kB/s | 803 kB 00:01 2024-08-17T16:59:29.195 INFO:teuthology.orchestra.run.smithi053.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 1.2 MB/s | 244 kB 00:00 2024-08-17T16:59:29.480 INFO:teuthology.orchestra.run.smithi003.stdout:(1/12): libgfortran-11.4.1-3.el9.x86_64.rpm 5.1 MB/s | 803 kB 00:00 2024-08-17T16:59:29.520 INFO:teuthology.orchestra.run.smithi053.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.0 MB/s | 3.0 MB 00:00 2024-08-17T16:59:29.530 INFO:teuthology.orchestra.run.smithi003.stdout:(2/12): libquadmath-11.4.1-3.el9.x86_64.rpm 963 kB/s | 195 kB 00:00 2024-08-17T16:59:29.637 INFO:teuthology.orchestra.run.smithi053.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 250 kB 00:00 2024-08-17T16:59:29.756 INFO:teuthology.orchestra.run.smithi003.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 119 kB/s | 33 kB 00:00 2024-08-17T16:59:29.839 INFO:teuthology.orchestra.run.smithi003.stdout:(4/12): ceph-mgr-diskprediction-local-17.2.7-14 14 MB/s | 7.4 MB 00:00 2024-08-17T16:59:30.040 INFO:teuthology.orchestra.run.smithi003.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 64 kB/s | 18 kB 00:00 2024-08-17T16:59:30.115 INFO:teuthology.orchestra.run.smithi003.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 128 kB/s | 35 kB 00:00 2024-08-17T16:59:30.365 INFO:teuthology.orchestra.run.smithi003.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.6 MB/s | 3.0 MB 00:00 2024-08-17T16:59:30.623 INFO:teuthology.orchestra.run.smithi003.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 480 kB/s | 244 kB 00:00 2024-08-17T16:59:30.863 INFO:teuthology.orchestra.run.smithi053.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 2.5 MB/s | 4.7 MB 00:01 2024-08-17T16:59:31.212 INFO:teuthology.orchestra.run.smithi053.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.5 MB/s | 5.1 MB 00:02 2024-08-17T16:59:31.249 INFO:teuthology.orchestra.run.smithi003.stdout:(9/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 3.9 MB/s | 4.7 MB 00:01 2024-08-17T16:59:31.274 INFO:teuthology.orchestra.run.smithi003.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 385 kB/s | 250 kB 00:00 2024-08-17T16:59:32.407 INFO:teuthology.orchestra.run.smithi003.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.5 MB/s | 5.1 MB 00:02 2024-08-17T16:59:33.062 INFO:teuthology.orchestra.run.smithi053.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 4.7 MB/s | 16 MB 00:03 2024-08-17T16:59:33.063 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-08-17T16:59:33.063 INFO:teuthology.orchestra.run.smithi053.stdout:Total 7.1 MB/s | 38 MB 00:05 2024-08-17T16:59:33.299 INFO:teuthology.orchestra.run.smithi003.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 7.9 MB/s | 16 MB 00:02 2024-08-17T16:59:33.300 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-08-17T16:59:33.300 INFO:teuthology.orchestra.run.smithi003.stdout:Total 8.7 MB/s | 38 MB 00:04 2024-08-17T16:59:33.430 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-08-17T16:59:33.457 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-08-17T16:59:33.458 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-08-17T16:59:33.668 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-08-17T16:59:33.695 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-08-17T16:59:33.695 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-08-17T16:59:33.996 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-08-17T16:59:33.996 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-08-17T16:59:34.235 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-08-17T16:59:34.236 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-08-17T16:59:34.788 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-08-17T16:59:34.864 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-08-17T16:59:34.937 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-08-17T16:59:35.029 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-08-17T16:59:35.048 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-08-17T16:59:35.087 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-08-17T16:59:35.100 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-08-17T16:59:35.150 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-08-17T16:59:35.250 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-08-17T16:59:35.273 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-08-17T16:59:35.282 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-08-17T16:59:35.310 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-08-17T16:59:35.403 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-08-17T16:59:35.465 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-08-17T16:59:35.497 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-08-17T16:59:35.587 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-08-17T16:59:36.099 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-08-17T16:59:36.211 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-08-17T16:59:36.301 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-08-17T16:59:36.422 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-08-17T16:59:37.547 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-08-17T16:59:37.739 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-08-17T16:59:38.640 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-08-17T16:59:38.667 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1404.gf2c39 12/12 2024-08-17T16:59:38.815 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-08-17T16:59:38.845 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1404.gf2c39 12/12 2024-08-17T16:59:41.100 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1404.gf2c39 12/12 2024-08-17T16:59:41.100 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1404.gf2c39 1/12 2024-08-17T16:59:41.100 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-08-17T16:59:41.100 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-08-17T16:59:41.100 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-08-17T16:59:41.100 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-08-17T16:59:41.100 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-08-17T16:59:41.100 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-08-17T16:59:41.100 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-08-17T16:59:41.101 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-08-17T16:59:41.101 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-08-17T16:59:41.101 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-08-17T16:59:41.454 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1404.gf2c39 12/12 2024-08-17T16:59:41.454 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1404.gf2c39 1/12 2024-08-17T16:59:41.455 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-08-17T16:59:41.455 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-08-17T16:59:41.455 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-08-17T16:59:41.455 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-08-17T16:59:41.455 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-08-17T16:59:41.455 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-08-17T16:59:41.455 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-08-17T16:59:41.455 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-08-17T16:59:41.455 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-08-17T16:59:41.455 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-08-17T16:59:41.492 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-08-17T16:59:41.492 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:41.492 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-08-17T16:59:41.492 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1404.gf2c39d4d.el9.noarch 2024-08-17T16:59:41.492 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-17T16:59:41.492 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-17T16:59:41.493 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-17T16:59:41.493 INFO:teuthology.orchestra.run.smithi053.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-08-17T16:59:41.493 INFO:teuthology.orchestra.run.smithi053.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-08-17T16:59:41.493 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-0.3.21-2.el9.x86_64 2024-08-17T16:59:41.493 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-08-17T16:59:41.493 INFO:teuthology.orchestra.run.smithi053.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-08-17T16:59:41.493 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-17T16:59:41.493 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-17T16:59:41.493 INFO:teuthology.orchestra.run.smithi053.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-17T16:59:41.493 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:41.493 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-08-17T16:59:41.878 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr-rook 2024-08-17T16:59:41.898 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-08-17T16:59:41.898 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:41.899 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-08-17T16:59:41.899 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1404.gf2c39d4d.el9.noarch 2024-08-17T16:59:41.899 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-17T16:59:41.899 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-17T16:59:41.899 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-17T16:59:41.899 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-08-17T16:59:41.899 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-08-17T16:59:41.899 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-0.3.21-2.el9.x86_64 2024-08-17T16:59:41.899 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-08-17T16:59:41.899 INFO:teuthology.orchestra.run.smithi003.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-08-17T16:59:41.899 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-17T16:59:41.899 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-17T16:59:41.900 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-17T16:59:41.900 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:41.900 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-08-17T16:59:42.277 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-rook 2024-08-17T16:59:42.460 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:16 ago on Sat 17 Aug 2024 04:58:26 PM UTC. 2024-08-17T16:59:42.598 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-08-17T16:59:42.599 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:59:42.599 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-08-17T16:59:42.599 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:59:42.599 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-08-17T16:59:42.600 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-rook noarch 2:17.2.7-1404.gf2c39d4d.el9 ceph-noarch 47 k 2024-08-17T16:59:42.600 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-08-17T16:59:42.600 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-17T16:59:42.600 INFO:teuthology.orchestra.run.smithi053.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-17T16:59:42.600 INFO:teuthology.orchestra.run.smithi053.stdout: python3-google-auth noarch 1:2.33.0-1.el9 epel 220 k 2024-08-17T16:59:42.600 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-17T16:59:42.600 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-17T16:59:42.600 INFO:teuthology.orchestra.run.smithi053.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-17T16:59:42.600 INFO:teuthology.orchestra.run.smithi053.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-17T16:59:42.600 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-17T16:59:42.600 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-17T16:59:42.600 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-17T16:59:42.601 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-17T16:59:42.601 INFO:teuthology.orchestra.run.smithi053.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-17T16:59:42.601 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:42.601 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-08-17T16:59:42.601 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:59:42.601 INFO:teuthology.orchestra.run.smithi053.stdout:Install 13 Packages 2024-08-17T16:59:42.601 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:42.601 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 2.2 M 2024-08-17T16:59:42.601 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 26 M 2024-08-17T16:59:42.602 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-08-17T16:59:42.859 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:15 ago on Sat 17 Aug 2024 04:58:27 PM UTC. 2024-08-17T16:59:42.883 INFO:teuthology.orchestra.run.smithi053.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 186 kB/s | 19 kB 00:00 2024-08-17T16:59:42.900 INFO:teuthology.orchestra.run.smithi053.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 213 kB/s | 26 kB 00:00 2024-08-17T16:59:42.959 INFO:teuthology.orchestra.run.smithi053.stdout:(3/13): ceph-mgr-rook-17.2.7-1404.gf2c39d4d.el9 266 kB/s | 47 kB 00:00 2024-08-17T16:59:42.992 INFO:teuthology.orchestra.run.smithi053.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-08-17T16:59:42.996 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-08-17T16:59:42.997 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:59:42.997 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-08-17T16:59:42.997 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:59:42.998 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-08-17T16:59:42.998 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook noarch 2:17.2.7-1404.gf2c39d4d.el9 ceph-noarch 47 k 2024-08-17T16:59:42.998 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-08-17T16:59:42.998 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-17T16:59:42.998 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-17T16:59:42.998 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth noarch 1:2.33.0-1.el9 epel 220 k 2024-08-17T16:59:42.998 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-17T16:59:42.998 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-17T16:59:42.998 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-17T16:59:42.998 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-17T16:59:42.998 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-17T16:59:42.999 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-17T16:59:42.999 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-17T16:59:42.999 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-17T16:59:42.999 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-17T16:59:42.999 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:42.999 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-08-17T16:59:42.999 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:59:42.999 INFO:teuthology.orchestra.run.smithi003.stdout:Install 13 Packages 2024-08-17T16:59:42.999 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:43.000 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 2.2 M 2024-08-17T16:59:43.000 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 26 M 2024-08-17T16:59:43.000 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-08-17T16:59:43.017 INFO:teuthology.orchestra.run.smithi053.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-08-17T16:59:43.059 INFO:teuthology.orchestra.run.smithi053.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 801 kB/s | 54 kB 00:00 2024-08-17T16:59:43.102 INFO:teuthology.orchestra.run.smithi053.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 385 kB/s | 32 kB 00:00 2024-08-17T16:59:43.118 INFO:teuthology.orchestra.run.smithi053.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 252 kB/s | 14 kB 00:00 2024-08-17T16:59:43.143 INFO:teuthology.orchestra.run.smithi053.stdout:(9/13): python3-google-auth-2.33.0-1.el9.noarch 5.4 MB/s | 220 kB 00:00 2024-08-17T16:59:43.169 INFO:teuthology.orchestra.run.smithi053.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-08-17T16:59:43.194 INFO:teuthology.orchestra.run.smithi053.stdout:(11/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.2 MB/s | 279 kB 00:00 2024-08-17T16:59:43.219 INFO:teuthology.orchestra.run.smithi053.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-08-17T16:59:43.252 INFO:teuthology.orchestra.run.smithi053.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.6 MB/s | 1.0 MB 00:00 2024-08-17T16:59:43.254 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-08-17T16:59:43.255 INFO:teuthology.orchestra.run.smithi053.stdout:Total 3.4 MB/s | 2.2 MB 00:00 2024-08-17T16:59:43.320 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-08-17T16:59:43.336 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-08-17T16:59:43.336 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-08-17T16:59:43.395 INFO:teuthology.orchestra.run.smithi003.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 198 kB/s | 26 kB 00:00 2024-08-17T16:59:43.420 INFO:teuthology.orchestra.run.smithi003.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 124 kB/s | 19 kB 00:00 2024-08-17T16:59:43.431 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-08-17T16:59:43.432 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-08-17T16:59:43.446 INFO:teuthology.orchestra.run.smithi003.stdout:(3/13): ceph-mgr-rook-17.2.7-1404.gf2c39d4d.el9 264 kB/s | 47 kB 00:00 2024-08-17T16:59:43.554 INFO:teuthology.orchestra.run.smithi003.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.6 MB/s | 222 kB 00:00 2024-08-17T16:59:43.605 INFO:teuthology.orchestra.run.smithi003.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 863 kB/s | 159 kB 00:00 2024-08-17T16:59:43.630 INFO:teuthology.orchestra.run.smithi003.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 713 kB/s | 54 kB 00:00 2024-08-17T16:59:43.714 INFO:teuthology.orchestra.run.smithi003.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.0 MB/s | 279 kB 00:00 2024-08-17T16:59:43.738 INFO:teuthology.orchestra.run.smithi003.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 241 kB/s | 32 kB 00:00 2024-08-17T16:59:43.789 INFO:teuthology.orchestra.run.smithi003.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 89 kB/s | 14 kB 00:00 2024-08-17T16:59:43.814 INFO:teuthology.orchestra.run.smithi003.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.6 MB/s | 59 kB 00:00 2024-08-17T16:59:43.839 INFO:teuthology.orchestra.run.smithi003.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.5 MB/s | 90 kB 00:00 2024-08-17T16:59:43.865 INFO:teuthology.orchestra.run.smithi003.stdout:(12/13): python3-google-auth-2.33.0-1.el9.noarc 1.4 MB/s | 220 kB 00:00 2024-08-17T16:59:43.907 INFO:teuthology.orchestra.run.smithi003.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.1 MB/s | 1.0 MB 00:00 2024-08-17T16:59:43.910 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-08-17T16:59:43.910 INFO:teuthology.orchestra.run.smithi003.stdout:Total 2.4 MB/s | 2.2 MB 00:00 2024-08-17T16:59:43.949 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-08-17T16:59:43.974 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-08-17T16:59:43.990 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-08-17T16:59:43.991 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-08-17T16:59:44.087 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-08-17T16:59:44.088 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-08-17T16:59:44.125 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-08-17T16:59:44.189 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-08-17T16:59:44.254 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-08-17T16:59:44.289 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-08-17T16:59:44.331 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-08-17T16:59:44.446 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-08-17T16:59:44.556 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-google-auth-1:2.33.0-1.el9.noarch 7/13 2024-08-17T16:59:44.608 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-08-17T16:59:44.734 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-08-17T16:59:44.911 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-08-17T16:59:44.976 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-08-17T16:59:45.041 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-08-17T16:59:45.084 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-08-17T16:59:45.126 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-08-17T16:59:45.233 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-08-17T16:59:45.342 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-google-auth-1:2.33.0-1.el9.noarch 7/13 2024-08-17T16:59:45.369 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-08-17T16:59:45.395 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-08-17T16:59:45.427 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-08-17T16:59:45.461 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-08-17T16:59:45.507 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-08-17T16:59:45.534 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-rook-2:17.2.7-1404.gf2c39d4d.el9.noarch 13/13 2024-08-17T16:59:46.162 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-08-17T16:59:46.215 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-08-17T16:59:46.266 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-08-17T16:59:46.314 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-08-17T16:59:46.332 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1404.gf2c39d4d.el9.noarch 13/13 2024-08-17T16:59:46.332 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1404.gf2c39d4d.el9.noarch 1/13 2024-08-17T16:59:46.332 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-08-17T16:59:46.333 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-08-17T16:59:46.333 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-08-17T16:59:46.333 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-08-17T16:59:46.333 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-08-17T16:59:46.333 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-08-17T16:59:46.333 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-08-17T16:59:46.333 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-08-17T16:59:46.333 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-google-auth-1:2.33.0-1.el9.noarch 10/13 2024-08-17T16:59:46.333 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-08-17T16:59:46.333 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-08-17T16:59:46.337 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-rook-2:17.2.7-1404.gf2c39d4d.el9.noarch 13/13 2024-08-17T16:59:46.657 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-08-17T16:59:46.657 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:46.657 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-08-17T16:59:46.657 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-rook-2:17.2.7-1404.gf2c39d4d.el9.noarch 2024-08-17T16:59:46.657 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-17T16:59:46.657 INFO:teuthology.orchestra.run.smithi053.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-17T16:59:46.657 INFO:teuthology.orchestra.run.smithi053.stdout: python3-google-auth-1:2.33.0-1.el9.noarch 2024-08-17T16:59:46.657 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-17T16:59:46.657 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-17T16:59:46.658 INFO:teuthology.orchestra.run.smithi053.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-17T16:59:46.658 INFO:teuthology.orchestra.run.smithi053.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-17T16:59:46.658 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-17T16:59:46.658 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-17T16:59:46.658 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-17T16:59:46.658 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-17T16:59:46.658 INFO:teuthology.orchestra.run.smithi053.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-17T16:59:46.658 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:46.658 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-08-17T16:59:46.898 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr-cephadm 2024-08-17T16:59:47.130 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1404.gf2c39d4d.el9.noarch 13/13 2024-08-17T16:59:47.130 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1404.gf2c39d4d.el9.noarch 1/13 2024-08-17T16:59:47.130 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-08-17T16:59:47.130 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-08-17T16:59:47.130 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-08-17T16:59:47.131 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-08-17T16:59:47.131 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-08-17T16:59:47.131 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-08-17T16:59:47.131 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-08-17T16:59:47.131 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-08-17T16:59:47.131 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-google-auth-1:2.33.0-1.el9.noarch 10/13 2024-08-17T16:59:47.131 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-08-17T16:59:47.131 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-08-17T16:59:47.462 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-08-17T16:59:47.462 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:47.462 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-08-17T16:59:47.462 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook-2:17.2.7-1404.gf2c39d4d.el9.noarch 2024-08-17T16:59:47.462 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-17T16:59:47.463 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-17T16:59:47.463 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth-1:2.33.0-1.el9.noarch 2024-08-17T16:59:47.463 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-17T16:59:47.463 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-17T16:59:47.463 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-17T16:59:47.463 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-17T16:59:47.463 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-17T16:59:47.463 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-17T16:59:47.463 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-17T16:59:47.463 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-17T16:59:47.463 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-17T16:59:47.463 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:47.463 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-08-17T16:59:47.475 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:21 ago on Sat 17 Aug 2024 04:58:26 PM UTC. 2024-08-17T16:59:47.610 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-08-17T16:59:47.612 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:59:47.612 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-08-17T16:59:47.612 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:59:47.612 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-08-17T16:59:47.612 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1404.gf2c39d4d.el9 ceph-noarch 124 k 2024-08-17T16:59:47.612 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-08-17T16:59:47.612 INFO:teuthology.orchestra.run.smithi053.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-17T16:59:47.612 INFO:teuthology.orchestra.run.smithi053.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-17T16:59:47.612 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-17T16:59:47.613 INFO:teuthology.orchestra.run.smithi053.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-17T16:59:47.613 INFO:teuthology.orchestra.run.smithi053.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-17T16:59:47.613 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:47.613 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-08-17T16:59:47.613 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:59:47.613 INFO:teuthology.orchestra.run.smithi053.stdout:Install 6 Packages 2024-08-17T16:59:47.613 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:47.613 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 7.0 M 2024-08-17T16:59:47.613 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 33 M 2024-08-17T16:59:47.613 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-08-17T16:59:47.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-cephadm 2024-08-17T16:59:47.992 INFO:teuthology.orchestra.run.smithi053.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.0 MB/s | 248 kB 00:00 2024-08-17T16:59:48.017 INFO:teuthology.orchestra.run.smithi053.stdout:(2/6): ceph-mgr-cephadm-17.2.7-1404.gf2c39d4d.e 476 kB/s | 124 kB 00:00 2024-08-17T16:59:48.100 INFO:teuthology.orchestra.run.smithi053.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 693 kB/s | 58 kB 00:00 2024-08-17T16:59:48.126 INFO:teuthology.orchestra.run.smithi053.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-08-17T16:59:48.151 INFO:teuthology.orchestra.run.smithi053.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.4 MB/s | 548 kB 00:00 2024-08-17T16:59:48.299 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:21 ago on Sat 17 Aug 2024 04:58:27 PM UTC. 2024-08-17T16:59:48.351 INFO:teuthology.orchestra.run.smithi053.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 10 MB/s | 6.0 MB 00:00 2024-08-17T16:59:48.353 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-08-17T16:59:48.354 INFO:teuthology.orchestra.run.smithi053.stdout:Total 9.4 MB/s | 7.0 MB 00:00 2024-08-17T16:59:48.427 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-08-17T16:59:48.432 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-08-17T16:59:48.432 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:59:48.433 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-08-17T16:59:48.433 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:59:48.433 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-08-17T16:59:48.433 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1404.gf2c39d4d.el9 ceph-noarch 124 k 2024-08-17T16:59:48.433 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-08-17T16:59:48.433 INFO:teuthology.orchestra.run.smithi003.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-17T16:59:48.433 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-17T16:59:48.433 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-17T16:59:48.433 INFO:teuthology.orchestra.run.smithi003.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-17T16:59:48.433 INFO:teuthology.orchestra.run.smithi003.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-17T16:59:48.433 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:48.433 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-08-17T16:59:48.434 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:59:48.434 INFO:teuthology.orchestra.run.smithi003.stdout:Install 6 Packages 2024-08-17T16:59:48.434 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:48.434 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 7.0 M 2024-08-17T16:59:48.434 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 33 M 2024-08-17T16:59:48.434 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-08-17T16:59:48.439 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-08-17T16:59:48.439 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-08-17T16:59:48.544 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-08-17T16:59:48.546 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-08-17T16:59:48.859 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-08-17T16:59:49.035 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-08-17T16:59:49.079 INFO:teuthology.orchestra.run.smithi003.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-08-17T16:59:49.088 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-08-17T16:59:49.129 INFO:teuthology.orchestra.run.smithi003.stdout:(2/6): ceph-mgr-cephadm-17.2.7-1404.gf2c39d4d.e 558 kB/s | 124 kB 00:00 2024-08-17T16:59:49.187 INFO:teuthology.orchestra.run.smithi003.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.9 MB/s | 548 kB 00:00 2024-08-17T16:59:49.213 INFO:teuthology.orchestra.run.smithi003.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 3.2 MB/s | 74 kB 00:00 2024-08-17T16:59:49.238 INFO:teuthology.orchestra.run.smithi003.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 539 kB/s | 58 kB 00:00 2024-08-17T16:59:49.314 INFO:teuthology.orchestra.run.smithi003.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-08-17T16:59:49.316 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-08-17T16:59:49.316 INFO:teuthology.orchestra.run.smithi003.stdout:Total 7.9 MB/s | 7.0 MB 00:00 2024-08-17T16:59:49.389 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-08-17T16:59:49.402 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-08-17T16:59:49.402 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-08-17T16:59:49.505 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-08-17T16:59:49.505 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-08-17T16:59:49.644 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-08-17T16:59:49.735 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-08-17T16:59:49.814 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-08-17T16:59:49.814 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-08-17T16:59:49.841 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1404.gf2c39d4d.el9.noarch 6/6 2024-08-17T16:59:49.983 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-08-17T16:59:50.034 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-08-17T16:59:50.604 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-08-17T16:59:50.629 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1404.gf2c39d4d.el9.noarch 6/6 2024-08-17T16:59:50.629 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1404.gf2c39d4d.el9.noarch 1/6 2024-08-17T16:59:50.629 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-08-17T16:59:50.629 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-08-17T16:59:50.629 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-08-17T16:59:50.629 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-08-17T16:59:50.699 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-08-17T16:59:50.760 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-08-17T16:59:50.786 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1404.gf2c39d4d.el9.noarch 6/6 2024-08-17T16:59:50.897 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-08-17T16:59:50.898 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:50.898 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-08-17T16:59:50.898 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-cephadm-2:17.2.7-1404.gf2c39d4d.el9.noarch 2024-08-17T16:59:50.898 INFO:teuthology.orchestra.run.smithi053.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-17T16:59:50.898 INFO:teuthology.orchestra.run.smithi053.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-17T16:59:50.899 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-17T16:59:50.899 INFO:teuthology.orchestra.run.smithi053.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-17T16:59:50.899 INFO:teuthology.orchestra.run.smithi053.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-17T16:59:50.899 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:50.899 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-08-17T16:59:51.153 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-fuse 2024-08-17T16:59:51.549 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1404.gf2c39d4d.el9.noarch 6/6 2024-08-17T16:59:51.549 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1404.gf2c39d4d.el9.noarch 1/6 2024-08-17T16:59:51.549 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-08-17T16:59:51.550 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-08-17T16:59:51.550 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-08-17T16:59:51.550 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-08-17T16:59:51.729 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:25 ago on Sat 17 Aug 2024 04:58:26 PM UTC. 2024-08-17T16:59:51.836 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-08-17T16:59:51.836 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:51.836 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-08-17T16:59:51.836 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm-2:17.2.7-1404.gf2c39d4d.el9.noarch 2024-08-17T16:59:51.836 INFO:teuthology.orchestra.run.smithi003.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-17T16:59:51.837 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-17T16:59:51.837 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-17T16:59:51.837 INFO:teuthology.orchestra.run.smithi003.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-17T16:59:51.837 INFO:teuthology.orchestra.run.smithi003.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-17T16:59:51.837 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:51.837 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-08-17T16:59:51.864 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-08-17T16:59:51.864 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:59:51.864 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-08-17T16:59:51.864 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:59:51.864 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-08-17T16:59:51.864 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-fuse x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 813 k 2024-08-17T16:59:51.865 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-08-17T16:59:51.865 INFO:teuthology.orchestra.run.smithi053.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-17T16:59:51.865 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:51.865 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-08-17T16:59:51.865 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:59:51.865 INFO:teuthology.orchestra.run.smithi053.stdout:Install 2 Packages 2024-08-17T16:59:51.865 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:51.865 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 893 k 2024-08-17T16:59:51.865 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 2.6 M 2024-08-17T16:59:51.866 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-08-17T16:59:52.076 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-fuse 2024-08-17T16:59:52.271 INFO:teuthology.orchestra.run.smithi053.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 613 kB/s | 80 kB 00:00 2024-08-17T16:59:52.463 INFO:teuthology.orchestra.run.smithi053.stdout:(2/2): ceph-fuse-17.2.7-1404.gf2c39d4d.el9.x86_ 2.5 MB/s | 813 kB 00:00 2024-08-17T16:59:52.464 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-08-17T16:59:52.464 INFO:teuthology.orchestra.run.smithi053.stdout:Total 1.5 MB/s | 893 kB 00:00 2024-08-17T16:59:52.476 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-08-17T16:59:52.500 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-08-17T16:59:52.500 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-08-17T16:59:52.585 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-08-17T16:59:52.585 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-08-17T16:59:52.658 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:25 ago on Sat 17 Aug 2024 04:58:27 PM UTC. 2024-08-17T16:59:52.776 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-08-17T16:59:52.792 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-08-17T16:59:52.793 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:59:52.793 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-08-17T16:59:52.793 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:59:52.793 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-08-17T16:59:52.793 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 813 k 2024-08-17T16:59:52.793 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-08-17T16:59:52.793 INFO:teuthology.orchestra.run.smithi003.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-17T16:59:52.793 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:52.793 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-08-17T16:59:52.793 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:59:52.793 INFO:teuthology.orchestra.run.smithi003.stdout:Install 2 Packages 2024-08-17T16:59:52.793 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:52.794 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 893 k 2024-08-17T16:59:52.794 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 2.6 M 2024-08-17T16:59:52.794 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-08-17T16:59:52.913 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-17T16:59:52.991 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-fuse-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2/2 2024-08-17T16:59:53.193 INFO:teuthology.orchestra.run.smithi003.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 640 kB/s | 80 kB 00:00 2024-08-17T16:59:53.428 INFO:teuthology.orchestra.run.smithi003.stdout:(2/2): ceph-fuse-17.2.7-1404.gf2c39d4d.el9.x86_ 2.2 MB/s | 813 kB 00:00 2024-08-17T16:59:53.429 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-08-17T16:59:53.429 INFO:teuthology.orchestra.run.smithi003.stdout:Total 1.4 MB/s | 893 kB 00:00 2024-08-17T16:59:53.441 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-08-17T16:59:53.464 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-08-17T16:59:53.464 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-08-17T16:59:53.547 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-08-17T16:59:53.547 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-08-17T16:59:53.652 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2/2 2024-08-17T16:59:53.652 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-fuse-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/2 2024-08-17T16:59:53.738 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-08-17T16:59:53.877 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-17T16:59:53.924 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-17T16:59:53.925 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:53.925 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-08-17T16:59:53.925 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-fuse-2:17.2.7-1404.gf2c39d4d.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-17T16:59:53.925 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:53.925 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-08-17T16:59:53.964 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-fuse-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2/2 2024-08-17T16:59:54.194 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install librados-devel 2024-08-17T16:59:54.673 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2/2 2024-08-17T16:59:54.674 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-fuse-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/2 2024-08-17T16:59:54.772 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:28 ago on Sat 17 Aug 2024 04:58:26 PM UTC. 2024-08-17T16:59:54.907 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-08-17T16:59:54.907 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:59:54.907 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-08-17T16:59:54.907 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:59:54.907 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-08-17T16:59:54.907 INFO:teuthology.orchestra.run.smithi053.stdout: librados-devel x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 126 k 2024-08-17T16:59:54.907 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:54.907 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-08-17T16:59:54.908 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:59:54.908 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-08-17T16:59:54.908 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:54.908 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 126 k 2024-08-17T16:59:54.908 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 448 k 2024-08-17T16:59:54.908 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-08-17T16:59:54.950 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-17T16:59:54.950 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:54.950 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-08-17T16:59:54.950 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse-2:17.2.7-1404.gf2c39d4d.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-17T16:59:54.950 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:54.951 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-08-17T16:59:55.135 INFO:teuthology.orchestra.run.smithi053.stdout:librados-devel-17.2.7-1404.gf2c39d4d.el9.x86_64 557 kB/s | 126 kB 00:00 2024-08-17T16:59:55.135 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-08-17T16:59:55.135 INFO:teuthology.orchestra.run.smithi053.stdout:Total 554 kB/s | 126 kB 00:00 2024-08-17T16:59:55.136 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-08-17T16:59:55.142 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-08-17T16:59:55.142 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-08-17T16:59:55.209 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-08-17T16:59:55.210 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-08-17T16:59:55.224 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librados-devel 2024-08-17T16:59:55.364 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-08-17T16:59:55.479 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librados-devel-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/1 2024-08-17T16:59:55.806 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:28 ago on Sat 17 Aug 2024 04:58:27 PM UTC. 2024-08-17T16:59:55.901 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados-devel-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/1 2024-08-17T16:59:55.939 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-08-17T16:59:55.940 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:59:55.940 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-08-17T16:59:55.940 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:59:55.940 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-08-17T16:59:55.940 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 126 k 2024-08-17T16:59:55.940 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:55.940 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-08-17T16:59:55.940 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:59:55.941 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-08-17T16:59:55.941 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:55.941 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 126 k 2024-08-17T16:59:55.941 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 448 k 2024-08-17T16:59:55.941 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-08-17T16:59:56.138 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados-devel-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/1 2024-08-17T16:59:56.138 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:56.138 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-08-17T16:59:56.138 INFO:teuthology.orchestra.run.smithi053.stdout: librados-devel-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:59:56.138 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:56.138 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-08-17T16:59:56.174 INFO:teuthology.orchestra.run.smithi003.stdout:librados-devel-17.2.7-1404.gf2c39d4d.el9.x86_64 542 kB/s | 126 kB 00:00 2024-08-17T16:59:56.174 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-08-17T16:59:56.175 INFO:teuthology.orchestra.run.smithi003.stdout:Total 539 kB/s | 126 kB 00:00 2024-08-17T16:59:56.175 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-08-17T16:59:56.180 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-08-17T16:59:56.180 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-08-17T16:59:56.247 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-08-17T16:59:56.248 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-08-17T16:59:56.380 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install libcephfs2 2024-08-17T16:59:56.404 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-08-17T16:59:56.634 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librados-devel-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/1 2024-08-17T16:59:56.956 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:30 ago on Sat 17 Aug 2024 04:58:26 PM UTC. 2024-08-17T16:59:57.047 INFO:teuthology.orchestra.run.smithi053.stdout:Package libcephfs2-2:17.2.7-1404.gf2c39d4d.el9.x86_64 is already installed. 2024-08-17T16:59:57.092 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-08-17T16:59:57.093 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-08-17T16:59:57.093 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-08-17T16:59:57.097 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados-devel-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/1 2024-08-17T16:59:57.159 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install libcephfs-devel 2024-08-17T16:59:57.352 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados-devel-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/1 2024-08-17T16:59:57.352 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:57.352 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-08-17T16:59:57.352 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:59:57.352 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:57.352 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-08-17T16:59:57.569 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install libcephfs2 2024-08-17T16:59:57.738 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:31 ago on Sat 17 Aug 2024 04:58:26 PM UTC. 2024-08-17T16:59:57.871 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-08-17T16:59:57.872 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:59:57.872 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-08-17T16:59:57.872 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:59:57.872 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-08-17T16:59:57.873 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs-devel x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 25 k 2024-08-17T16:59:57.873 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:57.873 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-08-17T16:59:57.873 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T16:59:57.873 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-08-17T16:59:57.873 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:57.873 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 25 k 2024-08-17T16:59:57.873 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 110 k 2024-08-17T16:59:57.873 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-08-17T16:59:58.039 INFO:teuthology.orchestra.run.smithi053.stdout:libcephfs-devel-17.2.7-1404.gf2c39d4d.el9.x86_6 151 kB/s | 25 kB 00:00 2024-08-17T16:59:58.039 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-08-17T16:59:58.039 INFO:teuthology.orchestra.run.smithi053.stdout:Total 149 kB/s | 25 kB 00:00 2024-08-17T16:59:58.040 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-08-17T16:59:58.043 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-08-17T16:59:58.043 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-08-17T16:59:58.045 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-08-17T16:59:58.045 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-08-17T16:59:58.121 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-08-17T16:59:58.146 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:31 ago on Sat 17 Aug 2024 04:58:27 PM UTC. 2024-08-17T16:59:58.239 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephfs-devel-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/1 2024-08-17T16:59:58.239 INFO:teuthology.orchestra.run.smithi003.stdout:Package libcephfs2-2:17.2.7-1404.gf2c39d4d.el9.x86_64 is already installed. 2024-08-17T16:59:58.284 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-08-17T16:59:58.285 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-08-17T16:59:58.285 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-08-17T16:59:58.349 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install libcephfs-devel 2024-08-17T16:59:58.612 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/1 2024-08-17T16:59:58.837 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephfs-devel-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/1 2024-08-17T16:59:58.838 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:58.838 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-08-17T16:59:58.838 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs-devel-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T16:59:58.838 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T16:59:58.838 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-08-17T16:59:58.919 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:31 ago on Sat 17 Aug 2024 04:58:27 PM UTC. 2024-08-17T16:59:59.006 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install librados2 2024-08-17T16:59:59.054 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-08-17T16:59:59.054 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:59:59.054 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-08-17T16:59:59.054 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:59:59.054 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-08-17T16:59:59.054 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 25 k 2024-08-17T16:59:59.054 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:59.055 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-08-17T16:59:59.055 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T16:59:59.055 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-08-17T16:59:59.055 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T16:59:59.055 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 25 k 2024-08-17T16:59:59.055 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 110 k 2024-08-17T16:59:59.055 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-08-17T16:59:59.225 INFO:teuthology.orchestra.run.smithi003.stdout:libcephfs-devel-17.2.7-1404.gf2c39d4d.el9.x86_6 147 kB/s | 25 kB 00:00 2024-08-17T16:59:59.226 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-08-17T16:59:59.226 INFO:teuthology.orchestra.run.smithi003.stdout:Total 146 kB/s | 25 kB 00:00 2024-08-17T16:59:59.227 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-08-17T16:59:59.230 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-08-17T16:59:59.230 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-08-17T16:59:59.231 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-08-17T16:59:59.232 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-08-17T16:59:59.311 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-08-17T16:59:59.424 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs-devel-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/1 2024-08-17T16:59:59.582 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:33 ago on Sat 17 Aug 2024 04:58:26 PM UTC. 2024-08-17T16:59:59.671 INFO:teuthology.orchestra.run.smithi053.stdout:Package librados2-2:17.2.7-1404.gf2c39d4d.el9.x86_64 is already installed. 2024-08-17T16:59:59.717 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-08-17T16:59:59.718 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-08-17T16:59:59.719 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-08-17T16:59:59.783 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install librbd1 2024-08-17T16:59:59.788 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/1 2024-08-17T17:00:00.018 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs-devel-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/1 2024-08-17T17:00:00.019 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:00:00.019 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-08-17T17:00:00.019 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T17:00:00.019 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:00:00.019 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-08-17T17:00:00.197 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librados2 2024-08-17T17:00:00.359 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:34 ago on Sat 17 Aug 2024 04:58:26 PM UTC. 2024-08-17T17:00:00.448 INFO:teuthology.orchestra.run.smithi053.stdout:Package librbd1-2:17.2.7-1404.gf2c39d4d.el9.x86_64 is already installed. 2024-08-17T17:00:00.493 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-08-17T17:00:00.495 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-08-17T17:00:00.495 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-08-17T17:00:00.567 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-rados 2024-08-17T17:00:00.774 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:33 ago on Sat 17 Aug 2024 04:58:27 PM UTC. 2024-08-17T17:00:00.862 INFO:teuthology.orchestra.run.smithi003.stdout:Package librados2-2:17.2.7-1404.gf2c39d4d.el9.x86_64 is already installed. 2024-08-17T17:00:00.906 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-08-17T17:00:00.907 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-08-17T17:00:00.908 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-08-17T17:00:00.970 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librbd1 2024-08-17T17:00:01.137 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:35 ago on Sat 17 Aug 2024 04:58:26 PM UTC. 2024-08-17T17:00:01.225 INFO:teuthology.orchestra.run.smithi053.stdout:Package python3-rados-2:17.2.7-1404.gf2c39d4d.el9.x86_64 is already installed. 2024-08-17T17:00:01.270 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-08-17T17:00:01.271 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-08-17T17:00:01.271 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-08-17T17:00:01.341 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-rgw 2024-08-17T17:00:01.547 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:34 ago on Sat 17 Aug 2024 04:58:27 PM UTC. 2024-08-17T17:00:01.636 INFO:teuthology.orchestra.run.smithi003.stdout:Package librbd1-2:17.2.7-1404.gf2c39d4d.el9.x86_64 is already installed. 2024-08-17T17:00:01.680 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-08-17T17:00:01.682 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-08-17T17:00:01.682 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-08-17T17:00:01.741 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rados 2024-08-17T17:00:01.916 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:35 ago on Sat 17 Aug 2024 04:58:26 PM UTC. 2024-08-17T17:00:02.006 INFO:teuthology.orchestra.run.smithi053.stdout:Package python3-rgw-2:17.2.7-1404.gf2c39d4d.el9.x86_64 is already installed. 2024-08-17T17:00:02.051 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-08-17T17:00:02.052 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-08-17T17:00:02.053 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-08-17T17:00:02.116 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-cephfs 2024-08-17T17:00:02.315 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:35 ago on Sat 17 Aug 2024 04:58:27 PM UTC. 2024-08-17T17:00:02.405 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rados-2:17.2.7-1404.gf2c39d4d.el9.x86_64 is already installed. 2024-08-17T17:00:02.449 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-08-17T17:00:02.450 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-08-17T17:00:02.451 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-08-17T17:00:02.514 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rgw 2024-08-17T17:00:02.693 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:36 ago on Sat 17 Aug 2024 04:58:26 PM UTC. 2024-08-17T17:00:02.780 INFO:teuthology.orchestra.run.smithi053.stdout:Package python3-cephfs-2:17.2.7-1404.gf2c39d4d.el9.x86_64 is already installed. 2024-08-17T17:00:02.825 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-08-17T17:00:02.827 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-08-17T17:00:02.827 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-08-17T17:00:02.890 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-rbd 2024-08-17T17:00:03.086 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:36 ago on Sat 17 Aug 2024 04:58:27 PM UTC. 2024-08-17T17:00:03.175 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rgw-2:17.2.7-1404.gf2c39d4d.el9.x86_64 is already installed. 2024-08-17T17:00:03.221 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-08-17T17:00:03.222 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-08-17T17:00:03.222 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-08-17T17:00:03.279 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-cephfs 2024-08-17T17:00:03.464 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:37 ago on Sat 17 Aug 2024 04:58:26 PM UTC. 2024-08-17T17:00:03.552 INFO:teuthology.orchestra.run.smithi053.stdout:Package python3-rbd-2:17.2.7-1404.gf2c39d4d.el9.x86_64 is already installed. 2024-08-17T17:00:03.596 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-08-17T17:00:03.597 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-08-17T17:00:03.597 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-08-17T17:00:03.666 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install rbd-fuse 2024-08-17T17:00:03.856 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:36 ago on Sat 17 Aug 2024 04:58:27 PM UTC. 2024-08-17T17:00:03.943 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-cephfs-2:17.2.7-1404.gf2c39d4d.el9.x86_64 is already installed. 2024-08-17T17:00:03.988 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-08-17T17:00:03.989 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-08-17T17:00:03.989 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-08-17T17:00:04.056 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rbd 2024-08-17T17:00:04.241 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:38 ago on Sat 17 Aug 2024 04:58:26 PM UTC. 2024-08-17T17:00:04.375 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-08-17T17:00:04.377 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T17:00:04.377 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-08-17T17:00:04.377 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T17:00:04.377 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-08-17T17:00:04.377 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-fuse x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 82 k 2024-08-17T17:00:04.377 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T17:00:04.378 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-08-17T17:00:04.378 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T17:00:04.378 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-08-17T17:00:04.378 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T17:00:04.378 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 82 k 2024-08-17T17:00:04.378 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 222 k 2024-08-17T17:00:04.379 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-08-17T17:00:04.584 INFO:teuthology.orchestra.run.smithi053.stdout:rbd-fuse-17.2.7-1404.gf2c39d4d.el9.x86_64.rpm 395 kB/s | 82 kB 00:00 2024-08-17T17:00:04.584 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:00:04.585 INFO:teuthology.orchestra.run.smithi053.stdout:Total 393 kB/s | 82 kB 00:00 2024-08-17T17:00:04.585 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-08-17T17:00:04.592 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-08-17T17:00:04.592 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-08-17T17:00:04.631 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:37 ago on Sat 17 Aug 2024 04:58:27 PM UTC. 2024-08-17T17:00:04.660 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-08-17T17:00:04.661 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-08-17T17:00:04.720 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rbd-2:17.2.7-1404.gf2c39d4d.el9.x86_64 is already installed. 2024-08-17T17:00:04.765 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-08-17T17:00:04.766 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-08-17T17:00:04.767 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-08-17T17:00:04.816 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-08-17T17:00:04.830 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-fuse 2024-08-17T17:00:04.926 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-fuse-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/1 2024-08-17T17:00:05.299 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/1 2024-08-17T17:00:05.399 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:38 ago on Sat 17 Aug 2024 04:58:27 PM UTC. 2024-08-17T17:00:05.532 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-08-17T17:00:05.533 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T17:00:05.533 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2024-08-17T17:00:05.533 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T17:00:05.533 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-08-17T17:00:05.533 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 82 k 2024-08-17T17:00:05.533 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:00:05.533 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-08-17T17:00:05.534 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T17:00:05.534 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-08-17T17:00:05.534 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:00:05.534 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 82 k 2024-08-17T17:00:05.534 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 222 k 2024-08-17T17:00:05.534 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-08-17T17:00:05.541 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-fuse-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/1 2024-08-17T17:00:05.542 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T17:00:05.542 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-08-17T17:00:05.542 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-fuse-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T17:00:05.542 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T17:00:05.542 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-08-17T17:00:05.745 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-fuse-17.2.7-1404.gf2c39d4d.el9.x86_64.rpm 387 kB/s | 82 kB 00:00 2024-08-17T17:00:05.745 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:00:05.746 INFO:teuthology.orchestra.run.smithi003.stdout:Total 385 kB/s | 82 kB 00:00 2024-08-17T17:00:05.746 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-08-17T17:00:05.753 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-08-17T17:00:05.753 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-08-17T17:00:05.761 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install rbd-mirror 2024-08-17T17:00:05.820 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-08-17T17:00:05.820 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-08-17T17:00:05.971 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-08-17T17:00:06.079 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-fuse-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/1 2024-08-17T17:00:06.334 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:40 ago on Sat 17 Aug 2024 04:58:26 PM UTC. 2024-08-17T17:00:06.455 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/1 2024-08-17T17:00:06.470 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-08-17T17:00:06.470 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T17:00:06.471 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2024-08-17T17:00:06.471 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T17:00:06.471 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-08-17T17:00:06.471 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-mirror x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 2.9 M 2024-08-17T17:00:06.471 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T17:00:06.471 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-08-17T17:00:06.471 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T17:00:06.471 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-08-17T17:00:06.471 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T17:00:06.471 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 2.9 M 2024-08-17T17:00:06.471 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 12 M 2024-08-17T17:00:06.471 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-08-17T17:00:06.709 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-fuse-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/1 2024-08-17T17:00:06.710 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:00:06.710 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-08-17T17:00:06.710 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T17:00:06.710 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:00:06.710 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-08-17T17:00:06.845 INFO:teuthology.orchestra.run.smithi053.stdout:rbd-mirror-17.2.7-1404.gf2c39d4d.el9.x86_64.rpm 7.9 MB/s | 2.9 MB 00:00 2024-08-17T17:00:06.846 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:00:06.846 INFO:teuthology.orchestra.run.smithi053.stdout:Total 7.8 MB/s | 2.9 MB 00:00 2024-08-17T17:00:06.846 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-08-17T17:00:06.856 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-08-17T17:00:06.856 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-08-17T17:00:06.936 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-mirror 2024-08-17T17:00:06.942 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-08-17T17:00:06.942 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-08-17T17:00:07.375 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-08-17T17:00:07.446 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-mirror-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/1 2024-08-17T17:00:07.472 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/1 2024-08-17T17:00:07.472 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-17T17:00:07.472 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-17T17:00:07.472 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-17T17:00:07.473 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-17T17:00:07.473 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T17:00:07.512 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:40 ago on Sat 17 Aug 2024 04:58:27 PM UTC. 2024-08-17T17:00:07.648 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-08-17T17:00:07.648 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T17:00:07.648 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-08-17T17:00:07.648 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T17:00:07.648 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-08-17T17:00:07.649 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 2.9 M 2024-08-17T17:00:07.649 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:00:07.649 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-08-17T17:00:07.649 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T17:00:07.649 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-08-17T17:00:07.649 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:00:07.649 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 2.9 M 2024-08-17T17:00:07.650 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 12 M 2024-08-17T17:00:07.650 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-08-17T17:00:07.989 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-mirror-17.2.7-1404.gf2c39d4d.el9.x86_64.rpm 8.7 MB/s | 2.9 MB 00:00 2024-08-17T17:00:07.990 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:00:07.990 INFO:teuthology.orchestra.run.smithi003.stdout:Total 8.6 MB/s | 2.9 MB 00:00 2024-08-17T17:00:07.990 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-08-17T17:00:08.000 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-08-17T17:00:08.000 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-08-17T17:00:08.084 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-08-17T17:00:08.085 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-08-17T17:00:08.524 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-08-17T17:00:08.528 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-mirror-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/1 2024-08-17T17:00:08.528 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T17:00:08.528 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-08-17T17:00:08.529 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-mirror-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T17:00:08.529 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T17:00:08.529 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-08-17T17:00:08.590 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-mirror-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/1 2024-08-17T17:00:08.617 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/1 2024-08-17T17:00:08.617 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-17T17:00:08.617 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-17T17:00:08.617 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-17T17:00:08.617 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-17T17:00:08.617 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:00:08.747 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install rbd-nbd 2024-08-17T17:00:09.325 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:43 ago on Sat 17 Aug 2024 04:58:26 PM UTC. 2024-08-17T17:00:09.460 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-08-17T17:00:09.461 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T17:00:09.461 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-08-17T17:00:09.461 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T17:00:09.461 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-08-17T17:00:09.462 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-nbd x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 165 k 2024-08-17T17:00:09.462 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T17:00:09.462 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-08-17T17:00:09.462 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-08-17T17:00:09.462 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2024-08-17T17:00:09.462 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T17:00:09.462 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 165 k 2024-08-17T17:00:09.462 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 477 k 2024-08-17T17:00:09.463 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-08-17T17:00:09.659 INFO:teuthology.orchestra.run.smithi053.stdout:rbd-nbd-17.2.7-1404.gf2c39d4d.el9.x86_64.rpm 837 kB/s | 165 kB 00:00 2024-08-17T17:00:09.659 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:00:09.659 INFO:teuthology.orchestra.run.smithi053.stdout:Total 832 kB/s | 165 kB 00:00 2024-08-17T17:00:09.660 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-08-17T17:00:09.667 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-08-17T17:00:09.668 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-08-17T17:00:09.684 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-mirror-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/1 2024-08-17T17:00:09.684 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:00:09.684 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-08-17T17:00:09.684 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T17:00:09.684 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:00:09.684 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-08-17T17:00:09.735 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-08-17T17:00:09.735 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-08-17T17:00:09.895 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-nbd 2024-08-17T17:00:09.898 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-08-17T17:00:09.991 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-nbd-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/1 2024-08-17T17:00:10.406 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/1 2024-08-17T17:00:10.474 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:43 ago on Sat 17 Aug 2024 04:58:27 PM UTC. 2024-08-17T17:00:10.608 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-08-17T17:00:10.609 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T17:00:10.609 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2024-08-17T17:00:10.609 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T17:00:10.609 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-08-17T17:00:10.609 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd x86_64 2:17.2.7-1404.gf2c39d4d.el9 ceph 165 k 2024-08-17T17:00:10.609 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:00:10.609 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-08-17T17:00:10.609 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-08-17T17:00:10.609 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-08-17T17:00:10.609 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:00:10.610 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 165 k 2024-08-17T17:00:10.610 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 477 k 2024-08-17T17:00:10.610 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-08-17T17:00:10.648 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-nbd-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/1 2024-08-17T17:00:10.648 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T17:00:10.648 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-08-17T17:00:10.649 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-nbd-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T17:00:10.649 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T17:00:10.649 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-08-17T17:00:10.826 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-nbd-17.2.7-1404.gf2c39d4d.el9.x86_64.rpm 764 kB/s | 165 kB 00:00 2024-08-17T17:00:10.826 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:00:10.827 INFO:teuthology.orchestra.run.smithi003.stdout:Total 758 kB/s | 165 kB 00:00 2024-08-17T17:00:10.827 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-08-17T17:00:10.835 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-08-17T17:00:10.835 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-08-17T17:00:10.890 DEBUG:teuthology.parallel:result is None 2024-08-17T17:00:10.903 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-08-17T17:00:10.904 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-08-17T17:00:11.063 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-08-17T17:00:11.162 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-nbd-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/1 2024-08-17T17:00:11.564 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/1 2024-08-17T17:00:11.808 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-nbd-2:17.2.7-1404.gf2c39d4d.el9.x86_64 1/1 2024-08-17T17:00:11.808 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:00:11.808 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-08-17T17:00:11.808 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd-2:17.2.7-1404.gf2c39d4d.el9.x86_64 2024-08-17T17:00:11.808 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:00:11.808 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-08-17T17:00:12.060 DEBUG:teuthology.parallel:result is None 2024-08-17T17:00:12.060 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-17T17:00:12.061 INFO:teuthology.packaging:ref: None 2024-08-17T17:00:12.061 INFO:teuthology.packaging:tag: None 2024-08-17T17:00:12.061 INFO:teuthology.packaging:branch: quincy 2024-08-17T17:00:12.061 INFO:teuthology.packaging:sha1: cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:00:12.061 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-08-17T17:00:12.283 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-17T17:00:12.306 INFO:teuthology.orchestra.run.smithi003.stdout:17.2.7-1404.gf2c39d4d.el9 2024-08-17T17:00:12.306 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1404.gf2c39d4d.el9 2024-08-17T17:00:12.307 INFO:teuthology.task.install:The correct ceph version 17.2.7-1404.gf2c39d4d is installed. 2024-08-17T17:00:12.308 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-17T17:00:12.308 INFO:teuthology.packaging:ref: None 2024-08-17T17:00:12.308 INFO:teuthology.packaging:tag: None 2024-08-17T17:00:12.309 INFO:teuthology.packaging:branch: quincy 2024-08-17T17:00:12.309 INFO:teuthology.packaging:sha1: cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:00:12.309 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-08-17T17:00:12.533 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-17T17:00:12.559 INFO:teuthology.orchestra.run.smithi053.stdout:17.2.7-1404.gf2c39d4d.el9 2024-08-17T17:00:12.559 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1404.gf2c39d4d.el9 2024-08-17T17:00:12.559 INFO:teuthology.task.install:The correct ceph version 17.2.7-1404.gf2c39d4d is installed. 2024-08-17T17:00:12.561 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-08-17T17:00:12.561 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-08-17T17:00:12.561 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-17T17:00:12.587 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-08-17T17:00:12.587 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-17T17:00:12.630 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-08-17T17:00:12.630 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-08-17T17:00:12.630 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/daemon-helper 2024-08-17T17:00:12.656 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-17T17:00:12.721 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-08-17T17:00:12.722 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/daemon-helper 2024-08-17T17:00:12.752 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-17T17:00:12.821 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-08-17T17:00:12.821 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-08-17T17:00:12.821 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-17T17:00:12.847 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-17T17:00:12.913 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-08-17T17:00:12.913 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-17T17:00:12.941 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-17T17:00:13.008 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-08-17T17:00:13.008 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-08-17T17:00:13.008 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/stdin-killer 2024-08-17T17:00:13.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-17T17:00:13.100 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-08-17T17:00:13.100 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/stdin-killer 2024-08-17T17:00:13.126 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-17T17:00:13.194 INFO:teuthology.run_tasks:Running task cephadm... 2024-08-17T17:00:13.285 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'create_rbd_pool': True, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'OSD_DOWN', 'mons down', 'mon down', 'MON_DOWN', 'out of quorum', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSDMAP_FLAGS', 'PG_AVAILABILITY'], 'sha1': 'cd8bdbeb9fbdab5733aa3aaadc042c965473e453'} 2024-08-17T17:00:13.285 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-08-17T17:00:13.285 INFO:tasks.cephadm:Cluster fsid is 2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:00:13.285 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-08-17T17:00:13.286 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.3', 'mon.c': '[v2:172.21.15.3:3301,v1:172.21.15.3:6790]', 'mon.b': '172.21.15.53'} 2024-08-17T17:00:13.286 INFO:tasks.cephadm:First mon is mon.a on smithi003 2024-08-17T17:00:13.286 INFO:tasks.cephadm:First mgr is y 2024-08-17T17:00:13.286 INFO:tasks.cephadm:Normalizing hostnames... 2024-08-17T17:00:13.286 DEBUG:teuthology.orchestra.run.smithi003:> sudo hostname $(hostname -s) 2024-08-17T17:00:13.312 DEBUG:teuthology.orchestra.run.smithi053:> sudo hostname $(hostname -s) 2024-08-17T17:00:13.338 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-08-17T17:00:13.339 DEBUG:teuthology.orchestra.run.smithi003:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-08-17T17:00:13.493 INFO:teuthology.orchestra.run.smithi003.stdout:-rw-r--r--. 1 ubuntu ubuntu 378585 Aug 17 17:00 /home/ubuntu/cephtest/cephadm 2024-08-17T17:00:13.493 DEBUG:teuthology.orchestra.run.smithi053:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-08-17T17:00:13.636 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r--. 1 ubuntu ubuntu 378585 Aug 17 17:00 /home/ubuntu/cephtest/cephadm 2024-08-17T17:00:13.636 DEBUG:teuthology.orchestra.run.smithi003:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-08-17T17:00:13.653 DEBUG:teuthology.orchestra.run.smithi053:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-08-17T17:00:13.678 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-08-17T17:00:13.678 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-08-17T17:00:13.695 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-08-17T17:00:14.623 INFO:teuthology.orchestra.run.smithi053.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-08-17T17:00:14.654 INFO:teuthology.orchestra.run.smithi003.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-08-17T17:00:41.036 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:00:41.036 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph_version": "ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)", 2024-08-17T17:00:41.036 INFO:teuthology.orchestra.run.smithi003.stdout: "image_id": "f69471c56189b063ba7a049adea817a12ba1a40187d11cab5cbef8f3aac57cce", 2024-08-17T17:00:41.036 INFO:teuthology.orchestra.run.smithi003.stdout: "repo_digests": [ 2024-08-17T17:00:41.037 INFO:teuthology.orchestra.run.smithi003.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251" 2024-08-17T17:00:41.037 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:00:41.037 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:00:45.603 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:00:45.603 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_version": "ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)", 2024-08-17T17:00:45.603 INFO:teuthology.orchestra.run.smithi053.stdout: "image_id": "f69471c56189b063ba7a049adea817a12ba1a40187d11cab5cbef8f3aac57cce", 2024-08-17T17:00:45.604 INFO:teuthology.orchestra.run.smithi053.stdout: "repo_digests": [ 2024-08-17T17:00:45.604 INFO:teuthology.orchestra.run.smithi053.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251" 2024-08-17T17:00:45.604 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:00:45.604 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:00:45.623 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /etc/ceph 2024-08-17T17:00:45.658 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /etc/ceph 2024-08-17T17:00:45.687 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 777 /etc/ceph 2024-08-17T17:00:45.728 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 777 /etc/ceph 2024-08-17T17:00:45.755 INFO:tasks.cephadm:Writing seed config... 2024-08-17T17:00:45.756 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-08-17T17:00:45.756 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-08-17T17:00:45.756 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-08-17T17:00:45.756 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-08-17T17:00:45.756 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-08-17T17:00:45.756 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-08-17T17:00:45.757 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-08-17T17:00:45.757 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-08-17T17:00:45.757 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-08-17T17:00:45.757 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-08-17T17:00:45.757 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-08-17T17:00:45.757 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-08-17T17:00:45.758 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-08-17T17:00:45.786 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 2b5b662e-5cba-11ef-bccf-c7b262605968 mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-08-17T17:00:45.787 DEBUG:teuthology.orchestra.run.smithi003:mon.a> sudo journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mon.a.service 2024-08-17T17:00:45.829 DEBUG:teuthology.orchestra.run.smithi003:mgr.y> sudo journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mgr.y.service 2024-08-17T17:00:45.871 INFO:tasks.cephadm:Bootstrapping... 2024-08-17T17:00:45.872 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.3 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-08-17T17:00:46.088 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-08-17T17:00:46.088 INFO:teuthology.orchestra.run.smithi003.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', '2b5b662e-5cba-11ef-bccf-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.3', '--skip-admin-label'] 2024-08-17T17:00:46.114 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stdout 5.1.2 2024-08-17T17:00:46.115 INFO:teuthology.orchestra.run.smithi003.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-08-17T17:00:46.115 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying podman|docker is present... 2024-08-17T17:00:46.140 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stdout 5.1.2 2024-08-17T17:00:46.140 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying lvm2 is present... 2024-08-17T17:00:46.140 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying time synchronization is in place... 2024-08-17T17:00:46.148 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-08-17T17:00:46.148 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-08-17T17:00:46.155 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-08-17T17:00:46.155 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2024-08-17T17:00:46.162 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout enabled 2024-08-17T17:00:46.169 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout active 2024-08-17T17:00:46.169 INFO:teuthology.orchestra.run.smithi003.stdout:Unit chronyd.service is enabled and running 2024-08-17T17:00:46.170 INFO:teuthology.orchestra.run.smithi003.stdout:Repeating the final host check... 2024-08-17T17:00:46.194 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stdout 5.1.2 2024-08-17T17:00:46.194 INFO:teuthology.orchestra.run.smithi003.stdout:podman (/bin/podman) version 5.1.2 is present 2024-08-17T17:00:46.195 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl is present 2024-08-17T17:00:46.195 INFO:teuthology.orchestra.run.smithi003.stdout:lvcreate is present 2024-08-17T17:00:46.202 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-08-17T17:00:46.202 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-08-17T17:00:46.208 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-08-17T17:00:46.209 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2024-08-17T17:00:46.216 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout enabled 2024-08-17T17:00:46.223 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout active 2024-08-17T17:00:46.223 INFO:teuthology.orchestra.run.smithi003.stdout:Unit chronyd.service is enabled and running 2024-08-17T17:00:46.223 INFO:teuthology.orchestra.run.smithi003.stdout:Host looks OK 2024-08-17T17:00:46.223 INFO:teuthology.orchestra.run.smithi003.stdout:Cluster fsid: 2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:00:46.223 INFO:teuthology.orchestra.run.smithi003.stdout:Acquiring lock 140142775214960 on /run/cephadm/2b5b662e-5cba-11ef-bccf-c7b262605968.lock 2024-08-17T17:00:46.223 INFO:teuthology.orchestra.run.smithi003.stdout:Lock 140142775214960 acquired on /run/cephadm/2b5b662e-5cba-11ef-bccf-c7b262605968.lock 2024-08-17T17:00:46.224 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying IP 172.21.15.3 port 3300 ... 2024-08-17T17:00:46.224 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying IP 172.21.15.3 port 6789 ... 2024-08-17T17:00:46.224 INFO:teuthology.orchestra.run.smithi003.stdout:Base mon IP(s) is [172.21.15.3:3300, 172.21.15.3:6789], mon addrv is [v2:172.21.15.3:3300,v1:172.21.15.3:6789] 2024-08-17T17:00:46.227 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.3 metric 100 2024-08-17T17:00:46.227 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.3 metric 100 2024-08-17T17:00:46.230 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-08-17T17:00:46.230 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-08-17T17:00:46.231 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-08-17T17:00:46.233 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-08-17T17:00:46.233 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-08-17T17:00:46.233 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-08-17T17:00:46.233 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-08-17T17:00:46.234 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15dc/64 scope link noprefixroute 2024-08-17T17:00:46.234 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-08-17T17:00:46.234 INFO:teuthology.orchestra.run.smithi003.stdout:Mon IP `172.21.15.3` is in CIDR network `172.21.0.0/20` 2024-08-17T17:00:46.234 INFO:teuthology.orchestra.run.smithi003.stdout:Mon IP `172.21.15.3` is in CIDR network `172.21.0.0/20` 2024-08-17T17:00:46.234 INFO:teuthology.orchestra.run.smithi003.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-08-17T17:00:46.235 INFO:teuthology.orchestra.run.smithi003.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-08-17T17:00:46.235 INFO:teuthology.orchestra.run.smithi003.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-08-17T17:00:48.445 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stdout f69471c56189b063ba7a049adea817a12ba1a40187d11cab5cbef8f3aac57cce 2024-08-17T17:00:48.445 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-08-17T17:00:48.445 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Getting image source signatures 2024-08-17T17:00:48.445 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Copying blob sha256:11a3dd1ce731c9342fb0d936b067311689a4bd18187bbc27c4f481cc3262ce2c 2024-08-17T17:00:48.445 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-08-17T17:00:48.445 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Copying config sha256:f69471c56189b063ba7a049adea817a12ba1a40187d11cab5cbef8f3aac57cce 2024-08-17T17:00:48.445 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Writing manifest to image destination 2024-08-17T17:00:49.306 INFO:teuthology.orchestra.run.smithi003.stdout:ceph: stdout ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable) 2024-08-17T17:00:49.306 INFO:teuthology.orchestra.run.smithi003.stdout:Ceph version: ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable) 2024-08-17T17:00:49.306 INFO:teuthology.orchestra.run.smithi003.stdout:Extracting ceph user uid/gid from container image... 2024-08-17T17:00:50.279 INFO:teuthology.orchestra.run.smithi003.stdout:stat: stdout 167 167 2024-08-17T17:00:50.279 INFO:teuthology.orchestra.run.smithi003.stdout:Creating initial keys... 2024-08-17T17:00:51.250 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph-authtool: stdout AQBC18BmpjVJIhAA3kMCZjwXuaJaGQSOfnYglA== 2024-08-17T17:00:52.072 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph-authtool: stdout AQBD18Bmh0W3HxAApCMcY/gdGTZS3IgeoSXXVg== 2024-08-17T17:00:53.046 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph-authtool: stdout AQBE18BmDxSBFxAA8rluAHW1Z6Ayjnts/6WjSg== 2024-08-17T17:00:53.047 INFO:teuthology.orchestra.run.smithi003.stdout:Creating initial monmap... 2024-08-17T17:00:54.119 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-08-17T17:00:54.120 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-08-17T17:00:54.120 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:00:54.120 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-08-17T17:00:54.120 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool for a [v2:172.21.15.3:3300,v1:172.21.15.3:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-08-17T17:00:54.120 INFO:teuthology.orchestra.run.smithi003.stdout:setting min_mon_release = octopus 2024-08-17T17:00:54.120 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: set fsid to 2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:00:54.120 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-08-17T17:00:54.120 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:00:54.121 INFO:teuthology.orchestra.run.smithi003.stdout:Creating mon... 2024-08-17T17:00:54.975 INFO:teuthology.orchestra.run.smithi003.stdout:create mon.a on 2024-08-17T17:00:55.189 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-08-17T17:00:55.365 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-08-17T17:00:55.528 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-2b5b662e-5cba-11ef-bccf-c7b262605968.target → /etc/systemd/system/ceph-2b5b662e-5cba-11ef-bccf-c7b262605968.target. 2024-08-17T17:00:55.528 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-2b5b662e-5cba-11ef-bccf-c7b262605968.target → /etc/systemd/system/ceph-2b5b662e-5cba-11ef-bccf-c7b262605968.target. 2024-08-17T17:00:55.729 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mon.a 2024-08-17T17:00:55.729 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Failed to reset failed state of unit ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mon.a.service: Unit ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mon.a.service not loaded. 2024-08-17T17:00:55.910 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-2b5b662e-5cba-11ef-bccf-c7b262605968.target.wants/ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mon.a.service → /etc/systemd/system/ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@.service. 2024-08-17T17:00:55.975 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:00:55 smithi003 systemd[1]: Starting Ceph mon.a for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:00:56.246 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:00:56 smithi003 podman[30402]: 2024-08-17 17:00:56.156292168 +0000 UTC m=+0.106069815 container create d527c771e438794b8349d0e501a5c5bc8d09875796c29770614e2511846ddc1d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, RELEASE=quincy-f2c39d4, io.buildah.version=1.36.0) 2024-08-17T17:00:56.246 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:00:56 smithi003 podman[30402]: 2024-08-17 17:00:56.068322812 +0000 UTC m=+0.018100474 image pull f69471c56189b063ba7a049adea817a12ba1a40187d11cab5cbef8f3aac57cce quay.ceph.io/ceph-ci/ceph:quincy 2024-08-17T17:00:56.434 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-08-17T17:00:56.434 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout disabled 2024-08-17T17:00:56.443 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-08-17T17:00:56.443 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2024-08-17T17:00:56.443 INFO:teuthology.orchestra.run.smithi003.stdout:firewalld.service is not enabled 2024-08-17T17:00:56.443 INFO:teuthology.orchestra.run.smithi003.stdout:Not possible to enable service . firewalld.service is not available 2024-08-17T17:00:56.443 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mon to start... 2024-08-17T17:00:56.443 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mon... 2024-08-17T17:00:56.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:00:56 smithi003 podman[30402]: 2024-08-17 17:00:56.33390539 +0000 UTC m=+0.283683058 container init d527c771e438794b8349d0e501a5c5bc8d09875796c29770614e2511846ddc1d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-a, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-f2c39d4, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-08-17T17:00:56.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:00:56 smithi003 podman[30402]: 2024-08-17 17:00:56.337235703 +0000 UTC m=+0.287013353 container start d527c771e438794b8349d0e501a5c5bc8d09875796c29770614e2511846ddc1d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=quincy-f2c39d4) 2024-08-17T17:00:56.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:00:56 smithi003 ceph-mon[30420]: mkfs 2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:00:56.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:00:56 smithi003 ceph-mon[30420]: mon.a is new leader, mons a in quorum (ranks 0) 2024-08-17T17:00:56.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:00:56 smithi003 bash[30402]: d527c771e438794b8349d0e501a5c5bc8d09875796c29770614e2511846ddc1d 2024-08-17T17:00:56.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:00:56 smithi003 systemd[1]: Started Ceph mon.a for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:00:57.562 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout cluster: 2024-08-17T17:00:57.562 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout id: 2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:00:57.562 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-08-17T17:00:57.562 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-08-17T17:00:57.562 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout services: 2024-08-17T17:00:57.562 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.655567s) 2024-08-17T17:00:57.563 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-08-17T17:00:57.563 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-08-17T17:00:57.563 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-08-17T17:00:57.563 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout data: 2024-08-17T17:00:57.563 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-08-17T17:00:57.563 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-08-17T17:00:57.563 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-08-17T17:00:57.563 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout pgs: 2024-08-17T17:00:57.563 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-08-17T17:00:57.563 INFO:teuthology.orchestra.run.smithi003.stdout:mon is available 2024-08-17T17:00:57.563 INFO:teuthology.orchestra.run.smithi003.stdout:Assimilating anything we can from ceph.conf... 2024-08-17T17:00:57.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:00:57 smithi003 ceph-mon[30420]: mon.a is new leader, mons a in quorum (ranks 0) 2024-08-17T17:00:57.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:00:57 smithi003 ceph-mon[30420]: monmap e1: 1 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-08-17T17:00:57.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:00:57 smithi003 ceph-mon[30420]: fsmap 2024-08-17T17:00:57.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:00:57 smithi003 ceph-mon[30420]: osdmap e1: 0 total, 0 up, 0 in 2024-08-17T17:00:57.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:00:57 smithi003 ceph-mon[30420]: mgrmap e1: no daemons active 2024-08-17T17:00:57.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:00:57 smithi003 ceph-mon[30420]: from='client.? 172.21.15.3:0/3738828356' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-08-17T17:00:58.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:00:58 smithi003 ceph-mon[30420]: from='client.? 172.21.15.3:0/767307416' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-08-17T17:00:58.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:00:58 smithi003 ceph-mon[30420]: from='client.? 172.21.15.3:0/767307416' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-08-17T17:00:59.969 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-08-17T17:00:59.969 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [global] 2024-08-17T17:00:59.970 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout fsid = 2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:00:59.970 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-08-17T17:00:59.970 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.3:3300,v1:172.21.15.3:6789] 2024-08-17T17:00:59.970 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-08-17T17:00:59.970 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-08-17T17:00:59.970 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-08-17T17:00:59.970 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-08-17T17:00:59.970 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-08-17T17:00:59.970 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [mgr] 2024-08-17T17:00:59.970 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-08-17T17:00:59.970 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-08-17T17:00:59.970 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [osd] 2024-08-17T17:00:59.970 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-08-17T17:00:59.971 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-08-17T17:00:59.971 INFO:teuthology.orchestra.run.smithi003.stdout:Generating new minimal ceph.conf... 2024-08-17T17:01:03.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:02 smithi003 ceph-mon[30420]: from='client.? 172.21.15.3:0/1151649135' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:01:08.752 INFO:teuthology.orchestra.run.smithi003.stdout:Restarting the monitor... 2024-08-17T17:01:09.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:08 smithi003 systemd[1]: Stopping Ceph mon.a for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:01:10.155 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:09 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-a[30416]: 2024-08-17T17:01:09.804+0000 7f3fafe53640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T17:01:10.155 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:09 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-a[30416]: 2024-08-17T17:01:09.804+0000 7f3fafe53640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-08-17T17:01:10.155 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:09 smithi003 podman[30675]: 2024-08-17 17:01:09.886969338 +0000 UTC m=+0.957860396 container died d527c771e438794b8349d0e501a5c5bc8d09875796c29770614e2511846ddc1d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, RELEASE=quincy-f2c39d4, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-08-17T17:01:10.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:10 smithi003 podman[30675]: 2024-08-17 17:01:10.228087117 +0000 UTC m=+1.298978165 container cleanup d527c771e438794b8349d0e501a5c5bc8d09875796c29770614e2511846ddc1d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-a, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, RELEASE=quincy-f2c39d4, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-17T17:01:10.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:10 smithi003 bash[30675]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-a 2024-08-17T17:01:10.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:10 smithi003 podman[30690]: 2024-08-17 17:01:10.395690519 +0000 UTC m=+0.506586388 container remove d527c771e438794b8349d0e501a5c5bc8d09875796c29770614e2511846ddc1d (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-a, RELEASE=quincy-f2c39d4, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-08-17T17:01:10.918 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:10 smithi003 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mon.a.service: Deactivated successfully. 2024-08-17T17:01:10.918 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:10 smithi003 systemd[1]: Stopped Ceph mon.a for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:01:10.918 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:10 smithi003 systemd[1]: Starting Ceph mon.a for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:01:10.918 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:10 smithi003 podman[30784]: 2024-08-17 17:01:10.846399736 +0000 UTC m=+0.111599381 container create a8b52ab645d32066a4c2bae858d2be3a8353be59f545703bac701da0f07d1193 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-a, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-f2c39d4, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-08-17T17:01:10.918 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:10 smithi003 podman[30784]: 2024-08-17 17:01:10.75269548 +0000 UTC m=+0.017895141 image pull f69471c56189b063ba7a049adea817a12ba1a40187d11cab5cbef8f3aac57cce quay.ceph.io/ceph-ci/ceph:quincy 2024-08-17T17:01:11.094 INFO:teuthology.orchestra.run.smithi003.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-08-17T17:01:11.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 podman[30784]: 2024-08-17 17:01:11.01540729 +0000 UTC m=+0.280606938 container init a8b52ab645d32066a4c2bae858d2be3a8353be59f545703bac701da0f07d1193 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=quincy-f2c39d4) 2024-08-17T17:01:11.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 podman[30784]: 2024-08-17 17:01:11.018500683 +0000 UTC m=+0.283700329 container start a8b52ab645d32066a4c2bae858d2be3a8353be59f545703bac701da0f07d1193 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-a, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=quincy-f2c39d4, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-08-17T17:01:11.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: set uid:gid to 167:167 (ceph:ceph) 2024-08-17T17:01:11.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable), process ceph-mon, pid 2 2024-08-17T17:01:11.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: pidfile_write: ignore empty --pid-file 2024-08-17T17:01:11.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: load: jerasure load: lrc 2024-08-17T17:01:11.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: RocksDB version: 6.15.5 2024-08-17T17:01:11.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-08-17T17:01:11.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Compile date Aug 17 2024 2024-08-17T17:01:11.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: DB SUMMARY 2024-08-17T17:01:11.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: DB Session ID: D9W7JCGN24M5JIIEW7QM 2024-08-17T17:01:11.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: CURRENT file: CURRENT 2024-08-17T17:01:11.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: IDENTITY file: IDENTITY 2024-08-17T17:01:11.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-08-17T17:01:11.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-08-17T17:01:11.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 83653 ; 2024-08-17T17:01:11.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.error_if_exists: 0 2024-08-17T17:01:11.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.create_if_missing: 0 2024-08-17T17:01:11.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.paranoid_checks: 1 2024-08-17T17:01:11.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-08-17T17:01:11.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.env: 0x5568ebdf3480 2024-08-17T17:01:11.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.fs: Posix File System 2024-08-17T17:01:11.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.info_log: 0x5568ed646f80 2024-08-17T17:01:11.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.max_file_opening_threads: 16 2024-08-17T17:01:11.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.statistics: (nil) 2024-08-17T17:01:11.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.use_fsync: 0 2024-08-17T17:01:11.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.max_log_file_size: 0 2024-08-17T17:01:11.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-08-17T17:01:11.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.log_file_time_to_roll: 0 2024-08-17T17:01:11.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.keep_log_file_num: 1000 2024-08-17T17:01:11.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.recycle_log_file_num: 0 2024-08-17T17:01:11.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.allow_fallocate: 1 2024-08-17T17:01:11.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.allow_mmap_reads: 0 2024-08-17T17:01:11.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.allow_mmap_writes: 0 2024-08-17T17:01:11.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.use_direct_reads: 0 2024-08-17T17:01:11.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-08-17T17:01:11.277 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.create_missing_column_families: 0 2024-08-17T17:01:11.277 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.db_log_dir: 2024-08-17T17:01:11.277 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-08-17T17:01:11.277 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.table_cache_numshardbits: 6 2024-08-17T17:01:11.277 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.WAL_ttl_seconds: 0 2024-08-17T17:01:11.277 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.WAL_size_limit_MB: 0 2024-08-17T17:01:11.277 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-08-17T17:01:11.277 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-08-17T17:01:11.277 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.is_fd_close_on_exec: 1 2024-08-17T17:01:11.277 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.advise_random_on_open: 1 2024-08-17T17:01:11.277 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.db_write_buffer_size: 0 2024-08-17T17:01:11.277 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.write_buffer_manager: 0x5568ed704780 2024-08-17T17:01:11.277 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-08-17T17:01:11.278 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-08-17T17:01:11.278 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-08-17T17:01:11.278 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.use_adaptive_mutex: 0 2024-08-17T17:01:11.278 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.rate_limiter: (nil) 2024-08-17T17:01:11.278 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-08-17T17:01:11.278 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.wal_recovery_mode: 2 2024-08-17T17:01:11.278 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.enable_thread_tracking: 0 2024-08-17T17:01:11.278 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.enable_pipelined_write: 0 2024-08-17T17:01:11.278 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.unordered_write: 0 2024-08-17T17:01:11.278 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-08-17T17:01:11.278 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-08-17T17:01:11.278 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-08-17T17:01:11.278 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-08-17T17:01:11.279 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.row_cache: None 2024-08-17T17:01:11.279 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.wal_filter: None 2024-08-17T17:01:11.279 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-08-17T17:01:11.279 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.allow_ingest_behind: 0 2024-08-17T17:01:11.279 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.preserve_deletes: 0 2024-08-17T17:01:11.280 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.two_write_queues: 0 2024-08-17T17:01:11.280 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.manual_wal_flush: 0 2024-08-17T17:01:11.280 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.atomic_flush: 0 2024-08-17T17:01:11.280 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-08-17T17:01:11.280 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.persist_stats_to_disk: 0 2024-08-17T17:01:11.280 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.write_dbid_to_manifest: 0 2024-08-17T17:01:11.280 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.log_readahead_size: 0 2024-08-17T17:01:11.280 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-08-17T17:01:11.281 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.best_efforts_recovery: 0 2024-08-17T17:01:11.281 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-08-17T17:01:11.281 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-08-17T17:01:11.281 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.allow_data_in_errors: 0 2024-08-17T17:01:11.281 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.db_host_id: __hostname__ 2024-08-17T17:01:11.281 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.max_background_jobs: 2 2024-08-17T17:01:11.281 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.max_background_compactions: -1 2024-08-17T17:01:11.281 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.max_subcompactions: 1 2024-08-17T17:01:11.281 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-08-17T17:01:11.281 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-08-17T17:01:11.281 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.delayed_write_rate : 16777216 2024-08-17T17:01:11.282 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.max_total_wal_size: 0 2024-08-17T17:01:11.282 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-08-17T17:01:11.282 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.stats_dump_period_sec: 600 2024-08-17T17:01:11.282 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.stats_persist_period_sec: 600 2024-08-17T17:01:11.282 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-08-17T17:01:11.282 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.max_open_files: -1 2024-08-17T17:01:11.282 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.bytes_per_sync: 0 2024-08-17T17:01:11.282 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.wal_bytes_per_sync: 0 2024-08-17T17:01:11.282 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.strict_bytes_per_sync: 0 2024-08-17T17:01:11.282 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.compaction_readahead_size: 0 2024-08-17T17:01:11.282 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.max_background_flushes: -1 2024-08-17T17:01:11.282 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Compression algorithms supported: 2024-08-17T17:01:11.282 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: kZSTD supported: 0 2024-08-17T17:01:11.282 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: kXpressCompression supported: 0 2024-08-17T17:01:11.283 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: kBZip2Compression supported: 0 2024-08-17T17:01:11.283 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-08-17T17:01:11.283 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: kLZ4Compression supported: 1 2024-08-17T17:01:11.283 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: kZlibCompression supported: 1 2024-08-17T17:01:11.283 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: kLZ4HCCompression supported: 1 2024-08-17T17:01:11.283 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: kSnappyCompression supported: 1 2024-08-17T17:01:11.283 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Fast CRC32 supported: Supported on x86 2024-08-17T17:01:11.283 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-08-17T17:01:11.283 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-08-17T17:01:11.283 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-08-17T17:01:11.283 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.merge_operator: 2024-08-17T17:01:11.283 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.compaction_filter: None 2024-08-17T17:01:11.283 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.compaction_filter_factory: None 2024-08-17T17:01:11.283 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.sst_partitioner_factory: None 2024-08-17T17:01:11.284 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.memtable_factory: SkipListFactory 2024-08-17T17:01:11.284 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.table_factory: BlockBasedTable 2024-08-17T17:01:11.284 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5568ed5eda78) 2024-08-17T17:01:11.284 INFO:journalctl@ceph.mon.a.smithi003.stdout: cache_index_and_filter_blocks: 1 2024-08-17T17:01:11.284 INFO:journalctl@ceph.mon.a.smithi003.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-08-17T17:01:11.284 INFO:journalctl@ceph.mon.a.smithi003.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-08-17T17:01:11.284 INFO:journalctl@ceph.mon.a.smithi003.stdout: pin_top_level_index_and_filter: 1 2024-08-17T17:01:11.284 INFO:journalctl@ceph.mon.a.smithi003.stdout: index_type: 0 2024-08-17T17:01:11.284 INFO:journalctl@ceph.mon.a.smithi003.stdout: data_block_index_type: 0 2024-08-17T17:01:11.284 INFO:journalctl@ceph.mon.a.smithi003.stdout: index_shortening: 1 2024-08-17T17:01:11.284 INFO:journalctl@ceph.mon.a.smithi003.stdout: data_block_hash_table_util_ratio: 0.750000 2024-08-17T17:01:11.284 INFO:journalctl@ceph.mon.a.smithi003.stdout: hash_index_allow_collision: 1 2024-08-17T17:01:11.284 INFO:journalctl@ceph.mon.a.smithi003.stdout: checksum: 1 2024-08-17T17:01:11.284 INFO:journalctl@ceph.mon.a.smithi003.stdout: no_block_cache: 0 2024-08-17T17:01:11.284 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache: 0x5568ed6231f0 2024-08-17T17:01:11.284 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache_name: BinnedLRUCache 2024-08-17T17:01:11.285 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache_options: 2024-08-17T17:01:11.285 INFO:journalctl@ceph.mon.a.smithi003.stdout: capacity : 536870912 2024-08-17T17:01:11.285 INFO:journalctl@ceph.mon.a.smithi003.stdout: num_shard_bits : 4 2024-08-17T17:01:11.285 INFO:journalctl@ceph.mon.a.smithi003.stdout: strict_capacity_limit : 0 2024-08-17T17:01:11.285 INFO:journalctl@ceph.mon.a.smithi003.stdout: high_pri_pool_ratio: 0.000 2024-08-17T17:01:11.285 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache_compressed: (nil) 2024-08-17T17:01:11.285 INFO:journalctl@ceph.mon.a.smithi003.stdout: persistent_cache: (nil) 2024-08-17T17:01:11.285 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_size: 4096 2024-08-17T17:01:11.285 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_size_deviation: 10 2024-08-17T17:01:11.285 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_restart_interval: 16 2024-08-17T17:01:11.285 INFO:journalctl@ceph.mon.a.smithi003.stdout: index_block_restart_interval: 1 2024-08-17T17:01:11.285 INFO:journalctl@ceph.mon.a.smithi003.stdout: metadata_block_size: 4096 2024-08-17T17:01:11.285 INFO:journalctl@ceph.mon.a.smithi003.stdout: partition_filters: 0 2024-08-17T17:01:11.285 INFO:journalctl@ceph.mon.a.smithi003.stdout: use_delta_encoding: 1 2024-08-17T17:01:11.285 INFO:journalctl@ceph.mon.a.smithi003.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-08-17T17:01:11.286 INFO:journalctl@ceph.mon.a.smithi003.stdout: whole_key_filtering: 1 2024-08-17T17:01:11.286 INFO:journalctl@ceph.mon.a.smithi003.stdout: verify_compression: 0 2024-08-17T17:01:11.286 INFO:journalctl@ceph.mon.a.smithi003.stdout: read_amp_bytes_per_bit: 0 2024-08-17T17:01:11.286 INFO:journalctl@ceph.mon.a.smithi003.stdout: format_version: 4 2024-08-17T17:01:11.286 INFO:journalctl@ceph.mon.a.smithi003.stdout: enable_index_compression: 1 2024-08-17T17:01:11.286 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_align: 0 2024-08-17T17:01:11.286 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.write_buffer_size: 33554432 2024-08-17T17:01:11.286 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.max_write_buffer_number: 2 2024-08-17T17:01:11.286 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.compression: NoCompression 2024-08-17T17:01:11.286 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.bottommost_compression: Disabled 2024-08-17T17:01:11.286 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.prefix_extractor: nullptr 2024-08-17T17:01:11.287 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-08-17T17:01:11.287 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.num_levels: 7 2024-08-17T17:01:11.287 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-08-17T17:01:11.287 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-08-17T17:01:11.287 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-08-17T17:01:11.287 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-08-17T17:01:11.287 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-08-17T17:01:11.287 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-08-17T17:01:11.287 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-08-17T17:01:11.287 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-08-17T17:01:11.287 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-08-17T17:01:11.287 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-08-17T17:01:11.287 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.compression_opts.window_bits: -14 2024-08-17T17:01:11.287 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.compression_opts.level: 32767 2024-08-17T17:01:11.288 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.compression_opts.strategy: 0 2024-08-17T17:01:11.288 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-08-17T17:01:11.288 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-08-17T17:01:11.288 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-08-17T17:01:11.288 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.compression_opts.enabled: false 2024-08-17T17:01:11.288 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-08-17T17:01:11.288 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-08-17T17:01:11.288 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-08-17T17:01:11.288 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.target_file_size_base: 67108864 2024-08-17T17:01:11.288 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.target_file_size_multiplier: 1 2024-08-17T17:01:11.288 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-08-17T17:01:11.288 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-08-17T17:01:11.288 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-08-17T17:01:11.288 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-08-17T17:01:11.288 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-08-17T17:01:11.289 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-08-17T17:01:11.289 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-08-17T17:01:11.289 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-08-17T17:01:11.289 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-08-17T17:01:11.289 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-08-17T17:01:11.289 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-08-17T17:01:11.289 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-08-17T17:01:11.289 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.arena_block_size: 4194304 2024-08-17T17:01:11.289 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-08-17T17:01:11.289 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-08-17T17:01:11.289 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-08-17T17:01:11.289 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.disable_auto_compactions: 0 2024-08-17T17:01:11.289 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-08-17T17:01:11.289 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-08-17T17:01:11.289 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-08-17T17:01:11.290 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-08-17T17:01:11.290 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-08-17T17:01:11.290 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-08-17T17:01:11.290 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-08-17T17:01:11.290 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-08-17T17:01:11.290 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-08-17T17:01:11.290 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-08-17T17:01:11.290 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-08-17T17:01:11.290 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.inplace_update_support: 0 2024-08-17T17:01:11.290 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.inplace_update_num_locks: 10000 2024-08-17T17:01:11.290 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-08-17T17:01:11.290 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-08-17T17:01:11.290 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.memtable_huge_page_size: 0 2024-08-17T17:01:11.290 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.bloom_locality: 0 2024-08-17T17:01:11.291 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.max_successive_merges: 0 2024-08-17T17:01:11.291 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.optimize_filters_for_hits: 0 2024-08-17T17:01:11.291 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.paranoid_file_checks: 0 2024-08-17T17:01:11.291 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.force_consistency_checks: 1 2024-08-17T17:01:11.291 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.report_bg_io_stats: 0 2024-08-17T17:01:11.291 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.ttl: 2592000 2024-08-17T17:01:11.291 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.periodic_compaction_seconds: 0 2024-08-17T17:01:11.291 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.enable_blob_files: false 2024-08-17T17:01:11.291 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.min_blob_size: 0 2024-08-17T17:01:11.291 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.blob_file_size: 268435456 2024-08-17T17:01:11.291 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.blob_compression_type: NoCompression 2024-08-17T17:01:11.292 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.enable_blob_garbage_collection: false 2024-08-17T17:01:11.292 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-08-17T17:01:11.292 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-08-17T17:01:11.292 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-08-17T17:01:11.292 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-08-17T17:01:11.292 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723914071042540, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-08-17T17:01:11.292 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-08-17T17:01:11.292 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-08-17T17:01:11.292 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723914071043317, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 80226, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 78468, "index_size": 220, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10130, "raw_average_key_size": 47, "raw_value_size": 72682, "raw_average_value_size": 341, "num_data_blocks": 10, "num_entries": 213, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1723914071, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "00c0f85f-041b-452f-9aa9-de5ee3f16f85", "db_session_id": "D9W7JCGN24M5JIIEW7QM"}} 2024-08-17T17:01:11.292 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-08-17T17:01:11.292 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723914071043709, "job": 1, "event": "recovery_finished"} 2024-08-17T17:01:11.292 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-08-17T17:01:11.292 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x5568ed656700 2024-08-17T17:01:11.292 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: DB pointer 0x5568ed626000 2024-08-17T17:01:11.292 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-08-17T17:01:11.293 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-08-17T17:01:11.293 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** DB Stats ** 2024-08-17T17:01:11.293 INFO:journalctl@ceph.mon.a.smithi003.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-17T17:01:11.293 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-08-17T17:01:11.293 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-17T17:01:11.293 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-17T17:01:11.293 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-08-17T17:01:11.293 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-08-17T17:01:11.293 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-17T17:01:11.293 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-08-17T17:01:11.293 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** Compaction Stats [default] ** 2024-08-17T17:01:11.293 INFO:journalctl@ceph.mon.a.smithi003.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-08-17T17:01:11.293 INFO:journalctl@ceph.mon.a.smithi003.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-08-17T17:01:11.293 INFO:journalctl@ceph.mon.a.smithi003.stdout: L0 2/0 80.13 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 152.4 0.00 0.00 1 0.001 0 0 2024-08-17T17:01:11.293 INFO:journalctl@ceph.mon.a.smithi003.stdout: Sum 2/0 80.13 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 152.4 0.00 0.00 1 0.001 0 0 2024-08-17T17:01:11.294 INFO:journalctl@ceph.mon.a.smithi003.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 152.4 0.00 0.00 1 0.001 0 0 2024-08-17T17:01:11.294 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-08-17T17:01:11.294 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** Compaction Stats [default] ** 2024-08-17T17:01:11.294 INFO:journalctl@ceph.mon.a.smithi003.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-08-17T17:01:11.294 INFO:journalctl@ceph.mon.a.smithi003.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-08-17T17:01:11.294 INFO:journalctl@ceph.mon.a.smithi003.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 152.4 0.00 0.00 1 0.001 0 0 2024-08-17T17:01:11.294 INFO:journalctl@ceph.mon.a.smithi003.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-17T17:01:11.294 INFO:journalctl@ceph.mon.a.smithi003.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-08-17T17:01:11.294 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-08-17T17:01:11.294 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-08-17T17:01:11.294 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-08-17T17:01:11.294 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(Keys): cumulative 0, interval 0 2024-08-17T17:01:11.294 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative compaction: 0.00 GB write, 24.03 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-17T17:01:11.295 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval compaction: 0.00 GB write, 24.03 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-17T17:01:11.295 INFO:journalctl@ceph.mon.a.smithi003.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-08-17T17:01:11.295 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-08-17T17:01:11.295 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** File Read Latency Histogram By Level [default] ** 2024-08-17T17:01:11.295 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-08-17T17:01:11.295 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** Compaction Stats [default] ** 2024-08-17T17:01:11.295 INFO:journalctl@ceph.mon.a.smithi003.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-08-17T17:01:11.295 INFO:journalctl@ceph.mon.a.smithi003.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-08-17T17:01:11.295 INFO:journalctl@ceph.mon.a.smithi003.stdout: L0 2/0 80.13 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 152.4 0.00 0.00 1 0.001 0 0 2024-08-17T17:01:11.295 INFO:journalctl@ceph.mon.a.smithi003.stdout: Sum 2/0 80.13 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 152.4 0.00 0.00 1 0.001 0 0 2024-08-17T17:01:11.295 INFO:journalctl@ceph.mon.a.smithi003.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-08-17T17:01:11.295 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-08-17T17:01:11.295 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** Compaction Stats [default] ** 2024-08-17T17:01:11.298 INFO:journalctl@ceph.mon.a.smithi003.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-08-17T17:01:11.298 INFO:journalctl@ceph.mon.a.smithi003.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-08-17T17:01:11.298 INFO:journalctl@ceph.mon.a.smithi003.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 152.4 0.00 0.00 1 0.001 0 0 2024-08-17T17:01:11.298 INFO:journalctl@ceph.mon.a.smithi003.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-17T17:01:11.298 INFO:journalctl@ceph.mon.a.smithi003.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-08-17T17:01:11.299 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-08-17T17:01:11.299 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-08-17T17:01:11.299 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-08-17T17:01:11.299 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(Keys): cumulative 0, interval 0 2024-08-17T17:01:11.299 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative compaction: 0.00 GB write, 23.78 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-17T17:01:11.299 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-17T17:01:11.299 INFO:journalctl@ceph.mon.a.smithi003.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-08-17T17:01:11.299 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-08-17T17:01:11.299 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** File Read Latency Histogram By Level [default] ** 2024-08-17T17:01:11.299 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: starting mon.a rank 0 at public addrs [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] at bind addrs [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:01:11.299 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: mon.a@-1(???) e1 preinit fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:01:11.299 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: mon.a@-1(???).mds e1 new map 2024-08-17T17:01:11.299 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: mon.a@-1(???).mds e1 print_map 2024-08-17T17:01:11.300 INFO:journalctl@ceph.mon.a.smithi003.stdout: e1 2024-08-17T17:01:11.300 INFO:journalctl@ceph.mon.a.smithi003.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-08-17T17:01:11.300 INFO:journalctl@ceph.mon.a.smithi003.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-08-17T17:01:11.300 INFO:journalctl@ceph.mon.a.smithi003.stdout: legacy client fscid: -1 2024-08-17T17:01:11.300 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-08-17T17:01:11.300 INFO:journalctl@ceph.mon.a.smithi003.stdout: No filesystems configured 2024-08-17T17:01:11.300 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-08-17T17:01:11.300 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-08-17T17:01:11.300 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-08-17T17:01:11.300 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-08-17T17:01:11.300 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-08-17T17:01:11.300 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 bash[30784]: a8b52ab645d32066a4c2bae858d2be3a8353be59f545703bac701da0f07d1193 2024-08-17T17:01:11.300 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 systemd[1]: Started Ceph mon.a for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:01:11.300 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: mon.a is new leader, mons a in quorum (ranks 0) 2024-08-17T17:01:11.301 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: monmap e1: 1 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-08-17T17:01:11.301 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: fsmap 2024-08-17T17:01:11.301 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: osdmap e1: 0 total, 0 up, 0 in 2024-08-17T17:01:11.301 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:11 smithi003 ceph-mon[30803]: mgrmap e1: no daemons active 2024-08-17T17:01:12.284 INFO:teuthology.orchestra.run.smithi003.stdout:Wrote config to /etc/ceph/ceph.conf 2024-08-17T17:01:12.284 INFO:teuthology.orchestra.run.smithi003.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-08-17T17:01:12.285 INFO:teuthology.orchestra.run.smithi003.stdout:Creating mgr... 2024-08-17T17:01:12.285 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying port 0.0.0.0:9283 ... 2024-08-17T17:01:12.285 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying port 0.0.0.0:8765 ... 2024-08-17T17:01:12.471 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mgr.y 2024-08-17T17:01:12.471 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Failed to reset failed state of unit ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mgr.y.service: Unit ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mgr.y.service not loaded. 2024-08-17T17:01:12.629 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-2b5b662e-5cba-11ef-bccf-c7b262605968.target.wants/ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@.service. 2024-08-17T17:01:12.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:12 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/2441750214' entity='client.admin' 2024-08-17T17:01:12.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:12 smithi003 systemd[1]: Starting Ceph mgr.y for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:01:13.143 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:12 smithi003 podman[31015]: 2024-08-17 17:01:12.864943429 +0000 UTC m=+0.109516053 container create 8cb7113bf953089166faaac456449b9d60ed53dc66b49333d89a0b40b6dad626 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=quincy-f2c39d4, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-08-17T17:01:13.143 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:12 smithi003 podman[31015]: 2024-08-17 17:01:12.773574067 +0000 UTC m=+0.018146645 image pull f69471c56189b063ba7a049adea817a12ba1a40187d11cab5cbef8f3aac57cce quay.ceph.io/ceph-ci/ceph:quincy 2024-08-17T17:01:13.143 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:13 smithi003 podman[31015]: 2024-08-17 17:01:13.042585857 +0000 UTC m=+0.287158447 container init 8cb7113bf953089166faaac456449b9d60ed53dc66b49333d89a0b40b6dad626 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-f2c39d4, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-17T17:01:13.143 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:13 smithi003 podman[31015]: 2024-08-17 17:01:13.045385176 +0000 UTC m=+0.289957754 container start 8cb7113bf953089166faaac456449b9d60ed53dc66b49333d89a0b40b6dad626 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, RELEASE=quincy-f2c39d4, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-17T17:01:13.143 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:13 smithi003 bash[31015]: 8cb7113bf953089166faaac456449b9d60ed53dc66b49333d89a0b40b6dad626 2024-08-17T17:01:13.157 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-08-17T17:01:13.157 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout disabled 2024-08-17T17:01:13.165 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-08-17T17:01:13.165 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2024-08-17T17:01:13.165 INFO:teuthology.orchestra.run.smithi003.stdout:firewalld.service is not enabled 2024-08-17T17:01:13.166 INFO:teuthology.orchestra.run.smithi003.stdout:Not possible to enable service . firewalld.service is not available 2024-08-17T17:01:13.175 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-08-17T17:01:13.175 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout disabled 2024-08-17T17:01:13.183 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-08-17T17:01:13.183 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2024-08-17T17:01:13.184 INFO:teuthology.orchestra.run.smithi003.stdout:firewalld.service is not enabled 2024-08-17T17:01:13.184 INFO:teuthology.orchestra.run.smithi003.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-08-17T17:01:13.184 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mgr to start... 2024-08-17T17:01:13.184 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mgr... 2024-08-17T17:01:13.406 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:13 smithi003 systemd[1]: Started Ceph mgr.y for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:01:13.406 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:13 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:13.166+0000 7f3ea569fec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-17T17:01:13.406 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:13 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:13.298+0000 7f3ea569fec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-17T17:01:13.406 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:13 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:13.366+0000 7f3ea569fec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-17T17:01:13.691 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:13 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:13.629+0000 7f3ea569fec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-17T17:01:13.990 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:13 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/2702226177' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-08-17T17:01:13.991 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:13 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:13.693+0000 7f3ea569fec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-17T17:01:14.258 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-08-17T17:01:14.258 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-08-17T17:01:14.258 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsid": "2b5b662e-5cba-11ef-bccf-c7b262605968", 2024-08-17T17:01:14.259 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "health": { 2024-08-17T17:01:14.259 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-08-17T17:01:14.259 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-08-17T17:01:14.259 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-08-17T17:01:14.259 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:01:14.259 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-08-17T17:01:14.259 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-08-17T17:01:14.259 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 0 2024-08-17T17:01:14.259 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-08-17T17:01:14.259 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-08-17T17:01:14.259 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "a" 2024-08-17T17:01:14.259 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-08-17T17:01:14.259 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-08-17T17:01:14.259 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "monmap": { 2024-08-17T17:01:14.260 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-17T17:01:14.260 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-08-17T17:01:14.260 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-08-17T17:01:14.260 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:01:14.260 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-08-17T17:01:14.260 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-17T17:01:14.260 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-08-17T17:01:14.260 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-08-17T17:01:14.260 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-08-17T17:01:14.260 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-08-17T17:01:14.260 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-08-17T17:01:14.260 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-08-17T17:01:14.260 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:01:14.260 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-08-17T17:01:14.260 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-08-17T17:01:14.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-08-17T17:01:14.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-08-17T17:01:14.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-08-17T17:01:14.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-08-17T17:01:14.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-08-17T17:01:14.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-08-17T17:01:14.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-08-17T17:01:14.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:01:14.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-08-17T17:01:14.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-17T17:01:14.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-08-17T17:01:14.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-08-17T17:01:14.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:01:14.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-08-17T17:01:14.262 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": false, 2024-08-17T17:01:14.262 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-08-17T17:01:14.262 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modules": [ 2024-08-17T17:01:14.262 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "iostat", 2024-08-17T17:01:14.262 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "nfs", 2024-08-17T17:01:14.262 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "restful" 2024-08-17T17:01:14.262 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-08-17T17:01:14.262 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-17T17:01:14.262 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:01:14.262 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-08-17T17:01:14.262 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-17T17:01:14.262 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modified": "2024-08-17T17:00:56.373373+0000", 2024-08-17T17:01:14.262 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-17T17:01:14.263 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:01:14.263 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-08-17T17:01:14.263 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-08-17T17:01:14.263 INFO:teuthology.orchestra.run.smithi003.stdout:mgr not available, waiting (1/15)... 2024-08-17T17:01:15.021 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:14 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:14.650+0000 7f3ea569fec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-17T17:01:15.021 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:14 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:14.766+0000 7f3ea569fec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-17T17:01:15.021 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:14 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:14.823+0000 7f3ea569fec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-17T17:01:15.021 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:15.019+0000 7f3ea569fec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-17T17:01:15.273 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:15.083+0000 7f3ea569fec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-17T17:01:15.273 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:15.152+0000 7f3ea569fec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-17T17:01:15.724 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:15.282+0000 7f3ea569fec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-17T17:01:16.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:15.722+0000 7f3ea569fec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-17T17:01:16.287 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:16 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:16.027+0000 7f3ea569fec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-17T17:01:16.287 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:16 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:16.096+0000 7f3ea569fec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-17T17:01:16.287 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:16 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:16.228+0000 7f3ea569fec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-17T17:01:16.560 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:16 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:16.496+0000 7f3ea569fec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-17T17:01:16.560 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:16 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:16.559+0000 7f3ea569fec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-17T17:01:16.821 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:16 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:16.753+0000 7f3ea569fec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-17T17:01:16.821 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:16 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:16.819+0000 7f3ea569fec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-17T17:01:17.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:16 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/2956738695' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-08-17T17:01:17.273 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:16 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:16.907+0000 7f3ea569fec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-17T17:01:17.273 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:16 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:16.968+0000 7f3ea569fec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-17T17:01:17.400 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-08-17T17:01:17.400 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-08-17T17:01:17.400 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsid": "2b5b662e-5cba-11ef-bccf-c7b262605968", 2024-08-17T17:01:17.400 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "health": { 2024-08-17T17:01:17.400 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-08-17T17:01:17.400 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-08-17T17:01:17.400 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-08-17T17:01:17.400 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:01:17.400 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-08-17T17:01:17.400 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-08-17T17:01:17.401 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 0 2024-08-17T17:01:17.401 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-08-17T17:01:17.401 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-08-17T17:01:17.401 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "a" 2024-08-17T17:01:17.401 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-08-17T17:01:17.401 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-08-17T17:01:17.401 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "monmap": { 2024-08-17T17:01:17.401 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-17T17:01:17.401 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-08-17T17:01:17.401 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-08-17T17:01:17.401 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:01:17.401 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-08-17T17:01:17.402 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-17T17:01:17.402 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-08-17T17:01:17.402 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-08-17T17:01:17.402 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-08-17T17:01:17.402 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-08-17T17:01:17.402 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-08-17T17:01:17.402 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-08-17T17:01:17.405 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:01:17.405 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-08-17T17:01:17.405 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-08-17T17:01:17.405 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-08-17T17:01:17.405 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-08-17T17:01:17.405 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-08-17T17:01:17.405 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-08-17T17:01:17.405 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-08-17T17:01:17.405 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-08-17T17:01:17.406 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-08-17T17:01:17.406 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:01:17.406 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-08-17T17:01:17.406 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-17T17:01:17.406 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-08-17T17:01:17.406 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-08-17T17:01:17.406 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:01:17.406 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-08-17T17:01:17.406 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": false, 2024-08-17T17:01:17.406 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-08-17T17:01:17.406 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modules": [ 2024-08-17T17:01:17.406 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "iostat", 2024-08-17T17:01:17.406 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "nfs", 2024-08-17T17:01:17.406 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "restful" 2024-08-17T17:01:17.406 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-08-17T17:01:17.407 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-17T17:01:17.407 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:01:17.407 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-08-17T17:01:17.407 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-17T17:01:17.407 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modified": "2024-08-17T17:00:56.373373+0000", 2024-08-17T17:01:17.407 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-17T17:01:17.407 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:01:17.407 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-08-17T17:01:17.407 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-08-17T17:01:17.407 INFO:teuthology.orchestra.run.smithi003.stdout:mgr not available, waiting (2/15)... 2024-08-17T17:01:17.656 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:17 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:17.514+0000 7f3ea569fec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-17T17:01:17.656 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:17 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:17.655+0000 7f3ea569fec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-17T17:01:18.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:17 smithi003 ceph-mon[30803]: Activating manager daemon y 2024-08-17T17:01:18.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:17 smithi003 ceph-mon[30803]: mgrmap e2: y(active, starting, since 0.00297257s) 2024-08-17T17:01:18.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:17 smithi003 ceph-mon[30803]: from='mgr.14100 172.21.15.3:0/1613577453' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-17T17:01:18.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:17 smithi003 ceph-mon[30803]: from='mgr.14100 172.21.15.3:0/1613577453' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-17T17:01:18.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:17 smithi003 ceph-mon[30803]: from='mgr.14100 172.21.15.3:0/1613577453' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-17T17:01:18.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:17 smithi003 ceph-mon[30803]: from='mgr.14100 172.21.15.3:0/1613577453' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:01:18.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:17 smithi003 ceph-mon[30803]: from='mgr.14100 172.21.15.3:0/1613577453' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-17T17:01:18.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:17 smithi003 ceph-mon[30803]: Manager daemon y is now available 2024-08-17T17:01:18.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:17 smithi003 ceph-mon[30803]: from='mgr.14100 172.21.15.3:0/1613577453' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:01:18.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:17 smithi003 ceph-mon[30803]: from='mgr.14100 172.21.15.3:0/1613577453' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-08-17T17:01:18.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:17 smithi003 ceph-mon[30803]: from='mgr.14100 172.21.15.3:0/1613577453' entity='mgr.y' 2024-08-17T17:01:18.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:17 smithi003 ceph-mon[30803]: from='mgr.14100 172.21.15.3:0/1613577453' entity='mgr.y' 2024-08-17T17:01:18.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:17 smithi003 ceph-mon[30803]: from='mgr.14100 172.21.15.3:0/1613577453' entity='mgr.y' 2024-08-17T17:01:19.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:19 smithi003 ceph-mon[30803]: mgrmap e3: y(active, since 1.00703s) 2024-08-17T17:01:20.642 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-08-17T17:01:20.643 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-08-17T17:01:20.643 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsid": "2b5b662e-5cba-11ef-bccf-c7b262605968", 2024-08-17T17:01:20.643 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "health": { 2024-08-17T17:01:20.643 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-08-17T17:01:20.643 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-08-17T17:01:20.643 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-08-17T17:01:20.643 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:01:20.643 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-08-17T17:01:20.643 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-08-17T17:01:20.643 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 0 2024-08-17T17:01:20.644 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-08-17T17:01:20.644 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-08-17T17:01:20.644 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "a" 2024-08-17T17:01:20.644 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-08-17T17:01:20.644 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-08-17T17:01:20.644 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "monmap": { 2024-08-17T17:01:20.644 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-17T17:01:20.644 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-08-17T17:01:20.644 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-08-17T17:01:20.644 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:01:20.644 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-08-17T17:01:20.644 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-17T17:01:20.644 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-08-17T17:01:20.645 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-08-17T17:01:20.645 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-08-17T17:01:20.645 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-08-17T17:01:20.645 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-08-17T17:01:20.645 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-08-17T17:01:20.645 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:01:20.645 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-08-17T17:01:20.645 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-08-17T17:01:20.645 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-08-17T17:01:20.645 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-08-17T17:01:20.645 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-08-17T17:01:20.645 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-08-17T17:01:20.645 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-08-17T17:01:20.645 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-08-17T17:01:20.646 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-08-17T17:01:20.646 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:01:20.646 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-08-17T17:01:20.646 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-17T17:01:20.646 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-08-17T17:01:20.646 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-08-17T17:01:20.646 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:01:20.646 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-08-17T17:01:20.646 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": true, 2024-08-17T17:01:20.646 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-08-17T17:01:20.646 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modules": [ 2024-08-17T17:01:20.646 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "iostat", 2024-08-17T17:01:20.646 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "nfs", 2024-08-17T17:01:20.647 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "restful" 2024-08-17T17:01:20.647 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-08-17T17:01:20.647 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-17T17:01:20.647 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:01:20.647 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-08-17T17:01:20.647 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-17T17:01:20.647 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modified": "2024-08-17T17:00:56.373373+0000", 2024-08-17T17:01:20.647 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-17T17:01:20.647 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-08-17T17:01:20.647 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-08-17T17:01:20.648 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-08-17T17:01:20.648 INFO:teuthology.orchestra.run.smithi003.stdout:mgr is available 2024-08-17T17:01:20.833 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:20 smithi003 ceph-mon[30803]: mgrmap e4: y(active, since 2s) 2024-08-17T17:01:20.833 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:20 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/2066371622' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-08-17T17:01:22.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:21 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/869950070' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-08-17T17:01:22.205 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-08-17T17:01:22.205 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [global] 2024-08-17T17:01:22.205 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout fsid = 2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:01:22.205 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-08-17T17:01:22.205 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-08-17T17:01:22.205 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-08-17T17:01:22.205 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-08-17T17:01:22.206 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-08-17T17:01:22.206 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-08-17T17:01:22.206 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [mgr] 2024-08-17T17:01:22.206 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-08-17T17:01:22.206 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-08-17T17:01:22.206 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [osd] 2024-08-17T17:01:22.206 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-08-17T17:01:22.206 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-08-17T17:01:22.206 INFO:teuthology.orchestra.run.smithi003.stdout:Enabling cephadm module... 2024-08-17T17:01:23.156 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:22 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/1761273789' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-08-17T17:01:23.156 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:22 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: ignoring --setuser ceph since I am not root 2024-08-17T17:01:23.157 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:22 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: ignoring --setgroup ceph since I am not root 2024-08-17T17:01:23.157 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:23 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:23.028+0000 7f66aef2eec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-17T17:01:23.157 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:23 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:23.155+0000 7f66aef2eec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-17T17:01:23.447 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:23 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:23.222+0000 7f66aef2eec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-17T17:01:23.699 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:23 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:23.475+0000 7f66aef2eec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-17T17:01:23.699 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:23 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:23.535+0000 7f66aef2eec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-17T17:01:24.094 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:23 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/1761273789' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-08-17T17:01:24.095 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:23 smithi003 ceph-mon[30803]: mgrmap e5: y(active, since 5s) 2024-08-17T17:01:24.666 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-08-17T17:01:24.666 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-08-17T17:01:24.666 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": true, 2024-08-17T17:01:24.666 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-08-17T17:01:24.666 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-08-17T17:01:24.666 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-08-17T17:01:24.666 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for the mgr to restart... 2024-08-17T17:01:24.666 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mgr epoch 5... 2024-08-17T17:01:24.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:24 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:24.481+0000 7f66aef2eec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-17T17:01:24.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:24 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:24.598+0000 7f66aef2eec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-17T17:01:24.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:24 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:24.655+0000 7f66aef2eec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-17T17:01:25.115 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:24 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/335699943' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-08-17T17:01:25.115 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:24 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:24.854+0000 7f66aef2eec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-17T17:01:25.115 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:24 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:24.918+0000 7f66aef2eec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-17T17:01:25.116 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:24 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:24.986+0000 7f66aef2eec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-17T17:01:25.523 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:25.116+0000 7f66aef2eec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-17T17:01:25.846 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:25.545+0000 7f66aef2eec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-17T17:01:25.847 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:25.845+0000 7f66aef2eec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-17T17:01:26.273 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:25.912+0000 7f66aef2eec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-17T17:01:26.273 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:26 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:26.042+0000 7f66aef2eec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-17T17:01:26.558 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:26 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:26.305+0000 7f66aef2eec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-17T17:01:26.558 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:26 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:26.369+0000 7f66aef2eec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-17T17:01:27.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:26 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:26.557+0000 7f66aef2eec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-17T17:01:27.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:26 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:26.621+0000 7f66aef2eec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-17T17:01:27.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:26 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:26.705+0000 7f66aef2eec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-17T17:01:27.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:26 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:26.763+0000 7f66aef2eec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-17T17:01:27.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:27 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:27.300+0000 7f66aef2eec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-17T17:01:27.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:27 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:27.440+0000 7f66aef2eec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-17T17:01:27.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:27 smithi003 ceph-mon[30803]: Active manager daemon y restarted 2024-08-17T17:01:27.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:27 smithi003 ceph-mon[30803]: Activating manager daemon y 2024-08-17T17:01:27.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:27 smithi003 ceph-mon[30803]: osdmap e2: 0 total, 0 up, 0 in 2024-08-17T17:01:27.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:27 smithi003 ceph-mon[30803]: mgrmap e6: y(active, starting, since 0.00313723s) 2024-08-17T17:01:27.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:27 smithi003 ceph-mon[30803]: from='mgr.14118 172.21.15.3:0/2297936839' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:01:27.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:27 smithi003 ceph-mon[30803]: from='mgr.14118 172.21.15.3:0/2297936839' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-17T17:01:27.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:27 smithi003 ceph-mon[30803]: from='mgr.14118 172.21.15.3:0/2297936839' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-17T17:01:27.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:27 smithi003 ceph-mon[30803]: from='mgr.14118 172.21.15.3:0/2297936839' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-17T17:01:27.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:27 smithi003 ceph-mon[30803]: from='mgr.14118 172.21.15.3:0/2297936839' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-17T17:01:27.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:27 smithi003 ceph-mon[30803]: Manager daemon y is now available 2024-08-17T17:01:27.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:27 smithi003 ceph-mon[30803]: from='mgr.14118 172.21.15.3:0/2297936839' entity='mgr.y' 2024-08-17T17:01:27.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:27 smithi003 ceph-mon[30803]: from='mgr.14118 172.21.15.3:0/2297936839' entity='mgr.y' 2024-08-17T17:01:27.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:27 smithi003 ceph-mon[30803]: from='mgr.14118 172.21.15.3:0/2297936839' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:01:27.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:27 smithi003 ceph-mon[30803]: from='mgr.14118 172.21.15.3:0/2297936839' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:01:27.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:27 smithi003 ceph-mon[30803]: from='mgr.14118 172.21.15.3:0/2297936839' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:01:27.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:27 smithi003 ceph-mon[30803]: from='mgr.14118 172.21.15.3:0/2297936839' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-08-17T17:01:28.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:28 smithi003 ceph-mon[30803]: Found migration_current of "None". Setting to last migration. 2024-08-17T17:01:28.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:28 smithi003 ceph-mon[30803]: mgrmap e7: y(active, since 1.00494s) 2024-08-17T17:01:29.022 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-08-17T17:01:29.022 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-08-17T17:01:29.022 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "initialized": true 2024-08-17T17:01:29.022 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-08-17T17:01:29.022 INFO:teuthology.orchestra.run.smithi003.stdout:mgr epoch 5 is available 2024-08-17T17:01:29.022 INFO:teuthology.orchestra.run.smithi003.stdout:Setting orchestrator backend to cephadm... 2024-08-17T17:01:29.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:29 smithi003 ceph-mon[30803]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-08-17T17:01:29.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:29 smithi003 ceph-mon[30803]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-08-17T17:01:30.437 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:30 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:30.364+0000 7f6661275640 -1 log_channel(cephadm) log [ERR] : [17/Aug/2024:17:01:30] ENGINE Error in HTTPServer.serve 2024-08-17T17:01:30.437 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:30 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: Traceback (most recent call last): 2024-08-17T17:01:30.437 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:30 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:01:30.437 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:30 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: self._connections.run(self.expiration_interval) 2024-08-17T17:01:30.437 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:30 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:01:30.437 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:30 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: self._run(expiration_interval) 2024-08-17T17:01:30.437 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:30 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:01:30.437 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:30 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:01:30.437 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:30 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:01:30.437 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:30 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:01:30.437 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:30 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:01:30.437 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:30 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: s = self.context.wrap_socket( 2024-08-17T17:01:30.438 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:30 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:01:30.438 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:30 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: return self.sslsocket_class._create( 2024-08-17T17:01:30.438 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:30 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:01:30.438 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:30 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: self.do_handshake() 2024-08-17T17:01:30.438 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:30 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:01:30.438 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:30 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: self._sslobj.do_handshake() 2024-08-17T17:01:30.438 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:30 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:01:30.438 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:30 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:30.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:30 smithi003 ceph-mon[30803]: mgrmap e8: y(active, since 2s) 2024-08-17T17:01:30.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:30 smithi003 ceph-mon[30803]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:01:30.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:30 smithi003 ceph-mon[30803]: from='mgr.14118 172.21.15.3:0/2297936839' entity='mgr.y' 2024-08-17T17:01:30.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:30 smithi003 ceph-mon[30803]: from='mgr.14118 172.21.15.3:0/2297936839' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:01:30.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:30 smithi003 ceph-mon[30803]: from='mgr.14118 172.21.15.3:0/2297936839' entity='mgr.y' 2024-08-17T17:01:30.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:30 smithi003 ceph-mon[30803]: from='mgr.14118 172.21.15.3:0/2297936839' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:01:31.276 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout value unchanged 2024-08-17T17:01:31.276 INFO:teuthology.orchestra.run.smithi003.stdout:Generating ssh key... 2024-08-17T17:01:31.593 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:31 smithi003 ceph-mon[30803]: [17/Aug/2024:17:01:30] ENGINE Bus STARTING 2024-08-17T17:01:31.593 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:31 smithi003 ceph-mon[30803]: [17/Aug/2024:17:01:30] ENGINE Serving on https://172.21.15.3:7150 2024-08-17T17:01:31.593 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:31 smithi003 ceph-mon[30803]: [17/Aug/2024:17:01:30] ENGINE Bus STARTED 2024-08-17T17:01:31.593 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:31 smithi003 ceph-mon[30803]: [17/Aug/2024:17:01:30] ENGINE Error in HTTPServer.serve 2024-08-17T17:01:31.593 INFO:journalctl@ceph.mon.a.smithi003.stdout: Traceback (most recent call last): 2024-08-17T17:01:31.593 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:01:31.593 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._connections.run(self.expiration_interval) 2024-08-17T17:01:31.594 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:01:31.594 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._run(expiration_interval) 2024-08-17T17:01:31.594 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:01:31.594 INFO:journalctl@ceph.mon.a.smithi003.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:01:31.594 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:01:31.594 INFO:journalctl@ceph.mon.a.smithi003.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:01:31.594 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:01:31.594 INFO:journalctl@ceph.mon.a.smithi003.stdout: s = self.context.wrap_socket( 2024-08-17T17:01:31.594 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:01:31.594 INFO:journalctl@ceph.mon.a.smithi003.stdout: return self.sslsocket_class._create( 2024-08-17T17:01:31.594 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:01:31.594 INFO:journalctl@ceph.mon.a.smithi003.stdout: self.do_handshake() 2024-08-17T17:01:31.594 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:01:31.595 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._sslobj.do_handshake() 2024-08-17T17:01:31.595 INFO:journalctl@ceph.mon.a.smithi003.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:01:31.595 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:31 smithi003 ceph-mon[30803]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:01:32.338 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:32 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: Generating public/private rsa key pair. 2024-08-17T17:01:32.338 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:32 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: Your identification has been saved in /tmp/tmpjd857b8y/key 2024-08-17T17:01:32.338 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:32 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: Your public key has been saved in /tmp/tmpjd857b8y/key.pub 2024-08-17T17:01:32.338 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:32 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: The key fingerprint is: 2024-08-17T17:01:32.338 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:32 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: SHA256:5ND09CXzj1EgqKw6YJyrkbKtwLWZa8ULejo47N/JQwo ceph-2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:01:32.338 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:32 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: The key's randomart image is: 2024-08-17T17:01:32.338 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:32 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: +---[RSA 3072]----+ 2024-08-17T17:01:32.338 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:32 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: | . ..+ o..| 2024-08-17T17:01:32.338 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:32 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: | o o.. * . | 2024-08-17T17:01:32.339 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:32 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: | ..o.. . o | 2024-08-17T17:01:32.339 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:32 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: | +o + | 2024-08-17T17:01:32.339 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:32 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: | . o. .S . .| 2024-08-17T17:01:32.339 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:32 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: |.E*.++. | 2024-08-17T17:01:32.339 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:32 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: |Boo*=.. | 2024-08-17T17:01:32.339 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:32 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: |=B.+*o. | 2024-08-17T17:01:32.339 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:32 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: |=*Bo =. | 2024-08-17T17:01:32.339 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:32 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: +----[SHA256]-----+ 2024-08-17T17:01:32.630 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:32 smithi003 ceph-mon[30803]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:01:32.630 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:32 smithi003 ceph-mon[30803]: Generating ssh key... 2024-08-17T17:01:32.630 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:32 smithi003 ceph-mon[30803]: from='mgr.14118 172.21.15.3:0/2297936839' entity='mgr.y' 2024-08-17T17:01:32.630 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:32 smithi003 ceph-mon[30803]: from='mgr.14118 172.21.15.3:0/2297936839' entity='mgr.y' 2024-08-17T17:01:33.690 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgMPqUE5x2Yw96xO+OHChQNFbNwLaBlK0DP8ZQAKqIEi87/zkiKpUuMN8RJ0DN0PxNm8qx0t4Oa673UqGHuSVQYSdlPTNHns4JITWcdgAPqjQx9H+RjytzS53h/P5VapOgaL1azWlv+xv1KwRGGKgi1kkhgA+6CFVZ0Ox4GGPX6O1rCFbg4mfoEs7e1Rg39/D1FXv5BUE9WB5Y50bV30CYaRWJAosp4b/RCXfnl8cZdKmkparsEuGf1sV1nGVeXi+pH+B8s5axXdQwD8O4HXPruNyw+AxdFns5kMrw0eve2FlwxCVpDCJBpFmX9lFHOonZOPf35eepWXJHQj5Flrg5jHdKOYloYoUoaC5nMW8WzfAPCjfAnOOEelVFkFGpn+t5bLIF3yP/YL6QN0LJHc7FhsDEScwYDLGFCNjsGh9wF67eHdUi4HZOhZLWXCN57BY2DsWS/9FMUWtfDLENPfUvwbDrmvHMNUiMxZPPBsMMs0xllGoBDDQs3yOc4GvnCCM= ceph-2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:01:33.690 INFO:teuthology.orchestra.run.smithi003.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-08-17T17:01:33.690 INFO:teuthology.orchestra.run.smithi003.stdout:Adding key to root@localhost authorized_keys... 2024-08-17T17:01:33.691 INFO:teuthology.orchestra.run.smithi003.stdout:Adding host smithi003... 2024-08-17T17:01:34.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:33 smithi003 ceph-mon[30803]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:01:35.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:34 smithi003 ceph-mon[30803]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi003", "addr": "172.21.15.3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:01:36.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:35 smithi003 ceph-mon[30803]: Deploying cephadm binary to smithi003 2024-08-17T17:01:38.265 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout Added host 'smithi003' with addr '172.21.15.3' 2024-08-17T17:01:38.265 INFO:teuthology.orchestra.run.smithi003.stdout:Deploying unmanaged mon service... 2024-08-17T17:01:38.860 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:38 smithi003 ceph-mon[30803]: from='mgr.14118 172.21.15.3:0/2297936839' entity='mgr.y' 2024-08-17T17:01:38.860 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:38 smithi003 ceph-mon[30803]: Added host smithi003 2024-08-17T17:01:38.860 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:38 smithi003 ceph-mon[30803]: from='mgr.14118 172.21.15.3:0/2297936839' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:01:39.597 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-08-17T17:01:39.597 INFO:teuthology.orchestra.run.smithi003.stdout:Deploying unmanaged mgr service... 2024-08-17T17:01:40.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:39 smithi003 ceph-mon[30803]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:01:40.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:39 smithi003 ceph-mon[30803]: Saving service mon spec with placement count:5 2024-08-17T17:01:40.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:39 smithi003 ceph-mon[30803]: from='mgr.14118 172.21.15.3:0/2297936839' entity='mgr.y' 2024-08-17T17:01:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:41 smithi003 ceph-mon[30803]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:01:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:41 smithi003 ceph-mon[30803]: Saving service mgr spec with placement count:2 2024-08-17T17:01:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:41 smithi003 ceph-mon[30803]: from='mgr.14118 172.21.15.3:0/2297936839' entity='mgr.y' 2024-08-17T17:01:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:41 smithi003 ceph-mon[30803]: from='mgr.14118 172.21.15.3:0/2297936839' entity='mgr.y' 2024-08-17T17:01:41.628 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-08-17T17:01:43.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:43 smithi003 ceph-mon[30803]: from='mgr.14118 172.21.15.3:0/2297936839' entity='mgr.y' 2024-08-17T17:01:43.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:43 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/243687389' entity='client.admin' 2024-08-17T17:01:44.462 INFO:teuthology.orchestra.run.smithi003.stdout:Enabling the dashboard module... 2024-08-17T17:01:44.491 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:44 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/266507820' entity='client.admin' 2024-08-17T17:01:44.491 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:44 smithi003 ceph-mon[30803]: from='mgr.14118 172.21.15.3:0/2297936839' entity='mgr.y' 2024-08-17T17:01:46.054 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:46 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: ignoring --setuser ceph since I am not root 2024-08-17T17:01:46.055 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:46 smithi003 ceph-mon[30803]: from='mgr.14118 172.21.15.3:0/2297936839' entity='mgr.y' 2024-08-17T17:01:46.055 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:46 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/1908486160' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-08-17T17:01:46.356 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:46 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: ignoring --setgroup ceph since I am not root 2024-08-17T17:01:46.357 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:46 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:46.155+0000 7ff412c60ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-17T17:01:46.357 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:46 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:46.285+0000 7ff412c60ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-17T17:01:46.357 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:46 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:46.354+0000 7ff412c60ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-17T17:01:47.006 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:46 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:46.609+0000 7ff412c60ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-17T17:01:47.007 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:46 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:46.672+0000 7ff412c60ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-17T17:01:47.258 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:47 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/1908486160' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-08-17T17:01:47.258 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:47 smithi003 ceph-mon[30803]: mgrmap e9: y(active, since 18s) 2024-08-17T17:01:48.009 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:47 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:47.640+0000 7ff412c60ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-17T17:01:48.010 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:47 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:47.758+0000 7ff412c60ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-17T17:01:48.010 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:47 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:47.815+0000 7ff412c60ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-17T17:01:48.234 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-08-17T17:01:48.234 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-08-17T17:01:48.234 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": true, 2024-08-17T17:01:48.234 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-08-17T17:01:48.234 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-08-17T17:01:48.234 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-08-17T17:01:48.234 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for the mgr to restart... 2024-08-17T17:01:48.234 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mgr epoch 9... 2024-08-17T17:01:48.272 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:48 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/1478824367' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-08-17T17:01:48.273 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:48.014+0000 7ff412c60ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-17T17:01:48.273 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:48.077+0000 7ff412c60ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-17T17:01:48.273 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:48.145+0000 7ff412c60ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-17T17:01:48.558 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:48.276+0000 7ff412c60ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-17T17:01:49.021 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:48.719+0000 7ff412c60ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-17T17:01:49.021 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:49 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:49.019+0000 7ff412c60ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-17T17:01:49.273 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:49 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:49.086+0000 7ff412c60ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-17T17:01:49.273 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:49 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:49.218+0000 7ff412c60ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-17T17:01:49.737 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:49 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:49.484+0000 7ff412c60ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-17T17:01:49.737 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:49 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:49.546+0000 7ff412c60ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-17T17:01:50.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:49 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:49.735+0000 7ff412c60ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-17T17:01:50.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:49 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:49.800+0000 7ff412c60ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-17T17:01:50.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:49 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:49.884+0000 7ff412c60ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-17T17:01:50.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:49 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:49.943+0000 7ff412c60ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-17T17:01:50.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:50 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:50.488+0000 7ff412c60ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-17T17:01:50.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:50 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:50.628+0000 7ff412c60ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-17T17:01:50.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:50 smithi003 ceph-mon[30803]: Active manager daemon y restarted 2024-08-17T17:01:50.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:50 smithi003 ceph-mon[30803]: Activating manager daemon y 2024-08-17T17:01:50.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:50 smithi003 ceph-mon[30803]: osdmap e3: 0 total, 0 up, 0 in 2024-08-17T17:01:50.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:50 smithi003 ceph-mon[30803]: mgrmap e10: y(active, starting, since 0.00281037s) 2024-08-17T17:01:50.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:50 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:01:50.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:50 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-17T17:01:50.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:50 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-17T17:01:50.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:50 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-17T17:01:50.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:50 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-17T17:01:50.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:50 smithi003 ceph-mon[30803]: Manager daemon y is now available 2024-08-17T17:01:50.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:50 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:01:50.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:50 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:01:50.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:50 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-08-17T17:01:52.543 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-08-17T17:01:52.543 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-08-17T17:01:52.543 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "initialized": true 2024-08-17T17:01:52.543 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-08-17T17:01:52.543 INFO:teuthology.orchestra.run.smithi003.stdout:mgr epoch 9 is available 2024-08-17T17:01:52.543 INFO:teuthology.orchestra.run.smithi003.stdout:Generating a dashboard self-signed certificate... 2024-08-17T17:01:52.635 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:52 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:52.540+0000 7ff3c3796640 -1 log_channel(cephadm) log [ERR] : [17/Aug/2024:17:01:52] ENGINE Error in HTTPServer.serve 2024-08-17T17:01:52.635 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:52 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: Traceback (most recent call last): 2024-08-17T17:01:52.635 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:52 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:01:52.635 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:52 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: self._connections.run(self.expiration_interval) 2024-08-17T17:01:52.635 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:52 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:01:52.635 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:52 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: self._run(expiration_interval) 2024-08-17T17:01:52.635 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:52 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:01:52.635 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:52 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:01:52.635 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:52 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:01:52.636 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:52 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:01:52.636 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:52 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:01:52.636 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:52 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: s = self.context.wrap_socket( 2024-08-17T17:01:52.636 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:52 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:01:52.636 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:52 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: return self.sslsocket_class._create( 2024-08-17T17:01:52.636 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:52 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:01:52.636 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:52 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: self.do_handshake() 2024-08-17T17:01:52.636 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:52 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:01:52.636 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:52 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: self._sslobj.do_handshake() 2024-08-17T17:01:52.636 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:52 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:01:52.636 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:01:52 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:01:52.885 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:52 smithi003 ceph-mon[30803]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-08-17T17:01:52.885 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:52 smithi003 ceph-mon[30803]: mgrmap e11: y(active, since 1.00529s) 2024-08-17T17:01:52.885 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:52 smithi003 ceph-mon[30803]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-08-17T17:01:52.885 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:52 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:01:52.885 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:52 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:01:53.816 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-08-17T17:01:53.816 INFO:teuthology.orchestra.run.smithi003.stdout:Creating initial admin user... 2024-08-17T17:01:54.225 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:54 smithi003 ceph-mon[30803]: [17/Aug/2024:17:01:52] ENGINE Bus STARTING 2024-08-17T17:01:54.225 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:54 smithi003 ceph-mon[30803]: [17/Aug/2024:17:01:52] ENGINE Serving on https://172.21.15.3:7150 2024-08-17T17:01:54.225 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:54 smithi003 ceph-mon[30803]: [17/Aug/2024:17:01:52] ENGINE Bus STARTED 2024-08-17T17:01:54.225 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:54 smithi003 ceph-mon[30803]: [17/Aug/2024:17:01:52] ENGINE Error in HTTPServer.serve 2024-08-17T17:01:54.225 INFO:journalctl@ceph.mon.a.smithi003.stdout: Traceback (most recent call last): 2024-08-17T17:01:54.226 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:01:54.226 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._connections.run(self.expiration_interval) 2024-08-17T17:01:54.226 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:01:54.226 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._run(expiration_interval) 2024-08-17T17:01:54.226 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:01:54.226 INFO:journalctl@ceph.mon.a.smithi003.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:01:54.226 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:01:54.226 INFO:journalctl@ceph.mon.a.smithi003.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:01:54.226 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:01:54.226 INFO:journalctl@ceph.mon.a.smithi003.stdout: s = self.context.wrap_socket( 2024-08-17T17:01:54.226 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:01:54.226 INFO:journalctl@ceph.mon.a.smithi003.stdout: return self.sslsocket_class._create( 2024-08-17T17:01:54.227 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:01:54.227 INFO:journalctl@ceph.mon.a.smithi003.stdout: self.do_handshake() 2024-08-17T17:01:54.227 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:01:54.227 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._sslobj.do_handshake() 2024-08-17T17:01:54.227 INFO:journalctl@ceph.mon.a.smithi003.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:01:54.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:54 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:01:54.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:54 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:01:54.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:54 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:01:54.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:54 smithi003 ceph-mon[30803]: mgrmap e12: y(active, since 2s) 2024-08-17T17:01:55.107 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:55 smithi003 ceph-mon[30803]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:01:56.181 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$AHch2itswdPYdfn8NzPmleMCkmoJsR0mGF07LhX5e3RONJHKQFvGu", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1723914115, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-08-17T17:01:56.181 INFO:teuthology.orchestra.run.smithi003.stdout:Fetching dashboard port number... 2024-08-17T17:01:56.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:56 smithi003 ceph-mon[30803]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:01:56.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:56 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:01:57.346 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 8443 2024-08-17T17:01:57.356 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-08-17T17:01:57.357 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout disabled 2024-08-17T17:01:57.364 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-08-17T17:01:57.365 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2024-08-17T17:01:57.365 INFO:teuthology.orchestra.run.smithi003.stdout:firewalld.service is not enabled 2024-08-17T17:01:57.365 INFO:teuthology.orchestra.run.smithi003.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-08-17T17:01:57.367 INFO:teuthology.orchestra.run.smithi003.stdout:Ceph Dashboard is now available at: 2024-08-17T17:01:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:01:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: URL: https://smithi003.front.sepia.ceph.com:8443/ 2024-08-17T17:01:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: User: admin 2024-08-17T17:01:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: Password: lgz7eh580g 2024-08-17T17:01:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:01:57.368 INFO:teuthology.orchestra.run.smithi003.stdout:Saving cluster configuration to /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config directory 2024-08-17T17:01:57.368 INFO:teuthology.orchestra.run.smithi003.stdout:Enabling autotune for osd_memory_target 2024-08-17T17:01:57.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:57 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/2074025117' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-08-17T17:01:58.848 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:01:58 smithi003 ceph-mon[30803]: mgrmap e13: y(active, since 7s) 2024-08-17T17:01:59.704 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-08-17T17:01:59.704 INFO:teuthology.orchestra.run.smithi003.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-08-17T17:01:59.704 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:01:59.704 INFO:teuthology.orchestra.run.smithi003.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-08-17T17:01:59.704 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:01:59.704 INFO:teuthology.orchestra.run.smithi003.stdout:Or, if you are only running a single cluster on this host: 2024-08-17T17:01:59.704 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:01:59.705 INFO:teuthology.orchestra.run.smithi003.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-08-17T17:01:59.705 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:01:59.705 INFO:teuthology.orchestra.run.smithi003.stdout:Please consider enabling telemetry to help improve Ceph: 2024-08-17T17:01:59.705 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:01:59.705 INFO:teuthology.orchestra.run.smithi003.stdout: ceph telemetry on 2024-08-17T17:01:59.705 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:01:59.705 INFO:teuthology.orchestra.run.smithi003.stdout:For more information see: 2024-08-17T17:01:59.705 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:01:59.705 INFO:teuthology.orchestra.run.smithi003.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-08-17T17:01:59.705 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:01:59.705 INFO:teuthology.orchestra.run.smithi003.stdout:Bootstrap complete. 2024-08-17T17:01:59.743 INFO:tasks.cephadm:Fetching config... 2024-08-17T17:01:59.744 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-08-17T17:01:59.744 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-08-17T17:01:59.761 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-08-17T17:01:59.761 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-08-17T17:01:59.761 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-08-17T17:01:59.817 INFO:tasks.cephadm:Fetching mon keyring... 2024-08-17T17:01:59.817 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-08-17T17:01:59.817 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/keyring of=/dev/stdout 2024-08-17T17:01:59.883 INFO:tasks.cephadm:Fetching pub ssh key... 2024-08-17T17:01:59.883 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-08-17T17:01:59.883 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-08-17T17:01:59.938 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-08-17T17:01:59.939 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgMPqUE5x2Yw96xO+OHChQNFbNwLaBlK0DP8ZQAKqIEi87/zkiKpUuMN8RJ0DN0PxNm8qx0t4Oa673UqGHuSVQYSdlPTNHns4JITWcdgAPqjQx9H+RjytzS53h/P5VapOgaL1azWlv+xv1KwRGGKgi1kkhgA+6CFVZ0Ox4GGPX6O1rCFbg4mfoEs7e1Rg39/D1FXv5BUE9WB5Y50bV30CYaRWJAosp4b/RCXfnl8cZdKmkparsEuGf1sV1nGVeXi+pH+B8s5axXdQwD8O4HXPruNyw+AxdFns5kMrw0eve2FlwxCVpDCJBpFmX9lFHOonZOPf35eepWXJHQj5Flrg5jHdKOYloYoUoaC5nMW8WzfAPCjfAnOOEelVFkFGpn+t5bLIF3yP/YL6QN0LJHc7FhsDEScwYDLGFCNjsGh9wF67eHdUi4HZOhZLWXCN57BY2DsWS/9FMUWtfDLENPfUvwbDrmvHMNUiMxZPPBsMMs0xllGoBDDQs3yOc4GvnCCM= ceph-2b5b662e-5cba-11ef-bccf-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-08-17T17:02:00.033 INFO:teuthology.orchestra.run.smithi003.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgMPqUE5x2Yw96xO+OHChQNFbNwLaBlK0DP8ZQAKqIEi87/zkiKpUuMN8RJ0DN0PxNm8qx0t4Oa673UqGHuSVQYSdlPTNHns4JITWcdgAPqjQx9H+RjytzS53h/P5VapOgaL1azWlv+xv1KwRGGKgi1kkhgA+6CFVZ0Ox4GGPX6O1rCFbg4mfoEs7e1Rg39/D1FXv5BUE9WB5Y50bV30CYaRWJAosp4b/RCXfnl8cZdKmkparsEuGf1sV1nGVeXi+pH+B8s5axXdQwD8O4HXPruNyw+AxdFns5kMrw0eve2FlwxCVpDCJBpFmX9lFHOonZOPf35eepWXJHQj5Flrg5jHdKOYloYoUoaC5nMW8WzfAPCjfAnOOEelVFkFGpn+t5bLIF3yP/YL6QN0LJHc7FhsDEScwYDLGFCNjsGh9wF67eHdUi4HZOhZLWXCN57BY2DsWS/9FMUWtfDLENPfUvwbDrmvHMNUiMxZPPBsMMs0xllGoBDDQs3yOc4GvnCCM= ceph-2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:02:00.044 DEBUG:teuthology.orchestra.run.smithi053:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgMPqUE5x2Yw96xO+OHChQNFbNwLaBlK0DP8ZQAKqIEi87/zkiKpUuMN8RJ0DN0PxNm8qx0t4Oa673UqGHuSVQYSdlPTNHns4JITWcdgAPqjQx9H+RjytzS53h/P5VapOgaL1azWlv+xv1KwRGGKgi1kkhgA+6CFVZ0Ox4GGPX6O1rCFbg4mfoEs7e1Rg39/D1FXv5BUE9WB5Y50bV30CYaRWJAosp4b/RCXfnl8cZdKmkparsEuGf1sV1nGVeXi+pH+B8s5axXdQwD8O4HXPruNyw+AxdFns5kMrw0eve2FlwxCVpDCJBpFmX9lFHOonZOPf35eepWXJHQj5Flrg5jHdKOYloYoUoaC5nMW8WzfAPCjfAnOOEelVFkFGpn+t5bLIF3yP/YL6QN0LJHc7FhsDEScwYDLGFCNjsGh9wF67eHdUi4HZOhZLWXCN57BY2DsWS/9FMUWtfDLENPfUvwbDrmvHMNUiMxZPPBsMMs0xllGoBDDQs3yOc4GvnCCM= ceph-2b5b662e-5cba-11ef-bccf-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-08-17T17:02:00.087 INFO:teuthology.orchestra.run.smithi053.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgMPqUE5x2Yw96xO+OHChQNFbNwLaBlK0DP8ZQAKqIEi87/zkiKpUuMN8RJ0DN0PxNm8qx0t4Oa673UqGHuSVQYSdlPTNHns4JITWcdgAPqjQx9H+RjytzS53h/P5VapOgaL1azWlv+xv1KwRGGKgi1kkhgA+6CFVZ0Ox4GGPX6O1rCFbg4mfoEs7e1Rg39/D1FXv5BUE9WB5Y50bV30CYaRWJAosp4b/RCXfnl8cZdKmkparsEuGf1sV1nGVeXi+pH+B8s5axXdQwD8O4HXPruNyw+AxdFns5kMrw0eve2FlwxCVpDCJBpFmX9lFHOonZOPf35eepWXJHQj5Flrg5jHdKOYloYoUoaC5nMW8WzfAPCjfAnOOEelVFkFGpn+t5bLIF3yP/YL6QN0LJHc7FhsDEScwYDLGFCNjsGh9wF67eHdUi4HZOhZLWXCN57BY2DsWS/9FMUWtfDLENPfUvwbDrmvHMNUiMxZPPBsMMs0xllGoBDDQs3yOc4GvnCCM= ceph-2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:02:00.101 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-08-17T17:02:00.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:00 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/1601522810' entity='client.admin' 2024-08-17T17:02:00.367 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:02:03.171 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-08-17T17:02:03.172 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-08-17T17:02:03.406 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:03 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/591373623' entity='client.admin' 2024-08-17T17:02:03.406 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:03 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:03.406 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:03 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:03.406 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:03 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi003", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:02:03.406 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:03 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:03.406 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:03 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:02:03.407 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:03 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:03.650 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:02:06.579 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi053 2024-08-17T17:02:06.579 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-08-17T17:02:06.579 DEBUG:teuthology.orchestra.run.smithi053:> dd of=/etc/ceph/ceph.conf 2024-08-17T17:02:06.601 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-08-17T17:02:06.601 DEBUG:teuthology.orchestra.run.smithi053:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:02:06.660 INFO:tasks.cephadm:Adding host smithi053 to orchestrator... 2024-08-17T17:02:06.660 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph orch host add smithi053 2024-08-17T17:02:06.925 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:02:07.122 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:06 smithi003 ceph-mon[30803]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:02:07.122 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:06 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:07.122 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:06 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:02:07.122 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:06 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:07.122 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:06 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:02:07.122 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:06 smithi003 ceph-mon[30803]: Updating smithi003:/etc/ceph/ceph.conf 2024-08-17T17:02:07.123 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:06 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:07.123 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:06 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:07.123 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:06 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:08.250 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:07 smithi003 ceph-mon[30803]: Updating smithi003:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:02:08.250 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:07 smithi003 ceph-mon[30803]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:02:08.250 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:07 smithi003 ceph-mon[30803]: Updating smithi003:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:02:09.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:09 smithi003 ceph-mon[30803]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi053", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:02:10.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:10 smithi003 ceph-mon[30803]: Deploying cephadm binary to smithi053 2024-08-17T17:02:12.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:11 smithi003 ceph-mon[30803]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:12.229 INFO:teuthology.orchestra.run.smithi003.stdout:Added host 'smithi053' with addr '172.21.15.53' 2024-08-17T17:02:12.852 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph orch host ls --format=json 2024-08-17T17:02:13.116 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:02:13.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:13 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:13.452 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:13 smithi003 ceph-mon[30803]: Added host smithi053 2024-08-17T17:02:13.452 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:13 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:02:13.452 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:13 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:13.452 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:13 smithi003 ceph-mon[30803]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:13.452 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:13 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:14.796 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:02:14.796 INFO:teuthology.orchestra.run.smithi003.stdout:[{"addr": "172.21.15.3", "hostname": "smithi003", "labels": [], "status": ""}, {"addr": "172.21.15.53", "hostname": "smithi053", "labels": [], "status": ""}] 2024-08-17T17:02:15.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:14 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:15.379 INFO:tasks.cephadm:Setting crush tunables to default 2024-08-17T17:02:15.379 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd crush tunables default 2024-08-17T17:02:15.636 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:02:15.759 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:15 smithi003 ceph-mon[30803]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:15.759 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:15 smithi003 ceph-mon[30803]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:02:17.435 INFO:teuthology.orchestra.run.smithi003.stderr:adjusted tunables profile to default 2024-08-17T17:02:17.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:17 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:17.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:17 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:17.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:17 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:17.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:17 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:17.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:17 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi053", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:02:17.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:17 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:17.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:17 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:02:17.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:17 smithi003 ceph-mon[30803]: Updating smithi053:/etc/ceph/ceph.conf 2024-08-17T17:02:17.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:17 smithi003 ceph-mon[30803]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:17.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:17 smithi003 ceph-mon[30803]: Updating smithi053:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:02:17.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:17 smithi003 ceph-mon[30803]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:02:17.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:17 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/3598863111' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-08-17T17:02:17.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:17 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:17.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:17 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:17.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:17 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:17.982 INFO:tasks.cephadm:Adding mon.a on smithi003 2024-08-17T17:02:17.983 INFO:tasks.cephadm:Adding mon.c on smithi003 2024-08-17T17:02:17.983 INFO:tasks.cephadm:Adding mon.b on smithi053 2024-08-17T17:02:17.983 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph orch apply mon '3;smithi003:172.21.15.3=a;smithi003:[v2:172.21.15.3:3301,v1:172.21.15.3:6790]=c;smithi053:172.21.15.53=b' 2024-08-17T17:02:18.238 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:02:18.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:18 smithi003 ceph-mon[30803]: Updating smithi053:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:02:18.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:18 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/3598863111' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-08-17T17:02:18.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:18 smithi003 ceph-mon[30803]: osdmap e4: 0 total, 0 up, 0 in 2024-08-17T17:02:19.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:19 smithi003 ceph-mon[30803]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:19.886 INFO:teuthology.orchestra.run.smithi053.stdout:Scheduled mon update... 2024-08-17T17:02:20.446 DEBUG:teuthology.orchestra.run.smithi003:mon.c> sudo journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mon.c.service 2024-08-17T17:02:20.450 DEBUG:teuthology.orchestra.run.smithi053:mon.b> sudo journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mon.b.service 2024-08-17T17:02:20.452 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-08-17T17:02:20.452 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph mon dump -f json 2024-08-17T17:02:21.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:20 smithi003 ceph-mon[30803]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi003:172.21.15.3=a;smithi003:[v2:172.21.15.3:3301,v1:172.21.15.3:6790]=c;smithi053:172.21.15.53=b", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:02:21.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:20 smithi003 ceph-mon[30803]: Saving service mon spec with placement smithi003:172.21.15.3=a;smithi003:[v2:172.21.15.3:3301,v1:172.21.15.3:6790]=c;smithi053:172.21.15.53=b;count:3 2024-08-17T17:02:21.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:20 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:21.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:20 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:02:21.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:20 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:21.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:20 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:02:21.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:20 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:21.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:20 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:02:21.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:20 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:21.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:20 smithi003 ceph-mon[30803]: Deploying daemon mon.b on smithi053 2024-08-17T17:02:21.643 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:02:22.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:21 smithi003 ceph-mon[30803]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:23.810 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:23 smithi053 systemd[1]: Starting Ceph mon.b for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:02:24.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:23 smithi003 ceph-mon[30803]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:24.339 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 podman[34667]: 2024-08-17 17:02:24.229001222 +0000 UTC m=+0.104285911 container create 471447c55e315b6fe8a114c5903a18b0ac37a7f6222b05c5c85d77981c6c8b54 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-b, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=quincy-f2c39d4, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-08-17T17:02:24.339 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 podman[34667]: 2024-08-17 17:02:24.142479979 +0000 UTC m=+0.017764670 image pull f69471c56189b063ba7a049adea817a12ba1a40187d11cab5cbef8f3aac57cce quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251 2024-08-17T17:02:24.343 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T17:02:24.343 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":1,"fsid":"2b5b662e-5cba-11ef-bccf-c7b262605968","modified":"2024-08-17T17:00:53.391169Z","created":"2024-08-17T17:00:53.391169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-08-17T17:02:24.344 INFO:teuthology.orchestra.run.smithi053.stderr:dumped monmap epoch 1 2024-08-17T17:02:24.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 podman[34667]: 2024-08-17 17:02:24.464990092 +0000 UTC m=+0.340274790 container init 471447c55e315b6fe8a114c5903a18b0ac37a7f6222b05c5c85d77981c6c8b54 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-b, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=quincy-f2c39d4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-08-17T17:02:24.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 podman[34667]: 2024-08-17 17:02:24.468782216 +0000 UTC m=+0.344066908 container start 471447c55e315b6fe8a114c5903a18b0ac37a7f6222b05c5c85d77981c6c8b54 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, ceph=True, RELEASE=quincy-f2c39d4, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD) 2024-08-17T17:02:24.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: set uid:gid to 167:167 (ceph:ceph) 2024-08-17T17:02:24.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable), process ceph-mon, pid 2 2024-08-17T17:02:24.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: pidfile_write: ignore empty --pid-file 2024-08-17T17:02:24.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: load: jerasure load: lrc 2024-08-17T17:02:24.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: RocksDB version: 6.15.5 2024-08-17T17:02:24.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-08-17T17:02:24.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Compile date Aug 17 2024 2024-08-17T17:02:24.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: DB SUMMARY 2024-08-17T17:02:24.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: DB Session ID: VUG65SVX4VEEAW2SN6Z1 2024-08-17T17:02:24.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: CURRENT file: CURRENT 2024-08-17T17:02:24.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: IDENTITY file: IDENTITY 2024-08-17T17:02:24.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-08-17T17:02:24.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-08-17T17:02:24.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-08-17T17:02:24.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.error_if_exists: 0 2024-08-17T17:02:24.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.create_if_missing: 0 2024-08-17T17:02:24.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.paranoid_checks: 1 2024-08-17T17:02:24.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-08-17T17:02:24.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.env: 0x55c6b1581480 2024-08-17T17:02:24.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.fs: Posix File System 2024-08-17T17:02:24.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.info_log: 0x55c6b2f8cf80 2024-08-17T17:02:24.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.max_file_opening_threads: 16 2024-08-17T17:02:24.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.statistics: (nil) 2024-08-17T17:02:24.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.use_fsync: 0 2024-08-17T17:02:24.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.max_log_file_size: 0 2024-08-17T17:02:24.721 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-08-17T17:02:24.721 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.log_file_time_to_roll: 0 2024-08-17T17:02:24.721 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.keep_log_file_num: 1000 2024-08-17T17:02:24.721 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.recycle_log_file_num: 0 2024-08-17T17:02:24.721 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.allow_fallocate: 1 2024-08-17T17:02:24.721 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.allow_mmap_reads: 0 2024-08-17T17:02:24.721 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.allow_mmap_writes: 0 2024-08-17T17:02:24.721 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.use_direct_reads: 0 2024-08-17T17:02:24.721 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-08-17T17:02:24.721 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.create_missing_column_families: 0 2024-08-17T17:02:24.722 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.db_log_dir: 2024-08-17T17:02:24.722 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2024-08-17T17:02:24.722 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.table_cache_numshardbits: 6 2024-08-17T17:02:24.722 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.WAL_ttl_seconds: 0 2024-08-17T17:02:24.722 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.WAL_size_limit_MB: 0 2024-08-17T17:02:24.722 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-08-17T17:02:24.722 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-08-17T17:02:24.722 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.is_fd_close_on_exec: 1 2024-08-17T17:02:24.722 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.advise_random_on_open: 1 2024-08-17T17:02:24.723 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.db_write_buffer_size: 0 2024-08-17T17:02:24.723 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.write_buffer_manager: 0x55c6b304a780 2024-08-17T17:02:24.723 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-08-17T17:02:24.723 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-08-17T17:02:24.723 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-08-17T17:02:24.723 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.use_adaptive_mutex: 0 2024-08-17T17:02:24.723 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.rate_limiter: (nil) 2024-08-17T17:02:24.723 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-08-17T17:02:24.723 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.wal_recovery_mode: 2 2024-08-17T17:02:24.723 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.enable_thread_tracking: 0 2024-08-17T17:02:24.723 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.enable_pipelined_write: 0 2024-08-17T17:02:24.723 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.unordered_write: 0 2024-08-17T17:02:24.724 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-08-17T17:02:24.724 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-08-17T17:02:24.724 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-08-17T17:02:24.724 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-08-17T17:02:24.724 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.row_cache: None 2024-08-17T17:02:24.724 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.wal_filter: None 2024-08-17T17:02:24.724 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-08-17T17:02:24.724 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.allow_ingest_behind: 0 2024-08-17T17:02:24.724 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.preserve_deletes: 0 2024-08-17T17:02:24.724 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.two_write_queues: 0 2024-08-17T17:02:24.724 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.manual_wal_flush: 0 2024-08-17T17:02:24.725 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.atomic_flush: 0 2024-08-17T17:02:24.725 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-08-17T17:02:24.725 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.persist_stats_to_disk: 0 2024-08-17T17:02:24.725 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.write_dbid_to_manifest: 0 2024-08-17T17:02:24.725 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.log_readahead_size: 0 2024-08-17T17:02:24.725 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-08-17T17:02:24.725 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.best_efforts_recovery: 0 2024-08-17T17:02:24.725 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-08-17T17:02:24.725 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-08-17T17:02:24.725 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.allow_data_in_errors: 0 2024-08-17T17:02:24.725 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.db_host_id: __hostname__ 2024-08-17T17:02:24.726 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.max_background_jobs: 2 2024-08-17T17:02:24.726 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.max_background_compactions: -1 2024-08-17T17:02:24.726 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.max_subcompactions: 1 2024-08-17T17:02:24.726 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-08-17T17:02:24.726 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-08-17T17:02:24.726 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.delayed_write_rate : 16777216 2024-08-17T17:02:24.726 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.max_total_wal_size: 0 2024-08-17T17:02:24.726 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-08-17T17:02:24.727 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.stats_dump_period_sec: 600 2024-08-17T17:02:24.727 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.stats_persist_period_sec: 600 2024-08-17T17:02:24.727 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-08-17T17:02:24.727 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.max_open_files: -1 2024-08-17T17:02:24.727 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.bytes_per_sync: 0 2024-08-17T17:02:24.727 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.wal_bytes_per_sync: 0 2024-08-17T17:02:24.727 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.strict_bytes_per_sync: 0 2024-08-17T17:02:24.727 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.compaction_readahead_size: 0 2024-08-17T17:02:24.728 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.max_background_flushes: -1 2024-08-17T17:02:24.728 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Compression algorithms supported: 2024-08-17T17:02:24.728 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: kZSTD supported: 0 2024-08-17T17:02:24.728 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: kXpressCompression supported: 0 2024-08-17T17:02:24.728 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: kBZip2Compression supported: 0 2024-08-17T17:02:24.729 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-08-17T17:02:24.729 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: kLZ4Compression supported: 1 2024-08-17T17:02:24.729 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: kZlibCompression supported: 1 2024-08-17T17:02:24.729 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: kLZ4HCCompression supported: 1 2024-08-17T17:02:24.729 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: kSnappyCompression supported: 1 2024-08-17T17:02:24.729 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Fast CRC32 supported: Supported on x86 2024-08-17T17:02:24.729 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2024-08-17T17:02:24.730 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-08-17T17:02:24.730 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-08-17T17:02:24.730 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.merge_operator: 2024-08-17T17:02:24.730 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.compaction_filter: None 2024-08-17T17:02:24.730 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.compaction_filter_factory: None 2024-08-17T17:02:24.730 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.sst_partitioner_factory: None 2024-08-17T17:02:24.731 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.memtable_factory: SkipListFactory 2024-08-17T17:02:24.731 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.table_factory: BlockBasedTable 2024-08-17T17:02:24.731 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c6b2f33a78) 2024-08-17T17:02:24.731 INFO:journalctl@ceph.mon.b.smithi053.stdout: cache_index_and_filter_blocks: 1 2024-08-17T17:02:24.731 INFO:journalctl@ceph.mon.b.smithi053.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-08-17T17:02:24.731 INFO:journalctl@ceph.mon.b.smithi053.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-08-17T17:02:24.731 INFO:journalctl@ceph.mon.b.smithi053.stdout: pin_top_level_index_and_filter: 1 2024-08-17T17:02:24.731 INFO:journalctl@ceph.mon.b.smithi053.stdout: index_type: 0 2024-08-17T17:02:24.731 INFO:journalctl@ceph.mon.b.smithi053.stdout: data_block_index_type: 0 2024-08-17T17:02:24.732 INFO:journalctl@ceph.mon.b.smithi053.stdout: index_shortening: 1 2024-08-17T17:02:24.732 INFO:journalctl@ceph.mon.b.smithi053.stdout: data_block_hash_table_util_ratio: 0.750000 2024-08-17T17:02:24.732 INFO:journalctl@ceph.mon.b.smithi053.stdout: hash_index_allow_collision: 1 2024-08-17T17:02:24.732 INFO:journalctl@ceph.mon.b.smithi053.stdout: checksum: 1 2024-08-17T17:02:24.732 INFO:journalctl@ceph.mon.b.smithi053.stdout: no_block_cache: 0 2024-08-17T17:02:24.732 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_cache: 0x55c6b2f691f0 2024-08-17T17:02:24.732 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_cache_name: BinnedLRUCache 2024-08-17T17:02:24.732 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_cache_options: 2024-08-17T17:02:24.732 INFO:journalctl@ceph.mon.b.smithi053.stdout: capacity : 536870912 2024-08-17T17:02:24.732 INFO:journalctl@ceph.mon.b.smithi053.stdout: num_shard_bits : 4 2024-08-17T17:02:24.732 INFO:journalctl@ceph.mon.b.smithi053.stdout: strict_capacity_limit : 0 2024-08-17T17:02:24.733 INFO:journalctl@ceph.mon.b.smithi053.stdout: high_pri_pool_ratio: 0.000 2024-08-17T17:02:24.733 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_cache_compressed: (nil) 2024-08-17T17:02:24.733 INFO:journalctl@ceph.mon.b.smithi053.stdout: persistent_cache: (nil) 2024-08-17T17:02:24.733 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_size: 4096 2024-08-17T17:02:24.733 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_size_deviation: 10 2024-08-17T17:02:24.733 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_restart_interval: 16 2024-08-17T17:02:24.733 INFO:journalctl@ceph.mon.b.smithi053.stdout: index_block_restart_interval: 1 2024-08-17T17:02:24.733 INFO:journalctl@ceph.mon.b.smithi053.stdout: metadata_block_size: 4096 2024-08-17T17:02:24.733 INFO:journalctl@ceph.mon.b.smithi053.stdout: partition_filters: 0 2024-08-17T17:02:24.733 INFO:journalctl@ceph.mon.b.smithi053.stdout: use_delta_encoding: 1 2024-08-17T17:02:24.734 INFO:journalctl@ceph.mon.b.smithi053.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-08-17T17:02:24.734 INFO:journalctl@ceph.mon.b.smithi053.stdout: whole_key_filtering: 1 2024-08-17T17:02:24.734 INFO:journalctl@ceph.mon.b.smithi053.stdout: verify_compression: 0 2024-08-17T17:02:24.734 INFO:journalctl@ceph.mon.b.smithi053.stdout: read_amp_bytes_per_bit: 0 2024-08-17T17:02:24.734 INFO:journalctl@ceph.mon.b.smithi053.stdout: format_version: 4 2024-08-17T17:02:24.734 INFO:journalctl@ceph.mon.b.smithi053.stdout: enable_index_compression: 1 2024-08-17T17:02:24.734 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_align: 0 2024-08-17T17:02:24.734 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.write_buffer_size: 33554432 2024-08-17T17:02:24.734 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.max_write_buffer_number: 2 2024-08-17T17:02:24.734 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.compression: NoCompression 2024-08-17T17:02:24.735 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.bottommost_compression: Disabled 2024-08-17T17:02:24.735 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.prefix_extractor: nullptr 2024-08-17T17:02:24.735 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-08-17T17:02:24.735 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.num_levels: 7 2024-08-17T17:02:24.735 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-08-17T17:02:24.735 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-08-17T17:02:24.735 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-08-17T17:02:24.735 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-08-17T17:02:24.735 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-08-17T17:02:24.736 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-08-17T17:02:24.736 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-08-17T17:02:24.736 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-08-17T17:02:24.736 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-08-17T17:02:24.736 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-08-17T17:02:24.736 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.compression_opts.window_bits: -14 2024-08-17T17:02:24.736 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.compression_opts.level: 32767 2024-08-17T17:02:24.736 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.compression_opts.strategy: 0 2024-08-17T17:02:24.736 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-08-17T17:02:24.736 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-08-17T17:02:24.736 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-08-17T17:02:24.736 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.compression_opts.enabled: false 2024-08-17T17:02:24.736 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-08-17T17:02:24.736 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-08-17T17:02:24.736 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-08-17T17:02:24.737 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.target_file_size_base: 67108864 2024-08-17T17:02:24.737 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.target_file_size_multiplier: 1 2024-08-17T17:02:24.737 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-08-17T17:02:24.737 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-08-17T17:02:24.737 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-08-17T17:02:24.737 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-08-17T17:02:24.737 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-08-17T17:02:24.737 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-08-17T17:02:24.737 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-08-17T17:02:24.737 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-08-17T17:02:24.737 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-08-17T17:02:24.737 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-08-17T17:02:24.737 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-08-17T17:02:24.737 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-08-17T17:02:24.738 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.arena_block_size: 4194304 2024-08-17T17:02:24.738 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-08-17T17:02:24.738 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-08-17T17:02:24.738 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-08-17T17:02:24.738 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.disable_auto_compactions: 0 2024-08-17T17:02:24.738 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-08-17T17:02:24.738 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-08-17T17:02:24.738 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-08-17T17:02:24.738 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-08-17T17:02:24.738 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-08-17T17:02:24.739 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-08-17T17:02:24.739 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-08-17T17:02:24.739 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-08-17T17:02:24.739 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-08-17T17:02:24.739 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-08-17T17:02:24.739 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-08-17T17:02:24.739 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.inplace_update_support: 0 2024-08-17T17:02:24.739 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.inplace_update_num_locks: 10000 2024-08-17T17:02:24.739 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-08-17T17:02:24.739 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-08-17T17:02:24.739 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.memtable_huge_page_size: 0 2024-08-17T17:02:24.740 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.bloom_locality: 0 2024-08-17T17:02:24.740 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.max_successive_merges: 0 2024-08-17T17:02:24.740 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.optimize_filters_for_hits: 0 2024-08-17T17:02:24.740 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.paranoid_file_checks: 0 2024-08-17T17:02:24.740 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.force_consistency_checks: 1 2024-08-17T17:02:24.740 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.report_bg_io_stats: 0 2024-08-17T17:02:24.740 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.ttl: 2592000 2024-08-17T17:02:24.740 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.periodic_compaction_seconds: 0 2024-08-17T17:02:24.740 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.enable_blob_files: false 2024-08-17T17:02:24.740 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.min_blob_size: 0 2024-08-17T17:02:24.740 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.blob_file_size: 268435456 2024-08-17T17:02:24.740 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.blob_compression_type: NoCompression 2024-08-17T17:02:24.741 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.enable_blob_garbage_collection: false 2024-08-17T17:02:24.741 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-08-17T17:02:24.741 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-08-17T17:02:24.741 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2024-08-17T17:02:24.741 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2024-08-17T17:02:24.741 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723914144492838, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-08-17T17:02:24.741 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2024-08-17T17:02:24.741 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-08-17T17:02:24.741 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723914144493177, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1571, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1723914144, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "8e09e9cd-2065-4c43-aa34-5ac2c8ff7843", "db_session_id": "VUG65SVX4VEEAW2SN6Z1"}} 2024-08-17T17:02:24.741 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2024-08-17T17:02:24.741 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723914144493537, "job": 1, "event": "recovery_finished"} 2024-08-17T17:02:24.742 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-08-17T17:02:24.742 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55c6b2f9c700 2024-08-17T17:02:24.742 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: DB pointer 0x55c6b2f6c000 2024-08-17T17:02:24.742 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-08-17T17:02:24.742 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-08-17T17:02:24.742 INFO:journalctl@ceph.mon.b.smithi053.stdout: ** DB Stats ** 2024-08-17T17:02:24.742 INFO:journalctl@ceph.mon.b.smithi053.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-17T17:02:24.743 INFO:journalctl@ceph.mon.b.smithi053.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-08-17T17:02:24.743 INFO:journalctl@ceph.mon.b.smithi053.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-17T17:02:24.743 INFO:journalctl@ceph.mon.b.smithi053.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-17T17:02:24.743 INFO:journalctl@ceph.mon.b.smithi053.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-08-17T17:02:24.743 INFO:journalctl@ceph.mon.b.smithi053.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-08-17T17:02:24.743 INFO:journalctl@ceph.mon.b.smithi053.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-17T17:02:24.743 INFO:journalctl@ceph.mon.b.smithi053.stdout: 2024-08-17T17:02:24.743 INFO:journalctl@ceph.mon.b.smithi053.stdout: ** Compaction Stats [default] ** 2024-08-17T17:02:24.743 INFO:journalctl@ceph.mon.b.smithi053.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-08-17T17:02:24.744 INFO:journalctl@ceph.mon.b.smithi053.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-08-17T17:02:24.744 INFO:journalctl@ceph.mon.b.smithi053.stdout: L0 1/0 1.53 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 2024-08-17T17:02:24.744 INFO:journalctl@ceph.mon.b.smithi053.stdout: Sum 1/0 1.53 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 2024-08-17T17:02:24.744 INFO:journalctl@ceph.mon.b.smithi053.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 2024-08-17T17:02:24.744 INFO:journalctl@ceph.mon.b.smithi053.stdout: 2024-08-17T17:02:24.744 INFO:journalctl@ceph.mon.b.smithi053.stdout: ** Compaction Stats [default] ** 2024-08-17T17:02:24.744 INFO:journalctl@ceph.mon.b.smithi053.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-08-17T17:02:24.744 INFO:journalctl@ceph.mon.b.smithi053.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-08-17T17:02:24.744 INFO:journalctl@ceph.mon.b.smithi053.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.6 0.00 0.00 1 0.000 0 0 2024-08-17T17:02:24.744 INFO:journalctl@ceph.mon.b.smithi053.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-17T17:02:24.744 INFO:journalctl@ceph.mon.b.smithi053.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-08-17T17:02:24.744 INFO:journalctl@ceph.mon.b.smithi053.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-08-17T17:02:24.745 INFO:journalctl@ceph.mon.b.smithi053.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-08-17T17:02:24.745 INFO:journalctl@ceph.mon.b.smithi053.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-08-17T17:02:24.745 INFO:journalctl@ceph.mon.b.smithi053.stdout: AddFile(Keys): cumulative 0, interval 0 2024-08-17T17:02:24.745 INFO:journalctl@ceph.mon.b.smithi053.stdout: Cumulative compaction: 0.00 GB write, 0.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-17T17:02:24.745 INFO:journalctl@ceph.mon.b.smithi053.stdout: Interval compaction: 0.00 GB write, 0.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-17T17:02:24.745 INFO:journalctl@ceph.mon.b.smithi053.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-08-17T17:02:24.745 INFO:journalctl@ceph.mon.b.smithi053.stdout: 2024-08-17T17:02:24.745 INFO:journalctl@ceph.mon.b.smithi053.stdout: ** File Read Latency Histogram By Level [default] ** 2024-08-17T17:02:24.745 INFO:journalctl@ceph.mon.b.smithi053.stdout: 2024-08-17T17:02:24.746 INFO:journalctl@ceph.mon.b.smithi053.stdout: ** Compaction Stats [default] ** 2024-08-17T17:02:24.746 INFO:journalctl@ceph.mon.b.smithi053.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-08-17T17:02:24.746 INFO:journalctl@ceph.mon.b.smithi053.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-08-17T17:02:24.746 INFO:journalctl@ceph.mon.b.smithi053.stdout: L0 1/0 1.53 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 2024-08-17T17:02:24.746 INFO:journalctl@ceph.mon.b.smithi053.stdout: Sum 1/0 1.53 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.6 0.00 0.00 1 0.000 0 0 2024-08-17T17:02:24.746 INFO:journalctl@ceph.mon.b.smithi053.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-08-17T17:02:24.746 INFO:journalctl@ceph.mon.b.smithi053.stdout: 2024-08-17T17:02:24.746 INFO:journalctl@ceph.mon.b.smithi053.stdout: ** Compaction Stats [default] ** 2024-08-17T17:02:24.746 INFO:journalctl@ceph.mon.b.smithi053.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-08-17T17:02:24.746 INFO:journalctl@ceph.mon.b.smithi053.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-08-17T17:02:24.746 INFO:journalctl@ceph.mon.b.smithi053.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.6 0.00 0.00 1 0.000 0 0 2024-08-17T17:02:24.746 INFO:journalctl@ceph.mon.b.smithi053.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-17T17:02:24.747 INFO:journalctl@ceph.mon.b.smithi053.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-08-17T17:02:24.747 INFO:journalctl@ceph.mon.b.smithi053.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-08-17T17:02:24.747 INFO:journalctl@ceph.mon.b.smithi053.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-08-17T17:02:24.747 INFO:journalctl@ceph.mon.b.smithi053.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-08-17T17:02:24.747 INFO:journalctl@ceph.mon.b.smithi053.stdout: AddFile(Keys): cumulative 0, interval 0 2024-08-17T17:02:24.747 INFO:journalctl@ceph.mon.b.smithi053.stdout: Cumulative compaction: 0.00 GB write, 0.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-17T17:02:24.747 INFO:journalctl@ceph.mon.b.smithi053.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-17T17:02:24.747 INFO:journalctl@ceph.mon.b.smithi053.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-08-17T17:02:24.747 INFO:journalctl@ceph.mon.b.smithi053.stdout: 2024-08-17T17:02:24.747 INFO:journalctl@ceph.mon.b.smithi053.stdout: ** File Read Latency Histogram By Level [default] ** 2024-08-17T17:02:24.747 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-08-17T17:02:24.747 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: using public_addr v2:172.21.15.53:0/0 -> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] 2024-08-17T17:02:24.747 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: starting mon.b rank -1 at public addrs [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] at bind addrs [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:02:24.747 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: mon.b@-1(???) e0 preinit fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:02:24.748 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: mon.b@-1(synchronizing).mds e1 new map 2024-08-17T17:02:24.748 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: mon.b@-1(synchronizing).mds e1 print_map 2024-08-17T17:02:24.748 INFO:journalctl@ceph.mon.b.smithi053.stdout: e1 2024-08-17T17:02:24.748 INFO:journalctl@ceph.mon.b.smithi053.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-08-17T17:02:24.748 INFO:journalctl@ceph.mon.b.smithi053.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-08-17T17:02:24.748 INFO:journalctl@ceph.mon.b.smithi053.stdout: legacy client fscid: -1 2024-08-17T17:02:24.748 INFO:journalctl@ceph.mon.b.smithi053.stdout: 2024-08-17T17:02:24.748 INFO:journalctl@ceph.mon.b.smithi053.stdout: No filesystems configured 2024-08-17T17:02:24.748 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-08-17T17:02:24.748 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-08-17T17:02:24.748 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-08-17T17:02:24.748 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-08-17T17:02:24.748 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-08-17T17:02:24.749 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-08-17T17:02:24.749 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-08-17T17:02:24.749 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-08-17T17:02:24.749 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-08-17T17:02:24.749 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-08-17T17:02:24.749 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: Updating smithi053:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:02:24.749 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/3598863111' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-08-17T17:02:24.749 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: osdmap e4: 0 total, 0 up, 0 in 2024-08-17T17:02:24.749 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:24.749 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi003:172.21.15.3=a;smithi003:[v2:172.21.15.3:3301,v1:172.21.15.3:6790]=c;smithi053:172.21.15.53=b", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:02:24.749 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: Saving service mon spec with placement smithi003:172.21.15.3=a;smithi003:[v2:172.21.15.3:3301,v1:172.21.15.3:6790]=c;smithi053:172.21.15.53=b;count:3 2024-08-17T17:02:24.749 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:24.750 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:02:24.750 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:24.750 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:02:24.750 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:24.750 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:02:24.750 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:24.751 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: Deploying daemon mon.b on smithi053 2024-08-17T17:02:24.751 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:24.751 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:24.751 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 ceph-mon[34699]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-08-17T17:02:24.751 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 bash[34667]: 471447c55e315b6fe8a114c5903a18b0ac37a7f6222b05c5c85d77981c6c8b54 2024-08-17T17:02:24.751 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:24 smithi053 systemd[1]: Started Ceph mon.b for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:02:26.162 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-08-17T17:02:26.163 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph mon dump -f json 2024-08-17T17:02:26.433 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:02:29.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:29 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:02:29.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:29 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:02:29.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:29 smithi053 ceph-mon[34699]: mon.a calling monitor election 2024-08-17T17:02:29.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:29 smithi053 ceph-mon[34699]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:29.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:29 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:02:29.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:29 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:02:29.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:29 smithi053 ceph-mon[34699]: mon.b calling monitor election 2024-08-17T17:02:29.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:29 smithi053 ceph-mon[34699]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:29.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:29 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:02:29.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:29 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:02:29.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:29 smithi053 ceph-mon[34699]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:29.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:29 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:02:29.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:29 smithi053 ceph-mon[34699]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-08-17T17:02:29.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:29 smithi053 ceph-mon[34699]: monmap e2: 2 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-08-17T17:02:29.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:29 smithi053 ceph-mon[34699]: fsmap 2024-08-17T17:02:29.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:29 smithi053 ceph-mon[34699]: osdmap e4: 0 total, 0 up, 0 in 2024-08-17T17:02:29.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:29 smithi053 ceph-mon[34699]: mgrmap e13: y(active, since 38s) 2024-08-17T17:02:29.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:29 smithi053 ceph-mon[34699]: overall HEALTH_OK 2024-08-17T17:02:29.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:29 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:29.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:29 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:29.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:29 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:29.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:29 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:02:29.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:29 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:29.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:29 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:02:29.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:29 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:02:29.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:29 smithi003 ceph-mon[30803]: mon.a calling monitor election 2024-08-17T17:02:29.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:29 smithi003 ceph-mon[30803]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:29.987 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:29 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:02:29.987 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:29 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:02:29.987 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:29 smithi003 ceph-mon[30803]: mon.b calling monitor election 2024-08-17T17:02:29.987 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:29 smithi003 ceph-mon[30803]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:29.987 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:29 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:02:29.987 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:29 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:02:29.987 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:29 smithi003 ceph-mon[30803]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:29.987 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:29 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:02:29.987 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:29 smithi003 ceph-mon[30803]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-08-17T17:02:29.987 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:29 smithi003 ceph-mon[30803]: monmap e2: 2 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-08-17T17:02:29.987 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:29 smithi003 ceph-mon[30803]: fsmap 2024-08-17T17:02:29.987 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:29 smithi003 ceph-mon[30803]: osdmap e4: 0 total, 0 up, 0 in 2024-08-17T17:02:29.987 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:29 smithi003 ceph-mon[30803]: mgrmap e13: y(active, since 38s) 2024-08-17T17:02:29.988 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:29 smithi003 ceph-mon[30803]: overall HEALTH_OK 2024-08-17T17:02:29.988 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:29 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:29.988 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:29 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:29.988 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:29 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:29.988 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:29 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:02:29.988 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:29 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:30.867 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:30 smithi053 ceph-mon[34699]: Deploying daemon mon.c on smithi003 2024-08-17T17:02:30.868 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:30 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:02:30.897 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:30 smithi003 ceph-mon[30803]: Deploying daemon mon.c on smithi003 2024-08-17T17:02:30.897 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:30 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:02:31.124 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T17:02:31.124 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":2,"fsid":"2b5b662e-5cba-11ef-bccf-c7b262605968","modified":"2024-08-17T17:02:24.515516Z","created":"2024-08-17T17:00:53.391169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-08-17T17:02:31.124 INFO:teuthology.orchestra.run.smithi053.stderr:dumped monmap epoch 2 2024-08-17T17:02:31.698 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:31 smithi053 ceph-mon[34699]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:31.699 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:31 smithi053 ceph-mon[34699]: from='client.? 172.21.15.53:0/873914816' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-17T17:02:31.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:31 smithi003 ceph-mon[30803]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:31.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:31 smithi003 ceph-mon[30803]: from='client.? 172.21.15.53:0/873914816' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-17T17:02:32.726 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-08-17T17:02:32.727 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph mon dump -f json 2024-08-17T17:02:32.998 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:02:33.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:32 smithi003 systemd[1]: Started Ceph mon.c for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:02:38.200 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T17:02:38.200 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":3,"fsid":"2b5b662e-5cba-11ef-bccf-c7b262605968","modified":"2024-08-17T17:02:32.825077Z","created":"2024-08-17T17:00:53.391169Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:3300","nonce":0},{"type":"v1","addr":"172.21.15.53:6789","nonce":0}]},"addr":"172.21.15.53:6789/0","public_addr":"172.21.15.53:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3301","nonce":0},{"type":"v1","addr":"172.21.15.3:6790","nonce":0}]},"addr":"172.21.15.3:6790/0","public_addr":"172.21.15.3:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-08-17T17:02:38.201 INFO:teuthology.orchestra.run.smithi053.stderr:dumped monmap epoch 3 2024-08-17T17:02:38.201 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:37 smithi053 ceph-mon[34699]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:38.201 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:37 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:02:38.201 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:37 smithi053 ceph-mon[34699]: mon.a calling monitor election 2024-08-17T17:02:38.201 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:37 smithi053 ceph-mon[34699]: mon.b calling monitor election 2024-08-17T17:02:38.201 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:37 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:02:38.201 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:37 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:02:38.201 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:37 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:02:38.202 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:37 smithi053 ceph-mon[34699]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:38.202 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:37 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:02:38.202 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:37 smithi053 ceph-mon[34699]: mon.c calling monitor election 2024-08-17T17:02:38.202 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:37 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:02:38.202 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:37 smithi053 ceph-mon[34699]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:38.202 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:37 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:02:38.202 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:37 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:02:38.202 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:37 smithi053 ceph-mon[34699]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-08-17T17:02:38.202 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:37 smithi053 ceph-mon[34699]: monmap e3: 3 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0],c=[v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-08-17T17:02:38.202 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:37 smithi053 ceph-mon[34699]: fsmap 2024-08-17T17:02:38.202 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:37 smithi053 ceph-mon[34699]: osdmap e4: 0 total, 0 up, 0 in 2024-08-17T17:02:38.202 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:37 smithi053 ceph-mon[34699]: mgrmap e13: y(active, since 47s) 2024-08-17T17:02:38.203 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:37 smithi053 ceph-mon[34699]: overall HEALTH_OK 2024-08-17T17:02:38.203 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:37 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:38.203 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:37 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:38.203 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:37 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:38.203 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:37 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:38.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[30803]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:38.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:02:38.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[30803]: mon.a calling monitor election 2024-08-17T17:02:38.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[30803]: mon.b calling monitor election 2024-08-17T17:02:38.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:02:38.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:02:38.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:02:38.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[30803]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:38.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:02:38.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[30803]: mon.c calling monitor election 2024-08-17T17:02:38.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:02:38.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[30803]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:38.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:02:38.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:02:38.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[30803]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-08-17T17:02:38.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[30803]: monmap e3: 3 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0],c=[v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-08-17T17:02:38.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[30803]: fsmap 2024-08-17T17:02:38.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[30803]: osdmap e4: 0 total, 0 up, 0 in 2024-08-17T17:02:38.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[30803]: mgrmap e13: y(active, since 47s) 2024-08-17T17:02:38.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[30803]: overall HEALTH_OK 2024-08-17T17:02:38.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:38.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:38.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:38.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:38.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[41530]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:38.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:02:38.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[41530]: mon.a calling monitor election 2024-08-17T17:02:38.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[41530]: mon.b calling monitor election 2024-08-17T17:02:38.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:02:38.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:02:38.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:02:38.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[41530]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:38.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:02:38.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[41530]: mon.c calling monitor election 2024-08-17T17:02:38.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:02:38.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[41530]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:38.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:02:38.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:02:38.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[41530]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-08-17T17:02:38.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[41530]: monmap e3: 3 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0],c=[v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-08-17T17:02:38.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[41530]: fsmap 2024-08-17T17:02:38.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[41530]: osdmap e4: 0 total, 0 up, 0 in 2024-08-17T17:02:38.277 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[41530]: mgrmap e13: y(active, since 47s) 2024-08-17T17:02:38.277 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[41530]: overall HEALTH_OK 2024-08-17T17:02:38.277 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:38.277 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:38.277 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:38.277 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:37 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:38.698 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-08-17T17:02:38.698 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph config generate-minimal-conf 2024-08-17T17:02:39.052 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:02:39.180 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:38 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:02:39.180 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:38 smithi003 ceph-mon[41530]: from='client.? 172.21.15.53:0/30433625' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-17T17:02:39.180 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:38 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:02:39.180 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:38 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:02:39.181 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:38 smithi003 ceph-mon[30803]: from='client.? 172.21.15.53:0/30433625' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-17T17:02:39.181 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:38 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:02:39.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:38 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:02:39.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:38 smithi053 ceph-mon[34699]: from='client.? 172.21.15.53:0/30433625' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-08-17T17:02:39.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:38 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:02:40.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:39 smithi053 ceph-mon[34699]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:40.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:39 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:40.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:39 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:40.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:39 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:40.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:39 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:40.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:39 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:40.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:39 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:02:40.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:39 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:40.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:39 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:40.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:39 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:40.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:39 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:40.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:39 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:40.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:39 smithi003 ceph-mon[30803]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:40.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:39 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:40.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:39 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:40.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:39 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:40.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:39 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:40.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:39 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:40.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:39 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:02:40.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:39 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:40.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:39 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:40.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:39 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:40.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:39 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:40.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:39 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:40.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:39 smithi003 ceph-mon[41530]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:40.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:39 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:40.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:39 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:40.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:39 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:40.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:39 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:40.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:39 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:40.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:39 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:02:40.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:39 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:40.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:39 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:40.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:39 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:40.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:39 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:40.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:39 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:40.920 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:40 smithi003 ceph-mon[30803]: Updating smithi003:/etc/ceph/ceph.conf 2024-08-17T17:02:40.920 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:40 smithi003 ceph-mon[30803]: Updating smithi053:/etc/ceph/ceph.conf 2024-08-17T17:02:40.920 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:40 smithi003 ceph-mon[30803]: Updating smithi053:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:02:40.920 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:40 smithi003 ceph-mon[30803]: Updating smithi003:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:02:40.920 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:40 smithi003 ceph-mon[30803]: Reconfiguring mon.a (unknown last config time)... 2024-08-17T17:02:40.920 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:40 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:02:40.920 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:40 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-17T17:02:40.920 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:40 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:40.920 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:40 smithi003 ceph-mon[30803]: Reconfiguring daemon mon.a on smithi003 2024-08-17T17:02:40.920 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:40 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:41.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:40 smithi053 ceph-mon[34699]: Updating smithi003:/etc/ceph/ceph.conf 2024-08-17T17:02:41.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:40 smithi053 ceph-mon[34699]: Updating smithi053:/etc/ceph/ceph.conf 2024-08-17T17:02:41.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:40 smithi053 ceph-mon[34699]: Updating smithi053:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:02:41.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:40 smithi053 ceph-mon[34699]: Updating smithi003:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:02:41.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:40 smithi053 ceph-mon[34699]: Reconfiguring mon.a (unknown last config time)... 2024-08-17T17:02:41.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:40 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:02:41.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:40 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-17T17:02:41.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:40 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:41.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:40 smithi053 ceph-mon[34699]: Reconfiguring daemon mon.a on smithi003 2024-08-17T17:02:41.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:40 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:41.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:40 smithi003 ceph-mon[41530]: Updating smithi003:/etc/ceph/ceph.conf 2024-08-17T17:02:41.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:40 smithi003 ceph-mon[41530]: Updating smithi053:/etc/ceph/ceph.conf 2024-08-17T17:02:41.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:40 smithi003 ceph-mon[41530]: Updating smithi053:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:02:41.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:40 smithi003 ceph-mon[41530]: Updating smithi003:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:02:41.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:40 smithi003 ceph-mon[41530]: Reconfiguring mon.a (unknown last config time)... 2024-08-17T17:02:41.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:40 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:02:41.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:40 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-17T17:02:41.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:40 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:41.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:40 smithi003 ceph-mon[41530]: Reconfiguring daemon mon.a on smithi003 2024-08-17T17:02:41.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:40 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:41.892 INFO:teuthology.orchestra.run.smithi003.stdout:# minimal ceph.conf for 2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:02:41.892 INFO:teuthology.orchestra.run.smithi003.stdout:[global] 2024-08-17T17:02:41.892 INFO:teuthology.orchestra.run.smithi003.stdout: fsid = 2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:02:41.892 INFO:teuthology.orchestra.run.smithi003.stdout: mon_host = [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2024-08-17T17:02:42.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:41 smithi053 ceph-mon[34699]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:42.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:41 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:42.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:41 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:42.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:41 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:02:42.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:41 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-17T17:02:42.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:41 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:42.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:41 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/3766519765' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:42.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:41 smithi003 ceph-mon[30803]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:42.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:41 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:42.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:41 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:42.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:41 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:02:42.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:41 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-17T17:02:42.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:41 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:42.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:41 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/3766519765' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:42.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:41 smithi003 ceph-mon[41530]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:42.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:41 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:42.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:41 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:42.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:41 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:02:42.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:41 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-17T17:02:42.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:41 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:42.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:41 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/3766519765' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:42.582 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-08-17T17:02:42.582 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-08-17T17:02:42.582 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/ceph/ceph.conf 2024-08-17T17:02:42.614 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-08-17T17:02:42.614 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:02:42.682 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-08-17T17:02:42.682 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/ceph/ceph.conf 2024-08-17T17:02:42.716 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-08-17T17:02:42.717 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:02:42.787 INFO:tasks.cephadm:Adding mgr.y on smithi003 2024-08-17T17:02:42.787 INFO:tasks.cephadm:Adding mgr.x on smithi053 2024-08-17T17:02:42.787 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph orch apply mgr '2;smithi003=y;smithi053=x' 2024-08-17T17:02:42.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:42 smithi053 ceph-mon[34699]: Reconfiguring mon.c (monmap changed)... 2024-08-17T17:02:42.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:42 smithi053 ceph-mon[34699]: Reconfiguring daemon mon.c on smithi003 2024-08-17T17:02:43.093 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:02:43.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:42 smithi003 ceph-mon[30803]: Reconfiguring mon.c (monmap changed)... 2024-08-17T17:02:43.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:42 smithi003 ceph-mon[30803]: Reconfiguring daemon mon.c on smithi003 2024-08-17T17:02:43.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:42 smithi003 ceph-mon[41530]: Reconfiguring mon.c (monmap changed)... 2024-08-17T17:02:43.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:42 smithi003 ceph-mon[41530]: Reconfiguring daemon mon.c on smithi003 2024-08-17T17:02:44.209 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:43 smithi053 ceph-mon[34699]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:44.209 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:43 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:44.209 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:43 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:44.209 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:43 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:02:44.209 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:43 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-17T17:02:44.209 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:43 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:44.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:43 smithi003 ceph-mon[30803]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:44.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:43 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:44.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:43 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:44.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:43 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:02:44.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:43 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-17T17:02:44.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:43 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:44.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:43 smithi003 ceph-mon[41530]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:44.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:43 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:44.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:43 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:44.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:43 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:02:44.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:43 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-17T17:02:44.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:43 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:45.209 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:44 smithi053 ceph-mon[34699]: Reconfiguring mon.b (monmap changed)... 2024-08-17T17:02:45.209 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:44 smithi053 ceph-mon[34699]: Reconfiguring daemon mon.b on smithi053 2024-08-17T17:02:45.213 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:44 smithi003 ceph-mon[30803]: Reconfiguring mon.b (monmap changed)... 2024-08-17T17:02:45.213 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:44 smithi003 ceph-mon[30803]: Reconfiguring daemon mon.b on smithi053 2024-08-17T17:02:45.213 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:44 smithi003 ceph-mon[41530]: Reconfiguring mon.b (monmap changed)... 2024-08-17T17:02:45.213 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:44 smithi003 ceph-mon[41530]: Reconfiguring daemon mon.b on smithi053 2024-08-17T17:02:45.835 INFO:teuthology.orchestra.run.smithi053.stdout:Scheduled mgr update... 2024-08-17T17:02:46.209 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:45 smithi053 ceph-mon[34699]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:46.209 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:45 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:46.209 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:45 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:46.209 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:45 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:02:46.210 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:45 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:46.210 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:45 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:02:46.210 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:45 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:46.210 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:45 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:46.210 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:45 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:02:46.210 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:45 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:46.210 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:45 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:02:46.210 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:45 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:46.210 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:45 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:02:46.210 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:45 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-08-17T17:02:46.210 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:45 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-17T17:02:46.210 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:45 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:46.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[41530]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:46.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:46.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:46.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:02:46.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:46.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:02:46.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:46.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:46.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:02:46.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:46.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:02:46.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:46.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:02:46.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-08-17T17:02:46.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-17T17:02:46.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:46.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[30803]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:46.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:46.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:46.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:02:46.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:46.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:02:46.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:46.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:46.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:02:46.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:46.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:02:46.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:46.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:02:46.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-08-17T17:02:46.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-17T17:02:46.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:45 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:46.383 DEBUG:teuthology.orchestra.run.smithi053:mgr.x> sudo journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mgr.x.service 2024-08-17T17:02:46.386 DEBUG:tasks.cephadm:set 0 configs 2024-08-17T17:02:46.386 INFO:tasks.cephadm:Deploying OSDs... 2024-08-17T17:02:46.386 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-08-17T17:02:46.386 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/scratch_devs of=/dev/stdout 2024-08-17T17:02:46.403 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-17T17:02:46.403 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_1 2024-08-17T17:02:46.458 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-17T17:02:46.458 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-17T17:02:46.459 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 972 Links: 1 2024-08-17T17:02:46.459 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-17T17:02:46.459 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-08-17T17:02:46.459 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-08-17 17:02:32.279318430 +0000 2024-08-17T17:02:46.459 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-08-17 16:58:05.168635667 +0000 2024-08-17T17:02:46.459 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-08-17 16:58:05.168635667 +0000 2024-08-17T17:02:46.459 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-08-17 16:58:05.168635667 +0000 2024-08-17T17:02:46.459 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-17T17:02:46.525 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-08-17T17:02:46.525 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-08-17T17:02:46.525 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000209776 s, 2.4 MB/s 2024-08-17T17:02:46.526 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-17T17:02:46.584 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_2 2024-08-17T17:02:46.639 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-17T17:02:46.639 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-17T17:02:46.639 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 979 Links: 1 2024-08-17T17:02:46.639 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-17T17:02:46.639 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-08-17T17:02:46.639 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-08-17 17:02:32.279318430 +0000 2024-08-17T17:02:46.639 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-08-17 16:58:05.168635667 +0000 2024-08-17T17:02:46.640 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-08-17 16:58:05.168635667 +0000 2024-08-17T17:02:46.640 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-08-17 16:58:05.168635667 +0000 2024-08-17T17:02:46.640 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-17T17:02:46.704 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-08-17T17:02:46.704 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-08-17T17:02:46.704 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000194249 s, 2.6 MB/s 2024-08-17T17:02:46.706 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-17T17:02:46.762 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_3 2024-08-17T17:02:46.818 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-17T17:02:46.819 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-17T17:02:46.819 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 973 Links: 1 2024-08-17T17:02:46.819 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-17T17:02:46.819 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-08-17T17:02:46.819 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-08-17 17:02:32.279318430 +0000 2024-08-17T17:02:46.820 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-08-17 16:58:05.168635667 +0000 2024-08-17T17:02:46.820 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-08-17 16:58:05.168635667 +0000 2024-08-17T17:02:46.820 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-08-17 16:58:05.168635667 +0000 2024-08-17T17:02:46.820 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-17T17:02:46.882 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-08-17T17:02:46.882 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-08-17T17:02:46.883 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000206121 s, 2.5 MB/s 2024-08-17T17:02:46.884 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-17T17:02:46.940 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_4 2024-08-17T17:02:46.997 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-17T17:02:46.997 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-17T17:02:46.997 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 971 Links: 1 2024-08-17T17:02:46.997 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-17T17:02:46.997 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-08-17T17:02:46.997 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-08-17 17:02:32.280318417 +0000 2024-08-17T17:02:46.997 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-08-17 16:58:05.168635667 +0000 2024-08-17T17:02:46.997 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-08-17 16:58:05.168635667 +0000 2024-08-17T17:02:46.997 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-08-17 16:58:05.168635667 +0000 2024-08-17T17:02:46.997 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-17T17:02:47.061 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-08-17T17:02:47.061 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-08-17T17:02:47.062 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000250619 s, 2.0 MB/s 2024-08-17T17:02:47.063 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-17T17:02:47.119 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-08-17T17:02:47.120 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/scratch_devs of=/dev/stdout 2024-08-17T17:02:47.138 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-17T17:02:47.138 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_1 2024-08-17T17:02:47.195 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-17T17:02:47.195 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-17T17:02:47.195 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 975 Links: 1 2024-08-17T17:02:47.195 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-17T17:02:47.195 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-08-17T17:02:47.195 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-08-17 17:02:23.631905126 +0000 2024-08-17T17:02:47.195 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-08-17 16:58:06.563739243 +0000 2024-08-17T17:02:47.196 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-08-17 16:58:06.563739243 +0000 2024-08-17T17:02:47.196 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-08-17 16:58:06.563739243 +0000 2024-08-17T17:02:47.196 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-17T17:02:47.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:46 smithi053 ceph-mon[34699]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi003=y;smithi053=x", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:02:47.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:46 smithi053 ceph-mon[34699]: Saving service mgr spec with placement smithi003=y;smithi053=x;count:2 2024-08-17T17:02:47.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:46 smithi053 ceph-mon[34699]: Deploying daemon mgr.x on smithi053 2024-08-17T17:02:47.241 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-08-17T17:02:47.242 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-08-17T17:02:47.242 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 9.423e-05 s, 5.4 MB/s 2024-08-17T17:02:47.243 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-17T17:02:47.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:46 smithi003 ceph-mon[30803]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi003=y;smithi053=x", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:02:47.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:46 smithi003 ceph-mon[30803]: Saving service mgr spec with placement smithi003=y;smithi053=x;count:2 2024-08-17T17:02:47.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:46 smithi003 ceph-mon[30803]: Deploying daemon mgr.x on smithi053 2024-08-17T17:02:47.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:46 smithi003 ceph-mon[41530]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi003=y;smithi053=x", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:02:47.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:46 smithi003 ceph-mon[41530]: Saving service mgr spec with placement smithi003=y;smithi053=x;count:2 2024-08-17T17:02:47.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:46 smithi003 ceph-mon[41530]: Deploying daemon mgr.x on smithi053 2024-08-17T17:02:47.300 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_2 2024-08-17T17:02:47.357 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-17T17:02:47.357 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-17T17:02:47.357 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 963 Links: 1 2024-08-17T17:02:47.357 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-17T17:02:47.357 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-08-17T17:02:47.357 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-08-17 17:02:23.632905108 +0000 2024-08-17T17:02:47.357 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-08-17 16:58:06.561739280 +0000 2024-08-17T17:02:47.357 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-08-17 16:58:06.561739280 +0000 2024-08-17T17:02:47.358 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-08-17 16:58:06.561739280 +0000 2024-08-17T17:02:47.358 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-17T17:02:47.422 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-08-17T17:02:47.422 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-08-17T17:02:47.422 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000124917 s, 4.1 MB/s 2024-08-17T17:02:47.423 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-17T17:02:47.479 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_3 2024-08-17T17:02:47.535 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-17T17:02:47.536 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-17T17:02:47.536 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 964 Links: 1 2024-08-17T17:02:47.536 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-17T17:02:47.536 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-08-17T17:02:47.536 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-08-17 17:02:23.632905108 +0000 2024-08-17T17:02:47.536 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-08-17 16:58:06.561739280 +0000 2024-08-17T17:02:47.536 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-08-17 16:58:06.561739280 +0000 2024-08-17T17:02:47.536 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-08-17 16:58:06.561739280 +0000 2024-08-17T17:02:47.536 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-17T17:02:47.736 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-08-17T17:02:47.736 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-08-17T17:02:47.736 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000114227 s, 4.5 MB/s 2024-08-17T17:02:47.737 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-17T17:02:47.754 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_4 2024-08-17T17:02:47.818 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-17T17:02:47.818 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-17T17:02:47.818 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 976 Links: 1 2024-08-17T17:02:47.818 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-17T17:02:47.818 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-08-17T17:02:47.818 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-08-17 17:02:47.687464146 +0000 2024-08-17T17:02:47.818 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-08-17 16:58:06.563739243 +0000 2024-08-17T17:02:47.818 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-08-17 16:58:06.563739243 +0000 2024-08-17T17:02:47.818 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-08-17 16:58:06.563739243 +0000 2024-08-17T17:02:47.819 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-17T17:02:47.922 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-08-17T17:02:47.922 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-08-17T17:02:47.922 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 9.7683e-05 s, 5.2 MB/s 2024-08-17T17:02:47.923 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-17T17:02:47.939 INFO:tasks.cephadm:Deploying osd.0 on smithi003 with /dev/vg_nvme/lv_4... 2024-08-17T17:02:47.940 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-08-17T17:02:48.164 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:47 smithi053 ceph-mon[34699]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:48.210 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:02:48.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:47 smithi003 ceph-mon[30803]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:48.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:47 smithi003 ceph-mon[41530]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:48.680 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:02:48 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:02:48.544+0000 7f5b4c277ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-17T17:02:48.681 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:02:48 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:02:48.679+0000 7f5b4c277ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-17T17:02:48.966 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:02:48 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:02:48.751+0000 7f5b4c277ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-17T17:02:49.322 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:02:49 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:02:49.028+0000 7f5b4c277ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-17T17:02:49.322 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:02:49 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:02:49.096+0000 7f5b4c277ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-17T17:02:49.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:49 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:49.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:49 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:49.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:49 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:49.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:49 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:49.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:49 smithi003 ceph-mon[30803]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:49.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:49 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:02:49.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:49 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:49.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:49 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:49.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:49 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:49.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:49 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:49.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:49 smithi003 ceph-mon[41530]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:49.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:49 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:02:49.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:49 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:49.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:49 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:49.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:49 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:49.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:49 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:49.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:49 smithi053 ceph-mon[34699]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:49.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:49 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:02:50.459 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:02:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:02:50.084+0000 7f5b4c277ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-17T17:02:50.460 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:02:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:02:50.202+0000 7f5b4c277ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-17T17:02:50.460 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:02:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:02:50.260+0000 7f5b4c277ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-17T17:02:50.460 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:02:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:02:50.458+0000 7f5b4c277ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-17T17:02:50.560 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:02:50.580 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi003:vg_nvme/lv_4 2024-08-17T17:02:50.716 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:02:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:02:50.521+0000 7f5b4c277ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-17T17:02:50.717 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:02:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:02:50.589+0000 7f5b4c277ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-17T17:02:50.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:50 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:50.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:50 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:50.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:50 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:50.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:50 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:02:50.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:50 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:50.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:50 smithi053 ceph-mon[34699]: Reconfiguring mgr.y (unknown last config time)... 2024-08-17T17:02:50.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:50 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:02:50.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:50 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-17T17:02:50.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:50 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:50.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:50 smithi053 ceph-mon[34699]: Reconfiguring daemon mgr.y on smithi003 2024-08-17T17:02:50.853 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:50 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:50.853 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:50 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:50.853 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:50 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:50.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:50 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:02:50.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:50 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:50.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:50 smithi003 ceph-mon[41530]: Reconfiguring mgr.y (unknown last config time)... 2024-08-17T17:02:50.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:50 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:02:50.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:50 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-17T17:02:50.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:50 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:50.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:50 smithi003 ceph-mon[41530]: Reconfiguring daemon mgr.y on smithi003 2024-08-17T17:02:50.854 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:50 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:50.854 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:50 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:50.854 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:50 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:50.854 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:50 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:02:50.855 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:50 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:50.855 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:50 smithi003 ceph-mon[30803]: Reconfiguring mgr.y (unknown last config time)... 2024-08-17T17:02:50.855 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:50 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:02:50.855 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:50 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-17T17:02:50.855 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:50 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:50.855 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:50 smithi003 ceph-mon[30803]: Reconfiguring daemon mgr.y on smithi003 2024-08-17T17:02:51.160 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:02:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:02:50.719+0000 7f5b4c277ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-17T17:02:51.465 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:02:51 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:02:51.159+0000 7f5b4c277ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-17T17:02:51.465 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:02:51 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:02:51.463+0000 7f5b4c277ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-17T17:02:51.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:51 smithi053 ceph-mon[34699]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:51.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:51 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:51.717 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:02:51 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:02:51.532+0000 7f5b4c277ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-17T17:02:51.717 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:02:51 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:02:51.665+0000 7f5b4c277ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-17T17:02:51.743 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:02:51.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:51 smithi003 ceph-mon[30803]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:51.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:51 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:51.986 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:51 smithi003 ceph-mon[41530]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:51.986 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:51 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:52.192 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:02:51 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:02:51.937+0000 7f5b4c277ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-17T17:02:52.192 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:02:52 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:02:52.000+0000 7f5b4c277ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-17T17:02:52.466 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:02:52 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:02:52.191+0000 7f5b4c277ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-17T17:02:52.467 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:02:52 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:02:52.256+0000 7f5b4c277ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-17T17:02:52.467 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:02:52 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:02:52.342+0000 7f5b4c277ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-17T17:02:52.467 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:02:52 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:02:52.402+0000 7f5b4c277ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-17T17:02:52.782 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:52 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:52.782 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:52 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:52.782 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:52 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:02:52.782 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:52 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:53.040 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:52 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:02:53.040 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:52 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:53.040 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:52 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:53.040 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:52 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:53.040 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:52 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:02:53.040 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:52 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:53.040 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:52 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:02:53.041 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:52 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:53.094 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:52 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:53.095 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:52 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:53.095 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:52 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:02:53.095 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:52 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:53.095 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:52 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:02:53.095 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:52 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:02:53.095 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:02:52 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:02:52.952+0000 7f5b4c277ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-17T17:02:53.095 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:02:53 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:02:53.093+0000 7f5b4c277ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-17T17:02:54.056 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:53 smithi003 ceph-mon[30803]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:54.056 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:53 smithi003 ceph-mon[30803]: Standby manager daemon x started 2024-08-17T17:02:54.056 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:53 smithi003 ceph-mon[30803]: from='mgr.? 172.21.15.53:0/1802369707' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:02:54.056 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:53 smithi003 ceph-mon[30803]: from='mgr.? 172.21.15.53:0/1802369707' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:02:54.056 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:53 smithi003 ceph-mon[30803]: from='mgr.? 172.21.15.53:0/1802369707' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:02:54.057 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:53 smithi003 ceph-mon[30803]: from='mgr.? 172.21.15.53:0/1802369707' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:02:54.057 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:53 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:02:54.057 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:53 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:02:54.057 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:53 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:54.057 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:53 smithi003 ceph-mon[41530]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:54.057 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:53 smithi003 ceph-mon[41530]: Standby manager daemon x started 2024-08-17T17:02:54.057 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:53 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.53:0/1802369707' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:02:54.057 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:53 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.53:0/1802369707' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:02:54.057 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:53 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.53:0/1802369707' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:02:54.057 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:53 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.53:0/1802369707' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:02:54.058 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:53 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:02:54.058 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:53 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:02:54.058 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:53 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:54.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:53 smithi053 ceph-mon[34699]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:54.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:53 smithi053 ceph-mon[34699]: Standby manager daemon x started 2024-08-17T17:02:54.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:53 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.53:0/1802369707' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:02:54.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:53 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.53:0/1802369707' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:02:54.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:53 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.53:0/1802369707' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:02:54.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:53 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.53:0/1802369707' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:02:54.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:53 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:02:54.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:53 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:02:54.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:53 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:02:55.078 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:54 smithi003 ceph-mon[30803]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:02:55.078 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:54 smithi003 ceph-mon[30803]: mgrmap e14: y(active, since 63s), standbys: x 2024-08-17T17:02:55.078 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:54 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-17T17:02:55.079 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:54 smithi003 ceph-mon[41530]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:02:55.079 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:54 smithi003 ceph-mon[41530]: mgrmap e14: y(active, since 63s), standbys: x 2024-08-17T17:02:55.079 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:54 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-17T17:02:55.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:54 smithi053 ceph-mon[34699]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:02:55.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:54 smithi053 ceph-mon[34699]: mgrmap e14: y(active, since 63s), standbys: x 2024-08-17T17:02:55.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:54 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-17T17:02:56.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:55 smithi053 ceph-mon[34699]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:56.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:55 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/651187101' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f9a7d76-f0ec-49fc-9540-2606e94758ab"}]: dispatch 2024-08-17T17:02:56.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:55 smithi053 ceph-mon[34699]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f9a7d76-f0ec-49fc-9540-2606e94758ab"}]: dispatch 2024-08-17T17:02:56.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:55 smithi053 ceph-mon[34699]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f9a7d76-f0ec-49fc-9540-2606e94758ab"}]': finished 2024-08-17T17:02:56.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:55 smithi053 ceph-mon[34699]: osdmap e5: 1 total, 0 up, 1 in 2024-08-17T17:02:56.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:55 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:02:56.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:55 smithi003 ceph-mon[41530]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:56.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:55 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/651187101' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f9a7d76-f0ec-49fc-9540-2606e94758ab"}]: dispatch 2024-08-17T17:02:56.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:55 smithi003 ceph-mon[41530]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f9a7d76-f0ec-49fc-9540-2606e94758ab"}]: dispatch 2024-08-17T17:02:56.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:55 smithi003 ceph-mon[41530]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f9a7d76-f0ec-49fc-9540-2606e94758ab"}]': finished 2024-08-17T17:02:56.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:55 smithi003 ceph-mon[41530]: osdmap e5: 1 total, 0 up, 1 in 2024-08-17T17:02:56.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:55 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:02:56.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:55 smithi003 ceph-mon[30803]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:56.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:55 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/651187101' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f9a7d76-f0ec-49fc-9540-2606e94758ab"}]: dispatch 2024-08-17T17:02:56.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:55 smithi003 ceph-mon[30803]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f9a7d76-f0ec-49fc-9540-2606e94758ab"}]: dispatch 2024-08-17T17:02:56.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:55 smithi003 ceph-mon[30803]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f9a7d76-f0ec-49fc-9540-2606e94758ab"}]': finished 2024-08-17T17:02:56.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:55 smithi003 ceph-mon[30803]: osdmap e5: 1 total, 0 up, 1 in 2024-08-17T17:02:56.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:55 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:02:57.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:56 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/4055936090' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:02:57.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:56 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/4055936090' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:02:57.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:56 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/4055936090' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:02:58.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:57 smithi053 ceph-mon[34699]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:58.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:57 smithi003 ceph-mon[30803]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:02:58.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:57 smithi003 ceph-mon[41530]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:03:00.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:02:59 smithi003 ceph-mon[30803]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:03:00.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:02:59 smithi003 ceph-mon[41530]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:03:00.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:02:59 smithi053 ceph-mon[34699]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:03:02.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:01 smithi053 ceph-mon[34699]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:03:02.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:01 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-08-17T17:03:02.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:01 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:03:02.267 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:01 smithi003 ceph-mon[30803]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:03:02.267 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:01 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-08-17T17:03:02.267 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:01 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:03:02.267 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:01 smithi003 ceph-mon[41530]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:03:02.267 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:01 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-08-17T17:03:02.267 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:01 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:03:02.919 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:02 smithi003 ceph-mon[30803]: Deploying daemon osd.0 on smithi003 2024-08-17T17:03:02.919 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:02 smithi003 ceph-mon[41530]: Deploying daemon osd.0 on smithi003 2024-08-17T17:03:03.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:02 smithi053 ceph-mon[34699]: Deploying daemon osd.0 on smithi003 2024-08-17T17:03:04.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:03 smithi003 ceph-mon[30803]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:03:04.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:03 smithi003 ceph-mon[41530]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:03:04.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:03 smithi053 ceph-mon[34699]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:03:06.122 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:05 smithi003 ceph-mon[30803]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:03:06.122 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:05 smithi003 ceph-mon[41530]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:03:06.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:05 smithi053 ceph-mon[34699]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:03:07.105 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:06 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:03:07.105 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:06 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:07.105 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:06 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:07.105 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:06 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:03:07.105 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:06 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:07.106 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:06 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:07.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:06 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:03:07.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:06 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:07.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:06 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:08.163 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:07 smithi003 ceph-mon[30803]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:03:08.163 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:07 smithi003 ceph-mon[41530]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:03:08.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:07 smithi053 ceph-mon[34699]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:03:09.873 INFO:teuthology.orchestra.run.smithi003.stdout:Created osd(s) 0 on host 'smithi003' 2024-08-17T17:03:10.145 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:09 smithi003 ceph-mon[30803]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:03:10.145 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:09 smithi003 ceph-mon[30803]: from='osd.0 [v2:172.21.15.3:6802/288807015,v1:172.21.15.3:6803/288807015]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-08-17T17:03:10.145 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:09 smithi003 ceph-mon[41530]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:03:10.145 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:09 smithi003 ceph-mon[41530]: from='osd.0 [v2:172.21.15.3:6802/288807015,v1:172.21.15.3:6803/288807015]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-08-17T17:03:10.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:09 smithi053 ceph-mon[34699]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:03:10.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:09 smithi053 ceph-mon[34699]: from='osd.0 [v2:172.21.15.3:6802/288807015,v1:172.21.15.3:6803/288807015]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-08-17T17:03:10.404 DEBUG:teuthology.orchestra.run.smithi003:osd.0> sudo journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.0.service 2024-08-17T17:03:10.407 INFO:tasks.cephadm:Deploying osd.1 on smithi003 with /dev/vg_nvme/lv_3... 2024-08-17T17:03:10.407 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-08-17T17:03:10.732 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:03:11.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:10 smithi003 ceph-mon[30803]: from='osd.0 [v2:172.21.15.3:6802/288807015,v1:172.21.15.3:6803/288807015]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-08-17T17:03:11.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:10 smithi003 ceph-mon[30803]: osdmap e6: 1 total, 0 up, 1 in 2024-08-17T17:03:11.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:10 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:03:11.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:10 smithi003 ceph-mon[30803]: from='osd.0 [v2:172.21.15.3:6802/288807015,v1:172.21.15.3:6803/288807015]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-08-17T17:03:11.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:10 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:11.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:10 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:11.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:10 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:11.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:10 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:11.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:10 smithi003 ceph-mon[30803]: from='osd.0 [v2:172.21.15.3:6802/288807015,v1:172.21.15.3:6803/288807015]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-08-17T17:03:11.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:10 smithi003 ceph-mon[30803]: osdmap e7: 1 total, 0 up, 1 in 2024-08-17T17:03:11.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:10 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:03:11.134 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:10 smithi003 ceph-mon[41530]: from='osd.0 [v2:172.21.15.3:6802/288807015,v1:172.21.15.3:6803/288807015]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-08-17T17:03:11.134 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:10 smithi003 ceph-mon[41530]: osdmap e6: 1 total, 0 up, 1 in 2024-08-17T17:03:11.134 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:10 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:03:11.134 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:10 smithi003 ceph-mon[41530]: from='osd.0 [v2:172.21.15.3:6802/288807015,v1:172.21.15.3:6803/288807015]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-08-17T17:03:11.134 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:10 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:11.134 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:10 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:11.134 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:10 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:11.135 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:10 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:11.135 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:10 smithi003 ceph-mon[41530]: from='osd.0 [v2:172.21.15.3:6802/288807015,v1:172.21.15.3:6803/288807015]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-08-17T17:03:11.135 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:10 smithi003 ceph-mon[41530]: osdmap e7: 1 total, 0 up, 1 in 2024-08-17T17:03:11.137 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:10 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:03:11.137 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:03:10 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0[45447]: 2024-08-17T17:03:10.853+0000 7f4bc7a68640 -1 osd.0 0 waiting for initial osdmap 2024-08-17T17:03:11.137 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:03:10 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0[45447]: 2024-08-17T17:03:10.860+0000 7f4bc38a1640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-17T17:03:11.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:10 smithi053 ceph-mon[34699]: from='osd.0 [v2:172.21.15.3:6802/288807015,v1:172.21.15.3:6803/288807015]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-08-17T17:03:11.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:10 smithi053 ceph-mon[34699]: osdmap e6: 1 total, 0 up, 1 in 2024-08-17T17:03:11.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:10 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:03:11.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:10 smithi053 ceph-mon[34699]: from='osd.0 [v2:172.21.15.3:6802/288807015,v1:172.21.15.3:6803/288807015]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-08-17T17:03:11.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:10 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:11.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:10 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:11.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:10 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:11.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:10 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:11.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:10 smithi053 ceph-mon[34699]: from='osd.0 [v2:172.21.15.3:6802/288807015,v1:172.21.15.3:6803/288807015]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-08-17T17:03:11.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:10 smithi053 ceph-mon[34699]: osdmap e7: 1 total, 0 up, 1 in 2024-08-17T17:03:11.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:10 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:03:12.081 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:11 smithi003 ceph-mon[30803]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:03:12.081 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:11 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:03:12.081 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:11 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:03:12.081 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:11 smithi003 ceph-mon[41530]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:03:12.081 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:11 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:03:12.082 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:11 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:03:12.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:11 smithi053 ceph-mon[34699]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:03:12.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:11 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:03:12.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:11 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:03:12.985 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:12 smithi003 ceph-mon[41530]: purged_snaps scrub starts 2024-08-17T17:03:12.985 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:12 smithi003 ceph-mon[41530]: purged_snaps scrub ok 2024-08-17T17:03:12.985 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:12 smithi003 ceph-mon[41530]: osd.0 [v2:172.21.15.3:6802/288807015,v1:172.21.15.3:6803/288807015] boot 2024-08-17T17:03:12.985 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:12 smithi003 ceph-mon[41530]: osdmap e8: 1 total, 1 up, 1 in 2024-08-17T17:03:12.985 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:12 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:03:12.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:12 smithi003 ceph-mon[30803]: purged_snaps scrub starts 2024-08-17T17:03:12.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:12 smithi003 ceph-mon[30803]: purged_snaps scrub ok 2024-08-17T17:03:12.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:12 smithi003 ceph-mon[30803]: osd.0 [v2:172.21.15.3:6802/288807015,v1:172.21.15.3:6803/288807015] boot 2024-08-17T17:03:12.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:12 smithi003 ceph-mon[30803]: osdmap e8: 1 total, 1 up, 1 in 2024-08-17T17:03:12.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:12 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:03:13.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:12 smithi053 ceph-mon[34699]: purged_snaps scrub starts 2024-08-17T17:03:13.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:12 smithi053 ceph-mon[34699]: purged_snaps scrub ok 2024-08-17T17:03:13.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:12 smithi053 ceph-mon[34699]: osd.0 [v2:172.21.15.3:6802/288807015,v1:172.21.15.3:6803/288807015] boot 2024-08-17T17:03:13.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:12 smithi053 ceph-mon[34699]: osdmap e8: 1 total, 1 up, 1 in 2024-08-17T17:03:13.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:12 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:03:13.931 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:03:13.955 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi003:vg_nvme/lv_3 2024-08-17T17:03:14.134 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:13 smithi003 ceph-mon[30803]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:03:14.134 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:13 smithi003 ceph-mon[30803]: osdmap e9: 1 total, 1 up, 1 in 2024-08-17T17:03:14.134 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:13 smithi003 ceph-mon[41530]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:03:14.134 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:13 smithi003 ceph-mon[41530]: osdmap e9: 1 total, 1 up, 1 in 2024-08-17T17:03:14.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:13 smithi053 ceph-mon[34699]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-17T17:03:14.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:13 smithi053 ceph-mon[34699]: osdmap e9: 1 total, 1 up, 1 in 2024-08-17T17:03:14.474 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:03:16.114 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:15 smithi003 ceph-mon[41530]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:16.114 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:15 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:16.114 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:15 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:16.114 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:15 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:03:16.115 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:15 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:16.115 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:15 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:03:16.115 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:15 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:03:16.115 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:15 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:16.115 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:15 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:03:16.115 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:15 smithi003 ceph-mon[30803]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:16.115 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:15 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:16.115 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:15 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:16.115 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:15 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:03:16.115 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:15 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:16.115 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:15 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:03:16.115 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:15 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:03:16.116 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:15 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:16.116 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:15 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:03:16.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:15 smithi053 ceph-mon[34699]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:16.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:15 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:16.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:15 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:16.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:15 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:03:16.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:15 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:16.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:15 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:03:16.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:15 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:03:16.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:15 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:16.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:15 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:03:17.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:16 smithi053 ceph-mon[34699]: Detected new or changed devices on smithi003 2024-08-17T17:03:17.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:16 smithi053 ceph-mon[34699]: Adjusting osd_memory_target on smithi003 to 16068M 2024-08-17T17:03:17.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:16 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:03:17.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:16 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:03:17.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:16 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:17.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:16 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:03:17.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:16 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:03:17.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:16 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:03:17.268 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:16 smithi003 ceph-mon[30803]: Detected new or changed devices on smithi003 2024-08-17T17:03:17.269 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:16 smithi003 ceph-mon[30803]: Adjusting osd_memory_target on smithi003 to 16068M 2024-08-17T17:03:17.269 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:16 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:03:17.269 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:16 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:03:17.269 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:16 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:17.269 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:16 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:03:17.269 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:16 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:03:17.269 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:16 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:03:17.269 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:16 smithi003 ceph-mon[41530]: Detected new or changed devices on smithi003 2024-08-17T17:03:17.269 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:16 smithi003 ceph-mon[41530]: Adjusting osd_memory_target on smithi003 to 16068M 2024-08-17T17:03:17.269 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:16 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:03:17.270 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:16 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:03:17.270 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:16 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:17.270 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:16 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:03:17.270 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:16 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:03:17.270 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:16 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:03:18.199 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:17 smithi003 ceph-mon[41530]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:18.199 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:17 smithi003 ceph-mon[41530]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:03:18.200 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:17 smithi003 ceph-mon[30803]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:18.200 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:17 smithi003 ceph-mon[30803]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:03:18.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:17 smithi053 ceph-mon[34699]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:18.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:17 smithi053 ceph-mon[34699]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:03:20.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:19 smithi003 ceph-mon[41530]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:20.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:19 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/2498436165' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "316b5c4e-3417-4b93-bbbb-c6ed6dd21b0e"}]: dispatch 2024-08-17T17:03:20.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:19 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/2498436165' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "316b5c4e-3417-4b93-bbbb-c6ed6dd21b0e"}]': finished 2024-08-17T17:03:20.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:19 smithi003 ceph-mon[41530]: osdmap e10: 2 total, 1 up, 2 in 2024-08-17T17:03:20.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:19 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:03:20.025 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:19 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/1065193152' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:03:20.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:19 smithi003 ceph-mon[30803]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:20.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:19 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/2498436165' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "316b5c4e-3417-4b93-bbbb-c6ed6dd21b0e"}]: dispatch 2024-08-17T17:03:20.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:19 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/2498436165' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "316b5c4e-3417-4b93-bbbb-c6ed6dd21b0e"}]': finished 2024-08-17T17:03:20.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:19 smithi003 ceph-mon[30803]: osdmap e10: 2 total, 1 up, 2 in 2024-08-17T17:03:20.027 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:19 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:03:20.027 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:19 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/1065193152' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:03:20.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:19 smithi053 ceph-mon[34699]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:20.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:19 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/2498436165' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "316b5c4e-3417-4b93-bbbb-c6ed6dd21b0e"}]: dispatch 2024-08-17T17:03:20.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:19 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/2498436165' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "316b5c4e-3417-4b93-bbbb-c6ed6dd21b0e"}]': finished 2024-08-17T17:03:20.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:19 smithi053 ceph-mon[34699]: osdmap e10: 2 total, 1 up, 2 in 2024-08-17T17:03:20.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:19 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:03:20.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:19 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/1065193152' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:03:22.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:21 smithi053 ceph-mon[34699]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:22.248 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:21 smithi003 ceph-mon[30803]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:22.249 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:21 smithi003 ceph-mon[41530]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:24.116 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:23 smithi003 ceph-mon[30803]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:24.116 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:23 smithi003 ceph-mon[41530]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:24.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:23 smithi053 ceph-mon[34699]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:26.027 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:25 smithi003 ceph-mon[30803]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:26.027 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:25 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-08-17T17:03:26.027 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:25 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:03:26.027 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:25 smithi003 ceph-mon[30803]: Deploying daemon osd.1 on smithi003 2024-08-17T17:03:26.027 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:25 smithi003 ceph-mon[41530]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:26.027 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:25 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-08-17T17:03:26.027 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:25 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:03:26.027 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:25 smithi003 ceph-mon[41530]: Deploying daemon osd.1 on smithi003 2024-08-17T17:03:26.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:25 smithi053 ceph-mon[34699]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:26.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:25 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-08-17T17:03:26.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:25 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:03:26.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:25 smithi053 ceph-mon[34699]: Deploying daemon osd.1 on smithi003 2024-08-17T17:03:28.176 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:27 smithi003 ceph-mon[30803]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:28.177 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:27 smithi003 ceph-mon[41530]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:28.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:27 smithi053 ceph-mon[34699]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:30.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:29 smithi003 ceph-mon[41530]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:30.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:29 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:03:30.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:29 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:30.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:29 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:30.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:29 smithi003 ceph-mon[30803]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:30.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:29 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:03:30.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:29 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:30.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:29 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:30.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:29 smithi053 ceph-mon[34699]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:30.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:29 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:03:30.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:29 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:30.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:29 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:32.207 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:31 smithi003 ceph-mon[30803]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:32.208 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:31 smithi003 ceph-mon[41530]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:32.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:31 smithi053 ceph-mon[34699]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:33.229 INFO:teuthology.orchestra.run.smithi003.stdout:Created osd(s) 1 on host 'smithi003' 2024-08-17T17:03:33.460 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:33 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:33.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:33 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:33.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:33 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:33.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:33 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:03:33.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:33 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:03:33.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:33 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:33.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:33 smithi053 ceph-mon[34699]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:33.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:33 smithi053 ceph-mon[34699]: from='osd.1 [v2:172.21.15.3:6810/2183964597,v1:172.21.15.3:6811/2183964597]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-08-17T17:03:33.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:33 smithi053 ceph-mon[34699]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-08-17T17:03:33.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:33 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:03:33.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:33 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:33.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:33 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:33.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:33 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:33.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:33 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:33.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:33 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:03:33.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:33 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:03:33.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:33 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:33.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:33 smithi003 ceph-mon[41530]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:33.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:33 smithi003 ceph-mon[41530]: from='osd.1 [v2:172.21.15.3:6810/2183964597,v1:172.21.15.3:6811/2183964597]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-08-17T17:03:33.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:33 smithi003 ceph-mon[41530]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-08-17T17:03:33.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:33 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:03:33.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:33 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:33.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:33 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:33.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:33 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:33.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:33 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:03:33.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:33 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:03:33.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:33 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:33.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:33 smithi003 ceph-mon[30803]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:33.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:33 smithi003 ceph-mon[30803]: from='osd.1 [v2:172.21.15.3:6810/2183964597,v1:172.21.15.3:6811/2183964597]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-08-17T17:03:33.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:33 smithi003 ceph-mon[30803]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-08-17T17:03:33.775 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:33 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:03:33.775 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:33 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:33.775 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:33 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:33.837 DEBUG:teuthology.orchestra.run.smithi003:osd.1> sudo journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.1.service 2024-08-17T17:03:33.839 INFO:tasks.cephadm:Deploying osd.2 on smithi003 with /dev/vg_nvme/lv_2... 2024-08-17T17:03:33.839 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-08-17T17:03:34.622 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:03:34.745 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:34 smithi003 ceph-mon[30803]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-17T17:03:34.745 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:34 smithi003 ceph-mon[30803]: osdmap e11: 2 total, 1 up, 2 in 2024-08-17T17:03:34.746 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:34 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:03:34.746 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:34 smithi003 ceph-mon[30803]: from='osd.1 [v2:172.21.15.3:6810/2183964597,v1:172.21.15.3:6811/2183964597]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-08-17T17:03:34.746 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:34 smithi003 ceph-mon[30803]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-08-17T17:03:34.746 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:34 smithi003 ceph-mon[41530]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-17T17:03:34.746 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:34 smithi003 ceph-mon[41530]: osdmap e11: 2 total, 1 up, 2 in 2024-08-17T17:03:34.747 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:34 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:03:34.747 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:34 smithi003 ceph-mon[41530]: from='osd.1 [v2:172.21.15.3:6810/2183964597,v1:172.21.15.3:6811/2183964597]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-08-17T17:03:34.747 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:34 smithi003 ceph-mon[41530]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-08-17T17:03:34.747 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:03:34 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1[51195]: 2024-08-17T17:03:34.480+0000 7f0f5cfe2640 -1 osd.1 0 waiting for initial osdmap 2024-08-17T17:03:34.747 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:03:34 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1[51195]: 2024-08-17T17:03:34.483+0000 7f0f57618640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-17T17:03:34.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:34 smithi053 ceph-mon[34699]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-17T17:03:34.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:34 smithi053 ceph-mon[34699]: osdmap e11: 2 total, 1 up, 2 in 2024-08-17T17:03:34.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:34 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:03:34.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:34 smithi053 ceph-mon[34699]: from='osd.1 [v2:172.21.15.3:6810/2183964597,v1:172.21.15.3:6811/2183964597]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-08-17T17:03:34.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:34 smithi053 ceph-mon[34699]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-08-17T17:03:35.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:35 smithi003 ceph-mon[41530]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-08-17T17:03:35.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:35 smithi003 ceph-mon[41530]: osdmap e12: 2 total, 1 up, 2 in 2024-08-17T17:03:35.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:35 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:03:35.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:35 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:03:35.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:35 smithi003 ceph-mon[41530]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:35.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:35 smithi003 ceph-mon[30803]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-08-17T17:03:35.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:35 smithi003 ceph-mon[30803]: osdmap e12: 2 total, 1 up, 2 in 2024-08-17T17:03:35.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:35 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:03:35.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:35 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:03:35.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:35 smithi003 ceph-mon[30803]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:35.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:35 smithi053 ceph-mon[34699]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-08-17T17:03:35.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:35 smithi053 ceph-mon[34699]: osdmap e12: 2 total, 1 up, 2 in 2024-08-17T17:03:35.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:35 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:03:35.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:35 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:03:35.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:35 smithi053 ceph-mon[34699]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-08-17T17:03:36.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:36 smithi003 ceph-mon[41530]: purged_snaps scrub starts 2024-08-17T17:03:36.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:36 smithi003 ceph-mon[41530]: purged_snaps scrub ok 2024-08-17T17:03:36.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:36 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:03:36.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:36 smithi003 ceph-mon[41530]: osd.1 [v2:172.21.15.3:6810/2183964597,v1:172.21.15.3:6811/2183964597] boot 2024-08-17T17:03:36.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:36 smithi003 ceph-mon[41530]: osdmap e13: 2 total, 2 up, 2 in 2024-08-17T17:03:36.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:36 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:03:36.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:36 smithi003 ceph-mon[30803]: purged_snaps scrub starts 2024-08-17T17:03:36.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:36 smithi003 ceph-mon[30803]: purged_snaps scrub ok 2024-08-17T17:03:36.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:36 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:03:36.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:36 smithi003 ceph-mon[30803]: osd.1 [v2:172.21.15.3:6810/2183964597,v1:172.21.15.3:6811/2183964597] boot 2024-08-17T17:03:36.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:36 smithi003 ceph-mon[30803]: osdmap e13: 2 total, 2 up, 2 in 2024-08-17T17:03:36.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:36 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:03:36.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:36 smithi053 ceph-mon[34699]: purged_snaps scrub starts 2024-08-17T17:03:36.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:36 smithi053 ceph-mon[34699]: purged_snaps scrub ok 2024-08-17T17:03:36.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:36 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:03:36.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:36 smithi053 ceph-mon[34699]: osd.1 [v2:172.21.15.3:6810/2183964597,v1:172.21.15.3:6811/2183964597] boot 2024-08-17T17:03:36.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:36 smithi053 ceph-mon[34699]: osdmap e13: 2 total, 2 up, 2 in 2024-08-17T17:03:36.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:36 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:03:37.618 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:37 smithi003 ceph-mon[30803]: osdmap e14: 2 total, 2 up, 2 in 2024-08-17T17:03:37.618 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:37 smithi003 ceph-mon[30803]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:37.618 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:37 smithi003 ceph-mon[41530]: osdmap e14: 2 total, 2 up, 2 in 2024-08-17T17:03:37.618 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:37 smithi003 ceph-mon[41530]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:37.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:37 smithi053 ceph-mon[34699]: osdmap e14: 2 total, 2 up, 2 in 2024-08-17T17:03:37.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:37 smithi053 ceph-mon[34699]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:38.942 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:38 smithi003 ceph-mon[41530]: Detected new or changed devices on smithi003 2024-08-17T17:03:38.942 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:38 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:38.943 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:38 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:38.943 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:38 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:03:38.943 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:38 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:03:38.943 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:38 smithi003 ceph-mon[41530]: Adjusting osd_memory_target on smithi003 to 8034M 2024-08-17T17:03:38.943 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:38 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:38.943 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:38 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:03:38.943 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:38 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:03:38.943 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:38 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:38.943 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:38 smithi003 ceph-mon[30803]: Detected new or changed devices on smithi003 2024-08-17T17:03:38.943 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:38 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:38.943 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:38 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:38.943 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:38 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:03:38.944 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:38 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:03:38.944 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:38 smithi003 ceph-mon[30803]: Adjusting osd_memory_target on smithi003 to 8034M 2024-08-17T17:03:38.944 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:38 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:38.944 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:38 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:03:38.944 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:38 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:03:38.944 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:38 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:38.948 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:03:38.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:38 smithi053 ceph-mon[34699]: Detected new or changed devices on smithi003 2024-08-17T17:03:38.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:38 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:38.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:38 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:38.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:38 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:03:38.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:38 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:03:38.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:38 smithi053 ceph-mon[34699]: Adjusting osd_memory_target on smithi003 to 8034M 2024-08-17T17:03:38.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:38 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:38.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:38 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:03:38.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:38 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:03:38.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:38 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:38.968 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi003:vg_nvme/lv_2 2024-08-17T17:03:39.232 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:03:39.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:39 smithi053 ceph-mon[34699]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:40.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:39 smithi003 ceph-mon[30803]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:40.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:39 smithi003 ceph-mon[41530]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:41.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:41 smithi003 ceph-mon[30803]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:41.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:41 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:03:41.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:41 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:03:41.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:41 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:03:41.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:41 smithi003 ceph-mon[41530]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:41.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:41 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:03:41.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:41 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:03:41.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:41 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:03:41.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:41 smithi053 ceph-mon[34699]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:41.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:41 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:03:41.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:41 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:03:41.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:41 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:03:42.955 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:42 smithi003 ceph-mon[30803]: from='client.24176 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:03:42.955 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:42 smithi003 ceph-mon[41530]: from='client.24176 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:03:42.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:42 smithi053 ceph-mon[34699]: from='client.24176 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:03:43.941 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:43 smithi003 ceph-mon[41530]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:43.941 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:43 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/172190594' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88af697d-38db-4c51-bfff-d0a776315596"}]: dispatch 2024-08-17T17:03:43.941 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:43 smithi003 ceph-mon[41530]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88af697d-38db-4c51-bfff-d0a776315596"}]: dispatch 2024-08-17T17:03:43.941 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:43 smithi003 ceph-mon[41530]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "88af697d-38db-4c51-bfff-d0a776315596"}]': finished 2024-08-17T17:03:43.941 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:43 smithi003 ceph-mon[41530]: osdmap e15: 3 total, 2 up, 3 in 2024-08-17T17:03:43.941 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:43 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:03:43.941 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:43 smithi003 ceph-mon[30803]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:43.942 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:43 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/172190594' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88af697d-38db-4c51-bfff-d0a776315596"}]: dispatch 2024-08-17T17:03:43.942 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:43 smithi003 ceph-mon[30803]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88af697d-38db-4c51-bfff-d0a776315596"}]: dispatch 2024-08-17T17:03:43.942 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:43 smithi003 ceph-mon[30803]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "88af697d-38db-4c51-bfff-d0a776315596"}]': finished 2024-08-17T17:03:43.942 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:43 smithi003 ceph-mon[30803]: osdmap e15: 3 total, 2 up, 3 in 2024-08-17T17:03:43.942 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:43 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:03:43.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:43 smithi053 ceph-mon[34699]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:43.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:43 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/172190594' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88af697d-38db-4c51-bfff-d0a776315596"}]: dispatch 2024-08-17T17:03:43.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:43 smithi053 ceph-mon[34699]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "88af697d-38db-4c51-bfff-d0a776315596"}]: dispatch 2024-08-17T17:03:43.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:43 smithi053 ceph-mon[34699]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "88af697d-38db-4c51-bfff-d0a776315596"}]': finished 2024-08-17T17:03:43.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:43 smithi053 ceph-mon[34699]: osdmap e15: 3 total, 2 up, 3 in 2024-08-17T17:03:43.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:43 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:03:44.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:44 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/2468091864' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:03:45.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:44 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/2468091864' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:03:45.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:44 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/2468091864' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:03:45.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:45 smithi053 ceph-mon[34699]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:46.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:45 smithi003 ceph-mon[41530]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:46.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:45 smithi003 ceph-mon[30803]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:47.892 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:47 smithi003 ceph-mon[30803]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:47.892 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:47 smithi003 ceph-mon[41530]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:47.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:47 smithi053 ceph-mon[34699]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:49.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:49 smithi053 ceph-mon[34699]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:49.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:49 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-08-17T17:03:49.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:49 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:03:50.016 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:49 smithi003 ceph-mon[30803]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:50.016 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:49 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-08-17T17:03:50.017 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:49 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:03:50.017 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:49 smithi003 ceph-mon[41530]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:50.017 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:49 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-08-17T17:03:50.017 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:49 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:03:50.865 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:50 smithi003 ceph-mon[30803]: Deploying daemon osd.2 on smithi003 2024-08-17T17:03:50.865 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:50 smithi003 ceph-mon[41530]: Deploying daemon osd.2 on smithi003 2024-08-17T17:03:50.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:50 smithi053 ceph-mon[34699]: Deploying daemon osd.2 on smithi003 2024-08-17T17:03:51.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:51 smithi053 ceph-mon[34699]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:52.013 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:51 smithi003 ceph-mon[41530]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:52.014 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:51 smithi003 ceph-mon[30803]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:53.790 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:53 smithi003 ceph-mon[30803]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:53.791 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:53 smithi003 ceph-mon[41530]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:53.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:53 smithi053 ceph-mon[34699]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:54.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:54 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:03:54.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:54 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:54.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:54 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:55.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:54 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:03:55.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:54 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:55.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:54 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:55.022 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:54 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:03:55.022 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:54 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:55.022 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:54 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:55.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:55 smithi053 ceph-mon[34699]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:56.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:55 smithi003 ceph-mon[30803]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:56.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:55 smithi003 ceph-mon[41530]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:57.601 INFO:teuthology.orchestra.run.smithi003.stdout:Created osd(s) 2 on host 'smithi003' 2024-08-17T17:03:57.897 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:57 smithi003 ceph-mon[30803]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:57.897 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:57 smithi003 ceph-mon[30803]: from='osd.2 [v2:172.21.15.3:6818/1747862993,v1:172.21.15.3:6819/1747862993]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-08-17T17:03:57.897 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:57 smithi003 ceph-mon[30803]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-08-17T17:03:57.897 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:57 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:57.897 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:57 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:57.897 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:57 smithi003 ceph-mon[41530]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:57.898 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:57 smithi003 ceph-mon[41530]: from='osd.2 [v2:172.21.15.3:6818/1747862993,v1:172.21.15.3:6819/1747862993]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-08-17T17:03:57.898 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:57 smithi003 ceph-mon[41530]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-08-17T17:03:57.898 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:57 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:57.898 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:57 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:57.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:57 smithi053 ceph-mon[34699]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:57.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:57 smithi053 ceph-mon[34699]: from='osd.2 [v2:172.21.15.3:6818/1747862993,v1:172.21.15.3:6819/1747862993]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-08-17T17:03:57.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:57 smithi053 ceph-mon[34699]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-08-17T17:03:57.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:57 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:57.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:57 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:58.191 DEBUG:teuthology.orchestra.run.smithi003:osd.2> sudo journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.2.service 2024-08-17T17:03:58.193 INFO:tasks.cephadm:Deploying osd.3 on smithi003 with /dev/vg_nvme/lv_1... 2024-08-17T17:03:58.194 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-08-17T17:03:58.502 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:03:58.917 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:58 smithi003 ceph-mon[30803]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-17T17:03:58.917 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:58 smithi003 ceph-mon[30803]: osdmap e16: 3 total, 2 up, 3 in 2024-08-17T17:03:58.918 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:58 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:03:58.918 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:58 smithi003 ceph-mon[30803]: from='osd.2 [v2:172.21.15.3:6818/1747862993,v1:172.21.15.3:6819/1747862993]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-08-17T17:03:58.918 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:58 smithi003 ceph-mon[30803]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-08-17T17:03:58.918 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:58 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:58.918 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:58 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:58.918 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:58 smithi003 ceph-mon[41530]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-17T17:03:58.918 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:58 smithi003 ceph-mon[41530]: osdmap e16: 3 total, 2 up, 3 in 2024-08-17T17:03:58.918 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:58 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:03:58.918 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:58 smithi003 ceph-mon[41530]: from='osd.2 [v2:172.21.15.3:6818/1747862993,v1:172.21.15.3:6819/1747862993]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-08-17T17:03:58.919 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:58 smithi003 ceph-mon[41530]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-08-17T17:03:58.919 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:58 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:58.919 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:58 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:58.919 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:03:58 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2[57066]: 2024-08-17T17:03:58.715+0000 7f446da50640 -1 osd.2 0 waiting for initial osdmap 2024-08-17T17:03:58.919 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:03:58 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2[57066]: 2024-08-17T17:03:58.718+0000 7f4468086640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-17T17:03:58.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:58 smithi053 ceph-mon[34699]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-17T17:03:58.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:58 smithi053 ceph-mon[34699]: osdmap e16: 3 total, 2 up, 3 in 2024-08-17T17:03:58.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:58 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:03:58.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:58 smithi053 ceph-mon[34699]: from='osd.2 [v2:172.21.15.3:6818/1747862993,v1:172.21.15.3:6819/1747862993]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-08-17T17:03:58.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:58 smithi053 ceph-mon[34699]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-08-17T17:03:58.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:58 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:58.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:58 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:03:59.874 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:59 smithi003 ceph-mon[30803]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:59.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:59 smithi003 ceph-mon[30803]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-08-17T17:03:59.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:59 smithi003 ceph-mon[30803]: osdmap e17: 3 total, 2 up, 3 in 2024-08-17T17:03:59.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:59 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:03:59.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:03:59 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:03:59.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:59 smithi003 ceph-mon[41530]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:03:59.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:59 smithi003 ceph-mon[41530]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-08-17T17:03:59.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:59 smithi003 ceph-mon[41530]: osdmap e17: 3 total, 2 up, 3 in 2024-08-17T17:03:59.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:59 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:03:59.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:03:59 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:04:00.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:59 smithi053 ceph-mon[34699]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:04:00.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:59 smithi053 ceph-mon[34699]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-08-17T17:04:00.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:59 smithi053 ceph-mon[34699]: osdmap e17: 3 total, 2 up, 3 in 2024-08-17T17:04:00.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:59 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:04:00.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:03:59 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:04:01.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:01 smithi053 ceph-mon[34699]: purged_snaps scrub starts 2024-08-17T17:04:01.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:01 smithi053 ceph-mon[34699]: purged_snaps scrub ok 2024-08-17T17:04:01.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:01 smithi053 ceph-mon[34699]: osd.2 [v2:172.21.15.3:6818/1747862993,v1:172.21.15.3:6819/1747862993] boot 2024-08-17T17:04:01.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:01 smithi053 ceph-mon[34699]: osdmap e18: 3 total, 3 up, 3 in 2024-08-17T17:04:01.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:01 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:04:01.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:01 smithi003 ceph-mon[30803]: purged_snaps scrub starts 2024-08-17T17:04:01.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:01 smithi003 ceph-mon[30803]: purged_snaps scrub ok 2024-08-17T17:04:01.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:01 smithi003 ceph-mon[30803]: osd.2 [v2:172.21.15.3:6818/1747862993,v1:172.21.15.3:6819/1747862993] boot 2024-08-17T17:04:01.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:01 smithi003 ceph-mon[30803]: osdmap e18: 3 total, 3 up, 3 in 2024-08-17T17:04:01.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:01 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:04:01.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:01 smithi003 ceph-mon[41530]: purged_snaps scrub starts 2024-08-17T17:04:01.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:01 smithi003 ceph-mon[41530]: purged_snaps scrub ok 2024-08-17T17:04:01.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:01 smithi003 ceph-mon[41530]: osd.2 [v2:172.21.15.3:6818/1747862993,v1:172.21.15.3:6819/1747862993] boot 2024-08-17T17:04:01.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:01 smithi003 ceph-mon[41530]: osdmap e18: 3 total, 3 up, 3 in 2024-08-17T17:04:01.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:01 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:04:01.842 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:04:01.868 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi003:vg_nvme/lv_1 2024-08-17T17:04:02.376 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:04:02.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:02 smithi053 ceph-mon[34699]: pgmap v73: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:04:02.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:02 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-08-17T17:04:02.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:02 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-08-17T17:04:02.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:02 smithi053 ceph-mon[34699]: osdmap e19: 3 total, 3 up, 3 in 2024-08-17T17:04:02.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:02 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:04:02.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:02 smithi003 ceph-mon[30803]: pgmap v73: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:04:02.506 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:02 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-08-17T17:04:02.506 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:02 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-08-17T17:04:02.506 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:02 smithi003 ceph-mon[30803]: osdmap e19: 3 total, 3 up, 3 in 2024-08-17T17:04:02.506 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:02 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:04:02.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:02 smithi003 ceph-mon[41530]: pgmap v73: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-08-17T17:04:02.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:02 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-08-17T17:04:02.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:02 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-08-17T17:04:02.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:02 smithi003 ceph-mon[41530]: osdmap e19: 3 total, 3 up, 3 in 2024-08-17T17:04:02.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:02 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-08-17T17:04:02.773 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:04:02 smithi003 sudo[59173]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-17T17:04:02.773 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:04:02 smithi003 sudo[59173]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-17T17:04:02.773 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:04:02 smithi003 sudo[59173]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-17T17:04:02.773 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:04:02 smithi003 sudo[59173]: pam_unix(sudo:session): session closed for user root 2024-08-17T17:04:03.413 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:03 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-08-17T17:04:03.414 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:03 smithi003 ceph-mon[41530]: osdmap e20: 3 total, 3 up, 3 in 2024-08-17T17:04:03.414 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:03 smithi003 ceph-mon[41530]: pgmap v76: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-08-17T17:04:03.414 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:04:03 smithi003 sudo[60703]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-17T17:04:03.414 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:04:03 smithi003 sudo[60703]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-17T17:04:03.414 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:04:03 smithi003 sudo[60703]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-17T17:04:03.414 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:04:03 smithi003 sudo[60703]: pam_unix(sudo:session): session closed for user root 2024-08-17T17:04:03.414 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:03 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-08-17T17:04:03.415 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:03 smithi003 ceph-mon[30803]: osdmap e20: 3 total, 3 up, 3 in 2024-08-17T17:04:03.415 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:03 smithi003 ceph-mon[30803]: pgmap v76: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-08-17T17:04:03.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:03 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-08-17T17:04:03.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:03 smithi053 ceph-mon[34699]: osdmap e20: 3 total, 3 up, 3 in 2024-08-17T17:04:03.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:03 smithi053 ceph-mon[34699]: pgmap v76: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-08-17T17:04:03.773 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:04:03 smithi003 sudo[60716]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-17T17:04:03.773 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:04:03 smithi003 sudo[60716]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-17T17:04:03.773 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:04:03 smithi003 sudo[60716]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-17T17:04:03.773 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:04:03 smithi003 sudo[60716]: pam_unix(sudo:session): session closed for user root 2024-08-17T17:04:04.163 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:04:04 smithi003 sudo[60788]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-17T17:04:04.163 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:04:04 smithi003 sudo[60788]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-17T17:04:04.163 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:04:04 smithi003 sudo[60788]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-17T17:04:04.163 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:04:04 smithi003 sudo[60788]: pam_unix(sudo:session): session closed for user root 2024-08-17T17:04:04.424 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[41530]: mgrmap e15: y(active, since 2m), standbys: x 2024-08-17T17:04:04.424 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[41530]: osdmap e21: 3 total, 3 up, 3 in 2024-08-17T17:04:04.424 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:04.424 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[41530]: Detected new or changed devices on smithi003 2024-08-17T17:04:04.424 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:04.424 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:04.424 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:04:04.424 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:04:04.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:04:04.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[41530]: Adjusting osd_memory_target on smithi003 to 5356M 2024-08-17T17:04:04.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:04.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:04.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:04:04.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:04.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:04:04.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:04.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:04:04.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:04.425 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[30803]: mgrmap e15: y(active, since 2m), standbys: x 2024-08-17T17:04:04.425 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[30803]: osdmap e21: 3 total, 3 up, 3 in 2024-08-17T17:04:04.426 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:04.426 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[30803]: Detected new or changed devices on smithi003 2024-08-17T17:04:04.426 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:04.426 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:04.426 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:04:04.426 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:04:04.426 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:04:04.426 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[30803]: Adjusting osd_memory_target on smithi003 to 5356M 2024-08-17T17:04:04.426 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:04.426 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:04.426 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:04:04.426 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:04.426 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:04:04.427 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:04.427 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:04:04.427 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:04 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:04.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:04 smithi053 ceph-mon[34699]: mgrmap e15: y(active, since 2m), standbys: x 2024-08-17T17:04:04.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:04 smithi053 ceph-mon[34699]: osdmap e21: 3 total, 3 up, 3 in 2024-08-17T17:04:04.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:04 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:04.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:04 smithi053 ceph-mon[34699]: Detected new or changed devices on smithi003 2024-08-17T17:04:04.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:04 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:04.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:04 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:04.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:04 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:04:04.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:04 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:04:04.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:04 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:04:04.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:04 smithi053 ceph-mon[34699]: Adjusting osd_memory_target on smithi003 to 5356M 2024-08-17T17:04:04.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:04 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:04.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:04 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:04.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:04 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:04:04.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:04 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:04.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:04 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:04:04.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:04 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:04.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:04 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:04:04.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:04 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:04.773 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:04:04 smithi003 sudo[60817]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-17T17:04:04.773 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:04:04 smithi003 sudo[60817]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-17T17:04:04.774 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:04:04 smithi003 sudo[60817]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-17T17:04:04.774 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:04:04 smithi003 sudo[60817]: pam_unix(sudo:session): session closed for user root 2024-08-17T17:04:05.273 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:04:04 smithi003 sudo[60908]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-17T17:04:05.273 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:04:04 smithi003 sudo[60908]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-17T17:04:05.273 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:04:04 smithi003 sudo[60908]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-17T17:04:05.273 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:04:04 smithi003 sudo[60908]: pam_unix(sudo:session): session closed for user root 2024-08-17T17:04:05.572 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:05 smithi003 sudo[60972]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-17T17:04:05.572 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:05 smithi003 sudo[60972]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-17T17:04:05.572 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:05 smithi003 sudo[60972]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-17T17:04:05.572 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:05 smithi003 sudo[60972]: pam_unix(sudo:session): session closed for user root 2024-08-17T17:04:06.159 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:05 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:04:06.159 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:05 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:04:06.159 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:05 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:06.159 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:05 smithi003 ceph-mon[41530]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-17T17:04:06.159 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:05 smithi003 sudo[60989]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-17T17:04:06.159 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:05 smithi003 sudo[60989]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-17T17:04:06.159 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:05 smithi003 sudo[60989]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-17T17:04:06.159 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:05 smithi003 sudo[60989]: pam_unix(sudo:session): session closed for user root 2024-08-17T17:04:06.159 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:05 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:04:06.160 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:05 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:04:06.160 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:05 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:06.160 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:05 smithi003 ceph-mon[30803]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-17T17:04:06.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:06 smithi053 sudo[37207]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-17T17:04:06.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:06 smithi053 sudo[37207]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-17T17:04:06.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:06 smithi053 sudo[37207]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-17T17:04:06.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:06 smithi053 sudo[37207]: pam_unix(sudo:session): session closed for user root 2024-08-17T17:04:07.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:06 smithi053 sudo[37210]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-17T17:04:07.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:06 smithi053 sudo[37210]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-17T17:04:07.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:06 smithi053 sudo[37210]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-17T17:04:07.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:06 smithi053 sudo[37210]: pam_unix(sudo:session): session closed for user root 2024-08-17T17:04:07.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:06 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:04:07.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:06 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:04:07.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:06 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:07.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:06 smithi053 ceph-mon[34699]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-17T17:04:07.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:07 smithi003 sudo[61062]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-17T17:04:07.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:07 smithi003 sudo[61062]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-17T17:04:07.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:07 smithi003 sudo[61062]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-17T17:04:07.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:07 smithi003 sudo[61062]: pam_unix(sudo:session): session closed for user root 2024-08-17T17:04:07.976 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:07 smithi003 ceph-mon[30803]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-08-17T17:04:07.976 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:07 smithi003 ceph-mon[30803]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:04:07.977 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:07 smithi003 ceph-mon[30803]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-17T17:04:07.977 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:07 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:04:07.977 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:07 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:07.977 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:07 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:04:07.977 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:07 smithi003 ceph-mon[30803]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-17T17:04:07.977 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:07 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:04:07.977 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:07 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:07.977 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:07 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:04:07.977 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:07 smithi003 ceph-mon[30803]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-17T17:04:07.977 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:07 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:07.977 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:07 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:04:07.977 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:07 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:07.977 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:07 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:04:07.978 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:07 smithi003 sudo[61065]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-17T17:04:07.978 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:07 smithi003 sudo[61065]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-17T17:04:07.978 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:07 smithi003 sudo[61065]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-17T17:04:07.978 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:07 smithi003 sudo[61065]: pam_unix(sudo:session): session closed for user root 2024-08-17T17:04:07.978 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:07 smithi003 ceph-mon[41530]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-08-17T17:04:07.978 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:07 smithi003 ceph-mon[41530]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:04:07.978 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:07 smithi003 ceph-mon[41530]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-17T17:04:07.978 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:07 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:04:07.978 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:07 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:07.978 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:07 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:04:07.978 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:07 smithi003 ceph-mon[41530]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-17T17:04:07.978 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:07 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:04:07.978 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:07 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:07.979 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:07 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:04:07.979 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:07 smithi003 ceph-mon[41530]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-17T17:04:07.979 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:07 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:07.979 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:07 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:04:07.979 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:07 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:07.979 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:07 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:04:08.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:07 smithi053 ceph-mon[34699]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-08-17T17:04:08.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:07 smithi053 ceph-mon[34699]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:04:08.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:07 smithi053 ceph-mon[34699]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-17T17:04:08.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:07 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:04:08.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:07 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:08.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:07 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:04:08.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:07 smithi053 ceph-mon[34699]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-17T17:04:08.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:07 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:04:08.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:07 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:08.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:07 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:04:08.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:07 smithi053 ceph-mon[34699]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-17T17:04:08.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:07 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:08.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:07 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:04:08.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:07 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:04:08.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:07 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:04:09.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:08 smithi003 ceph-mon[30803]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-08-17T17:04:09.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:08 smithi003 ceph-mon[30803]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-17T17:04:09.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:08 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/2154270367' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e20cf42-6a13-4ad5-9f01-203a30c692d6"}]: dispatch 2024-08-17T17:04:09.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:08 smithi003 ceph-mon[30803]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-17T17:04:09.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:08 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/2154270367' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e20cf42-6a13-4ad5-9f01-203a30c692d6"}]': finished 2024-08-17T17:04:09.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:08 smithi003 ceph-mon[30803]: osdmap e22: 4 total, 3 up, 4 in 2024-08-17T17:04:09.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:08 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:04:09.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:08 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/3343364207' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:04:09.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:08 smithi003 ceph-mon[41530]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-08-17T17:04:09.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:08 smithi003 ceph-mon[41530]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-17T17:04:09.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:08 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/2154270367' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e20cf42-6a13-4ad5-9f01-203a30c692d6"}]: dispatch 2024-08-17T17:04:09.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:08 smithi003 ceph-mon[41530]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-17T17:04:09.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:08 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/2154270367' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e20cf42-6a13-4ad5-9f01-203a30c692d6"}]': finished 2024-08-17T17:04:09.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:08 smithi003 ceph-mon[41530]: osdmap e22: 4 total, 3 up, 4 in 2024-08-17T17:04:09.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:08 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:04:09.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:08 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/3343364207' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:04:09.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:08 smithi053 ceph-mon[34699]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-08-17T17:04:09.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:08 smithi053 ceph-mon[34699]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-17T17:04:09.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:08 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/2154270367' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8e20cf42-6a13-4ad5-9f01-203a30c692d6"}]: dispatch 2024-08-17T17:04:09.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:08 smithi053 ceph-mon[34699]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-17T17:04:09.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:08 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/2154270367' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8e20cf42-6a13-4ad5-9f01-203a30c692d6"}]': finished 2024-08-17T17:04:09.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:08 smithi053 ceph-mon[34699]: osdmap e22: 4 total, 3 up, 4 in 2024-08-17T17:04:09.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:08 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:04:09.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:08 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/3343364207' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:04:10.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:09 smithi003 ceph-mon[30803]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-08-17T17:04:10.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:09 smithi003 ceph-mon[41530]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-08-17T17:04:10.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:09 smithi053 ceph-mon[34699]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-08-17T17:04:12.012 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:11 smithi003 ceph-mon[30803]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-08-17T17:04:12.013 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:11 smithi003 ceph-mon[41530]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-08-17T17:04:12.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:11 smithi053 ceph-mon[34699]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-08-17T17:04:13.766 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:13 smithi003 ceph-mon[30803]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-08-17T17:04:13.766 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:13 smithi003 ceph-mon[41530]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-08-17T17:04:14.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:13 smithi053 ceph-mon[34699]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-08-17T17:04:15.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:14 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-08-17T17:04:15.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:14 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:15.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:14 smithi003 ceph-mon[30803]: Deploying daemon osd.3 on smithi003 2024-08-17T17:04:15.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:14 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-08-17T17:04:15.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:14 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:15.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:14 smithi003 ceph-mon[41530]: Deploying daemon osd.3 on smithi003 2024-08-17T17:04:15.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:14 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-08-17T17:04:15.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:14 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:15.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:14 smithi053 ceph-mon[34699]: Deploying daemon osd.3 on smithi003 2024-08-17T17:04:16.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:15 smithi003 ceph-mon[41530]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-08-17T17:04:16.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:15 smithi003 ceph-mon[30803]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-08-17T17:04:16.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:15 smithi053 ceph-mon[34699]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-08-17T17:04:17.897 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:17 smithi003 ceph-mon[30803]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-08-17T17:04:17.897 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:17 smithi003 ceph-mon[41530]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-08-17T17:04:18.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:17 smithi053 ceph-mon[34699]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-08-17T17:04:19.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:19 smithi003 ceph-mon[30803]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.6 KiB/s rd, 72 KiB/s wr, 5 op/s 2024-08-17T17:04:19.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:19 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:04:19.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:19 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:19.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:19 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:19.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:19 smithi003 ceph-mon[41530]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.6 KiB/s rd, 72 KiB/s wr, 5 op/s 2024-08-17T17:04:19.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:19 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:04:19.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:19 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:19.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:19 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:20.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:19 smithi053 ceph-mon[34699]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.6 KiB/s rd, 72 KiB/s wr, 5 op/s 2024-08-17T17:04:20.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:19 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:04:20.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:19 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:20.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:19 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:21.969 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:21 smithi003 ceph-mon[30803]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-08-17T17:04:21.969 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:21 smithi003 ceph-mon[41530]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-08-17T17:04:22.021 INFO:teuthology.orchestra.run.smithi003.stdout:Created osd(s) 3 on host 'smithi003' 2024-08-17T17:04:22.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:21 smithi053 ceph-mon[34699]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-08-17T17:04:22.586 DEBUG:teuthology.orchestra.run.smithi003:osd.3> sudo journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.3.service 2024-08-17T17:04:22.589 INFO:tasks.cephadm:Deploying osd.4 on smithi053 with /dev/vg_nvme/lv_4... 2024-08-17T17:04:22.589 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-08-17T17:04:22.859 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:04:22.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:22 smithi053 ceph-mon[34699]: from='osd.3 [v2:172.21.15.3:6826/2572629546,v1:172.21.15.3:6827/2572629546]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-08-17T17:04:22.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:22 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:22.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:22 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:23.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:22 smithi003 ceph-mon[41530]: from='osd.3 [v2:172.21.15.3:6826/2572629546,v1:172.21.15.3:6827/2572629546]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-08-17T17:04:23.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:22 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:23.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:22 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:23.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:22 smithi003 ceph-mon[30803]: from='osd.3 [v2:172.21.15.3:6826/2572629546,v1:172.21.15.3:6827/2572629546]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-08-17T17:04:23.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:22 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:23.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:22 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:23.961 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:23 smithi003 ceph-mon[41530]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-08-17T17:04:23.962 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:23 smithi003 ceph-mon[41530]: from='osd.3 [v2:172.21.15.3:6826/2572629546,v1:172.21.15.3:6827/2572629546]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-08-17T17:04:23.962 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:23 smithi003 ceph-mon[41530]: osdmap e23: 4 total, 3 up, 4 in 2024-08-17T17:04:23.962 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:23 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:04:23.962 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:23 smithi003 ceph-mon[41530]: from='osd.3 [v2:172.21.15.3:6826/2572629546,v1:172.21.15.3:6827/2572629546]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-08-17T17:04:23.962 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:23 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:23.962 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:23 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:23.962 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:23 smithi003 ceph-mon[30803]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-08-17T17:04:23.962 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:23 smithi003 ceph-mon[30803]: from='osd.3 [v2:172.21.15.3:6826/2572629546,v1:172.21.15.3:6827/2572629546]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-08-17T17:04:23.962 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:23 smithi003 ceph-mon[30803]: osdmap e23: 4 total, 3 up, 4 in 2024-08-17T17:04:23.962 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:23 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:04:23.963 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:23 smithi003 ceph-mon[30803]: from='osd.3 [v2:172.21.15.3:6826/2572629546,v1:172.21.15.3:6827/2572629546]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-08-17T17:04:23.963 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:23 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:23.963 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:23 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:23.963 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:04:23 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3[62740]: 2024-08-17T17:04:23.764+0000 7f2a8771d640 -1 osd.3 0 waiting for initial osdmap 2024-08-17T17:04:23.963 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:04:23 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3[62740]: 2024-08-17T17:04:23.771+0000 7f2a82554640 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-17T17:04:24.069 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:23 smithi053 ceph-mon[34699]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-08-17T17:04:24.070 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:23 smithi053 ceph-mon[34699]: from='osd.3 [v2:172.21.15.3:6826/2572629546,v1:172.21.15.3:6827/2572629546]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-08-17T17:04:24.070 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:23 smithi053 ceph-mon[34699]: osdmap e23: 4 total, 3 up, 4 in 2024-08-17T17:04:24.070 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:23 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:04:24.070 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:23 smithi053 ceph-mon[34699]: from='osd.3 [v2:172.21.15.3:6826/2572629546,v1:172.21.15.3:6827/2572629546]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-08-17T17:04:24.070 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:23 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:24.070 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:23 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:24.825 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:24 smithi003 ceph-mon[41530]: from='osd.3 [v2:172.21.15.3:6826/2572629546,v1:172.21.15.3:6827/2572629546]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-08-17T17:04:24.825 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:24 smithi003 ceph-mon[41530]: osdmap e24: 4 total, 3 up, 4 in 2024-08-17T17:04:24.825 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:24 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:04:24.825 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:24 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:04:24.825 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:24 smithi003 ceph-mon[30803]: from='osd.3 [v2:172.21.15.3:6826/2572629546,v1:172.21.15.3:6827/2572629546]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-08-17T17:04:24.826 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:24 smithi003 ceph-mon[30803]: osdmap e24: 4 total, 3 up, 4 in 2024-08-17T17:04:24.826 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:24 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:04:24.826 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:24 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:04:24.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:24 smithi053 ceph-mon[34699]: from='osd.3 [v2:172.21.15.3:6826/2572629546,v1:172.21.15.3:6827/2572629546]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-08-17T17:04:24.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:24 smithi053 ceph-mon[34699]: osdmap e24: 4 total, 3 up, 4 in 2024-08-17T17:04:24.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:24 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:04:24.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:24 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:04:25.253 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T17:04:25.275 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi053:vg_nvme/lv_4 2024-08-17T17:04:25.545 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:04:25.851 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:25 smithi053 ceph-mon[34699]: purged_snaps scrub starts 2024-08-17T17:04:25.851 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:25 smithi053 ceph-mon[34699]: purged_snaps scrub ok 2024-08-17T17:04:25.851 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:25 smithi053 ceph-mon[34699]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-08-17T17:04:25.851 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:25 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:04:25.851 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:25 smithi053 ceph-mon[34699]: osd.3 [v2:172.21.15.3:6826/2572629546,v1:172.21.15.3:6827/2572629546] boot 2024-08-17T17:04:25.851 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:25 smithi053 ceph-mon[34699]: osdmap e25: 4 total, 4 up, 4 in 2024-08-17T17:04:25.852 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:25 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:04:26.017 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:25 smithi003 ceph-mon[41530]: purged_snaps scrub starts 2024-08-17T17:04:26.018 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:25 smithi003 ceph-mon[41530]: purged_snaps scrub ok 2024-08-17T17:04:26.018 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:25 smithi003 ceph-mon[41530]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-08-17T17:04:26.018 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:25 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:04:26.018 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:25 smithi003 ceph-mon[41530]: osd.3 [v2:172.21.15.3:6826/2572629546,v1:172.21.15.3:6827/2572629546] boot 2024-08-17T17:04:26.018 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:25 smithi003 ceph-mon[41530]: osdmap e25: 4 total, 4 up, 4 in 2024-08-17T17:04:26.018 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:25 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:04:26.018 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:25 smithi003 ceph-mon[30803]: purged_snaps scrub starts 2024-08-17T17:04:26.018 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:25 smithi003 ceph-mon[30803]: purged_snaps scrub ok 2024-08-17T17:04:26.018 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:25 smithi003 ceph-mon[30803]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-08-17T17:04:26.018 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:25 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:04:26.019 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:25 smithi003 ceph-mon[30803]: osd.3 [v2:172.21.15.3:6826/2572629546,v1:172.21.15.3:6827/2572629546] boot 2024-08-17T17:04:26.019 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:25 smithi003 ceph-mon[30803]: osdmap e25: 4 total, 4 up, 4 in 2024-08-17T17:04:26.019 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:25 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:04:27.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:26 smithi053 ceph-mon[34699]: osdmap e26: 4 total, 4 up, 4 in 2024-08-17T17:04:27.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:26 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:27.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:26 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:27.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:26 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:04:27.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:26 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:04:27.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:26 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:04:27.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:26 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:04:27.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:26 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:27.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:26 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:27.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:26 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:04:27.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:26 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:27.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:26 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:04:27.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:26 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:27.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:26 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:04:27.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:26 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:27.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:26 smithi003 ceph-mon[41530]: osdmap e26: 4 total, 4 up, 4 in 2024-08-17T17:04:27.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:26 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:27.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:26 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:27.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:26 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:04:27.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:26 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:04:27.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:26 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:04:27.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:26 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:04:27.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:26 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:27.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:26 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:27.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:26 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:04:27.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:26 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:27.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:26 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:04:27.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:26 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:27.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:26 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:04:27.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:26 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:27.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:26 smithi003 ceph-mon[30803]: osdmap e26: 4 total, 4 up, 4 in 2024-08-17T17:04:27.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:26 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:27.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:26 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:27.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:26 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:04:27.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:26 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:04:27.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:26 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:04:27.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:26 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:04:27.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:26 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:27.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:26 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:27.277 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:26 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:04:27.277 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:26 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:27.277 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:26 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:04:27.277 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:26 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:27.277 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:26 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:04:27.277 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:26 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:27.852 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:27 smithi053 ceph-mon[34699]: Detected new or changed devices on smithi003 2024-08-17T17:04:27.852 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:27 smithi053 ceph-mon[34699]: Adjusting osd_memory_target on smithi003 to 4017M 2024-08-17T17:04:27.853 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:27 smithi053 ceph-mon[34699]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-08-17T17:04:27.853 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:27 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:04:27.853 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:27 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:04:27.853 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:27 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:28.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:27 smithi003 ceph-mon[41530]: Detected new or changed devices on smithi003 2024-08-17T17:04:28.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:27 smithi003 ceph-mon[41530]: Adjusting osd_memory_target on smithi003 to 4017M 2024-08-17T17:04:28.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:27 smithi003 ceph-mon[41530]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-08-17T17:04:28.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:27 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:04:28.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:27 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:04:28.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:27 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:28.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:27 smithi003 ceph-mon[30803]: Detected new or changed devices on smithi003 2024-08-17T17:04:28.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:27 smithi003 ceph-mon[30803]: Adjusting osd_memory_target on smithi003 to 4017M 2024-08-17T17:04:28.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:27 smithi003 ceph-mon[30803]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-08-17T17:04:28.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:27 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:04:28.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:27 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:04:28.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:27 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:29.019 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:28 smithi053 ceph-mon[34699]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:04:29.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:28 smithi003 ceph-mon[30803]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:04:29.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:28 smithi003 ceph-mon[41530]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:04:29.911 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:29 smithi053 ceph-mon[34699]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-08-17T17:04:29.911 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:29 smithi053 ceph-mon[34699]: from='client.? 172.21.15.53:0/1014022710' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e8eabd7-8258-4b78-bb21-1a7291e0f1d4"}]: dispatch 2024-08-17T17:04:29.911 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:29 smithi053 ceph-mon[34699]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e8eabd7-8258-4b78-bb21-1a7291e0f1d4"}]: dispatch 2024-08-17T17:04:29.911 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:29 smithi053 ceph-mon[34699]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2e8eabd7-8258-4b78-bb21-1a7291e0f1d4"}]': finished 2024-08-17T17:04:29.911 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:29 smithi053 ceph-mon[34699]: osdmap e27: 5 total, 4 up, 5 in 2024-08-17T17:04:29.911 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:29 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:04:30.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:29 smithi003 ceph-mon[30803]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-08-17T17:04:30.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:29 smithi003 ceph-mon[30803]: from='client.? 172.21.15.53:0/1014022710' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e8eabd7-8258-4b78-bb21-1a7291e0f1d4"}]: dispatch 2024-08-17T17:04:30.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:29 smithi003 ceph-mon[30803]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e8eabd7-8258-4b78-bb21-1a7291e0f1d4"}]: dispatch 2024-08-17T17:04:30.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:29 smithi003 ceph-mon[30803]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2e8eabd7-8258-4b78-bb21-1a7291e0f1d4"}]': finished 2024-08-17T17:04:30.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:29 smithi003 ceph-mon[30803]: osdmap e27: 5 total, 4 up, 5 in 2024-08-17T17:04:30.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:29 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:04:30.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:29 smithi003 ceph-mon[41530]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-08-17T17:04:30.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:29 smithi003 ceph-mon[41530]: from='client.? 172.21.15.53:0/1014022710' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e8eabd7-8258-4b78-bb21-1a7291e0f1d4"}]: dispatch 2024-08-17T17:04:30.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:29 smithi003 ceph-mon[41530]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e8eabd7-8258-4b78-bb21-1a7291e0f1d4"}]: dispatch 2024-08-17T17:04:30.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:29 smithi003 ceph-mon[41530]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2e8eabd7-8258-4b78-bb21-1a7291e0f1d4"}]': finished 2024-08-17T17:04:30.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:29 smithi003 ceph-mon[41530]: osdmap e27: 5 total, 4 up, 5 in 2024-08-17T17:04:30.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:29 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:04:31.118 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:30 smithi053 ceph-mon[34699]: from='client.? 172.21.15.53:0/3786813599' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:04:31.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:30 smithi003 ceph-mon[30803]: from='client.? 172.21.15.53:0/3786813599' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:04:31.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:30 smithi003 ceph-mon[41530]: from='client.? 172.21.15.53:0/3786813599' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:04:32.106 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:31 smithi053 ceph-mon[34699]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-08-17T17:04:32.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:31 smithi003 ceph-mon[30803]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-08-17T17:04:32.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:31 smithi003 ceph-mon[41530]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-08-17T17:04:33.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:33 smithi053 ceph-mon[34699]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-08-17T17:04:34.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:33 smithi003 ceph-mon[41530]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-08-17T17:04:34.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:33 smithi003 ceph-mon[30803]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-08-17T17:04:36.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:35 smithi053 ceph-mon[34699]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-08-17T17:04:36.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:35 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-08-17T17:04:36.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:35 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:36.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:35 smithi053 ceph-mon[34699]: Deploying daemon osd.4 on smithi053 2024-08-17T17:04:36.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:35 smithi003 ceph-mon[41530]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-08-17T17:04:36.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:35 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-08-17T17:04:36.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:35 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:36.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:35 smithi003 ceph-mon[41530]: Deploying daemon osd.4 on smithi053 2024-08-17T17:04:36.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:35 smithi003 ceph-mon[30803]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-08-17T17:04:36.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:35 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-08-17T17:04:36.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:35 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:36.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:35 smithi003 ceph-mon[30803]: Deploying daemon osd.4 on smithi053 2024-08-17T17:04:37.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:37 smithi053 ceph-mon[34699]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-08-17T17:04:38.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:37 smithi003 ceph-mon[30803]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-08-17T17:04:38.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:37 smithi003 ceph-mon[41530]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-08-17T17:04:39.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:39 smithi053 ceph-mon[34699]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-08-17T17:04:39.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:39 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:04:39.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:39 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:39.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:39 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:40.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:39 smithi003 ceph-mon[30803]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-08-17T17:04:40.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:39 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:04:40.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:39 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:40.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:39 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:40.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:39 smithi003 ceph-mon[41530]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-08-17T17:04:40.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:39 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:04:40.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:39 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:40.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:39 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:41.972 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:41 smithi053 ceph-mon[34699]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-08-17T17:04:42.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:41 smithi003 ceph-mon[30803]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-08-17T17:04:42.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:41 smithi003 ceph-mon[41530]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-08-17T17:04:42.936 INFO:teuthology.orchestra.run.smithi053.stdout:Created osd(s) 4 on host 'smithi053' 2024-08-17T17:04:43.512 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:43 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:43.512 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:43 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:43.512 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:43 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:43.512 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:43 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:04:43.512 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:43 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:43.513 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:43 smithi053 ceph-mon[34699]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-08-17T17:04:43.513 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:43 smithi053 ceph-mon[34699]: from='osd.4 [v2:172.21.15.53:6800/1387588062,v1:172.21.15.53:6801/1387588062]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-08-17T17:04:43.513 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:43 smithi053 ceph-mon[34699]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-08-17T17:04:43.513 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:43 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:04:43.513 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:43 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:43.513 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:43 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:43.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:43 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:43.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:43 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:43.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:43 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:43.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:43 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:04:43.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:43 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:43.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:43 smithi003 ceph-mon[41530]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-08-17T17:04:43.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:43 smithi003 ceph-mon[41530]: from='osd.4 [v2:172.21.15.53:6800/1387588062,v1:172.21.15.53:6801/1387588062]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-08-17T17:04:43.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:43 smithi003 ceph-mon[41530]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-08-17T17:04:43.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:43 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:04:43.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:43 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:43.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:43 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:43.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:43 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:43.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:43 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:43.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:43 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:43.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:43 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:04:43.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:43 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:43.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:43 smithi003 ceph-mon[30803]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-08-17T17:04:43.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:43 smithi003 ceph-mon[30803]: from='osd.4 [v2:172.21.15.53:6800/1387588062,v1:172.21.15.53:6801/1387588062]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-08-17T17:04:43.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:43 smithi003 ceph-mon[30803]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-08-17T17:04:43.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:43 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:04:43.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:43 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:43.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:43 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:43.538 DEBUG:teuthology.orchestra.run.smithi053:osd.4> sudo journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.4.service 2024-08-17T17:04:43.540 INFO:tasks.cephadm:Deploying osd.5 on smithi053 with /dev/vg_nvme/lv_3... 2024-08-17T17:04:43.540 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-08-17T17:04:44.216 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:04:44.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:44 smithi003 ceph-mon[30803]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-08-17T17:04:44.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:44 smithi003 ceph-mon[30803]: osdmap e28: 5 total, 4 up, 5 in 2024-08-17T17:04:44.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:44 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:04:44.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:44 smithi003 ceph-mon[30803]: from='osd.4 [v2:172.21.15.53:6800/1387588062,v1:172.21.15.53:6801/1387588062]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-08-17T17:04:44.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:44 smithi003 ceph-mon[30803]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-08-17T17:04:44.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:44 smithi003 ceph-mon[41530]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-08-17T17:04:44.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:44 smithi003 ceph-mon[41530]: osdmap e28: 5 total, 4 up, 5 in 2024-08-17T17:04:44.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:44 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:04:44.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:44 smithi003 ceph-mon[41530]: from='osd.4 [v2:172.21.15.53:6800/1387588062,v1:172.21.15.53:6801/1387588062]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-08-17T17:04:44.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:44 smithi003 ceph-mon[41530]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-08-17T17:04:44.552 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:44 smithi053 ceph-mon[34699]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-08-17T17:04:44.552 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:44 smithi053 ceph-mon[34699]: osdmap e28: 5 total, 4 up, 5 in 2024-08-17T17:04:44.552 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:44 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:04:44.552 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:44 smithi053 ceph-mon[34699]: from='osd.4 [v2:172.21.15.53:6800/1387588062,v1:172.21.15.53:6801/1387588062]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-08-17T17:04:44.552 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:44 smithi053 ceph-mon[34699]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-08-17T17:04:44.552 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:04:44 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-4[39627]: 2024-08-17T17:04:44.241+0000 7f78cefbd640 -1 osd.4 0 waiting for initial osdmap 2024-08-17T17:04:44.552 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:04:44 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-4[39627]: 2024-08-17T17:04:44.245+0000 7f78c8df2640 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-17T17:04:45.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:45 smithi003 ceph-mon[30803]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-08-17T17:04:45.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:45 smithi003 ceph-mon[30803]: osdmap e29: 5 total, 4 up, 5 in 2024-08-17T17:04:45.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:45 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:04:45.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:45 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:04:45.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:45 smithi003 ceph-mon[30803]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-08-17T17:04:45.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:45 smithi003 ceph-mon[41530]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-08-17T17:04:45.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:45 smithi003 ceph-mon[41530]: osdmap e29: 5 total, 4 up, 5 in 2024-08-17T17:04:45.527 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:45 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:04:45.527 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:45 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:04:45.527 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:45 smithi003 ceph-mon[41530]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-08-17T17:04:45.567 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:45 smithi053 ceph-mon[34699]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-08-17T17:04:45.567 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:45 smithi053 ceph-mon[34699]: osdmap e29: 5 total, 4 up, 5 in 2024-08-17T17:04:45.568 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:45 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:04:45.568 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:45 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:04:45.568 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:45 smithi053 ceph-mon[34699]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-08-17T17:04:46.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:46 smithi003 ceph-mon[41530]: purged_snaps scrub starts 2024-08-17T17:04:46.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:46 smithi003 ceph-mon[41530]: purged_snaps scrub ok 2024-08-17T17:04:46.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:46 smithi003 ceph-mon[41530]: osd.4 [v2:172.21.15.53:6800/1387588062,v1:172.21.15.53:6801/1387588062] boot 2024-08-17T17:04:46.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:46 smithi003 ceph-mon[41530]: osdmap e30: 5 total, 5 up, 5 in 2024-08-17T17:04:46.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:46 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:04:46.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:46 smithi003 ceph-mon[41530]: osdmap e31: 5 total, 5 up, 5 in 2024-08-17T17:04:46.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:46 smithi003 ceph-mon[30803]: purged_snaps scrub starts 2024-08-17T17:04:46.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:46 smithi003 ceph-mon[30803]: purged_snaps scrub ok 2024-08-17T17:04:46.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:46 smithi003 ceph-mon[30803]: osd.4 [v2:172.21.15.53:6800/1387588062,v1:172.21.15.53:6801/1387588062] boot 2024-08-17T17:04:46.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:46 smithi003 ceph-mon[30803]: osdmap e30: 5 total, 5 up, 5 in 2024-08-17T17:04:46.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:46 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:04:46.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:46 smithi003 ceph-mon[30803]: osdmap e31: 5 total, 5 up, 5 in 2024-08-17T17:04:46.706 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:46 smithi053 ceph-mon[34699]: purged_snaps scrub starts 2024-08-17T17:04:46.706 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:46 smithi053 ceph-mon[34699]: purged_snaps scrub ok 2024-08-17T17:04:46.707 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:46 smithi053 ceph-mon[34699]: osd.4 [v2:172.21.15.53:6800/1387588062,v1:172.21.15.53:6801/1387588062] boot 2024-08-17T17:04:46.707 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:46 smithi053 ceph-mon[34699]: osdmap e30: 5 total, 5 up, 5 in 2024-08-17T17:04:46.707 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:46 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:04:46.707 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:46 smithi053 ceph-mon[34699]: osdmap e31: 5 total, 5 up, 5 in 2024-08-17T17:04:48.290 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:48 smithi053 ceph-mon[34699]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-08-17T17:04:48.290 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:48 smithi053 ceph-mon[34699]: Detected new or changed devices on smithi053 2024-08-17T17:04:48.290 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:48 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:48.290 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:48 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:48.290 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:48 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:04:48.290 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:48 smithi053 ceph-mon[34699]: Adjusting osd_memory_target on smithi053 to 17092M 2024-08-17T17:04:48.290 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:48 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:48.290 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:48 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:48.290 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:48 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:04:48.290 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:48 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:48.290 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:48 smithi053 ceph-mon[34699]: osdmap e32: 5 total, 5 up, 5 in 2024-08-17T17:04:48.295 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T17:04:48.316 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi053:vg_nvme/lv_3 2024-08-17T17:04:48.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:48 smithi003 ceph-mon[30803]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-08-17T17:04:48.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:48 smithi003 ceph-mon[30803]: Detected new or changed devices on smithi053 2024-08-17T17:04:48.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:48 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:48.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:48 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:48.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:48 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:04:48.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:48 smithi003 ceph-mon[30803]: Adjusting osd_memory_target on smithi053 to 17092M 2024-08-17T17:04:48.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:48 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:48.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:48 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:48.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:48 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:04:48.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:48 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:48.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:48 smithi003 ceph-mon[30803]: osdmap e32: 5 total, 5 up, 5 in 2024-08-17T17:04:48.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:48 smithi003 ceph-mon[41530]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-08-17T17:04:48.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:48 smithi003 ceph-mon[41530]: Detected new or changed devices on smithi053 2024-08-17T17:04:48.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:48 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:48.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:48 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:48.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:48 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:04:48.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:48 smithi003 ceph-mon[41530]: Adjusting osd_memory_target on smithi053 to 17092M 2024-08-17T17:04:48.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:48 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:48.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:48 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:48.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:48 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:04:48.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:48 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:04:48.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:48 smithi003 ceph-mon[41530]: osdmap e32: 5 total, 5 up, 5 in 2024-08-17T17:04:48.577 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:04:49.363 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:49 smithi053 ceph-mon[34699]: osdmap e33: 5 total, 5 up, 5 in 2024-08-17T17:04:49.363 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:49 smithi053 ceph-mon[34699]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-08-17T17:04:49.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:49 smithi003 ceph-mon[30803]: osdmap e33: 5 total, 5 up, 5 in 2024-08-17T17:04:49.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:49 smithi003 ceph-mon[30803]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-08-17T17:04:49.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:49 smithi003 ceph-mon[41530]: osdmap e33: 5 total, 5 up, 5 in 2024-08-17T17:04:49.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:49 smithi003 ceph-mon[41530]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-08-17T17:04:50.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:50 smithi053 ceph-mon[34699]: osdmap e34: 5 total, 5 up, 5 in 2024-08-17T17:04:50.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:50 smithi003 ceph-mon[30803]: osdmap e34: 5 total, 5 up, 5 in 2024-08-17T17:04:50.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:50 smithi003 ceph-mon[41530]: osdmap e34: 5 total, 5 up, 5 in 2024-08-17T17:04:51.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:51 smithi053 ceph-mon[34699]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:04:51.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:51 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:04:51.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:51 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:04:51.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:51 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:51.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:51 smithi053 ceph-mon[34699]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-08-17T17:04:51.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:51 smithi003 ceph-mon[30803]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:04:51.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:51 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:04:51.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:51 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:04:51.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:51 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:51.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:51 smithi003 ceph-mon[30803]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-08-17T17:04:51.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:51 smithi003 ceph-mon[41530]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:04:51.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:51 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:04:51.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:51 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:04:51.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:51 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:51.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:51 smithi003 ceph-mon[41530]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-08-17T17:04:52.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:52 smithi053 ceph-mon[34699]: from='client.? 172.21.15.53:0/515123453' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "738f10f7-ff0d-4392-a818-eb23c9f65a63"}]: dispatch 2024-08-17T17:04:52.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:52 smithi053 ceph-mon[34699]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "738f10f7-ff0d-4392-a818-eb23c9f65a63"}]: dispatch 2024-08-17T17:04:52.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:52 smithi053 ceph-mon[34699]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "738f10f7-ff0d-4392-a818-eb23c9f65a63"}]': finished 2024-08-17T17:04:52.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:52 smithi053 ceph-mon[34699]: osdmap e35: 6 total, 5 up, 6 in 2024-08-17T17:04:52.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:52 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:04:52.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:52 smithi003 ceph-mon[30803]: from='client.? 172.21.15.53:0/515123453' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "738f10f7-ff0d-4392-a818-eb23c9f65a63"}]: dispatch 2024-08-17T17:04:52.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:52 smithi003 ceph-mon[30803]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "738f10f7-ff0d-4392-a818-eb23c9f65a63"}]: dispatch 2024-08-17T17:04:52.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:52 smithi003 ceph-mon[30803]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "738f10f7-ff0d-4392-a818-eb23c9f65a63"}]': finished 2024-08-17T17:04:52.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:52 smithi003 ceph-mon[30803]: osdmap e35: 6 total, 5 up, 6 in 2024-08-17T17:04:52.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:52 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:04:52.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:52 smithi003 ceph-mon[41530]: from='client.? 172.21.15.53:0/515123453' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "738f10f7-ff0d-4392-a818-eb23c9f65a63"}]: dispatch 2024-08-17T17:04:52.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:52 smithi003 ceph-mon[41530]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "738f10f7-ff0d-4392-a818-eb23c9f65a63"}]: dispatch 2024-08-17T17:04:52.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:52 smithi003 ceph-mon[41530]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "738f10f7-ff0d-4392-a818-eb23c9f65a63"}]': finished 2024-08-17T17:04:52.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:52 smithi003 ceph-mon[41530]: osdmap e35: 6 total, 5 up, 6 in 2024-08-17T17:04:52.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:52 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:04:53.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:53 smithi053 ceph-mon[34699]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-08-17T17:04:53.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:53 smithi053 ceph-mon[34699]: from='client.? 172.21.15.53:0/3559763350' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:04:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:53 smithi003 ceph-mon[30803]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-08-17T17:04:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:53 smithi003 ceph-mon[30803]: from='client.? 172.21.15.53:0/3559763350' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:04:53.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:53 smithi003 ceph-mon[41530]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-08-17T17:04:53.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:53 smithi003 ceph-mon[41530]: from='client.? 172.21.15.53:0/3559763350' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:04:56.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:55 smithi003 ceph-mon[41530]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-08-17T17:04:56.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:55 smithi003 ceph-mon[30803]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-08-17T17:04:56.060 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:55 smithi053 ceph-mon[34699]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-08-17T17:04:57.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:57 smithi053 ceph-mon[34699]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-08-17T17:04:58.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:57 smithi003 ceph-mon[30803]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-08-17T17:04:58.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:57 smithi003 ceph-mon[41530]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-08-17T17:04:58.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:58 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-08-17T17:04:58.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:58 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:58.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:58 smithi053 ceph-mon[34699]: Deploying daemon osd.5 on smithi053 2024-08-17T17:04:59.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:58 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-08-17T17:04:59.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:58 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:59.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:58 smithi003 ceph-mon[30803]: Deploying daemon osd.5 on smithi053 2024-08-17T17:04:59.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:58 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-08-17T17:04:59.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:58 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:04:59.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:58 smithi003 ceph-mon[41530]: Deploying daemon osd.5 on smithi053 2024-08-17T17:04:59.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:04:59 smithi053 ceph-mon[34699]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-08-17T17:05:00.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:04:59 smithi003 ceph-mon[30803]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-08-17T17:05:00.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:04:59 smithi003 ceph-mon[41530]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-08-17T17:05:02.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:01 smithi003 ceph-mon[30803]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-08-17T17:05:02.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:01 smithi003 ceph-mon[41530]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-08-17T17:05:02.045 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:01 smithi053 ceph-mon[34699]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-08-17T17:05:03.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:02 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:03.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:02 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:03.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:02 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:03.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:02 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:03.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:02 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:03.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:02 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:03.042 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:02 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:03.042 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:02 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:03.042 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:02 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:04.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:03 smithi003 ceph-mon[30803]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-08-17T17:05:04.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:03 smithi003 ceph-mon[41530]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-08-17T17:05:04.058 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:03 smithi053 ceph-mon[34699]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-08-17T17:05:05.762 INFO:teuthology.orchestra.run.smithi053.stdout:Created osd(s) 5 on host 'smithi053' 2024-08-17T17:05:06.068 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:05 smithi053 ceph-mon[34699]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-08-17T17:05:06.068 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:05 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:06.069 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:05 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:06.069 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:05 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:06.069 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:05 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:06.069 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:05 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:06.069 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:05 smithi053 ceph-mon[34699]: from='osd.5 [v2:172.21.15.53:6808/1263400356,v1:172.21.15.53:6809/1263400356]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-08-17T17:05:06.069 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:05 smithi053 ceph-mon[34699]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-08-17T17:05:06.069 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:05 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:06.069 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:05 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:06.069 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:05 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:06.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:05 smithi003 ceph-mon[41530]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-08-17T17:05:06.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:05 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:06.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:05 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:06.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:05 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:06.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:05 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:06.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:05 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:06.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:05 smithi003 ceph-mon[41530]: from='osd.5 [v2:172.21.15.53:6808/1263400356,v1:172.21.15.53:6809/1263400356]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-08-17T17:05:06.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:05 smithi003 ceph-mon[41530]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-08-17T17:05:06.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:05 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:06.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:05 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:06.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:05 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:06.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:05 smithi003 ceph-mon[30803]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-08-17T17:05:06.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:05 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:06.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:05 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:06.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:05 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:06.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:05 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:06.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:05 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:06.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:05 smithi003 ceph-mon[30803]: from='osd.5 [v2:172.21.15.53:6808/1263400356,v1:172.21.15.53:6809/1263400356]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-08-17T17:05:06.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:05 smithi003 ceph-mon[30803]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-08-17T17:05:06.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:05 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:06.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:05 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:06.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:05 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:06.396 DEBUG:teuthology.orchestra.run.smithi053:osd.5> sudo journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.5.service 2024-08-17T17:05:06.399 INFO:tasks.cephadm:Deploying osd.6 on smithi053 with /dev/vg_nvme/lv_2... 2024-08-17T17:05:06.399 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-08-17T17:05:06.871 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:06 smithi053 ceph-mon[34699]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-08-17T17:05:06.872 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:06 smithi053 ceph-mon[34699]: osdmap e36: 6 total, 5 up, 6 in 2024-08-17T17:05:06.872 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:06 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:05:06.872 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:06 smithi053 ceph-mon[34699]: from='osd.5 [v2:172.21.15.53:6808/1263400356,v1:172.21.15.53:6809/1263400356]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-08-17T17:05:06.872 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:06 smithi053 ceph-mon[34699]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-08-17T17:05:06.872 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:05:06 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:05:06.861+0000 7f59b8432640 -1 osd.5 0 waiting for initial osdmap 2024-08-17T17:05:06.872 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:05:06 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:05:06.866+0000 7f59b3269640 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-17T17:05:07.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:06 smithi003 ceph-mon[30803]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-08-17T17:05:07.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:06 smithi003 ceph-mon[30803]: osdmap e36: 6 total, 5 up, 6 in 2024-08-17T17:05:07.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:06 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:05:07.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:06 smithi003 ceph-mon[30803]: from='osd.5 [v2:172.21.15.53:6808/1263400356,v1:172.21.15.53:6809/1263400356]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-08-17T17:05:07.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:06 smithi003 ceph-mon[30803]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-08-17T17:05:07.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:06 smithi003 ceph-mon[41530]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-08-17T17:05:07.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:06 smithi003 ceph-mon[41530]: osdmap e36: 6 total, 5 up, 6 in 2024-08-17T17:05:07.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:06 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:05:07.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:06 smithi003 ceph-mon[41530]: from='osd.5 [v2:172.21.15.53:6808/1263400356,v1:172.21.15.53:6809/1263400356]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-08-17T17:05:07.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:06 smithi003 ceph-mon[41530]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-08-17T17:05:07.833 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:05:07.958 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:07 smithi053 ceph-mon[34699]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-08-17T17:05:07.958 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:07 smithi053 ceph-mon[34699]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-08-17T17:05:07.958 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:07 smithi053 ceph-mon[34699]: osdmap e37: 6 total, 5 up, 6 in 2024-08-17T17:05:07.958 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:07 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:05:07.958 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:07 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:05:08.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:07 smithi003 ceph-mon[30803]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-08-17T17:05:08.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:07 smithi003 ceph-mon[30803]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-08-17T17:05:08.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:07 smithi003 ceph-mon[30803]: osdmap e37: 6 total, 5 up, 6 in 2024-08-17T17:05:08.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:07 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:05:08.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:07 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:05:08.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:07 smithi003 ceph-mon[41530]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-08-17T17:05:08.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:07 smithi003 ceph-mon[41530]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-08-17T17:05:08.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:07 smithi003 ceph-mon[41530]: osdmap e37: 6 total, 5 up, 6 in 2024-08-17T17:05:08.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:07 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:05:08.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:07 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:05:09.011 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:08 smithi053 ceph-mon[34699]: purged_snaps scrub starts 2024-08-17T17:05:09.011 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:08 smithi053 ceph-mon[34699]: purged_snaps scrub ok 2024-08-17T17:05:09.011 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:08 smithi053 ceph-mon[34699]: osd.5 [v2:172.21.15.53:6808/1263400356,v1:172.21.15.53:6809/1263400356] boot 2024-08-17T17:05:09.011 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:08 smithi053 ceph-mon[34699]: osdmap e38: 6 total, 6 up, 6 in 2024-08-17T17:05:09.011 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:08 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:05:09.011 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:08 smithi053 ceph-mon[34699]: osdmap e39: 6 total, 6 up, 6 in 2024-08-17T17:05:09.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:08 smithi003 ceph-mon[30803]: purged_snaps scrub starts 2024-08-17T17:05:09.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:08 smithi003 ceph-mon[30803]: purged_snaps scrub ok 2024-08-17T17:05:09.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:08 smithi003 ceph-mon[30803]: osd.5 [v2:172.21.15.53:6808/1263400356,v1:172.21.15.53:6809/1263400356] boot 2024-08-17T17:05:09.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:08 smithi003 ceph-mon[30803]: osdmap e38: 6 total, 6 up, 6 in 2024-08-17T17:05:09.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:08 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:05:09.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:08 smithi003 ceph-mon[30803]: osdmap e39: 6 total, 6 up, 6 in 2024-08-17T17:05:09.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:08 smithi003 ceph-mon[41530]: purged_snaps scrub starts 2024-08-17T17:05:09.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:08 smithi003 ceph-mon[41530]: purged_snaps scrub ok 2024-08-17T17:05:09.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:08 smithi003 ceph-mon[41530]: osd.5 [v2:172.21.15.53:6808/1263400356,v1:172.21.15.53:6809/1263400356] boot 2024-08-17T17:05:09.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:08 smithi003 ceph-mon[41530]: osdmap e38: 6 total, 6 up, 6 in 2024-08-17T17:05:09.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:08 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:05:09.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:08 smithi003 ceph-mon[41530]: osdmap e39: 6 total, 6 up, 6 in 2024-08-17T17:05:09.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:09 smithi053 ceph-mon[34699]: pgmap v127: 1 pgs: 1 remapped+peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-08-17T17:05:09.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:09 smithi053 ceph-mon[34699]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-08-17T17:05:09.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:09 smithi053 ceph-mon[34699]: Detected new or changed devices on smithi053 2024-08-17T17:05:09.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:09 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:09.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:09 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:09.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:09 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:05:09.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:09 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:05:09.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:09 smithi053 ceph-mon[34699]: Adjusting osd_memory_target on smithi053 to 8546M 2024-08-17T17:05:09.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:09 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:09.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:09 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:09.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:09 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:09.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:09 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:09.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:09 smithi053 ceph-mon[34699]: osdmap e40: 6 total, 6 up, 6 in 2024-08-17T17:05:10.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:09 smithi003 ceph-mon[30803]: pgmap v127: 1 pgs: 1 remapped+peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-08-17T17:05:10.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:09 smithi003 ceph-mon[30803]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-08-17T17:05:10.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:09 smithi003 ceph-mon[30803]: Detected new or changed devices on smithi053 2024-08-17T17:05:10.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:09 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:10.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:09 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:10.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:09 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:05:10.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:09 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:05:10.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:09 smithi003 ceph-mon[30803]: Adjusting osd_memory_target on smithi053 to 8546M 2024-08-17T17:05:10.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:09 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:10.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:09 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:10.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:09 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:10.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:09 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:10.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:09 smithi003 ceph-mon[30803]: osdmap e40: 6 total, 6 up, 6 in 2024-08-17T17:05:10.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:09 smithi003 ceph-mon[41530]: pgmap v127: 1 pgs: 1 remapped+peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-08-17T17:05:10.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:09 smithi003 ceph-mon[41530]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-08-17T17:05:10.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:09 smithi003 ceph-mon[41530]: Detected new or changed devices on smithi053 2024-08-17T17:05:10.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:09 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:10.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:09 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:10.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:09 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:05:10.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:09 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:05:10.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:09 smithi003 ceph-mon[41530]: Adjusting osd_memory_target on smithi053 to 8546M 2024-08-17T17:05:10.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:09 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:10.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:09 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:10.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:09 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:10.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:09 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:10.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:09 smithi003 ceph-mon[41530]: osdmap e40: 6 total, 6 up, 6 in 2024-08-17T17:05:10.634 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T17:05:10.654 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi053:vg_nvme/lv_2 2024-08-17T17:05:10.917 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:05:12.190 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:11 smithi053 ceph-mon[34699]: pgmap v130: 1 pgs: 1 remapped+peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-08-17T17:05:12.190 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:11 smithi053 ceph-mon[34699]: osdmap e41: 6 total, 6 up, 6 in 2024-08-17T17:05:12.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:11 smithi003 ceph-mon[41530]: pgmap v130: 1 pgs: 1 remapped+peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-08-17T17:05:12.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:11 smithi003 ceph-mon[41530]: osdmap e41: 6 total, 6 up, 6 in 2024-08-17T17:05:12.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:11 smithi003 ceph-mon[30803]: pgmap v130: 1 pgs: 1 remapped+peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-08-17T17:05:12.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:11 smithi003 ceph-mon[30803]: osdmap e41: 6 total, 6 up, 6 in 2024-08-17T17:05:13.190 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:12 smithi053 ceph-mon[34699]: osdmap e42: 6 total, 6 up, 6 in 2024-08-17T17:05:13.191 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:12 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:05:13.191 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:12 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:05:13.191 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:12 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:13.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:12 smithi003 ceph-mon[30803]: osdmap e42: 6 total, 6 up, 6 in 2024-08-17T17:05:13.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:12 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:05:13.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:12 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:05:13.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:12 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:13.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:12 smithi003 ceph-mon[41530]: osdmap e42: 6 total, 6 up, 6 in 2024-08-17T17:05:13.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:12 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:05:13.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:12 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:05:13.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:12 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:14.179 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:13 smithi053 ceph-mon[34699]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:05:14.179 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:13 smithi053 ceph-mon[34699]: pgmap v133: 1 pgs: 1 remapped+peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-08-17T17:05:14.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:13 smithi003 ceph-mon[30803]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:05:14.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:13 smithi003 ceph-mon[30803]: pgmap v133: 1 pgs: 1 remapped+peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-08-17T17:05:14.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:13 smithi003 ceph-mon[41530]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:05:14.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:13 smithi003 ceph-mon[41530]: pgmap v133: 1 pgs: 1 remapped+peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-08-17T17:05:15.174 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:14 smithi003 ceph-mon[30803]: from='client.? 172.21.15.53:0/4018920194' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62903bee-59eb-4eed-b64d-9c884de87409"}]: dispatch 2024-08-17T17:05:15.174 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:14 smithi003 ceph-mon[30803]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62903bee-59eb-4eed-b64d-9c884de87409"}]: dispatch 2024-08-17T17:05:15.175 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:14 smithi003 ceph-mon[30803]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62903bee-59eb-4eed-b64d-9c884de87409"}]': finished 2024-08-17T17:05:15.175 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:14 smithi003 ceph-mon[30803]: osdmap e43: 7 total, 6 up, 7 in 2024-08-17T17:05:15.175 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:14 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:05:15.175 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:14 smithi003 ceph-mon[41530]: from='client.? 172.21.15.53:0/4018920194' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62903bee-59eb-4eed-b64d-9c884de87409"}]: dispatch 2024-08-17T17:05:15.175 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:14 smithi003 ceph-mon[41530]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62903bee-59eb-4eed-b64d-9c884de87409"}]: dispatch 2024-08-17T17:05:15.175 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:14 smithi003 ceph-mon[41530]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62903bee-59eb-4eed-b64d-9c884de87409"}]': finished 2024-08-17T17:05:15.175 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:14 smithi003 ceph-mon[41530]: osdmap e43: 7 total, 6 up, 7 in 2024-08-17T17:05:15.175 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:14 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:05:15.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:14 smithi053 ceph-mon[34699]: from='client.? 172.21.15.53:0/4018920194' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62903bee-59eb-4eed-b64d-9c884de87409"}]: dispatch 2024-08-17T17:05:15.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:14 smithi053 ceph-mon[34699]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "62903bee-59eb-4eed-b64d-9c884de87409"}]: dispatch 2024-08-17T17:05:15.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:14 smithi053 ceph-mon[34699]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "62903bee-59eb-4eed-b64d-9c884de87409"}]': finished 2024-08-17T17:05:15.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:14 smithi053 ceph-mon[34699]: osdmap e43: 7 total, 6 up, 7 in 2024-08-17T17:05:15.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:14 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:05:16.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:15 smithi053 ceph-mon[34699]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-08-17T17:05:16.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:15 smithi053 ceph-mon[34699]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-08-17T17:05:16.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:15 smithi053 ceph-mon[34699]: Cluster is now healthy 2024-08-17T17:05:16.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:15 smithi053 ceph-mon[34699]: from='client.? 172.21.15.53:0/2511136060' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:05:16.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:15 smithi003 ceph-mon[30803]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-08-17T17:05:16.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:15 smithi003 ceph-mon[30803]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-08-17T17:05:16.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:15 smithi003 ceph-mon[30803]: Cluster is now healthy 2024-08-17T17:05:16.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:15 smithi003 ceph-mon[30803]: from='client.? 172.21.15.53:0/2511136060' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:05:16.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:15 smithi003 ceph-mon[41530]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-08-17T17:05:16.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:15 smithi003 ceph-mon[41530]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-08-17T17:05:16.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:15 smithi003 ceph-mon[41530]: Cluster is now healthy 2024-08-17T17:05:16.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:15 smithi003 ceph-mon[41530]: from='client.? 172.21.15.53:0/2511136060' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:05:18.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:17 smithi053 ceph-mon[34699]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-08-17T17:05:18.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:17 smithi003 ceph-mon[30803]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-08-17T17:05:18.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:17 smithi003 ceph-mon[41530]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-08-17T17:05:20.115 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:19 smithi053 ceph-mon[34699]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-08-17T17:05:20.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:19 smithi003 ceph-mon[30803]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-08-17T17:05:20.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:19 smithi003 ceph-mon[41530]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-08-17T17:05:21.183 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:20 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-08-17T17:05:21.183 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:20 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:21.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:20 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-08-17T17:05:21.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:20 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:21.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:20 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-08-17T17:05:21.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:20 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:21.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:21 smithi053 ceph-mon[34699]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-08-17T17:05:21.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:21 smithi053 ceph-mon[34699]: Deploying daemon osd.6 on smithi053 2024-08-17T17:05:22.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:21 smithi003 ceph-mon[30803]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-08-17T17:05:22.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:21 smithi003 ceph-mon[30803]: Deploying daemon osd.6 on smithi053 2024-08-17T17:05:22.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:21 smithi003 ceph-mon[41530]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-08-17T17:05:22.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:21 smithi003 ceph-mon[41530]: Deploying daemon osd.6 on smithi053 2024-08-17T17:05:24.057 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:23 smithi053 ceph-mon[34699]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-08-17T17:05:24.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:23 smithi003 ceph-mon[30803]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-08-17T17:05:24.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:23 smithi003 ceph-mon[41530]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-08-17T17:05:26.125 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:25 smithi053 ceph-mon[34699]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-08-17T17:05:26.125 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:25 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:26.125 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:25 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:26.125 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:25 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:26.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:25 smithi003 ceph-mon[30803]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-08-17T17:05:26.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:25 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:26.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:25 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:26.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:25 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:26.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:25 smithi003 ceph-mon[41530]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-08-17T17:05:26.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:25 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:26.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:25 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:26.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:25 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:28.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:27 smithi053 ceph-mon[34699]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-08-17T17:05:28.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:27 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:28.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:27 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:28.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:27 smithi003 ceph-mon[41530]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-08-17T17:05:28.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:27 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:28.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:27 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:28.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:27 smithi003 ceph-mon[30803]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-08-17T17:05:28.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:27 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:28.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:27 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:28.537 INFO:teuthology.orchestra.run.smithi053.stdout:Created osd(s) 6 on host 'smithi053' 2024-08-17T17:05:29.142 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:28 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:29.142 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:28 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:29.142 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:28 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:29.142 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:28 smithi053 ceph-mon[34699]: from='osd.6 [v2:172.21.15.53:6816/3729099668,v1:172.21.15.53:6817/3729099668]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-08-17T17:05:29.142 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:28 smithi053 ceph-mon[34699]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-08-17T17:05:29.142 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:28 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:29.143 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:28 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:29.143 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:28 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:29.167 DEBUG:teuthology.orchestra.run.smithi053:osd.6> sudo journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.6.service 2024-08-17T17:05:29.169 INFO:tasks.cephadm:Deploying osd.7 on smithi053 with /dev/vg_nvme/lv_1... 2024-08-17T17:05:29.169 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-08-17T17:05:29.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:28 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:29.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:28 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:29.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:28 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:29.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:28 smithi003 ceph-mon[30803]: from='osd.6 [v2:172.21.15.53:6816/3729099668,v1:172.21.15.53:6817/3729099668]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-08-17T17:05:29.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:28 smithi003 ceph-mon[30803]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-08-17T17:05:29.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:28 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:29.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:28 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:29.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:28 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:29.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:28 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:29.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:28 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:29.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:28 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:29.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:28 smithi003 ceph-mon[41530]: from='osd.6 [v2:172.21.15.53:6816/3729099668,v1:172.21.15.53:6817/3729099668]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-08-17T17:05:29.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:28 smithi003 ceph-mon[41530]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-08-17T17:05:29.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:28 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:29.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:28 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:29.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:28 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:30.058 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:05:30.146 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:29 smithi053 ceph-mon[34699]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-08-17T17:05:30.147 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:29 smithi053 ceph-mon[34699]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-08-17T17:05:30.147 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:29 smithi053 ceph-mon[34699]: osdmap e44: 7 total, 6 up, 7 in 2024-08-17T17:05:30.147 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:29 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:05:30.147 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:29 smithi053 ceph-mon[34699]: from='osd.6 [v2:172.21.15.53:6816/3729099668,v1:172.21.15.53:6817/3729099668]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-08-17T17:05:30.147 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:29 smithi053 ceph-mon[34699]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-08-17T17:05:30.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:29 smithi003 ceph-mon[30803]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-08-17T17:05:30.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:29 smithi003 ceph-mon[30803]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-08-17T17:05:30.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:29 smithi003 ceph-mon[30803]: osdmap e44: 7 total, 6 up, 7 in 2024-08-17T17:05:30.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:29 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:05:30.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:29 smithi003 ceph-mon[30803]: from='osd.6 [v2:172.21.15.53:6816/3729099668,v1:172.21.15.53:6817/3729099668]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-08-17T17:05:30.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:29 smithi003 ceph-mon[30803]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-08-17T17:05:30.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:29 smithi003 ceph-mon[41530]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-08-17T17:05:30.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:29 smithi003 ceph-mon[41530]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-08-17T17:05:30.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:29 smithi003 ceph-mon[41530]: osdmap e44: 7 total, 6 up, 7 in 2024-08-17T17:05:30.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:29 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:05:30.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:29 smithi003 ceph-mon[41530]: from='osd.6 [v2:172.21.15.53:6816/3729099668,v1:172.21.15.53:6817/3729099668]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-08-17T17:05:30.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:29 smithi003 ceph-mon[41530]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-08-17T17:05:30.449 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:05:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:05:30.308+0000 7fc96ee44640 -1 osd.6 0 waiting for initial osdmap 2024-08-17T17:05:30.449 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:05:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:05:30.314+0000 7fc96a47c640 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-17T17:05:31.422 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:31 smithi053 ceph-mon[34699]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-08-17T17:05:31.422 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:31 smithi053 ceph-mon[34699]: osdmap e45: 7 total, 6 up, 7 in 2024-08-17T17:05:31.423 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:31 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:05:31.423 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:31 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:05:31.423 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:31 smithi053 ceph-mon[34699]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-08-17T17:05:31.423 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:31 smithi053 ceph-mon[34699]: osd.6 [v2:172.21.15.53:6816/3729099668,v1:172.21.15.53:6817/3729099668] boot 2024-08-17T17:05:31.423 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:31 smithi053 ceph-mon[34699]: osdmap e46: 7 total, 7 up, 7 in 2024-08-17T17:05:31.423 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:31 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:05:31.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:31 smithi003 ceph-mon[41530]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-08-17T17:05:31.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:31 smithi003 ceph-mon[41530]: osdmap e45: 7 total, 6 up, 7 in 2024-08-17T17:05:31.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:31 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:05:31.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:31 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:05:31.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:31 smithi003 ceph-mon[41530]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-08-17T17:05:31.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:31 smithi003 ceph-mon[41530]: osd.6 [v2:172.21.15.53:6816/3729099668,v1:172.21.15.53:6817/3729099668] boot 2024-08-17T17:05:31.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:31 smithi003 ceph-mon[41530]: osdmap e46: 7 total, 7 up, 7 in 2024-08-17T17:05:31.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:31 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:05:31.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:31 smithi003 ceph-mon[30803]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-08-17T17:05:31.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:31 smithi003 ceph-mon[30803]: osdmap e45: 7 total, 6 up, 7 in 2024-08-17T17:05:31.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:31 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:05:31.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:31 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:05:31.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:31 smithi003 ceph-mon[30803]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-08-17T17:05:31.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:31 smithi003 ceph-mon[30803]: osd.6 [v2:172.21.15.53:6816/3729099668,v1:172.21.15.53:6817/3729099668] boot 2024-08-17T17:05:31.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:31 smithi003 ceph-mon[30803]: osdmap e46: 7 total, 7 up, 7 in 2024-08-17T17:05:31.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:31 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:05:32.702 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:32 smithi053 ceph-mon[34699]: purged_snaps scrub starts 2024-08-17T17:05:32.702 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:32 smithi053 ceph-mon[34699]: purged_snaps scrub ok 2024-08-17T17:05:32.702 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:32 smithi053 ceph-mon[34699]: osdmap e47: 7 total, 7 up, 7 in 2024-08-17T17:05:32.708 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-08-17T17:05:32.730 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph orch daemon add osd smithi053:vg_nvme/lv_1 2024-08-17T17:05:32.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:32 smithi003 ceph-mon[30803]: purged_snaps scrub starts 2024-08-17T17:05:32.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:32 smithi003 ceph-mon[30803]: purged_snaps scrub ok 2024-08-17T17:05:32.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:32 smithi003 ceph-mon[30803]: osdmap e47: 7 total, 7 up, 7 in 2024-08-17T17:05:32.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:32 smithi003 ceph-mon[41530]: purged_snaps scrub starts 2024-08-17T17:05:32.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:32 smithi003 ceph-mon[41530]: purged_snaps scrub ok 2024-08-17T17:05:32.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:32 smithi003 ceph-mon[41530]: osdmap e47: 7 total, 7 up, 7 in 2024-08-17T17:05:33.255 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:05:33.619 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:33 smithi053 ceph-mon[34699]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-08-17T17:05:33.619 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:33 smithi053 ceph-mon[34699]: osdmap e48: 7 total, 7 up, 7 in 2024-08-17T17:05:33.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:33 smithi003 ceph-mon[30803]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-08-17T17:05:33.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:33 smithi003 ceph-mon[30803]: osdmap e48: 7 total, 7 up, 7 in 2024-08-17T17:05:33.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:33 smithi003 ceph-mon[41530]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-08-17T17:05:33.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:33 smithi003 ceph-mon[41530]: osdmap e48: 7 total, 7 up, 7 in 2024-08-17T17:05:35.591 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:35 smithi053 ceph-mon[34699]: osdmap e49: 7 total, 7 up, 7 in 2024-08-17T17:05:35.592 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:35 smithi053 ceph-mon[34699]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-08-17T17:05:35.592 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:35 smithi053 ceph-mon[34699]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:05:35.592 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:35 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:05:35.592 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:35 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:05:35.592 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:35 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:35.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:35 smithi003 ceph-mon[30803]: osdmap e49: 7 total, 7 up, 7 in 2024-08-17T17:05:35.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:35 smithi003 ceph-mon[30803]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-08-17T17:05:35.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:35 smithi003 ceph-mon[30803]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:05:35.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:35 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:05:35.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:35 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:05:35.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:35 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:35.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:35 smithi003 ceph-mon[41530]: osdmap e49: 7 total, 7 up, 7 in 2024-08-17T17:05:35.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:35 smithi003 ceph-mon[41530]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-08-17T17:05:35.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:35 smithi003 ceph-mon[41530]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi053:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:05:35.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:35 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-08-17T17:05:35.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:35 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-08-17T17:05:35.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:35 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:36.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:36 smithi053 ceph-mon[34699]: osdmap e50: 7 total, 7 up, 7 in 2024-08-17T17:05:36.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:36 smithi053 ceph-mon[34699]: Detected new or changed devices on smithi053 2024-08-17T17:05:36.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:36 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:36.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:36 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:36.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:36 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:05:36.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:36 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:05:36.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:36 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:05:36.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:36 smithi053 ceph-mon[34699]: Adjusting osd_memory_target on smithi053 to 5697M 2024-08-17T17:05:36.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:36 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:36.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:36 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:36.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:36 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:36.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:36 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:36.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:36 smithi003 ceph-mon[30803]: osdmap e50: 7 total, 7 up, 7 in 2024-08-17T17:05:36.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:36 smithi003 ceph-mon[30803]: Detected new or changed devices on smithi053 2024-08-17T17:05:36.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:36 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:36.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:36 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:36.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:36 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:05:36.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:36 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:05:36.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:36 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:05:36.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:36 smithi003 ceph-mon[30803]: Adjusting osd_memory_target on smithi053 to 5697M 2024-08-17T17:05:36.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:36 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:36.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:36 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:36.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:36 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:36.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:36 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:36.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:36 smithi003 ceph-mon[41530]: osdmap e50: 7 total, 7 up, 7 in 2024-08-17T17:05:36.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:36 smithi003 ceph-mon[41530]: Detected new or changed devices on smithi053 2024-08-17T17:05:36.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:36 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:36.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:36 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:36.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:36 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:05:36.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:36 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:05:36.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:36 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:05:36.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:36 smithi003 ceph-mon[41530]: Adjusting osd_memory_target on smithi053 to 5697M 2024-08-17T17:05:36.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:36 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:36.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:36 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:36.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:36 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:36.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:36 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:37.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:37 smithi053 ceph-mon[34699]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-08-17T17:05:37.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:37 smithi053 ceph-mon[34699]: from='client.? 172.21.15.53:0/3967858069' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4794ff08-e5d9-49c9-bea5-e158681b4004"}]: dispatch 2024-08-17T17:05:37.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:37 smithi053 ceph-mon[34699]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4794ff08-e5d9-49c9-bea5-e158681b4004"}]: dispatch 2024-08-17T17:05:37.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:37 smithi053 ceph-mon[34699]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4794ff08-e5d9-49c9-bea5-e158681b4004"}]': finished 2024-08-17T17:05:37.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:37 smithi053 ceph-mon[34699]: osdmap e51: 8 total, 7 up, 8 in 2024-08-17T17:05:37.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:37 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:05:37.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:37 smithi003 ceph-mon[30803]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-08-17T17:05:37.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:37 smithi003 ceph-mon[30803]: from='client.? 172.21.15.53:0/3967858069' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4794ff08-e5d9-49c9-bea5-e158681b4004"}]: dispatch 2024-08-17T17:05:37.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:37 smithi003 ceph-mon[30803]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4794ff08-e5d9-49c9-bea5-e158681b4004"}]: dispatch 2024-08-17T17:05:37.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:37 smithi003 ceph-mon[30803]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4794ff08-e5d9-49c9-bea5-e158681b4004"}]': finished 2024-08-17T17:05:37.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:37 smithi003 ceph-mon[30803]: osdmap e51: 8 total, 7 up, 8 in 2024-08-17T17:05:37.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:37 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:05:37.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:37 smithi003 ceph-mon[41530]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-08-17T17:05:37.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:37 smithi003 ceph-mon[41530]: from='client.? 172.21.15.53:0/3967858069' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4794ff08-e5d9-49c9-bea5-e158681b4004"}]: dispatch 2024-08-17T17:05:37.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:37 smithi003 ceph-mon[41530]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4794ff08-e5d9-49c9-bea5-e158681b4004"}]: dispatch 2024-08-17T17:05:37.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:37 smithi003 ceph-mon[41530]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4794ff08-e5d9-49c9-bea5-e158681b4004"}]': finished 2024-08-17T17:05:37.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:37 smithi003 ceph-mon[41530]: osdmap e51: 8 total, 7 up, 8 in 2024-08-17T17:05:37.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:37 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:05:38.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:38 smithi053 ceph-mon[34699]: from='client.? 172.21.15.53:0/1050527419' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:05:38.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:38 smithi003 ceph-mon[30803]: from='client.? 172.21.15.53:0/1050527419' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:05:38.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:38 smithi003 ceph-mon[41530]: from='client.? 172.21.15.53:0/1050527419' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-08-17T17:05:39.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:39 smithi053 ceph-mon[34699]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-08-17T17:05:39.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:39 smithi003 ceph-mon[30803]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-08-17T17:05:39.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:39 smithi003 ceph-mon[41530]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-08-17T17:05:42.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:41 smithi003 ceph-mon[41530]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-08-17T17:05:42.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:41 smithi003 ceph-mon[30803]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-08-17T17:05:42.076 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:41 smithi053 ceph-mon[34699]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-08-17T17:05:43.899 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:43 smithi053 ceph-mon[34699]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-08-17T17:05:43.899 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:43 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-08-17T17:05:43.899 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:43 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:44.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:43 smithi003 ceph-mon[41530]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-08-17T17:05:44.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:43 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-08-17T17:05:44.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:43 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:44.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:43 smithi003 ceph-mon[30803]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-08-17T17:05:44.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:43 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-08-17T17:05:44.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:43 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:44.955 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:44 smithi053 ceph-mon[34699]: Deploying daemon osd.7 on smithi053 2024-08-17T17:05:45.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:44 smithi003 ceph-mon[30803]: Deploying daemon osd.7 on smithi053 2024-08-17T17:05:45.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:44 smithi003 ceph-mon[41530]: Deploying daemon osd.7 on smithi053 2024-08-17T17:05:45.946 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:45 smithi053 ceph-mon[34699]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-08-17T17:05:46.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:45 smithi003 ceph-mon[30803]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-08-17T17:05:46.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:45 smithi003 ceph-mon[41530]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-08-17T17:05:47.870 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:47 smithi053 ceph-mon[34699]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-08-17T17:05:48.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:47 smithi003 ceph-mon[30803]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-08-17T17:05:48.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:47 smithi003 ceph-mon[41530]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-08-17T17:05:48.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:48 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:48.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:48 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:48.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:48 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:49.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:48 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:49.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:48 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:49.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:48 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:49.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:48 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:49.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:48 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:49.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:48 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:50.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:50 smithi003 ceph-mon[30803]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-08-17T17:05:50.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:50 smithi003 ceph-mon[41530]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-08-17T17:05:50.345 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:50 smithi053 ceph-mon[34699]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-08-17T17:05:51.031 INFO:teuthology.orchestra.run.smithi053.stdout:Created osd(s) 7 on host 'smithi053' 2024-08-17T17:05:51.631 DEBUG:teuthology.orchestra.run.smithi053:osd.7> sudo journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.7.service 2024-08-17T17:05:51.633 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-08-17T17:05:51.634 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd stat -f json 2024-08-17T17:05:51.913 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:05:52.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:52 smithi003 ceph-mon[30803]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-08-17T17:05:52.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:52 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:52.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:52 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:52.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:52 smithi003 ceph-mon[30803]: from='osd.7 [v2:172.21.15.53:6824/832291503,v1:172.21.15.53:6825/832291503]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-08-17T17:05:52.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:52 smithi003 ceph-mon[30803]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-08-17T17:05:52.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:52 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:52.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:52 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:52.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:52 smithi003 ceph-mon[41530]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-08-17T17:05:52.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:52 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:52.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:52 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:52.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:52 smithi003 ceph-mon[41530]: from='osd.7 [v2:172.21.15.53:6824/832291503,v1:172.21.15.53:6825/832291503]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-08-17T17:05:52.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:52 smithi003 ceph-mon[41530]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-08-17T17:05:52.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:52 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:52.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:52 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:52.279 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:52 smithi053 ceph-mon[34699]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-08-17T17:05:52.279 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:52 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:52.279 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:52 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:52.279 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:52 smithi053 ceph-mon[34699]: from='osd.7 [v2:172.21.15.53:6824/832291503,v1:172.21.15.53:6825/832291503]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-08-17T17:05:52.279 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:52 smithi053 ceph-mon[34699]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-08-17T17:05:52.279 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:52 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:52.279 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:52 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:53.270 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:53 smithi053 ceph-mon[34699]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-08-17T17:05:53.270 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:53 smithi053 ceph-mon[34699]: osdmap e52: 8 total, 7 up, 8 in 2024-08-17T17:05:53.270 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:53 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:05:53.270 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:53 smithi053 ceph-mon[34699]: from='osd.7 [v2:172.21.15.53:6824/832291503,v1:172.21.15.53:6825/832291503]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-08-17T17:05:53.270 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:53 smithi053 ceph-mon[34699]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-08-17T17:05:53.270 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:05:53 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:05:53.039+0000 7f383284b640 -1 osd.7 0 waiting for initial osdmap 2024-08-17T17:05:53.270 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:05:53 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:05:53.049+0000 7f382d682640 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-17T17:05:53.301 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:53 smithi003 ceph-mon[30803]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-08-17T17:05:53.302 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:53 smithi003 ceph-mon[30803]: osdmap e52: 8 total, 7 up, 8 in 2024-08-17T17:05:53.302 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:53 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:05:53.302 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:53 smithi003 ceph-mon[30803]: from='osd.7 [v2:172.21.15.53:6824/832291503,v1:172.21.15.53:6825/832291503]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-08-17T17:05:53.302 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:53 smithi003 ceph-mon[30803]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-08-17T17:05:53.302 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:53 smithi003 ceph-mon[41530]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-08-17T17:05:53.302 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:53 smithi003 ceph-mon[41530]: osdmap e52: 8 total, 7 up, 8 in 2024-08-17T17:05:53.302 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:53 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:05:53.302 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:53 smithi003 ceph-mon[41530]: from='osd.7 [v2:172.21.15.53:6824/832291503,v1:172.21.15.53:6825/832291503]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-08-17T17:05:53.302 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:53 smithi003 ceph-mon[41530]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]: dispatch 2024-08-17T17:05:53.908 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:05:54.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:54 smithi003 ceph-mon[41530]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-08-17T17:05:54.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:54 smithi003 ceph-mon[41530]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-08-17T17:05:54.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:54 smithi003 ceph-mon[41530]: osdmap e53: 8 total, 7 up, 8 in 2024-08-17T17:05:54.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:54 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:05:54.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:54 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:05:54.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:54 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/3498663661' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-08-17T17:05:54.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:54 smithi003 ceph-mon[41530]: osd.7 [v2:172.21.15.53:6824/832291503,v1:172.21.15.53:6825/832291503] boot 2024-08-17T17:05:54.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:54 smithi003 ceph-mon[41530]: osdmap e54: 8 total, 8 up, 8 in 2024-08-17T17:05:54.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:54 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:05:54.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:54 smithi003 ceph-mon[30803]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-08-17T17:05:54.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:54 smithi003 ceph-mon[30803]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-08-17T17:05:54.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:54 smithi003 ceph-mon[30803]: osdmap e53: 8 total, 7 up, 8 in 2024-08-17T17:05:54.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:54 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:05:54.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:54 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:05:54.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:54 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/3498663661' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-08-17T17:05:54.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:54 smithi003 ceph-mon[30803]: osd.7 [v2:172.21.15.53:6824/832291503,v1:172.21.15.53:6825/832291503] boot 2024-08-17T17:05:54.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:54 smithi003 ceph-mon[30803]: osdmap e54: 8 total, 8 up, 8 in 2024-08-17T17:05:54.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:54 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:05:54.304 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:54 smithi053 ceph-mon[34699]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-08-17T17:05:54.304 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:54 smithi053 ceph-mon[34699]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi053", "root=default"]}]': finished 2024-08-17T17:05:54.305 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:54 smithi053 ceph-mon[34699]: osdmap e53: 8 total, 7 up, 8 in 2024-08-17T17:05:54.305 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:54 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:05:54.305 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:54 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:05:54.305 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:54 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/3498663661' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-08-17T17:05:54.305 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:54 smithi053 ceph-mon[34699]: osd.7 [v2:172.21.15.53:6824/832291503,v1:172.21.15.53:6825/832291503] boot 2024-08-17T17:05:54.305 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:54 smithi053 ceph-mon[34699]: osdmap e54: 8 total, 8 up, 8 in 2024-08-17T17:05:54.305 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:54 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:05:54.556 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":53,"num_osds":8,"num_up_osds":7,"osd_up_since":1723914331,"num_in_osds":8,"osd_in_since":1723914337,"num_remapped_pgs":0} 2024-08-17T17:05:55.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[41530]: purged_snaps scrub starts 2024-08-17T17:05:55.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[41530]: purged_snaps scrub ok 2024-08-17T17:05:55.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:55.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:55.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:05:55.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:05:55.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:05:55.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:05:55.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:55.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:55.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:55.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:55.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:55.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:55.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:55.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:55.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[41530]: osdmap e55: 8 total, 8 up, 8 in 2024-08-17T17:05:55.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[30803]: purged_snaps scrub starts 2024-08-17T17:05:55.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[30803]: purged_snaps scrub ok 2024-08-17T17:05:55.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:55.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:55.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:05:55.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:05:55.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:05:55.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:05:55.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:55.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:55.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:55.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:55.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:55.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:55.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:55.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:55.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:55 smithi003 ceph-mon[30803]: osdmap e55: 8 total, 8 up, 8 in 2024-08-17T17:05:55.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:55 smithi053 ceph-mon[34699]: purged_snaps scrub starts 2024-08-17T17:05:55.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:55 smithi053 ceph-mon[34699]: purged_snaps scrub ok 2024-08-17T17:05:55.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:55 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:55.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:55 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:55.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:55 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:05:55.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:55 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:05:55.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:55 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:05:55.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:55 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:05:55.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:55 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:55.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:55 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:55.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:55 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:55.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:55 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:55.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:55 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:05:55.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:55 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:05:55.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:55 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:05:55.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:55 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:05:55.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:55 smithi053 ceph-mon[34699]: osdmap e55: 8 total, 8 up, 8 in 2024-08-17T17:05:55.558 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd stat -f json 2024-08-17T17:05:55.824 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:05:56.087 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:56 smithi003 ceph-mon[30803]: Detected new or changed devices on smithi053 2024-08-17T17:05:56.087 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:56 smithi003 ceph-mon[30803]: Adjusting osd_memory_target on smithi053 to 4273M 2024-08-17T17:05:56.087 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:56 smithi003 ceph-mon[30803]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:05:56.088 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:56 smithi003 ceph-mon[41530]: Detected new or changed devices on smithi053 2024-08-17T17:05:56.088 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:56 smithi003 ceph-mon[41530]: Adjusting osd_memory_target on smithi053 to 4273M 2024-08-17T17:05:56.088 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:56 smithi003 ceph-mon[41530]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:05:56.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:56 smithi053 ceph-mon[34699]: Detected new or changed devices on smithi053 2024-08-17T17:05:56.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:56 smithi053 ceph-mon[34699]: Adjusting osd_memory_target on smithi053 to 4273M 2024-08-17T17:05:56.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:56 smithi053 ceph-mon[34699]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:05:57.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:57 smithi003 ceph-mon[30803]: osdmap e56: 8 total, 8 up, 8 in 2024-08-17T17:05:57.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:57 smithi003 ceph-mon[30803]: osdmap e57: 8 total, 8 up, 8 in 2024-08-17T17:05:57.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:57 smithi003 ceph-mon[41530]: osdmap e56: 8 total, 8 up, 8 in 2024-08-17T17:05:57.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:57 smithi003 ceph-mon[41530]: osdmap e57: 8 total, 8 up, 8 in 2024-08-17T17:05:57.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:57 smithi053 ceph-mon[34699]: osdmap e56: 8 total, 8 up, 8 in 2024-08-17T17:05:57.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:57 smithi053 ceph-mon[34699]: osdmap e57: 8 total, 8 up, 8 in 2024-08-17T17:05:57.670 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:05:58.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:58 smithi053 ceph-mon[34699]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:05:58.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:58 smithi053 ceph-mon[34699]: osdmap e58: 8 total, 8 up, 8 in 2024-08-17T17:05:58.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:05:58 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/1547396307' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-08-17T17:05:58.490 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:58 smithi003 ceph-mon[41530]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:05:58.491 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:58 smithi003 ceph-mon[41530]: osdmap e58: 8 total, 8 up, 8 in 2024-08-17T17:05:58.491 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:05:58 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/1547396307' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-08-17T17:05:58.491 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:58 smithi003 ceph-mon[30803]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:05:58.491 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:58 smithi003 ceph-mon[30803]: osdmap e58: 8 total, 8 up, 8 in 2024-08-17T17:05:58.491 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:05:58 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/1547396307' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-08-17T17:05:58.522 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":58,"num_osds":8,"num_up_osds":8,"osd_up_since":1723914354,"num_in_osds":8,"osd_in_since":1723914337,"num_remapped_pgs":0} 2024-08-17T17:05:58.522 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-17T17:05:58.794 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:06:00.303 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:00 smithi003 ceph-mon[41530]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:00.303 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:00 smithi003 ceph-mon[30803]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:00.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:00 smithi053 ceph-mon[34699]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:00.657 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:06:00.657 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":58,"fsid":"2b5b662e-5cba-11ef-bccf-c7b262605968","created":"2024-08-17T17:00:56.372943+0000","modified":"2024-08-17T17:05:57.165191+0000","last_up_change":"2024-08-17T17:05:54.031028+0000","last_in_change":"2024-08-17T17:05:37.289378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-17T17:04:01.140951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"7f9a7d76-f0ec-49fc-9540-2606e94758ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":288807015},{"type":"v1","addr":"172.21.15.3:6803","nonce":288807015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":288807015},{"type":"v1","addr":"172.21.15.3:6805","nonce":288807015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":288807015},{"type":"v1","addr":"172.21.15.3:6809","nonce":288807015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":288807015},{"type":"v1","addr":"172.21.15.3:6807","nonce":288807015}]},"public_addr":"172.21.15.3:6803/288807015","cluster_addr":"172.21.15.3:6805/288807015","heartbeat_back_addr":"172.21.15.3:6809/288807015","heartbeat_front_addr":"172.21.15.3:6807/288807015","state":["exists","up"]},{"osd":1,"uuid":"316b5c4e-3417-4b93-bbbb-c6ed6dd21b0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2183964597},{"type":"v1","addr":"172.21.15.3:6811","nonce":2183964597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2183964597},{"type":"v1","addr":"172.21.15.3:6813","nonce":2183964597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":2183964597},{"type":"v1","addr":"172.21.15.3:6817","nonce":2183964597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2183964597},{"type":"v1","addr":"172.21.15.3:6815","nonce":2183964597}]},"public_addr":"172.21.15.3:6811/2183964597","cluster_addr":"172.21.15.3:6813/2183964597","heartbeat_back_addr":"172.21.15.3:6817/2183964597","heartbeat_front_addr":"172.21.15.3:6815/2183964597","state":["exists","up"]},{"osd":2,"uuid":"88af697d-38db-4c51-bfff-d0a776315596","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1747862993},{"type":"v1","addr":"172.21.15.3:6819","nonce":1747862993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1747862993},{"type":"v1","addr":"172.21.15.3:6821","nonce":1747862993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1747862993},{"type":"v1","addr":"172.21.15.3:6825","nonce":1747862993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1747862993},{"type":"v1","addr":"172.21.15.3:6823","nonce":1747862993}]},"public_addr":"172.21.15.3:6819/1747862993","cluster_addr":"172.21.15.3:6821/1747862993","heartbeat_back_addr":"172.21.15.3:6825/1747862993","heartbeat_front_addr":"172.21.15.3:6823/1747862993","state":["exists","up"]},{"osd":3,"uuid":"8e20cf42-6a13-4ad5-9f01-203a30c692d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2572629546},{"type":"v1","addr":"172.21.15.3:6827","nonce":2572629546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2572629546},{"type":"v1","addr":"172.21.15.3:6829","nonce":2572629546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2572629546},{"type":"v1","addr":"172.21.15.3:6833","nonce":2572629546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2572629546},{"type":"v1","addr":"172.21.15.3:6831","nonce":2572629546}]},"public_addr":"172.21.15.3:6827/2572629546","cluster_addr":"172.21.15.3:6829/2572629546","heartbeat_back_addr":"172.21.15.3:6833/2572629546","heartbeat_front_addr":"172.21.15.3:6831/2572629546","state":["exists","up"]},{"osd":4,"uuid":"2e8eabd7-8258-4b78-bb21-1a7291e0f1d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6801","nonce":1387588062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6803","nonce":1387588062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6807","nonce":1387588062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6805","nonce":1387588062}]},"public_addr":"172.21.15.53:6801/1387588062","cluster_addr":"172.21.15.53:6803/1387588062","heartbeat_back_addr":"172.21.15.53:6807/1387588062","heartbeat_front_addr":"172.21.15.53:6805/1387588062","state":["exists","up"]},{"osd":5,"uuid":"738f10f7-ff0d-4392-a818-eb23c9f65a63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6809","nonce":1263400356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6811","nonce":1263400356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6815","nonce":1263400356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6813","nonce":1263400356}]},"public_addr":"172.21.15.53:6809/1263400356","cluster_addr":"172.21.15.53:6811/1263400356","heartbeat_back_addr":"172.21.15.53:6815/1263400356","heartbeat_front_addr":"172.21.15.53:6813/1263400356","state":["exists","up"]},{"osd":6,"uuid":"62903bee-59eb-4eed-b64d-9c884de87409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6817","nonce":3729099668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6819","nonce":3729099668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6823","nonce":3729099668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6821","nonce":3729099668}]},"public_addr":"172.21.15.53:6817/3729099668","cluster_addr":"172.21.15.53:6819/3729099668","heartbeat_back_addr":"172.21.15.53:6823/3729099668","heartbeat_front_addr":"172.21.15.53:6821/3729099668","state":["exists","up"]},{"osd":7,"uuid":"4794ff08-e5d9-49c9-bea5-e158681b4004","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6825","nonce":832291503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6827","nonce":832291503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6831","nonce":832291503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6829","nonce":832291503}]},"public_addr":"172.21.15.53:6825/832291503","cluster_addr":"172.21.15.53:6827/832291503","heartbeat_back_addr":"172.21.15.53:6831/832291503","heartbeat_front_addr":"172.21.15.53:6829/832291503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:03:10.759301+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:03:33.981212+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:03:58.516966+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:04:22.965963+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:04:43.836760+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:06.691915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:29.472293+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:52.166848+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/2187177466":"2024-08-18T17:01:50.631242+0000","172.21.15.3:6800/2187177466":"2024-08-18T17:01:50.631242+0000","172.21.15.3:0/3985128953":"2024-08-18T17:01:50.631242+0000","172.21.15.3:0/3817753887":"2024-08-18T17:01:50.631242+0000","172.21.15.3:6800/160535497":"2024-08-18T17:01:27.442443+0000","172.21.15.3:0/1999667208":"2024-08-18T17:01:50.631242+0000","172.21.15.3:0/4194950757":"2024-08-18T17:01:27.442443+0000","172.21.15.3:6801/160535497":"2024-08-18T17:01:27.442443+0000","172.21.15.3:0/1476294447":"2024-08-18T17:01:27.442443+0000","172.21.15.3:0/1668133120":"2024-08-18T17:01:27.442443+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-17T17:06:01.296 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:01 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/1013600522' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-17T17:06:01.297 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:01 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/1013600522' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-17T17:06:01.325 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-17T17:04:01.140951+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-08-17T17:06:01.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-08-17T17:06:01.367 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:01 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/1013600522' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-17T17:06:01.591 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:06:02.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:02 smithi003 ceph-mon[30803]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-08-17T17:06:02.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:02 smithi003 ceph-mon[41530]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-08-17T17:06:02.385 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:02 smithi053 ceph-mon[34699]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-08-17T17:06:03.365 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-08-17T17:06:04.076 INFO:tasks.cephadm:Adding prometheus.a on smithi053 2024-08-17T17:06:04.076 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph orch apply prometheus '1;smithi053=a' 2024-08-17T17:06:04.346 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:06:04.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:04 smithi053 ceph-mon[34699]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-08-17T17:06:04.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:04 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/1664092654' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-08-17T17:06:04.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:04 smithi003 ceph-mon[30803]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-08-17T17:06:04.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:04 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/1664092654' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-08-17T17:06:04.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:04 smithi003 ceph-mon[41530]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-08-17T17:06:04.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:04 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/1664092654' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-08-17T17:06:06.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:06 smithi053 ceph-mon[34699]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-08-17T17:06:06.230 INFO:teuthology.orchestra.run.smithi053.stdout:Scheduled prometheus update... 2024-08-17T17:06:06.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:06 smithi003 ceph-mon[41530]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-08-17T17:06:06.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:06 smithi003 ceph-mon[30803]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-08-17T17:06:06.870 DEBUG:teuthology.orchestra.run.smithi053:prometheus.a> sudo journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@prometheus.a.service 2024-08-17T17:06:06.873 INFO:tasks.cephadm:Adding node-exporter.a on smithi003 2024-08-17T17:06:06.873 INFO:tasks.cephadm:Adding node-exporter.b on smithi053 2024-08-17T17:06:06.873 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph orch apply node-exporter '2;smithi003=a;smithi053=b' 2024-08-17T17:06:07.199 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:06:07.522 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:07 smithi053 ceph-mon[34699]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi053=a", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:06:07.522 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:07 smithi053 ceph-mon[34699]: Saving service prometheus spec with placement smithi053=a;count:1 2024-08-17T17:06:07.522 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:07 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:06:07.522 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:07 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:06:07.522 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:07 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:07.522 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:07 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:06:07.522 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:07 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:06:07.523 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:07 smithi053 ceph-mon[34699]: Deploying daemon prometheus.a on smithi053 2024-08-17T17:06:07.523 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:07 smithi053 ceph-mon[34699]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-08-17T17:06:07.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:07 smithi003 ceph-mon[41530]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi053=a", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:06:07.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:07 smithi003 ceph-mon[41530]: Saving service prometheus spec with placement smithi053=a;count:1 2024-08-17T17:06:07.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:07 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:06:07.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:07 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:06:07.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:07 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:07.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:07 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:06:07.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:07 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:06:07.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:07 smithi003 ceph-mon[41530]: Deploying daemon prometheus.a on smithi053 2024-08-17T17:06:07.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:07 smithi003 ceph-mon[41530]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-08-17T17:06:07.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:07 smithi003 ceph-mon[30803]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi053=a", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:06:07.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:07 smithi003 ceph-mon[30803]: Saving service prometheus spec with placement smithi053=a;count:1 2024-08-17T17:06:07.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:07 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:06:07.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:07 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:06:07.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:07 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:07.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:07 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:06:07.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:07 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:06:07.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:07 smithi003 ceph-mon[30803]: Deploying daemon prometheus.a on smithi053 2024-08-17T17:06:07.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:07 smithi003 ceph-mon[30803]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-08-17T17:06:09.240 INFO:teuthology.orchestra.run.smithi053.stdout:Scheduled node-exporter update... 2024-08-17T17:06:10.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:09 smithi003 ceph-mon[41530]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-08-17T17:06:10.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:09 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:06:10.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:09 smithi003 ceph-mon[30803]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-08-17T17:06:10.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:09 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:06:10.211 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:09 smithi053 ceph-mon[34699]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-08-17T17:06:10.211 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:09 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:06:10.242 DEBUG:teuthology.orchestra.run.smithi003:node-exporter.a> sudo journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@node-exporter.a.service 2024-08-17T17:06:10.245 DEBUG:teuthology.orchestra.run.smithi053:node-exporter.b> sudo journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@node-exporter.b.service 2024-08-17T17:06:10.247 INFO:tasks.cephadm:Adding alertmanager.a on smithi003 2024-08-17T17:06:10.248 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph orch apply alertmanager '1;smithi003=a' 2024-08-17T17:06:10.578 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:06:10.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:10 smithi053 ceph-mon[34699]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi003=a;smithi053=b", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:06:10.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:10 smithi053 ceph-mon[34699]: Saving service node-exporter spec with placement smithi003=a;smithi053=b;count:2 2024-08-17T17:06:11.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:10 smithi003 ceph-mon[41530]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi003=a;smithi053=b", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:06:11.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:10 smithi003 ceph-mon[41530]: Saving service node-exporter spec with placement smithi003=a;smithi053=b;count:2 2024-08-17T17:06:11.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:10 smithi003 ceph-mon[30803]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi003=a;smithi053=b", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:06:11.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:10 smithi003 ceph-mon[30803]: Saving service node-exporter spec with placement smithi003=a;smithi053=b;count:2 2024-08-17T17:06:11.884 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:11 smithi053 ceph-mon[34699]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-17T17:06:12.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:11 smithi003 ceph-mon[41530]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-17T17:06:12.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:11 smithi003 ceph-mon[30803]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-08-17T17:06:13.063 INFO:teuthology.orchestra.run.smithi053.stdout:Scheduled alertmanager update... 2024-08-17T17:06:14.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:14 smithi053 ceph-mon[34699]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:14.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:14 smithi053 ceph-mon[34699]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi003=a", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:06:14.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:14 smithi053 ceph-mon[34699]: Saving service alertmanager spec with placement smithi003=a;count:1 2024-08-17T17:06:14.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:14 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:06:14.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:14 smithi003 ceph-mon[41530]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:14.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:14 smithi003 ceph-mon[41530]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi003=a", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:06:14.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:14 smithi003 ceph-mon[41530]: Saving service alertmanager spec with placement smithi003=a;count:1 2024-08-17T17:06:14.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:14 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:06:14.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:14 smithi003 ceph-mon[30803]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:14.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:14 smithi003 ceph-mon[30803]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi003=a", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:06:14.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:14 smithi003 ceph-mon[30803]: Saving service alertmanager spec with placement smithi003=a;count:1 2024-08-17T17:06:14.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:14 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:06:14.628 DEBUG:teuthology.orchestra.run.smithi003:alertmanager.a> sudo journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@alertmanager.a.service 2024-08-17T17:06:14.631 INFO:tasks.cephadm:Adding grafana.a on smithi053 2024-08-17T17:06:14.632 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph orch apply grafana '1;smithi053=a' 2024-08-17T17:06:15.039 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:06:16.155 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:16 smithi053 ceph-mon[34699]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:16.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:16 smithi003 ceph-mon[30803]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:16.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:16 smithi003 ceph-mon[41530]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:18.330 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:18 smithi053 ceph-mon[34699]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:18.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:18 smithi003 ceph-mon[30803]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:18.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:18 smithi003 ceph-mon[41530]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:19.205 INFO:teuthology.orchestra.run.smithi053.stdout:Scheduled grafana update... 2024-08-17T17:06:19.958 DEBUG:teuthology.orchestra.run.smithi053:grafana.a> sudo journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@grafana.a.service 2024-08-17T17:06:19.961 INFO:tasks.cephadm:Setting up client nodes... 2024-08-17T17:06:19.961 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-08-17T17:06:20.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:20 smithi053 ceph-mon[34699]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:20.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:20 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:06:20.246 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:06:20.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:20 smithi003 ceph-mon[41530]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:20.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:20 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:06:20.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:20 smithi003 ceph-mon[30803]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:20.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:20 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:06:21.407 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:21 smithi003 ceph-mon[41530]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi053=a", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:06:21.407 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:21 smithi003 ceph-mon[41530]: Saving service grafana spec with placement smithi053=a;count:1 2024-08-17T17:06:21.407 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:21 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:06:21.408 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:21 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:06:21.408 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:21 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:06:21.408 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:21 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-08-17T17:06:21.408 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:21 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:06:21.409 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:21 smithi003 ceph-mon[30803]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi053=a", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:06:21.409 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:21 smithi003 ceph-mon[30803]: Saving service grafana spec with placement smithi053=a;count:1 2024-08-17T17:06:21.409 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:21 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:06:21.409 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:21 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:06:21.409 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:21 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:06:21.409 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:21 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-08-17T17:06:21.409 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:21 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:06:21.409 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:21 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: ignoring --setuser ceph since I am not root 2024-08-17T17:06:21.409 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:21 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: ignoring --setgroup ceph since I am not root 2024-08-17T17:06:21.409 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:21 smithi003 ceph-mgr[31034]: -- 172.21.15.3:0/3311367738 <== mon.0 v2:172.21.15.3:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f8721009c0 con 0x55f87210d000 2024-08-17T17:06:21.409 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:21 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:06:21.277+0000 7fcecd060ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-17T17:06:21.410 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:21 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:06:21.405+0000 7fcecd060ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-17T17:06:21.410 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:21 smithi053 ceph-mon[34699]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi053=a", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:06:21.410 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:21 smithi053 ceph-mon[34699]: Saving service grafana spec with placement smithi053=a;count:1 2024-08-17T17:06:21.410 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:21 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:06:21.410 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:21 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:06:21.410 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:21 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:06:21.410 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:21 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-08-17T17:06:21.410 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:21 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' 2024-08-17T17:06:21.411 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:21 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: ignoring --setuser ceph since I am not root 2024-08-17T17:06:21.411 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:21 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: ignoring --setgroup ceph since I am not root 2024-08-17T17:06:21.411 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:21 smithi053 ceph-mgr[36567]: -- 172.21.15.53:0/2488306302 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55a339b951e0 con 0x55a339af8800 2024-08-17T17:06:21.411 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:21 smithi053 ceph-mgr[36567]: -- 172.21.15.53:0/2488306302 <== mon.1 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a339aec9c0 con 0x55a339af8800 2024-08-17T17:06:21.411 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:21 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:06:21.282+0000 7f02d4acbec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-17T17:06:21.411 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:21 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:06:21.409+0000 7f02d4acbec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-17T17:06:21.716 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:21 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:06:21.477+0000 7f02d4acbec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-17T17:06:21.735 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:21 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:06:21.475+0000 7fcecd060ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-17T17:06:21.735 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:21 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:06:21.734+0000 7fcecd060ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-17T17:06:22.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:21 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:06:21.795+0000 7fcecd060ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-17T17:06:22.073 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:21 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:06:21.728+0000 7f02d4acbec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-17T17:06:22.073 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:21 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:06:21.788+0000 7f02d4acbec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-17T17:06:22.318 INFO:teuthology.orchestra.run.smithi003.stdout:[client.0] 2024-08-17T17:06:22.318 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQCO2MBm0c95EhAAxFkHnKN5UlsNGAXSPfM2QQ== 2024-08-17T17:06:22.329 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:22 smithi003 ceph-mon[30803]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:22.329 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:22 smithi003 ceph-mon[30803]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-08-17T17:06:22.329 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:22 smithi003 ceph-mon[30803]: mgrmap e16: y(active, since 4m), standbys: x 2024-08-17T17:06:22.329 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:22 smithi003 ceph-mon[41530]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:22.329 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:22 smithi003 ceph-mon[41530]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-08-17T17:06:22.329 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:22 smithi003 ceph-mon[41530]: mgrmap e16: y(active, since 4m), standbys: x 2024-08-17T17:06:22.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:22 smithi053 ceph-mon[34699]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:22.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:22 smithi053 ceph-mon[34699]: from='mgr.14150 172.21.15.3:0/2106519424' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-08-17T17:06:22.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:22 smithi053 ceph-mon[34699]: mgrmap e16: y(active, since 4m), standbys: x 2024-08-17T17:06:23.018 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:22 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:06:22.764+0000 7fcecd060ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-17T17:06:23.018 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:22 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:06:22.881+0000 7fcecd060ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-17T17:06:23.018 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:22 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:06:22.939+0000 7fcecd060ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-17T17:06:23.044 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-08-17T17:06:23.044 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-08-17T17:06:23.044 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-08-17T17:06:23.074 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:22 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:06:22.729+0000 7f02d4acbec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-17T17:06:23.074 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:22 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:06:22.844+0000 7f02d4acbec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-17T17:06:23.075 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:22 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:06:22.901+0000 7f02d4acbec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-17T17:06:23.084 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-08-17T17:06:23.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:23 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/201561794' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-17T17:06:23.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:23 smithi003 ceph-mon[41530]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-17T17:06:23.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:23 smithi003 ceph-mon[41530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-17T17:06:23.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:23 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/201561794' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-17T17:06:23.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:23 smithi003 ceph-mon[30803]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-17T17:06:23.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:23 smithi003 ceph-mon[30803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-17T17:06:23.273 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:23 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:06:23.140+0000 7fcecd060ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-17T17:06:23.273 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:23 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:06:23.203+0000 7fcecd060ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-17T17:06:23.273 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:23 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:06:23.271+0000 7fcecd060ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-17T17:06:23.356 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:06:23.373 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:23 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/201561794' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-17T17:06:23.373 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:23 smithi053 ceph-mon[34699]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-17T17:06:23.373 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:23 smithi053 ceph-mon[34699]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-17T17:06:23.373 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:23 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:06:23.098+0000 7f02d4acbec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-17T17:06:23.373 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:23 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:06:23.163+0000 7f02d4acbec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-17T17:06:23.373 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:23 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:06:23.232+0000 7f02d4acbec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-17T17:06:23.374 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:23 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:06:23.372+0000 7f02d4acbec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-17T17:06:23.523 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:23 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:06:23.399+0000 7fcecd060ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-17T17:06:23.966 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:23 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:06:23.814+0000 7f02d4acbec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-17T17:06:24.135 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:23 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:06:23.832+0000 7fcecd060ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-17T17:06:24.135 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:24 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:06:24.134+0000 7fcecd060ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-17T17:06:24.432 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:24 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:06:24.120+0000 7f02d4acbec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-17T17:06:24.432 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:24 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:06:24.189+0000 7f02d4acbec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-17T17:06:24.432 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:24 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:06:24.320+0000 7f02d4acbec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-17T17:06:24.523 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:24 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:06:24.202+0000 7fcecd060ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-17T17:06:24.523 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:24 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:06:24.333+0000 7fcecd060ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-17T17:06:24.716 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:24 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:06:24.594+0000 7f02d4acbec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-17T17:06:24.717 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:24 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:06:24.657+0000 7f02d4acbec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-17T17:06:24.852 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:24 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:06:24.599+0000 7fcecd060ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-17T17:06:24.853 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:24 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:06:24.661+0000 7fcecd060ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-17T17:06:25.006 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:24 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:06:24.850+0000 7f02d4acbec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-17T17:06:25.006 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:24 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:06:24.917+0000 7f02d4acbec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-17T17:06:25.006 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:25 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:06:25.004+0000 7f02d4acbec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-17T17:06:25.190 INFO:teuthology.orchestra.run.smithi053.stdout:[client.1] 2024-08-17T17:06:25.190 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCR2MBmY+4eCxAAPaE92G0IzlRuLGwev/lrdg== 2024-08-17T17:06:25.241 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:24 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:06:24.851+0000 7fcecd060ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-17T17:06:25.241 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:24 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:06:24.916+0000 7fcecd060ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-17T17:06:25.241 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:06:25.001+0000 7fcecd060ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-17T17:06:25.241 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:06:25.061+0000 7fcecd060ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-17T17:06:25.466 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:25 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:06:25.065+0000 7f02d4acbec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-17T17:06:25.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:25 smithi053 ceph-mon[34699]: from='client.? 172.21.15.53:0/2348047312' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-17T17:06:25.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:25 smithi053 ceph-mon[34699]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-17T17:06:25.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:25 smithi053 ceph-mon[34699]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-17T17:06:25.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:25 smithi003 ceph-mon[30803]: from='client.? 172.21.15.53:0/2348047312' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-17T17:06:25.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:25 smithi003 ceph-mon[30803]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-17T17:06:25.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:25 smithi003 ceph-mon[30803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-17T17:06:25.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:25 smithi003 ceph-mon[41530]: from='client.? 172.21.15.53:0/2348047312' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-17T17:06:25.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:25 smithi003 ceph-mon[41530]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-08-17T17:06:25.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:25 smithi003 ceph-mon[41530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-17T17:06:25.707 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-08-17T17:06:25.707 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-08-17T17:06:25.707 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-08-17T17:06:25.743 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:25 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:06:25.621+0000 7f02d4acbec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-17T17:06:25.747 INFO:tasks.cephadm:Creating RBD pool 2024-08-17T17:06:25.747 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd pool create rbd 8 2024-08-17T17:06:25.941 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:06:25.606+0000 7fcecd060ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-17T17:06:25.941 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:06:25.748+0000 7fcecd060ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-17T17:06:25.941 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:06:25] ENGINE Bus STARTING 2024-08-17T17:06:25.941 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: CherryPy Checker: 2024-08-17T17:06:25.941 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: The Application mounted at '' has an empty config. 2024-08-17T17:06:25.941 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:06:26.004 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:25 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:06:25.767+0000 7f02d4acbec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-17T17:06:26.004 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:25 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: [17/Aug/2024:17:06:25] ENGINE Bus STARTING 2024-08-17T17:06:26.004 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:25 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: CherryPy Checker: 2024-08-17T17:06:26.004 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:25 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: The Application mounted at '' has an empty config. 2024-08-17T17:06:26.004 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:25 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: 2024-08-17T17:06:26.004 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:25 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: [17/Aug/2024:17:06:25] ENGINE Serving on http://:::9283 2024-08-17T17:06:26.004 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:25 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: [17/Aug/2024:17:06:25] ENGINE Bus STARTED 2024-08-17T17:06:26.109 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:06:26.192 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:06:25] ENGINE Serving on http://:::9283 2024-08-17T17:06:26.192 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:06:25] ENGINE Bus STARTED 2024-08-17T17:06:26.312 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: Active manager daemon y restarted 2024-08-17T17:06:26.312 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: Activating manager daemon y 2024-08-17T17:06:26.312 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: osdmap e59: 8 total, 8 up, 8 in 2024-08-17T17:06:26.312 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: mgrmap e17: y(active, starting, since 0.00717659s), standbys: x 2024-08-17T17:06:26.312 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:06:26.312 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:06:26.313 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:06:26.313 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-17T17:06:26.313 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-17T17:06:26.313 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:06:26.313 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:06:26.313 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:06:26.313 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:06:26.313 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:06:26.313 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:06:26.313 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:06:26.313 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:06:26.313 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-17T17:06:26.313 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-17T17:06:26.314 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-17T17:06:26.314 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: Standby manager daemon x restarted 2024-08-17T17:06:26.314 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: Standby manager daemon x started 2024-08-17T17:06:26.314 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.53:0/2480831997' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:06:26.314 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: Manager daemon y is now available 2024-08-17T17:06:26.314 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.53:0/2480831997' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:06:26.314 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.53:0/2480831997' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:06:26.314 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.53:0/2480831997' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:06:26.314 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:06:26.314 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:26.314 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:06:26.314 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:06:26.314 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-08-17T17:06:26.472 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: Active manager daemon y restarted 2024-08-17T17:06:26.472 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: Activating manager daemon y 2024-08-17T17:06:26.472 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: osdmap e59: 8 total, 8 up, 8 in 2024-08-17T17:06:26.472 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: mgrmap e17: y(active, starting, since 0.00717659s), standbys: x 2024-08-17T17:06:26.472 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:06:26.472 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:06:26.472 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:06:26.472 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-17T17:06:26.472 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-17T17:06:26.472 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:06:26.473 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:06:26.473 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:06:26.473 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:06:26.473 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:06:26.473 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:06:26.473 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:06:26.473 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:06:26.473 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-17T17:06:26.473 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-17T17:06:26.473 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-17T17:06:26.473 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: Standby manager daemon x restarted 2024-08-17T17:06:26.473 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: Standby manager daemon x started 2024-08-17T17:06:26.473 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.53:0/2480831997' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:06:26.473 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: Manager daemon y is now available 2024-08-17T17:06:26.473 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.53:0/2480831997' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:06:26.474 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.53:0/2480831997' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:06:26.474 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.53:0/2480831997' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:06:26.474 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:06:26.474 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:26.474 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:06:26.474 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:06:26.474 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-08-17T17:06:26.474 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: Active manager daemon y restarted 2024-08-17T17:06:26.475 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: Activating manager daemon y 2024-08-17T17:06:26.475 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: osdmap e59: 8 total, 8 up, 8 in 2024-08-17T17:06:26.475 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: mgrmap e17: y(active, starting, since 0.00717659s), standbys: x 2024-08-17T17:06:26.475 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:06:26.475 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:06:26.475 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:06:26.475 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-17T17:06:26.475 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-17T17:06:26.475 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:06:26.475 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:06:26.475 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:06:26.476 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:06:26.476 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:06:26.476 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:06:26.476 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:06:26.476 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:06:26.476 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-17T17:06:26.476 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-17T17:06:26.476 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-17T17:06:26.476 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: Standby manager daemon x restarted 2024-08-17T17:06:26.476 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: Standby manager daemon x started 2024-08-17T17:06:26.476 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: from='mgr.? 172.21.15.53:0/2480831997' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:06:26.476 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: Manager daemon y is now available 2024-08-17T17:06:26.476 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: from='mgr.? 172.21.15.53:0/2480831997' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:06:26.476 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: from='mgr.? 172.21.15.53:0/2480831997' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:06:26.476 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: from='mgr.? 172.21.15.53:0/2480831997' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:06:26.476 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:06:26.477 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:26.477 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:06:26.477 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:06:26.477 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:26 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-08-17T17:06:26.717 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:06:26] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-08-17T17:06:27.837 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:27 smithi003 ceph-mon[41530]: mgrmap e18: y(active, since 1.01061s), standbys: x 2024-08-17T17:06:27.837 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:27 smithi003 ceph-mon[41530]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:27.837 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:27 smithi003 ceph-mon[30803]: mgrmap e18: y(active, since 1.01061s), standbys: x 2024-08-17T17:06:27.837 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:27 smithi003 ceph-mon[30803]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:28.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:27 smithi053 ceph-mon[34699]: mgrmap e18: y(active, since 1.01061s), standbys: x 2024-08-17T17:06:28.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:27 smithi053 ceph-mon[34699]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:28.524 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:06:28.334+0000 7fce6bdb3640 -1 log_channel(cephadm) log [ERR] : [17/Aug/2024:17:06:28] ENGINE Error in HTTPServer.serve 2024-08-17T17:06:28.524 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: Traceback (most recent call last): 2024-08-17T17:06:28.524 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:06:28.524 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: self._connections.run(self.expiration_interval) 2024-08-17T17:06:28.524 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:06:28.524 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: self._run(expiration_interval) 2024-08-17T17:06:28.524 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:06:28.524 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:06:28.525 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:06:28.525 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:06:28.525 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:06:28.525 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: s = self.context.wrap_socket( 2024-08-17T17:06:28.525 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:06:28.525 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: return self.sslsocket_class._create( 2024-08-17T17:06:28.525 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:06:28.525 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: self.do_handshake() 2024-08-17T17:06:28.525 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:06:28.525 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: self._sslobj.do_handshake() 2024-08-17T17:06:28.525 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:06:28.525 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:06:28.776 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'rbd' created 2024-08-17T17:06:29.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-mon[30803]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:29.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-mon[30803]: mgrmap e19: y(active, since 2s), standbys: x 2024-08-17T17:06:29.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/424253651' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-08-17T17:06:29.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-mon[30803]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-08-17T17:06:29.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:29.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:29.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:29.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:29.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-mon[41530]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:29.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-mon[41530]: mgrmap e19: y(active, since 2s), standbys: x 2024-08-17T17:06:29.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/424253651' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-08-17T17:06:29.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-mon[41530]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-08-17T17:06:29.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:29.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:29.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:29.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:28 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:29.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:28 smithi053 ceph-mon[34699]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:29.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:28 smithi053 ceph-mon[34699]: mgrmap e19: y(active, since 2s), standbys: x 2024-08-17T17:06:29.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:28 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/424253651' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-08-17T17:06:29.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:28 smithi053 ceph-mon[34699]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-08-17T17:06:29.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:28 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:29.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:28 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:29.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:28 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:29.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:28 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:29.465 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- rbd pool init rbd 2024-08-17T17:06:29.762 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:06:30.034 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:29 smithi003 ceph-mon[41530]: [17/Aug/2024:17:06:28] ENGINE Bus STARTING 2024-08-17T17:06:30.034 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:29 smithi003 ceph-mon[41530]: [17/Aug/2024:17:06:28] ENGINE Serving on https://172.21.15.3:7150 2024-08-17T17:06:30.035 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:29 smithi003 ceph-mon[41530]: [17/Aug/2024:17:06:28] ENGINE Bus STARTED 2024-08-17T17:06:30.035 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:29 smithi003 ceph-mon[41530]: [17/Aug/2024:17:06:28] ENGINE Error in HTTPServer.serve 2024-08-17T17:06:30.035 INFO:journalctl@ceph.mon.c.smithi003.stdout: Traceback (most recent call last): 2024-08-17T17:06:30.035 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:06:30.035 INFO:journalctl@ceph.mon.c.smithi003.stdout: self._connections.run(self.expiration_interval) 2024-08-17T17:06:30.035 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:06:30.035 INFO:journalctl@ceph.mon.c.smithi003.stdout: self._run(expiration_interval) 2024-08-17T17:06:30.035 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:06:30.035 INFO:journalctl@ceph.mon.c.smithi003.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:06:30.035 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:06:30.035 INFO:journalctl@ceph.mon.c.smithi003.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:06:30.035 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:06:30.035 INFO:journalctl@ceph.mon.c.smithi003.stdout: s = self.context.wrap_socket( 2024-08-17T17:06:30.035 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:06:30.035 INFO:journalctl@ceph.mon.c.smithi003.stdout: return self.sslsocket_class._create( 2024-08-17T17:06:30.036 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:06:30.036 INFO:journalctl@ceph.mon.c.smithi003.stdout: self.do_handshake() 2024-08-17T17:06:30.036 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:06:30.036 INFO:journalctl@ceph.mon.c.smithi003.stdout: self._sslobj.do_handshake() 2024-08-17T17:06:30.036 INFO:journalctl@ceph.mon.c.smithi003.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:06:30.036 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:29 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:30.036 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:29 smithi003 ceph-mon[41530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-08-17T17:06:30.036 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:29 smithi003 ceph-mon[41530]: osdmap e60: 8 total, 8 up, 8 in 2024-08-17T17:06:30.036 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:29 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:30.036 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:29 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:30.036 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:29 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:30.036 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:29 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:30.036 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:29 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:30.036 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:29 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:30.036 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:29 smithi003 ceph-mon[41530]: Adjusting osd_memory_target on smithi053 to 4017M 2024-08-17T17:06:30.037 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:29 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:30.037 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:29 smithi003 ceph-mon[30803]: [17/Aug/2024:17:06:28] ENGINE Bus STARTING 2024-08-17T17:06:30.037 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:29 smithi003 ceph-mon[30803]: [17/Aug/2024:17:06:28] ENGINE Serving on https://172.21.15.3:7150 2024-08-17T17:06:30.037 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:29 smithi003 ceph-mon[30803]: [17/Aug/2024:17:06:28] ENGINE Bus STARTED 2024-08-17T17:06:30.037 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:29 smithi003 ceph-mon[30803]: [17/Aug/2024:17:06:28] ENGINE Error in HTTPServer.serve 2024-08-17T17:06:30.037 INFO:journalctl@ceph.mon.a.smithi003.stdout: Traceback (most recent call last): 2024-08-17T17:06:30.037 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:06:30.037 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._connections.run(self.expiration_interval) 2024-08-17T17:06:30.037 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:06:30.038 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._run(expiration_interval) 2024-08-17T17:06:30.038 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:06:30.038 INFO:journalctl@ceph.mon.a.smithi003.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:06:30.038 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:06:30.038 INFO:journalctl@ceph.mon.a.smithi003.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:06:30.038 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:06:30.038 INFO:journalctl@ceph.mon.a.smithi003.stdout: s = self.context.wrap_socket( 2024-08-17T17:06:30.038 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:06:30.038 INFO:journalctl@ceph.mon.a.smithi003.stdout: return self.sslsocket_class._create( 2024-08-17T17:06:30.038 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:06:30.038 INFO:journalctl@ceph.mon.a.smithi003.stdout: self.do_handshake() 2024-08-17T17:06:30.038 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:06:30.038 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._sslobj.do_handshake() 2024-08-17T17:06:30.038 INFO:journalctl@ceph.mon.a.smithi003.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:06:30.039 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:29 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:30.039 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:29 smithi003 ceph-mon[30803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-08-17T17:06:30.039 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:29 smithi003 ceph-mon[30803]: osdmap e60: 8 total, 8 up, 8 in 2024-08-17T17:06:30.039 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:29 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:30.039 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:29 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:30.039 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:29 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:30.039 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:29 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:30.039 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:29 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:30.039 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:29 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:30.039 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:29 smithi003 ceph-mon[30803]: Adjusting osd_memory_target on smithi053 to 4017M 2024-08-17T17:06:30.039 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:29 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:30.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:29 smithi053 ceph-mon[34699]: [17/Aug/2024:17:06:28] ENGINE Bus STARTING 2024-08-17T17:06:30.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:29 smithi053 ceph-mon[34699]: [17/Aug/2024:17:06:28] ENGINE Serving on https://172.21.15.3:7150 2024-08-17T17:06:30.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:29 smithi053 ceph-mon[34699]: [17/Aug/2024:17:06:28] ENGINE Bus STARTED 2024-08-17T17:06:30.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:29 smithi053 ceph-mon[34699]: [17/Aug/2024:17:06:28] ENGINE Error in HTTPServer.serve 2024-08-17T17:06:30.217 INFO:journalctl@ceph.mon.b.smithi053.stdout: Traceback (most recent call last): 2024-08-17T17:06:30.217 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:06:30.217 INFO:journalctl@ceph.mon.b.smithi053.stdout: self._connections.run(self.expiration_interval) 2024-08-17T17:06:30.217 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:06:30.217 INFO:journalctl@ceph.mon.b.smithi053.stdout: self._run(expiration_interval) 2024-08-17T17:06:30.217 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:06:30.218 INFO:journalctl@ceph.mon.b.smithi053.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:06:30.218 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:06:30.218 INFO:journalctl@ceph.mon.b.smithi053.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:06:30.218 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:06:30.218 INFO:journalctl@ceph.mon.b.smithi053.stdout: s = self.context.wrap_socket( 2024-08-17T17:06:30.218 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:06:30.218 INFO:journalctl@ceph.mon.b.smithi053.stdout: return self.sslsocket_class._create( 2024-08-17T17:06:30.218 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:06:30.218 INFO:journalctl@ceph.mon.b.smithi053.stdout: self.do_handshake() 2024-08-17T17:06:30.218 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:06:30.218 INFO:journalctl@ceph.mon.b.smithi053.stdout: self._sslobj.do_handshake() 2024-08-17T17:06:30.218 INFO:journalctl@ceph.mon.b.smithi053.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:06:30.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:29 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:30.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:29 smithi053 ceph-mon[34699]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-08-17T17:06:30.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:29 smithi053 ceph-mon[34699]: osdmap e60: 8 total, 8 up, 8 in 2024-08-17T17:06:30.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:29 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:30.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:29 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:30.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:29 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:30.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:29 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:30.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:29 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:30.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:29 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:06:30.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:29 smithi053 ceph-mon[34699]: Adjusting osd_memory_target on smithi053 to 4017M 2024-08-17T17:06:30.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:29 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:31.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:30 smithi003 ceph-mon[41530]: pgmap v6: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:31.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:30 smithi003 ceph-mon[41530]: osdmap e61: 8 total, 8 up, 8 in 2024-08-17T17:06:31.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:30 smithi003 ceph-mon[41530]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:06:31.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:30 smithi003 ceph-mon[41530]: mgrmap e20: y(active, since 4s), standbys: x 2024-08-17T17:06:31.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:30 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:31.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:30 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:31.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:30 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:31.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:30 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:06:31.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:30 smithi003 ceph-mon[30803]: pgmap v6: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:31.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:30 smithi003 ceph-mon[30803]: osdmap e61: 8 total, 8 up, 8 in 2024-08-17T17:06:31.098 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:30 smithi003 ceph-mon[30803]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:06:31.098 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:30 smithi003 ceph-mon[30803]: mgrmap e20: y(active, since 4s), standbys: x 2024-08-17T17:06:31.098 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:30 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:31.098 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:30 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:31.098 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:30 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:31.098 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:30 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:06:31.117 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:30 smithi053 ceph-mon[34699]: pgmap v6: 9 pgs: 8 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:31.117 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:30 smithi053 ceph-mon[34699]: osdmap e61: 8 total, 8 up, 8 in 2024-08-17T17:06:31.117 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:30 smithi053 ceph-mon[34699]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:06:31.117 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:30 smithi053 ceph-mon[34699]: mgrmap e20: y(active, since 4s), standbys: x 2024-08-17T17:06:31.117 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:30 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:31.117 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:30 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:31.117 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:30 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:06:31.117 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:30 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:06:32.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:31 smithi053 ceph-mon[34699]: Updating smithi003:/etc/ceph/ceph.conf 2024-08-17T17:06:32.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:31 smithi053 ceph-mon[34699]: Updating smithi053:/etc/ceph/ceph.conf 2024-08-17T17:06:32.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:31 smithi053 ceph-mon[34699]: osdmap e62: 8 total, 8 up, 8 in 2024-08-17T17:06:32.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:31 smithi053 ceph-mon[34699]: Updating smithi003:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:06:32.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:31 smithi053 ceph-mon[34699]: Updating smithi053:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:06:32.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:31 smithi053 ceph-mon[34699]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:06:32.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:31 smithi053 ceph-mon[34699]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:06:32.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:31 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:32.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:31 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:32.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:31 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:32.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:31 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:32.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:31 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:32.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:31 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/1022727689' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-08-17T17:06:32.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:31 smithi053 ceph-mon[34699]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-08-17T17:06:32.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:31 smithi003 ceph-mon[30803]: Updating smithi003:/etc/ceph/ceph.conf 2024-08-17T17:06:32.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:31 smithi003 ceph-mon[30803]: Updating smithi053:/etc/ceph/ceph.conf 2024-08-17T17:06:32.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:31 smithi003 ceph-mon[30803]: osdmap e62: 8 total, 8 up, 8 in 2024-08-17T17:06:32.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:31 smithi003 ceph-mon[30803]: Updating smithi003:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:06:32.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:31 smithi003 ceph-mon[30803]: Updating smithi053:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:06:32.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:31 smithi003 ceph-mon[30803]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:06:32.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:31 smithi003 ceph-mon[30803]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:06:32.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:31 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:32.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:31 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:32.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:31 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:32.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:31 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:32.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:31 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:32.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:31 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/1022727689' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-08-17T17:06:32.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:31 smithi003 ceph-mon[30803]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-08-17T17:06:32.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:31 smithi003 ceph-mon[41530]: Updating smithi003:/etc/ceph/ceph.conf 2024-08-17T17:06:32.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:31 smithi003 ceph-mon[41530]: Updating smithi053:/etc/ceph/ceph.conf 2024-08-17T17:06:32.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:31 smithi003 ceph-mon[41530]: osdmap e62: 8 total, 8 up, 8 in 2024-08-17T17:06:32.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:31 smithi003 ceph-mon[41530]: Updating smithi003:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:06:32.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:31 smithi003 ceph-mon[41530]: Updating smithi053:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:06:32.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:31 smithi003 ceph-mon[41530]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:06:32.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:31 smithi003 ceph-mon[41530]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:06:32.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:31 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:32.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:31 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:32.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:31 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:32.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:31 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:32.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:31 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:32.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:31 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/1022727689' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-08-17T17:06:32.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:31 smithi003 ceph-mon[41530]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-08-17T17:06:33.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:32 smithi053 ceph-mon[34699]: Updating smithi003:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:06:33.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:32 smithi053 ceph-mon[34699]: Updating smithi053:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:06:33.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:32 smithi053 ceph-mon[34699]: Deploying daemon alertmanager.a on smithi003 2024-08-17T17:06:33.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:32 smithi053 ceph-mon[34699]: pgmap v9: 9 pgs: 6 unknown, 3 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:33.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:32 smithi053 ceph-mon[34699]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-08-17T17:06:33.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:32 smithi053 ceph-mon[34699]: osdmap e63: 8 total, 8 up, 8 in 2024-08-17T17:06:33.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:32 smithi003 ceph-mon[41530]: Updating smithi003:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:06:33.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:32 smithi003 ceph-mon[41530]: Updating smithi053:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:06:33.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:32 smithi003 ceph-mon[41530]: Deploying daemon alertmanager.a on smithi003 2024-08-17T17:06:33.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:32 smithi003 ceph-mon[41530]: pgmap v9: 9 pgs: 6 unknown, 3 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:33.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:32 smithi003 ceph-mon[41530]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-08-17T17:06:33.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:32 smithi003 ceph-mon[41530]: osdmap e63: 8 total, 8 up, 8 in 2024-08-17T17:06:33.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:32 smithi003 ceph-mon[30803]: Updating smithi003:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:06:33.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:32 smithi003 ceph-mon[30803]: Updating smithi053:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:06:33.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:32 smithi003 ceph-mon[30803]: Deploying daemon alertmanager.a on smithi003 2024-08-17T17:06:33.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:32 smithi003 ceph-mon[30803]: pgmap v9: 9 pgs: 6 unknown, 3 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:33.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:32 smithi003 ceph-mon[30803]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-08-17T17:06:33.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:32 smithi003 ceph-mon[30803]: osdmap e63: 8 total, 8 up, 8 in 2024-08-17T17:06:34.957 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:34 smithi003 ceph-mon[41530]: osdmap e64: 8 total, 8 up, 8 in 2024-08-17T17:06:34.958 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:34 smithi003 ceph-mon[41530]: pgmap v12: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 28 op/s 2024-08-17T17:06:34.958 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:34 smithi003 ceph-mon[30803]: osdmap e64: 8 total, 8 up, 8 in 2024-08-17T17:06:34.958 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:34 smithi003 ceph-mon[30803]: pgmap v12: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 28 op/s 2024-08-17T17:06:34.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:34 smithi053 ceph-mon[34699]: osdmap e64: 8 total, 8 up, 8 in 2024-08-17T17:06:34.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:34 smithi053 ceph-mon[34699]: pgmap v12: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 28 op/s 2024-08-17T17:06:35.860 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:35 smithi003 ceph-mon[30803]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:06:35.860 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:35 smithi003 ceph-mon[30803]: Cluster is now healthy 2024-08-17T17:06:35.860 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:35 smithi003 ceph-mon[30803]: osdmap e65: 8 total, 8 up, 8 in 2024-08-17T17:06:35.860 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:35 smithi003 ceph-mon[41530]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:06:35.860 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:35 smithi003 ceph-mon[41530]: Cluster is now healthy 2024-08-17T17:06:35.860 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:35 smithi003 ceph-mon[41530]: osdmap e65: 8 total, 8 up, 8 in 2024-08-17T17:06:35.889 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph config log 1 --format=json 2024-08-17T17:06:35.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:35 smithi053 ceph-mon[34699]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:06:35.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:35 smithi053 ceph-mon[34699]: Cluster is now healthy 2024-08-17T17:06:35.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:35 smithi053 ceph-mon[34699]: osdmap e65: 8 total, 8 up, 8 in 2024-08-17T17:06:36.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:36 smithi053 ceph-mon[34699]: pgmap v14: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-08-17T17:06:36.717 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:36 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:06:36] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-08-17T17:06:37.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:36 smithi003 ceph-mon[30803]: pgmap v14: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-08-17T17:06:37.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:36 smithi003 ceph-mon[41530]: pgmap v14: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-08-17T17:06:38.811 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:06:39.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:38 smithi003 ceph-mon[41530]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 34 KiB/s rd, 341 B/s wr, 19 op/s 2024-08-17T17:06:39.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:38 smithi003 ceph-mon[30803]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 34 KiB/s rd, 341 B/s wr, 19 op/s 2024-08-17T17:06:39.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:38 smithi053 ceph-mon[34699]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 34 KiB/s rd, 341 B/s wr, 19 op/s 2024-08-17T17:06:41.118 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:40 smithi003 ceph-mon[30803]: pgmap v16: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 282 B/s wr, 0 op/s 2024-08-17T17:06:41.118 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:40 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:06:41.118 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:40 smithi003 ceph-mon[41530]: pgmap v16: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 282 B/s wr, 0 op/s 2024-08-17T17:06:41.118 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:40 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:06:41.119 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:06:40 smithi003 systemd[1]: Started Ceph alertmanager.a for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:06:41.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:40 smithi053 ceph-mon[34699]: pgmap v16: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 282 B/s wr, 0 op/s 2024-08-17T17:06:41.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:40 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:06:41.380 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:06:41.380 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:41 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:06:41] ENGINE Bus STOPPING 2024-08-17T17:06:41.663 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:41 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:06:41] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-08-17T17:06:41.663 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:41 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:06:41] ENGINE Bus STOPPED 2024-08-17T17:06:41.663 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:41 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:06:41] ENGINE Bus STARTING 2024-08-17T17:06:41.663 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:41 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:06:41] ENGINE Serving on http://:::9283 2024-08-17T17:06:41.663 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:06:41 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:06:41] ENGINE Bus STARTED 2024-08-17T17:06:42.027 INFO:teuthology.orchestra.run.smithi003.stdout:[{"version":19,"timestamp":"2024-08-17T17:06:41.134075+0000","name":"","changes":[{"name":"mgr/mgr/dashboard/GRAFANA_API_SSL_VERIFY","new_value":"false"}]}] 2024-08-17T17:06:42.027 INFO:tasks.ceph_manager:config epoch is 19 2024-08-17T17:06:42.028 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-08-17T17:06:42.028 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-08-17T17:06:42.028 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph mgr dump --format=json 2024-08-17T17:06:42.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:42 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:42.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:42 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:42.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:42 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:42.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:42 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:42.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:42 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:42.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:42 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:42.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:42 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-08-17T17:06:42.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:42 smithi053 ceph-mon[34699]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-08-17T17:06:42.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:42 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:42.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:42 smithi053 ceph-mon[34699]: Deploying daemon grafana.a on smithi053 2024-08-17T17:06:42.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:42 smithi053 ceph-mon[34699]: osdmap e66: 8 total, 8 up, 8 in 2024-08-17T17:06:42.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:42 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/4187684772' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-08-17T17:06:42.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:42 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:42.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:42 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:42.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:42 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:42.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:42 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:42.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:42 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:42.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:42 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:42.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:42 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-08-17T17:06:42.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:42 smithi003 ceph-mon[41530]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-08-17T17:06:42.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:42 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:42.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:42 smithi003 ceph-mon[41530]: Deploying daemon grafana.a on smithi053 2024-08-17T17:06:42.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:42 smithi003 ceph-mon[41530]: osdmap e66: 8 total, 8 up, 8 in 2024-08-17T17:06:42.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:42 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/4187684772' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-08-17T17:06:42.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:42 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:42.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:42 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:42.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:42 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:42.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:42 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:42.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:42 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:42.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:42 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:42.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:42 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-08-17T17:06:42.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:42 smithi003 ceph-mon[30803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-08-17T17:06:42.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:42 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:42.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:42 smithi003 ceph-mon[30803]: Deploying daemon grafana.a on smithi053 2024-08-17T17:06:42.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:42 smithi003 ceph-mon[30803]: osdmap e66: 8 total, 8 up, 8 in 2024-08-17T17:06:42.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:42 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/4187684772' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-08-17T17:06:42.300 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:06:43.023 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:06:42 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a[69785]: ts=2024-08-17T17:06:42.867Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00011595s 2024-08-17T17:06:43.426 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:43 smithi003 ceph-mon[41530]: pgmap v18: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:06:43.426 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:43 smithi003 ceph-mon[30803]: pgmap v18: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:06:43.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:43 smithi053 ceph-mon[34699]: pgmap v18: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:06:44.123 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:06:44.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:44 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/465837320' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-08-17T17:06:44.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:44 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/465837320' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-08-17T17:06:44.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:44 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/465837320' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-08-17T17:06:44.958 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":20,"flags":0,"active_gid":14457,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":2670911286},{"type":"v1","addr":"172.21.15.3:6801","nonce":2670911286}]},"active_addr":"172.21.15.3:6801/2670911286","active_change":"2024-08-17T17:06:25.750739+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":24346,"name":"x","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.3:8443/","prometheus":"http://172.21.15.3:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":59,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":2029382827}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":601716019}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":3717779298}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":880444026}]}]} 2024-08-17T17:06:44.962 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-08-17T17:06:44.962 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-08-17T17:06:44.962 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-17T17:06:45.236 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:06:45.445 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:45 smithi003 ceph-mon[41530]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 221 B/s wr, 0 op/s 2024-08-17T17:06:45.445 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:45 smithi003 ceph-mon[30803]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 221 B/s wr, 0 op/s 2024-08-17T17:06:45.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:45 smithi053 ceph-mon[34699]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 221 B/s wr, 0 op/s 2024-08-17T17:06:46.716 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:46 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:06:46] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-08-17T17:06:46.962 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:06:46.962 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":66,"fsid":"2b5b662e-5cba-11ef-bccf-c7b262605968","created":"2024-08-17T17:00:56.372943+0000","modified":"2024-08-17T17:06:41.168524+0000","last_up_change":"2024-08-17T17:05:54.031028+0000","last_in_change":"2024-08-17T17:05:37.289378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-17T17:04:01.140951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-17T17:06:28.452624+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"7f9a7d76-f0ec-49fc-9540-2606e94758ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":288807015},{"type":"v1","addr":"172.21.15.3:6803","nonce":288807015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":288807015},{"type":"v1","addr":"172.21.15.3:6805","nonce":288807015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":288807015},{"type":"v1","addr":"172.21.15.3:6809","nonce":288807015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":288807015},{"type":"v1","addr":"172.21.15.3:6807","nonce":288807015}]},"public_addr":"172.21.15.3:6803/288807015","cluster_addr":"172.21.15.3:6805/288807015","heartbeat_back_addr":"172.21.15.3:6809/288807015","heartbeat_front_addr":"172.21.15.3:6807/288807015","state":["exists","up"]},{"osd":1,"uuid":"316b5c4e-3417-4b93-bbbb-c6ed6dd21b0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2183964597},{"type":"v1","addr":"172.21.15.3:6811","nonce":2183964597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2183964597},{"type":"v1","addr":"172.21.15.3:6813","nonce":2183964597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":2183964597},{"type":"v1","addr":"172.21.15.3:6817","nonce":2183964597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2183964597},{"type":"v1","addr":"172.21.15.3:6815","nonce":2183964597}]},"public_addr":"172.21.15.3:6811/2183964597","cluster_addr":"172.21.15.3:6813/2183964597","heartbeat_back_addr":"172.21.15.3:6817/2183964597","heartbeat_front_addr":"172.21.15.3:6815/2183964597","state":["exists","up"]},{"osd":2,"uuid":"88af697d-38db-4c51-bfff-d0a776315596","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1747862993},{"type":"v1","addr":"172.21.15.3:6819","nonce":1747862993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1747862993},{"type":"v1","addr":"172.21.15.3:6821","nonce":1747862993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1747862993},{"type":"v1","addr":"172.21.15.3:6825","nonce":1747862993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1747862993},{"type":"v1","addr":"172.21.15.3:6823","nonce":1747862993}]},"public_addr":"172.21.15.3:6819/1747862993","cluster_addr":"172.21.15.3:6821/1747862993","heartbeat_back_addr":"172.21.15.3:6825/1747862993","heartbeat_front_addr":"172.21.15.3:6823/1747862993","state":["exists","up"]},{"osd":3,"uuid":"8e20cf42-6a13-4ad5-9f01-203a30c692d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2572629546},{"type":"v1","addr":"172.21.15.3:6827","nonce":2572629546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2572629546},{"type":"v1","addr":"172.21.15.3:6829","nonce":2572629546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2572629546},{"type":"v1","addr":"172.21.15.3:6833","nonce":2572629546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2572629546},{"type":"v1","addr":"172.21.15.3:6831","nonce":2572629546}]},"public_addr":"172.21.15.3:6827/2572629546","cluster_addr":"172.21.15.3:6829/2572629546","heartbeat_back_addr":"172.21.15.3:6833/2572629546","heartbeat_front_addr":"172.21.15.3:6831/2572629546","state":["exists","up"]},{"osd":4,"uuid":"2e8eabd7-8258-4b78-bb21-1a7291e0f1d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6801","nonce":1387588062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6803","nonce":1387588062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6807","nonce":1387588062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6805","nonce":1387588062}]},"public_addr":"172.21.15.53:6801/1387588062","cluster_addr":"172.21.15.53:6803/1387588062","heartbeat_back_addr":"172.21.15.53:6807/1387588062","heartbeat_front_addr":"172.21.15.53:6805/1387588062","state":["exists","up"]},{"osd":5,"uuid":"738f10f7-ff0d-4392-a818-eb23c9f65a63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6809","nonce":1263400356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6811","nonce":1263400356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6815","nonce":1263400356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6813","nonce":1263400356}]},"public_addr":"172.21.15.53:6809/1263400356","cluster_addr":"172.21.15.53:6811/1263400356","heartbeat_back_addr":"172.21.15.53:6815/1263400356","heartbeat_front_addr":"172.21.15.53:6813/1263400356","state":["exists","up"]},{"osd":6,"uuid":"62903bee-59eb-4eed-b64d-9c884de87409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6817","nonce":3729099668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6819","nonce":3729099668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6823","nonce":3729099668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6821","nonce":3729099668}]},"public_addr":"172.21.15.53:6817/3729099668","cluster_addr":"172.21.15.53:6819/3729099668","heartbeat_back_addr":"172.21.15.53:6823/3729099668","heartbeat_front_addr":"172.21.15.53:6821/3729099668","state":["exists","up"]},{"osd":7,"uuid":"4794ff08-e5d9-49c9-bea5-e158681b4004","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6825","nonce":832291503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6827","nonce":832291503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6831","nonce":832291503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6829","nonce":832291503}]},"public_addr":"172.21.15.53:6825/832291503","cluster_addr":"172.21.15.53:6827/832291503","heartbeat_back_addr":"172.21.15.53:6831/832291503","heartbeat_front_addr":"172.21.15.53:6829/832291503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:03:10.759301+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:03:33.981212+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:03:58.516966+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:04:22.965963+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:04:43.836760+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:06.691915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:29.472293+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:52.166848+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:0/1583211207":"2024-08-18T17:06:25.750593+0000","172.21.15.3:6801/1736600360":"2024-08-18T17:06:25.750593+0000","172.21.15.3:0/1668133120":"2024-08-18T17:01:27.442443+0000","172.21.15.3:0/1476294447":"2024-08-18T17:01:27.442443+0000","172.21.15.3:6801/160535497":"2024-08-18T17:01:27.442443+0000","172.21.15.3:0/1999667208":"2024-08-18T17:01:50.631242+0000","172.21.15.3:6800/160535497":"2024-08-18T17:01:27.442443+0000","172.21.15.3:0/3946755108":"2024-08-18T17:06:25.750593+0000","172.21.15.3:0/3817753887":"2024-08-18T17:01:50.631242+0000","172.21.15.3:0/3953105554":"2024-08-18T17:06:25.750593+0000","172.21.15.3:6800/1736600360":"2024-08-18T17:06:25.750593+0000","172.21.15.3:0/3985128953":"2024-08-18T17:01:50.631242+0000","172.21.15.3:0/4194950757":"2024-08-18T17:01:27.442443+0000","172.21.15.3:6800/2187177466":"2024-08-18T17:01:50.631242+0000","172.21.15.3:6801/2187177466":"2024-08-18T17:01:50.631242+0000","172.21.15.3:0/3116588554":"2024-08-18T17:06:25.750593+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-17T17:06:47.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:46 smithi053 ceph-mon[34699]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 204 B/s wr, 0 op/s 2024-08-17T17:06:47.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:46 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:47.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:46 smithi003 ceph-mon[30803]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 204 B/s wr, 0 op/s 2024-08-17T17:06:47.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:46 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:47.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:46 smithi003 ceph-mon[41530]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 204 B/s wr, 0 op/s 2024-08-17T17:06:47.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:46 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:47.645 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-08-17T17:06:47.646 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-17T17:06:47.917 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:06:48.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:47 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/3471732510' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-17T17:06:48.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:47 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/3471732510' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-17T17:06:48.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:47 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/3471732510' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-17T17:06:48.943 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:48 smithi003 ceph-mon[41530]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:48.943 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:48 smithi003 ceph-mon[30803]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:49.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:48 smithi053 ceph-mon[34699]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:49.619 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:06:49.619 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":66,"fsid":"2b5b662e-5cba-11ef-bccf-c7b262605968","created":"2024-08-17T17:00:56.372943+0000","modified":"2024-08-17T17:06:41.168524+0000","last_up_change":"2024-08-17T17:05:54.031028+0000","last_in_change":"2024-08-17T17:05:37.289378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-17T17:04:01.140951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-17T17:06:28.452624+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"7f9a7d76-f0ec-49fc-9540-2606e94758ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":288807015},{"type":"v1","addr":"172.21.15.3:6803","nonce":288807015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":288807015},{"type":"v1","addr":"172.21.15.3:6805","nonce":288807015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":288807015},{"type":"v1","addr":"172.21.15.3:6809","nonce":288807015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":288807015},{"type":"v1","addr":"172.21.15.3:6807","nonce":288807015}]},"public_addr":"172.21.15.3:6803/288807015","cluster_addr":"172.21.15.3:6805/288807015","heartbeat_back_addr":"172.21.15.3:6809/288807015","heartbeat_front_addr":"172.21.15.3:6807/288807015","state":["exists","up"]},{"osd":1,"uuid":"316b5c4e-3417-4b93-bbbb-c6ed6dd21b0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2183964597},{"type":"v1","addr":"172.21.15.3:6811","nonce":2183964597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2183964597},{"type":"v1","addr":"172.21.15.3:6813","nonce":2183964597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":2183964597},{"type":"v1","addr":"172.21.15.3:6817","nonce":2183964597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2183964597},{"type":"v1","addr":"172.21.15.3:6815","nonce":2183964597}]},"public_addr":"172.21.15.3:6811/2183964597","cluster_addr":"172.21.15.3:6813/2183964597","heartbeat_back_addr":"172.21.15.3:6817/2183964597","heartbeat_front_addr":"172.21.15.3:6815/2183964597","state":["exists","up"]},{"osd":2,"uuid":"88af697d-38db-4c51-bfff-d0a776315596","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1747862993},{"type":"v1","addr":"172.21.15.3:6819","nonce":1747862993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1747862993},{"type":"v1","addr":"172.21.15.3:6821","nonce":1747862993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1747862993},{"type":"v1","addr":"172.21.15.3:6825","nonce":1747862993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1747862993},{"type":"v1","addr":"172.21.15.3:6823","nonce":1747862993}]},"public_addr":"172.21.15.3:6819/1747862993","cluster_addr":"172.21.15.3:6821/1747862993","heartbeat_back_addr":"172.21.15.3:6825/1747862993","heartbeat_front_addr":"172.21.15.3:6823/1747862993","state":["exists","up"]},{"osd":3,"uuid":"8e20cf42-6a13-4ad5-9f01-203a30c692d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2572629546},{"type":"v1","addr":"172.21.15.3:6827","nonce":2572629546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2572629546},{"type":"v1","addr":"172.21.15.3:6829","nonce":2572629546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2572629546},{"type":"v1","addr":"172.21.15.3:6833","nonce":2572629546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2572629546},{"type":"v1","addr":"172.21.15.3:6831","nonce":2572629546}]},"public_addr":"172.21.15.3:6827/2572629546","cluster_addr":"172.21.15.3:6829/2572629546","heartbeat_back_addr":"172.21.15.3:6833/2572629546","heartbeat_front_addr":"172.21.15.3:6831/2572629546","state":["exists","up"]},{"osd":4,"uuid":"2e8eabd7-8258-4b78-bb21-1a7291e0f1d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6801","nonce":1387588062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6803","nonce":1387588062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6807","nonce":1387588062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6805","nonce":1387588062}]},"public_addr":"172.21.15.53:6801/1387588062","cluster_addr":"172.21.15.53:6803/1387588062","heartbeat_back_addr":"172.21.15.53:6807/1387588062","heartbeat_front_addr":"172.21.15.53:6805/1387588062","state":["exists","up"]},{"osd":5,"uuid":"738f10f7-ff0d-4392-a818-eb23c9f65a63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6809","nonce":1263400356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6811","nonce":1263400356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6815","nonce":1263400356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6813","nonce":1263400356}]},"public_addr":"172.21.15.53:6809/1263400356","cluster_addr":"172.21.15.53:6811/1263400356","heartbeat_back_addr":"172.21.15.53:6815/1263400356","heartbeat_front_addr":"172.21.15.53:6813/1263400356","state":["exists","up"]},{"osd":6,"uuid":"62903bee-59eb-4eed-b64d-9c884de87409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6817","nonce":3729099668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6819","nonce":3729099668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6823","nonce":3729099668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6821","nonce":3729099668}]},"public_addr":"172.21.15.53:6817/3729099668","cluster_addr":"172.21.15.53:6819/3729099668","heartbeat_back_addr":"172.21.15.53:6823/3729099668","heartbeat_front_addr":"172.21.15.53:6821/3729099668","state":["exists","up"]},{"osd":7,"uuid":"4794ff08-e5d9-49c9-bea5-e158681b4004","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6825","nonce":832291503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6827","nonce":832291503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6831","nonce":832291503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6829","nonce":832291503}]},"public_addr":"172.21.15.53:6825/832291503","cluster_addr":"172.21.15.53:6827/832291503","heartbeat_back_addr":"172.21.15.53:6831/832291503","heartbeat_front_addr":"172.21.15.53:6829/832291503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:03:10.759301+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:03:33.981212+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:03:58.516966+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:04:22.965963+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:04:43.836760+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:06.691915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:29.472293+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:52.166848+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:0/1583211207":"2024-08-18T17:06:25.750593+0000","172.21.15.3:6801/1736600360":"2024-08-18T17:06:25.750593+0000","172.21.15.3:0/1668133120":"2024-08-18T17:01:27.442443+0000","172.21.15.3:0/1476294447":"2024-08-18T17:01:27.442443+0000","172.21.15.3:6801/160535497":"2024-08-18T17:01:27.442443+0000","172.21.15.3:0/1999667208":"2024-08-18T17:01:50.631242+0000","172.21.15.3:6800/160535497":"2024-08-18T17:01:27.442443+0000","172.21.15.3:0/3946755108":"2024-08-18T17:06:25.750593+0000","172.21.15.3:0/3817753887":"2024-08-18T17:01:50.631242+0000","172.21.15.3:0/3953105554":"2024-08-18T17:06:25.750593+0000","172.21.15.3:6800/1736600360":"2024-08-18T17:06:25.750593+0000","172.21.15.3:0/3985128953":"2024-08-18T17:01:50.631242+0000","172.21.15.3:0/4194950757":"2024-08-18T17:01:27.442443+0000","172.21.15.3:6800/2187177466":"2024-08-18T17:01:50.631242+0000","172.21.15.3:6801/2187177466":"2024-08-18T17:01:50.631242+0000","172.21.15.3:0/3116588554":"2024-08-18T17:06:25.750593+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-17T17:06:49.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:49 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/3612720613' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-17T17:06:49.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:49 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/3612720613' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-17T17:06:50.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:49 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/3612720613' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-17T17:06:50.246 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-08-17T17:06:50.246 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-08-17T17:06:50.247 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-08-17T17:06:50.247 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-08-17T17:06:50.248 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-08-17T17:06:50.248 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-08-17T17:06:50.248 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-08-17T17:06:50.249 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-08-17T17:06:50.798 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:06:50.799 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:06:50.799 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:06:50.800 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:06:50.805 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:06:50.805 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:06:50.805 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:06:50.830 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:06:51.112 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:50 smithi003 ceph-mon[41530]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:51.112 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:06:50 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a[69785]: ts=2024-08-17T17:06:50.871Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003164786s 2024-08-17T17:06:51.112 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:50 smithi003 ceph-mon[30803]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:51.140 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:50 smithi053 ceph-mon[34699]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:52.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:52 smithi053 ceph-mon[34699]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:53.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:52 smithi003 ceph-mon[30803]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:53.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:52 smithi003 ceph-mon[41530]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:54.711 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 systemd[1]: Started Ceph grafana.a for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:06:54.963 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-mon[34699]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:54.963 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:54.963 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:54.963 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:54.963 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:54.964 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=settings t=2024-08-17T17:06:54.711454316Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-08-17T17:06:54.964 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=settings t=2024-08-17T17:06:54.711649437Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-08-17T17:06:54.964 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=settings t=2024-08-17T17:06:54.711676218Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-08-17T17:06:54.964 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=settings t=2024-08-17T17:06:54.711690719Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-08-17T17:06:54.964 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=settings t=2024-08-17T17:06:54.711701913Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-08-17T17:06:54.964 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=settings t=2024-08-17T17:06:54.711711448Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-08-17T17:06:54.964 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=settings t=2024-08-17T17:06:54.711720545Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-08-17T17:06:54.964 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=settings t=2024-08-17T17:06:54.711730418Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-08-17T17:06:54.964 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=settings t=2024-08-17T17:06:54.711742018Z level=info msg="Path Home" path=/usr/share/grafana 2024-08-17T17:06:54.964 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=settings t=2024-08-17T17:06:54.711753726Z level=info msg="Path Data" path=/var/lib/grafana 2024-08-17T17:06:54.965 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=settings t=2024-08-17T17:06:54.711763321Z level=info msg="Path Logs" path=/var/log/grafana 2024-08-17T17:06:54.965 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=settings t=2024-08-17T17:06:54.711773962Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-08-17T17:06:54.965 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=settings t=2024-08-17T17:06:54.711784373Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-08-17T17:06:54.965 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=settings t=2024-08-17T17:06:54.711794902Z level=info msg="App mode production" 2024-08-17T17:06:54.965 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=sqlstore t=2024-08-17T17:06:54.711860561Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-08-17T17:06:54.965 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=sqlstore t=2024-08-17T17:06:54.711889691Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-08-17T17:06:54.965 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.713177724Z level=info msg="Starting DB migrations" 2024-08-17T17:06:54.965 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.713276789Z level=info msg="Executing migration" id="create migration_log table" 2024-08-17T17:06:54.965 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.713847074Z level=info msg="Executing migration" id="create user table" 2024-08-17T17:06:54.965 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.714370328Z level=info msg="Executing migration" id="add unique index user.login" 2024-08-17T17:06:54.965 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.714841254Z level=info msg="Executing migration" id="add unique index user.email" 2024-08-17T17:06:54.965 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.715287482Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-08-17T17:06:54.966 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.715787663Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-08-17T17:06:54.966 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.716181899Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-08-17T17:06:54.966 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.716892538Z level=info msg="Executing migration" id="create user table v2" 2024-08-17T17:06:54.966 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.71729645Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-08-17T17:06:54.966 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.717674465Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-08-17T17:06:54.966 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.718118184Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-08-17T17:06:54.966 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.718476937Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-08-17T17:06:54.966 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.71880762Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-08-17T17:06:54.966 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.719236366Z level=info msg="Executing migration" id="Update user table charset" 2024-08-17T17:06:54.966 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.719544353Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-08-17T17:06:54.966 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.719955742Z level=info msg="Executing migration" id="Add missing user data" 2024-08-17T17:06:54.966 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.720314752Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-08-17T17:06:54.966 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.720708225Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-08-17T17:06:54.967 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.721128942Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-08-17T17:06:54.967 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.721572409Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-08-17T17:06:54.967 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.72272787Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-08-17T17:06:54.969 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.723168071Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-08-17T17:06:54.969 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.723609642Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-08-17T17:06:54.969 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.724058447Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-08-17T17:06:54.969 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.724494707Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-08-17T17:06:54.969 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.72492719Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-08-17T17:06:54.969 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.725224142Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-08-17T17:06:54.969 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.725574509Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-08-17T17:06:54.969 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.725944214Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-08-17T17:06:54.969 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.726366021Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-08-17T17:06:54.969 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.72672289Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-08-17T17:06:54.969 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.727403235Z level=info msg="Executing migration" id="create temp_user v2" 2024-08-17T17:06:54.970 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.727813086Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-08-17T17:06:54.970 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.728227989Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-08-17T17:06:54.970 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.728642151Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-08-17T17:06:54.970 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.72906462Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-08-17T17:06:54.970 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.729504402Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-08-17T17:06:54.970 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.729806787Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-08-17T17:06:54.970 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.730195479Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-08-17T17:06:54.970 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.730528794Z level=info msg="Executing migration" id="create star table" 2024-08-17T17:06:54.970 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.730873033Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-08-17T17:06:54.970 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.731337239Z level=info msg="Executing migration" id="create org table v1" 2024-08-17T17:06:54.970 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.731754552Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-08-17T17:06:54.971 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.732181583Z level=info msg="Executing migration" id="create org_user table v1" 2024-08-17T17:06:54.971 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.732571331Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-08-17T17:06:54.971 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.732990917Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-08-17T17:06:54.971 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.733419014Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-08-17T17:06:54.971 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.733829751Z level=info msg="Executing migration" id="Update org table charset" 2024-08-17T17:06:54.971 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.734124621Z level=info msg="Executing migration" id="Update org_user table charset" 2024-08-17T17:06:54.971 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.734417696Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-08-17T17:06:54.971 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.734700126Z level=info msg="Executing migration" id="create dashboard table" 2024-08-17T17:06:54.971 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.735109454Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-08-17T17:06:54.971 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.735530696Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-08-17T17:06:54.971 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.735962547Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-08-17T17:06:54.971 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.73638529Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-08-17T17:06:54.971 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.73681505Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-08-17T17:06:54.972 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.737220104Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-08-17T17:06:54.972 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.738091269Z level=info msg="Executing migration" id="create dashboard v2" 2024-08-17T17:06:54.972 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.738500471Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-08-17T17:06:54.972 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.738972532Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-08-17T17:06:54.972 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.739410294Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-08-17T17:06:54.972 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.739715998Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-08-17T17:06:54.972 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.740123358Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-08-17T17:06:54.972 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.740425002Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-08-17T17:06:54.972 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.74087711Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-08-17T17:06:54.972 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.741383323Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-08-17T17:06:54.972 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.741831914Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-08-17T17:06:54.972 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.742280437Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-08-17T17:06:54.972 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.742781902Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-08-17T17:06:54.973 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.743264247Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-08-17T17:06:54.973 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.743686464Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-08-17T17:06:54.973 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.743956836Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-08-17T17:06:54.973 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.744254979Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-08-17T17:06:54.973 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.744759442Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-08-17T17:06:54.973 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.745264566Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-08-17T17:06:54.973 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.745761732Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-08-17T17:06:54.973 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.746327552Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-08-17T17:06:54.973 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.746648709Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-08-17T17:06:54.973 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.747077502Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-08-17T17:06:54.973 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.747481421Z level=info msg="Executing migration" id="Update dashboard title length" 2024-08-17T17:06:54.973 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.747747709Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-08-17T17:06:54.973 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.748208251Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-08-17T17:06:54.974 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.748595539Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-08-17T17:06:54.974 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.749511495Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-08-17T17:06:54.974 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.749912427Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-08-17T17:06:54.974 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.750332802Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-08-17T17:06:54.974 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.750745173Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-08-17T17:06:54.974 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.751082235Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-08-17T17:06:54.974 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.751442654Z level=info msg="Executing migration" id="Add check_sum column" 2024-08-17T17:06:54.974 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.751925661Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-08-17T17:06:54.974 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.752363564Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-08-17T17:06:54.974 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.752647395Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-08-17T17:06:54.974 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.752953469Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-08-17T17:06:54.974 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.753371529Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-08-17T17:06:54.974 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.75388821Z level=info msg="Executing migration" id="create data_source table" 2024-08-17T17:06:54.975 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.754332903Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-08-17T17:06:54.975 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.754767533Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-08-17T17:06:54.975 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.75519001Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-08-17T17:06:54.975 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.755602311Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-08-17T17:06:54.975 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.755990257Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-08-17T17:06:54.975 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.757018883Z level=info msg="Executing migration" id="create data_source table v2" 2024-08-17T17:06:54.975 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.757466047Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-08-17T17:06:54.975 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.757854682Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-08-17T17:06:54.975 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.758332842Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-08-17T17:06:54.975 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.758661031Z level=info msg="Executing migration" id="Add column with_credentials" 2024-08-17T17:06:54.975 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.759203162Z level=info msg="Executing migration" id="Add secure json data column" 2024-08-17T17:06:54.975 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.759745225Z level=info msg="Executing migration" id="Update data_source table charset" 2024-08-17T17:06:54.975 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.760052Z level=info msg="Executing migration" id="Update initial version to 1" 2024-08-17T17:06:54.975 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.760366256Z level=info msg="Executing migration" id="Add read_only data column" 2024-08-17T17:06:54.976 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.76085732Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-08-17T17:06:54.976 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.761209833Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-08-17T17:06:54.976 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.761492689Z level=info msg="Executing migration" id="Add uid column" 2024-08-17T17:06:54.976 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.762075051Z level=info msg="Executing migration" id="Update uid value" 2024-08-17T17:06:54.976 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.762394436Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-08-17T17:06:54.976 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.762788646Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-08-17T17:06:54.976 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.763216243Z level=info msg="Executing migration" id="create api_key table" 2024-08-17T17:06:54.976 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.763667439Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-08-17T17:06:54.976 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.76412008Z level=info msg="Executing migration" id="add index api_key.key" 2024-08-17T17:06:54.976 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.764549606Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-08-17T17:06:54.976 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.765024319Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-08-17T17:06:54.976 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.765441351Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-08-17T17:06:54.976 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.765821859Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-08-17T17:06:54.977 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.76624965Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-08-17T17:06:54.977 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.76730993Z level=info msg="Executing migration" id="create api_key table v2" 2024-08-17T17:06:54.977 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.767699987Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-08-17T17:06:54.977 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.768127849Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-08-17T17:06:54.977 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.768541217Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-08-17T17:06:54.977 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.768959725Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-08-17T17:06:54.977 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.769288033Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-08-17T17:06:54.977 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.769602928Z level=info msg="Executing migration" id="Update api_key table charset" 2024-08-17T17:06:54.977 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.769850438Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-08-17T17:06:54.977 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.77039114Z level=info msg="Executing migration" id="Add service account foreign key" 2024-08-17T17:06:54.977 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.770887296Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-08-17T17:06:54.977 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.771207759Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-08-17T17:06:54.977 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.771700279Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-08-17T17:06:54.977 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.772248084Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-08-17T17:06:54.977 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.772632267Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-08-17T17:06:54.978 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.772972662Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-08-17T17:06:54.978 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.773373045Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-08-17T17:06:54.978 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.773800191Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-08-17T17:06:54.978 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.774216955Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-08-17T17:06:54.978 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.774635829Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-08-17T17:06:54.978 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.77488414Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-08-17T17:06:54.978 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.775138079Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-08-17T17:06:54.978 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.775644998Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-08-17T17:06:54.978 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.776194726Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-08-17T17:06:54.978 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.77650571Z level=info msg="Executing migration" id="create quota table v1" 2024-08-17T17:06:54.978 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.776894992Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-08-17T17:06:54.978 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.77737758Z level=info msg="Executing migration" id="Update quota table charset" 2024-08-17T17:06:54.979 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.777635274Z level=info msg="Executing migration" id="create plugin_setting table" 2024-08-17T17:06:54.979 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.778046966Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-08-17T17:06:54.979 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.778490813Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-08-17T17:06:54.979 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.779065782Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-08-17T17:06:54.979 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.779364281Z level=info msg="Executing migration" id="create session table" 2024-08-17T17:06:54.979 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.779793761Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-08-17T17:06:54.979 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.78007932Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-08-17T17:06:54.979 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.780337179Z level=info msg="Executing migration" id="create playlist table v2" 2024-08-17T17:06:54.979 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.780726317Z level=info msg="Executing migration" id="create playlist item table v2" 2024-08-17T17:06:54.979 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.781119665Z level=info msg="Executing migration" id="Update playlist table charset" 2024-08-17T17:06:54.979 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.781396129Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-08-17T17:06:54.979 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.781653764Z level=info msg="Executing migration" id="drop preferences table v2" 2024-08-17T17:06:54.979 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.78192722Z level=info msg="Executing migration" id="drop preferences table v3" 2024-08-17T17:06:54.980 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.78220159Z level=info msg="Executing migration" id="create preferences table v3" 2024-08-17T17:06:54.980 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.78260532Z level=info msg="Executing migration" id="Update preferences table charset" 2024-08-17T17:06:54.980 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.782864496Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-08-17T17:06:54.980 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.783502486Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-08-17T17:06:54.980 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.783791315Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-08-17T17:06:54.980 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.784404863Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-08-17T17:06:54.980 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.784995924Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-08-17T17:06:54.980 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.785298373Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-08-17T17:06:54.980 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.785750254Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-08-17T17:06:54.981 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.786285931Z level=info msg="Executing migration" id="create alert table v1" 2024-08-17T17:06:54.981 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.786746266Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-08-17T17:06:54.981 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.787218154Z level=info msg="Executing migration" id="add index alert state" 2024-08-17T17:06:54.981 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.787668218Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-08-17T17:06:54.981 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.788155519Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-08-17T17:06:54.981 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.788544521Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-08-17T17:06:54.981 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.788991279Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-08-17T17:06:54.981 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.789439514Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-08-17T17:06:54.981 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.79085435Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-08-17T17:06:54.981 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.791233359Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-08-17T17:06:54.981 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.791657155Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-08-17T17:06:54.981 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.791973169Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-08-17T17:06:54.981 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.792345628Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-08-17T17:06:54.981 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.792703344Z level=info msg="Executing migration" id="Add column is_default" 2024-08-17T17:06:54.982 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.793328188Z level=info msg="Executing migration" id="Add column frequency" 2024-08-17T17:06:54.982 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.793917358Z level=info msg="Executing migration" id="Add column send_reminder" 2024-08-17T17:06:54.982 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.794564791Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-08-17T17:06:54.982 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.795195259Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-08-17T17:06:54.982 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.795647365Z level=info msg="Executing migration" id="Update alert table charset" 2024-08-17T17:06:54.982 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.795961265Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-08-17T17:06:54.982 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.796261344Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-08-17T17:06:54.982 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.796673062Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-08-17T17:06:54.982 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.797130638Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-08-17T17:06:54.982 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.797535446Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-08-17T17:06:54.982 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.797891029Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-08-17T17:06:54.982 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.798349998Z level=info msg="Executing migration" id="Add for to alert table" 2024-08-17T17:06:54.982 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.798974405Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-08-17T17:06:54.983 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.799619447Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-08-17T17:06:54.983 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.799902349Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-08-17T17:06:54.983 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.80035403Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-08-17T17:06:54.983 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.800746724Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-08-17T17:06:54.983 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.801407373Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-08-17T17:06:54.983 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.801703356Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-08-17T17:06:54.983 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.802138039Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-08-17T17:06:54.983 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.802580246Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-08-17T17:06:54.983 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.802865521Z level=info msg="Executing migration" id="create annotation table v5" 2024-08-17T17:06:54.983 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.803318439Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-08-17T17:06:54.983 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.803783004Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-08-17T17:06:54.983 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.80425513Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-08-17T17:06:54.983 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.804713643Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-08-17T17:06:54.983 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.805183524Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-08-17T17:06:54.983 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.805674233Z level=info msg="Executing migration" id="Update annotation table charset" 2024-08-17T17:06:54.984 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.805937317Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-08-17T17:06:54.984 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.806614098Z level=info msg="Executing migration" id="Drop category_id index" 2024-08-17T17:06:54.984 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.8070342Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-08-17T17:06:54.984 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.807711902Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-08-17T17:06:54.984 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.808122107Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-08-17T17:06:54.984 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.80857836Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-08-17T17:06:54.984 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.808971983Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-08-17T17:06:54.984 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.810538618Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-08-17T17:06:54.984 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.810939475Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-08-17T17:06:54.984 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.811395889Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-08-17T17:06:54.984 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.811695744Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-08-17T17:06:54.984 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.812058953Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-08-17T17:06:54.984 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.812369571Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-08-17T17:06:54.985 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.813029053Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-08-17T17:06:54.985 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.813711909Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-08-17T17:06:54.985 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.814183504Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-08-17T17:06:54.985 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.814644425Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-08-17T17:06:54.985 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.81493231Z level=info msg="Executing migration" id="Add epoch_end column" 2024-08-17T17:06:54.985 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.81561096Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-08-17T17:06:54.985 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.816047617Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-08-17T17:06:54.985 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.81636133Z level=info msg="Executing migration" id="Move region to single row" 2024-08-17T17:06:54.985 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.816692344Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-08-17T17:06:54.985 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.817167266Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-08-17T17:06:54.985 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.817609044Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-08-17T17:06:54.985 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.818068555Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-08-17T17:06:54.985 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.818524151Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-08-17T17:06:54.985 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.818913444Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-08-17T17:06:54.985 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.819340258Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-08-17T17:06:54.986 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.819606854Z level=info msg="Executing migration" id="create test_data table" 2024-08-17T17:06:54.986 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.820020289Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-08-17T17:06:54.986 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.820437809Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-08-17T17:06:54.986 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.820891632Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-08-17T17:06:54.986 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.821364599Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-08-17T17:06:54.986 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.821640859Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-08-17T17:06:54.986 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.821963596Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-08-17T17:06:54.986 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.822260232Z level=info msg="Executing migration" id="create team table" 2024-08-17T17:06:54.986 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.822639629Z level=info msg="Executing migration" id="add index team.org_id" 2024-08-17T17:06:54.986 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.82309345Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-08-17T17:06:54.986 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.823552244Z level=info msg="Executing migration" id="create team member table" 2024-08-17T17:06:54.986 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.823948845Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-08-17T17:06:54.986 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.824390188Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-08-17T17:06:54.987 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.824830327Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-08-17T17:06:54.987 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.825262739Z level=info msg="Executing migration" id="Add column email to team table" 2024-08-17T17:06:54.987 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.826000852Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-08-17T17:06:54.987 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.826661054Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-08-17T17:06:54.987 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.827407597Z level=info msg="Executing migration" id="create dashboard acl table" 2024-08-17T17:06:54.987 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.827794893Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-08-17T17:06:54.987 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.828273842Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-08-17T17:06:54.987 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.82875404Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-08-17T17:06:54.987 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.829461091Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-08-17T17:06:54.987 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.829994918Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-08-17T17:06:54.987 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.830541897Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-08-17T17:06:54.987 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.83113162Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-08-17T17:06:54.987 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.831810608Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-08-17T17:06:54.987 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.832397246Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-08-17T17:06:54.987 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.832780141Z level=info msg="Executing migration" id="create tag table" 2024-08-17T17:06:54.988 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.83333606Z level=info msg="Executing migration" id="add index tag.key_value" 2024-08-17T17:06:54.989 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.833801374Z level=info msg="Executing migration" id="create login attempt table" 2024-08-17T17:06:54.989 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.834178651Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-08-17T17:06:54.990 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.834591363Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-08-17T17:06:54.990 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.834994581Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-08-17T17:06:54.990 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.836881266Z level=info msg="Executing migration" id="create login_attempt v2" 2024-08-17T17:06:54.990 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.837260694Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-08-17T17:06:54.990 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.837659316Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-08-17T17:06:54.990 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.837939883Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-08-17T17:06:54.990 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.838282847Z level=info msg="Executing migration" id="create user auth table" 2024-08-17T17:06:54.990 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.838611622Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-08-17T17:06:54.990 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.839040666Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-08-17T17:06:54.990 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.839350841Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-08-17T17:06:54.991 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.840102187Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-08-17T17:06:54.991 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.840812818Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-08-17T17:06:54.991 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.841534157Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-08-17T17:06:54.991 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.842310869Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-08-17T17:06:54.991 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.842725633Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-08-17T17:06:54.991 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.843460133Z level=info msg="Executing migration" id="create server_lock table" 2024-08-17T17:06:54.991 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.843830283Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-08-17T17:06:54.991 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.844251735Z level=info msg="Executing migration" id="create user auth token table" 2024-08-17T17:06:54.991 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.844659267Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-08-17T17:06:54.991 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.845076935Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-08-17T17:06:54.991 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.845478535Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-08-17T17:06:54.991 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.845898383Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-08-17T17:06:54.992 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.846655865Z level=info msg="Executing migration" id="create cache_data table" 2024-08-17T17:06:54.992 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.847054702Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-08-17T17:06:54.992 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.847468248Z level=info msg="Executing migration" id="create short_url table v1" 2024-08-17T17:06:54.992 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.847864843Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-08-17T17:06:54.992 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.848280811Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-08-17T17:06:54.992 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.848521985Z level=info msg="Executing migration" id="delete alert_definition table" 2024-08-17T17:06:54.992 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.848768334Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-08-17T17:06:54.992 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.849202021Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-08-17T17:06:54.992 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.849608596Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-08-17T17:06:54.992 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.850016056Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-08-17T17:06:54.992 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.850290947Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-08-17T17:06:54.992 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.850645925Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-08-17T17:06:54.992 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.851068743Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-08-17T17:06:54.992 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.851484014Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-08-17T17:06:54.993 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.851889092Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-08-17T17:06:54.993 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.85268378Z level=info msg="Executing migration" id="drop alert_definition table" 2024-08-17T17:06:54.993 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.853220529Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-08-17T17:06:54.993 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.853510877Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-08-17T17:06:54.993 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.853862648Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-08-17T17:06:54.993 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.854284699Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-08-17T17:06:54.993 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.854654476Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-08-17T17:06:54.993 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.854890291Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-08-17T17:06:54.993 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.855309759Z level=info msg="Executing migration" id="create alert_instance table" 2024-08-17T17:06:54.993 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.855674372Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-08-17T17:06:54.993 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.856088849Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-08-17T17:06:54.993 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.856494553Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-08-17T17:06:54.993 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.857273362Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-08-17T17:06:54.993 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.857666511Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-08-17T17:06:54.993 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.858069973Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-08-17T17:06:54.994 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.861474397Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-08-17T17:06:54.994 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.864885784Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-08-17T17:06:54.994 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.865324032Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-08-17T17:06:54.994 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.865712078Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-08-17T17:06:54.994 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.866510643Z level=info msg="Executing migration" id="create alert_rule table" 2024-08-17T17:06:54.994 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.866915156Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-08-17T17:06:54.994 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.867333706Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-08-17T17:06:54.994 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.867737688Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-08-17T17:06:54.994 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.868151123Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-08-17T17:06:54.994 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.868440018Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-08-17T17:06:54.994 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.869306368Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-08-17T17:06:54.995 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.87009882Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-08-17T17:06:54.995 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.870877494Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-08-17T17:06:54.995 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.871293684Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-08-17T17:06:54.995 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.871666044Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-08-17T17:06:54.995 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.872458036Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-08-17T17:06:54.995 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.873261928Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-08-17T17:06:54.995 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.873677572Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-08-17T17:06:54.995 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.874475493Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-08-17T17:06:54.995 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.875282459Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-08-17T17:06:54.995 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.875611288Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-08-17T17:06:54.995 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.87604565Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-08-17T17:06:54.995 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.876459396Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-08-17T17:06:54.995 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.876872512Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-08-17T17:06:54.995 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.877161654Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-08-17T17:06:54.995 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.877979742Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-08-17T17:06:54.996 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.878781397Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-08-17T17:06:54.996 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.87959251Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-08-17T17:06:54.996 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.880421248Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-08-17T17:06:54.996 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.881259862Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-08-17T17:06:54.996 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.881550657Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-08-17T17:06:54.996 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.881923247Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-08-17T17:06:54.996 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.882729809Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-08-17T17:06:54.996 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.883016947Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-08-17T17:06:54.996 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.883859113Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-08-17T17:06:54.996 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.884280457Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-08-17T17:06:54.996 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.885108636Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-08-17T17:06:54.996 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.885484371Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-08-17T17:06:54.996 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.885890097Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-08-17T17:06:54.996 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.886727048Z level=info msg="Executing migration" id="create provenance_type table" 2024-08-17T17:06:54.996 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.887111933Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-08-17T17:06:54.997 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.887545651Z level=info msg="Executing migration" id="create alert_image table" 2024-08-17T17:06:54.997 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.887920878Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-08-17T17:06:54.997 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.888330329Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-08-17T17:06:54.997 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.888576111Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-08-17T17:06:54.997 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.888979358Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-08-17T17:06:54.997 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.88940109Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-08-17T17:06:54.997 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.889479931Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-08-17T17:06:54.997 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.889695787Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-08-17T17:06:54.997 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.890065036Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-08-17T17:06:54.997 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.890510835Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-08-17T17:06:54.997 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.890630627Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-08-17T17:06:54.997 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.89074059Z level=info msg="alerts found to migrate" alerts=0 2024-08-17T17:06:54.997 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.891110907Z level=info msg="Executing migration" id="create library_element table v1" 2024-08-17T17:06:54.997 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.891528837Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-08-17T17:06:54.998 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.891946488Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-08-17T17:06:54.998 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.892335835Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-08-17T17:06:54.998 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.892751088Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-08-17T17:06:54.998 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.89318798Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-08-17T17:06:54.998 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.893452174Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-08-17T17:06:54.998 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.89376232Z level=info msg="Executing migration" id="create data_keys table" 2024-08-17T17:06:54.998 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.894164064Z level=info msg="Executing migration" id="create secrets table" 2024-08-17T17:06:54.998 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.894586721Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-08-17T17:06:54.998 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.89892465Z level=info msg="Executing migration" id="add name column into data_keys" 2024-08-17T17:06:54.998 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.899822973Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-08-17T17:06:54.998 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.900117538Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-08-17T17:06:54.998 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.904429657Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-08-17T17:06:54.998 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.908750048Z level=info msg="Executing migration" id="create kv_store table v1" 2024-08-17T17:06:54.999 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.909210842Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-08-17T17:06:54.999 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.909656523Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-08-17T17:06:54.999 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.909955035Z level=info msg="Executing migration" id="create permission table" 2024-08-17T17:06:54.999 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.910347354Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-08-17T17:06:54.999 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.910779591Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-08-17T17:06:54.999 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.911211501Z level=info msg="Executing migration" id="create role table" 2024-08-17T17:06:54.999 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.911631923Z level=info msg="Executing migration" id="add column display_name" 2024-08-17T17:06:54.999 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.912546078Z level=info msg="Executing migration" id="add column group_name" 2024-08-17T17:06:54.999 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.9134689Z level=info msg="Executing migration" id="add index role.org_id" 2024-08-17T17:06:54.999 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.913899118Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-08-17T17:06:54.999 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.914354328Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-08-17T17:06:54.999 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.914781795Z level=info msg="Executing migration" id="create team role table" 2024-08-17T17:06:55.000 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.915200229Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-08-17T17:06:55.000 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.915625852Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-08-17T17:06:55.000 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.916073082Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-08-17T17:06:55.000 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.916495408Z level=info msg="Executing migration" id="create user role table" 2024-08-17T17:06:55.000 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.916873842Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-08-17T17:06:55.000 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.917315244Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-08-17T17:06:55.000 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.91773522Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-08-17T17:06:55.000 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.918155225Z level=info msg="Executing migration" id="create builtin role table" 2024-08-17T17:06:55.000 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.918568708Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-08-17T17:06:55.000 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.9189932Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-08-17T17:06:55.000 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.91941573Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-08-17T17:06:55.000 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.920386482Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-08-17T17:06:55.001 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.920856185Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-08-17T17:06:55.001 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.921313586Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-08-17T17:06:55.001 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.921681112Z level=info msg="Executing migration" id="add unique index role.uid" 2024-08-17T17:06:55.001 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.922124812Z level=info msg="Executing migration" id="create seed assignment table" 2024-08-17T17:06:55.001 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.922507258Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-08-17T17:06:55.001 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.922957805Z level=info msg="Executing migration" id="add column hidden to role table" 2024-08-17T17:06:55.001 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.923924177Z level=info msg="Executing migration" id="create query_history table v1" 2024-08-17T17:06:55.001 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.924324009Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-08-17T17:06:55.001 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.924759706Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-08-17T17:06:55.001 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.92503788Z level=info msg="Executing migration" id="teams permissions migration" 2024-08-17T17:06:55.001 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.925418304Z level=info msg="Executing migration" id="dashboard permissions" 2024-08-17T17:06:55.001 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.925811356Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-08-17T17:06:55.001 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.926162228Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-08-17T17:06:55.002 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.926440429Z level=info msg="Executing migration" id="alerting notification permissions" 2024-08-17T17:06:55.002 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.926740285Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-08-17T17:06:55.002 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.927143629Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-08-17T17:06:55.002 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.927575778Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-08-17T17:06:55.002 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.928556219Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-08-17T17:06:55.002 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.928833409Z level=info msg="Executing migration" id="create correlation table v1" 2024-08-17T17:06:55.002 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.929232373Z level=info msg="Executing migration" id="add index correlations.uid" 2024-08-17T17:06:55.002 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.929660181Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-08-17T17:06:55.002 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.930079952Z level=info msg="Executing migration" id="add correlation config column" 2024-08-17T17:06:55.003 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.931071918Z level=info msg="Executing migration" id="create entity_events table" 2024-08-17T17:06:55.003 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.931514199Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-08-17T17:06:55.003 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.931923742Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-08-17T17:06:55.003 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.932010078Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-08-17T17:06:55.003 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.932275291Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-08-17T17:06:55.003 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.932354619Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-08-17T17:06:55.003 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.932568725Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-08-17T17:06:55.003 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.932939779Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-08-17T17:06:55.003 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.933346665Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-08-17T17:06:55.003 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.933767177Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-08-17T17:06:55.003 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.934187029Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-08-17T17:06:55.003 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.934590182Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-08-17T17:06:55.003 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.934968232Z level=info msg="Executing migration" id="Drop public config table" 2024-08-17T17:06:55.004 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.935402827Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-08-17T17:06:55.004 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.935778981Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-08-17T17:06:55.004 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.936225769Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-08-17T17:06:55.004 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.936680735Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-08-17T17:06:55.004 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.937293668Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-08-17T17:06:55.004 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.940344972Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-08-17T17:06:55.004 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.941346888Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-08-17T17:06:55.004 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.942351027Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-08-17T17:06:55.004 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.942646491Z level=info msg="Executing migration" id="add share column" 2024-08-17T17:06:55.004 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.943644816Z level=info msg="Executing migration" id="create default alerting folders" 2024-08-17T17:06:55.004 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.94412731Z level=info msg="Executing migration" id="create file table" 2024-08-17T17:06:55.004 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.944553343Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-08-17T17:06:55.004 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.94498411Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-08-17T17:06:55.005 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.945449993Z level=info msg="Executing migration" id="create file_meta table" 2024-08-17T17:06:55.005 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.945813122Z level=info msg="Executing migration" id="file table idx: path key" 2024-08-17T17:06:55.005 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.946257628Z level=info msg="Executing migration" id="set path collation in file table" 2024-08-17T17:06:55.005 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.946522936Z level=info msg="Executing migration" id="managed permissions migration" 2024-08-17T17:06:55.005 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.946829976Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-08-17T17:06:55.005 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.947133701Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-08-17T17:06:55.005 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.947561257Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-08-17T17:06:55.005 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.948592668Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-08-17T17:06:55.005 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.948879812Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-08-17T17:06:55.005 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.949324794Z level=info msg="Executing migration" id="update group index for alert rules" 2024-08-17T17:06:55.005 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.949651539Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-08-17T17:06:55.005 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.949942089Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-08-17T17:06:55.005 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.950283619Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-08-17T17:06:55.006 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.951296301Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-08-17T17:06:55.006 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.95231191Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-08-17T17:06:55.006 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.952731771Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-08-17T17:06:55.006 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.962796975Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-08-17T17:06:55.006 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.963298049Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-08-17T17:06:55.006 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.963746068Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-08-17T17:06:55.006 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.967672914Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-08-17T17:06:55.006 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.968005404Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-08-17T17:06:55.006 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.968346543Z level=info msg="Executing migration" id="create folder table" 2024-08-17T17:06:55.006 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.968710555Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-08-17T17:06:55.007 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.969170852Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-08-17T17:06:55.007 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.969620771Z level=info msg="Executing migration" id="Update folder title length" 2024-08-17T17:06:55.007 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.96985814Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-08-17T17:06:55.007 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=migrator t=2024-08-17T17:06:54.970293535Z level=info msg="migrations completed" performed=485 skipped=0 duration=257.045902ms 2024-08-17T17:06:55.007 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=sqlstore t=2024-08-17T17:06:54.970732534Z level=info msg="Created default organization" 2024-08-17T17:06:55.192 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:54 smithi003 ceph-mon[41530]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:55.192 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:54 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:55.193 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:54 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:55.193 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:54 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:55.193 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:54 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:55.193 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:54 smithi003 ceph-mon[30803]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:55.193 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:54 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:55.193 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:54 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:55.193 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:54 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:55.193 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:54 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:55.216 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=plugin.loader t=2024-08-17T17:06:55.031180046Z level=info msg="Plugin registered" pluginID=input 2024-08-17T17:06:55.216 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=plugin.loader t=2024-08-17T17:06:55.061181923Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-08-17T17:06:55.216 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=plugin.loader t=2024-08-17T17:06:55.061211093Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-08-17T17:06:55.216 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=secrets t=2024-08-17T17:06:55.061245058Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-08-17T17:06:55.216 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=query_data t=2024-08-17T17:06:55.062888962Z level=info msg="Query Service initialization" 2024-08-17T17:06:55.216 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=live.push_http t=2024-08-17T17:06:55.065137468Z level=info msg="Live Push Gateway initialization" 2024-08-17T17:06:55.568 INFO:teuthology.orchestra.run.smithi003.stdout:163208757270 2024-08-17T17:06:55.568 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-08-17T17:06:56.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:55 smithi053 ceph-mon[34699]: Deploying daemon node-exporter.a on smithi003 2024-08-17T17:06:56.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:55 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:06:56.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:55 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:56.217 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=infra.usagestats.collector t=2024-08-17T17:06:55.830362037Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-08-17T17:06:56.217 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=provisioning.datasources t=2024-08-17T17:06:55.830691908Z level=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-08-17T17:06:56.217 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=provisioning.datasources t=2024-08-17T17:06:55.863495531Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-08-17T17:06:56.217 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=provisioning.datasources t=2024-08-17T17:06:55.895350718Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-08-17T17:06:56.217 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=provisioning.alerting t=2024-08-17T17:06:55.917435697Z level=info msg="starting to provision alerting" 2024-08-17T17:06:56.217 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=provisioning.alerting t=2024-08-17T17:06:55.917461512Z level=info msg="finished to provision alerting" 2024-08-17T17:06:56.217 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=grafanaStorageLogger t=2024-08-17T17:06:55.917553237Z level=info msg="storage starting" 2024-08-17T17:06:56.217 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=ngalert.state.manager t=2024-08-17T17:06:55.917599927Z level=info msg="Warming state cache for startup" 2024-08-17T17:06:56.217 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=http.server t=2024-08-17T17:06:55.919358314Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-08-17T17:06:56.217 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=sqlstore.transactions t=2024-08-17T17:06:55.928109341Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-08-17T17:06:56.217 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=ngalert.state.manager t=2024-08-17T17:06:55.92829421Z level=info msg="State cache has been initialized" states=0 duration=10.693147ms 2024-08-17T17:06:56.217 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=ticker t=2024-08-17T17:06:55.92835497Z level=info msg=starting first_tick=2024-08-17T17:07:00Z 2024-08-17T17:06:56.217 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=ngalert.multiorg.alertmanager t=2024-08-17T17:06:55.928353358Z level=info msg="starting MultiOrg Alertmanager" 2024-08-17T17:06:56.217 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:06:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[63455]: logger=sqlstore.transactions t=2024-08-17T17:06:55.963336111Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-08-17T17:06:56.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:55 smithi003 ceph-mon[41530]: Deploying daemon node-exporter.a on smithi003 2024-08-17T17:06:56.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:55 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:06:56.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:55 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:56.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:55 smithi003 ceph-mon[30803]: Deploying daemon node-exporter.a on smithi003 2024-08-17T17:06:56.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:55 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:06:56.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:55 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:06:56.698 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:06:56 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:06:56] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-08-17T17:06:56.846 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:56 smithi003 ceph-mon[30803]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:56.847 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:56 smithi003 ceph-mon[41530]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:56.957 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:56 smithi053 ceph-mon[34699]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:59.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:06:58 smithi003 ceph-mon[41530]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:59.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:06:58 smithi003 ceph-mon[30803]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:59.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:06:58 smithi053 ceph-mon[34699]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:06:59.446 INFO:teuthology.orchestra.run.smithi003.stdout:128849018909 2024-08-17T17:06:59.446 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-08-17T17:07:01.067 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:00 smithi003 ceph-mon[41530]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:01.068 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:00 smithi003 ceph-mon[30803]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:01.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:00 smithi053 ceph-mon[34699]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:01.466 INFO:teuthology.orchestra.run.smithi003.stdout:107374182434 2024-08-17T17:07:01.466 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-08-17T17:07:03.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:02 smithi053 ceph-mon[34699]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:03.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:02 smithi003 ceph-mon[30803]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:03.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:02 smithi003 ceph-mon[41530]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:03.346 INFO:teuthology.orchestra.run.smithi003.stdout:77309411368 2024-08-17T17:07:03.347 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-08-17T17:07:04.523 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:04 smithi003 bash[71679]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-08-17T17:07:05.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:04 smithi003 ceph-mon[41530]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:05.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:04 smithi003 ceph-mon[30803]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:05.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:04 smithi053 ceph-mon[34699]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:05.264 INFO:teuthology.orchestra.run.smithi003.stdout:34359738417 2024-08-17T17:07:05.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-08-17T17:07:06.523 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:06 smithi003 bash[71679]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-08-17T17:07:06.716 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:07:06 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:07:06] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-08-17T17:07:07.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:06 smithi003 ceph-mon[30803]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:07.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:06 smithi003 ceph-mon[41530]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:07.024 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:06 smithi003 bash[71679]: Writing manifest to image destination 2024-08-17T17:07:07.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:06 smithi053 ceph-mon[34699]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:07.487 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 podman[71679]: 2024-08-17 17:07:07.203158786 +0000 UTC m=+4.637427595 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-08-17T17:07:07.487 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 podman[71679]: 2024-08-17 17:07:07.402998145 +0000 UTC m=+4.837266944 container create 6d9a244ec8ca400fd5c8ba934861a0c0605695a805bad9553a3f16e6e842f19a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-08-17T17:07:07.868 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 podman[71679]: 2024-08-17 17:07:07.758365694 +0000 UTC m=+5.192634494 container init 6d9a244ec8ca400fd5c8ba934861a0c0605695a805bad9553a3f16e6e842f19a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-08-17T17:07:07.868 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 podman[71679]: 2024-08-17 17:07:07.765840336 +0000 UTC m=+5.200109138 container start 6d9a244ec8ca400fd5c8ba934861a0c0605695a805bad9553a3f16e6e842f19a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-08-17T17:07:07.868 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.770Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-08-17T17:07:07.869 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.770Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-08-17T17:07:07.869 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.770Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-08-17T17:07:07.869 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.770Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-08-17T17:07:07.869 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-08-17T17:07:07.869 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-08-17T17:07:07.869 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-08-17T17:07:07.869 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=arp 2024-08-17T17:07:07.869 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=bcache 2024-08-17T17:07:07.869 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=bonding 2024-08-17T17:07:07.869 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=btrfs 2024-08-17T17:07:07.869 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=conntrack 2024-08-17T17:07:07.869 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=cpu 2024-08-17T17:07:07.870 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-08-17T17:07:07.870 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=diskstats 2024-08-17T17:07:07.870 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=dmi 2024-08-17T17:07:07.870 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=edac 2024-08-17T17:07:07.870 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=entropy 2024-08-17T17:07:07.870 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-08-17T17:07:07.870 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=filefd 2024-08-17T17:07:07.870 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=filesystem 2024-08-17T17:07:07.870 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=hwmon 2024-08-17T17:07:07.870 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=infiniband 2024-08-17T17:07:07.870 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=ipvs 2024-08-17T17:07:07.870 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=loadavg 2024-08-17T17:07:07.870 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=mdadm 2024-08-17T17:07:07.871 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=meminfo 2024-08-17T17:07:07.871 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=netclass 2024-08-17T17:07:07.871 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=netdev 2024-08-17T17:07:07.871 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=netstat 2024-08-17T17:07:07.871 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=nfs 2024-08-17T17:07:07.871 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=nfsd 2024-08-17T17:07:07.871 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=nvme 2024-08-17T17:07:07.871 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=os 2024-08-17T17:07:07.871 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-08-17T17:07:07.871 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=pressure 2024-08-17T17:07:07.871 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=rapl 2024-08-17T17:07:07.872 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=schedstat 2024-08-17T17:07:07.872 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=selinux 2024-08-17T17:07:07.872 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=sockstat 2024-08-17T17:07:07.872 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=softnet 2024-08-17T17:07:07.872 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=stat 2024-08-17T17:07:07.872 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=tapestats 2024-08-17T17:07:07.872 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=textfile 2024-08-17T17:07:07.872 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-08-17T17:07:07.872 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=time 2024-08-17T17:07:07.872 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-08-17T17:07:07.872 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=uname 2024-08-17T17:07:07.873 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=vmstat 2024-08-17T17:07:07.873 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=xfs 2024-08-17T17:07:07.873 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.771Z caller=node_exporter.go:117 level=info collector=zfs 2024-08-17T17:07:07.873 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.772Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-08-17T17:07:07.873 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[71991]: ts=2024-08-17T17:07:07.772Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-08-17T17:07:08.273 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 bash[71679]: 6d9a244ec8ca400fd5c8ba934861a0c0605695a805bad9553a3f16e6e842f19a 2024-08-17T17:07:08.273 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:07:07 smithi003 systemd[1]: Started Ceph node-exporter.a for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:07:08.697 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:07:08.697 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:07:08.697 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:07:08.734 INFO:teuthology.orchestra.run.smithi003.stdout:197568495636 2024-08-17T17:07:08.734 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-08-17T17:07:09.060 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:08 smithi053 bash[63717]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-08-17T17:07:09.086 INFO:teuthology.orchestra.run.smithi003.stdout:55834574892 2024-08-17T17:07:09.086 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-08-17T17:07:09.393 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:09 smithi003 ceph-mon[30803]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:09.393 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:09 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:09.393 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:09 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:09.393 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:09 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:09.393 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:09 smithi003 ceph-mon[30803]: Deploying daemon node-exporter.b on smithi053 2024-08-17T17:07:09.394 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:09 smithi003 ceph-mon[41530]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:09.394 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:09 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:09.394 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:09 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:09.394 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:09 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:09.394 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:09 smithi003 ceph-mon[41530]: Deploying daemon node-exporter.b on smithi053 2024-08-17T17:07:09.467 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:09 smithi053 bash[63717]: Getting image source signatures 2024-08-17T17:07:09.498 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:09 smithi053 bash[63717]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-08-17T17:07:09.498 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:09 smithi053 bash[63717]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-08-17T17:07:09.499 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:09 smithi053 bash[63717]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-08-17T17:07:09.499 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:09 smithi053 ceph-mon[34699]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:09.499 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:09 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:09.499 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:09 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:09.499 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:09 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:09.499 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:09 smithi053 ceph-mon[34699]: Deploying daemon node-exporter.b on smithi053 2024-08-17T17:07:10.967 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:10 smithi053 bash[63717]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-08-17T17:07:10.967 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:10 smithi053 bash[63717]: Writing manifest to image destination 2024-08-17T17:07:11.302 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-mon[34699]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:11.302 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:07:11.303 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 podman[63717]: 2024-08-17 17:07:10.993447606 +0000 UTC m=+2.114402862 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-08-17T17:07:11.303 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 podman[63717]: 2024-08-17 17:07:11.100217408 +0000 UTC m=+2.221172709 container create 791e642c268d628edb7c965de3ba235301f5a3faedc02ea3a3adcb6e89a25c87 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-08-17T17:07:11.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:11 smithi003 ceph-mon[41530]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:11.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:11 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:07:11.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:11 smithi003 ceph-mon[30803]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:11.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:11 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:07:11.717 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 podman[63717]: 2024-08-17 17:07:11.302265735 +0000 UTC m=+2.423221002 container init 791e642c268d628edb7c965de3ba235301f5a3faedc02ea3a3adcb6e89a25c87 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-08-17T17:07:11.717 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 podman[63717]: 2024-08-17 17:07:11.305775028 +0000 UTC m=+2.426730292 container start 791e642c268d628edb7c965de3ba235301f5a3faedc02ea3a3adcb6e89a25c87 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-08-17T17:07:11.717 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.310Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-08-17T17:07:11.717 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.310Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-08-17T17:07:11.717 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.311Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-08-17T17:07:11.717 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.311Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-08-17T17:07:11.717 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.311Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-08-17T17:07:11.717 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.311Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-08-17T17:07:11.717 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.311Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-08-17T17:07:11.718 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.311Z caller=node_exporter.go:117 level=info collector=arp 2024-08-17T17:07:11.718 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.311Z caller=node_exporter.go:117 level=info collector=bcache 2024-08-17T17:07:11.718 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.311Z caller=node_exporter.go:117 level=info collector=bonding 2024-08-17T17:07:11.718 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.311Z caller=node_exporter.go:117 level=info collector=btrfs 2024-08-17T17:07:11.718 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.311Z caller=node_exporter.go:117 level=info collector=conntrack 2024-08-17T17:07:11.718 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.311Z caller=node_exporter.go:117 level=info collector=cpu 2024-08-17T17:07:11.718 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.311Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-08-17T17:07:11.719 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.311Z caller=node_exporter.go:117 level=info collector=diskstats 2024-08-17T17:07:11.719 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.311Z caller=node_exporter.go:117 level=info collector=dmi 2024-08-17T17:07:11.719 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.311Z caller=node_exporter.go:117 level=info collector=edac 2024-08-17T17:07:11.719 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=entropy 2024-08-17T17:07:11.719 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-08-17T17:07:11.719 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=filefd 2024-08-17T17:07:11.719 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=filesystem 2024-08-17T17:07:11.719 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=hwmon 2024-08-17T17:07:11.719 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=infiniband 2024-08-17T17:07:11.719 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=ipvs 2024-08-17T17:07:11.719 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=loadavg 2024-08-17T17:07:11.719 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=mdadm 2024-08-17T17:07:11.719 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=meminfo 2024-08-17T17:07:11.720 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=netclass 2024-08-17T17:07:11.720 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=netdev 2024-08-17T17:07:11.720 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=netstat 2024-08-17T17:07:11.720 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=nfs 2024-08-17T17:07:11.720 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=nfsd 2024-08-17T17:07:11.720 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=nvme 2024-08-17T17:07:11.720 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=os 2024-08-17T17:07:11.720 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-08-17T17:07:11.720 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=pressure 2024-08-17T17:07:11.720 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=rapl 2024-08-17T17:07:11.720 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=schedstat 2024-08-17T17:07:11.720 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=selinux 2024-08-17T17:07:11.720 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=sockstat 2024-08-17T17:07:11.720 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=softnet 2024-08-17T17:07:11.721 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=stat 2024-08-17T17:07:11.721 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=tapestats 2024-08-17T17:07:11.721 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=textfile 2024-08-17T17:07:11.721 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-08-17T17:07:11.721 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=time 2024-08-17T17:07:11.721 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-08-17T17:07:11.721 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=uname 2024-08-17T17:07:11.721 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=vmstat 2024-08-17T17:07:11.721 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=xfs 2024-08-17T17:07:11.721 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=node_exporter.go:117 level=info collector=zfs 2024-08-17T17:07:11.721 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-08-17T17:07:11.721 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[63770]: ts=2024-08-17T17:07:11.312Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-08-17T17:07:11.722 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 bash[63717]: 791e642c268d628edb7c965de3ba235301f5a3faedc02ea3a3adcb6e89a25c87 2024-08-17T17:07:11.722 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:07:11 smithi053 systemd[1]: Started Ceph node-exporter.b for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:07:12.492 INFO:teuthology.orchestra.run.smithi003.stdout:231928234001 2024-08-17T17:07:12.492 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-08-17T17:07:12.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:12 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:12.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:12 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:12.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:12 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:12.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:12 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:12.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:12 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:12.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:12 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:07:12.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:12 smithi053 ceph-mon[34699]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:12.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:12 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:12.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:12 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:12.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:12 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:12.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:12 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:12.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:12 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:12.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:12 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:07:12.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:12 smithi003 ceph-mon[30803]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:12.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:12 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:12.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:12 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:12.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:12 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:12.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:12 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:12.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:12 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:12.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:12 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:07:12.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:12 smithi003 ceph-mon[41530]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:12.940 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:07:13.454 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:07:15.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:14 smithi053 ceph-mon[34699]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:15.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:14 smithi003 ceph-mon[41530]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:15.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:14 smithi003 ceph-mon[30803]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:15.618 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:07:15.618 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:07:15.807 INFO:teuthology.orchestra.run.smithi003.stdout:128849018912 2024-08-17T17:07:16.076 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:15 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:16.076 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:15 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:16.076 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:15 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/2522548756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-08-17T17:07:16.076 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:15 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:16.077 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:15 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:16.077 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:15 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:16.077 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:15 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/2522548756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-08-17T17:07:16.077 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:15 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:16.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:15 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:16.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:15 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:16.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:15 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/2522548756' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-08-17T17:07:16.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:15 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:16.716 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:07:16 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:07:16] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-08-17T17:07:17.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:16 smithi003 ceph-mon[30803]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:17.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:16 smithi003 ceph-mon[41530]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:17.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:16 smithi053 ceph-mon[34699]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:18.777 INFO:teuthology.orchestra.run.smithi003.stdout:77309411371 2024-08-17T17:07:19.110 INFO:teuthology.orchestra.run.smithi003.stdout:107374182438 2024-08-17T17:07:19.110 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:18 smithi003 ceph-mon[41530]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:19.110 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:18 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/2490352088' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-08-17T17:07:19.110 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:18 smithi003 ceph-mon[30803]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:19.111 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:18 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/2490352088' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-08-17T17:07:19.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:18 smithi053 ceph-mon[34699]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:19.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:18 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/2490352088' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-08-17T17:07:19.755 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182434 got 107374182438 for osd.3 2024-08-17T17:07:19.755 DEBUG:teuthology.parallel:result is None 2024-08-17T17:07:20.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:19 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/1237617750' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-08-17T17:07:20.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:19 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/1237617750' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-08-17T17:07:20.362 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018909 got 128849018912 for osd.4 2024-08-17T17:07:20.362 DEBUG:teuthology.parallel:result is None 2024-08-17T17:07:20.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:19 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/1237617750' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-08-17T17:07:21.045 INFO:teuthology.orchestra.run.smithi003.stdout:34359738420 2024-08-17T17:07:21.213 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:20 smithi003 ceph-mon[41530]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:21.213 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:20 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/3903957481' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-08-17T17:07:21.213 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:20 smithi003 ceph-mon[30803]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:21.213 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:20 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/3903957481' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-08-17T17:07:21.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:20 smithi053 ceph-mon[34699]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:21.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:20 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/3903957481' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-08-17T17:07:21.540 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411368 got 77309411371 for osd.2 2024-08-17T17:07:21.540 DEBUG:teuthology.parallel:result is None 2024-08-17T17:07:21.588 INFO:teuthology.orchestra.run.smithi003.stdout:163208757275 2024-08-17T17:07:22.102 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:22 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/2712989848' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-08-17T17:07:22.102 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:22 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/2712989848' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-08-17T17:07:22.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:22 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/2712989848' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-08-17T17:07:22.491 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757270 got 163208757275 for osd.5 2024-08-17T17:07:22.492 DEBUG:teuthology.parallel:result is None 2024-08-17T17:07:22.827 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738417 got 34359738420 for osd.0 2024-08-17T17:07:22.827 DEBUG:teuthology.parallel:result is None 2024-08-17T17:07:23.215 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:23 smithi003 ceph-mon[30803]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:23.216 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:23 smithi003 ceph-mon[41530]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:23.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:23 smithi053 ceph-mon[34699]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:24.278 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:07:25.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:25 smithi003 ceph-mon[41530]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:25.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:25 smithi003 ceph-mon[30803]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:25.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:25 smithi053 ceph-mon[34699]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:25.680 INFO:teuthology.orchestra.run.smithi003.stdout:197568495639 2024-08-17T17:07:26.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:26 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/997876365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-08-17T17:07:26.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:26 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:07:26.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:26 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/997876365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-08-17T17:07:26.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:26 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:07:26.312 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:26 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/997876365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-08-17T17:07:26.312 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:26 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:07:26.407 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495636 got 197568495639 for osd.6 2024-08-17T17:07:26.407 DEBUG:teuthology.parallel:result is None 2024-08-17T17:07:26.716 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:07:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:07:26] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-08-17T17:07:27.138 INFO:teuthology.orchestra.run.smithi003.stdout:55834574896 2024-08-17T17:07:27.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:27 smithi003 ceph-mon[30803]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:27.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:27 smithi003 ceph-mon[41530]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:27.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:27 smithi053 ceph-mon[34699]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:27.644 INFO:teuthology.orchestra.run.smithi003.stdout:231928234004 2024-08-17T17:07:27.968 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574892 got 55834574896 for osd.1 2024-08-17T17:07:27.968 DEBUG:teuthology.parallel:result is None 2024-08-17T17:07:28.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:28 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/3078074916' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-08-17T17:07:28.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:28 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/2715418113' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-08-17T17:07:28.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:28 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/3078074916' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-08-17T17:07:28.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:28 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/2715418113' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-08-17T17:07:28.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:28 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/3078074916' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-08-17T17:07:28.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:28 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/2715418113' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-08-17T17:07:28.611 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234001 got 231928234004 for osd.7 2024-08-17T17:07:28.611 DEBUG:teuthology.parallel:result is None 2024-08-17T17:07:28.611 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-08-17T17:07:28.611 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:07:29.216 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:29 smithi003 ceph-mon[41530]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:29.217 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:29 smithi003 ceph-mon[30803]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:29.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:29 smithi053 ceph-mon[34699]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:30.174 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:07:31.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:31 smithi003 ceph-mon[30803]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:31.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:31 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:31.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:31 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:31.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:31 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:31.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:31 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:07:31.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:31 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:31.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:31 smithi003 ceph-mon[41530]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:31.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:31 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:31.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:31 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:31.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:31 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:31.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:31 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:07:31.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:31 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:31.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:31 smithi053 ceph-mon[34699]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:31.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:31 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:31.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:31 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:31.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:31 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:31.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:31 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:07:31.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:31 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:32.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:32 smithi053 ceph-mon[34699]: Reconfiguring alertmanager.a (dependencies changed)... 2024-08-17T17:07:32.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:32 smithi053 ceph-mon[34699]: Reconfiguring daemon alertmanager.a on smithi003 2024-08-17T17:07:32.486 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:32 smithi003 ceph-mon[41530]: Reconfiguring alertmanager.a (dependencies changed)... 2024-08-17T17:07:32.486 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:32 smithi003 ceph-mon[41530]: Reconfiguring daemon alertmanager.a on smithi003 2024-08-17T17:07:32.487 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:32 smithi003 ceph-mon[30803]: Reconfiguring alertmanager.a (dependencies changed)... 2024-08-17T17:07:32.487 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:32 smithi003 ceph-mon[30803]: Reconfiguring daemon alertmanager.a on smithi003 2024-08-17T17:07:33.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:33 smithi053 ceph-mon[34699]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:33.500 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:33 smithi003 ceph-mon[41530]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:33.500 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:33 smithi003 ceph-mon[30803]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:34.434 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:07:34 smithi003 systemd[1]: Stopping Ceph alertmanager.a for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:07:34.460 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:07:34.460 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:07:35.051 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-08-17T17:07:33.774270+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":181,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131,"ondisk_log_size":131,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385796,"kb_used_data":6952,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747575420,"statfs":{"total":767960285184,"available":765517230080,"internally_reserved":0,"allocated":7118848,"data_stored":4028361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001832"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-08-17T17:06:41.669746+0000","last_change":"2024-08-17T17:06:34.535627+0000","last_active":"2024-08-17T17:06:41.669746+0000","last_peered":"2024-08-17T17:06:41.669746+0000","last_clean":"2024-08-17T17:06:41.669746+0000","last_became_active":"2024-08-17T17:06:29.779504+0000","last_became_peered":"2024-08-17T17:06:29.779504+0000","last_unstale":"2024-08-17T17:06:41.669746+0000","last_undegraded":"2024-08-17T17:06:41.669746+0000","last_fullsized":"2024-08-17T17:06:41.669746+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:21:33.158491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000362901,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-08-17T17:06:41.686916+0000","last_change":"2024-08-17T17:06:34.530258+0000","last_active":"2024-08-17T17:06:41.686916+0000","last_peered":"2024-08-17T17:06:41.686916+0000","last_clean":"2024-08-17T17:06:41.686916+0000","last_became_active":"2024-08-17T17:06:29.831452+0000","last_became_peered":"2024-08-17T17:06:29.831452+0000","last_unstale":"2024-08-17T17:06:41.686916+0000","last_undegraded":"2024-08-17T17:06:41.686916+0000","last_fullsized":"2024-08-17T17:06:41.686916+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:36:53.341363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027697000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-08-17T17:06:41.730247+0000","last_change":"2024-08-17T17:06:34.999020+0000","last_active":"2024-08-17T17:06:41.730247+0000","last_peered":"2024-08-17T17:06:41.730247+0000","last_clean":"2024-08-17T17:06:41.730247+0000","last_became_active":"2024-08-17T17:06:29.830625+0000","last_became_peered":"2024-08-17T17:06:29.830625+0000","last_unstale":"2024-08-17T17:06:41.730247+0000","last_undegraded":"2024-08-17T17:06:41.730247+0000","last_fullsized":"2024-08-17T17:06:41.730247+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:23:50.149657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000290981,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-08-17T17:06:41.686903+0000","last_change":"2024-08-17T17:06:34.530210+0000","last_active":"2024-08-17T17:06:41.686903+0000","last_peered":"2024-08-17T17:06:41.686903+0000","last_clean":"2024-08-17T17:06:41.686903+0000","last_became_active":"2024-08-17T17:06:29.779126+0000","last_became_peered":"2024-08-17T17:06:29.779126+0000","last_unstale":"2024-08-17T17:06:41.686903+0000","last_undegraded":"2024-08-17T17:06:41.686903+0000","last_fullsized":"2024-08-17T17:06:41.686903+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:53:27.601004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028588399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":21,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-08-17T17:06:41.408142+0000","last_change":"2024-08-17T17:06:34.534595+0000","last_active":"2024-08-17T17:06:41.408142+0000","last_peered":"2024-08-17T17:06:41.408142+0000","last_clean":"2024-08-17T17:06:41.408142+0000","last_became_active":"2024-08-17T17:06:29.777933+0000","last_became_peered":"2024-08-17T17:06:29.777933+0000","last_unstale":"2024-08-17T17:06:41.408142+0000","last_undegraded":"2024-08-17T17:06:41.408142+0000","last_fullsized":"2024-08-17T17:06:41.408142+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:33:08.360328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087151500000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-08-17T17:06:41.172225+0000","last_change":"2024-08-17T17:06:34.600672+0000","last_active":"2024-08-17T17:06:41.172225+0000","last_peered":"2024-08-17T17:06:41.172225+0000","last_clean":"2024-08-17T17:06:41.172225+0000","last_became_active":"2024-08-17T17:06:29.830105+0000","last_became_peered":"2024-08-17T17:06:29.830105+0000","last_unstale":"2024-08-17T17:06:41.172225+0000","last_undegraded":"2024-08-17T17:06:41.172225+0000","last_fullsized":"2024-08-17T17:06:41.172225+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:42:07.154002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000234334,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-08-17T17:06:41.730238+0000","last_change":"2024-08-17T17:06:34.999031+0000","last_active":"2024-08-17T17:06:41.730238+0000","last_peered":"2024-08-17T17:06:41.730238+0000","last_clean":"2024-08-17T17:06:41.730238+0000","last_became_active":"2024-08-17T17:06:29.785264+0000","last_became_peered":"2024-08-17T17:06:29.785264+0000","last_unstale":"2024-08-17T17:06:41.730238+0000","last_undegraded":"2024-08-17T17:06:41.730238+0000","last_fullsized":"2024-08-17T17:06:41.730238+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:59:39.460037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026064700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-08-17T17:06:41.408148+0000","last_change":"2024-08-17T17:06:34.534038+0000","last_active":"2024-08-17T17:06:41.408148+0000","last_peered":"2024-08-17T17:06:41.408148+0000","last_clean":"2024-08-17T17:06:41.408148+0000","last_became_active":"2024-08-17T17:06:29.779691+0000","last_became_peered":"2024-08-17T17:06:29.779691+0000","last_unstale":"2024-08-17T17:06:41.408148+0000","last_undegraded":"2024-08-17T17:06:41.408148+0000","last_fullsized":"2024-08-17T17:06:41.408148+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:51:04.192689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032951500000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"59'128","reported_seq":274,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-08-17T17:06:34.998668+0000","last_change":"2024-08-17T17:05:57.181832+0000","last_active":"2024-08-17T17:06:34.998668+0000","last_peered":"2024-08-17T17:06:34.998668+0000","last_clean":"2024-08-17T17:06:34.998668+0000","last_became_active":"2024-08-17T17:05:57.181461+0000","last_became_peered":"2024-08-17T17:05:57.181461+0000","last_unstale":"2024-08-17T17:06:34.998668+0000","last_undegraded":"2024-08-17T17:06:34.998668+0000","last_fullsized":"2024-08-17T17:06:34.998668+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:04:01.141401+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:10:53.855422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234006,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298584,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446568,"statfs":{"total":95995035648,"available":95689285632,"internally_reserved":0,"allocated":1257472,"data_stored":872342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:06:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.63600000000000001}]},{"osd":1,"last update":"Sat Aug 17 17:06:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.56799999999999995}]},{"osd":2,"last update":"Sat Aug 17 17:06:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.45400000000000001}]},{"osd":3,"last update":"Sat Aug 17 17:06:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.50900000000000001}]},{"osd":4,"last update":"Sat Aug 17 17:06:55 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.53400000000000003}]},{"osd":5,"last update":"Sat Aug 17 17:06:55 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.51900000000000002}]},{"osd":6,"last update":"Sat Aug 17 17:06:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.65900000000000003}]}]},{"osd":6,"up_from":46,"seq":197568495641,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298584,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446568,"statfs":{"total":95995035648,"available":95689285632,"internally_reserved":0,"allocated":1257472,"data_stored":872154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.505}]},{"osd":1,"last update":"Sat Aug 17 17:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.55100000000000005}]},{"osd":2,"last update":"Sat Aug 17 17:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.435},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.46899999999999997}]},{"osd":3,"last update":"Sat Aug 17 17:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.57599999999999996}]},{"osd":4,"last update":"Sat Aug 17 17:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.65500000000000003}]},{"osd":5,"last update":"Sat Aug 17 17:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.59299999999999997}]},{"osd":7,"last update":"Sat Aug 17 17:06:54 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.628},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.63900000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757278,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298008,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447144,"statfs":{"total":95995035648,"available":95689875456,"internally_reserved":0,"allocated":667648,"data_stored":281993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:07:12 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.59799999999999998,"15min":0.60999999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.60499999999999998,"15min":0.59899999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":3.5779999999999998,"5min":3.5779999999999998,"15min":3.5779999999999998},"last":0.62}]},{"osd":1,"last update":"Sat Aug 17 17:07:12 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.622,"15min":0.622},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":3.5950000000000002,"5min":3.5950000000000002,"15min":3.5950000000000002},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.56999999999999995,"15min":0.57899999999999996},"min":{"1min":0.36799999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.78100000000000003,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.503}]},{"osd":2,"last update":"Sat Aug 17 17:07:12 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.63700000000000001,"15min":0.63100000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.6160000000000001,"5min":3.6160000000000001,"15min":3.6160000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.621,"15min":0.63600000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.80700000000000005,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.63500000000000001}]},{"osd":3,"last update":"Sat Aug 17 17:07:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.60899999999999999,"15min":0.61899999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.61899999999999999,"15min":0.63500000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.57099999999999995}]},{"osd":4,"last update":"Sat Aug 17 17:07:12 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.432,"15min":0.42699999999999999},"min":{"1min":0.28899999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.69799999999999995,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.438,"15min":0.42799999999999999},"min":{"1min":0.32500000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.67700000000000005,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.36299999999999999}]},{"osd":6,"last update":"Sat Aug 17 17:06:32 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.32400000000000001}]},{"osd":7,"last update":"Sat Aug 17 17:06:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.377},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.38900000000000001}]}]},{"osd":4,"up_from":30,"seq":128849018916,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298008,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447144,"statfs":{"total":95995035648,"available":95689875456,"internally_reserved":0,"allocated":667648,"data_stored":282297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:06:52 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.51500000000000001,"15min":0.51900000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.69799999999999995,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.54100000000000004,"15min":0.54200000000000004},"min":{"1min":0.32500000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.66400000000000003}]},{"osd":1,"last update":"Sat Aug 17 17:06:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.54800000000000004,"15min":0.55300000000000005},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.38600000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.623}]},{"osd":2,"last update":"Sat Aug 17 17:06:52 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.53600000000000003,"15min":0.54400000000000004},"min":{"1min":0.373,"5min":0.313,"15min":0.313},"max":{"1min":0.67800000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52000000000000002,"15min":0.51700000000000002},"min":{"1min":0.41099999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.56599999999999995}]},{"osd":3,"last update":"Sat Aug 17 17:06:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.55500000000000005,"15min":0.56000000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.67500000000000004,"5min":0.752,"15min":0.752},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.34000000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.66900000000000004,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.69999999999999996}]},{"osd":5,"last update":"Sat Aug 17 17:07:09 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.40600000000000003,"15min":0.40400000000000003},"min":{"1min":0.30399999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.44500000000000001,"15min":0.45100000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.66400000000000003,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.47599999999999998}]},{"osd":6,"last update":"Sat Aug 17 17:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.436},{"interface":"front","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.55000000000000004}]},{"osd":7,"last update":"Sat Aug 17 17:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.45500000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182441,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298008,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447144,"statfs":{"total":95995035648,"available":95689875456,"internally_reserved":0,"allocated":667648,"data_stored":282297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.40000000000000002,"15min":0.40699999999999997},"min":{"1min":0.248,"5min":0.247,"15min":0.247},"max":{"1min":0.55000000000000004,"5min":0.62,"15min":0.62},"last":0.246},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.42099999999999999,"15min":0.42699999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.505,"5min":0.629,"15min":0.629},"last":0.28999999999999998}]},{"osd":1,"last update":"Sat Aug 17 17:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.44400000000000001,"15min":0.44700000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.42399999999999999,"15min":0.42899999999999999},"min":{"1min":0.308,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.56000000000000005,"5min":0.63,"15min":0.63},"last":0.32300000000000001}]},{"osd":2,"last update":"Sat Aug 17 17:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.45700000000000002,"15min":0.45900000000000002},"min":{"1min":0.34499999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.56999999999999995,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.442,"15min":0.44800000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.54400000000000004,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.308}]},{"osd":4,"last update":"Sat Aug 17 17:06:51 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56799999999999995,"15min":0.56599999999999995},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.57699999999999996,"15min":0.57299999999999995},"min":{"1min":0.40100000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.68100000000000005}]},{"osd":5,"last update":"Sat Aug 17 17:07:14 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.57699999999999996,"15min":0.58599999999999997},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72199999999999998,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.625},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.58499999999999996,"15min":0.59299999999999997},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83699999999999997,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.64100000000000001}]},{"osd":6,"last update":"Sat Aug 17 17:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.59299999999999997}]},{"osd":7,"last update":"Sat Aug 17 17:06:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.70899999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298008,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447144,"statfs":{"total":95995035648,"available":95689875456,"internally_reserved":0,"allocated":667648,"data_stored":282297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.39700000000000002,"15min":0.39600000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.39700000000000002,"15min":0.39300000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.39100000000000001}]},{"osd":1,"last update":"Sat Aug 17 17:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.40200000000000002,"15min":0.40400000000000003},"min":{"1min":0.26200000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56399999999999995,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.373},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.439,"15min":0.44800000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.35899999999999999}]},{"osd":3,"last update":"Sat Aug 17 17:06:30 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.496,"15min":0.503},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.47699999999999998,"15min":0.47899999999999998},"min":{"1min":0.34300000000000003,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.624,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.434}]},{"osd":4,"last update":"Sat Aug 17 17:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54100000000000004,"15min":0.54200000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57699999999999996,"15min":0.57899999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.83399999999999996,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.63700000000000001}]},{"osd":5,"last update":"Sat Aug 17 17:07:15 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57499999999999996,"15min":0.57699999999999996},"min":{"1min":0.40699999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.55200000000000005,"15min":0.55500000000000005},"min":{"1min":0.36399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.84599999999999997,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.69299999999999995}]},{"osd":6,"last update":"Sat Aug 17 17:06:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.71699999999999997}]},{"osd":7,"last update":"Sat Aug 17 17:06:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.74199999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298012,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447140,"statfs":{"total":95995035648,"available":95689871360,"internally_reserved":0,"allocated":671744,"data_stored":282316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.36399999999999999,"15min":0.33200000000000002},"min":{"1min":0.23899999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.60599999999999998,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.437},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.35699999999999998,"15min":0.33100000000000002},"min":{"1min":0.28100000000000003,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.55800000000000005,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.41699999999999998}]},{"osd":2,"last update":"Sat Aug 17 17:07:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.379,"15min":0.36299999999999999},"min":{"1min":0.29399999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.61899999999999999,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.372,"15min":0.35799999999999998},"min":{"1min":0.25800000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.54200000000000004,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.38500000000000001}]},{"osd":3,"last update":"Sat Aug 17 17:07:27 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.46800000000000003,"15min":0.47599999999999998},"min":{"1min":0.33100000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.55800000000000005,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.45800000000000002,"15min":0.46700000000000003},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.57299999999999995,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.495}]},{"osd":4,"last update":"Sat Aug 17 17:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58999999999999997,"15min":0.58899999999999997},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60399999999999998,"15min":0.60299999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.55500000000000005}]},{"osd":5,"last update":"Sat Aug 17 17:07:09 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.63200000000000001,"15min":0.63800000000000001},"min":{"1min":0.39900000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.82199999999999995,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.59299999999999997,"15min":0.59499999999999997},"min":{"1min":0.42399999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.84799999999999998,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.51100000000000001}]},{"osd":6,"last update":"Sat Aug 17 17:06:35 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.53300000000000003}]},{"osd":7,"last update":"Sat Aug 17 17:06:54 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.67100000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738423,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298584,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446568,"statfs":{"total":95995035648,"available":95689285632,"internally_reserved":0,"allocated":1261568,"data_stored":872665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 17 17:06:40 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.33100000000000002,"15min":0.29799999999999999},"min":{"1min":0.23499999999999999,"5min":0.192,"15min":0.192},"max":{"1min":0.56399999999999995,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.34100000000000003,"15min":0.309},"min":{"1min":0.27300000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.34799999999999998}]},{"osd":2,"last update":"Sat Aug 17 17:07:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.377,"15min":0.35999999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.40899999999999997,"15min":0.39900000000000002},"min":{"1min":0.30299999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.30199999999999999}]},{"osd":3,"last update":"Sat Aug 17 17:07:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.433,"15min":0.441},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.374},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.47099999999999997,"15min":0.48999999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.33600000000000002}]},{"osd":4,"last update":"Sat Aug 17 17:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.59199999999999997,"15min":0.59599999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.755,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.499},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.59299999999999997,"15min":0.59899999999999998},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.72899999999999998,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.59499999999999997}]},{"osd":5,"last update":"Sat Aug 17 17:07:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.57999999999999996,"15min":0.58799999999999997},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.75800000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.59599999999999997,"15min":0.60599999999999998},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.69799999999999995,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.65300000000000002}]},{"osd":6,"last update":"Sat Aug 17 17:06:35 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.73199999999999998}]},{"osd":7,"last update":"Sat Aug 17 17:06:54 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.56999999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:07:35.076 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:07:35.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:35 smithi053 ceph-mon[34699]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:35.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:35 smithi003 ceph-mon[41530]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:35.523 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:07:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a[69785]: ts=2024-08-17T17:07:35.189Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-08-17T17:07:35.523 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:07:35 smithi003 podman[74025]: 2024-08-17 17:07:35.210356335 +0000 UTC m=+0.170433478 container died 59c6ba22282d06aee6bc60d90e538804785a0de214798e3c8b0c90600520d1dc (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-08-17T17:07:35.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:35 smithi003 ceph-mon[30803]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:35.849 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:07:35 smithi003 podman[74025]: 2024-08-17 17:07:35.573238377 +0000 UTC m=+0.533315523 container cleanup 59c6ba22282d06aee6bc60d90e538804785a0de214798e3c8b0c90600520d1dc (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-08-17T17:07:35.849 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:07:35 smithi003 bash[74025]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a 2024-08-17T17:07:36.026 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:07:36.108 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:07:35 smithi003 podman[74057]: 2024-08-17 17:07:35.849281454 +0000 UTC m=+0.634609086 container remove 59c6ba22282d06aee6bc60d90e538804785a0de214798e3c8b0c90600520d1dc (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-08-17T17:07:36.108 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:07:35 smithi003 podman[74057]: 2024-08-17 17:07:35.950105729 +0000 UTC m=+0.735433363 volume remove f4ae61d4d1e528734ff503e60bb624d9cc7491bc9e1d64ec03cc4df999b684d0 2024-08-17T17:07:36.403 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:36 smithi003 ceph-mon[41530]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:07:36.404 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:36 smithi003 ceph-mon[30803]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:07:36.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:36 smithi053 ceph-mon[34699]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:07:36.467 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:07:36 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:07:36] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-08-17T17:07:36.673 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:07:36 smithi003 podman[74124]: 2024-08-17 17:07:36.503907679 +0000 UTC m=+0.512414754 container died 5429c30766236d725dcaf2795b67b22dba1828bf0aef7f65b2956f08fcde79c3 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=eloquent_meninsky, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-f2c39d4, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-08-17T17:07:37.023 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:07:36 smithi003 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-08-17T17:07:37.023 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:07:36 smithi003 systemd[1]: Stopped Ceph alertmanager.a for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:07:37.023 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:07:36 smithi003 systemd[1]: Starting Ceph alertmanager.a for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:07:37.426 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:37 smithi003 ceph-mon[41530]: pgmap v45: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:37.426 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:37 smithi003 ceph-mon[30803]: pgmap v45: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:37.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:37 smithi053 ceph-mon[34699]: pgmap v45: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:37.773 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:07:37 smithi003 podman[74240]: 2024-08-17 17:07:37.426661288 +0000 UTC m=+0.198127373 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-08-17T17:07:37.773 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:07:37 smithi003 podman[74240]: 2024-08-17 17:07:37.56864072 +0000 UTC m=+0.340106807 volume create b73b516a3fa3119fc3cadb631209731b5b1e8036fb0a966098986130c85d8a58 2024-08-17T17:07:37.774 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:07:37 smithi003 podman[74240]: 2024-08-17 17:07:37.660736502 +0000 UTC m=+0.432202645 container create be92e7808bcf8cf80ad5f6bfd25d176e41bee230964774fb79534150fa6abd18 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-08-17T17:07:38.215 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:07:37 smithi003 podman[74240]: 2024-08-17 17:07:37.971914393 +0000 UTC m=+0.743380477 container init be92e7808bcf8cf80ad5f6bfd25d176e41bee230964774fb79534150fa6abd18 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-08-17T17:07:38.215 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:07:37 smithi003 podman[74240]: 2024-08-17 17:07:37.97496295 +0000 UTC m=+0.746429034 container start be92e7808bcf8cf80ad5f6bfd25d176e41bee230964774fb79534150fa6abd18 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-08-17T17:07:38.216 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:07:37 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a[74261]: ts=2024-08-17T17:07:37.987Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-08-17T17:07:38.216 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:07:37 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a[74261]: ts=2024-08-17T17:07:37.987Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-08-17T17:07:38.216 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:07:37 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a[74261]: ts=2024-08-17T17:07:37.988Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.3 port=9094 2024-08-17T17:07:38.216 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:07:37 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a[74261]: ts=2024-08-17T17:07:37.989Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-08-17T17:07:38.216 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:07:38 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a[74261]: ts=2024-08-17T17:07:38.013Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-08-17T17:07:38.216 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:07:38 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a[74261]: ts=2024-08-17T17:07:38.013Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-08-17T17:07:38.216 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:07:38 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a[74261]: ts=2024-08-17T17:07:38.016Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-08-17T17:07:38.216 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:07:38 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a[74261]: ts=2024-08-17T17:07:38.016Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-08-17T17:07:38.216 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:07:38 smithi003 bash[74240]: be92e7808bcf8cf80ad5f6bfd25d176e41bee230964774fb79534150fa6abd18 2024-08-17T17:07:38.216 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:07:38 smithi003 systemd[1]: Started Ceph alertmanager.a for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:07:38.467 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:07:38.468 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:07:39.133 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":46,"stamp":"2024-08-17T17:07:37.774813+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":181,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131,"ondisk_log_size":131,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385796,"kb_used_data":6952,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747575420,"statfs":{"total":767960285184,"available":765517230080,"internally_reserved":0,"allocated":7118848,"data_stored":4028361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001747"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-08-17T17:06:41.669746+0000","last_change":"2024-08-17T17:06:34.535627+0000","last_active":"2024-08-17T17:06:41.669746+0000","last_peered":"2024-08-17T17:06:41.669746+0000","last_clean":"2024-08-17T17:06:41.669746+0000","last_became_active":"2024-08-17T17:06:29.779504+0000","last_became_peered":"2024-08-17T17:06:29.779504+0000","last_unstale":"2024-08-17T17:06:41.669746+0000","last_undegraded":"2024-08-17T17:06:41.669746+0000","last_fullsized":"2024-08-17T17:06:41.669746+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:21:33.158491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000362901,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-08-17T17:06:41.686916+0000","last_change":"2024-08-17T17:06:34.530258+0000","last_active":"2024-08-17T17:06:41.686916+0000","last_peered":"2024-08-17T17:06:41.686916+0000","last_clean":"2024-08-17T17:06:41.686916+0000","last_became_active":"2024-08-17T17:06:29.831452+0000","last_became_peered":"2024-08-17T17:06:29.831452+0000","last_unstale":"2024-08-17T17:06:41.686916+0000","last_undegraded":"2024-08-17T17:06:41.686916+0000","last_fullsized":"2024-08-17T17:06:41.686916+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:36:53.341363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027697000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-08-17T17:06:41.730247+0000","last_change":"2024-08-17T17:06:34.999020+0000","last_active":"2024-08-17T17:06:41.730247+0000","last_peered":"2024-08-17T17:06:41.730247+0000","last_clean":"2024-08-17T17:06:41.730247+0000","last_became_active":"2024-08-17T17:06:29.830625+0000","last_became_peered":"2024-08-17T17:06:29.830625+0000","last_unstale":"2024-08-17T17:06:41.730247+0000","last_undegraded":"2024-08-17T17:06:41.730247+0000","last_fullsized":"2024-08-17T17:06:41.730247+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:23:50.149657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000290981,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-08-17T17:06:41.686903+0000","last_change":"2024-08-17T17:06:34.530210+0000","last_active":"2024-08-17T17:06:41.686903+0000","last_peered":"2024-08-17T17:06:41.686903+0000","last_clean":"2024-08-17T17:06:41.686903+0000","last_became_active":"2024-08-17T17:06:29.779126+0000","last_became_peered":"2024-08-17T17:06:29.779126+0000","last_unstale":"2024-08-17T17:06:41.686903+0000","last_undegraded":"2024-08-17T17:06:41.686903+0000","last_fullsized":"2024-08-17T17:06:41.686903+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:53:27.601004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028588399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":21,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-08-17T17:06:41.408142+0000","last_change":"2024-08-17T17:06:34.534595+0000","last_active":"2024-08-17T17:06:41.408142+0000","last_peered":"2024-08-17T17:06:41.408142+0000","last_clean":"2024-08-17T17:06:41.408142+0000","last_became_active":"2024-08-17T17:06:29.777933+0000","last_became_peered":"2024-08-17T17:06:29.777933+0000","last_unstale":"2024-08-17T17:06:41.408142+0000","last_undegraded":"2024-08-17T17:06:41.408142+0000","last_fullsized":"2024-08-17T17:06:41.408142+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:33:08.360328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087151500000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-08-17T17:06:41.172225+0000","last_change":"2024-08-17T17:06:34.600672+0000","last_active":"2024-08-17T17:06:41.172225+0000","last_peered":"2024-08-17T17:06:41.172225+0000","last_clean":"2024-08-17T17:06:41.172225+0000","last_became_active":"2024-08-17T17:06:29.830105+0000","last_became_peered":"2024-08-17T17:06:29.830105+0000","last_unstale":"2024-08-17T17:06:41.172225+0000","last_undegraded":"2024-08-17T17:06:41.172225+0000","last_fullsized":"2024-08-17T17:06:41.172225+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:42:07.154002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000234334,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-08-17T17:06:41.730238+0000","last_change":"2024-08-17T17:06:34.999031+0000","last_active":"2024-08-17T17:06:41.730238+0000","last_peered":"2024-08-17T17:06:41.730238+0000","last_clean":"2024-08-17T17:06:41.730238+0000","last_became_active":"2024-08-17T17:06:29.785264+0000","last_became_peered":"2024-08-17T17:06:29.785264+0000","last_unstale":"2024-08-17T17:06:41.730238+0000","last_undegraded":"2024-08-17T17:06:41.730238+0000","last_fullsized":"2024-08-17T17:06:41.730238+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:59:39.460037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026064700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":20,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-08-17T17:06:41.408148+0000","last_change":"2024-08-17T17:06:34.534038+0000","last_active":"2024-08-17T17:06:41.408148+0000","last_peered":"2024-08-17T17:06:41.408148+0000","last_clean":"2024-08-17T17:06:41.408148+0000","last_became_active":"2024-08-17T17:06:29.779691+0000","last_became_peered":"2024-08-17T17:06:29.779691+0000","last_unstale":"2024-08-17T17:06:41.408148+0000","last_undegraded":"2024-08-17T17:06:41.408148+0000","last_fullsized":"2024-08-17T17:06:41.408148+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:51:04.192689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032951500000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"59'128","reported_seq":274,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-08-17T17:06:34.998668+0000","last_change":"2024-08-17T17:05:57.181832+0000","last_active":"2024-08-17T17:06:34.998668+0000","last_peered":"2024-08-17T17:06:34.998668+0000","last_clean":"2024-08-17T17:06:34.998668+0000","last_became_active":"2024-08-17T17:05:57.181461+0000","last_became_peered":"2024-08-17T17:05:57.181461+0000","last_unstale":"2024-08-17T17:06:34.998668+0000","last_undegraded":"2024-08-17T17:06:34.998668+0000","last_fullsized":"2024-08-17T17:06:34.998668+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:04:01.141401+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:10:53.855422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234007,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298584,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446568,"statfs":{"total":95995035648,"available":95689285632,"internally_reserved":0,"allocated":1257472,"data_stored":872342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:06:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.53900000000000003}]},{"osd":1,"last update":"Sat Aug 17 17:06:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.55900000000000005}]},{"osd":2,"last update":"Sat Aug 17 17:06:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.52600000000000002}]},{"osd":3,"last update":"Sat Aug 17 17:06:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.505},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.56999999999999995}]},{"osd":4,"last update":"Sat Aug 17 17:06:55 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.43099999999999999}]},{"osd":5,"last update":"Sat Aug 17 17:06:55 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.51500000000000001}]},{"osd":6,"last update":"Sat Aug 17 17:06:55 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.47299999999999998}]}]},{"osd":6,"up_from":46,"seq":197568495642,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298584,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446568,"statfs":{"total":95995035648,"available":95689285632,"internally_reserved":0,"allocated":1257472,"data_stored":872154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.52400000000000002,"15min":0.52300000000000002},"min":{"1min":0.34999999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52500000000000002,"15min":0.52300000000000002},"min":{"1min":0.40400000000000003,"5min":0.373,"15min":0.373},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.58999999999999997}]},{"osd":1,"last update":"Sat Aug 17 17:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55500000000000005,"15min":0.55300000000000005},"min":{"1min":0.39300000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.51000000000000001,"15min":0.50600000000000001},"min":{"1min":0.40100000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.61399999999999999}]},{"osd":2,"last update":"Sat Aug 17 17:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.374,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.54500000000000004,"15min":0.54100000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.69099999999999995}]},{"osd":3,"last update":"Sat Aug 17 17:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.52600000000000002,"15min":0.52800000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.373,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.74299999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.746}]},{"osd":4,"last update":"Sat Aug 17 17:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.44,"15min":0.442},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.69999999999999996,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.45200000000000001,"15min":0.45300000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.65500000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.43099999999999999}]},{"osd":5,"last update":"Sat Aug 17 17:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.46200000000000002,"15min":0.45900000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.64400000000000002,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.46500000000000002,"15min":0.46100000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.68799999999999994,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.53400000000000003}]},{"osd":7,"last update":"Sat Aug 17 17:06:54 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.63500000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757279,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298008,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447144,"statfs":{"total":95995035648,"available":95689875456,"internally_reserved":0,"allocated":667648,"data_stored":281993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:07:12 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.59799999999999998,"15min":0.60999999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.60499999999999998,"15min":0.59899999999999998},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":3.5779999999999998,"5min":3.5779999999999998,"15min":3.5779999999999998},"last":0.55300000000000005}]},{"osd":1,"last update":"Sat Aug 17 17:07:12 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.622,"15min":0.622},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":3.5950000000000002,"5min":3.5950000000000002,"15min":3.5950000000000002},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.56999999999999995,"15min":0.57899999999999996},"min":{"1min":0.36799999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.78100000000000003,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.60099999999999998}]},{"osd":2,"last update":"Sat Aug 17 17:07:12 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.63700000000000001,"15min":0.63100000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.6160000000000001,"5min":3.6160000000000001,"15min":3.6160000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.621,"15min":0.63600000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.80700000000000005,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.56799999999999995}]},{"osd":3,"last update":"Sat Aug 17 17:07:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.60899999999999999,"15min":0.61899999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.61899999999999999,"15min":0.63500000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.61699999999999999}]},{"osd":4,"last update":"Sat Aug 17 17:07:12 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.432,"15min":0.42699999999999999},"min":{"1min":0.28899999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.69799999999999995,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.438,"15min":0.42799999999999999},"min":{"1min":0.32500000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.67700000000000005,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.41799999999999998}]},{"osd":6,"last update":"Sat Aug 17 17:07:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.47599999999999998,"15min":0.46999999999999997},"min":{"1min":0.34999999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.69999999999999996,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.47799999999999998,"15min":0.47399999999999998},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.68100000000000005,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.374}]},{"osd":7,"last update":"Sat Aug 17 17:06:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.318},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.39900000000000002}]}]},{"osd":4,"up_from":30,"seq":128849018917,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298008,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447144,"statfs":{"total":95995035648,"available":95689875456,"internally_reserved":0,"allocated":667648,"data_stored":282297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:06:52 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.51500000000000001,"15min":0.51900000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.69799999999999995,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.54100000000000004,"15min":0.54200000000000004},"min":{"1min":0.32500000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.499}]},{"osd":1,"last update":"Sat Aug 17 17:06:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.54800000000000004,"15min":0.55300000000000005},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.38600000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.60499999999999998}]},{"osd":2,"last update":"Sat Aug 17 17:06:52 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.53600000000000003,"15min":0.54400000000000004},"min":{"1min":0.373,"5min":0.313,"15min":0.313},"max":{"1min":0.67800000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52000000000000002,"15min":0.51700000000000002},"min":{"1min":0.41099999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.56899999999999995}]},{"osd":3,"last update":"Sat Aug 17 17:06:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.55500000000000005,"15min":0.56000000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.67500000000000004,"5min":0.752,"15min":0.752},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.34000000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.66900000000000004,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.58099999999999996}]},{"osd":5,"last update":"Sat Aug 17 17:07:09 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.40600000000000003,"15min":0.40400000000000003},"min":{"1min":0.30399999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.437},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.44500000000000001,"15min":0.45100000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.66400000000000003,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.40600000000000003}]},{"osd":6,"last update":"Sat Aug 17 17:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.46999999999999997,"15min":0.46800000000000003},"min":{"1min":0.32000000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.45000000000000001,"15min":0.443},"min":{"1min":0.318,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.35499999999999998}]},{"osd":7,"last update":"Sat Aug 17 17:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.629}]}]},{"osd":3,"up_from":25,"seq":107374182442,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298008,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447144,"statfs":{"total":95995035648,"available":95689875456,"internally_reserved":0,"allocated":667648,"data_stored":282297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:07:35 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.38600000000000001,"15min":0.40200000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.59599999999999997,"5min":0.62,"15min":0.62},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.40300000000000002,"15min":0.42099999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.501,"5min":0.629,"15min":0.629},"last":0.25600000000000001}]},{"osd":1,"last update":"Sat Aug 17 17:07:35 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.42399999999999999,"15min":0.44},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.49099999999999999,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.41599999999999998,"15min":0.42599999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.59399999999999997,"5min":0.63,"15min":0.63},"last":0.33700000000000002}]},{"osd":2,"last update":"Sat Aug 17 17:07:35 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.44400000000000001,"15min":0.45500000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.55100000000000005,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.374,"5min":0.42699999999999999,"15min":0.443},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.52700000000000002,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.39200000000000002}]},{"osd":4,"last update":"Sat Aug 17 17:06:51 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56799999999999995,"15min":0.56599999999999995},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.57699999999999996,"15min":0.57299999999999995},"min":{"1min":0.40100000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.58899999999999997}]},{"osd":5,"last update":"Sat Aug 17 17:07:14 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.57699999999999996,"15min":0.58599999999999997},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.72199999999999998,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.58499999999999996,"15min":0.59299999999999997},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.83699999999999997,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.53700000000000003}]},{"osd":6,"last update":"Sat Aug 17 17:07:35 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.58099999999999996,"15min":0.57899999999999996},"min":{"1min":0.38800000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.71999999999999997,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.61599999999999999}]},{"osd":7,"last update":"Sat Aug 17 17:06:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.64400000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298008,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447144,"statfs":{"total":95995035648,"available":95689875456,"internally_reserved":0,"allocated":667648,"data_stored":282297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.39700000000000002,"15min":0.39600000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.39700000000000002,"15min":0.39300000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.56200000000000006}]},{"osd":1,"last update":"Sat Aug 17 17:07:06 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.40200000000000002,"15min":0.40400000000000003},"min":{"1min":0.26200000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56399999999999995,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.439,"15min":0.44800000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.77200000000000002}]},{"osd":3,"last update":"Sat Aug 17 17:07:32 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.48299999999999998,"15min":0.499},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.45900000000000002,"15min":0.47299999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.56100000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.49199999999999999}]},{"osd":4,"last update":"Sat Aug 17 17:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54100000000000004,"15min":0.54200000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57699999999999996,"15min":0.57899999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.83399999999999996,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.72799999999999998}]},{"osd":5,"last update":"Sat Aug 17 17:07:15 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57499999999999996,"15min":0.57699999999999996},"min":{"1min":0.40699999999999997,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.55200000000000005,"15min":0.55500000000000005},"min":{"1min":0.36399999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.84599999999999997,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.70999999999999996}]},{"osd":6,"last update":"Sat Aug 17 17:06:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.65700000000000003}]},{"osd":7,"last update":"Sat Aug 17 17:06:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.746}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298012,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447140,"statfs":{"total":95995035648,"available":95689871360,"internally_reserved":0,"allocated":671744,"data_stored":282316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.36399999999999999,"15min":0.33200000000000002},"min":{"1min":0.23899999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.60599999999999998,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.35699999999999998,"15min":0.33100000000000002},"min":{"1min":0.28100000000000003,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.55800000000000005,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.28100000000000003}]},{"osd":2,"last update":"Sat Aug 17 17:07:09 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.379,"15min":0.36299999999999999},"min":{"1min":0.29399999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.61899999999999999,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.372,"15min":0.35799999999999998},"min":{"1min":0.25800000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.54200000000000004,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.32900000000000001}]},{"osd":3,"last update":"Sat Aug 17 17:07:27 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.46800000000000003,"15min":0.47599999999999998},"min":{"1min":0.33100000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.55800000000000005,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.45800000000000002,"15min":0.46700000000000003},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.57299999999999995,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.45800000000000002}]},{"osd":4,"last update":"Sat Aug 17 17:06:47 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.58999999999999997,"15min":0.58899999999999997},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.498},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60399999999999998,"15min":0.60299999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.54500000000000004}]},{"osd":5,"last update":"Sat Aug 17 17:07:09 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.63200000000000001,"15min":0.63800000000000001},"min":{"1min":0.39900000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.82199999999999995,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.59299999999999997,"15min":0.59499999999999997},"min":{"1min":0.42399999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.84799999999999998,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.58899999999999997}]},{"osd":6,"last update":"Sat Aug 17 17:07:36 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.64900000000000002,"15min":0.66000000000000003},"min":{"1min":0.45400000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.68799999999999994,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.61199999999999999,"15min":0.61799999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.77300000000000002,"5min":0.877,"15min":0.877},"last":0.64700000000000002}]},{"osd":7,"last update":"Sat Aug 17 17:06:54 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.53000000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298584,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446568,"statfs":{"total":95995035648,"available":95689285632,"internally_reserved":0,"allocated":1261568,"data_stored":872665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 17 17:06:40 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.33100000000000002,"15min":0.29799999999999999},"min":{"1min":0.23499999999999999,"5min":0.192,"15min":0.192},"max":{"1min":0.56399999999999995,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.34100000000000003,"15min":0.309},"min":{"1min":0.27300000000000002,"5min":0.223,"15min":0.223},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.28999999999999998}]},{"osd":2,"last update":"Sat Aug 17 17:07:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.377,"15min":0.35999999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.40899999999999997,"15min":0.39900000000000002},"min":{"1min":0.30299999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.27800000000000002}]},{"osd":3,"last update":"Sat Aug 17 17:07:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.433,"15min":0.441},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.47099999999999997,"15min":0.48999999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.255}]},{"osd":4,"last update":"Sat Aug 17 17:06:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.59199999999999997,"15min":0.59599999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.755,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.59299999999999997,"15min":0.59899999999999998},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.72899999999999998,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.627}]},{"osd":5,"last update":"Sat Aug 17 17:07:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.57999999999999996,"15min":0.58799999999999997},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.75800000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.59599999999999997,"15min":0.60599999999999998},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.69799999999999995,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.66500000000000004}]},{"osd":6,"last update":"Sat Aug 17 17:07:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.60299999999999998,"15min":0.60799999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.74299999999999999,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.69699999999999995}]},{"osd":7,"last update":"Sat Aug 17 17:06:54 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.68300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:07:39.135 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-08-17T17:07:39.135 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-08-17T17:07:39.135 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-08-17T17:07:39.135 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph health --format=json 2024-08-17T17:07:39.407 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:07:39.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:39 smithi053 ceph-mon[34699]: pgmap v46: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:39.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:39 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:39.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:39 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:39.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:39 smithi053 ceph-mon[34699]: Reconfiguring prometheus.a (dependencies changed)... 2024-08-17T17:07:39.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:39 smithi053 ceph-mon[34699]: Reconfiguring daemon prometheus.a on smithi053 2024-08-17T17:07:39.520 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:39 smithi003 ceph-mon[41530]: pgmap v46: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:39.521 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:39 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:39.521 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:39 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:39.521 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:39 smithi003 ceph-mon[41530]: Reconfiguring prometheus.a (dependencies changed)... 2024-08-17T17:07:39.521 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:39 smithi003 ceph-mon[41530]: Reconfiguring daemon prometheus.a on smithi053 2024-08-17T17:07:39.521 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:39 smithi003 ceph-mon[30803]: pgmap v46: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:39.521 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:39 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:39.521 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:39 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:39.521 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:39 smithi003 ceph-mon[30803]: Reconfiguring prometheus.a (dependencies changed)... 2024-08-17T17:07:39.521 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:39 smithi003 ceph-mon[30803]: Reconfiguring daemon prometheus.a on smithi053 2024-08-17T17:07:40.107 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:07:39 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a[74261]: ts=2024-08-17T17:07:39.990Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000546038s 2024-08-17T17:07:40.177 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:40 smithi053 ceph-mon[34699]: from='client.24542 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:07:40.410 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:40 smithi003 ceph-mon[41530]: from='client.24542 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:07:40.411 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:40 smithi003 ceph-mon[30803]: from='client.24542 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:07:41.346 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:07:41.346 INFO:teuthology.orchestra.run.smithi003.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-08-17T17:07:41.443 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:41 smithi053 ceph-mon[34699]: pgmap v47: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:41.443 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:41 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:07:41.443 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:41 smithi053 systemd[1]: Stopping Ceph prometheus.a for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:07:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:41 smithi003 ceph-mon[30803]: pgmap v47: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:41 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:07:41.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:41 smithi003 ceph-mon[41530]: pgmap v47: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:41.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:41 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:07:41.717 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:41 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[61486]: ts=2024-08-17T17:07:41.441Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-08-17T17:07:41.717 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:41 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[61486]: ts=2024-08-17T17:07:41.442Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-08-17T17:07:41.717 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:41 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[61486]: ts=2024-08-17T17:07:41.442Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-08-17T17:07:41.717 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:41 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[61486]: ts=2024-08-17T17:07:41.442Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-08-17T17:07:41.717 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:41 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[61486]: ts=2024-08-17T17:07:41.442Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-08-17T17:07:41.717 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:41 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[61486]: ts=2024-08-17T17:07:41.442Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-08-17T17:07:41.717 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:41 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[61486]: ts=2024-08-17T17:07:41.442Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-08-17T17:07:41.717 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:41 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[61486]: ts=2024-08-17T17:07:41.442Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-08-17T17:07:41.717 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:41 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[61486]: ts=2024-08-17T17:07:41.442Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-08-17T17:07:41.717 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:41 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[61486]: ts=2024-08-17T17:07:41.442Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-08-17T17:07:41.717 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:41 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[61486]: ts=2024-08-17T17:07:41.442Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-08-17T17:07:41.718 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:41 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[61486]: ts=2024-08-17T17:07:41.442Z caller=main.go:1147 level=info msg="See you next time!" 2024-08-17T17:07:41.718 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:41 smithi053 podman[64535]: 2024-08-17 17:07:41.452116517 +0000 UTC m=+0.131399967 container died 05c2c4c5ce7387f7cf3a4f94add9ce51da5fb0af7301fe5dddaba695c5768c64 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:07:41.987 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-08-17T17:07:41.987 INFO:tasks.cephadm:Setup complete, yielding 2024-08-17T17:07:41.987 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-08-17T17:07:41.997 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi003.front.sepia.ceph.com 2024-08-17T17:07:41.997 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- bash -c 'ceph fs volume create foo' 2024-08-17T17:07:42.122 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:41 smithi053 podman[64535]: 2024-08-17 17:07:41.753597551 +0000 UTC m=+0.432881005 container cleanup 05c2c4c5ce7387f7cf3a4f94add9ce51da5fb0af7301fe5dddaba695c5768c64 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:07:42.122 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:41 smithi053 bash[64535]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a 2024-08-17T17:07:42.268 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:07:42.272 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:42 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/1738040627' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-08-17T17:07:42.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:42 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/1738040627' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-08-17T17:07:42.430 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:42 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/1738040627' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-08-17T17:07:42.431 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 podman[64547]: 2024-08-17 17:07:42.130058041 +0000 UTC m=+0.679387837 container remove 05c2c4c5ce7387f7cf3a4f94add9ce51da5fb0af7301fe5dddaba695c5768c64 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:07:42.716 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-08-17T17:07:42.716 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 systemd[1]: Stopped Ceph prometheus.a for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:07:42.716 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@prometheus.a.service: Consumed 1.079s CPU time. 2024-08-17T17:07:42.717 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 systemd[1]: Starting Ceph prometheus.a for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:07:42.717 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 podman[64640]: 2024-08-17 17:07:42.56232589 +0000 UTC m=+0.019066439 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-08-17T17:07:42.717 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 podman[64640]: 2024-08-17 17:07:42.664499655 +0000 UTC m=+0.121240189 container create af8f221d5152dc7cb9656ddb848de6660580632eefe976f733106312230e0add (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:07:43.123 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 podman[64640]: 2024-08-17 17:07:42.858455568 +0000 UTC m=+0.315196116 container init af8f221d5152dc7cb9656ddb848de6660580632eefe976f733106312230e0add (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:07:43.124 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 podman[64640]: 2024-08-17 17:07:42.861345119 +0000 UTC m=+0.318085656 container start af8f221d5152dc7cb9656ddb848de6660580632eefe976f733106312230e0add (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:07:43.124 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:07:42.885Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-08-17T17:07:43.124 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:07:42.885Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-08-17T17:07:43.124 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:07:42.885Z caller=main.go:570 level=info host_details="(Linux 5.14.0-480.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024 x86_64 smithi053 (none))" 2024-08-17T17:07:43.124 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:07:42.885Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-08-17T17:07:43.124 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:07:42.885Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-08-17T17:07:43.124 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:07:42.886Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-08-17T17:07:43.124 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:07:42.886Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-08-17T17:07:43.125 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:07:42.888Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-08-17T17:07:43.125 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:07:42.888Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-08-17T17:07:43.125 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:07:42.890Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-08-17T17:07:43.125 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:07:42.890Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.75µs 2024-08-17T17:07:43.125 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:07:42.890Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-08-17T17:07:43.125 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:07:42.890Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-08-17T17:07:43.125 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:07:42.890Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-08-17T17:07:43.125 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:07:42.890Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=20.39µs wal_replay_duration=421.008µs wbl_replay_duration=129ns total_replay_duration=460.207µs 2024-08-17T17:07:43.125 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:07:42.891Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-08-17T17:07:43.125 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:07:42.891Z caller=main.go:1029 level=info msg="TSDB started" 2024-08-17T17:07:43.125 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:07:42.891Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-08-17T17:07:43.125 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:07:42.905Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=14.556678ms db_storage=733ns remote_storage=1.124µs web_handler=304ns query_engine=517ns scrape=407.201µs scrape_sd=25.231µs notify=17.912µs notify_sd=7.135µs rules=13.866249ms tracing=5.843µs 2024-08-17T17:07:43.125 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:07:42.905Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-08-17T17:07:43.126 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:07:42.905Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-08-17T17:07:43.126 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 bash[64640]: af8f221d5152dc7cb9656ddb848de6660580632eefe976f733106312230e0add 2024-08-17T17:07:43.126 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:07:42 smithi053 systemd[1]: Started Ceph prometheus.a for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:07:43.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-mon[41530]: pgmap v48: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:43.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:43.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:43.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-08-17T17:07:43.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi003.front.sepia.ceph.com:9093"}]: dispatch 2024-08-17T17:07:43.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:43.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:07:43.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi053.front.sepia.ceph.com:9095"}]: dispatch 2024-08-17T17:07:43.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:43.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-08-17T17:07:43.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi053.front.sepia.ceph.com:3000"}]: dispatch 2024-08-17T17:07:43.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:43.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:07:43.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-mon[30803]: pgmap v48: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:43.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:43.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:43.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-08-17T17:07:43.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi003.front.sepia.ceph.com:9093"}]: dispatch 2024-08-17T17:07:43.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:43.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:07:43.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi053.front.sepia.ceph.com:9095"}]: dispatch 2024-08-17T17:07:43.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:43.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-08-17T17:07:43.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi053.front.sepia.ceph.com:3000"}]: dispatch 2024-08-17T17:07:43.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:43.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:07:43.275 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:07:43] ENGINE Bus STOPPING 2024-08-17T17:07:43.275 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:07:43] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-08-17T17:07:43.276 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:07:43] ENGINE Bus STOPPED 2024-08-17T17:07:43.276 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:07:43] ENGINE Bus STARTING 2024-08-17T17:07:43.276 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:07:43] ENGINE Serving on http://:::9283 2024-08-17T17:07:43.276 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:07:43] ENGINE Bus STARTED 2024-08-17T17:07:43.276 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:07:43] ENGINE Bus STOPPING 2024-08-17T17:07:43.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:43 smithi053 ceph-mon[34699]: pgmap v48: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:43 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:43 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:43 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-08-17T17:07:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:43 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi003.front.sepia.ceph.com:9093"}]: dispatch 2024-08-17T17:07:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:43 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:43 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:07:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:43 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi053.front.sepia.ceph.com:9095"}]: dispatch 2024-08-17T17:07:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:43 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:43 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-08-17T17:07:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:43 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi053.front.sepia.ceph.com:3000"}]: dispatch 2024-08-17T17:07:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:43 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:43 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:07:43.739 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:07:43] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-08-17T17:07:43.740 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:07:43] ENGINE Bus STOPPED 2024-08-17T17:07:43.740 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:07:43] ENGINE Bus STARTING 2024-08-17T17:07:44.021 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:07:43] ENGINE Serving on http://:::9283 2024-08-17T17:07:44.021 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:07:43] ENGINE Bus STARTED 2024-08-17T17:07:44.021 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:07:43 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:07:43] ENGINE Bus STOPPING 2024-08-17T17:07:44.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:44 smithi003 ceph-mon[30803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-08-17T17:07:44.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:44 smithi003 ceph-mon[30803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi003.front.sepia.ceph.com:9093"}]: dispatch 2024-08-17T17:07:44.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:44 smithi003 ceph-mon[30803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:07:44.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:44 smithi003 ceph-mon[30803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi053.front.sepia.ceph.com:9095"}]: dispatch 2024-08-17T17:07:44.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:44 smithi003 ceph-mon[30803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-08-17T17:07:44.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:44 smithi003 ceph-mon[30803]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi053.front.sepia.ceph.com:3000"}]: dispatch 2024-08-17T17:07:44.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:44 smithi003 ceph-mon[41530]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-08-17T17:07:44.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:44 smithi003 ceph-mon[41530]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi003.front.sepia.ceph.com:9093"}]: dispatch 2024-08-17T17:07:44.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:44 smithi003 ceph-mon[41530]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:07:44.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:44 smithi003 ceph-mon[41530]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi053.front.sepia.ceph.com:9095"}]: dispatch 2024-08-17T17:07:44.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:44 smithi003 ceph-mon[41530]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-08-17T17:07:44.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:44 smithi003 ceph-mon[41530]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi053.front.sepia.ceph.com:3000"}]: dispatch 2024-08-17T17:07:44.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:44 smithi053 ceph-mon[34699]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-08-17T17:07:44.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:44 smithi053 ceph-mon[34699]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi003.front.sepia.ceph.com:9093"}]: dispatch 2024-08-17T17:07:44.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:44 smithi053 ceph-mon[34699]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:07:44.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:44 smithi053 ceph-mon[34699]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi053.front.sepia.ceph.com:9095"}]: dispatch 2024-08-17T17:07:44.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:44 smithi053 ceph-mon[34699]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-08-17T17:07:44.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:44 smithi053 ceph-mon[34699]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi053.front.sepia.ceph.com:3000"}]: dispatch 2024-08-17T17:07:44.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:07:44 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:07:44] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-08-17T17:07:44.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:07:44 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:07:44] ENGINE Bus STOPPED 2024-08-17T17:07:44.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:07:44 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:07:44] ENGINE Bus STARTING 2024-08-17T17:07:44.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:07:44 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:07:44] ENGINE Serving on http://:::9283 2024-08-17T17:07:44.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:07:44 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:07:44] ENGINE Bus STARTED 2024-08-17T17:07:45.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:45 smithi053 ceph-mon[34699]: pgmap v49: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:45.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:45 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-08-17T17:07:45.461 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:45 smithi003 ceph-mon[41530]: pgmap v49: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:45.461 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:45 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-08-17T17:07:45.461 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:45 smithi003 ceph-mon[30803]: pgmap v49: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:45.461 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:45 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-08-17T17:07:46.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:46 smithi053 ceph-mon[34699]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:07:46.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:46 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-08-17T17:07:46.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:46 smithi053 ceph-mon[34699]: osdmap e67: 8 total, 8 up, 8 in 2024-08-17T17:07:46.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:46 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-08-17T17:07:46.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:46 smithi003 ceph-mon[41530]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:07:46.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:46 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-08-17T17:07:46.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:46 smithi003 ceph-mon[41530]: osdmap e67: 8 total, 8 up, 8 in 2024-08-17T17:07:46.450 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:46 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-08-17T17:07:46.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:46 smithi003 ceph-mon[30803]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:07:46.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:46 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-08-17T17:07:46.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:46 smithi003 ceph-mon[30803]: osdmap e67: 8 total, 8 up, 8 in 2024-08-17T17:07:46.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:46 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-08-17T17:07:46.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:46 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-a[30799]: 2024-08-17T17:07:46.146+0000 7f540e444640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-17T17:07:47.130 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-08-17T17:07:47.374 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[30803]: pgmap v51: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:47.374 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[30803]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:07:47.374 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-08-17T17:07:47.374 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[30803]: osdmap e68: 8 total, 8 up, 8 in 2024-08-17T17:07:47.374 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-08-17T17:07:47.374 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[30803]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-17T17:07:47.374 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[30803]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-08-17T17:07:47.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[30803]: osdmap e69: 8 total, 8 up, 8 in 2024-08-17T17:07:47.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-08-17T17:07:47.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[30803]: fsmap foo:0 2024-08-17T17:07:47.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[30803]: Saving service mds.foo spec with placement count:2 2024-08-17T17:07:47.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:47.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:47.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:47.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:47.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:47.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:47.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:07:47.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:47.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi003.civgvf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-17T17:07:47.376 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi003.civgvf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-17T17:07:47.376 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:47.376 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[41530]: pgmap v51: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:47.376 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[41530]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:07:47.376 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-08-17T17:07:47.376 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[41530]: osdmap e68: 8 total, 8 up, 8 in 2024-08-17T17:07:47.376 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-08-17T17:07:47.376 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[41530]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-17T17:07:47.376 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[41530]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-08-17T17:07:47.376 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[41530]: osdmap e69: 8 total, 8 up, 8 in 2024-08-17T17:07:47.376 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-08-17T17:07:47.376 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[41530]: fsmap foo:0 2024-08-17T17:07:47.377 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[41530]: Saving service mds.foo spec with placement count:2 2024-08-17T17:07:47.377 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:47.377 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:47.377 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:47.377 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:47.377 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:47.377 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:47.377 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:07:47.377 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:47.377 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi003.civgvf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-17T17:07:47.377 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi003.civgvf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-17T17:07:47.377 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:47.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:47 smithi053 ceph-mon[34699]: pgmap v51: 41 pgs: 32 unknown, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:47.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:47 smithi053 ceph-mon[34699]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:07:47.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:47 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-08-17T17:07:47.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:47 smithi053 ceph-mon[34699]: osdmap e68: 8 total, 8 up, 8 in 2024-08-17T17:07:47.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:47 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-08-17T17:07:47.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:47 smithi053 ceph-mon[34699]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-08-17T17:07:47.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:47 smithi053 ceph-mon[34699]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-08-17T17:07:47.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:47 smithi053 ceph-mon[34699]: osdmap e69: 8 total, 8 up, 8 in 2024-08-17T17:07:47.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:47 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-08-17T17:07:47.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:47 smithi053 ceph-mon[34699]: fsmap foo:0 2024-08-17T17:07:47.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:47 smithi053 ceph-mon[34699]: Saving service mds.foo spec with placement count:2 2024-08-17T17:07:47.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:47 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:47.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:47 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:47.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:47 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:47.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:47 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:47.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:47 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:47.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:47 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:47.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:47 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:07:47.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:47 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:47.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:47 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi003.civgvf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-17T17:07:47.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:47 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi003.civgvf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-17T17:07:47.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:47 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:48.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:48 smithi003 ceph-mon[30803]: Deploying daemon mds.foo.smithi003.civgvf on smithi003 2024-08-17T17:07:48.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:48 smithi003 ceph-mon[30803]: osdmap e70: 8 total, 8 up, 8 in 2024-08-17T17:07:48.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:48 smithi003 ceph-mon[41530]: Deploying daemon mds.foo.smithi003.civgvf on smithi003 2024-08-17T17:07:48.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:48 smithi003 ceph-mon[41530]: osdmap e70: 8 total, 8 up, 8 in 2024-08-17T17:07:48.274 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:07:47 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a[74261]: ts=2024-08-17T17:07:47.992Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002297972s 2024-08-17T17:07:48.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:48 smithi053 ceph-mon[34699]: Deploying daemon mds.foo.smithi003.civgvf on smithi003 2024-08-17T17:07:48.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:48 smithi053 ceph-mon[34699]: osdmap e70: 8 total, 8 up, 8 in 2024-08-17T17:07:48.530 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:07:49.165 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:49 smithi003 ceph-mon[41530]: pgmap v55: 73 pgs: 25 creating+peering, 48 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:49.165 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:49 smithi003 ceph-mon[30803]: pgmap v55: 73 pgs: 25 creating+peering, 48 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:49.165 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:49 smithi003 ceph-mon[30803]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:07:49.165 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:49 smithi003 ceph-mon[30803]: osdmap e71: 8 total, 8 up, 8 in 2024-08-17T17:07:49.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:49 smithi053 ceph-mon[34699]: pgmap v55: 73 pgs: 25 creating+peering, 48 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:49.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:49 smithi053 ceph-mon[34699]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:07:49.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:49 smithi053 ceph-mon[34699]: osdmap e71: 8 total, 8 up, 8 in 2024-08-17T17:07:49.468 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:49 smithi003 ceph-mon[41530]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:07:49.468 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:49 smithi003 ceph-mon[41530]: osdmap e71: 8 total, 8 up, 8 in 2024-08-17T17:07:51.448 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:51 smithi003 ceph-mon[41530]: pgmap v57: 73 pgs: 25 creating+peering, 48 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:51.448 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:51 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:51.448 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:51 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:51.448 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:51 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:51.448 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:51 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi053.ougbtc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-17T17:07:51.448 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:51 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi053.ougbtc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-17T17:07:51.448 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:51 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:51.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:51 smithi003 ceph-mon[30803]: pgmap v57: 73 pgs: 25 creating+peering, 48 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:51.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:51 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:51.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:51 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:51.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:51 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:51.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:51 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi053.ougbtc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-17T17:07:51.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:51 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi053.ougbtc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-17T17:07:51.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:51 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:51.458 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:51 smithi053 ceph-mon[34699]: pgmap v57: 73 pgs: 25 creating+peering, 48 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:07:51.458 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:51 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:51.458 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:51 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:51.459 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:51 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:51.459 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:51 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi053.ougbtc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-08-17T17:07:51.459 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:51 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi053.ougbtc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-08-17T17:07:51.459 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:51 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:51.473 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-08-17T17:07:51.741 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:07:52.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:52 smithi053 ceph-mon[34699]: Deploying daemon mds.foo.smithi053.ougbtc on smithi053 2024-08-17T17:07:52.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:52 smithi053 ceph-mon[34699]: mds.? [v2:172.21.15.3:6834/2800254789,v1:172.21.15.3:6835/2800254789] up:boot 2024-08-17T17:07:52.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:52 smithi053 ceph-mon[34699]: daemon mds.foo.smithi003.civgvf assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-08-17T17:07:52.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:52 smithi053 ceph-mon[34699]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-08-17T17:07:52.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:52 smithi053 ceph-mon[34699]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-08-17T17:07:52.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:52 smithi053 ceph-mon[34699]: Cluster is now healthy 2024-08-17T17:07:52.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:52 smithi053 ceph-mon[34699]: fsmap foo:0 1 up:standby 2024-08-17T17:07:52.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:52 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi003.civgvf"}]: dispatch 2024-08-17T17:07:52.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:52 smithi053 ceph-mon[34699]: fsmap foo:1 {0=foo.smithi003.civgvf=up:creating} 2024-08-17T17:07:52.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:52 smithi053 ceph-mon[34699]: daemon mds.foo.smithi003.civgvf is now active in filesystem foo as rank 0 2024-08-17T17:07:52.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:52 smithi003 ceph-mon[41530]: Deploying daemon mds.foo.smithi053.ougbtc on smithi053 2024-08-17T17:07:52.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:52 smithi003 ceph-mon[41530]: mds.? [v2:172.21.15.3:6834/2800254789,v1:172.21.15.3:6835/2800254789] up:boot 2024-08-17T17:07:52.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:52 smithi003 ceph-mon[41530]: daemon mds.foo.smithi003.civgvf assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-08-17T17:07:52.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:52 smithi003 ceph-mon[41530]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-08-17T17:07:52.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:52 smithi003 ceph-mon[41530]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-08-17T17:07:52.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:52 smithi003 ceph-mon[41530]: Cluster is now healthy 2024-08-17T17:07:52.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:52 smithi003 ceph-mon[41530]: fsmap foo:0 1 up:standby 2024-08-17T17:07:52.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:52 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi003.civgvf"}]: dispatch 2024-08-17T17:07:52.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:52 smithi003 ceph-mon[41530]: fsmap foo:1 {0=foo.smithi003.civgvf=up:creating} 2024-08-17T17:07:52.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:52 smithi003 ceph-mon[41530]: daemon mds.foo.smithi003.civgvf is now active in filesystem foo as rank 0 2024-08-17T17:07:52.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:52 smithi003 ceph-mon[30803]: Deploying daemon mds.foo.smithi053.ougbtc on smithi053 2024-08-17T17:07:52.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:52 smithi003 ceph-mon[30803]: mds.? [v2:172.21.15.3:6834/2800254789,v1:172.21.15.3:6835/2800254789] up:boot 2024-08-17T17:07:52.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:52 smithi003 ceph-mon[30803]: daemon mds.foo.smithi003.civgvf assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-08-17T17:07:52.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:52 smithi003 ceph-mon[30803]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-08-17T17:07:52.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:52 smithi003 ceph-mon[30803]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-08-17T17:07:52.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:52 smithi003 ceph-mon[30803]: Cluster is now healthy 2024-08-17T17:07:52.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:52 smithi003 ceph-mon[30803]: fsmap foo:0 1 up:standby 2024-08-17T17:07:52.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:52 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi003.civgvf"}]: dispatch 2024-08-17T17:07:52.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:52 smithi003 ceph-mon[30803]: fsmap foo:1 {0=foo.smithi003.civgvf=up:creating} 2024-08-17T17:07:52.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:52 smithi003 ceph-mon[30803]: daemon mds.foo.smithi003.civgvf is now active in filesystem foo as rank 0 2024-08-17T17:07:53.156 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:07:53 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:07:53] "GET /metrics HTTP/1.1" 200 203813 "" "Prometheus/2.43.0" 2024-08-17T17:07:53.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:53 smithi053 ceph-mon[34699]: pgmap v58: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.3 KiB/s wr, 7 op/s 2024-08-17T17:07:53.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:53 smithi053 ceph-mon[34699]: mds.? [v2:172.21.15.3:6834/2800254789,v1:172.21.15.3:6835/2800254789] up:active 2024-08-17T17:07:53.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:53 smithi053 ceph-mon[34699]: fsmap foo:1 {0=foo.smithi003.civgvf=up:active} 2024-08-17T17:07:53.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:53 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:53.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:53 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:53.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:53 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:53.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:53 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:53.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:53 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:53.476 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:53 smithi003 ceph-mon[41530]: pgmap v58: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.3 KiB/s wr, 7 op/s 2024-08-17T17:07:53.476 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:53 smithi003 ceph-mon[41530]: mds.? [v2:172.21.15.3:6834/2800254789,v1:172.21.15.3:6835/2800254789] up:active 2024-08-17T17:07:53.476 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:53 smithi003 ceph-mon[41530]: fsmap foo:1 {0=foo.smithi003.civgvf=up:active} 2024-08-17T17:07:53.476 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:53 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:53.476 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:53 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:53.476 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:53 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:53.476 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:53 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:53.476 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:53 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:53.476 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:53 smithi003 ceph-mon[30803]: pgmap v58: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.3 KiB/s wr, 7 op/s 2024-08-17T17:07:53.477 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:53 smithi003 ceph-mon[30803]: mds.? [v2:172.21.15.3:6834/2800254789,v1:172.21.15.3:6835/2800254789] up:active 2024-08-17T17:07:53.477 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:53 smithi003 ceph-mon[30803]: fsmap foo:1 {0=foo.smithi003.civgvf=up:active} 2024-08-17T17:07:53.477 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:53 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:53.477 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:53 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:53.477 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:53 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:53.477 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:53 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:53.477 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:53 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:54.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:54 smithi053 ceph-mon[34699]: mds.? [v2:172.21.15.53:6832/264432371,v1:172.21.15.53:6833/264432371] up:boot 2024-08-17T17:07:54.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:54 smithi053 ceph-mon[34699]: fsmap foo:1 {0=foo.smithi003.civgvf=up:active} 1 up:standby 2024-08-17T17:07:54.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:54 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi053.ougbtc"}]: dispatch 2024-08-17T17:07:54.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:54 smithi053 ceph-mon[34699]: fsmap foo:1 {0=foo.smithi003.civgvf=up:active} 1 up:standby 2024-08-17T17:07:54.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:54 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:07:54.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:54 smithi003 ceph-mon[41530]: mds.? [v2:172.21.15.53:6832/264432371,v1:172.21.15.53:6833/264432371] up:boot 2024-08-17T17:07:54.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:54 smithi003 ceph-mon[41530]: fsmap foo:1 {0=foo.smithi003.civgvf=up:active} 1 up:standby 2024-08-17T17:07:54.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:54 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi053.ougbtc"}]: dispatch 2024-08-17T17:07:54.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:54 smithi003 ceph-mon[41530]: fsmap foo:1 {0=foo.smithi003.civgvf=up:active} 1 up:standby 2024-08-17T17:07:54.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:54 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:07:54.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:54 smithi003 ceph-mon[30803]: mds.? [v2:172.21.15.53:6832/264432371,v1:172.21.15.53:6833/264432371] up:boot 2024-08-17T17:07:54.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:54 smithi003 ceph-mon[30803]: fsmap foo:1 {0=foo.smithi003.civgvf=up:active} 1 up:standby 2024-08-17T17:07:54.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:54 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi053.ougbtc"}]: dispatch 2024-08-17T17:07:54.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:54 smithi003 ceph-mon[30803]: fsmap foo:1 {0=foo.smithi003.civgvf=up:active} 1 up:standby 2024-08-17T17:07:54.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:54 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:07:55.046 INFO:teuthology.run_tasks:Running task ceph.healthy... 2024-08-17T17:07:55.055 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-08-17T17:07:55.055 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-08-17T17:07:55.055 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph mgr dump --format=json 2024-08-17T17:07:55.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:55 smithi003 ceph-mon[41530]: pgmap v59: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s wr, 5 op/s 2024-08-17T17:07:55.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:55 smithi003 ceph-mon[30803]: pgmap v59: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s wr, 5 op/s 2024-08-17T17:07:55.359 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:07:55.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:55 smithi053 ceph-mon[34699]: pgmap v59: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s wr, 5 op/s 2024-08-17T17:07:56.311 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:56 smithi053 ceph-mon[34699]: mds.? [v2:172.21.15.3:6834/2800254789,v1:172.21.15.3:6835/2800254789] up:active 2024-08-17T17:07:56.312 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:56 smithi053 ceph-mon[34699]: fsmap foo:1 {0=foo.smithi003.civgvf=up:active} 1 up:standby 2024-08-17T17:07:56.312 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:56 smithi053 ceph-mon[34699]: pgmap v60: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s wr, 5 op/s 2024-08-17T17:07:56.312 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:56 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:07:56.312 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:56 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:56.466 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:56 smithi003 ceph-mon[41530]: mds.? [v2:172.21.15.3:6834/2800254789,v1:172.21.15.3:6835/2800254789] up:active 2024-08-17T17:07:56.466 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:56 smithi003 ceph-mon[41530]: fsmap foo:1 {0=foo.smithi003.civgvf=up:active} 1 up:standby 2024-08-17T17:07:56.467 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:56 smithi003 ceph-mon[41530]: pgmap v60: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s wr, 5 op/s 2024-08-17T17:07:56.467 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:56 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:07:56.467 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:56 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:56.467 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:56 smithi003 ceph-mon[30803]: mds.? [v2:172.21.15.3:6834/2800254789,v1:172.21.15.3:6835/2800254789] up:active 2024-08-17T17:07:56.467 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:56 smithi003 ceph-mon[30803]: fsmap foo:1 {0=foo.smithi003.civgvf=up:active} 1 up:standby 2024-08-17T17:07:56.467 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:56 smithi003 ceph-mon[30803]: pgmap v60: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s wr, 5 op/s 2024-08-17T17:07:56.467 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:56 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:07:56.467 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:56 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:56.716 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:07:56 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:07:56] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-08-17T17:07:57.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:57 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:57.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:57 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:57.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:57 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:57.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:57 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:57.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:57 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:57.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:57 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:58.510 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:07:58.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:58 smithi053 ceph-mon[34699]: mds.? [v2:172.21.15.53:6832/264432371,v1:172.21.15.53:6833/264432371] up:standby 2024-08-17T17:07:58.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:58 smithi053 ceph-mon[34699]: fsmap foo:1 {0=foo.smithi003.civgvf=up:active} 1 up:standby 2024-08-17T17:07:58.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:58 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:58.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:58 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:58.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:58 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:58.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:58 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:07:58.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:58 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:58.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:58 smithi053 ceph-mon[34699]: pgmap v61: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-08-17T17:07:58.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:58 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:07:58.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:58 smithi003 ceph-mon[41530]: mds.? [v2:172.21.15.53:6832/264432371,v1:172.21.15.53:6833/264432371] up:standby 2024-08-17T17:07:58.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:58 smithi003 ceph-mon[41530]: fsmap foo:1 {0=foo.smithi003.civgvf=up:active} 1 up:standby 2024-08-17T17:07:58.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:58 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:58.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:58 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:58.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:58 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:58.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:58 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:07:58.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:58 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:58.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:58 smithi003 ceph-mon[41530]: pgmap v61: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-08-17T17:07:58.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:58 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:07:58.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:58 smithi003 ceph-mon[30803]: mds.? [v2:172.21.15.53:6832/264432371,v1:172.21.15.53:6833/264432371] up:standby 2024-08-17T17:07:58.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:58 smithi003 ceph-mon[30803]: fsmap foo:1 {0=foo.smithi003.civgvf=up:active} 1 up:standby 2024-08-17T17:07:58.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:58 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:58.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:58 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:58.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:58 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:07:58.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:58 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:07:58.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:58 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:07:58.775 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:58 smithi003 ceph-mon[30803]: pgmap v61: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-08-17T17:07:58.775 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:58 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:07:59.600 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":20,"flags":0,"active_gid":14457,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":2670911286},{"type":"v1","addr":"172.21.15.3:6801","nonce":2670911286}]},"active_addr":"172.21.15.3:6801/2670911286","active_change":"2024-08-17T17:06:25.750739+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":24346,"name":"x","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.3:8443/","prometheus":"http://172.21.15.3:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":59,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":2029382827}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":601716019}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":3717779298}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":880444026}]}]} 2024-08-17T17:07:59.604 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-08-17T17:07:59.604 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-08-17T17:07:59.605 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-17T17:07:59.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:07:59 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/2904555061' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-08-17T17:07:59.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:07:59 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/2904555061' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-08-17T17:07:59.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:07:59 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/2904555061' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-08-17T17:08:00.057 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:08:00.756 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:00 smithi053 ceph-mon[34699]: pgmap v62: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-08-17T17:08:00.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:00 smithi003 ceph-mon[30803]: pgmap v62: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-08-17T17:08:00.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:00 smithi003 ceph-mon[41530]: pgmap v62: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-08-17T17:08:02.015 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:08:02.015 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":71,"fsid":"2b5b662e-5cba-11ef-bccf-c7b262605968","created":"2024-08-17T17:00:56.372943+0000","modified":"2024-08-17T17:07:48.153878+0000","last_up_change":"2024-08-17T17:05:54.031028+0000","last_in_change":"2024-08-17T17:05:37.289378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-17T17:04:01.140951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-17T17:06:28.452624+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-08-17T17:07:44.635021+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-08-17T17:07:45.576097+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}}}],"osds":[{"osd":0,"uuid":"7f9a7d76-f0ec-49fc-9540-2606e94758ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":288807015},{"type":"v1","addr":"172.21.15.3:6803","nonce":288807015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":288807015},{"type":"v1","addr":"172.21.15.3:6805","nonce":288807015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":288807015},{"type":"v1","addr":"172.21.15.3:6809","nonce":288807015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":288807015},{"type":"v1","addr":"172.21.15.3:6807","nonce":288807015}]},"public_addr":"172.21.15.3:6803/288807015","cluster_addr":"172.21.15.3:6805/288807015","heartbeat_back_addr":"172.21.15.3:6809/288807015","heartbeat_front_addr":"172.21.15.3:6807/288807015","state":["exists","up"]},{"osd":1,"uuid":"316b5c4e-3417-4b93-bbbb-c6ed6dd21b0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2183964597},{"type":"v1","addr":"172.21.15.3:6811","nonce":2183964597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2183964597},{"type":"v1","addr":"172.21.15.3:6813","nonce":2183964597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":2183964597},{"type":"v1","addr":"172.21.15.3:6817","nonce":2183964597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2183964597},{"type":"v1","addr":"172.21.15.3:6815","nonce":2183964597}]},"public_addr":"172.21.15.3:6811/2183964597","cluster_addr":"172.21.15.3:6813/2183964597","heartbeat_back_addr":"172.21.15.3:6817/2183964597","heartbeat_front_addr":"172.21.15.3:6815/2183964597","state":["exists","up"]},{"osd":2,"uuid":"88af697d-38db-4c51-bfff-d0a776315596","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1747862993},{"type":"v1","addr":"172.21.15.3:6819","nonce":1747862993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1747862993},{"type":"v1","addr":"172.21.15.3:6821","nonce":1747862993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1747862993},{"type":"v1","addr":"172.21.15.3:6825","nonce":1747862993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1747862993},{"type":"v1","addr":"172.21.15.3:6823","nonce":1747862993}]},"public_addr":"172.21.15.3:6819/1747862993","cluster_addr":"172.21.15.3:6821/1747862993","heartbeat_back_addr":"172.21.15.3:6825/1747862993","heartbeat_front_addr":"172.21.15.3:6823/1747862993","state":["exists","up"]},{"osd":3,"uuid":"8e20cf42-6a13-4ad5-9f01-203a30c692d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2572629546},{"type":"v1","addr":"172.21.15.3:6827","nonce":2572629546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2572629546},{"type":"v1","addr":"172.21.15.3:6829","nonce":2572629546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2572629546},{"type":"v1","addr":"172.21.15.3:6833","nonce":2572629546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2572629546},{"type":"v1","addr":"172.21.15.3:6831","nonce":2572629546}]},"public_addr":"172.21.15.3:6827/2572629546","cluster_addr":"172.21.15.3:6829/2572629546","heartbeat_back_addr":"172.21.15.3:6833/2572629546","heartbeat_front_addr":"172.21.15.3:6831/2572629546","state":["exists","up"]},{"osd":4,"uuid":"2e8eabd7-8258-4b78-bb21-1a7291e0f1d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6801","nonce":1387588062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6803","nonce":1387588062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6807","nonce":1387588062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6805","nonce":1387588062}]},"public_addr":"172.21.15.53:6801/1387588062","cluster_addr":"172.21.15.53:6803/1387588062","heartbeat_back_addr":"172.21.15.53:6807/1387588062","heartbeat_front_addr":"172.21.15.53:6805/1387588062","state":["exists","up"]},{"osd":5,"uuid":"738f10f7-ff0d-4392-a818-eb23c9f65a63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6809","nonce":1263400356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6811","nonce":1263400356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6815","nonce":1263400356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6813","nonce":1263400356}]},"public_addr":"172.21.15.53:6809/1263400356","cluster_addr":"172.21.15.53:6811/1263400356","heartbeat_back_addr":"172.21.15.53:6815/1263400356","heartbeat_front_addr":"172.21.15.53:6813/1263400356","state":["exists","up"]},{"osd":6,"uuid":"62903bee-59eb-4eed-b64d-9c884de87409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6817","nonce":3729099668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6819","nonce":3729099668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6823","nonce":3729099668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6821","nonce":3729099668}]},"public_addr":"172.21.15.53:6817/3729099668","cluster_addr":"172.21.15.53:6819/3729099668","heartbeat_back_addr":"172.21.15.53:6823/3729099668","heartbeat_front_addr":"172.21.15.53:6821/3729099668","state":["exists","up"]},{"osd":7,"uuid":"4794ff08-e5d9-49c9-bea5-e158681b4004","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6825","nonce":832291503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6827","nonce":832291503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6831","nonce":832291503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6829","nonce":832291503}]},"public_addr":"172.21.15.53:6825/832291503","cluster_addr":"172.21.15.53:6827/832291503","heartbeat_back_addr":"172.21.15.53:6831/832291503","heartbeat_front_addr":"172.21.15.53:6829/832291503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:03:10.759301+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:03:33.981212+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:03:58.516966+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:04:22.965963+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:04:43.836760+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:06.691915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:29.472293+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:52.166848+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:0/1583211207":"2024-08-18T17:06:25.750593+0000","172.21.15.3:6801/1736600360":"2024-08-18T17:06:25.750593+0000","172.21.15.3:0/1668133120":"2024-08-18T17:01:27.442443+0000","172.21.15.3:0/1476294447":"2024-08-18T17:01:27.442443+0000","172.21.15.3:6801/160535497":"2024-08-18T17:01:27.442443+0000","172.21.15.3:0/1999667208":"2024-08-18T17:01:50.631242+0000","172.21.15.3:6800/160535497":"2024-08-18T17:01:27.442443+0000","172.21.15.3:0/3946755108":"2024-08-18T17:06:25.750593+0000","172.21.15.3:0/3817753887":"2024-08-18T17:01:50.631242+0000","172.21.15.3:0/3953105554":"2024-08-18T17:06:25.750593+0000","172.21.15.3:6800/1736600360":"2024-08-18T17:06:25.750593+0000","172.21.15.3:0/3985128953":"2024-08-18T17:01:50.631242+0000","172.21.15.3:0/4194950757":"2024-08-18T17:01:27.442443+0000","172.21.15.3:6800/2187177466":"2024-08-18T17:01:50.631242+0000","172.21.15.3:6801/2187177466":"2024-08-18T17:01:50.631242+0000","172.21.15.3:0/3116588554":"2024-08-18T17:06:25.750593+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-17T17:08:02.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:01 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:08:02.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:01 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:08:02.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:01 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:08:02.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:01 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:08:02.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:01 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:08:02.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:01 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:08:02.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:01 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:08:02.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:01 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:08:02.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:01 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:08:02.707 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-08-17T17:08:02.707 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-17T17:08:02.963 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:02 smithi003 ceph-mon[41530]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-08-17T17:08:02.963 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:02 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/3166063300' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-17T17:08:02.964 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:02 smithi003 ceph-mon[30803]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-08-17T17:08:02.964 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:02 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/3166063300' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-17T17:08:03.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:02 smithi053 ceph-mon[34699]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-08-17T17:08:03.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:02 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/3166063300' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-17T17:08:03.273 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:08:03 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:08:03] "GET /metrics HTTP/1.1" 200 239637 "" "Prometheus/2.43.0" 2024-08-17T17:08:04.127 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:08:05.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:05 smithi053 ceph-mon[34699]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-08-17T17:08:05.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:05 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:08:05.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:05 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:08:05.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:05 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:08:05.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:05 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:08:05.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:05 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:08:05.515 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:05 smithi003 ceph-mon[41530]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-08-17T17:08:05.515 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:05 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:08:05.515 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:05 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:08:05.515 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:05 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:08:05.515 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:05 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:08:05.515 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:05 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:08:05.515 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:05 smithi003 ceph-mon[30803]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-08-17T17:08:05.516 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:05 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:08:05.516 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:05 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:08:05.516 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:05 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:08:05.516 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:05 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:08:05.516 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:05 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:08:06.086 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:08:06.086 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":71,"fsid":"2b5b662e-5cba-11ef-bccf-c7b262605968","created":"2024-08-17T17:00:56.372943+0000","modified":"2024-08-17T17:07:48.153878+0000","last_up_change":"2024-08-17T17:05:54.031028+0000","last_in_change":"2024-08-17T17:05:37.289378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-17T17:04:01.140951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-17T17:06:28.452624+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":65,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-08-17T17:07:44.635021+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-08-17T17:07:45.576097+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}}}],"osds":[{"osd":0,"uuid":"7f9a7d76-f0ec-49fc-9540-2606e94758ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":288807015},{"type":"v1","addr":"172.21.15.3:6803","nonce":288807015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":288807015},{"type":"v1","addr":"172.21.15.3:6805","nonce":288807015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":288807015},{"type":"v1","addr":"172.21.15.3:6809","nonce":288807015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":288807015},{"type":"v1","addr":"172.21.15.3:6807","nonce":288807015}]},"public_addr":"172.21.15.3:6803/288807015","cluster_addr":"172.21.15.3:6805/288807015","heartbeat_back_addr":"172.21.15.3:6809/288807015","heartbeat_front_addr":"172.21.15.3:6807/288807015","state":["exists","up"]},{"osd":1,"uuid":"316b5c4e-3417-4b93-bbbb-c6ed6dd21b0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":2183964597},{"type":"v1","addr":"172.21.15.3:6811","nonce":2183964597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":2183964597},{"type":"v1","addr":"172.21.15.3:6813","nonce":2183964597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":2183964597},{"type":"v1","addr":"172.21.15.3:6817","nonce":2183964597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":2183964597},{"type":"v1","addr":"172.21.15.3:6815","nonce":2183964597}]},"public_addr":"172.21.15.3:6811/2183964597","cluster_addr":"172.21.15.3:6813/2183964597","heartbeat_back_addr":"172.21.15.3:6817/2183964597","heartbeat_front_addr":"172.21.15.3:6815/2183964597","state":["exists","up"]},{"osd":2,"uuid":"88af697d-38db-4c51-bfff-d0a776315596","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1747862993},{"type":"v1","addr":"172.21.15.3:6819","nonce":1747862993}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1747862993},{"type":"v1","addr":"172.21.15.3:6821","nonce":1747862993}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1747862993},{"type":"v1","addr":"172.21.15.3:6825","nonce":1747862993}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1747862993},{"type":"v1","addr":"172.21.15.3:6823","nonce":1747862993}]},"public_addr":"172.21.15.3:6819/1747862993","cluster_addr":"172.21.15.3:6821/1747862993","heartbeat_back_addr":"172.21.15.3:6825/1747862993","heartbeat_front_addr":"172.21.15.3:6823/1747862993","state":["exists","up"]},{"osd":3,"uuid":"8e20cf42-6a13-4ad5-9f01-203a30c692d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2572629546},{"type":"v1","addr":"172.21.15.3:6827","nonce":2572629546}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2572629546},{"type":"v1","addr":"172.21.15.3:6829","nonce":2572629546}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2572629546},{"type":"v1","addr":"172.21.15.3:6833","nonce":2572629546}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2572629546},{"type":"v1","addr":"172.21.15.3:6831","nonce":2572629546}]},"public_addr":"172.21.15.3:6827/2572629546","cluster_addr":"172.21.15.3:6829/2572629546","heartbeat_back_addr":"172.21.15.3:6833/2572629546","heartbeat_front_addr":"172.21.15.3:6831/2572629546","state":["exists","up"]},{"osd":4,"uuid":"2e8eabd7-8258-4b78-bb21-1a7291e0f1d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6801","nonce":1387588062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6803","nonce":1387588062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6807","nonce":1387588062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6805","nonce":1387588062}]},"public_addr":"172.21.15.53:6801/1387588062","cluster_addr":"172.21.15.53:6803/1387588062","heartbeat_back_addr":"172.21.15.53:6807/1387588062","heartbeat_front_addr":"172.21.15.53:6805/1387588062","state":["exists","up"]},{"osd":5,"uuid":"738f10f7-ff0d-4392-a818-eb23c9f65a63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6809","nonce":1263400356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6811","nonce":1263400356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6815","nonce":1263400356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6813","nonce":1263400356}]},"public_addr":"172.21.15.53:6809/1263400356","cluster_addr":"172.21.15.53:6811/1263400356","heartbeat_back_addr":"172.21.15.53:6815/1263400356","heartbeat_front_addr":"172.21.15.53:6813/1263400356","state":["exists","up"]},{"osd":6,"uuid":"62903bee-59eb-4eed-b64d-9c884de87409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6817","nonce":3729099668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6819","nonce":3729099668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6823","nonce":3729099668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6821","nonce":3729099668}]},"public_addr":"172.21.15.53:6817/3729099668","cluster_addr":"172.21.15.53:6819/3729099668","heartbeat_back_addr":"172.21.15.53:6823/3729099668","heartbeat_front_addr":"172.21.15.53:6821/3729099668","state":["exists","up"]},{"osd":7,"uuid":"4794ff08-e5d9-49c9-bea5-e158681b4004","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6825","nonce":832291503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6827","nonce":832291503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6831","nonce":832291503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6829","nonce":832291503}]},"public_addr":"172.21.15.53:6825/832291503","cluster_addr":"172.21.15.53:6827/832291503","heartbeat_back_addr":"172.21.15.53:6831/832291503","heartbeat_front_addr":"172.21.15.53:6829/832291503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:03:10.759301+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:03:33.981212+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:03:58.516966+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:04:22.965963+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:04:43.836760+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:06.691915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:29.472293+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:52.166848+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:0/1583211207":"2024-08-18T17:06:25.750593+0000","172.21.15.3:6801/1736600360":"2024-08-18T17:06:25.750593+0000","172.21.15.3:0/1668133120":"2024-08-18T17:01:27.442443+0000","172.21.15.3:0/1476294447":"2024-08-18T17:01:27.442443+0000","172.21.15.3:6801/160535497":"2024-08-18T17:01:27.442443+0000","172.21.15.3:0/1999667208":"2024-08-18T17:01:50.631242+0000","172.21.15.3:6800/160535497":"2024-08-18T17:01:27.442443+0000","172.21.15.3:0/3946755108":"2024-08-18T17:06:25.750593+0000","172.21.15.3:0/3817753887":"2024-08-18T17:01:50.631242+0000","172.21.15.3:0/3953105554":"2024-08-18T17:06:25.750593+0000","172.21.15.3:6800/1736600360":"2024-08-18T17:06:25.750593+0000","172.21.15.3:0/3985128953":"2024-08-18T17:01:50.631242+0000","172.21.15.3:0/4194950757":"2024-08-18T17:01:27.442443+0000","172.21.15.3:6800/2187177466":"2024-08-18T17:01:50.631242+0000","172.21.15.3:6801/2187177466":"2024-08-18T17:01:50.631242+0000","172.21.15.3:0/3116588554":"2024-08-18T17:06:25.750593+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-17T17:08:06.390 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:06 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/2108299791' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-17T17:08:06.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:06 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/2108299791' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-17T17:08:06.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:06 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/2108299791' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-08-17T17:08:06.467 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:08:06 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:08:06] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-08-17T17:08:06.686 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-08-17T17:08:06.687 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-08-17T17:08:06.687 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-08-17T17:08:06.688 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-08-17T17:08:06.688 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-08-17T17:08:06.688 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-08-17T17:08:06.689 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-08-17T17:08:06.689 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-08-17T17:08:07.256 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:08:07.264 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:08:07.264 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:08:07.265 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:08:07.265 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:08:07.298 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:08:07.307 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:08:07.308 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:08:07.398 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:07 smithi003 ceph-mon[30803]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-08-17T17:08:07.398 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:07 smithi003 ceph-mon[41530]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-08-17T17:08:07.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:07 smithi053 ceph-mon[34699]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-08-17T17:08:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:09 smithi053 ceph-mon[34699]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-08-17T17:08:09.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:09 smithi003 ceph-mon[30803]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-08-17T17:08:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:09 smithi003 ceph-mon[41530]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-08-17T17:08:11.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:11 smithi003 ceph-mon[41530]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:11.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:11 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:08:11.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:11 smithi003 ceph-mon[30803]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:11.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:11 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:08:11.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:11 smithi053 ceph-mon[34699]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:11.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:11 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:08:13.329 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:13 smithi003 ceph-mon[41530]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:13.329 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:13 smithi003 ceph-mon[30803]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:13.330 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:08:13 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:08:13] "GET /metrics HTTP/1.1" 200 239629 "" "Prometheus/2.43.0" 2024-08-17T17:08:13.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:13 smithi053 ceph-mon[34699]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:15.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:15 smithi053 ceph-mon[34699]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:15.479 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:15 smithi003 ceph-mon[41530]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:15.479 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:15 smithi003 ceph-mon[30803]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:16.716 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:08:16 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:08:16] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-08-17T17:08:17.421 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:17 smithi003 ceph-mon[30803]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:17.422 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:17 smithi003 ceph-mon[41530]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:17.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:17 smithi053 ceph-mon[34699]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:18.747 INFO:teuthology.orchestra.run.smithi003.stdout:231928234015 2024-08-17T17:08:18.748 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-08-17T17:08:19.340 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:19 smithi003 ceph-mon[41530]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:19.340 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:19 smithi003 ceph-mon[30803]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:19.365 INFO:teuthology.orchestra.run.smithi003.stdout:197568495651 2024-08-17T17:08:19.365 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-08-17T17:08:19.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:19 smithi053 ceph-mon[34699]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:21.322 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:21 smithi003 ceph-mon[41530]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:21.322 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:21 smithi003 ceph-mon[30803]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:21.325 INFO:teuthology.orchestra.run.smithi003.stdout:77309411384 2024-08-17T17:08:21.326 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-08-17T17:08:21.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:21 smithi053 ceph-mon[34699]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:23.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:23 smithi053 ceph-mon[34699]: pgmap v73: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:23.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:23 smithi003 ceph-mon[30803]: pgmap v73: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:23.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:23 smithi003 ceph-mon[41530]: pgmap v73: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:23.523 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:08:23 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:08:23] "GET /metrics HTTP/1.1" 200 239629 "" "Prometheus/2.43.0" 2024-08-17T17:08:23.670 INFO:teuthology.orchestra.run.smithi003.stdout:34359738433 2024-08-17T17:08:23.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-08-17T17:08:24.089 INFO:teuthology.orchestra.run.smithi003.stdout:128849018926 2024-08-17T17:08:24.090 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-08-17T17:08:24.608 INFO:teuthology.orchestra.run.smithi003.stdout:55834574908 2024-08-17T17:08:24.608 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-08-17T17:08:24.988 INFO:teuthology.orchestra.run.smithi003.stdout:163208757289 2024-08-17T17:08:24.988 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-08-17T17:08:25.047 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:08:25.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:25 smithi003 ceph-mon[30803]: pgmap v74: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:25.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:25 smithi003 ceph-mon[41530]: pgmap v74: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:25.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:25 smithi053 ceph-mon[34699]: pgmap v74: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:25.520 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:08:25.548 INFO:teuthology.orchestra.run.smithi003.stdout:107374182451 2024-08-17T17:08:25.548 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-08-17T17:08:26.192 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:08:26.466 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:08:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:08:26] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-08-17T17:08:26.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:26 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-08-17T17:08:26.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:26 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-08-17T17:08:26.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:26 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-08-17T17:08:26.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:26 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:08:26.513 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:26 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-08-17T17:08:26.513 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:26 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-08-17T17:08:26.513 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:26 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-08-17T17:08:26.513 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:26 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:08:26.513 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:26 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-08-17T17:08:26.513 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:26 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-08-17T17:08:26.513 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:26 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-08-17T17:08:26.513 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:26 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:08:27.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:27 smithi053 ceph-mon[34699]: pgmap v75: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:27.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:27 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-08-17T17:08:27.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:27 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-08-17T17:08:27.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:27 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-08-17T17:08:27.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:27 smithi053 ceph-mon[34699]: osdmap e72: 8 total, 8 up, 8 in 2024-08-17T17:08:27.491 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:27 smithi003 ceph-mon[41530]: pgmap v75: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:27.491 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:27 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-08-17T17:08:27.491 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:27 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-08-17T17:08:27.491 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:27 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-08-17T17:08:27.491 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:27 smithi003 ceph-mon[41530]: osdmap e72: 8 total, 8 up, 8 in 2024-08-17T17:08:27.491 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:27 smithi003 ceph-mon[30803]: pgmap v75: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:27.491 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:27 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-08-17T17:08:27.492 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:27 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-08-17T17:08:27.492 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:27 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-08-17T17:08:27.492 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:27 smithi003 ceph-mon[30803]: osdmap e72: 8 total, 8 up, 8 in 2024-08-17T17:08:28.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:28 smithi053 ceph-mon[34699]: osdmap e73: 8 total, 8 up, 8 in 2024-08-17T17:08:28.507 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:28 smithi003 ceph-mon[41530]: osdmap e73: 8 total, 8 up, 8 in 2024-08-17T17:08:28.508 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:28 smithi003 ceph-mon[30803]: osdmap e73: 8 total, 8 up, 8 in 2024-08-17T17:08:29.116 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:08:29.117 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:08:29.411 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:29 smithi003 ceph-mon[41530]: pgmap v78: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:29.411 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:29 smithi003 ceph-mon[30803]: pgmap v78: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:29.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:29 smithi053 ceph-mon[34699]: pgmap v78: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:29.473 INFO:teuthology.orchestra.run.smithi003.stdout:128849018928 2024-08-17T17:08:29.682 INFO:teuthology.orchestra.run.smithi003.stdout:231928234018 2024-08-17T17:08:30.079 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018926 got 128849018928 for osd.4 2024-08-17T17:08:30.079 DEBUG:teuthology.parallel:result is None 2024-08-17T17:08:30.303 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:30 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/1429053696' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-08-17T17:08:30.303 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:30 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/3210023120' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-08-17T17:08:30.303 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:30 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/1429053696' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-08-17T17:08:30.304 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:30 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/3210023120' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-08-17T17:08:30.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:30 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/1429053696' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-08-17T17:08:30.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:30 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/3210023120' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-08-17T17:08:31.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:31 smithi053 ceph-mon[34699]: pgmap v79: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:31.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:31 smithi003 ceph-mon[41530]: pgmap v79: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:31.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:31 smithi003 ceph-mon[30803]: pgmap v79: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:31.603 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234015 got 231928234018 for osd.7 2024-08-17T17:08:31.603 DEBUG:teuthology.parallel:result is None 2024-08-17T17:08:33.253 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:33 smithi003 ceph-mon[41530]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:33.253 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:08:33 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:08:33] "GET /metrics HTTP/1.1" 200 239599 "" "Prometheus/2.43.0" 2024-08-17T17:08:33.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:33 smithi003 ceph-mon[30803]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:33.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:33 smithi053 ceph-mon[34699]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:33.821 INFO:teuthology.orchestra.run.smithi003.stdout:34359738436 2024-08-17T17:08:34.456 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:34 smithi003 ceph-mon[30803]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:34.456 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:34 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/3102356810' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-08-17T17:08:34.456 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:34 smithi003 ceph-mon[41530]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:34.456 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:34 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/3102356810' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-08-17T17:08:34.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:34 smithi053 ceph-mon[34699]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:34.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:34 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/3102356810' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-08-17T17:08:34.934 INFO:teuthology.orchestra.run.smithi003.stdout:197568495654 2024-08-17T17:08:35.168 INFO:teuthology.orchestra.run.smithi003.stdout:77309411387 2024-08-17T17:08:35.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:35 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/2808944601' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-08-17T17:08:35.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:35 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/2125061852' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-08-17T17:08:35.493 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:35 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/2808944601' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-08-17T17:08:35.494 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:35 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/2125061852' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-08-17T17:08:35.494 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:35 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/2808944601' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-08-17T17:08:35.494 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:35 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/2125061852' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-08-17T17:08:35.892 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495651 got 197568495654 for osd.6 2024-08-17T17:08:35.892 DEBUG:teuthology.parallel:result is None 2024-08-17T17:08:36.327 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411384 got 77309411387 for osd.2 2024-08-17T17:08:36.327 DEBUG:teuthology.parallel:result is None 2024-08-17T17:08:36.336 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:08:36.336 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:08:36.340 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:08:36.466 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:08:36 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:08:36] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-08-17T17:08:36.473 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:36 smithi053 ceph-mon[34699]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:36.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:36 smithi003 ceph-mon[30803]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:36.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:36 smithi003 ceph-mon[41530]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:36.732 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738433 got 34359738436 for osd.0 2024-08-17T17:08:36.733 DEBUG:teuthology.parallel:result is None 2024-08-17T17:08:39.133 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:38 smithi003 ceph-mon[41530]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:39.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:38 smithi003 ceph-mon[30803]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:39.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:38 smithi053 ceph-mon[34699]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:40.773 INFO:teuthology.orchestra.run.smithi003.stdout:55834574912 2024-08-17T17:08:41.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:40 smithi053 ceph-mon[34699]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:41.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:40 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/3618005650' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-08-17T17:08:41.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:40 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:08:41.258 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:40 smithi003 ceph-mon[41530]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:41.258 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:40 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/3618005650' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-08-17T17:08:41.258 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:40 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:08:41.259 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:40 smithi003 ceph-mon[30803]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:41.259 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:40 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/3618005650' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-08-17T17:08:41.259 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:40 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:08:41.488 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574908 got 55834574912 for osd.1 2024-08-17T17:08:41.488 DEBUG:teuthology.parallel:result is None 2024-08-17T17:08:41.848 INFO:teuthology.orchestra.run.smithi003.stdout:163208757293 2024-08-17T17:08:42.768 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757289 got 163208757293 for osd.5 2024-08-17T17:08:42.768 DEBUG:teuthology.parallel:result is None 2024-08-17T17:08:43.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:42 smithi003 ceph-mon[41530]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:43.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:42 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/143689912' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-08-17T17:08:43.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:42 smithi003 ceph-mon[30803]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:43.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:42 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/143689912' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-08-17T17:08:43.113 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:42 smithi053 ceph-mon[34699]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:43.113 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:42 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/143689912' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-08-17T17:08:43.192 INFO:teuthology.orchestra.run.smithi003.stdout:107374182456 2024-08-17T17:08:43.523 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:08:43 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:08:43] "GET /metrics HTTP/1.1" 200 239646 "" "Prometheus/2.43.0" 2024-08-17T17:08:43.874 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182451 got 107374182456 for osd.3 2024-08-17T17:08:43.874 DEBUG:teuthology.parallel:result is None 2024-08-17T17:08:43.874 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-08-17T17:08:43.875 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:08:43.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:43 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/3596172389' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-08-17T17:08:44.147 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:08:44.270 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:43 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/3596172389' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-08-17T17:08:44.271 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:43 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/3596172389' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-08-17T17:08:44.997 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:44 smithi003 ceph-mon[41530]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:44.997 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:44 smithi003 ceph-mon[30803]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:45.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:44 smithi053 ceph-mon[34699]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:45.946 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:08:45.947 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:08:46.667 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":87,"stamp":"2024-08-17T17:08:45.788291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592725,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":226,"num_write_kb":2431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":170,"ondisk_log_size":170,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2386584,"kb_used_data":7720,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747574632,"statfs":{"total":767960285184,"available":765516423168,"internally_reserved":0,"allocated":7905280,"data_stored":4438791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002796"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.180946+0000","last_change":"2024-08-17T17:07:47.173693+0000","last_active":"2024-08-17T17:08:26.180946+0000","last_peered":"2024-08-17T17:08:26.180946+0000","last_clean":"2024-08-17T17:08:26.180946+0000","last_became_active":"2024-08-17T17:07:47.172508+0000","last_became_peered":"2024-08-17T17:07:47.172508+0000","last_unstale":"2024-08-17T17:08:26.180946+0000","last_undegraded":"2024-08-17T17:08:26.180946+0000","last_fullsized":"2024-08-17T17:08:26.180946+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:41:09.369848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.186059+0000","last_change":"2024-08-17T17:07:46.159359+0000","last_active":"2024-08-17T17:08:26.186059+0000","last_peered":"2024-08-17T17:08:26.186059+0000","last_clean":"2024-08-17T17:08:26.186059+0000","last_became_active":"2024-08-17T17:07:46.159068+0000","last_became_peered":"2024-08-17T17:07:46.159068+0000","last_unstale":"2024-08-17T17:08:26.186059+0000","last_undegraded":"2024-08-17T17:08:26.186059+0000","last_fullsized":"2024-08-17T17:08:26.186059+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:30.695234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.180767+0000","last_change":"2024-08-17T17:07:46.298932+0000","last_active":"2024-08-17T17:08:26.180767+0000","last_peered":"2024-08-17T17:08:26.180767+0000","last_clean":"2024-08-17T17:08:26.180767+0000","last_became_active":"2024-08-17T17:07:46.298633+0000","last_became_peered":"2024-08-17T17:07:46.298633+0000","last_unstale":"2024-08-17T17:08:26.180767+0000","last_undegraded":"2024-08-17T17:08:26.180767+0000","last_fullsized":"2024-08-17T17:08:26.180767+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:52:57.325225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.176783+0000","last_change":"2024-08-17T17:07:46.160669+0000","last_active":"2024-08-17T17:08:26.176783+0000","last_peered":"2024-08-17T17:08:26.176783+0000","last_clean":"2024-08-17T17:08:26.176783+0000","last_became_active":"2024-08-17T17:07:46.159930+0000","last_became_peered":"2024-08-17T17:07:46.159930+0000","last_unstale":"2024-08-17T17:08:26.176783+0000","last_undegraded":"2024-08-17T17:08:26.176783+0000","last_fullsized":"2024-08-17T17:08:26.176783+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:18:18.672383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.179384+0000","last_change":"2024-08-17T17:07:47.170972+0000","last_active":"2024-08-17T17:08:26.179384+0000","last_peered":"2024-08-17T17:08:26.179384+0000","last_clean":"2024-08-17T17:08:26.179384+0000","last_became_active":"2024-08-17T17:07:47.170721+0000","last_became_peered":"2024-08-17T17:07:47.170721+0000","last_unstale":"2024-08-17T17:08:26.179384+0000","last_undegraded":"2024-08-17T17:08:26.179384+0000","last_fullsized":"2024-08-17T17:08:26.179384+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:52:04.443843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.185954+0000","last_change":"2024-08-17T17:07:46.158514+0000","last_active":"2024-08-17T17:08:26.185954+0000","last_peered":"2024-08-17T17:08:26.185954+0000","last_clean":"2024-08-17T17:08:26.185954+0000","last_became_active":"2024-08-17T17:07:46.158256+0000","last_became_peered":"2024-08-17T17:07:46.158256+0000","last_unstale":"2024-08-17T17:08:26.185954+0000","last_undegraded":"2024-08-17T17:08:26.185954+0000","last_fullsized":"2024-08-17T17:08:26.185954+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:30:44.486515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.181400+0000","last_change":"2024-08-17T17:07:47.176779+0000","last_active":"2024-08-17T17:08:26.181400+0000","last_peered":"2024-08-17T17:08:26.181400+0000","last_clean":"2024-08-17T17:08:26.181400+0000","last_became_active":"2024-08-17T17:07:47.176535+0000","last_became_peered":"2024-08-17T17:07:47.176535+0000","last_unstale":"2024-08-17T17:08:26.181400+0000","last_undegraded":"2024-08-17T17:08:26.181400+0000","last_fullsized":"2024-08-17T17:08:26.181400+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:42:46.283019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":22,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.178856+0000","last_change":"2024-08-17T17:07:46.152367+0000","last_active":"2024-08-17T17:08:26.178856+0000","last_peered":"2024-08-17T17:08:26.178856+0000","last_clean":"2024-08-17T17:08:26.178856+0000","last_became_active":"2024-08-17T17:07:46.151885+0000","last_became_peered":"2024-08-17T17:07:46.151885+0000","last_unstale":"2024-08-17T17:08:26.178856+0000","last_undegraded":"2024-08-17T17:08:26.178856+0000","last_fullsized":"2024-08-17T17:08:26.178856+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:42:31.587176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.186388+0000","last_change":"2024-08-17T17:07:46.157185+0000","last_active":"2024-08-17T17:08:26.186388+0000","last_peered":"2024-08-17T17:08:26.186388+0000","last_clean":"2024-08-17T17:08:26.186388+0000","last_became_active":"2024-08-17T17:07:46.156901+0000","last_became_peered":"2024-08-17T17:07:46.156901+0000","last_unstale":"2024-08-17T17:08:26.186388+0000","last_undegraded":"2024-08-17T17:08:26.186388+0000","last_fullsized":"2024-08-17T17:08:26.186388+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:59.650169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.180568+0000","last_change":"2024-08-17T17:07:46.299427+0000","last_active":"2024-08-17T17:08:26.180568+0000","last_peered":"2024-08-17T17:08:26.180568+0000","last_clean":"2024-08-17T17:08:26.180568+0000","last_became_active":"2024-08-17T17:07:46.299008+0000","last_became_peered":"2024-08-17T17:07:46.299008+0000","last_unstale":"2024-08-17T17:08:26.180568+0000","last_undegraded":"2024-08-17T17:08:26.180568+0000","last_fullsized":"2024-08-17T17:08:26.180568+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:23:32.024874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.186292+0000","last_change":"2024-08-17T17:07:46.159280+0000","last_active":"2024-08-17T17:08:26.186292+0000","last_peered":"2024-08-17T17:08:26.186292+0000","last_clean":"2024-08-17T17:08:26.186292+0000","last_became_active":"2024-08-17T17:07:46.158988+0000","last_became_peered":"2024-08-17T17:07:46.158988+0000","last_unstale":"2024-08-17T17:08:26.186292+0000","last_undegraded":"2024-08-17T17:08:26.186292+0000","last_fullsized":"2024-08-17T17:08:26.186292+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:15:42.212909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.175879+0000","last_change":"2024-08-17T17:07:47.174653+0000","last_active":"2024-08-17T17:08:26.175879+0000","last_peered":"2024-08-17T17:08:26.175879+0000","last_clean":"2024-08-17T17:08:26.175879+0000","last_became_active":"2024-08-17T17:07:47.174034+0000","last_became_peered":"2024-08-17T17:07:47.174034+0000","last_unstale":"2024-08-17T17:08:26.175879+0000","last_undegraded":"2024-08-17T17:08:26.175879+0000","last_fullsized":"2024-08-17T17:08:26.175879+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:44:53.039509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.186283+0000","last_change":"2024-08-17T17:07:46.153923+0000","last_active":"2024-08-17T17:08:26.186283+0000","last_peered":"2024-08-17T17:08:26.186283+0000","last_clean":"2024-08-17T17:08:26.186283+0000","last_became_active":"2024-08-17T17:07:46.153614+0000","last_became_peered":"2024-08-17T17:07:46.153614+0000","last_unstale":"2024-08-17T17:08:26.186283+0000","last_undegraded":"2024-08-17T17:08:26.186283+0000","last_fullsized":"2024-08-17T17:08:26.186283+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:24:38.871907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.178145+0000","last_change":"2024-08-17T17:07:47.168592+0000","last_active":"2024-08-17T17:08:26.178145+0000","last_peered":"2024-08-17T17:08:26.178145+0000","last_clean":"2024-08-17T17:08:26.178145+0000","last_became_active":"2024-08-17T17:07:47.168264+0000","last_became_peered":"2024-08-17T17:07:47.168264+0000","last_unstale":"2024-08-17T17:08:26.178145+0000","last_undegraded":"2024-08-17T17:08:26.178145+0000","last_fullsized":"2024-08-17T17:08:26.178145+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:58:07.261379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.180803+0000","last_change":"2024-08-17T17:07:46.147836+0000","last_active":"2024-08-17T17:08:26.180803+0000","last_peered":"2024-08-17T17:08:26.180803+0000","last_clean":"2024-08-17T17:08:26.180803+0000","last_became_active":"2024-08-17T17:07:46.147559+0000","last_became_peered":"2024-08-17T17:07:46.147559+0000","last_unstale":"2024-08-17T17:08:26.180803+0000","last_undegraded":"2024-08-17T17:08:26.180803+0000","last_fullsized":"2024-08-17T17:08:26.180803+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:40:58.626980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.177832+0000","last_change":"2024-08-17T17:07:47.164620+0000","last_active":"2024-08-17T17:08:26.177832+0000","last_peered":"2024-08-17T17:08:26.177832+0000","last_clean":"2024-08-17T17:08:26.177832+0000","last_became_active":"2024-08-17T17:07:47.164325+0000","last_became_peered":"2024-08-17T17:07:47.164325+0000","last_unstale":"2024-08-17T17:08:26.177832+0000","last_undegraded":"2024-08-17T17:08:26.177832+0000","last_fullsized":"2024-08-17T17:08:26.177832+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:48:09.187177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.180712+0000","last_change":"2024-08-17T17:07:46.156846+0000","last_active":"2024-08-17T17:08:26.180712+0000","last_peered":"2024-08-17T17:08:26.180712+0000","last_clean":"2024-08-17T17:08:26.180712+0000","last_became_active":"2024-08-17T17:07:46.156421+0000","last_became_peered":"2024-08-17T17:07:46.156421+0000","last_unstale":"2024-08-17T17:08:26.180712+0000","last_undegraded":"2024-08-17T17:08:26.180712+0000","last_fullsized":"2024-08-17T17:08:26.180712+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:34:13.084499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.176108+0000","last_change":"2024-08-17T17:07:47.174543+0000","last_active":"2024-08-17T17:08:26.176108+0000","last_peered":"2024-08-17T17:08:26.176108+0000","last_clean":"2024-08-17T17:08:26.176108+0000","last_became_active":"2024-08-17T17:07:47.174103+0000","last_became_peered":"2024-08-17T17:07:47.174103+0000","last_unstale":"2024-08-17T17:08:26.176108+0000","last_undegraded":"2024-08-17T17:08:26.176108+0000","last_fullsized":"2024-08-17T17:08:26.176108+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:50:30.499357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.178293+0000","last_change":"2024-08-17T17:07:46.153140+0000","last_active":"2024-08-17T17:08:26.178293+0000","last_peered":"2024-08-17T17:08:26.178293+0000","last_clean":"2024-08-17T17:08:26.178293+0000","last_became_active":"2024-08-17T17:07:46.152695+0000","last_became_peered":"2024-08-17T17:07:46.152695+0000","last_unstale":"2024-08-17T17:08:26.178293+0000","last_undegraded":"2024-08-17T17:08:26.178293+0000","last_fullsized":"2024-08-17T17:08:26.178293+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:11:32.945713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.176250+0000","last_change":"2024-08-17T17:07:47.172506+0000","last_active":"2024-08-17T17:08:26.176250+0000","last_peered":"2024-08-17T17:08:26.176250+0000","last_clean":"2024-08-17T17:08:26.176250+0000","last_became_active":"2024-08-17T17:07:47.172164+0000","last_became_peered":"2024-08-17T17:07:47.172164+0000","last_unstale":"2024-08-17T17:08:26.176250+0000","last_undegraded":"2024-08-17T17:08:26.176250+0000","last_fullsized":"2024-08-17T17:08:26.176250+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:41:05.091599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.175994+0000","last_change":"2024-08-17T17:07:46.160296+0000","last_active":"2024-08-17T17:08:26.175994+0000","last_peered":"2024-08-17T17:08:26.175994+0000","last_clean":"2024-08-17T17:08:26.175994+0000","last_became_active":"2024-08-17T17:07:46.160031+0000","last_became_peered":"2024-08-17T17:07:46.160031+0000","last_unstale":"2024-08-17T17:08:26.175994+0000","last_undegraded":"2024-08-17T17:08:26.175994+0000","last_fullsized":"2024-08-17T17:08:26.175994+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:34:21.275585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.176386+0000","last_change":"2024-08-17T17:07:47.172575+0000","last_active":"2024-08-17T17:08:26.176386+0000","last_peered":"2024-08-17T17:08:26.176386+0000","last_clean":"2024-08-17T17:08:26.176386+0000","last_became_active":"2024-08-17T17:07:47.172117+0000","last_became_peered":"2024-08-17T17:07:47.172117+0000","last_unstale":"2024-08-17T17:08:26.176386+0000","last_undegraded":"2024-08-17T17:08:26.176386+0000","last_fullsized":"2024-08-17T17:08:26.176386+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:21:26.992513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.181714+0000","last_change":"2024-08-17T17:07:46.149804+0000","last_active":"2024-08-17T17:08:26.181714+0000","last_peered":"2024-08-17T17:08:26.181714+0000","last_clean":"2024-08-17T17:08:26.181714+0000","last_became_active":"2024-08-17T17:07:46.148863+0000","last_became_peered":"2024-08-17T17:07:46.148863+0000","last_unstale":"2024-08-17T17:08:26.181714+0000","last_undegraded":"2024-08-17T17:08:26.181714+0000","last_fullsized":"2024-08-17T17:08:26.181714+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:54.599139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.183189+0000","last_change":"2024-08-17T17:07:46.167064+0000","last_active":"2024-08-17T17:08:26.183189+0000","last_peered":"2024-08-17T17:08:26.183189+0000","last_clean":"2024-08-17T17:08:26.183189+0000","last_became_active":"2024-08-17T17:07:46.166758+0000","last_became_peered":"2024-08-17T17:07:46.166758+0000","last_unstale":"2024-08-17T17:08:26.183189+0000","last_undegraded":"2024-08-17T17:08:26.183189+0000","last_fullsized":"2024-08-17T17:08:26.183189+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:22:13.086529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"59'128","reported_seq":281,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.185928+0000","last_change":"2024-08-17T17:05:57.181832+0000","last_active":"2024-08-17T17:08:26.185928+0000","last_peered":"2024-08-17T17:08:26.185928+0000","last_clean":"2024-08-17T17:08:26.185928+0000","last_became_active":"2024-08-17T17:05:57.181461+0000","last_became_peered":"2024-08-17T17:05:57.181461+0000","last_unstale":"2024-08-17T17:08:26.185928+0000","last_undegraded":"2024-08-17T17:08:26.185928+0000","last_fullsized":"2024-08-17T17:08:26.185928+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:04:01.141401+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:10:53.855422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":27,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.180694+0000","last_change":"2024-08-17T17:06:34.534038+0000","last_active":"2024-08-17T17:08:26.180694+0000","last_peered":"2024-08-17T17:08:26.180694+0000","last_clean":"2024-08-17T17:08:26.180694+0000","last_became_active":"2024-08-17T17:06:29.779691+0000","last_became_peered":"2024-08-17T17:06:29.779691+0000","last_unstale":"2024-08-17T17:08:26.180694+0000","last_undegraded":"2024-08-17T17:08:26.180694+0000","last_fullsized":"2024-08-17T17:08:26.180694+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:51:04.192689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032951500000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.180948+0000","last_change":"2024-08-17T17:07:46.154584+0000","last_active":"2024-08-17T17:08:26.180948+0000","last_peered":"2024-08-17T17:08:26.180948+0000","last_clean":"2024-08-17T17:08:26.180948+0000","last_became_active":"2024-08-17T17:07:46.154296+0000","last_became_peered":"2024-08-17T17:07:46.154296+0000","last_unstale":"2024-08-17T17:08:26.180948+0000","last_undegraded":"2024-08-17T17:08:26.180948+0000","last_fullsized":"2024-08-17T17:08:26.180948+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:02:23.006828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.178431+0000","last_change":"2024-08-17T17:07:47.164375+0000","last_active":"2024-08-17T17:08:26.178431+0000","last_peered":"2024-08-17T17:08:26.178431+0000","last_clean":"2024-08-17T17:08:26.178431+0000","last_became_active":"2024-08-17T17:07:47.164077+0000","last_became_peered":"2024-08-17T17:07:47.164077+0000","last_unstale":"2024-08-17T17:08:26.178431+0000","last_undegraded":"2024-08-17T17:08:26.178431+0000","last_fullsized":"2024-08-17T17:08:26.178431+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:26:01.278743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":26,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.185860+0000","last_change":"2024-08-17T17:06:34.999031+0000","last_active":"2024-08-17T17:08:26.185860+0000","last_peered":"2024-08-17T17:08:26.185860+0000","last_clean":"2024-08-17T17:08:26.185860+0000","last_became_active":"2024-08-17T17:06:29.785264+0000","last_became_peered":"2024-08-17T17:06:29.785264+0000","last_unstale":"2024-08-17T17:08:26.185860+0000","last_undegraded":"2024-08-17T17:08:26.185860+0000","last_fullsized":"2024-08-17T17:08:26.185860+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:59:39.460037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026064700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.177230+0000","last_change":"2024-08-17T17:07:46.162672+0000","last_active":"2024-08-17T17:08:26.177230+0000","last_peered":"2024-08-17T17:08:26.177230+0000","last_clean":"2024-08-17T17:08:26.177230+0000","last_became_active":"2024-08-17T17:07:46.162058+0000","last_became_peered":"2024-08-17T17:07:46.162058+0000","last_unstale":"2024-08-17T17:08:26.177230+0000","last_undegraded":"2024-08-17T17:08:26.177230+0000","last_fullsized":"2024-08-17T17:08:26.177230+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:35:33.825034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-08-17T17:08:27.336966+0000","last_change":"2024-08-17T17:08:27.336966+0000","last_active":"2024-08-17T17:08:27.336966+0000","last_peered":"2024-08-17T17:08:27.336966+0000","last_clean":"2024-08-17T17:08:27.336966+0000","last_became_active":"2024-08-17T17:08:27.336581+0000","last_became_peered":"2024-08-17T17:08:27.336581+0000","last_unstale":"2024-08-17T17:08:27.336966+0000","last_undegraded":"2024-08-17T17:08:27.336966+0000","last_fullsized":"2024-08-17T17:08:27.336966+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:23:51.597327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.177255+0000","last_change":"2024-08-17T17:06:34.600672+0000","last_active":"2024-08-17T17:08:26.177255+0000","last_peered":"2024-08-17T17:08:26.177255+0000","last_clean":"2024-08-17T17:08:26.177255+0000","last_became_active":"2024-08-17T17:06:29.830105+0000","last_became_peered":"2024-08-17T17:06:29.830105+0000","last_unstale":"2024-08-17T17:08:26.177255+0000","last_undegraded":"2024-08-17T17:08:26.177255+0000","last_fullsized":"2024-08-17T17:08:26.177255+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:42:07.154002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000234334,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.180729+0000","last_change":"2024-08-17T17:07:46.148883+0000","last_active":"2024-08-17T17:08:26.180729+0000","last_peered":"2024-08-17T17:08:26.180729+0000","last_clean":"2024-08-17T17:08:26.180729+0000","last_became_active":"2024-08-17T17:07:46.148609+0000","last_became_peered":"2024-08-17T17:07:46.148609+0000","last_unstale":"2024-08-17T17:08:26.180729+0000","last_undegraded":"2024-08-17T17:08:26.180729+0000","last_fullsized":"2024-08-17T17:08:26.180729+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:03:35.797335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.183225+0000","last_change":"2024-08-17T17:07:46.300160+0000","last_active":"2024-08-17T17:08:26.183225+0000","last_peered":"2024-08-17T17:08:26.183225+0000","last_clean":"2024-08-17T17:08:26.183225+0000","last_became_active":"2024-08-17T17:07:46.299500+0000","last_became_peered":"2024-08-17T17:07:46.299500+0000","last_unstale":"2024-08-17T17:08:26.183225+0000","last_undegraded":"2024-08-17T17:08:26.183225+0000","last_fullsized":"2024-08-17T17:08:26.183225+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:56:50.286131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":28,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.180691+0000","last_change":"2024-08-17T17:06:34.534595+0000","last_active":"2024-08-17T17:08:26.180691+0000","last_peered":"2024-08-17T17:08:26.180691+0000","last_clean":"2024-08-17T17:08:26.180691+0000","last_became_active":"2024-08-17T17:06:29.777933+0000","last_became_peered":"2024-08-17T17:06:29.777933+0000","last_unstale":"2024-08-17T17:08:26.180691+0000","last_undegraded":"2024-08-17T17:08:26.180691+0000","last_fullsized":"2024-08-17T17:08:26.180691+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:33:08.360328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087151500000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.176503+0000","last_change":"2024-08-17T17:07:46.162284+0000","last_active":"2024-08-17T17:08:26.176503+0000","last_peered":"2024-08-17T17:08:26.176503+0000","last_clean":"2024-08-17T17:08:26.176503+0000","last_became_active":"2024-08-17T17:07:46.162009+0000","last_became_peered":"2024-08-17T17:07:46.162009+0000","last_unstale":"2024-08-17T17:08:26.176503+0000","last_undegraded":"2024-08-17T17:08:26.176503+0000","last_fullsized":"2024-08-17T17:08:26.176503+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:35:15.813252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.182788+0000","last_change":"2024-08-17T17:07:46.300897+0000","last_active":"2024-08-17T17:08:26.182788+0000","last_peered":"2024-08-17T17:08:26.182788+0000","last_clean":"2024-08-17T17:08:26.182788+0000","last_became_active":"2024-08-17T17:07:46.300558+0000","last_became_peered":"2024-08-17T17:07:46.300558+0000","last_unstale":"2024-08-17T17:08:26.182788+0000","last_undegraded":"2024-08-17T17:08:26.182788+0000","last_fullsized":"2024-08-17T17:08:26.182788+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:15:37.664039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.180939+0000","last_change":"2024-08-17T17:07:46.150254+0000","last_active":"2024-08-17T17:08:26.180939+0000","last_peered":"2024-08-17T17:08:26.180939+0000","last_clean":"2024-08-17T17:08:26.180939+0000","last_became_active":"2024-08-17T17:07:46.149875+0000","last_became_peered":"2024-08-17T17:07:46.149875+0000","last_unstale":"2024-08-17T17:08:26.180939+0000","last_undegraded":"2024-08-17T17:08:26.180939+0000","last_fullsized":"2024-08-17T17:08:26.180939+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:30:50.355502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.182950+0000","last_change":"2024-08-17T17:06:34.530210+0000","last_active":"2024-08-17T17:08:26.182950+0000","last_peered":"2024-08-17T17:08:26.182950+0000","last_clean":"2024-08-17T17:08:26.182950+0000","last_became_active":"2024-08-17T17:06:29.779126+0000","last_became_peered":"2024-08-17T17:06:29.779126+0000","last_unstale":"2024-08-17T17:08:26.182950+0000","last_undegraded":"2024-08-17T17:08:26.182950+0000","last_fullsized":"2024-08-17T17:08:26.182950+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:53:27.601004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028588399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.181750+0000","last_change":"2024-08-17T17:07:46.166149+0000","last_active":"2024-08-17T17:08:26.181750+0000","last_peered":"2024-08-17T17:08:26.181750+0000","last_clean":"2024-08-17T17:08:26.181750+0000","last_became_active":"2024-08-17T17:07:46.165844+0000","last_became_peered":"2024-08-17T17:07:46.165844+0000","last_unstale":"2024-08-17T17:08:26.181750+0000","last_undegraded":"2024-08-17T17:08:26.181750+0000","last_fullsized":"2024-08-17T17:08:26.181750+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:28:13.437271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.182882+0000","last_change":"2024-08-17T17:07:46.149720+0000","last_active":"2024-08-17T17:08:26.182882+0000","last_peered":"2024-08-17T17:08:26.182882+0000","last_clean":"2024-08-17T17:08:26.182882+0000","last_became_active":"2024-08-17T17:07:46.149023+0000","last_became_peered":"2024-08-17T17:07:46.149023+0000","last_unstale":"2024-08-17T17:08:26.182882+0000","last_undegraded":"2024-08-17T17:08:26.182882+0000","last_fullsized":"2024-08-17T17:08:26.182882+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:19:20.278134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":26,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.186310+0000","last_change":"2024-08-17T17:06:34.999020+0000","last_active":"2024-08-17T17:08:26.186310+0000","last_peered":"2024-08-17T17:08:26.186310+0000","last_clean":"2024-08-17T17:08:26.186310+0000","last_became_active":"2024-08-17T17:06:29.830625+0000","last_became_peered":"2024-08-17T17:06:29.830625+0000","last_unstale":"2024-08-17T17:08:26.186310+0000","last_undegraded":"2024-08-17T17:08:26.186310+0000","last_fullsized":"2024-08-17T17:08:26.186310+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:23:50.149657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000290981,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.177924+0000","last_change":"2024-08-17T17:07:47.227288+0000","last_active":"2024-08-17T17:08:26.177924+0000","last_peered":"2024-08-17T17:08:26.177924+0000","last_clean":"2024-08-17T17:08:26.177924+0000","last_became_active":"2024-08-17T17:07:47.226969+0000","last_became_peered":"2024-08-17T17:07:47.226969+0000","last_unstale":"2024-08-17T17:08:26.177924+0000","last_undegraded":"2024-08-17T17:08:26.177924+0000","last_fullsized":"2024-08-17T17:08:26.177924+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:28:51.453724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.181473+0000","last_change":"2024-08-17T17:07:46.157541+0000","last_active":"2024-08-17T17:08:26.181473+0000","last_peered":"2024-08-17T17:08:26.181473+0000","last_clean":"2024-08-17T17:08:26.181473+0000","last_became_active":"2024-08-17T17:07:46.157288+0000","last_became_peered":"2024-08-17T17:07:46.157288+0000","last_unstale":"2024-08-17T17:08:26.181473+0000","last_undegraded":"2024-08-17T17:08:26.181473+0000","last_fullsized":"2024-08-17T17:08:26.181473+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:18:42.512643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":26,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.183084+0000","last_change":"2024-08-17T17:06:34.530258+0000","last_active":"2024-08-17T17:08:26.183084+0000","last_peered":"2024-08-17T17:08:26.183084+0000","last_clean":"2024-08-17T17:08:26.183084+0000","last_became_active":"2024-08-17T17:06:29.831452+0000","last_became_peered":"2024-08-17T17:06:29.831452+0000","last_unstale":"2024-08-17T17:08:26.183084+0000","last_undegraded":"2024-08-17T17:08:26.183084+0000","last_fullsized":"2024-08-17T17:08:26.183084+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:36:53.341363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027697000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.181564+0000","last_change":"2024-08-17T17:07:47.172149+0000","last_active":"2024-08-17T17:08:26.181564+0000","last_peered":"2024-08-17T17:08:26.181564+0000","last_clean":"2024-08-17T17:08:26.181564+0000","last_became_active":"2024-08-17T17:07:47.171895+0000","last_became_peered":"2024-08-17T17:07:47.171895+0000","last_unstale":"2024-08-17T17:08:26.181564+0000","last_undegraded":"2024-08-17T17:08:26.181564+0000","last_fullsized":"2024-08-17T17:08:26.181564+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:33:59.514052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.179705+0000","last_change":"2024-08-17T17:07:46.160364+0000","last_active":"2024-08-17T17:08:26.179705+0000","last_peered":"2024-08-17T17:08:26.179705+0000","last_clean":"2024-08-17T17:08:26.179705+0000","last_became_active":"2024-08-17T17:07:46.160077+0000","last_became_peered":"2024-08-17T17:07:46.160077+0000","last_unstale":"2024-08-17T17:08:26.179705+0000","last_undegraded":"2024-08-17T17:08:26.179705+0000","last_fullsized":"2024-08-17T17:08:26.179705+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:46:56.843721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":26,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.178554+0000","last_change":"2024-08-17T17:06:34.535627+0000","last_active":"2024-08-17T17:08:26.178554+0000","last_peered":"2024-08-17T17:08:26.178554+0000","last_clean":"2024-08-17T17:08:26.178554+0000","last_became_active":"2024-08-17T17:06:29.779504+0000","last_became_peered":"2024-08-17T17:06:29.779504+0000","last_unstale":"2024-08-17T17:08:26.178554+0000","last_undegraded":"2024-08-17T17:08:26.178554+0000","last_fullsized":"2024-08-17T17:08:26.178554+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:21:33.158491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000362901,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.175789+0000","last_change":"2024-08-17T17:07:47.227287+0000","last_active":"2024-08-17T17:08:26.175789+0000","last_peered":"2024-08-17T17:08:26.175789+0000","last_clean":"2024-08-17T17:08:26.175789+0000","last_became_active":"2024-08-17T17:07:47.226983+0000","last_became_peered":"2024-08-17T17:07:47.226983+0000","last_unstale":"2024-08-17T17:08:26.175789+0000","last_undegraded":"2024-08-17T17:08:26.175789+0000","last_fullsized":"2024-08-17T17:08:26.175789+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:34:07.444142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.176190+0000","last_change":"2024-08-17T17:07:47.172582+0000","last_active":"2024-08-17T17:08:26.176190+0000","last_peered":"2024-08-17T17:08:26.176190+0000","last_clean":"2024-08-17T17:08:26.176190+0000","last_became_active":"2024-08-17T17:07:47.172267+0000","last_became_peered":"2024-08-17T17:07:47.172267+0000","last_unstale":"2024-08-17T17:08:26.176190+0000","last_undegraded":"2024-08-17T17:08:26.176190+0000","last_fullsized":"2024-08-17T17:08:26.176190+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:21:00.234634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"71'6","reported_seq":20,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.176239+0000","last_change":"2024-08-17T17:07:46.160694+0000","last_active":"2024-08-17T17:08:26.176239+0000","last_peered":"2024-08-17T17:08:26.176239+0000","last_clean":"2024-08-17T17:08:26.176239+0000","last_became_active":"2024-08-17T17:07:46.160407+0000","last_became_peered":"2024-08-17T17:07:46.160407+0000","last_unstale":"2024-08-17T17:08:26.176239+0000","last_undegraded":"2024-08-17T17:08:26.176239+0000","last_fullsized":"2024-08-17T17:08:26.176239+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:54:38.366094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1552,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.181892+0000","last_change":"2024-08-17T17:07:46.299965+0000","last_active":"2024-08-17T17:08:26.181892+0000","last_peered":"2024-08-17T17:08:26.181892+0000","last_clean":"2024-08-17T17:08:26.181892+0000","last_became_active":"2024-08-17T17:07:46.299319+0000","last_became_peered":"2024-08-17T17:07:46.299319+0000","last_unstale":"2024-08-17T17:08:26.181892+0000","last_undegraded":"2024-08-17T17:08:26.181892+0000","last_fullsized":"2024-08-17T17:08:26.181892+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:31:25.609194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.186200+0000","last_change":"2024-08-17T17:07:46.157895+0000","last_active":"2024-08-17T17:08:26.186200+0000","last_peered":"2024-08-17T17:08:26.186200+0000","last_clean":"2024-08-17T17:08:26.186200+0000","last_became_active":"2024-08-17T17:07:46.157636+0000","last_became_peered":"2024-08-17T17:07:46.157636+0000","last_unstale":"2024-08-17T17:08:26.186200+0000","last_undegraded":"2024-08-17T17:08:26.186200+0000","last_fullsized":"2024-08-17T17:08:26.186200+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:19:21.578951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.180826+0000","last_change":"2024-08-17T17:07:46.151440+0000","last_active":"2024-08-17T17:08:26.180826+0000","last_peered":"2024-08-17T17:08:26.180826+0000","last_clean":"2024-08-17T17:08:26.180826+0000","last_became_active":"2024-08-17T17:07:46.150468+0000","last_became_peered":"2024-08-17T17:07:46.150468+0000","last_unstale":"2024-08-17T17:08:26.180826+0000","last_undegraded":"2024-08-17T17:08:26.180826+0000","last_fullsized":"2024-08-17T17:08:26.180826+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:51.933623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.180946+0000","last_change":"2024-08-17T17:07:47.171412+0000","last_active":"2024-08-17T17:08:26.180946+0000","last_peered":"2024-08-17T17:08:26.180946+0000","last_clean":"2024-08-17T17:08:26.180946+0000","last_became_active":"2024-08-17T17:07:47.171143+0000","last_became_peered":"2024-08-17T17:07:47.171143+0000","last_unstale":"2024-08-17T17:08:26.180946+0000","last_undegraded":"2024-08-17T17:08:26.180946+0000","last_fullsized":"2024-08-17T17:08:26.180946+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:53:16.355093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.179294+0000","last_change":"2024-08-17T17:07:46.161427+0000","last_active":"2024-08-17T17:08:26.179294+0000","last_peered":"2024-08-17T17:08:26.179294+0000","last_clean":"2024-08-17T17:08:26.179294+0000","last_became_active":"2024-08-17T17:07:46.161140+0000","last_became_peered":"2024-08-17T17:07:46.161140+0000","last_unstale":"2024-08-17T17:08:26.179294+0000","last_undegraded":"2024-08-17T17:08:26.179294+0000","last_fullsized":"2024-08-17T17:08:26.179294+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:15.549311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.181489+0000","last_change":"2024-08-17T17:07:47.172334+0000","last_active":"2024-08-17T17:08:26.181489+0000","last_peered":"2024-08-17T17:08:26.181489+0000","last_clean":"2024-08-17T17:08:26.181489+0000","last_became_active":"2024-08-17T17:07:47.172071+0000","last_became_peered":"2024-08-17T17:07:47.172071+0000","last_unstale":"2024-08-17T17:08:26.181489+0000","last_undegraded":"2024-08-17T17:08:26.181489+0000","last_fullsized":"2024-08-17T17:08:26.181489+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:10:45.779777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.178548+0000","last_change":"2024-08-17T17:07:47.227351+0000","last_active":"2024-08-17T17:08:26.178548+0000","last_peered":"2024-08-17T17:08:26.178548+0000","last_clean":"2024-08-17T17:08:26.178548+0000","last_became_active":"2024-08-17T17:07:47.226799+0000","last_became_peered":"2024-08-17T17:07:47.226799+0000","last_unstale":"2024-08-17T17:08:26.178548+0000","last_undegraded":"2024-08-17T17:08:26.178548+0000","last_fullsized":"2024-08-17T17:08:26.178548+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:32:03.857904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.181638+0000","last_change":"2024-08-17T17:07:46.156840+0000","last_active":"2024-08-17T17:08:26.181638+0000","last_peered":"2024-08-17T17:08:26.181638+0000","last_clean":"2024-08-17T17:08:26.181638+0000","last_became_active":"2024-08-17T17:07:46.156566+0000","last_became_peered":"2024-08-17T17:07:46.156566+0000","last_unstale":"2024-08-17T17:08:26.181638+0000","last_undegraded":"2024-08-17T17:08:26.181638+0000","last_fullsized":"2024-08-17T17:08:26.181638+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:53:53.403475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.176730+0000","last_change":"2024-08-17T17:07:47.170033+0000","last_active":"2024-08-17T17:08:26.176730+0000","last_peered":"2024-08-17T17:08:26.176730+0000","last_clean":"2024-08-17T17:08:26.176730+0000","last_became_active":"2024-08-17T17:07:47.169729+0000","last_became_peered":"2024-08-17T17:07:47.169729+0000","last_unstale":"2024-08-17T17:08:26.176730+0000","last_undegraded":"2024-08-17T17:08:26.176730+0000","last_fullsized":"2024-08-17T17:08:26.176730+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:22:24.637462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.181885+0000","last_change":"2024-08-17T17:07:46.160245+0000","last_active":"2024-08-17T17:08:26.181885+0000","last_peered":"2024-08-17T17:08:26.181885+0000","last_clean":"2024-08-17T17:08:26.181885+0000","last_became_active":"2024-08-17T17:07:46.159973+0000","last_became_peered":"2024-08-17T17:07:46.159973+0000","last_unstale":"2024-08-17T17:08:26.181885+0000","last_undegraded":"2024-08-17T17:08:26.181885+0000","last_fullsized":"2024-08-17T17:08:26.181885+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:01:25.166219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":23,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-08-17T17:08:27.337450+0000","last_change":"2024-08-17T17:08:27.337450+0000","last_active":"2024-08-17T17:08:27.337450+0000","last_peered":"2024-08-17T17:08:27.337450+0000","last_clean":"2024-08-17T17:08:27.337450+0000","last_became_active":"2024-08-17T17:08:27.337024+0000","last_became_peered":"2024-08-17T17:08:27.337024+0000","last_unstale":"2024-08-17T17:08:27.337450+0000","last_undegraded":"2024-08-17T17:08:27.337450+0000","last_fullsized":"2024-08-17T17:08:27.337450+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:09:29.610727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.176058+0000","last_change":"2024-08-17T17:07:46.158700+0000","last_active":"2024-08-17T17:08:26.176058+0000","last_peered":"2024-08-17T17:08:26.176058+0000","last_clean":"2024-08-17T17:08:26.176058+0000","last_became_active":"2024-08-17T17:07:46.158397+0000","last_became_peered":"2024-08-17T17:07:46.158397+0000","last_unstale":"2024-08-17T17:08:26.176058+0000","last_undegraded":"2024-08-17T17:08:26.176058+0000","last_fullsized":"2024-08-17T17:08:26.176058+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:23:03.270667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.177461+0000","last_change":"2024-08-17T17:07:47.174396+0000","last_active":"2024-08-17T17:08:26.177461+0000","last_peered":"2024-08-17T17:08:26.177461+0000","last_clean":"2024-08-17T17:08:26.177461+0000","last_became_active":"2024-08-17T17:07:47.174025+0000","last_became_peered":"2024-08-17T17:07:47.174025+0000","last_unstale":"2024-08-17T17:08:26.177461+0000","last_undegraded":"2024-08-17T17:08:26.177461+0000","last_fullsized":"2024-08-17T17:08:26.177461+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:33:09.032654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.178091+0000","last_change":"2024-08-17T17:07:46.160470+0000","last_active":"2024-08-17T17:08:26.178091+0000","last_peered":"2024-08-17T17:08:26.178091+0000","last_clean":"2024-08-17T17:08:26.178091+0000","last_became_active":"2024-08-17T17:07:46.160214+0000","last_became_peered":"2024-08-17T17:07:46.160214+0000","last_unstale":"2024-08-17T17:08:26.178091+0000","last_undegraded":"2024-08-17T17:08:26.178091+0000","last_fullsized":"2024-08-17T17:08:26.178091+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:25:43.560945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.176138+0000","last_change":"2024-08-17T17:07:47.174519+0000","last_active":"2024-08-17T17:08:26.176138+0000","last_peered":"2024-08-17T17:08:26.176138+0000","last_clean":"2024-08-17T17:08:26.176138+0000","last_became_active":"2024-08-17T17:07:47.174128+0000","last_became_peered":"2024-08-17T17:07:47.174128+0000","last_unstale":"2024-08-17T17:08:26.176138+0000","last_undegraded":"2024-08-17T17:08:26.176138+0000","last_fullsized":"2024-08-17T17:08:26.176138+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:27:58.327661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.180705+0000","last_change":"2024-08-17T17:07:46.154087+0000","last_active":"2024-08-17T17:08:26.180705+0000","last_peered":"2024-08-17T17:08:26.180705+0000","last_clean":"2024-08-17T17:08:26.180705+0000","last_became_active":"2024-08-17T17:07:46.153779+0000","last_became_peered":"2024-08-17T17:07:46.153779+0000","last_unstale":"2024-08-17T17:08:26.180705+0000","last_undegraded":"2024-08-17T17:08:26.180705+0000","last_fullsized":"2024-08-17T17:08:26.180705+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:20:54.861570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.175881+0000","last_change":"2024-08-17T17:07:47.174452+0000","last_active":"2024-08-17T17:08:26.175881+0000","last_peered":"2024-08-17T17:08:26.175881+0000","last_clean":"2024-08-17T17:08:26.175881+0000","last_became_active":"2024-08-17T17:07:47.174140+0000","last_became_peered":"2024-08-17T17:07:47.174140+0000","last_unstale":"2024-08-17T17:08:26.175881+0000","last_undegraded":"2024-08-17T17:08:26.175881+0000","last_fullsized":"2024-08-17T17:08:26.175881+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:13:27.133343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.180895+0000","last_change":"2024-08-17T17:07:46.156750+0000","last_active":"2024-08-17T17:08:26.180895+0000","last_peered":"2024-08-17T17:08:26.180895+0000","last_clean":"2024-08-17T17:08:26.180895+0000","last_became_active":"2024-08-17T17:07:46.156253+0000","last_became_peered":"2024-08-17T17:07:46.156253+0000","last_unstale":"2024-08-17T17:08:26.180895+0000","last_undegraded":"2024-08-17T17:08:26.180895+0000","last_fullsized":"2024-08-17T17:08:26.180895+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:04:28.549130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.181088+0000","last_change":"2024-08-17T17:07:47.173463+0000","last_active":"2024-08-17T17:08:26.181088+0000","last_peered":"2024-08-17T17:08:26.181088+0000","last_clean":"2024-08-17T17:08:26.181088+0000","last_became_active":"2024-08-17T17:07:47.173151+0000","last_became_peered":"2024-08-17T17:07:47.173151+0000","last_unstale":"2024-08-17T17:08:26.181088+0000","last_undegraded":"2024-08-17T17:08:26.181088+0000","last_fullsized":"2024-08-17T17:08:26.181088+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:33:55.409780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.181309+0000","last_change":"2024-08-17T17:07:46.155673+0000","last_active":"2024-08-17T17:08:26.181309+0000","last_peered":"2024-08-17T17:08:26.181309+0000","last_clean":"2024-08-17T17:08:26.181309+0000","last_became_active":"2024-08-17T17:07:46.154508+0000","last_became_peered":"2024-08-17T17:07:46.154508+0000","last_unstale":"2024-08-17T17:08:26.181309+0000","last_undegraded":"2024-08-17T17:08:26.181309+0000","last_fullsized":"2024-08-17T17:08:26.181309+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:55:57.999423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":23,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-08-17T17:08:27.337685+0000","last_change":"2024-08-17T17:08:27.337685+0000","last_active":"2024-08-17T17:08:27.337685+0000","last_peered":"2024-08-17T17:08:27.337685+0000","last_clean":"2024-08-17T17:08:27.337685+0000","last_became_active":"2024-08-17T17:08:27.337163+0000","last_became_peered":"2024-08-17T17:08:27.337163+0000","last_unstale":"2024-08-17T17:08:27.337685+0000","last_undegraded":"2024-08-17T17:08:27.337685+0000","last_fullsized":"2024-08-17T17:08:27.337685+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:27:11.953158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":18,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.179133+0000","last_change":"2024-08-17T17:07:46.156321+0000","last_active":"2024-08-17T17:08:26.179133+0000","last_peered":"2024-08-17T17:08:26.179133+0000","last_clean":"2024-08-17T17:08:26.179133+0000","last_became_active":"2024-08-17T17:07:46.156066+0000","last_became_peered":"2024-08-17T17:07:46.156066+0000","last_unstale":"2024-08-17T17:08:26.179133+0000","last_undegraded":"2024-08-17T17:08:26.179133+0000","last_fullsized":"2024-08-17T17:08:26.179133+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:52.422656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2338,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":7014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234021,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298700,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446452,"statfs":{"total":95995035648,"available":95689166848,"internally_reserved":0,"allocated":1372160,"data_stored":924537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.55500000000000005,"15min":0.56200000000000006},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.72099999999999997,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.58399999999999996,"15min":0.59099999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.69799999999999995,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.66500000000000004}]},{"osd":1,"last update":"Sat Aug 17 17:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.58499999999999996,"15min":0.59399999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.59399999999999997,"15min":0.60199999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.76000000000000001,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.59099999999999997}]},{"osd":2,"last update":"Sat Aug 17 17:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.54600000000000004,"15min":0.55300000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.67800000000000005,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.59699999999999998,"15min":0.60699999999999998},"min":{"1min":0.39300000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.69699999999999995,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.72399999999999998}]},{"osd":3,"last update":"Sat Aug 17 17:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.56699999999999995,"15min":0.57199999999999995},"min":{"1min":0.41199999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.68300000000000005,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.60499999999999998,"15min":0.61099999999999999},"min":{"1min":0.38700000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.77900000000000003,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.68400000000000005}]},{"osd":4,"last update":"Sat Aug 17 17:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.48699999999999999,"15min":0.49299999999999999},"min":{"1min":0.29399999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.50700000000000001,"15min":0.51400000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.74099999999999999,"5min":0.878,"15min":0.878},"last":0.435}]},{"osd":5,"last update":"Sat Aug 17 17:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51200000000000001,"15min":0.51500000000000001},"min":{"1min":0.34499999999999997,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.81000000000000005,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.56499999999999995,"15min":0.57699999999999996},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.70199999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.45900000000000002}]},{"osd":6,"last update":"Sat Aug 17 17:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.58699999999999997,"15min":0.59699999999999998},"min":{"1min":0.372,"5min":0.316,"15min":0.316},"max":{"1min":0.747,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.54400000000000004,"15min":0.54800000000000004},"min":{"1min":0.32500000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.73699999999999999,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.47899999999999998}]}]},{"osd":6,"up_from":46,"seq":197568495656,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298688,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446464,"statfs":{"total":95995035648,"available":95689179136,"internally_reserved":0,"allocated":1359872,"data_stored":924203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.51800000000000002,"15min":0.52100000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.68600000000000005,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52500000000000002,"15min":0.52300000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.70499999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.57499999999999996}]},{"osd":1,"last update":"Sat Aug 17 17:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.54200000000000004,"15min":0.54900000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.71299999999999997,"5min":0.754,"15min":0.754},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.5,"5min":0.50900000000000001,"15min":0.505},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.54200000000000004}]},{"osd":2,"last update":"Sat Aug 17 17:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.70799999999999996,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.53800000000000003,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.69099999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.59299999999999997}]},{"osd":3,"last update":"Sat Aug 17 17:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.52100000000000002,"15min":0.52600000000000002},"min":{"1min":0.33300000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.73799999999999999,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.38800000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.746,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.64200000000000002}]},{"osd":4,"last update":"Sat Aug 17 17:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.44500000000000001,"15min":0.44400000000000001},"min":{"1min":0.32000000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.66200000000000003,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.45700000000000002,"15min":0.45500000000000002},"min":{"1min":0.315,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68799999999999994,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.42799999999999999}]},{"osd":5,"last update":"Sat Aug 17 17:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.47199999999999998,"15min":0.46200000000000002},"min":{"1min":0.33200000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.745,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.47399999999999998,"15min":0.46400000000000002},"min":{"1min":0.29699999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73399999999999999,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.495}]},{"osd":7,"last update":"Sat Aug 17 17:07:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.49199999999999999,"15min":0.48199999999999998},"min":{"1min":0.35999999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.54800000000000004,"15min":0.54200000000000004},"min":{"1min":0.38900000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.72699999999999998,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.625}]}]},{"osd":5,"up_from":38,"seq":163208757293,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298100,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447052,"statfs":{"total":95995035648,"available":95689781248,"internally_reserved":0,"allocated":761856,"data_stored":333060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:08:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.57799999999999996,"15min":0.60399999999999998},"min":{"1min":0.34999999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.66400000000000003,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.59199999999999997,"15min":0.59399999999999997},"min":{"1min":0.36699999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.66900000000000004,"5min":3.5779999999999998,"15min":3.5779999999999998},"last":0.56100000000000005}]},{"osd":1,"last update":"Sat Aug 17 17:08:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.60699999999999998,"15min":0.61699999999999999},"min":{"1min":0.42199999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.66200000000000003,"5min":3.5950000000000002,"15min":3.5950000000000002},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.55500000000000005,"15min":0.57399999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70799999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.72199999999999998}]},{"osd":2,"last update":"Sat Aug 17 17:08:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.61699999999999999,"15min":0.624},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.64900000000000002,"5min":3.6160000000000001,"15min":3.6160000000000001},"last":0.629},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.59999999999999998,"15min":0.629},"min":{"1min":0.42299999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.63500000000000001,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.57499999999999996}]},{"osd":3,"last update":"Sat Aug 17 17:08:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.59399999999999997,"15min":0.61399999999999999},"min":{"1min":0.38500000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.70099999999999996,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.59899999999999998,"15min":0.629},"min":{"1min":0.39400000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.68000000000000005,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.64600000000000002}]},{"osd":4,"last update":"Sat Aug 17 17:08:15 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.436,"15min":0.42799999999999999},"min":{"1min":0.307,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.439},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.432},"min":{"1min":0.312,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.69899999999999995,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.45700000000000002}]},{"osd":6,"last update":"Sat Aug 17 17:08:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.48499999999999999,"15min":0.47299999999999998},"min":{"1min":0.35799999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.76900000000000002,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.48399999999999999,"15min":0.47599999999999998},"min":{"1min":0.32600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.72899999999999998,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.34799999999999998}]},{"osd":7,"last update":"Sat Aug 17 17:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.78900000000000003,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54200000000000004,"15min":0.54600000000000004},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.74299999999999999,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.47999999999999998}]}]},{"osd":4,"up_from":30,"seq":128849018931,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298120,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447032,"statfs":{"total":95995035648,"available":95689760768,"internally_reserved":0,"allocated":778240,"data_stored":334422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.50900000000000001,"15min":0.51700000000000002},"min":{"1min":0.33600000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.66000000000000003,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.54800000000000004,"15min":0.54400000000000004},"min":{"1min":0.39900000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.81599999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.67200000000000004}]},{"osd":1,"last update":"Sat Aug 17 17:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.54200000000000004,"15min":0.55100000000000005},"min":{"1min":0.35299999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.64700000000000002,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.628},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.53900000000000003,"15min":0.53700000000000003},"min":{"1min":0.40999999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.68000000000000005,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.77300000000000002}]},{"osd":2,"last update":"Sat Aug 17 17:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.53600000000000003,"15min":0.54400000000000004},"min":{"1min":0.36599999999999999,"5min":0.313,"15min":0.313},"max":{"1min":0.71199999999999997,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.746},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.52400000000000002,"15min":0.51800000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.63700000000000001}]},{"osd":3,"last update":"Sat Aug 17 17:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55200000000000005,"15min":0.55900000000000005},"min":{"1min":0.42799999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.72799999999999998,"5min":0.752,"15min":0.752},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.41799999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69999999999999996,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.68300000000000005}]},{"osd":5,"last update":"Sat Aug 17 17:08:11 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.40999999999999998,"15min":0.40500000000000003},"min":{"1min":0.32100000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.56499999999999995,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.436},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.44400000000000001,"15min":0.45000000000000001},"min":{"1min":0.28899999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.48599999999999999}]},{"osd":6,"last update":"Sat Aug 17 17:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.46200000000000002,"15min":0.46500000000000002},"min":{"1min":0.34000000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.55000000000000004,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.44700000000000001,"15min":0.442},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.70799999999999996,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.56299999999999994}]},{"osd":7,"last update":"Sat Aug 17 17:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51000000000000001,"15min":0.50900000000000001},"min":{"1min":0.41799999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.50600000000000001,"15min":0.50800000000000001},"min":{"1min":0.33700000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.65700000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182456,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298100,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447052,"statfs":{"total":95995035648,"available":95689781248,"internally_reserved":0,"allocated":757760,"data_stored":332814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:08:38 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.38300000000000001,"15min":0.40100000000000002},"min":{"1min":0.26700000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.40100000000000002,"15min":0.41999999999999998},"min":{"1min":0.23499999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.23599999999999999}]},{"osd":1,"last update":"Sat Aug 17 17:08:38 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.42099999999999999,"15min":0.439},"min":{"1min":0.30299999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.41999999999999998,"15min":0.42799999999999999},"min":{"1min":0.29699999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.27200000000000002}]},{"osd":2,"last update":"Sat Aug 17 17:08:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.44500000000000001,"15min":0.45500000000000002},"min":{"1min":0.32800000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.307},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41899999999999998,"15min":0.441},"min":{"1min":0.30299999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.54500000000000004,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.379}]},{"osd":4,"last update":"Sat Aug 17 17:07:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.57199999999999995,"15min":0.56799999999999995},"min":{"1min":0.36599999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58199999999999996,"15min":0.57399999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.71899999999999997}]},{"osd":5,"last update":"Sat Aug 17 17:08:16 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57899999999999996,"15min":0.58699999999999997},"min":{"1min":0.41199999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.80800000000000005,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":3.4159999999999999},{"interface":"front","average":{"1min":1.5089999999999999,"5min":0.76700000000000002,"15min":0.65400000000000003},"min":{"1min":0.42599999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":18.376000000000001,"5min":18.376000000000001,"15min":18.376000000000001},"last":0.51900000000000002}]},{"osd":6,"last update":"Sat Aug 17 17:08:38 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.59799999999999998,"15min":0.58399999999999996},"min":{"1min":0.44,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.81200000000000006,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":3.3969999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.58499999999999996,"15min":0.57999999999999996},"min":{"1min":0.38900000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.80100000000000005,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.622}]},{"osd":7,"last update":"Sat Aug 17 17:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.60399999999999998,"15min":0.60199999999999998},"min":{"1min":0.41899999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.73699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":3.387},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.629,"15min":0.625},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.82999999999999996,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":3.4420000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298100,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447052,"statfs":{"total":95995035648,"available":95689781248,"internally_reserved":0,"allocated":761856,"data_stored":333328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.39200000000000002,"15min":0.39400000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.65900000000000003,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.20200000000000001},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.39600000000000002,"15min":0.39300000000000002},"min":{"1min":0.27300000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.56200000000000006,"5min":0.876,"15min":0.876},"last":0.23899999999999999}]},{"osd":1,"last update":"Sat Aug 17 17:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.40000000000000002,"15min":0.40400000000000003},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.51600000000000001,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.433,"15min":0.44600000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.34300000000000003}]},{"osd":3,"last update":"Sat Aug 17 17:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.46600000000000003,"15min":0.49299999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.44500000000000001,"15min":0.46899999999999997},"min":{"1min":0.27400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.30299999999999999}]},{"osd":4,"last update":"Sat Aug 17 17:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.34999999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57699999999999996,"15min":0.57899999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.627}]},{"osd":5,"last update":"Sat Aug 17 17:08:15 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.58199999999999996,"15min":0.57899999999999996},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.85299999999999998,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.55900000000000005,"15min":0.55700000000000005},"min":{"1min":0.39700000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.68100000000000005}]},{"osd":6,"last update":"Sat Aug 17 17:07:38 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.56100000000000005,"15min":0.55700000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59099999999999997,"15min":0.58899999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.51700000000000002}]},{"osd":7,"last update":"Sat Aug 17 17:07:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":0.754,"15min":0.65000000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":14.343,"5min":14.343,"15min":14.343},"last":0.497},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.58599999999999997,"15min":0.57799999999999996},"min":{"1min":0.44700000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.70499999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298096,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447056,"statfs":{"total":95995035648,"available":95689785344,"internally_reserved":0,"allocated":757760,"data_stored":332743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:07:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.377,"15min":0.33600000000000002},"min":{"1min":0.23799999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.60799999999999998,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.36799999999999999,"15min":0.33500000000000002},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.52400000000000002,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.46899999999999997}]},{"osd":2,"last update":"Sat Aug 17 17:08:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.38900000000000001,"15min":0.36599999999999999},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.64600000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.376,"15min":0.35899999999999999},"min":{"1min":0.27600000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.52100000000000002,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.373}]},{"osd":3,"last update":"Sat Aug 17 17:08:27 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.45900000000000002,"15min":0.47299999999999998},"min":{"1min":0.28899999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.60699999999999998,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.44500000000000001,"15min":0.46300000000000002},"min":{"1min":0.30099999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.57499999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.40799999999999997}]},{"osd":4,"last update":"Sat Aug 17 17:07:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59899999999999998,"15min":0.60099999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.80500000000000005,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.55400000000000005}]},{"osd":5,"last update":"Sat Aug 17 17:08:12 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61899999999999999,"15min":0.63400000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.748,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.84499999999999997,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.53900000000000003}]},{"osd":6,"last update":"Sat Aug 17 17:08:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.63500000000000001,"15min":0.65500000000000003},"min":{"1min":0.39100000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.60999999999999999,"15min":0.61799999999999999},"min":{"1min":0.40100000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.58099999999999996}]},{"osd":7,"last update":"Sat Aug 17 17:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.65100000000000002,"15min":0.65500000000000003},"min":{"1min":0.42599999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.502},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.64000000000000001,"15min":0.64500000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.59199999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298680,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446472,"statfs":{"total":95995035648,"available":95689187328,"internally_reserved":0,"allocated":1355776,"data_stored":923684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 17 17:07:43 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.34999999999999998,"15min":0.30399999999999999},"min":{"1min":0.19800000000000001,"5min":0.192,"15min":0.192},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.35299999999999998,"15min":0.313},"min":{"1min":0.22900000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.46600000000000003,"5min":0.628,"15min":0.628},"last":0.435}]},{"osd":2,"last update":"Sat Aug 17 17:08:05 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.378,"15min":0.35999999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.56499999999999995,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.375},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.39900000000000002,"15min":0.39600000000000002},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.50600000000000001,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.47799999999999998}]},{"osd":3,"last update":"Sat Aug 17 17:08:30 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.42499999999999999,"15min":0.438},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.60899999999999999,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.45600000000000002,"15min":0.48499999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.58899999999999997,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.49299999999999999}]},{"osd":4,"last update":"Sat Aug 17 17:07:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.58499999999999996,"15min":0.59299999999999997},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.76700000000000002,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58699999999999997,"15min":0.59699999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.82899999999999996,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.61599999999999999}]},{"osd":5,"last update":"Sat Aug 17 17:08:13 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.58599999999999997,"15min":0.58999999999999997},"min":{"1min":0.34799999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.622,"5min":0.59799999999999998,"15min":0.60699999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.66100000000000003}]},{"osd":6,"last update":"Sat Aug 17 17:08:36 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.61899999999999999,"15min":0.61399999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.85299999999999998,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.70199999999999996}]},{"osd":7,"last update":"Sat Aug 17 17:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.41099999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.751},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.77500000000000002,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.68999999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:08:46.672 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:08:46.716 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:08:46 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:08:46] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-08-17T17:08:46.950 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:08:47.114 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:46 smithi003 ceph-mon[41530]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:47.114 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:46 smithi003 ceph-mon[41530]: from='client.24695 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:08:47.115 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:46 smithi003 ceph-mon[30803]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:47.115 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:46 smithi003 ceph-mon[30803]: from='client.24695 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:08:47.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:46 smithi053 ceph-mon[34699]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:47.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:46 smithi053 ceph-mon[34699]: from='client.24695 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:08:48.799 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:08:48.800 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:08:49.128 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:48 smithi003 ceph-mon[41530]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:49.128 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:48 smithi003 ceph-mon[30803]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:49.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:48 smithi053 ceph-mon[34699]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:49.569 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2024-08-17T17:08:47.788658+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592725,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":226,"num_write_kb":2431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":170,"ondisk_log_size":170,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2386584,"kb_used_data":7720,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747574632,"statfs":{"total":767960285184,"available":765516423168,"internally_reserved":0,"allocated":7905280,"data_stored":4438791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002779"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.180946+0000","last_change":"2024-08-17T17:07:47.173693+0000","last_active":"2024-08-17T17:08:26.180946+0000","last_peered":"2024-08-17T17:08:26.180946+0000","last_clean":"2024-08-17T17:08:26.180946+0000","last_became_active":"2024-08-17T17:07:47.172508+0000","last_became_peered":"2024-08-17T17:07:47.172508+0000","last_unstale":"2024-08-17T17:08:26.180946+0000","last_undegraded":"2024-08-17T17:08:26.180946+0000","last_fullsized":"2024-08-17T17:08:26.180946+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:41:09.369848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.186059+0000","last_change":"2024-08-17T17:07:46.159359+0000","last_active":"2024-08-17T17:08:26.186059+0000","last_peered":"2024-08-17T17:08:26.186059+0000","last_clean":"2024-08-17T17:08:26.186059+0000","last_became_active":"2024-08-17T17:07:46.159068+0000","last_became_peered":"2024-08-17T17:07:46.159068+0000","last_unstale":"2024-08-17T17:08:26.186059+0000","last_undegraded":"2024-08-17T17:08:26.186059+0000","last_fullsized":"2024-08-17T17:08:26.186059+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:30.695234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.180767+0000","last_change":"2024-08-17T17:07:46.298932+0000","last_active":"2024-08-17T17:08:26.180767+0000","last_peered":"2024-08-17T17:08:26.180767+0000","last_clean":"2024-08-17T17:08:26.180767+0000","last_became_active":"2024-08-17T17:07:46.298633+0000","last_became_peered":"2024-08-17T17:07:46.298633+0000","last_unstale":"2024-08-17T17:08:26.180767+0000","last_undegraded":"2024-08-17T17:08:26.180767+0000","last_fullsized":"2024-08-17T17:08:26.180767+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:52:57.325225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.176783+0000","last_change":"2024-08-17T17:07:46.160669+0000","last_active":"2024-08-17T17:08:26.176783+0000","last_peered":"2024-08-17T17:08:26.176783+0000","last_clean":"2024-08-17T17:08:26.176783+0000","last_became_active":"2024-08-17T17:07:46.159930+0000","last_became_peered":"2024-08-17T17:07:46.159930+0000","last_unstale":"2024-08-17T17:08:26.176783+0000","last_undegraded":"2024-08-17T17:08:26.176783+0000","last_fullsized":"2024-08-17T17:08:26.176783+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:18:18.672383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.179384+0000","last_change":"2024-08-17T17:07:47.170972+0000","last_active":"2024-08-17T17:08:26.179384+0000","last_peered":"2024-08-17T17:08:26.179384+0000","last_clean":"2024-08-17T17:08:26.179384+0000","last_became_active":"2024-08-17T17:07:47.170721+0000","last_became_peered":"2024-08-17T17:07:47.170721+0000","last_unstale":"2024-08-17T17:08:26.179384+0000","last_undegraded":"2024-08-17T17:08:26.179384+0000","last_fullsized":"2024-08-17T17:08:26.179384+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:52:04.443843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.185954+0000","last_change":"2024-08-17T17:07:46.158514+0000","last_active":"2024-08-17T17:08:26.185954+0000","last_peered":"2024-08-17T17:08:26.185954+0000","last_clean":"2024-08-17T17:08:26.185954+0000","last_became_active":"2024-08-17T17:07:46.158256+0000","last_became_peered":"2024-08-17T17:07:46.158256+0000","last_unstale":"2024-08-17T17:08:26.185954+0000","last_undegraded":"2024-08-17T17:08:26.185954+0000","last_fullsized":"2024-08-17T17:08:26.185954+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:30:44.486515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.181400+0000","last_change":"2024-08-17T17:07:47.176779+0000","last_active":"2024-08-17T17:08:26.181400+0000","last_peered":"2024-08-17T17:08:26.181400+0000","last_clean":"2024-08-17T17:08:26.181400+0000","last_became_active":"2024-08-17T17:07:47.176535+0000","last_became_peered":"2024-08-17T17:07:47.176535+0000","last_unstale":"2024-08-17T17:08:26.181400+0000","last_undegraded":"2024-08-17T17:08:26.181400+0000","last_fullsized":"2024-08-17T17:08:26.181400+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:42:46.283019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":22,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.178856+0000","last_change":"2024-08-17T17:07:46.152367+0000","last_active":"2024-08-17T17:08:26.178856+0000","last_peered":"2024-08-17T17:08:26.178856+0000","last_clean":"2024-08-17T17:08:26.178856+0000","last_became_active":"2024-08-17T17:07:46.151885+0000","last_became_peered":"2024-08-17T17:07:46.151885+0000","last_unstale":"2024-08-17T17:08:26.178856+0000","last_undegraded":"2024-08-17T17:08:26.178856+0000","last_fullsized":"2024-08-17T17:08:26.178856+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:42:31.587176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.186388+0000","last_change":"2024-08-17T17:07:46.157185+0000","last_active":"2024-08-17T17:08:26.186388+0000","last_peered":"2024-08-17T17:08:26.186388+0000","last_clean":"2024-08-17T17:08:26.186388+0000","last_became_active":"2024-08-17T17:07:46.156901+0000","last_became_peered":"2024-08-17T17:07:46.156901+0000","last_unstale":"2024-08-17T17:08:26.186388+0000","last_undegraded":"2024-08-17T17:08:26.186388+0000","last_fullsized":"2024-08-17T17:08:26.186388+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:59.650169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.180568+0000","last_change":"2024-08-17T17:07:46.299427+0000","last_active":"2024-08-17T17:08:26.180568+0000","last_peered":"2024-08-17T17:08:26.180568+0000","last_clean":"2024-08-17T17:08:26.180568+0000","last_became_active":"2024-08-17T17:07:46.299008+0000","last_became_peered":"2024-08-17T17:07:46.299008+0000","last_unstale":"2024-08-17T17:08:26.180568+0000","last_undegraded":"2024-08-17T17:08:26.180568+0000","last_fullsized":"2024-08-17T17:08:26.180568+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:23:32.024874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.186292+0000","last_change":"2024-08-17T17:07:46.159280+0000","last_active":"2024-08-17T17:08:26.186292+0000","last_peered":"2024-08-17T17:08:26.186292+0000","last_clean":"2024-08-17T17:08:26.186292+0000","last_became_active":"2024-08-17T17:07:46.158988+0000","last_became_peered":"2024-08-17T17:07:46.158988+0000","last_unstale":"2024-08-17T17:08:26.186292+0000","last_undegraded":"2024-08-17T17:08:26.186292+0000","last_fullsized":"2024-08-17T17:08:26.186292+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:15:42.212909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.175879+0000","last_change":"2024-08-17T17:07:47.174653+0000","last_active":"2024-08-17T17:08:26.175879+0000","last_peered":"2024-08-17T17:08:26.175879+0000","last_clean":"2024-08-17T17:08:26.175879+0000","last_became_active":"2024-08-17T17:07:47.174034+0000","last_became_peered":"2024-08-17T17:07:47.174034+0000","last_unstale":"2024-08-17T17:08:26.175879+0000","last_undegraded":"2024-08-17T17:08:26.175879+0000","last_fullsized":"2024-08-17T17:08:26.175879+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:44:53.039509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.186283+0000","last_change":"2024-08-17T17:07:46.153923+0000","last_active":"2024-08-17T17:08:26.186283+0000","last_peered":"2024-08-17T17:08:26.186283+0000","last_clean":"2024-08-17T17:08:26.186283+0000","last_became_active":"2024-08-17T17:07:46.153614+0000","last_became_peered":"2024-08-17T17:07:46.153614+0000","last_unstale":"2024-08-17T17:08:26.186283+0000","last_undegraded":"2024-08-17T17:08:26.186283+0000","last_fullsized":"2024-08-17T17:08:26.186283+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:24:38.871907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.178145+0000","last_change":"2024-08-17T17:07:47.168592+0000","last_active":"2024-08-17T17:08:26.178145+0000","last_peered":"2024-08-17T17:08:26.178145+0000","last_clean":"2024-08-17T17:08:26.178145+0000","last_became_active":"2024-08-17T17:07:47.168264+0000","last_became_peered":"2024-08-17T17:07:47.168264+0000","last_unstale":"2024-08-17T17:08:26.178145+0000","last_undegraded":"2024-08-17T17:08:26.178145+0000","last_fullsized":"2024-08-17T17:08:26.178145+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:58:07.261379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.180803+0000","last_change":"2024-08-17T17:07:46.147836+0000","last_active":"2024-08-17T17:08:26.180803+0000","last_peered":"2024-08-17T17:08:26.180803+0000","last_clean":"2024-08-17T17:08:26.180803+0000","last_became_active":"2024-08-17T17:07:46.147559+0000","last_became_peered":"2024-08-17T17:07:46.147559+0000","last_unstale":"2024-08-17T17:08:26.180803+0000","last_undegraded":"2024-08-17T17:08:26.180803+0000","last_fullsized":"2024-08-17T17:08:26.180803+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:40:58.626980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.177832+0000","last_change":"2024-08-17T17:07:47.164620+0000","last_active":"2024-08-17T17:08:26.177832+0000","last_peered":"2024-08-17T17:08:26.177832+0000","last_clean":"2024-08-17T17:08:26.177832+0000","last_became_active":"2024-08-17T17:07:47.164325+0000","last_became_peered":"2024-08-17T17:07:47.164325+0000","last_unstale":"2024-08-17T17:08:26.177832+0000","last_undegraded":"2024-08-17T17:08:26.177832+0000","last_fullsized":"2024-08-17T17:08:26.177832+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:48:09.187177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.180712+0000","last_change":"2024-08-17T17:07:46.156846+0000","last_active":"2024-08-17T17:08:26.180712+0000","last_peered":"2024-08-17T17:08:26.180712+0000","last_clean":"2024-08-17T17:08:26.180712+0000","last_became_active":"2024-08-17T17:07:46.156421+0000","last_became_peered":"2024-08-17T17:07:46.156421+0000","last_unstale":"2024-08-17T17:08:26.180712+0000","last_undegraded":"2024-08-17T17:08:26.180712+0000","last_fullsized":"2024-08-17T17:08:26.180712+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:34:13.084499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.176108+0000","last_change":"2024-08-17T17:07:47.174543+0000","last_active":"2024-08-17T17:08:26.176108+0000","last_peered":"2024-08-17T17:08:26.176108+0000","last_clean":"2024-08-17T17:08:26.176108+0000","last_became_active":"2024-08-17T17:07:47.174103+0000","last_became_peered":"2024-08-17T17:07:47.174103+0000","last_unstale":"2024-08-17T17:08:26.176108+0000","last_undegraded":"2024-08-17T17:08:26.176108+0000","last_fullsized":"2024-08-17T17:08:26.176108+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:50:30.499357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.178293+0000","last_change":"2024-08-17T17:07:46.153140+0000","last_active":"2024-08-17T17:08:26.178293+0000","last_peered":"2024-08-17T17:08:26.178293+0000","last_clean":"2024-08-17T17:08:26.178293+0000","last_became_active":"2024-08-17T17:07:46.152695+0000","last_became_peered":"2024-08-17T17:07:46.152695+0000","last_unstale":"2024-08-17T17:08:26.178293+0000","last_undegraded":"2024-08-17T17:08:26.178293+0000","last_fullsized":"2024-08-17T17:08:26.178293+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:11:32.945713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.176250+0000","last_change":"2024-08-17T17:07:47.172506+0000","last_active":"2024-08-17T17:08:26.176250+0000","last_peered":"2024-08-17T17:08:26.176250+0000","last_clean":"2024-08-17T17:08:26.176250+0000","last_became_active":"2024-08-17T17:07:47.172164+0000","last_became_peered":"2024-08-17T17:07:47.172164+0000","last_unstale":"2024-08-17T17:08:26.176250+0000","last_undegraded":"2024-08-17T17:08:26.176250+0000","last_fullsized":"2024-08-17T17:08:26.176250+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:41:05.091599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.175994+0000","last_change":"2024-08-17T17:07:46.160296+0000","last_active":"2024-08-17T17:08:26.175994+0000","last_peered":"2024-08-17T17:08:26.175994+0000","last_clean":"2024-08-17T17:08:26.175994+0000","last_became_active":"2024-08-17T17:07:46.160031+0000","last_became_peered":"2024-08-17T17:07:46.160031+0000","last_unstale":"2024-08-17T17:08:26.175994+0000","last_undegraded":"2024-08-17T17:08:26.175994+0000","last_fullsized":"2024-08-17T17:08:26.175994+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:34:21.275585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.176386+0000","last_change":"2024-08-17T17:07:47.172575+0000","last_active":"2024-08-17T17:08:26.176386+0000","last_peered":"2024-08-17T17:08:26.176386+0000","last_clean":"2024-08-17T17:08:26.176386+0000","last_became_active":"2024-08-17T17:07:47.172117+0000","last_became_peered":"2024-08-17T17:07:47.172117+0000","last_unstale":"2024-08-17T17:08:26.176386+0000","last_undegraded":"2024-08-17T17:08:26.176386+0000","last_fullsized":"2024-08-17T17:08:26.176386+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:21:26.992513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.181714+0000","last_change":"2024-08-17T17:07:46.149804+0000","last_active":"2024-08-17T17:08:26.181714+0000","last_peered":"2024-08-17T17:08:26.181714+0000","last_clean":"2024-08-17T17:08:26.181714+0000","last_became_active":"2024-08-17T17:07:46.148863+0000","last_became_peered":"2024-08-17T17:07:46.148863+0000","last_unstale":"2024-08-17T17:08:26.181714+0000","last_undegraded":"2024-08-17T17:08:26.181714+0000","last_fullsized":"2024-08-17T17:08:26.181714+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:54.599139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.183189+0000","last_change":"2024-08-17T17:07:46.167064+0000","last_active":"2024-08-17T17:08:26.183189+0000","last_peered":"2024-08-17T17:08:26.183189+0000","last_clean":"2024-08-17T17:08:26.183189+0000","last_became_active":"2024-08-17T17:07:46.166758+0000","last_became_peered":"2024-08-17T17:07:46.166758+0000","last_unstale":"2024-08-17T17:08:26.183189+0000","last_undegraded":"2024-08-17T17:08:26.183189+0000","last_fullsized":"2024-08-17T17:08:26.183189+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:22:13.086529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"59'128","reported_seq":281,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.185928+0000","last_change":"2024-08-17T17:05:57.181832+0000","last_active":"2024-08-17T17:08:26.185928+0000","last_peered":"2024-08-17T17:08:26.185928+0000","last_clean":"2024-08-17T17:08:26.185928+0000","last_became_active":"2024-08-17T17:05:57.181461+0000","last_became_peered":"2024-08-17T17:05:57.181461+0000","last_unstale":"2024-08-17T17:08:26.185928+0000","last_undegraded":"2024-08-17T17:08:26.185928+0000","last_fullsized":"2024-08-17T17:08:26.185928+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:04:01.141401+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:10:53.855422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"63'1","reported_seq":27,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.180694+0000","last_change":"2024-08-17T17:06:34.534038+0000","last_active":"2024-08-17T17:08:26.180694+0000","last_peered":"2024-08-17T17:08:26.180694+0000","last_clean":"2024-08-17T17:08:26.180694+0000","last_became_active":"2024-08-17T17:06:29.779691+0000","last_became_peered":"2024-08-17T17:06:29.779691+0000","last_unstale":"2024-08-17T17:08:26.180694+0000","last_undegraded":"2024-08-17T17:08:26.180694+0000","last_fullsized":"2024-08-17T17:08:26.180694+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:51:04.192689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032951500000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.180948+0000","last_change":"2024-08-17T17:07:46.154584+0000","last_active":"2024-08-17T17:08:26.180948+0000","last_peered":"2024-08-17T17:08:26.180948+0000","last_clean":"2024-08-17T17:08:26.180948+0000","last_became_active":"2024-08-17T17:07:46.154296+0000","last_became_peered":"2024-08-17T17:07:46.154296+0000","last_unstale":"2024-08-17T17:08:26.180948+0000","last_undegraded":"2024-08-17T17:08:26.180948+0000","last_fullsized":"2024-08-17T17:08:26.180948+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:02:23.006828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.178431+0000","last_change":"2024-08-17T17:07:47.164375+0000","last_active":"2024-08-17T17:08:26.178431+0000","last_peered":"2024-08-17T17:08:26.178431+0000","last_clean":"2024-08-17T17:08:26.178431+0000","last_became_active":"2024-08-17T17:07:47.164077+0000","last_became_peered":"2024-08-17T17:07:47.164077+0000","last_unstale":"2024-08-17T17:08:26.178431+0000","last_undegraded":"2024-08-17T17:08:26.178431+0000","last_fullsized":"2024-08-17T17:08:26.178431+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:26:01.278743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":26,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.185860+0000","last_change":"2024-08-17T17:06:34.999031+0000","last_active":"2024-08-17T17:08:26.185860+0000","last_peered":"2024-08-17T17:08:26.185860+0000","last_clean":"2024-08-17T17:08:26.185860+0000","last_became_active":"2024-08-17T17:06:29.785264+0000","last_became_peered":"2024-08-17T17:06:29.785264+0000","last_unstale":"2024-08-17T17:08:26.185860+0000","last_undegraded":"2024-08-17T17:08:26.185860+0000","last_fullsized":"2024-08-17T17:08:26.185860+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:59:39.460037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026064700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.177230+0000","last_change":"2024-08-17T17:07:46.162672+0000","last_active":"2024-08-17T17:08:26.177230+0000","last_peered":"2024-08-17T17:08:26.177230+0000","last_clean":"2024-08-17T17:08:26.177230+0000","last_became_active":"2024-08-17T17:07:46.162058+0000","last_became_peered":"2024-08-17T17:07:46.162058+0000","last_unstale":"2024-08-17T17:08:26.177230+0000","last_undegraded":"2024-08-17T17:08:26.177230+0000","last_fullsized":"2024-08-17T17:08:26.177230+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:35:33.825034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-08-17T17:08:27.336966+0000","last_change":"2024-08-17T17:08:27.336966+0000","last_active":"2024-08-17T17:08:27.336966+0000","last_peered":"2024-08-17T17:08:27.336966+0000","last_clean":"2024-08-17T17:08:27.336966+0000","last_became_active":"2024-08-17T17:08:27.336581+0000","last_became_peered":"2024-08-17T17:08:27.336581+0000","last_unstale":"2024-08-17T17:08:27.336966+0000","last_undegraded":"2024-08-17T17:08:27.336966+0000","last_fullsized":"2024-08-17T17:08:27.336966+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:23:51.597327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.177255+0000","last_change":"2024-08-17T17:06:34.600672+0000","last_active":"2024-08-17T17:08:26.177255+0000","last_peered":"2024-08-17T17:08:26.177255+0000","last_clean":"2024-08-17T17:08:26.177255+0000","last_became_active":"2024-08-17T17:06:29.830105+0000","last_became_peered":"2024-08-17T17:06:29.830105+0000","last_unstale":"2024-08-17T17:08:26.177255+0000","last_undegraded":"2024-08-17T17:08:26.177255+0000","last_fullsized":"2024-08-17T17:08:26.177255+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:42:07.154002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000234334,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.180729+0000","last_change":"2024-08-17T17:07:46.148883+0000","last_active":"2024-08-17T17:08:26.180729+0000","last_peered":"2024-08-17T17:08:26.180729+0000","last_clean":"2024-08-17T17:08:26.180729+0000","last_became_active":"2024-08-17T17:07:46.148609+0000","last_became_peered":"2024-08-17T17:07:46.148609+0000","last_unstale":"2024-08-17T17:08:26.180729+0000","last_undegraded":"2024-08-17T17:08:26.180729+0000","last_fullsized":"2024-08-17T17:08:26.180729+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:03:35.797335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.183225+0000","last_change":"2024-08-17T17:07:46.300160+0000","last_active":"2024-08-17T17:08:26.183225+0000","last_peered":"2024-08-17T17:08:26.183225+0000","last_clean":"2024-08-17T17:08:26.183225+0000","last_became_active":"2024-08-17T17:07:46.299500+0000","last_became_peered":"2024-08-17T17:07:46.299500+0000","last_unstale":"2024-08-17T17:08:26.183225+0000","last_undegraded":"2024-08-17T17:08:26.183225+0000","last_fullsized":"2024-08-17T17:08:26.183225+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:56:50.286131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"65'2","reported_seq":28,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.180691+0000","last_change":"2024-08-17T17:06:34.534595+0000","last_active":"2024-08-17T17:08:26.180691+0000","last_peered":"2024-08-17T17:08:26.180691+0000","last_clean":"2024-08-17T17:08:26.180691+0000","last_became_active":"2024-08-17T17:06:29.777933+0000","last_became_peered":"2024-08-17T17:06:29.777933+0000","last_unstale":"2024-08-17T17:08:26.180691+0000","last_undegraded":"2024-08-17T17:08:26.180691+0000","last_fullsized":"2024-08-17T17:08:26.180691+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:33:08.360328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087151500000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.176503+0000","last_change":"2024-08-17T17:07:46.162284+0000","last_active":"2024-08-17T17:08:26.176503+0000","last_peered":"2024-08-17T17:08:26.176503+0000","last_clean":"2024-08-17T17:08:26.176503+0000","last_became_active":"2024-08-17T17:07:46.162009+0000","last_became_peered":"2024-08-17T17:07:46.162009+0000","last_unstale":"2024-08-17T17:08:26.176503+0000","last_undegraded":"2024-08-17T17:08:26.176503+0000","last_fullsized":"2024-08-17T17:08:26.176503+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:35:15.813252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.182788+0000","last_change":"2024-08-17T17:07:46.300897+0000","last_active":"2024-08-17T17:08:26.182788+0000","last_peered":"2024-08-17T17:08:26.182788+0000","last_clean":"2024-08-17T17:08:26.182788+0000","last_became_active":"2024-08-17T17:07:46.300558+0000","last_became_peered":"2024-08-17T17:07:46.300558+0000","last_unstale":"2024-08-17T17:08:26.182788+0000","last_undegraded":"2024-08-17T17:08:26.182788+0000","last_fullsized":"2024-08-17T17:08:26.182788+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:15:37.664039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.180939+0000","last_change":"2024-08-17T17:07:46.150254+0000","last_active":"2024-08-17T17:08:26.180939+0000","last_peered":"2024-08-17T17:08:26.180939+0000","last_clean":"2024-08-17T17:08:26.180939+0000","last_became_active":"2024-08-17T17:07:46.149875+0000","last_became_peered":"2024-08-17T17:07:46.149875+0000","last_unstale":"2024-08-17T17:08:26.180939+0000","last_undegraded":"2024-08-17T17:08:26.180939+0000","last_fullsized":"2024-08-17T17:08:26.180939+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:30:50.355502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.182950+0000","last_change":"2024-08-17T17:06:34.530210+0000","last_active":"2024-08-17T17:08:26.182950+0000","last_peered":"2024-08-17T17:08:26.182950+0000","last_clean":"2024-08-17T17:08:26.182950+0000","last_became_active":"2024-08-17T17:06:29.779126+0000","last_became_peered":"2024-08-17T17:06:29.779126+0000","last_unstale":"2024-08-17T17:08:26.182950+0000","last_undegraded":"2024-08-17T17:08:26.182950+0000","last_fullsized":"2024-08-17T17:08:26.182950+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:53:27.601004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028588399999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.181750+0000","last_change":"2024-08-17T17:07:46.166149+0000","last_active":"2024-08-17T17:08:26.181750+0000","last_peered":"2024-08-17T17:08:26.181750+0000","last_clean":"2024-08-17T17:08:26.181750+0000","last_became_active":"2024-08-17T17:07:46.165844+0000","last_became_peered":"2024-08-17T17:07:46.165844+0000","last_unstale":"2024-08-17T17:08:26.181750+0000","last_undegraded":"2024-08-17T17:08:26.181750+0000","last_fullsized":"2024-08-17T17:08:26.181750+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:28:13.437271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.182882+0000","last_change":"2024-08-17T17:07:46.149720+0000","last_active":"2024-08-17T17:08:26.182882+0000","last_peered":"2024-08-17T17:08:26.182882+0000","last_clean":"2024-08-17T17:08:26.182882+0000","last_became_active":"2024-08-17T17:07:46.149023+0000","last_became_peered":"2024-08-17T17:07:46.149023+0000","last_unstale":"2024-08-17T17:08:26.182882+0000","last_undegraded":"2024-08-17T17:08:26.182882+0000","last_fullsized":"2024-08-17T17:08:26.182882+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:19:20.278134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":26,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.186310+0000","last_change":"2024-08-17T17:06:34.999020+0000","last_active":"2024-08-17T17:08:26.186310+0000","last_peered":"2024-08-17T17:08:26.186310+0000","last_clean":"2024-08-17T17:08:26.186310+0000","last_became_active":"2024-08-17T17:06:29.830625+0000","last_became_peered":"2024-08-17T17:06:29.830625+0000","last_unstale":"2024-08-17T17:08:26.186310+0000","last_undegraded":"2024-08-17T17:08:26.186310+0000","last_fullsized":"2024-08-17T17:08:26.186310+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:23:50.149657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000290981,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.177924+0000","last_change":"2024-08-17T17:07:47.227288+0000","last_active":"2024-08-17T17:08:26.177924+0000","last_peered":"2024-08-17T17:08:26.177924+0000","last_clean":"2024-08-17T17:08:26.177924+0000","last_became_active":"2024-08-17T17:07:47.226969+0000","last_became_peered":"2024-08-17T17:07:47.226969+0000","last_unstale":"2024-08-17T17:08:26.177924+0000","last_undegraded":"2024-08-17T17:08:26.177924+0000","last_fullsized":"2024-08-17T17:08:26.177924+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:28:51.453724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.181473+0000","last_change":"2024-08-17T17:07:46.157541+0000","last_active":"2024-08-17T17:08:26.181473+0000","last_peered":"2024-08-17T17:08:26.181473+0000","last_clean":"2024-08-17T17:08:26.181473+0000","last_became_active":"2024-08-17T17:07:46.157288+0000","last_became_peered":"2024-08-17T17:07:46.157288+0000","last_unstale":"2024-08-17T17:08:26.181473+0000","last_undegraded":"2024-08-17T17:08:26.181473+0000","last_fullsized":"2024-08-17T17:08:26.181473+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:18:42.512643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":26,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.183084+0000","last_change":"2024-08-17T17:06:34.530258+0000","last_active":"2024-08-17T17:08:26.183084+0000","last_peered":"2024-08-17T17:08:26.183084+0000","last_clean":"2024-08-17T17:08:26.183084+0000","last_became_active":"2024-08-17T17:06:29.831452+0000","last_became_peered":"2024-08-17T17:06:29.831452+0000","last_unstale":"2024-08-17T17:08:26.183084+0000","last_undegraded":"2024-08-17T17:08:26.183084+0000","last_fullsized":"2024-08-17T17:08:26.183084+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:36:53.341363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027697000000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.181564+0000","last_change":"2024-08-17T17:07:47.172149+0000","last_active":"2024-08-17T17:08:26.181564+0000","last_peered":"2024-08-17T17:08:26.181564+0000","last_clean":"2024-08-17T17:08:26.181564+0000","last_became_active":"2024-08-17T17:07:47.171895+0000","last_became_peered":"2024-08-17T17:07:47.171895+0000","last_unstale":"2024-08-17T17:08:26.181564+0000","last_undegraded":"2024-08-17T17:08:26.181564+0000","last_fullsized":"2024-08-17T17:08:26.181564+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:33:59.514052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.179705+0000","last_change":"2024-08-17T17:07:46.160364+0000","last_active":"2024-08-17T17:08:26.179705+0000","last_peered":"2024-08-17T17:08:26.179705+0000","last_clean":"2024-08-17T17:08:26.179705+0000","last_became_active":"2024-08-17T17:07:46.160077+0000","last_became_peered":"2024-08-17T17:07:46.160077+0000","last_unstale":"2024-08-17T17:08:26.179705+0000","last_undegraded":"2024-08-17T17:08:26.179705+0000","last_fullsized":"2024-08-17T17:08:26.179705+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:46:56.843721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":26,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.178554+0000","last_change":"2024-08-17T17:06:34.535627+0000","last_active":"2024-08-17T17:08:26.178554+0000","last_peered":"2024-08-17T17:08:26.178554+0000","last_clean":"2024-08-17T17:08:26.178554+0000","last_became_active":"2024-08-17T17:06:29.779504+0000","last_became_peered":"2024-08-17T17:06:29.779504+0000","last_unstale":"2024-08-17T17:08:26.178554+0000","last_undegraded":"2024-08-17T17:08:26.178554+0000","last_fullsized":"2024-08-17T17:08:26.178554+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:21:33.158491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000362901,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.175789+0000","last_change":"2024-08-17T17:07:47.227287+0000","last_active":"2024-08-17T17:08:26.175789+0000","last_peered":"2024-08-17T17:08:26.175789+0000","last_clean":"2024-08-17T17:08:26.175789+0000","last_became_active":"2024-08-17T17:07:47.226983+0000","last_became_peered":"2024-08-17T17:07:47.226983+0000","last_unstale":"2024-08-17T17:08:26.175789+0000","last_undegraded":"2024-08-17T17:08:26.175789+0000","last_fullsized":"2024-08-17T17:08:26.175789+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:34:07.444142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.176190+0000","last_change":"2024-08-17T17:07:47.172582+0000","last_active":"2024-08-17T17:08:26.176190+0000","last_peered":"2024-08-17T17:08:26.176190+0000","last_clean":"2024-08-17T17:08:26.176190+0000","last_became_active":"2024-08-17T17:07:47.172267+0000","last_became_peered":"2024-08-17T17:07:47.172267+0000","last_unstale":"2024-08-17T17:08:26.176190+0000","last_undegraded":"2024-08-17T17:08:26.176190+0000","last_fullsized":"2024-08-17T17:08:26.176190+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:21:00.234634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"71'6","reported_seq":20,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.176239+0000","last_change":"2024-08-17T17:07:46.160694+0000","last_active":"2024-08-17T17:08:26.176239+0000","last_peered":"2024-08-17T17:08:26.176239+0000","last_clean":"2024-08-17T17:08:26.176239+0000","last_became_active":"2024-08-17T17:07:46.160407+0000","last_became_peered":"2024-08-17T17:07:46.160407+0000","last_unstale":"2024-08-17T17:08:26.176239+0000","last_undegraded":"2024-08-17T17:08:26.176239+0000","last_fullsized":"2024-08-17T17:08:26.176239+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:54:38.366094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1552,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.181892+0000","last_change":"2024-08-17T17:07:46.299965+0000","last_active":"2024-08-17T17:08:26.181892+0000","last_peered":"2024-08-17T17:08:26.181892+0000","last_clean":"2024-08-17T17:08:26.181892+0000","last_became_active":"2024-08-17T17:07:46.299319+0000","last_became_peered":"2024-08-17T17:07:46.299319+0000","last_unstale":"2024-08-17T17:08:26.181892+0000","last_undegraded":"2024-08-17T17:08:26.181892+0000","last_fullsized":"2024-08-17T17:08:26.181892+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:31:25.609194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.186200+0000","last_change":"2024-08-17T17:07:46.157895+0000","last_active":"2024-08-17T17:08:26.186200+0000","last_peered":"2024-08-17T17:08:26.186200+0000","last_clean":"2024-08-17T17:08:26.186200+0000","last_became_active":"2024-08-17T17:07:46.157636+0000","last_became_peered":"2024-08-17T17:07:46.157636+0000","last_unstale":"2024-08-17T17:08:26.186200+0000","last_undegraded":"2024-08-17T17:08:26.186200+0000","last_fullsized":"2024-08-17T17:08:26.186200+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:19:21.578951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.180826+0000","last_change":"2024-08-17T17:07:46.151440+0000","last_active":"2024-08-17T17:08:26.180826+0000","last_peered":"2024-08-17T17:08:26.180826+0000","last_clean":"2024-08-17T17:08:26.180826+0000","last_became_active":"2024-08-17T17:07:46.150468+0000","last_became_peered":"2024-08-17T17:07:46.150468+0000","last_unstale":"2024-08-17T17:08:26.180826+0000","last_undegraded":"2024-08-17T17:08:26.180826+0000","last_fullsized":"2024-08-17T17:08:26.180826+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:51.933623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.180946+0000","last_change":"2024-08-17T17:07:47.171412+0000","last_active":"2024-08-17T17:08:26.180946+0000","last_peered":"2024-08-17T17:08:26.180946+0000","last_clean":"2024-08-17T17:08:26.180946+0000","last_became_active":"2024-08-17T17:07:47.171143+0000","last_became_peered":"2024-08-17T17:07:47.171143+0000","last_unstale":"2024-08-17T17:08:26.180946+0000","last_undegraded":"2024-08-17T17:08:26.180946+0000","last_fullsized":"2024-08-17T17:08:26.180946+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:53:16.355093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.179294+0000","last_change":"2024-08-17T17:07:46.161427+0000","last_active":"2024-08-17T17:08:26.179294+0000","last_peered":"2024-08-17T17:08:26.179294+0000","last_clean":"2024-08-17T17:08:26.179294+0000","last_became_active":"2024-08-17T17:07:46.161140+0000","last_became_peered":"2024-08-17T17:07:46.161140+0000","last_unstale":"2024-08-17T17:08:26.179294+0000","last_undegraded":"2024-08-17T17:08:26.179294+0000","last_fullsized":"2024-08-17T17:08:26.179294+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:15.549311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.181489+0000","last_change":"2024-08-17T17:07:47.172334+0000","last_active":"2024-08-17T17:08:26.181489+0000","last_peered":"2024-08-17T17:08:26.181489+0000","last_clean":"2024-08-17T17:08:26.181489+0000","last_became_active":"2024-08-17T17:07:47.172071+0000","last_became_peered":"2024-08-17T17:07:47.172071+0000","last_unstale":"2024-08-17T17:08:26.181489+0000","last_undegraded":"2024-08-17T17:08:26.181489+0000","last_fullsized":"2024-08-17T17:08:26.181489+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:10:45.779777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.178548+0000","last_change":"2024-08-17T17:07:47.227351+0000","last_active":"2024-08-17T17:08:26.178548+0000","last_peered":"2024-08-17T17:08:26.178548+0000","last_clean":"2024-08-17T17:08:26.178548+0000","last_became_active":"2024-08-17T17:07:47.226799+0000","last_became_peered":"2024-08-17T17:07:47.226799+0000","last_unstale":"2024-08-17T17:08:26.178548+0000","last_undegraded":"2024-08-17T17:08:26.178548+0000","last_fullsized":"2024-08-17T17:08:26.178548+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:32:03.857904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.181638+0000","last_change":"2024-08-17T17:07:46.156840+0000","last_active":"2024-08-17T17:08:26.181638+0000","last_peered":"2024-08-17T17:08:26.181638+0000","last_clean":"2024-08-17T17:08:26.181638+0000","last_became_active":"2024-08-17T17:07:46.156566+0000","last_became_peered":"2024-08-17T17:07:46.156566+0000","last_unstale":"2024-08-17T17:08:26.181638+0000","last_undegraded":"2024-08-17T17:08:26.181638+0000","last_fullsized":"2024-08-17T17:08:26.181638+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:53:53.403475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.176730+0000","last_change":"2024-08-17T17:07:47.170033+0000","last_active":"2024-08-17T17:08:26.176730+0000","last_peered":"2024-08-17T17:08:26.176730+0000","last_clean":"2024-08-17T17:08:26.176730+0000","last_became_active":"2024-08-17T17:07:47.169729+0000","last_became_peered":"2024-08-17T17:07:47.169729+0000","last_unstale":"2024-08-17T17:08:26.176730+0000","last_undegraded":"2024-08-17T17:08:26.176730+0000","last_fullsized":"2024-08-17T17:08:26.176730+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:22:24.637462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.181885+0000","last_change":"2024-08-17T17:07:46.160245+0000","last_active":"2024-08-17T17:08:26.181885+0000","last_peered":"2024-08-17T17:08:26.181885+0000","last_clean":"2024-08-17T17:08:26.181885+0000","last_became_active":"2024-08-17T17:07:46.159973+0000","last_became_peered":"2024-08-17T17:07:46.159973+0000","last_unstale":"2024-08-17T17:08:26.181885+0000","last_undegraded":"2024-08-17T17:08:26.181885+0000","last_fullsized":"2024-08-17T17:08:26.181885+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:01:25.166219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":23,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-08-17T17:08:27.337450+0000","last_change":"2024-08-17T17:08:27.337450+0000","last_active":"2024-08-17T17:08:27.337450+0000","last_peered":"2024-08-17T17:08:27.337450+0000","last_clean":"2024-08-17T17:08:27.337450+0000","last_became_active":"2024-08-17T17:08:27.337024+0000","last_became_peered":"2024-08-17T17:08:27.337024+0000","last_unstale":"2024-08-17T17:08:27.337450+0000","last_undegraded":"2024-08-17T17:08:27.337450+0000","last_fullsized":"2024-08-17T17:08:27.337450+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:09:29.610727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.176058+0000","last_change":"2024-08-17T17:07:46.158700+0000","last_active":"2024-08-17T17:08:26.176058+0000","last_peered":"2024-08-17T17:08:26.176058+0000","last_clean":"2024-08-17T17:08:26.176058+0000","last_became_active":"2024-08-17T17:07:46.158397+0000","last_became_peered":"2024-08-17T17:07:46.158397+0000","last_unstale":"2024-08-17T17:08:26.176058+0000","last_undegraded":"2024-08-17T17:08:26.176058+0000","last_fullsized":"2024-08-17T17:08:26.176058+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:23:03.270667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.177461+0000","last_change":"2024-08-17T17:07:47.174396+0000","last_active":"2024-08-17T17:08:26.177461+0000","last_peered":"2024-08-17T17:08:26.177461+0000","last_clean":"2024-08-17T17:08:26.177461+0000","last_became_active":"2024-08-17T17:07:47.174025+0000","last_became_peered":"2024-08-17T17:07:47.174025+0000","last_unstale":"2024-08-17T17:08:26.177461+0000","last_undegraded":"2024-08-17T17:08:26.177461+0000","last_fullsized":"2024-08-17T17:08:26.177461+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:33:09.032654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.178091+0000","last_change":"2024-08-17T17:07:46.160470+0000","last_active":"2024-08-17T17:08:26.178091+0000","last_peered":"2024-08-17T17:08:26.178091+0000","last_clean":"2024-08-17T17:08:26.178091+0000","last_became_active":"2024-08-17T17:07:46.160214+0000","last_became_peered":"2024-08-17T17:07:46.160214+0000","last_unstale":"2024-08-17T17:08:26.178091+0000","last_undegraded":"2024-08-17T17:08:26.178091+0000","last_fullsized":"2024-08-17T17:08:26.178091+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:25:43.560945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.176138+0000","last_change":"2024-08-17T17:07:47.174519+0000","last_active":"2024-08-17T17:08:26.176138+0000","last_peered":"2024-08-17T17:08:26.176138+0000","last_clean":"2024-08-17T17:08:26.176138+0000","last_became_active":"2024-08-17T17:07:47.174128+0000","last_became_peered":"2024-08-17T17:07:47.174128+0000","last_unstale":"2024-08-17T17:08:26.176138+0000","last_undegraded":"2024-08-17T17:08:26.176138+0000","last_fullsized":"2024-08-17T17:08:26.176138+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:27:58.327661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.180705+0000","last_change":"2024-08-17T17:07:46.154087+0000","last_active":"2024-08-17T17:08:26.180705+0000","last_peered":"2024-08-17T17:08:26.180705+0000","last_clean":"2024-08-17T17:08:26.180705+0000","last_became_active":"2024-08-17T17:07:46.153779+0000","last_became_peered":"2024-08-17T17:07:46.153779+0000","last_unstale":"2024-08-17T17:08:26.180705+0000","last_undegraded":"2024-08-17T17:08:26.180705+0000","last_fullsized":"2024-08-17T17:08:26.180705+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:20:54.861570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.175881+0000","last_change":"2024-08-17T17:07:47.174452+0000","last_active":"2024-08-17T17:08:26.175881+0000","last_peered":"2024-08-17T17:08:26.175881+0000","last_clean":"2024-08-17T17:08:26.175881+0000","last_became_active":"2024-08-17T17:07:47.174140+0000","last_became_peered":"2024-08-17T17:07:47.174140+0000","last_unstale":"2024-08-17T17:08:26.175881+0000","last_undegraded":"2024-08-17T17:08:26.175881+0000","last_fullsized":"2024-08-17T17:08:26.175881+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:13:27.133343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":16,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.180895+0000","last_change":"2024-08-17T17:07:46.156750+0000","last_active":"2024-08-17T17:08:26.180895+0000","last_peered":"2024-08-17T17:08:26.180895+0000","last_clean":"2024-08-17T17:08:26.180895+0000","last_became_active":"2024-08-17T17:07:46.156253+0000","last_became_peered":"2024-08-17T17:07:46.156253+0000","last_unstale":"2024-08-17T17:08:26.180895+0000","last_undegraded":"2024-08-17T17:08:26.180895+0000","last_fullsized":"2024-08-17T17:08:26.180895+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:04:28.549130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.181088+0000","last_change":"2024-08-17T17:07:47.173463+0000","last_active":"2024-08-17T17:08:26.181088+0000","last_peered":"2024-08-17T17:08:26.181088+0000","last_clean":"2024-08-17T17:08:26.181088+0000","last_became_active":"2024-08-17T17:07:47.173151+0000","last_became_peered":"2024-08-17T17:07:47.173151+0000","last_unstale":"2024-08-17T17:08:26.181088+0000","last_undegraded":"2024-08-17T17:08:26.181088+0000","last_fullsized":"2024-08-17T17:08:26.181088+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:33:55.409780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.181309+0000","last_change":"2024-08-17T17:07:46.155673+0000","last_active":"2024-08-17T17:08:26.181309+0000","last_peered":"2024-08-17T17:08:26.181309+0000","last_clean":"2024-08-17T17:08:26.181309+0000","last_became_active":"2024-08-17T17:07:46.154508+0000","last_became_peered":"2024-08-17T17:07:46.154508+0000","last_unstale":"2024-08-17T17:08:26.181309+0000","last_undegraded":"2024-08-17T17:08:26.181309+0000","last_fullsized":"2024-08-17T17:08:26.181309+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:55:57.999423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":23,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-08-17T17:08:27.337685+0000","last_change":"2024-08-17T17:08:27.337685+0000","last_active":"2024-08-17T17:08:27.337685+0000","last_peered":"2024-08-17T17:08:27.337685+0000","last_clean":"2024-08-17T17:08:27.337685+0000","last_became_active":"2024-08-17T17:08:27.337163+0000","last_became_peered":"2024-08-17T17:08:27.337163+0000","last_unstale":"2024-08-17T17:08:27.337685+0000","last_undegraded":"2024-08-17T17:08:27.337685+0000","last_fullsized":"2024-08-17T17:08:27.337685+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:27:11.953158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":18,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-08-17T17:08:26.179133+0000","last_change":"2024-08-17T17:07:46.156321+0000","last_active":"2024-08-17T17:08:26.179133+0000","last_peered":"2024-08-17T17:08:26.179133+0000","last_clean":"2024-08-17T17:08:26.179133+0000","last_became_active":"2024-08-17T17:07:46.156066+0000","last_became_peered":"2024-08-17T17:07:46.156066+0000","last_unstale":"2024-08-17T17:08:26.179133+0000","last_undegraded":"2024-08-17T17:08:26.179133+0000","last_fullsized":"2024-08-17T17:08:26.179133+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:52.422656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2338,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":7014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234022,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298700,"kb_used_data":1340,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446452,"statfs":{"total":95995035648,"available":95689166848,"internally_reserved":0,"allocated":1372160,"data_stored":924537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.55500000000000005,"15min":0.56200000000000006},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.72099999999999997,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.58399999999999996,"15min":0.59099999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.69799999999999995,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.47899999999999998}]},{"osd":1,"last update":"Sat Aug 17 17:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.58499999999999996,"15min":0.59399999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.503},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.59399999999999997,"15min":0.60199999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.76000000000000001,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.54600000000000004}]},{"osd":2,"last update":"Sat Aug 17 17:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.54600000000000004,"15min":0.55300000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.67800000000000005,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.59699999999999998,"15min":0.60699999999999998},"min":{"1min":0.39300000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.69699999999999995,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.65600000000000003}]},{"osd":3,"last update":"Sat Aug 17 17:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.56699999999999995,"15min":0.57199999999999995},"min":{"1min":0.41199999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.68300000000000005,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.432},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.60499999999999998,"15min":0.61099999999999999},"min":{"1min":0.38700000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.77900000000000003,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.626}]},{"osd":4,"last update":"Sat Aug 17 17:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.48699999999999999,"15min":0.49299999999999999},"min":{"1min":0.29399999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.50700000000000001,"15min":0.51400000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.74099999999999999,"5min":0.878,"15min":0.878},"last":0.60099999999999998}]},{"osd":5,"last update":"Sat Aug 17 17:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51200000000000001,"15min":0.51500000000000001},"min":{"1min":0.34499999999999997,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.81000000000000005,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.56499999999999995,"15min":0.57699999999999996},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.70199999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.58499999999999996}]},{"osd":6,"last update":"Sat Aug 17 17:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.58699999999999997,"15min":0.59699999999999998},"min":{"1min":0.372,"5min":0.316,"15min":0.316},"max":{"1min":0.747,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.54400000000000004,"15min":0.54800000000000004},"min":{"1min":0.32500000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.73699999999999999,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.64300000000000002}]}]},{"osd":6,"up_from":46,"seq":197568495657,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298688,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446464,"statfs":{"total":95995035648,"available":95689179136,"internally_reserved":0,"allocated":1359872,"data_stored":924203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.51800000000000002,"15min":0.52100000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.68600000000000005,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52500000000000002,"15min":0.52300000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.70499999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.56499999999999995}]},{"osd":1,"last update":"Sat Aug 17 17:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.54200000000000004,"15min":0.54900000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.71299999999999997,"5min":0.754,"15min":0.754},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.5,"5min":0.50900000000000001,"15min":0.505},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.434}]},{"osd":2,"last update":"Sat Aug 17 17:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.70799999999999996,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.505,"5min":0.53800000000000003,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.69099999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.64200000000000002}]},{"osd":3,"last update":"Sat Aug 17 17:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.52100000000000002,"15min":0.52600000000000002},"min":{"1min":0.33300000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.73799999999999999,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.38800000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.746,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.60199999999999998}]},{"osd":4,"last update":"Sat Aug 17 17:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.44500000000000001,"15min":0.44400000000000001},"min":{"1min":0.32000000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.66200000000000003,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.45700000000000002,"15min":0.45500000000000002},"min":{"1min":0.315,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68799999999999994,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.36199999999999999}]},{"osd":5,"last update":"Sat Aug 17 17:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.47199999999999998,"15min":0.46200000000000002},"min":{"1min":0.33200000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.745,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.47399999999999998,"15min":0.46400000000000002},"min":{"1min":0.29699999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73399999999999999,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.58399999999999996}]},{"osd":7,"last update":"Sat Aug 17 17:07:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.49199999999999999,"15min":0.48199999999999998},"min":{"1min":0.35999999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.495},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.54800000000000004,"15min":0.54200000000000004},"min":{"1min":0.38900000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.72699999999999998,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.67700000000000005}]}]},{"osd":5,"up_from":38,"seq":163208757294,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298100,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447052,"statfs":{"total":95995035648,"available":95689781248,"internally_reserved":0,"allocated":761856,"data_stored":333060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:08:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.57799999999999996,"15min":0.60399999999999998},"min":{"1min":0.34999999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.66400000000000003,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.59199999999999997,"15min":0.59399999999999997},"min":{"1min":0.36699999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.66900000000000004,"5min":3.5779999999999998,"15min":3.5779999999999998},"last":0.373}]},{"osd":1,"last update":"Sat Aug 17 17:08:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.60699999999999998,"15min":0.61699999999999999},"min":{"1min":0.42199999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.66200000000000003,"5min":3.5950000000000002,"15min":3.5950000000000002},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.55500000000000005,"15min":0.57399999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70799999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.48899999999999999}]},{"osd":2,"last update":"Sat Aug 17 17:08:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.61699999999999999,"15min":0.624},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.64900000000000002,"5min":3.6160000000000001,"15min":3.6160000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.59999999999999998,"15min":0.629},"min":{"1min":0.42299999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.63500000000000001,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.40500000000000003}]},{"osd":3,"last update":"Sat Aug 17 17:08:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.59399999999999997,"15min":0.61399999999999999},"min":{"1min":0.38500000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.70099999999999996,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.59899999999999998,"15min":0.629},"min":{"1min":0.39400000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.68000000000000005,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.442}]},{"osd":4,"last update":"Sat Aug 17 17:08:15 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.436,"15min":0.42799999999999999},"min":{"1min":0.307,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.432},"min":{"1min":0.312,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.69899999999999995,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.50800000000000001}]},{"osd":6,"last update":"Sat Aug 17 17:08:38 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.48499999999999999,"15min":0.47299999999999998},"min":{"1min":0.35799999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.76900000000000002,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.48399999999999999,"15min":0.47599999999999998},"min":{"1min":0.32600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.72899999999999998,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.53000000000000003}]},{"osd":7,"last update":"Sat Aug 17 17:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.78900000000000003,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54200000000000004,"15min":0.54600000000000004},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.74299999999999999,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.56299999999999994}]}]},{"osd":4,"up_from":30,"seq":128849018932,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298120,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447032,"statfs":{"total":95995035648,"available":95689760768,"internally_reserved":0,"allocated":778240,"data_stored":334422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.50900000000000001,"15min":0.51700000000000002},"min":{"1min":0.33600000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.66000000000000003,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.54800000000000004,"15min":0.54400000000000004},"min":{"1min":0.39900000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.81599999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.42699999999999999}]},{"osd":1,"last update":"Sat Aug 17 17:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.54200000000000004,"15min":0.55100000000000005},"min":{"1min":0.35299999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.64700000000000002,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.53900000000000003,"15min":0.53700000000000003},"min":{"1min":0.40999999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.68000000000000005,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.55100000000000005}]},{"osd":2,"last update":"Sat Aug 17 17:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.53600000000000003,"15min":0.54400000000000004},"min":{"1min":0.36599999999999999,"5min":0.313,"15min":0.313},"max":{"1min":0.71199999999999997,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.502},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.52400000000000002,"15min":0.51800000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.53000000000000003}]},{"osd":3,"last update":"Sat Aug 17 17:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55200000000000005,"15min":0.55900000000000005},"min":{"1min":0.42799999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.72799999999999998,"5min":0.752,"15min":0.752},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.41799999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.69999999999999996,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.56399999999999995}]},{"osd":5,"last update":"Sat Aug 17 17:08:11 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.40999999999999998,"15min":0.40500000000000003},"min":{"1min":0.32100000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.56499999999999995,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.44400000000000001,"15min":0.45000000000000001},"min":{"1min":0.28899999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.58299999999999996}]},{"osd":6,"last update":"Sat Aug 17 17:08:32 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.46200000000000002,"15min":0.46500000000000002},"min":{"1min":0.34000000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.55000000000000004,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.372},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.44700000000000001,"15min":0.442},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.70799999999999996,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.39600000000000002}]},{"osd":7,"last update":"Sat Aug 17 17:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51000000000000001,"15min":0.50900000000000001},"min":{"1min":0.41799999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.439},{"interface":"front","average":{"1min":0.498,"5min":0.50600000000000001,"15min":0.50800000000000001},"min":{"1min":0.33700000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.60299999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182457,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298100,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447052,"statfs":{"total":95995035648,"available":95689781248,"internally_reserved":0,"allocated":757760,"data_stored":332814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:08:38 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.38300000000000001,"15min":0.40100000000000002},"min":{"1min":0.26700000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.40100000000000002,"15min":0.41999999999999998},"min":{"1min":0.23499999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.35899999999999999}]},{"osd":1,"last update":"Sat Aug 17 17:08:38 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.42099999999999999,"15min":0.439},"min":{"1min":0.30299999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.41999999999999998,"15min":0.42799999999999999},"min":{"1min":0.29699999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.379}]},{"osd":2,"last update":"Sat Aug 17 17:08:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.44500000000000001,"15min":0.45500000000000002},"min":{"1min":0.32800000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41899999999999998,"15min":0.441},"min":{"1min":0.30299999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.54500000000000004,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.28599999999999998}]},{"osd":4,"last update":"Sat Aug 17 17:07:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.57199999999999995,"15min":0.56799999999999995},"min":{"1min":0.36599999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58199999999999996,"15min":0.57399999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.625}]},{"osd":5,"last update":"Sat Aug 17 17:08:16 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57899999999999996,"15min":0.58699999999999997},"min":{"1min":0.41199999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.80800000000000005,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.61099999999999999},{"interface":"front","average":{"1min":1.5089999999999999,"5min":0.76700000000000002,"15min":0.65400000000000003},"min":{"1min":0.42599999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":18.376000000000001,"5min":18.376000000000001,"15min":18.376000000000001},"last":0.67400000000000004}]},{"osd":6,"last update":"Sat Aug 17 17:08:38 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.59799999999999998,"15min":0.58399999999999996},"min":{"1min":0.44,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.81200000000000006,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.58499999999999996,"15min":0.57999999999999996},"min":{"1min":0.38900000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.80100000000000005,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.56799999999999995}]},{"osd":7,"last update":"Sat Aug 17 17:07:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.60399999999999998,"15min":0.60199999999999998},"min":{"1min":0.41899999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.73699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.629,"15min":0.625},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.82999999999999996,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.66500000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298100,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447052,"statfs":{"total":95995035648,"available":95689781248,"internally_reserved":0,"allocated":761856,"data_stored":333328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.39200000000000002,"15min":0.39400000000000002},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.65900000000000003,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.223},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.39600000000000002,"15min":0.39300000000000002},"min":{"1min":0.27300000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.56200000000000006,"5min":0.876,"15min":0.876},"last":0.26300000000000001}]},{"osd":1,"last update":"Sat Aug 17 17:08:06 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.40000000000000002,"15min":0.40400000000000003},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.51600000000000001,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.433,"15min":0.44600000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.28799999999999998}]},{"osd":3,"last update":"Sat Aug 17 17:08:34 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.46600000000000003,"15min":0.49299999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.44500000000000001,"15min":0.46899999999999997},"min":{"1min":0.27400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.30299999999999999}]},{"osd":4,"last update":"Sat Aug 17 17:07:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.34999999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57699999999999996,"15min":0.57899999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.54100000000000004}]},{"osd":5,"last update":"Sat Aug 17 17:08:15 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.58199999999999996,"15min":0.57899999999999996},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.85299999999999998,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.503},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.55900000000000005,"15min":0.55700000000000005},"min":{"1min":0.39700000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.48899999999999999}]},{"osd":6,"last update":"Sat Aug 17 17:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56999999999999995,"15min":0.56000000000000005},"min":{"1min":0.433,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78600000000000003,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.59599999999999997,"15min":0.58999999999999997},"min":{"1min":0.443,"5min":0.379,"15min":0.379},"max":{"1min":0.89900000000000002,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.52300000000000002}]},{"osd":7,"last update":"Sat Aug 17 17:07:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":0.754,"15min":0.65000000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":14.343,"5min":14.343,"15min":14.343},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.58599999999999997,"15min":0.57799999999999996},"min":{"1min":0.44700000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.46300000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298096,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447056,"statfs":{"total":95995035648,"available":95689785344,"internally_reserved":0,"allocated":757760,"data_stored":332743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Aug 17 17:08:46 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.39300000000000002,"15min":0.34100000000000003},"min":{"1min":0.23899999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55200000000000005,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.38300000000000001,"15min":0.34000000000000002},"min":{"1min":0.25900000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.59499999999999997,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.41199999999999998}]},{"osd":2,"last update":"Sat Aug 17 17:08:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.38900000000000001,"15min":0.36599999999999999},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.64600000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.372,"5min":0.376,"15min":0.35899999999999999},"min":{"1min":0.27600000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.52100000000000002,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.439}]},{"osd":3,"last update":"Sat Aug 17 17:08:27 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.45900000000000002,"15min":0.47299999999999998},"min":{"1min":0.28899999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.60699999999999998,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.44500000000000001,"15min":0.46300000000000002},"min":{"1min":0.30099999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.57499999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.46800000000000003}]},{"osd":4,"last update":"Sat Aug 17 17:07:47 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.59899999999999998,"15min":0.60099999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.80500000000000005,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.76400000000000001}]},{"osd":5,"last update":"Sat Aug 17 17:08:12 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61899999999999999,"15min":0.63400000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.748,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.58799999999999997,"15min":0.59399999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.84499999999999997,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.71199999999999997}]},{"osd":6,"last update":"Sat Aug 17 17:08:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.63500000000000001,"15min":0.65500000000000003},"min":{"1min":0.39100000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.60999999999999999,"15min":0.61799999999999999},"min":{"1min":0.40100000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.80900000000000005}]},{"osd":7,"last update":"Sat Aug 17 17:07:55 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.65100000000000002,"15min":0.65500000000000003},"min":{"1min":0.42599999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.64000000000000001,"15min":0.64500000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.84899999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298680,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446472,"statfs":{"total":95995035648,"available":95689187328,"internally_reserved":0,"allocated":1355776,"data_stored":923684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Aug 17 17:08:46 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.375,"15min":0.313},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":0.60999999999999999,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.379,"15min":0.32200000000000001},"min":{"1min":0.30299999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.57099999999999995,"5min":0.628,"15min":0.628},"last":0.48699999999999999}]},{"osd":2,"last update":"Sat Aug 17 17:08:05 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.378,"15min":0.35999999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.56499999999999995,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.39900000000000002,"15min":0.39600000000000002},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.50600000000000001,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.46500000000000002}]},{"osd":3,"last update":"Sat Aug 17 17:08:30 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.42499999999999999,"15min":0.438},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.60899999999999999,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.45600000000000002,"15min":0.48499999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.58899999999999997,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.50900000000000001}]},{"osd":4,"last update":"Sat Aug 17 17:07:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.58499999999999996,"15min":0.59299999999999997},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.76700000000000002,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.58699999999999997,"15min":0.59699999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.82899999999999996,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.79100000000000004}]},{"osd":5,"last update":"Sat Aug 17 17:08:13 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.58599999999999997,"15min":0.58999999999999997},"min":{"1min":0.34799999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.59799999999999998,"15min":0.60699999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.80500000000000005}]},{"osd":6,"last update":"Sat Aug 17 17:08:36 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.61899999999999999,"15min":0.61399999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.85299999999999998,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.81999999999999995}]},{"osd":7,"last update":"Sat Aug 17 17:07:58 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.41099999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.77500000000000002,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.91400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:08:49.573 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-08-17T17:08:49.573 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-08-17T17:08:49.574 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-08-17T17:08:49.574 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph health --format=json 2024-08-17T17:08:49.849 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:08:50.165 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:49 smithi003 ceph-mon[41530]: from='client.24701 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:08:50.166 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:49 smithi003 ceph-mon[30803]: from='client.24701 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:08:50.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:49 smithi053 ceph-mon[34699]: from='client.24701 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:08:51.191 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:50 smithi003 ceph-mon[41530]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:51.191 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:50 smithi003 ceph-mon[30803]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:51.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:50 smithi053 ceph-mon[34699]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:51.604 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:08:51.604 INFO:teuthology.orchestra.run.smithi003.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-08-17T17:08:52.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:51 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/2082472399' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-08-17T17:08:52.261 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:51 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/2082472399' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-08-17T17:08:52.261 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:51 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/2082472399' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-08-17T17:08:52.292 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-08-17T17:08:52.292 INFO:teuthology.run_tasks:Running task print... 2024-08-17T17:08:52.304 INFO:teuthology.task.print:**** upgrading first half of cluster, with stress **** 2024-08-17T17:08:52.304 INFO:teuthology.run_tasks:Running task parallel... 2024-08-17T17:08:52.312 INFO:teuthology.task.parallel:starting parallel... 2024-08-17T17:08:52.313 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-08-17T17:08:52.313 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-08-17T17:08:52.319 INFO:tasks.workunit:Pulling workunits from ref quincy 2024-08-17T17:08:52.319 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-08-17T17:08:52.319 DEBUG:teuthology.orchestra.run.smithi003:> stat -- /home/ubuntu/cephtest/mnt.0 2024-08-17T17:08:52.320 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-08-17T17:08:52.320 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-08-17T17:08:52.321 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi003.front.sepia.ceph.com 2024-08-17T17:08:52.321 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'ceph config set mgr mgr/cephadm/daemon_cache_timeout 60' 2024-08-17T17:08:52.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:08:52.341 INFO:teuthology.orchestra.run.smithi003.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-17T17:08:52.342 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-08-17T17:08:52.398 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-08-17T17:08:52.399 DEBUG:teuthology.orchestra.run.smithi003:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-08-17T17:08:52.455 INFO:tasks.workunit:timeout=3h 2024-08-17T17:08:52.455 INFO:tasks.workunit:cleanup=True 2024-08-17T17:08:52.456 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch quincy https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 2024-08-17T17:08:52.512 INFO:tasks.workunit.client.0.smithi003.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-08-17T17:08:52.602 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:08:53.015 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:52 smithi003 ceph-mon[41530]: pgmap v90: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:53.015 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:52 smithi003 ceph-mon[30803]: pgmap v90: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:53.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:52 smithi053 ceph-mon[34699]: pgmap v90: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:53.273 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:08:53 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:08:53] "GET /metrics HTTP/1.1" 200 239646 "" "Prometheus/2.43.0" 2024-08-17T17:08:54.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:08:54 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:08:54] ENGINE Bus STOPPING 2024-08-17T17:08:54.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:08:54 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:08:54] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-08-17T17:08:54.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:08:54 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:08:54] ENGINE Bus STOPPED 2024-08-17T17:08:54.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:08:54 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:08:54] ENGINE Bus STARTING 2024-08-17T17:08:54.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:08:54 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:08:54] ENGINE Serving on http://:::9283 2024-08-17T17:08:54.774 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:08:54 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:08:54] ENGINE Bus STARTED 2024-08-17T17:08:55.137 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:54 smithi003 ceph-mon[41530]: pgmap v91: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:55.137 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:54 smithi003 ceph-mon[41530]: from='client.? ' entity='client.admin' 2024-08-17T17:08:55.137 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:54 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:08:55.137 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:54 smithi003 ceph-mon[30803]: pgmap v91: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:55.137 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:54 smithi003 ceph-mon[30803]: from='client.? ' entity='client.admin' 2024-08-17T17:08:55.137 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:54 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:08:55.161 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'ceph config set global log_to_journald false --force' 2024-08-17T17:08:55.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:54 smithi053 ceph-mon[34699]: pgmap v91: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:55.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:54 smithi053 ceph-mon[34699]: from='client.? ' entity='client.admin' 2024-08-17T17:08:55.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:54 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:08:55.491 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:08:56.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:55 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:08:56.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:55 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:08:56.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:55 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:08:56.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:55 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:08:56.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:55 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:08:56.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:55 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:08:56.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:55 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:08:56.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:55 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:08:56.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:55 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:08:56.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:55 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:08:56.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:55 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:08:56.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:55 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:08:56.716 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:08:56 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:08:56] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-08-17T17:08:56.905 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:56 smithi003 ceph-mon[41530]: pgmap v92: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:56.905 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:56 smithi003 ceph-mon[30803]: pgmap v92: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:57.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:56 smithi053 ceph-mon[34699]: pgmap v92: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:58.205 DEBUG:teuthology.orchestra.run.smithi003:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-08-17T17:08:58.265 INFO:tasks.workunit.client.0.smithi003.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-08-17T17:08:58.267 INFO:tasks.workunit.client.0.smithi003.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-08-17T17:08:58.267 INFO:tasks.workunit.client.0.smithi003.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-08-17T17:08:58.618 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-08-17T17:08:58.889 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:08:59.160 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:08:58 smithi003 ceph-mon[41530]: pgmap v93: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:59.160 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:08:58 smithi003 ceph-mon[30803]: pgmap v93: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:59.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:08:58 smithi053 ceph-mon[34699]: pgmap v93: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:08:59.821 INFO:tasks.workunit.client.0.smithi003.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-08-17T17:08:59.921 INFO:tasks.workunit.client.0.smithi003.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-08-17T17:08:59.963 INFO:tasks.workunit.client.0.smithi003.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-08-17T17:08:59.965 INFO:tasks.workunit.client.0.smithi003.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-08-17T17:08:59.965 INFO:tasks.workunit.client.0.smithi003.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-08-17T17:09:00.005 INFO:tasks.workunit.client.0.smithi003.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-08-17T17:09:00.008 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-08-17T17:09:00.008 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-08-17T17:09:00.025 INFO:tasks.workunit:Running workunits matching rbd/import_export.sh on client.0... 2024-08-17T17:09:00.026 INFO:tasks.workunit:Running workunit rbd/import_export.sh... 2024-08-17T17:09:00.026 DEBUG:teuthology.orchestra.run.smithi003:workunit test rbd/import_export.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_CREATE_ARGS=--new-format adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/import_export.sh 2024-08-17T17:09:00.122 INFO:tasks.workunit.client.0.smithi003.stderr:+ export RBD_FORCE_ALLOW_V1=1 2024-08-17T17:09:00.122 INFO:tasks.workunit.client.0.smithi003.stderr:+ RBD_FORCE_ALLOW_V1=1 2024-08-17T17:09:00.122 INFO:tasks.workunit.client.0.smithi003.stderr:+ TMPDIR=/tmp/rbd_import_export_80816 2024-08-17T17:09:00.122 INFO:tasks.workunit.client.0.smithi003.stderr:+ rm -rf /tmp/rbd_import_export_80816 2024-08-17T17:09:00.123 INFO:tasks.workunit.client.0.smithi003.stderr:+ mkdir /tmp/rbd_import_export_80816 2024-08-17T17:09:00.123 INFO:tasks.workunit.client.0.smithi003.stderr:+ trap 'rm -rf /tmp/rbd_import_export_80816' INT TERM EXIT 2024-08-17T17:09:00.123 INFO:tasks.workunit.client.0.smithi003.stderr:+ mkdir foo.80816 2024-08-17T17:09:00.124 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd import foo.80816 foo.dir 2024-08-17T17:09:00.228 INFO:tasks.workunit.client.0.smithi003.stderr:rbd: cannot import a directory 2024-08-17T17:09:00.229 INFO:tasks.workunit.client.0.smithi003.stderr: Importing image: 0% complete...failed. 2024-08-17T17:09:00.229 INFO:tasks.workunit.client.0.smithi003.stderr:rbd: import failed: (21) Is a directory 2024-08-17T17:09:00.237 INFO:tasks.workunit.client.0.smithi003.stderr:+ true 2024-08-17T17:09:00.237 INFO:tasks.workunit.client.0.smithi003.stderr:+ rmdir foo.80816 2024-08-17T17:09:00.253 INFO:tasks.workunit.client.0.smithi003.stderr:+ dd if=/bin/sh of=/tmp/rbd_import_export_80816/img bs=1k count=1 seek=10 2024-08-17T17:09:00.254 INFO:tasks.workunit.client.0.smithi003.stderr:1+0 records in 2024-08-17T17:09:00.254 INFO:tasks.workunit.client.0.smithi003.stderr:1+0 records out 2024-08-17T17:09:00.254 INFO:tasks.workunit.client.0.smithi003.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 4.5666e-05 s, 22.4 MB/s 2024-08-17T17:09:00.254 INFO:tasks.workunit.client.0.smithi003.stderr:+ dd if=/bin/dd of=/tmp/rbd_import_export_80816/img bs=1k count=10 seek=100 2024-08-17T17:09:00.255 INFO:tasks.workunit.client.0.smithi003.stderr:10+0 records in 2024-08-17T17:09:00.255 INFO:tasks.workunit.client.0.smithi003.stderr:10+0 records out 2024-08-17T17:09:00.255 INFO:tasks.workunit.client.0.smithi003.stderr:10240 bytes (10 kB, 10 KiB) copied, 5.5614e-05 s, 184 MB/s 2024-08-17T17:09:00.255 INFO:tasks.workunit.client.0.smithi003.stderr:+ dd if=/bin/rm of=/tmp/rbd_import_export_80816/img bs=1k count=100 seek=1000 2024-08-17T17:09:00.256 INFO:tasks.workunit.client.0.smithi003.stderr:60+1 records in 2024-08-17T17:09:00.256 INFO:tasks.workunit.client.0.smithi003.stderr:60+1 records out 2024-08-17T17:09:00.256 INFO:tasks.workunit.client.0.smithi003.stderr:61656 bytes (62 kB, 60 KiB) copied, 0.000167361 s, 368 MB/s 2024-08-17T17:09:00.256 INFO:tasks.workunit.client.0.smithi003.stderr:+ dd if=/bin/ls of=/tmp/rbd_import_export_80816/img bs=1k seek=10000 2024-08-17T17:09:00.257 INFO:tasks.workunit.client.0.smithi003.stderr:137+1 records in 2024-08-17T17:09:00.257 INFO:tasks.workunit.client.0.smithi003.stderr:137+1 records out 2024-08-17T17:09:00.257 INFO:tasks.workunit.client.0.smithi003.stderr:140952 bytes (141 kB, 138 KiB) copied, 0.000429726 s, 328 MB/s 2024-08-17T17:09:00.257 INFO:tasks.workunit.client.0.smithi003.stderr:+ dd if=/bin/ln of=/tmp/rbd_import_export_80816/img bs=1k seek=100000 2024-08-17T17:09:00.258 INFO:tasks.workunit.client.0.smithi003.stderr:60+1 records in 2024-08-17T17:09:00.258 INFO:tasks.workunit.client.0.smithi003.stderr:60+1 records out 2024-08-17T17:09:00.258 INFO:tasks.workunit.client.0.smithi003.stderr:61608 bytes (62 kB, 60 KiB) copied, 0.000172366 s, 357 MB/s 2024-08-17T17:09:00.259 INFO:tasks.workunit.client.0.smithi003.stderr:+ dd if=/bin/grep of=/tmp/rbd_import_export_80816/img bs=1k seek=1000000 2024-08-17T17:09:00.259 INFO:tasks.workunit.client.0.smithi003.stderr:154+1 records in 2024-08-17T17:09:00.259 INFO:tasks.workunit.client.0.smithi003.stderr:154+1 records out 2024-08-17T17:09:00.259 INFO:tasks.workunit.client.0.smithi003.stderr:158176 bytes (158 kB, 154 KiB) copied, 0.00038307 s, 413 MB/s 2024-08-17T17:09:00.259 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd rm testimg 2024-08-17T17:09:00.297 INFO:tasks.workunit.client.0.smithi003.stderr: Removing image: 0% complete...failed. 2024-08-17T17:09:00.297 INFO:tasks.workunit.client.0.smithi003.stderr:rbd: delete error: (2) No such file or directory 2024-08-17T17:09:00.300 INFO:tasks.workunit.client.0.smithi003.stderr:+ true 2024-08-17T17:09:00.300 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd import --new-format /tmp/rbd_import_export_80816/img testimg 2024-08-17T17:09:00.832 INFO:tasks.workunit.client.0.smithi003.stderr: Importing image: 1% complete... Importing image: 2% complete... Importing image: 3% complete... Importing image: 4% complete... Importing image: 5% complete... Importing image: 6% complete... Importing image: 7% complete... Importing image: 8% complete... Importing image: 9% complete... Importing image: 10% complete... Importing image: 11% complete... Importing image: 12% complete... Importing image: 13% complete... Importing image: 14% complete... Importing image: 15% complete... Importing image: 16% complete... Importing image: 17% complete... Importing image: 18% complete... Importing image: 19% complete... Importing image: 20% complete... Importing image: 21% complete... Importing image: 22% complete... Importing image: 23% complete... Importing image: 24% complete... Importing image: 25% complete... Importing image: 26% complete... Importing image: 27% complete... Importing image: 28% complete... Importing image: 29% complete... Importing image: 30% complete... Importing image: 31% complete... Importing image: 32% complete... Importing image: 33% complete... Importing image: 34% complete... Importing image: 35% complete... Importing image: 36% complete... Importing image: 37% complete... Importing image: 38% complete... Importing image: 39% complete... Importing image: 40% complete... Importing image: 41% complete... Importing image: 42% complete... Importing image: 43% complete... Importing image: 44% complete... Importing image: 45% complete... Importing image: 46% complete... Importing image: 47% complete... Importing image: 48% complete... Importing image: 49% complete... Importing image: 50% complete... Importing image: 51% complete... Importing image: 52% complete... Importing image: 53% complete... Importing image: 54% complete... Importing image: 55% complete... Importing image: 56% complete... Importing image: 57% complete... Importing image: 58% complete... Importing image: 59% complete... Importing image: 60% complete... Importing image: 61% complete... Importing image: 62% complete... Importing image: 63% complete... Importing image: 64% complete... Importing image: 65% complete... Importing image: 66% complete... Importing image: 67% complete... Importing image: 68% complete... Importing image: 69% complete... Importing image: 70% complete... Importing image: 71% complete... Importing image: 72% complete... Importing image: 73% complete... Importing image: 74% complete... Importing image: 75% complete... Importing image: 76% complete... Importing image: 77% complete... Importing image: 78% complete... Importing image: 79% complete... Importing image: 80% complete... Importing image: 81% complete... Importing image: 82% complete... Importing image: 83% complete... Importing image: 84% complete... Importing image: 85% complete... Importing image: 86% complete... Importing image: 87% complete... Importing image: 88% complete... Importing image: 89% complete... Importing image: 90% complete... Importing image: 91% complete... Importing image: 92% complete... Importing image: 93% complete... Importing image: 94% complete... Importing image: 95% complete... Importing image: 96% complete... Importing image: 97% complete... Importing image: 98% complete... Importing image: 99% complete... Importing image: 100% complete...done. 2024-08-17T17:09:00.833 INFO:tasks.workunit.client.0.smithi003.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-08-17T17:09:00.836 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd export testimg /tmp/rbd_import_export_80816/img2 2024-08-17T17:09:01.071 INFO:teuthology.orchestra.run.smithi003.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:09:01.086 INFO:tasks.workunit.client.0.smithi003.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-17T17:09:01.092 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd export testimg - 2024-08-17T17:09:01.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:00 smithi053 ceph-mon[34699]: pgmap v94: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:01.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:00 smithi003 ceph-mon[41530]: pgmap v94: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:01.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:00 smithi003 ceph-mon[30803]: pgmap v94: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:01.820 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'ceph orch ps' 2024-08-17T17:09:01.845 INFO:tasks.workunit.client.0.smithi003.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-17T17:09:01.849 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd rm testimg 2024-08-17T17:09:01.946 INFO:tasks.workunit.client.0.smithi003.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-08-17T17:09:01.949 INFO:tasks.workunit.client.0.smithi003.stderr:+ cmp /tmp/rbd_import_export_80816/img /tmp/rbd_import_export_80816/img2 2024-08-17T17:09:02.070 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:02 smithi003 ceph-mon[30803]: from='client.24746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:02.146 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:09:02.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:02 smithi053 ceph-mon[34699]: from='client.24746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:02.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:02 smithi053 ceph-mon[34699]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:09:02.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:02 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:09:02.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:02 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:09:02.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:02 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:09:02.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:02 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:09:02.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:02 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:09:02.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:02 smithi053 ceph-mon[34699]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:09:02.487 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:02 smithi003 ceph-mon[41530]: from='client.24746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:02.487 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:02 smithi003 ceph-mon[41530]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:09:02.487 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:02 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:09:02.487 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:02 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:09:02.487 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:02 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:09:02.487 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:02 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:09:02.487 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:02 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:09:02.488 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:02 smithi003 ceph-mon[41530]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:09:02.488 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:02 smithi003 ceph-mon[30803]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:09:02.488 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:02 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:09:02.488 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:02 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:09:02.488 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:02 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:09:02.488 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:02 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:09:02.488 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:02 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:09:02.488 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:02 smithi003 ceph-mon[30803]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:cd8bdbeb9fbdab5733aa3aaadc042c965473e453 2024-08-17T17:09:02.631 INFO:tasks.workunit.client.0.smithi003.stderr:+ cmp /tmp/rbd_import_export_80816/img /tmp/rbd_import_export_80816/img3 2024-08-17T17:09:03.145 INFO:tasks.workunit.client.0.smithi003.stderr:+ rm /tmp/rbd_import_export_80816/img2 /tmp/rbd_import_export_80816/img3 2024-08-17T17:09:03.324 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd import --new-format - testimg 2024-08-17T17:09:03.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:03 smithi053 ceph-mon[34699]: pgmap v95: 73 pgs: 73 active+clean; 5.9 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 69 KiB/s rd, 38 KiB/s wr, 12 op/s 2024-08-17T17:09:03.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:03 smithi003 ceph-mon[41530]: pgmap v95: 73 pgs: 73 active+clean; 5.9 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 69 KiB/s rd, 38 KiB/s wr, 12 op/s 2024-08-17T17:09:03.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:03 smithi003 ceph-mon[30803]: pgmap v95: 73 pgs: 73 active+clean; 5.9 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 69 KiB/s rd, 38 KiB/s wr, 12 op/s 2024-08-17T17:09:03.523 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:09:03 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:09:03] "GET /metrics HTTP/1.1" 200 239629 "" "Prometheus/2.43.0" 2024-08-17T17:09:03.710 INFO:tasks.workunit.client.0.smithi003.stderr: Importing image: 100% complete...done. 2024-08-17T17:09:03.711 INFO:tasks.workunit.client.0.smithi003.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-08-17T17:09:03.714 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd export testimg /tmp/rbd_import_export_80816/img2 2024-08-17T17:09:04.020 INFO:tasks.workunit.client.0.smithi003.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-17T17:09:04.024 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd export testimg - 2024-08-17T17:09:04.377 INFO:teuthology.orchestra.run.smithi003.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-08-17T17:09:04.377 INFO:teuthology.orchestra.run.smithi003.stdout:alertmanager.a smithi003 *:9093,9094 running (86s) 60s ago 2m 16.4M - 0.25.0 c8568f914cd2 be92e7808bcf 2024-08-17T17:09:04.377 INFO:teuthology.orchestra.run.smithi003.stdout:grafana.a smithi053 *:3000 running (2m) 63s ago 2m 80.3M - 9.4.7 954c08fa6188 2a28cf19852e 2024-08-17T17:09:04.377 INFO:teuthology.orchestra.run.smithi003.stdout:mds.foo.smithi003.civgvf smithi003 running (74s) 60s ago 73s 16.0M - 17.2.7-1404-gf2c39d4d f69471c56189 86a91b686f66 2024-08-17T17:09:04.377 INFO:teuthology.orchestra.run.smithi003.stdout:mds.foo.smithi053.ougbtc smithi053 running (71s) 63s ago 71s 13.8M - 17.2.7-1404-gf2c39d4d f69471c56189 41ad4cc0ce19 2024-08-17T17:09:04.377 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.x smithi053 *:8443 running (6m) 63s ago 6m 399M - 17.2.7-1404-gf2c39d4d f69471c56189 36a119b52ba2 2024-08-17T17:09:04.377 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.y smithi003 *:9283,8765 running (7m) 60s ago 7m 443M - 17.2.7-1404-gf2c39d4d f69471c56189 8cb7113bf953 2024-08-17T17:09:04.378 INFO:teuthology.orchestra.run.smithi003.stdout:mon.a smithi003 running (7m) 60s ago 8m 45.4M 2048M 17.2.7-1404-gf2c39d4d f69471c56189 a8b52ab645d3 2024-08-17T17:09:04.378 INFO:teuthology.orchestra.run.smithi003.stdout:mon.b smithi053 running (6m) 63s ago 6m 39.1M 2048M 17.2.7-1404-gf2c39d4d f69471c56189 471447c55e31 2024-08-17T17:09:04.378 INFO:teuthology.orchestra.run.smithi003.stdout:mon.c smithi003 running (6m) 60s ago 6m 37.8M 2048M 17.2.7-1404-gf2c39d4d f69471c56189 3cd9ac5ad1af 2024-08-17T17:09:04.378 INFO:teuthology.orchestra.run.smithi003.stdout:node-exporter.a smithi003 *:9100 running (117s) 60s ago 116s 12.1M - 1.5.0 0da6a335fe13 6d9a244ec8ca 2024-08-17T17:09:04.378 INFO:teuthology.orchestra.run.smithi003.stdout:node-exporter.b smithi053 *:9100 running (113s) 63s ago 112s 8074k - 1.5.0 0da6a335fe13 791e642c268d 2024-08-17T17:09:04.378 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0 smithi003 running (5m) 60s ago 5m 43.4M 4017M 17.2.7-1404-gf2c39d4d f69471c56189 dc8e5d51fe1c 2024-08-17T17:09:04.378 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1 smithi003 running (5m) 60s ago 5m 42.1M 4017M 17.2.7-1404-gf2c39d4d f69471c56189 7def3d9ed99f 2024-08-17T17:09:04.378 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2 smithi003 running (5m) 60s ago 5m 40.6M 4017M 17.2.7-1404-gf2c39d4d f69471c56189 3d65e6f7d660 2024-08-17T17:09:04.378 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3 smithi003 running (4m) 60s ago 4m 41.7M 4017M 17.2.7-1404-gf2c39d4d f69471c56189 61f40a5861f6 2024-08-17T17:09:04.378 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4 smithi053 running (4m) 63s ago 4m 40.8M 4017M 17.2.7-1404-gf2c39d4d f69471c56189 ced3b086d7f0 2024-08-17T17:09:04.378 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5 smithi053 running (4m) 63s ago 4m 40.5M 4017M 17.2.7-1404-gf2c39d4d f69471c56189 1c4ad574a193 2024-08-17T17:09:04.378 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6 smithi053 running (3m) 63s ago 3m 40.0M 4017M 17.2.7-1404-gf2c39d4d f69471c56189 5ed81ad77c62 2024-08-17T17:09:04.379 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7 smithi053 running (3m) 63s ago 3m 41.4M 4017M 17.2.7-1404-gf2c39d4d f69471c56189 f3e467b07793 2024-08-17T17:09:04.379 INFO:teuthology.orchestra.run.smithi003.stdout:prometheus.a smithi053 *:9095 running (81s) 63s ago 2m 33.6M - 2.43.0 a07b618ecd1d af8f221d5152 2024-08-17T17:09:04.766 INFO:tasks.workunit.client.0.smithi003.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-08-17T17:09:04.769 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd rm testimg 2024-08-17T17:09:04.868 INFO:tasks.workunit.client.0.smithi003.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-08-17T17:09:04.871 INFO:tasks.workunit.client.0.smithi003.stderr:+ cmp /tmp/rbd_import_export_80816/img /tmp/rbd_import_export_80816/img2 2024-08-17T17:09:05.065 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'echo wait for minority of mons to upgrade' 2024-08-17T17:09:05.410 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:09:05.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:05 smithi053 ceph-mon[34699]: pgmap v96: 73 pgs: 73 active+clean; 5.9 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 69 KiB/s rd, 38 KiB/s wr, 12 op/s 2024-08-17T17:09:05.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:05 smithi003 ceph-mon[30803]: pgmap v96: 73 pgs: 73 active+clean; 5.9 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 69 KiB/s rd, 38 KiB/s wr, 12 op/s 2024-08-17T17:09:05.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:05 smithi003 ceph-mon[41530]: pgmap v96: 73 pgs: 73 active+clean; 5.9 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 69 KiB/s rd, 38 KiB/s wr, 12 op/s 2024-08-17T17:09:05.578 INFO:tasks.workunit.client.0.smithi003.stderr:+ cmp /tmp/rbd_import_export_80816/img /tmp/rbd_import_export_80816/img3 2024-08-17T17:09:06.124 INFO:tasks.workunit.client.0.smithi003.stderr:+ rm /tmp/rbd_import_export_80816/img /tmp/rbd_import_export_80816/img2 /tmp/rbd_import_export_80816/img3 2024-08-17T17:09:06.356 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd help export 2024-08-17T17:09:06.356 INFO:tasks.workunit.client.0.smithi003.stderr:+ grep -q export-format 2024-08-17T17:09:06.377 INFO:tasks.workunit.client.0.smithi003.stderr:+ dd if=/bin/dd of=/tmp/rbd_import_export_80816/img bs=1k count=10 seek=100 2024-08-17T17:09:06.378 INFO:tasks.workunit.client.0.smithi003.stderr:10+0 records in 2024-08-17T17:09:06.378 INFO:tasks.workunit.client.0.smithi003.stderr:10+0 records out 2024-08-17T17:09:06.379 INFO:tasks.workunit.client.0.smithi003.stderr:10240 bytes (10 kB, 10 KiB) copied, 6.4613e-05 s, 158 MB/s 2024-08-17T17:09:06.379 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd import --new-format /tmp/rbd_import_export_80816/img testimg 2024-08-17T17:09:06.437 INFO:tasks.workunit.client.0.smithi003.stderr: Importing image: 100% complete...done. 2024-08-17T17:09:06.437 INFO:tasks.workunit.client.0.smithi003.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-08-17T17:09:06.440 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd snap create testimg@snap 2024-08-17T17:09:06.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:06 smithi053 ceph-mon[34699]: from='client.14880 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:06.466 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:09:06 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:09:06] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-08-17T17:09:06.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:06 smithi003 ceph-mon[41530]: from='client.14880 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:06.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:06 smithi003 ceph-mon[30803]: from='client.14880 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:07.223 INFO:tasks.workunit.client.0.smithi003.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-08-17T17:09:07.260 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd image-meta set testimg key1 value1 2024-08-17T17:09:07.288 INFO:tasks.workunit.client.0.smithi003.stderr:++ rbd image-meta list testimg 2024-08-17T17:09:07.324 INFO:teuthology.orchestra.run.smithi003.stdout:wait for minority of mons to upgrade 2024-08-17T17:09:07.332 INFO:tasks.workunit.client.0.smithi003.stderr:+ IMAGEMETA_BEFORE='There is 1 metadatum on this image: 2024-08-17T17:09:07.332 INFO:tasks.workunit.client.0.smithi003.stderr: 2024-08-17T17:09:07.332 INFO:tasks.workunit.client.0.smithi003.stderr:Key Value 2024-08-17T17:09:07.332 INFO:tasks.workunit.client.0.smithi003.stderr:key1 value1' 2024-08-17T17:09:07.333 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd export --export-format 2 testimg /tmp/rbd_import_export_80816/img_v2 2024-08-17T17:09:07.373 INFO:tasks.workunit.client.0.smithi003.stderr: Exporting image: 100% complete... Exporting image: 100% complete...done. 2024-08-17T17:09:07.377 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd import --export-format 2 /tmp/rbd_import_export_80816/img_v2 testimg_import 2024-08-17T17:09:07.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:07 smithi053 ceph-mon[34699]: pgmap v97: 73 pgs: 73 active+clean; 5.9 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 69 KiB/s rd, 38 KiB/s wr, 12 op/s 2024-08-17T17:09:07.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:07 smithi003 ceph-mon[30803]: pgmap v97: 73 pgs: 73 active+clean; 5.9 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 69 KiB/s rd, 38 KiB/s wr, 12 op/s 2024-08-17T17:09:07.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:07 smithi003 ceph-mon[41530]: pgmap v97: 73 pgs: 73 active+clean; 5.9 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 69 KiB/s rd, 38 KiB/s wr, 12 op/s 2024-08-17T17:09:08.035 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'while ! ceph mon versions | grep $sha1 && ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error; do sleep 2 ; done' 2024-08-17T17:09:08.235 INFO:tasks.workunit.client.0.smithi003.stderr: Importing image: 50% complete... Importing image: 100% complete... Importing image: 100% complete...done. 2024-08-17T17:09:08.238 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd info testimg_import 2024-08-17T17:09:08.272 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:08 smithi003 ceph-mon[30803]: osdmap e74: 8 total, 8 up, 8 in 2024-08-17T17:09:08.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:08 smithi003 ceph-mon[30803]: pgmap v99: 73 pgs: 73 active+clean; 689 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 239 KiB/s rd, 95 KiB/s wr, 89 op/s 2024-08-17T17:09:08.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:08 smithi003 ceph-mon[41530]: osdmap e74: 8 total, 8 up, 8 in 2024-08-17T17:09:08.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:08 smithi003 ceph-mon[41530]: pgmap v99: 73 pgs: 73 active+clean; 689 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 239 KiB/s rd, 95 KiB/s wr, 89 op/s 2024-08-17T17:09:08.277 INFO:tasks.workunit.client.0.smithi003.stdout:rbd image 'testimg_import': 2024-08-17T17:09:08.277 INFO:tasks.workunit.client.0.smithi003.stdout: size 110 KiB in 1 objects 2024-08-17T17:09:08.277 INFO:tasks.workunit.client.0.smithi003.stdout: order 22 (4 MiB objects) 2024-08-17T17:09:08.277 INFO:tasks.workunit.client.0.smithi003.stdout: snapshot_count: 1 2024-08-17T17:09:08.277 INFO:tasks.workunit.client.0.smithi003.stdout: id: 60fb12c2b187 2024-08-17T17:09:08.277 INFO:tasks.workunit.client.0.smithi003.stdout: block_name_prefix: rbd_data.60fb12c2b187 2024-08-17T17:09:08.277 INFO:tasks.workunit.client.0.smithi003.stdout: format: 2 2024-08-17T17:09:08.277 INFO:tasks.workunit.client.0.smithi003.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten 2024-08-17T17:09:08.277 INFO:tasks.workunit.client.0.smithi003.stdout: op_features: 2024-08-17T17:09:08.277 INFO:tasks.workunit.client.0.smithi003.stdout: flags: 2024-08-17T17:09:08.277 INFO:tasks.workunit.client.0.smithi003.stdout: create_timestamp: Sat Aug 17 17:09:07 2024 2024-08-17T17:09:08.278 INFO:tasks.workunit.client.0.smithi003.stdout: access_timestamp: Sat Aug 17 17:09:07 2024 2024-08-17T17:09:08.278 INFO:tasks.workunit.client.0.smithi003.stdout: modify_timestamp: Sat Aug 17 17:09:07 2024 2024-08-17T17:09:08.280 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd info testimg_import@snap 2024-08-17T17:09:08.325 INFO:tasks.workunit.client.0.smithi003.stdout:rbd image 'testimg_import': 2024-08-17T17:09:08.325 INFO:tasks.workunit.client.0.smithi003.stdout: size 110 KiB in 1 objects 2024-08-17T17:09:08.325 INFO:tasks.workunit.client.0.smithi003.stdout: order 22 (4 MiB objects) 2024-08-17T17:09:08.325 INFO:tasks.workunit.client.0.smithi003.stdout: snapshot_count: 1 2024-08-17T17:09:08.326 INFO:tasks.workunit.client.0.smithi003.stdout: id: 60fb12c2b187 2024-08-17T17:09:08.326 INFO:tasks.workunit.client.0.smithi003.stdout: block_name_prefix: rbd_data.60fb12c2b187 2024-08-17T17:09:08.326 INFO:tasks.workunit.client.0.smithi003.stdout: format: 2 2024-08-17T17:09:08.326 INFO:tasks.workunit.client.0.smithi003.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten 2024-08-17T17:09:08.326 INFO:tasks.workunit.client.0.smithi003.stdout: op_features: 2024-08-17T17:09:08.326 INFO:tasks.workunit.client.0.smithi003.stdout: flags: 2024-08-17T17:09:08.326 INFO:tasks.workunit.client.0.smithi003.stdout: create_timestamp: Sat Aug 17 17:09:07 2024 2024-08-17T17:09:08.326 INFO:tasks.workunit.client.0.smithi003.stdout: access_timestamp: Sat Aug 17 17:09:07 2024 2024-08-17T17:09:08.326 INFO:tasks.workunit.client.0.smithi003.stdout: modify_timestamp: Sat Aug 17 17:09:07 2024 2024-08-17T17:09:08.326 INFO:tasks.workunit.client.0.smithi003.stdout: protected: False 2024-08-17T17:09:08.330 INFO:tasks.workunit.client.0.smithi003.stderr:++ rbd image-meta list testimg_import 2024-08-17T17:09:08.372 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:09:08.377 INFO:tasks.workunit.client.0.smithi003.stderr:+ IMAGEMETA_AFTER='There is 1 metadatum on this image: 2024-08-17T17:09:08.377 INFO:tasks.workunit.client.0.smithi003.stderr: 2024-08-17T17:09:08.377 INFO:tasks.workunit.client.0.smithi003.stderr:Key Value 2024-08-17T17:09:08.377 INFO:tasks.workunit.client.0.smithi003.stderr:key1 value1' 2024-08-17T17:09:08.377 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 'There is 1 metadatum on this image: 2024-08-17T17:09:08.377 INFO:tasks.workunit.client.0.smithi003.stderr: 2024-08-17T17:09:08.377 INFO:tasks.workunit.client.0.smithi003.stderr:Key Value 2024-08-17T17:09:08.377 INFO:tasks.workunit.client.0.smithi003.stderr:key1 value1' = 'There is 1 metadatum on this image: 2024-08-17T17:09:08.378 INFO:tasks.workunit.client.0.smithi003.stderr: 2024-08-17T17:09:08.378 INFO:tasks.workunit.client.0.smithi003.stderr:Key Value 2024-08-17T17:09:08.378 INFO:tasks.workunit.client.0.smithi003.stderr:key1 value1' ']' 2024-08-17T17:09:08.378 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd export testimg_import /tmp/rbd_import_export_80816/img_import 2024-08-17T17:09:08.420 INFO:tasks.workunit.client.0.smithi003.stderr: Exporting image: 100% complete...done. 2024-08-17T17:09:08.424 INFO:tasks.workunit.client.0.smithi003.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_80816/img /tmp/rbd_import_export_80816/img_import 2024-08-17T17:09:08.424 INFO:tasks.workunit.client.0.smithi003.stderr:+ cmp -l /tmp/rbd_import_export_80816/img /tmp/rbd_import_export_80816/img_import 2024-08-17T17:09:08.425 INFO:tasks.workunit.client.0.smithi003.stderr:++ stat /tmp/rbd_import_export_80816/img --format %b 2024-08-17T17:09:08.426 INFO:tasks.workunit.client.0.smithi003.stderr:+ origsize=24 2024-08-17T17:09:08.426 INFO:tasks.workunit.client.0.smithi003.stderr:++ stat /tmp/rbd_import_export_80816/img_import --format %b 2024-08-17T17:09:08.427 INFO:tasks.workunit.client.0.smithi003.stderr:+ exportsize=224 2024-08-17T17:09:08.427 INFO:tasks.workunit.client.0.smithi003.stderr:+ difference=200 2024-08-17T17:09:08.428 INFO:tasks.workunit.client.0.smithi003.stderr:+ difference=200 2024-08-17T17:09:08.428 INFO:tasks.workunit.client.0.smithi003.stderr:+ test 200 -ge 0 -a 200 -lt 4096 2024-08-17T17:09:08.428 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd export testimg@snap /tmp/rbd_import_export_80816/img_snap 2024-08-17T17:09:08.466 INFO:tasks.workunit.client.0.smithi003.stderr: Exporting image: 100% complete...done. 2024-08-17T17:09:08.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:08 smithi053 ceph-mon[34699]: osdmap e74: 8 total, 8 up, 8 in 2024-08-17T17:09:08.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:08 smithi053 ceph-mon[34699]: pgmap v99: 73 pgs: 73 active+clean; 689 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 239 KiB/s rd, 95 KiB/s wr, 89 op/s 2024-08-17T17:09:08.469 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd export testimg_import@snap /tmp/rbd_import_export_80816/img_snap_import 2024-08-17T17:09:08.509 INFO:tasks.workunit.client.0.smithi003.stderr: Exporting image: 100% complete...done. 2024-08-17T17:09:08.512 INFO:tasks.workunit.client.0.smithi003.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_80816/img_snap /tmp/rbd_import_export_80816/img_snap_import 2024-08-17T17:09:08.512 INFO:tasks.workunit.client.0.smithi003.stderr:+ cmp -l /tmp/rbd_import_export_80816/img_snap /tmp/rbd_import_export_80816/img_snap_import 2024-08-17T17:09:08.513 INFO:tasks.workunit.client.0.smithi003.stderr:++ stat /tmp/rbd_import_export_80816/img_snap --format %b 2024-08-17T17:09:08.514 INFO:tasks.workunit.client.0.smithi003.stderr:+ origsize=224 2024-08-17T17:09:08.514 INFO:tasks.workunit.client.0.smithi003.stderr:++ stat /tmp/rbd_import_export_80816/img_snap_import --format %b 2024-08-17T17:09:08.515 INFO:tasks.workunit.client.0.smithi003.stderr:+ exportsize=224 2024-08-17T17:09:08.515 INFO:tasks.workunit.client.0.smithi003.stderr:+ difference=0 2024-08-17T17:09:08.515 INFO:tasks.workunit.client.0.smithi003.stderr:+ difference=0 2024-08-17T17:09:08.515 INFO:tasks.workunit.client.0.smithi003.stderr:+ test 0 -ge 0 -a 0 -lt 4096 2024-08-17T17:09:08.516 INFO:tasks.workunit.client.0.smithi003.stderr:+ rm /tmp/rbd_import_export_80816/img_v2 2024-08-17T17:09:08.516 INFO:tasks.workunit.client.0.smithi003.stderr:+ rm /tmp/rbd_import_export_80816/img_import 2024-08-17T17:09:08.516 INFO:tasks.workunit.client.0.smithi003.stderr:+ rm /tmp/rbd_import_export_80816/img_snap 2024-08-17T17:09:08.517 INFO:tasks.workunit.client.0.smithi003.stderr:+ rm /tmp/rbd_import_export_80816/img_snap_import 2024-08-17T17:09:08.518 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd snap rm testimg_import@snap 2024-08-17T17:09:09.228 INFO:tasks.workunit.client.0.smithi003.stderr: Removing snap: 100% complete...done. 2024-08-17T17:09:09.234 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd remove testimg_import 2024-08-17T17:09:09.327 INFO:tasks.workunit.client.0.smithi003.stderr: Removing image: 100% complete...done. 2024-08-17T17:09:09.330 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd snap rm testimg@snap 2024-08-17T17:09:09.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:09 smithi003 ceph-mon[30803]: osdmap e75: 8 total, 8 up, 8 in 2024-08-17T17:09:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:09 smithi003 ceph-mon[41530]: osdmap e75: 8 total, 8 up, 8 in 2024-08-17T17:09:09.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:09 smithi053 ceph-mon[34699]: osdmap e75: 8 total, 8 up, 8 in 2024-08-17T17:09:10.237 INFO:tasks.workunit.client.0.smithi003.stderr: Removing snap: 100% complete...done. 2024-08-17T17:09:10.245 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd rm testimg 2024-08-17T17:09:10.342 INFO:tasks.workunit.client.0.smithi003.stderr: Removing image: 100% complete...done. 2024-08-17T17:09:10.345 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd import --order 20 /tmp/rbd_import_export_80816/img testimg 2024-08-17T17:09:10.403 INFO:tasks.workunit.client.0.smithi003.stderr: Importing image: 100% complete...done. 2024-08-17T17:09:10.404 INFO:tasks.workunit.client.0.smithi003.stderr:rbd: --order is deprecated, use --object-size 2024-08-17T17:09:10.406 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd export --export-format 2 testimg /tmp/rbd_import_export_80816/img_v2 2024-08-17T17:09:10.451 INFO:tasks.workunit.client.0.smithi003.stderr: Exporting image: 100% complete... Exporting image: 100% complete...done. 2024-08-17T17:09:10.454 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd import --export-format 2 /tmp/rbd_import_export_80816/img_v2 testimg_import 2024-08-17T17:09:10.518 INFO:tasks.workunit.client.0.smithi003.stderr: Importing image: 100% complete... Importing image: 100% complete...done. 2024-08-17T17:09:10.522 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd info testimg_import 2024-08-17T17:09:10.522 INFO:tasks.workunit.client.0.smithi003.stderr:+ grep order 2024-08-17T17:09:10.522 INFO:tasks.workunit.client.0.smithi003.stderr:+ awk '{print $2}' 2024-08-17T17:09:10.522 INFO:tasks.workunit.client.0.smithi003.stderr:+ grep 20 2024-08-17T17:09:10.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:10 smithi003 ceph-mon[41530]: osdmap e76: 8 total, 8 up, 8 in 2024-08-17T17:09:10.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:10 smithi003 ceph-mon[41530]: pgmap v102: 73 pgs: 73 active+clean; 689 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 261 KiB/s rd, 83 KiB/s wr, 124 op/s 2024-08-17T17:09:10.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:10 smithi003 ceph-mon[30803]: osdmap e76: 8 total, 8 up, 8 in 2024-08-17T17:09:10.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:10 smithi003 ceph-mon[30803]: pgmap v102: 73 pgs: 73 active+clean; 689 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 261 KiB/s rd, 83 KiB/s wr, 124 op/s 2024-08-17T17:09:10.562 INFO:tasks.workunit.client.0.smithi003.stdout:20 2024-08-17T17:09:10.563 INFO:tasks.workunit.client.0.smithi003.stderr:+ rm /tmp/rbd_import_export_80816/img_v2 2024-08-17T17:09:10.563 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd remove testimg_import 2024-08-17T17:09:10.653 INFO:tasks.workunit.client.0.smithi003.stderr: Removing image: 100% complete...done. 2024-08-17T17:09:10.656 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd remove testimg 2024-08-17T17:09:10.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:10 smithi053 ceph-mon[34699]: osdmap e76: 8 total, 8 up, 8 in 2024-08-17T17:09:10.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:10 smithi053 ceph-mon[34699]: pgmap v102: 73 pgs: 73 active+clean; 689 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 261 KiB/s rd, 83 KiB/s wr, 124 op/s 2024-08-17T17:09:10.745 INFO:tasks.workunit.client.0.smithi003.stderr: Removing image: 100% complete...done. 2024-08-17T17:09:10.749 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd import --image-feature layering /tmp/rbd_import_export_80816/img testimg 2024-08-17T17:09:10.799 INFO:tasks.workunit.client.0.smithi003.stderr: Importing image: 100% complete...done. 2024-08-17T17:09:10.802 INFO:tasks.workunit.client.0.smithi003.stderr:++ rbd info testimg 2024-08-17T17:09:10.802 INFO:tasks.workunit.client.0.smithi003.stderr:++ grep features 2024-08-17T17:09:10.854 INFO:tasks.workunit.client.0.smithi003.stderr:+ FEATURES_BEFORE=' features: layering 2024-08-17T17:09:10.855 INFO:tasks.workunit.client.0.smithi003.stderr: op_features: ' 2024-08-17T17:09:10.855 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd export --export-format 2 testimg /tmp/rbd_import_export_80816/img_v2 2024-08-17T17:09:10.859 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:09:10.897 INFO:tasks.workunit.client.0.smithi003.stderr: Exporting image: 100% complete... Exporting image: 100% complete...done. 2024-08-17T17:09:10.901 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd import --export-format 2 /tmp/rbd_import_export_80816/img_v2 testimg_import 2024-08-17T17:09:10.961 INFO:tasks.workunit.client.0.smithi003.stderr: Importing image: 100% complete... Importing image: 100% complete...done. 2024-08-17T17:09:10.965 INFO:tasks.workunit.client.0.smithi003.stderr:++ rbd info testimg_import 2024-08-17T17:09:10.965 INFO:tasks.workunit.client.0.smithi003.stderr:++ grep features 2024-08-17T17:09:11.004 INFO:tasks.workunit.client.0.smithi003.stderr:+ FEATURES_AFTER=' features: layering 2024-08-17T17:09:11.004 INFO:tasks.workunit.client.0.smithi003.stderr: op_features: ' 2024-08-17T17:09:11.004 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' ' features: layering 2024-08-17T17:09:11.004 INFO:tasks.workunit.client.0.smithi003.stderr: op_features: ' '!=' ' features: layering 2024-08-17T17:09:11.004 INFO:tasks.workunit.client.0.smithi003.stderr: op_features: ' ']' 2024-08-17T17:09:11.005 INFO:tasks.workunit.client.0.smithi003.stderr:+ rm /tmp/rbd_import_export_80816/img_v2 2024-08-17T17:09:11.005 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd remove testimg_import 2024-08-17T17:09:11.080 INFO:tasks.workunit.client.0.smithi003.stderr: Removing image: 100% complete...done. 2024-08-17T17:09:11.083 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd remove testimg 2024-08-17T17:09:11.154 INFO:tasks.workunit.client.0.smithi003.stderr: Removing image: 100% complete...done. 2024-08-17T17:09:11.157 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd import --stripe-count 1000 --stripe-unit 4096 /tmp/rbd_import_export_80816/img testimg 2024-08-17T17:09:11.220 INFO:tasks.workunit.client.0.smithi003.stderr: Importing image: 100% complete...done. 2024-08-17T17:09:11.223 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd export --export-format 2 testimg /tmp/rbd_import_export_80816/img_v2 2024-08-17T17:09:11.267 INFO:tasks.workunit.client.0.smithi003.stderr: Exporting image: 100% complete... Exporting image: 100% complete...done. 2024-08-17T17:09:11.271 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd import --export-format 2 /tmp/rbd_import_export_80816/img_v2 testimg_import 2024-08-17T17:09:11.333 INFO:tasks.workunit.client.0.smithi003.stderr: Importing image: 100% complete... Importing image: 100% complete...done. 2024-08-17T17:09:11.337 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd info testimg_import 2024-08-17T17:09:11.337 INFO:tasks.workunit.client.0.smithi003.stderr:+ grep 'stripe unit' 2024-08-17T17:09:11.337 INFO:tasks.workunit.client.0.smithi003.stderr:+ grep -Ei '(4 KiB|4096)' 2024-08-17T17:09:11.373 INFO:tasks.workunit.client.0.smithi003.stdout: stripe unit: 4 KiB 2024-08-17T17:09:11.374 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd info testimg_import 2024-08-17T17:09:11.374 INFO:tasks.workunit.client.0.smithi003.stderr:+ grep 'stripe count' 2024-08-17T17:09:11.374 INFO:tasks.workunit.client.0.smithi003.stderr:+ awk '{print $3}' 2024-08-17T17:09:11.374 INFO:tasks.workunit.client.0.smithi003.stderr:+ grep 1000 2024-08-17T17:09:11.412 INFO:tasks.workunit.client.0.smithi003.stdout:1000 2024-08-17T17:09:11.413 INFO:tasks.workunit.client.0.smithi003.stderr:+ rm /tmp/rbd_import_export_80816/img_v2 2024-08-17T17:09:11.413 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd remove testimg_import 2024-08-17T17:09:11.501 INFO:tasks.workunit.client.0.smithi003.stderr: Removing image: 3% complete... Removing image: 7% complete... Removing image: 10% complete... Removing image: 14% complete... Removing image: 17% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 32% complete... Removing image: 35% complete... Removing image: 39% complete... Removing image: 42% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 57% complete... Removing image: 60% complete... Removing image: 64% complete... Removing image: 67% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 82% complete... Removing image: 85% complete... Removing image: 89% complete... Removing image: 92% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-17T17:09:11.504 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd remove testimg 2024-08-17T17:09:11.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:11 smithi003 ceph-mon[41530]: osdmap e77: 8 total, 8 up, 8 in 2024-08-17T17:09:11.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:11 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/222806562' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:11.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:11 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:09:11.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:11 smithi003 ceph-mon[41530]: from='client.24923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:11.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:11 smithi003 ceph-mon[30803]: osdmap e77: 8 total, 8 up, 8 in 2024-08-17T17:09:11.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:11 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/222806562' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:11.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:11 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:09:11.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:11 smithi003 ceph-mon[30803]: from='client.24923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:11.592 INFO:tasks.workunit.client.0.smithi003.stderr: Removing image: 3% complete... Removing image: 7% complete... Removing image: 10% complete... Removing image: 14% complete... Removing image: 17% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 32% complete... Removing image: 35% complete... Removing image: 39% complete... Removing image: 42% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 57% complete... Removing image: 60% complete... Removing image: 64% complete... Removing image: 67% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 82% complete... Removing image: 85% complete... Removing image: 89% complete... Removing image: 92% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-08-17T17:09:11.596 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd import --image-format=2 /tmp/rbd_import_export_80816/img testimg 2024-08-17T17:09:11.654 INFO:tasks.workunit.client.0.smithi003.stderr: Importing image: 100% complete...done. 2024-08-17T17:09:11.657 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd snap create testimg@snap1 2024-08-17T17:09:11.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:11 smithi053 ceph-mon[34699]: osdmap e77: 8 total, 8 up, 8 in 2024-08-17T17:09:11.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:11 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/222806562' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:11.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:11 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:09:11.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:11 smithi053 ceph-mon[34699]: from='client.24923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:12.245 INFO:tasks.workunit.client.0.smithi003.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-08-17T17:09:12.252 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd snap create testimg@snap2 2024-08-17T17:09:12.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:12 smithi003 ceph-mon[41530]: from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:12.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:12 smithi003 ceph-mon[41530]: pgmap v104: 73 pgs: 73 active+clean; 689 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 425 KiB/s rd, 46 KiB/s wr, 557 op/s 2024-08-17T17:09:12.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:12 smithi003 ceph-mon[30803]: from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:12.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:12 smithi003 ceph-mon[30803]: pgmap v104: 73 pgs: 73 active+clean; 689 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 425 KiB/s rd, 46 KiB/s wr, 557 op/s 2024-08-17T17:09:12.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:12 smithi053 ceph-mon[34699]: from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:12.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:12 smithi053 ceph-mon[34699]: pgmap v104: 73 pgs: 73 active+clean; 689 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 425 KiB/s rd, 46 KiB/s wr, 557 op/s 2024-08-17T17:09:13.260 INFO:tasks.workunit.client.0.smithi003.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-08-17T17:09:13.266 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd snap protect testimg@snap2 2024-08-17T17:09:13.311 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd export --export-format 2 testimg /tmp/rbd_import_export_80816/snap_protect 2024-08-17T17:09:13.361 INFO:tasks.workunit.client.0.smithi003.stderr: Exporting image: 33% complete... Exporting image: 100% complete... Exporting image: 100% complete...done. 2024-08-17T17:09:13.362 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd import --export-format 2 /tmp/rbd_import_export_80816/snap_protect testimg_import 2024-08-17T17:09:13.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:13 smithi003 ceph-mon[41530]: osdmap e78: 8 total, 8 up, 8 in 2024-08-17T17:09:13.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:13 smithi003 ceph-mon[30803]: osdmap e78: 8 total, 8 up, 8 in 2024-08-17T17:09:13.523 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:09:13 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:09:13] "GET /metrics HTTP/1.1" 200 239995 "" "Prometheus/2.43.0" 2024-08-17T17:09:13.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:13 smithi053 ceph-mon[34699]: osdmap e78: 8 total, 8 up, 8 in 2024-08-17T17:09:13.999 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:09:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:14 smithi003 ceph-mon[30803]: osdmap e79: 8 total, 8 up, 8 in 2024-08-17T17:09:14.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:14 smithi003 ceph-mon[41530]: osdmap e79: 8 total, 8 up, 8 in 2024-08-17T17:09:14.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:14 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/3837780991' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:14.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:14 smithi003 ceph-mon[41530]: pgmap v107: 73 pgs: 73 active+clean; 689 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 426 KiB/s rd, 46 KiB/s wr, 558 op/s 2024-08-17T17:09:14.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:14 smithi003 ceph-mon[41530]: from='client.15168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:14.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:14 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/3837780991' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:14.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:14 smithi003 ceph-mon[30803]: pgmap v107: 73 pgs: 73 active+clean; 689 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 426 KiB/s rd, 46 KiB/s wr, 558 op/s 2024-08-17T17:09:14.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:14 smithi003 ceph-mon[30803]: from='client.15168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:14.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:14 smithi053 ceph-mon[34699]: osdmap e79: 8 total, 8 up, 8 in 2024-08-17T17:09:14.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:14 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/3837780991' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:14.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:14 smithi053 ceph-mon[34699]: pgmap v107: 73 pgs: 73 active+clean; 689 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 426 KiB/s rd, 46 KiB/s wr, 558 op/s 2024-08-17T17:09:14.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:14 smithi053 ceph-mon[34699]: from='client.15168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:15.281 INFO:tasks.workunit.client.0.smithi003.stderr: Importing image: 33% complete... Importing image: 66% complete... Importing image: 100% complete... Importing image: 100% complete...done. 2024-08-17T17:09:15.284 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd info testimg_import@snap1 2024-08-17T17:09:15.284 INFO:tasks.workunit.client.0.smithi003.stderr:+ grep 'protected: False' 2024-08-17T17:09:15.324 INFO:tasks.workunit.client.0.smithi003.stdout: protected: False 2024-08-17T17:09:15.325 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd info testimg_import@snap2 2024-08-17T17:09:15.325 INFO:tasks.workunit.client.0.smithi003.stderr:+ grep 'protected: True' 2024-08-17T17:09:15.366 INFO:tasks.workunit.client.0.smithi003.stdout: protected: True 2024-08-17T17:09:15.366 INFO:tasks.workunit.client.0.smithi003.stderr:+ rm /tmp/rbd_import_export_80816/snap_protect 2024-08-17T17:09:15.367 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd snap unprotect testimg@snap2 2024-08-17T17:09:15.414 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd snap unprotect testimg_import@snap2 2024-08-17T17:09:15.461 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd snap purge testimg 2024-08-17T17:09:15.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:15 smithi003 ceph-mon[41530]: osdmap e80: 8 total, 8 up, 8 in 2024-08-17T17:09:15.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:15 smithi003 ceph-mon[41530]: from='client.25079 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:15.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:15 smithi003 ceph-mon[30803]: osdmap e80: 8 total, 8 up, 8 in 2024-08-17T17:09:15.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:15 smithi003 ceph-mon[30803]: from='client.25079 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:15.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:15 smithi053 ceph-mon[34699]: osdmap e80: 8 total, 8 up, 8 in 2024-08-17T17:09:15.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:15 smithi053 ceph-mon[34699]: from='client.25079 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:16.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:16 smithi003 ceph-mon[41530]: osdmap e81: 8 total, 8 up, 8 in 2024-08-17T17:09:16.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:16 smithi003 ceph-mon[41530]: pgmap v110: 73 pgs: 73 active+clean; 689 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:16.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:16 smithi003 ceph-mon[41530]: osdmap e82: 8 total, 8 up, 8 in 2024-08-17T17:09:16.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:16 smithi003 ceph-mon[30803]: osdmap e81: 8 total, 8 up, 8 in 2024-08-17T17:09:16.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:16 smithi003 ceph-mon[30803]: pgmap v110: 73 pgs: 73 active+clean; 689 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:16.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:16 smithi003 ceph-mon[30803]: osdmap e82: 8 total, 8 up, 8 in 2024-08-17T17:09:16.716 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:09:16 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:09:16] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-08-17T17:09:16.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:16 smithi053 ceph-mon[34699]: osdmap e81: 8 total, 8 up, 8 in 2024-08-17T17:09:16.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:16 smithi053 ceph-mon[34699]: pgmap v110: 73 pgs: 73 active+clean; 689 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:16.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:16 smithi053 ceph-mon[34699]: osdmap e82: 8 total, 8 up, 8 in 2024-08-17T17:09:17.012 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:09:17.204 INFO:tasks.workunit.client.0.smithi003.stderr: Removing all snapshots: 50% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-08-17T17:09:17.211 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd snap purge testimg_import 2024-08-17T17:09:17.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:17 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/4131589449' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:17.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:17 smithi003 ceph-mon[41530]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:17.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:17 smithi003 ceph-mon[41530]: osdmap e83: 8 total, 8 up, 8 in 2024-08-17T17:09:17.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:17 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/4131589449' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:17.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:17 smithi003 ceph-mon[30803]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:17.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:17 smithi003 ceph-mon[30803]: osdmap e83: 8 total, 8 up, 8 in 2024-08-17T17:09:17.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:17 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/4131589449' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:17.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:17 smithi053 ceph-mon[34699]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:17.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:17 smithi053 ceph-mon[34699]: osdmap e83: 8 total, 8 up, 8 in 2024-08-17T17:09:19.214 INFO:tasks.workunit.client.0.smithi003.stderr: Removing all snapshots: 50% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-08-17T17:09:19.220 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd remove testimg 2024-08-17T17:09:19.312 INFO:tasks.workunit.client.0.smithi003.stderr: Removing image: 100% complete...done. 2024-08-17T17:09:19.314 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd remove testimg_import 2024-08-17T17:09:19.407 INFO:tasks.workunit.client.0.smithi003.stderr: Removing image: 100% complete...done. 2024-08-17T17:09:19.410 INFO:tasks.workunit.client.0.smithi003.stderr:+ tiered=0 2024-08-17T17:09:19.411 INFO:tasks.workunit.client.0.smithi003.stderr:+ ceph osd dump 2024-08-17T17:09:19.411 INFO:tasks.workunit.client.0.smithi003.stderr:+ grep '^pool' 2024-08-17T17:09:19.411 INFO:tasks.workunit.client.0.smithi003.stderr:+ grep ''\''rbd'\''' 2024-08-17T17:09:19.411 INFO:tasks.workunit.client.0.smithi003.stderr:+ grep tier 2024-08-17T17:09:19.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:19 smithi053 ceph-mon[34699]: from='client.15222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:19.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:19 smithi053 ceph-mon[34699]: pgmap v113: 73 pgs: 73 active+clean; 799 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 127 KiB/s rd, 12 KiB/s wr, 168 op/s 2024-08-17T17:09:19.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:19 smithi053 ceph-mon[34699]: osdmap e84: 8 total, 8 up, 8 in 2024-08-17T17:09:19.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:19 smithi003 ceph-mon[30803]: from='client.15222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:19.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:19 smithi003 ceph-mon[30803]: pgmap v113: 73 pgs: 73 active+clean; 799 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 127 KiB/s rd, 12 KiB/s wr, 168 op/s 2024-08-17T17:09:19.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:19 smithi003 ceph-mon[30803]: osdmap e84: 8 total, 8 up, 8 in 2024-08-17T17:09:19.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:19 smithi003 ceph-mon[41530]: from='client.15222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:19.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:19 smithi003 ceph-mon[41530]: pgmap v113: 73 pgs: 73 active+clean; 799 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 127 KiB/s rd, 12 KiB/s wr, 168 op/s 2024-08-17T17:09:19.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:19 smithi003 ceph-mon[41530]: osdmap e84: 8 total, 8 up, 8 in 2024-08-17T17:09:19.839 INFO:tasks.workunit.client.0.smithi003.stderr:+ dd if=/dev/urandom bs=1M seek=1 count=1 of=/tmp/rbd_import_export_80816/sparse1 2024-08-17T17:09:19.843 INFO:tasks.workunit.client.0.smithi003.stderr:1+0 records in 2024-08-17T17:09:19.843 INFO:tasks.workunit.client.0.smithi003.stderr:1+0 records out 2024-08-17T17:09:19.843 INFO:tasks.workunit.client.0.smithi003.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0032459 s, 323 MB/s 2024-08-17T17:09:19.843 INFO:tasks.workunit.client.0.smithi003.stderr:+ dd if=/dev/urandom bs=1M count=1 of=/tmp/rbd_import_export_80816/sparse2 2024-08-17T17:09:19.847 INFO:tasks.workunit.client.0.smithi003.stderr:1+0 records in 2024-08-17T17:09:19.847 INFO:tasks.workunit.client.0.smithi003.stderr:1+0 records out 2024-08-17T17:09:19.847 INFO:tasks.workunit.client.0.smithi003.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00303792 s, 345 MB/s 2024-08-17T17:09:19.847 INFO:tasks.workunit.client.0.smithi003.stderr:+ truncate /tmp/rbd_import_export_80816/sparse2 -s 2M 2024-08-17T17:09:19.852 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd rm sparse1 2024-08-17T17:09:19.892 INFO:tasks.workunit.client.0.smithi003.stderr: Removing image: 0% complete...failed. 2024-08-17T17:09:19.892 INFO:tasks.workunit.client.0.smithi003.stderr:rbd: delete error: (2) No such file or directory 2024-08-17T17:09:19.895 INFO:tasks.workunit.client.0.smithi003.stderr:+ true 2024-08-17T17:09:19.895 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd import --new-format --order 20 /tmp/rbd_import_export_80816/sparse1 2024-08-17T17:09:19.959 INFO:tasks.workunit.client.0.smithi003.stderr: Importing image: 50% complete... Importing image: 100% complete...done. 2024-08-17T17:09:19.960 INFO:tasks.workunit.client.0.smithi003.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-08-17T17:09:19.960 INFO:tasks.workunit.client.0.smithi003.stderr:rbd: --order is deprecated, use --object-size 2024-08-17T17:09:19.962 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd ls -l 2024-08-17T17:09:19.963 INFO:tasks.workunit.client.0.smithi003.stderr:+ grep sparse1 2024-08-17T17:09:19.963 INFO:tasks.workunit.client.0.smithi003.stderr:+ grep -Ei '(2 MiB|2048k)' 2024-08-17T17:09:20.003 INFO:tasks.workunit.client.0.smithi003.stdout:sparse1 2 MiB 2 2024-08-17T17:09:20.004 INFO:tasks.workunit.client.0.smithi003.stderr:++ objects sparse1 2024-08-17T17:09:20.004 INFO:tasks.workunit.client.0.smithi003.stderr:++ image=sparse1 2024-08-17T17:09:20.004 INFO:tasks.workunit.client.0.smithi003.stderr:+++ rbd info sparse1 2024-08-17T17:09:20.004 INFO:tasks.workunit.client.0.smithi003.stderr:+++ grep block_name_prefix 2024-08-17T17:09:20.004 INFO:tasks.workunit.client.0.smithi003.stderr:+++ awk '{print $NF;}' 2024-08-17T17:09:20.022 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:09:20.049 INFO:tasks.workunit.client.0.smithi003.stderr:++ prefix=rbd_data.6257285cb9e 2024-08-17T17:09:20.049 INFO:tasks.workunit.client.0.smithi003.stderr:+++ grep rbd_data.6257285cb9e 2024-08-17T17:09:20.050 INFO:tasks.workunit.client.0.smithi003.stderr:++++ get_image_data_pool sparse1 2024-08-17T17:09:20.050 INFO:tasks.workunit.client.0.smithi003.stderr:++++ image=sparse1 2024-08-17T17:09:20.050 INFO:tasks.workunit.client.0.smithi003.stderr:+++ sed -e 's/rbd_data.6257285cb9e\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2024-08-17T17:09:20.050 INFO:tasks.workunit.client.0.smithi003.stderr:+++ sort -u 2024-08-17T17:09:20.051 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ rbd info sparse1 2024-08-17T17:09:20.051 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ awk -F: '{ print $NF }' 2024-08-17T17:09:20.051 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ grep 'data_pool: ' 2024-08-17T17:09:20.094 INFO:tasks.workunit.client.0.smithi003.stderr:++++ data_pool= 2024-08-17T17:09:20.094 INFO:tasks.workunit.client.0.smithi003.stderr:++++ '[' -z ']' 2024-08-17T17:09:20.094 INFO:tasks.workunit.client.0.smithi003.stderr:++++ data_pool=rbd 2024-08-17T17:09:20.094 INFO:tasks.workunit.client.0.smithi003.stderr:++++ echo rbd 2024-08-17T17:09:20.095 INFO:tasks.workunit.client.0.smithi003.stderr:+++ rados ls -p rbd 2024-08-17T17:09:20.134 INFO:tasks.workunit.client.0.smithi003.stderr:++ objects=1 2024-08-17T17:09:20.134 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo 1 2024-08-17T17:09:20.134 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 0 -eq 1 -o 1 = 1 ']' 2024-08-17T17:09:20.134 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd export sparse1 /tmp/rbd_import_export_80816/sparse1.out 2024-08-17T17:09:20.174 INFO:tasks.workunit.client.0.smithi003.stderr: Exporting image: 50% complete... Exporting image: 100% complete...done. 2024-08-17T17:09:20.177 INFO:tasks.workunit.client.0.smithi003.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_80816/sparse1 /tmp/rbd_import_export_80816/sparse1.out 2024-08-17T17:09:20.177 INFO:tasks.workunit.client.0.smithi003.stderr:+ cmp -l /tmp/rbd_import_export_80816/sparse1 /tmp/rbd_import_export_80816/sparse1.out 2024-08-17T17:09:20.179 INFO:tasks.workunit.client.0.smithi003.stderr:++ stat /tmp/rbd_import_export_80816/sparse1 --format %b 2024-08-17T17:09:20.180 INFO:tasks.workunit.client.0.smithi003.stderr:+ origsize=2048 2024-08-17T17:09:20.181 INFO:tasks.workunit.client.0.smithi003.stderr:++ stat /tmp/rbd_import_export_80816/sparse1.out --format %b 2024-08-17T17:09:20.182 INFO:tasks.workunit.client.0.smithi003.stderr:+ exportsize=2048 2024-08-17T17:09:20.182 INFO:tasks.workunit.client.0.smithi003.stderr:+ difference=0 2024-08-17T17:09:20.182 INFO:tasks.workunit.client.0.smithi003.stderr:+ difference=0 2024-08-17T17:09:20.182 INFO:tasks.workunit.client.0.smithi003.stderr:+ test 0 -ge 0 -a 0 -lt 4096 2024-08-17T17:09:20.182 INFO:tasks.workunit.client.0.smithi003.stderr:+ rm /tmp/rbd_import_export_80816/sparse1.out 2024-08-17T17:09:20.183 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd rm sparse1 2024-08-17T17:09:20.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:20 smithi003 ceph-mon[30803]: osdmap e85: 8 total, 8 up, 8 in 2024-08-17T17:09:20.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:20 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/2269262870' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:20.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:20 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/2050742451' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-08-17T17:09:20.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:20 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/2050742451' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-08-17T17:09:20.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:20 smithi003 ceph-mon[41530]: osdmap e85: 8 total, 8 up, 8 in 2024-08-17T17:09:20.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:20 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/2269262870' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:20.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:20 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/2050742451' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-08-17T17:09:20.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:20 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/2050742451' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-08-17T17:09:20.276 INFO:tasks.workunit.client.0.smithi003.stderr: Removing image: 50% complete... Removing image: 100% complete...done. 2024-08-17T17:09:20.279 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd rm sparse2 2024-08-17T17:09:20.319 INFO:tasks.workunit.client.0.smithi003.stderr: Removing image: 0% complete...failed. 2024-08-17T17:09:20.319 INFO:tasks.workunit.client.0.smithi003.stderr:rbd: delete error: (2) No such file or directory 2024-08-17T17:09:20.322 INFO:tasks.workunit.client.0.smithi003.stderr:+ true 2024-08-17T17:09:20.322 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd import --new-format --order 20 /tmp/rbd_import_export_80816/sparse2 2024-08-17T17:09:20.385 INFO:tasks.workunit.client.0.smithi003.stderr: Importing image: 50% complete... Importing image: 100% complete...done. 2024-08-17T17:09:20.386 INFO:tasks.workunit.client.0.smithi003.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-08-17T17:09:20.386 INFO:tasks.workunit.client.0.smithi003.stderr:rbd: --order is deprecated, use --object-size 2024-08-17T17:09:20.388 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd ls -l 2024-08-17T17:09:20.388 INFO:tasks.workunit.client.0.smithi003.stderr:+ grep sparse2 2024-08-17T17:09:20.389 INFO:tasks.workunit.client.0.smithi003.stderr:+ grep -Ei '(2 MiB|2048k)' 2024-08-17T17:09:20.426 INFO:tasks.workunit.client.0.smithi003.stdout:sparse2 2 MiB 2 2024-08-17T17:09:20.427 INFO:tasks.workunit.client.0.smithi003.stderr:++ objects sparse2 2024-08-17T17:09:20.427 INFO:tasks.workunit.client.0.smithi003.stderr:++ image=sparse2 2024-08-17T17:09:20.428 INFO:tasks.workunit.client.0.smithi003.stderr:+++ rbd info sparse2 2024-08-17T17:09:20.428 INFO:tasks.workunit.client.0.smithi003.stderr:+++ grep block_name_prefix 2024-08-17T17:09:20.428 INFO:tasks.workunit.client.0.smithi003.stderr:+++ awk '{print $NF;}' 2024-08-17T17:09:20.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:20 smithi053 ceph-mon[34699]: osdmap e85: 8 total, 8 up, 8 in 2024-08-17T17:09:20.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:20 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/2269262870' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:20.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:20 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/2050742451' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-08-17T17:09:20.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:20 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/2050742451' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-08-17T17:09:20.468 INFO:tasks.workunit.client.0.smithi003.stderr:++ prefix=rbd_data.3bdf2afbacaa 2024-08-17T17:09:20.469 INFO:tasks.workunit.client.0.smithi003.stderr:+++ grep rbd_data.3bdf2afbacaa 2024-08-17T17:09:20.469 INFO:tasks.workunit.client.0.smithi003.stderr:+++ sed -e 's/rbd_data.3bdf2afbacaa\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2024-08-17T17:09:20.469 INFO:tasks.workunit.client.0.smithi003.stderr:++++ get_image_data_pool sparse2 2024-08-17T17:09:20.469 INFO:tasks.workunit.client.0.smithi003.stderr:++++ image=sparse2 2024-08-17T17:09:20.469 INFO:tasks.workunit.client.0.smithi003.stderr:+++ sort -u 2024-08-17T17:09:20.470 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ rbd info sparse2 2024-08-17T17:09:20.470 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ grep 'data_pool: ' 2024-08-17T17:09:20.470 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ awk -F: '{ print $NF }' 2024-08-17T17:09:20.509 INFO:tasks.workunit.client.0.smithi003.stderr:++++ data_pool= 2024-08-17T17:09:20.509 INFO:tasks.workunit.client.0.smithi003.stderr:++++ '[' -z ']' 2024-08-17T17:09:20.510 INFO:tasks.workunit.client.0.smithi003.stderr:++++ data_pool=rbd 2024-08-17T17:09:20.510 INFO:tasks.workunit.client.0.smithi003.stderr:++++ echo rbd 2024-08-17T17:09:20.510 INFO:tasks.workunit.client.0.smithi003.stderr:+++ rados ls -p rbd 2024-08-17T17:09:20.549 INFO:tasks.workunit.client.0.smithi003.stderr:++ objects=0 2024-08-17T17:09:20.549 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo 0 2024-08-17T17:09:20.549 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 0 -eq 1 -o 0 = 0 ']' 2024-08-17T17:09:20.550 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd export sparse2 /tmp/rbd_import_export_80816/sparse2.out 2024-08-17T17:09:20.587 INFO:tasks.workunit.client.0.smithi003.stderr: Exporting image: 50% complete... Exporting image: 100% complete...done. 2024-08-17T17:09:20.591 INFO:tasks.workunit.client.0.smithi003.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_80816/sparse2 /tmp/rbd_import_export_80816/sparse2.out 2024-08-17T17:09:20.591 INFO:tasks.workunit.client.0.smithi003.stderr:+ cmp -l /tmp/rbd_import_export_80816/sparse2 /tmp/rbd_import_export_80816/sparse2.out 2024-08-17T17:09:20.593 INFO:tasks.workunit.client.0.smithi003.stderr:++ stat /tmp/rbd_import_export_80816/sparse2 --format %b 2024-08-17T17:09:20.594 INFO:tasks.workunit.client.0.smithi003.stderr:+ origsize=2048 2024-08-17T17:09:20.594 INFO:tasks.workunit.client.0.smithi003.stderr:++ stat /tmp/rbd_import_export_80816/sparse2.out --format %b 2024-08-17T17:09:20.596 INFO:tasks.workunit.client.0.smithi003.stderr:+ exportsize=2048 2024-08-17T17:09:20.596 INFO:tasks.workunit.client.0.smithi003.stderr:+ difference=0 2024-08-17T17:09:20.596 INFO:tasks.workunit.client.0.smithi003.stderr:+ difference=0 2024-08-17T17:09:20.596 INFO:tasks.workunit.client.0.smithi003.stderr:+ test 0 -ge 0 -a 0 -lt 4096 2024-08-17T17:09:20.596 INFO:tasks.workunit.client.0.smithi003.stderr:+ rm /tmp/rbd_import_export_80816/sparse2.out 2024-08-17T17:09:20.596 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd rm sparse2 2024-08-17T17:09:20.686 INFO:tasks.workunit.client.0.smithi003.stderr: Removing image: 50% complete... Removing image: 100% complete...done. 2024-08-17T17:09:20.689 INFO:tasks.workunit.client.0.smithi003.stderr:+ truncate /tmp/rbd_import_export_80816/sparse1 -s 10M 2024-08-17T17:09:20.690 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd import --new-format --order 20 - sparse1 2024-08-17T17:09:20.774 INFO:tasks.workunit.client.0.smithi003.stderr: Importing image: 100% complete...done. 2024-08-17T17:09:20.775 INFO:tasks.workunit.client.0.smithi003.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-08-17T17:09:20.775 INFO:tasks.workunit.client.0.smithi003.stderr:rbd: --order is deprecated, use --object-size 2024-08-17T17:09:20.778 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd ls -l 2024-08-17T17:09:20.778 INFO:tasks.workunit.client.0.smithi003.stderr:+ grep sparse1 2024-08-17T17:09:20.778 INFO:tasks.workunit.client.0.smithi003.stderr:+ grep -Ei '(10 MiB|10240k)' 2024-08-17T17:09:20.816 INFO:tasks.workunit.client.0.smithi003.stdout:sparse1 10 MiB 2 2024-08-17T17:09:20.816 INFO:tasks.workunit.client.0.smithi003.stderr:++ objects sparse1 2024-08-17T17:09:20.816 INFO:tasks.workunit.client.0.smithi003.stderr:++ image=sparse1 2024-08-17T17:09:20.817 INFO:tasks.workunit.client.0.smithi003.stderr:+++ rbd info sparse1 2024-08-17T17:09:20.817 INFO:tasks.workunit.client.0.smithi003.stderr:+++ grep block_name_prefix 2024-08-17T17:09:20.817 INFO:tasks.workunit.client.0.smithi003.stderr:+++ awk '{print $NF;}' 2024-08-17T17:09:20.853 INFO:tasks.workunit.client.0.smithi003.stderr:++ prefix=rbd_data.62bdfe1856c2 2024-08-17T17:09:20.854 INFO:tasks.workunit.client.0.smithi003.stderr:+++ grep rbd_data.62bdfe1856c2 2024-08-17T17:09:20.854 INFO:tasks.workunit.client.0.smithi003.stderr:+++ sed -e 's/rbd_data.62bdfe1856c2\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2024-08-17T17:09:20.854 INFO:tasks.workunit.client.0.smithi003.stderr:++++ get_image_data_pool sparse1 2024-08-17T17:09:20.854 INFO:tasks.workunit.client.0.smithi003.stderr:++++ image=sparse1 2024-08-17T17:09:20.855 INFO:tasks.workunit.client.0.smithi003.stderr:+++ sort -u 2024-08-17T17:09:20.855 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ rbd info sparse1 2024-08-17T17:09:20.855 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ grep 'data_pool: ' 2024-08-17T17:09:20.855 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ awk -F: '{ print $NF }' 2024-08-17T17:09:20.893 INFO:tasks.workunit.client.0.smithi003.stderr:++++ data_pool= 2024-08-17T17:09:20.893 INFO:tasks.workunit.client.0.smithi003.stderr:++++ '[' -z ']' 2024-08-17T17:09:20.893 INFO:tasks.workunit.client.0.smithi003.stderr:++++ data_pool=rbd 2024-08-17T17:09:20.893 INFO:tasks.workunit.client.0.smithi003.stderr:++++ echo rbd 2024-08-17T17:09:20.893 INFO:tasks.workunit.client.0.smithi003.stderr:+++ rados ls -p rbd 2024-08-17T17:09:20.934 INFO:tasks.workunit.client.0.smithi003.stderr:++ objects=1 2024-08-17T17:09:20.935 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo 1 2024-08-17T17:09:20.935 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 0 -eq 1 -o 1 = 1 ']' 2024-08-17T17:09:20.935 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd export sparse1 /tmp/rbd_import_export_80816/sparse1.out 2024-08-17T17:09:20.977 INFO:tasks.workunit.client.0.smithi003.stderr: Exporting image: 10% complete... Exporting image: 20% complete... Exporting image: 30% complete... Exporting image: 40% complete... Exporting image: 50% complete... Exporting image: 60% complete... Exporting image: 70% complete... Exporting image: 80% complete... Exporting image: 90% complete... Exporting image: 100% complete...done. 2024-08-17T17:09:20.980 INFO:tasks.workunit.client.0.smithi003.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_80816/sparse1 /tmp/rbd_import_export_80816/sparse1.out 2024-08-17T17:09:20.981 INFO:tasks.workunit.client.0.smithi003.stderr:+ cmp -l /tmp/rbd_import_export_80816/sparse1 /tmp/rbd_import_export_80816/sparse1.out 2024-08-17T17:09:20.988 INFO:tasks.workunit.client.0.smithi003.stderr:++ stat /tmp/rbd_import_export_80816/sparse1 --format %b 2024-08-17T17:09:20.989 INFO:tasks.workunit.client.0.smithi003.stderr:+ origsize=2048 2024-08-17T17:09:20.989 INFO:tasks.workunit.client.0.smithi003.stderr:++ stat /tmp/rbd_import_export_80816/sparse1.out --format %b 2024-08-17T17:09:20.991 INFO:tasks.workunit.client.0.smithi003.stderr:+ exportsize=2048 2024-08-17T17:09:20.991 INFO:tasks.workunit.client.0.smithi003.stderr:+ difference=0 2024-08-17T17:09:20.991 INFO:tasks.workunit.client.0.smithi003.stderr:+ difference=0 2024-08-17T17:09:20.991 INFO:tasks.workunit.client.0.smithi003.stderr:+ test 0 -ge 0 -a 0 -lt 4096 2024-08-17T17:09:20.991 INFO:tasks.workunit.client.0.smithi003.stderr:+ rm /tmp/rbd_import_export_80816/sparse1.out 2024-08-17T17:09:20.992 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd rm sparse1 2024-08-17T17:09:21.081 INFO:tasks.workunit.client.0.smithi003.stderr: Removing image: 10% complete... Removing image: 20% complete... Removing image: 30% complete... Removing image: 40% complete... Removing image: 50% complete... Removing image: 60% complete... Removing image: 70% complete... Removing image: 80% complete... Removing image: 90% complete... Removing image: 100% complete...done. 2024-08-17T17:09:21.084 INFO:tasks.workunit.client.0.smithi003.stderr:+ dd if=/dev/urandom bs=2M count=1 of=/tmp/rbd_import_export_80816/sparse2 oflag=append conv=notrunc 2024-08-17T17:09:21.090 INFO:tasks.workunit.client.0.smithi003.stderr:1+0 records in 2024-08-17T17:09:21.091 INFO:tasks.workunit.client.0.smithi003.stderr:1+0 records out 2024-08-17T17:09:21.091 INFO:tasks.workunit.client.0.smithi003.stderr:2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.00582698 s, 360 MB/s 2024-08-17T17:09:21.091 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd import --new-format --order 20 - sparse2 2024-08-17T17:09:21.170 INFO:tasks.workunit.client.0.smithi003.stderr: Importing image: 100% complete...done. 2024-08-17T17:09:21.170 INFO:tasks.workunit.client.0.smithi003.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-08-17T17:09:21.170 INFO:tasks.workunit.client.0.smithi003.stderr:rbd: --order is deprecated, use --object-size 2024-08-17T17:09:21.173 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd ls -l 2024-08-17T17:09:21.173 INFO:tasks.workunit.client.0.smithi003.stderr:+ grep sparse2 2024-08-17T17:09:21.173 INFO:tasks.workunit.client.0.smithi003.stderr:+ grep -Ei '(4 MiB|4096k)' 2024-08-17T17:09:21.219 INFO:tasks.workunit.client.0.smithi003.stdout:sparse2 4 MiB 2 2024-08-17T17:09:21.220 INFO:tasks.workunit.client.0.smithi003.stderr:++ objects sparse2 2024-08-17T17:09:21.220 INFO:tasks.workunit.client.0.smithi003.stderr:++ image=sparse2 2024-08-17T17:09:21.220 INFO:tasks.workunit.client.0.smithi003.stderr:+++ rbd info sparse2 2024-08-17T17:09:21.220 INFO:tasks.workunit.client.0.smithi003.stderr:+++ grep block_name_prefix 2024-08-17T17:09:21.220 INFO:tasks.workunit.client.0.smithi003.stderr:+++ awk '{print $NF;}' 2024-08-17T17:09:21.259 INFO:tasks.workunit.client.0.smithi003.stderr:++ prefix=rbd_data.62ea703d06ad 2024-08-17T17:09:21.260 INFO:tasks.workunit.client.0.smithi003.stderr:+++ grep rbd_data.62ea703d06ad 2024-08-17T17:09:21.260 INFO:tasks.workunit.client.0.smithi003.stderr:++++ get_image_data_pool sparse2 2024-08-17T17:09:21.261 INFO:tasks.workunit.client.0.smithi003.stderr:+++ sed -e 's/rbd_data.62ea703d06ad\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2024-08-17T17:09:21.261 INFO:tasks.workunit.client.0.smithi003.stderr:++++ image=sparse2 2024-08-17T17:09:21.261 INFO:tasks.workunit.client.0.smithi003.stderr:+++ sort -u 2024-08-17T17:09:21.261 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ rbd info sparse2 2024-08-17T17:09:21.261 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ grep 'data_pool: ' 2024-08-17T17:09:21.261 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ awk -F: '{ print $NF }' 2024-08-17T17:09:21.300 INFO:tasks.workunit.client.0.smithi003.stderr:++++ data_pool= 2024-08-17T17:09:21.300 INFO:tasks.workunit.client.0.smithi003.stderr:++++ '[' -z ']' 2024-08-17T17:09:21.300 INFO:tasks.workunit.client.0.smithi003.stderr:++++ data_pool=rbd 2024-08-17T17:09:21.300 INFO:tasks.workunit.client.0.smithi003.stderr:++++ echo rbd 2024-08-17T17:09:21.300 INFO:tasks.workunit.client.0.smithi003.stderr:+++ rados ls -p rbd 2024-08-17T17:09:21.342 INFO:tasks.workunit.client.0.smithi003.stderr:++ objects='0 2024-08-17T17:09:21.342 INFO:tasks.workunit.client.0.smithi003.stderr:2 2024-08-17T17:09:21.342 INFO:tasks.workunit.client.0.smithi003.stderr:3' 2024-08-17T17:09:21.342 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo 0 2 3 2024-08-17T17:09:21.342 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 0 -eq 1 -o '0 2 3' = '0 2 3' ']' 2024-08-17T17:09:21.342 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd export sparse2 /tmp/rbd_import_export_80816/sparse2.out 2024-08-17T17:09:21.383 INFO:tasks.workunit.client.0.smithi003.stderr: Exporting image: 25% complete... Exporting image: 50% complete... Exporting image: 75% complete... Exporting image: 100% complete...done. 2024-08-17T17:09:21.386 INFO:tasks.workunit.client.0.smithi003.stderr:+ compare_files_and_ondisk_sizes /tmp/rbd_import_export_80816/sparse2 /tmp/rbd_import_export_80816/sparse2.out 2024-08-17T17:09:21.387 INFO:tasks.workunit.client.0.smithi003.stderr:+ cmp -l /tmp/rbd_import_export_80816/sparse2 /tmp/rbd_import_export_80816/sparse2.out 2024-08-17T17:09:21.391 INFO:tasks.workunit.client.0.smithi003.stderr:++ stat /tmp/rbd_import_export_80816/sparse2 --format %b 2024-08-17T17:09:21.392 INFO:tasks.workunit.client.0.smithi003.stderr:+ origsize=6144 2024-08-17T17:09:21.392 INFO:tasks.workunit.client.0.smithi003.stderr:++ stat /tmp/rbd_import_export_80816/sparse2.out --format %b 2024-08-17T17:09:21.393 INFO:tasks.workunit.client.0.smithi003.stderr:+ exportsize=6144 2024-08-17T17:09:21.393 INFO:tasks.workunit.client.0.smithi003.stderr:+ difference=0 2024-08-17T17:09:21.393 INFO:tasks.workunit.client.0.smithi003.stderr:+ difference=0 2024-08-17T17:09:21.393 INFO:tasks.workunit.client.0.smithi003.stderr:+ test 0 -ge 0 -a 0 -lt 4096 2024-08-17T17:09:21.394 INFO:tasks.workunit.client.0.smithi003.stderr:+ rm /tmp/rbd_import_export_80816/sparse2.out 2024-08-17T17:09:21.394 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd rm sparse2 2024-08-17T17:09:21.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:21 smithi053 ceph-mon[34699]: pgmap v116: 73 pgs: 73 active+clean; 799 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 127 KiB/s rd, 12 KiB/s wr, 168 op/s 2024-08-17T17:09:21.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:21 smithi053 ceph-mon[34699]: from='client.25163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:21.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:21 smithi053 ceph-mon[34699]: from='client.25202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:21.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:21 smithi053 ceph-mon[34699]: osdmap e86: 8 total, 8 up, 8 in 2024-08-17T17:09:21.480 INFO:tasks.workunit.client.0.smithi003.stderr: Removing image: 25% complete... Removing image: 50% complete... Removing image: 75% complete... Removing image: 100% complete...done. 2024-08-17T17:09:21.483 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo 'partially-sparse file imports to partially-sparse image' 2024-08-17T17:09:21.483 INFO:tasks.workunit.client.0.smithi003.stdout:partially-sparse file imports to partially-sparse image 2024-08-17T17:09:21.484 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd import --new-format --order 20 /tmp/rbd_import_export_80816/sparse1 sparse 2024-08-17T17:09:21.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:21 smithi003 ceph-mon[30803]: pgmap v116: 73 pgs: 73 active+clean; 799 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 127 KiB/s rd, 12 KiB/s wr, 168 op/s 2024-08-17T17:09:21.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:21 smithi003 ceph-mon[30803]: from='client.25163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:21.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:21 smithi003 ceph-mon[30803]: from='client.25202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:21.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:21 smithi003 ceph-mon[30803]: osdmap e86: 8 total, 8 up, 8 in 2024-08-17T17:09:21.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:21 smithi003 ceph-mon[41530]: pgmap v116: 73 pgs: 73 active+clean; 799 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 127 KiB/s rd, 12 KiB/s wr, 168 op/s 2024-08-17T17:09:21.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:21 smithi003 ceph-mon[41530]: from='client.25163 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:21.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:21 smithi003 ceph-mon[41530]: from='client.25202 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:21.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:21 smithi003 ceph-mon[41530]: osdmap e86: 8 total, 8 up, 8 in 2024-08-17T17:09:21.549 INFO:tasks.workunit.client.0.smithi003.stderr: Importing image: 10% complete... Importing image: 20% complete... Importing image: 30% complete... Importing image: 40% complete... Importing image: 50% complete... Importing image: 60% complete... Importing image: 70% complete... Importing image: 80% complete... Importing image: 90% complete... Importing image: 100% complete...done. 2024-08-17T17:09:21.550 INFO:tasks.workunit.client.0.smithi003.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-08-17T17:09:21.550 INFO:tasks.workunit.client.0.smithi003.stderr:rbd: --order is deprecated, use --object-size 2024-08-17T17:09:21.553 INFO:tasks.workunit.client.0.smithi003.stderr:++ objects sparse 2024-08-17T17:09:21.553 INFO:tasks.workunit.client.0.smithi003.stderr:++ image=sparse 2024-08-17T17:09:21.554 INFO:tasks.workunit.client.0.smithi003.stderr:+++ rbd info sparse 2024-08-17T17:09:21.554 INFO:tasks.workunit.client.0.smithi003.stderr:+++ grep block_name_prefix 2024-08-17T17:09:21.554 INFO:tasks.workunit.client.0.smithi003.stderr:+++ awk '{print $NF;}' 2024-08-17T17:09:21.594 INFO:tasks.workunit.client.0.smithi003.stderr:++ prefix=rbd_data.3c66a8284825 2024-08-17T17:09:21.595 INFO:tasks.workunit.client.0.smithi003.stderr:+++ grep rbd_data.3c66a8284825 2024-08-17T17:09:21.595 INFO:tasks.workunit.client.0.smithi003.stderr:++++ get_image_data_pool sparse 2024-08-17T17:09:21.595 INFO:tasks.workunit.client.0.smithi003.stderr:+++ sed -e 's/rbd_data.3c66a8284825\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2024-08-17T17:09:21.595 INFO:tasks.workunit.client.0.smithi003.stderr:++++ image=sparse 2024-08-17T17:09:21.595 INFO:tasks.workunit.client.0.smithi003.stderr:+++ sort -u 2024-08-17T17:09:21.595 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ rbd info sparse 2024-08-17T17:09:21.596 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ grep 'data_pool: ' 2024-08-17T17:09:21.596 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ awk -F: '{ print $NF }' 2024-08-17T17:09:21.634 INFO:tasks.workunit.client.0.smithi003.stderr:++++ data_pool= 2024-08-17T17:09:21.634 INFO:tasks.workunit.client.0.smithi003.stderr:++++ '[' -z ']' 2024-08-17T17:09:21.634 INFO:tasks.workunit.client.0.smithi003.stderr:++++ data_pool=rbd 2024-08-17T17:09:21.634 INFO:tasks.workunit.client.0.smithi003.stderr:++++ echo rbd 2024-08-17T17:09:21.635 INFO:tasks.workunit.client.0.smithi003.stderr:+++ rados ls -p rbd 2024-08-17T17:09:21.669 INFO:tasks.workunit.client.0.smithi003.stderr:++ objects=1 2024-08-17T17:09:21.669 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo 1 2024-08-17T17:09:21.669 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 0 -eq 1 -o 1 = 1 ']' 2024-08-17T17:09:21.669 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd rm sparse 2024-08-17T17:09:21.752 INFO:tasks.workunit.client.0.smithi003.stderr: Removing image: 10% complete... Removing image: 20% complete... Removing image: 30% complete... Removing image: 40% complete... Removing image: 50% complete... Removing image: 60% complete... Removing image: 70% complete... Removing image: 80% complete... Removing image: 90% complete... Removing image: 100% complete...done. 2024-08-17T17:09:21.754 INFO:tasks.workunit.client.0.smithi003.stdout:zeros import through stdin to sparse image 2024-08-17T17:09:21.755 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo 'zeros import through stdin to sparse image' 2024-08-17T17:09:21.755 INFO:tasks.workunit.client.0.smithi003.stderr:+ dd if=/dev/zero bs=1M count=4 2024-08-17T17:09:21.755 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd import --new-format - sparse 2024-08-17T17:09:21.807 INFO:tasks.workunit.client.0.smithi003.stderr:4+0 records in 2024-08-17T17:09:21.807 INFO:tasks.workunit.client.0.smithi003.stderr:4+0 records out 2024-08-17T17:09:21.808 INFO:tasks.workunit.client.0.smithi003.stderr:4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0515837 s, 81.3 MB/s 2024-08-17T17:09:21.816 INFO:tasks.workunit.client.0.smithi003.stderr: Importing image: 100% complete...done. 2024-08-17T17:09:21.817 INFO:tasks.workunit.client.0.smithi003.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-08-17T17:09:21.820 INFO:tasks.workunit.client.0.smithi003.stderr:++ objects sparse 2024-08-17T17:09:21.820 INFO:tasks.workunit.client.0.smithi003.stderr:++ image=sparse 2024-08-17T17:09:21.820 INFO:tasks.workunit.client.0.smithi003.stderr:+++ rbd info sparse 2024-08-17T17:09:21.821 INFO:tasks.workunit.client.0.smithi003.stderr:+++ grep block_name_prefix 2024-08-17T17:09:21.821 INFO:tasks.workunit.client.0.smithi003.stderr:+++ awk '{print $NF;}' 2024-08-17T17:09:21.857 INFO:tasks.workunit.client.0.smithi003.stderr:++ prefix=rbd_data.6338edcbfdcc 2024-08-17T17:09:21.858 INFO:tasks.workunit.client.0.smithi003.stderr:+++ grep rbd_data.6338edcbfdcc 2024-08-17T17:09:21.859 INFO:tasks.workunit.client.0.smithi003.stderr:++++ get_image_data_pool sparse 2024-08-17T17:09:21.859 INFO:tasks.workunit.client.0.smithi003.stderr:++++ image=sparse 2024-08-17T17:09:21.859 INFO:tasks.workunit.client.0.smithi003.stderr:+++ sed -e 's/rbd_data.6338edcbfdcc\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2024-08-17T17:09:21.859 INFO:tasks.workunit.client.0.smithi003.stderr:+++ sort -u 2024-08-17T17:09:21.859 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ rbd info sparse 2024-08-17T17:09:21.859 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ grep 'data_pool: ' 2024-08-17T17:09:21.859 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ awk -F: '{ print $NF }' 2024-08-17T17:09:21.901 INFO:tasks.workunit.client.0.smithi003.stderr:++++ data_pool= 2024-08-17T17:09:21.901 INFO:tasks.workunit.client.0.smithi003.stderr:++++ '[' -z ']' 2024-08-17T17:09:21.901 INFO:tasks.workunit.client.0.smithi003.stderr:++++ data_pool=rbd 2024-08-17T17:09:21.901 INFO:tasks.workunit.client.0.smithi003.stderr:++++ echo rbd 2024-08-17T17:09:21.901 INFO:tasks.workunit.client.0.smithi003.stderr:+++ rados ls -p rbd 2024-08-17T17:09:21.934 INFO:tasks.workunit.client.0.smithi003.stderr:++ objects= 2024-08-17T17:09:21.934 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo 2024-08-17T17:09:21.934 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 0 -eq 1 -o '' = '' ']' 2024-08-17T17:09:21.934 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd rm sparse 2024-08-17T17:09:22.014 INFO:tasks.workunit.client.0.smithi003.stderr: Removing image: 100% complete...done. 2024-08-17T17:09:22.017 INFO:tasks.workunit.client.0.smithi003.stdout:zeros export to sparse file 2024-08-17T17:09:22.017 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo 'zeros export to sparse file' 2024-08-17T17:09:22.017 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd create --new-format sparse --size 4 2024-08-17T17:09:22.057 INFO:tasks.workunit.client.0.smithi003.stderr:rbd: --new-format is deprecated, use --image-format 2 2024-08-17T17:09:22.060 INFO:tasks.workunit.client.0.smithi003.stderr:++ rbd info sparse 2024-08-17T17:09:22.060 INFO:tasks.workunit.client.0.smithi003.stderr:++ grep block_name_prefix 2024-08-17T17:09:22.060 INFO:tasks.workunit.client.0.smithi003.stderr:++ awk '{print $NF;}' 2024-08-17T17:09:22.099 INFO:tasks.workunit.client.0.smithi003.stderr:+ prefix=rbd_data.3ca8f3c516d3 2024-08-17T17:09:22.100 INFO:tasks.workunit.client.0.smithi003.stderr:+ dd if=/dev/zero bs=4M count=1 2024-08-17T17:09:22.100 INFO:tasks.workunit.client.0.smithi003.stderr:++ get_image_data_pool sparse 2024-08-17T17:09:22.100 INFO:tasks.workunit.client.0.smithi003.stderr:++ image=sparse 2024-08-17T17:09:22.101 INFO:tasks.workunit.client.0.smithi003.stderr:+++ rbd info sparse 2024-08-17T17:09:22.101 INFO:tasks.workunit.client.0.smithi003.stderr:+++ grep 'data_pool: ' 2024-08-17T17:09:22.101 INFO:tasks.workunit.client.0.smithi003.stderr:+++ awk -F: '{ print $NF }' 2024-08-17T17:09:22.139 INFO:tasks.workunit.client.0.smithi003.stderr:++ data_pool= 2024-08-17T17:09:22.140 INFO:tasks.workunit.client.0.smithi003.stderr:++ '[' -z ']' 2024-08-17T17:09:22.140 INFO:tasks.workunit.client.0.smithi003.stderr:++ data_pool=rbd 2024-08-17T17:09:22.140 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo rbd 2024-08-17T17:09:22.140 INFO:tasks.workunit.client.0.smithi003.stderr:+ rados -p rbd put rbd_data.3ca8f3c516d3.000000000000 - 2024-08-17T17:09:22.174 INFO:tasks.workunit.client.0.smithi003.stderr:1+0 records in 2024-08-17T17:09:22.174 INFO:tasks.workunit.client.0.smithi003.stderr:1+0 records out 2024-08-17T17:09:22.174 INFO:tasks.workunit.client.0.smithi003.stderr:4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0739775 s, 56.7 MB/s 2024-08-17T17:09:22.193 INFO:tasks.workunit.client.0.smithi003.stderr:++ objects sparse 2024-08-17T17:09:22.193 INFO:tasks.workunit.client.0.smithi003.stderr:++ image=sparse 2024-08-17T17:09:22.193 INFO:tasks.workunit.client.0.smithi003.stderr:+++ rbd info sparse 2024-08-17T17:09:22.194 INFO:tasks.workunit.client.0.smithi003.stderr:+++ grep block_name_prefix 2024-08-17T17:09:22.194 INFO:tasks.workunit.client.0.smithi003.stderr:+++ awk '{print $NF;}' 2024-08-17T17:09:22.234 INFO:tasks.workunit.client.0.smithi003.stderr:++ prefix=rbd_data.3ca8f3c516d3 2024-08-17T17:09:22.235 INFO:tasks.workunit.client.0.smithi003.stderr:+++ grep rbd_data.3ca8f3c516d3 2024-08-17T17:09:22.235 INFO:tasks.workunit.client.0.smithi003.stderr:++++ get_image_data_pool sparse 2024-08-17T17:09:22.235 INFO:tasks.workunit.client.0.smithi003.stderr:++++ image=sparse 2024-08-17T17:09:22.235 INFO:tasks.workunit.client.0.smithi003.stderr:+++ sed -e 's/rbd_data.3ca8f3c516d3\.//' -e 's/^0*\([0-9a-f]\)/\1/' 2024-08-17T17:09:22.235 INFO:tasks.workunit.client.0.smithi003.stderr:+++ sort -u 2024-08-17T17:09:22.235 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ rbd info sparse 2024-08-17T17:09:22.235 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ grep 'data_pool: ' 2024-08-17T17:09:22.236 INFO:tasks.workunit.client.0.smithi003.stderr:+++++ awk -F: '{ print $NF }' 2024-08-17T17:09:22.276 INFO:tasks.workunit.client.0.smithi003.stderr:++++ data_pool= 2024-08-17T17:09:22.276 INFO:tasks.workunit.client.0.smithi003.stderr:++++ '[' -z ']' 2024-08-17T17:09:22.276 INFO:tasks.workunit.client.0.smithi003.stderr:++++ data_pool=rbd 2024-08-17T17:09:22.276 INFO:tasks.workunit.client.0.smithi003.stderr:++++ echo rbd 2024-08-17T17:09:22.276 INFO:tasks.workunit.client.0.smithi003.stderr:+++ rados ls -p rbd 2024-08-17T17:09:22.311 INFO:tasks.workunit.client.0.smithi003.stderr:++ objects=0 2024-08-17T17:09:22.311 INFO:tasks.workunit.client.0.smithi003.stderr:++ echo 0 2024-08-17T17:09:22.311 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 0 -eq 1 -o 0 = 0 ']' 2024-08-17T17:09:22.311 INFO:tasks.workunit.client.0.smithi003.stderr:+ rm /tmp/rbd_import_export_80816/sparse 2024-08-17T17:09:22.312 INFO:tasks.workunit.client.0.smithi003.stderr:rm: cannot remove '/tmp/rbd_import_export_80816/sparse': No such file or directory 2024-08-17T17:09:22.312 INFO:tasks.workunit.client.0.smithi003.stderr:+ true 2024-08-17T17:09:22.312 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd export sparse /tmp/rbd_import_export_80816/sparse 2024-08-17T17:09:22.349 INFO:tasks.workunit.client.0.smithi003.stderr: Exporting image: 100% complete...done. 2024-08-17T17:09:22.352 INFO:tasks.workunit.client.0.smithi003.stderr:++ stat /tmp/rbd_import_export_80816/sparse --format=%b 2024-08-17T17:09:22.353 INFO:tasks.workunit.client.0.smithi003.stderr:+ '[' 0 = 0 ']' 2024-08-17T17:09:22.353 INFO:tasks.workunit.client.0.smithi003.stderr:+ rbd rm sparse 2024-08-17T17:09:22.439 INFO:tasks.workunit.client.0.smithi003.stderr: Removing image: 100% complete...done. 2024-08-17T17:09:22.443 INFO:tasks.workunit.client.0.smithi003.stderr:+ rm /tmp/rbd_import_export_80816/sparse /tmp/rbd_import_export_80816/sparse1 /tmp/rbd_import_export_80816/sparse2 /tmp/rbd_import_export_80816/sparse3 2024-08-17T17:09:22.446 INFO:tasks.workunit.client.0.smithi003.stderr:rm: cannot remove '/tmp/rbd_import_export_80816/sparse3': No such file or directory 2024-08-17T17:09:22.446 INFO:tasks.workunit.client.0.smithi003.stderr:+ true 2024-08-17T17:09:22.446 INFO:tasks.workunit.client.0.smithi003.stdout:OK 2024-08-17T17:09:22.447 INFO:tasks.workunit.client.0.smithi003.stderr:+ echo OK 2024-08-17T17:09:22.447 INFO:tasks.workunit.client.0.smithi003.stderr:+ rm -rf /tmp/rbd_import_export_80816 2024-08-17T17:09:22.448 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-08-17T17:09:22.448 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-17T17:09:22.476 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:22 smithi003 ceph-mon[41530]: pgmap v118: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 474 op/s 2024-08-17T17:09:22.476 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:22 smithi003 ceph-mon[30803]: pgmap v118: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 474 op/s 2024-08-17T17:09:22.481 INFO:tasks.workunit:Stopping ['rbd/import_export.sh'] on client.0... 2024-08-17T17:09:22.482 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-08-17T17:09:22.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:22 smithi053 ceph-mon[34699]: pgmap v118: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 474 op/s 2024-08-17T17:09:22.759 DEBUG:teuthology.parallel:result is None 2024-08-17T17:09:22.760 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-08-17T17:09:22.787 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-08-17T17:09:22.787 DEBUG:teuthology.orchestra.run.smithi003:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-08-17T17:09:22.842 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-08-17T17:09:22.843 INFO:teuthology.task.sequential:In sequential, running task print... 2024-08-17T17:09:22.843 INFO:teuthology.task.print:**** done rbd/import_export.sh 5-workload 2024-08-17T17:09:22.843 DEBUG:teuthology.parallel:result is None 2024-08-17T17:09:23.039 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:09:23.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:23 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/3490896307' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:23.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:23 smithi003 ceph-mon[30803]: from='client.25499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:23.523 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:09:23 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:09:23] "GET /metrics HTTP/1.1" 200 239995 "" "Prometheus/2.43.0" 2024-08-17T17:09:23.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:23 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/3490896307' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:23.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:23 smithi003 ceph-mon[41530]: from='client.25499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:23 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/3490896307' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:23.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:23 smithi053 ceph-mon[34699]: from='client.25499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:24.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:24 smithi003 ceph-mon[41530]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:24.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:24 smithi003 ceph-mon[41530]: pgmap v119: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 363 op/s 2024-08-17T17:09:24.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:24 smithi003 ceph-mon[30803]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:24.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:24 smithi003 ceph-mon[30803]: pgmap v119: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 363 op/s 2024-08-17T17:09:24.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:24 smithi053 ceph-mon[34699]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:24.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:24 smithi053 ceph-mon[34699]: pgmap v119: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 363 op/s 2024-08-17T17:09:25.987 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:09:25.987 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:25 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/4258016369' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:25.988 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:25 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/4258016369' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:26.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:25 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/4258016369' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:26.716 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:09:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:09:26] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-08-17T17:09:27.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:26 smithi003 ceph-mon[30803]: pgmap v120: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1022 KiB/s rd, 958 KiB/s wr, 287 op/s 2024-08-17T17:09:27.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:26 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:09:27.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:26 smithi003 ceph-mon[30803]: from='client.15612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:27.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:26 smithi003 ceph-mon[30803]: osdmap e87: 8 total, 8 up, 8 in 2024-08-17T17:09:27.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:26 smithi003 ceph-mon[41530]: pgmap v120: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1022 KiB/s rd, 958 KiB/s wr, 287 op/s 2024-08-17T17:09:27.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:26 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:09:27.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:26 smithi003 ceph-mon[41530]: from='client.15612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:27.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:26 smithi003 ceph-mon[41530]: osdmap e87: 8 total, 8 up, 8 in 2024-08-17T17:09:27.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:26 smithi053 ceph-mon[34699]: pgmap v120: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1022 KiB/s rd, 958 KiB/s wr, 287 op/s 2024-08-17T17:09:27.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:26 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:09:27.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:26 smithi053 ceph-mon[34699]: from='client.15612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:27.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:26 smithi053 ceph-mon[34699]: osdmap e87: 8 total, 8 up, 8 in 2024-08-17T17:09:28.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:27 smithi003 ceph-mon[41530]: from='client.25523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:28.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:27 smithi003 ceph-mon[30803]: from='client.25523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:28.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:27 smithi053 ceph-mon[34699]: from='client.25523 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:28.996 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:09:28.997 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:28 smithi003 ceph-mon[30803]: pgmap v122: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1021 KiB/s rd, 1.4 MiB/s wr, 339 op/s 2024-08-17T17:09:28.997 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:28 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/580842472' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:28.997 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:28 smithi003 ceph-mon[41530]: pgmap v122: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1021 KiB/s rd, 1.4 MiB/s wr, 339 op/s 2024-08-17T17:09:28.997 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:28 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/580842472' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:29.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:28 smithi053 ceph-mon[34699]: pgmap v122: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1021 KiB/s rd, 1.4 MiB/s wr, 339 op/s 2024-08-17T17:09:29.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:28 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/580842472' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:30.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:29 smithi003 ceph-mon[41530]: from='client.25535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:30.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:29 smithi003 ceph-mon[30803]: from='client.25535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:30.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:29 smithi053 ceph-mon[34699]: from='client.25535 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:30.834 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:30 smithi003 ceph-mon[41530]: from='client.25541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:30.834 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:30 smithi003 ceph-mon[41530]: pgmap v123: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 950 KiB/s rd, 1.3 MiB/s wr, 315 op/s 2024-08-17T17:09:30.835 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:30 smithi003 ceph-mon[30803]: from='client.25541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:30.835 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:30 smithi003 ceph-mon[30803]: pgmap v123: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 950 KiB/s rd, 1.3 MiB/s wr, 315 op/s 2024-08-17T17:09:31.118 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:30 smithi053 ceph-mon[34699]: from='client.25541 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:31.118 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:30 smithi053 ceph-mon[34699]: pgmap v123: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 950 KiB/s rd, 1.3 MiB/s wr, 315 op/s 2024-08-17T17:09:31.985 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:09:31.986 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:31 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/755097744' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:31.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:31 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/755097744' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:32.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:31 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/755097744' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:33.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:32 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:09:33.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:32 smithi003 ceph-mon[30803]: Upgrade: Target is version 19.0.0-5836-gcd8bdbeb (unknown) 2024-08-17T17:09:33.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:32 smithi003 ceph-mon[30803]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d'] 2024-08-17T17:09:33.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:32 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:09:33.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:32 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:09:33.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:32 smithi003 ceph-mon[30803]: Upgrade: Need to upgrade myself (mgr.y) 2024-08-17T17:09:33.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:32 smithi003 ceph-mon[30803]: pgmap v124: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 40 KiB/s rd, 411 KiB/s wr, 53 op/s 2024-08-17T17:09:33.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:32 smithi003 ceph-mon[30803]: from='client.25553 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:33.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:32 smithi003 ceph-mon[30803]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d on smithi053 2024-08-17T17:09:33.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:32 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:09:33.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:32 smithi003 ceph-mon[41530]: Upgrade: Target is version 19.0.0-5836-gcd8bdbeb (unknown) 2024-08-17T17:09:33.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:32 smithi003 ceph-mon[41530]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d'] 2024-08-17T17:09:33.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:32 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:09:33.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:32 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:09:33.025 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:32 smithi003 ceph-mon[41530]: Upgrade: Need to upgrade myself (mgr.y) 2024-08-17T17:09:33.025 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:32 smithi003 ceph-mon[41530]: pgmap v124: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 40 KiB/s rd, 411 KiB/s wr, 53 op/s 2024-08-17T17:09:33.025 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:32 smithi003 ceph-mon[41530]: from='client.25553 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:33.025 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:32 smithi003 ceph-mon[41530]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d on smithi053 2024-08-17T17:09:33.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:32 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:09:33.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:32 smithi053 ceph-mon[34699]: Upgrade: Target is version 19.0.0-5836-gcd8bdbeb (unknown) 2024-08-17T17:09:33.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:32 smithi053 ceph-mon[34699]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, digests ['quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d'] 2024-08-17T17:09:33.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:32 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:09:33.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:32 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:09:33.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:32 smithi053 ceph-mon[34699]: Upgrade: Need to upgrade myself (mgr.y) 2024-08-17T17:09:33.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:32 smithi053 ceph-mon[34699]: pgmap v124: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 40 KiB/s rd, 411 KiB/s wr, 53 op/s 2024-08-17T17:09:33.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:32 smithi053 ceph-mon[34699]: from='client.25553 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:33.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:32 smithi053 ceph-mon[34699]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d on smithi053 2024-08-17T17:09:33.523 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:09:33 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:09:33] "GET /metrics HTTP/1.1" 200 240365 "" "Prometheus/2.43.0" 2024-08-17T17:09:34.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:33 smithi003 ceph-mon[30803]: from='client.25559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:34.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:33 smithi003 ceph-mon[41530]: from='client.25559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:34.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:33 smithi053 ceph-mon[34699]: from='client.25559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:34.862 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:09:35.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:34 smithi003 ceph-mon[41530]: pgmap v125: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 40 KiB/s rd, 411 KiB/s wr, 53 op/s 2024-08-17T17:09:35.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:34 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/433764286' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:35.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:34 smithi003 ceph-mon[30803]: pgmap v125: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 40 KiB/s rd, 411 KiB/s wr, 53 op/s 2024-08-17T17:09:35.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:34 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/433764286' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:35.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:34 smithi053 ceph-mon[34699]: pgmap v125: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 40 KiB/s rd, 411 KiB/s wr, 53 op/s 2024-08-17T17:09:35.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:34 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/433764286' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:36.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:35 smithi003 ceph-mon[30803]: from='client.25571 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:36.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:35 smithi003 ceph-mon[30803]: from='client.25577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:36.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:35 smithi003 ceph-mon[41530]: from='client.25571 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:36.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:35 smithi003 ceph-mon[41530]: from='client.25577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:36.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:35 smithi053 ceph-mon[34699]: from='client.25571 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:36.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:35 smithi053 ceph-mon[34699]: from='client.25577 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:36.716 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:09:36 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:09:36] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-08-17T17:09:37.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:36 smithi053 ceph-mon[34699]: pgmap v126: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 40 KiB/s rd, 411 KiB/s wr, 53 op/s 2024-08-17T17:09:37.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:36 smithi003 ceph-mon[41530]: pgmap v126: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 40 KiB/s rd, 411 KiB/s wr, 53 op/s 2024-08-17T17:09:37.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:36 smithi003 ceph-mon[30803]: pgmap v126: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 40 KiB/s rd, 411 KiB/s wr, 53 op/s 2024-08-17T17:09:37.738 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:09:38.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:37 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/2168415118' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:38.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:37 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/2168415118' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:38.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:37 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/2168415118' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:39.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:38 smithi053 ceph-mon[34699]: from='client.15684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:39.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:38 smithi053 ceph-mon[34699]: pgmap v127: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 35 KiB/s rd, 354 KiB/s wr, 45 op/s 2024-08-17T17:09:39.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:38 smithi053 ceph-mon[34699]: from='client.25595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:39.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:38 smithi003 ceph-mon[41530]: from='client.15684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:39.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:38 smithi003 ceph-mon[41530]: pgmap v127: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 35 KiB/s rd, 354 KiB/s wr, 45 op/s 2024-08-17T17:09:39.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:38 smithi003 ceph-mon[41530]: from='client.25595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:39.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:38 smithi003 ceph-mon[30803]: from='client.15684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:39.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:38 smithi003 ceph-mon[30803]: pgmap v127: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 35 KiB/s rd, 354 KiB/s wr, 45 op/s 2024-08-17T17:09:39.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:38 smithi003 ceph-mon[30803]: from='client.25595 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:40.616 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:09:41.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:40 smithi003 ceph-mon[41530]: pgmap v128: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:41.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:40 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/3789167162' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:41.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:40 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:09:41.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:40 smithi003 ceph-mon[30803]: pgmap v128: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:41.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:40 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/3789167162' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:41.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:40 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:09:41.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:40 smithi053 ceph-mon[34699]: pgmap v128: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:41.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:40 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/3789167162' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:41.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:40 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:09:42.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:41 smithi053 ceph-mon[34699]: from='client.25607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:42.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:41 smithi053 ceph-mon[34699]: from='client.25613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:42.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:41 smithi003 ceph-mon[41530]: from='client.25607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:42.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:41 smithi003 ceph-mon[41530]: from='client.25613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:42.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:41 smithi003 ceph-mon[30803]: from='client.25607 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:42.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:41 smithi003 ceph-mon[30803]: from='client.25613 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:43.140 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:42 smithi003 ceph-mon[41530]: pgmap v129: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:43.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:42 smithi003 ceph-mon[30803]: pgmap v129: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:43.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:42 smithi053 ceph-mon[34699]: pgmap v129: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:43.495 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:09:43.495 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:09:43 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:09:43] "GET /metrics HTTP/1.1" 200 240305 "" "Prometheus/2.43.0" 2024-08-17T17:09:44.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:43 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/1082114795' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:44.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:43 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/1082114795' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:44.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:43 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/1082114795' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:45.175 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:44 smithi003 ceph-mon[41530]: from='client.25625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:45.175 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:44 smithi003 ceph-mon[41530]: from='client.15726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:45.175 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:44 smithi003 ceph-mon[41530]: pgmap v130: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:45.175 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:44 smithi003 ceph-mon[30803]: from='client.25625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:45.175 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:44 smithi003 ceph-mon[30803]: from='client.15726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:45.175 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:44 smithi003 ceph-mon[30803]: pgmap v130: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:45.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:44 smithi053 ceph-mon[34699]: from='client.25625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:45.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:44 smithi053 ceph-mon[34699]: from='client.15726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:45.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:44 smithi053 ceph-mon[34699]: pgmap v130: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:46.369 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:09:46.716 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:09:46 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:09:46] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-08-17T17:09:47.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:46 smithi053 ceph-mon[34699]: pgmap v131: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:47.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:46 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/2186152446' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:47.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:46 smithi003 ceph-mon[30803]: pgmap v131: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:47.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:46 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/2186152446' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:47.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:46 smithi003 ceph-mon[41530]: pgmap v131: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:47.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:46 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/2186152446' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:48.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:47 smithi053 ceph-mon[34699]: from='client.25643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:48.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:47 smithi053 ceph-mon[34699]: from='client.25649 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:48.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:47 smithi003 ceph-mon[30803]: from='client.25643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:48.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:47 smithi003 ceph-mon[30803]: from='client.25649 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:48.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:47 smithi003 ceph-mon[41530]: from='client.25643 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:48.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:47 smithi003 ceph-mon[41530]: from='client.25649 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:49.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:48 smithi053 ceph-mon[34699]: pgmap v132: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:49.260 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:09:49.261 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:48 smithi003 ceph-mon[41530]: pgmap v132: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:49.261 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:48 smithi003 ceph-mon[30803]: pgmap v132: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:50.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:49 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/3038367540' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:50.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:49 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/3038367540' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:50.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:49 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/3038367540' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:51.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:50 smithi053 ceph-mon[34699]: from='client.25661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:51.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:50 smithi053 ceph-mon[34699]: from='client.25667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:51.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:50 smithi053 ceph-mon[34699]: pgmap v133: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:51.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:50 smithi003 ceph-mon[30803]: from='client.25661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:51.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:50 smithi003 ceph-mon[30803]: from='client.25667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:51.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:50 smithi003 ceph-mon[30803]: pgmap v133: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:51.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:50 smithi003 ceph-mon[41530]: from='client.25661 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:51.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:50 smithi003 ceph-mon[41530]: from='client.25667 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:51.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:50 smithi003 ceph-mon[41530]: pgmap v133: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:52.139 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:09:52.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:51 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/300875726' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:52.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:51 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/300875726' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:52.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:51 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/300875726' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:53.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:52 smithi053 ceph-mon[34699]: pgmap v134: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:53.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:52 smithi053 ceph-mon[34699]: from='client.25679 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:53.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:52 smithi003 ceph-mon[41530]: pgmap v134: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:53.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:52 smithi003 ceph-mon[41530]: from='client.25679 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:53.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:52 smithi003 ceph-mon[30803]: pgmap v134: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:53.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:52 smithi003 ceph-mon[30803]: from='client.25679 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:53.274 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:09:53 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:09:53] "GET /metrics HTTP/1.1" 200 240305 "" "Prometheus/2.43.0" 2024-08-17T17:09:54.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:53 smithi053 ceph-mon[34699]: from='client.25685 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:54.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:53 smithi003 ceph-mon[30803]: from='client.25685 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:54.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:53 smithi003 ceph-mon[41530]: from='client.25685 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:55.019 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:09:55.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:54 smithi053 ceph-mon[34699]: pgmap v135: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:55.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:54 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/2612711064' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:55.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:54 smithi003 ceph-mon[41530]: pgmap v135: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:55.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:54 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/2612711064' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:55.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:54 smithi003 ceph-mon[30803]: pgmap v135: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:55.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:54 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/2612711064' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:56.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:55 smithi053 ceph-mon[34699]: from='client.15792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:56.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:55 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:09:56.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:55 smithi003 ceph-mon[41530]: from='client.15792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:56.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:55 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:09:56.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:55 smithi003 ceph-mon[30803]: from='client.15792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:56.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:55 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:09:56.716 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:09:56 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[36563]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:09:56] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-08-17T17:09:57.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:56 smithi053 ceph-mon[34699]: from='client.15798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:57.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:56 smithi053 ceph-mon[34699]: pgmap v136: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:57.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:56 smithi003 ceph-mon[41530]: from='client.15798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:57.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:56 smithi003 ceph-mon[41530]: pgmap v136: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:57.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:56 smithi003 ceph-mon[30803]: from='client.15798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:57.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:56 smithi003 ceph-mon[30803]: pgmap v136: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:57.899 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:09:58.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:57 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/3524898028' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:58.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:57 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/3524898028' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:58.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:57 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/3524898028' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:09:59.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:58 smithi053 ceph-mon[34699]: pgmap v137: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:59.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:58 smithi053 ceph-mon[34699]: from='client.15810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:59.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:09:58 smithi053 ceph-mon[34699]: from='client.25721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:59.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:58 smithi003 ceph-mon[30803]: pgmap v137: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:59.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:58 smithi003 ceph-mon[30803]: from='client.15810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:59.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:09:58 smithi003 ceph-mon[30803]: from='client.25721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:59.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:58 smithi003 ceph-mon[41530]: pgmap v137: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:09:59.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:58 smithi003 ceph-mon[41530]: from='client.15810 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:09:59.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:09:58 smithi003 ceph-mon[41530]: from='client.25721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:00.775 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:10:01.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:00 smithi053 ceph-mon[34699]: pgmap v138: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:01.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:00 smithi053 ceph-mon[34699]: overall HEALTH_OK 2024-08-17T17:10:01.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:00 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/2535223684' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:10:01.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:00 smithi003 ceph-mon[30803]: pgmap v138: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:01.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:00 smithi003 ceph-mon[30803]: overall HEALTH_OK 2024-08-17T17:10:01.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:00 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/2535223684' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:10:01.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:00 smithi003 ceph-mon[41530]: pgmap v138: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:01.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:00 smithi003 ceph-mon[41530]: overall HEALTH_OK 2024-08-17T17:10:01.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:00 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/2535223684' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:10:02.168 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:01 smithi053 ceph-mon[34699]: from='client.25733 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:02.169 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:01 smithi053 ceph-mon[34699]: from='client.15834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:02.169 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:01 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:02.169 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:01 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:10:02.169 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:01 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-17T17:10:02.169 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:01 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:10:02.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:01 smithi003 ceph-mon[30803]: from='client.25733 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:02.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:01 smithi003 ceph-mon[30803]: from='client.15834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:02.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:01 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:02.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:01 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:10:02.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:01 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-17T17:10:02.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:01 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:10:02.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:01 smithi003 ceph-mon[41530]: from='client.25733 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:02.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:01 smithi003 ceph-mon[41530]: from='client.15834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:02.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:01 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:02.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:01 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:10:02.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:01 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-17T17:10:02.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:01 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:10:03.011 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:02 smithi053 ceph-mon[34699]: Upgrade: Updating mgr.x 2024-08-17T17:10:03.012 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:02 smithi053 ceph-mon[34699]: Deploying daemon mgr.x on smithi053 2024-08-17T17:10:03.012 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:02 smithi053 ceph-mon[34699]: pgmap v139: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:03.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:02 smithi003 ceph-mon[30803]: Upgrade: Updating mgr.x 2024-08-17T17:10:03.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:02 smithi003 ceph-mon[30803]: Deploying daemon mgr.x on smithi053 2024-08-17T17:10:03.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:02 smithi003 ceph-mon[30803]: pgmap v139: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:03.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:02 smithi003 ceph-mon[41530]: Upgrade: Updating mgr.x 2024-08-17T17:10:03.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:02 smithi003 ceph-mon[41530]: Deploying daemon mgr.x on smithi053 2024-08-17T17:10:03.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:02 smithi003 ceph-mon[41530]: pgmap v139: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:03.274 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:03 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:10:03] "GET /metrics HTTP/1.1" 200 240275 "" "Prometheus/2.43.0" 2024-08-17T17:10:03.466 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:03 smithi053 systemd[1]: Stopping Ceph mgr.x for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:10:03.657 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:10:03.773 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:10:03 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a[74261]: ts=2024-08-17T17:10:03.511Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi053.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.53:8443: connect: connection refused" 2024-08-17T17:10:03.782 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:03 smithi053 podman[67269]: 2024-08-17 17:10:03.517021824 +0000 UTC m=+0.135221254 container died 36a119b52ba24d1f4f378cb3fdd634e3185b9c108d08021bf94353646393ed45 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, RELEASE=quincy-f2c39d4, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.build-date=20240731) 2024-08-17T17:10:04.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:03 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/518451175' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:10:04.216 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:03 smithi053 podman[67269]: 2024-08-17 17:10:03.939623341 +0000 UTC m=+0.557822770 container cleanup 36a119b52ba24d1f4f378cb3fdd634e3185b9c108d08021bf94353646393ed45 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, RELEASE=quincy-f2c39d4, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-08-17T17:10:04.217 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:03 smithi053 bash[67269]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x 2024-08-17T17:10:04.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:03 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/518451175' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:10:04.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:03 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/518451175' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:10:04.510 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:04 smithi053 podman[67281]: 2024-08-17 17:10:04.232615586 +0000 UTC m=+0.719548853 container remove 36a119b52ba24d1f4f378cb3fdd634e3185b9c108d08021bf94353646393ed45 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=quincy-f2c39d4, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-08-17T17:10:04.510 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:04 smithi053 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-08-17T17:10:04.923 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:04 smithi053 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-08-17T17:10:04.923 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:04 smithi053 systemd[1]: Stopped Ceph mgr.x for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:10:04.924 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:04 smithi053 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mgr.x.service: Consumed 10.653s CPU time. 2024-08-17T17:10:04.924 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:04 smithi053 systemd[1]: Starting Ceph mgr.x for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:10:05.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:04 smithi053 ceph-mon[34699]: from='client.15846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:05.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:04 smithi053 ceph-mon[34699]: pgmap v140: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:05.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:04 smithi053 ceph-mon[34699]: from='client.25757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:05.217 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:04 smithi053 podman[67408]: 2024-08-17 17:10:04.863585477 +0000 UTC m=+0.017990583 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:10:05.217 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:04 smithi053 podman[67408]: 2024-08-17 17:10:04.975206905 +0000 UTC m=+0.129611999 container create 409905c993755fa2fa198682c610ab03d7ef585f5933a393fca5978ef854e83f (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=main-cd8bdbe, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0) 2024-08-17T17:10:05.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:04 smithi003 ceph-mon[41530]: from='client.15846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:05.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:04 smithi003 ceph-mon[41530]: pgmap v140: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:05.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:04 smithi003 ceph-mon[41530]: from='client.25757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:05.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:04 smithi003 ceph-mon[30803]: from='client.15846 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:05.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:04 smithi003 ceph-mon[30803]: pgmap v140: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:05.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:04 smithi003 ceph-mon[30803]: from='client.25757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:05.716 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:05 smithi053 podman[67408]: 2024-08-17 17:10:05.344651744 +0000 UTC m=+0.499056840 container init 409905c993755fa2fa198682c610ab03d7ef585f5933a393fca5978ef854e83f (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-cd8bdbe) 2024-08-17T17:10:05.716 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:05 smithi053 podman[67408]: 2024-08-17 17:10:05.347928307 +0000 UTC m=+0.502333410 container start 409905c993755fa2fa198682c610ab03d7ef585f5933a393fca5978ef854e83f (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-cd8bdbe, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, ceph=True) 2024-08-17T17:10:05.716 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:05 smithi053 ceph-mgr[67427]: -- 172.21.15.53:0/1202752914 <== mon.1 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5633df5cf0e0 con 0x5633df5ad400 2024-08-17T17:10:05.717 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:05 smithi053 bash[67408]: 409905c993755fa2fa198682c610ab03d7ef585f5933a393fca5978ef854e83f 2024-08-17T17:10:05.717 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:05 smithi053 systemd[1]: Started Ceph mgr.x for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:10:06.528 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:10:06.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:06 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:06.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:06 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:06.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:06 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:10:06.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:06 smithi003 ceph-mon[41530]: pgmap v141: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:06.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:06 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/2542075946' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:10:06.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:06 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:06.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:06 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:06.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:06 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:10:06.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:06 smithi003 ceph-mon[30803]: pgmap v141: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:06.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:06 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/2542075946' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:10:06.888 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:06 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:06.889 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:06 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:06.889 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:06 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:10:06.889 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:06 smithi053 ceph-mon[34699]: pgmap v141: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:06.889 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:06 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/2542075946' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:10:07.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:07 smithi053 ceph-mon[34699]: from='client.25772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:07.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:07 smithi053 ceph-mon[34699]: from='client.15873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:07.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:07 smithi003 ceph-mon[30803]: from='client.25772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:07.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:07 smithi003 ceph-mon[30803]: from='client.15873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:07.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:07 smithi003 ceph-mon[41530]: from='client.25772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:07.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:07 smithi003 ceph-mon[41530]: from='client.15873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:08.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:08 smithi003 ceph-mon[30803]: pgmap v142: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:08.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:08 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:08.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:08 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:08.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:08 smithi003 ceph-mon[41530]: pgmap v142: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:08.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:08 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:08.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:08 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:08.799 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:08 smithi053 ceph-mon[34699]: pgmap v142: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:08.799 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:08 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:08.799 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:08 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:09.414 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:10:09.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:09 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/1659797056' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:10:09.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:09 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/1659797056' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:10:09.794 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:09 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/1659797056' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:10:11.216 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: [17/Aug/2024:17:10:10] ENGINE Bus STARTING 2024-08-17T17:10:11.216 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: CherryPy Checker: 2024-08-17T17:10:11.216 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: The Application mounted at '' has an empty config. 2024-08-17T17:10:11.216 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: 2024-08-17T17:10:11.217 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: [17/Aug/2024:17:10:10] ENGINE Serving on http://:::9283 2024-08-17T17:10:11.217 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: [17/Aug/2024:17:10:10] ENGINE Bus STARTED 2024-08-17T17:10:11.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:11 smithi003 ceph-mon[41530]: from='client.25790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:11.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:11 smithi003 ceph-mon[41530]: from='client.25796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:11.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:11 smithi003 ceph-mon[41530]: pgmap v143: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:11.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:11 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:11.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:11 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:11.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:11 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:10:11.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:11 smithi003 ceph-mon[41530]: Standby manager daemon x restarted 2024-08-17T17:10:11.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:11 smithi003 ceph-mon[41530]: Standby manager daemon x started 2024-08-17T17:10:11.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:11 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:10:11.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:11 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:10:11.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:11 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:10:11.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:11 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:10:11.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:11 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:11.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:11 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:11.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:11 smithi003 ceph-mon[30803]: from='client.25790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:11.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:11 smithi003 ceph-mon[30803]: from='client.25796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:11.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:11 smithi003 ceph-mon[30803]: pgmap v143: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:11.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:11 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:11.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:11 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:11.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:11 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:10:11.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:11 smithi003 ceph-mon[30803]: Standby manager daemon x restarted 2024-08-17T17:10:11.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:11 smithi003 ceph-mon[30803]: Standby manager daemon x started 2024-08-17T17:10:11.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:11 smithi003 ceph-mon[30803]: from='mgr.? 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:10:11.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:11 smithi003 ceph-mon[30803]: from='mgr.? 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:10:11.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:11 smithi003 ceph-mon[30803]: from='mgr.? 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:10:11.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:11 smithi003 ceph-mon[30803]: from='mgr.? 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:10:11.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:11 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:11.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:11 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:11.541 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:11 smithi053 ceph-mon[34699]: from='client.25790 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:11.541 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:11 smithi053 ceph-mon[34699]: from='client.25796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:11.541 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:11 smithi053 ceph-mon[34699]: pgmap v143: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:11.541 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:11 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:11.541 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:11 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:11.541 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:11 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:10:11.541 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:11 smithi053 ceph-mon[34699]: Standby manager daemon x restarted 2024-08-17T17:10:11.541 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:11 smithi053 ceph-mon[34699]: Standby manager daemon x started 2024-08-17T17:10:11.541 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:11 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:10:11.542 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:11 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:10:11.542 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:11 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:10:11.542 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:11 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:10:11.542 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:11 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:11.542 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:11 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:12.293 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:10:12.294 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:10:12 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a[74261]: ts=2024-08-17T17:10:12.040Z caller=notify.go:743 level=info component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify success" attempts=7 2024-08-17T17:10:12.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:12 smithi053 ceph-mon[34699]: mgrmap e21: y(active, since 3m), standbys: x 2024-08-17T17:10:12.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:12 smithi053 ceph-mon[34699]: pgmap v144: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:12.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:12 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/1935381965' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:10:12.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:12 smithi003 ceph-mon[30803]: mgrmap e21: y(active, since 3m), standbys: x 2024-08-17T17:10:12.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:12 smithi003 ceph-mon[30803]: pgmap v144: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:12.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:12 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/1935381965' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:10:12.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:12 smithi003 ceph-mon[41530]: mgrmap e21: y(active, since 3m), standbys: x 2024-08-17T17:10:12.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:12 smithi003 ceph-mon[41530]: pgmap v144: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:12.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:12 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/1935381965' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:10:13.505 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:13 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:10:13] "GET /metrics HTTP/1.1" 200 240249 "" "Prometheus/2.43.0" 2024-08-17T17:10:13.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:13 smithi053 ceph-mon[34699]: from='client.15903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:13.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:13 smithi053 ceph-mon[34699]: from='client.25814 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:13.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:13 smithi003 ceph-mon[30803]: from='client.15903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:13.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:13 smithi003 ceph-mon[30803]: from='client.25814 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:13.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:13 smithi003 ceph-mon[41530]: from='client.15903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:13.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:13 smithi003 ceph-mon[41530]: from='client.25814 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:14.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:14 smithi003 ceph-mon[30803]: pgmap v145: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:14.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:14 smithi003 ceph-mon[41530]: pgmap v145: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:14.845 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:14 smithi053 ceph-mon[34699]: pgmap v145: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:15.168 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:10:15.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:15 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/3992896158' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:10:15.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:15 smithi003 ceph-mon[30803]: from='client.15921 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:15.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:15 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/3992896158' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:10:15.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:15 smithi003 ceph-mon[41530]: from='client.15921 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:15.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:15 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/3992896158' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:10:15.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:15 smithi053 ceph-mon[34699]: from='client.15921 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:16.716 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:16 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:10:16] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-08-17T17:10:16.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:16 smithi053 ceph-mon[34699]: from='client.15927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:16.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:16 smithi053 ceph-mon[34699]: pgmap v146: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:16.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:16 smithi003 ceph-mon[30803]: from='client.15927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:16.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:16 smithi003 ceph-mon[30803]: pgmap v146: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:16.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:16 smithi003 ceph-mon[41530]: from='client.15927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:16.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:16 smithi003 ceph-mon[41530]: pgmap v146: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:18.052 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:10:18.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:17 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/3646709935' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:10:18.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:17 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/3646709935' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:10:18.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:17 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/3646709935' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:10:19.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:18 smithi053 ceph-mon[34699]: pgmap v147: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:19.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:18 smithi053 ceph-mon[34699]: from='client.25844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:19.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:18 smithi003 ceph-mon[30803]: pgmap v147: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:19.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:18 smithi003 ceph-mon[30803]: from='client.25844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:19.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:18 smithi003 ceph-mon[41530]: pgmap v147: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:19.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:18 smithi003 ceph-mon[41530]: from='client.25844 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:20.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:19 smithi053 ceph-mon[34699]: from='client.15945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:20.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:19 smithi003 ceph-mon[30803]: from='client.15945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:20.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:19 smithi003 ceph-mon[41530]: from='client.15945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:20.931 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:10:21.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:20 smithi053 ceph-mon[34699]: pgmap v148: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:21.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:20 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/1409303356' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:10:21.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:20 smithi003 ceph-mon[30803]: pgmap v148: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:21.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:20 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/1409303356' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:10:21.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:20 smithi003 ceph-mon[41530]: pgmap v148: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:21.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:20 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/1409303356' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:10:22.156 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:21 smithi003 ceph-mon[41530]: from='client.25862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:22.156 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:21 smithi003 ceph-mon[41530]: from='client.15963 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:22.156 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:21 smithi003 ceph-mon[30803]: from='client.25862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:22.156 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:21 smithi003 ceph-mon[30803]: from='client.15963 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:22.156 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:10:22.002+0000 7fcec8001640 -1 mgr handle_mgr_map I was active but no longer am 2024-08-17T17:10:22.156 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: ignoring --setuser ceph since I am not root 2024-08-17T17:10:22.157 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: ignoring --setgroup ceph since I am not root 2024-08-17T17:10:22.157 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mgr[31034]: -- 172.21.15.3:0/1961301489 <== mon.2 v2:172.21.15.3:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ded30be9c0 con 0x55ded30cb000 2024-08-17T17:10:22.157 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:10:22.154+0000 7f8be5bd4ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-17T17:10:22.216 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: [17/Aug/2024:17:10:22] ENGINE Bus STOPPING 2024-08-17T17:10:22.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:21 smithi053 ceph-mon[34699]: from='client.25862 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:22.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:21 smithi053 ceph-mon[34699]: from='client.15963 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:22.523 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:10:22.287+0000 7f8be5bd4ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-17T17:10:22.523 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:10:22.356+0000 7f8be5bd4ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-17T17:10:22.716 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: [17/Aug/2024:17:10:22] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-08-17T17:10:22.716 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: [17/Aug/2024:17:10:22] ENGINE Bus STOPPED 2024-08-17T17:10:22.893 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:10:22.606+0000 7f8be5bd4ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-17T17:10:22.893 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:10:22.666+0000 7f8be5bd4ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-17T17:10:23.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:23.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:23.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:10:23.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:10:23.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:23.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:10:23.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-08-17T17:10:23.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-mon[34699]: Activating manager daemon x 2024-08-17T17:10:23.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-mon[34699]: osdmap e88: 8 total, 8 up, 8 in 2024-08-17T17:10:23.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-mon[34699]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-08-17T17:10:23.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:10:23.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-mon[34699]: mgrmap e22: x(active, starting, since 0.00885213s) 2024-08-17T17:10:23.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:10:23.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:10:23.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi003.civgvf"}]: dispatch 2024-08-17T17:10:23.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi053.ougbtc"}]: dispatch 2024-08-17T17:10:23.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-17T17:10:23.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:10:23.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:10:23.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:10:23.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:10:23.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:10:23.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:10:23.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:10:23.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:10:23.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-17T17:10:23.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-17T17:10:23.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-17T17:10:23.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:22 smithi053 ceph-mon[34699]: Manager daemon x is now available 2024-08-17T17:10:23.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:23.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:23.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:10:23.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:10:23.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:23.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:10:23.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-08-17T17:10:23.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[30803]: Activating manager daemon x 2024-08-17T17:10:23.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[30803]: osdmap e88: 8 total, 8 up, 8 in 2024-08-17T17:10:23.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[30803]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-08-17T17:10:23.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:10:23.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[30803]: mgrmap e22: x(active, starting, since 0.00885213s) 2024-08-17T17:10:23.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:10:23.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:10:23.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi003.civgvf"}]: dispatch 2024-08-17T17:10:23.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi053.ougbtc"}]: dispatch 2024-08-17T17:10:23.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-17T17:10:23.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:10:23.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:10:23.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:10:23.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:10:23.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:10:23.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:10:23.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:10:23.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:10:23.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-17T17:10:23.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-17T17:10:23.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-17T17:10:23.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[30803]: Manager daemon x is now available 2024-08-17T17:10:23.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:23.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:23.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:10:23.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:10:23.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' 2024-08-17T17:10:23.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:10:23.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-08-17T17:10:23.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[41530]: Activating manager daemon x 2024-08-17T17:10:23.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[41530]: osdmap e88: 8 total, 8 up, 8 in 2024-08-17T17:10:23.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[41530]: from='mgr.14457 172.21.15.3:0/571380618' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-08-17T17:10:23.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:10:23.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[41530]: mgrmap e22: x(active, starting, since 0.00885213s) 2024-08-17T17:10:23.277 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:10:23.277 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:10:23.277 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi003.civgvf"}]: dispatch 2024-08-17T17:10:23.277 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi053.ougbtc"}]: dispatch 2024-08-17T17:10:23.277 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-17T17:10:23.277 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:10:23.277 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:10:23.277 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:10:23.277 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:10:23.277 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:10:23.277 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:10:23.277 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:10:23.277 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:10:23.277 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-17T17:10:23.278 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-17T17:10:23.278 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-17T17:10:23.278 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:22 smithi003 ceph-mon[41530]: Manager daemon x is now available 2024-08-17T17:10:23.830 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENOTSUP: Module 'orchestrator' is not enabled/loaded (required by command 'orch upgrade status'): use `ceph mgr module enable orchestrator` to enable it 2024-08-17T17:10:24.006 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:23 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:10:23.624+0000 7f8be5bd4ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-17T17:10:24.006 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:23 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:10:23.742+0000 7f8be5bd4ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-17T17:10:24.006 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:23 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:10:23.800+0000 7f8be5bd4ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-17T17:10:24.174 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:23 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: 2024-08-17T17:10:23.828+0000 7f9c8d4f1640 -1 mgr.server reply reply (95) Operation not supported Module 'orchestrator' is not enabled/loaded (required by command 'orch upgrade status'): use `ceph mgr module enable orchestrator` to enable it 2024-08-17T17:10:24.175 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:23 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: [17/Aug/2024:17:10:23] ENGINE Bus STARTING 2024-08-17T17:10:24.175 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:23 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: 2024-08-17T17:10:23.967+0000 7f9c61499640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:10:24.175 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:23 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: 2024-08-17T17:10:23.967+0000 7f9c61499640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:10:24.175 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:23 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: 2024-08-17T17:10:23.967+0000 7f9c61499640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:10:24.175 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:23 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: 2024-08-17T17:10:23.967+0000 7f9c61499640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:10:24.175 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:23 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: 2024-08-17T17:10:23.967+0000 7f9c61499640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:10:24.175 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:23 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: 2024-08-17T17:10:23.967+0000 7f9c6349d640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:10:24.175 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:23 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: 2024-08-17T17:10:23.967+0000 7f9c6349d640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:10:24.175 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:23 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: 2024-08-17T17:10:23.967+0000 7f9c6349d640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:10:24.175 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:23 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: 2024-08-17T17:10:23.967+0000 7f9c6349d640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:10:24.175 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:23 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: 2024-08-17T17:10:23.967+0000 7f9c6349d640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:10:24.175 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:24 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: [17/Aug/2024:17:10:24] ENGINE Serving on http://:::9283 2024-08-17T17:10:24.176 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:24 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: [17/Aug/2024:17:10:24] ENGINE Bus STARTED 2024-08-17T17:10:24.176 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:24 smithi053 ceph-mon[34699]: mgrmap e23: x(active, since 1.01017s) 2024-08-17T17:10:24.176 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:24 smithi053 ceph-mon[34699]: pgmap v3: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:24.176 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:24 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/3689824267' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:10:24.176 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:24 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:24.176 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:24 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:24.176 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:24 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:24.176 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:24 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:24.176 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:24 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:24.176 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:24 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:10:24.176 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:24 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:10:24.177 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:24 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:10:24.177 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:24 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:10:24.177 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:24 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:10:24.177 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:24 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-08-17T17:10:24.177 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:24 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-08-17T17:10:24.271 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:10:24.005+0000 7f8be5bd4ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-17T17:10:24.271 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:10:24.070+0000 7f8be5bd4ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-17T17:10:24.271 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:10:24.139+0000 7f8be5bd4ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-17T17:10:24.271 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:10:24.269+0000 7f8be5bd4ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-17T17:10:24.271 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-mon[41530]: mgrmap e23: x(active, since 1.01017s) 2024-08-17T17:10:24.271 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-mon[41530]: pgmap v3: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:24.271 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/3689824267' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:10:24.271 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:24.271 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:24.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:24.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:24.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:24.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:10:24.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:10:24.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:10:24.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:10:24.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:10:24.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-08-17T17:10:24.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-08-17T17:10:24.272 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-mon[30803]: mgrmap e23: x(active, since 1.01017s) 2024-08-17T17:10:24.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-mon[30803]: pgmap v3: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:24.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/3689824267' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:10:24.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:24.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:24.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:24.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:24.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:24.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:10:24.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:10:24.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:10:24.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:10:24.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:10:24.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-08-17T17:10:24.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-08-17T17:10:24.426 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'ceph orch ps' 2024-08-17T17:10:24.706 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:10:25.015 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:24 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:10:24.728+0000 7f8be5bd4ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-17T17:10:25.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:25 smithi003 ceph-mon[30803]: Checking for cert/key for grafana.a 2024-08-17T17:10:25.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:25 smithi003 ceph-mon[30803]: Migrating grafana.a cert to cert store 2024-08-17T17:10:25.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:25 smithi003 ceph-mon[30803]: Migrating grafana.a key to cert store 2024-08-17T17:10:25.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:25 smithi003 ceph-mon[30803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:10:25.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:25 smithi003 ceph-mon[30803]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:25.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:25 smithi003 ceph-mon[41530]: Checking for cert/key for grafana.a 2024-08-17T17:10:25.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:25 smithi003 ceph-mon[41530]: Migrating grafana.a cert to cert store 2024-08-17T17:10:25.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:25 smithi003 ceph-mon[41530]: Migrating grafana.a key to cert store 2024-08-17T17:10:25.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:25 smithi003 ceph-mon[41530]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:10:25.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:25 smithi003 ceph-mon[41530]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:25.274 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:10:25.038+0000 7f8be5bd4ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-17T17:10:25.274 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:10:25.111+0000 7f8be5bd4ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-17T17:10:25.274 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:10:25.243+0000 7f8be5bd4ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-17T17:10:25.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:25 smithi053 ceph-mon[34699]: Checking for cert/key for grafana.a 2024-08-17T17:10:25.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:25 smithi053 ceph-mon[34699]: Migrating grafana.a cert to cert store 2024-08-17T17:10:25.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:25 smithi053 ceph-mon[34699]: Migrating grafana.a key to cert store 2024-08-17T17:10:25.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:25 smithi053 ceph-mon[34699]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:10:25.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:25 smithi053 ceph-mon[34699]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:25.764 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:10:25.509+0000 7f8be5bd4ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-17T17:10:25.764 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:10:25.572+0000 7f8be5bd4ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-17T17:10:26.015 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:10:25.763+0000 7f8be5bd4ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-17T17:10:26.015 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:10:25.829+0000 7f8be5bd4ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-17T17:10:26.015 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:10:25.918+0000 7f8be5bd4ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-17T17:10:26.015 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:10:25.979+0000 7f8be5bd4ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-17T17:10:26.268 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:26 smithi003 ceph-mon[41530]: Deploying cephadm binary to smithi053 2024-08-17T17:10:26.268 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:26 smithi003 ceph-mon[41530]: Deploying cephadm binary to smithi003 2024-08-17T17:10:26.268 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:26 smithi003 ceph-mon[41530]: mgrmap e24: x(active, since 3s) 2024-08-17T17:10:26.268 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:26 smithi003 ceph-mon[30803]: Deploying cephadm binary to smithi053 2024-08-17T17:10:26.268 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:26 smithi003 ceph-mon[30803]: Deploying cephadm binary to smithi003 2024-08-17T17:10:26.268 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:26 smithi003 ceph-mon[30803]: mgrmap e24: x(active, since 3s) 2024-08-17T17:10:26.311 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:26 smithi053 ceph-mon[34699]: Deploying cephadm binary to smithi053 2024-08-17T17:10:26.312 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:26 smithi053 ceph-mon[34699]: Deploying cephadm binary to smithi003 2024-08-17T17:10:26.312 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:26 smithi053 ceph-mon[34699]: mgrmap e24: x(active, since 3s) 2024-08-17T17:10:26.312 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: 2024-08-17T17:10:26.002+0000 7f9c479e3640 -1 log_channel(cephadm) log [ERR] : [17/Aug/2024:17:10:26] ENGINE Error in HTTPServer.serve 2024-08-17T17:10:26.312 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: Traceback (most recent call last): 2024-08-17T17:10:26.312 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:10:26.312 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: self._connections.run(self.expiration_interval) 2024-08-17T17:10:26.312 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:10:26.312 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: self._run(expiration_interval) 2024-08-17T17:10:26.313 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:10:26.313 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:10:26.313 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:10:26.313 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:10:26.313 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:10:26.313 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: s = self.context.wrap_socket( 2024-08-17T17:10:26.313 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:10:26.313 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: return self.sslsocket_class._create( 2024-08-17T17:10:26.313 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:10:26.313 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: self.do_handshake() 2024-08-17T17:10:26.314 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:10:26.314 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: self._sslobj.do_handshake() 2024-08-17T17:10:26.314 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:10:26.314 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: 2024-08-17T17:10:26.314 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:10:26] "GET /metrics HTTP/1.1" 200 33488 "" "Prometheus/2.43.0" 2024-08-17T17:10:26.589 INFO:teuthology.orchestra.run.smithi003.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-08-17T17:10:26.589 INFO:teuthology.orchestra.run.smithi003.stdout:alertmanager.a smithi003 *:9093,9094 running (2m) 18s ago 3m 25.9M - 0.25.0 c8568f914cd2 be92e7808bcf 2024-08-17T17:10:26.589 INFO:teuthology.orchestra.run.smithi003.stdout:grafana.a smithi053 *:3000 running (3m) 16s ago 3m 85.6M - 9.4.7 954c08fa6188 2a28cf19852e 2024-08-17T17:10:26.589 INFO:teuthology.orchestra.run.smithi003.stdout:mds.foo.smithi003.civgvf smithi003 running (2m) 18s ago 2m 16.9M - 17.2.7-1404-gf2c39d4d f69471c56189 86a91b686f66 2024-08-17T17:10:26.589 INFO:teuthology.orchestra.run.smithi003.stdout:mds.foo.smithi053.ougbtc smithi053 running (2m) 16s ago 2m 14.4M - 17.2.7-1404-gf2c39d4d f69471c56189 41ad4cc0ce19 2024-08-17T17:10:26.589 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.x smithi053 *:8443,9283 running (21s) 16s ago 7m 55.1M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 409905c99375 2024-08-17T17:10:26.590 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.y smithi003 *:9283,8765 running (9m) 18s ago 9m 449M - 17.2.7-1404-gf2c39d4d f69471c56189 8cb7113bf953 2024-08-17T17:10:26.590 INFO:teuthology.orchestra.run.smithi003.stdout:mon.a smithi003 running (9m) 18s ago 9m 52.3M 2048M 17.2.7-1404-gf2c39d4d f69471c56189 a8b52ab645d3 2024-08-17T17:10:26.590 INFO:teuthology.orchestra.run.smithi003.stdout:mon.b smithi053 running (8m) 16s ago 8m 42.3M 2048M 17.2.7-1404-gf2c39d4d f69471c56189 471447c55e31 2024-08-17T17:10:26.590 INFO:teuthology.orchestra.run.smithi003.stdout:mon.c smithi003 running (7m) 18s ago 7m 40.3M 2048M 17.2.7-1404-gf2c39d4d f69471c56189 3cd9ac5ad1af 2024-08-17T17:10:26.590 INFO:teuthology.orchestra.run.smithi003.stdout:node-exporter.a smithi003 *:9100 running (3m) 18s ago 3m 14.2M - 1.5.0 0da6a335fe13 6d9a244ec8ca 2024-08-17T17:10:26.590 INFO:teuthology.orchestra.run.smithi003.stdout:node-exporter.b smithi053 *:9100 running (3m) 16s ago 3m 14.5M - 1.5.0 0da6a335fe13 791e642c268d 2024-08-17T17:10:26.590 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0 smithi003 running (7m) 18s ago 7m 48.7M 4017M 17.2.7-1404-gf2c39d4d f69471c56189 dc8e5d51fe1c 2024-08-17T17:10:26.590 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1 smithi003 running (6m) 18s ago 6m 49.0M 4017M 17.2.7-1404-gf2c39d4d f69471c56189 7def3d9ed99f 2024-08-17T17:10:26.590 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2 smithi003 running (6m) 18s ago 6m 46.0M 4017M 17.2.7-1404-gf2c39d4d f69471c56189 3d65e6f7d660 2024-08-17T17:10:26.590 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3 smithi003 running (6m) 18s ago 6m 43.6M 4017M 17.2.7-1404-gf2c39d4d f69471c56189 61f40a5861f6 2024-08-17T17:10:26.590 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4 smithi053 running (5m) 16s ago 5m 45.3M 4017M 17.2.7-1404-gf2c39d4d f69471c56189 ced3b086d7f0 2024-08-17T17:10:26.590 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5 smithi053 running (5m) 16s ago 5m 45.4M 4017M 17.2.7-1404-gf2c39d4d f69471c56189 1c4ad574a193 2024-08-17T17:10:26.590 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6 smithi053 running (5m) 16s ago 5m 44.3M 4017M 17.2.7-1404-gf2c39d4d f69471c56189 5ed81ad77c62 2024-08-17T17:10:26.590 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7 smithi053 running (4m) 16s ago 4m 46.7M 4017M 17.2.7-1404-gf2c39d4d f69471c56189 f3e467b07793 2024-08-17T17:10:26.591 INFO:teuthology.orchestra.run.smithi003.stdout:prometheus.a smithi053 *:9095 running (2m) 16s ago 4m 51.7M - 2.43.0 a07b618ecd1d af8f221d5152 2024-08-17T17:10:26.829 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:26 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:10:26.575+0000 7f8be5bd4ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-17T17:10:26.829 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:26 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:10:26.722+0000 7f8be5bd4ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-17T17:10:26.829 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:26 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:10:26] ENGINE Bus STARTING 2024-08-17T17:10:26.830 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:26 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: CherryPy Checker: 2024-08-17T17:10:26.830 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:26 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: The Application mounted at '' has an empty config. 2024-08-17T17:10:26.830 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:26 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: 2024-08-17T17:10:27.230 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:26 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:10:26] ENGINE Serving on http://:::9283 2024-08-17T17:10:27.231 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:26 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: [17/Aug/2024:17:10:26] ENGINE Bus STARTED 2024-08-17T17:10:27.231 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:27 smithi003 ceph-mon[41530]: [17/Aug/2024:17:10:25] ENGINE Bus STARTING 2024-08-17T17:10:27.231 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:27 smithi003 ceph-mon[41530]: [17/Aug/2024:17:10:25] ENGINE Serving on http://172.21.15.53:8765 2024-08-17T17:10:27.231 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:27 smithi003 ceph-mon[41530]: [17/Aug/2024:17:10:26] ENGINE Serving on https://172.21.15.53:7150 2024-08-17T17:10:27.231 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:27 smithi003 ceph-mon[41530]: [17/Aug/2024:17:10:26] ENGINE Bus STARTED 2024-08-17T17:10:27.231 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:27 smithi003 ceph-mon[41530]: [17/Aug/2024:17:10:26] ENGINE Error in HTTPServer.serve 2024-08-17T17:10:27.231 INFO:journalctl@ceph.mon.c.smithi003.stdout: Traceback (most recent call last): 2024-08-17T17:10:27.231 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:10:27.231 INFO:journalctl@ceph.mon.c.smithi003.stdout: self._connections.run(self.expiration_interval) 2024-08-17T17:10:27.231 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:10:27.232 INFO:journalctl@ceph.mon.c.smithi003.stdout: self._run(expiration_interval) 2024-08-17T17:10:27.232 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:10:27.232 INFO:journalctl@ceph.mon.c.smithi003.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:10:27.232 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:10:27.232 INFO:journalctl@ceph.mon.c.smithi003.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:10:27.232 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:10:27.232 INFO:journalctl@ceph.mon.c.smithi003.stdout: s = self.context.wrap_socket( 2024-08-17T17:10:27.232 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:10:27.232 INFO:journalctl@ceph.mon.c.smithi003.stdout: return self.sslsocket_class._create( 2024-08-17T17:10:27.232 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:10:27.232 INFO:journalctl@ceph.mon.c.smithi003.stdout: self.do_handshake() 2024-08-17T17:10:27.232 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:10:27.232 INFO:journalctl@ceph.mon.c.smithi003.stdout: self._sslobj.do_handshake() 2024-08-17T17:10:27.232 INFO:journalctl@ceph.mon.c.smithi003.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:10:27.232 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:27 smithi003 ceph-mon[41530]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:27.232 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:27 smithi003 ceph-mon[41530]: Standby manager daemon y started 2024-08-17T17:10:27.233 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:27 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.3:0/2541253961' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-08-17T17:10:27.233 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:27 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.3:0/2541253961' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:10:27.233 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:27 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.3:0/2541253961' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-08-17T17:10:27.233 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:27 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.3:0/2541253961' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:10:27.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:27 smithi003 ceph-mon[30803]: [17/Aug/2024:17:10:25] ENGINE Bus STARTING 2024-08-17T17:10:27.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:27 smithi003 ceph-mon[30803]: [17/Aug/2024:17:10:25] ENGINE Serving on http://172.21.15.53:8765 2024-08-17T17:10:27.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:27 smithi003 ceph-mon[30803]: [17/Aug/2024:17:10:26] ENGINE Serving on https://172.21.15.53:7150 2024-08-17T17:10:27.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:27 smithi003 ceph-mon[30803]: [17/Aug/2024:17:10:26] ENGINE Bus STARTED 2024-08-17T17:10:27.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:27 smithi003 ceph-mon[30803]: [17/Aug/2024:17:10:26] ENGINE Error in HTTPServer.serve 2024-08-17T17:10:27.233 INFO:journalctl@ceph.mon.a.smithi003.stdout: Traceback (most recent call last): 2024-08-17T17:10:27.233 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:10:27.233 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._connections.run(self.expiration_interval) 2024-08-17T17:10:27.233 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:10:27.234 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._run(expiration_interval) 2024-08-17T17:10:27.234 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:10:27.234 INFO:journalctl@ceph.mon.a.smithi003.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:10:27.234 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:10:27.234 INFO:journalctl@ceph.mon.a.smithi003.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:10:27.234 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:10:27.234 INFO:journalctl@ceph.mon.a.smithi003.stdout: s = self.context.wrap_socket( 2024-08-17T17:10:27.234 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:10:27.234 INFO:journalctl@ceph.mon.a.smithi003.stdout: return self.sslsocket_class._create( 2024-08-17T17:10:27.234 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:10:27.234 INFO:journalctl@ceph.mon.a.smithi003.stdout: self.do_handshake() 2024-08-17T17:10:27.234 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:10:27.234 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._sslobj.do_handshake() 2024-08-17T17:10:27.234 INFO:journalctl@ceph.mon.a.smithi003.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:10:27.234 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:27 smithi003 ceph-mon[30803]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:27.235 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:27 smithi003 ceph-mon[30803]: Standby manager daemon y started 2024-08-17T17:10:27.235 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:27 smithi003 ceph-mon[30803]: from='mgr.? 172.21.15.3:0/2541253961' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-08-17T17:10:27.235 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:27 smithi003 ceph-mon[30803]: from='mgr.? 172.21.15.3:0/2541253961' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:10:27.235 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:27 smithi003 ceph-mon[30803]: from='mgr.? 172.21.15.3:0/2541253961' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-08-17T17:10:27.235 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:27 smithi003 ceph-mon[30803]: from='mgr.? 172.21.15.3:0/2541253961' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:10:27.380 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:27 smithi053 ceph-mon[34699]: [17/Aug/2024:17:10:25] ENGINE Bus STARTING 2024-08-17T17:10:27.380 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:27 smithi053 ceph-mon[34699]: [17/Aug/2024:17:10:25] ENGINE Serving on http://172.21.15.53:8765 2024-08-17T17:10:27.380 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:27 smithi053 ceph-mon[34699]: [17/Aug/2024:17:10:26] ENGINE Serving on https://172.21.15.53:7150 2024-08-17T17:10:27.380 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:27 smithi053 ceph-mon[34699]: [17/Aug/2024:17:10:26] ENGINE Bus STARTED 2024-08-17T17:10:27.380 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:27 smithi053 ceph-mon[34699]: [17/Aug/2024:17:10:26] ENGINE Error in HTTPServer.serve 2024-08-17T17:10:27.380 INFO:journalctl@ceph.mon.b.smithi053.stdout: Traceback (most recent call last): 2024-08-17T17:10:27.381 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:10:27.381 INFO:journalctl@ceph.mon.b.smithi053.stdout: self._connections.run(self.expiration_interval) 2024-08-17T17:10:27.381 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:10:27.381 INFO:journalctl@ceph.mon.b.smithi053.stdout: self._run(expiration_interval) 2024-08-17T17:10:27.381 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:10:27.381 INFO:journalctl@ceph.mon.b.smithi053.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:10:27.381 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:10:27.381 INFO:journalctl@ceph.mon.b.smithi053.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:10:27.381 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:10:27.381 INFO:journalctl@ceph.mon.b.smithi053.stdout: s = self.context.wrap_socket( 2024-08-17T17:10:27.381 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:10:27.381 INFO:journalctl@ceph.mon.b.smithi053.stdout: return self.sslsocket_class._create( 2024-08-17T17:10:27.381 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:10:27.381 INFO:journalctl@ceph.mon.b.smithi053.stdout: self.do_handshake() 2024-08-17T17:10:27.382 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:10:27.382 INFO:journalctl@ceph.mon.b.smithi053.stdout: self._sslobj.do_handshake() 2024-08-17T17:10:27.382 INFO:journalctl@ceph.mon.b.smithi053.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:10:27.382 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:27 smithi053 ceph-mon[34699]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:27.382 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:27 smithi053 ceph-mon[34699]: Standby manager daemon y started 2024-08-17T17:10:27.382 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:27 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.3:0/2541253961' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-08-17T17:10:27.382 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:27 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.3:0/2541253961' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:10:27.382 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:27 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.3:0/2541253961' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-08-17T17:10:27.382 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:27 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.3:0/2541253961' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:10:27.897 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'ceph orch upgrade pause' 2024-08-17T17:10:28.136 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:28 smithi003 ceph-mon[30803]: from='client.25546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:28.137 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:28 smithi003 ceph-mon[30803]: mgrmap e25: x(active, since 5s), standbys: y 2024-08-17T17:10:28.137 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:28 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-17T17:10:28.137 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:28 smithi003 ceph-mon[41530]: from='client.25546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:28.137 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:28 smithi003 ceph-mon[41530]: mgrmap e25: x(active, since 5s), standbys: y 2024-08-17T17:10:28.137 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:28 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-17T17:10:28.209 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:10:28.404 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:28 smithi053 ceph-mon[34699]: from='client.25546 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:28.404 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:28 smithi053 ceph-mon[34699]: mgrmap e25: x(active, since 5s), standbys: y 2024-08-17T17:10:28.404 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:28 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-17T17:10:29.073 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:29 smithi003 ceph-mon[41530]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 35 KiB/s rd, 614 B/s wr, 15 op/s 2024-08-17T17:10:29.074 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:29 smithi003 ceph-mon[30803]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 35 KiB/s rd, 614 B/s wr, 15 op/s 2024-08-17T17:10:29.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:29 smithi053 ceph-mon[34699]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 35 KiB/s rd, 614 B/s wr, 15 op/s 2024-08-17T17:10:30.687 INFO:teuthology.orchestra.run.smithi003.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:10:31.613 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:31 smithi003 ceph-mon[41530]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 438 B/s wr, 10 op/s 2024-08-17T17:10:31.613 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:31 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:31.613 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:31 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:31.613 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:31 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:31.614 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:31 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:31.614 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:31 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:31.614 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:31 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:31.614 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:31 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:31.614 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:31 smithi003 ceph-mon[30803]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 438 B/s wr, 10 op/s 2024-08-17T17:10:31.614 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:31 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:31.614 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:31 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:31.614 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:31 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:31.614 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:31 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:31.615 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:31 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:31.615 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:31 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:31.615 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:31 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:31.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'sleep 60' 2024-08-17T17:10:31.640 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:31 smithi053 ceph-mon[34699]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 438 B/s wr, 10 op/s 2024-08-17T17:10:31.640 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:31 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:31.640 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:31 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:31.640 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:31 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:31.640 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:31 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:31.640 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:31 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:31.640 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:31 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:31.641 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:31 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:31.895 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:10:32.676 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:32 smithi053 ceph-mon[34699]: from='client.25931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:32.677 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:32 smithi053 ceph-mon[34699]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:10:32.677 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:32 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:32.677 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:32 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:32.677 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:32 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:32.677 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:32 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:32.677 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:32 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:32.677 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:32 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:32.677 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:32 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:32.677 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:32 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:32.677 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:32 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:32.677 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:32 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:32.677 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:32 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:32.705 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:32 smithi003 ceph-mon[41530]: from='client.25931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:32.705 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:32 smithi003 ceph-mon[41530]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:10:32.705 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:32 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:32.705 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:32 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:32.705 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:32 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:32.705 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:32 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:32.706 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:32 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:32.706 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:32 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:32.706 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:32 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:32.706 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:32 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:32.706 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:32 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:32.706 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:32 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:32.706 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:32 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:32.706 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:32 smithi003 ceph-mon[30803]: from='client.25931 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:10:32.706 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:32 smithi003 ceph-mon[30803]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:10:32.706 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:32 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:32.706 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:32 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:32.707 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:32 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:32.707 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:32 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:32.707 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:32 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:32.707 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:32 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:32.707 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:32 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:32.707 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:32 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:32.707 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:32 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:32.707 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:32 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:32.707 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:32 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:33.273 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:33 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:10:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-08-17T17:10:33.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:33 smithi053 ceph-mon[34699]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 341 B/s wr, 8 op/s 2024-08-17T17:10:33.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:33 smithi053 ceph-mon[34699]: Adjusting osd_memory_target on smithi003 to 2481M 2024-08-17T17:10:33.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:33 smithi003 ceph-mon[30803]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 341 B/s wr, 8 op/s 2024-08-17T17:10:33.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:33 smithi003 ceph-mon[30803]: Adjusting osd_memory_target on smithi003 to 2481M 2024-08-17T17:10:33.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:33 smithi003 ceph-mon[41530]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 341 B/s wr, 8 op/s 2024-08-17T17:10:33.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:33 smithi003 ceph-mon[41530]: Adjusting osd_memory_target on smithi003 to 2481M 2024-08-17T17:10:35.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:35 smithi053 ceph-mon[34699]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 279 B/s wr, 6 op/s 2024-08-17T17:10:35.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:35 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:35.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:35 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:35.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:35 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:35.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:35 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:35.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:35 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:35.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:35 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:35.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:35 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:35.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:35 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:35.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:35 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:35.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:35 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:35.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:35 smithi053 ceph-mon[34699]: Adjusting osd_memory_target on smithi053 to 2481M 2024-08-17T17:10:35.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:35 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:35.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:35 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:10:35.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:35 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:10:35.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:35 smithi053 ceph-mon[34699]: Updating smithi003:/etc/ceph/ceph.conf 2024-08-17T17:10:35.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:35 smithi053 ceph-mon[34699]: Updating smithi053:/etc/ceph/ceph.conf 2024-08-17T17:10:35.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:35 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:35.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:35 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:35.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:35 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:35.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:35 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:35.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:35 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:35.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[41530]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 279 B/s wr, 6 op/s 2024-08-17T17:10:35.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:35.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:35.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:35.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:35.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:35.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:35.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:35.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:35.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:35.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:35.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[41530]: Adjusting osd_memory_target on smithi053 to 2481M 2024-08-17T17:10:35.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:35.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:10:35.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:10:35.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[41530]: Updating smithi003:/etc/ceph/ceph.conf 2024-08-17T17:10:35.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[41530]: Updating smithi053:/etc/ceph/ceph.conf 2024-08-17T17:10:35.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:35.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:35.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:35.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:35.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:35.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[30803]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 279 B/s wr, 6 op/s 2024-08-17T17:10:35.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:35.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:35.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:35.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:35.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:35.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:35.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:35.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:35.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:35.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-08-17T17:10:35.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[30803]: Adjusting osd_memory_target on smithi053 to 2481M 2024-08-17T17:10:35.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:35.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:10:35.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:10:35.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[30803]: Updating smithi003:/etc/ceph/ceph.conf 2024-08-17T17:10:35.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[30803]: Updating smithi053:/etc/ceph/ceph.conf 2024-08-17T17:10:35.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:35.527 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:35.527 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:35.527 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:35.527 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:35 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:36.466 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:36 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:10:36] "GET /metrics HTTP/1.1" 200 33488 "" "Prometheus/2.43.0" 2024-08-17T17:10:36.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:36 smithi053 ceph-mon[34699]: Updating smithi053:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:10:36.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:36 smithi053 ceph-mon[34699]: Updating smithi003:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:10:36.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:36 smithi053 ceph-mon[34699]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:10:36.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:36 smithi053 ceph-mon[34699]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:10:36.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:36 smithi053 ceph-mon[34699]: Updating smithi053:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:10:36.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:36 smithi053 ceph-mon[34699]: Updating smithi003:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:10:36.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:36 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:36.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:36 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:36.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:36 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:36.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:36 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:36.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:36 smithi053 ceph-mon[34699]: Reconfiguring alertmanager.a (dependencies changed)... 2024-08-17T17:10:36.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:36 smithi053 ceph-mon[34699]: Deploying daemon alertmanager.a on smithi003 2024-08-17T17:10:36.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:36 smithi003 ceph-mon[30803]: Updating smithi053:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:10:36.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:36 smithi003 ceph-mon[30803]: Updating smithi003:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:10:36.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:36 smithi003 ceph-mon[30803]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:10:36.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:36 smithi003 ceph-mon[30803]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:10:36.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:36 smithi003 ceph-mon[30803]: Updating smithi053:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:10:36.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:36 smithi003 ceph-mon[30803]: Updating smithi003:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:10:36.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:36 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:36.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:36 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:36.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:36 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:36.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:36 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:36.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:36 smithi003 ceph-mon[30803]: Reconfiguring alertmanager.a (dependencies changed)... 2024-08-17T17:10:36.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:36 smithi003 ceph-mon[30803]: Deploying daemon alertmanager.a on smithi003 2024-08-17T17:10:36.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:36 smithi003 ceph-mon[41530]: Updating smithi053:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:10:36.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:36 smithi003 ceph-mon[41530]: Updating smithi003:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:10:36.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:36 smithi003 ceph-mon[41530]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:10:36.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:36 smithi003 ceph-mon[41530]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:10:36.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:36 smithi003 ceph-mon[41530]: Updating smithi053:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:10:36.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:36 smithi003 ceph-mon[41530]: Updating smithi003:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:10:36.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:36 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:36.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:36 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:36.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:36 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:36.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:36 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:36.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:36 smithi003 ceph-mon[41530]: Reconfiguring alertmanager.a (dependencies changed)... 2024-08-17T17:10:36.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:36 smithi003 ceph-mon[41530]: Deploying daemon alertmanager.a on smithi003 2024-08-17T17:10:37.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:37 smithi003 ceph-mon[30803]: pgmap v10: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 6 op/s 2024-08-17T17:10:37.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:37 smithi003 ceph-mon[41530]: pgmap v10: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 6 op/s 2024-08-17T17:10:37.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:37 smithi053 ceph-mon[34699]: pgmap v10: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 6 op/s 2024-08-17T17:10:39.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:39 smithi003 ceph-mon[30803]: pgmap v11: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 6 op/s 2024-08-17T17:10:39.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:39 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:10:39.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:39 smithi003 ceph-mon[41530]: pgmap v11: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 6 op/s 2024-08-17T17:10:39.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:39 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:10:39.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:39 smithi053 ceph-mon[34699]: pgmap v11: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 255 B/s wr, 6 op/s 2024-08-17T17:10:39.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:39 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:10:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:41 smithi003 ceph-mon[30803]: pgmap v12: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:41.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:41 smithi003 ceph-mon[41530]: pgmap v12: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:41.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:41 smithi053 ceph-mon[34699]: pgmap v12: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:42.467 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:10:42 smithi003 systemd[1]: Stopping Ceph alertmanager.a for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:10:42.467 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:10:42 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a[74261]: ts=2024-08-17T17:10:42.405Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-08-17T17:10:42.757 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:10:42 smithi003 podman[88802]: 2024-08-17 17:10:42.467046239 +0000 UTC m=+0.155530464 container died be92e7808bcf8cf80ad5f6bfd25d176e41bee230964774fb79534150fa6abd18 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-08-17T17:10:43.023 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:10:42 smithi003 podman[88802]: 2024-08-17 17:10:42.847950686 +0000 UTC m=+0.536434900 container cleanup be92e7808bcf8cf80ad5f6bfd25d176e41bee230964774fb79534150fa6abd18 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-08-17T17:10:43.023 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:10:42 smithi003 bash[88802]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a 2024-08-17T17:10:43.419 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:43 smithi003 ceph-mon[30803]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:43.419 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:43 smithi003 ceph-mon[41530]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:43.420 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:10:43 smithi003 podman[88814]: 2024-08-17 17:10:43.124538141 +0000 UTC m=+0.651401423 container remove be92e7808bcf8cf80ad5f6bfd25d176e41bee230964774fb79534150fa6abd18 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-08-17T17:10:43.420 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:10:43 smithi003 podman[88814]: 2024-08-17 17:10:43.225218581 +0000 UTC m=+0.752081818 volume remove b73b516a3fa3119fc3cadb631209731b5b1e8036fb0a966098986130c85d8a58 2024-08-17T17:10:43.420 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:43 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:10:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-08-17T17:10:43.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:43 smithi053 ceph-mon[34699]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:43.773 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:10:43 smithi003 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-08-17T17:10:43.773 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:10:43 smithi003 systemd[1]: Stopped Ceph alertmanager.a for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:10:43.773 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:10:43 smithi003 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@alertmanager.a.service: Consumed 1.057s CPU time. 2024-08-17T17:10:43.773 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:10:43 smithi003 systemd[1]: Starting Ceph alertmanager.a for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:10:44.142 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:10:43 smithi003 podman[88945]: 2024-08-17 17:10:43.764692038 +0000 UTC m=+0.018138138 image pull 11f11916f8cdfb17c1da74b20deb2075d778357b4d60474d82453303f8d461f5 quay.io/prometheus/alertmanager:v0.27.0 2024-08-17T17:10:44.142 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:10:43 smithi003 podman[88945]: 2024-08-17 17:10:43.893355166 +0000 UTC m=+0.146801254 volume create e07e8079dd63183350a6f68d282814c973a555113d0cad74e48ade468cdd92be 2024-08-17T17:10:44.142 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:10:43 smithi003 podman[88945]: 2024-08-17 17:10:43.977003104 +0000 UTC m=+0.230449192 container create fb501c5aa873b83351542940e567ab8568f25fc1dfde32471c4a8e3bd3894cfe (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-08-17T17:10:44.523 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:10:44 smithi003 podman[88945]: 2024-08-17 17:10:44.238184042 +0000 UTC m=+0.491630132 container init fb501c5aa873b83351542940e567ab8568f25fc1dfde32471c4a8e3bd3894cfe (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-08-17T17:10:44.523 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:10:44 smithi003 podman[88945]: 2024-08-17 17:10:44.241504479 +0000 UTC m=+0.494950571 container start fb501c5aa873b83351542940e567ab8568f25fc1dfde32471c4a8e3bd3894cfe (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-08-17T17:10:44.523 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:10:44 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a[88960]: ts=2024-08-17T17:10:44.253Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)" 2024-08-17T17:10:44.524 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:10:44 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a[88960]: ts=2024-08-17T17:10:44.253Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)" 2024-08-17T17:10:44.524 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:10:44 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a[88960]: ts=2024-08-17T17:10:44.254Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.3 port=9094 2024-08-17T17:10:44.524 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:10:44 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a[88960]: ts=2024-08-17T17:10:44.256Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-08-17T17:10:44.524 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:10:44 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a[88960]: ts=2024-08-17T17:10:44.277Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-08-17T17:10:44.524 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:10:44 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a[88960]: ts=2024-08-17T17:10:44.277Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-08-17T17:10:44.524 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:10:44 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a[88960]: ts=2024-08-17T17:10:44.279Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093 2024-08-17T17:10:44.524 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:10:44 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a[88960]: ts=2024-08-17T17:10:44.279Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-08-17T17:10:44.524 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:10:44 smithi003 bash[88945]: fb501c5aa873b83351542940e567ab8568f25fc1dfde32471c4a8e3bd3894cfe 2024-08-17T17:10:44.524 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:10:44 smithi003 systemd[1]: Started Ceph alertmanager.a for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:10:45.023 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:44 smithi003 systemd[1]: Stopping Ceph node-exporter.a for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:10:45.476 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:45 smithi003 ceph-mon[41530]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:45.477 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:45 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:45.477 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:45 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:45.477 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:45 smithi003 podman[89123]: 2024-08-17 17:10:45.211649062 +0000 UTC m=+0.117057627 container died 6d9a244ec8ca400fd5c8ba934861a0c0605695a805bad9553a3f16e6e842f19a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-08-17T17:10:45.477 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:45 smithi003 ceph-mon[30803]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:45.477 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:45 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:45.477 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:45 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:45.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:45 smithi053 ceph-mon[34699]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:45.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:45 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:45.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:45 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:45.773 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:45 smithi003 podman[89123]: 2024-08-17 17:10:45.55972908 +0000 UTC m=+0.465137642 container cleanup 6d9a244ec8ca400fd5c8ba934861a0c0605695a805bad9553a3f16e6e842f19a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-08-17T17:10:45.773 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:45 smithi003 bash[89123]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a 2024-08-17T17:10:46.177 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:45 smithi003 podman[89137]: 2024-08-17 17:10:45.819395994 +0000 UTC m=+0.607528314 container remove 6d9a244ec8ca400fd5c8ba934861a0c0605695a805bad9553a3f16e6e842f19a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-08-17T17:10:46.177 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:45 smithi003 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-08-17T17:10:46.177 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:45 smithi003 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-08-17T17:10:46.177 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:45 smithi003 systemd[1]: Stopped Ceph node-exporter.a for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:10:46.178 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:45 smithi003 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@node-exporter.a.service: Consumed 2.200s CPU time. 2024-08-17T17:10:46.469 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:46 smithi003 ceph-mon[41530]: Reconfiguring node-exporter.a (dependencies changed)... 2024-08-17T17:10:46.469 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:46 smithi003 ceph-mon[41530]: Deploying daemon node-exporter.a on smithi003 2024-08-17T17:10:46.469 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:10:46 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a[88960]: ts=2024-08-17T17:10:46.256Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00017284s 2024-08-17T17:10:46.469 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:46 smithi003 systemd[1]: Starting Ceph node-exporter.a for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:10:46.469 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:46 smithi003 bash[89264]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-08-17T17:10:46.470 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:46 smithi003 ceph-mon[30803]: Reconfiguring node-exporter.a (dependencies changed)... 2024-08-17T17:10:46.470 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:46 smithi003 ceph-mon[30803]: Deploying daemon node-exporter.a on smithi003 2024-08-17T17:10:46.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:46 smithi053 ceph-mon[34699]: Reconfiguring node-exporter.a (dependencies changed)... 2024-08-17T17:10:46.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:46 smithi053 ceph-mon[34699]: Deploying daemon node-exporter.a on smithi003 2024-08-17T17:10:46.717 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:46 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:10:46] "GET /metrics HTTP/1.1" 200 33835 "" "Prometheus/2.43.0" 2024-08-17T17:10:47.023 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:46 smithi003 bash[89264]: Getting image source signatures 2024-08-17T17:10:47.023 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:46 smithi003 bash[89264]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-08-17T17:10:47.023 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:46 smithi003 bash[89264]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-08-17T17:10:47.023 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:46 smithi003 bash[89264]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-08-17T17:10:47.499 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:47 smithi003 ceph-mon[30803]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:47.500 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:47 smithi003 ceph-mon[41530]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:47.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:47 smithi053 ceph-mon[34699]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:48.523 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 bash[89264]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-08-17T17:10:48.523 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 bash[89264]: Writing manifest to image destination 2024-08-17T17:10:49.023 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 podman[89264]: 2024-08-17 17:10:48.63035504 +0000 UTC m=+2.343667540 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-08-17T17:10:49.023 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 podman[89264]: 2024-08-17 17:10:48.728706573 +0000 UTC m=+2.442019065 container create d19c9e3d51f8621030c7630545c94c8ef014e969991bd4504e8943aa6b5c7555 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-08-17T17:10:49.023 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 podman[89264]: 2024-08-17 17:10:48.931309397 +0000 UTC m=+2.644621889 container init d19c9e3d51f8621030c7630545c94c8ef014e969991bd4504e8943aa6b5c7555 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-08-17T17:10:49.023 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 podman[89264]: 2024-08-17 17:10:48.934901455 +0000 UTC m=+2.648213955 container start d19c9e3d51f8621030c7630545c94c8ef014e969991bd4504e8943aa6b5c7555 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-08-17T17:10:49.024 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.938Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-08-17T17:10:49.024 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.938Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-08-17T17:10:49.024 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.939Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-08-17T17:10:49.024 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.939Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-08-17T17:10:49.024 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-08-17T17:10:49.024 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-08-17T17:10:49.024 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-08-17T17:10:49.024 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=arp 2024-08-17T17:10:49.024 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=bcache 2024-08-17T17:10:49.024 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=bonding 2024-08-17T17:10:49.025 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=btrfs 2024-08-17T17:10:49.025 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=conntrack 2024-08-17T17:10:49.025 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=cpu 2024-08-17T17:10:49.025 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-08-17T17:10:49.025 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=diskstats 2024-08-17T17:10:49.025 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=dmi 2024-08-17T17:10:49.025 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=edac 2024-08-17T17:10:49.025 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=entropy 2024-08-17T17:10:49.026 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-08-17T17:10:49.026 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=filefd 2024-08-17T17:10:49.026 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=filesystem 2024-08-17T17:10:49.026 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=hwmon 2024-08-17T17:10:49.026 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=infiniband 2024-08-17T17:10:49.026 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=ipvs 2024-08-17T17:10:49.026 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=loadavg 2024-08-17T17:10:49.026 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=mdadm 2024-08-17T17:10:49.026 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=meminfo 2024-08-17T17:10:49.026 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=netclass 2024-08-17T17:10:49.026 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=netdev 2024-08-17T17:10:49.026 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=netstat 2024-08-17T17:10:49.027 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=nfs 2024-08-17T17:10:49.027 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=nfsd 2024-08-17T17:10:49.027 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=nvme 2024-08-17T17:10:49.027 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=os 2024-08-17T17:10:49.027 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-08-17T17:10:49.027 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=pressure 2024-08-17T17:10:49.027 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=rapl 2024-08-17T17:10:49.027 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=schedstat 2024-08-17T17:10:49.027 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=selinux 2024-08-17T17:10:49.027 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=sockstat 2024-08-17T17:10:49.027 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=softnet 2024-08-17T17:10:49.027 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=stat 2024-08-17T17:10:49.027 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=tapestats 2024-08-17T17:10:49.027 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=textfile 2024-08-17T17:10:49.028 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-08-17T17:10:49.028 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=time 2024-08-17T17:10:49.028 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.940Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-08-17T17:10:49.028 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.941Z caller=node_exporter.go:117 level=info collector=uname 2024-08-17T17:10:49.028 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.941Z caller=node_exporter.go:117 level=info collector=vmstat 2024-08-17T17:10:49.028 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.941Z caller=node_exporter.go:117 level=info collector=xfs 2024-08-17T17:10:49.028 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.941Z caller=node_exporter.go:117 level=info collector=zfs 2024-08-17T17:10:49.028 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.941Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-08-17T17:10:49.028 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a[89325]: ts=2024-08-17T17:10:48.941Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-08-17T17:10:49.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:49 smithi003 ceph-mon[41530]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:49.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:49 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:49.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:49 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:49.523 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:49 smithi003 bash[89264]: d19c9e3d51f8621030c7630545c94c8ef014e969991bd4504e8943aa6b5c7555 2024-08-17T17:10:49.524 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:10:49 smithi003 systemd[1]: Started Ceph node-exporter.a for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:10:49.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:49 smithi003 ceph-mon[30803]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:49.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:49 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:49.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:49 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:49.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:49 smithi053 ceph-mon[34699]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:49.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:49 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:49.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:49 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:10:50.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:50 smithi003 ceph-mon[30803]: Reconfiguring prometheus.a (dependencies changed)... 2024-08-17T17:10:50.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:50 smithi003 ceph-mon[41530]: Reconfiguring prometheus.a (dependencies changed)... 2024-08-17T17:10:50.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:50 smithi053 ceph-mon[34699]: Reconfiguring prometheus.a (dependencies changed)... 2024-08-17T17:10:51.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:51 smithi003 ceph-mon[30803]: Deploying daemon prometheus.a on smithi053 2024-08-17T17:10:51.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:51 smithi003 ceph-mon[30803]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:51.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:51 smithi003 ceph-mon[41530]: Deploying daemon prometheus.a on smithi053 2024-08-17T17:10:51.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:51 smithi003 ceph-mon[41530]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:51.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:51 smithi053 ceph-mon[34699]: Deploying daemon prometheus.a on smithi053 2024-08-17T17:10:51.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:51 smithi053 ceph-mon[34699]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:53.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:53 smithi003 ceph-mon[30803]: pgmap v18: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:53.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:53 smithi003 ceph-mon[41530]: pgmap v18: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:53.523 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:10:53 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[31030]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:10:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-08-17T17:10:53.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:53 smithi053 ceph-mon[34699]: pgmap v18: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:54.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:54 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:10:54.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:54 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:10:54.523 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:10:54 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a[88960]: ts=2024-08-17T17:10:54.259Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003802718s 2024-08-17T17:10:54.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:54 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:10:55.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:55 smithi003 ceph-mon[30803]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:55.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:55 smithi003 ceph-mon[41530]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:55.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:55 smithi053 ceph-mon[34699]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:56.716 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:10:56 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:10:56] "GET /metrics HTTP/1.1" 200 33836 "" "Prometheus/2.43.0" 2024-08-17T17:10:57.514 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:57 smithi053 ceph-mon[34699]: pgmap v20: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:57.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:57 smithi003 ceph-mon[30803]: pgmap v20: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:57.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:57 smithi003 ceph-mon[41530]: pgmap v20: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:59.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:10:59 smithi003 ceph-mon[30803]: pgmap v21: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:59.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:10:59 smithi003 ceph-mon[41530]: pgmap v21: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:10:59.551 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:10:59 smithi053 ceph-mon[34699]: pgmap v21: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:00.846 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:00 smithi053 systemd[1]: Stopping Ceph prometheus.a for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:11:00.846 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:11:00.835Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-08-17T17:11:00.846 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:11:00.835Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-08-17T17:11:00.846 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:11:00.835Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-08-17T17:11:00.846 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:11:00.835Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-08-17T17:11:00.846 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:11:00.835Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-08-17T17:11:00.846 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:11:00.835Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-08-17T17:11:00.846 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:11:00.835Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-08-17T17:11:00.846 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:11:00.835Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-08-17T17:11:00.846 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:11:00.835Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-08-17T17:11:00.846 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:11:00.835Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-08-17T17:11:00.846 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:11:00.835Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-08-17T17:11:00.847 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[64655]: ts=2024-08-17T17:11:00.836Z caller=main.go:1147 level=info msg="See you next time!" 2024-08-17T17:11:00.847 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:00 smithi053 podman[76192]: 2024-08-17 17:11:00.845209789 +0000 UTC m=+0.104403107 container died af8f221d5152dc7cb9656ddb848de6660580632eefe976f733106312230e0add (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:11:01.514 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:01 smithi053 ceph-mon[34699]: pgmap v22: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:01.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:01 smithi003 ceph-mon[30803]: pgmap v22: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:01.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:01 smithi003 ceph-mon[41530]: pgmap v22: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:02.716 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:02 smithi053 podman[76192]: 2024-08-17 17:11:02.355204972 +0000 UTC m=+1.614398287 container cleanup af8f221d5152dc7cb9656ddb848de6660580632eefe976f733106312230e0add (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:11:02.716 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:02 smithi053 bash[76192]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a 2024-08-17T17:11:03.452 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:03 smithi053 podman[76207]: 2024-08-17 17:11:03.198631104 +0000 UTC m=+2.355357782 container remove af8f221d5152dc7cb9656ddb848de6660580632eefe976f733106312230e0add (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:11:03.452 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:03 smithi053 ceph-mon[34699]: pgmap v23: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:03.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:03 smithi003 ceph-mon[30803]: pgmap v23: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:03.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:03 smithi003 ceph-mon[41530]: pgmap v23: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:03.717 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:03 smithi053 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-08-17T17:11:03.717 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:03 smithi053 systemd[1]: Stopped Ceph prometheus.a for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:11:03.717 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:03 smithi053 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@prometheus.a.service: Consumed 1.675s CPU time. 2024-08-17T17:11:03.717 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:03 smithi053 systemd[1]: Starting Ceph prometheus.a for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:11:04.030 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:03 smithi053 podman[76334]: 2024-08-17 17:11:03.797980849 +0000 UTC m=+0.019334404 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-08-17T17:11:04.030 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:03 smithi053 podman[76334]: 2024-08-17 17:11:03.916681679 +0000 UTC m=+0.138035219 container create fb2a55f98a6f6c66b04a667a118ef6566a3cbbb8cc949837757803c38ea7cd83 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:11:04.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:04 smithi053 podman[76334]: 2024-08-17 17:11:04.118786006 +0000 UTC m=+0.340139562 container init fb2a55f98a6f6c66b04a667a118ef6566a3cbbb8cc949837757803c38ea7cd83 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:11:04.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:04 smithi053 podman[76334]: 2024-08-17 17:11:04.121965612 +0000 UTC m=+0.343319154 container start fb2a55f98a6f6c66b04a667a118ef6566a3cbbb8cc949837757803c38ea7cd83 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:11:04.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:04.142Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-08-17T17:11:04.468 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:04.142Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-08-17T17:11:04.468 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:04.142Z caller=main.go:623 level=info host_details="(Linux 5.14.0-480.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024 x86_64 smithi053 (none))" 2024-08-17T17:11:04.468 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:04.142Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-08-17T17:11:04.468 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:04.142Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-08-17T17:11:04.468 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:04.144Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-08-17T17:11:04.468 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:04.144Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-08-17T17:11:04.468 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:04.147Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-08-17T17:11:04.468 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:04.147Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-08-17T17:11:04.468 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:04.148Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-08-17T17:11:04.468 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:04.148Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.196µs 2024-08-17T17:11:04.468 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:04.148Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-08-17T17:11:04.468 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:04.148Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-08-17T17:11:04.468 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:04.161Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-08-17T17:11:04.469 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:04.161Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-08-17T17:11:04.469 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:04.161Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=38.119µs wal_replay_duration=13.145251ms wbl_replay_duration=131ns total_replay_duration=13.213847ms 2024-08-17T17:11:04.469 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:04.164Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-08-17T17:11:04.469 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:04.164Z caller=main.go:1153 level=info msg="TSDB started" 2024-08-17T17:11:04.469 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:04.164Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-08-17T17:11:04.469 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:04.181Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=17.315454ms db_storage=933ns remote_storage=1.14µs web_handler=361ns query_engine=679ns scrape=428.326µs scrape_sd=85.741µs notify=10.07µs notify_sd=8.302µs rules=16.410135ms tracing=8.858µs 2024-08-17T17:11:04.469 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:04.182Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-08-17T17:11:04.469 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:04.182Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-08-17T17:11:04.469 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:04 smithi053 bash[76334]: fb2a55f98a6f6c66b04a667a118ef6566a3cbbb8cc949837757803c38ea7cd83 2024-08-17T17:11:04.469 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:04 smithi053 systemd[1]: Started Ceph prometheus.a for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:11:05.451 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:05 smithi053 ceph-mon[34699]: pgmap v24: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:05.451 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:05 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:05.451 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:05 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:05.451 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:05 smithi053 ceph-mon[34699]: Reconfiguring grafana.a (dependencies changed)... 2024-08-17T17:11:05.452 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:05 smithi053 ceph-mon[34699]: Reconfiguring daemon grafana.a on smithi053 2024-08-17T17:11:05.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:05 smithi003 ceph-mon[30803]: pgmap v24: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:05.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:05 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:05.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:05 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:05.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:05 smithi003 ceph-mon[30803]: Reconfiguring grafana.a (dependencies changed)... 2024-08-17T17:11:05.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:05 smithi003 ceph-mon[30803]: Reconfiguring daemon grafana.a on smithi053 2024-08-17T17:11:05.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:05 smithi003 ceph-mon[41530]: pgmap v24: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:05.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:05 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:05.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:05 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:05.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:05 smithi003 ceph-mon[41530]: Reconfiguring grafana.a (dependencies changed)... 2024-08-17T17:11:05.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:05 smithi003 ceph-mon[41530]: Reconfiguring daemon grafana.a on smithi053 2024-08-17T17:11:07.466 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:07 smithi053 systemd[1]: Stopping Ceph grafana.a for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:11:07.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:07 smithi053 ceph-mon[34699]: pgmap v25: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:07.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:07 smithi003 ceph-mon[30803]: pgmap v25: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:07.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:07 smithi003 ceph-mon[41530]: pgmap v25: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:07.762 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:07 smithi053 podman[76570]: 2024-08-17 17:11:07.480281913 +0000 UTC m=+0.093051203 container died 2a28cf19852e4d4784e59112a4570d55f6f7405c3f24964efa868d503f543201 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, architecture=x86_64, com.redhat.component=ubi8-container, release=239.1651231664, maintainer=Paul Cuzner , build-date=2022-04-29T11:48:11.055897, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, io.buildah.version=1.32.0, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, vendor=Red Hat, Inc.) 2024-08-17T17:11:08.180 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:07 smithi053 podman[76570]: 2024-08-17 17:11:07.928185848 +0000 UTC m=+0.540955180 container cleanup 2a28cf19852e4d4784e59112a4570d55f6f7405c3f24964efa868d503f543201 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, vcs-type=git, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, architecture=x86_64, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, description=Ceph Grafana Container, release=239.1651231664, build-date=2022-04-29T11:48:11.055897, name=ubi8, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, maintainer=Paul Cuzner ) 2024-08-17T17:11:08.180 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:07 smithi053 bash[76570]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a 2024-08-17T17:11:08.466 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:08 smithi053 podman[76583]: 2024-08-17 17:11:08.179835394 +0000 UTC m=+0.701136830 container remove 2a28cf19852e4d4784e59112a4570d55f6f7405c3f24964efa868d503f543201 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=239.1651231664, name=ubi8, description=Ceph Grafana Container, architecture=x86_64, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-04-29T11:48:11.055897, distribution-scope=public, vendor=Red Hat, Inc., vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, io.buildah.version=1.32.0, io.openshift.expose-services=) 2024-08-17T17:11:08.466 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:08 smithi053 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-08-17T17:11:08.467 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:08 smithi053 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-08-17T17:11:08.467 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:08 smithi053 systemd[1]: Stopped Ceph grafana.a for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:11:08.467 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:08 smithi053 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@grafana.a.service: Consumed 5.333s CPU time. 2024-08-17T17:11:08.467 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:08 smithi053 systemd[1]: Starting Ceph grafana.a for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:11:08.833 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:08 smithi053 podman[76677]: 2024-08-17 17:11:08.436886819 +0000 UTC m=+0.017367093 image pull 954c08fa618887fe9a6fe572ae541197c77e625b05f4193df7898f5ac649ada9 quay.io/ceph/ceph-grafana:9.4.7 2024-08-17T17:11:08.834 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:08 smithi053 podman[76677]: 2024-08-17 17:11:08.556001652 +0000 UTC m=+0.136481922 container create 9b5ef97a5cd51ec719907596d8d5b699ce19b02274f6a48b5c907d7721ee5254 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a, release=239.1651231664, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, distribution-scope=public, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.buildah.version=1.32.0, architecture=x86_64, build-date=2022-04-29T11:48:11.055897, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, maintainer=Paul Cuzner , description=Ceph Grafana Container, io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, com.redhat.component=ubi8-container) 2024-08-17T17:11:09.112 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:08 smithi053 podman[76677]: 2024-08-17 17:11:08.833537635 +0000 UTC m=+0.414017917 container init 9b5ef97a5cd51ec719907596d8d5b699ce19b02274f6a48b5c907d7721ee5254 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a, release=239.1651231664, description=Ceph Grafana Container, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, io.openshift.expose-services=, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., vcs-type=git, name=ubi8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, architecture=x86_64) 2024-08-17T17:11:09.112 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:08 smithi053 podman[76677]: 2024-08-17 17:11:08.836521394 +0000 UTC m=+0.417001668 container start 9b5ef97a5cd51ec719907596d8d5b699ce19b02274f6a48b5c907d7721ee5254 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a, name=ubi8, architecture=x86_64, release=239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-04-29T11:48:11.055897, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.32.0, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, com.redhat.component=ubi8-container, io.openshift.expose-services=, distribution-scope=public, maintainer=Paul Cuzner , vendor=Red Hat, Inc.) 2024-08-17T17:11:09.112 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:08 smithi053 bash[76677]: 9b5ef97a5cd51ec719907596d8d5b699ce19b02274f6a48b5c907d7721ee5254 2024-08-17T17:11:09.112 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:08 smithi053 systemd[1]: Started Ceph grafana.a for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:11:09.112 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:08 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=settings t=2024-08-17T17:11:08.999094724Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-08-17T17:11:09.112 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:08 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=settings t=2024-08-17T17:11:08.999275249Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-08-17T17:11:09.112 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:08 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=settings t=2024-08-17T17:11:08.999296219Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-08-17T17:11:09.113 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:08 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=settings t=2024-08-17T17:11:08.999304521Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-08-17T17:11:09.113 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:08 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=settings t=2024-08-17T17:11:08.999311104Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-08-17T17:11:09.113 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:08 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=settings t=2024-08-17T17:11:08.999336559Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-08-17T17:11:09.113 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:08 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=settings t=2024-08-17T17:11:08.999343018Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-08-17T17:11:09.113 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:08 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=settings t=2024-08-17T17:11:08.999349398Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-08-17T17:11:09.113 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:08 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=settings t=2024-08-17T17:11:08.999356553Z level=info msg="Path Home" path=/usr/share/grafana 2024-08-17T17:11:09.113 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:08 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=settings t=2024-08-17T17:11:08.999363122Z level=info msg="Path Data" path=/var/lib/grafana 2024-08-17T17:11:09.113 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:08 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=settings t=2024-08-17T17:11:08.999369221Z level=info msg="Path Logs" path=/var/log/grafana 2024-08-17T17:11:09.113 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:08 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=settings t=2024-08-17T17:11:08.999375902Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-08-17T17:11:09.113 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:08 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=settings t=2024-08-17T17:11:08.999382268Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-08-17T17:11:09.113 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:08 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=settings t=2024-08-17T17:11:08.999388664Z level=info msg="App mode production" 2024-08-17T17:11:09.113 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:08 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=sqlstore t=2024-08-17T17:11:08.999440242Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-08-17T17:11:09.113 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:08 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=sqlstore t=2024-08-17T17:11:08.999459431Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-08-17T17:11:09.114 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=migrator t=2024-08-17T17:11:09.01386871Z level=info msg="Starting DB migrations" 2024-08-17T17:11:09.114 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=migrator t=2024-08-17T17:11:09.01849223Z level=info msg="migrations completed" performed=0 skipped=481 duration=419.109µs 2024-08-17T17:11:09.114 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=sqlstore t=2024-08-17T17:11:09.018874643Z level=info msg="Created default organization" 2024-08-17T17:11:09.114 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=plugin.loader t=2024-08-17T17:11:09.080357771Z level=info msg="Plugin registered" pluginID=input 2024-08-17T17:11:09.114 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=plugin.loader t=2024-08-17T17:11:09.111459009Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-08-17T17:11:09.114 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=plugin.loader t=2024-08-17T17:11:09.111494748Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-08-17T17:11:09.114 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=secrets t=2024-08-17T17:11:09.111533523Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-08-17T17:11:09.384 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=query_data t=2024-08-17T17:11:09.113651907Z level=info msg="Query Service initialization" 2024-08-17T17:11:09.384 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=live.push_http t=2024-08-17T17:11:09.116164669Z level=info msg="Live Push Gateway initialization" 2024-08-17T17:11:09.384 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:09 smithi053 ceph-mon[34699]: pgmap v26: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:09.384 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:09 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:11:09.385 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:09 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:09.385 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:09 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:09.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:09 smithi003 ceph-mon[30803]: pgmap v26: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:09.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:09 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:11:09.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:09 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:09.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:09 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:09 smithi003 ceph-mon[41530]: pgmap v26: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:09 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:11:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:09 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:09 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:09.716 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:09 smithi053 systemd[1]: Stopping Ceph node-exporter.b for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:11:10.158 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:09 smithi053 podman[76854]: 2024-08-17 17:11:09.874775785 +0000 UTC m=+0.141692831 container died 791e642c268d628edb7c965de3ba235301f5a3faedc02ea3a3adcb6e89a25c87 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-08-17T17:11:10.158 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=infra.usagestats.collector t=2024-08-17T17:11:09.904097418Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-08-17T17:11:10.159 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=provisioning.datasources t=2024-08-17T17:11:09.990350062Z level=info msg="deleted correlations based on configuration" ds_name=Dashboard1 2024-08-17T17:11:10.159 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=provisioning.datasources t=2024-08-17T17:11:09.990378361Z level=info msg="deleted datasource based on configuration" name=Dashboard1 2024-08-17T17:11:10.159 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=provisioning.datasources t=2024-08-17T17:11:10.000528858Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-08-17T17:11:10.159 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=provisioning.alerting t=2024-08-17T17:11:10.081843736Z level=info msg="starting to provision alerting" 2024-08-17T17:11:10.159 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=provisioning.alerting t=2024-08-17T17:11:10.081871173Z level=info msg="finished to provision alerting" 2024-08-17T17:11:10.159 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=grafanaStorageLogger t=2024-08-17T17:11:10.081989407Z level=info msg="storage starting" 2024-08-17T17:11:10.159 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=ngalert.state.manager t=2024-08-17T17:11:10.082153529Z level=info msg="Warming state cache for startup" 2024-08-17T17:11:10.159 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=http.server t=2024-08-17T17:11:10.083901506Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-08-17T17:11:10.159 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=ngalert.state.manager t=2024-08-17T17:11:10.093051089Z level=info msg="State cache has been initialized" states=0 duration=10.896274ms 2024-08-17T17:11:10.159 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=ngalert.multiorg.alertmanager t=2024-08-17T17:11:10.093111518Z level=info msg="starting MultiOrg Alertmanager" 2024-08-17T17:11:10.159 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:11:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=ticker t=2024-08-17T17:11:10.093136403Z level=info msg=starting first_tick=2024-08-17T17:11:20Z 2024-08-17T17:11:10.466 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:10 smithi053 podman[76854]: 2024-08-17 17:11:10.357215679 +0000 UTC m=+0.624132726 container cleanup 791e642c268d628edb7c965de3ba235301f5a3faedc02ea3a3adcb6e89a25c87 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-08-17T17:11:10.466 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:10 smithi053 bash[76854]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b 2024-08-17T17:11:10.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:10 smithi053 ceph-mon[34699]: Reconfiguring node-exporter.b (dependencies changed)... 2024-08-17T17:11:10.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:10 smithi053 ceph-mon[34699]: Deploying daemon node-exporter.b on smithi053 2024-08-17T17:11:10.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:10 smithi003 ceph-mon[30803]: Reconfiguring node-exporter.b (dependencies changed)... 2024-08-17T17:11:10.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:10 smithi003 ceph-mon[30803]: Deploying daemon node-exporter.b on smithi053 2024-08-17T17:11:10.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:10 smithi003 ceph-mon[41530]: Reconfiguring node-exporter.b (dependencies changed)... 2024-08-17T17:11:10.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:10 smithi003 ceph-mon[41530]: Deploying daemon node-exporter.b on smithi053 2024-08-17T17:11:10.953 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:10 smithi053 podman[76869]: 2024-08-17 17:11:10.583694969 +0000 UTC m=+0.712139334 container remove 791e642c268d628edb7c965de3ba235301f5a3faedc02ea3a3adcb6e89a25c87 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-08-17T17:11:10.953 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:10 smithi053 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-08-17T17:11:10.954 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:10 smithi053 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-08-17T17:11:10.954 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:10 smithi053 systemd[1]: Stopped Ceph node-exporter.b for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:11:10.954 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:10 smithi053 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@node-exporter.b.service: Consumed 2.127s CPU time. 2024-08-17T17:11:11.216 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:10 smithi053 systemd[1]: Starting Ceph node-exporter.b for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:11:11.216 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:11 smithi053 bash[76998]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-08-17T17:11:11.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:11 smithi003 ceph-mon[30803]: pgmap v27: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:11.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:11 smithi003 ceph-mon[41530]: pgmap v27: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:11.716 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:11 smithi053 bash[76998]: Getting image source signatures 2024-08-17T17:11:11.717 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:11 smithi053 bash[76998]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-08-17T17:11:11.717 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:11 smithi053 bash[76998]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-08-17T17:11:11.717 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:11 smithi053 bash[76998]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-08-17T17:11:11.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:11 smithi053 ceph-mon[34699]: pgmap v27: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:13.216 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:12 smithi053 bash[76998]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-08-17T17:11:13.216 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 bash[76998]: Writing manifest to image destination 2024-08-17T17:11:13.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:13 smithi003 ceph-mon[30803]: pgmap v28: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:13.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:13 smithi003 ceph-mon[41530]: pgmap v28: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:13.568 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-mon[34699]: pgmap v28: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:13.568 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 podman[76998]: 2024-08-17 17:11:13.369663432 +0000 UTC m=+2.317027984 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-08-17T17:11:13.568 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 podman[76998]: 2024-08-17 17:11:13.467902164 +0000 UTC m=+2.415266709 container create edfcc96e3983071081e38b1902b06f2a2f58b27e50531e67418ceb75810f9860 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-08-17T17:11:13.967 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 podman[76998]: 2024-08-17 17:11:13.687080898 +0000 UTC m=+2.634445444 container init edfcc96e3983071081e38b1902b06f2a2f58b27e50531e67418ceb75810f9860 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-08-17T17:11:13.967 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 podman[76998]: 2024-08-17 17:11:13.690701726 +0000 UTC m=+2.638066279 container start edfcc96e3983071081e38b1902b06f2a2f58b27e50531e67418ceb75810f9860 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-08-17T17:11:13.967 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.695Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-08-17T17:11:13.967 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.695Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-08-17T17:11:13.967 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.695Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-08-17T17:11:13.967 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.695Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-08-17T17:11:13.967 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-08-17T17:11:13.967 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-08-17T17:11:13.967 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-08-17T17:11:13.967 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=arp 2024-08-17T17:11:13.968 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=bcache 2024-08-17T17:11:13.968 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=bonding 2024-08-17T17:11:13.968 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=btrfs 2024-08-17T17:11:13.968 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=conntrack 2024-08-17T17:11:13.968 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=cpu 2024-08-17T17:11:13.968 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-08-17T17:11:13.968 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=diskstats 2024-08-17T17:11:13.969 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=dmi 2024-08-17T17:11:13.969 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=edac 2024-08-17T17:11:13.969 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=entropy 2024-08-17T17:11:13.969 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-08-17T17:11:13.969 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=filefd 2024-08-17T17:11:13.969 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=filesystem 2024-08-17T17:11:13.969 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=hwmon 2024-08-17T17:11:13.969 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=infiniband 2024-08-17T17:11:13.969 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=ipvs 2024-08-17T17:11:13.969 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=loadavg 2024-08-17T17:11:13.969 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=mdadm 2024-08-17T17:11:13.969 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=meminfo 2024-08-17T17:11:13.970 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=netclass 2024-08-17T17:11:13.970 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=netdev 2024-08-17T17:11:13.970 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=netstat 2024-08-17T17:11:13.970 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=nfs 2024-08-17T17:11:13.970 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=nfsd 2024-08-17T17:11:13.970 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=nvme 2024-08-17T17:11:13.970 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=os 2024-08-17T17:11:13.970 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-08-17T17:11:13.970 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=pressure 2024-08-17T17:11:13.970 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=rapl 2024-08-17T17:11:13.970 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=schedstat 2024-08-17T17:11:13.970 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=selinux 2024-08-17T17:11:13.970 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=sockstat 2024-08-17T17:11:13.971 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=softnet 2024-08-17T17:11:13.971 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=stat 2024-08-17T17:11:13.971 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=tapestats 2024-08-17T17:11:13.971 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=textfile 2024-08-17T17:11:13.971 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-08-17T17:11:13.971 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=time 2024-08-17T17:11:13.971 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-08-17T17:11:13.971 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=uname 2024-08-17T17:11:13.971 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=vmstat 2024-08-17T17:11:13.971 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=xfs 2024-08-17T17:11:13.971 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.696Z caller=node_exporter.go:117 level=info collector=zfs 2024-08-17T17:11:13.971 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.697Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-08-17T17:11:13.972 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b[77050]: ts=2024-08-17T17:11:13.697Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-08-17T17:11:13.972 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 bash[76998]: edfcc96e3983071081e38b1902b06f2a2f58b27e50531e67418ceb75810f9860 2024-08-17T17:11:13.972 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:11:13 smithi053 systemd[1]: Started Ceph node-exporter.b for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:11:15.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:14 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:15.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:14 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:15.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:14 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-08-17T17:11:15.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:14 smithi053 ceph-mon[34699]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-08-17T17:11:15.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:14 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:11:15.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:14 smithi053 ceph-mon[34699]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:11:15.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:14 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-08-17T17:11:15.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:14 smithi053 ceph-mon[34699]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-08-17T17:11:15.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:14 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:11:15.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:14 smithi053 ceph-mon[34699]: pgmap v29: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:15.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:14 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:15.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:14 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:15.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:14 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-08-17T17:11:15.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:14 smithi003 ceph-mon[30803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-08-17T17:11:15.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:14 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:11:15.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:14 smithi003 ceph-mon[30803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:11:15.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:14 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-08-17T17:11:15.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:14 smithi003 ceph-mon[30803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-08-17T17:11:15.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:14 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:11:15.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:14 smithi003 ceph-mon[30803]: pgmap v29: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:15.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:14 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:15.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:14 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:15.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:14 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-08-17T17:11:15.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:14 smithi003 ceph-mon[41530]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-08-17T17:11:15.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:14 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:11:15.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:14 smithi003 ceph-mon[41530]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:11:15.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:14 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-08-17T17:11:15.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:14 smithi003 ceph-mon[41530]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-08-17T17:11:15.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:14 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:11:15.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:14 smithi003 ceph-mon[41530]: pgmap v29: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:17.066 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:16 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:16.950Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi053\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:11:17.466 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:11:17 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:11:17] "GET /metrics HTTP/1.1" 200 33837 "" "Prometheus/2.51.0" 2024-08-17T17:11:17.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:17 smithi053 ceph-mon[34699]: pgmap v30: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:17.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:17 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:17.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:17 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:17.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:17 smithi003 ceph-mon[30803]: pgmap v30: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:17.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:17 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:17.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:17 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:17.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:17 smithi003 ceph-mon[41530]: pgmap v30: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:17.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:17 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:17.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:17 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:19.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:18 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:19.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:18 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:19.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:18 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:11:19.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:18 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:11:19.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:18 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:19.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:18 smithi003 ceph-mon[30803]: pgmap v31: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:19.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:18 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:19.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:18 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:19.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:18 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:11:19.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:18 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:11:19.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:18 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:19.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:18 smithi003 ceph-mon[41530]: pgmap v31: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:19.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:18 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:19.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:18 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:19.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:18 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:11:19.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:18 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:11:19.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:18 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:19.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:18 smithi053 ceph-mon[34699]: pgmap v31: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:21.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:21 smithi053 ceph-mon[34699]: pgmap v32: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:21.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:21 smithi003 ceph-mon[30803]: pgmap v32: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:21.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:21 smithi003 ceph-mon[41530]: pgmap v32: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:23.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:23 smithi053 ceph-mon[34699]: pgmap v33: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:23.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:23 smithi003 ceph-mon[30803]: pgmap v33: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:23.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:23 smithi003 ceph-mon[41530]: pgmap v33: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:24.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:24 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:11:24.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:24 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:24.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:11:24.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:24 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:11:24.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:24 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:11:25.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:25 smithi053 ceph-mon[34699]: pgmap v34: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:25.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:25 smithi003 ceph-mon[30803]: pgmap v34: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:25.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:25 smithi003 ceph-mon[41530]: pgmap v34: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:26.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:25 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:25.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:11:26.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:25 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:25.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:11:27.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:27 smithi053 ceph-mon[34699]: pgmap v35: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:27.217 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:11:27 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:11:27] "GET /metrics HTTP/1.1" 200 33832 "" "Prometheus/2.51.0" 2024-08-17T17:11:27.217 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:26.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi053\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:11:27.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:27 smithi003 ceph-mon[30803]: pgmap v35: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:27.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:27 smithi003 ceph-mon[41530]: pgmap v35: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:29.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:29 smithi053 ceph-mon[34699]: pgmap v36: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:29.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:29 smithi003 ceph-mon[41530]: pgmap v36: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:29.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:29 smithi003 ceph-mon[30803]: pgmap v36: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:30.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:11:31.368 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:31 smithi053 ceph-mon[34699]: pgmap v37: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:31.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:31 smithi003 ceph-mon[30803]: pgmap v37: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:31.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:31 smithi003 ceph-mon[41530]: pgmap v37: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:33.399 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:33 smithi003 ceph-mon[41530]: pgmap v38: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:33.400 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:33 smithi003 ceph-mon[30803]: pgmap v38: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:33.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:33 smithi053 ceph-mon[34699]: pgmap v38: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:34.013 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'ceph orch upgrade resume' 2024-08-17T17:11:34.291 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:11:34.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:34 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:34.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:11:35.441 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:35 smithi003 ceph-mon[41530]: pgmap v39: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:35.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:35 smithi003 ceph-mon[30803]: pgmap v39: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:35.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:35 smithi053 ceph-mon[34699]: pgmap v39: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:36.139 INFO:teuthology.orchestra.run.smithi003.stdout:Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:11:36.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:35 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:35.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:11:36.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:35 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:35.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:11:36.857 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'echo wait for majority of mons to upgrade' 2024-08-17T17:11:37.066 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:36 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:36.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi053\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:11:37.347 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:11:37 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:11:37] "GET /metrics HTTP/1.1" 200 33832 "" "Prometheus/2.51.0" 2024-08-17T17:11:37.348 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:37 smithi053 ceph-mon[34699]: pgmap v40: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:37.348 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:37 smithi053 ceph-mon[34699]: from='client.16032 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:37.348 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:37 smithi053 ceph-mon[34699]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:11:37.348 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:37 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:37.348 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:37 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:11:37.441 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:37 smithi003 ceph-mon[41530]: pgmap v40: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:37.441 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:37 smithi003 ceph-mon[41530]: from='client.16032 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:37.441 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:37 smithi003 ceph-mon[41530]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:11:37.441 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:37 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:37.442 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:37 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:11:37.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:37 smithi003 ceph-mon[30803]: pgmap v40: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:37.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:37 smithi003 ceph-mon[30803]: from='client.16032 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:37.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:37 smithi003 ceph-mon[30803]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:11:37.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:37 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:37.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:37 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:11:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:11:39.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:39 smithi053 ceph-mon[34699]: pgmap v41: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:39.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:39 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:11:39.245 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:39 smithi003 ceph-mon[41530]: pgmap v41: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:39.245 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:39 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:11:39.245 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:39 smithi003 ceph-mon[30803]: pgmap v41: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:39.245 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:39 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:11:39.440 INFO:teuthology.orchestra.run.smithi003.stdout:wait for majority of mons to upgrade 2024-08-17T17:11:40.202 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'while ! ceph mon versions | grep $sha1 | egrep '"'"': [23]'"'"' && ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error; do sleep 2 ; done' 2024-08-17T17:11:40.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:11:40.919 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:11:41.469 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:41 smithi053 ceph-mon[34699]: pgmap v42: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:41.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:41 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:41.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:41 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:41 smithi003 ceph-mon[30803]: pgmap v42: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:41 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:41 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:41.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:41 smithi003 ceph-mon[41530]: pgmap v42: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:41.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:41 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:41.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:41 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:42.986 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:42 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:42.986 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:42 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:42.986 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:42 smithi003 ceph-mon[41530]: pgmap v43: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:42.986 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:42 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:11:42.986 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:42 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:11:42.986 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:42 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:42.986 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:42 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:11:42.987 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:42 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:42.987 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:42 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:42.987 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:42 smithi003 ceph-mon[30803]: pgmap v43: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:42.987 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:42 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:11:42.987 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:42 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:11:42.987 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:42 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:42.987 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:42 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:11:43.115 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:42 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:43.115 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:42 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:43.115 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:42 smithi053 ceph-mon[34699]: pgmap v43: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:43.115 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:42 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:11:43.115 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:42 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:11:43.115 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:42 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:43.116 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:42 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:11:43.887 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:11:43.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:43 smithi053 ceph-mon[34699]: Upgrade: Need to upgrade myself (mgr.x) 2024-08-17T17:11:43.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:43 smithi053 ceph-mon[34699]: Upgrade: Need to upgrade myself (mgr.x) 2024-08-17T17:11:43.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:43 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/4207581643' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:11:44.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:43 smithi003 ceph-mon[41530]: Upgrade: Need to upgrade myself (mgr.x) 2024-08-17T17:11:44.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:43 smithi003 ceph-mon[41530]: Upgrade: Need to upgrade myself (mgr.x) 2024-08-17T17:11:44.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:43 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/4207581643' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:11:44.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:43 smithi003 ceph-mon[30803]: Upgrade: Need to upgrade myself (mgr.x) 2024-08-17T17:11:44.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:43 smithi003 ceph-mon[30803]: Upgrade: Need to upgrade myself (mgr.x) 2024-08-17T17:11:44.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:43 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/4207581643' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:11:44.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:44 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:44.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:11:45.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:44 smithi003 ceph-mon[41530]: from='client.25949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:45.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:44 smithi003 ceph-mon[41530]: pgmap v44: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:45.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:44 smithi003 ceph-mon[41530]: from='client.16050 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:45.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:44 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:45.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:44 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:11:45.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:44 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:11:45.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:44 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-17T17:11:45.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:44 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:11:45.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:44 smithi003 ceph-mon[30803]: from='client.25949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:45.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:44 smithi003 ceph-mon[30803]: pgmap v44: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:45.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:44 smithi003 ceph-mon[30803]: from='client.16050 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:45.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:44 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:45.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:44 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:11:45.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:44 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:11:45.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:44 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-17T17:11:45.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:44 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:11:45.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:44 smithi053 ceph-mon[34699]: from='client.25949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:45.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:44 smithi053 ceph-mon[34699]: pgmap v44: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:45.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:44 smithi053 ceph-mon[34699]: from='client.16050 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:45.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:44 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:45.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:44 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:11:45.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:44 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:11:45.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:44 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-17T17:11:45.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:44 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:11:45.950 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:45 smithi003 ceph-mon[30803]: Upgrade: Updating mgr.y 2024-08-17T17:11:45.950 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:45 smithi003 ceph-mon[30803]: Deploying daemon mgr.y on smithi003 2024-08-17T17:11:45.950 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:45 smithi003 ceph-mon[41530]: Upgrade: Updating mgr.y 2024-08-17T17:11:45.950 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:45 smithi003 ceph-mon[41530]: Deploying daemon mgr.y on smithi003 2024-08-17T17:11:46.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:45 smithi053 ceph-mon[34699]: Upgrade: Updating mgr.y 2024-08-17T17:11:46.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:45 smithi053 ceph-mon[34699]: Deploying daemon mgr.y on smithi003 2024-08-17T17:11:46.217 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:45 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:45.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:11:46.217 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:45 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:45.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:11:46.273 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:45 smithi003 systemd[1]: Stopping Ceph mgr.y for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:11:46.559 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:46 smithi003 podman[91218]: 2024-08-17 17:11:46.289308805 +0000 UTC m=+0.153561879 container died 8cb7113bf953089166faaac456449b9d60ed53dc66b49333d89a0b40b6dad626 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=quincy-f2c39d4, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0) 2024-08-17T17:11:46.835 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:11:46.835 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:46 smithi003 podman[91218]: 2024-08-17 17:11:46.724224739 +0000 UTC m=+0.588477813 container cleanup 8cb7113bf953089166faaac456449b9d60ed53dc66b49333d89a0b40b6dad626 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, RELEASE=quincy-f2c39d4, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:11:46.836 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:46 smithi003 bash[91218]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y 2024-08-17T17:11:47.209 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:46 smithi003 ceph-mon[41530]: pgmap v45: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:47.209 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:46 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/1189518307' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:11:47.209 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:46 smithi003 ceph-mon[30803]: pgmap v45: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:47.209 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:46 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/1189518307' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:11:47.209 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:46 smithi003 podman[91254]: 2024-08-17 17:11:46.950981305 +0000 UTC m=+0.663933397 container remove 8cb7113bf953089166faaac456449b9d60ed53dc66b49333d89a0b40b6dad626 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, RELEASE=quincy-f2c39d4, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:11:47.209 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:46 smithi003 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-08-17T17:11:47.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:46 smithi053 ceph-mon[34699]: pgmap v45: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:47.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:46 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/1189518307' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:11:47.216 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:11:47 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:11:47] "GET /metrics HTTP/1.1" 200 33834 "" "Prometheus/2.51.0" 2024-08-17T17:11:47.217 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:46 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:46.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi053\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:11:47.523 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:47 smithi003 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mgr.y.service: Failed with result 'exit-code'. 2024-08-17T17:11:47.523 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:47 smithi003 systemd[1]: Stopped Ceph mgr.y for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:11:47.523 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:47 smithi003 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mgr.y.service: Consumed 45.560s CPU time. 2024-08-17T17:11:47.523 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:47 smithi003 systemd[1]: Starting Ceph mgr.y for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:11:47.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:47 smithi003 ceph-mon[30803]: from='client.16062 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:47.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:47 smithi003 ceph-mon[30803]: from='client.25973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:47.939 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:47 smithi003 ceph-mon[41530]: from='client.16062 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:47.939 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:47 smithi003 ceph-mon[41530]: from='client.25973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:47.939 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:47 smithi003 podman[91431]: 2024-08-17 17:11:47.56911875 +0000 UTC m=+0.017821633 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:11:47.939 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:47 smithi003 podman[91431]: 2024-08-17 17:11:47.67719356 +0000 UTC m=+0.125896478 container create f491ea24df2839cb7e000efbd2f7292d509d2f75f07e408387955bdab5c29907 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-cd8bdbe, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-08-17T17:11:47.939 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:47 smithi003 podman[91431]: 2024-08-17 17:11:47.93840939 +0000 UTC m=+0.387112264 container init f491ea24df2839cb7e000efbd2f7292d509d2f75f07e408387955bdab5c29907 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-cd8bdbe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-08-17T17:11:48.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:47 smithi053 ceph-mon[34699]: from='client.16062 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:48.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:47 smithi053 ceph-mon[34699]: from='client.25973 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:48.273 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:47 smithi003 podman[91431]: 2024-08-17 17:11:47.941393315 +0000 UTC m=+0.390096189 container start f491ea24df2839cb7e000efbd2f7292d509d2f75f07e408387955bdab5c29907 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-cd8bdbe, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-08-17T17:11:48.273 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:47 smithi003 ceph-mgr[91450]: -- 172.21.15.3:0/630817019 <== mon.0 v2:172.21.15.3:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dd203650e0 con 0x55dd20343000 2024-08-17T17:11:48.273 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:48 smithi003 bash[91431]: f491ea24df2839cb7e000efbd2f7292d509d2f75f07e408387955bdab5c29907 2024-08-17T17:11:48.273 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:48 smithi003 systemd[1]: Started Ceph mgr.y for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:11:49.406 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:49 smithi053 ceph-mon[34699]: pgmap v46: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:49.406 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:49 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:49.406 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:49 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:49.407 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:49 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:11:49.407 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:49 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:49.407 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:49 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:49.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:49 smithi003 ceph-mon[41530]: pgmap v46: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:49.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:49 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:49.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:49 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:49.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:49 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:11:49.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:49 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:49.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:49 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:49.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:49 smithi003 ceph-mon[30803]: pgmap v46: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:49.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:49 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:49.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:49 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:49.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:49 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:11:49.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:49 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:49.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:49 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:49.817 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:11:50.408 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:50 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/2831866981' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:11:50.408 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:11:50.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:50 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/2831866981' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:11:50.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:50 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/2831866981' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:11:51.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:51 smithi053 ceph-mon[34699]: from='client.25991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:51.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:51 smithi053 ceph-mon[34699]: pgmap v47: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:51.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:51 smithi053 ceph-mon[34699]: from='client.25997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:51.511 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:51 smithi003 ceph-mon[30803]: from='client.25991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:51.512 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:51 smithi003 ceph-mon[30803]: pgmap v47: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:51.512 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:51 smithi003 ceph-mon[30803]: from='client.25997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:51.512 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:51 smithi003 ceph-mon[41530]: from='client.25991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:51.512 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:51 smithi003 ceph-mon[41530]: pgmap v47: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:51.512 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:51 smithi003 ceph-mon[41530]: from='client.25997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:52.808 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:11:53.073 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:52 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:53.073 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:52 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:53.074 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:52 smithi003 ceph-mon[41530]: pgmap v48: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:53.074 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:52 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:53.074 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:52 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:53.074 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:52 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/3219103806' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:11:53.074 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:52 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:53.074 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:52 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:53.074 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:52 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:53.074 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:52 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:53.075 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:52 smithi003 ceph-mon[30803]: pgmap v48: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:53.075 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:52 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:53.075 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:52 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:53.075 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:52 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/3219103806' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:11:53.075 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:52 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:53.075 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:52 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:53.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:52 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:53.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:52 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:53.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:52 smithi053 ceph-mon[34699]: pgmap v48: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:53.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:52 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:53.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:52 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:53.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:52 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/3219103806' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:11:53.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:52 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:53.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:52 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:53.599 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:53 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: [17/Aug/2024:17:11:53] ENGINE Bus STARTING 2024-08-17T17:11:53.599 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:53 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: CherryPy Checker: 2024-08-17T17:11:53.599 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:53 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: The Application mounted at '' has an empty config. 2024-08-17T17:11:53.599 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:53 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:11:53.600 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:53 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: [17/Aug/2024:17:11:53] ENGINE Serving on http://:::9283 2024-08-17T17:11:53.962 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:53 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: [17/Aug/2024:17:11:53] ENGINE Bus STARTED 2024-08-17T17:11:54.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:54.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:11:54.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:53 smithi053 ceph-mon[34699]: from='client.26009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:54.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:53 smithi053 ceph-mon[34699]: from='client.16110 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:54.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:53 smithi053 ceph-mon[34699]: Standby manager daemon y restarted 2024-08-17T17:11:54.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:53 smithi053 ceph-mon[34699]: Standby manager daemon y started 2024-08-17T17:11:54.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:53 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-08-17T17:11:54.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:53 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:11:54.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:53 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-08-17T17:11:54.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:53 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:11:54.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:53 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:11:54.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:53 smithi003 ceph-mon[41530]: from='client.26009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:54.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:53 smithi003 ceph-mon[41530]: from='client.16110 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:54.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:53 smithi003 ceph-mon[41530]: Standby manager daemon y restarted 2024-08-17T17:11:54.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:53 smithi003 ceph-mon[41530]: Standby manager daemon y started 2024-08-17T17:11:54.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:53 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-08-17T17:11:54.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:53 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:11:54.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:53 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-08-17T17:11:54.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:53 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:11:54.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:53 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:11:54.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:53 smithi003 ceph-mon[30803]: from='client.26009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:54.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:53 smithi003 ceph-mon[30803]: from='client.16110 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:54.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:53 smithi003 ceph-mon[30803]: Standby manager daemon y restarted 2024-08-17T17:11:54.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:53 smithi003 ceph-mon[30803]: Standby manager daemon y started 2024-08-17T17:11:54.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:53 smithi003 ceph-mon[30803]: from='mgr.? 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-08-17T17:11:54.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:53 smithi003 ceph-mon[30803]: from='mgr.? 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:11:54.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:53 smithi003 ceph-mon[30803]: from='mgr.? 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-08-17T17:11:54.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:53 smithi003 ceph-mon[30803]: from='mgr.? 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:11:54.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:53 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:11:55.264 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:54 smithi003 ceph-mon[41530]: mgrmap e26: x(active, since 91s), standbys: y 2024-08-17T17:11:55.264 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:54 smithi003 ceph-mon[41530]: pgmap v49: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:55.265 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:54 smithi003 ceph-mon[30803]: mgrmap e26: x(active, since 91s), standbys: y 2024-08-17T17:11:55.265 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:54 smithi003 ceph-mon[30803]: pgmap v49: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:55.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:54 smithi053 ceph-mon[34699]: mgrmap e26: x(active, since 91s), standbys: y 2024-08-17T17:11:55.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:54 smithi053 ceph-mon[34699]: pgmap v49: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:55.850 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:11:56.187 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:55.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:11:56.188 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:55.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:11:56.188 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:55 smithi053 ceph-mon[34699]: mgrmap e27: x(active, since 92s), standbys: y 2024-08-17T17:11:56.188 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:55 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/2047908419' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:11:56.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:55 smithi003 ceph-mon[30803]: mgrmap e27: x(active, since 92s), standbys: y 2024-08-17T17:11:56.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:55 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/2047908419' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:11:56.274 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:56 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: [17/Aug/2024:17:11:56] ENGINE Bus STOPPING 2024-08-17T17:11:56.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:55 smithi003 ceph-mon[41530]: mgrmap e27: x(active, since 92s), standbys: y 2024-08-17T17:11:56.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:55 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/2047908419' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:11:56.466 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:11:56 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: 2024-08-17T17:11:56.185+0000 7f9cf616d640 -1 mgr handle_mgr_map I was active but no longer am 2024-08-17T17:11:56.466 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:11:56 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: ignoring --setuser ceph since I am not root 2024-08-17T17:11:56.466 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:11:56 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: ignoring --setgroup ceph since I am not root 2024-08-17T17:11:56.467 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:11:56 smithi053 ceph-mgr[67427]: -- 172.21.15.53:0/2720104803 <== mon.1 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55aa279210e0 con 0x55aa278ff400 2024-08-17T17:11:56.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:56 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: [17/Aug/2024:17:11:56] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-08-17T17:11:56.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:56 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: [17/Aug/2024:17:11:56] ENGINE Bus STOPPED 2024-08-17T17:11:57.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:11:56 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:11:56.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi053\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:11:57.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:57 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:57.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:57 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:57.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:57 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:11:57.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:57 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:11:57.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:57 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:57.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:57 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:11:57.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:57 smithi053 ceph-mon[34699]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-08-17T17:11:57.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:57 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-08-17T17:11:57.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:57 smithi053 ceph-mon[34699]: Activating manager daemon y 2024-08-17T17:11:57.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:57 smithi053 ceph-mon[34699]: osdmap e89: 8 total, 8 up, 8 in 2024-08-17T17:11:57.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:57 smithi053 ceph-mon[34699]: from='mgr.25417 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-08-17T17:11:57.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:57 smithi053 ceph-mon[34699]: mgrmap e28: y(active, starting, since 0.006934s) 2024-08-17T17:11:57.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:57 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:11:57.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:57 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:11:57.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:57 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:11:57.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:57 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi003.civgvf"}]: dispatch 2024-08-17T17:11:57.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:57 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi053.ougbtc"}]: dispatch 2024-08-17T17:11:57.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:57 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-17T17:11:57.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:57 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:11:57.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:57 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:11:57.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:57 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:11:57.218 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:57 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:11:57.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:57 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:11:57.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:57 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:11:57.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:57 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:11:57.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:57 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:11:57.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:57 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-17T17:11:57.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:57 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-17T17:11:57.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:57 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-17T17:11:57.219 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:57 smithi053 ceph-mon[34699]: Manager daemon y is now available 2024-08-17T17:11:57.311 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:57.311 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:57.311 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:11:57.311 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:11:57.311 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:57.311 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:11:57.311 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[41530]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-08-17T17:11:57.311 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-08-17T17:11:57.311 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[41530]: Activating manager daemon y 2024-08-17T17:11:57.312 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[41530]: osdmap e89: 8 total, 8 up, 8 in 2024-08-17T17:11:57.312 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[41530]: from='mgr.25417 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-08-17T17:11:57.312 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[41530]: mgrmap e28: y(active, starting, since 0.006934s) 2024-08-17T17:11:57.312 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:11:57.312 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:11:57.312 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:11:57.312 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi003.civgvf"}]: dispatch 2024-08-17T17:11:57.312 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi053.ougbtc"}]: dispatch 2024-08-17T17:11:57.312 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-17T17:11:57.312 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:11:57.312 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:11:57.312 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:11:57.312 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:11:57.312 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:11:57.313 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:11:57.313 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:11:57.313 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:11:57.313 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-17T17:11:57.313 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-17T17:11:57.313 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-17T17:11:57.313 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[41530]: Manager daemon y is now available 2024-08-17T17:11:57.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:57.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:57.314 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:11:57.314 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:11:57.314 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' 2024-08-17T17:11:57.314 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:11:57.314 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[30803]: from='mgr.25417 172.21.15.53:0/3324158388' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-08-17T17:11:57.314 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-08-17T17:11:57.314 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[30803]: Activating manager daemon y 2024-08-17T17:11:57.314 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[30803]: osdmap e89: 8 total, 8 up, 8 in 2024-08-17T17:11:57.314 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[30803]: from='mgr.25417 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-08-17T17:11:57.314 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[30803]: mgrmap e28: y(active, starting, since 0.006934s) 2024-08-17T17:11:57.314 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-08-17T17:11:57.314 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-08-17T17:11:57.314 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-08-17T17:11:57.315 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi003.civgvf"}]: dispatch 2024-08-17T17:11:57.315 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi053.ougbtc"}]: dispatch 2024-08-17T17:11:57.315 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-08-17T17:11:57.315 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-08-17T17:11:57.315 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-08-17T17:11:57.315 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-08-17T17:11:57.315 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-08-17T17:11:57.315 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-08-17T17:11:57.315 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-08-17T17:11:57.315 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-08-17T17:11:57.315 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-08-17T17:11:57.315 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-08-17T17:11:57.315 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-08-17T17:11:57.316 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-08-17T17:11:57.316 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-mon[30803]: Manager daemon y is now available 2024-08-17T17:11:57.316 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:11:57.168+0000 7f63bd220640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:11:57.316 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:11:57.168+0000 7f63bd220640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:11:57.316 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:11:57.168+0000 7f63bd220640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:11:57.316 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:11:57.168+0000 7f63bd220640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:11:57.316 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:11:57.168+0000 7f63bd220640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:11:57.316 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:11:57.169+0000 7f63b8a17640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:11:57.316 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:11:57.169+0000 7f63b8a17640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:11:57.316 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:11:57.169+0000 7f63b8a17640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:11:57.316 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:11:57.169+0000 7f63b8a17640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:11:57.317 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:11:57.169+0000 7f63b8a17640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:11:57.317 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: [17/Aug/2024:17:11:57] ENGINE Bus STARTING 2024-08-17T17:11:57.317 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: [17/Aug/2024:17:11:57] ENGINE Serving on http://:::9283 2024-08-17T17:11:57.317 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:57 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: [17/Aug/2024:17:11:57] ENGINE Bus STARTED 2024-08-17T17:11:58.323 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:58 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:11:58.324 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:58 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:11:58.324 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:58 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:11:58.324 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:58 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:11:58.324 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:58 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:11:58.324 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:58 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-08-17T17:11:58.324 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:58 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-08-17T17:11:58.324 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:58 smithi053 ceph-mon[34699]: mgrmap e29: y(active, since 1.01013s) 2024-08-17T17:11:58.324 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:58 smithi053 ceph-mon[34699]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:11:58.324 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:58 smithi053 ceph-mon[34699]: pgmap v3: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:58.377 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:11:58.377 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:11:58.377 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:11:58.377 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:11:58.377 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:11:58.377 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-08-17T17:11:58.377 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-08-17T17:11:58.377 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-mon[30803]: mgrmap e29: y(active, since 1.01013s) 2024-08-17T17:11:58.377 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-mon[30803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:11:58.377 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-mon[30803]: pgmap v3: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:58.378 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:11:58.378 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:11:58.378 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:11:58.378 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:11:58.378 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-08-17T17:11:58.378 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-08-17T17:11:58.378 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-08-17T17:11:58.378 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-mon[41530]: mgrmap e29: y(active, since 1.01013s) 2024-08-17T17:11:58.378 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-mon[41530]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-08-17T17:11:58.378 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-mon[41530]: pgmap v3: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:11:58.816 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:11:59.105 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:11:58.964+0000 7f639af59640 -1 log_channel(cephadm) log [ERR] : [17/Aug/2024:17:11:58] ENGINE Error in HTTPServer.serve 2024-08-17T17:11:59.105 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: Traceback (most recent call last): 2024-08-17T17:11:59.105 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:11:59.105 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: self._connections.run(self.expiration_interval) 2024-08-17T17:11:59.105 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:11:59.106 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: self._run(expiration_interval) 2024-08-17T17:11:59.106 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:11:59.106 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:11:59.106 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:11:59.106 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:11:59.106 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:11:59.106 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: s = self.context.wrap_socket( 2024-08-17T17:11:59.106 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:11:59.106 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: return self.sslsocket_class._create( 2024-08-17T17:11:59.106 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:11:59.106 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: self.do_handshake() 2024-08-17T17:11:59.106 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:11:59.106 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: self._sslobj.do_handshake() 2024-08-17T17:11:59.107 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:11:59.107 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:11:58 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:11:59.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:59 smithi053 ceph-mon[34699]: mgrmap e30: y(active, since 2s) 2024-08-17T17:11:59.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:59 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/3159641260' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:11:59.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:59 smithi053 ceph-mon[34699]: [17/Aug/2024:17:11:58] ENGINE Bus STARTING 2024-08-17T17:11:59.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:59 smithi053 ceph-mon[34699]: from='client.16173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:59.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:59 smithi053 ceph-mon[34699]: [17/Aug/2024:17:11:58] ENGINE Serving on http://172.21.15.3:8765 2024-08-17T17:11:59.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:59 smithi053 ceph-mon[34699]: [17/Aug/2024:17:11:58] ENGINE Serving on https://172.21.15.3:7150 2024-08-17T17:11:59.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:59 smithi053 ceph-mon[34699]: [17/Aug/2024:17:11:58] ENGINE Bus STARTED 2024-08-17T17:11:59.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:11:59 smithi053 ceph-mon[34699]: [17/Aug/2024:17:11:58] ENGINE Error in HTTPServer.serve 2024-08-17T17:11:59.467 INFO:journalctl@ceph.mon.b.smithi053.stdout: Traceback (most recent call last): 2024-08-17T17:11:59.467 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:11:59.467 INFO:journalctl@ceph.mon.b.smithi053.stdout: self._connections.run(self.expiration_interval) 2024-08-17T17:11:59.467 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:11:59.467 INFO:journalctl@ceph.mon.b.smithi053.stdout: self._run(expiration_interval) 2024-08-17T17:11:59.467 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:11:59.467 INFO:journalctl@ceph.mon.b.smithi053.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:11:59.468 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:11:59.468 INFO:journalctl@ceph.mon.b.smithi053.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:11:59.468 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:11:59.468 INFO:journalctl@ceph.mon.b.smithi053.stdout: s = self.context.wrap_socket( 2024-08-17T17:11:59.468 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:11:59.468 INFO:journalctl@ceph.mon.b.smithi053.stdout: return self.sslsocket_class._create( 2024-08-17T17:11:59.468 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:11:59.468 INFO:journalctl@ceph.mon.b.smithi053.stdout: self.do_handshake() 2024-08-17T17:11:59.468 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:11:59.468 INFO:journalctl@ceph.mon.b.smithi053.stdout: self._sslobj.do_handshake() 2024-08-17T17:11:59.468 INFO:journalctl@ceph.mon.b.smithi053.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:11:59.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:59 smithi003 ceph-mon[30803]: mgrmap e30: y(active, since 2s) 2024-08-17T17:11:59.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:59 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/3159641260' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:11:59.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:59 smithi003 ceph-mon[30803]: [17/Aug/2024:17:11:58] ENGINE Bus STARTING 2024-08-17T17:11:59.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:59 smithi003 ceph-mon[30803]: from='client.16173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:59.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:59 smithi003 ceph-mon[30803]: [17/Aug/2024:17:11:58] ENGINE Serving on http://172.21.15.3:8765 2024-08-17T17:11:59.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:59 smithi003 ceph-mon[30803]: [17/Aug/2024:17:11:58] ENGINE Serving on https://172.21.15.3:7150 2024-08-17T17:11:59.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:59 smithi003 ceph-mon[30803]: [17/Aug/2024:17:11:58] ENGINE Bus STARTED 2024-08-17T17:11:59.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:11:59 smithi003 ceph-mon[30803]: [17/Aug/2024:17:11:58] ENGINE Error in HTTPServer.serve 2024-08-17T17:11:59.523 INFO:journalctl@ceph.mon.a.smithi003.stdout: Traceback (most recent call last): 2024-08-17T17:11:59.523 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:11:59.524 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._connections.run(self.expiration_interval) 2024-08-17T17:11:59.524 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:11:59.524 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._run(expiration_interval) 2024-08-17T17:11:59.524 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:11:59.524 INFO:journalctl@ceph.mon.a.smithi003.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:11:59.524 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:11:59.524 INFO:journalctl@ceph.mon.a.smithi003.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:11:59.524 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:11:59.524 INFO:journalctl@ceph.mon.a.smithi003.stdout: s = self.context.wrap_socket( 2024-08-17T17:11:59.524 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:11:59.524 INFO:journalctl@ceph.mon.a.smithi003.stdout: return self.sslsocket_class._create( 2024-08-17T17:11:59.524 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:11:59.524 INFO:journalctl@ceph.mon.a.smithi003.stdout: self.do_handshake() 2024-08-17T17:11:59.525 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:11:59.525 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._sslobj.do_handshake() 2024-08-17T17:11:59.525 INFO:journalctl@ceph.mon.a.smithi003.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:11:59.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:59 smithi003 ceph-mon[41530]: mgrmap e30: y(active, since 2s) 2024-08-17T17:11:59.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:59 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/3159641260' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:11:59.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:59 smithi003 ceph-mon[41530]: [17/Aug/2024:17:11:58] ENGINE Bus STARTING 2024-08-17T17:11:59.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:59 smithi003 ceph-mon[41530]: from='client.16173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:11:59.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:59 smithi003 ceph-mon[41530]: [17/Aug/2024:17:11:58] ENGINE Serving on http://172.21.15.3:8765 2024-08-17T17:11:59.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:59 smithi003 ceph-mon[41530]: [17/Aug/2024:17:11:58] ENGINE Serving on https://172.21.15.3:7150 2024-08-17T17:11:59.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:59 smithi003 ceph-mon[41530]: [17/Aug/2024:17:11:58] ENGINE Bus STARTED 2024-08-17T17:11:59.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:11:59 smithi003 ceph-mon[41530]: [17/Aug/2024:17:11:58] ENGINE Error in HTTPServer.serve 2024-08-17T17:11:59.526 INFO:journalctl@ceph.mon.c.smithi003.stdout: Traceback (most recent call last): 2024-08-17T17:11:59.526 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:11:59.526 INFO:journalctl@ceph.mon.c.smithi003.stdout: self._connections.run(self.expiration_interval) 2024-08-17T17:11:59.526 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:11:59.526 INFO:journalctl@ceph.mon.c.smithi003.stdout: self._run(expiration_interval) 2024-08-17T17:11:59.526 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:11:59.526 INFO:journalctl@ceph.mon.c.smithi003.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:11:59.526 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:11:59.526 INFO:journalctl@ceph.mon.c.smithi003.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:11:59.526 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:11:59.527 INFO:journalctl@ceph.mon.c.smithi003.stdout: s = self.context.wrap_socket( 2024-08-17T17:11:59.527 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:11:59.527 INFO:journalctl@ceph.mon.c.smithi003.stdout: return self.sslsocket_class._create( 2024-08-17T17:11:59.527 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:11:59.527 INFO:journalctl@ceph.mon.c.smithi003.stdout: self.do_handshake() 2024-08-17T17:11:59.527 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:11:59.527 INFO:journalctl@ceph.mon.c.smithi003.stdout: self._sslobj.do_handshake() 2024-08-17T17:11:59.527 INFO:journalctl@ceph.mon.c.smithi003.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:12:00.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:00 smithi053 ceph-mon[34699]: from='client.16179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:00.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:12:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:12:00.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:00 smithi003 ceph-mon[30803]: from='client.16179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:00.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:00 smithi003 ceph-mon[41530]: from='client.16179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:01.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:01 smithi053 ceph-mon[34699]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:01.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:01 smithi053 ceph-mon[34699]: mgrmap e31: y(active, since 4s) 2024-08-17T17:12:01.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:01 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:01.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:01 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:01.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:01 smithi003 ceph-mon[30803]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:01.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:01 smithi003 ceph-mon[30803]: mgrmap e31: y(active, since 4s) 2024-08-17T17:12:01.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:01 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:01.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:01 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:01.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:01 smithi003 ceph-mon[41530]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:01.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:01 smithi003 ceph-mon[41530]: mgrmap e31: y(active, since 4s) 2024-08-17T17:12:01.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:01 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:01.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:01 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:01.746 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:01 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: [17/Aug/2024:17:12:01] ENGINE Bus STARTING 2024-08-17T17:12:01.746 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:01 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: CherryPy Checker: 2024-08-17T17:12:01.747 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:01 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: The Application mounted at '' has an empty config. 2024-08-17T17:12:01.747 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:01 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: 2024-08-17T17:12:01.770 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:12:02.205 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:01 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: [17/Aug/2024:17:12:01] ENGINE Serving on http://:::9283 2024-08-17T17:12:02.205 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:01 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: [17/Aug/2024:17:12:01] ENGINE Bus STARTED 2024-08-17T17:12:02.374 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:02 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/1067433952' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:02.374 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:02 smithi003 ceph-mon[30803]: Standby manager daemon x started 2024-08-17T17:12:02.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:02 smithi003 ceph-mon[30803]: from='mgr.? 172.21.15.53:0/1162734524' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:12:02.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:02 smithi003 ceph-mon[30803]: from='mgr.? 172.21.15.53:0/1162734524' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:12:02.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:02 smithi003 ceph-mon[30803]: from='mgr.? 172.21.15.53:0/1162734524' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:12:02.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:02 smithi003 ceph-mon[30803]: from='mgr.? 172.21.15.53:0/1162734524' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:12:02.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:02 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:02.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:02 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:02.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:02 smithi003 ceph-mon[30803]: from='client.16197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:02.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:02 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:02.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:02 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:02.375 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:02 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/1067433952' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:02.375 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:02 smithi003 ceph-mon[41530]: Standby manager daemon x started 2024-08-17T17:12:02.376 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:02 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.53:0/1162734524' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:12:02.376 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:02 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.53:0/1162734524' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:12:02.376 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:02 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.53:0/1162734524' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:12:02.376 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:02 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.53:0/1162734524' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:12:02.376 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:02 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:02.376 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:02 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:02.376 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:02 smithi003 ceph-mon[41530]: from='client.16197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:02.376 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:02 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:02.376 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:02 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:02.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:02 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/1067433952' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:02.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:02 smithi053 ceph-mon[34699]: Standby manager daemon x started 2024-08-17T17:12:02.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:02 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.53:0/1162734524' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:12:02.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:02 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.53:0/1162734524' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:12:02.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:02 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.53:0/1162734524' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:12:02.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:02 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.53:0/1162734524' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:12:02.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:02 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:02.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:02 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:02.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:02 smithi053 ceph-mon[34699]: from='client.16197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:02.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:02 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:02.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:02 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:03.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:03 smithi053 ceph-mon[34699]: from='client.16203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:03.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:03 smithi053 ceph-mon[34699]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 14 op/s 2024-08-17T17:12:03.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:03 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-17T17:12:03.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:03 smithi053 ceph-mon[34699]: mgrmap e32: y(active, since 6s), standbys: x 2024-08-17T17:12:03.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:03 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:03.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:03 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:03.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:03 smithi003 ceph-mon[30803]: from='client.16203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:03.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:03 smithi003 ceph-mon[30803]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 14 op/s 2024-08-17T17:12:03.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:03 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-17T17:12:03.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:03 smithi003 ceph-mon[30803]: mgrmap e32: y(active, since 6s), standbys: x 2024-08-17T17:12:03.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:03 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:03.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:03 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:03.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:03 smithi003 ceph-mon[41530]: from='client.16203 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:03.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:03 smithi003 ceph-mon[41530]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 14 op/s 2024-08-17T17:12:03.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:03 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-08-17T17:12:03.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:03 smithi003 ceph-mon[41530]: mgrmap e32: y(active, since 6s), standbys: x 2024-08-17T17:12:03.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:03 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:03.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:03 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:04.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:12:04.169Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://172.21.15.53:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 172.21.15.53:8765: connect: connection refused" 2024-08-17T17:12:04.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:12:04.171Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.53:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.53:8765: connect: connection refused" 2024-08-17T17:12:04.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:12:04.172Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.53:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.53:8765: connect: connection refused" 2024-08-17T17:12:04.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:12:04.174Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.53:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.53:8765: connect: connection refused" 2024-08-17T17:12:04.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:12:04.174Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.53:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.53:8765: connect: connection refused" 2024-08-17T17:12:04.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:12:04.174Z caller=refresh.go:90 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.53:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.53:8765: connect: connection refused" 2024-08-17T17:12:04.725 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:12:05.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:05 smithi053 ceph-mon[34699]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-08-17T17:12:05.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:05 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/507952226' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:05.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:05 smithi053 ceph-mon[34699]: from='client.26114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:05.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:05 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:05.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:05 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:05.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:05 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:12:05.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:05 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:12:05.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:05 smithi003 ceph-mon[30803]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-08-17T17:12:05.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:05 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/507952226' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:05.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:05 smithi003 ceph-mon[30803]: from='client.26114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:05.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:05 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:05.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:05 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:05.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:05 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:12:05.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:05 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:12:05.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:05 smithi003 ceph-mon[41530]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-08-17T17:12:05.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:05 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/507952226' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:05.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:05 smithi003 ceph-mon[41530]: from='client.26114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:05.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:05 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:05.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:05 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:05.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:05 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:12:05.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:05 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:12:06.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:06 smithi053 ceph-mon[34699]: from='client.16221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:06.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:06 smithi053 ceph-mon[34699]: Updating smithi003:/etc/ceph/ceph.conf 2024-08-17T17:12:06.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:06 smithi053 ceph-mon[34699]: Updating smithi053:/etc/ceph/ceph.conf 2024-08-17T17:12:06.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:06 smithi053 ceph-mon[34699]: Updating smithi003:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:12:06.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:06 smithi053 ceph-mon[34699]: Updating smithi053:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:12:06.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:06 smithi053 ceph-mon[34699]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:12:06.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:06 smithi053 ceph-mon[34699]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:12:06.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:06 smithi053 ceph-mon[34699]: Updating smithi003:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:12:06.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:06 smithi053 ceph-mon[34699]: Updating smithi053:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:12:06.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:06 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:06.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:06 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:06.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:06 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:06.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:06 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:06.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:06 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:06.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:06 smithi003 ceph-mon[30803]: from='client.16221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:06.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:06 smithi003 ceph-mon[30803]: Updating smithi003:/etc/ceph/ceph.conf 2024-08-17T17:12:06.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:06 smithi003 ceph-mon[30803]: Updating smithi053:/etc/ceph/ceph.conf 2024-08-17T17:12:06.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:06 smithi003 ceph-mon[30803]: Updating smithi003:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:12:06.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:06 smithi003 ceph-mon[30803]: Updating smithi053:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:12:06.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:06 smithi003 ceph-mon[30803]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:12:06.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:06 smithi003 ceph-mon[30803]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:12:06.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:06 smithi003 ceph-mon[30803]: Updating smithi003:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:12:06.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:06 smithi003 ceph-mon[30803]: Updating smithi053:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:12:06.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:06 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:06.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:06 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:06.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:06 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:06.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:06 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:06.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:06 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:06.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:06 smithi003 ceph-mon[41530]: from='client.16221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:06.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:06 smithi003 ceph-mon[41530]: Updating smithi003:/etc/ceph/ceph.conf 2024-08-17T17:12:06.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:06 smithi003 ceph-mon[41530]: Updating smithi053:/etc/ceph/ceph.conf 2024-08-17T17:12:06.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:06 smithi003 ceph-mon[41530]: Updating smithi003:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:12:06.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:06 smithi003 ceph-mon[41530]: Updating smithi053:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:12:06.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:06 smithi003 ceph-mon[41530]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:12:06.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:06 smithi003 ceph-mon[41530]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:12:06.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:06 smithi003 ceph-mon[41530]: Updating smithi003:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:12:06.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:06 smithi003 ceph-mon[41530]: Updating smithi053:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:12:06.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:06 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:06.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:06 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:06.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:06 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:06.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:06 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:06.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:06 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:07.074 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:07 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[67423]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:12:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.51.0" 2024-08-17T17:12:07.075 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:06 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:12:06.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi053\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:12:07.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:07 smithi053 ceph-mon[34699]: Reconfiguring prometheus.a (dependencies changed)... 2024-08-17T17:12:07.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:07 smithi053 ceph-mon[34699]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-08-17T17:12:07.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:07 smithi053 ceph-mon[34699]: Reconfiguring daemon prometheus.a on smithi053 2024-08-17T17:12:07.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:07 smithi003 ceph-mon[41530]: Reconfiguring prometheus.a (dependencies changed)... 2024-08-17T17:12:07.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:07 smithi003 ceph-mon[41530]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-08-17T17:12:07.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:07 smithi003 ceph-mon[41530]: Reconfiguring daemon prometheus.a on smithi053 2024-08-17T17:12:07.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:07 smithi003 ceph-mon[30803]: Reconfiguring prometheus.a (dependencies changed)... 2024-08-17T17:12:07.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:07 smithi003 ceph-mon[30803]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-08-17T17:12:07.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:07 smithi003 ceph-mon[30803]: Reconfiguring daemon prometheus.a on smithi053 2024-08-17T17:12:07.687 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:12:08.335 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:08 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/3185568659' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:08.335 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:08 smithi053 ceph-mon[34699]: from='client.26132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:08.335 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:08 smithi053 ceph-mon[34699]: from='client.26138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:08.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:08 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/3185568659' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:08.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:08 smithi003 ceph-mon[41530]: from='client.26132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:08.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:08 smithi003 ceph-mon[41530]: from='client.26138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:08.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:08 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/3185568659' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:08.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:08 smithi003 ceph-mon[30803]: from='client.26132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:08.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:08 smithi003 ceph-mon[30803]: from='client.26138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:09 smithi003 ceph-mon[41530]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-08-17T17:12:09.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:09 smithi003 ceph-mon[30803]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-08-17T17:12:09.703 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:09 smithi053 ceph-mon[34699]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-08-17T17:12:09.703 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:09 smithi053 systemd[1]: Stopping Ceph prometheus.a for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:12:09.966 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:12:09.702Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-08-17T17:12:09.967 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:12:09.702Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-08-17T17:12:09.967 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:12:09.702Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-08-17T17:12:09.967 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:12:09.702Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-08-17T17:12:09.967 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:12:09.702Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-08-17T17:12:09.967 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:12:09.702Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-08-17T17:12:09.967 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:12:09.702Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-08-17T17:12:09.967 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:12:09.702Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-08-17T17:12:09.967 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:12:09.703Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-08-17T17:12:09.967 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:12:09.704Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-08-17T17:12:09.967 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:12:09.704Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-08-17T17:12:09.967 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[76357]: ts=2024-08-17T17:12:09.704Z caller=main.go:1273 level=info msg="See you next time!" 2024-08-17T17:12:09.968 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:09 smithi053 podman[80736]: 2024-08-17 17:12:09.712935609 +0000 UTC m=+0.168516309 container died fb2a55f98a6f6c66b04a667a118ef6566a3cbbb8cc949837757803c38ea7cd83 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:12:10.354 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:10 smithi053 podman[80736]: 2024-08-17 17:12:10.060690318 +0000 UTC m=+0.516271063 container cleanup fb2a55f98a6f6c66b04a667a118ef6566a3cbbb8cc949837757803c38ea7cd83 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:12:10.354 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:10 smithi053 bash[80736]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a 2024-08-17T17:12:10.636 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:12:10.716 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:10 smithi053 podman[80749]: 2024-08-17 17:12:10.353619054 +0000 UTC m=+0.641942178 container remove fb2a55f98a6f6c66b04a667a118ef6566a3cbbb8cc949837757803c38ea7cd83 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:12:10.716 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:10 smithi053 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-08-17T17:12:10.717 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:10 smithi053 systemd[1]: Stopped Ceph prometheus.a for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:12:10.717 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:10 smithi053 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@prometheus.a.service: Consumed 1.255s CPU time. 2024-08-17T17:12:10.717 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:10 smithi053 systemd[1]: Starting Ceph prometheus.a for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:12:10.997 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:10 smithi053 podman[80842]: 2024-08-17 17:12:10.624639769 +0000 UTC m=+0.019085637 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-08-17T17:12:10.997 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:10 smithi053 podman[80842]: 2024-08-17 17:12:10.80854879 +0000 UTC m=+0.202994652 container create 41146206916a72307856f122da5d27ae859b7c1e840eb6e0961e6939c07ceb0e (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:12:11.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:11 smithi053 ceph-mon[34699]: pgmap v10: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-08-17T17:12:11.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:11 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/300021432' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:11.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:11 smithi053 ceph-mon[34699]: from='client.16251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:11.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:11 smithi053 ceph-mon[34699]: from='client.16257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:11.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:11 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:11.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:11 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:11.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:11 smithi053 podman[80842]: 2024-08-17 17:12:11.090279571 +0000 UTC m=+0.484725449 container init 41146206916a72307856f122da5d27ae859b7c1e840eb6e0961e6939c07ceb0e (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:12:11.468 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:11 smithi053 podman[80842]: 2024-08-17 17:12:11.093240591 +0000 UTC m=+0.487686450 container start 41146206916a72307856f122da5d27ae859b7c1e840eb6e0961e6939c07ceb0e (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:12:11.468 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:11.113Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-08-17T17:12:11.468 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:11.113Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-08-17T17:12:11.468 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:11.113Z caller=main.go:623 level=info host_details="(Linux 5.14.0-480.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024 x86_64 smithi053 (none))" 2024-08-17T17:12:11.468 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:11.113Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-08-17T17:12:11.468 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:11.113Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-08-17T17:12:11.468 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:11.115Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-08-17T17:12:11.468 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:11.116Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-08-17T17:12:11.469 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:11.118Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-08-17T17:12:11.469 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:11.118Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-08-17T17:12:11.469 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:11.119Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-08-17T17:12:11.469 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:11.119Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.671µs 2024-08-17T17:12:11.469 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:11.119Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-08-17T17:12:11.469 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:11.120Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-08-17T17:12:11.469 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:11.131Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-08-17T17:12:11.469 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:11.138Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-08-17T17:12:11.469 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:11.139Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-08-17T17:12:11.469 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:11.139Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=38.849µs wal_replay_duration=19.513168ms wbl_replay_duration=122ns total_replay_duration=19.584887ms 2024-08-17T17:12:11.469 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:11.142Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-08-17T17:12:11.470 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:11.142Z caller=main.go:1153 level=info msg="TSDB started" 2024-08-17T17:12:11.470 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:11.142Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-08-17T17:12:11.470 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:11.156Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=14.252184ms db_storage=850ns remote_storage=1.408µs web_handler=351ns query_engine=531ns scrape=476.978µs scrape_sd=102.745µs notify=11.268µs notify_sd=9.979µs rules=13.32336ms tracing=6.06µs 2024-08-17T17:12:11.470 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:11.156Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-08-17T17:12:11.470 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:11.156Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-08-17T17:12:11.470 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:11 smithi053 bash[80842]: 41146206916a72307856f122da5d27ae859b7c1e840eb6e0961e6939c07ceb0e 2024-08-17T17:12:11.470 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:11 smithi053 systemd[1]: Started Ceph prometheus.a for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:12:11.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:11 smithi003 ceph-mon[30803]: pgmap v10: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-08-17T17:12:11.593 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:11 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/300021432' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:11.593 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:11 smithi003 ceph-mon[30803]: from='client.16251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:11.593 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:11 smithi003 ceph-mon[30803]: from='client.16257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:11.593 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:11 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:11.593 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:11 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:11.593 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:11 smithi003 ceph-mon[41530]: pgmap v10: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-08-17T17:12:11.594 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:11 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/300021432' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:11.594 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:11 smithi003 ceph-mon[41530]: from='client.16251 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:11.594 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:11 smithi003 ceph-mon[41530]: from='client.16257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:11.594 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:11 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:11.594 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:11 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:12.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:12 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:12:12.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:12 smithi003 ceph-mon[30803]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:12:12.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:12 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:12:12.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:12 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:12:12.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:12 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:12:12.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:12 smithi003 ceph-mon[41530]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:12:12.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:12 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:12:12.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:12 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:12:12.549 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:12 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:12:12.550 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:12 smithi053 ceph-mon[34699]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-08-17T17:12:12.550 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:12 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:12:12.550 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:12 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-08-17T17:12:13.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:13 smithi003 ceph-mon[41530]: pgmap v11: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-08-17T17:12:13.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:13 smithi003 ceph-mon[41530]: Upgrade: Updating mgr.x 2024-08-17T17:12:13.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:13 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:12:13.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:13 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:12:13.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:13 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-17T17:12:13.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:13 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:12:13.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:13 smithi003 ceph-mon[41530]: Deploying daemon mgr.x on smithi053 2024-08-17T17:12:13.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:13 smithi003 ceph-mon[30803]: pgmap v11: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-08-17T17:12:13.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:13 smithi003 ceph-mon[30803]: Upgrade: Updating mgr.x 2024-08-17T17:12:13.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:13 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:12:13.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:13 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:12:13.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:13 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-17T17:12:13.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:13 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:12:13.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:13 smithi003 ceph-mon[30803]: Deploying daemon mgr.x on smithi053 2024-08-17T17:12:13.571 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:12:13.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:13 smithi053 ceph-mon[34699]: pgmap v11: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-08-17T17:12:13.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:13 smithi053 ceph-mon[34699]: Upgrade: Updating mgr.x 2024-08-17T17:12:13.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:13 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:12:13.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:13 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-08-17T17:12:13.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:13 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-08-17T17:12:13.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:13 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:12:13.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:13 smithi053 ceph-mon[34699]: Deploying daemon mgr.x on smithi053 2024-08-17T17:12:14.309 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:14 smithi053 systemd[1]: Stopping Ceph mgr.x for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:12:14.309 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:14 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/1329677211' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:14.309 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:14 smithi053 ceph-mon[34699]: from='client.26168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:14.309 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:14 smithi053 ceph-mon[34699]: from='client.26174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:14.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:14 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/1329677211' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:14.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:14 smithi003 ceph-mon[41530]: from='client.26168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:14.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:14 smithi003 ceph-mon[41530]: from='client.26174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:14.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:14 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/1329677211' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:14.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:14 smithi003 ceph-mon[30803]: from='client.26168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:14.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:14 smithi003 ceph-mon[30803]: from='client.26174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:14.569 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:14 smithi053 podman[81178]: 2024-08-17 17:12:14.307874579 +0000 UTC m=+0.131898111 container died 409905c993755fa2fa198682c610ab03d7ef585f5933a393fca5978ef854e83f (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.36.0, RELEASE=main-cd8bdbe, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS) 2024-08-17T17:12:14.966 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:14 smithi053 podman[81178]: 2024-08-17 17:12:14.651408495 +0000 UTC m=+0.475432025 container cleanup 409905c993755fa2fa198682c610ab03d7ef585f5933a393fca5978ef854e83f (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-cd8bdbe, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0) 2024-08-17T17:12:14.967 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:14 smithi053 bash[81178]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x 2024-08-17T17:12:15.242 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:14 smithi053 podman[81191]: 2024-08-17 17:12:14.969744432 +0000 UTC m=+0.662998540 container remove 409905c993755fa2fa198682c610ab03d7ef585f5933a393fca5978ef854e83f (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-cd8bdbe, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0) 2024-08-17T17:12:15.242 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:14 smithi053 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-08-17T17:12:15.242 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:15 smithi053 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-08-17T17:12:15.242 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:15 smithi053 systemd[1]: Stopped Ceph mgr.x for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:12:15.242 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:15 smithi053 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mgr.x.service: Consumed 21.093s CPU time. 2024-08-17T17:12:15.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:15 smithi003 ceph-mon[41530]: pgmap v12: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:15.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:15 smithi003 ceph-mon[30803]: pgmap v12: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:15.562 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:15 smithi053 systemd[1]: Starting Ceph mgr.x for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:12:15.563 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:15 smithi053 ceph-mon[34699]: pgmap v12: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:15.873 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:15 smithi053 podman[81320]: 2024-08-17 17:12:15.464929285 +0000 UTC m=+0.018152747 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:12:15.873 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:15 smithi053 podman[81320]: 2024-08-17 17:12:15.587705081 +0000 UTC m=+0.140928542 container create 874be7d3a23839f9f1b0d6f5cec91f50ea58a84f6dba32cc116f8250c1868679 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=main-cd8bdbe, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.36.0) 2024-08-17T17:12:15.874 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:15 smithi053 podman[81320]: 2024-08-17 17:12:15.76489869 +0000 UTC m=+0.318122155 container init 874be7d3a23839f9f1b0d6f5cec91f50ea58a84f6dba32cc116f8250c1868679 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=main-cd8bdbe, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0) 2024-08-17T17:12:15.874 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:15 smithi053 podman[81320]: 2024-08-17 17:12:15.768021709 +0000 UTC m=+0.321245171 container start 874be7d3a23839f9f1b0d6f5cec91f50ea58a84f6dba32cc116f8250c1868679 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-cd8bdbe, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0) 2024-08-17T17:12:15.874 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:15 smithi053 ceph-mgr[81339]: -- 172.21.15.53:0/375169298 <== mon.1 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55edc6ad70e0 con 0x55edc6ab4800 2024-08-17T17:12:16.216 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:15 smithi053 bash[81320]: 874be7d3a23839f9f1b0d6f5cec91f50ea58a84f6dba32cc116f8250c1868679 2024-08-17T17:12:16.216 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:15 smithi053 systemd[1]: Started Ceph mgr.x for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:12:16.522 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:12:17.186 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:16 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:16.947Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi053\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:12:17.186 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:16 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:17.186 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:16 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:17.186 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:16 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:12:17.186 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:16 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/1759134915' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:17.186 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:16 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:17.186 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:16 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:17.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:16 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:17.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:16 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:17.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:16 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:12:17.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:16 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/1759134915' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:17.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:16 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:17.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:16 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:17.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:16 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:17.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:16 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:17.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:16 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:12:17.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:16 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/1759134915' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:17.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:16 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:17.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:16 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:18.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:17 smithi003 ceph-mon[41530]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:18.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:17 smithi003 ceph-mon[41530]: from='client.26192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:18.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:17 smithi003 ceph-mon[41530]: from='client.26198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:18.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:17 smithi003 ceph-mon[30803]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:18.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:17 smithi003 ceph-mon[30803]: from='client.26192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:18.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:17 smithi003 ceph-mon[30803]: from='client.26198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:18.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:17 smithi053 ceph-mon[34699]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:18.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:17 smithi053 ceph-mon[34699]: from='client.26192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:18.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:17 smithi053 ceph-mon[34699]: from='client.26198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:19.469 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:12:20.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:19 smithi053 ceph-mon[34699]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:20.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:19 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/3419169212' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:20.217 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:12:20.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:19 smithi003 ceph-mon[30803]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:20.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:19 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/3419169212' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:20.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:19 smithi003 ceph-mon[41530]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:20.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:19 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/3419169212' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:21.233 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:21 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[81335]: [17/Aug/2024:17:12:21] ENGINE Bus STARTING 2024-08-17T17:12:21.233 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:21 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[81335]: CherryPy Checker: 2024-08-17T17:12:21.233 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:21 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[81335]: The Application mounted at '' has an empty config. 2024-08-17T17:12:21.233 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:21 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[81335]: 2024-08-17T17:12:21.234 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:20 smithi053 ceph-mon[34699]: from='client.26210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:21.234 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:20 smithi053 ceph-mon[34699]: from='client.26216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:21.234 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:20 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:21.234 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:20 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:21.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:20 smithi003 ceph-mon[30803]: from='client.26210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:21.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:20 smithi003 ceph-mon[30803]: from='client.26216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:21.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:20 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:21.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:20 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:21.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:20 smithi003 ceph-mon[41530]: from='client.26210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:21.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:20 smithi003 ceph-mon[41530]: from='client.26216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:21.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:20 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:21.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:20 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:21.599 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:21 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[81335]: [17/Aug/2024:17:12:21] ENGINE Serving on http://:::9283 2024-08-17T17:12:21.599 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:21 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[81335]: [17/Aug/2024:17:12:21] ENGINE Bus STARTED 2024-08-17T17:12:22.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:21 smithi053 ceph-mon[34699]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:22.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:21 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:22.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:21 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:22.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:21 smithi053 ceph-mon[34699]: Standby manager daemon x restarted 2024-08-17T17:12:22.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:21 smithi053 ceph-mon[34699]: Standby manager daemon x started 2024-08-17T17:12:22.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:21 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.53:0/1985468090' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:12:22.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:21 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.53:0/1985468090' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:12:22.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:21 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.53:0/1985468090' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:12:22.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:21 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.53:0/1985468090' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:12:22.238 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:21 smithi003 ceph-mon[30803]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:22.238 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:21 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:22.238 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:21 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:22.238 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:21 smithi003 ceph-mon[30803]: Standby manager daemon x restarted 2024-08-17T17:12:22.238 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:21 smithi003 ceph-mon[30803]: Standby manager daemon x started 2024-08-17T17:12:22.238 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:21 smithi003 ceph-mon[30803]: from='mgr.? 172.21.15.53:0/1985468090' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:12:22.238 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:21 smithi003 ceph-mon[30803]: from='mgr.? 172.21.15.53:0/1985468090' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:12:22.238 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:21 smithi003 ceph-mon[30803]: from='mgr.? 172.21.15.53:0/1985468090' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:12:22.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:21 smithi003 ceph-mon[30803]: from='mgr.? 172.21.15.53:0/1985468090' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:12:22.239 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:21 smithi003 ceph-mon[41530]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:22.239 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:21 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:22.239 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:21 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:22.239 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:21 smithi003 ceph-mon[41530]: Standby manager daemon x restarted 2024-08-17T17:12:22.239 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:21 smithi003 ceph-mon[41530]: Standby manager daemon x started 2024-08-17T17:12:22.239 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:21 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.53:0/1985468090' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:12:22.239 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:21 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.53:0/1985468090' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:12:22.239 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:21 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.53:0/1985468090' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:12:22.239 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:21 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.53:0/1985468090' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:12:22.427 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:12:23.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:22 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/347533468' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:23.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:22 smithi053 ceph-mon[34699]: mgrmap e33: y(active, since 25s), standbys: x 2024-08-17T17:12:23.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:22 smithi053 ceph-mon[34699]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:23.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:22 smithi053 ceph-mon[34699]: from='client.16329 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:23.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:22 smithi053 ceph-mon[34699]: from='client.26234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:23.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:22 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/347533468' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:23.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:22 smithi003 ceph-mon[30803]: mgrmap e33: y(active, since 25s), standbys: x 2024-08-17T17:12:23.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:22 smithi003 ceph-mon[30803]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:23.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:22 smithi003 ceph-mon[30803]: from='client.16329 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:23.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:22 smithi003 ceph-mon[30803]: from='client.26234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:23.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:22 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/347533468' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:23.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:22 smithi003 ceph-mon[41530]: mgrmap e33: y(active, since 25s), standbys: x 2024-08-17T17:12:23.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:22 smithi003 ceph-mon[41530]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:23.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:22 smithi003 ceph-mon[41530]: from='client.16329 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:23.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:22 smithi003 ceph-mon[41530]: from='client.26234 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:25.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:25.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:25.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:12:25.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:12:25.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:25.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[41530]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:25.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:12:25.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:12:25.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[41530]: Upgrade: Setting container_image for all mgr 2024-08-17T17:12:25.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:25.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-08-17T17:12:25.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-08-17T17:12:25.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-08-17T17:12:25.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-08-17T17:12:25.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-08-17T17:12:25.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-08-17T17:12:25.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-08-17T17:12:25.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-08-17T17:12:25.277 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[41530]: Upgrade: It appears safe to stop mon.a 2024-08-17T17:12:25.277 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/3088322048' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:25.277 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:25.277 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:25.278 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:12:25.278 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:12:25.278 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:25.278 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[30803]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:25.278 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:12:25.278 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:12:25.278 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[30803]: Upgrade: Setting container_image for all mgr 2024-08-17T17:12:25.278 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:25.279 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-08-17T17:12:25.279 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-08-17T17:12:25.279 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-08-17T17:12:25.279 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-08-17T17:12:25.279 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-08-17T17:12:25.279 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-08-17T17:12:25.279 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-08-17T17:12:25.279 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-08-17T17:12:25.279 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[30803]: Upgrade: It appears safe to stop mon.a 2024-08-17T17:12:25.279 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-mon[30803]: from='client.? 172.21.15.3:0/3088322048' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:25.373 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:12:25.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:25 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:25.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:25 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:25.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:25 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:12:25.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:25 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-08-17T17:12:25.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:25 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:25.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:25 smithi053 ceph-mon[34699]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:25.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:25 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-08-17T17:12:25.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:25 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-08-17T17:12:25.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:25 smithi053 ceph-mon[34699]: Upgrade: Setting container_image for all mgr 2024-08-17T17:12:25.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:25 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:25.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:25 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-08-17T17:12:25.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:25 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-08-17T17:12:25.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:25 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-08-17T17:12:25.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:25 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-08-17T17:12:25.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:25 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-08-17T17:12:25.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:25 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-08-17T17:12:25.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:25 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-08-17T17:12:25.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:25 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-08-17T17:12:25.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:25 smithi053 ceph-mon[34699]: Upgrade: It appears safe to stop mon.a 2024-08-17T17:12:25.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:25 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/3088322048' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:26.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:12:25] "GET /metrics HTTP/1.1" 200 33840 "" "Prometheus/2.51.0" 2024-08-17T17:12:26.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:25 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:25.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:12:26.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:25 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:25.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:12:27.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:26 smithi003 ceph-mon[41530]: from='client.16347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:27.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:26 smithi003 ceph-mon[41530]: from='client.26252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:27.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:26 smithi003 ceph-mon[41530]: Upgrade: Updating mon.a 2024-08-17T17:12:27.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:26 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:27.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:26 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:12:27.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:26 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-17T17:12:27.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:26 smithi003 ceph-mon[41530]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:12:27.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:26 smithi003 ceph-mon[41530]: Deploying daemon mon.a on smithi003 2024-08-17T17:12:27.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:26 smithi003 ceph-mon[30803]: from='client.16347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:27.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:26 smithi003 ceph-mon[30803]: from='client.26252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:27.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:26 smithi003 ceph-mon[30803]: Upgrade: Updating mon.a 2024-08-17T17:12:27.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:26 smithi003 ceph-mon[30803]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:27.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:26 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:12:27.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:26 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-17T17:12:27.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:26 smithi003 ceph-mon[30803]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:12:27.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:26 smithi003 ceph-mon[30803]: Deploying daemon mon.a on smithi003 2024-08-17T17:12:27.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:26.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi053\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:12:27.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:26 smithi053 ceph-mon[34699]: from='client.16347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:27.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:26 smithi053 ceph-mon[34699]: from='client.26252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:27.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:26 smithi053 ceph-mon[34699]: Upgrade: Updating mon.a 2024-08-17T17:12:27.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:26 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='mgr.y' 2024-08-17T17:12:27.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:26 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-08-17T17:12:27.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:26 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-08-17T17:12:27.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:26 smithi053 ceph-mon[34699]: from='mgr.25979 172.21.15.3:0/808301578' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-08-17T17:12:27.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:26 smithi053 ceph-mon[34699]: Deploying daemon mon.a on smithi003 2024-08-17T17:12:27.713 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:27 smithi003 systemd[1]: Stopping Ceph mon.a for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:12:28.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:27 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-a[30799]: 2024-08-17T17:12:27.710+0000 7f5413c4f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T17:12:28.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:27 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-a[30799]: 2024-08-17T17:12:27.710+0000 7f5413c4f640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-08-17T17:12:28.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:27 smithi003 podman[100381]: 2024-08-17 17:12:27.732778443 +0000 UTC m=+0.134023762 container died a8b52ab645d32066a4c2bae858d2be3a8353be59f545703bac701da0f07d1193 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-a, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=quincy-f2c39d4, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731) 2024-08-17T17:12:28.367 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:28 smithi003 podman[100381]: 2024-08-17 17:12:28.212657895 +0000 UTC m=+0.613903202 container cleanup a8b52ab645d32066a4c2bae858d2be3a8353be59f545703bac701da0f07d1193 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=quincy-f2c39d4, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:12:28.367 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:28 smithi003 bash[100381]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-a 2024-08-17T17:12:28.367 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:12:28.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:28 smithi003 podman[100397]: 2024-08-17 17:12:28.456101453 +0000 UTC m=+0.714468435 container remove a8b52ab645d32066a4c2bae858d2be3a8353be59f545703bac701da0f07d1193 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-a, org.label-schema.license=GPLv2, RELEASE=quincy-f2c39d4, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.36.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-08-17T17:12:28.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:28 smithi003 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mon.a.service: Deactivated successfully. 2024-08-17T17:12:28.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:28 smithi003 systemd[1]: Stopped Ceph mon.a for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:12:28.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:28 smithi003 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mon.a.service: Consumed 13.741s CPU time. 2024-08-17T17:12:29.160 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:28 smithi003 systemd[1]: Starting Ceph mon.a for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:12:29.160 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 podman[100596]: 2024-08-17 17:12:28.928884325 +0000 UTC m=+0.018126176 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:12:29.160 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 podman[100596]: 2024-08-17 17:12:29.031984719 +0000 UTC m=+0.121226557 container create c1cfbc2d47f8446b609e615b86be102c47a3b842c3e7bab3ca9290f7353dff3d (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, RELEASE=main-cd8bdbe, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True) 2024-08-17T17:12:29.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 podman[100596]: 2024-08-17 17:12:29.275698418 +0000 UTC m=+0.364940269 container init c1cfbc2d47f8446b609e615b86be102c47a3b842c3e7bab3ca9290f7353dff3d (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-cd8bdbe, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-08-17T17:12:29.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 podman[100596]: 2024-08-17 17:12:29.279167271 +0000 UTC m=+0.368409119 container start c1cfbc2d47f8446b609e615b86be102c47a3b842c3e7bab3ca9290f7353dff3d (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-a, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-cd8bdbe, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-08-17T17:12:29.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: set uid:gid to 167:167 (ceph:ceph) 2024-08-17T17:12:29.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev), process ceph-mon, pid 2 2024-08-17T17:12:29.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: pidfile_write: ignore empty --pid-file 2024-08-17T17:12:29.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: load: jerasure load: lrc 2024-08-17T17:12:29.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: RocksDB version: 7.9.2 2024-08-17T17:12:29.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Git sha 0 2024-08-17T17:12:29.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Compile date 2024-08-15 20:43:31 2024-08-17T17:12:29.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: DB SUMMARY 2024-08-17T17:12:29.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: DB Session ID: 9LJ7FFS12MAM3NIK3BNB 2024-08-17T17:12:29.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: CURRENT file: CURRENT 2024-08-17T17:12:29.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: IDENTITY file: IDENTITY 2024-08-17T17:12:29.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: MANIFEST file: MANIFEST-000015 size: 1172 Bytes 2024-08-17T17:12:29.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000033.sst 2024-08-17T17:12:29.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000031.log size: 678122 ; 2024-08-17T17:12:29.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.error_if_exists: 0 2024-08-17T17:12:29.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.create_if_missing: 0 2024-08-17T17:12:29.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.paranoid_checks: 1 2024-08-17T17:12:29.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.flush_verify_memtable_count: 1 2024-08-17T17:12:29.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-08-17T17:12:29.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-08-17T17:12:29.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.env: 0x5624c298bbe0 2024-08-17T17:12:29.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.fs: PosixFileSystem 2024-08-17T17:12:29.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.info_log: 0x5624c392b700 2024-08-17T17:12:29.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.max_file_opening_threads: 16 2024-08-17T17:12:29.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.statistics: (nil) 2024-08-17T17:12:29.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.use_fsync: 0 2024-08-17T17:12:29.527 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.max_log_file_size: 0 2024-08-17T17:12:29.527 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-08-17T17:12:29.527 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.log_file_time_to_roll: 0 2024-08-17T17:12:29.527 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.keep_log_file_num: 1000 2024-08-17T17:12:29.527 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.recycle_log_file_num: 0 2024-08-17T17:12:29.527 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.allow_fallocate: 1 2024-08-17T17:12:29.527 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.allow_mmap_reads: 0 2024-08-17T17:12:29.527 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.allow_mmap_writes: 0 2024-08-17T17:12:29.527 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.use_direct_reads: 0 2024-08-17T17:12:29.527 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-08-17T17:12:29.527 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.create_missing_column_families: 0 2024-08-17T17:12:29.527 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.db_log_dir: 2024-08-17T17:12:29.527 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.wal_dir: 2024-08-17T17:12:29.528 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.table_cache_numshardbits: 6 2024-08-17T17:12:29.528 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.WAL_ttl_seconds: 0 2024-08-17T17:12:29.528 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.WAL_size_limit_MB: 0 2024-08-17T17:12:29.528 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-08-17T17:12:29.528 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-08-17T17:12:29.528 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.is_fd_close_on_exec: 1 2024-08-17T17:12:29.528 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.advise_random_on_open: 1 2024-08-17T17:12:29.528 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.db_write_buffer_size: 0 2024-08-17T17:12:29.528 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.write_buffer_manager: 0x5624c392ee60 2024-08-17T17:12:29.528 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-08-17T17:12:29.528 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-08-17T17:12:29.528 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.use_adaptive_mutex: 0 2024-08-17T17:12:29.528 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.rate_limiter: (nil) 2024-08-17T17:12:29.529 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-08-17T17:12:29.529 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.wal_recovery_mode: 2 2024-08-17T17:12:29.529 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.enable_thread_tracking: 0 2024-08-17T17:12:29.529 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.enable_pipelined_write: 0 2024-08-17T17:12:29.529 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.unordered_write: 0 2024-08-17T17:12:29.529 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-08-17T17:12:29.529 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-08-17T17:12:29.529 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-08-17T17:12:29.529 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-08-17T17:12:29.529 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.row_cache: None 2024-08-17T17:12:29.529 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.wal_filter: None 2024-08-17T17:12:29.529 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-08-17T17:12:29.530 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.allow_ingest_behind: 0 2024-08-17T17:12:29.530 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.two_write_queues: 0 2024-08-17T17:12:29.530 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.manual_wal_flush: 0 2024-08-17T17:12:29.530 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.wal_compression: 0 2024-08-17T17:12:29.530 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.atomic_flush: 0 2024-08-17T17:12:29.530 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-08-17T17:12:29.530 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.persist_stats_to_disk: 0 2024-08-17T17:12:29.530 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.write_dbid_to_manifest: 0 2024-08-17T17:12:29.530 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.log_readahead_size: 0 2024-08-17T17:12:29.530 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-08-17T17:12:29.530 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.best_efforts_recovery: 0 2024-08-17T17:12:29.530 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-08-17T17:12:29.530 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-08-17T17:12:29.531 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.allow_data_in_errors: 0 2024-08-17T17:12:29.531 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.db_host_id: __hostname__ 2024-08-17T17:12:29.531 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.enforce_single_del_contracts: true 2024-08-17T17:12:29.531 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.max_background_jobs: 2 2024-08-17T17:12:29.531 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.max_background_compactions: -1 2024-08-17T17:12:29.531 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.max_subcompactions: 1 2024-08-17T17:12:29.531 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-08-17T17:12:29.531 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-08-17T17:12:29.531 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.delayed_write_rate : 16777216 2024-08-17T17:12:29.531 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.max_total_wal_size: 0 2024-08-17T17:12:29.531 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-08-17T17:12:29.531 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.stats_dump_period_sec: 600 2024-08-17T17:12:29.531 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.stats_persist_period_sec: 600 2024-08-17T17:12:29.531 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-08-17T17:12:29.531 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.max_open_files: -1 2024-08-17T17:12:29.532 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.bytes_per_sync: 0 2024-08-17T17:12:29.532 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.wal_bytes_per_sync: 0 2024-08-17T17:12:29.532 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.strict_bytes_per_sync: 0 2024-08-17T17:12:29.532 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.compaction_readahead_size: 0 2024-08-17T17:12:29.532 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.max_background_flushes: -1 2024-08-17T17:12:29.532 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Compression algorithms supported: 2024-08-17T17:12:29.532 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: kZSTD supported: 0 2024-08-17T17:12:29.532 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: kXpressCompression supported: 0 2024-08-17T17:12:29.532 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: kBZip2Compression supported: 0 2024-08-17T17:12:29.532 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-08-17T17:12:29.532 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: kLZ4Compression supported: 1 2024-08-17T17:12:29.532 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: kZlibCompression supported: 1 2024-08-17T17:12:29.532 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: kLZ4HCCompression supported: 1 2024-08-17T17:12:29.532 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: kSnappyCompression supported: 1 2024-08-17T17:12:29.533 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Fast CRC32 supported: Supported on x86 2024-08-17T17:12:29.533 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: DMutex implementation: pthread_mutex_t 2024-08-17T17:12:29.533 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-08-17T17:12:29.533 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-08-17T17:12:29.533 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-08-17T17:12:29.533 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.merge_operator: 2024-08-17T17:12:29.533 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.compaction_filter: None 2024-08-17T17:12:29.533 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.compaction_filter_factory: None 2024-08-17T17:12:29.533 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.sst_partitioner_factory: None 2024-08-17T17:12:29.533 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.memtable_factory: SkipListFactory 2024-08-17T17:12:29.533 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.table_factory: BlockBasedTable 2024-08-17T17:12:29.533 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5624c392b380) 2024-08-17T17:12:29.533 INFO:journalctl@ceph.mon.a.smithi003.stdout: cache_index_and_filter_blocks: 1 2024-08-17T17:12:29.534 INFO:journalctl@ceph.mon.a.smithi003.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-08-17T17:12:29.534 INFO:journalctl@ceph.mon.a.smithi003.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-08-17T17:12:29.534 INFO:journalctl@ceph.mon.a.smithi003.stdout: pin_top_level_index_and_filter: 1 2024-08-17T17:12:29.534 INFO:journalctl@ceph.mon.a.smithi003.stdout: index_type: 0 2024-08-17T17:12:29.534 INFO:journalctl@ceph.mon.a.smithi003.stdout: data_block_index_type: 0 2024-08-17T17:12:29.534 INFO:journalctl@ceph.mon.a.smithi003.stdout: index_shortening: 1 2024-08-17T17:12:29.534 INFO:journalctl@ceph.mon.a.smithi003.stdout: data_block_hash_table_util_ratio: 0.750000 2024-08-17T17:12:29.534 INFO:journalctl@ceph.mon.a.smithi003.stdout: checksum: 4 2024-08-17T17:12:29.534 INFO:journalctl@ceph.mon.a.smithi003.stdout: no_block_cache: 0 2024-08-17T17:12:29.534 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache: 0x5624c391f610 2024-08-17T17:12:29.534 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache_name: BinnedLRUCache 2024-08-17T17:12:29.534 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache_options: 2024-08-17T17:12:29.534 INFO:journalctl@ceph.mon.a.smithi003.stdout: capacity : 536870912 2024-08-17T17:12:29.535 INFO:journalctl@ceph.mon.a.smithi003.stdout: num_shard_bits : 4 2024-08-17T17:12:29.535 INFO:journalctl@ceph.mon.a.smithi003.stdout: strict_capacity_limit : 0 2024-08-17T17:12:29.535 INFO:journalctl@ceph.mon.a.smithi003.stdout: high_pri_pool_ratio: 0.000 2024-08-17T17:12:29.535 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache_compressed: (nil) 2024-08-17T17:12:29.535 INFO:journalctl@ceph.mon.a.smithi003.stdout: persistent_cache: (nil) 2024-08-17T17:12:29.535 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_size: 4096 2024-08-17T17:12:29.535 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_size_deviation: 10 2024-08-17T17:12:29.535 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_restart_interval: 16 2024-08-17T17:12:29.535 INFO:journalctl@ceph.mon.a.smithi003.stdout: index_block_restart_interval: 1 2024-08-17T17:12:29.535 INFO:journalctl@ceph.mon.a.smithi003.stdout: metadata_block_size: 4096 2024-08-17T17:12:29.535 INFO:journalctl@ceph.mon.a.smithi003.stdout: partition_filters: 0 2024-08-17T17:12:29.535 INFO:journalctl@ceph.mon.a.smithi003.stdout: use_delta_encoding: 1 2024-08-17T17:12:29.535 INFO:journalctl@ceph.mon.a.smithi003.stdout: filter_policy: bloomfilter 2024-08-17T17:12:29.535 INFO:journalctl@ceph.mon.a.smithi003.stdout: whole_key_filtering: 1 2024-08-17T17:12:29.536 INFO:journalctl@ceph.mon.a.smithi003.stdout: verify_compression: 0 2024-08-17T17:12:29.536 INFO:journalctl@ceph.mon.a.smithi003.stdout: read_amp_bytes_per_bit: 0 2024-08-17T17:12:29.536 INFO:journalctl@ceph.mon.a.smithi003.stdout: format_version: 5 2024-08-17T17:12:29.536 INFO:journalctl@ceph.mon.a.smithi003.stdout: enable_index_compression: 1 2024-08-17T17:12:29.536 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_align: 0 2024-08-17T17:12:29.536 INFO:journalctl@ceph.mon.a.smithi003.stdout: max_auto_readahead_size: 262144 2024-08-17T17:12:29.536 INFO:journalctl@ceph.mon.a.smithi003.stdout: prepopulate_block_cache: 0 2024-08-17T17:12:29.536 INFO:journalctl@ceph.mon.a.smithi003.stdout: initial_auto_readahead_size: 8192 2024-08-17T17:12:29.536 INFO:journalctl@ceph.mon.a.smithi003.stdout: num_file_reads_for_auto_readahead: 2 2024-08-17T17:12:29.536 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.write_buffer_size: 33554432 2024-08-17T17:12:29.536 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.max_write_buffer_number: 2 2024-08-17T17:12:29.536 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.compression: NoCompression 2024-08-17T17:12:29.536 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.bottommost_compression: Disabled 2024-08-17T17:12:29.536 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.prefix_extractor: nullptr 2024-08-17T17:12:29.536 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-08-17T17:12:29.537 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.num_levels: 7 2024-08-17T17:12:29.537 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-08-17T17:12:29.537 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-08-17T17:12:29.537 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-08-17T17:12:29.537 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-08-17T17:12:29.537 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-08-17T17:12:29.537 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-08-17T17:12:29.537 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-08-17T17:12:29.537 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-08-17T17:12:29.537 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-08-17T17:12:29.537 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-08-17T17:12:29.537 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-08-17T17:12:29.537 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-08-17T17:12:29.537 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.compression_opts.window_bits: -14 2024-08-17T17:12:29.538 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.compression_opts.level: 32767 2024-08-17T17:12:29.538 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.compression_opts.strategy: 0 2024-08-17T17:12:29.538 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-08-17T17:12:29.538 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-08-17T17:12:29.538 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-08-17T17:12:29.538 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-08-17T17:12:29.538 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.compression_opts.enabled: false 2024-08-17T17:12:29.538 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-08-17T17:12:29.538 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-08-17T17:12:29.538 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-08-17T17:12:29.538 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-08-17T17:12:29.538 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.target_file_size_base: 67108864 2024-08-17T17:12:29.538 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.target_file_size_multiplier: 1 2024-08-17T17:12:29.539 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-08-17T17:12:29.539 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-08-17T17:12:29.539 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-08-17T17:12:29.539 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-08-17T17:12:29.539 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-08-17T17:12:29.539 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-08-17T17:12:29.539 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-08-17T17:12:29.539 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-08-17T17:12:29.539 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-08-17T17:12:29.539 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-08-17T17:12:29.539 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-08-17T17:12:29.539 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-08-17T17:12:29.539 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-08-17T17:12:29.539 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.arena_block_size: 1048576 2024-08-17T17:12:29.540 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-08-17T17:12:29.540 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-08-17T17:12:29.540 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.disable_auto_compactions: 0 2024-08-17T17:12:29.540 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-08-17T17:12:29.540 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-08-17T17:12:29.540 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-08-17T17:12:29.540 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-08-17T17:12:29.540 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-08-17T17:12:29.540 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-08-17T17:12:29.540 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-08-17T17:12:29.540 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-08-17T17:12:29.540 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-08-17T17:12:29.540 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-08-17T17:12:29.540 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-08-17T17:12:29.541 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.inplace_update_support: 0 2024-08-17T17:12:29.541 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.inplace_update_num_locks: 10000 2024-08-17T17:12:29.541 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-08-17T17:12:29.541 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-08-17T17:12:29.541 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.memtable_huge_page_size: 0 2024-08-17T17:12:29.541 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.bloom_locality: 0 2024-08-17T17:12:29.541 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.max_successive_merges: 0 2024-08-17T17:12:29.541 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.optimize_filters_for_hits: 0 2024-08-17T17:12:29.541 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.paranoid_file_checks: 0 2024-08-17T17:12:29.541 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.force_consistency_checks: 1 2024-08-17T17:12:29.541 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.report_bg_io_stats: 0 2024-08-17T17:12:29.541 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.ttl: 2592000 2024-08-17T17:12:29.541 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.periodic_compaction_seconds: 0 2024-08-17T17:12:29.542 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-08-17T17:12:29.542 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-08-17T17:12:29.542 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.enable_blob_files: false 2024-08-17T17:12:29.542 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.min_blob_size: 0 2024-08-17T17:12:29.542 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.blob_file_size: 268435456 2024-08-17T17:12:29.542 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.blob_compression_type: NoCompression 2024-08-17T17:12:29.542 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.enable_blob_garbage_collection: false 2024-08-17T17:12:29.542 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-08-17T17:12:29.542 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-08-17T17:12:29.542 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-08-17T17:12:29.542 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.blob_file_starting_level: 0 2024-08-17T17:12:29.542 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-08-17T17:12:29.542 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 33.sst 2024-08-17T17:12:29.542 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-08-17T17:12:29.543 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 35, last_sequence is 12278, log_number is 31,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-08-17T17:12:29.543 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 31 2024-08-17T17:12:29.543 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 00c0f85f-041b-452f-9aa9-de5ee3f16f85 2024-08-17T17:12:29.543 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723914749306569, "job": 1, "event": "recovery_started", "wal_files": [31]} 2024-08-17T17:12:29.543 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #31 mode 2 2024-08-17T17:12:29.543 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723914749309731, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 36, "file_size": 615408, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12280, "largest_seqno": 12648, "table_properties": {"data_size": 612754, "index_size": 1102, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 517, "raw_key_size": 4669, "raw_average_key_size": 24, "raw_value_size": 608639, "raw_average_value_size": 3220, "num_data_blocks": 50, "num_entries": 189, "num_filter_entries": 189, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1723914749, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "00c0f85f-041b-452f-9aa9-de5ee3f16f85", "db_session_id": "9LJ7FFS12MAM3NIK3BNB", "orig_file_number": 36, "seqno_to_time_mapping": "N/A"}} 2024-08-17T17:12:29.543 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723914749309844, "job": 1, "event": "recovery_finished"} 2024-08-17T17:12:29.543 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: [db/version_set.cc:5047] Creating manifest 38 2024-08-17T17:12:29.543 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-08-17T17:12:29.543 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000031.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-08-17T17:12:29.543 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5624c394ee00 2024-08-17T17:12:29.543 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: DB pointer 0x5624c3a4c000 2024-08-17T17:12:29.543 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-08-17T17:12:29.543 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-08-17T17:12:29.544 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** DB Stats ** 2024-08-17T17:12:29.544 INFO:journalctl@ceph.mon.a.smithi003.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-17T17:12:29.544 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-08-17T17:12:29.544 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-17T17:12:29.544 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-17T17:12:29.544 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-08-17T17:12:29.544 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-17T17:12:29.544 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-17T17:12:29.544 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-08-17T17:12:29.544 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** Compaction Stats [default] ** 2024-08-17T17:12:29.544 INFO:journalctl@ceph.mon.a.smithi003.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-17T17:12:29.544 INFO:journalctl@ceph.mon.a.smithi003.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-08-17T17:12:29.544 INFO:journalctl@ceph.mon.a.smithi003.stdout: L0 1/0 600.98 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 309.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-08-17T17:12:29.545 INFO:journalctl@ceph.mon.a.smithi003.stdout: L6 1/0 7.35 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-08-17T17:12:29.545 INFO:journalctl@ceph.mon.a.smithi003.stdout: Sum 2/0 7.93 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 309.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-08-17T17:12:29.545 INFO:journalctl@ceph.mon.a.smithi003.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 309.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-08-17T17:12:29.545 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-08-17T17:12:29.545 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** Compaction Stats [default] ** 2024-08-17T17:12:29.545 INFO:journalctl@ceph.mon.a.smithi003.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-17T17:12:29.545 INFO:journalctl@ceph.mon.a.smithi003.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-08-17T17:12:29.545 INFO:journalctl@ceph.mon.a.smithi003.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 309.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-08-17T17:12:29.545 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-08-17T17:12:29.545 INFO:journalctl@ceph.mon.a.smithi003.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-08-17T17:12:29.545 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-08-17T17:12:29.545 INFO:journalctl@ceph.mon.a.smithi003.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-17T17:12:29.545 INFO:journalctl@ceph.mon.a.smithi003.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-08-17T17:12:29.546 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-08-17T17:12:29.546 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-08-17T17:12:29.546 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-08-17T17:12:29.546 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(Keys): cumulative 0, interval 0 2024-08-17T17:12:29.546 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative compaction: 0.00 GB write, 105.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-17T17:12:29.546 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval compaction: 0.00 GB write, 105.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-17T17:12:29.546 INFO:journalctl@ceph.mon.a.smithi003.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-08-17T17:12:29.546 INFO:journalctl@ceph.mon.a.smithi003.stdout: Block cache BinnedLRUCache@0x5624c391f610#2 capacity: 512.00 MB usage: 1.80 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-08-17T17:12:29.546 INFO:journalctl@ceph.mon.a.smithi003.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.59 KB,0.000113249%) IndexBlock(1,1.20 KB,0.000229478%) Misc(1,0.00 KB,0%) 2024-08-17T17:12:29.546 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-08-17T17:12:29.546 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** File Read Latency Histogram By Level [default] ** 2024-08-17T17:12:29.546 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: starting mon.a rank 0 at public addrs [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] at bind addrs [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:12:29.546 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: mon.a@-1(???) e3 preinit fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:12:29.546 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: mon.a@-1(???).mds e9 new map 2024-08-17T17:12:29.547 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: mon.a@-1(???).mds e9 print_map 2024-08-17T17:12:29.547 INFO:journalctl@ceph.mon.a.smithi003.stdout: e9 2024-08-17T17:12:29.547 INFO:journalctl@ceph.mon.a.smithi003.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-08-17T17:12:29.547 INFO:journalctl@ceph.mon.a.smithi003.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-08-17T17:12:29.547 INFO:journalctl@ceph.mon.a.smithi003.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-08-17T17:12:29.547 INFO:journalctl@ceph.mon.a.smithi003.stdout: legacy client fscid: 1 2024-08-17T17:12:29.547 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-08-17T17:12:29.547 INFO:journalctl@ceph.mon.a.smithi003.stdout: Filesystem 'foo' (1) 2024-08-17T17:12:29.547 INFO:journalctl@ceph.mon.a.smithi003.stdout: fs_name foo 2024-08-17T17:12:29.547 INFO:journalctl@ceph.mon.a.smithi003.stdout: epoch 8 2024-08-17T17:12:29.547 INFO:journalctl@ceph.mon.a.smithi003.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-08-17T17:12:29.547 INFO:journalctl@ceph.mon.a.smithi003.stdout: created 2024-08-17T17:07:46.147672+0000 2024-08-17T17:12:29.547 INFO:journalctl@ceph.mon.a.smithi003.stdout: modified 2024-08-17T17:07:55.190877+0000 2024-08-17T17:12:29.547 INFO:journalctl@ceph.mon.a.smithi003.stdout: tableserver 0 2024-08-17T17:12:29.548 INFO:journalctl@ceph.mon.a.smithi003.stdout: root 0 2024-08-17T17:12:29.548 INFO:journalctl@ceph.mon.a.smithi003.stdout: session_timeout 60 2024-08-17T17:12:29.548 INFO:journalctl@ceph.mon.a.smithi003.stdout: session_autoclose 300 2024-08-17T17:12:29.548 INFO:journalctl@ceph.mon.a.smithi003.stdout: max_file_size 1099511627776 2024-08-17T17:12:29.548 INFO:journalctl@ceph.mon.a.smithi003.stdout: max_xattr_size 65536 2024-08-17T17:12:29.548 INFO:journalctl@ceph.mon.a.smithi003.stdout: required_client_features {} 2024-08-17T17:12:29.548 INFO:journalctl@ceph.mon.a.smithi003.stdout: last_failure 0 2024-08-17T17:12:29.548 INFO:journalctl@ceph.mon.a.smithi003.stdout: last_failure_osd_epoch 0 2024-08-17T17:12:29.548 INFO:journalctl@ceph.mon.a.smithi003.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-08-17T17:12:29.548 INFO:journalctl@ceph.mon.a.smithi003.stdout: max_mds 1 2024-08-17T17:12:29.548 INFO:journalctl@ceph.mon.a.smithi003.stdout: in 0 2024-08-17T17:12:29.548 INFO:journalctl@ceph.mon.a.smithi003.stdout: up {0=14664} 2024-08-17T17:12:29.548 INFO:journalctl@ceph.mon.a.smithi003.stdout: failed 2024-08-17T17:12:29.549 INFO:journalctl@ceph.mon.a.smithi003.stdout: damaged 2024-08-17T17:12:29.549 INFO:journalctl@ceph.mon.a.smithi003.stdout: stopped 2024-08-17T17:12:29.549 INFO:journalctl@ceph.mon.a.smithi003.stdout: data_pools [4] 2024-08-17T17:12:29.549 INFO:journalctl@ceph.mon.a.smithi003.stdout: metadata_pool 3 2024-08-17T17:12:29.549 INFO:journalctl@ceph.mon.a.smithi003.stdout: inline_data disabled 2024-08-17T17:12:29.549 INFO:journalctl@ceph.mon.a.smithi003.stdout: balancer 2024-08-17T17:12:29.549 INFO:journalctl@ceph.mon.a.smithi003.stdout: bal_rank_mask -1 2024-08-17T17:12:29.549 INFO:journalctl@ceph.mon.a.smithi003.stdout: standby_count_wanted 1 2024-08-17T17:12:29.549 INFO:journalctl@ceph.mon.a.smithi003.stdout: qdb_cluster leader: 0 members: 2024-08-17T17:12:29.549 INFO:journalctl@ceph.mon.a.smithi003.stdout: [mds.foo.smithi003.civgvf{0:14664} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.3:6834/2800254789,v1:172.21.15.3:6835/2800254789] compat {c=[1],r=[1],i=[7ff]}] 2024-08-17T17:12:29.549 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-08-17T17:12:29.549 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-08-17T17:12:29.549 INFO:journalctl@ceph.mon.a.smithi003.stdout: Standby daemons: 2024-08-17T17:12:29.549 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-08-17T17:12:29.550 INFO:journalctl@ceph.mon.a.smithi003.stdout: [mds.foo.smithi053.ougbtc{-1:24511} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.53:6832/264432371,v1:172.21.15.53:6833/264432371] compat {c=[1],r=[1],i=[7ff]}] 2024-08-17T17:12:29.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: mon.a@-1(???).osd e89 crush map has features 3314933000854323200, adjusting msgr requires 2024-08-17T17:12:29.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: mon.a@-1(???).osd e89 crush map has features 432629239337189376, adjusting msgr requires 2024-08-17T17:12:29.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: mon.a@-1(???).osd e89 crush map has features 432629239337189376, adjusting msgr requires 2024-08-17T17:12:29.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: mon.a@-1(???).osd e89 crush map has features 432629239337189376, adjusting msgr requires 2024-08-17T17:12:29.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 ceph-mon[100615]: mon.a@-1(???).paxosservice(auth 1..18) refresh upgraded, format 0 -> 3 2024-08-17T17:12:29.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 bash[100596]: c1cfbc2d47f8446b609e615b86be102c47a3b842c3e7bab3ca9290f7353dff3d 2024-08-17T17:12:29.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:29 smithi003 systemd[1]: Started Ceph mon.a for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:12:30.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:12:31.339 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:12:32.716 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:32 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[81335]: ignoring --setuser ceph since I am not root 2024-08-17T17:12:32.716 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:32 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[81335]: ignoring --setgroup ceph since I am not root 2024-08-17T17:12:32.717 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:32 smithi053 ceph-mgr[81339]: -- 172.21.15.53:0/1318377574 <== mon.1 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5591a5b610e0 con 0x5591a5b3e800 2024-08-17T17:12:32.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:32 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/1030055260' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:32.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:32 smithi053 ceph-mon[34699]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:32.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:32 smithi053 ceph-mon[34699]: from='client.26264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:32.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:32 smithi053 ceph-mon[34699]: from='client.26270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:32.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:32 smithi053 ceph-mon[34699]: pgmap v20: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:32.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:32 smithi053 ceph-mon[34699]: mon.a calling monitor election 2024-08-17T17:12:32.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:32 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/1003424646' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:32.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:32 smithi053 ceph-mon[34699]: from='client.25867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:32.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:32 smithi053 ceph-mon[34699]: from='client.25873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:32.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:32 smithi053 ceph-mon[34699]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-08-17T17:12:32.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:32 smithi053 ceph-mon[34699]: monmap epoch 3 2024-08-17T17:12:32.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:32 smithi053 ceph-mon[34699]: fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:12:32.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:32 smithi053 ceph-mon[34699]: last_changed 2024-08-17T17:02:32.825077+0000 2024-08-17T17:12:32.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:32 smithi053 ceph-mon[34699]: created 2024-08-17T17:00:53.391169+0000 2024-08-17T17:12:32.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:32 smithi053 ceph-mon[34699]: min_mon_release 17 (quincy) 2024-08-17T17:12:32.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:32 smithi053 ceph-mon[34699]: election_strategy: 1 2024-08-17T17:12:32.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:32 smithi053 ceph-mon[34699]: 0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2024-08-17T17:12:32.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:32 smithi053 ceph-mon[34699]: 1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-08-17T17:12:32.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:32 smithi053 ceph-mon[34699]: 2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2024-08-17T17:12:32.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:32 smithi053 ceph-mon[34699]: fsmap foo:1 {0=foo.smithi003.civgvf=up:active} 1 up:standby 2024-08-17T17:12:32.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:32 smithi053 ceph-mon[34699]: osdmap e89: 8 total, 8 up, 8 in 2024-08-17T17:12:32.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:32 smithi053 ceph-mon[34699]: mgrmap e33: y(active, since 36s), standbys: x 2024-08-17T17:12:32.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:32 smithi053 ceph-mon[34699]: overall HEALTH_OK 2024-08-17T17:12:32.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:32 smithi053 ceph-mon[34699]: from='mgr.25979 ' entity='' 2024-08-17T17:12:32.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:32 smithi053 ceph-mon[34699]: mgrmap e34: y(active, since 36s), standbys: x 2024-08-17T17:12:32.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ignoring --setuser ceph since I am not root 2024-08-17T17:12:32.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ignoring --setgroup ceph since I am not root 2024-08-17T17:12:32.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mgr[91450]: -- 172.21.15.3:0/1423474364 <== mon.2 v2:172.21.15.3:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e7cf9f10e0 con 0x55e7cf9cf000 2024-08-17T17:12:32.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1030055260' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:32.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[100615]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:32.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[100615]: from='client.26264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:32.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[100615]: from='client.26270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:32.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[100615]: pgmap v20: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:32.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[100615]: mon.a calling monitor election 2024-08-17T17:12:32.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1003424646' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:32.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[100615]: from='client.25867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:32.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[100615]: from='client.25873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:32.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[100615]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-08-17T17:12:32.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[100615]: monmap epoch 3 2024-08-17T17:12:32.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[100615]: fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:12:32.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[100615]: last_changed 2024-08-17T17:02:32.825077+0000 2024-08-17T17:12:32.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[100615]: created 2024-08-17T17:00:53.391169+0000 2024-08-17T17:12:32.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[100615]: min_mon_release 17 (quincy) 2024-08-17T17:12:32.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[100615]: election_strategy: 1 2024-08-17T17:12:32.775 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[100615]: 0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2024-08-17T17:12:32.775 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[100615]: 1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-08-17T17:12:32.775 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[100615]: 2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2024-08-17T17:12:32.775 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[100615]: fsmap foo:1 {0=foo.smithi003.civgvf=up:active} 1 up:standby 2024-08-17T17:12:32.775 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[100615]: osdmap e89: 8 total, 8 up, 8 in 2024-08-17T17:12:32.775 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[100615]: mgrmap e33: y(active, since 36s), standbys: x 2024-08-17T17:12:32.775 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[100615]: overall HEALTH_OK 2024-08-17T17:12:32.775 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[100615]: from='mgr.25979 ' entity='' 2024-08-17T17:12:32.775 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[100615]: mgrmap e34: y(active, since 36s), standbys: x 2024-08-17T17:12:32.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/1030055260' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:32.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[41530]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:32.776 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[41530]: from='client.26264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:32.776 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[41530]: from='client.26270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:32.776 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[41530]: pgmap v20: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:32.776 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[41530]: mon.a calling monitor election 2024-08-17T17:12:32.776 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/1003424646' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:32.776 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[41530]: from='client.25867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:32.776 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[41530]: from='client.25873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:32.776 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[41530]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-08-17T17:12:32.776 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[41530]: monmap epoch 3 2024-08-17T17:12:32.776 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[41530]: fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:12:32.776 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[41530]: last_changed 2024-08-17T17:02:32.825077+0000 2024-08-17T17:12:32.776 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[41530]: created 2024-08-17T17:00:53.391169+0000 2024-08-17T17:12:32.776 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[41530]: min_mon_release 17 (quincy) 2024-08-17T17:12:32.776 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[41530]: election_strategy: 1 2024-08-17T17:12:32.777 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[41530]: 0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2024-08-17T17:12:32.777 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[41530]: 1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-08-17T17:12:32.777 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[41530]: 2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2024-08-17T17:12:32.777 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[41530]: fsmap foo:1 {0=foo.smithi003.civgvf=up:active} 1 up:standby 2024-08-17T17:12:32.777 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[41530]: osdmap e89: 8 total, 8 up, 8 in 2024-08-17T17:12:32.777 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[41530]: mgrmap e33: y(active, since 36s), standbys: x 2024-08-17T17:12:32.777 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[41530]: overall HEALTH_OK 2024-08-17T17:12:32.777 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[41530]: from='mgr.25979 ' entity='' 2024-08-17T17:12:32.777 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:32 smithi003 ceph-mon[41530]: mgrmap e34: y(active, since 36s), standbys: x 2024-08-17T17:12:34.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:34 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/2206380614' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-08-17T17:12:34.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:34 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:34.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:12:34.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:34 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2206380614' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-08-17T17:12:34.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:34 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/2206380614' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-08-17T17:12:37.217 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:36 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:36.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi053\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:12:37.966 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[81335]: [17/Aug/2024:17:12:37] ENGINE Bus STARTING 2024-08-17T17:12:37.967 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[81335]: CherryPy Checker: 2024-08-17T17:12:37.967 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[81335]: The Application mounted at '' has an empty config. 2024-08-17T17:12:37.967 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[81335]: 2024-08-17T17:12:37.967 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[81335]: [17/Aug/2024:17:12:37] ENGINE Serving on http://:::9283 2024-08-17T17:12:37.967 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x[81335]: [17/Aug/2024:17:12:37] ENGINE Bus STARTED 2024-08-17T17:12:37.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-mon[34699]: Standby manager daemon x restarted 2024-08-17T17:12:37.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-mon[34699]: Standby manager daemon x started 2024-08-17T17:12:37.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.53:0/2765398458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:12:37.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.53:0/2765398458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:12:37.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.53:0/2765398458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:12:37.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-mon[34699]: from='mgr.? 172.21.15.53:0/2765398458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:12:37.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-mon[34699]: Active manager daemon y restarted 2024-08-17T17:12:37.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-mon[34699]: Activating manager daemon y 2024-08-17T17:12:37.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-mon[34699]: osdmap e90: 8 total, 8 up, 8 in 2024-08-17T17:12:37.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-mon[34699]: mgrmap e35: y(active, starting, since 0.00968559s), standbys: x 2024-08-17T17:12:37.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-17T17:12:37.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-17T17:12:37.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-17T17:12:37.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi003.civgvf"} : dispatch 2024-08-17T17:12:37.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi053.ougbtc"} : dispatch 2024-08-17T17:12:37.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-08-17T17:12:37.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-08-17T17:12:37.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-17T17:12:37.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-17T17:12:37.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-17T17:12:37.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-17T17:12:37.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-17T17:12:37.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-17T17:12:37.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-17T17:12:37.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-17T17:12:37.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-08-17T17:12:37.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-08-17T17:12:37.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-08-17T17:12:37.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:37 smithi053 ceph-mon[34699]: Manager daemon y is now available 2024-08-17T17:12:38.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[41530]: Standby manager daemon x restarted 2024-08-17T17:12:38.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[41530]: Standby manager daemon x started 2024-08-17T17:12:38.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.53:0/2765398458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:12:38.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.53:0/2765398458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:12:38.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.53:0/2765398458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:12:38.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[41530]: from='mgr.? 172.21.15.53:0/2765398458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:12:38.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[41530]: Active manager daemon y restarted 2024-08-17T17:12:38.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[41530]: Activating manager daemon y 2024-08-17T17:12:38.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[41530]: osdmap e90: 8 total, 8 up, 8 in 2024-08-17T17:12:38.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[41530]: mgrmap e35: y(active, starting, since 0.00968559s), standbys: x 2024-08-17T17:12:38.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-17T17:12:38.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-17T17:12:38.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-17T17:12:38.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi003.civgvf"} : dispatch 2024-08-17T17:12:38.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi053.ougbtc"} : dispatch 2024-08-17T17:12:38.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-08-17T17:12:38.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-08-17T17:12:38.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-17T17:12:38.025 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-17T17:12:38.025 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-17T17:12:38.025 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-17T17:12:38.025 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-17T17:12:38.025 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-17T17:12:38.025 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-17T17:12:38.025 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-17T17:12:38.025 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-08-17T17:12:38.025 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-08-17T17:12:38.025 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-08-17T17:12:38.025 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[41530]: Manager daemon y is now available 2024-08-17T17:12:38.025 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[100615]: Standby manager daemon x restarted 2024-08-17T17:12:38.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[100615]: Standby manager daemon x started 2024-08-17T17:12:38.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[100615]: from='mgr.? 172.21.15.53:0/2765398458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-08-17T17:12:38.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[100615]: from='mgr.? 172.21.15.53:0/2765398458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-08-17T17:12:38.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[100615]: from='mgr.? 172.21.15.53:0/2765398458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-08-17T17:12:38.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[100615]: from='mgr.? 172.21.15.53:0/2765398458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-08-17T17:12:38.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[100615]: Active manager daemon y restarted 2024-08-17T17:12:38.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[100615]: Activating manager daemon y 2024-08-17T17:12:38.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[100615]: osdmap e90: 8 total, 8 up, 8 in 2024-08-17T17:12:38.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[100615]: mgrmap e35: y(active, starting, since 0.00968559s), standbys: x 2024-08-17T17:12:38.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-17T17:12:38.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-17T17:12:38.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-17T17:12:38.027 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi003.civgvf"} : dispatch 2024-08-17T17:12:38.027 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi053.ougbtc"} : dispatch 2024-08-17T17:12:38.027 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-08-17T17:12:38.027 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-08-17T17:12:38.027 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-17T17:12:38.027 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-17T17:12:38.027 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-17T17:12:38.027 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-17T17:12:38.027 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-17T17:12:38.027 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-17T17:12:38.027 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-17T17:12:38.027 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-17T17:12:38.027 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-08-17T17:12:38.027 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-08-17T17:12:38.028 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-08-17T17:12:38.028 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:37 smithi003 ceph-mon[100615]: Manager daemon y is now available 2024-08-17T17:12:38.446 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:38 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:12:38.085+0000 7f285a14a640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:12:38.446 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:38 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:12:38.085+0000 7f285a14a640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:12:38.446 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:38 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:12:38.085+0000 7f285a14a640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:12:38.446 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:38 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:12:38.085+0000 7f285a14a640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:12:38.446 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:38 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:12:38.085+0000 7f285a14a640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:12:38.446 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:38 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:12:38.087+0000 7f285f154640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:12:38.446 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:38 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:12:38.087+0000 7f285f154640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:12:38.447 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:38 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:12:38.087+0000 7f285f154640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:12:38.447 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:38 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:12:38.087+0000 7f285f154640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:12:38.447 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:38 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:12:38.087+0000 7f285f154640 -1 client.0 error registering admin socket command: (17) File exists 2024-08-17T17:12:38.626 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:12:38.719 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:38 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:12:38.719 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:38 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:12:38.720 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:38 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-08-17T17:12:38.720 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:38 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-08-17T17:12:38.720 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:38 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-08-17T17:12:38.720 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:38 smithi003 ceph-mon[41530]: mgrmap e36: y(active, since 1.01434s), standbys: x 2024-08-17T17:12:38.720 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:12:38.720 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:12:38.720 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-08-17T17:12:38.720 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-08-17T17:12:38.720 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-08-17T17:12:38.721 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:38 smithi003 ceph-mon[100615]: mgrmap e36: y(active, since 1.01434s), standbys: x 2024-08-17T17:12:38.721 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:38 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:12:38.612+0000 7f288dc05640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-08-17T17:12:38.721 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:38 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: [17/Aug/2024:17:12:38] ENGINE Bus STARTING 2024-08-17T17:12:38.721 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:38 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: CherryPy Checker: 2024-08-17T17:12:38.721 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:38 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: The Application mounted at '' has an empty config. 2024-08-17T17:12:38.721 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:38 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:12:38.721 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:38 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: [17/Aug/2024:17:12:38] ENGINE Serving on http://:::9283 2024-08-17T17:12:38.721 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:38 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: [17/Aug/2024:17:12:38] ENGINE Bus STARTED 2024-08-17T17:12:38.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:38 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:12:38.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:38 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:12:38.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:38 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-08-17T17:12:38.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:38 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-08-17T17:12:38.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:38 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-08-17T17:12:38.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:38 smithi053 ceph-mon[34699]: mgrmap e36: y(active, since 1.01434s), standbys: x 2024-08-17T17:12:39.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:39 smithi003 ceph-mon[100615]: [17/Aug/2024:17:12:39] ENGINE Bus STARTING 2024-08-17T17:12:39.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:39 smithi003 ceph-mon[100615]: [17/Aug/2024:17:12:39] ENGINE Serving on http://172.21.15.3:8765 2024-08-17T17:12:39.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:39 smithi003 ceph-mon[41530]: [17/Aug/2024:17:12:39] ENGINE Bus STARTING 2024-08-17T17:12:39.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:39 smithi003 ceph-mon[41530]: [17/Aug/2024:17:12:39] ENGINE Serving on http://172.21.15.3:8765 2024-08-17T17:12:39.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:39 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:12:39.454+0000 7f283bf14640 -1 log_channel(cephadm) log [ERR] : [17/Aug/2024:17:12:39] ENGINE Error in HTTPServer.serve 2024-08-17T17:12:39.774 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:39 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: Traceback (most recent call last): 2024-08-17T17:12:39.774 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:39 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:12:39.774 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:39 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: self._connections.run(self.expiration_interval) 2024-08-17T17:12:39.774 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:39 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:12:39.774 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:39 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: self._run(expiration_interval) 2024-08-17T17:12:39.774 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:39 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:12:39.774 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:39 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:12:39.774 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:39 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:12:39.774 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:39 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:12:39.774 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:39 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:12:39.774 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:39 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: s = self.context.wrap_socket( 2024-08-17T17:12:39.774 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:39 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:12:39.775 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:39 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: return self.sslsocket_class._create( 2024-08-17T17:12:39.775 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:39 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:12:39.775 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:39 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: self.do_handshake() 2024-08-17T17:12:39.775 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:39 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:12:39.775 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:39 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: self._sslobj.do_handshake() 2024-08-17T17:12:39.775 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:39 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:12:39.775 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:39 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:12:39.938 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:39 smithi053 ceph-mon[34699]: [17/Aug/2024:17:12:39] ENGINE Bus STARTING 2024-08-17T17:12:39.939 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:39 smithi053 ceph-mon[34699]: [17/Aug/2024:17:12:39] ENGINE Serving on http://172.21.15.3:8765 2024-08-17T17:12:40.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:12:40.910 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:40 smithi003 ceph-mon[100615]: [17/Aug/2024:17:12:39] ENGINE Serving on https://172.21.15.3:7150 2024-08-17T17:12:40.911 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:40 smithi003 ceph-mon[100615]: [17/Aug/2024:17:12:39] ENGINE Bus STARTED 2024-08-17T17:12:40.911 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:40 smithi003 ceph-mon[100615]: [17/Aug/2024:17:12:39] ENGINE Error in HTTPServer.serve 2024-08-17T17:12:40.911 INFO:journalctl@ceph.mon.a.smithi003.stdout: Traceback (most recent call last): 2024-08-17T17:12:40.911 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:12:40.911 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._connections.run(self.expiration_interval) 2024-08-17T17:12:40.911 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:12:40.911 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._run(expiration_interval) 2024-08-17T17:12:40.911 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:12:40.911 INFO:journalctl@ceph.mon.a.smithi003.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:12:40.911 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:12:40.911 INFO:journalctl@ceph.mon.a.smithi003.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:12:40.911 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:12:40.911 INFO:journalctl@ceph.mon.a.smithi003.stdout: s = self.context.wrap_socket( 2024-08-17T17:12:40.912 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:12:40.912 INFO:journalctl@ceph.mon.a.smithi003.stdout: return self.sslsocket_class._create( 2024-08-17T17:12:40.912 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:12:40.912 INFO:journalctl@ceph.mon.a.smithi003.stdout: self.do_handshake() 2024-08-17T17:12:40.912 INFO:journalctl@ceph.mon.a.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:12:40.912 INFO:journalctl@ceph.mon.a.smithi003.stdout: self._sslobj.do_handshake() 2024-08-17T17:12:40.912 INFO:journalctl@ceph.mon.a.smithi003.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:12:40.912 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:40 smithi003 ceph-mon[100615]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:40.912 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:40 smithi003 ceph-mon[100615]: mgrmap e37: y(active, since 2s), standbys: x 2024-08-17T17:12:40.912 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:40 smithi003 ceph-mon[41530]: [17/Aug/2024:17:12:39] ENGINE Serving on https://172.21.15.3:7150 2024-08-17T17:12:40.913 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:40 smithi003 ceph-mon[41530]: [17/Aug/2024:17:12:39] ENGINE Bus STARTED 2024-08-17T17:12:40.913 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:40 smithi003 ceph-mon[41530]: [17/Aug/2024:17:12:39] ENGINE Error in HTTPServer.serve 2024-08-17T17:12:40.913 INFO:journalctl@ceph.mon.c.smithi003.stdout: Traceback (most recent call last): 2024-08-17T17:12:40.913 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:12:40.913 INFO:journalctl@ceph.mon.c.smithi003.stdout: self._connections.run(self.expiration_interval) 2024-08-17T17:12:40.913 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:12:40.913 INFO:journalctl@ceph.mon.c.smithi003.stdout: self._run(expiration_interval) 2024-08-17T17:12:40.913 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:12:40.913 INFO:journalctl@ceph.mon.c.smithi003.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:12:40.913 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:12:40.913 INFO:journalctl@ceph.mon.c.smithi003.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:12:40.913 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:12:40.913 INFO:journalctl@ceph.mon.c.smithi003.stdout: s = self.context.wrap_socket( 2024-08-17T17:12:40.913 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:12:40.914 INFO:journalctl@ceph.mon.c.smithi003.stdout: return self.sslsocket_class._create( 2024-08-17T17:12:40.914 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:12:40.914 INFO:journalctl@ceph.mon.c.smithi003.stdout: self.do_handshake() 2024-08-17T17:12:40.914 INFO:journalctl@ceph.mon.c.smithi003.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:12:40.914 INFO:journalctl@ceph.mon.c.smithi003.stdout: self._sslobj.do_handshake() 2024-08-17T17:12:40.914 INFO:journalctl@ceph.mon.c.smithi003.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:12:40.914 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:40 smithi003 ceph-mon[41530]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:40.914 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:40 smithi003 ceph-mon[41530]: mgrmap e37: y(active, since 2s), standbys: x 2024-08-17T17:12:40.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:40 smithi053 ceph-mon[34699]: [17/Aug/2024:17:12:39] ENGINE Serving on https://172.21.15.3:7150 2024-08-17T17:12:40.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:40 smithi053 ceph-mon[34699]: [17/Aug/2024:17:12:39] ENGINE Bus STARTED 2024-08-17T17:12:40.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:40 smithi053 ceph-mon[34699]: [17/Aug/2024:17:12:39] ENGINE Error in HTTPServer.serve 2024-08-17T17:12:40.967 INFO:journalctl@ceph.mon.b.smithi053.stdout: Traceback (most recent call last): 2024-08-17T17:12:40.967 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-17T17:12:40.967 INFO:journalctl@ceph.mon.b.smithi053.stdout: self._connections.run(self.expiration_interval) 2024-08-17T17:12:40.967 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-17T17:12:40.967 INFO:journalctl@ceph.mon.b.smithi053.stdout: self._run(expiration_interval) 2024-08-17T17:12:40.967 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-17T17:12:40.967 INFO:journalctl@ceph.mon.b.smithi053.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-17T17:12:40.967 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-17T17:12:40.967 INFO:journalctl@ceph.mon.b.smithi053.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-17T17:12:40.967 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-17T17:12:40.967 INFO:journalctl@ceph.mon.b.smithi053.stdout: s = self.context.wrap_socket( 2024-08-17T17:12:40.968 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-17T17:12:40.968 INFO:journalctl@ceph.mon.b.smithi053.stdout: return self.sslsocket_class._create( 2024-08-17T17:12:40.968 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-17T17:12:40.968 INFO:journalctl@ceph.mon.b.smithi053.stdout: self.do_handshake() 2024-08-17T17:12:40.968 INFO:journalctl@ceph.mon.b.smithi053.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-17T17:12:40.968 INFO:journalctl@ceph.mon.b.smithi053.stdout: self._sslobj.do_handshake() 2024-08-17T17:12:40.968 INFO:journalctl@ceph.mon.b.smithi053.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-17T17:12:40.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:40 smithi053 ceph-mon[34699]: pgmap v4: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:40.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:40 smithi053 ceph-mon[34699]: mgrmap e37: y(active, since 2s), standbys: x 2024-08-17T17:12:41.639 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:12:41.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:41 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/3781102630' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-08-17T17:12:41.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:41 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3781102630' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-08-17T17:12:41.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:41 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/3781102630' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-08-17T17:12:42.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:42 smithi053 ceph-mon[34699]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:42.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:42 smithi053 ceph-mon[34699]: from='client.34170 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:42.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:42 smithi053 ceph-mon[34699]: mgrmap e38: y(active, since 4s), standbys: x 2024-08-17T17:12:42.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:42 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:42.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:42 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:42.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:42 smithi053 ceph-mon[34699]: from='client.34176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:42.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:42 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:42.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:42 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:42.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:42 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:42.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:42 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:43.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:42 smithi003 ceph-mon[100615]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:43.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:42 smithi003 ceph-mon[100615]: from='client.34170 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:43.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:42 smithi003 ceph-mon[100615]: mgrmap e38: y(active, since 4s), standbys: x 2024-08-17T17:12:43.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:42 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:43.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:42 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:43.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:42 smithi003 ceph-mon[100615]: from='client.34176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:43.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:42 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:43.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:42 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:43.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:42 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:43.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:42 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:43.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:42 smithi003 ceph-mon[41530]: pgmap v5: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:43.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:42 smithi003 ceph-mon[41530]: from='client.34170 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:43.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:42 smithi003 ceph-mon[41530]: mgrmap e38: y(active, since 4s), standbys: x 2024-08-17T17:12:43.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:42 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:43.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:42 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:43.025 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:42 smithi003 ceph-mon[41530]: from='client.34176 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:43.025 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:42 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:43.025 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:42 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:43.025 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:42 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:43.025 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:42 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:44.463 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:44 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:44.463 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:44 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:44.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:44 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:44.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:44 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:44.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:44 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:44.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:44 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:44.612 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:12:45.301 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:45 smithi003 ceph-mon[100615]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:45.301 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:45 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/28971567' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:45.301 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:45 smithi003 ceph-mon[41530]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:45.301 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:45 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/28971567' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:45.312 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:45 smithi053 ceph-mon[34699]: pgmap v6: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:45.312 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:45 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/28971567' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:46.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:12:45] "GET /metrics HTTP/1.1" 200 31140 "" "Prometheus/2.51.0" 2024-08-17T17:12:46.103 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:45 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:45.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:12:46.103 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:45 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:45.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:12:46.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:46 smithi053 ceph-mon[34699]: from='client.26366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:46.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:46 smithi053 ceph-mon[34699]: from='client.26372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:46.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:46 smithi003 ceph-mon[41530]: from='client.26366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:46.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:46 smithi003 ceph-mon[41530]: from='client.26372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:46.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:46 smithi003 ceph-mon[100615]: from='client.26366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:46.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:46 smithi003 ceph-mon[100615]: from='client.26372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:47.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:46 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:46.947Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi053\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:12:47.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:47 smithi053 ceph-mon[34699]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-08-17T17:12:47.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:47 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:47.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:47 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:47.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:47 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:12:47.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:47 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:12:47.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:47 smithi003 ceph-mon[100615]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-08-17T17:12:47.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:47 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:47.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:47 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:47.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:47 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:12:47.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:47 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:12:47.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:47 smithi003 ceph-mon[41530]: pgmap v7: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-08-17T17:12:47.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:47 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:47.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:47 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:47.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:47 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:12:47.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:47 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:12:47.689 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:12:48.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[41530]: Updating smithi003:/etc/ceph/ceph.conf 2024-08-17T17:12:48.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[41530]: Updating smithi053:/etc/ceph/ceph.conf 2024-08-17T17:12:48.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[41530]: Updating smithi003:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:12:48.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[41530]: Updating smithi053:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:12:48.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[41530]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:12:48.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[41530]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:12:48.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[41530]: Updating smithi003:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:12:48.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[41530]: Updating smithi053:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:12:48.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[41530]: from='client.? 172.21.15.3:0/2735843643' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:48.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:48.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:48.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:48.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:48.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:48.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:12:48.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:12:48.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-08-17T17:12:48.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["c"]} : dispatch 2024-08-17T17:12:48.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[100615]: Updating smithi003:/etc/ceph/ceph.conf 2024-08-17T17:12:48.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[100615]: Updating smithi053:/etc/ceph/ceph.conf 2024-08-17T17:12:48.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[100615]: Updating smithi003:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:12:48.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[100615]: Updating smithi053:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:12:48.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[100615]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:12:48.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[100615]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:12:48.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[100615]: Updating smithi003:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:12:48.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[100615]: Updating smithi053:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:12:48.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2735843643' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:48.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:48.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:48.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:48.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:48.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:48.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:12:48.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:12:48.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-08-17T17:12:48.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:48 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["c"]} : dispatch 2024-08-17T17:12:48.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:48 smithi053 ceph-mon[34699]: Updating smithi003:/etc/ceph/ceph.conf 2024-08-17T17:12:48.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:48 smithi053 ceph-mon[34699]: Updating smithi053:/etc/ceph/ceph.conf 2024-08-17T17:12:48.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:48 smithi053 ceph-mon[34699]: Updating smithi003:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:12:48.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:48 smithi053 ceph-mon[34699]: Updating smithi053:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.conf 2024-08-17T17:12:48.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:48 smithi053 ceph-mon[34699]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:12:48.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:48 smithi053 ceph-mon[34699]: Updating smithi053:/etc/ceph/ceph.client.admin.keyring 2024-08-17T17:12:48.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:48 smithi053 ceph-mon[34699]: Updating smithi003:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:12:48.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:48 smithi053 ceph-mon[34699]: Updating smithi053:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/config/ceph.client.admin.keyring 2024-08-17T17:12:48.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:48 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/2735843643' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:48.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:48 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:48.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:48 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:48.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:48 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:48.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:48 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:48.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:48 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:48.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:48 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:12:48.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:48 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:12:48.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:48 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-08-17T17:12:48.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:48 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["c"]} : dispatch 2024-08-17T17:12:49.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:49 smithi003 ceph-mon[41530]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-08-17T17:12:49.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:49 smithi003 ceph-mon[41530]: Upgrade: It appears safe to stop mon.c 2024-08-17T17:12:49.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:49 smithi003 ceph-mon[41530]: from='client.34206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:49.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:49 smithi003 ceph-mon[41530]: from='client.34212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:49.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:49 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:49.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:49 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-17T17:12:49.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:49 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-17T17:12:49.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:49 smithi003 ceph-mon[41530]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:12:49.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:49 smithi003 ceph-mon[100615]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-08-17T17:12:49.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:49 smithi003 ceph-mon[100615]: Upgrade: It appears safe to stop mon.c 2024-08-17T17:12:49.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:49 smithi003 ceph-mon[100615]: from='client.34206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:49.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:49 smithi003 ceph-mon[100615]: from='client.34212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:49.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:49 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:49.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:49 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-17T17:12:49.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:49 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-17T17:12:49.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:49 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:12:49.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:49 smithi053 ceph-mon[34699]: pgmap v8: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-08-17T17:12:49.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:49 smithi053 ceph-mon[34699]: Upgrade: It appears safe to stop mon.c 2024-08-17T17:12:49.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:49 smithi053 ceph-mon[34699]: from='client.34206 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:49.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:49 smithi053 ceph-mon[34699]: from='client.34212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:49.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:49 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:49.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:49 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-17T17:12:49.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:49 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-17T17:12:49.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:49 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:12:50.451 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:50 smithi003 ceph-mon[41530]: Upgrade: Updating mon.c 2024-08-17T17:12:50.451 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:50 smithi003 ceph-mon[41530]: Deploying daemon mon.c on smithi003 2024-08-17T17:12:50.452 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:50 smithi003 ceph-mon[100615]: Upgrade: Updating mon.c 2024-08-17T17:12:50.452 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:50 smithi003 ceph-mon[100615]: Deploying daemon mon.c on smithi003 2024-08-17T17:12:50.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:12:50.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:50 smithi053 ceph-mon[34699]: Upgrade: Updating mon.c 2024-08-17T17:12:50.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:50 smithi053 ceph-mon[34699]: Deploying daemon mon.c on smithi003 2024-08-17T17:12:50.674 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:12:51.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:50 smithi003 systemd[1]: Stopping Ceph mon.c for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:12:51.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:50 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-c[41525]: 2024-08-17T17:12:50.964+0000 7f750db12640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T17:12:51.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:50 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-c[41525]: 2024-08-17T17:12:50.964+0000 7f750db12640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-08-17T17:12:51.257 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:51 smithi003 podman[103180]: 2024-08-17 17:12:51.005389918 +0000 UTC m=+0.128102344 container died 3cd9ac5ad1afc1a22f4512bc6c47b3a1e1153bd3c43babb782f8a9972ac5dd3b (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-c, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=quincy-f2c39d4, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-08-17T17:12:51.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:51 smithi003 podman[103180]: 2024-08-17 17:12:51.372243047 +0000 UTC m=+0.494955473 container cleanup 3cd9ac5ad1afc1a22f4512bc6c47b3a1e1153bd3c43babb782f8a9972ac5dd3b (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-c, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=quincy-f2c39d4, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-08-17T17:12:51.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:51 smithi003 bash[103180]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-c 2024-08-17T17:12:52.018 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:51 smithi003 podman[103194]: 2024-08-17 17:12:51.623649082 +0000 UTC m=+0.620569843 container remove 3cd9ac5ad1afc1a22f4512bc6c47b3a1e1153bd3c43babb782f8a9972ac5dd3b (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-c, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=quincy-f2c39d4, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-08-17T17:12:52.018 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:51 smithi003 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mon.c.service: Deactivated successfully. 2024-08-17T17:12:52.018 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:51 smithi003 systemd[1]: Stopped Ceph mon.c for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:12:52.019 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:51 smithi003 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mon.c.service: Consumed 7.815s CPU time. 2024-08-17T17:12:52.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 systemd[1]: Starting Ceph mon.c for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:12:52.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 podman[103325]: 2024-08-17 17:12:52.140572479 +0000 UTC m=+0.018176616 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:12:52.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 podman[103325]: 2024-08-17 17:12:52.259047722 +0000 UTC m=+0.136651884 container create 54a6cfe9eb9aa657457be3d687d1b78dcb57c3d174cda0c5dad6ac13004dccf2 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-c, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-cd8bdbe, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.36.0) 2024-08-17T17:12:52.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[100615]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-08-17T17:12:52.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3562970401' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:52.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 podman[103325]: 2024-08-17 17:12:52.47000508 +0000 UTC m=+0.347609214 container init 54a6cfe9eb9aa657457be3d687d1b78dcb57c3d174cda0c5dad6ac13004dccf2 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-c, RELEASE=main-cd8bdbe, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-08-17T17:12:52.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 podman[103325]: 2024-08-17 17:12:52.473630201 +0000 UTC m=+0.351234322 container start 54a6cfe9eb9aa657457be3d687d1b78dcb57c3d174cda0c5dad6ac13004dccf2 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-c, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=main-cd8bdbe, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-17T17:12:52.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: set uid:gid to 167:167 (ceph:ceph) 2024-08-17T17:12:52.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev), process ceph-mon, pid 2 2024-08-17T17:12:52.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: pidfile_write: ignore empty --pid-file 2024-08-17T17:12:52.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: load: jerasure load: lrc 2024-08-17T17:12:52.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: RocksDB version: 7.9.2 2024-08-17T17:12:52.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Git sha 0 2024-08-17T17:12:52.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Compile date 2024-08-15 20:43:31 2024-08-17T17:12:52.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: DB SUMMARY 2024-08-17T17:12:52.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: DB Session ID: 53FVU5RKM1QUWUJER6GN 2024-08-17T17:12:52.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: CURRENT file: CURRENT 2024-08-17T17:12:52.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: IDENTITY file: IDENTITY 2024-08-17T17:12:52.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1097 Bytes 2024-08-17T17:12:52.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000027.sst 2024-08-17T17:12:52.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000025.log size: 3736885 ; 2024-08-17T17:12:52.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.error_if_exists: 0 2024-08-17T17:12:52.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.create_if_missing: 0 2024-08-17T17:12:52.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.paranoid_checks: 1 2024-08-17T17:12:52.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.flush_verify_memtable_count: 1 2024-08-17T17:12:52.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-08-17T17:12:52.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-08-17T17:12:52.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.env: 0x55aa836e8be0 2024-08-17T17:12:52.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.fs: PosixFileSystem 2024-08-17T17:12:52.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.info_log: 0x55aa84a11700 2024-08-17T17:12:52.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.max_file_opening_threads: 16 2024-08-17T17:12:52.776 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.statistics: (nil) 2024-08-17T17:12:52.776 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.use_fsync: 0 2024-08-17T17:12:52.776 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.max_log_file_size: 0 2024-08-17T17:12:52.776 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-08-17T17:12:52.776 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.log_file_time_to_roll: 0 2024-08-17T17:12:52.776 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.keep_log_file_num: 1000 2024-08-17T17:12:52.776 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.recycle_log_file_num: 0 2024-08-17T17:12:52.776 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.allow_fallocate: 1 2024-08-17T17:12:52.776 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.allow_mmap_reads: 0 2024-08-17T17:12:52.776 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.allow_mmap_writes: 0 2024-08-17T17:12:52.777 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.use_direct_reads: 0 2024-08-17T17:12:52.777 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-08-17T17:12:52.777 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.create_missing_column_families: 0 2024-08-17T17:12:52.777 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.db_log_dir: 2024-08-17T17:12:52.777 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.wal_dir: 2024-08-17T17:12:52.777 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.table_cache_numshardbits: 6 2024-08-17T17:12:52.777 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.WAL_ttl_seconds: 0 2024-08-17T17:12:52.777 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.WAL_size_limit_MB: 0 2024-08-17T17:12:52.777 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-08-17T17:12:52.777 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-08-17T17:12:52.777 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.is_fd_close_on_exec: 1 2024-08-17T17:12:52.777 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.advise_random_on_open: 1 2024-08-17T17:12:52.777 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.db_write_buffer_size: 0 2024-08-17T17:12:52.778 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.write_buffer_manager: 0x55aa84a14e60 2024-08-17T17:12:52.778 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-08-17T17:12:52.778 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-08-17T17:12:52.778 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.use_adaptive_mutex: 0 2024-08-17T17:12:52.778 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.rate_limiter: (nil) 2024-08-17T17:12:52.778 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-08-17T17:12:52.778 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.wal_recovery_mode: 2 2024-08-17T17:12:52.778 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.enable_thread_tracking: 0 2024-08-17T17:12:52.778 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.enable_pipelined_write: 0 2024-08-17T17:12:52.778 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.unordered_write: 0 2024-08-17T17:12:52.778 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-08-17T17:12:52.778 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-08-17T17:12:52.779 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-08-17T17:12:52.779 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-08-17T17:12:52.779 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.row_cache: None 2024-08-17T17:12:52.779 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.wal_filter: None 2024-08-17T17:12:52.779 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-08-17T17:12:52.779 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.allow_ingest_behind: 0 2024-08-17T17:12:52.779 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.two_write_queues: 0 2024-08-17T17:12:52.779 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.manual_wal_flush: 0 2024-08-17T17:12:52.779 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.wal_compression: 0 2024-08-17T17:12:52.779 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.atomic_flush: 0 2024-08-17T17:12:52.779 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-08-17T17:12:52.779 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.persist_stats_to_disk: 0 2024-08-17T17:12:52.779 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.write_dbid_to_manifest: 0 2024-08-17T17:12:52.779 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.log_readahead_size: 0 2024-08-17T17:12:52.780 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-08-17T17:12:52.780 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.best_efforts_recovery: 0 2024-08-17T17:12:52.780 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-08-17T17:12:52.780 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-08-17T17:12:52.780 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.allow_data_in_errors: 0 2024-08-17T17:12:52.780 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.db_host_id: __hostname__ 2024-08-17T17:12:52.780 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.enforce_single_del_contracts: true 2024-08-17T17:12:52.780 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.max_background_jobs: 2 2024-08-17T17:12:52.780 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.max_background_compactions: -1 2024-08-17T17:12:52.780 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.max_subcompactions: 1 2024-08-17T17:12:52.780 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-08-17T17:12:52.780 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-08-17T17:12:52.780 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.delayed_write_rate : 16777216 2024-08-17T17:12:52.780 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.max_total_wal_size: 0 2024-08-17T17:12:52.780 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-08-17T17:12:52.781 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.stats_dump_period_sec: 600 2024-08-17T17:12:52.781 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.stats_persist_period_sec: 600 2024-08-17T17:12:52.781 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-08-17T17:12:52.781 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.max_open_files: -1 2024-08-17T17:12:52.781 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.bytes_per_sync: 0 2024-08-17T17:12:52.781 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.wal_bytes_per_sync: 0 2024-08-17T17:12:52.781 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.strict_bytes_per_sync: 0 2024-08-17T17:12:52.781 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.compaction_readahead_size: 0 2024-08-17T17:12:52.781 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.max_background_flushes: -1 2024-08-17T17:12:52.781 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Compression algorithms supported: 2024-08-17T17:12:52.781 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: kZSTD supported: 0 2024-08-17T17:12:52.781 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: kXpressCompression supported: 0 2024-08-17T17:12:52.781 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: kBZip2Compression supported: 0 2024-08-17T17:12:52.782 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-08-17T17:12:52.782 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: kLZ4Compression supported: 1 2024-08-17T17:12:52.782 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: kZlibCompression supported: 1 2024-08-17T17:12:52.782 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: kLZ4HCCompression supported: 1 2024-08-17T17:12:52.782 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: kSnappyCompression supported: 1 2024-08-17T17:12:52.782 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Fast CRC32 supported: Supported on x86 2024-08-17T17:12:52.782 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: DMutex implementation: pthread_mutex_t 2024-08-17T17:12:52.782 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-08-17T17:12:52.782 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-08-17T17:12:52.782 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-08-17T17:12:52.782 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.merge_operator: 2024-08-17T17:12:52.782 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.compaction_filter: None 2024-08-17T17:12:52.782 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.compaction_filter_factory: None 2024-08-17T17:12:52.783 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.sst_partitioner_factory: None 2024-08-17T17:12:52.783 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.memtable_factory: SkipListFactory 2024-08-17T17:12:52.783 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.table_factory: BlockBasedTable 2024-08-17T17:12:52.783 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55aa84a11380) 2024-08-17T17:12:52.783 INFO:journalctl@ceph.mon.c.smithi003.stdout: cache_index_and_filter_blocks: 1 2024-08-17T17:12:52.783 INFO:journalctl@ceph.mon.c.smithi003.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-08-17T17:12:52.783 INFO:journalctl@ceph.mon.c.smithi003.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-08-17T17:12:52.783 INFO:journalctl@ceph.mon.c.smithi003.stdout: pin_top_level_index_and_filter: 1 2024-08-17T17:12:52.783 INFO:journalctl@ceph.mon.c.smithi003.stdout: index_type: 0 2024-08-17T17:12:52.783 INFO:journalctl@ceph.mon.c.smithi003.stdout: data_block_index_type: 0 2024-08-17T17:12:52.783 INFO:journalctl@ceph.mon.c.smithi003.stdout: index_shortening: 1 2024-08-17T17:12:52.783 INFO:journalctl@ceph.mon.c.smithi003.stdout: data_block_hash_table_util_ratio: 0.750000 2024-08-17T17:12:52.783 INFO:journalctl@ceph.mon.c.smithi003.stdout: checksum: 4 2024-08-17T17:12:52.783 INFO:journalctl@ceph.mon.c.smithi003.stdout: no_block_cache: 0 2024-08-17T17:12:52.784 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_cache: 0x55aa84a05610 2024-08-17T17:12:52.784 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_cache_name: BinnedLRUCache 2024-08-17T17:12:52.784 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_cache_options: 2024-08-17T17:12:52.784 INFO:journalctl@ceph.mon.c.smithi003.stdout: capacity : 536870912 2024-08-17T17:12:52.784 INFO:journalctl@ceph.mon.c.smithi003.stdout: num_shard_bits : 4 2024-08-17T17:12:52.784 INFO:journalctl@ceph.mon.c.smithi003.stdout: strict_capacity_limit : 0 2024-08-17T17:12:52.784 INFO:journalctl@ceph.mon.c.smithi003.stdout: high_pri_pool_ratio: 0.000 2024-08-17T17:12:52.784 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_cache_compressed: (nil) 2024-08-17T17:12:52.784 INFO:journalctl@ceph.mon.c.smithi003.stdout: persistent_cache: (nil) 2024-08-17T17:12:52.784 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_size: 4096 2024-08-17T17:12:52.784 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_size_deviation: 10 2024-08-17T17:12:52.784 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_restart_interval: 16 2024-08-17T17:12:52.784 INFO:journalctl@ceph.mon.c.smithi003.stdout: index_block_restart_interval: 1 2024-08-17T17:12:52.784 INFO:journalctl@ceph.mon.c.smithi003.stdout: metadata_block_size: 4096 2024-08-17T17:12:52.785 INFO:journalctl@ceph.mon.c.smithi003.stdout: partition_filters: 0 2024-08-17T17:12:52.785 INFO:journalctl@ceph.mon.c.smithi003.stdout: use_delta_encoding: 1 2024-08-17T17:12:52.785 INFO:journalctl@ceph.mon.c.smithi003.stdout: filter_policy: bloomfilter 2024-08-17T17:12:52.785 INFO:journalctl@ceph.mon.c.smithi003.stdout: whole_key_filtering: 1 2024-08-17T17:12:52.785 INFO:journalctl@ceph.mon.c.smithi003.stdout: verify_compression: 0 2024-08-17T17:12:52.785 INFO:journalctl@ceph.mon.c.smithi003.stdout: read_amp_bytes_per_bit: 0 2024-08-17T17:12:52.785 INFO:journalctl@ceph.mon.c.smithi003.stdout: format_version: 5 2024-08-17T17:12:52.785 INFO:journalctl@ceph.mon.c.smithi003.stdout: enable_index_compression: 1 2024-08-17T17:12:52.785 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_align: 0 2024-08-17T17:12:52.785 INFO:journalctl@ceph.mon.c.smithi003.stdout: max_auto_readahead_size: 262144 2024-08-17T17:12:52.785 INFO:journalctl@ceph.mon.c.smithi003.stdout: prepopulate_block_cache: 0 2024-08-17T17:12:52.787 INFO:journalctl@ceph.mon.c.smithi003.stdout: initial_auto_readahead_size: 8192 2024-08-17T17:12:52.787 INFO:journalctl@ceph.mon.c.smithi003.stdout: num_file_reads_for_auto_readahead: 2 2024-08-17T17:12:52.787 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.write_buffer_size: 33554432 2024-08-17T17:12:52.787 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.max_write_buffer_number: 2 2024-08-17T17:12:52.787 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.compression: NoCompression 2024-08-17T17:12:52.787 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.bottommost_compression: Disabled 2024-08-17T17:12:52.787 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.prefix_extractor: nullptr 2024-08-17T17:12:52.788 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-08-17T17:12:52.788 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.num_levels: 7 2024-08-17T17:12:52.788 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-08-17T17:12:52.788 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-08-17T17:12:52.788 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-08-17T17:12:52.788 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-08-17T17:12:52.788 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-08-17T17:12:52.788 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-08-17T17:12:52.788 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-08-17T17:12:52.788 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-08-17T17:12:52.789 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-08-17T17:12:52.789 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-08-17T17:12:52.789 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-08-17T17:12:52.789 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-08-17T17:12:52.789 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.compression_opts.window_bits: -14 2024-08-17T17:12:52.789 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.compression_opts.level: 32767 2024-08-17T17:12:52.789 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.compression_opts.strategy: 0 2024-08-17T17:12:52.789 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-08-17T17:12:52.789 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-08-17T17:12:52.789 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-08-17T17:12:52.789 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-08-17T17:12:52.789 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.compression_opts.enabled: false 2024-08-17T17:12:52.789 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-08-17T17:12:52.789 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-08-17T17:12:52.790 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-08-17T17:12:52.790 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-08-17T17:12:52.790 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.target_file_size_base: 67108864 2024-08-17T17:12:52.790 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.target_file_size_multiplier: 1 2024-08-17T17:12:52.790 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-08-17T17:12:52.790 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-08-17T17:12:52.790 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-08-17T17:12:52.790 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-08-17T17:12:52.790 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-08-17T17:12:52.790 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-08-17T17:12:52.790 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-08-17T17:12:52.790 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-08-17T17:12:52.790 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-08-17T17:12:52.790 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-08-17T17:12:52.791 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-08-17T17:12:52.791 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-08-17T17:12:52.791 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-08-17T17:12:52.791 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.arena_block_size: 1048576 2024-08-17T17:12:52.791 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-08-17T17:12:52.791 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-08-17T17:12:52.791 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.disable_auto_compactions: 0 2024-08-17T17:12:52.791 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-08-17T17:12:52.791 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-08-17T17:12:52.791 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-08-17T17:12:52.791 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-08-17T17:12:52.791 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-08-17T17:12:52.791 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-08-17T17:12:52.791 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-08-17T17:12:52.792 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-08-17T17:12:52.792 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-08-17T17:12:52.792 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-08-17T17:12:52.792 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-08-17T17:12:52.792 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.inplace_update_support: 0 2024-08-17T17:12:52.792 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.inplace_update_num_locks: 10000 2024-08-17T17:12:52.792 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-08-17T17:12:52.792 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-08-17T17:12:52.792 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.memtable_huge_page_size: 0 2024-08-17T17:12:52.792 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.bloom_locality: 0 2024-08-17T17:12:52.792 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.max_successive_merges: 0 2024-08-17T17:12:52.792 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.optimize_filters_for_hits: 0 2024-08-17T17:12:52.792 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.paranoid_file_checks: 0 2024-08-17T17:12:52.792 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.force_consistency_checks: 1 2024-08-17T17:12:52.793 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.report_bg_io_stats: 0 2024-08-17T17:12:52.793 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.ttl: 2592000 2024-08-17T17:12:52.793 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.periodic_compaction_seconds: 0 2024-08-17T17:12:52.793 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-08-17T17:12:52.793 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-08-17T17:12:52.793 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.enable_blob_files: false 2024-08-17T17:12:52.793 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.min_blob_size: 0 2024-08-17T17:12:52.793 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.blob_file_size: 268435456 2024-08-17T17:12:52.793 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.blob_compression_type: NoCompression 2024-08-17T17:12:52.793 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.enable_blob_garbage_collection: false 2024-08-17T17:12:52.793 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-08-17T17:12:52.793 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-08-17T17:12:52.793 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-08-17T17:12:52.793 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.blob_file_starting_level: 0 2024-08-17T17:12:52.794 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-08-17T17:12:52.794 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-08-17T17:12:52.794 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-08-17T17:12:52.794 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 12895, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-08-17T17:12:52.794 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-08-17T17:12:52.794 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 4c2cc502-67db-4cfb-b186-81841cab664b 2024-08-17T17:12:52.794 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723914772501334, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-08-17T17:12:52.794 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-08-17T17:12:52.794 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723914772510242, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 2346573, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12900, "largest_seqno": 13792, "table_properties": {"data_size": 2341780, "index_size": 2661, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1093, "raw_key_size": 10126, "raw_average_key_size": 24, "raw_value_size": 2332773, "raw_average_value_size": 5580, "num_data_blocks": 121, "num_entries": 418, "num_filter_entries": 418, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1723914772, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "4c2cc502-67db-4cfb-b186-81841cab664b", "db_session_id": "53FVU5RKM1QUWUJER6GN", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-08-17T17:12:52.794 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723914772510369, "job": 1, "event": "recovery_finished"} 2024-08-17T17:12:52.794 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-08-17T17:12:52.794 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-08-17T17:12:52.794 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-08-17T17:12:52.795 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55aa84a34e00 2024-08-17T17:12:52.795 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: DB pointer 0x55aa84b32000 2024-08-17T17:12:52.795 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-08-17T17:12:52.795 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-08-17T17:12:52.795 INFO:journalctl@ceph.mon.c.smithi003.stdout: ** DB Stats ** 2024-08-17T17:12:52.795 INFO:journalctl@ceph.mon.c.smithi003.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-17T17:12:52.795 INFO:journalctl@ceph.mon.c.smithi003.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-08-17T17:12:52.795 INFO:journalctl@ceph.mon.c.smithi003.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-17T17:12:52.795 INFO:journalctl@ceph.mon.c.smithi003.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-17T17:12:52.795 INFO:journalctl@ceph.mon.c.smithi003.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-08-17T17:12:52.795 INFO:journalctl@ceph.mon.c.smithi003.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-17T17:12:52.795 INFO:journalctl@ceph.mon.c.smithi003.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-17T17:12:52.795 INFO:journalctl@ceph.mon.c.smithi003.stdout: 2024-08-17T17:12:52.795 INFO:journalctl@ceph.mon.c.smithi003.stdout: ** Compaction Stats [default] ** 2024-08-17T17:12:52.796 INFO:journalctl@ceph.mon.c.smithi003.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-17T17:12:52.796 INFO:journalctl@ceph.mon.c.smithi003.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-08-17T17:12:52.796 INFO:journalctl@ceph.mon.c.smithi003.stdout: L0 1/0 2.24 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 629.5 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-08-17T17:12:52.796 INFO:journalctl@ceph.mon.c.smithi003.stdout: L6 1/0 7.35 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-08-17T17:12:52.796 INFO:journalctl@ceph.mon.c.smithi003.stdout: Sum 2/0 9.58 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 629.5 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-08-17T17:12:52.796 INFO:journalctl@ceph.mon.c.smithi003.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 629.5 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-08-17T17:12:52.796 INFO:journalctl@ceph.mon.c.smithi003.stdout: 2024-08-17T17:12:52.796 INFO:journalctl@ceph.mon.c.smithi003.stdout: ** Compaction Stats [default] ** 2024-08-17T17:12:52.796 INFO:journalctl@ceph.mon.c.smithi003.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-17T17:12:52.796 INFO:journalctl@ceph.mon.c.smithi003.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-08-17T17:12:52.796 INFO:journalctl@ceph.mon.c.smithi003.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 629.5 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-08-17T17:12:52.796 INFO:journalctl@ceph.mon.c.smithi003.stdout: 2024-08-17T17:12:52.796 INFO:journalctl@ceph.mon.c.smithi003.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-08-17T17:12:52.796 INFO:journalctl@ceph.mon.c.smithi003.stdout: 2024-08-17T17:12:52.797 INFO:journalctl@ceph.mon.c.smithi003.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-17T17:12:52.797 INFO:journalctl@ceph.mon.c.smithi003.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-08-17T17:12:52.797 INFO:journalctl@ceph.mon.c.smithi003.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-08-17T17:12:52.797 INFO:journalctl@ceph.mon.c.smithi003.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-08-17T17:12:52.797 INFO:journalctl@ceph.mon.c.smithi003.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-08-17T17:12:52.797 INFO:journalctl@ceph.mon.c.smithi003.stdout: AddFile(Keys): cumulative 0, interval 0 2024-08-17T17:12:52.797 INFO:journalctl@ceph.mon.c.smithi003.stdout: Cumulative compaction: 0.00 GB write, 190.77 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-17T17:12:52.797 INFO:journalctl@ceph.mon.c.smithi003.stdout: Interval compaction: 0.00 GB write, 190.77 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-17T17:12:52.797 INFO:journalctl@ceph.mon.c.smithi003.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-08-17T17:12:52.797 INFO:journalctl@ceph.mon.c.smithi003.stdout: Block cache BinnedLRUCache@0x55aa84a05610#2 capacity: 512.00 MB usage: 3.98 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-08-17T17:12:52.797 INFO:journalctl@ceph.mon.c.smithi003.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.16 KB,0.000220537%) IndexBlock(1,2.83 KB,0.000539422%) Misc(1,0.00 KB,0%) 2024-08-17T17:12:52.797 INFO:journalctl@ceph.mon.c.smithi003.stdout: 2024-08-17T17:12:52.797 INFO:journalctl@ceph.mon.c.smithi003.stdout: ** File Read Latency Histogram By Level [default] ** 2024-08-17T17:12:52.797 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: starting mon.c rank 2 at public addrs [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] at bind addrs [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:12:52.798 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: mon.c@-1(???) e3 preinit fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:12:52.798 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: mon.c@-1(???).mds e9 new map 2024-08-17T17:12:52.798 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: mon.c@-1(???).mds e9 print_map 2024-08-17T17:12:52.798 INFO:journalctl@ceph.mon.c.smithi003.stdout: e9 2024-08-17T17:12:52.798 INFO:journalctl@ceph.mon.c.smithi003.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-08-17T17:12:52.798 INFO:journalctl@ceph.mon.c.smithi003.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-08-17T17:12:52.798 INFO:journalctl@ceph.mon.c.smithi003.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-08-17T17:12:52.798 INFO:journalctl@ceph.mon.c.smithi003.stdout: legacy client fscid: 1 2024-08-17T17:12:52.798 INFO:journalctl@ceph.mon.c.smithi003.stdout: 2024-08-17T17:12:52.798 INFO:journalctl@ceph.mon.c.smithi003.stdout: Filesystem 'foo' (1) 2024-08-17T17:12:52.798 INFO:journalctl@ceph.mon.c.smithi003.stdout: fs_name foo 2024-08-17T17:12:52.798 INFO:journalctl@ceph.mon.c.smithi003.stdout: epoch 8 2024-08-17T17:12:52.798 INFO:journalctl@ceph.mon.c.smithi003.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-08-17T17:12:52.798 INFO:journalctl@ceph.mon.c.smithi003.stdout: created 2024-08-17T17:07:46.147672+0000 2024-08-17T17:12:52.799 INFO:journalctl@ceph.mon.c.smithi003.stdout: modified 2024-08-17T17:07:55.190877+0000 2024-08-17T17:12:52.799 INFO:journalctl@ceph.mon.c.smithi003.stdout: tableserver 0 2024-08-17T17:12:52.799 INFO:journalctl@ceph.mon.c.smithi003.stdout: root 0 2024-08-17T17:12:52.799 INFO:journalctl@ceph.mon.c.smithi003.stdout: session_timeout 60 2024-08-17T17:12:52.799 INFO:journalctl@ceph.mon.c.smithi003.stdout: session_autoclose 300 2024-08-17T17:12:52.799 INFO:journalctl@ceph.mon.c.smithi003.stdout: max_file_size 1099511627776 2024-08-17T17:12:52.799 INFO:journalctl@ceph.mon.c.smithi003.stdout: max_xattr_size 65536 2024-08-17T17:12:52.799 INFO:journalctl@ceph.mon.c.smithi003.stdout: required_client_features {} 2024-08-17T17:12:52.799 INFO:journalctl@ceph.mon.c.smithi003.stdout: last_failure 0 2024-08-17T17:12:52.799 INFO:journalctl@ceph.mon.c.smithi003.stdout: last_failure_osd_epoch 0 2024-08-17T17:12:52.799 INFO:journalctl@ceph.mon.c.smithi003.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-08-17T17:12:52.799 INFO:journalctl@ceph.mon.c.smithi003.stdout: max_mds 1 2024-08-17T17:12:52.799 INFO:journalctl@ceph.mon.c.smithi003.stdout: in 0 2024-08-17T17:12:52.799 INFO:journalctl@ceph.mon.c.smithi003.stdout: up {0=14664} 2024-08-17T17:12:52.799 INFO:journalctl@ceph.mon.c.smithi003.stdout: failed 2024-08-17T17:12:52.800 INFO:journalctl@ceph.mon.c.smithi003.stdout: damaged 2024-08-17T17:12:52.800 INFO:journalctl@ceph.mon.c.smithi003.stdout: stopped 2024-08-17T17:12:52.800 INFO:journalctl@ceph.mon.c.smithi003.stdout: data_pools [4] 2024-08-17T17:12:52.800 INFO:journalctl@ceph.mon.c.smithi003.stdout: metadata_pool 3 2024-08-17T17:12:52.800 INFO:journalctl@ceph.mon.c.smithi003.stdout: inline_data disabled 2024-08-17T17:12:52.800 INFO:journalctl@ceph.mon.c.smithi003.stdout: balancer 2024-08-17T17:12:52.800 INFO:journalctl@ceph.mon.c.smithi003.stdout: bal_rank_mask -1 2024-08-17T17:12:52.800 INFO:journalctl@ceph.mon.c.smithi003.stdout: standby_count_wanted 1 2024-08-17T17:12:52.800 INFO:journalctl@ceph.mon.c.smithi003.stdout: qdb_cluster leader: 0 members: 2024-08-17T17:12:52.800 INFO:journalctl@ceph.mon.c.smithi003.stdout: [mds.foo.smithi003.civgvf{0:14664} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.3:6834/2800254789,v1:172.21.15.3:6835/2800254789] compat {c=[1],r=[1],i=[7ff]}] 2024-08-17T17:12:52.800 INFO:journalctl@ceph.mon.c.smithi003.stdout: 2024-08-17T17:12:52.800 INFO:journalctl@ceph.mon.c.smithi003.stdout: 2024-08-17T17:12:52.800 INFO:journalctl@ceph.mon.c.smithi003.stdout: Standby daemons: 2024-08-17T17:12:52.800 INFO:journalctl@ceph.mon.c.smithi003.stdout: 2024-08-17T17:12:52.801 INFO:journalctl@ceph.mon.c.smithi003.stdout: [mds.foo.smithi053.ougbtc{-1:24511} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.53:6832/264432371,v1:172.21.15.53:6833/264432371] compat {c=[1],r=[1],i=[7ff]}] 2024-08-17T17:12:52.801 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: mon.c@-1(???).osd e90 crush map has features 3314933000854323200, adjusting msgr requires 2024-08-17T17:12:52.801 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: mon.c@-1(???).osd e90 crush map has features 432629239337189376, adjusting msgr requires 2024-08-17T17:12:52.801 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: mon.c@-1(???).osd e90 crush map has features 432629239337189376, adjusting msgr requires 2024-08-17T17:12:52.801 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: mon.c@-1(???).osd e90 crush map has features 432629239337189376, adjusting msgr requires 2024-08-17T17:12:52.801 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: mon.c@-1(???).paxosservice(auth 1..19) refresh upgraded, format 0 -> 3 2024-08-17T17:12:52.801 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 bash[103325]: 54a6cfe9eb9aa657457be3d687d1b78dcb57c3d174cda0c5dad6ac13004dccf2 2024-08-17T17:12:52.801 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 systemd[1]: Started Ceph mon.c for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:12:52.801 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-08-17T17:12:52.801 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:52 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3562970401' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:52.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:52 smithi053 ceph-mon[34699]: pgmap v9: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-08-17T17:12:52.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:52 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/3562970401' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-08-17T17:12:53.345 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 2 2024-08-17T17:12:53.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:53 smithi053 ceph-mon[34699]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:53 smithi053 ceph-mon[34699]: from='client.26408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:53 smithi053 ceph-mon[34699]: pgmap v10: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-08-17T17:12:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:53 smithi053 ceph-mon[34699]: mon.c calling monitor election 2024-08-17T17:12:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:53 smithi053 ceph-mon[34699]: mon.a calling monitor election 2024-08-17T17:12:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:53 smithi053 ceph-mon[34699]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-08-17T17:12:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:53 smithi053 ceph-mon[34699]: monmap epoch 3 2024-08-17T17:12:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:53 smithi053 ceph-mon[34699]: fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:12:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:53 smithi053 ceph-mon[34699]: last_changed 2024-08-17T17:02:32.825077+0000 2024-08-17T17:12:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:53 smithi053 ceph-mon[34699]: created 2024-08-17T17:00:53.391169+0000 2024-08-17T17:12:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:53 smithi053 ceph-mon[34699]: min_mon_release 17 (quincy) 2024-08-17T17:12:53.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:53 smithi053 ceph-mon[34699]: election_strategy: 1 2024-08-17T17:12:53.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:53 smithi053 ceph-mon[34699]: 0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2024-08-17T17:12:53.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:53 smithi053 ceph-mon[34699]: 1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-08-17T17:12:53.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:53 smithi053 ceph-mon[34699]: 2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2024-08-17T17:12:53.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:53 smithi053 ceph-mon[34699]: fsmap foo:1 {0=foo.smithi003.civgvf=up:active} 1 up:standby 2024-08-17T17:12:53.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:53 smithi053 ceph-mon[34699]: osdmap e90: 8 total, 8 up, 8 in 2024-08-17T17:12:53.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:53 smithi053 ceph-mon[34699]: mgrmap e38: y(active, since 15s), standbys: x 2024-08-17T17:12:53.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:53 smithi053 ceph-mon[34699]: overall HEALTH_OK 2024-08-17T17:12:53.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:53 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:53.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:53 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:53.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:53 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:12:53.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:53 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:12:53.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:53 smithi053 ceph-mon[34699]: from='client.? 172.21.15.3:0/3166720608' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-08-17T17:12:54.012 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[103344]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:54.012 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[103344]: from='client.26408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:54.012 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[103344]: pgmap v10: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-08-17T17:12:54.012 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[103344]: mon.c calling monitor election 2024-08-17T17:12:54.012 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[103344]: mon.a calling monitor election 2024-08-17T17:12:54.012 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[103344]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-08-17T17:12:54.012 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[103344]: monmap epoch 3 2024-08-17T17:12:54.012 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[103344]: fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:12:54.012 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[103344]: last_changed 2024-08-17T17:02:32.825077+0000 2024-08-17T17:12:54.012 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[103344]: created 2024-08-17T17:00:53.391169+0000 2024-08-17T17:12:54.012 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[103344]: min_mon_release 17 (quincy) 2024-08-17T17:12:54.013 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[103344]: election_strategy: 1 2024-08-17T17:12:54.013 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[103344]: 0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2024-08-17T17:12:54.013 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[103344]: 1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-08-17T17:12:54.013 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[103344]: 2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2024-08-17T17:12:54.013 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[103344]: fsmap foo:1 {0=foo.smithi003.civgvf=up:active} 1 up:standby 2024-08-17T17:12:54.013 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[103344]: osdmap e90: 8 total, 8 up, 8 in 2024-08-17T17:12:54.013 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[103344]: mgrmap e38: y(active, since 15s), standbys: x 2024-08-17T17:12:54.013 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[103344]: overall HEALTH_OK 2024-08-17T17:12:54.013 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:54.013 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:54.013 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:12:54.013 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:12:54.013 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3166720608' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-08-17T17:12:54.014 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[100615]: from='client.34224 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:54.014 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[100615]: from='client.26408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:54.014 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[100615]: pgmap v10: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-08-17T17:12:54.014 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[100615]: mon.c calling monitor election 2024-08-17T17:12:54.014 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[100615]: mon.a calling monitor election 2024-08-17T17:12:54.014 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[100615]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-08-17T17:12:54.014 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[100615]: monmap epoch 3 2024-08-17T17:12:54.014 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[100615]: fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:12:54.014 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[100615]: last_changed 2024-08-17T17:02:32.825077+0000 2024-08-17T17:12:54.014 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[100615]: created 2024-08-17T17:00:53.391169+0000 2024-08-17T17:12:54.014 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[100615]: min_mon_release 17 (quincy) 2024-08-17T17:12:54.015 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[100615]: election_strategy: 1 2024-08-17T17:12:54.015 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[100615]: 0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2024-08-17T17:12:54.015 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[100615]: 1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-08-17T17:12:54.015 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[100615]: 2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2024-08-17T17:12:54.015 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[100615]: fsmap foo:1 {0=foo.smithi003.civgvf=up:active} 1 up:standby 2024-08-17T17:12:54.015 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[100615]: osdmap e90: 8 total, 8 up, 8 in 2024-08-17T17:12:54.015 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[100615]: mgrmap e38: y(active, since 15s), standbys: x 2024-08-17T17:12:54.015 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[100615]: overall HEALTH_OK 2024-08-17T17:12:54.015 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:54.015 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:54.015 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:12:54.015 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:12:54.015 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:53 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3166720608' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-08-17T17:12:54.414 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'ceph orch ps' 2024-08-17T17:12:54.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:54.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:12:54.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:54 smithi003 ceph-mon[100615]: pgmap v11: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-08-17T17:12:54.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:54 smithi003 ceph-mon[103344]: pgmap v11: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-08-17T17:12:54.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:54 smithi053 ceph-mon[34699]: pgmap v11: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-08-17T17:12:55.326 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:12:55.718 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:12:55] "GET /metrics HTTP/1.1" 200 33762 "" "Prometheus/2.51.0" 2024-08-17T17:12:56.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:55.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:12:56.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:55.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:12:56.730 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:56 smithi003 ceph-mon[103344]: pgmap v12: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-08-17T17:12:56.730 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:56 smithi003 ceph-mon[100615]: pgmap v12: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-08-17T17:12:56.948 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:56 smithi053 ceph-mon[34699]: pgmap v12: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 5 op/s 2024-08-17T17:12:57.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:12:56 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:12:56.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi053\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:12:57.771 INFO:teuthology.orchestra.run.smithi003.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-08-17T17:12:57.771 INFO:teuthology.orchestra.run.smithi003.stdout:alertmanager.a smithi003 *:9093,9094 running (2m) 15s ago 6m 18.8M - 0.27.0 11f11916f8cd fb501c5aa873 2024-08-17T17:12:57.771 INFO:teuthology.orchestra.run.smithi003.stdout:grafana.a smithi053 *:3000 running (109s) 15s ago 6m 82.5M - 9.4.7 954c08fa6188 9b5ef97a5cd5 2024-08-17T17:12:57.771 INFO:teuthology.orchestra.run.smithi003.stdout:mds.foo.smithi003.civgvf smithi003 running (5m) 15s ago 5m 19.0M - 17.2.7-1404-gf2c39d4d f69471c56189 86a91b686f66 2024-08-17T17:12:57.771 INFO:teuthology.orchestra.run.smithi003.stdout:mds.foo.smithi053.ougbtc smithi053 running (5m) 15s ago 5m 16.0M - 17.2.7-1404-gf2c39d4d f69471c56189 41ad4cc0ce19 2024-08-17T17:12:57.771 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.x smithi053 *:8443,9283,8765 running (42s) 15s ago 10m 450M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 874be7d3a238 2024-08-17T17:12:57.771 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.y smithi003 *:8443,9283,8765 running (70s) 15s ago 11m 487M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb f491ea24df28 2024-08-17T17:12:57.772 INFO:teuthology.orchestra.run.smithi003.stdout:mon.a smithi003 running (28s) 15s ago 12m 30.8M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb c1cfbc2d47f8 2024-08-17T17:12:57.772 INFO:teuthology.orchestra.run.smithi003.stdout:mon.b smithi053 running (10m) 15s ago 10m 51.4M 2048M 17.2.7-1404-gf2c39d4d f69471c56189 471447c55e31 2024-08-17T17:12:57.772 INFO:teuthology.orchestra.run.smithi003.stdout:mon.c smithi003 starting - - - 2048M 2024-08-17T17:12:57.772 INFO:teuthology.orchestra.run.smithi003.stdout:node-exporter.a smithi003 *:9100 running (2m) 15s ago 5m 11.6M - 1.7.0 72c9c2088986 d19c9e3d51f8 2024-08-17T17:12:57.772 INFO:teuthology.orchestra.run.smithi003.stdout:node-exporter.b smithi053 *:9100 running (104s) 15s ago 5m 10.1M - 1.7.0 72c9c2088986 edfcc96e3983 2024-08-17T17:12:57.772 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0 smithi003 running (9m) 15s ago 9m 50.6M 2481M 17.2.7-1404-gf2c39d4d f69471c56189 dc8e5d51fe1c 2024-08-17T17:12:57.772 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1 smithi003 running (9m) 15s ago 9m 50.4M 2481M 17.2.7-1404-gf2c39d4d f69471c56189 7def3d9ed99f 2024-08-17T17:12:57.772 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2 smithi003 running (9m) 15s ago 9m 47.7M 2481M 17.2.7-1404-gf2c39d4d f69471c56189 3d65e6f7d660 2024-08-17T17:12:57.772 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3 smithi003 running (8m) 15s ago 8m 45.4M 2481M 17.2.7-1404-gf2c39d4d f69471c56189 61f40a5861f6 2024-08-17T17:12:57.772 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4 smithi053 running (8m) 15s ago 8m 47.4M 2481M 17.2.7-1404-gf2c39d4d f69471c56189 ced3b086d7f0 2024-08-17T17:12:57.772 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5 smithi053 running (7m) 15s ago 7m 47.0M 2481M 17.2.7-1404-gf2c39d4d f69471c56189 1c4ad574a193 2024-08-17T17:12:57.772 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6 smithi053 running (7m) 15s ago 7m 46.5M 2481M 17.2.7-1404-gf2c39d4d f69471c56189 5ed81ad77c62 2024-08-17T17:12:57.772 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7 smithi053 running (7m) 15s ago 7m 48.5M 2481M 17.2.7-1404-gf2c39d4d f69471c56189 f3e467b07793 2024-08-17T17:12:57.773 INFO:teuthology.orchestra.run.smithi003.stdout:prometheus.a smithi053 *:9095 running (47s) 15s ago 6m 43.3M - 2.51.0 1d3b7f56885b 41146206916a 2024-08-17T17:12:57.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:12:57 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:12:57.520+0000 7f288dc05640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-08-17T17:12:58.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:58 smithi003 ceph-mon[103344]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:58.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:58 smithi003 ceph-mon[103344]: from='client.34242 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:58.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:58 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:58.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:12:58 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:58.855 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:58 smithi003 ceph-mon[100615]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:58.855 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:58 smithi003 ceph-mon[100615]: from='client.34242 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:58.855 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:58 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:58.855 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:12:58 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:58.880 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'ceph orch upgrade pause' 2024-08-17T17:12:58.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:58 smithi053 ceph-mon[34699]: pgmap v13: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:12:58.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:58 smithi053 ceph-mon[34699]: from='client.34242 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:12:58.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:58 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:58.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:12:58 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:12:59.158 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:13:00.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:13:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:13:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:13:00.642 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:00 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:13:00.642 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:00 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:13:00.642 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:00 smithi003 ceph-mon[103344]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:00.642 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:00 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:13:00.642 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:00 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:13:00.643 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:00 smithi003 ceph-mon[100615]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:00.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:00 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:13:00.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:00 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:13:00.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:00 smithi053 ceph-mon[34699]: pgmap v14: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:01.049 INFO:teuthology.orchestra.run.smithi003.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:13:02.090 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:02 smithi003 ceph-mon[103344]: from='client.44108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:13:02.091 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:02 smithi003 ceph-mon[103344]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:13:02.091 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:02 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:13:02.091 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:02 smithi003 ceph-mon[100615]: from='client.44108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:13:02.091 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:02 smithi003 ceph-mon[100615]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:13:02.091 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:02 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:13:02.121 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'sleep 60' 2024-08-17T17:13:02.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:02 smithi053 ceph-mon[34699]: from='client.44108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:13:02.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:02 smithi053 ceph-mon[34699]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:13:02.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:02 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:13:03.109 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:13:03.234 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:03 smithi003 ceph-mon[100615]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:03.234 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:03 smithi003 ceph-mon[103344]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:03.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:03 smithi053 ceph-mon[34699]: pgmap v15: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:04.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:13:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:13:04.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:13:05.297 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:05 smithi003 ceph-mon[100615]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:05.297 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:05 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:13:05.297 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:05 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:13:05.297 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:05 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:13:05.297 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:05 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:13:05.298 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:05 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:13:05.298 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:05 smithi003 ceph-mon[103344]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:05.298 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:05 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:13:05.298 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:05 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:13:05.298 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:05 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:13:05.298 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:05 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:13:05.298 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:05 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:13:05.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:05 smithi053 ceph-mon[34699]: pgmap v16: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:05.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:05 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:13:05.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:05 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:13:05.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:05 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:13:05.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:05 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:13:05.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:05 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:13:06.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:13:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:13:05] "GET /metrics HTTP/1.1" 200 33762 "" "Prometheus/2.51.0" 2024-08-17T17:13:06.060 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:13:05 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:13:05.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:13:06.060 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:13:05 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:13:05.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:13:06.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:06 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:13:06.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:06 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:13:06.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:06 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:13:06.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:06 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:13:06.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:06 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:13:06.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:06 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:13:06.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:06 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:13:06.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:06 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:13:06.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:06 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:13:06.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:06 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:13:06.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:06 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:13:06.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:06 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:13:07.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:13:06 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:13:06.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi053\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:13:07.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:07 smithi053 ceph-mon[34699]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:07.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:07 smithi003 ceph-mon[103344]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:07.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:07 smithi003 ceph-mon[100615]: pgmap v17: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:08.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:08 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:13:08.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:13:08.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:13:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:09 smithi053 ceph-mon[34699]: pgmap v18: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:09.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:09 smithi003 ceph-mon[100615]: pgmap v18: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:09 smithi003 ceph-mon[103344]: pgmap v18: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:10.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:13:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:13:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:13:11.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:11 smithi053 ceph-mon[34699]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:11.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:11 smithi003 ceph-mon[100615]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:11.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:11 smithi003 ceph-mon[103344]: pgmap v19: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:13.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:13 smithi053 ceph-mon[34699]: pgmap v20: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:13.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:13 smithi003 ceph-mon[100615]: pgmap v20: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:13.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:13 smithi003 ceph-mon[103344]: pgmap v20: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:14.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:13:14 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:13:14.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:13:15.342 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:15 smithi003 ceph-mon[100615]: pgmap v21: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:15.343 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:15 smithi003 ceph-mon[103344]: pgmap v21: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:15.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:15 smithi053 ceph-mon[34699]: pgmap v21: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:16.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:13:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:13:15] "GET /metrics HTTP/1.1" 200 33827 "" "Prometheus/2.51.0" 2024-08-17T17:13:16.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:13:15 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:13:15.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:13:16.217 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:13:15 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:13:15.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:13:17.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:17 smithi053 ceph-mon[34699]: pgmap v22: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:17.217 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:13:16 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:13:16.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi053\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:13:17.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:17 smithi003 ceph-mon[100615]: pgmap v22: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:17.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:17 smithi003 ceph-mon[103344]: pgmap v22: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:19.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:19 smithi053 ceph-mon[34699]: pgmap v23: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:19.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:19 smithi003 ceph-mon[100615]: pgmap v23: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:19.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:19 smithi003 ceph-mon[103344]: pgmap v23: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:20.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:13:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:13:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:13:21.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:21 smithi053 ceph-mon[34699]: pgmap v24: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:21.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:21 smithi003 ceph-mon[100615]: pgmap v24: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:21.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:21 smithi003 ceph-mon[103344]: pgmap v24: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:23.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:23 smithi053 ceph-mon[34699]: pgmap v25: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:23.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:23 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:13:23.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:23 smithi003 ceph-mon[103344]: pgmap v25: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:23.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:13:23.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:23 smithi003 ceph-mon[100615]: pgmap v25: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:23.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:13:24.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:13:24 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:13:24.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:13:25.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:25 smithi053 ceph-mon[34699]: pgmap v26: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:25.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:25 smithi003 ceph-mon[103344]: pgmap v26: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:25.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:25 smithi003 ceph-mon[100615]: pgmap v26: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:26.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:13:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:13:25] "GET /metrics HTTP/1.1" 200 33826 "" "Prometheus/2.51.0" 2024-08-17T17:13:26.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:13:25 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:13:25.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:13:26.217 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:13:25 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:13:25.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:13:27.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:27 smithi053 ceph-mon[34699]: pgmap v27: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:27.217 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:13:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:13:26.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi053\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:13:27.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:27 smithi003 ceph-mon[103344]: pgmap v27: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:27.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:27 smithi003 ceph-mon[100615]: pgmap v27: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:29.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:29 smithi053 ceph-mon[34699]: pgmap v28: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:29.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:29 smithi003 ceph-mon[103344]: pgmap v28: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:29.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:29 smithi003 ceph-mon[100615]: pgmap v28: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:30.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:13:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:13:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:13:31.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:31 smithi053 ceph-mon[34699]: pgmap v29: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:31.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:31 smithi003 ceph-mon[100615]: pgmap v29: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:31.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:31 smithi003 ceph-mon[103344]: pgmap v29: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:33.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:33 smithi053 ceph-mon[34699]: pgmap v30: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:33.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:33 smithi003 ceph-mon[103344]: pgmap v30: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:33.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:33 smithi003 ceph-mon[100615]: pgmap v30: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:34.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:13:34 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:13:34.149Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:13:35.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:35 smithi053 ceph-mon[34699]: pgmap v31: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:35.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:35 smithi003 ceph-mon[103344]: pgmap v31: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:35.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:35 smithi003 ceph-mon[100615]: pgmap v31: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:36.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:13:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:13:35] "GET /metrics HTTP/1.1" 200 33826 "" "Prometheus/2.51.0" 2024-08-17T17:13:36.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:13:35 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:13:35.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:13:36.217 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:13:35 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:13:35.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:13:37.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:37 smithi053 ceph-mon[34699]: pgmap v32: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:37.217 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:13:36 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:13:36.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi053\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:13:37.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:37 smithi003 ceph-mon[103344]: pgmap v32: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:37.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:37 smithi003 ceph-mon[100615]: pgmap v32: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:38.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:38 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:13:38.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:13:38.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:13:39.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:39 smithi053 ceph-mon[34699]: pgmap v33: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:39.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:39 smithi003 ceph-mon[103344]: pgmap v33: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:39.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:39 smithi003 ceph-mon[100615]: pgmap v33: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:40.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:13:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:13:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:13:41.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:41 smithi053 ceph-mon[34699]: pgmap v34: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:41.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:41 smithi003 ceph-mon[103344]: pgmap v34: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:41 smithi003 ceph-mon[100615]: pgmap v34: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:43.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:43 smithi053 ceph-mon[34699]: pgmap v35: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:43.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:43 smithi003 ceph-mon[103344]: pgmap v35: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:43.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:43 smithi003 ceph-mon[100615]: pgmap v35: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:44.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:13:44 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:13:44.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:13:45.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:45 smithi053 ceph-mon[34699]: pgmap v36: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:45.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:45 smithi003 ceph-mon[103344]: pgmap v36: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:45.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:45 smithi003 ceph-mon[100615]: pgmap v36: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:46.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:13:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:13:45] "GET /metrics HTTP/1.1" 200 33826 "" "Prometheus/2.51.0" 2024-08-17T17:13:46.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:13:45 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:13:45.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:13:46.217 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:13:45 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:13:45.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:13:47.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:13:46 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:13:46.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi053\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:13:47.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:47 smithi053 ceph-mon[34699]: pgmap v37: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:47.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:47 smithi003 ceph-mon[103344]: pgmap v37: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:47.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:47 smithi003 ceph-mon[100615]: pgmap v37: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:49.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:49 smithi053 ceph-mon[34699]: pgmap v38: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:49.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:49 smithi003 ceph-mon[103344]: pgmap v38: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:49.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:49 smithi003 ceph-mon[100615]: pgmap v38: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:50.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:13:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:13:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:13:51.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:51 smithi053 ceph-mon[34699]: pgmap v39: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:51.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:51 smithi003 ceph-mon[100615]: pgmap v39: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:51.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:51 smithi003 ceph-mon[103344]: pgmap v39: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:53.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:53 smithi053 ceph-mon[34699]: pgmap v40: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:53.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:53 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:13:53.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:53 smithi003 ceph-mon[103344]: pgmap v40: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:53.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:13:53.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:53 smithi003 ceph-mon[100615]: pgmap v40: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:53.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:13:54.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:13:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:13:54.146Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:13:55.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:55 smithi053 ceph-mon[34699]: pgmap v41: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:55.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:55 smithi003 ceph-mon[103344]: pgmap v41: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:55.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:55 smithi003 ceph-mon[100615]: pgmap v41: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:56.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:13:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:13:55] "GET /metrics HTTP/1.1" 200 33825 "" "Prometheus/2.51.0" 2024-08-17T17:13:56.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:13:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:13:55.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:13:56.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:13:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:13:55.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:13:57.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:57 smithi053 ceph-mon[34699]: pgmap v42: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:57.217 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:13:56 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:13:56.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi053\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:13:57.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:57 smithi003 ceph-mon[103344]: pgmap v42: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:57.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:57 smithi003 ceph-mon[100615]: pgmap v42: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:59.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:13:59 smithi053 ceph-mon[34699]: pgmap v43: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:59.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:13:59 smithi003 ceph-mon[103344]: pgmap v43: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:13:59.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:13:59 smithi003 ceph-mon[100615]: pgmap v43: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:00.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:14:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:14:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:14:01.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:01 smithi053 ceph-mon[34699]: pgmap v44: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:01.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:01 smithi003 ceph-mon[100615]: pgmap v44: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:01.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:01 smithi003 ceph-mon[103344]: pgmap v44: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:03.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:03 smithi053 ceph-mon[34699]: pgmap v45: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:03.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:03 smithi003 ceph-mon[100615]: pgmap v45: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:03.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:03 smithi003 ceph-mon[103344]: pgmap v45: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:04.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:14:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:14:04.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:14:05.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:05 smithi053 ceph-mon[34699]: pgmap v46: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:05.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:05 smithi003 ceph-mon[100615]: pgmap v46: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:05.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:05 smithi003 ceph-mon[103344]: pgmap v46: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:06.014 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:14:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:14:05] "GET /metrics HTTP/1.1" 200 33825 "" "Prometheus/2.51.0" 2024-08-17T17:14:06.018 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:14:05 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:14:05.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:14:06.018 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:14:05 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:14:05.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:14:06.043 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'ceph orch upgrade resume' 2024-08-17T17:14:06.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:06 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:14:06.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:06 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:14:06.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:06 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:14:06.974 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:14:07.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:07 smithi053 ceph-mon[34699]: pgmap v47: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:07.217 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:14:06 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:14:06.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi053\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:14:07.413 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:07 smithi003 ceph-mon[103344]: pgmap v47: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:07.413 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:07 smithi003 ceph-mon[100615]: pgmap v47: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:08.363 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:14:08.363 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:14:08.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:08 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:14:09.300 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:09 smithi003 ceph-mon[100615]: pgmap v48: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:09.300 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:09 smithi003 ceph-mon[103344]: pgmap v48: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:09.382 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:09 smithi053 ceph-mon[34699]: pgmap v48: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:09.495 INFO:teuthology.orchestra.run.smithi003.stdout:Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:14:10.274 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'echo wait for all mons to upgrade' 2024-08-17T17:14:10.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:14:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:14:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:14:10.498 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:10 smithi003 ceph-mon[100615]: from='client.44114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:10.498 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:10 smithi003 ceph-mon[100615]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:14:10.498 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:10 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:10.499 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:10 smithi003 ceph-mon[100615]: pgmap v49: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:14:10.499 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:10 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:10.499 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:10 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:10.663 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:14:10.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:10 smithi003 ceph-mon[103344]: from='client.44114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:10.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:10 smithi003 ceph-mon[103344]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:14:10.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:10 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:10.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:10 smithi003 ceph-mon[103344]: pgmap v49: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:14:10.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:10 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:10.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:10 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:10.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:10 smithi053 ceph-mon[34699]: from='client.44114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:10.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:10 smithi053 ceph-mon[34699]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:14:10.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:10 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:10.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:10 smithi053 ceph-mon[34699]: pgmap v49: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:14:10.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:10 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:10.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:10 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:11.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:11 smithi003 ceph-mon[103344]: mgrmap e39: y(active, since 92s), standbys: x 2024-08-17T17:14:11.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:11 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:11.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:11 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:11.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:11 smithi003 ceph-mon[100615]: mgrmap e39: y(active, since 92s), standbys: x 2024-08-17T17:14:11.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:11 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:11.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:11 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:11.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:11 smithi053 ceph-mon[34699]: mgrmap e39: y(active, since 92s), standbys: x 2024-08-17T17:14:11.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:11 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:11.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:11 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:12.593 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:12 smithi053 ceph-mon[34699]: pgmap v50: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:14:12.593 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:12 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:12.593 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:12 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:14:12.593 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:12 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:12.593 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:12 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:14:12.593 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:12 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:14:12.593 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:12 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-08-17T17:14:12.594 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:12 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["b"]} : dispatch 2024-08-17T17:14:12.594 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:12 smithi053 ceph-mon[34699]: Upgrade: It appears safe to stop mon.b 2024-08-17T17:14:12.750 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:12 smithi003 ceph-mon[103344]: pgmap v50: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:14:12.750 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:12 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:12.750 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:12 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:14:12.750 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:12 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:12.750 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:12 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:14:12.751 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:12 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:14:12.751 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:12 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-08-17T17:14:12.751 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:12 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["b"]} : dispatch 2024-08-17T17:14:12.751 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:12 smithi003 ceph-mon[103344]: Upgrade: It appears safe to stop mon.b 2024-08-17T17:14:12.751 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:12 smithi003 ceph-mon[100615]: pgmap v50: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:14:12.751 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:12 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:12.751 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:12 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:14:12.751 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:12 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:12.751 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:12 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:14:12.751 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:12 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:14:12.752 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:12 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-08-17T17:14:12.752 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:12 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["b"]} : dispatch 2024-08-17T17:14:12.752 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:12 smithi003 ceph-mon[100615]: Upgrade: It appears safe to stop mon.b 2024-08-17T17:14:12.956 INFO:teuthology.orchestra.run.smithi003.stdout:wait for all mons to upgrade 2024-08-17T17:14:13.546 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'while ! ceph mon versions | grep $sha1 | grep '"'"': 3'"'"' && ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error; do sleep 2 ; done' 2024-08-17T17:14:13.814 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:14:14.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:14:14 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:14:14.146Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:14:14.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:14 smithi003 ceph-mon[103344]: Upgrade: Updating mon.b 2024-08-17T17:14:14.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:14 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:14.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:14 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-17T17:14:14.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:14 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-17T17:14:14.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:14 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:14.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:14 smithi003 ceph-mon[103344]: Deploying daemon mon.b on smithi053 2024-08-17T17:14:14.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:14 smithi003 ceph-mon[100615]: Upgrade: Updating mon.b 2024-08-17T17:14:14.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:14 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:14.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:14 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-17T17:14:14.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:14 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-17T17:14:14.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:14 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:14.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:14 smithi003 ceph-mon[100615]: Deploying daemon mon.b on smithi053 2024-08-17T17:14:14.572 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:14 smithi053 ceph-mon[34699]: Upgrade: Updating mon.b 2024-08-17T17:14:14.572 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:14 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:14.572 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:14 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-17T17:14:14.572 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:14 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-17T17:14:14.572 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:14 smithi053 ceph-mon[34699]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:14.572 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:14 smithi053 ceph-mon[34699]: Deploying daemon mon.b on smithi053 2024-08-17T17:14:15.152 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:14 smithi053 systemd[1]: Stopping Ceph mon.b for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:14:15.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:15 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-b[34682]: 2024-08-17T17:14:15.150+0000 7f1b3bba1640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T17:14:15.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:15 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-b[34682]: 2024-08-17T17:14:15.150+0000 7f1b3bba1640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-08-17T17:14:15.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:15 smithi053 podman[89803]: 2024-08-17 17:14:15.325230886 +0000 UTC m=+0.280336963 container died 471447c55e315b6fe8a114c5903a18b0ac37a7f6222b05c5c85d77981c6c8b54 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=quincy-f2c39d4, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.license=GPLv2) 2024-08-17T17:14:15.852 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:15 smithi053 podman[89803]: 2024-08-17 17:14:15.650522118 +0000 UTC m=+0.605628182 container cleanup 471447c55e315b6fe8a114c5903a18b0ac37a7f6222b05c5c85d77981c6c8b54 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-b, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-f2c39d4, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-08-17T17:14:15.852 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:15 smithi053 bash[89803]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-b 2024-08-17T17:14:15.853 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:14:15 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:14:15.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:14:15.853 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:14:15 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:14:15.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:14:15.958 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:14:15.958 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:14:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:14:15] "GET /metrics HTTP/1.1" 200 33823 "" "Prometheus/2.51.0" 2024-08-17T17:14:16.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:15 smithi053 podman[89815]: 2024-08-17 17:14:15.851999323 +0000 UTC m=+0.521904868 container remove 471447c55e315b6fe8a114c5903a18b0ac37a7f6222b05c5c85d77981c6c8b54 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-b, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, RELEASE=quincy-f2c39d4, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.36.0) 2024-08-17T17:14:16.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mon.b.service: Deactivated successfully. 2024-08-17T17:14:16.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 systemd[1]: Stopped Ceph mon.b for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:14:16.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mon.b.service: Consumed 9.343s CPU time. 2024-08-17T17:14:16.617 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 systemd[1]: Starting Ceph mon.b for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:14:16.617 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 podman[89945]: 2024-08-17 17:14:16.420507238 +0000 UTC m=+0.018215692 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:14:16.617 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 podman[89945]: 2024-08-17 17:14:16.519587406 +0000 UTC m=+0.117295845 container create 9dc96112f3afceec9ace658d33f047e7cde585c3c68bf30a416d01767198c245 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, RELEASE=main-cd8bdbe, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:14:16.949 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 podman[89945]: 2024-08-17 17:14:16.738640465 +0000 UTC m=+0.336348907 container init 9dc96112f3afceec9ace658d33f047e7cde585c3c68bf30a416d01767198c245 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-b, RELEASE=main-cd8bdbe, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-08-17T17:14:16.949 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 podman[89945]: 2024-08-17 17:14:16.742190062 +0000 UTC m=+0.339898512 container start 9dc96112f3afceec9ace658d33f047e7cde585c3c68bf30a416d01767198c245 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-b, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, ceph=True, RELEASE=main-cd8bdbe, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-17T17:14:16.949 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: set uid:gid to 167:167 (ceph:ceph) 2024-08-17T17:14:16.949 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev), process ceph-mon, pid 2 2024-08-17T17:14:16.949 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: pidfile_write: ignore empty --pid-file 2024-08-17T17:14:16.949 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: load: jerasure load: lrc 2024-08-17T17:14:16.950 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: RocksDB version: 7.9.2 2024-08-17T17:14:16.950 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Git sha 0 2024-08-17T17:14:16.950 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Compile date 2024-08-15 20:43:31 2024-08-17T17:14:16.950 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: DB SUMMARY 2024-08-17T17:14:16.950 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: DB Session ID: 5VW6IGC2I334UYL786SO 2024-08-17T17:14:16.950 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: CURRENT file: CURRENT 2024-08-17T17:14:16.950 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: IDENTITY file: IDENTITY 2024-08-17T17:14:16.950 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1097 Bytes 2024-08-17T17:14:16.950 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000027.sst 2024-08-17T17:14:16.950 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000025.log size: 5411236 ; 2024-08-17T17:14:16.950 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.error_if_exists: 0 2024-08-17T17:14:16.951 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.create_if_missing: 0 2024-08-17T17:14:16.951 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.paranoid_checks: 1 2024-08-17T17:14:16.951 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.flush_verify_memtable_count: 1 2024-08-17T17:14:16.951 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-08-17T17:14:16.951 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-08-17T17:14:16.951 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.env: 0x564ad331bbe0 2024-08-17T17:14:16.951 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.fs: PosixFileSystem 2024-08-17T17:14:16.951 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.info_log: 0x564ad44ab700 2024-08-17T17:14:16.951 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.max_file_opening_threads: 16 2024-08-17T17:14:16.951 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.statistics: (nil) 2024-08-17T17:14:16.951 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.use_fsync: 0 2024-08-17T17:14:16.951 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.max_log_file_size: 0 2024-08-17T17:14:16.952 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-08-17T17:14:16.952 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.log_file_time_to_roll: 0 2024-08-17T17:14:16.952 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.keep_log_file_num: 1000 2024-08-17T17:14:16.952 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.recycle_log_file_num: 0 2024-08-17T17:14:16.952 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.allow_fallocate: 1 2024-08-17T17:14:16.952 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.allow_mmap_reads: 0 2024-08-17T17:14:16.952 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.allow_mmap_writes: 0 2024-08-17T17:14:16.952 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.use_direct_reads: 0 2024-08-17T17:14:16.952 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-08-17T17:14:16.952 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.create_missing_column_families: 0 2024-08-17T17:14:16.952 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.db_log_dir: 2024-08-17T17:14:16.952 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.wal_dir: 2024-08-17T17:14:16.952 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.table_cache_numshardbits: 6 2024-08-17T17:14:16.952 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.WAL_ttl_seconds: 0 2024-08-17T17:14:16.953 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.WAL_size_limit_MB: 0 2024-08-17T17:14:16.953 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-08-17T17:14:16.953 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-08-17T17:14:16.953 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.is_fd_close_on_exec: 1 2024-08-17T17:14:16.953 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.advise_random_on_open: 1 2024-08-17T17:14:16.953 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.db_write_buffer_size: 0 2024-08-17T17:14:16.953 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.write_buffer_manager: 0x564ad44aee60 2024-08-17T17:14:16.953 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-08-17T17:14:16.953 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-08-17T17:14:16.953 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.use_adaptive_mutex: 0 2024-08-17T17:14:16.953 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.rate_limiter: (nil) 2024-08-17T17:14:16.953 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-08-17T17:14:16.953 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.wal_recovery_mode: 2 2024-08-17T17:14:16.953 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.enable_thread_tracking: 0 2024-08-17T17:14:16.953 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.enable_pipelined_write: 0 2024-08-17T17:14:16.954 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.unordered_write: 0 2024-08-17T17:14:16.954 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-08-17T17:14:16.954 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-08-17T17:14:16.954 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-08-17T17:14:16.954 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-08-17T17:14:16.954 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.row_cache: None 2024-08-17T17:14:16.954 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.wal_filter: None 2024-08-17T17:14:16.954 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-08-17T17:14:16.955 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.allow_ingest_behind: 0 2024-08-17T17:14:16.956 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.two_write_queues: 0 2024-08-17T17:14:16.956 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.manual_wal_flush: 0 2024-08-17T17:14:16.956 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.wal_compression: 0 2024-08-17T17:14:16.956 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.atomic_flush: 0 2024-08-17T17:14:16.956 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-08-17T17:14:16.956 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.persist_stats_to_disk: 0 2024-08-17T17:14:16.956 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.write_dbid_to_manifest: 0 2024-08-17T17:14:16.956 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.log_readahead_size: 0 2024-08-17T17:14:16.956 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-08-17T17:14:16.956 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.best_efforts_recovery: 0 2024-08-17T17:14:16.956 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-08-17T17:14:16.956 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-08-17T17:14:16.956 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.allow_data_in_errors: 0 2024-08-17T17:14:16.957 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.db_host_id: __hostname__ 2024-08-17T17:14:16.957 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.enforce_single_del_contracts: true 2024-08-17T17:14:16.957 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.max_background_jobs: 2 2024-08-17T17:14:16.957 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.max_background_compactions: -1 2024-08-17T17:14:16.957 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.max_subcompactions: 1 2024-08-17T17:14:16.957 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-08-17T17:14:16.957 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-08-17T17:14:16.957 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.delayed_write_rate : 16777216 2024-08-17T17:14:16.957 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.max_total_wal_size: 0 2024-08-17T17:14:16.957 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-08-17T17:14:16.957 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.stats_dump_period_sec: 600 2024-08-17T17:14:16.957 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.stats_persist_period_sec: 600 2024-08-17T17:14:16.958 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-08-17T17:14:16.958 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.max_open_files: -1 2024-08-17T17:14:16.958 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.bytes_per_sync: 0 2024-08-17T17:14:16.958 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.wal_bytes_per_sync: 0 2024-08-17T17:14:16.958 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.strict_bytes_per_sync: 0 2024-08-17T17:14:16.958 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.compaction_readahead_size: 0 2024-08-17T17:14:16.958 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.max_background_flushes: -1 2024-08-17T17:14:16.958 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Compression algorithms supported: 2024-08-17T17:14:16.958 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: kZSTD supported: 0 2024-08-17T17:14:16.958 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: kXpressCompression supported: 0 2024-08-17T17:14:16.958 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: kBZip2Compression supported: 0 2024-08-17T17:14:16.958 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-08-17T17:14:16.958 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: kLZ4Compression supported: 1 2024-08-17T17:14:16.959 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: kZlibCompression supported: 1 2024-08-17T17:14:16.959 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: kLZ4HCCompression supported: 1 2024-08-17T17:14:16.959 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: kSnappyCompression supported: 1 2024-08-17T17:14:16.959 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Fast CRC32 supported: Supported on x86 2024-08-17T17:14:16.959 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: DMutex implementation: pthread_mutex_t 2024-08-17T17:14:16.959 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-08-17T17:14:16.959 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-08-17T17:14:16.959 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-08-17T17:14:16.959 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.merge_operator: 2024-08-17T17:14:16.959 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.compaction_filter: None 2024-08-17T17:14:16.959 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.compaction_filter_factory: None 2024-08-17T17:14:16.959 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.sst_partitioner_factory: None 2024-08-17T17:14:16.959 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.memtable_factory: SkipListFactory 2024-08-17T17:14:16.959 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.table_factory: BlockBasedTable 2024-08-17T17:14:16.960 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564ad44ab380) 2024-08-17T17:14:16.960 INFO:journalctl@ceph.mon.b.smithi053.stdout: cache_index_and_filter_blocks: 1 2024-08-17T17:14:16.960 INFO:journalctl@ceph.mon.b.smithi053.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-08-17T17:14:16.960 INFO:journalctl@ceph.mon.b.smithi053.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-08-17T17:14:16.960 INFO:journalctl@ceph.mon.b.smithi053.stdout: pin_top_level_index_and_filter: 1 2024-08-17T17:14:16.960 INFO:journalctl@ceph.mon.b.smithi053.stdout: index_type: 0 2024-08-17T17:14:16.960 INFO:journalctl@ceph.mon.b.smithi053.stdout: data_block_index_type: 0 2024-08-17T17:14:16.960 INFO:journalctl@ceph.mon.b.smithi053.stdout: index_shortening: 1 2024-08-17T17:14:16.960 INFO:journalctl@ceph.mon.b.smithi053.stdout: data_block_hash_table_util_ratio: 0.750000 2024-08-17T17:14:16.960 INFO:journalctl@ceph.mon.b.smithi053.stdout: checksum: 4 2024-08-17T17:14:16.960 INFO:journalctl@ceph.mon.b.smithi053.stdout: no_block_cache: 0 2024-08-17T17:14:16.960 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_cache: 0x564ad449f610 2024-08-17T17:14:16.960 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_cache_name: BinnedLRUCache 2024-08-17T17:14:16.960 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_cache_options: 2024-08-17T17:14:16.960 INFO:journalctl@ceph.mon.b.smithi053.stdout: capacity : 536870912 2024-08-17T17:14:16.961 INFO:journalctl@ceph.mon.b.smithi053.stdout: num_shard_bits : 4 2024-08-17T17:14:16.961 INFO:journalctl@ceph.mon.b.smithi053.stdout: strict_capacity_limit : 0 2024-08-17T17:14:16.961 INFO:journalctl@ceph.mon.b.smithi053.stdout: high_pri_pool_ratio: 0.000 2024-08-17T17:14:16.961 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_cache_compressed: (nil) 2024-08-17T17:14:16.961 INFO:journalctl@ceph.mon.b.smithi053.stdout: persistent_cache: (nil) 2024-08-17T17:14:16.961 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_size: 4096 2024-08-17T17:14:16.961 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_size_deviation: 10 2024-08-17T17:14:16.961 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_restart_interval: 16 2024-08-17T17:14:16.961 INFO:journalctl@ceph.mon.b.smithi053.stdout: index_block_restart_interval: 1 2024-08-17T17:14:16.961 INFO:journalctl@ceph.mon.b.smithi053.stdout: metadata_block_size: 4096 2024-08-17T17:14:16.961 INFO:journalctl@ceph.mon.b.smithi053.stdout: partition_filters: 0 2024-08-17T17:14:16.961 INFO:journalctl@ceph.mon.b.smithi053.stdout: use_delta_encoding: 1 2024-08-17T17:14:16.961 INFO:journalctl@ceph.mon.b.smithi053.stdout: filter_policy: bloomfilter 2024-08-17T17:14:16.962 INFO:journalctl@ceph.mon.b.smithi053.stdout: whole_key_filtering: 1 2024-08-17T17:14:16.962 INFO:journalctl@ceph.mon.b.smithi053.stdout: verify_compression: 0 2024-08-17T17:14:16.962 INFO:journalctl@ceph.mon.b.smithi053.stdout: read_amp_bytes_per_bit: 0 2024-08-17T17:14:16.962 INFO:journalctl@ceph.mon.b.smithi053.stdout: format_version: 5 2024-08-17T17:14:16.962 INFO:journalctl@ceph.mon.b.smithi053.stdout: enable_index_compression: 1 2024-08-17T17:14:16.962 INFO:journalctl@ceph.mon.b.smithi053.stdout: block_align: 0 2024-08-17T17:14:16.962 INFO:journalctl@ceph.mon.b.smithi053.stdout: max_auto_readahead_size: 262144 2024-08-17T17:14:16.962 INFO:journalctl@ceph.mon.b.smithi053.stdout: prepopulate_block_cache: 0 2024-08-17T17:14:16.962 INFO:journalctl@ceph.mon.b.smithi053.stdout: initial_auto_readahead_size: 8192 2024-08-17T17:14:16.962 INFO:journalctl@ceph.mon.b.smithi053.stdout: num_file_reads_for_auto_readahead: 2 2024-08-17T17:14:16.962 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.write_buffer_size: 33554432 2024-08-17T17:14:16.962 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.max_write_buffer_number: 2 2024-08-17T17:14:16.963 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.compression: NoCompression 2024-08-17T17:14:16.963 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.bottommost_compression: Disabled 2024-08-17T17:14:16.963 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.prefix_extractor: nullptr 2024-08-17T17:14:16.963 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-08-17T17:14:16.963 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.num_levels: 7 2024-08-17T17:14:16.963 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-08-17T17:14:16.963 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-08-17T17:14:16.963 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-08-17T17:14:16.963 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-08-17T17:14:16.963 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-08-17T17:14:16.963 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-08-17T17:14:16.963 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-08-17T17:14:16.963 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-08-17T17:14:16.964 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-08-17T17:14:16.964 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-08-17T17:14:16.964 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-08-17T17:14:16.964 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-08-17T17:14:16.964 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.compression_opts.window_bits: -14 2024-08-17T17:14:16.964 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.compression_opts.level: 32767 2024-08-17T17:14:16.964 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.compression_opts.strategy: 0 2024-08-17T17:14:16.964 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-08-17T17:14:16.964 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-08-17T17:14:16.964 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-08-17T17:14:16.964 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-08-17T17:14:16.964 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.compression_opts.enabled: false 2024-08-17T17:14:16.964 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-08-17T17:14:16.964 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-08-17T17:14:16.965 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-08-17T17:14:16.965 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-08-17T17:14:16.965 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.target_file_size_base: 67108864 2024-08-17T17:14:16.965 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.target_file_size_multiplier: 1 2024-08-17T17:14:16.965 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-08-17T17:14:16.965 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-08-17T17:14:16.965 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-08-17T17:14:16.965 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-08-17T17:14:16.965 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-08-17T17:14:16.965 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-08-17T17:14:16.965 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-08-17T17:14:16.965 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-08-17T17:14:16.965 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-08-17T17:14:16.965 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-08-17T17:14:16.965 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-08-17T17:14:16.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-08-17T17:14:16.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-08-17T17:14:16.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.arena_block_size: 1048576 2024-08-17T17:14:16.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-08-17T17:14:16.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-08-17T17:14:16.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.disable_auto_compactions: 0 2024-08-17T17:14:16.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-08-17T17:14:16.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-08-17T17:14:16.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-08-17T17:14:16.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-08-17T17:14:16.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-08-17T17:14:16.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-08-17T17:14:16.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-08-17T17:14:16.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-08-17T17:14:16.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-08-17T17:14:16.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-08-17T17:14:16.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-08-17T17:14:16.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.inplace_update_support: 0 2024-08-17T17:14:16.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.inplace_update_num_locks: 10000 2024-08-17T17:14:16.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-08-17T17:14:16.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-08-17T17:14:16.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.memtable_huge_page_size: 0 2024-08-17T17:14:16.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.bloom_locality: 0 2024-08-17T17:14:16.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.max_successive_merges: 0 2024-08-17T17:14:16.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.optimize_filters_for_hits: 0 2024-08-17T17:14:16.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.paranoid_file_checks: 0 2024-08-17T17:14:16.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.force_consistency_checks: 1 2024-08-17T17:14:16.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.report_bg_io_stats: 0 2024-08-17T17:14:16.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.ttl: 2592000 2024-08-17T17:14:16.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.periodic_compaction_seconds: 0 2024-08-17T17:14:16.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-08-17T17:14:16.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-08-17T17:14:16.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.enable_blob_files: false 2024-08-17T17:14:16.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.min_blob_size: 0 2024-08-17T17:14:16.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.blob_file_size: 268435456 2024-08-17T17:14:16.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.blob_compression_type: NoCompression 2024-08-17T17:14:16.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.enable_blob_garbage_collection: false 2024-08-17T17:14:16.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-08-17T17:14:16.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-08-17T17:14:16.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-08-17T17:14:16.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.blob_file_starting_level: 0 2024-08-17T17:14:16.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-08-17T17:14:16.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-08-17T17:14:16.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-08-17T17:14:16.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 12972, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-08-17T17:14:16.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-08-17T17:14:16.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 8e09e9cd-2065-4c43-aa34-5ac2c8ff7843 2024-08-17T17:14:16.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723914856769068, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-08-17T17:14:16.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-08-17T17:14:16.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723914856781922, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 3368059, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12977, "largest_seqno": 14862, "table_properties": {"data_size": 3359805, "index_size": 5288, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1925, "raw_key_size": 17366, "raw_average_key_size": 22, "raw_value_size": 3343473, "raw_average_value_size": 4410, "num_data_blocks": 240, "num_entries": 758, "num_filter_entries": 758, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1723914856, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "8e09e9cd-2065-4c43-aa34-5ac2c8ff7843", "db_session_id": "5VW6IGC2I334UYL786SO", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-08-17T17:14:16.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: EVENT_LOG_v1 {"time_micros": 1723914856782049, "job": 1, "event": "recovery_finished"} 2024-08-17T17:14:16.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-08-17T17:14:16.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-08-17T17:14:16.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-08-17T17:14:16.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x564ad44cee00 2024-08-17T17:14:16.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: DB pointer 0x564ad45cc000 2024-08-17T17:14:16.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-08-17T17:14:16.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-08-17T17:14:16.970 INFO:journalctl@ceph.mon.b.smithi053.stdout: ** DB Stats ** 2024-08-17T17:14:16.970 INFO:journalctl@ceph.mon.b.smithi053.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-17T17:14:16.970 INFO:journalctl@ceph.mon.b.smithi053.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-08-17T17:14:16.970 INFO:journalctl@ceph.mon.b.smithi053.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-17T17:14:16.970 INFO:journalctl@ceph.mon.b.smithi053.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-17T17:14:16.970 INFO:journalctl@ceph.mon.b.smithi053.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-08-17T17:14:16.970 INFO:journalctl@ceph.mon.b.smithi053.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-17T17:14:16.970 INFO:journalctl@ceph.mon.b.smithi053.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-17T17:14:16.970 INFO:journalctl@ceph.mon.b.smithi053.stdout: 2024-08-17T17:14:16.970 INFO:journalctl@ceph.mon.b.smithi053.stdout: ** Compaction Stats [default] ** 2024-08-17T17:14:16.970 INFO:journalctl@ceph.mon.b.smithi053.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-17T17:14:16.971 INFO:journalctl@ceph.mon.b.smithi053.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-08-17T17:14:16.971 INFO:journalctl@ceph.mon.b.smithi053.stdout: L0 1/0 3.21 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 635.3 0.01 0.00 1 0.005 0 0 0.0 0.0 2024-08-17T17:14:16.971 INFO:journalctl@ceph.mon.b.smithi053.stdout: L6 1/0 7.35 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-08-17T17:14:16.971 INFO:journalctl@ceph.mon.b.smithi053.stdout: Sum 2/0 10.56 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 635.3 0.01 0.00 1 0.005 0 0 0.0 0.0 2024-08-17T17:14:16.971 INFO:journalctl@ceph.mon.b.smithi053.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 635.3 0.01 0.00 1 0.005 0 0 0.0 0.0 2024-08-17T17:14:16.971 INFO:journalctl@ceph.mon.b.smithi053.stdout: 2024-08-17T17:14:16.971 INFO:journalctl@ceph.mon.b.smithi053.stdout: ** Compaction Stats [default] ** 2024-08-17T17:14:16.971 INFO:journalctl@ceph.mon.b.smithi053.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-17T17:14:16.971 INFO:journalctl@ceph.mon.b.smithi053.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-08-17T17:14:16.971 INFO:journalctl@ceph.mon.b.smithi053.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 635.3 0.01 0.00 1 0.005 0 0 0.0 0.0 2024-08-17T17:14:16.971 INFO:journalctl@ceph.mon.b.smithi053.stdout: 2024-08-17T17:14:16.971 INFO:journalctl@ceph.mon.b.smithi053.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-08-17T17:14:16.971 INFO:journalctl@ceph.mon.b.smithi053.stdout: 2024-08-17T17:14:16.971 INFO:journalctl@ceph.mon.b.smithi053.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-17T17:14:16.971 INFO:journalctl@ceph.mon.b.smithi053.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-08-17T17:14:16.972 INFO:journalctl@ceph.mon.b.smithi053.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-08-17T17:14:16.972 INFO:journalctl@ceph.mon.b.smithi053.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-08-17T17:14:16.972 INFO:journalctl@ceph.mon.b.smithi053.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-08-17T17:14:16.972 INFO:journalctl@ceph.mon.b.smithi053.stdout: AddFile(Keys): cumulative 0, interval 0 2024-08-17T17:14:16.972 INFO:journalctl@ceph.mon.b.smithi053.stdout: Cumulative compaction: 0.00 GB write, 206.76 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-17T17:14:16.972 INFO:journalctl@ceph.mon.b.smithi053.stdout: Interval compaction: 0.00 GB write, 206.76 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-17T17:14:16.972 INFO:journalctl@ceph.mon.b.smithi053.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-08-17T17:14:16.972 INFO:journalctl@ceph.mon.b.smithi053.stdout: Block cache BinnedLRUCache@0x564ad449f610#2 capacity: 512.00 MB usage: 8.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-08-17T17:14:16.972 INFO:journalctl@ceph.mon.b.smithi053.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.03 KB,0.00038743%) IndexBlock(1,6.08 KB,0.00115931%) Misc(1,0.00 KB,0%) 2024-08-17T17:14:16.972 INFO:journalctl@ceph.mon.b.smithi053.stdout: 2024-08-17T17:14:16.972 INFO:journalctl@ceph.mon.b.smithi053.stdout: ** File Read Latency Histogram By Level [default] ** 2024-08-17T17:14:16.972 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: starting mon.b rank 1 at public addrs [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] at bind addrs [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:14:16.972 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: mon.b@-1(???) e3 preinit fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:14:16.972 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: mon.b@-1(???).mds e9 new map 2024-08-17T17:14:16.973 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: mon.b@-1(???).mds e9 print_map 2024-08-17T17:14:16.973 INFO:journalctl@ceph.mon.b.smithi053.stdout: e9 2024-08-17T17:14:16.973 INFO:journalctl@ceph.mon.b.smithi053.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-08-17T17:14:16.973 INFO:journalctl@ceph.mon.b.smithi053.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-08-17T17:14:16.973 INFO:journalctl@ceph.mon.b.smithi053.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-08-17T17:14:16.973 INFO:journalctl@ceph.mon.b.smithi053.stdout: legacy client fscid: 1 2024-08-17T17:14:16.973 INFO:journalctl@ceph.mon.b.smithi053.stdout: 2024-08-17T17:14:16.973 INFO:journalctl@ceph.mon.b.smithi053.stdout: Filesystem 'foo' (1) 2024-08-17T17:14:16.973 INFO:journalctl@ceph.mon.b.smithi053.stdout: fs_name foo 2024-08-17T17:14:16.973 INFO:journalctl@ceph.mon.b.smithi053.stdout: epoch 8 2024-08-17T17:14:16.973 INFO:journalctl@ceph.mon.b.smithi053.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-08-17T17:14:16.973 INFO:journalctl@ceph.mon.b.smithi053.stdout: created 2024-08-17T17:07:46.147672+0000 2024-08-17T17:14:16.973 INFO:journalctl@ceph.mon.b.smithi053.stdout: modified 2024-08-17T17:07:55.190877+0000 2024-08-17T17:14:16.973 INFO:journalctl@ceph.mon.b.smithi053.stdout: tableserver 0 2024-08-17T17:14:16.974 INFO:journalctl@ceph.mon.b.smithi053.stdout: root 0 2024-08-17T17:14:16.974 INFO:journalctl@ceph.mon.b.smithi053.stdout: session_timeout 60 2024-08-17T17:14:16.974 INFO:journalctl@ceph.mon.b.smithi053.stdout: session_autoclose 300 2024-08-17T17:14:16.974 INFO:journalctl@ceph.mon.b.smithi053.stdout: max_file_size 1099511627776 2024-08-17T17:14:16.974 INFO:journalctl@ceph.mon.b.smithi053.stdout: max_xattr_size 65536 2024-08-17T17:14:16.974 INFO:journalctl@ceph.mon.b.smithi053.stdout: required_client_features {} 2024-08-17T17:14:16.974 INFO:journalctl@ceph.mon.b.smithi053.stdout: last_failure 0 2024-08-17T17:14:16.974 INFO:journalctl@ceph.mon.b.smithi053.stdout: last_failure_osd_epoch 0 2024-08-17T17:14:16.974 INFO:journalctl@ceph.mon.b.smithi053.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-08-17T17:14:16.974 INFO:journalctl@ceph.mon.b.smithi053.stdout: max_mds 1 2024-08-17T17:14:16.974 INFO:journalctl@ceph.mon.b.smithi053.stdout: in 0 2024-08-17T17:14:16.974 INFO:journalctl@ceph.mon.b.smithi053.stdout: up {0=14664} 2024-08-17T17:14:16.974 INFO:journalctl@ceph.mon.b.smithi053.stdout: failed 2024-08-17T17:14:16.975 INFO:journalctl@ceph.mon.b.smithi053.stdout: damaged 2024-08-17T17:14:16.975 INFO:journalctl@ceph.mon.b.smithi053.stdout: stopped 2024-08-17T17:14:16.975 INFO:journalctl@ceph.mon.b.smithi053.stdout: data_pools [4] 2024-08-17T17:14:16.975 INFO:journalctl@ceph.mon.b.smithi053.stdout: metadata_pool 3 2024-08-17T17:14:16.975 INFO:journalctl@ceph.mon.b.smithi053.stdout: inline_data disabled 2024-08-17T17:14:16.975 INFO:journalctl@ceph.mon.b.smithi053.stdout: balancer 2024-08-17T17:14:16.975 INFO:journalctl@ceph.mon.b.smithi053.stdout: bal_rank_mask -1 2024-08-17T17:14:16.975 INFO:journalctl@ceph.mon.b.smithi053.stdout: standby_count_wanted 1 2024-08-17T17:14:16.975 INFO:journalctl@ceph.mon.b.smithi053.stdout: qdb_cluster leader: 0 members: 2024-08-17T17:14:16.975 INFO:journalctl@ceph.mon.b.smithi053.stdout: [mds.foo.smithi003.civgvf{0:14664} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.3:6834/2800254789,v1:172.21.15.3:6835/2800254789] compat {c=[1],r=[1],i=[7ff]}] 2024-08-17T17:14:16.975 INFO:journalctl@ceph.mon.b.smithi053.stdout: 2024-08-17T17:14:16.975 INFO:journalctl@ceph.mon.b.smithi053.stdout: 2024-08-17T17:14:16.975 INFO:journalctl@ceph.mon.b.smithi053.stdout: Standby daemons: 2024-08-17T17:14:16.976 INFO:journalctl@ceph.mon.b.smithi053.stdout: 2024-08-17T17:14:16.976 INFO:journalctl@ceph.mon.b.smithi053.stdout: [mds.foo.smithi053.ougbtc{-1:24511} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.53:6832/264432371,v1:172.21.15.53:6833/264432371] compat {c=[1],r=[1],i=[7ff]}] 2024-08-17T17:14:16.976 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: mon.b@-1(???).osd e90 crush map has features 3314933000854323200, adjusting msgr requires 2024-08-17T17:14:16.976 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: mon.b@-1(???).osd e90 crush map has features 432629239337189376, adjusting msgr requires 2024-08-17T17:14:16.976 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: mon.b@-1(???).osd e90 crush map has features 432629239337189376, adjusting msgr requires 2024-08-17T17:14:16.976 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: mon.b@-1(???).osd e90 crush map has features 432629239337189376, adjusting msgr requires 2024-08-17T17:14:16.976 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-mon[89963]: mon.b@-1(???).paxosservice(auth 1..21) refresh upgraded, format 0 -> 3 2024-08-17T17:14:16.976 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 bash[89945]: 9dc96112f3afceec9ace658d33f047e7cde585c3c68bf30a416d01767198c245 2024-08-17T17:14:16.976 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:16 smithi053 systemd[1]: Started Ceph mon.b for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:14:17.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:14:16 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:14:16.947Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi053\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:14:18.619 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:18 smithi053 ceph-mon[89963]: pgmap v51: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:14:18.625 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 3 2024-08-17T17:14:18.625 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:18 smithi003 ceph-mon[103344]: pgmap v51: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:14:18.626 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:18 smithi003 ceph-mon[100615]: pgmap v51: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:14:19.197 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'ceph orch ps' 2024-08-17T17:14:19.470 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:14:19.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[100615]: pgmap v53: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:14:19.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-17T17:14:19.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[100615]: mon.b calling monitor election 2024-08-17T17:14:19.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[100615]: mon.c calling monitor election 2024-08-17T17:14:19.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[100615]: mon.a calling monitor election 2024-08-17T17:14:19.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-17T17:14:19.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-17T17:14:19.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[100615]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-08-17T17:14:19.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[100615]: monmap epoch 4 2024-08-17T17:14:19.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[100615]: fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:14:19.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[100615]: last_changed 2024-08-17T17:14:18.294385+0000 2024-08-17T17:14:19.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[100615]: created 2024-08-17T17:00:53.391169+0000 2024-08-17T17:14:19.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[100615]: min_mon_release 19 (squid) 2024-08-17T17:14:19.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[100615]: election_strategy: 1 2024-08-17T17:14:19.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[100615]: 0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2024-08-17T17:14:19.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[100615]: 1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-08-17T17:14:19.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[100615]: 2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2024-08-17T17:14:19.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[100615]: fsmap foo:1 {0=foo.smithi003.civgvf=up:active} 1 up:standby 2024-08-17T17:14:19.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[100615]: osdmap e90: 8 total, 8 up, 8 in 2024-08-17T17:14:19.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[100615]: mgrmap e39: y(active, since 100s), standbys: x 2024-08-17T17:14:19.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[100615]: overall HEALTH_OK 2024-08-17T17:14:19.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:19.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:19.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:14:19.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1737667394' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-08-17T17:14:19.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[103344]: pgmap v53: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:14:19.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-17T17:14:19.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[103344]: mon.b calling monitor election 2024-08-17T17:14:19.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[103344]: mon.c calling monitor election 2024-08-17T17:14:19.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[103344]: mon.a calling monitor election 2024-08-17T17:14:19.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-17T17:14:19.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-17T17:14:19.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[103344]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-08-17T17:14:19.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[103344]: monmap epoch 4 2024-08-17T17:14:19.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[103344]: fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:14:19.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[103344]: last_changed 2024-08-17T17:14:18.294385+0000 2024-08-17T17:14:19.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[103344]: created 2024-08-17T17:00:53.391169+0000 2024-08-17T17:14:19.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[103344]: min_mon_release 19 (squid) 2024-08-17T17:14:19.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[103344]: election_strategy: 1 2024-08-17T17:14:19.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[103344]: 0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2024-08-17T17:14:19.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[103344]: 1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-08-17T17:14:19.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[103344]: 2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2024-08-17T17:14:19.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[103344]: fsmap foo:1 {0=foo.smithi003.civgvf=up:active} 1 up:standby 2024-08-17T17:14:19.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[103344]: osdmap e90: 8 total, 8 up, 8 in 2024-08-17T17:14:19.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[103344]: mgrmap e39: y(active, since 100s), standbys: x 2024-08-17T17:14:19.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[103344]: overall HEALTH_OK 2024-08-17T17:14:19.527 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:19.527 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:19.527 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:14:19.527 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:19 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1737667394' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-08-17T17:14:19.633 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:19 smithi053 ceph-mon[89963]: pgmap v53: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:14:19.633 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:19 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-17T17:14:19.633 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:19 smithi053 ceph-mon[89963]: mon.b calling monitor election 2024-08-17T17:14:19.633 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:19 smithi053 ceph-mon[89963]: mon.c calling monitor election 2024-08-17T17:14:19.633 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:19 smithi053 ceph-mon[89963]: mon.a calling monitor election 2024-08-17T17:14:19.633 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:19 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-17T17:14:19.634 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:19 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-17T17:14:19.634 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:19 smithi053 ceph-mon[89963]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-08-17T17:14:19.634 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:19 smithi053 ceph-mon[89963]: monmap epoch 4 2024-08-17T17:14:19.634 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:19 smithi053 ceph-mon[89963]: fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:14:19.634 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:19 smithi053 ceph-mon[89963]: last_changed 2024-08-17T17:14:18.294385+0000 2024-08-17T17:14:19.634 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:19 smithi053 ceph-mon[89963]: created 2024-08-17T17:00:53.391169+0000 2024-08-17T17:14:19.634 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:19 smithi053 ceph-mon[89963]: min_mon_release 19 (squid) 2024-08-17T17:14:19.634 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:19 smithi053 ceph-mon[89963]: election_strategy: 1 2024-08-17T17:14:19.634 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:19 smithi053 ceph-mon[89963]: 0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2024-08-17T17:14:19.634 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:19 smithi053 ceph-mon[89963]: 1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-08-17T17:14:19.634 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:19 smithi053 ceph-mon[89963]: 2: [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon.c 2024-08-17T17:14:19.634 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:19 smithi053 ceph-mon[89963]: fsmap foo:1 {0=foo.smithi003.civgvf=up:active} 1 up:standby 2024-08-17T17:14:19.634 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:19 smithi053 ceph-mon[89963]: osdmap e90: 8 total, 8 up, 8 in 2024-08-17T17:14:19.635 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:19 smithi053 ceph-mon[89963]: mgrmap e39: y(active, since 100s), standbys: x 2024-08-17T17:14:19.635 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:19 smithi053 ceph-mon[89963]: overall HEALTH_OK 2024-08-17T17:14:19.635 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:19 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:19.635 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:19 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:19.635 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:19 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:14:19.635 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:19 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1737667394' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-08-17T17:14:20.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:14:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:14:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:14:21.162 INFO:teuthology.orchestra.run.smithi003.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-08-17T17:14:21.163 INFO:teuthology.orchestra.run.smithi003.stdout:alertmanager.a smithi003 *:9093,9094 running (3m) 9s ago 7m 18.8M - 0.27.0 11f11916f8cd fb501c5aa873 2024-08-17T17:14:21.163 INFO:teuthology.orchestra.run.smithi003.stdout:grafana.a smithi053 *:3000 running (3m) 11s ago 7m 88.8M - 9.4.7 954c08fa6188 9b5ef97a5cd5 2024-08-17T17:14:21.163 INFO:teuthology.orchestra.run.smithi003.stdout:mds.foo.smithi003.civgvf smithi003 running (6m) 9s ago 6m 19.6M - 17.2.7-1404-gf2c39d4d f69471c56189 86a91b686f66 2024-08-17T17:14:21.163 INFO:teuthology.orchestra.run.smithi003.stdout:mds.foo.smithi053.ougbtc smithi053 running (6m) 11s ago 6m 16.3M - 17.2.7-1404-gf2c39d4d f69471c56189 41ad4cc0ce19 2024-08-17T17:14:21.163 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.x smithi053 *:8443,9283,8765 running (2m) 11s ago 11m 452M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 874be7d3a238 2024-08-17T17:14:21.163 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.y smithi003 *:8443,9283,8765 running (2m) 9s ago 13m 507M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb f491ea24df28 2024-08-17T17:14:21.163 INFO:teuthology.orchestra.run.smithi003.stdout:mon.a smithi003 running (112s) 9s ago 13m 39.9M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb c1cfbc2d47f8 2024-08-17T17:14:21.163 INFO:teuthology.orchestra.run.smithi003.stdout:mon.b smithi053 starting - - - 2048M 2024-08-17T17:14:21.163 INFO:teuthology.orchestra.run.smithi003.stdout:mon.c smithi003 running (89s) 9s ago 11m 34.0M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 54a6cfe9eb9a 2024-08-17T17:14:21.163 INFO:teuthology.orchestra.run.smithi003.stdout:node-exporter.a smithi003 *:9100 running (3m) 9s ago 7m 11.6M - 1.7.0 72c9c2088986 d19c9e3d51f8 2024-08-17T17:14:21.163 INFO:teuthology.orchestra.run.smithi003.stdout:node-exporter.b smithi053 *:9100 running (3m) 11s ago 7m 10.5M - 1.7.0 72c9c2088986 edfcc96e3983 2024-08-17T17:14:21.163 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0 smithi003 running (11m) 9s ago 11m 51.3M 2481M 17.2.7-1404-gf2c39d4d f69471c56189 dc8e5d51fe1c 2024-08-17T17:14:21.164 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1 smithi003 running (10m) 9s ago 10m 51.1M 2481M 17.2.7-1404-gf2c39d4d f69471c56189 7def3d9ed99f 2024-08-17T17:14:21.164 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2 smithi003 running (10m) 9s ago 10m 48.2M 2481M 17.2.7-1404-gf2c39d4d f69471c56189 3d65e6f7d660 2024-08-17T17:14:21.164 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3 smithi003 running (10m) 9s ago 10m 46.1M 2481M 17.2.7-1404-gf2c39d4d f69471c56189 61f40a5861f6 2024-08-17T17:14:21.164 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4 smithi053 running (9m) 11s ago 9m 47.8M 2481M 17.2.7-1404-gf2c39d4d f69471c56189 ced3b086d7f0 2024-08-17T17:14:21.164 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5 smithi053 running (9m) 11s ago 9m 47.4M 2481M 17.2.7-1404-gf2c39d4d f69471c56189 1c4ad574a193 2024-08-17T17:14:21.164 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6 smithi053 running (8m) 11s ago 8m 47.1M 2481M 17.2.7-1404-gf2c39d4d f69471c56189 5ed81ad77c62 2024-08-17T17:14:21.164 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7 smithi053 running (8m) 11s ago 8m 48.9M 2481M 17.2.7-1404-gf2c39d4d f69471c56189 f3e467b07793 2024-08-17T17:14:21.164 INFO:teuthology.orchestra.run.smithi003.stdout:prometheus.a smithi053 *:9095 running (2m) 11s ago 8m 47.3M - 2.51.0 1d3b7f56885b 41146206916a 2024-08-17T17:14:21.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:21 smithi053 ceph-mon[89963]: pgmap v54: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:14:21.474 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:21 smithi003 ceph-mon[103344]: pgmap v54: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:14:21.474 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:21 smithi003 ceph-mon[100615]: pgmap v54: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-08-17T17:14:21.812 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'ceph orch upgrade pause' 2024-08-17T17:14:22.079 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:14:22.200 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:14:21 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:14:21.792+0000 7f288dc05640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-08-17T17:14:22.511 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:22 smithi003 ceph-mon[103344]: from='client.44144 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:22.512 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:22 smithi003 ceph-mon[100615]: from='client.44144 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:22.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:22 smithi053 ceph-mon[89963]: from='client.44144 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:23.491 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:23 smithi003 ceph-mon[103344]: pgmap v55: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:23.491 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:23.491 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:23.491 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:23.491 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:23.491 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:14:23.492 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:23 smithi003 ceph-mon[100615]: pgmap v55: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:23.492 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:23.492 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:23.492 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:23.492 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:23.492 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:14:23.677 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:23 smithi053 ceph-mon[89963]: pgmap v55: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:23.677 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:23.677 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:23.677 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:23.677 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:23.677 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:14:23.925 INFO:teuthology.orchestra.run.smithi003.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:14:24.461 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:14:24 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:14:24.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:14:24.536 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'sleep 60' 2024-08-17T17:14:24.807 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:14:25.026 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:24 smithi053 ceph-mon[89963]: pgmap v56: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:25.026 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:24 smithi053 ceph-mon[89963]: from='client.34290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:25.026 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:24 smithi053 ceph-mon[89963]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:14:25.026 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:24 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:25.226 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:24 smithi003 ceph-mon[103344]: pgmap v56: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:25.226 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:24 smithi003 ceph-mon[103344]: from='client.34290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:25.226 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:24 smithi003 ceph-mon[103344]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:14:25.226 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:24 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:25.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:24 smithi003 ceph-mon[100615]: pgmap v56: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:25.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:24 smithi003 ceph-mon[100615]: from='client.34290 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:14:25.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:24 smithi003 ceph-mon[100615]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:14:25.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:24 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:25.746 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:14:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:14:25] "GET /metrics HTTP/1.1" 200 33823 "" "Prometheus/2.51.0" 2024-08-17T17:14:25.963 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:14:25 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:14:25.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:14:25.963 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:14:25 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:14:25.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:14:27.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:14:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:14:26.952Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi053\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:14:27.475 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:27 smithi003 ceph-mon[103344]: pgmap v57: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:27.475 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:27 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:27.475 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:27 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:27.475 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:27 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:27.475 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:27 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:14:27.475 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:27 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:27.475 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:27 smithi003 ceph-mon[103344]: Reconfiguring mon.a (monmap changed)... 2024-08-17T17:14:27.475 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:27 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-17T17:14:27.475 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:27 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-17T17:14:27.475 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:27 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:27.475 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:27 smithi003 ceph-mon[103344]: Reconfiguring daemon mon.a on smithi003 2024-08-17T17:14:27.476 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:27 smithi003 ceph-mon[100615]: pgmap v57: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:27.476 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:27 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:27.476 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:27 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:27.476 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:27 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:27.476 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:27 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:14:27.476 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:27 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:27.476 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:27 smithi003 ceph-mon[100615]: Reconfiguring mon.a (monmap changed)... 2024-08-17T17:14:27.476 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:27 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-17T17:14:27.476 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:27 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-17T17:14:27.476 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:27 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:27.476 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:27 smithi003 ceph-mon[100615]: Reconfiguring daemon mon.a on smithi003 2024-08-17T17:14:27.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:27 smithi053 ceph-mon[89963]: pgmap v57: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:27.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:27 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:27.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:27 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:27.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:27 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:27.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:27 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:14:27.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:27 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:27.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:27 smithi053 ceph-mon[89963]: Reconfiguring mon.a (monmap changed)... 2024-08-17T17:14:27.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:27 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-17T17:14:27.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:27 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-17T17:14:27.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:27 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:27.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:27 smithi053 ceph-mon[89963]: Reconfiguring daemon mon.a on smithi003 2024-08-17T17:14:29.106 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:28 smithi053 ceph-mon[89963]: pgmap v58: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:29.107 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:28 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:29.107 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:28 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:29.107 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:28 smithi053 ceph-mon[89963]: Reconfiguring mgr.y (monmap changed)... 2024-08-17T17:14:29.107 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:28 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-17T17:14:29.107 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:28 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-08-17T17:14:29.107 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:28 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:29.107 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:28 smithi053 ceph-mon[89963]: Reconfiguring daemon mgr.y on smithi003 2024-08-17T17:14:29.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:28 smithi003 ceph-mon[103344]: pgmap v58: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:29.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:28 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:29.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:28 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:29.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:28 smithi003 ceph-mon[103344]: Reconfiguring mgr.y (monmap changed)... 2024-08-17T17:14:29.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:28 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-17T17:14:29.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:28 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-08-17T17:14:29.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:28 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:29.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:28 smithi003 ceph-mon[103344]: Reconfiguring daemon mgr.y on smithi003 2024-08-17T17:14:29.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:28 smithi003 ceph-mon[100615]: pgmap v58: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:29.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:28 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:29.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:28 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:29.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:28 smithi003 ceph-mon[100615]: Reconfiguring mgr.y (monmap changed)... 2024-08-17T17:14:29.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:28 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-17T17:14:29.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:28 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-08-17T17:14:29.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:28 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:29.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:28 smithi003 ceph-mon[100615]: Reconfiguring daemon mgr.y on smithi003 2024-08-17T17:14:30.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:14:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:14:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:14:30.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:30 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:30.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:30 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:30.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:30 smithi053 ceph-mon[89963]: Reconfiguring mon.c (monmap changed)... 2024-08-17T17:14:30.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:30 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-17T17:14:30.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:30 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-17T17:14:30.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:30 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:30.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:30 smithi053 ceph-mon[89963]: Reconfiguring daemon mon.c on smithi003 2024-08-17T17:14:30.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:30 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:30.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:30 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:30.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:30 smithi003 ceph-mon[103344]: Reconfiguring mon.c (monmap changed)... 2024-08-17T17:14:30.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:30 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-17T17:14:30.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:30 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-17T17:14:30.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:30 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:30.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:30 smithi003 ceph-mon[103344]: Reconfiguring daemon mon.c on smithi003 2024-08-17T17:14:30.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:30 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:30.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:30 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:30.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:30 smithi003 ceph-mon[100615]: Reconfiguring mon.c (monmap changed)... 2024-08-17T17:14:30.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:30 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-17T17:14:30.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:30 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-17T17:14:30.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:30 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:30.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:30 smithi003 ceph-mon[100615]: Reconfiguring daemon mon.c on smithi003 2024-08-17T17:14:31.510 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:31 smithi003 ceph-mon[103344]: pgmap v59: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:31.510 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:31 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:31.510 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:31 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:31.510 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:31 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-08-17T17:14:31.510 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:31 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:31.511 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:31 smithi003 ceph-mon[100615]: pgmap v59: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:31.511 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:31 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:31.511 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:31 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:31.511 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:31 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-08-17T17:14:31.511 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:31 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:31.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:31 smithi053 ceph-mon[89963]: pgmap v59: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:31.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:31 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:31.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:31 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:31.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:31 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-08-17T17:14:31.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:31 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:32.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:32 smithi003 ceph-mon[103344]: Reconfiguring osd.0 (monmap changed)... 2024-08-17T17:14:32.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:32 smithi003 ceph-mon[103344]: Reconfiguring daemon osd.0 on smithi003 2024-08-17T17:14:32.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:32 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:32.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:32 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:32.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:32 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-08-17T17:14:32.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:32 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:32.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:32 smithi003 ceph-mon[100615]: Reconfiguring osd.0 (monmap changed)... 2024-08-17T17:14:32.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:32 smithi003 ceph-mon[100615]: Reconfiguring daemon osd.0 on smithi003 2024-08-17T17:14:32.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:32 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:32.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:32 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:32.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:32 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-08-17T17:14:32.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:32 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:32.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:32 smithi053 ceph-mon[89963]: Reconfiguring osd.0 (monmap changed)... 2024-08-17T17:14:32.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:32 smithi053 ceph-mon[89963]: Reconfiguring daemon osd.0 on smithi003 2024-08-17T17:14:32.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:32 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:32.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:32 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:32.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:32 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-08-17T17:14:32.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:32 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:33.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:33 smithi053 ceph-mon[89963]: pgmap v60: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:33.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:33 smithi053 ceph-mon[89963]: Reconfiguring osd.1 (monmap changed)... 2024-08-17T17:14:33.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:33 smithi053 ceph-mon[89963]: Reconfiguring daemon osd.1 on smithi003 2024-08-17T17:14:33.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:33 smithi003 ceph-mon[103344]: pgmap v60: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:33.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:33 smithi003 ceph-mon[103344]: Reconfiguring osd.1 (monmap changed)... 2024-08-17T17:14:33.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:33 smithi003 ceph-mon[103344]: Reconfiguring daemon osd.1 on smithi003 2024-08-17T17:14:33.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:33 smithi003 ceph-mon[100615]: pgmap v60: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:33.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:33 smithi003 ceph-mon[100615]: Reconfiguring osd.1 (monmap changed)... 2024-08-17T17:14:33.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:33 smithi003 ceph-mon[100615]: Reconfiguring daemon osd.1 on smithi003 2024-08-17T17:14:34.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:14:34 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:14:34.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:14:34.901 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:34 smithi053 ceph-mon[89963]: pgmap v61: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:34.982 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:34 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:34.983 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:34 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:34.983 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:34 smithi053 ceph-mon[89963]: Reconfiguring osd.2 (monmap changed)... 2024-08-17T17:14:34.983 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:34 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-08-17T17:14:34.983 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:34 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:34.983 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:34 smithi053 ceph-mon[89963]: Reconfiguring daemon osd.2 on smithi003 2024-08-17T17:14:34.983 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:34 smithi003 ceph-mon[103344]: pgmap v61: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:34.984 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:34 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:34.984 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:34 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:34.984 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:34 smithi003 ceph-mon[103344]: Reconfiguring osd.2 (monmap changed)... 2024-08-17T17:14:34.984 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:34 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-08-17T17:14:34.984 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:34 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:34.984 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:34 smithi003 ceph-mon[103344]: Reconfiguring daemon osd.2 on smithi003 2024-08-17T17:14:34.984 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:34 smithi003 ceph-mon[100615]: pgmap v61: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:34.984 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:34 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:34.984 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:34 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:34.984 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:34 smithi003 ceph-mon[100615]: Reconfiguring osd.2 (monmap changed)... 2024-08-17T17:14:34.985 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:34 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-08-17T17:14:34.985 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:34 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:34.985 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:34 smithi003 ceph-mon[100615]: Reconfiguring daemon osd.2 on smithi003 2024-08-17T17:14:35.731 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:14:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:14:35] "GET /metrics HTTP/1.1" 200 33823 "" "Prometheus/2.51.0" 2024-08-17T17:14:36.031 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:14:35 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:14:35.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:14:36.032 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:14:35 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:14:35.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:14:36.369 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:36 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:36.369 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:36 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:36.369 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:36 smithi003 ceph-mon[103344]: Reconfiguring osd.3 (monmap changed)... 2024-08-17T17:14:36.369 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:36 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-08-17T17:14:36.369 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:36 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:36.369 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:36 smithi003 ceph-mon[103344]: Reconfiguring daemon osd.3 on smithi003 2024-08-17T17:14:36.370 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:36 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:36.370 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:36 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:36.370 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:36 smithi003 ceph-mon[100615]: Reconfiguring osd.3 (monmap changed)... 2024-08-17T17:14:36.370 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:36 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-08-17T17:14:36.370 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:36 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:36.370 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:36 smithi003 ceph-mon[100615]: Reconfiguring daemon osd.3 on smithi003 2024-08-17T17:14:36.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:36 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:36.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:36 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:36.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:36 smithi053 ceph-mon[89963]: Reconfiguring osd.3 (monmap changed)... 2024-08-17T17:14:36.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:36 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-08-17T17:14:36.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:36 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:36.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:36 smithi053 ceph-mon[89963]: Reconfiguring daemon osd.3 on smithi003 2024-08-17T17:14:37.109 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:37 smithi003 ceph-mon[103344]: pgmap v62: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:37.109 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:37 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:37.109 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:37 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:37.109 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:37 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi003.civgvf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-17T17:14:37.109 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:37 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:37.110 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:37 smithi003 ceph-mon[100615]: pgmap v62: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:37.110 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:37 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:37.110 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:37 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:37.110 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:37 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi003.civgvf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-17T17:14:37.110 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:37 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:37.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:14:36 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:14:36.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi053\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:14:37.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:37 smithi053 ceph-mon[89963]: pgmap v62: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:37.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:37 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:37.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:37 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:37.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:37 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi003.civgvf", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-17T17:14:37.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:37 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:38.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:38 smithi003 ceph-mon[100615]: Reconfiguring mds.foo.smithi003.civgvf (monmap changed)... 2024-08-17T17:14:38.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:38 smithi003 ceph-mon[100615]: Reconfiguring daemon mds.foo.smithi003.civgvf on smithi003 2024-08-17T17:14:38.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:38.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:38.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-17T17:14:38.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-17T17:14:38.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:38.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:38 smithi003 ceph-mon[103344]: Reconfiguring mds.foo.smithi003.civgvf (monmap changed)... 2024-08-17T17:14:38.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:38 smithi003 ceph-mon[103344]: Reconfiguring daemon mds.foo.smithi003.civgvf on smithi003 2024-08-17T17:14:38.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:38.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:38.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-17T17:14:38.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-17T17:14:38.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:38.400 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:38 smithi053 ceph-mon[89963]: Reconfiguring mds.foo.smithi003.civgvf (monmap changed)... 2024-08-17T17:14:38.400 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:38 smithi053 ceph-mon[89963]: Reconfiguring daemon mds.foo.smithi003.civgvf on smithi003 2024-08-17T17:14:38.400 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:38.401 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:38.401 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-17T17:14:38.401 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-17T17:14:38.401 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:39.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:39 smithi053 ceph-mon[89963]: pgmap v63: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:39.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:39 smithi053 ceph-mon[89963]: Reconfiguring mon.b (monmap changed)... 2024-08-17T17:14:39.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:39 smithi053 ceph-mon[89963]: Reconfiguring daemon mon.b on smithi053 2024-08-17T17:14:39.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:14:39.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:39 smithi003 ceph-mon[103344]: pgmap v63: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:39.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:39 smithi003 ceph-mon[103344]: Reconfiguring mon.b (monmap changed)... 2024-08-17T17:14:39.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:39 smithi003 ceph-mon[103344]: Reconfiguring daemon mon.b on smithi053 2024-08-17T17:14:39.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:14:39.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:39 smithi003 ceph-mon[100615]: pgmap v63: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:39.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:39 smithi003 ceph-mon[100615]: Reconfiguring mon.b (monmap changed)... 2024-08-17T17:14:39.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:39 smithi003 ceph-mon[100615]: Reconfiguring daemon mon.b on smithi053 2024-08-17T17:14:39.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:14:40.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:14:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:14:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:14:40.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:40 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:40.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:40 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:40.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:40 smithi053 ceph-mon[89963]: Reconfiguring mgr.x (monmap changed)... 2024-08-17T17:14:40.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:40 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-17T17:14:40.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:40 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-08-17T17:14:40.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:40 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:40.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:40 smithi053 ceph-mon[89963]: Reconfiguring daemon mgr.x on smithi053 2024-08-17T17:14:40.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:40 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:40.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:40 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:40.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:40 smithi003 ceph-mon[103344]: Reconfiguring mgr.x (monmap changed)... 2024-08-17T17:14:40.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:40 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-17T17:14:40.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:40 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-08-17T17:14:40.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:40 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:40.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:40 smithi003 ceph-mon[103344]: Reconfiguring daemon mgr.x on smithi053 2024-08-17T17:14:40.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:40 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:40.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:40 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:40.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:40 smithi003 ceph-mon[100615]: Reconfiguring mgr.x (monmap changed)... 2024-08-17T17:14:40.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:40 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-17T17:14:40.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:40 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-08-17T17:14:40.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:40 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:40.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:40 smithi003 ceph-mon[100615]: Reconfiguring daemon mgr.x on smithi053 2024-08-17T17:14:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:41 smithi003 ceph-mon[100615]: pgmap v64: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:41 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:41 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:41 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-08-17T17:14:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:41 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:41.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:41 smithi003 ceph-mon[103344]: pgmap v64: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:41.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:41 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:41.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:41 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:41.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:41 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-08-17T17:14:41.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:41 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:41.623 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:41 smithi053 ceph-mon[89963]: pgmap v64: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:41.623 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:41 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:41.623 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:41 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:41.623 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:41 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-08-17T17:14:41.623 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:41 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:42.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:42 smithi053 ceph-mon[89963]: Reconfiguring osd.4 (monmap changed)... 2024-08-17T17:14:42.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:42 smithi053 ceph-mon[89963]: Reconfiguring daemon osd.4 on smithi053 2024-08-17T17:14:42.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:42 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:42.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:42 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:42.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:42 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-08-17T17:14:42.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:42 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:42.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:42 smithi003 ceph-mon[100615]: Reconfiguring osd.4 (monmap changed)... 2024-08-17T17:14:42.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:42 smithi003 ceph-mon[100615]: Reconfiguring daemon osd.4 on smithi053 2024-08-17T17:14:42.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:42 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:42.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:42 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:42.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:42 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-08-17T17:14:42.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:42 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:42.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:42 smithi003 ceph-mon[103344]: Reconfiguring osd.4 (monmap changed)... 2024-08-17T17:14:42.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:42 smithi003 ceph-mon[103344]: Reconfiguring daemon osd.4 on smithi053 2024-08-17T17:14:42.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:42 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:42.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:42 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:42.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:42 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-08-17T17:14:42.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:42 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:43.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:43 smithi053 ceph-mon[89963]: pgmap v65: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:43.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:43 smithi053 ceph-mon[89963]: Reconfiguring osd.5 (monmap changed)... 2024-08-17T17:14:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:43 smithi053 ceph-mon[89963]: Reconfiguring daemon osd.5 on smithi053 2024-08-17T17:14:43.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:43 smithi003 ceph-mon[103344]: pgmap v65: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:43.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:43 smithi003 ceph-mon[103344]: Reconfiguring osd.5 (monmap changed)... 2024-08-17T17:14:43.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:43 smithi003 ceph-mon[103344]: Reconfiguring daemon osd.5 on smithi053 2024-08-17T17:14:43.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:43 smithi003 ceph-mon[100615]: pgmap v65: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:43.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:43 smithi003 ceph-mon[100615]: Reconfiguring osd.5 (monmap changed)... 2024-08-17T17:14:43.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:43 smithi003 ceph-mon[100615]: Reconfiguring daemon osd.5 on smithi053 2024-08-17T17:14:44.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:14:44 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:14:44.146Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:14:44.866 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:44 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:44.866 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:44 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:44.866 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:44 smithi053 ceph-mon[89963]: Reconfiguring osd.6 (monmap changed)... 2024-08-17T17:14:44.866 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:44 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-08-17T17:14:44.866 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:44 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:44.866 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:44 smithi053 ceph-mon[89963]: Reconfiguring daemon osd.6 on smithi053 2024-08-17T17:14:44.866 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:44 smithi053 ceph-mon[89963]: pgmap v66: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:45.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:44 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:45.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:44 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:45.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:44 smithi003 ceph-mon[103344]: Reconfiguring osd.6 (monmap changed)... 2024-08-17T17:14:45.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:44 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-08-17T17:14:45.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:44 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:45.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:44 smithi003 ceph-mon[103344]: Reconfiguring daemon osd.6 on smithi053 2024-08-17T17:14:45.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:44 smithi003 ceph-mon[103344]: pgmap v66: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:45.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:44 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:45.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:44 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:45.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:44 smithi003 ceph-mon[100615]: Reconfiguring osd.6 (monmap changed)... 2024-08-17T17:14:45.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:44 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-08-17T17:14:45.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:44 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:45.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:44 smithi003 ceph-mon[100615]: Reconfiguring daemon osd.6 on smithi053 2024-08-17T17:14:45.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:44 smithi003 ceph-mon[100615]: pgmap v66: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:45.949 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:14:45 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:14:45.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:14:45.949 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:14:45 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:14:45.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:14:45.966 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:14:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:14:45] "GET /metrics HTTP/1.1" 200 33816 "" "Prometheus/2.51.0" 2024-08-17T17:14:46.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:45 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:46.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:45 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:46.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:45 smithi053 ceph-mon[89963]: Reconfiguring osd.7 (monmap changed)... 2024-08-17T17:14:46.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:45 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-08-17T17:14:46.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:45 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:46.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:45 smithi053 ceph-mon[89963]: Reconfiguring daemon osd.7 on smithi053 2024-08-17T17:14:46.272 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:45 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:46.272 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:45 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:46.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:45 smithi003 ceph-mon[100615]: Reconfiguring osd.7 (monmap changed)... 2024-08-17T17:14:46.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:45 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-08-17T17:14:46.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:45 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:46.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:45 smithi003 ceph-mon[100615]: Reconfiguring daemon osd.7 on smithi053 2024-08-17T17:14:46.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:45 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:46.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:45 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:46.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:45 smithi003 ceph-mon[103344]: Reconfiguring osd.7 (monmap changed)... 2024-08-17T17:14:46.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:45 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-08-17T17:14:46.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:45 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:46.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:45 smithi003 ceph-mon[103344]: Reconfiguring daemon osd.7 on smithi053 2024-08-17T17:14:47.088 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:14:46 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:14:46.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi053\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:14:47.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:47 smithi053 ceph-mon[89963]: pgmap v67: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:47.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:47 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:47.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:47 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:47.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:47 smithi053 ceph-mon[89963]: Reconfiguring mds.foo.smithi053.ougbtc (monmap changed)... 2024-08-17T17:14:47.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:47 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi053.ougbtc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-17T17:14:47.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:47 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:47.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:47 smithi053 ceph-mon[89963]: Reconfiguring daemon mds.foo.smithi053.ougbtc on smithi053 2024-08-17T17:14:47.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:47 smithi003 ceph-mon[103344]: pgmap v67: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:47.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:47 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:47.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:47 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:47.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:47 smithi003 ceph-mon[103344]: Reconfiguring mds.foo.smithi053.ougbtc (monmap changed)... 2024-08-17T17:14:47.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:47 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi053.ougbtc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-17T17:14:47.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:47 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:47.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:47 smithi003 ceph-mon[103344]: Reconfiguring daemon mds.foo.smithi053.ougbtc on smithi053 2024-08-17T17:14:47.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:47 smithi003 ceph-mon[100615]: pgmap v67: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:47.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:47 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:47.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:47 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:47.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:47 smithi003 ceph-mon[100615]: Reconfiguring mds.foo.smithi053.ougbtc (monmap changed)... 2024-08-17T17:14:47.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:47 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi053.ougbtc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-08-17T17:14:47.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:47 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:47.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:47 smithi003 ceph-mon[100615]: Reconfiguring daemon mds.foo.smithi053.ougbtc on smithi053 2024-08-17T17:14:49.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:48 smithi053 ceph-mon[89963]: pgmap v68: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:49.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:48 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:49.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:48 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:49.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:48 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:14:49.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:48 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:49.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:48 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:14:49.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:48 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:49.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:48 smithi003 ceph-mon[103344]: pgmap v68: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:49.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:48 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:49.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:48 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:49.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:48 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:14:49.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:48 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:49.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:48 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:14:49.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:48 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:49.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:48 smithi003 ceph-mon[100615]: pgmap v68: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:49.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:48 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:49.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:48 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:49.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:48 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:14:49.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:48 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:14:49.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:48 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:14:49.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:48 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:14:50.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:14:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:14:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:14:51.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:50 smithi053 ceph-mon[89963]: pgmap v69: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:51.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:50 smithi003 ceph-mon[103344]: pgmap v69: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:51.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:50 smithi003 ceph-mon[100615]: pgmap v69: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:53.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:52 smithi053 ceph-mon[89963]: pgmap v70: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:53.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:52 smithi003 ceph-mon[103344]: pgmap v70: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:53.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:52 smithi003 ceph-mon[100615]: pgmap v70: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:54.147 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:14:54.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:14:54.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:14:54.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:14:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:14:54.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:14:55.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:54 smithi053 ceph-mon[89963]: pgmap v71: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:55.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:54 smithi003 ceph-mon[100615]: pgmap v71: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:55.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:54 smithi003 ceph-mon[103344]: pgmap v71: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:56.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:14:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:14:55] "GET /metrics HTTP/1.1" 200 33764 "" "Prometheus/2.51.0" 2024-08-17T17:14:56.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:14:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:14:55.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:14:56.217 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:14:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:14:55.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:14:57.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:14:56 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:14:56.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi053\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:14:57.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:56 smithi053 ceph-mon[89963]: pgmap v72: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:57.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:56 smithi003 ceph-mon[103344]: pgmap v72: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:57.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:56 smithi003 ceph-mon[100615]: pgmap v72: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:59.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:14:58 smithi053 ceph-mon[89963]: pgmap v73: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:59.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:14:58 smithi003 ceph-mon[100615]: pgmap v73: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:14:59.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:14:58 smithi003 ceph-mon[103344]: pgmap v73: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:00.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:15:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:15:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:15:01.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:00 smithi053 ceph-mon[89963]: pgmap v74: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:01.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:00 smithi003 ceph-mon[100615]: pgmap v74: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:01.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:00 smithi003 ceph-mon[103344]: pgmap v74: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:03.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:02 smithi053 ceph-mon[89963]: pgmap v75: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:03.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:02 smithi003 ceph-mon[100615]: pgmap v75: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:03.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:02 smithi003 ceph-mon[103344]: pgmap v75: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:04.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:15:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:15:04.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:15:05.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:04 smithi053 ceph-mon[89963]: pgmap v76: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:05.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:04 smithi003 ceph-mon[100615]: pgmap v76: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:05.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:04 smithi003 ceph-mon[103344]: pgmap v76: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:06.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:15:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:15:05] "GET /metrics HTTP/1.1" 200 33764 "" "Prometheus/2.51.0" 2024-08-17T17:15:06.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:15:05 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:15:05.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:15:06.217 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:15:05 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:15:05.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:15:07.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:06 smithi053 ceph-mon[89963]: pgmap v77: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:07.217 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:15:06 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:15:06.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi053\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:15:07.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:06 smithi003 ceph-mon[100615]: pgmap v77: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:07.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:06 smithi003 ceph-mon[103344]: pgmap v77: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:09.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:08 smithi053 ceph-mon[89963]: pgmap v78: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:09.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:15:09.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:08 smithi003 ceph-mon[100615]: pgmap v78: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:09.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:15:09.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:08 smithi003 ceph-mon[103344]: pgmap v78: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:09.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:15:10.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:15:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:15:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:15:11.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:10 smithi053 ceph-mon[89963]: pgmap v79: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:11.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:10 smithi003 ceph-mon[100615]: pgmap v79: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:11.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:10 smithi003 ceph-mon[103344]: pgmap v79: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:13.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:12 smithi053 ceph-mon[89963]: pgmap v80: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:13.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:12 smithi003 ceph-mon[100615]: pgmap v80: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:13.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:12 smithi003 ceph-mon[103344]: pgmap v80: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:14.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:15:14 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:15:14.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:15:15.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:14 smithi053 ceph-mon[89963]: pgmap v81: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:15.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:14 smithi003 ceph-mon[100615]: pgmap v81: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:15.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:14 smithi003 ceph-mon[103344]: pgmap v81: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:15.966 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:15:15 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:15:15.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:15:15.967 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:15:15 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:15:15.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:15:16.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:15:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:15:15] "GET /metrics HTTP/1.1" 200 33829 "" "Prometheus/2.51.0" 2024-08-17T17:15:17.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:16 smithi053 ceph-mon[89963]: pgmap v82: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:17.217 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:15:16 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:15:16.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi053\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:15:17.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:16 smithi003 ceph-mon[103344]: pgmap v82: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:17.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:16 smithi003 ceph-mon[100615]: pgmap v82: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:19.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:18 smithi053 ceph-mon[89963]: pgmap v83: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:19.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:18 smithi003 ceph-mon[100615]: pgmap v83: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:19.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:18 smithi003 ceph-mon[103344]: pgmap v83: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:20.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:15:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:15:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:15:21.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:20 smithi053 ceph-mon[89963]: pgmap v84: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:21.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:20 smithi003 ceph-mon[103344]: pgmap v84: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:21.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:20 smithi003 ceph-mon[100615]: pgmap v84: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:23.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:22 smithi053 ceph-mon[89963]: pgmap v85: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:23.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:22 smithi003 ceph-mon[100615]: pgmap v85: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:23.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:22 smithi003 ceph-mon[103344]: pgmap v85: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:24.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:15:24.217 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:15:24 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:15:24.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:15:24.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:15:24.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:15:25.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:24 smithi053 ceph-mon[89963]: pgmap v86: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:25.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:24 smithi003 ceph-mon[100615]: pgmap v86: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:25.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:24 smithi003 ceph-mon[103344]: pgmap v86: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:26.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:15:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:15:25] "GET /metrics HTTP/1.1" 200 33828 "" "Prometheus/2.51.0" 2024-08-17T17:15:26.217 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:15:25 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:15:25.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:15:26.217 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:15:25 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:15:25.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:15:26.869 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'ceph orch upgrade resume' 2024-08-17T17:15:27.150 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:15:27.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:26 smithi053 ceph-mon[89963]: pgmap v87: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:27.217 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:15:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:15:26.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi053\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:15:27.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:26 smithi003 ceph-mon[100615]: pgmap v87: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:27.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:26 smithi003 ceph-mon[103344]: pgmap v87: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:29.104 INFO:teuthology.orchestra.run.smithi003.stdout:Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:15:29.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:28 smithi053 ceph-mon[89963]: pgmap v88: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:29.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:28 smithi003 ceph-mon[103344]: pgmap v88: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:29.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:28 smithi003 ceph-mon[100615]: pgmap v88: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:29.819 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'echo wait for half of osds to upgrade' 2024-08-17T17:15:30.405 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:30 smithi003 ceph-mon[103344]: from='client.34296 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:30.405 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:30 smithi003 ceph-mon[103344]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:15:30.406 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:30 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:30.406 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:30 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:15:30.406 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:30 smithi003 ceph-mon[100615]: from='client.34296 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:30.406 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:30 smithi003 ceph-mon[100615]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:15:30.406 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:30 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:30.406 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:30 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:15:30.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:15:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:15:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:15:30.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:30 smithi053 ceph-mon[89963]: from='client.34296 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:30.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:30 smithi053 ceph-mon[89963]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:15:30.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:30 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:30.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:30 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:15:30.840 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:15:31.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:31 smithi003 ceph-mon[100615]: pgmap v89: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:31.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:31 smithi003 ceph-mon[103344]: pgmap v89: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:31.451 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:31 smithi053 ceph-mon[89963]: pgmap v89: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:33.104 INFO:teuthology.orchestra.run.smithi003.stdout:wait for half of osds to upgrade 2024-08-17T17:15:33.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:33 smithi053 ceph-mon[89963]: pgmap v90: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:33.511 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:33 smithi003 ceph-mon[103344]: pgmap v90: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:33.511 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:33 smithi003 ceph-mon[100615]: pgmap v90: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:33.920 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'while ! ceph osd versions | grep $sha1 | egrep '"'"': [45678]'"'"' && ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error; do sleep 2 ; done' 2024-08-17T17:15:34.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:34 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:34.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:34 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:34.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:15:34 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:15:34.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:15:34.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:34 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:34.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:34 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:34.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:34 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:34.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:34 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:35.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:35 smithi003 ceph-mon[100615]: pgmap v91: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:35.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:35 smithi003 ceph-mon[103344]: pgmap v91: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:35.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:35 smithi053 ceph-mon[89963]: pgmap v91: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:35.467 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:15:35.854 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:15:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:15:35] "GET /metrics HTTP/1.1" 200 33828 "" "Prometheus/2.51.0" 2024-08-17T17:15:36.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:15:35 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:15:35.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:15:36.217 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:15:35 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:15:35.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:15:36.600 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:36.600 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:36.600 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[100615]: pgmap v92: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:36.600 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:15:36.600 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:15:36.600 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:36.600 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:15:36.601 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:15:36.601 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:15:36.601 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[100615]: Upgrade: Setting container_image for all mon 2024-08-17T17:15:36.601 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:36.601 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-08-17T17:15:36.601 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-08-17T17:15:36.601 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-08-17T17:15:36.601 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-08-17T17:15:36.601 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-08-17T17:15:36.601 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-08-17T17:15:36.601 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:15:36.601 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[100615]: Upgrade: Setting container_image for all crash 2024-08-17T17:15:36.601 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:36.601 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-08-17T17:15:36.602 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[100615]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-08-17T17:15:36.602 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[100615]: Upgrade: osd.0 is safe to restart 2024-08-17T17:15:36.602 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:36.602 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:36.602 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[103344]: pgmap v92: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:36.602 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:15:36.602 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:15:36.602 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:36.602 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:15:36.602 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:15:36.602 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:15:36.603 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[103344]: Upgrade: Setting container_image for all mon 2024-08-17T17:15:36.603 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:36.603 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-08-17T17:15:36.603 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-08-17T17:15:36.603 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-08-17T17:15:36.603 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-08-17T17:15:36.603 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-08-17T17:15:36.603 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-08-17T17:15:36.603 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:15:36.603 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[103344]: Upgrade: Setting container_image for all crash 2024-08-17T17:15:36.603 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:36.603 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-08-17T17:15:36.603 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[103344]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-08-17T17:15:36.603 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:36 smithi003 ceph-mon[103344]: Upgrade: osd.0 is safe to restart 2024-08-17T17:15:36.948 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:36 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:36.949 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:36 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:36.949 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:36 smithi053 ceph-mon[89963]: pgmap v92: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:36.949 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:36 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:15:36.949 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:36 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:15:36.949 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:36 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:36.950 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:36 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:15:36.950 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:36 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:15:36.950 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:36 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:15:36.950 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:36 smithi053 ceph-mon[89963]: Upgrade: Setting container_image for all mon 2024-08-17T17:15:36.950 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:36 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:36.950 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:36 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-08-17T17:15:36.951 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:36 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-08-17T17:15:36.951 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:36 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-08-17T17:15:36.951 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:36 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-08-17T17:15:36.951 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:36 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-08-17T17:15:36.951 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:36 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-08-17T17:15:36.951 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:36 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:15:36.952 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:36 smithi053 ceph-mon[89963]: Upgrade: Setting container_image for all crash 2024-08-17T17:15:36.952 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:36 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:36.952 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:36 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-08-17T17:15:36.952 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:36 smithi053 ceph-mon[89963]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-08-17T17:15:36.952 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:36 smithi053 ceph-mon[89963]: Upgrade: osd.0 is safe to restart 2024-08-17T17:15:37.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:15:36 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:15:36.947Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi053\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:15:37.514 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:37 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1113142051' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:15:37.515 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:37 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1113142051' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:15:37.558 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:15:37.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:37 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1113142051' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:15:38.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:38 smithi003 ceph-mon[103344]: from='client.44168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:38.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:38 smithi003 ceph-mon[103344]: pgmap v93: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:38.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:38 smithi003 ceph-mon[103344]: from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:38.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:15:38.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:38 smithi003 ceph-mon[103344]: Upgrade: Updating osd.0 2024-08-17T17:15:38.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:38.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-08-17T17:15:38.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:15:38.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:38 smithi003 ceph-mon[103344]: Deploying daemon osd.0 on smithi003 2024-08-17T17:15:38.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:38 smithi003 ceph-mon[100615]: from='client.44168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:38.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:38 smithi003 ceph-mon[100615]: pgmap v93: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:38.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:38 smithi003 ceph-mon[100615]: from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:38.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:15:38.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:38 smithi003 ceph-mon[100615]: Upgrade: Updating osd.0 2024-08-17T17:15:38.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:38.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-08-17T17:15:38.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:15:38.775 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:38 smithi003 ceph-mon[100615]: Deploying daemon osd.0 on smithi003 2024-08-17T17:15:38.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:38 smithi053 ceph-mon[89963]: from='client.44168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:38.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:38 smithi053 ceph-mon[89963]: pgmap v93: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:38.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:38 smithi053 ceph-mon[89963]: from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:38.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:15:38.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:38 smithi053 ceph-mon[89963]: Upgrade: Updating osd.0 2024-08-17T17:15:38.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:38.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-08-17T17:15:38.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:15:38.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:38 smithi053 ceph-mon[89963]: Deploying daemon osd.0 on smithi003 2024-08-17T17:15:40.453 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:15:40.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:15:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:15:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:15:40.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:40 smithi053 ceph-mon[89963]: pgmap v94: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:40.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:40 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/4245585938' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:15:40.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:40 smithi053 ceph-mon[89963]: from='client.44186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:41.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:40 smithi003 ceph-mon[103344]: pgmap v94: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:41.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:40 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/4245585938' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:15:41.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:40 smithi003 ceph-mon[103344]: from='client.44186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:41.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:40 smithi003 ceph-mon[100615]: pgmap v94: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:41.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:40 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/4245585938' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:15:41.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:40 smithi003 ceph-mon[100615]: from='client.44186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:41.340 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:41 smithi003 systemd[1]: Stopping Ceph osd.0 for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:15:41.703 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:41 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0[45447]: 2024-08-17T17:15:41.338+0000 7f4bc8a8c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T17:15:41.703 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:41 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0[45447]: 2024-08-17T17:15:41.338+0000 7f4bc8a8c640 -1 osd.0 90 *** Got signal Terminated *** 2024-08-17T17:15:41.703 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:41 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0[45447]: 2024-08-17T17:15:41.338+0000 7f4bc8a8c640 -1 osd.0 90 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-17T17:15:41.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:41 smithi053 ceph-mon[89963]: from='client.34332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:41.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:41 smithi053 ceph-mon[89963]: osd.0 marked itself down and dead 2024-08-17T17:15:42.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:41 smithi003 ceph-mon[100615]: from='client.34332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:42.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:41 smithi003 ceph-mon[100615]: osd.0 marked itself down and dead 2024-08-17T17:15:42.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:41 smithi003 ceph-mon[103344]: from='client.34332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:42.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:41 smithi003 ceph-mon[103344]: osd.0 marked itself down and dead 2024-08-17T17:15:42.398 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:42 smithi003 podman[111244]: 2024-08-17 17:15:42.11625456 +0000 UTC m=+0.906142166 container died dc8e5d51fe1cdb18c761df7e5367d725a9161ac488f28cabc02472c82ef7c7d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0, org.label-schema.vendor=CentOS, RELEASE=quincy-f2c39d4, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:15:42.704 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:42 smithi003 podman[111244]: 2024-08-17 17:15:42.588781909 +0000 UTC m=+1.378669503 container cleanup dc8e5d51fe1cdb18c761df7e5367d725a9161ac488f28cabc02472c82ef7c7d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0, org.label-schema.vendor=CentOS, RELEASE=quincy-f2c39d4, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-08-17T17:15:42.704 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:42 smithi003 bash[111244]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0 2024-08-17T17:15:42.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:42 smithi053 ceph-mon[89963]: pgmap v95: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:42.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:42 smithi053 ceph-mon[89963]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T17:15:42.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:42 smithi053 ceph-mon[89963]: osdmap e91: 8 total, 7 up, 8 in 2024-08-17T17:15:43.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:42 smithi003 ceph-mon[103344]: pgmap v95: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:43.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:42 smithi003 ceph-mon[103344]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T17:15:43.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:42 smithi003 ceph-mon[103344]: osdmap e91: 8 total, 7 up, 8 in 2024-08-17T17:15:43.023 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:42 smithi003 podman[111259]: 2024-08-17 17:15:42.882358545 +0000 UTC m=+0.765157341 container remove dc8e5d51fe1cdb18c761df7e5367d725a9161ac488f28cabc02472c82ef7c7d8 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=quincy-f2c39d4, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:15:43.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:42 smithi003 ceph-mon[100615]: pgmap v95: 73 pgs: 73 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:43.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:42 smithi003 ceph-mon[100615]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T17:15:43.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:42 smithi003 ceph-mon[100615]: osdmap e91: 8 total, 7 up, 8 in 2024-08-17T17:15:43.371 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:15:43.523 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:43 smithi003 podman[111388]: 2024-08-17 17:15:43.155844718 +0000 UTC m=+0.023682709 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:15:43.523 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:43 smithi003 podman[111388]: 2024-08-17 17:15:43.266657047 +0000 UTC m=+0.134495033 container create 9197348683c98fc1e2fef569816e0633900b7d7558dcc26bbc8428eba2c6e758 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, RELEASE=main-cd8bdbe, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-17T17:15:43.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:43 smithi053 ceph-mon[89963]: osdmap e92: 8 total, 7 up, 8 in 2024-08-17T17:15:43.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:43 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/213764888' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:15:43.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:43 smithi053 ceph-mon[89963]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:44.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:43 smithi003 ceph-mon[100615]: osdmap e92: 8 total, 7 up, 8 in 2024-08-17T17:15:44.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:43 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/213764888' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:15:44.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:43 smithi003 ceph-mon[100615]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:44.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:43 smithi003 ceph-mon[103344]: osdmap e92: 8 total, 7 up, 8 in 2024-08-17T17:15:44.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:43 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/213764888' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:15:44.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:43 smithi003 ceph-mon[103344]: from='client.34344 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:44.024 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:43 smithi003 podman[111388]: 2024-08-17 17:15:43.536698699 +0000 UTC m=+0.404536688 container init 9197348683c98fc1e2fef569816e0633900b7d7558dcc26bbc8428eba2c6e758 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0-deactivate, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=main-cd8bdbe, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-08-17T17:15:44.024 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:43 smithi003 podman[111388]: 2024-08-17 17:15:43.540381149 +0000 UTC m=+0.408219134 container start 9197348683c98fc1e2fef569816e0633900b7d7558dcc26bbc8428eba2c6e758 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0-deactivate, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-cd8bdbe, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-08-17T17:15:44.024 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:43 smithi003 podman[111388]: 2024-08-17 17:15:43.594599036 +0000 UTC m=+0.462437015 container attach 9197348683c98fc1e2fef569816e0633900b7d7558dcc26bbc8428eba2c6e758 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=main-cd8bdbe, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-08-17T17:15:44.024 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:43 smithi003 podman[111388]: 2024-08-17 17:15:43.742171489 +0000 UTC m=+0.610009479 container died 9197348683c98fc1e2fef569816e0633900b7d7558dcc26bbc8428eba2c6e758 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-cd8bdbe, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-08-17T17:15:44.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:15:44 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:15:44.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:15:44.715 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:44 smithi003 podman[111446]: 2024-08-17 17:15:44.298136378 +0000 UTC m=+0.546251346 container remove 9197348683c98fc1e2fef569816e0633900b7d7558dcc26bbc8428eba2c6e758 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=main-cd8bdbe, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-17T17:15:44.715 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:44 smithi003 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.0.service: Deactivated successfully. 2024-08-17T17:15:44.715 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:44 smithi003 systemd[1]: Stopped Ceph osd.0 for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:15:44.715 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:44 smithi003 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.0.service: Consumed 6.058s CPU time. 2024-08-17T17:15:44.715 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:44 smithi003 systemd[1]: Starting Ceph osd.0 for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:15:44.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:44 smithi053 ceph-mon[89963]: pgmap v98: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-08-17T17:15:44.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:44 smithi053 ceph-mon[89963]: from='client.34350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:44.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:44 smithi053 ceph-mon[89963]: Health check failed: Degraded data redundancy: 13/84 objects degraded (15.476%), 8 pgs degraded (PG_DEGRADED) 2024-08-17T17:15:45.010 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:44 smithi003 ceph-mon[103344]: pgmap v98: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-08-17T17:15:45.011 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:44 smithi003 ceph-mon[103344]: from='client.34350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:45.011 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:44 smithi003 ceph-mon[103344]: Health check failed: Degraded data redundancy: 13/84 objects degraded (15.476%), 8 pgs degraded (PG_DEGRADED) 2024-08-17T17:15:45.011 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:44 smithi003 podman[111550]: 2024-08-17 17:15:44.658390168 +0000 UTC m=+0.017759934 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:15:45.011 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:44 smithi003 podman[111550]: 2024-08-17 17:15:44.796218208 +0000 UTC m=+0.155587964 container create ef5de078f32c7c6f8295734ffef1c39a46c95e213bf0c890aa3eafeb3e882e72 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0-activate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-cd8bdbe, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.36.0) 2024-08-17T17:15:45.011 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:44 smithi003 ceph-mon[100615]: pgmap v98: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-08-17T17:15:45.011 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:44 smithi003 ceph-mon[100615]: from='client.34350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:45.011 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:44 smithi003 ceph-mon[100615]: Health check failed: Degraded data redundancy: 13/84 objects degraded (15.476%), 8 pgs degraded (PG_DEGRADED) 2024-08-17T17:15:45.273 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:45 smithi003 podman[111550]: 2024-08-17 17:15:45.010501495 +0000 UTC m=+0.369871261 container init ef5de078f32c7c6f8295734ffef1c39a46c95e213bf0c890aa3eafeb3e882e72 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0-activate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=main-cd8bdbe, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.36.0) 2024-08-17T17:15:45.273 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:45 smithi003 podman[111550]: 2024-08-17 17:15:45.014294803 +0000 UTC m=+0.373664560 container start ef5de078f32c7c6f8295734ffef1c39a46c95e213bf0c890aa3eafeb3e882e72 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0-activate, RELEASE=main-cd8bdbe, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True) 2024-08-17T17:15:45.273 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:45 smithi003 podman[111550]: 2024-08-17 17:15:45.110332548 +0000 UTC m=+0.469702307 container attach ef5de078f32c7c6f8295734ffef1c39a46c95e213bf0c890aa3eafeb3e882e72 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0-activate, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-cd8bdbe, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-08-17T17:15:45.274 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0-activate[111564]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:15:45.274 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:45 smithi003 bash[111550]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:15:45.274 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0-activate[111564]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:15:45.274 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:45 smithi003 bash[111550]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:15:45.274 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0-activate[111564]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-17T17:15:45.274 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:45 smithi003 bash[111550]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-17T17:15:45.274 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0-activate[111564]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:15:45.274 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:45 smithi003 bash[111550]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:15:45.274 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0-activate[111564]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:15:45.274 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:45 smithi003 bash[111550]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:15:45.274 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0-activate[111564]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-08-17T17:15:45.274 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:45 smithi003 bash[111550]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-08-17T17:15:45.275 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0-activate[111564]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2024-08-17T17:15:45.275 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:45 smithi003 bash[111550]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_4 --path /var/lib/ceph/osd/ceph-0 --no-mon-config 2024-08-17T17:15:45.939 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:15:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:15:45] "GET /metrics HTTP/1.1" 200 33830 "" "Prometheus/2.51.0" 2024-08-17T17:15:45.940 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0-activate[111564]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2024-08-17T17:15:45.940 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:45 smithi003 bash[111550]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block 2024-08-17T17:15:45.940 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0-activate[111564]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2024-08-17T17:15:45.940 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:45 smithi003 bash[111550]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block 2024-08-17T17:15:45.940 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0-activate[111564]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-08-17T17:15:45.940 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:45 smithi003 bash[111550]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-08-17T17:15:45.940 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0-activate[111564]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-08-17T17:15:45.940 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:45 smithi003 bash[111550]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-08-17T17:15:45.940 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0-activate[111564]: --> ceph-volume lvm activate successful for osd ID: 0 2024-08-17T17:15:45.940 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:45 smithi003 bash[111550]: --> ceph-volume lvm activate successful for osd ID: 0 2024-08-17T17:15:45.940 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:45 smithi003 podman[111550]: 2024-08-17 17:15:45.575260406 +0000 UTC m=+0.934630162 container died ef5de078f32c7c6f8295734ffef1c39a46c95e213bf0c890aa3eafeb3e882e72 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0-activate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-cd8bdbe, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:15:46.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:15:45 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:15:45.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:15:46.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:15:45 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:15:45.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:15:46.256 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:15:46.522 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:46 smithi003 podman[111687]: 2024-08-17 17:15:46.373600453 +0000 UTC m=+0.791767546 container remove ef5de078f32c7c6f8295734ffef1c39a46c95e213bf0c890aa3eafeb3e882e72 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0-activate, ceph=True, org.label-schema.license=GPLv2, RELEASE=main-cd8bdbe, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-08-17T17:15:46.854 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:46 smithi003 ceph-mon[100615]: pgmap v99: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-08-17T17:15:46.854 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:46 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1732419484' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:15:46.854 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:46 smithi003 ceph-mon[100615]: from='client.44222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:46.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:46 smithi003 ceph-mon[103344]: pgmap v99: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-08-17T17:15:46.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:46 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1732419484' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:15:46.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:46 smithi003 ceph-mon[103344]: from='client.44222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:46.855 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:46 smithi003 podman[111820]: 2024-08-17 17:15:46.496680349 +0000 UTC m=+0.017891804 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:15:46.855 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:46 smithi003 podman[111820]: 2024-08-17 17:15:46.60795126 +0000 UTC m=+0.129162702 container create 5d56fa8e72d5142e94ee4b2fddf53066f1017279d67bf1c93d51061d14466054 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-cd8bdbe, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-08-17T17:15:46.855 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:46 smithi003 podman[111820]: 2024-08-17 17:15:46.827064971 +0000 UTC m=+0.348276422 container init 5d56fa8e72d5142e94ee4b2fddf53066f1017279d67bf1c93d51061d14466054 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-cd8bdbe, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-08-17T17:15:46.855 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:46 smithi003 podman[111820]: 2024-08-17 17:15:46.831261129 +0000 UTC m=+0.352472572 container start 5d56fa8e72d5142e94ee4b2fddf53066f1017279d67bf1c93d51061d14466054 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-cd8bdbe, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True) 2024-08-17T17:15:47.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:46 smithi053 ceph-mon[89963]: pgmap v99: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-08-17T17:15:47.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:46 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1732419484' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:15:47.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:46 smithi053 ceph-mon[89963]: from='client.44222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:47.217 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:15:46 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:15:46.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi053\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi053\", job=\"node\", machine=\"x86_64\", nodename=\"smithi053\", release=\"5.14.0-480.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:15:47.273 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:46 smithi003 ceph-osd[111843]: -- 172.21.15.3:0/466252648 <== mon.0 v2:172.21.15.3:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564efc00da40 con 0x564efcdda000 2024-08-17T17:15:47.273 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:46 smithi003 bash[111820]: 5d56fa8e72d5142e94ee4b2fddf53066f1017279d67bf1c93d51061d14466054 2024-08-17T17:15:47.273 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:46 smithi003 systemd[1]: Started Ceph osd.0 for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:15:48.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:47 smithi053 ceph-mon[89963]: from='client.34368 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:48.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:47 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:48.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:47 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:48.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:47 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:15:48.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:47 smithi003 ceph-mon[103344]: from='client.34368 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:48.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:47 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:48.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:47 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:48.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:47 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:15:48.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:47 smithi003 ceph-mon[100615]: from='client.34368 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:48.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:47 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:48.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:47 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:48.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:47 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:15:49.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:48 smithi003 ceph-mon[103344]: pgmap v100: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-08-17T17:15:49.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:48 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2835188823' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:15:49.023 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:48 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0[111839]: 2024-08-17T17:15:48.717+0000 7ff025825640 -1 Falling back to public interface 2024-08-17T17:15:49.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:48 smithi003 ceph-mon[100615]: pgmap v100: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-08-17T17:15:49.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:48 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2835188823' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:15:49.195 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:15:49.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:48 smithi053 ceph-mon[89963]: pgmap v100: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-08-17T17:15:49.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:48 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2835188823' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:15:50.214 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:49 smithi053 ceph-mon[89963]: from='client.34383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:50.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:49 smithi003 ceph-mon[103344]: from='client.34383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:50.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:49 smithi003 ceph-mon[100615]: from='client.34383 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:50.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:15:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:15:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:15:51.629 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:51 smithi003 ceph-mon[103344]: from='client.34389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:51.629 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:51 smithi003 ceph-mon[103344]: pgmap v101: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-08-17T17:15:51.629 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:51 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:51.629 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:51 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:51.630 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:51 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:51.630 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:51 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:51.630 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:51 smithi003 ceph-mon[100615]: from='client.34389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:51.630 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:51 smithi003 ceph-mon[100615]: pgmap v101: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-08-17T17:15:51.630 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:51 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:51.630 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:51 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:51.630 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:51 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:51.630 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:51 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:51.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:51 smithi053 ceph-mon[89963]: from='client.34389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:51.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:51 smithi053 ceph-mon[89963]: pgmap v101: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-08-17T17:15:51.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:51 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:51.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:51 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:51.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:51 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:51.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:51 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:52.100 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:15:52.272 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:51 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0[111839]: 2024-08-17T17:15:51.994+0000 7ff025825640 -1 osd.0 90 log_to_monitors true 2024-08-17T17:15:52.593 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:52 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2981341208' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:15:52.593 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:52 smithi003 ceph-mon[100615]: from='osd.0 [v2:172.21.15.3:6802/58175680,v1:172.21.15.3:6803/58175680]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-08-17T17:15:52.593 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:52 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2981341208' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:15:52.593 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:52 smithi003 ceph-mon[103344]: from='osd.0 [v2:172.21.15.3:6802/58175680,v1:172.21.15.3:6803/58175680]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-08-17T17:15:52.593 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:15:52 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0[111839]: 2024-08-17T17:15:52.360+0000 7ff01d5cf640 -1 osd.0 90 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-17T17:15:52.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:52 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2981341208' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:15:52.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:52 smithi053 ceph-mon[89963]: from='osd.0 [v2:172.21.15.3:6802/58175680,v1:172.21.15.3:6803/58175680]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-08-17T17:15:53.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:53 smithi053 ceph-mon[89963]: pgmap v102: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-08-17T17:15:53.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:53 smithi053 ceph-mon[89963]: from='client.34401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:53.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:53 smithi053 ceph-mon[89963]: from='osd.0 [v2:172.21.15.3:6802/58175680,v1:172.21.15.3:6803/58175680]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-08-17T17:15:53.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:53 smithi053 ceph-mon[89963]: osdmap e93: 8 total, 7 up, 8 in 2024-08-17T17:15:53.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:53 smithi053 ceph-mon[89963]: from='osd.0 [v2:172.21.15.3:6802/58175680,v1:172.21.15.3:6803/58175680]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-08-17T17:15:53.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:53 smithi053 ceph-mon[89963]: from='client.34410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:53.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:53.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:15:53.741 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:53 smithi003 ceph-mon[103344]: pgmap v102: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-08-17T17:15:53.741 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:53 smithi003 ceph-mon[103344]: from='client.34401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:53.741 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:53 smithi003 ceph-mon[103344]: from='osd.0 [v2:172.21.15.3:6802/58175680,v1:172.21.15.3:6803/58175680]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-08-17T17:15:53.741 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:53 smithi003 ceph-mon[103344]: osdmap e93: 8 total, 7 up, 8 in 2024-08-17T17:15:53.741 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:53 smithi003 ceph-mon[103344]: from='osd.0 [v2:172.21.15.3:6802/58175680,v1:172.21.15.3:6803/58175680]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-08-17T17:15:53.742 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:53 smithi003 ceph-mon[103344]: from='client.34410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:53.742 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:53.742 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:15:53.742 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:53 smithi003 ceph-mon[100615]: pgmap v102: 73 pgs: 18 active+undersized, 8 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13/84 objects degraded (15.476%) 2024-08-17T17:15:53.742 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:53 smithi003 ceph-mon[100615]: from='client.34401 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:53.742 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:53 smithi003 ceph-mon[100615]: from='osd.0 [v2:172.21.15.3:6802/58175680,v1:172.21.15.3:6803/58175680]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-08-17T17:15:53.742 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:53 smithi003 ceph-mon[100615]: osdmap e93: 8 total, 7 up, 8 in 2024-08-17T17:15:53.742 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:53 smithi003 ceph-mon[100615]: from='osd.0 [v2:172.21.15.3:6802/58175680,v1:172.21.15.3:6803/58175680]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-08-17T17:15:53.742 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:53 smithi003 ceph-mon[100615]: from='client.34410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:53.742 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:53.742 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:15:54.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:15:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:15:54.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)\", cluster_addr=\"172.21.15.3\", device_class=\"ssd\", hostname=\"smithi003\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.3\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:15:54.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:54 smithi053 ceph-mon[89963]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T17:15:54.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:54 smithi053 ceph-mon[89963]: osd.0 [v2:172.21.15.3:6802/58175680,v1:172.21.15.3:6803/58175680] boot 2024-08-17T17:15:54.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:54 smithi053 ceph-mon[89963]: osdmap e94: 8 total, 8 up, 8 in 2024-08-17T17:15:54.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:54 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-17T17:15:54.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:54 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:54.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:54 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:54.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:54 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:15:54.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:54 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:15:54.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:54 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:54.523 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:15:54 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:15:54.398+0000 7f288dc05640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-08-17T17:15:54.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:54 smithi003 ceph-mon[100615]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T17:15:54.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:54 smithi003 ceph-mon[100615]: osd.0 [v2:172.21.15.3:6802/58175680,v1:172.21.15.3:6803/58175680] boot 2024-08-17T17:15:54.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:54 smithi003 ceph-mon[100615]: osdmap e94: 8 total, 8 up, 8 in 2024-08-17T17:15:54.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:54 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-17T17:15:54.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:54 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:54.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:54 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:54.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:54 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:15:54.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:54 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:15:54.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:54 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:54.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:54 smithi003 ceph-mon[103344]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T17:15:54.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:54 smithi003 ceph-mon[103344]: osd.0 [v2:172.21.15.3:6802/58175680,v1:172.21.15.3:6803/58175680] boot 2024-08-17T17:15:54.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:54 smithi003 ceph-mon[103344]: osdmap e94: 8 total, 8 up, 8 in 2024-08-17T17:15:54.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:54 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-17T17:15:54.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:54 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:54.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:54 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:54.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:54 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:15:54.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:54 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:15:54.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:54 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:15:54.993 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:15:55.651 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:55 smithi003 ceph-mon[103344]: pgmap v105: 73 pgs: 15 peering, 9 active+undersized, 2 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3/84 objects degraded (3.571%) 2024-08-17T17:15:55.651 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:55 smithi003 ceph-mon[103344]: osdmap e95: 8 total, 8 up, 8 in 2024-08-17T17:15:55.651 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:55 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 3/84 objects degraded (3.571%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:15:55.651 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:55 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:15:55.651 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:55 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:15:55.651 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:55 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:15:55.651 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:55 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:15:55.651 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:55 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-08-17T17:15:55.651 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:55 smithi003 ceph-mon[103344]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-08-17T17:15:55.651 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:55 smithi003 ceph-mon[103344]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-08-17T17:15:55.651 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:55 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/4244054743' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:15:55.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:55 smithi003 ceph-mon[100615]: pgmap v105: 73 pgs: 15 peering, 9 active+undersized, 2 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3/84 objects degraded (3.571%) 2024-08-17T17:15:55.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:55 smithi003 ceph-mon[100615]: osdmap e95: 8 total, 8 up, 8 in 2024-08-17T17:15:55.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:55 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 3/84 objects degraded (3.571%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:15:55.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:55 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:15:55.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:55 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:15:55.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:55 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:15:55.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:55 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:15:55.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:55 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-08-17T17:15:55.653 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:55 smithi003 ceph-mon[100615]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-08-17T17:15:55.653 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:55 smithi003 ceph-mon[100615]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-08-17T17:15:55.653 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:55 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/4244054743' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:15:55.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:55 smithi053 ceph-mon[89963]: pgmap v105: 73 pgs: 15 peering, 9 active+undersized, 2 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3/84 objects degraded (3.571%) 2024-08-17T17:15:55.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:55 smithi053 ceph-mon[89963]: osdmap e95: 8 total, 8 up, 8 in 2024-08-17T17:15:55.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:55 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 3/84 objects degraded (3.571%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:15:55.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:55 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:15:55.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:55 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:15:55.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:55 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:15:55.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:55 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:15:55.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:55 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-08-17T17:15:55.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:55 smithi053 ceph-mon[89963]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-08-17T17:15:55.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:55 smithi053 ceph-mon[89963]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-08-17T17:15:55.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:55 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/4244054743' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:15:56.022 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:15:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:15:55] "GET /metrics HTTP/1.1" 200 33740 "" "Prometheus/2.51.0" 2024-08-17T17:15:56.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:15:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:15:55.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:15:56.216 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:15:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:15:55.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:15:56.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:56 smithi053 ceph-mon[89963]: from='client.34422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:56.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:56 smithi053 ceph-mon[89963]: from='client.44288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:56.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:56 smithi003 ceph-mon[103344]: from='client.34422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:56.899 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:56 smithi003 ceph-mon[103344]: from='client.44288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:56.899 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:56 smithi003 ceph-mon[100615]: from='client.34422 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:56.899 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:56 smithi003 ceph-mon[100615]: from='client.44288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:57.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:57 smithi053 ceph-mon[89963]: pgmap v107: 73 pgs: 15 peering, 9 active+undersized, 2 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3/84 objects degraded (3.571%) 2024-08-17T17:15:57.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:57 smithi003 ceph-mon[100615]: pgmap v107: 73 pgs: 15 peering, 9 active+undersized, 2 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3/84 objects degraded (3.571%) 2024-08-17T17:15:57.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:57 smithi003 ceph-mon[103344]: pgmap v107: 73 pgs: 15 peering, 9 active+undersized, 2 active+undersized+degraded, 47 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3/84 objects degraded (3.571%) 2024-08-17T17:15:57.905 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:15:58.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:58 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3828594480' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:15:58.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:58 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3828594480' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:15:58.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:58 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3828594480' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:15:59.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:59 smithi053 ceph-mon[89963]: pgmap v108: 73 pgs: 15 peering, 58 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:59.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:59 smithi053 ceph-mon[89963]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:59.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:59 smithi053 ceph-mon[89963]: from='client.44306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:59.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:59 smithi053 ceph-mon[89963]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/84 objects degraded (3.571%), 2 pgs degraded) 2024-08-17T17:15:59.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:15:59 smithi053 ceph-mon[89963]: Cluster is now healthy 2024-08-17T17:15:59.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:59 smithi003 ceph-mon[100615]: pgmap v108: 73 pgs: 15 peering, 58 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:59.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:59 smithi003 ceph-mon[100615]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:59.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:59 smithi003 ceph-mon[100615]: from='client.44306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:59.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:59 smithi003 ceph-mon[100615]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/84 objects degraded (3.571%), 2 pgs degraded) 2024-08-17T17:15:59.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:15:59 smithi003 ceph-mon[100615]: Cluster is now healthy 2024-08-17T17:15:59.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:59 smithi003 ceph-mon[103344]: pgmap v108: 73 pgs: 15 peering, 58 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:15:59.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:59 smithi003 ceph-mon[103344]: from='client.34440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:59.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:59 smithi003 ceph-mon[103344]: from='client.44306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:15:59.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:59 smithi003 ceph-mon[103344]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 3/84 objects degraded (3.571%), 2 pgs degraded) 2024-08-17T17:15:59.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:15:59 smithi003 ceph-mon[103344]: Cluster is now healthy 2024-08-17T17:16:00.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:16:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:16:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:16:00.790 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:16:01.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:01 smithi053 ceph-mon[89963]: pgmap v109: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:01.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:01 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/10454268' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:01.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:01 smithi003 ceph-mon[103344]: pgmap v109: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:01.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:01 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/10454268' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:01.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:01 smithi003 ceph-mon[100615]: pgmap v109: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:01.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:01 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/10454268' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:02.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:02 smithi053 ceph-mon[89963]: from='client.44318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:02.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:02 smithi053 ceph-mon[89963]: from='client.34464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:02.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:02 smithi003 ceph-mon[103344]: from='client.44318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:02.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:02 smithi003 ceph-mon[103344]: from='client.34464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:02.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:02 smithi003 ceph-mon[100615]: from='client.44318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:02.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:02 smithi003 ceph-mon[100615]: from='client.34464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:03.631 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:03 smithi053 ceph-mon[89963]: pgmap v110: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:03.631 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:03 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3494686139' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:03.680 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:16:03.681 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:03 smithi003 ceph-mon[103344]: pgmap v110: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:03.681 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:03 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3494686139' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:03.681 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:03 smithi003 ceph-mon[100615]: pgmap v110: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:03.681 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:03 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3494686139' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:05.651 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:05 smithi003 ceph-mon[103344]: pgmap v111: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:05.651 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:05 smithi003 ceph-mon[103344]: from='client.44336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:05.651 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:05 smithi003 ceph-mon[103344]: from='client.34482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:05.651 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:05 smithi003 ceph-mon[100615]: pgmap v111: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:05.651 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:05 smithi003 ceph-mon[100615]: from='client.44336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:05.651 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:05 smithi003 ceph-mon[100615]: from='client.34482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:05.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:05 smithi053 ceph-mon[89963]: pgmap v111: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:05.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:05 smithi053 ceph-mon[89963]: from='client.44336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:05.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:05 smithi053 ceph-mon[89963]: from='client.34482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:06.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:16:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:16:05] "GET /metrics HTTP/1.1" 200 33740 "" "Prometheus/2.51.0" 2024-08-17T17:16:06.575 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:16:06.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:06 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3872099308' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:06.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:06 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3872099308' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:06.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:06 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3872099308' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:07.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:07 smithi053 ceph-mon[89963]: pgmap v112: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:07.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:07 smithi003 ceph-mon[100615]: pgmap v112: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:07.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:07 smithi003 ceph-mon[103344]: pgmap v112: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:08.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:08 smithi053 ceph-mon[89963]: from='client.44354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:08.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:08 smithi053 ceph-mon[89963]: from='client.44360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:08.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:08.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:16:08.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:08.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:08 smithi003 ceph-mon[103344]: from='client.44354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:08.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:08 smithi003 ceph-mon[103344]: from='client.44360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:08.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:08.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:16:08.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:08.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:08 smithi003 ceph-mon[100615]: from='client.44354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:08.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:08 smithi003 ceph-mon[100615]: from='client.44360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:08.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:08.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:16:08.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:09.466 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:16:09.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:09 smithi053 ceph-mon[89963]: pgmap v113: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:09.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:09 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3361627675' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:09.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:09 smithi003 ceph-mon[100615]: pgmap v113: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:09.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:09 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3361627675' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:09.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:09 smithi003 ceph-mon[103344]: pgmap v113: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:09.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:09 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3361627675' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:10.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:10 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-08-17T17:16:10.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:10 smithi053 ceph-mon[89963]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-08-17T17:16:10.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:10 smithi053 ceph-mon[89963]: Upgrade: osd.1 is safe to restart 2024-08-17T17:16:10.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:10 smithi053 ceph-mon[89963]: from='client.44372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:10.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:16:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:16:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:16:10.700 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:10 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-08-17T17:16:10.700 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:10 smithi003 ceph-mon[103344]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-08-17T17:16:10.700 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:10 smithi003 ceph-mon[103344]: Upgrade: osd.1 is safe to restart 2024-08-17T17:16:10.700 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:10 smithi003 ceph-mon[103344]: from='client.44372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:10.700 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:10 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-08-17T17:16:10.701 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:10 smithi003 ceph-mon[100615]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-08-17T17:16:10.701 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:10 smithi003 ceph-mon[100615]: Upgrade: osd.1 is safe to restart 2024-08-17T17:16:10.701 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:10 smithi003 ceph-mon[100615]: from='client.44372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:11.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:11 smithi053 ceph-mon[89963]: pgmap v114: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:11.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:11 smithi053 ceph-mon[89963]: from='client.34518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:11.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:11 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:11.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:11 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-08-17T17:16:11.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:11 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:16:11.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:11 smithi003 ceph-mon[100615]: pgmap v114: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:11.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:11 smithi003 ceph-mon[100615]: from='client.34518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:11.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:11 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:11.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:11 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-08-17T17:16:11.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:11 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:16:11.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:11 smithi003 ceph-mon[103344]: pgmap v114: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:11.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:11 smithi003 ceph-mon[103344]: from='client.34518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:11.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:11 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:11.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:11 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-08-17T17:16:11.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:11 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:16:12.364 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:16:12.487 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:12 smithi003 ceph-mon[103344]: Upgrade: Updating osd.1 2024-08-17T17:16:12.487 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:12 smithi003 ceph-mon[103344]: Deploying daemon osd.1 on smithi003 2024-08-17T17:16:12.487 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:12 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1172397397' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:12.488 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:12 smithi003 ceph-mon[100615]: Upgrade: Updating osd.1 2024-08-17T17:16:12.488 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:12 smithi003 ceph-mon[100615]: Deploying daemon osd.1 on smithi003 2024-08-17T17:16:12.488 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:12 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1172397397' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:12.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:12 smithi053 ceph-mon[89963]: Upgrade: Updating osd.1 2024-08-17T17:16:12.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:12 smithi053 ceph-mon[89963]: Deploying daemon osd.1 on smithi003 2024-08-17T17:16:12.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:12 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1172397397' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:13.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:13 smithi053 ceph-mon[89963]: pgmap v115: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:13.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:13 smithi053 ceph-mon[89963]: from='client.34530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:13.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:13 smithi003 ceph-mon[103344]: pgmap v115: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:13.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:13 smithi003 ceph-mon[103344]: from='client.34530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:13.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:13 smithi003 ceph-mon[100615]: pgmap v115: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:13.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:13 smithi003 ceph-mon[100615]: from='client.34530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:13.756 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:13 smithi003 systemd[1]: Stopping Ceph osd.1 for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:16:14.023 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:13 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1[51195]: 2024-08-17T17:16:13.761+0000 7f0f5e006640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T17:16:14.023 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:13 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1[51195]: 2024-08-17T17:16:13.761+0000 7f0f5e006640 -1 osd.1 95 *** Got signal Terminated *** 2024-08-17T17:16:14.023 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:13 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1[51195]: 2024-08-17T17:16:13.761+0000 7f0f5e006640 -1 osd.1 95 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-17T17:16:14.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:14 smithi053 ceph-mon[89963]: from='client.44396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:14.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:14 smithi053 ceph-mon[89963]: osd.1 marked itself down and dead 2024-08-17T17:16:14.737 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:14 smithi003 ceph-mon[100615]: from='client.44396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:14.737 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:14 smithi003 ceph-mon[100615]: osd.1 marked itself down and dead 2024-08-17T17:16:14.737 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:14 smithi003 ceph-mon[103344]: from='client.44396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:14.737 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:14 smithi003 ceph-mon[103344]: osd.1 marked itself down and dead 2024-08-17T17:16:15.023 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:14 smithi003 podman[116528]: 2024-08-17 17:16:14.890972611 +0000 UTC m=+1.247104001 container died 7def3d9ed99f5ff88fc00c6ce93a900113c66fbfe5d5758e0416ae614c86e624 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, RELEASE=quincy-f2c39d4, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-17T17:16:15.271 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:16:15.590 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:15 smithi003 ceph-mon[100615]: pgmap v116: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:15.590 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:15 smithi003 ceph-mon[100615]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T17:16:15.590 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:15 smithi003 ceph-mon[100615]: osdmap e96: 8 total, 7 up, 8 in 2024-08-17T17:16:15.590 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:15 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1868376401' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:15.590 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:15 smithi003 ceph-mon[103344]: pgmap v116: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:15.590 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:15 smithi003 ceph-mon[103344]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T17:16:15.591 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:15 smithi003 ceph-mon[103344]: osdmap e96: 8 total, 7 up, 8 in 2024-08-17T17:16:15.591 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:15 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1868376401' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:15.591 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:15 smithi003 podman[116528]: 2024-08-17 17:16:15.286725903 +0000 UTC m=+1.642857284 container cleanup 7def3d9ed99f5ff88fc00c6ce93a900113c66fbfe5d5758e0416ae614c86e624 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1, org.label-schema.vendor=CentOS, RELEASE=quincy-f2c39d4, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2) 2024-08-17T17:16:15.591 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:15 smithi003 bash[116528]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1 2024-08-17T17:16:15.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:15 smithi053 ceph-mon[89963]: pgmap v116: 73 pgs: 73 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:15.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:15 smithi053 ceph-mon[89963]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T17:16:15.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:15 smithi053 ceph-mon[89963]: osdmap e96: 8 total, 7 up, 8 in 2024-08-17T17:16:15.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:15 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1868376401' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:15.907 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:15 smithi003 podman[116577]: 2024-08-17 17:16:15.588902103 +0000 UTC m=+0.699991701 container remove 7def3d9ed99f5ff88fc00c6ce93a900113c66fbfe5d5758e0416ae614c86e624 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1, RELEASE=quincy-f2c39d4, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-08-17T17:16:15.907 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:16:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:16:15] "GET /metrics HTTP/1.1" 200 33946 "" "Prometheus/2.51.0" 2024-08-17T17:16:16.176 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:15 smithi003 podman[116714]: 2024-08-17 17:16:15.906575421 +0000 UTC m=+0.106957739 container create ad712d99483d5e5e8df03dfd527021c33dbc8561b5f43da003cbb31ba8e6816b (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=main-cd8bdbe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0) 2024-08-17T17:16:16.176 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:15 smithi003 podman[116714]: 2024-08-17 17:16:15.817663337 +0000 UTC m=+0.018045627 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:16:16.176 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:16 smithi003 podman[116714]: 2024-08-17 17:16:16.100659688 +0000 UTC m=+0.301041974 container init ad712d99483d5e5e8df03dfd527021c33dbc8561b5f43da003cbb31ba8e6816b (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, RELEASE=main-cd8bdbe, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-08-17T17:16:16.176 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:16 smithi003 podman[116714]: 2024-08-17 17:16:16.104207486 +0000 UTC m=+0.304589777 container start ad712d99483d5e5e8df03dfd527021c33dbc8561b5f43da003cbb31ba8e6816b (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, RELEASE=main-cd8bdbe, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-08-17T17:16:16.176 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:16 smithi003 podman[116714]: 2024-08-17 17:16:16.175657664 +0000 UTC m=+0.376039949 container attach ad712d99483d5e5e8df03dfd527021c33dbc8561b5f43da003cbb31ba8e6816b (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, RELEASE=main-cd8bdbe, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:16:16.428 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:16 smithi003 podman[116714]: 2024-08-17 17:16:16.301608667 +0000 UTC m=+0.501990963 container died ad712d99483d5e5e8df03dfd527021c33dbc8561b5f43da003cbb31ba8e6816b (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-deactivate, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-cd8bdbe, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:16:16.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:16 smithi053 ceph-mon[89963]: from='client.44408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:16.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:16 smithi053 ceph-mon[89963]: osdmap e97: 8 total, 7 up, 8 in 2024-08-17T17:16:16.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:16 smithi003 ceph-mon[100615]: from='client.44408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:16.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:16 smithi003 ceph-mon[100615]: osdmap e97: 8 total, 7 up, 8 in 2024-08-17T17:16:16.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:16 smithi003 ceph-mon[103344]: from='client.44408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:16.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:16 smithi003 ceph-mon[103344]: osdmap e97: 8 total, 7 up, 8 in 2024-08-17T17:16:17.273 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:16 smithi003 podman[116736]: 2024-08-17 17:16:16.846344117 +0000 UTC m=+0.538460366 container remove ad712d99483d5e5e8df03dfd527021c33dbc8561b5f43da003cbb31ba8e6816b (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-deactivate, io.buildah.version=1.36.0, GIT_CLEAN=True, RELEASE=main-cd8bdbe, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-08-17T17:16:17.274 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:16 smithi003 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.1.service: Deactivated successfully. 2024-08-17T17:16:17.274 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:16 smithi003 systemd[1]: Stopped Ceph osd.1 for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:16:17.274 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:16 smithi003 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.1.service: Consumed 6.861s CPU time. 2024-08-17T17:16:17.274 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:17 smithi003 systemd[1]: Starting Ceph osd.1 for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:16:17.625 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:17 smithi003 ceph-mon[100615]: from='client.34554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:17.626 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:17 smithi003 ceph-mon[100615]: pgmap v119: 73 pgs: 10 stale+active+clean, 63 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:17.626 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:17 smithi003 ceph-mon[103344]: from='client.34554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:17.626 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:17 smithi003 ceph-mon[103344]: pgmap v119: 73 pgs: 10 stale+active+clean, 63 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:17.627 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:17 smithi003 podman[116836]: 2024-08-17 17:16:17.20169909 +0000 UTC m=+0.017694328 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:16:17.627 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:17 smithi003 podman[116836]: 2024-08-17 17:16:17.3223668 +0000 UTC m=+0.138362026 container create b7de754966a57046db35582067fca4034222338b4346dc6fb3dbc0727978ba4f (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate, RELEASE=main-cd8bdbe, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-08-17T17:16:17.627 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:17 smithi003 podman[116836]: 2024-08-17 17:16:17.533495084 +0000 UTC m=+0.349490308 container init b7de754966a57046db35582067fca4034222338b4346dc6fb3dbc0727978ba4f (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-cd8bdbe) 2024-08-17T17:16:17.627 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:17 smithi003 podman[116836]: 2024-08-17 17:16:17.537766592 +0000 UTC m=+0.353761824 container start b7de754966a57046db35582067fca4034222338b4346dc6fb3dbc0727978ba4f (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=main-cd8bdbe, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-17T17:16:17.627 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:17 smithi003 podman[116836]: 2024-08-17 17:16:17.62511838 +0000 UTC m=+0.441113622 container attach b7de754966a57046db35582067fca4034222338b4346dc6fb3dbc0727978ba4f (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-cd8bdbe, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-08-17T17:16:17.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:17 smithi053 ceph-mon[89963]: from='client.34554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:17.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:17 smithi053 ceph-mon[89963]: pgmap v119: 73 pgs: 10 stale+active+clean, 63 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:18.023 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:17 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate[116850]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:16:18.023 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:17 smithi003 bash[116836]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:16:18.023 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:17 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate[116850]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:16:18.023 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:17 smithi003 bash[116836]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:16:18.023 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:17 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate[116850]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-17T17:16:18.023 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:17 smithi003 bash[116836]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-17T17:16:18.024 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:17 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate[116850]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:16:18.024 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:17 smithi003 bash[116836]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:16:18.024 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:17 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate[116850]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:16:18.024 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:17 smithi003 bash[116836]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:16:18.024 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:17 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate[116850]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-08-17T17:16:18.024 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:17 smithi003 bash[116836]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-08-17T17:16:18.024 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:17 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate[116850]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2024-08-17T17:16:18.024 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:17 smithi003 bash[116836]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2024-08-17T17:16:18.200 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:16:18.421 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:18 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate[116850]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2024-08-17T17:16:18.422 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:18 smithi003 bash[116836]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2024-08-17T17:16:18.422 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:18 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate[116850]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2024-08-17T17:16:18.422 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:18 smithi003 bash[116836]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2024-08-17T17:16:18.422 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:18 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate[116850]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-08-17T17:16:18.422 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:18 smithi003 bash[116836]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-08-17T17:16:18.422 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:18 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate[116850]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-08-17T17:16:18.422 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:18 smithi003 bash[116836]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-08-17T17:16:18.422 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:18 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate[116850]: --> ceph-volume lvm activate successful for osd ID: 1 2024-08-17T17:16:18.422 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:18 smithi003 bash[116836]: --> ceph-volume lvm activate successful for osd ID: 1 2024-08-17T17:16:18.422 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:18 smithi003 podman[116836]: 2024-08-17 17:16:18.107957691 +0000 UTC m=+0.923952927 container died b7de754966a57046db35582067fca4034222338b4346dc6fb3dbc0727978ba4f (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, RELEASE=main-cd8bdbe, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-08-17T17:16:18.688 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:18 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/835395378' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:18.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:18 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/835395378' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:18.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:18 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/835395378' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:19.006 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:18 smithi003 podman[117012]: 2024-08-17 17:16:18.688156509 +0000 UTC m=+0.570059463 container remove b7de754966a57046db35582067fca4034222338b4346dc6fb3dbc0727978ba4f (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate, RELEASE=main-cd8bdbe, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-08-17T17:16:19.006 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:18 smithi003 podman[117101]: 2024-08-17 17:16:18.807540248 +0000 UTC m=+0.017871907 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:16:19.006 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:18 smithi003 podman[117101]: 2024-08-17 17:16:18.905419242 +0000 UTC m=+0.115750896 container create f83a0fcf5c4c6ff21a36865486567ac416fc67b9ae9ae2ce72cdf52eebd9cfc3 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-cd8bdbe, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_BRANCH=HEAD) 2024-08-17T17:16:19.273 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:19 smithi003 podman[117101]: 2024-08-17 17:16:19.124884557 +0000 UTC m=+0.335216219 container init f83a0fcf5c4c6ff21a36865486567ac416fc67b9ae9ae2ce72cdf52eebd9cfc3 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-cd8bdbe, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.build-date=20240731) 2024-08-17T17:16:19.273 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:19 smithi003 podman[117101]: 2024-08-17 17:16:19.128204074 +0000 UTC m=+0.338535722 container start f83a0fcf5c4c6ff21a36865486567ac416fc67b9ae9ae2ce72cdf52eebd9cfc3 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1, ceph=True, CEPH_POINT_RELEASE=, RELEASE=main-cd8bdbe, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0) 2024-08-17T17:16:19.273 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:19 smithi003 ceph-osd[117121]: -- 172.21.15.3:0/3291536514 <== mon.2 v2:172.21.15.3:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558cc32d1a40 con 0x558cc409e000 2024-08-17T17:16:19.273 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:19 smithi003 bash[117101]: f83a0fcf5c4c6ff21a36865486567ac416fc67b9ae9ae2ce72cdf52eebd9cfc3 2024-08-17T17:16:19.273 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:19 smithi003 systemd[1]: Started Ceph osd.1 for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:16:19.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:19 smithi053 ceph-mon[89963]: pgmap v120: 73 pgs: 1 active+undersized, 9 stale+active+clean, 2 active+undersized+degraded, 61 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2/84 objects degraded (2.381%) 2024-08-17T17:16:19.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:19 smithi053 ceph-mon[89963]: from='client.34566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:19.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:19 smithi053 ceph-mon[89963]: Health check failed: Degraded data redundancy: 2/84 objects degraded (2.381%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:16:19.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:19 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:19.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:19 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:19.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:19 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:16:19.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:19 smithi003 ceph-mon[100615]: pgmap v120: 73 pgs: 1 active+undersized, 9 stale+active+clean, 2 active+undersized+degraded, 61 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2/84 objects degraded (2.381%) 2024-08-17T17:16:19.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:19 smithi003 ceph-mon[100615]: from='client.34566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:19.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:19 smithi003 ceph-mon[100615]: Health check failed: Degraded data redundancy: 2/84 objects degraded (2.381%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:16:19.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:19 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:19.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:19 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:19.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:19 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:16:19.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:19 smithi003 ceph-mon[103344]: pgmap v120: 73 pgs: 1 active+undersized, 9 stale+active+clean, 2 active+undersized+degraded, 61 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2/84 objects degraded (2.381%) 2024-08-17T17:16:19.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:19 smithi003 ceph-mon[103344]: from='client.34566 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:19.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:19 smithi003 ceph-mon[103344]: Health check failed: Degraded data redundancy: 2/84 objects degraded (2.381%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:16:19.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:19 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:19.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:19 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:19.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:19 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:16:20.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:20 smithi053 ceph-mon[89963]: from='client.44432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:20.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:16:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:16:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:16:20.767 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:20 smithi003 ceph-mon[103344]: from='client.44432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:20.768 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:20 smithi003 ceph-mon[100615]: from='client.44432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:21.023 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:20 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1[117117]: 2024-08-17T17:16:20.766+0000 7f22fecf2640 -1 Falling back to public interface 2024-08-17T17:16:21.108 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:16:21.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:21 smithi053 ceph-mon[89963]: pgmap v121: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-08-17T17:16:21.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:21 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2643068527' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:21.753 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:21 smithi003 ceph-mon[103344]: pgmap v121: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-08-17T17:16:21.753 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:21 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2643068527' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:21.753 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:21 smithi003 ceph-mon[100615]: pgmap v121: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-08-17T17:16:21.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:21 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2643068527' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:22.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:22 smithi053 ceph-mon[89963]: from='client.34587 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:22.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:22 smithi053 ceph-mon[89963]: from='client.34593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:22.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:22 smithi003 ceph-mon[100615]: from='client.34587 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:22.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:22 smithi003 ceph-mon[100615]: from='client.34593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:22.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:22 smithi003 ceph-mon[103344]: from='client.34587 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:22.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:22 smithi003 ceph-mon[103344]: from='client.34593 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:23 smithi053 ceph-mon[89963]: pgmap v122: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-08-17T17:16:23.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:23.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:23.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:23.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:16:23.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:23.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:23.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:23 smithi003 ceph-mon[103344]: pgmap v122: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-08-17T17:16:23.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:23.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:23.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:23.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:16:23.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:23.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:23.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:23 smithi003 ceph-mon[100615]: pgmap v122: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-08-17T17:16:23.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:23.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:23.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:23.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:16:23.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:23.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:24.030 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:16:24.471 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:24 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1[117117]: 2024-08-17T17:16:24.311+0000 7f22fecf2640 -1 osd.1 95 log_to_monitors true 2024-08-17T17:16:24.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:24 smithi003 ceph-mon[100615]: pgmap v123: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-08-17T17:16:24.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:24 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/443775406' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:24.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:24 smithi003 ceph-mon[100615]: from='client.34605 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:24.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:24 smithi003 ceph-mon[100615]: from='osd.1 [v2:172.21.15.3:6810/1956079582,v1:172.21.15.3:6811/1956079582]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-08-17T17:16:24.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:24 smithi003 ceph-mon[100615]: from='client.44471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:24.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:24 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 5/84 objects degraded (5.952%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T17:16:24.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:24 smithi003 ceph-mon[103344]: pgmap v123: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-08-17T17:16:24.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:24 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/443775406' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:24.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:24 smithi003 ceph-mon[103344]: from='client.34605 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:24.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:24 smithi003 ceph-mon[103344]: from='osd.1 [v2:172.21.15.3:6810/1956079582,v1:172.21.15.3:6811/1956079582]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-08-17T17:16:24.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:24 smithi003 ceph-mon[103344]: from='client.44471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:24.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:24 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 5/84 objects degraded (5.952%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T17:16:24.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:24 smithi053 ceph-mon[89963]: pgmap v123: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-08-17T17:16:24.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:24 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/443775406' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:24.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:24 smithi053 ceph-mon[89963]: from='client.34605 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:24.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:24 smithi053 ceph-mon[89963]: from='osd.1 [v2:172.21.15.3:6810/1956079582,v1:172.21.15.3:6811/1956079582]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-08-17T17:16:24.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:24 smithi053 ceph-mon[89963]: from='client.44471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:24.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:24 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 5/84 objects degraded (5.952%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T17:16:25.178 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:16:24 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1[117117]: 2024-08-17T17:16:24.950+0000 7f22f2a94640 -1 osd.1 95 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-17T17:16:25.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:16:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:16:25] "GET /metrics HTTP/1.1" 200 33957 "" "Prometheus/2.51.0" 2024-08-17T17:16:25.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:25 smithi003 ceph-mon[100615]: from='osd.1 [v2:172.21.15.3:6810/1956079582,v1:172.21.15.3:6811/1956079582]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-17T17:16:25.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:25 smithi003 ceph-mon[100615]: osdmap e98: 8 total, 7 up, 8 in 2024-08-17T17:16:25.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:25 smithi003 ceph-mon[100615]: from='osd.1 [v2:172.21.15.3:6810/1956079582,v1:172.21.15.3:6811/1956079582]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-08-17T17:16:25.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:25 smithi003 ceph-mon[100615]: from='osd.1 [v2:172.21.15.3:6810/1956079582,v1:172.21.15.3:6811/1956079582]' entity='osd.1' 2024-08-17T17:16:25.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:25 smithi003 ceph-mon[103344]: from='osd.1 [v2:172.21.15.3:6810/1956079582,v1:172.21.15.3:6811/1956079582]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-17T17:16:25.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:25 smithi003 ceph-mon[103344]: osdmap e98: 8 total, 7 up, 8 in 2024-08-17T17:16:25.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:25 smithi003 ceph-mon[103344]: from='osd.1 [v2:172.21.15.3:6810/1956079582,v1:172.21.15.3:6811/1956079582]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-08-17T17:16:25.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:25 smithi003 ceph-mon[103344]: from='osd.1 [v2:172.21.15.3:6810/1956079582,v1:172.21.15.3:6811/1956079582]' entity='osd.1' 2024-08-17T17:16:25.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:25 smithi053 ceph-mon[89963]: from='osd.1 [v2:172.21.15.3:6810/1956079582,v1:172.21.15.3:6811/1956079582]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-17T17:16:25.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:25 smithi053 ceph-mon[89963]: osdmap e98: 8 total, 7 up, 8 in 2024-08-17T17:16:25.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:25 smithi053 ceph-mon[89963]: from='osd.1 [v2:172.21.15.3:6810/1956079582,v1:172.21.15.3:6811/1956079582]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-08-17T17:16:25.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:25 smithi053 ceph-mon[89963]: from='osd.1 [v2:172.21.15.3:6810/1956079582,v1:172.21.15.3:6811/1956079582]' entity='osd.1' 2024-08-17T17:16:26.741 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:26 smithi003 ceph-mon[103344]: pgmap v125: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-08-17T17:16:26.742 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:26 smithi003 ceph-mon[103344]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T17:16:26.742 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:26 smithi003 ceph-mon[103344]: osd.1 [v2:172.21.15.3:6810/1956079582,v1:172.21.15.3:6811/1956079582] boot 2024-08-17T17:16:26.742 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:26 smithi003 ceph-mon[103344]: osdmap e99: 8 total, 8 up, 8 in 2024-08-17T17:16:26.742 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:26 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-17T17:16:26.742 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:26 smithi003 ceph-mon[100615]: pgmap v125: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-08-17T17:16:26.743 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:26 smithi003 ceph-mon[100615]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T17:16:26.743 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:26 smithi003 ceph-mon[100615]: osd.1 [v2:172.21.15.3:6810/1956079582,v1:172.21.15.3:6811/1956079582] boot 2024-08-17T17:16:26.743 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:26 smithi003 ceph-mon[100615]: osdmap e99: 8 total, 8 up, 8 in 2024-08-17T17:16:26.743 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:26 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-17T17:16:26.926 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:16:26.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:26 smithi053 ceph-mon[89963]: pgmap v125: 73 pgs: 27 active+undersized, 5 active+undersized+degraded, 41 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5/84 objects degraded (5.952%) 2024-08-17T17:16:26.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:26 smithi053 ceph-mon[89963]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T17:16:26.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:26 smithi053 ceph-mon[89963]: osd.1 [v2:172.21.15.3:6810/1956079582,v1:172.21.15.3:6811/1956079582] boot 2024-08-17T17:16:26.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:26 smithi053 ceph-mon[89963]: osdmap e99: 8 total, 8 up, 8 in 2024-08-17T17:16:26.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:26 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-17T17:16:27.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:16:26 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:16:26.906+0000 7f288dc05640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-08-17T17:16:27.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3057023386' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:27.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:27.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:27.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:16:27.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:16:27.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:27.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:16:27.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:27.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:27.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:27.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-08-17T17:16:27.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[100615]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-08-17T17:16:27.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[100615]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-08-17T17:16:27.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[100615]: from='client.34626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:27.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[100615]: osdmap e100: 8 total, 8 up, 8 in 2024-08-17T17:16:27.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[100615]: from='client.44492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:27.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3057023386' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:27.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:27.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:27.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:16:27.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:16:27.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:27.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:16:27.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:27.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:27.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:27.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-08-17T17:16:27.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[103344]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-08-17T17:16:27.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[103344]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-08-17T17:16:27.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[103344]: from='client.34626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:27.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[103344]: osdmap e100: 8 total, 8 up, 8 in 2024-08-17T17:16:27.776 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:27 smithi003 ceph-mon[103344]: from='client.44492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:27.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:27 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3057023386' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:27.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:27 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:27.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:27 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:27.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:27 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:16:27.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:27 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:16:27.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:27 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:27.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:27 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:16:27.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:27 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:27.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:27 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:27.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:27 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:16:27.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:27 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-08-17T17:16:27.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:27 smithi053 ceph-mon[89963]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-08-17T17:16:27.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:27 smithi053 ceph-mon[89963]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-08-17T17:16:27.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:27 smithi053 ceph-mon[89963]: from='client.34626 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:27.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:27 smithi053 ceph-mon[89963]: osdmap e100: 8 total, 8 up, 8 in 2024-08-17T17:16:27.968 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:27 smithi053 ceph-mon[89963]: from='client.44492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:29.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:29 smithi053 ceph-mon[89963]: pgmap v128: 73 pgs: 26 active+undersized, 4 active+undersized+degraded, 43 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-08-17T17:16:29.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:29 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:29.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:29 smithi003 ceph-mon[103344]: pgmap v128: 73 pgs: 26 active+undersized, 4 active+undersized+degraded, 43 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-08-17T17:16:29.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:29 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:29.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:29 smithi003 ceph-mon[100615]: pgmap v128: 73 pgs: 26 active+undersized, 4 active+undersized+degraded, 43 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-08-17T17:16:29.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:29 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:29.827 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:16:30.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:30 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 4/84 objects degraded (4.762%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T17:16:30.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:30 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3773100933' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:30.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:16:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:16:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:16:30.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:30 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 4/84 objects degraded (4.762%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T17:16:30.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:30 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3773100933' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:30.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:30 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 4/84 objects degraded (4.762%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T17:16:30.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:30 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3773100933' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:31.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:31 smithi053 ceph-mon[89963]: pgmap v129: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:31.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:31 smithi053 ceph-mon[89963]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:31.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:31 smithi053 ceph-mon[89963]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/84 objects degraded (4.762%), 4 pgs degraded) 2024-08-17T17:16:31.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:31 smithi053 ceph-mon[89963]: Cluster is now healthy 2024-08-17T17:16:31.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:31 smithi053 ceph-mon[89963]: from='client.34650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:31.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:31 smithi003 ceph-mon[103344]: pgmap v129: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:31.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:31 smithi003 ceph-mon[103344]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:31.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:31 smithi003 ceph-mon[103344]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/84 objects degraded (4.762%), 4 pgs degraded) 2024-08-17T17:16:31.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:31 smithi003 ceph-mon[103344]: Cluster is now healthy 2024-08-17T17:16:31.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:31 smithi003 ceph-mon[103344]: from='client.34650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:31.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:31 smithi003 ceph-mon[100615]: pgmap v129: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:31.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:31 smithi003 ceph-mon[100615]: from='client.34644 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:31.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:31 smithi003 ceph-mon[100615]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/84 objects degraded (4.762%), 4 pgs degraded) 2024-08-17T17:16:31.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:31 smithi003 ceph-mon[100615]: Cluster is now healthy 2024-08-17T17:16:31.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:31 smithi003 ceph-mon[100615]: from='client.34650 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:32.719 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:16:33.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:33 smithi053 ceph-mon[89963]: pgmap v130: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:33.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:33 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2076648078' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:33.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:33 smithi003 ceph-mon[100615]: pgmap v130: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:33.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:33 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2076648078' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:33.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:33 smithi003 ceph-mon[103344]: pgmap v130: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:33.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:33 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2076648078' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:34.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:34 smithi053 ceph-mon[89963]: from='client.44522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:34.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:34 smithi053 ceph-mon[89963]: from='client.34668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:34.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:34 smithi003 ceph-mon[100615]: from='client.44522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:34.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:34 smithi003 ceph-mon[100615]: from='client.34668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:34.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:34 smithi003 ceph-mon[103344]: from='client.44522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:34.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:34 smithi003 ceph-mon[103344]: from='client.34668 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:35.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:35 smithi053 ceph-mon[89963]: pgmap v131: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:35.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:35 smithi003 ceph-mon[100615]: pgmap v131: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:35.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:35 smithi003 ceph-mon[103344]: pgmap v131: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:35.607 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:16:36.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:16:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:16:35] "GET /metrics HTTP/1.1" 200 33957 "" "Prometheus/2.51.0" 2024-08-17T17:16:36.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:36 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3719383290' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:36.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:36 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3719383290' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:36.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:36 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3719383290' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:37.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:37 smithi053 ceph-mon[89963]: from='client.44540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:37.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:37 smithi053 ceph-mon[89963]: pgmap v132: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:37.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:37 smithi053 ceph-mon[89963]: from='client.34686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:37.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:37 smithi003 ceph-mon[100615]: from='client.44540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:37.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:37 smithi003 ceph-mon[100615]: pgmap v132: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:37.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:37 smithi003 ceph-mon[100615]: from='client.34686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:37.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:37 smithi003 ceph-mon[103344]: from='client.44540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:37.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:37 smithi003 ceph-mon[103344]: pgmap v132: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:37.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:37 smithi003 ceph-mon[103344]: from='client.34686 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:38.497 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:16:39.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:39 smithi053 ceph-mon[89963]: pgmap v133: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:39.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:39.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:16:39.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:39.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:39 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/607392736' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:39.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:39 smithi003 ceph-mon[100615]: pgmap v133: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:39.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:39.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:16:39.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:39.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:39 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/607392736' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:39.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:39 smithi003 ceph-mon[103344]: pgmap v133: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:39.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:39.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:16:39.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:39.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:39 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/607392736' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:40.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:16:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:16:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:16:40.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:40 smithi053 ceph-mon[89963]: from='client.34698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:40.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:40 smithi053 ceph-mon[89963]: from='client.44564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:40.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:40 smithi003 ceph-mon[100615]: from='client.34698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:40.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:40 smithi003 ceph-mon[100615]: from='client.44564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:40.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:40 smithi003 ceph-mon[103344]: from='client.34698 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:40.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:40 smithi003 ceph-mon[103344]: from='client.44564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:41.384 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:16:41.384 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:41 smithi003 ceph-mon[103344]: pgmap v134: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:41.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:41 smithi003 ceph-mon[100615]: pgmap v134: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:41.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:41 smithi053 ceph-mon[89963]: pgmap v134: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:42.342 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:42 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3294903547' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:42.342 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:42 smithi003 ceph-mon[103344]: from='client.34716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:42.342 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:42 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-08-17T17:16:42.342 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:42 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3294903547' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:42.342 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:42 smithi003 ceph-mon[100615]: from='client.34716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:42.342 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:42 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-08-17T17:16:42.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:42 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3294903547' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:42.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:42 smithi053 ceph-mon[89963]: from='client.34716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:42.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:42 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-08-17T17:16:43.346 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:43 smithi003 ceph-mon[100615]: pgmap v135: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:43.346 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:43 smithi003 ceph-mon[100615]: from='client.34722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:43.346 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:43 smithi003 ceph-mon[100615]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-08-17T17:16:43.346 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:43 smithi003 ceph-mon[100615]: Upgrade: osd.2 is safe to restart 2024-08-17T17:16:43.346 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:43 smithi003 ceph-mon[103344]: pgmap v135: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:43.346 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:43 smithi003 ceph-mon[103344]: from='client.34722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:43.347 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:43 smithi003 ceph-mon[103344]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-08-17T17:16:43.347 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:43 smithi003 ceph-mon[103344]: Upgrade: osd.2 is safe to restart 2024-08-17T17:16:43.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:43 smithi053 ceph-mon[89963]: pgmap v135: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:43.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:43 smithi053 ceph-mon[89963]: from='client.34722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:43.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:43 smithi053 ceph-mon[89963]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-08-17T17:16:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:43 smithi053 ceph-mon[89963]: Upgrade: osd.2 is safe to restart 2024-08-17T17:16:44.270 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:16:44.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:44 smithi053 ceph-mon[89963]: Upgrade: Updating osd.2 2024-08-17T17:16:44.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:44 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:44.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:44 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-08-17T17:16:44.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:44 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:16:44.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:44 smithi053 ceph-mon[89963]: Deploying daemon osd.2 on smithi003 2024-08-17T17:16:44.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:44 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1828183391' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:44.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:44 smithi003 ceph-mon[100615]: Upgrade: Updating osd.2 2024-08-17T17:16:44.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:44 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:44.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:44 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-08-17T17:16:44.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:44 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:16:44.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:44 smithi003 ceph-mon[100615]: Deploying daemon osd.2 on smithi003 2024-08-17T17:16:44.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:44 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1828183391' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:44.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:44 smithi003 ceph-mon[103344]: Upgrade: Updating osd.2 2024-08-17T17:16:44.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:44 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:44.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:44 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-08-17T17:16:44.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:44 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:16:44.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:44 smithi003 ceph-mon[103344]: Deploying daemon osd.2 on smithi003 2024-08-17T17:16:44.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:44 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1828183391' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:45.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:45 smithi053 ceph-mon[89963]: pgmap v136: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:45.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:45 smithi053 ceph-mon[89963]: from='client.34734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:45.748 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:45 smithi003 ceph-mon[103344]: pgmap v136: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:45.748 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:45 smithi003 ceph-mon[103344]: from='client.34734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:45.749 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:45 smithi003 ceph-mon[100615]: pgmap v136: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:45.749 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:45 smithi003 ceph-mon[100615]: from='client.34734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:45.749 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:16:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:16:45] "GET /metrics HTTP/1.1" 200 33938 "" "Prometheus/2.51.0" 2024-08-17T17:16:46.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:46 smithi053 ceph-mon[89963]: from='client.44600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:46.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:46 smithi003 ceph-mon[103344]: from='client.44600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:46.773 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:46 smithi003 systemd[1]: Stopping Ceph osd.2 for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:16:46.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:46 smithi003 ceph-mon[100615]: from='client.44600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:47.187 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:46 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2[57066]: 2024-08-17T17:16:46.794+0000 7f446ea74640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T17:16:47.187 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:46 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2[57066]: 2024-08-17T17:16:46.794+0000 7f446ea74640 -1 osd.2 100 *** Got signal Terminated *** 2024-08-17T17:16:47.187 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:46 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2[57066]: 2024-08-17T17:16:46.794+0000 7f446ea74640 -1 osd.2 100 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-17T17:16:47.188 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:16:47.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:47 smithi053 ceph-mon[89963]: pgmap v137: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:47.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:47 smithi053 ceph-mon[89963]: osd.2 marked itself down and dead 2024-08-17T17:16:47.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:47 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1474278345' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:47.768 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:47 smithi003 ceph-mon[100615]: pgmap v137: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:47.769 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:47 smithi003 ceph-mon[100615]: osd.2 marked itself down and dead 2024-08-17T17:16:47.769 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:47 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1474278345' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:47.769 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:47 smithi003 ceph-mon[103344]: pgmap v137: 73 pgs: 73 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:47.769 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:47 smithi003 ceph-mon[103344]: osd.2 marked itself down and dead 2024-08-17T17:16:47.769 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:47 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1474278345' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:48.023 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:47 smithi003 podman[122249]: 2024-08-17 17:16:47.882486379 +0000 UTC m=+1.222600669 container died 3d65e6f7d660d4b2bf2e46eb744dfa1d09b9198f3c0510fd0442325596e862f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2, RELEASE=quincy-f2c39d4, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-08-17T17:16:48.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:48 smithi003 ceph-mon[103344]: from='client.34752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:48.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:48 smithi003 ceph-mon[103344]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T17:16:48.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:48 smithi003 ceph-mon[103344]: osdmap e101: 8 total, 7 up, 8 in 2024-08-17T17:16:48.523 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:48 smithi003 podman[122249]: 2024-08-17 17:16:48.377659004 +0000 UTC m=+1.717773289 container cleanup 3d65e6f7d660d4b2bf2e46eb744dfa1d09b9198f3c0510fd0442325596e862f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=quincy-f2c39d4, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:16:48.523 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:48 smithi003 bash[122249]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2 2024-08-17T17:16:48.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:48 smithi003 ceph-mon[100615]: from='client.34752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:48.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:48 smithi003 ceph-mon[100615]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T17:16:48.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:48 smithi003 ceph-mon[100615]: osdmap e101: 8 total, 7 up, 8 in 2024-08-17T17:16:48.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:48 smithi053 ceph-mon[89963]: from='client.34752 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:48.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:48 smithi053 ceph-mon[89963]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T17:16:48.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:48 smithi053 ceph-mon[89963]: osdmap e101: 8 total, 7 up, 8 in 2024-08-17T17:16:48.955 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:48 smithi003 podman[122330]: 2024-08-17 17:16:48.612724049 +0000 UTC m=+0.730815850 container remove 3d65e6f7d660d4b2bf2e46eb744dfa1d09b9198f3c0510fd0442325596e862f4 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=quincy-f2c39d4, ceph=True, org.label-schema.build-date=20240731) 2024-08-17T17:16:49.273 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:48 smithi003 podman[122415]: 2024-08-17 17:16:48.955393701 +0000 UTC m=+0.112636867 container create 0d9fb550be5abafe1fb44835c188621072b5a5c43d34d016bd5f8f59e863818f (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=main-cd8bdbe, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True) 2024-08-17T17:16:49.273 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:48 smithi003 podman[122415]: 2024-08-17 17:16:48.860376974 +0000 UTC m=+0.017620163 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:16:49.273 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:49 smithi003 podman[122415]: 2024-08-17 17:16:49.199793288 +0000 UTC m=+0.357036468 container init 0d9fb550be5abafe1fb44835c188621072b5a5c43d34d016bd5f8f59e863818f (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=main-cd8bdbe, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-17T17:16:49.273 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:49 smithi003 podman[122415]: 2024-08-17 17:16:49.203678932 +0000 UTC m=+0.360922107 container start 0d9fb550be5abafe1fb44835c188621072b5a5c43d34d016bd5f8f59e863818f (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, RELEASE=main-cd8bdbe, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.36.0) 2024-08-17T17:16:49.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:49 smithi053 ceph-mon[89963]: from='client.44618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:49.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:49 smithi053 ceph-mon[89963]: pgmap v139: 73 pgs: 2 stale+active+clean, 71 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:49.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:49 smithi053 ceph-mon[89963]: osdmap e102: 8 total, 7 up, 8 in 2024-08-17T17:16:49.745 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:49 smithi003 ceph-mon[103344]: from='client.44618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:49.745 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:49 smithi003 ceph-mon[103344]: pgmap v139: 73 pgs: 2 stale+active+clean, 71 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:49.745 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:49 smithi003 ceph-mon[103344]: osdmap e102: 8 total, 7 up, 8 in 2024-08-17T17:16:49.745 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:49 smithi003 podman[122415]: 2024-08-17 17:16:49.291479874 +0000 UTC m=+0.448723047 container attach 0d9fb550be5abafe1fb44835c188621072b5a5c43d34d016bd5f8f59e863818f (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=main-cd8bdbe, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS) 2024-08-17T17:16:49.745 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:49 smithi003 podman[122415]: 2024-08-17 17:16:49.406077626 +0000 UTC m=+0.563320796 container died 0d9fb550be5abafe1fb44835c188621072b5a5c43d34d016bd5f8f59e863818f (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-cd8bdbe, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-08-17T17:16:49.745 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:49 smithi003 ceph-mon[100615]: from='client.44618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:49.746 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:49 smithi003 ceph-mon[100615]: pgmap v139: 73 pgs: 2 stale+active+clean, 71 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-08-17T17:16:49.746 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:49 smithi003 ceph-mon[100615]: osdmap e102: 8 total, 7 up, 8 in 2024-08-17T17:16:50.075 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:16:50.460 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:50 smithi003 podman[122438]: 2024-08-17 17:16:50.037148054 +0000 UTC m=+0.620194925 container remove 0d9fb550be5abafe1fb44835c188621072b5a5c43d34d016bd5f8f59e863818f (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=main-cd8bdbe, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-17T17:16:50.460 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:50 smithi003 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.2.service: Deactivated successfully. 2024-08-17T17:16:50.460 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:50 smithi003 systemd[1]: Stopped Ceph osd.2 for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:16:50.460 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:50 smithi003 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.2.service: Consumed 5.853s CPU time. 2024-08-17T17:16:50.460 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:50 smithi003 systemd[1]: Starting Ceph osd.2 for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:16:50.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:50 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1628071570' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:50.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:16:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:16:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:16:50.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:50 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1628071570' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:50.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:50 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1628071570' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:50.773 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:50 smithi003 podman[122608]: 2024-08-17 17:16:50.399240463 +0000 UTC m=+0.018393718 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:16:50.774 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:50 smithi003 podman[122608]: 2024-08-17 17:16:50.513413848 +0000 UTC m=+0.132567132 container create d18bb3d740e19494cf5ec8f4e04eed3af2d2097afbb8064c5c2b703a586696ef (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2-activate, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, RELEASE=main-cd8bdbe, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-08-17T17:16:51.032 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:50 smithi003 podman[122608]: 2024-08-17 17:16:50.774098368 +0000 UTC m=+0.393251612 container init d18bb3d740e19494cf5ec8f4e04eed3af2d2097afbb8064c5c2b703a586696ef (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2-activate, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-cd8bdbe, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-08-17T17:16:51.032 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:50 smithi003 podman[122608]: 2024-08-17 17:16:50.777584873 +0000 UTC m=+0.396738129 container start d18bb3d740e19494cf5ec8f4e04eed3af2d2097afbb8064c5c2b703a586696ef (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2-activate, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-cd8bdbe, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-08-17T17:16:51.032 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:50 smithi003 podman[122608]: 2024-08-17 17:16:50.874559335 +0000 UTC m=+0.493712601 container attach d18bb3d740e19494cf5ec8f4e04eed3af2d2097afbb8064c5c2b703a586696ef (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2-activate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=main-cd8bdbe) 2024-08-17T17:16:51.032 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:50 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2-activate[122626]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:16:51.032 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:50 smithi003 bash[122608]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:16:51.032 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:50 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2-activate[122626]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:16:51.032 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:50 smithi003 bash[122608]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:16:51.032 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:50 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2-activate[122626]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-17T17:16:51.032 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:50 smithi003 bash[122608]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-17T17:16:51.033 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:50 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2-activate[122626]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:16:51.033 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:50 smithi003 bash[122608]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:16:51.033 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:50 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2-activate[122626]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:16:51.033 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:50 smithi003 bash[122608]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:16:51.033 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:51 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2-activate[122626]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-08-17T17:16:51.033 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:51 smithi003 bash[122608]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-08-17T17:16:51.317 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:51 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2-activate[122626]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2024-08-17T17:16:51.317 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:51 smithi003 bash[122608]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2024-08-17T17:16:51.317 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:51 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2-activate[122626]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2/block 2024-08-17T17:16:51.317 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:51 smithi003 bash[122608]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2/block 2024-08-17T17:16:51.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:51 smithi053 ceph-mon[89963]: pgmap v141: 73 pgs: 14 active+undersized, 3 active+undersized+degraded, 56 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-08-17T17:16:51.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:51 smithi053 ceph-mon[89963]: from='client.34770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:51.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:51 smithi053 ceph-mon[89963]: from='client.34776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:51.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:51 smithi053 ceph-mon[89963]: Health check failed: Degraded data redundancy: 4/84 objects degraded (4.762%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:16:51.746 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:51 smithi003 ceph-mon[100615]: pgmap v141: 73 pgs: 14 active+undersized, 3 active+undersized+degraded, 56 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-08-17T17:16:51.746 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:51 smithi003 ceph-mon[100615]: from='client.34770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:51.746 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:51 smithi003 ceph-mon[100615]: from='client.34776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:51.746 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:51 smithi003 ceph-mon[100615]: Health check failed: Degraded data redundancy: 4/84 objects degraded (4.762%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:16:51.746 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:51 smithi003 ceph-mon[103344]: pgmap v141: 73 pgs: 14 active+undersized, 3 active+undersized+degraded, 56 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-08-17T17:16:51.750 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:51 smithi003 ceph-mon[103344]: from='client.34770 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:51.750 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:51 smithi003 ceph-mon[103344]: from='client.34776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:51.750 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:51 smithi003 ceph-mon[103344]: Health check failed: Degraded data redundancy: 4/84 objects degraded (4.762%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:16:51.751 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:51 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2-activate[122626]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2024-08-17T17:16:51.751 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:51 smithi003 bash[122608]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2024-08-17T17:16:51.751 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:51 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2-activate[122626]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-08-17T17:16:51.751 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:51 smithi003 bash[122608]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-08-17T17:16:51.751 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:51 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2-activate[122626]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-08-17T17:16:51.751 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:51 smithi003 bash[122608]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-08-17T17:16:51.751 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:51 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2-activate[122626]: --> ceph-volume lvm activate successful for osd ID: 2 2024-08-17T17:16:51.751 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:51 smithi003 bash[122608]: --> ceph-volume lvm activate successful for osd ID: 2 2024-08-17T17:16:51.751 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:51 smithi003 podman[122608]: 2024-08-17 17:16:51.34972586 +0000 UTC m=+0.968879112 container died d18bb3d740e19494cf5ec8f4e04eed3af2d2097afbb8064c5c2b703a586696ef (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2-activate, RELEASE=main-cd8bdbe, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-17T17:16:52.422 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:52 smithi003 podman[122733]: 2024-08-17 17:16:52.053497906 +0000 UTC m=+0.692316537 container remove d18bb3d740e19494cf5ec8f4e04eed3af2d2097afbb8064c5c2b703a586696ef (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-cd8bdbe, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_CLEAN=True) 2024-08-17T17:16:52.422 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:52 smithi003 podman[122792]: 2024-08-17 17:16:52.179569635 +0000 UTC m=+0.018058229 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:16:52.422 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:52 smithi003 podman[122792]: 2024-08-17 17:16:52.287871154 +0000 UTC m=+0.126359793 container create b30ff78c142eeb8dfb1d7a4aff6bed2d937e2ebe7b7be60565e1a3cf9dba5af6 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=main-cd8bdbe, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-08-17T17:16:52.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:52 smithi003 ceph-mon[103344]: pgmap v142: 73 pgs: 14 active+undersized, 3 active+undersized+degraded, 56 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-08-17T17:16:52.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:52 smithi003 ceph-mon[100615]: pgmap v142: 73 pgs: 14 active+undersized, 3 active+undersized+degraded, 56 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-08-17T17:16:52.773 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:52 smithi003 podman[122792]: 2024-08-17 17:16:52.548849392 +0000 UTC m=+0.387337976 container init b30ff78c142eeb8dfb1d7a4aff6bed2d937e2ebe7b7be60565e1a3cf9dba5af6 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=main-cd8bdbe, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.36.0) 2024-08-17T17:16:52.773 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:52 smithi003 podman[122792]: 2024-08-17 17:16:52.552314852 +0000 UTC m=+0.390803437 container start b30ff78c142eeb8dfb1d7a4aff6bed2d937e2ebe7b7be60565e1a3cf9dba5af6 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, RELEASE=main-cd8bdbe, org.label-schema.vendor=CentOS) 2024-08-17T17:16:52.774 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:52 smithi003 ceph-osd[122833]: -- 172.21.15.3:0/214549469 <== mon.0 v2:172.21.15.3:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561caf507a40 con 0x561caf6aa000 2024-08-17T17:16:52.774 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:52 smithi003 bash[122792]: b30ff78c142eeb8dfb1d7a4aff6bed2d937e2ebe7b7be60565e1a3cf9dba5af6 2024-08-17T17:16:52.774 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:52 smithi003 systemd[1]: Started Ceph osd.2 for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:16:52.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:52 smithi053 ceph-mon[89963]: pgmap v142: 73 pgs: 14 active+undersized, 3 active+undersized+degraded, 56 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-08-17T17:16:53.021 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:16:53.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:53 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3422306597' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:53.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:53.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:53.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:16:53.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:53 smithi053 ceph-mon[89963]: from='client.34791 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:53.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:53.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:16:53.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:53 smithi053 ceph-mon[89963]: from='client.34797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:53 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3422306597' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:16:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:53 smithi003 ceph-mon[100615]: from='client.34791 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:16:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:53 smithi003 ceph-mon[100615]: from='client.34797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:53.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:53 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3422306597' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:53.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:53.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:53.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:16:53.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:53 smithi003 ceph-mon[103344]: from='client.34791 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:53.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:53.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:16:53.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:53 smithi003 ceph-mon[103344]: from='client.34797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:54.464 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:54 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2[122810]: 2024-08-17T17:16:54.195+0000 7fd2f30c4640 -1 Falling back to public interface 2024-08-17T17:16:54.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:54 smithi003 ceph-mon[100615]: pgmap v143: 73 pgs: 15 active+undersized, 4 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-08-17T17:16:54.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:54 smithi003 ceph-mon[103344]: pgmap v143: 73 pgs: 15 active+undersized, 4 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-08-17T17:16:54.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:54 smithi053 ceph-mon[89963]: pgmap v143: 73 pgs: 15 active+undersized, 4 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-08-17T17:16:55.905 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:55 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3848845348' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:55.905 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:55 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3848845348' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:55.905 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:16:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:16:55] "GET /metrics HTTP/1.1" 200 33948 "" "Prometheus/2.51.0" 2024-08-17T17:16:55.939 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:16:55.942 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:55 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3848845348' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:57.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:57 smithi003 ceph-mon[100615]: pgmap v144: 73 pgs: 15 active+undersized, 4 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-08-17T17:16:57.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:57 smithi003 ceph-mon[100615]: from='client.34809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:57.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:57 smithi003 ceph-mon[100615]: from='client.34815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:57.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:57 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:57.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:57 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:57.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:57 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:57.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:57 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:57.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:57 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:57.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:57 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:57.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:57 smithi003 ceph-mon[103344]: pgmap v144: 73 pgs: 15 active+undersized, 4 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-08-17T17:16:57.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:57 smithi003 ceph-mon[103344]: from='client.34809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:57.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:57 smithi003 ceph-mon[103344]: from='client.34815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:57.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:57 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:57.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:57 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:57.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:57 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:57.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:57 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:57.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:57 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:57.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:57 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:57.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:57 smithi053 ceph-mon[89963]: pgmap v144: 73 pgs: 15 active+undersized, 4 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-08-17T17:16:57.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:57 smithi053 ceph-mon[89963]: from='client.34809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:57.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:57 smithi053 ceph-mon[89963]: from='client.34815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:16:57.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:57 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:57.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:57 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:57.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:57 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:57.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:57 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:57.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:57 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:57.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:57 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:16:57.830 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:57 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2[122810]: 2024-08-17T17:16:57.762+0000 7fd2f30c4640 -1 osd.2 100 log_to_monitors true 2024-08-17T17:16:58.567 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:58 smithi003 ceph-mon[103344]: from='osd.2 [v2:172.21.15.3:6818/1825512279,v1:172.21.15.3:6819/1825512279]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-08-17T17:16:58.567 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:58 smithi003 ceph-mon[100615]: from='osd.2 [v2:172.21.15.3:6818/1825512279,v1:172.21.15.3:6819/1825512279]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-08-17T17:16:58.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:58 smithi053 ceph-mon[89963]: from='osd.2 [v2:172.21.15.3:6818/1825512279,v1:172.21.15.3:6819/1825512279]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-08-17T17:16:58.861 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:16:58 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2[122810]: 2024-08-17T17:16:58.767+0000 7fd2e6e66640 -1 osd.2 100 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-17T17:16:58.900 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:16:59.534 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:59 smithi003 ceph-mon[103344]: pgmap v145: 73 pgs: 15 active+undersized, 4 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-08-17T17:16:59.535 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:59 smithi003 ceph-mon[103344]: from='osd.2 [v2:172.21.15.3:6818/1825512279,v1:172.21.15.3:6819/1825512279]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-17T17:16:59.535 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:59 smithi003 ceph-mon[103344]: osdmap e103: 8 total, 7 up, 8 in 2024-08-17T17:16:59.535 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:59 smithi003 ceph-mon[103344]: from='osd.2 [v2:172.21.15.3:6818/1825512279,v1:172.21.15.3:6819/1825512279]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-08-17T17:16:59.535 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:59 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1461782136' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:59.535 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:16:59 smithi003 ceph-mon[103344]: from='osd.2 [v2:172.21.15.3:6818/1825512279,v1:172.21.15.3:6819/1825512279]' entity='osd.2' 2024-08-17T17:16:59.535 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:59 smithi003 ceph-mon[100615]: pgmap v145: 73 pgs: 15 active+undersized, 4 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-08-17T17:16:59.535 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:59 smithi003 ceph-mon[100615]: from='osd.2 [v2:172.21.15.3:6818/1825512279,v1:172.21.15.3:6819/1825512279]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-17T17:16:59.535 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:59 smithi003 ceph-mon[100615]: osdmap e103: 8 total, 7 up, 8 in 2024-08-17T17:16:59.535 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:59 smithi003 ceph-mon[100615]: from='osd.2 [v2:172.21.15.3:6818/1825512279,v1:172.21.15.3:6819/1825512279]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-08-17T17:16:59.535 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:59 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1461782136' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:59.535 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:16:59 smithi003 ceph-mon[100615]: from='osd.2 [v2:172.21.15.3:6818/1825512279,v1:172.21.15.3:6819/1825512279]' entity='osd.2' 2024-08-17T17:16:59.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:59 smithi053 ceph-mon[89963]: pgmap v145: 73 pgs: 15 active+undersized, 4 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-08-17T17:16:59.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:59 smithi053 ceph-mon[89963]: from='osd.2 [v2:172.21.15.3:6818/1825512279,v1:172.21.15.3:6819/1825512279]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-17T17:16:59.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:59 smithi053 ceph-mon[89963]: osdmap e103: 8 total, 7 up, 8 in 2024-08-17T17:16:59.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:59 smithi053 ceph-mon[89963]: from='osd.2 [v2:172.21.15.3:6818/1825512279,v1:172.21.15.3:6819/1825512279]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-08-17T17:16:59.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:59 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1461782136' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:16:59.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:16:59 smithi053 ceph-mon[89963]: from='osd.2 [v2:172.21.15.3:6818/1825512279,v1:172.21.15.3:6819/1825512279]' entity='osd.2' 2024-08-17T17:17:00.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:17:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:17:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:17:00.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:00 smithi053 ceph-mon[89963]: from='client.44690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:00.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:00 smithi053 ceph-mon[89963]: from='client.44696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:00.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:00 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 6/84 objects degraded (7.143%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T17:17:00.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:00 smithi053 ceph-mon[89963]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T17:17:00.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:00 smithi053 ceph-mon[89963]: osd.2 [v2:172.21.15.3:6818/1825512279,v1:172.21.15.3:6819/1825512279] boot 2024-08-17T17:17:00.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:00 smithi053 ceph-mon[89963]: osdmap e104: 8 total, 8 up, 8 in 2024-08-17T17:17:00.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:00 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-17T17:17:00.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:00 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:00.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:00 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:00.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:00 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:17:00.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:00 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:17:00.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:00 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:00.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:00 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:17:00.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:00 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:17:00.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:00 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:17:00.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:00 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:17:00.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:00 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-08-17T17:17:00.523 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:17:00.257+0000 7f288dc05640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-08-17T17:17:00.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[100615]: from='client.44690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:00.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[100615]: from='client.44696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:00.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 6/84 objects degraded (7.143%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T17:17:00.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[100615]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T17:17:00.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[100615]: osd.2 [v2:172.21.15.3:6818/1825512279,v1:172.21.15.3:6819/1825512279] boot 2024-08-17T17:17:00.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[100615]: osdmap e104: 8 total, 8 up, 8 in 2024-08-17T17:17:00.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-17T17:17:00.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:00.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:00.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:17:00.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:17:00.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:00.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:17:00.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:17:00.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:17:00.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:17:00.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-08-17T17:17:00.527 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[103344]: from='client.44690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:00.527 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[103344]: from='client.44696 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:00.527 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 6/84 objects degraded (7.143%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T17:17:00.527 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[103344]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T17:17:00.527 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[103344]: osd.2 [v2:172.21.15.3:6818/1825512279,v1:172.21.15.3:6819/1825512279] boot 2024-08-17T17:17:00.527 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[103344]: osdmap e104: 8 total, 8 up, 8 in 2024-08-17T17:17:00.527 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-17T17:17:00.528 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:00.528 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:00.528 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:17:00.528 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:17:00.528 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:00.528 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:17:00.528 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:17:00.528 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:17:00.529 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:17:00.529 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:00 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-08-17T17:17:01.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:01 smithi053 ceph-mon[89963]: pgmap v147: 73 pgs: 15 active+undersized, 4 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-08-17T17:17:01.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:01 smithi053 ceph-mon[89963]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-08-17T17:17:01.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:01 smithi053 ceph-mon[89963]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-08-17T17:17:01.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:01 smithi053 ceph-mon[89963]: osdmap e105: 8 total, 8 up, 8 in 2024-08-17T17:17:01.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:01 smithi003 ceph-mon[100615]: pgmap v147: 73 pgs: 15 active+undersized, 4 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-08-17T17:17:01.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:01 smithi003 ceph-mon[100615]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-08-17T17:17:01.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:01 smithi003 ceph-mon[100615]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-08-17T17:17:01.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:01 smithi003 ceph-mon[100615]: osdmap e105: 8 total, 8 up, 8 in 2024-08-17T17:17:01.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:01 smithi003 ceph-mon[103344]: pgmap v147: 73 pgs: 15 active+undersized, 4 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-08-17T17:17:01.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:01 smithi003 ceph-mon[103344]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-08-17T17:17:01.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:01 smithi003 ceph-mon[103344]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-08-17T17:17:01.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:01 smithi003 ceph-mon[103344]: osdmap e105: 8 total, 8 up, 8 in 2024-08-17T17:17:01.799 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:17:02.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:02 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/945050077' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:02.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:02 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/945050077' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:02.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:02 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/945050077' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:03.618 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:03 smithi053 ceph-mon[89963]: pgmap v150: 73 pgs: 15 active+undersized, 4 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-08-17T17:17:03.618 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:03 smithi053 ceph-mon[89963]: from='client.44708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:03.618 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:03 smithi053 ceph-mon[89963]: from='client.44714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:03.618 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:03 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:03 smithi003 ceph-mon[100615]: pgmap v150: 73 pgs: 15 active+undersized, 4 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-08-17T17:17:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:03 smithi003 ceph-mon[100615]: from='client.44708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:03 smithi003 ceph-mon[100615]: from='client.44714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:03 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:03.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:03 smithi003 ceph-mon[103344]: pgmap v150: 73 pgs: 15 active+undersized, 4 active+undersized+degraded, 54 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 6/84 objects degraded (7.143%) 2024-08-17T17:17:03.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:03 smithi003 ceph-mon[103344]: from='client.44708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:03.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:03 smithi003 ceph-mon[103344]: from='client.44714 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:03.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:03 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:04.689 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:17:04.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:04 smithi003 ceph-mon[100615]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/84 objects degraded (7.143%), 4 pgs degraded) 2024-08-17T17:17:04.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:04 smithi003 ceph-mon[100615]: Cluster is now healthy 2024-08-17T17:17:04.690 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:04 smithi003 ceph-mon[103344]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/84 objects degraded (7.143%), 4 pgs degraded) 2024-08-17T17:17:04.690 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:04 smithi003 ceph-mon[103344]: Cluster is now healthy 2024-08-17T17:17:04.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:04 smithi053 ceph-mon[89963]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/84 objects degraded (7.143%), 4 pgs degraded) 2024-08-17T17:17:04.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:04 smithi053 ceph-mon[89963]: Cluster is now healthy 2024-08-17T17:17:05.650 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:05 smithi003 ceph-mon[103344]: pgmap v151: 73 pgs: 4 active+undersized, 69 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:05.650 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:05 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2753386932' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:05.651 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:05 smithi003 ceph-mon[100615]: pgmap v151: 73 pgs: 4 active+undersized, 69 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:05.651 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:05 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2753386932' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:05.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:05 smithi053 ceph-mon[89963]: pgmap v151: 73 pgs: 4 active+undersized, 69 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:05.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:05 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2753386932' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:06.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:17:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:17:05] "GET /metrics HTTP/1.1" 200 33948 "" "Prometheus/2.51.0" 2024-08-17T17:17:06.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:06 smithi053 ceph-mon[89963]: from='client.34866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:06.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:06 smithi053 ceph-mon[89963]: from='client.44732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:06.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:06 smithi003 ceph-mon[100615]: from='client.34866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:06.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:06 smithi003 ceph-mon[100615]: from='client.44732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:06.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:06 smithi003 ceph-mon[103344]: from='client.34866 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:06.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:06 smithi003 ceph-mon[103344]: from='client.44732 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:07.582 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:17:07.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:07 smithi053 ceph-mon[89963]: pgmap v152: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:07.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:07 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2778670258' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:07.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:07 smithi003 ceph-mon[103344]: pgmap v152: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:07.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:07 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2778670258' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:07.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:07 smithi003 ceph-mon[100615]: pgmap v152: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:07.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:07 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2778670258' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:09 smithi053 ceph-mon[89963]: from='client.34884 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:09 smithi053 ceph-mon[89963]: pgmap v153: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:09.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:09 smithi053 ceph-mon[89963]: from='client.34890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:09.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:09.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:17:09.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:09 smithi003 ceph-mon[100615]: from='client.34884 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:09.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:09 smithi003 ceph-mon[100615]: pgmap v153: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:09.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:09 smithi003 ceph-mon[100615]: from='client.34890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:09.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:09.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:17:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:09 smithi003 ceph-mon[103344]: from='client.34884 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:09 smithi003 ceph-mon[103344]: pgmap v153: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:09 smithi003 ceph-mon[103344]: from='client.34890 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:17:10.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:17:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:17:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:17:10.468 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:17:11.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:11 smithi053 ceph-mon[89963]: pgmap v154: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:11.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:11 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1875088227' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:11.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:11 smithi053 ceph-mon[89963]: from='client.44762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:11.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:11 smithi003 ceph-mon[100615]: pgmap v154: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:11.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:11 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1875088227' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:11.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:11 smithi003 ceph-mon[100615]: from='client.44762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:11.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:11 smithi003 ceph-mon[103344]: pgmap v154: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:11.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:11 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1875088227' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:11.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:11 smithi003 ceph-mon[103344]: from='client.44762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:12.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:12 smithi053 ceph-mon[89963]: from='client.34908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:12.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:12 smithi003 ceph-mon[100615]: from='client.34908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:12.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:12 smithi003 ceph-mon[103344]: from='client.34908 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:13.357 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:17:13.357 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:13 smithi003 ceph-mon[103344]: pgmap v155: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:13.358 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:13 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1638011477' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:13.358 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:13 smithi003 ceph-mon[100615]: pgmap v155: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:13.358 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:13 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1638011477' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:13.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:13 smithi053 ceph-mon[89963]: pgmap v155: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:13.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:13 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1638011477' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:14.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:14 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:14.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:14 smithi053 ceph-mon[89963]: from='client.44780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:14.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:14 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:14.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:14 smithi003 ceph-mon[103344]: from='client.44780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:14.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:14 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:14.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:14 smithi003 ceph-mon[100615]: from='client.44780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:15.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:15 smithi003 ceph-mon[103344]: from='client.34926 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:15.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:15 smithi003 ceph-mon[103344]: pgmap v156: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:15.425 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:15 smithi003 ceph-mon[100615]: from='client.34926 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:15.425 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:15 smithi003 ceph-mon[100615]: pgmap v156: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:15.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:15 smithi053 ceph-mon[89963]: from='client.34926 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:15.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:15 smithi053 ceph-mon[89963]: pgmap v156: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:15.683 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:17:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:17:15] "GET /metrics HTTP/1.1" 200 33944 "" "Prometheus/2.51.0" 2024-08-17T17:17:16.252 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:17:16.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:16 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-08-17T17:17:16.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:16 smithi003 ceph-mon[103344]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-08-17T17:17:16.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:16 smithi003 ceph-mon[103344]: Upgrade: osd.3 is safe to restart 2024-08-17T17:17:16.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:16 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1388917430' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:16.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:16 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-08-17T17:17:16.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:16 smithi003 ceph-mon[100615]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-08-17T17:17:16.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:16 smithi003 ceph-mon[100615]: Upgrade: osd.3 is safe to restart 2024-08-17T17:17:16.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:16 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1388917430' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:16.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:16 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-08-17T17:17:16.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:16 smithi053 ceph-mon[89963]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-08-17T17:17:16.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:16 smithi053 ceph-mon[89963]: Upgrade: osd.3 is safe to restart 2024-08-17T17:17:16.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:16 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1388917430' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:17.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:17 smithi003 ceph-mon[103344]: pgmap v157: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:17.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:17 smithi003 ceph-mon[103344]: from='client.34938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:17.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:17 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:17.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:17 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-08-17T17:17:17.435 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:17 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:17:17.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:17 smithi003 ceph-mon[100615]: pgmap v157: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:17.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:17 smithi003 ceph-mon[100615]: from='client.34938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:17.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:17 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:17.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:17 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-08-17T17:17:17.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:17 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:17:17.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:17 smithi053 ceph-mon[89963]: pgmap v157: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:17.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:17 smithi053 ceph-mon[89963]: from='client.34938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:17.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:17 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:17.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:17 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-08-17T17:17:17.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:17 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:17:18.158 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:18 smithi003 ceph-mon[103344]: from='client.44804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:18.158 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:18 smithi003 ceph-mon[103344]: Upgrade: Updating osd.3 2024-08-17T17:17:18.158 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:18 smithi003 ceph-mon[103344]: Deploying daemon osd.3 on smithi003 2024-08-17T17:17:18.159 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:18 smithi003 ceph-mon[100615]: from='client.44804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:18.159 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:18 smithi003 ceph-mon[100615]: Upgrade: Updating osd.3 2024-08-17T17:17:18.159 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:18 smithi003 ceph-mon[100615]: Deploying daemon osd.3 on smithi003 2024-08-17T17:17:18.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:18 smithi053 ceph-mon[89963]: from='client.44804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:18.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:18 smithi053 ceph-mon[89963]: Upgrade: Updating osd.3 2024-08-17T17:17:18.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:18 smithi053 ceph-mon[89963]: Deploying daemon osd.3 on smithi003 2024-08-17T17:17:19.139 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:17:19.321 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:19 smithi003 ceph-mon[103344]: pgmap v158: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:19.321 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:19 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2451443681' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:19.321 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:19 smithi003 ceph-mon[100615]: pgmap v158: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:19.322 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:19 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2451443681' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:19.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:19 smithi053 ceph-mon[89963]: pgmap v158: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:19.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:19 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2451443681' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:19.994 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:19 smithi003 systemd[1]: Stopping Ceph osd.3 for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:17:20.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:20 smithi003 ceph-mon[103344]: from='client.34956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:20.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:20 smithi003 ceph-mon[103344]: from='client.34962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:20.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:20 smithi003 ceph-mon[103344]: osd.3 marked itself down and dead 2024-08-17T17:17:20.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:20 smithi003 ceph-mon[100615]: from='client.34956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:20.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:20 smithi003 ceph-mon[100615]: from='client.34962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:20.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:20 smithi003 ceph-mon[100615]: osd.3 marked itself down and dead 2024-08-17T17:17:20.274 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:19 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3[62740]: 2024-08-17T17:17:19.992+0000 7f2a88741640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T17:17:20.274 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:19 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3[62740]: 2024-08-17T17:17:19.992+0000 7f2a88741640 -1 osd.3 105 *** Got signal Terminated *** 2024-08-17T17:17:20.274 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:19 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3[62740]: 2024-08-17T17:17:19.992+0000 7f2a88741640 -1 osd.3 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-17T17:17:20.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:17:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:17:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:17:20.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:20 smithi053 ceph-mon[89963]: from='client.34956 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:20.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:20 smithi053 ceph-mon[89963]: from='client.34962 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:20.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:20 smithi053 ceph-mon[89963]: osd.3 marked itself down and dead 2024-08-17T17:17:20.959 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:20 smithi003 podman[128004]: 2024-08-17 17:17:20.689290022 +0000 UTC m=+0.812813042 container died 61f40a5861f64f945732ab14d1c5bb2041d2d9f746b46093b5633954c4493888 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-f2c39d4, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-17T17:17:21.219 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:21 smithi003 ceph-mon[100615]: pgmap v159: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:21.219 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:21 smithi003 ceph-mon[100615]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T17:17:21.220 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:21 smithi003 ceph-mon[100615]: osdmap e106: 8 total, 7 up, 8 in 2024-08-17T17:17:21.220 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:21 smithi003 ceph-mon[103344]: pgmap v159: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:21.220 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:21 smithi003 ceph-mon[103344]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T17:17:21.220 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:21 smithi003 ceph-mon[103344]: osdmap e106: 8 total, 7 up, 8 in 2024-08-17T17:17:21.220 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:20 smithi003 podman[128004]: 2024-08-17 17:17:20.958726188 +0000 UTC m=+1.082249207 container cleanup 61f40a5861f64f945732ab14d1c5bb2041d2d9f746b46093b5633954c4493888 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=quincy-f2c39d4, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, io.buildah.version=1.36.0) 2024-08-17T17:17:21.220 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:20 smithi003 bash[128004]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3 2024-08-17T17:17:21.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:21 smithi053 ceph-mon[89963]: pgmap v159: 73 pgs: 73 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:21.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:21 smithi053 ceph-mon[89963]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T17:17:21.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:21 smithi053 ceph-mon[89963]: osdmap e106: 8 total, 7 up, 8 in 2024-08-17T17:17:21.522 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:21 smithi003 podman[128021]: 2024-08-17 17:17:21.21898334 +0000 UTC m=+0.521234689 container remove 61f40a5861f64f945732ab14d1c5bb2041d2d9f746b46093b5633954c4493888 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, RELEASE=quincy-f2c39d4, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-17T17:17:21.802 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:21 smithi003 podman[128109]: 2024-08-17 17:17:21.452806963 +0000 UTC m=+0.018391195 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:17:21.803 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:21 smithi003 podman[128109]: 2024-08-17 17:17:21.569727973 +0000 UTC m=+0.135312213 container create f387bf8bc88fc942b6db04feec1c6e88f2338221f4c520d7d2c85603b3fb8dc7 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-cd8bdbe, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True) 2024-08-17T17:17:21.803 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:21 smithi003 podman[128109]: 2024-08-17 17:17:21.797807672 +0000 UTC m=+0.363391907 container init f387bf8bc88fc942b6db04feec1c6e88f2338221f4c520d7d2c85603b3fb8dc7 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-cd8bdbe, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-08-17T17:17:22.039 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:17:22.169 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:21 smithi003 podman[128109]: 2024-08-17 17:17:21.802250193 +0000 UTC m=+0.367834429 container start f387bf8bc88fc942b6db04feec1c6e88f2338221f4c520d7d2c85603b3fb8dc7 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3-deactivate, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-cd8bdbe, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-08-17T17:17:22.169 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:21 smithi003 podman[128109]: 2024-08-17 17:17:21.872851826 +0000 UTC m=+0.438436083 container attach f387bf8bc88fc942b6db04feec1c6e88f2338221f4c520d7d2c85603b3fb8dc7 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=main-cd8bdbe, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-08-17T17:17:22.169 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:22 smithi003 podman[128109]: 2024-08-17 17:17:22.002246662 +0000 UTC m=+0.567830898 container died f387bf8bc88fc942b6db04feec1c6e88f2338221f4c520d7d2c85603b3fb8dc7 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-cd8bdbe, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-08-17T17:17:22.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:22 smithi053 ceph-mon[89963]: osdmap e107: 8 total, 7 up, 8 in 2024-08-17T17:17:22.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:22 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/786033102' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:22.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:22 smithi003 ceph-mon[103344]: osdmap e107: 8 total, 7 up, 8 in 2024-08-17T17:17:22.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:22 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/786033102' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:22.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:22 smithi003 ceph-mon[100615]: osdmap e107: 8 total, 7 up, 8 in 2024-08-17T17:17:22.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:22 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/786033102' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:23.021 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:22 smithi003 podman[128178]: 2024-08-17 17:17:22.568704919 +0000 UTC m=+0.558400080 container remove f387bf8bc88fc942b6db04feec1c6e88f2338221f4c520d7d2c85603b3fb8dc7 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-cd8bdbe, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-08-17T17:17:23.022 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:22 smithi003 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.3.service: Deactivated successfully. 2024-08-17T17:17:23.022 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:22 smithi003 systemd[1]: Stopped Ceph osd.3 for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:17:23.022 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:22 smithi003 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.3.service: Consumed 5.153s CPU time. 2024-08-17T17:17:23.022 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:22 smithi003 systemd[1]: Starting Ceph osd.3 for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:17:23.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:23 smithi003 ceph-mon[100615]: pgmap v162: 73 pgs: 13 stale+active+clean, 60 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:23.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:23 smithi003 ceph-mon[100615]: from='client.44834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:23.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:23 smithi003 ceph-mon[100615]: from='client.44840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:23.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:23.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:17:23.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:23 smithi003 ceph-mon[103344]: pgmap v162: 73 pgs: 13 stale+active+clean, 60 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:23.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:23 smithi003 ceph-mon[103344]: from='client.44834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:23.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:23 smithi003 ceph-mon[103344]: from='client.44840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:23.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:23.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:17:23.275 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:23 smithi003 podman[128306]: 2024-08-17 17:17:22.923743552 +0000 UTC m=+0.018248913 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:17:23.275 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:23 smithi003 podman[128306]: 2024-08-17 17:17:23.062162607 +0000 UTC m=+0.156667988 container create 99c7b0925432187b4355b2b934b22fac5302f30bbc3d2b77dff421d400232854 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3-activate, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=main-cd8bdbe, GIT_CLEAN=True) 2024-08-17T17:17:23.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:23 smithi053 ceph-mon[89963]: pgmap v162: 73 pgs: 13 stale+active+clean, 60 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:23.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:23 smithi053 ceph-mon[89963]: from='client.44834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:23.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:23 smithi053 ceph-mon[89963]: from='client.44840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:23.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:23.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:17:23.773 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:23 smithi003 podman[128306]: 2024-08-17 17:17:23.306015053 +0000 UTC m=+0.400520395 container init 99c7b0925432187b4355b2b934b22fac5302f30bbc3d2b77dff421d400232854 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, RELEASE=main-cd8bdbe, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-08-17T17:17:23.773 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:23 smithi003 podman[128306]: 2024-08-17 17:17:23.309838465 +0000 UTC m=+0.404343812 container start 99c7b0925432187b4355b2b934b22fac5302f30bbc3d2b77dff421d400232854 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3-activate, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-cd8bdbe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS) 2024-08-17T17:17:23.774 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:23 smithi003 podman[128306]: 2024-08-17 17:17:23.414708119 +0000 UTC m=+0.509213474 container attach 99c7b0925432187b4355b2b934b22fac5302f30bbc3d2b77dff421d400232854 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3-activate, RELEASE=main-cd8bdbe, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-08-17T17:17:23.774 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:23 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3-activate[128321]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:17:23.774 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:23 smithi003 bash[128306]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:17:23.774 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:23 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3-activate[128321]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:17:23.774 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:23 smithi003 bash[128306]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:17:23.774 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:23 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3-activate[128321]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-17T17:17:23.774 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:23 smithi003 bash[128306]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-17T17:17:23.774 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:23 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3-activate[128321]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:17:23.774 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:23 smithi003 bash[128306]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:17:23.774 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:23 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3-activate[128321]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:17:23.774 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:23 smithi003 bash[128306]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:17:23.775 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:23 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3-activate[128321]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-08-17T17:17:23.775 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:23 smithi003 bash[128306]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-08-17T17:17:23.775 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:23 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3-activate[128321]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-3 --no-mon-config 2024-08-17T17:17:23.775 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:23 smithi003 bash[128306]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_1 --path /var/lib/ceph/osd/ceph-3 --no-mon-config 2024-08-17T17:17:24.152 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:23 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3-activate[128321]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3/block 2024-08-17T17:17:24.152 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:23 smithi003 bash[128306]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3/block 2024-08-17T17:17:24.152 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:23 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3-activate[128321]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block 2024-08-17T17:17:24.152 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:23 smithi003 bash[128306]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-3/block 2024-08-17T17:17:24.152 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:23 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3-activate[128321]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-08-17T17:17:24.152 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:23 smithi003 bash[128306]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-08-17T17:17:24.152 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:23 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3-activate[128321]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-08-17T17:17:24.152 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:23 smithi003 bash[128306]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-08-17T17:17:24.152 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:23 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3-activate[128321]: --> ceph-volume lvm activate successful for osd ID: 3 2024-08-17T17:17:24.152 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:23 smithi003 bash[128306]: --> ceph-volume lvm activate successful for osd ID: 3 2024-08-17T17:17:24.153 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:23 smithi003 podman[128427]: 2024-08-17 17:17:23.892847424 +0000 UTC m=+0.018889453 container died 99c7b0925432187b4355b2b934b22fac5302f30bbc3d2b77dff421d400232854 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3-activate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-cd8bdbe, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:17:24.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:24 smithi003 ceph-mon[100615]: Health check failed: Degraded data redundancy: 4/84 objects degraded (4.762%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:17:24.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:24 smithi003 ceph-mon[103344]: Health check failed: Degraded data redundancy: 4/84 objects degraded (4.762%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:17:24.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:24 smithi053 ceph-mon[89963]: Health check failed: Degraded data redundancy: 4/84 objects degraded (4.762%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:17:24.874 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:24 smithi003 podman[128427]: 2024-08-17 17:17:24.527474691 +0000 UTC m=+0.653516720 container remove 99c7b0925432187b4355b2b934b22fac5302f30bbc3d2b77dff421d400232854 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3-activate, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-cd8bdbe, io.buildah.version=1.36.0) 2024-08-17T17:17:24.874 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:24 smithi003 podman[128509]: 2024-08-17 17:17:24.659932482 +0000 UTC m=+0.022085979 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:17:24.874 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:24 smithi003 podman[128509]: 2024-08-17 17:17:24.761914834 +0000 UTC m=+0.124068313 container create fe4dcbe3219e018f2e8d9a3e2889b21a7c754b13c51cd73113cfffc972d10ab9 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-cd8bdbe, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True) 2024-08-17T17:17:24.956 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:17:25.223 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:25 smithi003 podman[128509]: 2024-08-17 17:17:25.006266609 +0000 UTC m=+0.368420089 container init fe4dcbe3219e018f2e8d9a3e2889b21a7c754b13c51cd73113cfffc972d10ab9 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, RELEASE=main-cd8bdbe, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-08-17T17:17:25.223 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:25 smithi003 podman[128509]: 2024-08-17 17:17:25.00973585 +0000 UTC m=+0.371889329 container start fe4dcbe3219e018f2e8d9a3e2889b21a7c754b13c51cd73113cfffc972d10ab9 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-cd8bdbe, ceph=True, io.buildah.version=1.36.0) 2024-08-17T17:17:25.223 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:25 smithi003 ceph-osd[128556]: -- 172.21.15.3:0/3013516802 <== mon.2 v2:172.21.15.3:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d2b6fe1a40 con 0x55d2b7dae000 2024-08-17T17:17:25.223 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:25 smithi003 bash[128509]: fe4dcbe3219e018f2e8d9a3e2889b21a7c754b13c51cd73113cfffc972d10ab9 2024-08-17T17:17:25.223 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:25 smithi003 systemd[1]: Started Ceph osd.3 for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:17:25.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:25 smithi003 ceph-mon[100615]: pgmap v163: 73 pgs: 22 active+undersized, 3 stale+active+clean, 3 active+undersized+degraded, 45 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-08-17T17:17:25.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:25 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/4196536809' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:25.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:25 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:25.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:25 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:25.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:25 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:17:25.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:25 smithi003 ceph-mon[103344]: pgmap v163: 73 pgs: 22 active+undersized, 3 stale+active+clean, 3 active+undersized+degraded, 45 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-08-17T17:17:25.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:25 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/4196536809' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:25.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:25 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:25.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:25 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:25.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:25 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:17:25.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:25 smithi053 ceph-mon[89963]: pgmap v163: 73 pgs: 22 active+undersized, 3 stale+active+clean, 3 active+undersized+degraded, 45 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-08-17T17:17:25.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:25 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/4196536809' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:25.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:25 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:25.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:25 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:25.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:25 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:17:25.919 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:17:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:17:25] "GET /metrics HTTP/1.1" 200 33944 "" "Prometheus/2.51.0" 2024-08-17T17:17:26.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:26 smithi003 ceph-mon[100615]: from='client.34992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:26.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:26 smithi003 ceph-mon[100615]: from='client.34998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:26.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:26 smithi003 ceph-mon[103344]: from='client.34992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:26.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:26 smithi003 ceph-mon[103344]: from='client.34998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:26.523 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:26 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3[128547]: 2024-08-17T17:17:26.424+0000 7f7f9825e640 -1 Falling back to public interface 2024-08-17T17:17:26.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:26 smithi053 ceph-mon[89963]: from='client.34992 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:26.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:26 smithi053 ceph-mon[89963]: from='client.34998 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:27.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:27 smithi003 ceph-mon[103344]: pgmap v164: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-08-17T17:17:27.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:27 smithi003 ceph-mon[100615]: pgmap v164: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-08-17T17:17:27.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:27 smithi053 ceph-mon[89963]: pgmap v164: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-08-17T17:17:27.871 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:17:28.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:28 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/492975013' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:28.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:28 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/492975013' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:28.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:28 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/492975013' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:29.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:29 smithi003 ceph-mon[103344]: pgmap v165: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-08-17T17:17:29.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:29 smithi003 ceph-mon[103344]: from='client.44873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:29.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:29 smithi003 ceph-mon[103344]: from='client.44879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:29.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:29 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:29.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:29 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:29.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:29 smithi003 ceph-mon[100615]: pgmap v165: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-08-17T17:17:29.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:29 smithi003 ceph-mon[100615]: from='client.44873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:29.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:29 smithi003 ceph-mon[100615]: from='client.44879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:29.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:29 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:29.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:29 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:29.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:29 smithi053 ceph-mon[89963]: pgmap v165: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-08-17T17:17:29.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:29 smithi053 ceph-mon[89963]: from='client.44873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:29.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:29 smithi053 ceph-mon[89963]: from='client.44879 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:29.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:29 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:29.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:29 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:30.023 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:29 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3[128547]: 2024-08-17T17:17:29.939+0000 7f7f9825e640 -1 osd.3 105 log_to_monitors true 2024-08-17T17:17:30.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:17:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:17:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:17:30.771 INFO:teuthology.orchestra.run.smithi003.stdout:true 2024-08-17T17:17:30.771 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:17:30 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3[128547]: 2024-08-17T17:17:30.504+0000 7f7f90008640 -1 osd.3 105 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-17T17:17:30.771 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:30 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:30.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:30 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:30.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:30 smithi003 ceph-mon[103344]: pgmap v166: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-08-17T17:17:30.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:30 smithi003 ceph-mon[103344]: from='osd.3 [v2:172.21.15.3:6826/1366347928,v1:172.21.15.3:6827/1366347928]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-08-17T17:17:30.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:30 smithi003 ceph-mon[103344]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-08-17T17:17:30.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:30 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3501014835' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:30.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:30 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:30.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:30 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:30.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:30 smithi003 ceph-mon[100615]: pgmap v166: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-08-17T17:17:30.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:30 smithi003 ceph-mon[100615]: from='osd.3 [v2:172.21.15.3:6826/1366347928,v1:172.21.15.3:6827/1366347928]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-08-17T17:17:30.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:30 smithi003 ceph-mon[100615]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-08-17T17:17:30.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:30 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3501014835' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:30.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:30 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:30.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:30 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:30.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:30 smithi053 ceph-mon[89963]: pgmap v166: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-08-17T17:17:30.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:30 smithi053 ceph-mon[89963]: from='osd.3 [v2:172.21.15.3:6826/1366347928,v1:172.21.15.3:6827/1366347928]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-08-17T17:17:30.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:30 smithi053 ceph-mon[89963]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-08-17T17:17:30.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:30 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3501014835' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:31.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:31 smithi003 ceph-mon[103344]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-08-17T17:17:31.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:31 smithi003 ceph-mon[103344]: osdmap e108: 8 total, 7 up, 8 in 2024-08-17T17:17:31.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:31 smithi003 ceph-mon[103344]: from='osd.3 [v2:172.21.15.3:6826/1366347928,v1:172.21.15.3:6827/1366347928]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-08-17T17:17:31.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:31 smithi003 ceph-mon[103344]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-08-17T17:17:31.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:31 smithi003 ceph-mon[103344]: from='client.35034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:31.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:31 smithi003 ceph-mon[103344]: from='client.44900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:31.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:31 smithi003 ceph-mon[100615]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-08-17T17:17:31.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:31 smithi003 ceph-mon[100615]: osdmap e108: 8 total, 7 up, 8 in 2024-08-17T17:17:31.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:31 smithi003 ceph-mon[100615]: from='osd.3 [v2:172.21.15.3:6826/1366347928,v1:172.21.15.3:6827/1366347928]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-08-17T17:17:31.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:31 smithi003 ceph-mon[100615]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-08-17T17:17:31.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:31 smithi003 ceph-mon[100615]: from='client.35034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:31.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:31 smithi003 ceph-mon[100615]: from='client.44900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:31.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:31 smithi053 ceph-mon[89963]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-08-17T17:17:31.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:31 smithi053 ceph-mon[89963]: osdmap e108: 8 total, 7 up, 8 in 2024-08-17T17:17:31.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:31 smithi053 ceph-mon[89963]: from='osd.3 [v2:172.21.15.3:6826/1366347928,v1:172.21.15.3:6827/1366347928]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-08-17T17:17:31.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:31 smithi053 ceph-mon[89963]: from='osd.3 ' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-08-17T17:17:31.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:31 smithi053 ceph-mon[89963]: from='client.35034 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:31.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:31 smithi053 ceph-mon[89963]: from='client.44900 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:32.757 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:32 smithi003 ceph-mon[103344]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T17:17:32.757 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:32 smithi003 ceph-mon[103344]: osd.3 [v2:172.21.15.3:6826/1366347928,v1:172.21.15.3:6827/1366347928] boot 2024-08-17T17:17:32.757 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:32 smithi003 ceph-mon[103344]: osdmap e109: 8 total, 8 up, 8 in 2024-08-17T17:17:32.757 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:32 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-17T17:17:32.757 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:32 smithi003 ceph-mon[103344]: pgmap v169: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-08-17T17:17:32.757 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:32 smithi003 ceph-mon[103344]: osdmap e110: 8 total, 8 up, 8 in 2024-08-17T17:17:32.757 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:32 smithi003 ceph-mon[100615]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T17:17:32.758 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:32 smithi003 ceph-mon[100615]: osd.3 [v2:172.21.15.3:6826/1366347928,v1:172.21.15.3:6827/1366347928] boot 2024-08-17T17:17:32.758 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:32 smithi003 ceph-mon[100615]: osdmap e109: 8 total, 8 up, 8 in 2024-08-17T17:17:32.758 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:32 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-17T17:17:32.758 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:32 smithi003 ceph-mon[100615]: pgmap v169: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-08-17T17:17:32.758 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:32 smithi003 ceph-mon[100615]: osdmap e110: 8 total, 8 up, 8 in 2024-08-17T17:17:32.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:32 smithi053 ceph-mon[89963]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T17:17:32.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:32 smithi053 ceph-mon[89963]: osd.3 [v2:172.21.15.3:6826/1366347928,v1:172.21.15.3:6827/1366347928] boot 2024-08-17T17:17:32.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:32 smithi053 ceph-mon[89963]: osdmap e109: 8 total, 8 up, 8 in 2024-08-17T17:17:32.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:32 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-17T17:17:32.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:32 smithi053 ceph-mon[89963]: pgmap v169: 73 pgs: 28 active+undersized, 3 active+undersized+degraded, 42 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4/84 objects degraded (4.762%) 2024-08-17T17:17:32.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:32 smithi053 ceph-mon[89963]: osdmap e110: 8 total, 8 up, 8 in 2024-08-17T17:17:33.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:17:32 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:17:32.939+0000 7f288dc05640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-08-17T17:17:33.362 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 4 2024-08-17T17:17:34.011 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:33 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:34.011 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:33 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:34.011 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:33 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:17:34.011 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:33 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:17:34.011 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:33 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:34.011 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:33 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:17:34.011 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:33 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:17:34.011 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:33 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:17:34.011 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:33 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:17:34.011 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:33 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-08-17T17:17:34.011 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:33 smithi003 ceph-mon[103344]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-08-17T17:17:34.011 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:33 smithi003 ceph-mon[103344]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-08-17T17:17:34.012 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:33 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1811185840' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:34.012 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:33 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:34.012 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:33 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:34.012 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:33 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:17:34.012 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:33 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:17:34.012 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:33 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:34.012 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:33 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:17:34.012 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:33 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:17:34.012 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:33 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:17:34.012 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:33 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:17:34.012 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:33 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-08-17T17:17:34.012 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:33 smithi003 ceph-mon[100615]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-08-17T17:17:34.013 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:33 smithi003 ceph-mon[100615]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-08-17T17:17:34.013 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:33 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1811185840' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:34.038 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'ceph orch upgrade pause' 2024-08-17T17:17:34.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:33 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:34.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:33 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:34.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:33 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:17:34.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:33 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:17:34.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:33 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:34.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:33 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:17:34.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:33 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:17:34.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:33 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:17:34.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:33 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:17:34.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:33 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-08-17T17:17:34.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:33 smithi053 ceph-mon[89963]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-08-17T17:17:34.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:33 smithi053 ceph-mon[89963]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-08-17T17:17:34.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:33 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1811185840' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-08-17T17:17:34.309 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:17:35.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:34 smithi003 ceph-mon[100615]: pgmap v171: 73 pgs: 5 peering, 14 active+undersized, 2 active+undersized+degraded, 52 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2/84 objects degraded (2.381%) 2024-08-17T17:17:35.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:34 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 2/84 objects degraded (2.381%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:17:35.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:34 smithi003 ceph-mon[103344]: pgmap v171: 73 pgs: 5 peering, 14 active+undersized, 2 active+undersized+degraded, 52 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2/84 objects degraded (2.381%) 2024-08-17T17:17:35.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:34 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 2/84 objects degraded (2.381%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:17:35.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:34 smithi053 ceph-mon[89963]: pgmap v171: 73 pgs: 5 peering, 14 active+undersized, 2 active+undersized+degraded, 52 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2/84 objects degraded (2.381%) 2024-08-17T17:17:35.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:34 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 2/84 objects degraded (2.381%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:17:35.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:35 smithi003 ceph-mon[103344]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/84 objects degraded (2.381%), 2 pgs degraded) 2024-08-17T17:17:35.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:35 smithi003 ceph-mon[103344]: Cluster is now healthy 2024-08-17T17:17:35.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:35 smithi003 ceph-mon[100615]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/84 objects degraded (2.381%), 2 pgs degraded) 2024-08-17T17:17:35.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:35 smithi003 ceph-mon[100615]: Cluster is now healthy 2024-08-17T17:17:35.933 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:17:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:17:35] "GET /metrics HTTP/1.1" 200 33944 "" "Prometheus/2.51.0" 2024-08-17T17:17:36.115 INFO:teuthology.orchestra.run.smithi003.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:17:36.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:35 smithi053 ceph-mon[89963]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/84 objects degraded (2.381%), 2 pgs degraded) 2024-08-17T17:17:36.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:35 smithi053 ceph-mon[89963]: Cluster is now healthy 2024-08-17T17:17:36.706 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'ceph orch ps' 2024-08-17T17:17:36.978 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:17:37.352 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:37 smithi003 ceph-mon[103344]: pgmap v172: 73 pgs: 5 peering, 68 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:37.353 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:37 smithi003 ceph-mon[103344]: from='client.35052 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:37.353 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:37 smithi003 ceph-mon[103344]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:17:37.353 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:37 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:37.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:37 smithi003 ceph-mon[100615]: pgmap v172: 73 pgs: 5 peering, 68 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:37.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:37 smithi003 ceph-mon[100615]: from='client.35052 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:37.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:37 smithi003 ceph-mon[100615]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:17:37.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:37 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:37.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:37 smithi053 ceph-mon[89963]: pgmap v172: 73 pgs: 5 peering, 68 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:37.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:37 smithi053 ceph-mon[89963]: from='client.35052 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:37.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:37 smithi053 ceph-mon[89963]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:17:37.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:37 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:38.770 INFO:teuthology.orchestra.run.smithi003.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-08-17T17:17:38.770 INFO:teuthology.orchestra.run.smithi003.stdout:alertmanager.a smithi003 *:9093,9094 running (6m) 9s ago 10m 18.0M - 0.27.0 11f11916f8cd fb501c5aa873 2024-08-17T17:17:38.770 INFO:teuthology.orchestra.run.smithi003.stdout:grafana.a smithi053 *:3000 running (6m) 41s ago 10m 86.8M - 9.4.7 954c08fa6188 9b5ef97a5cd5 2024-08-17T17:17:38.771 INFO:teuthology.orchestra.run.smithi003.stdout:mds.foo.smithi003.civgvf smithi003 running (9m) 9s ago 9m 20.8M - 17.2.7-1404-gf2c39d4d f69471c56189 86a91b686f66 2024-08-17T17:17:38.771 INFO:teuthology.orchestra.run.smithi003.stdout:mds.foo.smithi053.ougbtc smithi053 running (9m) 41s ago 9m 16.9M - 17.2.7-1404-gf2c39d4d f69471c56189 41ad4cc0ce19 2024-08-17T17:17:38.771 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.x smithi053 *:8443,9283,8765 running (5m) 41s ago 14m 454M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 874be7d3a238 2024-08-17T17:17:38.771 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.y smithi003 *:8443,9283,8765 running (5m) 9s ago 16m 518M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb f491ea24df28 2024-08-17T17:17:38.771 INFO:teuthology.orchestra.run.smithi003.stdout:mon.a smithi003 running (5m) 9s ago 16m 48.7M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb c1cfbc2d47f8 2024-08-17T17:17:38.771 INFO:teuthology.orchestra.run.smithi003.stdout:mon.b smithi053 running (3m) 41s ago 15m 35.6M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 9dc96112f3af 2024-08-17T17:17:38.771 INFO:teuthology.orchestra.run.smithi003.stdout:mon.c smithi003 running (4m) 9s ago 15m 42.6M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 54a6cfe9eb9a 2024-08-17T17:17:38.772 INFO:teuthology.orchestra.run.smithi003.stdout:node-exporter.a smithi003 *:9100 running (6m) 9s ago 10m 11.9M - 1.7.0 72c9c2088986 d19c9e3d51f8 2024-08-17T17:17:38.772 INFO:teuthology.orchestra.run.smithi003.stdout:node-exporter.b smithi053 *:9100 running (6m) 41s ago 10m 10.3M - 1.7.0 72c9c2088986 edfcc96e3983 2024-08-17T17:17:38.772 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0 smithi003 running (112s) 9s ago 14m 43.7M 2481M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 5d56fa8e72d5 2024-08-17T17:17:38.772 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1 smithi003 running (79s) 9s ago 14m 49.1M 2481M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb f83a0fcf5c4c 2024-08-17T17:17:38.772 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2 smithi003 running (46s) 9s ago 13m 45.0M 2481M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb b30ff78c142e 2024-08-17T17:17:38.772 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3 smithi003 running (14s) 9s ago 13m 14.4M 2481M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb fe4dcbe3219e 2024-08-17T17:17:38.772 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4 smithi053 running (12m) 41s ago 12m 50.7M 2481M 17.2.7-1404-gf2c39d4d f69471c56189 ced3b086d7f0 2024-08-17T17:17:38.772 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5 smithi053 running (12m) 41s ago 12m 49.0M 2481M 17.2.7-1404-gf2c39d4d f69471c56189 1c4ad574a193 2024-08-17T17:17:38.772 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6 smithi053 running (12m) 41s ago 12m 49.2M 2481M 17.2.7-1404-gf2c39d4d f69471c56189 5ed81ad77c62 2024-08-17T17:17:38.772 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7 smithi053 running (11m) 41s ago 11m 50.5M 2481M 17.2.7-1404-gf2c39d4d f69471c56189 f3e467b07793 2024-08-17T17:17:38.772 INFO:teuthology.orchestra.run.smithi003.stdout:prometheus.a smithi053 *:9095 running (5m) 41s ago 11m 54.0M - 2.51.0 1d3b7f56885b 41146206916a 2024-08-17T17:17:39.341 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:39 smithi003 ceph-mon[100615]: pgmap v173: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:39.342 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:39.342 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:17:39.342 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:39 smithi003 ceph-mon[103344]: pgmap v173: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:39.342 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:39.342 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:17:39.366 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'ceph orch ps' 2024-08-17T17:17:39.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:39 smithi053 ceph-mon[89963]: pgmap v173: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:39.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:39.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:17:39.631 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:17:40.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:40 smithi053 ceph-mon[89963]: from='client.35058 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:40.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:17:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:17:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:17:40.473 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:40 smithi003 ceph-mon[103344]: from='client.35058 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:40.473 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:40 smithi003 ceph-mon[100615]: from='client.35058 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:41.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:41 smithi053 ceph-mon[89963]: pgmap v174: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:41.488 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:41 smithi003 ceph-mon[100615]: pgmap v174: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:41.488 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:41 smithi003 ceph-mon[103344]: pgmap v174: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:41.489 INFO:teuthology.orchestra.run.smithi003.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-08-17T17:17:41.489 INFO:teuthology.orchestra.run.smithi003.stdout:alertmanager.a smithi003 *:9093,9094 running (6m) 12s ago 11m 18.0M - 0.27.0 11f11916f8cd fb501c5aa873 2024-08-17T17:17:41.489 INFO:teuthology.orchestra.run.smithi003.stdout:grafana.a smithi053 *:3000 running (6m) 44s ago 10m 86.8M - 9.4.7 954c08fa6188 9b5ef97a5cd5 2024-08-17T17:17:41.489 INFO:teuthology.orchestra.run.smithi003.stdout:mds.foo.smithi003.civgvf smithi003 running (9m) 12s ago 9m 20.8M - 17.2.7-1404-gf2c39d4d f69471c56189 86a91b686f66 2024-08-17T17:17:41.489 INFO:teuthology.orchestra.run.smithi003.stdout:mds.foo.smithi053.ougbtc smithi053 running (9m) 44s ago 9m 16.9M - 17.2.7-1404-gf2c39d4d f69471c56189 41ad4cc0ce19 2024-08-17T17:17:41.489 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.x smithi053 *:8443,9283,8765 running (5m) 44s ago 14m 454M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 874be7d3a238 2024-08-17T17:17:41.489 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.y smithi003 *:8443,9283,8765 running (5m) 12s ago 16m 518M - 19.0.0-5836-gcd8bdbeb 0566dfeff0cb f491ea24df28 2024-08-17T17:17:41.489 INFO:teuthology.orchestra.run.smithi003.stdout:mon.a smithi003 running (5m) 12s ago 16m 48.7M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb c1cfbc2d47f8 2024-08-17T17:17:41.489 INFO:teuthology.orchestra.run.smithi003.stdout:mon.b smithi053 running (3m) 44s ago 15m 35.6M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 9dc96112f3af 2024-08-17T17:17:41.489 INFO:teuthology.orchestra.run.smithi003.stdout:mon.c smithi003 running (4m) 12s ago 15m 42.6M 2048M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 54a6cfe9eb9a 2024-08-17T17:17:41.490 INFO:teuthology.orchestra.run.smithi003.stdout:node-exporter.a smithi003 *:9100 running (6m) 12s ago 10m 11.9M - 1.7.0 72c9c2088986 d19c9e3d51f8 2024-08-17T17:17:41.490 INFO:teuthology.orchestra.run.smithi003.stdout:node-exporter.b smithi053 *:9100 running (6m) 44s ago 10m 10.3M - 1.7.0 72c9c2088986 edfcc96e3983 2024-08-17T17:17:41.490 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0 smithi003 running (114s) 12s ago 14m 43.7M 2481M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb 5d56fa8e72d5 2024-08-17T17:17:41.490 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1 smithi003 running (82s) 12s ago 14m 49.1M 2481M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb f83a0fcf5c4c 2024-08-17T17:17:41.490 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2 smithi003 running (49s) 12s ago 13m 45.0M 2481M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb b30ff78c142e 2024-08-17T17:17:41.490 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3 smithi003 running (16s) 12s ago 13m 14.4M 2481M 19.0.0-5836-gcd8bdbeb 0566dfeff0cb fe4dcbe3219e 2024-08-17T17:17:41.490 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4 smithi053 running (13m) 44s ago 13m 50.7M 2481M 17.2.7-1404-gf2c39d4d f69471c56189 ced3b086d7f0 2024-08-17T17:17:41.490 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5 smithi053 running (12m) 44s ago 12m 49.0M 2481M 17.2.7-1404-gf2c39d4d f69471c56189 1c4ad574a193 2024-08-17T17:17:41.490 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6 smithi053 running (12m) 44s ago 12m 49.2M 2481M 17.2.7-1404-gf2c39d4d f69471c56189 5ed81ad77c62 2024-08-17T17:17:41.490 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7 smithi053 running (11m) 44s ago 11m 50.5M 2481M 17.2.7-1404-gf2c39d4d f69471c56189 f3e467b07793 2024-08-17T17:17:41.490 INFO:teuthology.orchestra.run.smithi003.stdout:prometheus.a smithi053 *:9095 running (5m) 44s ago 11m 54.0M - 2.51.0 1d3b7f56885b 41146206916a 2024-08-17T17:17:42.100 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -e sha1=cd8bdbeb9fbdab5733aa3aaadc042c965473e453 -- bash -c 'ceph versions' 2024-08-17T17:17:42.367 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:17:43.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:43 smithi053 ceph-mon[89963]: from='client.44924 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:43 smithi053 ceph-mon[89963]: pgmap v175: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:43.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:43 smithi003 ceph-mon[103344]: from='client.44924 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:43.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:43 smithi003 ceph-mon[103344]: pgmap v175: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:43.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:43 smithi003 ceph-mon[100615]: from='client.44924 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:17:43.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:43 smithi003 ceph-mon[100615]: pgmap v175: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:44.306 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:17:44.345 INFO:teuthology.orchestra.run.smithi003.stdout: "mon": { 2024-08-17T17:17:44.345 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 3 2024-08-17T17:17:44.345 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:17:44.345 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": { 2024-08-17T17:17:44.345 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 2 2024-08-17T17:17:44.345 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:17:44.345 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": { 2024-08-17T17:17:44.345 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)": 4, 2024-08-17T17:17:44.346 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 4 2024-08-17T17:17:44.346 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:17:44.346 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": { 2024-08-17T17:17:44.346 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)": 2 2024-08-17T17:17:44.346 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:17:44.346 INFO:teuthology.orchestra.run.smithi003.stdout: "overall": { 2024-08-17T17:17:44.346 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph version 17.2.7-1404-gf2c39d4d (f2c39d4d5abc084f975be26cc0141421e55727d8) quincy (stable)": 6, 2024-08-17T17:17:44.346 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph version 19.0.0-5836-gcd8bdbeb (cd8bdbeb9fbdab5733aa3aaadc042c965473e453) squid (dev)": 9 2024-08-17T17:17:44.346 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:17:44.346 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:17:44.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:44 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:44.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:44 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:44.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:44 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:44.957 DEBUG:teuthology.parallel:result is None 2024-08-17T17:17:44.957 INFO:teuthology.run_tasks:Running task print... 2024-08-17T17:17:44.967 INFO:teuthology.task.print:**** done upgrading first half of cluster **** 2024-08-17T17:17:44.967 INFO:teuthology.run_tasks:Running task ceph.healthy... 2024-08-17T17:17:44.976 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-08-17T17:17:44.976 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-08-17T17:17:44.976 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph mgr dump --format=json 2024-08-17T17:17:45.249 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:17:45.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:45 smithi053 ceph-mon[89963]: pgmap v176: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:45.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:45 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3096185156' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:17:45.489 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:45 smithi003 ceph-mon[103344]: pgmap v176: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:45.489 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:45 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3096185156' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:17:45.489 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:45 smithi003 ceph-mon[100615]: pgmap v176: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:45.489 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:45 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3096185156' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-08-17T17:17:45.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:17:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:17:45] "GET /metrics HTTP/1.1" 200 33941 "" "Prometheus/2.51.0" 2024-08-17T17:17:47.120 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:17:47.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:47 smithi003 ceph-mon[100615]: pgmap v177: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:47.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:47 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3996451709' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-08-17T17:17:47.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:47 smithi003 ceph-mon[103344]: pgmap v177: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:47.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:47 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3996451709' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-08-17T17:17:47.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:47 smithi053 ceph-mon[89963]: pgmap v177: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:47.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:47 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3996451709' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-08-17T17:17:47.760 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":39,"flags":0,"active_gid":26282,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":969732223},{"type":"v1","addr":"172.21.15.3:6801","nonce":969732223}]},"active_addr":"172.21.15.3:6801/969732223","active_change":"2024-08-17T17:12:37.591219+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":25879,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.3:8443/","prometheus":"http://172.21.15.3:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":90,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":2412048326}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":1819633124}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":3414077177}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":439180576}]}]} 2024-08-17T17:17:47.764 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-08-17T17:17:47.764 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-08-17T17:17:47.764 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-17T17:17:48.037 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:17:48.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:48 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:17:48.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:48 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:17:48.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:48 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:17:48.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:48 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:48.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:48 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:17:48.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:48 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:17:48.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:48 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:17:48.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:48 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:48.488 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:48 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:17:48.488 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:48 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:17:48.488 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:48 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:17:48.488 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:48 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:48.488 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:48 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:17:48.488 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:48 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:17:48.488 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:48 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:17:48.488 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:48 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:48.489 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:48 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:17:48.489 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:48 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:17:48.489 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:48 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:17:48.489 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:48 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:48.489 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:48 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:17:48.489 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:48 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:17:48.489 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:48 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:17:48.489 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:48 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:17:49.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:49 smithi053 ceph-mon[89963]: pgmap v178: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:49.485 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:49 smithi003 ceph-mon[103344]: pgmap v178: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:49.485 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:49 smithi003 ceph-mon[100615]: pgmap v178: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:49.900 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:17:49.900 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":110,"fsid":"2b5b662e-5cba-11ef-bccf-c7b262605968","created":"2024-08-17T17:00:56.372943+0000","modified":"2024-08-17T17:17:32.478620+0000","last_up_change":"2024-08-17T17:17:31.477073+0000","last_in_change":"2024-08-17T17:05:37.289378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-17T17:04:01.140951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-17T17:06:28.452624+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":85,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-08-17T17:07:44.635021+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-08-17T17:07:45.576097+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f9a7d76-f0ec-49fc-9540-2606e94758ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":90,"up_from":94,"up_thru":109,"down_at":91,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":58175680},{"type":"v1","addr":"172.21.15.3:6803","nonce":58175680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":58175680},{"type":"v1","addr":"172.21.15.3:6805","nonce":58175680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":58175680},{"type":"v1","addr":"172.21.15.3:6809","nonce":58175680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":58175680},{"type":"v1","addr":"172.21.15.3:6807","nonce":58175680}]},"public_addr":"172.21.15.3:6803/58175680","cluster_addr":"172.21.15.3:6805/58175680","heartbeat_back_addr":"172.21.15.3:6809/58175680","heartbeat_front_addr":"172.21.15.3:6807/58175680","state":["exists","up"]},{"osd":1,"uuid":"316b5c4e-3417-4b93-bbbb-c6ed6dd21b0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":95,"up_from":99,"up_thru":109,"down_at":96,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":1956079582},{"type":"v1","addr":"172.21.15.3:6811","nonce":1956079582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":1956079582},{"type":"v1","addr":"172.21.15.3:6813","nonce":1956079582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1956079582},{"type":"v1","addr":"172.21.15.3:6817","nonce":1956079582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":1956079582},{"type":"v1","addr":"172.21.15.3:6815","nonce":1956079582}]},"public_addr":"172.21.15.3:6811/1956079582","cluster_addr":"172.21.15.3:6813/1956079582","heartbeat_back_addr":"172.21.15.3:6817/1956079582","heartbeat_front_addr":"172.21.15.3:6815/1956079582","state":["exists","up"]},{"osd":2,"uuid":"88af697d-38db-4c51-bfff-d0a776315596","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":100,"up_from":104,"up_thru":109,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1825512279},{"type":"v1","addr":"172.21.15.3:6819","nonce":1825512279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1825512279},{"type":"v1","addr":"172.21.15.3:6821","nonce":1825512279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1825512279},{"type":"v1","addr":"172.21.15.3:6825","nonce":1825512279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1825512279},{"type":"v1","addr":"172.21.15.3:6823","nonce":1825512279}]},"public_addr":"172.21.15.3:6819/1825512279","cluster_addr":"172.21.15.3:6821/1825512279","heartbeat_back_addr":"172.21.15.3:6825/1825512279","heartbeat_front_addr":"172.21.15.3:6823/1825512279","state":["exists","up"]},{"osd":3,"uuid":"8e20cf42-6a13-4ad5-9f01-203a30c692d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":105,"up_from":109,"up_thru":109,"down_at":106,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":1366347928},{"type":"v1","addr":"172.21.15.3:6827","nonce":1366347928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":1366347928},{"type":"v1","addr":"172.21.15.3:6829","nonce":1366347928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":1366347928},{"type":"v1","addr":"172.21.15.3:6833","nonce":1366347928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":1366347928},{"type":"v1","addr":"172.21.15.3:6831","nonce":1366347928}]},"public_addr":"172.21.15.3:6827/1366347928","cluster_addr":"172.21.15.3:6829/1366347928","heartbeat_back_addr":"172.21.15.3:6833/1366347928","heartbeat_front_addr":"172.21.15.3:6831/1366347928","state":["exists","up"]},{"osd":4,"uuid":"2e8eabd7-8258-4b78-bb21-1a7291e0f1d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6801","nonce":1387588062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6803","nonce":1387588062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6807","nonce":1387588062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6805","nonce":1387588062}]},"public_addr":"172.21.15.53:6801/1387588062","cluster_addr":"172.21.15.53:6803/1387588062","heartbeat_back_addr":"172.21.15.53:6807/1387588062","heartbeat_front_addr":"172.21.15.53:6805/1387588062","state":["exists","up"]},{"osd":5,"uuid":"738f10f7-ff0d-4392-a818-eb23c9f65a63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6809","nonce":1263400356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6811","nonce":1263400356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6815","nonce":1263400356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6813","nonce":1263400356}]},"public_addr":"172.21.15.53:6809/1263400356","cluster_addr":"172.21.15.53:6811/1263400356","heartbeat_back_addr":"172.21.15.53:6815/1263400356","heartbeat_front_addr":"172.21.15.53:6813/1263400356","state":["exists","up"]},{"osd":6,"uuid":"62903bee-59eb-4eed-b64d-9c884de87409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6817","nonce":3729099668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6819","nonce":3729099668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6823","nonce":3729099668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6821","nonce":3729099668}]},"public_addr":"172.21.15.53:6817/3729099668","cluster_addr":"172.21.15.53:6819/3729099668","heartbeat_back_addr":"172.21.15.53:6823/3729099668","heartbeat_front_addr":"172.21.15.53:6821/3729099668","state":["exists","up"]},{"osd":7,"uuid":"4794ff08-e5d9-49c9-bea5-e158681b4004","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6825","nonce":832291503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6827","nonce":832291503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6831","nonce":832291503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6829","nonce":832291503}]},"public_addr":"172.21.15.53:6825/832291503","cluster_addr":"172.21.15.53:6827/832291503","heartbeat_back_addr":"172.21.15.53:6831/832291503","heartbeat_front_addr":"172.21.15.53:6829/832291503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:03:10.759301+0000","dead_epoch":90},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:03:33.981212+0000","dead_epoch":95},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:03:58.516966+0000","dead_epoch":100},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:04:22.965963+0000","dead_epoch":105},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:04:43.836760+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:06.691915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:29.472293+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:52.166848+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:0/2723667418":"2024-08-18T17:12:37.591006+0000","172.21.15.3:6801/2210133652":"2024-08-18T17:12:37.591006+0000","172.21.15.3:6800/2210133652":"2024-08-18T17:12:37.591006+0000","172.21.15.3:0/2965813587":"2024-08-18T17:12:37.591006+0000","172.21.15.3:0/4087675427":"2024-08-18T17:12:37.591006+0000","172.21.15.3:0/2847539733":"2024-08-18T17:12:37.591006+0000","172.21.15.53:0/1967774260":"2024-08-18T17:11:56.179835+0000","172.21.15.3:0/3985128953":"2024-08-18T17:01:50.631242+0000","172.21.15.3:0/1476294447":"2024-08-18T17:01:27.442443+0000","172.21.15.3:0/143641995":"2024-08-18T17:12:37.591006+0000","172.21.15.3:0/601716019":"2024-08-18T17:10:21.996108+0000","172.21.15.3:0/3946755108":"2024-08-18T17:06:25.750593+0000","172.21.15.3:6800/1736600360":"2024-08-18T17:06:25.750593+0000","172.21.15.3:0/3953105554":"2024-08-18T17:06:25.750593+0000","172.21.15.3:6801/2670911286":"2024-08-18T17:10:21.996108+0000","172.21.15.3:0/1583211207":"2024-08-18T17:06:25.750593+0000","172.21.15.3:0/3116588554":"2024-08-18T17:06:25.750593+0000","172.21.15.53:0/1330698653":"2024-08-18T17:11:56.179835+0000","172.21.15.53:6834/4265580715":"2024-08-18T17:11:56.179835+0000","172.21.15.3:0/3817753887":"2024-08-18T17:01:50.631242+0000","172.21.15.3:6801/2187177466":"2024-08-18T17:01:50.631242+0000","172.21.15.3:6800/2670911286":"2024-08-18T17:10:21.996108+0000","172.21.15.3:0/1999667208":"2024-08-18T17:01:50.631242+0000","172.21.15.53:0/721818304":"2024-08-18T17:11:56.179835+0000","172.21.15.3:6801/1736600360":"2024-08-18T17:06:25.750593+0000","172.21.15.3:0/880444026":"2024-08-18T17:10:21.996108+0000","172.21.15.3:0/3717779298":"2024-08-18T17:10:21.996108+0000","172.21.15.3:0/4194950757":"2024-08-18T17:01:27.442443+0000","172.21.15.3:6800/2187177466":"2024-08-18T17:01:50.631242+0000","172.21.15.3:6800/160535497":"2024-08-18T17:01:27.442443+0000","172.21.15.53:6835/4265580715":"2024-08-18T17:11:56.179835+0000","172.21.15.3:0/2029382827":"2024-08-18T17:10:21.996108+0000","172.21.15.53:0/2555682812":"2024-08-18T17:11:56.179835+0000","172.21.15.3:0/1668133120":"2024-08-18T17:01:27.442443+0000","172.21.15.3:6801/160535497":"2024-08-18T17:01:27.442443+0000","172.21.15.3:0/3608486402":"2024-08-18T17:12:37.591006+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-17T17:17:50.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:50 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3849236628' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-17T17:17:50.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:17:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:17:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:17:50.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:50 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3849236628' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-17T17:17:50.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:50 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3849236628' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-17T17:17:50.633 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-08-17T17:17:50.634 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-17T17:17:50.909 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:17:51.354 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:51 smithi003 ceph-mon[103344]: pgmap v179: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:51.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:51 smithi003 ceph-mon[100615]: pgmap v179: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:51.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:51 smithi053 ceph-mon[89963]: pgmap v179: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:52.751 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:17:52.751 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":110,"fsid":"2b5b662e-5cba-11ef-bccf-c7b262605968","created":"2024-08-17T17:00:56.372943+0000","modified":"2024-08-17T17:17:32.478620+0000","last_up_change":"2024-08-17T17:17:31.477073+0000","last_in_change":"2024-08-17T17:05:37.289378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-17T17:04:01.140951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-17T17:06:28.452624+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":85,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-08-17T17:07:44.635021+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-08-17T17:07:45.576097+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f9a7d76-f0ec-49fc-9540-2606e94758ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":90,"up_from":94,"up_thru":109,"down_at":91,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":58175680},{"type":"v1","addr":"172.21.15.3:6803","nonce":58175680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":58175680},{"type":"v1","addr":"172.21.15.3:6805","nonce":58175680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":58175680},{"type":"v1","addr":"172.21.15.3:6809","nonce":58175680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":58175680},{"type":"v1","addr":"172.21.15.3:6807","nonce":58175680}]},"public_addr":"172.21.15.3:6803/58175680","cluster_addr":"172.21.15.3:6805/58175680","heartbeat_back_addr":"172.21.15.3:6809/58175680","heartbeat_front_addr":"172.21.15.3:6807/58175680","state":["exists","up"]},{"osd":1,"uuid":"316b5c4e-3417-4b93-bbbb-c6ed6dd21b0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":95,"up_from":99,"up_thru":109,"down_at":96,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":1956079582},{"type":"v1","addr":"172.21.15.3:6811","nonce":1956079582}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":1956079582},{"type":"v1","addr":"172.21.15.3:6813","nonce":1956079582}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1956079582},{"type":"v1","addr":"172.21.15.3:6817","nonce":1956079582}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":1956079582},{"type":"v1","addr":"172.21.15.3:6815","nonce":1956079582}]},"public_addr":"172.21.15.3:6811/1956079582","cluster_addr":"172.21.15.3:6813/1956079582","heartbeat_back_addr":"172.21.15.3:6817/1956079582","heartbeat_front_addr":"172.21.15.3:6815/1956079582","state":["exists","up"]},{"osd":2,"uuid":"88af697d-38db-4c51-bfff-d0a776315596","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":100,"up_from":104,"up_thru":109,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1825512279},{"type":"v1","addr":"172.21.15.3:6819","nonce":1825512279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1825512279},{"type":"v1","addr":"172.21.15.3:6821","nonce":1825512279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1825512279},{"type":"v1","addr":"172.21.15.3:6825","nonce":1825512279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1825512279},{"type":"v1","addr":"172.21.15.3:6823","nonce":1825512279}]},"public_addr":"172.21.15.3:6819/1825512279","cluster_addr":"172.21.15.3:6821/1825512279","heartbeat_back_addr":"172.21.15.3:6825/1825512279","heartbeat_front_addr":"172.21.15.3:6823/1825512279","state":["exists","up"]},{"osd":3,"uuid":"8e20cf42-6a13-4ad5-9f01-203a30c692d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":105,"up_from":109,"up_thru":109,"down_at":106,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":1366347928},{"type":"v1","addr":"172.21.15.3:6827","nonce":1366347928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":1366347928},{"type":"v1","addr":"172.21.15.3:6829","nonce":1366347928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":1366347928},{"type":"v1","addr":"172.21.15.3:6833","nonce":1366347928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":1366347928},{"type":"v1","addr":"172.21.15.3:6831","nonce":1366347928}]},"public_addr":"172.21.15.3:6827/1366347928","cluster_addr":"172.21.15.3:6829/1366347928","heartbeat_back_addr":"172.21.15.3:6833/1366347928","heartbeat_front_addr":"172.21.15.3:6831/1366347928","state":["exists","up"]},{"osd":4,"uuid":"2e8eabd7-8258-4b78-bb21-1a7291e0f1d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6801","nonce":1387588062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6803","nonce":1387588062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6807","nonce":1387588062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6805","nonce":1387588062}]},"public_addr":"172.21.15.53:6801/1387588062","cluster_addr":"172.21.15.53:6803/1387588062","heartbeat_back_addr":"172.21.15.53:6807/1387588062","heartbeat_front_addr":"172.21.15.53:6805/1387588062","state":["exists","up"]},{"osd":5,"uuid":"738f10f7-ff0d-4392-a818-eb23c9f65a63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6809","nonce":1263400356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6811","nonce":1263400356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6815","nonce":1263400356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6813","nonce":1263400356}]},"public_addr":"172.21.15.53:6809/1263400356","cluster_addr":"172.21.15.53:6811/1263400356","heartbeat_back_addr":"172.21.15.53:6815/1263400356","heartbeat_front_addr":"172.21.15.53:6813/1263400356","state":["exists","up"]},{"osd":6,"uuid":"62903bee-59eb-4eed-b64d-9c884de87409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6817","nonce":3729099668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6819","nonce":3729099668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6823","nonce":3729099668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6821","nonce":3729099668}]},"public_addr":"172.21.15.53:6817/3729099668","cluster_addr":"172.21.15.53:6819/3729099668","heartbeat_back_addr":"172.21.15.53:6823/3729099668","heartbeat_front_addr":"172.21.15.53:6821/3729099668","state":["exists","up"]},{"osd":7,"uuid":"4794ff08-e5d9-49c9-bea5-e158681b4004","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6825","nonce":832291503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6827","nonce":832291503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6831","nonce":832291503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6829","nonce":832291503}]},"public_addr":"172.21.15.53:6825/832291503","cluster_addr":"172.21.15.53:6827/832291503","heartbeat_back_addr":"172.21.15.53:6831/832291503","heartbeat_front_addr":"172.21.15.53:6829/832291503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:03:10.759301+0000","dead_epoch":90},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:03:33.981212+0000","dead_epoch":95},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:03:58.516966+0000","dead_epoch":100},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:04:22.965963+0000","dead_epoch":105},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:04:43.836760+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:06.691915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:29.472293+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:52.166848+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:0/2555682812":"2024-08-18T17:11:56.179835+0000","172.21.15.3:6800/2187177466":"2024-08-18T17:01:50.631242+0000","172.21.15.3:0/3985128953":"2024-08-18T17:01:50.631242+0000","172.21.15.3:0/2723667418":"2024-08-18T17:12:37.591006+0000","172.21.15.53:0/1967774260":"2024-08-18T17:11:56.179835+0000","172.21.15.3:6801/2210133652":"2024-08-18T17:12:37.591006+0000","172.21.15.3:0/1476294447":"2024-08-18T17:01:27.442443+0000","172.21.15.3:6800/2210133652":"2024-08-18T17:12:37.591006+0000","172.21.15.3:0/3946755108":"2024-08-18T17:06:25.750593+0000","172.21.15.3:0/4087675427":"2024-08-18T17:12:37.591006+0000","172.21.15.3:6800/1736600360":"2024-08-18T17:06:25.750593+0000","172.21.15.3:0/3953105554":"2024-08-18T17:06:25.750593+0000","172.21.15.3:0/2847539733":"2024-08-18T17:12:37.591006+0000","172.21.15.3:0/3116588554":"2024-08-18T17:06:25.750593+0000","172.21.15.53:0/721818304":"2024-08-18T17:11:56.179835+0000","172.21.15.3:6801/160535497":"2024-08-18T17:01:27.442443+0000","172.21.15.3:0/3608486402":"2024-08-18T17:12:37.591006+0000","172.21.15.3:0/1668133120":"2024-08-18T17:01:27.442443+0000","172.21.15.3:6801/1736600360":"2024-08-18T17:06:25.750593+0000","172.21.15.3:0/880444026":"2024-08-18T17:10:21.996108+0000","172.21.15.3:0/1999667208":"2024-08-18T17:01:50.631242+0000","172.21.15.3:6800/2670911286":"2024-08-18T17:10:21.996108+0000","172.21.15.3:0/601716019":"2024-08-18T17:10:21.996108+0000","172.21.15.3:0/143641995":"2024-08-18T17:12:37.591006+0000","172.21.15.3:6801/2187177466":"2024-08-18T17:01:50.631242+0000","172.21.15.53:6834/4265580715":"2024-08-18T17:11:56.179835+0000","172.21.15.3:0/3817753887":"2024-08-18T17:01:50.631242+0000","172.21.15.53:0/1330698653":"2024-08-18T17:11:56.179835+0000","172.21.15.3:6801/2670911286":"2024-08-18T17:10:21.996108+0000","172.21.15.3:0/2965813587":"2024-08-18T17:12:37.591006+0000","172.21.15.3:0/1583211207":"2024-08-18T17:06:25.750593+0000","172.21.15.3:0/3717779298":"2024-08-18T17:10:21.996108+0000","172.21.15.3:0/4194950757":"2024-08-18T17:01:27.442443+0000","172.21.15.3:6800/160535497":"2024-08-18T17:01:27.442443+0000","172.21.15.53:6835/4265580715":"2024-08-18T17:11:56.179835+0000","172.21.15.3:0/2029382827":"2024-08-18T17:10:21.996108+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-17T17:17:53.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:53 smithi003 ceph-mon[100615]: pgmap v180: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:53.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:53 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3378910026' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-17T17:17:53.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:17:53.441 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:53 smithi003 ceph-mon[103344]: pgmap v180: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:53.441 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:53 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3378910026' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-17T17:17:53.442 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:17:53.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:53 smithi053 ceph-mon[89963]: pgmap v180: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:53.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:53 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3378910026' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-17T17:17:53.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:17:53.471 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-08-17T17:17:53.471 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-08-17T17:17:53.472 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-08-17T17:17:53.473 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-08-17T17:17:53.473 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-08-17T17:17:53.474 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-08-17T17:17:53.474 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-08-17T17:17:53.475 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-08-17T17:17:54.056 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:17:54.056 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:17:54.056 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:17:54.057 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:17:54.057 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:17:54.057 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:17:54.062 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:17:54.064 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:17:55.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:55 smithi053 ceph-mon[89963]: pgmap v181: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:55.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:55 smithi003 ceph-mon[100615]: pgmap v181: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:55.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:55 smithi003 ceph-mon[103344]: pgmap v181: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:55.921 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:17:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:17:55] "GET /metrics HTTP/1.1" 200 33939 "" "Prometheus/2.51.0" 2024-08-17T17:17:57.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:57 smithi053 ceph-mon[89963]: pgmap v182: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:57.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:57 smithi003 ceph-mon[103344]: pgmap v182: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:57.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:57 smithi003 ceph-mon[100615]: pgmap v182: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:59.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:17:59 smithi053 ceph-mon[89963]: pgmap v183: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:59.478 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:17:59 smithi003 ceph-mon[103344]: pgmap v183: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:17:59.479 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:17:59 smithi003 ceph-mon[100615]: pgmap v183: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:00.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:18:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:18:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:18:01.440 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:01 smithi003 ceph-mon[103344]: pgmap v184: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:01.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:01 smithi003 ceph-mon[100615]: pgmap v184: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:01.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:01 smithi053 ceph-mon[89963]: pgmap v184: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:03.116 INFO:teuthology.orchestra.run.smithi003.stdout:403726925851 2024-08-17T17:18:03.117 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-08-17T17:18:03.338 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:03 smithi003 ceph-mon[100615]: pgmap v185: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:03.338 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:03 smithi003 ceph-mon[103344]: pgmap v185: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:03.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:03 smithi053 ceph-mon[89963]: pgmap v185: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:05.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:05 smithi003 ceph-mon[100615]: pgmap v186: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:05.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:05 smithi003 ceph-mon[103344]: pgmap v186: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:05.437 INFO:teuthology.orchestra.run.smithi003.stdout:446676598798 2024-08-17T17:18:05.438 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-08-17T17:18:05.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:05 smithi053 ceph-mon[89963]: pgmap v186: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:05.973 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:18:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:18:05] "GET /metrics HTTP/1.1" 200 33939 "" "Prometheus/2.51.0" 2024-08-17T17:18:07.362 INFO:teuthology.orchestra.run.smithi003.stdout:197568495770 2024-08-17T17:18:07.363 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-08-17T17:18:07.459 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:07 smithi003 ceph-mon[103344]: pgmap v187: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:07.459 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:07 smithi003 ceph-mon[100615]: pgmap v187: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:07.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:07 smithi053 ceph-mon[89963]: pgmap v187: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:08.148 INFO:teuthology.orchestra.run.smithi003.stdout:425201762326 2024-08-17T17:18:08.148 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-08-17T17:18:08.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:18:08.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:18:08.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:18:09.229 INFO:teuthology.orchestra.run.smithi003.stdout:468151435272 2024-08-17T17:18:09.230 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-08-17T17:18:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:09 smithi053 ceph-mon[89963]: pgmap v188: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:09.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:09 smithi003 ceph-mon[100615]: pgmap v188: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:09 smithi003 ceph-mon[103344]: pgmap v188: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:09.657 INFO:teuthology.orchestra.run.smithi003.stdout:128849019045 2024-08-17T17:18:09.657 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-08-17T17:18:10.447 INFO:teuthology.orchestra.run.smithi003.stdout:231928234136 2024-08-17T17:18:10.448 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-08-17T17:18:10.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:18:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:18:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:18:10.597 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:18:10.597 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:18:10.666 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:18:11.104 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:18:11.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:11 smithi053 ceph-mon[89963]: pgmap v189: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:11.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:11 smithi003 ceph-mon[100615]: pgmap v189: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:11.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:11 smithi003 ceph-mon[103344]: pgmap v189: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:12.539 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:18:13.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:13 smithi053 ceph-mon[89963]: pgmap v190: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:13.496 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:13 smithi003 ceph-mon[100615]: pgmap v190: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:13.496 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:13 smithi003 ceph-mon[103344]: pgmap v190: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:13.689 INFO:teuthology.orchestra.run.smithi003.stdout:163208757408 2024-08-17T17:18:13.690 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-08-17T17:18:13.804 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:18:15.651 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:15 smithi003 ceph-mon[100615]: pgmap v191: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:15.652 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:15 smithi003 ceph-mon[103344]: pgmap v191: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:15.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:15 smithi053 ceph-mon[89963]: pgmap v191: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:16.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:18:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:18:15] "GET /metrics HTTP/1.1" 200 33940 "" "Prometheus/2.51.0" 2024-08-17T17:18:17.594 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:17 smithi003 ceph-mon[103344]: pgmap v192: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:17.595 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:17 smithi003 ceph-mon[100615]: pgmap v192: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:17.661 INFO:teuthology.orchestra.run.smithi003.stdout:446676598801 2024-08-17T17:18:17.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:17 smithi053 ceph-mon[89963]: pgmap v192: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:18.671 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:18 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2324039002' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-08-17T17:18:18.671 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:18 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2324039002' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-08-17T17:18:18.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:18 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2324039002' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-08-17T17:18:18.775 INFO:teuthology.orchestra.run.smithi003.stdout:128849019047 2024-08-17T17:18:19.495 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:19 smithi003 ceph-mon[103344]: pgmap v193: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:19.496 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:19 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3117696061' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-08-17T17:18:19.496 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:19 smithi003 ceph-mon[100615]: pgmap v193: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:19.496 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:19 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3117696061' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-08-17T17:18:19.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:19 smithi053 ceph-mon[89963]: pgmap v193: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:19.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:19 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3117696061' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-08-17T17:18:19.826 INFO:teuthology.orchestra.run.smithi003.stdout:425201762328 2024-08-17T17:18:20.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:20 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2141847655' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-08-17T17:18:20.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:20 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3850852318' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-08-17T17:18:20.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:18:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:18:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:18:20.670 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:20 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2141847655' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-08-17T17:18:20.670 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:20 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3850852318' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-08-17T17:18:20.670 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:20 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2141847655' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-08-17T17:18:20.670 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:20 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3850852318' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-08-17T17:18:20.698 INFO:tasks.cephadm.ceph_manager.ceph:need seq 425201762326 got 425201762328 for osd.1 2024-08-17T17:18:20.698 DEBUG:teuthology.parallel:result is None 2024-08-17T17:18:20.969 INFO:tasks.cephadm.ceph_manager.ceph:need seq 446676598798 got 446676598801 for osd.2 2024-08-17T17:18:20.969 DEBUG:teuthology.parallel:result is None 2024-08-17T17:18:21.055 INFO:teuthology.orchestra.run.smithi003.stdout:197568495772 2024-08-17T17:18:21.593 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:21 smithi003 ceph-mon[100615]: pgmap v194: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:21.593 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:21 smithi003 ceph-mon[103344]: pgmap v194: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:21.624 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495770 got 197568495772 for osd.6 2024-08-17T17:18:21.624 DEBUG:teuthology.parallel:result is None 2024-08-17T17:18:21.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:21 smithi053 ceph-mon[89963]: pgmap v194: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:22.256 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:18:22.256 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:18:22.274 INFO:teuthology.orchestra.run.smithi003.stdout:403726925855 2024-08-17T17:18:22.367 INFO:teuthology.orchestra.run.smithi003.stdout:468151435275 2024-08-17T17:18:22.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:22 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2215991042' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-08-17T17:18:22.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:22 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/4273471621' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-08-17T17:18:22.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:22 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2215991042' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-08-17T17:18:22.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:22 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/4273471621' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-08-17T17:18:22.685 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849019045 got 128849019047 for osd.4 2024-08-17T17:18:22.685 DEBUG:teuthology.parallel:result is None 2024-08-17T17:18:22.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:22 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2215991042' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-08-17T17:18:22.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:22 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/4273471621' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-08-17T17:18:23.600 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:23 smithi003 ceph-mon[103344]: pgmap v195: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:23.600 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:18:23.600 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:23 smithi003 ceph-mon[100615]: pgmap v195: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:23.600 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:18:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:23 smithi053 ceph-mon[89963]: pgmap v195: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:18:25.453 INFO:tasks.cephadm.ceph_manager.ceph:need seq 468151435272 got 468151435275 for osd.3 2024-08-17T17:18:25.454 DEBUG:teuthology.parallel:result is None 2024-08-17T17:18:25.669 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:25 smithi003 ceph-mon[103344]: pgmap v196: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:25.670 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:18:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:18:25] "GET /metrics HTTP/1.1" 200 33939 "" "Prometheus/2.51.0" 2024-08-17T17:18:25.670 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:25 smithi003 ceph-mon[100615]: pgmap v196: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:25.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:25 smithi053 ceph-mon[89963]: pgmap v196: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:26.049 INFO:teuthology.orchestra.run.smithi003.stdout:231928234139 2024-08-17T17:18:26.663 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:26 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2140030879' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-08-17T17:18:26.663 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:26 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2140030879' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-08-17T17:18:26.690 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234136 got 231928234139 for osd.7 2024-08-17T17:18:26.690 DEBUG:teuthology.parallel:result is None 2024-08-17T17:18:26.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:26 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2140030879' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-08-17T17:18:27.097 INFO:tasks.cephadm.ceph_manager.ceph:need seq 403726925851 got 403726925855 for osd.0 2024-08-17T17:18:27.097 DEBUG:teuthology.parallel:result is None 2024-08-17T17:18:27.518 INFO:teuthology.orchestra.run.smithi003.stdout:163208757411 2024-08-17T17:18:27.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:27 smithi053 ceph-mon[89963]: pgmap v197: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:27.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:27 smithi003 ceph-mon[100615]: pgmap v197: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:27.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:27 smithi003 ceph-mon[103344]: pgmap v197: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:28.098 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757408 got 163208757411 for osd.5 2024-08-17T17:18:28.098 DEBUG:teuthology.parallel:result is None 2024-08-17T17:18:28.099 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-08-17T17:18:28.099 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:18:28.368 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:18:28.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:28 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1560911335' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-08-17T17:18:28.723 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:28 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1560911335' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-08-17T17:18:28.723 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:28 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1560911335' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-08-17T17:18:29.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:29 smithi053 ceph-mon[89963]: pgmap v198: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:29.769 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:29 smithi003 ceph-mon[100615]: pgmap v198: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:29.769 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:29 smithi003 ceph-mon[103344]: pgmap v198: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:30.193 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:18:30.193 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:18:30.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:18:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:18:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:18:30.919 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-08-17T17:18:29.688041+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790221,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":6390,"num_read_kb":13845,"num_write":1122,"num_write_kb":15039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1181,"ondisk_log_size":1181,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1343892,"kb_used_data":23560,"kb_used_omap":34,"kb_used_meta":1320157,"kb_avail":748617324,"statfs":{"total":767960285184,"available":766584139776,"internally_reserved":0,"allocated":24125440,"data_stored":19381199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35772,"internal_metadata":1351840836},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002556"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.495711+0000","last_change":"2024-08-17T17:17:32.495652+0000","last_active":"2024-08-17T17:17:32.495711+0000","last_peered":"2024-08-17T17:17:32.495711+0000","last_clean":"2024-08-17T17:17:32.495711+0000","last_became_active":"2024-08-17T17:17:32.495218+0000","last_became_peered":"2024-08-17T17:17:32.495218+0000","last_unstale":"2024-08-17T17:17:32.495711+0000","last_undegraded":"2024-08-17T17:17:32.495711+0000","last_fullsized":"2024-08-17T17:17:32.495711+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:11:45.821282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":70,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.485328+0000","last_change":"2024-08-17T17:17:00.758108+0000","last_active":"2024-08-17T17:17:31.485328+0000","last_peered":"2024-08-17T17:17:31.485328+0000","last_clean":"2024-08-17T17:17:31.485328+0000","last_became_active":"2024-08-17T17:17:00.757714+0000","last_became_peered":"2024-08-17T17:17:00.757714+0000","last_unstale":"2024-08-17T17:17:31.485328+0000","last_undegraded":"2024-08-17T17:17:31.485328+0000","last_fullsized":"2024-08-17T17:17:31.485328+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:13:47.351728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":67,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.495336+0000","last_change":"2024-08-17T17:17:32.495336+0000","last_active":"2024-08-17T17:17:32.495336+0000","last_peered":"2024-08-17T17:17:32.495336+0000","last_clean":"2024-08-17T17:17:32.495336+0000","last_became_active":"2024-08-17T17:17:32.494975+0000","last_became_peered":"2024-08-17T17:17:32.494975+0000","last_unstale":"2024-08-17T17:17:32.495336+0000","last_undegraded":"2024-08-17T17:17:32.495336+0000","last_fullsized":"2024-08-17T17:17:32.495336+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:51:24.050993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.497616+0000","last_change":"2024-08-17T17:17:32.494012+0000","last_active":"2024-08-17T17:17:32.497616+0000","last_peered":"2024-08-17T17:17:32.497616+0000","last_clean":"2024-08-17T17:17:32.497616+0000","last_became_active":"2024-08-17T17:17:32.493663+0000","last_became_peered":"2024-08-17T17:17:32.493663+0000","last_unstale":"2024-08-17T17:17:32.497616+0000","last_undegraded":"2024-08-17T17:17:32.497616+0000","last_fullsized":"2024-08-17T17:17:32.497616+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:05:05.713213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.497986+0000","last_change":"2024-08-17T17:17:32.494930+0000","last_active":"2024-08-17T17:17:32.497986+0000","last_peered":"2024-08-17T17:17:32.497986+0000","last_clean":"2024-08-17T17:17:32.497986+0000","last_became_active":"2024-08-17T17:17:32.494590+0000","last_became_peered":"2024-08-17T17:17:32.494590+0000","last_unstale":"2024-08-17T17:17:32.497986+0000","last_undegraded":"2024-08-17T17:17:32.497986+0000","last_fullsized":"2024-08-17T17:17:32.497986+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:23:16.376415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":51,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.485438+0000","last_change":"2024-08-17T17:07:46.158514+0000","last_active":"2024-08-17T17:17:31.485438+0000","last_peered":"2024-08-17T17:17:31.485438+0000","last_clean":"2024-08-17T17:17:31.485438+0000","last_became_active":"2024-08-17T17:07:46.158256+0000","last_became_peered":"2024-08-17T17:07:46.158256+0000","last_unstale":"2024-08-17T17:17:31.485438+0000","last_undegraded":"2024-08-17T17:17:31.485438+0000","last_fullsized":"2024-08-17T17:17:31.485438+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:30:44.486515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.494098+0000","last_change":"2024-08-17T17:17:32.494030+0000","last_active":"2024-08-17T17:17:32.494098+0000","last_peered":"2024-08-17T17:17:32.494098+0000","last_clean":"2024-08-17T17:17:32.494098+0000","last_became_active":"2024-08-17T17:17:32.493580+0000","last_became_peered":"2024-08-17T17:17:32.493580+0000","last_unstale":"2024-08-17T17:17:32.494098+0000","last_undegraded":"2024-08-17T17:17:32.494098+0000","last_fullsized":"2024-08-17T17:17:32.494098+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:23:14.505301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":76,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.492950+0000","last_change":"2024-08-17T17:15:54.349784+0000","last_active":"2024-08-17T17:17:31.492950+0000","last_peered":"2024-08-17T17:17:31.492950+0000","last_clean":"2024-08-17T17:17:31.492950+0000","last_became_active":"2024-08-17T17:15:54.349305+0000","last_became_peered":"2024-08-17T17:15:54.349305+0000","last_unstale":"2024-08-17T17:17:31.492950+0000","last_undegraded":"2024-08-17T17:17:31.492950+0000","last_fullsized":"2024-08-17T17:17:31.492950+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:36:27.291719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":69,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.486671+0000","last_change":"2024-08-17T17:15:54.358793+0000","last_active":"2024-08-17T17:17:31.486671+0000","last_peered":"2024-08-17T17:17:31.486671+0000","last_clean":"2024-08-17T17:17:31.486671+0000","last_became_active":"2024-08-17T17:15:54.358280+0000","last_became_peered":"2024-08-17T17:15:54.358280+0000","last_unstale":"2024-08-17T17:17:31.486671+0000","last_undegraded":"2024-08-17T17:17:31.486671+0000","last_fullsized":"2024-08-17T17:17:31.486671+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:56:46.173026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":50,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.487824+0000","last_change":"2024-08-17T17:07:46.299427+0000","last_active":"2024-08-17T17:17:31.487824+0000","last_peered":"2024-08-17T17:17:31.487824+0000","last_clean":"2024-08-17T17:17:31.487824+0000","last_became_active":"2024-08-17T17:07:46.299008+0000","last_became_peered":"2024-08-17T17:07:46.299008+0000","last_unstale":"2024-08-17T17:17:31.487824+0000","last_undegraded":"2024-08-17T17:17:31.487824+0000","last_fullsized":"2024-08-17T17:17:31.487824+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:23:32.024874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":70,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.487123+0000","last_change":"2024-08-17T17:16:26.964049+0000","last_active":"2024-08-17T17:17:31.487123+0000","last_peered":"2024-08-17T17:17:31.487123+0000","last_clean":"2024-08-17T17:17:31.487123+0000","last_became_active":"2024-08-17T17:16:26.963554+0000","last_became_peered":"2024-08-17T17:16:26.963554+0000","last_unstale":"2024-08-17T17:17:31.487123+0000","last_undegraded":"2024-08-17T17:17:31.487123+0000","last_fullsized":"2024-08-17T17:17:31.487123+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:39:29.363488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":84,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.494445+0000","last_change":"2024-08-17T17:17:32.494445+0000","last_active":"2024-08-17T17:17:32.494445+0000","last_peered":"2024-08-17T17:17:32.494445+0000","last_clean":"2024-08-17T17:17:32.494445+0000","last_became_active":"2024-08-17T17:17:32.494080+0000","last_became_peered":"2024-08-17T17:17:32.494080+0000","last_unstale":"2024-08-17T17:17:32.494445+0000","last_undegraded":"2024-08-17T17:17:32.494445+0000","last_fullsized":"2024-08-17T17:17:32.494445+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:39:58.938883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":52,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.486546+0000","last_change":"2024-08-17T17:07:46.153923+0000","last_active":"2024-08-17T17:17:31.486546+0000","last_peered":"2024-08-17T17:17:31.486546+0000","last_clean":"2024-08-17T17:17:31.486546+0000","last_became_active":"2024-08-17T17:07:46.153614+0000","last_became_peered":"2024-08-17T17:07:46.153614+0000","last_unstale":"2024-08-17T17:17:31.486546+0000","last_undegraded":"2024-08-17T17:17:31.486546+0000","last_fullsized":"2024-08-17T17:17:31.486546+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:24:38.871907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":67,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.492465+0000","last_change":"2024-08-17T17:16:26.965847+0000","last_active":"2024-08-17T17:17:31.492465+0000","last_peered":"2024-08-17T17:17:31.492465+0000","last_clean":"2024-08-17T17:17:31.492465+0000","last_became_active":"2024-08-17T17:16:26.965522+0000","last_became_peered":"2024-08-17T17:16:26.965522+0000","last_unstale":"2024-08-17T17:17:31.492465+0000","last_undegraded":"2024-08-17T17:17:31.492465+0000","last_fullsized":"2024-08-17T17:17:31.492465+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:39:20.536460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.495428+0000","last_change":"2024-08-17T17:17:32.495428+0000","last_active":"2024-08-17T17:17:32.495428+0000","last_peered":"2024-08-17T17:17:32.495428+0000","last_clean":"2024-08-17T17:17:32.495428+0000","last_became_active":"2024-08-17T17:17:32.494978+0000","last_became_peered":"2024-08-17T17:17:32.494978+0000","last_unstale":"2024-08-17T17:17:32.495428+0000","last_undegraded":"2024-08-17T17:17:32.495428+0000","last_fullsized":"2024-08-17T17:17:32.495428+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:42:24.647812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.484182+0000","last_change":"2024-08-17T17:16:26.961623+0000","last_active":"2024-08-17T17:17:32.484182+0000","last_peered":"2024-08-17T17:17:32.484182+0000","last_clean":"2024-08-17T17:17:32.484182+0000","last_became_active":"2024-08-17T17:16:26.961284+0000","last_became_peered":"2024-08-17T17:16:26.961284+0000","last_unstale":"2024-08-17T17:17:32.484182+0000","last_undegraded":"2024-08-17T17:17:32.484182+0000","last_fullsized":"2024-08-17T17:17:32.484182+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:27:32.176916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.497027+0000","last_change":"2024-08-17T17:17:32.496970+0000","last_active":"2024-08-17T17:17:32.497027+0000","last_peered":"2024-08-17T17:17:32.497027+0000","last_clean":"2024-08-17T17:17:32.497027+0000","last_became_active":"2024-08-17T17:17:32.496610+0000","last_became_peered":"2024-08-17T17:17:32.496610+0000","last_unstale":"2024-08-17T17:17:32.497027+0000","last_undegraded":"2024-08-17T17:17:32.497027+0000","last_fullsized":"2024-08-17T17:17:32.497027+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:03:29.830609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":67,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.494935+0000","last_change":"2024-08-17T17:17:32.494935+0000","last_active":"2024-08-17T17:17:32.494935+0000","last_peered":"2024-08-17T17:17:32.494935+0000","last_clean":"2024-08-17T17:17:32.494935+0000","last_became_active":"2024-08-17T17:17:32.494516+0000","last_became_peered":"2024-08-17T17:17:32.494516+0000","last_unstale":"2024-08-17T17:17:32.494935+0000","last_undegraded":"2024-08-17T17:17:32.494935+0000","last_fullsized":"2024-08-17T17:17:32.494935+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:31:21.680055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":68,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.492670+0000","last_change":"2024-08-17T17:16:26.963141+0000","last_active":"2024-08-17T17:17:31.492670+0000","last_peered":"2024-08-17T17:17:31.492670+0000","last_clean":"2024-08-17T17:17:31.492670+0000","last_became_active":"2024-08-17T17:16:26.962778+0000","last_became_peered":"2024-08-17T17:16:26.962778+0000","last_unstale":"2024-08-17T17:17:31.492670+0000","last_undegraded":"2024-08-17T17:17:31.492670+0000","last_fullsized":"2024-08-17T17:17:31.492670+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:56:40.295525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":84,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.489477+0000","last_change":"2024-08-17T17:17:00.758598+0000","last_active":"2024-08-17T17:17:31.489477+0000","last_peered":"2024-08-17T17:17:31.489477+0000","last_clean":"2024-08-17T17:17:31.489477+0000","last_became_active":"2024-08-17T17:17:00.757663+0000","last_became_peered":"2024-08-17T17:17:00.757663+0000","last_unstale":"2024-08-17T17:17:31.489477+0000","last_undegraded":"2024-08-17T17:17:31.489477+0000","last_fullsized":"2024-08-17T17:17:31.489477+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:16:15.398787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":68,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.490060+0000","last_change":"2024-08-17T17:17:00.758630+0000","last_active":"2024-08-17T17:17:31.490060+0000","last_peered":"2024-08-17T17:17:31.490060+0000","last_clean":"2024-08-17T17:17:31.490060+0000","last_became_active":"2024-08-17T17:17:00.757829+0000","last_became_peered":"2024-08-17T17:17:00.757829+0000","last_unstale":"2024-08-17T17:17:31.490060+0000","last_undegraded":"2024-08-17T17:17:31.490060+0000","last_fullsized":"2024-08-17T17:17:31.490060+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:17:35.583797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":67,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.488838+0000","last_change":"2024-08-17T17:16:26.962625+0000","last_active":"2024-08-17T17:17:31.488838+0000","last_peered":"2024-08-17T17:17:31.488838+0000","last_clean":"2024-08-17T17:17:31.488838+0000","last_became_active":"2024-08-17T17:16:26.962259+0000","last_became_peered":"2024-08-17T17:16:26.962259+0000","last_unstale":"2024-08-17T17:17:31.488838+0000","last_undegraded":"2024-08-17T17:17:31.488838+0000","last_fullsized":"2024-08-17T17:17:31.488838+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:29:34.545710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":72,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.496427+0000","last_change":"2024-08-17T17:17:32.496361+0000","last_active":"2024-08-17T17:17:32.496427+0000","last_peered":"2024-08-17T17:17:32.496427+0000","last_clean":"2024-08-17T17:17:32.496427+0000","last_became_active":"2024-08-17T17:17:32.496056+0000","last_became_peered":"2024-08-17T17:17:32.496056+0000","last_unstale":"2024-08-17T17:17:32.496427+0000","last_undegraded":"2024-08-17T17:17:32.496427+0000","last_fullsized":"2024-08-17T17:17:32.496427+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:11:20.404589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":54,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.495711+0000","last_change":"2024-08-17T17:17:32.495625+0000","last_active":"2024-08-17T17:17:32.495711+0000","last_peered":"2024-08-17T17:17:32.495711+0000","last_clean":"2024-08-17T17:17:32.495711+0000","last_became_active":"2024-08-17T17:17:32.495298+0000","last_became_peered":"2024-08-17T17:17:32.495298+0000","last_unstale":"2024-08-17T17:17:32.495711+0000","last_undegraded":"2024-08-17T17:17:32.495711+0000","last_fullsized":"2024-08-17T17:17:32.495711+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:18:34.358254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":410,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.486090+0000","last_change":"2024-08-17T17:15:54.354936+0000","last_active":"2024-08-17T17:17:31.486090+0000","last_peered":"2024-08-17T17:17:31.486090+0000","last_clean":"2024-08-17T17:17:31.486090+0000","last_became_active":"2024-08-17T17:15:54.354238+0000","last_became_peered":"2024-08-17T17:15:54.354238+0000","last_unstale":"2024-08-17T17:17:31.486090+0000","last_undegraded":"2024-08-17T17:17:31.486090+0000","last_fullsized":"2024-08-17T17:17:31.486090+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:04:01.141401+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:58:05.159624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":602,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.488040+0000","last_change":"2024-08-17T17:17:00.759124+0000","last_active":"2024-08-17T17:17:31.488040+0000","last_peered":"2024-08-17T17:17:31.488040+0000","last_clean":"2024-08-17T17:17:31.488040+0000","last_became_active":"2024-08-17T17:17:00.758605+0000","last_became_peered":"2024-08-17T17:17:00.758605+0000","last_unstale":"2024-08-17T17:17:31.488040+0000","last_undegraded":"2024-08-17T17:17:31.488040+0000","last_fullsized":"2024-08-17T17:17:31.488040+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:05:10.050836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.500056+0000","last_change":"2024-08-17T17:17:32.499994+0000","last_active":"2024-08-17T17:17:32.500056+0000","last_peered":"2024-08-17T17:17:32.500056+0000","last_clean":"2024-08-17T17:17:32.500056+0000","last_became_active":"2024-08-17T17:17:32.499614+0000","last_became_peered":"2024-08-17T17:17:32.499614+0000","last_unstale":"2024-08-17T17:17:32.500056+0000","last_undegraded":"2024-08-17T17:17:32.500056+0000","last_fullsized":"2024-08-17T17:17:32.500056+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:18:05.307880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":50,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.493323+0000","last_change":"2024-08-17T17:07:47.164375+0000","last_active":"2024-08-17T17:17:31.493323+0000","last_peered":"2024-08-17T17:17:31.493323+0000","last_clean":"2024-08-17T17:17:31.493323+0000","last_became_active":"2024-08-17T17:07:47.164077+0000","last_became_peered":"2024-08-17T17:07:47.164077+0000","last_unstale":"2024-08-17T17:17:31.493323+0000","last_undegraded":"2024-08-17T17:17:31.493323+0000","last_fullsized":"2024-08-17T17:17:31.493323+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:26:01.278743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":700,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.485964+0000","last_change":"2024-08-17T17:16:26.958415+0000","last_active":"2024-08-17T17:17:31.485964+0000","last_peered":"2024-08-17T17:17:31.485964+0000","last_clean":"2024-08-17T17:17:31.485964+0000","last_became_active":"2024-08-17T17:16:26.957956+0000","last_became_peered":"2024-08-17T17:16:26.957956+0000","last_unstale":"2024-08-17T17:17:31.485964+0000","last_undegraded":"2024-08-17T17:17:31.485964+0000","last_fullsized":"2024-08-17T17:17:31.485964+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:22.478077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.499100+0000","last_change":"2024-08-17T17:17:32.493840+0000","last_active":"2024-08-17T17:17:32.499100+0000","last_peered":"2024-08-17T17:17:32.499100+0000","last_clean":"2024-08-17T17:17:32.499100+0000","last_became_active":"2024-08-17T17:17:32.493503+0000","last_became_peered":"2024-08-17T17:17:32.493503+0000","last_unstale":"2024-08-17T17:17:32.499100+0000","last_undegraded":"2024-08-17T17:17:32.499100+0000","last_fullsized":"2024-08-17T17:17:32.499100+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:01:06.908354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":79,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.484970+0000","last_change":"2024-08-17T17:17:00.761104+0000","last_active":"2024-08-17T17:17:32.484970+0000","last_peered":"2024-08-17T17:17:32.484970+0000","last_clean":"2024-08-17T17:17:32.484970+0000","last_became_active":"2024-08-17T17:17:00.760721+0000","last_became_peered":"2024-08-17T17:17:00.760721+0000","last_unstale":"2024-08-17T17:17:32.484970+0000","last_undegraded":"2024-08-17T17:17:32.484970+0000","last_fullsized":"2024-08-17T17:17:32.484970+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:06:49.900957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":555,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.495429+0000","last_change":"2024-08-17T17:17:32.495364+0000","last_active":"2024-08-17T17:17:32.495429+0000","last_peered":"2024-08-17T17:17:32.495429+0000","last_clean":"2024-08-17T17:17:32.495429+0000","last_became_active":"2024-08-17T17:17:32.494801+0000","last_became_peered":"2024-08-17T17:17:32.494801+0000","last_unstale":"2024-08-17T17:17:32.495429+0000","last_undegraded":"2024-08-17T17:17:32.495429+0000","last_fullsized":"2024-08-17T17:17:32.495429+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:11:06.683576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":55,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.488438+0000","last_change":"2024-08-17T17:17:00.758552+0000","last_active":"2024-08-17T17:17:32.488438+0000","last_peered":"2024-08-17T17:17:32.488438+0000","last_clean":"2024-08-17T17:17:32.488438+0000","last_became_active":"2024-08-17T17:17:00.758200+0000","last_became_peered":"2024-08-17T17:17:00.758200+0000","last_unstale":"2024-08-17T17:17:32.488438+0000","last_undegraded":"2024-08-17T17:17:32.488438+0000","last_fullsized":"2024-08-17T17:17:32.488438+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:13:27.648231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":85,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.488032+0000","last_change":"2024-08-17T17:16:26.964140+0000","last_active":"2024-08-17T17:17:32.488032+0000","last_peered":"2024-08-17T17:17:32.488032+0000","last_clean":"2024-08-17T17:17:32.488032+0000","last_became_active":"2024-08-17T17:16:26.963807+0000","last_became_peered":"2024-08-17T17:16:26.963807+0000","last_unstale":"2024-08-17T17:17:32.488032+0000","last_undegraded":"2024-08-17T17:17:32.488032+0000","last_fullsized":"2024-08-17T17:17:32.488032+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:30:50.386236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":936,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.487871+0000","last_change":"2024-08-17T17:16:26.965149+0000","last_active":"2024-08-17T17:17:31.487871+0000","last_peered":"2024-08-17T17:17:31.487871+0000","last_clean":"2024-08-17T17:17:31.487871+0000","last_became_active":"2024-08-17T17:16:26.964674+0000","last_became_peered":"2024-08-17T17:16:26.964674+0000","last_unstale":"2024-08-17T17:17:31.487871+0000","last_undegraded":"2024-08-17T17:17:31.487871+0000","last_fullsized":"2024-08-17T17:17:31.487871+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:28:57.549403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1392,"num_read_kb":2863,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":68,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.490674+0000","last_change":"2024-08-17T17:15:54.359640+0000","last_active":"2024-08-17T17:17:31.490674+0000","last_peered":"2024-08-17T17:17:31.490674+0000","last_clean":"2024-08-17T17:17:31.490674+0000","last_became_active":"2024-08-17T17:15:54.359163+0000","last_became_peered":"2024-08-17T17:15:54.359163+0000","last_unstale":"2024-08-17T17:17:31.490674+0000","last_undegraded":"2024-08-17T17:17:31.490674+0000","last_fullsized":"2024-08-17T17:17:31.490674+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:13:52.163231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.500703+0000","last_change":"2024-08-17T17:17:32.499715+0000","last_active":"2024-08-17T17:17:32.500703+0000","last_peered":"2024-08-17T17:17:32.500703+0000","last_clean":"2024-08-17T17:17:32.500703+0000","last_became_active":"2024-08-17T17:17:32.499412+0000","last_became_peered":"2024-08-17T17:17:32.499412+0000","last_unstale":"2024-08-17T17:17:32.500703+0000","last_undegraded":"2024-08-17T17:17:32.500703+0000","last_fullsized":"2024-08-17T17:17:32.500703+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:39:38.456970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":85,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.495728+0000","last_change":"2024-08-17T17:17:32.495728+0000","last_active":"2024-08-17T17:17:32.495728+0000","last_peered":"2024-08-17T17:17:32.495728+0000","last_clean":"2024-08-17T17:17:32.495728+0000","last_became_active":"2024-08-17T17:17:32.495357+0000","last_became_peered":"2024-08-17T17:17:32.495357+0000","last_unstale":"2024-08-17T17:17:32.495728+0000","last_undegraded":"2024-08-17T17:17:32.495728+0000","last_fullsized":"2024-08-17T17:17:32.495728+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:27:45.561567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":302,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.488216+0000","last_change":"2024-08-17T17:16:26.964621+0000","last_active":"2024-08-17T17:17:32.488216+0000","last_peered":"2024-08-17T17:17:32.488216+0000","last_clean":"2024-08-17T17:17:32.488216+0000","last_became_active":"2024-08-17T17:16:26.964058+0000","last_became_peered":"2024-08-17T17:16:26.964058+0000","last_unstale":"2024-08-17T17:17:32.488216+0000","last_undegraded":"2024-08-17T17:17:32.488216+0000","last_fullsized":"2024-08-17T17:17:32.488216+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:00:04.465644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":44,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.488723+0000","last_change":"2024-08-17T17:16:26.965872+0000","last_active":"2024-08-17T17:17:32.488723+0000","last_peered":"2024-08-17T17:17:32.488723+0000","last_clean":"2024-08-17T17:17:32.488723+0000","last_became_active":"2024-08-17T17:16:26.965580+0000","last_became_peered":"2024-08-17T17:16:26.965580+0000","last_unstale":"2024-08-17T17:17:32.488723+0000","last_undegraded":"2024-08-17T17:17:32.488723+0000","last_fullsized":"2024-08-17T17:17:32.488723+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:21:52.311284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":55,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.488106+0000","last_change":"2024-08-17T17:17:00.758843+0000","last_active":"2024-08-17T17:17:32.488106+0000","last_peered":"2024-08-17T17:17:32.488106+0000","last_clean":"2024-08-17T17:17:32.488106+0000","last_became_active":"2024-08-17T17:17:00.758350+0000","last_became_peered":"2024-08-17T17:17:00.758350+0000","last_unstale":"2024-08-17T17:17:32.488106+0000","last_undegraded":"2024-08-17T17:17:32.488106+0000","last_fullsized":"2024-08-17T17:17:32.488106+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:36:47.412787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":411,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.487103+0000","last_change":"2024-08-17T17:15:54.358998+0000","last_active":"2024-08-17T17:17:31.487103+0000","last_peered":"2024-08-17T17:17:31.487103+0000","last_clean":"2024-08-17T17:17:31.487103+0000","last_became_active":"2024-08-17T17:15:54.358141+0000","last_became_peered":"2024-08-17T17:15:54.358141+0000","last_unstale":"2024-08-17T17:17:31.487103+0000","last_undegraded":"2024-08-17T17:17:31.487103+0000","last_fullsized":"2024-08-17T17:17:31.487103+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:32:08.078428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":55,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.484086+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:17:32.484086+0000","last_peered":"2024-08-17T17:17:32.484086+0000","last_clean":"2024-08-17T17:17:32.484086+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:17:32.484086+0000","last_undegraded":"2024-08-17T17:17:32.484086+0000","last_fullsized":"2024-08-17T17:17:32.484086+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.496087+0000","last_change":"2024-08-17T17:17:32.496023+0000","last_active":"2024-08-17T17:17:32.496087+0000","last_peered":"2024-08-17T17:17:32.496087+0000","last_clean":"2024-08-17T17:17:32.496087+0000","last_became_active":"2024-08-17T17:17:32.495652+0000","last_became_peered":"2024-08-17T17:17:32.495652+0000","last_unstale":"2024-08-17T17:17:32.496087+0000","last_undegraded":"2024-08-17T17:17:32.496087+0000","last_fullsized":"2024-08-17T17:17:32.496087+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:42:52.056527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":564,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.488044+0000","last_change":"2024-08-17T17:16:26.967134+0000","last_active":"2024-08-17T17:17:32.488044+0000","last_peered":"2024-08-17T17:17:32.488044+0000","last_clean":"2024-08-17T17:17:32.488044+0000","last_became_active":"2024-08-17T17:16:26.966517+0000","last_became_peered":"2024-08-17T17:16:26.966517+0000","last_unstale":"2024-08-17T17:17:32.488044+0000","last_undegraded":"2024-08-17T17:17:32.488044+0000","last_fullsized":"2024-08-17T17:17:32.488044+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:38:52.366564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.501049+0000","last_change":"2024-08-17T17:17:32.500970+0000","last_active":"2024-08-17T17:17:32.501049+0000","last_peered":"2024-08-17T17:17:32.501049+0000","last_clean":"2024-08-17T17:17:32.501049+0000","last_became_active":"2024-08-17T17:17:32.499033+0000","last_became_peered":"2024-08-17T17:17:32.499033+0000","last_unstale":"2024-08-17T17:17:32.501049+0000","last_undegraded":"2024-08-17T17:17:32.501049+0000","last_fullsized":"2024-08-17T17:17:32.501049+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:26:01.543899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.484380+0000","last_change":"2024-08-17T17:16:26.962885+0000","last_active":"2024-08-17T17:17:32.484380+0000","last_peered":"2024-08-17T17:17:32.484380+0000","last_clean":"2024-08-17T17:17:32.484380+0000","last_became_active":"2024-08-17T17:16:26.961919+0000","last_became_peered":"2024-08-17T17:16:26.961919+0000","last_unstale":"2024-08-17T17:17:32.484380+0000","last_undegraded":"2024-08-17T17:17:32.484380+0000","last_fullsized":"2024-08-17T17:17:32.484380+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:03:43.614359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":291,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.492024+0000","last_change":"2024-08-17T17:17:00.755772+0000","last_active":"2024-08-17T17:17:31.492024+0000","last_peered":"2024-08-17T17:17:31.492024+0000","last_clean":"2024-08-17T17:17:31.492024+0000","last_became_active":"2024-08-17T17:17:00.755281+0000","last_became_peered":"2024-08-17T17:17:00.755281+0000","last_unstale":"2024-08-17T17:17:31.492024+0000","last_undegraded":"2024-08-17T17:17:31.492024+0000","last_fullsized":"2024-08-17T17:17:31.492024+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:56:54.600755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":50,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.488953+0000","last_change":"2024-08-17T17:07:47.227287+0000","last_active":"2024-08-17T17:17:31.488953+0000","last_peered":"2024-08-17T17:17:31.488953+0000","last_clean":"2024-08-17T17:17:31.488953+0000","last_became_active":"2024-08-17T17:07:47.226983+0000","last_became_peered":"2024-08-17T17:07:47.226983+0000","last_unstale":"2024-08-17T17:17:31.488953+0000","last_undegraded":"2024-08-17T17:17:31.488953+0000","last_fullsized":"2024-08-17T17:17:31.488953+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:34:07.444142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":67,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.490105+0000","last_change":"2024-08-17T17:16:26.961292+0000","last_active":"2024-08-17T17:17:31.490105+0000","last_peered":"2024-08-17T17:17:31.490105+0000","last_clean":"2024-08-17T17:17:31.490105+0000","last_became_active":"2024-08-17T17:16:26.960977+0000","last_became_peered":"2024-08-17T17:16:26.960977+0000","last_unstale":"2024-08-17T17:17:31.490105+0000","last_undegraded":"2024-08-17T17:17:31.490105+0000","last_fullsized":"2024-08-17T17:17:31.490105+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:58:50.075227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":73,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.489768+0000","last_change":"2024-08-17T17:07:46.160694+0000","last_active":"2024-08-17T17:17:31.489768+0000","last_peered":"2024-08-17T17:17:31.489768+0000","last_clean":"2024-08-17T17:17:31.489768+0000","last_became_active":"2024-08-17T17:07:46.160407+0000","last_became_peered":"2024-08-17T17:07:46.160407+0000","last_unstale":"2024-08-17T17:17:31.489768+0000","last_undegraded":"2024-08-17T17:17:31.489768+0000","last_fullsized":"2024-08-17T17:17:31.489768+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:54:38.366094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":54,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.487584+0000","last_change":"2024-08-17T17:17:00.756909+0000","last_active":"2024-08-17T17:17:32.487584+0000","last_peered":"2024-08-17T17:17:32.487584+0000","last_clean":"2024-08-17T17:17:32.487584+0000","last_became_active":"2024-08-17T17:17:00.756532+0000","last_became_peered":"2024-08-17T17:17:00.756532+0000","last_unstale":"2024-08-17T17:17:32.487584+0000","last_undegraded":"2024-08-17T17:17:32.487584+0000","last_fullsized":"2024-08-17T17:17:32.487584+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:11:25.294907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.496800+0000","last_change":"2024-08-17T17:17:32.496800+0000","last_active":"2024-08-17T17:17:32.496800+0000","last_peered":"2024-08-17T17:17:32.496800+0000","last_clean":"2024-08-17T17:17:32.496800+0000","last_became_active":"2024-08-17T17:17:32.496405+0000","last_became_peered":"2024-08-17T17:17:32.496405+0000","last_unstale":"2024-08-17T17:17:32.496800+0000","last_undegraded":"2024-08-17T17:17:32.496800+0000","last_fullsized":"2024-08-17T17:17:32.496800+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:03:56.618806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":68,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.489488+0000","last_change":"2024-08-17T17:16:26.964939+0000","last_active":"2024-08-17T17:17:31.489488+0000","last_peered":"2024-08-17T17:17:31.489488+0000","last_clean":"2024-08-17T17:17:31.489488+0000","last_became_active":"2024-08-17T17:16:26.964598+0000","last_became_peered":"2024-08-17T17:16:26.964598+0000","last_unstale":"2024-08-17T17:17:31.489488+0000","last_undegraded":"2024-08-17T17:17:31.489488+0000","last_fullsized":"2024-08-17T17:17:31.489488+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:01:09.949950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":24,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.501097+0000","last_change":"2024-08-17T17:17:32.501043+0000","last_active":"2024-08-17T17:17:32.501097+0000","last_peered":"2024-08-17T17:17:32.501097+0000","last_clean":"2024-08-17T17:17:32.501097+0000","last_became_active":"2024-08-17T17:17:32.499400+0000","last_became_peered":"2024-08-17T17:17:32.499400+0000","last_unstale":"2024-08-17T17:17:32.501097+0000","last_undegraded":"2024-08-17T17:17:32.501097+0000","last_fullsized":"2024-08-17T17:17:32.501097+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:47:35.040523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.496833+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:17:32.496833+0000","last_peered":"2024-08-17T17:17:32.496833+0000","last_clean":"2024-08-17T17:17:32.496833+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:17:32.496833+0000","last_undegraded":"2024-08-17T17:17:32.496833+0000","last_fullsized":"2024-08-17T17:17:32.496833+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.501087+0000","last_change":"2024-08-17T17:17:32.501012+0000","last_active":"2024-08-17T17:17:32.501087+0000","last_peered":"2024-08-17T17:17:32.501087+0000","last_clean":"2024-08-17T17:17:32.501087+0000","last_became_active":"2024-08-17T17:17:32.499399+0000","last_became_peered":"2024-08-17T17:17:32.499399+0000","last_unstale":"2024-08-17T17:17:32.501087+0000","last_undegraded":"2024-08-17T17:17:32.501087+0000","last_fullsized":"2024-08-17T17:17:32.501087+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:19:57.035025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":67,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.491928+0000","last_change":"2024-08-17T17:16:26.963696+0000","last_active":"2024-08-17T17:17:31.491928+0000","last_peered":"2024-08-17T17:17:31.491928+0000","last_clean":"2024-08-17T17:17:31.491928+0000","last_became_active":"2024-08-17T17:16:26.963280+0000","last_became_peered":"2024-08-17T17:16:26.963280+0000","last_unstale":"2024-08-17T17:17:31.491928+0000","last_undegraded":"2024-08-17T17:17:31.491928+0000","last_fullsized":"2024-08-17T17:17:31.491928+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:22:01.474840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.500549+0000","last_change":"2024-08-17T17:17:32.500494+0000","last_active":"2024-08-17T17:17:32.500549+0000","last_peered":"2024-08-17T17:17:32.500549+0000","last_clean":"2024-08-17T17:17:32.500549+0000","last_became_active":"2024-08-17T17:17:32.498949+0000","last_became_peered":"2024-08-17T17:17:32.498949+0000","last_unstale":"2024-08-17T17:17:32.500549+0000","last_undegraded":"2024-08-17T17:17:32.500549+0000","last_fullsized":"2024-08-17T17:17:32.500549+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:44.681212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.498006+0000","last_change":"2024-08-17T17:17:32.495908+0000","last_active":"2024-08-17T17:17:32.498006+0000","last_peered":"2024-08-17T17:17:32.498006+0000","last_clean":"2024-08-17T17:17:32.498006+0000","last_became_active":"2024-08-17T17:17:32.495593+0000","last_became_peered":"2024-08-17T17:17:32.495593+0000","last_unstale":"2024-08-17T17:17:32.498006+0000","last_undegraded":"2024-08-17T17:17:32.498006+0000","last_fullsized":"2024-08-17T17:17:32.498006+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:41:39.009044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":55,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.498046+0000","last_change":"2024-08-17T17:17:32.497986+0000","last_active":"2024-08-17T17:17:32.498046+0000","last_peered":"2024-08-17T17:17:32.498046+0000","last_clean":"2024-08-17T17:17:32.498046+0000","last_became_active":"2024-08-17T17:17:32.497639+0000","last_became_peered":"2024-08-17T17:17:32.497639+0000","last_unstale":"2024-08-17T17:17:32.498046+0000","last_undegraded":"2024-08-17T17:17:32.498046+0000","last_fullsized":"2024-08-17T17:17:32.498046+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:03:25.854983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":85,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.495810+0000","last_change":"2024-08-17T17:17:32.495727+0000","last_active":"2024-08-17T17:17:32.495810+0000","last_peered":"2024-08-17T17:17:32.495810+0000","last_clean":"2024-08-17T17:17:32.495810+0000","last_became_active":"2024-08-17T17:17:32.494437+0000","last_became_peered":"2024-08-17T17:17:32.494437+0000","last_unstale":"2024-08-17T17:17:32.495810+0000","last_undegraded":"2024-08-17T17:17:32.495810+0000","last_fullsized":"2024-08-17T17:17:32.495810+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:56:22.902797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":86,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.489170+0000","last_change":"2024-08-17T17:17:00.758212+0000","last_active":"2024-08-17T17:17:31.489170+0000","last_peered":"2024-08-17T17:17:31.489170+0000","last_clean":"2024-08-17T17:17:31.489170+0000","last_became_active":"2024-08-17T17:17:00.757787+0000","last_became_peered":"2024-08-17T17:17:00.757787+0000","last_unstale":"2024-08-17T17:17:31.489170+0000","last_undegraded":"2024-08-17T17:17:31.489170+0000","last_fullsized":"2024-08-17T17:17:31.489170+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:19:11.289350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.495253+0000","last_change":"2024-08-17T17:17:32.495192+0000","last_active":"2024-08-17T17:17:32.495253+0000","last_peered":"2024-08-17T17:17:32.495253+0000","last_clean":"2024-08-17T17:17:32.495253+0000","last_became_active":"2024-08-17T17:17:32.494866+0000","last_became_peered":"2024-08-17T17:17:32.494866+0000","last_unstale":"2024-08-17T17:17:32.495253+0000","last_undegraded":"2024-08-17T17:17:32.495253+0000","last_fullsized":"2024-08-17T17:17:32.495253+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:21:13.274144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":58,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.483940+0000","last_change":"2024-08-17T17:15:54.357458+0000","last_active":"2024-08-17T17:17:32.483940+0000","last_peered":"2024-08-17T17:17:32.483940+0000","last_clean":"2024-08-17T17:17:32.483940+0000","last_became_active":"2024-08-17T17:15:54.356621+0000","last_became_peered":"2024-08-17T17:15:54.356621+0000","last_unstale":"2024-08-17T17:17:32.483940+0000","last_undegraded":"2024-08-17T17:17:32.483940+0000","last_fullsized":"2024-08-17T17:17:32.483940+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:16:10.254328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":84,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.493417+0000","last_change":"2024-08-17T17:17:32.493417+0000","last_active":"2024-08-17T17:17:32.493417+0000","last_peered":"2024-08-17T17:17:32.493417+0000","last_clean":"2024-08-17T17:17:32.493417+0000","last_became_active":"2024-08-17T17:17:32.493097+0000","last_became_peered":"2024-08-17T17:17:32.493097+0000","last_unstale":"2024-08-17T17:17:32.493417+0000","last_undegraded":"2024-08-17T17:17:32.493417+0000","last_fullsized":"2024-08-17T17:17:32.493417+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:22:01.622261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":70,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.488375+0000","last_change":"2024-08-17T17:16:26.964339+0000","last_active":"2024-08-17T17:17:31.488375+0000","last_peered":"2024-08-17T17:17:31.488375+0000","last_clean":"2024-08-17T17:17:31.488375+0000","last_became_active":"2024-08-17T17:16:26.963964+0000","last_became_peered":"2024-08-17T17:16:26.963964+0000","last_unstale":"2024-08-17T17:17:31.488375+0000","last_undegraded":"2024-08-17T17:17:31.488375+0000","last_fullsized":"2024-08-17T17:17:31.488375+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:24:14.417213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":84,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.492899+0000","last_change":"2024-08-17T17:17:32.492899+0000","last_active":"2024-08-17T17:17:32.492899+0000","last_peered":"2024-08-17T17:17:32.492899+0000","last_clean":"2024-08-17T17:17:32.492899+0000","last_became_active":"2024-08-17T17:17:32.492522+0000","last_became_peered":"2024-08-17T17:17:32.492522+0000","last_unstale":"2024-08-17T17:17:32.492899+0000","last_undegraded":"2024-08-17T17:17:32.492899+0000","last_fullsized":"2024-08-17T17:17:32.492899+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:27:57.515343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":53,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.490249+0000","last_change":"2024-08-17T17:07:46.156750+0000","last_active":"2024-08-17T17:17:31.490249+0000","last_peered":"2024-08-17T17:17:31.490249+0000","last_clean":"2024-08-17T17:17:31.490249+0000","last_became_active":"2024-08-17T17:07:46.156253+0000","last_became_peered":"2024-08-17T17:07:46.156253+0000","last_unstale":"2024-08-17T17:17:31.490249+0000","last_undegraded":"2024-08-17T17:17:31.490249+0000","last_fullsized":"2024-08-17T17:17:31.490249+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:04:28.549130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":92,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.499059+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:17:32.499059+0000","last_peered":"2024-08-17T17:17:32.499059+0000","last_clean":"2024-08-17T17:17:32.499059+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:17:32.499059+0000","last_undegraded":"2024-08-17T17:17:32.499059+0000","last_fullsized":"2024-08-17T17:17:32.499059+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":77,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.500444+0000","last_change":"2024-08-17T17:17:32.500387+0000","last_active":"2024-08-17T17:17:32.500444+0000","last_peered":"2024-08-17T17:17:32.500444+0000","last_clean":"2024-08-17T17:17:32.500444+0000","last_became_active":"2024-08-17T17:17:32.500062+0000","last_became_peered":"2024-08-17T17:17:32.500062+0000","last_unstale":"2024-08-17T17:17:32.500444+0000","last_undegraded":"2024-08-17T17:17:32.500444+0000","last_fullsized":"2024-08-17T17:17:32.500444+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:12:25.410392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":77,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.488249+0000","last_change":"2024-08-17T17:17:00.758339+0000","last_active":"2024-08-17T17:17:31.488249+0000","last_peered":"2024-08-17T17:17:31.488249+0000","last_clean":"2024-08-17T17:17:31.488249+0000","last_became_active":"2024-08-17T17:17:00.757987+0000","last_became_peered":"2024-08-17T17:17:00.757987+0000","last_unstale":"2024-08-17T17:17:31.488249+0000","last_undegraded":"2024-08-17T17:17:31.488249+0000","last_fullsized":"2024-08-17T17:17:31.488249+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:55:40.761984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.484386+0000","last_change":"2024-08-17T17:17:00.760208+0000","last_active":"2024-08-17T17:17:32.484386+0000","last_peered":"2024-08-17T17:17:32.484386+0000","last_clean":"2024-08-17T17:17:32.484386+0000","last_became_active":"2024-08-17T17:17:00.759800+0000","last_became_peered":"2024-08-17T17:17:00.759800+0000","last_unstale":"2024-08-17T17:17:32.484386+0000","last_undegraded":"2024-08-17T17:17:32.484386+0000","last_fullsized":"2024-08-17T17:17:32.484386+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:01:36.990816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17971,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234140,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307404,"kb_used_data":5880,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93437748,"statfs":{"total":95995035648,"available":95680253952,"internally_reserved":0,"allocated":6021120,"data_stored":5415758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495775,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307388,"kb_used_data":5868,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93437764,"statfs":{"total":95995035648,"available":95680270336,"internally_reserved":0,"allocated":6008832,"data_stored":5415424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757412,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298552,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446600,"statfs":{"total":95995035648,"available":95689318400,"internally_reserved":0,"allocated":1216512,"data_stored":626785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019050,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302720,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93442432,"statfs":{"total":95995035648,"available":95685050368,"internally_reserved":0,"allocated":1232896,"data_stored":631339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435277,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28660,"kb_used_data":1184,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":93716492,"statfs":{"total":95995035648,"available":95965687808,"internally_reserved":0,"allocated":1212416,"data_stored":626559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":28109834},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676598804,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36980,"kb_used_data":5284,"kb_used_omap":7,"kb_used_meta":31672,"kb_avail":93708172,"statfs":{"total":95995035648,"available":95957168128,"internally_reserved":0,"allocated":5410816,"data_stored":4821385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":32432290},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762331,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32884,"kb_used_data":1184,"kb_used_omap":7,"kb_used_meta":31672,"kb_avail":93712268,"statfs":{"total":95995035648,"available":95961362432,"internally_reserved":0,"allocated":1212416,"data_stored":626504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7945,"internal_metadata":32432375},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726925857,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":29304,"kb_used_data":1768,"kb_used_omap":7,"kb_used_meta":27512,"kb_avail":93715848,"statfs":{"total":95995035648,"available":95965028352,"internally_reserved":0,"allocated":1810432,"data_stored":1217445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":28172609},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6241,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:18:30.923 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:18:31.188 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:18:31.561 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:31 smithi003 ceph-mon[103344]: pgmap v199: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:31.561 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:31 smithi003 ceph-mon[103344]: from='client.45050 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:18:31.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:31 smithi003 ceph-mon[100615]: pgmap v199: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:31.562 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:31 smithi003 ceph-mon[100615]: from='client.45050 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:18:31.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:31 smithi053 ceph-mon[89963]: pgmap v199: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:31.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:31 smithi053 ceph-mon[89963]: from='client.45050 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:18:33.011 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:18:33.012 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:18:33.599 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:33 smithi003 ceph-mon[103344]: pgmap v200: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:33.599 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:33 smithi003 ceph-mon[100615]: pgmap v200: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:33.626 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-08-17T17:18:31.688555+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790221,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":6390,"num_read_kb":13845,"num_write":1122,"num_write_kb":15039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1181,"ondisk_log_size":1181,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1343892,"kb_used_data":23560,"kb_used_omap":34,"kb_used_meta":1320157,"kb_avail":748617324,"statfs":{"total":767960285184,"available":766584139776,"internally_reserved":0,"allocated":24125440,"data_stored":19381199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35772,"internal_metadata":1351840836},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002707"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.495711+0000","last_change":"2024-08-17T17:17:32.495652+0000","last_active":"2024-08-17T17:17:32.495711+0000","last_peered":"2024-08-17T17:17:32.495711+0000","last_clean":"2024-08-17T17:17:32.495711+0000","last_became_active":"2024-08-17T17:17:32.495218+0000","last_became_peered":"2024-08-17T17:17:32.495218+0000","last_unstale":"2024-08-17T17:17:32.495711+0000","last_undegraded":"2024-08-17T17:17:32.495711+0000","last_fullsized":"2024-08-17T17:17:32.495711+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:11:45.821282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":70,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.485328+0000","last_change":"2024-08-17T17:17:00.758108+0000","last_active":"2024-08-17T17:17:31.485328+0000","last_peered":"2024-08-17T17:17:31.485328+0000","last_clean":"2024-08-17T17:17:31.485328+0000","last_became_active":"2024-08-17T17:17:00.757714+0000","last_became_peered":"2024-08-17T17:17:00.757714+0000","last_unstale":"2024-08-17T17:17:31.485328+0000","last_undegraded":"2024-08-17T17:17:31.485328+0000","last_fullsized":"2024-08-17T17:17:31.485328+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:13:47.351728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":67,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.495336+0000","last_change":"2024-08-17T17:17:32.495336+0000","last_active":"2024-08-17T17:17:32.495336+0000","last_peered":"2024-08-17T17:17:32.495336+0000","last_clean":"2024-08-17T17:17:32.495336+0000","last_became_active":"2024-08-17T17:17:32.494975+0000","last_became_peered":"2024-08-17T17:17:32.494975+0000","last_unstale":"2024-08-17T17:17:32.495336+0000","last_undegraded":"2024-08-17T17:17:32.495336+0000","last_fullsized":"2024-08-17T17:17:32.495336+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:51:24.050993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.497616+0000","last_change":"2024-08-17T17:17:32.494012+0000","last_active":"2024-08-17T17:17:32.497616+0000","last_peered":"2024-08-17T17:17:32.497616+0000","last_clean":"2024-08-17T17:17:32.497616+0000","last_became_active":"2024-08-17T17:17:32.493663+0000","last_became_peered":"2024-08-17T17:17:32.493663+0000","last_unstale":"2024-08-17T17:17:32.497616+0000","last_undegraded":"2024-08-17T17:17:32.497616+0000","last_fullsized":"2024-08-17T17:17:32.497616+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:05:05.713213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.497986+0000","last_change":"2024-08-17T17:17:32.494930+0000","last_active":"2024-08-17T17:17:32.497986+0000","last_peered":"2024-08-17T17:17:32.497986+0000","last_clean":"2024-08-17T17:17:32.497986+0000","last_became_active":"2024-08-17T17:17:32.494590+0000","last_became_peered":"2024-08-17T17:17:32.494590+0000","last_unstale":"2024-08-17T17:17:32.497986+0000","last_undegraded":"2024-08-17T17:17:32.497986+0000","last_fullsized":"2024-08-17T17:17:32.497986+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:23:16.376415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":51,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.485438+0000","last_change":"2024-08-17T17:07:46.158514+0000","last_active":"2024-08-17T17:17:31.485438+0000","last_peered":"2024-08-17T17:17:31.485438+0000","last_clean":"2024-08-17T17:17:31.485438+0000","last_became_active":"2024-08-17T17:07:46.158256+0000","last_became_peered":"2024-08-17T17:07:46.158256+0000","last_unstale":"2024-08-17T17:17:31.485438+0000","last_undegraded":"2024-08-17T17:17:31.485438+0000","last_fullsized":"2024-08-17T17:17:31.485438+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:30:44.486515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.494098+0000","last_change":"2024-08-17T17:17:32.494030+0000","last_active":"2024-08-17T17:17:32.494098+0000","last_peered":"2024-08-17T17:17:32.494098+0000","last_clean":"2024-08-17T17:17:32.494098+0000","last_became_active":"2024-08-17T17:17:32.493580+0000","last_became_peered":"2024-08-17T17:17:32.493580+0000","last_unstale":"2024-08-17T17:17:32.494098+0000","last_undegraded":"2024-08-17T17:17:32.494098+0000","last_fullsized":"2024-08-17T17:17:32.494098+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:23:14.505301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":76,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.492950+0000","last_change":"2024-08-17T17:15:54.349784+0000","last_active":"2024-08-17T17:17:31.492950+0000","last_peered":"2024-08-17T17:17:31.492950+0000","last_clean":"2024-08-17T17:17:31.492950+0000","last_became_active":"2024-08-17T17:15:54.349305+0000","last_became_peered":"2024-08-17T17:15:54.349305+0000","last_unstale":"2024-08-17T17:17:31.492950+0000","last_undegraded":"2024-08-17T17:17:31.492950+0000","last_fullsized":"2024-08-17T17:17:31.492950+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:36:27.291719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":69,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.486671+0000","last_change":"2024-08-17T17:15:54.358793+0000","last_active":"2024-08-17T17:17:31.486671+0000","last_peered":"2024-08-17T17:17:31.486671+0000","last_clean":"2024-08-17T17:17:31.486671+0000","last_became_active":"2024-08-17T17:15:54.358280+0000","last_became_peered":"2024-08-17T17:15:54.358280+0000","last_unstale":"2024-08-17T17:17:31.486671+0000","last_undegraded":"2024-08-17T17:17:31.486671+0000","last_fullsized":"2024-08-17T17:17:31.486671+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:56:46.173026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":50,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.487824+0000","last_change":"2024-08-17T17:07:46.299427+0000","last_active":"2024-08-17T17:17:31.487824+0000","last_peered":"2024-08-17T17:17:31.487824+0000","last_clean":"2024-08-17T17:17:31.487824+0000","last_became_active":"2024-08-17T17:07:46.299008+0000","last_became_peered":"2024-08-17T17:07:46.299008+0000","last_unstale":"2024-08-17T17:17:31.487824+0000","last_undegraded":"2024-08-17T17:17:31.487824+0000","last_fullsized":"2024-08-17T17:17:31.487824+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:23:32.024874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":70,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.487123+0000","last_change":"2024-08-17T17:16:26.964049+0000","last_active":"2024-08-17T17:17:31.487123+0000","last_peered":"2024-08-17T17:17:31.487123+0000","last_clean":"2024-08-17T17:17:31.487123+0000","last_became_active":"2024-08-17T17:16:26.963554+0000","last_became_peered":"2024-08-17T17:16:26.963554+0000","last_unstale":"2024-08-17T17:17:31.487123+0000","last_undegraded":"2024-08-17T17:17:31.487123+0000","last_fullsized":"2024-08-17T17:17:31.487123+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:39:29.363488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":84,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.494445+0000","last_change":"2024-08-17T17:17:32.494445+0000","last_active":"2024-08-17T17:17:32.494445+0000","last_peered":"2024-08-17T17:17:32.494445+0000","last_clean":"2024-08-17T17:17:32.494445+0000","last_became_active":"2024-08-17T17:17:32.494080+0000","last_became_peered":"2024-08-17T17:17:32.494080+0000","last_unstale":"2024-08-17T17:17:32.494445+0000","last_undegraded":"2024-08-17T17:17:32.494445+0000","last_fullsized":"2024-08-17T17:17:32.494445+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:39:58.938883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":52,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.486546+0000","last_change":"2024-08-17T17:07:46.153923+0000","last_active":"2024-08-17T17:17:31.486546+0000","last_peered":"2024-08-17T17:17:31.486546+0000","last_clean":"2024-08-17T17:17:31.486546+0000","last_became_active":"2024-08-17T17:07:46.153614+0000","last_became_peered":"2024-08-17T17:07:46.153614+0000","last_unstale":"2024-08-17T17:17:31.486546+0000","last_undegraded":"2024-08-17T17:17:31.486546+0000","last_fullsized":"2024-08-17T17:17:31.486546+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:24:38.871907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":67,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.492465+0000","last_change":"2024-08-17T17:16:26.965847+0000","last_active":"2024-08-17T17:17:31.492465+0000","last_peered":"2024-08-17T17:17:31.492465+0000","last_clean":"2024-08-17T17:17:31.492465+0000","last_became_active":"2024-08-17T17:16:26.965522+0000","last_became_peered":"2024-08-17T17:16:26.965522+0000","last_unstale":"2024-08-17T17:17:31.492465+0000","last_undegraded":"2024-08-17T17:17:31.492465+0000","last_fullsized":"2024-08-17T17:17:31.492465+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:39:20.536460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.495428+0000","last_change":"2024-08-17T17:17:32.495428+0000","last_active":"2024-08-17T17:17:32.495428+0000","last_peered":"2024-08-17T17:17:32.495428+0000","last_clean":"2024-08-17T17:17:32.495428+0000","last_became_active":"2024-08-17T17:17:32.494978+0000","last_became_peered":"2024-08-17T17:17:32.494978+0000","last_unstale":"2024-08-17T17:17:32.495428+0000","last_undegraded":"2024-08-17T17:17:32.495428+0000","last_fullsized":"2024-08-17T17:17:32.495428+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:42:24.647812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.484182+0000","last_change":"2024-08-17T17:16:26.961623+0000","last_active":"2024-08-17T17:17:32.484182+0000","last_peered":"2024-08-17T17:17:32.484182+0000","last_clean":"2024-08-17T17:17:32.484182+0000","last_became_active":"2024-08-17T17:16:26.961284+0000","last_became_peered":"2024-08-17T17:16:26.961284+0000","last_unstale":"2024-08-17T17:17:32.484182+0000","last_undegraded":"2024-08-17T17:17:32.484182+0000","last_fullsized":"2024-08-17T17:17:32.484182+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:27:32.176916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.497027+0000","last_change":"2024-08-17T17:17:32.496970+0000","last_active":"2024-08-17T17:17:32.497027+0000","last_peered":"2024-08-17T17:17:32.497027+0000","last_clean":"2024-08-17T17:17:32.497027+0000","last_became_active":"2024-08-17T17:17:32.496610+0000","last_became_peered":"2024-08-17T17:17:32.496610+0000","last_unstale":"2024-08-17T17:17:32.497027+0000","last_undegraded":"2024-08-17T17:17:32.497027+0000","last_fullsized":"2024-08-17T17:17:32.497027+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:03:29.830609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":67,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.494935+0000","last_change":"2024-08-17T17:17:32.494935+0000","last_active":"2024-08-17T17:17:32.494935+0000","last_peered":"2024-08-17T17:17:32.494935+0000","last_clean":"2024-08-17T17:17:32.494935+0000","last_became_active":"2024-08-17T17:17:32.494516+0000","last_became_peered":"2024-08-17T17:17:32.494516+0000","last_unstale":"2024-08-17T17:17:32.494935+0000","last_undegraded":"2024-08-17T17:17:32.494935+0000","last_fullsized":"2024-08-17T17:17:32.494935+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:31:21.680055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":68,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.492670+0000","last_change":"2024-08-17T17:16:26.963141+0000","last_active":"2024-08-17T17:17:31.492670+0000","last_peered":"2024-08-17T17:17:31.492670+0000","last_clean":"2024-08-17T17:17:31.492670+0000","last_became_active":"2024-08-17T17:16:26.962778+0000","last_became_peered":"2024-08-17T17:16:26.962778+0000","last_unstale":"2024-08-17T17:17:31.492670+0000","last_undegraded":"2024-08-17T17:17:31.492670+0000","last_fullsized":"2024-08-17T17:17:31.492670+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:56:40.295525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":84,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.489477+0000","last_change":"2024-08-17T17:17:00.758598+0000","last_active":"2024-08-17T17:17:31.489477+0000","last_peered":"2024-08-17T17:17:31.489477+0000","last_clean":"2024-08-17T17:17:31.489477+0000","last_became_active":"2024-08-17T17:17:00.757663+0000","last_became_peered":"2024-08-17T17:17:00.757663+0000","last_unstale":"2024-08-17T17:17:31.489477+0000","last_undegraded":"2024-08-17T17:17:31.489477+0000","last_fullsized":"2024-08-17T17:17:31.489477+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:16:15.398787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":68,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.490060+0000","last_change":"2024-08-17T17:17:00.758630+0000","last_active":"2024-08-17T17:17:31.490060+0000","last_peered":"2024-08-17T17:17:31.490060+0000","last_clean":"2024-08-17T17:17:31.490060+0000","last_became_active":"2024-08-17T17:17:00.757829+0000","last_became_peered":"2024-08-17T17:17:00.757829+0000","last_unstale":"2024-08-17T17:17:31.490060+0000","last_undegraded":"2024-08-17T17:17:31.490060+0000","last_fullsized":"2024-08-17T17:17:31.490060+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:17:35.583797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":67,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.488838+0000","last_change":"2024-08-17T17:16:26.962625+0000","last_active":"2024-08-17T17:17:31.488838+0000","last_peered":"2024-08-17T17:17:31.488838+0000","last_clean":"2024-08-17T17:17:31.488838+0000","last_became_active":"2024-08-17T17:16:26.962259+0000","last_became_peered":"2024-08-17T17:16:26.962259+0000","last_unstale":"2024-08-17T17:17:31.488838+0000","last_undegraded":"2024-08-17T17:17:31.488838+0000","last_fullsized":"2024-08-17T17:17:31.488838+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:29:34.545710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":72,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.496427+0000","last_change":"2024-08-17T17:17:32.496361+0000","last_active":"2024-08-17T17:17:32.496427+0000","last_peered":"2024-08-17T17:17:32.496427+0000","last_clean":"2024-08-17T17:17:32.496427+0000","last_became_active":"2024-08-17T17:17:32.496056+0000","last_became_peered":"2024-08-17T17:17:32.496056+0000","last_unstale":"2024-08-17T17:17:32.496427+0000","last_undegraded":"2024-08-17T17:17:32.496427+0000","last_fullsized":"2024-08-17T17:17:32.496427+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:11:20.404589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":54,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.495711+0000","last_change":"2024-08-17T17:17:32.495625+0000","last_active":"2024-08-17T17:17:32.495711+0000","last_peered":"2024-08-17T17:17:32.495711+0000","last_clean":"2024-08-17T17:17:32.495711+0000","last_became_active":"2024-08-17T17:17:32.495298+0000","last_became_peered":"2024-08-17T17:17:32.495298+0000","last_unstale":"2024-08-17T17:17:32.495711+0000","last_undegraded":"2024-08-17T17:17:32.495711+0000","last_fullsized":"2024-08-17T17:17:32.495711+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:18:34.358254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":410,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.486090+0000","last_change":"2024-08-17T17:15:54.354936+0000","last_active":"2024-08-17T17:17:31.486090+0000","last_peered":"2024-08-17T17:17:31.486090+0000","last_clean":"2024-08-17T17:17:31.486090+0000","last_became_active":"2024-08-17T17:15:54.354238+0000","last_became_peered":"2024-08-17T17:15:54.354238+0000","last_unstale":"2024-08-17T17:17:31.486090+0000","last_undegraded":"2024-08-17T17:17:31.486090+0000","last_fullsized":"2024-08-17T17:17:31.486090+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:04:01.141401+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:58:05.159624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":602,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.488040+0000","last_change":"2024-08-17T17:17:00.759124+0000","last_active":"2024-08-17T17:17:31.488040+0000","last_peered":"2024-08-17T17:17:31.488040+0000","last_clean":"2024-08-17T17:17:31.488040+0000","last_became_active":"2024-08-17T17:17:00.758605+0000","last_became_peered":"2024-08-17T17:17:00.758605+0000","last_unstale":"2024-08-17T17:17:31.488040+0000","last_undegraded":"2024-08-17T17:17:31.488040+0000","last_fullsized":"2024-08-17T17:17:31.488040+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:05:10.050836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.500056+0000","last_change":"2024-08-17T17:17:32.499994+0000","last_active":"2024-08-17T17:17:32.500056+0000","last_peered":"2024-08-17T17:17:32.500056+0000","last_clean":"2024-08-17T17:17:32.500056+0000","last_became_active":"2024-08-17T17:17:32.499614+0000","last_became_peered":"2024-08-17T17:17:32.499614+0000","last_unstale":"2024-08-17T17:17:32.500056+0000","last_undegraded":"2024-08-17T17:17:32.500056+0000","last_fullsized":"2024-08-17T17:17:32.500056+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:18:05.307880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":50,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.493323+0000","last_change":"2024-08-17T17:07:47.164375+0000","last_active":"2024-08-17T17:17:31.493323+0000","last_peered":"2024-08-17T17:17:31.493323+0000","last_clean":"2024-08-17T17:17:31.493323+0000","last_became_active":"2024-08-17T17:07:47.164077+0000","last_became_peered":"2024-08-17T17:07:47.164077+0000","last_unstale":"2024-08-17T17:17:31.493323+0000","last_undegraded":"2024-08-17T17:17:31.493323+0000","last_fullsized":"2024-08-17T17:17:31.493323+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:26:01.278743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":700,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.485964+0000","last_change":"2024-08-17T17:16:26.958415+0000","last_active":"2024-08-17T17:17:31.485964+0000","last_peered":"2024-08-17T17:17:31.485964+0000","last_clean":"2024-08-17T17:17:31.485964+0000","last_became_active":"2024-08-17T17:16:26.957956+0000","last_became_peered":"2024-08-17T17:16:26.957956+0000","last_unstale":"2024-08-17T17:17:31.485964+0000","last_undegraded":"2024-08-17T17:17:31.485964+0000","last_fullsized":"2024-08-17T17:17:31.485964+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:22.478077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.499100+0000","last_change":"2024-08-17T17:17:32.493840+0000","last_active":"2024-08-17T17:17:32.499100+0000","last_peered":"2024-08-17T17:17:32.499100+0000","last_clean":"2024-08-17T17:17:32.499100+0000","last_became_active":"2024-08-17T17:17:32.493503+0000","last_became_peered":"2024-08-17T17:17:32.493503+0000","last_unstale":"2024-08-17T17:17:32.499100+0000","last_undegraded":"2024-08-17T17:17:32.499100+0000","last_fullsized":"2024-08-17T17:17:32.499100+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:01:06.908354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":79,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.484970+0000","last_change":"2024-08-17T17:17:00.761104+0000","last_active":"2024-08-17T17:17:32.484970+0000","last_peered":"2024-08-17T17:17:32.484970+0000","last_clean":"2024-08-17T17:17:32.484970+0000","last_became_active":"2024-08-17T17:17:00.760721+0000","last_became_peered":"2024-08-17T17:17:00.760721+0000","last_unstale":"2024-08-17T17:17:32.484970+0000","last_undegraded":"2024-08-17T17:17:32.484970+0000","last_fullsized":"2024-08-17T17:17:32.484970+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:06:49.900957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":555,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.495429+0000","last_change":"2024-08-17T17:17:32.495364+0000","last_active":"2024-08-17T17:17:32.495429+0000","last_peered":"2024-08-17T17:17:32.495429+0000","last_clean":"2024-08-17T17:17:32.495429+0000","last_became_active":"2024-08-17T17:17:32.494801+0000","last_became_peered":"2024-08-17T17:17:32.494801+0000","last_unstale":"2024-08-17T17:17:32.495429+0000","last_undegraded":"2024-08-17T17:17:32.495429+0000","last_fullsized":"2024-08-17T17:17:32.495429+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:11:06.683576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":55,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.488438+0000","last_change":"2024-08-17T17:17:00.758552+0000","last_active":"2024-08-17T17:17:32.488438+0000","last_peered":"2024-08-17T17:17:32.488438+0000","last_clean":"2024-08-17T17:17:32.488438+0000","last_became_active":"2024-08-17T17:17:00.758200+0000","last_became_peered":"2024-08-17T17:17:00.758200+0000","last_unstale":"2024-08-17T17:17:32.488438+0000","last_undegraded":"2024-08-17T17:17:32.488438+0000","last_fullsized":"2024-08-17T17:17:32.488438+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:13:27.648231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":85,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.488032+0000","last_change":"2024-08-17T17:16:26.964140+0000","last_active":"2024-08-17T17:17:32.488032+0000","last_peered":"2024-08-17T17:17:32.488032+0000","last_clean":"2024-08-17T17:17:32.488032+0000","last_became_active":"2024-08-17T17:16:26.963807+0000","last_became_peered":"2024-08-17T17:16:26.963807+0000","last_unstale":"2024-08-17T17:17:32.488032+0000","last_undegraded":"2024-08-17T17:17:32.488032+0000","last_fullsized":"2024-08-17T17:17:32.488032+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:30:50.386236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":936,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.487871+0000","last_change":"2024-08-17T17:16:26.965149+0000","last_active":"2024-08-17T17:17:31.487871+0000","last_peered":"2024-08-17T17:17:31.487871+0000","last_clean":"2024-08-17T17:17:31.487871+0000","last_became_active":"2024-08-17T17:16:26.964674+0000","last_became_peered":"2024-08-17T17:16:26.964674+0000","last_unstale":"2024-08-17T17:17:31.487871+0000","last_undegraded":"2024-08-17T17:17:31.487871+0000","last_fullsized":"2024-08-17T17:17:31.487871+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:28:57.549403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1392,"num_read_kb":2863,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":68,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.490674+0000","last_change":"2024-08-17T17:15:54.359640+0000","last_active":"2024-08-17T17:17:31.490674+0000","last_peered":"2024-08-17T17:17:31.490674+0000","last_clean":"2024-08-17T17:17:31.490674+0000","last_became_active":"2024-08-17T17:15:54.359163+0000","last_became_peered":"2024-08-17T17:15:54.359163+0000","last_unstale":"2024-08-17T17:17:31.490674+0000","last_undegraded":"2024-08-17T17:17:31.490674+0000","last_fullsized":"2024-08-17T17:17:31.490674+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:13:52.163231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.500703+0000","last_change":"2024-08-17T17:17:32.499715+0000","last_active":"2024-08-17T17:17:32.500703+0000","last_peered":"2024-08-17T17:17:32.500703+0000","last_clean":"2024-08-17T17:17:32.500703+0000","last_became_active":"2024-08-17T17:17:32.499412+0000","last_became_peered":"2024-08-17T17:17:32.499412+0000","last_unstale":"2024-08-17T17:17:32.500703+0000","last_undegraded":"2024-08-17T17:17:32.500703+0000","last_fullsized":"2024-08-17T17:17:32.500703+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:39:38.456970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":85,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.495728+0000","last_change":"2024-08-17T17:17:32.495728+0000","last_active":"2024-08-17T17:17:32.495728+0000","last_peered":"2024-08-17T17:17:32.495728+0000","last_clean":"2024-08-17T17:17:32.495728+0000","last_became_active":"2024-08-17T17:17:32.495357+0000","last_became_peered":"2024-08-17T17:17:32.495357+0000","last_unstale":"2024-08-17T17:17:32.495728+0000","last_undegraded":"2024-08-17T17:17:32.495728+0000","last_fullsized":"2024-08-17T17:17:32.495728+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:27:45.561567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":302,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.488216+0000","last_change":"2024-08-17T17:16:26.964621+0000","last_active":"2024-08-17T17:17:32.488216+0000","last_peered":"2024-08-17T17:17:32.488216+0000","last_clean":"2024-08-17T17:17:32.488216+0000","last_became_active":"2024-08-17T17:16:26.964058+0000","last_became_peered":"2024-08-17T17:16:26.964058+0000","last_unstale":"2024-08-17T17:17:32.488216+0000","last_undegraded":"2024-08-17T17:17:32.488216+0000","last_fullsized":"2024-08-17T17:17:32.488216+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:00:04.465644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":44,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.488723+0000","last_change":"2024-08-17T17:16:26.965872+0000","last_active":"2024-08-17T17:17:32.488723+0000","last_peered":"2024-08-17T17:17:32.488723+0000","last_clean":"2024-08-17T17:17:32.488723+0000","last_became_active":"2024-08-17T17:16:26.965580+0000","last_became_peered":"2024-08-17T17:16:26.965580+0000","last_unstale":"2024-08-17T17:17:32.488723+0000","last_undegraded":"2024-08-17T17:17:32.488723+0000","last_fullsized":"2024-08-17T17:17:32.488723+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:21:52.311284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":55,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.488106+0000","last_change":"2024-08-17T17:17:00.758843+0000","last_active":"2024-08-17T17:17:32.488106+0000","last_peered":"2024-08-17T17:17:32.488106+0000","last_clean":"2024-08-17T17:17:32.488106+0000","last_became_active":"2024-08-17T17:17:00.758350+0000","last_became_peered":"2024-08-17T17:17:00.758350+0000","last_unstale":"2024-08-17T17:17:32.488106+0000","last_undegraded":"2024-08-17T17:17:32.488106+0000","last_fullsized":"2024-08-17T17:17:32.488106+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:36:47.412787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":411,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.487103+0000","last_change":"2024-08-17T17:15:54.358998+0000","last_active":"2024-08-17T17:17:31.487103+0000","last_peered":"2024-08-17T17:17:31.487103+0000","last_clean":"2024-08-17T17:17:31.487103+0000","last_became_active":"2024-08-17T17:15:54.358141+0000","last_became_peered":"2024-08-17T17:15:54.358141+0000","last_unstale":"2024-08-17T17:17:31.487103+0000","last_undegraded":"2024-08-17T17:17:31.487103+0000","last_fullsized":"2024-08-17T17:17:31.487103+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:32:08.078428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":55,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.484086+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:17:32.484086+0000","last_peered":"2024-08-17T17:17:32.484086+0000","last_clean":"2024-08-17T17:17:32.484086+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:17:32.484086+0000","last_undegraded":"2024-08-17T17:17:32.484086+0000","last_fullsized":"2024-08-17T17:17:32.484086+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.496087+0000","last_change":"2024-08-17T17:17:32.496023+0000","last_active":"2024-08-17T17:17:32.496087+0000","last_peered":"2024-08-17T17:17:32.496087+0000","last_clean":"2024-08-17T17:17:32.496087+0000","last_became_active":"2024-08-17T17:17:32.495652+0000","last_became_peered":"2024-08-17T17:17:32.495652+0000","last_unstale":"2024-08-17T17:17:32.496087+0000","last_undegraded":"2024-08-17T17:17:32.496087+0000","last_fullsized":"2024-08-17T17:17:32.496087+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:42:52.056527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":564,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.488044+0000","last_change":"2024-08-17T17:16:26.967134+0000","last_active":"2024-08-17T17:17:32.488044+0000","last_peered":"2024-08-17T17:17:32.488044+0000","last_clean":"2024-08-17T17:17:32.488044+0000","last_became_active":"2024-08-17T17:16:26.966517+0000","last_became_peered":"2024-08-17T17:16:26.966517+0000","last_unstale":"2024-08-17T17:17:32.488044+0000","last_undegraded":"2024-08-17T17:17:32.488044+0000","last_fullsized":"2024-08-17T17:17:32.488044+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:38:52.366564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.501049+0000","last_change":"2024-08-17T17:17:32.500970+0000","last_active":"2024-08-17T17:17:32.501049+0000","last_peered":"2024-08-17T17:17:32.501049+0000","last_clean":"2024-08-17T17:17:32.501049+0000","last_became_active":"2024-08-17T17:17:32.499033+0000","last_became_peered":"2024-08-17T17:17:32.499033+0000","last_unstale":"2024-08-17T17:17:32.501049+0000","last_undegraded":"2024-08-17T17:17:32.501049+0000","last_fullsized":"2024-08-17T17:17:32.501049+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:26:01.543899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.484380+0000","last_change":"2024-08-17T17:16:26.962885+0000","last_active":"2024-08-17T17:17:32.484380+0000","last_peered":"2024-08-17T17:17:32.484380+0000","last_clean":"2024-08-17T17:17:32.484380+0000","last_became_active":"2024-08-17T17:16:26.961919+0000","last_became_peered":"2024-08-17T17:16:26.961919+0000","last_unstale":"2024-08-17T17:17:32.484380+0000","last_undegraded":"2024-08-17T17:17:32.484380+0000","last_fullsized":"2024-08-17T17:17:32.484380+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:03:43.614359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":291,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.492024+0000","last_change":"2024-08-17T17:17:00.755772+0000","last_active":"2024-08-17T17:17:31.492024+0000","last_peered":"2024-08-17T17:17:31.492024+0000","last_clean":"2024-08-17T17:17:31.492024+0000","last_became_active":"2024-08-17T17:17:00.755281+0000","last_became_peered":"2024-08-17T17:17:00.755281+0000","last_unstale":"2024-08-17T17:17:31.492024+0000","last_undegraded":"2024-08-17T17:17:31.492024+0000","last_fullsized":"2024-08-17T17:17:31.492024+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:56:54.600755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":50,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.488953+0000","last_change":"2024-08-17T17:07:47.227287+0000","last_active":"2024-08-17T17:17:31.488953+0000","last_peered":"2024-08-17T17:17:31.488953+0000","last_clean":"2024-08-17T17:17:31.488953+0000","last_became_active":"2024-08-17T17:07:47.226983+0000","last_became_peered":"2024-08-17T17:07:47.226983+0000","last_unstale":"2024-08-17T17:17:31.488953+0000","last_undegraded":"2024-08-17T17:17:31.488953+0000","last_fullsized":"2024-08-17T17:17:31.488953+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:34:07.444142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":67,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.490105+0000","last_change":"2024-08-17T17:16:26.961292+0000","last_active":"2024-08-17T17:17:31.490105+0000","last_peered":"2024-08-17T17:17:31.490105+0000","last_clean":"2024-08-17T17:17:31.490105+0000","last_became_active":"2024-08-17T17:16:26.960977+0000","last_became_peered":"2024-08-17T17:16:26.960977+0000","last_unstale":"2024-08-17T17:17:31.490105+0000","last_undegraded":"2024-08-17T17:17:31.490105+0000","last_fullsized":"2024-08-17T17:17:31.490105+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:58:50.075227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":73,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.489768+0000","last_change":"2024-08-17T17:07:46.160694+0000","last_active":"2024-08-17T17:17:31.489768+0000","last_peered":"2024-08-17T17:17:31.489768+0000","last_clean":"2024-08-17T17:17:31.489768+0000","last_became_active":"2024-08-17T17:07:46.160407+0000","last_became_peered":"2024-08-17T17:07:46.160407+0000","last_unstale":"2024-08-17T17:17:31.489768+0000","last_undegraded":"2024-08-17T17:17:31.489768+0000","last_fullsized":"2024-08-17T17:17:31.489768+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:54:38.366094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":54,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.487584+0000","last_change":"2024-08-17T17:17:00.756909+0000","last_active":"2024-08-17T17:17:32.487584+0000","last_peered":"2024-08-17T17:17:32.487584+0000","last_clean":"2024-08-17T17:17:32.487584+0000","last_became_active":"2024-08-17T17:17:00.756532+0000","last_became_peered":"2024-08-17T17:17:00.756532+0000","last_unstale":"2024-08-17T17:17:32.487584+0000","last_undegraded":"2024-08-17T17:17:32.487584+0000","last_fullsized":"2024-08-17T17:17:32.487584+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:11:25.294907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.496800+0000","last_change":"2024-08-17T17:17:32.496800+0000","last_active":"2024-08-17T17:17:32.496800+0000","last_peered":"2024-08-17T17:17:32.496800+0000","last_clean":"2024-08-17T17:17:32.496800+0000","last_became_active":"2024-08-17T17:17:32.496405+0000","last_became_peered":"2024-08-17T17:17:32.496405+0000","last_unstale":"2024-08-17T17:17:32.496800+0000","last_undegraded":"2024-08-17T17:17:32.496800+0000","last_fullsized":"2024-08-17T17:17:32.496800+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:03:56.618806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":68,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.489488+0000","last_change":"2024-08-17T17:16:26.964939+0000","last_active":"2024-08-17T17:17:31.489488+0000","last_peered":"2024-08-17T17:17:31.489488+0000","last_clean":"2024-08-17T17:17:31.489488+0000","last_became_active":"2024-08-17T17:16:26.964598+0000","last_became_peered":"2024-08-17T17:16:26.964598+0000","last_unstale":"2024-08-17T17:17:31.489488+0000","last_undegraded":"2024-08-17T17:17:31.489488+0000","last_fullsized":"2024-08-17T17:17:31.489488+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:01:09.949950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":24,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.501097+0000","last_change":"2024-08-17T17:17:32.501043+0000","last_active":"2024-08-17T17:17:32.501097+0000","last_peered":"2024-08-17T17:17:32.501097+0000","last_clean":"2024-08-17T17:17:32.501097+0000","last_became_active":"2024-08-17T17:17:32.499400+0000","last_became_peered":"2024-08-17T17:17:32.499400+0000","last_unstale":"2024-08-17T17:17:32.501097+0000","last_undegraded":"2024-08-17T17:17:32.501097+0000","last_fullsized":"2024-08-17T17:17:32.501097+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:47:35.040523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.496833+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:17:32.496833+0000","last_peered":"2024-08-17T17:17:32.496833+0000","last_clean":"2024-08-17T17:17:32.496833+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:17:32.496833+0000","last_undegraded":"2024-08-17T17:17:32.496833+0000","last_fullsized":"2024-08-17T17:17:32.496833+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.501087+0000","last_change":"2024-08-17T17:17:32.501012+0000","last_active":"2024-08-17T17:17:32.501087+0000","last_peered":"2024-08-17T17:17:32.501087+0000","last_clean":"2024-08-17T17:17:32.501087+0000","last_became_active":"2024-08-17T17:17:32.499399+0000","last_became_peered":"2024-08-17T17:17:32.499399+0000","last_unstale":"2024-08-17T17:17:32.501087+0000","last_undegraded":"2024-08-17T17:17:32.501087+0000","last_fullsized":"2024-08-17T17:17:32.501087+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:19:57.035025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":67,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.491928+0000","last_change":"2024-08-17T17:16:26.963696+0000","last_active":"2024-08-17T17:17:31.491928+0000","last_peered":"2024-08-17T17:17:31.491928+0000","last_clean":"2024-08-17T17:17:31.491928+0000","last_became_active":"2024-08-17T17:16:26.963280+0000","last_became_peered":"2024-08-17T17:16:26.963280+0000","last_unstale":"2024-08-17T17:17:31.491928+0000","last_undegraded":"2024-08-17T17:17:31.491928+0000","last_fullsized":"2024-08-17T17:17:31.491928+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:22:01.474840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.500549+0000","last_change":"2024-08-17T17:17:32.500494+0000","last_active":"2024-08-17T17:17:32.500549+0000","last_peered":"2024-08-17T17:17:32.500549+0000","last_clean":"2024-08-17T17:17:32.500549+0000","last_became_active":"2024-08-17T17:17:32.498949+0000","last_became_peered":"2024-08-17T17:17:32.498949+0000","last_unstale":"2024-08-17T17:17:32.500549+0000","last_undegraded":"2024-08-17T17:17:32.500549+0000","last_fullsized":"2024-08-17T17:17:32.500549+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:44.681212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.498006+0000","last_change":"2024-08-17T17:17:32.495908+0000","last_active":"2024-08-17T17:17:32.498006+0000","last_peered":"2024-08-17T17:17:32.498006+0000","last_clean":"2024-08-17T17:17:32.498006+0000","last_became_active":"2024-08-17T17:17:32.495593+0000","last_became_peered":"2024-08-17T17:17:32.495593+0000","last_unstale":"2024-08-17T17:17:32.498006+0000","last_undegraded":"2024-08-17T17:17:32.498006+0000","last_fullsized":"2024-08-17T17:17:32.498006+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:41:39.009044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":55,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.498046+0000","last_change":"2024-08-17T17:17:32.497986+0000","last_active":"2024-08-17T17:17:32.498046+0000","last_peered":"2024-08-17T17:17:32.498046+0000","last_clean":"2024-08-17T17:17:32.498046+0000","last_became_active":"2024-08-17T17:17:32.497639+0000","last_became_peered":"2024-08-17T17:17:32.497639+0000","last_unstale":"2024-08-17T17:17:32.498046+0000","last_undegraded":"2024-08-17T17:17:32.498046+0000","last_fullsized":"2024-08-17T17:17:32.498046+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:03:25.854983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":85,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.495810+0000","last_change":"2024-08-17T17:17:32.495727+0000","last_active":"2024-08-17T17:17:32.495810+0000","last_peered":"2024-08-17T17:17:32.495810+0000","last_clean":"2024-08-17T17:17:32.495810+0000","last_became_active":"2024-08-17T17:17:32.494437+0000","last_became_peered":"2024-08-17T17:17:32.494437+0000","last_unstale":"2024-08-17T17:17:32.495810+0000","last_undegraded":"2024-08-17T17:17:32.495810+0000","last_fullsized":"2024-08-17T17:17:32.495810+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:56:22.902797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":86,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.489170+0000","last_change":"2024-08-17T17:17:00.758212+0000","last_active":"2024-08-17T17:17:31.489170+0000","last_peered":"2024-08-17T17:17:31.489170+0000","last_clean":"2024-08-17T17:17:31.489170+0000","last_became_active":"2024-08-17T17:17:00.757787+0000","last_became_peered":"2024-08-17T17:17:00.757787+0000","last_unstale":"2024-08-17T17:17:31.489170+0000","last_undegraded":"2024-08-17T17:17:31.489170+0000","last_fullsized":"2024-08-17T17:17:31.489170+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:19:11.289350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.495253+0000","last_change":"2024-08-17T17:17:32.495192+0000","last_active":"2024-08-17T17:17:32.495253+0000","last_peered":"2024-08-17T17:17:32.495253+0000","last_clean":"2024-08-17T17:17:32.495253+0000","last_became_active":"2024-08-17T17:17:32.494866+0000","last_became_peered":"2024-08-17T17:17:32.494866+0000","last_unstale":"2024-08-17T17:17:32.495253+0000","last_undegraded":"2024-08-17T17:17:32.495253+0000","last_fullsized":"2024-08-17T17:17:32.495253+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:21:13.274144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":58,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.483940+0000","last_change":"2024-08-17T17:15:54.357458+0000","last_active":"2024-08-17T17:17:32.483940+0000","last_peered":"2024-08-17T17:17:32.483940+0000","last_clean":"2024-08-17T17:17:32.483940+0000","last_became_active":"2024-08-17T17:15:54.356621+0000","last_became_peered":"2024-08-17T17:15:54.356621+0000","last_unstale":"2024-08-17T17:17:32.483940+0000","last_undegraded":"2024-08-17T17:17:32.483940+0000","last_fullsized":"2024-08-17T17:17:32.483940+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:16:10.254328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":84,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.493417+0000","last_change":"2024-08-17T17:17:32.493417+0000","last_active":"2024-08-17T17:17:32.493417+0000","last_peered":"2024-08-17T17:17:32.493417+0000","last_clean":"2024-08-17T17:17:32.493417+0000","last_became_active":"2024-08-17T17:17:32.493097+0000","last_became_peered":"2024-08-17T17:17:32.493097+0000","last_unstale":"2024-08-17T17:17:32.493417+0000","last_undegraded":"2024-08-17T17:17:32.493417+0000","last_fullsized":"2024-08-17T17:17:32.493417+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:22:01.622261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":70,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.488375+0000","last_change":"2024-08-17T17:16:26.964339+0000","last_active":"2024-08-17T17:17:31.488375+0000","last_peered":"2024-08-17T17:17:31.488375+0000","last_clean":"2024-08-17T17:17:31.488375+0000","last_became_active":"2024-08-17T17:16:26.963964+0000","last_became_peered":"2024-08-17T17:16:26.963964+0000","last_unstale":"2024-08-17T17:17:31.488375+0000","last_undegraded":"2024-08-17T17:17:31.488375+0000","last_fullsized":"2024-08-17T17:17:31.488375+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:24:14.417213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":84,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.492899+0000","last_change":"2024-08-17T17:17:32.492899+0000","last_active":"2024-08-17T17:17:32.492899+0000","last_peered":"2024-08-17T17:17:32.492899+0000","last_clean":"2024-08-17T17:17:32.492899+0000","last_became_active":"2024-08-17T17:17:32.492522+0000","last_became_peered":"2024-08-17T17:17:32.492522+0000","last_unstale":"2024-08-17T17:17:32.492899+0000","last_undegraded":"2024-08-17T17:17:32.492899+0000","last_fullsized":"2024-08-17T17:17:32.492899+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:27:57.515343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":53,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.490249+0000","last_change":"2024-08-17T17:07:46.156750+0000","last_active":"2024-08-17T17:17:31.490249+0000","last_peered":"2024-08-17T17:17:31.490249+0000","last_clean":"2024-08-17T17:17:31.490249+0000","last_became_active":"2024-08-17T17:07:46.156253+0000","last_became_peered":"2024-08-17T17:07:46.156253+0000","last_unstale":"2024-08-17T17:17:31.490249+0000","last_undegraded":"2024-08-17T17:17:31.490249+0000","last_fullsized":"2024-08-17T17:17:31.490249+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:04:28.549130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":92,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.499059+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:17:32.499059+0000","last_peered":"2024-08-17T17:17:32.499059+0000","last_clean":"2024-08-17T17:17:32.499059+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:17:32.499059+0000","last_undegraded":"2024-08-17T17:17:32.499059+0000","last_fullsized":"2024-08-17T17:17:32.499059+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":77,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.500444+0000","last_change":"2024-08-17T17:17:32.500387+0000","last_active":"2024-08-17T17:17:32.500444+0000","last_peered":"2024-08-17T17:17:32.500444+0000","last_clean":"2024-08-17T17:17:32.500444+0000","last_became_active":"2024-08-17T17:17:32.500062+0000","last_became_peered":"2024-08-17T17:17:32.500062+0000","last_unstale":"2024-08-17T17:17:32.500444+0000","last_undegraded":"2024-08-17T17:17:32.500444+0000","last_fullsized":"2024-08-17T17:17:32.500444+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:12:25.410392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":77,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.488249+0000","last_change":"2024-08-17T17:17:00.758339+0000","last_active":"2024-08-17T17:17:31.488249+0000","last_peered":"2024-08-17T17:17:31.488249+0000","last_clean":"2024-08-17T17:17:31.488249+0000","last_became_active":"2024-08-17T17:17:00.757987+0000","last_became_peered":"2024-08-17T17:17:00.757987+0000","last_unstale":"2024-08-17T17:17:31.488249+0000","last_undegraded":"2024-08-17T17:17:31.488249+0000","last_fullsized":"2024-08-17T17:17:31.488249+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:55:40.761984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.484386+0000","last_change":"2024-08-17T17:17:00.760208+0000","last_active":"2024-08-17T17:17:32.484386+0000","last_peered":"2024-08-17T17:17:32.484386+0000","last_clean":"2024-08-17T17:17:32.484386+0000","last_became_active":"2024-08-17T17:17:00.759800+0000","last_became_peered":"2024-08-17T17:17:00.759800+0000","last_unstale":"2024-08-17T17:17:32.484386+0000","last_undegraded":"2024-08-17T17:17:32.484386+0000","last_fullsized":"2024-08-17T17:17:32.484386+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:01:36.990816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17971,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234140,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307404,"kb_used_data":5880,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93437748,"statfs":{"total":95995035648,"available":95680253952,"internally_reserved":0,"allocated":6021120,"data_stored":5415758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495775,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307388,"kb_used_data":5868,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93437764,"statfs":{"total":95995035648,"available":95680270336,"internally_reserved":0,"allocated":6008832,"data_stored":5415424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757412,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298552,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446600,"statfs":{"total":95995035648,"available":95689318400,"internally_reserved":0,"allocated":1216512,"data_stored":626785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019050,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302720,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93442432,"statfs":{"total":95995035648,"available":95685050368,"internally_reserved":0,"allocated":1232896,"data_stored":631339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435278,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28660,"kb_used_data":1184,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":93716492,"statfs":{"total":95995035648,"available":95965687808,"internally_reserved":0,"allocated":1212416,"data_stored":626559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":28109834},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676598804,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36980,"kb_used_data":5284,"kb_used_omap":7,"kb_used_meta":31672,"kb_avail":93708172,"statfs":{"total":95995035648,"available":95957168128,"internally_reserved":0,"allocated":5410816,"data_stored":4821385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":32432290},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762331,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32884,"kb_used_data":1184,"kb_used_omap":7,"kb_used_meta":31672,"kb_avail":93712268,"statfs":{"total":95995035648,"available":95961362432,"internally_reserved":0,"allocated":1212416,"data_stored":626504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7945,"internal_metadata":32432375},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726925857,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":29304,"kb_used_data":1768,"kb_used_omap":7,"kb_used_meta":27512,"kb_avail":93715848,"statfs":{"total":95995035648,"available":95965028352,"internally_reserved":0,"allocated":1810432,"data_stored":1217445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":28172609},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6241,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:18:33.629 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-08-17T17:18:33.629 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-08-17T17:18:33.629 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-08-17T17:18:33.629 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph health --format=json 2024-08-17T17:18:33.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:33 smithi053 ceph-mon[89963]: pgmap v200: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:33.900 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:18:34.656 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:34 smithi003 ceph-mon[103344]: from='client.35196 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:18:34.656 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:34 smithi003 ceph-mon[100615]: from='client.35196 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:18:34.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:34 smithi053 ceph-mon[89963]: from='client.35196 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:18:35.658 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:18:35.658 INFO:teuthology.orchestra.run.smithi003.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-08-17T17:18:35.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:35 smithi053 ceph-mon[89963]: pgmap v201: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:35.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:35 smithi003 ceph-mon[100615]: pgmap v201: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:35.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:35 smithi003 ceph-mon[103344]: pgmap v201: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:35.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:18:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:18:35] "GET /metrics HTTP/1.1" 200 33939 "" "Prometheus/2.51.0" 2024-08-17T17:18:36.307 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-08-17T17:18:36.308 INFO:teuthology.run_tasks:Running task print... 2024-08-17T17:18:36.321 INFO:teuthology.task.print:**** applying stress + thrashing to mixed-version cluster **** 2024-08-17T17:18:36.321 INFO:teuthology.run_tasks:Running task parallel... 2024-08-17T17:18:36.328 INFO:teuthology.task.parallel:starting parallel... 2024-08-17T17:18:36.329 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-08-17T17:18:36.329 INFO:teuthology.task.sequential:In sequential, running task thrashosds... 2024-08-17T17:18:36.331 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_force_recovery': 0, 'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_thrash_pg_upmap_items': 0, 'disable_objectstore_tool_tests': True, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-08-17T17:18:36.331 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'create_rbd_pool': True, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'OSD_DOWN', 'mons down', 'mon down', 'MON_DOWN', 'out of quorum', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSDMAP_FLAGS', 'PG_AVAILABILITY'], 'sha1': 'cd8bdbeb9fbdab5733aa3aaadc042c965473e453'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'cd8bdbeb9fbdab5733aa3aaadc042c965473e453'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'main', 'sha1': '91f88e4d132ac08dc53355fa3d9717a53236a3a9'}} 2024-08-17T17:18:36.332 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_force_recovery': 0, 'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_thrash_pg_upmap_items': 0, 'disable_objectstore_tool_tests': True, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-08-17T17:18:36.332 INFO:tasks.thrashosds:Beginning thrashosds... 2024-08-17T17:18:36.332 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-08-17T17:18:36.332 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:18:36.605 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:18:36.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:36 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1808659647' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-08-17T17:18:36.728 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:36 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1808659647' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-08-17T17:18:36.728 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:36 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1808659647' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-08-17T17:18:37.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:37 smithi053 ceph-mon[89963]: pgmap v202: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:37.733 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:37 smithi003 ceph-mon[103344]: pgmap v202: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:37.733 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:37 smithi003 ceph-mon[100615]: pgmap v202: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:38.374 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:18:38.375 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:18:38.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:18:38.733 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:18:38.733 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:18:39.087 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":203,"stamp":"2024-08-17T17:18:37.689882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790221,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":6390,"num_read_kb":13845,"num_write":1122,"num_write_kb":15039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1181,"ondisk_log_size":1181,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1343892,"kb_used_data":23560,"kb_used_omap":34,"kb_used_meta":1320157,"kb_avail":748617324,"statfs":{"total":767960285184,"available":766584139776,"internally_reserved":0,"allocated":24125440,"data_stored":19381199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35772,"internal_metadata":1351840836},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002649"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.495711+0000","last_change":"2024-08-17T17:17:32.495652+0000","last_active":"2024-08-17T17:17:32.495711+0000","last_peered":"2024-08-17T17:17:32.495711+0000","last_clean":"2024-08-17T17:17:32.495711+0000","last_became_active":"2024-08-17T17:17:32.495218+0000","last_became_peered":"2024-08-17T17:17:32.495218+0000","last_unstale":"2024-08-17T17:17:32.495711+0000","last_undegraded":"2024-08-17T17:17:32.495711+0000","last_fullsized":"2024-08-17T17:17:32.495711+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:11:45.821282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":70,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.485328+0000","last_change":"2024-08-17T17:17:00.758108+0000","last_active":"2024-08-17T17:17:31.485328+0000","last_peered":"2024-08-17T17:17:31.485328+0000","last_clean":"2024-08-17T17:17:31.485328+0000","last_became_active":"2024-08-17T17:17:00.757714+0000","last_became_peered":"2024-08-17T17:17:00.757714+0000","last_unstale":"2024-08-17T17:17:31.485328+0000","last_undegraded":"2024-08-17T17:17:31.485328+0000","last_fullsized":"2024-08-17T17:17:31.485328+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:13:47.351728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":67,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.495336+0000","last_change":"2024-08-17T17:17:32.495336+0000","last_active":"2024-08-17T17:17:32.495336+0000","last_peered":"2024-08-17T17:17:32.495336+0000","last_clean":"2024-08-17T17:17:32.495336+0000","last_became_active":"2024-08-17T17:17:32.494975+0000","last_became_peered":"2024-08-17T17:17:32.494975+0000","last_unstale":"2024-08-17T17:17:32.495336+0000","last_undegraded":"2024-08-17T17:17:32.495336+0000","last_fullsized":"2024-08-17T17:17:32.495336+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:51:24.050993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.497616+0000","last_change":"2024-08-17T17:17:32.494012+0000","last_active":"2024-08-17T17:17:32.497616+0000","last_peered":"2024-08-17T17:17:32.497616+0000","last_clean":"2024-08-17T17:17:32.497616+0000","last_became_active":"2024-08-17T17:17:32.493663+0000","last_became_peered":"2024-08-17T17:17:32.493663+0000","last_unstale":"2024-08-17T17:17:32.497616+0000","last_undegraded":"2024-08-17T17:17:32.497616+0000","last_fullsized":"2024-08-17T17:17:32.497616+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:05:05.713213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.497986+0000","last_change":"2024-08-17T17:17:32.494930+0000","last_active":"2024-08-17T17:17:32.497986+0000","last_peered":"2024-08-17T17:17:32.497986+0000","last_clean":"2024-08-17T17:17:32.497986+0000","last_became_active":"2024-08-17T17:17:32.494590+0000","last_became_peered":"2024-08-17T17:17:32.494590+0000","last_unstale":"2024-08-17T17:17:32.497986+0000","last_undegraded":"2024-08-17T17:17:32.497986+0000","last_fullsized":"2024-08-17T17:17:32.497986+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:23:16.376415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":51,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.485438+0000","last_change":"2024-08-17T17:07:46.158514+0000","last_active":"2024-08-17T17:17:31.485438+0000","last_peered":"2024-08-17T17:17:31.485438+0000","last_clean":"2024-08-17T17:17:31.485438+0000","last_became_active":"2024-08-17T17:07:46.158256+0000","last_became_peered":"2024-08-17T17:07:46.158256+0000","last_unstale":"2024-08-17T17:17:31.485438+0000","last_undegraded":"2024-08-17T17:17:31.485438+0000","last_fullsized":"2024-08-17T17:17:31.485438+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:30:44.486515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.494098+0000","last_change":"2024-08-17T17:17:32.494030+0000","last_active":"2024-08-17T17:17:32.494098+0000","last_peered":"2024-08-17T17:17:32.494098+0000","last_clean":"2024-08-17T17:17:32.494098+0000","last_became_active":"2024-08-17T17:17:32.493580+0000","last_became_peered":"2024-08-17T17:17:32.493580+0000","last_unstale":"2024-08-17T17:17:32.494098+0000","last_undegraded":"2024-08-17T17:17:32.494098+0000","last_fullsized":"2024-08-17T17:17:32.494098+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:23:14.505301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":76,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.492950+0000","last_change":"2024-08-17T17:15:54.349784+0000","last_active":"2024-08-17T17:17:31.492950+0000","last_peered":"2024-08-17T17:17:31.492950+0000","last_clean":"2024-08-17T17:17:31.492950+0000","last_became_active":"2024-08-17T17:15:54.349305+0000","last_became_peered":"2024-08-17T17:15:54.349305+0000","last_unstale":"2024-08-17T17:17:31.492950+0000","last_undegraded":"2024-08-17T17:17:31.492950+0000","last_fullsized":"2024-08-17T17:17:31.492950+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:36:27.291719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":69,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.486671+0000","last_change":"2024-08-17T17:15:54.358793+0000","last_active":"2024-08-17T17:17:31.486671+0000","last_peered":"2024-08-17T17:17:31.486671+0000","last_clean":"2024-08-17T17:17:31.486671+0000","last_became_active":"2024-08-17T17:15:54.358280+0000","last_became_peered":"2024-08-17T17:15:54.358280+0000","last_unstale":"2024-08-17T17:17:31.486671+0000","last_undegraded":"2024-08-17T17:17:31.486671+0000","last_fullsized":"2024-08-17T17:17:31.486671+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:56:46.173026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":50,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.487824+0000","last_change":"2024-08-17T17:07:46.299427+0000","last_active":"2024-08-17T17:17:31.487824+0000","last_peered":"2024-08-17T17:17:31.487824+0000","last_clean":"2024-08-17T17:17:31.487824+0000","last_became_active":"2024-08-17T17:07:46.299008+0000","last_became_peered":"2024-08-17T17:07:46.299008+0000","last_unstale":"2024-08-17T17:17:31.487824+0000","last_undegraded":"2024-08-17T17:17:31.487824+0000","last_fullsized":"2024-08-17T17:17:31.487824+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:23:32.024874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":70,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.487123+0000","last_change":"2024-08-17T17:16:26.964049+0000","last_active":"2024-08-17T17:17:31.487123+0000","last_peered":"2024-08-17T17:17:31.487123+0000","last_clean":"2024-08-17T17:17:31.487123+0000","last_became_active":"2024-08-17T17:16:26.963554+0000","last_became_peered":"2024-08-17T17:16:26.963554+0000","last_unstale":"2024-08-17T17:17:31.487123+0000","last_undegraded":"2024-08-17T17:17:31.487123+0000","last_fullsized":"2024-08-17T17:17:31.487123+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:39:29.363488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":84,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.494445+0000","last_change":"2024-08-17T17:17:32.494445+0000","last_active":"2024-08-17T17:17:32.494445+0000","last_peered":"2024-08-17T17:17:32.494445+0000","last_clean":"2024-08-17T17:17:32.494445+0000","last_became_active":"2024-08-17T17:17:32.494080+0000","last_became_peered":"2024-08-17T17:17:32.494080+0000","last_unstale":"2024-08-17T17:17:32.494445+0000","last_undegraded":"2024-08-17T17:17:32.494445+0000","last_fullsized":"2024-08-17T17:17:32.494445+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:39:58.938883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":52,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.486546+0000","last_change":"2024-08-17T17:07:46.153923+0000","last_active":"2024-08-17T17:17:31.486546+0000","last_peered":"2024-08-17T17:17:31.486546+0000","last_clean":"2024-08-17T17:17:31.486546+0000","last_became_active":"2024-08-17T17:07:46.153614+0000","last_became_peered":"2024-08-17T17:07:46.153614+0000","last_unstale":"2024-08-17T17:17:31.486546+0000","last_undegraded":"2024-08-17T17:17:31.486546+0000","last_fullsized":"2024-08-17T17:17:31.486546+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:24:38.871907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":67,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.492465+0000","last_change":"2024-08-17T17:16:26.965847+0000","last_active":"2024-08-17T17:17:31.492465+0000","last_peered":"2024-08-17T17:17:31.492465+0000","last_clean":"2024-08-17T17:17:31.492465+0000","last_became_active":"2024-08-17T17:16:26.965522+0000","last_became_peered":"2024-08-17T17:16:26.965522+0000","last_unstale":"2024-08-17T17:17:31.492465+0000","last_undegraded":"2024-08-17T17:17:31.492465+0000","last_fullsized":"2024-08-17T17:17:31.492465+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:39:20.536460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.495428+0000","last_change":"2024-08-17T17:17:32.495428+0000","last_active":"2024-08-17T17:17:32.495428+0000","last_peered":"2024-08-17T17:17:32.495428+0000","last_clean":"2024-08-17T17:17:32.495428+0000","last_became_active":"2024-08-17T17:17:32.494978+0000","last_became_peered":"2024-08-17T17:17:32.494978+0000","last_unstale":"2024-08-17T17:17:32.495428+0000","last_undegraded":"2024-08-17T17:17:32.495428+0000","last_fullsized":"2024-08-17T17:17:32.495428+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:42:24.647812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.484182+0000","last_change":"2024-08-17T17:16:26.961623+0000","last_active":"2024-08-17T17:17:32.484182+0000","last_peered":"2024-08-17T17:17:32.484182+0000","last_clean":"2024-08-17T17:17:32.484182+0000","last_became_active":"2024-08-17T17:16:26.961284+0000","last_became_peered":"2024-08-17T17:16:26.961284+0000","last_unstale":"2024-08-17T17:17:32.484182+0000","last_undegraded":"2024-08-17T17:17:32.484182+0000","last_fullsized":"2024-08-17T17:17:32.484182+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:27:32.176916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.497027+0000","last_change":"2024-08-17T17:17:32.496970+0000","last_active":"2024-08-17T17:17:32.497027+0000","last_peered":"2024-08-17T17:17:32.497027+0000","last_clean":"2024-08-17T17:17:32.497027+0000","last_became_active":"2024-08-17T17:17:32.496610+0000","last_became_peered":"2024-08-17T17:17:32.496610+0000","last_unstale":"2024-08-17T17:17:32.497027+0000","last_undegraded":"2024-08-17T17:17:32.497027+0000","last_fullsized":"2024-08-17T17:17:32.497027+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:03:29.830609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":67,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.494935+0000","last_change":"2024-08-17T17:17:32.494935+0000","last_active":"2024-08-17T17:17:32.494935+0000","last_peered":"2024-08-17T17:17:32.494935+0000","last_clean":"2024-08-17T17:17:32.494935+0000","last_became_active":"2024-08-17T17:17:32.494516+0000","last_became_peered":"2024-08-17T17:17:32.494516+0000","last_unstale":"2024-08-17T17:17:32.494935+0000","last_undegraded":"2024-08-17T17:17:32.494935+0000","last_fullsized":"2024-08-17T17:17:32.494935+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:31:21.680055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":68,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.492670+0000","last_change":"2024-08-17T17:16:26.963141+0000","last_active":"2024-08-17T17:17:31.492670+0000","last_peered":"2024-08-17T17:17:31.492670+0000","last_clean":"2024-08-17T17:17:31.492670+0000","last_became_active":"2024-08-17T17:16:26.962778+0000","last_became_peered":"2024-08-17T17:16:26.962778+0000","last_unstale":"2024-08-17T17:17:31.492670+0000","last_undegraded":"2024-08-17T17:17:31.492670+0000","last_fullsized":"2024-08-17T17:17:31.492670+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:56:40.295525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":84,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.489477+0000","last_change":"2024-08-17T17:17:00.758598+0000","last_active":"2024-08-17T17:17:31.489477+0000","last_peered":"2024-08-17T17:17:31.489477+0000","last_clean":"2024-08-17T17:17:31.489477+0000","last_became_active":"2024-08-17T17:17:00.757663+0000","last_became_peered":"2024-08-17T17:17:00.757663+0000","last_unstale":"2024-08-17T17:17:31.489477+0000","last_undegraded":"2024-08-17T17:17:31.489477+0000","last_fullsized":"2024-08-17T17:17:31.489477+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:16:15.398787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":68,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.490060+0000","last_change":"2024-08-17T17:17:00.758630+0000","last_active":"2024-08-17T17:17:31.490060+0000","last_peered":"2024-08-17T17:17:31.490060+0000","last_clean":"2024-08-17T17:17:31.490060+0000","last_became_active":"2024-08-17T17:17:00.757829+0000","last_became_peered":"2024-08-17T17:17:00.757829+0000","last_unstale":"2024-08-17T17:17:31.490060+0000","last_undegraded":"2024-08-17T17:17:31.490060+0000","last_fullsized":"2024-08-17T17:17:31.490060+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:17:35.583797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":67,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.488838+0000","last_change":"2024-08-17T17:16:26.962625+0000","last_active":"2024-08-17T17:17:31.488838+0000","last_peered":"2024-08-17T17:17:31.488838+0000","last_clean":"2024-08-17T17:17:31.488838+0000","last_became_active":"2024-08-17T17:16:26.962259+0000","last_became_peered":"2024-08-17T17:16:26.962259+0000","last_unstale":"2024-08-17T17:17:31.488838+0000","last_undegraded":"2024-08-17T17:17:31.488838+0000","last_fullsized":"2024-08-17T17:17:31.488838+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:29:34.545710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":72,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.496427+0000","last_change":"2024-08-17T17:17:32.496361+0000","last_active":"2024-08-17T17:17:32.496427+0000","last_peered":"2024-08-17T17:17:32.496427+0000","last_clean":"2024-08-17T17:17:32.496427+0000","last_became_active":"2024-08-17T17:17:32.496056+0000","last_became_peered":"2024-08-17T17:17:32.496056+0000","last_unstale":"2024-08-17T17:17:32.496427+0000","last_undegraded":"2024-08-17T17:17:32.496427+0000","last_fullsized":"2024-08-17T17:17:32.496427+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:11:20.404589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":54,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.495711+0000","last_change":"2024-08-17T17:17:32.495625+0000","last_active":"2024-08-17T17:17:32.495711+0000","last_peered":"2024-08-17T17:17:32.495711+0000","last_clean":"2024-08-17T17:17:32.495711+0000","last_became_active":"2024-08-17T17:17:32.495298+0000","last_became_peered":"2024-08-17T17:17:32.495298+0000","last_unstale":"2024-08-17T17:17:32.495711+0000","last_undegraded":"2024-08-17T17:17:32.495711+0000","last_fullsized":"2024-08-17T17:17:32.495711+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:18:34.358254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":410,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.486090+0000","last_change":"2024-08-17T17:15:54.354936+0000","last_active":"2024-08-17T17:17:31.486090+0000","last_peered":"2024-08-17T17:17:31.486090+0000","last_clean":"2024-08-17T17:17:31.486090+0000","last_became_active":"2024-08-17T17:15:54.354238+0000","last_became_peered":"2024-08-17T17:15:54.354238+0000","last_unstale":"2024-08-17T17:17:31.486090+0000","last_undegraded":"2024-08-17T17:17:31.486090+0000","last_fullsized":"2024-08-17T17:17:31.486090+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:04:01.141401+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:58:05.159624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":602,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.488040+0000","last_change":"2024-08-17T17:17:00.759124+0000","last_active":"2024-08-17T17:17:31.488040+0000","last_peered":"2024-08-17T17:17:31.488040+0000","last_clean":"2024-08-17T17:17:31.488040+0000","last_became_active":"2024-08-17T17:17:00.758605+0000","last_became_peered":"2024-08-17T17:17:00.758605+0000","last_unstale":"2024-08-17T17:17:31.488040+0000","last_undegraded":"2024-08-17T17:17:31.488040+0000","last_fullsized":"2024-08-17T17:17:31.488040+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:05:10.050836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.500056+0000","last_change":"2024-08-17T17:17:32.499994+0000","last_active":"2024-08-17T17:17:32.500056+0000","last_peered":"2024-08-17T17:17:32.500056+0000","last_clean":"2024-08-17T17:17:32.500056+0000","last_became_active":"2024-08-17T17:17:32.499614+0000","last_became_peered":"2024-08-17T17:17:32.499614+0000","last_unstale":"2024-08-17T17:17:32.500056+0000","last_undegraded":"2024-08-17T17:17:32.500056+0000","last_fullsized":"2024-08-17T17:17:32.500056+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:18:05.307880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":50,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.493323+0000","last_change":"2024-08-17T17:07:47.164375+0000","last_active":"2024-08-17T17:17:31.493323+0000","last_peered":"2024-08-17T17:17:31.493323+0000","last_clean":"2024-08-17T17:17:31.493323+0000","last_became_active":"2024-08-17T17:07:47.164077+0000","last_became_peered":"2024-08-17T17:07:47.164077+0000","last_unstale":"2024-08-17T17:17:31.493323+0000","last_undegraded":"2024-08-17T17:17:31.493323+0000","last_fullsized":"2024-08-17T17:17:31.493323+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:26:01.278743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":700,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.485964+0000","last_change":"2024-08-17T17:16:26.958415+0000","last_active":"2024-08-17T17:17:31.485964+0000","last_peered":"2024-08-17T17:17:31.485964+0000","last_clean":"2024-08-17T17:17:31.485964+0000","last_became_active":"2024-08-17T17:16:26.957956+0000","last_became_peered":"2024-08-17T17:16:26.957956+0000","last_unstale":"2024-08-17T17:17:31.485964+0000","last_undegraded":"2024-08-17T17:17:31.485964+0000","last_fullsized":"2024-08-17T17:17:31.485964+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:22.478077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.499100+0000","last_change":"2024-08-17T17:17:32.493840+0000","last_active":"2024-08-17T17:17:32.499100+0000","last_peered":"2024-08-17T17:17:32.499100+0000","last_clean":"2024-08-17T17:17:32.499100+0000","last_became_active":"2024-08-17T17:17:32.493503+0000","last_became_peered":"2024-08-17T17:17:32.493503+0000","last_unstale":"2024-08-17T17:17:32.499100+0000","last_undegraded":"2024-08-17T17:17:32.499100+0000","last_fullsized":"2024-08-17T17:17:32.499100+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:01:06.908354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":79,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.484970+0000","last_change":"2024-08-17T17:17:00.761104+0000","last_active":"2024-08-17T17:17:32.484970+0000","last_peered":"2024-08-17T17:17:32.484970+0000","last_clean":"2024-08-17T17:17:32.484970+0000","last_became_active":"2024-08-17T17:17:00.760721+0000","last_became_peered":"2024-08-17T17:17:00.760721+0000","last_unstale":"2024-08-17T17:17:32.484970+0000","last_undegraded":"2024-08-17T17:17:32.484970+0000","last_fullsized":"2024-08-17T17:17:32.484970+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:06:49.900957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":555,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.495429+0000","last_change":"2024-08-17T17:17:32.495364+0000","last_active":"2024-08-17T17:17:32.495429+0000","last_peered":"2024-08-17T17:17:32.495429+0000","last_clean":"2024-08-17T17:17:32.495429+0000","last_became_active":"2024-08-17T17:17:32.494801+0000","last_became_peered":"2024-08-17T17:17:32.494801+0000","last_unstale":"2024-08-17T17:17:32.495429+0000","last_undegraded":"2024-08-17T17:17:32.495429+0000","last_fullsized":"2024-08-17T17:17:32.495429+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:11:06.683576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":55,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.488438+0000","last_change":"2024-08-17T17:17:00.758552+0000","last_active":"2024-08-17T17:17:32.488438+0000","last_peered":"2024-08-17T17:17:32.488438+0000","last_clean":"2024-08-17T17:17:32.488438+0000","last_became_active":"2024-08-17T17:17:00.758200+0000","last_became_peered":"2024-08-17T17:17:00.758200+0000","last_unstale":"2024-08-17T17:17:32.488438+0000","last_undegraded":"2024-08-17T17:17:32.488438+0000","last_fullsized":"2024-08-17T17:17:32.488438+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:13:27.648231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":86,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.488077+0000","last_change":"2024-08-17T17:16:26.964140+0000","last_active":"2024-08-17T17:17:32.488077+0000","last_peered":"2024-08-17T17:17:32.488077+0000","last_clean":"2024-08-17T17:17:32.488077+0000","last_became_active":"2024-08-17T17:16:26.963807+0000","last_became_peered":"2024-08-17T17:16:26.963807+0000","last_unstale":"2024-08-17T17:17:32.488077+0000","last_undegraded":"2024-08-17T17:17:32.488077+0000","last_fullsized":"2024-08-17T17:17:32.488077+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:30:50.386236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":936,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.487871+0000","last_change":"2024-08-17T17:16:26.965149+0000","last_active":"2024-08-17T17:17:31.487871+0000","last_peered":"2024-08-17T17:17:31.487871+0000","last_clean":"2024-08-17T17:17:31.487871+0000","last_became_active":"2024-08-17T17:16:26.964674+0000","last_became_peered":"2024-08-17T17:16:26.964674+0000","last_unstale":"2024-08-17T17:17:31.487871+0000","last_undegraded":"2024-08-17T17:17:31.487871+0000","last_fullsized":"2024-08-17T17:17:31.487871+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:28:57.549403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1392,"num_read_kb":2863,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":68,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.490674+0000","last_change":"2024-08-17T17:15:54.359640+0000","last_active":"2024-08-17T17:17:31.490674+0000","last_peered":"2024-08-17T17:17:31.490674+0000","last_clean":"2024-08-17T17:17:31.490674+0000","last_became_active":"2024-08-17T17:15:54.359163+0000","last_became_peered":"2024-08-17T17:15:54.359163+0000","last_unstale":"2024-08-17T17:17:31.490674+0000","last_undegraded":"2024-08-17T17:17:31.490674+0000","last_fullsized":"2024-08-17T17:17:31.490674+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:13:52.163231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.500703+0000","last_change":"2024-08-17T17:17:32.499715+0000","last_active":"2024-08-17T17:17:32.500703+0000","last_peered":"2024-08-17T17:17:32.500703+0000","last_clean":"2024-08-17T17:17:32.500703+0000","last_became_active":"2024-08-17T17:17:32.499412+0000","last_became_peered":"2024-08-17T17:17:32.499412+0000","last_unstale":"2024-08-17T17:17:32.500703+0000","last_undegraded":"2024-08-17T17:17:32.500703+0000","last_fullsized":"2024-08-17T17:17:32.500703+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:39:38.456970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":85,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.495728+0000","last_change":"2024-08-17T17:17:32.495728+0000","last_active":"2024-08-17T17:17:32.495728+0000","last_peered":"2024-08-17T17:17:32.495728+0000","last_clean":"2024-08-17T17:17:32.495728+0000","last_became_active":"2024-08-17T17:17:32.495357+0000","last_became_peered":"2024-08-17T17:17:32.495357+0000","last_unstale":"2024-08-17T17:17:32.495728+0000","last_undegraded":"2024-08-17T17:17:32.495728+0000","last_fullsized":"2024-08-17T17:17:32.495728+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:27:45.561567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":302,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.488216+0000","last_change":"2024-08-17T17:16:26.964621+0000","last_active":"2024-08-17T17:17:32.488216+0000","last_peered":"2024-08-17T17:17:32.488216+0000","last_clean":"2024-08-17T17:17:32.488216+0000","last_became_active":"2024-08-17T17:16:26.964058+0000","last_became_peered":"2024-08-17T17:16:26.964058+0000","last_unstale":"2024-08-17T17:17:32.488216+0000","last_undegraded":"2024-08-17T17:17:32.488216+0000","last_fullsized":"2024-08-17T17:17:32.488216+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:00:04.465644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":44,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.488723+0000","last_change":"2024-08-17T17:16:26.965872+0000","last_active":"2024-08-17T17:17:32.488723+0000","last_peered":"2024-08-17T17:17:32.488723+0000","last_clean":"2024-08-17T17:17:32.488723+0000","last_became_active":"2024-08-17T17:16:26.965580+0000","last_became_peered":"2024-08-17T17:16:26.965580+0000","last_unstale":"2024-08-17T17:17:32.488723+0000","last_undegraded":"2024-08-17T17:17:32.488723+0000","last_fullsized":"2024-08-17T17:17:32.488723+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:21:52.311284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":55,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.488106+0000","last_change":"2024-08-17T17:17:00.758843+0000","last_active":"2024-08-17T17:17:32.488106+0000","last_peered":"2024-08-17T17:17:32.488106+0000","last_clean":"2024-08-17T17:17:32.488106+0000","last_became_active":"2024-08-17T17:17:00.758350+0000","last_became_peered":"2024-08-17T17:17:00.758350+0000","last_unstale":"2024-08-17T17:17:32.488106+0000","last_undegraded":"2024-08-17T17:17:32.488106+0000","last_fullsized":"2024-08-17T17:17:32.488106+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:36:47.412787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":411,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.487103+0000","last_change":"2024-08-17T17:15:54.358998+0000","last_active":"2024-08-17T17:17:31.487103+0000","last_peered":"2024-08-17T17:17:31.487103+0000","last_clean":"2024-08-17T17:17:31.487103+0000","last_became_active":"2024-08-17T17:15:54.358141+0000","last_became_peered":"2024-08-17T17:15:54.358141+0000","last_unstale":"2024-08-17T17:17:31.487103+0000","last_undegraded":"2024-08-17T17:17:31.487103+0000","last_fullsized":"2024-08-17T17:17:31.487103+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:32:08.078428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":55,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.484086+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:17:32.484086+0000","last_peered":"2024-08-17T17:17:32.484086+0000","last_clean":"2024-08-17T17:17:32.484086+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:17:32.484086+0000","last_undegraded":"2024-08-17T17:17:32.484086+0000","last_fullsized":"2024-08-17T17:17:32.484086+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.496087+0000","last_change":"2024-08-17T17:17:32.496023+0000","last_active":"2024-08-17T17:17:32.496087+0000","last_peered":"2024-08-17T17:17:32.496087+0000","last_clean":"2024-08-17T17:17:32.496087+0000","last_became_active":"2024-08-17T17:17:32.495652+0000","last_became_peered":"2024-08-17T17:17:32.495652+0000","last_unstale":"2024-08-17T17:17:32.496087+0000","last_undegraded":"2024-08-17T17:17:32.496087+0000","last_fullsized":"2024-08-17T17:17:32.496087+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:42:52.056527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":565,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.488091+0000","last_change":"2024-08-17T17:16:26.967134+0000","last_active":"2024-08-17T17:17:32.488091+0000","last_peered":"2024-08-17T17:17:32.488091+0000","last_clean":"2024-08-17T17:17:32.488091+0000","last_became_active":"2024-08-17T17:16:26.966517+0000","last_became_peered":"2024-08-17T17:16:26.966517+0000","last_unstale":"2024-08-17T17:17:32.488091+0000","last_undegraded":"2024-08-17T17:17:32.488091+0000","last_fullsized":"2024-08-17T17:17:32.488091+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:38:52.366564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.501049+0000","last_change":"2024-08-17T17:17:32.500970+0000","last_active":"2024-08-17T17:17:32.501049+0000","last_peered":"2024-08-17T17:17:32.501049+0000","last_clean":"2024-08-17T17:17:32.501049+0000","last_became_active":"2024-08-17T17:17:32.499033+0000","last_became_peered":"2024-08-17T17:17:32.499033+0000","last_unstale":"2024-08-17T17:17:32.501049+0000","last_undegraded":"2024-08-17T17:17:32.501049+0000","last_fullsized":"2024-08-17T17:17:32.501049+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:26:01.543899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.484380+0000","last_change":"2024-08-17T17:16:26.962885+0000","last_active":"2024-08-17T17:17:32.484380+0000","last_peered":"2024-08-17T17:17:32.484380+0000","last_clean":"2024-08-17T17:17:32.484380+0000","last_became_active":"2024-08-17T17:16:26.961919+0000","last_became_peered":"2024-08-17T17:16:26.961919+0000","last_unstale":"2024-08-17T17:17:32.484380+0000","last_undegraded":"2024-08-17T17:17:32.484380+0000","last_fullsized":"2024-08-17T17:17:32.484380+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:03:43.614359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":291,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.492024+0000","last_change":"2024-08-17T17:17:00.755772+0000","last_active":"2024-08-17T17:17:31.492024+0000","last_peered":"2024-08-17T17:17:31.492024+0000","last_clean":"2024-08-17T17:17:31.492024+0000","last_became_active":"2024-08-17T17:17:00.755281+0000","last_became_peered":"2024-08-17T17:17:00.755281+0000","last_unstale":"2024-08-17T17:17:31.492024+0000","last_undegraded":"2024-08-17T17:17:31.492024+0000","last_fullsized":"2024-08-17T17:17:31.492024+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:56:54.600755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":50,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.488953+0000","last_change":"2024-08-17T17:07:47.227287+0000","last_active":"2024-08-17T17:17:31.488953+0000","last_peered":"2024-08-17T17:17:31.488953+0000","last_clean":"2024-08-17T17:17:31.488953+0000","last_became_active":"2024-08-17T17:07:47.226983+0000","last_became_peered":"2024-08-17T17:07:47.226983+0000","last_unstale":"2024-08-17T17:17:31.488953+0000","last_undegraded":"2024-08-17T17:17:31.488953+0000","last_fullsized":"2024-08-17T17:17:31.488953+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:34:07.444142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":67,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.490105+0000","last_change":"2024-08-17T17:16:26.961292+0000","last_active":"2024-08-17T17:17:31.490105+0000","last_peered":"2024-08-17T17:17:31.490105+0000","last_clean":"2024-08-17T17:17:31.490105+0000","last_became_active":"2024-08-17T17:16:26.960977+0000","last_became_peered":"2024-08-17T17:16:26.960977+0000","last_unstale":"2024-08-17T17:17:31.490105+0000","last_undegraded":"2024-08-17T17:17:31.490105+0000","last_fullsized":"2024-08-17T17:17:31.490105+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:58:50.075227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":73,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.489768+0000","last_change":"2024-08-17T17:07:46.160694+0000","last_active":"2024-08-17T17:17:31.489768+0000","last_peered":"2024-08-17T17:17:31.489768+0000","last_clean":"2024-08-17T17:17:31.489768+0000","last_became_active":"2024-08-17T17:07:46.160407+0000","last_became_peered":"2024-08-17T17:07:46.160407+0000","last_unstale":"2024-08-17T17:17:31.489768+0000","last_undegraded":"2024-08-17T17:17:31.489768+0000","last_fullsized":"2024-08-17T17:17:31.489768+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:54:38.366094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":54,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.487584+0000","last_change":"2024-08-17T17:17:00.756909+0000","last_active":"2024-08-17T17:17:32.487584+0000","last_peered":"2024-08-17T17:17:32.487584+0000","last_clean":"2024-08-17T17:17:32.487584+0000","last_became_active":"2024-08-17T17:17:00.756532+0000","last_became_peered":"2024-08-17T17:17:00.756532+0000","last_unstale":"2024-08-17T17:17:32.487584+0000","last_undegraded":"2024-08-17T17:17:32.487584+0000","last_fullsized":"2024-08-17T17:17:32.487584+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:11:25.294907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.496800+0000","last_change":"2024-08-17T17:17:32.496800+0000","last_active":"2024-08-17T17:17:32.496800+0000","last_peered":"2024-08-17T17:17:32.496800+0000","last_clean":"2024-08-17T17:17:32.496800+0000","last_became_active":"2024-08-17T17:17:32.496405+0000","last_became_peered":"2024-08-17T17:17:32.496405+0000","last_unstale":"2024-08-17T17:17:32.496800+0000","last_undegraded":"2024-08-17T17:17:32.496800+0000","last_fullsized":"2024-08-17T17:17:32.496800+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:03:56.618806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":68,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.489488+0000","last_change":"2024-08-17T17:16:26.964939+0000","last_active":"2024-08-17T17:17:31.489488+0000","last_peered":"2024-08-17T17:17:31.489488+0000","last_clean":"2024-08-17T17:17:31.489488+0000","last_became_active":"2024-08-17T17:16:26.964598+0000","last_became_peered":"2024-08-17T17:16:26.964598+0000","last_unstale":"2024-08-17T17:17:31.489488+0000","last_undegraded":"2024-08-17T17:17:31.489488+0000","last_fullsized":"2024-08-17T17:17:31.489488+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:01:09.949950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":24,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.501097+0000","last_change":"2024-08-17T17:17:32.501043+0000","last_active":"2024-08-17T17:17:32.501097+0000","last_peered":"2024-08-17T17:17:32.501097+0000","last_clean":"2024-08-17T17:17:32.501097+0000","last_became_active":"2024-08-17T17:17:32.499400+0000","last_became_peered":"2024-08-17T17:17:32.499400+0000","last_unstale":"2024-08-17T17:17:32.501097+0000","last_undegraded":"2024-08-17T17:17:32.501097+0000","last_fullsized":"2024-08-17T17:17:32.501097+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:47:35.040523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.496833+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:17:32.496833+0000","last_peered":"2024-08-17T17:17:32.496833+0000","last_clean":"2024-08-17T17:17:32.496833+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:17:32.496833+0000","last_undegraded":"2024-08-17T17:17:32.496833+0000","last_fullsized":"2024-08-17T17:17:32.496833+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.501087+0000","last_change":"2024-08-17T17:17:32.501012+0000","last_active":"2024-08-17T17:17:32.501087+0000","last_peered":"2024-08-17T17:17:32.501087+0000","last_clean":"2024-08-17T17:17:32.501087+0000","last_became_active":"2024-08-17T17:17:32.499399+0000","last_became_peered":"2024-08-17T17:17:32.499399+0000","last_unstale":"2024-08-17T17:17:32.501087+0000","last_undegraded":"2024-08-17T17:17:32.501087+0000","last_fullsized":"2024-08-17T17:17:32.501087+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:19:57.035025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":67,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.491928+0000","last_change":"2024-08-17T17:16:26.963696+0000","last_active":"2024-08-17T17:17:31.491928+0000","last_peered":"2024-08-17T17:17:31.491928+0000","last_clean":"2024-08-17T17:17:31.491928+0000","last_became_active":"2024-08-17T17:16:26.963280+0000","last_became_peered":"2024-08-17T17:16:26.963280+0000","last_unstale":"2024-08-17T17:17:31.491928+0000","last_undegraded":"2024-08-17T17:17:31.491928+0000","last_fullsized":"2024-08-17T17:17:31.491928+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:22:01.474840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.500549+0000","last_change":"2024-08-17T17:17:32.500494+0000","last_active":"2024-08-17T17:17:32.500549+0000","last_peered":"2024-08-17T17:17:32.500549+0000","last_clean":"2024-08-17T17:17:32.500549+0000","last_became_active":"2024-08-17T17:17:32.498949+0000","last_became_peered":"2024-08-17T17:17:32.498949+0000","last_unstale":"2024-08-17T17:17:32.500549+0000","last_undegraded":"2024-08-17T17:17:32.500549+0000","last_fullsized":"2024-08-17T17:17:32.500549+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:44.681212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.498006+0000","last_change":"2024-08-17T17:17:32.495908+0000","last_active":"2024-08-17T17:17:32.498006+0000","last_peered":"2024-08-17T17:17:32.498006+0000","last_clean":"2024-08-17T17:17:32.498006+0000","last_became_active":"2024-08-17T17:17:32.495593+0000","last_became_peered":"2024-08-17T17:17:32.495593+0000","last_unstale":"2024-08-17T17:17:32.498006+0000","last_undegraded":"2024-08-17T17:17:32.498006+0000","last_fullsized":"2024-08-17T17:17:32.498006+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:41:39.009044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":55,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.498046+0000","last_change":"2024-08-17T17:17:32.497986+0000","last_active":"2024-08-17T17:17:32.498046+0000","last_peered":"2024-08-17T17:17:32.498046+0000","last_clean":"2024-08-17T17:17:32.498046+0000","last_became_active":"2024-08-17T17:17:32.497639+0000","last_became_peered":"2024-08-17T17:17:32.497639+0000","last_unstale":"2024-08-17T17:17:32.498046+0000","last_undegraded":"2024-08-17T17:17:32.498046+0000","last_fullsized":"2024-08-17T17:17:32.498046+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:03:25.854983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":85,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.495810+0000","last_change":"2024-08-17T17:17:32.495727+0000","last_active":"2024-08-17T17:17:32.495810+0000","last_peered":"2024-08-17T17:17:32.495810+0000","last_clean":"2024-08-17T17:17:32.495810+0000","last_became_active":"2024-08-17T17:17:32.494437+0000","last_became_peered":"2024-08-17T17:17:32.494437+0000","last_unstale":"2024-08-17T17:17:32.495810+0000","last_undegraded":"2024-08-17T17:17:32.495810+0000","last_fullsized":"2024-08-17T17:17:32.495810+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:56:22.902797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":86,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.489170+0000","last_change":"2024-08-17T17:17:00.758212+0000","last_active":"2024-08-17T17:17:31.489170+0000","last_peered":"2024-08-17T17:17:31.489170+0000","last_clean":"2024-08-17T17:17:31.489170+0000","last_became_active":"2024-08-17T17:17:00.757787+0000","last_became_peered":"2024-08-17T17:17:00.757787+0000","last_unstale":"2024-08-17T17:17:31.489170+0000","last_undegraded":"2024-08-17T17:17:31.489170+0000","last_fullsized":"2024-08-17T17:17:31.489170+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:19:11.289350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.495253+0000","last_change":"2024-08-17T17:17:32.495192+0000","last_active":"2024-08-17T17:17:32.495253+0000","last_peered":"2024-08-17T17:17:32.495253+0000","last_clean":"2024-08-17T17:17:32.495253+0000","last_became_active":"2024-08-17T17:17:32.494866+0000","last_became_peered":"2024-08-17T17:17:32.494866+0000","last_unstale":"2024-08-17T17:17:32.495253+0000","last_undegraded":"2024-08-17T17:17:32.495253+0000","last_fullsized":"2024-08-17T17:17:32.495253+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:21:13.274144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":58,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.483940+0000","last_change":"2024-08-17T17:15:54.357458+0000","last_active":"2024-08-17T17:17:32.483940+0000","last_peered":"2024-08-17T17:17:32.483940+0000","last_clean":"2024-08-17T17:17:32.483940+0000","last_became_active":"2024-08-17T17:15:54.356621+0000","last_became_peered":"2024-08-17T17:15:54.356621+0000","last_unstale":"2024-08-17T17:17:32.483940+0000","last_undegraded":"2024-08-17T17:17:32.483940+0000","last_fullsized":"2024-08-17T17:17:32.483940+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:16:10.254328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":84,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.493417+0000","last_change":"2024-08-17T17:17:32.493417+0000","last_active":"2024-08-17T17:17:32.493417+0000","last_peered":"2024-08-17T17:17:32.493417+0000","last_clean":"2024-08-17T17:17:32.493417+0000","last_became_active":"2024-08-17T17:17:32.493097+0000","last_became_peered":"2024-08-17T17:17:32.493097+0000","last_unstale":"2024-08-17T17:17:32.493417+0000","last_undegraded":"2024-08-17T17:17:32.493417+0000","last_fullsized":"2024-08-17T17:17:32.493417+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:22:01.622261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":70,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.488375+0000","last_change":"2024-08-17T17:16:26.964339+0000","last_active":"2024-08-17T17:17:31.488375+0000","last_peered":"2024-08-17T17:17:31.488375+0000","last_clean":"2024-08-17T17:17:31.488375+0000","last_became_active":"2024-08-17T17:16:26.963964+0000","last_became_peered":"2024-08-17T17:16:26.963964+0000","last_unstale":"2024-08-17T17:17:31.488375+0000","last_undegraded":"2024-08-17T17:17:31.488375+0000","last_fullsized":"2024-08-17T17:17:31.488375+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:24:14.417213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":84,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.492899+0000","last_change":"2024-08-17T17:17:32.492899+0000","last_active":"2024-08-17T17:17:32.492899+0000","last_peered":"2024-08-17T17:17:32.492899+0000","last_clean":"2024-08-17T17:17:32.492899+0000","last_became_active":"2024-08-17T17:17:32.492522+0000","last_became_peered":"2024-08-17T17:17:32.492522+0000","last_unstale":"2024-08-17T17:17:32.492899+0000","last_undegraded":"2024-08-17T17:17:32.492899+0000","last_fullsized":"2024-08-17T17:17:32.492899+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:27:57.515343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":53,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.490249+0000","last_change":"2024-08-17T17:07:46.156750+0000","last_active":"2024-08-17T17:17:31.490249+0000","last_peered":"2024-08-17T17:17:31.490249+0000","last_clean":"2024-08-17T17:17:31.490249+0000","last_became_active":"2024-08-17T17:07:46.156253+0000","last_became_peered":"2024-08-17T17:07:46.156253+0000","last_unstale":"2024-08-17T17:17:31.490249+0000","last_undegraded":"2024-08-17T17:17:31.490249+0000","last_fullsized":"2024-08-17T17:17:31.490249+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:04:28.549130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":92,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.499059+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:17:32.499059+0000","last_peered":"2024-08-17T17:17:32.499059+0000","last_clean":"2024-08-17T17:17:32.499059+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:17:32.499059+0000","last_undegraded":"2024-08-17T17:17:32.499059+0000","last_fullsized":"2024-08-17T17:17:32.499059+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":77,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.500444+0000","last_change":"2024-08-17T17:17:32.500387+0000","last_active":"2024-08-17T17:17:32.500444+0000","last_peered":"2024-08-17T17:17:32.500444+0000","last_clean":"2024-08-17T17:17:32.500444+0000","last_became_active":"2024-08-17T17:17:32.500062+0000","last_became_peered":"2024-08-17T17:17:32.500062+0000","last_unstale":"2024-08-17T17:17:32.500444+0000","last_undegraded":"2024-08-17T17:17:32.500444+0000","last_fullsized":"2024-08-17T17:17:32.500444+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:12:25.410392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":77,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.488249+0000","last_change":"2024-08-17T17:17:00.758339+0000","last_active":"2024-08-17T17:17:31.488249+0000","last_peered":"2024-08-17T17:17:31.488249+0000","last_clean":"2024-08-17T17:17:31.488249+0000","last_became_active":"2024-08-17T17:17:00.757987+0000","last_became_peered":"2024-08-17T17:17:00.757987+0000","last_unstale":"2024-08-17T17:17:31.488249+0000","last_undegraded":"2024-08-17T17:17:31.488249+0000","last_fullsized":"2024-08-17T17:17:31.488249+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:55:40.761984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.484386+0000","last_change":"2024-08-17T17:17:00.760208+0000","last_active":"2024-08-17T17:17:32.484386+0000","last_peered":"2024-08-17T17:17:32.484386+0000","last_clean":"2024-08-17T17:17:32.484386+0000","last_became_active":"2024-08-17T17:17:00.759800+0000","last_became_peered":"2024-08-17T17:17:00.759800+0000","last_unstale":"2024-08-17T17:17:32.484386+0000","last_undegraded":"2024-08-17T17:17:32.484386+0000","last_fullsized":"2024-08-17T17:17:32.484386+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:01:36.990816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17971,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234141,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307404,"kb_used_data":5880,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93437748,"statfs":{"total":95995035648,"available":95680253952,"internally_reserved":0,"allocated":6021120,"data_stored":5415758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495776,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307388,"kb_used_data":5868,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93437764,"statfs":{"total":95995035648,"available":95680270336,"internally_reserved":0,"allocated":6008832,"data_stored":5415424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757413,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298552,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446600,"statfs":{"total":95995035648,"available":95689318400,"internally_reserved":0,"allocated":1216512,"data_stored":626785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019051,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302720,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93442432,"statfs":{"total":95995035648,"available":95685050368,"internally_reserved":0,"allocated":1232896,"data_stored":631339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435279,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28660,"kb_used_data":1184,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":93716492,"statfs":{"total":95995035648,"available":95965687808,"internally_reserved":0,"allocated":1212416,"data_stored":626559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":28109834},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676598805,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36980,"kb_used_data":5284,"kb_used_omap":7,"kb_used_meta":31672,"kb_avail":93708172,"statfs":{"total":95995035648,"available":95957168128,"internally_reserved":0,"allocated":5410816,"data_stored":4821385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":32432290},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762332,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32884,"kb_used_data":1184,"kb_used_omap":7,"kb_used_meta":31672,"kb_avail":93712268,"statfs":{"total":95995035648,"available":95961362432,"internally_reserved":0,"allocated":1212416,"data_stored":626504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7945,"internal_metadata":32432375},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726925859,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":29304,"kb_used_data":1768,"kb_used_omap":7,"kb_used_meta":27512,"kb_avail":93715848,"statfs":{"total":95995035648,"available":95965028352,"internally_reserved":0,"allocated":1810432,"data_stored":1217445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":28172609},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6241,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:18:39.090 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:18:39.358 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:18:39.701 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:39 smithi003 ceph-mon[103344]: pgmap v203: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:39.701 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:39 smithi003 ceph-mon[103344]: from='client.45068 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:18:39.701 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:39 smithi003 ceph-mon[100615]: pgmap v203: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:39.701 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:39 smithi003 ceph-mon[100615]: from='client.45068 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:18:39.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:39 smithi053 ceph-mon[89963]: pgmap v203: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:39.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:39 smithi053 ceph-mon[89963]: from='client.45068 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:18:40.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:18:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:18:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:18:41.175 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:18:41.176 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:18:41.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:41 smithi053 ceph-mon[89963]: pgmap v204: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:41.724 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:41 smithi003 ceph-mon[103344]: pgmap v204: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:41.724 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:41 smithi003 ceph-mon[100615]: pgmap v204: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:41.750 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":204,"stamp":"2024-08-17T17:18:39.690296+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790221,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":6390,"num_read_kb":13845,"num_write":1122,"num_write_kb":15039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1181,"ondisk_log_size":1181,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1343892,"kb_used_data":23560,"kb_used_omap":34,"kb_used_meta":1320157,"kb_avail":748617324,"statfs":{"total":767960285184,"available":766584139776,"internally_reserved":0,"allocated":24125440,"data_stored":19381199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35772,"internal_metadata":1351840836},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002636"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.495711+0000","last_change":"2024-08-17T17:17:32.495652+0000","last_active":"2024-08-17T17:17:32.495711+0000","last_peered":"2024-08-17T17:17:32.495711+0000","last_clean":"2024-08-17T17:17:32.495711+0000","last_became_active":"2024-08-17T17:17:32.495218+0000","last_became_peered":"2024-08-17T17:17:32.495218+0000","last_unstale":"2024-08-17T17:17:32.495711+0000","last_undegraded":"2024-08-17T17:17:32.495711+0000","last_fullsized":"2024-08-17T17:17:32.495711+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:11:45.821282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":70,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.485328+0000","last_change":"2024-08-17T17:17:00.758108+0000","last_active":"2024-08-17T17:17:31.485328+0000","last_peered":"2024-08-17T17:17:31.485328+0000","last_clean":"2024-08-17T17:17:31.485328+0000","last_became_active":"2024-08-17T17:17:00.757714+0000","last_became_peered":"2024-08-17T17:17:00.757714+0000","last_unstale":"2024-08-17T17:17:31.485328+0000","last_undegraded":"2024-08-17T17:17:31.485328+0000","last_fullsized":"2024-08-17T17:17:31.485328+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:13:47.351728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":67,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.495336+0000","last_change":"2024-08-17T17:17:32.495336+0000","last_active":"2024-08-17T17:17:32.495336+0000","last_peered":"2024-08-17T17:17:32.495336+0000","last_clean":"2024-08-17T17:17:32.495336+0000","last_became_active":"2024-08-17T17:17:32.494975+0000","last_became_peered":"2024-08-17T17:17:32.494975+0000","last_unstale":"2024-08-17T17:17:32.495336+0000","last_undegraded":"2024-08-17T17:17:32.495336+0000","last_fullsized":"2024-08-17T17:17:32.495336+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:51:24.050993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.497616+0000","last_change":"2024-08-17T17:17:32.494012+0000","last_active":"2024-08-17T17:17:32.497616+0000","last_peered":"2024-08-17T17:17:32.497616+0000","last_clean":"2024-08-17T17:17:32.497616+0000","last_became_active":"2024-08-17T17:17:32.493663+0000","last_became_peered":"2024-08-17T17:17:32.493663+0000","last_unstale":"2024-08-17T17:17:32.497616+0000","last_undegraded":"2024-08-17T17:17:32.497616+0000","last_fullsized":"2024-08-17T17:17:32.497616+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:05:05.713213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.497986+0000","last_change":"2024-08-17T17:17:32.494930+0000","last_active":"2024-08-17T17:17:32.497986+0000","last_peered":"2024-08-17T17:17:32.497986+0000","last_clean":"2024-08-17T17:17:32.497986+0000","last_became_active":"2024-08-17T17:17:32.494590+0000","last_became_peered":"2024-08-17T17:17:32.494590+0000","last_unstale":"2024-08-17T17:17:32.497986+0000","last_undegraded":"2024-08-17T17:17:32.497986+0000","last_fullsized":"2024-08-17T17:17:32.497986+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:23:16.376415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":51,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.485438+0000","last_change":"2024-08-17T17:07:46.158514+0000","last_active":"2024-08-17T17:17:31.485438+0000","last_peered":"2024-08-17T17:17:31.485438+0000","last_clean":"2024-08-17T17:17:31.485438+0000","last_became_active":"2024-08-17T17:07:46.158256+0000","last_became_peered":"2024-08-17T17:07:46.158256+0000","last_unstale":"2024-08-17T17:17:31.485438+0000","last_undegraded":"2024-08-17T17:17:31.485438+0000","last_fullsized":"2024-08-17T17:17:31.485438+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:30:44.486515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.494098+0000","last_change":"2024-08-17T17:17:32.494030+0000","last_active":"2024-08-17T17:17:32.494098+0000","last_peered":"2024-08-17T17:17:32.494098+0000","last_clean":"2024-08-17T17:17:32.494098+0000","last_became_active":"2024-08-17T17:17:32.493580+0000","last_became_peered":"2024-08-17T17:17:32.493580+0000","last_unstale":"2024-08-17T17:17:32.494098+0000","last_undegraded":"2024-08-17T17:17:32.494098+0000","last_fullsized":"2024-08-17T17:17:32.494098+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:23:14.505301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":76,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.492950+0000","last_change":"2024-08-17T17:15:54.349784+0000","last_active":"2024-08-17T17:17:31.492950+0000","last_peered":"2024-08-17T17:17:31.492950+0000","last_clean":"2024-08-17T17:17:31.492950+0000","last_became_active":"2024-08-17T17:15:54.349305+0000","last_became_peered":"2024-08-17T17:15:54.349305+0000","last_unstale":"2024-08-17T17:17:31.492950+0000","last_undegraded":"2024-08-17T17:17:31.492950+0000","last_fullsized":"2024-08-17T17:17:31.492950+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:36:27.291719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":69,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.486671+0000","last_change":"2024-08-17T17:15:54.358793+0000","last_active":"2024-08-17T17:17:31.486671+0000","last_peered":"2024-08-17T17:17:31.486671+0000","last_clean":"2024-08-17T17:17:31.486671+0000","last_became_active":"2024-08-17T17:15:54.358280+0000","last_became_peered":"2024-08-17T17:15:54.358280+0000","last_unstale":"2024-08-17T17:17:31.486671+0000","last_undegraded":"2024-08-17T17:17:31.486671+0000","last_fullsized":"2024-08-17T17:17:31.486671+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:56:46.173026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":50,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.487824+0000","last_change":"2024-08-17T17:07:46.299427+0000","last_active":"2024-08-17T17:17:31.487824+0000","last_peered":"2024-08-17T17:17:31.487824+0000","last_clean":"2024-08-17T17:17:31.487824+0000","last_became_active":"2024-08-17T17:07:46.299008+0000","last_became_peered":"2024-08-17T17:07:46.299008+0000","last_unstale":"2024-08-17T17:17:31.487824+0000","last_undegraded":"2024-08-17T17:17:31.487824+0000","last_fullsized":"2024-08-17T17:17:31.487824+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:23:32.024874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":70,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.487123+0000","last_change":"2024-08-17T17:16:26.964049+0000","last_active":"2024-08-17T17:17:31.487123+0000","last_peered":"2024-08-17T17:17:31.487123+0000","last_clean":"2024-08-17T17:17:31.487123+0000","last_became_active":"2024-08-17T17:16:26.963554+0000","last_became_peered":"2024-08-17T17:16:26.963554+0000","last_unstale":"2024-08-17T17:17:31.487123+0000","last_undegraded":"2024-08-17T17:17:31.487123+0000","last_fullsized":"2024-08-17T17:17:31.487123+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:39:29.363488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":84,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.494445+0000","last_change":"2024-08-17T17:17:32.494445+0000","last_active":"2024-08-17T17:17:32.494445+0000","last_peered":"2024-08-17T17:17:32.494445+0000","last_clean":"2024-08-17T17:17:32.494445+0000","last_became_active":"2024-08-17T17:17:32.494080+0000","last_became_peered":"2024-08-17T17:17:32.494080+0000","last_unstale":"2024-08-17T17:17:32.494445+0000","last_undegraded":"2024-08-17T17:17:32.494445+0000","last_fullsized":"2024-08-17T17:17:32.494445+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:39:58.938883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":52,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.486546+0000","last_change":"2024-08-17T17:07:46.153923+0000","last_active":"2024-08-17T17:17:31.486546+0000","last_peered":"2024-08-17T17:17:31.486546+0000","last_clean":"2024-08-17T17:17:31.486546+0000","last_became_active":"2024-08-17T17:07:46.153614+0000","last_became_peered":"2024-08-17T17:07:46.153614+0000","last_unstale":"2024-08-17T17:17:31.486546+0000","last_undegraded":"2024-08-17T17:17:31.486546+0000","last_fullsized":"2024-08-17T17:17:31.486546+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:24:38.871907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":67,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.492465+0000","last_change":"2024-08-17T17:16:26.965847+0000","last_active":"2024-08-17T17:17:31.492465+0000","last_peered":"2024-08-17T17:17:31.492465+0000","last_clean":"2024-08-17T17:17:31.492465+0000","last_became_active":"2024-08-17T17:16:26.965522+0000","last_became_peered":"2024-08-17T17:16:26.965522+0000","last_unstale":"2024-08-17T17:17:31.492465+0000","last_undegraded":"2024-08-17T17:17:31.492465+0000","last_fullsized":"2024-08-17T17:17:31.492465+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:39:20.536460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":68,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.495428+0000","last_change":"2024-08-17T17:17:32.495428+0000","last_active":"2024-08-17T17:17:32.495428+0000","last_peered":"2024-08-17T17:17:32.495428+0000","last_clean":"2024-08-17T17:17:32.495428+0000","last_became_active":"2024-08-17T17:17:32.494978+0000","last_became_peered":"2024-08-17T17:17:32.494978+0000","last_unstale":"2024-08-17T17:17:32.495428+0000","last_undegraded":"2024-08-17T17:17:32.495428+0000","last_fullsized":"2024-08-17T17:17:32.495428+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:42:24.647812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.484182+0000","last_change":"2024-08-17T17:16:26.961623+0000","last_active":"2024-08-17T17:17:32.484182+0000","last_peered":"2024-08-17T17:17:32.484182+0000","last_clean":"2024-08-17T17:17:32.484182+0000","last_became_active":"2024-08-17T17:16:26.961284+0000","last_became_peered":"2024-08-17T17:16:26.961284+0000","last_unstale":"2024-08-17T17:17:32.484182+0000","last_undegraded":"2024-08-17T17:17:32.484182+0000","last_fullsized":"2024-08-17T17:17:32.484182+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:27:32.176916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.497027+0000","last_change":"2024-08-17T17:17:32.496970+0000","last_active":"2024-08-17T17:17:32.497027+0000","last_peered":"2024-08-17T17:17:32.497027+0000","last_clean":"2024-08-17T17:17:32.497027+0000","last_became_active":"2024-08-17T17:17:32.496610+0000","last_became_peered":"2024-08-17T17:17:32.496610+0000","last_unstale":"2024-08-17T17:17:32.497027+0000","last_undegraded":"2024-08-17T17:17:32.497027+0000","last_fullsized":"2024-08-17T17:17:32.497027+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:03:29.830609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":67,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.494935+0000","last_change":"2024-08-17T17:17:32.494935+0000","last_active":"2024-08-17T17:17:32.494935+0000","last_peered":"2024-08-17T17:17:32.494935+0000","last_clean":"2024-08-17T17:17:32.494935+0000","last_became_active":"2024-08-17T17:17:32.494516+0000","last_became_peered":"2024-08-17T17:17:32.494516+0000","last_unstale":"2024-08-17T17:17:32.494935+0000","last_undegraded":"2024-08-17T17:17:32.494935+0000","last_fullsized":"2024-08-17T17:17:32.494935+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:31:21.680055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":68,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.492670+0000","last_change":"2024-08-17T17:16:26.963141+0000","last_active":"2024-08-17T17:17:31.492670+0000","last_peered":"2024-08-17T17:17:31.492670+0000","last_clean":"2024-08-17T17:17:31.492670+0000","last_became_active":"2024-08-17T17:16:26.962778+0000","last_became_peered":"2024-08-17T17:16:26.962778+0000","last_unstale":"2024-08-17T17:17:31.492670+0000","last_undegraded":"2024-08-17T17:17:31.492670+0000","last_fullsized":"2024-08-17T17:17:31.492670+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:56:40.295525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":84,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.489477+0000","last_change":"2024-08-17T17:17:00.758598+0000","last_active":"2024-08-17T17:17:31.489477+0000","last_peered":"2024-08-17T17:17:31.489477+0000","last_clean":"2024-08-17T17:17:31.489477+0000","last_became_active":"2024-08-17T17:17:00.757663+0000","last_became_peered":"2024-08-17T17:17:00.757663+0000","last_unstale":"2024-08-17T17:17:31.489477+0000","last_undegraded":"2024-08-17T17:17:31.489477+0000","last_fullsized":"2024-08-17T17:17:31.489477+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:16:15.398787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":68,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.490060+0000","last_change":"2024-08-17T17:17:00.758630+0000","last_active":"2024-08-17T17:17:31.490060+0000","last_peered":"2024-08-17T17:17:31.490060+0000","last_clean":"2024-08-17T17:17:31.490060+0000","last_became_active":"2024-08-17T17:17:00.757829+0000","last_became_peered":"2024-08-17T17:17:00.757829+0000","last_unstale":"2024-08-17T17:17:31.490060+0000","last_undegraded":"2024-08-17T17:17:31.490060+0000","last_fullsized":"2024-08-17T17:17:31.490060+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:17:35.583797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":67,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.488838+0000","last_change":"2024-08-17T17:16:26.962625+0000","last_active":"2024-08-17T17:17:31.488838+0000","last_peered":"2024-08-17T17:17:31.488838+0000","last_clean":"2024-08-17T17:17:31.488838+0000","last_became_active":"2024-08-17T17:16:26.962259+0000","last_became_peered":"2024-08-17T17:16:26.962259+0000","last_unstale":"2024-08-17T17:17:31.488838+0000","last_undegraded":"2024-08-17T17:17:31.488838+0000","last_fullsized":"2024-08-17T17:17:31.488838+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:29:34.545710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":72,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.496427+0000","last_change":"2024-08-17T17:17:32.496361+0000","last_active":"2024-08-17T17:17:32.496427+0000","last_peered":"2024-08-17T17:17:32.496427+0000","last_clean":"2024-08-17T17:17:32.496427+0000","last_became_active":"2024-08-17T17:17:32.496056+0000","last_became_peered":"2024-08-17T17:17:32.496056+0000","last_unstale":"2024-08-17T17:17:32.496427+0000","last_undegraded":"2024-08-17T17:17:32.496427+0000","last_fullsized":"2024-08-17T17:17:32.496427+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:11:20.404589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":55,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.496130+0000","last_change":"2024-08-17T17:17:32.495625+0000","last_active":"2024-08-17T17:17:32.496130+0000","last_peered":"2024-08-17T17:17:32.496130+0000","last_clean":"2024-08-17T17:17:32.496130+0000","last_became_active":"2024-08-17T17:17:32.495298+0000","last_became_peered":"2024-08-17T17:17:32.495298+0000","last_unstale":"2024-08-17T17:17:32.496130+0000","last_undegraded":"2024-08-17T17:17:32.496130+0000","last_fullsized":"2024-08-17T17:17:32.496130+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:18:34.358254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":410,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.486090+0000","last_change":"2024-08-17T17:15:54.354936+0000","last_active":"2024-08-17T17:17:31.486090+0000","last_peered":"2024-08-17T17:17:31.486090+0000","last_clean":"2024-08-17T17:17:31.486090+0000","last_became_active":"2024-08-17T17:15:54.354238+0000","last_became_peered":"2024-08-17T17:15:54.354238+0000","last_unstale":"2024-08-17T17:17:31.486090+0000","last_undegraded":"2024-08-17T17:17:31.486090+0000","last_fullsized":"2024-08-17T17:17:31.486090+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:04:01.141401+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:58:05.159624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":602,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.488040+0000","last_change":"2024-08-17T17:17:00.759124+0000","last_active":"2024-08-17T17:17:31.488040+0000","last_peered":"2024-08-17T17:17:31.488040+0000","last_clean":"2024-08-17T17:17:31.488040+0000","last_became_active":"2024-08-17T17:17:00.758605+0000","last_became_peered":"2024-08-17T17:17:00.758605+0000","last_unstale":"2024-08-17T17:17:31.488040+0000","last_undegraded":"2024-08-17T17:17:31.488040+0000","last_fullsized":"2024-08-17T17:17:31.488040+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:05:10.050836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.500056+0000","last_change":"2024-08-17T17:17:32.499994+0000","last_active":"2024-08-17T17:17:32.500056+0000","last_peered":"2024-08-17T17:17:32.500056+0000","last_clean":"2024-08-17T17:17:32.500056+0000","last_became_active":"2024-08-17T17:17:32.499614+0000","last_became_peered":"2024-08-17T17:17:32.499614+0000","last_unstale":"2024-08-17T17:17:32.500056+0000","last_undegraded":"2024-08-17T17:17:32.500056+0000","last_fullsized":"2024-08-17T17:17:32.500056+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:18:05.307880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":50,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.493323+0000","last_change":"2024-08-17T17:07:47.164375+0000","last_active":"2024-08-17T17:17:31.493323+0000","last_peered":"2024-08-17T17:17:31.493323+0000","last_clean":"2024-08-17T17:17:31.493323+0000","last_became_active":"2024-08-17T17:07:47.164077+0000","last_became_peered":"2024-08-17T17:07:47.164077+0000","last_unstale":"2024-08-17T17:17:31.493323+0000","last_undegraded":"2024-08-17T17:17:31.493323+0000","last_fullsized":"2024-08-17T17:17:31.493323+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:26:01.278743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":700,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.485964+0000","last_change":"2024-08-17T17:16:26.958415+0000","last_active":"2024-08-17T17:17:31.485964+0000","last_peered":"2024-08-17T17:17:31.485964+0000","last_clean":"2024-08-17T17:17:31.485964+0000","last_became_active":"2024-08-17T17:16:26.957956+0000","last_became_peered":"2024-08-17T17:16:26.957956+0000","last_unstale":"2024-08-17T17:17:31.485964+0000","last_undegraded":"2024-08-17T17:17:31.485964+0000","last_fullsized":"2024-08-17T17:17:31.485964+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:22.478077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.499100+0000","last_change":"2024-08-17T17:17:32.493840+0000","last_active":"2024-08-17T17:17:32.499100+0000","last_peered":"2024-08-17T17:17:32.499100+0000","last_clean":"2024-08-17T17:17:32.499100+0000","last_became_active":"2024-08-17T17:17:32.493503+0000","last_became_peered":"2024-08-17T17:17:32.493503+0000","last_unstale":"2024-08-17T17:17:32.499100+0000","last_undegraded":"2024-08-17T17:17:32.499100+0000","last_fullsized":"2024-08-17T17:17:32.499100+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:01:06.908354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":79,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.484970+0000","last_change":"2024-08-17T17:17:00.761104+0000","last_active":"2024-08-17T17:17:32.484970+0000","last_peered":"2024-08-17T17:17:32.484970+0000","last_clean":"2024-08-17T17:17:32.484970+0000","last_became_active":"2024-08-17T17:17:00.760721+0000","last_became_peered":"2024-08-17T17:17:00.760721+0000","last_unstale":"2024-08-17T17:17:32.484970+0000","last_undegraded":"2024-08-17T17:17:32.484970+0000","last_fullsized":"2024-08-17T17:17:32.484970+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:06:49.900957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":555,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.495429+0000","last_change":"2024-08-17T17:17:32.495364+0000","last_active":"2024-08-17T17:17:32.495429+0000","last_peered":"2024-08-17T17:17:32.495429+0000","last_clean":"2024-08-17T17:17:32.495429+0000","last_became_active":"2024-08-17T17:17:32.494801+0000","last_became_peered":"2024-08-17T17:17:32.494801+0000","last_unstale":"2024-08-17T17:17:32.495429+0000","last_undegraded":"2024-08-17T17:17:32.495429+0000","last_fullsized":"2024-08-17T17:17:32.495429+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:11:06.683576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":55,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.488438+0000","last_change":"2024-08-17T17:17:00.758552+0000","last_active":"2024-08-17T17:17:32.488438+0000","last_peered":"2024-08-17T17:17:32.488438+0000","last_clean":"2024-08-17T17:17:32.488438+0000","last_became_active":"2024-08-17T17:17:00.758200+0000","last_became_peered":"2024-08-17T17:17:00.758200+0000","last_unstale":"2024-08-17T17:17:32.488438+0000","last_undegraded":"2024-08-17T17:17:32.488438+0000","last_fullsized":"2024-08-17T17:17:32.488438+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:13:27.648231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":86,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.488077+0000","last_change":"2024-08-17T17:16:26.964140+0000","last_active":"2024-08-17T17:17:32.488077+0000","last_peered":"2024-08-17T17:17:32.488077+0000","last_clean":"2024-08-17T17:17:32.488077+0000","last_became_active":"2024-08-17T17:16:26.963807+0000","last_became_peered":"2024-08-17T17:16:26.963807+0000","last_unstale":"2024-08-17T17:17:32.488077+0000","last_undegraded":"2024-08-17T17:17:32.488077+0000","last_fullsized":"2024-08-17T17:17:32.488077+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:30:50.386236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":936,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.487871+0000","last_change":"2024-08-17T17:16:26.965149+0000","last_active":"2024-08-17T17:17:31.487871+0000","last_peered":"2024-08-17T17:17:31.487871+0000","last_clean":"2024-08-17T17:17:31.487871+0000","last_became_active":"2024-08-17T17:16:26.964674+0000","last_became_peered":"2024-08-17T17:16:26.964674+0000","last_unstale":"2024-08-17T17:17:31.487871+0000","last_undegraded":"2024-08-17T17:17:31.487871+0000","last_fullsized":"2024-08-17T17:17:31.487871+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:28:57.549403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1392,"num_read_kb":2863,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":68,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.490674+0000","last_change":"2024-08-17T17:15:54.359640+0000","last_active":"2024-08-17T17:17:31.490674+0000","last_peered":"2024-08-17T17:17:31.490674+0000","last_clean":"2024-08-17T17:17:31.490674+0000","last_became_active":"2024-08-17T17:15:54.359163+0000","last_became_peered":"2024-08-17T17:15:54.359163+0000","last_unstale":"2024-08-17T17:17:31.490674+0000","last_undegraded":"2024-08-17T17:17:31.490674+0000","last_fullsized":"2024-08-17T17:17:31.490674+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:13:52.163231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.500703+0000","last_change":"2024-08-17T17:17:32.499715+0000","last_active":"2024-08-17T17:17:32.500703+0000","last_peered":"2024-08-17T17:17:32.500703+0000","last_clean":"2024-08-17T17:17:32.500703+0000","last_became_active":"2024-08-17T17:17:32.499412+0000","last_became_peered":"2024-08-17T17:17:32.499412+0000","last_unstale":"2024-08-17T17:17:32.500703+0000","last_undegraded":"2024-08-17T17:17:32.500703+0000","last_fullsized":"2024-08-17T17:17:32.500703+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:39:38.456970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":85,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.495728+0000","last_change":"2024-08-17T17:17:32.495728+0000","last_active":"2024-08-17T17:17:32.495728+0000","last_peered":"2024-08-17T17:17:32.495728+0000","last_clean":"2024-08-17T17:17:32.495728+0000","last_became_active":"2024-08-17T17:17:32.495357+0000","last_became_peered":"2024-08-17T17:17:32.495357+0000","last_unstale":"2024-08-17T17:17:32.495728+0000","last_undegraded":"2024-08-17T17:17:32.495728+0000","last_fullsized":"2024-08-17T17:17:32.495728+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:27:45.561567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":302,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.488216+0000","last_change":"2024-08-17T17:16:26.964621+0000","last_active":"2024-08-17T17:17:32.488216+0000","last_peered":"2024-08-17T17:17:32.488216+0000","last_clean":"2024-08-17T17:17:32.488216+0000","last_became_active":"2024-08-17T17:16:26.964058+0000","last_became_peered":"2024-08-17T17:16:26.964058+0000","last_unstale":"2024-08-17T17:17:32.488216+0000","last_undegraded":"2024-08-17T17:17:32.488216+0000","last_fullsized":"2024-08-17T17:17:32.488216+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:00:04.465644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":44,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.488723+0000","last_change":"2024-08-17T17:16:26.965872+0000","last_active":"2024-08-17T17:17:32.488723+0000","last_peered":"2024-08-17T17:17:32.488723+0000","last_clean":"2024-08-17T17:17:32.488723+0000","last_became_active":"2024-08-17T17:16:26.965580+0000","last_became_peered":"2024-08-17T17:16:26.965580+0000","last_unstale":"2024-08-17T17:17:32.488723+0000","last_undegraded":"2024-08-17T17:17:32.488723+0000","last_fullsized":"2024-08-17T17:17:32.488723+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:21:52.311284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":55,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.488106+0000","last_change":"2024-08-17T17:17:00.758843+0000","last_active":"2024-08-17T17:17:32.488106+0000","last_peered":"2024-08-17T17:17:32.488106+0000","last_clean":"2024-08-17T17:17:32.488106+0000","last_became_active":"2024-08-17T17:17:00.758350+0000","last_became_peered":"2024-08-17T17:17:00.758350+0000","last_unstale":"2024-08-17T17:17:32.488106+0000","last_undegraded":"2024-08-17T17:17:32.488106+0000","last_fullsized":"2024-08-17T17:17:32.488106+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:36:47.412787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":411,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.487103+0000","last_change":"2024-08-17T17:15:54.358998+0000","last_active":"2024-08-17T17:17:31.487103+0000","last_peered":"2024-08-17T17:17:31.487103+0000","last_clean":"2024-08-17T17:17:31.487103+0000","last_became_active":"2024-08-17T17:15:54.358141+0000","last_became_peered":"2024-08-17T17:15:54.358141+0000","last_unstale":"2024-08-17T17:17:31.487103+0000","last_undegraded":"2024-08-17T17:17:31.487103+0000","last_fullsized":"2024-08-17T17:17:31.487103+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:32:08.078428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":55,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.484086+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:17:32.484086+0000","last_peered":"2024-08-17T17:17:32.484086+0000","last_clean":"2024-08-17T17:17:32.484086+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:17:32.484086+0000","last_undegraded":"2024-08-17T17:17:32.484086+0000","last_fullsized":"2024-08-17T17:17:32.484086+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.496087+0000","last_change":"2024-08-17T17:17:32.496023+0000","last_active":"2024-08-17T17:17:32.496087+0000","last_peered":"2024-08-17T17:17:32.496087+0000","last_clean":"2024-08-17T17:17:32.496087+0000","last_became_active":"2024-08-17T17:17:32.495652+0000","last_became_peered":"2024-08-17T17:17:32.495652+0000","last_unstale":"2024-08-17T17:17:32.496087+0000","last_undegraded":"2024-08-17T17:17:32.496087+0000","last_fullsized":"2024-08-17T17:17:32.496087+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:42:52.056527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":565,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.488091+0000","last_change":"2024-08-17T17:16:26.967134+0000","last_active":"2024-08-17T17:17:32.488091+0000","last_peered":"2024-08-17T17:17:32.488091+0000","last_clean":"2024-08-17T17:17:32.488091+0000","last_became_active":"2024-08-17T17:16:26.966517+0000","last_became_peered":"2024-08-17T17:16:26.966517+0000","last_unstale":"2024-08-17T17:17:32.488091+0000","last_undegraded":"2024-08-17T17:17:32.488091+0000","last_fullsized":"2024-08-17T17:17:32.488091+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:38:52.366564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.501049+0000","last_change":"2024-08-17T17:17:32.500970+0000","last_active":"2024-08-17T17:17:32.501049+0000","last_peered":"2024-08-17T17:17:32.501049+0000","last_clean":"2024-08-17T17:17:32.501049+0000","last_became_active":"2024-08-17T17:17:32.499033+0000","last_became_peered":"2024-08-17T17:17:32.499033+0000","last_unstale":"2024-08-17T17:17:32.501049+0000","last_undegraded":"2024-08-17T17:17:32.501049+0000","last_fullsized":"2024-08-17T17:17:32.501049+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:26:01.543899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.484380+0000","last_change":"2024-08-17T17:16:26.962885+0000","last_active":"2024-08-17T17:17:32.484380+0000","last_peered":"2024-08-17T17:17:32.484380+0000","last_clean":"2024-08-17T17:17:32.484380+0000","last_became_active":"2024-08-17T17:16:26.961919+0000","last_became_peered":"2024-08-17T17:16:26.961919+0000","last_unstale":"2024-08-17T17:17:32.484380+0000","last_undegraded":"2024-08-17T17:17:32.484380+0000","last_fullsized":"2024-08-17T17:17:32.484380+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:03:43.614359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":291,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.492024+0000","last_change":"2024-08-17T17:17:00.755772+0000","last_active":"2024-08-17T17:17:31.492024+0000","last_peered":"2024-08-17T17:17:31.492024+0000","last_clean":"2024-08-17T17:17:31.492024+0000","last_became_active":"2024-08-17T17:17:00.755281+0000","last_became_peered":"2024-08-17T17:17:00.755281+0000","last_unstale":"2024-08-17T17:17:31.492024+0000","last_undegraded":"2024-08-17T17:17:31.492024+0000","last_fullsized":"2024-08-17T17:17:31.492024+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:56:54.600755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":50,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.488953+0000","last_change":"2024-08-17T17:07:47.227287+0000","last_active":"2024-08-17T17:17:31.488953+0000","last_peered":"2024-08-17T17:17:31.488953+0000","last_clean":"2024-08-17T17:17:31.488953+0000","last_became_active":"2024-08-17T17:07:47.226983+0000","last_became_peered":"2024-08-17T17:07:47.226983+0000","last_unstale":"2024-08-17T17:17:31.488953+0000","last_undegraded":"2024-08-17T17:17:31.488953+0000","last_fullsized":"2024-08-17T17:17:31.488953+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:34:07.444142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":67,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.490105+0000","last_change":"2024-08-17T17:16:26.961292+0000","last_active":"2024-08-17T17:17:31.490105+0000","last_peered":"2024-08-17T17:17:31.490105+0000","last_clean":"2024-08-17T17:17:31.490105+0000","last_became_active":"2024-08-17T17:16:26.960977+0000","last_became_peered":"2024-08-17T17:16:26.960977+0000","last_unstale":"2024-08-17T17:17:31.490105+0000","last_undegraded":"2024-08-17T17:17:31.490105+0000","last_fullsized":"2024-08-17T17:17:31.490105+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:58:50.075227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":73,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.489768+0000","last_change":"2024-08-17T17:07:46.160694+0000","last_active":"2024-08-17T17:17:31.489768+0000","last_peered":"2024-08-17T17:17:31.489768+0000","last_clean":"2024-08-17T17:17:31.489768+0000","last_became_active":"2024-08-17T17:07:46.160407+0000","last_became_peered":"2024-08-17T17:07:46.160407+0000","last_unstale":"2024-08-17T17:17:31.489768+0000","last_undegraded":"2024-08-17T17:17:31.489768+0000","last_fullsized":"2024-08-17T17:17:31.489768+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:54:38.366094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":54,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.487584+0000","last_change":"2024-08-17T17:17:00.756909+0000","last_active":"2024-08-17T17:17:32.487584+0000","last_peered":"2024-08-17T17:17:32.487584+0000","last_clean":"2024-08-17T17:17:32.487584+0000","last_became_active":"2024-08-17T17:17:00.756532+0000","last_became_peered":"2024-08-17T17:17:00.756532+0000","last_unstale":"2024-08-17T17:17:32.487584+0000","last_undegraded":"2024-08-17T17:17:32.487584+0000","last_fullsized":"2024-08-17T17:17:32.487584+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:11:25.294907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.496800+0000","last_change":"2024-08-17T17:17:32.496800+0000","last_active":"2024-08-17T17:17:32.496800+0000","last_peered":"2024-08-17T17:17:32.496800+0000","last_clean":"2024-08-17T17:17:32.496800+0000","last_became_active":"2024-08-17T17:17:32.496405+0000","last_became_peered":"2024-08-17T17:17:32.496405+0000","last_unstale":"2024-08-17T17:17:32.496800+0000","last_undegraded":"2024-08-17T17:17:32.496800+0000","last_fullsized":"2024-08-17T17:17:32.496800+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:03:56.618806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":68,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.489488+0000","last_change":"2024-08-17T17:16:26.964939+0000","last_active":"2024-08-17T17:17:31.489488+0000","last_peered":"2024-08-17T17:17:31.489488+0000","last_clean":"2024-08-17T17:17:31.489488+0000","last_became_active":"2024-08-17T17:16:26.964598+0000","last_became_peered":"2024-08-17T17:16:26.964598+0000","last_unstale":"2024-08-17T17:17:31.489488+0000","last_undegraded":"2024-08-17T17:17:31.489488+0000","last_fullsized":"2024-08-17T17:17:31.489488+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:01:09.949950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":24,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.501097+0000","last_change":"2024-08-17T17:17:32.501043+0000","last_active":"2024-08-17T17:17:32.501097+0000","last_peered":"2024-08-17T17:17:32.501097+0000","last_clean":"2024-08-17T17:17:32.501097+0000","last_became_active":"2024-08-17T17:17:32.499400+0000","last_became_peered":"2024-08-17T17:17:32.499400+0000","last_unstale":"2024-08-17T17:17:32.501097+0000","last_undegraded":"2024-08-17T17:17:32.501097+0000","last_fullsized":"2024-08-17T17:17:32.501097+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:47:35.040523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":65,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.496833+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:17:32.496833+0000","last_peered":"2024-08-17T17:17:32.496833+0000","last_clean":"2024-08-17T17:17:32.496833+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:17:32.496833+0000","last_undegraded":"2024-08-17T17:17:32.496833+0000","last_fullsized":"2024-08-17T17:17:32.496833+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":71,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.501087+0000","last_change":"2024-08-17T17:17:32.501012+0000","last_active":"2024-08-17T17:17:32.501087+0000","last_peered":"2024-08-17T17:17:32.501087+0000","last_clean":"2024-08-17T17:17:32.501087+0000","last_became_active":"2024-08-17T17:17:32.499399+0000","last_became_peered":"2024-08-17T17:17:32.499399+0000","last_unstale":"2024-08-17T17:17:32.501087+0000","last_undegraded":"2024-08-17T17:17:32.501087+0000","last_fullsized":"2024-08-17T17:17:32.501087+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:19:57.035025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":67,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.491928+0000","last_change":"2024-08-17T17:16:26.963696+0000","last_active":"2024-08-17T17:17:31.491928+0000","last_peered":"2024-08-17T17:17:31.491928+0000","last_clean":"2024-08-17T17:17:31.491928+0000","last_became_active":"2024-08-17T17:16:26.963280+0000","last_became_peered":"2024-08-17T17:16:26.963280+0000","last_unstale":"2024-08-17T17:17:31.491928+0000","last_undegraded":"2024-08-17T17:17:31.491928+0000","last_fullsized":"2024-08-17T17:17:31.491928+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:22:01.474840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":89,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.500549+0000","last_change":"2024-08-17T17:17:32.500494+0000","last_active":"2024-08-17T17:17:32.500549+0000","last_peered":"2024-08-17T17:17:32.500549+0000","last_clean":"2024-08-17T17:17:32.500549+0000","last_became_active":"2024-08-17T17:17:32.498949+0000","last_became_peered":"2024-08-17T17:17:32.498949+0000","last_unstale":"2024-08-17T17:17:32.500549+0000","last_undegraded":"2024-08-17T17:17:32.500549+0000","last_fullsized":"2024-08-17T17:17:32.500549+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:44.681212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":66,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.498006+0000","last_change":"2024-08-17T17:17:32.495908+0000","last_active":"2024-08-17T17:17:32.498006+0000","last_peered":"2024-08-17T17:17:32.498006+0000","last_clean":"2024-08-17T17:17:32.498006+0000","last_became_active":"2024-08-17T17:17:32.495593+0000","last_became_peered":"2024-08-17T17:17:32.495593+0000","last_unstale":"2024-08-17T17:17:32.498006+0000","last_undegraded":"2024-08-17T17:17:32.498006+0000","last_fullsized":"2024-08-17T17:17:32.498006+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:41:39.009044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":55,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.498046+0000","last_change":"2024-08-17T17:17:32.497986+0000","last_active":"2024-08-17T17:17:32.498046+0000","last_peered":"2024-08-17T17:17:32.498046+0000","last_clean":"2024-08-17T17:17:32.498046+0000","last_became_active":"2024-08-17T17:17:32.497639+0000","last_became_peered":"2024-08-17T17:17:32.497639+0000","last_unstale":"2024-08-17T17:17:32.498046+0000","last_undegraded":"2024-08-17T17:17:32.498046+0000","last_fullsized":"2024-08-17T17:17:32.498046+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:03:25.854983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":85,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.495810+0000","last_change":"2024-08-17T17:17:32.495727+0000","last_active":"2024-08-17T17:17:32.495810+0000","last_peered":"2024-08-17T17:17:32.495810+0000","last_clean":"2024-08-17T17:17:32.495810+0000","last_became_active":"2024-08-17T17:17:32.494437+0000","last_became_peered":"2024-08-17T17:17:32.494437+0000","last_unstale":"2024-08-17T17:17:32.495810+0000","last_undegraded":"2024-08-17T17:17:32.495810+0000","last_fullsized":"2024-08-17T17:17:32.495810+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:56:22.902797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":86,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.489170+0000","last_change":"2024-08-17T17:17:00.758212+0000","last_active":"2024-08-17T17:17:31.489170+0000","last_peered":"2024-08-17T17:17:31.489170+0000","last_clean":"2024-08-17T17:17:31.489170+0000","last_became_active":"2024-08-17T17:17:00.757787+0000","last_became_peered":"2024-08-17T17:17:00.757787+0000","last_unstale":"2024-08-17T17:17:31.489170+0000","last_undegraded":"2024-08-17T17:17:31.489170+0000","last_fullsized":"2024-08-17T17:17:31.489170+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:19:11.289350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":91,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.495253+0000","last_change":"2024-08-17T17:17:32.495192+0000","last_active":"2024-08-17T17:17:32.495253+0000","last_peered":"2024-08-17T17:17:32.495253+0000","last_clean":"2024-08-17T17:17:32.495253+0000","last_became_active":"2024-08-17T17:17:32.494866+0000","last_became_peered":"2024-08-17T17:17:32.494866+0000","last_unstale":"2024-08-17T17:17:32.495253+0000","last_undegraded":"2024-08-17T17:17:32.495253+0000","last_fullsized":"2024-08-17T17:17:32.495253+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:21:13.274144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":58,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.483940+0000","last_change":"2024-08-17T17:15:54.357458+0000","last_active":"2024-08-17T17:17:32.483940+0000","last_peered":"2024-08-17T17:17:32.483940+0000","last_clean":"2024-08-17T17:17:32.483940+0000","last_became_active":"2024-08-17T17:15:54.356621+0000","last_became_peered":"2024-08-17T17:15:54.356621+0000","last_unstale":"2024-08-17T17:17:32.483940+0000","last_undegraded":"2024-08-17T17:17:32.483940+0000","last_fullsized":"2024-08-17T17:17:32.483940+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:16:10.254328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":84,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.493417+0000","last_change":"2024-08-17T17:17:32.493417+0000","last_active":"2024-08-17T17:17:32.493417+0000","last_peered":"2024-08-17T17:17:32.493417+0000","last_clean":"2024-08-17T17:17:32.493417+0000","last_became_active":"2024-08-17T17:17:32.493097+0000","last_became_peered":"2024-08-17T17:17:32.493097+0000","last_unstale":"2024-08-17T17:17:32.493417+0000","last_undegraded":"2024-08-17T17:17:32.493417+0000","last_fullsized":"2024-08-17T17:17:32.493417+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:22:01.622261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":70,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.488375+0000","last_change":"2024-08-17T17:16:26.964339+0000","last_active":"2024-08-17T17:17:31.488375+0000","last_peered":"2024-08-17T17:17:31.488375+0000","last_clean":"2024-08-17T17:17:31.488375+0000","last_became_active":"2024-08-17T17:16:26.963964+0000","last_became_peered":"2024-08-17T17:16:26.963964+0000","last_unstale":"2024-08-17T17:17:31.488375+0000","last_undegraded":"2024-08-17T17:17:31.488375+0000","last_fullsized":"2024-08-17T17:17:31.488375+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:24:14.417213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":84,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.492899+0000","last_change":"2024-08-17T17:17:32.492899+0000","last_active":"2024-08-17T17:17:32.492899+0000","last_peered":"2024-08-17T17:17:32.492899+0000","last_clean":"2024-08-17T17:17:32.492899+0000","last_became_active":"2024-08-17T17:17:32.492522+0000","last_became_peered":"2024-08-17T17:17:32.492522+0000","last_unstale":"2024-08-17T17:17:32.492899+0000","last_undegraded":"2024-08-17T17:17:32.492899+0000","last_fullsized":"2024-08-17T17:17:32.492899+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:27:57.515343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":53,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.490249+0000","last_change":"2024-08-17T17:07:46.156750+0000","last_active":"2024-08-17T17:17:31.490249+0000","last_peered":"2024-08-17T17:17:31.490249+0000","last_clean":"2024-08-17T17:17:31.490249+0000","last_became_active":"2024-08-17T17:07:46.156253+0000","last_became_peered":"2024-08-17T17:07:46.156253+0000","last_unstale":"2024-08-17T17:17:31.490249+0000","last_undegraded":"2024-08-17T17:17:31.490249+0000","last_fullsized":"2024-08-17T17:17:31.490249+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:04:28.549130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":92,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.499059+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:17:32.499059+0000","last_peered":"2024-08-17T17:17:32.499059+0000","last_clean":"2024-08-17T17:17:32.499059+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:17:32.499059+0000","last_undegraded":"2024-08-17T17:17:32.499059+0000","last_fullsized":"2024-08-17T17:17:32.499059+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":77,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.500444+0000","last_change":"2024-08-17T17:17:32.500387+0000","last_active":"2024-08-17T17:17:32.500444+0000","last_peered":"2024-08-17T17:17:32.500444+0000","last_clean":"2024-08-17T17:17:32.500444+0000","last_became_active":"2024-08-17T17:17:32.500062+0000","last_became_peered":"2024-08-17T17:17:32.500062+0000","last_unstale":"2024-08-17T17:17:32.500444+0000","last_undegraded":"2024-08-17T17:17:32.500444+0000","last_fullsized":"2024-08-17T17:17:32.500444+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:12:25.410392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":77,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-08-17T17:17:31.488249+0000","last_change":"2024-08-17T17:17:00.758339+0000","last_active":"2024-08-17T17:17:31.488249+0000","last_peered":"2024-08-17T17:17:31.488249+0000","last_clean":"2024-08-17T17:17:31.488249+0000","last_became_active":"2024-08-17T17:17:00.757987+0000","last_became_peered":"2024-08-17T17:17:00.757987+0000","last_unstale":"2024-08-17T17:17:31.488249+0000","last_undegraded":"2024-08-17T17:17:31.488249+0000","last_fullsized":"2024-08-17T17:17:31.488249+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:55:40.761984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":69,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-08-17T17:17:32.484386+0000","last_change":"2024-08-17T17:17:00.760208+0000","last_active":"2024-08-17T17:17:32.484386+0000","last_peered":"2024-08-17T17:17:32.484386+0000","last_clean":"2024-08-17T17:17:32.484386+0000","last_became_active":"2024-08-17T17:17:00.759800+0000","last_became_peered":"2024-08-17T17:17:00.759800+0000","last_unstale":"2024-08-17T17:17:32.484386+0000","last_undegraded":"2024-08-17T17:17:32.484386+0000","last_fullsized":"2024-08-17T17:17:32.484386+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:01:36.990816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17971,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":392,"num_read_kb":834,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234142,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307404,"kb_used_data":5880,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93437748,"statfs":{"total":95995035648,"available":95680253952,"internally_reserved":0,"allocated":6021120,"data_stored":5415758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495777,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307388,"kb_used_data":5868,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93437764,"statfs":{"total":95995035648,"available":95680270336,"internally_reserved":0,"allocated":6008832,"data_stored":5415424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757414,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298552,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446600,"statfs":{"total":95995035648,"available":95689318400,"internally_reserved":0,"allocated":1216512,"data_stored":626785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019052,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302720,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93442432,"statfs":{"total":95995035648,"available":95685050368,"internally_reserved":0,"allocated":1232896,"data_stored":631339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435279,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28660,"kb_used_data":1184,"kb_used_omap":4,"kb_used_meta":27451,"kb_avail":93716492,"statfs":{"total":95995035648,"available":95965687808,"internally_reserved":0,"allocated":1212416,"data_stored":626559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":28109834},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676598806,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36980,"kb_used_data":5284,"kb_used_omap":7,"kb_used_meta":31672,"kb_avail":93708172,"statfs":{"total":95995035648,"available":95957168128,"internally_reserved":0,"allocated":5410816,"data_stored":4821385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":32432290},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762333,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32884,"kb_used_data":1184,"kb_used_omap":7,"kb_used_meta":31672,"kb_avail":93712268,"statfs":{"total":95995035648,"available":95961362432,"internally_reserved":0,"allocated":1212416,"data_stored":626504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7945,"internal_metadata":32432375},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726925859,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":29304,"kb_used_data":1768,"kb_used_omap":7,"kb_used_meta":27512,"kb_avail":93715848,"statfs":{"total":95995035648,"available":95965028352,"internally_reserved":0,"allocated":1810432,"data_stored":1217445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":28172609},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6241,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:18:41.753 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-08-17T17:18:41.754 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd dump 2024-08-17T17:18:42.021 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:18:42.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:42 smithi053 ceph-mon[89963]: from='client.45074 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:18:42.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:42 smithi003 ceph-mon[100615]: from='client.45074 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:18:42.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:42 smithi003 ceph-mon[103344]: from='client.45074 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:18:43.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:43 smithi053 ceph-mon[89963]: pgmap v205: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:43.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:43 smithi003 ceph-mon[103344]: pgmap v205: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:43.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:43 smithi003 ceph-mon[100615]: pgmap v205: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:43.811 INFO:teuthology.orchestra.run.smithi003.stdout:epoch 110 2024-08-17T17:18:43.811 INFO:teuthology.orchestra.run.smithi003.stdout:fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 2024-08-17T17:18:43.811 INFO:teuthology.orchestra.run.smithi003.stdout:created 2024-08-17T17:00:56.372943+0000 2024-08-17T17:18:43.811 INFO:teuthology.orchestra.run.smithi003.stdout:modified 2024-08-17T17:17:32.478620+0000 2024-08-17T17:18:43.811 INFO:teuthology.orchestra.run.smithi003.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-08-17T17:18:43.811 INFO:teuthology.orchestra.run.smithi003.stdout:crush_version 22 2024-08-17T17:18:43.811 INFO:teuthology.orchestra.run.smithi003.stdout:full_ratio 0.95 2024-08-17T17:18:43.811 INFO:teuthology.orchestra.run.smithi003.stdout:backfillfull_ratio 0.9 2024-08-17T17:18:43.811 INFO:teuthology.orchestra.run.smithi003.stdout:nearfull_ratio 0.85 2024-08-17T17:18:43.812 INFO:teuthology.orchestra.run.smithi003.stdout:require_min_compat_client luminous 2024-08-17T17:18:43.812 INFO:teuthology.orchestra.run.smithi003.stdout:min_compat_client luminous 2024-08-17T17:18:43.812 INFO:teuthology.orchestra.run.smithi003.stdout:require_osd_release quincy 2024-08-17T17:18:43.812 INFO:teuthology.orchestra.run.smithi003.stdout:stretch_mode_enabled false 2024-08-17T17:18:43.812 INFO:teuthology.orchestra.run.smithi003.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-08-17T17:18:43.812 INFO:teuthology.orchestra.run.smithi003.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 85 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.00 2024-08-17T17:18:43.812 INFO:teuthology.orchestra.run.smithi003.stdout:pool 3 'cephfs.foo.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 70 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.50 2024-08-17T17:18:43.812 INFO:teuthology.orchestra.run.smithi003.stdout:pool 4 'cephfs.foo.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 71 flags hashpspool,bulk stripe_width 0 application cephfs read_balance_score 2.25 2024-08-17T17:18:43.812 INFO:teuthology.orchestra.run.smithi003.stdout:max_osd 8 2024-08-17T17:18:43.812 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0 up in weight 1 up_from 94 up_thru 109 down_at 91 last_clean_interval [8,90) [v2:172.21.15.3:6802/58175680,v1:172.21.15.3:6803/58175680] [v2:172.21.15.3:6804/58175680,v1:172.21.15.3:6805/58175680] exists,up 7f9a7d76-f0ec-49fc-9540-2606e94758ab 2024-08-17T17:18:43.812 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1 up in weight 1 up_from 99 up_thru 109 down_at 96 last_clean_interval [13,95) [v2:172.21.15.3:6810/1956079582,v1:172.21.15.3:6811/1956079582] [v2:172.21.15.3:6812/1956079582,v1:172.21.15.3:6813/1956079582] exists,up 316b5c4e-3417-4b93-bbbb-c6ed6dd21b0e 2024-08-17T17:18:43.812 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2 up in weight 1 up_from 104 up_thru 109 down_at 101 last_clean_interval [18,100) [v2:172.21.15.3:6818/1825512279,v1:172.21.15.3:6819/1825512279] [v2:172.21.15.3:6820/1825512279,v1:172.21.15.3:6821/1825512279] exists,up 88af697d-38db-4c51-bfff-d0a776315596 2024-08-17T17:18:43.812 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3 up in weight 1 up_from 109 up_thru 109 down_at 106 last_clean_interval [25,105) [v2:172.21.15.3:6826/1366347928,v1:172.21.15.3:6827/1366347928] [v2:172.21.15.3:6828/1366347928,v1:172.21.15.3:6829/1366347928] exists,up 8e20cf42-6a13-4ad5-9f01-203a30c692d6 2024-08-17T17:18:43.812 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4 up in weight 1 up_from 30 up_thru 109 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6800/1387588062,v1:172.21.15.53:6801/1387588062] [v2:172.21.15.53:6802/1387588062,v1:172.21.15.53:6803/1387588062] exists,up 2e8eabd7-8258-4b78-bb21-1a7291e0f1d4 2024-08-17T17:18:43.813 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5 up in weight 1 up_from 38 up_thru 109 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6808/1263400356,v1:172.21.15.53:6809/1263400356] [v2:172.21.15.53:6810/1263400356,v1:172.21.15.53:6811/1263400356] exists,up 738f10f7-ff0d-4392-a818-eb23c9f65a63 2024-08-17T17:18:43.813 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6 up in weight 1 up_from 46 up_thru 106 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6816/3729099668,v1:172.21.15.53:6817/3729099668] [v2:172.21.15.53:6818/3729099668,v1:172.21.15.53:6819/3729099668] exists,up 62903bee-59eb-4eed-b64d-9c884de87409 2024-08-17T17:18:43.813 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7 up in weight 1 up_from 54 up_thru 109 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6824/832291503,v1:172.21.15.53:6825/832291503] [v2:172.21.15.53:6826/832291503,v1:172.21.15.53:6827/832291503] exists,up 4794ff08-e5d9-49c9-bea5-e158681b4004 2024-08-17T17:18:43.813 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 4.6 [1,5] 2024-08-17T17:18:43.813 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 4.18 [1,2] 2024-08-17T17:18:43.813 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 4.1d [1,2] 2024-08-17T17:18:43.813 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/2723667418 expires 2024-08-18T17:12:37.591006+0000 2024-08-17T17:18:43.813 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:6801/2210133652 expires 2024-08-18T17:12:37.591006+0000 2024-08-17T17:18:43.813 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:6800/2210133652 expires 2024-08-18T17:12:37.591006+0000 2024-08-17T17:18:43.813 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/2965813587 expires 2024-08-18T17:12:37.591006+0000 2024-08-17T17:18:43.813 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/4087675427 expires 2024-08-18T17:12:37.591006+0000 2024-08-17T17:18:43.814 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/2847539733 expires 2024-08-18T17:12:37.591006+0000 2024-08-17T17:18:43.814 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.53:0/1967774260 expires 2024-08-18T17:11:56.179835+0000 2024-08-17T17:18:43.814 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/3985128953 expires 2024-08-18T17:01:50.631242+0000 2024-08-17T17:18:43.814 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/1476294447 expires 2024-08-18T17:01:27.442443+0000 2024-08-17T17:18:43.814 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/143641995 expires 2024-08-18T17:12:37.591006+0000 2024-08-17T17:18:43.814 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/601716019 expires 2024-08-18T17:10:21.996108+0000 2024-08-17T17:18:43.814 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/3946755108 expires 2024-08-18T17:06:25.750593+0000 2024-08-17T17:18:43.814 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:6800/1736600360 expires 2024-08-18T17:06:25.750593+0000 2024-08-17T17:18:43.814 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/3953105554 expires 2024-08-18T17:06:25.750593+0000 2024-08-17T17:18:43.814 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:6801/2670911286 expires 2024-08-18T17:10:21.996108+0000 2024-08-17T17:18:43.815 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/1583211207 expires 2024-08-18T17:06:25.750593+0000 2024-08-17T17:18:43.815 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/3116588554 expires 2024-08-18T17:06:25.750593+0000 2024-08-17T17:18:43.815 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.53:0/1330698653 expires 2024-08-18T17:11:56.179835+0000 2024-08-17T17:18:43.815 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.53:6834/4265580715 expires 2024-08-18T17:11:56.179835+0000 2024-08-17T17:18:43.815 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/3817753887 expires 2024-08-18T17:01:50.631242+0000 2024-08-17T17:18:43.815 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:6801/2187177466 expires 2024-08-18T17:01:50.631242+0000 2024-08-17T17:18:43.815 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:6800/2670911286 expires 2024-08-18T17:10:21.996108+0000 2024-08-17T17:18:43.815 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/1999667208 expires 2024-08-18T17:01:50.631242+0000 2024-08-17T17:18:43.815 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.53:0/721818304 expires 2024-08-18T17:11:56.179835+0000 2024-08-17T17:18:43.815 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:6801/1736600360 expires 2024-08-18T17:06:25.750593+0000 2024-08-17T17:18:43.815 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/880444026 expires 2024-08-18T17:10:21.996108+0000 2024-08-17T17:18:43.816 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/3717779298 expires 2024-08-18T17:10:21.996108+0000 2024-08-17T17:18:43.816 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/4194950757 expires 2024-08-18T17:01:27.442443+0000 2024-08-17T17:18:43.816 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:6800/2187177466 expires 2024-08-18T17:01:50.631242+0000 2024-08-17T17:18:43.816 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:6800/160535497 expires 2024-08-18T17:01:27.442443+0000 2024-08-17T17:18:43.816 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.53:6835/4265580715 expires 2024-08-18T17:11:56.179835+0000 2024-08-17T17:18:43.816 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/2029382827 expires 2024-08-18T17:10:21.996108+0000 2024-08-17T17:18:43.816 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.53:0/2555682812 expires 2024-08-18T17:11:56.179835+0000 2024-08-17T17:18:43.816 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/1668133120 expires 2024-08-18T17:01:27.442443+0000 2024-08-17T17:18:43.816 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:6801/160535497 expires 2024-08-18T17:01:27.442443+0000 2024-08-17T17:18:43.817 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/3608486402 expires 2024-08-18T17:12:37.591006+0000 2024-08-17T17:18:44.470 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 94 up_thru 109 down_at 91 last_clean_interval [8,90) [v2:172.21.15.3:6802/58175680,v1:172.21.15.3:6803/58175680] [v2:172.21.15.3:6804/58175680,v1:172.21.15.3:6805/58175680] exists,up 7f9a7d76-f0ec-49fc-9540-2606e94758ab', 'osd.1 up in weight 1 up_from 99 up_thru 109 down_at 96 last_clean_interval [13,95) [v2:172.21.15.3:6810/1956079582,v1:172.21.15.3:6811/1956079582] [v2:172.21.15.3:6812/1956079582,v1:172.21.15.3:6813/1956079582] exists,up 316b5c4e-3417-4b93-bbbb-c6ed6dd21b0e', 'osd.2 up in weight 1 up_from 104 up_thru 109 down_at 101 last_clean_interval [18,100) [v2:172.21.15.3:6818/1825512279,v1:172.21.15.3:6819/1825512279] [v2:172.21.15.3:6820/1825512279,v1:172.21.15.3:6821/1825512279] exists,up 88af697d-38db-4c51-bfff-d0a776315596', 'osd.3 up in weight 1 up_from 109 up_thru 109 down_at 106 last_clean_interval [25,105) [v2:172.21.15.3:6826/1366347928,v1:172.21.15.3:6827/1366347928] [v2:172.21.15.3:6828/1366347928,v1:172.21.15.3:6829/1366347928] exists,up 8e20cf42-6a13-4ad5-9f01-203a30c692d6', 'osd.4 up in weight 1 up_from 30 up_thru 109 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6800/1387588062,v1:172.21.15.53:6801/1387588062] [v2:172.21.15.53:6802/1387588062,v1:172.21.15.53:6803/1387588062] exists,up 2e8eabd7-8258-4b78-bb21-1a7291e0f1d4', 'osd.5 up in weight 1 up_from 38 up_thru 109 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6808/1263400356,v1:172.21.15.53:6809/1263400356] [v2:172.21.15.53:6810/1263400356,v1:172.21.15.53:6811/1263400356] exists,up 738f10f7-ff0d-4392-a818-eb23c9f65a63', 'osd.6 up in weight 1 up_from 46 up_thru 106 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6816/3729099668,v1:172.21.15.53:6817/3729099668] [v2:172.21.15.53:6818/3729099668,v1:172.21.15.53:6819/3729099668] exists,up 62903bee-59eb-4eed-b64d-9c884de87409', 'osd.7 up in weight 1 up_from 54 up_thru 109 down_at 0 last_clean_interval [0,0) [v2:172.21.15.53:6824/832291503,v1:172.21.15.53:6825/832291503] [v2:172.21.15.53:6826/832291503,v1:172.21.15.53:6827/832291503] exists,up 4794ff08-e5d9-49c9-bea5-e158681b4004'] 2024-08-17T17:18:44.471 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon mon.a config show 2024-08-17T17:18:44.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:44 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/4182948283' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-08-17T17:18:44.741 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:18:44.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:44 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/4182948283' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-08-17T17:18:44.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:44 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/4182948283' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-08-17T17:18:45.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:45 smithi053 ceph-mon[89963]: pgmap v206: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:45.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:45 smithi003 ceph-mon[100615]: pgmap v206: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:45.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:45 smithi003 ceph-mon[103344]: pgmap v206: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:45.774 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:18:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:18:45] "GET /metrics HTTP/1.1" 200 33938 "" "Prometheus/2.51.0" 2024-08-17T17:18:46.366 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:18:46.366 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mon.a", 2024-08-17T17:18:46.367 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster": "ceph", 2024-08-17T17:18:46.367 INFO:teuthology.orchestra.run.smithi003.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-08-17T17:18:46.367 INFO:teuthology.orchestra.run.smithi003.stdout: "admin_socket_mode": "", 2024-08-17T17:18:46.367 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-08-17T17:18:46.367 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_client_required": "cephx, none", 2024-08-17T17:18:46.367 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_cluster_required": "cephx", 2024-08-17T17:18:46.367 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_debug": "true", 2024-08-17T17:18:46.367 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-08-17T17:18:46.367 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-08-17T17:18:46.367 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_service_required": "cephx", 2024-08-17T17:18:46.367 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_service_ticket_ttl": "240.000000", 2024-08-17T17:18:46.367 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_supported": "", 2024-08-17T17:18:46.368 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_aio": "true", 2024-08-17T17:18:46.368 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_aio_max_queue_depth": "1024", 2024-08-17T17:18:46.368 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_aio_poll_ms": "250", 2024-08-17T17:18:46.368 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_aio_reap_max": "16", 2024-08-17T17:18:46.368 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_async_discard_threads": "0", 2024-08-17T17:18:46.368 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_block_size": "4096", 2024-08-17T17:18:46.368 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_debug_aio": "false", 2024-08-17T17:18:46.368 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-08-17T17:18:46.368 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-08-17T17:18:46.368 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_debug_inflight_ios": "false", 2024-08-17T17:18:46.368 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_enable_discard": "false", 2024-08-17T17:18:46.368 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_flock_retry": "3", 2024-08-17T17:18:46.368 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_flock_retry_interval": "0.100000", 2024-08-17T17:18:46.369 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_inject_crash": "0", 2024-08-17T17:18:46.369 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_inject_crash_flush_delay": "2", 2024-08-17T17:18:46.369 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_ioring": "false", 2024-08-17T17:18:46.369 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_ioring_hipri": "false", 2024-08-17T17:18:46.369 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_ioring_sqthread_poll": "false", 2024-08-17T17:18:46.369 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-08-17T17:18:46.369 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_read_buffer_alignment": "4096", 2024-08-17T17:18:46.369 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-08-17T17:18:46.369 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-08-17T17:18:46.369 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-08-17T17:18:46.369 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_type": "", 2024-08-17T17:18:46.369 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_alloc_size": "1048576", 2024-08-17T17:18:46.369 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_allocator": "hybrid", 2024-08-17T17:18:46.370 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_buffered_io": "true", 2024-08-17T17:18:46.370 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_check_for_zeros": "false", 2024-08-17T17:18:46.370 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_check_volume_selector_often": "false", 2024-08-17T17:18:46.370 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-08-17T17:18:46.370 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_compact_log_sync": "false", 2024-08-17T17:18:46.370 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-08-17T17:18:46.370 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-08-17T17:18:46.370 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_log_compact_min_size": "16777216", 2024-08-17T17:18:46.370 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_log_replay_check_allocations": "true", 2024-08-17T17:18:46.370 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_max_log_runway": "4194304", 2024-08-17T17:18:46.370 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_max_prefetch": "1048576", 2024-08-17T17:18:46.371 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_min_flush_size": "524288", 2024-08-17T17:18:46.371 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_min_log_runway": "1048576", 2024-08-17T17:18:46.371 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_replay_recovery": "false", 2024-08-17T17:18:46.371 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-08-17T17:18:46.371 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_shared_alloc_size": "65536", 2024-08-17T17:18:46.371 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_sync_write": "false", 2024-08-17T17:18:46.371 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-08-17T17:18:46.371 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-08-17T17:18:46.371 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-08-17T17:18:46.371 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_allocation_from_file": "true", 2024-08-17T17:18:46.371 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_allocator": "hybrid", 2024-08-17T17:18:46.371 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_async_db_compaction": "true", 2024-08-17T17:18:46.372 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-08-17T17:18:46.372 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-08-17T17:18:46.372 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-08-17T17:18:46.372 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-08-17T17:18:46.372 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bdev_label_multi": "true", 2024-08-17T17:18:46.372 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-08-17T17:18:46.372 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bdev_label_require_all": "true", 2024-08-17T17:18:46.372 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-08-17T17:18:46.372 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-08-17T17:18:46.372 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_blobid_prealloc": "10240", 2024-08-17T17:18:46.372 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_create": "true", 2024-08-17T17:18:46.372 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_db_create": "false", 2024-08-17T17:18:46.373 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_db_path": "", 2024-08-17T17:18:46.373 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_db_size": "0", 2024-08-17T17:18:46.373 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_path": "", 2024-08-17T17:18:46.373 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_preallocate_file": "false", 2024-08-17T17:18:46.373 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_size": "107374182400", 2024-08-17T17:18:46.373 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_wal_create": "false", 2024-08-17T17:18:46.373 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_wal_path": "", 2024-08-17T17:18:46.373 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_wal_size": "100663296", 2024-08-17T17:18:46.373 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bluefs": "true", 2024-08-17T17:18:46.373 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-08-17T17:18:46.373 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bluefs_env_mirror": "false", 2024-08-17T17:18:46.374 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-08-17T17:18:46.374 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-08-17T17:18:46.374 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-08-17T17:18:46.374 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-08-17T17:18:46.374 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-08-17T17:18:46.374 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-08-17T17:18:46.374 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-08-17T17:18:46.374 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_autotune": "true", 2024-08-17T17:18:46.374 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-08-17T17:18:46.374 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-08-17T17:18:46.374 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-08-17T17:18:46.375 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-08-17T17:18:46.375 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_size": "0", 2024-08-17T17:18:46.375 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-08-17T17:18:46.375 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-08-17T17:18:46.375 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-08-17T17:18:46.375 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-08-17T17:18:46.375 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_type": "2q", 2024-08-17T17:18:46.375 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-08-17T17:18:46.375 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_clone_cow": "true", 2024-08-17T17:18:46.375 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_algorithm": "snappy", 2024-08-17T17:18:46.375 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_max_blob_size": "0", 2024-08-17T17:18:46.375 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-08-17T17:18:46.375 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-08-17T17:18:46.376 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_min_blob_size": "0", 2024-08-17T17:18:46.376 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-08-17T17:18:46.376 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-08-17T17:18:46.376 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_mode": "none", 2024-08-17T17:18:46.376 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-08-17T17:18:46.376 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_csum_type": "crc32c", 2024-08-17T17:18:46.376 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-08-17T17:18:46.376 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_enforce_settings": "default", 2024-08-17T17:18:46.376 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_freelist": "false", 2024-08-17T17:18:46.376 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_fsck_abort": "false", 2024-08-17T17:18:46.376 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-08-17T17:18:46.376 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-08-17T17:18:46.376 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_inject_read_err": "false", 2024-08-17T17:18:46.377 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_legacy_omap": "false", 2024-08-17T17:18:46.377 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-08-17T17:18:46.377 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-08-17T17:18:46.377 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-08-17T17:18:46.377 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-08-17T17:18:46.377 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-08-17T17:18:46.377 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-08-17T17:18:46.377 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-08-17T17:18:46.377 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_small_allocations": "0", 2024-08-17T17:18:46.377 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-08-17T17:18:46.377 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_default_buffered_read": "true", 2024-08-17T17:18:46.377 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_default_buffered_write": "false", 2024-08-17T17:18:46.378 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_deferred_batch_ops": "0", 2024-08-17T17:18:46.378 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-08-17T17:18:46.378 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-08-17T17:18:46.378 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_elastic_shared_blobs": "true", 2024-08-17T17:18:46.378 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-08-17T17:18:46.378 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-08-17T17:18:46.378 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-08-17T17:18:46.378 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-08-17T17:18:46.378 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-08-17T17:18:46.379 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fail_eio": "false", 2024-08-17T17:18:46.379 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-08-17T17:18:46.379 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-08-17T17:18:46.379 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-08-17T17:18:46.379 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-08-17T17:18:46.379 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_on_mkfs": "true", 2024-08-17T17:18:46.379 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-08-17T17:18:46.379 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_on_mount": "false", 2024-08-17T17:18:46.379 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-08-17T17:18:46.379 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_on_umount": "false", 2024-08-17T17:18:46.379 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-08-17T17:18:46.379 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-08-17T17:18:46.379 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-08-17T17:18:46.380 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-08-17T17:18:46.380 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-08-17T17:18:46.380 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-08-17T17:18:46.380 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-08-17T17:18:46.380 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-08-17T17:18:46.380 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_ignore_data_csum": "false", 2024-08-17T17:18:46.380 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-08-17T17:18:46.380 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_kvbackend": "rocksdb", 2024-08-17T17:18:46.380 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-08-17T17:18:46.380 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-08-17T17:18:46.380 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_log_op_age": "5.000000", 2024-08-17T17:18:46.381 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_max_alloc_size": "0", 2024-08-17T17:18:46.381 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_max_blob_size": "0", 2024-08-17T17:18:46.381 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-08-17T17:18:46.381 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-08-17T17:18:46.381 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_max_defer_interval": "3.000000", 2024-08-17T17:18:46.381 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_max_deferred_txc": "32", 2024-08-17T17:18:46.381 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_min_alloc_size": "0", 2024-08-17T17:18:46.381 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-08-17T17:18:46.381 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-08-17T17:18:46.381 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_nid_prealloc": "1024", 2024-08-17T17:18:46.382 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_prefer_deferred_size": "0", 2024-08-17T17:18:46.382 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-08-17T17:18:46.382 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-08-17T17:18:46.382 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_qfsck_on_mount": "true", 2024-08-17T17:18:46.382 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_retry_disk_reads": "3", 2024-08-17T17:18:46.382 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_rocksdb_cf": "true", 2024-08-17T17:18:46.382 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-08-17T17:18:46.382 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-08-17T17:18:46.383 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_rocksdb_options_annex": "", 2024-08-17T17:18:46.383 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-08-17T17:18:46.383 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-08-17T17:18:46.383 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_spdk_coremask": "0x1", 2024-08-17T17:18:46.383 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_spdk_io_sleep": "5", 2024-08-17T17:18:46.383 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_spdk_max_io_completion": "0", 2024-08-17T17:18:46.383 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_spdk_mem": "512", 2024-08-17T17:18:46.383 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_sync_submit_transaction": "false", 2024-08-17T17:18:46.383 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_throttle_bytes": "67108864", 2024-08-17T17:18:46.383 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_throttle_cost_per_io": "0", 2024-08-17T17:18:46.383 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-08-17T17:18:46.383 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-08-17T17:18:46.384 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-08-17T17:18:46.384 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-08-17T17:18:46.384 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_tracing": "false", 2024-08-17T17:18:46.384 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-08-17T17:18:46.384 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-08-17T17:18:46.384 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_volume_selection_reserved": "0", 2024-08-17T17:18:46.384 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-08-17T17:18:46.384 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-08-17T17:18:46.384 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-08-17T17:18:46.384 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-08-17T17:18:46.384 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-08-17T17:18:46.384 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-08-17T17:18:46.384 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_write_v2": "false", 2024-08-17T17:18:46.385 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_write_v2_random": "false", 2024-08-17T17:18:46.385 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_zero_block_detection": "false", 2024-08-17T17:18:46.385 INFO:teuthology.orchestra.run.smithi003.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-08-17T17:18:46.385 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_action_update_interval": "2", 2024-08-17T17:18:46.385 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-08-17T17:18:46.385 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-08-17T17:18:46.385 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-08-17T17:18:46.385 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-08-17T17:18:46.385 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_mount_timeout": "10", 2024-08-17T17:18:46.385 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-08-17T17:18:46.385 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-08-17T17:18:46.386 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-08-17T17:18:46.386 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-08-17T17:18:46.386 INFO:teuthology.orchestra.run.smithi003.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-08-17T17:18:46.386 INFO:teuthology.orchestra.run.smithi003.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-08-17T17:18:46.386 INFO:teuthology.orchestra.run.smithi003.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-08-17T17:18:46.386 INFO:teuthology.orchestra.run.smithi003.stdout: "cephx_cluster_require_signatures": "false", 2024-08-17T17:18:46.386 INFO:teuthology.orchestra.run.smithi003.stdout: "cephx_cluster_require_version": "2", 2024-08-17T17:18:46.386 INFO:teuthology.orchestra.run.smithi003.stdout: "cephx_require_signatures": "false", 2024-08-17T17:18:46.386 INFO:teuthology.orchestra.run.smithi003.stdout: "cephx_require_version": "2", 2024-08-17T17:18:46.386 INFO:teuthology.orchestra.run.smithi003.stdout: "cephx_service_require_signatures": "false", 2024-08-17T17:18:46.387 INFO:teuthology.orchestra.run.smithi003.stdout: "cephx_service_require_version": "2", 2024-08-17T17:18:46.387 INFO:teuthology.orchestra.run.smithi003.stdout: "cephx_sign_messages": "true", 2024-08-17T17:18:46.387 INFO:teuthology.orchestra.run.smithi003.stdout: "chdir": "", 2024-08-17T17:18:46.387 INFO:teuthology.orchestra.run.smithi003.stdout: "client_acl_type": "", 2024-08-17T17:18:46.387 INFO:teuthology.orchestra.run.smithi003.stdout: "client_asio_thread_count": "2", 2024-08-17T17:18:46.387 INFO:teuthology.orchestra.run.smithi003.stdout: "client_cache_mid": "0.750000", 2024-08-17T17:18:46.387 INFO:teuthology.orchestra.run.smithi003.stdout: "client_cache_size": "16384", 2024-08-17T17:18:46.387 INFO:teuthology.orchestra.run.smithi003.stdout: "client_caps_release_delay": "5", 2024-08-17T17:18:46.387 INFO:teuthology.orchestra.run.smithi003.stdout: "client_check_pool_perm": "true", 2024-08-17T17:18:46.387 INFO:teuthology.orchestra.run.smithi003.stdout: "client_collect_and_send_global_metrics": "false", 2024-08-17T17:18:46.387 INFO:teuthology.orchestra.run.smithi003.stdout: "client_debug_force_sync_read": "false", 2024-08-17T17:18:46.387 INFO:teuthology.orchestra.run.smithi003.stdout: "client_debug_getattr_caps": "false", 2024-08-17T17:18:46.387 INFO:teuthology.orchestra.run.smithi003.stdout: "client_debug_inject_features": "", 2024-08-17T17:18:46.387 INFO:teuthology.orchestra.run.smithi003.stdout: "client_debug_inject_tick_delay": "0", 2024-08-17T17:18:46.388 INFO:teuthology.orchestra.run.smithi003.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-08-17T17:18:46.388 INFO:teuthology.orchestra.run.smithi003.stdout: "client_die_on_failed_remount": "false", 2024-08-17T17:18:46.388 INFO:teuthology.orchestra.run.smithi003.stdout: "client_dirsize_rbytes": "true", 2024-08-17T17:18:46.388 INFO:teuthology.orchestra.run.smithi003.stdout: "client_force_lazyio": "false", 2024-08-17T17:18:46.388 INFO:teuthology.orchestra.run.smithi003.stdout: "client_fs": "", 2024-08-17T17:18:46.388 INFO:teuthology.orchestra.run.smithi003.stdout: "client_inject_fixed_oldest_tid": "false", 2024-08-17T17:18:46.388 INFO:teuthology.orchestra.run.smithi003.stdout: "client_inject_release_failure": "false", 2024-08-17T17:18:46.388 INFO:teuthology.orchestra.run.smithi003.stdout: "client_max_inline_size": "4096", 2024-08-17T17:18:46.388 INFO:teuthology.orchestra.run.smithi003.stdout: "client_max_retries_on_remount_failure": "5", 2024-08-17T17:18:46.388 INFO:teuthology.orchestra.run.smithi003.stdout: "client_mds_namespace": "", 2024-08-17T17:18:46.388 INFO:teuthology.orchestra.run.smithi003.stdout: "client_metadata": "", 2024-08-17T17:18:46.388 INFO:teuthology.orchestra.run.smithi003.stdout: "client_mount_gid": "-1", 2024-08-17T17:18:46.388 INFO:teuthology.orchestra.run.smithi003.stdout: "client_mount_timeout": "300", 2024-08-17T17:18:46.388 INFO:teuthology.orchestra.run.smithi003.stdout: "client_mount_uid": "-1", 2024-08-17T17:18:46.389 INFO:teuthology.orchestra.run.smithi003.stdout: "client_mountpoint": "/", 2024-08-17T17:18:46.389 INFO:teuthology.orchestra.run.smithi003.stdout: "client_notify_timeout": "10", 2024-08-17T17:18:46.389 INFO:teuthology.orchestra.run.smithi003.stdout: "client_oc": "true", 2024-08-17T17:18:46.389 INFO:teuthology.orchestra.run.smithi003.stdout: "client_oc_max_dirty": "104857600", 2024-08-17T17:18:46.389 INFO:teuthology.orchestra.run.smithi003.stdout: "client_oc_max_dirty_age": "5.000000", 2024-08-17T17:18:46.389 INFO:teuthology.orchestra.run.smithi003.stdout: "client_oc_max_objects": "1000", 2024-08-17T17:18:46.389 INFO:teuthology.orchestra.run.smithi003.stdout: "client_oc_size": "209715200", 2024-08-17T17:18:46.389 INFO:teuthology.orchestra.run.smithi003.stdout: "client_oc_target_dirty": "8388608", 2024-08-17T17:18:46.389 INFO:teuthology.orchestra.run.smithi003.stdout: "client_permissions": "true", 2024-08-17T17:18:46.389 INFO:teuthology.orchestra.run.smithi003.stdout: "client_quota": "true", 2024-08-17T17:18:46.389 INFO:teuthology.orchestra.run.smithi003.stdout: "client_quota_df": "true", 2024-08-17T17:18:46.389 INFO:teuthology.orchestra.run.smithi003.stdout: "client_readahead_max_bytes": "0", 2024-08-17T17:18:46.389 INFO:teuthology.orchestra.run.smithi003.stdout: "client_readahead_max_periods": "4", 2024-08-17T17:18:46.390 INFO:teuthology.orchestra.run.smithi003.stdout: "client_readahead_min": "131072", 2024-08-17T17:18:46.390 INFO:teuthology.orchestra.run.smithi003.stdout: "client_reconnect_stale": "false", 2024-08-17T17:18:46.390 INFO:teuthology.orchestra.run.smithi003.stdout: "client_shutdown_timeout": "30", 2024-08-17T17:18:46.390 INFO:teuthology.orchestra.run.smithi003.stdout: "client_snapdir": ".snap", 2024-08-17T17:18:46.390 INFO:teuthology.orchestra.run.smithi003.stdout: "client_tick_interval": "1", 2024-08-17T17:18:46.390 INFO:teuthology.orchestra.run.smithi003.stdout: "client_trace": "", 2024-08-17T17:18:46.390 INFO:teuthology.orchestra.run.smithi003.stdout: "client_try_dentry_invalidate": "false", 2024-08-17T17:18:46.390 INFO:teuthology.orchestra.run.smithi003.stdout: "client_use_faked_inos": "false", 2024-08-17T17:18:46.390 INFO:teuthology.orchestra.run.smithi003.stdout: "client_use_random_mds": "false", 2024-08-17T17:18:46.390 INFO:teuthology.orchestra.run.smithi003.stdout: "clog_to_graylog": "false", 2024-08-17T17:18:46.390 INFO:teuthology.orchestra.run.smithi003.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-08-17T17:18:46.390 INFO:teuthology.orchestra.run.smithi003.stdout: "clog_to_graylog_port": "12201", 2024-08-17T17:18:46.390 INFO:teuthology.orchestra.run.smithi003.stdout: "clog_to_monitors": "default=true", 2024-08-17T17:18:46.391 INFO:teuthology.orchestra.run.smithi003.stdout: "clog_to_syslog": "false", 2024-08-17T17:18:46.391 INFO:teuthology.orchestra.run.smithi003.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-08-17T17:18:46.391 INFO:teuthology.orchestra.run.smithi003.stdout: "clog_to_syslog_level": "info", 2024-08-17T17:18:46.391 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "-", 2024-08-17T17:18:46.391 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_network": "", 2024-08-17T17:18:46.391 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_network_interface": "", 2024-08-17T17:18:46.391 INFO:teuthology.orchestra.run.smithi003.stdout: "compressor_zlib_isal": "false", 2024-08-17T17:18:46.391 INFO:teuthology.orchestra.run.smithi003.stdout: "compressor_zlib_level": "5", 2024-08-17T17:18:46.391 INFO:teuthology.orchestra.run.smithi003.stdout: "compressor_zlib_winsize": "-15", 2024-08-17T17:18:46.391 INFO:teuthology.orchestra.run.smithi003.stdout: "compressor_zstd_level": "1", 2024-08-17T17:18:46.391 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image": "quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d", 2024-08-17T17:18:46.391 INFO:teuthology.orchestra.run.smithi003.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-08-17T17:18:46.391 INFO:teuthology.orchestra.run.smithi003.stdout: "crimson_alien_op_num_threads": "6", 2024-08-17T17:18:46.392 INFO:teuthology.orchestra.run.smithi003.stdout: "crimson_alien_thread_cpu_cores": "", 2024-08-17T17:18:46.392 INFO:teuthology.orchestra.run.smithi003.stdout: "crimson_osd_obc_lru_size": "512", 2024-08-17T17:18:46.392 INFO:teuthology.orchestra.run.smithi003.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-08-17T17:18:46.392 INFO:teuthology.orchestra.run.smithi003.stdout: "crimson_osd_stat_interval": "0", 2024-08-17T17:18:46.392 INFO:teuthology.orchestra.run.smithi003.stdout: "crimson_seastar_cpu_cores": "", 2024-08-17T17:18:46.392 INFO:teuthology.orchestra.run.smithi003.stdout: "crimson_seastar_num_threads": "0", 2024-08-17T17:18:46.392 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "", 2024-08-17T17:18:46.392 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location_hook": "", 2024-08-17T17:18:46.392 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location_hook_timeout": "10", 2024-08-17T17:18:46.392 INFO:teuthology.orchestra.run.smithi003.stdout: "daemonize": "false", 2024-08-17T17:18:46.392 INFO:teuthology.orchestra.run.smithi003.stdout: "daos_pool": "tank", 2024-08-17T17:18:46.392 INFO:teuthology.orchestra.run.smithi003.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-08-17T17:18:46.392 INFO:teuthology.orchestra.run.smithi003.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-08-17T17:18:46.393 INFO:teuthology.orchestra.run.smithi003.stdout: "dbstore_db_name_prefix": "dbstore", 2024-08-17T17:18:46.393 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_alienstore": "0/5", 2024-08-17T17:18:46.393 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_allow_any_pool_priority": "false", 2024-08-17T17:18:46.393 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_asok": "1/5", 2024-08-17T17:18:46.393 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_asok_assert_abort": "false", 2024-08-17T17:18:46.393 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_asserts_on_shutdown": "true", 2024-08-17T17:18:46.393 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_auth": "1/5", 2024-08-17T17:18:46.393 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_bdev": "1/3", 2024-08-17T17:18:46.393 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_bluefs": "1/5", 2024-08-17T17:18:46.393 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_bluestore": "1/5", 2024-08-17T17:18:46.393 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_buffer": "0/1", 2024-08-17T17:18:46.393 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_ceph_dedup": "0/5", 2024-08-17T17:18:46.393 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_ceph_exporter": "1/5", 2024-08-17T17:18:46.394 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_cephfs_mirror": "0/5", 2024-08-17T17:18:46.394 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_cephsqlite": "0/5", 2024-08-17T17:18:46.394 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_client": "0/5", 2024-08-17T17:18:46.394 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_compressor": "1/5", 2024-08-17T17:18:46.394 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_context": "0/1", 2024-08-17T17:18:46.394 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_crimson_interrupt": "0/5", 2024-08-17T17:18:46.394 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_crush": "1/1", 2024-08-17T17:18:46.394 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_crypto": "1/5", 2024-08-17T17:18:46.394 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_cyanstore": "0/5", 2024-08-17T17:18:46.394 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_deliberately_leak_memory": "false", 2024-08-17T17:18:46.394 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_disable_randomized_ping": "false", 2024-08-17T17:18:46.394 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_dpdk": "1/5", 2024-08-17T17:18:46.394 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_eventtrace": "1/5", 2024-08-17T17:18:46.395 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_filer": "0/1", 2024-08-17T17:18:46.395 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_filestore": "1/3", 2024-08-17T17:18:46.395 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_finisher": "1/1", 2024-08-17T17:18:46.395 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_fuse": "1/5", 2024-08-17T17:18:46.395 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_heartbeat_testing_span": "0", 2024-08-17T17:18:46.395 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_heartbeatmap": "1/5", 2024-08-17T17:18:46.395 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_immutable_obj_cache": "0/5", 2024-08-17T17:18:46.395 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_javaclient": "1/5", 2024-08-17T17:18:46.395 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_journal": "1/3", 2024-08-17T17:18:46.395 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_journaler": "0/5", 2024-08-17T17:18:46.395 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_kstore": "1/5", 2024-08-17T17:18:46.396 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_lockdep": "0/1", 2024-08-17T17:18:46.396 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mclock": "1/5", 2024-08-17T17:18:46.396 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mds": "1/5", 2024-08-17T17:18:46.396 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mds_balancer": "1/5", 2024-08-17T17:18:46.396 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mds_locker": "1/5", 2024-08-17T17:18:46.396 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mds_log": "1/5", 2024-08-17T17:18:46.396 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mds_log_expire": "1/5", 2024-08-17T17:18:46.396 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mds_migrator": "1/5", 2024-08-17T17:18:46.396 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mds_quiesce": "3/5", 2024-08-17T17:18:46.396 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_memstore": "1/5", 2024-08-17T17:18:46.396 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mgr": "2/5", 2024-08-17T17:18:46.397 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mgrc": "1/5", 2024-08-17T17:18:46.397 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mon": "20/20", 2024-08-17T17:18:46.397 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_monc": "0/10", 2024-08-17T17:18:46.397 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_ms": "1/1", 2024-08-17T17:18:46.397 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_none": "0/5", 2024-08-17T17:18:46.397 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_objclass": "0/5", 2024-08-17T17:18:46.397 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_objectcacher": "0/5", 2024-08-17T17:18:46.397 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_objecter": "0/1", 2024-08-17T17:18:46.397 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_optracker": "0/5", 2024-08-17T17:18:46.397 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_osd": "1/5", 2024-08-17T17:18:46.397 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_paxos": "20/20", 2024-08-17T17:18:46.397 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_perfcounter": "1/5", 2024-08-17T17:18:46.397 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_prioritycache": "1/5", 2024-08-17T17:18:46.397 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rados": "0/5", 2024-08-17T17:18:46.398 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rbd": "0/5", 2024-08-17T17:18:46.398 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rbd_mirror": "0/5", 2024-08-17T17:18:46.398 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rbd_pwl": "0/5", 2024-08-17T17:18:46.398 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rbd_replay": "0/5", 2024-08-17T17:18:46.398 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_refs": "0/0", 2024-08-17T17:18:46.398 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_reserver": "1/1", 2024-08-17T17:18:46.398 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rgw": "1/5", 2024-08-17T17:18:46.398 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rgw_access": "1/5", 2024-08-17T17:18:46.398 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rgw_datacache": "1/5", 2024-08-17T17:18:46.398 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rgw_dbstore": "1/5", 2024-08-17T17:18:46.398 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rgw_flight": "1/5", 2024-08-17T17:18:46.398 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rgw_lifecycle": "1/5", 2024-08-17T17:18:46.399 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rgw_notification": "1/5", 2024-08-17T17:18:46.399 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rgw_sync": "1/5", 2024-08-17T17:18:46.399 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rocksdb": "4/5", 2024-08-17T17:18:46.399 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore": "0/5", 2024-08-17T17:18:46.399 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_backref": "0/5", 2024-08-17T17:18:46.399 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_cache": "0/5", 2024-08-17T17:18:46.399 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_cleaner": "0/5", 2024-08-17T17:18:46.399 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_device": "0/5", 2024-08-17T17:18:46.399 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_epm": "0/5", 2024-08-17T17:18:46.399 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-08-17T17:18:46.399 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_journal": "0/5", 2024-08-17T17:18:46.399 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_lba": "0/5", 2024-08-17T17:18:46.399 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_odata": "0/5", 2024-08-17T17:18:46.400 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_omap": "0/5", 2024-08-17T17:18:46.400 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_onode": "0/5", 2024-08-17T17:18:46.400 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_t": "0/5", 2024-08-17T17:18:46.400 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_tm": "0/5", 2024-08-17T17:18:46.400 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_striper": "0/1", 2024-08-17T17:18:46.400 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_test": "0/5", 2024-08-17T17:18:46.400 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_throttle": "1/1", 2024-08-17T17:18:46.400 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_timer": "0/1", 2024-08-17T17:18:46.400 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_tp": "0/5", 2024-08-17T17:18:46.400 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_trace": "1/5", 2024-08-17T17:18:46.401 INFO:teuthology.orchestra.run.smithi003.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-08-17T17:18:46.401 INFO:teuthology.orchestra.run.smithi003.stdout: "device_failure_prediction_mode": "none", 2024-08-17T17:18:46.401 INFO:teuthology.orchestra.run.smithi003.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-08-17T17:18:46.401 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-08-17T17:18:46.401 INFO:teuthology.orchestra.run.smithi003.stdout: "err_to_graylog": "false", 2024-08-17T17:18:46.401 INFO:teuthology.orchestra.run.smithi003.stdout: "err_to_journald": "false", 2024-08-17T17:18:46.401 INFO:teuthology.orchestra.run.smithi003.stdout: "err_to_stderr": "true", 2024-08-17T17:18:46.401 INFO:teuthology.orchestra.run.smithi003.stdout: "err_to_syslog": "false", 2024-08-17T17:18:46.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event_tracing": "false", 2024-08-17T17:18:46.401 INFO:teuthology.orchestra.run.smithi003.stdout: "exporter_addr": "0.0.0.0", 2024-08-17T17:18:46.401 INFO:teuthology.orchestra.run.smithi003.stdout: "exporter_cert_file": "", 2024-08-17T17:18:46.401 INFO:teuthology.orchestra.run.smithi003.stdout: "exporter_http_port": "9926", 2024-08-17T17:18:46.402 INFO:teuthology.orchestra.run.smithi003.stdout: "exporter_key_file": "", 2024-08-17T17:18:46.402 INFO:teuthology.orchestra.run.smithi003.stdout: "exporter_prio_limit": "5", 2024-08-17T17:18:46.402 INFO:teuthology.orchestra.run.smithi003.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-08-17T17:18:46.402 INFO:teuthology.orchestra.run.smithi003.stdout: "exporter_sort_metrics": "true", 2024-08-17T17:18:46.402 INFO:teuthology.orchestra.run.smithi003.stdout: "exporter_stats_period": "5", 2024-08-17T17:18:46.402 INFO:teuthology.orchestra.run.smithi003.stdout: "fake_statfs_for_testing": "0", 2024-08-17T17:18:46.402 INFO:teuthology.orchestra.run.smithi003.stdout: "fatal_signal_handlers": "true", 2024-08-17T17:18:46.402 INFO:teuthology.orchestra.run.smithi003.stdout: "filer_max_purge_ops": "10", 2024-08-17T17:18:46.402 INFO:teuthology.orchestra.run.smithi003.stdout: "filer_max_truncate_ops": "128", 2024-08-17T17:18:46.402 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_apply_finisher_threads": "1", 2024-08-17T17:18:46.402 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_blackhole": "false", 2024-08-17T17:18:46.402 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_btrfs_clone_range": "true", 2024-08-17T17:18:46.403 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_btrfs_snap": "true", 2024-08-17T17:18:46.403 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_caller_concurrency": "10", 2024-08-17T17:18:46.403 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_collect_device_partition_information": "true", 2024-08-17T17:18:46.403 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_commit_timeout": "600.000000", 2024-08-17T17:18:46.403 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_debug_inject_read_err": "false", 2024-08-17T17:18:46.403 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_debug_omap_check": "false", 2024-08-17T17:18:46.403 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_debug_verify_split": "false", 2024-08-17T17:18:46.403 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_dump_file": "", 2024-08-17T17:18:46.403 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-08-17T17:18:46.403 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-08-17T17:18:46.403 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_fadvise": "true", 2024-08-17T17:18:46.403 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_fail_eio": "true", 2024-08-17T17:18:46.403 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_fd_cache_shards": "16", 2024-08-17T17:18:46.404 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_fd_cache_size": "128", 2024-08-17T17:18:46.404 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_fiemap": "false", 2024-08-17T17:18:46.404 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_fiemap_threshold": "4096", 2024-08-17T17:18:46.404 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-08-17T17:18:46.404 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_index_retry_probability": "0.000000", 2024-08-17T17:18:46.404 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_inject_stall": "0", 2024-08-17T17:18:46.404 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_journal_parallel": "false", 2024-08-17T17:18:46.404 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_journal_trailing": "false", 2024-08-17T17:18:46.404 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_journal_writeahead": "false", 2024-08-17T17:18:46.404 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_kill_at": "0", 2024-08-17T17:18:46.404 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-08-17T17:18:46.404 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattr_size": "0", 2024-08-17T17:18:46.404 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-08-17T17:18:46.405 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-08-17T17:18:46.405 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-08-17T17:18:46.405 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattrs": "0", 2024-08-17T17:18:46.405 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-08-17T17:18:46.405 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattrs_other": "2", 2024-08-17T17:18:46.405 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-08-17T17:18:46.405 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_sync_interval": "5.000000", 2024-08-17T17:18:46.405 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_xattr_value_size": "0", 2024-08-17T17:18:46.405 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-08-17T17:18:46.405 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-08-17T17:18:46.405 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-08-17T17:18:46.405 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_merge_threshold": "-10", 2024-08-17T17:18:46.406 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_min_sync_interval": "0.010000", 2024-08-17T17:18:46.406 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_odsync_write": "false", 2024-08-17T17:18:46.406 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_omap_backend": "rocksdb", 2024-08-17T17:18:46.406 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_omap_backend_path": "", 2024-08-17T17:18:46.406 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_omap_header_cache_size": "1024", 2024-08-17T17:18:46.406 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_ondisk_finisher_threads": "1", 2024-08-17T17:18:46.406 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-08-17T17:18:46.406 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_op_thread_timeout": "60", 2024-08-17T17:18:46.406 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_op_threads": "2", 2024-08-17T17:18:46.406 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_punch_hole": "false", 2024-08-17T17:18:46.406 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-08-17T17:18:46.406 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-08-17T17:18:46.406 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-08-17T17:18:46.407 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-08-17T17:18:46.407 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-08-17T17:18:46.407 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_max_bytes": "104857600", 2024-08-17T17:18:46.407 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-08-17T17:18:46.407 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-08-17T17:18:46.407 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-08-17T17:18:46.407 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_max_ops": "50", 2024-08-17T17:18:46.407 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-08-17T17:18:46.407 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_seek_data_hole": "false", 2024-08-17T17:18:46.407 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_sloppy_crc": "false", 2024-08-17T17:18:46.407 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-08-17T17:18:46.407 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_splice": "false", 2024-08-17T17:18:46.407 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_split_multiple": "2", 2024-08-17T17:18:46.407 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_split_rand_factor": "20", 2024-08-17T17:18:46.408 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_update_to": "1000", 2024-08-17T17:18:46.408 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-08-17T17:18:46.408 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-08-17T17:18:46.408 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-08-17T17:18:46.408 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-08-17T17:18:46.408 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-08-17T17:18:46.408 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-08-17T17:18:46.408 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_enable": "true", 2024-08-17T17:18:46.408 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-08-17T17:18:46.408 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-08-17T17:18:46.409 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-08-17T17:18:46.409 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-08-17T17:18:46.409 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-08-17T17:18:46.409 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-08-17T17:18:46.409 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_xfs_extsize": "false", 2024-08-17T17:18:46.409 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_zfs_snap": "false", 2024-08-17T17:18:46.409 INFO:teuthology.orchestra.run.smithi003.stdout: "fio_dir": "/tmp/fio", 2024-08-17T17:18:46.409 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "2b5b662e-5cba-11ef-bccf-c7b262605968", 2024-08-17T17:18:46.409 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_allow_other": "true", 2024-08-17T17:18:46.409 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_atomic_o_trunc": "true", 2024-08-17T17:18:46.409 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_big_writes": "true", 2024-08-17T17:18:46.409 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_debug": "false", 2024-08-17T17:18:46.409 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_default_permissions": "false", 2024-08-17T17:18:46.409 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_disable_pagecache": "false", 2024-08-17T17:18:46.410 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_max_write": "0", 2024-08-17T17:18:46.410 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_multithreaded": "true", 2024-08-17T17:18:46.410 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_require_active_mds": "true", 2024-08-17T17:18:46.410 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_set_user_groups": "true", 2024-08-17T17:18:46.410 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_splice_move": "true", 2024-08-17T17:18:46.410 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_splice_read": "true", 2024-08-17T17:18:46.410 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_splice_write": "true", 2024-08-17T17:18:46.410 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_syncfs_on_mksnap": "true", 2024-08-17T17:18:46.410 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_use_invalidate_cb": "true", 2024-08-17T17:18:46.410 INFO:teuthology.orchestra.run.smithi003.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-08-17T17:18:46.410 INFO:teuthology.orchestra.run.smithi003.stdout: "gss_target_name": "ceph", 2024-08-17T17:18:46.410 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_file": "", 2024-08-17T17:18:46.410 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_inject_failure": "0", 2024-08-17T17:18:46.411 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_interval": "5", 2024-08-17T17:18:46.411 INFO:teuthology.orchestra.run.smithi003.stdout: "host": "smithi003", 2024-08-17T17:18:46.411 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-08-17T17:18:46.411 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-08-17T17:18:46.411 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_max_size": "1073741824", 2024-08-17T17:18:46.411 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-08-17T17:18:46.411 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-08-17T17:18:46.411 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-08-17T17:18:46.411 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-08-17T17:18:46.411 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-08-17T17:18:46.411 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-08-17T17:18:46.411 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-08-17T17:18:46.411 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-08-17T17:18:46.411 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-08-17T17:18:46.412 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_watermark": "0.900000", 2024-08-17T17:18:46.412 INFO:teuthology.orchestra.run.smithi003.stdout: "inject_early_sigterm": "false", 2024-08-17T17:18:46.412 INFO:teuthology.orchestra.run.smithi003.stdout: "jaeger_agent_port": "6799", 2024-08-17T17:18:46.412 INFO:teuthology.orchestra.run.smithi003.stdout: "jaeger_tracing_enable": "false", 2024-08-17T17:18:46.412 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_aio": "true", 2024-08-17T17:18:46.412 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_align_min_size": "65536", 2024-08-17T17:18:46.412 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_block_align": "true", 2024-08-17T17:18:46.412 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_block_size": "4096", 2024-08-17T17:18:46.412 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_dio": "true", 2024-08-17T17:18:46.412 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_discard": "false", 2024-08-17T17:18:46.412 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_force_aio": "false", 2024-08-17T17:18:46.412 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_ignore_corruption": "false", 2024-08-17T17:18:46.412 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_max_write_bytes": "10485760", 2024-08-17T17:18:46.412 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_max_write_entries": "100", 2024-08-17T17:18:46.413 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_replay_from": "0", 2024-08-17T17:18:46.413 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_throttle_high_multiple": "0.000000", 2024-08-17T17:18:46.413 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-08-17T17:18:46.413 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-08-17T17:18:46.413 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_throttle_max_multiple": "0.000000", 2024-08-17T17:18:46.413 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_write_header_frequency": "0", 2024-08-17T17:18:46.413 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_zero_on_create": "false", 2024-08-17T17:18:46.413 INFO:teuthology.orchestra.run.smithi003.stdout: "journaler_prefetch_periods": "10", 2024-08-17T17:18:46.413 INFO:teuthology.orchestra.run.smithi003.stdout: "journaler_prezero_periods": "5", 2024-08-17T17:18:46.413 INFO:teuthology.orchestra.run.smithi003.stdout: "journaler_write_head_interval": "15", 2024-08-17T17:18:46.413 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "", 2024-08-17T17:18:46.413 INFO:teuthology.orchestra.run.smithi003.stdout: "keyfile": "", 2024-08-17T17:18:46.413 INFO:teuthology.orchestra.run.smithi003.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-08-17T17:18:46.414 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_backend": "rocksdb", 2024-08-17T17:18:46.414 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_default_stripe_size": "65536", 2024-08-17T17:18:46.414 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_fsck_on_mount": "false", 2024-08-17T17:18:46.415 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_fsck_on_mount_deep": "true", 2024-08-17T17:18:46.415 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_max_bytes": "67108864", 2024-08-17T17:18:46.415 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_max_ops": "512", 2024-08-17T17:18:46.415 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_nid_prealloc": "1024", 2024-08-17T17:18:46.416 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_onode_map_size": "1024", 2024-08-17T17:18:46.416 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-08-17T17:18:46.416 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_sync_submit_transaction": "false", 2024-08-17T17:18:46.416 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_sync_transaction": "false", 2024-08-17T17:18:46.416 INFO:teuthology.orchestra.run.smithi003.stdout: "librados_thread_count": "2", 2024-08-17T17:18:46.416 INFO:teuthology.orchestra.run.smithi003.stdout: "lockdep": "false", 2024-08-17T17:18:46.416 INFO:teuthology.orchestra.run.smithi003.stdout: "lockdep_force_backtrace": "false", 2024-08-17T17:18:46.416 INFO:teuthology.orchestra.run.smithi003.stdout: "log_coarse_timestamps": "true", 2024-08-17T17:18:46.416 INFO:teuthology.orchestra.run.smithi003.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-08-17T17:18:46.416 INFO:teuthology.orchestra.run.smithi003.stdout: "log_flush_on_exit": "false", 2024-08-17T17:18:46.416 INFO:teuthology.orchestra.run.smithi003.stdout: "log_graylog_host": "127.0.0.1", 2024-08-17T17:18:46.416 INFO:teuthology.orchestra.run.smithi003.stdout: "log_graylog_port": "12201", 2024-08-17T17:18:46.416 INFO:teuthology.orchestra.run.smithi003.stdout: "log_max_new": "1000", 2024-08-17T17:18:46.417 INFO:teuthology.orchestra.run.smithi003.stdout: "log_max_recent": "10000", 2024-08-17T17:18:46.417 INFO:teuthology.orchestra.run.smithi003.stdout: "log_stderr_prefix": "", 2024-08-17T17:18:46.417 INFO:teuthology.orchestra.run.smithi003.stdout: "log_stop_at_utilization": "0.970000", 2024-08-17T17:18:46.417 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": "true", 2024-08-17T17:18:46.417 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_graylog": "false", 2024-08-17T17:18:46.417 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_journald": "false", 2024-08-17T17:18:46.417 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_stderr": "false", 2024-08-17T17:18:46.417 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_syslog": "false", 2024-08-17T17:18:46.417 INFO:teuthology.orchestra.run.smithi003.stdout: "max_rotating_auth_attempts": "10", 2024-08-17T17:18:46.417 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-08-17T17:18:46.417 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_action_on_write_error": "1", 2024-08-17T17:18:46.417 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_alternate_name_max": "8192", 2024-08-17T17:18:46.417 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_asio_thread_count": "2", 2024-08-17T17:18:46.417 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_export_pin": "true", 2024-08-17T17:18:46.418 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_fragment_dirs": "true", 2024-08-17T17:18:46.418 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-08-17T17:18:46.418 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_fragment_interval": "5", 2024-08-17T17:18:46.418 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_fragment_size_max": "100000", 2024-08-17T17:18:46.418 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_idle_threshold": "0.000000", 2024-08-17T17:18:46.418 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_interval": "10", 2024-08-17T17:18:46.418 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_max": "-1", 2024-08-17T17:18:46.418 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_max_until": "-1", 2024-08-17T17:18:46.418 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_merge_size": "50", 2024-08-17T17:18:46.418 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_midchunk": "0.300000", 2024-08-17T17:18:46.418 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_min_rebalance": "0.100000", 2024-08-17T17:18:46.418 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_min_start": "0.200000", 2024-08-17T17:18:46.418 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_minchunk": "0.001000", 2024-08-17T17:18:46.419 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_mode": "0", 2024-08-17T17:18:46.419 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_need_max": "1.200000", 2024-08-17T17:18:46.419 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_need_min": "0.800000", 2024-08-17T17:18:46.419 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_overload_epochs": "2", 2024-08-17T17:18:46.419 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-08-17T17:18:46.419 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_sample_interval": "3.000000", 2024-08-17T17:18:46.419 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_split_bits": "3", 2024-08-17T17:18:46.419 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_split_rd": "25000.000000", 2024-08-17T17:18:46.419 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_split_size": "10000", 2024-08-17T17:18:46.419 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_split_wr": "10000.000000", 2024-08-17T17:18:46.419 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_target_decay": "10.000000", 2024-08-17T17:18:46.419 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-08-17T17:18:46.419 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_beacon_grace": "15.000000", 2024-08-17T17:18:46.419 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_beacon_interval": "4.000000", 2024-08-17T17:18:46.420 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_beacon_mon_down_grace": "60", 2024-08-17T17:18:46.420 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_memory_limit": "4294967296", 2024-08-17T17:18:46.420 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_mid": "0.700000", 2024-08-17T17:18:46.420 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-08-17T17:18:46.420 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_quiesce_delay": "0", 2024-08-17T17:18:46.420 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_quiesce_sleep": "200", 2024-08-17T17:18:46.420 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_quiesce_splitauth": "true", 2024-08-17T17:18:46.420 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_quiesce_threshold": "524288", 2024-08-17T17:18:46.420 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_release_free_interval": "10", 2024-08-17T17:18:46.420 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_reservation": "0.050000", 2024-08-17T17:18:46.420 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-08-17T17:18:46.420 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_trim_interval": "1", 2024-08-17T17:18:46.420 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_trim_threshold": "262144", 2024-08-17T17:18:46.420 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-08-17T17:18:46.420 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-08-17T17:18:46.421 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_client_delegate_inos_pct": "50", 2024-08-17T17:18:46.421 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_client_prealloc_inos": "1000", 2024-08-17T17:18:46.421 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-08-17T17:18:46.421 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_connect_bootstrapping": "false", 2024-08-17T17:18:46.421 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_damage_table_max_entries": "10000", 2024-08-17T17:18:46.421 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-08-17T17:18:46.421 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_debug_auth_pins": "false", 2024-08-17T17:18:46.421 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_debug_frag": "false", 2024-08-17T17:18:46.421 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_debug_scatterstat": "false", 2024-08-17T17:18:46.421 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_debug_subtrees": "false", 2024-08-17T17:18:46.421 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_decay_halflife": "5.000000", 2024-08-17T17:18:46.421 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_default_dir_hash": "2", 2024-08-17T17:18:46.421 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_defer_session_stale": "true", 2024-08-17T17:18:46.422 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_deny_all_reconnect": "false", 2024-08-17T17:18:46.422 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dir_keys_per_op": "16384", 2024-08-17T17:18:46.422 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dir_max_commit_size": "10", 2024-08-17T17:18:46.422 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dir_max_entries": "0", 2024-08-17T17:18:46.422 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dir_prefetch": "true", 2024-08-17T17:18:46.422 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dirstat_min_interval": "1.000000", 2024-08-17T17:18:46.422 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dump_cache_after_rejoin": "false", 2024-08-17T17:18:46.422 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dump_cache_on_map": "false", 2024-08-17T17:18:46.422 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dump_cache_threshold_file": "0", 2024-08-17T17:18:46.422 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-08-17T17:18:46.422 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_early_reply": "true", 2024-08-17T17:18:46.422 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_enable_op_tracker": "true", 2024-08-17T17:18:46.422 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_enforce_unique_name": "true", 2024-08-17T17:18:46.423 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_export_ephemeral_distributed": "true", 2024-08-17T17:18:46.423 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-08-17T17:18:46.423 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_export_ephemeral_random": "true", 2024-08-17T17:18:46.423 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-08-17T17:18:46.423 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-08-17T17:18:46.423 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_forward_all_requests_to_auth": "false", 2024-08-17T17:18:46.423 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-08-17T17:18:46.423 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-08-17T17:18:46.423 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-08-17T17:18:46.423 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-08-17T17:18:46.423 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_health_cache_threshold": "1.500000", 2024-08-17T17:18:46.423 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_health_summarize_threshold": "10", 2024-08-17T17:18:46.423 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_heartbeat_grace": "15.000000", 2024-08-17T17:18:46.423 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_heartbeat_reset_grace": "1000", 2024-08-17T17:18:46.424 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_inject_health_dummy": "false", 2024-08-17T17:18:46.424 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-08-17T17:18:46.424 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_inject_migrator_session_race": "false", 2024-08-17T17:18:46.424 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-08-17T17:18:46.424 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-08-17T17:18:46.424 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-08-17T17:18:46.424 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_join_fs": "", 2024-08-17T17:18:46.424 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_journal_format": "1", 2024-08-17T17:18:46.424 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-08-17T17:18:46.424 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_create_at": "0", 2024-08-17T17:18:46.424 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_dirfrag_at": "0", 2024-08-17T17:18:46.424 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_export_at": "0", 2024-08-17T17:18:46.424 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_import_at": "0", 2024-08-17T17:18:46.424 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_journal_at": "0", 2024-08-17T17:18:46.425 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_journal_expire_at": "0", 2024-08-17T17:18:46.425 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_journal_replay_at": "0", 2024-08-17T17:18:46.425 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_link_at": "0", 2024-08-17T17:18:46.425 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_mdstable_at": "0", 2024-08-17T17:18:46.425 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_openc_at": "0", 2024-08-17T17:18:46.425 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_rename_at": "0", 2024-08-17T17:18:46.425 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_shutdown_at": "0", 2024-08-17T17:18:46.425 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_event_large_threshold": "524288", 2024-08-17T17:18:46.425 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_events_per_segment": "1024", 2024-08-17T17:18:46.425 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_major_segment_event_ratio": "12", 2024-08-17T17:18:46.425 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_max_events": "-1", 2024-08-17T17:18:46.425 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_max_segments": "128", 2024-08-17T17:18:46.426 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_pause": "false", 2024-08-17T17:18:46.426 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_segment_size": "0", 2024-08-17T17:18:46.426 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_skip_corrupt_events": "false", 2024-08-17T17:18:46.426 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_skip_unbounded_events": "false", 2024-08-17T17:18:46.426 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-08-17T17:18:46.426 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_trim_threshold": "128", 2024-08-17T17:18:46.426 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-08-17T17:18:46.426 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_warn_factor": "2.000000", 2024-08-17T17:18:46.426 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_caps_per_client": "1048576", 2024-08-17T17:18:46.427 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_completed_flushes": "100000", 2024-08-17T17:18:46.427 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_completed_requests": "100000", 2024-08-17T17:18:46.427 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_export_size": "20971520", 2024-08-17T17:18:46.427 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_file_recover": "32", 2024-08-17T17:18:46.427 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_purge_files": "64", 2024-08-17T17:18:46.427 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_purge_ops": "8192", 2024-08-17T17:18:46.427 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-08-17T17:18:46.427 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-08-17T17:18:46.427 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_snaps_per_dir": "100", 2024-08-17T17:18:46.427 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_metrics_update_interval": "2", 2024-08-17T17:18:46.427 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_min_caps_per_client": "100", 2024-08-17T17:18:46.427 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_min_caps_working_set": "10000", 2024-08-17T17:18:46.427 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-08-17T17:18:46.427 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_numa_node": "-1", 2024-08-17T17:18:46.428 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-08-17T17:18:46.428 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_op_complaint_time": "30.000000", 2024-08-17T17:18:46.428 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_op_history_duration": "600", 2024-08-17T17:18:46.428 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_op_history_size": "20", 2024-08-17T17:18:46.428 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_op_history_slow_op_size": "20", 2024-08-17T17:18:46.428 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_op_history_slow_op_threshold": "10", 2024-08-17T17:18:46.428 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_op_log_threshold": "5", 2024-08-17T17:18:46.428 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_ping_grace": "15", 2024-08-17T17:18:46.428 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_ping_interval": "5", 2024-08-17T17:18:46.428 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-08-17T17:18:46.428 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-08-17T17:18:46.428 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_recall_max_caps": "30000", 2024-08-17T17:18:46.428 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-08-17T17:18:46.428 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_recall_max_decay_threshold": "131072", 2024-08-17T17:18:46.428 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-08-17T17:18:46.429 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_recall_warning_threshold": "262144", 2024-08-17T17:18:46.429 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_reconnect_timeout": "45.000000", 2024-08-17T17:18:46.429 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_replay_interval": "1.000000", 2024-08-17T17:18:46.429 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-08-17T17:18:46.429 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-08-17T17:18:46.429 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_root_ino_gid": "0", 2024-08-17T17:18:46.429 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_root_ino_uid": "0", 2024-08-17T17:18:46.429 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-08-17T17:18:46.429 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-08-17T17:18:46.429 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-08-17T17:18:46.429 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_blocklist_on_evict": "true", 2024-08-17T17:18:46.429 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_blocklist_on_timeout": "true", 2024-08-17T17:18:46.429 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-08-17T17:18:46.430 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-08-17T17:18:46.430 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-08-17T17:18:46.430 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-08-17T17:18:46.430 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-08-17T17:18:46.430 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_metadata_threshold": "16777216", 2024-08-17T17:18:46.430 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-08-17T17:18:46.430 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_shutdown_check": "0", 2024-08-17T17:18:46.430 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_skip_ino": "0", 2024-08-17T17:18:46.430 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_sleep_rank_change": "0.000000", 2024-08-17T17:18:46.430 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_snap_max_uid": "4294967294", 2024-08-17T17:18:46.430 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_snap_min_uid": "0", 2024-08-17T17:18:46.430 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_snap_rstat": "false", 2024-08-17T17:18:46.430 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_standby_replay_damaged": "false", 2024-08-17T17:18:46.431 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_symlink_recovery": "true", 2024-08-17T17:18:46.431 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_task_status_update_interval": "2.000000", 2024-08-17T17:18:46.431 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_thrash_exports": "0", 2024-08-17T17:18:46.431 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_thrash_fragments": "0", 2024-08-17T17:18:46.431 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_tick_interval": "5.000000", 2024-08-17T17:18:46.431 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_valgrind_exit": "false", 2024-08-17T17:18:46.431 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_verify_backtrace": "1", 2024-08-17T17:18:46.431 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_verify_scatter": "false", 2024-08-17T17:18:46.431 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_wipe_ino_prealloc": "false", 2024-08-17T17:18:46.431 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_wipe_sessions": "false", 2024-08-17T17:18:46.431 INFO:teuthology.orchestra.run.smithi003.stdout: "mempool_debug": "false", 2024-08-17T17:18:46.432 INFO:teuthology.orchestra.run.smithi003.stdout: "memstore_debug_omit_block_device_write": "false", 2024-08-17T17:18:46.432 INFO:teuthology.orchestra.run.smithi003.stdout: "memstore_device_bytes": "1073741824", 2024-08-17T17:18:46.432 INFO:teuthology.orchestra.run.smithi003.stdout: "memstore_page_set": "false", 2024-08-17T17:18:46.432 INFO:teuthology.orchestra.run.smithi003.stdout: "memstore_page_size": "65536", 2024-08-17T17:18:46.432 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_client_bytes": "134217728", 2024-08-17T17:18:46.432 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_client_messages": "512", 2024-08-17T17:18:46.432 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-08-17T17:18:46.432 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_connect_retry_interval": "1.000000", 2024-08-17T17:18:46.432 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-08-17T17:18:46.432 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-08-17T17:18:46.432 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_disabled_modules": "", 2024-08-17T17:18:46.432 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_enable_op_tracker": "true", 2024-08-17T17:18:46.433 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-08-17T17:18:46.433 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_max_pg_creating": "1024", 2024-08-17T17:18:46.433 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_max_pg_num_change": "128", 2024-08-17T17:18:46.433 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_mds_bytes": "134217728", 2024-08-17T17:18:46.433 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_mds_messages": "128", 2024-08-17T17:18:46.433 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-08-17T17:18:46.433 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_mon_bytes": "134217728", 2024-08-17T17:18:46.433 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_mon_messages": "128", 2024-08-17T17:18:46.433 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_num_op_tracker_shard": "32", 2024-08-17T17:18:46.433 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_op_complaint_time": "30.000000", 2024-08-17T17:18:46.433 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_op_history_duration": "600", 2024-08-17T17:18:46.433 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_op_history_size": "20", 2024-08-17T17:18:46.433 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_op_history_slow_op_size": "20", 2024-08-17T17:18:46.434 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-08-17T17:18:46.434 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_op_log_threshold": "5", 2024-08-17T17:18:46.434 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_osd_bytes": "536870912", 2024-08-17T17:18:46.434 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_osd_messages": "8192", 2024-08-17T17:18:46.434 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_pool": "true", 2024-08-17T17:18:46.434 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_service_beacon_grace": "60.000000", 2024-08-17T17:18:46.434 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_standby_modules": "true", 2024-08-17T17:18:46.434 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_stats_period": "5", 2024-08-17T17:18:46.434 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_stats_threshold": "5", 2024-08-17T17:18:46.434 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_tick_period": "2", 2024-08-17T17:18:46.434 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-08-17T17:18:46.434 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_accept_timeout_factor": "2.000000", 2024-08-17T17:18:46.435 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_allow_pool_delete": "true", 2024-08-17T17:18:46.435 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_allow_pool_size_one": "false", 2024-08-17T17:18:46.435 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_auth_validate_all_caps": "true", 2024-08-17T17:18:46.435 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-08-17T17:18:46.435 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-08-17T17:18:46.435 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_bytes": "104857600", 2024-08-17T17:18:46.435 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_directed_command_retry": "2", 2024-08-17T17:18:46.435 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_hunt_interval": "3.000000", 2024-08-17T17:18:46.435 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-08-17T17:18:46.435 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-08-17T17:18:46.435 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-08-17T17:18:46.435 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_hunt_parallel": "3", 2024-08-17T17:18:46.435 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_log_interval": "1.000000", 2024-08-17T17:18:46.436 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-08-17T17:18:46.436 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_ping_interval": "10.000000", 2024-08-17T17:18:46.436 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_ping_timeout": "30.000000", 2024-08-17T17:18:46.436 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_target_rank": "-1", 2024-08-17T17:18:46.436 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_clock_drift_allowed": "1.000000", 2024-08-17T17:18:46.436 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-08-17T17:18:46.436 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-08-17T17:18:46.436 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_level": "debug", 2024-08-17T17:18:46.436 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_file": "true", 2024-08-17T17:18:46.436 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_graylog": "false", 2024-08-17T17:18:46.436 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-08-17T17:18:46.436 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-08-17T17:18:46.437 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_journald": "true", 2024-08-17T17:18:46.437 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_stderr": "false", 2024-08-17T17:18:46.437 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-08-17T17:18:46.437 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-08-17T17:18:46.437 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_compact_on_bootstrap": "false", 2024-08-17T17:18:46.437 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_compact_on_start": "false", 2024-08-17T17:18:46.437 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_compact_on_trim": "true", 2024-08-17T17:18:46.437 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_con_tracker_persist_interval": "10", 2024-08-17T17:18:46.437 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_con_tracker_score_halflife": "43200", 2024-08-17T17:18:46.437 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_config_key_max_entry_size": "65536", 2024-08-17T17:18:46.437 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cpu_threads": "4", 2024-08-17T17:18:46.438 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_crush_min_required_version": "hammer", 2024-08-17T17:18:46.438 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_daemon_bytes": "419430400", 2024-08-17T17:18:46.438 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-08-17T17:18:46.438 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_data_avail_crit": "5", 2024-08-17T17:18:46.438 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_data_avail_warn": "5", 2024-08-17T17:18:46.438 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_data_size_warn": "16106127360", 2024-08-17T17:18:46.438 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_block_osdmap_trim": "false", 2024-08-17T17:18:46.438 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-08-17T17:18:46.438 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_dump_json": "false", 2024-08-17T17:18:46.438 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-08-17T17:18:46.438 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_dump_transactions": "false", 2024-08-17T17:18:46.438 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_extra_checks": "false", 2024-08-17T17:18:46.439 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-08-17T17:18:46.439 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-08-17T17:18:46.439 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_no_require_reef": "false", 2024-08-17T17:18:46.439 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_no_require_squid": "false", 2024-08-17T17:18:46.439 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-08-17T17:18:46.439 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_delta_reset_interval": "10.000000", 2024-08-17T17:18:46.439 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_dns_srv_name": "ceph-mon", 2024-08-17T17:18:46.439 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_down_mkfs_grace": "60", 2024-08-17T17:18:46.441 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_down_uptime_grace": "60", 2024-08-17T17:18:46.441 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_election_default_strategy": "3", 2024-08-17T17:18:46.441 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_election_timeout": "5.000000", 2024-08-17T17:18:46.441 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-08-17T17:18:46.442 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_elector_ping_divisor": "2", 2024-08-17T17:18:46.442 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_elector_ping_timeout": "2.000000", 2024-08-17T17:18:46.442 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_enable_op_tracker": "true", 2024-08-17T17:18:46.442 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_fake_pool_delete": "false", 2024-08-17T17:18:46.442 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_force_quorum_join": "false", 2024-08-17T17:18:46.442 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_fsmap_prune_threshold": "300", 2024-08-17T17:18:46.442 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_globalid_prealloc": "10000", 2024-08-17T17:18:46.442 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_health_detail_to_clog": "true", 2024-08-17T17:18:46.442 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_health_log_update_period": "5", 2024-08-17T17:18:46.442 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_health_max_detail": "50", 2024-08-17T17:18:46.442 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_health_to_clog": "true", 2024-08-17T17:18:46.442 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_health_to_clog_interval": "600", 2024-08-17T17:18:46.442 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-08-17T17:18:46.443 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_host": "[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0]", 2024-08-17T17:18:46.443 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_host_override": "", 2024-08-17T17:18:46.443 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_initial_members": "", 2024-08-17T17:18:46.443 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-08-17T17:18:46.443 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-08-17T17:18:46.443 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-08-17T17:18:46.443 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-08-17T17:18:46.443 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_keyvaluedb": "rocksdb", 2024-08-17T17:18:46.443 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_lease": "5.000000", 2024-08-17T17:18:46.443 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-08-17T17:18:46.443 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-08-17T17:18:46.443 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_log_full_interval": "50", 2024-08-17T17:18:46.443 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_log_max": "10000", 2024-08-17T17:18:46.443 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_log_max_summary": "50", 2024-08-17T17:18:46.444 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_log_entries_per_event": "4096", 2024-08-17T17:18:46.444 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_log_epochs": "500", 2024-08-17T17:18:46.444 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_mdsmap_epochs": "500", 2024-08-17T17:18:46.444 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_mgrmap_epochs": "500", 2024-08-17T17:18:46.444 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_nvmeof_epochs": "500", 2024-08-17T17:18:46.444 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_osd": "10000", 2024-08-17T17:18:46.444 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_pg_per_osd": "10000", 2024-08-17T17:18:46.444 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_pool_pg_num": "65536", 2024-08-17T17:18:46.444 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-08-17T17:18:46.445 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-08-17T17:18:46.445 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mds_force_trim_to": "0", 2024-08-17T17:18:46.445 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mds_skip_sanity": "true", 2024-08-17T17:18:46.445 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_memory_autotune": "true", 2024-08-17T17:18:46.445 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_memory_target": "2147483648", 2024-08-17T17:18:46.445 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mgr_beacon_grace": "30", 2024-08-17T17:18:46.445 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-08-17T17:18:46.445 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mgr_digest_period": "5", 2024-08-17T17:18:46.445 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mgr_inactive_grace": "60", 2024-08-17T17:18:46.445 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mgr_mkfs_grace": "240", 2024-08-17T17:18:46.445 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-08-17T17:18:46.445 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_min_osdmap_epochs": "500", 2024-08-17T17:18:46.445 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-08-17T17:18:46.446 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-08-17T17:18:46.446 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_op_complaint_time": "30", 2024-08-17T17:18:46.446 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_op_history_duration": "600", 2024-08-17T17:18:46.446 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_op_history_size": "20", 2024-08-17T17:18:46.446 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_op_history_slow_op_size": "20", 2024-08-17T17:18:46.446 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_op_history_slow_op_threshold": "10", 2024-08-17T17:18:46.446 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_op_log_threshold": "5", 2024-08-17T17:18:46.446 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-08-17T17:18:46.446 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-08-17T17:18:46.446 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-08-17T17:18:46.446 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_auto_mark_in": "false", 2024-08-17T17:18:46.446 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_auto_mark_new_in": "true", 2024-08-17T17:18:46.447 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-08-17T17:18:46.447 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-08-17T17:18:46.447 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_cache_size": "500", 2024-08-17T17:18:46.447 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_cache_size_min": "134217728", 2024-08-17T17:18:46.447 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_crush_smoke_test": "true", 2024-08-17T17:18:46.447 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_destroyed_out_interval": "600", 2024-08-17T17:18:46.447 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_down_out_interval": "600", 2024-08-17T17:18:46.447 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-08-17T17:18:46.447 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-08-17T17:18:46.447 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_force_trim_to": "0", 2024-08-17T17:18:46.447 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_full_ratio": "0.950000", 2024-08-17T17:18:46.447 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-08-17T17:18:46.447 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_laggy_halflife": "3600", 2024-08-17T17:18:46.448 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_laggy_max_interval": "300", 2024-08-17T17:18:46.448 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_laggy_weight": "0.300000", 2024-08-17T17:18:46.448 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-08-17T17:18:46.448 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_max_initial_pgs": "1024", 2024-08-17T17:18:46.448 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_min_down_reporters": "2", 2024-08-17T17:18:46.448 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-08-17T17:18:46.448 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-08-17T17:18:46.448 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-08-17T17:18:46.448 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_prime_pg_temp": "true", 2024-08-17T17:18:46.448 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-08-17T17:18:46.448 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-08-17T17:18:46.448 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_report_timeout": "900", 2024-08-17T17:18:46.449 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-08-17T17:18:46.449 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-08-17T17:18:46.449 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_warn_num_repaired": "10", 2024-08-17T17:18:46.449 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_warn_op_age": "32.000000", 2024-08-17T17:18:46.449 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-08-17T17:18:46.449 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osdmap_full_prune_interval": "10", 2024-08-17T17:18:46.449 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osdmap_full_prune_min": "10000", 2024-08-17T17:18:46.449 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-08-17T17:18:46.449 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-08-17T17:18:46.449 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pg_stuck_threshold": "60", 2024-08-17T17:18:46.449 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-08-17T17:18:46.449 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pg_warn_min_objects": "10000", 2024-08-17T17:18:46.449 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pg_warn_min_per_osd": "0", 2024-08-17T17:18:46.450 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-08-17T17:18:46.450 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pool_quota_crit_threshold": "0", 2024-08-17T17:18:46.450 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pool_quota_warn_threshold": "0", 2024-08-17T17:18:46.450 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_probe_timeout": "2.000000", 2024-08-17T17:18:46.450 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_reweight_max_change": "0.050000", 2024-08-17T17:18:46.450 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_reweight_max_osds": "4", 2024-08-17T17:18:46.450 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-08-17T17:18:46.450 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-08-17T17:18:46.450 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-08-17T17:18:46.450 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-08-17T17:18:46.450 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-08-17T17:18:46.450 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_scrub_interval": "86400", 2024-08-17T17:18:46.450 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_scrub_max_keys": "100", 2024-08-17T17:18:46.451 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_scrub_timeout": "300", 2024-08-17T17:18:46.451 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_session_timeout": "300", 2024-08-17T17:18:46.451 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_smart_report_timeout": "5", 2024-08-17T17:18:46.451 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_stat_smooth_intervals": "6", 2024-08-17T17:18:46.451 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-08-17T17:18:46.451 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_stretch_pool_min_size": "2", 2024-08-17T17:18:46.451 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_stretch_pool_size": "4", 2024-08-17T17:18:46.451 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-08-17T17:18:46.451 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_subscribe_interval": "86400.000000", 2024-08-17T17:18:46.451 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_sync_debug": "false", 2024-08-17T17:18:46.451 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_sync_max_payload_keys": "2000", 2024-08-17T17:18:46.451 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_sync_max_payload_size": "1048576", 2024-08-17T17:18:46.451 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_sync_provider_kill_at": "0", 2024-08-17T17:18:46.451 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_sync_requester_kill_at": "0", 2024-08-17T17:18:46.452 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_sync_timeout": "60.000000", 2024-08-17T17:18:46.452 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_target_pg_per_osd": "100", 2024-08-17T17:18:46.452 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_tick_interval": "5", 2024-08-17T17:18:46.452 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_timecheck_interval": "300.000000", 2024-08-17T17:18:46.452 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-08-17T17:18:46.452 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_use_min_delay_socket": "false", 2024-08-17T17:18:46.452 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_older_version_delay": "604800", 2024-08-17T17:18:46.452 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-08-17T17:18:46.452 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-08-17T17:18:46.452 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-08-17T17:18:46.452 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_filestore_osds": "true", 2024-08-17T17:18:46.452 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-08-17T17:18:46.452 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-08-17T17:18:46.452 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-08-17T17:18:46.453 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_misplaced": "false", 2024-08-17T17:18:46.453 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-08-17T17:18:46.453 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_older_version": "true", 2024-08-17T17:18:46.453 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-08-17T17:18:46.453 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_pool_no_app": "true", 2024-08-17T17:18:46.453 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-08-17T17:18:46.453 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-08-17T17:18:46.453 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-08-17T17:18:46.453 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-08-17T17:18:46.453 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_too_few_osds": "false", 2024-08-17T17:18:46.453 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-08-17T17:18:46.453 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-08-17T17:18:46.453 INFO:teuthology.orchestra.run.smithi003.stdout: "monmap": "", 2024-08-17T17:18:46.454 INFO:teuthology.orchestra.run.smithi003.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-08-17T17:18:46.454 INFO:teuthology.orchestra.run.smithi003.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-08-17T17:18:46.454 INFO:teuthology.orchestra.run.smithi003.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-08-17T17:18:46.454 INFO:teuthology.orchestra.run.smithi003.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-08-17T17:18:46.454 INFO:teuthology.orchestra.run.smithi003.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-08-17T17:18:46.454 INFO:teuthology.orchestra.run.smithi003.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-08-17T17:18:46.454 INFO:teuthology.orchestra.run.smithi003.stdout: "motr_tracing_enabled": "false", 2024-08-17T17:18:46.454 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_op_threads": "3", 2024-08-17T17:18:46.454 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_buffer_size": "131072", 2024-08-17T17:18:46.454 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_cm": "false", 2024-08-17T17:18:46.454 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_device_name": "", 2024-08-17T17:18:46.454 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_dscp": "96", 2024-08-17T17:18:46.454 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-08-17T17:18:46.454 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_gid_idx": "0", 2024-08-17T17:18:46.455 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_local_gid": "", 2024-08-17T17:18:46.455 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_polling_us": "1000", 2024-08-17T17:18:46.455 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_port_num": "1", 2024-08-17T17:18:46.455 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-08-17T17:18:46.455 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-08-17T17:18:46.455 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_roce_ver": "1", 2024-08-17T17:18:46.455 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_send_buffers": "1024", 2024-08-17T17:18:46.455 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_sl": "3", 2024-08-17T17:18:46.455 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_support_srq": "true", 2024-08-17T17:18:46.455 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_type": "ib", 2024-08-17T17:18:46.455 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_reap_threshold": "5", 2024-08-17T17:18:46.455 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_before_connect": "false", 2024-08-17T17:18:46.455 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_ipv4": "true", 2024-08-17T17:18:46.455 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_ipv6": "false", 2024-08-17T17:18:46.455 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_msgr1": "true", 2024-08-17T17:18:46.456 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_msgr2": "true", 2024-08-17T17:18:46.456 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_port_max": "7568", 2024-08-17T17:18:46.456 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_port_min": "6800", 2024-08-17T17:18:46.456 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_prefer_ipv4": "false", 2024-08-17T17:18:46.456 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_retry_count": "3", 2024-08-17T17:18:46.456 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_retry_delay": "5", 2024-08-17T17:18:46.456 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_blackhole_client": "false", 2024-08-17T17:18:46.456 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_blackhole_mds": "false", 2024-08-17T17:18:46.456 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_blackhole_mgr": "false", 2024-08-17T17:18:46.456 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_blackhole_mon": "false", 2024-08-17T17:18:46.456 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_blackhole_osd": "false", 2024-08-17T17:18:46.456 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_client_mode": "crc secure", 2024-08-17T17:18:46.456 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-08-17T17:18:46.457 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_cluster_mode": "crc secure", 2024-08-17T17:18:46.457 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_cluster_type": "", 2024-08-17T17:18:46.457 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_compress_secure": "false", 2024-08-17T17:18:46.457 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_connection_idle_timeout": "900", 2024-08-17T17:18:46.457 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_connection_ready_timeout": "10", 2024-08-17T17:18:46.457 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_crc_data": "true", 2024-08-17T17:18:46.457 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_crc_header": "true", 2024-08-17T17:18:46.457 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_die_on_bad_msg": "false", 2024-08-17T17:18:46.457 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_die_on_bug": "true", 2024-08-17T17:18:46.457 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_die_on_old_message": "true", 2024-08-17T17:18:46.457 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_die_on_skipped_message": "false", 2024-08-17T17:18:46.457 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_die_on_unhandled_msg": "false", 2024-08-17T17:18:46.457 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-08-17T17:18:46.457 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_coremask": "0xF", 2024-08-17T17:18:46.458 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-08-17T17:18:46.458 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_devs_allowlist": "", 2024-08-17T17:18:46.458 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_enable_tso": "true", 2024-08-17T17:18:46.458 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-08-17T17:18:46.458 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-08-17T17:18:46.458 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_hugepages": "", 2024-08-17T17:18:46.458 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_hw_flow_control": "true", 2024-08-17T17:18:46.458 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-08-17T17:18:46.458 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_lro": "true", 2024-08-17T17:18:46.458 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_memory_channel": "4", 2024-08-17T17:18:46.458 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-08-17T17:18:46.458 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_pmd": "", 2024-08-17T17:18:46.458 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_port_id": "0", 2024-08-17T17:18:46.459 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-08-17T17:18:46.459 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dump_corrupt_message_level": "1", 2024-08-17T17:18:46.459 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dump_on_send": "false", 2024-08-17T17:18:46.459 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_initial_backoff": "0.200000", 2024-08-17T17:18:46.459 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_inject_delay_max": "1.000000", 2024-08-17T17:18:46.459 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_inject_delay_probability": "0.000000", 2024-08-17T17:18:46.459 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_inject_delay_type": "", 2024-08-17T17:18:46.459 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_inject_internal_delays": "0.000000", 2024-08-17T17:18:46.459 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_inject_network_congestion": "0", 2024-08-17T17:18:46.459 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_inject_socket_failures": "0", 2024-08-17T17:18:46.459 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_learn_addr_from_peer": "true", 2024-08-17T17:18:46.459 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_max_accept_failures": "4", 2024-08-17T17:18:46.459 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_max_backoff": "15.000000", 2024-08-17T17:18:46.459 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_mon_client_mode": "secure crc", 2024-08-17T17:18:46.460 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_mon_cluster_mode": "secure crc", 2024-08-17T17:18:46.460 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_mon_service_mode": "secure crc", 2024-08-17T17:18:46.460 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_osd_compress_min_size": "1024", 2024-08-17T17:18:46.460 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_osd_compress_mode": "none", 2024-08-17T17:18:46.460 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_osd_compression_algorithm": "snappy", 2024-08-17T17:18:46.460 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-08-17T17:18:46.460 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_pq_min_cost": "65536", 2024-08-17T17:18:46.460 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_public_type": "", 2024-08-17T17:18:46.460 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_service_mode": "crc secure", 2024-08-17T17:18:46.460 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_tcp_listen_backlog": "512", 2024-08-17T17:18:46.460 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_tcp_nodelay": "true", 2024-08-17T17:18:46.460 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-08-17T17:18:46.460 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_tcp_rcvbuf": "0", 2024-08-17T17:18:46.460 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_time_events_min_wait_interval": "1000", 2024-08-17T17:18:46.460 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_type": "async+posix", 2024-08-17T17:18:46.461 INFO:teuthology.orchestra.run.smithi003.stdout: "no_config_file": "false", 2024-08-17T17:18:46.461 INFO:teuthology.orchestra.run.smithi003.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-08-17T17:18:46.461 INFO:teuthology.orchestra.run.smithi003.stdout: "nvmeof_mon_client_tick_period": "2", 2024-08-17T17:18:46.461 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_completion_locks_per_session": "32", 2024-08-17T17:18:46.461 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_debug_inject_relock_delay": "false", 2024-08-17T17:18:46.461 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_inflight_op_bytes": "104857600", 2024-08-17T17:18:46.461 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_inflight_ops": "1024", 2024-08-17T17:18:46.461 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_inject_no_watch_ping": "false", 2024-08-17T17:18:46.461 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-08-17T17:18:46.461 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_tick_interval": "5.000000", 2024-08-17T17:18:46.461 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_timeout": "10.000000", 2024-08-17T17:18:46.461 INFO:teuthology.orchestra.run.smithi003.stdout: "objectstore_blackhole": "false", 2024-08-17T17:18:46.461 INFO:teuthology.orchestra.run.smithi003.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-08-17T17:18:46.462 INFO:teuthology.orchestra.run.smithi003.stdout: "openssl_engine_opts": "", 2024-08-17T17:18:46.462 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_agent_delay_time": "5.000000", 2024-08-17T17:18:46.462 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_agent_hist_halflife": "1000", 2024-08-17T17:18:46.462 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_agent_max_low_ops": "2", 2024-08-17T17:18:46.462 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_agent_max_ops": "4", 2024-08-17T17:18:46.462 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-08-17T17:18:46.462 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_agent_quantize_effort": "0.100000", 2024-08-17T17:18:46.462 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_agent_slop": "0.020000", 2024-08-17T17:18:46.462 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-08-17T17:18:46.462 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_allow_recovery_below_min_size": "true", 2024-08-17T17:18:46.462 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_asio_thread_count": "2", 2024-08-17T17:18:46.462 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_async_recovery_min_cost": "100", 2024-08-17T17:18:46.462 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_auto_mark_unfound_lost": "false", 2024-08-17T17:18:46.463 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_backfill_retry_interval": "30.000000", 2024-08-17T17:18:46.463 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_backfill_scan_max": "512", 2024-08-17T17:18:46.463 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_backfill_scan_min": "64", 2024-08-17T17:18:46.463 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_backoff_on_degraded": "false", 2024-08-17T17:18:46.463 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_backoff_on_peering": "false", 2024-08-17T17:18:46.463 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_backoff_on_unfound": "true", 2024-08-17T17:18:46.463 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_beacon_report_interval": "300", 2024-08-17T17:18:46.463 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_bench_duration": "30", 2024-08-17T17:18:46.463 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-08-17T17:18:46.463 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_bench_max_block_size": "67108864", 2024-08-17T17:18:46.463 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_bench_small_size_max_iops": "100", 2024-08-17T17:18:46.463 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_blkin_trace_all": "false", 2024-08-17T17:18:46.463 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_blocked_scrub_grace_period": "120", 2024-08-17T17:18:46.464 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-08-17T17:18:46.464 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-08-17T17:18:46.464 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-08-17T17:18:46.464 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_check_for_log_corruption": "false", 2024-08-17T17:18:46.464 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-08-17T17:18:46.464 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-08-17T17:18:46.464 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-08-17T17:18:46.464 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-08-17T17:18:46.464 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_class_update_on_start": "true", 2024-08-17T17:18:46.464 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_client_message_cap": "256", 2024-08-17T17:18:46.464 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_client_message_size_cap": "524288000", 2024-08-17T17:18:46.464 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_client_op_priority": "63", 2024-08-17T17:18:46.464 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_client_watch_timeout": "30", 2024-08-17T17:18:46.465 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_command_max_records": "256", 2024-08-17T17:18:46.465 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_compact_on_start": "false", 2024-08-17T17:18:46.465 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-08-17T17:18:46.465 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_crush_chooseleaf_type": "1", 2024-08-17T17:18:46.465 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_crush_initial_weight": "-1.000000", 2024-08-17T17:18:46.465 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_crush_update_on_start": "true", 2024-08-17T17:18:46.465 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_crush_update_weight_set": "true", 2024-08-17T17:18:46.465 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-08-17T17:18:46.465 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-08-17T17:18:46.465 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-08-17T17:18:46.465 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_drop_ping_duration": "0", 2024-08-17T17:18:46.465 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-08-17T17:18:46.465 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_feed_pullee": "-1", 2024-08-17T17:18:46.466 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-08-17T17:18:46.466 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-08-17T17:18:46.466 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-08-17T17:18:46.466 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_misdirected_ops": "false", 2024-08-17T17:18:46.466 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_no_acting_change": "false", 2024-08-17T17:18:46.466 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_no_purge_strays": "false", 2024-08-17T17:18:46.466 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_op_order": "false", 2024-08-17T17:18:46.466 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_pg_log_writeout": "false", 2024-08-17T17:18:46.466 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_pretend_recovery_active": "false", 2024-08-17T17:18:46.466 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-08-17T17:18:46.466 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-08-17T17:18:46.466 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_shutdown": "false", 2024-08-17T17:18:46.466 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-08-17T17:18:46.466 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-08-17T17:18:46.467 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_trim_objects": "false", 2024-08-17T17:18:46.467 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_verify_cached_snaps": "false", 2024-08-17T17:18:46.467 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_verify_missing_on_start": "false", 2024-08-17T17:18:46.467 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_verify_snaps": "false", 2024-08-17T17:18:46.467 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-08-17T17:18:46.467 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-08-17T17:18:46.467 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_deep_scrub_keys": "1024", 2024-08-17T17:18:46.467 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-08-17T17:18:46.467 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-08-17T17:18:46.467 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-08-17T17:18:46.467 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_deep_scrub_stride": "524288", 2024-08-17T17:18:46.467 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-08-17T17:18:46.467 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_default_data_pool_replay_window": "45", 2024-08-17T17:18:46.468 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_default_notify_timeout": "30", 2024-08-17T17:18:46.468 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_delete_sleep": "0.000000", 2024-08-17T17:18:46.468 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-08-17T17:18:46.468 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-08-17T17:18:46.468 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-08-17T17:18:46.468 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_discard_disconnected_ops": "true", 2024-08-17T17:18:46.468 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_ec_partial_reads": "true", 2024-08-17T17:18:46.468 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_enable_op_tracker": "true", 2024-08-17T17:18:46.468 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-08-17T17:18:46.468 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_extblkdev_plugins": "vdo", 2024-08-17T17:18:46.468 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-08-17T17:18:46.468 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-08-17T17:18:46.468 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_fast_info": "true", 2024-08-17T17:18:46.468 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_fast_shutdown": "true", 2024-08-17T17:18:46.469 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-08-17T17:18:46.469 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_fast_shutdown_timeout": "15", 2024-08-17T17:18:46.469 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-08-17T17:18:46.469 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-08-17T17:18:46.469 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-08-17T17:18:46.469 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_function_tracing": "false", 2024-08-17T17:18:46.469 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_heartbeat_grace": "20", 2024-08-17T17:18:46.469 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_heartbeat_interval": "6", 2024-08-17T17:18:46.469 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-08-17T17:18:46.469 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_heartbeat_min_peers": "10", 2024-08-17T17:18:46.469 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_heartbeat_min_size": "2000", 2024-08-17T17:18:46.469 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_heartbeat_stale": "600", 2024-08-17T17:18:46.469 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-08-17T17:18:46.469 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_hit_set_max_size": "100000", 2024-08-17T17:18:46.469 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_hit_set_min_size": "1000", 2024-08-17T17:18:46.470 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-08-17T17:18:46.470 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-08-17T17:18:46.470 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-08-17T17:18:46.470 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-08-17T17:18:46.470 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-08-17T17:18:46.470 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_journal_flush_on_shutdown": "true", 2024-08-17T17:18:46.470 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_journal_size": "5120", 2024-08-17T17:18:46.470 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_kill_backfill_at": "0", 2024-08-17T17:18:46.470 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_loop_before_reset_tphandle": "64", 2024-08-17T17:18:46.470 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_map_cache_size": "50", 2024-08-17T17:18:46.470 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_map_dedup": "true", 2024-08-17T17:18:46.470 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_map_message_max": "40", 2024-08-17T17:18:46.470 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_map_message_max_bytes": "10485760", 2024-08-17T17:18:46.471 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_map_share_max_epochs": "40", 2024-08-17T17:18:46.471 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_attr_name_len": "100", 2024-08-17T17:18:46.471 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_attr_size": "0", 2024-08-17T17:18:46.471 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_backfills": "1", 2024-08-17T17:18:46.471 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_markdown_count": "5", 2024-08-17T17:18:46.471 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_markdown_period": "600", 2024-08-17T17:18:46.471 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_object_name_len": "2048", 2024-08-17T17:18:46.471 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_object_namespace_len": "256", 2024-08-17T17:18:46.471 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_object_size": "134217728", 2024-08-17T17:18:46.471 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-08-17T17:18:46.471 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_omap_entries_per_request": "1024", 2024-08-17T17:18:46.471 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_pg_blocked_by": "16", 2024-08-17T17:18:46.471 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_pg_log_entries": "10000", 2024-08-17T17:18:46.471 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-08-17T17:18:46.472 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_pgls": "1024", 2024-08-17T17:18:46.472 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_push_cost": "8388608", 2024-08-17T17:18:46.472 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_push_objects": "10", 2024-08-17T17:18:46.472 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_scrubs": "3", 2024-08-17T17:18:46.472 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-08-17T17:18:46.472 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_trimming_pgs": "2", 2024-08-17T17:18:46.472 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_write_op_reply_len": "64", 2024-08-17T17:18:46.472 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_write_size": "90", 2024-08-17T17:18:46.472 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-08-17T17:18:46.472 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-08-17T17:18:46.472 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-08-17T17:18:46.472 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-08-17T17:18:46.472 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-08-17T17:18:46.472 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-08-17T17:18:46.473 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-08-17T17:18:46.473 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_override_recovery_settings": "false", 2024-08-17T17:18:46.473 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_profile": "balanced", 2024-08-17T17:18:46.473 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-08-17T17:18:46.473 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-08-17T17:18:46.473 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-08-17T17:18:46.473 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-08-17T17:18:46.473 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-08-17T17:18:46.473 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-08-17T17:18:46.473 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-08-17T17:18:46.473 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-08-17T17:18:46.473 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-08-17T17:18:46.473 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-08-17T17:18:46.473 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_skip_benchmark": "false", 2024-08-17T17:18:46.473 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_memory_base": "805306368", 2024-08-17T17:18:46.474 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_memory_cache_min": "134217728", 2024-08-17T17:18:46.474 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-08-17T17:18:46.474 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-08-17T17:18:46.474 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_memory_target": "4294967296", 2024-08-17T17:18:46.474 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_memory_target_autotune": "false", 2024-08-17T17:18:46.474 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-08-17T17:18:46.474 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_min_pg_log_entries": "250", 2024-08-17T17:18:46.474 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_min_recovery_priority": "0", 2024-08-17T17:18:46.474 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mon_heartbeat_interval": "30", 2024-08-17T17:18:46.474 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-08-17T17:18:46.474 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mon_report_interval": "5", 2024-08-17T17:18:46.474 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mon_report_max_in_flight": "2", 2024-08-17T17:18:46.474 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-08-17T17:18:46.474 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_num_cache_shards": "32", 2024-08-17T17:18:46.475 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_num_op_tracker_shard": "32", 2024-08-17T17:18:46.475 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_numa_auto_affinity": "true", 2024-08-17T17:18:46.475 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_numa_node": "-1", 2024-08-17T17:18:46.475 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_numa_prefer_iface": "true", 2024-08-17T17:18:46.475 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-08-17T17:18:46.475 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_objecter_finishers": "1", 2024-08-17T17:18:46.475 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_objectstore": "bluestore", 2024-08-17T17:18:46.475 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_objectstore_fuse": "false", 2024-08-17T17:18:46.475 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_objectstore_ideal_list_max": "64", 2024-08-17T17:18:46.475 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_objectstore_tracing": "false", 2024-08-17T17:18:46.475 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_complaint_time": "30.000000", 2024-08-17T17:18:46.475 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_history_duration": "600", 2024-08-17T17:18:46.475 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_history_size": "20", 2024-08-17T17:18:46.475 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_history_slow_op_size": "20", 2024-08-17T17:18:46.475 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-08-17T17:18:46.476 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_log_threshold": "5", 2024-08-17T17:18:46.476 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_num_shards": "0", 2024-08-17T17:18:46.476 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_num_shards_hdd": "5", 2024-08-17T17:18:46.476 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_num_shards_ssd": "8", 2024-08-17T17:18:46.476 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_num_threads_per_shard": "0", 2024-08-17T17:18:46.476 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-08-17T17:18:46.476 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-08-17T17:18:46.476 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-08-17T17:18:46.476 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_pq_min_cost": "65536", 2024-08-17T17:18:46.476 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_queue": "mclock_scheduler", 2024-08-17T17:18:46.476 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_queue_cut_off": "high", 2024-08-17T17:18:46.476 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_thread_suicide_timeout": "150", 2024-08-17T17:18:46.476 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_thread_timeout": "15", 2024-08-17T17:18:46.476 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_open_classes_on_start": "true", 2024-08-17T17:18:46.477 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_os_flags": "0", 2024-08-17T17:18:46.477 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_peering_op_priority": "255", 2024-08-17T17:18:46.477 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_delete_cost": "1048576", 2024-08-17T17:18:46.477 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_delete_priority": "5", 2024-08-17T17:18:46.477 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-08-17T17:18:46.477 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-08-17T17:18:46.477 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_log_dups_tracked": "3000", 2024-08-17T17:18:46.477 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_log_trim_max": "10000", 2024-08-17T17:18:46.477 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_log_trim_min": "100", 2024-08-17T17:18:46.477 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-08-17T17:18:46.477 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_object_context_cache_count": "64", 2024-08-17T17:18:46.478 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-08-17T17:18:46.478 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-08-17T17:18:46.478 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-08-17T17:18:46.478 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-08-17T17:18:46.478 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-08-17T17:18:46.478 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-08-17T17:18:46.478 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-08-17T17:18:46.478 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-08-17T17:18:46.478 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_crimson": "false", 2024-08-17T17:18:46.478 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_crush_rule": "-1", 2024-08-17T17:18:46.478 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_ec_fast_read": "false", 2024-08-17T17:18:46.478 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-08-17T17:18:46.479 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_flag_bulk": "false", 2024-08-17T17:18:46.479 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-08-17T17:18:46.479 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_flag_nodelete": "false", 2024-08-17T17:18:46.479 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-08-17T17:18:46.479 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-08-17T17:18:46.479 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_flags": "0", 2024-08-17T17:18:46.479 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-08-17T17:18:46.479 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_min_size": "0", 2024-08-17T17:18:46.479 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-08-17T17:18:46.479 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_pg_num": "32", 2024-08-17T17:18:46.479 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_pgp_num": "0", 2024-08-17T17:18:46.479 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-08-17T17:18:46.480 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_read_ratio": "70", 2024-08-17T17:18:46.480 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_size": "3", 2024-08-17T17:18:46.480 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_type": "replicated", 2024-08-17T17:18:46.480 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-08-17T17:18:46.480 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_use_gmt_hitset": "true", 2024-08-17T17:18:46.480 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_push_per_object_cost": "1000", 2024-08-17T17:18:46.480 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_read_ec_check_for_errors": "false", 2024-08-17T17:18:46.480 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recover_clone_overlap": "true", 2024-08-17T17:18:46.480 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recover_clone_overlap_limit": "10", 2024-08-17T17:18:46.480 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_cost": "20971520", 2024-08-17T17:18:46.480 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_delay_start": "0.000000", 2024-08-17T17:18:46.480 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_max_active": "0", 2024-08-17T17:18:46.480 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_max_active_hdd": "3", 2024-08-17T17:18:46.480 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_max_active_ssd": "10", 2024-08-17T17:18:46.480 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_max_chunk": "8388608", 2024-08-17T17:18:46.481 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-08-17T17:18:46.481 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_max_single_start": "1", 2024-08-17T17:18:46.481 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_op_priority": "3", 2024-08-17T17:18:46.481 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_op_warn_multiple": "16", 2024-08-17T17:18:46.481 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_priority": "5", 2024-08-17T17:18:46.481 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_retry_interval": "30.000000", 2024-08-17T17:18:46.481 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_sleep": "0.000000", 2024-08-17T17:18:46.481 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-08-17T17:18:46.481 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-08-17T17:18:46.481 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-08-17T17:18:46.481 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_repair_during_recovery": "false", 2024-08-17T17:18:46.481 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_requested_scrub_priority": "120", 2024-08-17T17:18:46.481 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-08-17T17:18:46.482 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_rollback_to_cluster_snap": "", 2024-08-17T17:18:46.482 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_auto_repair": "false", 2024-08-17T17:18:46.482 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-08-17T17:18:46.482 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-08-17T17:18:46.482 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_begin_hour": "0", 2024-08-17T17:18:46.482 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_begin_week_day": "0", 2024-08-17T17:18:46.482 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_chunk_max": "25", 2024-08-17T17:18:46.482 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_chunk_min": "5", 2024-08-17T17:18:46.482 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_cost": "52428800", 2024-08-17T17:18:46.482 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-08-17T17:18:46.483 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_during_recovery": "false", 2024-08-17T17:18:46.483 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_end_hour": "0", 2024-08-17T17:18:46.483 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_end_week_day": "0", 2024-08-17T17:18:46.483 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_event_cost": "4096", 2024-08-17T17:18:46.483 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-08-17T17:18:46.483 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-08-17T17:18:46.483 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_invalid_stats": "true", 2024-08-17T17:18:46.483 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_load_threshold": "0.500000", 2024-08-17T17:18:46.483 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_max_interval": "604800.000000", 2024-08-17T17:18:46.483 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_max_preemptions": "5", 2024-08-17T17:18:46.483 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_min_interval": "86400.000000", 2024-08-17T17:18:46.483 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_priority": "5", 2024-08-17T17:18:46.483 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_sleep": "0.000000", 2024-08-17T17:18:46.483 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-08-17T17:18:46.483 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-08-17T17:18:46.484 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_shutdown_pgref_assert": "false", 2024-08-17T17:18:46.484 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-08-17T17:18:46.484 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_skip_data_digest": "false", 2024-08-17T17:18:46.484 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_smart_report_timeout": "5", 2024-08-17T17:18:46.484 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_snap_trim_cost": "1048576", 2024-08-17T17:18:46.484 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_snap_trim_priority": "5", 2024-08-17T17:18:46.484 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_snap_trim_sleep": "0.000000", 2024-08-17T17:18:46.484 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-08-17T17:18:46.484 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-08-17T17:18:46.484 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-08-17T17:18:46.484 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-08-17T17:18:46.484 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_stats_update_period_scrubbing": "15", 2024-08-17T17:18:46.484 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-08-17T17:18:46.484 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_target_transaction_size": "30", 2024-08-17T17:18:46.484 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-08-17T17:18:46.485 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-08-17T17:18:46.485 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-08-17T17:18:46.485 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-08-17T17:18:46.485 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-08-17T17:18:46.485 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-08-17T17:18:46.485 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-08-17T17:18:46.485 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_mode": "writeback", 2024-08-17T17:18:46.485 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-08-17T17:18:46.485 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-08-17T17:18:46.485 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tracing": "false", 2024-08-17T17:18:46.485 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_use_stale_snap": "false", 2024-08-17T17:18:46.485 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-08-17T17:18:46.485 INFO:teuthology.orchestra.run.smithi003.stdout: "osdc_blkin_trace_all": "false", 2024-08-17T17:18:46.485 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_kill_at": "0", 2024-08-17T17:18:46.486 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_max_join_drift": "10", 2024-08-17T17:18:46.486 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_min": "500", 2024-08-17T17:18:46.486 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_min_wait": "0.050000", 2024-08-17T17:18:46.486 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_propose_interval": "1.000000", 2024-08-17T17:18:46.486 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_service_trim_max": "500", 2024-08-17T17:18:46.486 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_service_trim_max_multiplier": "20", 2024-08-17T17:18:46.486 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_service_trim_min": "250", 2024-08-17T17:18:46.486 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_stash_full_interval": "25", 2024-08-17T17:18:46.486 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_trim_max": "500", 2024-08-17T17:18:46.486 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_trim_min": "250", 2024-08-17T17:18:46.486 INFO:teuthology.orchestra.run.smithi003.stdout: "perf": "true", 2024-08-17T17:18:46.486 INFO:teuthology.orchestra.run.smithi003.stdout: "pid_file": "", 2024-08-17T17:18:46.486 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-08-17T17:18:46.486 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-08-17T17:18:46.486 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "-", 2024-08-17T17:18:46.487 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrv": "", 2024-08-17T17:18:46.487 INFO:teuthology.orchestra.run.smithi003.stdout: "public_bind_addr": "-", 2024-08-17T17:18:46.487 INFO:teuthology.orchestra.run.smithi003.stdout: "public_network": "172.21.0.0/20", 2024-08-17T17:18:46.487 INFO:teuthology.orchestra.run.smithi003.stdout: "public_network_interface": "", 2024-08-17T17:18:46.487 INFO:teuthology.orchestra.run.smithi003.stdout: "qat_compressor_busy_polling": "false", 2024-08-17T17:18:46.487 INFO:teuthology.orchestra.run.smithi003.stdout: "qat_compressor_enabled": "false", 2024-08-17T17:18:46.487 INFO:teuthology.orchestra.run.smithi003.stdout: "qat_compressor_session_max_number": "256", 2024-08-17T17:18:46.487 INFO:teuthology.orchestra.run.smithi003.stdout: "rados_mon_op_timeout": "0", 2024-08-17T17:18:46.487 INFO:teuthology.orchestra.run.smithi003.stdout: "rados_osd_op_timeout": "0", 2024-08-17T17:18:46.487 INFO:teuthology.orchestra.run.smithi003.stdout: "rados_tracing": "false", 2024-08-17T17:18:46.487 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_atime_update_interval": "60", 2024-08-17T17:18:46.487 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-08-17T17:18:46.487 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_balance_parent_reads": "false", 2024-08-17T17:18:46.487 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_balance_snap_reads": "false", 2024-08-17T17:18:46.487 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_blkin_trace_all": "false", 2024-08-17T17:18:46.488 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_blocklist_expire_seconds": "0", 2024-08-17T17:18:46.488 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_blocklist_on_break_lock": "true", 2024-08-17T17:18:46.488 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache": "true", 2024-08-17T17:18:46.488 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_block_writes_upfront": "false", 2024-08-17T17:18:46.488 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_max_dirty": "25165824", 2024-08-17T17:18:46.488 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-08-17T17:18:46.488 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_max_dirty_object": "0", 2024-08-17T17:18:46.488 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_policy": "writearound", 2024-08-17T17:18:46.488 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_size": "33554432", 2024-08-17T17:18:46.488 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_target_dirty": "16777216", 2024-08-17T17:18:46.488 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-08-17T17:18:46.488 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_clone_copy_on_read": "false", 2024-08-17T17:18:46.488 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_compression_hint": "none", 2024-08-17T17:18:46.488 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_concurrent_management_ops": "10", 2024-08-17T17:18:46.488 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-08-17T17:18:46.489 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_clone_format": "auto", 2024-08-17T17:18:46.489 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_data_pool": "", 2024-08-17T17:18:46.489 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_features": "61", 2024-08-17T17:18:46.489 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_format": "2", 2024-08-17T17:18:46.489 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_map_options": "", 2024-08-17T17:18:46.489 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_order": "22", 2024-08-17T17:18:46.489 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_pool": "rbd", 2024-08-17T17:18:46.489 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-08-17T17:18:46.489 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_stripe_count": "0", 2024-08-17T17:18:46.489 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_stripe_unit": "0", 2024-08-17T17:18:46.489 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_disable_zero_copy_writes": "true", 2024-08-17T17:18:46.489 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_discard_granularity_bytes": "65536", 2024-08-17T17:18:46.489 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-08-17T17:18:46.489 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_enable_alloc_hint": "true", 2024-08-17T17:18:46.489 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-08-17T17:18:46.490 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_io_scheduler": "simple", 2024-08-17T17:18:46.490 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-08-17T17:18:46.490 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_commit_age": "5.000000", 2024-08-17T17:18:46.490 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-08-17T17:18:46.490 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-08-17T17:18:46.490 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-08-17T17:18:46.490 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-08-17T17:18:46.490 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_object_flush_interval": "0", 2024-08-17T17:18:46.490 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-08-17T17:18:46.490 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-08-17T17:18:46.490 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_order": "24", 2024-08-17T17:18:46.490 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_pool": "", 2024-08-17T17:18:46.490 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_splay_width": "4", 2024-08-17T17:18:46.490 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_localize_parent_reads": "false", 2024-08-17T17:18:46.490 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_localize_snap_reads": "false", 2024-08-17T17:18:46.491 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-08-17T17:18:46.491 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-08-17T17:18:46.491 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-08-17T17:18:46.491 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-08-17T17:18:46.491 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-08-17T17:18:46.491 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-08-17T17:18:46.491 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_image_policy_type": "simple", 2024-08-17T17:18:46.491 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-08-17T17:18:46.491 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-08-17T17:18:46.491 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-08-17T17:18:46.491 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-08-17T17:18:46.491 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-08-17T17:18:46.491 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-08-17T17:18:46.491 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-08-17T17:18:46.492 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_memory_autotune": "true", 2024-08-17T17:18:46.492 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_memory_base": "805306368", 2024-08-17T17:18:46.492 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-08-17T17:18:46.492 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-08-17T17:18:46.492 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-08-17T17:18:46.492 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-08-17T17:18:46.492 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_memory_target": "4294967296", 2024-08-17T17:18:46.492 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-08-17T17:18:46.492 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-08-17T17:18:46.492 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-08-17T17:18:46.492 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirroring_delete_delay": "0", 2024-08-17T17:18:46.492 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-08-17T17:18:46.493 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirroring_replay_delay": "0", 2024-08-17T17:18:46.493 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-08-17T17:18:46.493 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-08-17T17:18:46.493 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_move_to_trash_on_remove": "false", 2024-08-17T17:18:46.493 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-08-17T17:18:46.493 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mtime_update_interval": "60", 2024-08-17T17:18:46.493 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_non_blocking_aio": "true", 2024-08-17T17:18:46.493 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_op_thread_timeout": "60", 2024-08-17T17:18:46.493 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_op_threads": "1", 2024-08-17T17:18:46.493 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_parent_cache_enabled": "false", 2024-08-17T17:18:46.493 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_persistent_cache_mode": "disabled", 2024-08-17T17:18:46.493 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_persistent_cache_path": "/tmp", 2024-08-17T17:18:46.493 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_persistent_cache_size": "1073741824", 2024-08-17T17:18:46.493 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_plugins": "", 2024-08-17T17:18:46.494 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_bps_burst": "0", 2024-08-17T17:18:46.494 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-08-17T17:18:46.494 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_bps_limit": "0", 2024-08-17T17:18:46.494 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_exclude_ops": "0", 2024-08-17T17:18:46.494 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_iops_burst": "0", 2024-08-17T17:18:46.494 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-08-17T17:18:46.494 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_iops_limit": "0", 2024-08-17T17:18:46.494 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_read_bps_burst": "0", 2024-08-17T17:18:46.494 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-08-17T17:18:46.494 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_read_bps_limit": "0", 2024-08-17T17:18:46.494 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_read_iops_burst": "0", 2024-08-17T17:18:46.494 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-08-17T17:18:46.495 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_read_iops_limit": "0", 2024-08-17T17:18:46.495 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_schedule_tick_min": "50", 2024-08-17T17:18:46.495 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_write_bps_burst": "0", 2024-08-17T17:18:46.495 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-08-17T17:18:46.495 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_write_bps_limit": "0", 2024-08-17T17:18:46.495 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_write_iops_burst": "0", 2024-08-17T17:18:46.495 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-08-17T17:18:46.495 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_write_iops_limit": "0", 2024-08-17T17:18:46.495 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_quiesce_notification_attempts": "10", 2024-08-17T17:18:46.495 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_read_from_replica_policy": "default", 2024-08-17T17:18:46.495 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-08-17T17:18:46.495 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_readahead_max_bytes": "524288", 2024-08-17T17:18:46.495 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_readahead_trigger_requests": "10", 2024-08-17T17:18:46.495 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_request_timed_out_seconds": "30", 2024-08-17T17:18:46.495 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_skip_partial_discard": "true", 2024-08-17T17:18:46.496 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-08-17T17:18:46.496 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_tracing": "false", 2024-08-17T17:18:46.496 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_validate_names": "true", 2024-08-17T17:18:46.496 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_validate_pool": "true", 2024-08-17T17:18:46.496 INFO:teuthology.orchestra.run.smithi003.stdout: "restapi_base_url": "", 2024-08-17T17:18:46.496 INFO:teuthology.orchestra.run.smithi003.stdout: "restapi_log_level": "", 2024-08-17T17:18:46.496 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-08-17T17:18:46.496 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_account_default_quota_max_size": "-1", 2024-08-17T17:18:46.496 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_acl_grants_max_num": "100", 2024-08-17T17:18:46.496 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_admin_entry": "admin", 2024-08-17T17:18:46.496 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-08-17T17:18:46.496 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_asio_assert_yielding": "false", 2024-08-17T17:18:46.496 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_backend_store": "rados", 2024-08-17T17:18:46.496 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_barbican_url": "", 2024-08-17T17:18:46.496 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_beast_enable_async": "true", 2024-08-17T17:18:46.497 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_counters_cache": "false", 2024-08-17T17:18:46.497 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-08-17T17:18:46.497 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-08-17T17:18:46.497 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-08-17T17:18:46.497 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_index_max_aio": "128", 2024-08-17T17:18:46.497 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-08-17T17:18:46.497 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-08-17T17:18:46.497 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_quota_ttl": "600", 2024-08-17T17:18:46.497 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-08-17T17:18:46.497 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_cache_enabled": "true", 2024-08-17T17:18:46.497 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_cache_expiry_interval": "900", 2024-08-17T17:18:46.497 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_cache_lru_size": "25000", 2024-08-17T17:18:46.497 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_config_store": "rados", 2024-08-17T17:18:46.497 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_content_length_compat": "false", 2024-08-17T17:18:46.497 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_copy_obj_progress": "true", 2024-08-17T17:18:46.498 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-08-17T17:18:46.498 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_cors_rules_max_num": "100", 2024-08-17T17:18:46.498 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_cross_domain_policy": "", 2024-08-17T17:18:46.498 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_default_encryption_key": "", 2024-08-17T17:18:46.498 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_addr": "", 2024-08-17T17:18:46.498 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_ca_path": "", 2024-08-17T17:18:46.498 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_client_cert": "", 2024-08-17T17:18:46.498 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_client_key": "", 2024-08-17T17:18:46.498 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-08-17T17:18:46.498 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_password": "", 2024-08-17T17:18:46.498 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-08-17T17:18:46.498 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_username": "", 2024-08-17T17:18:46.499 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_require_ssl": "true", 2024-08-17T17:18:46.499 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-08-17T17:18:46.499 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-08-17T17:18:46.499 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-08-17T17:18:46.499 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-08-17T17:18:46.499 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-08-17T17:18:46.499 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-08-17T17:18:46.499 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-08-17T17:18:46.499 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-08-17T17:18:46.499 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-08-17T17:18:46.499 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-08-17T17:18:46.499 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-08-17T17:18:46.499 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-08-17T17:18:46.499 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-08-17T17:18:46.500 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-08-17T17:18:46.500 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_suppress_logs": "true", 2024-08-17T17:18:46.500 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_addr": "", 2024-08-17T17:18:46.500 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_auth": "token", 2024-08-17T17:18:46.500 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_namespace": "", 2024-08-17T17:18:46.500 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_prefix": "", 2024-08-17T17:18:46.500 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-08-17T17:18:46.500 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-08-17T17:18:46.500 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-08-17T17:18:46.500 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-08-17T17:18:46.500 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_token_file": "", 2024-08-17T17:18:46.500 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-08-17T17:18:46.500 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_curl_buffersize": "524288", 2024-08-17T17:18:46.500 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_curl_low_speed_limit": "1024", 2024-08-17T17:18:46.501 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_curl_low_speed_time": "300", 2024-08-17T17:18:46.501 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_curl_tcp_keepalive": "0", 2024-08-17T17:18:46.501 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-08-17T17:18:46.501 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-08-17T17:18:46.501 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-08-17T17:18:46.501 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-08-17T17:18:46.501 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-08-17T17:18:46.501 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_l1_fadvise": "4", 2024-08-17T17:18:46.501 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-08-17T17:18:46.501 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-08-17T17:18:46.501 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-08-17T17:18:46.501 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-08-17T17:18:46.501 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-08-17T17:18:46.501 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-08-17T17:18:46.502 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-08-17T17:18:46.502 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-08-17T17:18:46.502 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d4n_l1_fadvise": "4", 2024-08-17T17:18:46.502 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-08-17T17:18:46.502 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-08-17T17:18:46.502 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-08-17T17:18:46.502 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data_log_changes_size": "1000", 2024-08-17T17:18:46.502 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data_log_num_shards": "128", 2024-08-17T17:18:46.502 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-08-17T17:18:46.502 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data_log_window": "30", 2024-08-17T17:18:46.502 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data_notify_interval_msec": "0", 2024-08-17T17:18:46.502 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data_sync_poll_interval": "20", 2024-08-17T17:18:46.503 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data_sync_spawn_window": "20", 2024-08-17T17:18:46.503 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-08-17T17:18:46.503 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-08-17T17:18:46.503 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-08-17T17:18:46.503 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_default_data_log_backing": "fifo", 2024-08-17T17:18:46.503 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-08-17T17:18:46.503 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_default_region_info_oid": "default.region", 2024-08-17T17:18:46.503 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-08-17T17:18:46.503 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-08-17T17:18:46.503 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_defer_to_bucket_acls": "", 2024-08-17T17:18:46.503 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-08-17T17:18:46.503 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_disable_s3select": "false", 2024-08-17T17:18:46.503 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-08-17T17:18:46.503 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-08-17T17:18:46.504 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-08-17T17:18:46.504 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-08-17T17:18:46.504 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-08-17T17:18:46.504 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-08-17T17:18:46.504 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-08-17T17:18:46.504 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_data_res": "500.000000", 2024-08-17T17:18:46.504 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-08-17T17:18:46.504 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-08-17T17:18:46.504 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-08-17T17:18:46.504 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-08-17T17:18:46.504 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dns_name": "", 2024-08-17T17:18:46.504 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dns_s3website_name": "", 2024-08-17T17:18:46.504 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dynamic_resharding": "true", 2024-08-17T17:18:46.504 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-08-17T17:18:46.504 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-08-17T17:18:46.505 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-08-17T17:18:46.505 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enable_gc_threads": "true", 2024-08-17T17:18:46.505 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enable_lc_threads": "true", 2024-08-17T17:18:46.505 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enable_mdsearch": "true", 2024-08-17T17:18:46.505 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enable_ops_log": "false", 2024-08-17T17:18:46.505 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enable_quota_threads": "true", 2024-08-17T17:18:46.505 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enable_static_website": "false", 2024-08-17T17:18:46.505 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enable_usage_log": "false", 2024-08-17T17:18:46.505 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enforce_swift_acls": "true", 2024-08-17T17:18:46.505 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_exit_timeout_secs": "120", 2024-08-17T17:18:46.505 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_expose_bucket": "false", 2024-08-17T17:18:46.505 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_extended_http_attrs": "", 2024-08-17T17:18:46.505 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_filter": "none", 2024-08-17T17:18:46.505 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-08-17T17:18:46.506 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_frontends": "beast port=7480", 2024-08-17T17:18:46.506 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_max_concurrent_io": "10", 2024-08-17T17:18:46.506 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_max_deferred": "50", 2024-08-17T17:18:46.506 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-08-17T17:18:46.506 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_max_objs": "32", 2024-08-17T17:18:46.506 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_max_queue_size": "134213632", 2024-08-17T17:18:46.506 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_max_trim_chunk": "16", 2024-08-17T17:18:46.506 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_obj_min_wait": "7200", 2024-08-17T17:18:46.506 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_processor_max_time": "3600", 2024-08-17T17:18:46.506 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_processor_period": "3600", 2024-08-17T17:18:46.506 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-08-17T17:18:46.506 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_get_obj_window_size": "16777216", 2024-08-17T17:18:46.507 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_healthcheck_disabling_path": "", 2024-08-17T17:18:46.507 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ignore_get_invalid_range": "false", 2024-08-17T17:18:46.507 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_init_timeout": "300", 2024-08-17T17:18:46.507 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-08-17T17:18:46.507 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-08-17T17:18:46.507 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_kafka_connection_idle": "300", 2024-08-17T17:18:46.507 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_kafka_message_timeout": "5000", 2024-08-17T17:18:46.507 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_kafka_sleep_timeout": "10", 2024-08-17T17:18:46.507 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-08-17T17:18:46.507 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-08-17T17:18:46.507 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-08-17T17:18:46.507 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_domain": "", 2024-08-17T17:18:46.507 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_password": "", 2024-08-17T17:18:46.507 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_password_path": "", 2024-08-17T17:18:46.508 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_project": "", 2024-08-17T17:18:46.508 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_tenant": "", 2024-08-17T17:18:46.508 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_token": "", 2024-08-17T17:18:46.508 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_token_path": "", 2024-08-17T17:18:46.508 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_user": "", 2024-08-17T17:18:46.508 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_api_version": "2", 2024-08-17T17:18:46.508 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_barbican_domain": "", 2024-08-17T17:18:46.508 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_barbican_password": "", 2024-08-17T17:18:46.508 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_barbican_project": "", 2024-08-17T17:18:46.508 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_barbican_tenant": "", 2024-08-17T17:18:46.508 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_barbican_user": "", 2024-08-17T17:18:46.508 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-08-17T17:18:46.508 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_implicit_tenants": "false", 2024-08-17T17:18:46.508 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-08-17T17:18:46.509 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_service_token_enabled": "false", 2024-08-17T17:18:46.509 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_token_cache_size": "10000", 2024-08-17T17:18:46.509 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_url": "", 2024-08-17T17:18:46.509 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_verify_ssl": "true", 2024-08-17T17:18:46.509 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lc_debug_interval": "-1", 2024-08-17T17:18:46.509 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lc_lock_max_time": "90", 2024-08-17T17:18:46.509 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lc_max_objs": "32", 2024-08-17T17:18:46.509 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lc_max_rules": "1000", 2024-08-17T17:18:46.509 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lc_max_worker": "3", 2024-08-17T17:18:46.509 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lc_max_wp_worker": "3", 2024-08-17T17:18:46.509 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lc_thread_delay": "0", 2024-08-17T17:18:46.509 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-08-17T17:18:46.509 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ldap_dnattr": "uid", 2024-08-17T17:18:46.509 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-08-17T17:18:46.510 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ldap_searchfilter": "", 2024-08-17T17:18:46.510 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-08-17T17:18:46.510 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ldap_uri": "", 2024-08-17T17:18:46.510 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lfuda_sync_frequency": "60", 2024-08-17T17:18:46.510 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-08-17T17:18:46.510 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-08-17T17:18:46.510 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-08-17T17:18:46.510 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_log_http_headers": "", 2024-08-17T17:18:46.510 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_log_nonexistent_bucket": "false", 2024-08-17T17:18:46.510 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-08-17T17:18:46.510 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_log_object_name_utc": "false", 2024-08-17T17:18:46.510 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-08-17T17:18:46.510 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-08-17T17:18:46.511 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_attr_name_len": "0", 2024-08-17T17:18:46.511 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_attr_size": "0", 2024-08-17T17:18:46.511 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_attrs_num_in_req": "0", 2024-08-17T17:18:46.511 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_chunk_size": "4194304", 2024-08-17T17:18:46.511 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_concurrent_requests": "1024", 2024-08-17T17:18:46.511 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-08-17T17:18:46.511 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_dynamic_shards": "1999", 2024-08-17T17:18:46.511 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_listing_results": "1000", 2024-08-17T17:18:46.511 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_notify_retries": "10", 2024-08-17T17:18:46.511 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_objs_per_shard": "100000", 2024-08-17T17:18:46.511 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_put_param_size": "1048576", 2024-08-17T17:18:46.511 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_put_size": "5368709120", 2024-08-17T17:18:46.512 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_slo_entries": "1000", 2024-08-17T17:18:46.512 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_md_log_max_shards": "64", 2024-08-17T17:18:46.512 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_md_notify_interval_msec": "200", 2024-08-17T17:18:46.512 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_meta_sync_poll_interval": "20", 2024-08-17T17:18:46.512 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_meta_sync_spawn_window": "20", 2024-08-17T17:18:46.512 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-08-17T17:18:46.512 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_mp_lock_max_time": "600", 2024-08-17T17:18:46.512 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-08-17T17:18:46.512 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_multipart_min_part_size": "5242880", 2024-08-17T17:18:46.512 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-08-17T17:18:46.512 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-08-17T17:18:46.512 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_fhcache_size": "2017", 2024-08-17T17:18:46.512 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-08-17T17:18:46.512 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-08-17T17:18:46.513 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_lru_lanes": "5", 2024-08-17T17:18:46.513 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_max_gc": "300", 2024-08-17T17:18:46.513 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-08-17T17:18:46.513 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_run_gc_threads": "false", 2024-08-17T17:18:46.513 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_run_lc_threads": "false", 2024-08-17T17:18:46.513 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_run_quota_threads": "false", 2024-08-17T17:18:46.513 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_run_sync_thread": "false", 2024-08-17T17:18:46.513 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-08-17T17:18:46.513 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-08-17T17:18:46.513 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_num_async_rados_threads": "32", 2024-08-17T17:18:46.513 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_num_control_oids": "8", 2024-08-17T17:18:46.513 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_numa_node": "-1", 2024-08-17T17:18:46.513 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_obj_stripe_size": "4194304", 2024-08-17T17:18:46.513 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-08-17T17:18:46.514 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_objexp_chunk_size": "100", 2024-08-17T17:18:46.514 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_objexp_gc_interval": "600", 2024-08-17T17:18:46.514 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_objexp_hints_num_shards": "127", 2024-08-17T17:18:46.514 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-08-17T17:18:46.514 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-08-17T17:18:46.514 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_op_thread_timeout": "600", 2024-08-17T17:18:46.514 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_op_tracing": "false", 2024-08-17T17:18:46.514 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_opa_token": "", 2024-08-17T17:18:46.514 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_opa_url": "", 2024-08-17T17:18:46.514 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_opa_verify_ssl": "true", 2024-08-17T17:18:46.514 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-08-17T17:18:46.514 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-08-17T17:18:46.514 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ops_log_rados": "false", 2024-08-17T17:18:46.514 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ops_log_socket_path": "", 2024-08-17T17:18:46.515 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-08-17T17:18:46.515 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-08-17T17:18:46.515 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-08-17T17:18:46.515 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_period_push_interval": "2.000000", 2024-08-17T17:18:46.515 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_period_push_interval_max": "30.000000", 2024-08-17T17:18:46.515 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_period_root_pool": ".rgw.root", 2024-08-17T17:18:46.515 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-08-17T17:18:46.515 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-08-17T17:18:46.515 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_posix_cache_lanes": "3", 2024-08-17T17:18:46.515 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-08-17T17:18:46.515 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_posix_cache_max_buckets": "100", 2024-08-17T17:18:46.515 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_posix_cache_partitions": "3", 2024-08-17T17:18:46.515 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-08-17T17:18:46.515 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_print_continue": "true", 2024-08-17T17:18:46.515 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_print_prohibited_content_length": "false", 2024-08-17T17:18:46.515 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-08-17T17:18:46.516 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-08-17T17:18:46.516 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-08-17T17:18:46.516 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-08-17T17:18:46.516 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_rados_tracing": "false", 2024-08-17T17:18:46.516 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_realm": "", 2024-08-17T17:18:46.516 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_realm_id": "", 2024-08-17T17:18:46.516 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-08-17T17:18:46.516 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_region": "", 2024-08-17T17:18:46.516 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_region_root_pool": ".rgw.root", 2024-08-17T17:18:46.516 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_relaxed_region_enforcement": "false", 2024-08-17T17:18:46.516 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-08-17T17:18:46.516 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-08-17T17:18:46.516 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_request_uri": "", 2024-08-17T17:18:46.516 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_reshard_batch_size": "64", 2024-08-17T17:18:46.516 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-08-17T17:18:46.517 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_reshard_debug_interval": "-1", 2024-08-17T17:18:46.517 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_reshard_max_aio": "128", 2024-08-17T17:18:46.517 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_reshard_num_logs": "16", 2024-08-17T17:18:46.517 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_reshard_thread_interval": "600", 2024-08-17T17:18:46.517 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_resolve_cname": "false", 2024-08-17T17:18:46.517 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_rest_getusage_op_compat": "false", 2024-08-17T17:18:46.517 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_run_sync_thread": "true", 2024-08-17T17:18:46.517 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-08-17T17:18:46.517 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-08-17T17:18:46.517 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_auth_use_keystone": "false", 2024-08-17T17:18:46.517 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_auth_use_ldap": "false", 2024-08-17T17:18:46.517 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_auth_use_rados": "true", 2024-08-17T17:18:46.517 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_auth_use_sts": "false", 2024-08-17T17:18:46.518 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-08-17T17:18:46.518 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_success_create_obj_status": "0", 2024-08-17T17:18:46.518 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-08-17T17:18:46.518 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_scheduler_type": "throttler", 2024-08-17T17:18:46.518 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_script_uri": "", 2024-08-17T17:18:46.518 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_service_provider_name": "", 2024-08-17T17:18:46.518 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-08-17T17:18:46.518 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sts_client_id": "", 2024-08-17T17:18:46.518 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sts_client_secret": "", 2024-08-17T17:18:46.518 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sts_entry": "sts", 2024-08-17T17:18:46.518 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sts_key": "sts", 2024-08-17T17:18:46.518 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sts_max_session_duration": "43200", 2024-08-17T17:18:46.518 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sts_min_session_duration": "900", 2024-08-17T17:18:46.518 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sts_token_introspection_url": "", 2024-08-17T17:18:46.519 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_account_in_url": "false", 2024-08-17T17:18:46.519 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_auth_entry": "auth", 2024-08-17T17:18:46.519 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_auth_url": "", 2024-08-17T17:18:46.519 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_custom_header": "", 2024-08-17T17:18:46.519 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_enforce_content_length": "false", 2024-08-17T17:18:46.519 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_need_stats": "true", 2024-08-17T17:18:46.519 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_tenant_name": "", 2024-08-17T17:18:46.519 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_token_expiration": "86400", 2024-08-17T17:18:46.519 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_url": "", 2024-08-17T17:18:46.519 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_url_prefix": "swift", 2024-08-17T17:18:46.519 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_versioning_enabled": "false", 2024-08-17T17:18:46.519 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-08-17T17:18:46.519 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-08-17T17:18:46.520 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_lease_period": "120", 2024-08-17T17:18:46.520 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-08-17T17:18:46.520 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_log_trim_interval": "1200", 2024-08-17T17:18:46.520 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-08-17T17:18:46.520 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-08-17T17:18:46.520 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-08-17T17:18:46.520 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_obj_etag_verify": "false", 2024-08-17T17:18:46.520 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_trace_history_size": "4096", 2024-08-17T17:18:46.520 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-08-17T17:18:46.520 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-08-17T17:18:46.520 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_thread_pool_size": "512", 2024-08-17T17:18:46.520 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_topic_persistency_max_retries": "0", 2024-08-17T17:18:46.520 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-08-17T17:18:46.521 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-08-17T17:18:46.521 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_topic_require_publish_policy": "false", 2024-08-17T17:18:46.521 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_comment": "", 2024-08-17T17:18:46.521 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_createby": "", 2024-08-17T17:18:46.521 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_encoding": "", 2024-08-17T17:18:46.521 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_flag": "false", 2024-08-17T17:18:46.521 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_max_size": "5368709120", 2024-08-17T17:18:46.521 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_origin": "", 2024-08-17T17:18:46.521 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_sha_unit": "524288", 2024-08-17T17:18:46.521 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_tracker": "", 2024-08-17T17:18:46.521 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_trust_forwarded_https": "false", 2024-08-17T17:18:46.521 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-08-17T17:18:46.522 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_usage_log_tick_interval": "30", 2024-08-17T17:18:46.522 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_usage_max_shards": "32", 2024-08-17T17:18:46.522 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_usage_max_user_shards": "1", 2024-08-17T17:18:46.522 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_use_opa_authz": "false", 2024-08-17T17:18:46.522 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_counters_cache": "false", 2024-08-17T17:18:46.522 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_counters_cache_size": "10000", 2024-08-17T17:18:46.522 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-08-17T17:18:46.522 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_default_quota_max_size": "-1", 2024-08-17T17:18:46.522 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_max_buckets": "1000", 2024-08-17T17:18:46.522 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_policies_max_num": "100", 2024-08-17T17:18:46.522 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-08-17T17:18:46.522 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-08-17T17:18:46.522 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_quota_sync_interval": "86400", 2024-08-17T17:18:46.522 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-08-17T17:18:46.522 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_unique_email": "true", 2024-08-17T17:18:46.523 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_verify_ssl": "true", 2024-08-17T17:18:46.523 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_website_routing_rules_max_num": "50", 2024-08-17T17:18:46.523 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_zone": "", 2024-08-17T17:18:46.523 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_zone_id": "", 2024-08-17T17:18:46.523 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-08-17T17:18:46.523 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_zonegroup": "", 2024-08-17T17:18:46.523 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_zonegroup_id": "", 2024-08-17T17:18:46.523 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-08-17T17:18:46.523 INFO:teuthology.orchestra.run.smithi003.stdout: "rgwlc_auto_session_clear": "true", 2024-08-17T17:18:46.523 INFO:teuthology.orchestra.run.smithi003.stdout: "rgwlc_skip_bucket_step": "false", 2024-08-17T17:18:46.523 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_block_size": "4096", 2024-08-17T17:18:46.523 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-08-17T17:18:46.523 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-08-17T17:18:46.523 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-08-17T17:18:46.523 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-08-17T17:18:46.524 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cache_shard_bits": "4", 2024-08-17T17:18:46.524 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cache_size": "536870912", 2024-08-17T17:18:46.524 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cache_type": "binned_lru", 2024-08-17T17:18:46.524 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-08-17T17:18:46.524 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-08-17T17:18:46.524 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-08-17T17:18:46.524 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_collect_compaction_stats": "false", 2024-08-17T17:18:46.524 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_collect_extended_stats": "false", 2024-08-17T17:18:46.524 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_collect_memory_stats": "false", 2024-08-17T17:18:46.524 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-08-17T17:18:46.524 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_index_type": "binary_search", 2024-08-17T17:18:46.524 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_log_to_ceph_log": "true", 2024-08-17T17:18:46.524 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_metadata_block_size": "4096", 2024-08-17T17:18:46.524 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_partition_filters": "false", 2024-08-17T17:18:46.524 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_perf": "false", 2024-08-17T17:18:46.525 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-08-17T17:18:46.525 INFO:teuthology.orchestra.run.smithi003.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-08-17T17:18:46.525 INFO:teuthology.orchestra.run.smithi003.stdout: "rotating_keys_renewal_timeout": "10", 2024-08-17T17:18:46.525 INFO:teuthology.orchestra.run.smithi003.stdout: "run_dir": "/var/run/ceph", 2024-08-17T17:18:46.525 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_block_create": "true", 2024-08-17T17:18:46.525 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_cache_lru_size": "67108864", 2024-08-17T17:18:46.525 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_cbjournal_size": "5368709120", 2024-08-17T17:18:46.525 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_data_delta_based_overwrite": "0", 2024-08-17T17:18:46.525 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_default_max_object_size": "16777216", 2024-08-17T17:18:46.525 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-08-17T17:18:46.525 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_device_size": "53687091200", 2024-08-17T17:18:46.525 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-08-17T17:18:46.525 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_full_integrity_check": "false", 2024-08-17T17:18:46.525 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_journal_batch_capacity": "16", 2024-08-17T17:18:46.525 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-08-17T17:18:46.526 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-08-17T17:18:46.526 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_journal_iodepth_limit": "5", 2024-08-17T17:18:46.526 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_main_device_type": "SSD", 2024-08-17T17:18:46.526 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_max_concurrent_transactions": "8", 2024-08-17T17:18:46.526 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_max_data_allocation_size": "32768", 2024-08-17T17:18:46.526 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-08-17T17:18:46.526 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-08-17T17:18:46.526 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-08-17T17:18:46.526 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-08-17T17:18:46.526 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_segment_size": "67108864", 2024-08-17T17:18:46.526 INFO:teuthology.orchestra.run.smithi003.stdout: "set_keepcaps": "false", 2024-08-17T17:18:46.526 INFO:teuthology.orchestra.run.smithi003.stdout: "setgroup": "ceph", 2024-08-17T17:18:46.526 INFO:teuthology.orchestra.run.smithi003.stdout: "setuser": "ceph", 2024-08-17T17:18:46.526 INFO:teuthology.orchestra.run.smithi003.stdout: "setuser_match_path": "", 2024-08-17T17:18:46.526 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_misplaced_ratio": "0.050000", 2024-08-17T17:18:46.527 INFO:teuthology.orchestra.run.smithi003.stdout: "thp": "false", 2024-08-17T17:18:46.527 INFO:teuthology.orchestra.run.smithi003.stdout: "threadpool_default_timeout": "60", 2024-08-17T17:18:46.527 INFO:teuthology.orchestra.run.smithi003.stdout: "threadpool_empty_queue_max_wait": "2", 2024-08-17T17:18:46.527 INFO:teuthology.orchestra.run.smithi003.stdout: "throttler_perf_counter": "true", 2024-08-17T17:18:46.527 INFO:teuthology.orchestra.run.smithi003.stdout: "tmp_dir": "/tmp", 2024-08-17T17:18:46.527 INFO:teuthology.orchestra.run.smithi003.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX" 2024-08-17T17:18:46.527 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:18:47.380 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-08-17T17:18:47.660 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:18:47.667 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:47 smithi003 ceph-mon[103344]: pgmap v207: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:47.667 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:47 smithi003 ceph-mon[100615]: pgmap v207: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:47.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:47 smithi053 ceph-mon[89963]: pgmap v207: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:48.638 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:48 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:18:48.638 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:48 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:18:48.798 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:48 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:18:49.132 INFO:teuthology.orchestra.run.smithi003.stdout:mon.a: {} 2024-08-17T17:18:49.133 INFO:teuthology.orchestra.run.smithi003.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-17T17:18:49.133 INFO:teuthology.orchestra.run.smithi003.stdout:mon.b: {} 2024-08-17T17:18:49.133 INFO:teuthology.orchestra.run.smithi003.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-17T17:18:49.143 INFO:teuthology.orchestra.run.smithi003.stdout:mon.c: {} 2024-08-17T17:18:49.143 INFO:teuthology.orchestra.run.smithi003.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-17T17:18:49.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:49 smithi003 ceph-mon[100615]: pgmap v208: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:49.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:49 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1244451564' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-17T17:18:49.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:49 smithi003 ceph-mon[103344]: pgmap v208: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:49.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:49 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1244451564' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-17T17:18:49.776 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:49 smithi053 ceph-mon[89963]: pgmap v208: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:49.777 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:49 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1244451564' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-17T17:18:49.829 INFO:tasks.thrashosds.thrasher:Unable to test ceph-objectstore-tool, not available on all OSD nodes 2024-08-17T17:18:49.830 INFO:teuthology.task.sequential:In sequential, running task full_sequential... 2024-08-17T17:18:49.831 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-08-17T17:18:49.832 INFO:tasks.radosbench:Beginning radosbench... 2024-08-17T17:18:49.832 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-08-17T17:18:49.833 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-08-17T17:18:49.833 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-08-17T17:18:49.833 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-17T17:18:49.833 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:18:49.833 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-08-17T17:18:49.834 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight-by-utilization 110 3.0 2024-08-17T17:18:49.834 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-08-17T17:18:49.834 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:18:49.835 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-08-17T17:18:49.835 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:18:49.835 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-08-17T17:18:49.836 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-17T17:18:49.836 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-08-17T17:18:49.836 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:18:49.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:49.861 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:49.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:18:49.962 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:18:49.984 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:49.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:49.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:18:50.086 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:18:50.107 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:50.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:50.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:18:50.209 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:18:50.232 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:50.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:50.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:18:50.334 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:18:50.357 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:50.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:50.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:18:50.459 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:18:50.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:18:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:18:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:18:50.482 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:50.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:50.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:18:50.583 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:18:50.605 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:50.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:50.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:18:50.707 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:18:50.738 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:50.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:50.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:18:50.761 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:18:50.764 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:18:50.765 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:18:50.828 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:18:50.840 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:18:50.863 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:50.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:50.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:18:50.965 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:18:50.988 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:50.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:50.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:18:51.063 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:18:51.089 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:18:51.112 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:51.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:51.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:18:51.213 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:18:51.236 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:51.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:51.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:18:51.338 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:18:51.361 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:51.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:51.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:18:51.463 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:18:51.485 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:51.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:51.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:18:51.587 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:18:51.609 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:51.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:51.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:18:51.713 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:18:51.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:51 smithi053 ceph-mon[89963]: pgmap v209: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:51.736 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:51.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:51.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:18:51.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:51 smithi003 ceph-mon[100615]: pgmap v209: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:51.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:51 smithi003 ceph-mon[103344]: pgmap v209: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:51.839 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:18:51.861 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:51.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:51.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:18:51.963 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:18:51.987 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:51.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:51.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:18:52.089 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:18:52.110 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:52.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:52.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:18:52.212 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:18:52.235 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:52.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:52.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:18:52.337 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:18:52.358 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:52.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:52.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:18:52.460 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:18:52.484 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:52.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:52.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:18:52.585 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:18:52.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:52.607 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:52.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:18:52.708 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:18:52.730 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:52.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:52.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:18:52.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:52 smithi003 ceph-mon[100615]: pgmap v210: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:52.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:52 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:18:52.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:52 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:18:52.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:52 smithi003 ceph-mon[103344]: pgmap v210: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:52.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:52 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:18:52.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:52 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:18:52.832 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:18:52.854 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:52.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:52.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:18:52.956 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:18:52.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:52 smithi053 ceph-mon[89963]: pgmap v210: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:52.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:52 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:18:52.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:52 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:18:52.978 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:52.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:52.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:18:53.080 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:18:53.102 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:53.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:53.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:18:53.204 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:18:53.226 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:53.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:53.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:18:53.328 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:18:53.351 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:53.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:53.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:18:53.453 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:18:53.476 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:53.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:53.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:18:53.578 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:18:53.601 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:53.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:53.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:18:53.703 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:18:53.726 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:53.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:53.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:18:53.828 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:18:53.851 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:53.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:53.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:18:53.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:18:53.940 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:18:53.953 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:18:53.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:18:53.978 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:53.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:53.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:18:54.080 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:18:54.103 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:54.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:54.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:18:54.205 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:18:54.231 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:54.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:54.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:18:54.333 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:18:54.360 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:54.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:54.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:18:54.461 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:18:54.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:54.485 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:54.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:18:54.586 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:18:54.613 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:54.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:54.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:18:54.715 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:18:54.741 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:54.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:54.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:18:54.815 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:54 smithi003 ceph-mon[103344]: pgmap v211: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:54.815 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:54 smithi003 ceph-mon[100615]: pgmap v211: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:54.843 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:18:54.869 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:54.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:54.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:18:54.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:54 smithi053 ceph-mon[89963]: pgmap v211: 73 pgs: 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:54.971 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:18:54.998 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:54.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:54.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:18:55.057 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:18:55.057 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-08-17T17:18:55.058 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-08-17T17:18:55.058 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:18:55.099 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:18:55.125 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:55.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:55.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:18:55.226 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:18:55.250 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:55.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:55.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:18:55.352 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:18:55.378 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:55.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:55.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:18:55.479 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:18:55.503 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:55.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:55.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:18:55.552 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_0' created 2024-08-17T17:18:55.604 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:18:55.630 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:55.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:55.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:18:55.638 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:55 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1006635694' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-08-17T17:18:55.638 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:55 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1006635694' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-08-17T17:18:55.664 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-17T17:18:55.732 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:18:55.758 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:55.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:55.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:18:55.860 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:18:55.882 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:55.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:55.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:18:55.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:55 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1006635694' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-08-17T17:18:55.983 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:18:56.010 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:56.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:56.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:18:56.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:18:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:18:55] "GET /metrics HTTP/1.1" 200 33940 "" "Prometheus/2.51.0" 2024-08-17T17:18:56.111 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:18:56.133 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:56.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:56.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:18:56.235 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:18:56.258 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:56.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:56.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:18:56.359 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:18:56.386 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:56.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:56.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:18:56.402 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-08-17T17:18:56.487 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:18:56.513 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:56.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:56.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:18:56.615 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:18:56.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:56.638 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:56.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:18:56.677 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:56 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1006635694' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-08-17T17:18:56.677 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:56 smithi003 ceph-mon[103344]: osdmap e111: 8 total, 8 up, 8 in 2024-08-17T17:18:56.677 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:56 smithi003 ceph-mon[103344]: pgmap v213: 89 pgs: 16 unknown, 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:56.677 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:56 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1006635694' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-08-17T17:18:56.678 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:56 smithi003 ceph-mon[100615]: osdmap e111: 8 total, 8 up, 8 in 2024-08-17T17:18:56.678 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:56 smithi003 ceph-mon[100615]: pgmap v213: 89 pgs: 16 unknown, 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:56.738 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:18:56.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:56.767 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:56.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:18:56.868 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:18:56.890 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:56.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:56.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:18:56.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:56 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1006635694' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-08-17T17:18:56.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:56 smithi053 ceph-mon[89963]: osdmap e111: 8 total, 8 up, 8 in 2024-08-17T17:18:56.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:56 smithi053 ceph-mon[89963]: pgmap v213: 89 pgs: 16 unknown, 73 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:56.992 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:18:57.018 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:57.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:57.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:18:57.120 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:18:57.141 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:57.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:57.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:18:57.243 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:18:57.265 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:57.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:57.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:18:57.367 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:18:57.388 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:57.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:57.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:18:57.490 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:18:57.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:57.512 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:57.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:18:57.613 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:18:57.635 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:57.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:57.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:18:57.737 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:18:57.759 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:57.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:57.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:18:57.812 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:57 smithi003 ceph-mon[103344]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:18:57.812 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:57 smithi003 ceph-mon[103344]: osdmap e112: 8 total, 8 up, 8 in 2024-08-17T17:18:57.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:57 smithi003 ceph-mon[100615]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:18:57.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:57 smithi003 ceph-mon[100615]: osdmap e112: 8 total, 8 up, 8 in 2024-08-17T17:18:57.861 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:18:57.882 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:57.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:57.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:18:57.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:57 smithi053 ceph-mon[89963]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:18:57.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:57 smithi053 ceph-mon[89963]: osdmap e112: 8 total, 8 up, 8 in 2024-08-17T17:18:57.984 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:18:58.006 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:58.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:58.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:18:58.108 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:18:58.135 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:58.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:58.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:18:58.237 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:18:58.263 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:58.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:58.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:18:58.364 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:18:58.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:58.390 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:58.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:18:58.491 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:18:58.518 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:58.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:58.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:18:58.569 INFO:teuthology.orchestra.run.smithi003.stdout:moved 60 / 267 (22.4719%) 2024-08-17T17:18:58.569 INFO:teuthology.orchestra.run.smithi003.stdout:avg 33.375 2024-08-17T17:18:58.569 INFO:teuthology.orchestra.run.smithi003.stdout:stddev 4.49826 -> 11.8842 (expected baseline 5.40399) 2024-08-17T17:18:58.569 INFO:teuthology.orchestra.run.smithi003.stdout:min osd.2 with 25 -> 34 pgs (0.749064 -> 1.01873 * mean) 2024-08-17T17:18:58.569 INFO:teuthology.orchestra.run.smithi003.stdout:max osd.4 with 41 -> 29 pgs (1.22846 -> 0.868914 * mean) 2024-08-17T17:18:58.569 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:18:58.569 INFO:teuthology.orchestra.run.smithi003.stdout:oload 110 2024-08-17T17:18:58.569 INFO:teuthology.orchestra.run.smithi003.stdout:max_change 3 2024-08-17T17:18:58.569 INFO:teuthology.orchestra.run.smithi003.stdout:max_change_osds 4 2024-08-17T17:18:58.570 INFO:teuthology.orchestra.run.smithi003.stdout:average_utilization 0.0018 2024-08-17T17:18:58.570 INFO:teuthology.orchestra.run.smithi003.stdout:overload_utilization 0.0020 2024-08-17T17:18:58.570 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7 weight 1.0000 -> 0.5465 2024-08-17T17:18:58.570 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6 weight 1.0000 -> 0.5465 2024-08-17T17:18:58.570 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4 weight 1.0000 -> 0.5549 2024-08-17T17:18:58.570 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5 weight 1.0000 -> 0.5627 2024-08-17T17:18:58.619 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:18:58.644 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:58.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:58.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:18:58.745 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:18:58.769 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:58.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:58.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:18:58.852 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:58 smithi003 ceph-mon[103344]: osdmap e113: 8 total, 8 up, 8 in 2024-08-17T17:18:58.852 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:58 smithi003 ceph-mon[103344]: pgmap v216: 89 pgs: 14 unknown, 75 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:58.852 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:58 smithi003 ceph-mon[103344]: from='client.45104 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:18:58.852 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:58 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'4':'36368','5':'36875','6':'35815','7':'35813'}"} : dispatch 2024-08-17T17:18:58.852 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:58 smithi003 ceph-mon[100615]: osdmap e113: 8 total, 8 up, 8 in 2024-08-17T17:18:58.852 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:58 smithi003 ceph-mon[100615]: pgmap v216: 89 pgs: 14 unknown, 75 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:58.852 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:58 smithi003 ceph-mon[100615]: from='client.45104 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:18:58.852 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:58 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'4':'36368','5':'36875','6':'35815','7':'35813'}"} : dispatch 2024-08-17T17:18:58.871 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:18:58.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:58.895 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:58.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:18:58.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:58 smithi053 ceph-mon[89963]: osdmap e113: 8 total, 8 up, 8 in 2024-08-17T17:18:58.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:58 smithi053 ceph-mon[89963]: pgmap v216: 89 pgs: 14 unknown, 75 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-08-17T17:18:58.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:58 smithi053 ceph-mon[89963]: from='client.45104 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:18:58.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:58 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'4':'36368','5':'36875','6':'35815','7':'35813'}"} : dispatch 2024-08-17T17:18:58.996 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:18:59.021 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:59.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:59.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:18:59.123 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:18:59.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:59.151 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:59.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:18:59.252 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:18:59.278 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:59.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:59.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:18:59.380 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:18:59.394 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-08-17T17:18:59.418 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:59.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:59.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:18:59.519 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:18:59.543 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:59.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:59.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:18:59.644 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:18:59.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:59.667 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:59.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:18:59.768 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:18:59.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:59 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'4':'36368','5':'36875','6':'35815','7':'35813'}"}]': finished 2024-08-17T17:18:59.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:59 smithi003 ceph-mon[100615]: osdmap e114: 8 total, 8 up, 8 in 2024-08-17T17:18:59.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:59 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/203624822' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:18:59.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:59 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:18:59.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:59 smithi003 ceph-mon[100615]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:18:59.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:59 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:18:59.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:18:59 smithi003 ceph-mon[100615]: osdmap e115: 8 total, 8 up, 8 in 2024-08-17T17:18:59.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:59 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'4':'36368','5':'36875','6':'35815','7':'35813'}"}]': finished 2024-08-17T17:18:59.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:59 smithi003 ceph-mon[103344]: osdmap e114: 8 total, 8 up, 8 in 2024-08-17T17:18:59.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:59 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/203624822' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:18:59.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:59 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:18:59.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:59 smithi003 ceph-mon[103344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:18:59.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:59 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:18:59.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:18:59 smithi003 ceph-mon[103344]: osdmap e115: 8 total, 8 up, 8 in 2024-08-17T17:18:59.794 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:18:59.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:59.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:18:59.896 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:18:59.919 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:18:59.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:18:59.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:18:59.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:59 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'4':'36368','5':'36875','6':'35815','7':'35813'}"}]': finished 2024-08-17T17:18:59.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:59 smithi053 ceph-mon[89963]: osdmap e114: 8 total, 8 up, 8 in 2024-08-17T17:18:59.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:59 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/203624822' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:18:59.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:59 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:18:59.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:59 smithi053 ceph-mon[89963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:18:59.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:59 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:18:59.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:18:59 smithi053 ceph-mon[89963]: osdmap e115: 8 total, 8 up, 8 in 2024-08-17T17:19:00.020 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:00.042 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:00.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:00.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:00.144 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:00.166 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:00.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:00.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:00.268 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:00.289 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:00.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:00.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:00.374 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-08-17T17:19:00.374 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd test-reweight-by-utilization 110 1.0 2024-08-17T17:19:00.392 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:00.420 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:00.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:00.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:00.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:19:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:19:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:19:00.522 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:00.545 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:00.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:00.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:00.646 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:00.663 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:19:00.663 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:19:00.664 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:19:00.664 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:19:00.666 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:19:00.666 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:19:00.673 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:00.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:00.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:00.683 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:19:00.683 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:19:00.701 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:19:00.701 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:19:00.723 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:19:00.724 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:19:00.740 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:19:00.741 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:19:00.756 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:19:00.757 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:19:00.775 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:00.800 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:00.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:00.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:00.902 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:00.925 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:00.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:00.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:01.027 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:01.050 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:01.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:01.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:01.152 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:01.176 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:01.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:01.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:01.278 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:01.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:01.302 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:01.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:01.403 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:01.429 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:01.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:01.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:01.531 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:01.558 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:01.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:01.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:01.659 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:01.686 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:01.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:01.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:01.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:01 smithi053 ceph-mon[89963]: pgmap v219: 89 pgs: 20 peering, 2 unknown, 67 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-08-17T17:19:01.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:01 smithi053 ceph-mon[89963]: osdmap e116: 8 total, 8 up, 8 in 2024-08-17T17:19:01.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:01 smithi053 ceph-mon[89963]: Health check failed: Reduced data availability: 3 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-08-17T17:19:01.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:01 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1682221600' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:19:01.737 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:01 smithi003 ceph-mon[103344]: pgmap v219: 89 pgs: 20 peering, 2 unknown, 67 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-08-17T17:19:01.737 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:01 smithi003 ceph-mon[103344]: osdmap e116: 8 total, 8 up, 8 in 2024-08-17T17:19:01.737 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:01 smithi003 ceph-mon[103344]: Health check failed: Reduced data availability: 3 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-08-17T17:19:01.737 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:01 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1682221600' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:19:01.737 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:01 smithi003 ceph-mon[100615]: pgmap v219: 89 pgs: 20 peering, 2 unknown, 67 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-08-17T17:19:01.737 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:01 smithi003 ceph-mon[100615]: osdmap e116: 8 total, 8 up, 8 in 2024-08-17T17:19:01.738 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:01 smithi003 ceph-mon[100615]: Health check failed: Reduced data availability: 3 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-08-17T17:19:01.738 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:01 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1682221600' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:19:01.788 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:01.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:01.811 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:01.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:01.912 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:01.939 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:01.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:01.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:02.041 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:02.063 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:02.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:02.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:02.166 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:02.192 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:02.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:02.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:02.294 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:02.316 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:02.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:02.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:02.417 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:02.439 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:02.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:02.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:02.458 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:19:02.459 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:19:02.506 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:19:02.541 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:02.563 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:02.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:02.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:02.665 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:02.691 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:02.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:02.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:02.697 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:02 smithi003 ceph-mon[103344]: osdmap e117: 8 total, 8 up, 8 in 2024-08-17T17:19:02.697 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:02 smithi003 ceph-mon[100615]: osdmap e117: 8 total, 8 up, 8 in 2024-08-17T17:19:02.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:02 smithi053 ceph-mon[89963]: osdmap e117: 8 total, 8 up, 8 in 2024-08-17T17:19:02.793 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:02.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:02.817 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:02.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:02.918 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:02.945 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:02.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:02.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:03.046 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:03.072 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:03.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:03.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:03.174 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:03.201 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:03.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:03.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:03.303 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:03.330 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:03.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:03.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:03.432 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:03.458 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:03.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:03.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:03.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:03.583 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:03.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:03.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:03.685 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:03.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:03.708 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:03.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:03.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:03 smithi053 ceph-mon[89963]: pgmap v222: 89 pgs: 20 peering, 1 unknown, 68 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 22 B/s, 0 objects/s recovering 2024-08-17T17:19:03.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:03 smithi053 ceph-mon[89963]: osdmap e118: 8 total, 8 up, 8 in 2024-08-17T17:19:03.763 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:19:03.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:03 smithi003 ceph-mon[103344]: pgmap v222: 89 pgs: 20 peering, 1 unknown, 68 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 22 B/s, 0 objects/s recovering 2024-08-17T17:19:03.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:03 smithi003 ceph-mon[103344]: osdmap e118: 8 total, 8 up, 8 in 2024-08-17T17:19:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:03 smithi003 ceph-mon[100615]: pgmap v222: 89 pgs: 20 peering, 1 unknown, 68 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 22 B/s, 0 objects/s recovering 2024-08-17T17:19:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:03 smithi003 ceph-mon[100615]: osdmap e118: 8 total, 8 up, 8 in 2024-08-17T17:19:03.809 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:03.831 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:03.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:03.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:03.933 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:03.955 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:03.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:03.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:04.057 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:04.079 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:04.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:04.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:04.181 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:04.203 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:04.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:04.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:04.305 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:04.328 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:04.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:04.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:04.430 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:04.453 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:04.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:04.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:04.489 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:19:04.555 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:04.576 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:04.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:04.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:04.678 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:04.700 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:04.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:04.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:04.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:04 smithi053 ceph-mon[89963]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 5 pgs peering) 2024-08-17T17:19:04.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:04 smithi003 ceph-mon[100615]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 5 pgs peering) 2024-08-17T17:19:04.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:04 smithi003 ceph-mon[103344]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 5 pgs peering) 2024-08-17T17:19:04.801 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:04.823 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:04.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:04.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:04.925 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:04.947 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:04.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:04.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:05.049 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:05.071 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:05.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:05.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:05.173 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:05.196 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:05.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:05.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:05.297 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:05.320 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:05.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:05.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:05.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:05.444 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:05.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:05.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:05.546 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:05.569 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:05.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:05.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:05.670 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:05.697 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:05.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:05.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:05.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:05 smithi053 ceph-mon[89963]: pgmap v224: 89 pgs: 1 peering, 88 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s, 2 keys/s, 9 objects/s recovering 2024-08-17T17:19:05.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:05 smithi003 ceph-mon[100615]: pgmap v224: 89 pgs: 1 peering, 88 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s, 2 keys/s, 9 objects/s recovering 2024-08-17T17:19:05.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:05 smithi003 ceph-mon[103344]: pgmap v224: 89 pgs: 1 peering, 88 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s, 2 keys/s, 9 objects/s recovering 2024-08-17T17:19:05.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:19:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:19:05] "GET /metrics HTTP/1.1" 200 33940 "" "Prometheus/2.51.0" 2024-08-17T17:19:05.800 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:05.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:05.826 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:05.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:05.927 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:05.954 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:05.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:05.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:06.056 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:06.078 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:06.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:06.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:06.180 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:06.201 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:06.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:06.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:06.303 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:06.329 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:06.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:06.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:06.431 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:06.454 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:06.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:06.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:06.556 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:06.583 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:06.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:06.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:06.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:06.703 INFO:teuthology.orchestra.run.smithi003.stdout:moved 40 / 267 (14.9813%) 2024-08-17T17:19:06.703 INFO:teuthology.orchestra.run.smithi003.stdout:avg 33.375 2024-08-17T17:19:06.704 INFO:teuthology.orchestra.run.smithi003.stdout:stddev 11.8842 -> 19.767 (expected baseline 5.40399) 2024-08-17T17:19:06.704 INFO:teuthology.orchestra.run.smithi003.stdout:min osd.7 with 16 -> 10 pgs (0.479401 -> 0.299625 * mean) 2024-08-17T17:19:06.704 INFO:teuthology.orchestra.run.smithi003.stdout:max osd.1 with 49 -> 58 pgs (1.46816 -> 1.73783 * mean) 2024-08-17T17:19:06.704 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:19:06.704 INFO:teuthology.orchestra.run.smithi003.stdout:oload 110 2024-08-17T17:19:06.704 INFO:teuthology.orchestra.run.smithi003.stdout:max_change 1 2024-08-17T17:19:06.704 INFO:teuthology.orchestra.run.smithi003.stdout:max_change_osds 4 2024-08-17T17:19:06.704 INFO:teuthology.orchestra.run.smithi003.stdout:average_utilization 0.0018 2024-08-17T17:19:06.704 INFO:teuthology.orchestra.run.smithi003.stdout:overload_utilization 0.0020 2024-08-17T17:19:06.704 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6 weight 0.5465 -> 0.2997 2024-08-17T17:19:06.704 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7 weight 0.5465 -> 0.3042 2024-08-17T17:19:06.704 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5 weight 0.5627 -> 0.3133 2024-08-17T17:19:06.704 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4 weight 0.5549 -> 0.3090 2024-08-17T17:19:06.705 INFO:teuthology.orchestra.run.smithi003.stderr:no change 2024-08-17T17:19:06.708 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:06.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:06.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:06.810 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:06.836 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:06.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:06.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:06.938 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:06.965 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:06.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:06.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:07.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:19:06 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:19:06.702+0000 7f288dc05640 -1 mgr.server reply reply (4) Interrupted system call no change 2024-08-17T17:19:07.066 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:07.093 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:07.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:07.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:07.194 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:07.221 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:07.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:07.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:07.323 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:07.345 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:07.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:07.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:07.393 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:19:07.393 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:19:07.447 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:07.470 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:07.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:07.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:07.572 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:07.599 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:07.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:07.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:07.700 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:07.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:07 smithi053 ceph-mon[89963]: pgmap v225: 89 pgs: 1 peering, 88 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 779 KiB/s, 1 keys/s, 6 objects/s recovering 2024-08-17T17:19:07.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:07.727 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:07.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:07.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:07 smithi003 ceph-mon[100615]: pgmap v225: 89 pgs: 1 peering, 88 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 779 KiB/s, 1 keys/s, 6 objects/s recovering 2024-08-17T17:19:07.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:07 smithi003 ceph-mon[103344]: pgmap v225: 89 pgs: 1 peering, 88 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 779 KiB/s, 1 keys/s, 6 objects/s recovering 2024-08-17T17:19:07.828 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:07.855 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:07.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:07.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:07.857 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-08-17T17:19:07.857 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight-by-utilization 110 0.05 2024-08-17T17:19:07.957 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:07.979 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:07.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:07.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:08.081 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:08.107 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:08.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:08.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:08.209 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:08.231 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:08.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:08.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:08.333 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:08.355 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:08.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:08.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:08.457 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:08.480 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:08.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:08.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:08.582 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:08.604 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:08.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:08.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:08.706 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:08.731 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:08 smithi003 ceph-mon[100615]: from='client.45122 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:19:08.732 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:19:08.732 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:19:08.732 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:08 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/561043933' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:19:08.732 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:08 smithi003 ceph-mon[103344]: from='client.45122 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:19:08.732 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:19:08.732 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:19:08.732 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:08 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/561043933' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:19:08.733 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:08.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:08.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:08.835 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:08.861 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:08.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:08.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:08.962 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:08.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:08 smithi053 ceph-mon[89963]: from='client.45122 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:19:08.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:19:08.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:19:08.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:08 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/561043933' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:19:08.989 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:08.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:08.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:09.015 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-17T17:19:09.074 INFO:teuthology.orchestra.run.smithi003.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-08-17T17:19:09.076 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:19:09.077 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:19:09.090 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:09.117 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:09.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:09.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:09.219 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:09.245 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:09.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:09.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:09.347 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:09.370 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:09.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:09.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:09.472 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:09.499 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:09.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:09.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:09.600 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:09.623 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:09.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:09.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:09.725 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:09.729 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:09 smithi003 ceph-mon[103344]: pgmap v226: 89 pgs: 89 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 640 KiB/s, 1 keys/s, 5 objects/s recovering 2024-08-17T17:19:09.729 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:09 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/561043933' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T17:19:09.729 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:09 smithi003 ceph-mon[103344]: osdmap e119: 8 total, 8 up, 8 in 2024-08-17T17:19:09.729 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:09 smithi003 ceph-mon[100615]: pgmap v226: 89 pgs: 89 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 640 KiB/s, 1 keys/s, 5 objects/s recovering 2024-08-17T17:19:09.729 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:09 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/561043933' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T17:19:09.729 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:09 smithi003 ceph-mon[100615]: osdmap e119: 8 total, 8 up, 8 in 2024-08-17T17:19:09.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:09.755 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:09.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:09.858 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:09.880 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:09.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:09.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:09.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:09 smithi053 ceph-mon[89963]: pgmap v226: 89 pgs: 89 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 640 KiB/s, 1 keys/s, 5 objects/s recovering 2024-08-17T17:19:09.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:09 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/561043933' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T17:19:09.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:09 smithi053 ceph-mon[89963]: osdmap e119: 8 total, 8 up, 8 in 2024-08-17T17:19:09.982 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:10.008 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:10.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:10.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:10.110 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:10.136 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:10.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:10.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:10.238 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:10.265 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:10.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:10.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:10.367 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:10.393 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:10.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:10.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:10.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:19:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:19:10.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:19:10.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:10.518 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:10.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:10.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:10.620 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:10.646 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:10.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:10.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:10.702 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:10 smithi003 ceph-mon[103344]: pgmap v228: 89 pgs: 89 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 585 KiB/s, 1 keys/s, 4 objects/s recovering 2024-08-17T17:19:10.702 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:10 smithi003 ceph-mon[103344]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:19:10.703 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:10 smithi003 ceph-mon[100615]: pgmap v228: 89 pgs: 89 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 585 KiB/s, 1 keys/s, 4 objects/s recovering 2024-08-17T17:19:10.703 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:10 smithi003 ceph-mon[100615]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:19:10.748 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:10.771 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:10.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:10.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:10.873 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:10.895 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:10.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:10.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:10.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:10 smithi053 ceph-mon[89963]: pgmap v228: 89 pgs: 89 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 585 KiB/s, 1 keys/s, 4 objects/s recovering 2024-08-17T17:19:10.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:10 smithi053 ceph-mon[89963]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:19:10.997 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:11.024 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:11.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:11.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:11.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:11.151 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:11.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:11.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:11.253 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:11.280 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:11.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:11.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:11.382 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:11.409 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:11.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:11.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:11.512 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:11.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:11.537 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:11.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:11.638 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:11.665 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:11.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:11.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:11.730 DEBUG:teuthology.orchestra.run.smithi003:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-08-17T17:19:11.733 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-08-17T17:19:11.733 INFO:teuthology.orchestra.run:waiting for 3000 2024-08-17T17:19:11.766 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:11.782 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:19:11.784 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:hints = 1 2024-08-17T17:19:11.784 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-08-17T17:19:11.784 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object prefix: benchmark_data_smithi003_140793 2024-08-17T17:19:11.785 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:19:11.785 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 0 0 0 0 0 0 - 0 2024-08-17T17:19:11.796 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:11.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:11.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:11.898 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:11.943 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:11.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:11.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:12.045 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:12.074 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:12.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:12.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:12.176 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:12.205 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:12.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:12.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:12.307 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:12.353 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:12.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:12.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:12.455 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:12.481 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:12.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:12.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:12.583 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:12.624 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:12.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:12.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:12.726 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:12.756 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:12.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:12.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:12.785 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 1 16 3210 3194 199.614 199.625 0.00390263 0.00499765 2024-08-17T17:19:12.858 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:12.896 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:12.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:12.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:12.998 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:13.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:13.031 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:13.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:13.132 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:13.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:13.179 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:13.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:13.193 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:13 smithi003 ceph-mon[103344]: pgmap v229: 89 pgs: 89 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 441 KiB/s, 1 keys/s, 3 objects/s recovering 2024-08-17T17:19:13.193 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:13 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:19:13.193 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:13 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:19:13.193 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:13 smithi003 ceph-mon[100615]: pgmap v229: 89 pgs: 89 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 441 KiB/s, 1 keys/s, 3 objects/s recovering 2024-08-17T17:19:13.193 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:13 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:19:13.193 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:13 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:19:13.281 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:13.331 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:13.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:13.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:13.430 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:19:13.431 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:19:13.431 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:19:13.431 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:19:13.431 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:19:13.431 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:19:13.431 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:19:13.431 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:19:13.432 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:19:13.432 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:19:13.433 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:13.462 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:19:13.463 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:19:13.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:13 smithi053 ceph-mon[89963]: pgmap v229: 89 pgs: 89 active+clean; 4.6 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 441 KiB/s, 1 keys/s, 3 objects/s recovering 2024-08-17T17:19:13.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:13 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:19:13.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:13 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:19:13.469 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:13.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:13.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:13.498 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:19:13.498 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:19:13.528 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:19:13.529 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:19:13.529 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:19:13.572 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:13.609 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:13.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:13.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:13.711 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:13.751 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:13.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:13.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:13.785 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 2 16 5990 5974 186.677 173.75 0.00604828 0.00534706 2024-08-17T17:19:13.853 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:13.882 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:13.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:13.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:13.984 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:14.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:14.026 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:14.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:14.127 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:14.154 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:14.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:14.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:14.256 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:14.284 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:14.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:14.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:14.386 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:14.416 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:14.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:14.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:14.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:14 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:19:14.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:14 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:19:14.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:14 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:19:14.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:14 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:19:14.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:14 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3766863188' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:19:14.518 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:14.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:14 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:19:14.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:14 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:19:14.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:14 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:19:14.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:14 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:19:14.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:14 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3766863188' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:19:14.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:14 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:19:14.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:14 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:19:14.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:14 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:19:14.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:14 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:19:14.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:14 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3766863188' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:19:14.548 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:14.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:14.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:14.650 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:14.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:14.682 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:14.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:14.783 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:14.785 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 3 16 9046 9030 188.113 191 0.0155163 0.00529938 2024-08-17T17:19:14.811 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:14.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:14.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:14.913 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:14.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:14.955 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:14.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:15.056 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:15.094 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:15.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:15.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:15.196 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:15.224 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:15.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:15.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:15.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:15 smithi003 ceph-mon[103344]: pgmap v230: 89 pgs: 89 active+clean; 239 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s wr, 374 op/s 2024-08-17T17:19:15.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:15 smithi003 ceph-mon[100615]: pgmap v230: 89 pgs: 89 active+clean; 239 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s wr, 374 op/s 2024-08-17T17:19:15.325 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:15.354 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:15.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:15.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:15.457 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:15.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:15 smithi053 ceph-mon[89963]: pgmap v230: 89 pgs: 89 active+clean; 239 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s wr, 374 op/s 2024-08-17T17:19:15.486 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:15.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:15.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:15.588 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:15.615 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:15.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:15.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:15.717 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:15.745 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:15.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:15.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:15.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:19:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:19:15] "GET /metrics HTTP/1.1" 200 36452 "" "Prometheus/2.51.0" 2024-08-17T17:19:15.785 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 4 16 11985 11969 187.002 183.688 0.0030081 0.00533892 2024-08-17T17:19:15.846 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:15.886 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:15.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:15.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:15.990 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:16.020 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:16.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:16.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:16.122 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:16.164 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:16.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:16.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:16.266 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:16.307 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:16.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:16.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:16.409 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:16.452 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:16.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:16.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:16.554 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:16.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:16.599 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:16.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:16.700 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:16.728 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:16.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:16.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:16.785 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 5 16 14399 14383 179.774 150.875 0.00983651 0.00555192 2024-08-17T17:19:16.830 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:16.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:16.853 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:16.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:16.953 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:16.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:16.997 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:16.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:17.098 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:17.125 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:17.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:17.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:17.227 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:17.242 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-08-17T17:19:17.274 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:17.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:17.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:17.376 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:17.414 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:17.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:17.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:17.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:17 smithi003 ceph-mon[100615]: pgmap v231: 89 pgs: 89 active+clean; 347 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 34 MiB/s wr, 548 op/s 2024-08-17T17:19:17.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:17 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/432890700' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:19:17.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:17 smithi003 ceph-mon[103344]: pgmap v231: 89 pgs: 89 active+clean; 347 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 34 MiB/s wr, 548 op/s 2024-08-17T17:19:17.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:17 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/432890700' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:19:17.517 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:17.543 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:17.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:17.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:17.636 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:19:17.636 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-08-17T17:19:17.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-08-17T17:19:17.637 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:19:17.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.637 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:12504 5.5 5:a1c98502:::benchmark_data_smithi003_140793_object12503:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-17T17:19:17.637 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:15.967239+0000", 2024-08-17T17:19:17.637 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 1.46180438, 2024-08-17T17:19:17.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.014657765, 2024-08-17T17:19:17.637 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:17.637 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:17.637 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:17.637 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:19:17.637 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:19:17.637 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:19:17.637 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 12504 2024-08-17T17:19:17.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.638 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:17.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:17.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:15.967239+0000", 2024-08-17T17:19:17.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:17.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:15.967239+0000", 2024-08-17T17:19:17.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:17.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:15.967239+0000", 2024-08-17T17:19:17.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8500000000000001e-07 2024-08-17T17:19:17.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:17.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:15.967259+0000", 2024-08-17T17:19:17.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9154e-05 2024-08-17T17:19:17.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:17.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:15.967268+0000", 2024-08-17T17:19:17.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7769999999999997e-06 2024-08-17T17:19:17.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:17.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:15.967275+0000", 2024-08-17T17:19:17.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4810000000000003e-06 2024-08-17T17:19:17.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:17.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:15.975011+0000", 2024-08-17T17:19:17.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.007735802 2024-08-17T17:19:17.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:17.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:15.975132+0000", 2024-08-17T17:19:17.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000121539 2024-08-17T17:19:17.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:19:17.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:15.976235+0000", 2024-08-17T17:19:17.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0011027140000000001 2024-08-17T17:19:17.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:17.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:15.977615+0000", 2024-08-17T17:19:17.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001380582 2024-08-17T17:19:17.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:17.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:15.977764+0000", 2024-08-17T17:19:17.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014868300000000001 2024-08-17T17:19:17.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:19:17.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:15.981776+0000", 2024-08-17T17:19:17.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040114749999999996 2024-08-17T17:19:17.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:17.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:15.981853+0000", 2024-08-17T17:19:17.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7120999999999996e-05 2024-08-17T17:19:17.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:17.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:15.981897+0000", 2024-08-17T17:19:17.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3852e-05 2024-08-17T17:19:17.643 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.643 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:17.643 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.643 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.35277.0:13639 5.b e119/114 5:d87eb9a4:::benchmark_data_smithi003_140793_object13638:head v 119'896, mlcod=119'893)", 2024-08-17T17:19:17.644 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:16.412082+0000", 2024-08-17T17:19:17.644 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 1.016961271, 2024-08-17T17:19:17.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.014535408999999999, 2024-08-17T17:19:17.644 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:17.644 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:17.644 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:17.644 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:17.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:17.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.412082+0000", 2024-08-17T17:19:17.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:17.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.412082+0000", 2024-08-17T17:19:17.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:17.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.412082+0000", 2024-08-17T17:19:17.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6199999999999999e-07 2024-08-17T17:19:17.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:17.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.412739+0000", 2024-08-17T17:19:17.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00065684000000000001 2024-08-17T17:19:17.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:17.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.412749+0000", 2024-08-17T17:19:17.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9790000000000005e-06 2024-08-17T17:19:17.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:17.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.412755+0000", 2024-08-17T17:19:17.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.891e-06 2024-08-17T17:19:17.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:17.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.419766+0000", 2024-08-17T17:19:17.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0070111230000000002 2024-08-17T17:19:17.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:17.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.419785+0000", 2024-08-17T17:19:17.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8624999999999999e-05 2024-08-17T17:19:17.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:17.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.426580+0000", 2024-08-17T17:19:17.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0067956980000000002 2024-08-17T17:19:17.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:17.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.426617+0000", 2024-08-17T17:19:17.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6990999999999999e-05 2024-08-17T17:19:17.648 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.648 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:17.648 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.648 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:14121 5.5 5:a55af86d:::benchmark_data_smithi003_140793_object14120:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-17T17:19:17.648 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:16.655212+0000", 2024-08-17T17:19:17.648 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.77383164699999996, 2024-08-17T17:19:17.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.016588703999999999, 2024-08-17T17:19:17.648 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:17.648 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:17.649 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:17.649 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:19:17.649 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:19:17.649 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:19:17.649 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 14121 2024-08-17T17:19:17.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.649 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:17.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:17.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.655212+0000", 2024-08-17T17:19:17.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:17.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.655212+0000", 2024-08-17T17:19:17.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:17.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.655213+0000", 2024-08-17T17:19:17.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3200000000000001e-06 2024-08-17T17:19:17.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:17.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.655239+0000", 2024-08-17T17:19:17.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6122000000000002e-05 2024-08-17T17:19:17.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:17.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.655251+0000", 2024-08-17T17:19:17.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2153999999999999e-05 2024-08-17T17:19:17.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:17.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.655259+0000", 2024-08-17T17:19:17.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3540000000000002e-06 2024-08-17T17:19:17.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:17.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.662940+0000", 2024-08-17T17:19:17.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0076814709999999996 2024-08-17T17:19:17.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:17.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.663056+0000", 2024-08-17T17:19:17.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011594099999999999 2024-08-17T17:19:17.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:19:17.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.663512+0000", 2024-08-17T17:19:17.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00045606999999999999 2024-08-17T17:19:17.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:19:17.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.666264+0000", 2024-08-17T17:19:17.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027524759999999998 2024-08-17T17:19:17.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:17.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.666304+0000", 2024-08-17T17:19:17.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9771999999999998e-05 2024-08-17T17:19:17.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:17.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.671706+0000", 2024-08-17T17:19:17.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0054017659999999997 2024-08-17T17:19:17.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:17.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.671761+0000", 2024-08-17T17:19:17.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4957000000000003e-05 2024-08-17T17:19:17.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:17.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.671800+0000", 2024-08-17T17:19:17.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9301000000000002e-05 2024-08-17T17:19:17.654 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.654 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:17.654 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.654 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:14219 5.3 5:cfc48a4a:::benchmark_data_smithi003_140793_object14218:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-17T17:19:17.655 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:16.698880+0000", 2024-08-17T17:19:17.655 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.730162847, 2024-08-17T17:19:17.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.014485474999999999, 2024-08-17T17:19:17.655 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:17.655 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:17.655 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:17.655 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:19:17.655 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:19:17.655 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:19:17.655 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 14219 2024-08-17T17:19:17.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.655 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:17.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:17.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.698880+0000", 2024-08-17T17:19:17.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:17.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.698880+0000", 2024-08-17T17:19:17.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:17.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.698881+0000", 2024-08-17T17:19:17.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0399999999999996e-07 2024-08-17T17:19:17.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:17.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.698915+0000", 2024-08-17T17:19:17.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3958999999999998e-05 2024-08-17T17:19:17.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:17.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.698924+0000", 2024-08-17T17:19:17.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1759999999999999e-06 2024-08-17T17:19:17.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:17.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.698944+0000", 2024-08-17T17:19:17.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0360000000000002e-05 2024-08-17T17:19:17.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:17.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.699044+0000", 2024-08-17T17:19:17.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9134000000000003e-05 2024-08-17T17:19:17.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:17.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.699161+0000", 2024-08-17T17:19:17.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000117782 2024-08-17T17:19:17.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,6", 2024-08-17T17:19:17.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.700121+0000", 2024-08-17T17:19:17.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00095992099999999997 2024-08-17T17:19:17.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:17.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.707398+0000", 2024-08-17T17:19:17.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072768620000000003 2024-08-17T17:19:17.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:17.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.707475+0000", 2024-08-17T17:19:17.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6825999999999999e-05 2024-08-17T17:19:17.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:19:17.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.713269+0000", 2024-08-17T17:19:17.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0057937639999999999 2024-08-17T17:19:17.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:17.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.713323+0000", 2024-08-17T17:19:17.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4236e-05 2024-08-17T17:19:17.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:17.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.713366+0000", 2024-08-17T17:19:17.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2851e-05 2024-08-17T17:19:17.660 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.661 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:17.661 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.661 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:14220 5.3 5:c1a15624:::benchmark_data_smithi003_140793_object14219:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-17T17:19:17.661 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:16.698961+0000", 2024-08-17T17:19:17.661 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.73008265900000002, 2024-08-17T17:19:17.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.014481872999999999, 2024-08-17T17:19:17.661 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:17.661 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:17.661 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:17.661 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:19:17.661 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:19:17.661 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:19:17.662 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 14220 2024-08-17T17:19:17.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.662 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:17.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:17.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.698961+0000", 2024-08-17T17:19:17.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:17.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.698961+0000", 2024-08-17T17:19:17.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:17.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.698961+0000", 2024-08-17T17:19:17.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3799999999999997e-07 2024-08-17T17:19:17.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:17.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.698983+0000", 2024-08-17T17:19:17.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2160000000000001e-05 2024-08-17T17:19:17.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:17.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.698992+0000", 2024-08-17T17:19:17.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3659999999999992e-06 2024-08-17T17:19:17.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:17.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.698998+0000", 2024-08-17T17:19:17.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5649999999999998e-06 2024-08-17T17:19:17.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:17.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.701146+0000", 2024-08-17T17:19:17.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0021477409999999999 2024-08-17T17:19:17.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:17.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.701256+0000", 2024-08-17T17:19:17.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010959900000000001 2024-08-17T17:19:17.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,6", 2024-08-17T17:19:17.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.701386+0000", 2024-08-17T17:19:17.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013088999999999999 2024-08-17T17:19:17.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:17.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.707437+0000", 2024-08-17T17:19:17.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0060506270000000003 2024-08-17T17:19:17.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:17.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.707511+0000", 2024-08-17T17:19:17.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4187e-05 2024-08-17T17:19:17.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:19:17.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.713374+0000", 2024-08-17T17:19:17.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0058625839999999997 2024-08-17T17:19:17.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:17.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.713412+0000", 2024-08-17T17:19:17.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7965e-05 2024-08-17T17:19:17.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:17.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.713442+0000", 2024-08-17T17:19:17.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0651000000000002e-05 2024-08-17T17:19:17.667 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.667 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:17.667 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.668 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:14228 5.3 5:c48d6af6:::benchmark_data_smithi003_140793_object14227:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-17T17:19:17.668 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:16.702890+0000", 2024-08-17T17:19:17.668 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.72615317599999996, 2024-08-17T17:19:17.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017842468, 2024-08-17T17:19:17.668 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:17.668 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:17.668 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:17.668 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:19:17.668 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:19:17.668 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:19:17.668 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 14228 2024-08-17T17:19:17.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.668 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:17.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:17.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.702890+0000", 2024-08-17T17:19:17.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:17.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.702890+0000", 2024-08-17T17:19:17.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:17.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.702891+0000", 2024-08-17T17:19:17.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0299999999999999e-07 2024-08-17T17:19:17.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:17.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.702912+0000", 2024-08-17T17:19:17.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1134000000000001e-05 2024-08-17T17:19:17.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:17.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.702919+0000", 2024-08-17T17:19:17.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0500000000000003e-06 2024-08-17T17:19:17.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:17.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.702925+0000", 2024-08-17T17:19:17.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1700000000000002e-06 2024-08-17T17:19:17.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:17.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.713954+0000", 2024-08-17T17:19:17.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011029406 2024-08-17T17:19:17.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:17.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.714069+0000", 2024-08-17T17:19:17.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000114769 2024-08-17T17:19:17.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.672 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,6", 2024-08-17T17:19:17.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.714192+0000", 2024-08-17T17:19:17.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012274200000000001 2024-08-17T17:19:17.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.672 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:19:17.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.720560+0000", 2024-08-17T17:19:17.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063684800000000001 2024-08-17T17:19:17.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.672 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:17.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.720598+0000", 2024-08-17T17:19:17.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8071000000000002e-05 2024-08-17T17:19:17.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:17.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.720641+0000", 2024-08-17T17:19:17.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2833999999999997e-05 2024-08-17T17:19:17.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:17.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.720696+0000", 2024-08-17T17:19:17.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4421e-05 2024-08-17T17:19:17.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:17.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.720732+0000", 2024-08-17T17:19:17.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6888000000000003e-05 2024-08-17T17:19:17.674 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.674 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:17.674 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.674 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:14454 5.5 5:a984daf0:::benchmark_data_smithi003_140793_object14453:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-17T17:19:17.674 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:16.814470+0000", 2024-08-17T17:19:17.674 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.61457337000000001, 2024-08-17T17:19:17.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0405482, 2024-08-17T17:19:17.674 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:17.674 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:17.674 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:17.675 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:19:17.675 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:19:17.675 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:19:17.675 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 14454 2024-08-17T17:19:17.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.675 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:17.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:17.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.814470+0000", 2024-08-17T17:19:17.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:17.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.814470+0000", 2024-08-17T17:19:17.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:17.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.814471+0000", 2024-08-17T17:19:17.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5880000000000001e-06 2024-08-17T17:19:17.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:17.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.814494+0000", 2024-08-17T17:19:17.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2118999999999999e-05 2024-08-17T17:19:17.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:17.677 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.814506+0000", 2024-08-17T17:19:17.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2061e-05 2024-08-17T17:19:17.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.677 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:17.677 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.814513+0000", 2024-08-17T17:19:17.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5920000000000003e-06 2024-08-17T17:19:17.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.677 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:17.677 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.814866+0000", 2024-08-17T17:19:17.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00035303100000000002 2024-08-17T17:19:17.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:17.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.814982+0000", 2024-08-17T17:19:17.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011592299999999999 2024-08-17T17:19:17.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:19:17.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.815101+0000", 2024-08-17T17:19:17.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000118948 2024-08-17T17:19:17.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:19:17.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.816629+0000", 2024-08-17T17:19:17.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0015280960000000001 2024-08-17T17:19:17.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:17.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.817096+0000", 2024-08-17T17:19:17.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00046707999999999998 2024-08-17T17:19:17.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:17.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.854870+0000", 2024-08-17T17:19:17.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.037773436000000001 2024-08-17T17:19:17.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:17.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.854977+0000", 2024-08-17T17:19:17.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010742 2024-08-17T17:19:17.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:17.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.855018+0000", 2024-08-17T17:19:17.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0905999999999998e-05 2024-08-17T17:19:17.680 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.680 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:17.680 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.680 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:14458 5.3 5:cf7dc92c:::benchmark_data_smithi003_140793_object14457:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-17T17:19:17.680 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:16.817035+0000", 2024-08-17T17:19:17.680 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.61200771399999998, 2024-08-17T17:19:17.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.040060141, 2024-08-17T17:19:17.681 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:17.681 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:17.681 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:17.681 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:19:17.681 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:19:17.681 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:19:17.681 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 14458 2024-08-17T17:19:17.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.681 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:17.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:17.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.817035+0000", 2024-08-17T17:19:17.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:17.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.817035+0000", 2024-08-17T17:19:17.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:17.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.817038+0000", 2024-08-17T17:19:17.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1050000000000002e-06 2024-08-17T17:19:17.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:17.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.817064+0000", 2024-08-17T17:19:17.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6225e-05 2024-08-17T17:19:17.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:17.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.817076+0000", 2024-08-17T17:19:17.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2322e-05 2024-08-17T17:19:17.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:17.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.817083+0000", 2024-08-17T17:19:17.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.905e-06 2024-08-17T17:19:17.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:17.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.817144+0000", 2024-08-17T17:19:17.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0936000000000001e-05 2024-08-17T17:19:17.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:17.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.817273+0000", 2024-08-17T17:19:17.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012901699999999999 2024-08-17T17:19:17.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,6", 2024-08-17T17:19:17.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.854792+0000", 2024-08-17T17:19:17.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.037518595000000002 2024-08-17T17:19:17.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:17.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.856415+0000", 2024-08-17T17:19:17.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016229790000000001 2024-08-17T17:19:17.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:19:17.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.856436+0000", 2024-08-17T17:19:17.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1285999999999999e-05 2024-08-17T17:19:17.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:17.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.856950+0000", 2024-08-17T17:19:17.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000514315 2024-08-17T17:19:17.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:17.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.857048+0000", 2024-08-17T17:19:17.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7881999999999996e-05 2024-08-17T17:19:17.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:17.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.857096+0000", 2024-08-17T17:19:17.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7574000000000002e-05 2024-08-17T17:19:17.686 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.686 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:17.687 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.687 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.35277.0:14457 5.e e119/114 5:700f3353:::benchmark_data_smithi003_140793_object14456:head v 119'962, mlcod=119'958)", 2024-08-17T17:19:17.687 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:16.817484+0000", 2024-08-17T17:19:17.687 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.61155891699999998, 2024-08-17T17:19:17.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.03862115, 2024-08-17T17:19:17.687 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:17.687 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:17.687 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:17.687 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:17.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:17.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.817484+0000", 2024-08-17T17:19:17.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:17.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.817484+0000", 2024-08-17T17:19:17.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:17.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.817485+0000", 2024-08-17T17:19:17.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9199999999999999e-07 2024-08-17T17:19:17.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:17.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.817997+0000", 2024-08-17T17:19:17.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000512084 2024-08-17T17:19:17.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:17.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.854460+0000", 2024-08-17T17:19:17.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.036462940999999999 2024-08-17T17:19:17.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:17.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.854476+0000", 2024-08-17T17:19:17.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5489999999999999e-05 2024-08-17T17:19:17.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:17.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.854548+0000", 2024-08-17T17:19:17.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2438e-05 2024-08-17T17:19:17.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:17.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.854574+0000", 2024-08-17T17:19:17.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5701999999999999e-05 2024-08-17T17:19:17.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:17.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.856063+0000", 2024-08-17T17:19:17.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014889180000000001 2024-08-17T17:19:17.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.691 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:17.691 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.856105+0000", 2024-08-17T17:19:17.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2685000000000002e-05 2024-08-17T17:19:17.691 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.691 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:17.691 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.691 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.35277.0:14454 5.5 e119/114 ondisk, result = 0)", 2024-08-17T17:19:17.691 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:16.818123+0000", 2024-08-17T17:19:17.691 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.61092062899999999, 2024-08-17T17:19:17.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.036902298, 2024-08-17T17:19:17.692 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:17.692 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:17.692 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:19:17.692 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:17.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:17.692 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.818123+0000", 2024-08-17T17:19:17.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:17.692 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.818123+0000", 2024-08-17T17:19:17.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:17.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.818124+0000", 2024-08-17T17:19:17.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3799999999999996e-07 2024-08-17T17:19:17.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:17.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.818128+0000", 2024-08-17T17:19:17.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6779999999999999e-06 2024-08-17T17:19:17.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:17.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.854709+0000", 2024-08-17T17:19:17.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.036580488000000001 2024-08-17T17:19:17.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:17.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.854720+0000", 2024-08-17T17:19:17.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1323e-05 2024-08-17T17:19:17.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:17.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.854856+0000", 2024-08-17T17:19:17.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013576199999999999 2024-08-17T17:19:17.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:17.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.854864+0000", 2024-08-17T17:19:17.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9699999999999999e-06 2024-08-17T17:19:17.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:17.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.855025+0000", 2024-08-17T17:19:17.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000161139 2024-08-17T17:19:17.695 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.695 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:17.695 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.696 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.35277.0:14518 5.d e119/114 5:b43a6cf9:::benchmark_data_smithi003_140793_object14517:head v 119'914, mlcod=119'911)", 2024-08-17T17:19:17.696 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:16.917696+0000", 2024-08-17T17:19:17.696 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.511346984, 2024-08-17T17:19:17.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.015563914999999999, 2024-08-17T17:19:17.696 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:17.696 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:17.696 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:17.696 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:17.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:17.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.917696+0000", 2024-08-17T17:19:17.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:17.697 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.917696+0000", 2024-08-17T17:19:17.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:17.697 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.917696+0000", 2024-08-17T17:19:17.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5100000000000001e-07 2024-08-17T17:19:17.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:17.697 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.917721+0000", 2024-08-17T17:19:17.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4516e-05 2024-08-17T17:19:17.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:17.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.917727+0000", 2024-08-17T17:19:17.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3500000000000002e-06 2024-08-17T17:19:17.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:17.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.917732+0000", 2024-08-17T17:19:17.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6539999999999998e-06 2024-08-17T17:19:17.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:17.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.921846+0000", 2024-08-17T17:19:17.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0041137090000000001 2024-08-17T17:19:17.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:17.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.921869+0000", 2024-08-17T17:19:17.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3402999999999999e-05 2024-08-17T17:19:17.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.700 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:17.700 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.933221+0000", 2024-08-17T17:19:17.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011352352 2024-08-17T17:19:17.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.700 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:17.700 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.933260+0000", 2024-08-17T17:19:17.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8680000000000002e-05 2024-08-17T17:19:17.700 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.700 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:17.700 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.700 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:14529 5.5 5:ab70c4ad:::benchmark_data_smithi003_140793_object14528:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-17T17:19:17.701 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:16.918478+0000", 2024-08-17T17:19:17.701 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.51056537999999996, 2024-08-17T17:19:17.701 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.014654285, 2024-08-17T17:19:17.701 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:17.701 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:17.701 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:17.701 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:19:17.701 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:19:17.701 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:19:17.701 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 14529 2024-08-17T17:19:17.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.701 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:17.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:17.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.918478+0000", 2024-08-17T17:19:17.702 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:17.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.918478+0000", 2024-08-17T17:19:17.702 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:17.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.918479+0000", 2024-08-17T17:19:17.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.254e-06 2024-08-17T17:19:17.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.703 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:17.703 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.918500+0000", 2024-08-17T17:19:17.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0741999999999999e-05 2024-08-17T17:19:17.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.703 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:17.703 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.918567+0000", 2024-08-17T17:19:17.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7670999999999996e-05 2024-08-17T17:19:17.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.703 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:17.703 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.918576+0000", 2024-08-17T17:19:17.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5010000000000004e-06 2024-08-17T17:19:17.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.704 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:17.704 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.921714+0000", 2024-08-17T17:19:17.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031378909999999999 2024-08-17T17:19:17.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.704 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:17.704 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.921861+0000", 2024-08-17T17:19:17.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014689 2024-08-17T17:19:17.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.704 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:19:17.704 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.922151+0000", 2024-08-17T17:19:17.705 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00029066600000000003 2024-08-17T17:19:17.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.705 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:17.705 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.927214+0000", 2024-08-17T17:19:17.705 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0050622080000000003 2024-08-17T17:19:17.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.705 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:17.705 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.928830+0000", 2024-08-17T17:19:17.705 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016165649999999999 2024-08-17T17:19:17.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.705 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:19:17.705 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.933011+0000", 2024-08-17T17:19:17.706 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0041803129999999997 2024-08-17T17:19:17.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.706 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:17.706 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.933077+0000", 2024-08-17T17:19:17.706 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6842999999999997e-05 2024-08-17T17:19:17.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.706 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:17.706 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.933132+0000", 2024-08-17T17:19:17.706 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4741000000000003e-05 2024-08-17T17:19:17.706 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.706 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:17.707 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.707 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.35277.0:15215 5.7 e119/114 5:e05e4835:::benchmark_data_smithi003_140793_object15214:head v 119'990, mlcod=119'988)", 2024-08-17T17:19:17.707 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:17.239658+0000", 2024-08-17T17:19:17.707 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.18938492900000001, 2024-08-17T17:19:17.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.015361428, 2024-08-17T17:19:17.707 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:17.707 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:17.707 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:17.707 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:17.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:17.707 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.239658+0000", 2024-08-17T17:19:17.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.708 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.708 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.708 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:17.708 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.239658+0000", 2024-08-17T17:19:17.708 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.708 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.708 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.708 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:17.708 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.239659+0000", 2024-08-17T17:19:17.708 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2399999999999998e-07 2024-08-17T17:19:17.708 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.708 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.708 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:17.708 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.239682+0000", 2024-08-17T17:19:17.709 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3093999999999999e-05 2024-08-17T17:19:17.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.709 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.709 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:17.709 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.239690+0000", 2024-08-17T17:19:17.709 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3760000000000008e-06 2024-08-17T17:19:17.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.709 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.709 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:17.709 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.239696+0000", 2024-08-17T17:19:17.709 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1430000000000003e-06 2024-08-17T17:19:17.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.709 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.709 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:17.710 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.239753+0000", 2024-08-17T17:19:17.710 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6261999999999998e-05 2024-08-17T17:19:17.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.710 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.710 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:17.710 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.239773+0000", 2024-08-17T17:19:17.710 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0188000000000001e-05 2024-08-17T17:19:17.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.710 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.710 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:17.710 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.254974+0000", 2024-08-17T17:19:17.710 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.015201054 2024-08-17T17:19:17.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.710 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.711 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:17.711 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.255020+0000", 2024-08-17T17:19:17.711 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5787000000000001e-05 2024-08-17T17:19:17.711 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.711 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:17.711 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.711 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.711 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.711 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.35277.0:15222 5.8 e119/111 5:1431a9a9:::benchmark_data_smithi003_140793_object15221:head v 119'919, mlcod=119'917)", 2024-08-17T17:19:17.711 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:17.240725+0000", 2024-08-17T17:19:17.711 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.188318439, 2024-08-17T17:19:17.711 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017704599000000001, 2024-08-17T17:19:17.711 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:17.711 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:17.712 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:17.712 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:17.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.712 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:17.712 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.240725+0000", 2024-08-17T17:19:17.712 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.712 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:17.712 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.240725+0000", 2024-08-17T17:19:17.712 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.712 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:17.713 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.240726+0000", 2024-08-17T17:19:17.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0780000000000001e-06 2024-08-17T17:19:17.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.713 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:17.713 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.240753+0000", 2024-08-17T17:19:17.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7243e-05 2024-08-17T17:19:17.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.713 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:17.713 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.240765+0000", 2024-08-17T17:19:17.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2145e-05 2024-08-17T17:19:17.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.714 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:17.714 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.240772+0000", 2024-08-17T17:19:17.714 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2080000000000003e-06 2024-08-17T17:19:17.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.714 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.714 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:17.714 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.241286+0000", 2024-08-17T17:19:17.714 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00051402700000000002 2024-08-17T17:19:17.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.714 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.714 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:17.714 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.241307+0000", 2024-08-17T17:19:17.714 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0098e-05 2024-08-17T17:19:17.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.715 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:17.715 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.258373+0000", 2024-08-17T17:19:17.715 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017066089999999999 2024-08-17T17:19:17.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.715 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:17.715 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.258429+0000", 2024-08-17T17:19:17.715 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6709999999999997e-05 2024-08-17T17:19:17.715 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.715 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:17.715 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.715 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.35277.0:15234 5.a e120/114 5:5bbcccdf:::benchmark_data_smithi003_140793_object15233:head v 120'896, mlcod=119'894)", 2024-08-17T17:19:17.716 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:17.250263+0000", 2024-08-17T17:19:17.716 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.17878005, 2024-08-17T17:19:17.716 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.016195999999999999, 2024-08-17T17:19:17.716 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:17.716 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:17.716 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:17.716 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:17.716 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.716 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:17.716 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.250263+0000", 2024-08-17T17:19:17.716 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.716 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.716 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.716 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:17.717 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.250263+0000", 2024-08-17T17:19:17.717 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.717 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.717 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.717 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:17.717 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.250264+0000", 2024-08-17T17:19:17.717 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3099999999999998e-07 2024-08-17T17:19:17.717 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.717 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.717 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:17.717 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.250288+0000", 2024-08-17T17:19:17.717 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4749e-05 2024-08-17T17:19:17.717 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.717 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.718 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:17.718 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.250298+0000", 2024-08-17T17:19:17.718 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8409999999999995e-06 2024-08-17T17:19:17.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.718 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:17.718 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.250306+0000", 2024-08-17T17:19:17.718 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2590000000000003e-06 2024-08-17T17:19:17.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.718 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:17.718 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.250998+0000", 2024-08-17T17:19:17.718 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00069220100000000004 2024-08-17T17:19:17.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.719 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:17.719 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.251015+0000", 2024-08-17T17:19:17.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6781000000000001e-05 2024-08-17T17:19:17.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.719 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:17.719 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.260223+0000", 2024-08-17T17:19:17.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0092085169999999994 2024-08-17T17:19:17.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.719 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:17.719 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.266459+0000", 2024-08-17T17:19:17.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0062361209999999999 2024-08-17T17:19:17.719 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.719 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:17.720 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.720 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.720 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.720 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:15237 5.5 5:a0c91037:::benchmark_data_smithi003_140793_object15236:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-08-17T17:19:17.720 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:17.255752+0000", 2024-08-17T17:19:17.720 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.173291366, 2024-08-17T17:19:17.720 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.021989075, 2024-08-17T17:19:17.720 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:17.720 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:17.720 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:17.720 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:19:17.720 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:19:17.720 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:19:17.720 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 15237 2024-08-17T17:19:17.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.721 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:17.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:17.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.255752+0000", 2024-08-17T17:19:17.721 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:17.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.255752+0000", 2024-08-17T17:19:17.721 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.722 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:17.722 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.255754+0000", 2024-08-17T17:19:17.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7630000000000001e-06 2024-08-17T17:19:17.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.722 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:17.722 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.255780+0000", 2024-08-17T17:19:17.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6754000000000002e-05 2024-08-17T17:19:17.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.722 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:17.722 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.255794+0000", 2024-08-17T17:19:17.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3417e-05 2024-08-17T17:19:17.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.723 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:17.723 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.255811+0000", 2024-08-17T17:19:17.723 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6912999999999999e-05 2024-08-17T17:19:17.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.723 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:17.723 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.255895+0000", 2024-08-17T17:19:17.723 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4518000000000006e-05 2024-08-17T17:19:17.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.723 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op must wait for map", 2024-08-17T17:19:17.723 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.255902+0000", 2024-08-17T17:19:17.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0299999999999996e-06 2024-08-17T17:19:17.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.724 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:17.724 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.265666+0000", 2024-08-17T17:19:17.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0097635699999999992 2024-08-17T17:19:17.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.724 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:17.724 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.266462+0000", 2024-08-17T17:19:17.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00079658699999999997 2024-08-17T17:19:17.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.725 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:19:17.725 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.266615+0000", 2024-08-17T17:19:17.725 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000152929 2024-08-17T17:19:17.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.725 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:19:17.725 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.270042+0000", 2024-08-17T17:19:17.725 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034268810000000001 2024-08-17T17:19:17.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.725 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:17.725 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.273377+0000", 2024-08-17T17:19:17.725 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0033348620000000001 2024-08-17T17:19:17.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.726 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:17.726 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.276033+0000", 2024-08-17T17:19:17.726 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026559859999999999 2024-08-17T17:19:17.726 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.726 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:17.726 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.277683+0000", 2024-08-17T17:19:17.726 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016503589999999999 2024-08-17T17:19:17.726 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.726 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:17.726 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.277741+0000", 2024-08-17T17:19:17.726 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7506000000000001e-05 2024-08-17T17:19:17.727 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.727 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:17.727 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.727 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.727 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:15254 5.5 5:aa383707:::benchmark_data_smithi003_140793_object15253:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-08-17T17:19:17.727 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:17.267652+0000", 2024-08-17T17:19:17.727 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.16139126600000001, 2024-08-17T17:19:17.727 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.020297315999999999, 2024-08-17T17:19:17.727 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:17.727 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:17.727 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:17.727 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:19:17.728 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:19:17.728 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:19:17.728 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 15254 2024-08-17T17:19:17.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.728 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:17.728 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.728 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:17.728 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.267652+0000", 2024-08-17T17:19:17.728 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.728 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.728 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:17.728 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.267652+0000", 2024-08-17T17:19:17.728 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:17.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.267654+0000", 2024-08-17T17:19:17.729 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9039999999999999e-06 2024-08-17T17:19:17.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:17.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.267681+0000", 2024-08-17T17:19:17.729 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6690999999999999e-05 2024-08-17T17:19:17.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:17.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.267693+0000", 2024-08-17T17:19:17.730 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2306e-05 2024-08-17T17:19:17.730 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.730 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.730 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:17.730 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.267700+0000", 2024-08-17T17:19:17.730 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6499999999999996e-06 2024-08-17T17:19:17.730 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.730 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.730 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:17.730 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.267992+0000", 2024-08-17T17:19:17.730 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00029173399999999998 2024-08-17T17:19:17.730 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.730 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.730 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:17.731 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.268131+0000", 2024-08-17T17:19:17.731 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013875199999999999 2024-08-17T17:19:17.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.731 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.731 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:19:17.731 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.269143+0000", 2024-08-17T17:19:17.731 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010116579999999999 2024-08-17T17:19:17.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.731 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.731 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:19:17.731 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.270249+0000", 2024-08-17T17:19:17.731 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00110608 2024-08-17T17:19:17.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.731 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.731 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:17.732 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.277795+0000", 2024-08-17T17:19:17.732 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0075462100000000002 2024-08-17T17:19:17.732 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.732 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.732 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:17.732 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.286110+0000", 2024-08-17T17:19:17.732 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0083151389999999992 2024-08-17T17:19:17.732 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.732 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.732 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:17.732 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.287904+0000", 2024-08-17T17:19:17.732 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017934579999999999 2024-08-17T17:19:17.732 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.732 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.733 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:17.733 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.287949+0000", 2024-08-17T17:19:17.733 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5734e-05 2024-08-17T17:19:17.733 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.733 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:17.733 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.733 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.733 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.733 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.35277.0:15356 5.7 e120/114 5:e53e48d1:::benchmark_data_smithi003_140793_object15355:head v 120'1006, mlcod=120'1003)", 2024-08-17T17:19:17.733 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:17.331235+0000", 2024-08-17T17:19:17.733 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.097807896000000005, 2024-08-17T17:19:17.733 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.016736104000000002, 2024-08-17T17:19:17.733 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:17.733 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:17.734 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:17.734 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:17.734 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.734 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:17.734 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.331235+0000", 2024-08-17T17:19:17.734 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.734 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.734 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.734 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:17.734 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.331235+0000", 2024-08-17T17:19:17.734 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.734 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.734 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.734 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:17.735 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.331236+0000", 2024-08-17T17:19:17.735 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6300000000000004e-07 2024-08-17T17:19:17.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.735 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.735 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:17.735 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.331261+0000", 2024-08-17T17:19:17.735 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4457e-05 2024-08-17T17:19:17.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.735 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.735 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:17.735 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.331276+0000", 2024-08-17T17:19:17.735 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5409000000000001e-05 2024-08-17T17:19:17.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.735 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.736 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:17.736 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.331284+0000", 2024-08-17T17:19:17.736 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8590000000000005e-06 2024-08-17T17:19:17.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.736 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:17.736 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.331525+0000", 2024-08-17T17:19:17.736 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00024103199999999999 2024-08-17T17:19:17.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.736 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:17.736 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.331545+0000", 2024-08-17T17:19:17.736 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9585e-05 2024-08-17T17:19:17.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.737 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.737 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:17.737 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.347939+0000", 2024-08-17T17:19:17.737 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.016394272000000001 2024-08-17T17:19:17.737 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.737 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.737 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:17.737 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.347971+0000", 2024-08-17T17:19:17.737 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2626999999999998e-05 2024-08-17T17:19:17.737 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.737 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:17.737 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.737 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.737 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.737 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:15374 5.5 5:aacb4791:::benchmark_data_smithi003_140793_object15373:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-08-17T17:19:17.738 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:17.334797+0000", 2024-08-17T17:19:17.738 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.094246478999999994, 2024-08-17T17:19:17.738 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.016168786000000001, 2024-08-17T17:19:17.738 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:17.738 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:17.738 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:17.738 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:19:17.738 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:19:17.738 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:19:17.738 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 15374 2024-08-17T17:19:17.738 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.738 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:17.739 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.739 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:17.739 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.334797+0000", 2024-08-17T17:19:17.739 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.739 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.739 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.739 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:17.739 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.334797+0000", 2024-08-17T17:19:17.739 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.739 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.739 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.739 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:17.740 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.334798+0000", 2024-08-17T17:19:17.740 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1829999999999999e-06 2024-08-17T17:19:17.740 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.740 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.740 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:17.740 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.334829+0000", 2024-08-17T17:19:17.740 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0762000000000001e-05 2024-08-17T17:19:17.740 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.740 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.740 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:17.740 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.334841+0000", 2024-08-17T17:19:17.740 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1953999999999999e-05 2024-08-17T17:19:17.740 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.740 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.741 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:17.741 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.334848+0000", 2024-08-17T17:19:17.741 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2089999999999999e-06 2024-08-17T17:19:17.741 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.741 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.741 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:17.741 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.335951+0000", 2024-08-17T17:19:17.741 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0011029779999999999 2024-08-17T17:19:17.741 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.741 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.741 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:17.741 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.336069+0000", 2024-08-17T17:19:17.741 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000118251 2024-08-17T17:19:17.741 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.742 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.742 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:19:17.742 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.336206+0000", 2024-08-17T17:19:17.742 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000137229 2024-08-17T17:19:17.742 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.742 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.742 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:19:17.742 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.341578+0000", 2024-08-17T17:19:17.742 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0053715229999999996 2024-08-17T17:19:17.742 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.742 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.742 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:17.743 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.342411+0000", 2024-08-17T17:19:17.743 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00083340599999999997 2024-08-17T17:19:17.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.743 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.743 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:17.743 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.348156+0000", 2024-08-17T17:19:17.743 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0057448239999999999 2024-08-17T17:19:17.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.743 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.743 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:17.743 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.348213+0000", 2024-08-17T17:19:17.743 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6897000000000001e-05 2024-08-17T17:19:17.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.743 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.744 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:17.744 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.350966+0000", 2024-08-17T17:19:17.744 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027525700000000002 2024-08-17T17:19:17.744 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.744 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:17.744 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.744 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.744 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.744 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:15378 5.5 5:a45da2d6:::benchmark_data_smithi003_140793_object15377:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-08-17T17:19:17.744 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:17.335960+0000", 2024-08-17T17:19:17.744 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.093083508999999995, 2024-08-17T17:19:17.744 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.015803841999999999, 2024-08-17T17:19:17.744 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:17.744 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:17.744 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:17.745 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:19:17.745 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:19:17.745 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:19:17.745 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 15378 2024-08-17T17:19:17.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.745 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:17.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.745 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:17.745 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.335960+0000", 2024-08-17T17:19:17.745 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.745 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:17.745 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.335960+0000", 2024-08-17T17:19:17.746 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:17.746 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.746 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.746 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:17.746 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.335961+0000", 2024-08-17T17:19:17.746 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.61e-06 2024-08-17T17:19:17.746 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.746 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.746 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:17.746 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.335983+0000", 2024-08-17T17:19:17.746 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1237e-05 2024-08-17T17:19:17.746 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.746 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.746 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:17.747 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.335994+0000", 2024-08-17T17:19:17.747 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1568e-05 2024-08-17T17:19:17.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.747 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:17.747 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.336002+0000", 2024-08-17T17:19:17.747 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.418e-06 2024-08-17T17:19:17.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.747 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:17.747 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.336871+0000", 2024-08-17T17:19:17.747 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00086941999999999998 2024-08-17T17:19:17.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.748 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:17.748 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.337024+0000", 2024-08-17T17:19:17.748 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000153339 2024-08-17T17:19:17.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.748 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:19:17.748 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.337157+0000", 2024-08-17T17:19:17.748 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013320699999999999 2024-08-17T17:19:17.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.748 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:17.748 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.342444+0000", 2024-08-17T17:19:17.748 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0052866359999999999 2024-08-17T17:19:17.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.749 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:17.749 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.351019+0000", 2024-08-17T17:19:17.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0085748149999999995 2024-08-17T17:19:17.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.749 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:19:17.749 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.351662+0000", 2024-08-17T17:19:17.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00064353299999999995 2024-08-17T17:19:17.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.749 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:17.749 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.351719+0000", 2024-08-17T17:19:17.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6388000000000003e-05 2024-08-17T17:19:17.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:17.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:17.750 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:17.750 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.351764+0000", 2024-08-17T17:19:17.750 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4671e-05 2024-08-17T17:19:17.750 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.750 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:17.750 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.750 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:17.750 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:17.750 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:19:17.750 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:17.751 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:19:17.752 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:17 smithi053 ceph-mon[89963]: pgmap v231: 89 pgs: 89 active+clean; 347 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 34 MiB/s wr, 548 op/s 2024-08-17T17:19:17.752 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:17 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/432890700' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:19:17.785 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 6 16 16327 16311 169.893 120.5 0.0027184 0.00587972 2024-08-17T17:19:17.790 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:17.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:17.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:17.892 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:17.948 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:17.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:17.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:18.050 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:18.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:18.095 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:18.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:18.196 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:18.241 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:18.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:18.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:18.345 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:18.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:18.386 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:18.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:18.487 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:18.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:18 smithi003 ceph-mon[100615]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:19:18.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:18 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/432890700' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:19:18.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:18 smithi003 ceph-mon[100615]: osdmap e120: 8 total, 8 up, 8 in 2024-08-17T17:19:18.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:18 smithi003 ceph-mon[103344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:19:18.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:18 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/432890700' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:19:18.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:18 smithi003 ceph-mon[103344]: osdmap e120: 8 total, 8 up, 8 in 2024-08-17T17:19:18.527 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:18.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:18.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:18.618 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-17T17:19:18.629 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:18.654 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:18.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:18.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:18.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:18 smithi053 ceph-mon[89963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:19:18.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:18 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/432890700' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:19:18.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:18 smithi053 ceph-mon[89963]: osdmap e120: 8 total, 8 up, 8 in 2024-08-17T17:19:18.757 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:18.785 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:18.786 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 7 16 18489 18473 164.925 135.125 0.00771881 0.00605676 2024-08-17T17:19:18.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:18.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:18.887 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:18.934 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:18.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:18.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:19.036 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:19.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:19.080 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:19.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:19.181 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:19.208 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:19.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:19.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:19.310 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:19.340 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:19.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:19.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:19.442 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:19.483 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:19.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:19.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:19.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:19 smithi003 ceph-mon[100615]: pgmap v233: 89 pgs: 89 active+clean; 459 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 53 MiB/s wr, 842 op/s 2024-08-17T17:19:19.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:19 smithi003 ceph-mon[103344]: pgmap v233: 89 pgs: 89 active+clean; 459 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 53 MiB/s wr, 842 op/s 2024-08-17T17:19:19.586 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:19.617 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:19.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:19.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:19.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:19 smithi053 ceph-mon[89963]: pgmap v233: 89 pgs: 89 active+clean; 459 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 53 MiB/s wr, 842 op/s 2024-08-17T17:19:19.719 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:19.746 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:19.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:19.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:19.754 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:19:19.788 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 8 16 20784 20768 162.188 143.438 0.00897656 0.00615769 2024-08-17T17:19:19.809 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:19:19.848 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:19.892 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:19.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:19.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:19.995 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:20.035 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:20.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:20.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:20.137 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:20.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:20.165 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:20.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:20.266 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:20.294 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:20.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:20.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:20.395 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:20.427 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:20.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:20.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:20.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:19:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:19:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:19:20.529 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:20.556 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:20.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:20.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:20.658 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:20.699 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:20.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:20.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:20.788 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 9 15 22961 22946 159.291 136.125 0.00864436 0.00627114 2024-08-17T17:19:20.800 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:20.829 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:20.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:20.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:20.931 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:20.972 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:20.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:20.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:21.074 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:21.111 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:21.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:21.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:21.214 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:21.253 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:21.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:21.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:21.355 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:21.394 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:21.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:21.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:21.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:21.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:21 smithi003 ceph-mon[100615]: pgmap v234: 89 pgs: 89 active+clean; 1.1 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-08-17T17:19:21.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:21 smithi003 ceph-mon[103344]: pgmap v234: 89 pgs: 89 active+clean; 1.1 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-08-17T17:19:21.532 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:21.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:21.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:21.634 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:21.670 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:19:21.673 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:21.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:21.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:21.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:21 smithi053 ceph-mon[89963]: pgmap v234: 89 pgs: 89 active+clean; 1.1 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-08-17T17:19:21.776 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:21.788 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 10 16 25093 25077 156.679 133.188 0.0106545 0.00637444 2024-08-17T17:19:21.800 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:21.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:21.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:21.902 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:21.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:21.947 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:21.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:22.048 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:22.088 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:22.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:22.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:22.189 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:22.229 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:22.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:22.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:22.263 INFO:teuthology.orchestra.run.smithi003.stdout:moved 3 / 267 (1.1236%) 2024-08-17T17:19:22.263 INFO:teuthology.orchestra.run.smithi003.stdout:avg 33.375 2024-08-17T17:19:22.263 INFO:teuthology.orchestra.run.smithi003.stdout:stddev 11.8842 -> 12.1752 (expected baseline 5.40399) 2024-08-17T17:19:22.263 INFO:teuthology.orchestra.run.smithi003.stdout:min osd.7 with 16 -> 16 pgs (0.479401 -> 0.479401 * mean) 2024-08-17T17:19:22.263 INFO:teuthology.orchestra.run.smithi003.stdout:max osd.1 with 49 -> 50 pgs (1.46816 -> 1.49813 * mean) 2024-08-17T17:19:22.263 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:19:22.264 INFO:teuthology.orchestra.run.smithi003.stdout:oload 110 2024-08-17T17:19:22.264 INFO:teuthology.orchestra.run.smithi003.stdout:max_change 0.05 2024-08-17T17:19:22.264 INFO:teuthology.orchestra.run.smithi003.stdout:max_change_osds 4 2024-08-17T17:19:22.264 INFO:teuthology.orchestra.run.smithi003.stdout:average_utilization 0.0061 2024-08-17T17:19:22.264 INFO:teuthology.orchestra.run.smithi003.stdout:overload_utilization 0.0067 2024-08-17T17:19:22.264 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0 weight 1.0000 -> 0.9500 2024-08-17T17:19:22.264 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7 weight 0.5465 -> 0.5965 2024-08-17T17:19:22.264 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5 weight 0.5627 -> 0.5127 2024-08-17T17:19:22.264 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4 weight 0.5549 -> 0.5049 2024-08-17T17:19:22.331 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:22.358 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:22.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:22.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:22.460 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:22.484 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:22.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:22.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:22.522 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:22 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'62260','4':'33092','5':'33599','7':'39089'}"} : dispatch 2024-08-17T17:19:22.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:22 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'62260','4':'33092','5':'33599','7':'39089'}"} : dispatch 2024-08-17T17:19:22.586 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:22.608 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:22.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:22.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:22.710 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:22.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:22 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'62260','4':'33092','5':'33599','7':'39089'}"} : dispatch 2024-08-17T17:19:22.733 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:22.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:22.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:22.788 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 11 16 26360 26344 149.635 79.1875 0.00157725 0.00645279 2024-08-17T17:19:22.834 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:22.857 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:22.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:22.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:22.959 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:22.982 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:22.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:22.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:23.084 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:23.106 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:23.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:23.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:23.208 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:23.233 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:23.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:23.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:23.335 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:23.359 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:23.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:23.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:23.460 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:23.484 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:23.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:23.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:23.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:23 smithi003 ceph-mon[100615]: pgmap v235: 89 pgs: 89 active+clean; 1.1 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-08-17T17:19:23.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:23 smithi003 ceph-mon[100615]: from='client.45155 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:19:23.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'62260','4':'33092','5':'33599','7':'39089'}"}]': finished 2024-08-17T17:19:23.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:23 smithi003 ceph-mon[100615]: osdmap e121: 8 total, 8 up, 8 in 2024-08-17T17:19:23.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:19:23.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:19:23.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:23 smithi003 ceph-mon[103344]: pgmap v235: 89 pgs: 89 active+clean; 1.1 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-08-17T17:19:23.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:23 smithi003 ceph-mon[103344]: from='client.45155 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:19:23.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'62260','4':'33092','5':'33599','7':'39089'}"}]': finished 2024-08-17T17:19:23.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:23 smithi003 ceph-mon[103344]: osdmap e121: 8 total, 8 up, 8 in 2024-08-17T17:19:23.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:19:23.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:19:23.586 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:23.609 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:23.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:23.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:23.711 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:23 smithi053 ceph-mon[89963]: pgmap v235: 89 pgs: 89 active+clean; 1.1 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-08-17T17:19:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:23 smithi053 ceph-mon[89963]: from='client.45155 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:19:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'62260','4':'33092','5':'33599','7':'39089'}"}]': finished 2024-08-17T17:19:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:23 smithi053 ceph-mon[89963]: osdmap e121: 8 total, 8 up, 8 in 2024-08-17T17:19:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:19:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:19:23.734 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:23.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:23.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:23.788 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 12 16 26360 26344 137.168 0 - 0.00645279 2024-08-17T17:19:23.836 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:23.861 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:23.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:23.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:23.963 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:23.990 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:23.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:23.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:24.092 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:24.118 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:24.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:24.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:24.220 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:24.247 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:24.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:24.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:24.349 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:24.374 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:24.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:24.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:24.476 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:24.502 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:24.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:24.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:24.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:24 smithi003 ceph-mon[100615]: osdmap e122: 8 total, 8 up, 8 in 2024-08-17T17:19:24.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:24 smithi003 ceph-mon[103344]: osdmap e122: 8 total, 8 up, 8 in 2024-08-17T17:19:24.604 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:24.630 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:24.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:24.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:24.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:24 smithi053 ceph-mon[89963]: osdmap e122: 8 total, 8 up, 8 in 2024-08-17T17:19:24.733 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:24.760 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:24.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:24.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:24.788 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 13 16 26360 26344 126.618 0 - 0.00645279 2024-08-17T17:19:24.861 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:24.889 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:24.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:24.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:24.990 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:25.017 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:25.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:25.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:25.119 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:25.144 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:25.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:25.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:25.246 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:25.275 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:25.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:25.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:25.377 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:25.405 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:25.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:25.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:25.507 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:25.548 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:25.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:25.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:25.650 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:25.653 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:25 smithi003 ceph-mon[100615]: pgmap v238: 89 pgs: 89 active+clean; 1.5 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 187 MiB/s wr, 2.99k op/s 2024-08-17T17:19:25.653 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:25 smithi003 ceph-mon[100615]: osdmap e123: 8 total, 8 up, 8 in 2024-08-17T17:19:25.653 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:25 smithi003 ceph-mon[103344]: pgmap v238: 89 pgs: 89 active+clean; 1.5 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 187 MiB/s wr, 2.99k op/s 2024-08-17T17:19:25.653 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:25 smithi003 ceph-mon[103344]: osdmap e123: 8 total, 8 up, 8 in 2024-08-17T17:19:25.687 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:25.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:25.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:25.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:25 smithi053 ceph-mon[89963]: pgmap v238: 89 pgs: 89 active+clean; 1.5 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 187 MiB/s wr, 2.99k op/s 2024-08-17T17:19:25.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:25 smithi053 ceph-mon[89963]: osdmap e123: 8 total, 8 up, 8 in 2024-08-17T17:19:25.788 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:25.789 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 14 16 27255 27239 121.57 18.6458 0.0047204 0.0082202 2024-08-17T17:19:25.815 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:25.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:25.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:25.917 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:25.956 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:25.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:25.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:26.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:19:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:19:25] "GET /metrics HTTP/1.1" 200 36502 "" "Prometheus/2.51.0" 2024-08-17T17:19:26.058 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:26.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:26.103 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:26.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:26.204 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:26.231 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:26.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:26.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:26.332 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:26.372 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:26.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:26.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:26.475 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:26.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:26.533 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:26.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:26.634 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:26.679 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:26.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:26.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:26.781 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:26.789 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 15 16 29115 29099 121.214 116.25 0.0129253 0.00824216 2024-08-17T17:19:26.824 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:26.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:26.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:26.925 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:26.965 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:26.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:26.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:27.067 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:27.093 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:27.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:27.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:27.195 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:27.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:27.223 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:27.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:27.324 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:27.336 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:19:27.336 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:19:27.336 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:27.336 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:30082 5.c 5:3e3c6c71:::benchmark_data_smithi003_140793_object30081:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-08-17T17:19:27.336 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:27.331493+0000", 2024-08-17T17:19:27.336 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.002785311, 2024-08-17T17:19:27.336 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002817843, 2024-08-17T17:19:27.337 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:27.337 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:27.337 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:19:27.337 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:19:27.337 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:19:27.337 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:19:27.337 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 30082 2024-08-17T17:19:27.337 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:27.337 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:27.337 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:27.337 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:27.337 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:27.331493+0000", 2024-08-17T17:19:27.337 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:27.337 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:27.337 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:27.338 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:27.338 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:27.331493+0000", 2024-08-17T17:19:27.338 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:27.338 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:27.338 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:27.338 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:27.338 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:27.331494+0000", 2024-08-17T17:19:27.338 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.584e-06 2024-08-17T17:19:27.338 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:27.338 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:27.338 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:27.338 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:27.331515+0000", 2024-08-17T17:19:27.338 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0247000000000001e-05 2024-08-17T17:19:27.339 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:27.339 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:27.339 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:27.339 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:27.331527+0000", 2024-08-17T17:19:27.339 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2463000000000001e-05 2024-08-17T17:19:27.339 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:27.339 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:27.339 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:27.339 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:27.331535+0000", 2024-08-17T17:19:27.339 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4769999999999997e-06 2024-08-17T17:19:27.339 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:27.339 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:27.339 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:27.339 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:27.331621+0000", 2024-08-17T17:19:27.339 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6575000000000004e-05 2024-08-17T17:19:27.340 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:27.340 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:27.340 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:27.340 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:27.331771+0000", 2024-08-17T17:19:27.340 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014941299999999999 2024-08-17T17:19:27.340 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:27.340 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:27.340 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4", 2024-08-17T17:19:27.340 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:27.331923+0000", 2024-08-17T17:19:27.340 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000152576 2024-08-17T17:19:27.340 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:27.340 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:27.340 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:19:27.340 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:27.333472+0000", 2024-08-17T17:19:27.340 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0015482580000000001 2024-08-17T17:19:27.340 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:27.341 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:27.341 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:27.341 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:27.333562+0000", 2024-08-17T17:19:27.341 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0249999999999998e-05 2024-08-17T17:19:27.341 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:27.341 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:27.341 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:27.341 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:27.341 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-08-17T17:19:27.341 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 1 2024-08-17T17:19:27.341 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:19:27.350 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:27.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:27.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:27.452 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:27.479 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:27.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:27.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:27.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:27 smithi003 ceph-mon[103344]: pgmap v240: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.6 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s; 1632/79278 objects misplaced (2.059%); 139 KiB/s, 2 objects/s recovering 2024-08-17T17:19:27.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:27 smithi003 ceph-mon[100615]: pgmap v240: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.6 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s; 1632/79278 objects misplaced (2.059%); 139 KiB/s, 2 objects/s recovering 2024-08-17T17:19:27.580 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:27.622 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:27.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:27.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:27.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:27 smithi053 ceph-mon[89963]: pgmap v240: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.6 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s; 1632/79278 objects misplaced (2.059%); 139 KiB/s, 2 objects/s recovering 2024-08-17T17:19:27.724 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:27.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:27.765 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:27.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:27.789 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 16 16 30891 30875 120.575 111 0.00997979 0.00828723 2024-08-17T17:19:27.867 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:27.893 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:27.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:27.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:27.994 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:28.022 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:28.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:28.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:28.124 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:28.149 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:28.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:28.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:28.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:28.294 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:28.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:28.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:28.395 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:28.420 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:28.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:28.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:28.522 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:28.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:28.562 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:28.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:28.607 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-17T17:19:28.663 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:28.707 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:28.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:28.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:28.789 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 17 16 32642 32626 119.919 109.438 0.0108224 0.0083333 2024-08-17T17:19:28.808 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:28.835 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:28.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:28.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:28.937 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:28.962 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:28.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:28.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:29.048 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-08-17T17:19:29.048 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd test-reweight-by-utilization 110 0.05 2024-08-17T17:19:29.063 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:29.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:29.088 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:29.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:29.190 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:29.244 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:19:29.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:29.270 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:29.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:29.317 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:19:29.371 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:29.397 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:29.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:29.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:29.499 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:29.523 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:29.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:29.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:29.575 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:29 smithi003 ceph-mon[100615]: pgmap v241: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.6 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s; 1632/80709 objects misplaced (2.022%); 139 KiB/s, 2 objects/s recovering 2024-08-17T17:19:29.575 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:29 smithi003 ceph-mon[103344]: pgmap v241: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.6 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s; 1632/80709 objects misplaced (2.022%); 139 KiB/s, 2 objects/s recovering 2024-08-17T17:19:29.625 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:29.652 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:29.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:29.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:29.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:29 smithi053 ceph-mon[89963]: pgmap v241: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.6 GiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 92 MiB/s wr, 1.47k op/s; 1632/80709 objects misplaced (2.022%); 139 KiB/s, 2 objects/s recovering 2024-08-17T17:19:29.753 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:29.782 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:29.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:29.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:29.789 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 18 16 34158 34142 118.521 94.75 0.00531411 0.00843141 2024-08-17T17:19:29.883 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:29.921 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:29.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:29.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:30.023 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:30.045 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:30.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:30.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:30.147 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:30.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:30.188 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:30.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:30.288 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:30.329 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:30.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:30.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:30.431 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:30.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:19:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:19:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:19:30.470 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:30.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:30.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:30.571 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:30.596 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:30.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:30.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:30.698 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:30.722 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:30.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:30.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:30.789 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 19 16 35766 35750 117.572 100.5 0.0139889 0.0084985 2024-08-17T17:19:30.824 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:30.848 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:30.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:30.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:30.950 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:30.977 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:30.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:30.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:31.079 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:31.106 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:31.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:31.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:31.208 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:31.235 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:31.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:31.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:31.337 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:31.362 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:31.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:31.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:31.464 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:31.490 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:31.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:31.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:31.592 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:31.615 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:31.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:31.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:31.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:31 smithi053 ceph-mon[89963]: pgmap v242: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.9 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 112 MiB/s wr, 1.79k op/s; 1632/94257 objects misplaced (1.731%); 112 KiB/s, 1 objects/s recovering 2024-08-17T17:19:31.717 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:31.758 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:31.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:31.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:31.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:31 smithi003 ceph-mon[100615]: pgmap v242: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.9 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 112 MiB/s wr, 1.79k op/s; 1632/94257 objects misplaced (1.731%); 112 KiB/s, 1 objects/s recovering 2024-08-17T17:19:31.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:31 smithi003 ceph-mon[103344]: pgmap v242: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 1.9 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 112 MiB/s wr, 1.79k op/s; 1632/94257 objects misplaced (1.731%); 112 KiB/s, 1 objects/s recovering 2024-08-17T17:19:31.789 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-08-17T17:19:31.789733+0000 min lat: 0.0011526 max lat: 2.93219 avg lat: 0.00858369 2024-08-17T17:19:31.789 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:19:31.789 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 20 16 37278 37262 116.418 94.5 0.00346744 0.00858369 2024-08-17T17:19:31.860 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:31.902 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:31.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:31.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:32.004 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:32.032 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:32.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:32.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:32.134 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:32.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:32.176 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:32.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:32.182 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:19:32.183 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:19:32.221 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:19:32.221 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:19:32.263 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:19:32.263 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:19:32.277 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:32.297 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:19:32.298 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:19:32.315 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:32.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:32.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:32.328 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:19:32.328 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:19:32.360 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:19:32.361 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:19:32.396 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:19:32.396 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:19:32.417 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:32.433 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:19:32.434 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:19:32.459 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:32.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:32.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:32.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:32 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3071274590' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:19:32.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:32 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3071274590' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:19:32.561 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:32.589 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:32.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:32.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:32.690 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:32.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:32 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3071274590' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:19:32.731 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:32.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:32.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:32.789 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 21 16 38806 38790 115.422 95.5 0.00429072 0.0086574 2024-08-17T17:19:32.833 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:32.875 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:32.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:32.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:32.977 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:33.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:33.026 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:33.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:33.127 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:33.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:33.155 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:33.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:33.256 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:33.299 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:33.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:33.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:33.401 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:33.441 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:33.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:33.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:33.482 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:19:33.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:33 smithi003 ceph-mon[103344]: pgmap v243: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 2.0 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 97 MiB/s wr, 1.56k op/s; 1263/97626 objects misplaced (1.294%); 2.8 MiB/s, 45 objects/s recovering 2024-08-17T17:19:33.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:33 smithi003 ceph-mon[100615]: pgmap v243: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 2.0 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 97 MiB/s wr, 1.56k op/s; 1263/97626 objects misplaced (1.294%); 2.8 MiB/s, 45 objects/s recovering 2024-08-17T17:19:33.542 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:33.590 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:33.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:33.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:33.617 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:33 smithi053 ceph-mon[89963]: pgmap v243: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 2.0 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 97 MiB/s wr, 1.56k op/s; 1263/97626 objects misplaced (1.294%); 2.8 MiB/s, 45 objects/s recovering 2024-08-17T17:19:33.693 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:33.735 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:33.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:33.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:33.789 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 22 16 40377 40361 114.638 98.1875 0.00775554 0.00871682 2024-08-17T17:19:33.837 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:33.864 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:33.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:33.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:33.966 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:34.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:34.013 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:34.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:34.114 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:34.155 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:34.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:34.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:34.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:34.299 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:34.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:34.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:34.401 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:34.440 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:34.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:34.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:34.542 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:34.572 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:34.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:34.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:34.674 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:34.702 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:34.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:34.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:34.790 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 23 16 41951 41935 113.931 98.375 0.00547225 0.0087719 2024-08-17T17:19:34.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:34.849 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:34.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:34.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:34.951 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:34.998 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:34.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:34.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:35.100 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:35.136 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:35.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:35.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:35.238 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:35.266 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:35.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:35.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:35.368 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:35.407 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:35.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:35.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:35.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:35 smithi053 ceph-mon[89963]: pgmap v244: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 2.3 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s; 1263/115041 objects misplaced (1.098%); 2.4 MiB/s, 38 objects/s recovering 2024-08-17T17:19:35.510 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:35.548 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:35.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:35.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:35.649 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:35.651 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:35 smithi003 ceph-mon[103344]: pgmap v244: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 2.3 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s; 1263/115041 objects misplaced (1.098%); 2.4 MiB/s, 38 objects/s recovering 2024-08-17T17:19:35.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:35 smithi003 ceph-mon[100615]: pgmap v244: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 2.3 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s; 1263/115041 objects misplaced (1.098%); 2.4 MiB/s, 38 objects/s recovering 2024-08-17T17:19:35.677 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:35.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:35.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:35.779 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:35.790 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 24 16 43481 43465 113.167 95.625 0.0102066 0.00882952 2024-08-17T17:19:35.822 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:35.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:35.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:35.925 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:35.951 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:35.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:35.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:35.985 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:19:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:19:35] "GET /metrics HTTP/1.1" 200 36502 "" "Prometheus/2.51.0" 2024-08-17T17:19:36.053 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:36.080 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:36.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:36.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:36.182 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:36.226 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:36.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:36.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:36.328 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:36.354 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:36.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:36.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:36.458 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:36.500 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:36.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:36.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:36.602 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:36.626 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:36.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:36.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:36.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:36.769 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:36.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:36.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:36.790 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 25 16 44873 44857 112.12 87 0.00813951 0.00891355 2024-08-17T17:19:36.871 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:36.897 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:36.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:36.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:36.999 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:37.042 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:37.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:37.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:37.144 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:37.181 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:37.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:37.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:37.283 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:37.324 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:37.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:37.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:37.367 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:19:37.427 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:37.453 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:37.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:37.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:37.555 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:37.580 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:37.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:37.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:37.682 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:37.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:37 smithi053 ceph-mon[89963]: pgmap v245: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 2.5 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 85 MiB/s wr, 1.35k op/s; 889/120855 objects misplaced (0.736%); 4.1 MiB/s, 66 objects/s recovering 2024-08-17T17:19:37.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:37 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1087624381' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:19:37.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:37 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:19:37.730 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:37.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:37.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:37.777 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:37 smithi003 ceph-mon[100615]: pgmap v245: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 2.5 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 85 MiB/s wr, 1.35k op/s; 889/120855 objects misplaced (0.736%); 4.1 MiB/s, 66 objects/s recovering 2024-08-17T17:19:37.777 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:37 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1087624381' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:19:37.777 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:37 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:19:37.777 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:37 smithi003 ceph-mon[103344]: pgmap v245: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 2.5 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 85 MiB/s wr, 1.35k op/s; 889/120855 objects misplaced (0.736%); 4.1 MiB/s, 66 objects/s recovering 2024-08-17T17:19:37.777 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:37 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1087624381' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:19:37.777 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:37 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:19:37.790 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 26 16 46077 46061 110.7 75.25 0.00403673 0.00902575 2024-08-17T17:19:37.832 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:37.859 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:37.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:37.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:37.960 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:37.985 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:37.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:37.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:38.087 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:38.113 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:38.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:38.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:38.215 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:38.258 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:38.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:38.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:38.360 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:38.385 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:38.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:38.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:38.487 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:38.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:38.535 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:38.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:38.636 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:38.662 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:38.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:38.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:38.680 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-08-17T17:19:38.680 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:38 smithi003 ceph-mon[103344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:19:38.680 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:38 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:19:38.680 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:38 smithi003 ceph-mon[103344]: osdmap e124: 8 total, 8 up, 8 in 2024-08-17T17:19:38.680 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 5]} : dispatch 2024-08-17T17:19:38.680 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [4, 7, 6, 2]} : dispatch 2024-08-17T17:19:38.681 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 7]} : dispatch 2024-08-17T17:19:38.681 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 7]} : dispatch 2024-08-17T17:19:38.681 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2, 3, 7]} : dispatch 2024-08-17T17:19:38.681 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:19:38.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:38 smithi003 ceph-mon[100615]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:19:38.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:38 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:19:38.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:38 smithi003 ceph-mon[100615]: osdmap e124: 8 total, 8 up, 8 in 2024-08-17T17:19:38.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 5]} : dispatch 2024-08-17T17:19:38.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [4, 7, 6, 2]} : dispatch 2024-08-17T17:19:38.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 7]} : dispatch 2024-08-17T17:19:38.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 7]} : dispatch 2024-08-17T17:19:38.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2, 3, 7]} : dispatch 2024-08-17T17:19:38.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:19:38.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:38 smithi053 ceph-mon[89963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:19:38.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:38 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:19:38.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:38 smithi053 ceph-mon[89963]: osdmap e124: 8 total, 8 up, 8 in 2024-08-17T17:19:38.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 5]} : dispatch 2024-08-17T17:19:38.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [4, 7, 6, 2]} : dispatch 2024-08-17T17:19:38.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 7]} : dispatch 2024-08-17T17:19:38.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 7]} : dispatch 2024-08-17T17:19:38.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2, 3, 7]} : dispatch 2024-08-17T17:19:38.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:19:38.763 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:38.791 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 27 16 47503 47487 109.901 89.125 0.0121959 0.00909346 2024-08-17T17:19:38.807 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:38.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:38.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:38.911 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:38.953 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:38.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:38.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:39.032 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:19:39.032 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:19:39.054 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:39.097 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:39.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:39.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:39.199 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:39.224 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:39.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:39.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:39.326 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:39.366 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:39.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:39.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:39.467 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:39.507 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:39.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:39.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:39.608 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:39.625 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:39 smithi003 ceph-mon[103344]: pgmap v247: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 2.5 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 90 MiB/s wr, 1.43k op/s; 889/123750 objects misplaced (0.718%); 4.6 MiB/s, 74 objects/s recovering 2024-08-17T17:19:39.626 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 5]}]': finished 2024-08-17T17:19:39.626 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [4, 7, 6, 2]}]': finished 2024-08-17T17:19:39.626 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 7]}]': finished 2024-08-17T17:19:39.626 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 7]}]': finished 2024-08-17T17:19:39.626 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2, 3, 7]}]': finished 2024-08-17T17:19:39.626 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:39 smithi003 ceph-mon[103344]: osdmap e125: 8 total, 8 up, 8 in 2024-08-17T17:19:39.626 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:39 smithi003 ceph-mon[100615]: pgmap v247: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 2.5 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 90 MiB/s wr, 1.43k op/s; 889/123750 objects misplaced (0.718%); 4.6 MiB/s, 74 objects/s recovering 2024-08-17T17:19:39.626 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 5]}]': finished 2024-08-17T17:19:39.626 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [4, 7, 6, 2]}]': finished 2024-08-17T17:19:39.626 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 7]}]': finished 2024-08-17T17:19:39.627 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 7]}]': finished 2024-08-17T17:19:39.627 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2, 3, 7]}]': finished 2024-08-17T17:19:39.627 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:39 smithi003 ceph-mon[100615]: osdmap e125: 8 total, 8 up, 8 in 2024-08-17T17:19:39.668 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:39.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:39.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:39.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:39 smithi053 ceph-mon[89963]: pgmap v247: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 2.5 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 90 MiB/s wr, 1.43k op/s; 889/123750 objects misplaced (0.718%); 4.6 MiB/s, 74 objects/s recovering 2024-08-17T17:19:39.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 5]}]': finished 2024-08-17T17:19:39.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [4, 7, 6, 2]}]': finished 2024-08-17T17:19:39.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 7]}]': finished 2024-08-17T17:19:39.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 7]}]': finished 2024-08-17T17:19:39.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2, 3, 7]}]': finished 2024-08-17T17:19:39.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:39 smithi053 ceph-mon[89963]: osdmap e125: 8 total, 8 up, 8 in 2024-08-17T17:19:39.770 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:39.790 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 28 16 48943 48927 109.19 90 0.0233642 0.00915298 2024-08-17T17:19:39.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:39.798 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:39.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:39.899 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:39.927 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:39.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:39.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:40.029 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:40.064 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-17T17:19:40.084 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:40.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:40.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:40.185 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:40.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:40.238 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:40.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:40.339 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:40.364 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:40.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:40.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:40.465 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:40.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:40 smithi053 ceph-mon[89963]: osdmap e126: 8 total, 8 up, 8 in 2024-08-17T17:19:40.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:19:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:19:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:19:40.524 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:40.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:40.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:40.626 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:40.667 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:40.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:40.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:40.734 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:40 smithi003 ceph-mon[100615]: osdmap e126: 8 total, 8 up, 8 in 2024-08-17T17:19:40.735 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:40 smithi003 ceph-mon[103344]: osdmap e126: 8 total, 8 up, 8 in 2024-08-17T17:19:40.769 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:40.799 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 29 16 50302 50286 108.321 84.9375 0.0190547 0.00922309 2024-08-17T17:19:40.810 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:40.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:40.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:40.912 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:40.954 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:40.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:40.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:41.056 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:41.061 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:19:41.080 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:41.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:41.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:41.183 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:41.229 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:41.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:41.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:41.331 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:41.362 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:41.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:41.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:41.430 INFO:teuthology.orchestra.run.smithi003.stdout:moved 10 / 267 (3.74532%) 2024-08-17T17:19:41.430 INFO:teuthology.orchestra.run.smithi003.stdout:avg 33.375 2024-08-17T17:19:41.430 INFO:teuthology.orchestra.run.smithi003.stdout:stddev 11.521 -> 9.68165 (expected baseline 5.40399) 2024-08-17T17:19:41.430 INFO:teuthology.orchestra.run.smithi003.stdout:min osd.7 with 20 -> 23 pgs (0.599251 -> 0.689139 * mean) 2024-08-17T17:19:41.430 INFO:teuthology.orchestra.run.smithi003.stdout:max osd.1 with 49 -> 47 pgs (1.46816 -> 1.40824 * mean) 2024-08-17T17:19:41.430 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:19:41.430 INFO:teuthology.orchestra.run.smithi003.stdout:oload 110 2024-08-17T17:19:41.430 INFO:teuthology.orchestra.run.smithi003.stdout:max_change 0.05 2024-08-17T17:19:41.430 INFO:teuthology.orchestra.run.smithi003.stdout:max_change_osds 4 2024-08-17T17:19:41.431 INFO:teuthology.orchestra.run.smithi003.stdout:average_utilization 0.0137 2024-08-17T17:19:41.431 INFO:teuthology.orchestra.run.smithi003.stdout:overload_utilization 0.0151 2024-08-17T17:19:41.431 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0 weight 0.9500 -> 0.9000 2024-08-17T17:19:41.431 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7 weight 0.5965 -> 0.6464 2024-08-17T17:19:41.431 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2 weight 1.0000 -> 0.9500 2024-08-17T17:19:41.431 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6 weight 0.5465 -> 0.5965 2024-08-17T17:19:41.431 INFO:teuthology.orchestra.run.smithi003.stderr:no change 2024-08-17T17:19:41.464 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:41.491 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:41.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:41.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:41.593 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:41.635 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:41 smithi003 ceph-mon[103344]: pgmap v250: 89 pgs: 4 peering, 1 active+recovering+undersized+remapped, 84 active+clean; 2.8 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s; 889/139767 objects misplaced (0.636%); 3.9 MiB/s, 62 objects/s recovering 2024-08-17T17:19:41.636 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:41 smithi003 ceph-mon[100615]: pgmap v250: 89 pgs: 4 peering, 1 active+recovering+undersized+remapped, 84 active+clean; 2.8 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s; 889/139767 objects misplaced (0.636%); 3.9 MiB/s, 62 objects/s recovering 2024-08-17T17:19:41.636 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:19:41 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:19:41.428+0000 7f288dc05640 -1 mgr.server reply reply (4) Interrupted system call no change 2024-08-17T17:19:41.643 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:41.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:41.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:41.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:41 smithi053 ceph-mon[89963]: pgmap v250: 89 pgs: 4 peering, 1 active+recovering+undersized+remapped, 84 active+clean; 2.8 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s; 889/139767 objects misplaced (0.636%); 3.9 MiB/s, 62 objects/s recovering 2024-08-17T17:19:41.746 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:41.770 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:41.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:41.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:41.799 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 30 16 51560 51544 107.331 78.625 0.0105699 0.00931165 2024-08-17T17:19:41.873 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:41.902 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:41.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:41.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:42.004 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:42.030 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:42.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:42.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:42.132 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:42.173 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:42.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:42.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:42.263 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-08-17T17:19:42.263 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight-by-utilization 110 1.0 2024-08-17T17:19:42.273 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:19:42.274 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:42.297 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:42.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:42.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:42.400 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:42.425 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:42.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:42.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:42.493 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:42 smithi003 ceph-mon[100615]: from='client.54784 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:19:42.494 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:42 smithi003 ceph-mon[103344]: from='client.54784 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:19:42.506 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:19:42.526 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:42.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:42.554 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:42.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:42.655 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:42.681 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:42.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:42.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:42.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:42 smithi053 ceph-mon[89963]: from='client.54784 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:19:42.782 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:42.800 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 31 16 52862 52846 106.493 81.375 0.0106405 0.00938488 2024-08-17T17:19:42.827 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:42.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:42.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:42.930 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:42.956 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:42.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:42.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:43.058 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:43.103 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:43.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:43.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:43.205 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:43.230 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:43.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:43.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:43.332 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:43.374 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:43.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:43.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:43.476 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:43.517 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:43.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:43.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:43.619 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:43.645 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:43.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:43.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:43.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:43 smithi053 ceph-mon[89963]: pgmap v251: 89 pgs: 1 active+recovering+undersized+remapped, 4 peering, 84 active+clean; 2.9 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s; 498/142284 objects misplaced (0.350%); 4.1 MiB/s, 65 objects/s recovering 2024-08-17T17:19:43.747 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:43.772 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:43.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:43 smithi003 ceph-mon[103344]: pgmap v251: 89 pgs: 1 active+recovering+undersized+remapped, 4 peering, 84 active+clean; 2.9 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s; 498/142284 objects misplaced (0.350%); 4.1 MiB/s, 65 objects/s recovering 2024-08-17T17:19:43.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:43 smithi003 ceph-mon[100615]: pgmap v251: 89 pgs: 1 active+recovering+undersized+remapped, 4 peering, 84 active+clean; 2.9 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s; 498/142284 objects misplaced (0.350%); 4.1 MiB/s, 65 objects/s recovering 2024-08-17T17:19:43.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:43.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:43.800 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 32 16 54223 54207 105.823 85.0625 0.0102849 0.00944449 2024-08-17T17:19:43.874 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:43.898 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:43.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:43.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:44.000 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:44.045 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:44.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:44.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:44.147 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:44.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:44.186 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:44.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:44.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:44.331 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:44.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:44.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:44.372 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:19:44.396 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:19:44.433 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:44.492 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:44.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:44.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:44.594 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:44.638 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:44.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:44.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:44.740 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:44.783 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:44.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:44.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:44.800 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 33 16 55470 55454 104.978 77.9375 0.0113886 0.00951959 2024-08-17T17:19:44.886 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:44.926 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:44.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:44.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:45.028 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:45.066 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:45.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:45.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:45.168 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:45.192 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:45.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:45.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:45.294 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:45.318 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:45.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:45.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:45.420 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:45.444 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:45.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:45.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:45.546 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:45.588 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:45.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:45.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:45.690 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:45.694 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:45 smithi003 ceph-mon[103344]: pgmap v252: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 120 MiB/s wr, 1.92k op/s; 498/157590 objects misplaced (0.316%); 3.8 MiB/s, 61 objects/s recovering 2024-08-17T17:19:45.694 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:45 smithi003 ceph-mon[100615]: pgmap v252: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 120 MiB/s wr, 1.92k op/s; 498/157590 objects misplaced (0.316%); 3.8 MiB/s, 61 objects/s recovering 2024-08-17T17:19:45.694 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:19:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:19:45] "GET /metrics HTTP/1.1" 200 36517 "" "Prometheus/2.51.0" 2024-08-17T17:19:45.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:45 smithi053 ceph-mon[89963]: pgmap v252: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 120 MiB/s wr, 1.92k op/s; 498/157590 objects misplaced (0.316%); 3.8 MiB/s, 61 objects/s recovering 2024-08-17T17:19:45.726 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:45.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:45.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:45.800 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 34 16 56553 56537 103.881 67.6875 0.016474 0.00961979 2024-08-17T17:19:45.828 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:45.854 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:45.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:45.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:45.956 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:46.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:46.006 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:46.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:46.107 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:46.133 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:46.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:46.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:46.235 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:46.275 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:46.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:46.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:46.374 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:19:46.374 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:19:46.377 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:46.403 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:46.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:46.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:46.413 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:19:46.413 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:19:46.421 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:46 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/39165365' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:19:46.460 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:19:46.461 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:19:46.496 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:19:46.496 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:19:46.504 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:46.522 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:19:46.522 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:19:46.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:46.531 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:46.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:46.551 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:19:46.551 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:19:46.577 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:19:46.577 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:19:46.600 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:19:46.600 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:19:46.631 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:46.657 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:46.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:46.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:46.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:46 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/39165365' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:19:46.759 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:46.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:46 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/39165365' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:19:46.780 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:46.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:46.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:46.800 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 35 16 57451 57435 102.517 56.125 0.00248446 0.00966773 2024-08-17T17:19:46.882 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:46.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:46.905 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:46.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:47.006 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:47.030 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:47.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:47.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:47.131 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:47.155 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:47.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:47.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:47.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:47.282 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:47.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:47.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:47.384 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:47.411 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:47.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:47.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:47.513 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:47.543 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:47.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:47.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:47.645 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:47.667 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:47.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:47.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:47.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:47 smithi003 ceph-mon[103344]: pgmap v253: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 101 MiB/s wr, 1.61k op/s; 26/162369 objects misplaced (0.016%); 6.7 MiB/s, 108 objects/s recovering 2024-08-17T17:19:47.687 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:47 smithi003 ceph-mon[103344]: osdmap e127: 8 total, 8 up, 8 in 2024-08-17T17:19:47.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:47 smithi003 ceph-mon[100615]: pgmap v253: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 101 MiB/s wr, 1.61k op/s; 26/162369 objects misplaced (0.016%); 6.7 MiB/s, 108 objects/s recovering 2024-08-17T17:19:47.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:47 smithi003 ceph-mon[100615]: osdmap e127: 8 total, 8 up, 8 in 2024-08-17T17:19:47.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:47 smithi053 ceph-mon[89963]: pgmap v253: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 101 MiB/s wr, 1.61k op/s; 26/162369 objects misplaced (0.016%); 6.7 MiB/s, 108 objects/s recovering 2024-08-17T17:19:47.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:47 smithi053 ceph-mon[89963]: osdmap e127: 8 total, 8 up, 8 in 2024-08-17T17:19:47.769 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:47.793 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:47.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:47.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:47.800 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 36 16 57858 57842 100.377 25.4375 0.0101839 0.00995617 2024-08-17T17:19:47.896 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:47.921 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:47.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:47.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:48.023 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:48.050 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:48.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:48.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:48.152 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:48.177 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:48.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:48.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:48.279 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:48.302 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:48.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:48.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:48.404 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:48.427 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:48.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:48.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:48.529 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:48.570 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:48.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:48.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:48.602 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:19:48.602 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-08-17T17:19:48.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-08-17T17:19:48.602 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:19:48.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.603 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:14450 5.7 5:e4bdd575:::benchmark_data_smithi003_140793_object14449:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-17T17:19:48.603 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:16.811807+0000", 2024-08-17T17:19:48.603 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 30.870520498000001, 2024-08-17T17:19:48.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.042708128999999997, 2024-08-17T17:19:48.603 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:48.603 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:48.603 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:48.603 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:19:48.603 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:19:48.603 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:19:48.603 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 14450 2024-08-17T17:19:48.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.603 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:48.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:48.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.811807+0000", 2024-08-17T17:19:48.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:48.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.811807+0000", 2024-08-17T17:19:48.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:48.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.811809+0000", 2024-08-17T17:19:48.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5400000000000001e-06 2024-08-17T17:19:48.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.605 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:48.605 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.811841+0000", 2024-08-17T17:19:48.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2521000000000003e-05 2024-08-17T17:19:48.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.605 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:48.605 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.811853+0000", 2024-08-17T17:19:48.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1729000000000001e-05 2024-08-17T17:19:48.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.605 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:48.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.811860+0000", 2024-08-17T17:19:48.606 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0230000000000004e-06 2024-08-17T17:19:48.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:48.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.812115+0000", 2024-08-17T17:19:48.606 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00025542400000000003 2024-08-17T17:19:48.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:48.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.812229+0000", 2024-08-17T17:19:48.606 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011356700000000001 2024-08-17T17:19:48.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.607 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4", 2024-08-17T17:19:48.607 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.812364+0000", 2024-08-17T17:19:48.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013543 2024-08-17T17:19:48.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.607 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.607 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.813991+0000", 2024-08-17T17:19:48.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016268089999999999 2024-08-17T17:19:48.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.607 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.607 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.816506+0000", 2024-08-17T17:19:48.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025149479999999999 2024-08-17T17:19:48.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.608 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:19:48.608 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.818314+0000", 2024-08-17T17:19:48.608 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0018084799999999999 2024-08-17T17:19:48.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.608 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:48.608 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.818373+0000", 2024-08-17T17:19:48.608 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8999000000000002e-05 2024-08-17T17:19:48.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.608 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:48.609 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.854515+0000", 2024-08-17T17:19:48.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.036141659 2024-08-17T17:19:48.609 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.609 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:48.609 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.609 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:14453 5.7 5:eb283cbb:::benchmark_data_smithi003_140793_object14452:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-17T17:19:48.609 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:16.814129+0000", 2024-08-17T17:19:48.609 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 30.868198512999999, 2024-08-17T17:19:48.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.040560506000000003, 2024-08-17T17:19:48.609 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:48.609 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:48.609 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:48.610 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:19:48.610 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:19:48.610 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:19:48.610 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 14453 2024-08-17T17:19:48.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.610 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:48.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.610 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:48.610 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.814129+0000", 2024-08-17T17:19:48.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.610 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:48.610 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.814129+0000", 2024-08-17T17:19:48.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:48.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.814130+0000", 2024-08-17T17:19:48.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.266e-06 2024-08-17T17:19:48.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:48.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.814157+0000", 2024-08-17T17:19:48.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6543999999999999e-05 2024-08-17T17:19:48.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:48.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.814168+0000", 2024-08-17T17:19:48.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0903e-05 2024-08-17T17:19:48.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:48.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.814174+0000", 2024-08-17T17:19:48.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5899999999999996e-06 2024-08-17T17:19:48.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:48.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.814320+0000", 2024-08-17T17:19:48.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014548400000000001 2024-08-17T17:19:48.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.613 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:48.613 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.814459+0000", 2024-08-17T17:19:48.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013959700000000001 2024-08-17T17:19:48.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.613 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4", 2024-08-17T17:19:48.613 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.814607+0000", 2024-08-17T17:19:48.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014753999999999999 2024-08-17T17:19:48.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.613 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.613 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.816315+0000", 2024-08-17T17:19:48.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001707814 2024-08-17T17:19:48.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.614 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.614 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.817425+0000", 2024-08-17T17:19:48.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0011107510000000001 2024-08-17T17:19:48.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.614 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:19:48.614 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.854579+0000", 2024-08-17T17:19:48.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.037153835000000003 2024-08-17T17:19:48.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.614 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:48.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.854644+0000", 2024-08-17T17:19:48.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4592999999999996e-05 2024-08-17T17:19:48.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:48.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.854689+0000", 2024-08-17T17:19:48.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5589e-05 2024-08-17T17:19:48.615 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.615 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:48.615 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.615 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:14459 5.7 5:e0ae7e48:::benchmark_data_smithi003_140793_object14458:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-17T17:19:48.615 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:16.817345+0000", 2024-08-17T17:19:48.616 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 30.864982521999998, 2024-08-17T17:19:48.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.039378542000000002, 2024-08-17T17:19:48.616 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:48.616 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:48.616 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:48.616 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:19:48.616 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:19:48.616 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:19:48.616 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 14459 2024-08-17T17:19:48.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.616 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:48.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.616 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:48.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.817345+0000", 2024-08-17T17:19:48.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:48.617 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.817345+0000", 2024-08-17T17:19:48.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:48.617 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.817346+0000", 2024-08-17T17:19:48.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5370000000000001e-06 2024-08-17T17:19:48.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:48.618 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.817369+0000", 2024-08-17T17:19:48.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2711999999999999e-05 2024-08-17T17:19:48.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.618 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:48.618 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.817380+0000", 2024-08-17T17:19:48.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0939e-05 2024-08-17T17:19:48.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.618 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:48.618 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.817388+0000", 2024-08-17T17:19:48.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4569999999999999e-06 2024-08-17T17:19:48.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:48.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.817477+0000", 2024-08-17T17:19:48.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8924000000000004e-05 2024-08-17T17:19:48.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:48.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.817601+0000", 2024-08-17T17:19:48.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012412 2024-08-17T17:19:48.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4", 2024-08-17T17:19:48.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.817727+0000", 2024-08-17T17:19:48.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012608500000000001 2024-08-17T17:19:48.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.854565+0000", 2024-08-17T17:19:48.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.036838329000000003 2024-08-17T17:19:48.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:19:48.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.854698+0000", 2024-08-17T17:19:48.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013297800000000001 2024-08-17T17:19:48.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.856592+0000", 2024-08-17T17:19:48.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001894284 2024-08-17T17:19:48.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:48.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.856659+0000", 2024-08-17T17:19:48.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6681999999999996e-05 2024-08-17T17:19:48.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:48.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.856723+0000", 2024-08-17T17:19:48.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4494999999999998e-05 2024-08-17T17:19:48.621 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.622 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:48.622 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.622 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.35277.0:14459 5.7 e119/114 ondisk, result = 0)", 2024-08-17T17:19:48.622 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:16.818507+0000", 2024-08-17T17:19:48.622 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 30.863820094000001, 2024-08-17T17:19:48.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.036191512000000002, 2024-08-17T17:19:48.622 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:48.623 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:48.623 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:19:48.623 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:48.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.623 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:48.623 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.818507+0000", 2024-08-17T17:19:48.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.623 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:48.623 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.818507+0000", 2024-08-17T17:19:48.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:48.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.818508+0000", 2024-08-17T17:19:48.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7499999999999999e-07 2024-08-17T17:19:48.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:48.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.818513+0000", 2024-08-17T17:19:48.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4719999999999997e-06 2024-08-17T17:19:48.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:48.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.854385+0000", 2024-08-17T17:19:48.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.035872210000000002 2024-08-17T17:19:48.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.625 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:48.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.854403+0000", 2024-08-17T17:19:48.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7688000000000001e-05 2024-08-17T17:19:48.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.625 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:48.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.854549+0000", 2024-08-17T17:19:48.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000145938 2024-08-17T17:19:48.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:48.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.854558+0000", 2024-08-17T17:19:48.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4499999999999993e-06 2024-08-17T17:19:48.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:48.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:16.854699+0000", 2024-08-17T17:19:48.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000140879 2024-08-17T17:19:48.626 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.626 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:48.626 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.627 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:15228 5.7 5:e6be6871:::benchmark_data_smithi003_140793_object15227:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-17T17:19:48.627 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:17.237629+0000", 2024-08-17T17:19:48.627 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 30.444698523, 2024-08-17T17:19:48.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.028836343, 2024-08-17T17:19:48.627 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:48.627 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:48.627 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:48.627 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:19:48.627 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:19:48.627 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:19:48.627 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 15228 2024-08-17T17:19:48.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.627 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:48.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:48.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.237629+0000", 2024-08-17T17:19:48.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:48.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.237629+0000", 2024-08-17T17:19:48.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:48.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.237630+0000", 2024-08-17T17:19:48.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7700000000000004e-07 2024-08-17T17:19:48.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:48.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.237652+0000", 2024-08-17T17:19:48.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2853e-05 2024-08-17T17:19:48.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:48.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.237661+0000", 2024-08-17T17:19:48.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5709999999999998e-06 2024-08-17T17:19:48.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:48.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.237667+0000", 2024-08-17T17:19:48.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3779999999999998e-06 2024-08-17T17:19:48.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:48.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.239230+0000", 2024-08-17T17:19:48.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001562854 2024-08-17T17:19:48.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:48.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.239348+0000", 2024-08-17T17:19:48.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011770299999999999 2024-08-17T17:19:48.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4", 2024-08-17T17:19:48.631 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.243561+0000", 2024-08-17T17:19:48.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042126439999999998 2024-08-17T17:19:48.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.631 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.631 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.244838+0000", 2024-08-17T17:19:48.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0012775200000000001 2024-08-17T17:19:48.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.631 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:19:48.631 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.252381+0000", 2024-08-17T17:19:48.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00754303 2024-08-17T17:19:48.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.266368+0000", 2024-08-17T17:19:48.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013986887999999999 2024-08-17T17:19:48.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:48.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.266421+0000", 2024-08-17T17:19:48.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2648999999999997e-05 2024-08-17T17:19:48.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:48.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.266465+0000", 2024-08-17T17:19:48.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4675999999999997e-05 2024-08-17T17:19:48.633 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.633 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:48.633 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.633 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:15356 5.7 5:e53e48d1:::benchmark_data_smithi003_140793_object15355:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-08-17T17:19:48.633 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:17.328012+0000", 2024-08-17T17:19:48.633 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 30.354315801999999, 2024-08-17T17:19:48.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.026976709000000001, 2024-08-17T17:19:48.633 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:48.633 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:48.633 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:48.633 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:19:48.633 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:19:48.634 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:19:48.634 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 15356 2024-08-17T17:19:48.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.634 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:48.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:48.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.328012+0000", 2024-08-17T17:19:48.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:48.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.328012+0000", 2024-08-17T17:19:48.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:48.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.328013+0000", 2024-08-17T17:19:48.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.641e-06 2024-08-17T17:19:48.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:48.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.328041+0000", 2024-08-17T17:19:48.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7416e-05 2024-08-17T17:19:48.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:48.636 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.328051+0000", 2024-08-17T17:19:48.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0628999999999999e-05 2024-08-17T17:19:48.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.636 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:48.636 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.328058+0000", 2024-08-17T17:19:48.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4230000000000002e-06 2024-08-17T17:19:48.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.636 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:48.636 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.328141+0000", 2024-08-17T17:19:48.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3038e-05 2024-08-17T17:19:48.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:48.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.328277+0000", 2024-08-17T17:19:48.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013608899999999999 2024-08-17T17:19:48.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4", 2024-08-17T17:19:48.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.328417+0000", 2024-08-17T17:19:48.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000139636 2024-08-17T17:19:48.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.330280+0000", 2024-08-17T17:19:48.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00186363 2024-08-17T17:19:48.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:19:48.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.331684+0000", 2024-08-17T17:19:48.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014035300000000001 2024-08-17T17:19:48.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.354853+0000", 2024-08-17T17:19:48.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.023169428999999998 2024-08-17T17:19:48.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:48.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.354901+0000", 2024-08-17T17:19:48.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7579e-05 2024-08-17T17:19:48.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:48.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:17.354988+0000", 2024-08-17T17:19:48.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7669000000000004e-05 2024-08-17T17:19:48.639 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.639 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:48.639 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.639 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:22174 5.c 5:346f5b84:::benchmark_data_smithi003_140793_object22173:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-08-17T17:19:48.640 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:20.410238+0000", 2024-08-17T17:19:48.640 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.272089888, 2024-08-17T17:19:48.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.051141907, 2024-08-17T17:19:48.640 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:48.640 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:48.640 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:48.640 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:19:48.640 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:19:48.640 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:19:48.640 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 22174 2024-08-17T17:19:48.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.640 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:48.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:48.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.410238+0000", 2024-08-17T17:19:48.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:48.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.410238+0000", 2024-08-17T17:19:48.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:48.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.410240+0000", 2024-08-17T17:19:48.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9400000000000001e-06 2024-08-17T17:19:48.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:48.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.410272+0000", 2024-08-17T17:19:48.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2011000000000002e-05 2024-08-17T17:19:48.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:48.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.410283+0000", 2024-08-17T17:19:48.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1063999999999999e-05 2024-08-17T17:19:48.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:48.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.410289+0000", 2024-08-17T17:19:48.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6189999999999996e-06 2024-08-17T17:19:48.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:48.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.410541+0000", 2024-08-17T17:19:48.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00025191200000000002 2024-08-17T17:19:48.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:48.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.410646+0000", 2024-08-17T17:19:48.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010506 2024-08-17T17:19:48.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4", 2024-08-17T17:19:48.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.410761+0000", 2024-08-17T17:19:48.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011504 2024-08-17T17:19:48.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.413129+0000", 2024-08-17T17:19:48.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0023681150000000001 2024-08-17T17:19:48.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:19:48.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.414692+0000", 2024-08-17T17:19:48.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001562353 2024-08-17T17:19:48.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.461264+0000", 2024-08-17T17:19:48.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.046572077000000003 2024-08-17T17:19:48.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:48.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.461329+0000", 2024-08-17T17:19:48.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4733999999999994e-05 2024-08-17T17:19:48.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:48.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.461379+0000", 2024-08-17T17:19:48.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0982000000000001e-05 2024-08-17T17:19:48.646 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.646 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:48.646 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.646 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:22176 5.c 5:3131c163:::benchmark_data_smithi003_140793_object22175:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-08-17T17:19:48.647 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:20.410302+0000", 2024-08-17T17:19:48.647 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.272025202999998, 2024-08-17T17:19:48.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.051205428999999997, 2024-08-17T17:19:48.647 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:48.647 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:48.647 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:48.647 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:19:48.647 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:19:48.647 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:19:48.647 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 22176 2024-08-17T17:19:48.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.647 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:48.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:48.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.410302+0000", 2024-08-17T17:19:48.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:48.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.410302+0000", 2024-08-17T17:19:48.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:48.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.410303+0000", 2024-08-17T17:19:48.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9800000000000003e-07 2024-08-17T17:19:48.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:48.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.410327+0000", 2024-08-17T17:19:48.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3756000000000001e-05 2024-08-17T17:19:48.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:48.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.410335+0000", 2024-08-17T17:19:48.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7409999999999998e-06 2024-08-17T17:19:48.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:48.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.410342+0000", 2024-08-17T17:19:48.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1530000000000002e-06 2024-08-17T17:19:48.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:48.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.411761+0000", 2024-08-17T17:19:48.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001419795 2024-08-17T17:19:48.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:48.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.411937+0000", 2024-08-17T17:19:48.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017530399999999999 2024-08-17T17:19:48.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4", 2024-08-17T17:19:48.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.412299+0000", 2024-08-17T17:19:48.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00036262499999999999 2024-08-17T17:19:48.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.414670+0000", 2024-08-17T17:19:48.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0023703069999999999 2024-08-17T17:19:48.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:19:48.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.414699+0000", 2024-08-17T17:19:48.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8957000000000001e-05 2024-08-17T17:19:48.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.461419+0000", 2024-08-17T17:19:48.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.046720708 2024-08-17T17:19:48.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:48.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.461472+0000", 2024-08-17T17:19:48.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2890999999999999e-05 2024-08-17T17:19:48.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:48.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.461508+0000", 2024-08-17T17:19:48.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5593999999999998e-05 2024-08-17T17:19:48.653 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.653 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:48.653 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.653 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:22180 5.7 5:e1f0324f:::benchmark_data_smithi003_140793_object22179:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-08-17T17:19:48.653 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:20.411987+0000", 2024-08-17T17:19:48.653 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.270340251, 2024-08-17T17:19:48.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.049409092000000002, 2024-08-17T17:19:48.654 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:48.654 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:48.654 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:48.654 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:19:48.654 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:19:48.654 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:19:48.654 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 22180 2024-08-17T17:19:48.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.654 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:48.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:48.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.411987+0000", 2024-08-17T17:19:48.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:48.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.411987+0000", 2024-08-17T17:19:48.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:48.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.411988+0000", 2024-08-17T17:19:48.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4300000000000001e-07 2024-08-17T17:19:48.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:48.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.412010+0000", 2024-08-17T17:19:48.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2056e-05 2024-08-17T17:19:48.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:48.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.412141+0000", 2024-08-17T17:19:48.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013041600000000001 2024-08-17T17:19:48.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:48.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.412150+0000", 2024-08-17T17:19:48.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5799999999999998e-06 2024-08-17T17:19:48.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:48.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.412996+0000", 2024-08-17T17:19:48.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00084556300000000002 2024-08-17T17:19:48.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:48.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.413114+0000", 2024-08-17T17:19:48.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000117737 2024-08-17T17:19:48.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4", 2024-08-17T17:19:48.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.413251+0000", 2024-08-17T17:19:48.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000137494 2024-08-17T17:19:48.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.414396+0000", 2024-08-17T17:19:48.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001145313 2024-08-17T17:19:48.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:19:48.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.414832+0000", 2024-08-17T17:19:48.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00043551100000000003 2024-08-17T17:19:48.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.461303+0000", 2024-08-17T17:19:48.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.046471577 2024-08-17T17:19:48.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:48.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.461356+0000", 2024-08-17T17:19:48.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2117000000000003e-05 2024-08-17T17:19:48.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:48.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.461396+0000", 2024-08-17T17:19:48.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0785000000000003e-05 2024-08-17T17:19:48.659 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.659 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:48.660 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.660 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:22184 5.c 5:369fee11:::benchmark_data_smithi003_140793_object22183:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-08-17T17:19:48.660 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:20.413347+0000", 2024-08-17T17:19:48.660 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.268980008, 2024-08-17T17:19:48.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.048336494000000001, 2024-08-17T17:19:48.660 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:48.660 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:48.660 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:48.660 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:19:48.660 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:19:48.660 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:19:48.661 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 22184 2024-08-17T17:19:48.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.661 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:48.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:48.661 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.413347+0000", 2024-08-17T17:19:48.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:48.661 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.413347+0000", 2024-08-17T17:19:48.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:48.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.413348+0000", 2024-08-17T17:19:48.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.006e-06 2024-08-17T17:19:48.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:48.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.413370+0000", 2024-08-17T17:19:48.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1319999999999999e-05 2024-08-17T17:19:48.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:48.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.413380+0000", 2024-08-17T17:19:48.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0382e-05 2024-08-17T17:19:48.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:48.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.413388+0000", 2024-08-17T17:19:48.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5399999999999998e-06 2024-08-17T17:19:48.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:48.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.413536+0000", 2024-08-17T17:19:48.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014810100000000001 2024-08-17T17:19:48.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:48.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.413656+0000", 2024-08-17T17:19:48.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000120442 2024-08-17T17:19:48.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4", 2024-08-17T17:19:48.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.413806+0000", 2024-08-17T17:19:48.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015007 2024-08-17T17:19:48.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.414834+0000", 2024-08-17T17:19:48.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010276969999999999 2024-08-17T17:19:48.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:19:48.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.414854+0000", 2024-08-17T17:19:48.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0236000000000001e-05 2024-08-17T17:19:48.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.461551+0000", 2024-08-17T17:19:48.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.046697013000000002 2024-08-17T17:19:48.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:48.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.461602+0000", 2024-08-17T17:19:48.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0439000000000003e-05 2024-08-17T17:19:48.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:48.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:20.461684+0000", 2024-08-17T17:19:48.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2248000000000003e-05 2024-08-17T17:19:48.666 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.666 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:48.666 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.666 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26028 5.7 5:e0f7b3ef:::benchmark_data_smithi003_140793_object26027:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-08-17T17:19:48.666 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.254623+0000", 2024-08-17T17:19:48.667 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 25.427704399, 2024-08-17T17:19:48.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.079618548999999997, 2024-08-17T17:19:48.667 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:48.667 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:48.667 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:48.667 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:19:48.667 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:19:48.667 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:19:48.667 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26028 2024-08-17T17:19:48.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.667 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:48.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:48.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.254623+0000", 2024-08-17T17:19:48.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:48.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.254623+0000", 2024-08-17T17:19:48.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:48.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.254624+0000", 2024-08-17T17:19:48.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.11e-06 2024-08-17T17:19:48.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:48.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.254642+0000", 2024-08-17T17:19:48.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.789e-05 2024-08-17T17:19:48.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:48.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.254652+0000", 2024-08-17T17:19:48.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0108999999999999e-05 2024-08-17T17:19:48.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:48.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.254658+0000", 2024-08-17T17:19:48.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8420000000000001e-06 2024-08-17T17:19:48.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:48.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.254713+0000", 2024-08-17T17:19:48.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4685000000000002e-05 2024-08-17T17:19:48.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:48.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.254842+0000", 2024-08-17T17:19:48.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012882 2024-08-17T17:19:48.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4", 2024-08-17T17:19:48.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.254964+0000", 2024-08-17T17:19:48.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000121999 2024-08-17T17:19:48.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:19:48.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.257947+0000", 2024-08-17T17:19:48.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002983608 2024-08-17T17:19:48.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.257990+0000", 2024-08-17T17:19:48.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2660000000000002e-05 2024-08-17T17:19:48.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.672 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.334146+0000", 2024-08-17T17:19:48.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.076156009999999996 2024-08-17T17:19:48.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.672 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:48.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.334203+0000", 2024-08-17T17:19:48.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7000000000000003e-05 2024-08-17T17:19:48.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.672 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:48.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.334242+0000", 2024-08-17T17:19:48.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8816000000000002e-05 2024-08-17T17:19:48.673 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.673 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:48.673 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.673 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26031 5.7 5:e1c1336b:::benchmark_data_smithi003_140793_object26030:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-08-17T17:19:48.673 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.256308+0000", 2024-08-17T17:19:48.673 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 25.426019899, 2024-08-17T17:19:48.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.078312850000000003, 2024-08-17T17:19:48.673 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:48.673 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:48.673 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:48.673 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:19:48.674 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:19:48.674 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:19:48.674 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26031 2024-08-17T17:19:48.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.674 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:48.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:48.674 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.256308+0000", 2024-08-17T17:19:48.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:48.674 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.256308+0000", 2024-08-17T17:19:48.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:48.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.256309+0000", 2024-08-17T17:19:48.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.519e-06 2024-08-17T17:19:48.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:48.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.256332+0000", 2024-08-17T17:19:48.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2938999999999998e-05 2024-08-17T17:19:48.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:48.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.256344+0000", 2024-08-17T17:19:48.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2184000000000001e-05 2024-08-17T17:19:48.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:48.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.256352+0000", 2024-08-17T17:19:48.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4800000000000004e-06 2024-08-17T17:19:48.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:48.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.256518+0000", 2024-08-17T17:19:48.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016606600000000001 2024-08-17T17:19:48.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.677 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:48.677 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.256642+0000", 2024-08-17T17:19:48.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012402999999999999 2024-08-17T17:19:48.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.677 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4", 2024-08-17T17:19:48.677 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.256768+0000", 2024-08-17T17:19:48.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012617600000000001 2024-08-17T17:19:48.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.677 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:19:48.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.258008+0000", 2024-08-17T17:19:48.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00124044 2024-08-17T17:19:48.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.258572+0000", 2024-08-17T17:19:48.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00056349299999999998 2024-08-17T17:19:48.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.334279+0000", 2024-08-17T17:19:48.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.075706651999999999 2024-08-17T17:19:48.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:48.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.334565+0000", 2024-08-17T17:19:48.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00028626999999999998 2024-08-17T17:19:48.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:48.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.334620+0000", 2024-08-17T17:19:48.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5600999999999998e-05 2024-08-17T17:19:48.679 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.679 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:48.679 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.680 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26036 5.c 5:39880942:::benchmark_data_smithi003_140793_object26035:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-08-17T17:19:48.680 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.258354+0000", 2024-08-17T17:19:48.680 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 25.423973114999999, 2024-08-17T17:19:48.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.076247023999999997, 2024-08-17T17:19:48.680 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:48.680 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:48.680 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:48.680 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:19:48.680 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:19:48.680 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:19:48.680 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26036 2024-08-17T17:19:48.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.680 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:48.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:48.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.258354+0000", 2024-08-17T17:19:48.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:48.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.258354+0000", 2024-08-17T17:19:48.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:48.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.258356+0000", 2024-08-17T17:19:48.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6249999999999999e-06 2024-08-17T17:19:48.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:48.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.258378+0000", 2024-08-17T17:19:48.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.247e-05 2024-08-17T17:19:48.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:48.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.258390+0000", 2024-08-17T17:19:48.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1316e-05 2024-08-17T17:19:48.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:48.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.258397+0000", 2024-08-17T17:19:48.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4490000000000002e-06 2024-08-17T17:19:48.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:48.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.258460+0000", 2024-08-17T17:19:48.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2829000000000004e-05 2024-08-17T17:19:48.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:48.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.258559+0000", 2024-08-17T17:19:48.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8723999999999999e-05 2024-08-17T17:19:48.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4", 2024-08-17T17:19:48.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.258669+0000", 2024-08-17T17:19:48.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110364 2024-08-17T17:19:48.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.259813+0000", 2024-08-17T17:19:48.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0011439309999999999 2024-08-17T17:19:48.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:19:48.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.262251+0000", 2024-08-17T17:19:48.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002437551 2024-08-17T17:19:48.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.334521+0000", 2024-08-17T17:19:48.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.072270437000000007 2024-08-17T17:19:48.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:48.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.334569+0000", 2024-08-17T17:19:48.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.782e-05 2024-08-17T17:19:48.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:48.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.334601+0000", 2024-08-17T17:19:48.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2508000000000001e-05 2024-08-17T17:19:48.686 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.686 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:48.686 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.686 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26522 5.c 5:3f5f587f:::benchmark_data_smithi003_140793_object26521:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-08-17T17:19:48.686 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:25.387768+0000", 2024-08-17T17:19:48.686 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 22.294559868, 2024-08-17T17:19:48.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.030412268999999999, 2024-08-17T17:19:48.686 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:48.686 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:48.687 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:48.687 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:19:48.687 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:19:48.687 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:19:48.687 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26522 2024-08-17T17:19:48.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.687 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:48.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:48.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.387768+0000", 2024-08-17T17:19:48.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:48.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.387768+0000", 2024-08-17T17:19:48.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:48.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.387769+0000", 2024-08-17T17:19:48.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.779e-06 2024-08-17T17:19:48.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:48.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.387795+0000", 2024-08-17T17:19:48.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5724e-05 2024-08-17T17:19:48.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:48.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.387808+0000", 2024-08-17T17:19:48.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2825e-05 2024-08-17T17:19:48.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:48.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.387829+0000", 2024-08-17T17:19:48.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0789999999999999e-05 2024-08-17T17:19:48.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:48.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.387919+0000", 2024-08-17T17:19:48.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0023000000000002e-05 2024-08-17T17:19:48.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:48.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.388044+0000", 2024-08-17T17:19:48.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012486800000000001 2024-08-17T17:19:48.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4", 2024-08-17T17:19:48.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.388171+0000", 2024-08-17T17:19:48.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000127076 2024-08-17T17:19:48.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:19:48.691 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.389547+0000", 2024-08-17T17:19:48.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001376398 2024-08-17T17:19:48.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.691 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.691 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.389894+0000", 2024-08-17T17:19:48.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00034709600000000003 2024-08-17T17:19:48.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.691 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.691 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.418074+0000", 2024-08-17T17:19:48.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.028179932000000001 2024-08-17T17:19:48.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:48.692 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.418136+0000", 2024-08-17T17:19:48.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1626999999999998e-05 2024-08-17T17:19:48.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:48.692 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.418180+0000", 2024-08-17T17:19:48.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4131000000000001e-05 2024-08-17T17:19:48.692 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.692 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:48.692 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.693 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:33342 5.c 5:3306fe14:::benchmark_data_smithi003_140793_object33341:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-08-17T17:19:48.693 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:29.286437+0000", 2024-08-17T17:19:48.693 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 18.395890689000002, 2024-08-17T17:19:48.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.029969012999999999, 2024-08-17T17:19:48.693 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:48.693 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:48.693 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:48.693 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:19:48.693 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:19:48.693 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:19:48.693 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 33342 2024-08-17T17:19:48.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.693 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:48.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:48.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:29.286437+0000", 2024-08-17T17:19:48.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:48.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:29.286437+0000", 2024-08-17T17:19:48.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:48.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:29.286439+0000", 2024-08-17T17:19:48.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7400000000000001e-06 2024-08-17T17:19:48.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:48.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:29.286466+0000", 2024-08-17T17:19:48.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7418000000000001e-05 2024-08-17T17:19:48.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:48.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:29.286490+0000", 2024-08-17T17:19:48.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4383e-05 2024-08-17T17:19:48.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:48.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:29.286500+0000", 2024-08-17T17:19:48.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0056e-05 2024-08-17T17:19:48.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:48.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:29.289908+0000", 2024-08-17T17:19:48.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034073440000000001 2024-08-17T17:19:48.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:48.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:29.290069+0000", 2024-08-17T17:19:48.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000161495 2024-08-17T17:19:48.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4", 2024-08-17T17:19:48.697 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:29.290214+0000", 2024-08-17T17:19:48.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000144821 2024-08-17T17:19:48.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:19:48.697 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:29.292812+0000", 2024-08-17T17:19:48.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025982470000000001 2024-08-17T17:19:48.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.697 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:29.293342+0000", 2024-08-17T17:19:48.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00052969900000000001 2024-08-17T17:19:48.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:29.316323+0000", 2024-08-17T17:19:48.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.022981425 2024-08-17T17:19:48.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:48.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:29.316369+0000", 2024-08-17T17:19:48.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5862000000000002e-05 2024-08-17T17:19:48.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:48.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:29.316406+0000", 2024-08-17T17:19:48.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6523000000000003e-05 2024-08-17T17:19:48.699 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.699 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:48.699 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.699 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:33350 5.c 5:304163f3:::benchmark_data_smithi003_140793_object33349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-08-17T17:19:48.699 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:29.291097+0000", 2024-08-17T17:19:48.699 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 18.391230906000001, 2024-08-17T17:19:48.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.025409698000000001, 2024-08-17T17:19:48.699 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:48.699 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:48.699 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:48.699 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:19:48.699 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:19:48.700 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:19:48.700 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 33350 2024-08-17T17:19:48.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.700 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:48.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.700 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:48.700 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:29.291097+0000", 2024-08-17T17:19:48.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.700 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:48.700 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:29.291097+0000", 2024-08-17T17:19:48.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.701 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:48.701 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:29.291099+0000", 2024-08-17T17:19:48.701 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0339999999999999e-06 2024-08-17T17:19:48.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.701 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:48.701 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:29.291123+0000", 2024-08-17T17:19:48.701 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4477e-05 2024-08-17T17:19:48.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.701 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:48.701 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:29.291137+0000", 2024-08-17T17:19:48.701 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4268e-05 2024-08-17T17:19:48.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:48.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:29.291146+0000", 2024-08-17T17:19:48.702 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6510000000000007e-06 2024-08-17T17:19:48.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:48.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:29.291347+0000", 2024-08-17T17:19:48.702 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020087999999999999 2024-08-17T17:19:48.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:48.703 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:29.291468+0000", 2024-08-17T17:19:48.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000121149 2024-08-17T17:19:48.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.703 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4", 2024-08-17T17:19:48.703 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:29.291596+0000", 2024-08-17T17:19:48.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012778400000000001 2024-08-17T17:19:48.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.703 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.703 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:29.294915+0000", 2024-08-17T17:19:48.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0033188549999999999 2024-08-17T17:19:48.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.704 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:19:48.704 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:29.294926+0000", 2024-08-17T17:19:48.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0900000000000001e-05 2024-08-17T17:19:48.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.704 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.704 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:29.316439+0000", 2024-08-17T17:19:48.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.021513876000000001 2024-08-17T17:19:48.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.704 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:48.704 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:29.316477+0000", 2024-08-17T17:19:48.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7172000000000003e-05 2024-08-17T17:19:48.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.705 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:48.705 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:29.316506+0000", 2024-08-17T17:19:48.705 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9652e-05 2024-08-17T17:19:48.705 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.705 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:48.705 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.705 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.35277.0:35781 5.0 e123/111 5:0f302fdd:::benchmark_data_smithi003_140793_object35780:head v 123'2289, mlcod=123'2287)", 2024-08-17T17:19:48.705 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:30.796642+0000", 2024-08-17T17:19:48.705 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 16.885685444, 2024-08-17T17:19:48.705 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.036493310000000001, 2024-08-17T17:19:48.705 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:48.706 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:48.706 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:48.706 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:48.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.706 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:48.706 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:30.796642+0000", 2024-08-17T17:19:48.706 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.706 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:48.706 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:30.796642+0000", 2024-08-17T17:19:48.706 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:48.707 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:30.796643+0000", 2024-08-17T17:19:48.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5899999999999996e-07 2024-08-17T17:19:48.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:48.707 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:30.796662+0000", 2024-08-17T17:19:48.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.969e-05 2024-08-17T17:19:48.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:48.707 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:30.796670+0000", 2024-08-17T17:19:48.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6429999999999995e-06 2024-08-17T17:19:48.708 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.708 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.708 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:48.708 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:30.796674+0000", 2024-08-17T17:19:48.708 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.374e-06 2024-08-17T17:19:48.708 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.708 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.708 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:48.708 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:30.796722+0000", 2024-08-17T17:19:48.708 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7120999999999998e-05 2024-08-17T17:19:48.708 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.708 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.708 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:48.709 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:30.796740+0000", 2024-08-17T17:19:48.709 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8437999999999999e-05 2024-08-17T17:19:48.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.709 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.709 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:48.709 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:30.833061+0000", 2024-08-17T17:19:48.709 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.036320707000000001 2024-08-17T17:19:48.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.709 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.709 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:48.709 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:30.833135+0000", 2024-08-17T17:19:48.709 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4678000000000006e-05 2024-08-17T17:19:48.709 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.709 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:48.710 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.710 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.710 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:36979 5.c 5:38985aeb:::benchmark_data_smithi003_140793_object36978:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-08-17T17:19:48.710 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:31.585144+0000", 2024-08-17T17:19:48.710 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 16.097183285, 2024-08-17T17:19:48.710 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.025638030999999999, 2024-08-17T17:19:48.710 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:48.710 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:48.710 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:48.710 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:19:48.710 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:19:48.710 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:19:48.710 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 36979 2024-08-17T17:19:48.711 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.711 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:48.711 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.711 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:48.711 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.585144+0000", 2024-08-17T17:19:48.711 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.711 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.711 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.711 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:48.711 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.585144+0000", 2024-08-17T17:19:48.711 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.711 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.711 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.712 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:48.712 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.585145+0000", 2024-08-17T17:19:48.712 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.006e-06 2024-08-17T17:19:48.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.712 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:48.712 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.585169+0000", 2024-08-17T17:19:48.712 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4128e-05 2024-08-17T17:19:48.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.712 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:48.712 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.585179+0000", 2024-08-17T17:19:48.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2289999999999992e-06 2024-08-17T17:19:48.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.713 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:48.713 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.585185+0000", 2024-08-17T17:19:48.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8369999999999999e-06 2024-08-17T17:19:48.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.713 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:48.713 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.585966+0000", 2024-08-17T17:19:48.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00078078100000000003 2024-08-17T17:19:48.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.713 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:48.713 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.586080+0000", 2024-08-17T17:19:48.714 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000113921 2024-08-17T17:19:48.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.714 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.714 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4", 2024-08-17T17:19:48.714 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.586200+0000", 2024-08-17T17:19:48.714 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000119743 2024-08-17T17:19:48.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.714 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.714 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:19:48.714 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.589459+0000", 2024-08-17T17:19:48.714 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032587580000000001 2024-08-17T17:19:48.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.715 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.715 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.589560+0000", 2024-08-17T17:19:48.715 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000101173 2024-08-17T17:19:48.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.715 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.715 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.610694+0000", 2024-08-17T17:19:48.715 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.021134245999999999 2024-08-17T17:19:48.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.715 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:48.715 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.610736+0000", 2024-08-17T17:19:48.716 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2205e-05 2024-08-17T17:19:48.716 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.716 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.716 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:48.716 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.610782+0000", 2024-08-17T17:19:48.716 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6004000000000002e-05 2024-08-17T17:19:48.716 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.716 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:48.716 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.716 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.716 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.716 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:36984 5.7 5:ebd8682c:::benchmark_data_smithi003_140793_object36983:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-08-17T17:19:48.716 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:31.585200+0000", 2024-08-17T17:19:48.717 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 16.097127954000001, 2024-08-17T17:19:48.717 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.025626867000000001, 2024-08-17T17:19:48.717 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:48.717 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:48.717 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:48.717 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:19:48.717 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:19:48.717 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:19:48.717 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 36984 2024-08-17T17:19:48.717 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.717 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:48.717 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.718 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:48.718 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.585200+0000", 2024-08-17T17:19:48.718 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.718 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:48.718 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.585200+0000", 2024-08-17T17:19:48.718 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.718 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:48.718 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.585200+0000", 2024-08-17T17:19:48.718 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5899999999999996e-07 2024-08-17T17:19:48.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.719 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:48.719 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.585221+0000", 2024-08-17T17:19:48.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0899000000000001e-05 2024-08-17T17:19:48.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.719 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:48.719 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.585229+0000", 2024-08-17T17:19:48.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8259999999999993e-06 2024-08-17T17:19:48.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.719 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:48.720 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.585235+0000", 2024-08-17T17:19:48.720 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9839999999999998e-06 2024-08-17T17:19:48.720 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.720 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.720 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:48.720 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.585283+0000", 2024-08-17T17:19:48.720 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8063999999999999e-05 2024-08-17T17:19:48.720 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.720 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.720 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:48.720 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.585429+0000", 2024-08-17T17:19:48.720 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014582899999999999 2024-08-17T17:19:48.720 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4", 2024-08-17T17:19:48.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.585553+0000", 2024-08-17T17:19:48.721 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012463599999999999 2024-08-17T17:19:48.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:19:48.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.587048+0000", 2024-08-17T17:19:48.721 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014942829999999999 2024-08-17T17:19:48.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.589456+0000", 2024-08-17T17:19:48.721 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024078630000000001 2024-08-17T17:19:48.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.722 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.722 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.610713+0000", 2024-08-17T17:19:48.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.021257558999999999 2024-08-17T17:19:48.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.722 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:48.722 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.610771+0000", 2024-08-17T17:19:48.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7969e-05 2024-08-17T17:19:48.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.722 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:48.723 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.610826+0000", 2024-08-17T17:19:48.723 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5296e-05 2024-08-17T17:19:48.723 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.723 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:48.723 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.723 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:36986 5.7 5:e1346722:::benchmark_data_smithi003_140793_object36985:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-08-17T17:19:48.723 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:31.586983+0000", 2024-08-17T17:19:48.723 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 16.095344364999999, 2024-08-17T17:19:48.723 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.024266342999999999, 2024-08-17T17:19:48.723 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:19:48.723 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:19:48.724 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:19:48.724 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:19:48.724 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:19:48.724 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:19:48.724 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 36986 2024-08-17T17:19:48.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.724 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:19:48.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.724 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:19:48.724 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.586983+0000", 2024-08-17T17:19:48.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.724 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:19:48.725 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.586983+0000", 2024-08-17T17:19:48.725 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:19:48.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.725 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:19:48.725 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.586984+0000", 2024-08-17T17:19:48.725 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1200000000000003e-07 2024-08-17T17:19:48.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.725 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:19:48.725 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.587025+0000", 2024-08-17T17:19:48.725 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1402000000000001e-05 2024-08-17T17:19:48.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.726 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:19:48.726 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.587035+0000", 2024-08-17T17:19:48.726 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4089999999999992e-06 2024-08-17T17:19:48.726 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.726 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:19:48.726 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.587041+0000", 2024-08-17T17:19:48.726 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7259999999999996e-06 2024-08-17T17:19:48.726 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.726 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:19:48.726 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.587084+0000", 2024-08-17T17:19:48.726 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2970999999999999e-05 2024-08-17T17:19:48.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.727 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.727 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:19:48.727 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.587213+0000", 2024-08-17T17:19:48.727 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012887900000000001 2024-08-17T17:19:48.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.727 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.727 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4", 2024-08-17T17:19:48.727 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.587351+0000", 2024-08-17T17:19:48.727 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0001383 2024-08-17T17:19:48.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.727 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.727 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:19:48.727 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.589533+0000", 2024-08-17T17:19:48.728 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0021817239999999999 2024-08-17T17:19:48.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.728 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.728 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.728 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.589582+0000", 2024-08-17T17:19:48.728 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8832999999999998e-05 2024-08-17T17:19:48.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.728 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.728 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:19:48.728 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.611136+0000", 2024-08-17T17:19:48.728 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.021553624 2024-08-17T17:19:48.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.728 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:19:48.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.611224+0000", 2024-08-17T17:19:48.729 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8171000000000007e-05 2024-08-17T17:19:48.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:19:48.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:19:48.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:19:48.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:31.611249+0000", 2024-08-17T17:19:48.729 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5692000000000001e-05 2024-08-17T17:19:48.729 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.729 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:48.729 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.729 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:19:48.729 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:19:48.730 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:19:48.730 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:48.731 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:48 smithi053 ceph-mon[89963]: osdmap e128: 8 total, 8 up, 8 in 2024-08-17T17:19:48.752 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:48.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:48.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:48.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:48 smithi003 ceph-mon[100615]: osdmap e128: 8 total, 8 up, 8 in 2024-08-17T17:19:48.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:48 smithi003 ceph-mon[103344]: osdmap e128: 8 total, 8 up, 8 in 2024-08-17T17:19:48.800 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 37 16 58955 58939 99.5171 68.5625 0.0831457 0.0100437 2024-08-17T17:19:48.854 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:48.895 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:48.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:48.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:48.998 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:49.024 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:49.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:49.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:49.127 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:49.155 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:49.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:49.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:49.257 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:49.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:49.281 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:49.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:49.382 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:49.409 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:49.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:49.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:49.511 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:49.535 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:49.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:49.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:49.636 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:49.662 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:49.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:49.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:49.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:49 smithi053 ceph-mon[89963]: pgmap v256: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 3.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 26/164298 objects misplaced (0.016%); 6.7 MiB/s, 108 objects/s recovering 2024-08-17T17:19:49.764 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:49.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:49 smithi003 ceph-mon[103344]: pgmap v256: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 3.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 26/164298 objects misplaced (0.016%); 6.7 MiB/s, 108 objects/s recovering 2024-08-17T17:19:49.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:49 smithi003 ceph-mon[100615]: pgmap v256: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 3.3 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s; 26/164298 objects misplaced (0.016%); 6.7 MiB/s, 108 objects/s recovering 2024-08-17T17:19:49.800 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 38 16 60182 60166 98.9163 76.6875 0.00874729 0.0101036 2024-08-17T17:19:49.803 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:49.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:49.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:49.905 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:49.932 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:49.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:49.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:50.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:50.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:50.065 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:50.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:50.166 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:50.193 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:50.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:50.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:50.295 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:50.322 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:50.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:50.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:50.424 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:50.448 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:50.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:50.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:50.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:19:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:19:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:19:50.551 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:50.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:50.588 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:50.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:50.689 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:50.717 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:50.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:50.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:50.801 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 39 16 61388 61372 98.3127 75.375 0.00227111 0.0101648 2024-08-17T17:19:50.819 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:50.841 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:50.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:50.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:50.943 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:50.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:50.980 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:50.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:51.081 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:51.117 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:51.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:51.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:51.219 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:51.259 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:51.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:51.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:51.363 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:51.390 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:51.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:51.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:51.492 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:51.521 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:51.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:51.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:51.623 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:51.654 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:51.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:51.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:51.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:51 smithi053 ceph-mon[89963]: pgmap v257: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s; 26/175158 objects misplaced (0.015%); 3.7 MiB/s, 59 objects/s recovering 2024-08-17T17:19:51.756 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:51.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:51 smithi003 ceph-mon[100615]: pgmap v257: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s; 26/175158 objects misplaced (0.015%); 3.7 MiB/s, 59 objects/s recovering 2024-08-17T17:19:51.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:51 smithi003 ceph-mon[103344]: pgmap v257: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s; 26/175158 objects misplaced (0.015%); 3.7 MiB/s, 59 objects/s recovering 2024-08-17T17:19:51.792 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:51.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:51.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:51.801 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-08-17T17:19:51.801188+0000 min lat: 0.0011526 max lat: 2.93219 avg lat: 0.0102446 2024-08-17T17:19:51.801 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:19:51.801 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 40 16 62473 62457 97.5503 67.8125 0.0137326 0.0102446 2024-08-17T17:19:51.898 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:51.940 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:51.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:51.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:52.042 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:52.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:52.064 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:52.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:52.165 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:52.190 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:52.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:52.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:52.292 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:52.329 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:52.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:52.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:52.431 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:52.471 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:52.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:52.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:52.573 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:52.601 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:52.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:52.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:52.703 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:52.730 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:52.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:52.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:52.801 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 41 16 63747 63731 97.1131 79.625 0.00701218 0.0102917 2024-08-17T17:19:52.832 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:52.872 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:52.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:52.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:52.974 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:53.016 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:53.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:53.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:53.118 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:53.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:53.166 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:53.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:53.267 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:53.308 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:53.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:53.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:53.410 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:53.436 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:53.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:53.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:53.537 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:53.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:53.581 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:53.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:53.682 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:53.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:53 smithi053 ceph-mon[89963]: pgmap v258: 89 pgs: 89 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 50 MiB/s wr, 806 op/s; 3.9 MiB/s, 62 objects/s recovering 2024-08-17T17:19:53.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:19:53.722 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:53.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:53.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:53.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:53 smithi003 ceph-mon[103344]: pgmap v258: 89 pgs: 89 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 50 MiB/s wr, 806 op/s; 3.9 MiB/s, 62 objects/s recovering 2024-08-17T17:19:53.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:19:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:53 smithi003 ceph-mon[100615]: pgmap v258: 89 pgs: 89 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 50 MiB/s wr, 806 op/s; 3.9 MiB/s, 62 objects/s recovering 2024-08-17T17:19:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:19:53.801 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 42 16 65003 64987 96.6698 78.5 0.00602486 0.0103383 2024-08-17T17:19:53.824 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:53.853 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:53.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:53.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:53.955 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:53.981 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:53.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:53.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:54.083 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:54.110 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:54.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:54.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:54.212 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:54.236 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:54.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:54.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:54.338 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:54.362 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:54.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:54.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:54.464 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:54.490 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:54.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:54.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:54.567 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-17T17:19:54.592 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:54.616 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:54.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:54.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:54.718 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:54.768 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:54.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:54.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:54.801 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 43 16 66223 66207 96.1936 76.25 0.0179088 0.0103901 2024-08-17T17:19:54.871 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:54.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:54.925 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:54.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:55.026 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:55.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:55.071 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:55.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:55.172 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:55.212 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:55.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:55.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:55.314 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:55.355 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:55.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:55.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:55.457 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:55.481 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:55.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:55.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:55.554 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:55 smithi003 ceph-mon[103344]: pgmap v259: 89 pgs: 89 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s; 208 KiB/s, 3 objects/s recovering 2024-08-17T17:19:55.554 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:55 smithi003 ceph-mon[100615]: pgmap v259: 89 pgs: 89 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s; 208 KiB/s, 3 objects/s recovering 2024-08-17T17:19:55.583 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:19:55.609 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:55.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:55.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:19:55.711 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:55.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:55 smithi053 ceph-mon[89963]: pgmap v259: 89 pgs: 89 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s; 208 KiB/s, 3 objects/s recovering 2024-08-17T17:19:55.736 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:55.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:55.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:55.802 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 44 16 67396 67380 95.6736 73.3125 0.00549004 0.0104461 2024-08-17T17:19:55.837 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:55.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:55.881 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:55.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:55.982 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:56.007 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:56.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:56.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:56.008 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:19:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:19:55] "GET /metrics HTTP/1.1" 200 36523 "" "Prometheus/2.51.0" 2024-08-17T17:19:56.109 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:56.179 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:56.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:56.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:56.281 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:56.326 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:56.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:56.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:56.428 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:56.470 INFO:teuthology.orchestra.run.smithi003.stdout:moved 46 / 267 (17.2285%) 2024-08-17T17:19:56.470 INFO:teuthology.orchestra.run.smithi003.stdout:avg 33.375 2024-08-17T17:19:56.470 INFO:teuthology.orchestra.run.smithi003.stdout:stddev 11.521 -> 10.9194 (expected baseline 5.40399) 2024-08-17T17:19:56.470 INFO:teuthology.orchestra.run.smithi003.stdout:min osd.7 with 20 -> 46 pgs (0.599251 -> 1.37828 * mean) 2024-08-17T17:19:56.470 INFO:teuthology.orchestra.run.smithi003.stdout:max osd.1 with 49 -> 50 pgs (1.46816 -> 1.49813 * mean) 2024-08-17T17:19:56.470 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:19:56.470 INFO:teuthology.orchestra.run.smithi003.stdout:oload 110 2024-08-17T17:19:56.471 INFO:teuthology.orchestra.run.smithi003.stdout:max_change 1 2024-08-17T17:19:56.471 INFO:teuthology.orchestra.run.smithi003.stdout:max_change_osds 4 2024-08-17T17:19:56.471 INFO:teuthology.orchestra.run.smithi003.stdout:average_utilization 0.0185 2024-08-17T17:19:56.471 INFO:teuthology.orchestra.run.smithi003.stdout:overload_utilization 0.0204 2024-08-17T17:19:56.471 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0 weight 0.9500 -> 0.5399 2024-08-17T17:19:56.471 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7 weight 0.5965 -> 0.9607 2024-08-17T17:19:56.471 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2 weight 1.0000 -> 0.7258 2024-08-17T17:19:56.471 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6 weight 0.5465 -> 0.7042 2024-08-17T17:19:56.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:56.486 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:56.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:56.570 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:56 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'35386','2':'47565','6':'46152','7':'62963'}"} : dispatch 2024-08-17T17:19:56.571 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:56 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'35386','2':'47565','6':'46152','7':'62963'}"} : dispatch 2024-08-17T17:19:56.587 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:56.609 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:56.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:56.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:56.711 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:56.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:56 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'35386','2':'47565','6':'46152','7':'62963'}"} : dispatch 2024-08-17T17:19:56.733 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:56.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:56.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:56.802 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 45 16 68144 68128 94.5866 46.75 0.00288419 0.0104937 2024-08-17T17:19:56.836 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:56.844 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:19:56.858 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:56.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:56.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:56.960 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:56.982 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:56.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:56.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:57.084 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:57.107 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:57.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:57.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:57.209 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:57.233 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:57.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:57.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:57.334 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:57.359 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:57.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:57.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:57.461 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:57.475 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-08-17T17:19:57.500 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:57 smithi003 ceph-mon[100615]: pgmap v260: 89 pgs: 89 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 73 MiB/s wr, 1.16k op/s; 179 KiB/s, 2 objects/s recovering 2024-08-17T17:19:57.500 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:57 smithi003 ceph-mon[100615]: from='client.45182 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:19:57.500 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:57 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'35386','2':'47565','6':'46152','7':'62963'}"}]': finished 2024-08-17T17:19:57.500 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:57 smithi003 ceph-mon[100615]: osdmap e129: 8 total, 8 up, 8 in 2024-08-17T17:19:57.500 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:57 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3283387340' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:19:57.501 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:57 smithi003 ceph-mon[103344]: pgmap v260: 89 pgs: 89 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 73 MiB/s wr, 1.16k op/s; 179 KiB/s, 2 objects/s recovering 2024-08-17T17:19:57.501 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:57 smithi003 ceph-mon[103344]: from='client.45182 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:19:57.501 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:57 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'35386','2':'47565','6':'46152','7':'62963'}"}]': finished 2024-08-17T17:19:57.501 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:57 smithi003 ceph-mon[103344]: osdmap e129: 8 total, 8 up, 8 in 2024-08-17T17:19:57.501 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:57 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3283387340' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:19:57.511 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:57.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:57.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:57.613 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:57.636 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:57.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:57.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:57.738 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:57.763 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:57.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:57.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:57.802 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 46 16 68144 68128 92.531 0 - 0.0104937 2024-08-17T17:19:57.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:57.887 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:57.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:57.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:57.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:57 smithi053 ceph-mon[89963]: pgmap v260: 89 pgs: 89 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 73 MiB/s wr, 1.16k op/s; 179 KiB/s, 2 objects/s recovering 2024-08-17T17:19:57.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:57 smithi053 ceph-mon[89963]: from='client.45182 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-08-17T17:19:57.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:57 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'35386','2':'47565','6':'46152','7':'62963'}"}]': finished 2024-08-17T17:19:57.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:57 smithi053 ceph-mon[89963]: osdmap e129: 8 total, 8 up, 8 in 2024-08-17T17:19:57.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:57 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3283387340' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:19:57.988 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:58.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:58.011 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:58.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:58.101 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:19:58.112 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:58.138 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:58.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:58.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:58.241 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:19:58.263 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:58.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:58.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:19:58.365 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:58.387 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:58.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:58.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:58.489 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:19:58.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:58.528 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:58.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:19:58.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:58 smithi003 ceph-mon[100615]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:19:58.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:58 smithi003 ceph-mon[100615]: Cluster is now healthy 2024-08-17T17:19:58.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:58 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3283387340' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:19:58.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:58 smithi003 ceph-mon[100615]: osdmap e130: 8 total, 8 up, 8 in 2024-08-17T17:19:58.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:58 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:19:58.550 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:58 smithi003 ceph-mon[103344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:19:58.550 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:58 smithi003 ceph-mon[103344]: Cluster is now healthy 2024-08-17T17:19:58.551 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:58 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3283387340' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:19:58.551 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:58 smithi003 ceph-mon[103344]: osdmap e130: 8 total, 8 up, 8 in 2024-08-17T17:19:58.551 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:58 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:19:58.629 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:19:58.677 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:58.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:58.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:19:58.779 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:58.802 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 47 16 68195 68179 90.6306 1.59375 0.0112537 0.0109941 2024-08-17T17:19:58.807 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:58.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:58.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:58.909 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:19:58.946 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:58.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:58.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:19:58.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:58 smithi053 ceph-mon[89963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:19:58.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:58 smithi053 ceph-mon[89963]: Cluster is now healthy 2024-08-17T17:19:58.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:58 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3283387340' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:19:58.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:58 smithi053 ceph-mon[89963]: osdmap e130: 8 total, 8 up, 8 in 2024-08-17T17:19:58.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:58 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:19:59.048 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:59.095 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:59.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:59.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:59.197 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:59.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:59.234 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:59.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:59.335 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:19:59.368 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:59.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:59.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:19:59.470 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:59.514 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:59.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:59.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:59.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:19:59.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:59.659 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:19:59.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:19:59.743 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:59 smithi003 ceph-mon[100615]: pgmap v263: 89 pgs: 1 unknown, 88 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 53 MiB/s wr, 851 op/s; 208 KiB/s, 3 objects/s recovering 2024-08-17T17:19:59.743 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:59 smithi003 ceph-mon[100615]: osdmap e131: 8 total, 8 up, 8 in 2024-08-17T17:19:59.744 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:19:59 smithi003 ceph-mon[100615]: osdmap e132: 8 total, 8 up, 8 in 2024-08-17T17:19:59.744 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:59 smithi003 ceph-mon[103344]: pgmap v263: 89 pgs: 1 unknown, 88 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 53 MiB/s wr, 851 op/s; 208 KiB/s, 3 objects/s recovering 2024-08-17T17:19:59.744 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:59 smithi003 ceph-mon[103344]: osdmap e131: 8 total, 8 up, 8 in 2024-08-17T17:19:59.744 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:19:59 smithi003 ceph-mon[103344]: osdmap e132: 8 total, 8 up, 8 in 2024-08-17T17:19:59.759 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:59.789 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:59.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:59.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:59.802 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 48 16 69165 69149 90.0055 60.625 0.0194171 0.0111049 2024-08-17T17:19:59.891 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:19:59.923 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:19:59.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:19:59.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:19:59.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:59 smithi053 ceph-mon[89963]: pgmap v263: 89 pgs: 1 unknown, 88 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 53 MiB/s wr, 851 op/s; 208 KiB/s, 3 objects/s recovering 2024-08-17T17:19:59.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:59 smithi053 ceph-mon[89963]: osdmap e131: 8 total, 8 up, 8 in 2024-08-17T17:19:59.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:19:59 smithi053 ceph-mon[89963]: osdmap e132: 8 total, 8 up, 8 in 2024-08-17T17:20:00.026 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:00.060 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:00.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:00.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:00.162 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:00.190 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:00.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:00.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:00.292 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:00.325 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:00.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:00.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:00.427 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:00.458 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:00.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:00.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:00.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:20:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:20:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:20:00.560 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:00.597 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:00.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:00.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:00.629 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:20:00.629 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:20:00.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:00.630 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:69908 5.a 5:54fa4266:::benchmark_data_smithi003_140793_object69907:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-08-17T17:20:00.630 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:20:00.627096+0000", 2024-08-17T17:20:00.630 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.000458818, 2024-08-17T17:20:00.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00049181799999999999, 2024-08-17T17:20:00.630 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:00.630 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:00.630 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:20:00.630 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:00.630 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:00.630 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:00.630 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 69908 2024-08-17T17:20:00.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:00.630 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:00.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:00.631 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:00.631 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:20:00.627096+0000", 2024-08-17T17:20:00.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:00.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:00.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:00.631 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:00.631 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:20:00.627096+0000", 2024-08-17T17:20:00.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:00.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:00.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:00.631 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:00.631 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:20:00.627098+0000", 2024-08-17T17:20:00.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.902e-06 2024-08-17T17:20:00.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:00.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:00.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:00.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:20:00.627124+0000", 2024-08-17T17:20:00.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5556999999999999e-05 2024-08-17T17:20:00.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:00.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:00.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:00.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:20:00.627141+0000", 2024-08-17T17:20:00.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7181e-05 2024-08-17T17:20:00.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:00.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:00.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:00.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:20:00.627149+0000", 2024-08-17T17:20:00.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5669999999999997e-06 2024-08-17T17:20:00.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:00.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:00.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:00.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:20:00.627227+0000", 2024-08-17T17:20:00.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8022000000000006e-05 2024-08-17T17:20:00.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:00.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:00.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:00.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:20:00.627361+0000", 2024-08-17T17:20:00.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000134511 2024-08-17T17:20:00.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:00.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:00.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:20:00.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:20:00.627577+0000", 2024-08-17T17:20:00.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000215381 2024-08-17T17:20:00.634 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:00.634 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:00.634 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:00.634 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:00.634 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-08-17T17:20:00.634 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 1 2024-08-17T17:20:00.634 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:20:00.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:00.752 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:00.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:00.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:00.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:00 smithi003 ceph-mon[103344]: overall HEALTH_OK 2024-08-17T17:20:00.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:00 smithi003 ceph-mon[103344]: osdmap e133: 8 total, 8 up, 8 in 2024-08-17T17:20:00.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:00 smithi003 ceph-mon[100615]: overall HEALTH_OK 2024-08-17T17:20:00.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:00 smithi003 ceph-mon[100615]: osdmap e133: 8 total, 8 up, 8 in 2024-08-17T17:20:00.802 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 49 16 70056 70040 89.3052 55.6875 0.00406554 0.0111914 2024-08-17T17:20:00.854 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:00.881 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:00.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:00.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:00.886 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:20:00.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:00 smithi053 ceph-mon[89963]: overall HEALTH_OK 2024-08-17T17:20:00.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:00 smithi053 ceph-mon[89963]: osdmap e133: 8 total, 8 up, 8 in 2024-08-17T17:20:00.983 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:01.009 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:01.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:01.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:01.078 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:20:01.112 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:01.145 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:01.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:01.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:01.246 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:01.279 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:01.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:01.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:01.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-17T17:20:01.381 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:01.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:01.410 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:01.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:01.511 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:01.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:01.573 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:01.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:01.616 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:01 smithi003 ceph-mon[103344]: pgmap v266: 89 pgs: 1 active+recovering+undersized+remapped, 4 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 remapped+peering, 1 unknown, 78 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 51 MiB/s wr, 812 op/s; 17194/204591 objects degraded (8.404%); 4132/204591 objects misplaced (2.020%); 496 KiB/s, 11 objects/s recovering 2024-08-17T17:20:01.616 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:01 smithi003 ceph-mon[103344]: Health check failed: Degraded data redundancy: 17194/204591 objects degraded (8.404%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:01.619 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:01 smithi003 ceph-mon[100615]: pgmap v266: 89 pgs: 1 active+recovering+undersized+remapped, 4 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 remapped+peering, 1 unknown, 78 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 51 MiB/s wr, 812 op/s; 17194/204591 objects degraded (8.404%); 4132/204591 objects misplaced (2.020%); 496 KiB/s, 11 objects/s recovering 2024-08-17T17:20:01.619 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:01 smithi003 ceph-mon[100615]: Health check failed: Degraded data redundancy: 17194/204591 objects degraded (8.404%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:01.675 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:01.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:01.708 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:01.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:01.804 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 50 16 70891 70875 88.5597 52.1875 0.00681586 0.0112835 2024-08-17T17:20:01.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:01.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:01.865 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:01.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:01.966 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:01.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:01 smithi053 ceph-mon[89963]: pgmap v266: 89 pgs: 1 active+recovering+undersized+remapped, 4 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 remapped+peering, 1 unknown, 78 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 51 MiB/s wr, 812 op/s; 17194/204591 objects degraded (8.404%); 4132/204591 objects misplaced (2.020%); 496 KiB/s, 11 objects/s recovering 2024-08-17T17:20:01.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:01 smithi053 ceph-mon[89963]: Health check failed: Degraded data redundancy: 17194/204591 objects degraded (8.404%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:01.994 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:01.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:01.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:02.096 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:02.140 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:02.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:02.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:02.242 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:02.256 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-17T17:20:02.257 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:20:02.257 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-08-17T17:20:02.257 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.225212 2024-08-17T17:20:02.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 2 0.22521207160591294 2024-08-17T17:20:02.279 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:02.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:02.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:02.381 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:02.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:02.435 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:02.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:02.536 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:02.564 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:02.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:02.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:02.666 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:02.698 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:02.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:02.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:02.800 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:02.804 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 51 16 71752 71736 87.8784 53.8125 0.00678431 0.0113724 2024-08-17T17:20:02.839 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:02.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:02.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:02.941 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:02.967 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:20:03.016 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:03.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:03.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:03.118 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:03.167 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:03.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:03.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:03.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:03.325 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:03.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:03.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:03.427 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:03.463 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:03.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:03.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:03.493 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:20:03.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:03 smithi003 ceph-mon[103344]: pgmap v268: 89 pgs: 1 active+recovering+undersized+remapped, 4 activating+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 remapped+peering, 1 unknown, 77 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 49 MiB/s wr, 785 op/s; 21500/205599 objects degraded (10.457%); 4132/205599 objects misplaced (2.010%); 468 KiB/s, 11 objects/s recovering 2024-08-17T17:20:03.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:03 smithi003 ceph-mon[100615]: pgmap v268: 89 pgs: 1 active+recovering+undersized+remapped, 4 activating+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 remapped+peering, 1 unknown, 77 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 49 MiB/s wr, 785 op/s; 21500/205599 objects degraded (10.457%); 4132/205599 objects misplaced (2.010%); 468 KiB/s, 11 objects/s recovering 2024-08-17T17:20:03.565 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:03.598 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:03.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:03.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:03.700 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:03.734 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:03.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:03.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:03.768 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:03 smithi053 ceph-mon[89963]: pgmap v268: 89 pgs: 1 active+recovering+undersized+remapped, 4 activating+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 remapped+peering, 1 unknown, 77 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 49 MiB/s wr, 785 op/s; 21500/205599 objects degraded (10.457%); 4132/205599 objects misplaced (2.010%); 468 KiB/s, 11 objects/s recovering 2024-08-17T17:20:03.804 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 52 16 72618 72602 87.2293 54.125 0.00645639 0.0114573 2024-08-17T17:20:03.836 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:03.865 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:03.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:03.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:03.967 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:03.996 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:03.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:03.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:04.098 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:04.126 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:04.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:04.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:04.208 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:20:04.228 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:04.280 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:04.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:04.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:04.291 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:20:04.291 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:20:04.327 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:20:04.327 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:20:04.366 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:20:04.366 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:20:04.382 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:04.400 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:20:04.400 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:20:04.423 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:04.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:04.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:04.428 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:20:04.428 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:20:04.464 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:20:04.464 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:20:04.498 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:20:04.498 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:20:04.525 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:04.528 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:20:04.528 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:20:04.554 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:04.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:04.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:04.626 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:04 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1655811320' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:20:04.627 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:04 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1655811320' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:20:04.656 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:04.706 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:04.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:04.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:04.806 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 53 15 73448 73433 86.5606 51.9375 0.0122507 0.0115468 2024-08-17T17:20:04.809 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:04.840 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:04.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:04.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:04.942 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:04.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:04 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1655811320' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:20:04.971 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:04.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:04.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:05.073 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:05.109 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:05.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:05.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:05.211 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:05.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:05.241 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:05.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:05.342 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:05.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:05.382 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:05.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:05.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:05.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:05.521 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:05.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:05.610 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:05 smithi003 ceph-mon[100615]: pgmap v269: 89 pgs: 2 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s; 29853/214182 objects degraded (13.938%); 8241/214182 objects misplaced (3.848%); 2.5 MiB/s, 44 objects/s recovering 2024-08-17T17:20:05.610 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:05 smithi003 ceph-mon[103344]: pgmap v269: 89 pgs: 2 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s; 29853/214182 objects degraded (13.938%); 8241/214182 objects misplaced (3.848%); 2.5 MiB/s, 44 objects/s recovering 2024-08-17T17:20:05.622 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:05.677 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:05.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:05.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:05.779 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:05.807 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 54 16 74334 74318 85.9807 55.3125 0.0350247 0.0116241 2024-08-17T17:20:05.816 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:05.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:05.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:05.912 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:20:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:20:05] "GET /metrics HTTP/1.1" 200 36523 "" "Prometheus/2.51.0" 2024-08-17T17:20:05.918 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:05.942 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:05.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:05.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:05.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:05 smithi053 ceph-mon[89963]: pgmap v269: 89 pgs: 2 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s; 29853/214182 objects degraded (13.938%); 8241/214182 objects misplaced (3.848%); 2.5 MiB/s, 44 objects/s recovering 2024-08-17T17:20:06.045 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:06.076 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:06.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:06.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:06.178 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:06.225 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:06.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:06.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:06.327 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:06.372 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:06.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:06.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:06.473 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:06.504 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:06.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:06.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:06.522 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-08-17T17:20:06.606 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:06.637 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:06.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:06.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:06.729 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:06 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2661064655' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:20:06.729 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:06 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2661064655' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:20:06.739 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:06.786 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:06.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:06.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:06.807 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 55 16 75157 75141 85.3528 51.4375 0.00858537 0.0117099 2024-08-17T17:20:06.888 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:06.935 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:06.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:06.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:06.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:06 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2661064655' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:20:07.036 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:07.065 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:07.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:07.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:07.167 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:07.202 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:07.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:07.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:07.304 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:07.350 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:07.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:07.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:07.452 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:07.493 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:07.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:07.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:07.595 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:07.623 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:07.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:07.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:07.660 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:20:07.725 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:07.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:07 smithi003 ceph-mon[100615]: pgmap v270: 89 pgs: 2 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 58 MiB/s wr, 929 op/s; 29853/218370 objects degraded (13.671%); 8056/218370 objects misplaced (3.689%); 3.6 MiB/s, 62 objects/s recovering 2024-08-17T17:20:07.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:07 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 29853/218370 objects degraded (13.671%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:07.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:07 smithi003 ceph-mon[100615]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:20:07.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:07 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2661064655' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:20:07.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:07 smithi003 ceph-mon[100615]: osdmap e134: 8 total, 8 up, 8 in 2024-08-17T17:20:07.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:07 smithi003 ceph-mon[103344]: pgmap v270: 89 pgs: 2 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 58 MiB/s wr, 929 op/s; 29853/218370 objects degraded (13.671%); 8056/218370 objects misplaced (3.689%); 3.6 MiB/s, 62 objects/s recovering 2024-08-17T17:20:07.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:07 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 29853/218370 objects degraded (13.671%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:07.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:07 smithi003 ceph-mon[103344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:20:07.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:07 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2661064655' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:20:07.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:07 smithi003 ceph-mon[103344]: osdmap e134: 8 total, 8 up, 8 in 2024-08-17T17:20:07.780 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:07.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:07.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:07.807 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 56 16 76045 76029 84.8198 55.5 0.0351079 0.0117836 2024-08-17T17:20:07.882 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:07.907 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:07.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:07.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:07.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:07 smithi053 ceph-mon[89963]: pgmap v270: 89 pgs: 2 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 58 MiB/s wr, 929 op/s; 29853/218370 objects degraded (13.671%); 8056/218370 objects misplaced (3.689%); 3.6 MiB/s, 62 objects/s recovering 2024-08-17T17:20:07.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:07 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 29853/218370 objects degraded (13.671%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:07.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:07 smithi053 ceph-mon[89963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:20:07.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:07 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2661064655' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:20:07.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:07 smithi053 ceph-mon[89963]: osdmap e134: 8 total, 8 up, 8 in 2024-08-17T17:20:08.009 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:08.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:08.032 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:08.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:08.133 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:08.174 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:08.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:08.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:08.275 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:08.321 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:08.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:08.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:08.423 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:08.460 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:08.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:08.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:08.562 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:08.595 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:08.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:08.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:08.697 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:08.740 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:08.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:08.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:08.807 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 57 16 76891 76875 84.2595 52.875 0.0112316 0.0118624 2024-08-17T17:20:08.842 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:08.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:08.870 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:08.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:08.971 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:09.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:09.018 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:09.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:09.119 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:09.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:09.167 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:09.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:09.268 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:09.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:09 smithi003 ceph-mon[100615]: pgmap v272: 89 pgs: 2 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 38 MiB/s wr, 608 op/s; 29853/219189 objects degraded (13.620%); 8056/219189 objects misplaced (3.675%); 3.0 MiB/s, 50 objects/s recovering 2024-08-17T17:20:09.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:20:09.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:20:09.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:09 smithi003 ceph-mon[103344]: pgmap v272: 89 pgs: 2 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 38 MiB/s wr, 608 op/s; 29853/219189 objects degraded (13.620%); 8056/219189 objects misplaced (3.675%); 3.0 MiB/s, 50 objects/s recovering 2024-08-17T17:20:09.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:20:09.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:20:09.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:09.312 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:09.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:09.413 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:09.455 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:09.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:09.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:09 smithi053 ceph-mon[89963]: pgmap v272: 89 pgs: 2 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 38 MiB/s wr, 608 op/s; 29853/219189 objects degraded (13.620%); 8056/219189 objects misplaced (3.675%); 3.0 MiB/s, 50 objects/s recovering 2024-08-17T17:20:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:20:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:20:09.558 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:09.594 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:09.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:09.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:09.696 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:09.735 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:09.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:09.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:09.807 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 58 16 77721 77705 83.7012 51.875 0.0109296 0.0119412 2024-08-17T17:20:09.837 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:09.874 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:09.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:09.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:09.976 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:10.024 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:10.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:10.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:10.126 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:10.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:10.159 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:10.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:10.260 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:10.282 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:10.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:10.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:10.384 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:10.425 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:10.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:10.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:10.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:20:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:20:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:20:10.527 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:10.568 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:10.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:10.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:10.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:10.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:10.719 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:10.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:10.808 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 59 16 78551 78535 83.1618 51.875 0.0503895 0.0120185 2024-08-17T17:20:10.820 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:10.858 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:10.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:10.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:10.960 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:11.004 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:11.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:11.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:11.106 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:11.151 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:11.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:11.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:11.252 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:11.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:11 smithi003 ceph-mon[100615]: pgmap v273: 89 pgs: 2 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 56 MiB/s wr, 892 op/s; 29853/229479 objects degraded (13.009%); 7672/229479 objects misplaced (3.343%); 5.2 MiB/s, 84 objects/s recovering 2024-08-17T17:20:11.276 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:11 smithi003 ceph-mon[103344]: pgmap v273: 89 pgs: 2 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 56 MiB/s wr, 892 op/s; 29853/229479 objects degraded (13.009%); 7672/229479 objects misplaced (3.343%); 5.2 MiB/s, 84 objects/s recovering 2024-08-17T17:20:11.297 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:11.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:11.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:11.399 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:11.435 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:11.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:11.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:11.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:11 smithi053 ceph-mon[89963]: pgmap v273: 89 pgs: 2 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 56 MiB/s wr, 892 op/s; 29853/229479 objects degraded (13.009%); 7672/229479 objects misplaced (3.343%); 5.2 MiB/s, 84 objects/s recovering 2024-08-17T17:20:11.537 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:11.602 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:11.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:11.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:11.705 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:11.749 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:11.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:11.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:11.801 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:20:11.808 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-08-17T17:20:11.808577+0000 min lat: 0.0011526 max lat: 2.93219 avg lat: 0.0120913 2024-08-17T17:20:11.808 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:20:11.808 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 60 16 79403 79387 82.6627 53.25 0.0294647 0.0120913 2024-08-17T17:20:11.852 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:11.880 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:11.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:11.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:11.981 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:12.029 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:12.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:12.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:12.131 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:12.167 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:12.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:12.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:12.269 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:12.297 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:12.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:12.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:12.364 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:12 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 29853/231030 objects degraded (12.922%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:12.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:12 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 29853/231030 objects degraded (12.922%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:12.399 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:12.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:12.447 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:12.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:12.452 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:20:12.452 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:20:12.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:12 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 29853/231030 objects degraded (12.922%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:12.548 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:12.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:12.581 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:12.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:12.610 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:20:12.682 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:12.716 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:12.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:12.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:12.808 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 61 16 80261 80245 82.1867 53.625 0.00987047 0.0121611 2024-08-17T17:20:12.819 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:12.859 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:12.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:12.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:12.960 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:13.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:13.005 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:13.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:13.106 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:13.129 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-17T17:20:13.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:13.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:13.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:13.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:13.356 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:13.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:13.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:13.458 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:13.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:13 smithi053 ceph-mon[89963]: pgmap v274: 89 pgs: 2 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 53 MiB/s wr, 847 op/s; 29853/231030 objects degraded (12.922%); 7672/231030 objects misplaced (3.321%); 4.8 MiB/s, 78 objects/s recovering 2024-08-17T17:20:13.469 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:13 smithi003 ceph-mon[100615]: pgmap v274: 89 pgs: 2 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 53 MiB/s wr, 847 op/s; 29853/231030 objects degraded (12.922%); 7672/231030 objects misplaced (3.321%); 4.8 MiB/s, 78 objects/s recovering 2024-08-17T17:20:13.469 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:13 smithi003 ceph-mon[103344]: pgmap v274: 89 pgs: 2 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.7 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 53 MiB/s wr, 847 op/s; 29853/231030 objects degraded (12.922%); 7672/231030 objects misplaced (3.321%); 4.8 MiB/s, 78 objects/s recovering 2024-08-17T17:20:13.484 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:13.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:13.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:13.586 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:13.638 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:13.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:13.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:13.740 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:13.773 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:13.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:13.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:13.810 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 62 16 80963 80947 81.567 43.875 0.0154165 0.0122512 2024-08-17T17:20:13.874 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:13.907 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:13.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:13.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:14.009 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:14.049 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:14.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:14.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:14.151 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:14.163 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.2 primary-affinity to 0.225212 (8147592) 2024-08-17T17:20:14.187 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:14.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:14.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:14.288 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:14.314 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:14.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:14.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:14.416 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:14.441 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:14.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:14.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:14.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:14 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:20:14.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:14 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1003380133' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.22521207160591294} : dispatch 2024-08-17T17:20:14.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:14 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.22521207160591294} : dispatch 2024-08-17T17:20:14.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:14 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:20:14.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:14 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1003380133' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.22521207160591294} : dispatch 2024-08-17T17:20:14.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:14 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.22521207160591294} : dispatch 2024-08-17T17:20:14.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:14 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:20:14.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:14 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1003380133' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.22521207160591294} : dispatch 2024-08-17T17:20:14.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:14 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.22521207160591294} : dispatch 2024-08-17T17:20:14.543 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:14.573 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:14.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:14.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:14.675 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:14.699 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:14.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:14.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:14.800 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:14.810 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 63 16 81387 81371 80.6931 26.5 0.0045154 0.0123119 2024-08-17T17:20:14.824 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:14.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:14.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:14.925 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:14.950 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:14.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:14.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:15.052 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:15.075 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:15.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:15.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:15.177 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:15.202 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:15.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:15.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:15.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:15 smithi003 ceph-mon[100615]: pgmap v275: 89 pgs: 2 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 53 MiB/s wr, 847 op/s; 29853/239631 objects degraded (12.458%); 7467/239631 objects misplaced (3.116%); 4.8 MiB/s, 77 objects/s recovering 2024-08-17T17:20:15.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:15 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.22521207160591294}]': finished 2024-08-17T17:20:15.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:15 smithi003 ceph-mon[100615]: osdmap e135: 8 total, 8 up, 8 in 2024-08-17T17:20:15.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:15 smithi003 ceph-mon[100615]: osdmap e136: 8 total, 8 up, 8 in 2024-08-17T17:20:15.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:15 smithi003 ceph-mon[103344]: pgmap v275: 89 pgs: 2 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 53 MiB/s wr, 847 op/s; 29853/239631 objects degraded (12.458%); 7467/239631 objects misplaced (3.116%); 4.8 MiB/s, 77 objects/s recovering 2024-08-17T17:20:15.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:15 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.22521207160591294}]': finished 2024-08-17T17:20:15.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:15 smithi003 ceph-mon[103344]: osdmap e135: 8 total, 8 up, 8 in 2024-08-17T17:20:15.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:15 smithi003 ceph-mon[103344]: osdmap e136: 8 total, 8 up, 8 in 2024-08-17T17:20:15.304 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:15.327 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:15.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:15.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:15.414 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:15 smithi053 ceph-mon[89963]: pgmap v275: 89 pgs: 2 active+recovering+undersized+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 53 MiB/s wr, 847 op/s; 29853/239631 objects degraded (12.458%); 7467/239631 objects misplaced (3.116%); 4.8 MiB/s, 77 objects/s recovering 2024-08-17T17:20:15.415 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:15 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.22521207160591294}]': finished 2024-08-17T17:20:15.415 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:15 smithi053 ceph-mon[89963]: osdmap e135: 8 total, 8 up, 8 in 2024-08-17T17:20:15.415 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:15 smithi053 ceph-mon[89963]: osdmap e136: 8 total, 8 up, 8 in 2024-08-17T17:20:15.429 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:15.452 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:15.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:15.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:15.554 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:15.598 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:15.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:15.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:15.700 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:15.747 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:15.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:15.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:15.810 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 64 16 81669 81653 79.708 17.625 0.0189635 0.0125403 2024-08-17T17:20:15.849 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:15.897 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:15.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:15.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:15.914 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:20:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:20:15] "GET /metrics HTTP/1.1" 200 36552 "" "Prometheus/2.51.0" 2024-08-17T17:20:16.000 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:16.048 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:16.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:16.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:16.150 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:16.181 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:16.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:16.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:16.283 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:16.320 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:16.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:16.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:16.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:16.467 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:16.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:16.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:16.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:16 smithi003 ceph-mon[103344]: osdmap e137: 8 total, 8 up, 8 in 2024-08-17T17:20:16.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:16 smithi003 ceph-mon[100615]: osdmap e137: 8 total, 8 up, 8 in 2024-08-17T17:20:16.569 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:16.646 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:16.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:16.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:16.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:16 smithi053 ceph-mon[89963]: osdmap e137: 8 total, 8 up, 8 in 2024-08-17T17:20:16.748 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:16.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:16.800 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:16.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:16.812 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 65 16 82478 82462 79.2573 50.5625 0.0199801 0.0126112 2024-08-17T17:20:16.902 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:16.942 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:16.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:16.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:17.044 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:17.071 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:17.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:17.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:17.173 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:17.226 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:17.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:17.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:17.329 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:17.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:17.375 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:17.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:17.465 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:20:17.466 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:20:17.466 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:20:17.466 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:20:17.466 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:20:17.466 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:20:17.466 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:20:17.466 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:20:17.466 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:20:17.466 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:20:17.471 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:20:17.471 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:20:17.476 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:17.503 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:20:17.504 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:20:17.514 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:17.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:17.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:17.535 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:20:17.536 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:20:17.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:17.660 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:17.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:17.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:17.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:17 smithi053 ceph-mon[89963]: pgmap v279: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 47 MiB/s wr, 755 op/s; 29853/243093 objects degraded (12.280%); 7261/243093 objects misplaced (2.987%); 4.3 MiB/s, 68 objects/s recovering 2024-08-17T17:20:17.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:17 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3362812932' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:20:17.762 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:17.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:17 smithi003 ceph-mon[100615]: pgmap v279: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 47 MiB/s wr, 755 op/s; 29853/243093 objects degraded (12.280%); 7261/243093 objects misplaced (2.987%); 4.3 MiB/s, 68 objects/s recovering 2024-08-17T17:20:17.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:17 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3362812932' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:20:17.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:17 smithi003 ceph-mon[103344]: pgmap v279: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 47 MiB/s wr, 755 op/s; 29853/243093 objects degraded (12.280%); 7261/243093 objects misplaced (2.987%); 4.3 MiB/s, 68 objects/s recovering 2024-08-17T17:20:17.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:17 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3362812932' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:20:17.792 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:17.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:17.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:17.812 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 66 16 83211 83195 78.7506 45.8125 0.0115279 0.012692 2024-08-17T17:20:17.894 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:17.943 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:17.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:17.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:18.045 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:18.085 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:18.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:18.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:18.187 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:18.223 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:18.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:18.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:18.248 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:20:18.299 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:20:18.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:18.386 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:18.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:18.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:18.488 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:18.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:18.546 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:18.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:18.647 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:18.686 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:18.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:18.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:18.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:18 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 29853/243960 objects degraded (12.237%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:18.687 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:18 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 29853/243960 objects degraded (12.237%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:18.788 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:18.812 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 67 16 83975 83959 78.288 47.75 0.00682987 0.0127661 2024-08-17T17:20:18.821 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:18.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:18.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:18.923 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:18.954 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:18.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:18.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:18.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:18 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 29853/243960 objects degraded (12.237%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:19.056 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:19.097 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:19.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:19.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:19.200 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:19.236 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:19.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:19.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:19.337 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:19.371 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:19.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:19.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:19.473 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:19.516 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:19.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:19.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:19.618 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:19.653 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:19.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:19.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:19.754 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:19.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:19 smithi003 ceph-mon[100615]: pgmap v280: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 45 MiB/s wr, 717 op/s; 29853/243960 objects degraded (12.237%); 7261/243960 objects misplaced (2.976%); 4.3 MiB/s, 68 objects/s recovering 2024-08-17T17:20:19.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:19 smithi003 ceph-mon[103344]: pgmap v280: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 45 MiB/s wr, 717 op/s; 29853/243960 objects degraded (12.237%); 7261/243960 objects misplaced (2.976%); 4.3 MiB/s, 68 objects/s recovering 2024-08-17T17:20:19.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:19.815 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:19.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:19.815 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 68 16 84860 84844 77.9468 55.3125 0.0156226 0.0128222 2024-08-17T17:20:19.916 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:19.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:19 smithi053 ceph-mon[89963]: pgmap v280: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 45 MiB/s wr, 717 op/s; 29853/243960 objects degraded (12.237%); 7261/243960 objects misplaced (2.976%); 4.3 MiB/s, 68 objects/s recovering 2024-08-17T17:20:19.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:19.979 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:19.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:20.080 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:20.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:20.113 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:20.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:20.214 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:20.254 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:20.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:20.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:20.356 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:20.393 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:20.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:20.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:20.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:20:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:20:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:20:20.496 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:20.540 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:20.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:20.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:20.641 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:20.687 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:20.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:20.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:20.789 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:20.817 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:20.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:20.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:20.819 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 69 16 85747 85731 77.6161 55.4375 0.0177063 0.0128786 2024-08-17T17:20:20.842 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:20 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:20:20.842 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:20 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:20:20.842 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:20 smithi003 ceph-mon[100615]: pgmap v281: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 38 MiB/s wr, 602 op/s; 29853/250488 objects degraded (11.918%); 6720/250488 objects misplaced (2.683%); 7.8 MiB/s, 124 objects/s recovering 2024-08-17T17:20:20.842 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:20 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:20:20.842 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:20 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:20:20.842 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:20 smithi003 ceph-mon[103344]: pgmap v281: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 38 MiB/s wr, 602 op/s; 29853/250488 objects degraded (11.918%); 6720/250488 objects misplaced (2.683%); 7.8 MiB/s, 124 objects/s recovering 2024-08-17T17:20:20.919 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:20.948 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:20.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:20.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:20.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:20 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:20:20.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:20 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:20:20.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:20 smithi053 ceph-mon[89963]: pgmap v281: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 38 MiB/s wr, 602 op/s; 29853/250488 objects degraded (11.918%); 6720/250488 objects misplaced (2.683%); 7.8 MiB/s, 124 objects/s recovering 2024-08-17T17:20:21.021 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:20:21.050 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:21.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:21.134 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:21.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:21.235 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:21.265 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:21.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:21.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:21.367 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:21.395 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:21.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:21.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:21.496 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:21.553 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:21.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:21.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:21.587 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-17T17:20:21.588 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:20:21.588 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-08-17T17:20:21.588 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-17T17:20:21.588 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-17T17:20:21.588 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-08-17T17:20:21.655 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:21.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:21.705 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:21.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:21.806 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:21.819 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 70 16 86568 86552 77.2404 51.3125 0.0137666 0.0129411 2024-08-17T17:20:21.850 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:21.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:21.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:21.952 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:21.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:21.993 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:21.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:22.025 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:20:22.094 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:22.123 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:22.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:22.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:22.225 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:22.255 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:22.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:22.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:22.357 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:22.388 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:22.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:22.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:22.489 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:22.524 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:22.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:22.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:22.625 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:22.651 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:22.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:22.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:22.754 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:22.783 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:22.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:22.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:22.820 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 71 16 87353 87337 76.8436 49.0625 0.0152555 0.0130074 2024-08-17T17:20:22.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:22.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:22.928 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:22.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:22.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:22 smithi003 ceph-mon[100615]: pgmap v282: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 36 MiB/s wr, 578 op/s; 29853/252765 objects degraded (11.811%); 6720/252765 objects misplaced (2.659%); 6.2 MiB/s, 98 objects/s recovering 2024-08-17T17:20:22.986 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:22 smithi003 ceph-mon[103344]: pgmap v282: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 36 MiB/s wr, 578 op/s; 29853/252765 objects degraded (11.811%); 6720/252765 objects misplaced (2.659%); 6.2 MiB/s, 98 objects/s recovering 2024-08-17T17:20:23.029 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:23.073 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:23.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:23.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:23.174 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:23.205 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:22 smithi053 ceph-mon[89963]: pgmap v282: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 36 MiB/s wr, 578 op/s; 29853/252765 objects degraded (11.811%); 6720/252765 objects misplaced (2.659%); 6.2 MiB/s, 98 objects/s recovering 2024-08-17T17:20:23.215 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:23.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:23.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:23.317 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:23.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:23.365 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:23.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:23.466 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:23.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:23.511 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:23.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:23.612 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:23.637 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:20:23.637 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-08-17T17:20:23.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-08-17T17:20:23.637 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:20:23.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.637 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:58518 5.6 5:68c260f3:::benchmark_data_smithi003_140793_object58517:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-17T17:20:23.637 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:48.354524+0000", 2024-08-17T17:20:23.637 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 35.273887215000002, 2024-08-17T17:20:23.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.10235517099999999, 2024-08-17T17:20:23.638 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:23.638 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:23.638 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:23.638 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:23.638 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:23.638 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:23.638 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 58518 2024-08-17T17:20:23.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.638 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:23.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:23.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:48.354524+0000", 2024-08-17T17:20:23.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:23.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:48.354524+0000", 2024-08-17T17:20:23.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:23.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:48.354525+0000", 2024-08-17T17:20:23.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0729999999999999e-06 2024-08-17T17:20:23.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:23.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:48.354547+0000", 2024-08-17T17:20:23.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2217000000000001e-05 2024-08-17T17:20:23.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:23.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:48.354555+0000", 2024-08-17T17:20:23.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0949999999999996e-06 2024-08-17T17:20:23.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:23.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:48.354561+0000", 2024-08-17T17:20:23.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4450000000000004e-06 2024-08-17T17:20:23.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:48.362524+0000", 2024-08-17T17:20:23.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0079633140000000009 2024-08-17T17:20:23.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:23.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:48.362673+0000", 2024-08-17T17:20:23.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014918599999999999 2024-08-17T17:20:23.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-08-17T17:20:23.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:48.362803+0000", 2024-08-17T17:20:23.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013000199999999999 2024-08-17T17:20:23.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:23.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:48.367908+0000", 2024-08-17T17:20:23.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0051051330000000004 2024-08-17T17:20:23.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:23.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:48.368220+0000", 2024-08-17T17:20:23.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00031175400000000002 2024-08-17T17:20:23.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:23.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:48.456785+0000", 2024-08-17T17:20:23.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.088564670999999998 2024-08-17T17:20:23.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:23.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:48.456838+0000", 2024-08-17T17:20:23.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3350999999999998e-05 2024-08-17T17:20:23.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:23.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:48.456879+0000", 2024-08-17T17:20:23.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0930000000000003e-05 2024-08-17T17:20:23.644 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.644 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:23.644 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.644 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:59534 5.d 5:b8837190:::benchmark_data_smithi003_140793_object59533:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-17T17:20:23.644 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:49.246712+0000", 2024-08-17T17:20:23.644 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 34.381698989, 2024-08-17T17:20:23.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.102003304, 2024-08-17T17:20:23.644 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:23.644 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:23.644 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:23.644 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:23.644 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:23.645 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:23.645 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 59534 2024-08-17T17:20:23.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.645 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:23.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:23.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:49.246712+0000", 2024-08-17T17:20:23.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:23.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:49.246712+0000", 2024-08-17T17:20:23.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:23.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:49.246713+0000", 2024-08-17T17:20:23.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.02e-07 2024-08-17T17:20:23.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:23.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:49.246733+0000", 2024-08-17T17:20:23.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0276e-05 2024-08-17T17:20:23.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:23.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:49.246744+0000", 2024-08-17T17:20:23.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0625000000000001e-05 2024-08-17T17:20:23.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:23.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:49.246750+0000", 2024-08-17T17:20:23.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4659999999999996e-06 2024-08-17T17:20:23.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:49.249429+0000", 2024-08-17T17:20:23.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026784489999999998 2024-08-17T17:20:23.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:23.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:49.249546+0000", 2024-08-17T17:20:23.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000117732 2024-08-17T17:20:23.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,7", 2024-08-17T17:20:23.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:49.249679+0000", 2024-08-17T17:20:23.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000132834 2024-08-17T17:20:23.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:23.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:49.251569+0000", 2024-08-17T17:20:23.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0018900119999999999 2024-08-17T17:20:23.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:23.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:49.254391+0000", 2024-08-17T17:20:23.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002821502 2024-08-17T17:20:23.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:23.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:49.348614+0000", 2024-08-17T17:20:23.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.094222809000000005 2024-08-17T17:20:23.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:23.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:49.348666+0000", 2024-08-17T17:20:23.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2354000000000001e-05 2024-08-17T17:20:23.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:23.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:49.348715+0000", 2024-08-17T17:20:23.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9342999999999998e-05 2024-08-17T17:20:23.650 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.650 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:23.650 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.650 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:59548 5.b 5:d2f017de:::benchmark_data_smithi003_140793_object59547:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-17T17:20:23.651 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:49.257962+0000", 2024-08-17T17:20:23.651 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 34.370448574000001, 2024-08-17T17:20:23.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.10880440299999999, 2024-08-17T17:20:23.651 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:23.651 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:23.651 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:23.651 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:23.651 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:23.651 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:23.651 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 59548 2024-08-17T17:20:23.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.651 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:23.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:23.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:49.257962+0000", 2024-08-17T17:20:23.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:23.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:49.257962+0000", 2024-08-17T17:20:23.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:23.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:49.257963+0000", 2024-08-17T17:20:23.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5199999999999995e-07 2024-08-17T17:20:23.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:23.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:49.257983+0000", 2024-08-17T17:20:23.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0143999999999999e-05 2024-08-17T17:20:23.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:23.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:49.257995+0000", 2024-08-17T17:20:23.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1875999999999999e-05 2024-08-17T17:20:23.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:23.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:49.258004+0000", 2024-08-17T17:20:23.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7180000000000002e-06 2024-08-17T17:20:23.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:49.258057+0000", 2024-08-17T17:20:23.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3182e-05 2024-08-17T17:20:23.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:23.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:49.348314+0000", 2024-08-17T17:20:23.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.090257067999999996 2024-08-17T17:20:23.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:20:23.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:49.348460+0000", 2024-08-17T17:20:23.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014584999999999999 2024-08-17T17:20:23.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:23.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:49.351413+0000", 2024-08-17T17:20:23.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029532959999999998 2024-08-17T17:20:23.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:23.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:49.351999+0000", 2024-08-17T17:20:23.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00058608399999999995 2024-08-17T17:20:23.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:23.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:49.366664+0000", 2024-08-17T17:20:23.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.014664791999999999 2024-08-17T17:20:23.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:23.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:49.366722+0000", 2024-08-17T17:20:23.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7528999999999998e-05 2024-08-17T17:20:23.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:23.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:49.366767+0000", 2024-08-17T17:20:23.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5012000000000002e-05 2024-08-17T17:20:23.657 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.657 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:23.657 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.657 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:60499 5.a 5:58886896:::benchmark_data_smithi003_140793_object60498:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-17T17:20:23.657 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:50.027066+0000", 2024-08-17T17:20:23.657 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 33.601344587, 2024-08-17T17:20:23.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.108570272, 2024-08-17T17:20:23.657 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:23.657 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:23.657 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:23.657 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:23.657 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:23.658 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:23.658 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 60499 2024-08-17T17:20:23.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.658 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:23.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:23.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.027066+0000", 2024-08-17T17:20:23.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:23.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.027066+0000", 2024-08-17T17:20:23.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:23.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.027067+0000", 2024-08-17T17:20:23.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7599999999999996e-07 2024-08-17T17:20:23.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:23.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.029059+0000", 2024-08-17T17:20:23.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0019917839999999999 2024-08-17T17:20:23.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:23.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.029078+0000", 2024-08-17T17:20:23.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9164000000000002e-05 2024-08-17T17:20:23.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:23.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.029085+0000", 2024-08-17T17:20:23.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2969999999999998e-06 2024-08-17T17:20:23.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.029415+0000", 2024-08-17T17:20:23.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00032918699999999999 2024-08-17T17:20:23.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:23.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.029537+0000", 2024-08-17T17:20:23.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012220800000000001 2024-08-17T17:20:23.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,3", 2024-08-17T17:20:23.661 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.029663+0000", 2024-08-17T17:20:23.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000126184 2024-08-17T17:20:23.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:23.661 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.035456+0000", 2024-08-17T17:20:23.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0057928049999999998 2024-08-17T17:20:23.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:23.661 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.046986+0000", 2024-08-17T17:20:23.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011530452 2024-08-17T17:20:23.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:23.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.135544+0000", 2024-08-17T17:20:23.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.088558126000000001 2024-08-17T17:20:23.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:23.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.135589+0000", 2024-08-17T17:20:23.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4286999999999998e-05 2024-08-17T17:20:23.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:23.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.135637+0000", 2024-08-17T17:20:23.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8001999999999999e-05 2024-08-17T17:20:23.663 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.663 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:23.663 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.663 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:60504 5.b 5:d27c2726:::benchmark_data_smithi003_140793_object60503:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-17T17:20:23.663 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:50.031898+0000", 2024-08-17T17:20:23.663 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 33.596512642, 2024-08-17T17:20:23.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.10755693400000001, 2024-08-17T17:20:23.663 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:23.664 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:23.664 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:23.664 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:23.664 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:23.664 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:23.664 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 60504 2024-08-17T17:20:23.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.664 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:23.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:23.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.031898+0000", 2024-08-17T17:20:23.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:23.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.031898+0000", 2024-08-17T17:20:23.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:23.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.031900+0000", 2024-08-17T17:20:23.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.221e-06 2024-08-17T17:20:23.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:23.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.031924+0000", 2024-08-17T17:20:23.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4253999999999999e-05 2024-08-17T17:20:23.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:23.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.031933+0000", 2024-08-17T17:20:23.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3170000000000008e-06 2024-08-17T17:20:23.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:23.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.031940+0000", 2024-08-17T17:20:23.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8700000000000003e-06 2024-08-17T17:20:23.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.035078+0000", 2024-08-17T17:20:23.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003137955 2024-08-17T17:20:23.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:23.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.035264+0000", 2024-08-17T17:20:23.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000186131 2024-08-17T17:20:23.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:20:23.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.035389+0000", 2024-08-17T17:20:23.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012485400000000001 2024-08-17T17:20:23.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:23.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.041431+0000", 2024-08-17T17:20:23.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0060423209999999998 2024-08-17T17:20:23.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:23.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.042586+0000", 2024-08-17T17:20:23.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0011547199999999999 2024-08-17T17:20:23.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:23.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.139366+0000", 2024-08-17T17:20:23.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.096779717000000001 2024-08-17T17:20:23.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:23.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.139413+0000", 2024-08-17T17:20:23.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7026e-05 2024-08-17T17:20:23.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:23.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.139455+0000", 2024-08-17T17:20:23.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2548e-05 2024-08-17T17:20:23.669 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.669 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:23.670 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.670 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:60505 5.a 5:532026a0:::benchmark_data_smithi003_140793_object60504:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-17T17:20:23.670 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:50.031955+0000", 2024-08-17T17:20:23.670 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 33.596456261, 2024-08-17T17:20:23.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.103736547, 2024-08-17T17:20:23.670 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:23.670 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:23.670 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:23.670 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:23.670 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:23.670 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:23.671 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 60505 2024-08-17T17:20:23.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.671 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:23.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:23.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.031955+0000", 2024-08-17T17:20:23.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:23.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.031955+0000", 2024-08-17T17:20:23.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.672 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:23.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.031955+0000", 2024-08-17T17:20:23.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2500000000000005e-07 2024-08-17T17:20:23.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.672 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:23.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.031980+0000", 2024-08-17T17:20:23.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4363e-05 2024-08-17T17:20:23.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:23.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.031987+0000", 2024-08-17T17:20:23.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7940000000000003e-06 2024-08-17T17:20:23.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:23.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.031994+0000", 2024-08-17T17:20:23.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0460000000000002e-06 2024-08-17T17:20:23.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.035491+0000", 2024-08-17T17:20:23.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034970169999999998 2024-08-17T17:20:23.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:23.674 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.035615+0000", 2024-08-17T17:20:23.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012399700000000001 2024-08-17T17:20:23.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,3", 2024-08-17T17:20:23.674 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.035749+0000", 2024-08-17T17:20:23.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013421599999999999 2024-08-17T17:20:23.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:23.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.047023+0000", 2024-08-17T17:20:23.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011274246 2024-08-17T17:20:23.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:23.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.047060+0000", 2024-08-17T17:20:23.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7026000000000001e-05 2024-08-17T17:20:23.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:23.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.135645+0000", 2024-08-17T17:20:23.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.088585021 2024-08-17T17:20:23.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:23.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.135673+0000", 2024-08-17T17:20:23.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8126000000000001e-05 2024-08-17T17:20:23.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:23.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.135691+0000", 2024-08-17T17:20:23.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8070000000000001e-05 2024-08-17T17:20:23.676 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.676 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:23.676 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.677 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:60529 5.4 5:2c90e3b9:::benchmark_data_smithi003_140793_object60528:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-17T17:20:23.677 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:50.050420+0000", 2024-08-17T17:20:23.677 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 33.577991427999997, 2024-08-17T17:20:23.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.109017616, 2024-08-17T17:20:23.677 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:23.677 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:23.677 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:23.677 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:23.677 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:23.677 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:23.677 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 60529 2024-08-17T17:20:23.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.677 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:23.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:23.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.050420+0000", 2024-08-17T17:20:23.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:23.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.050420+0000", 2024-08-17T17:20:23.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:23.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.050420+0000", 2024-08-17T17:20:23.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4099999999999997e-07 2024-08-17T17:20:23.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:23.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.050440+0000", 2024-08-17T17:20:23.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9922999999999999e-05 2024-08-17T17:20:23.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:23.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.135782+0000", 2024-08-17T17:20:23.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.085341673000000007 2024-08-17T17:20:23.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:23.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.135792+0000", 2024-08-17T17:20:23.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6719999999999999e-06 2024-08-17T17:20:23.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.142648+0000", 2024-08-17T17:20:23.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0068557959999999999 2024-08-17T17:20:23.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:23.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.142757+0000", 2024-08-17T17:20:23.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010984699999999999 2024-08-17T17:20:23.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:20:23.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.142889+0000", 2024-08-17T17:20:23.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013205500000000001 2024-08-17T17:20:23.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:23.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.149342+0000", 2024-08-17T17:20:23.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0064527880000000001 2024-08-17T17:20:23.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:23.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.158693+0000", 2024-08-17T17:20:23.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0093503389999999992 2024-08-17T17:20:23.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:23.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.159342+0000", 2024-08-17T17:20:23.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00064970299999999996 2024-08-17T17:20:23.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:23.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.159394+0000", 2024-08-17T17:20:23.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2077000000000003e-05 2024-08-17T17:20:23.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:23.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.159437+0000", 2024-08-17T17:20:23.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2802000000000001e-05 2024-08-17T17:20:23.683 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.683 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:23.684 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.684 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:61360 5.d 5:bd89d4c5:::benchmark_data_smithi003_140793_object61359:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-17T17:20:23.684 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:50.778781+0000", 2024-08-17T17:20:23.684 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 32.849629749000002, 2024-08-17T17:20:23.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.131254864, 2024-08-17T17:20:23.684 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:23.684 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:23.684 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:23.684 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:23.684 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:23.684 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:23.685 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 61360 2024-08-17T17:20:23.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.685 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:23.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:23.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.778781+0000", 2024-08-17T17:20:23.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:23.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.778781+0000", 2024-08-17T17:20:23.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:23.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.778783+0000", 2024-08-17T17:20:23.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8419999999999999e-06 2024-08-17T17:20:23.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:23.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.778808+0000", 2024-08-17T17:20:23.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4810000000000001e-05 2024-08-17T17:20:23.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:23.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.778829+0000", 2024-08-17T17:20:23.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1569999999999998e-05 2024-08-17T17:20:23.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:23.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.778837+0000", 2024-08-17T17:20:23.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3119999999999996e-06 2024-08-17T17:20:23.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.778900+0000", 2024-08-17T17:20:23.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3577999999999993e-05 2024-08-17T17:20:23.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:23.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.779023+0000", 2024-08-17T17:20:23.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012223900000000001 2024-08-17T17:20:23.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,7", 2024-08-17T17:20:23.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.779325+0000", 2024-08-17T17:20:23.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00030238200000000002 2024-08-17T17:20:23.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:23.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.781873+0000", 2024-08-17T17:20:23.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025477109999999998 2024-08-17T17:20:23.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:23.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.782213+0000", 2024-08-17T17:20:23.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00034041099999999999 2024-08-17T17:20:23.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:23.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.909936+0000", 2024-08-17T17:20:23.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.12772302699999999 2024-08-17T17:20:23.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:23.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.909997+0000", 2024-08-17T17:20:23.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0609000000000002e-05 2024-08-17T17:20:23.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:23.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.910036+0000", 2024-08-17T17:20:23.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9372999999999997e-05 2024-08-17T17:20:23.690 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.690 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:23.690 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.691 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:61368 5.d 5:bf015eb7:::benchmark_data_smithi003_140793_object61367:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-17T17:20:23.691 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:50.784270+0000", 2024-08-17T17:20:23.691 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 32.844140842999998, 2024-08-17T17:20:23.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.132675028, 2024-08-17T17:20:23.691 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:23.691 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:23.691 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:23.691 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:23.691 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:23.691 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:23.691 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 61368 2024-08-17T17:20:23.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.691 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:23.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:23.692 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.784270+0000", 2024-08-17T17:20:23.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:23.692 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.784270+0000", 2024-08-17T17:20:23.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:23.692 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.784272+0000", 2024-08-17T17:20:23.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.403e-06 2024-08-17T17:20:23.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:23.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.784315+0000", 2024-08-17T17:20:23.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3727999999999999e-05 2024-08-17T17:20:23.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:23.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.784327+0000", 2024-08-17T17:20:23.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1695000000000001e-05 2024-08-17T17:20:23.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:23.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.784334+0000", 2024-08-17T17:20:23.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6379999999999998e-06 2024-08-17T17:20:23.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.784405+0000", 2024-08-17T17:20:23.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1027999999999998e-05 2024-08-17T17:20:23.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:23.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.784524+0000", 2024-08-17T17:20:23.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000119102 2024-08-17T17:20:23.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,7", 2024-08-17T17:20:23.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.784647+0000", 2024-08-17T17:20:23.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012282300000000001 2024-08-17T17:20:23.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:23.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.787272+0000", 2024-08-17T17:20:23.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002625063 2024-08-17T17:20:23.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:23.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.787308+0000", 2024-08-17T17:20:23.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6822000000000001e-05 2024-08-17T17:20:23.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:23.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.916824+0000", 2024-08-17T17:20:23.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.12951596300000001 2024-08-17T17:20:23.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:23.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.916900+0000", 2024-08-17T17:20:23.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6077999999999999e-05 2024-08-17T17:20:23.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:23.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.916945+0000", 2024-08-17T17:20:23.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4685000000000003e-05 2024-08-17T17:20:23.697 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.697 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:23.697 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.697 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:61374 5.8 5:1cffd916:::benchmark_data_smithi003_140793_object61373:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-17T17:20:23.697 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:50.787184+0000", 2024-08-17T17:20:23.697 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 32.841227230999998, 2024-08-17T17:20:23.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.11507941100000001, 2024-08-17T17:20:23.697 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:23.697 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:23.697 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:23.698 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:23.698 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:23.698 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:23.698 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 61374 2024-08-17T17:20:23.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.698 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:23.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:23.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.787184+0000", 2024-08-17T17:20:23.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:23.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.787184+0000", 2024-08-17T17:20:23.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:23.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.787209+0000", 2024-08-17T17:20:23.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5599e-05 2024-08-17T17:20:23.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:23.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.787273+0000", 2024-08-17T17:20:23.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3734999999999999e-05 2024-08-17T17:20:23.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.700 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:23.700 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.787285+0000", 2024-08-17T17:20:23.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1514e-05 2024-08-17T17:20:23.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.700 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:23.700 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.787292+0000", 2024-08-17T17:20:23.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8730000000000007e-06 2024-08-17T17:20:23.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.700 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.700 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.787510+0000", 2024-08-17T17:20:23.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000217421 2024-08-17T17:20:23.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.701 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:23.701 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.787640+0000", 2024-08-17T17:20:23.701 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013038200000000001 2024-08-17T17:20:23.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.701 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:20:23.701 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.787758+0000", 2024-08-17T17:20:23.701 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000117755 2024-08-17T17:20:23.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.701 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:23.701 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.794094+0000", 2024-08-17T17:20:23.701 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063357509999999997 2024-08-17T17:20:23.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:23.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.794535+0000", 2024-08-17T17:20:23.702 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00044143399999999999 2024-08-17T17:20:23.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:23.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.902140+0000", 2024-08-17T17:20:23.702 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.107604831 2024-08-17T17:20:23.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:23.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.902209+0000", 2024-08-17T17:20:23.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8937999999999996e-05 2024-08-17T17:20:23.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.703 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:23.703 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.902263+0000", 2024-08-17T17:20:23.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4178000000000002e-05 2024-08-17T17:20:23.703 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.703 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:23.703 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.703 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:61384 5.d 5:b67f0d5e:::benchmark_data_smithi003_140793_object61383:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-17T17:20:23.703 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:50.794595+0000", 2024-08-17T17:20:23.703 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 32.833815696999999, 2024-08-17T17:20:23.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.123014096, 2024-08-17T17:20:23.704 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:23.704 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:23.704 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:23.704 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:23.704 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:23.704 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:23.704 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 61384 2024-08-17T17:20:23.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.704 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:23.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.704 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:23.704 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.794595+0000", 2024-08-17T17:20:23.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.705 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:23.705 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.794595+0000", 2024-08-17T17:20:23.705 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.705 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:23.705 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.794597+0000", 2024-08-17T17:20:23.705 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.761e-06 2024-08-17T17:20:23.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.705 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:23.705 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.794620+0000", 2024-08-17T17:20:23.706 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3268000000000002e-05 2024-08-17T17:20:23.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.706 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:23.706 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.794631+0000", 2024-08-17T17:20:23.706 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0743e-05 2024-08-17T17:20:23.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.706 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:23.706 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.794638+0000", 2024-08-17T17:20:23.706 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0820000000000001e-06 2024-08-17T17:20:23.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.706 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.707 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.794697+0000", 2024-08-17T17:20:23.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8588999999999998e-05 2024-08-17T17:20:23.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:23.707 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.794834+0000", 2024-08-17T17:20:23.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000137553 2024-08-17T17:20:23.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,7", 2024-08-17T17:20:23.707 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.794973+0000", 2024-08-17T17:20:23.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013865899999999999 2024-08-17T17:20:23.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.708 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:23.708 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.796711+0000", 2024-08-17T17:20:23.708 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017377099999999999 2024-08-17T17:20:23.708 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.708 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.708 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:23.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:23.709 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:23.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:23.710 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.797072+0000", 2024-08-17T17:20:23.710 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00036177799999999999 2024-08-17T17:20:23.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.710 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.710 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:23.710 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.917510+0000", 2024-08-17T17:20:23.710 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.12043799500000001 2024-08-17T17:20:23.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.711 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.711 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:23.711 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.917574+0000", 2024-08-17T17:20:23.711 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3126999999999994e-05 2024-08-17T17:20:23.711 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.711 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.711 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:23.711 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.917609+0000", 2024-08-17T17:20:23.711 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5831000000000003e-05 2024-08-17T17:20:23.711 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.711 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:23.711 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.711 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.712 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:68131 5.a 5:5eb7e0b1:::benchmark_data_smithi003_140793_object68130:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-17T17:20:23.712 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:56.473603+0000", 2024-08-17T17:20:23.712 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.154807778999999, 2024-08-17T17:20:23.712 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3252823510000002, 2024-08-17T17:20:23.712 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:23.712 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:23.712 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:20:23.712 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:23.712 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:23.712 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:23.712 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68131 2024-08-17T17:20:23.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.712 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:23.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.713 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:23.713 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.473603+0000", 2024-08-17T17:20:23.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.713 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:23.713 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.473603+0000", 2024-08-17T17:20:23.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.713 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:23.713 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.473605+0000", 2024-08-17T17:20:23.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4100000000000001e-06 2024-08-17T17:20:23.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.714 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.714 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:23.714 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.473630+0000", 2024-08-17T17:20:23.714 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5717000000000001e-05 2024-08-17T17:20:23.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.714 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.714 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:23.714 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.473640+0000", 2024-08-17T17:20:23.714 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7869999999999996e-06 2024-08-17T17:20:23.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.714 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.714 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:23.715 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.473646+0000", 2024-08-17T17:20:23.715 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5099999999999998e-06 2024-08-17T17:20:23.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.715 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.715 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.480848+0000", 2024-08-17T17:20:23.715 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072023449999999998 2024-08-17T17:20:23.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.715 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:23.715 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.480987+0000", 2024-08-17T17:20:23.715 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013917199999999999 2024-08-17T17:20:23.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.716 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.716 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,3", 2024-08-17T17:20:23.716 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.481131+0000", 2024-08-17T17:20:23.716 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014377899999999999 2024-08-17T17:20:23.716 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.716 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.716 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.716 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.507492+0000", 2024-08-17T17:20:23.716 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.026361190999999999 2024-08-17T17:20:23.716 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.716 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.716 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:23.716 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.507503+0000", 2024-08-17T17:20:23.717 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1277000000000001e-05 2024-08-17T17:20:23.717 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.717 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.717 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.717 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.481848+0000", 2024-08-17T17:20:23.717 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97434455600000003 2024-08-17T17:20:23.717 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.717 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.717 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:23.717 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.481865+0000", 2024-08-17T17:20:23.717 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6639000000000001e-05 2024-08-17T17:20:23.717 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.717 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.718 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.718 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.778427+0000", 2024-08-17T17:20:23.718 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.296562913 2024-08-17T17:20:23.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.718 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for degraded object", 2024-08-17T17:20:23.718 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.779493+0000", 2024-08-17T17:20:23.718 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001065178 2024-08-17T17:20:23.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.718 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.718 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.798772+0000", 2024-08-17T17:20:23.718 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.019279580000000001 2024-08-17T17:20:23.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.719 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:23.719 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.798886+0000", 2024-08-17T17:20:23.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000113297 2024-08-17T17:20:23.719 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.719 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:23.719 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.719 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:68134 5.a 5:507a5fcd:::benchmark_data_smithi003_140793_object68133:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-08-17T17:20:23.719 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:56.497317+0000", 2024-08-17T17:20:23.719 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.131094049000001, 2024-08-17T17:20:23.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3050668550000002, 2024-08-17T17:20:23.720 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:23.720 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:23.720 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:23.720 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:23.720 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:23.720 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:23.720 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68134 2024-08-17T17:20:23.720 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.720 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:23.720 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.720 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:23.720 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.497317+0000", 2024-08-17T17:20:23.720 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.720 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:23.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.497317+0000", 2024-08-17T17:20:23.721 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:23.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.497319+0000", 2024-08-17T17:20:23.721 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.64e-06 2024-08-17T17:20:23.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:23.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.497345+0000", 2024-08-17T17:20:23.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6248e-05 2024-08-17T17:20:23.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.722 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:23.722 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.497357+0000", 2024-08-17T17:20:23.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.184e-05 2024-08-17T17:20:23.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.722 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:23.722 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.497366+0000", 2024-08-17T17:20:23.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2499999999999995e-06 2024-08-17T17:20:23.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.722 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.723 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.507534+0000", 2024-08-17T17:20:23.723 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010168089999999999 2024-08-17T17:20:23.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.723 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:23.723 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.507542+0000", 2024-08-17T17:20:23.723 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8029999999999997e-06 2024-08-17T17:20:23.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.723 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.723 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.481898+0000", 2024-08-17T17:20:23.723 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.974355949 2024-08-17T17:20:23.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.724 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:23.724 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.481906+0000", 2024-08-17T17:20:23.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.089e-06 2024-08-17T17:20:23.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.724 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.724 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.784148+0000", 2024-08-17T17:20:23.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3022419219999999 2024-08-17T17:20:23.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.724 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:23.724 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.784298+0000", 2024-08-17T17:20:23.725 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015066 2024-08-17T17:20:23.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.725 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:20:23.725 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.784462+0000", 2024-08-17T17:20:23.725 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000163676 2024-08-17T17:20:23.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.725 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:23.725 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.798884+0000", 2024-08-17T17:20:23.725 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.014422199 2024-08-17T17:20:23.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.726 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:23.726 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.798980+0000", 2024-08-17T17:20:23.726 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5964999999999993e-05 2024-08-17T17:20:23.726 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.726 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:23.726 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.799031+0000", 2024-08-17T17:20:23.726 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0421999999999999e-05 2024-08-17T17:20:23.726 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.726 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:23.726 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.799088+0000", 2024-08-17T17:20:23.726 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7260999999999999e-05 2024-08-17T17:20:23.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.727 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.727 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:23.727 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.802384+0000", 2024-08-17T17:20:23.727 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032958409999999999 2024-08-17T17:20:23.727 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.727 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:23.727 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.727 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.727 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:68144 5.a 5:5a80e405:::benchmark_data_smithi003_140793_object68143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-08-17T17:20:23.727 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:56.519883+0000", 2024-08-17T17:20:23.727 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.108527590000001, 2024-08-17T17:20:23.727 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3203096400000001, 2024-08-17T17:20:23.728 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:23.728 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:23.728 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:23.728 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:23.728 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:23.728 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:23.728 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68144 2024-08-17T17:20:23.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.728 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:23.728 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.728 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:23.728 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519883+0000", 2024-08-17T17:20:23.728 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:23.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519883+0000", 2024-08-17T17:20:23.729 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:23.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519886+0000", 2024-08-17T17:20:23.729 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4949999999999998e-06 2024-08-17T17:20:23.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:23.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519909+0000", 2024-08-17T17:20:23.729 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3366e-05 2024-08-17T17:20:23.730 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.730 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.730 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:23.730 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519926+0000", 2024-08-17T17:20:23.730 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6637999999999999e-05 2024-08-17T17:20:23.730 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.730 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.730 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:23.730 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519933+0000", 2024-08-17T17:20:23.730 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3710000000000002e-06 2024-08-17T17:20:23.730 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.730 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.730 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.731 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519988+0000", 2024-08-17T17:20:23.731 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4923999999999998e-05 2024-08-17T17:20:23.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.731 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.731 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:23.731 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519995+0000", 2024-08-17T17:20:23.731 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0369999999999997e-06 2024-08-17T17:20:23.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.731 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.731 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.731 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.482416+0000", 2024-08-17T17:20:23.731 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96242099800000003 2024-08-17T17:20:23.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.731 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.732 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:23.732 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.482427+0000", 2024-08-17T17:20:23.732 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1014e-05 2024-08-17T17:20:23.732 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.732 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.732 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.732 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.802434+0000", 2024-08-17T17:20:23.732 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.320006295 2024-08-17T17:20:23.732 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.732 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.732 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:23.732 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.802587+0000", 2024-08-17T17:20:23.732 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000153342 2024-08-17T17:20:23.733 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.733 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.733 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:20:23.733 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.802749+0000", 2024-08-17T17:20:23.733 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016180400000000001 2024-08-17T17:20:23.733 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.733 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.733 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:23.733 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.813063+0000", 2024-08-17T17:20:23.733 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010314591 2024-08-17T17:20:23.733 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.733 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.733 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:23.734 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.814540+0000", 2024-08-17T17:20:23.734 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014765500000000001 2024-08-17T17:20:23.734 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.734 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.734 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:23.734 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.836294+0000", 2024-08-17T17:20:23.734 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.021754141000000001 2024-08-17T17:20:23.734 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.734 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.734 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:23.734 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.836377+0000", 2024-08-17T17:20:23.734 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2698e-05 2024-08-17T17:20:23.734 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.734 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.735 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:23.735 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.840193+0000", 2024-08-17T17:20:23.735 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038163759999999998 2024-08-17T17:20:23.735 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.735 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:23.735 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.735 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.735 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:68113 5.4 5:2078877e:::benchmark_data_smithi003_140793_object68112:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e130)", 2024-08-17T17:20:23.735 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:57.488289+0000", 2024-08-17T17:20:23.735 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 26.140122011999999, 2024-08-17T17:20:23.735 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.249369143, 2024-08-17T17:20:23.735 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:23.736 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:23.736 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:20:23.736 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:23.736 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:23.736 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:23.736 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68113 2024-08-17T17:20:23.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.736 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:23.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.736 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:23.736 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488289+0000", 2024-08-17T17:20:23.736 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.737 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:23.737 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488289+0000", 2024-08-17T17:20:23.737 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.737 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.737 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.737 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:23.737 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488290+0000", 2024-08-17T17:20:23.737 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.463e-06 2024-08-17T17:20:23.737 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.737 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.737 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:23.737 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488320+0000", 2024-08-17T17:20:23.737 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9150999999999999e-05 2024-08-17T17:20:23.737 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.738 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.738 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:23.738 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488331+0000", 2024-08-17T17:20:23.738 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1396000000000001e-05 2024-08-17T17:20:23.738 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.738 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.738 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:23.738 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488341+0000", 2024-08-17T17:20:23.738 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0363999999999999e-05 2024-08-17T17:20:23.738 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.738 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.738 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.739 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.490414+0000", 2024-08-17T17:20:23.739 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002073068 2024-08-17T17:20:23.739 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.739 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.739 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:23.739 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.490424+0000", 2024-08-17T17:20:23.739 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4669999999999993e-06 2024-08-17T17:20:23.739 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.739 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.739 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.739 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.737532+0000", 2024-08-17T17:20:23.739 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2471080960000001 2024-08-17T17:20:23.739 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.740 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.740 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:23.740 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.737658+0000", 2024-08-17T17:20:23.740 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000126138 2024-08-17T17:20:23.740 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.740 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:23.740 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.740 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.740 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.740 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:68122 5.4 5:2519cdf9:::benchmark_data_smithi003_140793_object68121:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e130)", 2024-08-17T17:20:23.740 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:57.488356+0000", 2024-08-17T17:20:23.740 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 26.140054794000001, 2024-08-17T17:20:23.740 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.252854337, 2024-08-17T17:20:23.741 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:23.741 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:23.741 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:20:23.741 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:23.741 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:23.741 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:23.741 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68122 2024-08-17T17:20:23.741 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.741 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:23.741 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.741 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:23.741 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488356+0000", 2024-08-17T17:20:23.741 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.741 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.742 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.742 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:23.742 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488356+0000", 2024-08-17T17:20:23.742 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.742 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.742 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.742 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:23.742 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488357+0000", 2024-08-17T17:20:23.742 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.44e-07 2024-08-17T17:20:23.742 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.742 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.742 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:23.742 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488382+0000", 2024-08-17T17:20:23.742 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5321999999999999e-05 2024-08-17T17:20:23.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.743 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.743 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:23.743 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488391+0000", 2024-08-17T17:20:23.743 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8170000000000005e-06 2024-08-17T17:20:23.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.743 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.743 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:23.743 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488397+0000", 2024-08-17T17:20:23.743 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1580000000000002e-06 2024-08-17T17:20:23.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.743 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.743 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.744 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.490445+0000", 2024-08-17T17:20:23.744 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002048242 2024-08-17T17:20:23.744 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.744 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.744 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:23.744 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.490451+0000", 2024-08-17T17:20:23.744 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3460000000000001e-06 2024-08-17T17:20:23.744 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.744 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.744 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.744 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.741078+0000", 2024-08-17T17:20:23.744 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.250627181 2024-08-17T17:20:23.744 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.745 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:23.745 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.741211+0000", 2024-08-17T17:20:23.745 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013272699999999999 2024-08-17T17:20:23.745 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.745 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:23.745 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.745 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:68126 5.4 5:2abb837e:::benchmark_data_smithi003_140793_object68125:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e130)", 2024-08-17T17:20:23.745 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:57.488407+0000", 2024-08-17T17:20:23.745 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 26.140003827000001, 2024-08-17T17:20:23.745 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.287577712, 2024-08-17T17:20:23.746 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:23.746 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:23.746 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:20:23.746 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:23.746 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:23.746 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:23.746 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68126 2024-08-17T17:20:23.746 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.746 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:23.746 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.746 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:23.746 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488407+0000", 2024-08-17T17:20:23.746 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.747 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:23.747 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488407+0000", 2024-08-17T17:20:23.747 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.747 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:23.747 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488408+0000", 2024-08-17T17:20:23.747 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3600000000000004e-07 2024-08-17T17:20:23.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.747 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:23.747 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488432+0000", 2024-08-17T17:20:23.747 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3955999999999999e-05 2024-08-17T17:20:23.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.748 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:23.748 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488440+0000", 2024-08-17T17:20:23.748 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1550000000000007e-06 2024-08-17T17:20:23.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.748 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:23.748 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488446+0000", 2024-08-17T17:20:23.748 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2269999999999998e-06 2024-08-17T17:20:23.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.748 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.748 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.490496+0000", 2024-08-17T17:20:23.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020495449999999998 2024-08-17T17:20:23.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.749 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:23.749 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.490505+0000", 2024-08-17T17:20:23.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0450000000000006e-06 2024-08-17T17:20:23.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.749 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.749 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.741568+0000", 2024-08-17T17:20:23.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2510632020000001 2024-08-17T17:20:23.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.750 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.750 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for degraded object", 2024-08-17T17:20:23.750 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.742515+0000", 2024-08-17T17:20:23.750 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00094757200000000004 2024-08-17T17:20:23.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.750 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.750 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.750 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.775909+0000", 2024-08-17T17:20:23.750 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.033393513999999999 2024-08-17T17:20:23.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.750 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.750 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:23.750 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.775985+0000", 2024-08-17T17:20:23.750 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5959999999999998e-05 2024-08-17T17:20:23.751 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.751 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:23.751 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.751 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:68128 5.4 5:250eee02:::benchmark_data_smithi003_140793_object68127:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e130)", 2024-08-17T17:20:23.751 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:57.488455+0000", 2024-08-17T17:20:23.751 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 26.13995589, 2024-08-17T17:20:23.751 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2798230580000001, 2024-08-17T17:20:23.751 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:23.751 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:23.751 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:20:23.751 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:23.752 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:23.752 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:23.752 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68128 2024-08-17T17:20:23.752 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.752 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:23.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.752 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:23.753 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488455+0000", 2024-08-17T17:20:23.753 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.753 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.753 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.753 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:23.753 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488455+0000", 2024-08-17T17:20:23.753 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.753 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.753 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.754 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:23.754 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488456+0000", 2024-08-17T17:20:23.754 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2799999999999996e-07 2024-08-17T17:20:23.754 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.754 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.754 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:23.754 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488479+0000", 2024-08-17T17:20:23.754 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.368e-05 2024-08-17T17:20:23.754 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.754 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.754 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:23.754 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488488+0000", 2024-08-17T17:20:23.754 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2209999999999997e-06 2024-08-17T17:20:23.754 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.755 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.755 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:23.755 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488493+0000", 2024-08-17T17:20:23.755 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8080000000000001e-06 2024-08-17T17:20:23.755 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.755 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.755 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.755 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.490529+0000", 2024-08-17T17:20:23.755 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020353150000000002 2024-08-17T17:20:23.755 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.755 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.755 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:23.755 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.490535+0000", 2024-08-17T17:20:23.756 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2450000000000003e-06 2024-08-17T17:20:23.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.756 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.756 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.742562+0000", 2024-08-17T17:20:23.756 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.252026654 2024-08-17T17:20:23.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.756 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for degraded object", 2024-08-17T17:20:23.756 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.743351+0000", 2024-08-17T17:20:23.756 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00078913699999999998 2024-08-17T17:20:23.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.756 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.757 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.768179+0000", 2024-08-17T17:20:23.757 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.024828322999999999 2024-08-17T17:20:23.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.757 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:23.757 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.768278+0000", 2024-08-17T17:20:23.757 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9147000000000005e-05 2024-08-17T17:20:23.757 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.757 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:23.757 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.758 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:68137 5.4 5:243c5be5:::benchmark_data_smithi003_140793_object68136:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e130)", 2024-08-17T17:20:23.758 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:57.488502+0000", 2024-08-17T17:20:23.758 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 26.139908824999999, 2024-08-17T17:20:23.758 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.283335688, 2024-08-17T17:20:23.758 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:23.758 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:23.758 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:23.758 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:23.758 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:23.758 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:23.758 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68137 2024-08-17T17:20:23.758 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.758 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:23.759 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.759 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:23.759 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488502+0000", 2024-08-17T17:20:23.759 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.759 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.759 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.759 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:23.759 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488502+0000", 2024-08-17T17:20:23.759 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.759 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.759 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.759 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:23.759 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488503+0000", 2024-08-17T17:20:23.759 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2399999999999998e-07 2024-08-17T17:20:23.760 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.760 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.760 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:23.760 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488624+0000", 2024-08-17T17:20:23.760 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012089799999999999 2024-08-17T17:20:23.760 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.760 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.760 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:23.760 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488633+0000", 2024-08-17T17:20:23.760 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8660000000000001e-06 2024-08-17T17:20:23.760 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.760 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.760 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:23.761 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488640+0000", 2024-08-17T17:20:23.761 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8329999999999996e-06 2024-08-17T17:20:23.761 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.761 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.761 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.490556+0000", 2024-08-17T17:20:23.761 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0019160690000000001 2024-08-17T17:20:23.761 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.761 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:23.761 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.490562+0000", 2024-08-17T17:20:23.761 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.806e-06 2024-08-17T17:20:23.761 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.762 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.762 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.743428+0000", 2024-08-17T17:20:23.762 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.252865457 2024-08-17T17:20:23.762 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.762 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.762 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:23.762 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.743738+0000", 2024-08-17T17:20:23.762 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00031069099999999998 2024-08-17T17:20:23.762 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.762 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.762 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-08-17T17:20:23.762 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.743958+0000", 2024-08-17T17:20:23.762 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00021965299999999999 2024-08-17T17:20:23.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.763 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.763 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:23.763 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.761495+0000", 2024-08-17T17:20:23.763 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017537516 2024-08-17T17:20:23.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.763 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.763 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:23.763 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.761545+0000", 2024-08-17T17:20:23.763 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9064000000000003e-05 2024-08-17T17:20:23.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.763 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.764 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:23.764 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.768317+0000", 2024-08-17T17:20:23.764 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0067721500000000002 2024-08-17T17:20:23.764 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.764 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.764 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:23.764 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.768369+0000", 2024-08-17T17:20:23.764 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1873000000000003e-05 2024-08-17T17:20:23.764 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.764 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.764 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:23.764 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.771838+0000", 2024-08-17T17:20:23.764 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034692880000000001 2024-08-17T17:20:23.764 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.765 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:23.765 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.765 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.765 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:68153 5.a 5:5a530647:::benchmark_data_smithi003_140793_object68152:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-08-17T17:20:23.765 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:58.737452+0000", 2024-08-17T17:20:23.765 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 24.890959284000001, 2024-08-17T17:20:23.765 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.109671217, 2024-08-17T17:20:23.765 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:23.765 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:23.765 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:23.765 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:23.765 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:23.766 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:23.766 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68153 2024-08-17T17:20:23.766 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.766 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:23.766 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.766 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:23.766 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.737452+0000", 2024-08-17T17:20:23.766 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.766 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.766 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.766 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:23.766 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.737452+0000", 2024-08-17T17:20:23.766 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:23.767 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.767 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.767 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:23.767 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.737454+0000", 2024-08-17T17:20:23.767 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0030000000000001e-06 2024-08-17T17:20:23.767 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.767 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.767 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:23.767 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.737487+0000", 2024-08-17T17:20:23.767 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2826e-05 2024-08-17T17:20:23.767 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.767 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.767 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:23.767 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.737504+0000", 2024-08-17T17:20:23.768 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7893999999999998e-05 2024-08-17T17:20:23.768 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.768 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.768 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:23.768 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.737515+0000", 2024-08-17T17:20:23.768 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0338e-05 2024-08-17T17:20:23.768 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.768 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.768 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.768 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.737601+0000", 2024-08-17T17:20:23.768 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5929999999999999e-05 2024-08-17T17:20:23.768 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.768 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.768 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:23.768 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.737612+0000", 2024-08-17T17:20:23.769 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0998999999999999e-05 2024-08-17T17:20:23.769 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.769 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:23.769 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.804970+0000", 2024-08-17T17:20:23.769 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.067358630000000003 2024-08-17T17:20:23.769 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.769 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:23.769 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.805135+0000", 2024-08-17T17:20:23.769 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000164742 2024-08-17T17:20:23.769 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.769 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:20:23.770 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.805327+0000", 2024-08-17T17:20:23.770 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000192382 2024-08-17T17:20:23.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.770 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.770 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:23.770 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.813112+0000", 2024-08-17T17:20:23.770 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0077842440000000001 2024-08-17T17:20:23.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.770 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.770 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:23.770 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.814548+0000", 2024-08-17T17:20:23.770 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014368040000000001 2024-08-17T17:20:23.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.771 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:23.771 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.840255+0000", 2024-08-17T17:20:23.771 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.025706058 2024-08-17T17:20:23.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.771 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:23.771 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.840480+0000", 2024-08-17T17:20:23.771 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00022594 2024-08-17T17:20:23.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:23.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:23.771 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:23.771 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.847123+0000", 2024-08-17T17:20:23.771 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066424270000000002 2024-08-17T17:20:23.772 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.772 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:23.772 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.772 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:23.772 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:23.772 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:20:23.796 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-08-17T17:20:23.811 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:23.820 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 72 16 88074 88058 76.4023 45.0625 0.0265948 0.0130817 2024-08-17T17:20:23.841 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:23.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:23.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:23.943 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:23.977 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:23.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:23.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:24.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:20:24.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:23 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/4086431257' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:20:24.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:23 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:20:24.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:20:24.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:23 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/4086431257' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:20:24.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:23 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:20:24.024 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:20:24.024 INFO:teuthology.orchestra.run.smithi053.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-17T17:20:24.024 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:20:24.060 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:20:24.060 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:23 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/4086431257' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:20:24.060 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:23 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:20:24.080 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:24.133 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:24.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:24.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:24.235 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:24.276 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:24.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:24.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:24.378 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:24.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:24.417 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:24.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:24.518 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:24.559 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:24.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:24.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:24.661 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:24.700 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:24.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:24.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:24.802 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:24.820 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 73 16 88903 88887 76.0655 51.8125 0.0178223 0.0131381 2024-08-17T17:20:24.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:24.846 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:24.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:24.887 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:20:24.947 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:24.979 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:24.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:24.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:25.081 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:25.119 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:25.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:25.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:25.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:24 smithi053 ceph-mon[89963]: pgmap v283: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.3 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 47 MiB/s wr, 747 op/s; 29853/260499 objects degraded (11.460%); 6534/260499 objects misplaced (2.508%); 6.3 MiB/s, 100 objects/s recovering 2024-08-17T17:20:25.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:24 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 29853/260499 objects degraded (11.460%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:25.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:24 smithi053 ceph-mon[89963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:20:25.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:24 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:20:25.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:24 smithi053 ceph-mon[89963]: osdmap e138: 8 total, 8 up, 8 in 2024-08-17T17:20:25.222 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:25.254 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:25.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:25.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:25.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:24 smithi003 ceph-mon[103344]: pgmap v283: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.3 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 47 MiB/s wr, 747 op/s; 29853/260499 objects degraded (11.460%); 6534/260499 objects misplaced (2.508%); 6.3 MiB/s, 100 objects/s recovering 2024-08-17T17:20:25.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:24 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 29853/260499 objects degraded (11.460%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:25.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:24 smithi003 ceph-mon[103344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:20:25.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:24 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:20:25.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:24 smithi003 ceph-mon[103344]: osdmap e138: 8 total, 8 up, 8 in 2024-08-17T17:20:25.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:24 smithi003 ceph-mon[100615]: pgmap v283: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.3 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 47 MiB/s wr, 747 op/s; 29853/260499 objects degraded (11.460%); 6534/260499 objects misplaced (2.508%); 6.3 MiB/s, 100 objects/s recovering 2024-08-17T17:20:25.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:24 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 29853/260499 objects degraded (11.460%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:25.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:24 smithi003 ceph-mon[100615]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:20:25.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:24 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:20:25.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:24 smithi003 ceph-mon[100615]: osdmap e138: 8 total, 8 up, 8 in 2024-08-17T17:20:25.356 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:25.374 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-17T17:20:25.400 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:25.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:25.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:25.503 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:25.540 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:25.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:25.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:25.642 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:25.723 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:25.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:25.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:25.820 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 74 16 89674 89658 75.6888 48.1875 0.0130809 0.0132061 2024-08-17T17:20:25.827 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:25.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:25.867 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:25.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:25.968 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:26.002 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:26.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:26.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:26.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:20:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:20:25] "GET /metrics HTTP/1.1" 200 36540 "" "Prometheus/2.51.0" 2024-08-17T17:20:26.104 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:26.136 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:26.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:26.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:26.238 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:26.268 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:26.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:26.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:26.369 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:26.398 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:26.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:26.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:26.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:26.543 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:26.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:26.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:26.645 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:26.694 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:26.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:26.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:26.796 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:26.820 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 75 16 90518 90502 75.383 52.75 0.010596 0.0132593 2024-08-17T17:20:26.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:26.832 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:26.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:26.846 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:20:26.933 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:26.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:26.984 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:26.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:27.014 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:20:27.085 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:27.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:27.149 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:27.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:27.163 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:26 smithi003 ceph-mon[100615]: pgmap v285: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 45 MiB/s wr, 727 op/s; 29853/264936 objects degraded (11.268%); 6343/264936 objects misplaced (2.394%); 5.7 MiB/s, 91 objects/s recovering 2024-08-17T17:20:27.165 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:26 smithi003 ceph-mon[103344]: pgmap v285: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 45 MiB/s wr, 727 op/s; 29853/264936 objects degraded (11.268%); 6343/264936 objects misplaced (2.394%); 5.7 MiB/s, 91 objects/s recovering 2024-08-17T17:20:27.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:26 smithi053 ceph-mon[89963]: pgmap v285: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 45 MiB/s wr, 727 op/s; 29853/264936 objects degraded (11.268%); 6343/264936 objects misplaced (2.394%); 5.7 MiB/s, 91 objects/s recovering 2024-08-17T17:20:27.250 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:27.291 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:27.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:27.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:27.393 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:27.428 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:27.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:27.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:27.530 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:27.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:27.578 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:27.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:27.679 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:27.718 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:27.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:27.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:27.815 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:20:27.820 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:27.822 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 76 16 91270 91254 75.0086 47 0.0163539 0.0133258 2024-08-17T17:20:27.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:27.893 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:27.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:27.994 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:28.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:28.044 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:28.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:28.145 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:28.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:28.175 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:28.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:28.276 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:28.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:28.326 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:28.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:28.427 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:28.452 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:28.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:28.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:28.467 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:20:28.467 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:20:28.506 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:20:28.506 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:20:28.541 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:20:28.541 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:20:28.554 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:28.578 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:20:28.578 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:20:28.583 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:28.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:28.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:28.612 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:20:28.612 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:20:28.649 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:20:28.649 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:20:28.685 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:28.690 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:20:28.691 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:20:28.732 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:20:28.732 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:20:28.734 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:28.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:28.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:28.821 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 77 16 91975 91959 74.6068 44.0625 0.0284902 0.0133975 2024-08-17T17:20:28.836 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:28.877 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:28.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:28.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:28.979 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:29.022 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:28 smithi003 ceph-mon[100615]: pgmap v286: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 47 MiB/s wr, 747 op/s; 29853/266406 objects degraded (11.206%); 6343/266406 objects misplaced (2.381%); 5.7 MiB/s, 91 objects/s recovering 2024-08-17T17:20:29.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:28 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3103358179' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:20:29.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:28 smithi003 ceph-mon[103344]: pgmap v286: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 47 MiB/s wr, 747 op/s; 29853/266406 objects degraded (11.206%); 6343/266406 objects misplaced (2.381%); 5.7 MiB/s, 91 objects/s recovering 2024-08-17T17:20:29.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:28 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3103358179' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:20:29.026 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:29.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:29.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:29.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:29.165 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:29.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:29.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:29.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:28 smithi053 ceph-mon[89963]: pgmap v286: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 47 MiB/s wr, 747 op/s; 29853/266406 objects degraded (11.206%); 6343/266406 objects misplaced (2.381%); 5.7 MiB/s, 91 objects/s recovering 2024-08-17T17:20:29.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:28 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3103358179' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:20:29.267 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:29.299 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:29.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:29.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:29.401 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:29.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:29.445 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:29.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:29.547 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:29.576 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:29.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:29.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:29.678 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:29.719 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:29.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:29.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:29.820 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:29.821 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 78 16 92751 92735 74.2721 48.5 0.00884301 0.013457 2024-08-17T17:20:29.850 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-17T17:20:29.850 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:20:29.851 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-17T17:20:29.851 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd out 4 2024-08-17T17:20:29.861 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:29.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:29.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:29.962 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:30.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:30.017 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:30.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:30.118 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:30.158 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:30.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:30.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:30.215 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:29 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 29853/266406 objects degraded (11.206%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:30.260 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:30.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:29 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 29853/266406 objects degraded (11.206%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:30.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:29 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 29853/266406 objects degraded (11.206%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:30.289 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:30.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:30.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:30.391 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:30.419 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:30.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:30.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:30.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:20:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:20:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:20:30.522 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:30.569 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:30.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:30.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:30.671 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:30.702 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:30.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:30.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:30.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:30.823 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 79 16 93548 93532 73.9614 49.8125 0.00887333 0.0135137 2024-08-17T17:20:30.869 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:30.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:30.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:30.971 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:31.010 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:31.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:31.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:31.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:31.155 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:31.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:31.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:31.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:30 smithi053 ceph-mon[89963]: pgmap v287: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 50 MiB/s wr, 801 op/s; 29853/274554 objects degraded (10.873%); 5961/274554 objects misplaced (2.171%); 4.7 MiB/s, 75 objects/s recovering 2024-08-17T17:20:31.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:31.263 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:30 smithi003 ceph-mon[103344]: pgmap v287: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 50 MiB/s wr, 801 op/s; 29853/274554 objects degraded (10.873%); 5961/274554 objects misplaced (2.171%); 4.7 MiB/s, 75 objects/s recovering 2024-08-17T17:20:31.266 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:30 smithi003 ceph-mon[100615]: pgmap v287: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 50 MiB/s wr, 801 op/s; 29853/274554 objects degraded (10.873%); 5961/274554 objects misplaced (2.171%); 4.7 MiB/s, 75 objects/s recovering 2024-08-17T17:20:31.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:31.298 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:31.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:31.398 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:31.439 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:31.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:31.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:31.541 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:31.588 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:31.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:31.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:31.690 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:31.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:31.721 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:31.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:31.822 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:31.823 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-08-17T17:20:31.823227+0000 min lat: 0.0011526 max lat: 2.93219 avg lat: 0.0135634 2024-08-17T17:20:31.823 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:20:31.823 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 80 16 94391 94375 73.6955 52.6875 0.0248694 0.0135634 2024-08-17T17:20:31.860 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:31.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:31.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:31.962 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:31.992 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:31.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:31.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:32.094 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:32.116 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:32.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:32.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:32.218 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:32.262 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:32.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:32.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:32.364 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:32.413 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:32.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:32.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:32.515 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:32.548 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:32.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:32.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:32.650 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:32.680 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:32.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:32.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:32.782 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:32.823 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 81 16 95111 95095 73.3413 45 0.0207703 0.0136289 2024-08-17T17:20:32.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:32.827 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:32.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:32.928 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:32.968 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:32.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:32.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:33.070 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:33.084 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:32 smithi053 ceph-mon[89963]: pgmap v288: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 50 MiB/s wr, 795 op/s; 29853/276633 objects degraded (10.792%); 5961/276633 objects misplaced (2.155%); 4.7 MiB/s, 75 objects/s recovering 2024-08-17T17:20:33.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:33.106 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:33.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:33.207 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:33.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:33.248 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:33.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:33.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:32 smithi003 ceph-mon[103344]: pgmap v288: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 50 MiB/s wr, 795 op/s; 29853/276633 objects degraded (10.792%); 5961/276633 objects misplaced (2.155%); 4.7 MiB/s, 75 objects/s recovering 2024-08-17T17:20:33.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:32 smithi003 ceph-mon[100615]: pgmap v288: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 50 MiB/s wr, 795 op/s; 29853/276633 objects degraded (10.792%); 5961/276633 objects misplaced (2.155%); 4.7 MiB/s, 75 objects/s recovering 2024-08-17T17:20:33.349 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:33.393 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:33.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:33.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:33.495 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:33.522 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:33.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:33.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:33.624 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:33.659 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:33.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:33.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:33.761 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:33.797 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:33.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:33.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:33.823 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 82 16 95922 95906 73.0651 50.6875 0.00467574 0.0136769 2024-08-17T17:20:33.898 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:33.930 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:33.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:33.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:34.032 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:34.072 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:34.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:34.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:34.174 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:34.204 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:34.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:34.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:34.306 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:34.336 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:34.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:34.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:34.438 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:34.474 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:34.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:34.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:34.576 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:34.611 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:34.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:34.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:34.713 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:34.748 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:34.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:34.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:34.823 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 83 16 96691 96675 72.7639 48.0625 0.0230072 0.0137367 2024-08-17T17:20:34.849 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:34.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:34.898 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:34.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:34.999 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:35.026 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:35.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:35.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:35.128 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:35.163 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:35.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:35.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:35.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:34 smithi053 ceph-mon[89963]: pgmap v289: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 49 MiB/s wr, 779 op/s; 29853/283890 objects degraded (10.516%); 5760/283890 objects misplaced (2.029%); 4.8 MiB/s, 77 objects/s recovering 2024-08-17T17:20:35.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:34 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 29853/283890 objects degraded (10.516%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:35.264 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:35.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:34 smithi003 ceph-mon[103344]: pgmap v289: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 49 MiB/s wr, 779 op/s; 29853/283890 objects degraded (10.516%); 5760/283890 objects misplaced (2.029%); 4.8 MiB/s, 77 objects/s recovering 2024-08-17T17:20:35.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:34 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 29853/283890 objects degraded (10.516%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:35.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:34 smithi003 ceph-mon[100615]: pgmap v289: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 49 MiB/s wr, 779 op/s; 29853/283890 objects degraded (10.516%); 5760/283890 objects misplaced (2.029%); 4.8 MiB/s, 77 objects/s recovering 2024-08-17T17:20:35.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:34 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 29853/283890 objects degraded (10.516%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:35.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:35.303 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:35.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:35.404 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:35.440 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:35.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:35.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:35.541 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:35.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:35.581 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:35.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:35.682 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:35.728 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:35.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:35.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:35.823 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 84 16 97500 97484 72.4997 50.5625 0.00992952 0.0137865 2024-08-17T17:20:35.830 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:35.857 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:35.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:35.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:35.959 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:35.992 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:35.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:35.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:36.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:20:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:20:35] "GET /metrics HTTP/1.1" 200 36540 "" "Prometheus/2.51.0" 2024-08-17T17:20:36.094 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:36.117 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:36.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:36.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:36.219 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:36.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:36.254 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:36.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:36.355 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:36.385 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:36.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:36.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:36.487 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:36.529 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:36.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:36.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:36.631 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:36.679 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:36.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:36.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:36.781 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:36.810 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:36.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:36.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:36.823 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 85 16 98313 98297 72.2445 50.8125 0.0160049 0.0138345 2024-08-17T17:20:36.908 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:36 smithi003 ceph-mon[100615]: pgmap v290: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 49 MiB/s wr, 777 op/s; 29853/288345 objects degraded (10.353%); 5567/288345 objects misplaced (1.931%); 5.1 MiB/s, 81 objects/s recovering 2024-08-17T17:20:36.909 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:36 smithi003 ceph-mon[103344]: pgmap v290: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 49 MiB/s wr, 777 op/s; 29853/288345 objects degraded (10.353%); 5567/288345 objects misplaced (1.931%); 5.1 MiB/s, 81 objects/s recovering 2024-08-17T17:20:36.912 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:36.943 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:36.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:36.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:37.045 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:37.081 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:37.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:37.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:37.183 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:37.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:36 smithi053 ceph-mon[89963]: pgmap v290: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 49 MiB/s wr, 777 op/s; 29853/288345 objects degraded (10.353%); 5567/288345 objects misplaced (1.931%); 5.1 MiB/s, 81 objects/s recovering 2024-08-17T17:20:37.222 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:37.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:37.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:37.324 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:37.359 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:37.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:37.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:37.461 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:37.490 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:37.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:37.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:37.591 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:37.622 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:37.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:37.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:37.724 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:37.772 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:37.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:37.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:37.823 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 86 16 99093 99077 71.9714 48.75 0.0143126 0.0138884 2024-08-17T17:20:37.874 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:37.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:37.910 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:37.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:38.011 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:38.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:38.064 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:38.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:38.088 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:20:38.137 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:20:38.137 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:20:38.165 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:38.223 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:38.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:38.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:38.325 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:38.360 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:38.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:38.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:38.461 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:38.498 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:38.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:38.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:38.600 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:38.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:38.647 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:38.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:38.748 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:38.779 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:38.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:38.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:38.823 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 87 16 99861 99845 71.6958 48 0.00869922 0.0139412 2024-08-17T17:20:38.857 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:20:38.860 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-08-17T17:20:38.860 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:38 smithi003 ceph-mon[103344]: pgmap v291: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 43 MiB/s wr, 692 op/s; 29853/289881 objects degraded (10.298%); 5567/289881 objects misplaced (1.920%); 4.0 MiB/s, 64 objects/s recovering 2024-08-17T17:20:38.861 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:38 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1172329909' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:20:38.861 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:20:38.862 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:38 smithi003 ceph-mon[100615]: pgmap v291: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 43 MiB/s wr, 692 op/s; 29853/289881 objects degraded (10.298%); 5567/289881 objects misplaced (1.920%); 4.0 MiB/s, 64 objects/s recovering 2024-08-17T17:20:38.862 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:38 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1172329909' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:20:38.862 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:20:38.880 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:38.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:38.936 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:38.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:39.037 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:39.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:39.080 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:39.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:39.181 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:39.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:38 smithi053 ceph-mon[89963]: pgmap v291: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 43 MiB/s wr, 692 op/s; 29853/289881 objects degraded (10.298%); 5567/289881 objects misplaced (1.920%); 4.0 MiB/s, 64 objects/s recovering 2024-08-17T17:20:39.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:38 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1172329909' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:20:39.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:20:39.240 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:39.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:39.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:39.342 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:39.376 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:39.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:39.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:39.478 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:39.512 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:39.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:39.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:39.614 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:39.639 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:39.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:39.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:39.741 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:39.773 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:39.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:39.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:39.824 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 88 16 100534 100518 71.359 42.0625 0.0194154 0.0140081 2024-08-17T17:20:39.874 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:39.910 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:39.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:39.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:40.013 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:40.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:39 smithi003 ceph-mon[103344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:20:40.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:39 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1172329909' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:20:40.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:39 smithi003 ceph-mon[103344]: osdmap e139: 8 total, 8 up, 8 in 2024-08-17T17:20:40.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:20:40.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:20:40.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:39 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 29853/289881 objects degraded (10.298%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:40.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:39 smithi003 ceph-mon[100615]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:20:40.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:39 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1172329909' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:20:40.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:39 smithi003 ceph-mon[100615]: osdmap e139: 8 total, 8 up, 8 in 2024-08-17T17:20:40.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:20:40.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:20:40.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:39 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 29853/289881 objects degraded (10.298%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:40.048 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:40.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:40.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:40.150 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:40.191 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:40.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:40.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:40.213 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:39 smithi053 ceph-mon[89963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:20:40.213 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:39 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1172329909' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:20:40.213 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:39 smithi053 ceph-mon[89963]: osdmap e139: 8 total, 8 up, 8 in 2024-08-17T17:20:40.213 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:20:40.213 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:20:40.213 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:39 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 29853/289881 objects degraded (10.298%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:40.293 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:40.321 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:40.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:40.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:40.423 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:40.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:40.459 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:40.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:40.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:20:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:20:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:20:40.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:40.601 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:40.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:40.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:40.703 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:40.736 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:40.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:40.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:40.824 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 89 16 101319 101303 71.1086 49.0625 0.0244541 0.0140569 2024-08-17T17:20:40.838 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:40.877 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:40.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:40.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:40.978 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:41.011 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:41.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:41.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:41.102 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-17T17:20:41.113 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:41.204 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:41.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:41.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:41.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:41 smithi003 ceph-mon[100615]: pgmap v293: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 49 MiB/s wr, 780 op/s; 29853/297987 objects degraded (10.018%); 5150/297987 objects misplaced (1.728%); 5.1 MiB/s, 81 objects/s recovering 2024-08-17T17:20:41.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:41 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:20:41.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:41 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:20:41.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:41 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:20:41.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:41 smithi003 ceph-mon[103344]: pgmap v293: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 49 MiB/s wr, 780 op/s; 29853/297987 objects degraded (10.018%); 5150/297987 objects misplaced (1.728%); 5.1 MiB/s, 81 objects/s recovering 2024-08-17T17:20:41.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:41 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:20:41.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:41 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:20:41.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:41 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:20:41.306 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:41.353 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:41.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:41.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:41.455 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:41.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:41 smithi053 ceph-mon[89963]: pgmap v293: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 49 MiB/s wr, 780 op/s; 29853/297987 objects degraded (10.018%); 5150/297987 objects misplaced (1.728%); 5.1 MiB/s, 81 objects/s recovering 2024-08-17T17:20:41.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:41 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:20:41.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:41 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:20:41.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:41 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:20:41.506 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:41.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:41.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:41.608 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:41.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:41.654 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:41.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:41.755 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:41.799 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:41.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:41.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:41.824 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total time run: 90.0127 2024-08-17T17:20:41.824 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total writes made: 102054 2024-08-17T17:20:41.824 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Write size: 65536 2024-08-17T17:20:41.824 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object size: 65536 2024-08-17T17:20:41.824 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Bandwidth (MB/sec): 70.8608 2024-08-17T17:20:41.824 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Bandwidth: 39.8451 2024-08-17T17:20:41.825 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max bandwidth (MB/sec): 199.625 2024-08-17T17:20:41.825 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min bandwidth (MB/sec): 0 2024-08-17T17:20:41.825 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average IOPS: 1133 2024-08-17T17:20:41.825 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev IOPS: 637.537 2024-08-17T17:20:41.825 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max IOPS: 3194 2024-08-17T17:20:41.825 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min IOPS: 0 2024-08-17T17:20:41.825 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average Latency(s): 0.0141067 2024-08-17T17:20:41.825 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Latency(s): 0.0503758 2024-08-17T17:20:41.825 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max latency(s): 2.93219 2024-08-17T17:20:41.825 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min latency(s): 0.0011526 2024-08-17T17:20:41.828 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Cleaning up (deleting benchmark objects) 2024-08-17T17:20:41.900 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:41.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:41.926 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:41.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:42.027 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:42.057 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:42.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:42.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:42.160 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:42.194 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:42.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:42.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:42.296 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:42.339 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:42.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:42.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:42.441 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:42.476 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:42.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:42.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:42.578 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:42.622 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:42.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:42.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:42.724 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:42.755 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:42.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:42.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:42.857 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:42.897 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:42.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:42.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:42.999 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:43.030 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:43.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:43.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:43.132 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:43.179 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:43.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:43.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:43.281 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:43.318 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:43.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:43.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:43.421 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:43.448 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:43.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:43.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:43.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:43 smithi053 ceph-mon[89963]: pgmap v294: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 49 MiB/s wr, 778 op/s; 29853/300009 objects degraded (9.951%); 5150/300009 objects misplaced (1.717%); 5.1 MiB/s, 81 objects/s recovering 2024-08-17T17:20:43.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:43 smithi003 ceph-mon[100615]: pgmap v294: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 49 MiB/s wr, 778 op/s; 29853/300009 objects degraded (9.951%); 5150/300009 objects misplaced (1.717%); 5.1 MiB/s, 81 objects/s recovering 2024-08-17T17:20:43.543 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:43 smithi003 ceph-mon[103344]: pgmap v294: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 49 MiB/s wr, 778 op/s; 29853/300009 objects degraded (9.951%); 5150/300009 objects misplaced (1.717%); 5.1 MiB/s, 81 objects/s recovering 2024-08-17T17:20:43.550 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:43.579 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:43.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:43.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:43.615 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:20:43.681 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:43.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:43.707 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:43.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:43.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:43.856 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:43.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:43.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:43.958 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:43.990 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:43.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:43.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:44.092 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:44.130 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:44.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:44.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:44.232 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:44.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:44.263 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:44.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:44.364 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:44.402 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:44.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:44.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:44.504 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:44.534 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:44.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:44.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:44.629 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:20:44.635 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:44.663 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:44.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:44.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:44.680 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:20:44.682 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:20:44.765 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:44.822 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:44.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:44.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:44.924 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:44.953 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:44.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:44.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:45.056 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:45.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:45.094 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:45.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:45.195 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:45.222 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:45.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:45.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:45.324 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:45.351 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:45.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:45.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:45.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:45 smithi003 ceph-mon[100615]: pgmap v295: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 102 B/s rd, 43 MiB/s wr, 765 op/s; 29853/302511 objects degraded (9.868%); 4920/302511 objects misplaced (1.626%); 5.2 MiB/s, 83 objects/s recovering 2024-08-17T17:20:45.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:45 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 29853/302511 objects degraded (9.868%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:45.384 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:45 smithi003 ceph-mon[103344]: pgmap v295: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 102 B/s rd, 43 MiB/s wr, 765 op/s; 29853/302511 objects degraded (9.868%); 4920/302511 objects misplaced (1.626%); 5.2 MiB/s, 83 objects/s recovering 2024-08-17T17:20:45.384 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:45 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 29853/302511 objects degraded (9.868%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:45.453 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:45.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:45 smithi053 ceph-mon[89963]: pgmap v295: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 102 B/s rd, 43 MiB/s wr, 765 op/s; 29853/302511 objects degraded (9.868%); 4920/302511 objects misplaced (1.626%); 5.2 MiB/s, 83 objects/s recovering 2024-08-17T17:20:45.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:45 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 29853/302511 objects degraded (9.868%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:45.482 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:45.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:45.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:45.584 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:45.614 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:45.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:45.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:45.716 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:45.745 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:45.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:45.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:45.847 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:45.875 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:45.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:45.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:45.977 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:46.006 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:46.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:46.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:46.019 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:20:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:20:45] "GET /metrics HTTP/1.1" 200 36539 "" "Prometheus/2.51.0" 2024-08-17T17:20:46.107 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:46.153 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:46.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:46.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:46.255 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:46.304 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:46.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:46.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:46.407 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:46.435 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:46.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:46.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:46.537 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:46.564 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:46.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:46.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:46.666 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:46.711 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:46.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:46.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:46.813 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:46.862 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:46.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:46.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:46.964 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:46.995 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:46.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:46.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:47.096 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:47.124 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:47.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:47.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:47.225 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:47.251 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:47.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:47.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:47.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:47 smithi003 ceph-mon[103344]: pgmap v296: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 102 B/s rd, 37 MiB/s wr, 759 op/s; 29684/301341 objects degraded (9.851%); 4701/301341 objects misplaced (1.560%); 6.5 MiB/s, 103 objects/s recovering 2024-08-17T17:20:47.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:47 smithi003 ceph-mon[100615]: pgmap v296: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 102 B/s rd, 37 MiB/s wr, 759 op/s; 29684/301341 objects degraded (9.851%); 4701/301341 objects misplaced (1.560%); 6.5 MiB/s, 103 objects/s recovering 2024-08-17T17:20:47.353 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:47.401 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:47.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:47.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:47.503 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:47.545 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:47.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:47.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:47.647 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:47.677 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:47.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:47.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:47.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:47 smithi053 ceph-mon[89963]: pgmap v296: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 102 B/s rd, 37 MiB/s wr, 759 op/s; 29684/301341 objects degraded (9.851%); 4701/301341 objects misplaced (1.560%); 6.5 MiB/s, 103 objects/s recovering 2024-08-17T17:20:47.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:47 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-a[100611]: 2024-08-17T17:20:47.572+0000 7fc4d42c5640 -1 mon.a@0(leader).osd e139 definitely_dead 0 2024-08-17T17:20:47.778 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:47.822 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:47.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:47.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:47.924 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:47.965 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:47.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:47.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:47.970 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:20:47.970 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:20:48.067 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:48.100 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:48.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:48.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:48.202 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:48.242 INFO:teuthology.orchestra.run.smithi003.stderr:marked out osd.4. 2024-08-17T17:20:48.251 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:48.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:48.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:48.353 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:48.375 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:48.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:48.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:48.477 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:48.500 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:48.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:48.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:48.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:48 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1972999761' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-08-17T17:20:48.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:48 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-08-17T17:20:48.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:48 smithi003 ceph-mon[103344]: Client client.admin marked osd.4 out, while it was still marked up 2024-08-17T17:20:48.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:48 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1972999761' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-08-17T17:20:48.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:48 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-08-17T17:20:48.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:48 smithi003 ceph-mon[100615]: Client client.admin marked osd.4 out, while it was still marked up 2024-08-17T17:20:48.602 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:48.624 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:48.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:48.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:48.660 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-17T17:20:48.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:48 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1972999761' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-08-17T17:20:48.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:48 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-08-17T17:20:48.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:48 smithi053 ceph-mon[89963]: Client client.admin marked osd.4 out, while it was still marked up 2024-08-17T17:20:48.727 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:48.750 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:48.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:48.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:48.852 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:48.875 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:48.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:48.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:48.977 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:49.001 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:49.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:49.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:49.102 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:49.127 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:49.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:49.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:49.228 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:49.255 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:49.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:49.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:49.357 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:49.380 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:49.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:49.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:49.482 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:49.504 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:49.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:49.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:49.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:49 smithi003 ceph-mon[103344]: pgmap v297: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 102 B/s rd, 34 MiB/s wr, 755 op/s; 29684/299946 objects degraded (9.896%); 4701/299946 objects misplaced (1.567%); 6.5 MiB/s, 103 objects/s recovering 2024-08-17T17:20:49.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:49 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-08-17T17:20:49.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:49 smithi003 ceph-mon[103344]: osdmap e140: 8 total, 8 up, 7 in 2024-08-17T17:20:49.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:49 smithi003 ceph-mon[100615]: pgmap v297: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 102 B/s rd, 34 MiB/s wr, 755 op/s; 29684/299946 objects degraded (9.896%); 4701/299946 objects misplaced (1.567%); 6.5 MiB/s, 103 objects/s recovering 2024-08-17T17:20:49.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:49 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-08-17T17:20:49.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:49 smithi003 ceph-mon[100615]: osdmap e140: 8 total, 8 up, 7 in 2024-08-17T17:20:49.606 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:49.630 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:49.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:49.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:49.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:49 smithi053 ceph-mon[89963]: pgmap v297: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 102 B/s rd, 34 MiB/s wr, 755 op/s; 29684/299946 objects degraded (9.896%); 4701/299946 objects misplaced (1.567%); 6.5 MiB/s, 103 objects/s recovering 2024-08-17T17:20:49.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:49 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-08-17T17:20:49.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:49 smithi053 ceph-mon[89963]: osdmap e140: 8 total, 8 up, 7 in 2024-08-17T17:20:49.731 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:49.754 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:49.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:49.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:49.856 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:49.878 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:49.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:49.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:49.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:50.004 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:50.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:50.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:50.105 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:50.127 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:50.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:50.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:50.229 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:50.255 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:50.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:50.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:50.358 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:50.386 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:50.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:50.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:50.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:50 smithi053 ceph-mon[89963]: osdmap e141: 8 total, 8 up, 7 in 2024-08-17T17:20:50.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:50 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 29684/299946 objects degraded (9.896%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:50.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:20:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:20:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:20:50.488 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:50.514 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:50.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:50.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:50.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:50 smithi003 ceph-mon[103344]: osdmap e141: 8 total, 8 up, 7 in 2024-08-17T17:20:50.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:50 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 29684/299946 objects degraded (9.896%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:50.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:50 smithi003 ceph-mon[100615]: osdmap e141: 8 total, 8 up, 7 in 2024-08-17T17:20:50.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:50 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 29684/299946 objects degraded (9.896%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:50.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:50.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:50.653 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:50.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:50.754 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:50.783 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:50.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:50.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:50.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:50.921 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:50.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:50.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:51.023 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:51.066 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:51.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:51.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:51.168 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:51.199 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:51.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:51.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:51.301 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:51.331 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:51.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:51.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:51.433 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:51.473 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:51.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:51.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:51.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:51 smithi003 ceph-mon[103344]: pgmap v300: 89 pgs: 1 remapped+peering, 9 peering, 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 71 active+clean; 6.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 127 B/s rd, 16 MiB/s wr, 720 op/s; 25434/295194 objects degraded (8.616%); 4424/295194 objects misplaced (1.499%); 7.0 MiB/s, 111 objects/s recovering 2024-08-17T17:20:51.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:51 smithi003 ceph-mon[103344]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-08-17T17:20:51.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:51 smithi003 ceph-mon[103344]: osdmap e142: 8 total, 8 up, 7 in 2024-08-17T17:20:51.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:51 smithi003 ceph-mon[103344]: 5.c starting backfill to osd.6 from (0'0,0'0] MAX to 139'6629 2024-08-17T17:20:51.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:51 smithi003 ceph-mon[103344]: 5.7 starting backfill to osd.7 from (0'0,0'0] MAX to 139'6695 2024-08-17T17:20:51.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:51 smithi003 ceph-mon[103344]: 5.0 starting backfill to osd.6 from (0'0,0'0] MAX to 139'6643 2024-08-17T17:20:51.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:51 smithi003 ceph-mon[100615]: pgmap v300: 89 pgs: 1 remapped+peering, 9 peering, 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 71 active+clean; 6.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 127 B/s rd, 16 MiB/s wr, 720 op/s; 25434/295194 objects degraded (8.616%); 4424/295194 objects misplaced (1.499%); 7.0 MiB/s, 111 objects/s recovering 2024-08-17T17:20:51.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:51 smithi003 ceph-mon[100615]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-08-17T17:20:51.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:51 smithi003 ceph-mon[100615]: osdmap e142: 8 total, 8 up, 7 in 2024-08-17T17:20:51.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:51 smithi003 ceph-mon[100615]: 5.c starting backfill to osd.6 from (0'0,0'0] MAX to 139'6629 2024-08-17T17:20:51.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:51 smithi003 ceph-mon[100615]: 5.7 starting backfill to osd.7 from (0'0,0'0] MAX to 139'6695 2024-08-17T17:20:51.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:51 smithi003 ceph-mon[100615]: 5.0 starting backfill to osd.6 from (0'0,0'0] MAX to 139'6643 2024-08-17T17:20:51.574 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:51.607 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:51.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:51.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:51.708 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:51.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:51 smithi053 ceph-mon[89963]: pgmap v300: 89 pgs: 1 remapped+peering, 9 peering, 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 71 active+clean; 6.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 127 B/s rd, 16 MiB/s wr, 720 op/s; 25434/295194 objects degraded (8.616%); 4424/295194 objects misplaced (1.499%); 7.0 MiB/s, 111 objects/s recovering 2024-08-17T17:20:51.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:51 smithi053 ceph-mon[89963]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-08-17T17:20:51.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:51 smithi053 ceph-mon[89963]: osdmap e142: 8 total, 8 up, 7 in 2024-08-17T17:20:51.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:51 smithi053 ceph-mon[89963]: 5.c starting backfill to osd.6 from (0'0,0'0] MAX to 139'6629 2024-08-17T17:20:51.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:51 smithi053 ceph-mon[89963]: 5.7 starting backfill to osd.7 from (0'0,0'0] MAX to 139'6695 2024-08-17T17:20:51.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:51 smithi053 ceph-mon[89963]: 5.0 starting backfill to osd.6 from (0'0,0'0] MAX to 139'6643 2024-08-17T17:20:51.741 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:51.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:51.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:51.843 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:51.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:51.884 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:51.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:51.985 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:52.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:52.020 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:52.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:52.121 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:52.147 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:52.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:52.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:52.249 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:52.277 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:52.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:52.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:52.379 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:52.408 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:52.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:52.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:52.442 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:20:52.510 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:52.538 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:52.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:52.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:52.640 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:52.675 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:52.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:52.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:52.776 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:52.802 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:52.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:52.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:52.904 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:52.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:52.939 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:52.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:53.040 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:53.072 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:53.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:53.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:53.174 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:53.203 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:53.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:53.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:53.267 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-08-17T17:20:53.304 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:53.323 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:20:53.324 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:20:53.332 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:53.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:53.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:53.362 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:20:53.362 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:20:53.406 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:20:53.406 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:20:53.435 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:53.464 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:20:53.464 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:20:53.483 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:53.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:53.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:53.499 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:20:53.499 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:20:53.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:53 smithi003 ceph-mon[103344]: pgmap v302: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 4 remapped+peering, 9 peering, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 67 active+clean; 6.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 479 op/s; 27325/292716 objects degraded (9.335%); 8206/292716 objects misplaced (2.803%); 7.3 MiB/s, 116 objects/s recovering 2024-08-17T17:20:53.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:53 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2927599406' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:20:53.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:53 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:20:53.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:20:53.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:20:53.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:53 smithi003 ceph-mon[100615]: pgmap v302: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 4 remapped+peering, 9 peering, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 67 active+clean; 6.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 479 op/s; 27325/292716 objects degraded (9.335%); 8206/292716 objects misplaced (2.803%); 7.3 MiB/s, 116 objects/s recovering 2024-08-17T17:20:53.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:53 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2927599406' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:20:53.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:53 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:20:53.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:20:53.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:20:53.534 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:20:53.534 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:20:53.565 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:20:53.566 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:20:53.584 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:53.585 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:20:53.586 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:20:53.613 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:53.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:53.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:53.715 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:53.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:53 smithi053 ceph-mon[89963]: pgmap v302: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 4 remapped+peering, 9 peering, 1 active+recovering+undersized+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 67 active+clean; 6.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 479 op/s; 27325/292716 objects degraded (9.335%); 8206/292716 objects misplaced (2.803%); 7.3 MiB/s, 116 objects/s recovering 2024-08-17T17:20:53.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:53 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2927599406' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:20:53.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:53 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:20:53.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:20:53.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:20:53.753 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:53.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:53.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:53.855 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:53.886 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:53.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:53.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:53.988 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:54.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:54.026 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:54.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:54.127 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:54.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:54.162 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:54.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:54.263 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:54.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:54.301 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:54.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:54.402 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:54.431 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:54 smithi003 ceph-mon[100615]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:20:54.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:54 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:20:54.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:54 smithi003 ceph-mon[100615]: osdmap e143: 8 total, 8 up, 7 in 2024-08-17T17:20:54.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:54 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/26219515' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:20:54.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:54 smithi003 ceph-mon[103344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:20:54.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:54 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:20:54.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:54 smithi003 ceph-mon[103344]: osdmap e143: 8 total, 8 up, 7 in 2024-08-17T17:20:54.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:54 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/26219515' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:20:54.440 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:54.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:54.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:54.542 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:54.587 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:54.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:54.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:54.688 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:54.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:54 smithi053 ceph-mon[89963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:20:54.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:54 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:20:54.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:54 smithi053 ceph-mon[89963]: osdmap e143: 8 total, 8 up, 7 in 2024-08-17T17:20:54.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:54 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/26219515' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:20:54.721 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:54.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:54.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:54.823 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:54.867 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:54.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:54.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:54.969 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:55.009 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:55.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:55.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:55.111 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:55.146 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:55.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:55.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:55.248 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:55.281 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:55.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:55.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:55.383 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:55.418 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:55.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:55.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:55.519 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:55.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:55 smithi003 ceph-mon[103344]: pgmap v304: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 4 remapped+peering, 6 peering, 5 active+recovery_wait+undersized+degraded+remapped, 70 active+clean; 5.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 645 op/s; 21425/289305 objects degraded (7.406%); 13578/289305 objects misplaced (4.693%); 14 MiB/s, 223 objects/s recovering 2024-08-17T17:20:55.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:55 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 21425/289305 objects degraded (7.406%), 8 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:55.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:55 smithi003 ceph-mon[100615]: pgmap v304: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 4 remapped+peering, 6 peering, 5 active+recovery_wait+undersized+degraded+remapped, 70 active+clean; 5.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 645 op/s; 21425/289305 objects degraded (7.406%); 13578/289305 objects misplaced (4.693%); 14 MiB/s, 223 objects/s recovering 2024-08-17T17:20:55.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:55 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 21425/289305 objects degraded (7.406%), 8 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:55.556 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:55.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:55.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:55.657 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:55.687 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:55.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:55.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:55.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:55 smithi053 ceph-mon[89963]: pgmap v304: 89 pgs: 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovery_wait+remapped, 4 remapped+peering, 6 peering, 5 active+recovery_wait+undersized+degraded+remapped, 70 active+clean; 5.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 645 op/s; 21425/289305 objects degraded (7.406%); 13578/289305 objects misplaced (4.693%); 14 MiB/s, 223 objects/s recovering 2024-08-17T17:20:55.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:55 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 21425/289305 objects degraded (7.406%), 8 pgs degraded (PG_DEGRADED) 2024-08-17T17:20:55.789 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:55.833 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:55.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:55.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:55.842 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:20:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:20:55] "GET /metrics HTTP/1.1" 200 36604 "" "Prometheus/2.51.0" 2024-08-17T17:20:55.935 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:55.979 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:55.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:55.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:56.081 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:56.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:56.119 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:56.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:56.220 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:56.257 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:56.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:56.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:56.359 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:56.388 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:56.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:56.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:56.394 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-17T17:20:56.395 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:20:56.395 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-08-17T17:20:56.395 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-17T17:20:56.395 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-17T17:20:56.395 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-08-17T17:20:56.490 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:56.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:56.526 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:56.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:56.618 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:56 smithi003 ceph-mon[103344]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-17T17:20:56.619 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:56 smithi003 ceph-mon[100615]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-17T17:20:56.627 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:56.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:56.659 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:56.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:56.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:56 smithi053 ceph-mon[89963]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-08-17T17:20:56.760 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:20:56.800 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:56.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:56.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:20:56.837 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:20:56.902 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:56.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:56.938 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:56.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:57.039 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:57.089 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:57.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:57.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:57.190 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:57.230 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:57.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:57.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:57.332 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:57.346 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:20:57.346 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-08-17T17:20:57.346 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-08-17T17:20:57.346 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:20:57.346 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.346 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26082 5.1 5:8a74401c:::benchmark_data_smithi003_140793_object26081:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:20:57.346 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.358106+0000", 2024-08-17T17:20:57.346 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 94.982528404999996, 2024-08-17T17:20:57.346 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9241983769999997, 2024-08-17T17:20:57.346 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:57.346 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:57.346 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:57.346 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:57.347 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:57.347 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:57.347 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26082 2024-08-17T17:20:57.347 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.347 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:57.347 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.347 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:57.347 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.358106+0000", 2024-08-17T17:20:57.347 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.347 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.347 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.347 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:57.347 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.358106+0000", 2024-08-17T17:20:57.347 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.348 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.348 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.348 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:57.348 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.358107+0000", 2024-08-17T17:20:57.348 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4789999999999999e-06 2024-08-17T17:20:57.348 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.348 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.348 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:57.348 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.358131+0000", 2024-08-17T17:20:57.348 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4015e-05 2024-08-17T17:20:57.348 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.348 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.348 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:57.349 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.358143+0000", 2024-08-17T17:20:57.349 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.219e-05 2024-08-17T17:20:57.349 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.349 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.349 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:57.349 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.358151+0000", 2024-08-17T17:20:57.349 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2649999999999999e-06 2024-08-17T17:20:57.349 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.349 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.349 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.349 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.358226+0000", 2024-08-17T17:20:57.350 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5142999999999995e-05 2024-08-17T17:20:57.350 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.350 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.350 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.350 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.358238+0000", 2024-08-17T17:20:57.350 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2306e-05 2024-08-17T17:20:57.350 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.350 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.350 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.350 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.269985+0000", 2024-08-17T17:20:57.350 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.911747215 2024-08-17T17:20:57.350 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.350 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.350 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.351 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270001+0000", 2024-08-17T17:20:57.351 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5474000000000001e-05 2024-08-17T17:20:57.351 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.351 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.351 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.351 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.276689+0000", 2024-08-17T17:20:57.351 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0066882220000002 2024-08-17T17:20:57.351 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.351 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.351 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:57.351 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.277111+0000", 2024-08-17T17:20:57.351 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000421875 2024-08-17T17:20:57.351 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.351 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.352 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:20:57.352 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.277244+0000", 2024-08-17T17:20:57.352 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013283899999999999 2024-08-17T17:20:57.352 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.352 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.352 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:57.352 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.281296+0000", 2024-08-17T17:20:57.352 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040518719999999998 2024-08-17T17:20:57.352 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.352 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.352 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.352 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.281347+0000", 2024-08-17T17:20:57.352 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1427000000000001e-05 2024-08-17T17:20:57.352 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.353 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.353 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.353 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.281532+0000", 2024-08-17T17:20:57.353 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000185293 2024-08-17T17:20:57.353 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.353 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.353 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:57.353 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.281603+0000", 2024-08-17T17:20:57.353 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0518999999999993e-05 2024-08-17T17:20:57.353 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.353 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.353 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:57.353 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.282304+0000", 2024-08-17T17:20:57.353 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00070124300000000005 2024-08-17T17:20:57.353 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.353 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:57.354 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.354 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.354 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.354 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26098 5.1 5:89795322:::benchmark_data_smithi003_140793_object26097:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:20:57.354 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.365617+0000", 2024-08-17T17:20:57.354 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 94.975017299000001, 2024-08-17T17:20:57.354 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9216087019999999, 2024-08-17T17:20:57.354 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:57.354 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:57.354 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:57.354 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:57.354 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:57.354 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:57.354 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26098 2024-08-17T17:20:57.354 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.355 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:57.355 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.355 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:57.355 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365617+0000", 2024-08-17T17:20:57.355 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.355 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.355 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.355 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:57.355 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365617+0000", 2024-08-17T17:20:57.355 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.355 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.355 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.355 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:57.356 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365618+0000", 2024-08-17T17:20:57.356 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6249999999999999e-06 2024-08-17T17:20:57.356 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.356 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.356 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:57.356 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365655+0000", 2024-08-17T17:20:57.356 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6613999999999999e-05 2024-08-17T17:20:57.356 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.356 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.356 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:57.356 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365672+0000", 2024-08-17T17:20:57.356 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6844999999999999e-05 2024-08-17T17:20:57.356 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.356 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.356 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:57.357 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365679+0000", 2024-08-17T17:20:57.357 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4730000000000003e-06 2024-08-17T17:20:57.357 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.357 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.357 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.357 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366362+0000", 2024-08-17T17:20:57.357 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00068284800000000001 2024-08-17T17:20:57.357 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.357 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.357 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.357 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366374+0000", 2024-08-17T17:20:57.357 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1674e-05 2024-08-17T17:20:57.357 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.357 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.357 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.358 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270031+0000", 2024-08-17T17:20:57.358 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.90365675300000003 2024-08-17T17:20:57.358 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.358 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.358 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.358 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270039+0000", 2024-08-17T17:20:57.358 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1240000000000005e-06 2024-08-17T17:20:57.358 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.358 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.358 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.358 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.278729+0000", 2024-08-17T17:20:57.358 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0086905939999999 2024-08-17T17:20:57.358 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.358 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.359 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:57.359 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.278876+0000", 2024-08-17T17:20:57.359 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014623300000000001 2024-08-17T17:20:57.359 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.359 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.359 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:20:57.359 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.279011+0000", 2024-08-17T17:20:57.359 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013513899999999999 2024-08-17T17:20:57.359 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.359 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.359 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:57.359 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.281468+0000", 2024-08-17T17:20:57.359 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024572579999999999 2024-08-17T17:20:57.359 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.360 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.360 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.360 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.282380+0000", 2024-08-17T17:20:57.360 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00091197499999999998 2024-08-17T17:20:57.360 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.360 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.360 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.360 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.286511+0000", 2024-08-17T17:20:57.360 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0041307979999999998 2024-08-17T17:20:57.360 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.360 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.360 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:57.360 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.286586+0000", 2024-08-17T17:20:57.360 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5006000000000007e-05 2024-08-17T17:20:57.361 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.361 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.361 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:57.361 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.287226+0000", 2024-08-17T17:20:57.361 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00063974300000000002 2024-08-17T17:20:57.361 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.361 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:57.361 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.361 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.361 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.361 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26099 5.1 5:87830c36:::benchmark_data_smithi003_140793_object26098:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:20:57.361 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.365700+0000", 2024-08-17T17:20:57.361 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 94.974933883000006, 2024-08-17T17:20:57.361 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9222799620000002, 2024-08-17T17:20:57.361 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:57.362 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:57.362 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:57.362 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:57.362 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:57.362 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:57.362 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26099 2024-08-17T17:20:57.362 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.362 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:57.362 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.362 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:57.362 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365700+0000", 2024-08-17T17:20:57.362 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.362 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.362 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.362 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:57.363 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365700+0000", 2024-08-17T17:20:57.363 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.363 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.363 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.363 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:57.363 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365701+0000", 2024-08-17T17:20:57.363 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1799999999999995e-07 2024-08-17T17:20:57.363 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.363 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.363 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:57.363 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365722+0000", 2024-08-17T17:20:57.363 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0897999999999999e-05 2024-08-17T17:20:57.363 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.363 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.363 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:57.364 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365729+0000", 2024-08-17T17:20:57.364 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0539999999999997e-06 2024-08-17T17:20:57.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.364 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.364 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:57.364 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365734+0000", 2024-08-17T17:20:57.364 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9910000000000002e-06 2024-08-17T17:20:57.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.364 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.364 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.364 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366399+0000", 2024-08-17T17:20:57.364 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00066523799999999996 2024-08-17T17:20:57.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.364 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.364 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.365 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366406+0000", 2024-08-17T17:20:57.365 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3889999999999999e-06 2024-08-17T17:20:57.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.365 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.365 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270065+0000", 2024-08-17T17:20:57.365 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.90365837800000004 2024-08-17T17:20:57.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.365 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.365 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270072+0000", 2024-08-17T17:20:57.365 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5390000000000002e-06 2024-08-17T17:20:57.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.366 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.280026+0000", 2024-08-17T17:20:57.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.009954215 2024-08-17T17:20:57.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:57.366 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.280176+0000", 2024-08-17T17:20:57.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014965900000000001 2024-08-17T17:20:57.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:20:57.366 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.280355+0000", 2024-08-17T17:20:57.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000178586 2024-08-17T17:20:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.282543+0000", 2024-08-17T17:20:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0021886539999999999 2024-08-17T17:20:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.282560+0000", 2024-08-17T17:20:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6864e-05 2024-08-17T17:20:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.287385+0000", 2024-08-17T17:20:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0048250080000000004 2024-08-17T17:20:57.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.368 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.368 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:57.368 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.287456+0000", 2024-08-17T17:20:57.368 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0271000000000005e-05 2024-08-17T17:20:57.368 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.368 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.368 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:57.368 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.287980+0000", 2024-08-17T17:20:57.368 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00052470000000000001 2024-08-17T17:20:57.368 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.368 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:57.368 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.368 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.368 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.368 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26102 5.1 5:806d080d:::benchmark_data_smithi003_140793_object26101:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:20:57.369 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.366504+0000", 2024-08-17T17:20:57.369 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 94.974130095000007, 2024-08-17T17:20:57.369 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9223289210000001, 2024-08-17T17:20:57.369 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:57.369 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:57.369 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:57.369 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:57.369 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:57.369 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:57.369 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26102 2024-08-17T17:20:57.369 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.369 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:57.369 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.370 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:57.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366504+0000", 2024-08-17T17:20:57.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.370 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.370 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:57.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366504+0000", 2024-08-17T17:20:57.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.370 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.370 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:57.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366506+0000", 2024-08-17T17:20:57.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5149999999999999e-06 2024-08-17T17:20:57.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:57.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366531+0000", 2024-08-17T17:20:57.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5834e-05 2024-08-17T17:20:57.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:57.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366547+0000", 2024-08-17T17:20:57.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5235999999999999e-05 2024-08-17T17:20:57.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:57.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366554+0000", 2024-08-17T17:20:57.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2699999999999999e-06 2024-08-17T17:20:57.372 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.372 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.372 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366661+0000", 2024-08-17T17:20:57.372 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000107457 2024-08-17T17:20:57.372 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.372 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.373 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366671+0000", 2024-08-17T17:20:57.373 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3559999999999999e-06 2024-08-17T17:20:57.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.373 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.373 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.373 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270100+0000", 2024-08-17T17:20:57.373 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.90342893899999999 2024-08-17T17:20:57.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.374 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.374 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270108+0000", 2024-08-17T17:20:57.374 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6030000000000005e-06 2024-08-17T17:20:57.374 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.374 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.374 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.282611+0000", 2024-08-17T17:20:57.374 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.012502784 2024-08-17T17:20:57.374 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.374 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:57.374 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.282745+0000", 2024-08-17T17:20:57.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000133504 2024-08-17T17:20:57.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.375 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:20:57.375 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.282979+0000", 2024-08-17T17:20:57.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00023466900000000001 2024-08-17T17:20:57.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.375 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:57.375 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.287245+0000", 2024-08-17T17:20:57.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042655929999999998 2024-08-17T17:20:57.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.375 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.375 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.287308+0000", 2024-08-17T17:20:57.376 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2869999999999999e-05 2024-08-17T17:20:57.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.376 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.288280+0000", 2024-08-17T17:20:57.376 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00097271 2024-08-17T17:20:57.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:57.376 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.288377+0000", 2024-08-17T17:20:57.376 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6137000000000005e-05 2024-08-17T17:20:57.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:57.377 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.288833+0000", 2024-08-17T17:20:57.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00045644400000000002 2024-08-17T17:20:57.377 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.377 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:57.377 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.377 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26120 5.1 5:8459086c:::benchmark_data_smithi003_140793_object26119:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:20:57.377 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.375347+0000", 2024-08-17T17:20:57.377 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 94.965287099999998, 2024-08-17T17:20:57.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9140657679999999, 2024-08-17T17:20:57.377 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:57.377 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:57.377 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:57.377 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:57.378 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:57.378 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:57.378 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26120 2024-08-17T17:20:57.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.378 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:57.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:57.378 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375347+0000", 2024-08-17T17:20:57.378 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:57.378 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375347+0000", 2024-08-17T17:20:57.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:57.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375348+0000", 2024-08-17T17:20:57.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1939999999999999e-06 2024-08-17T17:20:57.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:57.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375371+0000", 2024-08-17T17:20:57.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2997e-05 2024-08-17T17:20:57.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:57.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375445+0000", 2024-08-17T17:20:57.380 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3657000000000003e-05 2024-08-17T17:20:57.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.380 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.380 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:57.380 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375456+0000", 2024-08-17T17:20:57.380 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0941000000000001e-05 2024-08-17T17:20:57.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.380 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.380 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.380 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375518+0000", 2024-08-17T17:20:57.380 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2533999999999995e-05 2024-08-17T17:20:57.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.381 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.381 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375531+0000", 2024-08-17T17:20:57.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.238e-05 2024-08-17T17:20:57.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.381 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.381 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270132+0000", 2024-08-17T17:20:57.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89460134800000002 2024-08-17T17:20:57.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.381 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.381 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270140+0000", 2024-08-17T17:20:57.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0299999999999994e-06 2024-08-17T17:20:57.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.382 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.382 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.382 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.284149+0000", 2024-08-17T17:20:57.382 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0140088189999998 2024-08-17T17:20:57.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.382 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.382 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:57.382 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.284323+0000", 2024-08-17T17:20:57.382 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017453600000000001 2024-08-17T17:20:57.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.382 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.382 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:20:57.383 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.284507+0000", 2024-08-17T17:20:57.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000183692 2024-08-17T17:20:57.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:57.383 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.288124+0000", 2024-08-17T17:20:57.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0036170659999999999 2024-08-17T17:20:57.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.383 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.288193+0000", 2024-08-17T17:20:57.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9238000000000004e-05 2024-08-17T17:20:57.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.288892+0000", 2024-08-17T17:20:57.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00069895500000000002 2024-08-17T17:20:57.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:57.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.288930+0000", 2024-08-17T17:20:57.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7125000000000001e-05 2024-08-17T17:20:57.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:57.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.289413+0000", 2024-08-17T17:20:57.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000483256 2024-08-17T17:20:57.384 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.384 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:57.384 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.385 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.385 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26121 5.1 5:8d0269c6:::benchmark_data_smithi003_140793_object26120:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:20:57.385 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.375480+0000", 2024-08-17T17:20:57.385 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 94.965153904999994, 2024-08-17T17:20:57.385 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.921705099, 2024-08-17T17:20:57.385 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:57.385 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:57.385 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:57.385 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:57.385 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:57.385 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:57.385 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26121 2024-08-17T17:20:57.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.385 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:57.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.386 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:57.386 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375480+0000", 2024-08-17T17:20:57.386 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.386 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:57.386 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375480+0000", 2024-08-17T17:20:57.386 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.386 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:57.386 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375481+0000", 2024-08-17T17:20:57.387 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.44e-07 2024-08-17T17:20:57.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.387 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:57.387 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375502+0000", 2024-08-17T17:20:57.387 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.086e-05 2024-08-17T17:20:57.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.387 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:57.387 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375511+0000", 2024-08-17T17:20:57.387 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5429999999999999e-06 2024-08-17T17:20:57.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.388 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:57.388 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375518+0000", 2024-08-17T17:20:57.388 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2949999999999999e-06 2024-08-17T17:20:57.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.388 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.388 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375557+0000", 2024-08-17T17:20:57.388 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9255999999999998e-05 2024-08-17T17:20:57.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.388 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.388 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375565+0000", 2024-08-17T17:20:57.388 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1289999999999996e-06 2024-08-17T17:20:57.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.389 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.389 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270164+0000", 2024-08-17T17:20:57.389 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89459907800000005 2024-08-17T17:20:57.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.389 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.389 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270172+0000", 2024-08-17T17:20:57.389 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8869999999999992e-06 2024-08-17T17:20:57.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.389 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.389 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.289461+0000", 2024-08-17T17:20:57.389 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0192893380000001 2024-08-17T17:20:57.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.390 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:57.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.289616+0000", 2024-08-17T17:20:57.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015440199999999999 2024-08-17T17:20:57.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.390 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:20:57.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.289854+0000", 2024-08-17T17:20:57.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000238271 2024-08-17T17:20:57.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.390 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.296410+0000", 2024-08-17T17:20:57.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006555735 2024-08-17T17:20:57.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.391 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:57.391 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.296424+0000", 2024-08-17T17:20:57.391 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4679e-05 2024-08-17T17:20:57.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.391 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.391 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.296471+0000", 2024-08-17T17:20:57.391 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6600000000000001e-05 2024-08-17T17:20:57.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.391 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:57.391 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.296537+0000", 2024-08-17T17:20:57.391 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5733000000000002e-05 2024-08-17T17:20:57.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.392 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:57.392 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.297185+0000", 2024-08-17T17:20:57.392 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00064864899999999995 2024-08-17T17:20:57.392 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.392 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:57.392 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.392 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26129 5.1 5:845981d1:::benchmark_data_smithi003_140793_object26128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:20:57.392 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.378910+0000", 2024-08-17T17:20:57.392 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 94.961723950999996, 2024-08-17T17:20:57.392 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9327440490000001, 2024-08-17T17:20:57.392 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:57.393 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:57.393 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:57.393 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:57.393 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:57.393 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:57.393 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26129 2024-08-17T17:20:57.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.393 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:57.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:57.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.378910+0000", 2024-08-17T17:20:57.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:57.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.378910+0000", 2024-08-17T17:20:57.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:57.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.378912+0000", 2024-08-17T17:20:57.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.657e-06 2024-08-17T17:20:57.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:57.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.378935+0000", 2024-08-17T17:20:57.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2795000000000001e-05 2024-08-17T17:20:57.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:57.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.378956+0000", 2024-08-17T17:20:57.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1744000000000001e-05 2024-08-17T17:20:57.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:57.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.378965+0000", 2024-08-17T17:20:57.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2930000000000001e-06 2024-08-17T17:20:57.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.379129+0000", 2024-08-17T17:20:57.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016463799999999999 2024-08-17T17:20:57.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.379143+0000", 2024-08-17T17:20:57.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.341e-05 2024-08-17T17:20:57.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.396 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.396 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270196+0000", 2024-08-17T17:20:57.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89105351399999999 2024-08-17T17:20:57.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.396 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.396 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270204+0000", 2024-08-17T17:20:57.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8159999999999994e-06 2024-08-17T17:20:57.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.396 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.396 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.294939+0000", 2024-08-17T17:20:57.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.024735352 2024-08-17T17:20:57.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:57.397 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.295108+0000", 2024-08-17T17:20:57.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016822500000000001 2024-08-17T17:20:57.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:20:57.397 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.295294+0000", 2024-08-17T17:20:57.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000186107 2024-08-17T17:20:57.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.398 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.398 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.301319+0000", 2024-08-17T17:20:57.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0060248339999999997 2024-08-17T17:20:57.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.398 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:57.398 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.304206+0000", 2024-08-17T17:20:57.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028874009999999999 2024-08-17T17:20:57.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.398 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.398 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.310650+0000", 2024-08-17T17:20:57.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0064438689999999996 2024-08-17T17:20:57.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:57.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.310709+0000", 2024-08-17T17:20:57.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9494000000000003e-05 2024-08-17T17:20:57.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:57.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.311654+0000", 2024-08-17T17:20:57.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00094490000000000004 2024-08-17T17:20:57.399 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.399 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:57.399 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:57.400 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:57.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:57.401 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26134 5.1 5:8a41a77b:::benchmark_data_smithi003_140793_object26133:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:20:57.401 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.380465+0000", 2024-08-17T17:20:57.401 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 94.960168951, 2024-08-17T17:20:57.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9324733580000002, 2024-08-17T17:20:57.401 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:57.401 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:57.401 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:57.401 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:57.401 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:57.402 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:57.402 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26134 2024-08-17T17:20:57.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.402 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:57.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:57.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.380465+0000", 2024-08-17T17:20:57.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:57.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.380465+0000", 2024-08-17T17:20:57.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:57.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.380466+0000", 2024-08-17T17:20:57.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.28e-06 2024-08-17T17:20:57.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:57.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.380490+0000", 2024-08-17T17:20:57.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3842999999999999e-05 2024-08-17T17:20:57.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:57.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.380506+0000", 2024-08-17T17:20:57.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6065e-05 2024-08-17T17:20:57.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:57.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.380514+0000", 2024-08-17T17:20:57.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8529999999999992e-06 2024-08-17T17:20:57.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.380591+0000", 2024-08-17T17:20:57.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6271000000000002e-05 2024-08-17T17:20:57.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.380603+0000", 2024-08-17T17:20:57.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2867999999999999e-05 2024-08-17T17:20:57.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270227+0000", 2024-08-17T17:20:57.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.88962385499999996 2024-08-17T17:20:57.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270235+0000", 2024-08-17T17:20:57.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6120000000000001e-06 2024-08-17T17:20:57.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.297325+0000", 2024-08-17T17:20:57.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0270902529999999 2024-08-17T17:20:57.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:57.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.297474+0000", 2024-08-17T17:20:57.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014895899999999999 2024-08-17T17:20:57.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:20:57.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.297667+0000", 2024-08-17T17:20:57.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019309500000000001 2024-08-17T17:20:57.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.304283+0000", 2024-08-17T17:20:57.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066158759999999997 2024-08-17T17:20:57.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:57.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.310600+0000", 2024-08-17T17:20:57.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063167420000000002 2024-08-17T17:20:57.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.312211+0000", 2024-08-17T17:20:57.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016113029999999999 2024-08-17T17:20:57.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:57.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.312376+0000", 2024-08-17T17:20:57.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000164804 2024-08-17T17:20:57.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:57.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.312939+0000", 2024-08-17T17:20:57.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00056267899999999996 2024-08-17T17:20:57.408 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.408 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:57.408 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.409 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26136 5.1 5:81387a84:::benchmark_data_smithi003_140793_object26135:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:20:57.409 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.382929+0000", 2024-08-17T17:20:57.409 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 94.957705028000007, 2024-08-17T17:20:57.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.930695842, 2024-08-17T17:20:57.409 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:57.409 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:57.409 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:57.409 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:57.409 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:57.409 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:57.409 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26136 2024-08-17T17:20:57.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.409 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:57.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:57.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.382929+0000", 2024-08-17T17:20:57.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:57.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.382929+0000", 2024-08-17T17:20:57.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:57.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.382931+0000", 2024-08-17T17:20:57.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5120000000000001e-06 2024-08-17T17:20:57.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:57.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.382982+0000", 2024-08-17T17:20:57.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1314999999999999e-05 2024-08-17T17:20:57.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:57.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.382998+0000", 2024-08-17T17:20:57.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6235999999999998e-05 2024-08-17T17:20:57.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:57.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.383006+0000", 2024-08-17T17:20:57.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4760000000000001e-06 2024-08-17T17:20:57.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.383064+0000", 2024-08-17T17:20:57.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8255999999999999e-05 2024-08-17T17:20:57.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.383073+0000", 2024-08-17T17:20:57.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5650000000000007e-06 2024-08-17T17:20:57.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270259+0000", 2024-08-17T17:20:57.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.88718503699999995 2024-08-17T17:20:57.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270266+0000", 2024-08-17T17:20:57.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.892e-06 2024-08-17T17:20:57.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.298758+0000", 2024-08-17T17:20:57.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0284916499999999 2024-08-17T17:20:57.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:57.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.298905+0000", 2024-08-17T17:20:57.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014658 2024-08-17T17:20:57.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:20:57.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.299067+0000", 2024-08-17T17:20:57.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016263700000000001 2024-08-17T17:20:57.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.304334+0000", 2024-08-17T17:20:57.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0052671059999999997 2024-08-17T17:20:57.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:57.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.310606+0000", 2024-08-17T17:20:57.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0062720830000000003 2024-08-17T17:20:57.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.313094+0000", 2024-08-17T17:20:57.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002487763 2024-08-17T17:20:57.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:57.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.313146+0000", 2024-08-17T17:20:57.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2079000000000001e-05 2024-08-17T17:20:57.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.416 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:57.416 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.313625+0000", 2024-08-17T17:20:57.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00047865499999999998 2024-08-17T17:20:57.416 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.416 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:57.416 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.416 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26145 5.1 5:896bb16f:::benchmark_data_smithi003_140793_object26144:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:20:57.416 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.386726+0000", 2024-08-17T17:20:57.416 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 94.953908041000005, 2024-08-17T17:20:57.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9275205729999998, 2024-08-17T17:20:57.417 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:57.417 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:57.417 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:57.417 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:57.417 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:57.417 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:57.417 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26145 2024-08-17T17:20:57.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.417 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:57.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.417 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:57.417 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.386726+0000", 2024-08-17T17:20:57.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:57.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.386726+0000", 2024-08-17T17:20:57.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:57.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.386727+0000", 2024-08-17T17:20:57.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.291e-06 2024-08-17T17:20:57.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:57.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.386775+0000", 2024-08-17T17:20:57.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7855999999999997e-05 2024-08-17T17:20:57.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:57.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.386791+0000", 2024-08-17T17:20:57.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6044000000000001e-05 2024-08-17T17:20:57.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:57.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.386799+0000", 2024-08-17T17:20:57.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7510000000000008e-06 2024-08-17T17:20:57.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.386873+0000", 2024-08-17T17:20:57.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4208000000000005e-05 2024-08-17T17:20:57.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.386887+0000", 2024-08-17T17:20:57.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3699e-05 2024-08-17T17:20:57.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270290+0000", 2024-08-17T17:20:57.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.88340287500000003 2024-08-17T17:20:57.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270298+0000", 2024-08-17T17:20:57.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0399999999999993e-06 2024-08-17T17:20:57.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.421 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.421 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.299986+0000", 2024-08-17T17:20:57.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0296885850000002 2024-08-17T17:20:57.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.421 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:57.421 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.300132+0000", 2024-08-17T17:20:57.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014511 2024-08-17T17:20:57.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.421 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:20:57.421 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.300298+0000", 2024-08-17T17:20:57.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016594799999999999 2024-08-17T17:20:57.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.422 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.304384+0000", 2024-08-17T17:20:57.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040865720000000001 2024-08-17T17:20:57.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:57.422 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.311674+0000", 2024-08-17T17:20:57.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072902210000000004 2024-08-17T17:20:57.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.422 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.313682+0000", 2024-08-17T17:20:57.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00200746 2024-08-17T17:20:57.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.423 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:57.423 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.313742+0000", 2024-08-17T17:20:57.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0265e-05 2024-08-17T17:20:57.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.423 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:57.423 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.314247+0000", 2024-08-17T17:20:57.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00050464800000000001 2024-08-17T17:20:57.423 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.423 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:57.423 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.424 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26147 5.1 5:85f2e61e:::benchmark_data_smithi003_140793_object26146:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:20:57.424 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.387915+0000", 2024-08-17T17:20:57.424 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 94.952719553999998, 2024-08-17T17:20:57.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9269894829999998, 2024-08-17T17:20:57.424 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:57.424 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:57.424 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:57.424 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:57.424 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:57.424 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:57.424 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26147 2024-08-17T17:20:57.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.424 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:57.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:57.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.387915+0000", 2024-08-17T17:20:57.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:57.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.387915+0000", 2024-08-17T17:20:57.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:57.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.387915+0000", 2024-08-17T17:20:57.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6899999999999996e-07 2024-08-17T17:20:57.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.426 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.426 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:57.426 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.387953+0000", 2024-08-17T17:20:57.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7447000000000002e-05 2024-08-17T17:20:57.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.426 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.426 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:57.426 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.387963+0000", 2024-08-17T17:20:57.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0302999999999999e-05 2024-08-17T17:20:57.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.426 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.426 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:57.426 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.387969+0000", 2024-08-17T17:20:57.427 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1090000000000003e-06 2024-08-17T17:20:57.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.427 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.427 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.388038+0000", 2024-08-17T17:20:57.427 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9042999999999996e-05 2024-08-17T17:20:57.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.427 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.427 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.388051+0000", 2024-08-17T17:20:57.427 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2498e-05 2024-08-17T17:20:57.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.427 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.427 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270321+0000", 2024-08-17T17:20:57.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.88227023299999996 2024-08-17T17:20:57.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.428 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270329+0000", 2024-08-17T17:20:57.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9240000000000007e-06 2024-08-17T17:20:57.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.428 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.301403+0000", 2024-08-17T17:20:57.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0310743539999998 2024-08-17T17:20:57.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:57.429 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.301580+0000", 2024-08-17T17:20:57.429 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017625900000000001 2024-08-17T17:20:57.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.429 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.429 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:20:57.429 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.301745+0000", 2024-08-17T17:20:57.429 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000165728 2024-08-17T17:20:57.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.429 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.429 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.429 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.307080+0000", 2024-08-17T17:20:57.429 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0053345570000000002 2024-08-17T17:20:57.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.430 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:57.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.311680+0000", 2024-08-17T17:20:57.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0046006270000000004 2024-08-17T17:20:57.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.430 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.314305+0000", 2024-08-17T17:20:57.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026242890000000001 2024-08-17T17:20:57.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.430 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:57.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.314364+0000", 2024-08-17T17:20:57.431 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8853e-05 2024-08-17T17:20:57.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.431 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:57.431 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.314904+0000", 2024-08-17T17:20:57.431 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00054049000000000002 2024-08-17T17:20:57.431 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.431 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:57.431 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.431 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26154 5.1 5:8b1cb6d6:::benchmark_data_smithi003_140793_object26153:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:20:57.431 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.390694+0000", 2024-08-17T17:20:57.432 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 94.949940362000007, 2024-08-17T17:20:57.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9249220940000002, 2024-08-17T17:20:57.432 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:57.432 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:57.432 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:57.432 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:57.432 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:57.432 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:57.432 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26154 2024-08-17T17:20:57.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.432 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:57.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.433 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:57.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.390694+0000", 2024-08-17T17:20:57.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.433 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:57.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.390694+0000", 2024-08-17T17:20:57.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.433 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:57.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.390695+0000", 2024-08-17T17:20:57.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3349999999999999e-06 2024-08-17T17:20:57.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:57.434 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.390716+0000", 2024-08-17T17:20:57.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0854999999999999e-05 2024-08-17T17:20:57.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:57.434 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.390728+0000", 2024-08-17T17:20:57.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2455e-05 2024-08-17T17:20:57.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:57.434 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.390736+0000", 2024-08-17T17:20:57.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0509999999999997e-06 2024-08-17T17:20:57.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.435 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.435 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.390824+0000", 2024-08-17T17:20:57.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7665999999999997e-05 2024-08-17T17:20:57.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.435 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.435 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.390835+0000", 2024-08-17T17:20:57.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0987e-05 2024-08-17T17:20:57.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.435 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.435 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270353+0000", 2024-08-17T17:20:57.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.879518146 2024-08-17T17:20:57.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.436 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.436 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270361+0000", 2024-08-17T17:20:57.436 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8779999999999998e-06 2024-08-17T17:20:57.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.436 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.436 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.302895+0000", 2024-08-17T17:20:57.436 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0325342339999999 2024-08-17T17:20:57.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.436 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:57.436 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.303046+0000", 2024-08-17T17:20:57.436 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015052899999999999 2024-08-17T17:20:57.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:20:57.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.303235+0000", 2024-08-17T17:20:57.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018953700000000001 2024-08-17T17:20:57.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.307121+0000", 2024-08-17T17:20:57.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038852779999999998 2024-08-17T17:20:57.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:57.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.311686+0000", 2024-08-17T17:20:57.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0045651889999999999 2024-08-17T17:20:57.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.315019+0000", 2024-08-17T17:20:57.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0033328609999999999 2024-08-17T17:20:57.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:57.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.315081+0000", 2024-08-17T17:20:57.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.198e-05 2024-08-17T17:20:57.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:57.439 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.315616+0000", 2024-08-17T17:20:57.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00053511300000000004 2024-08-17T17:20:57.439 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.439 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:57.439 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.439 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26175 5.1 5:8c6f3adb:::benchmark_data_smithi003_140793_object26174:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:20:57.439 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.403515+0000", 2024-08-17T17:20:57.439 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 94.937119291000002, 2024-08-17T17:20:57.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9127047070000001, 2024-08-17T17:20:57.440 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:57.440 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:57.440 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:57.440 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:57.440 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:57.440 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:57.440 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26175 2024-08-17T17:20:57.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.440 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:57.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:57.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.403515+0000", 2024-08-17T17:20:57.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.441 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:57.441 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.403515+0000", 2024-08-17T17:20:57.441 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.441 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:57.441 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.403517+0000", 2024-08-17T17:20:57.441 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7910000000000001e-06 2024-08-17T17:20:57.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.442 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:57.442 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.403543+0000", 2024-08-17T17:20:57.442 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6806999999999999e-05 2024-08-17T17:20:57.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.442 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:57.442 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.403558+0000", 2024-08-17T17:20:57.442 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5034999999999999e-05 2024-08-17T17:20:57.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.442 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:57.443 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.403567+0000", 2024-08-17T17:20:57.443 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8789999999999992e-06 2024-08-17T17:20:57.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.443 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.443 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.403634+0000", 2024-08-17T17:20:57.443 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6321000000000004e-05 2024-08-17T17:20:57.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.443 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.443 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.403646+0000", 2024-08-17T17:20:57.443 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2628e-05 2024-08-17T17:20:57.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.444 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.444 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270384+0000", 2024-08-17T17:20:57.444 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.86673805400000004 2024-08-17T17:20:57.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.444 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.444 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270392+0000", 2024-08-17T17:20:57.444 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8380000000000002e-06 2024-08-17T17:20:57.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.444 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.444 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.304419+0000", 2024-08-17T17:20:57.444 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0340264549999998 2024-08-17T17:20:57.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.445 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.445 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:57.445 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.304562+0000", 2024-08-17T17:20:57.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014378500000000001 2024-08-17T17:20:57.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.445 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.445 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:20:57.445 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.304715+0000", 2024-08-17T17:20:57.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015284700000000001 2024-08-17T17:20:57.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.445 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.445 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.445 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.310579+0000", 2024-08-17T17:20:57.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0058634339999999998 2024-08-17T17:20:57.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.446 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:57.446 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.311692+0000", 2024-08-17T17:20:57.446 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001112912 2024-08-17T17:20:57.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.446 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.446 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.315676+0000", 2024-08-17T17:20:57.446 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039842649999999999 2024-08-17T17:20:57.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.446 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:57.446 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.315747+0000", 2024-08-17T17:20:57.447 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0735000000000006e-05 2024-08-17T17:20:57.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.447 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:57.447 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.316220+0000", 2024-08-17T17:20:57.447 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00047292100000000001 2024-08-17T17:20:57.447 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.447 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:57.447 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.447 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26336 5.1 5:837f3a0a:::benchmark_data_smithi003_140793_object26335:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:20:57.447 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.566135+0000", 2024-08-17T17:20:57.447 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 94.774498777999995, 2024-08-17T17:20:57.448 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7524043960000002, 2024-08-17T17:20:57.448 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:57.448 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:57.448 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:57.448 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:57.448 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:57.448 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:57.448 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26336 2024-08-17T17:20:57.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.448 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:57.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.448 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:57.448 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.566135+0000", 2024-08-17T17:20:57.448 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.449 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:57.449 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.566135+0000", 2024-08-17T17:20:57.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.449 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:57.449 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.566137+0000", 2024-08-17T17:20:57.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.477e-06 2024-08-17T17:20:57.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.449 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:57.450 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.566159+0000", 2024-08-17T17:20:57.450 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1691e-05 2024-08-17T17:20:57.450 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.450 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.450 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:57.450 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.566175+0000", 2024-08-17T17:20:57.450 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6249999999999999e-05 2024-08-17T17:20:57.450 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.450 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.450 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:57.450 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.566184+0000", 2024-08-17T17:20:57.450 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0299999999999999e-06 2024-08-17T17:20:57.450 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.451 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.451 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.566258+0000", 2024-08-17T17:20:57.451 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3739999999999995e-05 2024-08-17T17:20:57.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.451 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.451 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.566270+0000", 2024-08-17T17:20:57.451 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2931e-05 2024-08-17T17:20:57.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.451 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.451 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270415+0000", 2024-08-17T17:20:57.451 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.70414461900000003 2024-08-17T17:20:57.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.452 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.452 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270423+0000", 2024-08-17T17:20:57.452 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4329999999999998e-06 2024-08-17T17:20:57.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.452 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.452 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.305724+0000", 2024-08-17T17:20:57.452 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.03530159 2024-08-17T17:20:57.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.452 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:57.452 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.305884+0000", 2024-08-17T17:20:57.452 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000159658 2024-08-17T17:20:57.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.453 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:20:57.453 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.306079+0000", 2024-08-17T17:20:57.453 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019524700000000001 2024-08-17T17:20:57.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.453 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:57.453 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.311698+0000", 2024-08-17T17:20:57.453 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0056185810000000001 2024-08-17T17:20:57.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.453 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.453 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.312166+0000", 2024-08-17T17:20:57.453 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000468314 2024-08-17T17:20:57.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.454 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.454 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.454 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.317902+0000", 2024-08-17T17:20:57.454 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0057355949999999996 2024-08-17T17:20:57.454 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.454 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.454 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:57.454 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.317959+0000", 2024-08-17T17:20:57.454 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7932e-05 2024-08-17T17:20:57.454 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.454 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.454 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:57.454 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.318540+0000", 2024-08-17T17:20:57.454 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00058030799999999997 2024-08-17T17:20:57.455 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.455 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:57.455 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.455 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.455 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.455 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:57 smithi003 ceph-mon[103344]: pgmap v305: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 531 op/s; 27690/284832 objects degraded (9.722%); 25150/284832 objects misplaced (8.830%); 12 MiB/s, 198 objects/s recovering 2024-08-17T17:20:57.455 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:57 smithi003 ceph-mon[100615]: pgmap v305: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 531 op/s; 27690/284832 objects degraded (9.722%); 25150/284832 objects misplaced (8.830%); 12 MiB/s, 198 objects/s recovering 2024-08-17T17:20:57.456 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26347 5.1 5:83125685:::benchmark_data_smithi003_140793_object26346:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:20:57.456 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.575398+0000", 2024-08-17T17:20:57.456 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 94.765235884999996, 2024-08-17T17:20:57.456 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7503322890000002, 2024-08-17T17:20:57.456 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:57.456 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:57.457 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:57.457 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:57.457 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:57.457 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:57.457 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26347 2024-08-17T17:20:57.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.457 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:57.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.457 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:57.457 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.575398+0000", 2024-08-17T17:20:57.457 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.457 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:57.458 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.575398+0000", 2024-08-17T17:20:57.458 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.458 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:57.458 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.575400+0000", 2024-08-17T17:20:57.458 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.683e-06 2024-08-17T17:20:57.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.458 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:57.458 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.575430+0000", 2024-08-17T17:20:57.458 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9808e-05 2024-08-17T17:20:57.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.459 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:57.459 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.575447+0000", 2024-08-17T17:20:57.459 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7257000000000001e-05 2024-08-17T17:20:57.459 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.459 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:57.459 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.575456+0000", 2024-08-17T17:20:57.459 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8230000000000001e-06 2024-08-17T17:20:57.459 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.459 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.459 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.575520+0000", 2024-08-17T17:20:57.460 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4515000000000001e-05 2024-08-17T17:20:57.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.460 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.460 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.575529+0000", 2024-08-17T17:20:57.460 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3119999999999994e-06 2024-08-17T17:20:57.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.460 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.460 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270446+0000", 2024-08-17T17:20:57.460 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.69491715499999995 2024-08-17T17:20:57.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.460 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.460 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270454+0000", 2024-08-17T17:20:57.461 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8099999999999998e-06 2024-08-17T17:20:57.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.461 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.461 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.307156+0000", 2024-08-17T17:20:57.461 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0367026579999998 2024-08-17T17:20:57.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.461 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:57.461 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.307253+0000", 2024-08-17T17:20:57.461 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6832999999999999e-05 2024-08-17T17:20:57.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.461 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:20:57.461 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.307393+0000", 2024-08-17T17:20:57.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014028399999999999 2024-08-17T17:20:57.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.462 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.462 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:57.462 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.311703+0000", 2024-08-17T17:20:57.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004309729 2024-08-17T17:20:57.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.462 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.462 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.462 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.312995+0000", 2024-08-17T17:20:57.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0012918820000000001 2024-08-17T17:20:57.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.462 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.463 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.463 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.324907+0000", 2024-08-17T17:20:57.463 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011912114999999999 2024-08-17T17:20:57.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.463 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.463 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:57.463 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.325027+0000", 2024-08-17T17:20:57.463 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000119506 2024-08-17T17:20:57.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.463 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.463 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:57.463 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.325731+0000", 2024-08-17T17:20:57.463 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00070391900000000001 2024-08-17T17:20:57.463 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.463 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:57.464 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.464 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26360 5.1 5:89f4cb1c:::benchmark_data_smithi003_140793_object26359:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:20:57.464 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.594844+0000", 2024-08-17T17:20:57.464 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 94.745790526999997, 2024-08-17T17:20:57.464 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7437470209999999, 2024-08-17T17:20:57.464 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:57.464 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:57.464 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:57.464 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:57.464 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:57.464 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:57.464 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26360 2024-08-17T17:20:57.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.465 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:57.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.465 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:57.465 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.594844+0000", 2024-08-17T17:20:57.465 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.465 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:57.465 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.594844+0000", 2024-08-17T17:20:57.465 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.465 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:57.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.594845+0000", 2024-08-17T17:20:57.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2929999999999999e-06 2024-08-17T17:20:57.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.466 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:57.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.594861+0000", 2024-08-17T17:20:57.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6368e-05 2024-08-17T17:20:57.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.466 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:57.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.594872+0000", 2024-08-17T17:20:57.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1079e-05 2024-08-17T17:20:57.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.467 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:57.467 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.594878+0000", 2024-08-17T17:20:57.467 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5629999999999999e-06 2024-08-17T17:20:57.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.467 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.467 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.594926+0000", 2024-08-17T17:20:57.467 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8498000000000002e-05 2024-08-17T17:20:57.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.467 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.467 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.594934+0000", 2024-08-17T17:20:57.467 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7999999999999999e-06 2024-08-17T17:20:57.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.468 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.468 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270477+0000", 2024-08-17T17:20:57.468 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.67554314800000004 2024-08-17T17:20:57.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.468 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.468 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270486+0000", 2024-08-17T17:20:57.468 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5650000000000002e-06 2024-08-17T17:20:57.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.468 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.468 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.316324+0000", 2024-08-17T17:20:57.468 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0458376779999998 2024-08-17T17:20:57.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.469 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.469 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:57.469 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.316469+0000", 2024-08-17T17:20:57.469 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014542299999999999 2024-08-17T17:20:57.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.469 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.469 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:20:57.469 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.316621+0000", 2024-08-17T17:20:57.469 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015184700000000001 2024-08-17T17:20:57.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.469 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.469 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.469 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.320092+0000", 2024-08-17T17:20:57.470 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034713589999999998 2024-08-17T17:20:57.470 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.470 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.470 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:57.470 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.320106+0000", 2024-08-17T17:20:57.470 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3382e-05 2024-08-17T17:20:57.470 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.470 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.470 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.470 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.337839+0000", 2024-08-17T17:20:57.470 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017733498 2024-08-17T17:20:57.470 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.470 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.471 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:57.471 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.337900+0000", 2024-08-17T17:20:57.471 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1266999999999995e-05 2024-08-17T17:20:57.471 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.471 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.471 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:57.471 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.338591+0000", 2024-08-17T17:20:57.471 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00069025300000000005 2024-08-17T17:20:57.471 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.471 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:57.471 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.471 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.471 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.471 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:57339 5.1 5:8dd4b2b4:::benchmark_data_smithi003_140793_object57338:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-08-17T17:20:57.471 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:46.499070+0000", 2024-08-17T17:20:57.471 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 70.841563965999995, 2024-08-17T17:20:57.472 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96002446600000002, 2024-08-17T17:20:57.472 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:57.472 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:57.472 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:57.472 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:57.472 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:57.472 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:57.472 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 57339 2024-08-17T17:20:57.472 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.472 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:57.472 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.472 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:57.472 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.499070+0000", 2024-08-17T17:20:57.472 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.472 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.473 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.473 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:57.473 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.499070+0000", 2024-08-17T17:20:57.473 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.473 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.473 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:57.473 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.499072+0000", 2024-08-17T17:20:57.473 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9860000000000001e-06 2024-08-17T17:20:57.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.473 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.473 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:57.473 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.499107+0000", 2024-08-17T17:20:57.473 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4891999999999997e-05 2024-08-17T17:20:57.474 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.474 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.474 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:57.474 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.499131+0000", 2024-08-17T17:20:57.474 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4409e-05 2024-08-17T17:20:57.474 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.474 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.474 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:57.474 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.499140+0000", 2024-08-17T17:20:57.474 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4999999999999999e-06 2024-08-17T17:20:57.474 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.474 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.474 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.475 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.499316+0000", 2024-08-17T17:20:57.475 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000175549 2024-08-17T17:20:57.475 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.475 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.475 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.475 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.499326+0000", 2024-08-17T17:20:57.475 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0834e-05 2024-08-17T17:20:57.475 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.475 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.475 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.475 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.445610+0000", 2024-08-17T17:20:57.475 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.94628382600000005 2024-08-17T17:20:57.475 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.475 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.475 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:57.476 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.445734+0000", 2024-08-17T17:20:57.476 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012335300000000001 2024-08-17T17:20:57.476 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.476 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.476 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:20:57.476 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.445884+0000", 2024-08-17T17:20:57.476 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015065900000000001 2024-08-17T17:20:57.476 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.476 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.476 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.476 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.453263+0000", 2024-08-17T17:20:57.476 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.007378356 2024-08-17T17:20:57.476 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.476 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.476 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:57.477 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.453482+0000", 2024-08-17T17:20:57.477 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00021963899999999999 2024-08-17T17:20:57.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.477 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.477 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.477 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.459021+0000", 2024-08-17T17:20:57.477 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0055392649999999998 2024-08-17T17:20:57.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.477 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.477 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:57.477 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.459055+0000", 2024-08-17T17:20:57.477 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3380000000000002e-05 2024-08-17T17:20:57.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.477 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.478 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:57.478 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.459095+0000", 2024-08-17T17:20:57.478 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9817999999999997e-05 2024-08-17T17:20:57.478 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.478 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:57.478 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.478 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.478 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:57364 5.1 5:88cf257a:::benchmark_data_smithi003_140793_object57363:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-08-17T17:20:57.478 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:46.521867+0000", 2024-08-17T17:20:57.478 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 70.818767046000005, 2024-08-17T17:20:57.478 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.94252200200000003, 2024-08-17T17:20:57.478 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:57.478 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:57.479 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:57.479 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:57.479 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:57.479 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:57.479 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 57364 2024-08-17T17:20:57.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.479 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:57.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.479 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:57.479 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.521867+0000", 2024-08-17T17:20:57.479 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.479 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:57.480 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.521867+0000", 2024-08-17T17:20:57.480 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.480 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.480 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:57.480 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.521868+0000", 2024-08-17T17:20:57.480 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.359e-06 2024-08-17T17:20:57.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.480 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.480 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:57.480 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.521897+0000", 2024-08-17T17:20:57.480 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8449000000000002e-05 2024-08-17T17:20:57.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.481 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:57.481 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.521910+0000", 2024-08-17T17:20:57.481 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3268000000000001e-05 2024-08-17T17:20:57.481 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.481 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:57.481 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.521917+0000", 2024-08-17T17:20:57.481 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4500000000000001e-06 2024-08-17T17:20:57.481 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.481 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.481 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.521994+0000", 2024-08-17T17:20:57.481 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7286000000000005e-05 2024-08-17T17:20:57.481 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.482 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.482 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.482 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.522016+0000", 2024-08-17T17:20:57.482 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1838999999999999e-05 2024-08-17T17:20:57.482 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.482 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.482 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.482 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.448024+0000", 2024-08-17T17:20:57.482 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.92600859899999999 2024-08-17T17:20:57.482 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.482 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.482 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:57.482 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.448158+0000", 2024-08-17T17:20:57.482 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013402999999999999 2024-08-17T17:20:57.483 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.483 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:20:57.483 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.448300+0000", 2024-08-17T17:20:57.483 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014166 2024-08-17T17:20:57.483 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.483 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.483 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.453544+0000", 2024-08-17T17:20:57.483 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0052435570000000003 2024-08-17T17:20:57.483 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.483 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:57.483 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.453556+0000", 2024-08-17T17:20:57.484 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2252000000000001e-05 2024-08-17T17:20:57.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.484 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.484 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.484 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.464279+0000", 2024-08-17T17:20:57.484 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010723534 2024-08-17T17:20:57.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.484 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.484 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:57.484 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.464343+0000", 2024-08-17T17:20:57.484 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3569000000000001e-05 2024-08-17T17:20:57.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.484 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.484 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:57.484 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.464389+0000", 2024-08-17T17:20:57.485 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6149999999999997e-05 2024-08-17T17:20:57.485 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.485 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:57.485 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.485 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.485 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.485 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:68138 5.1 5:8cd8b222:::benchmark_data_smithi003_140793_object68137:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-08-17T17:20:57.485 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:56.511667+0000", 2024-08-17T17:20:57.485 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 60.828966897999997, 2024-08-17T17:20:57.485 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2112100479999999, 2024-08-17T17:20:57.485 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:57.485 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:57.485 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:57.485 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:57.485 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:57.486 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:57.486 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68138 2024-08-17T17:20:57.486 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.486 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:57.486 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.486 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:57.486 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.511667+0000", 2024-08-17T17:20:57.486 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.486 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.486 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.486 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:57.486 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.511667+0000", 2024-08-17T17:20:57.486 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.486 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.487 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.487 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:57.487 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.511669+0000", 2024-08-17T17:20:57.487 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.61e-06 2024-08-17T17:20:57.487 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.487 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.487 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:57.487 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.511692+0000", 2024-08-17T17:20:57.487 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2768e-05 2024-08-17T17:20:57.487 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.487 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.487 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:57.487 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.511703+0000", 2024-08-17T17:20:57.487 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1060000000000001e-05 2024-08-17T17:20:57.487 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.488 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.488 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:57.488 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.511711+0000", 2024-08-17T17:20:57.488 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9470000000000003e-06 2024-08-17T17:20:57.488 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.488 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.488 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.488 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.512317+0000", 2024-08-17T17:20:57.488 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00060645199999999995 2024-08-17T17:20:57.488 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.488 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.488 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.488 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.512327+0000", 2024-08-17T17:20:57.488 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7159999999999998e-06 2024-08-17T17:20:57.489 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.489 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.489 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.489 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.497176+0000", 2024-08-17T17:20:57.489 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.98484927799999999 2024-08-17T17:20:57.489 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.489 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.489 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.489 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.497189+0000", 2024-08-17T17:20:57.489 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2729e-05 2024-08-17T17:20:57.489 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.489 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.489 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.489 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.711927+0000", 2024-08-17T17:20:57.490 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2147383940000001 2024-08-17T17:20:57.490 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.490 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.490 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:57.490 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.712104+0000", 2024-08-17T17:20:57.490 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017708700000000001 2024-08-17T17:20:57.490 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.490 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.490 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:20:57.490 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.712300+0000", 2024-08-17T17:20:57.490 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000195906 2024-08-17T17:20:57.490 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.490 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.490 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.491 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.719596+0000", 2024-08-17T17:20:57.491 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072962979999999997 2024-08-17T17:20:57.491 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.491 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.491 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.491 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.720426+0000", 2024-08-17T17:20:57.491 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00082983500000000001 2024-08-17T17:20:57.491 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.491 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.491 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:57.491 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.720670+0000", 2024-08-17T17:20:57.491 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000243856 2024-08-17T17:20:57.491 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.492 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.492 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:57.492 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.720724+0000", 2024-08-17T17:20:57.492 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4051000000000001e-05 2024-08-17T17:20:57.492 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.492 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.492 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:57.492 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.722877+0000", 2024-08-17T17:20:57.492 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002153061 2024-08-17T17:20:57.492 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.492 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:57.492 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.492 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.492 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.493 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:68143 5.1 5:8f270daa:::benchmark_data_smithi003_140793_object68142:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-08-17T17:20:57.493 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:56.519782+0000", 2024-08-17T17:20:57.493 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 60.820852170999999, 2024-08-17T17:20:57.493 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2052141999999999, 2024-08-17T17:20:57.493 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:20:57.493 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:20:57.493 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:20:57.493 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:20:57.493 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:20:57.493 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:20:57.493 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68143 2024-08-17T17:20:57.493 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.493 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:20:57.493 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.493 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:20:57.493 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519782+0000", 2024-08-17T17:20:57.494 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.494 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.494 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.494 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:20:57.494 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519782+0000", 2024-08-17T17:20:57.494 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:20:57.494 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.494 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.494 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:20:57.494 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519783+0000", 2024-08-17T17:20:57.494 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1310000000000001e-06 2024-08-17T17:20:57.494 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.494 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.494 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:20:57.495 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519802+0000", 2024-08-17T17:20:57.495 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9018999999999999e-05 2024-08-17T17:20:57.495 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.495 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.495 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:20:57.495 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519810+0000", 2024-08-17T17:20:57.495 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9370000000000004e-06 2024-08-17T17:20:57.495 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.495 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.495 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:20:57.495 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519877+0000", 2024-08-17T17:20:57.495 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7074000000000002e-05 2024-08-17T17:20:57.495 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.496 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.496 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.496 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519923+0000", 2024-08-17T17:20:57.496 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6057999999999998e-05 2024-08-17T17:20:57.496 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.496 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.496 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.496 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519930+0000", 2024-08-17T17:20:57.496 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5379999999999999e-06 2024-08-17T17:20:57.496 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.496 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.496 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.496 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.497221+0000", 2024-08-17T17:20:57.496 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.977291668 2024-08-17T17:20:57.496 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.497 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.497 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:20:57.497 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.497230+0000", 2024-08-17T17:20:57.497 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2570000000000007e-06 2024-08-17T17:20:57.497 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.497 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.497 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:20:57.497 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.716884+0000", 2024-08-17T17:20:57.497 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2196546719999999 2024-08-17T17:20:57.497 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.497 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.497 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:20:57.497 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.717065+0000", 2024-08-17T17:20:57.497 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000181163 2024-08-17T17:20:57.498 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.498 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.498 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:20:57.498 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.717263+0000", 2024-08-17T17:20:57.498 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019748699999999999 2024-08-17T17:20:57.498 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.498 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.498 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.498 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.720657+0000", 2024-08-17T17:20:57.498 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003393784 2024-08-17T17:20:57.498 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.498 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.498 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:20:57.499 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.722900+0000", 2024-08-17T17:20:57.499 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0022431959999999998 2024-08-17T17:20:57.499 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.499 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.499 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:20:57.499 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.722944+0000", 2024-08-17T17:20:57.499 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3656999999999999e-05 2024-08-17T17:20:57.499 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.499 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.499 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:20:57.499 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.722995+0000", 2024-08-17T17:20:57.499 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1264999999999998e-05 2024-08-17T17:20:57.499 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:20:57.499 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:20:57.499 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:20:57.500 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.724996+0000", 2024-08-17T17:20:57.500 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020012939999999998 2024-08-17T17:20:57.500 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.500 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:57.500 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.500 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:20:57.500 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:20:57.500 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:20:57.502 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:57.529 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:57.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:57.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:57.631 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:57.681 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:57.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:57.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:57.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:57 smithi053 ceph-mon[89963]: pgmap v305: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 531 op/s; 27690/284832 objects degraded (9.722%); 25150/284832 objects misplaced (8.830%); 12 MiB/s, 198 objects/s recovering 2024-08-17T17:20:57.783 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:57.811 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:57.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:57.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:57.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:20:57.913 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:57.942 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:57.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:57.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:58.026 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:20:58.043 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:58.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:58.134 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:58.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:58.235 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:20:58.270 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:58.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:58.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:20:58.372 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:58.402 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:58.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:58.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:58.497 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-17T17:20:58.504 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:58.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:58.579 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:58.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:58.611 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:20:58.653 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:20:58.679 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:20:58.718 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:58.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:58.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:20:58.820 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:58.833 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:20:58.833 INFO:teuthology.orchestra.run.smithi053.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-17T17:20:58.833 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:20:58.851 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:58.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:58.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:58.953 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:20:58.998 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:58.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:58.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:20:59.100 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:59.134 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:59.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:59.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:59.236 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:59.270 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:59.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:59.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:20:59.372 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:20:59.411 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:20:59.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:59.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:20:59.512 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:59.542 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:59.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:59.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:59.644 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:20:59.660 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:20:59 smithi053 ceph-mon[89963]: pgmap v306: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 486 op/s; 27690/283518 objects degraded (9.767%); 25150/283518 objects misplaced (8.871%); 10 MiB/s, 161 objects/s recovering 2024-08-17T17:20:59.672 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:59.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:59.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:20:59.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:20:59 smithi003 ceph-mon[100615]: pgmap v306: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 486 op/s; 27690/283518 objects degraded (9.767%); 25150/283518 objects misplaced (8.871%); 10 MiB/s, 161 objects/s recovering 2024-08-17T17:20:59.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:20:59 smithi003 ceph-mon[103344]: pgmap v306: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 486 op/s; 27690/283518 objects degraded (9.767%); 25150/283518 objects misplaced (8.871%); 10 MiB/s, 161 objects/s recovering 2024-08-17T17:20:59.774 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:20:59.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:59.806 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:59.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:20:59.907 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:20:59.944 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:20:59.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:20:59.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:00.046 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:00.088 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:00.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:00.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:00.190 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:00.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:00.238 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:00.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:00.339 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:00.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:00.382 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:00.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:00.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:00 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 27690/283518 objects degraded (9.767%), 11 pgs degraded (PG_DEGRADED) 2024-08-17T17:21:00.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:21:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:21:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:21:00.482 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:00.514 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:00.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:00.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:00.615 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:00.639 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:00.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:00.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:00.741 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:00.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:00 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 27690/283518 objects degraded (9.767%), 11 pgs degraded (PG_DEGRADED) 2024-08-17T17:21:00.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:00 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 27690/283518 objects degraded (9.767%), 11 pgs degraded (PG_DEGRADED) 2024-08-17T17:21:00.778 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:00.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:00.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:00.880 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:00.905 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:00.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:00.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:01.007 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:01.045 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:01.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:01.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:01.147 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:01.180 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:01.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:01.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:01.281 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:01.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:01.310 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:01.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:01.411 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:01.453 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:01.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:01.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:01.555 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:01.585 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:01.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:01.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:01.613 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:01 smithi053 ceph-mon[89963]: pgmap v307: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.7 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 548 op/s; 27492/278259 objects degraded (9.880%); 24822/278259 objects misplaced (8.920%); 9.9 MiB/s, 159 objects/s recovering 2024-08-17T17:21:01.687 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:01.727 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:01.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:01.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:01.728 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:21:01.760 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:01 smithi003 ceph-mon[100615]: pgmap v307: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.7 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 548 op/s; 27492/278259 objects degraded (9.880%); 24822/278259 objects misplaced (8.920%); 9.9 MiB/s, 159 objects/s recovering 2024-08-17T17:21:01.765 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:01 smithi003 ceph-mon[103344]: pgmap v307: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.7 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 548 op/s; 27492/278259 objects degraded (9.880%); 24822/278259 objects misplaced (8.920%); 9.9 MiB/s, 159 objects/s recovering 2024-08-17T17:21:01.829 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:01.859 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:01.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:01.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:01.962 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:02.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:02.002 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:02.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:02.103 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:02.144 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:02.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:02.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:02.246 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:02.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:02.286 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:02.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:02.387 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:02.416 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:02.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:02.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:02.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:02.545 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:02.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:02.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:02.648 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:02.689 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:02.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:02.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:02.791 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:02.829 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:02.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:02.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:02.931 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:02.964 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:02.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:02.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:03.066 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:03.101 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:03.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:03.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:03.203 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:03.238 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:03.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:03.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:03.340 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:03.375 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:03.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:03.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:03.420 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:03 smithi053 ceph-mon[89963]: pgmap v308: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 611 op/s; 27308/274365 objects degraded (9.953%); 24154/274365 objects misplaced (8.804%); 9.8 MiB/s, 157 objects/s recovering 2024-08-17T17:21:03.476 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:03.512 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:03.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:03.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:03.569 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:21:03.569 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2024-08-17T17:21:03.569 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:03.569 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.35277.0:115584 5.0 e143/141 5:04a20f0a:::benchmark_data_smithi003_140793_object13527:head v 143'7208, mlcod=143'7206)", 2024-08-17T17:21:03.570 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:21:03.564750+0000", 2024-08-17T17:21:03.570 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.0037867880000000001, 2024-08-17T17:21:03.570 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003822195, 2024-08-17T17:21:03.570 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:21:03.570 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2024-08-17T17:21:03.570 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:21:03.570 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:03.570 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:21:03.570 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:03.564750+0000", 2024-08-17T17:21:03.570 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:21:03.570 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:03.570 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:03.570 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:21:03.570 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:03.564750+0000", 2024-08-17T17:21:03.571 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:21:03.571 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:03.571 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:03.571 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:21:03.571 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:03.564750+0000", 2024-08-17T17:21:03.571 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:21:03.571 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:03.571 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:03.571 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:21:03.571 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:03.564756+0000", 2024-08-17T17:21:03.571 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.6719999999999998e-06 2024-08-17T17:21:03.571 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:03.571 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:03.571 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:21:03.571 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:03.564771+0000", 2024-08-17T17:21:03.572 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4324999999999999e-05 2024-08-17T17:21:03.572 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:03.572 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:03.572 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:21:03.572 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:03.564776+0000", 2024-08-17T17:21:03.572 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3179999999999997e-06 2024-08-17T17:21:03.572 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:03.572 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:03.572 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:21:03.572 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:03.564837+0000", 2024-08-17T17:21:03.572 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0924999999999997e-05 2024-08-17T17:21:03.572 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:03.572 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:03.572 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:21:03.572 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:03.564865+0000", 2024-08-17T17:21:03.573 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8413000000000001e-05 2024-08-17T17:21:03.573 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:21:03.573 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:21:03.573 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:21:03.573 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:21:03.573 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-08-17T17:21:03.573 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 1 2024-08-17T17:21:03.573 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:21:03.613 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:03.639 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:03.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:03.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:03.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:03 smithi003 ceph-mon[100615]: pgmap v308: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 611 op/s; 27308/274365 objects degraded (9.953%); 24154/274365 objects misplaced (8.804%); 9.8 MiB/s, 157 objects/s recovering 2024-08-17T17:21:03.688 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:03 smithi003 ceph-mon[103344]: pgmap v308: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 611 op/s; 27308/274365 objects degraded (9.953%); 24154/274365 objects misplaced (8.804%); 9.8 MiB/s, 157 objects/s recovering 2024-08-17T17:21:03.742 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:03.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:03.783 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:03.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:03.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:03.932 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:03.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:03.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:04.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:04.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:04.103 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:04.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:04.204 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:04.240 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:04.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:04.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:04.301 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:21:04.301 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:21:04.342 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:04.352 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:21:04.352 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:21:04.385 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:04.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:04.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:04.395 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:21:04.395 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:21:04.399 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-08-17T17:21:04.439 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:21:04.439 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:21:04.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:04 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1470791654' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:21:04.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:04 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:21:04.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:04 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/4113331202' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:21:04.474 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:21:04.474 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:21:04.487 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:04.504 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-17T17:21:04.523 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:21:04.523 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:21:04.533 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:04.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:04.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:04.568 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:21:04.568 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:21:04.568 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:04 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1470791654' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:21:04.568 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:04 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:21:04.569 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:04 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/4113331202' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:21:04.570 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:04 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1470791654' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:21:04.570 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:04 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:21:04.570 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:04 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/4113331202' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:21:04.602 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:21:04.602 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:21:04.636 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:04.669 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:04.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:04.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:04.771 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:04.815 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:04.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:04.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:04.917 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:04.918 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:21:04.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:04.960 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:04.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:05.061 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:05.106 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:05.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:05.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:05.208 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:05.250 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:05.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:05.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:05.352 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:05.401 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:05.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:05.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:05.503 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:05.526 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:05.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:05.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:05.627 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:05.655 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:05 smithi003 ceph-mon[100615]: pgmap v309: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 711 op/s; 27333/269499 objects degraded (10.142%); 23758/269499 objects misplaced (8.816%); 12 MiB/s, 185 objects/s recovering 2024-08-17T17:21:05.655 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:05 smithi003 ceph-mon[100615]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:21:05.655 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:05 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:21:05.655 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:05 smithi003 ceph-mon[100615]: osdmap e144: 8 total, 8 up, 7 in 2024-08-17T17:21:05.655 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:05 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 27333/269499 objects degraded (10.142%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:05.655 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:05 smithi003 ceph-mon[103344]: pgmap v309: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 711 op/s; 27333/269499 objects degraded (10.142%); 23758/269499 objects misplaced (8.816%); 12 MiB/s, 185 objects/s recovering 2024-08-17T17:21:05.655 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:05 smithi003 ceph-mon[103344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:21:05.655 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:05 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:21:05.655 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:05 smithi003 ceph-mon[103344]: osdmap e144: 8 total, 8 up, 7 in 2024-08-17T17:21:05.656 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:05 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 27333/269499 objects degraded (10.142%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:05.656 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:21:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:21:05] "GET /metrics HTTP/1.1" 200 36604 "" "Prometheus/2.51.0" 2024-08-17T17:21:05.662 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:05.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:05.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:05.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:05 smithi053 ceph-mon[89963]: pgmap v309: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 711 op/s; 27333/269499 objects degraded (10.142%); 23758/269499 objects misplaced (8.816%); 12 MiB/s, 185 objects/s recovering 2024-08-17T17:21:05.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:05 smithi053 ceph-mon[89963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:21:05.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:05 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:21:05.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:05 smithi053 ceph-mon[89963]: osdmap e144: 8 total, 8 up, 7 in 2024-08-17T17:21:05.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:05 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 27333/269499 objects degraded (10.142%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:05.764 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:05.818 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:05.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:05.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:05.922 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:05.966 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:05.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:05.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:06.068 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:06.110 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:06.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:06.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:06.212 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:06.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:06.263 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:06.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:06.364 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:06.401 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:06.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:06.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:06.502 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:06.542 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:06.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:06.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:06.644 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:06.657 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-17T17:21:06.657 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:21:06.657 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-08-17T17:21:06.657 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-17T17:21:06.657 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-17T17:21:06.657 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-08-17T17:21:06.680 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:06.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:06.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:06.781 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:06.815 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:06.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:06.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:06.917 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:06.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:06.973 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:06.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:07.074 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:07.080 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:21:07.080 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:21:07.109 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:07.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:07.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:07.130 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:21:07.211 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:07.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:07.245 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:07.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:07.346 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:07.376 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:07.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:07.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:07.478 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:07.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:07 smithi003 ceph-mon[100615]: pgmap v311: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 665 op/s; 26942/264855 objects degraded (10.172%); 23195/264855 objects misplaced (8.758%); 9.2 MiB/s, 147 objects/s recovering 2024-08-17T17:21:07.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:07 smithi003 ceph-mon[103344]: pgmap v311: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 665 op/s; 26942/264855 objects degraded (10.172%); 23195/264855 objects misplaced (8.758%); 9.2 MiB/s, 147 objects/s recovering 2024-08-17T17:21:07.528 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:07.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:07.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:07.628 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:21:07.630 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:07.690 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:07.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:07.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:07.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:07 smithi053 ceph-mon[89963]: pgmap v311: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 665 op/s; 26942/264855 objects degraded (10.172%); 23195/264855 objects misplaced (8.758%); 9.2 MiB/s, 147 objects/s recovering 2024-08-17T17:21:07.789 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:21:07.792 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:07.823 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-17T17:21:07.841 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:07.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:07.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:07.942 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:07.981 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:07.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:07.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:08.077 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:21:08.083 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:08.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:08.153 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:08.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:08.254 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:08.289 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:08.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:08.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:08.392 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:08.422 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:08.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:08.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:08.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:08.569 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:08.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:08.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:08.670 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:08.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:08.705 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:08.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:08.806 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:08.835 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:08.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:08.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:08.937 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:08.960 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:21:08.967 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:08.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:08.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:09.068 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:09.100 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:21:09.112 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:09.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:09.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:09.214 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:09.263 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:09.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:09.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:09.283 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:09 smithi053 ceph-mon[89963]: pgmap v312: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 659 op/s; 26942/263739 objects degraded (10.215%); 23195/263739 objects misplaced (8.795%); 9.2 MiB/s, 147 objects/s recovering 2024-08-17T17:21:09.283 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:21:09.283 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:21:09.365 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:09.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:09.402 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:09.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:09.411 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:09 smithi003 ceph-mon[103344]: pgmap v312: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 659 op/s; 26942/263739 objects degraded (10.215%); 23195/263739 objects misplaced (8.795%); 9.2 MiB/s, 147 objects/s recovering 2024-08-17T17:21:09.411 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:21:09.411 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:21:09.411 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:09 smithi003 ceph-mon[100615]: pgmap v312: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 659 op/s; 26942/263739 objects degraded (10.215%); 23195/263739 objects misplaced (8.795%); 9.2 MiB/s, 147 objects/s recovering 2024-08-17T17:21:09.412 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:21:09.412 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:21:09.503 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:09.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:09.541 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:09.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:09.642 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:09.673 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:09.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:09.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:09.775 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:09.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:09.814 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:09.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:09.908 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:21:09.909 INFO:teuthology.orchestra.run.smithi053.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-17T17:21:09.909 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:21:09.915 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:09.950 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:09.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:09.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:10.052 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:10.086 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:10.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:10.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:10.188 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:10.213 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:10 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 26942/263739 objects degraded (10.215%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:10.214 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:21:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=cleanup t=2024-08-17T17:21:10.095652629Z level=info msg="Completed cleanup jobs" duration=12.832508ms 2024-08-17T17:21:10.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:10.230 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:10.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:10.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:10 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 26942/263739 objects degraded (10.215%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:10.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:10 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 26942/263739 objects degraded (10.215%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:10.332 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:10.368 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:10.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:10.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:10.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:21:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:21:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:21:10.470 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:10.511 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:10.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:10.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:10.613 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:10.644 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:10.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:10.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:10.746 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:10.790 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:10.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:10.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:10.892 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:10.927 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:10.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:10.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:11.030 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:11.075 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:11.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:11.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:11.119 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-08-17T17:21:11.177 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:11.209 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:11.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:11.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:11.310 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:11.344 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:11.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:11.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:11.446 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:11.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:11 smithi053 ceph-mon[89963]: pgmap v313: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 647 op/s; 26782/258837 objects degraded (10.347%); 22804/258837 objects misplaced (8.810%); 9.4 MiB/s, 149 objects/s recovering 2024-08-17T17:21:11.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:11 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2088737713' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:21:11.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:11 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:21:11.474 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:11.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:11.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:11.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:11 smithi003 ceph-mon[103344]: pgmap v313: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 647 op/s; 26782/258837 objects degraded (10.347%); 22804/258837 objects misplaced (8.810%); 9.4 MiB/s, 149 objects/s recovering 2024-08-17T17:21:11.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:11 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2088737713' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:21:11.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:11 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:21:11.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:11 smithi003 ceph-mon[100615]: pgmap v313: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.3 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 647 op/s; 26782/258837 objects degraded (10.347%); 22804/258837 objects misplaced (8.810%); 9.4 MiB/s, 149 objects/s recovering 2024-08-17T17:21:11.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:11 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2088737713' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:21:11.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:11 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:21:11.576 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:11.605 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:11.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:11.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:11.706 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:11.744 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:11.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:11.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:11.845 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:11.886 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:11.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:11.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:11.988 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:12.021 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:12.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:12.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:12.123 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:12.162 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:12.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:12.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:12.263 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:12.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:12 smithi003 ceph-mon[103344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:21:12.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:12 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:21:12.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:12 smithi003 ceph-mon[103344]: osdmap e145: 8 total, 8 up, 7 in 2024-08-17T17:21:12.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:12 smithi003 ceph-mon[100615]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:21:12.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:12 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:21:12.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:12 smithi003 ceph-mon[100615]: osdmap e145: 8 total, 8 up, 7 in 2024-08-17T17:21:12.298 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:12.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:12.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:12.400 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:12.441 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:12.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:12.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:12.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:12 smithi053 ceph-mon[89963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:21:12.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:12 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:21:12.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:12 smithi053 ceph-mon[89963]: osdmap e145: 8 total, 8 up, 7 in 2024-08-17T17:21:12.543 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:12.592 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:12.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:12.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:12.693 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:12.733 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:12.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:12.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:12.835 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:12.872 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:12.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:12.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:12.974 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:13.006 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:13.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:13.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:13.018 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:21:13.018 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:21:13.108 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:13.154 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:21:13.154 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:21:13.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:13.158 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:13.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:13.199 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:21:13.200 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:21:13.228 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:13 smithi003 ceph-mon[103344]: pgmap v315: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.2 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 600 op/s; 26581/255072 objects degraded (10.421%); 22179/255072 objects misplaced (8.695%); 8.9 MiB/s, 142 objects/s recovering 2024-08-17T17:21:13.228 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:13 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/593062679' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:21:13.228 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:13 smithi003 ceph-mon[100615]: pgmap v315: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.2 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 600 op/s; 26581/255072 objects degraded (10.421%); 22179/255072 objects misplaced (8.695%); 8.9 MiB/s, 142 objects/s recovering 2024-08-17T17:21:13.229 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:13 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/593062679' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:21:13.242 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:21:13.243 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:21:13.259 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:13.280 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:21:13.281 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:21:13.303 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:13.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:13.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:13.313 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:21:13.313 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:21:13.346 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:21:13.347 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:21:13.377 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:21:13.377 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:21:13.406 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:13.414 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:21:13.414 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:21:13.446 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:13.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:13.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:13.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:13 smithi053 ceph-mon[89963]: pgmap v315: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.2 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 600 op/s; 26581/255072 objects degraded (10.421%); 22179/255072 objects misplaced (8.695%); 8.9 MiB/s, 142 objects/s recovering 2024-08-17T17:21:13.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:13 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/593062679' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:21:13.547 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:13.586 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:13.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:13.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:13.688 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:13.716 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-17T17:21:13.733 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:13.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:13.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:13.834 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:13.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:13.876 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:13.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:13.978 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:14.020 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:14.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:14.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:14.121 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:14.164 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:14.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:14.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:14.167 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:21:14.266 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:14.305 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:14.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:14.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:14.407 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:14.449 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:14.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:14.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:14.551 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:14.586 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:14.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:14.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:14.688 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:14.723 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:14.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:14.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:14.825 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:14.866 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:14.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:14.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:14.968 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:15.000 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:15.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:15.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:15.102 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:15.132 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:15.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:15.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:15.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:15.269 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:15.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:15.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:15.371 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:15.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:15.397 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:15.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:15.440 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:15 smithi053 ceph-mon[89963]: pgmap v316: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 660 op/s; 26666/249762 objects degraded (10.677%); 21753/249762 objects misplaced (8.709%); 8.5 MiB/s, 136 objects/s recovering 2024-08-17T17:21:15.441 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:15 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 26666/249762 objects degraded (10.677%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:15.498 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:15.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:15 smithi003 ceph-mon[103344]: pgmap v316: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 660 op/s; 26666/249762 objects degraded (10.677%); 21753/249762 objects misplaced (8.709%); 8.5 MiB/s, 136 objects/s recovering 2024-08-17T17:21:15.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:15 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 26666/249762 objects degraded (10.677%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:15.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:15 smithi003 ceph-mon[100615]: pgmap v316: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 660 op/s; 26666/249762 objects degraded (10.677%); 21753/249762 objects misplaced (8.709%); 8.5 MiB/s, 136 objects/s recovering 2024-08-17T17:21:15.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:15 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 26666/249762 objects degraded (10.677%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:15.534 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:15.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:15.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:15.636 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:15.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:15.675 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:15.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:15.776 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:15.813 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:15.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:15.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:15.913 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:21:15.915 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:15.942 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:15.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:15.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:15.955 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:21:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:21:15] "GET /metrics HTTP/1.1" 200 36602 "" "Prometheus/2.51.0" 2024-08-17T17:21:16.045 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:16.093 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:21:16.093 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2024-08-17T17:21:16.093 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:16.093 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.35277.0:123785 5.2 e145/111 5:4d30252f:::benchmark_data_smithi003_140793_object21728:head v 145'7954, mlcod=145'7952)", 2024-08-17T17:21:16.094 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:21:16.052552+0000", 2024-08-17T17:21:16.094 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.0094345309999999995, 2024-08-17T17:21:16.094 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0094551270000000007, 2024-08-17T17:21:16.094 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:21:16.094 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:21:16.094 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:21:16.094 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:16.094 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:21:16.094 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.052552+0000", 2024-08-17T17:21:16.094 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:21:16.094 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:16.094 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:16.095 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:21:16.095 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.052552+0000", 2024-08-17T17:21:16.095 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:21:16.095 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:16.095 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:16.095 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:21:16.095 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.052551+0000", 2024-08-17T17:21:16.095 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:21:16.095 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:16.095 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:16.095 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:21:16.095 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.052559+0000", 2024-08-17T17:21:16.095 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1769999999999998e-06 2024-08-17T17:21:16.095 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:16.096 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:16.096 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:21:16.096 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.052573+0000", 2024-08-17T17:21:16.096 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4100999999999999e-05 2024-08-17T17:21:16.096 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:16.096 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:16.096 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:21:16.096 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.052579+0000", 2024-08-17T17:21:16.096 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0159999999999997e-06 2024-08-17T17:21:16.096 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:16.096 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:16.096 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:21:16.096 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.052624+0000", 2024-08-17T17:21:16.096 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5385e-05 2024-08-17T17:21:16.096 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:16.096 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:16.097 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:21:16.097 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.052646+0000", 2024-08-17T17:21:16.097 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1639000000000001e-05 2024-08-17T17:21:16.097 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:16.097 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:16.097 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:21:16.097 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.061967+0000", 2024-08-17T17:21:16.097 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0093210129999999995 2024-08-17T17:21:16.097 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:21:16.097 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:21:16.097 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:21:16.097 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:16.097 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:16.097 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.35277.0:123787 5.2 e145/111 5:4ae50f80:::benchmark_data_smithi003_140793_object21730:head v 145'7955, mlcod=145'7952)", 2024-08-17T17:21:16.097 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:21:16.058484+0000", 2024-08-17T17:21:16.097 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.0035024330000000001, 2024-08-17T17:21:16.098 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0035676430000000001, 2024-08-17T17:21:16.098 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:21:16.098 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2024-08-17T17:21:16.098 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:21:16.098 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:16.098 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:21:16.098 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.058484+0000", 2024-08-17T17:21:16.098 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:21:16.098 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:16.098 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:16.098 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:21:16.098 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.058484+0000", 2024-08-17T17:21:16.098 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:21:16.098 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:16.098 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:16.099 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:21:16.099 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.058483+0000", 2024-08-17T17:21:16.099 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:21:16.099 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:16.099 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:16.099 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:21:16.099 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.058491+0000", 2024-08-17T17:21:16.099 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.2239999999999995e-06 2024-08-17T17:21:16.099 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:16.099 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:16.099 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:21:16.099 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.058504+0000", 2024-08-17T17:21:16.099 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2716e-05 2024-08-17T17:21:16.100 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:16.100 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:16.100 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:21:16.100 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.058509+0000", 2024-08-17T17:21:16.100 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9030000000000003e-06 2024-08-17T17:21:16.100 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:16.100 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:16.100 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:21:16.100 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.058828+0000", 2024-08-17T17:21:16.100 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00031970099999999998 2024-08-17T17:21:16.100 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:16.100 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:16.100 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:21:16.100 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.058854+0000", 2024-08-17T17:21:16.101 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5295000000000001e-05 2024-08-17T17:21:16.101 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:21:16.101 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:21:16.101 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:21:16.101 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:16.101 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:16.101 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:123789 5.e 5.3674db8e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-08-17T17:21:16.101 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:21:16.061990+0000", 2024-08-17T17:21:16.101 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4294967295.9999957, 2024-08-17T17:21:16.101 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5217999999999995e-05, 2024-08-17T17:21:16.101 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:21:16.101 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2024-08-17T17:21:16.101 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:21:16.101 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:21:16.101 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:21:16.102 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 123789 2024-08-17T17:21:16.102 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:16.102 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:21:16.102 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:16.102 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:21:16.102 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.061990+0000", 2024-08-17T17:21:16.102 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:21:16.102 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:16.102 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:16.102 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:21:16.102 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.061990+0000", 2024-08-17T17:21:16.102 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:21:16.102 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:16.102 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:16.103 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:21:16.103 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.061989+0000", 2024-08-17T17:21:16.103 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:21:16.103 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:16.103 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:16.103 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:21:16.103 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.061995+0000", 2024-08-17T17:21:16.103 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3600000000000001e-06 2024-08-17T17:21:16.103 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:16.103 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:16.103 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:21:16.103 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.062006+0000", 2024-08-17T17:21:16.103 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0981999999999999e-05 2024-08-17T17:21:16.103 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:16.103 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:16.104 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:21:16.104 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.062012+0000", 2024-08-17T17:21:16.104 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8089999999999998e-06 2024-08-17T17:21:16.104 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:16.104 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:16.104 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:21:16.104 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.062063+0000", 2024-08-17T17:21:16.104 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0983000000000003e-05 2024-08-17T17:21:16.104 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:21:16.104 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:21:16.104 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:21:16.104 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:21:16.104 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-08-17T17:21:16.104 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 3 2024-08-17T17:21:16.105 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:21:16.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:16.105 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:16.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:16.206 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:16.235 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:16.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:16.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:16.337 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:16.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:16.389 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:16.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:16.418 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:21:16.480 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:21:16.490 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:16.582 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:16.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:16.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:16.684 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:16.715 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:16.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:16.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:16.726 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-17T17:21:16.726 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:21:16.727 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-08-17T17:21:16.727 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:21:16.817 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:16.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:16.878 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:16.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:16.958 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-17T17:21:16.979 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:17.032 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:17.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:17.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:17.133 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:17.173 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:17.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:17.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:17.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:17 smithi053 ceph-mon[89963]: pgmap v317: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 668 op/s; 26247/244788 objects degraded (10.722%); 21108/244788 objects misplaced (8.623%); 9.4 MiB/s, 151 objects/s recovering 2024-08-17T17:21:17.275 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:17.311 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:17.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:17.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:17.361 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:21:17.413 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:17.448 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:17.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:17.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:17.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:17 smithi003 ceph-mon[100615]: pgmap v317: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 668 op/s; 26247/244788 objects degraded (10.722%); 21108/244788 objects misplaced (8.623%); 9.4 MiB/s, 151 objects/s recovering 2024-08-17T17:21:17.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:17 smithi003 ceph-mon[103344]: pgmap v317: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 668 op/s; 26247/244788 objects degraded (10.722%); 21108/244788 objects misplaced (8.623%); 9.4 MiB/s, 151 objects/s recovering 2024-08-17T17:21:17.551 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:17.587 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:17.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:17.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:17.689 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:17.723 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:17.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:17.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:17.825 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:17.862 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:17.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:17.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:17.963 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:18.002 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:18.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:18.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:18.104 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:18.144 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:18.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:18.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:18.246 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:18.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:18.278 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:18.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:18.379 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:18.408 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:18.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:18.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:18.510 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:18.522 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:21:18.563 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:18.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:18.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:18.664 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:18.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:18.716 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:18.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:18.817 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:18.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:18.864 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:18.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:18.965 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:18.999 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:19.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:19.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:19.101 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:19.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:19.145 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:19.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:19.207 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:21:19.208 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2024-08-17T17:21:19.208 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2024-08-17T17:21:19.208 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2024-08-17T17:21:19.208 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:21:19.230 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:19 smithi003 ceph-mon[103344]: pgmap v318: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 672 op/s; 26247/243552 objects degraded (10.777%); 21108/243552 objects misplaced (8.667%); 9.4 MiB/s, 151 objects/s recovering 2024-08-17T17:21:19.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:19 smithi003 ceph-mon[100615]: pgmap v318: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 672 op/s; 26247/243552 objects degraded (10.777%); 21108/243552 objects misplaced (8.667%); 9.4 MiB/s, 151 objects/s recovering 2024-08-17T17:21:19.246 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:19.284 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:19.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:19.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:19.386 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:19.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:19.414 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:19.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:19.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:19 smithi053 ceph-mon[89963]: pgmap v318: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 672 op/s; 26247/243552 objects degraded (10.777%); 21108/243552 objects misplaced (8.667%); 9.4 MiB/s, 151 objects/s recovering 2024-08-17T17:21:19.516 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:19.549 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:19.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:19.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:19.651 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:19.692 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:19.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:19.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:19.794 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:19.823 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:19.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:19.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:19.925 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:19.968 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:19.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:19.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:20.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:20.106 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:20.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:20.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:20.169 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-17T17:21:20.187 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-08-17T17:21:20.208 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:20.244 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:20.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:20.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:20.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:20 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 26247/243552 objects degraded (10.777%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:20.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:20 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1263136374' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:21:20.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:20 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:21:20.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:20 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 26247/243552 objects degraded (10.777%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:20.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:20 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1263136374' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:21:20.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:20 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:21:20.346 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:20.403 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:20.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:20.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:20.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:20 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 26247/243552 objects degraded (10.777%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:20.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:20 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1263136374' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:21:20.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:20 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:21:20.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:21:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:21:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:21:20.504 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:20.525 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:21:20.539 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:20.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:20.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:20.588 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:21:20.641 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:20.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:20.687 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:20.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:20.788 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:20.823 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:20.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:20.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:20.925 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:20.962 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:20.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:20.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:21.065 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:21.099 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:21.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:21.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:21.201 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:21.221 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:21 smithi003 ceph-mon[100615]: pgmap v319: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 667 op/s; 26086/238794 objects degraded (10.924%); 20697/238794 objects misplaced (8.667%); 9.6 MiB/s, 153 objects/s recovering 2024-08-17T17:21:21.221 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:21 smithi003 ceph-mon[100615]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:21:21.221 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:21 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:21:21.221 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:21 smithi003 ceph-mon[100615]: osdmap e146: 8 total, 8 up, 7 in 2024-08-17T17:21:21.222 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:21 smithi003 ceph-mon[103344]: pgmap v319: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 667 op/s; 26086/238794 objects degraded (10.924%); 20697/238794 objects misplaced (8.667%); 9.6 MiB/s, 153 objects/s recovering 2024-08-17T17:21:21.222 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:21 smithi003 ceph-mon[103344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:21:21.222 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:21 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:21:21.222 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:21 smithi003 ceph-mon[103344]: osdmap e146: 8 total, 8 up, 7 in 2024-08-17T17:21:21.235 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:21.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:21.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:21.336 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:21.380 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:21.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:21.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:21.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:21 smithi053 ceph-mon[89963]: pgmap v319: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 667 op/s; 26086/238794 objects degraded (10.924%); 20697/238794 objects misplaced (8.667%); 9.6 MiB/s, 153 objects/s recovering 2024-08-17T17:21:21.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:21 smithi053 ceph-mon[89963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:21:21.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:21 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:21:21.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:21 smithi053 ceph-mon[89963]: osdmap e146: 8 total, 8 up, 7 in 2024-08-17T17:21:21.482 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:21.517 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:21.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:21.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:21.619 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:21.655 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:21:21.655 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:21:21.659 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:21.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:21.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:21.689 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:21:21.689 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:21:21.733 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:21:21.733 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:21:21.761 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:21.769 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:21:21.769 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:21:21.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:21.797 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:21.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:21.800 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:21:21.801 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:21:21.840 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:21:21.840 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:21:21.873 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:21:21.873 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:21:21.897 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:21.908 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:21:21.908 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:21:21.947 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:21.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:21.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:22.049 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:22.087 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:22.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:22.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:22.188 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:22.219 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:22.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:22.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:22.321 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:22.361 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:22.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:22.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:22.462 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:22.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:22 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/462759951' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:21:22.505 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:22.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:22.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:22.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:22 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/462759951' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:21:22.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:22 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/462759951' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:21:22.606 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:22.645 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:22.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:22.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:22.656 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:21:22.656 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:21:22.746 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:22.786 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:22.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:22.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:22.888 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:22.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:22.928 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:22.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:23.014 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:21:23.029 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:23.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:23.079 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:23.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:23.179 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:23.213 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:23.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:23.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:23.315 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:23.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:23.378 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:23.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:23.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:23 smithi053 ceph-mon[89963]: pgmap v321: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.8 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 662 op/s; 25895/235203 objects degraded (11.010%); 20060/235203 objects misplaced (8.529%); 9.5 MiB/s, 152 objects/s recovering 2024-08-17T17:21:23.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:21:23.479 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:23.515 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:23.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:23.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:23.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:23 smithi003 ceph-mon[103344]: pgmap v321: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.8 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 662 op/s; 25895/235203 objects degraded (11.010%); 20060/235203 objects misplaced (8.529%); 9.5 MiB/s, 152 objects/s recovering 2024-08-17T17:21:23.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:21:23.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:23 smithi003 ceph-mon[100615]: pgmap v321: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.8 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 662 op/s; 25895/235203 objects degraded (11.010%); 20060/235203 objects misplaced (8.529%); 9.5 MiB/s, 152 objects/s recovering 2024-08-17T17:21:23.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:21:23.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:23.655 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:23.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:23.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:23.757 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:23.789 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:23.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:23.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:23.891 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:23.929 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:23.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:23.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:24.031 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:24.065 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:24.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:24.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:24.167 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:24.202 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:24.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:24.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:24.304 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:24.334 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:24.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:24.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:24.436 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:24.471 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:24.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:24.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:24.572 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:24.598 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:24.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:24.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:24.700 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:24.733 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:24.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:24.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:24.834 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:24.860 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:24.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:24.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:24.962 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:24.998 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:24.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:24.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:25.099 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:25.139 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:25.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:25.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:25.241 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:25.276 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:25.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:25.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:25.378 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:25.411 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:25.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:25.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:25.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:25 smithi053 ceph-mon[89963]: pgmap v322: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 635 op/s; 25947/230694 objects degraded (11.247%); 19842/230694 objects misplaced (8.601%); 8.4 MiB/s, 134 objects/s recovering 2024-08-17T17:21:25.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:25 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 25947/230694 objects degraded (11.247%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:25.513 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:25.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:25 smithi003 ceph-mon[103344]: pgmap v322: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 635 op/s; 25947/230694 objects degraded (11.247%); 19842/230694 objects misplaced (8.601%); 8.4 MiB/s, 134 objects/s recovering 2024-08-17T17:21:25.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:25 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 25947/230694 objects degraded (11.247%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:25.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:25 smithi003 ceph-mon[100615]: pgmap v322: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 635 op/s; 25947/230694 objects degraded (11.247%); 19842/230694 objects misplaced (8.601%); 8.4 MiB/s, 134 objects/s recovering 2024-08-17T17:21:25.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:25 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 25947/230694 objects degraded (11.247%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:25.547 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:25.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:25.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:25.649 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:25.678 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:25.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:25.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:25.780 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:25.812 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:25.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:25.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:25.914 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:25.952 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:25.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:25.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:26.002 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:21:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:21:25] "GET /metrics HTTP/1.1" 200 36604 "" "Prometheus/2.51.0" 2024-08-17T17:21:26.054 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:26.086 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:26.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:26.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:26.188 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:26.220 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:26.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:26.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:26.321 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:26.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:26.351 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:26.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:26.451 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:26.480 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:26.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:26.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:26.582 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:26.613 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:26.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:26.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:26.714 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:26.751 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:26.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:26.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:26.853 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:26.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:26.891 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:26.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:26.992 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:27.032 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:27.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:27.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:27.134 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:27.170 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:27.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:27.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:27.174 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-17T17:21:27.272 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:27.309 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:27.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:27.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:27.411 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:27.463 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:27.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:27.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:27.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:27 smithi053 ceph-mon[89963]: pgmap v323: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 656 op/s; 25560/225087 objects degraded (11.356%); 19020/225087 objects misplaced (8.450%); 9.6 MiB/s, 153 objects/s recovering 2024-08-17T17:21:27.501 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:21:27.527 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:27 smithi003 ceph-mon[100615]: pgmap v323: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 656 op/s; 25560/225087 objects degraded (11.356%); 19020/225087 objects misplaced (8.450%); 9.6 MiB/s, 153 objects/s recovering 2024-08-17T17:21:27.527 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:27 smithi003 ceph-mon[103344]: pgmap v323: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 656 op/s; 25560/225087 objects degraded (11.356%); 19020/225087 objects misplaced (8.450%); 9.6 MiB/s, 153 objects/s recovering 2024-08-17T17:21:27.564 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:27.604 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:27.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:27.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:27.626 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:21:27.705 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:27.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:27.751 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:27.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:27.815 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:21:27.817 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:21:27.852 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:27.881 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:27.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:27.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:27.985 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:28.018 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:28.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:28.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:28.120 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:28.151 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:28.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:28.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:28.253 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:28.287 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:28.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:28.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:28.390 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:28.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:28.430 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:28.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:28.531 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:28.553 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:28.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:28.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:28.655 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:28.679 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:28.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:28.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:28.780 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:28.821 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:28.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:28.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:28.923 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:28.952 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:28.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:28.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:29.054 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:29.090 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:29.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:29.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:29.192 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:29.201 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:21:29.223 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:29.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:29.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:29.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:29.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:29.371 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:29.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:29.414 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:29 smithi053 ceph-mon[89963]: pgmap v324: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 659 op/s; 25560/223746 objects degraded (11.424%); 19020/223746 objects misplaced (8.501%); 9.6 MiB/s, 153 objects/s recovering 2024-08-17T17:21:29.414 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:29 smithi053 ceph-mon[89963]: from='client.54859 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:21:29.472 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:29.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:29 smithi003 ceph-mon[100615]: pgmap v324: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 659 op/s; 25560/223746 objects degraded (11.424%); 19020/223746 objects misplaced (8.501%); 9.6 MiB/s, 153 objects/s recovering 2024-08-17T17:21:29.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:29 smithi003 ceph-mon[100615]: from='client.54859 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:21:29.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:29 smithi003 ceph-mon[103344]: pgmap v324: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 659 op/s; 25560/223746 objects degraded (11.424%); 19020/223746 objects misplaced (8.501%); 9.6 MiB/s, 153 objects/s recovering 2024-08-17T17:21:29.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:29 smithi003 ceph-mon[103344]: from='client.54859 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:21:29.526 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:29.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:29.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:29.553 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:21:29.554 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:21:29.628 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:29.672 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:29.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:29.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:29.774 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:29.810 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:29.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:29.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:29.913 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:29.954 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:29.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:29.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:30.055 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:30.093 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:30.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:30.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:30.194 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:30.240 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:30.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:30.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:30.256 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-17T17:21:30.343 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:30.375 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:30.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:30.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:30.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:21:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:21:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:21:30.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:30 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 25560/223746 objects degraded (11.424%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:30.477 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:30.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:30.518 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:30.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:30.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:30 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 25560/223746 objects degraded (11.424%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:30.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:30 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 25560/223746 objects degraded (11.424%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:30.619 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:30.681 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:30.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:30.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:30.719 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:21:30.783 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:30.824 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:30.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:30.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:30.926 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:30.966 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:30.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:30.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:31.068 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:31.112 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:31.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:31.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:31.126 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2024-08-17T17:21:27.750716+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4890695653,"num_objects":74582,"num_object_clones":0,"num_object_copies":223746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25560,"num_objects_misplaced":19020,"num_objects_unfound":0,"num_objects_dirty":74582,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":130678,"num_write_kb":6546496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11757,"num_bytes_recovered":771236048,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112165,"ondisk_log_size":112165,"up":267,"acting":257,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":280,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":15589988,"kb_used_data":14168684,"kb_used_omap":33,"kb_used_meta":1421214,"kb_avail":640626076,"statfs":{"total":671965249536,"available":656001101824,"internally_reserved":0,"allocated":14508732416,"data_stored":14503133317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34068,"internal_metadata":1455323884},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,2,5,2,3,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-432668672,"num_objects":-6602,"num_object_clones":0,"num_object_copies":-19806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-687,"num_objects_misplaced":-2088,"num_objects_unfound":0,"num_objects_dirty":-6602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6602,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1532,"num_bytes_recovered":100401152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003697"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":143,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.180390+0000","last_change":"2024-08-17T17:17:32.495652+0000","last_active":"2024-08-17T17:21:20.180390+0000","last_peered":"2024-08-17T17:21:20.180390+0000","last_clean":"2024-08-17T17:21:20.180390+0000","last_became_active":"2024-08-17T17:17:32.495218+0000","last_became_peered":"2024-08-17T17:17:32.495218+0000","last_unstale":"2024-08-17T17:21:20.180390+0000","last_undegraded":"2024-08-17T17:21:20.180390+0000","last_fullsized":"2024-08-17T17:21:20.180390+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:11:45.821282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":122,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-08-17T17:21:11.113852+0000","last_change":"2024-08-17T17:18:59.424887+0000","last_active":"2024-08-17T17:21:11.113852+0000","last_peered":"2024-08-17T17:21:11.113852+0000","last_clean":"2024-08-17T17:21:11.113852+0000","last_became_active":"2024-08-17T17:18:59.411539+0000","last_became_peered":"2024-08-17T17:18:59.411539+0000","last_unstale":"2024-08-17T17:21:11.113852+0000","last_undegraded":"2024-08-17T17:21:11.113852+0000","last_fullsized":"2024-08-17T17:21:11.113852+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:46:20.082241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":102,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-08-17T17:21:11.113726+0000","last_change":"2024-08-17T17:18:59.403259+0000","last_active":"2024-08-17T17:21:11.113726+0000","last_peered":"2024-08-17T17:21:11.113726+0000","last_clean":"2024-08-17T17:21:11.113726+0000","last_became_active":"2024-08-17T17:18:59.402706+0000","last_became_peered":"2024-08-17T17:18:59.402706+0000","last_unstale":"2024-08-17T17:21:11.113726+0000","last_undegraded":"2024-08-17T17:21:11.113726+0000","last_fullsized":"2024-08-17T17:21:11.113726+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:02:12.520417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":26,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-08-17T17:21:11.113573+0000","last_change":"2024-08-17T17:19:57.500646+0000","last_active":"2024-08-17T17:21:11.113573+0000","last_peered":"2024-08-17T17:21:11.113573+0000","last_clean":"2024-08-17T17:21:11.113573+0000","last_became_active":"2024-08-17T17:19:57.500332+0000","last_became_peered":"2024-08-17T17:19:57.500332+0000","last_unstale":"2024-08-17T17:21:11.113573+0000","last_undegraded":"2024-08-17T17:21:11.113573+0000","last_fullsized":"2024-08-17T17:21:11.113573+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:13:40.176690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":118,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.182143+0000","last_change":"2024-08-17T17:19:57.506750+0000","last_active":"2024-08-17T17:21:20.182143+0000","last_peered":"2024-08-17T17:21:20.182143+0000","last_clean":"2024-08-17T17:21:20.182143+0000","last_became_active":"2024-08-17T17:19:57.506360+0000","last_became_peered":"2024-08-17T17:19:57.506360+0000","last_unstale":"2024-08-17T17:21:20.182143+0000","last_undegraded":"2024-08-17T17:21:20.182143+0000","last_fullsized":"2024-08-17T17:21:20.182143+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:50:35.750066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":97,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-08-17T17:21:11.114755+0000","last_change":"2024-08-17T17:20:49.258206+0000","last_active":"2024-08-17T17:21:11.114755+0000","last_peered":"2024-08-17T17:21:11.114755+0000","last_clean":"2024-08-17T17:21:11.114755+0000","last_became_active":"2024-08-17T17:20:49.257761+0000","last_became_peered":"2024-08-17T17:20:49.257761+0000","last_unstale":"2024-08-17T17:21:11.114755+0000","last_undegraded":"2024-08-17T17:21:11.114755+0000","last_fullsized":"2024-08-17T17:21:11.114755+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:38:35.363748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":157,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.181215+0000","last_change":"2024-08-17T17:19:57.514773+0000","last_active":"2024-08-17T17:21:20.181215+0000","last_peered":"2024-08-17T17:21:20.181215+0000","last_clean":"2024-08-17T17:21:20.181215+0000","last_became_active":"2024-08-17T17:19:57.513839+0000","last_became_peered":"2024-08-17T17:19:57.513839+0000","last_unstale":"2024-08-17T17:21:20.181215+0000","last_undegraded":"2024-08-17T17:21:20.181215+0000","last_fullsized":"2024-08-17T17:21:20.181215+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:42:48.063169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":152,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-08-17T17:21:11.118248+0000","last_change":"2024-08-17T17:19:57.520685+0000","last_active":"2024-08-17T17:21:11.118248+0000","last_peered":"2024-08-17T17:21:11.118248+0000","last_clean":"2024-08-17T17:21:11.118248+0000","last_became_active":"2024-08-17T17:19:57.508226+0000","last_became_peered":"2024-08-17T17:19:57.508226+0000","last_unstale":"2024-08-17T17:21:11.118248+0000","last_undegraded":"2024-08-17T17:21:11.118248+0000","last_fullsized":"2024-08-17T17:21:11.118248+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:17:42.506273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"146'7916","reported_seq":10677,"reported_epoch":146,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:21:23.800667+0000","last_change":"2024-08-17T17:20:50.522509+0000","last_active":"2024-08-17T17:21:23.800667+0000","last_peered":"2024-08-17T17:21:23.800667+0000","last_clean":"2024-08-17T17:19:56.459843+0000","last_became_active":"2024-08-17T17:20:50.428702+0000","last_became_peered":"2024-08-17T17:20:50.428702+0000","last_unstale":"2024-08-17T17:21:23.800667+0000","last_undegraded":"2024-08-17T17:20:50.411703+0000","last_fullsized":"2024-08-17T17:20:50.249933+0000","mapping_epoch":141,"log_start":"123'1873","ondisk_log_start":"123'1873","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":6043,"log_dups_size":0,"ondisk_log_size":6043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:15:36.796575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308281344,"num_objects":4704,"num_object_clones":0,"num_object_copies":14112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7916,"num_write_kb":403840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1344,"num_bytes_recovered":88080384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"1,4,7","objects":2643},{"shards":"4,6,7","objects":2060},{"shards":"6,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":44,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-08-17T17:21:11.114089+0000","last_change":"2024-08-17T17:20:49.297856+0000","last_active":"2024-08-17T17:21:11.114089+0000","last_peered":"2024-08-17T17:21:11.114089+0000","last_clean":"2024-08-17T17:21:11.114089+0000","last_became_active":"2024-08-17T17:20:49.261859+0000","last_became_peered":"2024-08-17T17:20:49.261859+0000","last_unstale":"2024-08-17T17:21:11.114089+0000","last_undegraded":"2024-08-17T17:21:11.114089+0000","last_fullsized":"2024-08-17T17:21:11.114089+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:20:51.487220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":36,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-08-17T17:21:11.113340+0000","last_change":"2024-08-17T17:20:49.261329+0000","last_active":"2024-08-17T17:21:11.113340+0000","last_peered":"2024-08-17T17:21:11.113340+0000","last_clean":"2024-08-17T17:21:11.113340+0000","last_became_active":"2024-08-17T17:20:49.261070+0000","last_became_peered":"2024-08-17T17:20:49.261070+0000","last_unstale":"2024-08-17T17:21:11.113340+0000","last_undegraded":"2024-08-17T17:21:11.113340+0000","last_fullsized":"2024-08-17T17:21:11.113340+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:07:17.916979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"146'8215","reported_seq":8300,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:27.599299+0000","last_change":"2024-08-17T17:20:14.440439+0000","last_active":"2024-08-17T17:21:27.599299+0000","last_peered":"2024-08-17T17:21:27.599299+0000","last_clean":"2024-08-17T17:21:27.599299+0000","last_became_active":"2024-08-17T17:20:14.430994+0000","last_became_peered":"2024-08-17T17:20:14.430994+0000","last_unstale":"2024-08-17T17:21:27.599299+0000","last_undegraded":"2024-08-17T17:21:27.599299+0000","last_fullsized":"2024-08-17T17:21:27.599299+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":8215,"log_dups_size":0,"ondisk_log_size":8215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:01:41.276551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295370752,"num_objects":4507,"num_object_clones":0,"num_object_copies":13521,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8215,"num_write_kb":407104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":30,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-08-17T17:21:11.114089+0000","last_change":"2024-08-17T17:19:57.527977+0000","last_active":"2024-08-17T17:21:11.114089+0000","last_peered":"2024-08-17T17:21:11.114089+0000","last_clean":"2024-08-17T17:21:11.114089+0000","last_became_active":"2024-08-17T17:19:57.505766+0000","last_became_peered":"2024-08-17T17:19:57.505766+0000","last_unstale":"2024-08-17T17:21:11.114089+0000","last_undegraded":"2024-08-17T17:21:11.114089+0000","last_fullsized":"2024-08-17T17:21:11.114089+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:05:23.673641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":98,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.185454+0000","last_change":"2024-08-17T17:20:49.271540+0000","last_active":"2024-08-17T17:21:20.185454+0000","last_peered":"2024-08-17T17:21:20.185454+0000","last_clean":"2024-08-17T17:21:20.185454+0000","last_became_active":"2024-08-17T17:20:49.271130+0000","last_became_peered":"2024-08-17T17:20:49.271130+0000","last_unstale":"2024-08-17T17:21:20.185454+0000","last_undegraded":"2024-08-17T17:21:20.185454+0000","last_fullsized":"2024-08-17T17:21:20.185454+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:05.386705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"146'8307","reported_seq":8395,"reported_epoch":146,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:21:27.514532+0000","last_change":"2024-08-17T17:19:58.764064+0000","last_active":"2024-08-17T17:21:27.514532+0000","last_peered":"2024-08-17T17:21:27.514532+0000","last_clean":"2024-08-17T17:19:56.456607+0000","last_became_active":"2024-08-17T17:19:58.752544+0000","last_became_peered":"2024-08-17T17:19:58.752544+0000","last_unstale":"2024-08-17T17:21:27.514532+0000","last_undegraded":"2024-08-17T17:19:58.508594+0000","last_fullsized":"2024-08-17T17:19:58.482360+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":8307,"log_dups_size":0,"ondisk_log_size":8307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300220416,"num_objects":4581,"num_object_clones":0,"num_object_copies":13743,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8307,"num_write_kb":412416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,1],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1","objects":273},{"shards":"0,1,2","objects":2445}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":102,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-08-17T17:21:11.113893+0000","last_change":"2024-08-17T17:18:59.420539+0000","last_active":"2024-08-17T17:21:11.113893+0000","last_peered":"2024-08-17T17:21:11.113893+0000","last_clean":"2024-08-17T17:21:11.113893+0000","last_became_active":"2024-08-17T17:18:59.404078+0000","last_became_peered":"2024-08-17T17:18:59.404078+0000","last_unstale":"2024-08-17T17:21:11.113893+0000","last_undegraded":"2024-08-17T17:21:11.113893+0000","last_fullsized":"2024-08-17T17:21:11.113893+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:48:47.883684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":131,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.177735+0000","last_change":"2024-08-17T17:19:57.504404+0000","last_active":"2024-08-17T17:21:20.177735+0000","last_peered":"2024-08-17T17:21:20.177735+0000","last_clean":"2024-08-17T17:21:20.177735+0000","last_became_active":"2024-08-17T17:19:57.504036+0000","last_became_peered":"2024-08-17T17:19:57.504036+0000","last_unstale":"2024-08-17T17:21:20.177735+0000","last_undegraded":"2024-08-17T17:21:20.177735+0000","last_fullsized":"2024-08-17T17:21:20.177735+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:14:07.727290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"146'7945","reported_seq":11443,"reported_epoch":146,"state":"active+recovery_wait+degraded","last_fresh":"2024-08-17T17:21:25.692528+0000","last_change":"2024-08-17T17:20:15.471683+0000","last_active":"2024-08-17T17:21:25.692528+0000","last_peered":"2024-08-17T17:21:25.692528+0000","last_clean":"2024-08-17T17:19:56.442753+0000","last_became_active":"2024-08-17T17:20:15.438280+0000","last_became_peered":"2024-08-17T17:20:15.438280+0000","last_unstale":"2024-08-17T17:21:25.692528+0000","last_undegraded":"2024-08-17T17:20:15.430252+0000","last_fullsized":"2024-08-17T17:21:25.692528+0000","mapping_epoch":136,"log_start":"123'1925","ondisk_log_start":"123'1925","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":6020,"log_dups_size":0,"ondisk_log_size":6020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297598976,"num_objects":4541,"num_object_clones":0,"num_object_copies":13623,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2500,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7945,"num_write_kb":399552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1704,"num_bytes_recovered":111673344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":["3","0","2"],"object_location_counts":[{"shards":"0,2,3","objects":2500},{"shards":"1,2,3","objects":2041}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":104,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-08-17T17:21:11.134092+0000","last_change":"2024-08-17T17:17:32.495428+0000","last_active":"2024-08-17T17:21:11.134092+0000","last_peered":"2024-08-17T17:21:11.134092+0000","last_clean":"2024-08-17T17:21:11.134092+0000","last_became_active":"2024-08-17T17:17:32.494978+0000","last_became_peered":"2024-08-17T17:17:32.494978+0000","last_unstale":"2024-08-17T17:21:11.134092+0000","last_undegraded":"2024-08-17T17:21:11.134092+0000","last_fullsized":"2024-08-17T17:21:11.134092+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:42:24.647812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":146,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.194234+0000","last_change":"2024-08-17T17:20:49.272126+0000","last_active":"2024-08-17T17:21:20.194234+0000","last_peered":"2024-08-17T17:21:20.194234+0000","last_clean":"2024-08-17T17:21:20.194234+0000","last_became_active":"2024-08-17T17:20:49.271185+0000","last_became_peered":"2024-08-17T17:20:49.271185+0000","last_unstale":"2024-08-17T17:21:20.194234+0000","last_undegraded":"2024-08-17T17:21:20.194234+0000","last_fullsized":"2024-08-17T17:21:20.194234+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:43:58.184939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"146'8067","reported_seq":8150,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:23.814952+0000","last_change":"2024-08-17T17:20:14.437947+0000","last_active":"2024-08-17T17:21:23.814952+0000","last_peered":"2024-08-17T17:21:23.814952+0000","last_clean":"2024-08-17T17:21:23.814952+0000","last_became_active":"2024-08-17T17:20:14.428646+0000","last_became_peered":"2024-08-17T17:20:14.428646+0000","last_unstale":"2024-08-17T17:21:23.814952+0000","last_undegraded":"2024-08-17T17:21:23.814952+0000","last_fullsized":"2024-08-17T17:21:23.814952+0000","mapping_epoch":135,"log_start":"123'2063","ondisk_log_start":"123'2063","created":111,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":6004,"log_dups_size":0,"ondisk_log_size":6004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:06:56.380235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307363864,"num_objects":4691,"num_object_clones":0,"num_object_copies":14073,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4691,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8067,"num_write_kb":408193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":163,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.187213+0000","last_change":"2024-08-17T17:20:49.295255+0000","last_active":"2024-08-17T17:21:20.187213+0000","last_peered":"2024-08-17T17:21:20.187213+0000","last_clean":"2024-08-17T17:21:20.187213+0000","last_became_active":"2024-08-17T17:20:49.262267+0000","last_became_peered":"2024-08-17T17:20:49.262267+0000","last_unstale":"2024-08-17T17:21:20.187213+0000","last_undegraded":"2024-08-17T17:21:20.187213+0000","last_fullsized":"2024-08-17T17:21:20.187213+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:22:30.749208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":120,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.181642+0000","last_change":"2024-08-17T17:20:49.280596+0000","last_active":"2024-08-17T17:21:20.181642+0000","last_peered":"2024-08-17T17:21:20.181642+0000","last_clean":"2024-08-17T17:21:20.181642+0000","last_became_active":"2024-08-17T17:20:49.279553+0000","last_became_peered":"2024-08-17T17:20:49.279553+0000","last_unstale":"2024-08-17T17:21:20.181642+0000","last_undegraded":"2024-08-17T17:21:20.181642+0000","last_fullsized":"2024-08-17T17:21:20.181642+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:13:24.124392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"146'8016","reported_seq":8101,"reported_epoch":146,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T17:21:24.737495+0000","last_change":"2024-08-17T17:20:50.393430+0000","last_active":"2024-08-17T17:21:24.737495+0000","last_peered":"2024-08-17T17:21:24.737495+0000","last_clean":"2024-08-17T17:20:48.214722+0000","last_became_active":"2024-08-17T17:20:50.375136+0000","last_became_peered":"2024-08-17T17:20:50.375136+0000","last_unstale":"2024-08-17T17:21:24.737495+0000","last_undegraded":"2024-08-17T17:21:24.737495+0000","last_fullsized":"2024-08-17T17:21:24.737495+0000","mapping_epoch":141,"log_start":"123'2757","ondisk_log_start":"123'2757","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":5259,"log_dups_size":0,"ondisk_log_size":5259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305528832,"num_objects":4662,"num_object_clones":0,"num_object_copies":13986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4662,"num_objects_unfound":0,"num_objects_dirty":4662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8016,"num_write_kb":405696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,4],"avail_no_missing":["1","0","4"],"object_location_counts":[{"shards":"0,1,4","objects":4662}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":72,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.177963+0000","last_change":"2024-08-17T17:20:49.276769+0000","last_active":"2024-08-17T17:21:20.177963+0000","last_peered":"2024-08-17T17:21:20.177963+0000","last_clean":"2024-08-17T17:21:20.177963+0000","last_became_active":"2024-08-17T17:20:49.276498+0000","last_became_peered":"2024-08-17T17:20:49.276498+0000","last_unstale":"2024-08-17T17:21:20.177963+0000","last_undegraded":"2024-08-17T17:21:20.177963+0000","last_fullsized":"2024-08-17T17:21:20.177963+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:39:05.419440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":125,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-08-17T17:21:11.113984+0000","last_change":"2024-08-17T17:20:14.455738+0000","last_active":"2024-08-17T17:21:11.113984+0000","last_peered":"2024-08-17T17:21:11.113984+0000","last_clean":"2024-08-17T17:21:11.113984+0000","last_became_active":"2024-08-17T17:20:14.455469+0000","last_became_peered":"2024-08-17T17:20:14.455469+0000","last_unstale":"2024-08-17T17:21:11.113984+0000","last_undegraded":"2024-08-17T17:21:11.113984+0000","last_fullsized":"2024-08-17T17:21:11.113984+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:27:28.424336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"146'7969","reported_seq":8022,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:23.644413+0000","last_change":"2024-08-17T17:18:59.413691+0000","last_active":"2024-08-17T17:21:23.644413+0000","last_peered":"2024-08-17T17:21:23.644413+0000","last_clean":"2024-08-17T17:21:23.644413+0000","last_became_active":"2024-08-17T17:18:59.413358+0000","last_became_peered":"2024-08-17T17:18:59.413358+0000","last_unstale":"2024-08-17T17:21:23.644413+0000","last_undegraded":"2024-08-17T17:21:23.644413+0000","last_fullsized":"2024-08-17T17:21:23.644413+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":7969,"log_dups_size":0,"ondisk_log_size":7969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:51:20.829898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304807936,"num_objects":4651,"num_object_clones":0,"num_object_copies":13953,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7969,"num_write_kb":403840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":79,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-08-17T17:21:11.114677+0000","last_change":"2024-08-17T17:20:14.431140+0000","last_active":"2024-08-17T17:21:11.114677+0000","last_peered":"2024-08-17T17:21:11.114677+0000","last_clean":"2024-08-17T17:21:11.114677+0000","last_became_active":"2024-08-17T17:20:14.430839+0000","last_became_peered":"2024-08-17T17:20:14.430839+0000","last_unstale":"2024-08-17T17:21:11.114677+0000","last_undegraded":"2024-08-17T17:21:11.114677+0000","last_fullsized":"2024-08-17T17:21:11.114677+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:08:16.303616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":95,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.176835+0000","last_change":"2024-08-17T17:20:49.272325+0000","last_active":"2024-08-17T17:21:20.176835+0000","last_peered":"2024-08-17T17:21:20.176835+0000","last_clean":"2024-08-17T17:21:20.176835+0000","last_became_active":"2024-08-17T17:20:49.272002+0000","last_became_peered":"2024-08-17T17:20:49.272002+0000","last_unstale":"2024-08-17T17:21:20.176835+0000","last_undegraded":"2024-08-17T17:21:20.176835+0000","last_fullsized":"2024-08-17T17:21:20.176835+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:47:32.474887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"146'8140","reported_seq":12773,"reported_epoch":146,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:21:23.673807+0000","last_change":"2024-08-17T17:20:49.267398+0000","last_active":"2024-08-17T17:21:23.673807+0000","last_peered":"2024-08-17T17:21:23.673807+0000","last_clean":"2024-08-17T17:19:56.469841+0000","last_became_active":"2024-08-17T17:19:58.712493+0000","last_became_peered":"2024-08-17T17:19:58.712493+0000","last_unstale":"2024-08-17T17:21:23.673807+0000","last_undegraded":"2024-08-17T17:20:49.265798+0000","last_fullsized":"2024-08-17T17:19:58.486989+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":8140,"log_dups_size":0,"ondisk_log_size":8140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:48:09.444362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305135616,"num_objects":4656,"num_object_clones":0,"num_object_copies":13968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2015,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8140,"num_write_kb":409472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2288,"num_bytes_recovered":149946368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0],"avail_no_missing":["5","0","2"],"object_location_counts":[{"shards":"0,2,5","objects":2015},{"shards":"0,5","objects":2641}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":158,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.181045+0000","last_change":"2024-08-17T17:19:57.520745+0000","last_active":"2024-08-17T17:21:20.181045+0000","last_peered":"2024-08-17T17:21:20.181045+0000","last_clean":"2024-08-17T17:21:20.181045+0000","last_became_active":"2024-08-17T17:19:57.520152+0000","last_became_peered":"2024-08-17T17:19:57.520152+0000","last_unstale":"2024-08-17T17:21:20.181045+0000","last_undegraded":"2024-08-17T17:21:20.181045+0000","last_fullsized":"2024-08-17T17:21:20.181045+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:19:39.495644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":134,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.178036+0000","last_change":"2024-08-17T17:20:49.274372+0000","last_active":"2024-08-17T17:21:20.178036+0000","last_peered":"2024-08-17T17:21:20.178036+0000","last_clean":"2024-08-17T17:21:20.178036+0000","last_became_active":"2024-08-17T17:20:49.274023+0000","last_became_peered":"2024-08-17T17:20:49.274023+0000","last_unstale":"2024-08-17T17:21:20.178036+0000","last_undegraded":"2024-08-17T17:21:20.178036+0000","last_fullsized":"2024-08-17T17:21:20.178036+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:31:02.529766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"146'7917","reported_seq":10453,"reported_epoch":146,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-17T17:21:23.735849+0000","last_change":"2024-08-17T17:20:49.296622+0000","last_active":"2024-08-17T17:21:23.735849+0000","last_peered":"2024-08-17T17:21:23.735849+0000","last_clean":"2024-08-17T17:19:56.496869+0000","last_became_active":"2024-08-17T17:19:58.728339+0000","last_became_peered":"2024-08-17T17:19:58.728339+0000","last_unstale":"2024-08-17T17:21:23.735849+0000","last_undegraded":"2024-08-17T17:19:58.649778+0000","last_fullsized":"2024-08-17T17:19:58.487889+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":7917,"log_dups_size":0,"ondisk_log_size":7917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303366144,"num_objects":4629,"num_object_clones":0,"num_object_copies":13887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":419,"num_objects_misplaced":2510,"num_objects_unfound":0,"num_objects_dirty":4629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7917,"num_write_kb":401472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1228,"num_bytes_recovered":80478208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[2,5],"avail_no_missing":["2","0","5"],"object_location_counts":[{"shards":"0,2,5","objects":2510},{"shards":"2,5","objects":1700}],"blocked_by":[],"up_primary":7,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":447,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-08-17T17:21:11.113313+0000","last_change":"2024-08-17T17:20:00.425796+0000","last_active":"2024-08-17T17:21:11.113313+0000","last_peered":"2024-08-17T17:21:11.113313+0000","last_clean":"2024-08-17T17:21:11.113313+0000","last_became_active":"2024-08-17T17:20:00.425388+0000","last_became_peered":"2024-08-17T17:20:00.425388+0000","last_unstale":"2024-08-17T17:21:11.113313+0000","last_undegraded":"2024-08-17T17:21:11.113313+0000","last_fullsized":"2024-08-17T17:21:11.113313+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:04:01.141401+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:12.168702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":718,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-08-17T17:21:11.133846+0000","last_change":"2024-08-17T17:20:00.434774+0000","last_active":"2024-08-17T17:21:11.133846+0000","last_peered":"2024-08-17T17:21:11.133846+0000","last_clean":"2024-08-17T17:21:11.133846+0000","last_became_active":"2024-08-17T17:20:00.434343+0000","last_became_peered":"2024-08-17T17:20:00.434343+0000","last_unstale":"2024-08-17T17:21:11.133846+0000","last_undegraded":"2024-08-17T17:21:11.133846+0000","last_fullsized":"2024-08-17T17:21:11.133846+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:04.995152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":109,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.183500+0000","last_change":"2024-08-17T17:20:49.263797+0000","last_active":"2024-08-17T17:21:20.183500+0000","last_peered":"2024-08-17T17:21:20.183500+0000","last_clean":"2024-08-17T17:21:20.183500+0000","last_became_active":"2024-08-17T17:20:49.263488+0000","last_became_peered":"2024-08-17T17:20:49.263488+0000","last_unstale":"2024-08-17T17:21:20.183500+0000","last_undegraded":"2024-08-17T17:21:20.183500+0000","last_fullsized":"2024-08-17T17:21:20.183500+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:50:22.123993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":45,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-08-17T17:21:11.113807+0000","last_change":"2024-08-17T17:18:59.417834+0000","last_active":"2024-08-17T17:21:11.113807+0000","last_peered":"2024-08-17T17:21:11.113807+0000","last_clean":"2024-08-17T17:21:11.113807+0000","last_became_active":"2024-08-17T17:18:59.417456+0000","last_became_peered":"2024-08-17T17:18:59.417456+0000","last_unstale":"2024-08-17T17:21:11.113807+0000","last_undegraded":"2024-08-17T17:21:11.113807+0000","last_fullsized":"2024-08-17T17:21:11.113807+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:22:45.101077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"146'8152","reported_seq":8232,"reported_epoch":146,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T17:21:24.810430+0000","last_change":"2024-08-17T17:20:50.369539+0000","last_active":"2024-08-17T17:21:24.810430+0000","last_peered":"2024-08-17T17:21:24.810430+0000","last_clean":"2024-08-17T17:20:48.214899+0000","last_became_active":"2024-08-17T17:20:50.362743+0000","last_became_peered":"2024-08-17T17:20:50.362743+0000","last_unstale":"2024-08-17T17:21:24.810430+0000","last_undegraded":"2024-08-17T17:21:24.810430+0000","last_fullsized":"2024-08-17T17:21:24.810430+0000","mapping_epoch":141,"log_start":"125'2923","ondisk_log_start":"125'2923","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":5229,"log_dups_size":0,"ondisk_log_size":5229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306184192,"num_objects":4672,"num_object_clones":0,"num_object_copies":14016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4672,"num_objects_unfound":0,"num_objects_dirty":4672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8152,"num_write_kb":410368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,4],"avail_no_missing":["1","0","4"],"object_location_counts":[{"shards":"0,1,4","objects":4672}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":736,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-08-17T17:21:11.113442+0000","last_change":"2024-08-17T17:16:26.958415+0000","last_active":"2024-08-17T17:21:11.113442+0000","last_peered":"2024-08-17T17:21:11.113442+0000","last_clean":"2024-08-17T17:21:11.113442+0000","last_became_active":"2024-08-17T17:16:26.957956+0000","last_became_peered":"2024-08-17T17:16:26.957956+0000","last_unstale":"2024-08-17T17:21:11.113442+0000","last_undegraded":"2024-08-17T17:21:11.113442+0000","last_fullsized":"2024-08-17T17:21:11.113442+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:22.478077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":145,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.189760+0000","last_change":"2024-08-17T17:18:59.415331+0000","last_active":"2024-08-17T17:21:20.189760+0000","last_peered":"2024-08-17T17:21:20.189760+0000","last_clean":"2024-08-17T17:21:20.189760+0000","last_became_active":"2024-08-17T17:18:59.415008+0000","last_became_peered":"2024-08-17T17:18:59.415008+0000","last_unstale":"2024-08-17T17:21:20.189760+0000","last_undegraded":"2024-08-17T17:21:20.189760+0000","last_fullsized":"2024-08-17T17:21:20.189760+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:22:38.076866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":87,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-08-17T17:21:11.137020+0000","last_change":"2024-08-17T17:19:57.491741+0000","last_active":"2024-08-17T17:21:11.137020+0000","last_peered":"2024-08-17T17:21:11.137020+0000","last_clean":"2024-08-17T17:21:11.137020+0000","last_became_active":"2024-08-17T17:19:57.491070+0000","last_became_peered":"2024-08-17T17:19:57.491070+0000","last_unstale":"2024-08-17T17:21:11.137020+0000","last_undegraded":"2024-08-17T17:21:11.137020+0000","last_fullsized":"2024-08-17T17:21:11.137020+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:15:35.046763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"146'8099","reported_seq":8170,"reported_epoch":146,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:21:23.561259+0000","last_change":"2024-08-17T17:19:58.741198+0000","last_active":"2024-08-17T17:21:23.561259+0000","last_peered":"2024-08-17T17:21:23.561259+0000","last_clean":"2024-08-17T17:19:56.456745+0000","last_became_active":"2024-08-17T17:19:58.736100+0000","last_became_peered":"2024-08-17T17:19:58.736100+0000","last_unstale":"2024-08-17T17:21:23.561259+0000","last_undegraded":"2024-08-17T17:19:58.525523+0000","last_fullsized":"2024-08-17T17:19:58.486842+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":8099,"log_dups_size":0,"ondisk_log_size":8099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:42:51.931741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314114048,"num_objects":4793,"num_object_clones":0,"num_object_copies":14379,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8099,"num_write_kb":412544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7],"avail_no_missing":["5","2","7"],"object_location_counts":[{"shards":"2,5,7","objects":2663},{"shards":"5,7","objects":477}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":533,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.192015+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:21:20.192015+0000","last_peered":"2024-08-17T17:21:20.192015+0000","last_clean":"2024-08-17T17:21:20.192015+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:21:20.192015+0000","last_undegraded":"2024-08-17T17:21:20.192015+0000","last_fullsized":"2024-08-17T17:21:20.192015+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":141,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.179941+0000","last_change":"2024-08-17T17:19:57.515564+0000","last_active":"2024-08-17T17:21:20.179941+0000","last_peered":"2024-08-17T17:21:20.179941+0000","last_clean":"2024-08-17T17:21:20.179941+0000","last_became_active":"2024-08-17T17:19:57.515142+0000","last_became_peered":"2024-08-17T17:19:57.515142+0000","last_unstale":"2024-08-17T17:21:20.179941+0000","last_undegraded":"2024-08-17T17:21:20.179941+0000","last_fullsized":"2024-08-17T17:21:20.179941+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:59:33.506176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":165,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.177208+0000","last_change":"2024-08-17T17:19:57.506668+0000","last_active":"2024-08-17T17:21:20.177208+0000","last_peered":"2024-08-17T17:21:20.177208+0000","last_clean":"2024-08-17T17:21:20.177208+0000","last_became_active":"2024-08-17T17:19:57.505453+0000","last_became_peered":"2024-08-17T17:19:57.505453+0000","last_unstale":"2024-08-17T17:21:20.177208+0000","last_undegraded":"2024-08-17T17:21:20.177208+0000","last_fullsized":"2024-08-17T17:21:20.177208+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:28:48.696602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"146'8037","reported_seq":12435,"reported_epoch":146,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:21:24.777276+0000","last_change":"2024-08-17T17:20:49.284932+0000","last_active":"2024-08-17T17:21:24.777276+0000","last_peered":"2024-08-17T17:21:24.777276+0000","last_clean":"2024-08-17T17:19:56.454627+0000","last_became_active":"2024-08-17T17:19:58.696040+0000","last_became_peered":"2024-08-17T17:19:58.696040+0000","last_unstale":"2024-08-17T17:21:24.777276+0000","last_undegraded":"2024-08-17T17:20:49.283812+0000","last_fullsized":"2024-08-17T17:19:58.484961+0000","mapping_epoch":130,"log_start":"124'2785","ondisk_log_start":"124'2785","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":5252,"log_dups_size":0,"ondisk_log_size":5252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309002240,"num_objects":4715,"num_object_clones":0,"num_object_copies":14145,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2010,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8037,"num_write_kb":408064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2152,"num_bytes_recovered":141033472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6],"avail_no_missing":["1","0","6"],"object_location_counts":[{"shards":"0,1,6","objects":2010},{"shards":"1,6","objects":2705}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":1056,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-08-17T17:21:11.133815+0000","last_change":"2024-08-17T17:20:00.433870+0000","last_active":"2024-08-17T17:21:11.133815+0000","last_peered":"2024-08-17T17:21:11.133815+0000","last_clean":"2024-08-17T17:21:11.133815+0000","last_became_active":"2024-08-17T17:20:00.433386+0000","last_became_peered":"2024-08-17T17:20:00.433386+0000","last_unstale":"2024-08-17T17:21:11.133815+0000","last_undegraded":"2024-08-17T17:21:11.133815+0000","last_fullsized":"2024-08-17T17:21:11.133815+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:01:11.848862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1392,"num_read_kb":2863,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":67,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.193388+0000","last_change":"2024-08-17T17:20:49.272407+0000","last_active":"2024-08-17T17:21:20.193388+0000","last_peered":"2024-08-17T17:21:20.193388+0000","last_clean":"2024-08-17T17:21:20.193388+0000","last_became_active":"2024-08-17T17:20:49.271749+0000","last_became_peered":"2024-08-17T17:20:49.271749+0000","last_unstale":"2024-08-17T17:21:20.193388+0000","last_undegraded":"2024-08-17T17:21:20.193388+0000","last_fullsized":"2024-08-17T17:21:20.193388+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:36:01.443457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":138,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.176922+0000","last_change":"2024-08-17T17:17:32.499715+0000","last_active":"2024-08-17T17:21:20.176922+0000","last_peered":"2024-08-17T17:21:20.176922+0000","last_clean":"2024-08-17T17:21:20.176922+0000","last_became_active":"2024-08-17T17:17:32.499412+0000","last_became_peered":"2024-08-17T17:17:32.499412+0000","last_unstale":"2024-08-17T17:21:20.176922+0000","last_undegraded":"2024-08-17T17:21:20.176922+0000","last_fullsized":"2024-08-17T17:21:20.176922+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:39:38.456970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"146'8165","reported_seq":10841,"reported_epoch":146,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-17T17:21:23.701465+0000","last_change":"2024-08-17T17:20:49.283753+0000","last_active":"2024-08-17T17:21:23.701465+0000","last_peered":"2024-08-17T17:21:23.701465+0000","last_clean":"2024-08-17T17:19:56.442693+0000","last_became_active":"2024-08-17T17:19:58.754028+0000","last_became_peered":"2024-08-17T17:19:58.754028+0000","last_unstale":"2024-08-17T17:21:23.701465+0000","last_undegraded":"2024-08-17T17:19:58.515410+0000","last_fullsized":"2024-08-17T17:19:58.482756+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":8165,"log_dups_size":0,"ondisk_log_size":8165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:30:42.382781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309264384,"num_objects":4719,"num_object_clones":0,"num_object_copies":14157,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":430,"num_objects_misplaced":2587,"num_objects_unfound":0,"num_objects_dirty":4719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8165,"num_write_kb":412288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1298,"num_bytes_recovered":85065728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3],"avail_no_missing":["6","0","3"],"object_location_counts":[{"shards":"0,3,6","objects":2587},{"shards":"3,6","objects":1702}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":121,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-08-17T17:21:11.134005+0000","last_change":"2024-08-17T17:17:32.495728+0000","last_active":"2024-08-17T17:21:11.134005+0000","last_peered":"2024-08-17T17:21:11.134005+0000","last_clean":"2024-08-17T17:21:11.134005+0000","last_became_active":"2024-08-17T17:17:32.495357+0000","last_became_peered":"2024-08-17T17:17:32.495357+0000","last_unstale":"2024-08-17T17:21:11.134005+0000","last_undegraded":"2024-08-17T17:21:11.134005+0000","last_fullsized":"2024-08-17T17:21:11.134005+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:27:45.561567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":374,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.177121+0000","last_change":"2024-08-17T17:16:26.964621+0000","last_active":"2024-08-17T17:21:20.177121+0000","last_peered":"2024-08-17T17:21:20.177121+0000","last_clean":"2024-08-17T17:21:20.177121+0000","last_became_active":"2024-08-17T17:16:26.964058+0000","last_became_peered":"2024-08-17T17:16:26.964058+0000","last_unstale":"2024-08-17T17:21:20.177121+0000","last_undegraded":"2024-08-17T17:21:20.177121+0000","last_fullsized":"2024-08-17T17:21:20.177121+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:00:04.465644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":130,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.178067+0000","last_change":"2024-08-17T17:19:57.513601+0000","last_active":"2024-08-17T17:21:20.178067+0000","last_peered":"2024-08-17T17:21:20.178067+0000","last_clean":"2024-08-17T17:21:20.178067+0000","last_became_active":"2024-08-17T17:19:57.513260+0000","last_became_peered":"2024-08-17T17:19:57.513260+0000","last_unstale":"2024-08-17T17:21:20.178067+0000","last_undegraded":"2024-08-17T17:21:20.178067+0000","last_fullsized":"2024-08-17T17:21:20.178067+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:36:00.355650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"146'8274","reported_seq":8317,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:23.701278+0000","last_change":"2024-08-17T17:18:56.578793+0000","last_active":"2024-08-17T17:21:23.701278+0000","last_peered":"2024-08-17T17:21:23.701278+0000","last_clean":"2024-08-17T17:21:23.701278+0000","last_became_active":"2024-08-17T17:18:56.578433+0000","last_became_peered":"2024-08-17T17:18:56.578433+0000","last_unstale":"2024-08-17T17:21:23.701278+0000","last_undegraded":"2024-08-17T17:21:23.701278+0000","last_fullsized":"2024-08-17T17:21:23.701278+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":8274,"log_dups_size":0,"ondisk_log_size":8274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:25:02.009751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316145664,"num_objects":4824,"num_object_clones":0,"num_object_copies":14472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8274,"num_write_kb":419136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":148,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.177021+0000","last_change":"2024-08-17T17:20:49.266165+0000","last_active":"2024-08-17T17:21:20.177021+0000","last_peered":"2024-08-17T17:21:20.177021+0000","last_clean":"2024-08-17T17:21:20.177021+0000","last_became_active":"2024-08-17T17:20:49.265808+0000","last_became_peered":"2024-08-17T17:20:49.265808+0000","last_unstale":"2024-08-17T17:21:20.177021+0000","last_undegraded":"2024-08-17T17:21:20.177021+0000","last_fullsized":"2024-08-17T17:21:20.177021+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:17:06.083608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":499,"reported_epoch":145,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:21:11.114022+0000","last_change":"2024-08-17T17:20:50.267646+0000","last_active":"2024-08-17T17:21:11.114022+0000","last_peered":"2024-08-17T17:21:11.114022+0000","last_clean":"2024-08-17T17:20:38.859867+0000","last_became_active":"2024-08-17T17:20:50.267212+0000","last_became_peered":"2024-08-17T17:20:50.267212+0000","last_unstale":"2024-08-17T17:21:11.114022+0000","last_undegraded":"2024-08-17T17:20:50.251503+0000","last_fullsized":"2024-08-17T17:20:50.250987+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:39:10.599988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1],"avail_no_missing":["7","0","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":128,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.194146+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:21:20.194146+0000","last_peered":"2024-08-17T17:21:20.194146+0000","last_clean":"2024-08-17T17:21:20.194146+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:21:20.194146+0000","last_undegraded":"2024-08-17T17:21:20.194146+0000","last_fullsized":"2024-08-17T17:21:20.194146+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"146'8174","reported_seq":11558,"reported_epoch":146,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:21:25.664164+0000","last_change":"2024-08-17T17:20:49.297530+0000","last_active":"2024-08-17T17:21:25.664164+0000","last_peered":"2024-08-17T17:21:25.664164+0000","last_clean":"2024-08-17T17:19:56.461183+0000","last_became_active":"2024-08-17T17:19:58.704346+0000","last_became_peered":"2024-08-17T17:19:58.704346+0000","last_unstale":"2024-08-17T17:21:25.664164+0000","last_undegraded":"2024-08-17T17:19:58.512446+0000","last_fullsized":"2024-08-17T17:19:58.480807+0000","mapping_epoch":130,"log_start":"123'2103","ondisk_log_start":"123'2103","created":111,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":6071,"log_dups_size":0,"ondisk_log_size":6071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302776320,"num_objects":4620,"num_object_clones":0,"num_object_copies":13860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4306,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8174,"num_write_kb":409408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1645,"num_bytes_recovered":107806720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,2],"avail_no_missing":["3","0","2","4"],"object_location_counts":[{"shards":"0,2,3","objects":2529},{"shards":"2,3","objects":314}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":163,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.181397+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:21:20.181397+0000","last_peered":"2024-08-17T17:21:20.181397+0000","last_clean":"2024-08-17T17:21:20.181397+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:21:20.181397+0000","last_undegraded":"2024-08-17T17:21:20.181397+0000","last_fullsized":"2024-08-17T17:21:20.181397+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":640,"reported_epoch":146,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:21:20.176922+0000","last_change":"2024-08-17T17:20:50.272583+0000","last_active":"2024-08-17T17:21:20.176922+0000","last_peered":"2024-08-17T17:21:20.176922+0000","last_clean":"2024-08-17T17:20:44.742736+0000","last_became_active":"2024-08-17T17:20:50.271982+0000","last_became_peered":"2024-08-17T17:20:50.271982+0000","last_unstale":"2024-08-17T17:21:20.176922+0000","last_undegraded":"2024-08-17T17:20:50.261857+0000","last_fullsized":"2024-08-17T17:20:50.254164+0000","mapping_epoch":141,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6],"avail_no_missing":["1","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":152,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.180869+0000","last_change":"2024-08-17T17:19:57.520194+0000","last_active":"2024-08-17T17:21:20.180869+0000","last_peered":"2024-08-17T17:21:20.180869+0000","last_clean":"2024-08-17T17:21:20.180869+0000","last_became_active":"2024-08-17T17:19:57.519559+0000","last_became_peered":"2024-08-17T17:19:57.519559+0000","last_unstale":"2024-08-17T17:21:20.180869+0000","last_undegraded":"2024-08-17T17:21:20.180869+0000","last_fullsized":"2024-08-17T17:21:20.180869+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:48:33.839524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"146'8163","reported_seq":8248,"reported_epoch":146,"state":"active+recovery_wait+remapped","last_fresh":"2024-08-17T17:21:25.690876+0000","last_change":"2024-08-17T17:20:50.427272+0000","last_active":"2024-08-17T17:21:25.690876+0000","last_peered":"2024-08-17T17:21:25.690876+0000","last_clean":"2024-08-17T17:20:48.235622+0000","last_became_active":"2024-08-17T17:20:50.384650+0000","last_became_peered":"2024-08-17T17:20:50.384650+0000","last_unstale":"2024-08-17T17:21:25.690876+0000","last_undegraded":"2024-08-17T17:21:25.690876+0000","last_fullsized":"2024-08-17T17:21:25.690876+0000","mapping_epoch":141,"log_start":"123'2143","ondisk_log_start":"123'2143","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":6020,"log_dups_size":0,"ondisk_log_size":6020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300744704,"num_objects":4589,"num_object_clones":0,"num_object_copies":13767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4589,"num_objects_unfound":0,"num_objects_dirty":4589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8163,"num_write_kb":408064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,4],"avail_no_missing":["3","1","4"],"object_location_counts":[{"shards":"1,3,4","objects":4589}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":81,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.176597+0000","last_change":"2024-08-17T17:20:49.289009+0000","last_active":"2024-08-17T17:21:20.176597+0000","last_peered":"2024-08-17T17:21:20.176597+0000","last_clean":"2024-08-17T17:21:20.176597+0000","last_became_active":"2024-08-17T17:20:49.265734+0000","last_became_peered":"2024-08-17T17:20:49.265734+0000","last_unstale":"2024-08-17T17:21:20.176597+0000","last_undegraded":"2024-08-17T17:21:20.176597+0000","last_fullsized":"2024-08-17T17:21:20.176597+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:45:07.444277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":355,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-08-17T17:21:11.118491+0000","last_change":"2024-08-17T17:18:59.448779+0000","last_active":"2024-08-17T17:21:11.118491+0000","last_peered":"2024-08-17T17:21:11.118491+0000","last_clean":"2024-08-17T17:21:11.118491+0000","last_became_active":"2024-08-17T17:18:59.410280+0000","last_became_peered":"2024-08-17T17:18:59.410280+0000","last_unstale":"2024-08-17T17:21:11.118491+0000","last_undegraded":"2024-08-17T17:21:11.118491+0000","last_fullsized":"2024-08-17T17:21:11.118491+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:09:46.989594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":45,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-08-17T17:21:11.133719+0000","last_change":"2024-08-17T17:18:59.417183+0000","last_active":"2024-08-17T17:21:11.133719+0000","last_peered":"2024-08-17T17:21:11.133719+0000","last_clean":"2024-08-17T17:21:11.133719+0000","last_became_active":"2024-08-17T17:18:59.416843+0000","last_became_peered":"2024-08-17T17:18:59.416843+0000","last_unstale":"2024-08-17T17:21:11.133719+0000","last_undegraded":"2024-08-17T17:21:11.133719+0000","last_fullsized":"2024-08-17T17:21:11.133719+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:29:09.336335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":66,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-08-17T17:21:11.123218+0000","last_change":"2024-08-17T17:20:49.258048+0000","last_active":"2024-08-17T17:21:11.123218+0000","last_peered":"2024-08-17T17:21:11.123218+0000","last_clean":"2024-08-17T17:21:11.123218+0000","last_became_active":"2024-08-17T17:20:49.257672+0000","last_became_peered":"2024-08-17T17:20:49.257672+0000","last_unstale":"2024-08-17T17:21:11.123218+0000","last_undegraded":"2024-08-17T17:21:11.123218+0000","last_fullsized":"2024-08-17T17:21:11.123218+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:08:11.734316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":105,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-08-17T17:21:11.113833+0000","last_change":"2024-08-17T17:18:59.430969+0000","last_active":"2024-08-17T17:21:11.113833+0000","last_peered":"2024-08-17T17:21:11.113833+0000","last_clean":"2024-08-17T17:21:11.113833+0000","last_became_active":"2024-08-17T17:18:59.413475+0000","last_became_peered":"2024-08-17T17:18:59.413475+0000","last_unstale":"2024-08-17T17:21:11.113833+0000","last_undegraded":"2024-08-17T17:21:11.113833+0000","last_fullsized":"2024-08-17T17:21:11.113833+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:36:33.732645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":141,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.177877+0000","last_change":"2024-08-17T17:19:57.512029+0000","last_active":"2024-08-17T17:21:20.177877+0000","last_peered":"2024-08-17T17:21:20.177877+0000","last_clean":"2024-08-17T17:21:20.177877+0000","last_became_active":"2024-08-17T17:19:57.511671+0000","last_became_peered":"2024-08-17T17:19:57.511671+0000","last_unstale":"2024-08-17T17:21:20.177877+0000","last_undegraded":"2024-08-17T17:21:20.177877+0000","last_fullsized":"2024-08-17T17:21:20.177877+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:59:30.997886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":48,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-08-17T17:21:11.113531+0000","last_change":"2024-08-17T17:20:49.279458+0000","last_active":"2024-08-17T17:21:11.113531+0000","last_peered":"2024-08-17T17:21:11.113531+0000","last_clean":"2024-08-17T17:21:11.113531+0000","last_became_active":"2024-08-17T17:20:49.260790+0000","last_became_peered":"2024-08-17T17:20:49.260790+0000","last_unstale":"2024-08-17T17:21:11.113531+0000","last_undegraded":"2024-08-17T17:21:11.113531+0000","last_fullsized":"2024-08-17T17:21:11.113531+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:58.092945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":26,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-08-17T17:21:11.113886+0000","last_change":"2024-08-17T17:19:57.506117+0000","last_active":"2024-08-17T17:21:11.113886+0000","last_peered":"2024-08-17T17:21:11.113886+0000","last_clean":"2024-08-17T17:21:11.113886+0000","last_became_active":"2024-08-17T17:19:57.505063+0000","last_became_peered":"2024-08-17T17:19:57.505063+0000","last_unstale":"2024-08-17T17:21:11.113886+0000","last_undegraded":"2024-08-17T17:21:11.113886+0000","last_fullsized":"2024-08-17T17:21:11.113886+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:39:49.859925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":110,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.191752+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:21:20.191752+0000","last_peered":"2024-08-17T17:21:20.191752+0000","last_clean":"2024-08-17T17:21:20.191752+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:21:20.191752+0000","last_undegraded":"2024-08-17T17:21:20.191752+0000","last_fullsized":"2024-08-17T17:21:20.191752+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":138,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.191443+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:21:20.191443+0000","last_peered":"2024-08-17T17:21:20.191443+0000","last_clean":"2024-08-17T17:21:20.191443+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:21:20.191443+0000","last_undegraded":"2024-08-17T17:21:20.191443+0000","last_fullsized":"2024-08-17T17:21:20.191443+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":150,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.180769+0000","last_change":"2024-08-17T17:18:59.409548+0000","last_active":"2024-08-17T17:21:20.180769+0000","last_peered":"2024-08-17T17:21:20.180769+0000","last_clean":"2024-08-17T17:21:20.180769+0000","last_became_active":"2024-08-17T17:18:59.409161+0000","last_became_peered":"2024-08-17T17:18:59.409161+0000","last_unstale":"2024-08-17T17:21:20.180769+0000","last_undegraded":"2024-08-17T17:21:20.180769+0000","last_fullsized":"2024-08-17T17:21:20.180769+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:31:48.248795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":103,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-08-17T17:21:11.118421+0000","last_change":"2024-08-17T17:16:26.963696+0000","last_active":"2024-08-17T17:21:11.118421+0000","last_peered":"2024-08-17T17:21:11.118421+0000","last_clean":"2024-08-17T17:21:11.118421+0000","last_became_active":"2024-08-17T17:16:26.963280+0000","last_became_peered":"2024-08-17T17:16:26.963280+0000","last_unstale":"2024-08-17T17:21:11.118421+0000","last_undegraded":"2024-08-17T17:21:11.118421+0000","last_fullsized":"2024-08-17T17:21:11.118421+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:22:01.474840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":168,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.180953+0000","last_change":"2024-08-17T17:19:57.522181+0000","last_active":"2024-08-17T17:21:20.180953+0000","last_peered":"2024-08-17T17:21:20.180953+0000","last_clean":"2024-08-17T17:21:20.180953+0000","last_became_active":"2024-08-17T17:19:57.519621+0000","last_became_peered":"2024-08-17T17:19:57.519621+0000","last_unstale":"2024-08-17T17:21:20.180953+0000","last_undegraded":"2024-08-17T17:21:20.180953+0000","last_fullsized":"2024-08-17T17:21:20.180953+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:38:58.926689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-08-17T17:21:11.114143+0000","last_change":"2024-08-17T17:19:57.506328+0000","last_active":"2024-08-17T17:21:11.114143+0000","last_peered":"2024-08-17T17:21:11.114143+0000","last_clean":"2024-08-17T17:21:11.114143+0000","last_became_active":"2024-08-17T17:19:57.505305+0000","last_became_peered":"2024-08-17T17:19:57.505305+0000","last_unstale":"2024-08-17T17:21:11.114143+0000","last_undegraded":"2024-08-17T17:21:11.114143+0000","last_fullsized":"2024-08-17T17:21:11.114143+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:19:56.623573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":135,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.186071+0000","last_change":"2024-08-17T17:18:59.412463+0000","last_active":"2024-08-17T17:21:20.186071+0000","last_peered":"2024-08-17T17:21:20.186071+0000","last_clean":"2024-08-17T17:21:20.186071+0000","last_became_active":"2024-08-17T17:18:59.412134+0000","last_became_peered":"2024-08-17T17:18:59.412134+0000","last_unstale":"2024-08-17T17:21:20.186071+0000","last_undegraded":"2024-08-17T17:21:20.186071+0000","last_fullsized":"2024-08-17T17:21:20.186071+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:44.463153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":164,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.180785+0000","last_change":"2024-08-17T17:20:49.280657+0000","last_active":"2024-08-17T17:21:20.180785+0000","last_peered":"2024-08-17T17:21:20.180785+0000","last_clean":"2024-08-17T17:21:20.180785+0000","last_became_active":"2024-08-17T17:20:49.279617+0000","last_became_peered":"2024-08-17T17:20:49.279617+0000","last_unstale":"2024-08-17T17:21:20.180785+0000","last_undegraded":"2024-08-17T17:21:20.180785+0000","last_fullsized":"2024-08-17T17:21:20.180785+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:00:42.169630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":68,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.178156+0000","last_change":"2024-08-17T17:19:57.511196+0000","last_active":"2024-08-17T17:21:20.178156+0000","last_peered":"2024-08-17T17:21:20.178156+0000","last_clean":"2024-08-17T17:21:20.178156+0000","last_became_active":"2024-08-17T17:19:57.507497+0000","last_became_peered":"2024-08-17T17:19:57.507497+0000","last_unstale":"2024-08-17T17:21:20.178156+0000","last_undegraded":"2024-08-17T17:21:20.178156+0000","last_fullsized":"2024-08-17T17:21:20.178156+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:04:14.669068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":133,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.177275+0000","last_change":"2024-08-17T17:20:49.277439+0000","last_active":"2024-08-17T17:21:20.177275+0000","last_peered":"2024-08-17T17:21:20.177275+0000","last_clean":"2024-08-17T17:21:20.177275+0000","last_became_active":"2024-08-17T17:20:49.277150+0000","last_became_peered":"2024-08-17T17:20:49.277150+0000","last_unstale":"2024-08-17T17:21:20.177275+0000","last_undegraded":"2024-08-17T17:21:20.177275+0000","last_fullsized":"2024-08-17T17:21:20.177275+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:41:09.990039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":18,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-08-17T17:21:11.113766+0000","last_change":"2024-08-17T17:20:49.259891+0000","last_active":"2024-08-17T17:21:11.113766+0000","last_peered":"2024-08-17T17:21:11.113766+0000","last_clean":"2024-08-17T17:21:11.113766+0000","last_became_active":"2024-08-17T17:20:49.259400+0000","last_became_peered":"2024-08-17T17:20:49.259400+0000","last_unstale":"2024-08-17T17:21:11.113766+0000","last_undegraded":"2024-08-17T17:21:11.113766+0000","last_fullsized":"2024-08-17T17:21:11.113766+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:57:27.342244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":157,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.187114+0000","last_change":"2024-08-17T17:19:57.515293+0000","last_active":"2024-08-17T17:21:20.187114+0000","last_peered":"2024-08-17T17:21:20.187114+0000","last_clean":"2024-08-17T17:21:20.187114+0000","last_became_active":"2024-08-17T17:19:57.514809+0000","last_became_peered":"2024-08-17T17:19:57.514809+0000","last_unstale":"2024-08-17T17:21:20.187114+0000","last_undegraded":"2024-08-17T17:21:20.187114+0000","last_fullsized":"2024-08-17T17:21:20.187114+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:33:06.339296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":130,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-08-17T17:21:11.134025+0000","last_change":"2024-08-17T17:20:49.264033+0000","last_active":"2024-08-17T17:21:11.134025+0000","last_peered":"2024-08-17T17:21:11.134025+0000","last_clean":"2024-08-17T17:21:11.134025+0000","last_became_active":"2024-08-17T17:20:49.263684+0000","last_became_peered":"2024-08-17T17:20:49.263684+0000","last_unstale":"2024-08-17T17:21:11.134025+0000","last_undegraded":"2024-08-17T17:21:11.134025+0000","last_fullsized":"2024-08-17T17:21:11.134025+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:10:30.430767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":150,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.183684+0000","last_change":"2024-08-17T17:18:59.418434+0000","last_active":"2024-08-17T17:21:20.183684+0000","last_peered":"2024-08-17T17:21:20.183684+0000","last_clean":"2024-08-17T17:21:20.183684+0000","last_became_active":"2024-08-17T17:18:59.418043+0000","last_became_peered":"2024-08-17T17:18:59.418043+0000","last_unstale":"2024-08-17T17:21:20.183684+0000","last_undegraded":"2024-08-17T17:21:20.183684+0000","last_fullsized":"2024-08-17T17:21:20.183684+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:25:51.371820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":131,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-08-17T17:21:11.134109+0000","last_change":"2024-08-17T17:20:49.291167+0000","last_active":"2024-08-17T17:21:11.134109+0000","last_peered":"2024-08-17T17:21:11.134109+0000","last_clean":"2024-08-17T17:21:11.134109+0000","last_became_active":"2024-08-17T17:20:49.263750+0000","last_became_peered":"2024-08-17T17:20:49.263750+0000","last_unstale":"2024-08-17T17:21:11.134109+0000","last_undegraded":"2024-08-17T17:21:11.134109+0000","last_fullsized":"2024-08-17T17:21:11.134109+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:02.863246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":164,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.192110+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:21:20.192110+0000","last_peered":"2024-08-17T17:21:20.192110+0000","last_clean":"2024-08-17T17:21:20.192110+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:21:20.192110+0000","last_undegraded":"2024-08-17T17:21:20.192110+0000","last_fullsized":"2024-08-17T17:21:20.192110+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":156,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.182237+0000","last_change":"2024-08-17T17:18:59.413937+0000","last_active":"2024-08-17T17:21:20.182237+0000","last_peered":"2024-08-17T17:21:20.182237+0000","last_clean":"2024-08-17T17:21:20.182237+0000","last_became_active":"2024-08-17T17:18:59.413566+0000","last_became_peered":"2024-08-17T17:18:59.413566+0000","last_unstale":"2024-08-17T17:21:20.182237+0000","last_undegraded":"2024-08-17T17:21:20.182237+0000","last_fullsized":"2024-08-17T17:21:20.182237+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:17:39.877197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-08-17T17:21:20.177438+0000","last_change":"2024-08-17T17:20:49.276135+0000","last_active":"2024-08-17T17:21:20.177438+0000","last_peered":"2024-08-17T17:21:20.177438+0000","last_clean":"2024-08-17T17:21:20.177438+0000","last_became_active":"2024-08-17T17:20:49.275797+0000","last_became_peered":"2024-08-17T17:20:49.275797+0000","last_unstale":"2024-08-17T17:21:20.177438+0000","last_undegraded":"2024-08-17T17:21:20.177438+0000","last_fullsized":"2024-08-17T17:21:20.177438+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:38:46.858100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":58,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-08-17T17:21:11.134425+0000","last_change":"2024-08-17T17:19:57.526427+0000","last_active":"2024-08-17T17:21:11.134425+0000","last_peered":"2024-08-17T17:21:11.134425+0000","last_clean":"2024-08-17T17:21:11.134425+0000","last_became_active":"2024-08-17T17:19:57.506625+0000","last_became_peered":"2024-08-17T17:19:57.506625+0000","last_unstale":"2024-08-17T17:21:11.134425+0000","last_undegraded":"2024-08-17T17:21:11.134425+0000","last_fullsized":"2024-08-17T17:21:11.134425+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:25:21.357359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":570,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":4885905432,"num_objects":74554,"num_object_clones":0,"num_object_copies":223662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25533,"num_objects_misplaced":19020,"num_objects_unfound":0,"num_objects_dirty":74554,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129556,"num_write_kb":6531457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11660,"num_bytes_recovered":764084224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15816536064,"data_stored":15816523848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110984,"ondisk_log_size":110984,"up":48,"acting":40,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":5642,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":16702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17971,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":4194342,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12599296,"data_stored":12582988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":22,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234175,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1803000,"kb_used_data":1493168,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91942152,"statfs":{"total":95995035648,"available":94148763648,"internally_reserved":0,"allocated":1529004032,"data_stored":1528192186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,1,3,1,2,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495810,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1665712,"kb_used_data":1355880,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":92079440,"statfs":{"total":95995035648,"available":94289346560,"internally_reserved":0,"allocated":1388421120,"data_stored":1387617712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757447,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1830752,"kb_used_data":1520920,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91914400,"statfs":{"total":95995035648,"available":94120345600,"internally_reserved":0,"allocated":1557422080,"data_stored":1556629893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019085,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435313,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1458660,"kb_used_data":1350684,"kb_used_omap":4,"kb_used_meta":107963,"kb_avail":92286492,"statfs":{"total":95995035648,"available":94501367808,"internally_reserved":0,"allocated":1383100416,"data_stored":1382304595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":110554122},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":104,"seq":446676598839,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2743148,"kb_used_data":2639268,"kb_used_omap":7,"kb_used_meta":103864,"kb_avail":91002004,"statfs":{"total":95995035648,"available":93186052096,"internally_reserved":0,"allocated":2702610432,"data_stored":2701810499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":106356898},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762366,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2537572,"kb_used_data":2397596,"kb_used_omap":7,"kb_used_meta":139960,"kb_avail":91207580,"statfs":{"total":95995035648,"available":93396561920,"internally_reserved":0,"allocated":2455138304,"data_stored":2454342354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7945,"internal_metadata":143319287},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":94,"seq":403726925893,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3551144,"kb_used_data":3411168,"kb_used_omap":7,"kb_used_meta":139960,"kb_avail":90194008,"statfs":{"total":95995035648,"available":92358664192,"internally_reserved":0,"allocated":3493036032,"data_stored":3492236078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":143319361},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6241,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3470790656,"data_stored":3470786584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2436890624,"data_stored":2436890624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2682392576,"data_stored":2682388504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1371930624,"data_stored":1371930624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1409351680,"data_stored":1409351680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1543634944,"data_stored":1543634944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1376780288,"data_stored":1376780288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1524764672,"data_stored":1524760600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:21:31.131 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd pool set .mgr pgp_num 1 2024-08-17T17:21:31.215 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:31.246 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:31.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:31.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:31.348 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:31.371 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:31 smithi003 ceph-mon[103344]: pgmap v325: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 663 op/s; 25352/218877 objects degraded (11.583%); 18616/218877 objects misplaced (8.505%); 9.8 MiB/s, 157 objects/s recovering 2024-08-17T17:21:31.371 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:31 smithi003 ceph-mon[100615]: pgmap v325: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 663 op/s; 25352/218877 objects degraded (11.583%); 18616/218877 objects misplaced (8.505%); 9.8 MiB/s, 157 objects/s recovering 2024-08-17T17:21:31.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:31.409 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:31.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:31.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:31 smithi053 ceph-mon[89963]: pgmap v325: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 663 op/s; 25352/218877 objects degraded (11.583%); 18616/218877 objects misplaced (8.505%); 9.8 MiB/s, 157 objects/s recovering 2024-08-17T17:21:31.510 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:31.569 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:31.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:31.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:31.671 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:31.703 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:31.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:31.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:31.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:31.836 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:21:31.857 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:31.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:31.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:31.873 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:21:31.959 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:32.009 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:32.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:32.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:32.111 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:32.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:32.151 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:32.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:32.232 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-08-17T17:21:32.253 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:32.253 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:32 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3651989029' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:21:32.253 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:32 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:21:32.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:32 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3651989029' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:21:32.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:32 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:21:32.305 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:32.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:32.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:32.345 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:32 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3651989029' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:21:32.345 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:32 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:21:32.406 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:32.446 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:32.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:32.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:32.550 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:32.592 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:32.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:32.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:32.694 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:32.743 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:32.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:32.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:32.844 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:32.877 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:32.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:32.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:32.979 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:33.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:33.030 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:33.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:33.068 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:21:33.068 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:21:33.132 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:33.171 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:33.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:33.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:33.273 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:33.316 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:33.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:33.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:33.406 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:33 smithi003 ceph-mon[100615]: pgmap v326: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.4 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 623 op/s; 25159/215325 objects degraded (11.684%); 18014/215325 objects misplaced (8.366%); 9.5 MiB/s, 152 objects/s recovering 2024-08-17T17:21:33.406 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:33 smithi003 ceph-mon[100615]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:21:33.406 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:33 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:21:33.406 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:33 smithi003 ceph-mon[100615]: osdmap e147: 8 total, 8 up, 7 in 2024-08-17T17:21:33.407 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:33 smithi003 ceph-mon[103344]: pgmap v326: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.4 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 623 op/s; 25159/215325 objects degraded (11.684%); 18014/215325 objects misplaced (8.366%); 9.5 MiB/s, 152 objects/s recovering 2024-08-17T17:21:33.407 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:33 smithi003 ceph-mon[103344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:21:33.407 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:33 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:21:33.407 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:33 smithi003 ceph-mon[103344]: osdmap e147: 8 total, 8 up, 7 in 2024-08-17T17:21:33.419 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:33.450 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:33.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:33.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:33.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:33 smithi053 ceph-mon[89963]: pgmap v326: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.4 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 623 op/s; 25159/215325 objects degraded (11.684%); 18014/215325 objects misplaced (8.366%); 9.5 MiB/s, 152 objects/s recovering 2024-08-17T17:21:33.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:33 smithi053 ceph-mon[89963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:21:33.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:33 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:21:33.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:33 smithi053 ceph-mon[89963]: osdmap e147: 8 total, 8 up, 7 in 2024-08-17T17:21:33.552 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:33.592 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:33.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:33.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:33.694 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:33.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:33.733 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:33.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:33.834 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:33.894 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:33.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:33.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:33.996 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:34.020 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:34.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:34.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:34.122 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:34.155 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:34.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:34.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:34.257 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:34.290 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:34.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:34.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:34.392 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:34.430 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:34.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:34.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:34.531 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:34.558 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-17T17:21:34.570 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:34.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:34.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:34.672 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:34.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:34.706 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:34.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:34.807 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:34.838 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:34.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:34.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:34.940 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:34.977 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:34.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:34.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:34.993 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:21:35.080 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:35.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:35.123 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:35.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:35.224 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:35.261 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:35.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:35.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:35.353 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:35 smithi003 ceph-mon[103344]: pgmap v328: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 652 op/s; 25163/211110 objects degraded (11.919%); 17818/211110 objects misplaced (8.440%); 10 MiB/s, 159 objects/s recovering 2024-08-17T17:21:35.354 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:35 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 25163/211110 objects degraded (11.919%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:35.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:35 smithi003 ceph-mon[100615]: pgmap v328: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 652 op/s; 25163/211110 objects degraded (11.919%); 17818/211110 objects misplaced (8.440%); 10 MiB/s, 159 objects/s recovering 2024-08-17T17:21:35.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:35 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 25163/211110 objects degraded (11.919%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:35.363 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:35.374 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:35 smithi053 ceph-mon[89963]: pgmap v328: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 652 op/s; 25163/211110 objects degraded (11.919%); 17818/211110 objects misplaced (8.440%); 10 MiB/s, 159 objects/s recovering 2024-08-17T17:21:35.374 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:35 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 25163/211110 objects degraded (11.919%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:35.402 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:35.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:35.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:35.504 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:35.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:35.554 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:35.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:35.655 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:35.688 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:35.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:35.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:35.790 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:35.832 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:35.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:35.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:35.934 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:35.970 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:35.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:35.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:36.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:21:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:21:35] "GET /metrics HTTP/1.1" 200 36604 "" "Prometheus/2.51.0" 2024-08-17T17:21:36.071 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:36.113 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:36.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:36.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:36.215 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:36.280 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:36.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:36.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:36.382 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:36.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:36.444 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:36.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:36.545 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:36.575 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:36.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:36.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:36.677 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:36.730 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:36.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:36.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:36.749 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:21:36.749 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:21:36.832 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:36.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:36.867 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:36.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:36.968 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:37.004 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:37.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:37.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:37.106 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:37.131 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:37.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:37.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:37.159 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:21:37.159 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:21:37.199 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:21:37.199 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:21:37.232 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:37.233 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:21:37.234 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:21:37.241 INFO:teuthology.orchestra.run.smithi003.stderr:set pool 1 pgp_num to 1 2024-08-17T17:21:37.266 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:37.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:37.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:37.281 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:21:37.281 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:21:37.325 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:37 smithi003 ceph-mon[103344]: pgmap v329: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 656 op/s; 24731/205386 objects degraded (12.041%); 17003/205386 objects misplaced (8.279%); 10 MiB/s, 160 objects/s recovering 2024-08-17T17:21:37.325 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:37 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1773018603' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-08-17T17:21:37.325 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:37 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2076865136' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:21:37.325 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:21:37.326 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:21:37.326 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:37 smithi003 ceph-mon[100615]: pgmap v329: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 656 op/s; 24731/205386 objects degraded (12.041%); 17003/205386 objects misplaced (8.279%); 10 MiB/s, 160 objects/s recovering 2024-08-17T17:21:37.326 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:37 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1773018603' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-08-17T17:21:37.326 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:37 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2076865136' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:21:37.363 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:21:37.363 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:21:37.368 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:37.391 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:37.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:37.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:37.395 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:21:37.395 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:21:37.396 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:21:37.407 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:37 smithi053 ceph-mon[89963]: pgmap v329: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 656 op/s; 24731/205386 objects degraded (12.041%); 17003/205386 objects misplaced (8.279%); 10 MiB/s, 160 objects/s recovering 2024-08-17T17:21:37.407 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:37 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1773018603' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} : dispatch 2024-08-17T17:21:37.407 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:37 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2076865136' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:21:37.430 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:21:37.431 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:21:37.442 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-17T17:21:37.493 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:37.559 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:37.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:37.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:37.660 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:37.689 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:37.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:37.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:37.791 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:37.844 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:37.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:37.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:37.852 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:21:37.946 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:37.975 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:37.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:37.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:38.076 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:38.104 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:38.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:38.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:38.206 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:38.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:38.263 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:38.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:38.364 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:38.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:38.416 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:38.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:38.517 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:38.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:38 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1773018603' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-08-17T17:21:38.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:38 smithi003 ceph-mon[103344]: osdmap e148: 8 total, 8 up, 7 in 2024-08-17T17:21:38.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:21:38.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:21:38.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:38 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1773018603' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-08-17T17:21:38.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:38 smithi003 ceph-mon[100615]: osdmap e148: 8 total, 8 up, 7 in 2024-08-17T17:21:38.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:21:38.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:21:38.551 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:38.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:38.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:38.653 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:38.695 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:38.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:38.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:38.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:38 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1773018603' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-08-17T17:21:38.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:38 smithi053 ceph-mon[89963]: osdmap e148: 8 total, 8 up, 7 in 2024-08-17T17:21:38.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:21:38.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:21:38.775 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:21:38.797 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:38.828 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:38.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:38.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:38.929 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:38.973 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:38.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:38.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:39.075 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:39.104 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:39.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:39.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:39.205 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:39.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:39.246 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:39.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:39.347 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:39.379 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:39.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:39.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:39.481 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:39.515 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:39.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:39.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:39.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:39 smithi003 ceph-mon[103344]: pgmap v331: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 609 op/s; 24731/204240 objects degraded (12.109%); 17003/204240 objects misplaced (8.325%); 7.8 MiB/s, 124 objects/s recovering 2024-08-17T17:21:39.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:39 smithi003 ceph-mon[100615]: pgmap v331: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 609 op/s; 24731/204240 objects degraded (12.109%); 17003/204240 objects misplaced (8.325%); 7.8 MiB/s, 124 objects/s recovering 2024-08-17T17:21:39.568 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:39 smithi053 ceph-mon[89963]: pgmap v331: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 609 op/s; 24731/204240 objects degraded (12.109%); 17003/204240 objects misplaced (8.325%); 7.8 MiB/s, 124 objects/s recovering 2024-08-17T17:21:39.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:39.659 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:39.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:39.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:39.761 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:39.797 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:39.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:39.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:39.899 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:39.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:39.938 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:39.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:40.039 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:40.074 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:40.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:40.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:40.176 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:40.198 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:21:40.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:40.220 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:40.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:40.321 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:40.357 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:40.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:40.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:40.453 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:40 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 24731/204240 objects degraded (12.109%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:40.454 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:40 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 24731/204240 objects degraded (12.109%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:40.459 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:40.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:40 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 24731/204240 objects degraded (12.109%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:40.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:21:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:21:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:21:40.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:40.498 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:40.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:40.510 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:21:40.510 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2024-08-17T17:21:40.510 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.510 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:139704 5.9 5:9dd20ecd:::benchmark_data_smithi003_140793_object37647:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-08-17T17:21:40.510 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:21:40.450362+0000", 2024-08-17T17:21:40.510 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.057281876000000002, 2024-08-17T17:21:40.510 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.057313576999999997, 2024-08-17T17:21:40.510 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:21:40.510 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:21:40.511 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:21:40.511 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:21:40.511 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:21:40.511 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 139704 2024-08-17T17:21:40.511 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.511 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:21:40.511 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.511 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:21:40.511 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.450362+0000", 2024-08-17T17:21:40.511 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:21:40.511 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.511 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.511 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:21:40.511 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.450362+0000", 2024-08-17T17:21:40.512 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:21:40.512 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.512 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.512 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:21:40.512 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.450360+0000", 2024-08-17T17:21:40.512 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:21:40.512 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.512 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.512 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:21:40.512 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.450366+0000", 2024-08-17T17:21:40.512 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6169999999999998e-06 2024-08-17T17:21:40.512 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.512 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.512 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:21:40.513 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.450379+0000", 2024-08-17T17:21:40.513 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2612999999999999e-05 2024-08-17T17:21:40.513 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.513 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.513 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:21:40.513 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.450385+0000", 2024-08-17T17:21:40.513 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.7630000000000003e-06 2024-08-17T17:21:40.513 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.513 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.513 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:21:40.513 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.450441+0000", 2024-08-17T17:21:40.513 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5742000000000003e-05 2024-08-17T17:21:40.513 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.513 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.514 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for degraded object", 2024-08-17T17:21:40.514 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.450899+0000", 2024-08-17T17:21:40.514 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00045771199999999998 2024-08-17T17:21:40.514 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.514 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.514 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:21:40.514 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.462708+0000", 2024-08-17T17:21:40.514 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011808647 2024-08-17T17:21:40.514 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.514 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.514 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:21:40.514 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.462795+0000", 2024-08-17T17:21:40.514 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.7245999999999998e-05 2024-08-17T17:21:40.515 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.515 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.515 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:21:40.515 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.462940+0000", 2024-08-17T17:21:40.515 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014558099999999999 2024-08-17T17:21:40.515 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.515 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.515 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:21:40.515 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.468184+0000", 2024-08-17T17:21:40.515 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0052441579999999996 2024-08-17T17:21:40.515 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.515 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.515 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:21:40.515 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.471868+0000", 2024-08-17T17:21:40.516 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0036834139999999999 2024-08-17T17:21:40.516 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:21:40.516 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:21:40.516 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:21:40.516 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.516 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.516 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:139718 5.9 5:9c846c01:::benchmark_data_smithi003_140793_object37661:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-08-17T17:21:40.516 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:21:40.481479+0000", 2024-08-17T17:21:40.516 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.026164960000000001, 2024-08-17T17:21:40.516 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.026304389000000001, 2024-08-17T17:21:40.516 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:21:40.516 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:21:40.516 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:21:40.516 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:21:40.516 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:21:40.517 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 139718 2024-08-17T17:21:40.517 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.517 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:21:40.517 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.517 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:21:40.517 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.481479+0000", 2024-08-17T17:21:40.517 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:21:40.517 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.517 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.517 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:21:40.517 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.481479+0000", 2024-08-17T17:21:40.517 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:21:40.517 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.518 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.518 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:21:40.518 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.481477+0000", 2024-08-17T17:21:40.518 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:21:40.518 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.518 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.518 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:21:40.518 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.481487+0000", 2024-08-17T17:21:40.518 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0199e-05 2024-08-17T17:21:40.518 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.518 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.518 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:21:40.518 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.481512+0000", 2024-08-17T17:21:40.518 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5227000000000001e-05 2024-08-17T17:21:40.519 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.519 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.519 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:21:40.519 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.481519+0000", 2024-08-17T17:21:40.519 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.2520000000000002e-06 2024-08-17T17:21:40.519 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.519 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.519 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:21:40.519 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.481601+0000", 2024-08-17T17:21:40.519 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1478999999999997e-05 2024-08-17T17:21:40.519 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.519 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.519 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for degraded object", 2024-08-17T17:21:40.519 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.482334+0000", 2024-08-17T17:21:40.519 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00073301799999999999 2024-08-17T17:21:40.520 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.520 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.520 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:21:40.520 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.492721+0000", 2024-08-17T17:21:40.520 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010386819 2024-08-17T17:21:40.520 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.520 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.520 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:21:40.520 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.492804+0000", 2024-08-17T17:21:40.520 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.2836000000000004e-05 2024-08-17T17:21:40.520 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.520 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.520 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:21:40.520 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.493111+0000", 2024-08-17T17:21:40.520 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0003076 2024-08-17T17:21:40.521 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.521 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.521 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:21:40.521 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.498773+0000", 2024-08-17T17:21:40.521 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0056619979999999997 2024-08-17T17:21:40.521 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.521 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.521 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:21:40.521 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.498786+0000", 2024-08-17T17:21:40.521 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3281999999999999e-05 2024-08-17T17:21:40.521 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:21:40.521 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:21:40.521 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:21:40.521 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.521 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.522 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.35277.0:139729 5.b e148/130 5:d575f665:::benchmark_data_smithi003_140793_object37672:head v 148'8798, mlcod=0'0)", 2024-08-17T17:21:40.522 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:21:40.504014+0000", 2024-08-17T17:21:40.522 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.0036298889999999999, 2024-08-17T17:21:40.522 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0038627829999999998, 2024-08-17T17:21:40.522 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:21:40.522 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2024-08-17T17:21:40.522 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:21:40.522 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.522 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:21:40.522 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.504014+0000", 2024-08-17T17:21:40.522 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:21:40.523 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.523 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.523 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:21:40.523 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.504014+0000", 2024-08-17T17:21:40.523 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:21:40.523 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.523 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.523 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:21:40.523 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.504013+0000", 2024-08-17T17:21:40.523 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:21:40.523 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.524 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.524 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:21:40.524 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.504019+0000", 2024-08-17T17:21:40.524 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6949999999999997e-06 2024-08-17T17:21:40.524 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.524 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.524 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:21:40.524 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.504029+0000", 2024-08-17T17:21:40.525 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0135999999999999e-05 2024-08-17T17:21:40.525 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.525 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.525 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:21:40.525 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.504034+0000", 2024-08-17T17:21:40.525 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7099999999999998e-06 2024-08-17T17:21:40.525 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.526 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.526 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:21:40.526 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.504147+0000", 2024-08-17T17:21:40.526 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000112767 2024-08-17T17:21:40.526 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.526 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.526 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:21:40.526 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.504174+0000", 2024-08-17T17:21:40.526 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7441e-05 2024-08-17T17:21:40.526 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:21:40.526 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:21:40.526 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:21:40.526 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.526 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.527 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.35277.0:139725 5.5 e148/130 5:aff8a67b:::benchmark_data_smithi003_140793_object37668:head v 148'8635, mlcod=123'2152)", 2024-08-17T17:21:40.527 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:21:40.507426+0000", 2024-08-17T17:21:40.527 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.00021727500000000001, 2024-08-17T17:21:40.527 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00052205000000000003, 2024-08-17T17:21:40.527 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:21:40.527 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2024-08-17T17:21:40.527 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:21:40.528 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.528 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:21:40.528 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.507426+0000", 2024-08-17T17:21:40.528 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:21:40.528 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.528 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.528 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:21:40.528 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.507426+0000", 2024-08-17T17:21:40.528 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:21:40.528 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.528 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.529 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:21:40.529 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.507426+0000", 2024-08-17T17:21:40.529 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:21:40.529 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.529 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.529 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:21:40.529 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.507432+0000", 2024-08-17T17:21:40.529 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0700000000000003e-06 2024-08-17T17:21:40.530 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.530 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.530 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:21:40.530 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.507444+0000", 2024-08-17T17:21:40.530 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2067e-05 2024-08-17T17:21:40.530 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.530 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.530 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:21:40.530 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.507450+0000", 2024-08-17T17:21:40.531 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.6370000000000002e-06 2024-08-17T17:21:40.531 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.531 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.531 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:21:40.531 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.507530+0000", 2024-08-17T17:21:40.531 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0093999999999996e-05 2024-08-17T17:21:40.531 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.531 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.531 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:21:40.531 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.507555+0000", 2024-08-17T17:21:40.531 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4283000000000001e-05 2024-08-17T17:21:40.532 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:21:40.532 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:21:40.532 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:21:40.532 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.532 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.532 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.35277.0:139733 5.6 e148/130 5:60d0790a:::benchmark_data_smithi003_140793_object37676:head v 148'8797, mlcod=0'0)", 2024-08-17T17:21:40.532 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:21:40.507544+0000", 2024-08-17T17:21:40.533 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 9.9324e-05, 2024-08-17T17:21:40.533 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00045884799999999998, 2024-08-17T17:21:40.533 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:21:40.533 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2024-08-17T17:21:40.533 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:21:40.533 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.533 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:21:40.533 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.507544+0000", 2024-08-17T17:21:40.534 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:21:40.534 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.534 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.534 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:21:40.534 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.507544+0000", 2024-08-17T17:21:40.534 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:21:40.534 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.534 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.534 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:21:40.534 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.507544+0000", 2024-08-17T17:21:40.535 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:21:40.535 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.535 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.535 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:21:40.535 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.507551+0000", 2024-08-17T17:21:40.535 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0890000000000002e-06 2024-08-17T17:21:40.535 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.535 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.535 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:21:40.535 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.507561+0000", 2024-08-17T17:21:40.535 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0054000000000001e-05 2024-08-17T17:21:40.535 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.535 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.535 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:21:40.535 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.507566+0000", 2024-08-17T17:21:40.536 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1549999999999998e-06 2024-08-17T17:21:40.536 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.536 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.536 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:21:40.536 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.507616+0000", 2024-08-17T17:21:40.536 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9904000000000002e-05 2024-08-17T17:21:40.536 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.536 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.536 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:21:40.536 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.507636+0000", 2024-08-17T17:21:40.536 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0682e-05 2024-08-17T17:21:40.536 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:21:40.536 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:21:40.536 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:21:40.537 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.537 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.537 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.35277.0:139731 5.1 e148/130)", 2024-08-17T17:21:40.537 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:21:40.507970+0000", 2024-08-17T17:21:40.537 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4294967295.9996738, 2024-08-17T17:21:40.537 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.7867000000000004e-05, 2024-08-17T17:21:40.537 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:21:40.537 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2024-08-17T17:21:40.537 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:21:40.538 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.538 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:21:40.538 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.507970+0000", 2024-08-17T17:21:40.538 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:21:40.538 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.538 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.538 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:21:40.538 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.507970+0000", 2024-08-17T17:21:40.538 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:21:40.538 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.539 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.539 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:21:40.539 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.507969+0000", 2024-08-17T17:21:40.539 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:21:40.539 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.539 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.539 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:21:40.539 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.507975+0000", 2024-08-17T17:21:40.539 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.976e-06 2024-08-17T17:21:40.539 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.539 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.539 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:21:40.540 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.507988+0000", 2024-08-17T17:21:40.540 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3113e-05 2024-08-17T17:21:40.540 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.540 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.540 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:21:40.540 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.507993+0000", 2024-08-17T17:21:40.540 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7520000000000004e-06 2024-08-17T17:21:40.540 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:21:40.540 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:21:40.540 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:21:40.540 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:40.508055+0000", 2024-08-17T17:21:40.540 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2528999999999997e-05 2024-08-17T17:21:40.540 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:21:40.540 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:21:40.540 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:21:40.541 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:21:40.541 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-08-17T17:21:40.541 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 6 2024-08-17T17:21:40.541 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:21:40.599 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:40.626 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:40.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:40.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:40.729 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:40.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:40.761 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:40.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:40.818 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:21:40.862 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:40.929 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:40.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:40.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:41.030 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:41.097 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:41.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:41.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:41.200 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:41.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:41.252 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:41.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:41.305 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-08-17T17:21:41.354 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:41.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:41.392 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:41.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:41.406 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-17T17:21:41.494 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:41.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:41 smithi003 ceph-mon[103344]: pgmap v332: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 658 op/s; 24539/199509 objects degraded (12.300%); 16606/199509 objects misplaced (8.323%); 11 MiB/s, 174 objects/s recovering 2024-08-17T17:21:41.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:41 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:21:41.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:41 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1395634485' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:21:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:41 smithi003 ceph-mon[100615]: pgmap v332: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 658 op/s; 24539/199509 objects degraded (12.300%); 16606/199509 objects misplaced (8.323%); 11 MiB/s, 174 objects/s recovering 2024-08-17T17:21:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:41 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:21:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:41 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1395634485' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:21:41.534 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:41.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:41.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:41.636 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:41.700 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:41.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:41.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:41.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:41 smithi053 ceph-mon[89963]: pgmap v332: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 658 op/s; 24539/199509 objects degraded (12.300%); 16606/199509 objects misplaced (8.323%); 11 MiB/s, 174 objects/s recovering 2024-08-17T17:21:41.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:41 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:21:41.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:41 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1395634485' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:21:41.802 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:41.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:41.825 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:41.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:41.926 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:41.960 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:41.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:41.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:42.062 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:42.095 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:42.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:42.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:42.197 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:42.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:42.243 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:42.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:42.344 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:42.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:42.388 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:42.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:42.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:42 smithi053 ceph-mon[89963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:21:42.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:42 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1395634485' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:21:42.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:42 smithi053 ceph-mon[89963]: osdmap e149: 8 total, 8 up, 7 in 2024-08-17T17:21:42.490 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:42.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:42.522 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:42.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:42.624 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:42.683 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:21:42.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:42.688 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:42.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:42.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:42 smithi003 ceph-mon[103344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:21:42.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:42 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1395634485' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:21:42.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:42 smithi003 ceph-mon[103344]: osdmap e149: 8 total, 8 up, 7 in 2024-08-17T17:21:42.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:42 smithi003 ceph-mon[100615]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:21:42.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:42 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1395634485' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:21:42.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:42 smithi003 ceph-mon[100615]: osdmap e149: 8 total, 8 up, 7 in 2024-08-17T17:21:42.789 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:42.826 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:42.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:42.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:42.928 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:43.010 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:43.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:43.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:43.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:43.141 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:43.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:43.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:43.243 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:43.271 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:43.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:43.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:43.370 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:43 smithi003 ceph-mon[100615]: pgmap v334: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 631 op/s; 24349/195942 objects degraded (12.427%); 16002/195942 objects misplaced (8.167%); 11 MiB/s, 173 objects/s recovering 2024-08-17T17:21:43.370 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:43 smithi003 ceph-mon[103344]: pgmap v334: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 631 op/s; 24349/195942 objects degraded (12.427%); 16002/195942 objects misplaced (8.167%); 11 MiB/s, 173 objects/s recovering 2024-08-17T17:21:43.372 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:43.402 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:43.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:43.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:43.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:43 smithi053 ceph-mon[89963]: pgmap v334: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 631 op/s; 24349/195942 objects degraded (12.427%); 16002/195942 objects misplaced (8.167%); 11 MiB/s, 173 objects/s recovering 2024-08-17T17:21:43.504 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:43.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:43.558 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:43.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:43.566 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:21:43.567 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:21:43.605 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:21:43.606 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:21:43.646 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:21:43.646 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:21:43.659 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:43.689 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:21:43.689 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:21:43.691 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:43.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:43.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:43.696 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-17T17:21:43.696 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:21:43.696 INFO:tasks.thrashosds.thrasher:Reweighting osd 2 to 0.8326623301506796 2024-08-17T17:21:43.697 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 2 0.8326623301506796 2024-08-17T17:21:43.720 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:21:43.720 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:21:43.752 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:21:43.753 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:21:43.784 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:21:43.784 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:21:43.792 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:43.821 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:21:43.821 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:21:43.822 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:43.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:43.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:43.923 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:43.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:43.983 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:43.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:44.047 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:21:44.085 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:44.129 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:44.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:44.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:44.231 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:44.264 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:44.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:44.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:44.366 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:44.394 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:44.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:44.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:44.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:44 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1334103809' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:21:44.440 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:44 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1334103809' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:21:44.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:44 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1334103809' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:21:44.496 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:44.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:44.542 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:44.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:44.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:44.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:44.693 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:44.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:44.794 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:44.844 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:44.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:44.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:44.946 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:45.056 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:21:45.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:45.057 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:45.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:45.133 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:21:45.159 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:45.187 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:45.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:45.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:45.288 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:45.337 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:45.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:45.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:45.439 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:45.476 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:45.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:45.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:45.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:45 smithi003 ceph-mon[103344]: pgmap v335: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 3.9 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 569 op/s; 24370/191697 objects degraded (12.713%); 15797/191697 objects misplaced (8.241%); 7.5 MiB/s, 120 objects/s recovering 2024-08-17T17:21:45.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:45 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 24370/191697 objects degraded (12.713%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:45.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:45 smithi003 ceph-mon[100615]: pgmap v335: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 3.9 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 569 op/s; 24370/191697 objects degraded (12.713%); 15797/191697 objects misplaced (8.241%); 7.5 MiB/s, 120 objects/s recovering 2024-08-17T17:21:45.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:45 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 24370/191697 objects degraded (12.713%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:45.576 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:45 smithi053 ceph-mon[89963]: pgmap v335: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 3.9 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 569 op/s; 24370/191697 objects degraded (12.713%); 15797/191697 objects misplaced (8.241%); 7.5 MiB/s, 120 objects/s recovering 2024-08-17T17:21:45.577 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:45 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 24370/191697 objects degraded (12.713%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:45.578 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:45.609 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:45.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:45.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:45.711 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:45.750 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:45.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:45.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:45.840 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:21:45.840 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:21:45.852 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:45.911 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:45.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:45.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:45.996 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:21:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:21:45] "GET /metrics HTTP/1.1" 200 36605 "" "Prometheus/2.51.0" 2024-08-17T17:21:46.013 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:46.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:46.082 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:46.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:46.183 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:46.226 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:46.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:46.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:46.328 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:46.362 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:46.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:46.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:46.464 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:46.503 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:46.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:46.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:46.555 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-17T17:21:46.605 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:46.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:46.660 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:46.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:46.761 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:46.801 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:46.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:46.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:46.903 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:46.942 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:46.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:46.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:47.004 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:21:47.043 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:47.099 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:47.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:47.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:47.201 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:47.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:47.247 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:47.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:47.348 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:47.378 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:47.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:47.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:47.480 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:47.508 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:47.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:47.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:47.609 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:47.628 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:47 smithi003 ceph-mon[103344]: pgmap v336: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 748 op/s; 23955/186258 objects degraded (12.861%); 15002/186258 objects misplaced (8.054%); 12 MiB/s, 185 objects/s recovering 2024-08-17T17:21:47.629 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:47 smithi003 ceph-mon[100615]: pgmap v336: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 748 op/s; 23955/186258 objects degraded (12.861%); 15002/186258 objects misplaced (8.054%); 12 MiB/s, 185 objects/s recovering 2024-08-17T17:21:47.659 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:47.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:47.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:47.695 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:21:47.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:47 smithi053 ceph-mon[89963]: pgmap v336: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 748 op/s; 23955/186258 objects degraded (12.861%); 15002/186258 objects misplaced (8.054%); 12 MiB/s, 185 objects/s recovering 2024-08-17T17:21:47.762 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:47.790 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:47.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:47.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:47.892 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:47.936 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:47.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:47.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:48.038 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:48.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:48.081 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:48.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:48.182 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:48.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:48.227 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:48.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:48.328 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:48.368 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:48.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:48.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:48.470 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:48.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:48.529 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:48.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:48.630 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:48.661 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:48.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:48.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:48.729 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:21:48.762 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:48.811 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:48.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:48.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:48.913 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:48.951 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:48.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:48.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:49.054 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:49.083 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:49.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:49.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:49.185 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:49.217 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:49.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:49.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:49.319 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:49.339 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.2 to 0.832662 (d529) 2024-08-17T17:21:49.391 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:49.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:49.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:49.459 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:49 smithi003 ceph-mon[103344]: pgmap v337: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 639 op/s; 23955/185031 objects degraded (12.946%); 15002/185031 objects misplaced (8.108%); 9.9 MiB/s, 157 objects/s recovering 2024-08-17T17:21:49.460 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:49 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/4041319964' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.8326623301506796} : dispatch 2024-08-17T17:21:49.460 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:49 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.8326623301506796} : dispatch 2024-08-17T17:21:49.460 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:49 smithi003 ceph-mon[100615]: pgmap v337: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 639 op/s; 23955/185031 objects degraded (12.946%); 15002/185031 objects misplaced (8.108%); 9.9 MiB/s, 157 objects/s recovering 2024-08-17T17:21:49.461 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:49 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/4041319964' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.8326623301506796} : dispatch 2024-08-17T17:21:49.461 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:49 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.8326623301506796} : dispatch 2024-08-17T17:21:49.493 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:49.525 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:49.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:49.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:49.627 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:49.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:49.651 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:49.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:49.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:49 smithi053 ceph-mon[89963]: pgmap v337: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 639 op/s; 23955/185031 objects degraded (12.946%); 15002/185031 objects misplaced (8.108%); 9.9 MiB/s, 157 objects/s recovering 2024-08-17T17:21:49.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:49 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/4041319964' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.8326623301506796} : dispatch 2024-08-17T17:21:49.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:49 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 0.8326623301506796} : dispatch 2024-08-17T17:21:49.752 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:49.778 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:49.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:49.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:49.880 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:49.904 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:49.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:49.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:50.006 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:50.029 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:50.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:50.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:50.131 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:50.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:50.159 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:50.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:50.260 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:50.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:50.287 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:50.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:50.388 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:50.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:50.413 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:50.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:50.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:21:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:21:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:21:50.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:50 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.8326623301506796}]': finished 2024-08-17T17:21:50.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:50 smithi053 ceph-mon[89963]: osdmap e150: 8 total, 8 up, 7 in 2024-08-17T17:21:50.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:50 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 23955/185031 objects degraded (12.946%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:50.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:50 smithi053 ceph-mon[89963]: osdmap e151: 8 total, 8 up, 7 in 2024-08-17T17:21:50.514 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:50.537 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:50.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:50.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:50.639 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:50.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:50.679 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:50.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:50.766 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:50 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.8326623301506796}]': finished 2024-08-17T17:21:50.766 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:50 smithi003 ceph-mon[103344]: osdmap e150: 8 total, 8 up, 7 in 2024-08-17T17:21:50.766 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:50 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 23955/185031 objects degraded (12.946%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:50.766 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:50 smithi003 ceph-mon[103344]: osdmap e151: 8 total, 8 up, 7 in 2024-08-17T17:21:50.766 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:50 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 0.8326623301506796}]': finished 2024-08-17T17:21:50.767 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:50 smithi003 ceph-mon[100615]: osdmap e150: 8 total, 8 up, 7 in 2024-08-17T17:21:50.767 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:50 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 23955/185031 objects degraded (12.946%), 11 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:50.767 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:50 smithi003 ceph-mon[100615]: osdmap e151: 8 total, 8 up, 7 in 2024-08-17T17:21:50.780 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:50.809 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:50.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:50.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:50.833 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:21:50.833 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:21:50.911 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:50.943 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:50.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:50.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:51.044 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:51.089 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:51.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:51.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:51.191 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:51.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:51.238 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:51.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:51.339 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:51.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:51.383 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:51.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:51.483 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:51.514 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:51.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:51.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:51.545 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:51 smithi053 ceph-mon[89963]: pgmap v339: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 619 op/s; 23773/180216 objects degraded (13.191%); 14577/180216 objects misplaced (8.089%); 10 MiB/s, 166 objects/s recovering 2024-08-17T17:21:51.545 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:51 smithi053 ceph-mon[89963]: 5.0 starting backfill to osd.2 from (0'0,0'0] MAX to 149'9154 2024-08-17T17:21:51.584 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-17T17:21:51.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:51.637 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:51 smithi003 ceph-mon[100615]: pgmap v339: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 619 op/s; 23773/180216 objects degraded (13.191%); 14577/180216 objects misplaced (8.089%); 10 MiB/s, 166 objects/s recovering 2024-08-17T17:21:51.637 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:51 smithi003 ceph-mon[100615]: 5.0 starting backfill to osd.2 from (0'0,0'0] MAX to 149'9154 2024-08-17T17:21:51.638 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:51 smithi003 ceph-mon[103344]: pgmap v339: 89 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded, 2 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 75 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 619 op/s; 23773/180216 objects degraded (13.191%); 14577/180216 objects misplaced (8.089%); 10 MiB/s, 166 objects/s recovering 2024-08-17T17:21:51.638 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:51 smithi003 ceph-mon[103344]: 5.0 starting backfill to osd.2 from (0'0,0'0] MAX to 149'9154 2024-08-17T17:21:51.675 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:51.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:51.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:51.778 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:51.811 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:51.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:51.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:51.913 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:51.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:51.958 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:51.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:52.060 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:52.110 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:52.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:52.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:52.212 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:52.252 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:52.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:52.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:52.353 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:52.393 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:52.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:52.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:52.495 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:52.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:52.540 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:52.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:52.641 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:52.670 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:52.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:52.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:52.772 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:52.800 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:52.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:52.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:52.902 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:52.949 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:52.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:52.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:53.012 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:53 smithi003 ceph-mon[100615]: pgmap v341: 89 pgs: 1 peering, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 74 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 606 op/s; 22105/177126 objects degraded (12.480%); 15384/177126 objects misplaced (8.685%); 12 MiB/s, 195 objects/s recovering 2024-08-17T17:21:53.012 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:21:53.012 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:21:53.052 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:53.101 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:53.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:53.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:53.203 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:53.236 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:53.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:53.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:53.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:53 smithi003 ceph-mon[103344]: pgmap v341: 89 pgs: 1 peering, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 74 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 606 op/s; 22105/177126 objects degraded (12.480%); 15384/177126 objects misplaced (8.685%); 12 MiB/s, 195 objects/s recovering 2024-08-17T17:21:53.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:21:53.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:21:53.338 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:53.378 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:53.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:53.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:53.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:53 smithi053 ceph-mon[89963]: pgmap v341: 89 pgs: 1 peering, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 74 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 606 op/s; 22105/177126 objects degraded (12.480%); 15384/177126 objects misplaced (8.685%); 12 MiB/s, 195 objects/s recovering 2024-08-17T17:21:53.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:21:53.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:21:53.480 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:53.516 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:53.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:53.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:53.618 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:53.661 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:53.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:53.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:53.763 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:53.792 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:53.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:53.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:53.894 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:53.924 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:53.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:53.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:54.026 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:54.053 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:54.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:54.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:54.155 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:54.180 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:54.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:54.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:54.282 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:54.317 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:54.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:54.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:54.419 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:54.456 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:54.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:54.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:54.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:54 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:21:54.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:54 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:21:54.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:54 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:21:54.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:54 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:21:54.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:54 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:21:54.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:54 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:21:54.558 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:54.588 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:54.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:54.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:54.691 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:54.725 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:54.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:54.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:54.827 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:54.857 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:54.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:54.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:54.959 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:54.987 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:54.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:54.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:55.052 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:21:55.088 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:55.109 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-08-17T17:21:55.121 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:55.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:55.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:55.223 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:55.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:55.273 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:55.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:55.372 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:55 smithi003 ceph-mon[100615]: pgmap v342: 89 pgs: 1 peering, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 74 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 526 op/s; 22628/173619 objects degraded (13.033%); 15082/173619 objects misplaced (8.687%); 9.7 MiB/s, 155 objects/s recovering 2024-08-17T17:21:55.372 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:55 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1414834288' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:21:55.372 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:55 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 22628/173619 objects degraded (13.033%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:55.373 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:55 smithi003 ceph-mon[103344]: pgmap v342: 89 pgs: 1 peering, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 74 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 526 op/s; 22628/173619 objects degraded (13.033%); 15082/173619 objects misplaced (8.687%); 9.7 MiB/s, 155 objects/s recovering 2024-08-17T17:21:55.373 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:55 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1414834288' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:21:55.373 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:55 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 22628/173619 objects degraded (13.033%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:55.374 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:55.410 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:55.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:55.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:55.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:55 smithi053 ceph-mon[89963]: pgmap v342: 89 pgs: 1 peering, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 74 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 526 op/s; 22628/173619 objects degraded (13.033%); 15082/173619 objects misplaced (8.687%); 9.7 MiB/s, 155 objects/s recovering 2024-08-17T17:21:55.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:55 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1414834288' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:21:55.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:55 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 22628/173619 objects degraded (13.033%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-17T17:21:55.512 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:55.542 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:55.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:55.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:55.644 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:55.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:55.690 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:55.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:55.791 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:55.831 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:55.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:55.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:55.932 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:55.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:55.964 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:55.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:56.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:21:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:21:55] "GET /metrics HTTP/1.1" 200 36607 "" "Prometheus/2.51.0" 2024-08-17T17:21:56.065 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:56.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:56.106 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:56.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:56.207 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:56.238 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:56.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:56.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:56.340 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:56.376 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:56.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:56.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:56.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:56 smithi053 ceph-mon[89963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:21:56.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:56 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1414834288' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:21:56.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:56 smithi053 ceph-mon[89963]: osdmap e152: 8 total, 8 up, 7 in 2024-08-17T17:21:56.478 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:56.509 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:56.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:56.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:56.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:56 smithi003 ceph-mon[103344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:21:56.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:56 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1414834288' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:21:56.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:56 smithi003 ceph-mon[103344]: osdmap e152: 8 total, 8 up, 7 in 2024-08-17T17:21:56.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:56 smithi003 ceph-mon[100615]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:21:56.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:56 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1414834288' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:21:56.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:56 smithi003 ceph-mon[100615]: osdmap e152: 8 total, 8 up, 7 in 2024-08-17T17:21:56.611 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:56.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:56.637 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:56.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:56.738 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:56.769 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:56.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:56.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:56.871 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:56.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:56.894 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:56.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:56.995 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:57.027 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:57.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:57.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:57.047 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-17T17:21:57.048 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:21:57.048 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-08-17T17:21:57.048 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-08-17T17:21:57.129 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:21:57.164 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:57.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:57.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:21:57.266 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:57.301 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:57.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:57.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:57.402 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:57.452 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:57 smithi003 ceph-mon[100615]: pgmap v344: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 3.5 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 501 op/s; 22473/170526 objects degraded (13.179%); 14432/170526 objects misplaced (8.463%); 11 MiB/s, 168 objects/s recovering 2024-08-17T17:21:57.452 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:57 smithi003 ceph-mon[103344]: pgmap v344: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 3.5 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 501 op/s; 22473/170526 objects degraded (13.179%); 14432/170526 objects misplaced (8.463%); 11 MiB/s, 168 objects/s recovering 2024-08-17T17:21:57.462 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:57.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:57.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:57.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:57 smithi053 ceph-mon[89963]: pgmap v344: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 3.5 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 501 op/s; 22473/170526 objects degraded (13.179%); 14432/170526 objects misplaced (8.463%); 11 MiB/s, 168 objects/s recovering 2024-08-17T17:21:57.564 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:57.594 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:57.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:57.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:57.696 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:21:57.724 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:57.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:57.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:21:57.826 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:57.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:57.854 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:57.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:57.955 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:57.978 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:57.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:57.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:58.080 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:58.108 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:58.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:58.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:58.210 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:58.241 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:58.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:58.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:58.343 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:58.382 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:58.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:58.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:58.484 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:58.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:58.516 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:58.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:58.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:58.657 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:58.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:58.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:21:58.759 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:21:58.793 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:58.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:58.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:21:58.895 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:58.937 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:58.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:58.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:59.039 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:59.066 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:59.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:59.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:59.168 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:59.206 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:59.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:59.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:59.307 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:21:59.353 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:59.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:59.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:21:59.455 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:21:59.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:21:59 smithi053 ceph-mon[89963]: pgmap v345: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 531 op/s; 22473/167463 objects degraded (13.420%); 13995/167463 objects misplaced (8.357%); 10 MiB/s, 163 objects/s recovering 2024-08-17T17:21:59.486 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:21:59 smithi003 ceph-mon[100615]: pgmap v345: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 531 op/s; 22473/167463 objects degraded (13.420%); 13995/167463 objects misplaced (8.357%); 10 MiB/s, 163 objects/s recovering 2024-08-17T17:21:59.486 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:21:59 smithi003 ceph-mon[103344]: pgmap v345: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 531 op/s; 22473/167463 objects degraded (13.420%); 13995/167463 objects misplaced (8.357%); 10 MiB/s, 163 objects/s recovering 2024-08-17T17:21:59.490 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:59.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:59.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:21:59.592 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:21:59.624 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:21:59.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:59.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:21:59.725 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:21:59.763 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:59.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:59.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:21:59.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:21:59.903 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:21:59.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:21:59.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:00.005 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:00.037 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:00.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:00.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:00.138 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:00.168 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:00.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:00.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:00.270 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:00.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:00 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 22473/167463 objects degraded (13.420%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:00.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:00 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 22473/167463 objects degraded (13.420%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:00.310 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:00.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:00.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:00.412 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:00.456 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:00.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:00.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:00.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:00 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 22473/167463 objects degraded (13.420%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:00.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:22:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:22:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:22:00.558 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:00.590 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:00.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:00.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:00.692 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:00.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:00.737 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:00.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:00.838 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:00.872 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:00.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:00.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:00.974 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:01.022 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:01.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:01.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:01.124 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:01.171 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:01.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:01.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:01.273 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:01.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:01.324 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:01.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:01.398 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:01 smithi003 ceph-mon[100615]: pgmap v346: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 559 op/s; 22266/162696 objects degraded (13.686%); 13572/162696 objects misplaced (8.342%); 13 MiB/s, 205 objects/s recovering 2024-08-17T17:22:01.398 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:01 smithi003 ceph-mon[103344]: pgmap v346: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 559 op/s; 22266/162696 objects degraded (13.686%); 13572/162696 objects misplaced (8.342%); 13 MiB/s, 205 objects/s recovering 2024-08-17T17:22:01.424 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:01.450 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:01.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:01.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:01.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:01 smithi053 ceph-mon[89963]: pgmap v346: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 559 op/s; 22266/162696 objects degraded (13.686%); 13572/162696 objects misplaced (8.342%); 13 MiB/s, 205 objects/s recovering 2024-08-17T17:22:01.552 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:01.608 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:01.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:01.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:01.710 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:01.739 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:01.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:01.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:01.763 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:22:01.763 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:22:01.797 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:22:01.798 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:22:01.837 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:22:01.838 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:22:01.840 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:01.873 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:22:01.873 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:22:01.879 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:01.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:01.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:01.904 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:22:01.904 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:22:01.935 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:22:01.936 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:22:01.970 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:22:01.970 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:22:01.981 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:02.003 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:22:02.003 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:22:02.013 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:02.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:02.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:02.115 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:02.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:02.157 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:02.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:02.258 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:02.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:02 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1659405544' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:22:02.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:02 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1659405544' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:22:02.294 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:02.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:02.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:02.396 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:02.424 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:02.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:02.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:02.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:02 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1659405544' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:22:02.526 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:02.582 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:02.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:02.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:02.684 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:02.713 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:02.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:02.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:02.815 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:02.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:02.845 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:02.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:02.946 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:02.991 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:02.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:02.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:03.093 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:03.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:03.141 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:03.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:03.242 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:03.270 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:03.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:03.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:03.272 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:03 smithi003 ceph-mon[100615]: pgmap v347: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 600 op/s; 22266/159102 objects degraded (13.995%); 12702/159102 objects misplaced (7.984%); 12 MiB/s, 185 objects/s recovering 2024-08-17T17:22:03.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:03 smithi003 ceph-mon[103344]: pgmap v347: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 600 op/s; 22266/159102 objects degraded (13.995%); 12702/159102 objects misplaced (7.984%); 12 MiB/s, 185 objects/s recovering 2024-08-17T17:22:03.372 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:03.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:03.411 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:03.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:03.449 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:22:03.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:03 smithi053 ceph-mon[89963]: pgmap v347: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 600 op/s; 22266/159102 objects degraded (13.995%); 12702/159102 objects misplaced (7.984%); 12 MiB/s, 185 objects/s recovering 2024-08-17T17:22:03.512 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:03.553 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:03.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:03.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:03.655 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:03.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:03.708 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:03.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:03.809 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:03.862 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:03.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:03.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:03.964 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:03.995 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:03.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:03.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:04.097 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:04.132 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:04.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:04.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:04.233 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:04.277 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:04.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:04.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:04.379 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:04.415 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:04.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:04.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:04.517 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:04.558 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:04.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:04.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:04.660 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:04.699 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:04.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:04.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:04.801 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:04.829 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:04.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:04.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:04.921 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:22:04.921 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:22:04.921 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:04.921 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:154893 5.8 5:197b3d5f:::benchmark_data_smithi003_140793_object52836:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-08-17T17:22:04.921 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:22:04.756449+0000", 2024-08-17T17:22:04.922 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0077827790000000001, 2024-08-17T17:22:04.922 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0078168209999999998, 2024-08-17T17:22:04.922 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:22:04.922 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:22:04.922 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:22:04.922 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:22:04.922 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:22:04.922 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:22:04.922 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 154893 2024-08-17T17:22:04.922 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:04.922 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:22:04.922 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:04.923 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:22:04.923 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:04.756449+0000", 2024-08-17T17:22:04.923 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:22:04.923 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:04.923 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:04.923 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:22:04.923 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:04.756449+0000", 2024-08-17T17:22:04.923 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:22:04.923 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:04.923 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:04.923 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:22:04.923 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:04.756450+0000", 2024-08-17T17:22:04.923 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.313e-06 2024-08-17T17:22:04.923 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:04.924 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:04.924 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:22:04.924 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:04.756455+0000", 2024-08-17T17:22:04.924 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7520000000000004e-06 2024-08-17T17:22:04.924 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:04.924 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:04.924 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:22:04.924 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:04.756466+0000", 2024-08-17T17:22:04.924 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1029999999999999e-05 2024-08-17T17:22:04.924 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:04.924 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:04.924 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:22:04.924 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:04.756473+0000", 2024-08-17T17:22:04.924 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8399999999999997e-06 2024-08-17T17:22:04.925 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:04.925 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:04.925 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:22:04.925 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:04.756740+0000", 2024-08-17T17:22:04.925 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00026784999999999998 2024-08-17T17:22:04.925 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:04.925 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:04.925 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:22:04.925 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:04.756907+0000", 2024-08-17T17:22:04.925 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016710299999999999 2024-08-17T17:22:04.925 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:04.925 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:04.925 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:22:04.925 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:04.757044+0000", 2024-08-17T17:22:04.926 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013605 2024-08-17T17:22:04.926 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:22:04.926 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:22:04.926 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:22:04.926 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:04.926 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:04.926 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:154894 5.b 5:d353a844:::benchmark_data_smithi003_140793_object52837:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-08-17T17:22:04.926 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:22:04.756732+0000", 2024-08-17T17:22:04.926 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0074990730000000002, 2024-08-17T17:22:04.926 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0076056029999999998, 2024-08-17T17:22:04.926 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:22:04.926 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:22:04.926 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:22:04.927 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:22:04.927 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:22:04.927 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:22:04.927 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 154894 2024-08-17T17:22:04.927 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:04.927 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:22:04.927 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:04.927 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:22:04.927 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:04.756732+0000", 2024-08-17T17:22:04.927 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:22:04.927 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:04.927 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:04.927 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:22:04.927 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:04.756732+0000", 2024-08-17T17:22:04.928 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:22:04.928 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:04.928 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:04.928 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:22:04.928 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:04.756733+0000", 2024-08-17T17:22:04.928 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9100000000000003e-07 2024-08-17T17:22:04.928 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:04.928 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:04.928 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:22:04.928 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:04.756737+0000", 2024-08-17T17:22:04.928 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0629999999999999e-06 2024-08-17T17:22:04.928 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:04.928 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:04.928 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:22:04.929 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:04.756750+0000", 2024-08-17T17:22:04.929 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.322e-05 2024-08-17T17:22:04.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:04.929 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:04.929 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:22:04.929 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:04.756757+0000", 2024-08-17T17:22:04.929 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.041e-06 2024-08-17T17:22:04.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:04.929 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:04.929 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:22:04.929 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:04.756835+0000", 2024-08-17T17:22:04.929 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7490999999999994e-05 2024-08-17T17:22:04.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:04.929 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:04.929 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:22:04.930 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:04.760450+0000", 2024-08-17T17:22:04.930 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003615268 2024-08-17T17:22:04.930 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:04.930 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:04.930 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,7", 2024-08-17T17:22:04.930 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:04.760925+0000", 2024-08-17T17:22:04.930 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00047445400000000002 2024-08-17T17:22:04.930 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:22:04.930 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:22:04.930 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:22:04.930 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:04.930 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:04.930 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.35277.0:154893 5.8 e152/135)", 2024-08-17T17:22:04.930 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:22:04.763421+0000", 2024-08-17T17:22:04.930 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.00081061900000000005, 2024-08-17T17:22:04.931 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000975905, 2024-08-17T17:22:04.931 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:22:04.931 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:22:04.931 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-08-17T17:22:04.931 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:22:04.931 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:04.931 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:22:04.931 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:04.763421+0000", 2024-08-17T17:22:04.931 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:22:04.931 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:04.931 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:04.931 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:22:04.931 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:04.763421+0000", 2024-08-17T17:22:04.931 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:22:04.932 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:04.932 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:04.932 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:22:04.932 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:04.763422+0000", 2024-08-17T17:22:04.932 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4799999999999997e-07 2024-08-17T17:22:04.932 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:04.932 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:04.932 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:22:04.932 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:04.763429+0000", 2024-08-17T17:22:04.932 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1010000000000003e-06 2024-08-17T17:22:04.932 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:04.932 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:04.932 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:22:04.932 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:04.763444+0000", 2024-08-17T17:22:04.933 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4781e-05 2024-08-17T17:22:04.933 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:04.933 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:04.933 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:22:04.933 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:04.763451+0000", 2024-08-17T17:22:04.933 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6399999999999997e-06 2024-08-17T17:22:04.933 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:22:04.933 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:22:04.933 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:22:04.933 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:22:04.933 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-08-17T17:22:04.933 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 3 2024-08-17T17:22:04.933 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:22:04.934 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:04.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:04.976 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:04.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:05.077 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:05.116 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:05.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:05.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:05.218 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:05.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:05.256 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:05.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:05.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:05 smithi003 ceph-mon[103344]: pgmap v348: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 636 op/s; 22266/154521 objects degraded (14.410%); 12482/154521 objects misplaced (8.078%); 11 MiB/s, 177 objects/s recovering 2024-08-17T17:22:05.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:05 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 22266/154521 objects degraded (14.410%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:05.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:05 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:22:05.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:05 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:22:05.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:05 smithi003 ceph-mon[100615]: pgmap v348: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 636 op/s; 22266/154521 objects degraded (14.410%); 12482/154521 objects misplaced (8.078%); 11 MiB/s, 177 objects/s recovering 2024-08-17T17:22:05.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:05 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 22266/154521 objects degraded (14.410%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:05.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:05 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:22:05.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:05 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:22:05.357 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:05.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:05.392 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:05.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:05.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:05 smithi053 ceph-mon[89963]: pgmap v348: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 636 op/s; 22266/154521 objects degraded (14.410%); 12482/154521 objects misplaced (8.078%); 11 MiB/s, 177 objects/s recovering 2024-08-17T17:22:05.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:05 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 22266/154521 objects degraded (14.410%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:05.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:05 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:22:05.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:05 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:22:05.493 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:05.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:05.533 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:05.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:05.634 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:05.647 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-17T17:22:05.681 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:05.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:05.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:05.752 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:22:05.769 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:22:05.785 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:05.810 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:05.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:05.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:05.912 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:05.913 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:22:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:22:05] "GET /metrics HTTP/1.1" 200 36607 "" "Prometheus/2.51.0" 2024-08-17T17:22:05.939 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:05.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:05.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:05.992 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:22:06.041 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:06.065 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:06.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:06.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:06.167 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:06.198 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:06.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:06.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:06.299 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:06.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:06.349 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:06.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:06.450 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:06.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:06 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:22:06.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:06 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:22:06.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:06 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:22:06.469 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:06 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:22:06.469 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:06 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:22:06.469 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:06 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:22:06.469 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:06 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:22:06.469 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:06 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:22:06.469 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:06 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:22:06.487 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:06.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:06.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:06.589 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:06.621 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:06.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:06.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:06.723 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:06.770 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:06.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:06.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:06.872 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:06.919 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:06.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:06.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:07.021 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:07.052 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:07.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:07.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:07.154 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:07.194 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:07.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:07.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:07.243 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:07 smithi003 ceph-mon[103344]: pgmap v349: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 618 op/s; 22032/150732 objects degraded (14.617%); 11809/150732 objects misplaced (7.834%); 11 MiB/s, 170 objects/s recovering 2024-08-17T17:22:07.243 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:07 smithi003 ceph-mon[100615]: pgmap v349: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 618 op/s; 22032/150732 objects degraded (14.617%); 11809/150732 objects misplaced (7.834%); 11 MiB/s, 170 objects/s recovering 2024-08-17T17:22:07.296 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:07.331 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:07.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:07.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:07.433 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:07.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:07 smithi053 ceph-mon[89963]: pgmap v349: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 618 op/s; 22032/150732 objects degraded (14.617%); 11809/150732 objects misplaced (7.834%); 11 MiB/s, 170 objects/s recovering 2024-08-17T17:22:07.469 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:07.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:07.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:07.571 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:07.598 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:07.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:07.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:07.700 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:07.745 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:07.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:07.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:07.846 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:07.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:07.898 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:07.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:07.999 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:08.046 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:08.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:08.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:08.148 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:08.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:08.187 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:08.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:08.288 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:08.315 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:08.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:08.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:08.417 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:08.452 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:08.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:08.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:08.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:22:08.514 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:22:08.516 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:22:08.554 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:08.587 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:08.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:08.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:08.689 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:08.736 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:08.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:08.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:08.838 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:08.901 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:08.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:08.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:09.004 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:09.034 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:09.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:09.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:09.135 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:09.179 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:09.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:09.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:09.282 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:09.331 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:09.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:09.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:09.433 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:09.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:09 smithi003 ceph-mon[100615]: pgmap v350: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 3.0 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 637 op/s; 22032/147561 objects degraded (14.931%); 11366/147561 objects misplaced (7.703%); 11 MiB/s, 172 objects/s recovering 2024-08-17T17:22:09.440 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:09 smithi003 ceph-mon[103344]: pgmap v350: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 3.0 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 637 op/s; 22032/147561 objects degraded (14.931%); 11366/147561 objects misplaced (7.703%); 11 MiB/s, 172 objects/s recovering 2024-08-17T17:22:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:09 smithi053 ceph-mon[89963]: pgmap v350: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 3.0 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 637 op/s; 22032/147561 objects degraded (14.931%); 11366/147561 objects misplaced (7.703%); 11 MiB/s, 172 objects/s recovering 2024-08-17T17:22:09.486 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:09.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:09.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:09.588 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:09.636 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:09.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:09.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:09.738 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:09.783 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:09.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:09.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:09.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:09.926 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:09.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:09.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:10.029 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:10.055 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:10.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:10.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:10.156 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:10.200 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:10.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:10.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:10.226 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-08-17T17:22:10.302 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:10.326 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-08-17T17:22:10.332 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:10.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:10.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:10.434 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:10.459 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:10.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:10.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:10.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:10 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 22032/147561 objects degraded (14.931%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:10.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:10 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3773166245' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:22:10.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:10 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:22:10.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:22:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:22:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:22:10.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:10 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 22032/147561 objects degraded (14.931%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:10.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:10 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3773166245' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:22:10.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:10 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:22:10.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:10 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 22032/147561 objects degraded (14.931%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:10.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:10 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3773166245' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:22:10.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:10 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:22:10.561 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:10.592 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:10.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:10.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:10.694 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:10.726 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:10.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:10.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:10.828 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:10.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:10.889 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:10.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:10.991 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:11.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:11.025 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:11.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:11.076 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:22:11.084 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:22:11.092 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:22:11.126 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:11.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:11.159 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:11.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:11.260 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:11.294 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:11.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:11.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:11.401 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:11.445 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:11.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:11.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:11.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:11 smithi003 ceph-mon[103344]: pgmap v351: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 689 op/s; 21805/142608 objects degraded (15.290%); 10869/142608 objects misplaced (7.622%); 13 MiB/s, 214 objects/s recovering 2024-08-17T17:22:11.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:11 smithi003 ceph-mon[103344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:22:11.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:11 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:22:11.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:11 smithi003 ceph-mon[103344]: osdmap e153: 8 total, 8 up, 7 in 2024-08-17T17:22:11.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:11 smithi003 ceph-mon[100615]: pgmap v351: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 689 op/s; 21805/142608 objects degraded (15.290%); 10869/142608 objects misplaced (7.622%); 13 MiB/s, 214 objects/s recovering 2024-08-17T17:22:11.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:11 smithi003 ceph-mon[100615]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:22:11.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:11 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:22:11.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:11 smithi003 ceph-mon[100615]: osdmap e153: 8 total, 8 up, 7 in 2024-08-17T17:22:11.547 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:11.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:11.597 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:11.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:11.698 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:11.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:11 smithi053 ceph-mon[89963]: pgmap v351: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 689 op/s; 21805/142608 objects degraded (15.290%); 10869/142608 objects misplaced (7.622%); 13 MiB/s, 214 objects/s recovering 2024-08-17T17:22:11.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:11 smithi053 ceph-mon[89963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:22:11.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:11 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:22:11.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:11 smithi053 ceph-mon[89963]: osdmap e153: 8 total, 8 up, 7 in 2024-08-17T17:22:11.735 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:11.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:11.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:11.837 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:11.878 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:11.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:11.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:11.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:12.026 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:12.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:12.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:12.128 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:12.159 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:12.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:12.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:12.261 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:12.292 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:12.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:12.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:12.394 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:12.435 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:12.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:12.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:12.537 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:12.578 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:12.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:12.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:12.679 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:12.716 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:12.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:12.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:12.817 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:12.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:12.864 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:12.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:12.965 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:12.984 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:22:13.004 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:13.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:13.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:13.106 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:13.138 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:13.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:13.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:13.239 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:13.267 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:13.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:13.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:13.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:13 smithi003 ceph-mon[100615]: pgmap v353: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 2.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 669 op/s; 21805/138999 objects degraded (15.687%); 10062/138999 objects misplaced (7.239%); 12 MiB/s, 190 objects/s recovering 2024-08-17T17:22:13.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:13 smithi003 ceph-mon[103344]: pgmap v353: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 2.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 669 op/s; 21805/138999 objects degraded (15.687%); 10062/138999 objects misplaced (7.239%); 12 MiB/s, 190 objects/s recovering 2024-08-17T17:22:13.369 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:13.421 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:13.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:13.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:13.523 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:13.557 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:13.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:13.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:13.659 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:13.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:13.693 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:13.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:13.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:13 smithi053 ceph-mon[89963]: pgmap v353: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 2.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 669 op/s; 21805/138999 objects degraded (15.687%); 10062/138999 objects misplaced (7.239%); 12 MiB/s, 190 objects/s recovering 2024-08-17T17:22:13.794 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:13.828 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:13.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:13.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:13.930 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:13.964 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:13.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:13.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:14.066 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:14.115 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:14.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:14.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:14.216 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:14.276 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:14.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:14.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:14.378 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:14.409 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:14.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:14.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:14.511 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:14.546 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:14.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:14.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:14.647 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:14.685 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:14.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:14.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:14.787 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:14.822 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:14.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:14.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:14.924 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:14.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:14.975 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:14.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:15.076 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:15.121 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:15.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:15.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:15.223 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:15.272 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:15.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:15.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:15.374 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:15.409 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:15.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:15.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:15.481 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:15 smithi003 ceph-mon[100615]: pgmap v354: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 653 op/s; 21805/134907 objects degraded (16.163%); 9864/134907 objects misplaced (7.312%); 12 MiB/s, 188 objects/s recovering 2024-08-17T17:22:15.481 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:15 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 21805/134907 objects degraded (16.163%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:15.481 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:15 smithi003 ceph-mon[103344]: pgmap v354: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 653 op/s; 21805/134907 objects degraded (16.163%); 9864/134907 objects misplaced (7.312%); 12 MiB/s, 188 objects/s recovering 2024-08-17T17:22:15.481 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:15 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 21805/134907 objects degraded (16.163%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:15.511 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:15.549 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:15.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:15.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:15.598 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-08-17T17:22:15.651 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:15.711 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:15.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:15.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:15.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:15 smithi053 ceph-mon[89963]: pgmap v354: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 653 op/s; 21805/134907 objects degraded (16.163%); 9864/134907 objects misplaced (7.312%); 12 MiB/s, 188 objects/s recovering 2024-08-17T17:22:15.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:15 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 21805/134907 objects degraded (16.163%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:15.771 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:22:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:22:15] "GET /metrics HTTP/1.1" 200 36607 "" "Prometheus/2.51.0" 2024-08-17T17:22:15.814 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:15.838 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:15.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:15.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:15.863 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:22:15.863 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:22:15.896 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:22:15.896 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:22:15.937 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:22:15.937 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:22:15.940 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:15.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:15.969 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:15.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:15.972 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:22:15.972 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:22:16.002 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:22:16.002 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:22:16.037 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:22:16.037 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:22:16.069 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:16.072 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:22:16.073 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:22:16.101 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:16.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:16.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:16.107 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:22:16.107 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:22:16.203 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:16.257 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:16.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:16.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:16.342 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:16 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3754523195' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:22:16.342 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:16 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3754523195' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:22:16.359 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:16.431 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:16.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:16.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:16.532 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:16.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:16.580 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:16.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:16.681 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:16.713 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:16.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:16.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:16.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:16 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3754523195' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:22:16.815 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:16.863 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:16.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:16.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:16.964 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:16.984 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:22:16.995 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:16.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:16.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:17.097 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:17.145 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:17.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:17.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:17.247 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:17.289 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:17.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:17.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:17.390 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:17.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:17.423 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:17.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:17.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:17 smithi003 ceph-mon[103344]: pgmap v355: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 654 op/s; 21587/131076 objects degraded (16.469%); 9187/131076 objects misplaced (7.009%); 12 MiB/s, 187 objects/s recovering 2024-08-17T17:22:17.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:17 smithi003 ceph-mon[100615]: pgmap v355: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 654 op/s; 21587/131076 objects degraded (16.469%); 9187/131076 objects misplaced (7.009%); 12 MiB/s, 187 objects/s recovering 2024-08-17T17:22:17.524 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:17.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:17.559 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:17.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:17.660 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:17.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:17 smithi053 ceph-mon[89963]: pgmap v355: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 654 op/s; 21587/131076 objects degraded (16.469%); 9187/131076 objects misplaced (7.009%); 12 MiB/s, 187 objects/s recovering 2024-08-17T17:22:17.723 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:17.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:17.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:17.825 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:17.854 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:17.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:17.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:17.880 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:22:17.880 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:22:17.956 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:17.993 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:17.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:17.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:18.095 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:18.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:18.148 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:18.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:18.249 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:18.284 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:18.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:18.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:18.385 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:18.421 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:18.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:18.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:18.523 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:18.560 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:18.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:18.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:18.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-17T17:22:18.662 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:18.690 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:18.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:18.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:18.792 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:18.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:18.844 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:18.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:18.930 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:22:18.946 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:18.974 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:18.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:18.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:19.021 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:22:19.075 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:19.104 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:19.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:19.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:19.206 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:19.239 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:19.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:19.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:19.340 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:19.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:19.378 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:19.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:19.479 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:19.515 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:19.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:19.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:19.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:19 smithi003 ceph-mon[103344]: pgmap v356: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 657 op/s; 21587/127812 objects degraded (16.890%); 8675/127812 objects misplaced (6.787%); 12 MiB/s, 192 objects/s recovering 2024-08-17T17:22:19.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:19 smithi003 ceph-mon[100615]: pgmap v356: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 657 op/s; 21587/127812 objects degraded (16.890%); 8675/127812 objects misplaced (6.787%); 12 MiB/s, 192 objects/s recovering 2024-08-17T17:22:19.617 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:19.656 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:19.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:19.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:19.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:19 smithi053 ceph-mon[89963]: pgmap v356: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 657 op/s; 21587/127812 objects degraded (16.890%); 8675/127812 objects misplaced (6.787%); 12 MiB/s, 192 objects/s recovering 2024-08-17T17:22:19.759 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:19.790 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:19.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:19.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:19.892 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:19.930 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:19.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:19.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:20.032 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:20.071 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:20.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:20.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:20.172 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:20.210 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:20.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:20.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:20.312 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:20.342 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:20.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:20.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:20.443 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:20.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:22:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:22:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:22:20.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:20 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 21587/127812 objects degraded (16.890%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:20.479 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:20.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:20.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:20.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:20 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 21587/127812 objects degraded (16.890%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:20.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:20 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 21587/127812 objects degraded (16.890%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:20.581 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:20.622 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:20.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:20.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:20.724 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:20.773 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:20.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:20.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:20.875 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:20.975 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:20.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:20.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:20.986 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:22:21.077 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:21.112 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:21.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:21.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:21.213 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:21.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:21.275 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:21.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:21.376 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:21.425 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:21.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:21.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:21.474 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:21 smithi003 ceph-mon[103344]: pgmap v357: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 2.5 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 671 op/s; 21214/122433 objects degraded (17.327%); 8222/122433 objects misplaced (6.716%); 12 MiB/s, 202 objects/s recovering 2024-08-17T17:22:21.474 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:21 smithi003 ceph-mon[100615]: pgmap v357: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 2.5 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 671 op/s; 21214/122433 objects degraded (17.327%); 8222/122433 objects misplaced (6.716%); 12 MiB/s, 202 objects/s recovering 2024-08-17T17:22:21.527 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:21.558 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:21.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:21.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:21.660 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:21.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:21.700 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:21.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:21.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:21 smithi053 ceph-mon[89963]: pgmap v357: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 75 active+clean; 2.5 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 671 op/s; 21214/122433 objects degraded (17.327%); 8222/122433 objects misplaced (6.716%); 12 MiB/s, 202 objects/s recovering 2024-08-17T17:22:21.801 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:21.834 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:21.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:21.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:21.936 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:21.960 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:22:21.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:21.978 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:21.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:22.079 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:22.127 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:22.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:22.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:22.229 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:22.276 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:22.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:22.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:22.378 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:22.404 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:22.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:22.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:22.506 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:22.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:22.531 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:22.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:22.632 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:22.662 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:22.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:22.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:22.765 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:22.790 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:22.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:22.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:22.892 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:22.915 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:22.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:22.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:23.016 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:23.041 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:23.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:23.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:23.143 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:23.166 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:23.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:23.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:23.268 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:23.298 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:23.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:23.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:23.400 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:23.421 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:23.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:23.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:23.524 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:23.547 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:23 smithi003 ceph-mon[103344]: pgmap v358: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 2.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 638 op/s; 21214/118581 objects degraded (17.890%); 7538/118581 objects misplaced (6.357%); 10 MiB/s, 181 objects/s recovering 2024-08-17T17:22:23.547 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:23 smithi003 ceph-mon[103344]: osdmap e154: 8 total, 8 up, 7 in 2024-08-17T17:22:23.547 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:22:23.548 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:23 smithi003 ceph-mon[100615]: pgmap v358: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 2.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 638 op/s; 21214/118581 objects degraded (17.890%); 7538/118581 objects misplaced (6.357%); 10 MiB/s, 181 objects/s recovering 2024-08-17T17:22:23.548 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:23 smithi003 ceph-mon[100615]: osdmap e154: 8 total, 8 up, 7 in 2024-08-17T17:22:23.548 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:22:23.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:23.549 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:23.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:23.650 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:23.673 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:23.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:23.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:23 smithi053 ceph-mon[89963]: pgmap v358: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 2.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 638 op/s; 21214/118581 objects degraded (17.890%); 7538/118581 objects misplaced (6.357%); 10 MiB/s, 181 objects/s recovering 2024-08-17T17:22:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:23 smithi053 ceph-mon[89963]: osdmap e154: 8 total, 8 up, 7 in 2024-08-17T17:22:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:22:23.776 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:23.798 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:23.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:23.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:23.900 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:23.925 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:23.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:23.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:24.027 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:24.069 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:24.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:24.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:24.171 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:24.212 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:24.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:24.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:24.314 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:24.344 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:24.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:24.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:24.446 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:24.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:24.489 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:24.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:24.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:24 smithi003 ceph-mon[100615]: osdmap e155: 8 total, 8 up, 7 in 2024-08-17T17:22:24.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:24 smithi003 ceph-mon[103344]: osdmap e155: 8 total, 8 up, 7 in 2024-08-17T17:22:24.589 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:24.638 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:24.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:24.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:24.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:24 smithi053 ceph-mon[89963]: osdmap e155: 8 total, 8 up, 7 in 2024-08-17T17:22:24.740 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:24.790 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:24.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:24.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:24.892 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:24.922 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:24.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:24.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:25.024 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:25.064 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:25.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:25.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:25.166 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:25.212 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:25.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:25.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:25.314 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:25.340 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:25 smithi003 ceph-mon[100615]: pgmap v361: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 2.3 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 692 op/s; 20810/114438 objects degraded (18.185%); 7348/114438 objects misplaced (6.421%); 9.6 MiB/s, 223 objects/s recovering 2024-08-17T17:22:25.340 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:25 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 20810/114438 objects degraded (18.185%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:25.341 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:25 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1145737778' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:22:25.341 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:25 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:22:25.341 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:25 smithi003 ceph-mon[103344]: pgmap v361: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 2.3 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 692 op/s; 20810/114438 objects degraded (18.185%); 7348/114438 objects misplaced (6.421%); 9.6 MiB/s, 223 objects/s recovering 2024-08-17T17:22:25.341 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:25 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 20810/114438 objects degraded (18.185%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:25.341 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:25 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1145737778' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:22:25.341 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:25 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:22:25.347 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:25.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:25.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:25.448 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:25.481 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:25.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:25.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:25.583 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:25.614 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:25.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:25.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:25.715 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:25.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:25 smithi053 ceph-mon[89963]: pgmap v361: 89 pgs: 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 2.3 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 692 op/s; 20810/114438 objects degraded (18.185%); 7348/114438 objects misplaced (6.421%); 9.6 MiB/s, 223 objects/s recovering 2024-08-17T17:22:25.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:25 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 20810/114438 objects degraded (18.185%), 10 pgs degraded, 9 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:25.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:25 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1145737778' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:22:25.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:25 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:22:25.742 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:25.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:25.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:25.844 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:25.867 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:25.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:25.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:25.969 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:25.992 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:25.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:25.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:26.022 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:22:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:22:25] "GET /metrics HTTP/1.1" 200 36603 "" "Prometheus/2.51.0" 2024-08-17T17:22:26.093 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:26.116 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:26.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:26.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:26.218 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:26.242 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:26.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:26.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:26.344 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:26.370 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:26.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:26.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:26.472 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:26.513 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:26.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:26.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:26.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:26 smithi003 ceph-mon[103344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:22:26.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:26 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:22:26.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:26 smithi003 ceph-mon[103344]: osdmap e156: 8 total, 8 up, 7 in 2024-08-17T17:22:26.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:26 smithi003 ceph-mon[100615]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:22:26.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:26 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:22:26.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:26 smithi003 ceph-mon[100615]: osdmap e156: 8 total, 8 up, 7 in 2024-08-17T17:22:26.581 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-08-17T17:22:26.615 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:26.653 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:26.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:26.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:26.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:26 smithi053 ceph-mon[89963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:22:26.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:26 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:22:26.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:26 smithi053 ceph-mon[89963]: osdmap e156: 8 total, 8 up, 7 in 2024-08-17T17:22:26.755 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:26.791 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:26.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:26.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:26.893 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:26.930 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:26.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:26.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:27.032 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:27.067 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:27.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:27.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:27.169 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:27.215 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:27.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:27.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:27.316 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:27.342 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:27.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:27.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:27.444 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:27.484 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:27.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:27.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:27.522 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:27 smithi003 ceph-mon[100615]: pgmap v363: 89 pgs: 1 creating+activating+degraded, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 2.3 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 614 op/s; 20033/111360 objects degraded (17.989%); 6998/111360 objects misplaced (6.284%); 6.7 MiB/s, 239 objects/s recovering 2024-08-17T17:22:27.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:27 smithi003 ceph-mon[100615]: osdmap e157: 8 total, 8 up, 7 in 2024-08-17T17:22:27.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:27 smithi003 ceph-mon[103344]: pgmap v363: 89 pgs: 1 creating+activating+degraded, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 2.3 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 614 op/s; 20033/111360 objects degraded (17.989%); 6998/111360 objects misplaced (6.284%); 6.7 MiB/s, 239 objects/s recovering 2024-08-17T17:22:27.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:27 smithi003 ceph-mon[103344]: osdmap e157: 8 total, 8 up, 7 in 2024-08-17T17:22:27.587 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:27.626 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:27.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:27.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:27.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:27 smithi053 ceph-mon[89963]: pgmap v363: 89 pgs: 1 creating+activating+degraded, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 2.3 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 614 op/s; 20033/111360 objects degraded (17.989%); 6998/111360 objects misplaced (6.284%); 6.7 MiB/s, 239 objects/s recovering 2024-08-17T17:22:27.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:27 smithi053 ceph-mon[89963]: osdmap e157: 8 total, 8 up, 7 in 2024-08-17T17:22:27.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:27.768 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:27.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:27.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:27.869 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:27.895 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:27.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:27.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:27.997 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:28.028 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:28.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:28.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:28.129 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:28.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:28.167 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:28.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:28.268 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:28.309 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:28.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:28.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:28.410 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:28.441 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:28.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:28.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:28.543 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:28.583 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:28.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:28.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:28.685 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:28.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:28.724 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:28.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:28.825 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:28.854 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:28.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:28.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:28.956 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:28.985 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:28.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:28.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:29.087 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:29.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:29.125 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:29.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:29.226 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:29.267 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:29.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:29.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:29.369 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:29.427 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:29.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:29.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:29.475 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:29 smithi003 ceph-mon[100615]: pgmap v365: 89 pgs: 1 creating+activating+degraded, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 2.2 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 572 op/s; 20033/109098 objects degraded (18.362%); 6877/109098 objects misplaced (6.304%); 6.6 MiB/s, 248 objects/s recovering 2024-08-17T17:22:29.475 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:29 smithi003 ceph-mon[103344]: pgmap v365: 89 pgs: 1 creating+activating+degraded, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 2.2 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 572 op/s; 20033/109098 objects degraded (18.362%); 6877/109098 objects misplaced (6.304%); 6.6 MiB/s, 248 objects/s recovering 2024-08-17T17:22:29.529 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:29.557 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:29.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:29.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:29.634 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:22:29.634 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:22:29.658 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:29.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:29.701 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:29.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:29.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:29 smithi053 ceph-mon[89963]: pgmap v365: 89 pgs: 1 creating+activating+degraded, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 76 active+clean; 2.2 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 572 op/s; 20033/109098 objects degraded (18.362%); 6877/109098 objects misplaced (6.304%); 6.6 MiB/s, 248 objects/s recovering 2024-08-17T17:22:29.803 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:29.845 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:29.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:29.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:29.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:29.986 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:29.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:29.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:30.088 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:30.113 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:30.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:30.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:30.215 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:30.246 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:30.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:30.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:30.347 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:30.379 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:30.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:30.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:30.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:30 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 20033/109098 objects degraded (18.362%), 10 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:30.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:22:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:22:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:22:30.481 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:30.506 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:30.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:30.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:30.608 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:30.641 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:30.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:30.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:30.743 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:30.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:30 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 20033/109098 objects degraded (18.362%), 10 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:30.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:30 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 20033/109098 objects degraded (18.362%), 10 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:30.776 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:30.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:30.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:30.878 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:30.921 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:30.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:30.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:31.023 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:31.055 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:31.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:31.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:31.157 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:31.217 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:31.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:31.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:31.318 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:31.347 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:31.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:31.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:31.449 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:31.489 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:31.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:31.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:31.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:31 smithi003 ceph-mon[100615]: pgmap v366: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 2.1 GiB data, 9.3 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 648 op/s; 19463/104415 objects degraded (18.640%); 6877/104415 objects misplaced (6.586%); 8.9 MiB/s, 297 objects/s recovering 2024-08-17T17:22:31.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:31 smithi003 ceph-mon[103344]: pgmap v366: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 2.1 GiB data, 9.3 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 648 op/s; 19463/104415 objects degraded (18.640%); 6877/104415 objects misplaced (6.586%); 8.9 MiB/s, 297 objects/s recovering 2024-08-17T17:22:31.591 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:31.623 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:31.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:31.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:31.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:31 smithi053 ceph-mon[89963]: pgmap v366: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 2.1 GiB data, 9.3 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 648 op/s; 19463/104415 objects degraded (18.640%); 6877/104415 objects misplaced (6.586%); 8.9 MiB/s, 297 objects/s recovering 2024-08-17T17:22:31.726 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:31.759 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:31.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:31.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:31.861 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:31.901 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:31.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:31.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:32.003 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:32.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:32.035 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:32.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:32.040 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-08-17T17:22:32.136 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:32.182 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:32.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:32.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:32.284 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:32.345 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:32.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:32.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:32.446 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:32.501 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:32.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:32.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:32.603 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:32.638 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:32.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:32.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:32.739 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:32.778 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:32.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:32.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:32.880 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:32.898 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-17T17:22:32.936 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:32.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:32.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:33.038 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:33.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:33.126 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:33.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:33.152 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:22:33.152 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:22:33.183 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:22:33.200 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:22:33.200 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:22:33.228 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:33.236 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:22:33.236 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:22:33.275 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:22:33.275 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:22:33.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:33.294 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:33.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:33.307 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:22:33.307 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:22:33.355 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:22:33.355 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:22:33.382 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:22:33.383 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:22:33.396 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:33.416 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:33 smithi003 ceph-mon[100615]: pgmap v367: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 2.1 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 572 op/s; 18971/100695 objects degraded (18.840%); 6243/100695 objects misplaced (6.200%); 5.7 MiB/s, 229 objects/s recovering 2024-08-17T17:22:33.416 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:33 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/414163409' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:22:33.416 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:33 smithi003 ceph-mon[103344]: pgmap v367: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 2.1 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 572 op/s; 18971/100695 objects degraded (18.840%); 6243/100695 objects misplaced (6.200%); 5.7 MiB/s, 229 objects/s recovering 2024-08-17T17:22:33.416 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:33 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/414163409' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:22:33.417 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:22:33.417 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:22:33.454 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:33.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:33.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:33.556 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:33.602 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:33.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:33.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:33.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:22:33.704 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:33.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:33 smithi053 ceph-mon[89963]: pgmap v367: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 2.1 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 572 op/s; 18971/100695 objects degraded (18.840%); 6243/100695 objects misplaced (6.200%); 5.7 MiB/s, 229 objects/s recovering 2024-08-17T17:22:33.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:33 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/414163409' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:22:33.732 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:33.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:33.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:33.833 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:33.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:33.896 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:33.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:33.997 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:34.024 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:34.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:34.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:34.126 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:34.164 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:34.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:34.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:34.266 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:34.295 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:34.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:34.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:34.397 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:34.440 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:34.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:34.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:34.541 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:34.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:34.597 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:34.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:34.698 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:34.739 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:34.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:34.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:34.840 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:34.882 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:34.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:34.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:34.984 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:35.038 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:35.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:35.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:35.139 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:35.180 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:35.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:35.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:35.282 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:35.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:35.322 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:35.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:35.423 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:35.468 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:35.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:35.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:35.570 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:35.641 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:35.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:35.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:35.655 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:35 smithi003 ceph-mon[100615]: pgmap v368: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 2.0 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 609 op/s; 18749/95859 objects degraded (19.559%); 6243/95859 objects misplaced (6.513%); 2.5 MiB/s, 151 objects/s recovering 2024-08-17T17:22:35.655 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:35 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 18749/95859 objects degraded (19.559%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:35.655 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:22:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:22:35] "GET /metrics HTTP/1.1" 200 36603 "" "Prometheus/2.51.0" 2024-08-17T17:22:35.655 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:35 smithi003 ceph-mon[103344]: pgmap v368: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 2.0 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 609 op/s; 18749/95859 objects degraded (19.559%); 6243/95859 objects misplaced (6.513%); 2.5 MiB/s, 151 objects/s recovering 2024-08-17T17:22:35.655 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:35 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 18749/95859 objects degraded (19.559%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:35.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:35 smithi053 ceph-mon[89963]: pgmap v368: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 2.0 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 609 op/s; 18749/95859 objects degraded (19.559%); 6243/95859 objects misplaced (6.513%); 2.5 MiB/s, 151 objects/s recovering 2024-08-17T17:22:35.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:35 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 18749/95859 objects degraded (19.559%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:35.743 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:35.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:35.796 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:35.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:35.897 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:35.924 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:35.924 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:22:35.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:35.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:36.026 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:36.079 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:36.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:36.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:36.181 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:36.210 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:36.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:36.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:36.217 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:22:36.278 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:22:36.312 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:36.387 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:36.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:36.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:36.489 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:36.517 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:36.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:36.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:36.618 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:36.650 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:36.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:36.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:36.752 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:36.774 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:36.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:36.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:36.875 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:36.917 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:36.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:36.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:37.019 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:37.043 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:37.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:37.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:37.145 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:37.177 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:37.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:37.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:37.278 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:37.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:37.316 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:37.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:37.417 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:37.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:37.463 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:37.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:37.564 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:37.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:37.607 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:37.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:37.708 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:37.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:37 smithi053 ceph-mon[89963]: pgmap v369: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 1.9 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 650 op/s; 18335/91851 objects degraded (19.962%); 5787/91851 objects misplaced (6.300%); 2.2 MiB/s, 169 objects/s recovering 2024-08-17T17:22:37.746 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:37.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:37.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:37.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:37 smithi003 ceph-mon[100615]: pgmap v369: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 1.9 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 650 op/s; 18335/91851 objects degraded (19.962%); 5787/91851 objects misplaced (6.300%); 2.2 MiB/s, 169 objects/s recovering 2024-08-17T17:22:37.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:37 smithi003 ceph-mon[103344]: pgmap v369: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 1.9 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 650 op/s; 18335/91851 objects degraded (19.962%); 5787/91851 objects misplaced (6.300%); 2.2 MiB/s, 169 objects/s recovering 2024-08-17T17:22:37.848 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:37.891 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:37.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:37.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:37.993 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:38.033 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:38.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:38.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:38.135 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:38.170 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:38.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:38.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:38.272 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:38.312 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:38.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:38.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:38.414 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:38.461 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:38.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:38.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:38.563 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:38.586 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:38.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:38.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:38.688 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:38.715 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:38.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:38.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:38.817 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:38.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:38.849 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:38.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:38.950 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:38.979 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:38.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:38.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:39.081 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:39.130 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:39.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:39.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:39.232 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:39.268 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:39.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:39.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:39.370 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:39.412 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:39.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:39.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:39.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:39 smithi053 ceph-mon[89963]: pgmap v370: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 1.8 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 632 op/s; 18335/88227 objects degraded (20.782%); 5527/88227 objects misplaced (6.265%); 1.9 MiB/s, 147 objects/s recovering 2024-08-17T17:22:39.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:22:39.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:22:39.514 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:39.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:39 smithi003 ceph-mon[100615]: pgmap v370: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 1.8 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 632 op/s; 18335/88227 objects degraded (20.782%); 5527/88227 objects misplaced (6.265%); 1.9 MiB/s, 147 objects/s recovering 2024-08-17T17:22:39.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:22:39.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:22:39.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:39 smithi003 ceph-mon[103344]: pgmap v370: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 1.8 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 632 op/s; 18335/88227 objects degraded (20.782%); 5527/88227 objects misplaced (6.265%); 1.9 MiB/s, 147 objects/s recovering 2024-08-17T17:22:39.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:22:39.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:22:39.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:39.542 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:39.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:39.643 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:39.673 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:39.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:39.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:39.775 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:39.817 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:39.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:39.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:39.919 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:39.957 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:39.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:39.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:40.059 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:40.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:40.102 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:40.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:40.203 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:40.237 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:40.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:40.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:40.268 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:40 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 18335/88227 objects degraded (20.782%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:40.268 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:40 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 18335/88227 objects degraded (20.782%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:40.338 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:40.372 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:40.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:40.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:40.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:22:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:22:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:22:40.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:40 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 18335/88227 objects degraded (20.782%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:40.473 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:40.510 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:40.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:40.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:40.612 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:40.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:40.637 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:40.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:40.738 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:40.768 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:40.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:40.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:40.869 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:40.910 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:40.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:40.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:41.011 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:41.056 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:41.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:41.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:41.158 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:41.201 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:41.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:41.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:41.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:41 smithi003 ceph-mon[103344]: pgmap v371: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 1.7 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 730 op/s; 18131/82791 objects degraded (21.900%); 5527/82791 objects misplaced (6.676%); 1.8 MiB/s, 158 objects/s recovering 2024-08-17T17:22:41.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:41 smithi003 ceph-mon[100615]: pgmap v371: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 1.7 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 730 op/s; 18131/82791 objects degraded (21.900%); 5527/82791 objects misplaced (6.676%); 1.8 MiB/s, 158 objects/s recovering 2024-08-17T17:22:41.303 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:41.339 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:41.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:41.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:41.441 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:41.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:41 smithi053 ceph-mon[89963]: pgmap v371: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 1.7 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 730 op/s; 18131/82791 objects degraded (21.900%); 5527/82791 objects misplaced (6.676%); 1.8 MiB/s, 158 objects/s recovering 2024-08-17T17:22:41.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:41.471 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:41.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:41.572 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:41.589 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-08-17T17:22:41.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:41.623 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:41.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:41.725 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:41.752 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:41.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:41.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:41.853 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:41.912 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:41.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:41.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:42.014 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:42.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:42.060 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:42.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:42.161 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:42.186 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:42.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:42.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:42.288 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:42.338 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:42.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:42.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:42.440 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:42.471 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:42.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:42.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:42.573 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:42.614 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:42.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:42.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:42.716 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:42.747 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:42.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:42.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:42.848 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:42.862 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:22:42.875 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:42.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:42.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:42.917 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:22:42.977 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:43.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:43.023 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:43.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:43.123 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:43.162 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:43.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:43.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:43.264 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:43.295 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:43.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:43.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:43.396 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:43.412 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:43 smithi003 ceph-mon[100615]: pgmap v372: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 1.6 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 718 op/s; 17660/78555 objects degraded (22.481%); 4799/78555 objects misplaced (6.109%); 0 B/s, 150 objects/s recovering 2024-08-17T17:22:43.412 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:43 smithi003 ceph-mon[103344]: pgmap v372: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 1.6 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 718 op/s; 17660/78555 objects degraded (22.481%); 4799/78555 objects misplaced (6.109%); 0 B/s, 150 objects/s recovering 2024-08-17T17:22:43.425 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:43.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:43.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:43.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:43 smithi053 ceph-mon[89963]: pgmap v372: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 1.6 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 718 op/s; 17660/78555 objects degraded (22.481%); 4799/78555 objects misplaced (6.109%); 0 B/s, 150 objects/s recovering 2024-08-17T17:22:43.528 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:43.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:43.554 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:43.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:43.655 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:43.679 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:43.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:43.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:43.781 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:43.805 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:43.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:43.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:43.906 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:43.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:43.931 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:43.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:44.032 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:44.057 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:44.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:44.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:44.159 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:44.191 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:44.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:44.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:44.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:44 smithi003 ceph-mon[100615]: osdmap e158: 8 total, 8 up, 7 in 2024-08-17T17:22:44.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:44 smithi003 ceph-mon[103344]: osdmap e158: 8 total, 8 up, 7 in 2024-08-17T17:22:44.293 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:44.326 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:44.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:44.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:44.427 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:44.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:44 smithi053 ceph-mon[89963]: osdmap e158: 8 total, 8 up, 7 in 2024-08-17T17:22:44.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:44.477 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:44.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:44.578 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:44.630 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:44.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:44.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:44.732 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:44.767 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:44.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:44.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:44.868 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:44.918 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:44.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:44.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:45.020 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:45.062 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:45.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:45.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:45.146 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:45 smithi003 ceph-mon[100615]: pgmap v374: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 1.5 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 742 op/s; 17379/73587 objects degraded (23.617%); 4799/73587 objects misplaced (6.522%); 0 B/s, 136 objects/s recovering 2024-08-17T17:22:45.146 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:45 smithi003 ceph-mon[100615]: osdmap e159: 8 total, 8 up, 7 in 2024-08-17T17:22:45.146 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:45 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 17379/73587 objects degraded (23.617%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:45.146 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:45 smithi003 ceph-mon[103344]: pgmap v374: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 1.5 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 742 op/s; 17379/73587 objects degraded (23.617%); 4799/73587 objects misplaced (6.522%); 0 B/s, 136 objects/s recovering 2024-08-17T17:22:45.147 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:45 smithi003 ceph-mon[103344]: osdmap e159: 8 total, 8 up, 7 in 2024-08-17T17:22:45.147 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:45 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 17379/73587 objects degraded (23.617%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:45.164 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:45.192 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:45.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:45.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:45.294 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:45.330 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:45.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:45.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:45.432 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:45.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:45 smithi053 ceph-mon[89963]: pgmap v374: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 77 active+clean; 1.5 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 742 op/s; 17379/73587 objects degraded (23.617%); 4799/73587 objects misplaced (6.522%); 0 B/s, 136 objects/s recovering 2024-08-17T17:22:45.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:45 smithi053 ceph-mon[89963]: osdmap e159: 8 total, 8 up, 7 in 2024-08-17T17:22:45.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:45 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 17379/73587 objects degraded (23.617%), 8 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:45.477 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:22:45.477 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:22:45.477 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.477 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.35277.0:181995 5.8 e159/135 5:1e8008bb:::benchmark_data_smithi003_140793_object79938:head v 159'11414, mlcod=159'11411)", 2024-08-17T17:22:45.477 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:22:45.456030+0000", 2024-08-17T17:22:45.477 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.018567421000000001, 2024-08-17T17:22:45.477 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.018595831, 2024-08-17T17:22:45.477 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:22:45.477 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:22:45.477 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:22:45.478 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:22:45.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.478 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:22:45.478 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.456030+0000", 2024-08-17T17:22:45.478 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:22:45.478 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.478 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:22:45.478 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.456030+0000", 2024-08-17T17:22:45.478 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:22:45.478 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.478 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:22:45.479 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.456031+0000", 2024-08-17T17:22:45.479 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0270000000000001e-06 2024-08-17T17:22:45.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.479 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:22:45.479 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.456039+0000", 2024-08-17T17:22:45.479 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9330000000000001e-06 2024-08-17T17:22:45.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.479 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:22:45.479 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.456058+0000", 2024-08-17T17:22:45.479 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8318999999999999e-05 2024-08-17T17:22:45.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.480 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.480 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:22:45.480 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.456066+0000", 2024-08-17T17:22:45.480 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8510000000000005e-06 2024-08-17T17:22:45.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.480 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.480 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:22:45.480 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.456218+0000", 2024-08-17T17:22:45.480 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015197400000000001 2024-08-17T17:22:45.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.480 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.480 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:22:45.480 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.456262+0000", 2024-08-17T17:22:45.481 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3803000000000001e-05 2024-08-17T17:22:45.481 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.481 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:22:45.481 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.474152+0000", 2024-08-17T17:22:45.481 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017889882999999999 2024-08-17T17:22:45.481 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:22:45.481 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:22:45.481 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:22:45.481 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.482 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.35277.0:182005 5.8 e159/135 5:1d7ac5f5:::benchmark_data_smithi003_140793_object79948:head v 159'11415, mlcod=159'11411)", 2024-08-17T17:22:45.482 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:22:45.457884+0000", 2024-08-17T17:22:45.482 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.016713591, 2024-08-17T17:22:45.482 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.01680239, 2024-08-17T17:22:45.482 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:22:45.482 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:22:45.482 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:22:45.482 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:22:45.482 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.483 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:22:45.483 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.457884+0000", 2024-08-17T17:22:45.483 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:22:45.483 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.483 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:22:45.483 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.457884+0000", 2024-08-17T17:22:45.483 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:22:45.483 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.483 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:22:45.483 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.457885+0000", 2024-08-17T17:22:45.483 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0070000000000001e-06 2024-08-17T17:22:45.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.484 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.484 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:22:45.484 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.457892+0000", 2024-08-17T17:22:45.484 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6390000000000003e-06 2024-08-17T17:22:45.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.484 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.484 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:22:45.484 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.457909+0000", 2024-08-17T17:22:45.484 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7612e-05 2024-08-17T17:22:45.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.484 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.484 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:22:45.485 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.457919+0000", 2024-08-17T17:22:45.485 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4360000000000008e-06 2024-08-17T17:22:45.485 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.485 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.485 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:22:45.485 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.468208+0000", 2024-08-17T17:22:45.485 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.01028922 2024-08-17T17:22:45.485 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.485 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.485 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:22:45.485 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.468230+0000", 2024-08-17T17:22:45.485 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1605999999999999e-05 2024-08-17T17:22:45.485 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:22:45.485 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:22:45.486 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:22:45.486 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.486 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.486 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.35277.0:182004 5.b e159/130 5:d7da0fe6:::benchmark_data_smithi003_140793_object79947:head v 159'11503, mlcod=139'6593)", 2024-08-17T17:22:45.486 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:22:45.459105+0000", 2024-08-17T17:22:45.486 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.015492590000000001, 2024-08-17T17:22:45.486 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.015637614000000001, 2024-08-17T17:22:45.486 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:22:45.486 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:22:45.486 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:22:45.486 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:22:45.486 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.486 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:22:45.487 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.459105+0000", 2024-08-17T17:22:45.487 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:22:45.487 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.487 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.487 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:22:45.487 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.459105+0000", 2024-08-17T17:22:45.487 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:22:45.487 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.487 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.487 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:22:45.487 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.459106+0000", 2024-08-17T17:22:45.487 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.99e-07 2024-08-17T17:22:45.487 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.487 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.488 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:22:45.488 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.459112+0000", 2024-08-17T17:22:45.488 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2609999999999999e-06 2024-08-17T17:22:45.488 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.488 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.488 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:22:45.488 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.459125+0000", 2024-08-17T17:22:45.488 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3159000000000001e-05 2024-08-17T17:22:45.488 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.488 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.488 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:22:45.488 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.459134+0000", 2024-08-17T17:22:45.488 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0350000000000007e-06 2024-08-17T17:22:45.489 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.489 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.489 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:22:45.489 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.459819+0000", 2024-08-17T17:22:45.489 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00068516400000000004 2024-08-17T17:22:45.489 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.489 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.489 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:22:45.489 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.459843+0000", 2024-08-17T17:22:45.489 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3176999999999998e-05 2024-08-17T17:22:45.489 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:22:45.490 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:22:45.490 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:22:45.490 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.490 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.490 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.35277.0:182007 5.0 e159/150 5:0cc009ce:::benchmark_data_smithi003_140793_object79950:head v 159'11363, mlcod=159'11360)", 2024-08-17T17:22:45.490 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:22:45.467204+0000", 2024-08-17T17:22:45.490 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0073933769999999996, 2024-08-17T17:22:45.490 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0075875140000000001, 2024-08-17T17:22:45.490 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:22:45.490 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:22:45.491 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:22:45.491 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:22:45.491 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.491 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:22:45.491 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.467204+0000", 2024-08-17T17:22:45.491 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:22:45.491 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.491 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.491 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:22:45.491 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.467204+0000", 2024-08-17T17:22:45.491 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:22:45.491 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.491 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.492 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:22:45.492 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.467205+0000", 2024-08-17T17:22:45.492 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4400000000000001e-07 2024-08-17T17:22:45.492 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.492 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.492 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:22:45.492 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.467208+0000", 2024-08-17T17:22:45.492 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3239999999999999e-06 2024-08-17T17:22:45.492 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.492 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.492 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:22:45.493 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.467214+0000", 2024-08-17T17:22:45.493 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0000000000000002e-06 2024-08-17T17:22:45.493 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.493 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.493 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:22:45.494 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.467218+0000", 2024-08-17T17:22:45.494 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.33e-06 2024-08-17T17:22:45.494 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.494 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.494 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:22:45.494 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.467794+0000", 2024-08-17T17:22:45.494 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00057579300000000001 2024-08-17T17:22:45.494 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.494 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.495 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:22:45.495 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.467824+0000", 2024-08-17T17:22:45.495 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9543000000000002e-05 2024-08-17T17:22:45.495 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:22:45.495 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:22:45.495 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:22:45.495 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.495 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.495 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:182009 5.7 5:ee7e24e7:::benchmark_data_smithi003_140793_object79952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159)", 2024-08-17T17:22:45.495 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:22:45.470963+0000", 2024-08-17T17:22:45.495 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0036345240000000001, 2024-08-17T17:22:45.495 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038857359999999999, 2024-08-17T17:22:45.495 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:22:45.496 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:22:45.496 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:22:45.496 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:22:45.496 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:22:45.496 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:22:45.496 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 182009 2024-08-17T17:22:45.496 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.496 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:22:45.496 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.496 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:22:45.496 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.470963+0000", 2024-08-17T17:22:45.496 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:22:45.496 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.496 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.497 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:22:45.497 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.470963+0000", 2024-08-17T17:22:45.497 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:22:45.497 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.497 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.497 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:22:45.497 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.470965+0000", 2024-08-17T17:22:45.497 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.409e-06 2024-08-17T17:22:45.497 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.497 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.497 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:22:45.497 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.470970+0000", 2024-08-17T17:22:45.498 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2460000000000003e-06 2024-08-17T17:22:45.498 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.498 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.498 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:22:45.498 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.470981+0000", 2024-08-17T17:22:45.498 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1287999999999999e-05 2024-08-17T17:22:45.498 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.498 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.498 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:22:45.498 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.470988+0000", 2024-08-17T17:22:45.498 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4359999999999999e-06 2024-08-17T17:22:45.498 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.498 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.498 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:22:45.499 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.471168+0000", 2024-08-17T17:22:45.499 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000180442 2024-08-17T17:22:45.499 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.499 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.499 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:22:45.499 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.471288+0000", 2024-08-17T17:22:45.499 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000119487 2024-08-17T17:22:45.499 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.500 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.500 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4,7", 2024-08-17T17:22:45.500 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.471409+0000", 2024-08-17T17:22:45.500 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000121506 2024-08-17T17:22:45.500 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.500 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.500 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:22:45.500 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.474318+0000", 2024-08-17T17:22:45.500 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029093029999999998 2024-08-17T17:22:45.500 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:22:45.500 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:22:45.500 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:22:45.500 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.500 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.501 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:182011 5.c 5:32b451cc:::benchmark_data_smithi003_140793_object79954:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159)", 2024-08-17T17:22:45.501 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:22:45.471010+0000", 2024-08-17T17:22:45.501 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0035875600000000001, 2024-08-17T17:22:45.501 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039096560000000001, 2024-08-17T17:22:45.501 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:22:45.501 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:22:45.501 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:22:45.501 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:22:45.501 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:22:45.501 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:22:45.501 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 182011 2024-08-17T17:22:45.501 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.501 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:22:45.502 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.502 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:22:45.502 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.471010+0000", 2024-08-17T17:22:45.502 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:22:45.502 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.502 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.502 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:22:45.502 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.471010+0000", 2024-08-17T17:22:45.502 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:22:45.502 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.502 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.502 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:22:45.503 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.471011+0000", 2024-08-17T17:22:45.503 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9999999999999998e-07 2024-08-17T17:22:45.503 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.503 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.503 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:22:45.503 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.471013+0000", 2024-08-17T17:22:45.503 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.154e-06 2024-08-17T17:22:45.503 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.503 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.503 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:22:45.503 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.471019+0000", 2024-08-17T17:22:45.503 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0730000000000001e-06 2024-08-17T17:22:45.503 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.503 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.504 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:22:45.504 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.471024+0000", 2024-08-17T17:22:45.504 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6069999999999998e-06 2024-08-17T17:22:45.504 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.504 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.504 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:22:45.504 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.471130+0000", 2024-08-17T17:22:45.504 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000105451 2024-08-17T17:22:45.504 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.504 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.504 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:22:45.504 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.471268+0000", 2024-08-17T17:22:45.504 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013816699999999999 2024-08-17T17:22:45.504 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.505 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.505 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4,6", 2024-08-17T17:22:45.505 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.471406+0000", 2024-08-17T17:22:45.505 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013813200000000001 2024-08-17T17:22:45.505 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:22:45.505 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:22:45.505 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:22:45.505 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.505 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.505 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.35277.0:182010 5.3 e159/156 5:cab27f93:::benchmark_data_smithi003_140793_object79953:head v 159'11536, mlcod=159'11534)", 2024-08-17T17:22:45.505 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:22:45.471049+0000", 2024-08-17T17:22:45.505 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0035491289999999998, 2024-08-17T17:22:45.505 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039358489999999999, 2024-08-17T17:22:45.505 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:22:45.505 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:22:45.506 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:22:45.506 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:22:45.506 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.506 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:22:45.506 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.471049+0000", 2024-08-17T17:22:45.506 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:22:45.506 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.506 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.506 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:22:45.506 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.471049+0000", 2024-08-17T17:22:45.506 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:22:45.506 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.507 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.507 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:22:45.507 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.471049+0000", 2024-08-17T17:22:45.507 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7700000000000005e-07 2024-08-17T17:22:45.507 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.507 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.507 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:22:45.507 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.471054+0000", 2024-08-17T17:22:45.507 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1370000000000001e-06 2024-08-17T17:22:45.508 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.508 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.508 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:22:45.508 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.471061+0000", 2024-08-17T17:22:45.508 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8920000000000003e-06 2024-08-17T17:22:45.508 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.508 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.508 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:22:45.508 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.471066+0000", 2024-08-17T17:22:45.508 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7609999999999998e-06 2024-08-17T17:22:45.508 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.508 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.509 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:22:45.509 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.471611+0000", 2024-08-17T17:22:45.509 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00054525899999999998 2024-08-17T17:22:45.509 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.509 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.509 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:22:45.509 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.471629+0000", 2024-08-17T17:22:45.509 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8280000000000001e-05 2024-08-17T17:22:45.509 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:22:45.509 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:22:45.509 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:22:45.509 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.510 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.510 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.35277.0:182011 5.c e159/141)", 2024-08-17T17:22:45.510 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:22:45.474843+0000", 2024-08-17T17:22:45.510 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 4294967295.9997549, 2024-08-17T17:22:45.510 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000191757, 2024-08-17T17:22:45.510 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:22:45.510 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:22:45.510 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-08-17T17:22:45.510 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:22:45.510 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.510 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:22:45.510 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.474843+0000", 2024-08-17T17:22:45.510 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:22:45.511 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.511 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.511 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:22:45.511 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.474843+0000", 2024-08-17T17:22:45.511 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:22:45.511 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.511 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.511 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:22:45.511 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.474844+0000", 2024-08-17T17:22:45.511 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3769999999999999e-06 2024-08-17T17:22:45.511 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.511 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.511 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:22:45.512 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.474849+0000", 2024-08-17T17:22:45.512 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5700000000000003e-06 2024-08-17T17:22:45.512 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.512 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.512 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:22:45.512 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.474859+0000", 2024-08-17T17:22:45.512 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0611000000000001e-05 2024-08-17T17:22:45.512 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:22:45.512 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:22:45.512 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:22:45.512 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:45.474867+0000", 2024-08-17T17:22:45.512 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0700000000000007e-06 2024-08-17T17:22:45.512 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:22:45.513 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:22:45.513 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:22:45.513 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:22:45.513 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-08-17T17:22:45.513 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 8 2024-08-17T17:22:45.513 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:22:45.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:45.514 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:45.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:45.616 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:45.648 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:45.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:45.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:45.714 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:22:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:22:45] "GET /metrics HTTP/1.1" 200 36596 "" "Prometheus/2.51.0" 2024-08-17T17:22:45.750 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:45.790 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:45.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:45.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:45.892 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:45.921 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:45.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:45.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:46.023 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:46.050 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:46.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:46.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:46.135 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-08-17T17:22:46.152 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:46.202 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:46.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:46.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:46.304 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:46.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:46.346 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:46.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:46.447 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:46.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:46 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3055064124' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:22:46.472 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:46.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:46.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:46.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:46 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3055064124' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:22:46.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:46 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3055064124' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:22:46.574 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:46.616 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:46.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:46.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:46.718 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:46.747 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:46.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:46.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:46.849 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:46.882 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:46.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:46.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:46.984 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:47.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:47.036 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:47.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:47.137 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:47.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:47.173 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:47.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:47.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:47 smithi003 ceph-mon[103344]: pgmap v376: 89 pgs: 1 active+recovery_wait, 1 peering, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 1.4 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 763 op/s; 15150/69891 objects degraded (21.677%); 5809/69891 objects misplaced (8.312%); 0 B/s, 167 objects/s recovering 2024-08-17T17:22:47.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:47 smithi003 ceph-mon[103344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:22:47.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:47 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3055064124' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:22:47.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:47 smithi003 ceph-mon[103344]: osdmap e160: 8 total, 8 up, 7 in 2024-08-17T17:22:47.274 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:47.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:47 smithi003 ceph-mon[100615]: pgmap v376: 89 pgs: 1 active+recovery_wait, 1 peering, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 1.4 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 763 op/s; 15150/69891 objects degraded (21.677%); 5809/69891 objects misplaced (8.312%); 0 B/s, 167 objects/s recovering 2024-08-17T17:22:47.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:47 smithi003 ceph-mon[100615]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:22:47.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:47 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3055064124' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:22:47.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:47 smithi003 ceph-mon[100615]: osdmap e160: 8 total, 8 up, 7 in 2024-08-17T17:22:47.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:47.306 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:47.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:47.407 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:47.449 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:47.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:47.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:47.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:47 smithi053 ceph-mon[89963]: pgmap v376: 89 pgs: 1 active+recovery_wait, 1 peering, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 1.4 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 763 op/s; 15150/69891 objects degraded (21.677%); 5809/69891 objects misplaced (8.312%); 0 B/s, 167 objects/s recovering 2024-08-17T17:22:47.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:47 smithi053 ceph-mon[89963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:22:47.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:47 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3055064124' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:22:47.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:47 smithi053 ceph-mon[89963]: osdmap e160: 8 total, 8 up, 7 in 2024-08-17T17:22:47.552 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:47.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:47.586 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:47.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:47.687 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:47.714 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:47.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:47.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:47.816 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:47.850 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:47.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:47.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:47.952 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:48.006 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:48.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:48.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:48.108 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:48.138 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:48.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:48.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:48.240 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:48.280 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:48.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:48.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:48.382 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:48.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:48.427 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:48.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:48.528 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:48.576 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:48.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:48.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:48.678 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:48.725 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:48.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:48.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:48.827 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:48.857 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:48.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:48.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:48.959 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:48.989 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:48.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:48.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:49.047 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:22:49.055 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-17T17:22:49.091 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:49.118 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:49.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:49.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:49.220 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:49.269 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:49 smithi003 ceph-mon[103344]: pgmap v378: 89 pgs: 1 active+recovery_wait, 1 peering, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 1.4 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 575 op/s; 15150/68187 objects degraded (22.218%); 5627/68187 objects misplaced (8.252%); 0 B/s, 110 objects/s recovering 2024-08-17T17:22:49.270 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:49 smithi003 ceph-mon[100615]: pgmap v378: 89 pgs: 1 active+recovery_wait, 1 peering, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 1.4 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 575 op/s; 15150/68187 objects degraded (22.218%); 5627/68187 objects misplaced (8.252%); 0 B/s, 110 objects/s recovering 2024-08-17T17:22:49.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:49.306 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:49.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:49.407 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:49.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:49 smithi053 ceph-mon[89963]: pgmap v378: 89 pgs: 1 active+recovery_wait, 1 peering, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 78 active+clean; 1.4 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 575 op/s; 15150/68187 objects degraded (22.218%); 5627/68187 objects misplaced (8.252%); 0 B/s, 110 objects/s recovering 2024-08-17T17:22:49.496 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:22:49.496 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:22:49.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:49.503 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:49.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:49.540 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:22:49.540 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:22:49.583 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:22:49.583 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:22:49.605 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:49.626 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:22:49.626 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:22:49.633 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:49.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:49.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:49.656 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:22:49.656 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:22:49.699 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:22:49.700 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:22:49.735 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:49.736 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:22:49.736 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:22:49.761 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:49.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:49.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:49.779 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:22:49.779 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:22:49.863 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:49.920 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:49.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:49.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:50.023 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:50.052 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:50.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:50.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:50.154 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:50.178 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:50.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:50.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:50.280 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:50.322 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:50.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:50.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:50.424 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:50.452 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:50.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:50.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:50.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:50 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3837530376' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:22:50.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:50 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 15150/68187 objects degraded (22.218%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:50.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:22:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:22:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:22:50.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:50 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3837530376' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:22:50.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:50 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 15150/68187 objects degraded (22.218%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:50.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:50 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3837530376' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:22:50.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:50 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 15150/68187 objects degraded (22.218%), 6 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:50.554 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:50.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:50.594 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:50.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:50.695 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:50.734 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:50.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:50.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:50.835 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:50.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:50.866 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:50.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:50.967 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:51.003 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:51.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:51.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:51.104 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:51.133 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:51.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:51.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:51.173 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-08-17T17:22:51.235 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:51.262 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:51.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:51.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:51.364 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:51.406 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:51.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:51.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:51.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:51 smithi053 ceph-mon[89963]: pgmap v379: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 1.3 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 739 op/s; 14788/62229 objects degraded (23.764%); 5412/62229 objects misplaced (8.697%); 450 KiB/s, 209 objects/s recovering 2024-08-17T17:22:51.508 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:51.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:51 smithi003 ceph-mon[100615]: pgmap v379: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 1.3 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 739 op/s; 14788/62229 objects degraded (23.764%); 5412/62229 objects misplaced (8.697%); 450 KiB/s, 209 objects/s recovering 2024-08-17T17:22:51.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:51 smithi003 ceph-mon[103344]: pgmap v379: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 1.3 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 739 op/s; 14788/62229 objects degraded (23.764%); 5412/62229 objects misplaced (8.697%); 450 KiB/s, 209 objects/s recovering 2024-08-17T17:22:51.538 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:51.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:51.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:51.640 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:51.683 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:51.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:51.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:51.785 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:51.811 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:51.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:51.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:51.913 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:51.931 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:22:51.958 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:22:51.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:51.965 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:51.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:52.066 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:52.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:52.127 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:52.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:52.228 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:52.308 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:52.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:52.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:52.410 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:52.451 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:52.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:52.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:52.553 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:52.590 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:52.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:52.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:52.692 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:52.722 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:52.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:52.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:52.824 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:52.864 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:52.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:52.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:52.944 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:22:52.966 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:53.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:53.031 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:53.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:53.132 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:53.180 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:53.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:53.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:53.282 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:53.310 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:53.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:53.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:53.412 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:53.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:53 smithi053 ceph-mon[89963]: pgmap v380: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 1.2 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 642 op/s; 14788/58167 objects degraded (25.423%); 4754/58167 objects misplaced (8.173%); 376 KiB/s, 141 objects/s recovering 2024-08-17T17:22:53.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:22:53.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:22:53.469 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:53.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:53.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:53.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:53 smithi003 ceph-mon[103344]: pgmap v380: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 1.2 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 642 op/s; 14788/58167 objects degraded (25.423%); 4754/58167 objects misplaced (8.173%); 376 KiB/s, 141 objects/s recovering 2024-08-17T17:22:53.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:22:53.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:22:53.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:53 smithi003 ceph-mon[100615]: pgmap v380: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 1.2 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 642 op/s; 14788/58167 objects degraded (25.423%); 4754/58167 objects misplaced (8.173%); 376 KiB/s, 141 objects/s recovering 2024-08-17T17:22:53.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:22:53.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:22:53.571 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:53.610 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:53.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:53.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:53.712 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:53.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:53.761 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:53.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:53.829 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:22:53.862 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:53.891 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:53.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:53.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:53.993 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:54.019 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:54.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:54.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:54.121 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:54.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:54.171 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:54.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:54.272 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:54.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:54.329 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:54.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:54.430 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:54.482 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:54.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:54.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:54.584 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:54.609 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:54.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:54.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:54.711 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:54.743 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:54.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:54.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:54.844 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:54.886 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:54.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:54.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:54.988 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:55.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:55.019 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:55.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:55.120 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:55.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:55.157 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:55.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:55.258 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:55.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:55.328 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:55.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:55.429 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:55.464 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:55.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:55.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:55.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:55 smithi053 ceph-mon[89963]: pgmap v381: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 1.1 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 626 op/s; 14355/53463 objects degraded (26.850%); 4754/53463 objects misplaced (8.892%); 311 KiB/s, 150 objects/s recovering 2024-08-17T17:22:55.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:55 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 14355/53463 objects degraded (26.850%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:55.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:55 smithi003 ceph-mon[103344]: pgmap v381: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 1.1 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 626 op/s; 14355/53463 objects degraded (26.850%); 4754/53463 objects misplaced (8.892%); 311 KiB/s, 150 objects/s recovering 2024-08-17T17:22:55.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:55 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 14355/53463 objects degraded (26.850%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:55.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:55 smithi003 ceph-mon[100615]: pgmap v381: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 1.1 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 626 op/s; 14355/53463 objects degraded (26.850%); 4754/53463 objects misplaced (8.892%); 311 KiB/s, 150 objects/s recovering 2024-08-17T17:22:55.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:55 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 14355/53463 objects degraded (26.850%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:22:55.567 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:55.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:55.597 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:55.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:55.698 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:55.748 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:55.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:55.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:55.850 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:55.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:55.895 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:55.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:55.948 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:22:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:22:55] "GET /metrics HTTP/1.1" 200 36595 "" "Prometheus/2.51.0" 2024-08-17T17:22:55.996 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:56.055 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:56.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:56.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:56.157 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:56.185 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:56.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:56.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:56.287 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:56.313 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:56.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:56.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:56.414 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:56.445 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:56.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:56.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:56.547 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:56.572 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:56.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:56.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:56.626 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:22:56.626 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:22:56.674 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:56.715 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:56.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:56.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:56.817 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:56.859 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:56.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:56.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:56.960 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:57.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:57.013 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:57.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:57.114 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:57.147 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:57.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:57.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:57.249 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:57.301 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:57.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:57.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:57.403 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:57.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:57.435 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:57.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:57.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:57 smithi053 ceph-mon[89963]: pgmap v382: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 1.0 GiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 686 op/s; 14369/49299 objects degraded (29.147%); 4044/49299 objects misplaced (8.203%); 1.7 MiB/s, 139 objects/s recovering 2024-08-17T17:22:57.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:57 smithi003 ceph-mon[103344]: pgmap v382: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 1.0 GiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 686 op/s; 14369/49299 objects degraded (29.147%); 4044/49299 objects misplaced (8.203%); 1.7 MiB/s, 139 objects/s recovering 2024-08-17T17:22:57.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:57 smithi003 ceph-mon[100615]: pgmap v382: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 1.0 GiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 686 op/s; 14369/49299 objects degraded (29.147%); 4044/49299 objects misplaced (8.203%); 1.7 MiB/s, 139 objects/s recovering 2024-08-17T17:22:57.536 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:57.585 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:57.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:57.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:57.687 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:22:57.715 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:57.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:57.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:22:57.817 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:57.857 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:57.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:57.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:57.959 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:22:57.988 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:57.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:57.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:22:58.090 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:58.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:58.128 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:58.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:58.230 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:58.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:58.282 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:58.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:58.384 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:58.422 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:58.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:58.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:58.524 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:58.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:58.551 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:58.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:58.652 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:22:58.693 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:58.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:58.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:22:58.795 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:58.846 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:58.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:58.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:58.948 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:22:58.987 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:58.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:58.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:22:59.089 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:22:59.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:59.128 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:59.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:22:59.229 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:59.260 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:59.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:59.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:59.362 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:59.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:59.407 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:59.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:22:59.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:22:59 smithi053 ceph-mon[89963]: pgmap v383: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 988 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 597 op/s; 14369/47280 objects degraded (30.391%); 3800/47280 objects misplaced (8.037%); 1.4 MiB/s, 119 objects/s recovering 2024-08-17T17:22:59.485 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:22:59 smithi003 ceph-mon[103344]: pgmap v383: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 988 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 597 op/s; 14369/47280 objects degraded (30.391%); 3800/47280 objects misplaced (8.037%); 1.4 MiB/s, 119 objects/s recovering 2024-08-17T17:22:59.486 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:22:59 smithi003 ceph-mon[100615]: pgmap v383: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 988 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 597 op/s; 14369/47280 objects degraded (30.391%); 3800/47280 objects misplaced (8.037%); 1.4 MiB/s, 119 objects/s recovering 2024-08-17T17:22:59.508 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:22:59.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:59.538 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:59.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:22:59.641 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:59.669 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:59.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:59.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:59.771 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:22:59.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:59.801 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:22:59.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:22:59.902 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:22:59.951 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:22:59.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:22:59.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:00.053 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:00.088 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:00.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:00.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:00.190 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:00.220 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:00.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:00.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:00.322 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:00.351 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:00.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:00.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:00.453 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:00.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:00 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 14369/47280 objects degraded (30.391%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:23:00.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:23:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:23:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:23:00.470 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:00 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 14369/47280 objects degraded (30.391%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:23:00.470 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:00 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 14369/47280 objects degraded (30.391%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:23:00.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:00.497 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:00.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:00.598 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:00.644 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:00.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:00.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:00.746 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:00.783 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:00.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:00.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:00.885 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:00.911 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:00.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:00.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:01.013 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:01.041 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:01.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:01.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:01.143 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:01.176 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:01.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:01.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:01.278 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:01.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:01.320 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:01.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:01.421 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:01.449 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:01.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:01.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:01.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-17T17:23:01.551 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:01.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:01.595 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:01.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:01.696 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:01.726 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:01.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:01.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:01.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:01 smithi003 ceph-mon[100615]: pgmap v384: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 844 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 771 op/s; 13928/40389 objects degraded (34.485%); 3546/40389 objects misplaced (8.780%); 2.7 MiB/s, 174 objects/s recovering 2024-08-17T17:23:01.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:01 smithi003 ceph-mon[103344]: pgmap v384: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 844 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 771 op/s; 13928/40389 objects degraded (34.485%); 3546/40389 objects misplaced (8.780%); 2.7 MiB/s, 174 objects/s recovering 2024-08-17T17:23:01.828 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:01.878 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:01.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:01.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:01.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:01 smithi053 ceph-mon[89963]: pgmap v384: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 844 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 771 op/s; 13928/40389 objects degraded (34.485%); 3546/40389 objects misplaced (8.780%); 2.7 MiB/s, 174 objects/s recovering 2024-08-17T17:23:01.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:01.983 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-08-17T17:23:02.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:02.030 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:02.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:02.131 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:02.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:02.155 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:02.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:02.256 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:02.272 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:23:02.282 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:23:02.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:02.311 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:02.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:02.412 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:02.477 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:02.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:02.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:02.579 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:02.626 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:02.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:02.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:02.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:02.770 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:02.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:02.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:02.872 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:02.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:02.901 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:02.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:03.003 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:03.026 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:03.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:03.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:03.128 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:03.152 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:03.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:03.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:03.254 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:03.277 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:03.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:03.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:03.378 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:03.401 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:03.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:03.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:03.494 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-08-17T17:23:03.503 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:03.529 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:03.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:03.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:03.631 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:03.661 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:03.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:03.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:03.763 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:03.768 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:03 smithi053 ceph-mon[89963]: pgmap v385: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 760 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 719 op/s; 13928/36327 objects degraded (38.341%); 2879/36327 objects misplaced (7.925%); 2.5 MiB/s, 112 objects/s recovering 2024-08-17T17:23:03.768 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:03 smithi053 ceph-mon[89963]: osdmap e161: 8 total, 8 up, 7 in 2024-08-17T17:23:03.769 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:03 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/109462162' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:23:03.769 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:03 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:23:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:03 smithi003 ceph-mon[100615]: pgmap v385: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 760 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 719 op/s; 13928/36327 objects degraded (38.341%); 2879/36327 objects misplaced (7.925%); 2.5 MiB/s, 112 objects/s recovering 2024-08-17T17:23:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:03 smithi003 ceph-mon[100615]: osdmap e161: 8 total, 8 up, 7 in 2024-08-17T17:23:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:03 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/109462162' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:23:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:03 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:23:03.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:03 smithi003 ceph-mon[103344]: pgmap v385: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 79 active+clean; 760 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 719 op/s; 13928/36327 objects degraded (38.341%); 2879/36327 objects misplaced (7.925%); 2.5 MiB/s, 112 objects/s recovering 2024-08-17T17:23:03.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:03 smithi003 ceph-mon[103344]: osdmap e161: 8 total, 8 up, 7 in 2024-08-17T17:23:03.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:03 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/109462162' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:23:03.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:03 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:23:03.791 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:03.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:03.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:03.893 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:03.924 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:03.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:03.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:04.025 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:04.079 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:04.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:04.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:04.180 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:04.233 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:04.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:04.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:04.335 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:04.379 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:04.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:04.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:04.481 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:04.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:04.527 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:04.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:04.628 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:04.661 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:04.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:04.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:04.763 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:04.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:04 smithi003 ceph-mon[103344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:23:04.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:04 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:23:04.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:04 smithi003 ceph-mon[103344]: osdmap e162: 8 total, 8 up, 7 in 2024-08-17T17:23:04.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:04 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 13928/36327 objects degraded (38.341%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:23:04.775 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:04 smithi003 ceph-mon[100615]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:23:04.775 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:04 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:23:04.775 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:04 smithi003 ceph-mon[100615]: osdmap e162: 8 total, 8 up, 7 in 2024-08-17T17:23:04.776 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:04 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 13928/36327 objects degraded (38.341%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:23:04.804 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:04.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:04.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:04.906 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:04.954 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:04.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:04.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:04.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:04 smithi053 ceph-mon[89963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:23:04.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:04 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:23:04.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:04 smithi053 ceph-mon[89963]: osdmap e162: 8 total, 8 up, 7 in 2024-08-17T17:23:04.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:04 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 13928/36327 objects degraded (38.341%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:23:05.056 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:05.091 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:05.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:05.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:05.113 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:23:05.193 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:05.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:05.229 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:05.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:05.330 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:05.377 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:05.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:05.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:05.479 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:05.512 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:05.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:05.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:05.614 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:05.650 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:05.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:05.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:05.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:05 smithi003 ceph-mon[100615]: pgmap v388: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 675 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 708 op/s; 13511/32280 objects degraded (41.856%); 2879/32280 objects misplaced (8.919%); 2.0 MiB/s, 138 objects/s recovering 2024-08-17T17:23:05.652 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:05 smithi003 ceph-mon[103344]: pgmap v388: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 675 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 708 op/s; 13511/32280 objects degraded (41.856%); 2879/32280 objects misplaced (8.919%); 2.0 MiB/s, 138 objects/s recovering 2024-08-17T17:23:05.752 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:05.783 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:05.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:05.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:05.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:05.929 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:05.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:05.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:05.950 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:23:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:23:05] "GET /metrics HTTP/1.1" 200 36595 "" "Prometheus/2.51.0" 2024-08-17T17:23:05.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:05 smithi053 ceph-mon[89963]: pgmap v388: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 675 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 708 op/s; 13511/32280 objects degraded (41.856%); 2879/32280 objects misplaced (8.919%); 2.0 MiB/s, 138 objects/s recovering 2024-08-17T17:23:06.031 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:06.080 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:06.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:06.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:06.182 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:06.213 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:06.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:06.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:06.315 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:06.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:06.359 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:06.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:06.460 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:06.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:06.488 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:06.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:06.589 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:06.643 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:06.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:06.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:06.745 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:06.772 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:06.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:06.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:06.873 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:06.910 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:06.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:06.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:06.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:06 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:23:07.012 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:07.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:06 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:23:07.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:06 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:23:07.049 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:07.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:07.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:07.151 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:07.193 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:07.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:07.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:07.295 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:07.339 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:07.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:07.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:07.441 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:07.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:07.475 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:07.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:07.575 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:07.611 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:07.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:07.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:07.713 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:07.735 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:23:07.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:07.747 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:07.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:07.848 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:07.876 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:07.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:07.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:07.893 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:07 smithi003 ceph-mon[103344]: pgmap v389: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 599 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 777 op/s; 13511/28608 objects degraded (47.228%); 2101/28608 objects misplaced (7.344%); 4.9 MiB/s, 185 objects/s recovering 2024-08-17T17:23:07.893 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:07 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:23:07.893 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:07 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:23:07.894 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:07 smithi003 ceph-mon[100615]: pgmap v389: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 599 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 777 op/s; 13511/28608 objects degraded (47.228%); 2101/28608 objects misplaced (7.344%); 4.9 MiB/s, 185 objects/s recovering 2024-08-17T17:23:07.894 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:07 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:23:07.894 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:07 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:23:07.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:07 smithi053 ceph-mon[89963]: pgmap v389: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 599 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 777 op/s; 13511/28608 objects degraded (47.228%); 2101/28608 objects misplaced (7.344%); 4.9 MiB/s, 185 objects/s recovering 2024-08-17T17:23:07.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:07 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:23:07.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:07 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:23:07.978 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:08.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:08.009 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:08.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:08.087 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:23:08.087 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:23:08.111 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:08.156 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:08.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:08.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:08.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:08.298 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:08.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:08.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:08.400 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:08.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:08.442 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:08.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:08.543 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:08.573 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:08.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:08.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:08.674 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:08.713 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:08.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:08.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:08.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:23:08.815 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:08.845 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:23:08.845 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:23:08.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:08.854 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:08.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:08.888 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-17T17:23:08.955 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:09.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:09.011 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:09.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:09.112 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:09.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:09.145 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:09.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:09.246 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:09.272 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:09.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:09.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:09.374 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:09.400 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:09.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:09.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:09.501 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:09.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:09.565 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:09.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:09.666 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:09.694 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:09.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:09.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:09.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:09 smithi003 ceph-mon[100615]: pgmap v390: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 563 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 562 op/s; 13511/26880 objects degraded (50.264%); 1915/26880 objects misplaced (7.124%); 2.9 MiB/s, 98 objects/s recovering 2024-08-17T17:23:09.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:23:09.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:23:09.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:09 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 13511/26880 objects degraded (50.264%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:23:09.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:09 smithi003 ceph-mon[103344]: pgmap v390: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 563 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 562 op/s; 13511/26880 objects degraded (50.264%); 1915/26880 objects misplaced (7.124%); 2.9 MiB/s, 98 objects/s recovering 2024-08-17T17:23:09.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:23:09.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:23:09.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:09 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 13511/26880 objects degraded (50.264%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:23:09.796 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:09.836 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:09.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:09.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:09.878 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:09 smithi053 ceph-mon[89963]: pgmap v390: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 563 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 562 op/s; 13511/26880 objects degraded (50.264%); 1915/26880 objects misplaced (7.124%); 2.9 MiB/s, 98 objects/s recovering 2024-08-17T17:23:09.878 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:23:09.878 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:23:09.878 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:09 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 13511/26880 objects degraded (50.264%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:23:09.938 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:09.988 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:09.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:09.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:10.090 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:10.128 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:10.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:10.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:10.230 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:10.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:10.279 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:10.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:10.380 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:10.408 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:10.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:10.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:10.464 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:23:10.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:23:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:23:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:23:10.510 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:10.564 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:10.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:10.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:10.666 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:10.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:10.721 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:10.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:10.822 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:10.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:10.860 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:10.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:10.961 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:10.989 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:10.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:10.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:11.091 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:11.145 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:11.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:11.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:11.247 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:11.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:11.282 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:11.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:11.383 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:11.414 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:11.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:11.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:11.516 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:11.546 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:23:11.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:11.583 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:11.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:11.613 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:11 smithi003 ceph-mon[100615]: pgmap v391: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 419 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 681 op/s; 13260/19968 objects degraded (66.406%); 1646/19968 objects misplaced (8.243%); 5.0 MiB/s, 163 objects/s recovering 2024-08-17T17:23:11.613 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:11 smithi003 ceph-mon[103344]: pgmap v391: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 419 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 681 op/s; 13260/19968 objects degraded (66.406%); 1646/19968 objects misplaced (8.243%); 5.0 MiB/s, 163 objects/s recovering 2024-08-17T17:23:11.636 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:23:11.637 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:23:11.677 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:23:11.677 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:23:11.683 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:11.729 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:23:11.729 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:23:11.741 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:11.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:11.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:11.774 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:23:11.775 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:23:11.829 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:23:11.830 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:23:11.843 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:11.867 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:23:11.868 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:23:11.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:11.871 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:11.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:11.911 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:23:11.912 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:23:11.950 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:23:11.951 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:23:11.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:11 smithi053 ceph-mon[89963]: pgmap v391: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 419 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 681 op/s; 13260/19968 objects degraded (66.406%); 1646/19968 objects misplaced (8.243%); 5.0 MiB/s, 163 objects/s recovering 2024-08-17T17:23:11.971 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:12.002 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:12.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:12.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:12.104 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:12.148 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:12.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:12.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:12.249 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:12.285 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:12.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:12.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:12.387 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:12.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:12.429 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:12.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:12.530 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:12.570 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:12.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:12.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:12.672 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:12.713 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:12.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:12.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:12.815 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:12.828 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:12 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2541163284' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:23:12.828 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:12 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:23:12.828 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:12 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:23:12.828 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:12 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2541163284' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:23:12.828 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:12 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:23:12.828 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:12 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:23:12.845 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:12.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:12.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:12.947 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:12.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:12 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2541163284' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:23:12.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:12 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:23:12.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:12 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:23:12.976 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:12.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:12.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:13.078 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:13.125 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:13.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:13.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:13.228 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:13.261 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:13.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:13.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:13.363 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:13.395 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:13.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:13.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:13.497 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:13.540 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:13.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:13.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:13.642 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:13.695 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:13.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:13.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:13.798 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:13.799 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:13 smithi003 ceph-mon[100615]: pgmap v392: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 330 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 738 op/s; 13260/15687 objects degraded (84.529%); 924/15687 objects misplaced (5.890%); 4.3 MiB/s, 140 objects/s recovering 2024-08-17T17:23:13.799 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:13 smithi003 ceph-mon[103344]: pgmap v392: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 330 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 738 op/s; 13260/15687 objects degraded (84.529%); 924/15687 objects misplaced (5.890%); 4.3 MiB/s, 140 objects/s recovering 2024-08-17T17:23:13.828 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:13.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:13.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:13.930 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:13.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:13 smithi053 ceph-mon[89963]: pgmap v392: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 330 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 738 op/s; 13260/15687 objects degraded (84.529%); 924/15687 objects misplaced (5.890%); 4.3 MiB/s, 140 objects/s recovering 2024-08-17T17:23:13.984 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:13.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:13.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:14.086 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:14.132 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:14.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:14.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:14.234 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:14.262 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:14.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:14.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:14.364 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:14.394 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:14.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:14.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:14.496 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:14.546 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:14.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:14.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:14.648 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:14.694 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:14.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:14.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:14.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:14 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 13260/15687 objects degraded (84.529%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:23:14.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:14 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 13260/15687 objects degraded (84.529%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:23:14.796 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:14.827 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:14.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:14.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:14.829 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:23:14.929 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:14.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:14 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 13260/15687 objects degraded (84.529%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:23:14.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:14.985 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:14.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:15.086 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:15.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:15.113 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:15.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:15.214 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:15.266 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:15.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:15.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:15.368 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:15.400 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:15.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:15.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:15.502 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:15.527 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:15.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:15.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:15.585 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:15 smithi003 ceph-mon[100615]: pgmap v393: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 227 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 784 op/s; 13078/10740 objects degraded (121.769%); 924/10740 objects misplaced (8.603%); 3.9 MiB/s, 126 objects/s recovering 2024-08-17T17:23:15.628 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:15.687 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:15.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:15.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:15.789 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:15.818 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:15.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:15.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:15.920 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:15.945 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:15.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:15.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:15.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:15 smithi053 ceph-mon[89963]: pgmap v393: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 227 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 784 op/s; 13078/10740 objects degraded (121.769%); 924/10740 objects misplaced (8.603%); 3.9 MiB/s, 126 objects/s recovering 2024-08-17T17:23:16.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:15 smithi003 ceph-mon[103344]: pgmap v393: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 227 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 784 op/s; 13078/10740 objects degraded (121.769%); 924/10740 objects misplaced (8.603%); 3.9 MiB/s, 126 objects/s recovering 2024-08-17T17:23:16.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:23:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:23:15] "GET /metrics HTTP/1.1" 200 36591 "" "Prometheus/2.51.0" 2024-08-17T17:23:16.047 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:16.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:16.081 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:16.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:16.182 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:16.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:16.214 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:16.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:16.315 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:16.356 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:16.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:16.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:16.375 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Removed 102054 objects 2024-08-17T17:23:16.376 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Clean up completed and total clean up time :154.548 2024-08-17T17:23:16.457 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:16.479 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:16.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:16.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:16.581 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:16.608 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:16.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:16.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:16.710 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:16.734 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:16.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:16.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:16.836 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:16.858 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:16.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:16.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:16.959 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:16.981 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:16.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:16.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:17.083 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:17.105 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:17.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:17.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:17.214 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:17.237 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:17.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:17.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:17.339 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:17.361 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:17.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:17.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:17.463 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:17.485 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:17.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:17.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:17.587 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:17.610 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:17.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:17.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:17.712 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:17.737 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:17.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:17.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:17.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:17 smithi003 ceph-mon[103344]: pgmap v394: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 136 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 718 op/s; 12874/6414 objects degraded (200.717%); 388/6414 objects misplaced (6.049%); 3.7 MiB/s, 112 objects/s recovering 2024-08-17T17:23:17.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:17 smithi003 ceph-mon[100615]: pgmap v394: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 136 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 718 op/s; 12874/6414 objects degraded (200.717%); 388/6414 objects misplaced (6.049%); 3.7 MiB/s, 112 objects/s recovering 2024-08-17T17:23:17.839 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:17.862 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:17.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:17.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:17.963 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:17.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:17 smithi053 ceph-mon[89963]: pgmap v394: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 136 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 718 op/s; 12874/6414 objects degraded (200.717%); 388/6414 objects misplaced (6.049%); 3.7 MiB/s, 112 objects/s recovering 2024-08-17T17:23:17.986 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:17.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:17.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:18.088 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:18.110 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:18.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:18.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:18.212 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:18.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:18.235 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:18.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:18.336 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:18.360 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:18.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:18.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:18.461 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:18.483 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:18.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:18.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:18.585 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:18.586 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-17T17:23:18.586 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:23:18.587 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-08-17T17:23:18.587 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:23:18.613 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:18.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:18.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:18.715 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:18.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:18.738 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:18.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:18.839 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:18.861 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:18.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:18.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:18.963 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:18.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:18 smithi053 ceph-mon[89963]: pgmap v395: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 93 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 673 op/s; 12874/4347 objects degraded (296.158%); 130/4347 objects misplaced (2.991%); 1.8 MiB/s, 81 objects/s recovering 2024-08-17T17:23:18.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:18 smithi053 ceph-mon[89963]: osdmap e163: 8 total, 8 up, 7 in 2024-08-17T17:23:18.985 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:18.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:18.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:19.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:18 smithi003 ceph-mon[100615]: pgmap v395: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 93 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 673 op/s; 12874/4347 objects degraded (296.158%); 130/4347 objects misplaced (2.991%); 1.8 MiB/s, 81 objects/s recovering 2024-08-17T17:23:19.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:18 smithi003 ceph-mon[100615]: osdmap e163: 8 total, 8 up, 7 in 2024-08-17T17:23:19.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:18 smithi003 ceph-mon[103344]: pgmap v395: 89 pgs: 1 active+recovery_wait, 3 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 93 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 673 op/s; 12874/4347 objects degraded (296.158%); 130/4347 objects misplaced (2.991%); 1.8 MiB/s, 81 objects/s recovering 2024-08-17T17:23:19.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:18 smithi003 ceph-mon[103344]: osdmap e163: 8 total, 8 up, 7 in 2024-08-17T17:23:19.087 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:19.109 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:19.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:19.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:19.210 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:19.233 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:19.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:19.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:19.334 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:19.357 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:19.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:19.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:19.459 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:19.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:19.482 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:19.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:19.583 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:19.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:19.608 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:19.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:19.709 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:19.731 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:19.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:19.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:19.833 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:19.855 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:19.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:19.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:19.956 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:19.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:19 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 12874/4347 objects degraded (296.158%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:23:19.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:19 smithi053 ceph-mon[89963]: osdmap e164: 8 total, 8 up, 7 in 2024-08-17T17:23:19.978 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:19.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:19.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:20.022 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:19 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 12874/4347 objects degraded (296.158%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:23:20.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:19 smithi003 ceph-mon[100615]: osdmap e164: 8 total, 8 up, 7 in 2024-08-17T17:23:20.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:19 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 12874/4347 objects degraded (296.158%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:23:20.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:19 smithi003 ceph-mon[103344]: osdmap e164: 8 total, 8 up, 7 in 2024-08-17T17:23:20.080 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:20.102 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:20.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:20.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:20.204 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:20.227 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:20.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:20.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:20.328 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:20.351 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:20.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:20.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:20.453 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:20.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:23:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:23:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:23:20.475 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:20.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:20.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:20.577 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:20.601 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:20.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:20.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:20.703 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:20.725 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:20.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:20.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:20.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:20 smithi003 ceph-mon[100615]: pgmap v398: 89 pgs: 1 peering, 1 active+recovery_wait, 3 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 18 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 623 op/s; 11885/720 objects degraded (1650.694%); 130/720 objects misplaced (18.056%); 552 KiB/s, 180 objects/s recovering 2024-08-17T17:23:20.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:20 smithi003 ceph-mon[103344]: pgmap v398: 89 pgs: 1 peering, 1 active+recovery_wait, 3 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 18 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 623 op/s; 11885/720 objects degraded (1650.694%); 130/720 objects misplaced (18.056%); 552 KiB/s, 180 objects/s recovering 2024-08-17T17:23:20.826 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:20.848 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:20.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:20.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:20.950 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:20.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:20 smithi053 ceph-mon[89963]: pgmap v398: 89 pgs: 1 peering, 1 active+recovery_wait, 3 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 18 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 623 op/s; 11885/720 objects degraded (1650.694%); 130/720 objects misplaced (18.056%); 552 KiB/s, 180 objects/s recovering 2024-08-17T17:23:20.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:20.973 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:20.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:21.074 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:21.096 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:21.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:21.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:21.198 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:21.220 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:21.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:21.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:21.321 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:21.344 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:21.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:21.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:21.446 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:21.470 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:21.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:21.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:21.571 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:21.594 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:21.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:21.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:21.654 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-08-17T17:23:21.668 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:21 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3381869809' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:23:21.668 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:21 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3381869809' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:23:21.696 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:21.719 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:21.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:21.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:21.821 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:21.844 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:21.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:21.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:21.946 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:21.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:21 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3381869809' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:23:21.970 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:21.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:21.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:22.072 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:22.095 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:22.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:22.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:22.197 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:22.219 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:22.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:22.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:22.321 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:22.344 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:22.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:22.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:22.445 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:22.468 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:22.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:22.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:22.570 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:22.592 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:22.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:22.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:22.694 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:22.717 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:22.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:22.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:22.746 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:22 smithi003 ceph-mon[103344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:23:22.747 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:22 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3381869809' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:23:22.747 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:22 smithi003 ceph-mon[103344]: osdmap e165: 8 total, 8 up, 7 in 2024-08-17T17:23:22.747 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:22 smithi003 ceph-mon[103344]: pgmap v400: 89 pgs: 1 peering, 1 active+recovery_wait, 3 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 351 op/s; 11868/84 objects degraded (14128.571%); 0 B/s, 167 objects/s recovering 2024-08-17T17:23:22.747 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:22 smithi003 ceph-mon[100615]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:23:22.747 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:22 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3381869809' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:23:22.747 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:22 smithi003 ceph-mon[100615]: osdmap e165: 8 total, 8 up, 7 in 2024-08-17T17:23:22.747 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:22 smithi003 ceph-mon[100615]: pgmap v400: 89 pgs: 1 peering, 1 active+recovery_wait, 3 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 351 op/s; 11868/84 objects degraded (14128.571%); 0 B/s, 167 objects/s recovering 2024-08-17T17:23:22.754 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:23:22.754 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:23:22.774 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:23:22.774 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-08-17T17:23:22.774 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-08-17T17:23:22.774 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:23:22.819 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:22.841 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:22.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:22.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:22.942 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:22.964 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:22.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:22.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:22.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:22 smithi053 ceph-mon[89963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:23:22.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:22 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3381869809' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:23:22.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:22 smithi053 ceph-mon[89963]: osdmap e165: 8 total, 8 up, 7 in 2024-08-17T17:23:22.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:22 smithi053 ceph-mon[89963]: pgmap v400: 89 pgs: 1 peering, 1 active+recovery_wait, 3 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 351 op/s; 11868/84 objects degraded (14128.571%); 0 B/s, 167 objects/s recovering 2024-08-17T17:23:23.066 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:23.088 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:23.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:23.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:23.190 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:23.212 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:23.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:23.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:23.314 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:23.337 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:23.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:23.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:23.438 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:23.461 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:23.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:23.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:23.562 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:23.587 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:23.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:23.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:23.689 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:23.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:23.712 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:23.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:23.813 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:23.836 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:23.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:23.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:23.937 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:23.960 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:23.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:23.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:23.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:23:23.997 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:23:23.997 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:23:24.022 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-17T17:23:24.061 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:24.088 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:24.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:24.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:24.190 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:24.212 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:24.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:24.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:24.313 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:24.336 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:24.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:24.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:24.437 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:24.460 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:24.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:24.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:24.467 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:23:24.562 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:24.585 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:24.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:24.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:24.687 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:24.710 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:24.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:24.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:24.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:24 smithi003 ceph-mon[103344]: pgmap v401: 89 pgs: 1 peering, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 236 op/s; 10511/84 objects degraded (12513.095%); 0 B/s, 393 objects/s recovering 2024-08-17T17:23:24.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:24 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 11868/84 objects degraded (14128.571%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:23:24.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:24 smithi003 ceph-mon[100615]: pgmap v401: 89 pgs: 1 peering, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 236 op/s; 10511/84 objects degraded (12513.095%); 0 B/s, 393 objects/s recovering 2024-08-17T17:23:24.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:24 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 11868/84 objects degraded (14128.571%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:23:24.812 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:24.835 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:24.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:24.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:24.937 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:24.959 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:24.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:24.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:24.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:24 smithi053 ceph-mon[89963]: pgmap v401: 89 pgs: 1 peering, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 236 op/s; 10511/84 objects degraded (12513.095%); 0 B/s, 393 objects/s recovering 2024-08-17T17:23:24.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:24 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 11868/84 objects degraded (14128.571%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:23:25.062 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:25.084 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:25.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:25.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:25.185 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:23:25.186 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:25.215 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:25.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:25.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:25.317 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:25.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:25.340 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:25.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:25.441 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:25.464 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:25.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:25.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:25.565 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:25.588 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:25.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:25.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:25.690 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:25.712 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:25.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:25.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:25.813 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:25.836 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:25.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:25.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:25.937 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:25.960 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:25.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:25.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:26.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:23:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:23:25] "GET /metrics HTTP/1.1" 200 36574 "" "Prometheus/2.51.0" 2024-08-17T17:23:26.062 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:26.085 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:26.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:26.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:26.186 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:26.209 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:26.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:26.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:26.311 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:26.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:26.334 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:26.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:26.435 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:26.457 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:26.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:26.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:26.558 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:26.583 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:26.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:26.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:26.684 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:26.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:26.707 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:26.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:26.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:26.831 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:26.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:26.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:26.932 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:26.955 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:26.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:26.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:26.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:26 smithi053 ceph-mon[89963]: osdmap e166: 8 total, 8 up, 7 in 2024-08-17T17:23:26.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:26 smithi053 ceph-mon[89963]: pgmap v403: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 34 op/s; 10511/84 objects degraded (12513.095%); 0 B/s, 220 objects/s recovering 2024-08-17T17:23:27.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:26 smithi003 ceph-mon[103344]: osdmap e166: 8 total, 8 up, 7 in 2024-08-17T17:23:27.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:26 smithi003 ceph-mon[103344]: pgmap v403: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 34 op/s; 10511/84 objects degraded (12513.095%); 0 B/s, 220 objects/s recovering 2024-08-17T17:23:27.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:26 smithi003 ceph-mon[100615]: osdmap e166: 8 total, 8 up, 7 in 2024-08-17T17:23:27.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:26 smithi003 ceph-mon[100615]: pgmap v403: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 34 op/s; 10511/84 objects degraded (12513.095%); 0 B/s, 220 objects/s recovering 2024-08-17T17:23:27.057 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:27.080 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:27.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:27.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:27.181 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:27.203 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:27.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:27.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:27.305 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:27.327 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:27.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:27.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:27.429 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:27.452 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:27.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:27.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:27.554 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:27.577 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:27.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:27.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:27.678 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:27.704 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:27.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:27.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:27.806 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:27.828 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:27.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:27.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:27.930 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:27.951 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:27.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:27.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:27.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:27 smithi053 ceph-mon[89963]: osdmap e167: 8 total, 8 up, 7 in 2024-08-17T17:23:28.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:27 smithi003 ceph-mon[103344]: osdmap e167: 8 total, 8 up, 7 in 2024-08-17T17:23:28.007 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:27 smithi003 ceph-mon[100615]: osdmap e167: 8 total, 8 up, 7 in 2024-08-17T17:23:28.053 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:28.076 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:28.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:28.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:28.177 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:28.199 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:28.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:28.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:28.302 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:28.324 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:28.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:28.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:28.426 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:28.448 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:28.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:28.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:28.550 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:28.573 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:28.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:28.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:28.675 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:28.698 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:28.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:28.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:28.800 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:28.822 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:28.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:28.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:28.924 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:28.946 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:28.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:28.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:28.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:28 smithi053 ceph-mon[89963]: pgmap v405: 89 pgs: 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 10371/84 objects degraded (12346.429%); 0 B/s, 242 objects/s recovering 2024-08-17T17:23:29.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:28 smithi003 ceph-mon[103344]: pgmap v405: 89 pgs: 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 10371/84 objects degraded (12346.429%); 0 B/s, 242 objects/s recovering 2024-08-17T17:23:29.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:28 smithi003 ceph-mon[100615]: pgmap v405: 89 pgs: 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.6 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 10371/84 objects degraded (12346.429%); 0 B/s, 242 objects/s recovering 2024-08-17T17:23:29.048 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:29.071 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:29.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:29.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:29.173 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:29.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:29.196 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:29.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:29.298 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:29.320 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:29.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:29.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:29.422 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:29.444 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:29.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:29.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:29.545 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:23:29.546 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:23:29.546 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:23:29.546 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:23:29.546 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:23:29.546 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:23:29.546 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:23:29.546 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:23:29.547 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:29.547 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:23:29.548 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:23:29.567 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:23:29.567 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:23:29.569 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:29.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:29.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:29.589 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:23:29.589 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:23:29.605 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:23:29.605 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:23:29.672 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:29.694 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:29.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:29.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:29.796 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:29.819 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:29.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:29.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:29.921 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:29.943 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:29.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:29.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:29.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:29 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3713112621' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:23:29.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:29 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 10371/84 objects degraded (12346.429%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:23:30.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:29 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3713112621' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:23:30.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:29 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 10371/84 objects degraded (12346.429%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:23:30.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:29 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3713112621' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:23:30.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:29 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 10371/84 objects degraded (12346.429%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:23:30.045 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:30.067 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:30.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:30.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:30.169 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:30.192 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:30.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:30.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:30.293 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:30.315 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:30.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:30.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:30.417 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:30.440 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:30.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:30.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:30.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:23:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:23:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:23:30.514 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:23:30.542 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:30.564 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:30.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:30.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:30.666 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:30.691 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:30.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:30.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:30.725 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:30 smithi003 ceph-mon[103344]: pgmap v406: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 83 active+clean; 4.6 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 9000/84 objects degraded (10714.286%); 0 B/s, 358 objects/s recovering 2024-08-17T17:23:30.725 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:30 smithi003 ceph-mon[100615]: pgmap v406: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 83 active+clean; 4.6 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 9000/84 objects degraded (10714.286%); 0 B/s, 358 objects/s recovering 2024-08-17T17:23:30.793 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:30.816 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:30.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:30.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:30.918 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:30.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:30.941 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:30.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:30.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:30 smithi053 ceph-mon[89963]: pgmap v406: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 83 active+clean; 4.6 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 9000/84 objects degraded (10714.286%); 0 B/s, 358 objects/s recovering 2024-08-17T17:23:31.042 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:31.065 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:31.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:31.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:31.167 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:31.190 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:31.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:31.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:31.292 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:31.314 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:31.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:31.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:31.416 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:31.438 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:31.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:31.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:31.540 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:31.564 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:31.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:31.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:31.666 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:31.688 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:31.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:31.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:31.790 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:31.812 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:31.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:31.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:31.913 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:31.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:31.939 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:31.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:32.040 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:32.062 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:32.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:32.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:32.164 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:32.186 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:32.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:32.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:32.288 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:32.312 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:32.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:32.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:32.413 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:32.435 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:32.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:32.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:32.486 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:23:32.537 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:32.563 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:32.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:32.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:32.665 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:32.687 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:32.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:32.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:32.790 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:32.812 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:32.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:32.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:32.913 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:32.937 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:32.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:32.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:33.039 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:33.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:33.062 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:33.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:33.128 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:32 smithi003 ceph-mon[103344]: pgmap v407: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 83 active+clean; 4.6 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 8084/84 objects degraded (9623.810%); 0 B/s, 303 objects/s recovering 2024-08-17T17:23:33.128 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:32 smithi003 ceph-mon[100615]: pgmap v407: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 83 active+clean; 4.6 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 8084/84 objects degraded (9623.810%); 0 B/s, 303 objects/s recovering 2024-08-17T17:23:33.163 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:33.185 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:33.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:33.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:33.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:32 smithi053 ceph-mon[89963]: pgmap v407: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 83 active+clean; 4.6 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 8084/84 objects degraded (9623.810%); 0 B/s, 303 objects/s recovering 2024-08-17T17:23:33.288 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:33.310 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:33.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:33.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:33.412 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:33.435 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:33.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:33.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:33.537 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:33.560 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:33.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:33.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:33.661 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:33.685 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:33.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:33.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:33.787 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:33.810 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:33.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:33.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:33.912 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:33.934 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:33.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:33.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:34.036 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:34.058 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:34.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:34.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:34.160 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:34.182 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:34.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:34.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:34.284 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:34.306 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:34.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:34.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:34.407 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:34.429 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:34.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:34.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:34.531 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:34.553 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:34.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:34.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:34.655 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:34.678 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:34.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:34.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:34.780 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:34.801 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:34.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:34.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:34.903 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:34.927 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:34.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:34.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:35.029 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:35.051 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:35.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:35.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:35.152 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:35.174 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:35.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:35.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:35.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:34 smithi053 ceph-mon[89963]: pgmap v408: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 83 active+clean; 4.6 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 8084/84 objects degraded (9623.810%); 0 B/s, 297 objects/s recovering 2024-08-17T17:23:35.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:34 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 8084/84 objects degraded (9623.810%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:23:35.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:34 smithi003 ceph-mon[103344]: pgmap v408: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 83 active+clean; 4.6 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 8084/84 objects degraded (9623.810%); 0 B/s, 297 objects/s recovering 2024-08-17T17:23:35.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:34 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 8084/84 objects degraded (9623.810%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:23:35.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:34 smithi003 ceph-mon[100615]: pgmap v408: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 83 active+clean; 4.6 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 8084/84 objects degraded (9623.810%); 0 B/s, 297 objects/s recovering 2024-08-17T17:23:35.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:34 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 8084/84 objects degraded (9623.810%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:23:35.276 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:35.298 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:35.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:35.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:35.400 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:35.423 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:35.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:35.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:35.524 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:35.546 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:35.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:35.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:35.648 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:35.670 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:35.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:35.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:35.772 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:35.794 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:35.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:35.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:35.896 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:35.919 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:35.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:35.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:36.021 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:36.022 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:23:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:23:35] "GET /metrics HTTP/1.1" 200 36574 "" "Prometheus/2.51.0" 2024-08-17T17:23:36.043 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:36.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:36.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:36.145 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:36.169 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:36.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:36.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:36.271 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:36.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:36.293 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:36.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:36.394 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:36.417 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:36.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:36.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:36.519 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:36.541 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:36.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:36.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:36.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:36.666 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:36.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:36.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:36.768 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:36.790 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:36.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:36.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:36.892 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:36.914 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:36.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:36.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:37.015 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:37.038 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:37.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:37.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:37.140 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:37.162 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:37.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:37.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:37.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:36 smithi053 ceph-mon[89963]: pgmap v409: 89 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 83 active+clean; 4.6 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 8084/84 objects degraded (9623.810%); 0 B/s, 242 objects/s recovering 2024-08-17T17:23:37.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:36 smithi053 ceph-mon[89963]: from='client.35487 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:23:37.242 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:36 smithi003 ceph-mon[103344]: pgmap v409: 89 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 83 active+clean; 4.6 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 8084/84 objects degraded (9623.810%); 0 B/s, 242 objects/s recovering 2024-08-17T17:23:37.242 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:36 smithi003 ceph-mon[103344]: from='client.35487 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:23:37.242 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:36 smithi003 ceph-mon[100615]: pgmap v409: 89 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 83 active+clean; 4.6 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 8084/84 objects degraded (9623.810%); 0 B/s, 242 objects/s recovering 2024-08-17T17:23:37.242 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:36 smithi003 ceph-mon[100615]: from='client.35487 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:23:37.245 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:23:37.246 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:23:37.263 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:37.285 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:37.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:37.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:37.387 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:37.410 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:37.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:37.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:37.512 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:37.535 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:37.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:37.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:37.636 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:37.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:37.662 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:37.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:37.763 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:37.785 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:37.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:37.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:37.887 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:37.910 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:37.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:37.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:38.012 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:38.034 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:38.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:38.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:38.136 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:38.158 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:38.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:38.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:38.259 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:38.281 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:38.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:38.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:38.383 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:38.405 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:38.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:38.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:38.506 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:38.528 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:38.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:38.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:38.630 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:38.652 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:38.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:38.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:38.754 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:38.777 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:38.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:38.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:38.878 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:38.901 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:38.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:38.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:39.003 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:39.025 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:39.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:39.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:39.127 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:39.149 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:39.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:39.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:39.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:38 smithi053 ceph-mon[89963]: pgmap v410: 89 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 83 active+clean; 4.6 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 7052/84 objects degraded (8395.238%); 0 B/s, 297 objects/s recovering 2024-08-17T17:23:39.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:23:39.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:39.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:38 smithi003 ceph-mon[103344]: pgmap v410: 89 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 83 active+clean; 4.6 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 7052/84 objects degraded (8395.238%); 0 B/s, 297 objects/s recovering 2024-08-17T17:23:39.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:23:39.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:38 smithi003 ceph-mon[100615]: pgmap v410: 89 pgs: 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 83 active+clean; 4.6 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 7052/84 objects degraded (8395.238%); 0 B/s, 297 objects/s recovering 2024-08-17T17:23:39.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:23:39.273 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:39.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:39.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:39.375 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:39.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:39.399 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:39.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:39.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:39.523 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:39.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:39.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:39.625 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:39.647 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:39.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:39.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:39.748 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:39.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:39.772 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:39.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:39.873 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:39.895 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:39.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:39.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:39.997 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:40.020 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:40.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:40.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:40.122 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:40.143 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:40.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:40.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:40.214 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:39 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 7052/84 objects degraded (8395.238%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:23:40.245 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:40.267 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:40.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:40.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:40.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:39 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 7052/84 objects degraded (8395.238%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:23:40.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:39 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 7052/84 objects degraded (8395.238%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:23:40.369 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:40.391 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:40.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:40.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:40.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:23:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:23:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:23:40.493 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:40.515 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:40.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:40.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:40.617 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:40.639 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:40.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:40.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:40.741 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:40.763 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:40.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:40.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:40.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:40.888 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:40.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:40.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:40.990 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:41.013 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:41.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:41.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:41.115 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:41.137 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:41.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:41.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:41.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:40 smithi053 ceph-mon[89963]: pgmap v411: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 83 active+clean; 4.6 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 7052/84 objects degraded (8395.238%); 0 B/s, 276 objects/s recovering 2024-08-17T17:23:41.240 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:41.262 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:41.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:41.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:41.264 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:40 smithi003 ceph-mon[103344]: pgmap v411: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 83 active+clean; 4.6 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 7052/84 objects degraded (8395.238%); 0 B/s, 276 objects/s recovering 2024-08-17T17:23:41.264 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:40 smithi003 ceph-mon[100615]: pgmap v411: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 83 active+clean; 4.6 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 7052/84 objects degraded (8395.238%); 0 B/s, 276 objects/s recovering 2024-08-17T17:23:41.364 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:41.387 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:41.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:41.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:41.489 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:41.512 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:41.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:41.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:41.614 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:41.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:41.637 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:41.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:41.738 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:41.762 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:41.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:41.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:41.830 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2024-08-17T17:23:35.819346+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790221,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":205232,"num_write_kb":6546496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34586,"num_bytes_recovered":1542529232,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120603,"ondisk_log_size":120603,"up":267,"acting":265,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":278,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":3238300,"kb_used_data":1644808,"kb_used_omap":33,"kb_used_meta":1593374,"kb_avail":652977764,"statfs":{"total":671965249536,"available":668649230336,"internally_reserved":0,"allocated":1684283392,"data_stored":1677567163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34068,"internal_metadata":1631615724},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2427,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2427,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002232"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":186,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.681074+0000","last_change":"2024-08-17T17:17:32.495652+0000","last_active":"2024-08-17T17:23:26.681074+0000","last_peered":"2024-08-17T17:23:26.681074+0000","last_clean":"2024-08-17T17:23:26.681074+0000","last_became_active":"2024-08-17T17:17:32.495218+0000","last_became_peered":"2024-08-17T17:17:32.495218+0000","last_unstale":"2024-08-17T17:23:26.681074+0000","last_undegraded":"2024-08-17T17:23:26.681074+0000","last_fullsized":"2024-08-17T17:23:26.681074+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:11:45.821282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":143,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.671388+0000","last_change":"2024-08-17T17:18:59.424887+0000","last_active":"2024-08-17T17:23:25.671388+0000","last_peered":"2024-08-17T17:23:25.671388+0000","last_clean":"2024-08-17T17:23:25.671388+0000","last_became_active":"2024-08-17T17:18:59.411539+0000","last_became_peered":"2024-08-17T17:18:59.411539+0000","last_unstale":"2024-08-17T17:23:25.671388+0000","last_undegraded":"2024-08-17T17:23:25.671388+0000","last_fullsized":"2024-08-17T17:23:25.671388+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:46:20.082241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":123,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.671450+0000","last_change":"2024-08-17T17:18:59.403259+0000","last_active":"2024-08-17T17:23:25.671450+0000","last_peered":"2024-08-17T17:23:25.671450+0000","last_clean":"2024-08-17T17:23:25.671450+0000","last_became_active":"2024-08-17T17:18:59.402706+0000","last_became_peered":"2024-08-17T17:18:59.402706+0000","last_unstale":"2024-08-17T17:23:25.671450+0000","last_undegraded":"2024-08-17T17:23:25.671450+0000","last_fullsized":"2024-08-17T17:23:25.671450+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:02:12.520417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":47,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.671039+0000","last_change":"2024-08-17T17:19:57.500646+0000","last_active":"2024-08-17T17:23:25.671039+0000","last_peered":"2024-08-17T17:23:25.671039+0000","last_clean":"2024-08-17T17:23:25.671039+0000","last_became_active":"2024-08-17T17:19:57.500332+0000","last_became_peered":"2024-08-17T17:19:57.500332+0000","last_unstale":"2024-08-17T17:23:25.671039+0000","last_undegraded":"2024-08-17T17:23:25.671039+0000","last_fullsized":"2024-08-17T17:23:25.671039+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:13:40.176690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":161,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.681001+0000","last_change":"2024-08-17T17:19:57.506750+0000","last_active":"2024-08-17T17:23:26.681001+0000","last_peered":"2024-08-17T17:23:26.681001+0000","last_clean":"2024-08-17T17:23:26.681001+0000","last_became_active":"2024-08-17T17:19:57.506360+0000","last_became_peered":"2024-08-17T17:19:57.506360+0000","last_unstale":"2024-08-17T17:23:26.681001+0000","last_undegraded":"2024-08-17T17:23:26.681001+0000","last_fullsized":"2024-08-17T17:23:26.681001+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:50:35.750066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":118,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.672572+0000","last_change":"2024-08-17T17:20:49.258206+0000","last_active":"2024-08-17T17:23:25.672572+0000","last_peered":"2024-08-17T17:23:25.672572+0000","last_clean":"2024-08-17T17:23:25.672572+0000","last_became_active":"2024-08-17T17:20:49.257761+0000","last_became_peered":"2024-08-17T17:20:49.257761+0000","last_unstale":"2024-08-17T17:23:25.672572+0000","last_undegraded":"2024-08-17T17:23:25.672572+0000","last_fullsized":"2024-08-17T17:23:25.672572+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:38:35.363748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":200,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.680612+0000","last_change":"2024-08-17T17:19:57.514773+0000","last_active":"2024-08-17T17:23:26.680612+0000","last_peered":"2024-08-17T17:23:26.680612+0000","last_clean":"2024-08-17T17:23:26.680612+0000","last_became_active":"2024-08-17T17:19:57.513839+0000","last_became_peered":"2024-08-17T17:19:57.513839+0000","last_unstale":"2024-08-17T17:23:26.680612+0000","last_undegraded":"2024-08-17T17:23:26.680612+0000","last_fullsized":"2024-08-17T17:23:26.680612+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:42:48.063169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":173,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.741841+0000","last_change":"2024-08-17T17:19:57.520685+0000","last_active":"2024-08-17T17:23:25.741841+0000","last_peered":"2024-08-17T17:23:25.741841+0000","last_clean":"2024-08-17T17:23:25.741841+0000","last_became_active":"2024-08-17T17:19:57.508226+0000","last_became_peered":"2024-08-17T17:19:57.508226+0000","last_unstale":"2024-08-17T17:23:25.741841+0000","last_undegraded":"2024-08-17T17:23:25.741841+0000","last_fullsized":"2024-08-17T17:23:25.741841+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:17:42.506273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"162'12620","reported_seq":25609,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.672527+0000","last_change":"2024-08-17T17:23:19.660613+0000","last_active":"2024-08-17T17:23:25.672527+0000","last_peered":"2024-08-17T17:23:25.672527+0000","last_clean":"2024-08-17T17:23:25.672527+0000","last_became_active":"2024-08-17T17:23:19.632401+0000","last_became_peered":"2024-08-17T17:23:19.632401+0000","last_unstale":"2024-08-17T17:23:25.672527+0000","last_undegraded":"2024-08-17T17:23:25.672527+0000","last_fullsized":"2024-08-17T17:23:25.672527+0000","mapping_epoch":163,"log_start":"139'6573","ondisk_log_start":"139'6573","created":111,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":6047,"log_dups_size":0,"ondisk_log_size":6047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:52:31.234302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12620,"num_write_kb":403840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6310,"num_bytes_recovered":342032384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":65,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.671378+0000","last_change":"2024-08-17T17:20:49.297856+0000","last_active":"2024-08-17T17:23:25.671378+0000","last_peered":"2024-08-17T17:23:25.671378+0000","last_clean":"2024-08-17T17:23:25.671378+0000","last_became_active":"2024-08-17T17:20:49.261859+0000","last_became_peered":"2024-08-17T17:20:49.261859+0000","last_unstale":"2024-08-17T17:23:25.671378+0000","last_undegraded":"2024-08-17T17:23:25.671378+0000","last_fullsized":"2024-08-17T17:23:25.671378+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:20:51.487220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":57,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.670946+0000","last_change":"2024-08-17T17:20:49.261329+0000","last_active":"2024-08-17T17:23:25.670946+0000","last_peered":"2024-08-17T17:23:25.670946+0000","last_clean":"2024-08-17T17:23:25.670946+0000","last_became_active":"2024-08-17T17:20:49.261070+0000","last_became_peered":"2024-08-17T17:20:49.261070+0000","last_unstale":"2024-08-17T17:23:25.670946+0000","last_undegraded":"2024-08-17T17:23:25.670946+0000","last_fullsized":"2024-08-17T17:23:25.670946+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:07:17.916979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"162'12722","reported_seq":12850,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.678852+0000","last_change":"2024-08-17T17:20:14.440439+0000","last_active":"2024-08-17T17:23:26.678852+0000","last_peered":"2024-08-17T17:23:26.678852+0000","last_clean":"2024-08-17T17:23:26.678852+0000","last_became_active":"2024-08-17T17:20:14.430994+0000","last_became_peered":"2024-08-17T17:20:14.430994+0000","last_unstale":"2024-08-17T17:23:26.678852+0000","last_undegraded":"2024-08-17T17:23:26.678852+0000","last_fullsized":"2024-08-17T17:23:26.678852+0000","mapping_epoch":135,"log_start":"125'3000","ondisk_log_start":"125'3000","created":111,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":9722,"log_dups_size":0,"ondisk_log_size":9722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:01:41.276551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12722,"num_write_kb":407104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":51,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.671821+0000","last_change":"2024-08-17T17:19:57.527977+0000","last_active":"2024-08-17T17:23:25.671821+0000","last_peered":"2024-08-17T17:23:25.671821+0000","last_clean":"2024-08-17T17:23:25.671821+0000","last_became_active":"2024-08-17T17:19:57.505766+0000","last_became_peered":"2024-08-17T17:19:57.505766+0000","last_unstale":"2024-08-17T17:23:25.671821+0000","last_undegraded":"2024-08-17T17:23:25.671821+0000","last_fullsized":"2024-08-17T17:23:25.671821+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:05:23.673641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":141,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.681520+0000","last_change":"2024-08-17T17:20:49.271540+0000","last_active":"2024-08-17T17:23:26.681520+0000","last_peered":"2024-08-17T17:23:26.681520+0000","last_clean":"2024-08-17T17:23:26.681520+0000","last_became_active":"2024-08-17T17:20:49.271130+0000","last_became_peered":"2024-08-17T17:20:49.271130+0000","last_unstale":"2024-08-17T17:23:26.681520+0000","last_undegraded":"2024-08-17T17:23:26.681520+0000","last_fullsized":"2024-08-17T17:23:26.681520+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:05.386705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"162'12888","reported_seq":21644,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.714173+0000","last_change":"2024-08-17T17:23:26.714130+0000","last_active":"2024-08-17T17:23:26.714173+0000","last_peered":"2024-08-17T17:23:26.714173+0000","last_clean":"2024-08-17T17:23:26.714173+0000","last_became_active":"2024-08-17T17:23:26.682936+0000","last_became_peered":"2024-08-17T17:23:26.682936+0000","last_unstale":"2024-08-17T17:23:26.714173+0000","last_undegraded":"2024-08-17T17:23:26.714173+0000","last_fullsized":"2024-08-17T17:23:26.714173+0000","mapping_epoch":166,"log_start":"126'3200","ondisk_log_start":"126'3200","created":111,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":9688,"log_dups_size":0,"ondisk_log_size":9688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:09:53.693422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12888,"num_write_kb":412416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4308,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":123,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.671539+0000","last_change":"2024-08-17T17:18:59.420539+0000","last_active":"2024-08-17T17:23:25.671539+0000","last_peered":"2024-08-17T17:23:25.671539+0000","last_clean":"2024-08-17T17:23:25.671539+0000","last_became_active":"2024-08-17T17:18:59.404078+0000","last_became_peered":"2024-08-17T17:18:59.404078+0000","last_unstale":"2024-08-17T17:23:25.671539+0000","last_undegraded":"2024-08-17T17:23:25.671539+0000","last_fullsized":"2024-08-17T17:23:25.671539+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:48:47.883684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":174,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.682121+0000","last_change":"2024-08-17T17:19:57.504404+0000","last_active":"2024-08-17T17:23:26.682121+0000","last_peered":"2024-08-17T17:23:26.682121+0000","last_clean":"2024-08-17T17:23:26.682121+0000","last_became_active":"2024-08-17T17:19:57.504036+0000","last_became_peered":"2024-08-17T17:19:57.504036+0000","last_unstale":"2024-08-17T17:23:26.682121+0000","last_undegraded":"2024-08-17T17:23:26.682121+0000","last_fullsized":"2024-08-17T17:23:26.682121+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:14:07.727290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"162'12486","reported_seq":21032,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.680295+0000","last_change":"2024-08-17T17:22:16.915046+0000","last_active":"2024-08-17T17:23:26.680295+0000","last_peered":"2024-08-17T17:23:26.680295+0000","last_clean":"2024-08-17T17:23:26.680295+0000","last_became_active":"2024-08-17T17:20:15.438280+0000","last_became_peered":"2024-08-17T17:20:15.438280+0000","last_unstale":"2024-08-17T17:23:26.680295+0000","last_undegraded":"2024-08-17T17:23:26.680295+0000","last_fullsized":"2024-08-17T17:23:26.680295+0000","mapping_epoch":136,"log_start":"139'6425","ondisk_log_start":"139'6425","created":111,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":6061,"log_dups_size":0,"ondisk_log_size":6061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:33:56.173311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12486,"num_write_kb":399552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4204,"num_bytes_recovered":275513344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":125,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.741711+0000","last_change":"2024-08-17T17:17:32.495428+0000","last_active":"2024-08-17T17:23:25.741711+0000","last_peered":"2024-08-17T17:23:25.741711+0000","last_clean":"2024-08-17T17:23:25.741711+0000","last_became_active":"2024-08-17T17:17:32.494978+0000","last_became_peered":"2024-08-17T17:17:32.494978+0000","last_unstale":"2024-08-17T17:23:25.741711+0000","last_undegraded":"2024-08-17T17:23:25.741711+0000","last_fullsized":"2024-08-17T17:23:25.741711+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:42:24.647812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":189,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.680409+0000","last_change":"2024-08-17T17:20:49.272126+0000","last_active":"2024-08-17T17:23:26.680409+0000","last_peered":"2024-08-17T17:23:26.680409+0000","last_clean":"2024-08-17T17:23:26.680409+0000","last_became_active":"2024-08-17T17:20:49.271185+0000","last_became_peered":"2024-08-17T17:20:49.271185+0000","last_unstale":"2024-08-17T17:23:26.680409+0000","last_undegraded":"2024-08-17T17:23:26.680409+0000","last_fullsized":"2024-08-17T17:23:26.680409+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:43:58.184939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"162'12758","reported_seq":12862,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.671424+0000","last_change":"2024-08-17T17:20:14.437947+0000","last_active":"2024-08-17T17:23:25.671424+0000","last_peered":"2024-08-17T17:23:25.671424+0000","last_clean":"2024-08-17T17:23:25.671424+0000","last_became_active":"2024-08-17T17:20:14.428646+0000","last_became_peered":"2024-08-17T17:20:14.428646+0000","last_unstale":"2024-08-17T17:23:25.671424+0000","last_undegraded":"2024-08-17T17:23:25.671424+0000","last_fullsized":"2024-08-17T17:23:25.671424+0000","mapping_epoch":135,"log_start":"140'6663","ondisk_log_start":"140'6663","created":111,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":6095,"log_dups_size":0,"ondisk_log_size":6095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:06:56.380235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12758,"num_write_kb":408193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":206,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.679511+0000","last_change":"2024-08-17T17:20:49.295255+0000","last_active":"2024-08-17T17:23:26.679511+0000","last_peered":"2024-08-17T17:23:26.679511+0000","last_clean":"2024-08-17T17:23:26.679511+0000","last_became_active":"2024-08-17T17:20:49.262267+0000","last_became_peered":"2024-08-17T17:20:49.262267+0000","last_unstale":"2024-08-17T17:23:26.679511+0000","last_undegraded":"2024-08-17T17:23:26.679511+0000","last_fullsized":"2024-08-17T17:23:26.679511+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:22:30.749208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":163,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.681282+0000","last_change":"2024-08-17T17:20:49.280596+0000","last_active":"2024-08-17T17:23:26.681282+0000","last_peered":"2024-08-17T17:23:26.681282+0000","last_clean":"2024-08-17T17:23:26.681282+0000","last_became_active":"2024-08-17T17:20:49.279553+0000","last_became_peered":"2024-08-17T17:20:49.279553+0000","last_unstale":"2024-08-17T17:23:26.681282+0000","last_undegraded":"2024-08-17T17:23:26.681282+0000","last_fullsized":"2024-08-17T17:23:26.681282+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:13:24.124392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"162'12678","reported_seq":12806,"reported_epoch":167,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T17:23:26.682621+0000","last_change":"2024-08-17T17:20:50.393430+0000","last_active":"2024-08-17T17:23:26.682621+0000","last_peered":"2024-08-17T17:23:26.682621+0000","last_clean":"2024-08-17T17:20:48.214722+0000","last_became_active":"2024-08-17T17:20:50.375136+0000","last_became_peered":"2024-08-17T17:20:50.375136+0000","last_unstale":"2024-08-17T17:23:26.682621+0000","last_undegraded":"2024-08-17T17:23:26.682621+0000","last_fullsized":"2024-08-17T17:23:26.682621+0000","mapping_epoch":141,"log_start":"145'7457","ondisk_log_start":"145'7457","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":5221,"log_dups_size":0,"ondisk_log_size":5221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12678,"num_write_kb":405696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,4],"avail_no_missing":["1","0","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":115,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.682217+0000","last_change":"2024-08-17T17:20:49.276769+0000","last_active":"2024-08-17T17:23:26.682217+0000","last_peered":"2024-08-17T17:23:26.682217+0000","last_clean":"2024-08-17T17:23:26.682217+0000","last_became_active":"2024-08-17T17:20:49.276498+0000","last_became_peered":"2024-08-17T17:20:49.276498+0000","last_unstale":"2024-08-17T17:23:26.682217+0000","last_undegraded":"2024-08-17T17:23:26.682217+0000","last_fullsized":"2024-08-17T17:23:26.682217+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:39:05.419440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":146,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.671375+0000","last_change":"2024-08-17T17:20:14.455738+0000","last_active":"2024-08-17T17:23:25.671375+0000","last_peered":"2024-08-17T17:23:25.671375+0000","last_clean":"2024-08-17T17:23:25.671375+0000","last_became_active":"2024-08-17T17:20:14.455469+0000","last_became_peered":"2024-08-17T17:20:14.455469+0000","last_unstale":"2024-08-17T17:23:25.671375+0000","last_undegraded":"2024-08-17T17:23:25.671375+0000","last_fullsized":"2024-08-17T17:23:25.671375+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:27:28.424336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"162'12620","reported_seq":12694,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.741393+0000","last_change":"2024-08-17T17:18:59.413691+0000","last_active":"2024-08-17T17:23:25.741393+0000","last_peered":"2024-08-17T17:23:25.741393+0000","last_clean":"2024-08-17T17:23:25.741393+0000","last_became_active":"2024-08-17T17:18:59.413358+0000","last_became_peered":"2024-08-17T17:18:59.413358+0000","last_unstale":"2024-08-17T17:23:25.741393+0000","last_undegraded":"2024-08-17T17:23:25.741393+0000","last_fullsized":"2024-08-17T17:23:25.741393+0000","mapping_epoch":114,"log_start":"123'2600","ondisk_log_start":"123'2600","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:51:20.829898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12620,"num_write_kb":403840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":100,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.672536+0000","last_change":"2024-08-17T17:20:14.431140+0000","last_active":"2024-08-17T17:23:25.672536+0000","last_peered":"2024-08-17T17:23:25.672536+0000","last_clean":"2024-08-17T17:23:25.672536+0000","last_became_active":"2024-08-17T17:20:14.430839+0000","last_became_peered":"2024-08-17T17:20:14.430839+0000","last_unstale":"2024-08-17T17:23:25.672536+0000","last_undegraded":"2024-08-17T17:23:25.672536+0000","last_fullsized":"2024-08-17T17:23:25.672536+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:08:16.303616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":138,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.682626+0000","last_change":"2024-08-17T17:20:49.272325+0000","last_active":"2024-08-17T17:23:26.682626+0000","last_peered":"2024-08-17T17:23:26.682626+0000","last_clean":"2024-08-17T17:23:26.682626+0000","last_became_active":"2024-08-17T17:20:49.272002+0000","last_became_peered":"2024-08-17T17:20:49.272002+0000","last_unstale":"2024-08-17T17:23:26.682626+0000","last_undegraded":"2024-08-17T17:23:26.682626+0000","last_fullsized":"2024-08-17T17:23:26.682626+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:47:32.474887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"162'12796","reported_seq":21490,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.741639+0000","last_change":"2024-08-17T17:23:03.528083+0000","last_active":"2024-08-17T17:23:25.741639+0000","last_peered":"2024-08-17T17:23:25.741639+0000","last_clean":"2024-08-17T17:23:25.741639+0000","last_became_active":"2024-08-17T17:23:03.498966+0000","last_became_peered":"2024-08-17T17:23:03.498966+0000","last_unstale":"2024-08-17T17:23:25.741639+0000","last_undegraded":"2024-08-17T17:23:25.741639+0000","last_fullsized":"2024-08-17T17:23:25.741639+0000","mapping_epoch":161,"log_start":"123'2700","ondisk_log_start":"123'2700","created":111,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:41:21.801736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12796,"num_write_kb":409472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4303,"num_bytes_recovered":160628736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":201,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.680085+0000","last_change":"2024-08-17T17:19:57.520745+0000","last_active":"2024-08-17T17:23:26.680085+0000","last_peered":"2024-08-17T17:23:26.680085+0000","last_clean":"2024-08-17T17:23:26.680085+0000","last_became_active":"2024-08-17T17:19:57.520152+0000","last_became_peered":"2024-08-17T17:19:57.520152+0000","last_unstale":"2024-08-17T17:23:26.680085+0000","last_undegraded":"2024-08-17T17:23:26.680085+0000","last_fullsized":"2024-08-17T17:23:26.680085+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:19:39.495644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":177,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.682283+0000","last_change":"2024-08-17T17:20:49.274372+0000","last_active":"2024-08-17T17:23:26.682283+0000","last_peered":"2024-08-17T17:23:26.682283+0000","last_clean":"2024-08-17T17:23:26.682283+0000","last_became_active":"2024-08-17T17:20:49.274023+0000","last_became_peered":"2024-08-17T17:20:49.274023+0000","last_unstale":"2024-08-17T17:23:26.682283+0000","last_undegraded":"2024-08-17T17:23:26.682283+0000","last_fullsized":"2024-08-17T17:23:26.682283+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:31:02.529766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"162'12546","reported_seq":20981,"reported_epoch":166,"state":"active+recovery_wait","last_fresh":"2024-08-17T17:23:25.671137+0000","last_change":"2024-08-17T17:22:23.962607+0000","last_active":"2024-08-17T17:23:25.671137+0000","last_peered":"2024-08-17T17:23:25.671137+0000","last_clean":"2024-08-17T17:19:56.496869+0000","last_became_active":"2024-08-17T17:22:23.898645+0000","last_became_peered":"2024-08-17T17:22:23.898645+0000","last_unstale":"2024-08-17T17:23:25.671137+0000","last_undegraded":"2024-08-17T17:23:25.671137+0000","last_fullsized":"2024-08-17T17:23:25.671137+0000","mapping_epoch":154,"log_start":"139'6516","ondisk_log_start":"139'6516","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":6030,"log_dups_size":0,"ondisk_log_size":6030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:56:04.784052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12546,"num_write_kb":401472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4158,"num_bytes_recovered":244842496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":["7","0","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":468,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.670868+0000","last_change":"2024-08-17T17:20:00.425796+0000","last_active":"2024-08-17T17:23:25.670868+0000","last_peered":"2024-08-17T17:23:25.670868+0000","last_clean":"2024-08-17T17:23:25.670868+0000","last_became_active":"2024-08-17T17:20:00.425388+0000","last_became_peered":"2024-08-17T17:20:00.425388+0000","last_unstale":"2024-08-17T17:23:25.670868+0000","last_undegraded":"2024-08-17T17:23:25.670868+0000","last_fullsized":"2024-08-17T17:23:25.670868+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:04:01.141401+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:12.168702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":739,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.741466+0000","last_change":"2024-08-17T17:20:00.434774+0000","last_active":"2024-08-17T17:23:25.741466+0000","last_peered":"2024-08-17T17:23:25.741466+0000","last_clean":"2024-08-17T17:23:25.741466+0000","last_became_active":"2024-08-17T17:20:00.434343+0000","last_became_peered":"2024-08-17T17:20:00.434343+0000","last_unstale":"2024-08-17T17:23:25.741466+0000","last_undegraded":"2024-08-17T17:23:25.741466+0000","last_fullsized":"2024-08-17T17:23:25.741466+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:04.995152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":152,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.680400+0000","last_change":"2024-08-17T17:20:49.263797+0000","last_active":"2024-08-17T17:23:26.680400+0000","last_peered":"2024-08-17T17:23:26.680400+0000","last_clean":"2024-08-17T17:23:26.680400+0000","last_became_active":"2024-08-17T17:20:49.263488+0000","last_became_peered":"2024-08-17T17:20:49.263488+0000","last_unstale":"2024-08-17T17:23:26.680400+0000","last_undegraded":"2024-08-17T17:23:26.680400+0000","last_fullsized":"2024-08-17T17:23:26.680400+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:50:22.123993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":66,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.671504+0000","last_change":"2024-08-17T17:18:59.417834+0000","last_active":"2024-08-17T17:23:25.671504+0000","last_peered":"2024-08-17T17:23:25.671504+0000","last_clean":"2024-08-17T17:23:25.671504+0000","last_became_active":"2024-08-17T17:18:59.417456+0000","last_became_peered":"2024-08-17T17:18:59.417456+0000","last_unstale":"2024-08-17T17:23:25.671504+0000","last_undegraded":"2024-08-17T17:23:25.671504+0000","last_fullsized":"2024-08-17T17:23:25.671504+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:22:45.101077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"162'12824","reported_seq":12948,"reported_epoch":167,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T17:23:26.681840+0000","last_change":"2024-08-17T17:20:50.369539+0000","last_active":"2024-08-17T17:23:26.681840+0000","last_peered":"2024-08-17T17:23:26.681840+0000","last_clean":"2024-08-17T17:20:48.214899+0000","last_became_active":"2024-08-17T17:20:50.362743+0000","last_became_peered":"2024-08-17T17:20:50.362743+0000","last_unstale":"2024-08-17T17:23:26.681840+0000","last_undegraded":"2024-08-17T17:23:26.681840+0000","last_fullsized":"2024-08-17T17:23:26.681840+0000","mapping_epoch":141,"log_start":"145'7623","ondisk_log_start":"145'7623","created":111,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":5201,"log_dups_size":0,"ondisk_log_size":5201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12824,"num_write_kb":410368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,4],"avail_no_missing":["1","0","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":757,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.671004+0000","last_change":"2024-08-17T17:16:26.958415+0000","last_active":"2024-08-17T17:23:25.671004+0000","last_peered":"2024-08-17T17:23:25.671004+0000","last_clean":"2024-08-17T17:23:25.671004+0000","last_became_active":"2024-08-17T17:16:26.957956+0000","last_became_peered":"2024-08-17T17:16:26.957956+0000","last_unstale":"2024-08-17T17:23:25.671004+0000","last_undegraded":"2024-08-17T17:23:25.671004+0000","last_fullsized":"2024-08-17T17:23:25.671004+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:22.478077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":188,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.679209+0000","last_change":"2024-08-17T17:18:59.415331+0000","last_active":"2024-08-17T17:23:26.679209+0000","last_peered":"2024-08-17T17:23:26.679209+0000","last_clean":"2024-08-17T17:23:26.679209+0000","last_became_active":"2024-08-17T17:18:59.415008+0000","last_became_peered":"2024-08-17T17:18:59.415008+0000","last_unstale":"2024-08-17T17:23:26.679209+0000","last_undegraded":"2024-08-17T17:23:26.679209+0000","last_fullsized":"2024-08-17T17:23:26.679209+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:22:38.076866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":118,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.741812+0000","last_change":"2024-08-17T17:21:50.562469+0000","last_active":"2024-08-17T17:23:25.741812+0000","last_peered":"2024-08-17T17:23:25.741812+0000","last_clean":"2024-08-17T17:23:25.741812+0000","last_became_active":"2024-08-17T17:21:50.562136+0000","last_became_peered":"2024-08-17T17:21:50.562136+0000","last_unstale":"2024-08-17T17:23:25.741812+0000","last_undegraded":"2024-08-17T17:23:25.741812+0000","last_fullsized":"2024-08-17T17:23:25.741812+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:48:33.499542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"162'12892","reported_seq":12993,"reported_epoch":166,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:23:25.742164+0000","last_change":"2024-08-17T17:21:50.595337+0000","last_active":"2024-08-17T17:23:25.742164+0000","last_peered":"2024-08-17T17:23:25.742164+0000","last_clean":"2024-08-17T17:19:56.456745+0000","last_became_active":"2024-08-17T17:21:50.565097+0000","last_became_peered":"2024-08-17T17:21:50.565097+0000","last_unstale":"2024-08-17T17:23:25.742164+0000","last_undegraded":"2024-08-17T17:21:50.354726+0000","last_fullsized":"2024-08-17T17:21:50.338039+0000","mapping_epoch":150,"log_start":"123'2800","ondisk_log_start":"123'2800","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:39:43.756591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12892,"num_write_kb":412544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":-2130}],"blocked_by":[],"up_primary":2,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":576,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.681726+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:23:26.681726+0000","last_peered":"2024-08-17T17:23:26.681726+0000","last_clean":"2024-08-17T17:23:26.681726+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:23:26.681726+0000","last_undegraded":"2024-08-17T17:23:26.681726+0000","last_fullsized":"2024-08-17T17:23:26.681726+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":184,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.681947+0000","last_change":"2024-08-17T17:19:57.515564+0000","last_active":"2024-08-17T17:23:26.681947+0000","last_peered":"2024-08-17T17:23:26.681947+0000","last_clean":"2024-08-17T17:23:26.681947+0000","last_became_active":"2024-08-17T17:19:57.515142+0000","last_became_peered":"2024-08-17T17:19:57.515142+0000","last_unstale":"2024-08-17T17:23:26.681947+0000","last_undegraded":"2024-08-17T17:23:26.681947+0000","last_fullsized":"2024-08-17T17:23:26.681947+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:59:33.506176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":208,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.682103+0000","last_change":"2024-08-17T17:19:57.506668+0000","last_active":"2024-08-17T17:23:26.682103+0000","last_peered":"2024-08-17T17:23:26.682103+0000","last_clean":"2024-08-17T17:23:26.682103+0000","last_became_active":"2024-08-17T17:19:57.505453+0000","last_became_peered":"2024-08-17T17:19:57.505453+0000","last_unstale":"2024-08-17T17:23:26.682103+0000","last_undegraded":"2024-08-17T17:23:26.682103+0000","last_fullsized":"2024-08-17T17:23:26.682103+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:28:48.696602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"162'12752","reported_seq":21222,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.681628+0000","last_change":"2024-08-17T17:22:44.149376+0000","last_active":"2024-08-17T17:23:26.681628+0000","last_peered":"2024-08-17T17:23:26.681628+0000","last_clean":"2024-08-17T17:23:26.681628+0000","last_became_active":"2024-08-17T17:22:44.122999+0000","last_became_peered":"2024-08-17T17:22:44.122999+0000","last_unstale":"2024-08-17T17:23:26.681628+0000","last_undegraded":"2024-08-17T17:23:26.681628+0000","last_fullsized":"2024-08-17T17:23:26.681628+0000","mapping_epoch":158,"log_start":"144'7485","ondisk_log_start":"144'7485","created":111,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":5267,"log_dups_size":0,"ondisk_log_size":5267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:19:30.558444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12752,"num_write_kb":408064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4162,"num_bytes_recovered":155844608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":1077,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.741478+0000","last_change":"2024-08-17T17:20:00.433870+0000","last_active":"2024-08-17T17:23:25.741478+0000","last_peered":"2024-08-17T17:23:25.741478+0000","last_clean":"2024-08-17T17:23:25.741478+0000","last_became_active":"2024-08-17T17:20:00.433386+0000","last_became_peered":"2024-08-17T17:20:00.433386+0000","last_unstale":"2024-08-17T17:23:25.741478+0000","last_undegraded":"2024-08-17T17:23:25.741478+0000","last_fullsized":"2024-08-17T17:23:25.741478+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:01:11.848862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1392,"num_read_kb":2863,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":110,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.680154+0000","last_change":"2024-08-17T17:20:49.272407+0000","last_active":"2024-08-17T17:23:26.680154+0000","last_peered":"2024-08-17T17:23:26.680154+0000","last_clean":"2024-08-17T17:23:26.680154+0000","last_became_active":"2024-08-17T17:20:49.271749+0000","last_became_peered":"2024-08-17T17:20:49.271749+0000","last_unstale":"2024-08-17T17:23:26.680154+0000","last_undegraded":"2024-08-17T17:23:26.680154+0000","last_fullsized":"2024-08-17T17:23:26.680154+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:36:01.443457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":181,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.682520+0000","last_change":"2024-08-17T17:17:32.499715+0000","last_active":"2024-08-17T17:23:26.682520+0000","last_peered":"2024-08-17T17:23:26.682520+0000","last_clean":"2024-08-17T17:23:26.682520+0000","last_became_active":"2024-08-17T17:17:32.499412+0000","last_became_peered":"2024-08-17T17:17:32.499412+0000","last_unstale":"2024-08-17T17:23:26.682520+0000","last_undegraded":"2024-08-17T17:23:26.682520+0000","last_fullsized":"2024-08-17T17:23:26.682520+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:39:38.456970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"162'12884","reported_seq":21623,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.742324+0000","last_change":"2024-08-17T17:22:26.352693+0000","last_active":"2024-08-17T17:23:25.742324+0000","last_peered":"2024-08-17T17:23:25.742324+0000","last_clean":"2024-08-17T17:23:25.742324+0000","last_became_active":"2024-08-17T17:22:26.327438+0000","last_became_peered":"2024-08-17T17:22:26.327438+0000","last_unstale":"2024-08-17T17:23:25.742324+0000","last_undegraded":"2024-08-17T17:23:25.742324+0000","last_fullsized":"2024-08-17T17:23:25.742324+0000","mapping_epoch":156,"log_start":"128'4000","ondisk_log_start":"128'4000","created":111,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":8884,"log_dups_size":0,"ondisk_log_size":8884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:05:05.434890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12884,"num_write_kb":412288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4315,"num_bytes_recovered":248709120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":142,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.741423+0000","last_change":"2024-08-17T17:17:32.495728+0000","last_active":"2024-08-17T17:23:25.741423+0000","last_peered":"2024-08-17T17:23:25.741423+0000","last_clean":"2024-08-17T17:23:25.741423+0000","last_became_active":"2024-08-17T17:17:32.495357+0000","last_became_peered":"2024-08-17T17:17:32.495357+0000","last_unstale":"2024-08-17T17:23:25.741423+0000","last_undegraded":"2024-08-17T17:23:25.741423+0000","last_fullsized":"2024-08-17T17:23:25.741423+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:27:45.561567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":417,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.682727+0000","last_change":"2024-08-17T17:16:26.964621+0000","last_active":"2024-08-17T17:23:26.682727+0000","last_peered":"2024-08-17T17:23:26.682727+0000","last_clean":"2024-08-17T17:23:26.682727+0000","last_became_active":"2024-08-17T17:16:26.964058+0000","last_became_peered":"2024-08-17T17:16:26.964058+0000","last_unstale":"2024-08-17T17:23:26.682727+0000","last_undegraded":"2024-08-17T17:23:26.682727+0000","last_fullsized":"2024-08-17T17:23:26.682727+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:00:04.465644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":173,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.682303+0000","last_change":"2024-08-17T17:19:57.513601+0000","last_active":"2024-08-17T17:23:26.682303+0000","last_peered":"2024-08-17T17:23:26.682303+0000","last_clean":"2024-08-17T17:23:26.682303+0000","last_became_active":"2024-08-17T17:19:57.513260+0000","last_became_peered":"2024-08-17T17:19:57.513260+0000","last_unstale":"2024-08-17T17:23:26.682303+0000","last_undegraded":"2024-08-17T17:23:26.682303+0000","last_fullsized":"2024-08-17T17:23:26.682303+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:36:00.355650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"162'13098","reported_seq":13162,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.741995+0000","last_change":"2024-08-17T17:18:56.578793+0000","last_active":"2024-08-17T17:23:25.741995+0000","last_peered":"2024-08-17T17:23:25.741995+0000","last_clean":"2024-08-17T17:23:25.741995+0000","last_became_active":"2024-08-17T17:18:56.578433+0000","last_became_peered":"2024-08-17T17:18:56.578433+0000","last_unstale":"2024-08-17T17:23:25.741995+0000","last_undegraded":"2024-08-17T17:23:25.741995+0000","last_fullsized":"2024-08-17T17:23:25.741995+0000","mapping_epoch":111,"log_start":"128'4200","ondisk_log_start":"128'4200","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":8898,"log_dups_size":0,"ondisk_log_size":8898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:25:02.009751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13098,"num_write_kb":419136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":198,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.682628+0000","last_change":"2024-08-17T17:21:50.443989+0000","last_active":"2024-08-17T17:23:26.682628+0000","last_peered":"2024-08-17T17:23:26.682628+0000","last_clean":"2024-08-17T17:23:26.682628+0000","last_became_active":"2024-08-17T17:21:50.443652+0000","last_became_peered":"2024-08-17T17:21:50.443652+0000","last_unstale":"2024-08-17T17:23:26.682628+0000","last_undegraded":"2024-08-17T17:23:26.682628+0000","last_fullsized":"2024-08-17T17:23:26.682628+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:00:21.916690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":552,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.671587+0000","last_change":"2024-08-17T17:22:44.122893+0000","last_active":"2024-08-17T17:23:25.671587+0000","last_peered":"2024-08-17T17:23:25.671587+0000","last_clean":"2024-08-17T17:23:25.671587+0000","last_became_active":"2024-08-17T17:22:44.122354+0000","last_became_peered":"2024-08-17T17:22:44.122354+0000","last_unstale":"2024-08-17T17:23:25.671587+0000","last_undegraded":"2024-08-17T17:23:25.671587+0000","last_fullsized":"2024-08-17T17:23:25.671587+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:55:31.419118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":171,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.680314+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:23:26.680314+0000","last_peered":"2024-08-17T17:23:26.680314+0000","last_clean":"2024-08-17T17:23:26.680314+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:23:26.680314+0000","last_undegraded":"2024-08-17T17:23:26.680314+0000","last_fullsized":"2024-08-17T17:23:26.680314+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"162'12794","reported_seq":18334,"reported_epoch":167,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-17T17:23:30.981267+0000","last_change":"2024-08-17T17:23:25.234777+0000","last_active":"2024-08-17T17:23:30.981267+0000","last_peered":"2024-08-17T17:23:30.981267+0000","last_clean":"2024-08-17T17:19:56.461183+0000","last_became_active":"2024-08-17T17:19:58.704346+0000","last_became_peered":"2024-08-17T17:19:58.704346+0000","last_unstale":"2024-08-17T17:23:30.981267+0000","last_undegraded":"2024-08-17T17:19:58.512446+0000","last_fullsized":"2024-08-17T17:19:58.480807+0000","mapping_epoch":130,"log_start":"140'6703","ondisk_log_start":"140'6703","created":111,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":6091,"log_dups_size":0,"ondisk_log_size":6091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3250,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12794,"num_write_kb":409408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2701,"num_bytes_recovered":107806720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,2],"avail_no_missing":["3","0","2","4"],"object_location_counts":[{"shards":"2,3","objects":-3250}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":206,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.680772+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:23:26.680772+0000","last_peered":"2024-08-17T17:23:26.680772+0000","last_clean":"2024-08-17T17:23:26.680772+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:23:26.680772+0000","last_undegraded":"2024-08-17T17:23:26.680772+0000","last_fullsized":"2024-08-17T17:23:26.680772+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":726,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.682653+0000","last_change":"2024-08-17T17:23:19.633786+0000","last_active":"2024-08-17T17:23:26.682653+0000","last_peered":"2024-08-17T17:23:26.682653+0000","last_clean":"2024-08-17T17:23:26.682653+0000","last_became_active":"2024-08-17T17:23:19.633230+0000","last_became_peered":"2024-08-17T17:23:19.633230+0000","last_unstale":"2024-08-17T17:23:26.682653+0000","last_undegraded":"2024-08-17T17:23:26.682653+0000","last_fullsized":"2024-08-17T17:23:26.682653+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:30:50.377391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":196,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.679947+0000","last_change":"2024-08-17T17:19:57.520194+0000","last_active":"2024-08-17T17:23:26.679947+0000","last_peered":"2024-08-17T17:23:26.679947+0000","last_clean":"2024-08-17T17:23:26.679947+0000","last_became_active":"2024-08-17T17:19:57.519559+0000","last_became_peered":"2024-08-17T17:19:57.519559+0000","last_unstale":"2024-08-17T17:23:26.679947+0000","last_undegraded":"2024-08-17T17:23:26.679947+0000","last_fullsized":"2024-08-17T17:23:26.679947+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:48:33.839524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.0","version":"162'12752","reported_seq":12884,"reported_epoch":167,"state":"active+remapped+backfill_wait","last_fresh":"2024-08-17T17:23:26.679536+0000","last_change":"2024-08-17T17:21:50.550958+0000","last_active":"2024-08-17T17:23:26.679536+0000","last_peered":"2024-08-17T17:23:26.679536+0000","last_clean":"2024-08-17T17:20:48.235622+0000","last_became_active":"2024-08-17T17:21:50.527964+0000","last_became_peered":"2024-08-17T17:21:50.527964+0000","last_unstale":"2024-08-17T17:23:26.679536+0000","last_undegraded":"2024-08-17T17:23:26.679536+0000","last_fullsized":"2024-08-17T17:23:26.679536+0000","mapping_epoch":150,"log_start":"142'6743","ondisk_log_start":"142'6743","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:18:55.541598+0000","last_clean_scrub_stamp":"2024-08-17T17:18:55.541598+0000","objects_scrubbed":0,"log_size":6009,"log_dups_size":0,"ondisk_log_size":6009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12752,"num_write_kb":408064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,4],"avail_no_missing":["3","1","2","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":124,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.682514+0000","last_change":"2024-08-17T17:20:49.289009+0000","last_active":"2024-08-17T17:23:26.682514+0000","last_peered":"2024-08-17T17:23:26.682514+0000","last_clean":"2024-08-17T17:23:26.682514+0000","last_became_active":"2024-08-17T17:20:49.265734+0000","last_became_peered":"2024-08-17T17:20:49.265734+0000","last_unstale":"2024-08-17T17:23:26.682514+0000","last_undegraded":"2024-08-17T17:23:26.682514+0000","last_fullsized":"2024-08-17T17:23:26.682514+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:45:07.444277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":376,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.742394+0000","last_change":"2024-08-17T17:18:59.448779+0000","last_active":"2024-08-17T17:23:25.742394+0000","last_peered":"2024-08-17T17:23:25.742394+0000","last_clean":"2024-08-17T17:23:25.742394+0000","last_became_active":"2024-08-17T17:18:59.410280+0000","last_became_peered":"2024-08-17T17:18:59.410280+0000","last_unstale":"2024-08-17T17:23:25.742394+0000","last_undegraded":"2024-08-17T17:23:25.742394+0000","last_fullsized":"2024-08-17T17:23:25.742394+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:09:46.989594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":66,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.741247+0000","last_change":"2024-08-17T17:18:59.417183+0000","last_active":"2024-08-17T17:23:25.741247+0000","last_peered":"2024-08-17T17:23:25.741247+0000","last_clean":"2024-08-17T17:23:25.741247+0000","last_became_active":"2024-08-17T17:18:59.416843+0000","last_became_peered":"2024-08-17T17:18:59.416843+0000","last_unstale":"2024-08-17T17:23:25.741247+0000","last_undegraded":"2024-08-17T17:23:25.741247+0000","last_fullsized":"2024-08-17T17:23:25.741247+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:29:09.336335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":87,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.742409+0000","last_change":"2024-08-17T17:20:49.258048+0000","last_active":"2024-08-17T17:23:25.742409+0000","last_peered":"2024-08-17T17:23:25.742409+0000","last_clean":"2024-08-17T17:23:25.742409+0000","last_became_active":"2024-08-17T17:20:49.257672+0000","last_became_peered":"2024-08-17T17:20:49.257672+0000","last_unstale":"2024-08-17T17:23:25.742409+0000","last_undegraded":"2024-08-17T17:23:25.742409+0000","last_fullsized":"2024-08-17T17:23:25.742409+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:08:11.734316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":126,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.671671+0000","last_change":"2024-08-17T17:18:59.430969+0000","last_active":"2024-08-17T17:23:25.671671+0000","last_peered":"2024-08-17T17:23:25.671671+0000","last_clean":"2024-08-17T17:23:25.671671+0000","last_became_active":"2024-08-17T17:18:59.413475+0000","last_became_peered":"2024-08-17T17:18:59.413475+0000","last_unstale":"2024-08-17T17:23:25.671671+0000","last_undegraded":"2024-08-17T17:23:25.671671+0000","last_fullsized":"2024-08-17T17:23:25.671671+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:36:33.732645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":184,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.682212+0000","last_change":"2024-08-17T17:19:57.512029+0000","last_active":"2024-08-17T17:23:26.682212+0000","last_peered":"2024-08-17T17:23:26.682212+0000","last_clean":"2024-08-17T17:23:26.682212+0000","last_became_active":"2024-08-17T17:19:57.511671+0000","last_became_peered":"2024-08-17T17:19:57.511671+0000","last_unstale":"2024-08-17T17:23:26.682212+0000","last_undegraded":"2024-08-17T17:23:26.682212+0000","last_fullsized":"2024-08-17T17:23:26.682212+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:59:30.997886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":69,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.671280+0000","last_change":"2024-08-17T17:20:49.279458+0000","last_active":"2024-08-17T17:23:25.671280+0000","last_peered":"2024-08-17T17:23:25.671280+0000","last_clean":"2024-08-17T17:23:25.671280+0000","last_became_active":"2024-08-17T17:20:49.260790+0000","last_became_peered":"2024-08-17T17:20:49.260790+0000","last_unstale":"2024-08-17T17:23:25.671280+0000","last_undegraded":"2024-08-17T17:23:25.671280+0000","last_fullsized":"2024-08-17T17:23:25.671280+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:58.092945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":47,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.671725+0000","last_change":"2024-08-17T17:19:57.506117+0000","last_active":"2024-08-17T17:23:25.671725+0000","last_peered":"2024-08-17T17:23:25.671725+0000","last_clean":"2024-08-17T17:23:25.671725+0000","last_became_active":"2024-08-17T17:19:57.505063+0000","last_became_peered":"2024-08-17T17:19:57.505063+0000","last_unstale":"2024-08-17T17:23:25.671725+0000","last_undegraded":"2024-08-17T17:23:25.671725+0000","last_fullsized":"2024-08-17T17:23:25.671725+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:39:49.859925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":153,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.681581+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:23:26.681581+0000","last_peered":"2024-08-17T17:23:26.681581+0000","last_clean":"2024-08-17T17:23:26.681581+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:23:26.681581+0000","last_undegraded":"2024-08-17T17:23:26.681581+0000","last_fullsized":"2024-08-17T17:23:26.681581+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":181,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.679548+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:23:26.679548+0000","last_peered":"2024-08-17T17:23:26.679548+0000","last_clean":"2024-08-17T17:23:26.679548+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:23:26.679548+0000","last_undegraded":"2024-08-17T17:23:26.679548+0000","last_fullsized":"2024-08-17T17:23:26.679548+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":193,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.679821+0000","last_change":"2024-08-17T17:18:59.409548+0000","last_active":"2024-08-17T17:23:26.679821+0000","last_peered":"2024-08-17T17:23:26.679821+0000","last_clean":"2024-08-17T17:23:26.679821+0000","last_became_active":"2024-08-17T17:18:59.409161+0000","last_became_peered":"2024-08-17T17:18:59.409161+0000","last_unstale":"2024-08-17T17:23:26.679821+0000","last_undegraded":"2024-08-17T17:23:26.679821+0000","last_fullsized":"2024-08-17T17:23:26.679821+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:31:48.248795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":124,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.742336+0000","last_change":"2024-08-17T17:16:26.963696+0000","last_active":"2024-08-17T17:23:25.742336+0000","last_peered":"2024-08-17T17:23:25.742336+0000","last_clean":"2024-08-17T17:23:25.742336+0000","last_became_active":"2024-08-17T17:16:26.963280+0000","last_became_peered":"2024-08-17T17:16:26.963280+0000","last_unstale":"2024-08-17T17:23:25.742336+0000","last_undegraded":"2024-08-17T17:23:25.742336+0000","last_fullsized":"2024-08-17T17:23:25.742336+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:22:01.474840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":211,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.679996+0000","last_change":"2024-08-17T17:19:57.522181+0000","last_active":"2024-08-17T17:23:26.679996+0000","last_peered":"2024-08-17T17:23:26.679996+0000","last_clean":"2024-08-17T17:23:26.679996+0000","last_became_active":"2024-08-17T17:19:57.519621+0000","last_became_peered":"2024-08-17T17:19:57.519621+0000","last_unstale":"2024-08-17T17:23:26.679996+0000","last_undegraded":"2024-08-17T17:23:26.679996+0000","last_fullsized":"2024-08-17T17:23:26.679996+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:38:58.926689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":47,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.671725+0000","last_change":"2024-08-17T17:19:57.506328+0000","last_active":"2024-08-17T17:23:25.671725+0000","last_peered":"2024-08-17T17:23:25.671725+0000","last_clean":"2024-08-17T17:23:25.671725+0000","last_became_active":"2024-08-17T17:19:57.505305+0000","last_became_peered":"2024-08-17T17:19:57.505305+0000","last_unstale":"2024-08-17T17:23:25.671725+0000","last_undegraded":"2024-08-17T17:23:25.671725+0000","last_fullsized":"2024-08-17T17:23:25.671725+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:19:56.623573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":177,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.681616+0000","last_change":"2024-08-17T17:18:59.412463+0000","last_active":"2024-08-17T17:23:26.681616+0000","last_peered":"2024-08-17T17:23:26.681616+0000","last_clean":"2024-08-17T17:23:26.681616+0000","last_became_active":"2024-08-17T17:18:59.412134+0000","last_became_peered":"2024-08-17T17:18:59.412134+0000","last_unstale":"2024-08-17T17:23:26.681616+0000","last_undegraded":"2024-08-17T17:23:26.681616+0000","last_fullsized":"2024-08-17T17:23:26.681616+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:44.463153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":207,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.681067+0000","last_change":"2024-08-17T17:20:49.280657+0000","last_active":"2024-08-17T17:23:26.681067+0000","last_peered":"2024-08-17T17:23:26.681067+0000","last_clean":"2024-08-17T17:23:26.681067+0000","last_became_active":"2024-08-17T17:20:49.279617+0000","last_became_peered":"2024-08-17T17:20:49.279617+0000","last_unstale":"2024-08-17T17:23:26.681067+0000","last_undegraded":"2024-08-17T17:23:26.681067+0000","last_fullsized":"2024-08-17T17:23:26.681067+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:00:42.169630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":111,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.682310+0000","last_change":"2024-08-17T17:19:57.511196+0000","last_active":"2024-08-17T17:23:26.682310+0000","last_peered":"2024-08-17T17:23:26.682310+0000","last_clean":"2024-08-17T17:23:26.682310+0000","last_became_active":"2024-08-17T17:19:57.507497+0000","last_became_peered":"2024-08-17T17:19:57.507497+0000","last_unstale":"2024-08-17T17:23:26.682310+0000","last_undegraded":"2024-08-17T17:23:26.682310+0000","last_fullsized":"2024-08-17T17:23:26.682310+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:04:14.669068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":176,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.682831+0000","last_change":"2024-08-17T17:20:49.277439+0000","last_active":"2024-08-17T17:23:26.682831+0000","last_peered":"2024-08-17T17:23:26.682831+0000","last_clean":"2024-08-17T17:23:26.682831+0000","last_became_active":"2024-08-17T17:20:49.277150+0000","last_became_peered":"2024-08-17T17:20:49.277150+0000","last_unstale":"2024-08-17T17:23:26.682831+0000","last_undegraded":"2024-08-17T17:23:26.682831+0000","last_fullsized":"2024-08-17T17:23:26.682831+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:41:09.990039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":39,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.671289+0000","last_change":"2024-08-17T17:20:49.259891+0000","last_active":"2024-08-17T17:23:25.671289+0000","last_peered":"2024-08-17T17:23:25.671289+0000","last_clean":"2024-08-17T17:23:25.671289+0000","last_became_active":"2024-08-17T17:20:49.259400+0000","last_became_peered":"2024-08-17T17:20:49.259400+0000","last_unstale":"2024-08-17T17:23:25.671289+0000","last_undegraded":"2024-08-17T17:23:25.671289+0000","last_fullsized":"2024-08-17T17:23:25.671289+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:57:27.342244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":207,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.679411+0000","last_change":"2024-08-17T17:21:50.442535+0000","last_active":"2024-08-17T17:23:26.679411+0000","last_peered":"2024-08-17T17:23:26.679411+0000","last_clean":"2024-08-17T17:23:26.679411+0000","last_became_active":"2024-08-17T17:21:50.441891+0000","last_became_peered":"2024-08-17T17:21:50.441891+0000","last_unstale":"2024-08-17T17:23:26.679411+0000","last_undegraded":"2024-08-17T17:23:26.679411+0000","last_fullsized":"2024-08-17T17:23:26.679411+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:46:24.025407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":151,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.741635+0000","last_change":"2024-08-17T17:20:49.264033+0000","last_active":"2024-08-17T17:23:25.741635+0000","last_peered":"2024-08-17T17:23:25.741635+0000","last_clean":"2024-08-17T17:23:25.741635+0000","last_became_active":"2024-08-17T17:20:49.263684+0000","last_became_peered":"2024-08-17T17:20:49.263684+0000","last_unstale":"2024-08-17T17:23:25.741635+0000","last_undegraded":"2024-08-17T17:23:25.741635+0000","last_fullsized":"2024-08-17T17:23:25.741635+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:10:30.430767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":193,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.680913+0000","last_change":"2024-08-17T17:18:59.418434+0000","last_active":"2024-08-17T17:23:26.680913+0000","last_peered":"2024-08-17T17:23:26.680913+0000","last_clean":"2024-08-17T17:23:26.680913+0000","last_became_active":"2024-08-17T17:18:59.418043+0000","last_became_peered":"2024-08-17T17:18:59.418043+0000","last_unstale":"2024-08-17T17:23:26.680913+0000","last_undegraded":"2024-08-17T17:23:26.680913+0000","last_fullsized":"2024-08-17T17:23:26.680913+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:25:51.371820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":152,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.741343+0000","last_change":"2024-08-17T17:20:49.291167+0000","last_active":"2024-08-17T17:23:25.741343+0000","last_peered":"2024-08-17T17:23:25.741343+0000","last_clean":"2024-08-17T17:23:25.741343+0000","last_became_active":"2024-08-17T17:20:49.263750+0000","last_became_peered":"2024-08-17T17:20:49.263750+0000","last_unstale":"2024-08-17T17:23:25.741343+0000","last_undegraded":"2024-08-17T17:23:25.741343+0000","last_fullsized":"2024-08-17T17:23:25.741343+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:02.863246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":207,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.681723+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:23:26.681723+0000","last_peered":"2024-08-17T17:23:26.681723+0000","last_clean":"2024-08-17T17:23:26.681723+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:23:26.681723+0000","last_undegraded":"2024-08-17T17:23:26.681723+0000","last_fullsized":"2024-08-17T17:23:26.681723+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":200,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.681430+0000","last_change":"2024-08-17T17:18:59.413937+0000","last_active":"2024-08-17T17:23:26.681430+0000","last_peered":"2024-08-17T17:23:26.681430+0000","last_clean":"2024-08-17T17:23:26.681430+0000","last_became_active":"2024-08-17T17:18:59.413566+0000","last_became_peered":"2024-08-17T17:18:59.413566+0000","last_unstale":"2024-08-17T17:23:26.681430+0000","last_undegraded":"2024-08-17T17:23:26.681430+0000","last_fullsized":"2024-08-17T17:23:26.681430+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:17:39.877197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":65,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-08-17T17:23:26.681770+0000","last_change":"2024-08-17T17:20:49.276135+0000","last_active":"2024-08-17T17:23:26.681770+0000","last_peered":"2024-08-17T17:23:26.681770+0000","last_clean":"2024-08-17T17:23:26.681770+0000","last_became_active":"2024-08-17T17:20:49.275797+0000","last_became_peered":"2024-08-17T17:20:49.275797+0000","last_unstale":"2024-08-17T17:23:26.681770+0000","last_undegraded":"2024-08-17T17:23:26.681770+0000","last_fullsized":"2024-08-17T17:23:26.681770+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:38:46.858100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":79,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-08-17T17:23:25.741886+0000","last_change":"2024-08-17T17:19:57.526427+0000","last_active":"2024-08-17T17:23:25.741886+0000","last_peered":"2024-08-17T17:23:25.741886+0000","last_clean":"2024-08-17T17:23:25.741886+0000","last_became_active":"2024-08-17T17:19:57.506625+0000","last_became_peered":"2024-08-17T17:19:57.506625+0000","last_unstale":"2024-08-17T17:23:25.741886+0000","last_undegraded":"2024-08-17T17:23:25.741886+0000","last_fullsized":"2024-08-17T17:23:25.741886+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:25:21.357359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":570,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":204110,"num_write_kb":6531457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34462,"num_bytes_recovered":1535377408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2112421888,"data_stored":2112421888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119422,"ondisk_log_size":119422,"up":48,"acting":46,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":5642,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":16702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17971,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":4194342,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234201,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299972,"kb_used_data":2620,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93445180,"statfs":{"total":95995035648,"available":95687864320,"internally_reserved":0,"allocated":2682880,"data_stored":1712681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495836,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312392,"kb_used_data":6712,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93432760,"statfs":{"total":95995035648,"available":95675146240,"internally_reserved":0,"allocated":6873088,"data_stored":5907255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757473,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307700,"kb_used_data":2024,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93437452,"statfs":{"total":95995035648,"available":95679950848,"internally_reserved":0,"allocated":2072576,"data_stored":1117964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019111,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435338,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":136316,"kb_used_data":6124,"kb_used_omap":4,"kb_used_meta":130171,"kb_avail":93608836,"statfs":{"total":95995035648,"available":95855448064,"internally_reserved":0,"allocated":6270976,"data_stored":5312730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":133295114},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676598865,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":345532,"kb_used_data":179184,"kb_used_omap":7,"kb_used_meta":166328,"kb_avail":93399620,"statfs":{"total":95995035648,"available":95641210880,"internally_reserved":0,"allocated":183484416,"data_stored":182526130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":170320034},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762392,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":369144,"kb_used_data":138924,"kb_used_omap":7,"kb_used_meta":230200,"kb_avail":93376008,"statfs":{"total":95995035648,"available":95617032192,"internally_reserved":0,"allocated":142258176,"data_stored":141299801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7945,"internal_metadata":235725047},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726925918,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1467244,"kb_used_data":1309220,"kb_used_omap":7,"kb_used_meta":158008,"kb_avail":92277908,"statfs":{"total":95995035648,"available":94492577792,"internally_reserved":0,"allocated":1340641280,"data_stored":1339690602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":161800513},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6241,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1338572800,"data_stored":1338572800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139591680,"data_stored":139591680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":177209344,"data_stored":177209344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":457048064,"data_stored":457048064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:23:41.835 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd pool set unique_pool_0 pgp_num 16 2024-08-17T17:23:41.863 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:41.902 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:41.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:41.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:42.004 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:42.026 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:42.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:42.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:42.128 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:42.151 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:42.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:42.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:42.253 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:42.275 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:42.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:42.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:42.369 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:23:42.377 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:42.399 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:42.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:42.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:42.501 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:42.524 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:42.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:42.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:42.625 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:42.651 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:42.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:42.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:42.752 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:42.774 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:42.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:42.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:42.876 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:42.898 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:42.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:42.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:42.930 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:42 smithi003 ceph-mon[103344]: pgmap v412: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 83 active+clean; 4.6 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 5661/84 objects degraded (6739.286%); 0 B/s, 278 objects/s recovering 2024-08-17T17:23:42.930 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:42 smithi003 ceph-mon[100615]: pgmap v412: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 83 active+clean; 4.6 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 5661/84 objects degraded (6739.286%); 0 B/s, 278 objects/s recovering 2024-08-17T17:23:43.000 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:43.022 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:43.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:43.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:43.124 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:43.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:43.147 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:43.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:43.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:42 smithi053 ceph-mon[89963]: pgmap v412: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 83 active+clean; 4.6 MiB data, 3.1 GiB used, 623 GiB / 626 GiB avail; 5661/84 objects degraded (6739.286%); 0 B/s, 278 objects/s recovering 2024-08-17T17:23:43.248 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:43.270 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:43.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:43.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:43.372 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:43.396 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:43.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:43.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:43.497 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:43.520 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:43.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:43.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:43.621 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:43.644 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:43.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:43.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:43.746 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:43.768 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:43.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:43.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:43.870 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:43.893 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:43.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:43.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:43.995 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:44.017 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:44.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:44.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:44.119 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:44.141 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:44.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:44.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:44.243 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:44.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:44.267 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:44.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:44.368 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:44.391 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:44.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:44.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:44.493 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:44.516 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:44.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:44.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:44.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:44.640 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:44.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:44.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:44.742 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:44.764 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:44.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:44.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:44.866 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:44.890 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:44.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:44.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:44.896 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-08-17T17:23:44.992 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:45.014 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:45.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:45.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:45.116 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:45.138 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:45.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:45.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:45.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:44 smithi053 ceph-mon[89963]: pgmap v413: 89 pgs: 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 83 active+clean; 4.6 MiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 2957/84 objects degraded (3520.238%); 0 B/s, 209 objects/s recovering 2024-08-17T17:23:45.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:44 smithi053 ceph-mon[89963]: osdmap e168: 8 total, 8 up, 7 in 2024-08-17T17:23:45.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:44 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1285963829' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:23:45.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:44 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 2957/84 objects degraded (3520.238%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:23:45.240 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:45.263 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:45.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:45.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:45.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:44 smithi003 ceph-mon[100615]: pgmap v413: 89 pgs: 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 83 active+clean; 4.6 MiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 2957/84 objects degraded (3520.238%); 0 B/s, 209 objects/s recovering 2024-08-17T17:23:45.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:44 smithi003 ceph-mon[100615]: osdmap e168: 8 total, 8 up, 7 in 2024-08-17T17:23:45.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:44 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1285963829' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:23:45.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:44 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 2957/84 objects degraded (3520.238%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:23:45.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:44 smithi003 ceph-mon[103344]: pgmap v413: 89 pgs: 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait, 83 active+clean; 4.6 MiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 2957/84 objects degraded (3520.238%); 0 B/s, 209 objects/s recovering 2024-08-17T17:23:45.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:44 smithi003 ceph-mon[103344]: osdmap e168: 8 total, 8 up, 7 in 2024-08-17T17:23:45.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:44 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1285963829' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:23:45.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:44 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 2957/84 objects degraded (3520.238%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:23:45.365 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:45.389 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:45.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:45.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:45.403 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:23:45.403 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:23:45.492 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:45.514 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:45.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:45.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:45.615 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:45.638 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:45.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:45.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:45.739 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:45.761 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:45.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:45.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:45.863 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:45.885 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:45.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:45.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:45.916 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:23:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:23:45] "GET /metrics HTTP/1.1" 200 36575 "" "Prometheus/2.51.0" 2024-08-17T17:23:45.987 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:46.010 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:46.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:46.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:46.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:46.136 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:46.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:46.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:46.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:45 smithi053 ceph-mon[89963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:23:46.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:45 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1285963829' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:23:46.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:45 smithi053 ceph-mon[89963]: osdmap e169: 8 total, 8 up, 7 in 2024-08-17T17:23:46.237 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:46.260 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:46.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:46.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:46.272 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:45 smithi003 ceph-mon[100615]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:23:46.272 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:45 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1285963829' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:23:46.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:45 smithi003 ceph-mon[100615]: osdmap e169: 8 total, 8 up, 7 in 2024-08-17T17:23:46.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:45 smithi003 ceph-mon[103344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:23:46.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:45 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1285963829' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:23:46.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:45 smithi003 ceph-mon[103344]: osdmap e169: 8 total, 8 up, 7 in 2024-08-17T17:23:46.362 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:46.384 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:46.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:46.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:46.391 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:23:46.413 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-17T17:23:46.485 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:46.508 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:46.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:46.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:46.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:46.635 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:46.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:46.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:46.737 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:46.759 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:46.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:46.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:46.860 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:46.883 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:46.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:46.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:46.984 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:47.006 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:47.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:47.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:47.108 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:47.130 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:47.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:47.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:47.181 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:46 smithi003 ceph-mon[103344]: pgmap v416: 89 pgs: 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 84 active+clean; 4.6 MiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 2957/84 objects degraded (3520.238%); 0 B/s, 184 objects/s recovering 2024-08-17T17:23:47.181 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:46 smithi003 ceph-mon[100615]: pgmap v416: 89 pgs: 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 84 active+clean; 4.6 MiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 2957/84 objects degraded (3520.238%); 0 B/s, 184 objects/s recovering 2024-08-17T17:23:47.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:46 smithi053 ceph-mon[89963]: pgmap v416: 89 pgs: 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 84 active+clean; 4.6 MiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 2957/84 objects degraded (3520.238%); 0 B/s, 184 objects/s recovering 2024-08-17T17:23:47.231 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:47.255 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:47.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:47.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:47.309 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:23:47.309 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:23:47.310 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:23:47.310 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:23:47.310 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:23:47.310 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:23:47.310 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:23:47.310 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:23:47.310 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:23:47.311 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:23:47.311 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:23:47.311 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:23:47.311 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:23:47.311 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:23:47.311 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:23:47.311 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:23:47.357 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:47.379 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:47.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:47.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:47.481 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:47.507 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:47.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:47.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:47.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:47.632 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:47.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:47.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:47.734 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:47.756 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:47.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:47.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:47.858 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:47.880 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:47.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:47.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:47.981 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:48.004 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:48.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:48.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:48.106 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:48.129 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:48.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:48.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:48.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:47 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/27489159' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:23:48.231 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:48.253 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:48.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:48.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:48.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:47 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/27489159' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:23:48.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:47 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/27489159' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:23:48.355 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:48.378 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:48.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:48.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:48.480 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:48.501 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:48.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:48.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:48.603 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:48.626 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:48.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:48.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:48.728 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:48.751 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:48.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:48.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:48.853 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:48.875 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:48.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:48.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:48.977 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:49.003 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:49.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:49.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:49.007 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:23:49.107 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:49.130 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:49.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:49.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:49.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:48 smithi053 ceph-mon[89963]: pgmap v417: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 4.6 MiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 2130/84 objects degraded (2535.714%); 0 B/s, 288 objects/s recovering 2024-08-17T17:23:49.232 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:49.254 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:49.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:49.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:49.272 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:48 smithi003 ceph-mon[100615]: pgmap v417: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 4.6 MiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 2130/84 objects degraded (2535.714%); 0 B/s, 288 objects/s recovering 2024-08-17T17:23:49.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:48 smithi003 ceph-mon[103344]: pgmap v417: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 4.6 MiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 2130/84 objects degraded (2535.714%); 0 B/s, 288 objects/s recovering 2024-08-17T17:23:49.356 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:49.385 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:49.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:49.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:49.486 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:49.509 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:49.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:49.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:49.611 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:49.633 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:49.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:49.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:49.735 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:49.757 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:49.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:49.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:49.859 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:49.882 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:49.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:49.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:49.945 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:23:49.985 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:50.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:50.013 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:50.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:50.114 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:50.137 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:50.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:50.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:50.239 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:50.262 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:50.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:50.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:50.364 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:50.390 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:50.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:50.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:50.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:50 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:23:50.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:50 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:23:50.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:50 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 2130/84 objects degraded (2535.714%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:23:50.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:23:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:23:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:23:50.492 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:50.515 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:50.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:50.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:50.522 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:50 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:23:50.522 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:50 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:23:50.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:50 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 2130/84 objects degraded (2535.714%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:23:50.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:50 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:23:50.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:50 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:23:50.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:50 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 2130/84 objects degraded (2535.714%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:23:50.617 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:50.638 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:50.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:50.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:50.740 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:50.764 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:50.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:50.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:50.866 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:50.889 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:50.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:50.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:50.991 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:51.013 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:51.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:51.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:51.115 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:51.137 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:51.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:51.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:51.239 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:51.262 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:51.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:51.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:51.364 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:51.383 INFO:teuthology.orchestra.run.smithi003.stderr:set pool 5 pgp_num to 16 2024-08-17T17:23:51.389 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:51.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:51.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:51.491 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:51.514 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:51.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:51.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:51.522 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:51 smithi003 ceph-mon[100615]: pgmap v418: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 4.6 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 2130/84 objects degraded (2535.714%); 0 B/s, 233 objects/s recovering 2024-08-17T17:23:51.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:51 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/372294455' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"} : dispatch 2024-08-17T17:23:51.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:51 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"} : dispatch 2024-08-17T17:23:51.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:51 smithi003 ceph-mon[103344]: pgmap v418: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 4.6 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 2130/84 objects degraded (2535.714%); 0 B/s, 233 objects/s recovering 2024-08-17T17:23:51.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:51 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/372294455' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"} : dispatch 2024-08-17T17:23:51.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:51 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"} : dispatch 2024-08-17T17:23:51.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:51.641 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:51.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:51.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:51.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:51 smithi053 ceph-mon[89963]: pgmap v418: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 4.6 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 2130/84 objects degraded (2535.714%); 0 B/s, 233 objects/s recovering 2024-08-17T17:23:51.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:51 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/372294455' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"} : dispatch 2024-08-17T17:23:51.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:51 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"} : dispatch 2024-08-17T17:23:51.743 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:51.765 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:51.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:51.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:51.866 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:51.888 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:51.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:51.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:51.990 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:52.013 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:23:52.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:52.015 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:52.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:52.116 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:52.139 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:52.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:52.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:52.241 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:52.268 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:52.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:52.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:52.369 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:52.387 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-08-17T17:23:52.387 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-08-17T17:23:52.402 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:52.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:52.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:52.504 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:52.531 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:52.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:52.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:52.633 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:52.634 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:52 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-08-17T17:23:52.634 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:52 smithi003 ceph-mon[103344]: osdmap e170: 8 total, 8 up, 7 in 2024-08-17T17:23:52.635 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:52 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-08-17T17:23:52.635 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:52 smithi003 ceph-mon[100615]: osdmap e170: 8 total, 8 up, 7 in 2024-08-17T17:23:52.658 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:52.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:52.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:52.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:52 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-08-17T17:23:52.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:52 smithi053 ceph-mon[89963]: osdmap e170: 8 total, 8 up, 7 in 2024-08-17T17:23:52.760 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:52.783 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:52.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:52.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:52.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:52.896 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:23:52.896 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-08-17T17:23:52.896 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-08-17T17:23:52.896 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:23:52.896 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.896 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:61360 5.d 5:bd89d4c5:::benchmark_data_smithi003_140793_object61359:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-17T17:23:52.896 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:50.778781+0000", 2024-08-17T17:23:52.897 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 242.11522514000001, 2024-08-17T17:23:52.897 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.131254864, 2024-08-17T17:23:52.897 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:23:52.897 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:23:52.897 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:23:52.897 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:23:52.897 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:23:52.897 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:23:52.897 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 61360 2024-08-17T17:23:52.897 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.897 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:23:52.897 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.897 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:23:52.897 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.778781+0000", 2024-08-17T17:23:52.898 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:52.898 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.898 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.898 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:23:52.898 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.778781+0000", 2024-08-17T17:23:52.898 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:52.898 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.898 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.898 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:23:52.898 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.778783+0000", 2024-08-17T17:23:52.898 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8419999999999999e-06 2024-08-17T17:23:52.898 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.899 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.899 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:23:52.899 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.778808+0000", 2024-08-17T17:23:52.899 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4810000000000001e-05 2024-08-17T17:23:52.899 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.899 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.899 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:23:52.899 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.778829+0000", 2024-08-17T17:23:52.899 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1569999999999998e-05 2024-08-17T17:23:52.899 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.899 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.899 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:23:52.900 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.778837+0000", 2024-08-17T17:23:52.900 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3119999999999996e-06 2024-08-17T17:23:52.900 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.900 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.900 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:52.900 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.778900+0000", 2024-08-17T17:23:52.900 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3577999999999993e-05 2024-08-17T17:23:52.900 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.900 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.900 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:23:52.900 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.779023+0000", 2024-08-17T17:23:52.900 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012223900000000001 2024-08-17T17:23:52.900 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.901 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.901 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,7", 2024-08-17T17:23:52.901 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.779325+0000", 2024-08-17T17:23:52.901 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00030238200000000002 2024-08-17T17:23:52.901 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.901 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.901 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:23:52.901 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.781873+0000", 2024-08-17T17:23:52.901 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025477109999999998 2024-08-17T17:23:52.901 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.901 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.902 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:23:52.902 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.782213+0000", 2024-08-17T17:23:52.902 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00034041099999999999 2024-08-17T17:23:52.902 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.902 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.902 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:23:52.902 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.909936+0000", 2024-08-17T17:23:52.902 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.12772302699999999 2024-08-17T17:23:52.902 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.902 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.902 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:23:52.902 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.909997+0000", 2024-08-17T17:23:52.903 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0609000000000002e-05 2024-08-17T17:23:52.903 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.903 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.903 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:23:52.903 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.910036+0000", 2024-08-17T17:23:52.903 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9372999999999997e-05 2024-08-17T17:23:52.903 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:52.903 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:23:52.903 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:52.903 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.903 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.903 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:61368 5.d 5:bf015eb7:::benchmark_data_smithi003_140793_object61367:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-17T17:23:52.903 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:50.784270+0000", 2024-08-17T17:23:52.903 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 242.109736234, 2024-08-17T17:23:52.904 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.132675028, 2024-08-17T17:23:52.904 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:23:52.904 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:23:52.904 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:23:52.904 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:23:52.904 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:23:52.904 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:23:52.904 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 61368 2024-08-17T17:23:52.904 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.904 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:23:52.904 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.904 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:23:52.904 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.784270+0000", 2024-08-17T17:23:52.905 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:52.905 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.905 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.905 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:23:52.905 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.784270+0000", 2024-08-17T17:23:52.905 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:52.905 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.905 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.905 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:23:52.905 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.784272+0000", 2024-08-17T17:23:52.905 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.403e-06 2024-08-17T17:23:52.905 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.905 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.905 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:23:52.906 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.784315+0000", 2024-08-17T17:23:52.906 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3727999999999999e-05 2024-08-17T17:23:52.906 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.906 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.906 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:23:52.906 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.784327+0000", 2024-08-17T17:23:52.906 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1695000000000001e-05 2024-08-17T17:23:52.906 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.906 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.906 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:23:52.906 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.784334+0000", 2024-08-17T17:23:52.906 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6379999999999998e-06 2024-08-17T17:23:52.906 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.907 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.907 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:52.907 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.784405+0000", 2024-08-17T17:23:52.907 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1027999999999998e-05 2024-08-17T17:23:52.907 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.907 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.907 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:23:52.907 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.784524+0000", 2024-08-17T17:23:52.907 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000119102 2024-08-17T17:23:52.907 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.907 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.907 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,7", 2024-08-17T17:23:52.907 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.784647+0000", 2024-08-17T17:23:52.908 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012282300000000001 2024-08-17T17:23:52.908 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.908 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.908 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:23:52.908 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.787272+0000", 2024-08-17T17:23:52.908 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002625063 2024-08-17T17:23:52.908 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.908 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.908 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:23:52.908 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.787308+0000", 2024-08-17T17:23:52.908 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6822000000000001e-05 2024-08-17T17:23:52.908 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.908 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.909 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:23:52.909 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.916824+0000", 2024-08-17T17:23:52.909 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.12951596300000001 2024-08-17T17:23:52.909 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.909 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.909 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:23:52.909 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.916900+0000", 2024-08-17T17:23:52.909 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6077999999999999e-05 2024-08-17T17:23:52.909 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.909 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.909 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:23:52.909 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.916945+0000", 2024-08-17T17:23:52.909 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4685000000000003e-05 2024-08-17T17:23:52.910 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:52.910 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:23:52.910 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:52.910 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.910 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.910 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:61374 5.8 5:1cffd916:::benchmark_data_smithi003_140793_object61373:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-17T17:23:52.910 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:50.787184+0000", 2024-08-17T17:23:52.910 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 242.10682262200001, 2024-08-17T17:23:52.910 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.11507941100000001, 2024-08-17T17:23:52.910 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:23:52.910 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:23:52.911 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:23:52.911 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:23:52.911 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:23:52.911 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:23:52.911 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 61374 2024-08-17T17:23:52.911 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.912 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:23:52.912 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.912 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:23:52.912 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.787184+0000", 2024-08-17T17:23:52.912 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:52.912 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.912 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.912 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:23:52.912 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.787184+0000", 2024-08-17T17:23:52.912 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:52.912 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.912 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.912 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:23:52.913 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.787209+0000", 2024-08-17T17:23:52.913 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5599e-05 2024-08-17T17:23:52.913 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.913 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.913 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:23:52.913 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.787273+0000", 2024-08-17T17:23:52.913 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3734999999999999e-05 2024-08-17T17:23:52.913 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.913 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.913 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:23:52.913 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.787285+0000", 2024-08-17T17:23:52.913 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1514e-05 2024-08-17T17:23:52.913 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.913 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.914 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:23:52.914 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.787292+0000", 2024-08-17T17:23:52.914 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8730000000000007e-06 2024-08-17T17:23:52.914 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.914 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.914 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:52.914 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.787510+0000", 2024-08-17T17:23:52.914 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000217421 2024-08-17T17:23:52.914 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.914 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.914 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:23:52.914 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.787640+0000", 2024-08-17T17:23:52.915 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013038200000000001 2024-08-17T17:23:52.915 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.915 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.915 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:23:52.915 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.787758+0000", 2024-08-17T17:23:52.915 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000117755 2024-08-17T17:23:52.915 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.915 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.915 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:23:52.915 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.794094+0000", 2024-08-17T17:23:52.915 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063357509999999997 2024-08-17T17:23:52.915 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.915 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.915 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:23:52.916 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.794535+0000", 2024-08-17T17:23:52.916 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00044143399999999999 2024-08-17T17:23:52.916 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.916 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.916 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:23:52.916 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.902140+0000", 2024-08-17T17:23:52.916 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.107604831 2024-08-17T17:23:52.916 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.916 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.916 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:23:52.916 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.902209+0000", 2024-08-17T17:23:52.916 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8937999999999996e-05 2024-08-17T17:23:52.916 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.917 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.917 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:23:52.917 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.902263+0000", 2024-08-17T17:23:52.917 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4178000000000002e-05 2024-08-17T17:23:52.917 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:52.917 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:23:52.917 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:52.917 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.917 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.917 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:61384 5.d 5:b67f0d5e:::benchmark_data_smithi003_140793_object61383:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-17T17:23:52.917 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:50.794595+0000", 2024-08-17T17:23:52.917 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 242.09941108800001, 2024-08-17T17:23:52.917 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.123014096, 2024-08-17T17:23:52.917 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:23:52.918 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:23:52.918 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:23:52.918 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:23:52.918 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:23:52.918 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:23:52.918 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 61384 2024-08-17T17:23:52.918 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.918 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:23:52.918 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.918 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:23:52.918 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.794595+0000", 2024-08-17T17:23:52.918 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:52.918 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.919 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.919 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:23:52.919 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.794595+0000", 2024-08-17T17:23:52.919 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:52.919 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.919 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.919 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:23:52.919 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.794597+0000", 2024-08-17T17:23:52.919 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.761e-06 2024-08-17T17:23:52.919 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.919 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.919 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:23:52.919 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.794620+0000", 2024-08-17T17:23:52.919 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3268000000000002e-05 2024-08-17T17:23:52.919 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.920 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.920 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:23:52.920 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.794631+0000", 2024-08-17T17:23:52.920 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0743e-05 2024-08-17T17:23:52.920 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.920 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.920 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:23:52.920 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.794638+0000", 2024-08-17T17:23:52.920 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0820000000000001e-06 2024-08-17T17:23:52.920 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.920 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.920 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:52.920 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.794697+0000", 2024-08-17T17:23:52.920 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8588999999999998e-05 2024-08-17T17:23:52.920 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.920 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.921 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:23:52.921 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.794834+0000", 2024-08-17T17:23:52.921 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000137553 2024-08-17T17:23:52.921 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.921 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.921 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,7", 2024-08-17T17:23:52.921 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.794973+0000", 2024-08-17T17:23:52.921 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013865899999999999 2024-08-17T17:23:52.921 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.921 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.921 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:23:52.921 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.796711+0000", 2024-08-17T17:23:52.921 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017377099999999999 2024-08-17T17:23:52.921 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.921 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.922 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:23:52.922 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.797072+0000", 2024-08-17T17:23:52.922 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00036177799999999999 2024-08-17T17:23:52.922 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.922 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.922 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:23:52.922 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.917510+0000", 2024-08-17T17:23:52.922 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.12043799500000001 2024-08-17T17:23:52.922 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.922 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.922 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:23:52.922 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.917574+0000", 2024-08-17T17:23:52.922 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3126999999999994e-05 2024-08-17T17:23:52.922 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.922 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.923 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:23:52.923 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:50.917609+0000", 2024-08-17T17:23:52.923 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5831000000000003e-05 2024-08-17T17:23:52.923 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:52.923 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:23:52.923 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:52.923 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.923 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.923 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:68131 5.a 5:5eb7e0b1:::benchmark_data_smithi003_140793_object68130:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-17T17:23:52.923 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:56.473603+0000", 2024-08-17T17:23:52.923 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 236.42040316999999, 2024-08-17T17:23:52.923 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3252823510000002, 2024-08-17T17:23:52.923 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:23:52.924 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:23:52.924 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:23:52.924 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:23:52.924 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:23:52.924 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:23:52.924 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68131 2024-08-17T17:23:52.924 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.924 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:23:52.924 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.924 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:23:52.924 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.473603+0000", 2024-08-17T17:23:52.924 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:52.924 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.925 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.925 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:23:52.925 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.473603+0000", 2024-08-17T17:23:52.925 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:52.925 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.925 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.925 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:23:52.925 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.473605+0000", 2024-08-17T17:23:52.925 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4100000000000001e-06 2024-08-17T17:23:52.925 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.925 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.925 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:23:52.925 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.473630+0000", 2024-08-17T17:23:52.925 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5717000000000001e-05 2024-08-17T17:23:52.926 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.926 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.926 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:23:52.926 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.473640+0000", 2024-08-17T17:23:52.926 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7869999999999996e-06 2024-08-17T17:23:52.926 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.926 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.926 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:23:52.926 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.473646+0000", 2024-08-17T17:23:52.926 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5099999999999998e-06 2024-08-17T17:23:52.926 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.926 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.926 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:52.926 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.480848+0000", 2024-08-17T17:23:52.927 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072023449999999998 2024-08-17T17:23:52.927 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.927 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.927 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:23:52.927 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.480987+0000", 2024-08-17T17:23:52.927 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013917199999999999 2024-08-17T17:23:52.927 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.927 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.927 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,3", 2024-08-17T17:23:52.927 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.481131+0000", 2024-08-17T17:23:52.927 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014377899999999999 2024-08-17T17:23:52.927 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.927 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.927 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:52.928 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.507492+0000", 2024-08-17T17:23:52.928 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.026361190999999999 2024-08-17T17:23:52.928 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.928 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.928 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:23:52.928 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.507503+0000", 2024-08-17T17:23:52.928 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1277000000000001e-05 2024-08-17T17:23:52.928 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.928 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.928 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:52.928 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.481848+0000", 2024-08-17T17:23:52.928 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97434455600000003 2024-08-17T17:23:52.928 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.929 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.929 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:23:52.929 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.481865+0000", 2024-08-17T17:23:52.929 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6639000000000001e-05 2024-08-17T17:23:52.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.929 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.929 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:52.929 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.778427+0000", 2024-08-17T17:23:52.929 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.296562913 2024-08-17T17:23:52.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.929 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.929 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for degraded object", 2024-08-17T17:23:52.929 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.779493+0000", 2024-08-17T17:23:52.929 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001065178 2024-08-17T17:23:52.930 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.930 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.930 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:52.930 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.798772+0000", 2024-08-17T17:23:52.930 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.019279580000000001 2024-08-17T17:23:52.930 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.930 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.930 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:23:52.930 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.798886+0000", 2024-08-17T17:23:52.930 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000113297 2024-08-17T17:23:52.930 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:52.930 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:23:52.930 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:52.931 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.931 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.931 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:68134 5.a 5:507a5fcd:::benchmark_data_smithi003_140793_object68133:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-08-17T17:23:52.931 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:56.497317+0000", 2024-08-17T17:23:52.931 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 236.39668943999999, 2024-08-17T17:23:52.931 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3050668550000002, 2024-08-17T17:23:52.931 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:23:52.931 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:23:52.931 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:23:52.931 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:23:52.931 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:23:52.931 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:23:52.931 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68134 2024-08-17T17:23:52.932 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.932 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:23:52.932 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.932 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:23:52.932 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.497317+0000", 2024-08-17T17:23:52.932 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:52.932 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.932 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.933 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:23:52.933 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.497317+0000", 2024-08-17T17:23:52.933 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:52.933 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.933 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.933 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:23:52.933 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.497319+0000", 2024-08-17T17:23:52.933 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.64e-06 2024-08-17T17:23:52.933 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.933 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.933 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:23:52.933 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.497345+0000", 2024-08-17T17:23:52.934 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6248e-05 2024-08-17T17:23:52.934 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.934 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.934 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:23:52.934 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.497357+0000", 2024-08-17T17:23:52.934 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.184e-05 2024-08-17T17:23:52.934 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.934 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.934 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:23:52.934 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.497366+0000", 2024-08-17T17:23:52.934 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2499999999999995e-06 2024-08-17T17:23:52.934 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.934 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.934 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:52.935 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.507534+0000", 2024-08-17T17:23:52.935 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010168089999999999 2024-08-17T17:23:52.935 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.935 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.935 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:23:52.935 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.507542+0000", 2024-08-17T17:23:52.935 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8029999999999997e-06 2024-08-17T17:23:52.935 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.935 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.935 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:52.935 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.481898+0000", 2024-08-17T17:23:52.935 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.974355949 2024-08-17T17:23:52.935 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.936 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.936 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:23:52.936 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.481906+0000", 2024-08-17T17:23:52.936 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.089e-06 2024-08-17T17:23:52.936 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.936 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.936 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:52.936 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.784148+0000", 2024-08-17T17:23:52.936 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3022419219999999 2024-08-17T17:23:52.936 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.936 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.936 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:23:52.936 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.784298+0000", 2024-08-17T17:23:52.936 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015066 2024-08-17T17:23:52.936 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.937 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.937 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:23:52.937 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.784462+0000", 2024-08-17T17:23:52.937 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000163676 2024-08-17T17:23:52.937 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.937 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.937 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:23:52.937 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.798884+0000", 2024-08-17T17:23:52.937 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.014422199 2024-08-17T17:23:52.937 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.937 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.937 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:23:52.937 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.798980+0000", 2024-08-17T17:23:52.937 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5964999999999993e-05 2024-08-17T17:23:52.938 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.938 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.938 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:23:52.938 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.799031+0000", 2024-08-17T17:23:52.938 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0421999999999999e-05 2024-08-17T17:23:52.938 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.938 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.938 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:23:52.938 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.799088+0000", 2024-08-17T17:23:52.938 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7260999999999999e-05 2024-08-17T17:23:52.938 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.938 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.939 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:23:52.939 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.802384+0000", 2024-08-17T17:23:52.939 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032958409999999999 2024-08-17T17:23:52.939 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:52.939 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:23:52.939 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:52.939 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.939 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.939 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:68144 5.a 5:5a80e405:::benchmark_data_smithi003_140793_object68143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-08-17T17:23:52.939 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:56.519883+0000", 2024-08-17T17:23:52.939 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 236.374122981, 2024-08-17T17:23:52.939 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3203096400000001, 2024-08-17T17:23:52.939 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:23:52.939 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:23:52.940 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:23:52.940 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:23:52.940 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:23:52.940 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:23:52.940 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68144 2024-08-17T17:23:52.940 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.940 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:23:52.940 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.940 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:23:52.940 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519883+0000", 2024-08-17T17:23:52.940 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:52.940 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.940 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.940 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:23:52.941 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519883+0000", 2024-08-17T17:23:52.941 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:52.941 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.941 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.941 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:23:52.941 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519886+0000", 2024-08-17T17:23:52.941 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4949999999999998e-06 2024-08-17T17:23:52.941 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.941 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.941 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:23:52.941 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519909+0000", 2024-08-17T17:23:52.941 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3366e-05 2024-08-17T17:23:52.941 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.941 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.942 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:23:52.942 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519926+0000", 2024-08-17T17:23:52.942 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6637999999999999e-05 2024-08-17T17:23:52.942 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.942 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.942 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:23:52.942 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519933+0000", 2024-08-17T17:23:52.942 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3710000000000002e-06 2024-08-17T17:23:52.942 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.942 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.942 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:52.942 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519988+0000", 2024-08-17T17:23:52.942 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4923999999999998e-05 2024-08-17T17:23:52.943 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.943 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.943 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:23:52.943 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519995+0000", 2024-08-17T17:23:52.943 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0369999999999997e-06 2024-08-17T17:23:52.943 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.943 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.943 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:52.943 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.482416+0000", 2024-08-17T17:23:52.943 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96242099800000003 2024-08-17T17:23:52.943 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.943 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.943 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:23:52.943 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.482427+0000", 2024-08-17T17:23:52.943 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1014e-05 2024-08-17T17:23:52.944 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.944 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.944 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:52.944 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.802434+0000", 2024-08-17T17:23:52.944 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.320006295 2024-08-17T17:23:52.944 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.944 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.944 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:23:52.944 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.802587+0000", 2024-08-17T17:23:52.944 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000153342 2024-08-17T17:23:52.944 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.944 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.944 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:23:52.944 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.802749+0000", 2024-08-17T17:23:52.945 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016180400000000001 2024-08-17T17:23:52.945 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.945 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.945 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:23:52.945 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.813063+0000", 2024-08-17T17:23:52.945 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010314591 2024-08-17T17:23:52.945 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.945 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.945 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:23:52.945 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.814540+0000", 2024-08-17T17:23:52.945 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014765500000000001 2024-08-17T17:23:52.945 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.945 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.945 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:23:52.945 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.836294+0000", 2024-08-17T17:23:52.946 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.021754141000000001 2024-08-17T17:23:52.946 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.946 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.946 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:23:52.946 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.836377+0000", 2024-08-17T17:23:52.946 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2698e-05 2024-08-17T17:23:52.946 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.946 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.946 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:23:52.946 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.840193+0000", 2024-08-17T17:23:52.946 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038163759999999998 2024-08-17T17:23:52.946 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:52.947 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:23:52.947 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:52.947 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.947 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.947 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:68113 5.4 5:2078877e:::benchmark_data_smithi003_140793_object68112:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e130)", 2024-08-17T17:23:52.947 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:57.488289+0000", 2024-08-17T17:23:52.947 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 235.40571740300001, 2024-08-17T17:23:52.947 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.249369143, 2024-08-17T17:23:52.947 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:23:52.947 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:23:52.947 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:23:52.947 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:23:52.947 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:23:52.947 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:23:52.948 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68113 2024-08-17T17:23:52.948 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.948 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:23:52.948 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.948 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:23:52.948 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488289+0000", 2024-08-17T17:23:52.948 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:52.948 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.948 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.948 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:23:52.948 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488289+0000", 2024-08-17T17:23:52.948 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:52.948 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.948 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.948 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:23:52.948 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488290+0000", 2024-08-17T17:23:52.949 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.463e-06 2024-08-17T17:23:52.949 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.949 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.949 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:23:52.949 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488320+0000", 2024-08-17T17:23:52.949 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9150999999999999e-05 2024-08-17T17:23:52.949 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.949 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.949 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:23:52.949 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488331+0000", 2024-08-17T17:23:52.949 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1396000000000001e-05 2024-08-17T17:23:52.949 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.949 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.949 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:23:52.950 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488341+0000", 2024-08-17T17:23:52.950 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0363999999999999e-05 2024-08-17T17:23:52.950 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.950 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.950 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:52.950 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.490414+0000", 2024-08-17T17:23:52.950 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002073068 2024-08-17T17:23:52.950 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.950 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.950 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:23:52.950 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.490424+0000", 2024-08-17T17:23:52.950 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4669999999999993e-06 2024-08-17T17:23:52.950 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.951 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.951 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:52.951 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.737532+0000", 2024-08-17T17:23:52.951 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2471080960000001 2024-08-17T17:23:52.951 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.951 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.951 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:23:52.951 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.737658+0000", 2024-08-17T17:23:52.951 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000126138 2024-08-17T17:23:52.951 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:52.951 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:23:52.951 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:52.951 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.952 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.952 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:68122 5.4 5:2519cdf9:::benchmark_data_smithi003_140793_object68121:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e130)", 2024-08-17T17:23:52.952 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:57.488356+0000", 2024-08-17T17:23:52.952 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 235.40565018500001, 2024-08-17T17:23:52.952 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.252854337, 2024-08-17T17:23:52.952 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:23:52.952 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:23:52.952 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:23:52.952 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:23:52.952 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:23:52.952 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:23:52.952 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68122 2024-08-17T17:23:52.952 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.952 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:23:52.953 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.953 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:23:52.953 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488356+0000", 2024-08-17T17:23:52.953 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:52.953 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:52.954 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:52.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:52.954 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.955 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:23:52.955 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488356+0000", 2024-08-17T17:23:52.955 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:52.955 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.955 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.955 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:23:52.955 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488357+0000", 2024-08-17T17:23:52.955 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.44e-07 2024-08-17T17:23:52.955 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.955 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.955 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:23:52.955 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488382+0000", 2024-08-17T17:23:52.955 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5321999999999999e-05 2024-08-17T17:23:52.956 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.956 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.956 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:23:52.956 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488391+0000", 2024-08-17T17:23:52.956 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8170000000000005e-06 2024-08-17T17:23:52.956 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.956 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.956 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:23:52.956 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488397+0000", 2024-08-17T17:23:52.956 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1580000000000002e-06 2024-08-17T17:23:52.956 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.956 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.956 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:52.957 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.490445+0000", 2024-08-17T17:23:52.957 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002048242 2024-08-17T17:23:52.957 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.957 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.957 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:23:52.957 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.490451+0000", 2024-08-17T17:23:52.957 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3460000000000001e-06 2024-08-17T17:23:52.957 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.957 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.957 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:52.957 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.741078+0000", 2024-08-17T17:23:52.957 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.250627181 2024-08-17T17:23:52.957 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.958 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.958 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:23:52.958 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.741211+0000", 2024-08-17T17:23:52.958 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013272699999999999 2024-08-17T17:23:52.958 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:52.958 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:23:52.958 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:52.958 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.958 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.958 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:68126 5.4 5:2abb837e:::benchmark_data_smithi003_140793_object68125:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e130)", 2024-08-17T17:23:52.958 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:57.488407+0000", 2024-08-17T17:23:52.958 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 235.40559921799999, 2024-08-17T17:23:52.958 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.287577712, 2024-08-17T17:23:52.959 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:23:52.959 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:23:52.959 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:23:52.959 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:23:52.959 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:23:52.959 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:23:52.959 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68126 2024-08-17T17:23:52.959 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.959 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:23:52.959 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.959 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:23:52.959 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488407+0000", 2024-08-17T17:23:52.959 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:52.960 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.960 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.960 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:23:52.960 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488407+0000", 2024-08-17T17:23:52.960 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:52.960 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.960 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.960 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:23:52.960 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488408+0000", 2024-08-17T17:23:52.960 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3600000000000004e-07 2024-08-17T17:23:52.960 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.960 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.960 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:23:52.961 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488432+0000", 2024-08-17T17:23:52.961 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3955999999999999e-05 2024-08-17T17:23:52.961 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.961 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.961 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:23:52.961 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488440+0000", 2024-08-17T17:23:52.961 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1550000000000007e-06 2024-08-17T17:23:52.961 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.961 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.961 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:23:52.961 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488446+0000", 2024-08-17T17:23:52.961 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2269999999999998e-06 2024-08-17T17:23:52.962 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.962 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.962 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:52.962 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.490496+0000", 2024-08-17T17:23:52.962 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020495449999999998 2024-08-17T17:23:52.962 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.962 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.962 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:23:52.963 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.490505+0000", 2024-08-17T17:23:52.963 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0450000000000006e-06 2024-08-17T17:23:52.963 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.963 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.963 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:52.963 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.741568+0000", 2024-08-17T17:23:52.963 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2510632020000001 2024-08-17T17:23:52.963 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.963 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.963 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for degraded object", 2024-08-17T17:23:52.963 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.742515+0000", 2024-08-17T17:23:52.963 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00094757200000000004 2024-08-17T17:23:52.964 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.964 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.964 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:52.964 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.775909+0000", 2024-08-17T17:23:52.964 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.033393513999999999 2024-08-17T17:23:52.964 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.964 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.964 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:23:52.964 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.775985+0000", 2024-08-17T17:23:52.964 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5959999999999998e-05 2024-08-17T17:23:52.964 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:52.964 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:23:52.964 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:52.965 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.965 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.965 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:68128 5.4 5:250eee02:::benchmark_data_smithi003_140793_object68127:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e130)", 2024-08-17T17:23:52.965 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:57.488455+0000", 2024-08-17T17:23:52.965 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 235.40555128099999, 2024-08-17T17:23:52.965 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2798230580000001, 2024-08-17T17:23:52.965 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:23:52.965 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:23:52.965 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:23:52.965 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:23:52.965 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:23:52.965 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:23:52.966 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68128 2024-08-17T17:23:52.966 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.966 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:23:52.966 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.966 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:23:52.966 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488455+0000", 2024-08-17T17:23:52.966 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:52.966 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.966 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.966 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:23:52.966 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488455+0000", 2024-08-17T17:23:52.966 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:52.966 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.967 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.967 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:23:52.967 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488456+0000", 2024-08-17T17:23:52.967 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2799999999999996e-07 2024-08-17T17:23:52.967 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.967 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.967 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:23:52.967 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488479+0000", 2024-08-17T17:23:52.967 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.368e-05 2024-08-17T17:23:52.967 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.967 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.967 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:23:52.967 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488488+0000", 2024-08-17T17:23:52.968 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2209999999999997e-06 2024-08-17T17:23:52.968 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.968 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.968 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:23:52.968 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488493+0000", 2024-08-17T17:23:52.968 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8080000000000001e-06 2024-08-17T17:23:52.968 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.968 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.968 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:52.968 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.490529+0000", 2024-08-17T17:23:52.968 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020353150000000002 2024-08-17T17:23:52.968 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.968 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.968 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:23:52.969 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.490535+0000", 2024-08-17T17:23:52.969 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2450000000000003e-06 2024-08-17T17:23:52.969 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.969 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.969 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:52.969 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.742562+0000", 2024-08-17T17:23:52.969 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.252026654 2024-08-17T17:23:52.969 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.969 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.969 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for degraded object", 2024-08-17T17:23:52.969 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.743351+0000", 2024-08-17T17:23:52.969 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00078913699999999998 2024-08-17T17:23:52.970 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:23:52.970 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:23:52.971 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.971 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.971 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:52.971 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.768179+0000", 2024-08-17T17:23:52.971 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.024828322999999999 2024-08-17T17:23:52.971 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.971 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.971 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:23:52.971 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.768278+0000", 2024-08-17T17:23:52.971 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9147000000000005e-05 2024-08-17T17:23:52.971 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:52.971 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:23:52.972 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:52.972 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.972 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.972 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:68137 5.4 5:243c5be5:::benchmark_data_smithi003_140793_object68136:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e130)", 2024-08-17T17:23:52.972 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:57.488502+0000", 2024-08-17T17:23:52.972 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 235.405504216, 2024-08-17T17:23:52.972 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.283335688, 2024-08-17T17:23:52.972 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:23:52.972 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:23:52.972 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:23:52.972 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:23:52.972 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:23:52.972 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:23:52.972 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68137 2024-08-17T17:23:52.973 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.973 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:23:52.973 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.973 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:23:52.973 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488502+0000", 2024-08-17T17:23:52.973 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:52.973 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.973 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.973 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:23:52.973 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488502+0000", 2024-08-17T17:23:52.973 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:52.973 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.973 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.974 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:23:52.974 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488503+0000", 2024-08-17T17:23:52.974 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2399999999999998e-07 2024-08-17T17:23:52.974 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.974 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.974 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:23:52.974 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488624+0000", 2024-08-17T17:23:52.974 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012089799999999999 2024-08-17T17:23:52.974 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.974 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.974 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:23:52.974 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488633+0000", 2024-08-17T17:23:52.974 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8660000000000001e-06 2024-08-17T17:23:52.975 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.975 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.975 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:23:52.975 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.488640+0000", 2024-08-17T17:23:52.975 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8329999999999996e-06 2024-08-17T17:23:52.975 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.975 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.975 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:52.975 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.490556+0000", 2024-08-17T17:23:52.975 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0019160690000000001 2024-08-17T17:23:52.975 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.975 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.975 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:23:52.975 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.490562+0000", 2024-08-17T17:23:52.975 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.806e-06 2024-08-17T17:23:52.976 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.976 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.976 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:52.976 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.743428+0000", 2024-08-17T17:23:52.976 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.252865457 2024-08-17T17:23:52.976 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.976 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.976 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:23:52.976 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.743738+0000", 2024-08-17T17:23:52.976 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00031069099999999998 2024-08-17T17:23:52.976 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.976 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.976 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-08-17T17:23:52.976 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.743958+0000", 2024-08-17T17:23:52.976 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00021965299999999999 2024-08-17T17:23:52.977 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.977 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.977 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:23:52.977 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.761495+0000", 2024-08-17T17:23:52.977 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017537516 2024-08-17T17:23:52.977 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.977 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.977 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:23:52.977 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.761545+0000", 2024-08-17T17:23:52.977 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9064000000000003e-05 2024-08-17T17:23:52.977 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.977 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.977 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:23:52.978 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.768317+0000", 2024-08-17T17:23:52.978 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0067721500000000002 2024-08-17T17:23:52.978 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.978 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.978 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:23:52.978 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.768369+0000", 2024-08-17T17:23:52.978 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1873000000000003e-05 2024-08-17T17:23:52.978 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.978 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.978 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:23:52.978 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.771838+0000", 2024-08-17T17:23:52.978 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034692880000000001 2024-08-17T17:23:52.978 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:52.979 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:23:52.979 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:52.979 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.979 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.979 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:68153 5.a 5:5a530647:::benchmark_data_smithi003_140793_object68152:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-08-17T17:23:52.979 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:58.737452+0000", 2024-08-17T17:23:52.979 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 234.156554675, 2024-08-17T17:23:52.979 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.109671217, 2024-08-17T17:23:52.979 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:23:52.979 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:23:52.979 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:23:52.979 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:23:52.979 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:23:52.979 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:23:52.980 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68153 2024-08-17T17:23:52.980 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.980 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:23:52.980 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.980 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:23:52.980 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.737452+0000", 2024-08-17T17:23:52.980 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:52.980 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.980 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.980 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:23:52.980 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.737452+0000", 2024-08-17T17:23:52.980 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:52.980 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.980 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.980 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:23:52.981 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.737454+0000", 2024-08-17T17:23:52.981 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0030000000000001e-06 2024-08-17T17:23:52.981 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.981 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.981 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:23:52.981 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.737487+0000", 2024-08-17T17:23:52.981 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2826e-05 2024-08-17T17:23:52.981 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.981 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.981 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:23:52.981 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.737504+0000", 2024-08-17T17:23:52.981 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7893999999999998e-05 2024-08-17T17:23:52.981 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.981 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.981 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:23:52.982 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.737515+0000", 2024-08-17T17:23:52.982 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0338e-05 2024-08-17T17:23:52.982 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.982 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.982 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:52.982 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.737601+0000", 2024-08-17T17:23:52.982 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5929999999999999e-05 2024-08-17T17:23:52.982 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.982 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.982 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:23:52.982 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.737612+0000", 2024-08-17T17:23:52.982 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0998999999999999e-05 2024-08-17T17:23:52.982 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.983 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.983 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:52.983 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.804970+0000", 2024-08-17T17:23:52.983 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.067358630000000003 2024-08-17T17:23:52.983 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.983 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.983 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:23:52.983 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.805135+0000", 2024-08-17T17:23:52.983 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000164742 2024-08-17T17:23:52.983 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.983 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.983 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:23:52.983 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.805327+0000", 2024-08-17T17:23:52.983 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000192382 2024-08-17T17:23:52.983 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.984 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.984 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:23:52.984 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.813112+0000", 2024-08-17T17:23:52.984 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0077842440000000001 2024-08-17T17:23:52.984 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.984 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.984 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:23:52.984 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.814548+0000", 2024-08-17T17:23:52.984 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014368040000000001 2024-08-17T17:23:52.984 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.984 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.984 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:23:52.984 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.840255+0000", 2024-08-17T17:23:52.984 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.025706058 2024-08-17T17:23:52.985 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.985 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.985 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:23:52.985 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.840480+0000", 2024-08-17T17:23:52.985 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00022594 2024-08-17T17:23:52.985 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.985 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.985 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:23:52.985 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.847123+0000", 2024-08-17T17:23:52.985 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066424270000000002 2024-08-17T17:23:52.985 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:52.985 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:23:52.985 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:52.985 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.985 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.986 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:114038 5.4 5:2445ff84:::benchmark_data_smithi003_140793_object11981:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-08-17T17:23:52.986 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:21:01.212871+0000", 2024-08-17T17:23:52.986 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 171.68113519600001, 2024-08-17T17:23:52.986 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.11732702, 2024-08-17T17:23:52.986 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:23:52.986 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:23:52.986 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:23:52.986 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:23:52.986 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:23:52.986 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:23:52.986 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 114038 2024-08-17T17:23:52.986 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.986 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:23:52.986 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.987 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:23:52.987 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:21:01.212871+0000", 2024-08-17T17:23:52.987 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:52.987 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.987 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.987 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:23:52.987 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:21:01.212871+0000", 2024-08-17T17:23:52.987 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:52.987 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.987 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.987 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:23:52.987 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:21:01.212873+0000", 2024-08-17T17:23:52.987 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6419999999999999e-06 2024-08-17T17:23:52.988 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.988 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.988 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:23:52.988 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:21:01.212878+0000", 2024-08-17T17:23:52.988 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5450000000000003e-06 2024-08-17T17:23:52.988 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.988 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.988 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:23:52.988 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:21:01.212893+0000", 2024-08-17T17:23:52.988 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4212e-05 2024-08-17T17:23:52.988 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.988 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.989 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:23:52.989 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:21:01.212900+0000", 2024-08-17T17:23:52.989 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9100000000000005e-06 2024-08-17T17:23:52.989 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.989 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.989 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:52.989 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:21:01.214755+0000", 2024-08-17T17:23:52.989 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001854477 2024-08-17T17:23:52.989 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.989 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.989 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:23:52.989 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:21:01.214873+0000", 2024-08-17T17:23:52.989 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000118443 2024-08-17T17:23:52.989 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.990 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.990 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-08-17T17:23:52.990 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:21:01.215013+0000", 2024-08-17T17:23:52.990 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000139882 2024-08-17T17:23:52.990 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.990 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.990 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:23:52.990 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:21:01.218687+0000", 2024-08-17T17:23:52.990 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0036740240000000001 2024-08-17T17:23:52.991 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.991 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.991 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:23:52.991 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:21:01.219014+0000", 2024-08-17T17:23:52.991 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00032652199999999999 2024-08-17T17:23:52.991 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.991 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.991 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:23:52.991 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:21:01.327218+0000", 2024-08-17T17:23:52.991 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.10820421600000001 2024-08-17T17:23:52.991 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.991 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.992 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:23:52.992 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:21:01.327294+0000", 2024-08-17T17:23:52.992 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5574000000000005e-05 2024-08-17T17:23:52.992 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.992 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.992 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:23:52.992 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:21:01.330198+0000", 2024-08-17T17:23:52.992 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029045730000000001 2024-08-17T17:23:52.992 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:52.992 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:23:52.992 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:52.992 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.992 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.992 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:114043 5.4 5:2bbddf94:::benchmark_data_smithi003_140793_object11986:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-08-17T17:23:52.993 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:21:01.221421+0000", 2024-08-17T17:23:52.993 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 171.672584973, 2024-08-17T17:23:52.993 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.111304932, 2024-08-17T17:23:52.993 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:23:52.993 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:23:52.993 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:23:52.993 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:23:52.993 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:23:52.993 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:23:52.993 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 114043 2024-08-17T17:23:52.993 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.993 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:23:52.994 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.994 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:23:52.994 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:21:01.221421+0000", 2024-08-17T17:23:52.994 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:52.994 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.994 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.994 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:23:52.994 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:21:01.221421+0000", 2024-08-17T17:23:52.994 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:52.994 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.994 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.994 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:23:52.994 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:21:01.221423+0000", 2024-08-17T17:23:52.995 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.367e-06 2024-08-17T17:23:52.995 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.995 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.995 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:23:52.995 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:21:01.221428+0000", 2024-08-17T17:23:52.995 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9529999999999998e-06 2024-08-17T17:23:52.995 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.995 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.995 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:23:52.995 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:21:01.221439+0000", 2024-08-17T17:23:52.995 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1579e-05 2024-08-17T17:23:52.995 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.995 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.996 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:23:52.996 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:21:01.221448+0000", 2024-08-17T17:23:52.996 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2500000000000006e-06 2024-08-17T17:23:52.996 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.996 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.996 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:52.996 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:21:01.221524+0000", 2024-08-17T17:23:52.996 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6316000000000006e-05 2024-08-17T17:23:52.996 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.996 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.996 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:23:52.996 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:21:01.221639+0000", 2024-08-17T17:23:52.996 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011503800000000001 2024-08-17T17:23:52.997 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.997 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.997 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-08-17T17:23:52.997 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:21:01.221728+0000", 2024-08-17T17:23:52.997 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9173999999999997e-05 2024-08-17T17:23:52.997 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.997 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.997 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:23:52.997 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:21:01.224387+0000", 2024-08-17T17:23:52.997 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026588789999999998 2024-08-17T17:23:52.997 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.997 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.997 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:23:52.997 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:21:01.224408+0000", 2024-08-17T17:23:52.998 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0619999999999999e-05 2024-08-17T17:23:52.998 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.998 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.998 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:23:52.998 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:21:01.330324+0000", 2024-08-17T17:23:52.998 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.105916684 2024-08-17T17:23:52.998 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.998 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.998 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:23:52.998 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:21:01.330373+0000", 2024-08-17T17:23:52.998 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9228999999999999e-05 2024-08-17T17:23:52.998 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.999 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.999 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:23:52.999 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:21:01.332726+0000", 2024-08-17T17:23:52.999 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0023528429999999999 2024-08-17T17:23:52.999 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:52.999 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:23:52.999 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:52.999 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:52.999 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:52.999 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.35277.0:155041 5.6 e152/150 5:687221f3:::benchmark_data_smithi003_140793_object52984:head v 152'9754, mlcod=128'4316)", 2024-08-17T17:23:52.999 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:22:05.017695+0000", 2024-08-17T17:23:52.999 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 107.876311497, 2024-08-17T17:23:52.999 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.121108375, 2024-08-17T17:23:53.000 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:23:53.000 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:23:53.000 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:23:53.000 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:23:53.000 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.000 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:23:53.000 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:05.017695+0000", 2024-08-17T17:23:53.000 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:53.000 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.000 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.000 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:23:53.000 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:05.017695+0000", 2024-08-17T17:23:53.000 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:53.000 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.001 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.001 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:23:53.001 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:05.017696+0000", 2024-08-17T17:23:53.001 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5999999999999991e-07 2024-08-17T17:23:53.001 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.001 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.001 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:23:53.001 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:05.017702+0000", 2024-08-17T17:23:53.001 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6460000000000004e-06 2024-08-17T17:23:53.001 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.001 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.001 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:23:53.001 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:05.017714+0000", 2024-08-17T17:23:53.001 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1124999999999999e-05 2024-08-17T17:23:53.001 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.002 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.002 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:23:53.002 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:05.017721+0000", 2024-08-17T17:23:53.002 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4440000000000002e-06 2024-08-17T17:23:53.002 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.002 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.002 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:53.002 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:05.017880+0000", 2024-08-17T17:23:53.002 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015906399999999999 2024-08-17T17:23:53.002 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.002 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.002 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:23:53.002 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:05.017903+0000", 2024-08-17T17:23:53.002 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3252999999999999e-05 2024-08-17T17:23:53.003 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.003 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.003 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:23:53.003 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:05.138719+0000", 2024-08-17T17:23:53.003 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.12081560199999999 2024-08-17T17:23:53.003 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.003 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.003 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:23:53.003 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:05.138803+0000", 2024-08-17T17:23:53.003 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4281000000000002e-05 2024-08-17T17:23:53.003 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:53.003 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:23:53.003 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:53.003 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.003 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.004 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.35277.0:155073 5.6 e152/150 5:6e7ff565:::benchmark_data_smithi003_140793_object53016:head v 152'9758, mlcod=128'4316)", 2024-08-17T17:23:53.004 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:22:05.054509+0000", 2024-08-17T17:23:53.004 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 107.83949742, 2024-08-17T17:23:53.004 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.113264167, 2024-08-17T17:23:53.004 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:23:53.004 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:23:53.004 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:23:53.004 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:23:53.004 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.004 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:23:53.004 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:05.054509+0000", 2024-08-17T17:23:53.004 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:53.004 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.004 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.005 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:23:53.005 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:05.054509+0000", 2024-08-17T17:23:53.005 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:53.005 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.005 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.005 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:23:53.005 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:05.054509+0000", 2024-08-17T17:23:53.005 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6600000000000003e-07 2024-08-17T17:23:53.005 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.005 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.005 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:23:53.005 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:05.054513+0000", 2024-08-17T17:23:53.005 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4599999999999999e-06 2024-08-17T17:23:53.005 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.005 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.006 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:23:53.006 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:05.054519+0000", 2024-08-17T17:23:53.006 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1550000000000004e-06 2024-08-17T17:23:53.006 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.006 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.006 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:23:53.006 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:05.054523+0000", 2024-08-17T17:23:53.006 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4569999999999998e-06 2024-08-17T17:23:53.006 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.006 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.006 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:53.006 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:05.081080+0000", 2024-08-17T17:23:53.006 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.026557197000000001 2024-08-17T17:23:53.006 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.007 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.007 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:23:53.007 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:05.081107+0000", 2024-08-17T17:23:53.007 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6276999999999999e-05 2024-08-17T17:23:53.007 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.007 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.007 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:23:53.007 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:05.167662+0000", 2024-08-17T17:23:53.007 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.086555696000000001 2024-08-17T17:23:53.007 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.007 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.007 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:23:53.007 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:05.167773+0000", 2024-08-17T17:23:53.008 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110659 2024-08-17T17:23:53.008 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:53.008 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:23:53.008 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:53.008 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.008 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.008 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.35277.0:156849 5.6 e152/150 5:641d2640:::benchmark_data_smithi003_140793_object54792:head v 152'9868, mlcod=128'4316)", 2024-08-17T17:23:53.008 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:22:07.790540+0000", 2024-08-17T17:23:53.008 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 105.103465926, 2024-08-17T17:23:53.008 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.118630493, 2024-08-17T17:23:53.008 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:23:53.008 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:23:53.008 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:23:53.009 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:23:53.009 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.009 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:23:53.009 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:07.790540+0000", 2024-08-17T17:23:53.009 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:53.009 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.009 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.009 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:23:53.009 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:07.790540+0000", 2024-08-17T17:23:53.009 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:53.009 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.009 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.009 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:23:53.009 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:07.790542+0000", 2024-08-17T17:23:53.009 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.144e-06 2024-08-17T17:23:53.010 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.010 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.010 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:23:53.010 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:07.790550+0000", 2024-08-17T17:23:53.010 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4719999999999995e-06 2024-08-17T17:23:53.010 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.010 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.010 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:23:53.010 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:07.790568+0000", 2024-08-17T17:23:53.010 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8337000000000001e-05 2024-08-17T17:23:53.010 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.010 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.010 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:23:53.010 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:07.790577+0000", 2024-08-17T17:23:53.011 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9530000000000005e-06 2024-08-17T17:23:53.011 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.011 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.011 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:53.011 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:07.825809+0000", 2024-08-17T17:23:53.011 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.035231578999999999 2024-08-17T17:23:53.011 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.011 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.011 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:23:53.011 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:07.825849+0000", 2024-08-17T17:23:53.011 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9851000000000002e-05 2024-08-17T17:23:53.011 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.011 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.012 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:23:53.012 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:07.909126+0000", 2024-08-17T17:23:53.012 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.083276839000000005 2024-08-17T17:23:53.012 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.012 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.012 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:23:53.012 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:07.909171+0000", 2024-08-17T17:23:53.012 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5318000000000002e-05 2024-08-17T17:23:53.012 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:53.012 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:23:53.012 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:53.012 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.012 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.013 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.35277.0:156857 5.6 e152/150 5:6bc4dee8:::benchmark_data_smithi003_140793_object54800:head v 152'9869, mlcod=128'4316)", 2024-08-17T17:23:53.013 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:22:07.790595+0000", 2024-08-17T17:23:53.013 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 105.10341101900001, 2024-08-17T17:23:53.013 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.118613528, 2024-08-17T17:23:53.013 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:23:53.013 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:23:53.013 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:23:53.013 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:23:53.013 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.013 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:23:53.013 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:07.790595+0000", 2024-08-17T17:23:53.013 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:53.013 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.014 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.014 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:23:53.014 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:07.790595+0000", 2024-08-17T17:23:53.014 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:53.014 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.014 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.014 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:23:53.014 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:07.790596+0000", 2024-08-17T17:23:53.014 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2800000000000003e-07 2024-08-17T17:23:53.014 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.014 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.014 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:23:53.014 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:07.790610+0000", 2024-08-17T17:23:53.015 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4368e-05 2024-08-17T17:23:53.015 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.015 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.015 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:23:53.015 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:07.790621+0000", 2024-08-17T17:23:53.015 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1289e-05 2024-08-17T17:23:53.015 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.015 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.015 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:23:53.015 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:07.790627+0000", 2024-08-17T17:23:53.015 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4430000000000002e-06 2024-08-17T17:23:53.015 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.015 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.016 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:53.016 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:07.835296+0000", 2024-08-17T17:23:53.016 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.044668767999999998 2024-08-17T17:23:53.016 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.016 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.016 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:23:53.016 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:07.835334+0000", 2024-08-17T17:23:53.016 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8584000000000002e-05 2024-08-17T17:23:53.016 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.016 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.016 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:23:53.016 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:07.909177+0000", 2024-08-17T17:23:53.016 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.073842568999999997 2024-08-17T17:23:53.016 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.017 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.017 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:23:53.017 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:07.909209+0000", 2024-08-17T17:23:53.017 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2178999999999998e-05 2024-08-17T17:23:53.017 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:53.017 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:23:53.017 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:53.017 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.017 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.017 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.35277.0:177150 5.6 e157/150 5:6e12f559:::benchmark_data_smithi003_140793_object75093:head v 157'11177, mlcod=128'4316)", 2024-08-17T17:23:53.017 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:22:38.269025+0000", 2024-08-17T17:23:53.017 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 74.624980927999999, 2024-08-17T17:23:53.017 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.109402236, 2024-08-17T17:23:53.017 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:23:53.018 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:23:53.018 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:23:53.018 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:23:53.018 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.018 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:23:53.018 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:38.269025+0000", 2024-08-17T17:23:53.018 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:53.018 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.018 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.018 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:23:53.018 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:38.269025+0000", 2024-08-17T17:23:53.018 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:23:53.018 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.018 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.018 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:23:53.019 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:38.269027+0000", 2024-08-17T17:23:53.019 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4309999999999999e-06 2024-08-17T17:23:53.019 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.019 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.019 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:23:53.019 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:38.269035+0000", 2024-08-17T17:23:53.019 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4139999999999994e-06 2024-08-17T17:23:53.019 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.019 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.019 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:23:53.019 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:38.269053+0000", 2024-08-17T17:23:53.019 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7774999999999998e-05 2024-08-17T17:23:53.019 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.019 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.020 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:23:53.020 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:38.269062+0000", 2024-08-17T17:23:53.020 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8249999999999994e-06 2024-08-17T17:23:53.020 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.020 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.020 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:23:53.020 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:38.317143+0000", 2024-08-17T17:23:53.020 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.048081271000000002 2024-08-17T17:23:53.020 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.020 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.020 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:23:53.020 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:38.317167+0000", 2024-08-17T17:23:53.020 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3677999999999999e-05 2024-08-17T17:23:53.020 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.021 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.021 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:23:53.021 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:38.378376+0000", 2024-08-17T17:23:53.021 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.06120921 2024-08-17T17:23:53.021 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:23:53.021 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:23:53.021 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:23:53.021 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:22:38.378428+0000", 2024-08-17T17:23:53.021 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1632000000000003e-05 2024-08-17T17:23:53.021 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:53.021 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:23:53.021 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:53.021 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:23:53.021 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:23:53.022 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:23:53.055 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:53.078 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:53.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:53.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:53.179 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:53.189 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:23:53.206 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:53.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:53.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:53.308 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:53.330 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:53.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:53.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:53.432 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:53.457 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:53.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:53.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:53.559 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:53.581 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:53.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:53.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:53.636 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:53 smithi003 ceph-mon[103344]: pgmap v420: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 4.6 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 2130/84 objects degraded (2535.714%); 0 B/s, 224 objects/s recovering 2024-08-17T17:23:53.636 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:23:53.636 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:23:53.636 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:23:53.636 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:23:53.636 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:23:53.636 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:53 smithi003 ceph-mon[100615]: pgmap v420: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 4.6 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 2130/84 objects degraded (2535.714%); 0 B/s, 224 objects/s recovering 2024-08-17T17:23:53.636 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:23:53.637 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:23:53.637 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:23:53.637 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:23:53.637 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:23:53.682 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:53.704 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:53.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:53.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:53.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:53 smithi053 ceph-mon[89963]: pgmap v420: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 4.6 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 2130/84 objects degraded (2535.714%); 0 B/s, 224 objects/s recovering 2024-08-17T17:23:53.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:23:53.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:23:53.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:23:53.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:23:53.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:23:53.806 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:53.830 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:53.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:53.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:53.932 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:53.955 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:53.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:53.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:54.057 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:54.080 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:54.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:54.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:54.182 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:54.204 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:54.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:54.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:54.306 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:54.329 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:54.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:54.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:54.431 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:54.453 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:54.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:54.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:54.555 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:54.576 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:54.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:54.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:54.678 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:54.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:54.702 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:54.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:54.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:54.828 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:54.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:54.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:54.929 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:54.952 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:54.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:54.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:55.053 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:55.076 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:55.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:55.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:55.178 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:55.200 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:55.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:55.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:55.303 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:55.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:55.326 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:55.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:55.427 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:55.449 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:55.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:55.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:55.551 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:55.574 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:55.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:55.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:55.676 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:55.698 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:55.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:55.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:55.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:55 smithi003 ceph-mon[100615]: pgmap v421: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 4.6 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 2130/84 objects degraded (2535.714%); 0 B/s, 357 objects/s recovering 2024-08-17T17:23:55.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:23:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:23:55] "GET /metrics HTTP/1.1" 200 36576 "" "Prometheus/2.51.0" 2024-08-17T17:23:55.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:55 smithi003 ceph-mon[103344]: pgmap v421: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 4.6 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 2130/84 objects degraded (2535.714%); 0 B/s, 357 objects/s recovering 2024-08-17T17:23:55.801 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:55.823 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:55.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:55.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:55.925 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:55.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:55.949 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:55.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:55.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:55 smithi053 ceph-mon[89963]: pgmap v421: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 4.6 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 2130/84 objects degraded (2535.714%); 0 B/s, 357 objects/s recovering 2024-08-17T17:23:56.050 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:56.072 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:56.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:56.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:56.174 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:56.197 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:56.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:56.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:56.299 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:56.321 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:56.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:56.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:56.423 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:56.446 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:56.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:56.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:56.547 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:56.570 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:56.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:56.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:56.672 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:56.695 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:56.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:56.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:56.796 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:56.819 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:56.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:56.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:56.921 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:56.943 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:56.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:56.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:57.045 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:57.069 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:57.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:57.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:57.153 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:23:57.153 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:23:57.170 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:57.171 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:23:57.171 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:23:57.188 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:23:57.188 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:23:57.193 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:57.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:57.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:57.205 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:23:57.205 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:23:57.222 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:23:57.223 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:23:57.239 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:23:57.239 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:23:57.255 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:23:57.256 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:23:57.272 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:23:57.272 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:23:57.294 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:57.317 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:57.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:57.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:57.387 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-17T17:23:57.388 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 0 1 2024-08-17T17:23:57.418 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:57.454 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:57.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:57.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:57.557 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:57.580 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:57.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:57.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:57.682 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:57.704 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:57.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:57.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:57.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:57 smithi003 ceph-mon[100615]: pgmap v422: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 4.6 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 2130/84 objects degraded (2535.714%); 0 B/s, 318 objects/s recovering 2024-08-17T17:23:57.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:57 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3931023353' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:23:57.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:57 smithi003 ceph-mon[103344]: pgmap v422: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 4.6 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 2130/84 objects degraded (2535.714%); 0 B/s, 318 objects/s recovering 2024-08-17T17:23:57.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:57 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3931023353' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:23:57.806 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:23:57.829 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:57.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:57.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:23:57.931 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:57.953 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:57.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:57.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:57.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:57 smithi053 ceph-mon[89963]: pgmap v422: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 4.6 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 2130/84 objects degraded (2535.714%); 0 B/s, 318 objects/s recovering 2024-08-17T17:23:57.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:57 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3931023353' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:23:58.055 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:23:58.077 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:58.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:58.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:23:58.179 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:58.201 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:58.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:58.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:58.303 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:58.325 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:58.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:58.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:58.427 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:23:58.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:58.450 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:58.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:23:58.477 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-17T17:23:58.551 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:58.574 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:58.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:58.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:58.676 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:23:58.701 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:58.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:58.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:23:58.802 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:58.825 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:58.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:58.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:58.927 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:23:58.952 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:58.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:58.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:23:59.054 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:59.077 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:59.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:59.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:59.178 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:59.202 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:59.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:59.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:59.304 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:59.326 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:59.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:59.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:59.428 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:59.451 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:59.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:59.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:59.490 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-08-17T17:23:59.552 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:23:59.574 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:23:59.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:59.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:23:59.676 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:59.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:59 smithi003 ceph-mon[100615]: pgmap v423: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 4.6 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 2130/84 objects degraded (2535.714%); 0 B/s, 236 objects/s recovering 2024-08-17T17:23:59.684 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:59 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/522884059' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:23:59.685 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:23:59 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:23:59.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:59 smithi003 ceph-mon[103344]: pgmap v423: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 4.6 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 2130/84 objects degraded (2535.714%); 0 B/s, 236 objects/s recovering 2024-08-17T17:23:59.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:59 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/522884059' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:23:59.685 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:23:59 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:23:59.699 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:59.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:59.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:59.800 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:23:59.822 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:59.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:59.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:23:59.924 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:23:59.947 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:23:59.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:23:59.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:23:59.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:59 smithi053 ceph-mon[89963]: pgmap v423: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 4.6 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 2130/84 objects degraded (2535.714%); 0 B/s, 236 objects/s recovering 2024-08-17T17:23:59.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:59 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/522884059' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:23:59.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:23:59 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:24:00.049 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:00.072 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:00.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:00.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:00.174 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:00.197 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:00.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:00.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:00.299 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:00.321 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:00.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:00.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:00.422 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:00.446 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:00.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:00.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:00.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:24:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:24:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:24:00.495 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_0' removed 2024-08-17T17:24:00.549 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:00.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:00.572 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:00.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:00.673 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:00.695 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:00.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:00.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:00.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:00 smithi003 ceph-mon[103344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:24:00.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:00 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:24:00.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:00 smithi003 ceph-mon[103344]: osdmap e171: 8 total, 8 up, 7 in 2024-08-17T17:24:00.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:00 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3780953780' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:24:00.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:00 smithi003 ceph-mon[100615]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:24:00.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:00 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:24:00.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:00 smithi003 ceph-mon[100615]: osdmap e171: 8 total, 8 up, 7 in 2024-08-17T17:24:00.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:00 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3780953780' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:24:00.797 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:00.820 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:00.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:00.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:00.922 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:00.944 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:00.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:00.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:00.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:00 smithi053 ceph-mon[89963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:24:00.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:00 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:24:00.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:00 smithi053 ceph-mon[89963]: osdmap e171: 8 total, 8 up, 7 in 2024-08-17T17:24:00.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:00 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3780953780' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:24:01.046 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:01.072 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:01.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:01.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:01.173 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:01.195 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:01.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:01.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:01.297 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:01.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:01.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:01.421 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:01.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:01.448 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:01.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:01.549 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:01.578 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:01.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:01.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:01.618 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-08-17T17:24:01.619 INFO:tasks.radosbench:Beginning radosbench... 2024-08-17T17:24:01.620 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-08-17T17:24:01.620 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd pool create unique_pool_1 16 2024-08-17T17:24:01.635 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:24:01.635 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:24:01.681 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:01.703 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:01.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:01.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:01.712 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:24:01.763 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:01 smithi003 ceph-mon[103344]: pgmap v425: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 4.6 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 1208/84 objects degraded (1438.095%); 0 B/s, 305 objects/s recovering 2024-08-17T17:24:01.763 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:01 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 1208/84 objects degraded (1438.095%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:24:01.763 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:01 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3780953780' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T17:24:01.763 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:01 smithi003 ceph-mon[103344]: osdmap e172: 8 total, 8 up, 7 in 2024-08-17T17:24:01.763 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:01 smithi003 ceph-mon[100615]: pgmap v425: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 4.6 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 1208/84 objects degraded (1438.095%); 0 B/s, 305 objects/s recovering 2024-08-17T17:24:01.763 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:01 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 1208/84 objects degraded (1438.095%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:24:01.763 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:01 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3780953780' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T17:24:01.764 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:01 smithi003 ceph-mon[100615]: osdmap e172: 8 total, 8 up, 7 in 2024-08-17T17:24:01.805 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:01.829 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:01.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:01.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:01.931 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:01.956 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:01.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:01.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:01.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:01 smithi053 ceph-mon[89963]: pgmap v425: 89 pgs: 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 4.6 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 1208/84 objects degraded (1438.095%); 0 B/s, 305 objects/s recovering 2024-08-17T17:24:01.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:01 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 1208/84 objects degraded (1438.095%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:24:01.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:01 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3780953780' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T17:24:01.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:01 smithi053 ceph-mon[89963]: osdmap e172: 8 total, 8 up, 7 in 2024-08-17T17:24:02.058 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:02.080 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:02.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:02.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:02.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:02.206 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:02.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:02.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:02.307 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:02.330 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:02.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:02.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:02.432 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:02.454 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:02.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:02.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:02.556 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:02.582 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:02.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:02.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:02.684 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:02.696 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:02 smithi003 ceph-mon[103344]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1208/84 objects degraded (1438.095%), 1 pg degraded, 1 pg undersized) 2024-08-17T17:24:02.697 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:02 smithi003 ceph-mon[100615]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1208/84 objects degraded (1438.095%), 1 pg degraded, 1 pg undersized) 2024-08-17T17:24:02.710 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:02.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:02.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:02.812 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:02.834 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:02.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:02.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:02.844 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:24:02.936 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:02.962 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:02.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:02.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:02.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:02 smithi053 ceph-mon[89963]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1208/84 objects degraded (1438.095%), 1 pg degraded, 1 pg undersized) 2024-08-17T17:24:03.064 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:03.090 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:03.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:03.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:03.192 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:03.215 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:03.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:03.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:03.317 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:03.339 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:03.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:03.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:03.441 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:03.463 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:03.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:03.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:03.565 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:03.592 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:03.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:03.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:03.694 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:03.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:03.718 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:03.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:03.819 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:03.846 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:03.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:03.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:03.867 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:03 smithi053 ceph-mon[89963]: pgmap v427: 73 pgs: 73 active+clean; 4.6 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:03.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:03 smithi003 ceph-mon[103344]: pgmap v427: 73 pgs: 73 active+clean; 4.6 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:03.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:03 smithi003 ceph-mon[100615]: pgmap v427: 73 pgs: 73 active+clean; 4.6 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:03.948 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:03.971 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:03.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:03.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:04.073 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:04.095 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:04.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:04.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:04.197 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:04.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:04.223 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:04.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:04.324 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:04.346 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:04.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:04.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:04.448 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:04.471 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:04.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:04.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:04.573 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:04.595 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:04.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:04.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:04.697 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:04.724 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:04.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:04.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:04.826 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:04.842 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:24:04.842 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:24:04.848 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:04.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:04.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:04.950 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:04.972 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:04.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:04.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:05.074 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:05.078 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:24:05.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:05.103 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:05.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:05.204 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:05.227 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:05.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:05.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:05.329 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:05.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:05.358 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:05.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:05.460 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:05.482 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:05.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:05.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:05.558 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.0 to 1 (10000) 2024-08-17T17:24:05.577 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:05 smithi003 ceph-mon[103344]: pgmap v428: 73 pgs: 73 active+clean; 4.6 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:05.577 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:05 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3018356427' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T17:24:05.577 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:05 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T17:24:05.577 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:05 smithi003 ceph-mon[100615]: pgmap v428: 73 pgs: 73 active+clean; 4.6 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:05.577 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:05 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3018356427' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T17:24:05.577 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:05 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T17:24:05.584 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:05.608 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:05.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:05.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:05.619 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-17T17:24:05.710 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:05.736 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:05.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:05.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:05.838 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:05.865 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:05.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:05.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:05.955 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:24:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:24:05] "GET /metrics HTTP/1.1" 200 36576 "" "Prometheus/2.51.0" 2024-08-17T17:24:05.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:05 smithi053 ceph-mon[89963]: pgmap v428: 73 pgs: 73 active+clean; 4.6 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:05.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:05 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3018356427' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T17:24:05.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:05 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T17:24:05.967 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:05.993 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:05.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:05.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:06.095 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:06.121 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:06.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:06.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:06.223 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:06.250 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:06.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:06.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:06.288 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:24:06.288 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:24:06.324 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 1 1 2024-08-17T17:24:06.352 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:06.379 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:06.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:06.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:06.481 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:06.503 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:06.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:06.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:06.605 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:06.628 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:06.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:06.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:06.730 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:06.756 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:06.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:06.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:06.846 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:06 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-17T17:24:06.846 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:06 smithi003 ceph-mon[103344]: osdmap e173: 8 total, 8 up, 7 in 2024-08-17T17:24:06.846 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:06 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-17T17:24:06.846 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:06 smithi003 ceph-mon[100615]: osdmap e173: 8 total, 8 up, 7 in 2024-08-17T17:24:06.858 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:06.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:06.881 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:06.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:06.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:06 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-17T17:24:06.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:06 smithi053 ceph-mon[89963]: osdmap e173: 8 total, 8 up, 7 in 2024-08-17T17:24:06.982 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:07.003 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:07.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:07.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:07.105 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:07.128 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:07.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:07.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:07.230 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:07.257 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:07.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:07.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:07.359 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:07.385 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:07.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:07.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:07.487 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:07.514 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:07.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:07.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:07.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:07.639 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:07.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:07.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:07.740 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:07.763 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:07.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:07.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:07.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:07.886 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:07.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:07.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:07.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:07 smithi053 ceph-mon[89963]: pgmap v430: 73 pgs: 73 active+clean; 4.6 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:07.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:07 smithi053 ceph-mon[89963]: osdmap e174: 8 total, 8 up, 7 in 2024-08-17T17:24:07.988 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:08.009 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:07 smithi003 ceph-mon[100615]: pgmap v430: 73 pgs: 73 active+clean; 4.6 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:08.009 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:07 smithi003 ceph-mon[100615]: osdmap e174: 8 total, 8 up, 7 in 2024-08-17T17:24:08.010 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:07 smithi003 ceph-mon[103344]: pgmap v430: 73 pgs: 73 active+clean; 4.6 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:08.010 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:07 smithi003 ceph-mon[103344]: osdmap e174: 8 total, 8 up, 7 in 2024-08-17T17:24:08.015 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:08.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:08.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:08.116 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:08.143 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:08.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:08.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:08.245 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:08.270 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:08.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:08.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:08.371 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:08.399 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:08.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:08.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:08.500 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:08.527 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:08.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:08.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:08.629 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:08.656 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:08.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:08.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:08.730 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:08 smithi003 ceph-mon[103344]: osdmap e175: 8 total, 8 up, 7 in 2024-08-17T17:24:08.730 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:24:08.731 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:24:08.731 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:24:08.731 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:08 smithi003 ceph-mon[100615]: osdmap e175: 8 total, 8 up, 7 in 2024-08-17T17:24:08.731 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:24:08.731 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:24:08.731 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:24:08.758 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:08.785 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:08.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:08.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:08.887 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:08.910 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:08.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:08.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:08.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:08 smithi053 ceph-mon[89963]: osdmap e175: 8 total, 8 up, 7 in 2024-08-17T17:24:08.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:24:08.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:24:08.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:24:09.012 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:09.034 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:09.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:09.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:09.136 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:09.163 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:09.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:09.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:09.265 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:09.292 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:09.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:09.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:09.394 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:09.420 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:09.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:09.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:09.521 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:09.548 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:09.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:09.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:09.650 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:09.677 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:09.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:09.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:09.779 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:09.802 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:24:09.805 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:09.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:09.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:09.907 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:09.930 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:09 smithi003 ceph-mon[103344]: pgmap v433: 73 pgs: 3 peering, 70 active+clean; 4.6 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:09.930 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:09 smithi003 ceph-mon[103344]: osdmap e176: 8 total, 8 up, 7 in 2024-08-17T17:24:09.931 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:09 smithi003 ceph-mon[100615]: pgmap v433: 73 pgs: 3 peering, 70 active+clean; 4.6 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:09.931 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:09 smithi003 ceph-mon[100615]: osdmap e176: 8 total, 8 up, 7 in 2024-08-17T17:24:09.933 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:09.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:09.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:09.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:09 smithi053 ceph-mon[89963]: pgmap v433: 73 pgs: 3 peering, 70 active+clean; 4.6 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:09.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:09 smithi053 ceph-mon[89963]: osdmap e176: 8 total, 8 up, 7 in 2024-08-17T17:24:10.035 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:10.062 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:10.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:10.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:10.164 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:10.190 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:10.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:10.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:10.292 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:10.319 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:10.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:10.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:10.421 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:10.448 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:10.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:10.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:10.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:24:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:24:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:24:10.550 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:10.576 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:10.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:10.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:10.678 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:10.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:10.707 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:10.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:10.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:10.834 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:10.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:10.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:10.919 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:10 smithi003 ceph-mon[103344]: osdmap e177: 8 total, 8 up, 7 in 2024-08-17T17:24:10.919 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:10 smithi003 ceph-mon[100615]: osdmap e177: 8 total, 8 up, 7 in 2024-08-17T17:24:10.936 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:10.963 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:10.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:10.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:10.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:10 smithi053 ceph-mon[89963]: osdmap e177: 8 total, 8 up, 7 in 2024-08-17T17:24:11.065 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:11.093 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:11.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:11.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:11.195 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:11.223 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:11.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:11.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:11.325 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:11.351 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:11.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:11.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:11.454 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:11.480 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:11.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:11.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:11.582 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:11.609 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:11.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:11.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:11.711 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:11.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:11.739 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:11.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:11.839 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:11.866 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:11.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:11.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:11.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:11 smithi053 ceph-mon[89963]: pgmap v436: 73 pgs: 5 peering, 68 active+clean; 4.6 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 144 KiB/s, 4 objects/s recovering 2024-08-17T17:24:11.968 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:11.994 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:11.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:11.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:12.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:11 smithi003 ceph-mon[100615]: pgmap v436: 73 pgs: 5 peering, 68 active+clean; 4.6 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 144 KiB/s, 4 objects/s recovering 2024-08-17T17:24:12.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:11 smithi003 ceph-mon[103344]: pgmap v436: 73 pgs: 5 peering, 68 active+clean; 4.6 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 144 KiB/s, 4 objects/s recovering 2024-08-17T17:24:12.096 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:12.122 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:12.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:12.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:12.224 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:12.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:12.249 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:12.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:12.350 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:12.377 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:12.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:12.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:12.479 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:12.502 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:12.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:12.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:12.604 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:12.627 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:12.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:12.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:12.729 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:12.751 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:12.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:12.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:12.853 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:12.859 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:12 smithi003 ceph-mon[103344]: pgmap v437: 73 pgs: 2 peering, 71 active+clean; 4.6 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 110 KiB/s, 3 objects/s recovering 2024-08-17T17:24:12.860 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:12 smithi003 ceph-mon[100615]: pgmap v437: 73 pgs: 2 peering, 71 active+clean; 4.6 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 110 KiB/s, 3 objects/s recovering 2024-08-17T17:24:12.875 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:12.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:12.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:12.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:12 smithi053 ceph-mon[89963]: pgmap v437: 73 pgs: 2 peering, 71 active+clean; 4.6 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 110 KiB/s, 3 objects/s recovering 2024-08-17T17:24:12.977 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:12.999 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:13.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:13.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:13.101 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:13.128 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:13.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:13.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:13.230 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:13.257 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:13.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:13.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:13.358 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:13.385 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:13.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:13.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:13.487 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:13.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:13.515 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:13.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:13.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:13.617 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-08-17T17:24:13.626 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_1' created 2024-08-17T17:24:13.648 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:13.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:13.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:13.750 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:13.777 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:13.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:13.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:13.851 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:24:13.852 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:24:13.863 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:24:13.863 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:24:13.864 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:13 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2120173148' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-08-17T17:24:13.864 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:13 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-08-17T17:24:13.864 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:13 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1712711118' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:24:13.864 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:13 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2120173148' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-08-17T17:24:13.864 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:13 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-08-17T17:24:13.864 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:13 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1712711118' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:24:13.879 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:13.880 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:24:13.880 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:24:13.896 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:24:13.896 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:24:13.906 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:13.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:13.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:13.912 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:24:13.913 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:24:13.928 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:24:13.929 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:24:13.944 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:24:13.944 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:24:13.960 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:24:13.961 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:24:13.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:13 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2120173148' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-08-17T17:24:13.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:13 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-08-17T17:24:13.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:13 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1712711118' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:24:14.008 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:14.033 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:14.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:14.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:14.135 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:14.157 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:14.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:14.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:14.259 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:14.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:14.283 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:14.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:14.384 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:14.406 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:14.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:14.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:14.508 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:14.530 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:14.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:14.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:14.632 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:14.658 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:14.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:14.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:14.715 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-08-17T17:24:14.760 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:14.787 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:14.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:14.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:14.889 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:14.893 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:14 smithi003 ceph-mon[103344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:24:14.893 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:14 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-08-17T17:24:14.893 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:14 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1712711118' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:24:14.893 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:14 smithi003 ceph-mon[103344]: osdmap e178: 8 total, 8 up, 7 in 2024-08-17T17:24:14.893 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:14 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/133508825' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:24:14.893 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:14 smithi003 ceph-mon[103344]: pgmap v439: 89 pgs: 2 creating+peering, 14 unknown, 2 peering, 71 active+clean; 4.6 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 96 KiB/s, 3 objects/s recovering 2024-08-17T17:24:14.894 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:14 smithi003 ceph-mon[100615]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:24:14.894 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:14 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-08-17T17:24:14.894 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:14 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1712711118' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:24:14.894 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:14 smithi003 ceph-mon[100615]: osdmap e178: 8 total, 8 up, 7 in 2024-08-17T17:24:14.894 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:14 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/133508825' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:24:14.894 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:14 smithi003 ceph-mon[100615]: pgmap v439: 89 pgs: 2 creating+peering, 14 unknown, 2 peering, 71 active+clean; 4.6 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 96 KiB/s, 3 objects/s recovering 2024-08-17T17:24:14.911 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:24:14.911 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-08-17T17:24:14.911 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-08-17T17:24:14.911 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-08-17T17:24:14.911 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:24:14.915 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:14.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:14.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:14.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:14 smithi053 ceph-mon[89963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:24:14.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:14 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-08-17T17:24:14.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:14 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1712711118' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:24:14.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:14 smithi053 ceph-mon[89963]: osdmap e178: 8 total, 8 up, 7 in 2024-08-17T17:24:14.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:14 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/133508825' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:24:14.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:14 smithi053 ceph-mon[89963]: pgmap v439: 89 pgs: 2 creating+peering, 14 unknown, 2 peering, 71 active+clean; 4.6 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 96 KiB/s, 3 objects/s recovering 2024-08-17T17:24:15.016 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:15.044 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:15.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:15.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:15.146 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:15.172 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:15.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:15.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:15.274 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:15.297 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:15.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:15.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:15.399 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:15.428 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:15.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:15.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:15.530 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:15.552 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:15.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:15.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:15.654 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:15.676 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:15.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:15.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:15.721 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-17T17:24:15.756 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:24:15.779 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:15.806 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:15.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:15.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:15.900 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:24:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:24:15] "GET /metrics HTTP/1.1" 200 34204 "" "Prometheus/2.51.0" 2024-08-17T17:24:15.900 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:15 smithi003 ceph-mon[100615]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:24:15.900 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:15 smithi003 ceph-mon[100615]: osdmap e179: 8 total, 8 up, 7 in 2024-08-17T17:24:15.900 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:15 smithi003 ceph-mon[103344]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:24:15.900 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:15 smithi003 ceph-mon[103344]: osdmap e179: 8 total, 8 up, 7 in 2024-08-17T17:24:15.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:15.931 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:15.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:15.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:15.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:15 smithi053 ceph-mon[89963]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:24:15.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:15 smithi053 ceph-mon[89963]: osdmap e179: 8 total, 8 up, 7 in 2024-08-17T17:24:16.033 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:16.060 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:16.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:16.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:16.162 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:16.186 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:16.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:16.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:16.289 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:16.311 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:16.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:16.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:16.413 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:16.436 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:16.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:16.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:16.538 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:16.565 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:16.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:16.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:16.667 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:16.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:16.694 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:16.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:16.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:16 smithi003 ceph-mon[103344]: osdmap e180: 8 total, 8 up, 7 in 2024-08-17T17:24:16.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:16 smithi003 ceph-mon[103344]: pgmap v442: 89 pgs: 5 creating+peering, 7 unknown, 77 active+clean; 4.6 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 0 B/s, 0 objects/s recovering 2024-08-17T17:24:16.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:16 smithi003 ceph-mon[100615]: osdmap e180: 8 total, 8 up, 7 in 2024-08-17T17:24:16.757 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:16 smithi003 ceph-mon[100615]: pgmap v442: 89 pgs: 5 creating+peering, 7 unknown, 77 active+clean; 4.6 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 0 B/s, 0 objects/s recovering 2024-08-17T17:24:16.795 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:16.821 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:16.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:16.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:16.923 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:16.951 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:16.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:16.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:16.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:16 smithi053 ceph-mon[89963]: osdmap e180: 8 total, 8 up, 7 in 2024-08-17T17:24:16.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:16 smithi053 ceph-mon[89963]: pgmap v442: 89 pgs: 5 creating+peering, 7 unknown, 77 active+clean; 4.6 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 0 B/s, 0 objects/s recovering 2024-08-17T17:24:17.053 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:17.079 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:17.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:17.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:17.181 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:17.207 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:17.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:17.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:17.309 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:17.335 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:17.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:17.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:17.436 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:17.462 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:17.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:17.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:17.564 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:17.588 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:17.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:17.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:17.690 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:17.713 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:17.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:17.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:17.815 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:17.837 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:17.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:17.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:17.939 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:17.950 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:24:17.966 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:17.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:17.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:17.987 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:24:18.068 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:18.094 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:18.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:18.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:18.195 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:24:18.196 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:18.221 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:18.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:18.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:18.323 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:18.347 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:18.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:18.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:18.448 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:18.475 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:18.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:18.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:18.576 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:18.598 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:18.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:18.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:18.700 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:18.723 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:18.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:18.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:18.818 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:24:18.824 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:18.851 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:18.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:18.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:18.952 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:18.979 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:18.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:18.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:19.081 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:19.108 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:19.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:19.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:19.210 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:19.233 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:19.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:19.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:19.321 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:19 smithi003 ceph-mon[103344]: pgmap v443: 89 pgs: 5 creating+peering, 2 unknown, 82 active+clean; 4.6 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:19.321 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:19 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:24:19.322 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:19 smithi003 ceph-mon[100615]: pgmap v443: 89 pgs: 5 creating+peering, 2 unknown, 82 active+clean; 4.6 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:19.322 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:19 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:24:19.334 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:19.357 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:19.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:19.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:19.459 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:19.482 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:19.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:19.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:19.585 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:19.607 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:19.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:19.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:19.709 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:19.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:19 smithi053 ceph-mon[89963]: pgmap v443: 89 pgs: 5 creating+peering, 2 unknown, 82 active+clean; 4.6 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:19.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:19 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:24:19.736 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:19.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:19.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:19.763 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:24:19.765 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:24:19.837 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:19.864 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:19.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:19.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:19.965 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:19.992 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:19.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:19.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:20.094 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:20.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:20.119 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:20.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:20.220 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:20.242 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:20.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:20.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:20.263 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.1 to 1 (10000) 2024-08-17T17:24:20.344 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:20.366 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:20.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:20.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:20.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:24:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:24:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:24:20.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:20 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3249565105' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:24:20.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:20 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:24:20.468 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:20.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:20.491 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:20.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:20.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:20 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3249565105' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:24:20.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:20 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:24:20.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:20 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3249565105' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:24:20.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:20 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:24:20.592 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:20.619 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:20.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:20.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:20.721 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:20.747 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:20.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:20.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:20.848 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:20.871 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:20.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:20.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:20.973 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:21.000 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:21.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:21.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:21.102 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:21.125 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:21.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:21.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:21.227 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:21.249 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:21.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:21.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:21.351 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:21.374 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:21.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:21.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:21.476 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:21.503 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:21.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:21.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:21.522 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:21 smithi003 ceph-mon[100615]: pgmap v444: 89 pgs: 89 active+clean; 4.6 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:21.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:21 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-17T17:24:21.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:21 smithi003 ceph-mon[100615]: osdmap e181: 8 total, 8 up, 7 in 2024-08-17T17:24:21.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:21 smithi003 ceph-mon[103344]: pgmap v444: 89 pgs: 89 active+clean; 4.6 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:21.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:21 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-17T17:24:21.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:21 smithi003 ceph-mon[103344]: osdmap e181: 8 total, 8 up, 7 in 2024-08-17T17:24:21.604 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:21.627 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:21.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:21.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:21.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:21 smithi053 ceph-mon[89963]: pgmap v444: 89 pgs: 89 active+clean; 4.6 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:21.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:21 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-17T17:24:21.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:21 smithi053 ceph-mon[89963]: osdmap e181: 8 total, 8 up, 7 in 2024-08-17T17:24:21.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:21.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:21.751 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:21.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:21.790 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 2 1 2024-08-17T17:24:21.851 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:21.878 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:21.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:21.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:21.980 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:22.007 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:22.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:22.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:22.109 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:22.135 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:22.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:22.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:22.237 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:22.261 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:22.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:22.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:22.363 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:22.390 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:22.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:22.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:22.492 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:22.518 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:22.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:22.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:22.620 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:22.631 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:24:22.631 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-08-17T17:24:22.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-08-17T17:24:22.631 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:24:22.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.631 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26082 5.1 5:8a74401c:::benchmark_data_smithi003_140793_object26081:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:24:22.631 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.358106+0000", 2024-08-17T17:24:22.631 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 300.27035246999998, 2024-08-17T17:24:22.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9241983769999997, 2024-08-17T17:24:22.631 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:24:22.631 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:24:22.631 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:22.632 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:24:22.632 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:24:22.632 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:22.632 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26082 2024-08-17T17:24:22.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.632 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:24:22.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:24:22.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.358106+0000", 2024-08-17T17:24:22.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:24:22.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.358106+0000", 2024-08-17T17:24:22.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:24:22.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.358107+0000", 2024-08-17T17:24:22.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4789999999999999e-06 2024-08-17T17:24:22.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:24:22.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.358131+0000", 2024-08-17T17:24:22.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4015e-05 2024-08-17T17:24:22.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:24:22.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.358143+0000", 2024-08-17T17:24:22.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.219e-05 2024-08-17T17:24:22.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:24:22.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.358151+0000", 2024-08-17T17:24:22.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2649999999999999e-06 2024-08-17T17:24:22.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.358226+0000", 2024-08-17T17:24:22.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5142999999999995e-05 2024-08-17T17:24:22.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.358238+0000", 2024-08-17T17:24:22.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2306e-05 2024-08-17T17:24:22.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.269985+0000", 2024-08-17T17:24:22.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.911747215 2024-08-17T17:24:22.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270001+0000", 2024-08-17T17:24:22.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5474000000000001e-05 2024-08-17T17:24:22.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.636 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.636 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.276689+0000", 2024-08-17T17:24:22.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0066882220000002 2024-08-17T17:24:22.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.636 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:24:22.636 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.277111+0000", 2024-08-17T17:24:22.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000421875 2024-08-17T17:24:22.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.636 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:24:22.636 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.277244+0000", 2024-08-17T17:24:22.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013283899999999999 2024-08-17T17:24:22.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:24:22.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.281296+0000", 2024-08-17T17:24:22.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040518719999999998 2024-08-17T17:24:22.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.281347+0000", 2024-08-17T17:24:22.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1427000000000001e-05 2024-08-17T17:24:22.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.281532+0000", 2024-08-17T17:24:22.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000185293 2024-08-17T17:24:22.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:24:22.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.281603+0000", 2024-08-17T17:24:22.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0518999999999993e-05 2024-08-17T17:24:22.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:24:22.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.282304+0000", 2024-08-17T17:24:22.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00070124300000000005 2024-08-17T17:24:22.639 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.639 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:24:22.639 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.639 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26098 5.1 5:89795322:::benchmark_data_smithi003_140793_object26097:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:24:22.639 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.365617+0000", 2024-08-17T17:24:22.639 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 300.262841364, 2024-08-17T17:24:22.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9216087019999999, 2024-08-17T17:24:22.639 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:24:22.640 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:24:22.640 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:22.640 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:24:22.640 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:24:22.640 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:22.640 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26098 2024-08-17T17:24:22.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.640 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:24:22.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:24:22.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365617+0000", 2024-08-17T17:24:22.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:24:22.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365617+0000", 2024-08-17T17:24:22.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:24:22.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365618+0000", 2024-08-17T17:24:22.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6249999999999999e-06 2024-08-17T17:24:22.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:24:22.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365655+0000", 2024-08-17T17:24:22.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6613999999999999e-05 2024-08-17T17:24:22.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:24:22.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365672+0000", 2024-08-17T17:24:22.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6844999999999999e-05 2024-08-17T17:24:22.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:24:22.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365679+0000", 2024-08-17T17:24:22.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4730000000000003e-06 2024-08-17T17:24:22.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366362+0000", 2024-08-17T17:24:22.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00068284800000000001 2024-08-17T17:24:22.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366374+0000", 2024-08-17T17:24:22.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1674e-05 2024-08-17T17:24:22.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270031+0000", 2024-08-17T17:24:22.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.90365675300000003 2024-08-17T17:24:22.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270039+0000", 2024-08-17T17:24:22.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1240000000000005e-06 2024-08-17T17:24:22.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.278729+0000", 2024-08-17T17:24:22.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0086905939999999 2024-08-17T17:24:22.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:24:22.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.278876+0000", 2024-08-17T17:24:22.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014623300000000001 2024-08-17T17:24:22.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:24:22.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.279011+0000", 2024-08-17T17:24:22.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013513899999999999 2024-08-17T17:24:22.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:24:22.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.281468+0000", 2024-08-17T17:24:22.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024572579999999999 2024-08-17T17:24:22.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.282380+0000", 2024-08-17T17:24:22.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00091197499999999998 2024-08-17T17:24:22.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.286511+0000", 2024-08-17T17:24:22.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0041307979999999998 2024-08-17T17:24:22.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:24:22.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.286586+0000", 2024-08-17T17:24:22.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5006000000000007e-05 2024-08-17T17:24:22.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:24:22.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.287226+0000", 2024-08-17T17:24:22.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00063974300000000002 2024-08-17T17:24:22.648 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.648 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:24:22.648 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.648 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26099 5.1 5:87830c36:::benchmark_data_smithi003_140793_object26098:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:24:22.648 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.365700+0000", 2024-08-17T17:24:22.648 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 300.262757948, 2024-08-17T17:24:22.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9222799620000002, 2024-08-17T17:24:22.648 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:24:22.648 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:24:22.648 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:22.648 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:24:22.649 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:24:22.649 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:22.649 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26099 2024-08-17T17:24:22.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.649 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:24:22.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:24:22.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365700+0000", 2024-08-17T17:24:22.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:24:22.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365700+0000", 2024-08-17T17:24:22.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:24:22.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365701+0000", 2024-08-17T17:24:22.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1799999999999995e-07 2024-08-17T17:24:22.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:24:22.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365722+0000", 2024-08-17T17:24:22.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0897999999999999e-05 2024-08-17T17:24:22.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:24:22.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365729+0000", 2024-08-17T17:24:22.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0539999999999997e-06 2024-08-17T17:24:22.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:24:22.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365734+0000", 2024-08-17T17:24:22.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9910000000000002e-06 2024-08-17T17:24:22.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366399+0000", 2024-08-17T17:24:22.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00066523799999999996 2024-08-17T17:24:22.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366406+0000", 2024-08-17T17:24:22.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3889999999999999e-06 2024-08-17T17:24:22.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270065+0000", 2024-08-17T17:24:22.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.90365837800000004 2024-08-17T17:24:22.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270072+0000", 2024-08-17T17:24:22.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5390000000000002e-06 2024-08-17T17:24:22.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.280026+0000", 2024-08-17T17:24:22.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.009954215 2024-08-17T17:24:22.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:24:22.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.280176+0000", 2024-08-17T17:24:22.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014965900000000001 2024-08-17T17:24:22.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:24:22.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.280355+0000", 2024-08-17T17:24:22.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000178586 2024-08-17T17:24:22.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.282543+0000", 2024-08-17T17:24:22.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0021886539999999999 2024-08-17T17:24:22.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:24:22.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.282560+0000", 2024-08-17T17:24:22.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6864e-05 2024-08-17T17:24:22.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.287385+0000", 2024-08-17T17:24:22.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0048250080000000004 2024-08-17T17:24:22.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:24:22.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.287456+0000", 2024-08-17T17:24:22.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0271000000000005e-05 2024-08-17T17:24:22.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:24:22.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.287980+0000", 2024-08-17T17:24:22.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00052470000000000001 2024-08-17T17:24:22.656 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.656 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:24:22.656 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.656 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26102 5.1 5:806d080d:::benchmark_data_smithi003_140793_object26101:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:24:22.656 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.366504+0000", 2024-08-17T17:24:22.656 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 300.26195416000002, 2024-08-17T17:24:22.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9223289210000001, 2024-08-17T17:24:22.656 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:24:22.656 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:24:22.657 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:22.657 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:24:22.657 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:24:22.657 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:22.657 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26102 2024-08-17T17:24:22.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.657 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:24:22.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:24:22.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366504+0000", 2024-08-17T17:24:22.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:24:22.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366504+0000", 2024-08-17T17:24:22.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:24:22.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366506+0000", 2024-08-17T17:24:22.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5149999999999999e-06 2024-08-17T17:24:22.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:24:22.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366531+0000", 2024-08-17T17:24:22.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5834e-05 2024-08-17T17:24:22.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:24:22.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366547+0000", 2024-08-17T17:24:22.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5235999999999999e-05 2024-08-17T17:24:22.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:24:22.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366554+0000", 2024-08-17T17:24:22.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2699999999999999e-06 2024-08-17T17:24:22.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366661+0000", 2024-08-17T17:24:22.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000107457 2024-08-17T17:24:22.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366671+0000", 2024-08-17T17:24:22.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3559999999999999e-06 2024-08-17T17:24:22.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.661 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270100+0000", 2024-08-17T17:24:22.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.90342893899999999 2024-08-17T17:24:22.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.661 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270108+0000", 2024-08-17T17:24:22.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6030000000000005e-06 2024-08-17T17:24:22.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.661 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.282611+0000", 2024-08-17T17:24:22.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.012502784 2024-08-17T17:24:22.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:24:22.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.282745+0000", 2024-08-17T17:24:22.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000133504 2024-08-17T17:24:22.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:24:22.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.282979+0000", 2024-08-17T17:24:22.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00023466900000000001 2024-08-17T17:24:22.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:24:22.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.287245+0000", 2024-08-17T17:24:22.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042655929999999998 2024-08-17T17:24:22.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.287308+0000", 2024-08-17T17:24:22.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2869999999999999e-05 2024-08-17T17:24:22.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.288280+0000", 2024-08-17T17:24:22.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00097271 2024-08-17T17:24:22.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:24:22.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.288377+0000", 2024-08-17T17:24:22.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6137000000000005e-05 2024-08-17T17:24:22.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:24:22.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.288833+0000", 2024-08-17T17:24:22.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00045644400000000002 2024-08-17T17:24:22.664 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.664 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:24:22.664 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.665 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26120 5.1 5:8459086c:::benchmark_data_smithi003_140793_object26119:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:24:22.665 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.375347+0000", 2024-08-17T17:24:22.665 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 300.25311116500001, 2024-08-17T17:24:22.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9140657679999999, 2024-08-17T17:24:22.665 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:24:22.665 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:24:22.665 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:22.665 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:24:22.665 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:24:22.665 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:22.665 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26120 2024-08-17T17:24:22.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.665 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:24:22.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:24:22.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375347+0000", 2024-08-17T17:24:22.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:24:22.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375347+0000", 2024-08-17T17:24:22.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:24:22.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375348+0000", 2024-08-17T17:24:22.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1939999999999999e-06 2024-08-17T17:24:22.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:24:22.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375371+0000", 2024-08-17T17:24:22.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2997e-05 2024-08-17T17:24:22.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:24:22.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375445+0000", 2024-08-17T17:24:22.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3657000000000003e-05 2024-08-17T17:24:22.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:24:22.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375456+0000", 2024-08-17T17:24:22.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0941000000000001e-05 2024-08-17T17:24:22.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375518+0000", 2024-08-17T17:24:22.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2533999999999995e-05 2024-08-17T17:24:22.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375531+0000", 2024-08-17T17:24:22.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.238e-05 2024-08-17T17:24:22.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270132+0000", 2024-08-17T17:24:22.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89460134800000002 2024-08-17T17:24:22.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270140+0000", 2024-08-17T17:24:22.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0299999999999994e-06 2024-08-17T17:24:22.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.284149+0000", 2024-08-17T17:24:22.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0140088189999998 2024-08-17T17:24:22.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:24:22.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.284323+0000", 2024-08-17T17:24:22.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017453600000000001 2024-08-17T17:24:22.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:24:22.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.284507+0000", 2024-08-17T17:24:22.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000183692 2024-08-17T17:24:22.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:24:22.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.288124+0000", 2024-08-17T17:24:22.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0036170659999999999 2024-08-17T17:24:22.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.288193+0000", 2024-08-17T17:24:22.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9238000000000004e-05 2024-08-17T17:24:22.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.288892+0000", 2024-08-17T17:24:22.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00069895500000000002 2024-08-17T17:24:22.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.672 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:24:22.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.288930+0000", 2024-08-17T17:24:22.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7125000000000001e-05 2024-08-17T17:24:22.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:24:22.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.289413+0000", 2024-08-17T17:24:22.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000483256 2024-08-17T17:24:22.673 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.673 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:24:22.673 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.673 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26121 5.1 5:8d0269c6:::benchmark_data_smithi003_140793_object26120:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:24:22.673 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.375480+0000", 2024-08-17T17:24:22.673 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 300.25297797000002, 2024-08-17T17:24:22.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.921705099, 2024-08-17T17:24:22.673 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:24:22.674 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:24:22.674 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:22.674 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:24:22.674 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:24:22.674 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:22.674 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26121 2024-08-17T17:24:22.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.674 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:24:22.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:24:22.674 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375480+0000", 2024-08-17T17:24:22.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:24:22.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375480+0000", 2024-08-17T17:24:22.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:24:22.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375481+0000", 2024-08-17T17:24:22.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.44e-07 2024-08-17T17:24:22.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:24:22.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375502+0000", 2024-08-17T17:24:22.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.086e-05 2024-08-17T17:24:22.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:24:22.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375511+0000", 2024-08-17T17:24:22.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5429999999999999e-06 2024-08-17T17:24:22.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:24:22.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375518+0000", 2024-08-17T17:24:22.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2949999999999999e-06 2024-08-17T17:24:22.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.677 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.677 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375557+0000", 2024-08-17T17:24:22.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9255999999999998e-05 2024-08-17T17:24:22.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.677 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.677 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375565+0000", 2024-08-17T17:24:22.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1289999999999996e-06 2024-08-17T17:24:22.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.677 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.677 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270164+0000", 2024-08-17T17:24:22.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89459907800000005 2024-08-17T17:24:22.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270172+0000", 2024-08-17T17:24:22.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8869999999999992e-06 2024-08-17T17:24:22.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.289461+0000", 2024-08-17T17:24:22.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0192893380000001 2024-08-17T17:24:22.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:24:22.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.289616+0000", 2024-08-17T17:24:22.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015440199999999999 2024-08-17T17:24:22.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:24:22.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.289854+0000", 2024-08-17T17:24:22.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000238271 2024-08-17T17:24:22.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.296410+0000", 2024-08-17T17:24:22.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006555735 2024-08-17T17:24:22.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:24:22.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.296424+0000", 2024-08-17T17:24:22.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4679e-05 2024-08-17T17:24:22.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.296471+0000", 2024-08-17T17:24:22.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6600000000000001e-05 2024-08-17T17:24:22.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:24:22.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.296537+0000", 2024-08-17T17:24:22.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5733000000000002e-05 2024-08-17T17:24:22.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:24:22.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.297185+0000", 2024-08-17T17:24:22.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00064864899999999995 2024-08-17T17:24:22.681 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.681 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:24:22.681 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.681 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26129 5.1 5:845981d1:::benchmark_data_smithi003_140793_object26128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:24:22.681 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.378910+0000", 2024-08-17T17:24:22.681 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 300.24954801600001, 2024-08-17T17:24:22.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9327440490000001, 2024-08-17T17:24:22.682 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:24:22.682 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:24:22.682 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:22.682 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:24:22.682 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:24:22.682 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:22.682 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26129 2024-08-17T17:24:22.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.682 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:24:22.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:24:22.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.378910+0000", 2024-08-17T17:24:22.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:24:22.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.378910+0000", 2024-08-17T17:24:22.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:24:22.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.378912+0000", 2024-08-17T17:24:22.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.657e-06 2024-08-17T17:24:22.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:24:22.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.378935+0000", 2024-08-17T17:24:22.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2795000000000001e-05 2024-08-17T17:24:22.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:24:22.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.378956+0000", 2024-08-17T17:24:22.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1744000000000001e-05 2024-08-17T17:24:22.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:24:22.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.378965+0000", 2024-08-17T17:24:22.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2930000000000001e-06 2024-08-17T17:24:22.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.379129+0000", 2024-08-17T17:24:22.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016463799999999999 2024-08-17T17:24:22.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.379143+0000", 2024-08-17T17:24:22.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.341e-05 2024-08-17T17:24:22.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270196+0000", 2024-08-17T17:24:22.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89105351399999999 2024-08-17T17:24:22.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270204+0000", 2024-08-17T17:24:22.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8159999999999994e-06 2024-08-17T17:24:22.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.294939+0000", 2024-08-17T17:24:22.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.024735352 2024-08-17T17:24:22.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:24:22.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.295108+0000", 2024-08-17T17:24:22.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016822500000000001 2024-08-17T17:24:22.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:24:22.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.295294+0000", 2024-08-17T17:24:22.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000186107 2024-08-17T17:24:22.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.301319+0000", 2024-08-17T17:24:22.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0060248339999999997 2024-08-17T17:24:22.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:24:22.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.304206+0000", 2024-08-17T17:24:22.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028874009999999999 2024-08-17T17:24:22.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.310650+0000", 2024-08-17T17:24:22.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0064438689999999996 2024-08-17T17:24:22.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:24:22.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.310709+0000", 2024-08-17T17:24:22.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9494000000000003e-05 2024-08-17T17:24:22.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:24:22.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.311654+0000", 2024-08-17T17:24:22.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00094490000000000004 2024-08-17T17:24:22.689 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.689 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:24:22.689 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:22.690 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:22.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:22.692 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26134 5.1 5:8a41a77b:::benchmark_data_smithi003_140793_object26133:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:24:22.692 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.380465+0000", 2024-08-17T17:24:22.692 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 300.24799301600001, 2024-08-17T17:24:22.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9324733580000002, 2024-08-17T17:24:22.692 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:24:22.692 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:24:22.692 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:22.692 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:24:22.693 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:24:22.693 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:22.693 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26134 2024-08-17T17:24:22.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.693 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:24:22.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:24:22.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.380465+0000", 2024-08-17T17:24:22.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:24:22.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.380465+0000", 2024-08-17T17:24:22.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:24:22.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.380466+0000", 2024-08-17T17:24:22.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.28e-06 2024-08-17T17:24:22.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:24:22.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.380490+0000", 2024-08-17T17:24:22.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3842999999999999e-05 2024-08-17T17:24:22.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:24:22.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.380506+0000", 2024-08-17T17:24:22.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6065e-05 2024-08-17T17:24:22.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:24:22.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.380514+0000", 2024-08-17T17:24:22.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8529999999999992e-06 2024-08-17T17:24:22.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.380591+0000", 2024-08-17T17:24:22.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6271000000000002e-05 2024-08-17T17:24:22.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.380603+0000", 2024-08-17T17:24:22.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2867999999999999e-05 2024-08-17T17:24:22.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270227+0000", 2024-08-17T17:24:22.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.88962385499999996 2024-08-17T17:24:22.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270235+0000", 2024-08-17T17:24:22.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6120000000000001e-06 2024-08-17T17:24:22.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.697 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.297325+0000", 2024-08-17T17:24:22.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0270902529999999 2024-08-17T17:24:22.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:24:22.697 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.297474+0000", 2024-08-17T17:24:22.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014895899999999999 2024-08-17T17:24:22.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:24:22.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.297667+0000", 2024-08-17T17:24:22.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019309500000000001 2024-08-17T17:24:22.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.304283+0000", 2024-08-17T17:24:22.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066158759999999997 2024-08-17T17:24:22.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:24:22.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.310600+0000", 2024-08-17T17:24:22.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063167420000000002 2024-08-17T17:24:22.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.312211+0000", 2024-08-17T17:24:22.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016113029999999999 2024-08-17T17:24:22.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:24:22.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.312376+0000", 2024-08-17T17:24:22.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000164804 2024-08-17T17:24:22.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:24:22.700 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.312939+0000", 2024-08-17T17:24:22.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00056267899999999996 2024-08-17T17:24:22.700 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.700 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:24:22.700 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.700 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26136 5.1 5:81387a84:::benchmark_data_smithi003_140793_object26135:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:24:22.700 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.382929+0000", 2024-08-17T17:24:22.700 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 300.24552909300002, 2024-08-17T17:24:22.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.930695842, 2024-08-17T17:24:22.700 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:24:22.700 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:24:22.700 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:22.701 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:24:22.701 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:24:22.701 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:22.701 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26136 2024-08-17T17:24:22.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.701 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:24:22.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.701 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:24:22.701 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.382929+0000", 2024-08-17T17:24:22.701 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.701 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:24:22.701 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.382929+0000", 2024-08-17T17:24:22.702 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:24:22.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.382931+0000", 2024-08-17T17:24:22.702 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5120000000000001e-06 2024-08-17T17:24:22.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:24:22.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.382982+0000", 2024-08-17T17:24:22.702 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1314999999999999e-05 2024-08-17T17:24:22.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.703 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:24:22.703 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.382998+0000", 2024-08-17T17:24:22.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6235999999999998e-05 2024-08-17T17:24:22.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.703 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:24:22.703 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.383006+0000", 2024-08-17T17:24:22.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4760000000000001e-06 2024-08-17T17:24:22.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.703 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.703 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.383064+0000", 2024-08-17T17:24:22.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8255999999999999e-05 2024-08-17T17:24:22.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.704 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.704 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.383073+0000", 2024-08-17T17:24:22.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5650000000000007e-06 2024-08-17T17:24:22.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.704 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.704 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270259+0000", 2024-08-17T17:24:22.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.88718503699999995 2024-08-17T17:24:22.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.705 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.705 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270266+0000", 2024-08-17T17:24:22.705 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.892e-06 2024-08-17T17:24:22.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.705 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.705 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.298758+0000", 2024-08-17T17:24:22.705 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0284916499999999 2024-08-17T17:24:22.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.705 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:24:22.705 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.298905+0000", 2024-08-17T17:24:22.705 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014658 2024-08-17T17:24:22.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.706 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:24:22.706 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.299067+0000", 2024-08-17T17:24:22.706 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016263700000000001 2024-08-17T17:24:22.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.706 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.706 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.304334+0000", 2024-08-17T17:24:22.706 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0052671059999999997 2024-08-17T17:24:22.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.706 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:24:22.707 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.310606+0000", 2024-08-17T17:24:22.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0062720830000000003 2024-08-17T17:24:22.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.707 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.313094+0000", 2024-08-17T17:24:22.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002487763 2024-08-17T17:24:22.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:24:22.707 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.313146+0000", 2024-08-17T17:24:22.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2079000000000001e-05 2024-08-17T17:24:22.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.708 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.708 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:24:22.708 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.313625+0000", 2024-08-17T17:24:22.708 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00047865499999999998 2024-08-17T17:24:22.708 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.708 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:24:22.708 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.708 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.708 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.708 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26145 5.1 5:896bb16f:::benchmark_data_smithi003_140793_object26144:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:24:22.708 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.386726+0000", 2024-08-17T17:24:22.708 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 300.24173210599997, 2024-08-17T17:24:22.708 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9275205729999998, 2024-08-17T17:24:22.709 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:24:22.709 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:24:22.709 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:22.709 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:24:22.709 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:24:22.709 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:22.709 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26145 2024-08-17T17:24:22.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.709 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:24:22.709 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.709 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:24:22.709 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.386726+0000", 2024-08-17T17:24:22.709 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.710 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.710 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:24:22.710 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.386726+0000", 2024-08-17T17:24:22.710 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.710 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.710 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:24:22.710 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.386727+0000", 2024-08-17T17:24:22.710 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.291e-06 2024-08-17T17:24:22.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.710 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.711 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:24:22.711 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.386775+0000", 2024-08-17T17:24:22.711 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7855999999999997e-05 2024-08-17T17:24:22.711 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.711 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.711 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:24:22.711 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.386791+0000", 2024-08-17T17:24:22.711 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6044000000000001e-05 2024-08-17T17:24:22.711 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.711 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.711 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:24:22.711 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.386799+0000", 2024-08-17T17:24:22.711 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7510000000000008e-06 2024-08-17T17:24:22.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.712 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.712 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.386873+0000", 2024-08-17T17:24:22.712 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4208000000000005e-05 2024-08-17T17:24:22.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.712 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.712 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.386887+0000", 2024-08-17T17:24:22.712 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3699e-05 2024-08-17T17:24:22.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.712 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.713 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270290+0000", 2024-08-17T17:24:22.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.88340287500000003 2024-08-17T17:24:22.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.713 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.713 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270298+0000", 2024-08-17T17:24:22.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0399999999999993e-06 2024-08-17T17:24:22.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.713 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.713 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.299986+0000", 2024-08-17T17:24:22.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0296885850000002 2024-08-17T17:24:22.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.714 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.714 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:24:22.714 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.300132+0000", 2024-08-17T17:24:22.714 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014511 2024-08-17T17:24:22.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.714 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.714 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:24:22.714 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.300298+0000", 2024-08-17T17:24:22.714 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016594799999999999 2024-08-17T17:24:22.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.714 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.714 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.714 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.304384+0000", 2024-08-17T17:24:22.715 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040865720000000001 2024-08-17T17:24:22.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.715 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:24:22.715 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.311674+0000", 2024-08-17T17:24:22.715 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072902210000000004 2024-08-17T17:24:22.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.715 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.715 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.313682+0000", 2024-08-17T17:24:22.715 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00200746 2024-08-17T17:24:22.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.716 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:24:22.716 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.313742+0000", 2024-08-17T17:24:22.716 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0265e-05 2024-08-17T17:24:22.716 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.716 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.716 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:24:22.716 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.314247+0000", 2024-08-17T17:24:22.716 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00050464800000000001 2024-08-17T17:24:22.716 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.716 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:24:22.716 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.716 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.716 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.717 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26147 5.1 5:85f2e61e:::benchmark_data_smithi003_140793_object26146:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:24:22.717 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.387915+0000", 2024-08-17T17:24:22.717 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 300.24054361899999, 2024-08-17T17:24:22.717 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9269894829999998, 2024-08-17T17:24:22.717 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:24:22.717 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:24:22.717 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:22.717 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:24:22.717 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:24:22.717 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:22.717 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26147 2024-08-17T17:24:22.717 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.717 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:24:22.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.718 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:24:22.718 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.387915+0000", 2024-08-17T17:24:22.718 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.718 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:24:22.718 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.387915+0000", 2024-08-17T17:24:22.718 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.718 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:24:22.719 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.387915+0000", 2024-08-17T17:24:22.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6899999999999996e-07 2024-08-17T17:24:22.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.719 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:24:22.719 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.387953+0000", 2024-08-17T17:24:22.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7447000000000002e-05 2024-08-17T17:24:22.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.719 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:24:22.719 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.387963+0000", 2024-08-17T17:24:22.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0302999999999999e-05 2024-08-17T17:24:22.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.720 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.720 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:24:22.720 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.387969+0000", 2024-08-17T17:24:22.720 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1090000000000003e-06 2024-08-17T17:24:22.720 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.720 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.720 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.720 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.388038+0000", 2024-08-17T17:24:22.720 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9042999999999996e-05 2024-08-17T17:24:22.720 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.720 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.720 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.720 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.388051+0000", 2024-08-17T17:24:22.720 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2498e-05 2024-08-17T17:24:22.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270321+0000", 2024-08-17T17:24:22.721 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.88227023299999996 2024-08-17T17:24:22.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270329+0000", 2024-08-17T17:24:22.721 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9240000000000007e-06 2024-08-17T17:24:22.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.722 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.301403+0000", 2024-08-17T17:24:22.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0310743539999998 2024-08-17T17:24:22.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.722 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:24:22.722 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.301580+0000", 2024-08-17T17:24:22.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017625900000000001 2024-08-17T17:24:22.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.722 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:24:22.722 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.301745+0000", 2024-08-17T17:24:22.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000165728 2024-08-17T17:24:22.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.723 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.723 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.307080+0000", 2024-08-17T17:24:22.723 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0053345570000000002 2024-08-17T17:24:22.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.723 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:24:22.723 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.311680+0000", 2024-08-17T17:24:22.723 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0046006270000000004 2024-08-17T17:24:22.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.723 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.723 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.314305+0000", 2024-08-17T17:24:22.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026242890000000001 2024-08-17T17:24:22.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.724 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:24:22.724 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.314364+0000", 2024-08-17T17:24:22.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8853e-05 2024-08-17T17:24:22.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.724 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:24:22.724 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.314904+0000", 2024-08-17T17:24:22.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00054049000000000002 2024-08-17T17:24:22.724 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.724 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:24:22.724 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.725 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26154 5.1 5:8b1cb6d6:::benchmark_data_smithi003_140793_object26153:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:24:22.725 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.390694+0000", 2024-08-17T17:24:22.725 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 300.237764427, 2024-08-17T17:24:22.725 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9249220940000002, 2024-08-17T17:24:22.725 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:24:22.725 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:24:22.725 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:22.725 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:24:22.725 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:24:22.725 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:22.725 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26154 2024-08-17T17:24:22.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.725 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:24:22.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.726 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:24:22.726 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.390694+0000", 2024-08-17T17:24:22.726 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.726 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.726 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:24:22.726 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.390694+0000", 2024-08-17T17:24:22.726 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.726 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.726 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:24:22.726 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.390695+0000", 2024-08-17T17:24:22.726 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3349999999999999e-06 2024-08-17T17:24:22.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.727 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.727 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:24:22.727 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.390716+0000", 2024-08-17T17:24:22.727 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0854999999999999e-05 2024-08-17T17:24:22.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.727 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.727 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:24:22.727 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.390728+0000", 2024-08-17T17:24:22.727 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2455e-05 2024-08-17T17:24:22.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.727 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.727 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:24:22.728 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.390736+0000", 2024-08-17T17:24:22.728 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0509999999999997e-06 2024-08-17T17:24:22.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.728 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.728 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.728 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.390824+0000", 2024-08-17T17:24:22.728 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7665999999999997e-05 2024-08-17T17:24:22.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.728 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.728 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.728 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.390835+0000", 2024-08-17T17:24:22.728 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0987e-05 2024-08-17T17:24:22.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.728 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270353+0000", 2024-08-17T17:24:22.729 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.879518146 2024-08-17T17:24:22.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270361+0000", 2024-08-17T17:24:22.729 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8779999999999998e-06 2024-08-17T17:24:22.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.302895+0000", 2024-08-17T17:24:22.729 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0325342339999999 2024-08-17T17:24:22.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.730 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.730 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:24:22.730 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.303046+0000", 2024-08-17T17:24:22.730 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015052899999999999 2024-08-17T17:24:22.730 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.730 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.730 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:24:22.730 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.303235+0000", 2024-08-17T17:24:22.730 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018953700000000001 2024-08-17T17:24:22.730 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.730 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.730 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.731 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.307121+0000", 2024-08-17T17:24:22.731 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038852779999999998 2024-08-17T17:24:22.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.731 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.731 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:24:22.731 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.311686+0000", 2024-08-17T17:24:22.731 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0045651889999999999 2024-08-17T17:24:22.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.731 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.731 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.731 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.315019+0000", 2024-08-17T17:24:22.731 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0033328609999999999 2024-08-17T17:24:22.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.731 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.732 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:24:22.732 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.315081+0000", 2024-08-17T17:24:22.732 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.198e-05 2024-08-17T17:24:22.732 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.732 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.732 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:24:22.732 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.315616+0000", 2024-08-17T17:24:22.732 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00053511300000000004 2024-08-17T17:24:22.732 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.732 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:24:22.732 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.732 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.732 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.733 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26175 5.1 5:8c6f3adb:::benchmark_data_smithi003_140793_object26174:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:24:22.733 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.403515+0000", 2024-08-17T17:24:22.733 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 300.22494335599998, 2024-08-17T17:24:22.733 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9127047070000001, 2024-08-17T17:24:22.733 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:24:22.733 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:24:22.733 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:22.733 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:24:22.733 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:24:22.733 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:22.733 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26175 2024-08-17T17:24:22.733 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.734 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:24:22.734 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.734 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:24:22.734 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.403515+0000", 2024-08-17T17:24:22.734 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.734 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.734 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.734 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:24:22.734 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.403515+0000", 2024-08-17T17:24:22.734 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.734 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.734 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.734 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:24:22.735 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.403517+0000", 2024-08-17T17:24:22.735 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7910000000000001e-06 2024-08-17T17:24:22.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.735 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.735 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:24:22.735 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.403543+0000", 2024-08-17T17:24:22.735 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6806999999999999e-05 2024-08-17T17:24:22.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.735 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.735 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:24:22.735 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.403558+0000", 2024-08-17T17:24:22.735 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5034999999999999e-05 2024-08-17T17:24:22.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.736 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:24:22.736 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.403567+0000", 2024-08-17T17:24:22.736 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8789999999999992e-06 2024-08-17T17:24:22.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.736 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.736 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.403634+0000", 2024-08-17T17:24:22.736 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6321000000000004e-05 2024-08-17T17:24:22.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.736 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.736 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.403646+0000", 2024-08-17T17:24:22.737 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2628e-05 2024-08-17T17:24:22.737 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.737 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.737 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.737 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270384+0000", 2024-08-17T17:24:22.737 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.86673805400000004 2024-08-17T17:24:22.737 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.737 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.737 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.737 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270392+0000", 2024-08-17T17:24:22.737 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8380000000000002e-06 2024-08-17T17:24:22.737 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.737 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.738 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.738 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.304419+0000", 2024-08-17T17:24:22.738 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0340264549999998 2024-08-17T17:24:22.738 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.738 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.738 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:24:22.738 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.304562+0000", 2024-08-17T17:24:22.738 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014378500000000001 2024-08-17T17:24:22.738 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.738 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.738 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:24:22.738 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.304715+0000", 2024-08-17T17:24:22.739 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015284700000000001 2024-08-17T17:24:22.739 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.739 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.739 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.739 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.310579+0000", 2024-08-17T17:24:22.739 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0058634339999999998 2024-08-17T17:24:22.739 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.739 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.739 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:24:22.739 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.311692+0000", 2024-08-17T17:24:22.739 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001112912 2024-08-17T17:24:22.739 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.739 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.739 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.740 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.315676+0000", 2024-08-17T17:24:22.740 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039842649999999999 2024-08-17T17:24:22.740 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.740 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.740 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:24:22.740 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.315747+0000", 2024-08-17T17:24:22.740 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0735000000000006e-05 2024-08-17T17:24:22.740 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.740 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.740 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:24:22.740 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.316220+0000", 2024-08-17T17:24:22.740 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00047292100000000001 2024-08-17T17:24:22.740 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.740 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:24:22.741 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.741 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.741 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.741 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26336 5.1 5:837f3a0a:::benchmark_data_smithi003_140793_object26335:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:24:22.741 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.566135+0000", 2024-08-17T17:24:22.741 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 300.062322843, 2024-08-17T17:24:22.741 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7524043960000002, 2024-08-17T17:24:22.741 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:24:22.741 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:24:22.741 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:22.741 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:24:22.741 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:24:22.741 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:22.742 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26336 2024-08-17T17:24:22.742 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.742 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:24:22.742 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.742 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:24:22.742 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.566135+0000", 2024-08-17T17:24:22.742 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.742 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.742 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.742 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:24:22.742 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.566135+0000", 2024-08-17T17:24:22.742 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.742 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.742 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.743 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:24:22.743 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.566137+0000", 2024-08-17T17:24:22.743 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.477e-06 2024-08-17T17:24:22.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.743 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.743 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:24:22.743 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.566159+0000", 2024-08-17T17:24:22.743 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1691e-05 2024-08-17T17:24:22.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.743 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.743 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:24:22.743 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.566175+0000", 2024-08-17T17:24:22.743 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6249999999999999e-05 2024-08-17T17:24:22.744 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.744 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.744 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:24:22.744 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.566184+0000", 2024-08-17T17:24:22.744 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0299999999999999e-06 2024-08-17T17:24:22.744 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.744 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.744 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.744 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.566258+0000", 2024-08-17T17:24:22.744 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3739999999999995e-05 2024-08-17T17:24:22.744 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.744 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.744 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.744 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.566270+0000", 2024-08-17T17:24:22.745 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2931e-05 2024-08-17T17:24:22.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.745 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.745 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270415+0000", 2024-08-17T17:24:22.745 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.70414461900000003 2024-08-17T17:24:22.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.745 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.745 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270423+0000", 2024-08-17T17:24:22.745 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4329999999999998e-06 2024-08-17T17:24:22.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.746 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.746 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.305724+0000", 2024-08-17T17:24:22.746 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.03530159 2024-08-17T17:24:22.746 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.746 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.746 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:24:22.746 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.305884+0000", 2024-08-17T17:24:22.746 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000159658 2024-08-17T17:24:22.746 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.746 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.746 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:24:22.746 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.306079+0000", 2024-08-17T17:24:22.746 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019524700000000001 2024-08-17T17:24:22.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.747 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:24:22.747 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.311698+0000", 2024-08-17T17:24:22.747 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0056185810000000001 2024-08-17T17:24:22.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.747 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.747 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.312166+0000", 2024-08-17T17:24:22.747 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000468314 2024-08-17T17:24:22.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.747 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.748 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.317902+0000", 2024-08-17T17:24:22.748 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0057355949999999996 2024-08-17T17:24:22.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.748 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:24:22.748 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.317959+0000", 2024-08-17T17:24:22.748 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7932e-05 2024-08-17T17:24:22.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.748 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:24:22.748 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.318540+0000", 2024-08-17T17:24:22.748 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00058030799999999997 2024-08-17T17:24:22.748 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.749 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:24:22.749 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.749 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26347 5.1 5:83125685:::benchmark_data_smithi003_140793_object26346:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:24:22.749 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.575398+0000", 2024-08-17T17:24:22.749 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 300.05305994999998, 2024-08-17T17:24:22.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7503322890000002, 2024-08-17T17:24:22.749 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:24:22.749 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:24:22.749 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:22.749 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:24:22.749 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:24:22.749 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:22.750 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26347 2024-08-17T17:24:22.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.750 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:24:22.750 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.750 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:24:22.750 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.575398+0000", 2024-08-17T17:24:22.750 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.750 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.750 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:24:22.750 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.575398+0000", 2024-08-17T17:24:22.750 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.751 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:24:22.751 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.575400+0000", 2024-08-17T17:24:22.751 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.683e-06 2024-08-17T17:24:22.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.751 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:24:22.751 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.575430+0000", 2024-08-17T17:24:22.751 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9808e-05 2024-08-17T17:24:22.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.751 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:24:22.751 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.575447+0000", 2024-08-17T17:24:22.751 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7257000000000001e-05 2024-08-17T17:24:22.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.752 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:24:22.752 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.575456+0000", 2024-08-17T17:24:22.752 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8230000000000001e-06 2024-08-17T17:24:22.752 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.752 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.752 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.575520+0000", 2024-08-17T17:24:22.752 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4515000000000001e-05 2024-08-17T17:24:22.752 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.752 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.752 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.575529+0000", 2024-08-17T17:24:22.753 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3119999999999994e-06 2024-08-17T17:24:22.753 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.753 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.753 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.753 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270446+0000", 2024-08-17T17:24:22.753 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.69491715499999995 2024-08-17T17:24:22.753 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.753 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.753 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.753 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270454+0000", 2024-08-17T17:24:22.753 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8099999999999998e-06 2024-08-17T17:24:22.753 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.753 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.754 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.754 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.307156+0000", 2024-08-17T17:24:22.754 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0367026579999998 2024-08-17T17:24:22.754 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.754 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.754 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:24:22.754 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.307253+0000", 2024-08-17T17:24:22.754 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6832999999999999e-05 2024-08-17T17:24:22.754 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.754 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.754 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:24:22.754 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.307393+0000", 2024-08-17T17:24:22.754 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014028399999999999 2024-08-17T17:24:22.755 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.755 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.755 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:24:22.755 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.311703+0000", 2024-08-17T17:24:22.755 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004309729 2024-08-17T17:24:22.755 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.755 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.755 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.755 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.312995+0000", 2024-08-17T17:24:22.755 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0012918820000000001 2024-08-17T17:24:22.755 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.755 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.755 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.756 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.324907+0000", 2024-08-17T17:24:22.756 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011912114999999999 2024-08-17T17:24:22.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.756 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:24:22.756 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.325027+0000", 2024-08-17T17:24:22.756 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000119506 2024-08-17T17:24:22.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.756 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:24:22.756 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.325731+0000", 2024-08-17T17:24:22.756 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00070391900000000001 2024-08-17T17:24:22.756 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.756 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:24:22.757 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.757 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26360 5.1 5:89f4cb1c:::benchmark_data_smithi003_140793_object26359:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:24:22.757 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.594844+0000", 2024-08-17T17:24:22.757 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 300.03361459199999, 2024-08-17T17:24:22.757 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7437470209999999, 2024-08-17T17:24:22.757 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:24:22.757 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:24:22.757 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:22.757 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:24:22.757 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:24:22.757 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:22.757 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26360 2024-08-17T17:24:22.758 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.758 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:24:22.758 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.758 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:24:22.758 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.594844+0000", 2024-08-17T17:24:22.758 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.758 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.758 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.758 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:24:22.758 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.594844+0000", 2024-08-17T17:24:22.758 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.758 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.758 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.759 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:24:22.759 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.594845+0000", 2024-08-17T17:24:22.759 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2929999999999999e-06 2024-08-17T17:24:22.759 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.759 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.759 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:24:22.759 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.594861+0000", 2024-08-17T17:24:22.759 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6368e-05 2024-08-17T17:24:22.759 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.759 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.759 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:24:22.759 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.594872+0000", 2024-08-17T17:24:22.759 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1079e-05 2024-08-17T17:24:22.760 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.760 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.760 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:24:22.760 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.594878+0000", 2024-08-17T17:24:22.760 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5629999999999999e-06 2024-08-17T17:24:22.760 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.760 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.760 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.760 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.594926+0000", 2024-08-17T17:24:22.760 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8498000000000002e-05 2024-08-17T17:24:22.760 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.760 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.760 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.760 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.594934+0000", 2024-08-17T17:24:22.761 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7999999999999999e-06 2024-08-17T17:24:22.761 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.761 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.761 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270477+0000", 2024-08-17T17:24:22.761 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.67554314800000004 2024-08-17T17:24:22.761 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.761 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.761 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270486+0000", 2024-08-17T17:24:22.761 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5650000000000002e-06 2024-08-17T17:24:22.761 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.762 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.762 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.316324+0000", 2024-08-17T17:24:22.762 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0458376779999998 2024-08-17T17:24:22.762 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.762 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.762 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:24:22.762 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.316469+0000", 2024-08-17T17:24:22.762 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014542299999999999 2024-08-17T17:24:22.762 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.762 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.762 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:24:22.762 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.316621+0000", 2024-08-17T17:24:22.762 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015184700000000001 2024-08-17T17:24:22.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.763 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.763 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.763 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.320092+0000", 2024-08-17T17:24:22.763 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034713589999999998 2024-08-17T17:24:22.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.763 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.763 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:24:22.763 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.320106+0000", 2024-08-17T17:24:22.763 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3382e-05 2024-08-17T17:24:22.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.763 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.763 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.764 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.337839+0000", 2024-08-17T17:24:22.764 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017733498 2024-08-17T17:24:22.764 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.764 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.764 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:24:22.764 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.337900+0000", 2024-08-17T17:24:22.764 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1266999999999995e-05 2024-08-17T17:24:22.764 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.764 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.764 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:24:22.764 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.338591+0000", 2024-08-17T17:24:22.764 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00069025300000000005 2024-08-17T17:24:22.764 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.765 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:24:22.765 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.765 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.765 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:57339 5.1 5:8dd4b2b4:::benchmark_data_smithi003_140793_object57338:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-08-17T17:24:22.765 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:46.499070+0000", 2024-08-17T17:24:22.765 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 276.12938803100002, 2024-08-17T17:24:22.765 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96002446600000002, 2024-08-17T17:24:22.765 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:24:22.765 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:24:22.765 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:22.765 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:24:22.765 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:24:22.765 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:22.766 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 57339 2024-08-17T17:24:22.766 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.766 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:24:22.766 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.766 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:24:22.766 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.499070+0000", 2024-08-17T17:24:22.766 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.766 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.766 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.766 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:24:22.766 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.499070+0000", 2024-08-17T17:24:22.766 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.766 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.767 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.767 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:24:22.767 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.499072+0000", 2024-08-17T17:24:22.767 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9860000000000001e-06 2024-08-17T17:24:22.767 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.769 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:24:22.769 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.499107+0000", 2024-08-17T17:24:22.769 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4891999999999997e-05 2024-08-17T17:24:22.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.770 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.770 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:24:22.770 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.499131+0000", 2024-08-17T17:24:22.770 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4409e-05 2024-08-17T17:24:22.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.770 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.770 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:24:22.770 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.499140+0000", 2024-08-17T17:24:22.770 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4999999999999999e-06 2024-08-17T17:24:22.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.770 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.771 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.771 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.499316+0000", 2024-08-17T17:24:22.771 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000175549 2024-08-17T17:24:22.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.771 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.771 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.499326+0000", 2024-08-17T17:24:22.771 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0834e-05 2024-08-17T17:24:22.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.771 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.771 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.445610+0000", 2024-08-17T17:24:22.771 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.94628382600000005 2024-08-17T17:24:22.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.772 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.772 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:24:22.772 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.445734+0000", 2024-08-17T17:24:22.772 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012335300000000001 2024-08-17T17:24:22.772 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.772 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.772 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:24:22.772 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.445884+0000", 2024-08-17T17:24:22.772 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015065900000000001 2024-08-17T17:24:22.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.773 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.773 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.453263+0000", 2024-08-17T17:24:22.773 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.007378356 2024-08-17T17:24:22.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.773 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:24:22.773 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.453482+0000", 2024-08-17T17:24:22.773 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00021963899999999999 2024-08-17T17:24:22.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.773 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.774 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.459021+0000", 2024-08-17T17:24:22.774 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0055392649999999998 2024-08-17T17:24:22.774 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.774 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.774 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:24:22.774 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.459055+0000", 2024-08-17T17:24:22.774 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3380000000000002e-05 2024-08-17T17:24:22.774 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.774 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.774 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:24:22.774 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.459095+0000", 2024-08-17T17:24:22.774 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9817999999999997e-05 2024-08-17T17:24:22.774 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.775 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:24:22.775 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.775 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.775 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.775 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:57364 5.1 5:88cf257a:::benchmark_data_smithi003_140793_object57363:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-08-17T17:24:22.775 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:46.521867+0000", 2024-08-17T17:24:22.775 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 276.106591111, 2024-08-17T17:24:22.775 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.94252200200000003, 2024-08-17T17:24:22.775 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:24:22.775 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:24:22.775 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:22.775 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:24:22.775 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:24:22.776 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:22.776 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 57364 2024-08-17T17:24:22.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.776 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:24:22.776 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.776 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:24:22.776 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.521867+0000", 2024-08-17T17:24:22.776 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.776 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.776 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:24:22.776 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.521867+0000", 2024-08-17T17:24:22.776 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.777 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.777 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:24:22.777 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.521868+0000", 2024-08-17T17:24:22.777 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.359e-06 2024-08-17T17:24:22.777 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.777 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.777 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:24:22.777 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.521897+0000", 2024-08-17T17:24:22.777 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8449000000000002e-05 2024-08-17T17:24:22.777 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.777 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.777 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:24:22.777 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.521910+0000", 2024-08-17T17:24:22.778 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3268000000000001e-05 2024-08-17T17:24:22.778 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.778 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.778 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:24:22.778 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.521917+0000", 2024-08-17T17:24:22.778 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4500000000000001e-06 2024-08-17T17:24:22.778 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.778 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.778 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.778 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.521994+0000", 2024-08-17T17:24:22.778 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7286000000000005e-05 2024-08-17T17:24:22.778 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.778 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.779 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.779 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:46.522016+0000", 2024-08-17T17:24:22.779 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1838999999999999e-05 2024-08-17T17:24:22.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.779 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.779 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.779 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.448024+0000", 2024-08-17T17:24:22.779 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.92600859899999999 2024-08-17T17:24:22.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.779 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.779 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:24:22.779 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.448158+0000", 2024-08-17T17:24:22.779 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013402999999999999 2024-08-17T17:24:22.780 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.780 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.780 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:24:22.780 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.448300+0000", 2024-08-17T17:24:22.780 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014166 2024-08-17T17:24:22.780 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.780 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.780 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.780 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.453544+0000", 2024-08-17T17:24:22.780 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0052435570000000003 2024-08-17T17:24:22.780 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.780 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.780 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:24:22.781 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.453556+0000", 2024-08-17T17:24:22.781 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2252000000000001e-05 2024-08-17T17:24:22.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.781 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.781 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.781 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.464279+0000", 2024-08-17T17:24:22.781 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010723534 2024-08-17T17:24:22.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.781 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.781 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:24:22.781 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.464343+0000", 2024-08-17T17:24:22.781 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3569000000000001e-05 2024-08-17T17:24:22.782 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.782 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.782 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:24:22.782 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:47.464389+0000", 2024-08-17T17:24:22.782 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6149999999999997e-05 2024-08-17T17:24:22.782 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.782 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:24:22.782 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.782 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.782 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.782 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:68138 5.1 5:8cd8b222:::benchmark_data_smithi003_140793_object68137:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-08-17T17:24:22.782 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:56.511667+0000", 2024-08-17T17:24:22.782 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 266.11679096300003, 2024-08-17T17:24:22.783 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2112100479999999, 2024-08-17T17:24:22.783 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:24:22.783 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:24:22.783 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:22.783 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:24:22.783 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:24:22.783 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:22.783 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68138 2024-08-17T17:24:22.783 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.783 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:24:22.783 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.783 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:24:22.783 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.511667+0000", 2024-08-17T17:24:22.784 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.784 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.784 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.784 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:24:22.784 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.511667+0000", 2024-08-17T17:24:22.784 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.784 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.784 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.784 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:24:22.784 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.511669+0000", 2024-08-17T17:24:22.784 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.61e-06 2024-08-17T17:24:22.784 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.784 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.784 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:24:22.784 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.511692+0000", 2024-08-17T17:24:22.785 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2768e-05 2024-08-17T17:24:22.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.785 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:24:22.785 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.511703+0000", 2024-08-17T17:24:22.785 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1060000000000001e-05 2024-08-17T17:24:22.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.785 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:24:22.785 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.511711+0000", 2024-08-17T17:24:22.785 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9470000000000003e-06 2024-08-17T17:24:22.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.786 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.786 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.512317+0000", 2024-08-17T17:24:22.786 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00060645199999999995 2024-08-17T17:24:22.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.786 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.786 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.512327+0000", 2024-08-17T17:24:22.786 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7159999999999998e-06 2024-08-17T17:24:22.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.786 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.786 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.497176+0000", 2024-08-17T17:24:22.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.98484927799999999 2024-08-17T17:24:22.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.787 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.497189+0000", 2024-08-17T17:24:22.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2729e-05 2024-08-17T17:24:22.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.787 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.711927+0000", 2024-08-17T17:24:22.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2147383940000001 2024-08-17T17:24:22.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.788 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:24:22.788 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.712104+0000", 2024-08-17T17:24:22.788 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017708700000000001 2024-08-17T17:24:22.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.788 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.788 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:24:22.788 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.712300+0000", 2024-08-17T17:24:22.788 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000195906 2024-08-17T17:24:22.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.788 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.788 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.788 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.719596+0000", 2024-08-17T17:24:22.788 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072962979999999997 2024-08-17T17:24:22.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.789 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.789 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.720426+0000", 2024-08-17T17:24:22.789 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00082983500000000001 2024-08-17T17:24:22.789 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.789 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:24:22.789 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.720670+0000", 2024-08-17T17:24:22.789 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000243856 2024-08-17T17:24:22.789 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.789 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:24:22.790 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.720724+0000", 2024-08-17T17:24:22.790 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4051000000000001e-05 2024-08-17T17:24:22.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.790 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:24:22.790 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.722877+0000", 2024-08-17T17:24:22.790 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002153061 2024-08-17T17:24:22.790 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.790 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:24:22.790 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.790 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:68143 5.1 5:8f270daa:::benchmark_data_smithi003_140793_object68142:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-08-17T17:24:22.791 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:56.519782+0000", 2024-08-17T17:24:22.791 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 266.10867623600001, 2024-08-17T17:24:22.791 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2052141999999999, 2024-08-17T17:24:22.791 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:24:22.791 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:24:22.791 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:22.791 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:24:22.791 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:24:22.791 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:22.791 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68143 2024-08-17T17:24:22.791 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.791 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:24:22.791 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.792 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:24:22.792 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519782+0000", 2024-08-17T17:24:22.792 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.792 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.792 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.792 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:24:22.792 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519782+0000", 2024-08-17T17:24:22.792 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:24:22.792 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.792 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.792 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:24:22.792 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519783+0000", 2024-08-17T17:24:22.792 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1310000000000001e-06 2024-08-17T17:24:22.792 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.793 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.793 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:24:22.793 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519802+0000", 2024-08-17T17:24:22.793 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9018999999999999e-05 2024-08-17T17:24:22.793 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.793 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.793 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:24:22.793 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519810+0000", 2024-08-17T17:24:22.793 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9370000000000004e-06 2024-08-17T17:24:22.793 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.793 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.793 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:24:22.793 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519877+0000", 2024-08-17T17:24:22.794 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7074000000000002e-05 2024-08-17T17:24:22.794 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.794 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.794 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.794 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519923+0000", 2024-08-17T17:24:22.794 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6057999999999998e-05 2024-08-17T17:24:22.794 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.794 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.794 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.794 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519930+0000", 2024-08-17T17:24:22.794 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5379999999999999e-06 2024-08-17T17:24:22.794 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.795 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.795 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.795 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.497221+0000", 2024-08-17T17:24:22.795 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.977291668 2024-08-17T17:24:22.795 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.795 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.795 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:24:22.795 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.497230+0000", 2024-08-17T17:24:22.795 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2570000000000007e-06 2024-08-17T17:24:22.795 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.795 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.795 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:24:22.795 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.716884+0000", 2024-08-17T17:24:22.796 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2196546719999999 2024-08-17T17:24:22.796 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.796 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.796 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:24:22.796 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.717065+0000", 2024-08-17T17:24:22.796 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000181163 2024-08-17T17:24:22.796 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.796 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.796 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:24:22.796 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.717263+0000", 2024-08-17T17:24:22.796 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019748699999999999 2024-08-17T17:24:22.796 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.796 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.797 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.797 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.720657+0000", 2024-08-17T17:24:22.797 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003393784 2024-08-17T17:24:22.797 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.797 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.797 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:24:22.797 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.722900+0000", 2024-08-17T17:24:22.797 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0022431959999999998 2024-08-17T17:24:22.797 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.797 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.797 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:22.797 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.722944+0000", 2024-08-17T17:24:22.797 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3656999999999999e-05 2024-08-17T17:24:22.798 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.798 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.798 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:24:22.798 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.722995+0000", 2024-08-17T17:24:22.798 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1264999999999998e-05 2024-08-17T17:24:22.798 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:24:22.798 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:24:22.798 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:24:22.798 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.724996+0000", 2024-08-17T17:24:22.798 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020012939999999998 2024-08-17T17:24:22.798 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.798 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:24:22.799 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.799 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:24:22.799 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:24:22.799 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:24:22.799 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:22.826 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:22.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:22.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:22.928 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:22.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:22.955 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:22.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:23.056 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:23.082 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:23.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:23.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:23.184 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:23.213 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:23.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:23.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:23.315 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:23.337 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:23.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:23.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:23.439 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:23.465 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:23.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:23.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:23.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:23 smithi003 ceph-mon[103344]: pgmap v446: 89 pgs: 89 active+clean; 4.6 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:23.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:24:23.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:24:23.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:23 smithi003 ceph-mon[100615]: pgmap v446: 89 pgs: 89 active+clean; 4.6 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:23.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:24:23.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:24:23.567 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:23.588 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:23.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:23.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:23.690 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:23 smithi053 ceph-mon[89963]: pgmap v446: 89 pgs: 89 active+clean; 4.6 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:24:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:24:23.717 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:23.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:23.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:23.819 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:23.845 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:23.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:23.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:23.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:23.969 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:23.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:23.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:24.071 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:24.096 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:24.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:24.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:24.198 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:24.223 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:24.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:24.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:24.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:24.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:24.349 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:24.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:24.449 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:24.472 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:24.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:24.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:24.574 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:24.600 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:24.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:24.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:24.702 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:24.729 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:24.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:24.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:24.830 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:24.857 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:24.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:24.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:24.959 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:24.981 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:24.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:24.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:25.083 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:25.110 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:25.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:25.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:25.212 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:25.234 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:25.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:25.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:25.278 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-17T17:24:25.336 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:25.360 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:25.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:25.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:25.462 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:25.485 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:25.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:25.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:25.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:25 smithi003 ceph-mon[103344]: pgmap v447: 89 pgs: 89 active+clean; 4.6 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:25.506 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:25 smithi003 ceph-mon[100615]: pgmap v447: 89 pgs: 89 active+clean; 4.6 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:25.559 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:24:25.587 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:25.610 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:25.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:25.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:25.682 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:25 smithi053 ceph-mon[89963]: pgmap v447: 89 pgs: 89 active+clean; 4.6 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:25.711 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:25.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:25.735 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:25.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:25.772 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:24:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:24:25] "GET /metrics HTTP/1.1" 200 36495 "" "Prometheus/2.51.0" 2024-08-17T17:24:25.836 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:25.858 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:25.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:25.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:25.955 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:24:25.956 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:24:25.959 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:25.973 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:24:25.973 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:24:25.982 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:25.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:25.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:25.991 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:24:25.991 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:24:26.007 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:24:26.007 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:24:26.023 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:24:26.023 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:24:26.039 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:24:26.040 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:24:26.056 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:24:26.056 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:24:26.071 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:24:26.072 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:24:26.083 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:26.105 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:26.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:26.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:26.207 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:26.231 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:26.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:26.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:26.280 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-08-17T17:24:26.333 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:26.355 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:26.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:26.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:26.457 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:26.479 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:26.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:26.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:26.493 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:26 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/139272304' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:24:26.493 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:26 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:24:26.494 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:26 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1279592312' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:24:26.494 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:26 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/139272304' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:24:26.494 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:26 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:24:26.494 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:26 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1279592312' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:24:26.582 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:26.606 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:26.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:26.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:26.708 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:26.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:26 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/139272304' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:24:26.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:26 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:24:26.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:26 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1279592312' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:24:26.732 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:26.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:26.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:26.834 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:26.859 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:26.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:26.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:26.960 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:26.983 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:26.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:26.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:27.085 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:27.099 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:24:27.107 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:27.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:27.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:27.209 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:27.233 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:27.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:27.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:27.278 INFO:teuthology.orchestra.run.smithi003.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-08-17T17:24:27.335 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:27.349 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:24:27.350 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:24:27.356 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:27.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:27.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:27.458 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:27.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:27 smithi053 ceph-mon[89963]: pgmap v448: 89 pgs: 89 active+clean; 4.6 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:27.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:27 smithi053 ceph-mon[89963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:24:27.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:27 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:24:27.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:27 smithi053 ceph-mon[89963]: osdmap e182: 8 total, 8 up, 7 in 2024-08-17T17:24:27.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:27 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3935744126' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:24:27.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:27 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:24:27.480 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:27.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:27.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:27.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:27 smithi003 ceph-mon[100615]: pgmap v448: 89 pgs: 89 active+clean; 4.6 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:27.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:27 smithi003 ceph-mon[100615]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:24:27.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:27 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:24:27.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:27 smithi003 ceph-mon[100615]: osdmap e182: 8 total, 8 up, 7 in 2024-08-17T17:24:27.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:27 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3935744126' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:24:27.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:27 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:24:27.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:27 smithi003 ceph-mon[103344]: pgmap v448: 89 pgs: 89 active+clean; 4.6 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:27.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:27 smithi003 ceph-mon[103344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:24:27.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:27 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:24:27.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:27 smithi003 ceph-mon[103344]: osdmap e182: 8 total, 8 up, 7 in 2024-08-17T17:24:27.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:27 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3935744126' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:24:27.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:27 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:24:27.582 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:27.604 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:27.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:27.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:27.706 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:27.728 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:27.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:27.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:27.830 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:27.853 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:27.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:27.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:27.954 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:27.977 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:27.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:27.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:28.039 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-17T17:24:28.079 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:28.101 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:28.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:28.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:28.203 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:28.225 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:28.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:28.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:28.308 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:24:28.327 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:28.349 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:28.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:28.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:28.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:28.473 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:28.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:28.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:28.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:28 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T17:24:28.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:28 smithi003 ceph-mon[103344]: osdmap e183: 8 total, 8 up, 7 in 2024-08-17T17:24:28.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:28 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:24:28.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:28 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T17:24:28.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:28 smithi003 ceph-mon[100615]: osdmap e183: 8 total, 8 up, 7 in 2024-08-17T17:24:28.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:28 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:24:28.566 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:28 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T17:24:28.566 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:28 smithi053 ceph-mon[89963]: osdmap e183: 8 total, 8 up, 7 in 2024-08-17T17:24:28.566 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:28 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:24:28.575 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:28.597 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:28.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:28.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:28.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:28.721 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:28.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:28.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:28.823 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:28.913 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:24:28.918 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:28.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:28.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:29.004 DEBUG:teuthology.orchestra.run.smithi003:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-08-17T17:24:29.007 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-08-17T17:24:29.007 INFO:teuthology.orchestra.run:waiting for 3000 2024-08-17T17:24:29.024 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:29.049 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:29.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:29.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:29.055 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:hints = 1 2024-08-17T17:24:29.055 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-08-17T17:24:29.055 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object prefix: benchmark_data_smithi003_175043 2024-08-17T17:24:29.056 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:24:29.056 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 0 0 0 0 0 0 - 0 2024-08-17T17:24:29.150 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:29.198 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:29.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:29.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:29.300 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:29.344 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:29.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:29.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:29.355 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:29 smithi053 ceph-mon[89963]: pgmap v451: 89 pgs: 89 active+clean; 4.6 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:29.355 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:29 smithi053 ceph-mon[89963]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:24:29.446 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:29.500 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:29.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:29.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:29.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:29 smithi003 ceph-mon[100615]: pgmap v451: 89 pgs: 89 active+clean; 4.6 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:29.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:29 smithi003 ceph-mon[100615]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:24:29.551 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:29 smithi003 ceph-mon[103344]: pgmap v451: 89 pgs: 89 active+clean; 4.6 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:29.551 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:29 smithi003 ceph-mon[103344]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:24:29.602 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:29.644 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:29.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:29.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:29.731 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:24:29.746 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:29.795 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:29.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:29.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:29.897 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:29.934 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:29.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:29.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:30.006 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:24:30.006 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:24:30.035 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:30.056 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 1 16 2928 2912 181.954 182 0.00433077 0.00548121 2024-08-17T17:24:30.073 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:30.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:30.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:30.175 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:30.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:30.221 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:30.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:30.322 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:30.361 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:30.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:30.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:30.463 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:30.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:24:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:24:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:24:30.493 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:30.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:30.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:30.595 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:30.623 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:30.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:30.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:30.725 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:30.753 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:30.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:30.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:30.856 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:30.858 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-17T17:24:30.892 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:24:30.895 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:30.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:30.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:30.997 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:31.027 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:31.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:31.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:31.056 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 2 15 5655 5640 176.22 170.5 0.00924584 0.00566661 2024-08-17T17:24:31.129 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:31.160 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:31.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:31.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:31.218 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:24:31.262 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:31.291 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:31.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:31.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:31.299 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.2 to 1 (10000) 2024-08-17T17:24:31.393 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:31.417 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:31.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:31.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:31.519 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:31.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:31 smithi003 ceph-mon[103344]: pgmap v452: 89 pgs: 89 active+clean; 4.6 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:31.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:31 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1006964794' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:24:31.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:31 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:24:31.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:31 smithi003 ceph-mon[100615]: pgmap v452: 89 pgs: 89 active+clean; 4.6 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:31.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:31 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1006964794' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:24:31.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:31 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:24:31.541 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:31.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:31.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:31.576 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:31 smithi053 ceph-mon[89963]: pgmap v452: 89 pgs: 89 active+clean; 4.6 MiB data, 2.4 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:24:31.576 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:31 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1006964794' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:24:31.576 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:31 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:24:31.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:31.665 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:31.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:31.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:31.767 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:31.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:31.792 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:31.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:31.892 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:31.915 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:31.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:31.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:32.017 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:32.039 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:32.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:32.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:32.056 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 3 16 6556 6540 136.23 56.25 0.00121065 0.00562802 2024-08-17T17:24:32.141 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:32.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:32.166 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:32.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:32.267 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:32.291 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:32.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:32.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:32.393 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:32.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:32.419 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:32.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:32.520 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:32.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:32 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-17T17:24:32.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:32 smithi003 ceph-mon[100615]: osdmap e184: 8 total, 8 up, 7 in 2024-08-17T17:24:32.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:32 smithi003 ceph-mon[100615]: osdmap e185: 8 total, 8 up, 7 in 2024-08-17T17:24:32.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:32 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-17T17:24:32.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:32 smithi003 ceph-mon[103344]: osdmap e184: 8 total, 8 up, 7 in 2024-08-17T17:24:32.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:32 smithi003 ceph-mon[103344]: osdmap e185: 8 total, 8 up, 7 in 2024-08-17T17:24:32.542 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:32.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:32.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:32.644 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:32.668 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:32.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:32.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:32.675 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:32 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-17T17:24:32.675 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:32 smithi053 ceph-mon[89963]: osdmap e184: 8 total, 8 up, 7 in 2024-08-17T17:24:32.675 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:32 smithi053 ceph-mon[89963]: osdmap e185: 8 total, 8 up, 7 in 2024-08-17T17:24:32.717 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 3 1 2024-08-17T17:24:32.769 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:32.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:32.793 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:32.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:32.879 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:24:32.879 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:24:32.894 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:32.916 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:32.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:32.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:33.018 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:33.040 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:33.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:33.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:33.056 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 4 16 6556 6540 102.174 0 - 0.00562802 2024-08-17T17:24:33.142 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:33.163 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:33.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:33.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:33.265 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:33.287 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:33.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:33.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:33.389 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:33.411 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:33.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:33.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:33.414 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:24:33.414 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:24:33.420 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:24:33.431 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:24:33.432 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:24:33.449 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:24:33.449 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:24:33.468 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:24:33.468 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:24:33.484 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:24:33.485 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:24:33.500 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:24:33.501 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:24:33.513 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:33.516 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:24:33.516 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:24:33.531 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:24:33.531 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:24:33.535 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:33.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:33.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:33.565 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:33 smithi003 ceph-mon[103344]: pgmap v454: 89 pgs: 89 active+clean; 160 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 26 MiB/s wr, 414 op/s 2024-08-17T17:24:33.565 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:33 smithi003 ceph-mon[103344]: osdmap e186: 8 total, 8 up, 7 in 2024-08-17T17:24:33.565 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:33 smithi003 ceph-mon[100615]: pgmap v454: 89 pgs: 89 active+clean; 160 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 26 MiB/s wr, 414 op/s 2024-08-17T17:24:33.565 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:33 smithi003 ceph-mon[100615]: osdmap e186: 8 total, 8 up, 7 in 2024-08-17T17:24:33.637 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:33.660 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:33.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:33.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:33.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:33 smithi053 ceph-mon[89963]: pgmap v454: 89 pgs: 89 active+clean; 160 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 26 MiB/s wr, 414 op/s 2024-08-17T17:24:33.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:33 smithi053 ceph-mon[89963]: osdmap e186: 8 total, 8 up, 7 in 2024-08-17T17:24:33.761 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:33.783 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:33.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:33.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:33.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:33.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:33.918 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:33.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:33.958 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-17T17:24:34.019 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:34.051 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:34.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:34.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:34.056 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 5 16 7009 6993 87.4017 14.1562 0.00507782 0.0114302 2024-08-17T17:24:34.152 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:34.192 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:34.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:34.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:34.294 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:34.334 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:34.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:34.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:34.338 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:24:34.437 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:34.477 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:34.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:34.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:34.482 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:34 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/4090239894' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:24:34.483 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:34 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/4090239894' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:24:34.578 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:34.610 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:34 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/4090239894' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:24:34.620 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:34.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:34.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:34.722 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:34.756 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:34.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:34.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:34.858 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:34.882 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:34.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:34.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:34.984 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:35.014 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:35.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:35.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:35.056 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 6 16 9627 9611 100.103 163.625 0.00429409 0.00997959 2024-08-17T17:24:35.116 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:35.144 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:35.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:35.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:35.246 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:35.275 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:35.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:35.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:35.378 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:35.405 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:35.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:35.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:35.422 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:24:35.507 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:35.536 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:35.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:35.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:35.607 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:35 smithi003 ceph-mon[103344]: pgmap v457: 89 pgs: 89 active+clean; 265 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 43 MiB/s wr, 694 op/s; 94 B/s, 0 objects/s recovering 2024-08-17T17:24:35.608 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:35 smithi003 ceph-mon[100615]: pgmap v457: 89 pgs: 89 active+clean; 265 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 43 MiB/s wr, 694 op/s; 94 B/s, 0 objects/s recovering 2024-08-17T17:24:35.614 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:35 smithi053 ceph-mon[89963]: pgmap v457: 89 pgs: 89 active+clean; 265 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 43 MiB/s wr, 694 op/s; 94 B/s, 0 objects/s recovering 2024-08-17T17:24:35.638 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:35.683 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:35.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:35.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:35.785 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:35.813 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:35.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:35.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:35.860 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:24:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:24:35] "GET /metrics HTTP/1.1" 200 36495 "" "Prometheus/2.51.0" 2024-08-17T17:24:35.914 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:35.944 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:35.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:35.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:36.046 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:36.069 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:36.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:36.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:36.074 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 7 16 11992 11976 106.638 147.812 0.00221177 0.00934157 2024-08-17T17:24:36.108 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:24:36.109 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2024-08-17T17:24:36.109 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.109 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:12024 6.7 6:edef8d97:::benchmark_data_smithi003_175043_object12023:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-08-17T17:24:36.109 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:24:36.086399+0000", 2024-08-17T17:24:36.109 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.0063312430000000003, 2024-08-17T17:24:36.109 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0063542349999999997, 2024-08-17T17:24:36.109 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:24:36.109 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:24:36.109 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:24:36.109 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:24:36.110 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:24:36.110 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 12024 2024-08-17T17:24:36.110 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.110 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:24:36.110 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.110 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:24:36.110 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.086399+0000", 2024-08-17T17:24:36.110 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:36.110 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.110 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.110 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:24:36.110 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.086399+0000", 2024-08-17T17:24:36.111 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:36.111 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.111 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.111 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:24:36.111 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.086397+0000", 2024-08-17T17:24:36.111 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:24:36.111 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.111 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.111 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:24:36.111 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.086444+0000", 2024-08-17T17:24:36.111 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7058999999999998e-05 2024-08-17T17:24:36.111 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.111 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.111 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:24:36.112 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.086459+0000", 2024-08-17T17:24:36.112 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4769e-05 2024-08-17T17:24:36.112 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.112 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.112 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:24:36.112 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.086466+0000", 2024-08-17T17:24:36.112 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.4499999999999998e-06 2024-08-17T17:24:36.112 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.112 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.112 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:24:36.112 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.086514+0000", 2024-08-17T17:24:36.112 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7845e-05 2024-08-17T17:24:36.113 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.113 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.113 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:24:36.113 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.086626+0000", 2024-08-17T17:24:36.113 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000112038 2024-08-17T17:24:36.113 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.113 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.113 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:24:36.113 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.086788+0000", 2024-08-17T17:24:36.113 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016162300000000001 2024-08-17T17:24:36.113 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.113 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.113 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:24:36.114 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.088093+0000", 2024-08-17T17:24:36.114 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001305123 2024-08-17T17:24:36.114 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.114 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.114 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:36.114 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.090637+0000", 2024-08-17T17:24:36.114 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0025438349999999999 2024-08-17T17:24:36.114 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:36.114 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:24:36.115 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:36.115 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.115 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.115 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:12026 6.7 6:ea2da925:::benchmark_data_smithi003_175043_object12025:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-08-17T17:24:36.115 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:24:36.087103+0000", 2024-08-17T17:24:36.115 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.0056273820000000002, 2024-08-17T17:24:36.115 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0057152890000000001, 2024-08-17T17:24:36.115 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:24:36.115 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:24:36.115 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:24:36.115 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:24:36.115 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:24:36.115 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 12026 2024-08-17T17:24:36.115 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.116 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:24:36.116 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.116 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:24:36.116 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.087103+0000", 2024-08-17T17:24:36.116 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:36.116 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.116 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.116 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:24:36.116 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.087103+0000", 2024-08-17T17:24:36.116 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:36.117 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.117 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.117 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:24:36.117 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.087102+0000", 2024-08-17T17:24:36.117 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:24:36.117 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.117 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.117 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:24:36.117 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.087172+0000", 2024-08-17T17:24:36.117 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0572000000000001e-05 2024-08-17T17:24:36.117 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.117 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.117 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:24:36.117 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.087191+0000", 2024-08-17T17:24:36.118 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9159000000000001e-05 2024-08-17T17:24:36.118 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.118 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.118 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:24:36.118 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.087201+0000", 2024-08-17T17:24:36.118 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.3330000000000003e-06 2024-08-17T17:24:36.118 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.118 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.118 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:24:36.118 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.087420+0000", 2024-08-17T17:24:36.118 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021970900000000001 2024-08-17T17:24:36.118 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.118 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.119 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:24:36.119 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.087531+0000", 2024-08-17T17:24:36.119 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000110713 2024-08-17T17:24:36.119 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.119 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.119 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:24:36.119 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.087661+0000", 2024-08-17T17:24:36.119 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000130331 2024-08-17T17:24:36.119 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.120 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.120 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:24:36.120 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.088267+0000", 2024-08-17T17:24:36.120 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00060597600000000004 2024-08-17T17:24:36.120 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.120 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.120 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:36.120 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.090674+0000", 2024-08-17T17:24:36.120 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002406086 2024-08-17T17:24:36.120 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:36.120 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:24:36.120 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:36.120 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.121 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.121 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:12036 6.7 6:eedce5ab:::benchmark_data_smithi003_175043_object12035:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-08-17T17:24:36.121 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:24:36.091044+0000", 2024-08-17T17:24:36.121 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.001686276, 2024-08-17T17:24:36.121 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001878266, 2024-08-17T17:24:36.121 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:24:36.121 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:24:36.121 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:24:36.121 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:24:36.121 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:24:36.121 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 12036 2024-08-17T17:24:36.121 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.121 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:24:36.122 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.122 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:24:36.122 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.091044+0000", 2024-08-17T17:24:36.122 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:36.122 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.122 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.122 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:24:36.122 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.091044+0000", 2024-08-17T17:24:36.122 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:36.122 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.123 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.123 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:24:36.123 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.091042+0000", 2024-08-17T17:24:36.123 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:24:36.123 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.123 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.123 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:24:36.123 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.091096+0000", 2024-08-17T17:24:36.123 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4120999999999999e-05 2024-08-17T17:24:36.123 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.123 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.123 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:24:36.123 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.091106+0000", 2024-08-17T17:24:36.123 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.842e-06 2024-08-17T17:24:36.124 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.124 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.124 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:24:36.124 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.091110+0000", 2024-08-17T17:24:36.124 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9129999999999996e-06 2024-08-17T17:24:36.124 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.124 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.124 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:24:36.124 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.091156+0000", 2024-08-17T17:24:36.124 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5784000000000001e-05 2024-08-17T17:24:36.124 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.124 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.124 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:24:36.125 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.091279+0000", 2024-08-17T17:24:36.125 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000123297 2024-08-17T17:24:36.125 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.125 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.125 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:24:36.125 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.091473+0000", 2024-08-17T17:24:36.125 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000194234 2024-08-17T17:24:36.125 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.125 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.125 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:24:36.125 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.092203+0000", 2024-08-17T17:24:36.125 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000729968 2024-08-17T17:24:36.125 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:36.125 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:24:36.125 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:36.126 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.126 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.126 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop_reply(client.45491.0:12036 6.7 e186/178)", 2024-08-17T17:24:36.126 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:24:36.092950+0000", 2024-08-17T17:24:36.126 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4294967295.9997797, 2024-08-17T17:24:36.126 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0880000000000001e-05, 2024-08-17T17:24:36.126 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:24:36.126 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "queued for pg", 2024-08-17T17:24:36.126 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:24:36.126 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.126 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:24:36.126 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.092950+0000", 2024-08-17T17:24:36.126 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:36.127 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.127 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.127 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:24:36.127 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.092950+0000", 2024-08-17T17:24:36.127 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:36.127 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.127 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.127 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:24:36.127 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.092950+0000", 2024-08-17T17:24:36.127 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:24:36.127 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.127 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.127 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:24:36.128 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.092956+0000", 2024-08-17T17:24:36.128 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5189999999999998e-06 2024-08-17T17:24:36.128 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.128 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.128 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:24:36.128 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.092966+0000", 2024-08-17T17:24:36.128 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.7410000000000004e-06 2024-08-17T17:24:36.128 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:36.128 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:36.128 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:24:36.128 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:36.092970+0000", 2024-08-17T17:24:36.128 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.7859999999999998e-06 2024-08-17T17:24:36.128 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:36.129 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:24:36.129 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:36.129 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:36.129 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-08-17T17:24:36.129 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 4 2024-08-17T17:24:36.129 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:24:36.172 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:36.200 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:36.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:36.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:36.302 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:36.332 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:36.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:36.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:36.399 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:24:36.433 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:36.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:36.493 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:36.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:36.594 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:36.648 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:36.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:36.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:36.750 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:36.794 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:36.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:36.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:36.896 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:36.930 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-17T17:24:36.946 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:36.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:36.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:37.050 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:37.075 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 8 16 14139 14123 110.067 134.188 0.00543625 0.00907582 2024-08-17T17:24:37.091 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:37.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:37.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:37.193 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:37.222 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:37.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:37.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:37.269 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:24:37.275 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:24:37.324 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:37.338 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-08-17T17:24:37.351 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:37.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:37.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:37.372 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:37 smithi003 ceph-mon[103344]: pgmap v458: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 448 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s; 291/21369 objects misplaced (1.362%); 1.4 MiB/s, 23 objects/s recovering 2024-08-17T17:24:37.372 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:37 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3125200219' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:24:37.372 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:37 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:24:37.373 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:37 smithi003 ceph-mon[100615]: pgmap v458: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 448 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s; 291/21369 objects misplaced (1.362%); 1.4 MiB/s, 23 objects/s recovering 2024-08-17T17:24:37.373 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:37 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3125200219' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:24:37.373 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:37 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:24:37.453 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:37.495 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:37.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:37.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:37.597 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:37.623 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:37.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:37.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:37.652 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:37 smithi053 ceph-mon[89963]: pgmap v458: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 448 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s; 291/21369 objects misplaced (1.362%); 1.4 MiB/s, 23 objects/s recovering 2024-08-17T17:24:37.652 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:37 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3125200219' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:24:37.652 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:37 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:24:37.726 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:37.763 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:37.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:37.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:37.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:37.895 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:37.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:37.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:37.998 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:38.037 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:38.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:38.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:38.075 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 9 16 16338 16322 113.101 137.438 0.0055864 0.00883454 2024-08-17T17:24:38.139 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:38.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:38.175 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:38.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:38.276 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:38.303 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:38.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:38.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:38.405 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:38.424 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:38 smithi053 ceph-mon[89963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:24:38.424 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:38 smithi053 ceph-mon[89963]: Cluster is now healthy 2024-08-17T17:24:38.424 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:38 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:24:38.424 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:38 smithi053 ceph-mon[89963]: osdmap e187: 8 total, 8 up, 7 in 2024-08-17T17:24:38.424 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:24:38.424 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:24:38.432 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:38.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:38.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:38.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:38 smithi003 ceph-mon[100615]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:24:38.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:38 smithi003 ceph-mon[100615]: Cluster is now healthy 2024-08-17T17:24:38.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:38 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:24:38.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:38 smithi003 ceph-mon[100615]: osdmap e187: 8 total, 8 up, 7 in 2024-08-17T17:24:38.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:24:38.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:24:38.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:38 smithi003 ceph-mon[103344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:24:38.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:38 smithi003 ceph-mon[103344]: Cluster is now healthy 2024-08-17T17:24:38.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:38 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:24:38.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:38 smithi003 ceph-mon[103344]: osdmap e187: 8 total, 8 up, 7 in 2024-08-17T17:24:38.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:24:38.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:24:38.534 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:38.556 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:38.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:38.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:38.658 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:38.681 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:38.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:38.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:38.782 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:38.805 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:38.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:38.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:38.906 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:38.930 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:38.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:38.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:39.032 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:39.057 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:39.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:39.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:39.075 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 10 16 17154 17138 106.901 51 0.00148082 0.00874439 2024-08-17T17:24:39.159 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:39.182 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:39.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:39.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:39.284 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:39.308 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:39.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:39.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:39.409 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:39.436 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:39.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:39.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:39.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:39 smithi003 ceph-mon[100615]: pgmap v460: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 571 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 68 MiB/s wr, 1.09k op/s; 291/27258 objects misplaced (1.068%); 1.4 MiB/s, 23 objects/s recovering 2024-08-17T17:24:39.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:39 smithi003 ceph-mon[100615]: osdmap e188: 8 total, 8 up, 7 in 2024-08-17T17:24:39.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [1, 2]} : dispatch 2024-08-17T17:24:39.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [3, 2]} : dispatch 2024-08-17T17:24:39.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:39 smithi003 ceph-mon[103344]: pgmap v460: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 571 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 68 MiB/s wr, 1.09k op/s; 291/27258 objects misplaced (1.068%); 1.4 MiB/s, 23 objects/s recovering 2024-08-17T17:24:39.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:39 smithi003 ceph-mon[103344]: osdmap e188: 8 total, 8 up, 7 in 2024-08-17T17:24:39.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [1, 2]} : dispatch 2024-08-17T17:24:39.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [3, 2]} : dispatch 2024-08-17T17:24:39.538 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:39.563 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:39.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:39.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:39.665 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:39.686 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:39 smithi053 ceph-mon[89963]: pgmap v460: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 571 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 68 MiB/s wr, 1.09k op/s; 291/27258 objects misplaced (1.068%); 1.4 MiB/s, 23 objects/s recovering 2024-08-17T17:24:39.686 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:39 smithi053 ceph-mon[89963]: osdmap e188: 8 total, 8 up, 7 in 2024-08-17T17:24:39.686 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [1, 2]} : dispatch 2024-08-17T17:24:39.686 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [3, 2]} : dispatch 2024-08-17T17:24:39.703 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:39.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:39.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:39.805 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:39.828 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:24:39.828 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2024-08-17T17:24:39.829 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2024-08-17T17:24:39.829 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2024-08-17T17:24:39.829 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:24:39.837 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:39.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:39.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:39.940 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:39.968 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:39.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:39.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:40.070 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:40.075 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 11 16 18562 18546 105.185 88 0.0098153 0.00949987 2024-08-17T17:24:40.108 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:40.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:40.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:40.210 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:40.247 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:40.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:40.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:40.349 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:40.378 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:40.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:40.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:40.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:24:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:24:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:24:40.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:40 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [1, 2]}]': finished 2024-08-17T17:24:40.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:40 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [3, 2]}]': finished 2024-08-17T17:24:40.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:40 smithi053 ceph-mon[89963]: osdmap e189: 8 total, 8 up, 7 in 2024-08-17T17:24:40.482 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:40.512 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:40.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:40.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:40.514 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-17T17:24:40.614 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:40.623 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:40 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [1, 2]}]': finished 2024-08-17T17:24:40.623 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:40 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [3, 2]}]': finished 2024-08-17T17:24:40.623 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:40 smithi003 ceph-mon[103344]: osdmap e189: 8 total, 8 up, 7 in 2024-08-17T17:24:40.623 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:40 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [1, 2]}]': finished 2024-08-17T17:24:40.623 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:40 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [3, 2]}]': finished 2024-08-17T17:24:40.623 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:40 smithi003 ceph-mon[100615]: osdmap e189: 8 total, 8 up, 7 in 2024-08-17T17:24:40.642 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:40.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:40.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:40.744 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:40.774 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:40.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:40.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:40.876 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:40.908 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:24:40.920 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:40.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:40.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:41.021 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:41.051 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:41.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:41.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:41.075 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 12 16 20647 20631 107.275 130.312 0.0145458 0.00931509 2024-08-17T17:24:41.153 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:41.193 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:41.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:41.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:41.295 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:41.323 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:41.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:41.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:41.388 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.3 to 1 (10000) 2024-08-17T17:24:41.425 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:41.452 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:41.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:41.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:41.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:41 smithi003 ceph-mon[103344]: pgmap v463: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 851 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 98 MiB/s wr, 1.56k op/s; 291/40701 objects misplaced (0.715%); 1.4 MiB/s, 23 objects/s recovering 2024-08-17T17:24:41.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:41 smithi003 ceph-mon[103344]: osdmap e190: 8 total, 8 up, 7 in 2024-08-17T17:24:41.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:41 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/636559866' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:24:41.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:41 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:24:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:41 smithi003 ceph-mon[100615]: pgmap v463: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 851 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 98 MiB/s wr, 1.56k op/s; 291/40701 objects misplaced (0.715%); 1.4 MiB/s, 23 objects/s recovering 2024-08-17T17:24:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:41 smithi003 ceph-mon[100615]: osdmap e190: 8 total, 8 up, 7 in 2024-08-17T17:24:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:41 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/636559866' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:24:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:41 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:24:41.554 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:41.575 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:41.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:41.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:41.677 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:41.703 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:41.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:41.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:41.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:41 smithi053 ceph-mon[89963]: pgmap v463: 89 pgs: 1 active+recovering+undersized+remapped, 88 active+clean; 851 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 98 MiB/s wr, 1.56k op/s; 291/40701 objects misplaced (0.715%); 1.4 MiB/s, 23 objects/s recovering 2024-08-17T17:24:41.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:41 smithi053 ceph-mon[89963]: osdmap e190: 8 total, 8 up, 7 in 2024-08-17T17:24:41.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:41 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/636559866' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:24:41.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:41 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:24:41.805 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:41.845 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:41.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:41.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:41.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:41.991 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:41.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:41.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:42.075 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 13 16 22501 22485 107.934 115.875 0.00463305 0.00925883 2024-08-17T17:24:42.093 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:42.130 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:42.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:42.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:42.232 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:42.273 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:42.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:42.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:42.375 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:42.412 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:42.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:42.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:42.514 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:42.550 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:42.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:42.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:42.555 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:42 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-17T17:24:42.555 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:42 smithi053 ceph-mon[89963]: osdmap e191: 8 total, 8 up, 7 in 2024-08-17T17:24:42.652 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:42.667 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:24:42.691 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:42.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:42.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:42.732 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:42 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-17T17:24:42.732 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:42 smithi003 ceph-mon[100615]: osdmap e191: 8 total, 8 up, 7 in 2024-08-17T17:24:42.732 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:42 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-17T17:24:42.732 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:42 smithi003 ceph-mon[103344]: osdmap e191: 8 total, 8 up, 7 in 2024-08-17T17:24:42.793 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:42.818 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:42.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:42.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:42.924 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:42.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:42.963 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:42.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:43.031 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:24:43.031 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2024-08-17T17:24:43.031 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2024-08-17T17:24:43.031 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2024-08-17T17:24:43.031 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.031 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:68127 5.6 5:61dbb48d:::benchmark_data_smithi003_140793_object68126:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-08-17T17:24:43.031 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:19:56.478364+0000", 2024-08-17T17:24:43.031 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 286.54981722100001, 2024-08-17T17:24:43.032 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2649956150000001, 2024-08-17T17:24:43.032 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:24:43.032 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2024-08-17T17:24:43.032 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:24:43.032 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:24:43.032 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:43.032 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 68127 2024-08-17T17:24:43.032 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.032 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:24:43.032 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.032 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:24:43.032 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.478364+0000", 2024-08-17T17:24:43.032 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.032 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.033 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.033 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:24:43.033 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.478364+0000", 2024-08-17T17:24:43.033 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.033 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.033 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.033 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:24:43.033 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.478364+0000", 2024-08-17T17:24:43.033 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:24:43.033 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.033 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.033 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:24:43.033 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.478393+0000", 2024-08-17T17:24:43.034 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8975e-05 2024-08-17T17:24:43.034 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.034 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.034 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:24:43.034 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.478400+0000", 2024-08-17T17:24:43.034 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.5580000000000003e-06 2024-08-17T17:24:43.034 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.034 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.034 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:24:43.034 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.478404+0000", 2024-08-17T17:24:43.034 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3409999999999996e-06 2024-08-17T17:24:43.034 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.035 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.035 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:24:43.035 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.479536+0000", 2024-08-17T17:24:43.035 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0011312430000000001 2024-08-17T17:24:43.035 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.035 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.035 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:24:43.035 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.479544+0000", 2024-08-17T17:24:43.035 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0709999999999995e-06 2024-08-17T17:24:43.035 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.035 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.035 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:24:43.035 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:57.488821+0000", 2024-08-17T17:24:43.035 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.009277507 2024-08-17T17:24:43.036 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.036 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.036 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:24:43.036 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:57.488833+0000", 2024-08-17T17:24:43.036 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1421e-05 2024-08-17T17:24:43.036 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.036 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.036 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:24:43.036 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:58.742225+0000", 2024-08-17T17:24:43.036 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2533923570000001 2024-08-17T17:24:43.036 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.036 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.037 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:24:43.037 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:58.743360+0000", 2024-08-17T17:24:43.037 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001134681 2024-08-17T17:24:43.037 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.037 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:24:43.037 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.037 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.037 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.037 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:155041 5.6 5:687221f3:::benchmark_data_smithi003_140793_object52984:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-08-17T17:24:43.037 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:22:05.002991+0000", 2024-08-17T17:24:43.037 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 158.02519014399999, 2024-08-17T17:24:43.037 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.13823176400000001, 2024-08-17T17:24:43.038 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:24:43.038 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:43.038 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:24:43.038 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:24:43.038 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:43.038 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 155041 2024-08-17T17:24:43.038 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.038 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:24:43.038 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.038 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:24:43.038 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.002991+0000", 2024-08-17T17:24:43.038 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.038 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.039 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.039 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:24:43.039 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.002991+0000", 2024-08-17T17:24:43.039 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.039 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.039 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.039 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:24:43.039 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.002990+0000", 2024-08-17T17:24:43.039 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:24:43.039 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.039 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.039 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:24:43.039 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.002996+0000", 2024-08-17T17:24:43.039 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4269999999999996e-06 2024-08-17T17:24:43.039 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.040 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.040 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:24:43.040 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.003009+0000", 2024-08-17T17:24:43.040 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2608e-05 2024-08-17T17:24:43.040 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.040 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.040 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:24:43.040 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.003013+0000", 2024-08-17T17:24:43.040 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.69e-06 2024-08-17T17:24:43.040 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.040 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.040 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:24:43.040 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.003075+0000", 2024-08-17T17:24:43.041 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1433000000000006e-05 2024-08-17T17:24:43.041 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.041 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.041 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:24:43.041 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.003229+0000", 2024-08-17T17:24:43.041 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015471199999999999 2024-08-17T17:24:43.041 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.041 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.041 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:24:43.041 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.003374+0000", 2024-08-17T17:24:43.041 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000144423 2024-08-17T17:24:43.041 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.041 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.042 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.042 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.008557+0000", 2024-08-17T17:24:43.042 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.005182697 2024-08-17T17:24:43.042 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.042 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.042 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:24:43.042 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.008567+0000", 2024-08-17T17:24:43.042 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0712e-05 2024-08-17T17:24:43.042 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.042 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.042 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.042 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.139046+0000", 2024-08-17T17:24:43.042 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.13047855799999999 2024-08-17T17:24:43.043 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.043 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.043 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:24:43.043 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.139117+0000", 2024-08-17T17:24:43.043 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0895000000000005e-05 2024-08-17T17:24:43.043 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.043 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.043 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:24:43.043 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.141223+0000", 2024-08-17T17:24:43.043 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002106236 2024-08-17T17:24:43.043 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.043 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:24:43.044 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.044 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.044 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.044 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:155052 5.6 5:6efe07a0:::benchmark_data_smithi003_140793_object52995:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-08-17T17:24:43.044 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:22:05.025081+0000", 2024-08-17T17:24:43.044 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 158.003100437, 2024-08-17T17:24:43.044 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.118019527, 2024-08-17T17:24:43.044 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:24:43.044 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:43.044 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:24:43.044 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:24:43.044 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:43.044 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 155052 2024-08-17T17:24:43.045 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.045 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:24:43.045 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.045 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:24:43.045 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.025081+0000", 2024-08-17T17:24:43.045 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.045 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.045 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.045 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:24:43.045 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.025081+0000", 2024-08-17T17:24:43.045 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.045 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.045 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.045 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:24:43.046 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.025079+0000", 2024-08-17T17:24:43.046 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999976 2024-08-17T17:24:43.046 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.046 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.047 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:24:43.047 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.025087+0000", 2024-08-17T17:24:43.047 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.3159999999999997e-06 2024-08-17T17:24:43.047 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.047 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.047 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:24:43.047 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.025106+0000", 2024-08-17T17:24:43.047 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.8751999999999999e-05 2024-08-17T17:24:43.047 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.047 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.047 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:24:43.047 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.025114+0000", 2024-08-17T17:24:43.047 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1389999999999995e-06 2024-08-17T17:24:43.047 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.047 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.048 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:24:43.048 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.025194+0000", 2024-08-17T17:24:43.048 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0366999999999998e-05 2024-08-17T17:24:43.048 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.048 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.048 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:24:43.048 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.025344+0000", 2024-08-17T17:24:43.048 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014968499999999999 2024-08-17T17:24:43.048 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.048 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.048 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:24:43.048 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.025497+0000", 2024-08-17T17:24:43.048 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015284700000000001 2024-08-17T17:24:43.049 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.049 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.049 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:24:43.049 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.034479+0000", 2024-08-17T17:24:43.049 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0089826950000000006 2024-08-17T17:24:43.049 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.049 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.049 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.049 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.044451+0000", 2024-08-17T17:24:43.049 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0099711810000000008 2024-08-17T17:24:43.049 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.049 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.049 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.049 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.141267+0000", 2024-08-17T17:24:43.049 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.096816540000000006 2024-08-17T17:24:43.050 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.050 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.050 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:24:43.050 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.141324+0000", 2024-08-17T17:24:43.050 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6586000000000003e-05 2024-08-17T17:24:43.050 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.050 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.050 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:24:43.050 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.143100+0000", 2024-08-17T17:24:43.050 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001776602 2024-08-17T17:24:43.050 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.050 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:24:43.050 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.050 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.051 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.051 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:155055 5.6 5:6d10c853:::benchmark_data_smithi003_140793_object52998:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-08-17T17:24:43.051 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:22:05.025126+0000", 2024-08-17T17:24:43.051 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 158.00305534699999, 2024-08-17T17:24:43.051 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.119811187, 2024-08-17T17:24:43.051 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:24:43.051 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:43.051 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:24:43.051 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:24:43.051 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:43.051 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 155055 2024-08-17T17:24:43.051 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.051 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:24:43.051 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.051 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:24:43.052 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.025126+0000", 2024-08-17T17:24:43.052 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.052 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.052 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.052 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:24:43.052 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.025126+0000", 2024-08-17T17:24:43.052 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.052 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.052 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.052 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:24:43.052 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.025125+0000", 2024-08-17T17:24:43.052 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:24:43.052 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.052 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.052 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:24:43.053 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.025129+0000", 2024-08-17T17:24:43.053 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4189999999999998e-06 2024-08-17T17:24:43.053 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.053 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.053 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:24:43.053 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.025153+0000", 2024-08-17T17:24:43.053 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4090000000000001e-05 2024-08-17T17:24:43.053 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.053 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.053 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:24:43.053 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.025158+0000", 2024-08-17T17:24:43.053 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7979999999999996e-06 2024-08-17T17:24:43.053 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.053 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.054 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:24:43.054 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.029400+0000", 2024-08-17T17:24:43.054 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.004242427 2024-08-17T17:24:43.054 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.054 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.054 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:24:43.054 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.029561+0000", 2024-08-17T17:24:43.054 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016091 2024-08-17T17:24:43.054 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.054 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.054 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:24:43.054 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.029724+0000", 2024-08-17T17:24:43.054 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016306099999999999 2024-08-17T17:24:43.054 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.054 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.055 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:24:43.055 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.034581+0000", 2024-08-17T17:24:43.055 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0048567419999999998 2024-08-17T17:24:43.055 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.055 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.055 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.055 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.044483+0000", 2024-08-17T17:24:43.055 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.009901946 2024-08-17T17:24:43.055 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.055 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.055 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.055 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.143155+0000", 2024-08-17T17:24:43.055 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.098671657999999995 2024-08-17T17:24:43.056 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.056 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.056 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:24:43.056 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.143252+0000", 2024-08-17T17:24:43.056 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.7639000000000005e-05 2024-08-17T17:24:43.056 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.056 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.056 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:24:43.056 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.144937+0000", 2024-08-17T17:24:43.056 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0016850019999999999 2024-08-17T17:24:43.056 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.056 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:24:43.056 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.057 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.057 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.057 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:155064 5.6 5:6f11f153:::benchmark_data_smithi003_140793_object53007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-08-17T17:24:43.057 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:22:05.035280+0000", 2024-08-17T17:24:43.057 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 157.992901725, 2024-08-17T17:24:43.057 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.111619966, 2024-08-17T17:24:43.057 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:24:43.057 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:43.057 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:24:43.057 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:24:43.057 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:43.057 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 155064 2024-08-17T17:24:43.057 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.057 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:24:43.058 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.058 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:24:43.058 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.035280+0000", 2024-08-17T17:24:43.058 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.058 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.058 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.058 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:24:43.058 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.035280+0000", 2024-08-17T17:24:43.058 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.058 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.058 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.058 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:24:43.058 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.035278+0000", 2024-08-17T17:24:43.058 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:24:43.059 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.059 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.059 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:24:43.059 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.035286+0000", 2024-08-17T17:24:43.059 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.8110000000000003e-06 2024-08-17T17:24:43.059 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.059 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.059 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:24:43.059 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.035305+0000", 2024-08-17T17:24:43.059 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9437e-05 2024-08-17T17:24:43.059 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.059 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.059 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:24:43.060 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.035312+0000", 2024-08-17T17:24:43.060 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0999999999999998e-06 2024-08-17T17:24:43.060 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.060 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.060 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:24:43.060 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.035355+0000", 2024-08-17T17:24:43.060 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2938999999999997e-05 2024-08-17T17:24:43.060 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.060 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.060 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:24:43.060 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.035509+0000", 2024-08-17T17:24:43.060 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015360499999999999 2024-08-17T17:24:43.060 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.061 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.061 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:24:43.061 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.035654+0000", 2024-08-17T17:24:43.061 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014486100000000001 2024-08-17T17:24:43.061 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.061 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.061 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.061 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.044515+0000", 2024-08-17T17:24:43.061 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0088617100000000001 2024-08-17T17:24:43.061 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.061 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.061 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:24:43.061 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.045044+0000", 2024-08-17T17:24:43.061 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00052873999999999998 2024-08-17T17:24:43.062 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.062 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.062 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.062 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.145039+0000", 2024-08-17T17:24:43.062 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.099994686999999999 2024-08-17T17:24:43.062 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.062 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.062 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:24:43.062 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.145173+0000", 2024-08-17T17:24:43.062 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013378000000000001 2024-08-17T17:24:43.062 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.062 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.062 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:24:43.062 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.146900+0000", 2024-08-17T17:24:43.063 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001727033 2024-08-17T17:24:43.063 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.063 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:24:43.063 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.063 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.063 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.063 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:155073 5.6 5:6e7ff565:::benchmark_data_smithi003_140793_object53016:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-08-17T17:24:43.063 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:22:05.051110+0000", 2024-08-17T17:24:43.063 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 157.97707096400001, 2024-08-17T17:24:43.063 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.11889108399999999, 2024-08-17T17:24:43.063 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:24:43.063 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:43.063 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:24:43.063 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:24:43.064 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:43.064 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 155073 2024-08-17T17:24:43.064 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.064 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:24:43.064 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.064 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:24:43.064 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.051110+0000", 2024-08-17T17:24:43.064 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.064 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.064 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.064 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:24:43.064 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.051110+0000", 2024-08-17T17:24:43.064 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.064 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.065 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.065 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:24:43.065 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.051110+0000", 2024-08-17T17:24:43.065 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:24:43.065 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.065 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.065 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:24:43.065 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.051114+0000", 2024-08-17T17:24:43.065 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.8289999999999998e-06 2024-08-17T17:24:43.065 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.065 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.065 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:24:43.065 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.051130+0000", 2024-08-17T17:24:43.066 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5139e-05 2024-08-17T17:24:43.066 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.066 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.066 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:24:43.066 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.051135+0000", 2024-08-17T17:24:43.066 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5790000000000003e-06 2024-08-17T17:24:43.066 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.066 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.066 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:24:43.066 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.051182+0000", 2024-08-17T17:24:43.066 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6977000000000001e-05 2024-08-17T17:24:43.066 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.067 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.067 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:24:43.067 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.051304+0000", 2024-08-17T17:24:43.067 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012221700000000001 2024-08-17T17:24:43.067 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.067 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.067 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:24:43.067 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.051416+0000", 2024-08-17T17:24:43.067 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000112052 2024-08-17T17:24:43.067 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.067 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.067 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:24:43.067 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.056143+0000", 2024-08-17T17:24:43.067 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0047267480000000002 2024-08-17T17:24:43.068 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.068 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.068 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.068 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.056279+0000", 2024-08-17T17:24:43.068 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013566500000000001 2024-08-17T17:24:43.068 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.068 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.068 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.068 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.167919+0000", 2024-08-17T17:24:43.068 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.111640168 2024-08-17T17:24:43.068 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.068 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.069 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:24:43.069 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.167977+0000", 2024-08-17T17:24:43.069 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7785999999999999e-05 2024-08-17T17:24:43.069 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.069 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.069 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:24:43.069 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.170001+0000", 2024-08-17T17:24:43.069 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0020247070000000002 2024-08-17T17:24:43.069 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.069 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:24:43.069 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.069 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.069 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.069 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:155076 5.6 5:66158e1f:::benchmark_data_smithi003_140793_object53019:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-08-17T17:24:43.069 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:22:05.061032+0000", 2024-08-17T17:24:43.070 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 157.96714934100001, 2024-08-17T17:24:43.070 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.110307611, 2024-08-17T17:24:43.070 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:24:43.070 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:43.070 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:24:43.070 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:24:43.070 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:43.070 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 155076 2024-08-17T17:24:43.070 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.070 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:24:43.070 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.070 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:24:43.070 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.061032+0000", 2024-08-17T17:24:43.070 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.071 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.071 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.071 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:24:43.071 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.061032+0000", 2024-08-17T17:24:43.071 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.071 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.071 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.071 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:24:43.071 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.061031+0000", 2024-08-17T17:24:43.071 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:24:43.071 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.071 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.072 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:24:43.072 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.061038+0000", 2024-08-17T17:24:43.072 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.418e-06 2024-08-17T17:24:43.072 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.072 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.072 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:24:43.072 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.061054+0000", 2024-08-17T17:24:43.072 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5617999999999999e-05 2024-08-17T17:24:43.072 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.072 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.072 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:24:43.072 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.061060+0000", 2024-08-17T17:24:43.072 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.754e-06 2024-08-17T17:24:43.072 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.073 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.073 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:24:43.073 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.061119+0000", 2024-08-17T17:24:43.073 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8649e-05 2024-08-17T17:24:43.073 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.073 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.073 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:24:43.073 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.061258+0000", 2024-08-17T17:24:43.073 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000139276 2024-08-17T17:24:43.073 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.073 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.073 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:24:43.073 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.061339+0000", 2024-08-17T17:24:43.073 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1086000000000003e-05 2024-08-17T17:24:43.074 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.074 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.074 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.074 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.065823+0000", 2024-08-17T17:24:43.074 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0044834860000000001 2024-08-17T17:24:43.074 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.074 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.074 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:24:43.074 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.076055+0000", 2024-08-17T17:24:43.074 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010232109 2024-08-17T17:24:43.074 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.074 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.074 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.075 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.170055+0000", 2024-08-17T17:24:43.075 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.094000104000000001 2024-08-17T17:24:43.075 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.075 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.075 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:24:43.075 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.170097+0000", 2024-08-17T17:24:43.075 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1928999999999998e-05 2024-08-17T17:24:43.075 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.075 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.075 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:24:43.075 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.171340+0000", 2024-08-17T17:24:43.075 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0012424549999999999 2024-08-17T17:24:43.075 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.075 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:24:43.076 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.076 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.076 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.076 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:155078 5.6 5:6c568685:::benchmark_data_smithi003_140793_object53021:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-08-17T17:24:43.076 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:22:05.062052+0000", 2024-08-17T17:24:43.076 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 157.96612915099999, 2024-08-17T17:24:43.076 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.11017162699999999, 2024-08-17T17:24:43.076 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:24:43.076 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:43.076 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:24:43.076 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:24:43.076 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:43.077 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 155078 2024-08-17T17:24:43.077 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.077 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:24:43.077 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.077 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:24:43.077 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.062052+0000", 2024-08-17T17:24:43.077 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.077 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.077 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.077 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:24:43.077 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.062052+0000", 2024-08-17T17:24:43.077 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.077 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.077 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.078 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:24:43.078 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.062051+0000", 2024-08-17T17:24:43.078 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:24:43.078 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.078 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.078 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:24:43.078 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.062056+0000", 2024-08-17T17:24:43.078 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6979999999999997e-06 2024-08-17T17:24:43.078 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.078 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.078 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:24:43.078 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.062067+0000", 2024-08-17T17:24:43.078 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0859000000000001e-05 2024-08-17T17:24:43.079 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.079 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.079 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:24:43.079 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.062073+0000", 2024-08-17T17:24:43.079 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4409999999999998e-06 2024-08-17T17:24:43.079 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.079 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.079 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:24:43.079 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.065849+0000", 2024-08-17T17:24:43.079 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003775105 2024-08-17T17:24:43.079 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.079 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.079 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:24:43.080 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.066005+0000", 2024-08-17T17:24:43.080 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015673999999999999 2024-08-17T17:24:43.080 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.080 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.080 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:24:43.080 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.066129+0000", 2024-08-17T17:24:43.080 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012327799999999999 2024-08-17T17:24:43.080 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.080 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.080 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.080 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.076043+0000", 2024-08-17T17:24:43.080 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0099146570000000003 2024-08-17T17:24:43.080 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.080 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.080 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:24:43.081 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.081449+0000", 2024-08-17T17:24:43.081 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0054059920000000001 2024-08-17T17:24:43.081 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.081 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.081 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.081 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.171384+0000", 2024-08-17T17:24:43.081 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.089934699000000007 2024-08-17T17:24:43.081 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.081 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.081 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:24:43.081 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.171417+0000", 2024-08-17T17:24:43.081 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2712000000000001e-05 2024-08-17T17:24:43.081 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.081 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.082 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:24:43.082 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.172224+0000", 2024-08-17T17:24:43.082 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00080719799999999997 2024-08-17T17:24:43.082 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.082 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:24:43.082 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.082 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.082 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.082 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:155079 5.6 5:6fa27f2d:::benchmark_data_smithi003_140793_object53022:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-08-17T17:24:43.082 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:22:05.064324+0000", 2024-08-17T17:24:43.082 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 157.963856916, 2024-08-17T17:24:43.083 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.10876187800000001, 2024-08-17T17:24:43.083 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:24:43.083 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:43.083 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:24:43.083 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:24:43.083 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:43.083 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 155079 2024-08-17T17:24:43.083 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.083 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:24:43.083 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.083 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:24:43.083 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.064324+0000", 2024-08-17T17:24:43.084 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.084 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.084 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.084 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:24:43.084 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.064324+0000", 2024-08-17T17:24:43.084 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.084 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.084 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.084 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:24:43.084 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.064323+0000", 2024-08-17T17:24:43.084 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:24:43.084 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.084 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.085 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:24:43.085 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.064331+0000", 2024-08-17T17:24:43.085 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.8219999999999996e-06 2024-08-17T17:24:43.085 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.085 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.085 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:24:43.085 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.064353+0000", 2024-08-17T17:24:43.085 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1895999999999998e-05 2024-08-17T17:24:43.085 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.085 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.085 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:24:43.085 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.064361+0000", 2024-08-17T17:24:43.085 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.2899999999999997e-06 2024-08-17T17:24:43.086 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.086 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.086 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:24:43.086 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.070673+0000", 2024-08-17T17:24:43.086 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0063119889999999996 2024-08-17T17:24:43.086 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.086 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.086 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:24:43.086 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.070787+0000", 2024-08-17T17:24:43.086 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000114813 2024-08-17T17:24:43.086 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.086 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.086 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:24:43.087 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.070897+0000", 2024-08-17T17:24:43.087 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000109479 2024-08-17T17:24:43.087 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.087 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.087 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:24:43.087 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.086255+0000", 2024-08-17T17:24:43.087 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.015358569000000001 2024-08-17T17:24:43.087 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.087 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.087 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.087 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.086318+0000", 2024-08-17T17:24:43.087 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2265000000000001e-05 2024-08-17T17:24:43.087 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.088 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.088 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.088 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.172271+0000", 2024-08-17T17:24:43.088 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.085953272999999997 2024-08-17T17:24:43.088 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.088 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.088 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:24:43.088 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.172304+0000", 2024-08-17T17:24:43.088 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2778999999999999e-05 2024-08-17T17:24:43.088 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.088 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.088 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:24:43.089 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:43.090 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 14 16 24332 24316 108.392 114.438 0.0048079 0.00921983 2024-08-17T17:24:43.091 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.173086+0000", 2024-08-17T17:24:43.091 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000782514 2024-08-17T17:24:43.091 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.091 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:24:43.091 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.091 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.091 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.091 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:155080 5.6 5:6a70446c:::benchmark_data_smithi003_140793_object53023:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-08-17T17:24:43.091 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:22:05.065178+0000", 2024-08-17T17:24:43.091 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 157.96300340499999, 2024-08-17T17:24:43.091 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.10913120599999999, 2024-08-17T17:24:43.092 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:24:43.092 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:43.092 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:24:43.092 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:24:43.092 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:43.092 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 155080 2024-08-17T17:24:43.092 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.092 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:24:43.092 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.092 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:24:43.092 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.065178+0000", 2024-08-17T17:24:43.092 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.093 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.093 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.093 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:24:43.093 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.065178+0000", 2024-08-17T17:24:43.093 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.093 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.093 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.093 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:24:43.093 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.065177+0000", 2024-08-17T17:24:43.093 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:24:43.093 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.093 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.094 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:24:43.094 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.065182+0000", 2024-08-17T17:24:43.094 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1839999999999998e-06 2024-08-17T17:24:43.094 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.094 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.094 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:24:43.094 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.065195+0000", 2024-08-17T17:24:43.094 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2863e-05 2024-08-17T17:24:43.094 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.094 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.094 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:24:43.094 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.065200+0000", 2024-08-17T17:24:43.095 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1660000000000002e-06 2024-08-17T17:24:43.095 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.095 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.095 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:24:43.095 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.076918+0000", 2024-08-17T17:24:43.095 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011717389999999999 2024-08-17T17:24:43.095 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.095 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.095 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:24:43.095 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.077329+0000", 2024-08-17T17:24:43.095 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00041130099999999998 2024-08-17T17:24:43.095 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.096 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.096 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:24:43.096 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.077478+0000", 2024-08-17T17:24:43.096 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014906500000000001 2024-08-17T17:24:43.096 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.096 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.096 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.096 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.086368+0000", 2024-08-17T17:24:43.096 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0088899359999999993 2024-08-17T17:24:43.096 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.096 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.096 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:24:43.096 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.086377+0000", 2024-08-17T17:24:43.096 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.7739999999999993e-06 2024-08-17T17:24:43.097 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.097 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.097 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.097 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.173163+0000", 2024-08-17T17:24:43.097 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.086786549000000004 2024-08-17T17:24:43.097 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.097 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.097 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:24:43.097 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.173233+0000", 2024-08-17T17:24:43.097 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9441000000000001e-05 2024-08-17T17:24:43.097 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.097 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.097 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:24:43.097 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:05.174309+0000", 2024-08-17T17:24:43.098 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0010763280000000001 2024-08-17T17:24:43.098 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.098 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:24:43.098 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.098 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.098 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.098 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:156849 5.6 5:641d2640:::benchmark_data_smithi003_140793_object54792:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-08-17T17:24:43.098 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:22:07.771252+0000", 2024-08-17T17:24:43.098 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 155.25692909099999, 2024-08-17T17:24:43.098 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.14347606600000001, 2024-08-17T17:24:43.098 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:24:43.098 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:43.099 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:24:43.099 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:24:43.099 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:43.099 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 156849 2024-08-17T17:24:43.099 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.099 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:24:43.099 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.099 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:24:43.099 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.771252+0000", 2024-08-17T17:24:43.099 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.099 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.099 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.099 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:24:43.100 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.771252+0000", 2024-08-17T17:24:43.100 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.100 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.100 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.100 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:24:43.100 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.771251+0000", 2024-08-17T17:24:43.100 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:24:43.100 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.100 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.100 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:24:43.100 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.771256+0000", 2024-08-17T17:24:43.100 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2660000000000003e-06 2024-08-17T17:24:43.100 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.101 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.101 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:24:43.101 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.771264+0000", 2024-08-17T17:24:43.101 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.6810000000000004e-06 2024-08-17T17:24:43.101 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.101 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.101 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:24:43.101 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.771269+0000", 2024-08-17T17:24:43.101 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9699999999999998e-06 2024-08-17T17:24:43.101 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.101 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.101 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:24:43.101 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.783234+0000", 2024-08-17T17:24:43.101 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011964519 2024-08-17T17:24:43.102 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.102 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.102 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:24:43.102 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.783358+0000", 2024-08-17T17:24:43.102 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000124344 2024-08-17T17:24:43.102 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.102 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.102 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:24:43.102 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.783476+0000", 2024-08-17T17:24:43.102 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000117602 2024-08-17T17:24:43.102 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.102 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.102 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:24:43.103 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.793480+0000", 2024-08-17T17:24:43.103 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010004733 2024-08-17T17:24:43.103 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.103 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.103 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.103 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.793512+0000", 2024-08-17T17:24:43.103 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1912999999999998e-05 2024-08-17T17:24:43.103 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.103 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.103 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.103 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.912983+0000", 2024-08-17T17:24:43.103 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.119470807 2024-08-17T17:24:43.103 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.104 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.104 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:24:43.104 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.913037+0000", 2024-08-17T17:24:43.104 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4163999999999998e-05 2024-08-17T17:24:43.104 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.104 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.104 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:24:43.104 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.914728+0000", 2024-08-17T17:24:43.104 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001690906 2024-08-17T17:24:43.104 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.104 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:24:43.104 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.105 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.105 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.105 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:156857 5.6 5:6bc4dee8:::benchmark_data_smithi003_140793_object54800:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-08-17T17:24:43.105 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:22:07.781800+0000", 2024-08-17T17:24:43.105 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 155.246380837, 2024-08-17T17:24:43.105 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.13373579599999999, 2024-08-17T17:24:43.105 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:24:43.105 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:43.105 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:24:43.105 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:24:43.105 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:43.105 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 156857 2024-08-17T17:24:43.105 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.106 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:24:43.106 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.106 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:24:43.106 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.781800+0000", 2024-08-17T17:24:43.106 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.106 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.106 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.106 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:24:43.106 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.781800+0000", 2024-08-17T17:24:43.106 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.106 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.106 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.106 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:24:43.107 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.781799+0000", 2024-08-17T17:24:43.107 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:24:43.107 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.107 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.107 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:24:43.107 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.781806+0000", 2024-08-17T17:24:43.107 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5169999999999996e-06 2024-08-17T17:24:43.107 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.107 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.107 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:24:43.107 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.781818+0000", 2024-08-17T17:24:43.107 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2832e-05 2024-08-17T17:24:43.107 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.108 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.108 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:24:43.108 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.781824+0000", 2024-08-17T17:24:43.108 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8810000000000001e-06 2024-08-17T17:24:43.108 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.108 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.108 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:24:43.108 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.788082+0000", 2024-08-17T17:24:43.108 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0062576890000000003 2024-08-17T17:24:43.108 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.108 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.108 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:24:43.108 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.788238+0000", 2024-08-17T17:24:43.108 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000155861 2024-08-17T17:24:43.109 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.109 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.109 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:24:43.109 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.788388+0000", 2024-08-17T17:24:43.109 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014992900000000001 2024-08-17T17:24:43.109 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.109 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.109 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.109 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.793538+0000", 2024-08-17T17:24:43.109 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0051501489999999997 2024-08-17T17:24:43.109 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.109 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.109 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:24:43.109 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.793943+0000", 2024-08-17T17:24:43.109 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00040507500000000002 2024-08-17T17:24:43.109 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.110 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.110 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.110 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.914772+0000", 2024-08-17T17:24:43.110 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.120829108 2024-08-17T17:24:43.110 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.110 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.110 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:24:43.110 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.914807+0000", 2024-08-17T17:24:43.110 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4433e-05 2024-08-17T17:24:43.110 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.110 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.110 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:24:43.110 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.915536+0000", 2024-08-17T17:24:43.111 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00072964999999999998 2024-08-17T17:24:43.111 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.111 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:24:43.111 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.111 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.111 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.111 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:156871 5.6 5:60780fd7:::benchmark_data_smithi003_140793_object54814:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-08-17T17:24:43.111 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:22:07.808826+0000", 2024-08-17T17:24:43.111 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 155.219355522, 2024-08-17T17:24:43.111 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.107544047, 2024-08-17T17:24:43.111 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:24:43.111 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:43.111 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:24:43.111 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:24:43.111 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:43.111 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 156871 2024-08-17T17:24:43.112 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.112 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:24:43.112 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.112 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:24:43.112 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.808826+0000", 2024-08-17T17:24:43.112 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.112 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.112 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.112 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:24:43.112 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.808826+0000", 2024-08-17T17:24:43.112 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.112 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.112 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.112 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:24:43.112 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.808825+0000", 2024-08-17T17:24:43.112 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:24:43.113 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.113 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.113 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:24:43.113 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.808830+0000", 2024-08-17T17:24:43.113 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3190000000000002e-06 2024-08-17T17:24:43.113 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.113 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.113 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:24:43.113 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.808841+0000", 2024-08-17T17:24:43.113 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0944999999999999e-05 2024-08-17T17:24:43.113 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.113 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.113 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:24:43.113 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.808845+0000", 2024-08-17T17:24:43.114 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2490000000000003e-06 2024-08-17T17:24:43.114 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.114 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.114 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:24:43.114 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.808897+0000", 2024-08-17T17:24:43.114 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1634000000000001e-05 2024-08-17T17:24:43.114 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.114 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.114 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:24:43.114 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.809059+0000", 2024-08-17T17:24:43.114 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016229499999999999 2024-08-17T17:24:43.114 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.114 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.114 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:24:43.114 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.809211+0000", 2024-08-17T17:24:43.115 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015180599999999999 2024-08-17T17:24:43.115 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.115 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.115 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:24:43.115 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.813638+0000", 2024-08-17T17:24:43.115 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0044272490000000003 2024-08-17T17:24:43.115 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.115 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.115 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.115 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.813805+0000", 2024-08-17T17:24:43.115 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000166532 2024-08-17T17:24:43.115 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.115 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.115 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.116 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.915598+0000", 2024-08-17T17:24:43.116 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.101793289 2024-08-17T17:24:43.116 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.116 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.116 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:24:43.116 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.915643+0000", 2024-08-17T17:24:43.116 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5089999999999997e-05 2024-08-17T17:24:43.116 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.116 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.116 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:24:43.116 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:07.916370+0000", 2024-08-17T17:24:43.116 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00072683700000000003 2024-08-17T17:24:43.116 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.116 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:24:43.116 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.117 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.117 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.117 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:177148 5.6 5:66ca71a4:::benchmark_data_smithi003_140793_object75091:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-08-17T17:24:43.117 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:22:38.248422+0000", 2024-08-17T17:24:43.117 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 124.779758977, 2024-08-17T17:24:43.117 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.12687539, 2024-08-17T17:24:43.117 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:24:43.117 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:43.117 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:24:43.117 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:24:43.117 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:43.117 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 177148 2024-08-17T17:24:43.117 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.117 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:24:43.117 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.118 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:24:43.118 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.248422+0000", 2024-08-17T17:24:43.118 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.118 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.118 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.118 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:24:43.118 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.248422+0000", 2024-08-17T17:24:43.118 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.118 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.118 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.118 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:24:43.118 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.248421+0000", 2024-08-17T17:24:43.118 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:24:43.119 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.119 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.119 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:24:43.119 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.248426+0000", 2024-08-17T17:24:43.119 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.327e-06 2024-08-17T17:24:43.119 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.119 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.119 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:24:43.119 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.248441+0000", 2024-08-17T17:24:43.119 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5129e-05 2024-08-17T17:24:43.119 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.119 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.119 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:24:43.119 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.248447+0000", 2024-08-17T17:24:43.119 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4129999999999997e-06 2024-08-17T17:24:43.120 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.120 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.120 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:24:43.120 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.256828+0000", 2024-08-17T17:24:43.120 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0083808540000000001 2024-08-17T17:24:43.120 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.120 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.120 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:24:43.120 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.256981+0000", 2024-08-17T17:24:43.120 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015285 2024-08-17T17:24:43.120 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.120 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.120 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:24:43.120 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.257148+0000", 2024-08-17T17:24:43.120 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000167734 2024-08-17T17:24:43.120 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.121 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.121 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:24:43.121 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.268412+0000", 2024-08-17T17:24:43.121 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011263904 2024-08-17T17:24:43.121 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.121 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.121 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.121 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.273101+0000", 2024-08-17T17:24:43.121 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0046885690000000001 2024-08-17T17:24:43.121 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.121 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.121 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.121 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.373183+0000", 2024-08-17T17:24:43.121 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.10008175599999999 2024-08-17T17:24:43.121 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.121 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.122 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:24:43.122 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.373285+0000", 2024-08-17T17:24:43.122 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000102056 2024-08-17T17:24:43.122 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.122 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.122 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:24:43.122 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.375298+0000", 2024-08-17T17:24:43.122 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0020131279999999999 2024-08-17T17:24:43.122 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.122 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:24:43.122 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.122 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.122 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.123 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:177149 5.6 5:6b6b407b:::benchmark_data_smithi003_140793_object75092:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-08-17T17:24:43.123 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:22:38.249577+0000", 2024-08-17T17:24:43.123 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 124.77860454, 2024-08-17T17:24:43.123 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.12896480900000001, 2024-08-17T17:24:43.123 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:24:43.123 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:43.123 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:24:43.123 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:24:43.123 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:43.123 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 177149 2024-08-17T17:24:43.123 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.123 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:24:43.123 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.123 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:24:43.123 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.249577+0000", 2024-08-17T17:24:43.124 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.124 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.124 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.124 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:24:43.124 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.249577+0000", 2024-08-17T17:24:43.124 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.124 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.124 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.124 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:24:43.124 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.249575+0000", 2024-08-17T17:24:43.124 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:24:43.124 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.124 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.124 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:24:43.125 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.249581+0000", 2024-08-17T17:24:43.125 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8619999999999999e-06 2024-08-17T17:24:43.125 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.125 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.125 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:24:43.125 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.249598+0000", 2024-08-17T17:24:43.125 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6537000000000001e-05 2024-08-17T17:24:43.125 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.125 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.125 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:24:43.125 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.249604+0000", 2024-08-17T17:24:43.125 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9610000000000002e-06 2024-08-17T17:24:43.125 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.126 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.126 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:24:43.126 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.263048+0000", 2024-08-17T17:24:43.126 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.013444335 2024-08-17T17:24:43.126 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.126 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.126 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:24:43.126 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.263195+0000", 2024-08-17T17:24:43.126 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014668 2024-08-17T17:24:43.127 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.127 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.127 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:24:43.127 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.263354+0000", 2024-08-17T17:24:43.127 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000159226 2024-08-17T17:24:43.127 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.127 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.127 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:24:43.127 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.274592+0000", 2024-08-17T17:24:43.127 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011238242000000001 2024-08-17T17:24:43.127 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.127 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.127 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.127 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.276653+0000", 2024-08-17T17:24:43.128 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0020602839999999999 2024-08-17T17:24:43.128 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.128 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.128 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.128 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.375437+0000", 2024-08-17T17:24:43.128 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.098784602999999999 2024-08-17T17:24:43.128 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.128 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.128 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:24:43.128 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.376703+0000", 2024-08-17T17:24:43.128 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0012655850000000001 2024-08-17T17:24:43.128 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.128 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.129 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:24:43.129 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.378542+0000", 2024-08-17T17:24:43.129 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0018388359999999999 2024-08-17T17:24:43.129 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.129 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:24:43.129 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.129 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.129 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.129 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:177150 5.6 5:6e12f559:::benchmark_data_smithi003_140793_object75093:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-08-17T17:24:43.129 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:22:38.249795+0000", 2024-08-17T17:24:43.129 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 124.778386715, 2024-08-17T17:24:43.129 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.132585068, 2024-08-17T17:24:43.129 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:24:43.129 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:43.130 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:24:43.130 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:24:43.130 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:43.130 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 177150 2024-08-17T17:24:43.130 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.130 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:24:43.130 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.130 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:24:43.130 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.249795+0000", 2024-08-17T17:24:43.130 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.130 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.130 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.130 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:24:43.130 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.249795+0000", 2024-08-17T17:24:43.131 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.131 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.131 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.131 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:24:43.131 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.249794+0000", 2024-08-17T17:24:43.131 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:24:43.131 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.131 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.131 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:24:43.131 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.249799+0000", 2024-08-17T17:24:43.131 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4020000000000001e-06 2024-08-17T17:24:43.131 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.131 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.132 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:24:43.132 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.249815+0000", 2024-08-17T17:24:43.132 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5911000000000001e-05 2024-08-17T17:24:43.132 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.132 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.132 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:24:43.132 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.249821+0000", 2024-08-17T17:24:43.132 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0979999999999999e-06 2024-08-17T17:24:43.132 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.132 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.132 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:24:43.132 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.268467+0000", 2024-08-17T17:24:43.132 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.018645922999999998 2024-08-17T17:24:43.132 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.133 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.133 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:24:43.133 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.268660+0000", 2024-08-17T17:24:43.133 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000193382 2024-08-17T17:24:43.133 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.133 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.133 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:24:43.133 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.268872+0000", 2024-08-17T17:24:43.133 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021176999999999999 2024-08-17T17:24:43.133 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.133 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.133 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:24:43.133 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.276599+0000", 2024-08-17T17:24:43.133 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.007727093 2024-08-17T17:24:43.133 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.134 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.134 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.134 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.276686+0000", 2024-08-17T17:24:43.134 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.6779000000000004e-05 2024-08-17T17:24:43.134 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.134 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.134 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.134 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.380615+0000", 2024-08-17T17:24:43.134 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.103928567 2024-08-17T17:24:43.134 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.134 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.134 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:24:43.134 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.380656+0000", 2024-08-17T17:24:43.134 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1127000000000001e-05 2024-08-17T17:24:43.135 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.135 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.135 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:24:43.135 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:38.382380+0000", 2024-08-17T17:24:43.135 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001723877 2024-08-17T17:24:43.135 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.135 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:24:43.135 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.135 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.135 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.135 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:181624 5.6 5:6597c837:::benchmark_data_smithi003_140793_object79567:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159)", 2024-08-17T17:24:43.135 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:22:44.806093+0000", 2024-08-17T17:24:43.135 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 118.22208783799999, 2024-08-17T17:24:43.136 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.11199959600000001, 2024-08-17T17:24:43.136 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:24:43.136 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:43.136 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:24:43.136 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:24:43.136 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:43.136 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 181624 2024-08-17T17:24:43.136 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.136 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:24:43.136 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.136 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:24:43.136 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:44.806093+0000", 2024-08-17T17:24:43.136 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.136 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.137 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.137 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:24:43.137 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:44.806093+0000", 2024-08-17T17:24:43.137 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.137 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.137 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.137 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:24:43.137 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:44.806093+0000", 2024-08-17T17:24:43.137 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:24:43.137 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.137 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.137 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:24:43.137 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:44.806095+0000", 2024-08-17T17:24:43.137 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3819999999999998e-06 2024-08-17T17:24:43.137 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.138 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.138 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:24:43.138 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:44.806103+0000", 2024-08-17T17:24:43.138 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.2180000000000002e-06 2024-08-17T17:24:43.138 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.138 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.138 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:24:43.138 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:44.806106+0000", 2024-08-17T17:24:43.138 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.241e-06 2024-08-17T17:24:43.138 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.138 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.138 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:24:43.138 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:44.806311+0000", 2024-08-17T17:24:43.139 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00020495100000000001 2024-08-17T17:24:43.139 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.139 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.139 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:24:43.139 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:44.806422+0000", 2024-08-17T17:24:43.139 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011066200000000001 2024-08-17T17:24:43.139 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.139 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.139 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:24:43.139 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:44.806554+0000", 2024-08-17T17:24:43.139 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013239099999999999 2024-08-17T17:24:43.139 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.139 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.140 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.140 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:44.816359+0000", 2024-08-17T17:24:43.140 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0098046690000000002 2024-08-17T17:24:43.140 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.140 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.140 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:24:43.140 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:44.816367+0000", 2024-08-17T17:24:43.140 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.7420000000000003e-06 2024-08-17T17:24:43.140 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.140 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.140 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.140 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:44.916543+0000", 2024-08-17T17:24:43.140 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.100175653 2024-08-17T17:24:43.140 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.141 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.141 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:24:43.141 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:44.916614+0000", 2024-08-17T17:24:43.141 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.1236999999999996e-05 2024-08-17T17:24:43.141 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.141 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.141 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:24:43.141 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:44.918093+0000", 2024-08-17T17:24:43.141 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0014788119999999999 2024-08-17T17:24:43.141 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.141 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:24:43.141 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.141 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.141 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.141 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:181626 5.6 5:6490ce51:::benchmark_data_smithi003_140793_object79569:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159)", 2024-08-17T17:24:43.142 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:22:44.809326+0000", 2024-08-17T17:24:43.142 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 118.21885527800001, 2024-08-17T17:24:43.142 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.112102225, 2024-08-17T17:24:43.142 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:24:43.142 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:43.142 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:24:43.142 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:24:43.142 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:24:43.142 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 181626 2024-08-17T17:24:43.142 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.142 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:24:43.142 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.142 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:24:43.142 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:44.809326+0000", 2024-08-17T17:24:43.143 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.143 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.143 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.143 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:24:43.143 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:44.809326+0000", 2024-08-17T17:24:43.143 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.143 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.143 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.143 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:24:43.143 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:44.809325+0000", 2024-08-17T17:24:43.143 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:24:43.143 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.143 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.143 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:24:43.144 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:44.809331+0000", 2024-08-17T17:24:43.144 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2199999999999997e-06 2024-08-17T17:24:43.144 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.144 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.144 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:24:43.144 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:44.809345+0000", 2024-08-17T17:24:43.144 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3953e-05 2024-08-17T17:24:43.144 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.144 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.144 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:24:43.144 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:44.809352+0000", 2024-08-17T17:24:43.144 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9650000000000002e-06 2024-08-17T17:24:43.144 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.145 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.145 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:24:43.145 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:44.811329+0000", 2024-08-17T17:24:43.145 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0019762130000000001 2024-08-17T17:24:43.145 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.145 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.145 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:24:43.145 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:44.811459+0000", 2024-08-17T17:24:43.145 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013042699999999999 2024-08-17T17:24:43.145 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.145 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.145 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:24:43.145 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:44.811591+0000", 2024-08-17T17:24:43.145 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013217099999999999 2024-08-17T17:24:43.146 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.146 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.146 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:24:43.146 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:44.817055+0000", 2024-08-17T17:24:43.146 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0054633900000000003 2024-08-17T17:24:43.146 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.146 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.146 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.146 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:44.817701+0000", 2024-08-17T17:24:43.146 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00064659500000000003 2024-08-17T17:24:43.146 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.146 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.147 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.147 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:44.919901+0000", 2024-08-17T17:24:43.147 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.102199866 2024-08-17T17:24:43.147 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.147 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.147 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:24:43.147 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:44.919957+0000", 2024-08-17T17:24:43.147 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6164e-05 2024-08-17T17:24:43.147 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.147 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.147 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:24:43.147 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:44.921428+0000", 2024-08-17T17:24:43.147 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0014711139999999999 2024-08-17T17:24:43.147 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.148 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:24:43.148 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.148 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.148 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.148 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:21573 6.7 6:e497cb3d:::benchmark_data_smithi003_175043_object21572:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-17T17:24:43.148 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:24:41.500822+0000", 2024-08-17T17:24:43.148 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 1.5273596970000001, 2024-08-17T17:24:43.148 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.14131363699999999, 2024-08-17T17:24:43.148 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:24:43.148 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:43.148 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:24:43.148 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:24:43.148 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:24:43.148 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 21573 2024-08-17T17:24:43.149 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.149 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:24:43.149 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.149 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:24:43.149 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:41.500822+0000", 2024-08-17T17:24:43.149 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.149 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.149 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.149 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:24:43.149 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:41.500822+0000", 2024-08-17T17:24:43.149 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.149 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.149 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.150 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:24:43.150 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:41.500820+0000", 2024-08-17T17:24:43.150 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:24:43.150 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.150 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.150 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:24:43.150 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:41.500875+0000", 2024-08-17T17:24:43.150 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4910000000000001e-05 2024-08-17T17:24:43.150 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.150 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.150 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:24:43.150 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:41.500887+0000", 2024-08-17T17:24:43.150 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2235999999999999e-05 2024-08-17T17:24:43.151 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.151 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.151 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:24:43.151 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:41.500893+0000", 2024-08-17T17:24:43.151 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6930000000000004e-06 2024-08-17T17:24:43.151 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.151 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.151 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:24:43.151 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:41.500949+0000", 2024-08-17T17:24:43.151 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5719e-05 2024-08-17T17:24:43.151 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.151 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.152 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:24:43.152 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:41.501051+0000", 2024-08-17T17:24:43.152 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010171900000000001 2024-08-17T17:24:43.152 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.152 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.152 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:24:43.152 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:41.501135+0000", 2024-08-17T17:24:43.152 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.4889000000000007e-05 2024-08-17T17:24:43.152 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.152 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.152 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:24:43.152 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:41.503013+0000", 2024-08-17T17:24:43.152 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001877802 2024-08-17T17:24:43.152 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.153 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.153 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.153 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:41.503667+0000", 2024-08-17T17:24:43.153 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00065350599999999999 2024-08-17T17:24:43.153 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.153 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.153 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.153 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:41.642050+0000", 2024-08-17T17:24:43.153 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.138383434 2024-08-17T17:24:43.153 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.153 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.153 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:24:43.153 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:41.642099+0000", 2024-08-17T17:24:43.153 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9157999999999999e-05 2024-08-17T17:24:43.154 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.154 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.154 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:24:43.154 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:41.642135+0000", 2024-08-17T17:24:43.154 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5890999999999999e-05 2024-08-17T17:24:43.154 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.154 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:24:43.154 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.154 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.154 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.154 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:21574 6.7 6:ee0fa0b8:::benchmark_data_smithi003_175043_object21573:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-08-17T17:24:43.155 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:24:41.500899+0000", 2024-08-17T17:24:43.155 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 1.5272819069999999, 2024-08-17T17:24:43.155 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.14788504799999999, 2024-08-17T17:24:43.155 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:24:43.155 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:24:43.155 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:24:43.155 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:24:43.155 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:24:43.155 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 21574 2024-08-17T17:24:43.155 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.155 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:24:43.155 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.155 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:24:43.155 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:41.500899+0000", 2024-08-17T17:24:43.156 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.156 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.156 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.156 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:24:43.156 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:41.500899+0000", 2024-08-17T17:24:43.156 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:43.156 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.156 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.156 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:24:43.156 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:41.500899+0000", 2024-08-17T17:24:43.156 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:24:43.156 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.156 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.156 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:24:43.156 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:41.500939+0000", 2024-08-17T17:24:43.157 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0442999999999999e-05 2024-08-17T17:24:43.157 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.157 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.157 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:24:43.157 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:41.500946+0000", 2024-08-17T17:24:43.157 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1890000000000004e-06 2024-08-17T17:24:43.157 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.157 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.157 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:24:43.157 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:41.500950+0000", 2024-08-17T17:24:43.157 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5379999999999996e-06 2024-08-17T17:24:43.157 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.157 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.157 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:24:43.158 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:41.501984+0000", 2024-08-17T17:24:43.158 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0010338109999999999 2024-08-17T17:24:43.158 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.158 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.158 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:24:43.158 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:41.502115+0000", 2024-08-17T17:24:43.158 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013102099999999999 2024-08-17T17:24:43.158 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.158 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.158 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:24:43.158 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:41.502271+0000", 2024-08-17T17:24:43.158 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015559499999999999 2024-08-17T17:24:43.158 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.159 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.159 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:24:43.159 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:41.503140+0000", 2024-08-17T17:24:43.159 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00086966599999999995 2024-08-17T17:24:43.159 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.159 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.159 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.159 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:41.503861+0000", 2024-08-17T17:24:43.159 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00072092599999999999 2024-08-17T17:24:43.159 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.159 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.159 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:43.159 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:41.648733+0000", 2024-08-17T17:24:43.160 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.14487227699999999 2024-08-17T17:24:43.160 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.160 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.160 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:24:43.160 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:41.648765+0000", 2024-08-17T17:24:43.160 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1313000000000003e-05 2024-08-17T17:24:43.160 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:43.160 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:43.160 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:24:43.160 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:41.648784+0000", 2024-08-17T17:24:43.160 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9675000000000001e-05 2024-08-17T17:24:43.160 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.160 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:24:43.160 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.160 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:43.161 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:24:43.161 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:24:43.188 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:43.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:43.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:43.290 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:43.332 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:43.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:43.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:43.434 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:43.460 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:43.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:43.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:43.562 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:43.592 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:43.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:43.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:43.694 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:43.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:43 smithi053 ceph-mon[89963]: pgmap v466: 89 pgs: 89 active+clean; 1.1 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 147 MiB/s wr, 2.35k op/s; 4.5 MiB/s, 72 objects/s recovering 2024-08-17T17:24:43.723 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:43.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:43.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:43.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:43 smithi003 ceph-mon[100615]: pgmap v466: 89 pgs: 89 active+clean; 1.1 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 147 MiB/s wr, 2.35k op/s; 4.5 MiB/s, 72 objects/s recovering 2024-08-17T17:24:43.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:43 smithi003 ceph-mon[103344]: pgmap v466: 89 pgs: 89 active+clean; 1.1 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 147 MiB/s wr, 2.35k op/s; 4.5 MiB/s, 72 objects/s recovering 2024-08-17T17:24:43.825 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:43.876 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:43.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:43.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:43.886 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 5 1 2024-08-17T17:24:43.978 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:44.029 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:44.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:44.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:44.070 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-17T17:24:44.076 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 15 15 26299 26284 109.364 123 0.0114919 0.00913764 2024-08-17T17:24:44.131 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:44.161 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:44.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:44.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:44.229 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:24:44.263 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:44.288 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:44.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:44.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:44.390 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:44.423 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:24:44.430 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:44.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:44.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:44.532 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:44.549 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:24:44.549 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:24:44.559 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:44.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:44.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:44.589 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:24:44.589 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:24:44.627 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:24:44.627 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:24:44.661 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:44.671 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:24:44.671 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:24:44.690 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:44.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:44.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:44.701 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:24:44.701 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:24:44.734 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:24:44.734 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:24:44.764 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:24:44.765 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:24:44.792 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:44.798 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:24:44.798 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:24:44.833 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:44.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:44.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:44.935 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:44.978 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:44.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:44.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:45.077 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 16 16 28056 28040 109.387 109.75 0.00823534 0.0091355 2024-08-17T17:24:45.083 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:45.111 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:45.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:45.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:45.213 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:45.251 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:45.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:45.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:45.353 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:45.394 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:45.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:45.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:45.496 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:45.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:45 smithi003 ceph-mon[100615]: pgmap v467: 89 pgs: 89 active+clean; 1.3 GiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 119 MiB/s wr, 1.90k op/s; 3.3 MiB/s, 53 objects/s recovering 2024-08-17T17:24:45.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:45 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3647787443' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:24:45.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:45 smithi003 ceph-mon[103344]: pgmap v467: 89 pgs: 89 active+clean; 1.3 GiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 119 MiB/s wr, 1.90k op/s; 3.3 MiB/s, 53 objects/s recovering 2024-08-17T17:24:45.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:45 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3647787443' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:24:45.537 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:45.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:45.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:45.639 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:45.659 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:45 smithi053 ceph-mon[89963]: pgmap v467: 89 pgs: 89 active+clean; 1.3 GiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 119 MiB/s wr, 1.90k op/s; 3.3 MiB/s, 53 objects/s recovering 2024-08-17T17:24:45.659 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:45 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3647787443' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:24:45.667 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:45.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:45.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:45.769 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:45.796 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:45.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:45.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:45.872 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:24:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:24:45] "GET /metrics HTTP/1.1" 200 36555 "" "Prometheus/2.51.0" 2024-08-17T17:24:45.897 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:45.923 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:45.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:45.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:46.025 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:46.072 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:46.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:46.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:46.077 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 17 16 29888 29872 109.685 114.5 0.00575943 0.00910841 2024-08-17T17:24:46.174 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:46.213 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:46.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:46.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:46.315 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:46.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:46.356 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:46.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:46.457 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:46.499 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:46.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:46.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:46.600 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:46.626 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:46.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:46.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:46.728 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:46.755 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:46.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:46.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:46.857 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:46.881 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:46.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:46.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:46.982 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:47.013 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:47.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:47.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:47.078 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 18 16 31398 31382 108.835 94.375 0.00888739 0.00917949 2024-08-17T17:24:47.084 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:24:47.115 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:47.140 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:47.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:47.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:47.188 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:24:47.189 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:24:47.242 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:47.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:47.291 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:47.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:47.392 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:47.417 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:47.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:47.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:47.519 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:47.556 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:47.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:47.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:47.658 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:47.685 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:47.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:47.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:47.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:47 smithi053 ceph-mon[89963]: pgmap v468: 89 pgs: 89 active+clean; 1.5 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 106 MiB/s wr, 1.70k op/s; 2.8 MiB/s, 45 objects/s recovering 2024-08-17T17:24:47.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:47 smithi003 ceph-mon[103344]: pgmap v468: 89 pgs: 89 active+clean; 1.5 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 106 MiB/s wr, 1.70k op/s; 2.8 MiB/s, 45 objects/s recovering 2024-08-17T17:24:47.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:47 smithi003 ceph-mon[100615]: pgmap v468: 89 pgs: 89 active+clean; 1.5 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 106 MiB/s wr, 1.70k op/s; 2.8 MiB/s, 45 objects/s recovering 2024-08-17T17:24:47.787 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:47.813 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:47.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:47.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:47.915 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:47.941 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:47.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:47.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:47.990 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-17T17:24:48.043 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:48.066 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:48.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:48.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:48.077 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 19 16 32804 32788 107.732 87.875 0.00426674 0.00926109 2024-08-17T17:24:48.169 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:48.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:48.199 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:48.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:48.300 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:48.325 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:24:48.330 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:48.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:48.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:48.432 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:48.438 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-08-17T17:24:48.460 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:48.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:48.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:48.492 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:24:48.562 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:48.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:48.607 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:48.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:48.708 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:48.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:48 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3479004036' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:24:48.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:48 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:24:48.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:48 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:24:48.753 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:48.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:48.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:48.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:48 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3479004036' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:24:48.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:48 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:24:48.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:48 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:24:48.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:48 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3479004036' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:24:48.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:48 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:24:48.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:48 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:24:48.854 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:48.884 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:48.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:48.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:48.986 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:49.015 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:49.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:49.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:49.077 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-08-17T17:24:49.077719+0000 min lat: 0.00112466 max lat: 2.58819 avg lat: 0.0092755 2024-08-17T17:24:49.077 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:24:49.077 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 20 16 34533 34517 107.748 108.062 0.0047314 0.0092755 2024-08-17T17:24:49.117 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:49.149 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:49.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:49.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:49.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:49.277 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:49.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:49.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:49.379 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:49.419 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:49.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:49.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:49.521 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:49.544 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:49.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:49.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:49.647 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:49.680 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:49 smithi053 ceph-mon[89963]: pgmap v469: 89 pgs: 89 active+clean; 1.7 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 109 MiB/s wr, 1.74k op/s; 2.3 MiB/s, 36 objects/s recovering 2024-08-17T17:24:49.680 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:49 smithi053 ceph-mon[89963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:24:49.680 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:49 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:24:49.680 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:49 smithi053 ceph-mon[89963]: osdmap e192: 8 total, 8 up, 7 in 2024-08-17T17:24:49.684 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:49.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:49.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:49.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:49 smithi003 ceph-mon[103344]: pgmap v469: 89 pgs: 89 active+clean; 1.7 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 109 MiB/s wr, 1.74k op/s; 2.3 MiB/s, 36 objects/s recovering 2024-08-17T17:24:49.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:49 smithi003 ceph-mon[103344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:24:49.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:49 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:24:49.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:49 smithi003 ceph-mon[103344]: osdmap e192: 8 total, 8 up, 7 in 2024-08-17T17:24:49.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:49 smithi003 ceph-mon[100615]: pgmap v469: 89 pgs: 89 active+clean; 1.7 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 109 MiB/s wr, 1.74k op/s; 2.3 MiB/s, 36 objects/s recovering 2024-08-17T17:24:49.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:49 smithi003 ceph-mon[100615]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:24:49.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:49 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:24:49.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:49 smithi003 ceph-mon[100615]: osdmap e192: 8 total, 8 up, 7 in 2024-08-17T17:24:49.785 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:49.810 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:49.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:49.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:49.912 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:49.940 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:49.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:49.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:50.042 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:50.064 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:50.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:50.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:50.077 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 21 16 35959 35943 106.862 89.125 0.00225528 0.00933832 2024-08-17T17:24:50.147 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:24:50.147 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:24:50.167 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:50.193 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:50.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:50.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:50.295 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:50.322 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:50.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:50.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:50.424 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:50.459 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:50.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:50.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:50.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:24:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:24:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:24:50.561 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:50.598 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:50.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:50.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:50.700 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:50.740 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:50.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:50.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:50.825 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-17T17:24:50.841 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:50.888 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:50.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:50.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:50.990 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:51.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:51.018 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:51.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:51.077 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 22 16 37644 37628 106.791 105.312 0.00582246 0.00935839 2024-08-17T17:24:51.119 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:51.157 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:51.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:51.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:51.169 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:24:51.258 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:51.285 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:51.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:51.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:51.387 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:51.412 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:51.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:51.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:51.514 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:51.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:51 smithi003 ceph-mon[100615]: pgmap v471: 89 pgs: 89 active+clean; 1.9 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 93 MiB/s wr, 1.49k op/s; 0 B/s, 0 objects/s recovering 2024-08-17T17:24:51.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:51 smithi003 ceph-mon[103344]: pgmap v471: 89 pgs: 89 active+clean; 1.9 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 93 MiB/s wr, 1.49k op/s; 0 B/s, 0 objects/s recovering 2024-08-17T17:24:51.549 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:51 smithi053 ceph-mon[89963]: pgmap v471: 89 pgs: 89 active+clean; 1.9 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 93 MiB/s wr, 1.49k op/s; 0 B/s, 0 objects/s recovering 2024-08-17T17:24:51.550 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:51.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:51.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:51.652 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:51.691 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:51.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:51.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:51.793 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:51.817 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:24:51.838 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:51.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:51.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:51.940 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:51.966 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:51.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:51.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:52.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:52.079 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 23 16 39145 39129 106.221 93.8125 0.010316 0.00940749 2024-08-17T17:24:52.118 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:52.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:52.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:52.221 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:52.262 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:52.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:52.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:52.364 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:52.393 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:52.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:52.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:52.453 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.5 to 1 (10000) 2024-08-17T17:24:52.495 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:52.517 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:52.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:52.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:52.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:52 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2913430411' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:24:52.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:52 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2913430411' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:24:52.618 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:24:52.640 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:52.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:52.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:24:52.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:52 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2913430411' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:24:52.742 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:52.764 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:52.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:52.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:52.865 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:52.888 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:52.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:52.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:52.984 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:24:52.984 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:24:52.991 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:53.016 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:53.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:53.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:53.079 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 24 16 39758 39742 103.394 38.3125 0.00247352 0.00942779 2024-08-17T17:24:53.118 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:53.141 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:53.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:53.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:53.242 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:53.244 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:24:53.264 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:53.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:53.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:53.366 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:53.388 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:53.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:53.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:53.489 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:53.512 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:53.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:53.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:53.604 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-17T17:24:53.614 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:53.670 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:53.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:53.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:53.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:53 smithi053 ceph-mon[89963]: pgmap v472: 89 pgs: 89 active+clean; 2.2 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 107 MiB/s wr, 1.71k op/s; 0 B/s, 0 objects/s recovering 2024-08-17T17:24:53.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:53 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2913430411' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-17T17:24:53.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:53 smithi053 ceph-mon[89963]: osdmap e193: 8 total, 8 up, 7 in 2024-08-17T17:24:53.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:24:53.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:24:53.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:24:53.772 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:53 smithi003 ceph-mon[100615]: pgmap v472: 89 pgs: 89 active+clean; 2.2 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 107 MiB/s wr, 1.71k op/s; 0 B/s, 0 objects/s recovering 2024-08-17T17:24:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:53 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2913430411' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-17T17:24:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:53 smithi003 ceph-mon[100615]: osdmap e193: 8 total, 8 up, 7 in 2024-08-17T17:24:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:24:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:24:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:24:53.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:53 smithi003 ceph-mon[103344]: pgmap v472: 89 pgs: 89 active+clean; 2.2 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 107 MiB/s wr, 1.71k op/s; 0 B/s, 0 objects/s recovering 2024-08-17T17:24:53.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:53 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2913430411' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-17T17:24:53.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:53 smithi003 ceph-mon[103344]: osdmap e193: 8 total, 8 up, 7 in 2024-08-17T17:24:53.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:24:53.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:24:53.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:24:53.795 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:53.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:53.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:53.870 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 6 1 2024-08-17T17:24:53.897 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:53.939 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:53.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:53.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:54.041 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:54.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:54.066 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:54.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:54.079 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 25 16 39758 39742 99.2616 0 - 0.00942779 2024-08-17T17:24:54.166 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:54.186 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:24:54.194 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:54.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:54.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:54.296 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:54.318 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:54.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:54.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:54.389 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:24:54.390 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:24:54.409 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:24:54.409 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:24:54.419 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:54.425 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:24:54.426 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:24:54.442 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:24:54.442 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:24:54.444 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:54.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:54.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:54.448 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:24:54.463 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:24:54.463 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:24:54.494 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:24:54.494 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:24:54.522 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:24:54.522 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:24:54.539 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:24:54.539 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:24:54.547 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:54.569 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:54.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:54.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:54.670 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:54.693 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:54.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:54.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:54.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:54 smithi003 ceph-mon[100615]: osdmap e194: 8 total, 8 up, 7 in 2024-08-17T17:24:54.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:54 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/118388390' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:24:54.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:54 smithi003 ceph-mon[100615]: osdmap e195: 8 total, 8 up, 7 in 2024-08-17T17:24:54.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:54 smithi003 ceph-mon[103344]: osdmap e194: 8 total, 8 up, 7 in 2024-08-17T17:24:54.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:54 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/118388390' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:24:54.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:54 smithi003 ceph-mon[103344]: osdmap e195: 8 total, 8 up, 7 in 2024-08-17T17:24:54.796 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:54.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:54.829 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:54.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:54.848 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:54 smithi053 ceph-mon[89963]: osdmap e194: 8 total, 8 up, 7 in 2024-08-17T17:24:54.848 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:54 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/118388390' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:24:54.848 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:54 smithi053 ceph-mon[89963]: osdmap e195: 8 total, 8 up, 7 in 2024-08-17T17:24:54.930 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:54.971 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:54.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:54.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:55.073 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:55.079 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 26 16 40012 39996 96.057 7.9375 0.0237345 0.0104023 2024-08-17T17:24:55.107 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:55.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:55.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:55.209 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:55.239 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:55.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:55.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:55.341 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:55.370 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:55.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:55.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:55.473 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:55.504 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:55.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:55.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:55.606 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:55.632 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:55 smithi053 ceph-mon[89963]: pgmap v475: 89 pgs: 89 active+clean; 2.2 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 97 MiB/s wr, 1.55k op/s; 0 B/s, 0 objects/s recovering 2024-08-17T17:24:55.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:55.636 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:55.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:55.737 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:55.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:55 smithi003 ceph-mon[103344]: pgmap v475: 89 pgs: 89 active+clean; 2.2 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 97 MiB/s wr, 1.55k op/s; 0 B/s, 0 objects/s recovering 2024-08-17T17:24:55.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:55 smithi003 ceph-mon[100615]: pgmap v475: 89 pgs: 89 active+clean; 2.2 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 97 MiB/s wr, 1.55k op/s; 0 B/s, 0 objects/s recovering 2024-08-17T17:24:55.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:24:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:24:55] "GET /metrics HTTP/1.1" 200 36522 "" "Prometheus/2.51.0" 2024-08-17T17:24:55.778 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:55.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:55.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:55.880 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:55.913 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:55.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:55.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:56.015 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:56.044 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:56.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:56.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:56.079 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 27 16 41495 41479 95.932 92.6875 0.00931382 0.0104174 2024-08-17T17:24:56.146 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:56.178 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:56.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:56.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:56.280 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:56.319 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:56.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:56.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:56.421 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:56.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:56.461 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:56.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:56.562 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:56.588 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:56.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:56.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:56.690 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:56.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:56 smithi053 ceph-mon[89963]: Health check failed: Degraded data redundancy: 4956/116592 objects degraded (4.251%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:24:56.727 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:56.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:56.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:56.829 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:56.868 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:56.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:56.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:56.970 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:57.009 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:57.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:57.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:57.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:56 smithi003 ceph-mon[103344]: Health check failed: Degraded data redundancy: 4956/116592 objects degraded (4.251%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:24:57.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:56 smithi003 ceph-mon[100615]: Health check failed: Degraded data redundancy: 4956/116592 objects degraded (4.251%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:24:57.043 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:24:57.043 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2024-08-17T17:24:57.044 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.044 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.45491.0:42730 6.2 e195/178 6:48c5f616:::benchmark_data_smithi003_175043_object42729:head v 195'2832, mlcod=195'2830)", 2024-08-17T17:24:57.044 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:24:57.037715+0000", 2024-08-17T17:24:57.044 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.0028556190000000002, 2024-08-17T17:24:57.044 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0028839360000000001, 2024-08-17T17:24:57.044 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:24:57.044 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2024-08-17T17:24:57.044 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:24:57.044 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.044 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:24:57.044 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.037715+0000", 2024-08-17T17:24:57.044 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:57.044 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.044 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.044 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:24:57.045 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.037715+0000", 2024-08-17T17:24:57.045 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:57.045 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.045 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.045 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:24:57.045 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.037715+0000", 2024-08-17T17:24:57.045 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:24:57.045 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.045 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.045 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:24:57.045 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.037761+0000", 2024-08-17T17:24:57.045 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5809000000000001e-05 2024-08-17T17:24:57.045 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.045 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.045 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:24:57.046 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.037769+0000", 2024-08-17T17:24:57.046 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1310000000000006e-06 2024-08-17T17:24:57.046 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.046 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.046 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:24:57.046 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.037774+0000", 2024-08-17T17:24:57.046 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9609999999999996e-06 2024-08-17T17:24:57.046 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.046 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.046 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:24:57.046 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.037844+0000", 2024-08-17T17:24:57.046 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9641999999999995e-05 2024-08-17T17:24:57.046 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.047 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.047 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:24:57.047 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.037861+0000", 2024-08-17T17:24:57.047 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7078000000000001e-05 2024-08-17T17:24:57.047 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:57.047 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:24:57.047 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:57.047 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.047 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.047 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:42734 6.8 6:17ec0a72:::benchmark_data_smithi003_175043_object42733:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-08-17T17:24:57.047 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:24:57.037801+0000", 2024-08-17T17:24:57.047 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.0027698850000000001, 2024-08-17T17:24:57.047 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002867915, 2024-08-17T17:24:57.048 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:24:57.048 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:24:57.048 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:24:57.048 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:24:57.048 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:24:57.048 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 42734 2024-08-17T17:24:57.048 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.048 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:24:57.048 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.048 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:24:57.048 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.037801+0000", 2024-08-17T17:24:57.048 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:57.048 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.048 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.049 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:24:57.049 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.037801+0000", 2024-08-17T17:24:57.049 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:57.049 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.049 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.049 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:24:57.049 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.037799+0000", 2024-08-17T17:24:57.049 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:24:57.049 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.049 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.049 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:24:57.049 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.037835+0000", 2024-08-17T17:24:57.049 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5434e-05 2024-08-17T17:24:57.049 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.049 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.050 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:24:57.050 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.037849+0000", 2024-08-17T17:24:57.050 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3685e-05 2024-08-17T17:24:57.050 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.050 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.050 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:24:57.050 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.037854+0000", 2024-08-17T17:24:57.050 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.874e-06 2024-08-17T17:24:57.050 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.050 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.050 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:24:57.050 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.037913+0000", 2024-08-17T17:24:57.050 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8941999999999999e-05 2024-08-17T17:24:57.051 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.051 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.051 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:24:57.051 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.038057+0000", 2024-08-17T17:24:57.051 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014404700000000001 2024-08-17T17:24:57.051 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.051 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.051 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,3", 2024-08-17T17:24:57.051 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.038220+0000", 2024-08-17T17:24:57.051 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016210500000000001 2024-08-17T17:24:57.051 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.051 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.051 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:24:57.052 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.039969+0000", 2024-08-17T17:24:57.052 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001749879 2024-08-17T17:24:57.052 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.052 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.052 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:24:57.052 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.040566+0000", 2024-08-17T17:24:57.052 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00059608100000000004 2024-08-17T17:24:57.052 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:57.052 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:24:57.052 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:57.052 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.052 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.052 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.45491.0:42736 6.b e195/178 6:d429d27b:::benchmark_data_smithi003_175043_object42735:head v 195'2675, mlcod=195'2673)", 2024-08-17T17:24:57.052 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:24:57.039801+0000", 2024-08-17T17:24:57.053 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.00077014200000000003, 2024-08-17T17:24:57.053 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00094643699999999995, 2024-08-17T17:24:57.053 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:24:57.053 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2024-08-17T17:24:57.053 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:24:57.053 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.053 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:24:57.053 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.039801+0000", 2024-08-17T17:24:57.053 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:57.053 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.053 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.053 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:24:57.053 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.039801+0000", 2024-08-17T17:24:57.053 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:57.053 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.054 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.054 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:24:57.054 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.039800+0000", 2024-08-17T17:24:57.054 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:24:57.054 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.054 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.054 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:24:57.054 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.039844+0000", 2024-08-17T17:24:57.054 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3643000000000002e-05 2024-08-17T17:24:57.054 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.054 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.054 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:24:57.054 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.039858+0000", 2024-08-17T17:24:57.055 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.437e-05 2024-08-17T17:24:57.055 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.055 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.055 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:24:57.055 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.039864+0000", 2024-08-17T17:24:57.055 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0599999999999996e-06 2024-08-17T17:24:57.055 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.055 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.055 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:24:57.055 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.039980+0000", 2024-08-17T17:24:57.055 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000115482 2024-08-17T17:24:57.055 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.055 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.055 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:24:57.055 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.040001+0000", 2024-08-17T17:24:57.056 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0883999999999999e-05 2024-08-17T17:24:57.056 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:57.056 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:24:57.056 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:57.056 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.056 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.056 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.45491.0:42735 6.6 e195/194 6:65b84875:::benchmark_data_smithi003_175043_object42734:head v 195'2574, mlcod=0'0)", 2024-08-17T17:24:57.056 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:24:57.040109+0000", 2024-08-17T17:24:57.056 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.00046258200000000001, 2024-08-17T17:24:57.056 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00071460800000000004, 2024-08-17T17:24:57.056 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:24:57.056 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2024-08-17T17:24:57.056 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:24:57.056 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.056 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:24:57.057 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.040109+0000", 2024-08-17T17:24:57.057 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:57.057 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.057 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.057 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:24:57.057 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.040109+0000", 2024-08-17T17:24:57.057 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:57.057 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.057 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.057 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:24:57.057 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.040108+0000", 2024-08-17T17:24:57.057 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:24:57.057 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.057 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.057 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:24:57.058 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.040162+0000", 2024-08-17T17:24:57.058 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3848999999999999e-05 2024-08-17T17:24:57.058 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.058 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.058 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:24:57.058 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.040174+0000", 2024-08-17T17:24:57.058 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2303000000000001e-05 2024-08-17T17:24:57.058 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.058 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.058 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:24:57.058 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.040180+0000", 2024-08-17T17:24:57.058 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4439999999999999e-06 2024-08-17T17:24:57.058 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.058 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.059 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:24:57.059 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.040223+0000", 2024-08-17T17:24:57.059 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3392000000000001e-05 2024-08-17T17:24:57.059 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.059 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.059 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:24:57.059 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.040246+0000", 2024-08-17T17:24:57.059 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2855999999999999e-05 2024-08-17T17:24:57.059 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:57.059 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:24:57.059 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:57.059 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.059 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.059 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:42737 6.5 6:abc4ca9b:::benchmark_data_smithi003_175043_object42736:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-08-17T17:24:57.060 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:24:57.040224+0000", 2024-08-17T17:24:57.060 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.00034719500000000001, 2024-08-17T17:24:57.060 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00067627599999999996, 2024-08-17T17:24:57.060 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:24:57.060 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:24:57.060 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:24:57.060 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:24:57.060 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:24:57.060 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 42737 2024-08-17T17:24:57.060 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.060 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:24:57.060 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.060 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:24:57.060 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.040224+0000", 2024-08-17T17:24:57.060 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:57.061 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.061 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.061 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:24:57.061 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.040224+0000", 2024-08-17T17:24:57.061 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:24:57.061 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.061 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.061 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:24:57.061 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.040222+0000", 2024-08-17T17:24:57.061 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:24:57.061 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.061 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.061 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:24:57.061 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.040255+0000", 2024-08-17T17:24:57.061 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2672999999999997e-05 2024-08-17T17:24:57.061 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.062 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.062 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:24:57.062 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.040267+0000", 2024-08-17T17:24:57.062 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2038000000000001e-05 2024-08-17T17:24:57.062 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.062 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.062 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:24:57.062 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.040273+0000", 2024-08-17T17:24:57.062 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7130000000000002e-06 2024-08-17T17:24:57.062 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.062 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.062 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:24:57.062 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.040315+0000", 2024-08-17T17:24:57.063 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2404000000000003e-05 2024-08-17T17:24:57.063 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.063 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.063 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:24:57.063 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.040529+0000", 2024-08-17T17:24:57.063 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000213346 2024-08-17T17:24:57.063 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:24:57.063 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:24:57.063 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,3", 2024-08-17T17:24:57.063 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:24:57.040661+0000", 2024-08-17T17:24:57.063 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013275099999999999 2024-08-17T17:24:57.063 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:57.063 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:24:57.063 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:57.064 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:24:57.064 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-08-17T17:24:57.064 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 5 2024-08-17T17:24:57.064 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:24:57.079 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 28 16 42789 42773 95.3944 80.875 0.0063768 0.0104769 2024-08-17T17:24:57.111 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:57.141 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:57.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:57.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:57.243 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:57.274 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:57.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:57.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:57.376 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:57.400 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:57.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:57.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:57.502 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:57.534 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:57.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:57.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:57.636 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:57.671 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:57.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:57.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:57.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:57 smithi053 ceph-mon[89963]: pgmap v477: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.4 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 81 MiB/s wr, 1.30k op/s; 4956/116592 objects degraded (4.251%); 11 KiB/s, 0 objects/s recovering 2024-08-17T17:24:57.773 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:57.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:57.800 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:57.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:57.822 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-17T17:24:57.901 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:24:57.941 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:57.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:57.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:24:58.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:57 smithi003 ceph-mon[103344]: pgmap v477: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.4 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 81 MiB/s wr, 1.30k op/s; 4956/116592 objects degraded (4.251%); 11 KiB/s, 0 objects/s recovering 2024-08-17T17:24:58.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:57 smithi003 ceph-mon[100615]: pgmap v477: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 87 active+clean; 2.4 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 81 MiB/s wr, 1.30k op/s; 4956/116592 objects degraded (4.251%); 11 KiB/s, 0 objects/s recovering 2024-08-17T17:24:58.042 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:24:58.079 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 29 16 43871 43855 94.4373 67.625 0.0054326 0.010581 2024-08-17T17:24:58.083 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:58.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:58.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:24:58.184 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:58.222 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:58.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:58.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:58.324 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:58.362 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:58.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:58.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:24:58.464 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:58.496 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:58.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:58.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:58.598 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:24:58.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:58.647 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:58.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:24:58.688 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:24:58.748 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:58.771 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:58.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:58.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:58.873 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:58.906 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:58.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:58.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:59.008 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:59.045 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:59.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:59.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:59.079 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 30 16 45162 45146 93.9791 80.6875 0.0141138 0.0106348 2024-08-17T17:24:59.147 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:59.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:59.183 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:59.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:59.284 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:59.287 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:24:59.315 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:59.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:59.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:59.417 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:24:59.464 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:24:59.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:59.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:24:59.566 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:24:59.614 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:59.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:59.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:24:59.699 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:24:59 smithi003 ceph-mon[103344]: pgmap v478: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.4 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 47 MiB/s wr, 754 op/s; 7340/120240 objects degraded (6.104%); 2487/120240 objects misplaced (2.068%); 363 KiB/s, 6 objects/s recovering 2024-08-17T17:24:59.700 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:24:59 smithi003 ceph-mon[100615]: pgmap v478: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.4 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 47 MiB/s wr, 754 op/s; 7340/120240 objects degraded (6.104%); 2487/120240 objects misplaced (2.068%); 363 KiB/s, 6 objects/s recovering 2024-08-17T17:24:59.715 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:24:59.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:24:59 smithi053 ceph-mon[89963]: pgmap v478: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.4 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 47 MiB/s wr, 754 op/s; 7340/120240 objects degraded (6.104%); 2487/120240 objects misplaced (2.068%); 363 KiB/s, 6 objects/s recovering 2024-08-17T17:24:59.766 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:59.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:59.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:24:59.868 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:24:59.917 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:24:59.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:24:59.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:00.019 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:00.047 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:00.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:00.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:00.080 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 31 16 46469 46453 93.5826 81.6875 0.00833265 0.0106792 2024-08-17T17:25:00.149 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:00.190 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:00.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:00.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:00.292 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:00.330 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:00.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:00.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:00.432 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:00.463 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:00.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:00.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:00.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:25:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:25:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:25:00.566 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:00.600 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:00.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:00.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:00.635 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:25:00.702 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:00.728 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:00.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:00.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:00.829 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:00.875 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:00.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:00.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:00.977 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:01.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:01.018 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:01.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:01.081 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 32 16 47701 47685 93.0616 77 0.00840465 0.0107392 2024-08-17T17:25:01.119 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:01.164 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:01.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:01.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:01.218 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:25:01.218 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2024-08-17T17:25:01.218 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2024-08-17T17:25:01.218 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2024-08-17T17:25:01.218 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:25:01.266 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:01.294 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:01.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:01.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:01.396 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:01.427 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:01.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:01.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:01.529 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:01.567 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:01.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:01.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:01.669 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:01.711 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:01.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:01.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:01.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:01 smithi053 ceph-mon[89963]: pgmap v479: 89 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 2.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 58 MiB/s wr, 933 op/s; 9800/127359 objects degraded (7.695%); 2487/127359 objects misplaced (1.953%); 294 KiB/s, 5 objects/s recovering 2024-08-17T17:25:01.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:01 smithi003 ceph-mon[100615]: pgmap v479: 89 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 2.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 58 MiB/s wr, 933 op/s; 9800/127359 objects degraded (7.695%); 2487/127359 objects misplaced (1.953%); 294 KiB/s, 5 objects/s recovering 2024-08-17T17:25:01.808 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:01 smithi003 ceph-mon[103344]: pgmap v479: 89 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 2.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 58 MiB/s wr, 933 op/s; 9800/127359 objects degraded (7.695%); 2487/127359 objects misplaced (1.953%); 294 KiB/s, 5 objects/s recovering 2024-08-17T17:25:01.813 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:01.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:01.879 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:01.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:01.911 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:25:01.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:02.026 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:02.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:02.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:02.081 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 33 16 48877 48861 92.4677 73.5 0.0148045 0.0108086 2024-08-17T17:25:02.128 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:02.184 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:02.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:02.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:02.286 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:02.318 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:02.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:02.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:02.419 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:02.452 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:02.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:02.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:02.553 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:02.571 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-08-17T17:25:02.604 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:02 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2881903368' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:25:02.605 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:02 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2881903368' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:25:02.605 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:02.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:02.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:02.708 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:02.737 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:02.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:02.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:02.829 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:02 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2881903368' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:25:02.839 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:02.863 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-17T17:25:02.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:02.879 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:02.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:02.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:03.020 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:03.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:03.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:03.081 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 34 16 49939 49923 91.7006 66.375 0.017809 0.0108985 2024-08-17T17:25:03.122 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:03.170 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:03.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:03.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:03.268 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:25:03.273 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:03.313 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:03.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:03.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:03.414 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:03.479 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:03.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:03.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:03.581 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:03.616 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:03.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:03.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:03.718 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:03.759 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:03.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:03.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:03 smithi003 ceph-mon[100615]: pgmap v480: 89 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 2.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s; 9800/137904 objects degraded (7.106%); 2313/137904 objects misplaced (1.677%); 1.5 MiB/s, 25 objects/s recovering 2024-08-17T17:25:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:03 smithi003 ceph-mon[100615]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:25:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:03 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 9800/137904 objects degraded (7.106%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:03 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2881903368' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:25:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:03 smithi003 ceph-mon[100615]: osdmap e196: 8 total, 8 up, 7 in 2024-08-17T17:25:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:03 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:25:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:03 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:25:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:03 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1091402763' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:25:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:03 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:25:03.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:03 smithi003 ceph-mon[103344]: pgmap v480: 89 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 2.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s; 9800/137904 objects degraded (7.106%); 2313/137904 objects misplaced (1.677%); 1.5 MiB/s, 25 objects/s recovering 2024-08-17T17:25:03.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:03 smithi003 ceph-mon[103344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:25:03.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:03 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 9800/137904 objects degraded (7.106%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:03.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:03 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2881903368' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:25:03.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:03 smithi003 ceph-mon[103344]: osdmap e196: 8 total, 8 up, 7 in 2024-08-17T17:25:03.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:03 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:25:03.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:03 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:25:03.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:03 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1091402763' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:25:03.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:03 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:25:03.775 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:03 smithi053 ceph-mon[89963]: pgmap v480: 89 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 2.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s; 9800/137904 objects degraded (7.106%); 2313/137904 objects misplaced (1.677%); 1.5 MiB/s, 25 objects/s recovering 2024-08-17T17:25:03.775 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:03 smithi053 ceph-mon[89963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:25:03.775 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:03 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 9800/137904 objects degraded (7.106%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:03.775 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:03 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2881903368' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:25:03.775 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:03 smithi053 ceph-mon[89963]: osdmap e196: 8 total, 8 up, 7 in 2024-08-17T17:25:03.775 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:03 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:25:03.775 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:03 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:25:03.775 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:03 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1091402763' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:25:03.775 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:03 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:25:03.861 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:03.887 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:03.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:03.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:03.988 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:03.989 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.6 to 1 (10000) 2024-08-17T17:25:04.027 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:04.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:04.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:04.081 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 35 16 51069 51053 91.0986 70.625 0.00222734 0.0109488 2024-08-17T17:25:04.129 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:04.154 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:04.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:04.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:04.256 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:04.278 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:04.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:04.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:04.380 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:04.402 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:04.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:04.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:04.504 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:04.535 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:04.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:04.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:04.637 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:04.662 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:04.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:04.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:04.764 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:04.788 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:04.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:04.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:04.890 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:04.915 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:04.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:04.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:05.003 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:25:05.004 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2024-08-17T17:25:05.004 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2024-08-17T17:25:05.004 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2024-08-17T17:25:05.004 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.004 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:68129 5.9 5:9510659a:::benchmark_data_smithi003_140793_object68128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-17T17:25:05.004 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:19:56.467405+0000", 2024-08-17T17:25:05.004 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 308.533580739, 2024-08-17T17:25:05.004 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.23049352, 2024-08-17T17:25:05.004 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:25:05.004 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2024-08-17T17:25:05.004 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:25:05.004 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:25:05.004 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:25:05.005 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 68129 2024-08-17T17:25:05.005 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.005 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:25:05.005 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.005 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:25:05.005 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.467405+0000", 2024-08-17T17:25:05.005 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.005 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.005 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.005 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:25:05.005 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.467405+0000", 2024-08-17T17:25:05.005 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.005 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.005 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.006 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:25:05.006 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.467404+0000", 2024-08-17T17:25:05.006 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:25:05.006 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.006 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.006 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:25:05.006 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.467439+0000", 2024-08-17T17:25:05.006 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4745e-05 2024-08-17T17:25:05.006 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.006 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.006 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:25:05.006 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.467454+0000", 2024-08-17T17:25:05.007 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5764e-05 2024-08-17T17:25:05.007 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.007 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.007 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:25:05.007 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.467461+0000", 2024-08-17T17:25:05.007 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4409999999999998e-06 2024-08-17T17:25:05.007 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.007 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.007 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.007 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.467501+0000", 2024-08-17T17:25:05.007 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0166999999999997e-05 2024-08-17T17:25:05.007 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.007 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.007 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:25:05.008 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.467596+0000", 2024-08-17T17:25:05.008 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5434999999999997e-05 2024-08-17T17:25:05.008 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.008 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.008 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,4", 2024-08-17T17:25:05.008 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.467689+0000", 2024-08-17T17:25:05.008 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.2597999999999997e-05 2024-08-17T17:25:05.008 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.008 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.008 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:25:05.008 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.469945+0000", 2024-08-17T17:25:05.008 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0022558510000000001 2024-08-17T17:25:05.008 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.009 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.009 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:25:05.009 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.471143+0000", 2024-08-17T17:25:05.009 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0011986799999999999 2024-08-17T17:25:05.009 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.009 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.009 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.009 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.473992+0000", 2024-08-17T17:25:05.009 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0028489959999999999 2024-08-17T17:25:05.009 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.009 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.009 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:25:05.009 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.474008+0000", 2024-08-17T17:25:05.010 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5574e-05 2024-08-17T17:25:05.010 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.010 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.010 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.010 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:57.483854+0000", 2024-08-17T17:25:05.010 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.009845922 2024-08-17T17:25:05.010 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.010 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.010 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:25:05.010 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:57.483866+0000", 2024-08-17T17:25:05.010 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1804000000000001e-05 2024-08-17T17:25:05.010 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.010 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.010 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.011 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:58.697814+0000", 2024-08-17T17:25:05.011 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2139484229999999 2024-08-17T17:25:05.011 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.011 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.011 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:25:05.011 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:58.697899+0000", 2024-08-17T17:25:05.011 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.4734999999999994e-05 2024-08-17T17:25:05.011 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.011 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:25:05.011 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.011 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.011 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.011 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:68113 5.4 5.7ee11e04 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-08-17T17:25:05.012 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:19:56.477714+0000", 2024-08-17T17:25:05.012 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 308.523271889, 2024-08-17T17:25:05.012 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0063266660000001, 2024-08-17T17:25:05.012 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:25:05.012 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2024-08-17T17:25:05.012 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:25:05.012 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:25:05.012 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:25:05.012 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 68113 2024-08-17T17:25:05.012 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.012 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:25:05.012 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.012 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:25:05.012 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477714+0000", 2024-08-17T17:25:05.012 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.013 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.013 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.013 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:25:05.013 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477714+0000", 2024-08-17T17:25:05.013 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.013 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.013 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.013 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:25:05.013 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477713+0000", 2024-08-17T17:25:05.013 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:25:05.013 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.013 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.013 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:25:05.013 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477757+0000", 2024-08-17T17:25:05.013 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3979e-05 2024-08-17T17:25:05.013 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.014 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.014 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:25:05.014 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477795+0000", 2024-08-17T17:25:05.014 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8472e-05 2024-08-17T17:25:05.014 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.014 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.014 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:25:05.014 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477803+0000", 2024-08-17T17:25:05.014 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.8690000000000004e-06 2024-08-17T17:25:05.014 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.014 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.014 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.014 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477873+0000", 2024-08-17T17:25:05.014 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0020999999999999e-05 2024-08-17T17:25:05.014 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.015 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.015 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:25:05.015 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477893+0000", 2024-08-17T17:25:05.015 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9724999999999998e-05 2024-08-17T17:25:05.015 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.015 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.015 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.015 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:57.484032+0000", 2024-08-17T17:25:05.016 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0061393169999999 2024-08-17T17:25:05.016 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.016 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.016 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:25:05.016 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:57.484041+0000", 2024-08-17T17:25:05.016 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.6859999999999995e-06 2024-08-17T17:25:05.016 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.016 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:25:05.016 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.016 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.016 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.016 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:68122 5.4 5.9fb398a4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-08-17T17:25:05.016 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:19:56.477817+0000", 2024-08-17T17:25:05.016 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 308.52316898100003, 2024-08-17T17:25:05.017 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0062544259999999, 2024-08-17T17:25:05.017 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:25:05.017 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2024-08-17T17:25:05.017 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:25:05.017 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:25:05.017 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:25:05.017 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 68122 2024-08-17T17:25:05.017 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.017 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:25:05.017 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.017 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:25:05.017 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477817+0000", 2024-08-17T17:25:05.017 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.017 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.018 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.018 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:25:05.018 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477817+0000", 2024-08-17T17:25:05.018 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.018 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.018 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.018 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:05.020 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:25:05.020 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477816+0000", 2024-08-17T17:25:05.020 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:25:05.020 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.020 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.020 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:25:05.020 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477855+0000", 2024-08-17T17:25:05.020 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9004999999999997e-05 2024-08-17T17:25:05.020 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.020 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.020 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:25:05.020 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477866+0000", 2024-08-17T17:25:05.021 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0944000000000001e-05 2024-08-17T17:25:05.021 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.021 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.021 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:25:05.021 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477872+0000", 2024-08-17T17:25:05.021 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9200000000000001e-06 2024-08-17T17:25:05.021 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.021 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.021 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.021 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477924+0000", 2024-08-17T17:25:05.021 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1924e-05 2024-08-17T17:25:05.021 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.022 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.022 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:25:05.022 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477932+0000", 2024-08-17T17:25:05.022 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.5100000000000001e-06 2024-08-17T17:25:05.022 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.022 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.022 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.022 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:57.484065+0000", 2024-08-17T17:25:05.022 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0061333269999999 2024-08-17T17:25:05.022 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.022 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.023 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:25:05.023 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:57.484072+0000", 2024-08-17T17:25:05.023 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4790000000000001e-06 2024-08-17T17:25:05.023 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.023 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:25:05.023 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.023 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.023 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.023 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:68126 5.4 5.7ec1dd54 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-08-17T17:25:05.023 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:19:56.477878+0000", 2024-08-17T17:25:05.023 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 308.52310782199999, 2024-08-17T17:25:05.023 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0062189070000001, 2024-08-17T17:25:05.023 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:25:05.024 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2024-08-17T17:25:05.024 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:25:05.024 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:25:05.024 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:25:05.024 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 68126 2024-08-17T17:25:05.024 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.024 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:25:05.024 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.024 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:25:05.024 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477878+0000", 2024-08-17T17:25:05.024 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.024 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.024 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.025 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:25:05.025 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477878+0000", 2024-08-17T17:25:05.025 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.025 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.025 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.025 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:25:05.025 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477878+0000", 2024-08-17T17:25:05.025 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:25:05.025 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.025 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.025 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:25:05.025 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477922+0000", 2024-08-17T17:25:05.025 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3866999999999999e-05 2024-08-17T17:25:05.025 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.026 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.026 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:25:05.026 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477931+0000", 2024-08-17T17:25:05.026 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5429999999999999e-06 2024-08-17T17:25:05.026 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.026 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.026 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:25:05.026 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477936+0000", 2024-08-17T17:25:05.026 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.8280000000000002e-06 2024-08-17T17:25:05.026 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.026 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.026 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.027 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477980+0000", 2024-08-17T17:25:05.027 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3630000000000001e-05 2024-08-17T17:25:05.027 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.027 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.027 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:25:05.027 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477985+0000", 2024-08-17T17:25:05.027 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1320000000000002e-06 2024-08-17T17:25:05.027 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.027 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.027 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.027 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:57.484091+0000", 2024-08-17T17:25:05.027 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0061058940000001 2024-08-17T17:25:05.027 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.027 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.028 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:25:05.028 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:57.484097+0000", 2024-08-17T17:25:05.028 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5540000000000003e-06 2024-08-17T17:25:05.028 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.028 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:25:05.028 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.028 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.028 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.028 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:68128 5.4 5.407770a4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-08-17T17:25:05.028 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:19:56.477957+0000", 2024-08-17T17:25:05.028 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 308.52302940599998, 2024-08-17T17:25:05.028 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0061641539999999, 2024-08-17T17:25:05.029 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:25:05.029 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2024-08-17T17:25:05.029 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:25:05.029 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:25:05.029 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:25:05.029 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 68128 2024-08-17T17:25:05.029 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.029 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:25:05.029 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.029 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:25:05.029 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477957+0000", 2024-08-17T17:25:05.029 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.029 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.029 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.030 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:25:05.030 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477957+0000", 2024-08-17T17:25:05.030 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.030 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.030 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.030 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:25:05.030 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477956+0000", 2024-08-17T17:25:05.030 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:25:05.030 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.030 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.030 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:25:05.030 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477989+0000", 2024-08-17T17:25:05.030 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2774000000000002e-05 2024-08-17T17:25:05.031 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.031 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.031 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:25:05.031 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477998+0000", 2024-08-17T17:25:05.031 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1549999999999996e-06 2024-08-17T17:25:05.031 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.031 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.031 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:25:05.031 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.478003+0000", 2024-08-17T17:25:05.031 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6079999999999998e-06 2024-08-17T17:25:05.031 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.031 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.031 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.031 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.478043+0000", 2024-08-17T17:25:05.032 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0331999999999999e-05 2024-08-17T17:25:05.032 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.032 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.032 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:25:05.032 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.478048+0000", 2024-08-17T17:25:05.032 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1529999999999996e-06 2024-08-17T17:25:05.032 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.032 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.032 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.032 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:57.484115+0000", 2024-08-17T17:25:05.032 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0060665440000001 2024-08-17T17:25:05.032 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.032 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.032 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:25:05.033 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:57.484121+0000", 2024-08-17T17:25:05.033 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1279999999999996e-06 2024-08-17T17:25:05.033 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.033 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:25:05.033 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.033 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.033 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.033 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:68137 5.4 5.a7da3c24 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-08-17T17:25:05.033 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:19:56.501938+0000", 2024-08-17T17:25:05.033 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 308.49904783699998, 2024-08-17T17:25:05.033 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.98222817500000004, 2024-08-17T17:25:05.033 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:25:05.033 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2024-08-17T17:25:05.033 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:25:05.034 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:25:05.034 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:25:05.034 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 68137 2024-08-17T17:25:05.034 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.034 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:25:05.034 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.034 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:25:05.034 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.501938+0000", 2024-08-17T17:25:05.034 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.034 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.034 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.034 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:25:05.035 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.501938+0000", 2024-08-17T17:25:05.035 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.035 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.035 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.035 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:25:05.035 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.501937+0000", 2024-08-17T17:25:05.035 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:25:05.035 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.035 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.035 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:25:05.035 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.501998+0000", 2024-08-17T17:25:05.035 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1072000000000001e-05 2024-08-17T17:25:05.035 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.035 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.035 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:25:05.036 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.502012+0000", 2024-08-17T17:25:05.036 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.401e-05 2024-08-17T17:25:05.036 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.036 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.036 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:25:05.036 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.502019+0000", 2024-08-17T17:25:05.036 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.725e-06 2024-08-17T17:25:05.036 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.036 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.036 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.036 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.502059+0000", 2024-08-17T17:25:05.036 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.053e-05 2024-08-17T17:25:05.036 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.036 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.036 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:25:05.037 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.502067+0000", 2024-08-17T17:25:05.037 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7910000000000005e-06 2024-08-17T17:25:05.037 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.037 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.037 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.037 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:57.484160+0000", 2024-08-17T17:25:05.037 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.98209291200000004 2024-08-17T17:25:05.037 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.037 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.037 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:25:05.037 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:57.484166+0000", 2024-08-17T17:25:05.038 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5830000000000003e-06 2024-08-17T17:25:05.038 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.038 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:25:05.038 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.038 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.038 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.038 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:112683 5.9 5:9d8751d9:::benchmark_data_smithi003_140793_object10626:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-08-17T17:25:05.038 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:20:59.144624+0000", 2024-08-17T17:25:05.038 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 245.85636241099999, 2024-08-17T17:25:05.038 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.24938321399999999, 2024-08-17T17:25:05.038 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:25:05.038 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:25:05.038 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:25:05.039 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:25:05.039 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:25:05.039 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 112683 2024-08-17T17:25:05.039 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.039 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:25:05.039 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.039 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:25:05.039 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.144624+0000", 2024-08-17T17:25:05.039 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.039 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.039 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.039 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:25:05.039 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.144624+0000", 2024-08-17T17:25:05.040 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.040 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.040 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.040 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:25:05.040 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.144622+0000", 2024-08-17T17:25:05.040 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:25:05.040 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.040 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.040 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:25:05.040 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.144633+0000", 2024-08-17T17:25:05.040 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.118e-05 2024-08-17T17:25:05.040 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.040 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.041 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:25:05.041 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.144652+0000", 2024-08-17T17:25:05.041 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.925e-05 2024-08-17T17:25:05.041 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.041 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.041 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:25:05.041 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.144659+0000", 2024-08-17T17:25:05.041 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.7939999999999997e-06 2024-08-17T17:25:05.041 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.041 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.041 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.041 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.252458+0000", 2024-08-17T17:25:05.041 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.10779897500000001 2024-08-17T17:25:05.041 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.042 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.042 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for degraded object", 2024-08-17T17:25:05.042 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.252889+0000", 2024-08-17T17:25:05.042 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000430906 2024-08-17T17:25:05.042 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.042 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.042 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.042 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.268609+0000", 2024-08-17T17:25:05.042 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.015720357000000001 2024-08-17T17:25:05.042 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.042 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.042 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:25:05.042 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.268671+0000", 2024-08-17T17:25:05.043 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1861999999999998e-05 2024-08-17T17:25:05.043 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.043 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.043 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:25:05.043 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.268758+0000", 2024-08-17T17:25:05.043 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.6433000000000004e-05 2024-08-17T17:25:05.043 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.043 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.043 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:25:05.043 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.316211+0000", 2024-08-17T17:25:05.043 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.047453690999999999 2024-08-17T17:25:05.044 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.044 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.044 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:25:05.044 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.330353+0000", 2024-08-17T17:25:05.044 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.014141712000000001 2024-08-17T17:25:05.044 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.044 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.044 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:25:05.044 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.388973+0000", 2024-08-17T17:25:05.044 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.058620086000000002 2024-08-17T17:25:05.044 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.044 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.044 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:25:05.044 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.389028+0000", 2024-08-17T17:25:05.045 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4469000000000003e-05 2024-08-17T17:25:05.045 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.045 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.045 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:25:05.045 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.394007+0000", 2024-08-17T17:25:05.045 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0049793950000000002 2024-08-17T17:25:05.045 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.045 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:25:05.045 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.045 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.045 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.045 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:112696 5.9 5:9786f897:::benchmark_data_smithi003_140793_object10639:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-08-17T17:25:05.045 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:20:59.161610+0000", 2024-08-17T17:25:05.045 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 245.839376609, 2024-08-17T17:25:05.046 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.237310516, 2024-08-17T17:25:05.046 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:25:05.046 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:25:05.046 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:25:05.046 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:25:05.046 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:25:05.046 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 112696 2024-08-17T17:25:05.046 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.046 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:25:05.046 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.046 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:25:05.046 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.161610+0000", 2024-08-17T17:25:05.046 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.047 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.047 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.047 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:25:05.047 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.161610+0000", 2024-08-17T17:25:05.047 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.047 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.047 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.047 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:25:05.047 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.161607+0000", 2024-08-17T17:25:05.047 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:25:05.047 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.047 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.047 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:25:05.047 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.161616+0000", 2024-08-17T17:25:05.048 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.2730000000000002e-06 2024-08-17T17:25:05.048 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.048 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.048 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:25:05.048 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.161634+0000", 2024-08-17T17:25:05.048 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.8615999999999999e-05 2024-08-17T17:25:05.048 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.048 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.048 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:25:05.048 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.161641+0000", 2024-08-17T17:25:05.048 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5570000000000001e-06 2024-08-17T17:25:05.048 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.048 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.048 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.049 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.252927+0000", 2024-08-17T17:25:05.049 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.091286272000000002 2024-08-17T17:25:05.049 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.049 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.049 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for degraded object", 2024-08-17T17:25:05.049 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.253290+0000", 2024-08-17T17:25:05.049 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000362994 2024-08-17T17:25:05.049 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.049 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.049 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.049 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.288104+0000", 2024-08-17T17:25:05.049 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.034813802999999997 2024-08-17T17:25:05.049 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.049 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.050 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:25:05.050 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.288169+0000", 2024-08-17T17:25:05.050 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4731000000000001e-05 2024-08-17T17:25:05.050 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.050 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.050 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:25:05.050 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.288254+0000", 2024-08-17T17:25:05.050 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.5177000000000001e-05 2024-08-17T17:25:05.050 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.050 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.050 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:25:05.050 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.326648+0000", 2024-08-17T17:25:05.050 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.038393712000000003 2024-08-17T17:25:05.050 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.051 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.051 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:25:05.051 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.394089+0000", 2024-08-17T17:25:05.051 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.067441017000000006 2024-08-17T17:25:05.051 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.051 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.051 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:25:05.051 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.394131+0000", 2024-08-17T17:25:05.051 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2812000000000003e-05 2024-08-17T17:25:05.051 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.051 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.051 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:25:05.051 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.394166+0000", 2024-08-17T17:25:05.051 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4941000000000003e-05 2024-08-17T17:25:05.052 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.052 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.052 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:25:05.052 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.398920+0000", 2024-08-17T17:25:05.052 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0047537200000000003 2024-08-17T17:25:05.052 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.052 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:25:05.052 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.052 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.052 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.052 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:112710 5.9 5:95df58a9:::benchmark_data_smithi003_140793_object10653:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-08-17T17:25:05.052 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:20:59.186192+0000", 2024-08-17T17:25:05.052 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 245.814793936, 2024-08-17T17:25:05.052 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.21782403, 2024-08-17T17:25:05.053 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:25:05.053 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:25:05.053 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:25:05.053 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:25:05.053 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:25:05.053 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 112710 2024-08-17T17:25:05.053 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.053 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:25:05.053 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.053 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:25:05.053 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.186192+0000", 2024-08-17T17:25:05.053 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.053 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.054 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.054 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:25:05.054 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.186192+0000", 2024-08-17T17:25:05.054 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.054 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.054 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.054 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:25:05.054 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.186191+0000", 2024-08-17T17:25:05.054 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:25:05.054 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.054 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.054 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:25:05.054 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.186202+0000", 2024-08-17T17:25:05.054 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1209000000000001e-05 2024-08-17T17:25:05.055 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.055 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.055 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:25:05.055 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.186218+0000", 2024-08-17T17:25:05.055 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5892999999999998e-05 2024-08-17T17:25:05.055 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.055 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.055 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:25:05.055 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.186225+0000", 2024-08-17T17:25:05.055 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.5079999999999999e-06 2024-08-17T17:25:05.055 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.055 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.055 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.055 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.257659+0000", 2024-08-17T17:25:05.056 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.071434003999999995 2024-08-17T17:25:05.056 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.056 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.056 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for degraded object", 2024-08-17T17:25:05.056 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.258073+0000", 2024-08-17T17:25:05.056 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00041356599999999998 2024-08-17T17:25:05.056 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.056 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.056 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.056 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.326686+0000", 2024-08-17T17:25:05.056 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.068612772000000002 2024-08-17T17:25:05.056 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.056 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.056 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:25:05.056 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.326817+0000", 2024-08-17T17:25:05.057 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013136699999999999 2024-08-17T17:25:05.057 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.057 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.057 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:25:05.057 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.327032+0000", 2024-08-17T17:25:05.057 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021476899999999999 2024-08-17T17:25:05.057 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.057 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.057 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:25:05.057 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.358458+0000", 2024-08-17T17:25:05.057 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.031426035999999997 2024-08-17T17:25:05.057 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.057 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.058 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:25:05.058 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.398995+0000", 2024-08-17T17:25:05.058 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.040537205999999999 2024-08-17T17:25:05.058 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.058 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.058 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:25:05.058 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.399045+0000", 2024-08-17T17:25:05.058 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9650000000000001e-05 2024-08-17T17:25:05.058 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.058 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.058 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:25:05.059 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.399104+0000", 2024-08-17T17:25:05.059 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9852999999999998e-05 2024-08-17T17:25:05.059 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.059 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.059 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:25:05.059 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.404016+0000", 2024-08-17T17:25:05.059 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0049118169999999997 2024-08-17T17:25:05.059 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.059 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:25:05.059 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.059 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.059 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.060 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:113964 5.9 5:9063bcb1:::benchmark_data_smithi003_140793_object11907:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-08-17T17:25:05.060 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:21:01.112989+0000", 2024-08-17T17:25:05.060 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 243.887996684, 2024-08-17T17:25:05.060 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.23952488699999999, 2024-08-17T17:25:05.060 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:25:05.060 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:25:05.060 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:25:05.060 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:25:05.060 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:25:05.060 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 113964 2024-08-17T17:25:05.060 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.060 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:25:05.060 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.061 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:25:05.061 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.112989+0000", 2024-08-17T17:25:05.061 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.061 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.061 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.061 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:25:05.061 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.112989+0000", 2024-08-17T17:25:05.061 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.061 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.061 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.061 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:25:05.061 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.112989+0000", 2024-08-17T17:25:05.061 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:25:05.061 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.062 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.062 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:25:05.062 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.112995+0000", 2024-08-17T17:25:05.062 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1659999999999999e-06 2024-08-17T17:25:05.062 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.062 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.062 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:25:05.062 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.113007+0000", 2024-08-17T17:25:05.062 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1729000000000001e-05 2024-08-17T17:25:05.062 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.062 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.062 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:25:05.062 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.113012+0000", 2024-08-17T17:25:05.063 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7529999999999998e-06 2024-08-17T17:25:05.063 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.063 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.063 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.063 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.186592+0000", 2024-08-17T17:25:05.063 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.073580089000000001 2024-08-17T17:25:05.063 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.063 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.063 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for degraded object", 2024-08-17T17:25:05.063 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.187303+0000", 2024-08-17T17:25:05.063 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00071040899999999995 2024-08-17T17:25:05.063 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.064 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.064 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.064 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.212958+0000", 2024-08-17T17:25:05.064 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.025654966000000001 2024-08-17T17:25:05.064 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.064 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.064 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:25:05.064 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.213018+0000", 2024-08-17T17:25:05.064 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0143999999999999e-05 2024-08-17T17:25:05.064 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.064 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.064 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:25:05.064 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.213117+0000", 2024-08-17T17:25:05.065 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.9322999999999998e-05 2024-08-17T17:25:05.065 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.065 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.065 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:25:05.065 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.334107+0000", 2024-08-17T17:25:05.065 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.120990216 2024-08-17T17:25:05.065 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.065 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.065 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:25:05.065 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.347424+0000", 2024-08-17T17:25:05.065 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.013316721 2024-08-17T17:25:05.065 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.065 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.065 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:25:05.066 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.347458+0000", 2024-08-17T17:25:05.066 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4221999999999999e-05 2024-08-17T17:25:05.066 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.066 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.066 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:25:05.066 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.347514+0000", 2024-08-17T17:25:05.066 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5501000000000003e-05 2024-08-17T17:25:05.066 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.066 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.066 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:25:05.066 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.352514+0000", 2024-08-17T17:25:05.066 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0050004539999999997 2024-08-17T17:25:05.066 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.067 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:25:05.067 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.067 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.067 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.067 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:113971 5.9 5:9a18fcd4:::benchmark_data_smithi003_140793_object11914:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-08-17T17:25:05.067 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:21:01.120085+0000", 2024-08-17T17:25:05.067 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 243.880900959, 2024-08-17T17:25:05.067 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.23685455599999999, 2024-08-17T17:25:05.067 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:25:05.067 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:25:05.067 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:25:05.067 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:25:05.067 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:25:05.068 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 113971 2024-08-17T17:25:05.068 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.068 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:25:05.068 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.068 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:25:05.068 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.120085+0000", 2024-08-17T17:25:05.068 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.068 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.068 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.068 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:25:05.068 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.120085+0000", 2024-08-17T17:25:05.068 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.068 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.068 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.069 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:25:05.069 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.120084+0000", 2024-08-17T17:25:05.069 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:25:05.069 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.069 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.069 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:25:05.069 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.120090+0000", 2024-08-17T17:25:05.069 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2009999999999996e-06 2024-08-17T17:25:05.069 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.069 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.069 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:25:05.069 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.120102+0000", 2024-08-17T17:25:05.069 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2047e-05 2024-08-17T17:25:05.069 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.070 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.070 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:25:05.070 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.120107+0000", 2024-08-17T17:25:05.070 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3730000000000001e-06 2024-08-17T17:25:05.070 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.070 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.070 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.070 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.187331+0000", 2024-08-17T17:25:05.070 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.067223565999999998 2024-08-17T17:25:05.070 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.070 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.070 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for degraded object", 2024-08-17T17:25:05.070 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.187701+0000", 2024-08-17T17:25:05.071 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00037034299999999997 2024-08-17T17:25:05.071 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.071 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.071 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.071 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.304808+0000", 2024-08-17T17:25:05.071 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.117106919 2024-08-17T17:25:05.071 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.071 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.071 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:25:05.071 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.304897+0000", 2024-08-17T17:25:05.071 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.8435000000000003e-05 2024-08-17T17:25:05.071 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.071 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.071 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:25:05.071 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.305038+0000", 2024-08-17T17:25:05.072 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000141574 2024-08-17T17:25:05.072 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.072 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.072 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:25:05.072 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.334117+0000", 2024-08-17T17:25:05.072 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.029078366000000001 2024-08-17T17:25:05.072 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.072 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.072 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:25:05.072 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.352565+0000", 2024-08-17T17:25:05.072 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.018448101000000001 2024-08-17T17:25:05.072 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.072 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.072 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:25:05.073 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.352589+0000", 2024-08-17T17:25:05.073 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4555000000000001e-05 2024-08-17T17:25:05.073 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.073 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.073 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:25:05.073 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.352620+0000", 2024-08-17T17:25:05.073 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0957000000000002e-05 2024-08-17T17:25:05.073 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.073 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.073 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:25:05.073 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.356940+0000", 2024-08-17T17:25:05.073 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0043194790000000002 2024-08-17T17:25:05.073 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.073 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:25:05.074 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.074 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.074 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.074 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:113988 5.9 5:99701dcc:::benchmark_data_smithi003_140793_object11931:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-08-17T17:25:05.074 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:21:01.147053+0000", 2024-08-17T17:25:05.074 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 243.853933232, 2024-08-17T17:25:05.074 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.214886141, 2024-08-17T17:25:05.074 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:25:05.074 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:25:05.074 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:25:05.074 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:25:05.074 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:25:05.074 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 113988 2024-08-17T17:25:05.075 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.075 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:25:05.075 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.075 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:25:05.075 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.147053+0000", 2024-08-17T17:25:05.075 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.075 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.075 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.075 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:25:05.075 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.147053+0000", 2024-08-17T17:25:05.075 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.075 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.075 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.076 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:25:05.076 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.147051+0000", 2024-08-17T17:25:05.076 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:25:05.076 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.076 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.076 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:25:05.076 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.147069+0000", 2024-08-17T17:25:05.076 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7515000000000001e-05 2024-08-17T17:25:05.076 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.076 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.076 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:25:05.076 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.147083+0000", 2024-08-17T17:25:05.076 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4608e-05 2024-08-17T17:25:05.076 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.077 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.077 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:25:05.077 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.147090+0000", 2024-08-17T17:25:05.077 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.6610000000000003e-06 2024-08-17T17:25:05.077 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.077 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.078 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:04 smithi053 ceph-mon[89963]: pgmap v482: 89 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s; 9800/142614 objects degraded (6.872%); 2313/142614 objects misplaced (1.622%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:25:05.078 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:04 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-17T17:25:05.078 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:04 smithi053 ceph-mon[89963]: osdmap e197: 8 total, 8 up, 7 in 2024-08-17T17:25:05.078 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:04 smithi053 ceph-mon[89963]: osdmap e198: 8 total, 8 up, 7 in 2024-08-17T17:25:05.079 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.079 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.187726+0000", 2024-08-17T17:25:05.079 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.040636257000000002 2024-08-17T17:25:05.079 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.079 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.079 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for degraded object", 2024-08-17T17:25:05.079 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.188113+0000", 2024-08-17T17:25:05.079 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00038674899999999998 2024-08-17T17:25:05.079 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.080 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.080 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.080 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.318662+0000", 2024-08-17T17:25:05.080 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.130548734 2024-08-17T17:25:05.080 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.080 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.080 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:25:05.080 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.318771+0000", 2024-08-17T17:25:05.080 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000109394 2024-08-17T17:25:05.080 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.080 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.080 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:25:05.080 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.318945+0000", 2024-08-17T17:25:05.081 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000173597 2024-08-17T17:25:05.081 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.081 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.081 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:25:05.081 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.347390+0000", 2024-08-17T17:25:05.081 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.028444971999999999 2024-08-17T17:25:05.081 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.081 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.081 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:25:05.081 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.356991+0000", 2024-08-17T17:25:05.081 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0096016969999999993 2024-08-17T17:25:05.081 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.081 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.082 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:25:05.082 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.357431+0000", 2024-08-17T17:25:05.082 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00043941199999999997 2024-08-17T17:25:05.082 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.082 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.082 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:25:05.082 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.357493+0000", 2024-08-17T17:25:05.082 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2426000000000002e-05 2024-08-17T17:25:05.082 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.082 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.082 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:25:05.082 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.361939+0000", 2024-08-17T17:25:05.082 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0044456829999999998 2024-08-17T17:25:05.082 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.083 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:25:05.083 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.083 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.083 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.083 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:114294 5.9 5:9a6efe34:::benchmark_data_smithi003_140793_object12237:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-08-17T17:25:05.083 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:21:01.628248+0000", 2024-08-17T17:25:05.083 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 243.37273825099999, 2024-08-17T17:25:05.083 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.214618067, 2024-08-17T17:25:05.083 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:25:05.083 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:25:05.083 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:25:05.083 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:25:05.084 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:25:05.084 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 114294 2024-08-17T17:25:05.084 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.084 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:25:05.084 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.084 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:25:05.084 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.628248+0000", 2024-08-17T17:25:05.084 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.084 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.084 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.084 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:25:05.084 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.628248+0000", 2024-08-17T17:25:05.084 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.085 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.085 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.085 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:25:05.085 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.628247+0000", 2024-08-17T17:25:05.085 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:25:05.085 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.085 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.085 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:25:05.085 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.628254+0000", 2024-08-17T17:25:05.085 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0929999999999997e-06 2024-08-17T17:25:05.085 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.086 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.086 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:25:05.086 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.628264+0000", 2024-08-17T17:25:05.086 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0149999999999999e-05 2024-08-17T17:25:05.086 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.086 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.086 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:25:05.086 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.628269+0000", 2024-08-17T17:25:05.086 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3959999999999997e-06 2024-08-17T17:25:05.086 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.086 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.086 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.086 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.761136+0000", 2024-08-17T17:25:05.087 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.13286663900000001 2024-08-17T17:25:05.087 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.087 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.087 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for degraded object", 2024-08-17T17:25:05.087 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.761652+0000", 2024-08-17T17:25:05.087 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00051622300000000001 2024-08-17T17:25:05.087 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.087 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.087 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.087 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.780798+0000", 2024-08-17T17:25:05.087 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019146318999999998 2024-08-17T17:25:05.087 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.087 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.087 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:25:05.088 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.780885+0000", 2024-08-17T17:25:05.088 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.6273000000000006e-05 2024-08-17T17:25:05.088 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.088 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.088 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:25:05.088 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.781038+0000", 2024-08-17T17:25:05.088 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000152876 2024-08-17T17:25:05.088 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.088 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.088 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:25:05.088 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.805970+0000", 2024-08-17T17:25:05.088 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.024932691 2024-08-17T17:25:05.088 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.088 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.088 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:25:05.089 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.836398+0000", 2024-08-17T17:25:05.089 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.030428112 2024-08-17T17:25:05.089 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.089 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.089 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:25:05.089 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.836477+0000", 2024-08-17T17:25:05.089 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.8326999999999997e-05 2024-08-17T17:25:05.089 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.089 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.089 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:25:05.089 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.836557+0000", 2024-08-17T17:25:05.089 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0331e-05 2024-08-17T17:25:05.089 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.089 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.090 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:25:05.090 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:01.842866+0000", 2024-08-17T17:25:05.090 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0063089210000000003 2024-08-17T17:25:05.090 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.090 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:25:05.090 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.090 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.090 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.090 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:114739 5.9 5:91ff45cd:::benchmark_data_smithi003_140793_object12682:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-08-17T17:25:05.090 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:21:02.328991+0000", 2024-08-17T17:25:05.090 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 242.67199522499999, 2024-08-17T17:25:05.090 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.23855512500000001, 2024-08-17T17:25:05.091 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:25:05.091 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:25:05.091 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:25:05.091 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:25:05.091 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:25:05.091 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 114739 2024-08-17T17:25:05.091 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.091 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:25:05.091 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.091 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:25:05.091 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:02.328991+0000", 2024-08-17T17:25:05.091 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.092 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.092 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.092 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:25:05.092 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:02.328991+0000", 2024-08-17T17:25:05.092 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.092 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.092 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.092 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:25:05.092 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:02.328989+0000", 2024-08-17T17:25:05.092 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999976 2024-08-17T17:25:05.092 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.092 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.092 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:25:05.092 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:02.328997+0000", 2024-08-17T17:25:05.093 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.3669999999999997e-06 2024-08-17T17:25:05.093 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.093 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.093 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:25:05.093 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:02.329013+0000", 2024-08-17T17:25:05.093 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5863999999999999e-05 2024-08-17T17:25:05.093 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.093 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.093 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:25:05.093 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:02.329022+0000", 2024-08-17T17:25:05.093 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.6500000000000002e-06 2024-08-17T17:25:05.093 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.093 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.094 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.094 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:02.454958+0000", 2024-08-17T17:25:05.094 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.125936509 2024-08-17T17:25:05.094 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.094 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.094 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for degraded object", 2024-08-17T17:25:05.094 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:02.455347+0000", 2024-08-17T17:25:05.094 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00038867099999999999 2024-08-17T17:25:05.094 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.094 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.094 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.094 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:02.475404+0000", 2024-08-17T17:25:05.095 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.020056995000000001 2024-08-17T17:25:05.095 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.095 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.095 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:25:05.095 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:02.475504+0000", 2024-08-17T17:25:05.095 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010018600000000001 2024-08-17T17:25:05.095 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.095 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.095 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:25:05.095 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:02.475647+0000", 2024-08-17T17:25:05.095 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000142757 2024-08-17T17:25:05.095 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.095 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.096 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:25:05.096 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:02.489446+0000", 2024-08-17T17:25:05.096 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.013799448000000001 2024-08-17T17:25:05.096 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.096 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.096 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:25:05.096 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:02.514345+0000", 2024-08-17T17:25:05.096 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.024898478000000002 2024-08-17T17:25:05.096 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.096 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.096 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:25:05.096 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:02.563071+0000", 2024-08-17T17:25:05.096 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.048726751999999998 2024-08-17T17:25:05.097 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.097 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.097 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:25:05.097 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:02.563131+0000", 2024-08-17T17:25:05.097 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9499000000000001e-05 2024-08-17T17:25:05.097 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.097 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.097 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:25:05.097 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:02.567546+0000", 2024-08-17T17:25:05.097 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.004415174 2024-08-17T17:25:05.097 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.097 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:25:05.097 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.098 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.098 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.098 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:122519 5.9 5:9813af1f:::benchmark_data_smithi003_140793_object20462:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-08-17T17:25:05.098 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:21:14.073604+0000", 2024-08-17T17:25:05.098 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 230.927382534, 2024-08-17T17:25:05.098 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.21620722000000001, 2024-08-17T17:25:05.098 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:25:05.098 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:25:05.098 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:25:05.098 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:25:05.098 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:25:05.098 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 122519 2024-08-17T17:25:05.099 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.099 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:25:05.099 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.099 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:25:05.099 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:14.073604+0000", 2024-08-17T17:25:05.099 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.099 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.099 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.099 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:25:05.099 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:14.073604+0000", 2024-08-17T17:25:05.099 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.099 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.100 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.100 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:25:05.100 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:14.073602+0000", 2024-08-17T17:25:05.100 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:25:05.100 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.100 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.100 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:25:05.100 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:14.073609+0000", 2024-08-17T17:25:05.100 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9090000000000003e-06 2024-08-17T17:25:05.100 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.100 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.100 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:25:05.101 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:14.073625+0000", 2024-08-17T17:25:05.101 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5863999999999999e-05 2024-08-17T17:25:05.101 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.101 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.101 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:25:05.101 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:14.073631+0000", 2024-08-17T17:25:05.101 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.105e-06 2024-08-17T17:25:05.101 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.101 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.101 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.101 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:14.173963+0000", 2024-08-17T17:25:05.101 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.100332429 2024-08-17T17:25:05.102 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.102 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.102 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for degraded object", 2024-08-17T17:25:05.102 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:14.174518+0000", 2024-08-17T17:25:05.102 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00055506299999999995 2024-08-17T17:25:05.102 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.102 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.102 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.102 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:14.198862+0000", 2024-08-17T17:25:05.102 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.024343402 2024-08-17T17:25:05.102 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.102 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.102 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:25:05.102 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:14.199003+0000", 2024-08-17T17:25:05.103 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014138900000000001 2024-08-17T17:25:05.103 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.103 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.103 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:25:05.103 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:14.199218+0000", 2024-08-17T17:25:05.103 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000214587 2024-08-17T17:25:05.103 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.103 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.103 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:25:05.103 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:14.226645+0000", 2024-08-17T17:25:05.103 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.027427488999999999 2024-08-17T17:25:05.103 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.103 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.104 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:25:05.104 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:14.278399+0000", 2024-08-17T17:25:05.104 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.051753830000000001 2024-08-17T17:25:05.104 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.104 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.104 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:25:05.104 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:14.285112+0000", 2024-08-17T17:25:05.104 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0067126039999999996 2024-08-17T17:25:05.104 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.104 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.104 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:25:05.104 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:14.285181+0000", 2024-08-17T17:25:05.104 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.8750000000000004e-05 2024-08-17T17:25:05.104 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.105 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.105 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:25:05.105 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:14.289811+0000", 2024-08-17T17:25:05.105 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0046302909999999999 2024-08-17T17:25:05.105 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.105 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:25:05.105 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.105 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.105 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.105 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:123390 5.9 5:91763507:::benchmark_data_smithi003_140793_object21333:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-08-17T17:25:05.105 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:21:15.342371+0000", 2024-08-17T17:25:05.105 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 229.65861474900001, 2024-08-17T17:25:05.105 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.23271187199999999, 2024-08-17T17:25:05.105 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:25:05.106 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:25:05.106 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:25:05.106 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:25:05.106 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:25:05.106 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 123390 2024-08-17T17:25:05.106 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.106 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:25:05.106 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.106 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:25:05.106 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.342371+0000", 2024-08-17T17:25:05.106 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.106 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.106 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.106 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:25:05.107 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.342371+0000", 2024-08-17T17:25:05.107 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.107 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.107 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.107 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:25:05.107 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.342370+0000", 2024-08-17T17:25:05.107 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:25:05.107 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.107 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.107 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:25:05.107 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.342378+0000", 2024-08-17T17:25:05.107 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.8189999999999992e-06 2024-08-17T17:25:05.107 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.108 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.108 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:25:05.108 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.342398+0000", 2024-08-17T17:25:05.108 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9950999999999999e-05 2024-08-17T17:25:05.108 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.108 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.108 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:25:05.108 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.342404+0000", 2024-08-17T17:25:05.108 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3640000000000004e-06 2024-08-17T17:25:05.108 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.108 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.108 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.108 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.480861+0000", 2024-08-17T17:25:05.109 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.13845644700000001 2024-08-17T17:25:05.109 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.109 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.109 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for degraded object", 2024-08-17T17:25:05.109 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.481300+0000", 2024-08-17T17:25:05.109 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000439238 2024-08-17T17:25:05.109 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.109 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.109 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.109 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.499133+0000", 2024-08-17T17:25:05.109 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017833203999999998 2024-08-17T17:25:05.109 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.109 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.109 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:25:05.110 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.499249+0000", 2024-08-17T17:25:05.110 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000115487 2024-08-17T17:25:05.110 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.110 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.110 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:25:05.110 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.499434+0000", 2024-08-17T17:25:05.110 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00018509099999999999 2024-08-17T17:25:05.110 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.110 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.110 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:25:05.110 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.550475+0000", 2024-08-17T17:25:05.110 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.051041198000000003 2024-08-17T17:25:05.110 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.111 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.111 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:25:05.111 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.571352+0000", 2024-08-17T17:25:05.111 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.020877601999999999 2024-08-17T17:25:05.111 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.111 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.111 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:25:05.111 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.571399+0000", 2024-08-17T17:25:05.111 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6313000000000002e-05 2024-08-17T17:25:05.111 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.111 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.111 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:25:05.111 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.571438+0000", 2024-08-17T17:25:05.112 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.875e-05 2024-08-17T17:25:05.112 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.112 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.112 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:25:05.112 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.575083+0000", 2024-08-17T17:25:05.112 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0036457149999999999 2024-08-17T17:25:05.112 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.112 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:25:05.112 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.112 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.112 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.112 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:123398 5.9 5:987f7dcf:::benchmark_data_smithi003_140793_object21341:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-08-17T17:25:05.113 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:21:15.353598+0000", 2024-08-17T17:25:05.113 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 229.64738780499999, 2024-08-17T17:25:05.113 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.224712101, 2024-08-17T17:25:05.113 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:25:05.113 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:25:05.113 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:25:05.113 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:25:05.113 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:25:05.113 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 123398 2024-08-17T17:25:05.113 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.113 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:25:05.113 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.113 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:25:05.114 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.353598+0000", 2024-08-17T17:25:05.114 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.114 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.114 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.114 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:25:05.114 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.353598+0000", 2024-08-17T17:25:05.114 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.114 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.114 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.114 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:25:05.114 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.353597+0000", 2024-08-17T17:25:05.114 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:25:05.114 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.115 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.115 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:25:05.115 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.353603+0000", 2024-08-17T17:25:05.115 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.7800000000000003e-06 2024-08-17T17:25:05.115 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.115 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.115 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:25:05.115 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.353618+0000", 2024-08-17T17:25:05.116 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4586000000000001e-05 2024-08-17T17:25:05.116 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.116 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.116 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:25:05.116 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.353623+0000", 2024-08-17T17:25:05.116 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4600000000000002e-06 2024-08-17T17:25:05.116 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.116 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.116 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.116 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.481319+0000", 2024-08-17T17:25:05.116 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.12769579 2024-08-17T17:25:05.117 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.117 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.117 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for degraded object", 2024-08-17T17:25:05.117 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.481651+0000", 2024-08-17T17:25:05.117 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00033159999999999998 2024-08-17T17:25:05.117 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.117 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.117 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.117 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.512806+0000", 2024-08-17T17:25:05.117 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.031155527999999998 2024-08-17T17:25:05.118 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.118 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.118 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:25:05.118 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.512914+0000", 2024-08-17T17:25:05.118 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000108096 2024-08-17T17:25:05.118 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.118 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.118 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:25:05.118 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.513063+0000", 2024-08-17T17:25:05.118 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000149018 2024-08-17T17:25:05.118 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.118 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.118 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:25:05.118 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.550483+0000", 2024-08-17T17:25:05.119 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.037419397 2024-08-17T17:25:05.119 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.119 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.119 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:25:05.119 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.575125+0000", 2024-08-17T17:25:05.119 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.024642081 2024-08-17T17:25:05.119 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.119 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.119 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:25:05.119 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.575148+0000", 2024-08-17T17:25:05.119 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3257000000000001e-05 2024-08-17T17:25:05.119 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.119 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.120 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:25:05.120 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.575183+0000", 2024-08-17T17:25:05.120 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4714e-05 2024-08-17T17:25:05.120 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.120 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.120 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:25:05.120 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.578310+0000", 2024-08-17T17:25:05.120 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0031274599999999999 2024-08-17T17:25:05.120 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.120 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:25:05.120 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.120 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.120 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.121 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:123404 5.9 5:9c306129:::benchmark_data_smithi003_140793_object21347:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-08-17T17:25:05.121 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:21:15.365046+0000", 2024-08-17T17:25:05.121 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 229.635939721, 2024-08-17T17:25:05.121 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.229434849, 2024-08-17T17:25:05.121 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:25:05.121 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:25:05.121 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:25:05.121 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:25:05.121 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:25:05.121 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 123404 2024-08-17T17:25:05.121 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.121 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:25:05.121 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.121 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:25:05.122 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.365046+0000", 2024-08-17T17:25:05.122 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.122 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.122 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.122 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:25:05.122 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.365046+0000", 2024-08-17T17:25:05.122 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.122 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.122 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.122 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:25:05.122 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.365045+0000", 2024-08-17T17:25:05.122 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:25:05.122 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.122 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.123 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:25:05.123 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.365052+0000", 2024-08-17T17:25:05.123 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.2899999999999997e-06 2024-08-17T17:25:05.123 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.123 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.123 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:25:05.123 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.365066+0000", 2024-08-17T17:25:05.123 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3431000000000001e-05 2024-08-17T17:25:05.123 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.123 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.123 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:25:05.123 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.365070+0000", 2024-08-17T17:25:05.123 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.8099999999999997e-06 2024-08-17T17:25:05.123 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.124 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.124 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.124 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.481725+0000", 2024-08-17T17:25:05.124 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.116654723 2024-08-17T17:25:05.124 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.124 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.124 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for degraded object", 2024-08-17T17:25:05.124 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.482113+0000", 2024-08-17T17:25:05.124 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000387718 2024-08-17T17:25:05.124 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.124 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.124 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.124 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.525780+0000", 2024-08-17T17:25:05.124 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.043667435999999997 2024-08-17T17:25:05.125 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.125 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.125 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:25:05.125 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.525897+0000", 2024-08-17T17:25:05.125 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000116777 2024-08-17T17:25:05.125 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.125 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.125 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:25:05.125 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.526072+0000", 2024-08-17T17:25:05.125 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00017510900000000001 2024-08-17T17:25:05.125 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.125 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.125 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:25:05.125 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.560296+0000", 2024-08-17T17:25:05.126 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.034223441 2024-08-17T17:25:05.126 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.126 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.126 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:25:05.126 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.585404+0000", 2024-08-17T17:25:05.126 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.025108325000000001 2024-08-17T17:25:05.126 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.126 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.126 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:25:05.126 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.590781+0000", 2024-08-17T17:25:05.126 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0053768749999999997 2024-08-17T17:25:05.126 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.126 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.126 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:25:05.127 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.590842+0000", 2024-08-17T17:25:05.127 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.122e-05 2024-08-17T17:25:05.127 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.127 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.127 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:25:05.127 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.594481+0000", 2024-08-17T17:25:05.127 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003639144 2024-08-17T17:25:05.127 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.127 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:25:05.127 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.127 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.127 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.127 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:123412 5.9 5:956aa967:::benchmark_data_smithi003_140793_object21355:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-08-17T17:25:05.127 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:21:15.379092+0000", 2024-08-17T17:25:05.128 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 229.62189416499999, 2024-08-17T17:25:05.128 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.219860472, 2024-08-17T17:25:05.128 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:25:05.128 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:25:05.128 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:25:05.128 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:25:05.128 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:25:05.128 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 123412 2024-08-17T17:25:05.128 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.128 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:25:05.128 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.128 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:25:05.128 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.379092+0000", 2024-08-17T17:25:05.128 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.129 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.129 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.129 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:25:05.129 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.379092+0000", 2024-08-17T17:25:05.129 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.129 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.129 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.129 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:25:05.129 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.379091+0000", 2024-08-17T17:25:05.129 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:25:05.129 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.129 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.129 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:25:05.129 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.379097+0000", 2024-08-17T17:25:05.130 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6289999999999998e-06 2024-08-17T17:25:05.130 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.130 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.130 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:25:05.130 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.379108+0000", 2024-08-17T17:25:05.130 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1462e-05 2024-08-17T17:25:05.130 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.130 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.130 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:25:05.130 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.379113+0000", 2024-08-17T17:25:05.130 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4179999999999999e-06 2024-08-17T17:25:05.130 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.130 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.131 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.131 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.482141+0000", 2024-08-17T17:25:05.131 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.10302826700000001 2024-08-17T17:25:05.131 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.131 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.131 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for degraded object", 2024-08-17T17:25:05.131 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.482496+0000", 2024-08-17T17:25:05.131 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00035522499999999997 2024-08-17T17:25:05.131 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.131 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.131 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.131 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.537376+0000", 2024-08-17T17:25:05.131 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.054880197999999998 2024-08-17T17:25:05.131 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.132 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.132 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:25:05.132 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.537487+0000", 2024-08-17T17:25:05.132 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011077 2024-08-17T17:25:05.132 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.132 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.132 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:25:05.132 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.537644+0000", 2024-08-17T17:25:05.132 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000156737 2024-08-17T17:25:05.132 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.132 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.132 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:25:05.132 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.568536+0000", 2024-08-17T17:25:05.132 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.030891734000000001 2024-08-17T17:25:05.133 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.133 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.133 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:25:05.133 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.594545+0000", 2024-08-17T17:25:05.133 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.026009391999999999 2024-08-17T17:25:05.133 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.133 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.133 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:25:05.133 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.594585+0000", 2024-08-17T17:25:05.133 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9771999999999998e-05 2024-08-17T17:25:05.133 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.133 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.133 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:25:05.133 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.594642+0000", 2024-08-17T17:25:05.134 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7328999999999999e-05 2024-08-17T17:25:05.134 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.134 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.134 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:25:05.134 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.598952+0000", 2024-08-17T17:25:05.134 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0043104119999999996 2024-08-17T17:25:05.134 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.134 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:25:05.134 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.134 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.134 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.134 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:123413 5.9 5:9d0beec9:::benchmark_data_smithi003_140793_object21356:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-08-17T17:25:05.134 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:21:15.381672+0000", 2024-08-17T17:25:05.134 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 229.61931418399999, 2024-08-17T17:25:05.135 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.221532805, 2024-08-17T17:25:05.135 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:25:05.135 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:25:05.135 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:25:05.135 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:25:05.135 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:25:05.135 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 123413 2024-08-17T17:25:05.135 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.135 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:25:05.135 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.135 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:25:05.135 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.381672+0000", 2024-08-17T17:25:05.135 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.135 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.136 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.136 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:25:05.136 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.381672+0000", 2024-08-17T17:25:05.136 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:25:05.136 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.136 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.136 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:25:05.136 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.381671+0000", 2024-08-17T17:25:05.136 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:25:05.136 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.136 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.136 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:25:05.136 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.381676+0000", 2024-08-17T17:25:05.136 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5480000000000001e-06 2024-08-17T17:25:05.137 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.137 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.137 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:25:05.137 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.381690+0000", 2024-08-17T17:25:05.137 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3669000000000001e-05 2024-08-17T17:25:05.137 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.137 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.137 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:25:05.137 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.381695+0000", 2024-08-17T17:25:05.137 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2129999999999999e-06 2024-08-17T17:25:05.137 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.137 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.137 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.138 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.482521+0000", 2024-08-17T17:25:05.138 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.10082545599999999 2024-08-17T17:25:05.138 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.138 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.138 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for degraded object", 2024-08-17T17:25:05.138 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.482845+0000", 2024-08-17T17:25:05.138 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00032396699999999998 2024-08-17T17:25:05.138 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.138 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.138 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:25:05.138 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.547941+0000", 2024-08-17T17:25:05.138 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.065095960999999994 2024-08-17T17:25:05.138 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.139 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.139 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:25:05.139 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.548021+0000", 2024-08-17T17:25:05.139 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.9985000000000001e-05 2024-08-17T17:25:05.139 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.139 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.139 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:25:05.139 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.548128+0000", 2024-08-17T17:25:05.139 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000107617 2024-08-17T17:25:05.139 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.139 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.139 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:25:05.139 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.575093+0000", 2024-08-17T17:25:05.139 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.026964424000000001 2024-08-17T17:25:05.140 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.140 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.140 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:25:05.140 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.599019+0000", 2024-08-17T17:25:05.140 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.023926469999999998 2024-08-17T17:25:05.140 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.140 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.140 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:25:05.140 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.599061+0000", 2024-08-17T17:25:05.140 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1869999999999997e-05 2024-08-17T17:25:05.140 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.140 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.140 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:25:05.140 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.599144+0000", 2024-08-17T17:25:05.141 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.2696999999999998e-05 2024-08-17T17:25:05.141 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:25:05.141 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:25:05.141 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:25:05.141 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:15.603205+0000", 2024-08-17T17:25:05.141 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0040610020000000002 2024-08-17T17:25:05.141 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.141 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:25:05.141 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.141 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:25:05.141 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:25:05.141 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:25:05.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:05.143 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:05.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:05.143 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 36 16 51106 51090 88.6339 2.3125 0.00982214 0.0110161 2024-08-17T17:25:05.244 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:05.267 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:05.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:05.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:05.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:04 smithi003 ceph-mon[100615]: pgmap v482: 89 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s; 9800/142614 objects degraded (6.872%); 2313/142614 objects misplaced (1.622%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:25:05.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:04 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-17T17:25:05.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:04 smithi003 ceph-mon[100615]: osdmap e197: 8 total, 8 up, 7 in 2024-08-17T17:25:05.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:04 smithi003 ceph-mon[100615]: osdmap e198: 8 total, 8 up, 7 in 2024-08-17T17:25:05.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:04 smithi003 ceph-mon[103344]: pgmap v482: 89 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s; 9800/142614 objects degraded (6.872%); 2313/142614 objects misplaced (1.622%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:25:05.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:04 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-17T17:25:05.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:04 smithi003 ceph-mon[103344]: osdmap e197: 8 total, 8 up, 7 in 2024-08-17T17:25:05.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:04 smithi003 ceph-mon[103344]: osdmap e198: 8 total, 8 up, 7 in 2024-08-17T17:25:05.369 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:05.394 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:05.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:05.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:05.496 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:05.527 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:05.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:05.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:05.629 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:05.654 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:05.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:05.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:05.756 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:05.793 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:05.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:05.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:05.860 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 7 1 2024-08-17T17:25:05.895 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:05.903 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:25:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:25:05] "GET /metrics HTTP/1.1" 200 36522 "" "Prometheus/2.51.0" 2024-08-17T17:25:05.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:05.941 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:05.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:06.042 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:06.069 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:06.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:06.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:06.081 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 37 16 51562 51546 87.0096 28.5 0.00776998 0.0114852 2024-08-17T17:25:06.171 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:06.220 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:06.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:06.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:06.322 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:06.348 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:06.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:06.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:06.450 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:06.492 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:06.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:06.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:06.556 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:25:06.594 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:06.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:06.649 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:06.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:06.750 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:06.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:06 smithi003 ceph-mon[100615]: osdmap e199: 8 total, 8 up, 7 in 2024-08-17T17:25:06.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:06 smithi003 ceph-mon[103344]: osdmap e199: 8 total, 8 up, 7 in 2024-08-17T17:25:06.783 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:06 smithi053 ceph-mon[89963]: osdmap e199: 8 total, 8 up, 7 in 2024-08-17T17:25:06.785 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-17T17:25:06.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:06.790 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:06.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:06.891 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:06.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:06.956 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:06.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:07.058 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:07.082 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 38 16 52556 52540 86.354 62.125 0.034889 0.0115725 2024-08-17T17:25:07.086 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:07.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:07.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:07.188 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:07.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:07.227 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:07.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:07.328 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:07.361 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:07.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:07.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:07.463 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:07.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:07.486 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:07.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:07.587 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:07.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:07.632 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:07.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:07.733 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:07.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:07 smithi003 ceph-mon[100615]: pgmap v486: 89 pgs: 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.0 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 60 MiB/s wr, 956 op/s; 7431/149388 objects degraded (4.974%); 4495/149388 objects misplaced (3.009%); 2.9 MiB/s, 46 objects/s recovering 2024-08-17T17:25:07.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:07 smithi003 ceph-mon[103344]: pgmap v486: 89 pgs: 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.0 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 60 MiB/s wr, 956 op/s; 7431/149388 objects degraded (4.974%); 4495/149388 objects misplaced (3.009%); 2.9 MiB/s, 46 objects/s recovering 2024-08-17T17:25:07.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:07.774 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:07.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:07.875 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:07.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:07.910 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:07.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:07.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:07 smithi053 ceph-mon[89963]: pgmap v486: 89 pgs: 2 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.0 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 60 MiB/s wr, 956 op/s; 7431/149388 objects degraded (4.974%); 4495/149388 objects misplaced (3.009%); 2.9 MiB/s, 46 objects/s recovering 2024-08-17T17:25:08.011 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:08.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:08.072 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:08.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:08.082 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 39 16 53570 53554 85.765 63.375 0.0210134 0.0116522 2024-08-17T17:25:08.173 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:08.260 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:08.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:08.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:08.362 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:08.392 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:08.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:08.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:08.478 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:25:08.494 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:08.551 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:08.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:08.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:08.652 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:08.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:08.683 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:08.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:08.784 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:08.822 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:08.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:08.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:08.924 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:08.974 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:08.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:08.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:09.076 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:09.082 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-08-17T17:25:09.082917+0000 min lat: 0.00112466 max lat: 2.58819 avg lat: 0.0117362 2024-08-17T17:25:09.083 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:25:09.083 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 40 16 54556 54540 85.1613 61.625 0.0113794 0.0117362 2024-08-17T17:25:09.110 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:09.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:09.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:09.212 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:09.237 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:09 smithi003 ceph-mon[103344]: pgmap v487: 89 pgs: 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 60 MiB/s wr, 952 op/s; 13819/152964 objects degraded (9.034%); 5163/152964 objects misplaced (3.375%); 4.8 MiB/s, 76 objects/s recovering 2024-08-17T17:25:09.237 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:25:09.238 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:25:09.238 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:25:09.238 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:09 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 13819/152964 objects degraded (9.034%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:09.238 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:09 smithi003 ceph-mon[100615]: pgmap v487: 89 pgs: 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 60 MiB/s wr, 952 op/s; 13819/152964 objects degraded (9.034%); 5163/152964 objects misplaced (3.375%); 4.8 MiB/s, 76 objects/s recovering 2024-08-17T17:25:09.238 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:25:09.238 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:25:09.238 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:25:09.238 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:09 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 13819/152964 objects degraded (9.034%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:09.243 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:09.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:09.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:09.263 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:25:09.263 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:25:09.277 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:25:09.277 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:25:09.317 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:25:09.317 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:25:09.345 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:09.362 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:25:09.362 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:25:09.388 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:09.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:09.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:09.392 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:25:09.392 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:25:09.425 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:25:09.425 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:25:09.462 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:25:09.462 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:25:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:09 smithi053 ceph-mon[89963]: pgmap v487: 89 pgs: 2 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 60 MiB/s wr, 952 op/s; 13819/152964 objects degraded (9.034%); 5163/152964 objects misplaced (3.375%); 4.8 MiB/s, 76 objects/s recovering 2024-08-17T17:25:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:25:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:25:09.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:25:09.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:09 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 13819/152964 objects degraded (9.034%), 4 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:09.490 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:09.493 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:25:09.494 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:25:09.536 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:09.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:09.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:09.638 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:09.667 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:09.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:09.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:09.769 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:09.807 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:09.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:09.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:09.856 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:25:09.910 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:09.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:09.962 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:09.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:10.043 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:25:10.063 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:10.083 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 41 16 55608 55592 84.6875 65.75 0.00878317 0.0118004 2024-08-17T17:25:10.105 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:10 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/363655483' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:25:10.115 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:10.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:10.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:10.217 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:10.250 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:10.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:10.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:10.352 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:10.381 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:10.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:10.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:10.408 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:10 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/363655483' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:25:10.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:10 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/363655483' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:25:10.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:25:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:25:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:25:10.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:10.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:10.524 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:10.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:10.625 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:10.666 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:10.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:10.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:10.768 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:10.799 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:10.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:10.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:10.901 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:10.932 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:10.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:10.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:11.034 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:11.065 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:11.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:11.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:11.083 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 42 16 56756 56740 84.3795 71.75 0.00885068 0.0118449 2024-08-17T17:25:11.166 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:11.197 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:11.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:11.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:11.299 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:11.343 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:11.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:11.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:11.445 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:11.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:11 smithi053 ceph-mon[89963]: pgmap v488: 89 pgs: 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 52 MiB/s wr, 830 op/s; 16927/157563 objects degraded (10.743%); 5163/157563 objects misplaced (3.277%); 4.2 MiB/s, 67 objects/s recovering 2024-08-17T17:25:11.489 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:11.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:11.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:11.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:11 smithi003 ceph-mon[103344]: pgmap v488: 89 pgs: 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 52 MiB/s wr, 830 op/s; 16927/157563 objects degraded (10.743%); 5163/157563 objects misplaced (3.277%); 4.2 MiB/s, 67 objects/s recovering 2024-08-17T17:25:11.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:11 smithi003 ceph-mon[100615]: pgmap v488: 89 pgs: 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 52 MiB/s wr, 830 op/s; 16927/157563 objects degraded (10.743%); 5163/157563 objects misplaced (3.277%); 4.2 MiB/s, 67 objects/s recovering 2024-08-17T17:25:11.590 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:11.629 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:11.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:11.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:11.731 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:11.761 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:11.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:11.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:11.863 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:11.889 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:11.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:11.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:11.991 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:12.022 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:12.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:12.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:12.083 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 43 16 57880 57864 84.0509 70.25 0.00522044 0.0118914 2024-08-17T17:25:12.124 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:12.170 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:12.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:12.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:12.271 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:12.322 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:12.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:12.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:12.424 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:12.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:12.470 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:12.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:12.571 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:12.605 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:12.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:12.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:12.707 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:12.740 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:12.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:12.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:12.842 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:12.887 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:12.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:12.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:12.989 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:13.018 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:13.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:13.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:13.086 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 44 16 58948 58932 83.6528 66.75 0.0132593 0.0119468 2024-08-17T17:25:13.120 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:13.153 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:13.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:13.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:13.230 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:25:13.255 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:13.268 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:13 smithi003 ceph-mon[103344]: pgmap v489: 89 pgs: 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 53 MiB/s wr, 840 op/s; 16927/165819 objects degraded (10.208%); 4736/165819 objects misplaced (2.856%); 6.6 MiB/s, 105 objects/s recovering 2024-08-17T17:25:13.268 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:13 smithi003 ceph-mon[100615]: pgmap v489: 89 pgs: 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 53 MiB/s wr, 840 op/s; 16927/165819 objects degraded (10.208%); 4736/165819 objects misplaced (2.856%); 6.6 MiB/s, 105 objects/s recovering 2024-08-17T17:25:13.282 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:13.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:13.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:13.384 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:13.412 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:13.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:13.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:13.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:13 smithi053 ceph-mon[89963]: pgmap v489: 89 pgs: 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 53 MiB/s wr, 840 op/s; 16927/165819 objects degraded (10.208%); 4736/165819 objects misplaced (2.856%); 6.6 MiB/s, 105 objects/s recovering 2024-08-17T17:25:13.514 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:13.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:13.561 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:13.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:13.662 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:13.697 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:13.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:13.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:13.799 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:13.829 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:13.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:13.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:13.931 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:13.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:13.976 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:13.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:14.077 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:14.085 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 45 16 59975 59959 83.2204 64.1875 0.0100087 0.0120105 2024-08-17T17:25:14.119 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:14.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:14.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:14.221 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:14.244 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:14.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:14.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:14.346 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:14.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:14.388 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:14.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:14.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:14 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 16927/169995 objects degraded (9.957%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:14.489 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:14.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:14 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 16927/169995 objects degraded (9.957%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:14.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:14 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 16927/169995 objects degraded (9.957%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:14.529 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:14.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:14.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:14.630 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:14.671 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:14.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:14.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:14.773 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:14.807 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:14.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:14.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:14.909 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:14.944 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:14.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:14.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:15.046 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:15.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:15.086 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:15.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:15.087 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 46 16 60993 60977 82.7945 63.625 0.0115488 0.0120705 2024-08-17T17:25:15.188 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:15.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:15.216 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:15.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:15.318 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:15.359 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:15.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:15.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:15.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:15 smithi003 ceph-mon[103344]: pgmap v490: 89 pgs: 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 53 MiB/s wr, 855 op/s; 16927/169995 objects degraded (9.957%); 4736/169995 objects misplaced (2.786%); 5.6 MiB/s, 88 objects/s recovering 2024-08-17T17:25:15.425 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:15 smithi003 ceph-mon[100615]: pgmap v490: 89 pgs: 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 53 MiB/s wr, 855 op/s; 16927/169995 objects degraded (9.957%); 4736/169995 objects misplaced (2.786%); 5.6 MiB/s, 88 objects/s recovering 2024-08-17T17:25:15.461 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:15.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:15 smithi053 ceph-mon[89963]: pgmap v490: 89 pgs: 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 53 MiB/s wr, 855 op/s; 16927/169995 objects degraded (9.957%); 4736/169995 objects misplaced (2.786%); 5.6 MiB/s, 88 objects/s recovering 2024-08-17T17:25:15.487 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:15.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:15.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:15.589 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:15.620 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:15.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:15.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:15.722 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:15.744 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:15.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:15.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:15.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:25:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:25:15] "GET /metrics HTTP/1.1" 200 36539 "" "Prometheus/2.51.0" 2024-08-17T17:25:15.846 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:15.900 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:15.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:15.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:16.002 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:16.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:16.050 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:16.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:16.086 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 47 16 61826 61810 82.1409 52.0625 0.00566522 0.0121665 2024-08-17T17:25:16.152 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:16.189 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:16.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:16.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:16.291 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:16.319 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:16.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:16.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:16.421 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:16.450 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:16.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:16.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:16.552 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:16.577 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:16.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:16.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:16.680 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:16.707 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:16.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:16.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:16.810 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:16.835 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:16.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:16.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:16.938 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:16.980 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:16.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:16.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:17.082 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:17.086 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 48 16 62667 62651 81.5249 52.5625 0.0293259 0.0122592 2024-08-17T17:25:17.128 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:17.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:17.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:17.229 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:17.275 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:17.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:17.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:17.377 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:17.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:17.420 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:17.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:17.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:17 smithi053 ceph-mon[89963]: pgmap v491: 89 pgs: 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 56 MiB/s wr, 893 op/s; 16927/177141 objects degraded (9.556%); 4472/177141 objects misplaced (2.525%); 5.5 MiB/s, 87 objects/s recovering 2024-08-17T17:25:17.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:17 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3640893211' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:25:17.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:17 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:25:17.500 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-08-17T17:25:17.501 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:17 smithi003 ceph-mon[103344]: pgmap v491: 89 pgs: 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 56 MiB/s wr, 893 op/s; 16927/177141 objects degraded (9.556%); 4472/177141 objects misplaced (2.525%); 5.5 MiB/s, 87 objects/s recovering 2024-08-17T17:25:17.502 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:17 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3640893211' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:25:17.502 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:17 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:25:17.503 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:17 smithi003 ceph-mon[100615]: pgmap v491: 89 pgs: 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 56 MiB/s wr, 893 op/s; 16927/177141 objects degraded (9.556%); 4472/177141 objects misplaced (2.525%); 5.5 MiB/s, 87 objects/s recovering 2024-08-17T17:25:17.503 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:17 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3640893211' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:25:17.503 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:17 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:25:17.522 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:17.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:17.565 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:17.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:17.667 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:17.697 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:17.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:17.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:17.799 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:17.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:17.826 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:17.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:17.927 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:17.958 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:17.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:17.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:18.060 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:18.086 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 49 16 63501 63485 80.9252 52.125 0.0257457 0.0123507 2024-08-17T17:25:18.092 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:18.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:18.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:18.194 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:18.222 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:18.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:18.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:18.324 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:18.367 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:18.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:18.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:18.469 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:18.497 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:18.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:18.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:18.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:18 smithi003 ceph-mon[103344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:25:18.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:18 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:25:18.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:18 smithi003 ceph-mon[103344]: osdmap e200: 8 total, 8 up, 7 in 2024-08-17T17:25:18.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:18 smithi003 ceph-mon[100615]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:25:18.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:18 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:25:18.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:18 smithi003 ceph-mon[100615]: osdmap e200: 8 total, 8 up, 7 in 2024-08-17T17:25:18.599 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:18.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:18.629 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:18.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:18.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:18 smithi053 ceph-mon[89963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:25:18.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:18 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:25:18.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:18 smithi053 ceph-mon[89963]: osdmap e200: 8 total, 8 up, 7 in 2024-08-17T17:25:18.730 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:18.769 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:18.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:18.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:18.871 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:18.899 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:18.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:18.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:19.001 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:19.043 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:19.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:19.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:19.086 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 50 16 64477 64461 80.5268 61 0.00816262 0.012412 2024-08-17T17:25:19.145 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:19.185 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:19.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:19.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:19.288 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:19.319 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:19.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:19.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:19.420 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:19.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:19.453 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:19.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:19.554 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:19.598 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:19.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:19.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:19.700 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:19.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:19 smithi053 ceph-mon[89963]: pgmap v493: 89 pgs: 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 60 MiB/s wr, 962 op/s; 16927/181872 objects degraded (9.307%); 4319/181872 objects misplaced (2.375%); 5.3 MiB/s, 84 objects/s recovering 2024-08-17T17:25:19.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:19 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:25:19.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:19.752 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:19.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:19.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:19 smithi003 ceph-mon[103344]: pgmap v493: 89 pgs: 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 60 MiB/s wr, 962 op/s; 16927/181872 objects degraded (9.307%); 4319/181872 objects misplaced (2.375%); 5.3 MiB/s, 84 objects/s recovering 2024-08-17T17:25:19.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:19 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:25:19.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:19 smithi003 ceph-mon[100615]: pgmap v493: 89 pgs: 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 60 MiB/s wr, 962 op/s; 16927/181872 objects degraded (9.307%); 4319/181872 objects misplaced (2.375%); 5.3 MiB/s, 84 objects/s recovering 2024-08-17T17:25:19.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:19 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:25:19.853 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:19.884 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:25:19.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:19.898 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:19.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:20.000 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:20.035 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:20.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:20.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:20.087 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 51 16 65534 65518 80.2431 66.0625 0.0116297 0.0124565 2024-08-17T17:25:20.137 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:20.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:20.204 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:20.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:20.305 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:20.337 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:20.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:20.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:20.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:20.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:20 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 16927/181872 objects degraded (9.307%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:20.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:25:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:25:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:25:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:20.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:20.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:20.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:20 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 16927/181872 objects degraded (9.307%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:20.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:20 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 16927/181872 objects degraded (9.307%), 5 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:20.584 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:20.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:20.635 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:20.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:20.736 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:20.773 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:20.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:20.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:20.875 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:20.921 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:20.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:20.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:21.023 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:21.079 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:21.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:21.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:21.086 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 52 16 66456 66440 79.8082 57.625 0.0100219 0.0125233 2024-08-17T17:25:21.091 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:25:21.091 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:25:21.092 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:25:21.182 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:21.209 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:21.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:21.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:21.312 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:21.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:21.398 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:21.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:21.499 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:21.534 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:21.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:21.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:21.636 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:21.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:21.682 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:21.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:21.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:21 smithi053 ceph-mon[89963]: pgmap v494: 89 pgs: 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 63 MiB/s wr, 1.00k op/s; 16927/187686 objects degraded (9.019%); 4319/187686 objects misplaced (2.301%); 5.3 MiB/s, 84 objects/s recovering 2024-08-17T17:25:21.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:21 smithi003 ceph-mon[103344]: pgmap v494: 89 pgs: 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 63 MiB/s wr, 1.00k op/s; 16927/187686 objects degraded (9.019%); 4319/187686 objects misplaced (2.301%); 5.3 MiB/s, 84 objects/s recovering 2024-08-17T17:25:21.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:21 smithi003 ceph-mon[100615]: pgmap v494: 89 pgs: 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 63 MiB/s wr, 1.00k op/s; 16927/187686 objects degraded (9.019%); 4319/187686 objects misplaced (2.301%); 5.3 MiB/s, 84 objects/s recovering 2024-08-17T17:25:21.783 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:21.823 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:21.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:21.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:21.925 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:21.953 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:21.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:21.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:22.055 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:22.087 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 53 16 67395 67379 79.4097 58.6875 0.00833762 0.0125871 2024-08-17T17:25:22.088 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:22.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:22.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:22.190 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:22.229 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:22.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:22.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:22.330 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:22.334 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.7 to 1 (10000) 2024-08-17T17:25:22.358 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:22.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:22.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:22.459 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:22.460 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:22 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/4086443886' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:25:22.461 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:22 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:25:22.461 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:22 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/4086443886' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:25:22.461 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:22 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:25:22.495 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:22.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:22.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:22.597 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:22.629 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:22.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:22.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:22.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:22 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/4086443886' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:25:22.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:22 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:25:22.731 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:22.760 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:22.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:22.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:22.862 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:22.891 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:22.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:22.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:22.992 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:23.025 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:23.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:23.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:23.087 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 54 16 68422 68406 79.1269 64.1875 0.0103073 0.0126313 2024-08-17T17:25:23.127 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:23.154 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:23.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:23.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:23.256 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:23.293 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:23.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:23.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:23.395 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:23.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:23.441 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:23.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:23.542 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:23.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:23.579 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:23.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:23.603 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:23 smithi003 ceph-mon[103344]: pgmap v495: 89 pgs: 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 62 MiB/s wr, 990 op/s; 16927/195576 objects degraded (8.655%); 3871/195576 objects misplaced (1.979%); 5.4 MiB/s, 86 objects/s recovering 2024-08-17T17:25:23.603 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:23 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-17T17:25:23.603 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:23 smithi003 ceph-mon[103344]: osdmap e201: 8 total, 8 up, 7 in 2024-08-17T17:25:23.604 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:25:23.604 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:23 smithi003 ceph-mon[100615]: pgmap v495: 89 pgs: 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 62 MiB/s wr, 990 op/s; 16927/195576 objects degraded (8.655%); 3871/195576 objects misplaced (1.979%); 5.4 MiB/s, 86 objects/s recovering 2024-08-17T17:25:23.604 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:23 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-17T17:25:23.604 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:23 smithi003 ceph-mon[100615]: osdmap e201: 8 total, 8 up, 7 in 2024-08-17T17:25:23.604 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:25:23.605 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-17T17:25:23.680 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:23.707 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:23.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:23.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:23 smithi053 ceph-mon[89963]: pgmap v495: 89 pgs: 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 62 MiB/s wr, 990 op/s; 16927/195576 objects degraded (8.655%); 3871/195576 objects misplaced (1.979%); 5.4 MiB/s, 86 objects/s recovering 2024-08-17T17:25:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:23 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-17T17:25:23.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:23 smithi053 ceph-mon[89963]: osdmap e201: 8 total, 8 up, 7 in 2024-08-17T17:25:23.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:25:23.809 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:23.834 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:23.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:23.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:23.936 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:23.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:23.970 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:23.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:24.072 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:24.089 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 55 16 69404 69388 78.802 61.375 0.0437188 0.0126836 2024-08-17T17:25:24.122 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:24.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:24.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:24.224 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:24.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:24.264 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:24.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:24.365 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:24.407 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:24.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:24.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:24.509 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:24.538 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:24.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:24.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:24.640 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:24.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:24.662 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:24.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:24.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:24 smithi053 ceph-mon[89963]: osdmap e202: 8 total, 8 up, 7 in 2024-08-17T17:25:24.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:24 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:25:24.764 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:24.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:24 smithi003 ceph-mon[103344]: osdmap e202: 8 total, 8 up, 7 in 2024-08-17T17:25:24.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:24 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:25:24.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:24 smithi003 ceph-mon[100615]: osdmap e202: 8 total, 8 up, 7 in 2024-08-17T17:25:24.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:24 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:25:24.807 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:24.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:24.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:24.909 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:24.940 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:24.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:24.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:25.042 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:25.083 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:25.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:25.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:25.089 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 56 16 70200 70184 78.2834 49.75 0.00641007 0.0127676 2024-08-17T17:25:25.185 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:25.217 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:25.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:25.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:25.319 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:25.361 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:25.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:25.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:25.463 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:25.503 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:25.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:25.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:25.605 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:25.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:25.648 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:25.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:25.651 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:25 smithi003 ceph-mon[103344]: pgmap v498: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.1 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 55 MiB/s wr, 875 op/s; 16932/199374 objects degraded (8.493%); 3871/199374 objects misplaced (1.942%); 4.2 MiB/s, 67 objects/s recovering 2024-08-17T17:25:25.652 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:25 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:25:25.652 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:25 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 16932/199374 objects degraded (8.493%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:25.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:25 smithi003 ceph-mon[100615]: pgmap v498: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.1 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 55 MiB/s wr, 875 op/s; 16932/199374 objects degraded (8.493%); 3871/199374 objects misplaced (1.942%); 4.2 MiB/s, 67 objects/s recovering 2024-08-17T17:25:25.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:25 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:25:25.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:25 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 16932/199374 objects degraded (8.493%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:25.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:25 smithi053 ceph-mon[89963]: pgmap v498: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.1 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 55 MiB/s wr, 875 op/s; 16932/199374 objects degraded (8.493%); 3871/199374 objects misplaced (1.942%); 4.2 MiB/s, 67 objects/s recovering 2024-08-17T17:25:25.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:25 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:25:25.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:25 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 16932/199374 objects degraded (8.493%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:25.749 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:25.780 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:25.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:25.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:25.882 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:25.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:25.920 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:25.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:26.021 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:26.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:25:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:25:25] "GET /metrics HTTP/1.1" 200 36525 "" "Prometheus/2.51.0" 2024-08-17T17:25:26.054 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:26.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:26.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:26.089 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 57 16 71153 71137 77.955 59.5625 0.0187136 0.0128223 2024-08-17T17:25:26.156 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:26.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:26.189 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:26.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:26.290 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:26.319 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:26.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:26.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:26.420 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:26.453 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:26.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:26.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:26.555 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:26.590 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:26.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:26.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:26.692 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:26.734 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:26.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:26.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:26.836 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:26.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:26.866 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:26.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:26.967 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:27.023 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:27.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:27.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:27.089 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 58 16 72071 72055 77.6003 57.375 0.0176739 0.0128795 2024-08-17T17:25:27.125 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-08-17T17:25:27.125 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:25:27.125 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:27.152 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:27.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:27.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:27.173 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:25:27.254 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:27.280 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:27.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:27.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:27.382 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:27.409 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:27.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:27.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:27.510 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:27.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:27.559 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:27.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:27.660 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:27.685 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:27.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:27.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:27.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:27 smithi053 ceph-mon[89963]: pgmap v499: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 62 MiB/s wr, 989 op/s; 16932/205632 objects degraded (8.234%); 3568/205632 objects misplaced (1.735%); 5.9 MiB/s, 93 objects/s recovering 2024-08-17T17:25:27.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:27 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:25:27.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:27 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:25:27.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:27 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:25:27.760 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:27 smithi003 ceph-mon[100615]: pgmap v499: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 62 MiB/s wr, 989 op/s; 16932/205632 objects degraded (8.234%); 3568/205632 objects misplaced (1.735%); 5.9 MiB/s, 93 objects/s recovering 2024-08-17T17:25:27.760 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:27 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:25:27.760 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:27 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:25:27.760 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:27 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:25:27.761 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:27 smithi003 ceph-mon[103344]: pgmap v499: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 62 MiB/s wr, 989 op/s; 16932/205632 objects degraded (8.234%); 3568/205632 objects misplaced (1.735%); 5.9 MiB/s, 93 objects/s recovering 2024-08-17T17:25:27.761 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:27 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:25:27.761 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:27 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:25:27.761 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:27 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:25:27.787 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:27.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:27.842 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:27.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:27.943 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:27.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:27.987 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:27.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:28.088 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:28.089 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 59 16 72900 72884 77.1633 51.8125 0.0276225 0.012954 2024-08-17T17:25:28.114 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:28.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:28.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:28.216 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:28.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:28.270 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:28.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:28.372 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:28.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:28.405 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:28.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:28.506 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:28.535 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:28.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:28.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:28.637 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:28.671 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:28.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:28.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:28.773 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:28.812 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:28.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:28.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:28.896 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:25:28.914 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:28.942 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:28.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:28.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:29.043 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:29.072 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:29.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:29.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:29.090 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-08-17T17:25:29.089904+0000 min lat: 0.00112466 max lat: 2.58819 avg lat: 0.0130176 2024-08-17T17:25:29.090 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:25:29.091 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 60 16 73768 73752 76.7815 54.25 0.0202886 0.0130176 2024-08-17T17:25:29.174 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:29.217 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:25:29.217 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:25:29.223 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:29.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:29.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:29.256 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:25:29.257 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:25:29.301 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:25:29.301 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:25:29.326 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:29.339 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:25:29.339 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:25:29.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:29.359 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:29.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:29.371 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:25:29.371 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:25:29.405 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:25:29.405 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:25:29.439 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:25:29.439 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:25:29.460 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:29.471 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:25:29.472 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:25:29.493 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:29.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:29.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:29.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:29 smithi003 ceph-mon[103344]: pgmap v500: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 59 MiB/s wr, 937 op/s; 16932/210192 objects degraded (8.055%); 3393/210192 objects misplaced (1.614%); 7.2 MiB/s, 115 objects/s recovering 2024-08-17T17:25:29.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:29 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2906455742' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:25:29.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:29 smithi003 ceph-mon[100615]: pgmap v500: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 59 MiB/s wr, 937 op/s; 16932/210192 objects degraded (8.055%); 3393/210192 objects misplaced (1.614%); 7.2 MiB/s, 115 objects/s recovering 2024-08-17T17:25:29.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:29 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2906455742' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:25:29.595 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:29.621 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:29.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:29.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:29.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:29 smithi053 ceph-mon[89963]: pgmap v500: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 59 MiB/s wr, 937 op/s; 16932/210192 objects degraded (8.055%); 3393/210192 objects misplaced (1.614%); 7.2 MiB/s, 115 objects/s recovering 2024-08-17T17:25:29.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:29 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2906455742' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:25:29.723 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:29.772 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:29.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:29.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:29.875 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:29.879 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:25:29.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:29.932 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:29.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:30.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:30.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:30.083 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:30.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:30.089 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 61 16 74699 74683 76.4767 58.1875 0.00763282 0.0130694 2024-08-17T17:25:30.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:30.229 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:30.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:30.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:30.366 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:30.398 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:30.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:30.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:30.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:30 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 16932/210192 objects degraded (8.055%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:30.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:25:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:25:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:25:30.500 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:30.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:30 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 16932/210192 objects degraded (8.055%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:30.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:30 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 16932/210192 objects degraded (8.055%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:30.530 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:30.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:30.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:30.632 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:30.657 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:30.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:30.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:30.759 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:30.788 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:30.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:30.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:30.890 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:30.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:30.931 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:30.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:31.032 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:31.085 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:31.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:31.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:31.091 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 62 15 75532 75517 76.0823 52.125 0.0421903 0.013137 2024-08-17T17:25:31.188 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:31.213 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:31.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:31.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:31.314 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:31.341 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:31.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:31.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:31.443 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:31.458 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:31 smithi003 ceph-mon[100615]: pgmap v501: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 53 MiB/s wr, 845 op/s; 16932/215868 objects degraded (7.844%); 3393/215868 objects misplaced (1.572%); 3.7 MiB/s, 59 objects/s recovering 2024-08-17T17:25:31.459 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:31 smithi003 ceph-mon[103344]: pgmap v501: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 53 MiB/s wr, 845 op/s; 16932/215868 objects degraded (7.844%); 3393/215868 objects misplaced (1.572%); 3.7 MiB/s, 59 objects/s recovering 2024-08-17T17:25:31.497 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:31.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:31.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:31.599 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:31.635 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:31.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:31.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:31.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:31 smithi053 ceph-mon[89963]: pgmap v501: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 53 MiB/s wr, 845 op/s; 16932/215868 objects degraded (7.844%); 3393/215868 objects misplaced (1.572%); 3.7 MiB/s, 59 objects/s recovering 2024-08-17T17:25:31.737 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:31.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:31.786 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:31.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:31.887 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:31.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:31.931 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:31.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:32.032 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:32.058 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:32.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:32.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:32.091 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 63 16 76338 76322 75.6734 50.3125 0.0476686 0.0132075 2024-08-17T17:25:32.160 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:32.186 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:32.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:32.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:32.287 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:32.318 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:25:32.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:32.323 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:32.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:32.399 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:32 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2173217951' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:25:32.399 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:32 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:25:32.405 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-08-17T17:25:32.425 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:32.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:32.496 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:32.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:32.597 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:32.628 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:32.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:32.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:32.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:32 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2173217951' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:25:32.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:32 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:25:32.730 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:32.776 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:32 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2173217951' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:25:32.776 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:32 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:25:32.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:32.791 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:32.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:32.892 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:32.921 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:32.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:32.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:33.023 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:33.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:33.066 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:33.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:33.091 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 64 16 77235 77219 75.3671 56.0625 0.0115613 0.0132625 2024-08-17T17:25:33.167 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:33.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:33.215 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:33.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:33.316 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:33.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:33.388 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:33.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:33.489 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:33.528 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:33.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:33.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:33.630 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:33.645 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:33 smithi053 ceph-mon[89963]: pgmap v502: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 60 MiB/s wr, 964 op/s; 16932/223176 objects degraded (7.587%); 2850/223176 objects misplaced (1.277%); 6.7 MiB/s, 107 objects/s recovering 2024-08-17T17:25:33.645 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:33 smithi053 ceph-mon[89963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:25:33.645 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:33 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:25:33.645 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:33 smithi053 ceph-mon[89963]: osdmap e203: 8 total, 8 up, 7 in 2024-08-17T17:25:33.659 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:33 smithi003 ceph-mon[103344]: pgmap v502: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 60 MiB/s wr, 964 op/s; 16932/223176 objects degraded (7.587%); 2850/223176 objects misplaced (1.277%); 6.7 MiB/s, 107 objects/s recovering 2024-08-17T17:25:33.659 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:33 smithi003 ceph-mon[103344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:25:33.659 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:33 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:25:33.659 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:33 smithi003 ceph-mon[103344]: osdmap e203: 8 total, 8 up, 7 in 2024-08-17T17:25:33.660 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:33 smithi003 ceph-mon[100615]: pgmap v502: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 60 MiB/s wr, 964 op/s; 16932/223176 objects degraded (7.587%); 2850/223176 objects misplaced (1.277%); 6.7 MiB/s, 107 objects/s recovering 2024-08-17T17:25:33.660 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:33 smithi003 ceph-mon[100615]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:25:33.660 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:33 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:25:33.660 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:33 smithi003 ceph-mon[100615]: osdmap e203: 8 total, 8 up, 7 in 2024-08-17T17:25:33.668 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:33.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:33.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:33.770 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:33.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:33.810 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:33.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:33.911 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:33.957 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:33.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:33.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:34.059 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:34.091 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 65 16 78169 78153 75.1057 58.375 0.0182428 0.0133085 2024-08-17T17:25:34.099 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:34.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:34.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:34.201 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:34.248 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:34.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:34.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:34.350 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:34.391 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:34.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:34.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:34.493 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:34.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:34.516 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:34.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:34.617 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:34.654 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:34.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:34.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:34.756 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:34.797 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:34.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:34.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:34.899 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:34.941 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:34.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:34.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:35.042 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:35.082 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:35.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:35.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:35.091 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 66 16 79054 79038 74.8058 55.3125 0.033126 0.013362 2024-08-17T17:25:35.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:35.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:35.238 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:35.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:35.339 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:35.381 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:35.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:35.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:35.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:35.513 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:35.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:35.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:35.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:35 smithi003 ceph-mon[103344]: pgmap v504: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 56 MiB/s wr, 903 op/s; 16932/226479 objects degraded (7.476%); 2850/226479 objects misplaced (1.258%); 6.4 MiB/s, 102 objects/s recovering 2024-08-17T17:25:35.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:35 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 16932/226479 objects degraded (7.476%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:35.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:35 smithi003 ceph-mon[100615]: pgmap v504: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 56 MiB/s wr, 903 op/s; 16932/226479 objects degraded (7.476%); 2850/226479 objects misplaced (1.258%); 6.4 MiB/s, 102 objects/s recovering 2024-08-17T17:25:35.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:35 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 16932/226479 objects degraded (7.476%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:35.616 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:35.646 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:35.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:35.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:35.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:35 smithi053 ceph-mon[89963]: pgmap v504: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 56 MiB/s wr, 903 op/s; 16932/226479 objects degraded (7.476%); 2850/226479 objects misplaced (1.258%); 6.4 MiB/s, 102 objects/s recovering 2024-08-17T17:25:35.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:35 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 16932/226479 objects degraded (7.476%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:35.748 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:35.781 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:35.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:35.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:35.883 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:35.911 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:35.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:35.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:36.013 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:36.022 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:25:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:25:35] "GET /metrics HTTP/1.1" 200 36525 "" "Prometheus/2.51.0" 2024-08-17T17:25:36.048 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:36.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:36.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:36.092 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 67 16 79966 79950 74.5401 57 0.00889362 0.0134095 2024-08-17T17:25:36.149 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:36.194 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:36.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:36.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:36.247 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:25:36.249 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:25:36.296 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:36.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:36.336 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:36.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:36.437 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:36.477 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:36.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:36.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:36.579 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:36.601 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:36.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:36.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:36.703 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:36.746 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:36.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:36.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:36.848 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:36.876 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:36.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:36.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:36.978 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:37.011 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:37.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:37.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:37.092 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 68 16 80793 80777 74.2041 51.6875 0.0118898 0.0134704 2024-08-17T17:25:37.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:37.150 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:37.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:37.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:37.252 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:37.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:37.293 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:37.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:37.394 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:37.427 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:37.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:37.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:37.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:37 smithi003 ceph-mon[103344]: from='client.45569 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:25:37.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:37 smithi003 ceph-mon[103344]: pgmap v505: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 55 MiB/s wr, 884 op/s; 16932/232167 objects degraded (7.293%); 2454/232167 objects misplaced (1.057%); 7.0 MiB/s, 111 objects/s recovering 2024-08-17T17:25:37.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:37 smithi003 ceph-mon[100615]: from='client.45569 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:25:37.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:37 smithi003 ceph-mon[100615]: pgmap v505: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 55 MiB/s wr, 884 op/s; 16932/232167 objects degraded (7.293%); 2454/232167 objects misplaced (1.057%); 7.0 MiB/s, 111 objects/s recovering 2024-08-17T17:25:37.529 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:37.556 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:37.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:37.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:37.658 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:37.700 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:37.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:37.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:37.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:37 smithi053 ceph-mon[89963]: from='client.45569 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:25:37.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:37 smithi053 ceph-mon[89963]: pgmap v505: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 55 MiB/s wr, 884 op/s; 16932/232167 objects degraded (7.293%); 2454/232167 objects misplaced (1.057%); 7.0 MiB/s, 111 objects/s recovering 2024-08-17T17:25:37.802 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:37.831 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:37.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:37.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:37.932 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:37.954 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:37.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:37.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:38.056 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:38.089 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:38.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:38.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:38.091 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 69 16 81655 81639 73.9095 53.875 0.0235523 0.0135243 2024-08-17T17:25:38.190 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:38.222 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:38.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:38.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:38.326 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:38.367 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:38.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:38.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:38.468 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:38.498 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:38.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:38.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:38.600 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:38.632 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:38.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:38.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:38.734 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:38.763 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:38.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:38.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:38.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:38.910 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:38.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:38.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:39.012 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:39.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:39.057 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:39.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:39.092 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 70 16 82565 82549 73.6662 56.875 0.0312767 0.0135695 2024-08-17T17:25:39.158 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:39.192 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:39.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:39.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:39.294 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:39.339 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:39.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:39.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:39.373 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:39 smithi003 ceph-mon[100615]: pgmap v506: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 55 MiB/s wr, 873 op/s; 16932/236424 objects degraded (7.162%); 2267/236424 objects misplaced (0.959%); 7.0 MiB/s, 112 objects/s recovering 2024-08-17T17:25:39.373 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:25:39.373 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:25:39.373 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:39 smithi003 ceph-mon[103344]: pgmap v506: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 55 MiB/s wr, 873 op/s; 16932/236424 objects degraded (7.162%); 2267/236424 objects misplaced (0.959%); 7.0 MiB/s, 112 objects/s recovering 2024-08-17T17:25:39.373 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:25:39.373 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:25:39.441 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:39.463 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:39.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:39.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:39.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:39 smithi053 ceph-mon[89963]: pgmap v506: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 55 MiB/s wr, 873 op/s; 16932/236424 objects degraded (7.162%); 2267/236424 objects misplaced (0.959%); 7.0 MiB/s, 112 objects/s recovering 2024-08-17T17:25:39.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:25:39.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:25:39.489 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:25:39.489 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-08-17T17:25:39.489 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-08-17T17:25:39.489 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-08-17T17:25:39.489 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:25:39.565 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:39.587 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:39.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:39.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:39.689 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:39.713 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:39.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:39.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:39.815 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:39.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:39.838 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:39.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:39.940 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:39.964 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:39.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:39.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:40.066 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:40.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:40.089 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:40.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:40.092 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 71 16 82805 82789 72.8402 15 0.00384263 0.0135643 2024-08-17T17:25:40.189 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:40.212 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:40.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:40.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:40.314 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:40.336 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:40.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:40.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:40.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:40 smithi003 ceph-mon[100615]: osdmap e204: 8 total, 8 up, 7 in 2024-08-17T17:25:40.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:40 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 16932/236424 objects degraded (7.162%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:40.438 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:40 smithi003 ceph-mon[103344]: osdmap e204: 8 total, 8 up, 7 in 2024-08-17T17:25:40.438 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:40 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 16932/236424 objects degraded (7.162%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:40.439 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:40.462 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:40.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:40.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:40.466 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":504,"stamp":"2024-08-17T17:25:33.863931+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4950464461,"num_objects":75493,"num_object_clones":0,"num_object_copies":226479,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16932,"num_objects_misplaced":2850,"num_objects_unfound":0,"num_objects_dirty":75493,"num_whiteouts":0,"num_read":6376,"num_read_kb":13833,"num_write":76587,"num_write_kb":4844799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3508,"num_bytes_recovered":227616134,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76646,"ondisk_log_size":76646,"up":267,"acting":260,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":286,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":16923276,"kb_used_data":14960908,"kb_used_omap":33,"kb_used_meta":1962270,"kb_avail":639292788,"statfs":{"total":671965249536,"available":654635814912,"internally_reserved":0,"allocated":15319969792,"data_stored":15312000156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34068,"internal_metadata":2009365228},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,4,2,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":592117760,"num_objects":9035,"num_object_clones":0,"num_object_copies":27105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1021,"num_objects_unfound":0,"num_objects_dirty":9035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9035,"num_write_kb":578240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1021,"num_bytes_recovered":66912256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004473"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":257,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.343887+0000","last_change":"2024-08-17T17:17:32.495652+0000","last_active":"2024-08-17T17:25:23.343887+0000","last_peered":"2024-08-17T17:25:23.343887+0000","last_clean":"2024-08-17T17:25:23.343887+0000","last_became_active":"2024-08-17T17:17:32.495218+0000","last_became_peered":"2024-08-17T17:17:32.495218+0000","last_unstale":"2024-08-17T17:25:23.343887+0000","last_undegraded":"2024-08-17T17:25:23.343887+0000","last_fullsized":"2024-08-17T17:25:23.343887+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:11:45.821282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":178,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-08-17T17:25:22.327507+0000","last_change":"2024-08-17T17:18:59.424887+0000","last_active":"2024-08-17T17:25:22.327507+0000","last_peered":"2024-08-17T17:25:22.327507+0000","last_clean":"2024-08-17T17:25:22.327507+0000","last_became_active":"2024-08-17T17:18:59.411539+0000","last_became_peered":"2024-08-17T17:18:59.411539+0000","last_unstale":"2024-08-17T17:25:22.327507+0000","last_undegraded":"2024-08-17T17:25:22.327507+0000","last_fullsized":"2024-08-17T17:25:22.327507+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:46:20.082241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":19,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.345102+0000","last_change":"2024-08-17T17:24:53.460742+0000","last_active":"2024-08-17T17:25:23.345102+0000","last_peered":"2024-08-17T17:25:23.345102+0000","last_clean":"2024-08-17T17:25:23.345102+0000","last_became_active":"2024-08-17T17:24:53.460365+0000","last_became_peered":"2024-08-17T17:24:53.460365+0000","last_unstale":"2024-08-17T17:25:23.345102+0000","last_undegraded":"2024-08-17T17:25:23.345102+0000","last_fullsized":"2024-08-17T17:25:23.345102+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:24:08.116968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":70,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-08-17T17:25:32.400104+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:25:32.400104+0000","last_peered":"2024-08-17T17:25:32.400104+0000","last_clean":"2024-08-17T17:25:32.400104+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:25:32.400104+0000","last_undegraded":"2024-08-17T17:25:32.400104+0000","last_fullsized":"2024-08-17T17:25:32.400104+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":70,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-08-17T17:25:32.400419+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:25:32.400419+0000","last_peered":"2024-08-17T17:25:32.400419+0000","last_clean":"2024-08-17T17:25:32.400419+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:25:32.400419+0000","last_undegraded":"2024-08-17T17:25:32.400419+0000","last_fullsized":"2024-08-17T17:25:32.400419+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":153,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-08-17T17:25:22.327233+0000","last_change":"2024-08-17T17:20:49.258206+0000","last_active":"2024-08-17T17:25:22.327233+0000","last_peered":"2024-08-17T17:25:22.327233+0000","last_clean":"2024-08-17T17:25:22.327233+0000","last_became_active":"2024-08-17T17:20:49.257761+0000","last_became_peered":"2024-08-17T17:20:49.257761+0000","last_unstale":"2024-08-17T17:25:22.327233+0000","last_undegraded":"2024-08-17T17:25:22.327233+0000","last_fullsized":"2024-08-17T17:25:22.327233+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:38:35.363748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":271,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.341252+0000","last_change":"2024-08-17T17:19:57.514773+0000","last_active":"2024-08-17T17:25:23.341252+0000","last_peered":"2024-08-17T17:25:23.341252+0000","last_clean":"2024-08-17T17:25:23.341252+0000","last_became_active":"2024-08-17T17:19:57.513839+0000","last_became_peered":"2024-08-17T17:19:57.513839+0000","last_unstale":"2024-08-17T17:25:23.341252+0000","last_undegraded":"2024-08-17T17:25:23.341252+0000","last_fullsized":"2024-08-17T17:25:23.341252+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:42:48.063169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":249,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.336056+0000","last_change":"2024-08-17T17:24:53.465875+0000","last_active":"2024-08-17T17:25:23.336056+0000","last_peered":"2024-08-17T17:25:23.336056+0000","last_clean":"2024-08-17T17:25:23.336056+0000","last_became_active":"2024-08-17T17:24:53.458129+0000","last_became_peered":"2024-08-17T17:24:53.458129+0000","last_unstale":"2024-08-17T17:25:23.336056+0000","last_undegraded":"2024-08-17T17:25:23.336056+0000","last_fullsized":"2024-08-17T17:25:23.336056+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:51:33.584869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"203'4775","reported_seq":4820,"reported_epoch":203,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:25:32.878715+0000","last_change":"2024-08-17T17:24:54.892004+0000","last_active":"2024-08-17T17:25:32.878715+0000","last_peered":"2024-08-17T17:25:32.878715+0000","last_clean":"2024-08-17T17:24:52.438492+0000","last_became_active":"2024-08-17T17:24:54.887269+0000","last_became_peered":"2024-08-17T17:24:54.887269+0000","last_unstale":"2024-08-17T17:25:32.878715+0000","last_undegraded":"2024-08-17T17:24:54.477707+0000","last_fullsized":"2024-08-17T17:24:54.461071+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":4775,"log_dups_size":0,"ondisk_log_size":4775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312934400,"num_objects":4775,"num_object_clones":0,"num_object_copies":14325,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4775,"num_write_kb":305600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[0,6],"avail_no_missing":["0","2","6"],"object_location_counts":[{"shards":"0,2,6","objects":2460},{"shards":"0,6","objects":2315}],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":114,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-08-17T17:25:22.327977+0000","last_change":"2024-08-17T17:24:06.597382+0000","last_active":"2024-08-17T17:25:22.327977+0000","last_peered":"2024-08-17T17:25:22.327977+0000","last_clean":"2024-08-17T17:25:22.327977+0000","last_became_active":"2024-08-17T17:24:06.588881+0000","last_became_peered":"2024-08-17T17:24:06.588881+0000","last_unstale":"2024-08-17T17:25:22.327977+0000","last_undegraded":"2024-08-17T17:25:22.327977+0000","last_fullsized":"2024-08-17T17:25:22.327977+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:27.240751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":29,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.344847+0000","last_change":"2024-08-17T17:25:04.503780+0000","last_active":"2024-08-17T17:25:23.344847+0000","last_peered":"2024-08-17T17:25:23.344847+0000","last_clean":"2024-08-17T17:25:23.344847+0000","last_became_active":"2024-08-17T17:25:04.503503+0000","last_became_peered":"2024-08-17T17:25:04.503503+0000","last_unstale":"2024-08-17T17:25:23.344847+0000","last_undegraded":"2024-08-17T17:25:23.344847+0000","last_fullsized":"2024-08-17T17:25:23.344847+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:04:48.171216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"202'4717","reported_seq":4776,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:31.245786+0000","last_change":"2024-08-17T17:24:14.639361+0000","last_active":"2024-08-17T17:25:31.245786+0000","last_peered":"2024-08-17T17:25:31.245786+0000","last_clean":"2024-08-17T17:25:31.245786+0000","last_became_active":"2024-08-17T17:24:14.639074+0000","last_became_peered":"2024-08-17T17:24:14.639074+0000","last_unstale":"2024-08-17T17:25:31.245786+0000","last_undegraded":"2024-08-17T17:25:31.245786+0000","last_fullsized":"2024-08-17T17:25:31.245786+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":4717,"log_dups_size":0,"ondisk_log_size":4717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:15:05.377992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309133312,"num_objects":4717,"num_object_clones":0,"num_object_copies":14151,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4717,"num_write_kb":301888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":100,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-08-17T17:25:22.328361+0000","last_change":"2024-08-17T17:24:06.599023+0000","last_active":"2024-08-17T17:25:22.328361+0000","last_peered":"2024-08-17T17:25:22.328361+0000","last_clean":"2024-08-17T17:25:22.328361+0000","last_became_active":"2024-08-17T17:24:06.588954+0000","last_became_peered":"2024-08-17T17:24:06.588954+0000","last_unstale":"2024-08-17T17:25:22.328361+0000","last_undegraded":"2024-08-17T17:25:22.328361+0000","last_fullsized":"2024-08-17T17:25:22.328361+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:20:51.445576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":212,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.339683+0000","last_change":"2024-08-17T17:20:49.271540+0000","last_active":"2024-08-17T17:25:23.339683+0000","last_peered":"2024-08-17T17:25:23.339683+0000","last_clean":"2024-08-17T17:25:23.339683+0000","last_became_active":"2024-08-17T17:20:49.271130+0000","last_became_peered":"2024-08-17T17:20:49.271130+0000","last_unstale":"2024-08-17T17:25:23.339683+0000","last_undegraded":"2024-08-17T17:25:23.339683+0000","last_fullsized":"2024-08-17T17:25:23.339683+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:05.386705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"202'4544","reported_seq":4576,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:28.962455+0000","last_change":"2024-08-17T17:24:14.644070+0000","last_active":"2024-08-17T17:25:28.962455+0000","last_peered":"2024-08-17T17:25:28.962455+0000","last_clean":"2024-08-17T17:25:28.962455+0000","last_became_active":"2024-08-17T17:24:14.643799+0000","last_became_peered":"2024-08-17T17:24:14.643799+0000","last_unstale":"2024-08-17T17:25:28.962455+0000","last_undegraded":"2024-08-17T17:25:28.962455+0000","last_fullsized":"2024-08-17T17:25:28.962455+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":4544,"log_dups_size":0,"ondisk_log_size":4544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:25:38.445770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297795584,"num_objects":4544,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4544,"num_write_kb":290816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":172,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-08-17T17:25:22.328009+0000","last_change":"2024-08-17T17:24:53.467526+0000","last_active":"2024-08-17T17:25:22.328009+0000","last_peered":"2024-08-17T17:25:22.328009+0000","last_clean":"2024-08-17T17:25:22.328009+0000","last_became_active":"2024-08-17T17:24:53.460631+0000","last_became_peered":"2024-08-17T17:24:53.460631+0000","last_unstale":"2024-08-17T17:25:22.328009+0000","last_undegraded":"2024-08-17T17:25:22.328009+0000","last_fullsized":"2024-08-17T17:25:22.328009+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:05:24.938831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.339891+0000","last_change":"2024-08-17T17:25:04.507176+0000","last_active":"2024-08-17T17:25:23.339891+0000","last_peered":"2024-08-17T17:25:23.339891+0000","last_clean":"2024-08-17T17:25:23.339891+0000","last_became_active":"2024-08-17T17:25:04.506788+0000","last_became_peered":"2024-08-17T17:25:04.506788+0000","last_unstale":"2024-08-17T17:25:23.339891+0000","last_undegraded":"2024-08-17T17:25:23.339891+0000","last_fullsized":"2024-08-17T17:25:23.339891+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:22:04.658216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"203'4732","reported_seq":4793,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-08-17T17:25:32.843034+0000","last_change":"2024-08-17T17:24:14.642637+0000","last_active":"2024-08-17T17:25:32.843034+0000","last_peered":"2024-08-17T17:25:32.843034+0000","last_clean":"2024-08-17T17:25:32.843034+0000","last_became_active":"2024-08-17T17:24:14.642347+0000","last_became_peered":"2024-08-17T17:24:14.642347+0000","last_unstale":"2024-08-17T17:25:32.843034+0000","last_undegraded":"2024-08-17T17:25:32.843034+0000","last_fullsized":"2024-08-17T17:25:32.843034+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":4732,"log_dups_size":0,"ondisk_log_size":4732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:37:44.631335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310116352,"num_objects":4732,"num_object_clones":0,"num_object_copies":14196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4732,"num_write_kb":302848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":161,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.346031+0000","last_change":"2024-08-17T17:17:32.495428+0000","last_active":"2024-08-17T17:25:23.346031+0000","last_peered":"2024-08-17T17:25:23.346031+0000","last_clean":"2024-08-17T17:25:23.346031+0000","last_became_active":"2024-08-17T17:17:32.494978+0000","last_became_peered":"2024-08-17T17:17:32.494978+0000","last_unstale":"2024-08-17T17:25:23.346031+0000","last_undegraded":"2024-08-17T17:25:23.346031+0000","last_fullsized":"2024-08-17T17:25:23.346031+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:42:24.647812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":262,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-08-17T17:25:32.399947+0000","last_change":"2024-08-17T17:20:49.272126+0000","last_active":"2024-08-17T17:25:32.399947+0000","last_peered":"2024-08-17T17:25:32.399947+0000","last_clean":"2024-08-17T17:25:32.399947+0000","last_became_active":"2024-08-17T17:20:49.271185+0000","last_became_peered":"2024-08-17T17:20:49.271185+0000","last_unstale":"2024-08-17T17:25:32.399947+0000","last_undegraded":"2024-08-17T17:25:32.399947+0000","last_fullsized":"2024-08-17T17:25:32.399947+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:43:58.184939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"202'4864","reported_seq":5796,"reported_epoch":202,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:25:30.249832+0000","last_change":"2024-08-17T17:24:54.898729+0000","last_active":"2024-08-17T17:25:30.249832+0000","last_peered":"2024-08-17T17:25:30.249832+0000","last_clean":"2024-08-17T17:24:52.435411+0000","last_became_active":"2024-08-17T17:24:54.889545+0000","last_became_peered":"2024-08-17T17:24:54.889545+0000","last_unstale":"2024-08-17T17:25:30.249832+0000","last_undegraded":"2024-08-17T17:24:54.482592+0000","last_fullsized":"2024-08-17T17:24:54.459529+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":4864,"log_dups_size":0,"ondisk_log_size":4864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318767104,"num_objects":4864,"num_object_clones":0,"num_object_copies":14592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4864,"num_write_kb":311296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":430,"num_bytes_recovered":28180480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2],"avail_no_missing":["1","2","3","7"],"object_location_counts":[{"shards":"1,2","objects":2277},{"shards":"1,2,7","objects":2587}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":288,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.344530+0000","last_change":"2024-08-17T17:24:06.595380+0000","last_active":"2024-08-17T17:25:23.344530+0000","last_peered":"2024-08-17T17:25:23.344530+0000","last_clean":"2024-08-17T17:25:23.344530+0000","last_became_active":"2024-08-17T17:24:06.584747+0000","last_became_peered":"2024-08-17T17:24:06.584747+0000","last_unstale":"2024-08-17T17:25:23.344530+0000","last_undegraded":"2024-08-17T17:25:23.344530+0000","last_fullsized":"2024-08-17T17:25:23.344530+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:02:18.473980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":234,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.344251+0000","last_change":"2024-08-17T17:20:49.280596+0000","last_active":"2024-08-17T17:25:23.344251+0000","last_peered":"2024-08-17T17:25:23.344251+0000","last_clean":"2024-08-17T17:25:23.344251+0000","last_became_active":"2024-08-17T17:20:49.279553+0000","last_became_peered":"2024-08-17T17:20:49.279553+0000","last_unstale":"2024-08-17T17:25:23.344251+0000","last_undegraded":"2024-08-17T17:25:23.344251+0000","last_fullsized":"2024-08-17T17:25:23.344251+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:13:24.124392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"202'4607","reported_seq":4666,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:31.267263+0000","last_change":"2024-08-17T17:24:14.638890+0000","last_active":"2024-08-17T17:25:31.267263+0000","last_peered":"2024-08-17T17:25:31.267263+0000","last_clean":"2024-08-17T17:25:31.267263+0000","last_became_active":"2024-08-17T17:24:14.638594+0000","last_became_peered":"2024-08-17T17:24:14.638594+0000","last_unstale":"2024-08-17T17:25:31.267263+0000","last_undegraded":"2024-08-17T17:25:31.267263+0000","last_fullsized":"2024-08-17T17:25:31.267263+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":4607,"log_dups_size":0,"ondisk_log_size":4607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:53:34.277077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301924352,"num_objects":4607,"num_object_clones":0,"num_object_copies":13821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4607,"num_write_kb":294848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":186,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.338683+0000","last_change":"2024-08-17T17:20:49.276769+0000","last_active":"2024-08-17T17:25:23.338683+0000","last_peered":"2024-08-17T17:25:23.338683+0000","last_clean":"2024-08-17T17:25:23.338683+0000","last_became_active":"2024-08-17T17:20:49.276498+0000","last_became_peered":"2024-08-17T17:20:49.276498+0000","last_unstale":"2024-08-17T17:25:23.338683+0000","last_undegraded":"2024-08-17T17:25:23.338683+0000","last_fullsized":"2024-08-17T17:25:23.338683+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:39:05.419440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":181,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-08-17T17:25:22.327839+0000","last_change":"2024-08-17T17:20:14.455738+0000","last_active":"2024-08-17T17:25:22.327839+0000","last_peered":"2024-08-17T17:25:22.327839+0000","last_clean":"2024-08-17T17:25:22.327839+0000","last_became_active":"2024-08-17T17:20:14.455469+0000","last_became_peered":"2024-08-17T17:20:14.455469+0000","last_unstale":"2024-08-17T17:25:22.327839+0000","last_undegraded":"2024-08-17T17:25:22.327839+0000","last_fullsized":"2024-08-17T17:25:22.327839+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:27:28.424336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"202'4761","reported_seq":5434,"reported_epoch":202,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:25:31.267652+0000","last_change":"2024-08-17T17:25:05.678234+0000","last_active":"2024-08-17T17:25:31.267652+0000","last_peered":"2024-08-17T17:25:31.267652+0000","last_clean":"2024-08-17T17:24:52.436463+0000","last_became_active":"2024-08-17T17:25:05.673238+0000","last_became_peered":"2024-08-17T17:25:05.673238+0000","last_unstale":"2024-08-17T17:25:31.267652+0000","last_undegraded":"2024-08-17T17:25:05.585769+0000","last_fullsized":"2024-08-17T17:25:05.499112+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":4761,"log_dups_size":0,"ondisk_log_size":4761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312016896,"num_objects":4761,"num_object_clones":0,"num_object_copies":14283,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5456,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4761,"num_write_kb":304704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":304,"num_bytes_recovered":19922944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,5],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,5,7","objects":305},{"shards":"1,3,7","objects":2215},{"shards":"3,5","objects":1520},{"shards":"3,5,7","objects":721}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":135,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-08-17T17:25:22.327208+0000","last_change":"2024-08-17T17:20:14.431140+0000","last_active":"2024-08-17T17:25:22.327208+0000","last_peered":"2024-08-17T17:25:22.327208+0000","last_clean":"2024-08-17T17:25:22.327208+0000","last_became_active":"2024-08-17T17:20:14.430839+0000","last_became_peered":"2024-08-17T17:20:14.430839+0000","last_unstale":"2024-08-17T17:25:22.327208+0000","last_undegraded":"2024-08-17T17:25:22.327208+0000","last_fullsized":"2024-08-17T17:25:22.327208+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:08:16.303616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":15,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.336301+0000","last_change":"2024-08-17T17:25:04.509325+0000","last_active":"2024-08-17T17:25:23.336301+0000","last_peered":"2024-08-17T17:25:23.336301+0000","last_clean":"2024-08-17T17:25:23.336301+0000","last_became_active":"2024-08-17T17:25:04.508364+0000","last_became_peered":"2024-08-17T17:25:04.508364+0000","last_unstale":"2024-08-17T17:25:23.336301+0000","last_undegraded":"2024-08-17T17:25:23.336301+0000","last_fullsized":"2024-08-17T17:25:23.336301+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:48:56.415763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"202'4574","reported_seq":7980,"reported_epoch":202,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-17T17:25:30.279271+0000","last_change":"2024-08-17T17:25:04.021147+0000","last_active":"2024-08-17T17:25:30.279271+0000","last_peered":"2024-08-17T17:25:30.279271+0000","last_clean":"2024-08-17T17:24:52.450213+0000","last_became_active":"2024-08-17T17:24:54.880124+0000","last_became_peered":"2024-08-17T17:24:54.880124+0000","last_unstale":"2024-08-17T17:25:30.279271+0000","last_undegraded":"2024-08-17T17:25:30.279271+0000","last_fullsized":"2024-08-17T17:24:54.459660+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":4574,"log_dups_size":0,"ondisk_log_size":4574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299761664,"num_objects":4574,"num_object_clones":0,"num_object_copies":13722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":698,"num_objects_unfound":0,"num_objects_dirty":4574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4574,"num_write_kb":292736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1672,"num_bytes_recovered":109576192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[1,0],"avail_no_missing":["1","0","2"],"object_location_counts":[{"shards":"0,1","objects":3876},{"shards":"0,1,2","objects":698}],"blocked_by":[],"up_primary":5,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":272,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.339806+0000","last_change":"2024-08-17T17:19:57.520745+0000","last_active":"2024-08-17T17:25:23.339806+0000","last_peered":"2024-08-17T17:25:23.339806+0000","last_clean":"2024-08-17T17:25:23.339806+0000","last_became_active":"2024-08-17T17:19:57.520152+0000","last_became_peered":"2024-08-17T17:19:57.520152+0000","last_unstale":"2024-08-17T17:25:23.339806+0000","last_undegraded":"2024-08-17T17:25:23.339806+0000","last_fullsized":"2024-08-17T17:25:23.339806+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:19:39.495644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":248,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.338984+0000","last_change":"2024-08-17T17:20:49.274372+0000","last_active":"2024-08-17T17:25:23.338984+0000","last_peered":"2024-08-17T17:25:23.338984+0000","last_clean":"2024-08-17T17:25:23.338984+0000","last_became_active":"2024-08-17T17:20:49.274023+0000","last_became_peered":"2024-08-17T17:20:49.274023+0000","last_unstale":"2024-08-17T17:25:23.338984+0000","last_undegraded":"2024-08-17T17:25:23.338984+0000","last_fullsized":"2024-08-17T17:25:23.338984+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:31:02.529766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"203'5101","reported_seq":5134,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-08-17T17:25:33.793678+0000","last_change":"2024-08-17T17:24:14.643398+0000","last_active":"2024-08-17T17:25:33.793678+0000","last_peered":"2024-08-17T17:25:33.793678+0000","last_clean":"2024-08-17T17:25:33.793678+0000","last_became_active":"2024-08-17T17:24:14.643102+0000","last_became_peered":"2024-08-17T17:24:14.643102+0000","last_unstale":"2024-08-17T17:25:33.793678+0000","last_undegraded":"2024-08-17T17:25:33.793678+0000","last_fullsized":"2024-08-17T17:25:33.793678+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":5101,"log_dups_size":0,"ondisk_log_size":5101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:09:30.323709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334299136,"num_objects":5101,"num_object_clones":0,"num_object_copies":15303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5101,"num_write_kb":326464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":529,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-08-17T17:25:22.327041+0000","last_change":"2024-08-17T17:24:09.599615+0000","last_active":"2024-08-17T17:25:22.327041+0000","last_peered":"2024-08-17T17:25:22.327041+0000","last_clean":"2024-08-17T17:25:22.327041+0000","last_became_active":"2024-08-17T17:24:09.599085+0000","last_became_peered":"2024-08-17T17:24:09.599085+0000","last_unstale":"2024-08-17T17:25:22.327041+0000","last_undegraded":"2024-08-17T17:25:22.327041+0000","last_fullsized":"2024-08-17T17:25:22.327041+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:04:01.141401+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:30:09.569579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":775,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.344966+0000","last_change":"2024-08-17T17:20:00.434774+0000","last_active":"2024-08-17T17:25:23.344966+0000","last_peered":"2024-08-17T17:25:23.344966+0000","last_clean":"2024-08-17T17:25:23.344966+0000","last_became_active":"2024-08-17T17:20:00.434343+0000","last_became_peered":"2024-08-17T17:20:00.434343+0000","last_unstale":"2024-08-17T17:25:23.344966+0000","last_undegraded":"2024-08-17T17:25:23.344966+0000","last_fullsized":"2024-08-17T17:25:23.344966+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:04.995152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":230,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.341258+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:25:23.341258+0000","last_peered":"2024-08-17T17:25:23.341258+0000","last_clean":"2024-08-17T17:25:23.341258+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:25:23.341258+0000","last_undegraded":"2024-08-17T17:25:23.341258+0000","last_fullsized":"2024-08-17T17:25:23.341258+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":15,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.338793+0000","last_change":"2024-08-17T17:25:04.579426+0000","last_active":"2024-08-17T17:25:23.338793+0000","last_peered":"2024-08-17T17:25:23.338793+0000","last_clean":"2024-08-17T17:25:23.338793+0000","last_became_active":"2024-08-17T17:25:04.579012+0000","last_became_peered":"2024-08-17T17:25:04.579012+0000","last_unstale":"2024-08-17T17:25:23.338793+0000","last_undegraded":"2024-08-17T17:25:23.338793+0000","last_fullsized":"2024-08-17T17:25:23.338793+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:56:32.241321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"202'4620","reported_seq":4679,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:30.242671+0000","last_change":"2024-08-17T17:24:14.641829+0000","last_active":"2024-08-17T17:25:30.242671+0000","last_peered":"2024-08-17T17:25:30.242671+0000","last_clean":"2024-08-17T17:25:30.242671+0000","last_became_active":"2024-08-17T17:24:14.641532+0000","last_became_peered":"2024-08-17T17:24:14.641532+0000","last_unstale":"2024-08-17T17:25:30.242671+0000","last_undegraded":"2024-08-17T17:25:30.242671+0000","last_fullsized":"2024-08-17T17:25:30.242671+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":4620,"log_dups_size":0,"ondisk_log_size":4620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:43:05.232036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302776320,"num_objects":4620,"num_object_clones":0,"num_object_copies":13860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4620,"num_write_kb":295680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":792,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-08-17T17:25:22.327164+0000","last_change":"2024-08-17T17:16:26.958415+0000","last_active":"2024-08-17T17:25:22.327164+0000","last_peered":"2024-08-17T17:25:22.327164+0000","last_clean":"2024-08-17T17:25:22.327164+0000","last_became_active":"2024-08-17T17:16:26.957956+0000","last_became_peered":"2024-08-17T17:16:26.957956+0000","last_unstale":"2024-08-17T17:25:22.327164+0000","last_undegraded":"2024-08-17T17:25:22.327164+0000","last_fullsized":"2024-08-17T17:25:22.327164+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:22.478077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":261,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-08-17T17:25:32.411450+0000","last_change":"2024-08-17T17:18:59.415331+0000","last_active":"2024-08-17T17:25:32.411450+0000","last_peered":"2024-08-17T17:25:32.411450+0000","last_clean":"2024-08-17T17:25:32.411450+0000","last_became_active":"2024-08-17T17:18:59.415008+0000","last_became_peered":"2024-08-17T17:18:59.415008+0000","last_unstale":"2024-08-17T17:25:32.411450+0000","last_undegraded":"2024-08-17T17:25:32.411450+0000","last_fullsized":"2024-08-17T17:25:32.411450+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:22:38.076866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":70,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-08-17T17:25:32.400617+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:25:32.400617+0000","last_peered":"2024-08-17T17:25:32.400617+0000","last_clean":"2024-08-17T17:25:32.400617+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:25:32.400617+0000","last_undegraded":"2024-08-17T17:25:32.400617+0000","last_fullsized":"2024-08-17T17:25:32.400617+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"202'4522","reported_seq":4563,"reported_epoch":202,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:25:28.925873+0000","last_change":"2024-08-17T17:25:05.609285+0000","last_active":"2024-08-17T17:25:28.925873+0000","last_peered":"2024-08-17T17:25:28.925873+0000","last_clean":"2024-08-17T17:25:03.970571+0000","last_became_active":"2024-08-17T17:25:05.604478+0000","last_became_peered":"2024-08-17T17:25:05.604478+0000","last_unstale":"2024-08-17T17:25:28.925873+0000","last_undegraded":"2024-08-17T17:25:05.510212+0000","last_fullsized":"2024-08-17T17:25:05.498949+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":4522,"log_dups_size":0,"ondisk_log_size":4522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:29:09.928695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296353792,"num_objects":4522,"num_object_clones":0,"num_object_copies":13566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4522,"num_write_kb":289408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,3],"avail_no_missing":["7","2","3"],"object_location_counts":[{"shards":"2,3,7","objects":3108},{"shards":"3,7","objects":1414}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":647,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.339761+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:25:23.339761+0000","last_peered":"2024-08-17T17:25:23.339761+0000","last_clean":"2024-08-17T17:25:23.339761+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:25:23.339761+0000","last_undegraded":"2024-08-17T17:25:23.339761+0000","last_fullsized":"2024-08-17T17:25:23.339761+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":255,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.338527+0000","last_change":"2024-08-17T17:19:57.515564+0000","last_active":"2024-08-17T17:25:23.338527+0000","last_peered":"2024-08-17T17:25:23.338527+0000","last_clean":"2024-08-17T17:25:23.338527+0000","last_became_active":"2024-08-17T17:19:57.515142+0000","last_became_peered":"2024-08-17T17:19:57.515142+0000","last_unstale":"2024-08-17T17:25:23.338527+0000","last_undegraded":"2024-08-17T17:25:23.338527+0000","last_fullsized":"2024-08-17T17:25:23.338527+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:59:33.506176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":286,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.338740+0000","last_change":"2024-08-17T17:24:06.587903+0000","last_active":"2024-08-17T17:25:23.338740+0000","last_peered":"2024-08-17T17:25:23.338740+0000","last_clean":"2024-08-17T17:25:23.338740+0000","last_became_active":"2024-08-17T17:24:06.587503+0000","last_became_peered":"2024-08-17T17:24:06.587503+0000","last_unstale":"2024-08-17T17:25:23.338740+0000","last_undegraded":"2024-08-17T17:25:23.338740+0000","last_fullsized":"2024-08-17T17:25:23.338740+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:05:58.591992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"202'4637","reported_seq":6599,"reported_epoch":202,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-17T17:25:31.248748+0000","last_change":"2024-08-17T17:25:04.657561+0000","last_active":"2024-08-17T17:25:31.248748+0000","last_peered":"2024-08-17T17:25:31.248748+0000","last_clean":"2024-08-17T17:24:52.452117+0000","last_became_active":"2024-08-17T17:25:04.596208+0000","last_became_peered":"2024-08-17T17:25:04.596208+0000","last_unstale":"2024-08-17T17:25:31.248748+0000","last_undegraded":"2024-08-17T17:25:31.248748+0000","last_fullsized":"2024-08-17T17:25:04.500643+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303890432,"num_objects":4637,"num_object_clones":0,"num_object_copies":13911,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2152,"num_objects_unfound":0,"num_objects_dirty":4637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":296768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":949,"num_bytes_recovered":62193664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":1435},{"shards":"3,5,7","objects":717},{"shards":"3,7","objects":2485}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":1113,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.345040+0000","last_change":"2024-08-17T17:20:00.433870+0000","last_active":"2024-08-17T17:25:23.345040+0000","last_peered":"2024-08-17T17:25:23.345040+0000","last_clean":"2024-08-17T17:25:23.345040+0000","last_became_active":"2024-08-17T17:20:00.433386+0000","last_became_peered":"2024-08-17T17:20:00.433386+0000","last_unstale":"2024-08-17T17:25:23.345040+0000","last_undegraded":"2024-08-17T17:25:23.345040+0000","last_fullsized":"2024-08-17T17:25:23.345040+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:01:11.848862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1392,"num_read_kb":2863,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":183,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-08-17T17:25:32.399750+0000","last_change":"2024-08-17T17:20:49.272407+0000","last_active":"2024-08-17T17:25:32.399750+0000","last_peered":"2024-08-17T17:25:32.399750+0000","last_clean":"2024-08-17T17:25:32.399750+0000","last_became_active":"2024-08-17T17:20:49.271749+0000","last_became_peered":"2024-08-17T17:20:49.271749+0000","last_unstale":"2024-08-17T17:25:32.399750+0000","last_undegraded":"2024-08-17T17:25:32.399750+0000","last_fullsized":"2024-08-17T17:25:32.399750+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:36:01.443457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":252,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.343146+0000","last_change":"2024-08-17T17:17:32.499715+0000","last_active":"2024-08-17T17:25:23.343146+0000","last_peered":"2024-08-17T17:25:23.343146+0000","last_clean":"2024-08-17T17:25:23.343146+0000","last_became_active":"2024-08-17T17:17:32.499412+0000","last_became_peered":"2024-08-17T17:17:32.499412+0000","last_unstale":"2024-08-17T17:25:23.343146+0000","last_undegraded":"2024-08-17T17:25:23.343146+0000","last_fullsized":"2024-08-17T17:25:23.343146+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:39:38.456970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"203'4843","reported_seq":4904,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-08-17T17:25:32.908948+0000","last_change":"2024-08-17T17:24:14.642608+0000","last_active":"2024-08-17T17:25:32.908948+0000","last_peered":"2024-08-17T17:25:32.908948+0000","last_clean":"2024-08-17T17:25:32.908948+0000","last_became_active":"2024-08-17T17:24:14.642292+0000","last_became_peered":"2024-08-17T17:24:14.642292+0000","last_unstale":"2024-08-17T17:25:32.908948+0000","last_undegraded":"2024-08-17T17:25:32.908948+0000","last_fullsized":"2024-08-17T17:25:32.908948+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":4843,"log_dups_size":0,"ondisk_log_size":4843,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:53:36.848614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317390848,"num_objects":4843,"num_object_clones":0,"num_object_copies":14529,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4843,"num_write_kb":309952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":178,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.345175+0000","last_change":"2024-08-17T17:17:32.495728+0000","last_active":"2024-08-17T17:25:23.345175+0000","last_peered":"2024-08-17T17:25:23.345175+0000","last_clean":"2024-08-17T17:25:23.345175+0000","last_became_active":"2024-08-17T17:17:32.495357+0000","last_became_peered":"2024-08-17T17:17:32.495357+0000","last_unstale":"2024-08-17T17:25:23.345175+0000","last_undegraded":"2024-08-17T17:25:23.345175+0000","last_fullsized":"2024-08-17T17:25:23.345175+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:27:45.561567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":488,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.343314+0000","last_change":"2024-08-17T17:16:26.964621+0000","last_active":"2024-08-17T17:25:23.343314+0000","last_peered":"2024-08-17T17:25:23.343314+0000","last_clean":"2024-08-17T17:25:23.343314+0000","last_became_active":"2024-08-17T17:16:26.964058+0000","last_became_peered":"2024-08-17T17:16:26.964058+0000","last_unstale":"2024-08-17T17:25:23.343314+0000","last_undegraded":"2024-08-17T17:25:23.343314+0000","last_fullsized":"2024-08-17T17:25:23.343314+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:00:04.465644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":258,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.339344+0000","last_change":"2024-08-17T17:24:53.465550+0000","last_active":"2024-08-17T17:25:23.339344+0000","last_peered":"2024-08-17T17:25:23.339344+0000","last_clean":"2024-08-17T17:25:23.339344+0000","last_became_active":"2024-08-17T17:24:53.465228+0000","last_became_peered":"2024-08-17T17:24:53.465228+0000","last_unstale":"2024-08-17T17:25:23.339344+0000","last_undegraded":"2024-08-17T17:25:23.339344+0000","last_fullsized":"2024-08-17T17:25:23.339344+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:21:18.378359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"202'4678","reported_seq":4737,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:30.251436+0000","last_change":"2024-08-17T17:24:14.640584+0000","last_active":"2024-08-17T17:25:30.251436+0000","last_peered":"2024-08-17T17:25:30.251436+0000","last_clean":"2024-08-17T17:25:30.251436+0000","last_became_active":"2024-08-17T17:24:14.640168+0000","last_became_peered":"2024-08-17T17:24:14.640168+0000","last_unstale":"2024-08-17T17:25:30.251436+0000","last_undegraded":"2024-08-17T17:25:30.251436+0000","last_fullsized":"2024-08-17T17:25:30.251436+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":4678,"log_dups_size":0,"ondisk_log_size":4678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:38:36.176813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306577408,"num_objects":4678,"num_object_clones":0,"num_object_copies":14034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4678,"num_write_kb":299392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":283,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.343231+0000","last_change":"2024-08-17T17:24:53.464307+0000","last_active":"2024-08-17T17:25:23.343231+0000","last_peered":"2024-08-17T17:25:23.343231+0000","last_clean":"2024-08-17T17:25:23.343231+0000","last_became_active":"2024-08-17T17:24:53.463996+0000","last_became_peered":"2024-08-17T17:24:53.463996+0000","last_unstale":"2024-08-17T17:25:23.343231+0000","last_undegraded":"2024-08-17T17:25:23.343231+0000","last_fullsized":"2024-08-17T17:25:23.343231+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:04:18.993725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":629,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-08-17T17:25:22.328304+0000","last_change":"2024-08-17T17:24:09.599822+0000","last_active":"2024-08-17T17:25:22.328304+0000","last_peered":"2024-08-17T17:25:22.328304+0000","last_clean":"2024-08-17T17:25:22.328304+0000","last_became_active":"2024-08-17T17:24:09.599189+0000","last_became_peered":"2024-08-17T17:24:09.599189+0000","last_unstale":"2024-08-17T17:25:22.328304+0000","last_undegraded":"2024-08-17T17:25:22.328304+0000","last_fullsized":"2024-08-17T17:25:22.328304+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:04.465604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":244,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-08-17T17:25:32.399852+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:25:32.399852+0000","last_peered":"2024-08-17T17:25:32.399852+0000","last_clean":"2024-08-17T17:25:32.399852+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:25:32.399852+0000","last_undegraded":"2024-08-17T17:25:32.399852+0000","last_fullsized":"2024-08-17T17:25:32.399852+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"202'4882","reported_seq":4941,"reported_epoch":202,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:25:31.162891+0000","last_change":"2024-08-17T17:25:05.605653+0000","last_active":"2024-08-17T17:25:31.162891+0000","last_peered":"2024-08-17T17:25:31.162891+0000","last_clean":"2024-08-17T17:25:03.971607+0000","last_became_active":"2024-08-17T17:25:05.601333+0000","last_became_peered":"2024-08-17T17:25:05.601333+0000","last_unstale":"2024-08-17T17:25:31.162891+0000","last_undegraded":"2024-08-17T17:25:05.522007+0000","last_fullsized":"2024-08-17T17:25:05.499355+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":4882,"log_dups_size":0,"ondisk_log_size":4882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319946752,"num_objects":4882,"num_object_clones":0,"num_object_copies":14646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4882,"num_write_kb":312448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2],"avail_no_missing":["3","2","7"],"object_location_counts":[{"shards":"2,3","objects":1566},{"shards":"2,3,7","objects":3316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":277,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.341565+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:25:23.341565+0000","last_peered":"2024-08-17T17:25:23.341565+0000","last_clean":"2024-08-17T17:25:23.341565+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:25:23.341565+0000","last_undegraded":"2024-08-17T17:25:23.341565+0000","last_fullsized":"2024-08-17T17:25:23.341565+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":797,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.343841+0000","last_change":"2024-08-17T17:23:19.633786+0000","last_active":"2024-08-17T17:25:23.343841+0000","last_peered":"2024-08-17T17:25:23.343841+0000","last_clean":"2024-08-17T17:25:23.343841+0000","last_became_active":"2024-08-17T17:23:19.633230+0000","last_became_peered":"2024-08-17T17:23:19.633230+0000","last_unstale":"2024-08-17T17:25:23.343841+0000","last_undegraded":"2024-08-17T17:25:23.343841+0000","last_fullsized":"2024-08-17T17:25:23.343841+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:30:50.377391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":274,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.339594+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:25:23.339594+0000","last_peered":"2024-08-17T17:25:23.339594+0000","last_clean":"2024-08-17T17:25:23.339594+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:25:23.339594+0000","last_undegraded":"2024-08-17T17:25:23.339594+0000","last_fullsized":"2024-08-17T17:25:23.339594+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"202'4608","reported_seq":4640,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:28.941392+0000","last_change":"2024-08-17T17:24:14.639560+0000","last_active":"2024-08-17T17:25:28.941392+0000","last_peered":"2024-08-17T17:25:28.941392+0000","last_clean":"2024-08-17T17:25:28.941392+0000","last_became_active":"2024-08-17T17:24:14.639093+0000","last_became_peered":"2024-08-17T17:24:14.639093+0000","last_unstale":"2024-08-17T17:25:28.941392+0000","last_undegraded":"2024-08-17T17:25:28.941392+0000","last_fullsized":"2024-08-17T17:25:28.941392+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":4608,"log_dups_size":0,"ondisk_log_size":4608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:45:57.940550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301989888,"num_objects":4608,"num_object_clones":0,"num_object_copies":13824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4608,"num_write_kb":294912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":74,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-08-17T17:25:32.400517+0000","last_change":"2024-08-17T17:24:06.593660+0000","last_active":"2024-08-17T17:25:32.400517+0000","last_peered":"2024-08-17T17:25:32.400517+0000","last_clean":"2024-08-17T17:25:32.400517+0000","last_became_active":"2024-08-17T17:24:06.586215+0000","last_became_peered":"2024-08-17T17:24:06.586215+0000","last_unstale":"2024-08-17T17:25:32.400517+0000","last_undegraded":"2024-08-17T17:25:32.400517+0000","last_fullsized":"2024-08-17T17:25:32.400517+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:15:49.475395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":420,"reported_epoch":202,"state":"active+recovery_wait+degraded","last_fresh":"2024-08-17T17:25:23.359036+0000","last_change":"2024-08-17T17:25:23.359036+0000","last_active":"2024-08-17T17:25:23.359036+0000","last_peered":"2024-08-17T17:25:23.359036+0000","last_clean":"2024-08-17T17:25:17.220335+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:25:23.359036+0000","last_undegraded":"2024-08-17T17:25:23.336653+0000","last_fullsized":"2024-08-17T17:25:23.359036+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":["6","2","3"],"object_location_counts":[{"shards":"2,3,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":102,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.344858+0000","last_change":"2024-08-17T17:18:59.417183+0000","last_active":"2024-08-17T17:25:23.344858+0000","last_peered":"2024-08-17T17:25:23.344858+0000","last_clean":"2024-08-17T17:25:23.344858+0000","last_became_active":"2024-08-17T17:18:59.416843+0000","last_became_peered":"2024-08-17T17:18:59.416843+0000","last_unstale":"2024-08-17T17:25:23.344858+0000","last_undegraded":"2024-08-17T17:25:23.344858+0000","last_fullsized":"2024-08-17T17:25:23.344858+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:29:09.336335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":123,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.336099+0000","last_change":"2024-08-17T17:20:49.258048+0000","last_active":"2024-08-17T17:25:23.336099+0000","last_peered":"2024-08-17T17:25:23.336099+0000","last_clean":"2024-08-17T17:25:23.336099+0000","last_became_active":"2024-08-17T17:20:49.257672+0000","last_became_peered":"2024-08-17T17:20:49.257672+0000","last_unstale":"2024-08-17T17:25:23.336099+0000","last_undegraded":"2024-08-17T17:25:23.336099+0000","last_fullsized":"2024-08-17T17:25:23.336099+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:08:11.734316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":161,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-08-17T17:25:22.327548+0000","last_change":"2024-08-17T17:18:59.430969+0000","last_active":"2024-08-17T17:25:22.327548+0000","last_peered":"2024-08-17T17:25:22.327548+0000","last_clean":"2024-08-17T17:25:22.327548+0000","last_became_active":"2024-08-17T17:18:59.413475+0000","last_became_peered":"2024-08-17T17:18:59.413475+0000","last_unstale":"2024-08-17T17:25:22.327548+0000","last_undegraded":"2024-08-17T17:25:22.327548+0000","last_fullsized":"2024-08-17T17:25:22.327548+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:36:33.732645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":268,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.338601+0000","last_change":"2024-08-17T17:25:04.506413+0000","last_active":"2024-08-17T17:25:23.338601+0000","last_peered":"2024-08-17T17:25:23.338601+0000","last_clean":"2024-08-17T17:25:23.338601+0000","last_became_active":"2024-08-17T17:25:04.506104+0000","last_became_peered":"2024-08-17T17:25:04.506104+0000","last_unstale":"2024-08-17T17:25:23.338601+0000","last_undegraded":"2024-08-17T17:25:23.338601+0000","last_fullsized":"2024-08-17T17:25:23.338601+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:41:53.556223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":104,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-08-17T17:25:22.327770+0000","last_change":"2024-08-17T17:20:49.279458+0000","last_active":"2024-08-17T17:25:22.327770+0000","last_peered":"2024-08-17T17:25:22.327770+0000","last_clean":"2024-08-17T17:25:22.327770+0000","last_became_active":"2024-08-17T17:20:49.260790+0000","last_became_peered":"2024-08-17T17:20:49.260790+0000","last_unstale":"2024-08-17T17:25:22.327770+0000","last_undegraded":"2024-08-17T17:25:22.327770+0000","last_fullsized":"2024-08-17T17:25:22.327770+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:58.092945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":19,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.345263+0000","last_change":"2024-08-17T17:24:53.463713+0000","last_active":"2024-08-17T17:25:23.345263+0000","last_peered":"2024-08-17T17:25:23.345263+0000","last_clean":"2024-08-17T17:25:23.345263+0000","last_became_active":"2024-08-17T17:24:53.463419+0000","last_became_peered":"2024-08-17T17:24:53.463419+0000","last_unstale":"2024-08-17T17:25:23.345263+0000","last_undegraded":"2024-08-17T17:25:23.345263+0000","last_fullsized":"2024-08-17T17:25:23.345263+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:34:44.545175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":224,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.339522+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:25:23.339522+0000","last_peered":"2024-08-17T17:25:23.339522+0000","last_clean":"2024-08-17T17:25:23.339522+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:25:23.339522+0000","last_undegraded":"2024-08-17T17:25:23.339522+0000","last_fullsized":"2024-08-17T17:25:23.339522+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":254,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-08-17T17:25:32.400967+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:25:32.400967+0000","last_peered":"2024-08-17T17:25:32.400967+0000","last_clean":"2024-08-17T17:25:32.400967+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:25:32.400967+0000","last_undegraded":"2024-08-17T17:25:32.400967+0000","last_fullsized":"2024-08-17T17:25:32.400967+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":271,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.339481+0000","last_change":"2024-08-17T17:25:04.509422+0000","last_active":"2024-08-17T17:25:23.339481+0000","last_peered":"2024-08-17T17:25:23.339481+0000","last_clean":"2024-08-17T17:25:23.339481+0000","last_became_active":"2024-08-17T17:25:04.508742+0000","last_became_peered":"2024-08-17T17:25:04.508742+0000","last_unstale":"2024-08-17T17:25:23.339481+0000","last_undegraded":"2024-08-17T17:25:23.339481+0000","last_fullsized":"2024-08-17T17:25:23.339481+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:18:31.976907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":160,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.336269+0000","last_change":"2024-08-17T17:16:26.963696+0000","last_active":"2024-08-17T17:25:23.336269+0000","last_peered":"2024-08-17T17:25:23.336269+0000","last_clean":"2024-08-17T17:25:23.336269+0000","last_became_active":"2024-08-17T17:16:26.963280+0000","last_became_peered":"2024-08-17T17:16:26.963280+0000","last_unstale":"2024-08-17T17:25:23.336269+0000","last_undegraded":"2024-08-17T17:25:23.336269+0000","last_fullsized":"2024-08-17T17:25:23.336269+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:22:01.474840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":289,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.339706+0000","last_change":"2024-08-17T17:24:06.586886+0000","last_active":"2024-08-17T17:25:23.339706+0000","last_peered":"2024-08-17T17:25:23.339706+0000","last_clean":"2024-08-17T17:25:23.339706+0000","last_became_active":"2024-08-17T17:24:06.586516+0000","last_became_peered":"2024-08-17T17:24:06.586516+0000","last_unstale":"2024-08-17T17:25:23.339706+0000","last_undegraded":"2024-08-17T17:25:23.339706+0000","last_fullsized":"2024-08-17T17:25:23.339706+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:03:30.002726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":70,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-08-17T17:25:32.400274+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:25:32.400274+0000","last_peered":"2024-08-17T17:25:32.400274+0000","last_clean":"2024-08-17T17:25:32.400274+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:25:32.400274+0000","last_undegraded":"2024-08-17T17:25:32.400274+0000","last_fullsized":"2024-08-17T17:25:32.400274+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":248,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.339750+0000","last_change":"2024-08-17T17:18:59.412463+0000","last_active":"2024-08-17T17:25:23.339750+0000","last_peered":"2024-08-17T17:25:23.339750+0000","last_clean":"2024-08-17T17:25:23.339750+0000","last_became_active":"2024-08-17T17:18:59.412134+0000","last_became_peered":"2024-08-17T17:18:59.412134+0000","last_unstale":"2024-08-17T17:25:23.339750+0000","last_undegraded":"2024-08-17T17:25:23.339750+0000","last_fullsized":"2024-08-17T17:25:23.339750+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:44.463153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":292,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.347470+0000","last_change":"2024-08-17T17:25:04.510854+0000","last_active":"2024-08-17T17:25:23.347470+0000","last_peered":"2024-08-17T17:25:23.347470+0000","last_clean":"2024-08-17T17:25:23.347470+0000","last_became_active":"2024-08-17T17:25:04.510462+0000","last_became_peered":"2024-08-17T17:25:04.510462+0000","last_unstale":"2024-08-17T17:25:23.347470+0000","last_undegraded":"2024-08-17T17:25:23.347470+0000","last_fullsized":"2024-08-17T17:25:23.347470+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:55:54.762086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":189,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.339422+0000","last_change":"2024-08-17T17:24:06.591165+0000","last_active":"2024-08-17T17:25:23.339422+0000","last_peered":"2024-08-17T17:25:23.339422+0000","last_clean":"2024-08-17T17:25:23.339422+0000","last_became_active":"2024-08-17T17:24:06.590785+0000","last_became_peered":"2024-08-17T17:24:06.590785+0000","last_unstale":"2024-08-17T17:25:23.339422+0000","last_undegraded":"2024-08-17T17:25:23.339422+0000","last_fullsized":"2024-08-17T17:25:23.339422+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:54:05.775563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":261,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.343442+0000","last_change":"2024-08-17T17:24:32.300560+0000","last_active":"2024-08-17T17:25:23.343442+0000","last_peered":"2024-08-17T17:25:23.343442+0000","last_clean":"2024-08-17T17:25:23.343442+0000","last_became_active":"2024-08-17T17:24:32.300228+0000","last_became_peered":"2024-08-17T17:24:32.300228+0000","last_unstale":"2024-08-17T17:25:23.343442+0000","last_undegraded":"2024-08-17T17:25:23.343442+0000","last_fullsized":"2024-08-17T17:25:23.343442+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:18:05.475449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":70,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-08-17T17:25:32.399582+0000","last_change":"2024-08-17T17:24:06.586379+0000","last_active":"2024-08-17T17:25:32.399582+0000","last_peered":"2024-08-17T17:25:32.399582+0000","last_clean":"2024-08-17T17:25:32.399582+0000","last_became_active":"2024-08-17T17:24:06.586023+0000","last_became_peered":"2024-08-17T17:24:06.586023+0000","last_unstale":"2024-08-17T17:25:32.399582+0000","last_undegraded":"2024-08-17T17:25:32.399582+0000","last_fullsized":"2024-08-17T17:25:32.399582+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:45.701637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":278,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.344429+0000","last_change":"2024-08-17T17:21:50.442535+0000","last_active":"2024-08-17T17:25:23.344429+0000","last_peered":"2024-08-17T17:25:23.344429+0000","last_clean":"2024-08-17T17:25:23.344429+0000","last_became_active":"2024-08-17T17:21:50.441891+0000","last_became_peered":"2024-08-17T17:21:50.441891+0000","last_unstale":"2024-08-17T17:25:23.344429+0000","last_undegraded":"2024-08-17T17:25:23.344429+0000","last_fullsized":"2024-08-17T17:25:23.344429+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:46:24.025407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":215,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.345082+0000","last_change":"2024-08-17T17:25:04.508045+0000","last_active":"2024-08-17T17:25:23.345082+0000","last_peered":"2024-08-17T17:25:23.345082+0000","last_clean":"2024-08-17T17:25:23.345082+0000","last_became_active":"2024-08-17T17:25:04.505083+0000","last_became_peered":"2024-08-17T17:25:04.505083+0000","last_unstale":"2024-08-17T17:25:23.345082+0000","last_undegraded":"2024-08-17T17:25:23.345082+0000","last_fullsized":"2024-08-17T17:25:23.345082+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:23:42.993544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":264,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.351434+0000","last_change":"2024-08-17T17:18:59.418434+0000","last_active":"2024-08-17T17:25:23.351434+0000","last_peered":"2024-08-17T17:25:23.351434+0000","last_clean":"2024-08-17T17:25:23.351434+0000","last_became_active":"2024-08-17T17:18:59.418043+0000","last_became_peered":"2024-08-17T17:18:59.418043+0000","last_unstale":"2024-08-17T17:25:23.351434+0000","last_undegraded":"2024-08-17T17:25:23.351434+0000","last_fullsized":"2024-08-17T17:25:23.351434+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:25:51.371820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":188,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.345143+0000","last_change":"2024-08-17T17:20:49.291167+0000","last_active":"2024-08-17T17:25:23.345143+0000","last_peered":"2024-08-17T17:25:23.345143+0000","last_clean":"2024-08-17T17:25:23.345143+0000","last_became_active":"2024-08-17T17:20:49.263750+0000","last_became_peered":"2024-08-17T17:20:49.263750+0000","last_unstale":"2024-08-17T17:25:23.345143+0000","last_undegraded":"2024-08-17T17:25:23.345143+0000","last_fullsized":"2024-08-17T17:25:23.345143+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:02.863246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":278,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.339851+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:25:23.339851+0000","last_peered":"2024-08-17T17:25:23.339851+0000","last_clean":"2024-08-17T17:25:23.339851+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:25:23.339851+0000","last_undegraded":"2024-08-17T17:25:23.339851+0000","last_fullsized":"2024-08-17T17:25:23.339851+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":278,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.347656+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:25:23.347656+0000","last_peered":"2024-08-17T17:25:23.347656+0000","last_clean":"2024-08-17T17:25:23.347656+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:25:23.347656+0000","last_undegraded":"2024-08-17T17:25:23.347656+0000","last_fullsized":"2024-08-17T17:25:23.347656+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":15,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-08-17T17:25:23.335970+0000","last_change":"2024-08-17T17:25:04.503834+0000","last_active":"2024-08-17T17:25:23.335970+0000","last_peered":"2024-08-17T17:25:23.335970+0000","last_clean":"2024-08-17T17:25:23.335970+0000","last_became_active":"2024-08-17T17:25:04.502712+0000","last_became_peered":"2024-08-17T17:25:04.502712+0000","last_unstale":"2024-08-17T17:25:23.335970+0000","last_undegraded":"2024-08-17T17:25:23.335970+0000","last_fullsized":"2024-08-17T17:25:23.335970+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:45.389103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":89,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-08-17T17:25:32.400867+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:25:32.400867+0000","last_peered":"2024-08-17T17:25:32.400867+0000","last_clean":"2024-08-17T17:25:32.400867+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:25:32.400867+0000","last_undegraded":"2024-08-17T17:25:32.400867+0000","last_fullsized":"2024-08-17T17:25:32.400867+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":4945674240,"num_objects":75465,"num_object_clones":0,"num_object_copies":226395,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16927,"num_objects_misplaced":2850,"num_objects_unfound":0,"num_objects_dirty":75465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75465,"num_write_kb":4829760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3355,"num_bytes_recovered":219873280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15121907712,"data_stored":15121907712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75465,"ondisk_log_size":75465,"up":48,"acting":41,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":6304,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":16702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17971,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":4194342,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234224,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2619600,"kb_used_data":2322248,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":91125552,"statfs":{"total":95995035648,"available":93312565248,"internally_reserved":0,"allocated":2377981952,"data_stored":2376836054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495860,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2040408,"kb_used_data":1738896,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91704744,"statfs":{"total":95995035648,"available":93905657856,"internally_reserved":0,"allocated":1780629504,"data_stored":1779483660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757497,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1741776,"kb_used_data":1431940,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":92003376,"statfs":{"total":95995035648,"available":94211457024,"internally_reserved":0,"allocated":1466306560,"data_stored":1465168459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019135,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435362,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3222776,"kb_used_data":3028720,"kb_used_omap":4,"kb_used_meta":194043,"kb_avail":90522376,"statfs":{"total":95995035648,"available":92694913024,"internally_reserved":0,"allocated":3101409280,"data_stored":3100282887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":198700042},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,2,4,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":104,"seq":446676598888,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3153352,"kb_used_data":2862464,"kb_used_omap":7,"kb_used_meta":290872,"kb_avail":90591800,"statfs":{"total":95995035648,"available":92766003200,"internally_reserved":0,"allocated":2931163136,"data_stored":2930025625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":297853090},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762415,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2428028,"kb_used_data":2115892,"kb_used_omap":7,"kb_used_meta":312120,"kb_avail":91317124,"statfs":{"total":95995035648,"available":93508734976,"internally_reserved":0,"allocated":2166673408,"data_stored":2165542886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7945,"internal_metadata":319611127},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":94,"seq":403726925942,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1717336,"kb_used_data":1460748,"kb_used_omap":7,"kb_used_meta":256568,"kb_avail":92027816,"statfs":{"total":95995035648,"available":94236483584,"internally_reserved":0,"allocated":1495805952,"data_stored":1494660585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":262725953},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6241,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1249837056,"data_stored":1249837056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2176909312,"data_stored":2176909312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2950955008,"data_stored":2950955008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3103850496,"data_stored":3103850496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1474428928,"data_stored":1474428928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1775828992,"data_stored":1775828992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2390097920,"data_stored":2390097920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:25:40.471 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:25:40.471 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:40 smithi053 ceph-mon[89963]: osdmap e204: 8 total, 8 up, 7 in 2024-08-17T17:25:40.471 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:40 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 16932/236424 objects degraded (7.162%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:40.472 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:25:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:25:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:25:40.563 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:40.586 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:40.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:40.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:40.688 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:40.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:40.712 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:40.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:40.813 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:40.841 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:40.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:40.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:40.866 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:25:40.902 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-17T17:25:40.942 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:40.995 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:40.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:40.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:41.092 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 72 16 83054 83038 72.0448 15.5625 0.0280622 0.0138735 2024-08-17T17:25:41.098 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:41.108 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:41 smithi003 ceph-mon[103344]: pgmap v508: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 50 MiB/s wr, 798 op/s; 16932/242355 objects degraded (6.986%); 2267/242355 objects misplaced (0.935%); 4.6 MiB/s, 72 objects/s recovering 2024-08-17T17:25:41.108 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:41 smithi003 ceph-mon[103344]: osdmap e205: 8 total, 8 up, 7 in 2024-08-17T17:25:41.125 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:41.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:41.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:41.227 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:41.253 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:41.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:41.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:41.266 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:25:41.355 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:41.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:41 smithi003 ceph-mon[100615]: pgmap v508: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 50 MiB/s wr, 798 op/s; 16932/242355 objects degraded (6.986%); 2267/242355 objects misplaced (0.935%); 4.6 MiB/s, 72 objects/s recovering 2024-08-17T17:25:41.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:41 smithi003 ceph-mon[100615]: osdmap e205: 8 total, 8 up, 7 in 2024-08-17T17:25:41.442 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:41.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:41.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:41.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:41 smithi053 ceph-mon[89963]: pgmap v508: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 50 MiB/s wr, 798 op/s; 16932/242355 objects degraded (6.986%); 2267/242355 objects misplaced (0.935%); 4.6 MiB/s, 72 objects/s recovering 2024-08-17T17:25:41.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:41 smithi053 ceph-mon[89963]: osdmap e205: 8 total, 8 up, 7 in 2024-08-17T17:25:41.544 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:41.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:41.603 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:41.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:41.704 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:41.745 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:41.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:41.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:41.848 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:41.888 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:41.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:41.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:41.991 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:42.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:42.039 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:42.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:42.094 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 73 16 83943 83927 71.8173 55.5625 0.0158794 0.013918 2024-08-17T17:25:42.141 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:42.177 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:25:42.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:42.227 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:42.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:42.230 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:25:42.246 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:25:42.328 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:42.356 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:42.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:42.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:42.458 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:42.494 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:42.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:42.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:42.596 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:42.645 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:42.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:42.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:42.746 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:42.776 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:42.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:42.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:42.878 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:42.924 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:42.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:42.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:43.026 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:43.072 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:43.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:43.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:43.094 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 74 16 84801 84785 71.5715 53.625 0.0262292 0.0139664 2024-08-17T17:25:43.174 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:43.213 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:43.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:43.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:43.218 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:25:43.218 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:25:43.218 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:25:43.218 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:25:43.218 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:25:43.218 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:25:43.218 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:25:43.218 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:25:43.219 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:25:43.219 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:25:43.244 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:25:43.245 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:25:43.246 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:43 smithi003 ceph-mon[100615]: pgmap v510: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 54 MiB/s wr, 866 op/s; 16932/247275 objects degraded (6.847%); 1850/247275 objects misplaced (0.748%); 7.8 MiB/s, 124 objects/s recovering 2024-08-17T17:25:43.246 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:43 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1380003850' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:25:43.246 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:43 smithi003 ceph-mon[103344]: pgmap v510: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 54 MiB/s wr, 866 op/s; 16932/247275 objects degraded (6.847%); 1850/247275 objects misplaced (0.748%); 7.8 MiB/s, 124 objects/s recovering 2024-08-17T17:25:43.246 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:43 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1380003850' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:25:43.284 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:25:43.285 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:25:43.315 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:43.316 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:25:43.316 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:25:43.348 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:43.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:43.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:43.450 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:43.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:43 smithi053 ceph-mon[89963]: pgmap v510: 89 pgs: 1 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 81 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 54 MiB/s wr, 866 op/s; 16932/247275 objects degraded (6.847%); 1850/247275 objects misplaced (0.748%); 7.8 MiB/s, 124 objects/s recovering 2024-08-17T17:25:43.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:43 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1380003850' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:25:43.478 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:43.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:43.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:43.580 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:43.616 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:43.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:43.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:43.718 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:43.749 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:43.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:43.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:43.851 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:43.901 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:43.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:43.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:44.003 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:44.041 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:44.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:44.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:44.094 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 75 16 85724 85708 71.3864 57.6875 0.0306574 0.0140028 2024-08-17T17:25:44.143 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:44.179 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:44.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:44.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:44.281 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:44.303 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:44.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:44.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:44.405 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:44.446 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:44.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:44.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:44.548 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:44.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:44.595 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:44.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:44.696 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:44.734 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:44.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:44.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:44.835 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:44.879 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:44.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:44.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:44.981 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:45.023 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:45.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:45.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:45.094 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 76 16 86641 86625 71.2012 57.3125 0.0120135 0.0140394 2024-08-17T17:25:45.125 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:45.164 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:45.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:45.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:45.266 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:45.303 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:45.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:45.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:45.399 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:45 smithi003 ceph-mon[103344]: pgmap v511: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 48 MiB/s wr, 773 op/s; 16932/250728 objects degraded (6.753%); 1548/250728 objects misplaced (0.617%); 7.1 MiB/s, 113 objects/s recovering 2024-08-17T17:25:45.399 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:45 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 16932/250728 objects degraded (6.753%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:45.399 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:45 smithi003 ceph-mon[100615]: pgmap v511: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 48 MiB/s wr, 773 op/s; 16932/250728 objects degraded (6.753%); 1548/250728 objects misplaced (0.617%); 7.1 MiB/s, 113 objects/s recovering 2024-08-17T17:25:45.399 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:45 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 16932/250728 objects degraded (6.753%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:45.405 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:45.433 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:45.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:45.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:45.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:45 smithi053 ceph-mon[89963]: pgmap v511: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 48 MiB/s wr, 773 op/s; 16932/250728 objects degraded (6.753%); 1548/250728 objects misplaced (0.617%); 7.1 MiB/s, 113 objects/s recovering 2024-08-17T17:25:45.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:45 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 16932/250728 objects degraded (6.753%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:45.535 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:45.568 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:45.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:45.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:45.670 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:45.700 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:45.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:45.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:45.801 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:45.844 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:45.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:45.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:45.935 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:25:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:25:45] "GET /metrics HTTP/1.1" 200 36526 "" "Prometheus/2.51.0" 2024-08-17T17:25:45.945 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:45.974 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:45.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:45.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:46.076 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:46.095 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 77 16 87500 87484 70.9738 53.6875 0.0293638 0.0140836 2024-08-17T17:25:46.119 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:46.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:46.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:46.220 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:46.248 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:46.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:46.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:46.350 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:46.403 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:46.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:46.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:46.505 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:46.553 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:46.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:46.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:46.655 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:46.691 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:46.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:46.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:46.793 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:46.821 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:46.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:46.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:46.923 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:46.953 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:46.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:46.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:47.055 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:47.086 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:47.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:47.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:47.094 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 78 16 88270 88254 70.6809 48.125 0.0165282 0.0141417 2024-08-17T17:25:47.188 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:47.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:47.229 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:47.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:47.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:47 smithi003 ceph-mon[100615]: pgmap v512: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 48 MiB/s wr, 770 op/s; 16932/254925 objects degraded (6.642%); 1548/254925 objects misplaced (0.607%); 5.6 MiB/s, 89 objects/s recovering 2024-08-17T17:25:47.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:47 smithi003 ceph-mon[103344]: pgmap v512: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 48 MiB/s wr, 770 op/s; 16932/254925 objects degraded (6.642%); 1548/254925 objects misplaced (0.607%); 5.6 MiB/s, 89 objects/s recovering 2024-08-17T17:25:47.330 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:47.371 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:47.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:47.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:47.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:47 smithi053 ceph-mon[89963]: pgmap v512: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 48 MiB/s wr, 770 op/s; 16932/254925 objects degraded (6.642%); 1548/254925 objects misplaced (0.607%); 5.6 MiB/s, 89 objects/s recovering 2024-08-17T17:25:47.473 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:47.506 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:47.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:47.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:47.608 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:47.657 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:47.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:47.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:47.759 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:47.798 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:47.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:47.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:47.899 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:47.934 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:47.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:47.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:48.036 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:48.064 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:48.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:48.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:48.094 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 79 16 89213 89197 70.5323 58.9375 0.00726761 0.0141724 2024-08-17T17:25:48.166 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:48.209 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:48.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:48.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:48.311 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:48.346 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:48.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:48.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:48.448 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:48.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:48.497 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:48.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:48.598 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:48.652 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:48.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:48.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:48.754 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:48.782 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:48.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:48.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:48.884 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:48.913 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:48.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:48.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:49.015 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:49.061 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:49.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:49.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:49.095 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-08-17T17:25:49.095177+0000 min lat: 0.00112466 max lat: 2.58819 avg lat: 0.0142097 2024-08-17T17:25:49.095 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:25:49.095 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 80 16 90101 90085 70.3444 55.5 0.0108769 0.0142097 2024-08-17T17:25:49.164 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:49.193 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:49.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:49.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:49.295 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:49.326 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:49.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:49.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:49.428 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:49.457 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:49.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:49.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:49.460 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:49 smithi003 ceph-mon[100615]: pgmap v513: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 40 MiB/s wr, 634 op/s; 16932/259035 objects degraded (6.537%); 1306/259035 objects misplaced (0.504%); 6.9 MiB/s, 109 objects/s recovering 2024-08-17T17:25:49.460 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:49 smithi003 ceph-mon[103344]: pgmap v513: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 40 MiB/s wr, 634 op/s; 16932/259035 objects degraded (6.537%); 1306/259035 objects misplaced (0.504%); 6.9 MiB/s, 109 objects/s recovering 2024-08-17T17:25:49.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:49 smithi053 ceph-mon[89963]: pgmap v513: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 40 MiB/s wr, 634 op/s; 16932/259035 objects degraded (6.537%); 1306/259035 objects misplaced (0.504%); 6.9 MiB/s, 109 objects/s recovering 2024-08-17T17:25:49.559 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:49.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:49.588 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:49.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:49.690 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:49.746 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:49.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:49.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:49.848 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:49.874 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:49.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:49.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:49.976 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:50.018 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:50.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:50.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:50.095 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 81 16 90939 90923 70.1226 52.375 0.015316 0.0142549 2024-08-17T17:25:50.120 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:50.154 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-08-17T17:25:50.161 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:50.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:50.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:50.263 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:50.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:50 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2125930342' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:25:50.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:50 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:25:50.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:50 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 16932/259035 objects degraded (6.537%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:50.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:50 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2125930342' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:25:50.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:50 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:25:50.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:50 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 16932/259035 objects degraded (6.537%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:50.290 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:50.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:50.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:50.391 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:50.420 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:50.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:50.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:50.462 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:25:50.465 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:25:50.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:50 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2125930342' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:25:50.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:50 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:25:50.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:50 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 16932/259035 objects degraded (6.537%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:50.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:25:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:25:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:25:50.522 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:50.538 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:25:50.538 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:25:50.563 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:50.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:50.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:50.666 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:50.697 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:50.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:50.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:50.799 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:50.831 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:50.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:50.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:50.933 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:50.964 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:50.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:50.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:51.065 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:51.095 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 82 16 91834 91818 69.9496 55.9375 0.0148077 0.0142904 2024-08-17T17:25:51.095 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:51.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:51.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:51.197 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:51.215 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:25:51.239 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:51.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:51.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:51.340 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:51.351 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-17T17:25:51.366 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:51.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:51.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:51.404 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:51 smithi003 ceph-mon[100615]: pgmap v514: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 48 MiB/s wr, 769 op/s; 16932/265461 objects degraded (6.378%); 1306/265461 objects misplaced (0.492%); 6.0 MiB/s, 96 objects/s recovering 2024-08-17T17:25:51.405 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:51 smithi003 ceph-mon[100615]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:25:51.405 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:51 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:25:51.405 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:51 smithi003 ceph-mon[100615]: osdmap e206: 8 total, 8 up, 7 in 2024-08-17T17:25:51.405 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:51 smithi003 ceph-mon[100615]: from='client.55135 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:25:51.405 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:51 smithi003 ceph-mon[103344]: pgmap v514: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 48 MiB/s wr, 769 op/s; 16932/265461 objects degraded (6.378%); 1306/265461 objects misplaced (0.492%); 6.0 MiB/s, 96 objects/s recovering 2024-08-17T17:25:51.405 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:51 smithi003 ceph-mon[103344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:25:51.405 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:51 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:25:51.405 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:51 smithi003 ceph-mon[103344]: osdmap e206: 8 total, 8 up, 7 in 2024-08-17T17:25:51.405 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:51 smithi003 ceph-mon[103344]: from='client.55135 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:25:51.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:51 smithi053 ceph-mon[89963]: pgmap v514: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 48 MiB/s wr, 769 op/s; 16932/265461 objects degraded (6.378%); 1306/265461 objects misplaced (0.492%); 6.0 MiB/s, 96 objects/s recovering 2024-08-17T17:25:51.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:51 smithi053 ceph-mon[89963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:25:51.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:51 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:25:51.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:51 smithi053 ceph-mon[89963]: osdmap e206: 8 total, 8 up, 7 in 2024-08-17T17:25:51.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:51 smithi053 ceph-mon[89963]: from='client.55135 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:25:51.468 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:51.496 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:51.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:51.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:51.598 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:51.651 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:51.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:51.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:51.753 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:51.787 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:51.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:51.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:51.889 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:51.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:51.936 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:51.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:52.037 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:52.042 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":515,"stamp":"2024-08-17T17:25:50.146732+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5802039245,"num_objects":88487,"num_object_clones":0,"num_object_copies":265461,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16932,"num_objects_misplaced":1306,"num_objects_unfound":0,"num_objects_dirty":88487,"num_whiteouts":0,"num_read":6376,"num_read_kb":13833,"num_write":89581,"num_write_kb":5676415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5053,"num_bytes_recovered":328869254,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89640,"ondisk_log_size":89640,"up":267,"acting":261,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":284,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":19290852,"kb_used_data":17270508,"kb_used_omap":33,"kb_used_meta":2020254,"kb_avail":636925212,"statfs":{"total":671965249536,"available":652211417088,"internally_reserved":0,"allocated":17685000192,"data_stored":17676889325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34068,"internal_metadata":2068740844},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":504758272,"num_objects":7702,"num_object_clones":0,"num_object_copies":23106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-961,"num_objects_unfound":0,"num_objects_dirty":7702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7702,"num_write_kb":492928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":962,"num_bytes_recovered":63045632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.044062"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":263,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.109352+0000","last_change":"2024-08-17T17:17:32.495652+0000","last_active":"2024-08-17T17:25:40.109352+0000","last_peered":"2024-08-17T17:25:40.109352+0000","last_clean":"2024-08-17T17:25:40.109352+0000","last_became_active":"2024-08-17T17:17:32.495218+0000","last_became_peered":"2024-08-17T17:17:32.495218+0000","last_unstale":"2024-08-17T17:25:40.109352+0000","last_undegraded":"2024-08-17T17:25:40.109352+0000","last_fullsized":"2024-08-17T17:25:40.109352+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:11:45.821282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":181,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-08-17T17:25:39.119603+0000","last_change":"2024-08-17T17:18:59.424887+0000","last_active":"2024-08-17T17:25:39.119603+0000","last_peered":"2024-08-17T17:25:39.119603+0000","last_clean":"2024-08-17T17:25:39.119603+0000","last_became_active":"2024-08-17T17:18:59.411539+0000","last_became_peered":"2024-08-17T17:18:59.411539+0000","last_unstale":"2024-08-17T17:25:39.119603+0000","last_undegraded":"2024-08-17T17:25:39.119603+0000","last_fullsized":"2024-08-17T17:25:39.119603+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:46:20.082241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-08-17T17:25:39.113706+0000","last_change":"2024-08-17T17:24:53.460742+0000","last_active":"2024-08-17T17:25:39.113706+0000","last_peered":"2024-08-17T17:25:39.113706+0000","last_clean":"2024-08-17T17:25:39.113706+0000","last_became_active":"2024-08-17T17:24:53.460365+0000","last_became_peered":"2024-08-17T17:24:53.460365+0000","last_unstale":"2024-08-17T17:25:39.113706+0000","last_undegraded":"2024-08-17T17:25:39.113706+0000","last_fullsized":"2024-08-17T17:25:39.113706+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:24:08.116968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":74,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.766629+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:25:40.766629+0000","last_peered":"2024-08-17T17:25:40.766629+0000","last_clean":"2024-08-17T17:25:40.766629+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:25:40.766629+0000","last_undegraded":"2024-08-17T17:25:40.766629+0000","last_fullsized":"2024-08-17T17:25:40.766629+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":74,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.766208+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:25:40.766208+0000","last_peered":"2024-08-17T17:25:40.766208+0000","last_clean":"2024-08-17T17:25:40.766208+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:25:40.766208+0000","last_undegraded":"2024-08-17T17:25:40.766208+0000","last_fullsized":"2024-08-17T17:25:40.766208+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":156,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-08-17T17:25:39.119395+0000","last_change":"2024-08-17T17:20:49.258206+0000","last_active":"2024-08-17T17:25:39.119395+0000","last_peered":"2024-08-17T17:25:39.119395+0000","last_clean":"2024-08-17T17:25:39.119395+0000","last_became_active":"2024-08-17T17:20:49.257761+0000","last_became_peered":"2024-08-17T17:20:49.257761+0000","last_unstale":"2024-08-17T17:25:39.119395+0000","last_undegraded":"2024-08-17T17:25:39.119395+0000","last_fullsized":"2024-08-17T17:25:39.119395+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:38:35.363748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":277,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.109648+0000","last_change":"2024-08-17T17:19:57.514773+0000","last_active":"2024-08-17T17:25:40.109648+0000","last_peered":"2024-08-17T17:25:40.109648+0000","last_clean":"2024-08-17T17:25:40.109648+0000","last_became_active":"2024-08-17T17:19:57.513839+0000","last_became_peered":"2024-08-17T17:19:57.513839+0000","last_unstale":"2024-08-17T17:25:40.109648+0000","last_undegraded":"2024-08-17T17:25:40.109648+0000","last_fullsized":"2024-08-17T17:25:40.109648+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:42:48.063169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":251,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-08-17T17:25:39.115681+0000","last_change":"2024-08-17T17:24:53.465875+0000","last_active":"2024-08-17T17:25:39.115681+0000","last_peered":"2024-08-17T17:25:39.115681+0000","last_clean":"2024-08-17T17:25:39.115681+0000","last_became_active":"2024-08-17T17:24:53.458129+0000","last_became_peered":"2024-08-17T17:24:53.458129+0000","last_unstale":"2024-08-17T17:25:39.115681+0000","last_undegraded":"2024-08-17T17:25:39.115681+0000","last_fullsized":"2024-08-17T17:25:39.115681+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:51:33.584869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"205'5541","reported_seq":5590,"reported_epoch":205,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:25:47.911371+0000","last_change":"2024-08-17T17:24:54.892004+0000","last_active":"2024-08-17T17:25:47.911371+0000","last_peered":"2024-08-17T17:25:47.911371+0000","last_clean":"2024-08-17T17:24:52.438492+0000","last_became_active":"2024-08-17T17:24:54.887269+0000","last_became_peered":"2024-08-17T17:24:54.887269+0000","last_unstale":"2024-08-17T17:25:47.911371+0000","last_undegraded":"2024-08-17T17:24:54.477707+0000","last_fullsized":"2024-08-17T17:24:54.461071+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":5541,"log_dups_size":0,"ondisk_log_size":5541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363134976,"num_objects":5541,"num_object_clones":0,"num_object_copies":16623,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5541,"num_write_kb":354624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[0,6],"avail_no_missing":["0","2","6"],"object_location_counts":[{"shards":"0,2,6","objects":2460},{"shards":"0,6","objects":3081}],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":117,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-08-17T17:25:39.119435+0000","last_change":"2024-08-17T17:24:06.597382+0000","last_active":"2024-08-17T17:25:39.119435+0000","last_peered":"2024-08-17T17:25:39.119435+0000","last_clean":"2024-08-17T17:25:39.119435+0000","last_became_active":"2024-08-17T17:24:06.588881+0000","last_became_peered":"2024-08-17T17:24:06.588881+0000","last_unstale":"2024-08-17T17:25:39.119435+0000","last_undegraded":"2024-08-17T17:25:39.119435+0000","last_fullsized":"2024-08-17T17:25:39.119435+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:27.240751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":31,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-08-17T17:25:39.112783+0000","last_change":"2024-08-17T17:25:04.503780+0000","last_active":"2024-08-17T17:25:39.112783+0000","last_peered":"2024-08-17T17:25:39.112783+0000","last_clean":"2024-08-17T17:25:39.112783+0000","last_became_active":"2024-08-17T17:25:04.503503+0000","last_became_peered":"2024-08-17T17:25:04.503503+0000","last_unstale":"2024-08-17T17:25:39.112783+0000","last_undegraded":"2024-08-17T17:25:39.112783+0000","last_fullsized":"2024-08-17T17:25:39.112783+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:04:48.171216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"205'5493","reported_seq":5558,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:46.412558+0000","last_change":"2024-08-17T17:24:14.639361+0000","last_active":"2024-08-17T17:25:46.412558+0000","last_peered":"2024-08-17T17:25:46.412558+0000","last_clean":"2024-08-17T17:25:46.412558+0000","last_became_active":"2024-08-17T17:24:14.639074+0000","last_became_peered":"2024-08-17T17:24:14.639074+0000","last_unstale":"2024-08-17T17:25:46.412558+0000","last_undegraded":"2024-08-17T17:25:46.412558+0000","last_fullsized":"2024-08-17T17:25:46.412558+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":5493,"log_dups_size":0,"ondisk_log_size":5493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:15:05.377992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359989248,"num_objects":5493,"num_object_clones":0,"num_object_copies":16479,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5493,"num_write_kb":351552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":103,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-08-17T17:25:39.119789+0000","last_change":"2024-08-17T17:24:06.599023+0000","last_active":"2024-08-17T17:25:39.119789+0000","last_peered":"2024-08-17T17:25:39.119789+0000","last_clean":"2024-08-17T17:25:39.119789+0000","last_became_active":"2024-08-17T17:24:06.588954+0000","last_became_peered":"2024-08-17T17:24:06.588954+0000","last_unstale":"2024-08-17T17:25:39.119789+0000","last_undegraded":"2024-08-17T17:25:39.119789+0000","last_fullsized":"2024-08-17T17:25:39.119789+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:20:51.445576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":218,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.111404+0000","last_change":"2024-08-17T17:20:49.271540+0000","last_active":"2024-08-17T17:25:40.111404+0000","last_peered":"2024-08-17T17:25:40.111404+0000","last_clean":"2024-08-17T17:25:40.111404+0000","last_became_active":"2024-08-17T17:20:49.271130+0000","last_became_peered":"2024-08-17T17:20:49.271130+0000","last_unstale":"2024-08-17T17:25:40.111404+0000","last_undegraded":"2024-08-17T17:25:40.111404+0000","last_fullsized":"2024-08-17T17:25:40.111404+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:05.386705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"205'5565","reported_seq":5600,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:48.917036+0000","last_change":"2024-08-17T17:24:14.644070+0000","last_active":"2024-08-17T17:25:48.917036+0000","last_peered":"2024-08-17T17:25:48.917036+0000","last_clean":"2024-08-17T17:25:48.917036+0000","last_became_active":"2024-08-17T17:24:14.643799+0000","last_became_peered":"2024-08-17T17:24:14.643799+0000","last_unstale":"2024-08-17T17:25:48.917036+0000","last_undegraded":"2024-08-17T17:25:48.917036+0000","last_fullsized":"2024-08-17T17:25:48.917036+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":5565,"log_dups_size":0,"ondisk_log_size":5565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:25:38.445770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364707840,"num_objects":5565,"num_object_clones":0,"num_object_copies":16695,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5565,"num_write_kb":356160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":175,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-08-17T17:25:39.120260+0000","last_change":"2024-08-17T17:24:53.467526+0000","last_active":"2024-08-17T17:25:39.120260+0000","last_peered":"2024-08-17T17:25:39.120260+0000","last_clean":"2024-08-17T17:25:39.120260+0000","last_became_active":"2024-08-17T17:24:53.460631+0000","last_became_peered":"2024-08-17T17:24:53.460631+0000","last_unstale":"2024-08-17T17:25:39.120260+0000","last_undegraded":"2024-08-17T17:25:39.120260+0000","last_fullsized":"2024-08-17T17:25:39.120260+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:05:24.938831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-08-17T17:25:39.115957+0000","last_change":"2024-08-17T17:25:04.507176+0000","last_active":"2024-08-17T17:25:39.115957+0000","last_peered":"2024-08-17T17:25:39.115957+0000","last_clean":"2024-08-17T17:25:39.115957+0000","last_became_active":"2024-08-17T17:25:04.506788+0000","last_became_peered":"2024-08-17T17:25:04.506788+0000","last_unstale":"2024-08-17T17:25:39.115957+0000","last_undegraded":"2024-08-17T17:25:39.115957+0000","last_fullsized":"2024-08-17T17:25:39.115957+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:22:04.658216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"205'5445","reported_seq":5510,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:47.918756+0000","last_change":"2024-08-17T17:24:14.642637+0000","last_active":"2024-08-17T17:25:47.918756+0000","last_peered":"2024-08-17T17:25:47.918756+0000","last_clean":"2024-08-17T17:25:47.918756+0000","last_became_active":"2024-08-17T17:24:14.642347+0000","last_became_peered":"2024-08-17T17:24:14.642347+0000","last_unstale":"2024-08-17T17:25:47.918756+0000","last_undegraded":"2024-08-17T17:25:47.918756+0000","last_fullsized":"2024-08-17T17:25:47.918756+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":5445,"log_dups_size":0,"ondisk_log_size":5445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:37:44.631335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356843520,"num_objects":5445,"num_object_clones":0,"num_object_copies":16335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5445,"num_write_kb":348480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":163,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-08-17T17:25:39.113274+0000","last_change":"2024-08-17T17:17:32.495428+0000","last_active":"2024-08-17T17:25:39.113274+0000","last_peered":"2024-08-17T17:25:39.113274+0000","last_clean":"2024-08-17T17:25:39.113274+0000","last_became_active":"2024-08-17T17:17:32.494978+0000","last_became_peered":"2024-08-17T17:17:32.494978+0000","last_unstale":"2024-08-17T17:25:39.113274+0000","last_undegraded":"2024-08-17T17:25:39.113274+0000","last_fullsized":"2024-08-17T17:25:39.113274+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:42:24.647812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":266,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.766301+0000","last_change":"2024-08-17T17:20:49.272126+0000","last_active":"2024-08-17T17:25:40.766301+0000","last_peered":"2024-08-17T17:25:40.766301+0000","last_clean":"2024-08-17T17:25:40.766301+0000","last_became_active":"2024-08-17T17:20:49.271185+0000","last_became_peered":"2024-08-17T17:20:49.271185+0000","last_unstale":"2024-08-17T17:25:40.766301+0000","last_undegraded":"2024-08-17T17:25:40.766301+0000","last_fullsized":"2024-08-17T17:25:40.766301+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:43:58.184939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"205'5624","reported_seq":6564,"reported_epoch":205,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:25:45.242134+0000","last_change":"2024-08-17T17:24:54.898729+0000","last_active":"2024-08-17T17:25:45.242134+0000","last_peered":"2024-08-17T17:25:45.242134+0000","last_clean":"2024-08-17T17:24:52.435411+0000","last_became_active":"2024-08-17T17:24:54.889545+0000","last_became_peered":"2024-08-17T17:24:54.889545+0000","last_unstale":"2024-08-17T17:25:45.242134+0000","last_undegraded":"2024-08-17T17:24:54.482592+0000","last_fullsized":"2024-08-17T17:24:54.459529+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":5624,"log_dups_size":0,"ondisk_log_size":5624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368574464,"num_objects":5624,"num_object_clones":0,"num_object_copies":16872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5624,"num_write_kb":359936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":430,"num_bytes_recovered":28180480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2],"avail_no_missing":["1","2","3","7"],"object_location_counts":[{"shards":"1,2","objects":3037},{"shards":"1,2,7","objects":2587}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":294,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.109363+0000","last_change":"2024-08-17T17:24:06.595380+0000","last_active":"2024-08-17T17:25:40.109363+0000","last_peered":"2024-08-17T17:25:40.109363+0000","last_clean":"2024-08-17T17:25:40.109363+0000","last_became_active":"2024-08-17T17:24:06.584747+0000","last_became_peered":"2024-08-17T17:24:06.584747+0000","last_unstale":"2024-08-17T17:25:40.109363+0000","last_undegraded":"2024-08-17T17:25:40.109363+0000","last_fullsized":"2024-08-17T17:25:40.109363+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:02:18.473980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":240,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.109657+0000","last_change":"2024-08-17T17:20:49.280596+0000","last_active":"2024-08-17T17:25:40.109657+0000","last_peered":"2024-08-17T17:25:40.109657+0000","last_clean":"2024-08-17T17:25:40.109657+0000","last_became_active":"2024-08-17T17:20:49.279553+0000","last_became_peered":"2024-08-17T17:20:49.279553+0000","last_unstale":"2024-08-17T17:25:40.109657+0000","last_undegraded":"2024-08-17T17:25:40.109657+0000","last_fullsized":"2024-08-17T17:25:40.109657+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:13:24.124392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"205'5391","reported_seq":5456,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:46.380829+0000","last_change":"2024-08-17T17:24:14.638890+0000","last_active":"2024-08-17T17:25:46.380829+0000","last_peered":"2024-08-17T17:25:46.380829+0000","last_clean":"2024-08-17T17:25:46.380829+0000","last_became_active":"2024-08-17T17:24:14.638594+0000","last_became_peered":"2024-08-17T17:24:14.638594+0000","last_unstale":"2024-08-17T17:25:46.380829+0000","last_undegraded":"2024-08-17T17:25:46.380829+0000","last_fullsized":"2024-08-17T17:25:46.380829+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":5391,"log_dups_size":0,"ondisk_log_size":5391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:53:34.277077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353304576,"num_objects":5391,"num_object_clones":0,"num_object_copies":16173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5391,"num_write_kb":345024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":192,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.111557+0000","last_change":"2024-08-17T17:20:49.276769+0000","last_active":"2024-08-17T17:25:40.111557+0000","last_peered":"2024-08-17T17:25:40.111557+0000","last_clean":"2024-08-17T17:25:40.111557+0000","last_became_active":"2024-08-17T17:20:49.276498+0000","last_became_peered":"2024-08-17T17:20:49.276498+0000","last_unstale":"2024-08-17T17:25:40.111557+0000","last_undegraded":"2024-08-17T17:25:40.111557+0000","last_fullsized":"2024-08-17T17:25:40.111557+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:39:05.419440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":184,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-08-17T17:25:39.120102+0000","last_change":"2024-08-17T17:20:14.455738+0000","last_active":"2024-08-17T17:25:39.120102+0000","last_peered":"2024-08-17T17:25:39.120102+0000","last_clean":"2024-08-17T17:25:39.120102+0000","last_became_active":"2024-08-17T17:20:14.455469+0000","last_became_peered":"2024-08-17T17:20:14.455469+0000","last_unstale":"2024-08-17T17:25:39.120102+0000","last_undegraded":"2024-08-17T17:25:39.120102+0000","last_fullsized":"2024-08-17T17:25:39.120102+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:27:28.424336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"205'5491","reported_seq":6171,"reported_epoch":205,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:25:46.402366+0000","last_change":"2024-08-17T17:25:05.678234+0000","last_active":"2024-08-17T17:25:46.402366+0000","last_peered":"2024-08-17T17:25:46.402366+0000","last_clean":"2024-08-17T17:24:52.436463+0000","last_became_active":"2024-08-17T17:25:05.673238+0000","last_became_peered":"2024-08-17T17:25:05.673238+0000","last_unstale":"2024-08-17T17:25:46.402366+0000","last_undegraded":"2024-08-17T17:25:05.585769+0000","last_fullsized":"2024-08-17T17:25:05.499112+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":5491,"log_dups_size":0,"ondisk_log_size":5491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359858176,"num_objects":5491,"num_object_clones":0,"num_object_copies":16473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5456,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5491,"num_write_kb":351424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":304,"num_bytes_recovered":19922944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,5],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,5,7","objects":305},{"shards":"1,3,7","objects":2215},{"shards":"3,5","objects":2250},{"shards":"3,5,7","objects":721}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":138,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-08-17T17:25:39.119359+0000","last_change":"2024-08-17T17:20:14.431140+0000","last_active":"2024-08-17T17:25:39.119359+0000","last_peered":"2024-08-17T17:25:39.119359+0000","last_clean":"2024-08-17T17:25:39.119359+0000","last_became_active":"2024-08-17T17:20:14.430839+0000","last_became_peered":"2024-08-17T17:20:14.430839+0000","last_unstale":"2024-08-17T17:25:39.119359+0000","last_undegraded":"2024-08-17T17:25:39.119359+0000","last_fullsized":"2024-08-17T17:25:39.119359+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:08:16.303616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-08-17T17:25:39.116022+0000","last_change":"2024-08-17T17:25:04.509325+0000","last_active":"2024-08-17T17:25:39.116022+0000","last_peered":"2024-08-17T17:25:39.116022+0000","last_clean":"2024-08-17T17:25:39.116022+0000","last_became_active":"2024-08-17T17:25:04.508364+0000","last_became_peered":"2024-08-17T17:25:04.508364+0000","last_unstale":"2024-08-17T17:25:39.116022+0000","last_undegraded":"2024-08-17T17:25:39.116022+0000","last_fullsized":"2024-08-17T17:25:39.116022+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:48:56.415763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"205'5534","reported_seq":10357,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:48.805107+0000","last_change":"2024-08-17T17:25:40.811756+0000","last_active":"2024-08-17T17:25:48.805107+0000","last_peered":"2024-08-17T17:25:48.805107+0000","last_clean":"2024-08-17T17:25:48.805107+0000","last_became_active":"2024-08-17T17:25:40.771755+0000","last_became_peered":"2024-08-17T17:25:40.771755+0000","last_unstale":"2024-08-17T17:25:48.805107+0000","last_undegraded":"2024-08-17T17:25:48.805107+0000","last_fullsized":"2024-08-17T17:25:48.805107+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":5534,"log_dups_size":0,"ondisk_log_size":5534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:13:03.918487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362676224,"num_objects":5534,"num_object_clones":0,"num_object_copies":16602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5534,"num_write_kb":354176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2371,"num_bytes_recovered":155385856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":278,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.109114+0000","last_change":"2024-08-17T17:19:57.520745+0000","last_active":"2024-08-17T17:25:40.109114+0000","last_peered":"2024-08-17T17:25:40.109114+0000","last_clean":"2024-08-17T17:25:40.109114+0000","last_became_active":"2024-08-17T17:19:57.520152+0000","last_became_peered":"2024-08-17T17:19:57.520152+0000","last_unstale":"2024-08-17T17:25:40.109114+0000","last_undegraded":"2024-08-17T17:25:40.109114+0000","last_fullsized":"2024-08-17T17:25:40.109114+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:19:39.495644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":254,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.112314+0000","last_change":"2024-08-17T17:20:49.274372+0000","last_active":"2024-08-17T17:25:40.112314+0000","last_peered":"2024-08-17T17:25:40.112314+0000","last_clean":"2024-08-17T17:25:40.112314+0000","last_became_active":"2024-08-17T17:20:49.274023+0000","last_became_peered":"2024-08-17T17:20:49.274023+0000","last_unstale":"2024-08-17T17:25:40.112314+0000","last_undegraded":"2024-08-17T17:25:40.112314+0000","last_fullsized":"2024-08-17T17:25:40.112314+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:31:02.529766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"205'5855","reported_seq":5889,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:48.798230+0000","last_change":"2024-08-17T17:24:14.643398+0000","last_active":"2024-08-17T17:25:48.798230+0000","last_peered":"2024-08-17T17:25:48.798230+0000","last_clean":"2024-08-17T17:25:48.798230+0000","last_became_active":"2024-08-17T17:24:14.643102+0000","last_became_peered":"2024-08-17T17:24:14.643102+0000","last_unstale":"2024-08-17T17:25:48.798230+0000","last_undegraded":"2024-08-17T17:25:48.798230+0000","last_fullsized":"2024-08-17T17:25:48.798230+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":5855,"log_dups_size":0,"ondisk_log_size":5855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:09:30.323709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383713280,"num_objects":5855,"num_object_clones":0,"num_object_copies":17565,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5855,"num_write_kb":374720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":532,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-08-17T17:25:39.119002+0000","last_change":"2024-08-17T17:24:09.599615+0000","last_active":"2024-08-17T17:25:39.119002+0000","last_peered":"2024-08-17T17:25:39.119002+0000","last_clean":"2024-08-17T17:25:39.119002+0000","last_became_active":"2024-08-17T17:24:09.599085+0000","last_became_peered":"2024-08-17T17:24:09.599085+0000","last_unstale":"2024-08-17T17:25:39.119002+0000","last_undegraded":"2024-08-17T17:25:39.119002+0000","last_fullsized":"2024-08-17T17:25:39.119002+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:04:01.141401+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:30:09.569579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":777,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-08-17T17:25:39.113109+0000","last_change":"2024-08-17T17:20:00.434774+0000","last_active":"2024-08-17T17:25:39.113109+0000","last_peered":"2024-08-17T17:25:39.113109+0000","last_clean":"2024-08-17T17:25:39.113109+0000","last_became_active":"2024-08-17T17:20:00.434343+0000","last_became_peered":"2024-08-17T17:20:00.434343+0000","last_unstale":"2024-08-17T17:25:39.113109+0000","last_undegraded":"2024-08-17T17:25:39.113109+0000","last_fullsized":"2024-08-17T17:25:39.113109+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:04.995152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":236,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.109087+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:25:40.109087+0000","last_peered":"2024-08-17T17:25:40.109087+0000","last_clean":"2024-08-17T17:25:40.109087+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:25:40.109087+0000","last_undegraded":"2024-08-17T17:25:40.109087+0000","last_fullsized":"2024-08-17T17:25:40.109087+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-08-17T17:25:39.120937+0000","last_change":"2024-08-17T17:25:04.579426+0000","last_active":"2024-08-17T17:25:39.120937+0000","last_peered":"2024-08-17T17:25:39.120937+0000","last_clean":"2024-08-17T17:25:39.120937+0000","last_became_active":"2024-08-17T17:25:04.579012+0000","last_became_peered":"2024-08-17T17:25:04.579012+0000","last_unstale":"2024-08-17T17:25:39.120937+0000","last_undegraded":"2024-08-17T17:25:39.120937+0000","last_fullsized":"2024-08-17T17:25:39.120937+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:56:32.241321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"205'5329","reported_seq":5394,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:45.206978+0000","last_change":"2024-08-17T17:24:14.641829+0000","last_active":"2024-08-17T17:25:45.206978+0000","last_peered":"2024-08-17T17:25:45.206978+0000","last_clean":"2024-08-17T17:25:45.206978+0000","last_became_active":"2024-08-17T17:24:14.641532+0000","last_became_peered":"2024-08-17T17:24:14.641532+0000","last_unstale":"2024-08-17T17:25:45.206978+0000","last_undegraded":"2024-08-17T17:25:45.206978+0000","last_fullsized":"2024-08-17T17:25:45.206978+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":5329,"log_dups_size":0,"ondisk_log_size":5329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:43:05.232036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349241344,"num_objects":5329,"num_object_clones":0,"num_object_copies":15987,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5329,"num_write_kb":341056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":795,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-08-17T17:25:39.119134+0000","last_change":"2024-08-17T17:16:26.958415+0000","last_active":"2024-08-17T17:25:39.119134+0000","last_peered":"2024-08-17T17:25:39.119134+0000","last_clean":"2024-08-17T17:25:39.119134+0000","last_became_active":"2024-08-17T17:16:26.957956+0000","last_became_peered":"2024-08-17T17:16:26.957956+0000","last_unstale":"2024-08-17T17:25:39.119134+0000","last_undegraded":"2024-08-17T17:25:39.119134+0000","last_fullsized":"2024-08-17T17:25:39.119134+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:22.478077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":265,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.765974+0000","last_change":"2024-08-17T17:18:59.415331+0000","last_active":"2024-08-17T17:25:40.765974+0000","last_peered":"2024-08-17T17:25:40.765974+0000","last_clean":"2024-08-17T17:25:40.765974+0000","last_became_active":"2024-08-17T17:18:59.415008+0000","last_became_peered":"2024-08-17T17:18:59.415008+0000","last_unstale":"2024-08-17T17:25:40.765974+0000","last_undegraded":"2024-08-17T17:25:40.765974+0000","last_fullsized":"2024-08-17T17:25:40.765974+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:22:38.076866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":74,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.766409+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:25:40.766409+0000","last_peered":"2024-08-17T17:25:40.766409+0000","last_clean":"2024-08-17T17:25:40.766409+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:25:40.766409+0000","last_undegraded":"2024-08-17T17:25:40.766409+0000","last_fullsized":"2024-08-17T17:25:40.766409+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"205'5557","reported_seq":5601,"reported_epoch":205,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:25:48.917773+0000","last_change":"2024-08-17T17:25:05.609285+0000","last_active":"2024-08-17T17:25:48.917773+0000","last_peered":"2024-08-17T17:25:48.917773+0000","last_clean":"2024-08-17T17:25:03.970571+0000","last_became_active":"2024-08-17T17:25:05.604478+0000","last_became_peered":"2024-08-17T17:25:05.604478+0000","last_unstale":"2024-08-17T17:25:48.917773+0000","last_undegraded":"2024-08-17T17:25:05.510212+0000","last_fullsized":"2024-08-17T17:25:05.498949+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":5557,"log_dups_size":0,"ondisk_log_size":5557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:29:09.928695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":364183552,"num_objects":5557,"num_object_clones":0,"num_object_copies":16671,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5557,"num_write_kb":355648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,3],"avail_no_missing":["7","2","3"],"object_location_counts":[{"shards":"2,3,7","objects":3108},{"shards":"3,7","objects":2449}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":653,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.109142+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:25:40.109142+0000","last_peered":"2024-08-17T17:25:40.109142+0000","last_clean":"2024-08-17T17:25:40.109142+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:25:40.109142+0000","last_undegraded":"2024-08-17T17:25:40.109142+0000","last_fullsized":"2024-08-17T17:25:40.109142+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":261,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.110963+0000","last_change":"2024-08-17T17:19:57.515564+0000","last_active":"2024-08-17T17:25:40.110963+0000","last_peered":"2024-08-17T17:25:40.110963+0000","last_clean":"2024-08-17T17:25:40.110963+0000","last_became_active":"2024-08-17T17:19:57.515142+0000","last_became_peered":"2024-08-17T17:19:57.515142+0000","last_unstale":"2024-08-17T17:25:40.110963+0000","last_undegraded":"2024-08-17T17:25:40.110963+0000","last_fullsized":"2024-08-17T17:25:40.110963+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:59:33.506176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":292,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.112064+0000","last_change":"2024-08-17T17:24:06.587903+0000","last_active":"2024-08-17T17:25:40.112064+0000","last_peered":"2024-08-17T17:25:40.112064+0000","last_clean":"2024-08-17T17:25:40.112064+0000","last_became_active":"2024-08-17T17:24:06.587503+0000","last_became_peered":"2024-08-17T17:24:06.587503+0000","last_unstale":"2024-08-17T17:25:40.112064+0000","last_undegraded":"2024-08-17T17:25:40.112064+0000","last_fullsized":"2024-08-17T17:25:40.112064+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:05:58.591992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"205'5385","reported_seq":9047,"reported_epoch":205,"state":"active+recovering+undersized+remapped","last_fresh":"2024-08-17T17:25:46.440193+0000","last_change":"2024-08-17T17:25:04.657561+0000","last_active":"2024-08-17T17:25:46.440193+0000","last_peered":"2024-08-17T17:25:46.440193+0000","last_clean":"2024-08-17T17:24:52.452117+0000","last_became_active":"2024-08-17T17:25:04.596208+0000","last_became_peered":"2024-08-17T17:25:04.596208+0000","last_unstale":"2024-08-17T17:25:46.440193+0000","last_undegraded":"2024-08-17T17:25:46.440193+0000","last_fullsized":"2024-08-17T17:25:04.500643+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":5385,"log_dups_size":0,"ondisk_log_size":5385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352911360,"num_objects":5385,"num_object_clones":0,"num_object_copies":16155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1306,"num_objects_unfound":0,"num_objects_dirty":5385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5385,"num_write_kb":344640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1795,"num_bytes_recovered":117637120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":589},{"shards":"3,5,7","objects":717},{"shards":"3,7","objects":4079}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":1115,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-08-17T17:25:39.113017+0000","last_change":"2024-08-17T17:20:00.433870+0000","last_active":"2024-08-17T17:25:39.113017+0000","last_peered":"2024-08-17T17:25:39.113017+0000","last_clean":"2024-08-17T17:25:39.113017+0000","last_became_active":"2024-08-17T17:20:00.433386+0000","last_became_peered":"2024-08-17T17:20:00.433386+0000","last_unstale":"2024-08-17T17:25:39.113017+0000","last_undegraded":"2024-08-17T17:25:39.113017+0000","last_fullsized":"2024-08-17T17:25:39.113017+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:01:11.848862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1392,"num_read_kb":2863,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":187,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.766219+0000","last_change":"2024-08-17T17:20:49.272407+0000","last_active":"2024-08-17T17:25:40.766219+0000","last_peered":"2024-08-17T17:25:40.766219+0000","last_clean":"2024-08-17T17:25:40.766219+0000","last_became_active":"2024-08-17T17:20:49.271749+0000","last_became_peered":"2024-08-17T17:20:49.271749+0000","last_unstale":"2024-08-17T17:25:40.766219+0000","last_undegraded":"2024-08-17T17:25:40.766219+0000","last_fullsized":"2024-08-17T17:25:40.766219+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:36:01.443457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":258,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.111954+0000","last_change":"2024-08-17T17:17:32.499715+0000","last_active":"2024-08-17T17:25:40.111954+0000","last_peered":"2024-08-17T17:25:40.111954+0000","last_clean":"2024-08-17T17:25:40.111954+0000","last_became_active":"2024-08-17T17:17:32.499412+0000","last_became_peered":"2024-08-17T17:17:32.499412+0000","last_unstale":"2024-08-17T17:25:40.111954+0000","last_undegraded":"2024-08-17T17:25:40.111954+0000","last_fullsized":"2024-08-17T17:25:40.111954+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:39:38.456970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"205'5595","reported_seq":5660,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:47.891988+0000","last_change":"2024-08-17T17:24:14.642608+0000","last_active":"2024-08-17T17:25:47.891988+0000","last_peered":"2024-08-17T17:25:47.891988+0000","last_clean":"2024-08-17T17:25:47.891988+0000","last_became_active":"2024-08-17T17:24:14.642292+0000","last_became_peered":"2024-08-17T17:24:14.642292+0000","last_unstale":"2024-08-17T17:25:47.891988+0000","last_undegraded":"2024-08-17T17:25:47.891988+0000","last_fullsized":"2024-08-17T17:25:47.891988+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":5595,"log_dups_size":0,"ondisk_log_size":5595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:53:36.848614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366673920,"num_objects":5595,"num_object_clones":0,"num_object_copies":16785,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5595,"num_write_kb":358080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":180,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-08-17T17:25:39.113786+0000","last_change":"2024-08-17T17:17:32.495728+0000","last_active":"2024-08-17T17:25:39.113786+0000","last_peered":"2024-08-17T17:25:39.113786+0000","last_clean":"2024-08-17T17:25:39.113786+0000","last_became_active":"2024-08-17T17:17:32.495357+0000","last_became_peered":"2024-08-17T17:17:32.495357+0000","last_unstale":"2024-08-17T17:25:39.113786+0000","last_undegraded":"2024-08-17T17:25:39.113786+0000","last_fullsized":"2024-08-17T17:25:39.113786+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:27:45.561567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":494,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.112070+0000","last_change":"2024-08-17T17:16:26.964621+0000","last_active":"2024-08-17T17:25:40.112070+0000","last_peered":"2024-08-17T17:25:40.112070+0000","last_clean":"2024-08-17T17:25:40.112070+0000","last_became_active":"2024-08-17T17:16:26.964058+0000","last_became_peered":"2024-08-17T17:16:26.964058+0000","last_unstale":"2024-08-17T17:25:40.112070+0000","last_undegraded":"2024-08-17T17:25:40.112070+0000","last_fullsized":"2024-08-17T17:25:40.112070+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:00:04.465644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":264,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.111644+0000","last_change":"2024-08-17T17:24:53.465550+0000","last_active":"2024-08-17T17:25:40.111644+0000","last_peered":"2024-08-17T17:25:40.111644+0000","last_clean":"2024-08-17T17:25:40.111644+0000","last_became_active":"2024-08-17T17:24:53.465228+0000","last_became_peered":"2024-08-17T17:24:53.465228+0000","last_unstale":"2024-08-17T17:25:40.111644+0000","last_undegraded":"2024-08-17T17:25:40.111644+0000","last_fullsized":"2024-08-17T17:25:40.111644+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:21:18.378359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"205'5434","reported_seq":5499,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:45.215108+0000","last_change":"2024-08-17T17:24:14.640584+0000","last_active":"2024-08-17T17:25:45.215108+0000","last_peered":"2024-08-17T17:25:45.215108+0000","last_clean":"2024-08-17T17:25:45.215108+0000","last_became_active":"2024-08-17T17:24:14.640168+0000","last_became_peered":"2024-08-17T17:24:14.640168+0000","last_unstale":"2024-08-17T17:25:45.215108+0000","last_undegraded":"2024-08-17T17:25:45.215108+0000","last_fullsized":"2024-08-17T17:25:45.215108+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":5434,"log_dups_size":0,"ondisk_log_size":5434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:38:36.176813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356122624,"num_objects":5434,"num_object_clones":0,"num_object_copies":16302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5434,"num_write_kb":347776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":289,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.112034+0000","last_change":"2024-08-17T17:24:53.464307+0000","last_active":"2024-08-17T17:25:40.112034+0000","last_peered":"2024-08-17T17:25:40.112034+0000","last_clean":"2024-08-17T17:25:40.112034+0000","last_became_active":"2024-08-17T17:24:53.463996+0000","last_became_peered":"2024-08-17T17:24:53.463996+0000","last_unstale":"2024-08-17T17:25:40.112034+0000","last_undegraded":"2024-08-17T17:25:40.112034+0000","last_fullsized":"2024-08-17T17:25:40.112034+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:04:18.993725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":632,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-08-17T17:25:39.120320+0000","last_change":"2024-08-17T17:24:09.599822+0000","last_active":"2024-08-17T17:25:39.120320+0000","last_peered":"2024-08-17T17:25:39.120320+0000","last_clean":"2024-08-17T17:25:39.120320+0000","last_became_active":"2024-08-17T17:24:09.599189+0000","last_became_peered":"2024-08-17T17:24:09.599189+0000","last_unstale":"2024-08-17T17:25:39.120320+0000","last_undegraded":"2024-08-17T17:25:39.120320+0000","last_fullsized":"2024-08-17T17:25:39.120320+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:04.465604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":248,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.766222+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:25:40.766222+0000","last_peered":"2024-08-17T17:25:40.766222+0000","last_clean":"2024-08-17T17:25:40.766222+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:25:40.766222+0000","last_undegraded":"2024-08-17T17:25:40.766222+0000","last_fullsized":"2024-08-17T17:25:40.766222+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"205'5594","reported_seq":5659,"reported_epoch":205,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:25:46.413852+0000","last_change":"2024-08-17T17:25:05.605653+0000","last_active":"2024-08-17T17:25:46.413852+0000","last_peered":"2024-08-17T17:25:46.413852+0000","last_clean":"2024-08-17T17:25:03.971607+0000","last_became_active":"2024-08-17T17:25:05.601333+0000","last_became_peered":"2024-08-17T17:25:05.601333+0000","last_unstale":"2024-08-17T17:25:46.413852+0000","last_undegraded":"2024-08-17T17:25:05.522007+0000","last_fullsized":"2024-08-17T17:25:05.499355+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":5594,"log_dups_size":0,"ondisk_log_size":5594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366608384,"num_objects":5594,"num_object_clones":0,"num_object_copies":16782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5594,"num_write_kb":358016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2],"avail_no_missing":["3","2","7"],"object_location_counts":[{"shards":"2,3","objects":2278},{"shards":"2,3,7","objects":3316}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":283,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.109795+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:25:40.109795+0000","last_peered":"2024-08-17T17:25:40.109795+0000","last_clean":"2024-08-17T17:25:40.109795+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:25:40.109795+0000","last_undegraded":"2024-08-17T17:25:40.109795+0000","last_fullsized":"2024-08-17T17:25:40.109795+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":803,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.113961+0000","last_change":"2024-08-17T17:23:19.633786+0000","last_active":"2024-08-17T17:25:40.113961+0000","last_peered":"2024-08-17T17:25:40.113961+0000","last_clean":"2024-08-17T17:25:40.113961+0000","last_became_active":"2024-08-17T17:23:19.633230+0000","last_became_peered":"2024-08-17T17:23:19.633230+0000","last_unstale":"2024-08-17T17:25:40.113961+0000","last_undegraded":"2024-08-17T17:25:40.113961+0000","last_fullsized":"2024-08-17T17:25:40.113961+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:30:50.377391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":280,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.108973+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:25:40.108973+0000","last_peered":"2024-08-17T17:25:40.108973+0000","last_clean":"2024-08-17T17:25:40.108973+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:25:40.108973+0000","last_undegraded":"2024-08-17T17:25:40.108973+0000","last_fullsized":"2024-08-17T17:25:40.108973+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"205'5626","reported_seq":5661,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:48.909736+0000","last_change":"2024-08-17T17:24:14.639560+0000","last_active":"2024-08-17T17:25:48.909736+0000","last_peered":"2024-08-17T17:25:48.909736+0000","last_clean":"2024-08-17T17:25:48.909736+0000","last_became_active":"2024-08-17T17:24:14.639093+0000","last_became_peered":"2024-08-17T17:24:14.639093+0000","last_unstale":"2024-08-17T17:25:48.909736+0000","last_undegraded":"2024-08-17T17:25:48.909736+0000","last_fullsized":"2024-08-17T17:25:48.909736+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":5626,"log_dups_size":0,"ondisk_log_size":5626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:45:57.940550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368705536,"num_objects":5626,"num_object_clones":0,"num_object_copies":16878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5626,"num_write_kb":360064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":78,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.766304+0000","last_change":"2024-08-17T17:24:06.593660+0000","last_active":"2024-08-17T17:25:40.766304+0000","last_peered":"2024-08-17T17:25:40.766304+0000","last_clean":"2024-08-17T17:25:40.766304+0000","last_became_active":"2024-08-17T17:24:06.586215+0000","last_became_peered":"2024-08-17T17:24:06.586215+0000","last_unstale":"2024-08-17T17:25:40.766304+0000","last_undegraded":"2024-08-17T17:25:40.766304+0000","last_fullsized":"2024-08-17T17:25:40.766304+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:15:49.475395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":422,"reported_epoch":204,"state":"active+recovery_wait+degraded","last_fresh":"2024-08-17T17:25:39.116285+0000","last_change":"2024-08-17T17:25:23.359036+0000","last_active":"2024-08-17T17:25:39.116285+0000","last_peered":"2024-08-17T17:25:39.116285+0000","last_clean":"2024-08-17T17:25:17.220335+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:25:39.116285+0000","last_undegraded":"2024-08-17T17:25:23.336653+0000","last_fullsized":"2024-08-17T17:25:39.116285+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4194304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":["6","2","3"],"object_location_counts":[{"shards":"2,3,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":104,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-08-17T17:25:39.112878+0000","last_change":"2024-08-17T17:18:59.417183+0000","last_active":"2024-08-17T17:25:39.112878+0000","last_peered":"2024-08-17T17:25:39.112878+0000","last_clean":"2024-08-17T17:25:39.112878+0000","last_became_active":"2024-08-17T17:18:59.416843+0000","last_became_peered":"2024-08-17T17:18:59.416843+0000","last_unstale":"2024-08-17T17:25:39.112878+0000","last_undegraded":"2024-08-17T17:25:39.112878+0000","last_fullsized":"2024-08-17T17:25:39.112878+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:29:09.336335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":125,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-08-17T17:25:39.115732+0000","last_change":"2024-08-17T17:20:49.258048+0000","last_active":"2024-08-17T17:25:39.115732+0000","last_peered":"2024-08-17T17:25:39.115732+0000","last_clean":"2024-08-17T17:25:39.115732+0000","last_became_active":"2024-08-17T17:20:49.257672+0000","last_became_peered":"2024-08-17T17:20:49.257672+0000","last_unstale":"2024-08-17T17:25:39.115732+0000","last_undegraded":"2024-08-17T17:25:39.115732+0000","last_fullsized":"2024-08-17T17:25:39.115732+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:08:11.734316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":164,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-08-17T17:25:39.119733+0000","last_change":"2024-08-17T17:18:59.430969+0000","last_active":"2024-08-17T17:25:39.119733+0000","last_peered":"2024-08-17T17:25:39.119733+0000","last_clean":"2024-08-17T17:25:39.119733+0000","last_became_active":"2024-08-17T17:18:59.413475+0000","last_became_peered":"2024-08-17T17:18:59.413475+0000","last_unstale":"2024-08-17T17:25:39.119733+0000","last_undegraded":"2024-08-17T17:25:39.119733+0000","last_fullsized":"2024-08-17T17:25:39.119733+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:36:33.732645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":275,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.111560+0000","last_change":"2024-08-17T17:25:04.506413+0000","last_active":"2024-08-17T17:25:40.111560+0000","last_peered":"2024-08-17T17:25:40.111560+0000","last_clean":"2024-08-17T17:25:40.111560+0000","last_became_active":"2024-08-17T17:25:04.506104+0000","last_became_peered":"2024-08-17T17:25:04.506104+0000","last_unstale":"2024-08-17T17:25:40.111560+0000","last_undegraded":"2024-08-17T17:25:40.111560+0000","last_fullsized":"2024-08-17T17:25:40.111560+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:41:53.556223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":107,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-08-17T17:25:39.120022+0000","last_change":"2024-08-17T17:20:49.279458+0000","last_active":"2024-08-17T17:25:39.120022+0000","last_peered":"2024-08-17T17:25:39.120022+0000","last_clean":"2024-08-17T17:25:39.120022+0000","last_became_active":"2024-08-17T17:20:49.260790+0000","last_became_peered":"2024-08-17T17:20:49.260790+0000","last_unstale":"2024-08-17T17:25:39.120022+0000","last_undegraded":"2024-08-17T17:25:39.120022+0000","last_fullsized":"2024-08-17T17:25:39.120022+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:58.092945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-08-17T17:25:39.113322+0000","last_change":"2024-08-17T17:24:53.463713+0000","last_active":"2024-08-17T17:25:39.113322+0000","last_peered":"2024-08-17T17:25:39.113322+0000","last_clean":"2024-08-17T17:25:39.113322+0000","last_became_active":"2024-08-17T17:24:53.463419+0000","last_became_peered":"2024-08-17T17:24:53.463419+0000","last_unstale":"2024-08-17T17:25:39.113322+0000","last_undegraded":"2024-08-17T17:25:39.113322+0000","last_fullsized":"2024-08-17T17:25:39.113322+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:34:44.545175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":230,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.108982+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:25:40.108982+0000","last_peered":"2024-08-17T17:25:40.108982+0000","last_clean":"2024-08-17T17:25:40.108982+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:25:40.108982+0000","last_undegraded":"2024-08-17T17:25:40.108982+0000","last_fullsized":"2024-08-17T17:25:40.108982+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":258,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.765656+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:25:40.765656+0000","last_peered":"2024-08-17T17:25:40.765656+0000","last_clean":"2024-08-17T17:25:40.765656+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:25:40.765656+0000","last_undegraded":"2024-08-17T17:25:40.765656+0000","last_fullsized":"2024-08-17T17:25:40.765656+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":277,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.108932+0000","last_change":"2024-08-17T17:25:04.509422+0000","last_active":"2024-08-17T17:25:40.108932+0000","last_peered":"2024-08-17T17:25:40.108932+0000","last_clean":"2024-08-17T17:25:40.108932+0000","last_became_active":"2024-08-17T17:25:04.508742+0000","last_became_peered":"2024-08-17T17:25:04.508742+0000","last_unstale":"2024-08-17T17:25:40.108932+0000","last_undegraded":"2024-08-17T17:25:40.108932+0000","last_fullsized":"2024-08-17T17:25:40.108932+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:18:31.976907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":162,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-08-17T17:25:39.116229+0000","last_change":"2024-08-17T17:16:26.963696+0000","last_active":"2024-08-17T17:25:39.116229+0000","last_peered":"2024-08-17T17:25:39.116229+0000","last_clean":"2024-08-17T17:25:39.116229+0000","last_became_active":"2024-08-17T17:16:26.963280+0000","last_became_peered":"2024-08-17T17:16:26.963280+0000","last_unstale":"2024-08-17T17:25:39.116229+0000","last_undegraded":"2024-08-17T17:25:39.116229+0000","last_fullsized":"2024-08-17T17:25:39.116229+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:22:01.474840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":295,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.109083+0000","last_change":"2024-08-17T17:24:06.586886+0000","last_active":"2024-08-17T17:25:40.109083+0000","last_peered":"2024-08-17T17:25:40.109083+0000","last_clean":"2024-08-17T17:25:40.109083+0000","last_became_active":"2024-08-17T17:24:06.586516+0000","last_became_peered":"2024-08-17T17:24:06.586516+0000","last_unstale":"2024-08-17T17:25:40.109083+0000","last_undegraded":"2024-08-17T17:25:40.109083+0000","last_fullsized":"2024-08-17T17:25:40.109083+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:03:30.002726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":74,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.766378+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:25:40.766378+0000","last_peered":"2024-08-17T17:25:40.766378+0000","last_clean":"2024-08-17T17:25:40.766378+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:25:40.766378+0000","last_undegraded":"2024-08-17T17:25:40.766378+0000","last_fullsized":"2024-08-17T17:25:40.766378+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":254,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.111512+0000","last_change":"2024-08-17T17:18:59.412463+0000","last_active":"2024-08-17T17:25:40.111512+0000","last_peered":"2024-08-17T17:25:40.111512+0000","last_clean":"2024-08-17T17:25:40.111512+0000","last_became_active":"2024-08-17T17:18:59.412134+0000","last_became_peered":"2024-08-17T17:18:59.412134+0000","last_unstale":"2024-08-17T17:25:40.111512+0000","last_undegraded":"2024-08-17T17:25:40.111512+0000","last_fullsized":"2024-08-17T17:25:40.111512+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:44.463153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":298,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.109404+0000","last_change":"2024-08-17T17:25:04.510854+0000","last_active":"2024-08-17T17:25:40.109404+0000","last_peered":"2024-08-17T17:25:40.109404+0000","last_clean":"2024-08-17T17:25:40.109404+0000","last_became_active":"2024-08-17T17:25:04.510462+0000","last_became_peered":"2024-08-17T17:25:04.510462+0000","last_unstale":"2024-08-17T17:25:40.109404+0000","last_undegraded":"2024-08-17T17:25:40.109404+0000","last_fullsized":"2024-08-17T17:25:40.109404+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:55:54.762086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":195,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.111671+0000","last_change":"2024-08-17T17:24:06.591165+0000","last_active":"2024-08-17T17:25:40.111671+0000","last_peered":"2024-08-17T17:25:40.111671+0000","last_clean":"2024-08-17T17:25:40.111671+0000","last_became_active":"2024-08-17T17:24:06.590785+0000","last_became_peered":"2024-08-17T17:24:06.590785+0000","last_unstale":"2024-08-17T17:25:40.111671+0000","last_undegraded":"2024-08-17T17:25:40.111671+0000","last_fullsized":"2024-08-17T17:25:40.111671+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:54:05.775563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":267,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.112160+0000","last_change":"2024-08-17T17:24:32.300560+0000","last_active":"2024-08-17T17:25:40.112160+0000","last_peered":"2024-08-17T17:25:40.112160+0000","last_clean":"2024-08-17T17:25:40.112160+0000","last_became_active":"2024-08-17T17:24:32.300228+0000","last_became_peered":"2024-08-17T17:24:32.300228+0000","last_unstale":"2024-08-17T17:25:40.112160+0000","last_undegraded":"2024-08-17T17:25:40.112160+0000","last_fullsized":"2024-08-17T17:25:40.112160+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:18:05.475449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":74,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.766105+0000","last_change":"2024-08-17T17:24:06.586379+0000","last_active":"2024-08-17T17:25:40.766105+0000","last_peered":"2024-08-17T17:25:40.766105+0000","last_clean":"2024-08-17T17:25:40.766105+0000","last_became_active":"2024-08-17T17:24:06.586023+0000","last_became_peered":"2024-08-17T17:24:06.586023+0000","last_unstale":"2024-08-17T17:25:40.766105+0000","last_undegraded":"2024-08-17T17:25:40.766105+0000","last_fullsized":"2024-08-17T17:25:40.766105+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:45.701637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":284,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.109262+0000","last_change":"2024-08-17T17:21:50.442535+0000","last_active":"2024-08-17T17:25:40.109262+0000","last_peered":"2024-08-17T17:25:40.109262+0000","last_clean":"2024-08-17T17:25:40.109262+0000","last_became_active":"2024-08-17T17:21:50.441891+0000","last_became_peered":"2024-08-17T17:21:50.441891+0000","last_unstale":"2024-08-17T17:25:40.109262+0000","last_undegraded":"2024-08-17T17:25:40.109262+0000","last_fullsized":"2024-08-17T17:25:40.109262+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:46:24.025407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":217,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-08-17T17:25:39.113145+0000","last_change":"2024-08-17T17:25:04.508045+0000","last_active":"2024-08-17T17:25:39.113145+0000","last_peered":"2024-08-17T17:25:39.113145+0000","last_clean":"2024-08-17T17:25:39.113145+0000","last_became_active":"2024-08-17T17:25:04.505083+0000","last_became_peered":"2024-08-17T17:25:04.505083+0000","last_unstale":"2024-08-17T17:25:39.113145+0000","last_undegraded":"2024-08-17T17:25:39.113145+0000","last_fullsized":"2024-08-17T17:25:39.113145+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:23:42.993544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":270,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.109276+0000","last_change":"2024-08-17T17:18:59.418434+0000","last_active":"2024-08-17T17:25:40.109276+0000","last_peered":"2024-08-17T17:25:40.109276+0000","last_clean":"2024-08-17T17:25:40.109276+0000","last_became_active":"2024-08-17T17:18:59.418043+0000","last_became_peered":"2024-08-17T17:18:59.418043+0000","last_unstale":"2024-08-17T17:25:40.109276+0000","last_undegraded":"2024-08-17T17:25:40.109276+0000","last_fullsized":"2024-08-17T17:25:40.109276+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:25:51.371820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":190,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-08-17T17:25:39.113719+0000","last_change":"2024-08-17T17:20:49.291167+0000","last_active":"2024-08-17T17:25:39.113719+0000","last_peered":"2024-08-17T17:25:39.113719+0000","last_clean":"2024-08-17T17:25:39.113719+0000","last_became_active":"2024-08-17T17:20:49.263750+0000","last_became_peered":"2024-08-17T17:20:49.263750+0000","last_unstale":"2024-08-17T17:25:39.113719+0000","last_undegraded":"2024-08-17T17:25:39.113719+0000","last_fullsized":"2024-08-17T17:25:39.113719+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:02.863246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":284,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.109173+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:25:40.109173+0000","last_peered":"2024-08-17T17:25:40.109173+0000","last_clean":"2024-08-17T17:25:40.109173+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:25:40.109173+0000","last_undegraded":"2024-08-17T17:25:40.109173+0000","last_fullsized":"2024-08-17T17:25:40.109173+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":284,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.109857+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:25:40.109857+0000","last_peered":"2024-08-17T17:25:40.109857+0000","last_clean":"2024-08-17T17:25:40.109857+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:25:40.109857+0000","last_undegraded":"2024-08-17T17:25:40.109857+0000","last_fullsized":"2024-08-17T17:25:40.109857+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-08-17T17:25:39.115552+0000","last_change":"2024-08-17T17:25:04.503834+0000","last_active":"2024-08-17T17:25:39.115552+0000","last_peered":"2024-08-17T17:25:39.115552+0000","last_clean":"2024-08-17T17:25:39.115552+0000","last_became_active":"2024-08-17T17:25:04.502712+0000","last_became_peered":"2024-08-17T17:25:04.502712+0000","last_unstale":"2024-08-17T17:25:39.115552+0000","last_undegraded":"2024-08-17T17:25:39.115552+0000","last_fullsized":"2024-08-17T17:25:39.115552+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:45.389103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":93,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-08-17T17:25:40.766308+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:25:40.766308+0000","last_peered":"2024-08-17T17:25:40.766308+0000","last_clean":"2024-08-17T17:25:40.766308+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:25:40.766308+0000","last_undegraded":"2024-08-17T17:25:40.766308+0000","last_fullsized":"2024-08-17T17:25:40.766308+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":5797249024,"num_objects":88459,"num_object_clones":0,"num_object_copies":265377,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16927,"num_objects_misplaced":1306,"num_objects_unfound":0,"num_objects_dirty":88459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":88459,"num_write_kb":5661376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4900,"num_bytes_recovered":321126400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17655463936,"data_stored":17655463936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88459,"ondisk_log_size":88459,"up":48,"acting":42,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":6304,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":16702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17971,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":4194342,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234228,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3016704,"kb_used_data":2711032,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":90728448,"statfs":{"total":95995035648,"available":92905930752,"internally_reserved":0,"allocated":2776096768,"data_stored":2774927335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495863,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2455096,"kb_used_data":2149424,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91290056,"statfs":{"total":95995035648,"available":93481017344,"internally_reserved":0,"allocated":2201010176,"data_stored":2199848635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757500,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2045356,"kb_used_data":1748004,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":91699796,"statfs":{"total":95995035648,"available":93900591104,"internally_reserved":0,"allocated":1789956096,"data_stored":1788802298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019138,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435365,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3731112,"kb_used_data":3506400,"kb_used_omap":4,"kb_used_meta":224699,"kb_avail":90014040,"statfs":{"total":95995035648,"available":92174376960,"internally_reserved":0,"allocated":3590553600,"data_stored":3589403672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":230091786},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":104,"seq":446676598892,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3516344,"kb_used_data":3233776,"kb_used_omap":7,"kb_used_meta":282552,"kb_avail":90228808,"statfs":{"total":95995035648,"available":92394299392,"internally_reserved":0,"allocated":3311386624,"data_stored":3310225578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":289333410},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762418,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2735340,"kb_used_data":2405156,"kb_used_omap":7,"kb_used_meta":330168,"kb_avail":91009812,"statfs":{"total":95995035648,"available":93194047488,"internally_reserved":0,"allocated":2462879744,"data_stored":2461725687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7945,"internal_metadata":338092279},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":94,"seq":403726925945,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1790900,"kb_used_data":1516716,"kb_used_omap":7,"kb_used_meta":274168,"kb_avail":91954252,"statfs":{"total":95995035648,"available":94161154048,"internally_reserved":0,"allocated":1553117184,"data_stored":1551956120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":280748353},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6241,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1445199872,"data_stored":1445199872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2472280064,"data_stored":2472280064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3330736128,"data_stored":3330736128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3596484608,"data_stored":3596484608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1795293184,"data_stored":1795293184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2228092928,"data_stored":2228092928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2787377152,"data_stored":2787377152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:25:52.046 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph status --format=json 2024-08-17T17:25:52.080 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:52.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:52.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:52.095 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 83 16 92687 92671 69.7492 53.3125 0.0120623 0.0143302 2024-08-17T17:25:52.182 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:52.211 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:52.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:52.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:52.312 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:52.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:52.364 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:52.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:52.465 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:52.495 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:52.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:52.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:52.597 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:52.603 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:25:52.620 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:25:52.649 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:25:52.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:52.680 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:52.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:52.781 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:52.833 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:52.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:52.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:52.935 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:52.981 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:52.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:52.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:53.083 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:53.095 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 84 16 93510 93494 69.5312 51.4375 0.0250423 0.0143757 2024-08-17T17:25:53.118 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:53.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:53.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:53.220 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:53.253 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:53.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:53.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:53.355 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:53.391 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:53.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:53.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:53.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:53 smithi053 ceph-mon[89963]: pgmap v516: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 52 MiB/s wr, 828 op/s; 16932/272154 objects degraded (6.221%); 1054/272154 objects misplaced (0.387%); 5.0 MiB/s, 79 objects/s recovering 2024-08-17T17:25:53.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:25:53.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:25:53.493 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:53.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:53 smithi003 ceph-mon[103344]: pgmap v516: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 52 MiB/s wr, 828 op/s; 16932/272154 objects degraded (6.221%); 1054/272154 objects misplaced (0.387%); 5.0 MiB/s, 79 objects/s recovering 2024-08-17T17:25:53.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:25:53.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:25:53.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:53 smithi003 ceph-mon[100615]: pgmap v516: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 52 MiB/s wr, 828 op/s; 16932/272154 objects degraded (6.221%); 1054/272154 objects misplaced (0.387%); 5.0 MiB/s, 79 objects/s recovering 2024-08-17T17:25:53.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:25:53.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:25:53.539 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:53.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:53.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:53.640 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:53.675 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:53.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:53.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:53.777 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:53.806 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:53.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:53.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:53.908 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:53.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:53.946 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:53.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:54.046 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:54.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:54.082 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:54.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:54.097 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 85 16 94449 94433 69.4034 58.6875 0.0132366 0.0144023 2024-08-17T17:25:54.183 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:54.235 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:54.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:54.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:54.337 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:54.385 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:54.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:54.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:54.474 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:25:54.487 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:54.518 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:54.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:54.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:54.621 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:54.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:54.648 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:54.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:54.749 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:54.777 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:54.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:54.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:54.879 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:54.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:54.932 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:54.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:55.033 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:55.076 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:55.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:55.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:55.097 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 86 16 95343 95327 69.2446 55.875 0.0116235 0.0144349 2024-08-17T17:25:55.178 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:55.211 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:55.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:55.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:55.313 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:55.351 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:55 smithi003 ceph-mon[103344]: pgmap v517: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 52 MiB/s wr, 839 op/s; 16932/275928 objects degraded (6.136%); 1054/275928 objects misplaced (0.382%); 3.1 MiB/s, 49 objects/s recovering 2024-08-17T17:25:55.351 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:55 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 16932/275928 objects degraded (6.136%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:55.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:55 smithi003 ceph-mon[100615]: pgmap v517: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 52 MiB/s wr, 839 op/s; 16932/275928 objects degraded (6.136%); 1054/275928 objects misplaced (0.382%); 3.1 MiB/s, 49 objects/s recovering 2024-08-17T17:25:55.351 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:55 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 16932/275928 objects degraded (6.136%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:55.356 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:55.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:55.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:55.458 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:55.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:55 smithi053 ceph-mon[89963]: pgmap v517: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 52 MiB/s wr, 839 op/s; 16932/275928 objects degraded (6.136%); 1054/275928 objects misplaced (0.382%); 3.1 MiB/s, 49 objects/s recovering 2024-08-17T17:25:55.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:55 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 16932/275928 objects degraded (6.136%), 6 pgs degraded (PG_DEGRADED) 2024-08-17T17:25:55.486 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:55.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:55.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:55.588 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:55.616 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:55.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:55.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:55.718 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:55.749 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:55.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:55.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:55.851 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:55.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:55.892 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:55.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:55.907 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:25:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:25:55] "GET /metrics HTTP/1.1" 200 36525 "" "Prometheus/2.51.0" 2024-08-17T17:25:55.993 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:56.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:56.029 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:56.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:56.097 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 87 16 96248 96232 69.0988 56.5625 0.0132943 0.0144658 2024-08-17T17:25:56.130 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:56.159 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:56.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:56.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:56.260 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:56.293 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:56.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:56.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:56.395 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:56.422 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:56.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:56.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:56.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:56.566 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:56.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:56.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:56.668 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:56.693 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:56.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:56.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:56.722 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:25:56.722 INFO:teuthology.orchestra.run.smithi003.stdout:{"fsid":"2b5b662e-5cba-11ef-bccf-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 16932/281079 objects degraded (6.024%), 6 pgs degraded, 2 pgs undersized","count":8},"muted":false}},"mutes":[]},"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":698,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":206,"num_osds":8,"num_up_osds":8,"osd_up_since":1723915051,"num_in_osds":7,"osd_in_since":1723915248,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":82},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+recovering+undersized+remapped","count":1},{"state_name":"active+recovery_wait+degraded","count":1}],"num_pgs":89,"num_pools":5,"num_objects":93693,"data_bytes":6143219661,"bytes_used":20731682816,"bytes_avail":651233566720,"bytes_total":671965249536,"degraded_objects":16932,"degraded_total":281079,"degraded_ratio":0.060239292156297697,"misplaced_objects":1054,"misplaced_total":281079,"misplaced_ratio":0.003749835455512507,"recovering_objects_per_sec":49,"recovering_bytes_per_sec":3235525,"recovering_keys_per_sec":0,"num_objects_recovered":494,"num_bytes_recovered":32374784,"num_keys_recovered":0,"write_bytes_sec":57099812,"read_op_per_sec":0,"write_op_per_sec":871},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi003.civgvf","status":"up:active","gid":14664}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.3:8443/","prometheus":"http://172.21.15.3:9283/"}},"servicemap":{"epoch":23,"modified":"2024-08-17T17:18:31.688676+0000","services":{}},"progress_events":{"714d66b4-625a-4d60-aa1c-f38ff47ebe12":{"message":"Global Recovery Event (25s)\n [=========================...] (remaining: 2s)","progress":0.91011238098144531,"add_to_ceph_s":true},"fcbeb9b7-7baf-488b-9fe4-c01aca441f9c":{"message":"Upgrade to 19.0.0-5836-gcd8bdbeb (4m)\n [============................] (remaining: 5m)","progress":0.44999998807907104,"add_to_ceph_s":true}}} 2024-08-17T17:25:56.795 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:56.823 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:56.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:56.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:56.924 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:56.955 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:56.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:56.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:57.057 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:57.090 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:57.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:57.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:57.097 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 88 16 97072 97056 68.8988 51.5 0.0482198 0.0145075 2024-08-17T17:25:57.192 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:57.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:57.221 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:57.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:57.322 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:57.355 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:57.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:57.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:57.456 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:57.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:57 smithi053 ceph-mon[89963]: pgmap v518: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 54 MiB/s wr, 871 op/s; 16932/281079 objects degraded (6.024%); 1054/281079 objects misplaced (0.375%); 3.1 MiB/s, 49 objects/s recovering 2024-08-17T17:25:57.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:57 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3562865742' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T17:25:57.468 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:57 smithi003 ceph-mon[100615]: pgmap v518: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 54 MiB/s wr, 871 op/s; 16932/281079 objects degraded (6.024%); 1054/281079 objects misplaced (0.375%); 3.1 MiB/s, 49 objects/s recovering 2024-08-17T17:25:57.468 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:57 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3562865742' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T17:25:57.469 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:57 smithi003 ceph-mon[103344]: pgmap v518: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 54 MiB/s wr, 871 op/s; 16932/281079 objects degraded (6.024%); 1054/281079 objects misplaced (0.375%); 3.1 MiB/s, 49 objects/s recovering 2024-08-17T17:25:57.469 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:57 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3562865742' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T17:25:57.516 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:57.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:57.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:57.618 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:57.661 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:57.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:57.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:57.763 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:25:57.794 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:57.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:57.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:25:57.896 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:57.950 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:57.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:57.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:58.052 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:25:58.090 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:58.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:58.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:25:58.097 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 89 16 97926 97910 68.7244 53.375 0.0190715 0.0145453 2024-08-17T17:25:58.193 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:58.222 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:58.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:58.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:58.258 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:25:58.258 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:25:58.301 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:25:58.302 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:25:58.323 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:25:58.343 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:25:58.344 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:25:58.373 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:58.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:58.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:25:58.382 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:25:58.382 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:25:58.414 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:25:58.415 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:25:58.449 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:25:58.449 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:25:58.475 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:58.476 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:25:58.476 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:25:58.491 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:25:58.491 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:25:58.497 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:58.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:58.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:58.599 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:58.633 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:58.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:58.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:58.735 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:25:58.764 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:58.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:58.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:25:58.866 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:58.898 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:58.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:58.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:58.999 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:59.034 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:59.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:59.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:59.098 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total time run: 90.0148 2024-08-17T17:25:59.098 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total writes made: 98646 2024-08-17T17:25:59.098 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Write size: 65536 2024-08-17T17:25:59.098 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object size: 65536 2024-08-17T17:25:59.098 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Bandwidth (MB/sec): 68.4929 2024-08-17T17:25:59.098 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Bandwidth: 34.6888 2024-08-17T17:25:59.098 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max bandwidth (MB/sec): 182 2024-08-17T17:25:59.098 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min bandwidth (MB/sec): 0 2024-08-17T17:25:59.098 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average IOPS: 1095 2024-08-17T17:25:59.098 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev IOPS: 555.029 2024-08-17T17:25:59.099 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max IOPS: 2912 2024-08-17T17:25:59.099 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min IOPS: 0 2024-08-17T17:25:59.099 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average Latency(s): 0.0145948 2024-08-17T17:25:59.099 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Latency(s): 0.0544194 2024-08-17T17:25:59.099 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max latency(s): 2.58819 2024-08-17T17:25:59.099 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min latency(s): 0.00112466 2024-08-17T17:25:59.102 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Cleaning up (deleting benchmark objects) 2024-08-17T17:25:59.135 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:59.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:59.198 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:59.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:59.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:59 smithi003 ceph-mon[103344]: pgmap v519: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 55 MiB/s wr, 876 op/s; 16932/285345 objects degraded (5.934%); 776/285345 objects misplaced (0.272%); 3.3 MiB/s, 52 objects/s recovering 2024-08-17T17:25:59.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:25:59 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2221442175' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:25:59.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:59 smithi003 ceph-mon[100615]: pgmap v519: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 55 MiB/s wr, 876 op/s; 16932/285345 objects degraded (5.934%); 776/285345 objects misplaced (0.272%); 3.3 MiB/s, 52 objects/s recovering 2024-08-17T17:25:59.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:25:59 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2221442175' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:25:59.300 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:25:59.321 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:59.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:59.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:25:59.423 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:59.450 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:59.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:59.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:25:59.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:59 smithi053 ceph-mon[89963]: pgmap v519: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 55 MiB/s wr, 876 op/s; 16932/285345 objects degraded (5.934%); 776/285345 objects misplaced (0.272%); 3.3 MiB/s, 52 objects/s recovering 2024-08-17T17:25:59.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:25:59 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2221442175' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:25:59.552 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:25:59.582 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:59.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:59.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:25:59.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:59.721 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:59.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:59.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:59.823 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:25:59.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:59.865 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:25:59.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:25:59.966 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:25:59.993 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:25:59.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:25:59.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:00.095 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:00.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:00.136 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:00.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:00.237 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:00.274 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:00.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:00.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:00.376 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:00.394 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:26:00.394 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:26:00.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.394 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45491.0:98749 6.7 e206/178 6:e20619a3:::benchmark_data_smithi003_175043_object100:head v 206'6358, mlcod=206'6355)", 2024-08-17T17:26:00.394 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:25:59.239319+0000", 2024-08-17T17:26:00.394 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.047986893000000003, 2024-08-17T17:26:00.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.048014868000000002, 2024-08-17T17:26:00.394 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:26:00.394 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:26:00.394 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:26:00.395 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:26:00.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:26:00.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.239319+0000", 2024-08-17T17:26:00.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:26:00.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:26:00.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.239319+0000", 2024-08-17T17:26:00.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:26:00.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:26:00.396 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.239319+0000", 2024-08-17T17:26:00.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.08e-07 2024-08-17T17:26:00.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.396 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:26:00.396 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.239324+0000", 2024-08-17T17:26:00.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0950000000000003e-06 2024-08-17T17:26:00.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:26:00.397 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.239332+0000", 2024-08-17T17:26:00.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3900000000000004e-06 2024-08-17T17:26:00.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:26:00.397 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.239337+0000", 2024-08-17T17:26:00.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3650000000000003e-06 2024-08-17T17:26:00.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:26:00.397 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.246593+0000", 2024-08-17T17:26:00.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072563219999999999 2024-08-17T17:26:00.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.398 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:26:00.398 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.246613+0000", 2024-08-17T17:26:00.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9927000000000001e-05 2024-08-17T17:26:00.398 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:26:00.398 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:26:00.398 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:26:00.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.399 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45491.0:98752 6.7 e206/178 6:e7629068:::benchmark_data_smithi003_175043_object103:head v 206'6359, mlcod=206'6355)", 2024-08-17T17:26:00.399 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:25:59.242103+0000", 2024-08-17T17:26:00.399 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.045202702999999997, 2024-08-17T17:26:00.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.045266913999999998, 2024-08-17T17:26:00.399 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:26:00.399 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:26:00.399 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:26:00.399 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:26:00.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:26:00.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.242103+0000", 2024-08-17T17:26:00.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:26:00.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:26:00.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.242103+0000", 2024-08-17T17:26:00.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:26:00.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:26:00.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.242104+0000", 2024-08-17T17:26:00.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1880000000000001e-06 2024-08-17T17:26:00.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:26:00.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.242119+0000", 2024-08-17T17:26:00.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4523e-05 2024-08-17T17:26:00.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:26:00.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.242133+0000", 2024-08-17T17:26:00.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4595e-05 2024-08-17T17:26:00.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:26:00.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.242142+0000", 2024-08-17T17:26:00.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5019999999999992e-06 2024-08-17T17:26:00.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:26:00.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.249899+0000", 2024-08-17T17:26:00.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0077575939999999996 2024-08-17T17:26:00.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:26:00.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.249918+0000", 2024-08-17T17:26:00.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8355e-05 2024-08-17T17:26:00.402 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:26:00.402 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:26:00.402 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:26:00.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.402 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45491.0:98755 6.4 6:27bc67cf:::benchmark_data_smithi003_175043_object106:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-08-17T17:26:00.402 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:25:59.243927+0000", 2024-08-17T17:26:00.403 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.043379256999999997, 2024-08-17T17:26:00.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.043475706000000003, 2024-08-17T17:26:00.403 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:26:00.403 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:26:00.403 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:26:00.403 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:26:00.403 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45491", 2024-08-17T17:26:00.403 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:26:00.403 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 98755 2024-08-17T17:26:00.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.403 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:26:00.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:26:00.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.243927+0000", 2024-08-17T17:26:00.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:26:00.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:26:00.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.243927+0000", 2024-08-17T17:26:00.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:26:00.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:26:00.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.243928+0000", 2024-08-17T17:26:00.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1930000000000001e-06 2024-08-17T17:26:00.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:26:00.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.243932+0000", 2024-08-17T17:26:00.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4410000000000003e-06 2024-08-17T17:26:00.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:26:00.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.243945+0000", 2024-08-17T17:26:00.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2917e-05 2024-08-17T17:26:00.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:26:00.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.243953+0000", 2024-08-17T17:26:00.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4209999999999995e-06 2024-08-17T17:26:00.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:26:00.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.244031+0000", 2024-08-17T17:26:00.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7960000000000006e-05 2024-08-17T17:26:00.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:26:00.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.244146+0000", 2024-08-17T17:26:00.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011473599999999999 2024-08-17T17:26:00.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,5", 2024-08-17T17:26:00.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.244240+0000", 2024-08-17T17:26:00.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3790999999999997e-05 2024-08-17T17:26:00.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:26:00.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.247640+0000", 2024-08-17T17:26:00.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034004040000000001 2024-08-17T17:26:00.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:26:00.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.250978+0000", 2024-08-17T17:26:00.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0033374949999999998 2024-08-17T17:26:00.407 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:26:00.407 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:26:00.407 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:26:00.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.408 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45491.0:98758 6.e 6:73bfa864:::benchmark_data_smithi003_175043_object109:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-08-17T17:26:00.408 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:25:59.243975+0000", 2024-08-17T17:26:00.408 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.043330585999999997, 2024-08-17T17:26:00.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.043468711, 2024-08-17T17:26:00.408 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:26:00.408 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:26:00.408 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:26:00.408 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:26:00.408 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45491", 2024-08-17T17:26:00.408 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:26:00.408 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 98758 2024-08-17T17:26:00.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.409 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:26:00.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:26:00.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.243975+0000", 2024-08-17T17:26:00.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:26:00.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:26:00.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.243975+0000", 2024-08-17T17:26:00.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:26:00.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:26:00.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.243976+0000", 2024-08-17T17:26:00.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3099999999999997e-07 2024-08-17T17:26:00.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:26:00.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.243979+0000", 2024-08-17T17:26:00.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8009999999999999e-06 2024-08-17T17:26:00.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:26:00.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.243987+0000", 2024-08-17T17:26:00.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8310000000000007e-06 2024-08-17T17:26:00.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:26:00.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.243994+0000", 2024-08-17T17:26:00.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8639999999999998e-06 2024-08-17T17:26:00.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:26:00.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.244056+0000", 2024-08-17T17:26:00.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1329999999999997e-05 2024-08-17T17:26:00.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:26:00.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.244215+0000", 2024-08-17T17:26:00.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015910800000000001 2024-08-17T17:26:00.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:26:00.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.244378+0000", 2024-08-17T17:26:00.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016337900000000001 2024-08-17T17:26:00.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:26:00.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.250368+0000", 2024-08-17T17:26:00.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0059900200000000004 2024-08-17T17:26:00.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:26:00.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.251227+0000", 2024-08-17T17:26:00.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00085901300000000005 2024-08-17T17:26:00.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:26:00.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.251491+0000", 2024-08-17T17:26:00.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00026357600000000001 2024-08-17T17:26:00.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:26:00.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.251557+0000", 2024-08-17T17:26:00.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6202000000000001e-05 2024-08-17T17:26:00.413 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:26:00.413 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:26:00.413 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:26:00.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.413 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45491.0:98759 6.4 6:2277c227:::benchmark_data_smithi003_175043_object110:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-08-17T17:26:00.414 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:25:59.250460+0000", 2024-08-17T17:26:00.414 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.036845968999999999, 2024-08-17T17:26:00.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.037028833999999997, 2024-08-17T17:26:00.414 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:26:00.414 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:26:00.414 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:26:00.414 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:26:00.414 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45491", 2024-08-17T17:26:00.414 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:26:00.414 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 98759 2024-08-17T17:26:00.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.414 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:26:00.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:26:00.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.250460+0000", 2024-08-17T17:26:00.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:26:00.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:26:00.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.250460+0000", 2024-08-17T17:26:00.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:26:00.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:26:00.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.250462+0000", 2024-08-17T17:26:00.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7460000000000001e-06 2024-08-17T17:26:00.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.416 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:26:00.416 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.250467+0000", 2024-08-17T17:26:00.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9840000000000001e-06 2024-08-17T17:26:00.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.416 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:26:00.416 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.250480+0000", 2024-08-17T17:26:00.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3487e-05 2024-08-17T17:26:00.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.416 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:26:00.416 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.250489+0000", 2024-08-17T17:26:00.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7530000000000007e-06 2024-08-17T17:26:00.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.417 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:26:00.417 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.251091+0000", 2024-08-17T17:26:00.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00060220300000000005 2024-08-17T17:26:00.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.417 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:26:00.417 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.251222+0000", 2024-08-17T17:26:00.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000131216 2024-08-17T17:26:00.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,5", 2024-08-17T17:26:00.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.251331+0000", 2024-08-17T17:26:00.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000108923 2024-08-17T17:26:00.418 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:26:00.418 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:26:00.418 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:26:00.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.418 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.45491.0:98758 6.e e206/194 ondisk, result = 0)", 2024-08-17T17:26:00.418 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:25:59.251410+0000", 2024-08-17T17:26:00.418 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.035896142999999998, 2024-08-17T17:26:00.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.036110221999999997, 2024-08-17T17:26:00.418 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:26:00.419 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:26:00.419 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:26:00.419 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:26:00.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:26:00.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.251410+0000", 2024-08-17T17:26:00.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:26:00.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:26:00.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.251410+0000", 2024-08-17T17:26:00.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:26:00.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:26:00.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.251410+0000", 2024-08-17T17:26:00.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7100000000000002e-07 2024-08-17T17:26:00.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:26:00.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.251414+0000", 2024-08-17T17:26:00.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.664e-06 2024-08-17T17:26:00.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:26:00.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.251422+0000", 2024-08-17T17:26:00.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9489999999999996e-06 2024-08-17T17:26:00.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.421 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:26:00.421 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.251428+0000", 2024-08-17T17:26:00.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1800000000000001e-06 2024-08-17T17:26:00.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.421 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:26:00.421 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.251476+0000", 2024-08-17T17:26:00.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7630000000000003e-05 2024-08-17T17:26:00.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:26:00.422 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.251484+0000", 2024-08-17T17:26:00.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6459999999999999e-06 2024-08-17T17:26:00.422 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:26:00.422 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:26:00.422 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:26:00.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.422 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45491.0:98760 6.1 6:86d1671f:::benchmark_data_smithi003_175043_object111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-08-17T17:26:00.422 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:25:59.251767+0000", 2024-08-17T17:26:00.422 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.035538817, 2024-08-17T17:26:00.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.035779852000000001, 2024-08-17T17:26:00.422 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:26:00.423 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:26:00.423 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:26:00.423 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:26:00.423 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45491", 2024-08-17T17:26:00.423 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:26:00.423 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 98760 2024-08-17T17:26:00.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.423 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:26:00.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.423 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:26:00.423 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.251767+0000", 2024-08-17T17:26:00.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:26:00.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.424 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:26:00.424 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.251767+0000", 2024-08-17T17:26:00.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:26:00.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.424 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:26:00.424 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.251768+0000", 2024-08-17T17:26:00.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.054e-06 2024-08-17T17:26:00.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.424 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:26:00.424 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.251772+0000", 2024-08-17T17:26:00.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9210000000000002e-06 2024-08-17T17:26:00.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:26:00.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.251782+0000", 2024-08-17T17:26:00.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0057e-05 2024-08-17T17:26:00.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:26:00.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.251789+0000", 2024-08-17T17:26:00.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6669999999999999e-06 2024-08-17T17:26:00.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.426 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:26:00.426 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.251862+0000", 2024-08-17T17:26:00.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3292000000000002e-05 2024-08-17T17:26:00.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.426 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.426 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:26:00.426 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.251963+0000", 2024-08-17T17:26:00.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000100958 2024-08-17T17:26:00.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:00.426 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:00.426 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,6", 2024-08-17T17:26:00.426 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:59.252055+0000", 2024-08-17T17:26:00.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1742000000000004e-05 2024-08-17T17:26:00.427 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:26:00.427 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:26:00.427 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:26:00.427 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:26:00.427 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-08-17T17:26:00.427 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 7 2024-08-17T17:26:00.427 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:26:00.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:00.428 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:00.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:00.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:26:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:26:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:26:00.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:00 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 16932/285345 objects degraded (5.934%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:00.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:00 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 16932/285345 objects degraded (5.934%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:00.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:00 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 16932/285345 objects degraded (5.934%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:00.529 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:00.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:00.561 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:00.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:00.662 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:00.696 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:00.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:00.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:00.798 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:00.836 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:00.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:00.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:00.938 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:01.024 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:01.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:01.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:01.126 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:01.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:01.159 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:01.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:01.260 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:01.298 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:01.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:01.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:01.400 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:01.429 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:01.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:01.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:01.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:01 smithi053 ceph-mon[89963]: pgmap v520: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 54 MiB/s wr, 870 op/s; 16932/291609 objects degraded (5.806%); 776/291609 objects misplaced (0.266%); 3.3 MiB/s, 52 objects/s recovering 2024-08-17T17:26:01.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:01 smithi003 ceph-mon[103344]: pgmap v520: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 54 MiB/s wr, 870 op/s; 16932/291609 objects degraded (5.806%); 776/291609 objects misplaced (0.266%); 3.3 MiB/s, 52 objects/s recovering 2024-08-17T17:26:01.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:01 smithi003 ceph-mon[100615]: pgmap v520: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 54 MiB/s wr, 870 op/s; 16932/291609 objects degraded (5.806%); 776/291609 objects misplaced (0.266%); 3.3 MiB/s, 52 objects/s recovering 2024-08-17T17:26:01.531 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:01.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:01.573 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:01.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:01.674 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:01.718 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:01.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:01.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:01.820 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:01.861 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:01.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:01.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:01.962 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:01.996 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:01.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:01.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:02.098 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:02.121 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:02.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:02.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:02.223 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:02.270 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:02.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:02.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:02.372 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:02.394 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:02.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:02.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:02.495 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:02.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:02.530 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:02.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:02.631 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:02.669 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:02.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:02.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:02.771 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:02.809 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:02.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:02.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:02.911 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:02.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:02.946 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:02.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:03.047 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:03.079 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:03.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:03.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:03.181 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:03.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:03.210 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:03.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:03.311 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:03.352 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:03.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:03.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:03.454 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:03.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:03 smithi053 ceph-mon[89963]: pgmap v521: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 53 MiB/s wr, 915 op/s; 16932/293157 objects degraded (5.776%); 459/293157 objects misplaced (0.157%); 4.5 MiB/s, 72 objects/s recovering 2024-08-17T17:26:03.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:03.490 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:03.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:03.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:03 smithi003 ceph-mon[100615]: pgmap v521: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 53 MiB/s wr, 915 op/s; 16932/293157 objects degraded (5.776%); 459/293157 objects misplaced (0.157%); 4.5 MiB/s, 72 objects/s recovering 2024-08-17T17:26:03.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:03 smithi003 ceph-mon[103344]: pgmap v521: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 53 MiB/s wr, 915 op/s; 16932/293157 objects degraded (5.776%); 459/293157 objects misplaced (0.157%); 4.5 MiB/s, 72 objects/s recovering 2024-08-17T17:26:03.592 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:03.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:03.625 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:03.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:03.726 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:03.766 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:03.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:03.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:03.870 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:03.901 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:03.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:03.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:04.003 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:04.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:04.036 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:04.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:04.137 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:04.171 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:04.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:04.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:04.273 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:04.316 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:04.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:04.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:04.418 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:04.451 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:04.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:04.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:04.553 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:04.590 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:04.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:04.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:04.692 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:04.735 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:04.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:04.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:04.837 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:04.869 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:04.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:04.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:04.971 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:05.010 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:05.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:05.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:05.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:05.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:05.155 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:05.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:05.256 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:05.295 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:05.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:05.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:05.397 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:05.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:05 smithi003 ceph-mon[100615]: pgmap v522: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 85 B/s rd, 41 MiB/s wr, 813 op/s; 16932/290523 objects degraded (5.828%); 459/290523 objects misplaced (0.158%); 3.1 MiB/s, 49 objects/s recovering 2024-08-17T17:26:05.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:05 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 16932/290523 objects degraded (5.828%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:05.451 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:05 smithi003 ceph-mon[103344]: pgmap v522: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 85 B/s rd, 41 MiB/s wr, 813 op/s; 16932/290523 objects degraded (5.828%); 459/290523 objects misplaced (0.158%); 3.1 MiB/s, 49 objects/s recovering 2024-08-17T17:26:05.451 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:05 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 16932/290523 objects degraded (5.828%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:05.452 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-08-17T17:26:05.452 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:26:05.456 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:05.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:05.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:05.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:05 smithi053 ceph-mon[89963]: pgmap v522: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 85 B/s rd, 41 MiB/s wr, 813 op/s; 16932/290523 objects degraded (5.828%); 459/290523 objects misplaced (0.158%); 3.1 MiB/s, 49 objects/s recovering 2024-08-17T17:26:05.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:05 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 16932/290523 objects degraded (5.828%), 6 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:05.558 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:05.587 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:05.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:05.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:05.689 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:05.736 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:05.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:05.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:05.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:26:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:26:05] "GET /metrics HTTP/1.1" 200 36525 "" "Prometheus/2.51.0" 2024-08-17T17:26:05.838 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:05.869 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:05.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:05.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:05.971 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:06.009 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:06.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:06.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:06.110 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:06.134 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-17T17:26:06.159 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:06.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:06.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:06.261 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:06.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:06.318 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:06.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:06.354 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:26:06.419 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:06.488 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:06.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:06.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:06.590 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:06.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:06.639 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:06.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:06.740 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:06.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:06.784 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:06.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:06.885 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:06.917 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:06.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:06.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:07.018 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:07.047 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:07.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:07.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:07.149 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:07.188 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:07.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:07.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:07.290 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:07.326 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:07.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:07.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:07.428 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:07.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:07 smithi003 ceph-mon[100615]: pgmap v523: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 85 B/s rd, 35 MiB/s wr, 842 op/s; 16932/285771 objects degraded (5.925%); 459/285771 objects misplaced (0.161%); 3.1 MiB/s, 49 objects/s recovering 2024-08-17T17:26:07.451 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:07 smithi003 ceph-mon[103344]: pgmap v523: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 85 B/s rd, 35 MiB/s wr, 842 op/s; 16932/285771 objects degraded (5.925%); 459/285771 objects misplaced (0.161%); 3.1 MiB/s, 49 objects/s recovering 2024-08-17T17:26:07.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:07 smithi053 ceph-mon[89963]: pgmap v523: 89 pgs: 1 active+recovery_wait+degraded, 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 85 B/s rd, 35 MiB/s wr, 842 op/s; 16932/285771 objects degraded (5.925%); 459/285771 objects misplaced (0.161%); 3.1 MiB/s, 49 objects/s recovering 2024-08-17T17:26:07.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:07.468 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:07.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:07.537 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:26:07.569 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:07.597 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:07.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:07.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:07.698 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:07.723 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:07.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:07.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:07.825 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:07.848 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:07.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:07.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:07.949 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:07.972 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:07.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:07.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:08.039 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:26:08.074 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:08.098 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:08.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:08.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:08.200 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:08.224 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:08.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:08.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:08.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:08 smithi003 ceph-mon[103344]: osdmap e207: 8 total, 8 up, 7 in 2024-08-17T17:26:08.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:26:08.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:08 smithi003 ceph-mon[100615]: osdmap e207: 8 total, 8 up, 7 in 2024-08-17T17:26:08.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:26:08.326 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:08.359 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:08.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:08.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:08.461 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:08.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:08 smithi053 ceph-mon[89963]: osdmap e207: 8 total, 8 up, 7 in 2024-08-17T17:26:08.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:26:08.488 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:08.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:08.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:08.590 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:08.656 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:08.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:08.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:08.758 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:08.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:08.809 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:08.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:08.812 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:26:08.910 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:08.975 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:08.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:08.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:09.077 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:09.104 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:09.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:09.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:09.205 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:09.217 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:09 smithi003 ceph-mon[100615]: pgmap v525: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 102 B/s rd, 22 MiB/s wr, 826 op/s; 16760/281910 objects degraded (5.945%); 28/281910 objects misplaced (0.010%); 5.2 MiB/s, 83 objects/s recovering 2024-08-17T17:26:09.240 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:09.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:09.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:09.342 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:09.388 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:09.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:09.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:09.489 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:09.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:09 smithi003 ceph-mon[100615]: osdmap e208: 8 total, 8 up, 7 in 2024-08-17T17:26:09.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:09 smithi003 ceph-mon[103344]: pgmap v525: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 102 B/s rd, 22 MiB/s wr, 826 op/s; 16760/281910 objects degraded (5.945%); 28/281910 objects misplaced (0.010%); 5.2 MiB/s, 83 objects/s recovering 2024-08-17T17:26:09.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:09 smithi003 ceph-mon[103344]: osdmap e208: 8 total, 8 up, 7 in 2024-08-17T17:26:09.530 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:09.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:09.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:09.632 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:09.664 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:09.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:09.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:09.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:09 smithi053 ceph-mon[89963]: pgmap v525: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+degraded, 5 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 102 B/s rd, 22 MiB/s wr, 826 op/s; 16760/281910 objects degraded (5.945%); 28/281910 objects misplaced (0.010%); 5.2 MiB/s, 83 objects/s recovering 2024-08-17T17:26:09.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:09 smithi053 ceph-mon[89963]: osdmap e208: 8 total, 8 up, 7 in 2024-08-17T17:26:09.766 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:09.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:09.808 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:09.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:09.909 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:09.948 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:09.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:09.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:10.050 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:10.080 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:10.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:10.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:10.181 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:10.218 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:10.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:10.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:10.242 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-08-17T17:26:10.319 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:10.357 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:10.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:10.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:10.459 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:10.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:26:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:26:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:26:10.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:10 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/4203894546' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:26:10.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:10 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:26:10.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:10 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 16760/281910 objects degraded (5.945%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:10.492 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:10.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:10.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:10.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:10 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/4203894546' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:26:10.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:10 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:26:10.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:10 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 16760/281910 objects degraded (5.945%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:10.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:10 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/4203894546' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:26:10.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:10 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:26:10.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:10 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 16760/281910 objects degraded (5.945%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:10.594 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:10.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:10.633 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:10.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:10.734 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:10.771 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:10.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:10.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:10.873 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:10.904 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:10.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:10.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:11.006 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:11.057 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:11.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:11.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:11.159 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:11.191 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:11.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:11.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:11.293 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:11.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:11.330 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:11.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:11.431 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:11.462 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:11.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:11.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:11.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:11 smithi003 ceph-mon[103344]: pgmap v527: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 127 B/s rd, 1.6 MiB/s wr, 757 op/s; 14033/276186 objects degraded (5.081%); 2582/276186 objects misplaced (0.935%); 5.8 MiB/s, 86 objects/s recovering 2024-08-17T17:26:11.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:11 smithi003 ceph-mon[103344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:26:11.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:11 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:26:11.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:11 smithi003 ceph-mon[103344]: osdmap e209: 8 total, 8 up, 7 in 2024-08-17T17:26:11.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:11 smithi003 ceph-mon[100615]: pgmap v527: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 127 B/s rd, 1.6 MiB/s wr, 757 op/s; 14033/276186 objects degraded (5.081%); 2582/276186 objects misplaced (0.935%); 5.8 MiB/s, 86 objects/s recovering 2024-08-17T17:26:11.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:11 smithi003 ceph-mon[100615]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:26:11.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:11 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:26:11.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:11 smithi003 ceph-mon[100615]: osdmap e209: 8 total, 8 up, 7 in 2024-08-17T17:26:11.563 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:11.591 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:11.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:11.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:11.693 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:11.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:11 smithi053 ceph-mon[89963]: pgmap v527: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 127 B/s rd, 1.6 MiB/s wr, 757 op/s; 14033/276186 objects degraded (5.081%); 2582/276186 objects misplaced (0.935%); 5.8 MiB/s, 86 objects/s recovering 2024-08-17T17:26:11.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:11 smithi053 ceph-mon[89963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:26:11.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:11 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:26:11.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:11 smithi053 ceph-mon[89963]: osdmap e209: 8 total, 8 up, 7 in 2024-08-17T17:26:11.738 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:11.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:11.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:11.840 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:11.869 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:11.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:11.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:11.971 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:12.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:12.004 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:12.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:12.043 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:26:12.043 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-08-17T17:26:12.044 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-08-17T17:26:12.044 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-08-17T17:26:12.044 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:26:12.105 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:12.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:12.133 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:12.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:12.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:12.266 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:12.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:12.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:12.368 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:12.397 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:12.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:12.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:12.499 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:12.540 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:12.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:12.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:12.592 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:26:12.642 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:12.672 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:12.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:12.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:12.774 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:12.814 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:12.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:12.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:12.916 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:12.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:12.954 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:12.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:13.055 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:13.089 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:13.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:13.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:13.156 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-17T17:26:13.192 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:13.259 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:13.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:13.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:13.361 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:13.392 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:13.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:13.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:13.494 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:13.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:13 smithi003 ceph-mon[103344]: pgmap v529: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 823 op/s; 13611/270948 objects degraded (5.023%); 2554/270948 objects misplaced (0.943%); 10 MiB/s, 154 objects/s recovering 2024-08-17T17:26:13.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:13 smithi003 ceph-mon[100615]: pgmap v529: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 823 op/s; 13611/270948 objects degraded (5.023%); 2554/270948 objects misplaced (0.943%); 10 MiB/s, 154 objects/s recovering 2024-08-17T17:26:13.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:13.545 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:13.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:13.646 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:13.685 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:13.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:13.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:13.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:13 smithi053 ceph-mon[89963]: pgmap v529: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 823 op/s; 13611/270948 objects degraded (5.023%); 2554/270948 objects misplaced (0.943%); 10 MiB/s, 154 objects/s recovering 2024-08-17T17:26:13.787 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:13.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:13.828 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:13.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:13.929 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:13.967 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:13.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:13.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:14.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:14.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:14.111 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:14.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:14.212 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:14.257 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:14.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:14.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:14.266 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:26:14.267 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:26:14.359 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:14.396 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:14.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:14.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:14.498 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:14.532 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:14.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:14.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:14.634 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:14.666 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:14.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:14.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:14.768 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:14.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:14.801 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:14.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:14.902 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:14.942 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:14.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:14.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:15.044 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:15.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:15.084 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:15.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:15.185 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:15.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:15.235 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:15.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:15.337 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:15.374 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:15.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:15.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:15.426 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:15 smithi003 ceph-mon[103344]: pgmap v530: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 733 op/s; 13611/267189 objects degraded (5.094%); 2554/267189 objects misplaced (0.956%); 4.4 MiB/s, 61 objects/s recovering 2024-08-17T17:26:15.426 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:15 smithi003 ceph-mon[103344]: from='client.45602 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:26:15.426 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:15 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 13611/267189 objects degraded (5.094%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:15.429 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:15 smithi003 ceph-mon[100615]: pgmap v530: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 733 op/s; 13611/267189 objects degraded (5.094%); 2554/267189 objects misplaced (0.956%); 4.4 MiB/s, 61 objects/s recovering 2024-08-17T17:26:15.429 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:15 smithi003 ceph-mon[100615]: from='client.45602 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:26:15.429 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:15 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 13611/267189 objects degraded (5.094%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:15.476 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:15.517 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:15.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:15.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:15.619 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:15.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:15.653 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:15.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:15.682 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:26:15.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:15 smithi053 ceph-mon[89963]: pgmap v530: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 733 op/s; 13611/267189 objects degraded (5.094%); 2554/267189 objects misplaced (0.956%); 4.4 MiB/s, 61 objects/s recovering 2024-08-17T17:26:15.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:15 smithi053 ceph-mon[89963]: from='client.45602 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:26:15.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:15 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 13611/267189 objects degraded (5.094%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:15.725 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:26:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:26:15] "GET /metrics HTTP/1.1" 200 36526 "" "Prometheus/2.51.0" 2024-08-17T17:26:15.754 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:15.782 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:15.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:15.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:15.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:15.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:15.932 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:15.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:15.974 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":530,"stamp":"2024-08-17T17:26:13.881883+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5839722469,"num_objects":89063,"num_object_clones":0,"num_object_copies":267189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13611,"num_objects_misplaced":2554,"num_objects_unfound":0,"num_objects_dirty":89063,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":109381,"num_write_kb":6328384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6829,"num_bytes_recovered":449127814,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106240,"ondisk_log_size":106240,"up":267,"acting":262,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":282,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":19780116,"kb_used_data":17678984,"kb_used_omap":33,"kb_used_meta":2100766,"kb_avail":636435948,"statfs":{"total":671965249536,"available":651710410752,"internally_reserved":0,"allocated":18103279616,"data_stored":18094959595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34068,"internal_metadata":2151185132},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,8,11,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-321585152,"num_objects":-4907,"num_object_clones":0,"num_object_copies":-14721,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3149,"num_objects_misplaced":2526,"num_objects_unfound":0,"num_objects_dirty":-4907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4907,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":412,"num_bytes_recovered":30867456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.685581"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":271,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.260618+0000","last_change":"2024-08-17T17:17:32.495652+0000","last_active":"2024-08-17T17:26:10.260618+0000","last_peered":"2024-08-17T17:26:10.260618+0000","last_clean":"2024-08-17T17:26:10.260618+0000","last_became_active":"2024-08-17T17:17:32.495218+0000","last_became_peered":"2024-08-17T17:17:32.495218+0000","last_unstale":"2024-08-17T17:26:10.260618+0000","last_undegraded":"2024-08-17T17:26:10.260618+0000","last_fullsized":"2024-08-17T17:26:10.260618+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:11:45.821282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":184,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-08-17T17:26:07.210438+0000","last_change":"2024-08-17T17:18:59.424887+0000","last_active":"2024-08-17T17:26:07.210438+0000","last_peered":"2024-08-17T17:26:07.210438+0000","last_clean":"2024-08-17T17:26:07.210438+0000","last_became_active":"2024-08-17T17:18:59.411539+0000","last_became_peered":"2024-08-17T17:18:59.411539+0000","last_unstale":"2024-08-17T17:26:07.210438+0000","last_undegraded":"2024-08-17T17:26:07.210438+0000","last_fullsized":"2024-08-17T17:26:07.210438+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:46:20.082241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.239048+0000","last_change":"2024-08-17T17:24:53.460742+0000","last_active":"2024-08-17T17:26:08.239048+0000","last_peered":"2024-08-17T17:26:08.239048+0000","last_clean":"2024-08-17T17:26:08.239048+0000","last_became_active":"2024-08-17T17:24:53.460365+0000","last_became_peered":"2024-08-17T17:24:53.460365+0000","last_unstale":"2024-08-17T17:26:08.239048+0000","last_undegraded":"2024-08-17T17:26:08.239048+0000","last_fullsized":"2024-08-17T17:26:08.239048+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:24:08.116968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":82,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.259259+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:26:10.259259+0000","last_peered":"2024-08-17T17:26:10.259259+0000","last_clean":"2024-08-17T17:26:10.259259+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:26:10.259259+0000","last_undegraded":"2024-08-17T17:26:10.259259+0000","last_fullsized":"2024-08-17T17:26:10.259259+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":82,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.249047+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:26:10.249047+0000","last_peered":"2024-08-17T17:26:10.249047+0000","last_clean":"2024-08-17T17:26:10.249047+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:26:10.249047+0000","last_undegraded":"2024-08-17T17:26:10.249047+0000","last_fullsized":"2024-08-17T17:26:10.249047+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":159,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-08-17T17:26:07.210048+0000","last_change":"2024-08-17T17:20:49.258206+0000","last_active":"2024-08-17T17:26:07.210048+0000","last_peered":"2024-08-17T17:26:07.210048+0000","last_clean":"2024-08-17T17:26:07.210048+0000","last_became_active":"2024-08-17T17:20:49.257761+0000","last_became_peered":"2024-08-17T17:20:49.257761+0000","last_unstale":"2024-08-17T17:26:07.210048+0000","last_undegraded":"2024-08-17T17:26:07.210048+0000","last_fullsized":"2024-08-17T17:26:07.210048+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:38:35.363748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":285,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.266253+0000","last_change":"2024-08-17T17:19:57.514773+0000","last_active":"2024-08-17T17:26:10.266253+0000","last_peered":"2024-08-17T17:26:10.266253+0000","last_clean":"2024-08-17T17:26:10.266253+0000","last_became_active":"2024-08-17T17:19:57.513839+0000","last_became_peered":"2024-08-17T17:19:57.513839+0000","last_unstale":"2024-08-17T17:26:10.266253+0000","last_undegraded":"2024-08-17T17:26:10.266253+0000","last_fullsized":"2024-08-17T17:26:10.266253+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:42:48.063169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":255,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.221838+0000","last_change":"2024-08-17T17:24:53.465875+0000","last_active":"2024-08-17T17:26:08.221838+0000","last_peered":"2024-08-17T17:26:08.221838+0000","last_clean":"2024-08-17T17:26:08.221838+0000","last_became_active":"2024-08-17T17:24:53.458129+0000","last_became_peered":"2024-08-17T17:24:53.458129+0000","last_unstale":"2024-08-17T17:26:08.221838+0000","last_undegraded":"2024-08-17T17:26:08.221838+0000","last_fullsized":"2024-08-17T17:26:08.221838+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:51:33.584869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"209'6835","reported_seq":6892,"reported_epoch":209,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:26:12.916189+0000","last_change":"2024-08-17T17:24:54.892004+0000","last_active":"2024-08-17T17:26:12.916189+0000","last_peered":"2024-08-17T17:26:12.916189+0000","last_clean":"2024-08-17T17:24:52.438492+0000","last_became_active":"2024-08-17T17:24:54.887269+0000","last_became_peered":"2024-08-17T17:24:54.887269+0000","last_unstale":"2024-08-17T17:26:12.916189+0000","last_undegraded":"2024-08-17T17:24:54.477707+0000","last_fullsized":"2024-08-17T17:24:54.461071+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6835,"log_dups_size":0,"ondisk_log_size":6835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":354877440,"num_objects":5415,"num_object_clones":0,"num_object_copies":16245,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6835,"num_write_kb":392000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[0,6],"avail_no_missing":["0","2","6"],"object_location_counts":[{"shards":"0,2,6","objects":1750},{"shards":"0,6","objects":2955}],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":120,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-08-17T17:26:07.210420+0000","last_change":"2024-08-17T17:24:06.597382+0000","last_active":"2024-08-17T17:26:07.210420+0000","last_peered":"2024-08-17T17:26:07.210420+0000","last_clean":"2024-08-17T17:26:07.210420+0000","last_became_active":"2024-08-17T17:24:06.588881+0000","last_became_peered":"2024-08-17T17:24:06.588881+0000","last_unstale":"2024-08-17T17:26:07.210420+0000","last_undegraded":"2024-08-17T17:26:07.210420+0000","last_fullsized":"2024-08-17T17:26:07.210420+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:27.240751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":35,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.238949+0000","last_change":"2024-08-17T17:25:04.503780+0000","last_active":"2024-08-17T17:26:08.238949+0000","last_peered":"2024-08-17T17:26:08.238949+0000","last_clean":"2024-08-17T17:26:08.238949+0000","last_became_active":"2024-08-17T17:25:04.503503+0000","last_became_peered":"2024-08-17T17:25:04.503503+0000","last_unstale":"2024-08-17T17:26:08.238949+0000","last_undegraded":"2024-08-17T17:26:08.238949+0000","last_fullsized":"2024-08-17T17:26:08.238949+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:04:48.171216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"209'6734","reported_seq":6807,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:11.473640+0000","last_change":"2024-08-17T17:24:14.639361+0000","last_active":"2024-08-17T17:26:11.473640+0000","last_peered":"2024-08-17T17:26:11.473640+0000","last_clean":"2024-08-17T17:26:11.473640+0000","last_became_active":"2024-08-17T17:24:14.639074+0000","last_became_peered":"2024-08-17T17:24:14.639074+0000","last_unstale":"2024-08-17T17:26:11.473640+0000","last_undegraded":"2024-08-17T17:26:11.473640+0000","last_fullsized":"2024-08-17T17:26:11.473640+0000","mapping_epoch":178,"log_start":"183'300","ondisk_log_start":"183'300","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6434,"log_dups_size":0,"ondisk_log_size":6434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:15:05.377992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368967680,"num_objects":5630,"num_object_clones":0,"num_object_copies":16890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6734,"num_write_kb":395648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":106,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-08-17T17:26:07.211505+0000","last_change":"2024-08-17T17:24:06.599023+0000","last_active":"2024-08-17T17:26:07.211505+0000","last_peered":"2024-08-17T17:26:07.211505+0000","last_clean":"2024-08-17T17:26:07.211505+0000","last_became_active":"2024-08-17T17:24:06.588954+0000","last_became_peered":"2024-08-17T17:24:06.588954+0000","last_unstale":"2024-08-17T17:26:07.211505+0000","last_undegraded":"2024-08-17T17:26:07.211505+0000","last_fullsized":"2024-08-17T17:26:07.211505+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:20:51.445576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":226,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.265890+0000","last_change":"2024-08-17T17:20:49.271540+0000","last_active":"2024-08-17T17:26:10.265890+0000","last_peered":"2024-08-17T17:26:10.265890+0000","last_clean":"2024-08-17T17:26:10.265890+0000","last_became_active":"2024-08-17T17:20:49.271130+0000","last_became_peered":"2024-08-17T17:20:49.271130+0000","last_unstale":"2024-08-17T17:26:10.265890+0000","last_undegraded":"2024-08-17T17:26:10.265890+0000","last_fullsized":"2024-08-17T17:26:10.265890+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:05.386705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"208'6612","reported_seq":6650,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.926773+0000","last_change":"2024-08-17T17:24:14.644070+0000","last_active":"2024-08-17T17:26:08.926773+0000","last_peered":"2024-08-17T17:26:08.926773+0000","last_clean":"2024-08-17T17:26:08.926773+0000","last_became_active":"2024-08-17T17:24:14.643799+0000","last_became_peered":"2024-08-17T17:24:14.643799+0000","last_unstale":"2024-08-17T17:26:08.926773+0000","last_undegraded":"2024-08-17T17:26:08.926773+0000","last_fullsized":"2024-08-17T17:26:08.926773+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6612,"log_dups_size":0,"ondisk_log_size":6612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:25:38.445770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370278400,"num_objects":5650,"num_object_clones":0,"num_object_copies":16950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6612,"num_write_kb":392384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":178,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-08-17T17:26:07.214304+0000","last_change":"2024-08-17T17:24:53.467526+0000","last_active":"2024-08-17T17:26:07.214304+0000","last_peered":"2024-08-17T17:26:07.214304+0000","last_clean":"2024-08-17T17:26:07.214304+0000","last_became_active":"2024-08-17T17:24:53.460631+0000","last_became_peered":"2024-08-17T17:24:53.460631+0000","last_unstale":"2024-08-17T17:26:07.214304+0000","last_undegraded":"2024-08-17T17:26:07.214304+0000","last_fullsized":"2024-08-17T17:26:07.214304+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:05:24.938831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.221983+0000","last_change":"2024-08-17T17:25:04.507176+0000","last_active":"2024-08-17T17:26:08.221983+0000","last_peered":"2024-08-17T17:26:08.221983+0000","last_clean":"2024-08-17T17:26:08.221983+0000","last_became_active":"2024-08-17T17:25:04.506788+0000","last_became_peered":"2024-08-17T17:25:04.506788+0000","last_unstale":"2024-08-17T17:26:08.221983+0000","last_undegraded":"2024-08-17T17:26:08.221983+0000","last_fullsized":"2024-08-17T17:26:08.221983+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:22:04.658216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"209'6654","reported_seq":6727,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:12.967575+0000","last_change":"2024-08-17T17:24:14.642637+0000","last_active":"2024-08-17T17:26:12.967575+0000","last_peered":"2024-08-17T17:26:12.967575+0000","last_clean":"2024-08-17T17:26:12.967575+0000","last_became_active":"2024-08-17T17:24:14.642347+0000","last_became_peered":"2024-08-17T17:24:14.642347+0000","last_unstale":"2024-08-17T17:26:12.967575+0000","last_undegraded":"2024-08-17T17:26:12.967575+0000","last_fullsized":"2024-08-17T17:26:12.967575+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6654,"log_dups_size":0,"ondisk_log_size":6654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:37:44.631335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352321536,"num_objects":5376,"num_object_clones":0,"num_object_copies":16128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6654,"num_write_kb":384960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":167,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.239715+0000","last_change":"2024-08-17T17:17:32.495428+0000","last_active":"2024-08-17T17:26:08.239715+0000","last_peered":"2024-08-17T17:26:08.239715+0000","last_clean":"2024-08-17T17:26:08.239715+0000","last_became_active":"2024-08-17T17:17:32.494978+0000","last_became_peered":"2024-08-17T17:17:32.494978+0000","last_unstale":"2024-08-17T17:26:08.239715+0000","last_undegraded":"2024-08-17T17:26:08.239715+0000","last_fullsized":"2024-08-17T17:26:08.239715+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:42:24.647812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":274,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.248687+0000","last_change":"2024-08-17T17:20:49.272126+0000","last_active":"2024-08-17T17:26:10.248687+0000","last_peered":"2024-08-17T17:26:10.248687+0000","last_clean":"2024-08-17T17:26:10.248687+0000","last_became_active":"2024-08-17T17:20:49.271185+0000","last_became_peered":"2024-08-17T17:20:49.271185+0000","last_unstale":"2024-08-17T17:26:10.248687+0000","last_undegraded":"2024-08-17T17:26:10.248687+0000","last_fullsized":"2024-08-17T17:26:10.248687+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:43:58.184939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"209'6951","reported_seq":7899,"reported_epoch":209,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:26:10.268939+0000","last_change":"2024-08-17T17:24:54.898729+0000","last_active":"2024-08-17T17:26:10.268939+0000","last_peered":"2024-08-17T17:26:10.268939+0000","last_clean":"2024-08-17T17:24:52.435411+0000","last_became_active":"2024-08-17T17:24:54.889545+0000","last_became_peered":"2024-08-17T17:24:54.889545+0000","last_unstale":"2024-08-17T17:26:10.268939+0000","last_undegraded":"2024-08-17T17:24:54.482592+0000","last_fullsized":"2024-08-17T17:24:54.459529+0000","mapping_epoch":194,"log_start":"186'700","ondisk_log_start":"186'700","created":178,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6251,"log_dups_size":0,"ondisk_log_size":6251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380305408,"num_objects":5803,"num_object_clones":0,"num_object_copies":17409,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6951,"num_write_kb":408128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":430,"num_bytes_recovered":28180480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2],"avail_no_missing":["1","2","3","7"],"object_location_counts":[{"shards":"1,2","objects":3216},{"shards":"1,2,7","objects":2013}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":302,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.256038+0000","last_change":"2024-08-17T17:24:06.595380+0000","last_active":"2024-08-17T17:26:10.256038+0000","last_peered":"2024-08-17T17:26:10.256038+0000","last_clean":"2024-08-17T17:26:10.256038+0000","last_became_active":"2024-08-17T17:24:06.584747+0000","last_became_peered":"2024-08-17T17:24:06.584747+0000","last_unstale":"2024-08-17T17:26:10.256038+0000","last_undegraded":"2024-08-17T17:26:10.256038+0000","last_fullsized":"2024-08-17T17:26:10.256038+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:02:18.473980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":248,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.260897+0000","last_change":"2024-08-17T17:20:49.280596+0000","last_active":"2024-08-17T17:26:10.260897+0000","last_peered":"2024-08-17T17:26:10.260897+0000","last_clean":"2024-08-17T17:26:10.260897+0000","last_became_active":"2024-08-17T17:20:49.279553+0000","last_became_peered":"2024-08-17T17:20:49.279553+0000","last_unstale":"2024-08-17T17:26:10.260897+0000","last_undegraded":"2024-08-17T17:26:10.260897+0000","last_fullsized":"2024-08-17T17:26:10.260897+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:13:24.124392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"209'6717","reported_seq":6790,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:11.564520+0000","last_change":"2024-08-17T17:24:14.638890+0000","last_active":"2024-08-17T17:26:11.564520+0000","last_peered":"2024-08-17T17:26:11.564520+0000","last_clean":"2024-08-17T17:26:11.564520+0000","last_became_active":"2024-08-17T17:24:14.638594+0000","last_became_peered":"2024-08-17T17:24:14.638594+0000","last_unstale":"2024-08-17T17:26:11.564520+0000","last_undegraded":"2024-08-17T17:26:11.564520+0000","last_fullsized":"2024-08-17T17:26:11.564520+0000","mapping_epoch":178,"log_start":"183'300","ondisk_log_start":"183'300","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6417,"log_dups_size":0,"ondisk_log_size":6417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:53:34.277077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359727104,"num_objects":5489,"num_object_clones":0,"num_object_copies":16467,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6717,"num_write_kb":390592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":200,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.266358+0000","last_change":"2024-08-17T17:20:49.276769+0000","last_active":"2024-08-17T17:26:10.266358+0000","last_peered":"2024-08-17T17:26:10.266358+0000","last_clean":"2024-08-17T17:26:10.266358+0000","last_became_active":"2024-08-17T17:20:49.276498+0000","last_became_peered":"2024-08-17T17:20:49.276498+0000","last_unstale":"2024-08-17T17:26:10.266358+0000","last_undegraded":"2024-08-17T17:26:10.266358+0000","last_fullsized":"2024-08-17T17:26:10.266358+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:39:05.419440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":187,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-08-17T17:26:07.213869+0000","last_change":"2024-08-17T17:20:14.455738+0000","last_active":"2024-08-17T17:26:07.213869+0000","last_peered":"2024-08-17T17:26:07.213869+0000","last_clean":"2024-08-17T17:26:07.213869+0000","last_became_active":"2024-08-17T17:20:14.455469+0000","last_became_peered":"2024-08-17T17:20:14.455469+0000","last_unstale":"2024-08-17T17:26:07.213869+0000","last_undegraded":"2024-08-17T17:26:07.213869+0000","last_fullsized":"2024-08-17T17:26:07.213869+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:27:28.424336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"209'6785","reported_seq":8364,"reported_epoch":209,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:26:11.588746+0000","last_change":"2024-08-17T17:25:05.678234+0000","last_active":"2024-08-17T17:26:11.588746+0000","last_peered":"2024-08-17T17:26:11.588746+0000","last_clean":"2024-08-17T17:24:52.436463+0000","last_became_active":"2024-08-17T17:25:05.673238+0000","last_became_peered":"2024-08-17T17:25:05.673238+0000","last_unstale":"2024-08-17T17:26:11.588746+0000","last_undegraded":"2024-08-17T17:25:05.585769+0000","last_fullsized":"2024-08-17T17:25:05.499112+0000","mapping_epoch":198,"log_start":"183'400","ondisk_log_start":"183'400","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6385,"log_dups_size":0,"ondisk_log_size":6385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365887488,"num_objects":5583,"num_object_clones":0,"num_object_copies":16749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4862,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6785,"num_write_kb":395776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":601,"num_bytes_recovered":39387136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,5],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":1918},{"shards":"3,5","objects":2639},{"shards":"3,5,7","objects":721}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":141,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-08-17T17:26:07.209969+0000","last_change":"2024-08-17T17:20:14.431140+0000","last_active":"2024-08-17T17:26:07.209969+0000","last_peered":"2024-08-17T17:26:07.209969+0000","last_clean":"2024-08-17T17:26:07.209969+0000","last_became_active":"2024-08-17T17:20:14.430839+0000","last_became_peered":"2024-08-17T17:20:14.430839+0000","last_unstale":"2024-08-17T17:26:07.209969+0000","last_undegraded":"2024-08-17T17:26:07.209969+0000","last_fullsized":"2024-08-17T17:26:07.209969+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:08:16.303616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.222172+0000","last_change":"2024-08-17T17:25:04.509325+0000","last_active":"2024-08-17T17:26:08.222172+0000","last_peered":"2024-08-17T17:26:08.222172+0000","last_clean":"2024-08-17T17:26:08.222172+0000","last_became_active":"2024-08-17T17:25:04.508364+0000","last_became_peered":"2024-08-17T17:25:04.508364+0000","last_unstale":"2024-08-17T17:26:08.222172+0000","last_undegraded":"2024-08-17T17:26:08.222172+0000","last_fullsized":"2024-08-17T17:26:08.222172+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:48:56.415763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"209'6746","reported_seq":11574,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:13.734932+0000","last_change":"2024-08-17T17:25:40.811756+0000","last_active":"2024-08-17T17:26:13.734932+0000","last_peered":"2024-08-17T17:26:13.734932+0000","last_clean":"2024-08-17T17:26:13.734932+0000","last_became_active":"2024-08-17T17:25:40.771755+0000","last_became_peered":"2024-08-17T17:25:40.771755+0000","last_unstale":"2024-08-17T17:26:13.734932+0000","last_undegraded":"2024-08-17T17:26:13.734932+0000","last_fullsized":"2024-08-17T17:26:13.734932+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6746,"log_dups_size":0,"ondisk_log_size":6746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:13:03.918487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351993880,"num_objects":5372,"num_object_clones":0,"num_object_copies":16116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5372,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6746,"num_write_kb":387713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2371,"num_bytes_recovered":155385856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":287,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.254213+0000","last_change":"2024-08-17T17:19:57.520745+0000","last_active":"2024-08-17T17:26:10.254213+0000","last_peered":"2024-08-17T17:26:10.254213+0000","last_clean":"2024-08-17T17:26:10.254213+0000","last_became_active":"2024-08-17T17:19:57.520152+0000","last_became_peered":"2024-08-17T17:19:57.520152+0000","last_unstale":"2024-08-17T17:26:10.254213+0000","last_undegraded":"2024-08-17T17:26:10.254213+0000","last_fullsized":"2024-08-17T17:26:10.254213+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:19:39.495644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":262,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.264049+0000","last_change":"2024-08-17T17:20:49.274372+0000","last_active":"2024-08-17T17:26:10.264049+0000","last_peered":"2024-08-17T17:26:10.264049+0000","last_clean":"2024-08-17T17:26:10.264049+0000","last_became_active":"2024-08-17T17:20:49.274023+0000","last_became_peered":"2024-08-17T17:20:49.274023+0000","last_unstale":"2024-08-17T17:26:10.264049+0000","last_undegraded":"2024-08-17T17:26:10.264049+0000","last_fullsized":"2024-08-17T17:26:10.264049+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:31:02.529766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"209'7138","reported_seq":7177,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:13.771391+0000","last_change":"2024-08-17T17:24:14.643398+0000","last_active":"2024-08-17T17:26:13.771391+0000","last_peered":"2024-08-17T17:26:13.771391+0000","last_clean":"2024-08-17T17:26:13.771391+0000","last_became_active":"2024-08-17T17:24:14.643102+0000","last_became_peered":"2024-08-17T17:24:14.643102+0000","last_unstale":"2024-08-17T17:26:13.771391+0000","last_undegraded":"2024-08-17T17:26:13.771391+0000","last_fullsized":"2024-08-17T17:26:13.771391+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":7138,"log_dups_size":0,"ondisk_log_size":7138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:09:30.323709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365035520,"num_objects":5570,"num_object_clones":0,"num_object_copies":16710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7138,"num_write_kb":406656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":535,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-08-17T17:26:07.209345+0000","last_change":"2024-08-17T17:24:09.599615+0000","last_active":"2024-08-17T17:26:07.209345+0000","last_peered":"2024-08-17T17:26:07.209345+0000","last_clean":"2024-08-17T17:26:07.209345+0000","last_became_active":"2024-08-17T17:24:09.599085+0000","last_became_peered":"2024-08-17T17:24:09.599085+0000","last_unstale":"2024-08-17T17:26:07.209345+0000","last_undegraded":"2024-08-17T17:26:07.209345+0000","last_fullsized":"2024-08-17T17:26:07.209345+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:04:01.141401+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:30:09.569579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":781,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.239121+0000","last_change":"2024-08-17T17:20:00.434774+0000","last_active":"2024-08-17T17:26:08.239121+0000","last_peered":"2024-08-17T17:26:08.239121+0000","last_clean":"2024-08-17T17:26:08.239121+0000","last_became_active":"2024-08-17T17:20:00.434343+0000","last_became_peered":"2024-08-17T17:20:00.434343+0000","last_unstale":"2024-08-17T17:26:08.239121+0000","last_undegraded":"2024-08-17T17:26:08.239121+0000","last_fullsized":"2024-08-17T17:26:08.239121+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:04.995152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":245,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.255717+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:26:10.255717+0000","last_peered":"2024-08-17T17:26:10.255717+0000","last_clean":"2024-08-17T17:26:10.255717+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:26:10.255717+0000","last_undegraded":"2024-08-17T17:26:10.255717+0000","last_fullsized":"2024-08-17T17:26:10.255717+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":21,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.224011+0000","last_change":"2024-08-17T17:25:04.579426+0000","last_active":"2024-08-17T17:26:08.224011+0000","last_peered":"2024-08-17T17:26:08.224011+0000","last_clean":"2024-08-17T17:26:08.224011+0000","last_became_active":"2024-08-17T17:25:04.579012+0000","last_became_peered":"2024-08-17T17:25:04.579012+0000","last_unstale":"2024-08-17T17:26:08.224011+0000","last_undegraded":"2024-08-17T17:26:08.224011+0000","last_fullsized":"2024-08-17T17:26:08.224011+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:56:32.241321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"208'6647","reported_seq":6720,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.267078+0000","last_change":"2024-08-17T17:24:14.641829+0000","last_active":"2024-08-17T17:26:10.267078+0000","last_peered":"2024-08-17T17:26:10.267078+0000","last_clean":"2024-08-17T17:26:10.267078+0000","last_became_active":"2024-08-17T17:24:14.641532+0000","last_became_peered":"2024-08-17T17:24:14.641532+0000","last_unstale":"2024-08-17T17:26:10.267078+0000","last_undegraded":"2024-08-17T17:26:10.267078+0000","last_fullsized":"2024-08-17T17:26:10.267078+0000","mapping_epoch":178,"log_start":"183'300","ondisk_log_start":"183'300","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6347,"log_dups_size":0,"ondisk_log_size":6347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:43:05.232036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366411776,"num_objects":5591,"num_object_clones":0,"num_object_copies":16773,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6647,"num_write_kb":391616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":798,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-08-17T17:26:07.209520+0000","last_change":"2024-08-17T17:16:26.958415+0000","last_active":"2024-08-17T17:26:07.209520+0000","last_peered":"2024-08-17T17:26:07.209520+0000","last_clean":"2024-08-17T17:26:07.209520+0000","last_became_active":"2024-08-17T17:16:26.957956+0000","last_became_peered":"2024-08-17T17:16:26.957956+0000","last_unstale":"2024-08-17T17:26:07.209520+0000","last_undegraded":"2024-08-17T17:26:07.209520+0000","last_fullsized":"2024-08-17T17:26:07.209520+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:22.478077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":273,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.251093+0000","last_change":"2024-08-17T17:18:59.415331+0000","last_active":"2024-08-17T17:26:10.251093+0000","last_peered":"2024-08-17T17:26:10.251093+0000","last_clean":"2024-08-17T17:26:10.251093+0000","last_became_active":"2024-08-17T17:18:59.415008+0000","last_became_peered":"2024-08-17T17:18:59.415008+0000","last_unstale":"2024-08-17T17:26:10.251093+0000","last_undegraded":"2024-08-17T17:26:10.251093+0000","last_fullsized":"2024-08-17T17:26:10.251093+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:22:38.076866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":82,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.249218+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:26:10.249218+0000","last_peered":"2024-08-17T17:26:10.249218+0000","last_clean":"2024-08-17T17:26:10.249218+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:26:10.249218+0000","last_undegraded":"2024-08-17T17:26:10.249218+0000","last_fullsized":"2024-08-17T17:26:10.249218+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"208'6550","reported_seq":6934,"reported_epoch":208,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-17T17:26:08.987414+0000","last_change":"2024-08-17T17:26:06.842453+0000","last_active":"2024-08-17T17:26:08.987414+0000","last_peered":"2024-08-17T17:26:08.987414+0000","last_clean":"2024-08-17T17:25:03.970571+0000","last_became_active":"2024-08-17T17:25:05.604478+0000","last_became_peered":"2024-08-17T17:25:05.604478+0000","last_unstale":"2024-08-17T17:26:08.987414+0000","last_undegraded":"2024-08-17T17:25:05.510212+0000","last_fullsized":"2024-08-17T17:25:05.498949+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6550,"log_dups_size":0,"ondisk_log_size":6550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:29:09.928695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369098752,"num_objects":5632,"num_object_clones":0,"num_object_copies":16896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":386,"num_objects_misplaced":2554,"num_objects_unfound":0,"num_objects_dirty":5632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6550,"num_write_kb":389824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":11010048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,3],"avail_no_missing":["7","2","3"],"object_location_counts":[{"shards":"2,3,7","objects":2554},{"shards":"3,7","objects":2692}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":661,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.255766+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:26:10.255766+0000","last_peered":"2024-08-17T17:26:10.255766+0000","last_clean":"2024-08-17T17:26:10.255766+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:26:10.255766+0000","last_undegraded":"2024-08-17T17:26:10.255766+0000","last_fullsized":"2024-08-17T17:26:10.255766+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":269,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.263805+0000","last_change":"2024-08-17T17:19:57.515564+0000","last_active":"2024-08-17T17:26:10.263805+0000","last_peered":"2024-08-17T17:26:10.263805+0000","last_clean":"2024-08-17T17:26:10.263805+0000","last_became_active":"2024-08-17T17:19:57.515142+0000","last_became_peered":"2024-08-17T17:19:57.515142+0000","last_unstale":"2024-08-17T17:26:10.263805+0000","last_undegraded":"2024-08-17T17:26:10.263805+0000","last_fullsized":"2024-08-17T17:26:10.263805+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:59:33.506176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":300,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.263880+0000","last_change":"2024-08-17T17:24:06.587903+0000","last_active":"2024-08-17T17:26:10.263880+0000","last_peered":"2024-08-17T17:26:10.263880+0000","last_clean":"2024-08-17T17:26:10.263880+0000","last_became_active":"2024-08-17T17:24:06.587503+0000","last_became_peered":"2024-08-17T17:24:06.587503+0000","last_unstale":"2024-08-17T17:26:10.263880+0000","last_undegraded":"2024-08-17T17:26:10.263880+0000","last_fullsized":"2024-08-17T17:26:10.263880+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:05:58.591992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"209'6659","reported_seq":12949,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:11.495229+0000","last_change":"2024-08-17T17:26:08.233981+0000","last_active":"2024-08-17T17:26:11.495229+0000","last_peered":"2024-08-17T17:26:11.495229+0000","last_clean":"2024-08-17T17:26:11.495229+0000","last_became_active":"2024-08-17T17:26:08.225711+0000","last_became_peered":"2024-08-17T17:26:08.225711+0000","last_unstale":"2024-08-17T17:26:11.495229+0000","last_undegraded":"2024-08-17T17:26:11.495229+0000","last_fullsized":"2024-08-17T17:26:11.495229+0000","mapping_epoch":207,"log_start":"183'200","ondisk_log_start":"183'200","created":178,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6459,"log_dups_size":0,"ondisk_log_size":6459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:31:42.806869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":360644608,"num_objects":5503,"num_object_clones":0,"num_object_copies":16509,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6659,"num_write_kb":389184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3101,"num_bytes_recovered":203227136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":1119,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.239009+0000","last_change":"2024-08-17T17:20:00.433870+0000","last_active":"2024-08-17T17:26:08.239009+0000","last_peered":"2024-08-17T17:26:08.239009+0000","last_clean":"2024-08-17T17:26:08.239009+0000","last_became_active":"2024-08-17T17:20:00.433386+0000","last_became_peered":"2024-08-17T17:20:00.433386+0000","last_unstale":"2024-08-17T17:26:08.239009+0000","last_undegraded":"2024-08-17T17:26:08.239009+0000","last_fullsized":"2024-08-17T17:26:08.239009+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:01:11.848862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1392,"num_read_kb":2863,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":195,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.248530+0000","last_change":"2024-08-17T17:20:49.272407+0000","last_active":"2024-08-17T17:26:10.248530+0000","last_peered":"2024-08-17T17:26:10.248530+0000","last_clean":"2024-08-17T17:26:10.248530+0000","last_became_active":"2024-08-17T17:20:49.271749+0000","last_became_peered":"2024-08-17T17:20:49.271749+0000","last_unstale":"2024-08-17T17:26:10.248530+0000","last_undegraded":"2024-08-17T17:26:10.248530+0000","last_fullsized":"2024-08-17T17:26:10.248530+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:36:01.443457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":266,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.267288+0000","last_change":"2024-08-17T17:17:32.499715+0000","last_active":"2024-08-17T17:26:10.267288+0000","last_peered":"2024-08-17T17:26:10.267288+0000","last_clean":"2024-08-17T17:26:10.267288+0000","last_became_active":"2024-08-17T17:17:32.499412+0000","last_became_peered":"2024-08-17T17:17:32.499412+0000","last_unstale":"2024-08-17T17:26:10.267288+0000","last_undegraded":"2024-08-17T17:26:10.267288+0000","last_fullsized":"2024-08-17T17:26:10.267288+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:39:38.456970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"209'6886","reported_seq":6959,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:12.971825+0000","last_change":"2024-08-17T17:24:14.642608+0000","last_active":"2024-08-17T17:26:12.971825+0000","last_peered":"2024-08-17T17:26:12.971825+0000","last_clean":"2024-08-17T17:26:12.971825+0000","last_became_active":"2024-08-17T17:24:14.642292+0000","last_became_peered":"2024-08-17T17:24:14.642292+0000","last_unstale":"2024-08-17T17:26:12.971825+0000","last_undegraded":"2024-08-17T17:26:12.971825+0000","last_fullsized":"2024-08-17T17:26:12.971825+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6886,"log_dups_size":0,"ondisk_log_size":6886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:53:36.848614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356384768,"num_objects":5438,"num_object_clones":0,"num_object_copies":16314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6886,"num_write_kb":394368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":184,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.239220+0000","last_change":"2024-08-17T17:17:32.495728+0000","last_active":"2024-08-17T17:26:08.239220+0000","last_peered":"2024-08-17T17:26:08.239220+0000","last_clean":"2024-08-17T17:26:08.239220+0000","last_became_active":"2024-08-17T17:17:32.495357+0000","last_became_peered":"2024-08-17T17:17:32.495357+0000","last_unstale":"2024-08-17T17:26:08.239220+0000","last_undegraded":"2024-08-17T17:26:08.239220+0000","last_fullsized":"2024-08-17T17:26:08.239220+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:27:45.561567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":502,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.267457+0000","last_change":"2024-08-17T17:16:26.964621+0000","last_active":"2024-08-17T17:26:10.267457+0000","last_peered":"2024-08-17T17:26:10.267457+0000","last_clean":"2024-08-17T17:26:10.267457+0000","last_became_active":"2024-08-17T17:16:26.964058+0000","last_became_peered":"2024-08-17T17:16:26.964058+0000","last_unstale":"2024-08-17T17:26:10.267457+0000","last_undegraded":"2024-08-17T17:26:10.267457+0000","last_fullsized":"2024-08-17T17:26:10.267457+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:00:04.465644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":272,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.266443+0000","last_change":"2024-08-17T17:24:53.465550+0000","last_active":"2024-08-17T17:26:10.266443+0000","last_peered":"2024-08-17T17:26:10.266443+0000","last_clean":"2024-08-17T17:26:10.266443+0000","last_became_active":"2024-08-17T17:24:53.465228+0000","last_became_peered":"2024-08-17T17:24:53.465228+0000","last_unstale":"2024-08-17T17:26:10.266443+0000","last_undegraded":"2024-08-17T17:26:10.266443+0000","last_fullsized":"2024-08-17T17:26:10.266443+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:21:18.378359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"209'6776","reported_seq":6849,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.263204+0000","last_change":"2024-08-17T17:24:14.640584+0000","last_active":"2024-08-17T17:26:10.263204+0000","last_peered":"2024-08-17T17:26:10.263204+0000","last_clean":"2024-08-17T17:26:10.263204+0000","last_became_active":"2024-08-17T17:24:14.640168+0000","last_became_peered":"2024-08-17T17:24:14.640168+0000","last_unstale":"2024-08-17T17:26:10.263204+0000","last_undegraded":"2024-08-17T17:26:10.263204+0000","last_fullsized":"2024-08-17T17:26:10.263204+0000","mapping_epoch":178,"log_start":"186'500","ondisk_log_start":"186'500","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6276,"log_dups_size":0,"ondisk_log_size":6276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:38:36.176813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368836608,"num_objects":5628,"num_object_clones":0,"num_object_copies":16884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6776,"num_write_kb":396928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":297,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.267370+0000","last_change":"2024-08-17T17:24:53.464307+0000","last_active":"2024-08-17T17:26:10.267370+0000","last_peered":"2024-08-17T17:26:10.267370+0000","last_clean":"2024-08-17T17:26:10.267370+0000","last_became_active":"2024-08-17T17:24:53.463996+0000","last_became_peered":"2024-08-17T17:24:53.463996+0000","last_unstale":"2024-08-17T17:26:10.267370+0000","last_undegraded":"2024-08-17T17:26:10.267370+0000","last_fullsized":"2024-08-17T17:26:10.267370+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:04:18.993725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":635,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-08-17T17:26:07.214386+0000","last_change":"2024-08-17T17:24:09.599822+0000","last_active":"2024-08-17T17:26:07.214386+0000","last_peered":"2024-08-17T17:26:07.214386+0000","last_clean":"2024-08-17T17:26:07.214386+0000","last_became_active":"2024-08-17T17:24:09.599189+0000","last_became_peered":"2024-08-17T17:24:09.599189+0000","last_unstale":"2024-08-17T17:26:07.214386+0000","last_undegraded":"2024-08-17T17:26:07.214386+0000","last_fullsized":"2024-08-17T17:26:07.214386+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:04.465604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":256,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.248610+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:26:10.248610+0000","last_peered":"2024-08-17T17:26:10.248610+0000","last_clean":"2024-08-17T17:26:10.248610+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:26:10.248610+0000","last_undegraded":"2024-08-17T17:26:10.248610+0000","last_fullsized":"2024-08-17T17:26:10.248610+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"209'6898","reported_seq":6971,"reported_epoch":209,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:26:11.560540+0000","last_change":"2024-08-17T17:25:05.605653+0000","last_active":"2024-08-17T17:26:11.560540+0000","last_peered":"2024-08-17T17:26:11.560540+0000","last_clean":"2024-08-17T17:25:03.971607+0000","last_became_active":"2024-08-17T17:25:05.601333+0000","last_became_peered":"2024-08-17T17:25:05.601333+0000","last_unstale":"2024-08-17T17:26:11.560540+0000","last_undegraded":"2024-08-17T17:25:05.522007+0000","last_fullsized":"2024-08-17T17:25:05.499355+0000","mapping_epoch":198,"log_start":"186'500","ondisk_log_start":"186'500","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6398,"log_dups_size":0,"ondisk_log_size":6398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370278400,"num_objects":5650,"num_object_clones":0,"num_object_copies":16950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6898,"num_write_kb":401536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2],"avail_no_missing":["3","2","7"],"object_location_counts":[{"shards":"2,3","objects":2334},{"shards":"2,3,7","objects":2692}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":292,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.266509+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:26:10.266509+0000","last_peered":"2024-08-17T17:26:10.266509+0000","last_clean":"2024-08-17T17:26:10.266509+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:26:10.266509+0000","last_undegraded":"2024-08-17T17:26:10.266509+0000","last_fullsized":"2024-08-17T17:26:10.266509+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":811,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.265181+0000","last_change":"2024-08-17T17:23:19.633786+0000","last_active":"2024-08-17T17:26:10.265181+0000","last_peered":"2024-08-17T17:26:10.265181+0000","last_clean":"2024-08-17T17:26:10.265181+0000","last_became_active":"2024-08-17T17:23:19.633230+0000","last_became_peered":"2024-08-17T17:23:19.633230+0000","last_unstale":"2024-08-17T17:26:10.265181+0000","last_undegraded":"2024-08-17T17:26:10.265181+0000","last_fullsized":"2024-08-17T17:26:10.265181+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:30:50.377391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":288,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.253979+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:26:10.253979+0000","last_peered":"2024-08-17T17:26:10.253979+0000","last_clean":"2024-08-17T17:26:10.253979+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:26:10.253979+0000","last_undegraded":"2024-08-17T17:26:10.253979+0000","last_fullsized":"2024-08-17T17:26:10.253979+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"208'6671","reported_seq":6709,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.853614+0000","last_change":"2024-08-17T17:24:14.639560+0000","last_active":"2024-08-17T17:26:08.853614+0000","last_peered":"2024-08-17T17:26:08.853614+0000","last_clean":"2024-08-17T17:26:08.853614+0000","last_became_active":"2024-08-17T17:24:14.639093+0000","last_became_peered":"2024-08-17T17:24:14.639093+0000","last_unstale":"2024-08-17T17:26:08.853614+0000","last_undegraded":"2024-08-17T17:26:08.853614+0000","last_fullsized":"2024-08-17T17:26:08.853614+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6671,"log_dups_size":0,"ondisk_log_size":6671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:45:57.940550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373882880,"num_objects":5705,"num_object_clones":0,"num_object_copies":17115,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6671,"num_write_kb":396032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":86,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.249124+0000","last_change":"2024-08-17T17:24:06.593660+0000","last_active":"2024-08-17T17:26:10.249124+0000","last_peered":"2024-08-17T17:26:10.249124+0000","last_clean":"2024-08-17T17:26:10.249124+0000","last_became_active":"2024-08-17T17:24:06.586215+0000","last_became_peered":"2024-08-17T17:24:06.586215+0000","last_unstale":"2024-08-17T17:26:10.249124+0000","last_undegraded":"2024-08-17T17:26:10.249124+0000","last_fullsized":"2024-08-17T17:26:10.249124+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:15:49.475395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":440,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.222210+0000","last_change":"2024-08-17T17:26:06.855307+0000","last_active":"2024-08-17T17:26:08.222210+0000","last_peered":"2024-08-17T17:26:08.222210+0000","last_clean":"2024-08-17T17:26:08.222210+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:26:08.222210+0000","last_undegraded":"2024-08-17T17:26:08.222210+0000","last_fullsized":"2024-08-17T17:26:08.222210+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":108,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.238874+0000","last_change":"2024-08-17T17:18:59.417183+0000","last_active":"2024-08-17T17:26:08.238874+0000","last_peered":"2024-08-17T17:26:08.238874+0000","last_clean":"2024-08-17T17:26:08.238874+0000","last_became_active":"2024-08-17T17:18:59.416843+0000","last_became_peered":"2024-08-17T17:18:59.416843+0000","last_unstale":"2024-08-17T17:26:08.238874+0000","last_undegraded":"2024-08-17T17:26:08.238874+0000","last_fullsized":"2024-08-17T17:26:08.238874+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:29:09.336335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":129,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.222092+0000","last_change":"2024-08-17T17:20:49.258048+0000","last_active":"2024-08-17T17:26:08.222092+0000","last_peered":"2024-08-17T17:26:08.222092+0000","last_clean":"2024-08-17T17:26:08.222092+0000","last_became_active":"2024-08-17T17:20:49.257672+0000","last_became_peered":"2024-08-17T17:20:49.257672+0000","last_unstale":"2024-08-17T17:26:08.222092+0000","last_undegraded":"2024-08-17T17:26:08.222092+0000","last_fullsized":"2024-08-17T17:26:08.222092+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:08:11.734316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":167,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-08-17T17:26:07.210331+0000","last_change":"2024-08-17T17:18:59.430969+0000","last_active":"2024-08-17T17:26:07.210331+0000","last_peered":"2024-08-17T17:26:07.210331+0000","last_clean":"2024-08-17T17:26:07.210331+0000","last_became_active":"2024-08-17T17:18:59.413475+0000","last_became_peered":"2024-08-17T17:18:59.413475+0000","last_unstale":"2024-08-17T17:26:07.210331+0000","last_undegraded":"2024-08-17T17:26:07.210331+0000","last_fullsized":"2024-08-17T17:26:07.210331+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:36:33.732645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":283,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.266246+0000","last_change":"2024-08-17T17:25:04.506413+0000","last_active":"2024-08-17T17:26:10.266246+0000","last_peered":"2024-08-17T17:26:10.266246+0000","last_clean":"2024-08-17T17:26:10.266246+0000","last_became_active":"2024-08-17T17:25:04.506104+0000","last_became_peered":"2024-08-17T17:25:04.506104+0000","last_unstale":"2024-08-17T17:26:10.266246+0000","last_undegraded":"2024-08-17T17:26:10.266246+0000","last_fullsized":"2024-08-17T17:26:10.266246+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:41:53.556223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":110,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-08-17T17:26:07.213789+0000","last_change":"2024-08-17T17:20:49.279458+0000","last_active":"2024-08-17T17:26:07.213789+0000","last_peered":"2024-08-17T17:26:07.213789+0000","last_clean":"2024-08-17T17:26:07.213789+0000","last_became_active":"2024-08-17T17:20:49.260790+0000","last_became_peered":"2024-08-17T17:20:49.260790+0000","last_unstale":"2024-08-17T17:26:07.213789+0000","last_undegraded":"2024-08-17T17:26:07.213789+0000","last_fullsized":"2024-08-17T17:26:07.213789+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:58.092945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.239606+0000","last_change":"2024-08-17T17:24:53.463713+0000","last_active":"2024-08-17T17:26:08.239606+0000","last_peered":"2024-08-17T17:26:08.239606+0000","last_clean":"2024-08-17T17:26:08.239606+0000","last_became_active":"2024-08-17T17:24:53.463419+0000","last_became_peered":"2024-08-17T17:24:53.463419+0000","last_unstale":"2024-08-17T17:26:08.239606+0000","last_undegraded":"2024-08-17T17:26:08.239606+0000","last_fullsized":"2024-08-17T17:26:08.239606+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:34:44.545175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":238,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.255487+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:26:10.255487+0000","last_peered":"2024-08-17T17:26:10.255487+0000","last_clean":"2024-08-17T17:26:10.255487+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:26:10.255487+0000","last_undegraded":"2024-08-17T17:26:10.255487+0000","last_fullsized":"2024-08-17T17:26:10.255487+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":266,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.249522+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:26:10.249522+0000","last_peered":"2024-08-17T17:26:10.249522+0000","last_clean":"2024-08-17T17:26:10.249522+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:26:10.249522+0000","last_undegraded":"2024-08-17T17:26:10.249522+0000","last_fullsized":"2024-08-17T17:26:10.249522+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":285,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.253895+0000","last_change":"2024-08-17T17:25:04.509422+0000","last_active":"2024-08-17T17:26:10.253895+0000","last_peered":"2024-08-17T17:26:10.253895+0000","last_clean":"2024-08-17T17:26:10.253895+0000","last_became_active":"2024-08-17T17:25:04.508742+0000","last_became_peered":"2024-08-17T17:25:04.508742+0000","last_unstale":"2024-08-17T17:26:10.253895+0000","last_undegraded":"2024-08-17T17:26:10.253895+0000","last_fullsized":"2024-08-17T17:26:10.253895+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:18:31.976907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":166,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.222120+0000","last_change":"2024-08-17T17:16:26.963696+0000","last_active":"2024-08-17T17:26:08.222120+0000","last_peered":"2024-08-17T17:26:08.222120+0000","last_clean":"2024-08-17T17:26:08.222120+0000","last_became_active":"2024-08-17T17:16:26.963280+0000","last_became_peered":"2024-08-17T17:16:26.963280+0000","last_unstale":"2024-08-17T17:26:08.222120+0000","last_undegraded":"2024-08-17T17:26:08.222120+0000","last_fullsized":"2024-08-17T17:26:08.222120+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:22:01.474840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":303,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.254096+0000","last_change":"2024-08-17T17:24:06.586886+0000","last_active":"2024-08-17T17:26:10.254096+0000","last_peered":"2024-08-17T17:26:10.254096+0000","last_clean":"2024-08-17T17:26:10.254096+0000","last_became_active":"2024-08-17T17:24:06.586516+0000","last_became_peered":"2024-08-17T17:24:06.586516+0000","last_unstale":"2024-08-17T17:26:10.254096+0000","last_undegraded":"2024-08-17T17:26:10.254096+0000","last_fullsized":"2024-08-17T17:26:10.254096+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:03:30.002726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":83,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.248969+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:26:10.248969+0000","last_peered":"2024-08-17T17:26:10.248969+0000","last_clean":"2024-08-17T17:26:10.248969+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:26:10.248969+0000","last_undegraded":"2024-08-17T17:26:10.248969+0000","last_fullsized":"2024-08-17T17:26:10.248969+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":263,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.265978+0000","last_change":"2024-08-17T17:18:59.412463+0000","last_active":"2024-08-17T17:26:10.265978+0000","last_peered":"2024-08-17T17:26:10.265978+0000","last_clean":"2024-08-17T17:26:10.265978+0000","last_became_active":"2024-08-17T17:18:59.412134+0000","last_became_peered":"2024-08-17T17:18:59.412134+0000","last_unstale":"2024-08-17T17:26:10.265978+0000","last_undegraded":"2024-08-17T17:26:10.265978+0000","last_fullsized":"2024-08-17T17:26:10.265978+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:44.463153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":306,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.256520+0000","last_change":"2024-08-17T17:25:04.510854+0000","last_active":"2024-08-17T17:26:10.256520+0000","last_peered":"2024-08-17T17:26:10.256520+0000","last_clean":"2024-08-17T17:26:10.256520+0000","last_became_active":"2024-08-17T17:25:04.510462+0000","last_became_peered":"2024-08-17T17:25:04.510462+0000","last_unstale":"2024-08-17T17:26:10.256520+0000","last_undegraded":"2024-08-17T17:26:10.256520+0000","last_fullsized":"2024-08-17T17:26:10.256520+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:55:54.762086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":203,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.266522+0000","last_change":"2024-08-17T17:24:06.591165+0000","last_active":"2024-08-17T17:26:10.266522+0000","last_peered":"2024-08-17T17:26:10.266522+0000","last_clean":"2024-08-17T17:26:10.266522+0000","last_became_active":"2024-08-17T17:24:06.590785+0000","last_became_peered":"2024-08-17T17:24:06.590785+0000","last_unstale":"2024-08-17T17:26:10.266522+0000","last_undegraded":"2024-08-17T17:26:10.266522+0000","last_fullsized":"2024-08-17T17:26:10.266522+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:54:05.775563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":275,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.267588+0000","last_change":"2024-08-17T17:24:32.300560+0000","last_active":"2024-08-17T17:26:10.267588+0000","last_peered":"2024-08-17T17:26:10.267588+0000","last_clean":"2024-08-17T17:26:10.267588+0000","last_became_active":"2024-08-17T17:24:32.300228+0000","last_became_peered":"2024-08-17T17:24:32.300228+0000","last_unstale":"2024-08-17T17:26:10.267588+0000","last_undegraded":"2024-08-17T17:26:10.267588+0000","last_fullsized":"2024-08-17T17:26:10.267588+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:18:05.475449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":82,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.248425+0000","last_change":"2024-08-17T17:24:06.586379+0000","last_active":"2024-08-17T17:26:10.248425+0000","last_peered":"2024-08-17T17:26:10.248425+0000","last_clean":"2024-08-17T17:26:10.248425+0000","last_became_active":"2024-08-17T17:24:06.586023+0000","last_became_peered":"2024-08-17T17:24:06.586023+0000","last_unstale":"2024-08-17T17:26:10.248425+0000","last_undegraded":"2024-08-17T17:26:10.248425+0000","last_fullsized":"2024-08-17T17:26:10.248425+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:45.701637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":292,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.255885+0000","last_change":"2024-08-17T17:21:50.442535+0000","last_active":"2024-08-17T17:26:10.255885+0000","last_peered":"2024-08-17T17:26:10.255885+0000","last_clean":"2024-08-17T17:26:10.255885+0000","last_became_active":"2024-08-17T17:21:50.441891+0000","last_became_peered":"2024-08-17T17:21:50.441891+0000","last_unstale":"2024-08-17T17:26:10.255885+0000","last_undegraded":"2024-08-17T17:26:10.255885+0000","last_fullsized":"2024-08-17T17:26:10.255885+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:46:24.025407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":221,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.239639+0000","last_change":"2024-08-17T17:25:04.508045+0000","last_active":"2024-08-17T17:26:08.239639+0000","last_peered":"2024-08-17T17:26:08.239639+0000","last_clean":"2024-08-17T17:26:08.239639+0000","last_became_active":"2024-08-17T17:25:04.505083+0000","last_became_peered":"2024-08-17T17:25:04.505083+0000","last_unstale":"2024-08-17T17:26:08.239639+0000","last_undegraded":"2024-08-17T17:26:08.239639+0000","last_fullsized":"2024-08-17T17:26:08.239639+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:23:42.993544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":278,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.255938+0000","last_change":"2024-08-17T17:18:59.418434+0000","last_active":"2024-08-17T17:26:10.255938+0000","last_peered":"2024-08-17T17:26:10.255938+0000","last_clean":"2024-08-17T17:26:10.255938+0000","last_became_active":"2024-08-17T17:18:59.418043+0000","last_became_peered":"2024-08-17T17:18:59.418043+0000","last_unstale":"2024-08-17T17:26:10.255938+0000","last_undegraded":"2024-08-17T17:26:10.255938+0000","last_fullsized":"2024-08-17T17:26:10.255938+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:25:51.371820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":194,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.239139+0000","last_change":"2024-08-17T17:20:49.291167+0000","last_active":"2024-08-17T17:26:08.239139+0000","last_peered":"2024-08-17T17:26:08.239139+0000","last_clean":"2024-08-17T17:26:08.239139+0000","last_became_active":"2024-08-17T17:20:49.263750+0000","last_became_peered":"2024-08-17T17:20:49.263750+0000","last_unstale":"2024-08-17T17:26:08.239139+0000","last_undegraded":"2024-08-17T17:26:08.239139+0000","last_fullsized":"2024-08-17T17:26:08.239139+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:02.863246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":292,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.255873+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:26:10.255873+0000","last_peered":"2024-08-17T17:26:10.255873+0000","last_clean":"2024-08-17T17:26:10.255873+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:26:10.255873+0000","last_undegraded":"2024-08-17T17:26:10.255873+0000","last_fullsized":"2024-08-17T17:26:10.255873+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":292,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.256577+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:26:10.256577+0000","last_peered":"2024-08-17T17:26:10.256577+0000","last_clean":"2024-08-17T17:26:10.256577+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:26:10.256577+0000","last_undegraded":"2024-08-17T17:26:10.256577+0000","last_fullsized":"2024-08-17T17:26:10.256577+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.221701+0000","last_change":"2024-08-17T17:25:04.503834+0000","last_active":"2024-08-17T17:26:08.221701+0000","last_peered":"2024-08-17T17:26:08.221701+0000","last_clean":"2024-08-17T17:26:08.221701+0000","last_became_active":"2024-08-17T17:25:04.502712+0000","last_became_peered":"2024-08-17T17:25:04.502712+0000","last_unstale":"2024-08-17T17:26:08.221701+0000","last_undegraded":"2024-08-17T17:26:08.221701+0000","last_fullsized":"2024-08-17T17:26:08.221701+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:45.389103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":102,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.249475+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:26:10.249475+0000","last_peered":"2024-08-17T17:26:10.249475+0000","last_clean":"2024-08-17T17:26:10.249475+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:26:10.249475+0000","last_undegraded":"2024-08-17T17:26:10.249475+0000","last_fullsized":"2024-08-17T17:26:10.249475+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":5834932248,"num_objects":89035,"num_object_clones":0,"num_object_copies":267105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13611,"num_objects_misplaced":2554,"num_objects_unfound":0,"num_objects_dirty":89035,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108259,"num_write_kb":6313345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6671,"num_bytes_recovered":437190656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17985122304,"data_stored":17985110088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105059,"ondisk_log_size":105059,"up":48,"acting":43,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":6304,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":16702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17971,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16789504,"data_stored":16777273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234232,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3080180,"kb_used_data":2766184,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":90664972,"statfs":{"total":95995035648,"available":92840931328,"internally_reserved":0,"allocated":2832572416,"data_stored":2831379361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495868,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2628792,"kb_used_data":2323120,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91116360,"statfs":{"total":95995035648,"available":93303152640,"internally_reserved":0,"allocated":2378874880,"data_stored":2377681821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757505,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2095088,"kb_used_data":1797736,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":91650064,"statfs":{"total":95995035648,"available":93849665536,"internally_reserved":0,"allocated":1840881664,"data_stored":1839692276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019143,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435370,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3847724,"kb_used_data":3576992,"kb_used_omap":4,"kb_used_meta":270523,"kb_avail":89897428,"statfs":{"total":95995035648,"available":92054966272,"internally_reserved":0,"allocated":3662839808,"data_stored":3661658362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":277015562},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":104,"seq":446676598896,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3608176,"kb_used_data":3295072,"kb_used_omap":7,"kb_used_meta":313080,"kb_avail":90136976,"statfs":{"total":95995035648,"available":92300263424,"internally_reserved":0,"allocated":3374153728,"data_stored":3372969060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":320594082},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762423,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2833380,"kb_used_data":2507288,"kb_used_omap":7,"kb_used_meta":326008,"kb_avail":90911772,"statfs":{"total":95995035648,"available":93093654528,"internally_reserved":0,"allocated":2567462912,"data_stored":2566281161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7945,"internal_metadata":333832439},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":94,"seq":403726925950,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1686776,"kb_used_data":1412592,"kb_used_omap":7,"kb_used_meta":274168,"kb_avail":92058376,"statfs":{"total":95995035648,"available":94267777024,"internally_reserved":0,"allocated":1446494208,"data_stored":1445297554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":280748353},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6241,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1418334208,"data_stored":1418330136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2550665216,"data_stored":2550661144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3341680640,"data_stored":3341680640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3644588032,"data_stored":3644588032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1836453888,"data_stored":1836449816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2368929792,"data_stored":2368929792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2824470528,"data_stored":2824470528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:26:16.010 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:26:16.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:16.092 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:16.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:16.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:16.194 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:16.226 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:16.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:16.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:16.327 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:16.355 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:16.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:16.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:16.456 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:16.488 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:16.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:16.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:16.589 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:16.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:16.636 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:16.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:16.737 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:16.765 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:16.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:16.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:16.826 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:26:16.826 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:26:16.864 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:26:16.864 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:26:16.866 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:16.908 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:26:16.909 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:26:16.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:16.923 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:16.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:16.954 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:26:16.954 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:26:16.990 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:26:16.991 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:26:17.024 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:17.025 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:26:17.025 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:26:17.053 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:17.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:17.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:17.058 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:26:17.058 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:26:17.092 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:26:17.093 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:26:17.155 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:17.183 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:17.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:17.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:17.285 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:17.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:17.335 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:17.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:17.437 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:17.468 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:17.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:17.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:17.492 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:17 smithi003 ceph-mon[103344]: pgmap v531: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 795 op/s; 13611/262815 objects degraded (5.179%); 2216/262815 objects misplaced (0.843%); 6.3 MiB/s, 93 objects/s recovering 2024-08-17T17:26:17.493 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:17 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/4209074226' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:26:17.493 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:17 smithi003 ceph-mon[100615]: pgmap v531: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 795 op/s; 13611/262815 objects degraded (5.179%); 2216/262815 objects misplaced (0.843%); 6.3 MiB/s, 93 objects/s recovering 2024-08-17T17:26:17.493 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:17 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/4209074226' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:26:17.570 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:17.599 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:17.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:17.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:17.701 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:17.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:17 smithi053 ceph-mon[89963]: pgmap v531: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 795 op/s; 13611/262815 objects degraded (5.179%); 2216/262815 objects misplaced (0.843%); 6.3 MiB/s, 93 objects/s recovering 2024-08-17T17:26:17.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:17 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/4209074226' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:26:17.739 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:17.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:17.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:17.841 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:17.872 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:17.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:17.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:17.974 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:18.023 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:18.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:18.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:18.125 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:18.158 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:18.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:18.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:18.180 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:26:18.260 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:18.310 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:18.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:18.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:18.412 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:18.457 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:18.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:18.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:18.559 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:18.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:18.596 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:18.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:18.697 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:18.734 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:18.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:18.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:18.836 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:18.872 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:18.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:18.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:18.974 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:18.978 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:26:19.004 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:19.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:19.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:19.106 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:19.135 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:19.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:19.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:19.237 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:19.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:19 smithi003 ceph-mon[100615]: pgmap v532: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.3 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 704 op/s; 13127/259191 objects degraded (5.065%); 2216/259191 objects misplaced (0.855%); 6.8 MiB/s, 102 objects/s recovering 2024-08-17T17:26:19.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:19 smithi003 ceph-mon[103344]: pgmap v532: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.3 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 704 op/s; 13127/259191 objects degraded (5.065%); 2216/259191 objects misplaced (0.855%); 6.8 MiB/s, 102 objects/s recovering 2024-08-17T17:26:19.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:19.285 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:19.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:19.386 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:19.420 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:19.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:19.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:19.523 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:19.563 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:19.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:19.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:19.665 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:19.699 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:19.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:19.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:19.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:19 smithi053 ceph-mon[89963]: pgmap v532: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.3 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 704 op/s; 13127/259191 objects degraded (5.065%); 2216/259191 objects misplaced (0.855%); 6.8 MiB/s, 102 objects/s recovering 2024-08-17T17:26:19.801 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:19.833 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:19.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:19.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:19.935 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:19.975 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:19.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:19.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:20.077 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:20.132 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:20.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:20.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:20.182 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:26:20.234 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:20.263 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:20.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:20.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:20.322 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:26:20.322 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:26:20.365 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:20.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:20.419 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:20.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:20.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:20 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 13127/259191 objects degraded (5.065%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:20.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:26:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:26:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:26:20.520 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:20.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:20 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 13127/259191 objects degraded (5.065%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:20.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:20 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 13127/259191 objects degraded (5.065%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:20.552 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:20.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:20.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:20.654 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:20.685 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:20.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:20.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:20.787 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:20.821 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:20.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:20.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:20.923 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:20.967 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:20.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:20.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:21.069 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:21.101 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:21.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:21.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:21.203 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:21.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:21.251 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:21.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:21.352 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:21.381 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:21.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:21.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:21.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:21.499 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:21 smithi003 ceph-mon[103344]: pgmap v533: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 758 op/s; 13127/253416 objects degraded (5.180%); 1836/253416 objects misplaced (0.725%); 7.5 MiB/s, 119 objects/s recovering 2024-08-17T17:26:21.499 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:21 smithi003 ceph-mon[100615]: pgmap v533: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 758 op/s; 13127/253416 objects degraded (5.180%); 1836/253416 objects misplaced (0.725%); 7.5 MiB/s, 119 objects/s recovering 2024-08-17T17:26:21.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:21.523 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:21.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:21.624 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:21.660 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:21.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:21.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:21.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:21 smithi053 ceph-mon[89963]: pgmap v533: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 758 op/s; 13127/253416 objects degraded (5.180%); 1836/253416 objects misplaced (0.725%); 7.5 MiB/s, 119 objects/s recovering 2024-08-17T17:26:21.762 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:21.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:21.800 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:21.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:21.901 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:21.934 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:21.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:21.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:21.962 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:26:21.963 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-17T17:26:22.035 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:22.077 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:22.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:22.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:22.179 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:22.242 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:22.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:22.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:22.344 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:22.373 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:22.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:22.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:22.475 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:22.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:22.505 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:22.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:22.606 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:22.634 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:22.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:22.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:22.736 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:22.763 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:22.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:22.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:22.865 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:22.898 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:22.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:22.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:23.000 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:23.029 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:23.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:23.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:23.131 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:23.179 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:23.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:23.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:23.281 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:23.288 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-08-17T17:26:23.311 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:23.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:23.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:23.413 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:23.448 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:23.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:23.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:23.550 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:23.587 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:23.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:23.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:23.628 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:23 smithi003 ceph-mon[100615]: pgmap v534: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 822 op/s; 12635/247431 objects degraded (5.106%); 1836/247431 objects misplaced (0.742%); 7.8 MiB/s, 124 objects/s recovering 2024-08-17T17:26:23.628 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:26:23.629 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:23 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/4006815540' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:26:23.629 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:23 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:26:23.629 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:23 smithi003 ceph-mon[103344]: pgmap v534: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 822 op/s; 12635/247431 objects degraded (5.106%); 1836/247431 objects misplaced (0.742%); 7.8 MiB/s, 124 objects/s recovering 2024-08-17T17:26:23.629 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:26:23.629 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:23 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/4006815540' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:26:23.629 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:23 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:26:23.689 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:23 smithi053 ceph-mon[89963]: pgmap v534: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 822 op/s; 12635/247431 objects degraded (5.106%); 1836/247431 objects misplaced (0.742%); 7.8 MiB/s, 124 objects/s recovering 2024-08-17T17:26:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:26:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:23 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/4006815540' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:26:23.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:23 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:26:23.736 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:23.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:23.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:23.838 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:23.879 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:23.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:23.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:23.923 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:26:23.981 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:24.012 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:24.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:24.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:24.114 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:24.151 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:24.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:24.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:24.253 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:24.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:24.292 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:24.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:24.393 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:24.474 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:24.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:24.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:24.576 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:24.615 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:24.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:24.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:24.658 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:24 smithi003 ceph-mon[103344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:26:24.658 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:24 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:26:24.658 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:24 smithi003 ceph-mon[103344]: osdmap e210: 8 total, 8 up, 7 in 2024-08-17T17:26:24.658 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:24 smithi003 ceph-mon[100615]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:26:24.659 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:24 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:26:24.659 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:24 smithi003 ceph-mon[100615]: osdmap e210: 8 total, 8 up, 7 in 2024-08-17T17:26:24.716 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:24.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:24 smithi053 ceph-mon[89963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:26:24.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:24 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:26:24.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:24 smithi053 ceph-mon[89963]: osdmap e210: 8 total, 8 up, 7 in 2024-08-17T17:26:24.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:24.768 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:24.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:24.869 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:24.900 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:24.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:24.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:25.002 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:25.038 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:25.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:25.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:25.140 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:25.180 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:25.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:25.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:25.281 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:25.317 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:25.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:25.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:25.419 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:25.452 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:25.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:25.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:25.553 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:25.594 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:25.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:25.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:25.651 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:25 smithi003 ceph-mon[103344]: pgmap v536: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 784 op/s; 12635/243627 objects degraded (5.186%); 1836/243627 objects misplaced (0.754%); 7.5 MiB/s, 120 objects/s recovering 2024-08-17T17:26:25.651 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:25 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 12635/243627 objects degraded (5.186%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:25.651 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:25 smithi003 ceph-mon[100615]: pgmap v536: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 784 op/s; 12635/243627 objects degraded (5.186%); 1836/243627 objects misplaced (0.754%); 7.5 MiB/s, 120 objects/s recovering 2024-08-17T17:26:25.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:25 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 12635/243627 objects degraded (5.186%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:25.696 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:25.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:25 smithi053 ceph-mon[89963]: pgmap v536: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 784 op/s; 12635/243627 objects degraded (5.186%); 1836/243627 objects misplaced (0.754%); 7.5 MiB/s, 120 objects/s recovering 2024-08-17T17:26:25.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:25 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 12635/243627 objects degraded (5.186%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:25.725 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:25.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:25.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:25.827 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:25.856 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:25.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:25.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:25.944 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:26:25.945 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:26:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:26:25] "GET /metrics HTTP/1.1" 200 36526 "" "Prometheus/2.51.0" 2024-08-17T17:26:25.948 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:26:25.958 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:26.004 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:26.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:26.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:26.106 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:26.143 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:26.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:26.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:26.245 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:26.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:26.291 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:26.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:26.392 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:26.395 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:26:26.428 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:26.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:26.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:26.530 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:26.564 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:26.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:26.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:26.632 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:26:26.666 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:26.671 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":536,"stamp":"2024-08-17T17:26:23.888520+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5325002725,"num_objects":81209,"num_object_clones":0,"num_object_copies":243627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12635,"num_objects_misplaced":1836,"num_objects_unfound":0,"num_objects_dirty":81209,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":117235,"num_write_kb":6328384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8035,"num_bytes_recovered":528164230,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110094,"ondisk_log_size":110094,"up":267,"acting":262,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":280,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":18458456,"kb_used_data":16330904,"kb_used_omap":33,"kb_used_meta":2127134,"kb_avail":637757608,"statfs":{"total":671965249536,"available":653063790592,"internally_reserved":0,"allocated":16722845696,"data_stored":16714486195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34068,"internal_metadata":2178185964},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-514719744,"num_objects":-7854,"num_object_clones":0,"num_object_copies":-23562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-976,"num_objects_misplaced":-718,"num_objects_unfound":0,"num_objects_dirty":-7854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7854,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1206,"num_bytes_recovered":79036416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.006637"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":271,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.260618+0000","last_change":"2024-08-17T17:17:32.495652+0000","last_active":"2024-08-17T17:26:10.260618+0000","last_peered":"2024-08-17T17:26:10.260618+0000","last_clean":"2024-08-17T17:26:10.260618+0000","last_became_active":"2024-08-17T17:17:32.495218+0000","last_became_peered":"2024-08-17T17:17:32.495218+0000","last_unstale":"2024-08-17T17:26:10.260618+0000","last_undegraded":"2024-08-17T17:26:10.260618+0000","last_fullsized":"2024-08-17T17:26:10.260618+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:11:45.821282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":185,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.224852+0000","last_change":"2024-08-17T17:18:59.424887+0000","last_active":"2024-08-17T17:26:08.224852+0000","last_peered":"2024-08-17T17:26:08.224852+0000","last_clean":"2024-08-17T17:26:08.224852+0000","last_became_active":"2024-08-17T17:18:59.411539+0000","last_became_peered":"2024-08-17T17:18:59.411539+0000","last_unstale":"2024-08-17T17:26:08.224852+0000","last_undegraded":"2024-08-17T17:26:08.224852+0000","last_fullsized":"2024-08-17T17:26:08.224852+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:46:20.082241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":26,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.254733+0000","last_change":"2024-08-17T17:24:53.460742+0000","last_active":"2024-08-17T17:26:10.254733+0000","last_peered":"2024-08-17T17:26:10.254733+0000","last_clean":"2024-08-17T17:26:10.254733+0000","last_became_active":"2024-08-17T17:24:53.460365+0000","last_became_peered":"2024-08-17T17:24:53.460365+0000","last_unstale":"2024-08-17T17:26:10.254733+0000","last_undegraded":"2024-08-17T17:26:10.254733+0000","last_fullsized":"2024-08-17T17:26:10.254733+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:24:08.116968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":82,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.259259+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:26:10.259259+0000","last_peered":"2024-08-17T17:26:10.259259+0000","last_clean":"2024-08-17T17:26:10.259259+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:26:10.259259+0000","last_undegraded":"2024-08-17T17:26:10.259259+0000","last_fullsized":"2024-08-17T17:26:10.259259+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":82,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.249047+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:26:10.249047+0000","last_peered":"2024-08-17T17:26:10.249047+0000","last_clean":"2024-08-17T17:26:10.249047+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:26:10.249047+0000","last_undegraded":"2024-08-17T17:26:10.249047+0000","last_fullsized":"2024-08-17T17:26:10.249047+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":160,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.224526+0000","last_change":"2024-08-17T17:20:49.258206+0000","last_active":"2024-08-17T17:26:08.224526+0000","last_peered":"2024-08-17T17:26:08.224526+0000","last_clean":"2024-08-17T17:26:08.224526+0000","last_became_active":"2024-08-17T17:20:49.257761+0000","last_became_peered":"2024-08-17T17:20:49.257761+0000","last_unstale":"2024-08-17T17:26:08.224526+0000","last_undegraded":"2024-08-17T17:26:08.224526+0000","last_fullsized":"2024-08-17T17:26:08.224526+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:38:35.363748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":285,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.266253+0000","last_change":"2024-08-17T17:19:57.514773+0000","last_active":"2024-08-17T17:26:10.266253+0000","last_peered":"2024-08-17T17:26:10.266253+0000","last_clean":"2024-08-17T17:26:10.266253+0000","last_became_active":"2024-08-17T17:19:57.513839+0000","last_became_peered":"2024-08-17T17:19:57.513839+0000","last_unstale":"2024-08-17T17:26:10.266253+0000","last_undegraded":"2024-08-17T17:26:10.266253+0000","last_fullsized":"2024-08-17T17:26:10.266253+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:42:48.063169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":256,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.241892+0000","last_change":"2024-08-17T17:24:53.465875+0000","last_active":"2024-08-17T17:26:10.241892+0000","last_peered":"2024-08-17T17:26:10.241892+0000","last_clean":"2024-08-17T17:26:10.241892+0000","last_became_active":"2024-08-17T17:24:53.458129+0000","last_became_peered":"2024-08-17T17:24:53.458129+0000","last_unstale":"2024-08-17T17:26:10.241892+0000","last_undegraded":"2024-08-17T17:26:10.241892+0000","last_fullsized":"2024-08-17T17:26:10.241892+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:51:33.584869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"209'7346","reported_seq":7403,"reported_epoch":209,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:26:22.973085+0000","last_change":"2024-08-17T17:24:54.892004+0000","last_active":"2024-08-17T17:26:22.973085+0000","last_peered":"2024-08-17T17:26:22.973085+0000","last_clean":"2024-08-17T17:24:52.438492+0000","last_became_active":"2024-08-17T17:24:54.887269+0000","last_became_peered":"2024-08-17T17:24:54.887269+0000","last_unstale":"2024-08-17T17:26:22.973085+0000","last_undegraded":"2024-08-17T17:24:54.477707+0000","last_fullsized":"2024-08-17T17:24:54.461071+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":7346,"log_dups_size":0,"ondisk_log_size":7346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321388544,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7346,"num_write_kb":392000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[0,6],"avail_no_missing":["0","2","6"],"object_location_counts":[{"shards":"0,2,6","objects":1239},{"shards":"0,6","objects":2444}],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":121,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.225127+0000","last_change":"2024-08-17T17:24:06.597382+0000","last_active":"2024-08-17T17:26:08.225127+0000","last_peered":"2024-08-17T17:26:08.225127+0000","last_clean":"2024-08-17T17:26:08.225127+0000","last_became_active":"2024-08-17T17:24:06.588881+0000","last_became_peered":"2024-08-17T17:24:06.588881+0000","last_unstale":"2024-08-17T17:26:08.225127+0000","last_undegraded":"2024-08-17T17:26:08.225127+0000","last_fullsized":"2024-08-17T17:26:08.225127+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:27.240751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":36,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.254475+0000","last_change":"2024-08-17T17:25:04.503780+0000","last_active":"2024-08-17T17:26:10.254475+0000","last_peered":"2024-08-17T17:26:10.254475+0000","last_clean":"2024-08-17T17:26:10.254475+0000","last_became_active":"2024-08-17T17:25:04.503503+0000","last_became_peered":"2024-08-17T17:25:04.503503+0000","last_unstale":"2024-08-17T17:26:10.254475+0000","last_undegraded":"2024-08-17T17:26:10.254475+0000","last_fullsized":"2024-08-17T17:26:10.254475+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:04:48.171216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"209'7241","reported_seq":7314,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:21.568527+0000","last_change":"2024-08-17T17:24:14.639361+0000","last_active":"2024-08-17T17:26:21.568527+0000","last_peered":"2024-08-17T17:26:21.568527+0000","last_clean":"2024-08-17T17:26:21.568527+0000","last_became_active":"2024-08-17T17:24:14.639074+0000","last_became_peered":"2024-08-17T17:24:14.639074+0000","last_unstale":"2024-08-17T17:26:21.568527+0000","last_undegraded":"2024-08-17T17:26:21.568527+0000","last_fullsized":"2024-08-17T17:26:21.568527+0000","mapping_epoch":178,"log_start":"186'800","ondisk_log_start":"186'800","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6441,"log_dups_size":0,"ondisk_log_size":6441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:15:05.377992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335740928,"num_objects":5123,"num_object_clones":0,"num_object_copies":15369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7241,"num_write_kb":395648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":107,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.225111+0000","last_change":"2024-08-17T17:24:06.599023+0000","last_active":"2024-08-17T17:26:08.225111+0000","last_peered":"2024-08-17T17:26:08.225111+0000","last_clean":"2024-08-17T17:26:08.225111+0000","last_became_active":"2024-08-17T17:24:06.588954+0000","last_became_peered":"2024-08-17T17:24:06.588954+0000","last_unstale":"2024-08-17T17:26:08.225111+0000","last_undegraded":"2024-08-17T17:26:08.225111+0000","last_fullsized":"2024-08-17T17:26:08.225111+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:20:51.445576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":226,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.265890+0000","last_change":"2024-08-17T17:20:49.271540+0000","last_active":"2024-08-17T17:26:10.265890+0000","last_peered":"2024-08-17T17:26:10.265890+0000","last_clean":"2024-08-17T17:26:10.265890+0000","last_became_active":"2024-08-17T17:20:49.271130+0000","last_became_peered":"2024-08-17T17:20:49.271130+0000","last_unstale":"2024-08-17T17:26:10.265890+0000","last_undegraded":"2024-08-17T17:26:10.265890+0000","last_fullsized":"2024-08-17T17:26:10.265890+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:05.386705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"209'7095","reported_seq":7134,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:18.991109+0000","last_change":"2024-08-17T17:24:14.644070+0000","last_active":"2024-08-17T17:26:18.991109+0000","last_peered":"2024-08-17T17:26:18.991109+0000","last_clean":"2024-08-17T17:26:18.991109+0000","last_became_active":"2024-08-17T17:24:14.643799+0000","last_became_peered":"2024-08-17T17:24:14.643799+0000","last_unstale":"2024-08-17T17:26:18.991109+0000","last_undegraded":"2024-08-17T17:26:18.991109+0000","last_fullsized":"2024-08-17T17:26:18.991109+0000","mapping_epoch":178,"log_start":"183'100","ondisk_log_start":"183'100","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6995,"log_dups_size":0,"ondisk_log_size":6995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:25:38.445770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338624512,"num_objects":5167,"num_object_clones":0,"num_object_copies":15501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7095,"num_write_kb":392384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":179,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.227372+0000","last_change":"2024-08-17T17:24:53.467526+0000","last_active":"2024-08-17T17:26:08.227372+0000","last_peered":"2024-08-17T17:26:08.227372+0000","last_clean":"2024-08-17T17:26:08.227372+0000","last_became_active":"2024-08-17T17:24:53.460631+0000","last_became_peered":"2024-08-17T17:24:53.460631+0000","last_unstale":"2024-08-17T17:26:08.227372+0000","last_undegraded":"2024-08-17T17:26:08.227372+0000","last_fullsized":"2024-08-17T17:26:08.227372+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:05:24.938831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.242439+0000","last_change":"2024-08-17T17:25:04.507176+0000","last_active":"2024-08-17T17:26:10.242439+0000","last_peered":"2024-08-17T17:26:10.242439+0000","last_clean":"2024-08-17T17:26:10.242439+0000","last_became_active":"2024-08-17T17:25:04.506788+0000","last_became_peered":"2024-08-17T17:25:04.506788+0000","last_unstale":"2024-08-17T17:26:10.242439+0000","last_undegraded":"2024-08-17T17:26:10.242439+0000","last_fullsized":"2024-08-17T17:26:10.242439+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:22:04.658216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"209'7137","reported_seq":7210,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:22.945906+0000","last_change":"2024-08-17T17:24:14.642637+0000","last_active":"2024-08-17T17:26:22.945906+0000","last_peered":"2024-08-17T17:26:22.945906+0000","last_clean":"2024-08-17T17:26:22.945906+0000","last_became_active":"2024-08-17T17:24:14.642347+0000","last_became_peered":"2024-08-17T17:24:14.642347+0000","last_unstale":"2024-08-17T17:26:22.945906+0000","last_undegraded":"2024-08-17T17:26:22.945906+0000","last_fullsized":"2024-08-17T17:26:22.945906+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":7137,"log_dups_size":0,"ondisk_log_size":7137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:37:44.631335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320667648,"num_objects":4893,"num_object_clones":0,"num_object_copies":14679,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4893,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7137,"num_write_kb":384960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":168,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.254854+0000","last_change":"2024-08-17T17:17:32.495428+0000","last_active":"2024-08-17T17:26:10.254854+0000","last_peered":"2024-08-17T17:26:10.254854+0000","last_clean":"2024-08-17T17:26:10.254854+0000","last_became_active":"2024-08-17T17:17:32.494978+0000","last_became_peered":"2024-08-17T17:17:32.494978+0000","last_unstale":"2024-08-17T17:26:10.254854+0000","last_undegraded":"2024-08-17T17:26:10.254854+0000","last_fullsized":"2024-08-17T17:26:10.254854+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:42:24.647812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":274,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.248687+0000","last_change":"2024-08-17T17:20:49.272126+0000","last_active":"2024-08-17T17:26:10.248687+0000","last_peered":"2024-08-17T17:26:10.248687+0000","last_clean":"2024-08-17T17:26:10.248687+0000","last_became_active":"2024-08-17T17:20:49.271185+0000","last_became_peered":"2024-08-17T17:20:49.271185+0000","last_unstale":"2024-08-17T17:26:10.248687+0000","last_undegraded":"2024-08-17T17:26:10.248687+0000","last_fullsized":"2024-08-17T17:26:10.248687+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:43:58.184939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"209'7453","reported_seq":8401,"reported_epoch":209,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:26:20.236117+0000","last_change":"2024-08-17T17:24:54.898729+0000","last_active":"2024-08-17T17:26:20.236117+0000","last_peered":"2024-08-17T17:26:20.236117+0000","last_clean":"2024-08-17T17:24:52.435411+0000","last_became_active":"2024-08-17T17:24:54.889545+0000","last_became_peered":"2024-08-17T17:24:54.889545+0000","last_unstale":"2024-08-17T17:26:20.236117+0000","last_undegraded":"2024-08-17T17:24:54.482592+0000","last_fullsized":"2024-08-17T17:24:54.459529+0000","mapping_epoch":194,"log_start":"189'1200","ondisk_log_start":"189'1200","created":178,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6253,"log_dups_size":0,"ondisk_log_size":6253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347406336,"num_objects":5301,"num_object_clones":0,"num_object_copies":15903,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7453,"num_write_kb":408128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":430,"num_bytes_recovered":28180480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2],"avail_no_missing":["1","2","3","7"],"object_location_counts":[{"shards":"1,2","objects":2714},{"shards":"1,2,7","objects":1511}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":302,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.256038+0000","last_change":"2024-08-17T17:24:06.595380+0000","last_active":"2024-08-17T17:26:10.256038+0000","last_peered":"2024-08-17T17:26:10.256038+0000","last_clean":"2024-08-17T17:26:10.256038+0000","last_became_active":"2024-08-17T17:24:06.584747+0000","last_became_peered":"2024-08-17T17:24:06.584747+0000","last_unstale":"2024-08-17T17:26:10.256038+0000","last_undegraded":"2024-08-17T17:26:10.256038+0000","last_fullsized":"2024-08-17T17:26:10.256038+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:02:18.473980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":248,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.260897+0000","last_change":"2024-08-17T17:20:49.280596+0000","last_active":"2024-08-17T17:26:10.260897+0000","last_peered":"2024-08-17T17:26:10.260897+0000","last_clean":"2024-08-17T17:26:10.260897+0000","last_became_active":"2024-08-17T17:20:49.279553+0000","last_became_peered":"2024-08-17T17:20:49.279553+0000","last_unstale":"2024-08-17T17:26:10.260897+0000","last_undegraded":"2024-08-17T17:26:10.260897+0000","last_fullsized":"2024-08-17T17:26:10.260897+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:13:24.124392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"209'7149","reported_seq":7222,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:21.564633+0000","last_change":"2024-08-17T17:24:14.638890+0000","last_active":"2024-08-17T17:26:21.564633+0000","last_peered":"2024-08-17T17:26:21.564633+0000","last_clean":"2024-08-17T17:26:21.564633+0000","last_became_active":"2024-08-17T17:24:14.638594+0000","last_became_peered":"2024-08-17T17:24:14.638594+0000","last_unstale":"2024-08-17T17:26:21.564633+0000","last_undegraded":"2024-08-17T17:26:21.564633+0000","last_fullsized":"2024-08-17T17:26:21.564633+0000","mapping_epoch":178,"log_start":"186'700","ondisk_log_start":"186'700","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6449,"log_dups_size":0,"ondisk_log_size":6449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:53:34.277077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331415552,"num_objects":5057,"num_object_clones":0,"num_object_copies":15171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7149,"num_write_kb":390592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":200,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.266358+0000","last_change":"2024-08-17T17:20:49.276769+0000","last_active":"2024-08-17T17:26:10.266358+0000","last_peered":"2024-08-17T17:26:10.266358+0000","last_clean":"2024-08-17T17:26:10.266358+0000","last_became_active":"2024-08-17T17:20:49.276498+0000","last_became_peered":"2024-08-17T17:20:49.276498+0000","last_unstale":"2024-08-17T17:26:10.266358+0000","last_undegraded":"2024-08-17T17:26:10.266358+0000","last_fullsized":"2024-08-17T17:26:10.266358+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:39:05.419440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":188,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.227167+0000","last_change":"2024-08-17T17:20:14.455738+0000","last_active":"2024-08-17T17:26:08.227167+0000","last_peered":"2024-08-17T17:26:08.227167+0000","last_clean":"2024-08-17T17:26:08.227167+0000","last_became_active":"2024-08-17T17:20:14.455469+0000","last_became_peered":"2024-08-17T17:20:14.455469+0000","last_unstale":"2024-08-17T17:26:08.227167+0000","last_undegraded":"2024-08-17T17:26:08.227167+0000","last_fullsized":"2024-08-17T17:26:08.227167+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:27:28.424336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"209'7274","reported_seq":10317,"reported_epoch":209,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:26:21.586745+0000","last_change":"2024-08-17T17:25:05.678234+0000","last_active":"2024-08-17T17:26:21.586745+0000","last_peered":"2024-08-17T17:26:21.586745+0000","last_clean":"2024-08-17T17:24:52.436463+0000","last_became_active":"2024-08-17T17:25:05.673238+0000","last_became_peered":"2024-08-17T17:25:05.673238+0000","last_unstale":"2024-08-17T17:26:21.586745+0000","last_undegraded":"2024-08-17T17:25:05.585769+0000","last_fullsized":"2024-08-17T17:25:05.499112+0000","mapping_epoch":198,"log_start":"186'800","ondisk_log_start":"186'800","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6474,"log_dups_size":0,"ondisk_log_size":6474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333840384,"num_objects":5094,"num_object_clones":0,"num_object_copies":15282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3886,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7274,"num_write_kb":395776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1089,"num_bytes_recovered":71368704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,5],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":1430},{"shards":"3,5","objects":2638},{"shards":"3,5,7","objects":721}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":142,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.224478+0000","last_change":"2024-08-17T17:20:14.431140+0000","last_active":"2024-08-17T17:26:08.224478+0000","last_peered":"2024-08-17T17:26:08.224478+0000","last_clean":"2024-08-17T17:26:08.224478+0000","last_became_active":"2024-08-17T17:20:14.430839+0000","last_became_peered":"2024-08-17T17:20:14.430839+0000","last_unstale":"2024-08-17T17:26:08.224478+0000","last_undegraded":"2024-08-17T17:26:08.224478+0000","last_fullsized":"2024-08-17T17:26:08.224478+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:08:16.303616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":22,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.244125+0000","last_change":"2024-08-17T17:25:04.509325+0000","last_active":"2024-08-17T17:26:10.244125+0000","last_peered":"2024-08-17T17:26:10.244125+0000","last_clean":"2024-08-17T17:26:10.244125+0000","last_became_active":"2024-08-17T17:25:04.508364+0000","last_became_peered":"2024-08-17T17:25:04.508364+0000","last_unstale":"2024-08-17T17:26:10.244125+0000","last_undegraded":"2024-08-17T17:26:10.244125+0000","last_fullsized":"2024-08-17T17:26:10.244125+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:48:56.415763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"210'7198","reported_seq":12027,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-08-17T17:26:23.802142+0000","last_change":"2024-08-17T17:25:40.811756+0000","last_active":"2024-08-17T17:26:23.802142+0000","last_peered":"2024-08-17T17:26:23.802142+0000","last_clean":"2024-08-17T17:26:23.802142+0000","last_became_active":"2024-08-17T17:25:40.771755+0000","last_became_peered":"2024-08-17T17:25:40.771755+0000","last_unstale":"2024-08-17T17:26:23.802142+0000","last_undegraded":"2024-08-17T17:26:23.802142+0000","last_fullsized":"2024-08-17T17:26:23.802142+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":7198,"log_dups_size":0,"ondisk_log_size":7198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:13:03.918487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322371608,"num_objects":4920,"num_object_clones":0,"num_object_copies":14760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4920,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7198,"num_write_kb":387713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2371,"num_bytes_recovered":155385856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":287,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.254213+0000","last_change":"2024-08-17T17:19:57.520745+0000","last_active":"2024-08-17T17:26:10.254213+0000","last_peered":"2024-08-17T17:26:10.254213+0000","last_clean":"2024-08-17T17:26:10.254213+0000","last_became_active":"2024-08-17T17:19:57.520152+0000","last_became_peered":"2024-08-17T17:19:57.520152+0000","last_unstale":"2024-08-17T17:26:10.254213+0000","last_undegraded":"2024-08-17T17:26:10.254213+0000","last_fullsized":"2024-08-17T17:26:10.254213+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:19:39.495644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":262,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.264049+0000","last_change":"2024-08-17T17:20:49.274372+0000","last_active":"2024-08-17T17:26:10.264049+0000","last_peered":"2024-08-17T17:26:10.264049+0000","last_clean":"2024-08-17T17:26:10.264049+0000","last_became_active":"2024-08-17T17:20:49.274023+0000","last_became_peered":"2024-08-17T17:20:49.274023+0000","last_unstale":"2024-08-17T17:26:10.264049+0000","last_undegraded":"2024-08-17T17:26:10.264049+0000","last_fullsized":"2024-08-17T17:26:10.264049+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:31:02.529766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"210'7652","reported_seq":7688,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-08-17T17:26:23.814695+0000","last_change":"2024-08-17T17:24:14.643398+0000","last_active":"2024-08-17T17:26:23.814695+0000","last_peered":"2024-08-17T17:26:23.814695+0000","last_clean":"2024-08-17T17:26:23.814695+0000","last_became_active":"2024-08-17T17:24:14.643102+0000","last_became_peered":"2024-08-17T17:24:14.643102+0000","last_unstale":"2024-08-17T17:26:23.814695+0000","last_undegraded":"2024-08-17T17:26:23.814695+0000","last_fullsized":"2024-08-17T17:26:23.814695+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":7652,"log_dups_size":0,"ondisk_log_size":7652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:09:30.323709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331350016,"num_objects":5056,"num_object_clones":0,"num_object_copies":15168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7652,"num_write_kb":406656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":536,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.224515+0000","last_change":"2024-08-17T17:24:09.599615+0000","last_active":"2024-08-17T17:26:08.224515+0000","last_peered":"2024-08-17T17:26:08.224515+0000","last_clean":"2024-08-17T17:26:08.224515+0000","last_became_active":"2024-08-17T17:24:09.599085+0000","last_became_peered":"2024-08-17T17:24:09.599085+0000","last_unstale":"2024-08-17T17:26:08.224515+0000","last_undegraded":"2024-08-17T17:26:08.224515+0000","last_fullsized":"2024-08-17T17:26:08.224515+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:04:01.141401+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:30:09.569579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":782,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.254729+0000","last_change":"2024-08-17T17:20:00.434774+0000","last_active":"2024-08-17T17:26:10.254729+0000","last_peered":"2024-08-17T17:26:10.254729+0000","last_clean":"2024-08-17T17:26:10.254729+0000","last_became_active":"2024-08-17T17:20:00.434343+0000","last_became_peered":"2024-08-17T17:20:00.434343+0000","last_unstale":"2024-08-17T17:26:10.254729+0000","last_undegraded":"2024-08-17T17:26:10.254729+0000","last_fullsized":"2024-08-17T17:26:10.254729+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:04.995152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":245,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.255717+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:26:10.255717+0000","last_peered":"2024-08-17T17:26:10.255717+0000","last_clean":"2024-08-17T17:26:10.255717+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:26:10.255717+0000","last_undegraded":"2024-08-17T17:26:10.255717+0000","last_fullsized":"2024-08-17T17:26:10.255717+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":22,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.242993+0000","last_change":"2024-08-17T17:25:04.579426+0000","last_active":"2024-08-17T17:26:10.242993+0000","last_peered":"2024-08-17T17:26:10.242993+0000","last_clean":"2024-08-17T17:26:10.242993+0000","last_became_active":"2024-08-17T17:25:04.579012+0000","last_became_peered":"2024-08-17T17:25:04.579012+0000","last_unstale":"2024-08-17T17:26:10.242993+0000","last_undegraded":"2024-08-17T17:26:10.242993+0000","last_fullsized":"2024-08-17T17:26:10.242993+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:56:32.241321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"209'7126","reported_seq":7200,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:20.326952+0000","last_change":"2024-08-17T17:24:14.641829+0000","last_active":"2024-08-17T17:26:20.326952+0000","last_peered":"2024-08-17T17:26:20.326952+0000","last_clean":"2024-08-17T17:26:20.326952+0000","last_became_active":"2024-08-17T17:24:14.641532+0000","last_became_peered":"2024-08-17T17:24:14.641532+0000","last_unstale":"2024-08-17T17:26:20.326952+0000","last_undegraded":"2024-08-17T17:26:20.326952+0000","last_fullsized":"2024-08-17T17:26:20.326952+0000","mapping_epoch":178,"log_start":"186'800","ondisk_log_start":"186'800","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6326,"log_dups_size":0,"ondisk_log_size":6326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:43:05.232036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335020032,"num_objects":5112,"num_object_clones":0,"num_object_copies":15336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7126,"num_write_kb":391616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":799,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.224686+0000","last_change":"2024-08-17T17:16:26.958415+0000","last_active":"2024-08-17T17:26:08.224686+0000","last_peered":"2024-08-17T17:26:08.224686+0000","last_clean":"2024-08-17T17:26:08.224686+0000","last_became_active":"2024-08-17T17:16:26.957956+0000","last_became_peered":"2024-08-17T17:16:26.957956+0000","last_unstale":"2024-08-17T17:26:08.224686+0000","last_undegraded":"2024-08-17T17:26:08.224686+0000","last_fullsized":"2024-08-17T17:26:08.224686+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:22.478077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":274,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.251131+0000","last_change":"2024-08-17T17:18:59.415331+0000","last_active":"2024-08-17T17:26:10.251131+0000","last_peered":"2024-08-17T17:26:10.251131+0000","last_clean":"2024-08-17T17:26:10.251131+0000","last_became_active":"2024-08-17T17:18:59.415008+0000","last_became_peered":"2024-08-17T17:18:59.415008+0000","last_unstale":"2024-08-17T17:26:10.251131+0000","last_undegraded":"2024-08-17T17:26:10.251131+0000","last_fullsized":"2024-08-17T17:26:10.251131+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:22:38.076866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":83,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.249253+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:26:10.249253+0000","last_peered":"2024-08-17T17:26:10.249253+0000","last_clean":"2024-08-17T17:26:10.249253+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:26:10.249253+0000","last_undegraded":"2024-08-17T17:26:10.249253+0000","last_fullsized":"2024-08-17T17:26:10.249253+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"209'7013","reported_seq":8834,"reported_epoch":209,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-17T17:26:19.003034+0000","last_change":"2024-08-17T17:26:06.842453+0000","last_active":"2024-08-17T17:26:19.003034+0000","last_peered":"2024-08-17T17:26:19.003034+0000","last_clean":"2024-08-17T17:25:03.970571+0000","last_became_active":"2024-08-17T17:25:05.604478+0000","last_became_peered":"2024-08-17T17:25:05.604478+0000","last_unstale":"2024-08-17T17:26:19.003034+0000","last_undegraded":"2024-08-17T17:25:05.510212+0000","last_fullsized":"2024-08-17T17:25:05.498949+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":7013,"log_dups_size":0,"ondisk_log_size":7013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:29:09.928695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338755584,"num_objects":5169,"num_object_clones":0,"num_object_copies":15507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":386,"num_objects_misplaced":1836,"num_objects_unfound":0,"num_objects_dirty":5169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7013,"num_write_kb":389824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":886,"num_bytes_recovered":58064896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,3],"avail_no_missing":["7","2","3"],"object_location_counts":[{"shards":"2,3,7","objects":1836},{"shards":"3,7","objects":2947}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":661,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.255766+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:26:10.255766+0000","last_peered":"2024-08-17T17:26:10.255766+0000","last_clean":"2024-08-17T17:26:10.255766+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:26:10.255766+0000","last_undegraded":"2024-08-17T17:26:10.255766+0000","last_fullsized":"2024-08-17T17:26:10.255766+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":269,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.263805+0000","last_change":"2024-08-17T17:19:57.515564+0000","last_active":"2024-08-17T17:26:10.263805+0000","last_peered":"2024-08-17T17:26:10.263805+0000","last_clean":"2024-08-17T17:26:10.263805+0000","last_became_active":"2024-08-17T17:19:57.515142+0000","last_became_peered":"2024-08-17T17:19:57.515142+0000","last_unstale":"2024-08-17T17:26:10.263805+0000","last_undegraded":"2024-08-17T17:26:10.263805+0000","last_fullsized":"2024-08-17T17:26:10.263805+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:59:33.506176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":300,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.263880+0000","last_change":"2024-08-17T17:24:06.587903+0000","last_active":"2024-08-17T17:26:10.263880+0000","last_peered":"2024-08-17T17:26:10.263880+0000","last_clean":"2024-08-17T17:26:10.263880+0000","last_became_active":"2024-08-17T17:24:06.587503+0000","last_became_peered":"2024-08-17T17:24:06.587503+0000","last_unstale":"2024-08-17T17:26:10.263880+0000","last_undegraded":"2024-08-17T17:26:10.263880+0000","last_fullsized":"2024-08-17T17:26:10.263880+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:05:58.591992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"209'7154","reported_seq":13444,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:21.554371+0000","last_change":"2024-08-17T17:26:08.233981+0000","last_active":"2024-08-17T17:26:21.554371+0000","last_peered":"2024-08-17T17:26:21.554371+0000","last_clean":"2024-08-17T17:26:21.554371+0000","last_became_active":"2024-08-17T17:26:08.225711+0000","last_became_peered":"2024-08-17T17:26:08.225711+0000","last_unstale":"2024-08-17T17:26:21.554371+0000","last_undegraded":"2024-08-17T17:26:21.554371+0000","last_fullsized":"2024-08-17T17:26:21.554371+0000","mapping_epoch":207,"log_start":"186'700","ondisk_log_start":"186'700","created":178,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6454,"log_dups_size":0,"ondisk_log_size":6454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:31:42.806869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328204288,"num_objects":5008,"num_object_clones":0,"num_object_copies":15024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7154,"num_write_kb":389184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3101,"num_bytes_recovered":203227136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":1120,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.254948+0000","last_change":"2024-08-17T17:20:00.433870+0000","last_active":"2024-08-17T17:26:10.254948+0000","last_peered":"2024-08-17T17:26:10.254948+0000","last_clean":"2024-08-17T17:26:10.254948+0000","last_became_active":"2024-08-17T17:20:00.433386+0000","last_became_peered":"2024-08-17T17:20:00.433386+0000","last_unstale":"2024-08-17T17:26:10.254948+0000","last_undegraded":"2024-08-17T17:26:10.254948+0000","last_fullsized":"2024-08-17T17:26:10.254948+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:01:11.848862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1392,"num_read_kb":2863,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":195,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.248530+0000","last_change":"2024-08-17T17:20:49.272407+0000","last_active":"2024-08-17T17:26:10.248530+0000","last_peered":"2024-08-17T17:26:10.248530+0000","last_clean":"2024-08-17T17:26:10.248530+0000","last_became_active":"2024-08-17T17:20:49.271749+0000","last_became_peered":"2024-08-17T17:20:49.271749+0000","last_unstale":"2024-08-17T17:26:10.248530+0000","last_undegraded":"2024-08-17T17:26:10.248530+0000","last_fullsized":"2024-08-17T17:26:10.248530+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:36:01.443457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":266,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.267288+0000","last_change":"2024-08-17T17:17:32.499715+0000","last_active":"2024-08-17T17:26:10.267288+0000","last_peered":"2024-08-17T17:26:10.267288+0000","last_clean":"2024-08-17T17:26:10.267288+0000","last_became_active":"2024-08-17T17:17:32.499412+0000","last_became_peered":"2024-08-17T17:17:32.499412+0000","last_unstale":"2024-08-17T17:26:10.267288+0000","last_undegraded":"2024-08-17T17:26:10.267288+0000","last_fullsized":"2024-08-17T17:26:10.267288+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:39:38.456970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"209'7408","reported_seq":7482,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:22.946099+0000","last_change":"2024-08-17T17:24:14.642608+0000","last_active":"2024-08-17T17:26:22.946099+0000","last_peered":"2024-08-17T17:26:22.946099+0000","last_clean":"2024-08-17T17:26:22.946099+0000","last_became_active":"2024-08-17T17:24:14.642292+0000","last_became_peered":"2024-08-17T17:24:14.642292+0000","last_unstale":"2024-08-17T17:26:22.946099+0000","last_undegraded":"2024-08-17T17:26:22.946099+0000","last_fullsized":"2024-08-17T17:26:22.946099+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":7408,"log_dups_size":0,"ondisk_log_size":7408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:53:36.848614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322174976,"num_objects":4916,"num_object_clones":0,"num_object_copies":14748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7408,"num_write_kb":394368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":185,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.254928+0000","last_change":"2024-08-17T17:17:32.495728+0000","last_active":"2024-08-17T17:26:10.254928+0000","last_peered":"2024-08-17T17:26:10.254928+0000","last_clean":"2024-08-17T17:26:10.254928+0000","last_became_active":"2024-08-17T17:17:32.495357+0000","last_became_peered":"2024-08-17T17:17:32.495357+0000","last_unstale":"2024-08-17T17:26:10.254928+0000","last_undegraded":"2024-08-17T17:26:10.254928+0000","last_fullsized":"2024-08-17T17:26:10.254928+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:27:45.561567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":502,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.267457+0000","last_change":"2024-08-17T17:16:26.964621+0000","last_active":"2024-08-17T17:26:10.267457+0000","last_peered":"2024-08-17T17:26:10.267457+0000","last_clean":"2024-08-17T17:26:10.267457+0000","last_became_active":"2024-08-17T17:16:26.964058+0000","last_became_peered":"2024-08-17T17:16:26.964058+0000","last_unstale":"2024-08-17T17:26:10.267457+0000","last_undegraded":"2024-08-17T17:26:10.267457+0000","last_fullsized":"2024-08-17T17:26:10.267457+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:00:04.465644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":272,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.266443+0000","last_change":"2024-08-17T17:24:53.465550+0000","last_active":"2024-08-17T17:26:10.266443+0000","last_peered":"2024-08-17T17:26:10.266443+0000","last_clean":"2024-08-17T17:26:10.266443+0000","last_became_active":"2024-08-17T17:24:53.465228+0000","last_became_peered":"2024-08-17T17:24:53.465228+0000","last_unstale":"2024-08-17T17:26:10.266443+0000","last_undegraded":"2024-08-17T17:26:10.266443+0000","last_fullsized":"2024-08-17T17:26:10.266443+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:21:18.378359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"209'7275","reported_seq":7347,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:20.326620+0000","last_change":"2024-08-17T17:24:14.640584+0000","last_active":"2024-08-17T17:26:20.326620+0000","last_peered":"2024-08-17T17:26:20.326620+0000","last_clean":"2024-08-17T17:26:20.326620+0000","last_became_active":"2024-08-17T17:24:14.640168+0000","last_became_peered":"2024-08-17T17:24:14.640168+0000","last_unstale":"2024-08-17T17:26:20.326620+0000","last_undegraded":"2024-08-17T17:26:20.326620+0000","last_fullsized":"2024-08-17T17:26:20.326620+0000","mapping_epoch":178,"log_start":"187'1000","ondisk_log_start":"187'1000","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6275,"log_dups_size":0,"ondisk_log_size":6275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:38:36.176813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336134144,"num_objects":5129,"num_object_clones":0,"num_object_copies":15387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7275,"num_write_kb":396928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":297,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.267370+0000","last_change":"2024-08-17T17:24:53.464307+0000","last_active":"2024-08-17T17:26:10.267370+0000","last_peered":"2024-08-17T17:26:10.267370+0000","last_clean":"2024-08-17T17:26:10.267370+0000","last_became_active":"2024-08-17T17:24:53.463996+0000","last_became_peered":"2024-08-17T17:24:53.463996+0000","last_unstale":"2024-08-17T17:26:10.267370+0000","last_undegraded":"2024-08-17T17:26:10.267370+0000","last_fullsized":"2024-08-17T17:26:10.267370+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:04:18.993725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":636,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.227464+0000","last_change":"2024-08-17T17:24:09.599822+0000","last_active":"2024-08-17T17:26:08.227464+0000","last_peered":"2024-08-17T17:26:08.227464+0000","last_clean":"2024-08-17T17:26:08.227464+0000","last_became_active":"2024-08-17T17:24:09.599189+0000","last_became_peered":"2024-08-17T17:24:09.599189+0000","last_unstale":"2024-08-17T17:26:08.227464+0000","last_undegraded":"2024-08-17T17:26:08.227464+0000","last_fullsized":"2024-08-17T17:26:08.227464+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:04.465604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":257,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.248644+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:26:10.248644+0000","last_peered":"2024-08-17T17:26:10.248644+0000","last_clean":"2024-08-17T17:26:10.248644+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:26:10.248644+0000","last_undegraded":"2024-08-17T17:26:10.248644+0000","last_fullsized":"2024-08-17T17:26:10.248644+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"209'7431","reported_seq":7505,"reported_epoch":209,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:26:21.593836+0000","last_change":"2024-08-17T17:25:05.605653+0000","last_active":"2024-08-17T17:26:21.593836+0000","last_peered":"2024-08-17T17:26:21.593836+0000","last_clean":"2024-08-17T17:25:03.971607+0000","last_became_active":"2024-08-17T17:25:05.601333+0000","last_became_peered":"2024-08-17T17:25:05.601333+0000","last_unstale":"2024-08-17T17:26:21.593836+0000","last_undegraded":"2024-08-17T17:25:05.522007+0000","last_fullsized":"2024-08-17T17:25:05.499355+0000","mapping_epoch":198,"log_start":"187'1000","ondisk_log_start":"187'1000","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6431,"log_dups_size":0,"ondisk_log_size":6431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335347712,"num_objects":5117,"num_object_clones":0,"num_object_copies":15351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7431,"num_write_kb":401536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2],"avail_no_missing":["3","2","7"],"object_location_counts":[{"shards":"2,3","objects":1801},{"shards":"2,3,7","objects":2159}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":292,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.266509+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:26:10.266509+0000","last_peered":"2024-08-17T17:26:10.266509+0000","last_clean":"2024-08-17T17:26:10.266509+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:26:10.266509+0000","last_undegraded":"2024-08-17T17:26:10.266509+0000","last_fullsized":"2024-08-17T17:26:10.266509+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":811,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.265181+0000","last_change":"2024-08-17T17:23:19.633786+0000","last_active":"2024-08-17T17:26:10.265181+0000","last_peered":"2024-08-17T17:26:10.265181+0000","last_clean":"2024-08-17T17:26:10.265181+0000","last_became_active":"2024-08-17T17:23:19.633230+0000","last_became_peered":"2024-08-17T17:23:19.633230+0000","last_unstale":"2024-08-17T17:26:10.265181+0000","last_undegraded":"2024-08-17T17:26:10.265181+0000","last_fullsized":"2024-08-17T17:26:10.265181+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:30:50.377391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":288,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.253979+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:26:10.253979+0000","last_peered":"2024-08-17T17:26:10.253979+0000","last_clean":"2024-08-17T17:26:10.253979+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:26:10.253979+0000","last_undegraded":"2024-08-17T17:26:10.253979+0000","last_fullsized":"2024-08-17T17:26:10.253979+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"209'7161","reported_seq":7200,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:18.972638+0000","last_change":"2024-08-17T17:24:14.639560+0000","last_active":"2024-08-17T17:26:18.972638+0000","last_peered":"2024-08-17T17:26:18.972638+0000","last_clean":"2024-08-17T17:26:18.972638+0000","last_became_active":"2024-08-17T17:24:14.639093+0000","last_became_peered":"2024-08-17T17:24:14.639093+0000","last_unstale":"2024-08-17T17:26:18.972638+0000","last_undegraded":"2024-08-17T17:26:18.972638+0000","last_fullsized":"2024-08-17T17:26:18.972638+0000","mapping_epoch":178,"log_start":"183'100","ondisk_log_start":"183'100","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":7061,"log_dups_size":0,"ondisk_log_size":7061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:45:57.940550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341770240,"num_objects":5215,"num_object_clones":0,"num_object_copies":15645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7161,"num_write_kb":396032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":87,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.249166+0000","last_change":"2024-08-17T17:24:06.593660+0000","last_active":"2024-08-17T17:26:10.249166+0000","last_peered":"2024-08-17T17:26:10.249166+0000","last_clean":"2024-08-17T17:26:10.249166+0000","last_became_active":"2024-08-17T17:24:06.586215+0000","last_became_peered":"2024-08-17T17:24:06.586215+0000","last_unstale":"2024-08-17T17:26:10.249166+0000","last_undegraded":"2024-08-17T17:26:10.249166+0000","last_fullsized":"2024-08-17T17:26:10.249166+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:15:49.475395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":441,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.242120+0000","last_change":"2024-08-17T17:26:06.855307+0000","last_active":"2024-08-17T17:26:10.242120+0000","last_peered":"2024-08-17T17:26:10.242120+0000","last_clean":"2024-08-17T17:26:10.242120+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:26:10.242120+0000","last_undegraded":"2024-08-17T17:26:10.242120+0000","last_fullsized":"2024-08-17T17:26:10.242120+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":109,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.254535+0000","last_change":"2024-08-17T17:18:59.417183+0000","last_active":"2024-08-17T17:26:10.254535+0000","last_peered":"2024-08-17T17:26:10.254535+0000","last_clean":"2024-08-17T17:26:10.254535+0000","last_became_active":"2024-08-17T17:18:59.416843+0000","last_became_peered":"2024-08-17T17:18:59.416843+0000","last_unstale":"2024-08-17T17:26:10.254535+0000","last_undegraded":"2024-08-17T17:26:10.254535+0000","last_fullsized":"2024-08-17T17:26:10.254535+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:29:09.336335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":130,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.244036+0000","last_change":"2024-08-17T17:20:49.258048+0000","last_active":"2024-08-17T17:26:10.244036+0000","last_peered":"2024-08-17T17:26:10.244036+0000","last_clean":"2024-08-17T17:26:10.244036+0000","last_became_active":"2024-08-17T17:20:49.257672+0000","last_became_peered":"2024-08-17T17:20:49.257672+0000","last_unstale":"2024-08-17T17:26:10.244036+0000","last_undegraded":"2024-08-17T17:26:10.244036+0000","last_fullsized":"2024-08-17T17:26:10.244036+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:08:11.734316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":168,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.224785+0000","last_change":"2024-08-17T17:18:59.430969+0000","last_active":"2024-08-17T17:26:08.224785+0000","last_peered":"2024-08-17T17:26:08.224785+0000","last_clean":"2024-08-17T17:26:08.224785+0000","last_became_active":"2024-08-17T17:18:59.413475+0000","last_became_peered":"2024-08-17T17:18:59.413475+0000","last_unstale":"2024-08-17T17:26:08.224785+0000","last_undegraded":"2024-08-17T17:26:08.224785+0000","last_fullsized":"2024-08-17T17:26:08.224785+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:36:33.732645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":283,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.266246+0000","last_change":"2024-08-17T17:25:04.506413+0000","last_active":"2024-08-17T17:26:10.266246+0000","last_peered":"2024-08-17T17:26:10.266246+0000","last_clean":"2024-08-17T17:26:10.266246+0000","last_became_active":"2024-08-17T17:25:04.506104+0000","last_became_peered":"2024-08-17T17:25:04.506104+0000","last_unstale":"2024-08-17T17:26:10.266246+0000","last_undegraded":"2024-08-17T17:26:10.266246+0000","last_fullsized":"2024-08-17T17:26:10.266246+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:41:53.556223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":111,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-08-17T17:26:08.227078+0000","last_change":"2024-08-17T17:20:49.279458+0000","last_active":"2024-08-17T17:26:08.227078+0000","last_peered":"2024-08-17T17:26:08.227078+0000","last_clean":"2024-08-17T17:26:08.227078+0000","last_became_active":"2024-08-17T17:20:49.260790+0000","last_became_peered":"2024-08-17T17:20:49.260790+0000","last_unstale":"2024-08-17T17:26:08.227078+0000","last_undegraded":"2024-08-17T17:26:08.227078+0000","last_fullsized":"2024-08-17T17:26:08.227078+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:58.092945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":26,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.255027+0000","last_change":"2024-08-17T17:24:53.463713+0000","last_active":"2024-08-17T17:26:10.255027+0000","last_peered":"2024-08-17T17:26:10.255027+0000","last_clean":"2024-08-17T17:26:10.255027+0000","last_became_active":"2024-08-17T17:24:53.463419+0000","last_became_peered":"2024-08-17T17:24:53.463419+0000","last_unstale":"2024-08-17T17:26:10.255027+0000","last_undegraded":"2024-08-17T17:26:10.255027+0000","last_fullsized":"2024-08-17T17:26:10.255027+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:34:44.545175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":238,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.255487+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:26:10.255487+0000","last_peered":"2024-08-17T17:26:10.255487+0000","last_clean":"2024-08-17T17:26:10.255487+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:26:10.255487+0000","last_undegraded":"2024-08-17T17:26:10.255487+0000","last_fullsized":"2024-08-17T17:26:10.255487+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":266,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.249522+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:26:10.249522+0000","last_peered":"2024-08-17T17:26:10.249522+0000","last_clean":"2024-08-17T17:26:10.249522+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:26:10.249522+0000","last_undegraded":"2024-08-17T17:26:10.249522+0000","last_fullsized":"2024-08-17T17:26:10.249522+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":285,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.253895+0000","last_change":"2024-08-17T17:25:04.509422+0000","last_active":"2024-08-17T17:26:10.253895+0000","last_peered":"2024-08-17T17:26:10.253895+0000","last_clean":"2024-08-17T17:26:10.253895+0000","last_became_active":"2024-08-17T17:25:04.508742+0000","last_became_peered":"2024-08-17T17:25:04.508742+0000","last_unstale":"2024-08-17T17:26:10.253895+0000","last_undegraded":"2024-08-17T17:26:10.253895+0000","last_fullsized":"2024-08-17T17:26:10.253895+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:18:31.976907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":167,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.242036+0000","last_change":"2024-08-17T17:16:26.963696+0000","last_active":"2024-08-17T17:26:10.242036+0000","last_peered":"2024-08-17T17:26:10.242036+0000","last_clean":"2024-08-17T17:26:10.242036+0000","last_became_active":"2024-08-17T17:16:26.963280+0000","last_became_peered":"2024-08-17T17:16:26.963280+0000","last_unstale":"2024-08-17T17:26:10.242036+0000","last_undegraded":"2024-08-17T17:26:10.242036+0000","last_fullsized":"2024-08-17T17:26:10.242036+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:22:01.474840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":303,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.254096+0000","last_change":"2024-08-17T17:24:06.586886+0000","last_active":"2024-08-17T17:26:10.254096+0000","last_peered":"2024-08-17T17:26:10.254096+0000","last_clean":"2024-08-17T17:26:10.254096+0000","last_became_active":"2024-08-17T17:24:06.586516+0000","last_became_peered":"2024-08-17T17:24:06.586516+0000","last_unstale":"2024-08-17T17:26:10.254096+0000","last_undegraded":"2024-08-17T17:26:10.254096+0000","last_fullsized":"2024-08-17T17:26:10.254096+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:03:30.002726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":83,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.248969+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:26:10.248969+0000","last_peered":"2024-08-17T17:26:10.248969+0000","last_clean":"2024-08-17T17:26:10.248969+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:26:10.248969+0000","last_undegraded":"2024-08-17T17:26:10.248969+0000","last_fullsized":"2024-08-17T17:26:10.248969+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":263,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.265978+0000","last_change":"2024-08-17T17:18:59.412463+0000","last_active":"2024-08-17T17:26:10.265978+0000","last_peered":"2024-08-17T17:26:10.265978+0000","last_clean":"2024-08-17T17:26:10.265978+0000","last_became_active":"2024-08-17T17:18:59.412134+0000","last_became_peered":"2024-08-17T17:18:59.412134+0000","last_unstale":"2024-08-17T17:26:10.265978+0000","last_undegraded":"2024-08-17T17:26:10.265978+0000","last_fullsized":"2024-08-17T17:26:10.265978+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:44.463153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":306,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.256520+0000","last_change":"2024-08-17T17:25:04.510854+0000","last_active":"2024-08-17T17:26:10.256520+0000","last_peered":"2024-08-17T17:26:10.256520+0000","last_clean":"2024-08-17T17:26:10.256520+0000","last_became_active":"2024-08-17T17:25:04.510462+0000","last_became_peered":"2024-08-17T17:25:04.510462+0000","last_unstale":"2024-08-17T17:26:10.256520+0000","last_undegraded":"2024-08-17T17:26:10.256520+0000","last_fullsized":"2024-08-17T17:26:10.256520+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:55:54.762086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":203,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.266522+0000","last_change":"2024-08-17T17:24:06.591165+0000","last_active":"2024-08-17T17:26:10.266522+0000","last_peered":"2024-08-17T17:26:10.266522+0000","last_clean":"2024-08-17T17:26:10.266522+0000","last_became_active":"2024-08-17T17:24:06.590785+0000","last_became_peered":"2024-08-17T17:24:06.590785+0000","last_unstale":"2024-08-17T17:26:10.266522+0000","last_undegraded":"2024-08-17T17:26:10.266522+0000","last_fullsized":"2024-08-17T17:26:10.266522+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:54:05.775563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":275,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.267588+0000","last_change":"2024-08-17T17:24:32.300560+0000","last_active":"2024-08-17T17:26:10.267588+0000","last_peered":"2024-08-17T17:26:10.267588+0000","last_clean":"2024-08-17T17:26:10.267588+0000","last_became_active":"2024-08-17T17:24:32.300228+0000","last_became_peered":"2024-08-17T17:24:32.300228+0000","last_unstale":"2024-08-17T17:26:10.267588+0000","last_undegraded":"2024-08-17T17:26:10.267588+0000","last_fullsized":"2024-08-17T17:26:10.267588+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:18:05.475449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":82,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.248425+0000","last_change":"2024-08-17T17:24:06.586379+0000","last_active":"2024-08-17T17:26:10.248425+0000","last_peered":"2024-08-17T17:26:10.248425+0000","last_clean":"2024-08-17T17:26:10.248425+0000","last_became_active":"2024-08-17T17:24:06.586023+0000","last_became_peered":"2024-08-17T17:24:06.586023+0000","last_unstale":"2024-08-17T17:26:10.248425+0000","last_undegraded":"2024-08-17T17:26:10.248425+0000","last_fullsized":"2024-08-17T17:26:10.248425+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:45.701637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":292,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.255885+0000","last_change":"2024-08-17T17:21:50.442535+0000","last_active":"2024-08-17T17:26:10.255885+0000","last_peered":"2024-08-17T17:26:10.255885+0000","last_clean":"2024-08-17T17:26:10.255885+0000","last_became_active":"2024-08-17T17:21:50.441891+0000","last_became_peered":"2024-08-17T17:21:50.441891+0000","last_unstale":"2024-08-17T17:26:10.255885+0000","last_undegraded":"2024-08-17T17:26:10.255885+0000","last_fullsized":"2024-08-17T17:26:10.255885+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:46:24.025407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":222,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.254765+0000","last_change":"2024-08-17T17:25:04.508045+0000","last_active":"2024-08-17T17:26:10.254765+0000","last_peered":"2024-08-17T17:26:10.254765+0000","last_clean":"2024-08-17T17:26:10.254765+0000","last_became_active":"2024-08-17T17:25:04.505083+0000","last_became_peered":"2024-08-17T17:25:04.505083+0000","last_unstale":"2024-08-17T17:26:10.254765+0000","last_undegraded":"2024-08-17T17:26:10.254765+0000","last_fullsized":"2024-08-17T17:26:10.254765+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:23:42.993544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":278,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.255938+0000","last_change":"2024-08-17T17:18:59.418434+0000","last_active":"2024-08-17T17:26:10.255938+0000","last_peered":"2024-08-17T17:26:10.255938+0000","last_clean":"2024-08-17T17:26:10.255938+0000","last_became_active":"2024-08-17T17:18:59.418043+0000","last_became_peered":"2024-08-17T17:18:59.418043+0000","last_unstale":"2024-08-17T17:26:10.255938+0000","last_undegraded":"2024-08-17T17:26:10.255938+0000","last_fullsized":"2024-08-17T17:26:10.255938+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:25:51.371820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":195,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.254833+0000","last_change":"2024-08-17T17:20:49.291167+0000","last_active":"2024-08-17T17:26:10.254833+0000","last_peered":"2024-08-17T17:26:10.254833+0000","last_clean":"2024-08-17T17:26:10.254833+0000","last_became_active":"2024-08-17T17:20:49.263750+0000","last_became_peered":"2024-08-17T17:20:49.263750+0000","last_unstale":"2024-08-17T17:26:10.254833+0000","last_undegraded":"2024-08-17T17:26:10.254833+0000","last_fullsized":"2024-08-17T17:26:10.254833+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:02.863246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":292,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.255873+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:26:10.255873+0000","last_peered":"2024-08-17T17:26:10.255873+0000","last_clean":"2024-08-17T17:26:10.255873+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:26:10.255873+0000","last_undegraded":"2024-08-17T17:26:10.255873+0000","last_fullsized":"2024-08-17T17:26:10.255873+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":292,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.256577+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:26:10.256577+0000","last_peered":"2024-08-17T17:26:10.256577+0000","last_clean":"2024-08-17T17:26:10.256577+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:26:10.256577+0000","last_undegraded":"2024-08-17T17:26:10.256577+0000","last_fullsized":"2024-08-17T17:26:10.256577+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.241704+0000","last_change":"2024-08-17T17:25:04.503834+0000","last_active":"2024-08-17T17:26:10.241704+0000","last_peered":"2024-08-17T17:26:10.241704+0000","last_clean":"2024-08-17T17:26:10.241704+0000","last_became_active":"2024-08-17T17:25:04.502712+0000","last_became_peered":"2024-08-17T17:25:04.502712+0000","last_unstale":"2024-08-17T17:26:10.241704+0000","last_undegraded":"2024-08-17T17:26:10.241704+0000","last_fullsized":"2024-08-17T17:26:10.241704+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:45.389103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":102,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-08-17T17:26:10.249475+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:26:10.249475+0000","last_peered":"2024-08-17T17:26:10.249475+0000","last_clean":"2024-08-17T17:26:10.249475+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:26:10.249475+0000","last_undegraded":"2024-08-17T17:26:10.249475+0000","last_fullsized":"2024-08-17T17:26:10.249475+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":5320212504,"num_objects":81181,"num_object_clones":0,"num_object_copies":243543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12635,"num_objects_misplaced":1836,"num_objects_unfound":0,"num_objects_dirty":81181,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":116113,"num_write_kb":6313345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7877,"num_bytes_recovered":516227072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16641896448,"data_stored":16641884232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108913,"ondisk_log_size":108913,"up":48,"acting":43,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":6304,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":16702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17971,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16789504,"data_stored":16777273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234234,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2886336,"kb_used_data":2580664,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":90858816,"statfs":{"total":95995035648,"available":93039427584,"internally_reserved":0,"allocated":2642599936,"data_stored":2641398985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495870,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2501068,"kb_used_data":2191232,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91244084,"statfs":{"total":95995035648,"available":93433942016,"internally_reserved":0,"allocated":2243821568,"data_stored":2242620613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757507,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1996548,"kb_used_data":1686712,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91748604,"statfs":{"total":95995035648,"available":93950570496,"internally_reserved":0,"allocated":1727193088,"data_stored":1725995804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019145,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435372,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3556712,"kb_used_data":3267936,"kb_used_omap":4,"kb_used_meta":288571,"kb_avail":90188440,"statfs":{"total":95995035648,"available":92352962560,"internally_reserved":0,"allocated":3346366464,"data_stored":3345185018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":295496714},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":104,"seq":446676598898,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3332984,"kb_used_data":3019760,"kb_used_omap":7,"kb_used_meta":313080,"kb_avail":90412168,"statfs":{"total":95995035648,"available":92582060032,"internally_reserved":0,"allocated":3092234240,"data_stored":3091041676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":320594082},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762425,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2654576,"kb_used_data":2320232,"kb_used_omap":7,"kb_used_meta":334328,"kb_avail":91090576,"statfs":{"total":95995035648,"available":93276749824,"internally_reserved":0,"allocated":2375917568,"data_stored":2374727921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7945,"internal_metadata":342352119},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":94,"seq":403726925952,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1530232,"kb_used_data":1264368,"kb_used_omap":7,"kb_used_meta":265848,"kb_avail":92214920,"statfs":{"total":95995035648,"available":94428078080,"internally_reserved":0,"allocated":1294712832,"data_stored":1293516178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":272228673},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6241,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1288966144,"data_stored":1288962072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2355826688,"data_stored":2355822616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3084320768,"data_stored":3084320768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3323199488,"data_stored":3323199488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1721569280,"data_stored":1721565208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2235170816,"data_stored":2235170816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2632843264,"data_stored":2632843264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:26:26.737 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph status --format=json 2024-08-17T17:26:26.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:26.739 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:26.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:26.840 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:26.868 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:26.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:26.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:26.970 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:27.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:27.019 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:27.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:27.120 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:27.154 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:27.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:27.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:27.256 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:27.290 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:27.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:27.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:27.392 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:27.427 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:27.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:27.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:27.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:27 smithi003 ceph-mon[103344]: from='client.35691 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:26:27.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:27 smithi003 ceph-mon[103344]: pgmap v537: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 778 op/s; 12635/239445 objects degraded (5.277%); 1414/239445 objects misplaced (0.591%); 8.1 MiB/s, 128 objects/s recovering 2024-08-17T17:26:27.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:27 smithi003 ceph-mon[100615]: from='client.35691 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:26:27.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:27 smithi003 ceph-mon[100615]: pgmap v537: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 778 op/s; 12635/239445 objects degraded (5.277%); 1414/239445 objects misplaced (0.591%); 8.1 MiB/s, 128 objects/s recovering 2024-08-17T17:26:27.529 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:27.572 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:27.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:27.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:27.674 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:27.712 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:27.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:27.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:27.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:27 smithi053 ceph-mon[89963]: from='client.35691 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:26:27.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:27 smithi053 ceph-mon[89963]: pgmap v537: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 778 op/s; 12635/239445 objects degraded (5.277%); 1414/239445 objects misplaced (0.591%); 8.1 MiB/s, 128 objects/s recovering 2024-08-17T17:26:27.814 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:27.861 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:27.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:27.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:27.963 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:28.005 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:28.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:28.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:28.107 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:28.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:28.139 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:28.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:28.241 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:28.297 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:28.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:28.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:28.399 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:28.447 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:28.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:28.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:28.486 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:28 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:26:28.487 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:28 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:26:28.549 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:28.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:28.584 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:28.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:28.686 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:28.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:28 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:26:28.729 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:28.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:28.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:28.832 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:28.872 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:28.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:28.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:28.974 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:29.005 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:29.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:29.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:29.107 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:29.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:29.147 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:29.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:29.248 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:29.286 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:29.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:29.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:29.387 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:29.415 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:29.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:29.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:29.517 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:29.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:29.558 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:29.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:29.578 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:29 smithi003 ceph-mon[100615]: pgmap v538: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 784 op/s; 12093/235647 objects degraded (5.132%); 1414/235647 objects misplaced (0.600%); 8.2 MiB/s, 131 objects/s recovering 2024-08-17T17:26:29.578 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:29 smithi003 ceph-mon[103344]: pgmap v538: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 784 op/s; 12093/235647 objects degraded (5.132%); 1414/235647 objects misplaced (0.600%); 8.2 MiB/s, 131 objects/s recovering 2024-08-17T17:26:29.595 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:29 smithi053 ceph-mon[89963]: pgmap v538: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 784 op/s; 12093/235647 objects degraded (5.132%); 1414/235647 objects misplaced (0.600%); 8.2 MiB/s, 131 objects/s recovering 2024-08-17T17:26:29.659 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:29.696 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:29.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:29.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:29.798 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:29.844 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:29.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:29.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:29.946 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:29.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:29.989 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:29.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:30.071 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:26:30.071 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:26:30.071 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:26:30.071 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:26:30.071 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:26:30.071 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:26:30.071 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:26:30.072 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:26:30.072 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:26:30.072 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:26:30.072 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:26:30.072 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:26:30.072 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:26:30.072 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:26:30.072 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:26:30.072 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:26:30.091 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:30.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:30.132 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:30.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:30.233 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:30.278 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:30.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:30.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:30.380 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:30.409 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:30.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:30.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:30.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:26:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:26:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:26:30.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:30 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3273405479' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:26:30.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:30 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 12093/235647 objects degraded (5.132%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:30.511 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:30.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:30 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3273405479' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:26:30.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:30 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 12093/235647 objects degraded (5.132%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:30.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:30 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3273405479' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:26:30.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:30 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 12093/235647 objects degraded (5.132%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:30.563 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:30.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:30.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:30.665 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:30.705 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:30.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:30.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:30.808 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:30.834 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:30.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:30.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:30.936 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:30.968 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:30.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:30.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:31.071 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:31.094 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:31.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:31.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:31.196 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:31.225 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:31.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:31.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:31.328 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:31.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:31.367 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:31.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:31.468 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:31.507 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:31.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:31.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:31.582 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:31 smithi053 ceph-mon[89963]: pgmap v539: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 786 op/s; 12093/229806 objects degraded (5.262%); 924/229806 objects misplaced (0.402%); 8.9 MiB/s, 142 objects/s recovering 2024-08-17T17:26:31.608 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:31.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:31.717 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:31.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:31.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:31 smithi003 ceph-mon[103344]: pgmap v539: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 786 op/s; 12093/229806 objects degraded (5.262%); 924/229806 objects misplaced (0.402%); 8.9 MiB/s, 142 objects/s recovering 2024-08-17T17:26:31.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:31 smithi003 ceph-mon[100615]: pgmap v539: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 786 op/s; 12093/229806 objects degraded (5.262%); 924/229806 objects misplaced (0.402%); 8.9 MiB/s, 142 objects/s recovering 2024-08-17T17:26:31.818 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:31.822 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:26:31.836 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:26:31.844 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:31.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:31.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:31.946 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:31.992 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:31.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:31.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:32.093 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:32.118 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:32.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:32.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:32.220 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:32.253 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:32.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:32.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:32.355 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:32.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:32.395 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:32.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:32.496 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:32.528 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:32.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:32.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:32.629 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:32.664 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:32.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:32.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:32.766 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:32.802 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:32.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:32.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:32.803 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:26:32.904 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:32.937 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:32.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:32.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:33.039 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:33.069 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:33.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:33.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:33.171 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:33.222 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:33.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:33.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:33.324 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:33.372 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:33.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:33.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:33.473 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:33.505 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:33.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:33.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:33.607 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:33.618 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:33 smithi053 ceph-mon[89963]: pgmap v540: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 774 op/s; 11577/224172 objects degraded (5.164%); 924/224172 objects misplaced (0.412%); 9.0 MiB/s, 144 objects/s recovering 2024-08-17T17:26:33.618 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:33 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:26:33.618 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:33 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:26:33.641 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:33.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:33.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:33.742 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:33.757 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:26:33.757 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:26:33.758 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.758 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45491.0:124897 6.3 e210/178 6:c2cbe65e:::benchmark_data_smithi003_175043_object26248:head v 210'7806, mlcod=210'7804)", 2024-08-17T17:26:33.758 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:26:33.008626+0000", 2024-08-17T17:26:33.758 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.016458865999999999, 2024-08-17T17:26:33.758 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.016487132000000002, 2024-08-17T17:26:33.758 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:26:33.758 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:26:33.758 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:26:33.758 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:26:33.758 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.758 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:26:33.758 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.008626+0000", 2024-08-17T17:26:33.759 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:26:33.759 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.759 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.759 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:26:33.759 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.008626+0000", 2024-08-17T17:26:33.759 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:26:33.759 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.759 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.759 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:26:33.759 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.008627+0000", 2024-08-17T17:26:33.759 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3090000000000001e-06 2024-08-17T17:26:33.759 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.759 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.760 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:26:33.760 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.008636+0000", 2024-08-17T17:26:33.760 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6719999999999993e-06 2024-08-17T17:26:33.760 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.760 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.760 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:26:33.760 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.008666+0000", 2024-08-17T17:26:33.760 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9689e-05 2024-08-17T17:26:33.760 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.760 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:26:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.008675+0000", 2024-08-17T17:26:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1419999999999999e-06 2024-08-17T17:26:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:26:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.008755+0000", 2024-08-17T17:26:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0026999999999998e-05 2024-08-17T17:26:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:26:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.008778+0000", 2024-08-17T17:26:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2710000000000001e-05 2024-08-17T17:26:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:26:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:26:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:26:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45491.0:124898 6.3 e210/178 6:c22993ee:::benchmark_data_smithi003_175043_object26249:head v 210'7807, mlcod=210'7804)", 2024-08-17T17:26:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:26:33.009758+0000", 2024-08-17T17:26:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.015327031, 2024-08-17T17:26:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.015410263, 2024-08-17T17:26:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:26:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:26:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:26:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:26:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.763 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:26:33.763 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.009758+0000", 2024-08-17T17:26:33.763 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:26:33.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.763 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.763 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:26:33.763 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.009758+0000", 2024-08-17T17:26:33.763 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:26:33.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.763 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.763 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:26:33.763 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.009759+0000", 2024-08-17T17:26:33.763 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6000000000000003e-07 2024-08-17T17:26:33.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.763 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.764 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:26:33.764 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.009765+0000", 2024-08-17T17:26:33.764 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1229999999999997e-06 2024-08-17T17:26:33.764 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.764 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.764 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:26:33.764 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.009776+0000", 2024-08-17T17:26:33.764 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1507e-05 2024-08-17T17:26:33.764 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.764 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.764 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:26:33.764 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.009783+0000", 2024-08-17T17:26:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3060000000000003e-06 2024-08-17T17:26:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:26:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.012425+0000", 2024-08-17T17:26:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002642644 2024-08-17T17:26:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:26:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.012447+0000", 2024-08-17T17:26:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1724000000000001e-05 2024-08-17T17:26:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:26:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:26:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:26:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45491.0:124900 6.3 e210/178 6:cfa36eef:::benchmark_data_smithi003_175043_object26251:head v 210'7808, mlcod=210'7804)", 2024-08-17T17:26:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:26:33.017111+0000", 2024-08-17T17:26:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0079741640000000006, 2024-08-17T17:26:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0081093499999999995, 2024-08-17T17:26:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:26:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:26:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:26:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:26:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:26:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.017111+0000", 2024-08-17T17:26:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:26:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:26:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.017111+0000", 2024-08-17T17:26:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:26:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:26:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.017111+0000", 2024-08-17T17:26:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8299999999999998e-07 2024-08-17T17:26:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:26:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.017115+0000", 2024-08-17T17:26:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9700000000000001e-06 2024-08-17T17:26:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:26:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.017124+0000", 2024-08-17T17:26:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5520000000000005e-06 2024-08-17T17:26:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:26:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.017129+0000", 2024-08-17T17:26:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7999999999999998e-06 2024-08-17T17:26:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:26:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.020444+0000", 2024-08-17T17:26:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0033151959999999999 2024-08-17T17:26:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:26:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.020464+0000", 2024-08-17T17:26:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0679999999999999e-05 2024-08-17T17:26:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:26:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:26:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:26:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45491.0:124906 6.9 e210/178 6:9c047e44:::benchmark_data_smithi003_175043_object26257:head v 210'7630, mlcod=210'7628)", 2024-08-17T17:26:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:26:33.021480+0000", 2024-08-17T17:26:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0036047340000000001, 2024-08-17T17:26:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0037859389999999999, 2024-08-17T17:26:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:26:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:26:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:26:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:26:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:26:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.021480+0000", 2024-08-17T17:26:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:26:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:26:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.021480+0000", 2024-08-17T17:26:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:26:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:26:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.021481+0000", 2024-08-17T17:26:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5499999999999997e-07 2024-08-17T17:26:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:26:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.021488+0000", 2024-08-17T17:26:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4189999999999996e-06 2024-08-17T17:26:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:26:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.021499+0000", 2024-08-17T17:26:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0049e-05 2024-08-17T17:26:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:26:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.021506+0000", 2024-08-17T17:26:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0509999999999999e-06 2024-08-17T17:26:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:26:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.024563+0000", 2024-08-17T17:26:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0030573950000000001 2024-08-17T17:26:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:26:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.024583+0000", 2024-08-17T17:26:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9584000000000001e-05 2024-08-17T17:26:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:26:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:26:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:26:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45491.0:124902 6.1 e210/178)", 2024-08-17T17:26:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:26:33.021565+0000", 2024-08-17T17:26:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0035197900000000001, 2024-08-17T17:26:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0037462519999999998, 2024-08-17T17:26:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:26:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:26:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-08-17T17:26:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:26:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:26:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.021565+0000", 2024-08-17T17:26:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:26:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:26:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.021565+0000", 2024-08-17T17:26:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:26:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:26:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.021566+0000", 2024-08-17T17:26:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.75e-07 2024-08-17T17:26:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:26:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.021572+0000", 2024-08-17T17:26:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7049999999999996e-06 2024-08-17T17:26:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:26:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.021581+0000", 2024-08-17T17:26:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8039999999999996e-06 2024-08-17T17:26:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:26:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:26:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:26:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:26:33.021587+0000", 2024-08-17T17:26:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.105e-06 2024-08-17T17:26:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:26:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:26:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:26:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:26:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-08-17T17:26:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 5 2024-08-17T17:26:33.776 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:26:33.777 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:33 smithi003 ceph-mon[100615]: pgmap v540: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 774 op/s; 11577/224172 objects degraded (5.164%); 924/224172 objects misplaced (0.412%); 9.0 MiB/s, 144 objects/s recovering 2024-08-17T17:26:33.777 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:33 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:26:33.777 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:33 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:26:33.777 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:33 smithi003 ceph-mon[103344]: pgmap v540: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 774 op/s; 11577/224172 objects degraded (5.164%); 924/224172 objects misplaced (0.412%); 9.0 MiB/s, 144 objects/s recovering 2024-08-17T17:26:33.778 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:33 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:26:33.778 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:33 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:26:33.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:33.778 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:33.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:33.879 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:33.925 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:33.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:33.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:34.027 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:34.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:34.066 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:34.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:34.167 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:34.199 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:34.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:34.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:34.301 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:34.332 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:34.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:34.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:34.434 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:34.461 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:34.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:34.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:34.563 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:34.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:34.602 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:34.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:34.703 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:34.740 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:34.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:34.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:34.797 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-17T17:26:34.843 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:34.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:34.880 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:34.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:34.982 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:35.030 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:35.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:35.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:35.132 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:35.160 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:35.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:35.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:35.262 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:35.293 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:35.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:35.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:35.396 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:35.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:35.440 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:35.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:35.541 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:35.579 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:35.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:35.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:35.653 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:35 smithi003 ceph-mon[100615]: pgmap v541: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.5 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 768 op/s; 11577/220605 objects degraded (5.248%); 924/220605 objects misplaced (0.419%); 8.5 MiB/s, 135 objects/s recovering 2024-08-17T17:26:35.653 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:35 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 11577/220605 objects degraded (5.248%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:35.653 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:35 smithi003 ceph-mon[103344]: pgmap v541: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.5 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 768 op/s; 11577/220605 objects degraded (5.248%); 924/220605 objects misplaced (0.419%); 8.5 MiB/s, 135 objects/s recovering 2024-08-17T17:26:35.653 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:35 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 11577/220605 objects degraded (5.248%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:35.682 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:35.712 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:35.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:35.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:35.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:35 smithi053 ceph-mon[89963]: pgmap v541: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.5 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 768 op/s; 11577/220605 objects degraded (5.248%); 924/220605 objects misplaced (0.419%); 8.5 MiB/s, 135 objects/s recovering 2024-08-17T17:26:35.719 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:35 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 11577/220605 objects degraded (5.248%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:35.814 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:35.844 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:35.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:35.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:35.946 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:35.980 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:35.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:35.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:35.995 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:26:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:26:35] "GET /metrics HTTP/1.1" 200 36526 "" "Prometheus/2.51.0" 2024-08-17T17:26:36.082 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:36.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:36.118 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:36.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:36.219 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:36.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:36.254 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:36.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:36.355 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:36.398 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:36.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:36.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:36.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:36.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:36.538 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:36.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:36.640 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:36.674 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:36.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:36.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:36.775 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:36.819 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:36.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:36.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:36.920 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:36.955 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:36.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:36.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:37.057 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:37.091 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:37.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:37.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:37.193 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:37.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:37.243 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:37.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:37.344 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:37.383 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:37.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:37.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:37.484 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:37.513 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:37.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:37.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:37.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:37 smithi003 ceph-mon[103344]: pgmap v542: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 755 op/s; 11577/216408 objects degraded (5.350%); 352/216408 objects misplaced (0.163%); 10 MiB/s, 167 objects/s recovering 2024-08-17T17:26:37.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:37 smithi003 ceph-mon[100615]: pgmap v542: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 755 op/s; 11577/216408 objects degraded (5.350%); 352/216408 objects misplaced (0.163%); 10 MiB/s, 167 objects/s recovering 2024-08-17T17:26:37.614 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:37.652 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:37.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:37.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:37.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:37 smithi053 ceph-mon[89963]: pgmap v542: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 755 op/s; 11577/216408 objects degraded (5.350%); 352/216408 objects misplaced (0.163%); 10 MiB/s, 167 objects/s recovering 2024-08-17T17:26:37.754 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:37.781 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:37.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:37.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:37.883 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:37.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:37.914 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:37.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:38.015 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:38.044 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:38.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:38.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:38.146 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:38.181 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:38.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:38.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:38.283 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:38.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:38.324 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:38.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:38.425 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:38.457 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:38.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:38.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:38.559 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:38.586 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:38.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:38.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:38.627 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:26:38.627 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:26:38.687 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:38.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:26:38.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:38.733 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:38.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:38.834 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:38.861 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:38.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:38.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:38.963 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:38.991 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:38.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:38.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:38.993 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:26:38.999 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:26:39.093 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:39.119 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:39.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:39.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:39.221 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:39.253 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:39.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:39.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:39.355 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:39.400 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:39.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:39.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:39.410 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-08-17T17:26:39.502 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:39.545 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:39.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:39.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:39.647 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:39.672 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:39.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:39.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:39.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:39 smithi053 ceph-mon[89963]: pgmap v543: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 745 op/s; 11111/212574 objects degraded (5.227%); 352/212574 objects misplaced (0.166%); 9.5 MiB/s, 151 objects/s recovering 2024-08-17T17:26:39.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:39 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2857922814' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:26:39.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:39 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:26:39.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:39 smithi003 ceph-mon[103344]: pgmap v543: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 745 op/s; 11111/212574 objects degraded (5.227%); 352/212574 objects misplaced (0.166%); 9.5 MiB/s, 151 objects/s recovering 2024-08-17T17:26:39.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:39 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2857922814' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:26:39.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:39 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:26:39.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:39 smithi003 ceph-mon[100615]: pgmap v543: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 745 op/s; 11111/212574 objects degraded (5.227%); 352/212574 objects misplaced (0.166%); 9.5 MiB/s, 151 objects/s recovering 2024-08-17T17:26:39.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:39 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2857922814' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:26:39.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:39 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:26:39.773 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:39.799 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:39.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:39.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:39.901 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:39.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:39.924 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:39.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:40.025 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:40.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:40.048 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:40.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:40.150 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:40.172 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:40.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:40.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:40.273 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:40.296 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:40.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:40.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:40.397 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:40.421 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:40.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:40.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:40.430 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:26:40.431 INFO:teuthology.orchestra.run.smithi003.stdout:{"fsid":"2b5b662e-5cba-11ef-bccf-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 10725/206436 objects degraded (5.195%), 4 pgs degraded, 5 pgs undersized","count":9},"muted":false}},"mutes":[]},"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":742,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":212,"num_osds":8,"num_up_osds":8,"osd_up_since":1723915051,"num_in_osds":7,"osd_in_since":1723915248,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":84},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+undersized+remapped","count":1}],"num_pgs":89,"num_pools":5,"num_objects":68812,"data_bytes":4512552933,"bytes_used":16703979520,"bytes_avail":655261270016,"bytes_total":671965249536,"degraded_objects":10725,"degraded_total":206436,"degraded_ratio":0.051953147706795336,"recovering_objects_per_sec":179,"recovering_bytes_per_sec":9264833,"recovering_keys_per_sec":0,"num_objects_recovered":1801,"num_bytes_recovered":92733440,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":778},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi003.civgvf","status":"up:active","gid":14664}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.3:8443/","prometheus":"http://172.21.15.3:9283/"}},"servicemap":{"epoch":23,"modified":"2024-08-17T17:18:31.688676+0000","services":{}},"progress_events":{"714d66b4-625a-4d60-aa1c-f38ff47ebe12":{"message":"Global Recovery Event (80s)\n [==========================..] (remaining: 4s)","progress":0.94382023811340332,"add_to_ceph_s":true},"fcbeb9b7-7baf-488b-9fe4-c01aca441f9c":{"message":"Upgrade to 19.0.0-5836-gcd8bdbeb (4m)\n [============................] (remaining: 5m)","progress":0.44999998807907104,"add_to_ceph_s":true}}} 2024-08-17T17:26:40.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:26:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:26:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:26:40.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:40 smithi053 ceph-mon[89963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:26:40.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:40 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:26:40.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:40 smithi053 ceph-mon[89963]: osdmap e211: 8 total, 8 up, 7 in 2024-08-17T17:26:40.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:40 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 11111/212574 objects degraded (5.227%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:40.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:40 smithi003 ceph-mon[103344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:26:40.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:40 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:26:40.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:40 smithi003 ceph-mon[103344]: osdmap e211: 8 total, 8 up, 7 in 2024-08-17T17:26:40.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:40 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 11111/212574 objects degraded (5.227%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:40.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:40 smithi003 ceph-mon[100615]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:26:40.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:40 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:26:40.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:40 smithi003 ceph-mon[100615]: osdmap e211: 8 total, 8 up, 7 in 2024-08-17T17:26:40.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:40 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 11111/212574 objects degraded (5.227%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:40.524 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:40.554 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:40.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:40.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:40.656 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:40.685 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:40.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:40.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:40.787 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:40.829 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:40.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:40.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:40.930 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:40.971 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:40.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:40.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:41.073 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:41.102 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:41.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:41.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:41.204 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:41.238 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:41.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:41.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:41.242 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-08-17T17:26:41.242 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:26:41.340 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:41.366 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:41.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:41.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:41.469 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:41.496 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:41.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:41.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:41.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:41 smithi003 ceph-mon[103344]: pgmap v545: 89 pgs: 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 778 op/s; 10725/206436 objects degraded (5.195%); 8.8 MiB/s, 179 objects/s recovering 2024-08-17T17:26:41.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:41 smithi003 ceph-mon[103344]: osdmap e212: 8 total, 8 up, 7 in 2024-08-17T17:26:41.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:41 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/626932175' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T17:26:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:41 smithi003 ceph-mon[100615]: pgmap v545: 89 pgs: 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 778 op/s; 10725/206436 objects degraded (5.195%); 8.8 MiB/s, 179 objects/s recovering 2024-08-17T17:26:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:41 smithi003 ceph-mon[100615]: osdmap e212: 8 total, 8 up, 7 in 2024-08-17T17:26:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:41 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/626932175' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T17:26:41.598 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:41.646 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:41.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:41.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:41.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:41 smithi053 ceph-mon[89963]: pgmap v545: 89 pgs: 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 778 op/s; 10725/206436 objects degraded (5.195%); 8.8 MiB/s, 179 objects/s recovering 2024-08-17T17:26:41.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:41 smithi053 ceph-mon[89963]: osdmap e212: 8 total, 8 up, 7 in 2024-08-17T17:26:41.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:41 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/626932175' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T17:26:41.747 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:41.799 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:41.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:41.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:41.901 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:41.943 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:41.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:41.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:42.046 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:42.074 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:42.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:42.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:42.176 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:42.198 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:42.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:42.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:42.300 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:42.328 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:42.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:42.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:42.430 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:42.459 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:42.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:42.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:42.561 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:42.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:42.603 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:42.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:42.668 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:26:42.704 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:42.759 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:42.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:42.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:42.862 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:42.902 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:42.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:42.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:43.004 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:43.051 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:43.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:43.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:43.153 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:43.184 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:43.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:43.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:43.287 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:43.315 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:43.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:43.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:43.417 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:43.444 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:43.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:43.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:43.546 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:43.577 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:43.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:43.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:43.665 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:43 smithi003 ceph-mon[100615]: pgmap v547: 89 pgs: 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 818 op/s; 10255/200955 objects degraded (5.103%); 11 MiB/s, 222 objects/s recovering 2024-08-17T17:26:43.666 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:43 smithi003 ceph-mon[103344]: pgmap v547: 89 pgs: 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 818 op/s; 10255/200955 objects degraded (5.103%); 11 MiB/s, 222 objects/s recovering 2024-08-17T17:26:43.679 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:43.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:43 smithi053 ceph-mon[89963]: pgmap v547: 89 pgs: 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 818 op/s; 10255/200955 objects degraded (5.103%); 11 MiB/s, 222 objects/s recovering 2024-08-17T17:26:43.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:43.717 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:43.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:43.741 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:26:43.818 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:43.876 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:43.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:43.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:43.978 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:44.031 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:44.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:44.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:44.133 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:44.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:44.201 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:44.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:44.302 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:44.344 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:44.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:44.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:44.446 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:44.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:44.491 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:44.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:44.592 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:44.618 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:44.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:44.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:44.720 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:44.753 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:44.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:44.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:44.855 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:44.924 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:44.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:44.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:44.929 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:26:44.929 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-08-17T17:26:44.929 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-08-17T17:26:44.929 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-08-17T17:26:44.929 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:26:45.026 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:45.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:45.056 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:45.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:45.157 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:45.192 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:45.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:45.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:45.297 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:45.326 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:45.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:45.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:45.428 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:45.455 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:45.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:45.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:45.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:45 smithi003 ceph-mon[103344]: pgmap v548: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 768 op/s; 9739/197937 objects degraded (4.920%); 226/197937 objects misplaced (0.114%); 8.7 MiB/s, 186 objects/s recovering 2024-08-17T17:26:45.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:45 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 9739/197937 objects degraded (4.920%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:45.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:45 smithi003 ceph-mon[100615]: pgmap v548: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 768 op/s; 9739/197937 objects degraded (4.920%); 226/197937 objects misplaced (0.114%); 8.7 MiB/s, 186 objects/s recovering 2024-08-17T17:26:45.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:45 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 9739/197937 objects degraded (4.920%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:45.558 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:45.600 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:45.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:45.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:45.702 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:45.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:45 smithi053 ceph-mon[89963]: pgmap v548: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 768 op/s; 9739/197937 objects degraded (4.920%); 226/197937 objects misplaced (0.114%); 8.7 MiB/s, 186 objects/s recovering 2024-08-17T17:26:45.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:45 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 9739/197937 objects degraded (4.920%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:45.748 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:45.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:45.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:45.850 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:45.877 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:45.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:45.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:45.881 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:26:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:26:45] "GET /metrics HTTP/1.1" 200 36532 "" "Prometheus/2.51.0" 2024-08-17T17:26:45.886 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-17T17:26:45.978 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:46.002 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:26:46.002 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:26:46.003 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:26:46.003 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:26:46.003 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:26:46.003 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:26:46.003 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:26:46.003 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:26:46.003 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:26:46.003 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:26:46.003 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:26:46.003 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:26:46.003 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:26:46.003 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:26:46.004 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:26:46.004 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:26:46.004 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:46.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:46.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:46.106 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:46.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:46.137 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:46.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:46.239 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:46.241 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:26:46.266 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:46.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:46.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:46.368 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:46.393 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:46.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:46.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:46.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:46.553 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:46.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:46.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:46.655 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:46.656 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:46 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1252715656' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:26:46.657 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:46 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1252715656' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:26:46.681 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:46.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:46.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:46.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:46 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1252715656' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:26:46.783 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:46.810 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:46.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:46.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:46.912 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:46.936 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:46.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:46.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:47.037 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:47.059 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:47.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:47.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:47.161 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:47.188 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:47.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:47.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:47.289 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:47.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:47.335 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:47.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:47.436 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:47.468 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:47.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:47.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:47.570 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:47.601 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:47.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:47.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:47.704 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:47.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:47 smithi053 ceph-mon[89963]: pgmap v549: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 775 op/s; 9739/193950 objects degraded (5.021%); 226/193950 objects misplaced (0.117%); 6.8 MiB/s, 157 objects/s recovering 2024-08-17T17:26:47.734 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:47.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:47.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:47.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:47 smithi003 ceph-mon[103344]: pgmap v549: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 775 op/s; 9739/193950 objects degraded (5.021%); 226/193950 objects misplaced (0.117%); 6.8 MiB/s, 157 objects/s recovering 2024-08-17T17:26:47.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:47 smithi003 ceph-mon[100615]: pgmap v549: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 775 op/s; 9739/193950 objects degraded (5.021%); 226/193950 objects misplaced (0.117%); 6.8 MiB/s, 157 objects/s recovering 2024-08-17T17:26:47.835 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:47.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:47.880 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:47.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:47.981 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:48.023 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:48.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:48.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:48.125 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:48.165 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:48.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:48.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:48.267 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:48.309 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:48.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:48.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:48.410 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:48.441 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:48.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:48.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:48.542 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:48.569 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:48.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:48.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:48.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:48.717 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:48.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:48.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:48.819 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:48.859 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:48.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:48.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:48.961 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:48.987 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:48.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:48.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:49.089 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:49.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:49.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:49.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:49.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:49.314 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:49.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:49.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:49.416 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:49.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:49.470 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:49.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:49.571 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:49.614 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:49.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:49.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:49.647 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:49 smithi003 ceph-mon[103344]: pgmap v550: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 635 op/s; 9275/190245 objects degraded (4.875%); 226/190245 objects misplaced (0.119%); 5.6 MiB/s, 89 objects/s recovering 2024-08-17T17:26:49.647 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:49 smithi003 ceph-mon[100615]: pgmap v550: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 635 op/s; 9275/190245 objects degraded (4.875%); 226/190245 objects misplaced (0.119%); 5.6 MiB/s, 89 objects/s recovering 2024-08-17T17:26:49.716 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:49.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:49 smithi053 ceph-mon[89963]: pgmap v550: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 635 op/s; 9275/190245 objects degraded (4.875%); 226/190245 objects misplaced (0.119%); 5.6 MiB/s, 89 objects/s recovering 2024-08-17T17:26:49.743 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:49.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:49.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:49.844 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:49.880 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:49.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:49.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:49.982 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:50.024 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:50.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:50.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:50.126 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:50.182 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:50.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:50.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:50.284 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:50.314 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:50.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:50.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:50.415 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:50.451 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:50.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:50.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:50.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:26:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:26:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:26:50.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:50 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 9275/190245 objects degraded (4.875%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:50.554 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:50.595 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:50.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:50.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:50.697 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:50.726 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:50.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:50.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:50.733 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:50 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 9275/190245 objects degraded (4.875%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:50.733 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:50 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 9275/190245 objects degraded (4.875%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:50.818 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:26:50.828 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:50.853 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:50.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:50.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:50.955 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:50.983 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:50.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:50.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:51.085 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:51.112 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:51.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:51.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:51.127 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:26:51.128 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:26:51.214 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:51.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:51.246 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:51.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:51.347 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:51.374 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:51.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:51.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:51.475 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:51.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:51.534 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:51.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:51.635 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:51.664 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:51.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:51.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:51.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:51 smithi053 ceph-mon[89963]: pgmap v551: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 718 op/s; 9235/184872 objects degraded (4.995%); 6.1 MiB/s, 102 objects/s recovering 2024-08-17T17:26:51.767 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:51.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:51 smithi003 ceph-mon[103344]: pgmap v551: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 718 op/s; 9235/184872 objects degraded (4.995%); 6.1 MiB/s, 102 objects/s recovering 2024-08-17T17:26:51.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:51 smithi003 ceph-mon[100615]: pgmap v551: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 718 op/s; 9235/184872 objects degraded (4.995%); 6.1 MiB/s, 102 objects/s recovering 2024-08-17T17:26:51.799 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:51.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:51.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:51.901 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:51.929 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:51.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:51.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:51.972 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":551,"stamp":"2024-08-17T17:26:49.905428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4041480165,"num_objects":61624,"num_object_clones":0,"num_object_copies":184872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61624,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":136820,"num_write_kb":6328384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12042,"num_bytes_recovered":762848646,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114379,"ondisk_log_size":114379,"up":267,"acting":263,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":278,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":14923008,"kb_used_data":12789896,"kb_used_omap":33,"kb_used_meta":2132702,"kb_avail":641293056,"statfs":{"total":671965249536,"available":656684089344,"internally_reserved":0,"allocated":13096853504,"data_stored":13088343832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34068,"internal_metadata":2183887596},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,3,1,8,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-471072768,"num_objects":-7188,"num_object_clones":0,"num_object_copies":-21564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1490,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7188,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1023,"num_bytes_recovered":64421888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.005945"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":277,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.409906+0000","last_change":"2024-08-17T17:17:32.495652+0000","last_active":"2024-08-17T17:26:40.409906+0000","last_peered":"2024-08-17T17:26:40.409906+0000","last_clean":"2024-08-17T17:26:40.409906+0000","last_became_active":"2024-08-17T17:17:32.495218+0000","last_became_peered":"2024-08-17T17:17:32.495218+0000","last_unstale":"2024-08-17T17:26:40.409906+0000","last_undegraded":"2024-08-17T17:26:40.409906+0000","last_fullsized":"2024-08-17T17:26:40.409906+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:11:45.821282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":188,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-17T17:26:39.408439+0000","last_change":"2024-08-17T17:18:59.424887+0000","last_active":"2024-08-17T17:26:39.408439+0000","last_peered":"2024-08-17T17:26:39.408439+0000","last_clean":"2024-08-17T17:26:39.408439+0000","last_became_active":"2024-08-17T17:18:59.411539+0000","last_became_peered":"2024-08-17T17:18:59.411539+0000","last_unstale":"2024-08-17T17:26:39.408439+0000","last_undegraded":"2024-08-17T17:26:39.408439+0000","last_fullsized":"2024-08-17T17:26:39.408439+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:46:20.082241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":28,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-17T17:26:39.403072+0000","last_change":"2024-08-17T17:24:53.460742+0000","last_active":"2024-08-17T17:26:39.403072+0000","last_peered":"2024-08-17T17:26:39.403072+0000","last_clean":"2024-08-17T17:26:39.403072+0000","last_became_active":"2024-08-17T17:24:53.460365+0000","last_became_peered":"2024-08-17T17:24:53.460365+0000","last_unstale":"2024-08-17T17:26:39.403072+0000","last_undegraded":"2024-08-17T17:26:39.403072+0000","last_fullsized":"2024-08-17T17:26:39.403072+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:24:08.116968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":89,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.453920+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:26:40.453920+0000","last_peered":"2024-08-17T17:26:40.453920+0000","last_clean":"2024-08-17T17:26:40.453920+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:26:40.453920+0000","last_undegraded":"2024-08-17T17:26:40.453920+0000","last_fullsized":"2024-08-17T17:26:40.453920+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":89,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.452183+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:26:40.452183+0000","last_peered":"2024-08-17T17:26:40.452183+0000","last_clean":"2024-08-17T17:26:40.452183+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:26:40.452183+0000","last_undegraded":"2024-08-17T17:26:40.452183+0000","last_fullsized":"2024-08-17T17:26:40.452183+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":163,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-17T17:26:39.408194+0000","last_change":"2024-08-17T17:20:49.258206+0000","last_active":"2024-08-17T17:26:39.408194+0000","last_peered":"2024-08-17T17:26:39.408194+0000","last_clean":"2024-08-17T17:26:39.408194+0000","last_became_active":"2024-08-17T17:20:49.257761+0000","last_became_peered":"2024-08-17T17:20:49.257761+0000","last_unstale":"2024-08-17T17:26:39.408194+0000","last_undegraded":"2024-08-17T17:26:39.408194+0000","last_fullsized":"2024-08-17T17:26:39.408194+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:38:35.363748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":291,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.410538+0000","last_change":"2024-08-17T17:19:57.514773+0000","last_active":"2024-08-17T17:26:40.410538+0000","last_peered":"2024-08-17T17:26:40.410538+0000","last_clean":"2024-08-17T17:26:40.410538+0000","last_became_active":"2024-08-17T17:19:57.513839+0000","last_became_peered":"2024-08-17T17:19:57.513839+0000","last_unstale":"2024-08-17T17:26:40.410538+0000","last_undegraded":"2024-08-17T17:26:40.410538+0000","last_fullsized":"2024-08-17T17:26:40.410538+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:42:48.063169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":258,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-17T17:26:39.409978+0000","last_change":"2024-08-17T17:24:53.465875+0000","last_active":"2024-08-17T17:26:39.409978+0000","last_peered":"2024-08-17T17:26:39.409978+0000","last_clean":"2024-08-17T17:26:39.409978+0000","last_became_active":"2024-08-17T17:24:53.458129+0000","last_became_peered":"2024-08-17T17:24:53.458129+0000","last_unstale":"2024-08-17T17:26:39.409978+0000","last_undegraded":"2024-08-17T17:26:39.409978+0000","last_fullsized":"2024-08-17T17:26:39.409978+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:51:33.584869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"212'8457","reported_seq":9633,"reported_epoch":212,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-17T17:26:48.006714+0000","last_change":"2024-08-17T17:26:38.496637+0000","last_active":"2024-08-17T17:26:48.006714+0000","last_peered":"2024-08-17T17:26:48.006714+0000","last_clean":"2024-08-17T17:24:52.438492+0000","last_became_active":"2024-08-17T17:24:54.887269+0000","last_became_peered":"2024-08-17T17:24:54.887269+0000","last_unstale":"2024-08-17T17:26:48.006714+0000","last_undegraded":"2024-08-17T17:24:54.477707+0000","last_fullsized":"2024-08-17T17:24:54.461071+0000","mapping_epoch":194,"log_start":"186'500","ondisk_log_start":"186'500","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":7957,"log_dups_size":0,"ondisk_log_size":7957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248578048,"num_objects":3793,"num_object_clones":0,"num_object_copies":11379,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8457,"num_write_kb":392000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":556,"num_bytes_recovered":33816576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[0,6],"avail_no_missing":["0","2","6"],"object_location_counts":[{"shards":"0,6","objects":1889}],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":124,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-17T17:26:39.408787+0000","last_change":"2024-08-17T17:24:06.597382+0000","last_active":"2024-08-17T17:26:39.408787+0000","last_peered":"2024-08-17T17:26:39.408787+0000","last_clean":"2024-08-17T17:26:39.408787+0000","last_became_active":"2024-08-17T17:24:06.588881+0000","last_became_peered":"2024-08-17T17:24:06.588881+0000","last_unstale":"2024-08-17T17:26:39.408787+0000","last_undegraded":"2024-08-17T17:26:39.408787+0000","last_fullsized":"2024-08-17T17:26:39.408787+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:27.240751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":38,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-17T17:26:39.402545+0000","last_change":"2024-08-17T17:25:04.503780+0000","last_active":"2024-08-17T17:26:39.402545+0000","last_peered":"2024-08-17T17:26:39.402545+0000","last_clean":"2024-08-17T17:26:39.402545+0000","last_became_active":"2024-08-17T17:25:04.503503+0000","last_became_peered":"2024-08-17T17:25:04.503503+0000","last_unstale":"2024-08-17T17:26:39.402545+0000","last_undegraded":"2024-08-17T17:26:39.402545+0000","last_fullsized":"2024-08-17T17:26:39.402545+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:04:48.171216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"212'8475","reported_seq":8555,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:46.628604+0000","last_change":"2024-08-17T17:24:14.639361+0000","last_active":"2024-08-17T17:26:46.628604+0000","last_peered":"2024-08-17T17:26:46.628604+0000","last_clean":"2024-08-17T17:26:46.628604+0000","last_became_active":"2024-08-17T17:24:14.639074+0000","last_became_peered":"2024-08-17T17:24:14.639074+0000","last_unstale":"2024-08-17T17:26:46.628604+0000","last_undegraded":"2024-08-17T17:26:46.628604+0000","last_fullsized":"2024-08-17T17:26:46.628604+0000","mapping_epoch":178,"log_start":"191'2000","ondisk_log_start":"191'2000","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6475,"log_dups_size":0,"ondisk_log_size":6475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:15:05.377992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254869504,"num_objects":3889,"num_object_clones":0,"num_object_copies":11667,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8475,"num_write_kb":395648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":110,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-17T17:26:39.408805+0000","last_change":"2024-08-17T17:24:06.599023+0000","last_active":"2024-08-17T17:26:39.408805+0000","last_peered":"2024-08-17T17:26:39.408805+0000","last_clean":"2024-08-17T17:26:39.408805+0000","last_became_active":"2024-08-17T17:24:06.588954+0000","last_became_peered":"2024-08-17T17:24:06.588954+0000","last_unstale":"2024-08-17T17:26:39.408805+0000","last_undegraded":"2024-08-17T17:26:39.408805+0000","last_fullsized":"2024-08-17T17:26:39.408805+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:20:51.445576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":232,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.406318+0000","last_change":"2024-08-17T17:20:49.271540+0000","last_active":"2024-08-17T17:26:40.406318+0000","last_peered":"2024-08-17T17:26:40.406318+0000","last_clean":"2024-08-17T17:26:40.406318+0000","last_became_active":"2024-08-17T17:20:49.271130+0000","last_became_peered":"2024-08-17T17:20:49.271130+0000","last_unstale":"2024-08-17T17:26:40.406318+0000","last_undegraded":"2024-08-17T17:26:40.406318+0000","last_fullsized":"2024-08-17T17:26:40.406318+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:05.386705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"212'8441","reported_seq":8483,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:48.930176+0000","last_change":"2024-08-17T17:24:14.644070+0000","last_active":"2024-08-17T17:26:48.930176+0000","last_peered":"2024-08-17T17:26:48.930176+0000","last_clean":"2024-08-17T17:26:48.930176+0000","last_became_active":"2024-08-17T17:24:14.643799+0000","last_became_peered":"2024-08-17T17:24:14.643799+0000","last_unstale":"2024-08-17T17:26:48.930176+0000","last_undegraded":"2024-08-17T17:26:48.930176+0000","last_fullsized":"2024-08-17T17:26:48.930176+0000","mapping_epoch":178,"log_start":"191'1400","ondisk_log_start":"191'1400","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":7041,"log_dups_size":0,"ondisk_log_size":7041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:25:38.445770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250413056,"num_objects":3821,"num_object_clones":0,"num_object_copies":11463,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8441,"num_write_kb":392384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":182,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-17T17:26:39.408725+0000","last_change":"2024-08-17T17:24:53.467526+0000","last_active":"2024-08-17T17:26:39.408725+0000","last_peered":"2024-08-17T17:26:39.408725+0000","last_clean":"2024-08-17T17:26:39.408725+0000","last_became_active":"2024-08-17T17:24:53.460631+0000","last_became_peered":"2024-08-17T17:24:53.460631+0000","last_unstale":"2024-08-17T17:26:39.408725+0000","last_undegraded":"2024-08-17T17:26:39.408725+0000","last_fullsized":"2024-08-17T17:26:39.408725+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:05:24.938831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":24,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-17T17:26:39.413041+0000","last_change":"2024-08-17T17:25:04.507176+0000","last_active":"2024-08-17T17:26:39.413041+0000","last_peered":"2024-08-17T17:26:39.413041+0000","last_clean":"2024-08-17T17:26:39.413041+0000","last_became_active":"2024-08-17T17:25:04.506788+0000","last_became_peered":"2024-08-17T17:25:04.506788+0000","last_unstale":"2024-08-17T17:26:39.413041+0000","last_undegraded":"2024-08-17T17:26:39.413041+0000","last_fullsized":"2024-08-17T17:26:39.413041+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:22:04.658216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"212'8310","reported_seq":8390,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:47.989491+0000","last_change":"2024-08-17T17:24:14.642637+0000","last_active":"2024-08-17T17:26:47.989491+0000","last_peered":"2024-08-17T17:26:47.989491+0000","last_clean":"2024-08-17T17:26:47.989491+0000","last_became_active":"2024-08-17T17:24:14.642347+0000","last_became_peered":"2024-08-17T17:24:14.642347+0000","last_unstale":"2024-08-17T17:26:47.989491+0000","last_undegraded":"2024-08-17T17:26:47.989491+0000","last_fullsized":"2024-08-17T17:26:47.989491+0000","mapping_epoch":178,"log_start":"186'400","ondisk_log_start":"186'400","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":7910,"log_dups_size":0,"ondisk_log_size":7910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:37:44.631335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243793920,"num_objects":3720,"num_object_clones":0,"num_object_copies":11160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8310,"num_write_kb":384960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":170,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-17T17:26:39.405504+0000","last_change":"2024-08-17T17:17:32.495428+0000","last_active":"2024-08-17T17:26:39.405504+0000","last_peered":"2024-08-17T17:26:39.405504+0000","last_clean":"2024-08-17T17:26:39.405504+0000","last_became_active":"2024-08-17T17:17:32.494978+0000","last_became_peered":"2024-08-17T17:17:32.494978+0000","last_unstale":"2024-08-17T17:26:39.405504+0000","last_undegraded":"2024-08-17T17:26:39.405504+0000","last_fullsized":"2024-08-17T17:26:39.405504+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:42:24.647812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":280,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.452873+0000","last_change":"2024-08-17T17:20:49.272126+0000","last_active":"2024-08-17T17:26:40.452873+0000","last_peered":"2024-08-17T17:26:40.452873+0000","last_clean":"2024-08-17T17:26:40.452873+0000","last_became_active":"2024-08-17T17:20:49.271185+0000","last_became_peered":"2024-08-17T17:20:49.271185+0000","last_unstale":"2024-08-17T17:26:40.452873+0000","last_undegraded":"2024-08-17T17:26:40.452873+0000","last_fullsized":"2024-08-17T17:26:40.452873+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:43:58.184939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"212'8689","reported_seq":9644,"reported_epoch":212,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:26:45.274258+0000","last_change":"2024-08-17T17:24:54.898729+0000","last_active":"2024-08-17T17:26:45.274258+0000","last_peered":"2024-08-17T17:26:45.274258+0000","last_clean":"2024-08-17T17:24:52.435411+0000","last_became_active":"2024-08-17T17:24:54.889545+0000","last_became_peered":"2024-08-17T17:24:54.889545+0000","last_unstale":"2024-08-17T17:26:45.274258+0000","last_undegraded":"2024-08-17T17:24:54.482592+0000","last_fullsized":"2024-08-17T17:24:54.459529+0000","mapping_epoch":194,"log_start":"192'2400","ondisk_log_start":"192'2400","created":178,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6289,"log_dups_size":0,"ondisk_log_size":6289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266403840,"num_objects":4065,"num_object_clones":0,"num_object_copies":12195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8689,"num_write_kb":408128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":430,"num_bytes_recovered":28180480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2],"avail_no_missing":["1","2","3","7"],"object_location_counts":[{"shards":"1,2","objects":1478},{"shards":"1,2,7","objects":275}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":309,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.409910+0000","last_change":"2024-08-17T17:24:06.595380+0000","last_active":"2024-08-17T17:26:40.409910+0000","last_peered":"2024-08-17T17:26:40.409910+0000","last_clean":"2024-08-17T17:26:40.409910+0000","last_became_active":"2024-08-17T17:24:06.584747+0000","last_became_peered":"2024-08-17T17:24:06.584747+0000","last_unstale":"2024-08-17T17:26:40.409910+0000","last_undegraded":"2024-08-17T17:26:40.409910+0000","last_fullsized":"2024-08-17T17:26:40.409910+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:02:18.473980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":254,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.410225+0000","last_change":"2024-08-17T17:20:49.280596+0000","last_active":"2024-08-17T17:26:40.410225+0000","last_peered":"2024-08-17T17:26:40.410225+0000","last_clean":"2024-08-17T17:26:40.410225+0000","last_became_active":"2024-08-17T17:20:49.279553+0000","last_became_peered":"2024-08-17T17:20:49.279553+0000","last_unstale":"2024-08-17T17:26:40.410225+0000","last_undegraded":"2024-08-17T17:26:40.410225+0000","last_fullsized":"2024-08-17T17:26:40.410225+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:13:24.124392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"212'8362","reported_seq":8442,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:46.681714+0000","last_change":"2024-08-17T17:24:14.638890+0000","last_active":"2024-08-17T17:26:46.681714+0000","last_peered":"2024-08-17T17:26:46.681714+0000","last_clean":"2024-08-17T17:26:46.681714+0000","last_became_active":"2024-08-17T17:24:14.638594+0000","last_became_peered":"2024-08-17T17:24:14.638594+0000","last_unstale":"2024-08-17T17:26:46.681714+0000","last_undegraded":"2024-08-17T17:26:46.681714+0000","last_fullsized":"2024-08-17T17:26:46.681714+0000","mapping_epoch":178,"log_start":"191'1900","ondisk_log_start":"191'1900","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6462,"log_dups_size":0,"ondisk_log_size":6462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:53:34.277077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251920384,"num_objects":3844,"num_object_clones":0,"num_object_copies":11532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8362,"num_write_kb":390592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":206,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.405727+0000","last_change":"2024-08-17T17:20:49.276769+0000","last_active":"2024-08-17T17:26:40.405727+0000","last_peered":"2024-08-17T17:26:40.405727+0000","last_clean":"2024-08-17T17:26:40.405727+0000","last_became_active":"2024-08-17T17:20:49.276498+0000","last_became_peered":"2024-08-17T17:20:49.276498+0000","last_unstale":"2024-08-17T17:26:40.405727+0000","last_undegraded":"2024-08-17T17:26:40.405727+0000","last_fullsized":"2024-08-17T17:26:40.405727+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:39:05.419440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":191,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-17T17:26:39.408527+0000","last_change":"2024-08-17T17:20:14.455738+0000","last_active":"2024-08-17T17:26:39.408527+0000","last_peered":"2024-08-17T17:26:39.408527+0000","last_clean":"2024-08-17T17:26:39.408527+0000","last_became_active":"2024-08-17T17:20:14.455469+0000","last_became_peered":"2024-08-17T17:20:14.455469+0000","last_unstale":"2024-08-17T17:26:39.408527+0000","last_undegraded":"2024-08-17T17:26:39.408527+0000","last_fullsized":"2024-08-17T17:26:39.408527+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:27:28.424336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"212'8503","reported_seq":15240,"reported_epoch":212,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:26:46.692013+0000","last_change":"2024-08-17T17:25:05.678234+0000","last_active":"2024-08-17T17:26:46.692013+0000","last_peered":"2024-08-17T17:26:46.692013+0000","last_clean":"2024-08-17T17:24:52.436463+0000","last_became_active":"2024-08-17T17:25:05.673238+0000","last_became_peered":"2024-08-17T17:25:05.673238+0000","last_unstale":"2024-08-17T17:26:46.692013+0000","last_undegraded":"2024-08-17T17:25:05.585769+0000","last_fullsized":"2024-08-17T17:25:05.499112+0000","mapping_epoch":198,"log_start":"191'2100","ondisk_log_start":"191'2100","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6403,"log_dups_size":0,"ondisk_log_size":6403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253296640,"num_objects":3865,"num_object_clones":0,"num_object_copies":11595,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1428,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8503,"num_write_kb":395776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2318,"num_bytes_recovered":151912448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,5],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":201},{"shards":"3,5","objects":2638},{"shards":"3,5,7","objects":721}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":145,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-17T17:26:39.408161+0000","last_change":"2024-08-17T17:20:14.431140+0000","last_active":"2024-08-17T17:26:39.408161+0000","last_peered":"2024-08-17T17:26:39.408161+0000","last_clean":"2024-08-17T17:26:39.408161+0000","last_became_active":"2024-08-17T17:20:14.430839+0000","last_became_peered":"2024-08-17T17:20:14.430839+0000","last_unstale":"2024-08-17T17:26:39.408161+0000","last_undegraded":"2024-08-17T17:26:39.408161+0000","last_fullsized":"2024-08-17T17:26:39.408161+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:08:16.303616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":24,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-17T17:26:39.410313+0000","last_change":"2024-08-17T17:25:04.509325+0000","last_active":"2024-08-17T17:26:39.410313+0000","last_peered":"2024-08-17T17:26:39.410313+0000","last_clean":"2024-08-17T17:26:39.410313+0000","last_became_active":"2024-08-17T17:25:04.508364+0000","last_became_peered":"2024-08-17T17:25:04.508364+0000","last_unstale":"2024-08-17T17:26:39.410313+0000","last_undegraded":"2024-08-17T17:26:39.410313+0000","last_fullsized":"2024-08-17T17:26:39.410313+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:48:56.415763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"212'8327","reported_seq":13158,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:48.812115+0000","last_change":"2024-08-17T17:25:40.811756+0000","last_active":"2024-08-17T17:26:48.812115+0000","last_peered":"2024-08-17T17:26:48.812115+0000","last_clean":"2024-08-17T17:26:48.812115+0000","last_became_active":"2024-08-17T17:25:40.771755+0000","last_became_peered":"2024-08-17T17:25:40.771755+0000","last_unstale":"2024-08-17T17:26:48.812115+0000","last_undegraded":"2024-08-17T17:26:48.812115+0000","last_fullsized":"2024-08-17T17:26:48.812115+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":8327,"log_dups_size":0,"ondisk_log_size":8327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:13:03.918487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248381464,"num_objects":3791,"num_object_clones":0,"num_object_copies":11373,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3791,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8327,"num_write_kb":387713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2371,"num_bytes_recovered":155385856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":293,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.408009+0000","last_change":"2024-08-17T17:19:57.520745+0000","last_active":"2024-08-17T17:26:40.408009+0000","last_peered":"2024-08-17T17:26:40.408009+0000","last_clean":"2024-08-17T17:26:40.408009+0000","last_became_active":"2024-08-17T17:19:57.520152+0000","last_became_peered":"2024-08-17T17:19:57.520152+0000","last_unstale":"2024-08-17T17:26:40.408009+0000","last_undegraded":"2024-08-17T17:26:40.408009+0000","last_fullsized":"2024-08-17T17:26:40.408009+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:19:39.495644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":268,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.406215+0000","last_change":"2024-08-17T17:20:49.274372+0000","last_active":"2024-08-17T17:26:40.406215+0000","last_peered":"2024-08-17T17:26:40.406215+0000","last_clean":"2024-08-17T17:26:40.406215+0000","last_became_active":"2024-08-17T17:20:49.274023+0000","last_became_peered":"2024-08-17T17:20:49.274023+0000","last_unstale":"2024-08-17T17:26:40.406215+0000","last_undegraded":"2024-08-17T17:26:40.406215+0000","last_fullsized":"2024-08-17T17:26:40.406215+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:31:02.529766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"212'8890","reported_seq":8932,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:48.840892+0000","last_change":"2024-08-17T17:24:14.643398+0000","last_active":"2024-08-17T17:26:48.840892+0000","last_peered":"2024-08-17T17:26:48.840892+0000","last_clean":"2024-08-17T17:26:48.840892+0000","last_became_active":"2024-08-17T17:24:14.643102+0000","last_became_peered":"2024-08-17T17:24:14.643102+0000","last_unstale":"2024-08-17T17:26:48.840892+0000","last_undegraded":"2024-08-17T17:26:48.840892+0000","last_fullsized":"2024-08-17T17:26:48.840892+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":8890,"log_dups_size":0,"ondisk_log_size":8890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:09:30.323709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250216448,"num_objects":3818,"num_object_clones":0,"num_object_copies":11454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8890,"num_write_kb":406656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":539,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-17T17:26:39.408138+0000","last_change":"2024-08-17T17:24:09.599615+0000","last_active":"2024-08-17T17:26:39.408138+0000","last_peered":"2024-08-17T17:26:39.408138+0000","last_clean":"2024-08-17T17:26:39.408138+0000","last_became_active":"2024-08-17T17:24:09.599085+0000","last_became_peered":"2024-08-17T17:24:09.599085+0000","last_unstale":"2024-08-17T17:26:39.408138+0000","last_undegraded":"2024-08-17T17:26:39.408138+0000","last_fullsized":"2024-08-17T17:26:39.408138+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:04:01.141401+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:30:09.569579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":784,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-17T17:26:39.402836+0000","last_change":"2024-08-17T17:20:00.434774+0000","last_active":"2024-08-17T17:26:39.402836+0000","last_peered":"2024-08-17T17:26:39.402836+0000","last_clean":"2024-08-17T17:26:39.402836+0000","last_became_active":"2024-08-17T17:20:00.434343+0000","last_became_peered":"2024-08-17T17:20:00.434343+0000","last_unstale":"2024-08-17T17:26:39.402836+0000","last_undegraded":"2024-08-17T17:26:39.402836+0000","last_fullsized":"2024-08-17T17:26:39.402836+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:04.995152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":251,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.408334+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:26:40.408334+0000","last_peered":"2024-08-17T17:26:40.408334+0000","last_clean":"2024-08-17T17:26:40.408334+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:26:40.408334+0000","last_undegraded":"2024-08-17T17:26:40.408334+0000","last_fullsized":"2024-08-17T17:26:40.408334+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":24,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-17T17:26:39.410277+0000","last_change":"2024-08-17T17:25:04.579426+0000","last_active":"2024-08-17T17:26:39.410277+0000","last_peered":"2024-08-17T17:26:39.410277+0000","last_clean":"2024-08-17T17:26:39.410277+0000","last_became_active":"2024-08-17T17:25:04.579012+0000","last_became_peered":"2024-08-17T17:25:04.579012+0000","last_unstale":"2024-08-17T17:26:39.410277+0000","last_undegraded":"2024-08-17T17:26:39.410277+0000","last_fullsized":"2024-08-17T17:26:39.410277+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:56:32.241321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"212'8261","reported_seq":8338,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:45.342988+0000","last_change":"2024-08-17T17:24:14.641829+0000","last_active":"2024-08-17T17:26:45.342988+0000","last_peered":"2024-08-17T17:26:45.342988+0000","last_clean":"2024-08-17T17:26:45.342988+0000","last_became_active":"2024-08-17T17:24:14.641532+0000","last_became_peered":"2024-08-17T17:24:14.641532+0000","last_unstale":"2024-08-17T17:26:45.342988+0000","last_undegraded":"2024-08-17T17:26:45.342988+0000","last_fullsized":"2024-08-17T17:26:45.342988+0000","mapping_epoch":178,"log_start":"191'2000","ondisk_log_start":"191'2000","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6261,"log_dups_size":0,"ondisk_log_size":6261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:43:05.232036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260636672,"num_objects":3977,"num_object_clones":0,"num_object_copies":11931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8261,"num_write_kb":391616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":802,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-17T17:26:39.408323+0000","last_change":"2024-08-17T17:16:26.958415+0000","last_active":"2024-08-17T17:26:39.408323+0000","last_peered":"2024-08-17T17:26:39.408323+0000","last_clean":"2024-08-17T17:26:39.408323+0000","last_became_active":"2024-08-17T17:16:26.957956+0000","last_became_peered":"2024-08-17T17:16:26.957956+0000","last_unstale":"2024-08-17T17:26:39.408323+0000","last_undegraded":"2024-08-17T17:26:39.408323+0000","last_fullsized":"2024-08-17T17:26:39.408323+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:22.478077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":280,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.452581+0000","last_change":"2024-08-17T17:18:59.415331+0000","last_active":"2024-08-17T17:26:40.452581+0000","last_peered":"2024-08-17T17:26:40.452581+0000","last_clean":"2024-08-17T17:26:40.452581+0000","last_became_active":"2024-08-17T17:18:59.415008+0000","last_became_peered":"2024-08-17T17:18:59.415008+0000","last_unstale":"2024-08-17T17:26:40.452581+0000","last_undegraded":"2024-08-17T17:26:40.452581+0000","last_fullsized":"2024-08-17T17:26:40.452581+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:22:38.076866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":89,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.452355+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:26:40.452355+0000","last_peered":"2024-08-17T17:26:40.452355+0000","last_clean":"2024-08-17T17:26:40.452355+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:26:40.452355+0000","last_undegraded":"2024-08-17T17:26:40.452355+0000","last_fullsized":"2024-08-17T17:26:40.452355+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"212'8399","reported_seq":14677,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:48.948020+0000","last_change":"2024-08-17T17:26:40.426009+0000","last_active":"2024-08-17T17:26:48.948020+0000","last_peered":"2024-08-17T17:26:48.948020+0000","last_clean":"2024-08-17T17:26:48.948020+0000","last_became_active":"2024-08-17T17:26:40.411144+0000","last_became_peered":"2024-08-17T17:26:40.411144+0000","last_unstale":"2024-08-17T17:26:48.948020+0000","last_undegraded":"2024-08-17T17:26:48.948020+0000","last_fullsized":"2024-08-17T17:26:48.948020+0000","mapping_epoch":211,"log_start":"191'1400","ondisk_log_start":"191'1400","created":178,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6999,"log_dups_size":0,"ondisk_log_size":6999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:22.245859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247922688,"num_objects":3783,"num_object_clones":0,"num_object_copies":11349,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8399,"num_write_kb":389824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3108,"num_bytes_recovered":178388992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":668,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.408036+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:26:40.408036+0000","last_peered":"2024-08-17T17:26:40.408036+0000","last_clean":"2024-08-17T17:26:40.408036+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:26:40.408036+0000","last_undegraded":"2024-08-17T17:26:40.408036+0000","last_fullsized":"2024-08-17T17:26:40.408036+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":276,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.405704+0000","last_change":"2024-08-17T17:19:57.515564+0000","last_active":"2024-08-17T17:26:40.405704+0000","last_peered":"2024-08-17T17:26:40.405704+0000","last_clean":"2024-08-17T17:26:40.405704+0000","last_became_active":"2024-08-17T17:19:57.515142+0000","last_became_peered":"2024-08-17T17:19:57.515142+0000","last_unstale":"2024-08-17T17:26:40.405704+0000","last_undegraded":"2024-08-17T17:26:40.405704+0000","last_fullsized":"2024-08-17T17:26:40.405704+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:59:33.506176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":306,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.406036+0000","last_change":"2024-08-17T17:24:06.587903+0000","last_active":"2024-08-17T17:26:40.406036+0000","last_peered":"2024-08-17T17:26:40.406036+0000","last_clean":"2024-08-17T17:26:40.406036+0000","last_became_active":"2024-08-17T17:24:06.587503+0000","last_became_peered":"2024-08-17T17:24:06.587503+0000","last_unstale":"2024-08-17T17:26:40.406036+0000","last_undegraded":"2024-08-17T17:26:40.406036+0000","last_fullsized":"2024-08-17T17:26:40.406036+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:05:58.591992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"212'8276","reported_seq":14573,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:46.665066+0000","last_change":"2024-08-17T17:26:08.233981+0000","last_active":"2024-08-17T17:26:46.665066+0000","last_peered":"2024-08-17T17:26:46.665066+0000","last_clean":"2024-08-17T17:26:46.665066+0000","last_became_active":"2024-08-17T17:26:08.225711+0000","last_became_peered":"2024-08-17T17:26:08.225711+0000","last_unstale":"2024-08-17T17:26:46.665066+0000","last_undegraded":"2024-08-17T17:26:46.665066+0000","last_fullsized":"2024-08-17T17:26:46.665066+0000","mapping_epoch":207,"log_start":"191'1800","ondisk_log_start":"191'1800","created":178,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6476,"log_dups_size":0,"ondisk_log_size":6476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:31:42.806869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254672896,"num_objects":3886,"num_object_clones":0,"num_object_copies":11658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8276,"num_write_kb":389184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3101,"num_bytes_recovered":203227136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":1122,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-17T17:26:39.402864+0000","last_change":"2024-08-17T17:20:00.433870+0000","last_active":"2024-08-17T17:26:39.402864+0000","last_peered":"2024-08-17T17:26:39.402864+0000","last_clean":"2024-08-17T17:26:39.402864+0000","last_became_active":"2024-08-17T17:20:00.433386+0000","last_became_peered":"2024-08-17T17:20:00.433386+0000","last_unstale":"2024-08-17T17:26:39.402864+0000","last_undegraded":"2024-08-17T17:26:39.402864+0000","last_fullsized":"2024-08-17T17:26:39.402864+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:01:11.848862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1392,"num_read_kb":2863,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":201,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.452784+0000","last_change":"2024-08-17T17:20:49.272407+0000","last_active":"2024-08-17T17:26:40.452784+0000","last_peered":"2024-08-17T17:26:40.452784+0000","last_clean":"2024-08-17T17:26:40.452784+0000","last_became_active":"2024-08-17T17:20:49.271749+0000","last_became_peered":"2024-08-17T17:20:49.271749+0000","last_unstale":"2024-08-17T17:26:40.452784+0000","last_undegraded":"2024-08-17T17:26:40.452784+0000","last_fullsized":"2024-08-17T17:26:40.452784+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:36:01.443457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":272,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.406668+0000","last_change":"2024-08-17T17:17:32.499715+0000","last_active":"2024-08-17T17:26:40.406668+0000","last_peered":"2024-08-17T17:26:40.406668+0000","last_clean":"2024-08-17T17:26:40.406668+0000","last_became_active":"2024-08-17T17:17:32.499412+0000","last_became_peered":"2024-08-17T17:17:32.499412+0000","last_unstale":"2024-08-17T17:26:40.406668+0000","last_undegraded":"2024-08-17T17:26:40.406668+0000","last_fullsized":"2024-08-17T17:26:40.406668+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:39:38.456970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"212'8564","reported_seq":8644,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:47.979359+0000","last_change":"2024-08-17T17:24:14.642608+0000","last_active":"2024-08-17T17:26:47.979359+0000","last_peered":"2024-08-17T17:26:47.979359+0000","last_clean":"2024-08-17T17:26:47.979359+0000","last_became_active":"2024-08-17T17:24:14.642292+0000","last_became_peered":"2024-08-17T17:24:14.642292+0000","last_unstale":"2024-08-17T17:26:47.979359+0000","last_undegraded":"2024-08-17T17:26:47.979359+0000","last_fullsized":"2024-08-17T17:26:47.979359+0000","mapping_epoch":178,"log_start":"186'600","ondisk_log_start":"186'600","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":7964,"log_dups_size":0,"ondisk_log_size":7964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:53:36.848614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246415360,"num_objects":3760,"num_object_clones":0,"num_object_copies":11280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8564,"num_write_kb":394368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":187,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-17T17:26:39.403239+0000","last_change":"2024-08-17T17:17:32.495728+0000","last_active":"2024-08-17T17:26:39.403239+0000","last_peered":"2024-08-17T17:26:39.403239+0000","last_clean":"2024-08-17T17:26:39.403239+0000","last_became_active":"2024-08-17T17:17:32.495357+0000","last_became_peered":"2024-08-17T17:17:32.495357+0000","last_unstale":"2024-08-17T17:26:39.403239+0000","last_undegraded":"2024-08-17T17:26:39.403239+0000","last_fullsized":"2024-08-17T17:26:39.403239+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:27:45.561567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":508,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.406831+0000","last_change":"2024-08-17T17:16:26.964621+0000","last_active":"2024-08-17T17:26:40.406831+0000","last_peered":"2024-08-17T17:26:40.406831+0000","last_clean":"2024-08-17T17:26:40.406831+0000","last_became_active":"2024-08-17T17:16:26.964058+0000","last_became_peered":"2024-08-17T17:16:26.964058+0000","last_unstale":"2024-08-17T17:26:40.406831+0000","last_undegraded":"2024-08-17T17:26:40.406831+0000","last_fullsized":"2024-08-17T17:26:40.406831+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:00:04.465644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":278,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.405836+0000","last_change":"2024-08-17T17:24:53.465550+0000","last_active":"2024-08-17T17:26:40.405836+0000","last_peered":"2024-08-17T17:26:40.405836+0000","last_clean":"2024-08-17T17:26:40.405836+0000","last_became_active":"2024-08-17T17:24:53.465228+0000","last_became_peered":"2024-08-17T17:24:53.465228+0000","last_unstale":"2024-08-17T17:26:40.405836+0000","last_undegraded":"2024-08-17T17:26:40.405836+0000","last_fullsized":"2024-08-17T17:26:40.405836+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:21:18.378359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"212'8476","reported_seq":8556,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:45.338351+0000","last_change":"2024-08-17T17:24:14.640584+0000","last_active":"2024-08-17T17:26:45.338351+0000","last_peered":"2024-08-17T17:26:45.338351+0000","last_clean":"2024-08-17T17:26:45.338351+0000","last_became_active":"2024-08-17T17:24:14.640168+0000","last_became_peered":"2024-08-17T17:24:14.640168+0000","last_unstale":"2024-08-17T17:26:45.338351+0000","last_undegraded":"2024-08-17T17:26:45.338351+0000","last_fullsized":"2024-08-17T17:26:45.338351+0000","mapping_epoch":178,"log_start":"192'2200","ondisk_log_start":"192'2200","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6276,"log_dups_size":0,"ondisk_log_size":6276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:38:36.176813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257425408,"num_objects":3928,"num_object_clones":0,"num_object_copies":11784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8476,"num_write_kb":396928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":303,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.406744+0000","last_change":"2024-08-17T17:24:53.464307+0000","last_active":"2024-08-17T17:26:40.406744+0000","last_peered":"2024-08-17T17:26:40.406744+0000","last_clean":"2024-08-17T17:26:40.406744+0000","last_became_active":"2024-08-17T17:24:53.463996+0000","last_became_peered":"2024-08-17T17:24:53.463996+0000","last_unstale":"2024-08-17T17:26:40.406744+0000","last_undegraded":"2024-08-17T17:26:40.406744+0000","last_fullsized":"2024-08-17T17:26:40.406744+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:04:18.993725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":639,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-17T17:26:39.408803+0000","last_change":"2024-08-17T17:24:09.599822+0000","last_active":"2024-08-17T17:26:39.408803+0000","last_peered":"2024-08-17T17:26:39.408803+0000","last_clean":"2024-08-17T17:26:39.408803+0000","last_became_active":"2024-08-17T17:24:09.599189+0000","last_became_peered":"2024-08-17T17:24:09.599189+0000","last_unstale":"2024-08-17T17:26:39.408803+0000","last_undegraded":"2024-08-17T17:26:39.408803+0000","last_fullsized":"2024-08-17T17:26:39.408803+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:04.465604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":263,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.452893+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:26:40.452893+0000","last_peered":"2024-08-17T17:26:40.452893+0000","last_clean":"2024-08-17T17:26:40.452893+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:26:40.452893+0000","last_undegraded":"2024-08-17T17:26:40.452893+0000","last_fullsized":"2024-08-17T17:26:40.452893+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"212'8706","reported_seq":8783,"reported_epoch":212,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:26:46.694031+0000","last_change":"2024-08-17T17:25:05.605653+0000","last_active":"2024-08-17T17:26:46.694031+0000","last_peered":"2024-08-17T17:26:46.694031+0000","last_clean":"2024-08-17T17:25:03.971607+0000","last_became_active":"2024-08-17T17:25:05.601333+0000","last_became_peered":"2024-08-17T17:25:05.601333+0000","last_unstale":"2024-08-17T17:26:46.694031+0000","last_undegraded":"2024-08-17T17:25:05.522007+0000","last_fullsized":"2024-08-17T17:25:05.499355+0000","mapping_epoch":198,"log_start":"192'2300","ondisk_log_start":"192'2300","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6406,"log_dups_size":0,"ondisk_log_size":6406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251789312,"num_objects":3842,"num_object_clones":0,"num_object_copies":11526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8706,"num_write_kb":401536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2],"avail_no_missing":["3","2","7"],"object_location_counts":[{"shards":"2,3","objects":526},{"shards":"2,3,7","objects":884}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":298,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.410640+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:26:40.410640+0000","last_peered":"2024-08-17T17:26:40.410640+0000","last_clean":"2024-08-17T17:26:40.410640+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:26:40.410640+0000","last_undegraded":"2024-08-17T17:26:40.410640+0000","last_fullsized":"2024-08-17T17:26:40.410640+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":817,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.406107+0000","last_change":"2024-08-17T17:23:19.633786+0000","last_active":"2024-08-17T17:26:40.406107+0000","last_peered":"2024-08-17T17:26:40.406107+0000","last_clean":"2024-08-17T17:26:40.406107+0000","last_became_active":"2024-08-17T17:23:19.633230+0000","last_became_peered":"2024-08-17T17:23:19.633230+0000","last_unstale":"2024-08-17T17:26:40.406107+0000","last_undegraded":"2024-08-17T17:26:40.406107+0000","last_fullsized":"2024-08-17T17:26:40.406107+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:30:50.377391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":294,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.407770+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:26:40.407770+0000","last_peered":"2024-08-17T17:26:40.407770+0000","last_clean":"2024-08-17T17:26:40.407770+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:26:40.407770+0000","last_undegraded":"2024-08-17T17:26:40.407770+0000","last_fullsized":"2024-08-17T17:26:40.407770+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"212'8562","reported_seq":8604,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:49.032220+0000","last_change":"2024-08-17T17:24:14.639560+0000","last_active":"2024-08-17T17:26:49.032220+0000","last_peered":"2024-08-17T17:26:49.032220+0000","last_clean":"2024-08-17T17:26:49.032220+0000","last_became_active":"2024-08-17T17:24:14.639093+0000","last_became_peered":"2024-08-17T17:24:14.639093+0000","last_unstale":"2024-08-17T17:26:49.032220+0000","last_undegraded":"2024-08-17T17:26:49.032220+0000","last_fullsized":"2024-08-17T17:26:49.032220+0000","mapping_epoch":178,"log_start":"191'1500","ondisk_log_start":"191'1500","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":7062,"log_dups_size":0,"ondisk_log_size":7062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:45:57.940550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249954304,"num_objects":3814,"num_object_clones":0,"num_object_copies":11442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8562,"num_write_kb":396032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":93,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.452260+0000","last_change":"2024-08-17T17:24:06.593660+0000","last_active":"2024-08-17T17:26:40.452260+0000","last_peered":"2024-08-17T17:26:40.452260+0000","last_clean":"2024-08-17T17:26:40.452260+0000","last_became_active":"2024-08-17T17:24:06.586215+0000","last_became_peered":"2024-08-17T17:24:06.586215+0000","last_unstale":"2024-08-17T17:26:40.452260+0000","last_undegraded":"2024-08-17T17:26:40.452260+0000","last_fullsized":"2024-08-17T17:26:40.452260+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:15:49.475395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":443,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-17T17:26:39.410396+0000","last_change":"2024-08-17T17:26:06.855307+0000","last_active":"2024-08-17T17:26:39.410396+0000","last_peered":"2024-08-17T17:26:39.410396+0000","last_clean":"2024-08-17T17:26:39.410396+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:26:39.410396+0000","last_undegraded":"2024-08-17T17:26:39.410396+0000","last_fullsized":"2024-08-17T17:26:39.410396+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":111,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-17T17:26:39.402709+0000","last_change":"2024-08-17T17:18:59.417183+0000","last_active":"2024-08-17T17:26:39.402709+0000","last_peered":"2024-08-17T17:26:39.402709+0000","last_clean":"2024-08-17T17:26:39.402709+0000","last_became_active":"2024-08-17T17:18:59.416843+0000","last_became_peered":"2024-08-17T17:18:59.416843+0000","last_unstale":"2024-08-17T17:26:39.402709+0000","last_undegraded":"2024-08-17T17:26:39.402709+0000","last_fullsized":"2024-08-17T17:26:39.402709+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:29:09.336335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":132,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-17T17:26:39.410028+0000","last_change":"2024-08-17T17:20:49.258048+0000","last_active":"2024-08-17T17:26:39.410028+0000","last_peered":"2024-08-17T17:26:39.410028+0000","last_clean":"2024-08-17T17:26:39.410028+0000","last_became_active":"2024-08-17T17:20:49.257672+0000","last_became_peered":"2024-08-17T17:20:49.257672+0000","last_unstale":"2024-08-17T17:26:39.410028+0000","last_undegraded":"2024-08-17T17:26:39.410028+0000","last_fullsized":"2024-08-17T17:26:39.410028+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:08:11.734316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":171,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-17T17:26:39.408418+0000","last_change":"2024-08-17T17:18:59.430969+0000","last_active":"2024-08-17T17:26:39.408418+0000","last_peered":"2024-08-17T17:26:39.408418+0000","last_clean":"2024-08-17T17:26:39.408418+0000","last_became_active":"2024-08-17T17:18:59.413475+0000","last_became_peered":"2024-08-17T17:18:59.413475+0000","last_unstale":"2024-08-17T17:26:39.408418+0000","last_undegraded":"2024-08-17T17:26:39.408418+0000","last_fullsized":"2024-08-17T17:26:39.408418+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:36:33.732645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":289,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.405708+0000","last_change":"2024-08-17T17:25:04.506413+0000","last_active":"2024-08-17T17:26:40.405708+0000","last_peered":"2024-08-17T17:26:40.405708+0000","last_clean":"2024-08-17T17:26:40.405708+0000","last_became_active":"2024-08-17T17:25:04.506104+0000","last_became_peered":"2024-08-17T17:25:04.506104+0000","last_unstale":"2024-08-17T17:26:40.405708+0000","last_undegraded":"2024-08-17T17:26:40.405708+0000","last_fullsized":"2024-08-17T17:26:40.405708+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:41:53.556223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":114,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-17T17:26:39.408448+0000","last_change":"2024-08-17T17:20:49.279458+0000","last_active":"2024-08-17T17:26:39.408448+0000","last_peered":"2024-08-17T17:26:39.408448+0000","last_clean":"2024-08-17T17:26:39.408448+0000","last_became_active":"2024-08-17T17:20:49.260790+0000","last_became_peered":"2024-08-17T17:20:49.260790+0000","last_unstale":"2024-08-17T17:26:39.408448+0000","last_undegraded":"2024-08-17T17:26:39.408448+0000","last_fullsized":"2024-08-17T17:26:39.408448+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:58.092945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":28,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-17T17:26:39.404160+0000","last_change":"2024-08-17T17:24:53.463713+0000","last_active":"2024-08-17T17:26:39.404160+0000","last_peered":"2024-08-17T17:26:39.404160+0000","last_clean":"2024-08-17T17:26:39.404160+0000","last_became_active":"2024-08-17T17:24:53.463419+0000","last_became_peered":"2024-08-17T17:24:53.463419+0000","last_unstale":"2024-08-17T17:26:39.404160+0000","last_undegraded":"2024-08-17T17:26:39.404160+0000","last_fullsized":"2024-08-17T17:26:39.404160+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:34:44.545175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":244,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.407809+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:26:40.407809+0000","last_peered":"2024-08-17T17:26:40.407809+0000","last_clean":"2024-08-17T17:26:40.407809+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:26:40.407809+0000","last_undegraded":"2024-08-17T17:26:40.407809+0000","last_fullsized":"2024-08-17T17:26:40.407809+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":272,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.453117+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:26:40.453117+0000","last_peered":"2024-08-17T17:26:40.453117+0000","last_clean":"2024-08-17T17:26:40.453117+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:26:40.453117+0000","last_undegraded":"2024-08-17T17:26:40.453117+0000","last_fullsized":"2024-08-17T17:26:40.453117+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":291,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.407680+0000","last_change":"2024-08-17T17:25:04.509422+0000","last_active":"2024-08-17T17:26:40.407680+0000","last_peered":"2024-08-17T17:26:40.407680+0000","last_clean":"2024-08-17T17:26:40.407680+0000","last_became_active":"2024-08-17T17:25:04.508742+0000","last_became_peered":"2024-08-17T17:25:04.508742+0000","last_unstale":"2024-08-17T17:26:40.407680+0000","last_undegraded":"2024-08-17T17:26:40.407680+0000","last_fullsized":"2024-08-17T17:26:40.407680+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:18:31.976907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":169,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-17T17:26:39.410283+0000","last_change":"2024-08-17T17:16:26.963696+0000","last_active":"2024-08-17T17:26:39.410283+0000","last_peered":"2024-08-17T17:26:39.410283+0000","last_clean":"2024-08-17T17:26:39.410283+0000","last_became_active":"2024-08-17T17:16:26.963280+0000","last_became_peered":"2024-08-17T17:16:26.963280+0000","last_unstale":"2024-08-17T17:26:39.410283+0000","last_undegraded":"2024-08-17T17:26:39.410283+0000","last_fullsized":"2024-08-17T17:26:39.410283+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:22:01.474840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":309,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.407919+0000","last_change":"2024-08-17T17:24:06.586886+0000","last_active":"2024-08-17T17:26:40.407919+0000","last_peered":"2024-08-17T17:26:40.407919+0000","last_clean":"2024-08-17T17:26:40.407919+0000","last_became_active":"2024-08-17T17:24:06.586516+0000","last_became_peered":"2024-08-17T17:24:06.586516+0000","last_unstale":"2024-08-17T17:26:40.407919+0000","last_undegraded":"2024-08-17T17:26:40.407919+0000","last_fullsized":"2024-08-17T17:26:40.407919+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:03:30.002726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":89,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.452090+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:26:40.452090+0000","last_peered":"2024-08-17T17:26:40.452090+0000","last_clean":"2024-08-17T17:26:40.452090+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:26:40.452090+0000","last_undegraded":"2024-08-17T17:26:40.452090+0000","last_fullsized":"2024-08-17T17:26:40.452090+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":269,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.406385+0000","last_change":"2024-08-17T17:18:59.412463+0000","last_active":"2024-08-17T17:26:40.406385+0000","last_peered":"2024-08-17T17:26:40.406385+0000","last_clean":"2024-08-17T17:26:40.406385+0000","last_became_active":"2024-08-17T17:18:59.412134+0000","last_became_peered":"2024-08-17T17:18:59.412134+0000","last_unstale":"2024-08-17T17:26:40.406385+0000","last_undegraded":"2024-08-17T17:26:40.406385+0000","last_fullsized":"2024-08-17T17:26:40.406385+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:44.463153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":312,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.410097+0000","last_change":"2024-08-17T17:25:04.510854+0000","last_active":"2024-08-17T17:26:40.410097+0000","last_peered":"2024-08-17T17:26:40.410097+0000","last_clean":"2024-08-17T17:26:40.410097+0000","last_became_active":"2024-08-17T17:25:04.510462+0000","last_became_peered":"2024-08-17T17:25:04.510462+0000","last_unstale":"2024-08-17T17:26:40.410097+0000","last_undegraded":"2024-08-17T17:26:40.410097+0000","last_fullsized":"2024-08-17T17:26:40.410097+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:55:54.762086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":209,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.405855+0000","last_change":"2024-08-17T17:24:06.591165+0000","last_active":"2024-08-17T17:26:40.405855+0000","last_peered":"2024-08-17T17:26:40.405855+0000","last_clean":"2024-08-17T17:26:40.405855+0000","last_became_active":"2024-08-17T17:24:06.590785+0000","last_became_peered":"2024-08-17T17:24:06.590785+0000","last_unstale":"2024-08-17T17:26:40.405855+0000","last_undegraded":"2024-08-17T17:26:40.405855+0000","last_fullsized":"2024-08-17T17:26:40.405855+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:54:05.775563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":281,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.406968+0000","last_change":"2024-08-17T17:24:32.300560+0000","last_active":"2024-08-17T17:26:40.406968+0000","last_peered":"2024-08-17T17:26:40.406968+0000","last_clean":"2024-08-17T17:26:40.406968+0000","last_became_active":"2024-08-17T17:24:32.300228+0000","last_became_peered":"2024-08-17T17:24:32.300228+0000","last_unstale":"2024-08-17T17:26:40.406968+0000","last_undegraded":"2024-08-17T17:26:40.406968+0000","last_fullsized":"2024-08-17T17:26:40.406968+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:18:05.475449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":89,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.452725+0000","last_change":"2024-08-17T17:24:06.586379+0000","last_active":"2024-08-17T17:26:40.452725+0000","last_peered":"2024-08-17T17:26:40.452725+0000","last_clean":"2024-08-17T17:26:40.452725+0000","last_became_active":"2024-08-17T17:24:06.586023+0000","last_became_peered":"2024-08-17T17:24:06.586023+0000","last_unstale":"2024-08-17T17:26:40.452725+0000","last_undegraded":"2024-08-17T17:26:40.452725+0000","last_fullsized":"2024-08-17T17:26:40.452725+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:45.701637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":299,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.409919+0000","last_change":"2024-08-17T17:21:50.442535+0000","last_active":"2024-08-17T17:26:40.409919+0000","last_peered":"2024-08-17T17:26:40.409919+0000","last_clean":"2024-08-17T17:26:40.409919+0000","last_became_active":"2024-08-17T17:21:50.441891+0000","last_became_peered":"2024-08-17T17:21:50.441891+0000","last_unstale":"2024-08-17T17:26:40.409919+0000","last_undegraded":"2024-08-17T17:26:40.409919+0000","last_fullsized":"2024-08-17T17:26:40.409919+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:46:24.025407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":224,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-17T17:26:39.405462+0000","last_change":"2024-08-17T17:25:04.508045+0000","last_active":"2024-08-17T17:26:39.405462+0000","last_peered":"2024-08-17T17:26:39.405462+0000","last_clean":"2024-08-17T17:26:39.405462+0000","last_became_active":"2024-08-17T17:25:04.505083+0000","last_became_peered":"2024-08-17T17:25:04.505083+0000","last_unstale":"2024-08-17T17:26:39.405462+0000","last_undegraded":"2024-08-17T17:26:39.405462+0000","last_fullsized":"2024-08-17T17:26:39.405462+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:23:42.993544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":285,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.408497+0000","last_change":"2024-08-17T17:18:59.418434+0000","last_active":"2024-08-17T17:26:40.408497+0000","last_peered":"2024-08-17T17:26:40.408497+0000","last_clean":"2024-08-17T17:26:40.408497+0000","last_became_active":"2024-08-17T17:18:59.418043+0000","last_became_peered":"2024-08-17T17:18:59.418043+0000","last_unstale":"2024-08-17T17:26:40.408497+0000","last_undegraded":"2024-08-17T17:26:40.408497+0000","last_fullsized":"2024-08-17T17:26:40.408497+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:25:51.371820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":197,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-17T17:26:39.403153+0000","last_change":"2024-08-17T17:20:49.291167+0000","last_active":"2024-08-17T17:26:39.403153+0000","last_peered":"2024-08-17T17:26:39.403153+0000","last_clean":"2024-08-17T17:26:39.403153+0000","last_became_active":"2024-08-17T17:20:49.263750+0000","last_became_peered":"2024-08-17T17:20:49.263750+0000","last_unstale":"2024-08-17T17:26:39.403153+0000","last_undegraded":"2024-08-17T17:26:39.403153+0000","last_fullsized":"2024-08-17T17:26:39.403153+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:02.863246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":298,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.408133+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:26:40.408133+0000","last_peered":"2024-08-17T17:26:40.408133+0000","last_clean":"2024-08-17T17:26:40.408133+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:26:40.408133+0000","last_undegraded":"2024-08-17T17:26:40.408133+0000","last_fullsized":"2024-08-17T17:26:40.408133+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":298,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.410425+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:26:40.410425+0000","last_peered":"2024-08-17T17:26:40.410425+0000","last_clean":"2024-08-17T17:26:40.410425+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:26:40.410425+0000","last_undegraded":"2024-08-17T17:26:40.410425+0000","last_fullsized":"2024-08-17T17:26:40.410425+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-08-17T17:26:39.409880+0000","last_change":"2024-08-17T17:25:04.503834+0000","last_active":"2024-08-17T17:26:39.409880+0000","last_peered":"2024-08-17T17:26:39.409880+0000","last_clean":"2024-08-17T17:26:39.409880+0000","last_became_active":"2024-08-17T17:25:04.502712+0000","last_became_peered":"2024-08-17T17:25:04.502712+0000","last_unstale":"2024-08-17T17:26:39.409880+0000","last_undegraded":"2024-08-17T17:26:39.409880+0000","last_fullsized":"2024-08-17T17:26:39.409880+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:45.389103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":108,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.453031+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:26:40.453031+0000","last_peered":"2024-08-17T17:26:40.453031+0000","last_clean":"2024-08-17T17:26:40.453031+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:26:40.453031+0000","last_undegraded":"2024-08-17T17:26:40.453031+0000","last_fullsized":"2024-08-17T17:26:40.453031+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":4036689944,"num_objects":61596,"num_object_clones":0,"num_object_copies":184788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61596,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135698,"num_write_kb":6313345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11884,"num_bytes_recovered":750911488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12980924416,"data_stored":12980912200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113198,"ondisk_log_size":113198,"up":48,"acting":44,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":6304,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":16702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17971,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16789504,"data_stored":16777273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234240,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2351728,"kb_used_data":2046056,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91393424,"statfs":{"total":95995035648,"available":93586866176,"internally_reserved":0,"allocated":2095161344,"data_stored":2093936684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495875,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2161256,"kb_used_data":1851424,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91583896,"statfs":{"total":95995035648,"available":93781909504,"internally_reserved":0,"allocated":1895858176,"data_stored":1894641408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757512,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1752800,"kb_used_data":1455448,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":91992352,"statfs":{"total":95995035648,"available":94200168448,"internally_reserved":0,"allocated":1490378752,"data_stored":1489165655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019150,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435377,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2798744,"kb_used_data":2501776,"kb_used_omap":4,"kb_used_meta":296891,"kb_avail":90946408,"statfs":{"total":95995035648,"available":93129121792,"internally_reserved":0,"allocated":2561818624,"data_stored":2560613469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":304016394},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,5,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":104,"seq":446676598904,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2431656,"kb_used_data":2108704,"kb_used_omap":7,"kb_used_meta":322808,"kb_avail":91313496,"statfs":{"total":95995035648,"available":93505019904,"internally_reserved":0,"allocated":2159312896,"data_stored":2158096623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":330555554},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762430,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2186400,"kb_used_data":1856216,"kb_used_omap":7,"kb_used_meta":330168,"kb_avail":91558752,"statfs":{"total":95995035648,"available":93756162048,"internally_reserved":0,"allocated":1900765184,"data_stored":1899551828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7945,"internal_metadata":338092279},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,3,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":94,"seq":403726925957,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1240424,"kb_used_data":970272,"kb_used_omap":7,"kb_used_meta":270008,"kb_avail":92504728,"statfs":{"total":95995035648,"available":94724841472,"internally_reserved":0,"allocated":993558528,"data_stored":992338165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":276488513},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6241,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":990121984,"data_stored":990117912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1884164096,"data_stored":1884160024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2126184448,"data_stored":2126184448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2539913216,"data_stored":2539913216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1483476992,"data_stored":1483472920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1872494592,"data_stored":1872494592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2084569088,"data_stored":2084569088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:26:52.031 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:52.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:52.079 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:52.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:52.180 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:52.208 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:52.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:52.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:52.310 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:52.362 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:52.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:52.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:52.464 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:52.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:52.504 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:52.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:52.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:52 smithi003 ceph-mon[103344]: from='client.35715 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:26:52.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:52 smithi003 ceph-mon[100615]: from='client.35715 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:26:52.544 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:26:52.606 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:52.636 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:52.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:52.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:52.738 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:52.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:52.817 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:52.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:52.918 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:52.954 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:52.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:52.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:52.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:52 smithi053 ceph-mon[89963]: from='client.35715 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:26:53.056 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:53.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:53.106 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:53.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:53.207 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:53.234 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:53.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:53.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:53.335 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:53.362 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:53.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:53.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:53.464 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:53.503 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:53.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:53.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:53.605 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:53.638 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:53.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:53.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:53.740 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:53.762 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:53.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:53.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:53 smithi003 ceph-mon[100615]: pgmap v552: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 733 op/s; 8833/178971 objects degraded (4.935%); 6.4 MiB/s, 106 objects/s recovering 2024-08-17T17:26:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:26:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:26:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:53 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1035028290' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:26:53.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:53 smithi003 ceph-mon[103344]: pgmap v552: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 733 op/s; 8833/178971 objects degraded (4.935%); 6.4 MiB/s, 106 objects/s recovering 2024-08-17T17:26:53.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:26:53.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:26:53.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:53 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1035028290' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:26:53.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:53.903 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:53.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:53.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:53.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:53 smithi053 ceph-mon[89963]: pgmap v552: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 733 op/s; 8833/178971 objects degraded (4.935%); 6.4 MiB/s, 106 objects/s recovering 2024-08-17T17:26:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:26:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:26:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:53 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1035028290' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:26:54.005 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:54.046 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:54.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:54.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:54.148 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:54.179 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:54.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:54.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:54.281 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:54.318 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:54.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:54.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:54.420 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:54.452 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:54.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:54.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:54.554 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:54.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:54.578 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:54.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:54.679 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:54.709 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:54.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:54.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:54.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:54 smithi003 ceph-mon[103344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:26:54.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:54 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1035028290' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:26:54.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:54 smithi003 ceph-mon[103344]: osdmap e213: 8 total, 8 up, 7 in 2024-08-17T17:26:54.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:54 smithi003 ceph-mon[100615]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:26:54.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:54 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1035028290' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:26:54.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:54 smithi003 ceph-mon[100615]: osdmap e213: 8 total, 8 up, 7 in 2024-08-17T17:26:54.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:54.833 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:54.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:54.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:54.935 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:54.966 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:54.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:54 smithi053 ceph-mon[89963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:26:54.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:54 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1035028290' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:26:54.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:54 smithi053 ceph-mon[89963]: osdmap e213: 8 total, 8 up, 7 in 2024-08-17T17:26:54.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:54.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:54.978 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:26:55.068 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:55.091 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:55.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:55.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:55.193 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:55.220 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:55.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:55.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:55.322 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:55.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:55.365 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:55.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:55.466 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:55.494 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:55.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:55.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:55.596 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:55.627 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:55.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:55.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:55.729 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:55.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:55.766 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:55.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:55.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:55 smithi003 ceph-mon[103344]: pgmap v553: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 710 op/s; 8511/175356 objects degraded (4.854%); 4.9 MiB/s, 109 objects/s recovering 2024-08-17T17:26:55.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:55 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 8833/178971 objects degraded (4.935%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:55.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:55 smithi003 ceph-mon[100615]: pgmap v553: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 710 op/s; 8511/175356 objects degraded (4.854%); 4.9 MiB/s, 109 objects/s recovering 2024-08-17T17:26:55.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:55 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 8833/178971 objects degraded (4.935%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:55.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:26:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:26:55] "GET /metrics HTTP/1.1" 200 36526 "" "Prometheus/2.51.0" 2024-08-17T17:26:55.867 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:55.896 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:55.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:55.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:55.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:55 smithi053 ceph-mon[89963]: pgmap v553: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 710 op/s; 8511/175356 objects degraded (4.854%); 4.9 MiB/s, 109 objects/s recovering 2024-08-17T17:26:55.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:55 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 8833/178971 objects degraded (4.935%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T17:26:55.998 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:56.035 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:56.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:56.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:56.137 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:56.162 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:56.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:56.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:56.265 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:56.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:56.295 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:56.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:56.397 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:56.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:56.430 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:56.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:56.530 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:56.552 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:56.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:56.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:56.654 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:26:56.681 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:56.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:56.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:26:56.783 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:56.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:56.840 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:56.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:56.941 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:56.981 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:56.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:56.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:57.082 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:57.107 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:57.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:57.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:57.209 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:57.252 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:57.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:57.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:57.354 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:57.388 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:57.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:57.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:57.489 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:57.519 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:57.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:57.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:57.621 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:57.651 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:57.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:57.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:57.753 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:26:57.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:57 smithi003 ceph-mon[103344]: pgmap v555: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.5 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 762 op/s; 8511/171072 objects degraded (4.975%); 4.1 MiB/s, 102 objects/s recovering 2024-08-17T17:26:57.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:57 smithi003 ceph-mon[100615]: pgmap v555: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.5 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 762 op/s; 8511/171072 objects degraded (4.975%); 4.1 MiB/s, 102 objects/s recovering 2024-08-17T17:26:57.780 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:57.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:57.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:26:57.882 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:57.910 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:57.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:57.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:57.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:57 smithi053 ceph-mon[89963]: pgmap v555: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.5 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 762 op/s; 8511/171072 objects degraded (4.975%); 4.1 MiB/s, 102 objects/s recovering 2024-08-17T17:26:58.012 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:58.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:58.040 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:58.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:58.140 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:58.170 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:58.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:58.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:58.272 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:26:58.319 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:58.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:58.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:26:58.421 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:26:58.458 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:58.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:58.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:26:58.560 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:58.588 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:58.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:58.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:58.690 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:58.712 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:58.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:58.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:58.813 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:58.842 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:58.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:58.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:58.944 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:58.979 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:58.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:58.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:59.081 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:26:59.121 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:59.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:59.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:26:59.223 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:59.245 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:59.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:59.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:59.329 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-08-17T17:26:59.347 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:59.373 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:59.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:59.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:59.475 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:59.505 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:59.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:59.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:59.607 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:26:59.642 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:26:59.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:59.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:26:59.744 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:26:59.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:26:59 smithi003 ceph-mon[103344]: pgmap v556: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.4 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 752 op/s; 8511/167649 objects degraded (5.077%); 2.7 MiB/s, 78 objects/s recovering 2024-08-17T17:26:59.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:26:59 smithi003 ceph-mon[100615]: pgmap v556: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.4 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 752 op/s; 8511/167649 objects degraded (5.077%); 2.7 MiB/s, 78 objects/s recovering 2024-08-17T17:26:59.773 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:59.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:59.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:26:59.875 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:26:59.904 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:26:59.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:26:59.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:26:59.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:26:59 smithi053 ceph-mon[89963]: pgmap v556: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.4 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 752 op/s; 8511/167649 objects degraded (5.077%); 2.7 MiB/s, 78 objects/s recovering 2024-08-17T17:27:00.006 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:00.037 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:00.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:00.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:00.139 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:00.171 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:00.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:00.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:00.273 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:00.304 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:00.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:00.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:00.406 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:00.434 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:00.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:00.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:00.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:27:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:27:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:27:00.536 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:00.564 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:00.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:00.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:00.666 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:00.699 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:00.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:00.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:00.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:00 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 8511/167649 objects degraded (5.077%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:00.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:00 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 8511/167649 objects degraded (5.077%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:00.801 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:00.828 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:00.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:00.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:00.930 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:00.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:00.961 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:00.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:00.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:00 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 8511/167649 objects degraded (5.077%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:01.061 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:01.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:01.099 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:01.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:01.200 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:01.229 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:01.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:01.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:01.330 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:01.369 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:01.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:01.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:01.471 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:01.500 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:01.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:01.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:01.602 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:01.630 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:01.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:01.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:01.732 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:01.762 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:01.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:01.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:01.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:01 smithi003 ceph-mon[103344]: pgmap v557: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 749 op/s; 8099/162384 objects degraded (4.988%); 1.3 MiB/s, 93 objects/s recovering 2024-08-17T17:27:01.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:01 smithi003 ceph-mon[100615]: pgmap v557: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 749 op/s; 8099/162384 objects degraded (4.988%); 1.3 MiB/s, 93 objects/s recovering 2024-08-17T17:27:01.862 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:27:01.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:01.871 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:27:01.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:01.918 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:01.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:01.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:01 smithi053 ceph-mon[89963]: pgmap v557: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 749 op/s; 8099/162384 objects degraded (4.988%); 1.3 MiB/s, 93 objects/s recovering 2024-08-17T17:27:02.019 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:02.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:02.063 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:02.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:02.164 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:02.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:02.202 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:02.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:02.303 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:02.330 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:02.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:02.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:02.432 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:02.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:02.481 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:02.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:02.502 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:27:02.502 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:27:02.583 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:02.608 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:02.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:02.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:02.710 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:02.739 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:02.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:02.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:02.841 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:02.867 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:02.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:02.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:02.969 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:03.006 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:03.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:03.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:03.013 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:27:03.109 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:03.155 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:03.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:03.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:03.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:03.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:03.299 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:03.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:03.400 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:03.428 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:03.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:03.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:03.530 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:03.558 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:03.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:03.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:03.660 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:03.702 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:03.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:03.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:03.769 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:03 smithi053 ceph-mon[89963]: pgmap v558: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 738 op/s; 8099/156810 objects degraded (5.165%); 0 B/s, 73 objects/s recovering 2024-08-17T17:27:03.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:03 smithi003 ceph-mon[103344]: pgmap v558: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 738 op/s; 8099/156810 objects degraded (5.165%); 0 B/s, 73 objects/s recovering 2024-08-17T17:27:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:03 smithi003 ceph-mon[100615]: pgmap v558: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 738 op/s; 8099/156810 objects degraded (5.165%); 0 B/s, 73 objects/s recovering 2024-08-17T17:27:03.804 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:03.838 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:03.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:03.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:03.940 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:03.976 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:03.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:03.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:04.078 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:04.081 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-17T17:27:04.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:04.126 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:04.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:04.227 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:04.313 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:04.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:04.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:04.415 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:04.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:04.472 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:04.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:04.573 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:04.611 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:04.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:04.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:04.715 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:04.765 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:04.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:04.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:04.868 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:04.897 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:04.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:04.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:04.999 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:05.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:05.031 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:05.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:05.132 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:05.163 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:05.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:05.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:05.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:05.302 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:05.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:05.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:05.404 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:05.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:05.475 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:05.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:05.576 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:05.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:05.626 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:05.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:05.727 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:05.763 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:05.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:05.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:05.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:05 smithi003 ceph-mon[103344]: pgmap v559: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 747 op/s; 7706/152910 objects degraded (5.040%); 0 B/s, 80 objects/s recovering 2024-08-17T17:27:05.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:05 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 7706/152910 objects degraded (5.040%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:05.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:05 smithi003 ceph-mon[100615]: pgmap v559: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 747 op/s; 7706/152910 objects degraded (5.040%); 0 B/s, 80 objects/s recovering 2024-08-17T17:27:05.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:05 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 7706/152910 objects degraded (5.040%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:05.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:27:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:27:05] "GET /metrics HTTP/1.1" 200 36526 "" "Prometheus/2.51.0" 2024-08-17T17:27:05.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:05.900 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:05.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:05.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:05.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:05 smithi053 ceph-mon[89963]: pgmap v559: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 747 op/s; 7706/152910 objects degraded (5.040%); 0 B/s, 80 objects/s recovering 2024-08-17T17:27:05.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:05 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 7706/152910 objects degraded (5.040%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:06.002 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:06.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:06.058 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:06.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:06.159 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:06.193 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:06.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:06.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:06.295 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:06.335 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:06.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:06.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:06.437 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:06.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:06.480 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:06.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:06.581 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:06.584 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:27:06.607 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:06.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:06.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:06.709 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:06.735 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:06.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:06.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:06.837 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:06.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:06.882 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:06.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:06.983 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:07.017 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:07.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:07.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:07.119 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:07.147 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:07.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:07.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:07.249 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:07.281 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:07.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:07.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:07.382 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:07.408 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:07.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:07.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:07.510 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:07.536 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:07.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:07.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:07.638 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:07.669 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:07.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:07.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:07.771 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:07.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:07 smithi003 ceph-mon[100615]: pgmap v560: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 696 op/s; 7706/148482 objects degraded (5.190%); 0 B/s, 68 objects/s recovering 2024-08-17T17:27:07.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:07 smithi003 ceph-mon[103344]: pgmap v560: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 696 op/s; 7706/148482 objects degraded (5.190%); 0 B/s, 68 objects/s recovering 2024-08-17T17:27:07.796 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:07.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:07.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:07.898 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:07.929 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:07.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:07.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:07.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:07 smithi053 ceph-mon[89963]: pgmap v560: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 696 op/s; 7706/148482 objects degraded (5.190%); 0 B/s, 68 objects/s recovering 2024-08-17T17:27:08.031 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:08.065 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:08.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:08.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:08.166 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:08.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:08.209 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:08.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:08.310 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:08.336 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:08.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:08.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:08.438 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:08.460 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:08.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:08.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:08.561 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:08.604 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:08.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:08.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:08.706 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:08.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:08.737 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:08.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:08.838 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:08.860 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:08.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:08.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:08.962 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:08.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:08 smithi053 ceph-mon[89963]: from='client.35727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:27:08.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:27:08.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:27:08.991 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:08.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:08.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:09.022 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:08 smithi003 ceph-mon[100615]: from='client.35727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:27:09.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:27:09.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:27:09.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:08 smithi003 ceph-mon[103344]: from='client.35727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:27:09.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:27:09.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:27:09.094 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:09.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:09.136 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:09.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:09.237 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:09.282 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:09.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:09.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:09.384 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:09.424 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:09.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:09.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:09.526 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:09.571 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:09.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:09.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:09.673 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:09.702 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:09.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:09.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:09.803 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:09.826 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:09.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:09.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:09.928 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:09.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:09 smithi053 ceph-mon[89963]: pgmap v561: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 727 op/s; 7706/144867 objects degraded (5.319%); 0 B/s, 67 objects/s recovering 2024-08-17T17:27:09.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:09.976 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:09.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:10.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:09 smithi003 ceph-mon[100615]: pgmap v561: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 727 op/s; 7706/144867 objects degraded (5.319%); 0 B/s, 67 objects/s recovering 2024-08-17T17:27:10.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:09 smithi003 ceph-mon[103344]: pgmap v561: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 727 op/s; 7706/144867 objects degraded (5.319%); 0 B/s, 67 objects/s recovering 2024-08-17T17:27:10.077 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:10.108 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:10.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:10.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:10.210 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:10.238 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:10.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:10.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:10.340 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:10.374 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:10.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:10.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:10.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:27:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:27:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:27:10.476 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:10.515 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:10.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:10.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:10.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:10.639 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:27:10.641 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:27:10.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:10.685 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:10.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:10.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:10 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 7706/144867 objects degraded (5.319%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:10.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:10 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 7706/144867 objects degraded (5.319%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:10.786 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:10.814 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:10.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:10.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:10.916 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:10.946 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:10.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:10.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:10.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:10 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 7706/144867 objects degraded (5.319%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:11.048 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:11.081 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:11.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:11.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:11.184 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:11.198 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:27:11.210 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:11.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:11.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:11.312 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:11.342 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:11.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:11.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:11.444 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:11.470 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:11.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:11.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:11.572 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:11.613 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:11.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:11.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:11.714 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:11.737 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:11.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:11.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:11.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:11 smithi003 ceph-mon[100615]: pgmap v562: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 798 op/s; 7289/138888 objects degraded (5.248%); 0 B/s, 101 objects/s recovering 2024-08-17T17:27:11.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:11 smithi003 ceph-mon[103344]: pgmap v562: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 798 op/s; 7289/138888 objects degraded (5.248%); 0 B/s, 101 objects/s recovering 2024-08-17T17:27:11.839 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:11.861 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:11.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:11.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:11.963 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:11.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:11 smithi053 ceph-mon[89963]: pgmap v562: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 798 op/s; 7289/138888 objects degraded (5.248%); 0 B/s, 101 objects/s recovering 2024-08-17T17:27:11.996 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:11.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:11.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:12.098 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:12.121 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:12.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:12.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:12.224 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:12.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:12.247 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:12.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:12.348 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:12.370 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:12.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:12.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:12.472 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:12.494 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:12.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:12.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:12.518 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":560,"stamp":"2024-08-17T17:27:05.914533+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3246528485,"num_objects":49494,"num_object_clones":0,"num_object_copies":148482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7706,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49494,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":148950,"num_write_kb":6328384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13370,"num_bytes_recovered":776021382,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115709,"ondisk_log_size":115709,"up":267,"acting":263,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":278,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":12781488,"kb_used_data":10608184,"kb_used_omap":33,"kb_used_meta":2172958,"kb_avail":643434576,"statfs":{"total":671965249536,"available":658877005824,"internally_reserved":0,"allocated":10862780416,"data_stored":10854215269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34068,"internal_metadata":2225109740},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-539557888,"num_objects":-8233,"num_object_clones":0,"num_object_copies":-24699,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8233,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":805,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.815367"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":280,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.108115+0000","last_change":"2024-08-17T17:17:32.495652+0000","last_active":"2024-08-17T17:26:54.108115+0000","last_peered":"2024-08-17T17:26:54.108115+0000","last_clean":"2024-08-17T17:26:54.108115+0000","last_became_active":"2024-08-17T17:17:32.495218+0000","last_became_peered":"2024-08-17T17:17:32.495218+0000","last_unstale":"2024-08-17T17:26:54.108115+0000","last_undegraded":"2024-08-17T17:26:54.108115+0000","last_fullsized":"2024-08-17T17:26:54.108115+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:11:45.821282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":189,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.408643+0000","last_change":"2024-08-17T17:18:59.424887+0000","last_active":"2024-08-17T17:26:40.408643+0000","last_peered":"2024-08-17T17:26:40.408643+0000","last_clean":"2024-08-17T17:26:40.408643+0000","last_became_active":"2024-08-17T17:18:59.411539+0000","last_became_peered":"2024-08-17T17:18:59.411539+0000","last_unstale":"2024-08-17T17:26:40.408643+0000","last_undegraded":"2024-08-17T17:26:40.408643+0000","last_fullsized":"2024-08-17T17:26:40.408643+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:46:20.082241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":29,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.444371+0000","last_change":"2024-08-17T17:24:53.460742+0000","last_active":"2024-08-17T17:26:40.444371+0000","last_peered":"2024-08-17T17:26:40.444371+0000","last_clean":"2024-08-17T17:26:40.444371+0000","last_became_active":"2024-08-17T17:24:53.460365+0000","last_became_peered":"2024-08-17T17:24:53.460365+0000","last_unstale":"2024-08-17T17:26:40.444371+0000","last_undegraded":"2024-08-17T17:26:40.444371+0000","last_fullsized":"2024-08-17T17:26:40.444371+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:24:08.116968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":91,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.109014+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:26:54.109014+0000","last_peered":"2024-08-17T17:26:54.109014+0000","last_clean":"2024-08-17T17:26:54.109014+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:26:54.109014+0000","last_undegraded":"2024-08-17T17:26:54.109014+0000","last_fullsized":"2024-08-17T17:26:54.109014+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":91,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.108456+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:26:54.108456+0000","last_peered":"2024-08-17T17:26:54.108456+0000","last_clean":"2024-08-17T17:26:54.108456+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:26:54.108456+0000","last_undegraded":"2024-08-17T17:26:54.108456+0000","last_fullsized":"2024-08-17T17:26:54.108456+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":164,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.408342+0000","last_change":"2024-08-17T17:20:49.258206+0000","last_active":"2024-08-17T17:26:40.408342+0000","last_peered":"2024-08-17T17:26:40.408342+0000","last_clean":"2024-08-17T17:26:40.408342+0000","last_became_active":"2024-08-17T17:20:49.257761+0000","last_became_peered":"2024-08-17T17:20:49.257761+0000","last_unstale":"2024-08-17T17:26:40.408342+0000","last_undegraded":"2024-08-17T17:26:40.408342+0000","last_fullsized":"2024-08-17T17:26:40.408342+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:38:35.363748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":293,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.106199+0000","last_change":"2024-08-17T17:19:57.514773+0000","last_active":"2024-08-17T17:26:54.106199+0000","last_peered":"2024-08-17T17:26:54.106199+0000","last_clean":"2024-08-17T17:26:54.106199+0000","last_became_active":"2024-08-17T17:19:57.513839+0000","last_became_peered":"2024-08-17T17:19:57.513839+0000","last_unstale":"2024-08-17T17:26:54.106199+0000","last_undegraded":"2024-08-17T17:26:54.106199+0000","last_fullsized":"2024-08-17T17:26:54.106199+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:42:48.063169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":259,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.410121+0000","last_change":"2024-08-17T17:24:53.465875+0000","last_active":"2024-08-17T17:26:40.410121+0000","last_peered":"2024-08-17T17:26:40.410121+0000","last_clean":"2024-08-17T17:26:40.410121+0000","last_became_active":"2024-08-17T17:24:53.458129+0000","last_became_peered":"2024-08-17T17:24:53.458129+0000","last_unstale":"2024-08-17T17:26:40.410121+0000","last_undegraded":"2024-08-17T17:26:40.410121+0000","last_fullsized":"2024-08-17T17:26:40.410121+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:51:33.584869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"213'9138","reported_seq":12569,"reported_epoch":213,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-17T17:27:03.048400+0000","last_change":"2024-08-17T17:26:38.496637+0000","last_active":"2024-08-17T17:27:03.048400+0000","last_peered":"2024-08-17T17:27:03.048400+0000","last_clean":"2024-08-17T17:24:52.438492+0000","last_became_active":"2024-08-17T17:24:54.887269+0000","last_became_peered":"2024-08-17T17:24:54.887269+0000","last_unstale":"2024-08-17T17:27:03.048400+0000","last_undegraded":"2024-08-17T17:24:54.477707+0000","last_fullsized":"2024-08-17T17:24:54.461071+0000","mapping_epoch":194,"log_start":"189'1200","ondisk_log_start":"189'1200","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":7938,"log_dups_size":0,"ondisk_log_size":7938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203948032,"num_objects":3112,"num_object_clones":0,"num_object_copies":9336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":777,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9138,"num_write_kb":392000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1683,"num_bytes_recovered":33816576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[0,6],"avail_no_missing":["0","2","6"],"object_location_counts":[{"shards":"0,6","objects":2335}],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":125,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.409054+0000","last_change":"2024-08-17T17:24:06.597382+0000","last_active":"2024-08-17T17:26:40.409054+0000","last_peered":"2024-08-17T17:26:40.409054+0000","last_clean":"2024-08-17T17:26:40.409054+0000","last_became_active":"2024-08-17T17:24:06.588881+0000","last_became_peered":"2024-08-17T17:24:06.588881+0000","last_unstale":"2024-08-17T17:26:40.409054+0000","last_undegraded":"2024-08-17T17:26:40.409054+0000","last_fullsized":"2024-08-17T17:26:40.409054+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:27.240751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":39,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.443960+0000","last_change":"2024-08-17T17:25:04.503780+0000","last_active":"2024-08-17T17:26:40.443960+0000","last_peered":"2024-08-17T17:26:40.443960+0000","last_clean":"2024-08-17T17:26:40.443960+0000","last_became_active":"2024-08-17T17:25:04.503503+0000","last_became_peered":"2024-08-17T17:25:04.503503+0000","last_unstale":"2024-08-17T17:26:40.443960+0000","last_undegraded":"2024-08-17T17:26:40.443960+0000","last_fullsized":"2024-08-17T17:26:40.443960+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:04:48.171216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"213'9174","reported_seq":9256,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:27:01.754056+0000","last_change":"2024-08-17T17:24:14.639361+0000","last_active":"2024-08-17T17:27:01.754056+0000","last_peered":"2024-08-17T17:27:01.754056+0000","last_clean":"2024-08-17T17:27:01.754056+0000","last_became_active":"2024-08-17T17:24:14.639074+0000","last_became_peered":"2024-08-17T17:24:14.639074+0000","last_unstale":"2024-08-17T17:27:01.754056+0000","last_undegraded":"2024-08-17T17:27:01.754056+0000","last_fullsized":"2024-08-17T17:27:01.754056+0000","mapping_epoch":178,"log_start":"195'2700","ondisk_log_start":"195'2700","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6474,"log_dups_size":0,"ondisk_log_size":6474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:15:05.377992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209059840,"num_objects":3190,"num_object_clones":0,"num_object_copies":9570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9174,"num_write_kb":395648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":111,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.408959+0000","last_change":"2024-08-17T17:24:06.599023+0000","last_active":"2024-08-17T17:26:40.408959+0000","last_peered":"2024-08-17T17:26:40.408959+0000","last_clean":"2024-08-17T17:26:40.408959+0000","last_became_active":"2024-08-17T17:24:06.588954+0000","last_became_peered":"2024-08-17T17:24:06.588954+0000","last_unstale":"2024-08-17T17:26:40.408959+0000","last_undegraded":"2024-08-17T17:26:40.408959+0000","last_fullsized":"2024-08-17T17:26:40.408959+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:20:51.445576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":234,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.114033+0000","last_change":"2024-08-17T17:20:49.271540+0000","last_active":"2024-08-17T17:26:54.114033+0000","last_peered":"2024-08-17T17:26:54.114033+0000","last_clean":"2024-08-17T17:26:54.114033+0000","last_became_active":"2024-08-17T17:20:49.271130+0000","last_became_peered":"2024-08-17T17:20:49.271130+0000","last_unstale":"2024-08-17T17:26:54.114033+0000","last_undegraded":"2024-08-17T17:26:54.114033+0000","last_fullsized":"2024-08-17T17:26:54.114033+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:05.386705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"213'9180","reported_seq":9223,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:27:03.960731+0000","last_change":"2024-08-17T17:24:14.644070+0000","last_active":"2024-08-17T17:27:03.960731+0000","last_peered":"2024-08-17T17:27:03.960731+0000","last_clean":"2024-08-17T17:27:03.960731+0000","last_became_active":"2024-08-17T17:24:14.643799+0000","last_became_peered":"2024-08-17T17:24:14.643799+0000","last_unstale":"2024-08-17T17:27:03.960731+0000","last_undegraded":"2024-08-17T17:27:03.960731+0000","last_fullsized":"2024-08-17T17:27:03.960731+0000","mapping_epoch":178,"log_start":"192'2200","ondisk_log_start":"192'2200","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6980,"log_dups_size":0,"ondisk_log_size":6980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:25:38.445770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201981952,"num_objects":3082,"num_object_clones":0,"num_object_copies":9246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9180,"num_write_kb":392384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":183,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.409008+0000","last_change":"2024-08-17T17:24:53.467526+0000","last_active":"2024-08-17T17:26:40.409008+0000","last_peered":"2024-08-17T17:26:40.409008+0000","last_clean":"2024-08-17T17:26:40.409008+0000","last_became_active":"2024-08-17T17:24:53.460631+0000","last_became_peered":"2024-08-17T17:24:53.460631+0000","last_unstale":"2024-08-17T17:26:40.409008+0000","last_undegraded":"2024-08-17T17:26:40.409008+0000","last_fullsized":"2024-08-17T17:26:40.409008+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:05:24.938831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.410313+0000","last_change":"2024-08-17T17:25:04.507176+0000","last_active":"2024-08-17T17:26:40.410313+0000","last_peered":"2024-08-17T17:26:40.410313+0000","last_clean":"2024-08-17T17:26:40.410313+0000","last_became_active":"2024-08-17T17:25:04.506788+0000","last_became_peered":"2024-08-17T17:25:04.506788+0000","last_unstale":"2024-08-17T17:26:40.410313+0000","last_undegraded":"2024-08-17T17:26:40.410313+0000","last_fullsized":"2024-08-17T17:26:40.410313+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:22:04.658216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"213'9029","reported_seq":9111,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:27:02.981092+0000","last_change":"2024-08-17T17:24:14.642637+0000","last_active":"2024-08-17T17:27:02.981092+0000","last_peered":"2024-08-17T17:27:02.981092+0000","last_clean":"2024-08-17T17:27:02.981092+0000","last_became_active":"2024-08-17T17:24:14.642347+0000","last_became_peered":"2024-08-17T17:24:14.642347+0000","last_unstale":"2024-08-17T17:27:02.981092+0000","last_undegraded":"2024-08-17T17:27:02.981092+0000","last_fullsized":"2024-08-17T17:27:02.981092+0000","mapping_epoch":178,"log_start":"189'1100","ondisk_log_start":"189'1100","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":7929,"log_dups_size":0,"ondisk_log_size":7929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:37:44.631335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196673536,"num_objects":3001,"num_object_clones":0,"num_object_copies":9003,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9029,"num_write_kb":384960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":171,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.444537+0000","last_change":"2024-08-17T17:17:32.495428+0000","last_active":"2024-08-17T17:26:40.444537+0000","last_peered":"2024-08-17T17:26:40.444537+0000","last_clean":"2024-08-17T17:26:40.444537+0000","last_became_active":"2024-08-17T17:17:32.494978+0000","last_became_peered":"2024-08-17T17:17:32.494978+0000","last_unstale":"2024-08-17T17:26:40.444537+0000","last_undegraded":"2024-08-17T17:26:40.444537+0000","last_fullsized":"2024-08-17T17:26:40.444537+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:42:24.647812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":282,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.108151+0000","last_change":"2024-08-17T17:20:49.272126+0000","last_active":"2024-08-17T17:26:54.108151+0000","last_peered":"2024-08-17T17:26:54.108151+0000","last_clean":"2024-08-17T17:26:54.108151+0000","last_became_active":"2024-08-17T17:20:49.271185+0000","last_became_peered":"2024-08-17T17:20:49.271185+0000","last_unstale":"2024-08-17T17:26:54.108151+0000","last_undegraded":"2024-08-17T17:26:54.108151+0000","last_fullsized":"2024-08-17T17:26:54.108151+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:43:58.184939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"213'9674","reported_seq":10631,"reported_epoch":213,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:27:05.370910+0000","last_change":"2024-08-17T17:24:54.898729+0000","last_active":"2024-08-17T17:27:05.370910+0000","last_peered":"2024-08-17T17:27:05.370910+0000","last_clean":"2024-08-17T17:24:52.435411+0000","last_became_active":"2024-08-17T17:24:54.889545+0000","last_became_peered":"2024-08-17T17:24:54.889545+0000","last_unstale":"2024-08-17T17:27:05.370910+0000","last_undegraded":"2024-08-17T17:24:54.482592+0000","last_fullsized":"2024-08-17T17:24:54.459529+0000","mapping_epoch":194,"log_start":"199'3400","ondisk_log_start":"199'3400","created":178,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6274,"log_dups_size":0,"ondisk_log_size":6274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201850880,"num_objects":3080,"num_object_clones":0,"num_object_copies":9240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2587,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9674,"num_write_kb":408128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":430,"num_bytes_recovered":28180480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2],"avail_no_missing":["1","2","3","7"],"object_location_counts":[{"shards":"1,2","objects":493}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":311,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.105717+0000","last_change":"2024-08-17T17:24:06.595380+0000","last_active":"2024-08-17T17:26:54.105717+0000","last_peered":"2024-08-17T17:26:54.105717+0000","last_clean":"2024-08-17T17:26:54.105717+0000","last_became_active":"2024-08-17T17:24:06.584747+0000","last_became_peered":"2024-08-17T17:24:06.584747+0000","last_unstale":"2024-08-17T17:26:54.105717+0000","last_undegraded":"2024-08-17T17:26:54.105717+0000","last_fullsized":"2024-08-17T17:26:54.105717+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:02:18.473980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":256,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.108356+0000","last_change":"2024-08-17T17:20:49.280596+0000","last_active":"2024-08-17T17:26:54.108356+0000","last_peered":"2024-08-17T17:26:54.108356+0000","last_clean":"2024-08-17T17:26:54.108356+0000","last_became_active":"2024-08-17T17:20:49.279553+0000","last_became_peered":"2024-08-17T17:20:49.279553+0000","last_unstale":"2024-08-17T17:26:54.108356+0000","last_undegraded":"2024-08-17T17:26:54.108356+0000","last_fullsized":"2024-08-17T17:26:54.108356+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:13:24.124392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"213'9056","reported_seq":9138,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:27:01.732045+0000","last_change":"2024-08-17T17:24:14.638890+0000","last_active":"2024-08-17T17:27:01.732045+0000","last_peered":"2024-08-17T17:27:01.732045+0000","last_clean":"2024-08-17T17:27:01.732045+0000","last_became_active":"2024-08-17T17:24:14.638594+0000","last_became_peered":"2024-08-17T17:24:14.638594+0000","last_unstale":"2024-08-17T17:27:01.732045+0000","last_undegraded":"2024-08-17T17:27:01.732045+0000","last_fullsized":"2024-08-17T17:27:01.732045+0000","mapping_epoch":178,"log_start":"195'2600","ondisk_log_start":"195'2600","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6456,"log_dups_size":0,"ondisk_log_size":6456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:53:34.277077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206438400,"num_objects":3150,"num_object_clones":0,"num_object_copies":9450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9056,"num_write_kb":390592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":208,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.109865+0000","last_change":"2024-08-17T17:20:49.276769+0000","last_active":"2024-08-17T17:26:54.109865+0000","last_peered":"2024-08-17T17:26:54.109865+0000","last_clean":"2024-08-17T17:26:54.109865+0000","last_became_active":"2024-08-17T17:20:49.276498+0000","last_became_peered":"2024-08-17T17:20:49.276498+0000","last_unstale":"2024-08-17T17:26:54.109865+0000","last_undegraded":"2024-08-17T17:26:54.109865+0000","last_fullsized":"2024-08-17T17:26:54.109865+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:39:05.419440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":192,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.408818+0000","last_change":"2024-08-17T17:20:14.455738+0000","last_active":"2024-08-17T17:26:40.408818+0000","last_peered":"2024-08-17T17:26:40.408818+0000","last_clean":"2024-08-17T17:26:40.408818+0000","last_became_active":"2024-08-17T17:20:14.455469+0000","last_became_peered":"2024-08-17T17:20:14.455469+0000","last_unstale":"2024-08-17T17:26:40.408818+0000","last_undegraded":"2024-08-17T17:26:40.408818+0000","last_fullsized":"2024-08-17T17:26:40.408818+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:27:28.424336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"213'9228","reported_seq":16570,"reported_epoch":213,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:27:01.752144+0000","last_change":"2024-08-17T17:25:05.678234+0000","last_active":"2024-08-17T17:27:01.752144+0000","last_peered":"2024-08-17T17:27:01.752144+0000","last_clean":"2024-08-17T17:24:52.436463+0000","last_became_active":"2024-08-17T17:25:05.673238+0000","last_became_peered":"2024-08-17T17:25:05.673238+0000","last_unstale":"2024-08-17T17:27:01.752144+0000","last_undegraded":"2024-08-17T17:25:05.585769+0000","last_fullsized":"2024-08-17T17:25:05.499112+0000","mapping_epoch":198,"log_start":"195'2800","ondisk_log_start":"195'2800","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6428,"log_dups_size":0,"ondisk_log_size":6428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205783040,"num_objects":3140,"num_object_clones":0,"num_object_copies":9420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9228,"num_write_kb":395776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2519,"num_bytes_recovered":165085184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,5],"avail_no_missing":["3","1","5","7"],"object_location_counts":[{"shards":"3,5","objects":2114},{"shards":"3,5,7","objects":197}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":146,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.408273+0000","last_change":"2024-08-17T17:20:14.431140+0000","last_active":"2024-08-17T17:26:40.408273+0000","last_peered":"2024-08-17T17:26:40.408273+0000","last_clean":"2024-08-17T17:26:40.408273+0000","last_became_active":"2024-08-17T17:20:14.430839+0000","last_became_peered":"2024-08-17T17:20:14.430839+0000","last_unstale":"2024-08-17T17:26:40.408273+0000","last_undegraded":"2024-08-17T17:26:40.408273+0000","last_fullsized":"2024-08-17T17:26:40.408273+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:08:16.303616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.410813+0000","last_change":"2024-08-17T17:25:04.509325+0000","last_active":"2024-08-17T17:26:40.410813+0000","last_peered":"2024-08-17T17:26:40.410813+0000","last_clean":"2024-08-17T17:26:40.410813+0000","last_became_active":"2024-08-17T17:25:04.508364+0000","last_became_peered":"2024-08-17T17:25:04.508364+0000","last_unstale":"2024-08-17T17:26:40.410813+0000","last_undegraded":"2024-08-17T17:26:40.410813+0000","last_fullsized":"2024-08-17T17:26:40.410813+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:48:56.415763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"213'9040","reported_seq":13872,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:27:03.785030+0000","last_change":"2024-08-17T17:25:40.811756+0000","last_active":"2024-08-17T17:27:03.785030+0000","last_peered":"2024-08-17T17:27:03.785030+0000","last_clean":"2024-08-17T17:27:03.785030+0000","last_became_active":"2024-08-17T17:25:40.771755+0000","last_became_peered":"2024-08-17T17:25:40.771755+0000","last_unstale":"2024-08-17T17:27:03.785030+0000","last_undegraded":"2024-08-17T17:27:03.785030+0000","last_fullsized":"2024-08-17T17:27:03.785030+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":9040,"log_dups_size":0,"ondisk_log_size":9040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:13:03.918487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201654296,"num_objects":3078,"num_object_clones":0,"num_object_copies":9234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3078,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9040,"num_write_kb":387713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2371,"num_bytes_recovered":155385856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":295,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.106540+0000","last_change":"2024-08-17T17:19:57.520745+0000","last_active":"2024-08-17T17:26:54.106540+0000","last_peered":"2024-08-17T17:26:54.106540+0000","last_clean":"2024-08-17T17:26:54.106540+0000","last_became_active":"2024-08-17T17:19:57.520152+0000","last_became_peered":"2024-08-17T17:19:57.520152+0000","last_unstale":"2024-08-17T17:26:54.106540+0000","last_undegraded":"2024-08-17T17:26:54.106540+0000","last_fullsized":"2024-08-17T17:26:54.106540+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:19:39.495644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":270,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.109767+0000","last_change":"2024-08-17T17:20:49.274372+0000","last_active":"2024-08-17T17:26:54.109767+0000","last_peered":"2024-08-17T17:26:54.109767+0000","last_clean":"2024-08-17T17:26:54.109767+0000","last_became_active":"2024-08-17T17:20:49.274023+0000","last_became_peered":"2024-08-17T17:20:49.274023+0000","last_unstale":"2024-08-17T17:26:54.109767+0000","last_undegraded":"2024-08-17T17:26:54.109767+0000","last_fullsized":"2024-08-17T17:26:54.109767+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:31:02.529766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"213'9652","reported_seq":9692,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:27:03.839366+0000","last_change":"2024-08-17T17:24:14.643398+0000","last_active":"2024-08-17T17:27:03.839366+0000","last_peered":"2024-08-17T17:27:03.839366+0000","last_clean":"2024-08-17T17:27:03.839366+0000","last_became_active":"2024-08-17T17:24:14.643102+0000","last_became_peered":"2024-08-17T17:24:14.643102+0000","last_unstale":"2024-08-17T17:27:03.839366+0000","last_undegraded":"2024-08-17T17:27:03.839366+0000","last_fullsized":"2024-08-17T17:27:03.839366+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":9652,"log_dups_size":0,"ondisk_log_size":9652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:09:30.323709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200278016,"num_objects":3056,"num_object_clones":0,"num_object_copies":9168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9652,"num_write_kb":406656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":540,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.408309+0000","last_change":"2024-08-17T17:24:09.599615+0000","last_active":"2024-08-17T17:26:40.408309+0000","last_peered":"2024-08-17T17:26:40.408309+0000","last_clean":"2024-08-17T17:26:40.408309+0000","last_became_active":"2024-08-17T17:24:09.599085+0000","last_became_peered":"2024-08-17T17:24:09.599085+0000","last_unstale":"2024-08-17T17:26:40.408309+0000","last_undegraded":"2024-08-17T17:26:40.408309+0000","last_fullsized":"2024-08-17T17:26:40.408309+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:04:01.141401+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:30:09.569579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":785,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.444165+0000","last_change":"2024-08-17T17:20:00.434774+0000","last_active":"2024-08-17T17:26:40.444165+0000","last_peered":"2024-08-17T17:26:40.444165+0000","last_clean":"2024-08-17T17:26:40.444165+0000","last_became_active":"2024-08-17T17:20:00.434343+0000","last_became_peered":"2024-08-17T17:20:00.434343+0000","last_unstale":"2024-08-17T17:26:40.444165+0000","last_undegraded":"2024-08-17T17:26:40.444165+0000","last_fullsized":"2024-08-17T17:26:40.444165+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:04.995152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":253,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.105197+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:26:54.105197+0000","last_peered":"2024-08-17T17:26:54.105197+0000","last_clean":"2024-08-17T17:26:54.105197+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:26:54.105197+0000","last_undegraded":"2024-08-17T17:26:54.105197+0000","last_fullsized":"2024-08-17T17:26:54.105197+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":25,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.410728+0000","last_change":"2024-08-17T17:25:04.579426+0000","last_active":"2024-08-17T17:26:40.410728+0000","last_peered":"2024-08-17T17:26:40.410728+0000","last_clean":"2024-08-17T17:26:40.410728+0000","last_became_active":"2024-08-17T17:25:04.579012+0000","last_became_peered":"2024-08-17T17:25:04.579012+0000","last_unstale":"2024-08-17T17:26:40.410728+0000","last_undegraded":"2024-08-17T17:26:40.410728+0000","last_fullsized":"2024-08-17T17:26:40.410728+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:56:32.241321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"213'9210","reported_seq":9291,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:27:05.382938+0000","last_change":"2024-08-17T17:24:14.641829+0000","last_active":"2024-08-17T17:27:05.382938+0000","last_peered":"2024-08-17T17:27:05.382938+0000","last_clean":"2024-08-17T17:27:05.382938+0000","last_became_active":"2024-08-17T17:24:14.641532+0000","last_became_peered":"2024-08-17T17:24:14.641532+0000","last_unstale":"2024-08-17T17:27:05.382938+0000","last_undegraded":"2024-08-17T17:27:05.382938+0000","last_fullsized":"2024-08-17T17:27:05.382938+0000","mapping_epoch":178,"log_start":"195'2900","ondisk_log_start":"195'2900","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6310,"log_dups_size":0,"ondisk_log_size":6310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:43:05.232036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198443008,"num_objects":3028,"num_object_clones":0,"num_object_copies":9084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9210,"num_write_kb":391616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":803,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.408409+0000","last_change":"2024-08-17T17:16:26.958415+0000","last_active":"2024-08-17T17:26:40.408409+0000","last_peered":"2024-08-17T17:26:40.408409+0000","last_clean":"2024-08-17T17:26:40.408409+0000","last_became_active":"2024-08-17T17:16:26.957956+0000","last_became_peered":"2024-08-17T17:16:26.957956+0000","last_unstale":"2024-08-17T17:26:40.408409+0000","last_undegraded":"2024-08-17T17:26:40.408409+0000","last_fullsized":"2024-08-17T17:26:40.408409+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:22.478077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":282,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.107750+0000","last_change":"2024-08-17T17:18:59.415331+0000","last_active":"2024-08-17T17:26:54.107750+0000","last_peered":"2024-08-17T17:26:54.107750+0000","last_clean":"2024-08-17T17:26:54.107750+0000","last_became_active":"2024-08-17T17:18:59.415008+0000","last_became_peered":"2024-08-17T17:18:59.415008+0000","last_unstale":"2024-08-17T17:26:54.107750+0000","last_undegraded":"2024-08-17T17:26:54.107750+0000","last_fullsized":"2024-08-17T17:26:54.107750+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:22:38.076866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":91,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.108653+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:26:54.108653+0000","last_peered":"2024-08-17T17:26:54.108653+0000","last_clean":"2024-08-17T17:26:54.108653+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:26:54.108653+0000","last_undegraded":"2024-08-17T17:26:54.108653+0000","last_fullsized":"2024-08-17T17:26:54.108653+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"213'9110","reported_seq":15389,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:27:03.949610+0000","last_change":"2024-08-17T17:26:40.426009+0000","last_active":"2024-08-17T17:27:03.949610+0000","last_peered":"2024-08-17T17:27:03.949610+0000","last_clean":"2024-08-17T17:27:03.949610+0000","last_became_active":"2024-08-17T17:26:40.411144+0000","last_became_peered":"2024-08-17T17:26:40.411144+0000","last_unstale":"2024-08-17T17:27:03.949610+0000","last_undegraded":"2024-08-17T17:27:03.949610+0000","last_fullsized":"2024-08-17T17:27:03.949610+0000","mapping_epoch":211,"log_start":"192'2100","ondisk_log_start":"192'2100","created":178,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":7010,"log_dups_size":0,"ondisk_log_size":7010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:22.245859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201326592,"num_objects":3072,"num_object_clones":0,"num_object_copies":9216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9110,"num_write_kb":389824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3108,"num_bytes_recovered":178388992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":670,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.105238+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:26:54.105238+0000","last_peered":"2024-08-17T17:26:54.105238+0000","last_clean":"2024-08-17T17:26:54.105238+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:26:54.105238+0000","last_undegraded":"2024-08-17T17:26:54.105238+0000","last_fullsized":"2024-08-17T17:26:54.105238+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":278,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.109559+0000","last_change":"2024-08-17T17:19:57.515564+0000","last_active":"2024-08-17T17:26:54.109559+0000","last_peered":"2024-08-17T17:26:54.109559+0000","last_clean":"2024-08-17T17:26:54.109559+0000","last_became_active":"2024-08-17T17:19:57.515142+0000","last_became_peered":"2024-08-17T17:19:57.515142+0000","last_unstale":"2024-08-17T17:26:54.109559+0000","last_undegraded":"2024-08-17T17:26:54.109559+0000","last_fullsized":"2024-08-17T17:26:54.109559+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:59:33.506176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":308,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.109331+0000","last_change":"2024-08-17T17:24:06.587903+0000","last_active":"2024-08-17T17:26:54.109331+0000","last_peered":"2024-08-17T17:26:54.109331+0000","last_clean":"2024-08-17T17:26:54.109331+0000","last_became_active":"2024-08-17T17:24:06.587503+0000","last_became_peered":"2024-08-17T17:24:06.587503+0000","last_unstale":"2024-08-17T17:26:54.109331+0000","last_undegraded":"2024-08-17T17:26:54.109331+0000","last_fullsized":"2024-08-17T17:26:54.109331+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:05:58.591992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"213'8990","reported_seq":15289,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:27:01.771022+0000","last_change":"2024-08-17T17:26:08.233981+0000","last_active":"2024-08-17T17:27:01.771022+0000","last_peered":"2024-08-17T17:27:01.771022+0000","last_clean":"2024-08-17T17:27:01.771022+0000","last_became_active":"2024-08-17T17:26:08.225711+0000","last_became_peered":"2024-08-17T17:26:08.225711+0000","last_unstale":"2024-08-17T17:27:01.771022+0000","last_undegraded":"2024-08-17T17:27:01.771022+0000","last_fullsized":"2024-08-17T17:27:01.771022+0000","mapping_epoch":207,"log_start":"195'2600","ondisk_log_start":"195'2600","created":178,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6390,"log_dups_size":0,"ondisk_log_size":6390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:31:42.806869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207880192,"num_objects":3172,"num_object_clones":0,"num_object_copies":9516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8990,"num_write_kb":389184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3101,"num_bytes_recovered":203227136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":1123,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.444102+0000","last_change":"2024-08-17T17:20:00.433870+0000","last_active":"2024-08-17T17:26:40.444102+0000","last_peered":"2024-08-17T17:26:40.444102+0000","last_clean":"2024-08-17T17:26:40.444102+0000","last_became_active":"2024-08-17T17:20:00.433386+0000","last_became_peered":"2024-08-17T17:20:00.433386+0000","last_unstale":"2024-08-17T17:26:40.444102+0000","last_undegraded":"2024-08-17T17:26:40.444102+0000","last_fullsized":"2024-08-17T17:26:40.444102+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:01:11.848862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1392,"num_read_kb":2863,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":203,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.108014+0000","last_change":"2024-08-17T17:20:49.272407+0000","last_active":"2024-08-17T17:26:54.108014+0000","last_peered":"2024-08-17T17:26:54.108014+0000","last_clean":"2024-08-17T17:26:54.108014+0000","last_became_active":"2024-08-17T17:20:49.271749+0000","last_became_peered":"2024-08-17T17:20:49.271749+0000","last_unstale":"2024-08-17T17:26:54.108014+0000","last_undegraded":"2024-08-17T17:26:54.108014+0000","last_fullsized":"2024-08-17T17:26:54.108014+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:36:01.443457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":275,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.110085+0000","last_change":"2024-08-17T17:17:32.499715+0000","last_active":"2024-08-17T17:26:54.110085+0000","last_peered":"2024-08-17T17:26:54.110085+0000","last_clean":"2024-08-17T17:26:54.110085+0000","last_became_active":"2024-08-17T17:17:32.499412+0000","last_became_peered":"2024-08-17T17:17:32.499412+0000","last_unstale":"2024-08-17T17:26:54.110085+0000","last_undegraded":"2024-08-17T17:26:54.110085+0000","last_fullsized":"2024-08-17T17:26:54.110085+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:39:38.456970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"213'9234","reported_seq":9316,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:27:03.014682+0000","last_change":"2024-08-17T17:24:14.642608+0000","last_active":"2024-08-17T17:27:03.014682+0000","last_peered":"2024-08-17T17:27:03.014682+0000","last_clean":"2024-08-17T17:27:03.014682+0000","last_became_active":"2024-08-17T17:24:14.642292+0000","last_became_peered":"2024-08-17T17:24:14.642292+0000","last_unstale":"2024-08-17T17:27:03.014682+0000","last_undegraded":"2024-08-17T17:27:03.014682+0000","last_fullsized":"2024-08-17T17:27:03.014682+0000","mapping_epoch":178,"log_start":"190'1300","ondisk_log_start":"190'1300","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":7934,"log_dups_size":0,"ondisk_log_size":7934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:53:36.848614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202506240,"num_objects":3090,"num_object_clones":0,"num_object_copies":9270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9234,"num_write_kb":394368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":188,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.444624+0000","last_change":"2024-08-17T17:17:32.495728+0000","last_active":"2024-08-17T17:26:40.444624+0000","last_peered":"2024-08-17T17:26:40.444624+0000","last_clean":"2024-08-17T17:26:40.444624+0000","last_became_active":"2024-08-17T17:17:32.495357+0000","last_became_peered":"2024-08-17T17:17:32.495357+0000","last_unstale":"2024-08-17T17:26:40.444624+0000","last_undegraded":"2024-08-17T17:26:40.444624+0000","last_fullsized":"2024-08-17T17:26:40.444624+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:27:45.561567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":511,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.110252+0000","last_change":"2024-08-17T17:16:26.964621+0000","last_active":"2024-08-17T17:26:54.110252+0000","last_peered":"2024-08-17T17:26:54.110252+0000","last_clean":"2024-08-17T17:26:54.110252+0000","last_became_active":"2024-08-17T17:16:26.964058+0000","last_became_peered":"2024-08-17T17:16:26.964058+0000","last_unstale":"2024-08-17T17:26:54.110252+0000","last_undegraded":"2024-08-17T17:26:54.110252+0000","last_fullsized":"2024-08-17T17:26:54.110252+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:00:04.465644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":281,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.109952+0000","last_change":"2024-08-17T17:24:53.465550+0000","last_active":"2024-08-17T17:26:54.109952+0000","last_peered":"2024-08-17T17:26:54.109952+0000","last_clean":"2024-08-17T17:26:54.109952+0000","last_became_active":"2024-08-17T17:24:53.465228+0000","last_became_peered":"2024-08-17T17:24:53.465228+0000","last_unstale":"2024-08-17T17:26:54.109952+0000","last_undegraded":"2024-08-17T17:26:54.109952+0000","last_fullsized":"2024-08-17T17:26:54.109952+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:21:18.378359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"213'9417","reported_seq":9499,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:27:05.346798+0000","last_change":"2024-08-17T17:24:14.640584+0000","last_active":"2024-08-17T17:27:05.346798+0000","last_peered":"2024-08-17T17:27:05.346798+0000","last_clean":"2024-08-17T17:27:05.346798+0000","last_became_active":"2024-08-17T17:24:14.640168+0000","last_became_peered":"2024-08-17T17:24:14.640168+0000","last_unstale":"2024-08-17T17:27:05.346798+0000","last_undegraded":"2024-08-17T17:27:05.346798+0000","last_fullsized":"2024-08-17T17:27:05.346798+0000","mapping_epoch":178,"log_start":"195'3100","ondisk_log_start":"195'3100","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6317,"log_dups_size":0,"ondisk_log_size":6317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:38:36.176813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195756032,"num_objects":2987,"num_object_clones":0,"num_object_copies":8961,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9417,"num_write_kb":396928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":306,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.110163+0000","last_change":"2024-08-17T17:24:53.464307+0000","last_active":"2024-08-17T17:26:54.110163+0000","last_peered":"2024-08-17T17:26:54.110163+0000","last_clean":"2024-08-17T17:26:54.110163+0000","last_became_active":"2024-08-17T17:24:53.463996+0000","last_became_peered":"2024-08-17T17:24:53.463996+0000","last_unstale":"2024-08-17T17:26:54.110163+0000","last_undegraded":"2024-08-17T17:26:54.110163+0000","last_fullsized":"2024-08-17T17:26:54.110163+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:04:18.993725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":640,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.409001+0000","last_change":"2024-08-17T17:24:09.599822+0000","last_active":"2024-08-17T17:26:40.409001+0000","last_peered":"2024-08-17T17:26:40.409001+0000","last_clean":"2024-08-17T17:26:40.409001+0000","last_became_active":"2024-08-17T17:24:09.599189+0000","last_became_peered":"2024-08-17T17:24:09.599189+0000","last_unstale":"2024-08-17T17:26:40.409001+0000","last_undegraded":"2024-08-17T17:26:40.409001+0000","last_fullsized":"2024-08-17T17:26:40.409001+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:04.465604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":265,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.108087+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:26:54.108087+0000","last_peered":"2024-08-17T17:26:54.108087+0000","last_clean":"2024-08-17T17:26:54.108087+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:26:54.108087+0000","last_undegraded":"2024-08-17T17:26:54.108087+0000","last_fullsized":"2024-08-17T17:26:54.108087+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"213'9405","reported_seq":9487,"reported_epoch":213,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:27:01.746102+0000","last_change":"2024-08-17T17:25:05.605653+0000","last_active":"2024-08-17T17:27:01.746102+0000","last_peered":"2024-08-17T17:27:01.746102+0000","last_clean":"2024-08-17T17:25:03.971607+0000","last_became_active":"2024-08-17T17:25:05.601333+0000","last_became_peered":"2024-08-17T17:25:05.601333+0000","last_unstale":"2024-08-17T17:27:01.746102+0000","last_undegraded":"2024-08-17T17:25:05.522007+0000","last_fullsized":"2024-08-17T17:25:05.499355+0000","mapping_epoch":198,"log_start":"195'3000","ondisk_log_start":"195'3000","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6405,"log_dups_size":0,"ondisk_log_size":6405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205979648,"num_objects":3143,"num_object_clones":0,"num_object_copies":9429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9405,"num_write_kb":401536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2],"avail_no_missing":["3","2","7"],"object_location_counts":[{"shards":"2,3","objects":-173},{"shards":"2,3,7","objects":185}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":300,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.106103+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:26:54.106103+0000","last_peered":"2024-08-17T17:26:54.106103+0000","last_clean":"2024-08-17T17:26:54.106103+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:26:54.106103+0000","last_undegraded":"2024-08-17T17:26:54.106103+0000","last_fullsized":"2024-08-17T17:26:54.106103+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":819,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.113074+0000","last_change":"2024-08-17T17:23:19.633786+0000","last_active":"2024-08-17T17:26:54.113074+0000","last_peered":"2024-08-17T17:26:54.113074+0000","last_clean":"2024-08-17T17:26:54.113074+0000","last_became_active":"2024-08-17T17:23:19.633230+0000","last_became_peered":"2024-08-17T17:23:19.633230+0000","last_unstale":"2024-08-17T17:26:54.113074+0000","last_undegraded":"2024-08-17T17:26:54.113074+0000","last_fullsized":"2024-08-17T17:26:54.113074+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:30:50.377391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":296,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.106298+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:26:54.106298+0000","last_peered":"2024-08-17T17:26:54.106298+0000","last_clean":"2024-08-17T17:26:54.106298+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:26:54.106298+0000","last_undegraded":"2024-08-17T17:26:54.106298+0000","last_fullsized":"2024-08-17T17:26:54.106298+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"213'9291","reported_seq":9334,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:27:04.006930+0000","last_change":"2024-08-17T17:24:14.639560+0000","last_active":"2024-08-17T17:27:04.006930+0000","last_peered":"2024-08-17T17:27:04.006930+0000","last_clean":"2024-08-17T17:27:04.006930+0000","last_became_active":"2024-08-17T17:24:14.639093+0000","last_became_peered":"2024-08-17T17:24:14.639093+0000","last_unstale":"2024-08-17T17:27:04.006930+0000","last_undegraded":"2024-08-17T17:27:04.006930+0000","last_fullsized":"2024-08-17T17:27:04.006930+0000","mapping_epoch":178,"log_start":"192'2300","ondisk_log_start":"192'2300","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6991,"log_dups_size":0,"ondisk_log_size":6991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:45:57.940550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202178560,"num_objects":3085,"num_object_clones":0,"num_object_copies":9255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9291,"num_write_kb":396032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":95,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.113830+0000","last_change":"2024-08-17T17:24:06.593660+0000","last_active":"2024-08-17T17:26:54.113830+0000","last_peered":"2024-08-17T17:26:54.113830+0000","last_clean":"2024-08-17T17:26:54.113830+0000","last_became_active":"2024-08-17T17:24:06.586215+0000","last_became_peered":"2024-08-17T17:24:06.586215+0000","last_unstale":"2024-08-17T17:26:54.113830+0000","last_undegraded":"2024-08-17T17:26:54.113830+0000","last_fullsized":"2024-08-17T17:26:54.113830+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:15:49.475395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":444,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.410890+0000","last_change":"2024-08-17T17:26:06.855307+0000","last_active":"2024-08-17T17:26:40.410890+0000","last_peered":"2024-08-17T17:26:40.410890+0000","last_clean":"2024-08-17T17:26:40.410890+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:26:40.410890+0000","last_undegraded":"2024-08-17T17:26:40.410890+0000","last_fullsized":"2024-08-17T17:26:40.410890+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":112,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.443974+0000","last_change":"2024-08-17T17:18:59.417183+0000","last_active":"2024-08-17T17:26:40.443974+0000","last_peered":"2024-08-17T17:26:40.443974+0000","last_clean":"2024-08-17T17:26:40.443974+0000","last_became_active":"2024-08-17T17:18:59.416843+0000","last_became_peered":"2024-08-17T17:18:59.416843+0000","last_unstale":"2024-08-17T17:26:40.443974+0000","last_undegraded":"2024-08-17T17:26:40.443974+0000","last_fullsized":"2024-08-17T17:26:40.443974+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:29:09.336335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":133,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.410355+0000","last_change":"2024-08-17T17:20:49.258048+0000","last_active":"2024-08-17T17:26:40.410355+0000","last_peered":"2024-08-17T17:26:40.410355+0000","last_clean":"2024-08-17T17:26:40.410355+0000","last_became_active":"2024-08-17T17:20:49.257672+0000","last_became_peered":"2024-08-17T17:20:49.257672+0000","last_unstale":"2024-08-17T17:26:40.410355+0000","last_undegraded":"2024-08-17T17:26:40.410355+0000","last_fullsized":"2024-08-17T17:26:40.410355+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:08:11.734316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":172,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.408571+0000","last_change":"2024-08-17T17:18:59.430969+0000","last_active":"2024-08-17T17:26:40.408571+0000","last_peered":"2024-08-17T17:26:40.408571+0000","last_clean":"2024-08-17T17:26:40.408571+0000","last_became_active":"2024-08-17T17:18:59.413475+0000","last_became_peered":"2024-08-17T17:18:59.413475+0000","last_unstale":"2024-08-17T17:26:40.408571+0000","last_undegraded":"2024-08-17T17:26:40.408571+0000","last_fullsized":"2024-08-17T17:26:40.408571+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:36:33.732645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":291,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.109776+0000","last_change":"2024-08-17T17:25:04.506413+0000","last_active":"2024-08-17T17:26:54.109776+0000","last_peered":"2024-08-17T17:26:54.109776+0000","last_clean":"2024-08-17T17:26:54.109776+0000","last_became_active":"2024-08-17T17:25:04.506104+0000","last_became_peered":"2024-08-17T17:25:04.506104+0000","last_unstale":"2024-08-17T17:26:54.109776+0000","last_undegraded":"2024-08-17T17:26:54.109776+0000","last_fullsized":"2024-08-17T17:26:54.109776+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:41:53.556223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":115,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.408719+0000","last_change":"2024-08-17T17:20:49.279458+0000","last_active":"2024-08-17T17:26:40.408719+0000","last_peered":"2024-08-17T17:26:40.408719+0000","last_clean":"2024-08-17T17:26:40.408719+0000","last_became_active":"2024-08-17T17:20:49.260790+0000","last_became_peered":"2024-08-17T17:20:49.260790+0000","last_unstale":"2024-08-17T17:26:40.408719+0000","last_undegraded":"2024-08-17T17:26:40.408719+0000","last_fullsized":"2024-08-17T17:26:40.408719+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:58.092945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":29,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.444714+0000","last_change":"2024-08-17T17:24:53.463713+0000","last_active":"2024-08-17T17:26:40.444714+0000","last_peered":"2024-08-17T17:26:40.444714+0000","last_clean":"2024-08-17T17:26:40.444714+0000","last_became_active":"2024-08-17T17:24:53.463419+0000","last_became_peered":"2024-08-17T17:24:53.463419+0000","last_unstale":"2024-08-17T17:26:40.444714+0000","last_undegraded":"2024-08-17T17:26:40.444714+0000","last_fullsized":"2024-08-17T17:26:40.444714+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:34:44.545175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":247,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.104973+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:26:54.104973+0000","last_peered":"2024-08-17T17:26:54.104973+0000","last_clean":"2024-08-17T17:26:54.104973+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:26:54.104973+0000","last_undegraded":"2024-08-17T17:26:54.104973+0000","last_fullsized":"2024-08-17T17:26:54.104973+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":274,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.113693+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:26:54.113693+0000","last_peered":"2024-08-17T17:26:54.113693+0000","last_clean":"2024-08-17T17:26:54.113693+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:26:54.113693+0000","last_undegraded":"2024-08-17T17:26:54.113693+0000","last_fullsized":"2024-08-17T17:26:54.113693+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":294,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.106211+0000","last_change":"2024-08-17T17:25:04.509422+0000","last_active":"2024-08-17T17:26:54.106211+0000","last_peered":"2024-08-17T17:26:54.106211+0000","last_clean":"2024-08-17T17:26:54.106211+0000","last_became_active":"2024-08-17T17:25:04.508742+0000","last_became_peered":"2024-08-17T17:25:04.508742+0000","last_unstale":"2024-08-17T17:26:54.106211+0000","last_undegraded":"2024-08-17T17:26:54.106211+0000","last_fullsized":"2024-08-17T17:26:54.106211+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:18:31.976907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":170,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.410798+0000","last_change":"2024-08-17T17:16:26.963696+0000","last_active":"2024-08-17T17:26:40.410798+0000","last_peered":"2024-08-17T17:26:40.410798+0000","last_clean":"2024-08-17T17:26:40.410798+0000","last_became_active":"2024-08-17T17:16:26.963280+0000","last_became_peered":"2024-08-17T17:16:26.963280+0000","last_unstale":"2024-08-17T17:26:40.410798+0000","last_undegraded":"2024-08-17T17:26:40.410798+0000","last_fullsized":"2024-08-17T17:26:40.410798+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:22:01.474840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":311,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.106443+0000","last_change":"2024-08-17T17:24:06.586886+0000","last_active":"2024-08-17T17:26:54.106443+0000","last_peered":"2024-08-17T17:26:54.106443+0000","last_clean":"2024-08-17T17:26:54.106443+0000","last_became_active":"2024-08-17T17:24:06.586516+0000","last_became_peered":"2024-08-17T17:24:06.586516+0000","last_unstale":"2024-08-17T17:26:54.106443+0000","last_undegraded":"2024-08-17T17:26:54.106443+0000","last_fullsized":"2024-08-17T17:26:54.106443+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:03:30.002726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":91,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.108372+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:26:54.108372+0000","last_peered":"2024-08-17T17:26:54.108372+0000","last_clean":"2024-08-17T17:26:54.108372+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:26:54.108372+0000","last_undegraded":"2024-08-17T17:26:54.108372+0000","last_fullsized":"2024-08-17T17:26:54.108372+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":271,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.114125+0000","last_change":"2024-08-17T17:18:59.412463+0000","last_active":"2024-08-17T17:26:54.114125+0000","last_peered":"2024-08-17T17:26:54.114125+0000","last_clean":"2024-08-17T17:26:54.114125+0000","last_became_active":"2024-08-17T17:18:59.412134+0000","last_became_peered":"2024-08-17T17:18:59.412134+0000","last_unstale":"2024-08-17T17:26:54.114125+0000","last_undegraded":"2024-08-17T17:26:54.114125+0000","last_fullsized":"2024-08-17T17:26:54.114125+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:44.463153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":314,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.105356+0000","last_change":"2024-08-17T17:25:04.510854+0000","last_active":"2024-08-17T17:26:54.105356+0000","last_peered":"2024-08-17T17:26:54.105356+0000","last_clean":"2024-08-17T17:26:54.105356+0000","last_became_active":"2024-08-17T17:25:04.510462+0000","last_became_peered":"2024-08-17T17:25:04.510462+0000","last_unstale":"2024-08-17T17:26:54.105356+0000","last_undegraded":"2024-08-17T17:26:54.105356+0000","last_fullsized":"2024-08-17T17:26:54.105356+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:55:54.762086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":212,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.110088+0000","last_change":"2024-08-17T17:24:06.591165+0000","last_active":"2024-08-17T17:26:54.110088+0000","last_peered":"2024-08-17T17:26:54.110088+0000","last_clean":"2024-08-17T17:26:54.110088+0000","last_became_active":"2024-08-17T17:24:06.590785+0000","last_became_peered":"2024-08-17T17:24:06.590785+0000","last_unstale":"2024-08-17T17:26:54.110088+0000","last_undegraded":"2024-08-17T17:26:54.110088+0000","last_fullsized":"2024-08-17T17:26:54.110088+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:54:05.775563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":283,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.110488+0000","last_change":"2024-08-17T17:24:32.300560+0000","last_active":"2024-08-17T17:26:54.110488+0000","last_peered":"2024-08-17T17:26:54.110488+0000","last_clean":"2024-08-17T17:26:54.110488+0000","last_became_active":"2024-08-17T17:24:32.300228+0000","last_became_peered":"2024-08-17T17:24:32.300228+0000","last_unstale":"2024-08-17T17:26:54.110488+0000","last_undegraded":"2024-08-17T17:26:54.110488+0000","last_fullsized":"2024-08-17T17:26:54.110488+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:18:05.475449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":91,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.107925+0000","last_change":"2024-08-17T17:24:06.586379+0000","last_active":"2024-08-17T17:26:54.107925+0000","last_peered":"2024-08-17T17:26:54.107925+0000","last_clean":"2024-08-17T17:26:54.107925+0000","last_became_active":"2024-08-17T17:24:06.586023+0000","last_became_peered":"2024-08-17T17:24:06.586023+0000","last_unstale":"2024-08-17T17:26:54.107925+0000","last_undegraded":"2024-08-17T17:26:54.107925+0000","last_fullsized":"2024-08-17T17:26:54.107925+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:45.701637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":301,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.105628+0000","last_change":"2024-08-17T17:21:50.442535+0000","last_active":"2024-08-17T17:26:54.105628+0000","last_peered":"2024-08-17T17:26:54.105628+0000","last_clean":"2024-08-17T17:26:54.105628+0000","last_became_active":"2024-08-17T17:21:50.441891+0000","last_became_peered":"2024-08-17T17:21:50.441891+0000","last_unstale":"2024-08-17T17:26:54.105628+0000","last_undegraded":"2024-08-17T17:26:54.105628+0000","last_fullsized":"2024-08-17T17:26:54.105628+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:46:24.025407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":225,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.444396+0000","last_change":"2024-08-17T17:25:04.508045+0000","last_active":"2024-08-17T17:26:40.444396+0000","last_peered":"2024-08-17T17:26:40.444396+0000","last_clean":"2024-08-17T17:26:40.444396+0000","last_became_active":"2024-08-17T17:25:04.505083+0000","last_became_peered":"2024-08-17T17:25:04.505083+0000","last_unstale":"2024-08-17T17:26:40.444396+0000","last_undegraded":"2024-08-17T17:26:40.444396+0000","last_fullsized":"2024-08-17T17:26:40.444396+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:23:42.993544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":287,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.106064+0000","last_change":"2024-08-17T17:18:59.418434+0000","last_active":"2024-08-17T17:26:54.106064+0000","last_peered":"2024-08-17T17:26:54.106064+0000","last_clean":"2024-08-17T17:26:54.106064+0000","last_became_active":"2024-08-17T17:18:59.418043+0000","last_became_peered":"2024-08-17T17:18:59.418043+0000","last_unstale":"2024-08-17T17:26:54.106064+0000","last_undegraded":"2024-08-17T17:26:54.106064+0000","last_fullsized":"2024-08-17T17:26:54.106064+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:25:51.371820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":198,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.444551+0000","last_change":"2024-08-17T17:20:49.291167+0000","last_active":"2024-08-17T17:26:40.444551+0000","last_peered":"2024-08-17T17:26:40.444551+0000","last_clean":"2024-08-17T17:26:40.444551+0000","last_became_active":"2024-08-17T17:20:49.263750+0000","last_became_peered":"2024-08-17T17:20:49.263750+0000","last_unstale":"2024-08-17T17:26:40.444551+0000","last_undegraded":"2024-08-17T17:26:40.444551+0000","last_fullsized":"2024-08-17T17:26:40.444551+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:02.863246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":300,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.105328+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:26:54.105328+0000","last_peered":"2024-08-17T17:26:54.105328+0000","last_clean":"2024-08-17T17:26:54.105328+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:26:54.105328+0000","last_undegraded":"2024-08-17T17:26:54.105328+0000","last_fullsized":"2024-08-17T17:26:54.105328+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":300,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.105736+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:26:54.105736+0000","last_peered":"2024-08-17T17:26:54.105736+0000","last_clean":"2024-08-17T17:26:54.105736+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:26:54.105736+0000","last_undegraded":"2024-08-17T17:26:54.105736+0000","last_fullsized":"2024-08-17T17:26:54.105736+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-08-17T17:26:40.409934+0000","last_change":"2024-08-17T17:25:04.503834+0000","last_active":"2024-08-17T17:26:40.409934+0000","last_peered":"2024-08-17T17:26:40.409934+0000","last_clean":"2024-08-17T17:26:40.409934+0000","last_became_active":"2024-08-17T17:25:04.502712+0000","last_became_peered":"2024-08-17T17:25:04.502712+0000","last_unstale":"2024-08-17T17:26:40.409934+0000","last_undegraded":"2024-08-17T17:26:40.409934+0000","last_fullsized":"2024-08-17T17:26:40.409934+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:45.389103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":110,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-08-17T17:26:54.113607+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:26:54.113607+0000","last_peered":"2024-08-17T17:26:54.113607+0000","last_clean":"2024-08-17T17:26:54.113607+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:26:54.113607+0000","last_undegraded":"2024-08-17T17:26:54.113607+0000","last_fullsized":"2024-08-17T17:26:54.113607+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":3241738264,"num_objects":49466,"num_object_clones":0,"num_object_copies":148398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7706,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49466,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147828,"num_write_kb":6313345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13212,"num_bytes_recovered":764084224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10755584000,"data_stored":10755571784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114528,"ondisk_log_size":114528,"up":48,"acting":44,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":6304,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":16702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17971,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16789504,"data_stored":16777273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234243,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2083392,"kb_used_data":1773560,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91661760,"statfs":{"total":95995035648,"available":93861642240,"internally_reserved":0,"allocated":1816125440,"data_stored":1814892855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495878,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1878200,"kb_used_data":1580848,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":91866952,"statfs":{"total":95995035648,"available":94071758848,"internally_reserved":0,"allocated":1618788352,"data_stored":1617563659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757515,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1509872,"kb_used_data":1204200,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92235280,"statfs":{"total":95995035648,"available":94448926720,"internally_reserved":0,"allocated":1233100800,"data_stored":1231879778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019153,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435380,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2328488,"kb_used_data":2043936,"kb_used_omap":4,"kb_used_meta":284411,"kb_avail":91416664,"statfs":{"total":95995035648,"available":93610663936,"internally_reserved":0,"allocated":2092990464,"data_stored":2091777384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":291236874},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":104,"seq":446676598907,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2052024,"kb_used_data":1724976,"kb_used_omap":7,"kb_used_meta":326968,"kb_avail":91693128,"statfs":{"total":95995035648,"available":93893763072,"internally_reserved":0,"allocated":1766375424,"data_stored":1765151226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":334815394},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762434,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1858928,"kb_used_data":1488424,"kb_used_omap":7,"kb_used_meta":370424,"kb_avail":91886224,"statfs":{"total":95995035648,"available":94091493376,"internally_reserved":0,"allocated":1524146176,"data_stored":1522924895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7945,"internal_metadata":379314423},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":94,"seq":403726925960,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1070584,"kb_used_data":792240,"kb_used_omap":7,"kb_used_meta":278328,"kb_avail":92674568,"statfs":{"total":95995035648,"available":94898757632,"internally_reserved":0,"allocated":811253760,"data_stored":810025472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":285008193},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6241,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":806555648,"data_stored":806551576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1505562624,"data_stored":1505558552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1757478912,"data_stored":1757478912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2075721728,"data_stored":2075721728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1214189568,"data_stored":1214185496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1593769984,"data_stored":1593769984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1802305536,"data_stored":1802305536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:27:12.523 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph status --format=json 2024-08-17T17:27:12.596 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:12.625 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:12.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:12.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:12.727 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:12.769 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:12.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:12.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:12.774 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:27:12.774 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:27:12.804 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:27:12.804 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:27:12.872 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:12.887 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:27:12.887 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:27:12.887 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:12 smithi003 ceph-mon[103344]: osdmap e214: 8 total, 8 up, 7 in 2024-08-17T17:27:12.888 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:12 smithi003 ceph-mon[100615]: osdmap e214: 8 total, 8 up, 7 in 2024-08-17T17:27:12.911 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:12.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:12.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:12.931 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:27:12.932 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:27:12.962 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:27:12.962 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:27:12.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:12 smithi053 ceph-mon[89963]: osdmap e214: 8 total, 8 up, 7 in 2024-08-17T17:27:12.990 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:27:12.991 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:27:13.013 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:13.025 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:27:13.026 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:27:13.055 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:27:13.055 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:27:13.065 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:13.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:13.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:13.167 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:13.195 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:13.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:13.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:13.297 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:13.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:13.338 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:13.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:13.439 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:13.483 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:13.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:13.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:13.584 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:13.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:13.631 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:13.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:13.716 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:13 smithi003 ceph-mon[100615]: pgmap v564: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.7 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 783 op/s; 7289/133305 objects degraded (5.468%); 0 B/s, 80 objects/s recovering 2024-08-17T17:27:13.716 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:13 smithi003 ceph-mon[100615]: osdmap e215: 8 total, 8 up, 7 in 2024-08-17T17:27:13.716 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:13 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2345062439' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:27:13.716 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:13 smithi003 ceph-mon[103344]: pgmap v564: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.7 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 783 op/s; 7289/133305 objects degraded (5.468%); 0 B/s, 80 objects/s recovering 2024-08-17T17:27:13.716 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:13 smithi003 ceph-mon[103344]: osdmap e215: 8 total, 8 up, 7 in 2024-08-17T17:27:13.716 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:13 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2345062439' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:27:13.733 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:13.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:13.778 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:13.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:13.879 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:13.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:13.910 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:13.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:13.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:13 smithi053 ceph-mon[89963]: pgmap v564: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 2.7 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 783 op/s; 7289/133305 objects degraded (5.468%); 0 B/s, 80 objects/s recovering 2024-08-17T17:27:13.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:13 smithi053 ceph-mon[89963]: osdmap e215: 8 total, 8 up, 7 in 2024-08-17T17:27:13.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:13 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2345062439' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:27:14.011 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:14.079 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:14.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:14.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:14.180 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:14.213 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:14.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:14.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:14.315 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:14.343 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:14.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:14.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:14.445 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:14.478 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:14.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:14.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:14.579 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:14.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:14.635 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:14.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:14.736 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:14.761 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:14.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:14.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:14.863 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:14.911 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:14.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:14.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:15.013 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:15.038 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:15.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:15.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:15.140 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:15.181 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:15.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:15.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:15.286 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:15.324 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:15.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:15.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:15.426 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:15.468 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:15.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:15.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:15.570 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:15.606 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:15.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:15.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:15.708 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:15.760 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:15.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:15.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:15.765 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:15 smithi003 ceph-mon[103344]: pgmap v566: 89 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 763 op/s; 6929/130149 objects degraded (5.324%); 0 B/s, 97 objects/s recovering 2024-08-17T17:27:15.765 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:15 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 6929/130149 objects degraded (5.324%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:15.766 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:15 smithi003 ceph-mon[100615]: pgmap v566: 89 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 763 op/s; 6929/130149 objects degraded (5.324%); 0 B/s, 97 objects/s recovering 2024-08-17T17:27:15.766 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:15 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 6929/130149 objects degraded (5.324%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:15.766 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:27:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:27:15] "GET /metrics HTTP/1.1" 200 36514 "" "Prometheus/2.51.0" 2024-08-17T17:27:15.861 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:15.904 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:15.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:15.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:15.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:15 smithi053 ceph-mon[89963]: pgmap v566: 89 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 763 op/s; 6929/130149 objects degraded (5.324%); 0 B/s, 97 objects/s recovering 2024-08-17T17:27:15.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:15 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 6929/130149 objects degraded (5.324%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:16.006 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:16.034 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:16.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:16.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:16.136 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:16.175 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:16.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:16.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:16.277 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:16.304 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:16.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:16.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:16.405 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:16.433 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:16.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:16.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:16.535 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:16.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:16.581 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:16.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:16.682 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:16.707 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:16.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:16.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:16.809 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:16.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:16.877 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:16.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:16.978 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:17.004 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:17.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:17.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:17.107 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:17.134 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:17.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:17.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:17.236 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:17.265 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:17.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:17.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:17.367 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:17.414 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:17.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:17.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:17.516 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:17.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:17 smithi003 ceph-mon[103344]: pgmap v567: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 768 op/s; 6529/126408 objects degraded (5.165%); 0 B/s, 147 objects/s recovering 2024-08-17T17:27:17.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:17 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:27:17.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:17 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:27:17.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:17 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:27:17.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:17 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:27:17.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:17 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:27:17.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:27:17.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:17 smithi003 ceph-mon[100615]: pgmap v567: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 768 op/s; 6529/126408 objects degraded (5.165%); 0 B/s, 147 objects/s recovering 2024-08-17T17:27:17.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:17 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:27:17.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:17 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:27:17.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:17 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:27:17.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:17 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:27:17.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:17 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:27:17.541 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:17.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:17.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:17.643 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:17.669 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:17.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:17.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:17.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:17 smithi053 ceph-mon[89963]: pgmap v567: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 768 op/s; 6529/126408 objects degraded (5.165%); 0 B/s, 147 objects/s recovering 2024-08-17T17:27:17.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:17 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:27:17.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:17 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:27:17.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:17 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:27:17.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:17 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:27:17.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:17 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:27:17.771 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:17.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:17.809 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:17.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:17.910 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:17.941 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:17.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:17.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:18.042 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:18.065 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:18.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:18.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:18.167 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:18.211 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:18.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:18.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:18.313 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:18.338 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:18.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:18.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:18.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:18.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:18.480 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:18.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:18.581 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:18.612 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:18.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:18.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:18.713 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:18.742 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:18.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:18.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:18.843 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:18.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:18.878 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:18.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:18.979 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:19.017 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:19.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:19.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:19.119 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:19.143 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:19.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:19.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:19.245 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:19.289 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:19.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:19.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:19.391 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:19.426 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:19.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:19.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:19.496 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:19 smithi003 ceph-mon[100615]: pgmap v568: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 648 op/s; 6529/123315 objects degraded (5.295%); 0 B/s, 94 objects/s recovering 2024-08-17T17:27:19.496 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:19 smithi003 ceph-mon[103344]: pgmap v568: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 648 op/s; 6529/123315 objects degraded (5.295%); 0 B/s, 94 objects/s recovering 2024-08-17T17:27:19.529 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:19.557 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:27:19.585 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:19.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:19.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:19.687 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:19.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:19 smithi053 ceph-mon[89963]: pgmap v568: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 648 op/s; 6529/123315 objects degraded (5.295%); 0 B/s, 94 objects/s recovering 2024-08-17T17:27:19.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:19.744 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:19.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:19.845 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:19.871 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:19.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:19.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:19.972 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:20.001 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:20.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:20.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:20.103 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:20.143 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:20.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:20.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:20.245 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:20.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:20.300 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:20.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:20.401 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:20.439 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:20.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:20.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:20.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:20 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 6529/123315 objects degraded (5.295%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:20.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:27:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:27:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:27:20.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:20 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 6529/123315 objects degraded (5.295%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:20.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:20 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 6529/123315 objects degraded (5.295%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:20.541 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:20.567 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:20.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:20.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:20.669 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:20.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:20.717 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:20.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:20.818 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:20.840 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:20.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:20.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:20.942 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:20.971 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:20.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:20.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:21.073 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:21.099 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:21.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:21.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:21.201 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:21.231 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:21.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:21.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:21.333 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:21.380 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:21.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:21.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:21.482 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:21.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:21 smithi003 ceph-mon[103344]: pgmap v569: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 773 op/s; 6529/117312 objects degraded (5.566%); 0 B/s, 91 objects/s recovering 2024-08-17T17:27:21.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:21 smithi003 ceph-mon[100615]: pgmap v569: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 773 op/s; 6529/117312 objects degraded (5.566%); 0 B/s, 91 objects/s recovering 2024-08-17T17:27:21.525 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:21.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:21.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:21.627 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:21.668 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:21.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:21.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:21.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:21 smithi053 ceph-mon[89963]: pgmap v569: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 773 op/s; 6529/117312 objects degraded (5.566%); 0 B/s, 91 objects/s recovering 2024-08-17T17:27:21.770 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:21.799 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:21.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:21.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:21.901 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:21.929 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:21.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:21.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:21.984 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:27:21.985 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:27:21.985 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.985 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45491.0:161399 6.6 6:6854780b:::benchmark_data_smithi003_175043_object62750:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-08-17T17:27:21.985 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:27:21.746576+0000", 2024-08-17T17:27:21.985 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.011124393, 2024-08-17T17:27:21.985 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011153412, 2024-08-17T17:27:21.985 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:27:21.985 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:27:21.985 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:27:21.985 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:27:21.985 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45491", 2024-08-17T17:27:21.986 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:27:21.986 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 161399 2024-08-17T17:27:21.986 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:21.986 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:27:21.986 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.986 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:27:21.986 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.746576+0000", 2024-08-17T17:27:21.986 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:21.986 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:21.986 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.986 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:27:21.986 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.746576+0000", 2024-08-17T17:27:21.986 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:21.986 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:21.987 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.987 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:27:21.987 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.746577+0000", 2024-08-17T17:27:21.987 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4780000000000001e-06 2024-08-17T17:27:21.987 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:21.987 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.987 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:27:21.987 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.746582+0000", 2024-08-17T17:27:21.987 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8840000000000002e-06 2024-08-17T17:27:21.988 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:21.988 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.988 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:27:21.988 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.746598+0000", 2024-08-17T17:27:21.988 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6036e-05 2024-08-17T17:27:21.988 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:21.988 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.988 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:27:21.988 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.746606+0000", 2024-08-17T17:27:21.988 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3719999999999999e-06 2024-08-17T17:27:21.988 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:21.988 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.988 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:21.988 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.746653+0000", 2024-08-17T17:27:21.989 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7729000000000003e-05 2024-08-17T17:27:21.989 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:21.989 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.989 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:27:21.989 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.746760+0000", 2024-08-17T17:27:21.989 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010678100000000001 2024-08-17T17:27:21.989 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:21.989 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.989 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 6,7", 2024-08-17T17:27:21.989 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.746875+0000", 2024-08-17T17:27:21.990 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000114477 2024-08-17T17:27:21.990 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:21.990 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.990 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:21.990 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.756549+0000", 2024-08-17T17:27:21.990 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0096747169999999993 2024-08-17T17:27:21.990 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:21.990 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.990 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:21.990 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.756595+0000", 2024-08-17T17:27:21.990 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6112000000000002e-05 2024-08-17T17:27:21.990 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:21.990 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:27:21.990 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:21.991 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:21.991 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.991 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45491.0:161402 6.2 6:4dbf2830:::benchmark_data_smithi003_175043_object62753:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-08-17T17:27:21.991 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:27:21.751510+0000", 2024-08-17T17:27:21.991 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0061905190000000002, 2024-08-17T17:27:21.991 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063027150000000004, 2024-08-17T17:27:21.991 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:27:21.991 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:27:21.991 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:27:21.991 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:27:21.991 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45491", 2024-08-17T17:27:21.991 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:27:21.991 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 161402 2024-08-17T17:27:21.992 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:21.992 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:27:21.992 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.992 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:27:21.992 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.751510+0000", 2024-08-17T17:27:21.992 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:21.992 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:21.992 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.992 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:27:21.992 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.751510+0000", 2024-08-17T17:27:21.992 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:21.992 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:21.992 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.992 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:27:21.993 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.751511+0000", 2024-08-17T17:27:21.993 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.488e-06 2024-08-17T17:27:21.993 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:21.993 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.993 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:27:21.993 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.751517+0000", 2024-08-17T17:27:21.993 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0809999999999999e-06 2024-08-17T17:27:21.993 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:21.993 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.993 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:27:21.993 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.751530+0000", 2024-08-17T17:27:21.993 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2228999999999999e-05 2024-08-17T17:27:21.993 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:21.993 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.993 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:27:21.993 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.751537+0000", 2024-08-17T17:27:21.994 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7419999999999997e-06 2024-08-17T17:27:21.994 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:21.994 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.994 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:21.994 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.751806+0000", 2024-08-17T17:27:21.994 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00026861700000000002 2024-08-17T17:27:21.994 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:21.994 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.994 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:27:21.994 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.751962+0000", 2024-08-17T17:27:21.994 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000156205 2024-08-17T17:27:21.994 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:21.995 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.995 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,6", 2024-08-17T17:27:21.995 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.752097+0000", 2024-08-17T17:27:21.995 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013510900000000001 2024-08-17T17:27:21.995 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:21.995 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.995 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:21.995 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.757081+0000", 2024-08-17T17:27:21.995 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004983805 2024-08-17T17:27:21.995 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:21.995 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:27:21.995 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:21.995 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:21.995 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.996 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45491.0:161404 6.c 6:36ed452e:::benchmark_data_smithi003_175043_object62755:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-08-17T17:27:21.996 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:27:21.752629+0000", 2024-08-17T17:27:21.996 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0050713570000000003, 2024-08-17T17:27:21.996 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0052549240000000002, 2024-08-17T17:27:21.996 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:27:21.996 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:27:21.996 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:27:21.996 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:27:21.996 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45491", 2024-08-17T17:27:21.996 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:27:21.996 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 161404 2024-08-17T17:27:21.996 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:21.996 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:27:21.996 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.997 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:27:21.997 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.752629+0000", 2024-08-17T17:27:21.997 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:21.997 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:21.997 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.997 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:27:21.997 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.752629+0000", 2024-08-17T17:27:21.997 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:21.997 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:21.997 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.997 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:27:21.997 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.752630+0000", 2024-08-17T17:27:21.998 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0359999999999999e-06 2024-08-17T17:27:21.998 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:21.998 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.998 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:27:21.998 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.752634+0000", 2024-08-17T17:27:21.998 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5240000000000002e-06 2024-08-17T17:27:21.998 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:21.998 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.998 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:27:21.998 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.752645+0000", 2024-08-17T17:27:21.998 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0755e-05 2024-08-17T17:27:21.998 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:21.998 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.998 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:27:21.999 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.752652+0000", 2024-08-17T17:27:21.999 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9770000000000003e-06 2024-08-17T17:27:21.999 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:21.999 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.999 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:21.999 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.752844+0000", 2024-08-17T17:27:21.999 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019141199999999999 2024-08-17T17:27:21.999 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:21.999 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:21.999 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:27:22.000 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.752966+0000", 2024-08-17T17:27:22.000 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012232399999999999 2024-08-17T17:27:22.000 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:22.000 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:22.000 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,6", 2024-08-17T17:27:22.000 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.753092+0000", 2024-08-17T17:27:22.000 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000125597 2024-08-17T17:27:22.000 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:22.000 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:22.000 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:22.000 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.756972+0000", 2024-08-17T17:27:22.000 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038803190000000001 2024-08-17T17:27:22.000 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:22.001 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:22.001 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:27:22.001 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.757876+0000", 2024-08-17T17:27:22.001 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00090406000000000004 2024-08-17T17:27:22.001 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:22.001 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:27:22.001 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:22.001 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:22.001 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:22.001 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45491.0:161405 6.7 e215/178 6:e4adc8ce:::benchmark_data_smithi003_175043_object62756:head v 215'10507, mlcod=215'10505)", 2024-08-17T17:27:22.001 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:27:21.754920+0000", 2024-08-17T17:27:22.001 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0027806469999999998, 2024-08-17T17:27:22.001 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0030343879999999998, 2024-08-17T17:27:22.001 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:27:22.002 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:27:22.002 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:27:22.002 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:27:22.002 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:22.002 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:27:22.002 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.754920+0000", 2024-08-17T17:27:22.002 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:22.002 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:22.002 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:22.002 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:27:22.002 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.754920+0000", 2024-08-17T17:27:22.002 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:22.003 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:22.003 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:22.003 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:27:22.003 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.754920+0000", 2024-08-17T17:27:22.003 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2399999999999997e-07 2024-08-17T17:27:22.003 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:22.003 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:22.003 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:27:22.003 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.754924+0000", 2024-08-17T17:27:22.003 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9920000000000001e-06 2024-08-17T17:27:22.003 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:22.003 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:22.003 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:27:22.004 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.754935+0000", 2024-08-17T17:27:22.004 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1102e-05 2024-08-17T17:27:22.004 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:22.004 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:22.004 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:27:22.004 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.754943+0000", 2024-08-17T17:27:22.004 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3880000000000002e-06 2024-08-17T17:27:22.004 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:22.004 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:22.004 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:22.004 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.755089+0000", 2024-08-17T17:27:22.004 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014639199999999999 2024-08-17T17:27:22.004 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:22.004 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:22.004 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:27:22.005 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.755111+0000", 2024-08-17T17:27:22.005 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1690000000000001e-05 2024-08-17T17:27:22.005 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:22.005 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:27:22.005 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:22.005 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:22.005 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:22.005 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45491.0:161406 6.0 e215/178 6:08811e6c:::benchmark_data_smithi003_175043_object62757:head v 215'10129, mlcod=215'10126)", 2024-08-17T17:27:22.005 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:27:21.757583+0000", 2024-08-17T17:27:22.005 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.000117754, 2024-08-17T17:27:22.005 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00041851799999999999, 2024-08-17T17:27:22.005 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:27:22.005 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:27:22.006 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:27:22.006 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:27:22.006 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:22.006 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:27:22.006 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.757583+0000", 2024-08-17T17:27:22.006 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:22.006 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:22.006 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:22.006 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:27:22.007 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.757583+0000", 2024-08-17T17:27:22.007 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:22.007 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:22.007 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:22.007 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:27:22.007 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.757583+0000", 2024-08-17T17:27:22.007 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4000000000000001e-07 2024-08-17T17:27:22.007 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:22.007 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:22.007 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:27:22.007 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.757589+0000", 2024-08-17T17:27:22.007 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6799999999999998e-06 2024-08-17T17:27:22.007 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:22.007 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:22.008 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:27:22.008 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.757599+0000", 2024-08-17T17:27:22.008 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.007e-05 2024-08-17T17:27:22.008 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:22.008 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:22.008 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:27:22.008 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.757605+0000", 2024-08-17T17:27:22.008 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3690000000000003e-06 2024-08-17T17:27:22.008 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:22.008 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:22.008 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:22.008 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.757933+0000", 2024-08-17T17:27:22.009 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00032745300000000001 2024-08-17T17:27:22.009 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:22.009 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:22.009 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:27:22.009 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:21.757951+0000", 2024-08-17T17:27:22.009 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7978999999999998e-05 2024-08-17T17:27:22.009 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:22.009 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:27:22.009 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:22.009 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:22.009 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-08-17T17:27:22.009 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 5 2024-08-17T17:27:22.009 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:27:22.031 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:22.069 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:22.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:22.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:22.171 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:22.210 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:22.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:22.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:22.312 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:22.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:22.342 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:22.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:22.443 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:22.473 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:22.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:22.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:22.574 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:22.620 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:22.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:22.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:22.722 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:22.747 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:22.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:22.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:22.849 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:22.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:22.911 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:22.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:23.012 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:23.042 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:23.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:23.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:23.144 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:23.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:23.174 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:23.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:23.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:23 smithi003 ceph-mon[103344]: pgmap v570: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.3 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 727 op/s; 6232/111459 objects degraded (5.591%); 0 B/s, 105 objects/s recovering 2024-08-17T17:27:23.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:27:23.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:23 smithi003 ceph-mon[100615]: pgmap v570: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.3 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 727 op/s; 6232/111459 objects degraded (5.591%); 0 B/s, 105 objects/s recovering 2024-08-17T17:27:23.275 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:27:23.275 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:23.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:23.301 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:23.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:23.402 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:23.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:23.429 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:23.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:23.530 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:23.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:23.580 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:23.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:23.681 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:23.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:23.709 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:23.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:23 smithi053 ceph-mon[89963]: pgmap v570: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.3 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 727 op/s; 6232/111459 objects degraded (5.591%); 0 B/s, 105 objects/s recovering 2024-08-17T17:27:23.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:27:23.810 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:23.853 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:23.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:23.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:23.955 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:23.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:23.979 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:23.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:24.080 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:24.103 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:24.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:24.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:24.205 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:24.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:24.247 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:24.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:24.260 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-08-17T17:27:24.347 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:24.374 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:24.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:24.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:24.476 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:24.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:24.506 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:24.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:24.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:24 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3337083469' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:27:24.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:24 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:27:24.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:24 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3337083469' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:27:24.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:24 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:27:24.608 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:24.650 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:24.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:24.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:24.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:24 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3337083469' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:27:24.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:24 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:27:24.752 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:24.796 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:24.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:24.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:24.821 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-17T17:27:24.898 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:24.927 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:24.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:24.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:25.029 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:25.059 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:25.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:25.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:25.161 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:25.230 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:25.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:25.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:25.308 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:27:25.331 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:25.356 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:25.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:25.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:25.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:25 smithi003 ceph-mon[103344]: pgmap v571: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 746 op/s; 6232/107922 objects degraded (5.775%); 0 B/s, 93 objects/s recovering 2024-08-17T17:27:25.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:25 smithi003 ceph-mon[103344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:27:25.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:25 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:27:25.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:25 smithi003 ceph-mon[103344]: osdmap e216: 8 total, 8 up, 7 in 2024-08-17T17:27:25.434 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:25 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 6232/107922 objects degraded (5.775%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:25.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:25 smithi003 ceph-mon[100615]: pgmap v571: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 746 op/s; 6232/107922 objects degraded (5.775%); 0 B/s, 93 objects/s recovering 2024-08-17T17:27:25.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:25 smithi003 ceph-mon[100615]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:27:25.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:25 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:27:25.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:25 smithi003 ceph-mon[100615]: osdmap e216: 8 total, 8 up, 7 in 2024-08-17T17:27:25.435 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:25 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 6232/107922 objects degraded (5.775%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:25.458 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:25.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:25.515 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:25.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:25.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:25.663 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:25.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:25.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:25.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:25 smithi053 ceph-mon[89963]: pgmap v571: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.2 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 746 op/s; 6232/107922 objects degraded (5.775%); 0 B/s, 93 objects/s recovering 2024-08-17T17:27:25.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:25 smithi053 ceph-mon[89963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:27:25.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:25 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:27:25.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:25 smithi053 ceph-mon[89963]: osdmap e216: 8 total, 8 up, 7 in 2024-08-17T17:27:25.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:25 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 6232/107922 objects degraded (5.775%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:25.743 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:27:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:27:25] "GET /metrics HTTP/1.1" 200 36508 "" "Prometheus/2.51.0" 2024-08-17T17:27:25.765 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:25.812 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:25.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:25.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:25.915 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:25.940 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:25.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:25.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:26.042 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:26.088 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:26.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:26.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:26.190 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:26.222 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:26.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:26.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:26.324 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:26.366 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:26.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:26.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:26.395 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:27:26.395 INFO:teuthology.orchestra.run.smithi003.stdout:{"fsid":"2b5b662e-5cba-11ef-bccf-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5799/103872 objects degraded (5.583%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":788,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":216,"num_osds":8,"num_up_osds":8,"osd_up_since":1723915051,"num_in_osds":7,"osd_in_since":1723915248,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":86},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":89,"num_pools":5,"num_objects":34624,"data_bytes":2272008165,"bytes_used":10027196416,"bytes_avail":661938053120,"bytes_total":671965249536,"degraded_objects":5799,"degraded_total":103872,"degraded_ratio":0.055828327171903884,"recovering_objects_per_sec":72,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":730,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":750},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi003.civgvf","status":"up:active","gid":14664}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.3:8443/","prometheus":"http://172.21.15.3:9283/"}},"servicemap":{"epoch":23,"modified":"2024-08-17T17:18:31.688676+0000","services":{}},"progress_events":{"714d66b4-625a-4d60-aa1c-f38ff47ebe12":{"message":"Global Recovery Event (2m)\n [===========================.] (remaining: 4s)","progress":0.96629214286804199,"add_to_ceph_s":true},"fcbeb9b7-7baf-488b-9fe4-c01aca441f9c":{"message":"Upgrade to 19.0.0-5836-gcd8bdbeb (4m)\n [============................] (remaining: 5m)","progress":0.44999998807907104,"add_to_ceph_s":true}}} 2024-08-17T17:27:26.468 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:26.497 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:26.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:26.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:26.599 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:26.627 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:26.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:26.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:26.729 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:26.777 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:26.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:26.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:26.879 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:26.909 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:26.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:26.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:27.011 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:27.038 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:27.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:27.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:27.140 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:27.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:27.176 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:27.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:27.277 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:27.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:27.309 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:27.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:27.410 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:27.437 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:27.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:27.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:27.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:27 smithi003 ceph-mon[103344]: pgmap v573: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.1 GiB data, 9.3 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 750 op/s; 5799/103872 objects degraded (5.583%); 0 B/s, 72 objects/s recovering 2024-08-17T17:27:27.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:27 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/188811589' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T17:27:27.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:27 smithi003 ceph-mon[100615]: pgmap v573: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.1 GiB data, 9.3 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 750 op/s; 5799/103872 objects degraded (5.583%); 0 B/s, 72 objects/s recovering 2024-08-17T17:27:27.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:27 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/188811589' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T17:27:27.540 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:27.585 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:27.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:27.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:27.687 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:27.714 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:27.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:27.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:27.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:27 smithi053 ceph-mon[89963]: pgmap v573: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.1 GiB data, 9.3 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 750 op/s; 5799/103872 objects degraded (5.583%); 0 B/s, 72 objects/s recovering 2024-08-17T17:27:27.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:27 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/188811589' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T17:27:27.816 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:27.848 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:27.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:27.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:27.950 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:27.977 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:28.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:28.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:28.107 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:28.150 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:28.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:28.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:28.252 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:28.279 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:28.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:28.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:28.381 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:28.410 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:27:28.411 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:28.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:28.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:28.456 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-08-17T17:27:28.456 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:27:28.513 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:28.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:28.542 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:28.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:28.643 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:28.668 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:28.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:28.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:28.770 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:28.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:28.799 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:28.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:28.899 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:28.929 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:28.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:28.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:29.031 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:29.077 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:29.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:29.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:29.179 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:29.210 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:29.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:29.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:29.312 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:29.341 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:29.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:29.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:29.443 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:29.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:29.474 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:29.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:29.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:29 smithi003 ceph-mon[103344]: pgmap v574: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.0 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 775 op/s; 5799/100041 objects degraded (5.797%); 0 B/s, 72 objects/s recovering 2024-08-17T17:27:29.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:29 smithi003 ceph-mon[100615]: pgmap v574: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.0 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 775 op/s; 5799/100041 objects degraded (5.797%); 0 B/s, 72 objects/s recovering 2024-08-17T17:27:29.575 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:29.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:29.614 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:29.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:29.715 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:29.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:29 smithi053 ceph-mon[89963]: pgmap v574: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 2.0 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 775 op/s; 5799/100041 objects degraded (5.797%); 0 B/s, 72 objects/s recovering 2024-08-17T17:27:29.747 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:29.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:29.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:29.764 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:27:29.849 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:29.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:29.948 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:29.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:30.049 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:30.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:30.098 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:30.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:30.199 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:30.248 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:30.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:30.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:30.353 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:30.397 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:30.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:30.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:30.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:30 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 5799/100041 objects degraded (5.797%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:30.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:27:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:27:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:27:30.499 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:30.524 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:30.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:30.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:30.626 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:30.658 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:30 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 5799/100041 objects degraded (5.797%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:30.659 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:30 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 5799/100041 objects degraded (5.797%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:30.676 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:30.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:30.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:30.778 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:30.833 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:30.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:30.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:30.836 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:27:30.934 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:30.989 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:30.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:30.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:31.091 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:31.132 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:31.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:31.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:31.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:31.321 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:31.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:31.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:31.423 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:31.479 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:31.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:31.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:31.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:31 smithi003 ceph-mon[103344]: pgmap v575: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.9 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 779 op/s; 5799/93909 objects degraded (6.175%); 0 B/s, 72 objects/s recovering 2024-08-17T17:27:31.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:31 smithi003 ceph-mon[100615]: pgmap v575: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.9 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 779 op/s; 5799/93909 objects degraded (6.175%); 0 B/s, 72 objects/s recovering 2024-08-17T17:27:31.581 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:31.606 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:31.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:31.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:31.708 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:31.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:31 smithi053 ceph-mon[89963]: pgmap v575: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.9 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 779 op/s; 5799/93909 objects degraded (6.175%); 0 B/s, 72 objects/s recovering 2024-08-17T17:27:31.749 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:31.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:31.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:31.851 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:31.892 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:31.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:31.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:31.994 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:32.034 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:32.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:32.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:32.136 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:32.165 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:32.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:32.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:32.267 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:32.293 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:32.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:32.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:32.323 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:27:32.395 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:32.401 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:27:32.402 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:27:32.423 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:32.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:32.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:32.441 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:27:32.442 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:27:32.484 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:27:32.485 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:27:32.525 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:32.529 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:27:32.529 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:27:32.560 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:32.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:32.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:32.563 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:27:32.563 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:27:32.596 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:27:32.596 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:27:32.635 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:27:32.635 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:27:32.662 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:32.669 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:27:32.669 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:27:32.707 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:32.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:32.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:32.809 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:32.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:32.842 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:32.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:32.943 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:32.972 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:32.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:32.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:33.074 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:33.105 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:33.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:33.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:33.207 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:33.234 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:33.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:33.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:33.336 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:33.373 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:33.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:33.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:33.455 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:33 smithi003 ceph-mon[100615]: pgmap v576: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.8 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 797 op/s; 5430/87519 objects degraded (6.204%); 0 B/s, 80 objects/s recovering 2024-08-17T17:27:33.455 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:33 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/4012835020' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:27:33.455 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:33 smithi003 ceph-mon[103344]: pgmap v576: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.8 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 797 op/s; 5430/87519 objects degraded (6.204%); 0 B/s, 80 objects/s recovering 2024-08-17T17:27:33.455 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:33 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/4012835020' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:27:33.475 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:33.501 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:33.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:33.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:33.603 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:33.633 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:33.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:33.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:33.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:33 smithi053 ceph-mon[89963]: pgmap v576: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.8 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 797 op/s; 5430/87519 objects degraded (6.204%); 0 B/s, 80 objects/s recovering 2024-08-17T17:27:33.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:33 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/4012835020' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:27:33.735 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:33.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:33.780 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:33.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:33.881 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:33.928 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:33.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:33.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:34.030 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:34.056 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:34.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:34.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:34.158 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:34.200 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:34.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:34.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:34.302 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:34.329 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:34.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:34.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:34.431 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:34.460 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:34.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:34.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:34.562 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:34.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:34.591 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:34.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:34.692 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:34.718 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:34.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:34.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:34.820 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:34.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:34.863 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:34.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:34.964 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:34.998 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:34.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:34.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:35.100 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:35.128 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:35.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:35.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:35.230 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:35.259 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:35.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:35.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:35.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:35.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:35.411 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:35.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:35.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:27:35.512 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:35.552 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:35 smithi003 ceph-mon[100615]: pgmap v577: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.7 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 809 op/s; 5430/83619 objects degraded (6.494%); 0 B/s, 80 objects/s recovering 2024-08-17T17:27:35.553 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:35 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 5430/83619 objects degraded (6.494%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:35.554 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:35 smithi003 ceph-mon[103344]: pgmap v577: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.7 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 809 op/s; 5430/83619 objects degraded (6.494%); 0 B/s, 80 objects/s recovering 2024-08-17T17:27:35.554 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:35 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 5430/83619 objects degraded (6.494%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:35.603 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:35.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:35.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:35.705 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:35.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:35 smithi053 ceph-mon[89963]: pgmap v577: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.7 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 809 op/s; 5430/83619 objects degraded (6.494%); 0 B/s, 80 objects/s recovering 2024-08-17T17:27:35.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:35 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 5430/83619 objects degraded (6.494%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:35.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:35.749 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:35.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:35.850 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:35.864 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:27:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:27:35] "GET /metrics HTTP/1.1" 200 36508 "" "Prometheus/2.51.0" 2024-08-17T17:27:35.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:35.903 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:35.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:36.004 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:36.032 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:36.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:36.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:36.134 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:36.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:36.173 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:36.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:36.274 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:36.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:36.324 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:36.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:36.425 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:36.453 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:36.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:36.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:36.555 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:36.618 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:36.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:36.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:36.720 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:36.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:36.746 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:36.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:36.847 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:36.891 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:36.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:36.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:36.993 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:37.015 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:37.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:37.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:37.117 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:37.153 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:37.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:37.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:37.255 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:37.282 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:37.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:37.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:37.384 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:37.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:37.411 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:37.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:37.512 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:37.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:37.543 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:37.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:37.645 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:37.664 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:27:37.665 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:37 smithi003 ceph-mon[103344]: pgmap v578: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.6 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 774 op/s; 5010/78774 objects degraded (6.360%); 0 B/s, 104 objects/s recovering 2024-08-17T17:27:37.665 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:37 smithi003 ceph-mon[103344]: from='client.45674 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:27:37.667 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:37 smithi003 ceph-mon[100615]: pgmap v578: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.6 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 774 op/s; 5010/78774 objects degraded (6.360%); 0 B/s, 104 objects/s recovering 2024-08-17T17:27:37.667 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:37 smithi003 ceph-mon[100615]: from='client.45674 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:27:37.670 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:27:37.672 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:37.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:37.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:37.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:37 smithi053 ceph-mon[89963]: pgmap v578: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.6 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 774 op/s; 5010/78774 objects degraded (6.360%); 0 B/s, 104 objects/s recovering 2024-08-17T17:27:37.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:37 smithi053 ceph-mon[89963]: from='client.45674 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:27:37.775 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:37.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:37.803 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:37.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:37.904 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:37.933 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:37.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:37.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:38.035 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:38.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:38.080 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:38.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:38.182 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:38.210 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:38.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:38.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:38.312 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:38.355 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:38.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:38.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:38.457 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:38.497 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:38.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:38.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:38.579 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":578,"stamp":"2024-08-17T17:27:35.926936+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1723733989,"num_objects":26258,"num_object_clones":0,"num_object_copies":78774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5010,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26258,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":172186,"num_write_kb":6328384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16066,"num_bytes_recovered":776021382,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":118845,"ondisk_log_size":118845,"up":267,"acting":264,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":276,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":8218816,"kb_used_data":6012936,"kb_used_omap":33,"kb_used_meta":2204894,"kb_avail":647997248,"statfs":{"total":671965249536,"available":663549181952,"internally_reserved":0,"allocated":6157246464,"data_stored":6148514542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34068,"internal_metadata":2257812204},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,1,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-592642048,"num_objects":-9043,"num_object_clones":0,"num_object_copies":-27129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1222,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9043,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1222,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.679558"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":286,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.256472+0000","last_change":"2024-08-17T17:17:32.495652+0000","last_active":"2024-08-17T17:27:24.256472+0000","last_peered":"2024-08-17T17:27:24.256472+0000","last_clean":"2024-08-17T17:27:24.256472+0000","last_became_active":"2024-08-17T17:17:32.495218+0000","last_became_peered":"2024-08-17T17:17:32.495218+0000","last_unstale":"2024-08-17T17:27:24.256472+0000","last_undegraded":"2024-08-17T17:27:24.256472+0000","last_fullsized":"2024-08-17T17:27:24.256472+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:11:45.821282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":192,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-17T17:27:12.676358+0000","last_change":"2024-08-17T17:18:59.424887+0000","last_active":"2024-08-17T17:27:12.676358+0000","last_peered":"2024-08-17T17:27:12.676358+0000","last_clean":"2024-08-17T17:27:12.676358+0000","last_became_active":"2024-08-17T17:18:59.411539+0000","last_became_peered":"2024-08-17T17:18:59.411539+0000","last_unstale":"2024-08-17T17:27:12.676358+0000","last_undegraded":"2024-08-17T17:27:12.676358+0000","last_fullsized":"2024-08-17T17:27:12.676358+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:46:20.082241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":32,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-17T17:27:12.600846+0000","last_change":"2024-08-17T17:24:53.460742+0000","last_active":"2024-08-17T17:27:12.600846+0000","last_peered":"2024-08-17T17:27:12.600846+0000","last_clean":"2024-08-17T17:27:12.600846+0000","last_became_active":"2024-08-17T17:24:53.460365+0000","last_became_peered":"2024-08-17T17:24:53.460365+0000","last_unstale":"2024-08-17T17:27:12.600846+0000","last_undegraded":"2024-08-17T17:27:12.600846+0000","last_fullsized":"2024-08-17T17:27:12.600846+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:24:08.116968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":97,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.268179+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:27:24.268179+0000","last_peered":"2024-08-17T17:27:24.268179+0000","last_clean":"2024-08-17T17:27:24.268179+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:27:24.268179+0000","last_undegraded":"2024-08-17T17:27:24.268179+0000","last_fullsized":"2024-08-17T17:27:24.268179+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":97,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.262005+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:27:24.262005+0000","last_peered":"2024-08-17T17:27:24.262005+0000","last_clean":"2024-08-17T17:27:24.262005+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:27:24.262005+0000","last_undegraded":"2024-08-17T17:27:24.262005+0000","last_fullsized":"2024-08-17T17:27:24.262005+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":167,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-17T17:27:12.675784+0000","last_change":"2024-08-17T17:20:49.258206+0000","last_active":"2024-08-17T17:27:12.675784+0000","last_peered":"2024-08-17T17:27:12.675784+0000","last_clean":"2024-08-17T17:27:12.675784+0000","last_became_active":"2024-08-17T17:20:49.257761+0000","last_became_peered":"2024-08-17T17:20:49.257761+0000","last_unstale":"2024-08-17T17:27:12.675784+0000","last_undegraded":"2024-08-17T17:27:12.675784+0000","last_fullsized":"2024-08-17T17:27:12.675784+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:38:35.363748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":300,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.275018+0000","last_change":"2024-08-17T17:19:57.514773+0000","last_active":"2024-08-17T17:27:24.275018+0000","last_peered":"2024-08-17T17:27:24.275018+0000","last_clean":"2024-08-17T17:27:24.275018+0000","last_became_active":"2024-08-17T17:19:57.513839+0000","last_became_peered":"2024-08-17T17:19:57.513839+0000","last_unstale":"2024-08-17T17:27:24.275018+0000","last_undegraded":"2024-08-17T17:27:24.275018+0000","last_fullsized":"2024-08-17T17:27:24.275018+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:42:48.063169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":262,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-17T17:27:12.602357+0000","last_change":"2024-08-17T17:24:53.465875+0000","last_active":"2024-08-17T17:27:12.602357+0000","last_peered":"2024-08-17T17:27:12.602357+0000","last_clean":"2024-08-17T17:27:12.602357+0000","last_became_active":"2024-08-17T17:24:53.458129+0000","last_became_peered":"2024-08-17T17:24:53.458129+0000","last_unstale":"2024-08-17T17:27:12.602357+0000","last_undegraded":"2024-08-17T17:27:12.602357+0000","last_fullsized":"2024-08-17T17:27:12.602357+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:51:33.584869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"216'10629","reported_seq":15632,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:33.872649+0000","last_change":"2024-08-17T17:27:12.673270+0000","last_active":"2024-08-17T17:27:33.872649+0000","last_peered":"2024-08-17T17:27:33.872649+0000","last_clean":"2024-08-17T17:27:33.872649+0000","last_became_active":"2024-08-17T17:27:12.603527+0000","last_became_peered":"2024-08-17T17:27:12.603527+0000","last_unstale":"2024-08-17T17:27:33.872649+0000","last_undegraded":"2024-08-17T17:27:33.872649+0000","last_fullsized":"2024-08-17T17:27:33.872649+0000","mapping_epoch":214,"log_start":"191'1600","ondisk_log_start":"191'1600","created":178,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":9029,"log_dups_size":0,"ondisk_log_size":9029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:35:47.074384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106233856,"num_objects":1621,"num_object_clones":0,"num_object_copies":4863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10629,"num_write_kb":392000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2460,"num_bytes_recovered":33816576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":128,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-17T17:27:12.676685+0000","last_change":"2024-08-17T17:24:06.597382+0000","last_active":"2024-08-17T17:27:12.676685+0000","last_peered":"2024-08-17T17:27:12.676685+0000","last_clean":"2024-08-17T17:27:12.676685+0000","last_became_active":"2024-08-17T17:24:06.588881+0000","last_became_peered":"2024-08-17T17:24:06.588881+0000","last_unstale":"2024-08-17T17:27:12.676685+0000","last_undegraded":"2024-08-17T17:27:12.676685+0000","last_fullsized":"2024-08-17T17:27:12.676685+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:27.240751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":42,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-17T17:27:12.600371+0000","last_change":"2024-08-17T17:25:04.503780+0000","last_active":"2024-08-17T17:27:12.600371+0000","last_peered":"2024-08-17T17:27:12.600371+0000","last_clean":"2024-08-17T17:27:12.600371+0000","last_became_active":"2024-08-17T17:25:04.503503+0000","last_became_peered":"2024-08-17T17:25:04.503503+0000","last_unstale":"2024-08-17T17:27:12.600371+0000","last_undegraded":"2024-08-17T17:27:12.600371+0000","last_fullsized":"2024-08-17T17:27:12.600371+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:04:48.171216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"216'10624","reported_seq":10712,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:31.858945+0000","last_change":"2024-08-17T17:24:14.639361+0000","last_active":"2024-08-17T17:27:31.858945+0000","last_peered":"2024-08-17T17:27:31.858945+0000","last_clean":"2024-08-17T17:27:31.858945+0000","last_became_active":"2024-08-17T17:24:14.639074+0000","last_became_peered":"2024-08-17T17:24:14.639074+0000","last_unstale":"2024-08-17T17:27:31.858945+0000","last_undegraded":"2024-08-17T17:27:31.858945+0000","last_fullsized":"2024-08-17T17:27:31.858945+0000","mapping_epoch":178,"log_start":"200'4200","ondisk_log_start":"200'4200","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6424,"log_dups_size":0,"ondisk_log_size":6424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:15:05.377992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114032640,"num_objects":1740,"num_object_clones":0,"num_object_copies":5220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10624,"num_write_kb":395648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":114,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-17T17:27:12.677225+0000","last_change":"2024-08-17T17:24:06.599023+0000","last_active":"2024-08-17T17:27:12.677225+0000","last_peered":"2024-08-17T17:27:12.677225+0000","last_clean":"2024-08-17T17:27:12.677225+0000","last_became_active":"2024-08-17T17:24:06.588954+0000","last_became_peered":"2024-08-17T17:24:06.588954+0000","last_unstale":"2024-08-17T17:27:12.677225+0000","last_undegraded":"2024-08-17T17:27:12.677225+0000","last_fullsized":"2024-08-17T17:27:12.677225+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:20:51.445576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":241,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.259642+0000","last_change":"2024-08-17T17:20:49.271540+0000","last_active":"2024-08-17T17:27:24.259642+0000","last_peered":"2024-08-17T17:27:24.259642+0000","last_clean":"2024-08-17T17:27:24.259642+0000","last_became_active":"2024-08-17T17:20:49.271130+0000","last_became_peered":"2024-08-17T17:20:49.271130+0000","last_unstale":"2024-08-17T17:27:24.259642+0000","last_undegraded":"2024-08-17T17:27:24.259642+0000","last_fullsized":"2024-08-17T17:27:24.259642+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:05.386705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"216'10611","reported_seq":10657,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:34.005587+0000","last_change":"2024-08-17T17:24:14.644070+0000","last_active":"2024-08-17T17:27:34.005587+0000","last_peered":"2024-08-17T17:27:34.005587+0000","last_clean":"2024-08-17T17:27:34.005587+0000","last_became_active":"2024-08-17T17:24:14.643799+0000","last_became_peered":"2024-08-17T17:24:14.643799+0000","last_unstale":"2024-08-17T17:27:34.005587+0000","last_undegraded":"2024-08-17T17:27:34.005587+0000","last_fullsized":"2024-08-17T17:27:34.005587+0000","mapping_epoch":178,"log_start":"199'3600","ondisk_log_start":"199'3600","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":7011,"log_dups_size":0,"ondisk_log_size":7011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:25:38.445770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108199936,"num_objects":1651,"num_object_clones":0,"num_object_copies":4953,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10611,"num_write_kb":392384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":186,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-17T17:27:12.677869+0000","last_change":"2024-08-17T17:24:53.467526+0000","last_active":"2024-08-17T17:27:12.677869+0000","last_peered":"2024-08-17T17:27:12.677869+0000","last_clean":"2024-08-17T17:27:12.677869+0000","last_became_active":"2024-08-17T17:24:53.460631+0000","last_became_peered":"2024-08-17T17:24:53.460631+0000","last_unstale":"2024-08-17T17:27:12.677869+0000","last_undegraded":"2024-08-17T17:27:12.677869+0000","last_fullsized":"2024-08-17T17:27:12.677869+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:05:24.938831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":28,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-17T17:27:12.602756+0000","last_change":"2024-08-17T17:25:04.507176+0000","last_active":"2024-08-17T17:27:12.602756+0000","last_peered":"2024-08-17T17:27:12.602756+0000","last_clean":"2024-08-17T17:27:12.602756+0000","last_became_active":"2024-08-17T17:25:04.506788+0000","last_became_peered":"2024-08-17T17:25:04.506788+0000","last_unstale":"2024-08-17T17:27:12.602756+0000","last_undegraded":"2024-08-17T17:27:12.602756+0000","last_fullsized":"2024-08-17T17:27:12.602756+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:22:04.658216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"216'10457","reported_seq":10545,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:33.071329+0000","last_change":"2024-08-17T17:24:14.642637+0000","last_active":"2024-08-17T17:27:33.071329+0000","last_peered":"2024-08-17T17:27:33.071329+0000","last_clean":"2024-08-17T17:27:33.071329+0000","last_became_active":"2024-08-17T17:24:14.642347+0000","last_became_peered":"2024-08-17T17:24:14.642347+0000","last_unstale":"2024-08-17T17:27:33.071329+0000","last_undegraded":"2024-08-17T17:27:33.071329+0000","last_fullsized":"2024-08-17T17:27:33.071329+0000","mapping_epoch":178,"log_start":"192'2300","ondisk_log_start":"192'2300","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":8157,"log_dups_size":0,"ondisk_log_size":8157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:37:44.631335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103088128,"num_objects":1573,"num_object_clones":0,"num_object_copies":4719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10457,"num_write_kb":384960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":174,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-17T17:27:12.600636+0000","last_change":"2024-08-17T17:17:32.495428+0000","last_active":"2024-08-17T17:27:12.600636+0000","last_peered":"2024-08-17T17:27:12.600636+0000","last_clean":"2024-08-17T17:27:12.600636+0000","last_became_active":"2024-08-17T17:17:32.494978+0000","last_became_peered":"2024-08-17T17:17:32.494978+0000","last_unstale":"2024-08-17T17:27:12.600636+0000","last_undegraded":"2024-08-17T17:27:12.600636+0000","last_fullsized":"2024-08-17T17:27:12.600636+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:42:24.647812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":289,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.261618+0000","last_change":"2024-08-17T17:20:49.272126+0000","last_active":"2024-08-17T17:27:24.261618+0000","last_peered":"2024-08-17T17:27:24.261618+0000","last_clean":"2024-08-17T17:27:24.261618+0000","last_became_active":"2024-08-17T17:20:49.271185+0000","last_became_peered":"2024-08-17T17:20:49.271185+0000","last_unstale":"2024-08-17T17:27:24.261618+0000","last_undegraded":"2024-08-17T17:27:24.261618+0000","last_fullsized":"2024-08-17T17:27:24.261618+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:43:58.184939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"216'11204","reported_seq":16006,"reported_epoch":216,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-17T17:27:35.480289+0000","last_change":"2024-08-17T17:27:10.732177+0000","last_active":"2024-08-17T17:27:35.480289+0000","last_peered":"2024-08-17T17:27:35.480289+0000","last_clean":"2024-08-17T17:24:52.435411+0000","last_became_active":"2024-08-17T17:24:54.889545+0000","last_became_peered":"2024-08-17T17:24:54.889545+0000","last_unstale":"2024-08-17T17:27:35.480289+0000","last_undegraded":"2024-08-17T17:24:54.482592+0000","last_fullsized":"2024-08-17T17:24:54.459529+0000","mapping_epoch":194,"log_start":"202'4900","ondisk_log_start":"202'4900","created":178,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6304,"log_dups_size":0,"ondisk_log_size":6304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101580800,"num_objects":1550,"num_object_clones":0,"num_object_copies":4650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":668,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11204,"num_write_kb":408128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2349,"num_bytes_recovered":28180480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2],"avail_no_missing":["1","2","3","7"],"object_location_counts":[{"shards":"1,2","objects":882}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":317,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.257902+0000","last_change":"2024-08-17T17:24:06.595380+0000","last_active":"2024-08-17T17:27:24.257902+0000","last_peered":"2024-08-17T17:27:24.257902+0000","last_clean":"2024-08-17T17:27:24.257902+0000","last_became_active":"2024-08-17T17:24:06.584747+0000","last_became_peered":"2024-08-17T17:24:06.584747+0000","last_unstale":"2024-08-17T17:27:24.257902+0000","last_undegraded":"2024-08-17T17:27:24.257902+0000","last_fullsized":"2024-08-17T17:27:24.257902+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:02:18.473980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":262,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.256750+0000","last_change":"2024-08-17T17:20:49.280596+0000","last_active":"2024-08-17T17:27:24.256750+0000","last_peered":"2024-08-17T17:27:24.256750+0000","last_clean":"2024-08-17T17:27:24.256750+0000","last_became_active":"2024-08-17T17:20:49.279553+0000","last_became_peered":"2024-08-17T17:20:49.279553+0000","last_unstale":"2024-08-17T17:27:24.256750+0000","last_undegraded":"2024-08-17T17:27:24.256750+0000","last_fullsized":"2024-08-17T17:27:24.256750+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:13:24.124392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"216'10416","reported_seq":10504,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:31.806862+0000","last_change":"2024-08-17T17:24:14.638890+0000","last_active":"2024-08-17T17:27:31.806862+0000","last_peered":"2024-08-17T17:27:31.806862+0000","last_clean":"2024-08-17T17:27:31.806862+0000","last_became_active":"2024-08-17T17:24:14.638594+0000","last_became_peered":"2024-08-17T17:24:14.638594+0000","last_unstale":"2024-08-17T17:27:31.806862+0000","last_undegraded":"2024-08-17T17:27:31.806862+0000","last_fullsized":"2024-08-17T17:27:31.806862+0000","mapping_epoch":178,"log_start":"200'4000","ondisk_log_start":"200'4000","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6416,"log_dups_size":0,"ondisk_log_size":6416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:53:34.277077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117309440,"num_objects":1790,"num_object_clones":0,"num_object_copies":5370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10416,"num_write_kb":390592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":215,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.259073+0000","last_change":"2024-08-17T17:20:49.276769+0000","last_active":"2024-08-17T17:27:24.259073+0000","last_peered":"2024-08-17T17:27:24.259073+0000","last_clean":"2024-08-17T17:27:24.259073+0000","last_became_active":"2024-08-17T17:20:49.276498+0000","last_became_peered":"2024-08-17T17:20:49.276498+0000","last_unstale":"2024-08-17T17:27:24.259073+0000","last_undegraded":"2024-08-17T17:27:24.259073+0000","last_fullsized":"2024-08-17T17:27:24.259073+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:39:05.419440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":195,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-17T17:27:12.677713+0000","last_change":"2024-08-17T17:20:14.455738+0000","last_active":"2024-08-17T17:27:12.677713+0000","last_peered":"2024-08-17T17:27:12.677713+0000","last_clean":"2024-08-17T17:27:12.677713+0000","last_became_active":"2024-08-17T17:20:14.455469+0000","last_became_peered":"2024-08-17T17:20:14.455469+0000","last_unstale":"2024-08-17T17:27:12.677713+0000","last_undegraded":"2024-08-17T17:27:12.677713+0000","last_fullsized":"2024-08-17T17:27:12.677713+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:27:28.424336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"216'10674","reported_seq":18022,"reported_epoch":216,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:27:31.857405+0000","last_change":"2024-08-17T17:25:05.678234+0000","last_active":"2024-08-17T17:27:31.857405+0000","last_peered":"2024-08-17T17:27:31.857405+0000","last_clean":"2024-08-17T17:24:52.436463+0000","last_became_active":"2024-08-17T17:25:05.673238+0000","last_became_peered":"2024-08-17T17:25:05.673238+0000","last_unstale":"2024-08-17T17:27:31.857405+0000","last_undegraded":"2024-08-17T17:25:05.585769+0000","last_fullsized":"2024-08-17T17:25:05.499112+0000","mapping_epoch":198,"log_start":"200'4200","ondisk_log_start":"200'4200","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6474,"log_dups_size":0,"ondisk_log_size":6474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111017984,"num_objects":1694,"num_object_clones":0,"num_object_copies":5082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10674,"num_write_kb":395776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2519,"num_bytes_recovered":165085184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,5],"avail_no_missing":["3","1","5","7"],"object_location_counts":[{"shards":"3,5","objects":668}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":149,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-17T17:27:12.675704+0000","last_change":"2024-08-17T17:20:14.431140+0000","last_active":"2024-08-17T17:27:12.675704+0000","last_peered":"2024-08-17T17:27:12.675704+0000","last_clean":"2024-08-17T17:27:12.675704+0000","last_became_active":"2024-08-17T17:20:14.430839+0000","last_became_peered":"2024-08-17T17:20:14.430839+0000","last_unstale":"2024-08-17T17:27:12.675704+0000","last_undegraded":"2024-08-17T17:27:12.675704+0000","last_fullsized":"2024-08-17T17:27:12.675704+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:08:16.303616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":28,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-17T17:27:12.602520+0000","last_change":"2024-08-17T17:25:04.509325+0000","last_active":"2024-08-17T17:27:12.602520+0000","last_peered":"2024-08-17T17:27:12.602520+0000","last_clean":"2024-08-17T17:27:12.602520+0000","last_became_active":"2024-08-17T17:25:04.508364+0000","last_became_peered":"2024-08-17T17:25:04.508364+0000","last_unstale":"2024-08-17T17:27:12.602520+0000","last_undegraded":"2024-08-17T17:27:12.602520+0000","last_fullsized":"2024-08-17T17:27:12.602520+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:48:56.415763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"216'10509","reported_seq":15344,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:33.841520+0000","last_change":"2024-08-17T17:25:40.811756+0000","last_active":"2024-08-17T17:27:33.841520+0000","last_peered":"2024-08-17T17:27:33.841520+0000","last_clean":"2024-08-17T17:27:33.841520+0000","last_became_active":"2024-08-17T17:25:40.771755+0000","last_became_peered":"2024-08-17T17:25:40.771755+0000","last_unstale":"2024-08-17T17:27:33.841520+0000","last_undegraded":"2024-08-17T17:27:33.841520+0000","last_fullsized":"2024-08-17T17:27:33.841520+0000","mapping_epoch":204,"log_start":"186'500","ondisk_log_start":"186'500","created":178,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:13:03.918487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105381912,"num_objects":1609,"num_object_clones":0,"num_object_copies":4827,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1609,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10509,"num_write_kb":387713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2371,"num_bytes_recovered":155385856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":301,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.261446+0000","last_change":"2024-08-17T17:19:57.520745+0000","last_active":"2024-08-17T17:27:24.261446+0000","last_peered":"2024-08-17T17:27:24.261446+0000","last_clean":"2024-08-17T17:27:24.261446+0000","last_became_active":"2024-08-17T17:19:57.520152+0000","last_became_peered":"2024-08-17T17:19:57.520152+0000","last_unstale":"2024-08-17T17:27:24.261446+0000","last_undegraded":"2024-08-17T17:27:24.261446+0000","last_fullsized":"2024-08-17T17:27:24.261446+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:19:39.495644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":276,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.263034+0000","last_change":"2024-08-17T17:20:49.274372+0000","last_active":"2024-08-17T17:27:24.263034+0000","last_peered":"2024-08-17T17:27:24.263034+0000","last_clean":"2024-08-17T17:27:24.263034+0000","last_became_active":"2024-08-17T17:20:49.274023+0000","last_became_peered":"2024-08-17T17:20:49.274023+0000","last_unstale":"2024-08-17T17:27:24.263034+0000","last_undegraded":"2024-08-17T17:27:24.263034+0000","last_fullsized":"2024-08-17T17:27:24.263034+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:31:02.529766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"216'11132","reported_seq":11178,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:33.725632+0000","last_change":"2024-08-17T17:24:14.643398+0000","last_active":"2024-08-17T17:27:33.725632+0000","last_peered":"2024-08-17T17:27:33.725632+0000","last_clean":"2024-08-17T17:27:33.725632+0000","last_became_active":"2024-08-17T17:24:14.643102+0000","last_became_peered":"2024-08-17T17:24:14.643102+0000","last_unstale":"2024-08-17T17:27:33.725632+0000","last_undegraded":"2024-08-17T17:27:33.725632+0000","last_fullsized":"2024-08-17T17:27:33.725632+0000","mapping_epoch":178,"log_start":"187'1100","ondisk_log_start":"187'1100","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:09:30.323709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103284736,"num_objects":1576,"num_object_clones":0,"num_object_copies":4728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11132,"num_write_kb":406656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":543,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-17T17:27:12.675551+0000","last_change":"2024-08-17T17:24:09.599615+0000","last_active":"2024-08-17T17:27:12.675551+0000","last_peered":"2024-08-17T17:27:12.675551+0000","last_clean":"2024-08-17T17:27:12.675551+0000","last_became_active":"2024-08-17T17:24:09.599085+0000","last_became_peered":"2024-08-17T17:24:09.599085+0000","last_unstale":"2024-08-17T17:27:12.675551+0000","last_undegraded":"2024-08-17T17:27:12.675551+0000","last_fullsized":"2024-08-17T17:27:12.675551+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:04:01.141401+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:30:09.569579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":788,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-17T17:27:12.600611+0000","last_change":"2024-08-17T17:20:00.434774+0000","last_active":"2024-08-17T17:27:12.600611+0000","last_peered":"2024-08-17T17:27:12.600611+0000","last_clean":"2024-08-17T17:27:12.600611+0000","last_became_active":"2024-08-17T17:20:00.434343+0000","last_became_peered":"2024-08-17T17:20:00.434343+0000","last_unstale":"2024-08-17T17:27:12.600611+0000","last_undegraded":"2024-08-17T17:27:12.600611+0000","last_fullsized":"2024-08-17T17:27:12.600611+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:04.995152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":259,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.253411+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:27:24.253411+0000","last_peered":"2024-08-17T17:27:24.253411+0000","last_clean":"2024-08-17T17:27:24.253411+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:27:24.253411+0000","last_undegraded":"2024-08-17T17:27:24.253411+0000","last_fullsized":"2024-08-17T17:27:24.253411+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":28,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-17T17:27:12.602622+0000","last_change":"2024-08-17T17:25:04.579426+0000","last_active":"2024-08-17T17:27:12.602622+0000","last_peered":"2024-08-17T17:27:12.602622+0000","last_clean":"2024-08-17T17:27:12.602622+0000","last_became_active":"2024-08-17T17:25:04.579012+0000","last_became_peered":"2024-08-17T17:25:04.579012+0000","last_unstale":"2024-08-17T17:27:12.602622+0000","last_undegraded":"2024-08-17T17:27:12.602622+0000","last_fullsized":"2024-08-17T17:27:12.602622+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:56:32.241321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"216'10695","reported_seq":10783,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:35.474856+0000","last_change":"2024-08-17T17:24:14.641829+0000","last_active":"2024-08-17T17:27:35.474856+0000","last_peered":"2024-08-17T17:27:35.474856+0000","last_clean":"2024-08-17T17:27:35.474856+0000","last_became_active":"2024-08-17T17:24:14.641532+0000","last_became_peered":"2024-08-17T17:24:14.641532+0000","last_unstale":"2024-08-17T17:27:35.474856+0000","last_undegraded":"2024-08-17T17:27:35.474856+0000","last_fullsized":"2024-08-17T17:27:35.474856+0000","mapping_epoch":178,"log_start":"202'4400","ondisk_log_start":"202'4400","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6295,"log_dups_size":0,"ondisk_log_size":6295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:43:05.232036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101122048,"num_objects":1543,"num_object_clones":0,"num_object_copies":4629,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10695,"num_write_kb":391616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":806,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-17T17:27:12.675761+0000","last_change":"2024-08-17T17:16:26.958415+0000","last_active":"2024-08-17T17:27:12.675761+0000","last_peered":"2024-08-17T17:27:12.675761+0000","last_clean":"2024-08-17T17:27:12.675761+0000","last_became_active":"2024-08-17T17:16:26.957956+0000","last_became_peered":"2024-08-17T17:16:26.957956+0000","last_unstale":"2024-08-17T17:27:12.675761+0000","last_undegraded":"2024-08-17T17:27:12.675761+0000","last_fullsized":"2024-08-17T17:27:12.675761+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:22.478077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":288,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.261414+0000","last_change":"2024-08-17T17:18:59.415331+0000","last_active":"2024-08-17T17:27:24.261414+0000","last_peered":"2024-08-17T17:27:24.261414+0000","last_clean":"2024-08-17T17:27:24.261414+0000","last_became_active":"2024-08-17T17:18:59.415008+0000","last_became_peered":"2024-08-17T17:18:59.415008+0000","last_unstale":"2024-08-17T17:27:24.261414+0000","last_undegraded":"2024-08-17T17:27:24.261414+0000","last_fullsized":"2024-08-17T17:27:24.261414+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:22:38.076866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":97,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.262224+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:27:24.262224+0000","last_peered":"2024-08-17T17:27:24.262224+0000","last_clean":"2024-08-17T17:27:24.262224+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:27:24.262224+0000","last_undegraded":"2024-08-17T17:27:24.262224+0000","last_fullsized":"2024-08-17T17:27:24.262224+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"216'10558","reported_seq":16840,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:33.950750+0000","last_change":"2024-08-17T17:26:40.426009+0000","last_active":"2024-08-17T17:27:33.950750+0000","last_peered":"2024-08-17T17:27:33.950750+0000","last_clean":"2024-08-17T17:27:33.950750+0000","last_became_active":"2024-08-17T17:26:40.411144+0000","last_became_peered":"2024-08-17T17:26:40.411144+0000","last_unstale":"2024-08-17T17:27:33.950750+0000","last_undegraded":"2024-08-17T17:27:33.950750+0000","last_fullsized":"2024-08-17T17:27:33.950750+0000","mapping_epoch":211,"log_start":"199'3500","ondisk_log_start":"199'3500","created":178,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":7058,"log_dups_size":0,"ondisk_log_size":7058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:22.245859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106430464,"num_objects":1624,"num_object_clones":0,"num_object_copies":4872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10558,"num_write_kb":389824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3108,"num_bytes_recovered":178388992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":676,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.253566+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:27:24.253566+0000","last_peered":"2024-08-17T17:27:24.253566+0000","last_clean":"2024-08-17T17:27:24.253566+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:27:24.253566+0000","last_undegraded":"2024-08-17T17:27:24.253566+0000","last_fullsized":"2024-08-17T17:27:24.253566+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":284,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.259206+0000","last_change":"2024-08-17T17:19:57.515564+0000","last_active":"2024-08-17T17:27:24.259206+0000","last_peered":"2024-08-17T17:27:24.259206+0000","last_clean":"2024-08-17T17:27:24.259206+0000","last_became_active":"2024-08-17T17:19:57.515142+0000","last_became_peered":"2024-08-17T17:19:57.515142+0000","last_unstale":"2024-08-17T17:27:24.259206+0000","last_undegraded":"2024-08-17T17:27:24.259206+0000","last_fullsized":"2024-08-17T17:27:24.259206+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:59:33.506176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":315,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.262723+0000","last_change":"2024-08-17T17:24:06.587903+0000","last_active":"2024-08-17T17:27:24.262723+0000","last_peered":"2024-08-17T17:27:24.262723+0000","last_clean":"2024-08-17T17:27:24.262723+0000","last_became_active":"2024-08-17T17:24:06.587503+0000","last_became_peered":"2024-08-17T17:24:06.587503+0000","last_unstale":"2024-08-17T17:27:24.262723+0000","last_undegraded":"2024-08-17T17:27:24.262723+0000","last_fullsized":"2024-08-17T17:27:24.262723+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:05:58.591992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"216'10420","reported_seq":16723,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:31.857759+0000","last_change":"2024-08-17T17:26:08.233981+0000","last_active":"2024-08-17T17:27:31.857759+0000","last_peered":"2024-08-17T17:27:31.857759+0000","last_clean":"2024-08-17T17:27:31.857759+0000","last_became_active":"2024-08-17T17:26:08.225711+0000","last_became_peered":"2024-08-17T17:26:08.225711+0000","last_unstale":"2024-08-17T17:27:31.857759+0000","last_undegraded":"2024-08-17T17:27:31.857759+0000","last_fullsized":"2024-08-17T17:27:31.857759+0000","mapping_epoch":207,"log_start":"200'4000","ondisk_log_start":"200'4000","created":178,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6420,"log_dups_size":0,"ondisk_log_size":6420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:31:42.806869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114163712,"num_objects":1742,"num_object_clones":0,"num_object_copies":5226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10420,"num_write_kb":389184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3101,"num_bytes_recovered":203227136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":1126,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-17T17:27:12.600618+0000","last_change":"2024-08-17T17:20:00.433870+0000","last_active":"2024-08-17T17:27:12.600618+0000","last_peered":"2024-08-17T17:27:12.600618+0000","last_clean":"2024-08-17T17:27:12.600618+0000","last_became_active":"2024-08-17T17:20:00.433386+0000","last_became_peered":"2024-08-17T17:20:00.433386+0000","last_unstale":"2024-08-17T17:27:12.600618+0000","last_undegraded":"2024-08-17T17:27:12.600618+0000","last_fullsized":"2024-08-17T17:27:12.600618+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:01:11.848862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1392,"num_read_kb":2863,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":210,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.261458+0000","last_change":"2024-08-17T17:20:49.272407+0000","last_active":"2024-08-17T17:27:24.261458+0000","last_peered":"2024-08-17T17:27:24.261458+0000","last_clean":"2024-08-17T17:27:24.261458+0000","last_became_active":"2024-08-17T17:20:49.271749+0000","last_became_peered":"2024-08-17T17:20:49.271749+0000","last_unstale":"2024-08-17T17:27:24.261458+0000","last_undegraded":"2024-08-17T17:27:24.261458+0000","last_fullsized":"2024-08-17T17:27:24.261458+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:36:01.443457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":281,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.259423+0000","last_change":"2024-08-17T17:17:32.499715+0000","last_active":"2024-08-17T17:27:24.259423+0000","last_peered":"2024-08-17T17:27:24.259423+0000","last_clean":"2024-08-17T17:27:24.259423+0000","last_became_active":"2024-08-17T17:17:32.499412+0000","last_became_peered":"2024-08-17T17:17:32.499412+0000","last_unstale":"2024-08-17T17:27:24.259423+0000","last_undegraded":"2024-08-17T17:27:24.259423+0000","last_fullsized":"2024-08-17T17:27:24.259423+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:39:38.456970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"216'10680","reported_seq":10766,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:33.077811+0000","last_change":"2024-08-17T17:24:14.642608+0000","last_active":"2024-08-17T17:27:33.077811+0000","last_peered":"2024-08-17T17:27:33.077811+0000","last_clean":"2024-08-17T17:27:33.077811+0000","last_became_active":"2024-08-17T17:24:14.642292+0000","last_became_peered":"2024-08-17T17:24:14.642292+0000","last_unstale":"2024-08-17T17:27:33.077811+0000","last_undegraded":"2024-08-17T17:27:33.077811+0000","last_fullsized":"2024-08-17T17:27:33.077811+0000","mapping_epoch":178,"log_start":"192'2500","ondisk_log_start":"192'2500","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":8180,"log_dups_size":0,"ondisk_log_size":8180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:53:36.848614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107741184,"num_objects":1644,"num_object_clones":0,"num_object_copies":4932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10680,"num_write_kb":394368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":191,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-17T17:27:12.600930+0000","last_change":"2024-08-17T17:17:32.495728+0000","last_active":"2024-08-17T17:27:12.600930+0000","last_peered":"2024-08-17T17:27:12.600930+0000","last_clean":"2024-08-17T17:27:12.600930+0000","last_became_active":"2024-08-17T17:17:32.495357+0000","last_became_peered":"2024-08-17T17:17:32.495357+0000","last_unstale":"2024-08-17T17:27:12.600930+0000","last_undegraded":"2024-08-17T17:27:12.600930+0000","last_fullsized":"2024-08-17T17:27:12.600930+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:27:45.561567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":517,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.262732+0000","last_change":"2024-08-17T17:16:26.964621+0000","last_active":"2024-08-17T17:27:24.262732+0000","last_peered":"2024-08-17T17:27:24.262732+0000","last_clean":"2024-08-17T17:27:24.262732+0000","last_became_active":"2024-08-17T17:16:26.964058+0000","last_became_peered":"2024-08-17T17:16:26.964058+0000","last_unstale":"2024-08-17T17:27:24.262732+0000","last_undegraded":"2024-08-17T17:27:24.262732+0000","last_fullsized":"2024-08-17T17:27:24.262732+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:00:04.465644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":287,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.259160+0000","last_change":"2024-08-17T17:24:53.465550+0000","last_active":"2024-08-17T17:27:24.259160+0000","last_peered":"2024-08-17T17:27:24.259160+0000","last_clean":"2024-08-17T17:27:24.259160+0000","last_became_active":"2024-08-17T17:24:53.465228+0000","last_became_peered":"2024-08-17T17:24:53.465228+0000","last_unstale":"2024-08-17T17:27:24.259160+0000","last_undegraded":"2024-08-17T17:27:24.259160+0000","last_fullsized":"2024-08-17T17:27:24.259160+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:21:18.378359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"216'10823","reported_seq":10911,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:35.396921+0000","last_change":"2024-08-17T17:24:14.640584+0000","last_active":"2024-08-17T17:27:35.396921+0000","last_peered":"2024-08-17T17:27:35.396921+0000","last_clean":"2024-08-17T17:27:35.396921+0000","last_became_active":"2024-08-17T17:24:14.640168+0000","last_became_peered":"2024-08-17T17:24:14.640168+0000","last_unstale":"2024-08-17T17:27:35.396921+0000","last_undegraded":"2024-08-17T17:27:35.396921+0000","last_fullsized":"2024-08-17T17:27:35.396921+0000","mapping_epoch":178,"log_start":"202'4500","ondisk_log_start":"202'4500","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6323,"log_dups_size":0,"ondisk_log_size":6323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:38:36.176813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103612416,"num_objects":1581,"num_object_clones":0,"num_object_copies":4743,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10823,"num_write_kb":396928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":312,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.259551+0000","last_change":"2024-08-17T17:24:53.464307+0000","last_active":"2024-08-17T17:27:24.259551+0000","last_peered":"2024-08-17T17:27:24.259551+0000","last_clean":"2024-08-17T17:27:24.259551+0000","last_became_active":"2024-08-17T17:24:53.463996+0000","last_became_peered":"2024-08-17T17:24:53.463996+0000","last_unstale":"2024-08-17T17:27:24.259551+0000","last_undegraded":"2024-08-17T17:27:24.259551+0000","last_fullsized":"2024-08-17T17:27:24.259551+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:04:18.993725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":643,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-17T17:27:12.677948+0000","last_change":"2024-08-17T17:24:09.599822+0000","last_active":"2024-08-17T17:27:12.677948+0000","last_peered":"2024-08-17T17:27:12.677948+0000","last_clean":"2024-08-17T17:27:12.677948+0000","last_became_active":"2024-08-17T17:24:09.599189+0000","last_became_peered":"2024-08-17T17:24:09.599189+0000","last_unstale":"2024-08-17T17:27:12.677948+0000","last_undegraded":"2024-08-17T17:27:12.677948+0000","last_fullsized":"2024-08-17T17:27:12.677948+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:04.465604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":271,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.261543+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:27:24.261543+0000","last_peered":"2024-08-17T17:27:24.261543+0000","last_clean":"2024-08-17T17:27:24.261543+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:27:24.261543+0000","last_undegraded":"2024-08-17T17:27:24.261543+0000","last_fullsized":"2024-08-17T17:27:24.261543+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"216'10880","reported_seq":10968,"reported_epoch":216,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:27:31.818604+0000","last_change":"2024-08-17T17:25:05.605653+0000","last_active":"2024-08-17T17:27:31.818604+0000","last_peered":"2024-08-17T17:27:31.818604+0000","last_clean":"2024-08-17T17:25:03.971607+0000","last_became_active":"2024-08-17T17:25:05.601333+0000","last_became_peered":"2024-08-17T17:25:05.601333+0000","last_unstale":"2024-08-17T17:27:31.818604+0000","last_undegraded":"2024-08-17T17:25:05.522007+0000","last_fullsized":"2024-08-17T17:25:05.499355+0000","mapping_epoch":198,"log_start":"201'4400","ondisk_log_start":"201'4400","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6480,"log_dups_size":0,"ondisk_log_size":6480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109314048,"num_objects":1668,"num_object_clones":0,"num_object_copies":5004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3316,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1668,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10880,"num_write_kb":401536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2],"avail_no_missing":["3","2","7"],"object_location_counts":[{"shards":"2,3","objects":-1648}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":306,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.275135+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:27:24.275135+0000","last_peered":"2024-08-17T17:27:24.275135+0000","last_clean":"2024-08-17T17:27:24.275135+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:27:24.275135+0000","last_undegraded":"2024-08-17T17:27:24.275135+0000","last_fullsized":"2024-08-17T17:27:24.275135+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":826,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.264364+0000","last_change":"2024-08-17T17:23:19.633786+0000","last_active":"2024-08-17T17:27:24.264364+0000","last_peered":"2024-08-17T17:27:24.264364+0000","last_clean":"2024-08-17T17:27:24.264364+0000","last_became_active":"2024-08-17T17:23:19.633230+0000","last_became_peered":"2024-08-17T17:23:19.633230+0000","last_unstale":"2024-08-17T17:27:24.264364+0000","last_undegraded":"2024-08-17T17:27:24.264364+0000","last_fullsized":"2024-08-17T17:27:24.264364+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:30:50.377391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":303,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.256494+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:27:24.256494+0000","last_peered":"2024-08-17T17:27:24.256494+0000","last_clean":"2024-08-17T17:27:24.256494+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:27:24.256494+0000","last_undegraded":"2024-08-17T17:27:24.256494+0000","last_fullsized":"2024-08-17T17:27:24.256494+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"216'10752","reported_seq":10798,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:34.014028+0000","last_change":"2024-08-17T17:24:14.639560+0000","last_active":"2024-08-17T17:27:34.014028+0000","last_peered":"2024-08-17T17:27:34.014028+0000","last_clean":"2024-08-17T17:27:34.014028+0000","last_became_active":"2024-08-17T17:24:14.639093+0000","last_became_peered":"2024-08-17T17:24:14.639093+0000","last_unstale":"2024-08-17T17:27:34.014028+0000","last_undegraded":"2024-08-17T17:27:34.014028+0000","last_fullsized":"2024-08-17T17:27:34.014028+0000","mapping_epoch":178,"log_start":"199'3700","ondisk_log_start":"199'3700","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":7052,"log_dups_size":0,"ondisk_log_size":7052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:45:57.940550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106430464,"num_objects":1624,"num_object_clones":0,"num_object_copies":4872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10752,"num_write_kb":396032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":101,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.262096+0000","last_change":"2024-08-17T17:24:06.593660+0000","last_active":"2024-08-17T17:27:24.262096+0000","last_peered":"2024-08-17T17:27:24.262096+0000","last_clean":"2024-08-17T17:27:24.262096+0000","last_became_active":"2024-08-17T17:24:06.586215+0000","last_became_peered":"2024-08-17T17:24:06.586215+0000","last_unstale":"2024-08-17T17:27:24.262096+0000","last_undegraded":"2024-08-17T17:27:24.262096+0000","last_fullsized":"2024-08-17T17:27:24.262096+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:15:49.475395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":447,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-17T17:27:12.602663+0000","last_change":"2024-08-17T17:26:06.855307+0000","last_active":"2024-08-17T17:27:12.602663+0000","last_peered":"2024-08-17T17:27:12.602663+0000","last_clean":"2024-08-17T17:27:12.602663+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:27:12.602663+0000","last_undegraded":"2024-08-17T17:27:12.602663+0000","last_fullsized":"2024-08-17T17:27:12.602663+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":115,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-17T17:27:12.600760+0000","last_change":"2024-08-17T17:18:59.417183+0000","last_active":"2024-08-17T17:27:12.600760+0000","last_peered":"2024-08-17T17:27:12.600760+0000","last_clean":"2024-08-17T17:27:12.600760+0000","last_became_active":"2024-08-17T17:18:59.416843+0000","last_became_peered":"2024-08-17T17:18:59.416843+0000","last_unstale":"2024-08-17T17:27:12.600760+0000","last_undegraded":"2024-08-17T17:27:12.600760+0000","last_fullsized":"2024-08-17T17:27:12.600760+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:29:09.336335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":136,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-17T17:27:12.602497+0000","last_change":"2024-08-17T17:20:49.258048+0000","last_active":"2024-08-17T17:27:12.602497+0000","last_peered":"2024-08-17T17:27:12.602497+0000","last_clean":"2024-08-17T17:27:12.602497+0000","last_became_active":"2024-08-17T17:20:49.257672+0000","last_became_peered":"2024-08-17T17:20:49.257672+0000","last_unstale":"2024-08-17T17:27:12.602497+0000","last_undegraded":"2024-08-17T17:27:12.602497+0000","last_fullsized":"2024-08-17T17:27:12.602497+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:08:11.734316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":175,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-17T17:27:12.676345+0000","last_change":"2024-08-17T17:18:59.430969+0000","last_active":"2024-08-17T17:27:12.676345+0000","last_peered":"2024-08-17T17:27:12.676345+0000","last_clean":"2024-08-17T17:27:12.676345+0000","last_became_active":"2024-08-17T17:18:59.413475+0000","last_became_peered":"2024-08-17T17:18:59.413475+0000","last_unstale":"2024-08-17T17:27:12.676345+0000","last_undegraded":"2024-08-17T17:27:12.676345+0000","last_fullsized":"2024-08-17T17:27:12.676345+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:36:33.732645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":297,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.258987+0000","last_change":"2024-08-17T17:25:04.506413+0000","last_active":"2024-08-17T17:27:24.258987+0000","last_peered":"2024-08-17T17:27:24.258987+0000","last_clean":"2024-08-17T17:27:24.258987+0000","last_became_active":"2024-08-17T17:25:04.506104+0000","last_became_peered":"2024-08-17T17:25:04.506104+0000","last_unstale":"2024-08-17T17:27:24.258987+0000","last_undegraded":"2024-08-17T17:27:24.258987+0000","last_fullsized":"2024-08-17T17:27:24.258987+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:41:53.556223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":118,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-17T17:27:12.677646+0000","last_change":"2024-08-17T17:20:49.279458+0000","last_active":"2024-08-17T17:27:12.677646+0000","last_peered":"2024-08-17T17:27:12.677646+0000","last_clean":"2024-08-17T17:27:12.677646+0000","last_became_active":"2024-08-17T17:20:49.260790+0000","last_became_peered":"2024-08-17T17:20:49.260790+0000","last_unstale":"2024-08-17T17:27:12.677646+0000","last_undegraded":"2024-08-17T17:27:12.677646+0000","last_fullsized":"2024-08-17T17:27:12.677646+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:58.092945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":32,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-17T17:27:12.601519+0000","last_change":"2024-08-17T17:24:53.463713+0000","last_active":"2024-08-17T17:27:12.601519+0000","last_peered":"2024-08-17T17:27:12.601519+0000","last_clean":"2024-08-17T17:27:12.601519+0000","last_became_active":"2024-08-17T17:24:53.463419+0000","last_became_peered":"2024-08-17T17:24:53.463419+0000","last_unstale":"2024-08-17T17:27:12.601519+0000","last_undegraded":"2024-08-17T17:27:12.601519+0000","last_fullsized":"2024-08-17T17:27:12.601519+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:34:44.545175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":253,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.253324+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:27:24.253324+0000","last_peered":"2024-08-17T17:27:24.253324+0000","last_clean":"2024-08-17T17:27:24.253324+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:27:24.253324+0000","last_undegraded":"2024-08-17T17:27:24.253324+0000","last_fullsized":"2024-08-17T17:27:24.253324+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":280,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.262537+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:27:24.262537+0000","last_peered":"2024-08-17T17:27:24.262537+0000","last_clean":"2024-08-17T17:27:24.262537+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:27:24.262537+0000","last_undegraded":"2024-08-17T17:27:24.262537+0000","last_fullsized":"2024-08-17T17:27:24.262537+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":300,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.256370+0000","last_change":"2024-08-17T17:25:04.509422+0000","last_active":"2024-08-17T17:27:24.256370+0000","last_peered":"2024-08-17T17:27:24.256370+0000","last_clean":"2024-08-17T17:27:24.256370+0000","last_became_active":"2024-08-17T17:25:04.508742+0000","last_became_peered":"2024-08-17T17:25:04.508742+0000","last_unstale":"2024-08-17T17:27:24.256370+0000","last_undegraded":"2024-08-17T17:27:24.256370+0000","last_fullsized":"2024-08-17T17:27:24.256370+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:18:31.976907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":173,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-17T17:27:12.602580+0000","last_change":"2024-08-17T17:16:26.963696+0000","last_active":"2024-08-17T17:27:12.602580+0000","last_peered":"2024-08-17T17:27:12.602580+0000","last_clean":"2024-08-17T17:27:12.602580+0000","last_became_active":"2024-08-17T17:16:26.963280+0000","last_became_peered":"2024-08-17T17:16:26.963280+0000","last_unstale":"2024-08-17T17:27:12.602580+0000","last_undegraded":"2024-08-17T17:27:12.602580+0000","last_fullsized":"2024-08-17T17:27:12.602580+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:22:01.474840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":318,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.261134+0000","last_change":"2024-08-17T17:24:06.586886+0000","last_active":"2024-08-17T17:27:24.261134+0000","last_peered":"2024-08-17T17:27:24.261134+0000","last_clean":"2024-08-17T17:27:24.261134+0000","last_became_active":"2024-08-17T17:24:06.586516+0000","last_became_peered":"2024-08-17T17:24:06.586516+0000","last_unstale":"2024-08-17T17:27:24.261134+0000","last_undegraded":"2024-08-17T17:27:24.261134+0000","last_fullsized":"2024-08-17T17:27:24.261134+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:03:30.002726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":97,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.261961+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:27:24.261961+0000","last_peered":"2024-08-17T17:27:24.261961+0000","last_clean":"2024-08-17T17:27:24.261961+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:27:24.261961+0000","last_undegraded":"2024-08-17T17:27:24.261961+0000","last_fullsized":"2024-08-17T17:27:24.261961+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":277,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.259717+0000","last_change":"2024-08-17T17:18:59.412463+0000","last_active":"2024-08-17T17:27:24.259717+0000","last_peered":"2024-08-17T17:27:24.259717+0000","last_clean":"2024-08-17T17:27:24.259717+0000","last_became_active":"2024-08-17T17:18:59.412134+0000","last_became_peered":"2024-08-17T17:18:59.412134+0000","last_unstale":"2024-08-17T17:27:24.259717+0000","last_undegraded":"2024-08-17T17:27:24.259717+0000","last_fullsized":"2024-08-17T17:27:24.259717+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:44.463153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":321,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.256284+0000","last_change":"2024-08-17T17:25:04.510854+0000","last_active":"2024-08-17T17:27:24.256284+0000","last_peered":"2024-08-17T17:27:24.256284+0000","last_clean":"2024-08-17T17:27:24.256284+0000","last_became_active":"2024-08-17T17:25:04.510462+0000","last_became_peered":"2024-08-17T17:25:04.510462+0000","last_unstale":"2024-08-17T17:27:24.256284+0000","last_undegraded":"2024-08-17T17:27:24.256284+0000","last_fullsized":"2024-08-17T17:27:24.256284+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:55:54.762086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":218,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.259249+0000","last_change":"2024-08-17T17:24:06.591165+0000","last_active":"2024-08-17T17:27:24.259249+0000","last_peered":"2024-08-17T17:27:24.259249+0000","last_clean":"2024-08-17T17:27:24.259249+0000","last_became_active":"2024-08-17T17:24:06.590785+0000","last_became_peered":"2024-08-17T17:24:06.590785+0000","last_unstale":"2024-08-17T17:27:24.259249+0000","last_undegraded":"2024-08-17T17:27:24.259249+0000","last_fullsized":"2024-08-17T17:27:24.259249+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:54:05.775563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":290,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.262881+0000","last_change":"2024-08-17T17:24:32.300560+0000","last_active":"2024-08-17T17:27:24.262881+0000","last_peered":"2024-08-17T17:27:24.262881+0000","last_clean":"2024-08-17T17:27:24.262881+0000","last_became_active":"2024-08-17T17:24:32.300228+0000","last_became_peered":"2024-08-17T17:24:32.300228+0000","last_unstale":"2024-08-17T17:27:24.262881+0000","last_undegraded":"2024-08-17T17:27:24.262881+0000","last_fullsized":"2024-08-17T17:27:24.262881+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:18:05.475449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":97,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.261357+0000","last_change":"2024-08-17T17:24:06.586379+0000","last_active":"2024-08-17T17:27:24.261357+0000","last_peered":"2024-08-17T17:27:24.261357+0000","last_clean":"2024-08-17T17:27:24.261357+0000","last_became_active":"2024-08-17T17:24:06.586023+0000","last_became_peered":"2024-08-17T17:24:06.586023+0000","last_unstale":"2024-08-17T17:27:24.261357+0000","last_undegraded":"2024-08-17T17:27:24.261357+0000","last_fullsized":"2024-08-17T17:27:24.261357+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:45.701637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":307,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.257796+0000","last_change":"2024-08-17T17:21:50.442535+0000","last_active":"2024-08-17T17:27:24.257796+0000","last_peered":"2024-08-17T17:27:24.257796+0000","last_clean":"2024-08-17T17:27:24.257796+0000","last_became_active":"2024-08-17T17:21:50.441891+0000","last_became_peered":"2024-08-17T17:21:50.441891+0000","last_unstale":"2024-08-17T17:27:24.257796+0000","last_undegraded":"2024-08-17T17:27:24.257796+0000","last_fullsized":"2024-08-17T17:27:24.257796+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:46:24.025407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":228,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-17T17:27:12.600553+0000","last_change":"2024-08-17T17:25:04.508045+0000","last_active":"2024-08-17T17:27:12.600553+0000","last_peered":"2024-08-17T17:27:12.600553+0000","last_clean":"2024-08-17T17:27:12.600553+0000","last_became_active":"2024-08-17T17:25:04.505083+0000","last_became_peered":"2024-08-17T17:25:04.505083+0000","last_unstale":"2024-08-17T17:27:12.600553+0000","last_undegraded":"2024-08-17T17:27:12.600553+0000","last_fullsized":"2024-08-17T17:27:12.600553+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:23:42.993544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":293,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.256644+0000","last_change":"2024-08-17T17:18:59.418434+0000","last_active":"2024-08-17T17:27:24.256644+0000","last_peered":"2024-08-17T17:27:24.256644+0000","last_clean":"2024-08-17T17:27:24.256644+0000","last_became_active":"2024-08-17T17:18:59.418043+0000","last_became_peered":"2024-08-17T17:18:59.418043+0000","last_unstale":"2024-08-17T17:27:24.256644+0000","last_undegraded":"2024-08-17T17:27:24.256644+0000","last_fullsized":"2024-08-17T17:27:24.256644+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:25:51.371820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":201,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-17T17:27:12.600920+0000","last_change":"2024-08-17T17:20:49.291167+0000","last_active":"2024-08-17T17:27:12.600920+0000","last_peered":"2024-08-17T17:27:12.600920+0000","last_clean":"2024-08-17T17:27:12.600920+0000","last_became_active":"2024-08-17T17:20:49.263750+0000","last_became_peered":"2024-08-17T17:20:49.263750+0000","last_unstale":"2024-08-17T17:27:12.600920+0000","last_undegraded":"2024-08-17T17:27:12.600920+0000","last_fullsized":"2024-08-17T17:27:12.600920+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:02.863246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":307,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.253649+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:27:24.253649+0000","last_peered":"2024-08-17T17:27:24.253649+0000","last_clean":"2024-08-17T17:27:24.253649+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:27:24.253649+0000","last_undegraded":"2024-08-17T17:27:24.253649+0000","last_fullsized":"2024-08-17T17:27:24.253649+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":307,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.269354+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:27:24.269354+0000","last_peered":"2024-08-17T17:27:24.269354+0000","last_clean":"2024-08-17T17:27:24.269354+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:27:24.269354+0000","last_undegraded":"2024-08-17T17:27:24.269354+0000","last_fullsized":"2024-08-17T17:27:24.269354+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":28,"reported_epoch":215,"state":"active+clean","last_fresh":"2024-08-17T17:27:12.602161+0000","last_change":"2024-08-17T17:25:04.503834+0000","last_active":"2024-08-17T17:27:12.602161+0000","last_peered":"2024-08-17T17:27:12.602161+0000","last_clean":"2024-08-17T17:27:12.602161+0000","last_became_active":"2024-08-17T17:25:04.502712+0000","last_became_peered":"2024-08-17T17:25:04.502712+0000","last_unstale":"2024-08-17T17:27:12.602161+0000","last_undegraded":"2024-08-17T17:27:12.602161+0000","last_fullsized":"2024-08-17T17:27:12.602161+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:45.389103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":116,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-08-17T17:27:24.262446+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:27:24.262446+0000","last_peered":"2024-08-17T17:27:24.262446+0000","last_clean":"2024-08-17T17:27:24.262446+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:27:24.262446+0000","last_undegraded":"2024-08-17T17:27:24.262446+0000","last_fullsized":"2024-08-17T17:27:24.262446+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":1718943768,"num_objects":26230,"num_object_clones":0,"num_object_copies":78690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5010,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26230,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171064,"num_write_kb":6313345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15908,"num_bytes_recovered":764084224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6043086848,"data_stored":6043074632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117664,"ondisk_log_size":117664,"up":48,"acting":45,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":6304,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":16702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17971,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16789504,"data_stored":16777273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234249,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1529264,"kb_used_data":1219368,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":92215888,"statfs":{"total":95995035648,"available":94429069312,"internally_reserved":0,"allocated":1248632832,"data_stored":1247376422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495884,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1174632,"kb_used_data":864800,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":92570520,"statfs":{"total":95995035648,"available":94792212480,"internally_reserved":0,"allocated":885555200,"data_stored":884306682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757521,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":951520,"kb_used_data":641688,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":92793632,"statfs":{"total":95995035648,"available":95020679168,"internally_reserved":0,"allocated":657088512,"data_stored":655843665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019159,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435386,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1426136,"kb_used_data":1123216,"kb_used_omap":4,"kb_used_meta":302459,"kb_avail":92319016,"statfs":{"total":95995035648,"available":94534672384,"internally_reserved":0,"allocated":1150173184,"data_stored":1148936279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":309718026},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":104,"seq":446676598913,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1159016,"kb_used_data":822112,"kb_used_omap":7,"kb_used_meta":336696,"kb_avail":92586136,"statfs":{"total":95995035648,"available":94808203264,"internally_reserved":0,"allocated":841842688,"data_stored":840594665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":344776866},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762440,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1289824,"kb_used_data":919256,"kb_used_omap":7,"kb_used_meta":370424,"kb_avail":92455328,"statfs":{"total":95995035648,"available":94674255872,"internally_reserved":0,"allocated":941318144,"data_stored":940073038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7945,"internal_metadata":379314423},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":94,"seq":403726925966,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":688424,"kb_used_data":422496,"kb_used_omap":7,"kb_used_meta":265848,"kb_avail":93056728,"statfs":{"total":95995035648,"available":95290089472,"internally_reserved":0,"allocated":432635904,"data_stored":431383791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":272228673},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6241,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":428085248,"data_stored":428081176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":927010816,"data_stored":927006744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":833028096,"data_stored":833028096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1135149056,"data_stored":1135149056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":633147392,"data_stored":633143320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":852164608,"data_stored":852164608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1234501632,"data_stored":1234501632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:27:38.599 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:38.625 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:38.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:38.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:38.711 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:27:38.711 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:27:38.712 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:27:38.712 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:27:38.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:27:38.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:38.765 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:38.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:38.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:38.867 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:38.912 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:38.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:38.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:39.014 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:39.059 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:39.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:39.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:39.161 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:39.200 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:39.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:39.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:39.302 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:39.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:39.330 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:39.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:39.395 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-17T17:27:39.431 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:39.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:39.463 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:39.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:39.564 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:39.591 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:39.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:39.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:39.693 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:39.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:39 smithi053 ceph-mon[89963]: pgmap v579: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.5 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 801 op/s; 5010/75021 objects degraded (6.678%); 0 B/s, 65 objects/s recovering 2024-08-17T17:27:39.717 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:39.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:39.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:39.758 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:39 smithi003 ceph-mon[103344]: pgmap v579: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.5 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 801 op/s; 5010/75021 objects degraded (6.678%); 0 B/s, 65 objects/s recovering 2024-08-17T17:27:39.758 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:39 smithi003 ceph-mon[100615]: pgmap v579: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.5 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 801 op/s; 5010/75021 objects degraded (6.678%); 0 B/s, 65 objects/s recovering 2024-08-17T17:27:39.819 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:39.863 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:27:39.886 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:39.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:39.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:39.988 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:40.017 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:40.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:40.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:40.080 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:27:40.119 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:40.147 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:40.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:40.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:40.249 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:40.275 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:40.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:40.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:40.378 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:40.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:40.408 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:40.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:40.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:27:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:27:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:27:40.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:40 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 5010/75021 objects degraded (6.678%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:40.509 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:40.557 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:40.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:40.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:40.567 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:40 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 5010/75021 objects degraded (6.678%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:40.568 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:40 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 5010/75021 objects degraded (6.678%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:40.659 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:40.698 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:40.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:40.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:40.800 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:40.846 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:40.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:40.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:40.948 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:40.989 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:40.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:40.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:41.092 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:41.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:41.146 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:41.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:41.247 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:41.276 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:41.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:41.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:41.348 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-08-17T17:27:41.377 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:41.424 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:41.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:41.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:41.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:41 smithi003 ceph-mon[103344]: pgmap v580: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.4 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 861 op/s; 5010/69021 objects degraded (7.259%); 0 B/s, 65 objects/s recovering 2024-08-17T17:27:41.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:41 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/201827237' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:27:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:41 smithi003 ceph-mon[100615]: pgmap v580: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.4 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 861 op/s; 5010/69021 objects degraded (7.259%); 0 B/s, 65 objects/s recovering 2024-08-17T17:27:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:41 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/201827237' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:27:41.527 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:41.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:41.575 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:41.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:41.584 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:27:41.676 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:41.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:41.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:41.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:41 smithi053 ceph-mon[89963]: pgmap v580: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 86 active+clean; 1.4 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 861 op/s; 5010/69021 objects degraded (7.259%); 0 B/s, 65 objects/s recovering 2024-08-17T17:27:41.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:41 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/201827237' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:27:41.815 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:41.867 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:41.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:41.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:41.968 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:42.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:42.025 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:42.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:42.126 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:42.149 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:42.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:42.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:42.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:42.273 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:42.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:42.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:42.375 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:42.397 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:42.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:42.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:42.499 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:42.521 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:42.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:42.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:42.623 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:42.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:42.662 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:42.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:42.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:42 smithi053 ceph-mon[89963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:27:42.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:42 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/201827237' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:27:42.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:42 smithi053 ceph-mon[89963]: osdmap e217: 8 total, 8 up, 7 in 2024-08-17T17:27:42.763 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:42.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:42 smithi003 ceph-mon[103344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:27:42.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:42 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/201827237' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:27:42.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:42 smithi003 ceph-mon[103344]: osdmap e217: 8 total, 8 up, 7 in 2024-08-17T17:27:42.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:42 smithi003 ceph-mon[100615]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:27:42.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:42 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/201827237' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:27:42.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:42 smithi003 ceph-mon[100615]: osdmap e217: 8 total, 8 up, 7 in 2024-08-17T17:27:42.788 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:42.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:42.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:42.890 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:42.914 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:42.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:42.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:43.016 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:43.039 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:43.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:43.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:43.141 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:43.163 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:43.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:43.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:43.265 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:43.287 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:43.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:43.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:43.389 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:43.537 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:43.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:43.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:43.639 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:43.677 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:43.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:43.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:43.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:43 smithi053 ceph-mon[89963]: pgmap v582: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 86 active+clean; 1.3 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 808 op/s; 4508/63234 objects degraded (7.129%); 0 B/s, 92 objects/s recovering 2024-08-17T17:27:43.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:43 smithi053 ceph-mon[89963]: osdmap e218: 8 total, 8 up, 7 in 2024-08-17T17:27:43.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:43 smithi003 ceph-mon[100615]: pgmap v582: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 86 active+clean; 1.3 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 808 op/s; 4508/63234 objects degraded (7.129%); 0 B/s, 92 objects/s recovering 2024-08-17T17:27:43.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:43 smithi003 ceph-mon[100615]: osdmap e218: 8 total, 8 up, 7 in 2024-08-17T17:27:43.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:43 smithi003 ceph-mon[103344]: pgmap v582: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 86 active+clean; 1.3 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 808 op/s; 4508/63234 objects degraded (7.129%); 0 B/s, 92 objects/s recovering 2024-08-17T17:27:43.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:43 smithi003 ceph-mon[103344]: osdmap e218: 8 total, 8 up, 7 in 2024-08-17T17:27:43.779 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:43.807 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:43.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:43.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:43.909 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:43.938 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:43.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:43.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:44.040 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:44.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:44.083 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:44.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:44.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:44.227 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:44.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:44.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:44.329 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:44.366 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:44.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:44.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:44.468 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:44.496 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:44.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:44.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:44.598 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:44.624 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:44.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:44.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:44.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:44 smithi053 ceph-mon[89963]: osdmap e219: 8 total, 8 up, 7 in 2024-08-17T17:27:44.726 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:44.749 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:44.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:44.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:44.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:44 smithi003 ceph-mon[100615]: osdmap e219: 8 total, 8 up, 7 in 2024-08-17T17:27:44.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:44 smithi003 ceph-mon[103344]: osdmap e219: 8 total, 8 up, 7 in 2024-08-17T17:27:44.850 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:44.879 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:44.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:44.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:44.981 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:45.004 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:45.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:45.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:45.106 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:45.136 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:45.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:45.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:45.238 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:45.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:45.268 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:45.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:45.369 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:45.415 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:45.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:45.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:45.517 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:45.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:45.560 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:45.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:45.651 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:45 smithi003 ceph-mon[103344]: pgmap v585: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 86 active+clean; 1.2 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 817 op/s; 4508/60291 objects degraded (7.477%); 0 B/s, 83 objects/s recovering 2024-08-17T17:27:45.651 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:45 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 4508/60291 objects degraded (7.477%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:45.651 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:45 smithi003 ceph-mon[100615]: pgmap v585: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 86 active+clean; 1.2 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 817 op/s; 4508/60291 objects degraded (7.477%); 0 B/s, 83 objects/s recovering 2024-08-17T17:27:45.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:45 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 4508/60291 objects degraded (7.477%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:45.653 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:27:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:27:45] "GET /metrics HTTP/1.1" 200 36505 "" "Prometheus/2.51.0" 2024-08-17T17:27:45.661 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:45.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:45.687 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:45.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:45.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:45 smithi053 ceph-mon[89963]: pgmap v585: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 86 active+clean; 1.2 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 817 op/s; 4508/60291 objects degraded (7.477%); 0 B/s, 83 objects/s recovering 2024-08-17T17:27:45.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:45 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 4508/60291 objects degraded (7.477%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:45.788 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:45.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:45.827 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:45.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:45.928 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:45.956 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:45.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:45.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:46.058 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:46.086 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:46.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:46.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:46.188 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:46.228 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:46.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:46.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:46.330 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:46.374 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:46.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:46.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:46.476 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:46.502 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:46.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:46.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:46.603 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:46.630 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:46.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:46.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:46.732 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:46.754 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:46.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:46.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:46.856 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:46.894 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:46.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:46.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:46.996 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:47.040 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:47.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:47.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:47.141 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:47.178 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:47.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:47.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:47.279 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:47.306 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:47.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:47.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:47.408 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:47.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:47.436 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:47.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:47.537 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:47.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:47.567 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:47.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:47.658 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:47 smithi003 ceph-mon[103344]: pgmap v586: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.2 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 680 op/s; 4340/56763 objects degraded (7.646%); 0 B/s, 111 objects/s recovering 2024-08-17T17:27:47.659 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:47 smithi003 ceph-mon[100615]: pgmap v586: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.2 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 680 op/s; 4340/56763 objects degraded (7.646%); 0 B/s, 111 objects/s recovering 2024-08-17T17:27:47.668 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:47.710 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:47.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:47.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:47.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:47 smithi053 ceph-mon[89963]: pgmap v586: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.2 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 680 op/s; 4340/56763 objects degraded (7.646%); 0 B/s, 111 objects/s recovering 2024-08-17T17:27:47.812 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:47.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:47.856 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:47.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:47.873 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:27:47.874 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:27:47.910 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:27:47.910 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:27:47.944 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:27:47.944 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:27:47.957 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:47.985 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:27:47.985 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:27:48.001 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:48.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:48.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:48.018 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:27:48.018 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:27:48.056 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:27:48.056 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:27:48.095 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:27:48.095 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:27:48.102 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:48.129 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:27:48.129 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:27:48.144 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:48.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:48.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:48.246 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:48.273 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:48.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:48.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:48.375 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:48.402 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:48.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:48.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:48.504 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:48.550 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:48.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:48.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:48.651 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:48.697 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:48.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:48.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:48.700 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:48 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/335549121' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:27:48.700 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:48 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/335549121' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:27:48.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:48 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/335549121' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:27:48.799 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:48.829 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:48.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:48.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:48.931 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:48.963 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:48.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:48.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:49.064 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:49.090 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:49.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:49.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:49.191 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:49.231 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:49.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:49.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:49.333 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:49.385 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:49.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:49.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:49.487 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:49.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:49.528 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:49.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:49.628 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:49 smithi003 ceph-mon[103344]: pgmap v587: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.1 GiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 657 op/s; 3953/54012 objects degraded (7.319%); 0 B/s, 84 objects/s recovering 2024-08-17T17:27:49.629 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:49.630 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:49 smithi003 ceph-mon[100615]: pgmap v587: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.1 GiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 657 op/s; 3953/54012 objects degraded (7.319%); 0 B/s, 84 objects/s recovering 2024-08-17T17:27:49.630 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:27:49.630 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-08-17T17:27:49.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-08-17T17:27:49.631 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:27:49.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.631 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26082 5.1 5:8a74401c:::benchmark_data_smithi003_140793_object26081:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:27:49.631 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.358106+0000", 2024-08-17T17:27:49.631 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 507.26654256500001, 2024-08-17T17:27:49.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9241983769999997, 2024-08-17T17:27:49.631 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:27:49.631 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:27:49.631 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:27:49.631 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:27:49.632 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:27:49.632 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:27:49.632 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26082 2024-08-17T17:27:49.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.632 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:27:49.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:27:49.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.358106+0000", 2024-08-17T17:27:49.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:27:49.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.358106+0000", 2024-08-17T17:27:49.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:27:49.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.358107+0000", 2024-08-17T17:27:49.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4789999999999999e-06 2024-08-17T17:27:49.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:27:49.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.358131+0000", 2024-08-17T17:27:49.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4015e-05 2024-08-17T17:27:49.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:27:49.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.358143+0000", 2024-08-17T17:27:49.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.219e-05 2024-08-17T17:27:49.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:27:49.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.358151+0000", 2024-08-17T17:27:49.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2649999999999999e-06 2024-08-17T17:27:49.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.358226+0000", 2024-08-17T17:27:49.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5142999999999995e-05 2024-08-17T17:27:49.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.358238+0000", 2024-08-17T17:27:49.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2306e-05 2024-08-17T17:27:49.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.269985+0000", 2024-08-17T17:27:49.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.911747215 2024-08-17T17:27:49.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.636 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.636 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270001+0000", 2024-08-17T17:27:49.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5474000000000001e-05 2024-08-17T17:27:49.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.636 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.636 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.276689+0000", 2024-08-17T17:27:49.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0066882220000002 2024-08-17T17:27:49.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:27:49.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.277111+0000", 2024-08-17T17:27:49.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000421875 2024-08-17T17:27:49.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:27:49.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.277244+0000", 2024-08-17T17:27:49.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013283899999999999 2024-08-17T17:27:49.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:27:49.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.281296+0000", 2024-08-17T17:27:49.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040518719999999998 2024-08-17T17:27:49.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.281347+0000", 2024-08-17T17:27:49.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1427000000000001e-05 2024-08-17T17:27:49.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.281532+0000", 2024-08-17T17:27:49.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000185293 2024-08-17T17:27:49.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:27:49.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.281603+0000", 2024-08-17T17:27:49.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0518999999999993e-05 2024-08-17T17:27:49.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:27:49.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.282304+0000", 2024-08-17T17:27:49.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00070124300000000005 2024-08-17T17:27:49.639 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.639 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:27:49.639 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.639 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26098 5.1 5:89795322:::benchmark_data_smithi003_140793_object26097:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:27:49.639 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.365617+0000", 2024-08-17T17:27:49.640 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 507.25903145900003, 2024-08-17T17:27:49.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9216087019999999, 2024-08-17T17:27:49.640 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:27:49.640 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:27:49.640 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:27:49.640 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:27:49.640 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:27:49.640 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:27:49.640 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26098 2024-08-17T17:27:49.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.640 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:27:49.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:27:49.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365617+0000", 2024-08-17T17:27:49.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:27:49.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365617+0000", 2024-08-17T17:27:49.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:27:49.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365618+0000", 2024-08-17T17:27:49.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6249999999999999e-06 2024-08-17T17:27:49.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:27:49.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365655+0000", 2024-08-17T17:27:49.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6613999999999999e-05 2024-08-17T17:27:49.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:27:49.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365672+0000", 2024-08-17T17:27:49.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6844999999999999e-05 2024-08-17T17:27:49.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:27:49.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365679+0000", 2024-08-17T17:27:49.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4730000000000003e-06 2024-08-17T17:27:49.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366362+0000", 2024-08-17T17:27:49.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00068284800000000001 2024-08-17T17:27:49.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366374+0000", 2024-08-17T17:27:49.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1674e-05 2024-08-17T17:27:49.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270031+0000", 2024-08-17T17:27:49.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.90365675300000003 2024-08-17T17:27:49.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270039+0000", 2024-08-17T17:27:49.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1240000000000005e-06 2024-08-17T17:27:49.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.278729+0000", 2024-08-17T17:27:49.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0086905939999999 2024-08-17T17:27:49.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:27:49.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.278876+0000", 2024-08-17T17:27:49.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014623300000000001 2024-08-17T17:27:49.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:27:49.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.279011+0000", 2024-08-17T17:27:49.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013513899999999999 2024-08-17T17:27:49.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:27:49.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.281468+0000", 2024-08-17T17:27:49.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024572579999999999 2024-08-17T17:27:49.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.282380+0000", 2024-08-17T17:27:49.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00091197499999999998 2024-08-17T17:27:49.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.286511+0000", 2024-08-17T17:27:49.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0041307979999999998 2024-08-17T17:27:49.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:27:49.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.286586+0000", 2024-08-17T17:27:49.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5006000000000007e-05 2024-08-17T17:27:49.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:27:49.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.287226+0000", 2024-08-17T17:27:49.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00063974300000000002 2024-08-17T17:27:49.647 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.647 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:27:49.647 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.647 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26099 5.1 5:87830c36:::benchmark_data_smithi003_140793_object26098:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:27:49.647 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.365700+0000", 2024-08-17T17:27:49.647 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 507.25894804299998, 2024-08-17T17:27:49.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9222799620000002, 2024-08-17T17:27:49.648 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:27:49.648 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:27:49.648 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:27:49.648 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:27:49.648 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:27:49.648 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:27:49.648 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26099 2024-08-17T17:27:49.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.648 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:27:49.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:27:49.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365700+0000", 2024-08-17T17:27:49.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:27:49.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365700+0000", 2024-08-17T17:27:49.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:27:49.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365701+0000", 2024-08-17T17:27:49.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1799999999999995e-07 2024-08-17T17:27:49.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:27:49.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365722+0000", 2024-08-17T17:27:49.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0897999999999999e-05 2024-08-17T17:27:49.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:27:49.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365729+0000", 2024-08-17T17:27:49.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0539999999999997e-06 2024-08-17T17:27:49.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:27:49.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.365734+0000", 2024-08-17T17:27:49.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9910000000000002e-06 2024-08-17T17:27:49.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366399+0000", 2024-08-17T17:27:49.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00066523799999999996 2024-08-17T17:27:49.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366406+0000", 2024-08-17T17:27:49.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3889999999999999e-06 2024-08-17T17:27:49.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270065+0000", 2024-08-17T17:27:49.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.90365837800000004 2024-08-17T17:27:49.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270072+0000", 2024-08-17T17:27:49.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5390000000000002e-06 2024-08-17T17:27:49.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.280026+0000", 2024-08-17T17:27:49.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.009954215 2024-08-17T17:27:49.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:27:49.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.280176+0000", 2024-08-17T17:27:49.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014965900000000001 2024-08-17T17:27:49.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:27:49.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.280355+0000", 2024-08-17T17:27:49.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000178586 2024-08-17T17:27:49.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.282543+0000", 2024-08-17T17:27:49.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0021886539999999999 2024-08-17T17:27:49.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:27:49.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.282560+0000", 2024-08-17T17:27:49.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6864e-05 2024-08-17T17:27:49.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.287385+0000", 2024-08-17T17:27:49.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0048250080000000004 2024-08-17T17:27:49.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:27:49.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.287456+0000", 2024-08-17T17:27:49.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0271000000000005e-05 2024-08-17T17:27:49.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:27:49.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.287980+0000", 2024-08-17T17:27:49.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00052470000000000001 2024-08-17T17:27:49.655 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.655 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:27:49.655 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.655 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26102 5.1 5:806d080d:::benchmark_data_smithi003_140793_object26101:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:27:49.655 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.366504+0000", 2024-08-17T17:27:49.655 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 507.25814425499999, 2024-08-17T17:27:49.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9223289210000001, 2024-08-17T17:27:49.655 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:27:49.655 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:27:49.656 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:27:49.656 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:27:49.656 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:27:49.656 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:27:49.656 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26102 2024-08-17T17:27:49.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.656 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:27:49.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:27:49.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366504+0000", 2024-08-17T17:27:49.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:27:49.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366504+0000", 2024-08-17T17:27:49.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:27:49.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366506+0000", 2024-08-17T17:27:49.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5149999999999999e-06 2024-08-17T17:27:49.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:27:49.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366531+0000", 2024-08-17T17:27:49.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5834e-05 2024-08-17T17:27:49.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:27:49.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366547+0000", 2024-08-17T17:27:49.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5235999999999999e-05 2024-08-17T17:27:49.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:27:49.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366554+0000", 2024-08-17T17:27:49.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2699999999999999e-06 2024-08-17T17:27:49.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366661+0000", 2024-08-17T17:27:49.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000107457 2024-08-17T17:27:49.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.366671+0000", 2024-08-17T17:27:49.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3559999999999999e-06 2024-08-17T17:27:49.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270100+0000", 2024-08-17T17:27:49.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.90342893899999999 2024-08-17T17:27:49.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.661 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270108+0000", 2024-08-17T17:27:49.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6030000000000005e-06 2024-08-17T17:27:49.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.661 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.282611+0000", 2024-08-17T17:27:49.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.012502784 2024-08-17T17:27:49.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:27:49.661 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.282745+0000", 2024-08-17T17:27:49.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000133504 2024-08-17T17:27:49.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:27:49.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.282979+0000", 2024-08-17T17:27:49.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00023466900000000001 2024-08-17T17:27:49.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:27:49.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.287245+0000", 2024-08-17T17:27:49.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042655929999999998 2024-08-17T17:27:49.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.287308+0000", 2024-08-17T17:27:49.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2869999999999999e-05 2024-08-17T17:27:49.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.288280+0000", 2024-08-17T17:27:49.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00097271 2024-08-17T17:27:49.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:27:49.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.288377+0000", 2024-08-17T17:27:49.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6137000000000005e-05 2024-08-17T17:27:49.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:27:49.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.288833+0000", 2024-08-17T17:27:49.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00045644400000000002 2024-08-17T17:27:49.664 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.664 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:27:49.664 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.664 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26120 5.1 5:8459086c:::benchmark_data_smithi003_140793_object26119:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:27:49.664 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.375347+0000", 2024-08-17T17:27:49.664 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 507.24930125999998, 2024-08-17T17:27:49.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9140657679999999, 2024-08-17T17:27:49.664 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:27:49.665 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:27:49.665 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:27:49.665 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:27:49.665 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:27:49.665 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:27:49.665 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26120 2024-08-17T17:27:49.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.665 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:27:49.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:27:49.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375347+0000", 2024-08-17T17:27:49.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:27:49.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375347+0000", 2024-08-17T17:27:49.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:27:49.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375348+0000", 2024-08-17T17:27:49.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1939999999999999e-06 2024-08-17T17:27:49.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:27:49.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375371+0000", 2024-08-17T17:27:49.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2997e-05 2024-08-17T17:27:49.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:27:49.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375445+0000", 2024-08-17T17:27:49.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3657000000000003e-05 2024-08-17T17:27:49.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:27:49.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375456+0000", 2024-08-17T17:27:49.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0941000000000001e-05 2024-08-17T17:27:49.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375518+0000", 2024-08-17T17:27:49.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2533999999999995e-05 2024-08-17T17:27:49.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375531+0000", 2024-08-17T17:27:49.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.238e-05 2024-08-17T17:27:49.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270132+0000", 2024-08-17T17:27:49.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89460134800000002 2024-08-17T17:27:49.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270140+0000", 2024-08-17T17:27:49.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0299999999999994e-06 2024-08-17T17:27:49.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.284149+0000", 2024-08-17T17:27:49.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0140088189999998 2024-08-17T17:27:49.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:27:49.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.284323+0000", 2024-08-17T17:27:49.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017453600000000001 2024-08-17T17:27:49.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:27:49.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.284507+0000", 2024-08-17T17:27:49.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000183692 2024-08-17T17:27:49.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:27:49.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.288124+0000", 2024-08-17T17:27:49.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0036170659999999999 2024-08-17T17:27:49.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.288193+0000", 2024-08-17T17:27:49.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9238000000000004e-05 2024-08-17T17:27:49.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.288892+0000", 2024-08-17T17:27:49.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00069895500000000002 2024-08-17T17:27:49.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:27:49.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.288930+0000", 2024-08-17T17:27:49.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7125000000000001e-05 2024-08-17T17:27:49.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:27:49.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.289413+0000", 2024-08-17T17:27:49.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000483256 2024-08-17T17:27:49.671 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.671 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:27:49.672 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.672 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26121 5.1 5:8d0269c6:::benchmark_data_smithi003_140793_object26120:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:27:49.672 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.375480+0000", 2024-08-17T17:27:49.672 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 507.24916806499999, 2024-08-17T17:27:49.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.921705099, 2024-08-17T17:27:49.672 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:27:49.672 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:27:49.672 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:27:49.672 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:27:49.672 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:27:49.672 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:27:49.672 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26121 2024-08-17T17:27:49.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.673 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:27:49.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:27:49.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375480+0000", 2024-08-17T17:27:49.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:27:49.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375480+0000", 2024-08-17T17:27:49.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:27:49.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375481+0000", 2024-08-17T17:27:49.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.44e-07 2024-08-17T17:27:49.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:27:49.674 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375502+0000", 2024-08-17T17:27:49.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.086e-05 2024-08-17T17:27:49.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:27:49.674 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375511+0000", 2024-08-17T17:27:49.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5429999999999999e-06 2024-08-17T17:27:49.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:27:49.674 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375518+0000", 2024-08-17T17:27:49.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2949999999999999e-06 2024-08-17T17:27:49.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375557+0000", 2024-08-17T17:27:49.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9255999999999998e-05 2024-08-17T17:27:49.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.375565+0000", 2024-08-17T17:27:49.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1289999999999996e-06 2024-08-17T17:27:49.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270164+0000", 2024-08-17T17:27:49.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89459907800000005 2024-08-17T17:27:49.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270172+0000", 2024-08-17T17:27:49.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8869999999999992e-06 2024-08-17T17:27:49.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.289461+0000", 2024-08-17T17:27:49.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0192893380000001 2024-08-17T17:27:49.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.677 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:27:49.677 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.289616+0000", 2024-08-17T17:27:49.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015440199999999999 2024-08-17T17:27:49.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.677 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:27:49.677 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.289854+0000", 2024-08-17T17:27:49.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000238271 2024-08-17T17:27:49.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.677 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.677 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.296410+0000", 2024-08-17T17:27:49.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006555735 2024-08-17T17:27:49.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:27:49.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.296424+0000", 2024-08-17T17:27:49.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4679e-05 2024-08-17T17:27:49.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.296471+0000", 2024-08-17T17:27:49.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6600000000000001e-05 2024-08-17T17:27:49.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:27:49.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.296537+0000", 2024-08-17T17:27:49.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5733000000000002e-05 2024-08-17T17:27:49.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:27:49.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.297185+0000", 2024-08-17T17:27:49.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00064864899999999995 2024-08-17T17:27:49.679 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.679 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:27:49.679 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.679 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26129 5.1 5:845981d1:::benchmark_data_smithi003_140793_object26128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:27:49.679 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.378910+0000", 2024-08-17T17:27:49.679 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 507.24573811099998, 2024-08-17T17:27:49.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9327440490000001, 2024-08-17T17:27:49.680 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:27:49.680 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:27:49.680 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:27:49.680 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:27:49.680 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:27:49.680 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:27:49.680 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26129 2024-08-17T17:27:49.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.680 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:27:49.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:27:49.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.378910+0000", 2024-08-17T17:27:49.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:27:49.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.378910+0000", 2024-08-17T17:27:49.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:27:49.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.378912+0000", 2024-08-17T17:27:49.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.657e-06 2024-08-17T17:27:49.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:27:49.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.378935+0000", 2024-08-17T17:27:49.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2795000000000001e-05 2024-08-17T17:27:49.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:27:49.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.378956+0000", 2024-08-17T17:27:49.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1744000000000001e-05 2024-08-17T17:27:49.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:27:49.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.378965+0000", 2024-08-17T17:27:49.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2930000000000001e-06 2024-08-17T17:27:49.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.379129+0000", 2024-08-17T17:27:49.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016463799999999999 2024-08-17T17:27:49.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.379143+0000", 2024-08-17T17:27:49.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.341e-05 2024-08-17T17:27:49.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270196+0000", 2024-08-17T17:27:49.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89105351399999999 2024-08-17T17:27:49.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270204+0000", 2024-08-17T17:27:49.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8159999999999994e-06 2024-08-17T17:27:49.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.294939+0000", 2024-08-17T17:27:49.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.024735352 2024-08-17T17:27:49.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:27:49.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.295108+0000", 2024-08-17T17:27:49.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016822500000000001 2024-08-17T17:27:49.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:27:49.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.295294+0000", 2024-08-17T17:27:49.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000186107 2024-08-17T17:27:49.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.301319+0000", 2024-08-17T17:27:49.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0060248339999999997 2024-08-17T17:27:49.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:27:49.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.304206+0000", 2024-08-17T17:27:49.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028874009999999999 2024-08-17T17:27:49.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.310650+0000", 2024-08-17T17:27:49.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0064438689999999996 2024-08-17T17:27:49.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:27:49.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.310709+0000", 2024-08-17T17:27:49.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9494000000000003e-05 2024-08-17T17:27:49.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:27:49.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.311654+0000", 2024-08-17T17:27:49.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00094490000000000004 2024-08-17T17:27:49.687 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.687 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:27:49.687 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:49.688 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:49.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:49.689 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26134 5.1 5:8a41a77b:::benchmark_data_smithi003_140793_object26133:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:27:49.689 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.380465+0000", 2024-08-17T17:27:49.689 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 507.24418311099998, 2024-08-17T17:27:49.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9324733580000002, 2024-08-17T17:27:49.689 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:27:49.689 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:27:49.689 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:27:49.689 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:27:49.689 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:27:49.689 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:27:49.689 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26134 2024-08-17T17:27:49.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.690 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:27:49.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:27:49.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.380465+0000", 2024-08-17T17:27:49.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:27:49.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.380465+0000", 2024-08-17T17:27:49.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.691 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:27:49.691 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.380466+0000", 2024-08-17T17:27:49.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.28e-06 2024-08-17T17:27:49.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.691 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:27:49.691 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.380490+0000", 2024-08-17T17:27:49.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3842999999999999e-05 2024-08-17T17:27:49.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.691 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:27:49.691 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.380506+0000", 2024-08-17T17:27:49.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6065e-05 2024-08-17T17:27:49.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:27:49.692 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.380514+0000", 2024-08-17T17:27:49.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8529999999999992e-06 2024-08-17T17:27:49.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.692 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.380591+0000", 2024-08-17T17:27:49.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6271000000000002e-05 2024-08-17T17:27:49.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.692 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.380603+0000", 2024-08-17T17:27:49.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2867999999999999e-05 2024-08-17T17:27:49.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270227+0000", 2024-08-17T17:27:49.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.88962385499999996 2024-08-17T17:27:49.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270235+0000", 2024-08-17T17:27:49.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6120000000000001e-06 2024-08-17T17:27:49.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.297325+0000", 2024-08-17T17:27:49.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0270902529999999 2024-08-17T17:27:49.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:27:49.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.297474+0000", 2024-08-17T17:27:49.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014895899999999999 2024-08-17T17:27:49.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:27:49.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.297667+0000", 2024-08-17T17:27:49.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019309500000000001 2024-08-17T17:27:49.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.304283+0000", 2024-08-17T17:27:49.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066158759999999997 2024-08-17T17:27:49.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:27:49.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.310600+0000", 2024-08-17T17:27:49.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063167420000000002 2024-08-17T17:27:49.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.312211+0000", 2024-08-17T17:27:49.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016113029999999999 2024-08-17T17:27:49.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:27:49.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.312376+0000", 2024-08-17T17:27:49.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000164804 2024-08-17T17:27:49.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:27:49.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.312939+0000", 2024-08-17T17:27:49.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00056267899999999996 2024-08-17T17:27:49.696 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.696 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:27:49.696 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.696 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26136 5.1 5:81387a84:::benchmark_data_smithi003_140793_object26135:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:27:49.696 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.382929+0000", 2024-08-17T17:27:49.696 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 507.24171918799999, 2024-08-17T17:27:49.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.930695842, 2024-08-17T17:27:49.696 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:27:49.696 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:27:49.697 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:27:49.697 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:27:49.697 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:27:49.697 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:27:49.697 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26136 2024-08-17T17:27:49.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.697 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:27:49.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:27:49.697 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.382929+0000", 2024-08-17T17:27:49.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:27:49.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.382929+0000", 2024-08-17T17:27:49.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:27:49.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.382931+0000", 2024-08-17T17:27:49.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5120000000000001e-06 2024-08-17T17:27:49.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:27:49.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.382982+0000", 2024-08-17T17:27:49.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1314999999999999e-05 2024-08-17T17:27:49.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:27:49.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.382998+0000", 2024-08-17T17:27:49.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6235999999999998e-05 2024-08-17T17:27:49.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:27:49.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.383006+0000", 2024-08-17T17:27:49.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4760000000000001e-06 2024-08-17T17:27:49.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.383064+0000", 2024-08-17T17:27:49.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8255999999999999e-05 2024-08-17T17:27:49.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.700 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.700 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.383073+0000", 2024-08-17T17:27:49.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5650000000000007e-06 2024-08-17T17:27:49.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.700 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.700 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270259+0000", 2024-08-17T17:27:49.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.88718503699999995 2024-08-17T17:27:49.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.700 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.701 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270266+0000", 2024-08-17T17:27:49.701 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.892e-06 2024-08-17T17:27:49.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.701 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.701 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.298758+0000", 2024-08-17T17:27:49.701 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0284916499999999 2024-08-17T17:27:49.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.701 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:27:49.701 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.298905+0000", 2024-08-17T17:27:49.701 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014658 2024-08-17T17:27:49.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:27:49.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.299067+0000", 2024-08-17T17:27:49.702 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016263700000000001 2024-08-17T17:27:49.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.304334+0000", 2024-08-17T17:27:49.702 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0052671059999999997 2024-08-17T17:27:49.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:27:49.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.310606+0000", 2024-08-17T17:27:49.702 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0062720830000000003 2024-08-17T17:27:49.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.703 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.703 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.313094+0000", 2024-08-17T17:27:49.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002487763 2024-08-17T17:27:49.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.703 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:27:49.703 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.313146+0000", 2024-08-17T17:27:49.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2079000000000001e-05 2024-08-17T17:27:49.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.704 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:27:49.704 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.313625+0000", 2024-08-17T17:27:49.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00047865499999999998 2024-08-17T17:27:49.704 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.704 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:27:49.704 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.704 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26145 5.1 5:896bb16f:::benchmark_data_smithi003_140793_object26144:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:27:49.704 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.386726+0000", 2024-08-17T17:27:49.705 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 507.237922201, 2024-08-17T17:27:49.705 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9275205729999998, 2024-08-17T17:27:49.705 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:27:49.705 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:27:49.705 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:27:49.705 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:27:49.705 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:27:49.705 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:27:49.705 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26145 2024-08-17T17:27:49.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.705 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:27:49.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.705 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:27:49.705 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.386726+0000", 2024-08-17T17:27:49.706 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.706 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:27:49.706 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.386726+0000", 2024-08-17T17:27:49.706 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.706 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:27:49.706 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.386727+0000", 2024-08-17T17:27:49.706 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.291e-06 2024-08-17T17:27:49.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:27:49.707 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.386775+0000", 2024-08-17T17:27:49.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7855999999999997e-05 2024-08-17T17:27:49.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:27:49.707 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.386791+0000", 2024-08-17T17:27:49.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6044000000000001e-05 2024-08-17T17:27:49.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:27:49.707 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.386799+0000", 2024-08-17T17:27:49.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7510000000000008e-06 2024-08-17T17:27:49.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.708 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.708 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.708 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.386873+0000", 2024-08-17T17:27:49.708 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4208000000000005e-05 2024-08-17T17:27:49.708 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.708 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.708 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.708 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.386887+0000", 2024-08-17T17:27:49.708 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3699e-05 2024-08-17T17:27:49.708 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.708 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.708 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.708 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270290+0000", 2024-08-17T17:27:49.709 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.88340287500000003 2024-08-17T17:27:49.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.709 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.709 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.709 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270298+0000", 2024-08-17T17:27:49.709 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0399999999999993e-06 2024-08-17T17:27:49.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.709 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.709 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.709 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.299986+0000", 2024-08-17T17:27:49.709 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0296885850000002 2024-08-17T17:27:49.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.709 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.709 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:27:49.710 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.300132+0000", 2024-08-17T17:27:49.710 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014511 2024-08-17T17:27:49.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.710 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.710 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:27:49.710 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.300298+0000", 2024-08-17T17:27:49.710 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016594799999999999 2024-08-17T17:27:49.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.710 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.710 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.710 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.304384+0000", 2024-08-17T17:27:49.710 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040865720000000001 2024-08-17T17:27:49.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.711 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.711 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:27:49.711 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.311674+0000", 2024-08-17T17:27:49.711 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072902210000000004 2024-08-17T17:27:49.711 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.711 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.711 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.711 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.313682+0000", 2024-08-17T17:27:49.711 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00200746 2024-08-17T17:27:49.711 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.711 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.711 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:27:49.711 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.313742+0000", 2024-08-17T17:27:49.711 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0265e-05 2024-08-17T17:27:49.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.712 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:27:49.712 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.314247+0000", 2024-08-17T17:27:49.712 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00050464800000000001 2024-08-17T17:27:49.712 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.712 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:27:49.712 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.712 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26147 5.1 5:85f2e61e:::benchmark_data_smithi003_140793_object26146:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:27:49.712 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.387915+0000", 2024-08-17T17:27:49.712 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 507.23673371400002, 2024-08-17T17:27:49.712 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9269894829999998, 2024-08-17T17:27:49.713 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:27:49.713 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:27:49.713 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:27:49.713 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:27:49.713 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:27:49.713 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:27:49.713 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26147 2024-08-17T17:27:49.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.713 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:27:49.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.713 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:27:49.713 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.387915+0000", 2024-08-17T17:27:49.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.714 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.714 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:27:49.714 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.387915+0000", 2024-08-17T17:27:49.714 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.714 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.714 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:27:49.714 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.387915+0000", 2024-08-17T17:27:49.714 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6899999999999996e-07 2024-08-17T17:27:49.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.714 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.714 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:27:49.714 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.387953+0000", 2024-08-17T17:27:49.715 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7447000000000002e-05 2024-08-17T17:27:49.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.715 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:27:49.715 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.387963+0000", 2024-08-17T17:27:49.715 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0302999999999999e-05 2024-08-17T17:27:49.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.715 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:27:49.715 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.387969+0000", 2024-08-17T17:27:49.715 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1090000000000003e-06 2024-08-17T17:27:49.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.715 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.716 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.388038+0000", 2024-08-17T17:27:49.716 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9042999999999996e-05 2024-08-17T17:27:49.716 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.716 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.716 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.716 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.388051+0000", 2024-08-17T17:27:49.716 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2498e-05 2024-08-17T17:27:49.716 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.716 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.716 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.716 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270321+0000", 2024-08-17T17:27:49.716 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.88227023299999996 2024-08-17T17:27:49.716 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.716 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.717 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.717 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270329+0000", 2024-08-17T17:27:49.717 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9240000000000007e-06 2024-08-17T17:27:49.717 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.717 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.717 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.717 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.301403+0000", 2024-08-17T17:27:49.717 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0310743539999998 2024-08-17T17:27:49.717 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.717 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.717 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:27:49.717 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.301580+0000", 2024-08-17T17:27:49.717 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017625900000000001 2024-08-17T17:27:49.717 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.718 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:27:49.718 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.301745+0000", 2024-08-17T17:27:49.718 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000165728 2024-08-17T17:27:49.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.718 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.718 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.307080+0000", 2024-08-17T17:27:49.718 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0053345570000000002 2024-08-17T17:27:49.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.718 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:27:49.718 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.311680+0000", 2024-08-17T17:27:49.718 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0046006270000000004 2024-08-17T17:27:49.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.719 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.719 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.314305+0000", 2024-08-17T17:27:49.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026242890000000001 2024-08-17T17:27:49.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.719 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:27:49.719 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.314364+0000", 2024-08-17T17:27:49.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8853e-05 2024-08-17T17:27:49.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.719 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:27:49.720 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.314904+0000", 2024-08-17T17:27:49.720 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00054049000000000002 2024-08-17T17:27:49.720 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.720 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:27:49.720 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.720 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.720 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.720 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26154 5.1 5:8b1cb6d6:::benchmark_data_smithi003_140793_object26153:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:27:49.720 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.390694+0000", 2024-08-17T17:27:49.720 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 507.23395452199998, 2024-08-17T17:27:49.720 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9249220940000002, 2024-08-17T17:27:49.720 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:27:49.720 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:27:49.720 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:27:49.721 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:27:49.721 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:27:49.721 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:27:49.721 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26154 2024-08-17T17:27:49.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.721 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:27:49.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:27:49.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.390694+0000", 2024-08-17T17:27:49.721 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:27:49.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.390694+0000", 2024-08-17T17:27:49.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.722 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:27:49.722 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.390695+0000", 2024-08-17T17:27:49.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3349999999999999e-06 2024-08-17T17:27:49.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.722 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:27:49.722 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.390716+0000", 2024-08-17T17:27:49.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0854999999999999e-05 2024-08-17T17:27:49.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.723 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:27:49.723 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.390728+0000", 2024-08-17T17:27:49.723 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2455e-05 2024-08-17T17:27:49.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.723 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:27:49.723 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.390736+0000", 2024-08-17T17:27:49.723 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0509999999999997e-06 2024-08-17T17:27:49.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.723 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.723 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.390824+0000", 2024-08-17T17:27:49.723 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7665999999999997e-05 2024-08-17T17:27:49.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.724 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.724 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.390835+0000", 2024-08-17T17:27:49.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0987e-05 2024-08-17T17:27:49.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.724 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.724 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270353+0000", 2024-08-17T17:27:49.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.879518146 2024-08-17T17:27:49.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.724 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.724 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270361+0000", 2024-08-17T17:27:49.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8779999999999998e-06 2024-08-17T17:27:49.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.725 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.725 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.302895+0000", 2024-08-17T17:27:49.725 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0325342339999999 2024-08-17T17:27:49.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.725 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:27:49.725 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.303046+0000", 2024-08-17T17:27:49.725 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015052899999999999 2024-08-17T17:27:49.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.725 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:27:49.725 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.303235+0000", 2024-08-17T17:27:49.725 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018953700000000001 2024-08-17T17:27:49.726 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.726 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.726 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.307121+0000", 2024-08-17T17:27:49.726 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038852779999999998 2024-08-17T17:27:49.726 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.726 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:27:49.726 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.311686+0000", 2024-08-17T17:27:49.726 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0045651889999999999 2024-08-17T17:27:49.726 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.726 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.726 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.315019+0000", 2024-08-17T17:27:49.727 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0033328609999999999 2024-08-17T17:27:49.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.727 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.727 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:27:49.727 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.315081+0000", 2024-08-17T17:27:49.727 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.198e-05 2024-08-17T17:27:49.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.727 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.727 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:27:49.727 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.315616+0000", 2024-08-17T17:27:49.727 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00053511300000000004 2024-08-17T17:27:49.727 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.727 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:27:49.727 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.728 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.728 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26175 5.1 5:8c6f3adb:::benchmark_data_smithi003_140793_object26174:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:27:49.728 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.403515+0000", 2024-08-17T17:27:49.728 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 507.22113345100001, 2024-08-17T17:27:49.728 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9127047070000001, 2024-08-17T17:27:49.728 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:27:49.728 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:27:49.728 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:27:49.728 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:27:49.728 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:27:49.728 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:27:49.728 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26175 2024-08-17T17:27:49.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.729 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:27:49.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:27:49.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.403515+0000", 2024-08-17T17:27:49.729 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:27:49.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.403515+0000", 2024-08-17T17:27:49.729 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:27:49.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.403517+0000", 2024-08-17T17:27:49.730 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:49 smithi053 ceph-mon[89963]: pgmap v587: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1.1 GiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 657 op/s; 3953/54012 objects degraded (7.319%); 0 B/s, 84 objects/s recovering 2024-08-17T17:27:49.731 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7910000000000001e-06 2024-08-17T17:27:49.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.731 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.731 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:27:49.731 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.403543+0000", 2024-08-17T17:27:49.731 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6806999999999999e-05 2024-08-17T17:27:49.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.731 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.731 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:27:49.731 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.403558+0000", 2024-08-17T17:27:49.731 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5034999999999999e-05 2024-08-17T17:27:49.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.731 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.732 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:27:49.732 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.403567+0000", 2024-08-17T17:27:49.732 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8789999999999992e-06 2024-08-17T17:27:49.732 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.732 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.732 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.732 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.403634+0000", 2024-08-17T17:27:49.732 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6321000000000004e-05 2024-08-17T17:27:49.732 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.732 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.732 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.732 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.403646+0000", 2024-08-17T17:27:49.732 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2628e-05 2024-08-17T17:27:49.732 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.733 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.733 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.733 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270384+0000", 2024-08-17T17:27:49.733 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.86673805400000004 2024-08-17T17:27:49.733 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.733 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.733 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.733 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270392+0000", 2024-08-17T17:27:49.733 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8380000000000002e-06 2024-08-17T17:27:49.733 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.733 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.733 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.733 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.304419+0000", 2024-08-17T17:27:49.733 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0340264549999998 2024-08-17T17:27:49.734 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.734 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.734 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:27:49.734 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.304562+0000", 2024-08-17T17:27:49.734 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014378500000000001 2024-08-17T17:27:49.734 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.734 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.734 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:27:49.734 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.304715+0000", 2024-08-17T17:27:49.734 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015284700000000001 2024-08-17T17:27:49.734 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.734 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.734 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.735 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.310579+0000", 2024-08-17T17:27:49.735 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0058634339999999998 2024-08-17T17:27:49.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.735 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.735 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:27:49.735 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.311692+0000", 2024-08-17T17:27:49.735 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001112912 2024-08-17T17:27:49.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.735 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.735 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.735 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.315676+0000", 2024-08-17T17:27:49.735 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039842649999999999 2024-08-17T17:27:49.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.735 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.736 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:27:49.736 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.315747+0000", 2024-08-17T17:27:49.736 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0735000000000006e-05 2024-08-17T17:27:49.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.736 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:27:49.736 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.316220+0000", 2024-08-17T17:27:49.736 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00047292100000000001 2024-08-17T17:27:49.736 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.736 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:27:49.736 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.737 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.737 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26336 5.1 5:837f3a0a:::benchmark_data_smithi003_140793_object26335:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:27:49.737 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.566135+0000", 2024-08-17T17:27:49.737 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 507.05851293799998, 2024-08-17T17:27:49.737 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7524043960000002, 2024-08-17T17:27:49.737 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:27:49.737 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:27:49.737 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:27:49.737 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:27:49.737 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:27:49.737 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:27:49.737 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26336 2024-08-17T17:27:49.737 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.738 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:27:49.738 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.738 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:27:49.738 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.566135+0000", 2024-08-17T17:27:49.738 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.738 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.738 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.738 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:27:49.738 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.566135+0000", 2024-08-17T17:27:49.738 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.738 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.738 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.738 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:27:49.738 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.566137+0000", 2024-08-17T17:27:49.739 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.477e-06 2024-08-17T17:27:49.739 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.739 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.739 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:27:49.739 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.566159+0000", 2024-08-17T17:27:49.739 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1691e-05 2024-08-17T17:27:49.739 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.739 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.739 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:27:49.739 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.566175+0000", 2024-08-17T17:27:49.739 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6249999999999999e-05 2024-08-17T17:27:49.739 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.739 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.739 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:27:49.740 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.566184+0000", 2024-08-17T17:27:49.740 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0299999999999999e-06 2024-08-17T17:27:49.740 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.740 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.740 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.740 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.566258+0000", 2024-08-17T17:27:49.740 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3739999999999995e-05 2024-08-17T17:27:49.740 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.740 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.740 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.740 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.566270+0000", 2024-08-17T17:27:49.740 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2931e-05 2024-08-17T17:27:49.740 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.740 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.741 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.741 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270415+0000", 2024-08-17T17:27:49.741 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.70414461900000003 2024-08-17T17:27:49.741 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.741 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.741 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.741 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270423+0000", 2024-08-17T17:27:49.741 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4329999999999998e-06 2024-08-17T17:27:49.741 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.741 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.741 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.741 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.305724+0000", 2024-08-17T17:27:49.741 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.03530159 2024-08-17T17:27:49.741 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.741 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.742 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:27:49.742 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.305884+0000", 2024-08-17T17:27:49.742 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000159658 2024-08-17T17:27:49.742 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.742 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.742 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:27:49.742 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.306079+0000", 2024-08-17T17:27:49.742 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019524700000000001 2024-08-17T17:27:49.742 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.742 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.742 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:27:49.742 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.311698+0000", 2024-08-17T17:27:49.742 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0056185810000000001 2024-08-17T17:27:49.742 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.743 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.743 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.743 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.312166+0000", 2024-08-17T17:27:49.743 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000468314 2024-08-17T17:27:49.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.743 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.743 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.743 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.317902+0000", 2024-08-17T17:27:49.743 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0057355949999999996 2024-08-17T17:27:49.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.743 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.743 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:27:49.743 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.317959+0000", 2024-08-17T17:27:49.743 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7932e-05 2024-08-17T17:27:49.744 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.744 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.744 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:27:49.744 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.318540+0000", 2024-08-17T17:27:49.744 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00058030799999999997 2024-08-17T17:27:49.744 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.744 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:27:49.744 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.744 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.744 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.744 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26347 5.1 5:83125685:::benchmark_data_smithi003_140793_object26346:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:27:49.744 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.575398+0000", 2024-08-17T17:27:49.744 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 507.04925004500001, 2024-08-17T17:27:49.744 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7503322890000002, 2024-08-17T17:27:49.745 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:27:49.745 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:27:49.745 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:27:49.745 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:27:49.745 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:27:49.745 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:27:49.745 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26347 2024-08-17T17:27:49.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.745 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:27:49.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.745 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:27:49.745 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.575398+0000", 2024-08-17T17:27:49.745 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.746 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.746 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:27:49.746 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.575398+0000", 2024-08-17T17:27:49.746 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.746 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.746 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.746 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:27:49.746 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.575400+0000", 2024-08-17T17:27:49.746 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.683e-06 2024-08-17T17:27:49.746 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.746 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.746 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:27:49.746 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.575430+0000", 2024-08-17T17:27:49.746 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9808e-05 2024-08-17T17:27:49.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.747 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:27:49.747 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.575447+0000", 2024-08-17T17:27:49.747 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7257000000000001e-05 2024-08-17T17:27:49.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.747 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:27:49.747 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.575456+0000", 2024-08-17T17:27:49.747 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8230000000000001e-06 2024-08-17T17:27:49.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.747 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.747 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.575520+0000", 2024-08-17T17:27:49.748 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4515000000000001e-05 2024-08-17T17:27:49.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.748 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.748 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.575529+0000", 2024-08-17T17:27:49.748 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3119999999999994e-06 2024-08-17T17:27:49.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.748 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.748 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270446+0000", 2024-08-17T17:27:49.748 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.69491715499999995 2024-08-17T17:27:49.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.748 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.749 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270454+0000", 2024-08-17T17:27:49.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8099999999999998e-06 2024-08-17T17:27:49.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.749 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.749 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.307156+0000", 2024-08-17T17:27:49.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0367026579999998 2024-08-17T17:27:49.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.749 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:27:49.749 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.307253+0000", 2024-08-17T17:27:49.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6832999999999999e-05 2024-08-17T17:27:49.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.750 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:27:49.750 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.307393+0000", 2024-08-17T17:27:49.750 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014028399999999999 2024-08-17T17:27:49.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.750 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.750 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:27:49.750 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.311703+0000", 2024-08-17T17:27:49.750 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004309729 2024-08-17T17:27:49.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.750 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.750 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.750 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.312995+0000", 2024-08-17T17:27:49.750 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0012918820000000001 2024-08-17T17:27:49.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.751 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.751 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.324907+0000", 2024-08-17T17:27:49.751 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011912114999999999 2024-08-17T17:27:49.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.751 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:27:49.751 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.325027+0000", 2024-08-17T17:27:49.751 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000119506 2024-08-17T17:27:49.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.751 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:27:49.751 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.325731+0000", 2024-08-17T17:27:49.751 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00070391900000000001 2024-08-17T17:27:49.752 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.752 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:27:49.752 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.752 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.752 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:26360 5.1 5:89f4cb1c:::benchmark_data_smithi003_140793_object26359:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-08-17T17:27:49.752 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:22.594844+0000", 2024-08-17T17:27:49.752 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 507.02980468700002, 2024-08-17T17:27:49.752 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7437470209999999, 2024-08-17T17:27:49.752 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:27:49.752 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:27:49.752 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:27:49.752 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:27:49.752 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:27:49.753 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:27:49.753 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26360 2024-08-17T17:27:49.753 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.753 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:27:49.753 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.753 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:27:49.753 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.594844+0000", 2024-08-17T17:27:49.753 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.753 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.753 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.753 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:27:49.753 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.594844+0000", 2024-08-17T17:27:49.753 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.753 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.754 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.754 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:27:49.754 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.594845+0000", 2024-08-17T17:27:49.754 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2929999999999999e-06 2024-08-17T17:27:49.754 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.754 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.754 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:27:49.754 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.594861+0000", 2024-08-17T17:27:49.754 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6368e-05 2024-08-17T17:27:49.754 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.754 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.754 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:27:49.754 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.594872+0000", 2024-08-17T17:27:49.754 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1079e-05 2024-08-17T17:27:49.755 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.755 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.755 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:27:49.755 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.594878+0000", 2024-08-17T17:27:49.755 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5629999999999999e-06 2024-08-17T17:27:49.755 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.755 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.755 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.755 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.594926+0000", 2024-08-17T17:27:49.755 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8498000000000002e-05 2024-08-17T17:27:49.755 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.755 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.755 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.755 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:22.594934+0000", 2024-08-17T17:27:49.756 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7999999999999999e-06 2024-08-17T17:27:49.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.756 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.756 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270477+0000", 2024-08-17T17:27:49.756 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.67554314800000004 2024-08-17T17:27:49.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.756 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.756 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:23.270486+0000", 2024-08-17T17:27:49.756 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5650000000000002e-06 2024-08-17T17:27:49.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.756 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.757 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.316324+0000", 2024-08-17T17:27:49.757 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0458376779999998 2024-08-17T17:27:49.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.757 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:27:49.757 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.316469+0000", 2024-08-17T17:27:49.757 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014542299999999999 2024-08-17T17:27:49.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.757 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:27:49.757 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.316621+0000", 2024-08-17T17:27:49.757 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015184700000000001 2024-08-17T17:27:49.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.758 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.758 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.320092+0000", 2024-08-17T17:27:49.758 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034713589999999998 2024-08-17T17:27:49.758 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.758 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.758 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:27:49.758 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.320106+0000", 2024-08-17T17:27:49.758 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3382e-05 2024-08-17T17:27:49.758 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.758 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.758 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.758 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.337839+0000", 2024-08-17T17:27:49.759 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017733498 2024-08-17T17:27:49.759 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.759 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.759 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:27:49.759 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.337900+0000", 2024-08-17T17:27:49.759 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1266999999999995e-05 2024-08-17T17:27:49.759 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.759 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.759 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:27:49.759 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:25.338591+0000", 2024-08-17T17:27:49.759 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00069025300000000005 2024-08-17T17:27:49.759 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.759 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:27:49.759 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.760 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.760 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.760 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:68138 5.1 5:8cd8b222:::benchmark_data_smithi003_140793_object68137:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-08-17T17:27:49.760 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:56.511667+0000", 2024-08-17T17:27:49.760 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 473.112981058, 2024-08-17T17:27:49.760 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2112100479999999, 2024-08-17T17:27:49.760 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:27:49.760 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:27:49.760 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:27:49.760 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:27:49.760 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:27:49.761 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:27:49.761 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68138 2024-08-17T17:27:49.761 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.761 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:27:49.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.761 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:27:49.761 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.511667+0000", 2024-08-17T17:27:49.761 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.761 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.761 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:27:49.761 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.511667+0000", 2024-08-17T17:27:49.761 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.761 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.762 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.762 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:27:49.762 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.511669+0000", 2024-08-17T17:27:49.762 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.61e-06 2024-08-17T17:27:49.762 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.762 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.762 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:27:49.762 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.511692+0000", 2024-08-17T17:27:49.762 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2768e-05 2024-08-17T17:27:49.762 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.762 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.762 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:27:49.762 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.511703+0000", 2024-08-17T17:27:49.763 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1060000000000001e-05 2024-08-17T17:27:49.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.763 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.763 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:27:49.763 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.511711+0000", 2024-08-17T17:27:49.763 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9470000000000003e-06 2024-08-17T17:27:49.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.763 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.763 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.763 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.512317+0000", 2024-08-17T17:27:49.763 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00060645199999999995 2024-08-17T17:27:49.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.763 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.763 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.764 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.512327+0000", 2024-08-17T17:27:49.764 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7159999999999998e-06 2024-08-17T17:27:49.764 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.764 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.764 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.764 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.497176+0000", 2024-08-17T17:27:49.764 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.98484927799999999 2024-08-17T17:27:49.764 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.764 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.764 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.764 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.497189+0000", 2024-08-17T17:27:49.764 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2729e-05 2024-08-17T17:27:49.764 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.764 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.765 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.765 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.711927+0000", 2024-08-17T17:27:49.765 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2147383940000001 2024-08-17T17:27:49.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.765 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.765 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:27:49.765 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.712104+0000", 2024-08-17T17:27:49.765 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017708700000000001 2024-08-17T17:27:49.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.765 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.765 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:27:49.765 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.712300+0000", 2024-08-17T17:27:49.765 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000195906 2024-08-17T17:27:49.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.766 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.766 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.766 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.719596+0000", 2024-08-17T17:27:49.766 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072962979999999997 2024-08-17T17:27:49.766 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.766 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.766 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.766 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.720426+0000", 2024-08-17T17:27:49.766 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00082983500000000001 2024-08-17T17:27:49.766 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.766 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.766 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:27:49.766 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.720670+0000", 2024-08-17T17:27:49.767 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000243856 2024-08-17T17:27:49.767 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.767 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.767 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:27:49.767 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.720724+0000", 2024-08-17T17:27:49.767 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4051000000000001e-05 2024-08-17T17:27:49.767 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.767 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.767 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:27:49.767 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.722877+0000", 2024-08-17T17:27:49.767 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002153061 2024-08-17T17:27:49.767 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.767 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:27:49.767 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.768 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.768 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.768 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.35277.0:68143 5.1 5:8f270daa:::benchmark_data_smithi003_140793_object68142:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-08-17T17:27:49.768 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:19:56.519782+0000", 2024-08-17T17:27:49.768 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 473.10486633099998, 2024-08-17T17:27:49.768 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2052141999999999, 2024-08-17T17:27:49.768 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:27:49.768 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:27:49.768 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:27:49.768 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:27:49.768 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.35277", 2024-08-17T17:27:49.768 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:27:49.768 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68143 2024-08-17T17:27:49.768 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.769 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:27:49.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.769 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:27:49.769 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519782+0000", 2024-08-17T17:27:49.769 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.769 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.769 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:27:49.769 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519782+0000", 2024-08-17T17:27:49.769 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.769 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.769 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:27:49.769 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519783+0000", 2024-08-17T17:27:49.770 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1310000000000001e-06 2024-08-17T17:27:49.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.770 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.770 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:27:49.770 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519802+0000", 2024-08-17T17:27:49.770 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9018999999999999e-05 2024-08-17T17:27:49.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.770 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.770 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:27:49.770 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519810+0000", 2024-08-17T17:27:49.770 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9370000000000004e-06 2024-08-17T17:27:49.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.770 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.770 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:27:49.771 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519877+0000", 2024-08-17T17:27:49.771 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7074000000000002e-05 2024-08-17T17:27:49.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.771 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.771 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519923+0000", 2024-08-17T17:27:49.771 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6057999999999998e-05 2024-08-17T17:27:49.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.771 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.771 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:56.519930+0000", 2024-08-17T17:27:49.771 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5379999999999999e-06 2024-08-17T17:27:49.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.772 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.772 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.497221+0000", 2024-08-17T17:27:49.772 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.977291668 2024-08-17T17:27:49.772 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.772 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.772 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.773 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:57.497230+0000", 2024-08-17T17:27:49.773 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2570000000000007e-06 2024-08-17T17:27:49.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.773 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.773 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.716884+0000", 2024-08-17T17:27:49.773 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2196546719999999 2024-08-17T17:27:49.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.773 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:27:49.773 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.717065+0000", 2024-08-17T17:27:49.773 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000181163 2024-08-17T17:27:49.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.774 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.774 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:27:49.774 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.717263+0000", 2024-08-17T17:27:49.774 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019748699999999999 2024-08-17T17:27:49.774 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.774 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.774 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.774 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.720657+0000", 2024-08-17T17:27:49.774 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003393784 2024-08-17T17:27:49.774 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.774 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.774 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:27:49.774 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.722900+0000", 2024-08-17T17:27:49.775 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0022431959999999998 2024-08-17T17:27:49.775 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.775 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.775 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.775 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.722944+0000", 2024-08-17T17:27:49.775 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3656999999999999e-05 2024-08-17T17:27:49.775 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.775 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.775 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:27:49.775 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.722995+0000", 2024-08-17T17:27:49.775 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1264999999999998e-05 2024-08-17T17:27:49.775 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.775 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.775 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:27:49.776 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:19:58.724996+0000", 2024-08-17T17:27:49.776 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020012939999999998 2024-08-17T17:27:49.776 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.776 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:27:49.776 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.776 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.776 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45491.0:51014 6.c 6:3b1e47a1:::benchmark_data_smithi003_175043_object51013:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-08-17T17:27:49.776 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:25:03.970975+0000", 2024-08-17T17:27:49.776 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 165.65367357, 2024-08-17T17:27:49.776 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.707060298, 2024-08-17T17:27:49.776 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:27:49.776 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:27:49.777 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:27:49.777 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:27:49.777 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45491", 2024-08-17T17:27:49.777 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:27:49.777 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 51014 2024-08-17T17:27:49.777 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.777 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:27:49.777 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.777 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:27:49.777 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.970975+0000", 2024-08-17T17:27:49.777 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.777 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.777 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.778 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:27:49.778 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.970975+0000", 2024-08-17T17:27:49.778 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.778 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.778 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.778 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:27:49.778 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.970976+0000", 2024-08-17T17:27:49.778 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.443e-06 2024-08-17T17:27:49.778 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.778 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.778 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:27:49.778 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.971002+0000", 2024-08-17T17:27:49.778 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5477999999999999e-05 2024-08-17T17:27:49.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.779 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.779 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:27:49.779 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.971015+0000", 2024-08-17T17:27:49.779 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3555e-05 2024-08-17T17:27:49.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.779 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.779 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:27:49.779 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.971023+0000", 2024-08-17T17:27:49.779 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3900000000000004e-06 2024-08-17T17:27:49.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.779 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.779 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.779 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.975076+0000", 2024-08-17T17:27:49.780 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040535240000000002 2024-08-17T17:27:49.780 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.780 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.780 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:27:49.780 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.975207+0000", 2024-08-17T17:27:49.780 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000130992 2024-08-17T17:27:49.780 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.780 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.780 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-08-17T17:27:49.780 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.975338+0000", 2024-08-17T17:27:49.780 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000130987 2024-08-17T17:27:49.780 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.780 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.780 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.781 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.990507+0000", 2024-08-17T17:27:49.781 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.015168921 2024-08-17T17:27:49.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.781 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.781 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:27:49.781 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.990522+0000", 2024-08-17T17:27:49.781 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5148e-05 2024-08-17T17:27:49.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.781 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.781 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.781 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.995576+0000", 2024-08-17T17:27:49.781 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0050539399999999998 2024-08-17T17:27:49.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.781 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.782 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.782 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.995590+0000", 2024-08-17T17:27:49.782 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3723000000000001e-05 2024-08-17T17:27:49.782 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.782 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.782 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.782 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:04.505851+0000", 2024-08-17T17:27:49.782 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.51026153500000004 2024-08-17T17:27:49.782 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.782 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.782 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.782 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:04.505866+0000", 2024-08-17T17:27:49.782 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4335000000000001e-05 2024-08-17T17:27:49.782 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.783 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.783 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.783 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:05.677960+0000", 2024-08-17T17:27:49.783 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.172094752 2024-08-17T17:27:49.783 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.783 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.783 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:27:49.783 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:05.678035+0000", 2024-08-17T17:27:49.783 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4574999999999997e-05 2024-08-17T17:27:49.783 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.783 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:27:49.783 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.783 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.783 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.784 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45491.0:51030 6.c 6:39c4a7e1:::benchmark_data_smithi003_175043_object51029:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-08-17T17:27:49.784 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:25:03.982807+0000", 2024-08-17T17:27:49.784 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 165.641841618, 2024-08-17T17:27:49.784 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7154847520000001, 2024-08-17T17:27:49.784 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:27:49.784 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:27:49.784 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:27:49.784 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:27:49.784 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45491", 2024-08-17T17:27:49.784 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:27:49.784 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 51030 2024-08-17T17:27:49.784 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.784 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:27:49.784 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.785 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:27:49.785 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.982807+0000", 2024-08-17T17:27:49.785 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.785 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:27:49.785 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.982807+0000", 2024-08-17T17:27:49.785 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:27:49.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.785 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:27:49.785 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.982807+0000", 2024-08-17T17:27:49.785 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3500000000000002e-07 2024-08-17T17:27:49.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.786 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:27:49.786 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.982837+0000", 2024-08-17T17:27:49.786 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9655999999999999e-05 2024-08-17T17:27:49.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.786 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:27:49.786 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.982844+0000", 2024-08-17T17:27:49.786 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1029999999999996e-06 2024-08-17T17:27:49.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.786 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:27:49.786 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.982849+0000", 2024-08-17T17:27:49.786 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8670000000000002e-06 2024-08-17T17:27:49.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.787 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.996164+0000", 2024-08-17T17:27:49.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013315268 2024-08-17T17:27:49.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.787 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.996178+0000", 2024-08-17T17:27:49.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3876999999999999e-05 2024-08-17T17:27:49.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.787 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:04.506236+0000", 2024-08-17T17:27:49.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.51005823800000005 2024-08-17T17:27:49.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.788 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.788 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:27:49.788 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:04.506250+0000", 2024-08-17T17:27:49.788 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3875e-05 2024-08-17T17:27:49.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.788 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.788 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:27:49.788 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:05.680158+0000", 2024-08-17T17:27:49.788 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1739081920000001 2024-08-17T17:27:49.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.788 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.788 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:27:49.788 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:05.680328+0000", 2024-08-17T17:27:49.788 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016995000000000001 2024-08-17T17:27:49.789 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.789 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,6", 2024-08-17T17:27:49.789 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:05.680581+0000", 2024-08-17T17:27:49.789 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00025260199999999999 2024-08-17T17:27:49.789 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.789 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:27:49.789 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:05.686170+0000", 2024-08-17T17:27:49.789 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0055897220000000001 2024-08-17T17:27:49.789 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.789 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.789 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:05.695445+0000", 2024-08-17T17:27:49.790 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0092747639999999996 2024-08-17T17:27:49.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.790 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:27:49.790 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:05.695511+0000", 2024-08-17T17:27:49.790 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6025000000000006e-05 2024-08-17T17:27:49.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.790 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:27:49.790 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:05.695579+0000", 2024-08-17T17:27:49.790 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7936999999999997e-05 2024-08-17T17:27:49.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:27:49.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:27:49.791 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:27:49.791 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:05.698291+0000", 2024-08-17T17:27:49.791 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027122409999999998 2024-08-17T17:27:49.791 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.791 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:27:49.791 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.791 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:27:49.791 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:27:49.791 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:27:49.791 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:49.818 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:49.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:49.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:49.920 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:49.948 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:49.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:49.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:50.050 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:50.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:50.091 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:50.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:50.194 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:50.238 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:50.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:50.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:50.340 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:50.365 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:50.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:50.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:50.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:50 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 3953/54012 objects degraded (7.319%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:50.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:27:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:27:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:27:50.467 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:50.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:50.507 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:50.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:50.608 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:50.632 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:50.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:50.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:50.723 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:27:50.734 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:50.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:50 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 3953/54012 objects degraded (7.319%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:50.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:50 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 3953/54012 objects degraded (7.319%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:50.785 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:50.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:50.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:50.888 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:50.932 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:50.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:50.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:51.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:51.073 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:51.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:51.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:51.099 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:27:51.176 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:51.202 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:51.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:51.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:51.304 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:51.359 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:51.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:51.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:51.434 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-17T17:27:51.461 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:51.509 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:51.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:51.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:51.611 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:51.655 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:51.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:51.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:51.691 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:51 smithi003 ceph-mon[103344]: pgmap v588: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1004 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 631 op/s; 3953/48081 objects degraded (8.222%); 0 B/s, 69 objects/s recovering 2024-08-17T17:27:51.692 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:51 smithi003 ceph-mon[100615]: pgmap v588: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1004 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 631 op/s; 3953/48081 objects degraded (8.222%); 0 B/s, 69 objects/s recovering 2024-08-17T17:27:51.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:51 smithi053 ceph-mon[89963]: pgmap v588: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 1004 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 631 op/s; 3953/48081 objects degraded (8.222%); 0 B/s, 69 objects/s recovering 2024-08-17T17:27:51.757 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:51.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:51.799 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:51.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:51.808 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:27:51.900 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:51.940 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:51.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:51.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:52.042 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:52.072 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:52.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:52.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:52.174 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:52.204 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:52.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:52.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:52.306 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:52.335 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:52.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:52.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:52.390 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:27:52.437 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:52.482 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:52.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:52.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:52.584 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:52.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:52.638 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:52.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:52.739 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:52.772 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:52.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:52.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:52.874 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:52.915 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:52.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:52.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:53.017 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:53.062 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:53.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:53.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:53.164 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:53.200 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:53.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:53.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:53.301 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:53.329 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:53.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:53.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:53.399 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:27:53.430 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:53.457 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:53.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:53.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:53.556 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:53 smithi003 ceph-mon[103344]: pgmap v589: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 877 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 741 op/s; 3715/41955 objects degraded (8.855%); 0 B/s, 82 objects/s recovering 2024-08-17T17:27:53.556 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:27:53.556 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:53 smithi003 ceph-mon[100615]: pgmap v589: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 877 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 741 op/s; 3715/41955 objects degraded (8.855%); 0 B/s, 82 objects/s recovering 2024-08-17T17:27:53.556 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:27:53.559 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:53.581 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:53.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:53.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:53.683 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:53.707 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:53.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:53.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:53.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:53 smithi053 ceph-mon[89963]: pgmap v589: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 877 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 741 op/s; 3715/41955 objects degraded (8.855%); 0 B/s, 82 objects/s recovering 2024-08-17T17:27:53.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:27:53.809 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:53.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:53.861 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:53.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:53.962 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:54.002 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:54.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:54.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:54.104 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:54.148 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:54.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:54.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:54.250 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:54.279 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:54.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:54.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:54.381 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:54.410 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:54.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:54.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:54.512 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:54.543 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:54.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:54.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:54.633 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:27:54.634 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:27:54.645 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:54.691 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:54.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:54.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:54.793 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:54.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:54.837 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:54.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:54.922 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:27:54.922 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2024-08-17T17:27:54.922 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 0 2024-08-17T17:27:54.922 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:27:54.938 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:54.973 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:54.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:54.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:55.075 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:55.106 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:55.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:55.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:55.208 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:55.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:55.252 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:55.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:55.353 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:55.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:55.408 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:55.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:55.509 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:55.543 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:55.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:55.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:55.560 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:55 smithi003 ceph-mon[103344]: pgmap v590: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 797 MiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 753 op/s; 3715/38139 objects degraded (9.741%); 0 B/s, 75 objects/s recovering 2024-08-17T17:27:55.560 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:55 smithi003 ceph-mon[103344]: from='client.45689 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:27:55.560 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:55 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 3715/38139 objects degraded (9.741%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:55.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:55 smithi003 ceph-mon[100615]: pgmap v590: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 797 MiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 753 op/s; 3715/38139 objects degraded (9.741%); 0 B/s, 75 objects/s recovering 2024-08-17T17:27:55.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:55 smithi003 ceph-mon[100615]: from='client.45689 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:27:55.561 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:55 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 3715/38139 objects degraded (9.741%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:55.568 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:55 smithi053 ceph-mon[89963]: pgmap v590: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 797 MiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 753 op/s; 3715/38139 objects degraded (9.741%); 0 B/s, 75 objects/s recovering 2024-08-17T17:27:55.569 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:55 smithi053 ceph-mon[89963]: from='client.45689 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:27:55.569 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:55 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 3715/38139 objects degraded (9.741%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:27:55.613 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-17T17:27:55.645 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:55.688 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:55.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:55.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:55.790 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:55.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:55.834 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:55.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:55.935 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:55.962 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:27:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:27:55] "GET /metrics HTTP/1.1" 200 36505 "" "Prometheus/2.51.0" 2024-08-17T17:27:55.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:55.979 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:55.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:56.005 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":590,"stamp":"2024-08-17T17:27:53.937848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":836048869,"num_objects":12713,"num_object_clones":0,"num_object_copies":38139,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3715,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12713,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":185731,"num_write_kb":6328384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17361,"num_bytes_recovered":776021382,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120990,"ondisk_log_size":120990,"up":267,"acting":265,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":273,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":5644696,"kb_used_data":3399456,"kb_used_omap":33,"kb_used_meta":2245150,"kb_avail":650571368,"statfs":{"total":671965249536,"available":666185080832,"internally_reserved":0,"allocated":3481042944,"data_stored":3472201185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34068,"internal_metadata":2299034348},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,6,3,8,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-521732096,"num_objects":-7961,"num_object_clones":0,"num_object_copies":-23883,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-793,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7961,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":793,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.568228"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":292,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.372284+0000","last_change":"2024-08-17T17:17:32.495652+0000","last_active":"2024-08-17T17:27:43.372284+0000","last_peered":"2024-08-17T17:27:43.372284+0000","last_clean":"2024-08-17T17:27:43.372284+0000","last_became_active":"2024-08-17T17:17:32.495218+0000","last_became_peered":"2024-08-17T17:17:32.495218+0000","last_unstale":"2024-08-17T17:27:43.372284+0000","last_undegraded":"2024-08-17T17:27:43.372284+0000","last_fullsized":"2024-08-17T17:27:43.372284+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:11:45.821282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":195,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-08-17T17:27:42.374290+0000","last_change":"2024-08-17T17:18:59.424887+0000","last_active":"2024-08-17T17:27:42.374290+0000","last_peered":"2024-08-17T17:27:42.374290+0000","last_clean":"2024-08-17T17:27:42.374290+0000","last_became_active":"2024-08-17T17:18:59.411539+0000","last_became_peered":"2024-08-17T17:18:59.411539+0000","last_unstale":"2024-08-17T17:27:42.374290+0000","last_undegraded":"2024-08-17T17:27:42.374290+0000","last_fullsized":"2024-08-17T17:27:42.374290+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:46:20.082241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":35,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-08-17T17:27:42.373662+0000","last_change":"2024-08-17T17:24:53.460742+0000","last_active":"2024-08-17T17:27:42.373662+0000","last_peered":"2024-08-17T17:27:42.373662+0000","last_clean":"2024-08-17T17:27:42.373662+0000","last_became_active":"2024-08-17T17:24:53.460365+0000","last_became_peered":"2024-08-17T17:24:53.460365+0000","last_unstale":"2024-08-17T17:27:42.373662+0000","last_undegraded":"2024-08-17T17:27:42.373662+0000","last_fullsized":"2024-08-17T17:27:42.373662+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:24:08.116968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":103,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.419643+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:27:43.419643+0000","last_peered":"2024-08-17T17:27:43.419643+0000","last_clean":"2024-08-17T17:27:43.419643+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:27:43.419643+0000","last_undegraded":"2024-08-17T17:27:43.419643+0000","last_fullsized":"2024-08-17T17:27:43.419643+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":103,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.419882+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:27:43.419882+0000","last_peered":"2024-08-17T17:27:43.419882+0000","last_clean":"2024-08-17T17:27:43.419882+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:27:43.419882+0000","last_undegraded":"2024-08-17T17:27:43.419882+0000","last_fullsized":"2024-08-17T17:27:43.419882+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":170,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-08-17T17:27:42.374018+0000","last_change":"2024-08-17T17:20:49.258206+0000","last_active":"2024-08-17T17:27:42.374018+0000","last_peered":"2024-08-17T17:27:42.374018+0000","last_clean":"2024-08-17T17:27:42.374018+0000","last_became_active":"2024-08-17T17:20:49.257761+0000","last_became_peered":"2024-08-17T17:20:49.257761+0000","last_unstale":"2024-08-17T17:27:42.374018+0000","last_undegraded":"2024-08-17T17:27:42.374018+0000","last_fullsized":"2024-08-17T17:27:42.374018+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:38:35.363748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":306,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.371961+0000","last_change":"2024-08-17T17:19:57.514773+0000","last_active":"2024-08-17T17:27:43.371961+0000","last_peered":"2024-08-17T17:27:43.371961+0000","last_clean":"2024-08-17T17:27:43.371961+0000","last_became_active":"2024-08-17T17:19:57.513839+0000","last_became_peered":"2024-08-17T17:19:57.513839+0000","last_unstale":"2024-08-17T17:27:43.371961+0000","last_undegraded":"2024-08-17T17:27:43.371961+0000","last_fullsized":"2024-08-17T17:27:43.371961+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:42:48.063169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":265,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-08-17T17:27:42.461747+0000","last_change":"2024-08-17T17:24:53.465875+0000","last_active":"2024-08-17T17:27:42.461747+0000","last_peered":"2024-08-17T17:27:42.461747+0000","last_clean":"2024-08-17T17:27:42.461747+0000","last_became_active":"2024-08-17T17:24:53.458129+0000","last_became_peered":"2024-08-17T17:24:53.458129+0000","last_unstale":"2024-08-17T17:27:42.461747+0000","last_undegraded":"2024-08-17T17:27:42.461747+0000","last_fullsized":"2024-08-17T17:27:42.461747+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:51:33.584869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"219'11586","reported_seq":16592,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:53.835538+0000","last_change":"2024-08-17T17:27:12.673270+0000","last_active":"2024-08-17T17:27:53.835538+0000","last_peered":"2024-08-17T17:27:53.835538+0000","last_clean":"2024-08-17T17:27:53.835538+0000","last_became_active":"2024-08-17T17:27:12.603527+0000","last_became_peered":"2024-08-17T17:27:12.603527+0000","last_unstale":"2024-08-17T17:27:53.835538+0000","last_undegraded":"2024-08-17T17:27:53.835538+0000","last_fullsized":"2024-08-17T17:27:53.835538+0000","mapping_epoch":214,"log_start":"191'1600","ondisk_log_start":"191'1600","created":178,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":9986,"log_dups_size":0,"ondisk_log_size":9986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:35:47.074384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43515904,"num_objects":664,"num_object_clones":0,"num_object_copies":1992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11586,"num_write_kb":392000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2460,"num_bytes_recovered":33816576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":131,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-08-17T17:27:42.374780+0000","last_change":"2024-08-17T17:24:06.597382+0000","last_active":"2024-08-17T17:27:42.374780+0000","last_peered":"2024-08-17T17:27:42.374780+0000","last_clean":"2024-08-17T17:27:42.374780+0000","last_became_active":"2024-08-17T17:24:06.588881+0000","last_became_peered":"2024-08-17T17:24:06.588881+0000","last_unstale":"2024-08-17T17:27:42.374780+0000","last_undegraded":"2024-08-17T17:27:42.374780+0000","last_fullsized":"2024-08-17T17:27:42.374780+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:27.240751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":45,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-08-17T17:27:42.373487+0000","last_change":"2024-08-17T17:25:04.503780+0000","last_active":"2024-08-17T17:27:42.373487+0000","last_peered":"2024-08-17T17:27:42.373487+0000","last_clean":"2024-08-17T17:27:42.373487+0000","last_became_active":"2024-08-17T17:25:04.503503+0000","last_became_peered":"2024-08-17T17:25:04.503503+0000","last_unstale":"2024-08-17T17:27:42.373487+0000","last_undegraded":"2024-08-17T17:27:42.373487+0000","last_fullsized":"2024-08-17T17:27:42.373487+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:04:48.171216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"219'11571","reported_seq":11665,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:51.895062+0000","last_change":"2024-08-17T17:24:14.639361+0000","last_active":"2024-08-17T17:27:51.895062+0000","last_peered":"2024-08-17T17:27:51.895062+0000","last_clean":"2024-08-17T17:27:51.895062+0000","last_became_active":"2024-08-17T17:24:14.639074+0000","last_became_peered":"2024-08-17T17:24:14.639074+0000","last_unstale":"2024-08-17T17:27:51.895062+0000","last_undegraded":"2024-08-17T17:27:51.895062+0000","last_fullsized":"2024-08-17T17:27:51.895062+0000","mapping_epoch":178,"log_start":"203'5100","ondisk_log_start":"203'5100","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6471,"log_dups_size":0,"ondisk_log_size":6471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:15:05.377992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51970048,"num_objects":793,"num_object_clones":0,"num_object_copies":2379,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11571,"num_write_kb":395648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":117,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-08-17T17:27:42.374809+0000","last_change":"2024-08-17T17:24:06.599023+0000","last_active":"2024-08-17T17:27:42.374809+0000","last_peered":"2024-08-17T17:27:42.374809+0000","last_clean":"2024-08-17T17:27:42.374809+0000","last_became_active":"2024-08-17T17:24:06.588954+0000","last_became_peered":"2024-08-17T17:24:06.588954+0000","last_unstale":"2024-08-17T17:27:42.374809+0000","last_undegraded":"2024-08-17T17:27:42.374809+0000","last_fullsized":"2024-08-17T17:27:42.374809+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:20:51.445576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":247,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.373800+0000","last_change":"2024-08-17T17:20:49.271540+0000","last_active":"2024-08-17T17:27:43.373800+0000","last_peered":"2024-08-17T17:27:43.373800+0000","last_clean":"2024-08-17T17:27:43.373800+0000","last_became_active":"2024-08-17T17:20:49.271130+0000","last_became_peered":"2024-08-17T17:20:49.271130+0000","last_unstale":"2024-08-17T17:27:43.373800+0000","last_undegraded":"2024-08-17T17:27:43.373800+0000","last_fullsized":"2024-08-17T17:27:43.373800+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:05.386705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"219'11330","reported_seq":11379,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:49.014412+0000","last_change":"2024-08-17T17:24:14.644070+0000","last_active":"2024-08-17T17:27:49.014412+0000","last_peered":"2024-08-17T17:27:49.014412+0000","last_clean":"2024-08-17T17:27:49.014412+0000","last_became_active":"2024-08-17T17:24:14.643799+0000","last_became_peered":"2024-08-17T17:24:14.643799+0000","last_unstale":"2024-08-17T17:27:49.014412+0000","last_undegraded":"2024-08-17T17:27:49.014412+0000","last_fullsized":"2024-08-17T17:27:49.014412+0000","mapping_epoch":178,"log_start":"202'4300","ondisk_log_start":"202'4300","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":7030,"log_dups_size":0,"ondisk_log_size":7030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:25:38.445770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61079552,"num_objects":932,"num_object_clones":0,"num_object_copies":2796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11330,"num_write_kb":392384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":189,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-08-17T17:27:42.374584+0000","last_change":"2024-08-17T17:24:53.467526+0000","last_active":"2024-08-17T17:27:42.374584+0000","last_peered":"2024-08-17T17:27:42.374584+0000","last_clean":"2024-08-17T17:27:42.374584+0000","last_became_active":"2024-08-17T17:24:53.460631+0000","last_became_peered":"2024-08-17T17:24:53.460631+0000","last_unstale":"2024-08-17T17:27:42.374584+0000","last_undegraded":"2024-08-17T17:27:42.374584+0000","last_fullsized":"2024-08-17T17:27:42.374584+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:05:24.938831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":31,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-08-17T17:27:42.461929+0000","last_change":"2024-08-17T17:25:04.507176+0000","last_active":"2024-08-17T17:27:42.461929+0000","last_peered":"2024-08-17T17:27:42.461929+0000","last_clean":"2024-08-17T17:27:42.461929+0000","last_became_active":"2024-08-17T17:25:04.506788+0000","last_became_peered":"2024-08-17T17:25:04.506788+0000","last_unstale":"2024-08-17T17:27:42.461929+0000","last_undegraded":"2024-08-17T17:27:42.461929+0000","last_fullsized":"2024-08-17T17:27:42.461929+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:22:04.658216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"219'11356","reported_seq":11450,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:53.050404+0000","last_change":"2024-08-17T17:24:14.642637+0000","last_active":"2024-08-17T17:27:53.050404+0000","last_peered":"2024-08-17T17:27:53.050404+0000","last_clean":"2024-08-17T17:27:53.050404+0000","last_became_active":"2024-08-17T17:24:14.642347+0000","last_became_peered":"2024-08-17T17:24:14.642347+0000","last_unstale":"2024-08-17T17:27:53.050404+0000","last_undegraded":"2024-08-17T17:27:53.050404+0000","last_fullsized":"2024-08-17T17:27:53.050404+0000","mapping_epoch":178,"log_start":"199'3200","ondisk_log_start":"199'3200","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":8156,"log_dups_size":0,"ondisk_log_size":8156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:37:44.631335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44171264,"num_objects":674,"num_object_clones":0,"num_object_copies":2022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11356,"num_write_kb":384960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":177,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-08-17T17:27:42.373988+0000","last_change":"2024-08-17T17:17:32.495428+0000","last_active":"2024-08-17T17:27:42.373988+0000","last_peered":"2024-08-17T17:27:42.373988+0000","last_clean":"2024-08-17T17:27:42.373988+0000","last_became_active":"2024-08-17T17:17:32.494978+0000","last_became_peered":"2024-08-17T17:17:32.494978+0000","last_unstale":"2024-08-17T17:27:42.373988+0000","last_undegraded":"2024-08-17T17:27:42.373988+0000","last_fullsized":"2024-08-17T17:27:42.373988+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:42:24.647812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":295,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.419746+0000","last_change":"2024-08-17T17:20:49.272126+0000","last_active":"2024-08-17T17:27:43.419746+0000","last_peered":"2024-08-17T17:27:43.419746+0000","last_clean":"2024-08-17T17:27:43.419746+0000","last_became_active":"2024-08-17T17:20:49.271185+0000","last_became_peered":"2024-08-17T17:20:49.271185+0000","last_unstale":"2024-08-17T17:27:43.419746+0000","last_undegraded":"2024-08-17T17:27:43.419746+0000","last_fullsized":"2024-08-17T17:27:43.419746+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:43:58.184939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"219'11888","reported_seq":18039,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:50.512506+0000","last_change":"2024-08-17T17:27:43.414735+0000","last_active":"2024-08-17T17:27:50.512506+0000","last_peered":"2024-08-17T17:27:50.512506+0000","last_clean":"2024-08-17T17:27:50.512506+0000","last_became_active":"2024-08-17T17:27:43.376560+0000","last_became_peered":"2024-08-17T17:27:43.376560+0000","last_unstale":"2024-08-17T17:27:50.512506+0000","last_undegraded":"2024-08-17T17:27:50.512506+0000","last_fullsized":"2024-08-17T17:27:50.512506+0000","mapping_epoch":218,"log_start":"203'5200","ondisk_log_start":"203'5200","created":178,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6688,"log_dups_size":0,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:16:55.157437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56754176,"num_objects":866,"num_object_clones":0,"num_object_copies":2598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":866,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11888,"num_write_kb":408128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3017,"num_bytes_recovered":28180480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":323,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.371606+0000","last_change":"2024-08-17T17:24:06.595380+0000","last_active":"2024-08-17T17:27:43.371606+0000","last_peered":"2024-08-17T17:27:43.371606+0000","last_clean":"2024-08-17T17:27:43.371606+0000","last_became_active":"2024-08-17T17:24:06.584747+0000","last_became_peered":"2024-08-17T17:24:06.584747+0000","last_unstale":"2024-08-17T17:27:43.371606+0000","last_undegraded":"2024-08-17T17:27:43.371606+0000","last_fullsized":"2024-08-17T17:27:43.371606+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:02:18.473980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":269,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.372496+0000","last_change":"2024-08-17T17:20:49.280596+0000","last_active":"2024-08-17T17:27:43.372496+0000","last_peered":"2024-08-17T17:27:43.372496+0000","last_clean":"2024-08-17T17:27:43.372496+0000","last_became_active":"2024-08-17T17:20:49.279553+0000","last_became_peered":"2024-08-17T17:20:49.279553+0000","last_unstale":"2024-08-17T17:27:43.372496+0000","last_undegraded":"2024-08-17T17:27:43.372496+0000","last_fullsized":"2024-08-17T17:27:43.372496+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:13:24.124392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"219'11403","reported_seq":11497,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:51.930079+0000","last_change":"2024-08-17T17:24:14.638890+0000","last_active":"2024-08-17T17:27:51.930079+0000","last_peered":"2024-08-17T17:27:51.930079+0000","last_clean":"2024-08-17T17:27:51.930079+0000","last_became_active":"2024-08-17T17:24:14.638594+0000","last_became_peered":"2024-08-17T17:24:14.638594+0000","last_unstale":"2024-08-17T17:27:51.930079+0000","last_undegraded":"2024-08-17T17:27:51.930079+0000","last_fullsized":"2024-08-17T17:27:51.930079+0000","mapping_epoch":178,"log_start":"203'5000","ondisk_log_start":"203'5000","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6403,"log_dups_size":0,"ondisk_log_size":6403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:53:34.277077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52625408,"num_objects":803,"num_object_clones":0,"num_object_copies":2409,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11403,"num_write_kb":390592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":221,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.373654+0000","last_change":"2024-08-17T17:20:49.276769+0000","last_active":"2024-08-17T17:27:43.373654+0000","last_peered":"2024-08-17T17:27:43.373654+0000","last_clean":"2024-08-17T17:27:43.373654+0000","last_became_active":"2024-08-17T17:20:49.276498+0000","last_became_peered":"2024-08-17T17:20:49.276498+0000","last_unstale":"2024-08-17T17:27:43.373654+0000","last_undegraded":"2024-08-17T17:27:43.373654+0000","last_fullsized":"2024-08-17T17:27:43.373654+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:39:05.419440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":198,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-08-17T17:27:42.374439+0000","last_change":"2024-08-17T17:20:14.455738+0000","last_active":"2024-08-17T17:27:42.374439+0000","last_peered":"2024-08-17T17:27:42.374439+0000","last_clean":"2024-08-17T17:27:42.374439+0000","last_became_active":"2024-08-17T17:20:14.455469+0000","last_became_peered":"2024-08-17T17:20:14.455469+0000","last_unstale":"2024-08-17T17:27:42.374439+0000","last_undegraded":"2024-08-17T17:27:42.374439+0000","last_fullsized":"2024-08-17T17:27:42.374439+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:27:28.424336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"219'11560","reported_seq":18914,"reported_epoch":219,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:27:51.909072+0000","last_change":"2024-08-17T17:25:05.678234+0000","last_active":"2024-08-17T17:27:51.909072+0000","last_peered":"2024-08-17T17:27:51.909072+0000","last_clean":"2024-08-17T17:24:52.436463+0000","last_became_active":"2024-08-17T17:25:05.673238+0000","last_became_peered":"2024-08-17T17:25:05.673238+0000","last_unstale":"2024-08-17T17:27:51.909072+0000","last_undegraded":"2024-08-17T17:25:05.585769+0000","last_fullsized":"2024-08-17T17:25:05.499112+0000","mapping_epoch":198,"log_start":"203'5100","ondisk_log_start":"203'5100","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6460,"log_dups_size":0,"ondisk_log_size":6460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52953088,"num_objects":808,"num_object_clones":0,"num_object_copies":2424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11560,"num_write_kb":395776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2519,"num_bytes_recovered":165085184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,5],"avail_no_missing":["3","1","5","7"],"object_location_counts":[{"shards":"3,5","objects":-218}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":152,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-08-17T17:27:42.373972+0000","last_change":"2024-08-17T17:20:14.431140+0000","last_active":"2024-08-17T17:27:42.373972+0000","last_peered":"2024-08-17T17:27:42.373972+0000","last_clean":"2024-08-17T17:27:42.373972+0000","last_became_active":"2024-08-17T17:20:14.430839+0000","last_became_peered":"2024-08-17T17:20:14.430839+0000","last_unstale":"2024-08-17T17:27:42.373972+0000","last_undegraded":"2024-08-17T17:27:42.373972+0000","last_fullsized":"2024-08-17T17:27:42.373972+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:08:16.303616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":31,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-08-17T17:27:42.461895+0000","last_change":"2024-08-17T17:25:04.509325+0000","last_active":"2024-08-17T17:27:42.461895+0000","last_peered":"2024-08-17T17:27:42.461895+0000","last_clean":"2024-08-17T17:27:42.461895+0000","last_became_active":"2024-08-17T17:25:04.508364+0000","last_became_peered":"2024-08-17T17:25:04.508364+0000","last_unstale":"2024-08-17T17:27:42.461895+0000","last_undegraded":"2024-08-17T17:27:42.461895+0000","last_fullsized":"2024-08-17T17:27:42.461895+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:48:56.415763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"219'11439","reported_seq":16277,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:53.836497+0000","last_change":"2024-08-17T17:25:40.811756+0000","last_active":"2024-08-17T17:27:53.836497+0000","last_peered":"2024-08-17T17:27:53.836497+0000","last_clean":"2024-08-17T17:27:53.836497+0000","last_became_active":"2024-08-17T17:25:40.771755+0000","last_became_peered":"2024-08-17T17:25:40.771755+0000","last_unstale":"2024-08-17T17:27:53.836497+0000","last_undegraded":"2024-08-17T17:27:53.836497+0000","last_fullsized":"2024-08-17T17:27:53.836497+0000","mapping_epoch":204,"log_start":"191'1400","ondisk_log_start":"191'1400","created":178,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:13:03.918487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44433432,"num_objects":679,"num_object_clones":0,"num_object_copies":2037,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":679,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11439,"num_write_kb":387713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2371,"num_bytes_recovered":155385856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":307,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.371038+0000","last_change":"2024-08-17T17:19:57.520745+0000","last_active":"2024-08-17T17:27:43.371038+0000","last_peered":"2024-08-17T17:27:43.371038+0000","last_clean":"2024-08-17T17:27:43.371038+0000","last_became_active":"2024-08-17T17:19:57.520152+0000","last_became_peered":"2024-08-17T17:19:57.520152+0000","last_unstale":"2024-08-17T17:27:43.371038+0000","last_undegraded":"2024-08-17T17:27:43.371038+0000","last_fullsized":"2024-08-17T17:27:43.371038+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:19:39.495644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":283,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.373792+0000","last_change":"2024-08-17T17:20:49.274372+0000","last_active":"2024-08-17T17:27:43.373792+0000","last_peered":"2024-08-17T17:27:43.373792+0000","last_clean":"2024-08-17T17:27:43.373792+0000","last_became_active":"2024-08-17T17:20:49.274023+0000","last_became_peered":"2024-08-17T17:20:49.274023+0000","last_unstale":"2024-08-17T17:27:43.373792+0000","last_undegraded":"2024-08-17T17:27:43.373792+0000","last_fullsized":"2024-08-17T17:27:43.373792+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:31:02.529766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"219'12093","reported_seq":12142,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:53.863030+0000","last_change":"2024-08-17T17:24:14.643398+0000","last_active":"2024-08-17T17:27:53.863030+0000","last_peered":"2024-08-17T17:27:53.863030+0000","last_clean":"2024-08-17T17:27:53.863030+0000","last_became_active":"2024-08-17T17:24:14.643102+0000","last_became_peered":"2024-08-17T17:24:14.643102+0000","last_unstale":"2024-08-17T17:27:53.863030+0000","last_undegraded":"2024-08-17T17:27:53.863030+0000","last_fullsized":"2024-08-17T17:27:53.863030+0000","mapping_epoch":178,"log_start":"191'2000","ondisk_log_start":"191'2000","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:09:30.323709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40304640,"num_objects":615,"num_object_clones":0,"num_object_copies":1845,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12093,"num_write_kb":406656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":546,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-08-17T17:27:42.373726+0000","last_change":"2024-08-17T17:24:09.599615+0000","last_active":"2024-08-17T17:27:42.373726+0000","last_peered":"2024-08-17T17:27:42.373726+0000","last_clean":"2024-08-17T17:27:42.373726+0000","last_became_active":"2024-08-17T17:24:09.599085+0000","last_became_peered":"2024-08-17T17:24:09.599085+0000","last_unstale":"2024-08-17T17:27:42.373726+0000","last_undegraded":"2024-08-17T17:27:42.373726+0000","last_fullsized":"2024-08-17T17:27:42.373726+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:04:01.141401+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:30:09.569579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":791,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-08-17T17:27:42.373841+0000","last_change":"2024-08-17T17:20:00.434774+0000","last_active":"2024-08-17T17:27:42.373841+0000","last_peered":"2024-08-17T17:27:42.373841+0000","last_clean":"2024-08-17T17:27:42.373841+0000","last_became_active":"2024-08-17T17:20:00.434343+0000","last_became_peered":"2024-08-17T17:20:00.434343+0000","last_unstale":"2024-08-17T17:27:42.373841+0000","last_undegraded":"2024-08-17T17:27:42.373841+0000","last_fullsized":"2024-08-17T17:27:42.373841+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:04.995152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":265,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.371204+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:27:43.371204+0000","last_peered":"2024-08-17T17:27:43.371204+0000","last_clean":"2024-08-17T17:27:43.371204+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:27:43.371204+0000","last_undegraded":"2024-08-17T17:27:43.371204+0000","last_fullsized":"2024-08-17T17:27:43.371204+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":31,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-08-17T17:27:42.463569+0000","last_change":"2024-08-17T17:25:04.579426+0000","last_active":"2024-08-17T17:27:42.463569+0000","last_peered":"2024-08-17T17:27:42.463569+0000","last_clean":"2024-08-17T17:27:42.463569+0000","last_became_active":"2024-08-17T17:25:04.579012+0000","last_became_peered":"2024-08-17T17:25:04.579012+0000","last_unstale":"2024-08-17T17:27:42.463569+0000","last_undegraded":"2024-08-17T17:27:42.463569+0000","last_fullsized":"2024-08-17T17:27:42.463569+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:56:32.241321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"219'11350","reported_seq":11442,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:50.533139+0000","last_change":"2024-08-17T17:24:14.641829+0000","last_active":"2024-08-17T17:27:50.533139+0000","last_peered":"2024-08-17T17:27:50.533139+0000","last_clean":"2024-08-17T17:27:50.533139+0000","last_became_active":"2024-08-17T17:24:14.641532+0000","last_became_peered":"2024-08-17T17:24:14.641532+0000","last_unstale":"2024-08-17T17:27:50.533139+0000","last_undegraded":"2024-08-17T17:27:50.533139+0000","last_fullsized":"2024-08-17T17:27:50.533139+0000","mapping_epoch":178,"log_start":"202'4700","ondisk_log_start":"202'4700","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6650,"log_dups_size":0,"ondisk_log_size":6650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:43:05.232036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58195968,"num_objects":888,"num_object_clones":0,"num_object_copies":2664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11350,"num_write_kb":391616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":809,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-08-17T17:27:42.373963+0000","last_change":"2024-08-17T17:16:26.958415+0000","last_active":"2024-08-17T17:27:42.373963+0000","last_peered":"2024-08-17T17:27:42.373963+0000","last_clean":"2024-08-17T17:27:42.373963+0000","last_became_active":"2024-08-17T17:16:26.957956+0000","last_became_peered":"2024-08-17T17:16:26.957956+0000","last_unstale":"2024-08-17T17:27:42.373963+0000","last_undegraded":"2024-08-17T17:27:42.373963+0000","last_fullsized":"2024-08-17T17:27:42.373963+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:22.478077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":294,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.420297+0000","last_change":"2024-08-17T17:18:59.415331+0000","last_active":"2024-08-17T17:27:43.420297+0000","last_peered":"2024-08-17T17:27:43.420297+0000","last_clean":"2024-08-17T17:27:43.420297+0000","last_became_active":"2024-08-17T17:18:59.415008+0000","last_became_peered":"2024-08-17T17:18:59.415008+0000","last_unstale":"2024-08-17T17:27:43.420297+0000","last_undegraded":"2024-08-17T17:27:43.420297+0000","last_fullsized":"2024-08-17T17:27:43.420297+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:22:38.076866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":103,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.420052+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:27:43.420052+0000","last_peered":"2024-08-17T17:27:43.420052+0000","last_clean":"2024-08-17T17:27:43.420052+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:27:43.420052+0000","last_undegraded":"2024-08-17T17:27:43.420052+0000","last_fullsized":"2024-08-17T17:27:43.420052+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"219'11278","reported_seq":17563,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:49.043722+0000","last_change":"2024-08-17T17:26:40.426009+0000","last_active":"2024-08-17T17:27:49.043722+0000","last_peered":"2024-08-17T17:27:49.043722+0000","last_clean":"2024-08-17T17:27:49.043722+0000","last_became_active":"2024-08-17T17:26:40.411144+0000","last_became_peered":"2024-08-17T17:26:40.411144+0000","last_unstale":"2024-08-17T17:27:49.043722+0000","last_undegraded":"2024-08-17T17:27:49.043722+0000","last_fullsized":"2024-08-17T17:27:49.043722+0000","mapping_epoch":211,"log_start":"202'4300","ondisk_log_start":"202'4300","created":178,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6978,"log_dups_size":0,"ondisk_log_size":6978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:22.245859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59244544,"num_objects":904,"num_object_clones":0,"num_object_copies":2712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11278,"num_write_kb":389824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3108,"num_bytes_recovered":178388992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":682,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.371121+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:27:43.371121+0000","last_peered":"2024-08-17T17:27:43.371121+0000","last_clean":"2024-08-17T17:27:43.371121+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:27:43.371121+0000","last_undegraded":"2024-08-17T17:27:43.371121+0000","last_fullsized":"2024-08-17T17:27:43.371121+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":290,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.373156+0000","last_change":"2024-08-17T17:19:57.515564+0000","last_active":"2024-08-17T17:27:43.373156+0000","last_peered":"2024-08-17T17:27:43.373156+0000","last_clean":"2024-08-17T17:27:43.373156+0000","last_became_active":"2024-08-17T17:19:57.515142+0000","last_became_peered":"2024-08-17T17:19:57.515142+0000","last_unstale":"2024-08-17T17:27:43.373156+0000","last_undegraded":"2024-08-17T17:27:43.373156+0000","last_fullsized":"2024-08-17T17:27:43.373156+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:59:33.506176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":321,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.373638+0000","last_change":"2024-08-17T17:24:06.587903+0000","last_active":"2024-08-17T17:27:43.373638+0000","last_peered":"2024-08-17T17:27:43.373638+0000","last_clean":"2024-08-17T17:27:43.373638+0000","last_became_active":"2024-08-17T17:24:06.587503+0000","last_became_peered":"2024-08-17T17:24:06.587503+0000","last_unstale":"2024-08-17T17:27:43.373638+0000","last_undegraded":"2024-08-17T17:27:43.373638+0000","last_fullsized":"2024-08-17T17:27:43.373638+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:05:58.591992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"219'11372","reported_seq":17683,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:51.930774+0000","last_change":"2024-08-17T17:26:08.233981+0000","last_active":"2024-08-17T17:27:51.930774+0000","last_peered":"2024-08-17T17:27:51.930774+0000","last_clean":"2024-08-17T17:27:51.930774+0000","last_became_active":"2024-08-17T17:26:08.225711+0000","last_became_peered":"2024-08-17T17:26:08.225711+0000","last_unstale":"2024-08-17T17:27:51.930774+0000","last_undegraded":"2024-08-17T17:27:51.930774+0000","last_fullsized":"2024-08-17T17:27:51.930774+0000","mapping_epoch":207,"log_start":"203'4900","ondisk_log_start":"203'4900","created":178,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6472,"log_dups_size":0,"ondisk_log_size":6472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:31:42.806869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51773440,"num_objects":790,"num_object_clones":0,"num_object_copies":2370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11372,"num_write_kb":389184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3101,"num_bytes_recovered":203227136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":1129,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-08-17T17:27:42.373900+0000","last_change":"2024-08-17T17:20:00.433870+0000","last_active":"2024-08-17T17:27:42.373900+0000","last_peered":"2024-08-17T17:27:42.373900+0000","last_clean":"2024-08-17T17:27:42.373900+0000","last_became_active":"2024-08-17T17:20:00.433386+0000","last_became_peered":"2024-08-17T17:20:00.433386+0000","last_unstale":"2024-08-17T17:27:42.373900+0000","last_undegraded":"2024-08-17T17:27:42.373900+0000","last_fullsized":"2024-08-17T17:27:42.373900+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:01:11.848862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1392,"num_read_kb":2863,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":216,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.419578+0000","last_change":"2024-08-17T17:20:49.272407+0000","last_active":"2024-08-17T17:27:43.419578+0000","last_peered":"2024-08-17T17:27:43.419578+0000","last_clean":"2024-08-17T17:27:43.419578+0000","last_became_active":"2024-08-17T17:20:49.271749+0000","last_became_peered":"2024-08-17T17:20:49.271749+0000","last_unstale":"2024-08-17T17:27:43.419578+0000","last_undegraded":"2024-08-17T17:27:43.419578+0000","last_fullsized":"2024-08-17T17:27:43.419578+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:36:01.443457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":287,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.374081+0000","last_change":"2024-08-17T17:17:32.499715+0000","last_active":"2024-08-17T17:27:43.374081+0000","last_peered":"2024-08-17T17:27:43.374081+0000","last_clean":"2024-08-17T17:27:43.374081+0000","last_became_active":"2024-08-17T17:17:32.499412+0000","last_became_peered":"2024-08-17T17:17:32.499412+0000","last_unstale":"2024-08-17T17:27:43.374081+0000","last_undegraded":"2024-08-17T17:27:43.374081+0000","last_fullsized":"2024-08-17T17:27:43.374081+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:39:38.456970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"219'11630","reported_seq":11724,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:53.085622+0000","last_change":"2024-08-17T17:24:14.642608+0000","last_active":"2024-08-17T17:27:53.085622+0000","last_peered":"2024-08-17T17:27:53.085622+0000","last_clean":"2024-08-17T17:27:53.085622+0000","last_became_active":"2024-08-17T17:24:14.642292+0000","last_became_peered":"2024-08-17T17:24:14.642292+0000","last_unstale":"2024-08-17T17:27:53.085622+0000","last_undegraded":"2024-08-17T17:27:53.085622+0000","last_fullsized":"2024-08-17T17:27:53.085622+0000","mapping_epoch":178,"log_start":"199'3500","ondisk_log_start":"199'3500","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":8130,"log_dups_size":0,"ondisk_log_size":8130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:53:36.848614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45481984,"num_objects":694,"num_object_clones":0,"num_object_copies":2082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11630,"num_write_kb":394368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":194,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-08-17T17:27:42.373843+0000","last_change":"2024-08-17T17:17:32.495728+0000","last_active":"2024-08-17T17:27:42.373843+0000","last_peered":"2024-08-17T17:27:42.373843+0000","last_clean":"2024-08-17T17:27:42.373843+0000","last_became_active":"2024-08-17T17:17:32.495357+0000","last_became_peered":"2024-08-17T17:17:32.495357+0000","last_unstale":"2024-08-17T17:27:42.373843+0000","last_undegraded":"2024-08-17T17:27:42.373843+0000","last_fullsized":"2024-08-17T17:27:42.373843+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:27:45.561567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":523,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.374175+0000","last_change":"2024-08-17T17:16:26.964621+0000","last_active":"2024-08-17T17:27:43.374175+0000","last_peered":"2024-08-17T17:27:43.374175+0000","last_clean":"2024-08-17T17:27:43.374175+0000","last_became_active":"2024-08-17T17:16:26.964058+0000","last_became_peered":"2024-08-17T17:16:26.964058+0000","last_unstale":"2024-08-17T17:27:43.374175+0000","last_undegraded":"2024-08-17T17:27:43.374175+0000","last_fullsized":"2024-08-17T17:27:43.374175+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:00:04.465644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":293,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.373758+0000","last_change":"2024-08-17T17:24:53.465550+0000","last_active":"2024-08-17T17:27:43.373758+0000","last_peered":"2024-08-17T17:27:43.373758+0000","last_clean":"2024-08-17T17:27:43.373758+0000","last_became_active":"2024-08-17T17:24:53.465228+0000","last_became_peered":"2024-08-17T17:24:53.465228+0000","last_unstale":"2024-08-17T17:27:43.373758+0000","last_undegraded":"2024-08-17T17:27:43.373758+0000","last_fullsized":"2024-08-17T17:27:43.373758+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:21:18.378359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"219'11516","reported_seq":11610,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:50.512116+0000","last_change":"2024-08-17T17:24:14.640584+0000","last_active":"2024-08-17T17:27:50.512116+0000","last_peered":"2024-08-17T17:27:50.512116+0000","last_clean":"2024-08-17T17:27:50.512116+0000","last_became_active":"2024-08-17T17:24:14.640168+0000","last_became_peered":"2024-08-17T17:24:14.640168+0000","last_unstale":"2024-08-17T17:27:50.512116+0000","last_undegraded":"2024-08-17T17:27:50.512116+0000","last_fullsized":"2024-08-17T17:27:50.512116+0000","mapping_epoch":178,"log_start":"203'4800","ondisk_log_start":"203'4800","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6716,"log_dups_size":0,"ondisk_log_size":6716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:38:36.176813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58195968,"num_objects":888,"num_object_clones":0,"num_object_copies":2664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11516,"num_write_kb":396928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":318,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.374115+0000","last_change":"2024-08-17T17:24:53.464307+0000","last_active":"2024-08-17T17:27:43.374115+0000","last_peered":"2024-08-17T17:27:43.374115+0000","last_clean":"2024-08-17T17:27:43.374115+0000","last_became_active":"2024-08-17T17:24:53.463996+0000","last_became_peered":"2024-08-17T17:24:53.463996+0000","last_unstale":"2024-08-17T17:27:43.374115+0000","last_undegraded":"2024-08-17T17:27:43.374115+0000","last_fullsized":"2024-08-17T17:27:43.374115+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:04:18.993725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":646,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-08-17T17:27:42.374585+0000","last_change":"2024-08-17T17:24:09.599822+0000","last_active":"2024-08-17T17:27:42.374585+0000","last_peered":"2024-08-17T17:27:42.374585+0000","last_clean":"2024-08-17T17:27:42.374585+0000","last_became_active":"2024-08-17T17:24:09.599189+0000","last_became_peered":"2024-08-17T17:24:09.599189+0000","last_unstale":"2024-08-17T17:27:42.374585+0000","last_undegraded":"2024-08-17T17:27:42.374585+0000","last_fullsized":"2024-08-17T17:27:42.374585+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:04.465604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":277,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.419663+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:27:43.419663+0000","last_peered":"2024-08-17T17:27:43.419663+0000","last_clean":"2024-08-17T17:27:43.419663+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:27:43.419663+0000","last_undegraded":"2024-08-17T17:27:43.419663+0000","last_fullsized":"2024-08-17T17:27:43.419663+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"219'11775","reported_seq":13123,"reported_epoch":219,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-17T17:27:51.919690+0000","last_change":"2024-08-17T17:27:41.806697+0000","last_active":"2024-08-17T17:27:51.919690+0000","last_peered":"2024-08-17T17:27:51.919690+0000","last_clean":"2024-08-17T17:25:03.971607+0000","last_became_active":"2024-08-17T17:25:05.601333+0000","last_became_peered":"2024-08-17T17:25:05.601333+0000","last_unstale":"2024-08-17T17:27:51.919690+0000","last_undegraded":"2024-08-17T17:25:05.522007+0000","last_fullsized":"2024-08-17T17:25:05.499355+0000","mapping_epoch":198,"log_start":"205'5300","ondisk_log_start":"205'5300","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":6475,"log_dups_size":0,"ondisk_log_size":6475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50659328,"num_objects":773,"num_object_clones":0,"num_object_copies":2319,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2689,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11775,"num_write_kb":401536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":627,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2],"avail_no_missing":["3","2","7"],"object_location_counts":[{"shards":"2,3","objects":-1916}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":312,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.372114+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:27:43.372114+0000","last_peered":"2024-08-17T17:27:43.372114+0000","last_clean":"2024-08-17T17:27:43.372114+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:27:43.372114+0000","last_undegraded":"2024-08-17T17:27:43.372114+0000","last_fullsized":"2024-08-17T17:27:43.372114+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":832,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.373778+0000","last_change":"2024-08-17T17:23:19.633786+0000","last_active":"2024-08-17T17:27:43.373778+0000","last_peered":"2024-08-17T17:27:43.373778+0000","last_clean":"2024-08-17T17:27:43.373778+0000","last_became_active":"2024-08-17T17:23:19.633230+0000","last_became_peered":"2024-08-17T17:23:19.633230+0000","last_unstale":"2024-08-17T17:27:43.373778+0000","last_undegraded":"2024-08-17T17:27:43.373778+0000","last_fullsized":"2024-08-17T17:27:43.373778+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:30:50.377391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":309,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.370924+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:27:43.370924+0000","last_peered":"2024-08-17T17:27:43.370924+0000","last_clean":"2024-08-17T17:27:43.370924+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:27:43.370924+0000","last_undegraded":"2024-08-17T17:27:43.370924+0000","last_fullsized":"2024-08-17T17:27:43.370924+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"219'11462","reported_seq":11510,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:49.042742+0000","last_change":"2024-08-17T17:24:14.639560+0000","last_active":"2024-08-17T17:27:49.042742+0000","last_peered":"2024-08-17T17:27:49.042742+0000","last_clean":"2024-08-17T17:27:49.042742+0000","last_became_active":"2024-08-17T17:24:14.639093+0000","last_became_peered":"2024-08-17T17:24:14.639093+0000","last_unstale":"2024-08-17T17:27:49.042742+0000","last_undegraded":"2024-08-17T17:27:49.042742+0000","last_fullsized":"2024-08-17T17:27:49.042742+0000","mapping_epoch":178,"log_start":"202'4400","ondisk_log_start":"202'4400","created":178,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:24:13.606349+0000","last_clean_scrub_stamp":"2024-08-17T17:24:13.606349+0000","objects_scrubbed":0,"log_size":7062,"log_dups_size":0,"ondisk_log_size":7062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:45:57.940550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59899904,"num_objects":914,"num_object_clones":0,"num_object_copies":2742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11462,"num_write_kb":396032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":107,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.419961+0000","last_change":"2024-08-17T17:24:06.593660+0000","last_active":"2024-08-17T17:27:43.419961+0000","last_peered":"2024-08-17T17:27:43.419961+0000","last_clean":"2024-08-17T17:27:43.419961+0000","last_became_active":"2024-08-17T17:24:06.586215+0000","last_became_peered":"2024-08-17T17:24:06.586215+0000","last_unstale":"2024-08-17T17:27:43.419961+0000","last_undegraded":"2024-08-17T17:27:43.419961+0000","last_fullsized":"2024-08-17T17:27:43.419961+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:15:49.475395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":450,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-08-17T17:27:42.462088+0000","last_change":"2024-08-17T17:26:06.855307+0000","last_active":"2024-08-17T17:27:42.462088+0000","last_peered":"2024-08-17T17:27:42.462088+0000","last_clean":"2024-08-17T17:27:42.462088+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:27:42.462088+0000","last_undegraded":"2024-08-17T17:27:42.462088+0000","last_fullsized":"2024-08-17T17:27:42.462088+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":118,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-08-17T17:27:42.373555+0000","last_change":"2024-08-17T17:18:59.417183+0000","last_active":"2024-08-17T17:27:42.373555+0000","last_peered":"2024-08-17T17:27:42.373555+0000","last_clean":"2024-08-17T17:27:42.373555+0000","last_became_active":"2024-08-17T17:18:59.416843+0000","last_became_peered":"2024-08-17T17:18:59.416843+0000","last_unstale":"2024-08-17T17:27:42.373555+0000","last_undegraded":"2024-08-17T17:27:42.373555+0000","last_fullsized":"2024-08-17T17:27:42.373555+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:29:09.336335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":139,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-08-17T17:27:42.462122+0000","last_change":"2024-08-17T17:20:49.258048+0000","last_active":"2024-08-17T17:27:42.462122+0000","last_peered":"2024-08-17T17:27:42.462122+0000","last_clean":"2024-08-17T17:27:42.462122+0000","last_became_active":"2024-08-17T17:20:49.257672+0000","last_became_peered":"2024-08-17T17:20:49.257672+0000","last_unstale":"2024-08-17T17:27:42.462122+0000","last_undegraded":"2024-08-17T17:27:42.462122+0000","last_fullsized":"2024-08-17T17:27:42.462122+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:08:11.734316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":178,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-08-17T17:27:42.374316+0000","last_change":"2024-08-17T17:18:59.430969+0000","last_active":"2024-08-17T17:27:42.374316+0000","last_peered":"2024-08-17T17:27:42.374316+0000","last_clean":"2024-08-17T17:27:42.374316+0000","last_became_active":"2024-08-17T17:18:59.413475+0000","last_became_peered":"2024-08-17T17:18:59.413475+0000","last_unstale":"2024-08-17T17:27:42.374316+0000","last_undegraded":"2024-08-17T17:27:42.374316+0000","last_fullsized":"2024-08-17T17:27:42.374316+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:36:33.732645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":303,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.373460+0000","last_change":"2024-08-17T17:25:04.506413+0000","last_active":"2024-08-17T17:27:43.373460+0000","last_peered":"2024-08-17T17:27:43.373460+0000","last_clean":"2024-08-17T17:27:43.373460+0000","last_became_active":"2024-08-17T17:25:04.506104+0000","last_became_peered":"2024-08-17T17:25:04.506104+0000","last_unstale":"2024-08-17T17:27:43.373460+0000","last_undegraded":"2024-08-17T17:27:43.373460+0000","last_fullsized":"2024-08-17T17:27:43.373460+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:41:53.556223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":121,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-08-17T17:27:42.374370+0000","last_change":"2024-08-17T17:20:49.279458+0000","last_active":"2024-08-17T17:27:42.374370+0000","last_peered":"2024-08-17T17:27:42.374370+0000","last_clean":"2024-08-17T17:27:42.374370+0000","last_became_active":"2024-08-17T17:20:49.260790+0000","last_became_peered":"2024-08-17T17:20:49.260790+0000","last_unstale":"2024-08-17T17:27:42.374370+0000","last_undegraded":"2024-08-17T17:27:42.374370+0000","last_fullsized":"2024-08-17T17:27:42.374370+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:58.092945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":35,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-08-17T17:27:42.373830+0000","last_change":"2024-08-17T17:24:53.463713+0000","last_active":"2024-08-17T17:27:42.373830+0000","last_peered":"2024-08-17T17:27:42.373830+0000","last_clean":"2024-08-17T17:27:42.373830+0000","last_became_active":"2024-08-17T17:24:53.463419+0000","last_became_peered":"2024-08-17T17:24:53.463419+0000","last_unstale":"2024-08-17T17:27:42.373830+0000","last_undegraded":"2024-08-17T17:27:42.373830+0000","last_fullsized":"2024-08-17T17:27:42.373830+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:34:44.545175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":259,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.370893+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:27:43.370893+0000","last_peered":"2024-08-17T17:27:43.370893+0000","last_clean":"2024-08-17T17:27:43.370893+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:27:43.370893+0000","last_undegraded":"2024-08-17T17:27:43.370893+0000","last_fullsized":"2024-08-17T17:27:43.370893+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":286,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.420047+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:27:43.420047+0000","last_peered":"2024-08-17T17:27:43.420047+0000","last_clean":"2024-08-17T17:27:43.420047+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:27:43.420047+0000","last_undegraded":"2024-08-17T17:27:43.420047+0000","last_fullsized":"2024-08-17T17:27:43.420047+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":306,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.370854+0000","last_change":"2024-08-17T17:25:04.509422+0000","last_active":"2024-08-17T17:27:43.370854+0000","last_peered":"2024-08-17T17:27:43.370854+0000","last_clean":"2024-08-17T17:27:43.370854+0000","last_became_active":"2024-08-17T17:25:04.508742+0000","last_became_peered":"2024-08-17T17:25:04.508742+0000","last_unstale":"2024-08-17T17:27:43.370854+0000","last_undegraded":"2024-08-17T17:27:43.370854+0000","last_fullsized":"2024-08-17T17:27:43.370854+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:18:31.976907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":176,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-08-17T17:27:42.462004+0000","last_change":"2024-08-17T17:16:26.963696+0000","last_active":"2024-08-17T17:27:42.462004+0000","last_peered":"2024-08-17T17:27:42.462004+0000","last_clean":"2024-08-17T17:27:42.462004+0000","last_became_active":"2024-08-17T17:16:26.963280+0000","last_became_peered":"2024-08-17T17:16:26.963280+0000","last_unstale":"2024-08-17T17:27:42.462004+0000","last_undegraded":"2024-08-17T17:27:42.462004+0000","last_fullsized":"2024-08-17T17:27:42.462004+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:22:01.474840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":324,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.371019+0000","last_change":"2024-08-17T17:24:06.586886+0000","last_active":"2024-08-17T17:27:43.371019+0000","last_peered":"2024-08-17T17:27:43.371019+0000","last_clean":"2024-08-17T17:27:43.371019+0000","last_became_active":"2024-08-17T17:24:06.586516+0000","last_became_peered":"2024-08-17T17:24:06.586516+0000","last_unstale":"2024-08-17T17:27:43.371019+0000","last_undegraded":"2024-08-17T17:27:43.371019+0000","last_fullsized":"2024-08-17T17:27:43.371019+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:03:30.002726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":103,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.419787+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:27:43.419787+0000","last_peered":"2024-08-17T17:27:43.419787+0000","last_clean":"2024-08-17T17:27:43.419787+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:27:43.419787+0000","last_undegraded":"2024-08-17T17:27:43.419787+0000","last_fullsized":"2024-08-17T17:27:43.419787+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":283,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.373635+0000","last_change":"2024-08-17T17:18:59.412463+0000","last_active":"2024-08-17T17:27:43.373635+0000","last_peered":"2024-08-17T17:27:43.373635+0000","last_clean":"2024-08-17T17:27:43.373635+0000","last_became_active":"2024-08-17T17:18:59.412134+0000","last_became_peered":"2024-08-17T17:18:59.412134+0000","last_unstale":"2024-08-17T17:27:43.373635+0000","last_undegraded":"2024-08-17T17:27:43.373635+0000","last_fullsized":"2024-08-17T17:27:43.373635+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:44.463153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":327,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.371627+0000","last_change":"2024-08-17T17:25:04.510854+0000","last_active":"2024-08-17T17:27:43.371627+0000","last_peered":"2024-08-17T17:27:43.371627+0000","last_clean":"2024-08-17T17:27:43.371627+0000","last_became_active":"2024-08-17T17:25:04.510462+0000","last_became_peered":"2024-08-17T17:25:04.510462+0000","last_unstale":"2024-08-17T17:27:43.371627+0000","last_undegraded":"2024-08-17T17:27:43.371627+0000","last_fullsized":"2024-08-17T17:27:43.371627+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:55:54.762086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":224,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.373855+0000","last_change":"2024-08-17T17:24:06.591165+0000","last_active":"2024-08-17T17:27:43.373855+0000","last_peered":"2024-08-17T17:27:43.373855+0000","last_clean":"2024-08-17T17:27:43.373855+0000","last_became_active":"2024-08-17T17:24:06.590785+0000","last_became_peered":"2024-08-17T17:24:06.590785+0000","last_unstale":"2024-08-17T17:27:43.373855+0000","last_undegraded":"2024-08-17T17:27:43.373855+0000","last_fullsized":"2024-08-17T17:27:43.373855+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:54:05.775563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":296,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.374255+0000","last_change":"2024-08-17T17:24:32.300560+0000","last_active":"2024-08-17T17:27:43.374255+0000","last_peered":"2024-08-17T17:27:43.374255+0000","last_clean":"2024-08-17T17:27:43.374255+0000","last_became_active":"2024-08-17T17:24:32.300228+0000","last_became_peered":"2024-08-17T17:24:32.300228+0000","last_unstale":"2024-08-17T17:27:43.374255+0000","last_undegraded":"2024-08-17T17:27:43.374255+0000","last_fullsized":"2024-08-17T17:27:43.374255+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:18:05.475449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":103,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.419475+0000","last_change":"2024-08-17T17:24:06.586379+0000","last_active":"2024-08-17T17:27:43.419475+0000","last_peered":"2024-08-17T17:27:43.419475+0000","last_clean":"2024-08-17T17:27:43.419475+0000","last_became_active":"2024-08-17T17:24:06.586023+0000","last_became_peered":"2024-08-17T17:24:06.586023+0000","last_unstale":"2024-08-17T17:27:43.419475+0000","last_undegraded":"2024-08-17T17:27:43.419475+0000","last_fullsized":"2024-08-17T17:27:43.419475+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:45.701637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":313,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.371516+0000","last_change":"2024-08-17T17:21:50.442535+0000","last_active":"2024-08-17T17:27:43.371516+0000","last_peered":"2024-08-17T17:27:43.371516+0000","last_clean":"2024-08-17T17:27:43.371516+0000","last_became_active":"2024-08-17T17:21:50.441891+0000","last_became_peered":"2024-08-17T17:21:50.441891+0000","last_unstale":"2024-08-17T17:27:43.371516+0000","last_undegraded":"2024-08-17T17:27:43.371516+0000","last_fullsized":"2024-08-17T17:27:43.371516+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:46:24.025407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":231,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-08-17T17:27:42.373706+0000","last_change":"2024-08-17T17:25:04.508045+0000","last_active":"2024-08-17T17:27:42.373706+0000","last_peered":"2024-08-17T17:27:42.373706+0000","last_clean":"2024-08-17T17:27:42.373706+0000","last_became_active":"2024-08-17T17:25:04.505083+0000","last_became_peered":"2024-08-17T17:25:04.505083+0000","last_unstale":"2024-08-17T17:27:42.373706+0000","last_undegraded":"2024-08-17T17:27:42.373706+0000","last_fullsized":"2024-08-17T17:27:42.373706+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:23:42.993544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":299,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.373071+0000","last_change":"2024-08-17T17:18:59.418434+0000","last_active":"2024-08-17T17:27:43.373071+0000","last_peered":"2024-08-17T17:27:43.373071+0000","last_clean":"2024-08-17T17:27:43.373071+0000","last_became_active":"2024-08-17T17:18:59.418043+0000","last_became_peered":"2024-08-17T17:18:59.418043+0000","last_unstale":"2024-08-17T17:27:43.373071+0000","last_undegraded":"2024-08-17T17:27:43.373071+0000","last_fullsized":"2024-08-17T17:27:43.373071+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:25:51.371820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":204,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-08-17T17:27:42.373763+0000","last_change":"2024-08-17T17:20:49.291167+0000","last_active":"2024-08-17T17:27:42.373763+0000","last_peered":"2024-08-17T17:27:42.373763+0000","last_clean":"2024-08-17T17:27:42.373763+0000","last_became_active":"2024-08-17T17:20:49.263750+0000","last_became_peered":"2024-08-17T17:20:49.263750+0000","last_unstale":"2024-08-17T17:27:42.373763+0000","last_undegraded":"2024-08-17T17:27:42.373763+0000","last_fullsized":"2024-08-17T17:27:42.373763+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:02.863246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":313,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.371352+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:27:43.371352+0000","last_peered":"2024-08-17T17:27:43.371352+0000","last_clean":"2024-08-17T17:27:43.371352+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:27:43.371352+0000","last_undegraded":"2024-08-17T17:27:43.371352+0000","last_fullsized":"2024-08-17T17:27:43.371352+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":313,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.372055+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:27:43.372055+0000","last_peered":"2024-08-17T17:27:43.372055+0000","last_clean":"2024-08-17T17:27:43.372055+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:27:43.372055+0000","last_undegraded":"2024-08-17T17:27:43.372055+0000","last_fullsized":"2024-08-17T17:27:43.372055+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":31,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-08-17T17:27:42.461549+0000","last_change":"2024-08-17T17:25:04.503834+0000","last_active":"2024-08-17T17:27:42.461549+0000","last_peered":"2024-08-17T17:27:42.461549+0000","last_clean":"2024-08-17T17:27:42.461549+0000","last_became_active":"2024-08-17T17:25:04.502712+0000","last_became_peered":"2024-08-17T17:25:04.502712+0000","last_unstale":"2024-08-17T17:27:42.461549+0000","last_undegraded":"2024-08-17T17:27:42.461549+0000","last_fullsized":"2024-08-17T17:27:42.461549+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:45.389103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":122,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-08-17T17:27:43.419956+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:27:43.419956+0000","last_peered":"2024-08-17T17:27:43.419956+0000","last_clean":"2024-08-17T17:27:43.419956+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:27:43.419956+0000","last_undegraded":"2024-08-17T17:27:43.419956+0000","last_fullsized":"2024-08-17T17:27:43.419956+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":831258648,"num_objects":12685,"num_object_clones":0,"num_object_copies":38055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3715,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12685,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":184609,"num_write_kb":6313345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17203,"num_bytes_recovered":764084224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3374788608,"data_stored":3374776392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119809,"ondisk_log_size":119809,"up":48,"acting":46,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":6304,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":16702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17971,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16789504,"data_stored":16777273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234252,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1251608,"kb_used_data":941776,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":92493544,"statfs":{"total":95995035648,"available":94713389056,"internally_reserved":0,"allocated":964378624,"data_stored":963106523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495887,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":796304,"kb_used_data":490632,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92948848,"statfs":{"total":95995035648,"available":95179620352,"internally_reserved":0,"allocated":502407168,"data_stored":501142959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757525,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":585800,"kb_used_data":280128,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93159352,"statfs":{"total":95995035648,"available":95395176448,"internally_reserved":0,"allocated":286851072,"data_stored":285590534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019163,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435390,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":833408,"kb_used_data":518456,"kb_used_omap":4,"kb_used_meta":314939,"kb_avail":92911744,"statfs":{"total":95995035648,"available":95141625856,"internally_reserved":0,"allocated":530898944,"data_stored":529646348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":322497546},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":104,"seq":446676598916,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":811216,"kb_used_data":470344,"kb_used_omap":7,"kb_used_meta":340856,"kb_avail":92933936,"statfs":{"total":95995035648,"available":95164350464,"internally_reserved":0,"allocated":481632256,"data_stored":480368542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":349036706},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762443,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":870408,"kb_used_data":504128,"kb_used_omap":7,"kb_used_meta":366264,"kb_avail":92874744,"statfs":{"total":95995035648,"available":95103737856,"internally_reserved":0,"allocated":516227072,"data_stored":514966275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7945,"internal_metadata":375054583},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":94,"seq":403726925970,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":495952,"kb_used_data":193992,"kb_used_omap":7,"kb_used_meta":301944,"kb_avail":93249200,"statfs":{"total":95995035648,"available":95487180800,"internally_reserved":0,"allocated":198647808,"data_stored":197380004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":309190977},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6241,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":182718464,"data_stored":182714392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":503058432,"data_stored":503054360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":471859200,"data_stored":471859200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":512294912,"data_stored":512294912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266866688,"data_stored":266862616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":491782144,"data_stored":491782144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":946208768,"data_stored":946208768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:27:56.009 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph status --format=json 2024-08-17T17:27:56.055 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:27:56.080 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:56.150 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:56.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:56.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:56.253 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:56.309 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:56.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:56.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:56.344 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:27:56.411 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:56.440 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:56.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:56.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:56.542 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:56.588 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:56.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:56.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:56.690 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:56.720 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:56.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:56.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:56.822 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:27:56.864 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:56.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:56.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:27:56.965 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:56.996 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:56.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:56.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:57.099 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:57.126 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:57.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:57.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:57.228 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:57.259 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:57.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:57.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:57.361 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:27:57.395 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:57.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:57.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:27:57.450 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-08-17T17:27:57.464 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:27:57.497 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:57.529 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:57.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:57.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:57.553 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:57 smithi053 ceph-mon[89963]: pgmap v591: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 704 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 738 op/s; 3715/33678 objects degraded (11.031%); 0 B/s, 66 objects/s recovering 2024-08-17T17:27:57.553 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:57 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2384292903' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:27:57.631 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:57.663 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:57.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:57.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:57.702 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:57 smithi003 ceph-mon[103344]: pgmap v591: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 704 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 738 op/s; 3715/33678 objects degraded (11.031%); 0 B/s, 66 objects/s recovering 2024-08-17T17:27:57.702 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:57 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2384292903' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:27:57.703 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:57 smithi003 ceph-mon[100615]: pgmap v591: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 704 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 738 op/s; 3715/33678 objects degraded (11.031%); 0 B/s, 66 objects/s recovering 2024-08-17T17:27:57.703 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:57 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2384292903' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:27:57.765 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:57.808 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:57.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:57.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:57.910 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:57.957 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:57.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:57.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:57.964 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:27:57.964 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2024-08-17T17:27:57.964 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2024-08-17T17:27:57.964 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2024-08-17T17:27:57.964 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:27:58.059 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:58.088 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:58.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:58.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:58.191 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:58.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:58.229 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:58.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:58.330 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:58.360 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:58.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:58.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:58.462 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:58.499 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:58.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:58.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:27:58.601 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:58.631 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:58.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:58.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:58.636 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:58 smithi053 ceph-mon[89963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:27:58.636 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:58 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2384292903' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:27:58.637 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:58 smithi053 ceph-mon[89963]: osdmap e220: 8 total, 8 up, 7 in 2024-08-17T17:27:58.662 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-17T17:27:58.733 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:58.770 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:58.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:58.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:58.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:58 smithi003 ceph-mon[103344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:27:58.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:58 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2384292903' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:27:58.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:58 smithi003 ceph-mon[103344]: osdmap e220: 8 total, 8 up, 7 in 2024-08-17T17:27:58.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:58 smithi003 ceph-mon[100615]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:27:58.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:58 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2384292903' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:27:58.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:58 smithi003 ceph-mon[100615]: osdmap e220: 8 total, 8 up, 7 in 2024-08-17T17:27:58.872 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:27:58.899 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:58.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:58.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:27:59.000 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:59.040 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:59.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:59.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:59.060 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:27:59.142 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:59.174 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:59.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:59.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:59.276 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:27:59.302 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:59.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:59.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:27:59.404 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:59.450 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:59.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:59.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:59.552 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:27:59.581 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:27:59.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:59.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:27:59.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:27:59.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:27:59 smithi053 ceph-mon[89963]: pgmap v593: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 626 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 803 op/s; 3486/29904 objects degraded (11.657%); 0 B/s, 46 objects/s recovering 2024-08-17T17:27:59.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:59.718 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:59.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:27:59.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:27:59 smithi003 ceph-mon[100615]: pgmap v593: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 626 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 803 op/s; 3486/29904 objects degraded (11.657%); 0 B/s, 46 objects/s recovering 2024-08-17T17:27:59.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:27:59 smithi003 ceph-mon[103344]: pgmap v593: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 626 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 803 op/s; 3486/29904 objects degraded (11.657%); 0 B/s, 46 objects/s recovering 2024-08-17T17:27:59.819 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:27:59.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:59.860 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:59.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:27:59.961 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:27:59.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:27:59.985 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:27:59.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:00.086 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:00.109 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:00.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:00.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:00.212 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:00.234 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:00.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:00.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:00.336 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:00.375 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:00.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:00.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:00.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:28:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:28:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:28:00.477 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:00.507 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:00.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:00.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:00.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:00.635 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:00.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:00.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:00.736 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:00 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 3486/29904 objects degraded (11.657%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:28:00.738 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:00.761 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:00.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:00.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:00.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:00 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 3486/29904 objects degraded (11.657%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:28:00.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:00 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 3486/29904 objects degraded (11.657%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:28:00.863 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:00.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:00.899 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:00.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:01.000 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:01.027 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:01.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:01.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:01.129 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:01.168 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:01.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:01.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:01.173 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:28:01.173 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2024-08-17T17:28:01.173 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2024-08-17T17:28:01.173 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2024-08-17T17:28:01.173 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.173 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:15020 5.1 5:85f3cc14:::benchmark_data_smithi003_140793_object15019:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-17T17:28:01.173 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:19:17.145842+0000", 2024-08-17T17:28:01.173 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 524.02312776999997, 2024-08-17T17:28:01.174 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.014211249, 2024-08-17T17:28:01.174 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:01.174 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:01.174 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:01.174 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:28:01.174 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:28:01.174 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 15020 2024-08-17T17:28:01.174 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.174 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:01.174 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.174 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:01.174 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.145842+0000", 2024-08-17T17:28:01.175 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.175 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.175 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.175 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:01.175 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.145842+0000", 2024-08-17T17:28:01.175 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.178 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.178 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.178 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:01.178 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.145841+0000", 2024-08-17T17:28:01.178 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:28:01.178 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.178 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.178 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:01.178 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.145900+0000", 2024-08-17T17:28:01.179 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9515000000000002e-05 2024-08-17T17:28:01.179 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.179 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.179 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:01.179 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.145922+0000", 2024-08-17T17:28:01.179 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1715000000000001e-05 2024-08-17T17:28:01.179 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.179 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.179 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:01.179 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.145926+0000", 2024-08-17T17:28:01.179 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1910000000000002e-06 2024-08-17T17:28:01.179 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.179 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.179 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:01.180 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.145976+0000", 2024-08-17T17:28:01.180 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9359e-05 2024-08-17T17:28:01.180 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.180 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.180 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:01.180 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.146116+0000", 2024-08-17T17:28:01.180 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014038599999999999 2024-08-17T17:28:01.180 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.180 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.180 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,3", 2024-08-17T17:28:01.180 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.146238+0000", 2024-08-17T17:28:01.180 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000121623 2024-08-17T17:28:01.180 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.181 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.181 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:01.181 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.146884+0000", 2024-08-17T17:28:01.181 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00064646300000000001 2024-08-17T17:28:01.181 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.181 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.181 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:01.181 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.147453+0000", 2024-08-17T17:28:01.181 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00056866700000000002 2024-08-17T17:28:01.181 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.181 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.181 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:01.181 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.159991+0000", 2024-08-17T17:28:01.181 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012538046000000001 2024-08-17T17:28:01.181 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.182 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.182 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:01.182 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.160019+0000", 2024-08-17T17:28:01.182 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8487999999999999e-05 2024-08-17T17:28:01.182 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.182 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.182 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:01.182 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.160053+0000", 2024-08-17T17:28:01.182 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4282999999999997e-05 2024-08-17T17:28:01.183 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.183 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:01.183 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.183 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.183 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.183 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:15226 5.1 5:8f8e69e6:::benchmark_data_smithi003_140793_object15225:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-17T17:28:01.183 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:19:17.237585+0000", 2024-08-17T17:28:01.183 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 523.93138538300002, 2024-08-17T17:28:01.183 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.014643912, 2024-08-17T17:28:01.183 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:01.183 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:01.183 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:01.183 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:28:01.184 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:28:01.184 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 15226 2024-08-17T17:28:01.184 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.184 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:01.184 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.184 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:01.184 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.237585+0000", 2024-08-17T17:28:01.184 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.184 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.184 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.184 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:01.184 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.237585+0000", 2024-08-17T17:28:01.184 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.184 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.184 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.185 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:01.185 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.237583+0000", 2024-08-17T17:28:01.185 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:28:01.185 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.185 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.185 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:01.185 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.237642+0000", 2024-08-17T17:28:01.185 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8810000000000001e-05 2024-08-17T17:28:01.185 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.185 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.185 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:01.185 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.237658+0000", 2024-08-17T17:28:01.185 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6226e-05 2024-08-17T17:28:01.186 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.186 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.186 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:01.186 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.237665+0000", 2024-08-17T17:28:01.186 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4309999999999999e-06 2024-08-17T17:28:01.186 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.186 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.186 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:01.186 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.237720+0000", 2024-08-17T17:28:01.186 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5862000000000001e-05 2024-08-17T17:28:01.186 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.186 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.186 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:01.187 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.237848+0000", 2024-08-17T17:28:01.187 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000127179 2024-08-17T17:28:01.187 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.187 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.187 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,3", 2024-08-17T17:28:01.187 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.237984+0000", 2024-08-17T17:28:01.187 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000135964 2024-08-17T17:28:01.187 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.187 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.187 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:01.187 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.239525+0000", 2024-08-17T17:28:01.187 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001540912 2024-08-17T17:28:01.187 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.187 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.188 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:01.188 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.242534+0000", 2024-08-17T17:28:01.188 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0030097209999999999 2024-08-17T17:28:01.188 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.188 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.188 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:01.188 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.252182+0000", 2024-08-17T17:28:01.188 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0096481009999999992 2024-08-17T17:28:01.188 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.188 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.188 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:01.188 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.252209+0000", 2024-08-17T17:28:01.188 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6815e-05 2024-08-17T17:28:01.189 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.189 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.189 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:01.189 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.252229+0000", 2024-08-17T17:28:01.189 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.933e-05 2024-08-17T17:28:01.189 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.189 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:01.189 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.189 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.189 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.189 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:15233 5.1 5:8af853be:::benchmark_data_smithi003_140793_object15232:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-08-17T17:28:01.189 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:19:17.245633+0000", 2024-08-17T17:28:01.189 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 523.92333661500004, 2024-08-17T17:28:01.189 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.014090702, 2024-08-17T17:28:01.190 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:01.190 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:01.190 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:01.190 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:28:01.190 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:28:01.191 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 15233 2024-08-17T17:28:01.191 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.191 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:01.191 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.191 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:01.191 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.245633+0000", 2024-08-17T17:28:01.191 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.191 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.191 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.192 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:01.192 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.245633+0000", 2024-08-17T17:28:01.192 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.192 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.192 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.192 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:01.192 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.245624+0000", 2024-08-17T17:28:01.192 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.99999 2024-08-17T17:28:01.192 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.192 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.192 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:01.192 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.245665+0000", 2024-08-17T17:28:01.192 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1904999999999999e-05 2024-08-17T17:28:01.193 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.193 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.193 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:01.193 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.245682+0000", 2024-08-17T17:28:01.193 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6084999999999999e-05 2024-08-17T17:28:01.193 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.193 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.193 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:01.193 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.245688+0000", 2024-08-17T17:28:01.193 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4980000000000003e-06 2024-08-17T17:28:01.193 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.193 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.193 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:01.193 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.245724+0000", 2024-08-17T17:28:01.194 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5451000000000003e-05 2024-08-17T17:28:01.194 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.194 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.194 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:01.194 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.245807+0000", 2024-08-17T17:28:01.194 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.3698999999999999e-05 2024-08-17T17:28:01.194 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.194 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.194 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,3", 2024-08-17T17:28:01.194 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.245932+0000", 2024-08-17T17:28:01.194 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012473200000000001 2024-08-17T17:28:01.194 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.195 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.195 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:01.195 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.246707+0000", 2024-08-17T17:28:01.195 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00077458799999999999 2024-08-17T17:28:01.195 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.195 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.195 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:01.195 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.251998+0000", 2024-08-17T17:28:01.195 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0052913839999999997 2024-08-17T17:28:01.195 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.195 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.195 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:01.195 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.259670+0000", 2024-08-17T17:28:01.195 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.007672295 2024-08-17T17:28:01.196 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.196 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.196 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:01.196 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.259701+0000", 2024-08-17T17:28:01.196 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1066999999999999e-05 2024-08-17T17:28:01.196 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.196 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.196 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:01.196 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:17.259724+0000", 2024-08-17T17:28:01.196 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2785e-05 2024-08-17T17:28:01.196 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.196 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:01.196 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.196 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.196 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.197 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:25002 5.1 5:84c6bd71:::benchmark_data_smithi003_140793_object25001:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-08-17T17:28:01.197 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:19:21.727993+0000", 2024-08-17T17:28:01.197 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 519.44097740699999, 2024-08-17T17:28:01.197 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012271251, 2024-08-17T17:28:01.197 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:01.197 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:01.197 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:01.197 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:28:01.197 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:28:01.197 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 25002 2024-08-17T17:28:01.197 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.197 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:01.197 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.197 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:01.198 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:21.727993+0000", 2024-08-17T17:28:01.198 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.198 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.198 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.198 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:01.198 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:21.727993+0000", 2024-08-17T17:28:01.198 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.198 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.198 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.198 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:01.198 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:21.727992+0000", 2024-08-17T17:28:01.198 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:28:01.198 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.198 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.199 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:01.199 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:21.728049+0000", 2024-08-17T17:28:01.199 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7445000000000003e-05 2024-08-17T17:28:01.199 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.199 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.199 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:01.199 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:21.728059+0000", 2024-08-17T17:28:01.199 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0159e-05 2024-08-17T17:28:01.199 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.199 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.199 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:01.199 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:21.728063+0000", 2024-08-17T17:28:01.199 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.844e-06 2024-08-17T17:28:01.199 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.200 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.200 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:01.200 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:21.728105+0000", 2024-08-17T17:28:01.200 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1975999999999999e-05 2024-08-17T17:28:01.200 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.200 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.200 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:01.200 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:21.728260+0000", 2024-08-17T17:28:01.200 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015447099999999999 2024-08-17T17:28:01.200 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.200 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.200 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,3", 2024-08-17T17:28:01.200 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:21.728401+0000", 2024-08-17T17:28:01.200 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014178300000000001 2024-08-17T17:28:01.201 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.201 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.201 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:01.201 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:21.729329+0000", 2024-08-17T17:28:01.201 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00092803299999999998 2024-08-17T17:28:01.201 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.201 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.201 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:01.201 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:21.733074+0000", 2024-08-17T17:28:01.201 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0037445740000000001 2024-08-17T17:28:01.201 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.201 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.201 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:01.201 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:21.740203+0000", 2024-08-17T17:28:01.202 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0071293069999999997 2024-08-17T17:28:01.202 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.202 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.202 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:01.202 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:21.740235+0000", 2024-08-17T17:28:01.202 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1427999999999998e-05 2024-08-17T17:28:01.202 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.202 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.202 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:01.202 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:21.740264+0000", 2024-08-17T17:28:01.202 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9179999999999998e-05 2024-08-17T17:28:01.202 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.202 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:01.203 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.203 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.203 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.203 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:26037 5.1 5:8e2cbde0:::benchmark_data_smithi003_140793_object26036:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-08-17T17:28:01.203 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:19:22.258546+0000", 2024-08-17T17:28:01.203 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 518.91042414900005, 2024-08-17T17:28:01.203 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.056420265999999997, 2024-08-17T17:28:01.203 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:01.203 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:01.203 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:01.203 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:28:01.203 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:28:01.203 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 26037 2024-08-17T17:28:01.203 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.204 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:01.204 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.204 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:01.204 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:22.258546+0000", 2024-08-17T17:28:01.204 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.204 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.204 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.204 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:01.204 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:22.258546+0000", 2024-08-17T17:28:01.204 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.204 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.204 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.204 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:01.204 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:22.258545+0000", 2024-08-17T17:28:01.205 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:28:01.205 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.205 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.205 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:01.205 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:22.258602+0000", 2024-08-17T17:28:01.205 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7116e-05 2024-08-17T17:28:01.205 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.205 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.205 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:01.205 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:22.258614+0000", 2024-08-17T17:28:01.205 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1685000000000001e-05 2024-08-17T17:28:01.205 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.205 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.206 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:01.206 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:22.258620+0000", 2024-08-17T17:28:01.206 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2060000000000004e-06 2024-08-17T17:28:01.206 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.206 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.206 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:01.206 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:22.258653+0000", 2024-08-17T17:28:01.206 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3030000000000001e-05 2024-08-17T17:28:01.206 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.206 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.206 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:01.206 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:22.258731+0000", 2024-08-17T17:28:01.207 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.8131000000000001e-05 2024-08-17T17:28:01.207 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.207 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.207 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,3", 2024-08-17T17:28:01.207 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:22.258807+0000", 2024-08-17T17:28:01.207 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.6297000000000005e-05 2024-08-17T17:28:01.207 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.207 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.207 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:01.207 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:22.259638+0000", 2024-08-17T17:28:01.207 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00083096199999999998 2024-08-17T17:28:01.207 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.207 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.207 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:01.208 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:22.262794+0000", 2024-08-17T17:28:01.208 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0031562399999999998 2024-08-17T17:28:01.208 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.208 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.208 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:01.208 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:22.314851+0000", 2024-08-17T17:28:01.208 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.052056142 2024-08-17T17:28:01.208 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.208 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.208 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:01.208 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:22.314917+0000", 2024-08-17T17:28:01.208 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5988999999999994e-05 2024-08-17T17:28:01.208 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.208 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.209 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:01.209 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:22.314966+0000", 2024-08-17T17:28:01.209 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9573999999999997e-05 2024-08-17T17:28:01.209 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.209 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:01.209 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.209 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.209 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.209 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.35277.0:84085 5.0 e137/111 5:0af5c770:::benchmark_data_smithi003_140793_object84084:head v 137'5232, mlcod=137'5229)", 2024-08-17T17:28:01.209 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:20:18.940473+0000", 2024-08-17T17:28:01.209 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 462.22849708500001, 2024-08-17T17:28:01.209 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.013070686999999999, 2024-08-17T17:28:01.209 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:01.210 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:01.210 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:01.210 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.210 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:01.210 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:18.940473+0000", 2024-08-17T17:28:01.210 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.210 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.210 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.210 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:01.210 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:18.940473+0000", 2024-08-17T17:28:01.210 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.210 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.210 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.210 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:01.211 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:18.940471+0000", 2024-08-17T17:28:01.211 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:28:01.211 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.211 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.211 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:01.211 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:18.940515+0000", 2024-08-17T17:28:01.211 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3695000000000001e-05 2024-08-17T17:28:01.211 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.211 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.211 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:01.211 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:18.940537+0000", 2024-08-17T17:28:01.211 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1849999999999999e-05 2024-08-17T17:28:01.211 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.212 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.212 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:01.212 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:18.940544+0000", 2024-08-17T17:28:01.212 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.1300000000000003e-06 2024-08-17T17:28:01.212 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.212 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.212 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:01.212 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:18.940604+0000", 2024-08-17T17:28:01.212 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0149000000000003e-05 2024-08-17T17:28:01.212 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.212 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.212 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:01.212 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:18.940624+0000", 2024-08-17T17:28:01.213 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0146999999999999e-05 2024-08-17T17:28:01.213 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.213 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.213 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:01.213 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:18.953505+0000", 2024-08-17T17:28:01.213 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.01288102 2024-08-17T17:28:01.213 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.213 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.213 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:01.213 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:18.953544+0000", 2024-08-17T17:28:01.213 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8340000000000002e-05 2024-08-17T17:28:01.213 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.213 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:01.213 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.214 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.214 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.214 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.35277.0:84480 5.c e137/111 5:30fd82b5:::benchmark_data_smithi003_140793_object84479:head v 137'5238, mlcod=137'5234)", 2024-08-17T17:28:01.214 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:20:19.372990+0000", 2024-08-17T17:28:01.214 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 461.79598028499998, 2024-08-17T17:28:01.214 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012034385, 2024-08-17T17:28:01.214 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:01.214 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:01.214 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:01.214 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.214 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:01.214 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:19.372990+0000", 2024-08-17T17:28:01.214 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.214 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.215 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.215 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:01.215 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:19.372990+0000", 2024-08-17T17:28:01.215 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.215 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.215 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.215 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:01.215 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:19.372989+0000", 2024-08-17T17:28:01.215 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:28:01.215 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.215 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.215 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:01.215 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:19.373061+0000", 2024-08-17T17:28:01.215 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.2719999999999995e-05 2024-08-17T17:28:01.216 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.216 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.216 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:01.216 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:19.373079+0000", 2024-08-17T17:28:01.216 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7833e-05 2024-08-17T17:28:01.216 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.216 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.216 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:01.216 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:19.373086+0000", 2024-08-17T17:28:01.216 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5610000000000004e-06 2024-08-17T17:28:01.216 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.216 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.216 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:01.216 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:19.373128+0000", 2024-08-17T17:28:01.217 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2447999999999997e-05 2024-08-17T17:28:01.217 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.217 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.217 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:01.217 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:19.373147+0000", 2024-08-17T17:28:01.217 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.8377e-05 2024-08-17T17:28:01.217 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.217 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.217 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:01.217 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:19.384981+0000", 2024-08-17T17:28:01.217 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011834269999999999 2024-08-17T17:28:01.217 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.217 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.218 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:01.218 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:19.385024+0000", 2024-08-17T17:28:01.218 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3189000000000003e-05 2024-08-17T17:28:01.218 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.218 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:01.218 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.218 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.218 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.218 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.35277.0:84552 5.9 e137/130 5:9a0c6aa7:::benchmark_data_smithi003_140793_object84551:head v 137'5246, mlcod=0'0)", 2024-08-17T17:28:01.218 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:20:19.464918+0000", 2024-08-17T17:28:01.218 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 461.70405246799999, 2024-08-17T17:28:01.218 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012072039999999999, 2024-08-17T17:28:01.218 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:01.218 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:01.219 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:01.219 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.219 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:01.219 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:19.464918+0000", 2024-08-17T17:28:01.219 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.219 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.219 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.219 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:01.219 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:19.464918+0000", 2024-08-17T17:28:01.219 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.219 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.219 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.219 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:01.220 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:19.464917+0000", 2024-08-17T17:28:01.220 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:28:01.220 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.220 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.220 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:01.220 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:19.464944+0000", 2024-08-17T17:28:01.220 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7382999999999999e-05 2024-08-17T17:28:01.220 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.220 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.220 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:01.220 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:19.464954+0000", 2024-08-17T17:28:01.220 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0217999999999999e-05 2024-08-17T17:28:01.220 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.221 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.221 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:01.221 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:19.464960+0000", 2024-08-17T17:28:01.221 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4500000000000003e-06 2024-08-17T17:28:01.221 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.221 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.221 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:01.221 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:19.465088+0000", 2024-08-17T17:28:01.221 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000128718 2024-08-17T17:28:01.221 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.221 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.221 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:01.221 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:19.465113+0000", 2024-08-17T17:28:01.222 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4779000000000001e-05 2024-08-17T17:28:01.222 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.222 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.222 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:01.222 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:19.473501+0000", 2024-08-17T17:28:01.222 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0083876019999999992 2024-08-17T17:28:01.222 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.222 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.222 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:01.222 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:19.476990+0000", 2024-08-17T17:28:01.222 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0034887 2024-08-17T17:28:01.222 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.222 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:01.223 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.223 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.223 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.223 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.35277.0:84609 5.c e137/111 5:3aa788f9:::benchmark_data_smithi003_140793_object84608:head v 137'5245, mlcod=137'5243)", 2024-08-17T17:28:01.223 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:20:19.517041+0000", 2024-08-17T17:28:01.223 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 461.65192872699998, 2024-08-17T17:28:01.223 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.013698333, 2024-08-17T17:28:01.223 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:01.223 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:01.223 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:01.223 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.223 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:01.223 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:19.517041+0000", 2024-08-17T17:28:01.223 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.224 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.224 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.224 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:01.224 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:19.517041+0000", 2024-08-17T17:28:01.224 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.224 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.224 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.224 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:01.224 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:19.517041+0000", 2024-08-17T17:28:01.224 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:28:01.224 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.224 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.224 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:01.224 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:19.517101+0000", 2024-08-17T17:28:01.225 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0591999999999999e-05 2024-08-17T17:28:01.225 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.225 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.225 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:01.225 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:19.517119+0000", 2024-08-17T17:28:01.225 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7631000000000001e-05 2024-08-17T17:28:01.225 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.225 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.225 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:01.225 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:19.517125+0000", 2024-08-17T17:28:01.225 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7409999999999998e-06 2024-08-17T17:28:01.225 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.225 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.225 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:01.226 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:19.517183+0000", 2024-08-17T17:28:01.226 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8727999999999997e-05 2024-08-17T17:28:01.226 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.226 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.226 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:01.226 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:19.517202+0000", 2024-08-17T17:28:01.226 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.8944999999999999e-05 2024-08-17T17:28:01.226 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.226 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.226 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:01.226 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:19.530646+0000", 2024-08-17T17:28:01.226 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.013444255 2024-08-17T17:28:01.226 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.227 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.227 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:01.227 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:19.530740+0000", 2024-08-17T17:28:01.227 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.3095000000000002e-05 2024-08-17T17:28:01.227 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.227 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:01.227 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.227 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.227 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.227 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.35277.0:87105 5.c e137/111 5:3b49071d:::benchmark_data_smithi003_140793_object87104:head v 137'5412, mlcod=137'5409)", 2024-08-17T17:28:01.228 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:20:22.511829+0000", 2024-08-17T17:28:01.228 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 458.65714069000001, 2024-08-17T17:28:01.228 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011880594, 2024-08-17T17:28:01.228 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:01.228 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:01.228 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:01.228 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.228 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:01.228 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:22.511829+0000", 2024-08-17T17:28:01.229 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.229 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.229 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.229 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:01.229 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:22.511829+0000", 2024-08-17T17:28:01.229 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.229 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.229 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.229 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:01.229 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:22.511828+0000", 2024-08-17T17:28:01.230 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:28:01.230 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.230 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.230 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:01.230 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:22.511933+0000", 2024-08-17T17:28:01.230 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000104555 2024-08-17T17:28:01.230 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.230 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.230 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:01.230 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:22.511949+0000", 2024-08-17T17:28:01.231 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.594e-05 2024-08-17T17:28:01.231 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.231 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.231 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:01.231 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:22.511955+0000", 2024-08-17T17:28:01.231 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.891e-06 2024-08-17T17:28:01.231 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.231 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.231 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:01.231 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:22.512165+0000", 2024-08-17T17:28:01.231 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00020991 2024-08-17T17:28:01.231 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.231 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.232 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:01.232 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:22.512215+0000", 2024-08-17T17:28:01.232 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9903e-05 2024-08-17T17:28:01.232 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.232 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.232 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:01.232 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:22.523665+0000", 2024-08-17T17:28:01.232 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011450297 2024-08-17T17:28:01.232 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.232 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.232 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:01.233 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:22.523710+0000", 2024-08-17T17:28:01.233 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5012999999999997e-05 2024-08-17T17:28:01.233 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.233 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:01.233 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.233 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.233 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.233 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.35277.0:90798 5.9 e138/130 5:95a1b3a8:::benchmark_data_smithi003_140793_object90797:head v 138'5610, mlcod=0'0)", 2024-08-17T17:28:01.233 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:20:27.252253+0000", 2024-08-17T17:28:01.233 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 453.91671741599998, 2024-08-17T17:28:01.233 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012475096, 2024-08-17T17:28:01.233 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:01.233 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:01.233 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:01.234 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.234 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:01.234 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:27.252253+0000", 2024-08-17T17:28:01.234 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.234 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.234 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.234 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:01.234 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:27.252253+0000", 2024-08-17T17:28:01.234 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.234 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.234 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.234 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:01.234 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:27.252252+0000", 2024-08-17T17:28:01.234 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:28:01.235 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.235 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.236 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:01.236 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:27.252281+0000", 2024-08-17T17:28:01.236 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9006e-05 2024-08-17T17:28:01.236 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.236 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.236 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:01.236 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:27.252299+0000", 2024-08-17T17:28:01.236 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.8386e-05 2024-08-17T17:28:01.237 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.237 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.237 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:01.237 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:27.252305+0000", 2024-08-17T17:28:01.237 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3779999999999998e-06 2024-08-17T17:28:01.237 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.237 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.237 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:01.237 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:27.252411+0000", 2024-08-17T17:28:01.237 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010584799999999999 2024-08-17T17:28:01.237 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.237 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.237 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:01.237 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:27.252436+0000", 2024-08-17T17:28:01.238 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4677000000000001e-05 2024-08-17T17:28:01.238 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.238 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.238 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:01.238 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:27.264685+0000", 2024-08-17T17:28:01.238 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012249042999999999 2024-08-17T17:28:01.238 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.238 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.238 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:01.238 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:27.264728+0000", 2024-08-17T17:28:01.238 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2784000000000002e-05 2024-08-17T17:28:01.238 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.238 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:01.239 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.239 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.239 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.239 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.35277.0:112908 5.0 e143/141 5:0de7b4a9:::benchmark_data_smithi003_140793_object10851:head v 143'7044, mlcod=143'7042)", 2024-08-17T17:28:01.239 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:20:59.553417+0000", 2024-08-17T17:28:01.239 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 421.61555259400001, 2024-08-17T17:28:01.239 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.01269575, 2024-08-17T17:28:01.239 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:01.239 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:01.239 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:01.239 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.239 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:01.239 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.553417+0000", 2024-08-17T17:28:01.239 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.240 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.240 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.240 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:01.240 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.553417+0000", 2024-08-17T17:28:01.240 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.240 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.240 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.240 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:01.240 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.553417+0000", 2024-08-17T17:28:01.240 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:28:01.240 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.240 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.240 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:01.240 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.553421+0000", 2024-08-17T17:28:01.241 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5510000000000001e-06 2024-08-17T17:28:01.241 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.241 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.241 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:01.241 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.553429+0000", 2024-08-17T17:28:01.241 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.024e-06 2024-08-17T17:28:01.241 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.241 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.241 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:01.241 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.553432+0000", 2024-08-17T17:28:01.241 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8249999999999998e-06 2024-08-17T17:28:01.241 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.241 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.242 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:01.242 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.553479+0000", 2024-08-17T17:28:01.242 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6473e-05 2024-08-17T17:28:01.242 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.242 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.242 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:01.242 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.553506+0000", 2024-08-17T17:28:01.242 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7189999999999999e-05 2024-08-17T17:28:01.242 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.242 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.242 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:01.242 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.566065+0000", 2024-08-17T17:28:01.242 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012559305999999999 2024-08-17T17:28:01.242 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.243 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.243 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:01.243 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.566113+0000", 2024-08-17T17:28:01.243 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7840000000000003e-05 2024-08-17T17:28:01.243 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.243 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:01.243 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.243 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.243 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.243 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.35277.0:112945 5.7 e143/141 5:ef5dd467:::benchmark_data_smithi003_140793_object10888:head v 143'7108, mlcod=143'7106)", 2024-08-17T17:28:01.243 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:20:59.606935+0000", 2024-08-17T17:28:01.243 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 421.56203538199998, 2024-08-17T17:28:01.243 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012337344, 2024-08-17T17:28:01.243 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:01.244 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:01.244 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:01.244 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.244 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:01.244 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.606935+0000", 2024-08-17T17:28:01.244 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.244 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.244 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.244 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:01.244 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.606935+0000", 2024-08-17T17:28:01.244 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.244 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.244 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.245 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:01.245 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.606934+0000", 2024-08-17T17:28:01.245 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:28:01.245 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.245 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.245 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:01.245 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.606941+0000", 2024-08-17T17:28:01.245 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9419999999999998e-06 2024-08-17T17:28:01.245 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.245 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.245 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:01.245 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.606962+0000", 2024-08-17T17:28:01.245 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1478999999999999e-05 2024-08-17T17:28:01.245 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.246 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.246 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:01.246 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.606969+0000", 2024-08-17T17:28:01.246 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0350000000000004e-06 2024-08-17T17:28:01.246 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.246 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.246 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:01.246 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.607019+0000", 2024-08-17T17:28:01.246 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9413999999999998e-05 2024-08-17T17:28:01.246 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.246 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.246 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:01.246 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.607044+0000", 2024-08-17T17:28:01.247 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4916e-05 2024-08-17T17:28:01.247 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.247 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.247 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:01.247 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.619138+0000", 2024-08-17T17:28:01.247 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012094737 2024-08-17T17:28:01.247 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.247 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.247 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:01.247 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.619272+0000", 2024-08-17T17:28:01.247 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013368399999999999 2024-08-17T17:28:01.247 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.247 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:01.248 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.248 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.248 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.248 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.35277.0:112946 5.7 e143/141 5:e222d0bd:::benchmark_data_smithi003_140793_object10889:head v 143'7109, mlcod=143'7106)", 2024-08-17T17:28:01.248 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:20:59.606982+0000", 2024-08-17T17:28:01.248 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 421.56198762999998, 2024-08-17T17:28:01.248 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012322955, 2024-08-17T17:28:01.248 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:01.248 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:01.248 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:01.248 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.248 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:01.248 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.606982+0000", 2024-08-17T17:28:01.249 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.249 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.249 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.249 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:01.249 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.606982+0000", 2024-08-17T17:28:01.249 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.249 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.249 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.249 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:01.249 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.606982+0000", 2024-08-17T17:28:01.249 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:28:01.249 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.249 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.249 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:01.250 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.606986+0000", 2024-08-17T17:28:01.250 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9430000000000002e-06 2024-08-17T17:28:01.250 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.250 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.250 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:01.250 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.606992+0000", 2024-08-17T17:28:01.250 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0970000000000002e-06 2024-08-17T17:28:01.250 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.250 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.250 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:01.250 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.606996+0000", 2024-08-17T17:28:01.250 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.053e-06 2024-08-17T17:28:01.251 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.251 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.251 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:01.251 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.609520+0000", 2024-08-17T17:28:01.251 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0025242179999999999 2024-08-17T17:28:01.251 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.251 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.251 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:01.251 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.609592+0000", 2024-08-17T17:28:01.251 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.1718999999999996e-05 2024-08-17T17:28:01.251 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.251 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.251 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:01.252 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.619277+0000", 2024-08-17T17:28:01.252 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0096851469999999999 2024-08-17T17:28:01.252 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.252 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.252 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:01.252 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.619305+0000", 2024-08-17T17:28:01.252 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8076e-05 2024-08-17T17:28:01.252 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.252 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:01.252 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.252 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.252 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.252 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.35277.0:112949 5.7 e143/141 5:ef515977:::benchmark_data_smithi003_140793_object10892:head v 143'7111, mlcod=143'7106)", 2024-08-17T17:28:01.253 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:20:59.612010+0000", 2024-08-17T17:28:01.253 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 421.55696011999999, 2024-08-17T17:28:01.253 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012005284999999999, 2024-08-17T17:28:01.253 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:01.253 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:01.253 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:01.253 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.253 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:01.253 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.612010+0000", 2024-08-17T17:28:01.253 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.253 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.253 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.253 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:01.253 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.612010+0000", 2024-08-17T17:28:01.254 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.254 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.254 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.254 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:01.254 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.612009+0000", 2024-08-17T17:28:01.254 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:28:01.254 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.254 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.254 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:01.254 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.612016+0000", 2024-08-17T17:28:01.254 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.7209999999999997e-06 2024-08-17T17:28:01.254 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.254 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.255 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:01.255 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.612031+0000", 2024-08-17T17:28:01.255 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.471e-05 2024-08-17T17:28:01.255 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.255 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.255 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:01.255 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.612037+0000", 2024-08-17T17:28:01.255 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1639999999999998e-06 2024-08-17T17:28:01.255 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.255 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.255 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:01.255 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.619368+0000", 2024-08-17T17:28:01.255 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.007330937 2024-08-17T17:28:01.256 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.256 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.256 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:01.256 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.619394+0000", 2024-08-17T17:28:01.256 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6673e-05 2024-08-17T17:28:01.256 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.256 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.256 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:01.256 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.623960+0000", 2024-08-17T17:28:01.256 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0045657129999999999 2024-08-17T17:28:01.256 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.256 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.256 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:01.256 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:20:59.624015+0000", 2024-08-17T17:28:01.257 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5028000000000002e-05 2024-08-17T17:28:01.257 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.257 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:01.257 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.257 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.257 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.257 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.35277.0:113379 5.7 e143/141 5:e17073df:::benchmark_data_smithi003_140793_object11322:head v 143'7137, mlcod=143'7134)", 2024-08-17T17:28:01.257 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:21:00.200433+0000", 2024-08-17T17:28:01.257 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 420.968537266, 2024-08-17T17:28:01.257 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012001871000000001, 2024-08-17T17:28:01.257 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:01.257 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:01.257 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:01.257 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.258 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:01.258 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:00.200433+0000", 2024-08-17T17:28:01.258 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.258 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.258 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.258 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:01.258 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:00.200433+0000", 2024-08-17T17:28:01.258 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.258 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.258 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.258 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:01.258 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:00.200432+0000", 2024-08-17T17:28:01.258 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:28:01.259 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.259 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.259 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:01.259 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:00.200437+0000", 2024-08-17T17:28:01.259 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5349999999999998e-06 2024-08-17T17:28:01.259 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.259 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.259 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:01.259 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:00.200443+0000", 2024-08-17T17:28:01.259 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7849999999999997e-06 2024-08-17T17:28:01.259 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.259 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.259 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:01.259 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:00.200446+0000", 2024-08-17T17:28:01.259 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3440000000000001e-06 2024-08-17T17:28:01.260 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.260 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.260 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:01.260 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:00.205927+0000", 2024-08-17T17:28:01.260 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0054805560000000001 2024-08-17T17:28:01.260 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.260 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.260 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:01.260 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:00.208211+0000", 2024-08-17T17:28:01.260 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0022845349999999999 2024-08-17T17:28:01.260 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.260 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.260 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:01.261 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:00.212391+0000", 2024-08-17T17:28:01.261 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0041795299999999999 2024-08-17T17:28:01.261 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.261 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.261 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:01.261 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:00.212435+0000", 2024-08-17T17:28:01.261 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3890999999999997e-05 2024-08-17T17:28:01.261 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.261 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:01.262 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.262 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.262 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.262 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.35277.0:122452 5.0 e145/141 5:0b0145b6:::benchmark_data_smithi003_140793_object20395:head v 145'7648, mlcod=145'7646)", 2024-08-17T17:28:01.262 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:21:13.979429+0000", 2024-08-17T17:28:01.262 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 407.18954142899997, 2024-08-17T17:28:01.262 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012890697, 2024-08-17T17:28:01.262 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:01.262 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:01.262 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:01.262 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.262 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:01.262 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:13.979429+0000", 2024-08-17T17:28:01.263 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.263 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.263 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.263 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:01.263 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:13.979429+0000", 2024-08-17T17:28:01.263 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.263 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.263 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.263 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:01.263 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:13.979427+0000", 2024-08-17T17:28:01.263 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:28:01.263 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.263 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.264 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:01.264 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:13.979439+0000", 2024-08-17T17:28:01.264 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2023e-05 2024-08-17T17:28:01.264 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.264 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.264 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:01.264 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:13.979463+0000", 2024-08-17T17:28:01.264 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3754999999999999e-05 2024-08-17T17:28:01.264 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.264 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.264 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:01.264 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:13.979472+0000", 2024-08-17T17:28:01.264 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.3999999999999998e-06 2024-08-17T17:28:01.264 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.265 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.265 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:01.265 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:13.979619+0000", 2024-08-17T17:28:01.265 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014733199999999999 2024-08-17T17:28:01.265 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.265 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.265 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:01.265 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:13.979699+0000", 2024-08-17T17:28:01.265 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.9178000000000007e-05 2024-08-17T17:28:01.265 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.265 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.265 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:01.265 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:13.992191+0000", 2024-08-17T17:28:01.266 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012492754999999999 2024-08-17T17:28:01.266 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.266 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.266 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:01.266 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:13.992319+0000", 2024-08-17T17:28:01.266 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012788599999999999 2024-08-17T17:28:01.266 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.266 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:01.266 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.266 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.266 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.266 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.35277.0:122453 5.0 e145/141 5:0ee0ca18:::benchmark_data_smithi003_140793_object20396:head v 145'7649, mlcod=145'7646)", 2024-08-17T17:28:01.267 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:21:13.983089+0000", 2024-08-17T17:28:01.267 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 407.18588123199999, 2024-08-17T17:28:01.267 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011904247999999999, 2024-08-17T17:28:01.267 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:01.267 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:01.267 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:01.267 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.267 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:01.267 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:13.983089+0000", 2024-08-17T17:28:01.267 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.267 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.267 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.267 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:01.267 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:13.983089+0000", 2024-08-17T17:28:01.268 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.268 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.268 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.268 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:01.268 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:13.983087+0000", 2024-08-17T17:28:01.268 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:28:01.268 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.268 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.268 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:01.268 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:13.983100+0000", 2024-08-17T17:28:01.268 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2316e-05 2024-08-17T17:28:01.268 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.268 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.269 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:01.269 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:13.983122+0000", 2024-08-17T17:28:01.269 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2087e-05 2024-08-17T17:28:01.269 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.269 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.269 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:01.269 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:13.983129+0000", 2024-08-17T17:28:01.269 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.695e-06 2024-08-17T17:28:01.269 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.269 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.269 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:01.269 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:13.983913+0000", 2024-08-17T17:28:01.269 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00078339000000000002 2024-08-17T17:28:01.269 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.270 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.270 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:01.270 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:13.983939+0000", 2024-08-17T17:28:01.270 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6112e-05 2024-08-17T17:28:01.270 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.270 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.270 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:01.270 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:13.994949+0000", 2024-08-17T17:28:01.270 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011010000000000001 2024-08-17T17:28:01.270 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.270 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.270 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:01.271 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:13.994993+0000", 2024-08-17T17:28:01.271 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4150999999999998e-05 2024-08-17T17:28:01.271 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.271 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:01.271 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.271 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.271 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.271 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.35277.0:126161 5.0 e145/141 5:0c196b4d:::benchmark_data_smithi003_140793_object24104:head v 145'7906, mlcod=145'7904)", 2024-08-17T17:28:01.271 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:21:19.749659+0000", 2024-08-17T17:28:01.271 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 401.41931142599998, 2024-08-17T17:28:01.271 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012166372, 2024-08-17T17:28:01.271 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:01.271 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:01.271 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:01.272 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.272 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:01.272 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:19.749659+0000", 2024-08-17T17:28:01.272 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.272 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.272 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.272 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:01.272 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:19.749659+0000", 2024-08-17T17:28:01.272 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.272 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.272 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.272 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:01.272 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:19.749657+0000", 2024-08-17T17:28:01.272 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:28:01.273 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.273 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.273 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:01.273 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:19.749668+0000", 2024-08-17T17:28:01.273 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.045e-05 2024-08-17T17:28:01.273 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.273 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.273 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:01.273 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:19.749689+0000", 2024-08-17T17:28:01.273 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1359999999999999e-05 2024-08-17T17:28:01.273 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.273 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.273 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:01.273 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:19.749696+0000", 2024-08-17T17:28:01.274 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.1720000000000001e-06 2024-08-17T17:28:01.274 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.274 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.274 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:01.274 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:19.752099+0000", 2024-08-17T17:28:01.274 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0024025689999999998 2024-08-17T17:28:01.274 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.274 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.274 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:01.274 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:19.752128+0000", 2024-08-17T17:28:01.274 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9643999999999999e-05 2024-08-17T17:28:01.274 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.274 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.275 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:01.275 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:19.761738+0000", 2024-08-17T17:28:01.275 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0096097040000000002 2024-08-17T17:28:01.275 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.275 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.275 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:01.275 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:19.761825+0000", 2024-08-17T17:28:01.275 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.6842999999999995e-05 2024-08-17T17:28:01.275 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.275 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:01.275 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.275 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.275 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.275 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.35277.0:141363 5.c e149/141 5:3abf1c19:::benchmark_data_smithi003_140793_object39306:head v 149'8775, mlcod=149'8773)", 2024-08-17T17:28:01.276 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:21:43.115743+0000", 2024-08-17T17:28:01.276 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 378.05322713499999, 2024-08-17T17:28:01.276 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012915163, 2024-08-17T17:28:01.276 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:01.276 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:01.276 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:01.276 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.276 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:01.276 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:43.115743+0000", 2024-08-17T17:28:01.276 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.276 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.276 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.276 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:01.276 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:43.115743+0000", 2024-08-17T17:28:01.277 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:01.277 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.277 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.277 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:01.277 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:43.115742+0000", 2024-08-17T17:28:01.277 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:28:01.277 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.277 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.277 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:01.277 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:43.115759+0000", 2024-08-17T17:28:01.277 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7422e-05 2024-08-17T17:28:01.277 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.277 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.277 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:01.278 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:43.115787+0000", 2024-08-17T17:28:01.278 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7627000000000002e-05 2024-08-17T17:28:01.278 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.278 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.278 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:01.278 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:43.115795+0000", 2024-08-17T17:28:01.278 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.3529999999999995e-06 2024-08-17T17:28:01.278 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.278 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.278 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:01.278 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:43.118659+0000", 2024-08-17T17:28:01.278 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0028640290000000001 2024-08-17T17:28:01.278 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.279 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.279 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:01.279 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:43.118716+0000", 2024-08-17T17:28:01.279 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7142000000000003e-05 2024-08-17T17:28:01.279 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.279 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.279 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:01.279 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:43.128458+0000", 2024-08-17T17:28:01.279 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0097419380000000003 2024-08-17T17:28:01.279 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:01.279 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:01.279 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:01.279 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:43.128658+0000", 2024-08-17T17:28:01.280 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000199891 2024-08-17T17:28:01.280 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.280 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:01.280 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.280 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:01.280 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:01.280 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:28:01.280 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:01.305 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:01.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:01.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:01.406 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:01.442 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:01.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:01.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:01.544 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:01.573 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:01.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:01.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:01.675 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:01.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:01.706 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:01.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:01.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:01 smithi003 ceph-mon[100615]: pgmap v594: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 505 MiB data, 4.3 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 799 op/s; 3486/24096 objects degraded (14.467%); 0 B/s, 46 objects/s recovering 2024-08-17T17:28:01.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:01 smithi003 ceph-mon[103344]: pgmap v594: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 505 MiB data, 4.3 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 799 op/s; 3486/24096 objects degraded (14.467%); 0 B/s, 46 objects/s recovering 2024-08-17T17:28:01.807 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:01.843 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:01.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:01.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:01.945 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:01.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:01 smithi053 ceph-mon[89963]: pgmap v594: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 505 MiB data, 4.3 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 799 op/s; 3486/24096 objects degraded (14.467%); 0 B/s, 46 objects/s recovering 2024-08-17T17:28:01.977 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:01.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:01.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:02.079 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:02.081 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-17T17:28:02.111 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:02.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:02.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:02.213 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:02.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:02.236 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:02.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:02.337 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:02.371 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:02.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:02.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:02.444 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:28:02.472 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:02.510 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:02.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:02.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:02.612 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:02.657 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:02.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:02.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:02.759 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:02.795 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:02.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:02.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:02.897 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:02.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:02.937 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:02.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:03.038 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:03.087 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:03.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:03.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:03.189 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:03.222 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:03.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:03.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:03.324 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:03.361 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:03.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:03.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:03.463 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:03.487 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:03.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:03.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:03.589 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:03.617 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:03.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:03.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:03.719 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:03.744 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:03.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:03.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:03.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:03 smithi003 ceph-mon[103344]: pgmap v595: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 458 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 670 op/s; 3486/21846 objects degraded (15.957%); 0 B/s, 22 objects/s recovering 2024-08-17T17:28:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:03 smithi003 ceph-mon[100615]: pgmap v595: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 458 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 670 op/s; 3486/21846 objects degraded (15.957%); 0 B/s, 22 objects/s recovering 2024-08-17T17:28:03.846 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:03.870 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:03.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:03.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:03.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:03 smithi053 ceph-mon[89963]: pgmap v595: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 458 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 670 op/s; 3486/21846 objects degraded (15.957%); 0 B/s, 22 objects/s recovering 2024-08-17T17:28:03.972 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:04.000 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:04.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:04.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:04.102 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:04.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:04.146 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:04.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:04.248 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:04.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:04.308 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:04.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:04.410 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:04.442 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:04.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:04.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:04.544 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:04.580 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:04.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:04.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:04.647 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:28:04.647 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2024-08-17T17:28:04.647 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.647 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:194698 6.7 6:e6731fdb:::benchmark_data_smithi003_175043_object96049:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-08-17T17:28:04.647 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:28:04.629877+0000", 2024-08-17T17:28:04.647 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.014932632, 2024-08-17T17:28:04.647 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.014963405000000001, 2024-08-17T17:28:04.648 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:04.648 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:28:04.648 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:04.648 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:28:04.648 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:28:04.648 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 194698 2024-08-17T17:28:04.648 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.648 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:04.648 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.648 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:04.648 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.629877+0000", 2024-08-17T17:28:04.648 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:04.648 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.649 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.649 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:04.649 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.629877+0000", 2024-08-17T17:28:04.649 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:04.649 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.649 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.649 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:04.649 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.629875+0000", 2024-08-17T17:28:04.649 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:28:04.649 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.649 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.649 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:04.649 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.629882+0000", 2024-08-17T17:28:04.650 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3489999999999997e-06 2024-08-17T17:28:04.650 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.650 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.650 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:04.650 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.629897+0000", 2024-08-17T17:28:04.650 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5169e-05 2024-08-17T17:28:04.650 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.650 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.650 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:04.650 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.629903+0000", 2024-08-17T17:28:04.650 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0440000000000001e-06 2024-08-17T17:28:04.650 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.650 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.651 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:04.651 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.629955+0000", 2024-08-17T17:28:04.651 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.202e-05 2024-08-17T17:28:04.651 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.651 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.651 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:04.651 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.630077+0000", 2024-08-17T17:28:04.651 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012245800000000001 2024-08-17T17:28:04.651 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.651 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.651 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:28:04.651 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.630209+0000", 2024-08-17T17:28:04.651 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013162699999999999 2024-08-17T17:28:04.652 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.652 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.652 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:04.652 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.640285+0000", 2024-08-17T17:28:04.652 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010076039 2024-08-17T17:28:04.652 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:04.652 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:04.652 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:04.652 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.652 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.652 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:194701 6.7 6:e3b317e3:::benchmark_data_smithi003_175043_object96052:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-08-17T17:28:04.652 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:28:04.634205+0000", 2024-08-17T17:28:04.652 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.010604924999999999, 2024-08-17T17:28:04.652 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010725405, 2024-08-17T17:28:04.653 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:04.653 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:28:04.653 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:04.653 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:28:04.653 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:28:04.653 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 194701 2024-08-17T17:28:04.653 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.653 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:04.653 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.653 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:04.653 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.634205+0000", 2024-08-17T17:28:04.653 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:04.653 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.653 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.654 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:04.654 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.634205+0000", 2024-08-17T17:28:04.654 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:04.654 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.654 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.654 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:04.654 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.634204+0000", 2024-08-17T17:28:04.654 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:28:04.654 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.654 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.654 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:04.654 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.634209+0000", 2024-08-17T17:28:04.654 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1259999999999997e-06 2024-08-17T17:28:04.654 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.655 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.655 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:04.655 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.634220+0000", 2024-08-17T17:28:04.655 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0672e-05 2024-08-17T17:28:04.655 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.655 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.655 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:04.655 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.634226+0000", 2024-08-17T17:28:04.655 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0390000000000001e-06 2024-08-17T17:28:04.655 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.655 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.655 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:04.655 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.634888+0000", 2024-08-17T17:28:04.656 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000662541 2024-08-17T17:28:04.656 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.656 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.656 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:04.656 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.635012+0000", 2024-08-17T17:28:04.656 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012392699999999999 2024-08-17T17:28:04.656 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.656 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.656 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:28:04.656 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.635123+0000", 2024-08-17T17:28:04.656 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000110926 2024-08-17T17:28:04.656 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.656 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.656 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:04.657 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.640400+0000", 2024-08-17T17:28:04.657 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0052769109999999996 2024-08-17T17:28:04.657 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:04.657 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:04.657 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:04.657 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.657 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.657 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:194712 6.7 6:efcf7081:::benchmark_data_smithi003_175043_object96063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-08-17T17:28:04.657 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:28:04.643288+0000", 2024-08-17T17:28:04.657 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.0015220469999999999, 2024-08-17T17:28:04.657 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0017163689999999999, 2024-08-17T17:28:04.657 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:04.657 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:28:04.657 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:04.658 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:28:04.658 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:28:04.658 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 194712 2024-08-17T17:28:04.658 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.658 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:04.658 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.658 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:04.658 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.643288+0000", 2024-08-17T17:28:04.658 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:04.658 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.658 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.658 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:04.658 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.643288+0000", 2024-08-17T17:28:04.659 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:04.659 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.659 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.659 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:04.659 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.643286+0000", 2024-08-17T17:28:04.659 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:28:04.659 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.659 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.659 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:04.659 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.643294+0000", 2024-08-17T17:28:04.659 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.3690000000000007e-06 2024-08-17T17:28:04.659 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.659 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.659 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:04.660 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.643317+0000", 2024-08-17T17:28:04.660 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.285e-05 2024-08-17T17:28:04.660 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.660 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.660 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:04.660 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.643325+0000", 2024-08-17T17:28:04.660 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.2369999999999992e-06 2024-08-17T17:28:04.660 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.660 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.660 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:04.660 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.643398+0000", 2024-08-17T17:28:04.660 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.2243999999999995e-05 2024-08-17T17:28:04.660 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.660 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.660 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:04.661 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.643535+0000", 2024-08-17T17:28:04.661 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013761700000000001 2024-08-17T17:28:04.661 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.661 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.661 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:28:04.661 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.643672+0000", 2024-08-17T17:28:04.661 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013660500000000001 2024-08-17T17:28:04.661 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:04.661 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:04.661 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:04.661 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.661 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.661 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:194713 6.7 6.b3456e17 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-08-17T17:28:04.662 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:28:04.643341+0000", 2024-08-17T17:28:04.662 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.0014686809999999999, 2024-08-17T17:28:04.662 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0017310069999999999, 2024-08-17T17:28:04.662 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:04.662 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "queued for pg", 2024-08-17T17:28:04.662 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:04.662 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:28:04.662 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:28:04.662 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 194713 2024-08-17T17:28:04.662 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.662 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:04.662 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.662 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:04.663 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.643341+0000", 2024-08-17T17:28:04.663 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:04.663 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.663 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.663 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:04.663 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.643341+0000", 2024-08-17T17:28:04.663 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:04.663 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.663 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.663 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:04.663 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.643341+0000", 2024-08-17T17:28:04.663 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:28:04.663 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.664 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.664 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:04.664 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.643344+0000", 2024-08-17T17:28:04.664 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3440000000000001e-06 2024-08-17T17:28:04.664 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.664 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.664 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:04.664 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.643351+0000", 2024-08-17T17:28:04.664 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3719999999999999e-06 2024-08-17T17:28:04.664 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.664 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.664 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:04.664 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.643356+0000", 2024-08-17T17:28:04.664 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5190000000000003e-06 2024-08-17T17:28:04.665 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:04.665 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:04.665 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:04.665 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.665 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.665 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.45491.0:194709 6.4 e220/178 6:22a105ff:::benchmark_data_smithi003_175043_object96060:head v 220'12055, mlcod=220'12052)", 2024-08-17T17:28:04.665 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:28:04.644962+0000", 2024-08-17T17:28:04.665 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4294967295.9998484, 2024-08-17T17:28:04.665 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000162862, 2024-08-17T17:28:04.665 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:04.665 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2024-08-17T17:28:04.665 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:04.665 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.665 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:04.666 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.644962+0000", 2024-08-17T17:28:04.666 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:04.666 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.666 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.666 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:04.666 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.644962+0000", 2024-08-17T17:28:04.666 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:04.666 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.666 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.666 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:04.666 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.644961+0000", 2024-08-17T17:28:04.666 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:28:04.666 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.666 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.667 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:04.667 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.644968+0000", 2024-08-17T17:28:04.667 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9999999999999999e-06 2024-08-17T17:28:04.667 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.667 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.667 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:04.667 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.644982+0000", 2024-08-17T17:28:04.667 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4797000000000001e-05 2024-08-17T17:28:04.667 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.667 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.667 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:04.667 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.644988+0000", 2024-08-17T17:28:04.667 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0379999999999996e-06 2024-08-17T17:28:04.667 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.667 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.668 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:04.668 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.645032+0000", 2024-08-17T17:28:04.668 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3409999999999999e-05 2024-08-17T17:28:04.668 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:04.668 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:04.668 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:04.668 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:04.645053+0000", 2024-08-17T17:28:04.668 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1004000000000001e-05 2024-08-17T17:28:04.668 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:04.668 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:04.668 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:04.668 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:04.668 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-08-17T17:28:04.669 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 5 2024-08-17T17:28:04.669 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:28:04.682 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:04.716 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:04.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:04.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:04.818 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:04.846 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:04.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:04.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:04.950 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:04.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:04.982 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:04.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:05.083 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:05.129 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:05.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:05.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:05.231 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:05.259 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:05.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:05.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:05.361 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:05.389 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:05.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:05.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:05.491 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:05.549 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:05.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:05.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:05.595 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:05 smithi003 ceph-mon[100615]: pgmap v596: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 316 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 769 op/s; 3189/15051 objects degraded (21.188%); 0 B/s, 52 objects/s recovering 2024-08-17T17:28:05.595 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:05 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 3189/15051 objects degraded (21.188%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:28:05.595 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:05 smithi003 ceph-mon[103344]: pgmap v596: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 316 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 769 op/s; 3189/15051 objects degraded (21.188%); 0 B/s, 52 objects/s recovering 2024-08-17T17:28:05.595 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:05 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 3189/15051 objects degraded (21.188%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:28:05.650 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:05.679 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:05.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:05.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:05.781 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:05.792 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:28:05.793 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:28:05.809 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:05.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:05.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:05.829 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:28:05.829 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:28:05.867 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:28:05.867 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:28:05.867 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:28:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:28:05] "GET /metrics HTTP/1.1" 200 36505 "" "Prometheus/2.51.0" 2024-08-17T17:28:05.911 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:05.917 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:28:05.917 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:28:05.943 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:05.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:05.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:05.950 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:28:05.950 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:28:05.964 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:05 smithi053 ceph-mon[89963]: pgmap v596: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 316 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 769 op/s; 3189/15051 objects degraded (21.188%); 0 B/s, 52 objects/s recovering 2024-08-17T17:28:05.964 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:05 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 3189/15051 objects degraded (21.188%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:28:05.980 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:28:05.980 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:28:05.999 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-17T17:28:06.014 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:28:06.014 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:28:06.045 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:06.047 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:28:06.047 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:28:06.118 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:06.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:06.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:06.220 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:06.250 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:06.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:06.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:06.352 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:06.393 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:06.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:06.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:06.425 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:28:06.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:06.541 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:06.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:06.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:06.638 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:06 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/669721281' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:28:06.639 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:06 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/669721281' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:28:06.643 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:06.669 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:06.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:06.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:06.772 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:28:06.772 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:06.817 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:06.818 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:06 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/669721281' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:28:06.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:06.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:06.920 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:06.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:06.959 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:06.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:07.060 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:07.085 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:07.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:07.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:07.186 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:07.218 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:07.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:07.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:07.320 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:07.352 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:07.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:07.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:07.454 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:07.479 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:07.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:07.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:07.581 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:07.633 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:07.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:07.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:07.735 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:07.764 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:07.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:07.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:07.772 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:07 smithi053 ceph-mon[89963]: pgmap v597: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 221 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 773 op/s; 3189/10467 objects degraded (30.467%); 0 B/s, 52 objects/s recovering 2024-08-17T17:28:07.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:07 smithi003 ceph-mon[103344]: pgmap v597: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 221 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 773 op/s; 3189/10467 objects degraded (30.467%); 0 B/s, 52 objects/s recovering 2024-08-17T17:28:07.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:07 smithi003 ceph-mon[100615]: pgmap v597: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 221 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 773 op/s; 3189/10467 objects degraded (30.467%); 0 B/s, 52 objects/s recovering 2024-08-17T17:28:07.866 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:07.892 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:07.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:07.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:07.901 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Removed 98646 objects 2024-08-17T17:28:07.902 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Clean up completed and total clean up time :128.8 2024-08-17T17:28:07.995 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:08.018 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:08.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:08.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:08.120 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:08.148 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:08.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:08.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:08.170 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:28:08.170 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:28:08.179 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:28:08.250 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:08.273 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:08.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:08.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:08.375 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:08.397 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:08.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:08.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:08.425 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:28:08.425 INFO:teuthology.orchestra.run.smithi003.stdout:{"fsid":"2b5b662e-5cba-11ef-bccf-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3189/10467 objects degraded (30.467%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":830,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":220,"num_osds":8,"num_up_osds":8,"osd_up_since":1723915051,"num_in_osds":7,"osd_in_since":1723915248,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":87},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":89,"num_pools":5,"num_objects":3489,"data_bytes":231544805,"bytes_used":3816534016,"bytes_avail":668148715520,"bytes_total":671965249536,"degraded_objects":3189,"degraded_total":10467,"degraded_ratio":0.30467182573803381,"recovering_objects_per_sec":52,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":526,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":773},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi003.civgvf","status":"up:active","gid":14664}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.3:8443/","prometheus":"http://172.21.15.3:9283/"}},"servicemap":{"epoch":23,"modified":"2024-08-17T17:18:31.688676+0000","services":{}},"progress_events":{"714d66b4-625a-4d60-aa1c-f38ff47ebe12":{"message":"Global Recovery Event (2m)\n [===========================.] (remaining: 3s)","progress":0.97752809524536133,"add_to_ceph_s":true},"fcbeb9b7-7baf-488b-9fe4-c01aca441f9c":{"message":"Upgrade to 19.0.0-5836-gcd8bdbeb (4m)\n [============................] (remaining: 5m)","progress":0.44999998807907104,"add_to_ceph_s":true}}} 2024-08-17T17:28:08.499 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:08.522 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:08.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:08.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:08.624 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:08.648 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:08.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:08.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:08.661 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-08-17T17:28:08.661 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-08-17T17:28:08.750 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:08.772 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:08.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:08.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:08.825 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-17T17:28:08.874 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:08.897 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:08.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:08.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:08.999 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:09.022 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:09.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:09.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:09.091 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:09 smithi053 ceph-mon[89963]: pgmap v598: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 139 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 741 op/s; 2917/6549 objects degraded (44.541%); 0 B/s, 54 objects/s recovering 2024-08-17T17:28:09.091 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:09.104 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:28:09.124 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:09.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:09.151 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:09.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:09.252 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:09.275 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:09.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:09.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:09.377 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:09.399 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:09.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:09.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:28:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:09 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1771360644' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T17:28:09.501 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:09.503 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:28:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:09 smithi003 ceph-mon[103344]: pgmap v598: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 139 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 741 op/s; 2917/6549 objects degraded (44.541%); 0 B/s, 54 objects/s recovering 2024-08-17T17:28:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:28:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:09 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1771360644' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T17:28:09.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:09 smithi003 ceph-mon[100615]: pgmap v598: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 139 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 741 op/s; 2917/6549 objects degraded (44.541%); 0 B/s, 54 objects/s recovering 2024-08-17T17:28:09.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:09.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:28:09.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:09 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1771360644' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T17:28:09.526 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:09.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:09.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:09.628 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:09.650 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:09.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:09.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:09.752 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:09.775 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:09.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:09.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:09.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:09.901 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:09.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:09.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:10.003 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:10.025 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:10.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:10.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:10.127 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:10.151 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:10.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:10.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:10.215 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:10 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 2917/6549 objects degraded (44.541%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:28:10.215 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:10 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 2917/6549 objects degraded (44.541%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:28:10.252 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:10.274 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:10.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:10.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:10.377 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:10.399 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:10.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:10.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:10.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:10 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 2917/6549 objects degraded (44.541%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:28:10.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:28:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:28:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:28:10.501 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:10.525 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:10.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:10.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:10.627 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:10.649 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:10.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:10.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:10.750 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:10.773 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:10.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:10.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:10.874 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:10.897 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:10.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:10.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:10.998 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:11.021 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:11.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:11.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:11.123 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:11.145 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:11.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:11.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:11.205 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:11 smithi053 ceph-mon[89963]: pgmap v599: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 38 MiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 783 op/s; 2917/1686 objects degraded (173.013%); 0 B/s, 47 objects/s recovering 2024-08-17T17:28:11.247 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:11.269 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:11.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:11.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:11.305 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:28:11.305 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:28:11.371 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:11.393 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:11.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:11.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:11.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:11.517 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:11.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:11.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:11.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:11 smithi003 ceph-mon[103344]: pgmap v599: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 38 MiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 783 op/s; 2917/1686 objects degraded (173.013%); 0 B/s, 47 objects/s recovering 2024-08-17T17:28:11.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:11 smithi003 ceph-mon[100615]: pgmap v599: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 38 MiB data, 3.0 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 783 op/s; 2917/1686 objects degraded (173.013%); 0 B/s, 47 objects/s recovering 2024-08-17T17:28:11.620 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:11.642 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:11.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:11.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:11.744 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:11.767 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:11.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:11.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:11.869 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:11.895 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:11.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:11.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:11.986 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-17T17:28:11.996 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:12.051 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:12.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:12.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:12.153 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:12.177 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:12.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:12.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:12.265 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:28:12.279 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:12.302 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:12.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:12.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:12.404 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:12.427 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:12.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:12.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:12.530 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:12.554 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:12.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:12.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:12.656 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:12.679 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:12.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:12.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:12.781 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:12.804 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:12.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:12.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:12.906 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:12.930 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:12.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:12.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:12.945 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-08-17T17:28:12.946 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:28:13.032 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:13.055 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:13.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:13.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:13.157 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:13.179 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:13.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:13.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:13.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:13 smithi003 ceph-mon[100615]: pgmap v600: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 18 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 648 op/s; 2917/729 objects degraded (400.137%); 0 B/s, 47 objects/s recovering 2024-08-17T17:28:13.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:13 smithi003 ceph-mon[103344]: pgmap v600: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 18 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 648 op/s; 2917/729 objects degraded (400.137%); 0 B/s, 47 objects/s recovering 2024-08-17T17:28:13.281 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:13.303 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:13.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:13.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:13.405 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:13.427 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:13.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:13.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:13.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:13 smithi053 ceph-mon[89963]: pgmap v600: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 18 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 648 op/s; 2917/729 objects degraded (400.137%); 0 B/s, 47 objects/s recovering 2024-08-17T17:28:13.529 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:13.555 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:13.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:13.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:13.657 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:13.684 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:13.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:13.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:13.726 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:28:13.729 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:28:13.778 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:28:13.785 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:13.809 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:13.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:13.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:13.911 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:13.933 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:13.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:13.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:14.035 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:14.058 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:28:14.186 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:28:14.187 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-08-17T17:28:14.187 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:14 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3704830484' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:28:14.187 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:14 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:28:14.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:14 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3704830484' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:28:14.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:14 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:28:14.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:14.189 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:14.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:14.290 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:14.315 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:14.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:14.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:14.377 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:14 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3704830484' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:28:14.377 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:14 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:28:14.417 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:14.442 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:14.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:14.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:14.544 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:14.567 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:14.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:14.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:14.669 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:14.692 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:14.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:14.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:14.706 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-17T17:28:14.794 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:14.817 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:14.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:14.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:14.919 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:14.941 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:14.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:14.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:14.999 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:28:15.043 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:15.065 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:15.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:15.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:15.167 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:15.191 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:15.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:15.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:15.293 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:15.316 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:15.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:15.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:15.398 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:15 smithi053 ceph-mon[89963]: pgmap v601: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 4.6 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 604 op/s; 1728/84 objects degraded (2057.143%); 0 B/s, 146 objects/s recovering 2024-08-17T17:28:15.398 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:15 smithi053 ceph-mon[89963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:28:15.398 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:15 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:28:15.398 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:15 smithi053 ceph-mon[89963]: osdmap e221: 8 total, 8 up, 7 in 2024-08-17T17:28:15.398 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:15 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 1728/84 objects degraded (2057.143%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:28:15.419 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:15.439 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:15 smithi003 ceph-mon[103344]: pgmap v601: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 4.6 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 604 op/s; 1728/84 objects degraded (2057.143%); 0 B/s, 146 objects/s recovering 2024-08-17T17:28:15.439 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:15 smithi003 ceph-mon[103344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:28:15.439 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:15 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:28:15.439 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:15 smithi003 ceph-mon[103344]: osdmap e221: 8 total, 8 up, 7 in 2024-08-17T17:28:15.439 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:15 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 1728/84 objects degraded (2057.143%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:28:15.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:15 smithi003 ceph-mon[100615]: pgmap v601: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 4.6 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 604 op/s; 1728/84 objects degraded (2057.143%); 0 B/s, 146 objects/s recovering 2024-08-17T17:28:15.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:15 smithi003 ceph-mon[100615]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:28:15.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:15 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:28:15.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:15 smithi003 ceph-mon[100615]: osdmap e221: 8 total, 8 up, 7 in 2024-08-17T17:28:15.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:15 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 1728/84 objects degraded (2057.143%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-08-17T17:28:15.441 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:15.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:15.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:15.543 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:15.567 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:15.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:15.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:15.668 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:15.691 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:15.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:15.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:15.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:28:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:28:15] "GET /metrics HTTP/1.1" 200 36507 "" "Prometheus/2.51.0" 2024-08-17T17:28:15.792 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:15.815 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:15.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:15.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:15.917 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:15.943 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:15.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:15.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:16.044 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:16.071 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:16.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:16.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:16.173 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:16.196 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:16.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:16.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:16.297 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:16.322 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:16.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:16.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:16.391 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:16 smithi053 ceph-mon[89963]: osdmap e222: 8 total, 8 up, 7 in 2024-08-17T17:28:16.423 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:16.446 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:16.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:16.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:16.486 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:16 smithi003 ceph-mon[103344]: osdmap e222: 8 total, 8 up, 7 in 2024-08-17T17:28:16.487 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:16 smithi003 ceph-mon[100615]: osdmap e222: 8 total, 8 up, 7 in 2024-08-17T17:28:16.548 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:16.571 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:16.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:16.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:16.673 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:16.698 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:16.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:16.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:16.781 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:28:16.781 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:28:16.799 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:16.823 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:16.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:16.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:16.925 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:16.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:16.948 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:16.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:17.049 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:17.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:17.072 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:17.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:17.173 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:17.195 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:17.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:17.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:17.243 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:17 smithi003 ceph-mon[103344]: pgmap v604: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 4.6 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 269 op/s; 1728/84 objects degraded (2057.143%); 0 B/s, 148 objects/s recovering 2024-08-17T17:28:17.243 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:17 smithi003 ceph-mon[103344]: osdmap e223: 8 total, 8 up, 7 in 2024-08-17T17:28:17.243 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:17 smithi003 ceph-mon[100615]: pgmap v604: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 4.6 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 269 op/s; 1728/84 objects degraded (2057.143%); 0 B/s, 148 objects/s recovering 2024-08-17T17:28:17.243 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:17 smithi003 ceph-mon[100615]: osdmap e223: 8 total, 8 up, 7 in 2024-08-17T17:28:17.297 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:17.318 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:17.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:17.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:17.386 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:17 smithi053 ceph-mon[89963]: pgmap v604: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 87 active+clean; 4.6 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 269 op/s; 1728/84 objects degraded (2057.143%); 0 B/s, 148 objects/s recovering 2024-08-17T17:28:17.386 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:17 smithi053 ceph-mon[89963]: osdmap e223: 8 total, 8 up, 7 in 2024-08-17T17:28:17.411 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-17T17:28:17.420 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:17.442 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:17.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:17.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:17.544 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:17.567 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:17.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:17.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:17.669 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:17.691 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:17.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:17.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:17.793 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:17.816 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:17.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:17.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:17.917 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:17.940 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:17.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:17.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:18.042 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:18.063 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:18.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:18.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:18.165 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:18.188 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:18.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:18.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:18.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:18 smithi003 ceph-mon[100615]: osdmap e224: 8 total, 8 up, 7 in 2024-08-17T17:28:18.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:18 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:28:18.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:18 smithi003 ceph-mon[103344]: osdmap e224: 8 total, 8 up, 7 in 2024-08-17T17:28:18.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:18 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:28:18.290 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:18.314 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:18.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:18.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:18.328 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:28:18.415 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:18.438 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:18.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:18.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:18.443 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:18 smithi053 ceph-mon[89963]: osdmap e224: 8 total, 8 up, 7 in 2024-08-17T17:28:18.443 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:18 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:28:18.516 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:28:18.540 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:18.581 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:18.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:18.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:18.683 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:18.708 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:18.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:18.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:18.809 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:18.833 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:18.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:18.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:18.935 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:18.958 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:18.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:18.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:19.060 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:19.082 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:19.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:19.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:19.141 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_1' removed 2024-08-17T17:28:19.179 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:19 smithi003 ceph-mon[103344]: pgmap v607: 89 pgs: 1 peering, 88 active+clean; 4.6 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 0 B/s, 431 objects/s recovering 2024-08-17T17:28:19.179 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:19 smithi003 ceph-mon[103344]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1728/84 objects degraded (2057.143%), 2 pgs degraded, 2 pgs undersized) 2024-08-17T17:28:19.179 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:19 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1603855701' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:28:19.179 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:19 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:28:19.186 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:19.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:19 smithi003 ceph-mon[100615]: pgmap v607: 89 pgs: 1 peering, 88 active+clean; 4.6 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 0 B/s, 431 objects/s recovering 2024-08-17T17:28:19.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:19 smithi003 ceph-mon[100615]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1728/84 objects degraded (2057.143%), 2 pgs degraded, 2 pgs undersized) 2024-08-17T17:28:19.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:19 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1603855701' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:28:19.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:19 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:28:19.209 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:19.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:19.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:19.264 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:28:19.265 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:28:19.311 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:19.335 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:19.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:19.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:19.422 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:19 smithi053 ceph-mon[89963]: pgmap v607: 89 pgs: 1 peering, 88 active+clean; 4.6 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 0 B/s, 431 objects/s recovering 2024-08-17T17:28:19.422 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:19 smithi053 ceph-mon[89963]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1728/84 objects degraded (2057.143%), 2 pgs degraded, 2 pgs undersized) 2024-08-17T17:28:19.422 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:19 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1603855701' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:28:19.422 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:19 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:28:19.437 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:19.459 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:19.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:19.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:19.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:19.583 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:19.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:19.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:19.685 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:19.708 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:19.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:19.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:19.791 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:28:19.792 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:28:19.809 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:28:19.809 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:28:19.811 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:19.825 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:28:19.825 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:28:19.833 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:19.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:19.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:19.841 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:28:19.842 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:28:19.857 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:28:19.857 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:28:19.873 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:28:19.873 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:28:19.891 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:28:19.892 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:28:19.907 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:28:19.908 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:28:19.934 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:19.956 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:19.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:19.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:20.058 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:20.080 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:20.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:20.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:20.182 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:20.204 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:20.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:20.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:20.306 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:20.329 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:20.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:20.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:20.377 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:28:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:28:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:28:20.377 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:20 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T17:28:20.377 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:20 smithi053 ceph-mon[89963]: osdmap e225: 8 total, 8 up, 7 in 2024-08-17T17:28:20.377 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:20 smithi053 ceph-mon[89963]: from='client.35787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:28:20.378 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:20 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/315299906' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:28:20.431 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:20.455 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:20.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:20.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:20.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:20 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T17:28:20.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:20 smithi003 ceph-mon[103344]: osdmap e225: 8 total, 8 up, 7 in 2024-08-17T17:28:20.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:20 smithi003 ceph-mon[103344]: from='client.35787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:28:20.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:20 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/315299906' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:28:20.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:20 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T17:28:20.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:20 smithi003 ceph-mon[100615]: osdmap e225: 8 total, 8 up, 7 in 2024-08-17T17:28:20.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:20 smithi003 ceph-mon[100615]: from='client.35787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:28:20.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:20 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/315299906' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:28:20.556 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:20.579 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:20.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:20.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:20.635 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:28:20.635 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2024-08-17T17:28:20.635 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2024-08-17T17:28:20.635 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2024-08-17T17:28:20.635 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.635 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:59549 5.2 5:4e305829:::benchmark_data_smithi003_140793_object59548:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-17T17:28:20.635 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:19:49.258169+0000", 2024-08-17T17:28:20.635 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 511.37403830099998, 2024-08-17T17:28:20.635 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.10804612099999999, 2024-08-17T17:28:20.635 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:20.636 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:20.636 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:20.636 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:28:20.636 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:28:20.636 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 59549 2024-08-17T17:28:20.636 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.636 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:20.636 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.636 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:20.636 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.258169+0000", 2024-08-17T17:28:20.636 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.636 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.636 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.637 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:20.637 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.258169+0000", 2024-08-17T17:28:20.637 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.637 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.637 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.637 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:20.637 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.258168+0000", 2024-08-17T17:28:20.637 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:28:20.637 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.637 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.637 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:20.637 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.258247+0000", 2024-08-17T17:28:20.638 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.9614e-05 2024-08-17T17:28:20.638 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.638 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.638 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:20.638 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.258259+0000", 2024-08-17T17:28:20.638 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1985e-05 2024-08-17T17:28:20.638 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.638 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.638 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:20.638 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.258266+0000", 2024-08-17T17:28:20.638 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0119999999999999e-06 2024-08-17T17:28:20.638 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.639 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.639 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:20.639 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.258321+0000", 2024-08-17T17:28:20.639 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4846000000000003e-05 2024-08-17T17:28:20.639 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.639 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.639 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:20.639 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.258441+0000", 2024-08-17T17:28:20.639 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011964399999999999 2024-08-17T17:28:20.639 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.639 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.639 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:28:20.639 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.258570+0000", 2024-08-17T17:28:20.639 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000129443 2024-08-17T17:28:20.640 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.640 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.640 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.640 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.262760+0000", 2024-08-17T17:28:20.640 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0041898120000000002 2024-08-17T17:28:20.640 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.640 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.640 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:20.640 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.264445+0000", 2024-08-17T17:28:20.640 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001684472 2024-08-17T17:28:20.640 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.640 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.640 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.641 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.366103+0000", 2024-08-17T17:28:20.641 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.101658544 2024-08-17T17:28:20.641 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.641 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.641 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:20.641 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.366172+0000", 2024-08-17T17:28:20.641 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9239999999999994e-05 2024-08-17T17:28:20.641 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.641 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.641 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:20.641 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.366215+0000", 2024-08-17T17:28:20.641 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.261e-05 2024-08-17T17:28:20.641 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.641 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:20.641 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.642 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.642 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.642 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:59553 5.2 5:46b62487:::benchmark_data_smithi003_140793_object59552:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-17T17:28:20.642 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:19:49.258394+0000", 2024-08-17T17:28:20.642 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 511.373813025, 2024-08-17T17:28:20.642 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.107943279, 2024-08-17T17:28:20.642 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:20.642 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:20.642 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:20.642 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:28:20.642 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:28:20.642 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 59553 2024-08-17T17:28:20.642 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.642 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:20.643 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.643 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:20.643 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.258394+0000", 2024-08-17T17:28:20.643 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.643 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.643 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.643 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:20.643 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.258394+0000", 2024-08-17T17:28:20.643 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.643 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.643 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.643 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:20.643 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.258393+0000", 2024-08-17T17:28:20.643 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:28:20.643 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.644 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.644 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:20.644 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.258454+0000", 2024-08-17T17:28:20.644 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0856999999999997e-05 2024-08-17T17:28:20.644 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.644 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.644 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:20.644 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.258466+0000", 2024-08-17T17:28:20.644 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1653e-05 2024-08-17T17:28:20.644 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.644 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.644 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:20.644 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.258472+0000", 2024-08-17T17:28:20.644 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2099999999999998e-06 2024-08-17T17:28:20.644 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.645 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.645 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:20.645 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.260373+0000", 2024-08-17T17:28:20.645 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0019012059999999999 2024-08-17T17:28:20.645 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.645 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.645 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:20.645 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.260554+0000", 2024-08-17T17:28:20.645 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00018076 2024-08-17T17:28:20.645 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.645 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.645 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:28:20.645 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.260682+0000", 2024-08-17T17:28:20.645 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012760700000000001 2024-08-17T17:28:20.645 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.645 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.646 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:20.646 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.264451+0000", 2024-08-17T17:28:20.646 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0037690229999999998 2024-08-17T17:28:20.646 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.646 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.646 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.646 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.264953+0000", 2024-08-17T17:28:20.646 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00050264999999999999 2024-08-17T17:28:20.646 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.646 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.646 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.646 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.366254+0000", 2024-08-17T17:28:20.646 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.101301004 2024-08-17T17:28:20.646 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.647 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.647 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:20.647 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.366302+0000", 2024-08-17T17:28:20.647 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7330999999999998e-05 2024-08-17T17:28:20.647 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.647 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.647 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:20.647 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.366337+0000", 2024-08-17T17:28:20.647 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5905999999999998e-05 2024-08-17T17:28:20.647 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.647 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:20.647 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.647 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.647 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.648 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:59558 5.2 5:44b251fd:::benchmark_data_smithi003_140793_object59557:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-17T17:28:20.648 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:19:49.261652+0000", 2024-08-17T17:28:20.648 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 511.370554707, 2024-08-17T17:28:20.648 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.10960434500000001, 2024-08-17T17:28:20.648 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:20.648 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:20.648 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:20.648 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:28:20.648 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:28:20.648 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 59558 2024-08-17T17:28:20.648 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.648 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:20.648 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.648 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:20.649 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.261652+0000", 2024-08-17T17:28:20.649 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.649 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.649 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.649 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:20.649 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.261652+0000", 2024-08-17T17:28:20.649 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.649 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.649 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.649 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:20.649 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.261651+0000", 2024-08-17T17:28:20.649 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:28:20.649 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.649 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.649 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:20.650 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.261692+0000", 2024-08-17T17:28:20.650 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1383e-05 2024-08-17T17:28:20.650 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.650 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.650 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:20.650 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.261707+0000", 2024-08-17T17:28:20.650 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5065000000000001e-05 2024-08-17T17:28:20.650 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.650 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.650 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:20.650 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.261713+0000", 2024-08-17T17:28:20.650 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9100000000000002e-06 2024-08-17T17:28:20.651 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.651 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.651 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:20.651 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.262798+0000", 2024-08-17T17:28:20.651 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0010844559999999999 2024-08-17T17:28:20.651 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.651 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.651 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:20.651 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.262920+0000", 2024-08-17T17:28:20.651 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012288199999999999 2024-08-17T17:28:20.651 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.651 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.651 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:28:20.651 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.263072+0000", 2024-08-17T17:28:20.651 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000151337 2024-08-17T17:28:20.652 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.652 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.652 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:20.652 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.264558+0000", 2024-08-17T17:28:20.652 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001486575 2024-08-17T17:28:20.652 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.652 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.652 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.652 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.265079+0000", 2024-08-17T17:28:20.652 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00052085100000000002 2024-08-17T17:28:20.652 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.652 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.652 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.652 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.371169+0000", 2024-08-17T17:28:20.653 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.106089432 2024-08-17T17:28:20.653 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.653 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.653 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:20.653 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.371220+0000", 2024-08-17T17:28:20.653 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1038999999999997e-05 2024-08-17T17:28:20.653 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.653 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.653 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:20.653 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:49.371257+0000", 2024-08-17T17:28:20.653 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.7165000000000001e-05 2024-08-17T17:28:20.653 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.653 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:20.653 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.654 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.654 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.654 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:61377 5.2 5:4702e049:::benchmark_data_smithi003_140793_object61376:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-17T17:28:20.654 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:19:50.790321+0000", 2024-08-17T17:28:20.654 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 509.84188598200001, 2024-08-17T17:28:20.654 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.11930178399999999, 2024-08-17T17:28:20.654 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:20.654 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:20.654 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:20.654 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:28:20.654 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:28:20.654 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 61377 2024-08-17T17:28:20.654 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.655 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:20.655 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.655 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:20.655 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:50.790321+0000", 2024-08-17T17:28:20.655 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.655 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.655 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.655 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:20.655 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:50.790321+0000", 2024-08-17T17:28:20.655 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.655 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.655 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.655 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:20.655 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:50.790320+0000", 2024-08-17T17:28:20.656 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:28:20.656 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.656 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.656 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:20.656 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:50.790398+0000", 2024-08-17T17:28:20.656 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.8220999999999995e-05 2024-08-17T17:28:20.656 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.656 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.656 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:20.656 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:50.790405+0000", 2024-08-17T17:28:20.656 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3899999999999998e-06 2024-08-17T17:28:20.656 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.656 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.656 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:20.656 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:50.790407+0000", 2024-08-17T17:28:20.657 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8909999999999999e-06 2024-08-17T17:28:20.657 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.657 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.657 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:20.657 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:50.790558+0000", 2024-08-17T17:28:20.657 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000150397 2024-08-17T17:28:20.657 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.657 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.657 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:20.657 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:50.790646+0000", 2024-08-17T17:28:20.657 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.8444999999999998e-05 2024-08-17T17:28:20.657 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.657 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.657 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:28:20.658 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:50.790733+0000", 2024-08-17T17:28:20.658 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.6267999999999995e-05 2024-08-17T17:28:20.658 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.658 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.658 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.658 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:50.794895+0000", 2024-08-17T17:28:20.658 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0041628460000000004 2024-08-17T17:28:20.658 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.658 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.658 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:20.658 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:50.794946+0000", 2024-08-17T17:28:20.658 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0859000000000002e-05 2024-08-17T17:28:20.659 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.659 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.659 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.659 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:50.909518+0000", 2024-08-17T17:28:20.659 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.114571348 2024-08-17T17:28:20.659 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.659 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.659 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:20.659 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:50.909575+0000", 2024-08-17T17:28:20.659 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7114000000000002e-05 2024-08-17T17:28:20.659 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.659 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.659 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:20.659 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:50.909623+0000", 2024-08-17T17:28:20.660 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.8293e-05 2024-08-17T17:28:20.660 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.660 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:20.660 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.660 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.660 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.660 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:61379 5.2 5:41fc91fb:::benchmark_data_smithi003_140793_object61378:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-17T17:28:20.660 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:19:50.790417+0000", 2024-08-17T17:28:20.660 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 509.84179066899998, 2024-08-17T17:28:20.660 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.12019479, 2024-08-17T17:28:20.660 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:20.660 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:20.660 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:20.660 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:28:20.661 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:28:20.661 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 61379 2024-08-17T17:28:20.661 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.661 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:20.661 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.661 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:20.661 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:50.790417+0000", 2024-08-17T17:28:20.661 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.661 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.661 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.661 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:20.661 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:50.790417+0000", 2024-08-17T17:28:20.661 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.661 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.661 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.662 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:20.662 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:50.790416+0000", 2024-08-17T17:28:20.662 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:28:20.662 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.662 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.663 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:20.663 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:50.790472+0000", 2024-08-17T17:28:20.663 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5546e-05 2024-08-17T17:28:20.663 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.663 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.663 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:20.663 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:50.790477+0000", 2024-08-17T17:28:20.664 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8259999999999998e-06 2024-08-17T17:28:20.664 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.664 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.664 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:20.664 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:50.790482+0000", 2024-08-17T17:28:20.664 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3039999999999998e-06 2024-08-17T17:28:20.664 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.664 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.664 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:20.664 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:50.792657+0000", 2024-08-17T17:28:20.664 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0021749249999999999 2024-08-17T17:28:20.664 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.664 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.665 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:20.665 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:50.792777+0000", 2024-08-17T17:28:20.665 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012012599999999999 2024-08-17T17:28:20.665 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.665 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.665 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:28:20.665 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:50.793000+0000", 2024-08-17T17:28:20.665 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00022319700000000001 2024-08-17T17:28:20.665 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.665 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.665 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.665 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:50.794937+0000", 2024-08-17T17:28:20.665 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0019371919999999999 2024-08-17T17:28:20.665 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.665 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.666 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:20.666 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:50.795047+0000", 2024-08-17T17:28:20.666 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010942000000000001 2024-08-17T17:28:20.666 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.666 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.666 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.666 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:50.910527+0000", 2024-08-17T17:28:20.666 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.11548050999999999 2024-08-17T17:28:20.666 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.666 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.666 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:20.666 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:50.910575+0000", 2024-08-17T17:28:20.666 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7336999999999998e-05 2024-08-17T17:28:20.666 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.667 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.667 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:20.667 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:50.910611+0000", 2024-08-17T17:28:20.667 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6801999999999998e-05 2024-08-17T17:28:20.667 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.667 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:20.667 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.667 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.667 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.667 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:113760 5.3 5:c77f0da7:::benchmark_data_smithi003_140793_object11703:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-08-17T17:28:20.667 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:21:00.767571+0000", 2024-08-17T17:28:20.667 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 439.864636383, 2024-08-17T17:28:20.667 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.093483020999999999, 2024-08-17T17:28:20.667 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:20.668 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:20.668 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:20.668 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:28:20.668 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:28:20.668 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 113760 2024-08-17T17:28:20.668 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.668 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:20.668 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.668 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:20.668 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:00.767571+0000", 2024-08-17T17:28:20.668 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.668 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.668 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.668 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:20.669 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:00.767571+0000", 2024-08-17T17:28:20.669 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.669 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.669 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.669 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:20.669 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:00.767569+0000", 2024-08-17T17:28:20.669 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:28:20.669 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.669 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.669 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:20.669 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:00.767578+0000", 2024-08-17T17:28:20.669 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.7730000000000005e-06 2024-08-17T17:28:20.669 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.669 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.669 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:20.670 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:00.767591+0000", 2024-08-17T17:28:20.670 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2802e-05 2024-08-17T17:28:20.670 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.670 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.670 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:20.670 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:00.767598+0000", 2024-08-17T17:28:20.670 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.7900000000000002e-06 2024-08-17T17:28:20.670 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.670 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.670 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:20.670 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:00.777141+0000", 2024-08-17T17:28:20.670 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.009543213 2024-08-17T17:28:20.670 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.670 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.671 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for rw locks", 2024-08-17T17:28:20.671 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:00.777260+0000", 2024-08-17T17:28:20.671 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000119622 2024-08-17T17:28:20.671 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.671 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.671 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:20.671 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:00.802616+0000", 2024-08-17T17:28:20.671 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.025355998000000001 2024-08-17T17:28:20.671 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.671 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.671 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:20.671 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:00.802695+0000", 2024-08-17T17:28:20.671 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.8676000000000004e-05 2024-08-17T17:28:20.671 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.671 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.671 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:28:20.672 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:00.802836+0000", 2024-08-17T17:28:20.672 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000141326 2024-08-17T17:28:20.672 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.672 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.672 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:20.672 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:00.810852+0000", 2024-08-17T17:28:20.672 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0080153259999999997 2024-08-17T17:28:20.672 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.672 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.672 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.672 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:00.810884+0000", 2024-08-17T17:28:20.672 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2413999999999998e-05 2024-08-17T17:28:20.672 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.672 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.672 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.673 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:00.859286+0000", 2024-08-17T17:28:20.673 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.048401762000000001 2024-08-17T17:28:20.673 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.673 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.673 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:20.673 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:00.859341+0000", 2024-08-17T17:28:20.673 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4598999999999997e-05 2024-08-17T17:28:20.673 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.673 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.673 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:20.673 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:00.861054+0000", 2024-08-17T17:28:20.673 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00171327 2024-08-17T17:28:20.673 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.673 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:20.673 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.674 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.674 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.674 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:123796 5.3 5:ce1a31d7:::benchmark_data_smithi003_140793_object21739:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-08-17T17:28:20.674 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:21:16.067385+0000", 2024-08-17T17:28:20.674 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 424.56482208599999, 2024-08-17T17:28:20.674 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.089326504000000001, 2024-08-17T17:28:20.674 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:20.674 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:20.674 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:20.674 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:28:20.674 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:28:20.674 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 123796 2024-08-17T17:28:20.674 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.674 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:20.675 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.675 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:20.675 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.067385+0000", 2024-08-17T17:28:20.675 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.675 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.675 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.675 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:20.675 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.067385+0000", 2024-08-17T17:28:20.675 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.675 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.675 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.675 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:20.675 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.067385+0000", 2024-08-17T17:28:20.675 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:28:20.676 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.676 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.676 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:20.676 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.067388+0000", 2024-08-17T17:28:20.676 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2399999999999999e-06 2024-08-17T17:28:20.676 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.676 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.676 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:20.676 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.067400+0000", 2024-08-17T17:28:20.676 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2282e-05 2024-08-17T17:28:20.676 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.676 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.676 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:20.677 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.067405+0000", 2024-08-17T17:28:20.677 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3089999999999997e-06 2024-08-17T17:28:20.677 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.677 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.677 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:20.677 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.077540+0000", 2024-08-17T17:28:20.677 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010135467 2024-08-17T17:28:20.677 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.677 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.677 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:20.677 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.077655+0000", 2024-08-17T17:28:20.677 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000115523 2024-08-17T17:28:20.677 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.678 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.678 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:28:20.678 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.077816+0000", 2024-08-17T17:28:20.678 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016057800000000001 2024-08-17T17:28:20.678 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.678 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.678 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:20.678 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.093713+0000", 2024-08-17T17:28:20.678 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.015896987000000001 2024-08-17T17:28:20.678 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.678 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.678 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.679 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.120037+0000", 2024-08-17T17:28:20.679 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.026324009999999998 2024-08-17T17:28:20.679 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.679 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.679 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.679 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.155508+0000", 2024-08-17T17:28:20.679 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.035470698000000002 2024-08-17T17:28:20.679 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.679 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.679 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:20.679 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.155559+0000", 2024-08-17T17:28:20.679 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0744000000000001e-05 2024-08-17T17:28:20.679 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.679 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.680 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:20.680 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:21:16.156712+0000", 2024-08-17T17:28:20.680 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0011530990000000001 2024-08-17T17:28:20.680 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.680 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:20.680 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.680 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.680 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.680 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:153566 5.2 5:4f012f1e:::benchmark_data_smithi003_140793_object51509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-08-17T17:28:20.680 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:22:02.733431+0000", 2024-08-17T17:28:20.680 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 377.89877618000003, 2024-08-17T17:28:20.680 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.11735413, 2024-08-17T17:28:20.680 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:20.680 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:20.681 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:20.681 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:28:20.681 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:28:20.681 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 153566 2024-08-17T17:28:20.681 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.681 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:20.681 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.681 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:20.681 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.733431+0000", 2024-08-17T17:28:20.681 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.681 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.681 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.681 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:20.681 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.733431+0000", 2024-08-17T17:28:20.682 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.682 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.682 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.682 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:20.682 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.733430+0000", 2024-08-17T17:28:20.682 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:28:20.682 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.682 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.682 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:20.682 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.733436+0000", 2024-08-17T17:28:20.682 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4790000000000001e-06 2024-08-17T17:28:20.682 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.682 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.683 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:20.683 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.733453+0000", 2024-08-17T17:28:20.683 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6509000000000001e-05 2024-08-17T17:28:20.683 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.683 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.683 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:20.683 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.733459+0000", 2024-08-17T17:28:20.683 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0129999999999999e-06 2024-08-17T17:28:20.683 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.683 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.683 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:20.683 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.733505+0000", 2024-08-17T17:28:20.684 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6545999999999998e-05 2024-08-17T17:28:20.684 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.684 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.684 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:20.684 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.733633+0000", 2024-08-17T17:28:20.684 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000127322 2024-08-17T17:28:20.684 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.684 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.684 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:28:20.684 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.733753+0000", 2024-08-17T17:28:20.684 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000119916 2024-08-17T17:28:20.684 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.684 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.685 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.685 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.738221+0000", 2024-08-17T17:28:20.685 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0044680839999999998 2024-08-17T17:28:20.685 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.685 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.685 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:20.685 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.738249+0000", 2024-08-17T17:28:20.685 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8747000000000001e-05 2024-08-17T17:28:20.685 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.685 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.685 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.685 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.850691+0000", 2024-08-17T17:28:20.685 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.11244172199999999 2024-08-17T17:28:20.685 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.686 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.686 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:20.686 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.850745+0000", 2024-08-17T17:28:20.686 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4135000000000003e-05 2024-08-17T17:28:20.686 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.686 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.686 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:20.686 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.850785+0000", 2024-08-17T17:28:20.686 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9805000000000003e-05 2024-08-17T17:28:20.686 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.686 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:20.686 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.686 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.687 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.687 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:153570 5.2 5:4f5e89a3:::benchmark_data_smithi003_140793_object51513:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-08-17T17:28:20.687 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:22:02.743390+0000", 2024-08-17T17:28:20.687 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 377.88881748199998, 2024-08-17T17:28:20.687 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.1119482, 2024-08-17T17:28:20.687 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:20.687 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:20.687 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:20.687 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:28:20.687 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:28:20.687 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 153570 2024-08-17T17:28:20.687 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.688 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:20.688 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.688 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:20.688 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.743390+0000", 2024-08-17T17:28:20.688 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.688 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.688 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.688 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:20.688 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.743390+0000", 2024-08-17T17:28:20.688 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.688 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.688 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.688 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:20.688 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.743389+0000", 2024-08-17T17:28:20.689 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:28:20.689 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.689 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.689 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:20.689 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.743394+0000", 2024-08-17T17:28:20.689 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2900000000000002e-06 2024-08-17T17:28:20.689 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.689 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.689 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:20.689 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.743406+0000", 2024-08-17T17:28:20.689 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1779e-05 2024-08-17T17:28:20.689 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.689 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.690 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:20.690 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.743411+0000", 2024-08-17T17:28:20.690 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9729999999999997e-06 2024-08-17T17:28:20.690 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.690 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.690 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:20.690 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.743464+0000", 2024-08-17T17:28:20.690 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3025000000000001e-05 2024-08-17T17:28:20.690 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.690 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.690 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:20.690 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.743600+0000", 2024-08-17T17:28:20.690 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013659699999999999 2024-08-17T17:28:20.690 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.691 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.691 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:28:20.691 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.743744+0000", 2024-08-17T17:28:20.691 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014353900000000001 2024-08-17T17:28:20.691 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.691 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.691 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.691 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.748190+0000", 2024-08-17T17:28:20.691 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0044460039999999999 2024-08-17T17:28:20.691 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.691 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.691 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:20.691 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.748520+0000", 2024-08-17T17:28:20.692 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00033048799999999998 2024-08-17T17:28:20.692 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.692 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.692 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.692 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.855260+0000", 2024-08-17T17:28:20.692 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.106739537 2024-08-17T17:28:20.692 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.692 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.692 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:20.692 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.855303+0000", 2024-08-17T17:28:20.692 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3640000000000002e-05 2024-08-17T17:28:20.692 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.692 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.693 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:20.693 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.855338+0000", 2024-08-17T17:28:20.693 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4527999999999999e-05 2024-08-17T17:28:20.693 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.693 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:20.693 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.693 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.693 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.693 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:153583 5.2 5:4ca8c1fd:::benchmark_data_smithi003_140793_object51526:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-08-17T17:28:20.693 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:22:02.758291+0000", 2024-08-17T17:28:20.693 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 377.87391606199998, 2024-08-17T17:28:20.693 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.097153820000000002, 2024-08-17T17:28:20.693 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:20.694 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:20.694 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:20.694 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:28:20.694 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:28:20.694 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 153583 2024-08-17T17:28:20.694 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.694 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:20.694 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.694 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:20.694 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.758291+0000", 2024-08-17T17:28:20.694 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.694 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.694 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.694 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:20.695 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.758291+0000", 2024-08-17T17:28:20.695 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.695 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.695 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.695 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:20.695 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.758289+0000", 2024-08-17T17:28:20.695 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:28:20.695 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.695 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.695 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:20.695 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.758295+0000", 2024-08-17T17:28:20.695 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5670000000000002e-06 2024-08-17T17:28:20.695 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.695 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.695 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:20.696 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.758307+0000", 2024-08-17T17:28:20.696 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2469999999999999e-05 2024-08-17T17:28:20.696 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.696 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.696 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:20.696 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.758314+0000", 2024-08-17T17:28:20.696 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1199999999999999e-06 2024-08-17T17:28:20.696 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.696 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.696 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:20.696 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.758370+0000", 2024-08-17T17:28:20.696 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6428000000000003e-05 2024-08-17T17:28:20.696 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.696 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.697 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:20.697 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.758474+0000", 2024-08-17T17:28:20.697 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000103798 2024-08-17T17:28:20.697 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.697 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.697 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:28:20.697 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.759173+0000", 2024-08-17T17:28:20.697 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00069921000000000002 2024-08-17T17:28:20.697 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.697 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.697 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.697 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.766158+0000", 2024-08-17T17:28:20.697 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.006985046 2024-08-17T17:28:20.697 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.698 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.698 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:20.698 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.766345+0000", 2024-08-17T17:28:20.698 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00018708799999999999 2024-08-17T17:28:20.698 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.698 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.698 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.698 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.855380+0000", 2024-08-17T17:28:20.698 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.089034778999999994 2024-08-17T17:28:20.698 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.698 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.698 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:20.698 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.855415+0000", 2024-08-17T17:28:20.698 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5494999999999998e-05 2024-08-17T17:28:20.699 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.699 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.699 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:20.699 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.855445+0000", 2024-08-17T17:28:20.699 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9507999999999999e-05 2024-08-17T17:28:20.699 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.699 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:20.699 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.699 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.699 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.699 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:153590 5.2 5:4771b403:::benchmark_data_smithi003_140793_object51533:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-08-17T17:28:20.699 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:22:02.771087+0000", 2024-08-17T17:28:20.699 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 377.86112015999998, 2024-08-17T17:28:20.699 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.084568156000000005, 2024-08-17T17:28:20.699 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:20.700 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:20.700 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:20.700 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:28:20.700 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:28:20.700 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 153590 2024-08-17T17:28:20.700 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.700 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:20.700 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.700 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:20.700 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.771087+0000", 2024-08-17T17:28:20.700 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.700 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.700 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.700 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:20.701 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.771087+0000", 2024-08-17T17:28:20.701 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.701 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.701 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.701 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:20.701 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.771086+0000", 2024-08-17T17:28:20.701 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:28:20.701 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.701 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.701 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:20.701 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.771093+0000", 2024-08-17T17:28:20.701 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3520000000000001e-06 2024-08-17T17:28:20.701 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.701 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.701 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:20.702 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.771106+0000", 2024-08-17T17:28:20.702 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2816999999999999e-05 2024-08-17T17:28:20.702 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.702 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.702 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:20.702 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.771112+0000", 2024-08-17T17:28:20.702 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1e-06 2024-08-17T17:28:20.702 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.702 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.702 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:20.702 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.771218+0000", 2024-08-17T17:28:20.702 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000106001 2024-08-17T17:28:20.702 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.702 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.703 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:20.703 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.771337+0000", 2024-08-17T17:28:20.703 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000119227 2024-08-17T17:28:20.703 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.703 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.703 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:28:20.703 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.771453+0000", 2024-08-17T17:28:20.703 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011580499999999999 2024-08-17T17:28:20.703 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.703 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.703 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:20.703 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.775447+0000", 2024-08-17T17:28:20.703 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0039939399999999996 2024-08-17T17:28:20.703 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.703 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.703 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.704 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.776543+0000", 2024-08-17T17:28:20.704 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0010957720000000001 2024-08-17T17:28:20.704 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.704 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.704 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.704 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.855568+0000", 2024-08-17T17:28:20.704 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.079025552999999998 2024-08-17T17:28:20.704 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.704 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.704 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:20.704 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.855620+0000", 2024-08-17T17:28:20.704 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2027999999999997e-05 2024-08-17T17:28:20.704 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.704 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.704 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:20.705 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.855655+0000", 2024-08-17T17:28:20.705 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5002000000000001e-05 2024-08-17T17:28:20.705 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.705 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:20.705 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.705 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.705 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.705 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:176027 5.3 5:c374dd00:::benchmark_data_smithi003_140793_object73970:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-08-17T17:28:20.705 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:22:36.724311+0000", 2024-08-17T17:28:20.705 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 343.90789602699999, 2024-08-17T17:28:20.705 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.089565759999999994, 2024-08-17T17:28:20.705 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:20.705 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:20.705 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:20.706 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:28:20.706 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:28:20.706 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 176027 2024-08-17T17:28:20.706 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.706 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:20.706 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.706 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:20.706 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:36.724311+0000", 2024-08-17T17:28:20.706 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.706 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.706 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.706 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:20.706 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:36.724311+0000", 2024-08-17T17:28:20.707 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.707 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.707 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.707 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:20.707 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:36.724309+0000", 2024-08-17T17:28:20.707 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:28:20.707 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.707 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.707 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:20.707 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:36.724318+0000", 2024-08-17T17:28:20.707 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.4300000000000006e-06 2024-08-17T17:28:20.707 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.707 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.707 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:20.707 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:36.724335+0000", 2024-08-17T17:28:20.708 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6949e-05 2024-08-17T17:28:20.708 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.708 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.708 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:20.708 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:36.724341+0000", 2024-08-17T17:28:20.708 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1299999999999998e-06 2024-08-17T17:28:20.708 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.708 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.708 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:20.708 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:36.724389+0000", 2024-08-17T17:28:20.708 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.8440999999999999e-05 2024-08-17T17:28:20.708 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.708 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.708 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:20.708 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:36.724520+0000", 2024-08-17T17:28:20.709 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000130901 2024-08-17T17:28:20.709 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.709 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.709 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:28:20.709 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:36.724666+0000", 2024-08-17T17:28:20.709 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014577700000000001 2024-08-17T17:28:20.709 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.709 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.709 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:20.709 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:36.728981+0000", 2024-08-17T17:28:20.709 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.004315106 2024-08-17T17:28:20.709 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.709 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.709 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.709 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:36.807924+0000", 2024-08-17T17:28:20.710 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.078942929999999994 2024-08-17T17:28:20.710 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.710 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.710 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.710 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:36.813793+0000", 2024-08-17T17:28:20.710 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0058691189999999999 2024-08-17T17:28:20.710 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.710 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.710 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:20.710 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:36.813840+0000", 2024-08-17T17:28:20.710 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6625999999999997e-05 2024-08-17T17:28:20.710 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.710 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.710 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:20.710 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:36.813877+0000", 2024-08-17T17:28:20.711 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.7101000000000003e-05 2024-08-17T17:28:20.711 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.711 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:20.711 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.711 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.711 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.711 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:176034 5.3 5:c263f8f3:::benchmark_data_smithi003_140793_object73977:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-08-17T17:28:20.711 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:22:36.733474+0000", 2024-08-17T17:28:20.711 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 343.89873303399997, 2024-08-17T17:28:20.711 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.089284200999999994, 2024-08-17T17:28:20.711 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:20.711 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:20.711 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:20.711 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:28:20.712 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:28:20.712 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 176034 2024-08-17T17:28:20.712 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.712 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:20.712 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.712 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:20.712 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:36.733474+0000", 2024-08-17T17:28:20.712 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.712 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.712 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.712 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:20.713 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:36.733474+0000", 2024-08-17T17:28:20.713 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.713 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.713 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.713 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:20.713 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:36.733473+0000", 2024-08-17T17:28:20.713 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:28:20.713 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.713 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.713 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:20.713 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:36.733478+0000", 2024-08-17T17:28:20.714 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0309999999999998e-06 2024-08-17T17:28:20.714 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.714 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.714 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:20.714 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:36.733487+0000", 2024-08-17T17:28:20.714 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1770000000000004e-06 2024-08-17T17:28:20.714 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.714 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.714 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:20.714 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:36.733491+0000", 2024-08-17T17:28:20.714 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.7740000000000002e-06 2024-08-17T17:28:20.715 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.715 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.715 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:20.715 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:36.733527+0000", 2024-08-17T17:28:20.715 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5574999999999997e-05 2024-08-17T17:28:20.715 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.715 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.715 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:20.715 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:36.733615+0000", 2024-08-17T17:28:20.715 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.7726999999999995e-05 2024-08-17T17:28:20.715 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.715 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.715 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:28:20.715 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:36.733688+0000", 2024-08-17T17:28:20.716 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3720000000000006e-05 2024-08-17T17:28:20.716 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.716 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.716 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:20.716 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:36.736884+0000", 2024-08-17T17:28:20.716 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0031952510000000001 2024-08-17T17:28:20.716 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.716 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.716 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.716 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:36.813940+0000", 2024-08-17T17:28:20.716 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.077056155000000001 2024-08-17T17:28:20.716 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.716 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.716 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.716 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:36.822701+0000", 2024-08-17T17:28:20.716 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0087616080000000006 2024-08-17T17:28:20.717 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.717 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.717 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:20.717 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:36.822735+0000", 2024-08-17T17:28:20.717 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3519000000000002e-05 2024-08-17T17:28:20.717 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.717 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.717 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:20.717 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:36.822758+0000", 2024-08-17T17:28:20.717 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3561999999999999e-05 2024-08-17T17:28:20.717 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.717 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:20.717 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.717 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.718 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.718 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:176345 5.3 5:c50318b5:::benchmark_data_smithi003_140793_object74288:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-08-17T17:28:20.718 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:22:37.196942+0000", 2024-08-17T17:28:20.718 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 343.43526511900001, 2024-08-17T17:28:20.718 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.087642827000000006, 2024-08-17T17:28:20.718 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:20.718 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:20.718 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:20.718 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:28:20.718 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:28:20.718 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 176345 2024-08-17T17:28:20.718 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.718 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:20.719 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.719 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:20.719 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:37.196942+0000", 2024-08-17T17:28:20.719 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.719 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.719 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.719 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:20.719 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:37.196942+0000", 2024-08-17T17:28:20.719 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.719 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.719 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.719 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:20.719 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:37.196941+0000", 2024-08-17T17:28:20.719 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:28:20.719 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.720 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.720 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:20.720 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:37.196947+0000", 2024-08-17T17:28:20.720 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4529999999999999e-06 2024-08-17T17:28:20.720 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.720 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.720 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:20.720 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:37.196958+0000", 2024-08-17T17:28:20.720 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0796e-05 2024-08-17T17:28:20.720 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.720 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.720 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:20.720 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:37.196962+0000", 2024-08-17T17:28:20.720 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4479999999999996e-06 2024-08-17T17:28:20.721 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.721 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.721 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:20.721 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:37.197007+0000", 2024-08-17T17:28:20.721 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4378000000000001e-05 2024-08-17T17:28:20.721 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.721 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.721 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:20.721 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:37.197134+0000", 2024-08-17T17:28:20.721 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000127634 2024-08-17T17:28:20.721 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.721 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.721 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:28:20.721 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:37.197254+0000", 2024-08-17T17:28:20.721 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000120104 2024-08-17T17:28:20.721 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.722 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.722 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:20.722 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:37.201799+0000", 2024-08-17T17:28:20.722 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0045448659999999998 2024-08-17T17:28:20.722 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.722 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.722 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.722 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:37.205062+0000", 2024-08-17T17:28:20.722 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0032628150000000001 2024-08-17T17:28:20.722 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.722 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.722 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.722 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:37.284526+0000", 2024-08-17T17:28:20.722 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.079464149999999997 2024-08-17T17:28:20.722 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.723 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.723 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:20.723 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:37.284561+0000", 2024-08-17T17:28:20.723 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4484999999999999e-05 2024-08-17T17:28:20.723 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.723 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.723 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:20.723 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:37.284585+0000", 2024-08-17T17:28:20.723 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.421e-05 2024-08-17T17:28:20.723 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.723 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:20.723 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.723 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.724 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.724 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:203831 5.3 5:ce38c9d1:::benchmark_data_smithi003_140793_object101774:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-08-17T17:28:20.724 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:23:15.893546+0000", 2024-08-17T17:28:20.724 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 304.73866140799998, 2024-08-17T17:28:20.724 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.088895242999999999, 2024-08-17T17:28:20.724 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:20.724 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:20.724 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:20.724 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:28:20.724 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:28:20.724 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 203831 2024-08-17T17:28:20.724 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.724 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:20.725 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.725 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:20.725 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.893546+0000", 2024-08-17T17:28:20.725 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.725 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.725 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.725 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:20.725 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.893546+0000", 2024-08-17T17:28:20.725 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.725 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.725 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.725 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:20.726 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.893544+0000", 2024-08-17T17:28:20.726 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:28:20.726 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.726 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.726 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:20.726 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.893551+0000", 2024-08-17T17:28:20.726 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5239999999999997e-06 2024-08-17T17:28:20.726 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.726 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.726 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:20.726 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.893567+0000", 2024-08-17T17:28:20.726 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6226999999999999e-05 2024-08-17T17:28:20.727 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.727 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.727 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:20.727 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.893575+0000", 2024-08-17T17:28:20.727 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.2530000000000004e-06 2024-08-17T17:28:20.727 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.727 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.727 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:20.727 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.893901+0000", 2024-08-17T17:28:20.727 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00032573900000000002 2024-08-17T17:28:20.727 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.727 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.728 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:20.728 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.894008+0000", 2024-08-17T17:28:20.728 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000107225 2024-08-17T17:28:20.728 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.728 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.728 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:28:20.728 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.894106+0000", 2024-08-17T17:28:20.728 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.7213999999999995e-05 2024-08-17T17:28:20.728 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.728 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.728 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:20.729 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.905560+0000", 2024-08-17T17:28:20.729 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011454041999999999 2024-08-17T17:28:20.729 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.729 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.729 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.729 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.905595+0000", 2024-08-17T17:28:20.729 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5508e-05 2024-08-17T17:28:20.729 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.729 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.729 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.729 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.982339+0000", 2024-08-17T17:28:20.729 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.076743594999999998 2024-08-17T17:28:20.729 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.729 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.730 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:20.730 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.982400+0000", 2024-08-17T17:28:20.730 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1270000000000001e-05 2024-08-17T17:28:20.730 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.730 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.730 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:20.730 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.982441+0000", 2024-08-17T17:28:20.730 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.108e-05 2024-08-17T17:28:20.730 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.730 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:20.730 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.730 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.730 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.730 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:203832 5.3 5:c9630e6b:::benchmark_data_smithi003_140793_object101775:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-08-17T17:28:20.731 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:23:15.893901+0000", 2024-08-17T17:28:20.731 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 304.738306077, 2024-08-17T17:28:20.731 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.088645783000000006, 2024-08-17T17:28:20.731 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:20.731 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:20.731 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:20.731 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:28:20.731 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:28:20.731 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 203832 2024-08-17T17:28:20.731 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.731 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:20.731 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.731 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:20.732 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.893901+0000", 2024-08-17T17:28:20.732 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.732 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.732 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.732 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:20.732 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.893901+0000", 2024-08-17T17:28:20.732 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.732 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:20.733 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.733 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.733 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:20.733 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.893900+0000", 2024-08-17T17:28:20.733 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:28:20.734 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.734 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.734 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:20.734 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.893913+0000", 2024-08-17T17:28:20.734 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2489e-05 2024-08-17T17:28:20.734 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.734 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.734 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:20.734 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.893924+0000", 2024-08-17T17:28:20.734 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1229999999999999e-05 2024-08-17T17:28:20.734 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.734 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.734 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:20.734 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.893929+0000", 2024-08-17T17:28:20.735 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5489999999999997e-06 2024-08-17T17:28:20.735 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.735 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.735 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:20.735 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.900886+0000", 2024-08-17T17:28:20.735 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0069563180000000004 2024-08-17T17:28:20.735 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.735 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.735 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:20.735 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.901009+0000", 2024-08-17T17:28:20.735 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012338599999999999 2024-08-17T17:28:20.735 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.736 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.736 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:28:20.736 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.901124+0000", 2024-08-17T17:28:20.736 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000114372 2024-08-17T17:28:20.736 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.736 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.736 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:20.736 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.906225+0000", 2024-08-17T17:28:20.736 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0051016220000000001 2024-08-17T17:28:20.736 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.736 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.736 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.736 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.907309+0000", 2024-08-17T17:28:20.736 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0010833500000000001 2024-08-17T17:28:20.736 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.737 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.737 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.737 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.982483+0000", 2024-08-17T17:28:20.737 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.075174793000000004 2024-08-17T17:28:20.737 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.737 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.737 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:20.737 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.982514+0000", 2024-08-17T17:28:20.737 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0355e-05 2024-08-17T17:28:20.737 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.737 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.737 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:20.737 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.982547+0000", 2024-08-17T17:28:20.737 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3213999999999997e-05 2024-08-17T17:28:20.737 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.738 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:20.738 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.738 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.738 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.738 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:203867 5.2 5:4646fd85:::benchmark_data_smithi003_140793_object101810:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-08-17T17:28:20.738 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:23:15.934270+0000", 2024-08-17T17:28:20.738 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 304.69793755799998, 2024-08-17T17:28:20.738 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.093295378999999998, 2024-08-17T17:28:20.738 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:20.738 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:20.738 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:20.738 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:28:20.738 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:28:20.739 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 203867 2024-08-17T17:28:20.739 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.739 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:20.739 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.739 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:20.739 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.934270+0000", 2024-08-17T17:28:20.739 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.739 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.739 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.739 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:20.739 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.934270+0000", 2024-08-17T17:28:20.739 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.739 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.739 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.740 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:20.740 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.934267+0000", 2024-08-17T17:28:20.740 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999976 2024-08-17T17:28:20.740 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.740 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.740 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:20.740 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.934277+0000", 2024-08-17T17:28:20.740 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.9089999999999995e-06 2024-08-17T17:28:20.740 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.740 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.740 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:20.740 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.934299+0000", 2024-08-17T17:28:20.740 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1902000000000001e-05 2024-08-17T17:28:20.740 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.741 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.741 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:20.741 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.934308+0000", 2024-08-17T17:28:20.741 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.8400000000000001e-06 2024-08-17T17:28:20.741 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.741 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.741 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:20.741 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.934362+0000", 2024-08-17T17:28:20.741 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4317999999999997e-05 2024-08-17T17:28:20.741 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.741 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.741 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:20.741 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.934490+0000", 2024-08-17T17:28:20.741 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012787900000000001 2024-08-17T17:28:20.742 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.742 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.742 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:28:20.742 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.934622+0000", 2024-08-17T17:28:20.742 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013141300000000001 2024-08-17T17:28:20.742 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.742 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.742 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.742 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.938764+0000", 2024-08-17T17:28:20.742 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0041419919999999997 2024-08-17T17:28:20.742 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.742 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.742 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:20.743 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:15.938838+0000", 2024-08-17T17:28:20.743 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.4420999999999998e-05 2024-08-17T17:28:20.743 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.743 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.743 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:20.743 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:16.027474+0000", 2024-08-17T17:28:20.743 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.088636309999999996 2024-08-17T17:28:20.743 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.743 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.743 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:20.743 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:16.027521+0000", 2024-08-17T17:28:20.743 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7046000000000003e-05 2024-08-17T17:28:20.743 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.743 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.743 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:20.744 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:23:16.027565+0000", 2024-08-17T17:28:20.744 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3612000000000002e-05 2024-08-17T17:28:20.744 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.744 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:20.744 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.744 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.744 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.744 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.45491.0:131824 6.2 e212/198 6:450b32f2:::benchmark_data_smithi003_175043_object33175:head v 212'8475, mlcod=0'0)", 2024-08-17T17:28:20.744 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:26:42.166400+0000", 2024-08-17T17:28:20.744 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 98.465807076999994, 2024-08-17T17:28:20.744 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.099398654000000003, 2024-08-17T17:28:20.744 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:20.744 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:20.744 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:20.744 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.745 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:20.745 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.166400+0000", 2024-08-17T17:28:20.745 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.745 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.745 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.745 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:20.745 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.166400+0000", 2024-08-17T17:28:20.745 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.745 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.745 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.745 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:20.745 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.166400+0000", 2024-08-17T17:28:20.745 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:28:20.745 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.745 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.745 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:20.746 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.166405+0000", 2024-08-17T17:28:20.746 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0830000000000003e-06 2024-08-17T17:28:20.746 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.746 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.746 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:20.746 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.166412+0000", 2024-08-17T17:28:20.746 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7530000000000001e-06 2024-08-17T17:28:20.746 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.746 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.746 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:20.746 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.166416+0000", 2024-08-17T17:28:20.746 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6069999999999999e-06 2024-08-17T17:28:20.746 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.747 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.747 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:20.747 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.166449+0000", 2024-08-17T17:28:20.747 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2462000000000002e-05 2024-08-17T17:28:20.747 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.747 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.747 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:20.747 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.166464+0000", 2024-08-17T17:28:20.747 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5637e-05 2024-08-17T17:28:20.747 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.747 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.747 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:20.747 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.170152+0000", 2024-08-17T17:28:20.748 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0036880039999999999 2024-08-17T17:28:20.748 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.748 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.748 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:20.748 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.265799+0000", 2024-08-17T17:28:20.748 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.095646615000000004 2024-08-17T17:28:20.748 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.748 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:20.748 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.748 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.748 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.748 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.45491.0:131827 6.6 e212/207 6:68c53fdf:::benchmark_data_smithi003_175043_object33178:head v 212'8072, mlcod=212'8070)", 2024-08-17T17:28:20.748 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:26:42.170765+0000", 2024-08-17T17:28:20.749 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 98.461441715999996, 2024-08-17T17:28:20.749 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.10206145599999999, 2024-08-17T17:28:20.749 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:20.749 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:20.749 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:20.749 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.749 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:20.749 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.170765+0000", 2024-08-17T17:28:20.749 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.749 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.749 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.749 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:20.749 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.170765+0000", 2024-08-17T17:28:20.749 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.749 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.750 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.750 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:20.750 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.170765+0000", 2024-08-17T17:28:20.750 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:28:20.750 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.750 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.750 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:20.750 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.170770+0000", 2024-08-17T17:28:20.750 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.8019999999999999e-06 2024-08-17T17:28:20.750 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.750 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.750 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:20.750 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.265511+0000", 2024-08-17T17:28:20.750 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.094741685000000006 2024-08-17T17:28:20.751 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.751 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.751 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:20.751 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.265531+0000", 2024-08-17T17:28:20.751 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9959e-05 2024-08-17T17:28:20.751 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.751 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.751 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:20.751 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.265763+0000", 2024-08-17T17:28:20.751 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00023125899999999999 2024-08-17T17:28:20.751 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.751 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.751 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:20.751 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.265948+0000", 2024-08-17T17:28:20.751 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000185105 2024-08-17T17:28:20.752 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.752 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.752 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:20.752 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.272764+0000", 2024-08-17T17:28:20.752 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0068160190000000004 2024-08-17T17:28:20.752 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.752 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.752 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:20.752 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.272827+0000", 2024-08-17T17:28:20.752 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3184999999999999e-05 2024-08-17T17:28:20.752 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.752 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:20.752 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.752 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.752 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.753 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.45491.0:131829 6.1 e212/178 6:82d98170:::benchmark_data_smithi003_175043_object33180:head v 212'8326, mlcod=212'8324)", 2024-08-17T17:28:20.753 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:26:42.171012+0000", 2024-08-17T17:28:20.753 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 98.461194898000002, 2024-08-17T17:28:20.753 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.10184240999999999, 2024-08-17T17:28:20.753 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:20.753 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:20.753 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:20.753 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.753 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:20.753 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.171012+0000", 2024-08-17T17:28:20.753 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.753 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.753 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.753 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:20.753 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.171012+0000", 2024-08-17T17:28:20.753 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:20.754 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.754 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.754 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:20.754 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.171012+0000", 2024-08-17T17:28:20.754 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:28:20.754 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.754 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.754 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:20.754 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.171017+0000", 2024-08-17T17:28:20.754 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8599999999999998e-06 2024-08-17T17:28:20.754 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.754 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.754 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:20.754 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.265874+0000", 2024-08-17T17:28:20.755 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.094856313999999997 2024-08-17T17:28:20.755 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.755 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.755 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:20.755 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.265883+0000", 2024-08-17T17:28:20.755 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.0999999999999993e-06 2024-08-17T17:28:20.755 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.755 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.755 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:20.755 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.265966+0000", 2024-08-17T17:28:20.755 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.2618000000000001e-05 2024-08-17T17:28:20.755 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.755 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.755 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:20.755 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.265987+0000", 2024-08-17T17:28:20.756 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1098999999999999e-05 2024-08-17T17:28:20.756 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.756 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.756 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:20.756 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.272820+0000", 2024-08-17T17:28:20.756 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0068330480000000004 2024-08-17T17:28:20.756 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:20.756 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:20.756 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:20.756 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.272855+0000", 2024-08-17T17:28:20.756 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5049999999999998e-05 2024-08-17T17:28:20.756 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.756 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:20.756 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.756 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:20.757 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:20.757 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:28:20.780 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:20.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:20.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:20.881 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:20.906 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:20.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:20.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:21.008 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:21.031 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:21.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:21.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:21.133 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:21.159 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:21.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:21.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:21.186 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-08-17T17:28:21.187 INFO:tasks.radosbench:Beginning radosbench... 2024-08-17T17:28:21.187 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-08-17T17:28:21.187 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd pool create unique_pool_2 16 2024-08-17T17:28:21.212 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:21 smithi053 ceph-mon[89963]: pgmap v609: 73 pgs: 73 active+clean; 4.6 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:21.261 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:21.291 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:21.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:21.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:21.393 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:21.417 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:21.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:21.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:21.519 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:21.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:21 smithi003 ceph-mon[103344]: pgmap v609: 73 pgs: 73 active+clean; 4.6 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:21.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:21 smithi003 ceph-mon[100615]: pgmap v609: 73 pgs: 73 active+clean; 4.6 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:21.541 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:21.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:21.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:21.643 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:21.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:21.666 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:21.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:21.766 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:21.793 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:21.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:21.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:21.807 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":608,"stamp":"2024-08-17T17:28:19.129828+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790221,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":6376,"num_read_kb":13833,"num_write":1122,"num_write_kb":15039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":11937158,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1181,"ondisk_log_size":1181,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":273,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2967128,"kb_used_data":699648,"kb_used_omap":33,"kb_used_meta":2267358,"kb_avail":653248936,"statfs":{"total":671965249536,"available":668926910464,"internally_reserved":0,"allocated":716439552,"data_stored":707512434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34068,"internal_metadata":2321775340},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1728,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-197294,"num_write_kb":-6313345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-19190,"num_bytes_recovered":-764084224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.022495"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":300,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.120984+0000","last_change":"2024-08-17T17:17:32.495652+0000","last_active":"2024-08-17T17:28:16.120984+0000","last_peered":"2024-08-17T17:28:16.120984+0000","last_clean":"2024-08-17T17:28:16.120984+0000","last_became_active":"2024-08-17T17:17:32.495218+0000","last_became_peered":"2024-08-17T17:17:32.495218+0000","last_unstale":"2024-08-17T17:28:16.120984+0000","last_undegraded":"2024-08-17T17:28:16.120984+0000","last_fullsized":"2024-08-17T17:28:16.120984+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:11:45.821282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":200,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.122894+0000","last_change":"2024-08-17T17:18:59.424887+0000","last_active":"2024-08-17T17:28:16.122894+0000","last_peered":"2024-08-17T17:28:16.122894+0000","last_clean":"2024-08-17T17:28:16.122894+0000","last_became_active":"2024-08-17T17:18:59.411539+0000","last_became_peered":"2024-08-17T17:18:59.411539+0000","last_unstale":"2024-08-17T17:28:16.122894+0000","last_undegraded":"2024-08-17T17:28:16.122894+0000","last_fullsized":"2024-08-17T17:28:16.122894+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:46:20.082241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":40,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.114536+0000","last_change":"2024-08-17T17:24:53.460742+0000","last_active":"2024-08-17T17:28:16.114536+0000","last_peered":"2024-08-17T17:28:16.114536+0000","last_clean":"2024-08-17T17:28:16.114536+0000","last_became_active":"2024-08-17T17:24:53.460365+0000","last_became_peered":"2024-08-17T17:24:53.460365+0000","last_unstale":"2024-08-17T17:28:16.114536+0000","last_undegraded":"2024-08-17T17:28:16.114536+0000","last_fullsized":"2024-08-17T17:28:16.114536+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:24:08.116968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":113,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-08-17T17:28:17.800351+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:28:17.800351+0000","last_peered":"2024-08-17T17:28:17.800351+0000","last_clean":"2024-08-17T17:28:17.800351+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:28:17.800351+0000","last_undegraded":"2024-08-17T17:28:17.800351+0000","last_fullsized":"2024-08-17T17:28:17.800351+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":113,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-08-17T17:28:17.800561+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:28:17.800561+0000","last_peered":"2024-08-17T17:28:17.800561+0000","last_clean":"2024-08-17T17:28:17.800561+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:28:17.800561+0000","last_undegraded":"2024-08-17T17:28:17.800561+0000","last_fullsized":"2024-08-17T17:28:17.800561+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":175,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.122870+0000","last_change":"2024-08-17T17:20:49.258206+0000","last_active":"2024-08-17T17:28:16.122870+0000","last_peered":"2024-08-17T17:28:16.122870+0000","last_clean":"2024-08-17T17:28:16.122870+0000","last_became_active":"2024-08-17T17:20:49.257761+0000","last_became_peered":"2024-08-17T17:20:49.257761+0000","last_unstale":"2024-08-17T17:28:16.122870+0000","last_undegraded":"2024-08-17T17:28:16.122870+0000","last_fullsized":"2024-08-17T17:28:16.122870+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:38:35.363748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":314,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.117784+0000","last_change":"2024-08-17T17:19:57.514773+0000","last_active":"2024-08-17T17:28:16.117784+0000","last_peered":"2024-08-17T17:28:16.117784+0000","last_clean":"2024-08-17T17:28:16.117784+0000","last_became_active":"2024-08-17T17:19:57.513839+0000","last_became_peered":"2024-08-17T17:19:57.513839+0000","last_unstale":"2024-08-17T17:28:16.117784+0000","last_undegraded":"2024-08-17T17:28:16.117784+0000","last_fullsized":"2024-08-17T17:28:16.117784+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:42:48.063169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":270,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.122261+0000","last_change":"2024-08-17T17:24:53.465875+0000","last_active":"2024-08-17T17:28:16.122261+0000","last_peered":"2024-08-17T17:28:16.122261+0000","last_clean":"2024-08-17T17:28:16.122261+0000","last_became_active":"2024-08-17T17:24:53.458129+0000","last_became_peered":"2024-08-17T17:24:53.458129+0000","last_unstale":"2024-08-17T17:28:16.122261+0000","last_undegraded":"2024-08-17T17:28:16.122261+0000","last_fullsized":"2024-08-17T17:28:16.122261+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:51:33.584869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":136,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.123197+0000","last_change":"2024-08-17T17:24:06.597382+0000","last_active":"2024-08-17T17:28:16.123197+0000","last_peered":"2024-08-17T17:28:16.123197+0000","last_clean":"2024-08-17T17:28:16.123197+0000","last_became_active":"2024-08-17T17:24:06.588881+0000","last_became_peered":"2024-08-17T17:24:06.588881+0000","last_unstale":"2024-08-17T17:28:16.123197+0000","last_undegraded":"2024-08-17T17:28:16.123197+0000","last_fullsized":"2024-08-17T17:28:16.123197+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:27.240751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":50,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.114208+0000","last_change":"2024-08-17T17:25:04.503780+0000","last_active":"2024-08-17T17:28:16.114208+0000","last_peered":"2024-08-17T17:28:16.114208+0000","last_clean":"2024-08-17T17:28:16.114208+0000","last_became_active":"2024-08-17T17:25:04.503503+0000","last_became_peered":"2024-08-17T17:25:04.503503+0000","last_unstale":"2024-08-17T17:28:16.114208+0000","last_undegraded":"2024-08-17T17:28:16.114208+0000","last_fullsized":"2024-08-17T17:28:16.114208+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:04:48.171216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":122,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.123392+0000","last_change":"2024-08-17T17:24:06.599023+0000","last_active":"2024-08-17T17:28:16.123392+0000","last_peered":"2024-08-17T17:28:16.123392+0000","last_clean":"2024-08-17T17:28:16.123392+0000","last_became_active":"2024-08-17T17:24:06.588954+0000","last_became_peered":"2024-08-17T17:24:06.588954+0000","last_unstale":"2024-08-17T17:28:16.123392+0000","last_undegraded":"2024-08-17T17:28:16.123392+0000","last_fullsized":"2024-08-17T17:28:16.123392+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:20:51.445576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":253,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-08-17T17:28:15.213206+0000","last_change":"2024-08-17T17:20:49.271540+0000","last_active":"2024-08-17T17:28:15.213206+0000","last_peered":"2024-08-17T17:28:15.213206+0000","last_clean":"2024-08-17T17:28:15.213206+0000","last_became_active":"2024-08-17T17:20:49.271130+0000","last_became_peered":"2024-08-17T17:20:49.271130+0000","last_unstale":"2024-08-17T17:28:15.213206+0000","last_undegraded":"2024-08-17T17:28:15.213206+0000","last_fullsized":"2024-08-17T17:28:15.213206+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:05.386705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":194,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.123035+0000","last_change":"2024-08-17T17:24:53.467526+0000","last_active":"2024-08-17T17:28:16.123035+0000","last_peered":"2024-08-17T17:28:16.123035+0000","last_clean":"2024-08-17T17:28:16.123035+0000","last_became_active":"2024-08-17T17:24:53.460631+0000","last_became_peered":"2024-08-17T17:24:53.460631+0000","last_unstale":"2024-08-17T17:28:16.123035+0000","last_undegraded":"2024-08-17T17:28:16.123035+0000","last_fullsized":"2024-08-17T17:28:16.123035+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:05:24.938831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":36,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.122614+0000","last_change":"2024-08-17T17:25:04.507176+0000","last_active":"2024-08-17T17:28:16.122614+0000","last_peered":"2024-08-17T17:28:16.122614+0000","last_clean":"2024-08-17T17:28:16.122614+0000","last_became_active":"2024-08-17T17:25:04.506788+0000","last_became_peered":"2024-08-17T17:25:04.506788+0000","last_unstale":"2024-08-17T17:28:16.122614+0000","last_undegraded":"2024-08-17T17:28:16.122614+0000","last_fullsized":"2024-08-17T17:28:16.122614+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:22:04.658216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":182,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.114640+0000","last_change":"2024-08-17T17:17:32.495428+0000","last_active":"2024-08-17T17:28:16.114640+0000","last_peered":"2024-08-17T17:28:16.114640+0000","last_clean":"2024-08-17T17:28:16.114640+0000","last_became_active":"2024-08-17T17:17:32.494978+0000","last_became_peered":"2024-08-17T17:17:32.494978+0000","last_unstale":"2024-08-17T17:28:16.114640+0000","last_undegraded":"2024-08-17T17:28:16.114640+0000","last_fullsized":"2024-08-17T17:28:16.114640+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:42:24.647812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":305,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-08-17T17:28:17.800025+0000","last_change":"2024-08-17T17:20:49.272126+0000","last_active":"2024-08-17T17:28:17.800025+0000","last_peered":"2024-08-17T17:28:17.800025+0000","last_clean":"2024-08-17T17:28:17.800025+0000","last_became_active":"2024-08-17T17:20:49.271185+0000","last_became_peered":"2024-08-17T17:20:49.271185+0000","last_unstale":"2024-08-17T17:28:17.800025+0000","last_undegraded":"2024-08-17T17:28:17.800025+0000","last_fullsized":"2024-08-17T17:28:17.800025+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:43:58.184939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":331,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.117483+0000","last_change":"2024-08-17T17:24:06.595380+0000","last_active":"2024-08-17T17:28:16.117483+0000","last_peered":"2024-08-17T17:28:16.117483+0000","last_clean":"2024-08-17T17:28:16.117483+0000","last_became_active":"2024-08-17T17:24:06.584747+0000","last_became_peered":"2024-08-17T17:24:06.584747+0000","last_unstale":"2024-08-17T17:28:16.117483+0000","last_undegraded":"2024-08-17T17:28:16.117483+0000","last_fullsized":"2024-08-17T17:28:16.117483+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:02:18.473980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":277,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.121153+0000","last_change":"2024-08-17T17:20:49.280596+0000","last_active":"2024-08-17T17:28:16.121153+0000","last_peered":"2024-08-17T17:28:16.121153+0000","last_clean":"2024-08-17T17:28:16.121153+0000","last_became_active":"2024-08-17T17:20:49.279553+0000","last_became_peered":"2024-08-17T17:20:49.279553+0000","last_unstale":"2024-08-17T17:28:16.121153+0000","last_undegraded":"2024-08-17T17:28:16.121153+0000","last_fullsized":"2024-08-17T17:28:16.121153+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:13:24.124392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":227,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-08-17T17:28:15.213277+0000","last_change":"2024-08-17T17:20:49.276769+0000","last_active":"2024-08-17T17:28:15.213277+0000","last_peered":"2024-08-17T17:28:15.213277+0000","last_clean":"2024-08-17T17:28:15.213277+0000","last_became_active":"2024-08-17T17:20:49.276498+0000","last_became_peered":"2024-08-17T17:20:49.276498+0000","last_unstale":"2024-08-17T17:28:15.213277+0000","last_undegraded":"2024-08-17T17:28:15.213277+0000","last_fullsized":"2024-08-17T17:28:15.213277+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:39:05.419440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":203,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.122786+0000","last_change":"2024-08-17T17:20:14.455738+0000","last_active":"2024-08-17T17:28:16.122786+0000","last_peered":"2024-08-17T17:28:16.122786+0000","last_clean":"2024-08-17T17:28:16.122786+0000","last_became_active":"2024-08-17T17:20:14.455469+0000","last_became_peered":"2024-08-17T17:20:14.455469+0000","last_unstale":"2024-08-17T17:28:16.122786+0000","last_undegraded":"2024-08-17T17:28:16.122786+0000","last_fullsized":"2024-08-17T17:28:16.122786+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:27:28.424336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":157,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.122831+0000","last_change":"2024-08-17T17:20:14.431140+0000","last_active":"2024-08-17T17:28:16.122831+0000","last_peered":"2024-08-17T17:28:16.122831+0000","last_clean":"2024-08-17T17:28:16.122831+0000","last_became_active":"2024-08-17T17:20:14.430839+0000","last_became_peered":"2024-08-17T17:20:14.430839+0000","last_unstale":"2024-08-17T17:28:16.122831+0000","last_undegraded":"2024-08-17T17:28:16.122831+0000","last_fullsized":"2024-08-17T17:28:16.122831+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:08:16.303616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":36,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.123529+0000","last_change":"2024-08-17T17:25:04.509325+0000","last_active":"2024-08-17T17:28:16.123529+0000","last_peered":"2024-08-17T17:28:16.123529+0000","last_clean":"2024-08-17T17:28:16.123529+0000","last_became_active":"2024-08-17T17:25:04.508364+0000","last_became_peered":"2024-08-17T17:25:04.508364+0000","last_unstale":"2024-08-17T17:28:16.123529+0000","last_undegraded":"2024-08-17T17:28:16.123529+0000","last_fullsized":"2024-08-17T17:28:16.123529+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:48:56.415763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":315,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.116932+0000","last_change":"2024-08-17T17:19:57.520745+0000","last_active":"2024-08-17T17:28:16.116932+0000","last_peered":"2024-08-17T17:28:16.116932+0000","last_clean":"2024-08-17T17:28:16.116932+0000","last_became_active":"2024-08-17T17:19:57.520152+0000","last_became_peered":"2024-08-17T17:19:57.520152+0000","last_unstale":"2024-08-17T17:28:16.116932+0000","last_undegraded":"2024-08-17T17:28:16.116932+0000","last_fullsized":"2024-08-17T17:28:16.116932+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:19:39.495644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":289,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-08-17T17:28:15.213750+0000","last_change":"2024-08-17T17:20:49.274372+0000","last_active":"2024-08-17T17:28:15.213750+0000","last_peered":"2024-08-17T17:28:15.213750+0000","last_clean":"2024-08-17T17:28:15.213750+0000","last_became_active":"2024-08-17T17:20:49.274023+0000","last_became_peered":"2024-08-17T17:20:49.274023+0000","last_unstale":"2024-08-17T17:28:15.213750+0000","last_undegraded":"2024-08-17T17:28:15.213750+0000","last_fullsized":"2024-08-17T17:28:15.213750+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:31:02.529766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":551,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.123180+0000","last_change":"2024-08-17T17:24:09.599615+0000","last_active":"2024-08-17T17:28:16.123180+0000","last_peered":"2024-08-17T17:28:16.123180+0000","last_clean":"2024-08-17T17:28:16.123180+0000","last_became_active":"2024-08-17T17:24:09.599085+0000","last_became_peered":"2024-08-17T17:24:09.599085+0000","last_unstale":"2024-08-17T17:28:16.123180+0000","last_undegraded":"2024-08-17T17:28:16.123180+0000","last_fullsized":"2024-08-17T17:28:16.123180+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:04:01.141401+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:30:09.569579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":796,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.114440+0000","last_change":"2024-08-17T17:20:00.434774+0000","last_active":"2024-08-17T17:28:16.114440+0000","last_peered":"2024-08-17T17:28:16.114440+0000","last_clean":"2024-08-17T17:28:16.114440+0000","last_became_active":"2024-08-17T17:20:00.434343+0000","last_became_peered":"2024-08-17T17:20:00.434343+0000","last_unstale":"2024-08-17T17:28:16.114440+0000","last_undegraded":"2024-08-17T17:28:16.114440+0000","last_fullsized":"2024-08-17T17:28:16.114440+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:04.995152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":274,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.117243+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:28:16.117243+0000","last_peered":"2024-08-17T17:28:16.117243+0000","last_clean":"2024-08-17T17:28:16.117243+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:28:16.117243+0000","last_undegraded":"2024-08-17T17:28:16.117243+0000","last_fullsized":"2024-08-17T17:28:16.117243+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":36,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.122975+0000","last_change":"2024-08-17T17:25:04.579426+0000","last_active":"2024-08-17T17:28:16.122975+0000","last_peered":"2024-08-17T17:28:16.122975+0000","last_clean":"2024-08-17T17:28:16.122975+0000","last_became_active":"2024-08-17T17:25:04.579012+0000","last_became_peered":"2024-08-17T17:25:04.579012+0000","last_unstale":"2024-08-17T17:28:16.122975+0000","last_undegraded":"2024-08-17T17:28:16.122975+0000","last_fullsized":"2024-08-17T17:28:16.122975+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:56:32.241321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":814,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.123157+0000","last_change":"2024-08-17T17:16:26.958415+0000","last_active":"2024-08-17T17:28:16.123157+0000","last_peered":"2024-08-17T17:28:16.123157+0000","last_clean":"2024-08-17T17:28:16.123157+0000","last_became_active":"2024-08-17T17:16:26.957956+0000","last_became_peered":"2024-08-17T17:16:26.957956+0000","last_unstale":"2024-08-17T17:28:16.123157+0000","last_undegraded":"2024-08-17T17:28:16.123157+0000","last_fullsized":"2024-08-17T17:28:16.123157+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:22.478077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":304,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-08-17T17:28:17.800248+0000","last_change":"2024-08-17T17:18:59.415331+0000","last_active":"2024-08-17T17:28:17.800248+0000","last_peered":"2024-08-17T17:28:17.800248+0000","last_clean":"2024-08-17T17:28:17.800248+0000","last_became_active":"2024-08-17T17:18:59.415008+0000","last_became_peered":"2024-08-17T17:18:59.415008+0000","last_unstale":"2024-08-17T17:28:17.800248+0000","last_undegraded":"2024-08-17T17:28:17.800248+0000","last_fullsized":"2024-08-17T17:28:17.800248+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:22:38.076866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":113,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-08-17T17:28:17.800694+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:28:17.800694+0000","last_peered":"2024-08-17T17:28:17.800694+0000","last_clean":"2024-08-17T17:28:17.800694+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:28:17.800694+0000","last_undegraded":"2024-08-17T17:28:17.800694+0000","last_fullsized":"2024-08-17T17:28:17.800694+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":690,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.117243+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:28:16.117243+0000","last_peered":"2024-08-17T17:28:16.117243+0000","last_clean":"2024-08-17T17:28:16.117243+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:28:16.117243+0000","last_undegraded":"2024-08-17T17:28:16.117243+0000","last_fullsized":"2024-08-17T17:28:16.117243+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":296,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-08-17T17:28:15.213123+0000","last_change":"2024-08-17T17:19:57.515564+0000","last_active":"2024-08-17T17:28:15.213123+0000","last_peered":"2024-08-17T17:28:15.213123+0000","last_clean":"2024-08-17T17:28:15.213123+0000","last_became_active":"2024-08-17T17:19:57.515142+0000","last_became_peered":"2024-08-17T17:19:57.515142+0000","last_unstale":"2024-08-17T17:28:15.213123+0000","last_undegraded":"2024-08-17T17:28:15.213123+0000","last_fullsized":"2024-08-17T17:28:15.213123+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:59:33.506176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":327,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-08-17T17:28:15.213558+0000","last_change":"2024-08-17T17:24:06.587903+0000","last_active":"2024-08-17T17:28:15.213558+0000","last_peered":"2024-08-17T17:28:15.213558+0000","last_clean":"2024-08-17T17:28:15.213558+0000","last_became_active":"2024-08-17T17:24:06.587503+0000","last_became_peered":"2024-08-17T17:24:06.587503+0000","last_unstale":"2024-08-17T17:28:15.213558+0000","last_undegraded":"2024-08-17T17:28:15.213558+0000","last_fullsized":"2024-08-17T17:28:15.213558+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:05:58.591992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":1134,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.114454+0000","last_change":"2024-08-17T17:20:00.433870+0000","last_active":"2024-08-17T17:28:16.114454+0000","last_peered":"2024-08-17T17:28:16.114454+0000","last_clean":"2024-08-17T17:28:16.114454+0000","last_became_active":"2024-08-17T17:20:00.433386+0000","last_became_peered":"2024-08-17T17:20:00.433386+0000","last_unstale":"2024-08-17T17:28:16.114454+0000","last_undegraded":"2024-08-17T17:28:16.114454+0000","last_fullsized":"2024-08-17T17:28:16.114454+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:01:11.848862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1392,"num_read_kb":2863,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":226,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-08-17T17:28:17.800157+0000","last_change":"2024-08-17T17:20:49.272407+0000","last_active":"2024-08-17T17:28:17.800157+0000","last_peered":"2024-08-17T17:28:17.800157+0000","last_clean":"2024-08-17T17:28:17.800157+0000","last_became_active":"2024-08-17T17:20:49.271749+0000","last_became_peered":"2024-08-17T17:20:49.271749+0000","last_unstale":"2024-08-17T17:28:17.800157+0000","last_undegraded":"2024-08-17T17:28:17.800157+0000","last_fullsized":"2024-08-17T17:28:17.800157+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:36:01.443457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":293,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-08-17T17:28:15.213609+0000","last_change":"2024-08-17T17:17:32.499715+0000","last_active":"2024-08-17T17:28:15.213609+0000","last_peered":"2024-08-17T17:28:15.213609+0000","last_clean":"2024-08-17T17:28:15.213609+0000","last_became_active":"2024-08-17T17:17:32.499412+0000","last_became_peered":"2024-08-17T17:17:32.499412+0000","last_unstale":"2024-08-17T17:28:15.213609+0000","last_undegraded":"2024-08-17T17:28:15.213609+0000","last_fullsized":"2024-08-17T17:28:15.213609+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:39:38.456970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":199,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.114768+0000","last_change":"2024-08-17T17:17:32.495728+0000","last_active":"2024-08-17T17:28:16.114768+0000","last_peered":"2024-08-17T17:28:16.114768+0000","last_clean":"2024-08-17T17:28:16.114768+0000","last_became_active":"2024-08-17T17:17:32.495357+0000","last_became_peered":"2024-08-17T17:17:32.495357+0000","last_unstale":"2024-08-17T17:28:16.114768+0000","last_undegraded":"2024-08-17T17:28:16.114768+0000","last_fullsized":"2024-08-17T17:28:16.114768+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:27:45.561567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":529,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-08-17T17:28:15.213727+0000","last_change":"2024-08-17T17:16:26.964621+0000","last_active":"2024-08-17T17:28:15.213727+0000","last_peered":"2024-08-17T17:28:15.213727+0000","last_clean":"2024-08-17T17:28:15.213727+0000","last_became_active":"2024-08-17T17:16:26.964058+0000","last_became_peered":"2024-08-17T17:16:26.964058+0000","last_unstale":"2024-08-17T17:28:15.213727+0000","last_undegraded":"2024-08-17T17:28:15.213727+0000","last_fullsized":"2024-08-17T17:28:15.213727+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:00:04.465644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":299,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-08-17T17:28:15.213324+0000","last_change":"2024-08-17T17:24:53.465550+0000","last_active":"2024-08-17T17:28:15.213324+0000","last_peered":"2024-08-17T17:28:15.213324+0000","last_clean":"2024-08-17T17:28:15.213324+0000","last_became_active":"2024-08-17T17:24:53.465228+0000","last_became_peered":"2024-08-17T17:24:53.465228+0000","last_unstale":"2024-08-17T17:28:15.213324+0000","last_undegraded":"2024-08-17T17:28:15.213324+0000","last_fullsized":"2024-08-17T17:28:15.213324+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:21:18.378359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":324,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-08-17T17:28:15.213630+0000","last_change":"2024-08-17T17:24:53.464307+0000","last_active":"2024-08-17T17:28:15.213630+0000","last_peered":"2024-08-17T17:28:15.213630+0000","last_clean":"2024-08-17T17:28:15.213630+0000","last_became_active":"2024-08-17T17:24:53.463996+0000","last_became_peered":"2024-08-17T17:24:53.463996+0000","last_unstale":"2024-08-17T17:28:15.213630+0000","last_undegraded":"2024-08-17T17:28:15.213630+0000","last_fullsized":"2024-08-17T17:28:15.213630+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:04:18.993725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":651,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.123129+0000","last_change":"2024-08-17T17:24:09.599822+0000","last_active":"2024-08-17T17:28:16.123129+0000","last_peered":"2024-08-17T17:28:16.123129+0000","last_clean":"2024-08-17T17:28:16.123129+0000","last_became_active":"2024-08-17T17:24:09.599189+0000","last_became_peered":"2024-08-17T17:24:09.599189+0000","last_unstale":"2024-08-17T17:28:16.123129+0000","last_undegraded":"2024-08-17T17:28:16.123129+0000","last_fullsized":"2024-08-17T17:28:16.123129+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:04.465604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":287,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-08-17T17:28:17.800461+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:28:17.800461+0000","last_peered":"2024-08-17T17:28:17.800461+0000","last_clean":"2024-08-17T17:28:17.800461+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:28:17.800461+0000","last_undegraded":"2024-08-17T17:28:17.800461+0000","last_fullsized":"2024-08-17T17:28:17.800461+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":320,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.117902+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:28:16.117902+0000","last_peered":"2024-08-17T17:28:16.117902+0000","last_clean":"2024-08-17T17:28:16.117902+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:28:16.117902+0000","last_undegraded":"2024-08-17T17:28:16.117902+0000","last_fullsized":"2024-08-17T17:28:16.117902+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":838,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-08-17T17:28:15.213646+0000","last_change":"2024-08-17T17:23:19.633786+0000","last_active":"2024-08-17T17:28:15.213646+0000","last_peered":"2024-08-17T17:28:15.213646+0000","last_clean":"2024-08-17T17:28:15.213646+0000","last_became_active":"2024-08-17T17:23:19.633230+0000","last_became_peered":"2024-08-17T17:23:19.633230+0000","last_unstale":"2024-08-17T17:28:15.213646+0000","last_undegraded":"2024-08-17T17:28:15.213646+0000","last_fullsized":"2024-08-17T17:28:15.213646+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:30:50.377391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":317,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.116571+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:28:16.116571+0000","last_peered":"2024-08-17T17:28:16.116571+0000","last_clean":"2024-08-17T17:28:16.116571+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:28:16.116571+0000","last_undegraded":"2024-08-17T17:28:16.116571+0000","last_fullsized":"2024-08-17T17:28:16.116571+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":117,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-08-17T17:28:17.800674+0000","last_change":"2024-08-17T17:24:06.593660+0000","last_active":"2024-08-17T17:28:17.800674+0000","last_peered":"2024-08-17T17:28:17.800674+0000","last_clean":"2024-08-17T17:28:17.800674+0000","last_became_active":"2024-08-17T17:24:06.586215+0000","last_became_peered":"2024-08-17T17:24:06.586215+0000","last_unstale":"2024-08-17T17:28:17.800674+0000","last_undegraded":"2024-08-17T17:28:17.800674+0000","last_fullsized":"2024-08-17T17:28:17.800674+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:15:49.475395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":455,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.123036+0000","last_change":"2024-08-17T17:26:06.855307+0000","last_active":"2024-08-17T17:28:16.123036+0000","last_peered":"2024-08-17T17:28:16.123036+0000","last_clean":"2024-08-17T17:28:16.123036+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:28:16.123036+0000","last_undegraded":"2024-08-17T17:28:16.123036+0000","last_fullsized":"2024-08-17T17:28:16.123036+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":123,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.114253+0000","last_change":"2024-08-17T17:18:59.417183+0000","last_active":"2024-08-17T17:28:16.114253+0000","last_peered":"2024-08-17T17:28:16.114253+0000","last_clean":"2024-08-17T17:28:16.114253+0000","last_became_active":"2024-08-17T17:18:59.416843+0000","last_became_peered":"2024-08-17T17:18:59.416843+0000","last_unstale":"2024-08-17T17:28:16.114253+0000","last_undegraded":"2024-08-17T17:28:16.114253+0000","last_fullsized":"2024-08-17T17:28:16.114253+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:29:09.336335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":144,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.122626+0000","last_change":"2024-08-17T17:20:49.258048+0000","last_active":"2024-08-17T17:28:16.122626+0000","last_peered":"2024-08-17T17:28:16.122626+0000","last_clean":"2024-08-17T17:28:16.122626+0000","last_became_active":"2024-08-17T17:20:49.257672+0000","last_became_peered":"2024-08-17T17:20:49.257672+0000","last_unstale":"2024-08-17T17:28:16.122626+0000","last_undegraded":"2024-08-17T17:28:16.122626+0000","last_fullsized":"2024-08-17T17:28:16.122626+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:08:11.734316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":183,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.122740+0000","last_change":"2024-08-17T17:18:59.430969+0000","last_active":"2024-08-17T17:28:16.122740+0000","last_peered":"2024-08-17T17:28:16.122740+0000","last_clean":"2024-08-17T17:28:16.122740+0000","last_became_active":"2024-08-17T17:18:59.413475+0000","last_became_peered":"2024-08-17T17:18:59.413475+0000","last_unstale":"2024-08-17T17:28:16.122740+0000","last_undegraded":"2024-08-17T17:28:16.122740+0000","last_fullsized":"2024-08-17T17:28:16.122740+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:36:33.732645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":310,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-08-17T17:28:15.213240+0000","last_change":"2024-08-17T17:25:04.506413+0000","last_active":"2024-08-17T17:28:15.213240+0000","last_peered":"2024-08-17T17:28:15.213240+0000","last_clean":"2024-08-17T17:28:15.213240+0000","last_became_active":"2024-08-17T17:25:04.506104+0000","last_became_peered":"2024-08-17T17:25:04.506104+0000","last_unstale":"2024-08-17T17:28:15.213240+0000","last_undegraded":"2024-08-17T17:28:15.213240+0000","last_fullsized":"2024-08-17T17:28:15.213240+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:41:53.556223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":126,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.122637+0000","last_change":"2024-08-17T17:20:49.279458+0000","last_active":"2024-08-17T17:28:16.122637+0000","last_peered":"2024-08-17T17:28:16.122637+0000","last_clean":"2024-08-17T17:28:16.122637+0000","last_became_active":"2024-08-17T17:20:49.260790+0000","last_became_peered":"2024-08-17T17:20:49.260790+0000","last_unstale":"2024-08-17T17:28:16.122637+0000","last_undegraded":"2024-08-17T17:28:16.122637+0000","last_fullsized":"2024-08-17T17:28:16.122637+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:58.092945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":40,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.114806+0000","last_change":"2024-08-17T17:24:53.463713+0000","last_active":"2024-08-17T17:28:16.114806+0000","last_peered":"2024-08-17T17:28:16.114806+0000","last_clean":"2024-08-17T17:28:16.114806+0000","last_became_active":"2024-08-17T17:24:53.463419+0000","last_became_peered":"2024-08-17T17:24:53.463419+0000","last_unstale":"2024-08-17T17:28:16.114806+0000","last_undegraded":"2024-08-17T17:28:16.114806+0000","last_fullsized":"2024-08-17T17:28:16.114806+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:34:44.545175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":267,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.116951+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:28:16.116951+0000","last_peered":"2024-08-17T17:28:16.116951+0000","last_clean":"2024-08-17T17:28:16.116951+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:28:16.116951+0000","last_undegraded":"2024-08-17T17:28:16.116951+0000","last_fullsized":"2024-08-17T17:28:16.116951+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":297,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-08-17T17:28:17.800611+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:28:17.800611+0000","last_peered":"2024-08-17T17:28:17.800611+0000","last_clean":"2024-08-17T17:28:17.800611+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:28:17.800611+0000","last_undegraded":"2024-08-17T17:28:17.800611+0000","last_fullsized":"2024-08-17T17:28:17.800611+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":314,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.116471+0000","last_change":"2024-08-17T17:25:04.509422+0000","last_active":"2024-08-17T17:28:16.116471+0000","last_peered":"2024-08-17T17:28:16.116471+0000","last_clean":"2024-08-17T17:28:16.116471+0000","last_became_active":"2024-08-17T17:25:04.508742+0000","last_became_peered":"2024-08-17T17:25:04.508742+0000","last_unstale":"2024-08-17T17:28:16.116471+0000","last_undegraded":"2024-08-17T17:28:16.116471+0000","last_fullsized":"2024-08-17T17:28:16.116471+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:18:31.976907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":181,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.122886+0000","last_change":"2024-08-17T17:16:26.963696+0000","last_active":"2024-08-17T17:28:16.122886+0000","last_peered":"2024-08-17T17:28:16.122886+0000","last_clean":"2024-08-17T17:28:16.122886+0000","last_became_active":"2024-08-17T17:16:26.963280+0000","last_became_peered":"2024-08-17T17:16:26.963280+0000","last_unstale":"2024-08-17T17:28:16.122886+0000","last_undegraded":"2024-08-17T17:28:16.122886+0000","last_fullsized":"2024-08-17T17:28:16.122886+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:22:01.474840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":332,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.116791+0000","last_change":"2024-08-17T17:24:06.586886+0000","last_active":"2024-08-17T17:28:16.116791+0000","last_peered":"2024-08-17T17:28:16.116791+0000","last_clean":"2024-08-17T17:28:16.116791+0000","last_became_active":"2024-08-17T17:24:06.586516+0000","last_became_peered":"2024-08-17T17:24:06.586516+0000","last_unstale":"2024-08-17T17:28:16.116791+0000","last_undegraded":"2024-08-17T17:28:16.116791+0000","last_fullsized":"2024-08-17T17:28:16.116791+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:03:30.002726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":114,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-08-17T17:28:17.800559+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:28:17.800559+0000","last_peered":"2024-08-17T17:28:17.800559+0000","last_clean":"2024-08-17T17:28:17.800559+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:28:17.800559+0000","last_undegraded":"2024-08-17T17:28:17.800559+0000","last_fullsized":"2024-08-17T17:28:17.800559+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":289,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-08-17T17:28:15.213314+0000","last_change":"2024-08-17T17:18:59.412463+0000","last_active":"2024-08-17T17:28:15.213314+0000","last_peered":"2024-08-17T17:28:15.213314+0000","last_clean":"2024-08-17T17:28:15.213314+0000","last_became_active":"2024-08-17T17:18:59.412134+0000","last_became_peered":"2024-08-17T17:18:59.412134+0000","last_unstale":"2024-08-17T17:28:15.213314+0000","last_undegraded":"2024-08-17T17:28:15.213314+0000","last_fullsized":"2024-08-17T17:28:15.213314+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:44.463153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":335,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.117499+0000","last_change":"2024-08-17T17:25:04.510854+0000","last_active":"2024-08-17T17:28:16.117499+0000","last_peered":"2024-08-17T17:28:16.117499+0000","last_clean":"2024-08-17T17:28:16.117499+0000","last_became_active":"2024-08-17T17:25:04.510462+0000","last_became_peered":"2024-08-17T17:25:04.510462+0000","last_unstale":"2024-08-17T17:28:16.117499+0000","last_undegraded":"2024-08-17T17:28:16.117499+0000","last_fullsized":"2024-08-17T17:28:16.117499+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:55:54.762086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":230,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-08-17T17:28:15.213390+0000","last_change":"2024-08-17T17:24:06.591165+0000","last_active":"2024-08-17T17:28:15.213390+0000","last_peered":"2024-08-17T17:28:15.213390+0000","last_clean":"2024-08-17T17:28:15.213390+0000","last_became_active":"2024-08-17T17:24:06.590785+0000","last_became_peered":"2024-08-17T17:24:06.590785+0000","last_unstale":"2024-08-17T17:28:15.213390+0000","last_undegraded":"2024-08-17T17:28:15.213390+0000","last_fullsized":"2024-08-17T17:28:15.213390+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:54:05.775563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":302,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-08-17T17:28:15.213796+0000","last_change":"2024-08-17T17:24:32.300560+0000","last_active":"2024-08-17T17:28:15.213796+0000","last_peered":"2024-08-17T17:28:15.213796+0000","last_clean":"2024-08-17T17:28:15.213796+0000","last_became_active":"2024-08-17T17:24:32.300228+0000","last_became_peered":"2024-08-17T17:24:32.300228+0000","last_unstale":"2024-08-17T17:28:15.213796+0000","last_undegraded":"2024-08-17T17:28:15.213796+0000","last_fullsized":"2024-08-17T17:28:15.213796+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:18:05.475449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":113,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-08-17T17:28:17.800402+0000","last_change":"2024-08-17T17:24:06.586379+0000","last_active":"2024-08-17T17:28:17.800402+0000","last_peered":"2024-08-17T17:28:17.800402+0000","last_clean":"2024-08-17T17:28:17.800402+0000","last_became_active":"2024-08-17T17:24:06.586023+0000","last_became_peered":"2024-08-17T17:24:06.586023+0000","last_unstale":"2024-08-17T17:28:17.800402+0000","last_undegraded":"2024-08-17T17:28:17.800402+0000","last_fullsized":"2024-08-17T17:28:17.800402+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:45.701637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":321,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.117443+0000","last_change":"2024-08-17T17:21:50.442535+0000","last_active":"2024-08-17T17:28:16.117443+0000","last_peered":"2024-08-17T17:28:16.117443+0000","last_clean":"2024-08-17T17:28:16.117443+0000","last_became_active":"2024-08-17T17:21:50.441891+0000","last_became_peered":"2024-08-17T17:21:50.441891+0000","last_unstale":"2024-08-17T17:28:16.117443+0000","last_undegraded":"2024-08-17T17:28:16.117443+0000","last_fullsized":"2024-08-17T17:28:16.117443+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:46:24.025407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":236,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.114440+0000","last_change":"2024-08-17T17:25:04.508045+0000","last_active":"2024-08-17T17:28:16.114440+0000","last_peered":"2024-08-17T17:28:16.114440+0000","last_clean":"2024-08-17T17:28:16.114440+0000","last_became_active":"2024-08-17T17:25:04.505083+0000","last_became_peered":"2024-08-17T17:25:04.505083+0000","last_unstale":"2024-08-17T17:28:16.114440+0000","last_undegraded":"2024-08-17T17:28:16.114440+0000","last_fullsized":"2024-08-17T17:28:16.114440+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:23:42.993544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":307,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.117453+0000","last_change":"2024-08-17T17:18:59.418434+0000","last_active":"2024-08-17T17:28:16.117453+0000","last_peered":"2024-08-17T17:28:16.117453+0000","last_clean":"2024-08-17T17:28:16.117453+0000","last_became_active":"2024-08-17T17:18:59.418043+0000","last_became_peered":"2024-08-17T17:18:59.418043+0000","last_unstale":"2024-08-17T17:28:16.117453+0000","last_undegraded":"2024-08-17T17:28:16.117453+0000","last_fullsized":"2024-08-17T17:28:16.117453+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:25:51.371820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":209,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.114639+0000","last_change":"2024-08-17T17:20:49.291167+0000","last_active":"2024-08-17T17:28:16.114639+0000","last_peered":"2024-08-17T17:28:16.114639+0000","last_clean":"2024-08-17T17:28:16.114639+0000","last_became_active":"2024-08-17T17:20:49.263750+0000","last_became_peered":"2024-08-17T17:20:49.263750+0000","last_unstale":"2024-08-17T17:28:16.114639+0000","last_undegraded":"2024-08-17T17:28:16.114639+0000","last_fullsized":"2024-08-17T17:28:16.114639+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:02.863246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":321,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.117269+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:28:16.117269+0000","last_peered":"2024-08-17T17:28:16.117269+0000","last_clean":"2024-08-17T17:28:16.117269+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:28:16.117269+0000","last_undegraded":"2024-08-17T17:28:16.117269+0000","last_fullsized":"2024-08-17T17:28:16.117269+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":321,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.117864+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:28:16.117864+0000","last_peered":"2024-08-17T17:28:16.117864+0000","last_clean":"2024-08-17T17:28:16.117864+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:28:16.117864+0000","last_undegraded":"2024-08-17T17:28:16.117864+0000","last_fullsized":"2024-08-17T17:28:16.117864+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":36,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-08-17T17:28:16.122037+0000","last_change":"2024-08-17T17:25:04.503834+0000","last_active":"2024-08-17T17:28:16.122037+0000","last_peered":"2024-08-17T17:28:16.122037+0000","last_clean":"2024-08-17T17:28:16.122037+0000","last_became_active":"2024-08-17T17:25:04.502712+0000","last_became_peered":"2024-08-17T17:25:04.502712+0000","last_unstale":"2024-08-17T17:28:16.122037+0000","last_undegraded":"2024-08-17T17:28:16.122037+0000","last_fullsized":"2024-08-17T17:28:16.122037+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:45.389103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":133,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-08-17T17:28:17.800654+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:28:17.800654+0000","last_peered":"2024-08-17T17:28:17.800654+0000","last_clean":"2024-08-17T17:28:17.800654+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:28:17.800654+0000","last_undegraded":"2024-08-17T17:28:17.800654+0000","last_fullsized":"2024-08-17T17:28:17.800654+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":6304,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":16702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17971,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234258,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":809624,"kb_used_data":512268,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":92935528,"statfs":{"total":95995035648,"available":95165980672,"internally_reserved":0,"allocated":524562432,"data_stored":523270796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495893,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309012,"kb_used_data":7492,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93436140,"statfs":{"total":95995035648,"available":95678607360,"internally_reserved":0,"allocated":7671808,"data_stored":6388064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757530,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312652,"kb_used_data":2808,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93432500,"statfs":{"total":95995035648,"available":95674880000,"internally_reserved":0,"allocated":2875392,"data_stored":1599391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019168,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435395,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":327476,"kb_used_data":2792,"kb_used_omap":4,"kb_used_meta":324667,"kb_avail":93417676,"statfs":{"total":95995035648,"available":95659700224,"internally_reserved":0,"allocated":2859008,"data_stored":1590814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":332459018},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676598921,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":356060,"kb_used_data":6868,"kb_used_omap":7,"kb_used_meta":349176,"kb_avail":93389092,"statfs":{"total":95995035648,"available":95630430208,"internally_reserved":0,"allocated":7032832,"data_stored":5765231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":357556386},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762448,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":534480,"kb_used_data":164040,"kb_used_omap":7,"kb_used_meta":370424,"kb_avail":93210672,"statfs":{"total":95995035648,"available":95447728128,"internally_reserved":0,"allocated":167976960,"data_stored":166716348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7945,"internal_metadata":379314423},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726925975,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":317824,"kb_used_data":3380,"kb_used_omap":7,"kb_used_meta":314424,"kb_avail":93427328,"statfs":{"total":95995035648,"available":95669583872,"internally_reserved":0,"allocated":3461120,"data_stored":2181790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":321970497},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6241,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:28:21.895 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:21.917 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:21.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:21.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:22.019 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:22.041 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:22.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:22.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:22.064 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-17T17:28:22.143 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:22.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:22.170 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:22.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:22.271 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:22.294 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:22.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:22.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:22.395 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:22.418 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:22.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:22.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:22.520 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:22.543 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:22.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:22.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:22.644 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:22.667 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:22.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:22.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:22.769 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:22.795 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:22.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:22.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:22.818 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:28:22.897 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:22.920 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:22.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:22.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:22.952 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:28:23.022 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:23.044 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:23.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:23.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:23.146 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:23.168 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:23.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:23.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:23.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:23 smithi053 ceph-mon[89963]: pgmap v610: 73 pgs: 73 active+clean; 4.6 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:23.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:23.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:28:23.270 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:23.297 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:23.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:23.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:23.334 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:23 smithi003 ceph-mon[103344]: pgmap v610: 73 pgs: 73 active+clean; 4.6 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:23.334 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:23.334 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:28:23.334 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:23 smithi003 ceph-mon[100615]: pgmap v610: 73 pgs: 73 active+clean; 4.6 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:23.334 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:23.334 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:28:23.399 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:23.422 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:23.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:23.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:23.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:23.546 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:23.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:23.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:23.632 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:28:23.647 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:23.672 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:23.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:23.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:23.774 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:23.798 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:23.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:23.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:23.900 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:23.923 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:23.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:23.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:24.025 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:24.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:24.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:24.149 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:24.172 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:24.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:24.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:24.274 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:24.298 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:24.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:24.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:24.400 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:24.421 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:24.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:24.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:24.523 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:24.545 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:24.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:24.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:24.612 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:28:24.647 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:24.676 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:24.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:24.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:24.778 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:24.801 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:24.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:24.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:24.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:28:24.903 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:24.925 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:24.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:24.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:25.027 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:25.050 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:25.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:25.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:25.151 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:25.175 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:25.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:25.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:25.276 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:25.299 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:25.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:25.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:25.401 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:25.413 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:28:25.414 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2024-08-17T17:28:25.414 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 0 2024-08-17T17:28:25.414 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:28:25.423 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:25.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:25.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:25.525 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:25.550 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:25.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:25.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:25.652 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:25.674 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:25.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:25.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:25.721 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:25 smithi053 ceph-mon[89963]: pgmap v611: 73 pgs: 73 active+clean; 4.6 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:25.721 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:25 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:25.721 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:25 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:25.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:25 smithi003 ceph-mon[100615]: pgmap v611: 73 pgs: 73 active+clean; 4.6 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:25.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:25 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:25.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:25 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:25.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:25 smithi003 ceph-mon[103344]: pgmap v611: 73 pgs: 73 active+clean; 4.6 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:25.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:25 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:25.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:25 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:25.774 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:28:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:28:25] "GET /metrics HTTP/1.1" 200 34148 "" "Prometheus/2.51.0" 2024-08-17T17:28:25.776 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:25.798 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:25.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:25.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:25.899 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:25.921 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:25.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:25.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:26.023 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:26.028 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-17T17:28:26.045 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:26.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:26.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:26.147 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:26.169 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:26.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:26.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:26.189 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:28:26.239 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:28:26.274 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:26.297 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:26.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:26.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:26.311 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:28:26.399 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:26.421 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:26.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:26.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:26.493 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_2' created 2024-08-17T17:28:26.500 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-08-17T17:28:26.523 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:26.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:26.548 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:26.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:26.649 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:26.672 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:26.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:26.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:26.688 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:26 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3482853102' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:28:26.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:26 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:28:26.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:26 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3551355652' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-08-17T17:28:26.689 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:26 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3482853102' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:28:26.689 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:26 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:28:26.689 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:26 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3551355652' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-08-17T17:28:26.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:26 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3482853102' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:28:26.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:26 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:28:26.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:26 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3551355652' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-08-17T17:28:26.774 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:26.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:26.799 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:26.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:26.900 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:26.922 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:26.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:26.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:27.024 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:27.046 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:27.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:27.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:27.148 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:27.170 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:27.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:27.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:27.272 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:27.294 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:27.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:27.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:27.396 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:27.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:27.421 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:27.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:27.522 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:27.544 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:27.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:27.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:27.646 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:27.649 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:27 smithi053 ceph-mon[89963]: pgmap v612: 73 pgs: 73 active+clean; 4.6 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:27.649 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:27 smithi053 ceph-mon[89963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:28:27.649 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:27 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:28:27.649 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:27 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3551355652' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-08-17T17:28:27.649 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:27 smithi053 ceph-mon[89963]: osdmap e226: 8 total, 8 up, 7 in 2024-08-17T17:28:27.668 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:27.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:27.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:27.675 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:27 smithi003 ceph-mon[100615]: pgmap v612: 73 pgs: 73 active+clean; 4.6 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:27.676 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:27 smithi003 ceph-mon[100615]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:28:27.676 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:27 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:28:27.676 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:27 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3551355652' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-08-17T17:28:27.676 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:27 smithi003 ceph-mon[100615]: osdmap e226: 8 total, 8 up, 7 in 2024-08-17T17:28:27.676 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:27 smithi003 ceph-mon[103344]: pgmap v612: 73 pgs: 73 active+clean; 4.6 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:27.676 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:27 smithi003 ceph-mon[103344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:28:27.676 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:27 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:28:27.676 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:27 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3551355652' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-08-17T17:28:27.676 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:27 smithi003 ceph-mon[103344]: osdmap e226: 8 total, 8 up, 7 in 2024-08-17T17:28:27.700 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-08-17T17:28:27.770 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:27.798 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:27.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:27.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:27.889 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:28:27.889 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2024-08-17T17:28:27.890 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2024-08-17T17:28:27.890 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2024-08-17T17:28:27.890 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:28:27.900 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:27.922 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:27.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:27.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:28.024 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:28.047 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:28.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:28.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:28.148 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:28.170 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:28.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:28.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:28.272 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:28.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:28.295 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:28.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:28.396 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:28.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:28.419 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:28.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:28.484 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-17T17:28:28.520 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:28.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:28 smithi003 ceph-mon[100615]: osdmap e227: 8 total, 8 up, 7 in 2024-08-17T17:28:28.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:28 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:28.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:28 smithi003 ceph-mon[103344]: osdmap e227: 8 total, 8 up, 7 in 2024-08-17T17:28:28.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:28 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:28.549 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:28.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:28.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:28.650 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:28.673 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:28.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:28.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:28.713 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:28 smithi053 ceph-mon[89963]: osdmap e227: 8 total, 8 up, 7 in 2024-08-17T17:28:28.713 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:28 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:28.756 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:28:28.776 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:28.798 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:28.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:28.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:28.899 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:28.922 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:28.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:28.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:29.024 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:29.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:29.048 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:29.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:29.149 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:29.172 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:29.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:29.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:29.274 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:29.297 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:29.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:29.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:29.399 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:29.421 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:29.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:29.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:29.523 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:29.545 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:29.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:29.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:29.646 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:29.652 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:29 smithi053 ceph-mon[89963]: pgmap v615: 89 pgs: 4 creating+peering, 12 unknown, 73 active+clean; 4.6 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:29.652 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:29 smithi053 ceph-mon[89963]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:28:29.652 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:29 smithi053 ceph-mon[89963]: osdmap e228: 8 total, 8 up, 7 in 2024-08-17T17:28:29.669 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:29.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:29.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:29.771 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:29.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:29 smithi003 ceph-mon[103344]: pgmap v615: 89 pgs: 4 creating+peering, 12 unknown, 73 active+clean; 4.6 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:29.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:29 smithi003 ceph-mon[103344]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:28:29.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:29 smithi003 ceph-mon[103344]: osdmap e228: 8 total, 8 up, 7 in 2024-08-17T17:28:29.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:29 smithi003 ceph-mon[100615]: pgmap v615: 89 pgs: 4 creating+peering, 12 unknown, 73 active+clean; 4.6 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:29.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:29 smithi003 ceph-mon[100615]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:28:29.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:29 smithi003 ceph-mon[100615]: osdmap e228: 8 total, 8 up, 7 in 2024-08-17T17:28:29.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:29.794 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:29.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:29.895 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:29.917 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:29.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:29.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:30.019 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:30.041 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:30.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:30.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:30.143 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:30.165 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:30.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:30.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:30.267 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:30.290 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:30.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:30.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:30.338 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:28:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:28:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:28:30.360 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:28:30.360 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2024-08-17T17:28:30.361 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2024-08-17T17:28:30.361 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2024-08-17T17:28:30.361 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.361 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:68129 5.9 5:9510659a:::benchmark_data_smithi003_140793_object68128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-08-17T17:28:30.361 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:19:56.467405+0000", 2024-08-17T17:28:30.361 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 513.89082476299996, 2024-08-17T17:28:30.361 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.23049352, 2024-08-17T17:28:30.361 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:30.361 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2024-08-17T17:28:30.361 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:30.361 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:28:30.361 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:28:30.361 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 68129 2024-08-17T17:28:30.362 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.362 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:30.362 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.362 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:30.362 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.467405+0000", 2024-08-17T17:28:30.362 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.362 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.362 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.362 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:30.362 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.467405+0000", 2024-08-17T17:28:30.362 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.362 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.362 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.363 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:30.363 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.467404+0000", 2024-08-17T17:28:30.363 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:28:30.363 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.363 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.363 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:30.363 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.467439+0000", 2024-08-17T17:28:30.363 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4745e-05 2024-08-17T17:28:30.363 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.363 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.363 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:30.363 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.467454+0000", 2024-08-17T17:28:30.363 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5764e-05 2024-08-17T17:28:30.363 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.364 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.364 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:30.364 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.467461+0000", 2024-08-17T17:28:30.364 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4409999999999998e-06 2024-08-17T17:28:30.364 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.364 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.364 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.364 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.467501+0000", 2024-08-17T17:28:30.364 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0166999999999997e-05 2024-08-17T17:28:30.364 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.364 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.364 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:30.364 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.467596+0000", 2024-08-17T17:28:30.364 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5434999999999997e-05 2024-08-17T17:28:30.365 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.365 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.365 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,4", 2024-08-17T17:28:30.365 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.467689+0000", 2024-08-17T17:28:30.365 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.2597999999999997e-05 2024-08-17T17:28:30.365 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.365 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.365 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:30.365 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.469945+0000", 2024-08-17T17:28:30.365 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0022558510000000001 2024-08-17T17:28:30.365 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.365 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.365 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:30.366 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.471143+0000", 2024-08-17T17:28:30.366 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0011986799999999999 2024-08-17T17:28:30.366 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.366 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.366 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.366 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.473992+0000", 2024-08-17T17:28:30.366 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0028489959999999999 2024-08-17T17:28:30.366 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.366 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.366 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:28:30.366 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.474008+0000", 2024-08-17T17:28:30.366 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5574e-05 2024-08-17T17:28:30.367 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.367 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.367 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.367 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:57.483854+0000", 2024-08-17T17:28:30.367 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.009845922 2024-08-17T17:28:30.367 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.367 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.367 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:28:30.367 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:57.483866+0000", 2024-08-17T17:28:30.367 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1804000000000001e-05 2024-08-17T17:28:30.367 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.367 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.367 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.367 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:58.697814+0000", 2024-08-17T17:28:30.368 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2139484229999999 2024-08-17T17:28:30.368 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.368 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.368 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:30.368 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:58.697899+0000", 2024-08-17T17:28:30.368 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.4734999999999994e-05 2024-08-17T17:28:30.368 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.368 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:30.368 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.368 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.368 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.368 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:68113 5.4 5.7ee11e04 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-08-17T17:28:30.368 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:19:56.477714+0000", 2024-08-17T17:28:30.368 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 513.88051591299995, 2024-08-17T17:28:30.369 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0063266660000001, 2024-08-17T17:28:30.369 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:30.369 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2024-08-17T17:28:30.369 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:30.369 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:28:30.369 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:28:30.369 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 68113 2024-08-17T17:28:30.369 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.369 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:30.369 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.369 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:30.369 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477714+0000", 2024-08-17T17:28:30.369 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.369 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.370 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.370 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:30.370 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477714+0000", 2024-08-17T17:28:30.370 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.370 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.370 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.370 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:30.370 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477713+0000", 2024-08-17T17:28:30.370 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:28:30.370 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.370 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.370 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:30.370 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477757+0000", 2024-08-17T17:28:30.370 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3979e-05 2024-08-17T17:28:30.371 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.371 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.371 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:30.371 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477795+0000", 2024-08-17T17:28:30.371 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8472e-05 2024-08-17T17:28:30.371 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.371 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.371 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:30.371 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477803+0000", 2024-08-17T17:28:30.371 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.8690000000000004e-06 2024-08-17T17:28:30.371 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.371 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.371 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.372 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477873+0000", 2024-08-17T17:28:30.372 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0020999999999999e-05 2024-08-17T17:28:30.372 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.372 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.372 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:28:30.372 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477893+0000", 2024-08-17T17:28:30.372 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9724999999999998e-05 2024-08-17T17:28:30.372 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.372 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.372 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.372 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:57.484032+0000", 2024-08-17T17:28:30.372 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0061393169999999 2024-08-17T17:28:30.372 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.372 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.373 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:30.373 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:57.484041+0000", 2024-08-17T17:28:30.373 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.6859999999999995e-06 2024-08-17T17:28:30.373 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.373 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:30.373 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.373 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.373 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.373 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:68122 5.4 5.9fb398a4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-08-17T17:28:30.373 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:19:56.477817+0000", 2024-08-17T17:28:30.373 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 513.88041300500004, 2024-08-17T17:28:30.373 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0062544259999999, 2024-08-17T17:28:30.373 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:30.373 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2024-08-17T17:28:30.374 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:30.374 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:28:30.374 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:28:30.374 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 68122 2024-08-17T17:28:30.374 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.374 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:30.374 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.374 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:30.374 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477817+0000", 2024-08-17T17:28:30.374 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.374 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.374 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.374 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:30.374 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477817+0000", 2024-08-17T17:28:30.375 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.375 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.375 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.376 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:30.376 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477816+0000", 2024-08-17T17:28:30.376 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:28:30.376 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.376 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.376 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:30.376 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477855+0000", 2024-08-17T17:28:30.376 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9004999999999997e-05 2024-08-17T17:28:30.376 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.376 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.376 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:30.376 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477866+0000", 2024-08-17T17:28:30.377 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0944000000000001e-05 2024-08-17T17:28:30.377 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.377 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.377 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:30.377 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477872+0000", 2024-08-17T17:28:30.377 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9200000000000001e-06 2024-08-17T17:28:30.377 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.377 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.377 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.377 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477924+0000", 2024-08-17T17:28:30.377 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1924e-05 2024-08-17T17:28:30.377 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.377 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.377 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:28:30.378 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477932+0000", 2024-08-17T17:28:30.378 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.5100000000000001e-06 2024-08-17T17:28:30.378 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.378 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.378 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.378 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:57.484065+0000", 2024-08-17T17:28:30.378 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0061333269999999 2024-08-17T17:28:30.378 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.378 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.378 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:30.378 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:57.484072+0000", 2024-08-17T17:28:30.378 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4790000000000001e-06 2024-08-17T17:28:30.378 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.378 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:30.379 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.379 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.379 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.379 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:68126 5.4 5.7ec1dd54 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-08-17T17:28:30.379 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:19:56.477878+0000", 2024-08-17T17:28:30.379 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 513.88035184600005, 2024-08-17T17:28:30.379 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0062189070000001, 2024-08-17T17:28:30.379 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:30.379 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2024-08-17T17:28:30.379 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:30.379 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:28:30.379 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:28:30.379 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 68126 2024-08-17T17:28:30.380 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.380 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:30.380 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.380 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:30.380 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477878+0000", 2024-08-17T17:28:30.380 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.380 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.380 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.380 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:30.380 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477878+0000", 2024-08-17T17:28:30.380 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.380 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.380 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.381 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:30.381 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477878+0000", 2024-08-17T17:28:30.381 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:28:30.381 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.381 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.381 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:30.381 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477922+0000", 2024-08-17T17:28:30.381 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3866999999999999e-05 2024-08-17T17:28:30.381 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.381 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.381 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:30.381 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477931+0000", 2024-08-17T17:28:30.381 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5429999999999999e-06 2024-08-17T17:28:30.381 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.382 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.382 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:30.382 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477936+0000", 2024-08-17T17:28:30.382 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.8280000000000002e-06 2024-08-17T17:28:30.382 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.382 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.382 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.382 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477980+0000", 2024-08-17T17:28:30.382 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3630000000000001e-05 2024-08-17T17:28:30.382 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.382 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.382 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:28:30.382 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477985+0000", 2024-08-17T17:28:30.383 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1320000000000002e-06 2024-08-17T17:28:30.383 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.383 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.383 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.383 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:57.484091+0000", 2024-08-17T17:28:30.383 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0061058940000001 2024-08-17T17:28:30.383 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.383 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.383 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:30.383 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:57.484097+0000", 2024-08-17T17:28:30.383 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5540000000000003e-06 2024-08-17T17:28:30.383 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.383 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:30.384 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.384 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.384 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.384 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:68128 5.4 5.407770a4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e129)", 2024-08-17T17:28:30.384 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:19:56.477957+0000", 2024-08-17T17:28:30.384 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 513.88027342999999, 2024-08-17T17:28:30.384 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0061641539999999, 2024-08-17T17:28:30.384 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:30.384 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2024-08-17T17:28:30.384 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:30.384 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:28:30.384 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:28:30.384 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 68128 2024-08-17T17:28:30.385 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.385 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:30.385 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.385 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:30.385 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477957+0000", 2024-08-17T17:28:30.385 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.385 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.385 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.385 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:30.385 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477957+0000", 2024-08-17T17:28:30.385 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.385 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.385 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.386 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:30.386 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477956+0000", 2024-08-17T17:28:30.386 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:28:30.386 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.386 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.386 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:30.386 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477989+0000", 2024-08-17T17:28:30.386 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2774000000000002e-05 2024-08-17T17:28:30.386 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.386 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.386 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:30.386 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.477998+0000", 2024-08-17T17:28:30.386 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1549999999999996e-06 2024-08-17T17:28:30.387 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.387 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.387 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:30.387 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.478003+0000", 2024-08-17T17:28:30.387 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6079999999999998e-06 2024-08-17T17:28:30.387 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.387 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.387 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.387 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.478043+0000", 2024-08-17T17:28:30.387 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0331999999999999e-05 2024-08-17T17:28:30.387 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.387 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.387 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:28:30.387 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.478048+0000", 2024-08-17T17:28:30.388 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1529999999999996e-06 2024-08-17T17:28:30.388 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.388 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.388 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.388 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:57.484115+0000", 2024-08-17T17:28:30.388 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0060665440000001 2024-08-17T17:28:30.388 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.388 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.388 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:30.388 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:57.484121+0000", 2024-08-17T17:28:30.388 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1279999999999996e-06 2024-08-17T17:28:30.388 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.388 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:30.388 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.389 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.389 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.389 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:68137 5.4 5.a7da3c24 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-08-17T17:28:30.389 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:19:56.501938+0000", 2024-08-17T17:28:30.389 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 513.85629186100005, 2024-08-17T17:28:30.389 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.98222817500000004, 2024-08-17T17:28:30.389 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:30.389 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2024-08-17T17:28:30.389 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:30.389 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:28:30.389 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:28:30.389 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 68137 2024-08-17T17:28:30.389 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.389 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:30.390 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.390 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:30.390 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.501938+0000", 2024-08-17T17:28:30.390 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.390 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.390 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.390 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:30.390 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.501938+0000", 2024-08-17T17:28:30.390 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.390 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.390 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.390 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:30.390 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.501937+0000", 2024-08-17T17:28:30.390 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:28:30.391 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.391 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.391 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:30.391 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.501998+0000", 2024-08-17T17:28:30.391 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1072000000000001e-05 2024-08-17T17:28:30.391 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.391 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.391 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:30.391 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.502012+0000", 2024-08-17T17:28:30.391 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.401e-05 2024-08-17T17:28:30.391 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.391 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.391 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:30.391 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.502019+0000", 2024-08-17T17:28:30.392 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.725e-06 2024-08-17T17:28:30.392 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.392 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.392 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.392 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.502059+0000", 2024-08-17T17:28:30.392 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.053e-05 2024-08-17T17:28:30.392 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.392 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.392 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:28:30.392 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:56.502067+0000", 2024-08-17T17:28:30.392 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7910000000000005e-06 2024-08-17T17:28:30.392 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.392 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.392 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.392 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:57.484160+0000", 2024-08-17T17:28:30.393 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.98209291200000004 2024-08-17T17:28:30.393 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.393 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.393 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:30.393 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:19:57.484166+0000", 2024-08-17T17:28:30.393 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5830000000000003e-06 2024-08-17T17:28:30.393 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.393 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:30.393 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.393 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.393 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.393 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:51027 6.5 6:a62b8487:::benchmark_data_smithi003_175043_object51026:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-08-17T17:28:30.393 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:03.980042+0000", 2024-08-17T17:28:30.394 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 206.37818790399999, 2024-08-17T17:28:30.394 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.629104286, 2024-08-17T17:28:30.394 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:30.394 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2024-08-17T17:28:30.394 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:30.394 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:28:30.394 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:28:30.394 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 51027 2024-08-17T17:28:30.394 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.394 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:30.394 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.394 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:30.394 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:03.980042+0000", 2024-08-17T17:28:30.395 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.395 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.395 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.395 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:30.395 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:03.980042+0000", 2024-08-17T17:28:30.395 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.395 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.395 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.395 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:30.395 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:03.980041+0000", 2024-08-17T17:28:30.395 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:28:30.395 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.395 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.396 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:30.396 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:03.980090+0000", 2024-08-17T17:28:30.396 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9597e-05 2024-08-17T17:28:30.396 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.396 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.396 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:30.396 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:03.980101+0000", 2024-08-17T17:28:30.396 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1056000000000001e-05 2024-08-17T17:28:30.396 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.396 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.396 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:30.396 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:03.980106+0000", 2024-08-17T17:28:30.396 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1439999999999996e-06 2024-08-17T17:28:30.397 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.397 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.397 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.397 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:03.980149+0000", 2024-08-17T17:28:30.397 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3813999999999998e-05 2024-08-17T17:28:30.397 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.397 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.397 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:30.397 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:03.980282+0000", 2024-08-17T17:28:30.397 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013306400000000001 2024-08-17T17:28:30.397 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.397 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.397 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,3", 2024-08-17T17:28:30.397 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:03.980413+0000", 2024-08-17T17:28:30.398 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013048399999999999 2024-08-17T17:28:30.398 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.398 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.398 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:30.398 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:03.981965+0000", 2024-08-17T17:28:30.398 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0015519030000000001 2024-08-17T17:28:30.398 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.398 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.398 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:30.398 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:03.987450+0000", 2024-08-17T17:28:30.398 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0054850009999999998 2024-08-17T17:28:30.398 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.398 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.398 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.399 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:03.994306+0000", 2024-08-17T17:28:30.399 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0068563549999999997 2024-08-17T17:28:30.399 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.399 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.399 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:28:30.399 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:03.994316+0000", 2024-08-17T17:28:30.399 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.9750000000000002e-06 2024-08-17T17:28:30.399 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.399 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.399 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.399 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.498819+0000", 2024-08-17T17:28:30.399 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.50450318999999999 2024-08-17T17:28:30.399 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.399 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.400 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:28:30.400 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.498830+0000", 2024-08-17T17:28:30.400 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0706999999999999e-05 2024-08-17T17:28:30.400 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.400 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.400 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.400 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.609045+0000", 2024-08-17T17:28:30.400 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1102146129999999 2024-08-17T17:28:30.400 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.400 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.400 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:30.400 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.609147+0000", 2024-08-17T17:28:30.400 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000101789 2024-08-17T17:28:30.400 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.401 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:30.401 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.401 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.401 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.401 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:51039 6.5 6:acbe4258:::benchmark_data_smithi003_175043_object51038:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-08-17T17:28:30.401 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:04.000857+0000", 2024-08-17T17:28:30.401 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 206.35737277699999, 2024-08-17T17:28:30.401 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6213081819999999, 2024-08-17T17:28:30.401 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:30.401 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:30.401 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:30.401 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:28:30.401 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:28:30.401 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 51039 2024-08-17T17:28:30.402 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.402 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:30.402 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.402 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:30.402 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.000857+0000", 2024-08-17T17:28:30.402 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.402 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.402 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.402 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:30.402 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.000857+0000", 2024-08-17T17:28:30.402 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.402 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.402 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.403 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:30.403 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.000857+0000", 2024-08-17T17:28:30.403 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:28:30.403 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.403 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.403 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:30.403 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.000884+0000", 2024-08-17T17:28:30.403 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7793999999999999e-05 2024-08-17T17:28:30.403 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.403 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.403 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:30.403 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.000896+0000", 2024-08-17T17:28:30.403 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1399e-05 2024-08-17T17:28:30.403 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.404 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.404 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:30.404 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.000900+0000", 2024-08-17T17:28:30.404 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1609999999999996e-06 2024-08-17T17:28:30.404 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.404 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.404 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.404 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.000953+0000", 2024-08-17T17:28:30.404 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2871000000000003e-05 2024-08-17T17:28:30.404 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.404 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.404 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:28:30.404 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.000962+0000", 2024-08-17T17:28:30.404 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.8070000000000006e-06 2024-08-17T17:28:30.405 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.405 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.405 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.405 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.498857+0000", 2024-08-17T17:28:30.405 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.49789539199999999 2024-08-17T17:28:30.405 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.405 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.405 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:28:30.405 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.498879+0000", 2024-08-17T17:28:30.405 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1815e-05 2024-08-17T17:28:30.405 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.405 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.405 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.405 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.610645+0000", 2024-08-17T17:28:30.405 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.111765892 2024-08-17T17:28:30.406 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.406 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.406 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:30.406 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.610784+0000", 2024-08-17T17:28:30.406 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000138889 2024-08-17T17:28:30.406 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.406 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.406 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,6", 2024-08-17T17:28:30.406 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.610899+0000", 2024-08-17T17:28:30.406 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000115567 2024-08-17T17:28:30.406 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.406 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.406 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:30.406 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.614497+0000", 2024-08-17T17:28:30.407 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003598319 2024-08-17T17:28:30.407 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.407 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.407 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:30.407 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.618508+0000", 2024-08-17T17:28:30.407 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0040104010000000002 2024-08-17T17:28:30.407 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.407 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.407 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:30.407 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.619872+0000", 2024-08-17T17:28:30.407 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0013636659999999999 2024-08-17T17:28:30.407 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.407 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.407 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:30.408 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.619939+0000", 2024-08-17T17:28:30.408 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.7108999999999997e-05 2024-08-17T17:28:30.408 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.408 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.408 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:30.408 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.622166+0000", 2024-08-17T17:28:30.408 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0022268740000000002 2024-08-17T17:28:30.408 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.408 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:30.408 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.408 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.408 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.408 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:51079 6.5 6:ac56d4c0:::benchmark_data_smithi003_175043_object51078:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-08-17T17:28:30.408 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:04.620257+0000", 2024-08-17T17:28:30.409 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 205.73797298700001, 2024-08-17T17:28:30.409 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0032013129999999, 2024-08-17T17:28:30.409 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:30.409 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:30.409 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:30.409 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:28:30.409 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:28:30.409 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 51079 2024-08-17T17:28:30.409 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.409 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:30.409 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.409 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:30.409 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.620257+0000", 2024-08-17T17:28:30.409 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.409 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.409 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.410 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:30.410 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.620257+0000", 2024-08-17T17:28:30.410 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.410 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.410 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.410 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:30.410 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.620257+0000", 2024-08-17T17:28:30.410 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:28:30.410 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.410 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.410 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:30.410 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.620280+0000", 2024-08-17T17:28:30.410 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3575999999999999e-05 2024-08-17T17:28:30.410 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.410 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.411 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:30.411 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.620286+0000", 2024-08-17T17:28:30.411 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1430000000000003e-06 2024-08-17T17:28:30.411 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.411 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.411 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:30.411 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.620289+0000", 2024-08-17T17:28:30.411 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8820000000000001e-06 2024-08-17T17:28:30.411 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.411 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.411 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.411 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.620343+0000", 2024-08-17T17:28:30.411 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3631999999999997e-05 2024-08-17T17:28:30.411 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.411 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.412 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:28:30.412 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.620353+0000", 2024-08-17T17:28:30.412 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6690000000000001e-06 2024-08-17T17:28:30.412 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.412 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.412 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.412 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.612654+0000", 2024-08-17T17:28:30.412 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.99230118300000003 2024-08-17T17:28:30.412 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.412 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.412 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:30.412 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.612791+0000", 2024-08-17T17:28:30.412 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013729899999999999 2024-08-17T17:28:30.412 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.413 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.413 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,6", 2024-08-17T17:28:30.413 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.612967+0000", 2024-08-17T17:28:30.413 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00017623500000000001 2024-08-17T17:28:30.413 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.413 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.413 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:30.413 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.616374+0000", 2024-08-17T17:28:30.413 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003406708 2024-08-17T17:28:30.413 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.413 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.413 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:30.413 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.618561+0000", 2024-08-17T17:28:30.413 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0021864879999999999 2024-08-17T17:28:30.413 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.414 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.414 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:30.414 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.622299+0000", 2024-08-17T17:28:30.414 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0037389419999999999 2024-08-17T17:28:30.414 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.414 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.414 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:30.414 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.622346+0000", 2024-08-17T17:28:30.414 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6230999999999998e-05 2024-08-17T17:28:30.414 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.414 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.414 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:30.414 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.623458+0000", 2024-08-17T17:28:30.415 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001112785 2024-08-17T17:28:30.415 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.415 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:30.415 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.415 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.415 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.415 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:51103 6.5 6:adee92f5:::benchmark_data_smithi003_175043_object51102:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-08-17T17:28:30.415 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:04.672180+0000", 2024-08-17T17:28:30.415 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 205.68604974900001, 2024-08-17T17:28:30.415 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.95579269499999997, 2024-08-17T17:28:30.415 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:30.415 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:30.415 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:30.415 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:28:30.416 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:28:30.416 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 51103 2024-08-17T17:28:30.416 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.416 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:30.416 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.416 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:30.416 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.672180+0000", 2024-08-17T17:28:30.416 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.416 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.416 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.416 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:30.416 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.672180+0000", 2024-08-17T17:28:30.416 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.416 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.417 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.417 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:30.417 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.672180+0000", 2024-08-17T17:28:30.417 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:28:30.417 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.417 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.417 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:30.417 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.672245+0000", 2024-08-17T17:28:30.417 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5209000000000005e-05 2024-08-17T17:28:30.417 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.417 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.417 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:30.417 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.672258+0000", 2024-08-17T17:28:30.417 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2855e-05 2024-08-17T17:28:30.418 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.418 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.418 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:30.418 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.672263+0000", 2024-08-17T17:28:30.418 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4079999999999997e-06 2024-08-17T17:28:30.418 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.418 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.418 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.418 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.672307+0000", 2024-08-17T17:28:30.418 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4305999999999999e-05 2024-08-17T17:28:30.418 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.418 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.418 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:28:30.418 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.672315+0000", 2024-08-17T17:28:30.419 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3900000000000004e-06 2024-08-17T17:28:30.419 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.419 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.419 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.419 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.614552+0000", 2024-08-17T17:28:30.419 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.94223699100000002 2024-08-17T17:28:30.419 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.419 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.419 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:30.419 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.614660+0000", 2024-08-17T17:28:30.419 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000107824 2024-08-17T17:28:30.419 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.419 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.419 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,6", 2024-08-17T17:28:30.420 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.614765+0000", 2024-08-17T17:28:30.420 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000105938 2024-08-17T17:28:30.420 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.420 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.420 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:30.420 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.616527+0000", 2024-08-17T17:28:30.420 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0017612249999999999 2024-08-17T17:28:30.420 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.420 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.420 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:30.420 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.623505+0000", 2024-08-17T17:28:30.420 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0069780909999999996 2024-08-17T17:28:30.420 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.420 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.421 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:30.421 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.625477+0000", 2024-08-17T17:28:30.421 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0019721510000000001 2024-08-17T17:28:30.421 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.421 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.421 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:30.421 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.625547+0000", 2024-08-17T17:28:30.421 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0289000000000004e-05 2024-08-17T17:28:30.421 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.421 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.421 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:30.421 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.627973+0000", 2024-08-17T17:28:30.421 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0024258679999999999 2024-08-17T17:28:30.421 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.421 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:30.422 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.422 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.422 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.422 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:130180 6.5 6:aab40d0b:::benchmark_data_smithi003_175043_object31531:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-08-17T17:28:30.422 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:26:39.395823+0000", 2024-08-17T17:28:30.422 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 110.962407115, 2024-08-17T17:28:30.422 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0299970549999999, 2024-08-17T17:28:30.422 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:30.422 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2024-08-17T17:28:30.422 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:30.422 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:28:30.422 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:28:30.426 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 130180 2024-08-17T17:28:30.426 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.426 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:30.426 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.426 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:30.426 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.395823+0000", 2024-08-17T17:28:30.426 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.426 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.427 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.427 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:30.427 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.395823+0000", 2024-08-17T17:28:30.427 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.427 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.427 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.427 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:30.427 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.395822+0000", 2024-08-17T17:28:30.427 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:28:30.427 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.427 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.427 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:30.428 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.395828+0000", 2024-08-17T17:28:30.428 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6110000000000001e-06 2024-08-17T17:28:30.428 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.428 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.428 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:30.428 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.395839+0000", 2024-08-17T17:28:30.428 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.168e-05 2024-08-17T17:28:30.428 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.428 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.428 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:30.428 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.395846+0000", 2024-08-17T17:28:30.428 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0700000000000001e-06 2024-08-17T17:28:30.428 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.428 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.428 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.429 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.395889+0000", 2024-08-17T17:28:30.429 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2938000000000001e-05 2024-08-17T17:28:30.429 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.429 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.429 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:30.429 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.395999+0000", 2024-08-17T17:28:30.429 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000109954 2024-08-17T17:28:30.429 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.429 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.429 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,6", 2024-08-17T17:28:30.429 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.396086+0000", 2024-08-17T17:28:30.429 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.6395000000000002e-05 2024-08-17T17:28:30.429 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.429 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.430 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:30.430 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.399219+0000", 2024-08-17T17:28:30.430 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0031329470000000001 2024-08-17T17:28:30.430 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.430 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.430 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:30.430 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.399351+0000", 2024-08-17T17:28:30.430 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000131965 2024-08-17T17:28:30.430 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.430 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.430 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.430 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.412194+0000", 2024-08-17T17:28:30.430 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012843108000000001 2024-08-17T17:28:30.430 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.431 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.431 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:28:30.431 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.412206+0000", 2024-08-17T17:28:30.431 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2860999999999999e-05 2024-08-17T17:28:30.431 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.431 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.431 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.431 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.425771+0000", 2024-08-17T17:28:30.431 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.013564382 2024-08-17T17:28:30.431 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.431 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.431 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:30.431 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.425820+0000", 2024-08-17T17:28:30.431 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9215000000000002e-05 2024-08-17T17:28:30.432 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.432 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:30.432 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.432 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.432 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.432 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:130209 6.5 6:a6e9550c:::benchmark_data_smithi003_175043_object31560:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-08-17T17:28:30.432 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:26:39.452057+0000", 2024-08-17T17:28:30.432 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 110.90617356200001, 2024-08-17T17:28:30.432 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.98617804200000003, 2024-08-17T17:28:30.432 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:30.432 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:30.432 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:30.432 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:28:30.432 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:28:30.432 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 130209 2024-08-17T17:28:30.433 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.433 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:30.433 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.433 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:30.433 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.452057+0000", 2024-08-17T17:28:30.433 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.433 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.433 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.433 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:30.433 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.452057+0000", 2024-08-17T17:28:30.433 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.433 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.433 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.433 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:30.433 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.452056+0000", 2024-08-17T17:28:30.434 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:28:30.434 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.434 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.434 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:30.434 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.452060+0000", 2024-08-17T17:28:30.434 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:30.435 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1189999999999999e-06 2024-08-17T17:28:30.436 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.436 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.436 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:30.436 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.452069+0000", 2024-08-17T17:28:30.436 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.7190000000000007e-06 2024-08-17T17:28:30.436 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.436 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.436 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:30.436 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.452072+0000", 2024-08-17T17:28:30.436 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2200000000000001e-06 2024-08-17T17:28:30.436 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.436 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.436 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.437 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.452128+0000", 2024-08-17T17:28:30.437 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6128999999999997e-05 2024-08-17T17:28:30.437 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.437 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.437 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:28:30.437 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.452137+0000", 2024-08-17T17:28:30.437 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1090000000000004e-06 2024-08-17T17:28:30.437 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.437 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.437 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.437 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.426086+0000", 2024-08-17T17:28:30.437 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.973948429 2024-08-17T17:28:30.437 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.438 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.438 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:30.438 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.426212+0000", 2024-08-17T17:28:30.438 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000126489 2024-08-17T17:28:30.438 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.438 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.438 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,6", 2024-08-17T17:28:30.438 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.426345+0000", 2024-08-17T17:28:30.438 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013292100000000001 2024-08-17T17:28:30.438 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.438 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.438 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:30.438 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.432225+0000", 2024-08-17T17:28:30.439 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0058797520000000002 2024-08-17T17:28:30.439 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.439 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.439 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:30.439 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.434985+0000", 2024-08-17T17:28:30.439 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0027605540000000001 2024-08-17T17:28:30.439 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.439 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.439 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:30.439 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.438156+0000", 2024-08-17T17:28:30.439 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003170679 2024-08-17T17:28:30.439 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.439 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.439 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:30.440 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.438199+0000", 2024-08-17T17:28:30.440 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2528999999999999e-05 2024-08-17T17:28:30.440 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.440 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.440 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:30.440 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.438235+0000", 2024-08-17T17:28:30.440 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6001999999999999e-05 2024-08-17T17:28:30.440 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.440 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:30.440 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.440 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.440 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.440 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:130221 6.5 6:ab1fee0c:::benchmark_data_smithi003_175043_object31572:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-08-17T17:28:30.441 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:26:39.467209+0000", 2024-08-17T17:28:30.441 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 110.891021112, 2024-08-17T17:28:30.441 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.97115574800000004, 2024-08-17T17:28:30.441 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:30.441 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:30.441 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:30.441 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:28:30.441 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:28:30.441 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 130221 2024-08-17T17:28:30.441 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.441 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:30.441 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.442 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:30.442 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.467209+0000", 2024-08-17T17:28:30.442 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.442 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.442 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.442 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:30.442 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.467209+0000", 2024-08-17T17:28:30.442 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.442 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.442 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.442 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:30.442 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.467208+0000", 2024-08-17T17:28:30.443 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:28:30.443 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.443 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.443 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:30.443 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.467214+0000", 2024-08-17T17:28:30.443 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6829999999999996e-06 2024-08-17T17:28:30.443 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.443 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.443 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:30.443 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.467228+0000", 2024-08-17T17:28:30.443 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4124000000000001e-05 2024-08-17T17:28:30.443 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.443 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.444 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:30.444 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.467235+0000", 2024-08-17T17:28:30.444 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.7320000000000001e-06 2024-08-17T17:28:30.444 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.444 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.444 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.444 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.467292+0000", 2024-08-17T17:28:30.444 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7442999999999999e-05 2024-08-17T17:28:30.444 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.444 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.444 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:28:30.444 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.467304+0000", 2024-08-17T17:28:30.444 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.163e-05 2024-08-17T17:28:30.444 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.445 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.445 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.445 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.429166+0000", 2024-08-17T17:28:30.445 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.96186255099999995 2024-08-17T17:28:30.445 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.445 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.445 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:30.445 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.429300+0000", 2024-08-17T17:28:30.445 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000133867 2024-08-17T17:28:30.445 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.445 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.445 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,6", 2024-08-17T17:28:30.445 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.429438+0000", 2024-08-17T17:28:30.445 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000137508 2024-08-17T17:28:30.446 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.446 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.446 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:30.446 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.434988+0000", 2024-08-17T17:28:30.446 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0055507059999999999 2024-08-17T17:28:30.446 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.446 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.446 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:30.446 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.438099+0000", 2024-08-17T17:28:30.446 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0031103350000000001 2024-08-17T17:28:30.446 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.446 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.446 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:30.447 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.438298+0000", 2024-08-17T17:28:30.447 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00019952500000000001 2024-08-17T17:28:30.447 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.447 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.447 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:30.447 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.438333+0000", 2024-08-17T17:28:30.447 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5238999999999999e-05 2024-08-17T17:28:30.447 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.447 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.447 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:30.447 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.438365+0000", 2024-08-17T17:28:30.447 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1327e-05 2024-08-17T17:28:30.447 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.447 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:30.448 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.448 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.448 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.448 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:130224 6.5 6:a603362e:::benchmark_data_smithi003_175043_object31575:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-08-17T17:28:30.448 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:26:39.472409+0000", 2024-08-17T17:28:30.448 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 110.88582161399999, 2024-08-17T17:28:30.448 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.97190938100000002, 2024-08-17T17:28:30.448 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:30.448 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:30.448 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:30.448 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:28:30.448 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:28:30.448 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 130224 2024-08-17T17:28:30.448 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.449 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:30.449 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.449 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:30.449 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.472409+0000", 2024-08-17T17:28:30.449 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.449 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.449 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.449 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:30.449 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.472409+0000", 2024-08-17T17:28:30.449 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.449 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.449 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.449 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:30.449 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.472407+0000", 2024-08-17T17:28:30.449 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:28:30.449 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.450 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.450 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:30.450 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.472414+0000", 2024-08-17T17:28:30.450 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1879999999999999e-06 2024-08-17T17:28:30.450 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.450 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.450 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:30.450 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.472426+0000", 2024-08-17T17:28:30.450 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2283000000000001e-05 2024-08-17T17:28:30.450 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.450 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.450 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:30.450 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.472432+0000", 2024-08-17T17:28:30.450 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.472e-06 2024-08-17T17:28:30.451 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.451 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.451 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.451 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.472487+0000", 2024-08-17T17:28:30.451 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4193000000000001e-05 2024-08-17T17:28:30.451 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.451 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.451 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:28:30.451 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.472496+0000", 2024-08-17T17:28:30.451 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1749999999999994e-06 2024-08-17T17:28:30.451 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.451 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.451 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.452 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.432270+0000", 2024-08-17T17:28:30.452 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.95977414999999999 2024-08-17T17:28:30.452 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.452 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.452 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:30.452 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.432362+0000", 2024-08-17T17:28:30.452 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1762999999999995e-05 2024-08-17T17:28:30.452 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.452 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.452 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,6", 2024-08-17T17:28:30.452 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.432432+0000", 2024-08-17T17:28:30.452 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0352000000000007e-05 2024-08-17T17:28:30.452 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.453 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.453 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:30.453 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.438269+0000", 2024-08-17T17:28:30.453 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0058365830000000002 2024-08-17T17:28:30.453 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.453 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.453 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:30.453 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.438274+0000", 2024-08-17T17:28:30.453 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1590000000000001e-06 2024-08-17T17:28:30.453 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.453 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.453 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:30.453 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.444187+0000", 2024-08-17T17:28:30.453 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0059136500000000003 2024-08-17T17:28:30.454 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.454 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.454 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:30.454 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.444280+0000", 2024-08-17T17:28:30.454 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.2147999999999999e-05 2024-08-17T17:28:30.454 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.454 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.454 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:30.454 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.444318+0000", 2024-08-17T17:28:30.454 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8347999999999999e-05 2024-08-17T17:28:30.454 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.454 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:30.454 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.455 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.455 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.455 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:130247 6.5 6:af8bf089:::benchmark_data_smithi003_175043_object31598:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-08-17T17:28:30.455 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:26:39.498424+0000", 2024-08-17T17:28:30.455 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 110.85980601200001, 2024-08-17T17:28:30.455 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.94608363699999998, 2024-08-17T17:28:30.455 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:30.455 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:30.455 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:30.455 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:28:30.455 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:28:30.455 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 130247 2024-08-17T17:28:30.456 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.456 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:30.456 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.456 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:30.456 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.498424+0000", 2024-08-17T17:28:30.456 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.456 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.456 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.456 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:30.456 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.498424+0000", 2024-08-17T17:28:30.456 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.456 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.456 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.456 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:30.456 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.498424+0000", 2024-08-17T17:28:30.457 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:28:30.457 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.457 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.457 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:30.457 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.498427+0000", 2024-08-17T17:28:30.457 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8389999999999998e-06 2024-08-17T17:28:30.457 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.457 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.457 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:30.457 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.498432+0000", 2024-08-17T17:28:30.457 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6030000000000004e-06 2024-08-17T17:28:30.457 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.457 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.458 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:30.458 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.498437+0000", 2024-08-17T17:28:30.458 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2340000000000002e-06 2024-08-17T17:28:30.458 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.458 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.458 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.458 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.498499+0000", 2024-08-17T17:28:30.458 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1822000000000005e-05 2024-08-17T17:28:30.458 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.458 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.458 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:28:30.458 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.498507+0000", 2024-08-17T17:28:30.458 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0600000000000008e-06 2024-08-17T17:28:30.459 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.459 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.459 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.459 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.435029+0000", 2024-08-17T17:28:30.459 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.93652218499999995 2024-08-17T17:28:30.459 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.459 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.459 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:30.459 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.435151+0000", 2024-08-17T17:28:30.459 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000121217 2024-08-17T17:28:30.459 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.459 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.459 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,6", 2024-08-17T17:28:30.459 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.435282+0000", 2024-08-17T17:28:30.459 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000130918 2024-08-17T17:28:30.459 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.460 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.460 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:30.460 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.438410+0000", 2024-08-17T17:28:30.460 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0031286439999999999 2024-08-17T17:28:30.460 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.460 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.460 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:30.460 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.438415+0000", 2024-08-17T17:28:30.460 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3460000000000004e-06 2024-08-17T17:28:30.460 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.460 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.460 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:30.460 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.444399+0000", 2024-08-17T17:28:30.460 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0059842430000000002 2024-08-17T17:28:30.460 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.461 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.461 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:30.461 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.444473+0000", 2024-08-17T17:28:30.461 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3987999999999997e-05 2024-08-17T17:28:30.461 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.461 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.461 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:30.461 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.444508+0000", 2024-08-17T17:28:30.461 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4992e-05 2024-08-17T17:28:30.461 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.461 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:30.461 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.461 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.461 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.461 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:130260 6.5 6:a2aa193e:::benchmark_data_smithi003_175043_object31611:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-08-17T17:28:30.461 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:26:39.514808+0000", 2024-08-17T17:28:30.462 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 110.843421734, 2024-08-17T17:28:30.462 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.92982520000000002, 2024-08-17T17:28:30.462 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:30.462 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:30.462 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:30.462 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:28:30.462 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:28:30.462 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 130260 2024-08-17T17:28:30.462 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.462 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:30.462 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.462 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:30.462 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.514808+0000", 2024-08-17T17:28:30.463 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.463 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.463 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.463 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:30.463 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.514808+0000", 2024-08-17T17:28:30.463 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.463 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.463 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.463 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:30.463 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.514807+0000", 2024-08-17T17:28:30.463 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:28:30.463 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.463 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.463 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:30.464 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.514813+0000", 2024-08-17T17:28:30.464 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7690000000000002e-06 2024-08-17T17:28:30.464 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.464 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.464 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:30.464 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.514825+0000", 2024-08-17T17:28:30.464 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2503999999999999e-05 2024-08-17T17:28:30.464 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.464 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.464 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:30.464 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.514831+0000", 2024-08-17T17:28:30.464 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3240000000000002e-06 2024-08-17T17:28:30.464 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.465 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.465 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.465 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.514874+0000", 2024-08-17T17:28:30.465 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2984e-05 2024-08-17T17:28:30.465 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.465 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.465 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:28:30.465 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.514882+0000", 2024-08-17T17:28:30.465 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.8849999999999999e-06 2024-08-17T17:28:30.465 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.465 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.465 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.465 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.438431+0000", 2024-08-17T17:28:30.465 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.92354913000000005 2024-08-17T17:28:30.465 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.465 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.466 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:30.466 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.438527+0000", 2024-08-17T17:28:30.466 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5915999999999994e-05 2024-08-17T17:28:30.466 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.466 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.466 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,6", 2024-08-17T17:28:30.466 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.438631+0000", 2024-08-17T17:28:30.466 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010438999999999999 2024-08-17T17:28:30.466 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.466 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.466 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:30.466 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.444332+0000", 2024-08-17T17:28:30.466 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0057008429999999997 2024-08-17T17:28:30.467 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.467 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.467 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:30.467 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.444369+0000", 2024-08-17T17:28:30.467 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6724000000000003e-05 2024-08-17T17:28:30.467 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.467 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.467 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:30.467 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.444564+0000", 2024-08-17T17:28:30.467 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000195582 2024-08-17T17:28:30.467 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.467 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.467 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:30.467 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.444602+0000", 2024-08-17T17:28:30.468 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.7768000000000002e-05 2024-08-17T17:28:30.468 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.468 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.468 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:30.468 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.444634+0000", 2024-08-17T17:28:30.468 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1613999999999999e-05 2024-08-17T17:28:30.468 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.468 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:30.468 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.468 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.468 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.468 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:130267 6.5 6:ad16fc1b:::benchmark_data_smithi003_175043_object31618:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-08-17T17:28:30.469 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:26:39.521044+0000", 2024-08-17T17:28:30.469 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 110.83718578, 2024-08-17T17:28:30.469 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.93381886199999997, 2024-08-17T17:28:30.469 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:30.469 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:30.469 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:30.469 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:28:30.469 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:28:30.469 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 130267 2024-08-17T17:28:30.469 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.469 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:30.469 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.469 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:30.469 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.521044+0000", 2024-08-17T17:28:30.469 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.470 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.470 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.470 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:30.470 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.521044+0000", 2024-08-17T17:28:30.470 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.470 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.470 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.470 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:30.470 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.521044+0000", 2024-08-17T17:28:30.470 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:28:30.470 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.470 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.470 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:30.471 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.521047+0000", 2024-08-17T17:28:30.471 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9409999999999999e-06 2024-08-17T17:28:30.471 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.471 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.471 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:30.471 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.521054+0000", 2024-08-17T17:28:30.471 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9299999999999997e-06 2024-08-17T17:28:30.471 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.471 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.471 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:30.471 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.521058+0000", 2024-08-17T17:28:30.471 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5900000000000001e-06 2024-08-17T17:28:30.471 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.471 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.472 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.472 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.521117+0000", 2024-08-17T17:28:30.472 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8403999999999999e-05 2024-08-17T17:28:30.472 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.472 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.472 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:28:30.472 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.521127+0000", 2024-08-17T17:28:30.472 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0023e-05 2024-08-17T17:28:30.472 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.472 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.472 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.472 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.441476+0000", 2024-08-17T17:28:30.472 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.92034908999999998 2024-08-17T17:28:30.472 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.473 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.473 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:30.473 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.441571+0000", 2024-08-17T17:28:30.473 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.4856000000000001e-05 2024-08-17T17:28:30.473 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.473 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.473 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,6", 2024-08-17T17:28:30.473 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.441699+0000", 2024-08-17T17:28:30.473 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012802899999999999 2024-08-17T17:28:30.473 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.473 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.473 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:30.473 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.448154+0000", 2024-08-17T17:28:30.473 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0064556489999999999 2024-08-17T17:28:30.474 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.474 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.474 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:30.474 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.450964+0000", 2024-08-17T17:28:30.474 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00280949 2024-08-17T17:28:30.474 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.474 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.474 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:30.474 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.454778+0000", 2024-08-17T17:28:30.474 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003814362 2024-08-17T17:28:30.474 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.474 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.474 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:30.474 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.454825+0000", 2024-08-17T17:28:30.475 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6424000000000001e-05 2024-08-17T17:28:30.475 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.475 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.475 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:30.475 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.454863+0000", 2024-08-17T17:28:30.475 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8622999999999999e-05 2024-08-17T17:28:30.475 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.475 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:30.475 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.475 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.475 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.475 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:130296 6.5 6:a964ae2d:::benchmark_data_smithi003_175043_object31647:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-08-17T17:28:30.475 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:26:39.561033+0000", 2024-08-17T17:28:30.475 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 110.79719699499999, 2024-08-17T17:28:30.476 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.90031448700000005, 2024-08-17T17:28:30.476 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:30.476 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:30.476 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:30.476 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:28:30.476 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:28:30.476 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 130296 2024-08-17T17:28:30.476 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.476 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:30.476 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.476 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:30.476 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.561033+0000", 2024-08-17T17:28:30.476 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.476 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.477 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.477 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:30.477 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.561033+0000", 2024-08-17T17:28:30.477 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.477 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.477 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.477 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:30.477 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.561031+0000", 2024-08-17T17:28:30.477 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:28:30.477 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.477 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.477 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:30.477 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.561040+0000", 2024-08-17T17:28:30.478 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.7329999999999992e-06 2024-08-17T17:28:30.478 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.478 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.478 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:30.478 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.561056+0000", 2024-08-17T17:28:30.478 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6164999999999999e-05 2024-08-17T17:28:30.478 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.478 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.478 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:30.478 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.561063+0000", 2024-08-17T17:28:30.478 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.1790000000000002e-06 2024-08-17T17:28:30.478 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.478 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.479 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.479 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.561114+0000", 2024-08-17T17:28:30.479 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0864e-05 2024-08-17T17:28:30.479 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.479 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.479 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:28:30.479 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.561127+0000", 2024-08-17T17:28:30.479 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3145999999999999e-05 2024-08-17T17:28:30.479 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.479 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.479 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.479 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.444677+0000", 2024-08-17T17:28:30.479 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.88354925299999998 2024-08-17T17:28:30.479 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.480 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.480 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:30.480 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.444774+0000", 2024-08-17T17:28:30.480 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6825999999999998e-05 2024-08-17T17:28:30.480 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.480 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.480 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,6", 2024-08-17T17:28:30.480 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.444893+0000", 2024-08-17T17:28:30.480 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011908099999999999 2024-08-17T17:28:30.480 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.480 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.480 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:30.480 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.454678+0000", 2024-08-17T17:28:30.480 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0097858660000000007 2024-08-17T17:28:30.481 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.481 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.481 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:30.481 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.454687+0000", 2024-08-17T17:28:30.481 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.5329999999999994e-06 2024-08-17T17:28:30.481 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.481 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.481 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:30.481 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.461279+0000", 2024-08-17T17:28:30.481 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.006592194 2024-08-17T17:28:30.481 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.481 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.481 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:30.481 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.461312+0000", 2024-08-17T17:28:30.482 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3081999999999999e-05 2024-08-17T17:28:30.482 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.482 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.482 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:30.482 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.461348+0000", 2024-08-17T17:28:30.482 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5355999999999998e-05 2024-08-17T17:28:30.482 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.482 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:30.482 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.482 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.482 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.482 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:130341 6.5 6:aac8c266:::benchmark_data_smithi003_175043_object31692:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-08-17T17:28:30.482 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:26:39.624358+0000", 2024-08-17T17:28:30.482 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 110.733871663, 2024-08-17T17:28:30.483 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.84451184400000001, 2024-08-17T17:28:30.483 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:30.483 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:30.483 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:30.483 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:28:30.483 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:28:30.483 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 130341 2024-08-17T17:28:30.483 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.483 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:30.483 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.483 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:30.483 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.624358+0000", 2024-08-17T17:28:30.483 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.484 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.484 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.484 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:30.484 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.624358+0000", 2024-08-17T17:28:30.484 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.484 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.484 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.484 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:30.484 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.624358+0000", 2024-08-17T17:28:30.484 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:28:30.484 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.484 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.484 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:30.484 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.624363+0000", 2024-08-17T17:28:30.485 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9420000000000003e-06 2024-08-17T17:28:30.485 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.485 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.485 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:30.485 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.624375+0000", 2024-08-17T17:28:30.485 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1846e-05 2024-08-17T17:28:30.485 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.485 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.485 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:30.485 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.624380+0000", 2024-08-17T17:28:30.485 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2440000000000001e-06 2024-08-17T17:28:30.485 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.485 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.486 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.486 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.624438+0000", 2024-08-17T17:28:30.486 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8575000000000001e-05 2024-08-17T17:28:30.486 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.486 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.486 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:28:30.486 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.624450+0000", 2024-08-17T17:28:30.486 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1609e-05 2024-08-17T17:28:30.486 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.486 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.486 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.486 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.448181+0000", 2024-08-17T17:28:30.486 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.82373111799999998 2024-08-17T17:28:30.487 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.487 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.487 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:30.487 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.448289+0000", 2024-08-17T17:28:30.487 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010764600000000001 2024-08-17T17:28:30.487 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.487 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.487 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,6", 2024-08-17T17:28:30.487 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.448396+0000", 2024-08-17T17:28:30.487 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000106796 2024-08-17T17:28:30.487 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.487 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.487 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:30.487 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.454737+0000", 2024-08-17T17:28:30.488 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0063414450000000002 2024-08-17T17:28:30.488 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.488 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.488 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:30.488 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.457949+0000", 2024-08-17T17:28:30.488 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0032123429999999999 2024-08-17T17:28:30.488 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.488 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.488 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:30.488 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.468776+0000", 2024-08-17T17:28:30.488 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010827041000000001 2024-08-17T17:28:30.488 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.488 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.489 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:30.489 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.468828+0000", 2024-08-17T17:28:30.489 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1746000000000002e-05 2024-08-17T17:28:30.489 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.489 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.489 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:30.489 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.468870+0000", 2024-08-17T17:28:30.489 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2222000000000003e-05 2024-08-17T17:28:30.489 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.489 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:30.489 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.489 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.489 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.490 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:130373 6.5 6:a027210d:::benchmark_data_smithi003_175043_object31724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-08-17T17:28:30.490 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:26:39.667523+0000", 2024-08-17T17:28:30.490 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 110.69070712600001, 2024-08-17T17:28:30.490 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.81279908999999995, 2024-08-17T17:28:30.490 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:28:30.490 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:28:30.490 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:28:30.490 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:28:30.490 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:28:30.490 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 130373 2024-08-17T17:28:30.490 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.490 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:28:30.490 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.490 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:28:30.491 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.667523+0000", 2024-08-17T17:28:30.491 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.491 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.491 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.491 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:28:30.491 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.667523+0000", 2024-08-17T17:28:30.491 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:28:30.491 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.491 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.491 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:28:30.491 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.667522+0000", 2024-08-17T17:28:30.491 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:28:30.492 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.492 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.492 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:28:30.492 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.667527+0000", 2024-08-17T17:28:30.492 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.109e-06 2024-08-17T17:28:30.492 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.492 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.492 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:28:30.492 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.667536+0000", 2024-08-17T17:28:30.492 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1870000000000003e-06 2024-08-17T17:28:30.492 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.492 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.492 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:28:30.492 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.667542+0000", 2024-08-17T17:28:30.492 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.601e-06 2024-08-17T17:28:30.493 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.493 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.493 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.493 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.667591+0000", 2024-08-17T17:28:30.493 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.8463e-05 2024-08-17T17:28:30.493 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.493 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.493 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:28:30.493 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.667600+0000", 2024-08-17T17:28:30.493 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.2520000000000005e-06 2024-08-17T17:28:30.493 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.493 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.493 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:28:30.493 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.454925+0000", 2024-08-17T17:28:30.494 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.78732473300000005 2024-08-17T17:28:30.494 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.494 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.494 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:28:30.494 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.455030+0000", 2024-08-17T17:28:30.494 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010560699999999999 2024-08-17T17:28:30.494 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.494 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.494 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,6", 2024-08-17T17:28:30.494 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.455139+0000", 2024-08-17T17:28:30.494 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000108921 2024-08-17T17:28:30.494 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.494 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.495 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:30.495 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.461260+0000", 2024-08-17T17:28:30.495 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0061209690000000004 2024-08-17T17:28:30.495 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.495 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.495 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:28:30.495 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.461395+0000", 2024-08-17T17:28:30.495 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013491600000000001 2024-08-17T17:28:30.495 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.495 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.495 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:28:30.495 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.480250+0000", 2024-08-17T17:28:30.495 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.018854427 2024-08-17T17:28:30.495 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.495 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.496 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:28:30.496 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.480297+0000", 2024-08-17T17:28:30.496 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7583999999999997e-05 2024-08-17T17:28:30.496 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:28:30.496 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:28:30.496 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:28:30.496 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.480322+0000", 2024-08-17T17:28:30.496 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4930999999999999e-05 2024-08-17T17:28:30.496 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.496 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:30.496 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.496 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:28:30.496 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:28:30.497 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:28:30.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:30.520 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:30.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:30.601 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:28:30.621 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:30.643 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:30.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:30.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:30.745 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:30.767 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:30.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:30.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:30.869 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:30.891 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:30.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:30.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:30.993 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:31.014 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:31.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:31.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:31.116 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:31.138 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:31.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:31.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:31.201 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:28:31.201 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:28:31.201 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:28:31.201 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:28:31.201 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:28:31.202 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:28:31.202 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:28:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:28:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:28:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:28:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:28:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:28:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:28:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:28:31.203 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:28:31.203 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:28:31.204 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-17T17:28:31.239 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:31.262 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:31.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:31.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:31.311 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:28:31.312 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:28:31.363 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:31.385 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:31.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:31.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:31.487 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:31.509 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:31.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:31.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:31.611 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:31.634 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:31.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:31.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:31.729 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:28:31.736 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:31.762 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:31.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:31.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:31.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:31 smithi003 ceph-mon[100615]: pgmap v617: 89 pgs: 4 creating+peering, 1 unknown, 84 active+clean; 4.6 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:31.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:31 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3612729244' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:28:31.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:31 smithi003 ceph-mon[103344]: pgmap v617: 89 pgs: 4 creating+peering, 1 unknown, 84 active+clean; 4.6 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:31.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:31 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3612729244' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:28:31.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:31.887 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:31.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:31.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:31.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:31 smithi053 ceph-mon[89963]: pgmap v617: 89 pgs: 4 creating+peering, 1 unknown, 84 active+clean; 4.6 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:31.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:31 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3612729244' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:28:31.988 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:32.010 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:32.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:32.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:32.111 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:32.133 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:32.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:32.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:32.191 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":617,"stamp":"2024-08-17T17:28:29.950425+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790221,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":6376,"num_read_kb":13833,"num_write":1122,"num_write_kb":15039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":11937158,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1181,"ondisk_log_size":1181,"up":264,"acting":264,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":225,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2771028,"kb_used_data":481336,"kb_used_omap":33,"kb_used_meta":2289566,"kb_avail":653445036,"statfs":{"total":671965249536,"available":669127716864,"internally_reserved":0,"allocated":492888064,"data_stored":483820843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34068,"internal_metadata":2344516332},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001128"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":306,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.497472+0000","last_change":"2024-08-17T17:17:32.495652+0000","last_active":"2024-08-17T17:28:26.497472+0000","last_peered":"2024-08-17T17:28:26.497472+0000","last_clean":"2024-08-17T17:28:26.497472+0000","last_became_active":"2024-08-17T17:17:32.495218+0000","last_became_peered":"2024-08-17T17:17:32.495218+0000","last_unstale":"2024-08-17T17:28:26.497472+0000","last_undegraded":"2024-08-17T17:28:26.497472+0000","last_fullsized":"2024-08-17T17:28:26.497472+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:11:45.821282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":204,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.500801+0000","last_change":"2024-08-17T17:18:59.424887+0000","last_active":"2024-08-17T17:28:27.500801+0000","last_peered":"2024-08-17T17:28:27.500801+0000","last_clean":"2024-08-17T17:28:27.500801+0000","last_became_active":"2024-08-17T17:18:59.411539+0000","last_became_peered":"2024-08-17T17:18:59.411539+0000","last_unstale":"2024-08-17T17:28:27.500801+0000","last_undegraded":"2024-08-17T17:28:27.500801+0000","last_fullsized":"2024-08-17T17:28:27.500801+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:46:20.082241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":43,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.498220+0000","last_change":"2024-08-17T17:24:53.460742+0000","last_active":"2024-08-17T17:28:26.498220+0000","last_peered":"2024-08-17T17:28:26.498220+0000","last_clean":"2024-08-17T17:28:26.498220+0000","last_became_active":"2024-08-17T17:24:53.460365+0000","last_became_peered":"2024-08-17T17:24:53.460365+0000","last_unstale":"2024-08-17T17:28:26.498220+0000","last_undegraded":"2024-08-17T17:28:26.498220+0000","last_fullsized":"2024-08-17T17:28:26.498220+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:24:08.116968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":119,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.502421+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:28:27.502421+0000","last_peered":"2024-08-17T17:28:27.502421+0000","last_clean":"2024-08-17T17:28:27.502421+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:28:27.502421+0000","last_undegraded":"2024-08-17T17:28:27.502421+0000","last_fullsized":"2024-08-17T17:28:27.502421+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":119,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.504154+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:28:27.504154+0000","last_peered":"2024-08-17T17:28:27.504154+0000","last_clean":"2024-08-17T17:28:27.504154+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:28:27.504154+0000","last_undegraded":"2024-08-17T17:28:27.504154+0000","last_fullsized":"2024-08-17T17:28:27.504154+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":179,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.500683+0000","last_change":"2024-08-17T17:20:49.258206+0000","last_active":"2024-08-17T17:28:27.500683+0000","last_peered":"2024-08-17T17:28:27.500683+0000","last_clean":"2024-08-17T17:28:27.500683+0000","last_became_active":"2024-08-17T17:20:49.257761+0000","last_became_peered":"2024-08-17T17:20:49.257761+0000","last_unstale":"2024-08-17T17:28:27.500683+0000","last_undegraded":"2024-08-17T17:28:27.500683+0000","last_fullsized":"2024-08-17T17:28:27.500683+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:38:35.363748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":320,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.497593+0000","last_change":"2024-08-17T17:19:57.514773+0000","last_active":"2024-08-17T17:28:26.497593+0000","last_peered":"2024-08-17T17:28:26.497593+0000","last_clean":"2024-08-17T17:28:26.497593+0000","last_became_active":"2024-08-17T17:19:57.513839+0000","last_became_peered":"2024-08-17T17:19:57.513839+0000","last_unstale":"2024-08-17T17:28:26.497593+0000","last_undegraded":"2024-08-17T17:28:26.497593+0000","last_fullsized":"2024-08-17T17:28:26.497593+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:42:48.063169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":273,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.496878+0000","last_change":"2024-08-17T17:24:53.465875+0000","last_active":"2024-08-17T17:28:26.496878+0000","last_peered":"2024-08-17T17:28:26.496878+0000","last_clean":"2024-08-17T17:28:26.496878+0000","last_became_active":"2024-08-17T17:24:53.458129+0000","last_became_peered":"2024-08-17T17:24:53.458129+0000","last_unstale":"2024-08-17T17:28:26.496878+0000","last_undegraded":"2024-08-17T17:28:26.496878+0000","last_fullsized":"2024-08-17T17:28:26.496878+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:51:33.584869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-08-17T17:28:26.486401+0000","last_change":"2024-08-17T17:28:26.486401+0000","last_active":"2024-08-17T17:28:26.486401+0000","last_peered":"2024-08-17T17:28:26.486401+0000","last_clean":"2024-08-17T17:28:26.486401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T17:28:26.486401+0000","last_undegraded":"2024-08-17T17:28:26.486401+0000","last_fullsized":"2024-08-17T17:28:26.486401+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":140,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.501000+0000","last_change":"2024-08-17T17:24:06.597382+0000","last_active":"2024-08-17T17:28:27.501000+0000","last_peered":"2024-08-17T17:28:27.501000+0000","last_clean":"2024-08-17T17:28:27.501000+0000","last_became_active":"2024-08-17T17:24:06.588881+0000","last_became_peered":"2024-08-17T17:24:06.588881+0000","last_unstale":"2024-08-17T17:28:27.501000+0000","last_undegraded":"2024-08-17T17:28:27.501000+0000","last_fullsized":"2024-08-17T17:28:27.501000+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:27.240751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":53,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.497893+0000","last_change":"2024-08-17T17:25:04.503780+0000","last_active":"2024-08-17T17:28:26.497893+0000","last_peered":"2024-08-17T17:28:26.497893+0000","last_clean":"2024-08-17T17:28:26.497893+0000","last_became_active":"2024-08-17T17:25:04.503503+0000","last_became_peered":"2024-08-17T17:25:04.503503+0000","last_unstale":"2024-08-17T17:28:26.497893+0000","last_undegraded":"2024-08-17T17:28:26.497893+0000","last_fullsized":"2024-08-17T17:28:26.497893+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:04:48.171216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"creating+peering","last_fresh":"2024-08-17T17:28:26.495614+0000","last_change":"2024-08-17T17:28:26.495614+0000","last_active":"2024-08-17T17:28:26.486401+0000","last_peered":"2024-08-17T17:28:26.486401+0000","last_clean":"2024-08-17T17:28:26.486401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T17:28:26.495614+0000","last_undegraded":"2024-08-17T17:28:26.495614+0000","last_fullsized":"2024-08-17T17:28:26.495614+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":126,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.501180+0000","last_change":"2024-08-17T17:24:06.599023+0000","last_active":"2024-08-17T17:28:27.501180+0000","last_peered":"2024-08-17T17:28:27.501180+0000","last_clean":"2024-08-17T17:28:27.501180+0000","last_became_active":"2024-08-17T17:24:06.588954+0000","last_became_peered":"2024-08-17T17:24:06.588954+0000","last_unstale":"2024-08-17T17:28:27.501180+0000","last_undegraded":"2024-08-17T17:28:27.501180+0000","last_fullsized":"2024-08-17T17:28:27.501180+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:20:51.445576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":259,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-17T17:28:19.705947+0000","last_change":"2024-08-17T17:20:49.271540+0000","last_active":"2024-08-17T17:28:19.705947+0000","last_peered":"2024-08-17T17:28:19.705947+0000","last_clean":"2024-08-17T17:28:19.705947+0000","last_became_active":"2024-08-17T17:20:49.271130+0000","last_became_peered":"2024-08-17T17:20:49.271130+0000","last_unstale":"2024-08-17T17:28:19.705947+0000","last_undegraded":"2024-08-17T17:28:19.705947+0000","last_fullsized":"2024-08-17T17:28:19.705947+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:05.386705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.510534+0000","last_change":"2024-08-17T17:28:27.508033+0000","last_active":"2024-08-17T17:28:27.510534+0000","last_peered":"2024-08-17T17:28:27.510534+0000","last_clean":"2024-08-17T17:28:27.510534+0000","last_became_active":"2024-08-17T17:28:27.507657+0000","last_became_peered":"2024-08-17T17:28:27.507657+0000","last_unstale":"2024-08-17T17:28:27.510534+0000","last_undegraded":"2024-08-17T17:28:27.510534+0000","last_fullsized":"2024-08-17T17:28:27.510534+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:32:00.826804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":198,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.501305+0000","last_change":"2024-08-17T17:24:53.467526+0000","last_active":"2024-08-17T17:28:27.501305+0000","last_peered":"2024-08-17T17:28:27.501305+0000","last_clean":"2024-08-17T17:28:27.501305+0000","last_became_active":"2024-08-17T17:24:53.460631+0000","last_became_peered":"2024-08-17T17:24:53.460631+0000","last_unstale":"2024-08-17T17:28:27.501305+0000","last_undegraded":"2024-08-17T17:28:27.501305+0000","last_fullsized":"2024-08-17T17:28:27.501305+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:05:24.938831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":39,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.496976+0000","last_change":"2024-08-17T17:25:04.507176+0000","last_active":"2024-08-17T17:28:26.496976+0000","last_peered":"2024-08-17T17:28:26.496976+0000","last_clean":"2024-08-17T17:28:26.496976+0000","last_became_active":"2024-08-17T17:25:04.506788+0000","last_became_peered":"2024-08-17T17:25:04.506788+0000","last_unstale":"2024-08-17T17:28:26.496976+0000","last_undegraded":"2024-08-17T17:28:26.496976+0000","last_fullsized":"2024-08-17T17:28:26.496976+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:22:04.658216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.510163+0000","last_change":"2024-08-17T17:28:27.509743+0000","last_active":"2024-08-17T17:28:27.510163+0000","last_peered":"2024-08-17T17:28:27.510163+0000","last_clean":"2024-08-17T17:28:27.510163+0000","last_became_active":"2024-08-17T17:28:27.509414+0000","last_became_peered":"2024-08-17T17:28:27.509414+0000","last_unstale":"2024-08-17T17:28:27.510163+0000","last_undegraded":"2024-08-17T17:28:27.510163+0000","last_fullsized":"2024-08-17T17:28:27.510163+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:37:42.377123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":185,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.498181+0000","last_change":"2024-08-17T17:17:32.495428+0000","last_active":"2024-08-17T17:28:26.498181+0000","last_peered":"2024-08-17T17:28:26.498181+0000","last_clean":"2024-08-17T17:28:26.498181+0000","last_became_active":"2024-08-17T17:17:32.494978+0000","last_became_peered":"2024-08-17T17:17:32.494978+0000","last_unstale":"2024-08-17T17:28:26.498181+0000","last_undegraded":"2024-08-17T17:28:26.498181+0000","last_fullsized":"2024-08-17T17:28:26.498181+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:42:24.647812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":311,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.502866+0000","last_change":"2024-08-17T17:20:49.272126+0000","last_active":"2024-08-17T17:28:27.502866+0000","last_peered":"2024-08-17T17:28:27.502866+0000","last_clean":"2024-08-17T17:28:27.502866+0000","last_became_active":"2024-08-17T17:20:49.271185+0000","last_became_peered":"2024-08-17T17:20:49.271185+0000","last_unstale":"2024-08-17T17:28:27.502866+0000","last_undegraded":"2024-08-17T17:28:27.502866+0000","last_fullsized":"2024-08-17T17:28:27.502866+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:43:58.184939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.513194+0000","last_change":"2024-08-17T17:28:27.512879+0000","last_active":"2024-08-17T17:28:27.513194+0000","last_peered":"2024-08-17T17:28:27.513194+0000","last_clean":"2024-08-17T17:28:27.513194+0000","last_became_active":"2024-08-17T17:28:27.512586+0000","last_became_peered":"2024-08-17T17:28:27.512586+0000","last_unstale":"2024-08-17T17:28:27.513194+0000","last_undegraded":"2024-08-17T17:28:27.513194+0000","last_fullsized":"2024-08-17T17:28:27.513194+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:28:55.521766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":337,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.497479+0000","last_change":"2024-08-17T17:24:06.595380+0000","last_active":"2024-08-17T17:28:26.497479+0000","last_peered":"2024-08-17T17:28:26.497479+0000","last_clean":"2024-08-17T17:28:26.497479+0000","last_became_active":"2024-08-17T17:24:06.584747+0000","last_became_peered":"2024-08-17T17:24:06.584747+0000","last_unstale":"2024-08-17T17:28:26.497479+0000","last_undegraded":"2024-08-17T17:28:26.497479+0000","last_fullsized":"2024-08-17T17:28:26.497479+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:02:18.473980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":283,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.497640+0000","last_change":"2024-08-17T17:20:49.280596+0000","last_active":"2024-08-17T17:28:26.497640+0000","last_peered":"2024-08-17T17:28:26.497640+0000","last_clean":"2024-08-17T17:28:26.497640+0000","last_became_active":"2024-08-17T17:20:49.279553+0000","last_became_peered":"2024-08-17T17:20:49.279553+0000","last_unstale":"2024-08-17T17:28:26.497640+0000","last_undegraded":"2024-08-17T17:28:26.497640+0000","last_fullsized":"2024-08-17T17:28:26.497640+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:13:24.124392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-17T17:28:29.014615+0000","last_change":"2024-08-17T17:28:27.506638+0000","last_active":"2024-08-17T17:28:29.014615+0000","last_peered":"2024-08-17T17:28:29.014615+0000","last_clean":"2024-08-17T17:28:29.014615+0000","last_became_active":"2024-08-17T17:28:27.506343+0000","last_became_peered":"2024-08-17T17:28:27.506343+0000","last_unstale":"2024-08-17T17:28:29.014615+0000","last_undegraded":"2024-08-17T17:28:29.014615+0000","last_fullsized":"2024-08-17T17:28:29.014615+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:09:58.041735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":233,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-17T17:28:19.706021+0000","last_change":"2024-08-17T17:20:49.276769+0000","last_active":"2024-08-17T17:28:19.706021+0000","last_peered":"2024-08-17T17:28:19.706021+0000","last_clean":"2024-08-17T17:28:19.706021+0000","last_became_active":"2024-08-17T17:20:49.276498+0000","last_became_peered":"2024-08-17T17:20:49.276498+0000","last_unstale":"2024-08-17T17:28:19.706021+0000","last_undegraded":"2024-08-17T17:28:19.706021+0000","last_fullsized":"2024-08-17T17:28:19.706021+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:39:05.419440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":207,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.501190+0000","last_change":"2024-08-17T17:20:14.455738+0000","last_active":"2024-08-17T17:28:27.501190+0000","last_peered":"2024-08-17T17:28:27.501190+0000","last_clean":"2024-08-17T17:28:27.501190+0000","last_became_active":"2024-08-17T17:20:14.455469+0000","last_became_peered":"2024-08-17T17:20:14.455469+0000","last_unstale":"2024-08-17T17:28:27.501190+0000","last_undegraded":"2024-08-17T17:28:27.501190+0000","last_fullsized":"2024-08-17T17:28:27.501190+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:27:28.424336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.510620+0000","last_change":"2024-08-17T17:28:27.510620+0000","last_active":"2024-08-17T17:28:27.510620+0000","last_peered":"2024-08-17T17:28:27.510620+0000","last_clean":"2024-08-17T17:28:27.510620+0000","last_became_active":"2024-08-17T17:28:27.510351+0000","last_became_peered":"2024-08-17T17:28:27.510351+0000","last_unstale":"2024-08-17T17:28:27.510620+0000","last_undegraded":"2024-08-17T17:28:27.510620+0000","last_fullsized":"2024-08-17T17:28:27.510620+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:14:19.994370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":161,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.500604+0000","last_change":"2024-08-17T17:20:14.431140+0000","last_active":"2024-08-17T17:28:27.500604+0000","last_peered":"2024-08-17T17:28:27.500604+0000","last_clean":"2024-08-17T17:28:27.500604+0000","last_became_active":"2024-08-17T17:20:14.430839+0000","last_became_peered":"2024-08-17T17:20:14.430839+0000","last_unstale":"2024-08-17T17:28:27.500604+0000","last_undegraded":"2024-08-17T17:28:27.500604+0000","last_fullsized":"2024-08-17T17:28:27.500604+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:08:16.303616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":39,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.497323+0000","last_change":"2024-08-17T17:25:04.509325+0000","last_active":"2024-08-17T17:28:26.497323+0000","last_peered":"2024-08-17T17:28:26.497323+0000","last_clean":"2024-08-17T17:28:26.497323+0000","last_became_active":"2024-08-17T17:25:04.508364+0000","last_became_peered":"2024-08-17T17:25:04.508364+0000","last_unstale":"2024-08-17T17:28:26.497323+0000","last_undegraded":"2024-08-17T17:28:26.497323+0000","last_fullsized":"2024-08-17T17:28:26.497323+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:48:56.415763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.512014+0000","last_change":"2024-08-17T17:28:27.512014+0000","last_active":"2024-08-17T17:28:27.512014+0000","last_peered":"2024-08-17T17:28:27.512014+0000","last_clean":"2024-08-17T17:28:27.512014+0000","last_became_active":"2024-08-17T17:28:27.511692+0000","last_became_peered":"2024-08-17T17:28:27.511692+0000","last_unstale":"2024-08-17T17:28:27.512014+0000","last_undegraded":"2024-08-17T17:28:27.512014+0000","last_fullsized":"2024-08-17T17:28:27.512014+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:38:59.458491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":322,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.496519+0000","last_change":"2024-08-17T17:19:57.520745+0000","last_active":"2024-08-17T17:28:26.496519+0000","last_peered":"2024-08-17T17:28:26.496519+0000","last_clean":"2024-08-17T17:28:26.496519+0000","last_became_active":"2024-08-17T17:19:57.520152+0000","last_became_peered":"2024-08-17T17:19:57.520152+0000","last_unstale":"2024-08-17T17:28:26.496519+0000","last_undegraded":"2024-08-17T17:28:26.496519+0000","last_fullsized":"2024-08-17T17:28:26.496519+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:19:39.495644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":295,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-17T17:28:19.706752+0000","last_change":"2024-08-17T17:20:49.274372+0000","last_active":"2024-08-17T17:28:19.706752+0000","last_peered":"2024-08-17T17:28:19.706752+0000","last_clean":"2024-08-17T17:28:19.706752+0000","last_became_active":"2024-08-17T17:20:49.274023+0000","last_became_peered":"2024-08-17T17:20:49.274023+0000","last_unstale":"2024-08-17T17:28:19.706752+0000","last_undegraded":"2024-08-17T17:28:19.706752+0000","last_fullsized":"2024-08-17T17:28:19.706752+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:31:02.529766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.512270+0000","last_change":"2024-08-17T17:28:27.511026+0000","last_active":"2024-08-17T17:28:27.512270+0000","last_peered":"2024-08-17T17:28:27.512270+0000","last_clean":"2024-08-17T17:28:27.512270+0000","last_became_active":"2024-08-17T17:28:27.510738+0000","last_became_peered":"2024-08-17T17:28:27.510738+0000","last_unstale":"2024-08-17T17:28:27.512270+0000","last_undegraded":"2024-08-17T17:28:27.512270+0000","last_fullsized":"2024-08-17T17:28:27.512270+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:40:30.859320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":555,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.500582+0000","last_change":"2024-08-17T17:24:09.599615+0000","last_active":"2024-08-17T17:28:27.500582+0000","last_peered":"2024-08-17T17:28:27.500582+0000","last_clean":"2024-08-17T17:28:27.500582+0000","last_became_active":"2024-08-17T17:24:09.599085+0000","last_became_peered":"2024-08-17T17:24:09.599085+0000","last_unstale":"2024-08-17T17:28:27.500582+0000","last_undegraded":"2024-08-17T17:28:27.500582+0000","last_fullsized":"2024-08-17T17:28:27.500582+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:04:01.141401+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:30:09.569579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":799,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.498092+0000","last_change":"2024-08-17T17:20:00.434774+0000","last_active":"2024-08-17T17:28:26.498092+0000","last_peered":"2024-08-17T17:28:26.498092+0000","last_clean":"2024-08-17T17:28:26.498092+0000","last_became_active":"2024-08-17T17:20:00.434343+0000","last_became_peered":"2024-08-17T17:20:00.434343+0000","last_unstale":"2024-08-17T17:28:26.498092+0000","last_undegraded":"2024-08-17T17:28:26.498092+0000","last_fullsized":"2024-08-17T17:28:26.498092+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:04.995152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":280,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.496733+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:28:26.496733+0000","last_peered":"2024-08-17T17:28:26.496733+0000","last_clean":"2024-08-17T17:28:26.496733+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:28:26.496733+0000","last_undegraded":"2024-08-17T17:28:26.496733+0000","last_fullsized":"2024-08-17T17:28:26.496733+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":39,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.497122+0000","last_change":"2024-08-17T17:25:04.579426+0000","last_active":"2024-08-17T17:28:26.497122+0000","last_peered":"2024-08-17T17:28:26.497122+0000","last_clean":"2024-08-17T17:28:26.497122+0000","last_became_active":"2024-08-17T17:25:04.579012+0000","last_became_peered":"2024-08-17T17:25:04.579012+0000","last_unstale":"2024-08-17T17:28:26.497122+0000","last_undegraded":"2024-08-17T17:28:26.497122+0000","last_fullsized":"2024-08-17T17:28:26.497122+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:56:32.241321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":10,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.507036+0000","last_change":"2024-08-17T17:28:27.506760+0000","last_active":"2024-08-17T17:28:27.507036+0000","last_peered":"2024-08-17T17:28:27.507036+0000","last_clean":"2024-08-17T17:28:27.507036+0000","last_became_active":"2024-08-17T17:28:27.506306+0000","last_became_peered":"2024-08-17T17:28:27.506306+0000","last_unstale":"2024-08-17T17:28:27.507036+0000","last_undegraded":"2024-08-17T17:28:27.507036+0000","last_fullsized":"2024-08-17T17:28:27.507036+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:41:13.520504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":818,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.500746+0000","last_change":"2024-08-17T17:16:26.958415+0000","last_active":"2024-08-17T17:28:27.500746+0000","last_peered":"2024-08-17T17:28:27.500746+0000","last_clean":"2024-08-17T17:28:27.500746+0000","last_became_active":"2024-08-17T17:16:26.957956+0000","last_became_peered":"2024-08-17T17:16:26.957956+0000","last_unstale":"2024-08-17T17:28:27.500746+0000","last_undegraded":"2024-08-17T17:28:27.500746+0000","last_fullsized":"2024-08-17T17:28:27.500746+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:22.478077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":311,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.503062+0000","last_change":"2024-08-17T17:18:59.415331+0000","last_active":"2024-08-17T17:28:27.503062+0000","last_peered":"2024-08-17T17:28:27.503062+0000","last_clean":"2024-08-17T17:28:27.503062+0000","last_became_active":"2024-08-17T17:18:59.415008+0000","last_became_peered":"2024-08-17T17:18:59.415008+0000","last_unstale":"2024-08-17T17:28:27.503062+0000","last_undegraded":"2024-08-17T17:28:27.503062+0000","last_fullsized":"2024-08-17T17:28:27.503062+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:22:38.076866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":120,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.504375+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:28:27.504375+0000","last_peered":"2024-08-17T17:28:27.504375+0000","last_clean":"2024-08-17T17:28:27.504375+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:28:27.504375+0000","last_undegraded":"2024-08-17T17:28:27.504375+0000","last_fullsized":"2024-08-17T17:28:27.504375+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.511251+0000","last_change":"2024-08-17T17:28:27.509856+0000","last_active":"2024-08-17T17:28:27.511251+0000","last_peered":"2024-08-17T17:28:27.511251+0000","last_clean":"2024-08-17T17:28:27.511251+0000","last_became_active":"2024-08-17T17:28:27.509524+0000","last_became_peered":"2024-08-17T17:28:27.509524+0000","last_unstale":"2024-08-17T17:28:27.511251+0000","last_undegraded":"2024-08-17T17:28:27.511251+0000","last_fullsized":"2024-08-17T17:28:27.511251+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:57:02.551772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":696,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.496745+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:28:26.496745+0000","last_peered":"2024-08-17T17:28:26.496745+0000","last_clean":"2024-08-17T17:28:26.496745+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:28:26.496745+0000","last_undegraded":"2024-08-17T17:28:26.496745+0000","last_fullsized":"2024-08-17T17:28:26.496745+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":303,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-17T17:28:19.705937+0000","last_change":"2024-08-17T17:19:57.515564+0000","last_active":"2024-08-17T17:28:19.705937+0000","last_peered":"2024-08-17T17:28:19.705937+0000","last_clean":"2024-08-17T17:28:19.705937+0000","last_became_active":"2024-08-17T17:19:57.515142+0000","last_became_peered":"2024-08-17T17:19:57.515142+0000","last_unstale":"2024-08-17T17:28:19.705937+0000","last_undegraded":"2024-08-17T17:28:19.705937+0000","last_fullsized":"2024-08-17T17:28:19.705937+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:59:33.506176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":333,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-17T17:28:19.706548+0000","last_change":"2024-08-17T17:24:06.587903+0000","last_active":"2024-08-17T17:28:19.706548+0000","last_peered":"2024-08-17T17:28:19.706548+0000","last_clean":"2024-08-17T17:28:19.706548+0000","last_became_active":"2024-08-17T17:24:06.587503+0000","last_became_peered":"2024-08-17T17:24:06.587503+0000","last_unstale":"2024-08-17T17:28:19.706548+0000","last_undegraded":"2024-08-17T17:28:19.706548+0000","last_fullsized":"2024-08-17T17:28:19.706548+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:05:58.591992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.511940+0000","last_change":"2024-08-17T17:28:27.511940+0000","last_active":"2024-08-17T17:28:27.511940+0000","last_peered":"2024-08-17T17:28:27.511940+0000","last_clean":"2024-08-17T17:28:27.511940+0000","last_became_active":"2024-08-17T17:28:27.511636+0000","last_became_peered":"2024-08-17T17:28:27.511636+0000","last_unstale":"2024-08-17T17:28:27.511940+0000","last_undegraded":"2024-08-17T17:28:27.511940+0000","last_fullsized":"2024-08-17T17:28:27.511940+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:51:38.705815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":1137,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.498280+0000","last_change":"2024-08-17T17:20:00.433870+0000","last_active":"2024-08-17T17:28:26.498280+0000","last_peered":"2024-08-17T17:28:26.498280+0000","last_clean":"2024-08-17T17:28:26.498280+0000","last_became_active":"2024-08-17T17:20:00.433386+0000","last_became_peered":"2024-08-17T17:20:00.433386+0000","last_unstale":"2024-08-17T17:28:26.498280+0000","last_undegraded":"2024-08-17T17:28:26.498280+0000","last_fullsized":"2024-08-17T17:28:26.498280+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:01:11.848862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1392,"num_read_kb":2863,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":232,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.502656+0000","last_change":"2024-08-17T17:20:49.272407+0000","last_active":"2024-08-17T17:28:27.502656+0000","last_peered":"2024-08-17T17:28:27.502656+0000","last_clean":"2024-08-17T17:28:27.502656+0000","last_became_active":"2024-08-17T17:20:49.271749+0000","last_became_peered":"2024-08-17T17:20:49.271749+0000","last_unstale":"2024-08-17T17:28:27.502656+0000","last_undegraded":"2024-08-17T17:28:27.502656+0000","last_fullsized":"2024-08-17T17:28:27.502656+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:36:01.443457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":299,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-17T17:28:19.706247+0000","last_change":"2024-08-17T17:17:32.499715+0000","last_active":"2024-08-17T17:28:19.706247+0000","last_peered":"2024-08-17T17:28:19.706247+0000","last_clean":"2024-08-17T17:28:19.706247+0000","last_became_active":"2024-08-17T17:17:32.499412+0000","last_became_peered":"2024-08-17T17:17:32.499412+0000","last_unstale":"2024-08-17T17:28:19.706247+0000","last_undegraded":"2024-08-17T17:28:19.706247+0000","last_fullsized":"2024-08-17T17:28:19.706247+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:39:38.456970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"creating+peering","last_fresh":"2024-08-17T17:28:26.495146+0000","last_change":"2024-08-17T17:28:26.495146+0000","last_active":"2024-08-17T17:28:26.486401+0000","last_peered":"2024-08-17T17:28:26.486401+0000","last_clean":"2024-08-17T17:28:26.486401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T17:28:26.495146+0000","last_undegraded":"2024-08-17T17:28:26.495146+0000","last_fullsized":"2024-08-17T17:28:26.495146+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":202,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.498410+0000","last_change":"2024-08-17T17:17:32.495728+0000","last_active":"2024-08-17T17:28:26.498410+0000","last_peered":"2024-08-17T17:28:26.498410+0000","last_clean":"2024-08-17T17:28:26.498410+0000","last_became_active":"2024-08-17T17:17:32.495357+0000","last_became_peered":"2024-08-17T17:17:32.495357+0000","last_unstale":"2024-08-17T17:28:26.498410+0000","last_undegraded":"2024-08-17T17:28:26.498410+0000","last_fullsized":"2024-08-17T17:28:26.498410+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:27:45.561567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":535,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-17T17:28:19.706392+0000","last_change":"2024-08-17T17:16:26.964621+0000","last_active":"2024-08-17T17:28:19.706392+0000","last_peered":"2024-08-17T17:28:19.706392+0000","last_clean":"2024-08-17T17:28:19.706392+0000","last_became_active":"2024-08-17T17:16:26.964058+0000","last_became_peered":"2024-08-17T17:16:26.964058+0000","last_unstale":"2024-08-17T17:28:19.706392+0000","last_undegraded":"2024-08-17T17:28:19.706392+0000","last_fullsized":"2024-08-17T17:28:19.706392+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:00:04.465644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":305,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-17T17:28:19.706053+0000","last_change":"2024-08-17T17:24:53.465550+0000","last_active":"2024-08-17T17:28:19.706053+0000","last_peered":"2024-08-17T17:28:19.706053+0000","last_clean":"2024-08-17T17:28:19.706053+0000","last_became_active":"2024-08-17T17:24:53.465228+0000","last_became_peered":"2024-08-17T17:24:53.465228+0000","last_unstale":"2024-08-17T17:28:19.706053+0000","last_undegraded":"2024-08-17T17:28:19.706053+0000","last_fullsized":"2024-08-17T17:28:19.706053+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:21:18.378359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.509315+0000","last_change":"2024-08-17T17:28:27.509315+0000","last_active":"2024-08-17T17:28:27.509315+0000","last_peered":"2024-08-17T17:28:27.509315+0000","last_clean":"2024-08-17T17:28:27.509315+0000","last_became_active":"2024-08-17T17:28:27.509018+0000","last_became_peered":"2024-08-17T17:28:27.509018+0000","last_unstale":"2024-08-17T17:28:27.509315+0000","last_undegraded":"2024-08-17T17:28:27.509315+0000","last_fullsized":"2024-08-17T17:28:27.509315+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:26:22.379668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":330,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-17T17:28:19.706316+0000","last_change":"2024-08-17T17:24:53.464307+0000","last_active":"2024-08-17T17:28:19.706316+0000","last_peered":"2024-08-17T17:28:19.706316+0000","last_clean":"2024-08-17T17:28:19.706316+0000","last_became_active":"2024-08-17T17:24:53.463996+0000","last_became_peered":"2024-08-17T17:24:53.463996+0000","last_unstale":"2024-08-17T17:28:19.706316+0000","last_undegraded":"2024-08-17T17:28:19.706316+0000","last_fullsized":"2024-08-17T17:28:19.706316+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:04:18.993725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":655,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.501324+0000","last_change":"2024-08-17T17:24:09.599822+0000","last_active":"2024-08-17T17:28:27.501324+0000","last_peered":"2024-08-17T17:28:27.501324+0000","last_clean":"2024-08-17T17:28:27.501324+0000","last_became_active":"2024-08-17T17:24:09.599189+0000","last_became_peered":"2024-08-17T17:24:09.599189+0000","last_unstale":"2024-08-17T17:28:27.501324+0000","last_undegraded":"2024-08-17T17:28:27.501324+0000","last_fullsized":"2024-08-17T17:28:27.501324+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:04.465604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":294,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.502763+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:28:27.502763+0000","last_peered":"2024-08-17T17:28:27.502763+0000","last_clean":"2024-08-17T17:28:27.502763+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:28:27.502763+0000","last_undegraded":"2024-08-17T17:28:27.502763+0000","last_fullsized":"2024-08-17T17:28:27.502763+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"creating+peering","last_fresh":"2024-08-17T17:28:26.495088+0000","last_change":"2024-08-17T17:28:26.495088+0000","last_active":"2024-08-17T17:28:26.486401+0000","last_peered":"2024-08-17T17:28:26.486401+0000","last_clean":"2024-08-17T17:28:26.486401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T17:28:26.495088+0000","last_undegraded":"2024-08-17T17:28:26.495088+0000","last_fullsized":"2024-08-17T17:28:26.495088+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":327,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.497791+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:28:26.497791+0000","last_peered":"2024-08-17T17:28:26.497791+0000","last_clean":"2024-08-17T17:28:26.497791+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:28:26.497791+0000","last_undegraded":"2024-08-17T17:28:26.497791+0000","last_fullsized":"2024-08-17T17:28:26.497791+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":844,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-17T17:28:19.706509+0000","last_change":"2024-08-17T17:23:19.633786+0000","last_active":"2024-08-17T17:28:19.706509+0000","last_peered":"2024-08-17T17:28:19.706509+0000","last_clean":"2024-08-17T17:28:19.706509+0000","last_became_active":"2024-08-17T17:23:19.633230+0000","last_became_peered":"2024-08-17T17:23:19.633230+0000","last_unstale":"2024-08-17T17:28:19.706509+0000","last_undegraded":"2024-08-17T17:28:19.706509+0000","last_fullsized":"2024-08-17T17:28:19.706509+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:30:50.377391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":323,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.496402+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:28:26.496402+0000","last_peered":"2024-08-17T17:28:26.496402+0000","last_clean":"2024-08-17T17:28:26.496402+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:28:26.496402+0000","last_undegraded":"2024-08-17T17:28:26.496402+0000","last_fullsized":"2024-08-17T17:28:26.496402+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"creating+peering","last_fresh":"2024-08-17T17:28:26.495086+0000","last_change":"2024-08-17T17:28:26.495086+0000","last_active":"2024-08-17T17:28:26.486401+0000","last_peered":"2024-08-17T17:28:26.486401+0000","last_clean":"2024-08-17T17:28:26.486401+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T17:28:26.495086+0000","last_undegraded":"2024-08-17T17:28:26.495086+0000","last_fullsized":"2024-08-17T17:28:26.495086+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":124,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.504287+0000","last_change":"2024-08-17T17:24:06.593660+0000","last_active":"2024-08-17T17:28:27.504287+0000","last_peered":"2024-08-17T17:28:27.504287+0000","last_clean":"2024-08-17T17:28:27.504287+0000","last_became_active":"2024-08-17T17:24:06.586215+0000","last_became_peered":"2024-08-17T17:24:06.586215+0000","last_unstale":"2024-08-17T17:28:27.504287+0000","last_undegraded":"2024-08-17T17:28:27.504287+0000","last_fullsized":"2024-08-17T17:28:27.504287+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:15:49.475395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":458,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.497262+0000","last_change":"2024-08-17T17:26:06.855307+0000","last_active":"2024-08-17T17:28:26.497262+0000","last_peered":"2024-08-17T17:28:26.497262+0000","last_clean":"2024-08-17T17:28:26.497262+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:28:26.497262+0000","last_undegraded":"2024-08-17T17:28:26.497262+0000","last_fullsized":"2024-08-17T17:28:26.497262+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":126,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.497948+0000","last_change":"2024-08-17T17:18:59.417183+0000","last_active":"2024-08-17T17:28:26.497948+0000","last_peered":"2024-08-17T17:28:26.497948+0000","last_clean":"2024-08-17T17:28:26.497948+0000","last_became_active":"2024-08-17T17:18:59.416843+0000","last_became_peered":"2024-08-17T17:18:59.416843+0000","last_unstale":"2024-08-17T17:28:26.497948+0000","last_undegraded":"2024-08-17T17:28:26.497948+0000","last_fullsized":"2024-08-17T17:28:26.497948+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:29:09.336335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":147,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.496942+0000","last_change":"2024-08-17T17:20:49.258048+0000","last_active":"2024-08-17T17:28:26.496942+0000","last_peered":"2024-08-17T17:28:26.496942+0000","last_clean":"2024-08-17T17:28:26.496942+0000","last_became_active":"2024-08-17T17:20:49.257672+0000","last_became_peered":"2024-08-17T17:20:49.257672+0000","last_unstale":"2024-08-17T17:28:26.496942+0000","last_undegraded":"2024-08-17T17:28:26.496942+0000","last_fullsized":"2024-08-17T17:28:26.496942+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:08:11.734316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":187,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.500901+0000","last_change":"2024-08-17T17:18:59.430969+0000","last_active":"2024-08-17T17:28:27.500901+0000","last_peered":"2024-08-17T17:28:27.500901+0000","last_clean":"2024-08-17T17:28:27.500901+0000","last_became_active":"2024-08-17T17:18:59.413475+0000","last_became_peered":"2024-08-17T17:18:59.413475+0000","last_unstale":"2024-08-17T17:28:27.500901+0000","last_undegraded":"2024-08-17T17:28:27.500901+0000","last_fullsized":"2024-08-17T17:28:27.500901+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:36:33.732645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":316,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-17T17:28:19.705945+0000","last_change":"2024-08-17T17:25:04.506413+0000","last_active":"2024-08-17T17:28:19.705945+0000","last_peered":"2024-08-17T17:28:19.705945+0000","last_clean":"2024-08-17T17:28:19.705945+0000","last_became_active":"2024-08-17T17:25:04.506104+0000","last_became_peered":"2024-08-17T17:25:04.506104+0000","last_unstale":"2024-08-17T17:28:19.705945+0000","last_undegraded":"2024-08-17T17:28:19.705945+0000","last_fullsized":"2024-08-17T17:28:19.705945+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:41:53.556223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":130,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.501164+0000","last_change":"2024-08-17T17:20:49.279458+0000","last_active":"2024-08-17T17:28:27.501164+0000","last_peered":"2024-08-17T17:28:27.501164+0000","last_clean":"2024-08-17T17:28:27.501164+0000","last_became_active":"2024-08-17T17:20:49.260790+0000","last_became_peered":"2024-08-17T17:20:49.260790+0000","last_unstale":"2024-08-17T17:28:27.501164+0000","last_undegraded":"2024-08-17T17:28:27.501164+0000","last_fullsized":"2024-08-17T17:28:27.501164+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:58.092945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":43,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.498330+0000","last_change":"2024-08-17T17:24:53.463713+0000","last_active":"2024-08-17T17:28:26.498330+0000","last_peered":"2024-08-17T17:28:26.498330+0000","last_clean":"2024-08-17T17:28:26.498330+0000","last_became_active":"2024-08-17T17:24:53.463419+0000","last_became_peered":"2024-08-17T17:24:53.463419+0000","last_unstale":"2024-08-17T17:28:26.498330+0000","last_undegraded":"2024-08-17T17:28:26.498330+0000","last_fullsized":"2024-08-17T17:28:26.498330+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:34:44.545175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":273,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.496528+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:28:26.496528+0000","last_peered":"2024-08-17T17:28:26.496528+0000","last_clean":"2024-08-17T17:28:26.496528+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:28:26.496528+0000","last_undegraded":"2024-08-17T17:28:26.496528+0000","last_fullsized":"2024-08-17T17:28:26.496528+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":303,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.505560+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:28:27.505560+0000","last_peered":"2024-08-17T17:28:27.505560+0000","last_clean":"2024-08-17T17:28:27.505560+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:28:27.505560+0000","last_undegraded":"2024-08-17T17:28:27.505560+0000","last_fullsized":"2024-08-17T17:28:27.505560+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":320,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.496289+0000","last_change":"2024-08-17T17:25:04.509422+0000","last_active":"2024-08-17T17:28:26.496289+0000","last_peered":"2024-08-17T17:28:26.496289+0000","last_clean":"2024-08-17T17:28:26.496289+0000","last_became_active":"2024-08-17T17:25:04.508742+0000","last_became_peered":"2024-08-17T17:25:04.508742+0000","last_unstale":"2024-08-17T17:28:26.496289+0000","last_undegraded":"2024-08-17T17:28:26.496289+0000","last_fullsized":"2024-08-17T17:28:26.496289+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:18:31.976907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":184,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.497171+0000","last_change":"2024-08-17T17:16:26.963696+0000","last_active":"2024-08-17T17:28:26.497171+0000","last_peered":"2024-08-17T17:28:26.497171+0000","last_clean":"2024-08-17T17:28:26.497171+0000","last_became_active":"2024-08-17T17:16:26.963280+0000","last_became_peered":"2024-08-17T17:16:26.963280+0000","last_unstale":"2024-08-17T17:28:26.497171+0000","last_undegraded":"2024-08-17T17:28:26.497171+0000","last_fullsized":"2024-08-17T17:28:26.497171+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:22:01.474840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":338,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.496481+0000","last_change":"2024-08-17T17:24:06.586886+0000","last_active":"2024-08-17T17:28:26.496481+0000","last_peered":"2024-08-17T17:28:26.496481+0000","last_clean":"2024-08-17T17:28:26.496481+0000","last_became_active":"2024-08-17T17:24:06.586516+0000","last_became_peered":"2024-08-17T17:24:06.586516+0000","last_unstale":"2024-08-17T17:28:26.496481+0000","last_undegraded":"2024-08-17T17:28:26.496481+0000","last_fullsized":"2024-08-17T17:28:26.496481+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:03:30.002726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":120,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.504066+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:28:27.504066+0000","last_peered":"2024-08-17T17:28:27.504066+0000","last_clean":"2024-08-17T17:28:27.504066+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:28:27.504066+0000","last_undegraded":"2024-08-17T17:28:27.504066+0000","last_fullsized":"2024-08-17T17:28:27.504066+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":295,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-17T17:28:19.705701+0000","last_change":"2024-08-17T17:18:59.412463+0000","last_active":"2024-08-17T17:28:19.705701+0000","last_peered":"2024-08-17T17:28:19.705701+0000","last_clean":"2024-08-17T17:28:19.705701+0000","last_became_active":"2024-08-17T17:18:59.412134+0000","last_became_peered":"2024-08-17T17:18:59.412134+0000","last_unstale":"2024-08-17T17:28:19.705701+0000","last_undegraded":"2024-08-17T17:28:19.705701+0000","last_fullsized":"2024-08-17T17:28:19.705701+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:44.463153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":341,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.497313+0000","last_change":"2024-08-17T17:25:04.510854+0000","last_active":"2024-08-17T17:28:26.497313+0000","last_peered":"2024-08-17T17:28:26.497313+0000","last_clean":"2024-08-17T17:28:26.497313+0000","last_became_active":"2024-08-17T17:25:04.510462+0000","last_became_peered":"2024-08-17T17:25:04.510462+0000","last_unstale":"2024-08-17T17:28:26.497313+0000","last_undegraded":"2024-08-17T17:28:26.497313+0000","last_fullsized":"2024-08-17T17:28:26.497313+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:55:54.762086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":236,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-17T17:28:19.706158+0000","last_change":"2024-08-17T17:24:06.591165+0000","last_active":"2024-08-17T17:28:19.706158+0000","last_peered":"2024-08-17T17:28:19.706158+0000","last_clean":"2024-08-17T17:28:19.706158+0000","last_became_active":"2024-08-17T17:24:06.590785+0000","last_became_peered":"2024-08-17T17:24:06.590785+0000","last_unstale":"2024-08-17T17:28:19.706158+0000","last_undegraded":"2024-08-17T17:28:19.706158+0000","last_fullsized":"2024-08-17T17:28:19.706158+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:54:05.775563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":308,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-17T17:28:19.707122+0000","last_change":"2024-08-17T17:24:32.300560+0000","last_active":"2024-08-17T17:28:19.707122+0000","last_peered":"2024-08-17T17:28:19.707122+0000","last_clean":"2024-08-17T17:28:19.707122+0000","last_became_active":"2024-08-17T17:24:32.300228+0000","last_became_peered":"2024-08-17T17:24:32.300228+0000","last_unstale":"2024-08-17T17:28:19.707122+0000","last_undegraded":"2024-08-17T17:28:19.707122+0000","last_fullsized":"2024-08-17T17:28:19.707122+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:18:05.475449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":119,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.502538+0000","last_change":"2024-08-17T17:24:06.586379+0000","last_active":"2024-08-17T17:28:27.502538+0000","last_peered":"2024-08-17T17:28:27.502538+0000","last_clean":"2024-08-17T17:28:27.502538+0000","last_became_active":"2024-08-17T17:24:06.586023+0000","last_became_peered":"2024-08-17T17:24:06.586023+0000","last_unstale":"2024-08-17T17:28:27.502538+0000","last_undegraded":"2024-08-17T17:28:27.502538+0000","last_fullsized":"2024-08-17T17:28:27.502538+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:45.701637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":327,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.497384+0000","last_change":"2024-08-17T17:21:50.442535+0000","last_active":"2024-08-17T17:28:26.497384+0000","last_peered":"2024-08-17T17:28:26.497384+0000","last_clean":"2024-08-17T17:28:26.497384+0000","last_became_active":"2024-08-17T17:21:50.441891+0000","last_became_peered":"2024-08-17T17:21:50.441891+0000","last_unstale":"2024-08-17T17:28:26.497384+0000","last_undegraded":"2024-08-17T17:28:26.497384+0000","last_fullsized":"2024-08-17T17:28:26.497384+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:46:24.025407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":239,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.498145+0000","last_change":"2024-08-17T17:25:04.508045+0000","last_active":"2024-08-17T17:28:26.498145+0000","last_peered":"2024-08-17T17:28:26.498145+0000","last_clean":"2024-08-17T17:28:26.498145+0000","last_became_active":"2024-08-17T17:25:04.505083+0000","last_became_peered":"2024-08-17T17:25:04.505083+0000","last_unstale":"2024-08-17T17:28:26.498145+0000","last_undegraded":"2024-08-17T17:28:26.498145+0000","last_fullsized":"2024-08-17T17:28:26.498145+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:23:42.993544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":313,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.496942+0000","last_change":"2024-08-17T17:18:59.418434+0000","last_active":"2024-08-17T17:28:26.496942+0000","last_peered":"2024-08-17T17:28:26.496942+0000","last_clean":"2024-08-17T17:28:26.496942+0000","last_became_active":"2024-08-17T17:18:59.418043+0000","last_became_peered":"2024-08-17T17:18:59.418043+0000","last_unstale":"2024-08-17T17:28:26.496942+0000","last_undegraded":"2024-08-17T17:28:26.496942+0000","last_fullsized":"2024-08-17T17:28:26.496942+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:25:51.371820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":212,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.498343+0000","last_change":"2024-08-17T17:20:49.291167+0000","last_active":"2024-08-17T17:28:26.498343+0000","last_peered":"2024-08-17T17:28:26.498343+0000","last_clean":"2024-08-17T17:28:26.498343+0000","last_became_active":"2024-08-17T17:20:49.263750+0000","last_became_peered":"2024-08-17T17:20:49.263750+0000","last_unstale":"2024-08-17T17:28:26.498343+0000","last_undegraded":"2024-08-17T17:28:26.498343+0000","last_fullsized":"2024-08-17T17:28:26.498343+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:02.863246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":327,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.496877+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:28:26.496877+0000","last_peered":"2024-08-17T17:28:26.496877+0000","last_clean":"2024-08-17T17:28:26.496877+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:28:26.496877+0000","last_undegraded":"2024-08-17T17:28:26.496877+0000","last_fullsized":"2024-08-17T17:28:26.496877+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":327,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.497792+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:28:26.497792+0000","last_peered":"2024-08-17T17:28:26.497792+0000","last_clean":"2024-08-17T17:28:26.497792+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:28:26.497792+0000","last_undegraded":"2024-08-17T17:28:26.497792+0000","last_fullsized":"2024-08-17T17:28:26.497792+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":39,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-08-17T17:28:26.496741+0000","last_change":"2024-08-17T17:25:04.503834+0000","last_active":"2024-08-17T17:28:26.496741+0000","last_peered":"2024-08-17T17:28:26.496741+0000","last_clean":"2024-08-17T17:28:26.496741+0000","last_became_active":"2024-08-17T17:25:04.502712+0000","last_became_peered":"2024-08-17T17:25:04.502712+0000","last_unstale":"2024-08-17T17:28:26.496741+0000","last_undegraded":"2024-08-17T17:28:26.496741+0000","last_fullsized":"2024-08-17T17:28:26.496741+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:45.389103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":139,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-08-17T17:28:27.505457+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:28:27.505457+0000","last_peered":"2024-08-17T17:28:27.505457+0000","last_clean":"2024-08-17T17:28:27.505457+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:28:27.505457+0000","last_undegraded":"2024-08-17T17:28:27.505457+0000","last_fullsized":"2024-08-17T17:28:27.505457+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":45,"acting":45,"num_store_stats":6},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":6304,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":16702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17971,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234260,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":752272,"kb_used_data":454912,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":92992880,"statfs":{"total":95995035648,"available":95224709120,"internally_reserved":0,"allocated":465829888,"data_stored":464519100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495895,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309052,"kb_used_data":7532,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93436100,"statfs":{"total":95995035648,"available":95678566400,"internally_reserved":0,"allocated":7712768,"data_stored":6413503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757532,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312684,"kb_used_data":2848,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93432468,"statfs":{"total":95995035648,"available":95674847232,"internally_reserved":0,"allocated":2916352,"data_stored":1624830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019170,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435397,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":331664,"kb_used_data":2820,"kb_used_omap":4,"kb_used_meta":328827,"kb_avail":93413488,"statfs":{"total":95995035648,"available":95655411712,"internally_reserved":0,"allocated":2887680,"data_stored":1607516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":336718858},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676598924,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":374212,"kb_used_data":6968,"kb_used_omap":7,"kb_used_meta":367224,"kb_avail":93370940,"statfs":{"total":95995035648,"available":95611842560,"internally_reserved":0,"allocated":7135232,"data_stored":5832831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":376037538},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762450,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":373268,"kb_used_data":2824,"kb_used_omap":7,"kb_used_meta":370424,"kb_avail":93371884,"statfs":{"total":95995035648,"available":95612809216,"internally_reserved":0,"allocated":2891776,"data_stored":1607483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7945,"internal_metadata":379314423},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726925977,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":317876,"kb_used_data":3432,"kb_used_omap":7,"kb_used_meta":314424,"kb_avail":93427276,"statfs":{"total":95995035648,"available":95669530624,"internally_reserved":0,"allocated":3514368,"data_stored":2215580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":321970497},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6241,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:28:32.196 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph status --format=json 2024-08-17T17:28:32.236 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:32.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:32.263 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:32.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:32.364 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:32.387 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:32.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:32.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:32.489 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:32.513 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:32.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:32.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:32.615 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:32.637 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:32.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:32.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:32.739 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:32.761 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:32.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:32.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:32.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:32 smithi003 ceph-mon[100615]: from='client.45752 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:28:32.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:32 smithi003 ceph-mon[103344]: from='client.45752 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:28:32.863 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:32.890 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:32.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:32.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:32.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:32 smithi053 ceph-mon[89963]: from='client.45752 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:28:32.992 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:33.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:33.017 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:33.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:33.118 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:33.145 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:33.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:33.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:33.247 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:33.273 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:33.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:33.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:33.375 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:33.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:33.400 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:33.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:33.501 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:33.524 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:33.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:33.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:33.626 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:33.653 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:33.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:33.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:33.755 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:33.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:33 smithi003 ceph-mon[103344]: pgmap v618: 89 pgs: 4 creating+peering, 85 active+clean; 4.6 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:33.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:33 smithi003 ceph-mon[100615]: pgmap v618: 89 pgs: 4 creating+peering, 85 active+clean; 4.6 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:33.779 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:33.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:33.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:33.867 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:33 smithi053 ceph-mon[89963]: pgmap v618: 89 pgs: 4 creating+peering, 85 active+clean; 4.6 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:33.882 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:33.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:33.912 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:33.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:34.013 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:34.039 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:34.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:34.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:34.100 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:28:34.140 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:34.163 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:34.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:34.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:34.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:34.287 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:34.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:34.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:34.389 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:34.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:34.413 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:34.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:34.472 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:28:34.512 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:28:34.515 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:34.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:34.588 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:34.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:34.689 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:34.716 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:34.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:34.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:34.818 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:34.843 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:34.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:34.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:34.945 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:34.970 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:34.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:34.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:35.072 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:35.097 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:35.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:35.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:35.198 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:35.223 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:35.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:35.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:35.325 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:35.349 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:35.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:35.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:35.451 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:35.476 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:35.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:35.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:35.577 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:35.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:35.603 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:35.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:35.704 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:35.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:35.728 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:35.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:35.767 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:35 smithi003 ceph-mon[103344]: pgmap v619: 89 pgs: 89 active+clean; 4.6 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:35.768 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:35 smithi003 ceph-mon[100615]: pgmap v619: 89 pgs: 89 active+clean; 4.6 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:35.768 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:28:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:28:35] "GET /metrics HTTP/1.1" 200 34148 "" "Prometheus/2.51.0" 2024-08-17T17:28:35.829 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:35.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:35.853 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:35.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:35.954 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:35.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:35 smithi053 ceph-mon[89963]: pgmap v619: 89 pgs: 89 active+clean; 4.6 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:35.978 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:35.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:35.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:36.079 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:36.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:36.102 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:36.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:36.203 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:36.225 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:36.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:36.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:36.327 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:36.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:36.355 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:36.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:36.456 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:36.479 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:36.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:36.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:36.501 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:28:36.521 INFO:teuthology.orchestra.run.smithi003.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-08-17T17:28:36.581 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:36.607 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:36.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:36.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:36.709 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:36.736 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:36.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:36.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:36.743 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:36 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1412750803' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:28:36.743 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:36 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1412750803' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:28:36.837 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:36.863 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:36.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:36.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:36.965 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:36.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:36 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1412750803' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:28:36.992 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:36.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:36.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:37.094 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:37.117 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:37.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:37.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:37.219 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:37.241 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:37.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:37.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:37.343 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:37.365 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:37.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:37.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:37.467 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:37.494 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:37.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:37.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:37.596 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:37.621 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:37.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:37.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:37.642 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:37 smithi003 ceph-mon[103344]: pgmap v620: 89 pgs: 89 active+clean; 4.6 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:37.642 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:37 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1412750803' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T17:28:37.643 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:37 smithi003 ceph-mon[103344]: osdmap e229: 8 total, 8 up, 7 in 2024-08-17T17:28:37.643 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:37 smithi003 ceph-mon[100615]: pgmap v620: 89 pgs: 89 active+clean; 4.6 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:37.643 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:37 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1412750803' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T17:28:37.643 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:37 smithi003 ceph-mon[100615]: osdmap e229: 8 total, 8 up, 7 in 2024-08-17T17:28:37.724 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:37.750 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:37.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:37.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:37.852 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:37.878 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:37.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:37.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:37.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:37 smithi053 ceph-mon[89963]: pgmap v620: 89 pgs: 89 active+clean; 4.6 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:37.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:37 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1412750803' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T17:28:37.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:37 smithi053 ceph-mon[89963]: osdmap e229: 8 total, 8 up, 7 in 2024-08-17T17:28:37.980 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:38.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:38.008 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:38.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:38.109 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:38.134 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:38.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:38.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:38.235 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:38.260 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:38.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:38.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:38.362 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:38.386 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:38.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:38.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:38.467 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:28:38.467 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:28:38.489 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:38.512 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:38.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:38.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:38.614 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:38.636 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:38.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:38.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:38.738 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:38.765 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:38.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:38.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:38.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:38.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:38.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:38.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:28:38.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:38.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:38.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:38.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:38.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:28:38.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:38.867 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:38.894 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:38.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:38.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:38.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:38.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:38.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:38.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:28:38.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:38.996 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:39.018 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:39.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:39.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:39.120 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:39.143 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:39.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:39.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:39.245 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:39.271 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:39.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:39.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:39.373 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:39.396 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:39.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:39.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:39.498 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:39.525 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:39.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:39.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:39.627 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:39.650 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:39.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:39.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:39.724 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:39 smithi003 ceph-mon[103344]: pgmap v622: 89 pgs: 89 active+clean; 4.6 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:39.724 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:39 smithi003 ceph-mon[103344]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:28:39.724 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2, 3, 5]} : dispatch 2024-08-17T17:28:39.724 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:39 smithi003 ceph-mon[100615]: pgmap v622: 89 pgs: 89 active+clean; 4.6 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:39.724 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:39 smithi003 ceph-mon[100615]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:28:39.724 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2, 3, 5]} : dispatch 2024-08-17T17:28:39.752 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:39.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:39.776 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:39.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:39.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:39.899 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:39.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:39.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:39.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:39 smithi053 ceph-mon[89963]: pgmap v622: 89 pgs: 89 active+clean; 4.6 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:39.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:39 smithi053 ceph-mon[89963]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:28:39.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2, 3, 5]} : dispatch 2024-08-17T17:28:40.001 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:40.028 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:40.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:40.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:40.035 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-17T17:28:40.130 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:40.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:40.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:40.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:40.255 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:40.278 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:40.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:40.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:40.379 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:40.406 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:40.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:40.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:40.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:28:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:28:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:28:40.508 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:40.535 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:40.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:40.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:40.636 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:40.658 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:40.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:40.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:40.760 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:40.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:40 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2, 3, 5]}]': finished 2024-08-17T17:28:40.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:40 smithi003 ceph-mon[103344]: osdmap e230: 8 total, 8 up, 7 in 2024-08-17T17:28:40.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:40 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2, 3, 5]}]': finished 2024-08-17T17:28:40.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:40 smithi003 ceph-mon[100615]: osdmap e230: 8 total, 8 up, 7 in 2024-08-17T17:28:40.784 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:40.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:40.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:40.885 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:40.912 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:40.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:40.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:40.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:40 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2, 3, 5]}]': finished 2024-08-17T17:28:40.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:40 smithi053 ceph-mon[89963]: osdmap e230: 8 total, 8 up, 7 in 2024-08-17T17:28:41.014 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:41.037 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:41.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:41.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:41.139 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:41.162 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:41.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:41.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:41.264 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:41.286 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:41.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:41.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:41.388 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:41.411 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:41.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:41.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:41.478 DEBUG:teuthology.orchestra.run.smithi003:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-08-17T17:28:41.481 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-08-17T17:28:41.481 INFO:teuthology.orchestra.run:waiting for 3000 2024-08-17T17:28:41.512 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:41.540 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:41.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:41.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:41.548 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:hints = 1 2024-08-17T17:28:41.548 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-08-17T17:28:41.548 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object prefix: benchmark_data_smithi003_201798 2024-08-17T17:28:41.549 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:28:41.549 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 0 0 0 0 0 0 - 0 2024-08-17T17:28:41.641 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:41.681 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:41.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:41.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:41.698 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:41 smithi003 ceph-mon[103344]: pgmap v624: 89 pgs: 89 active+clean; 4.6 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:41.698 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:41 smithi003 ceph-mon[103344]: osdmap e231: 8 total, 8 up, 7 in 2024-08-17T17:28:41.698 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:41 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:28:41.698 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:41 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:28:41.698 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:41 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:41.699 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:41 smithi003 ceph-mon[100615]: pgmap v624: 89 pgs: 89 active+clean; 4.6 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:41.699 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:41 smithi003 ceph-mon[100615]: osdmap e231: 8 total, 8 up, 7 in 2024-08-17T17:28:41.699 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:41 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:28:41.699 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:41 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:28:41.699 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:41 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:41.783 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:41.824 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:41.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:41.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:41.926 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:41.955 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:41.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:41.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:41.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:41 smithi053 ceph-mon[89963]: pgmap v624: 89 pgs: 89 active+clean; 4.6 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:41.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:41 smithi053 ceph-mon[89963]: osdmap e231: 8 total, 8 up, 7 in 2024-08-17T17:28:41.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:41 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:28:41.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:41 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:28:41.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:41 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:42.056 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:42.083 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:42.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:42.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:42.186 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:42.196 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:28:42.212 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:42.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:42.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:42.260 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:28:42.313 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:42.343 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:42.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:42.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:42.445 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:42.474 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:42.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:42.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:42.548 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 1 16 2935 2919 182.418 182.438 0.00297928 0.00546409 2024-08-17T17:28:42.576 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:42.611 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-08-17T17:28:42.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:42.620 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:42.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:42.721 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:42.748 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:42.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:42.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:42.807 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:28:42.808 INFO:teuthology.orchestra.run.smithi003.stdout:{"fsid":"2b5b662e-5cba-11ef-bccf-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false}},"mutes":[]},"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":864,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":232,"num_osds":8,"num_up_osds":8,"osd_up_since":1723915051,"num_in_osds":7,"osd_in_since":1723915248,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":89}],"num_pgs":89,"num_pools":5,"num_objects":28,"data_bytes":4790221,"bytes_used":2784628736,"bytes_avail":669180620800,"bytes_total":671965249536},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi003.civgvf","status":"up:active","gid":14664}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.3:8443/","prometheus":"http://172.21.15.3:9283/"}},"servicemap":{"epoch":24,"modified":"2024-08-17T17:28:39.953105+0000","services":{"mon":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{"fcbeb9b7-7baf-488b-9fe4-c01aca441f9c":{"message":"Upgrade to 19.0.0-5836-gcd8bdbeb (4m)\n [============................] (remaining: 5m)","progress":0.44999998807907104,"add_to_ceph_s":true}}} 2024-08-17T17:28:42.850 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:42.891 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:42.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:42.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:42.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:42 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2188991555' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:28:42.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:42 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:28:42.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:42 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2188991555' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:28:42.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:42 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:28:42.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:42 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2188991555' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:28:42.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:42 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:28:42.993 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:43.024 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:43.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:43.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:43.126 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:43.165 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:43.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:43.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:43.267 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:43.306 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:43.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:43.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:43.407 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:43.438 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:43.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:43.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:43.540 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:43.548 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 2 16 6003 5987 187.075 191.75 0.00356093 0.0053346 2024-08-17T17:28:43.572 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:43.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:43.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:43.673 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:43.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:43.717 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:43.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:43.818 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:43.858 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:43.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:43.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:43.951 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:43 smithi053 ceph-mon[89963]: pgmap v626: 89 pgs: 89 active+clean; 4.6 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:43.951 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:43 smithi053 ceph-mon[89963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:28:43.951 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:43 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:28:43.951 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:43 smithi053 ceph-mon[89963]: osdmap e232: 8 total, 8 up, 7 in 2024-08-17T17:28:43.951 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:43 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1076582488' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T17:28:43.960 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:43.994 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:43.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:43.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:44.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:43 smithi003 ceph-mon[103344]: pgmap v626: 89 pgs: 89 active+clean; 4.6 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:44.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:43 smithi003 ceph-mon[103344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:28:44.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:43 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:28:44.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:43 smithi003 ceph-mon[103344]: osdmap e232: 8 total, 8 up, 7 in 2024-08-17T17:28:44.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:43 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1076582488' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T17:28:44.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:43 smithi003 ceph-mon[100615]: pgmap v626: 89 pgs: 89 active+clean; 4.6 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail 2024-08-17T17:28:44.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:43 smithi003 ceph-mon[100615]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:28:44.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:43 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:28:44.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:43 smithi003 ceph-mon[100615]: osdmap e232: 8 total, 8 up, 7 in 2024-08-17T17:28:44.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:43 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1076582488' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T17:28:44.096 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:44.138 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:44.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:44.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:44.240 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:44.281 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:44.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:44.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:44.383 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:44.420 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:44.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:44.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:44.523 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:44.548 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 3 16 9038 9022 187.94 189.688 0.00415735 0.00531217 2024-08-17T17:28:44.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:44.556 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:44.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:44.657 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:44.671 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-08-17T17:28:44.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:28:44.685 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:44.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:44.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:44.786 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:44.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:44.826 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:44.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:44.927 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:44.954 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:44.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:44.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:45.057 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:45.101 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:45.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:45.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:45.203 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:45.242 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:45.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:45.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:45.344 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:45.375 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:45.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:45.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:45.476 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:45.506 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:45.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:45.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:45.548 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 4 16 11786 11770 183.889 171.75 0.00503496 0.00542992 2024-08-17T17:28:45.608 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:45.647 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:45.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:45.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:45.750 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:45.791 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:45.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:45.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:45.893 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:45.948 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:45.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:45.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:45.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:45 smithi053 ceph-mon[89963]: pgmap v628: 89 pgs: 89 active+clean; 131 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 21 MiB/s wr, 336 op/s 2024-08-17T17:28:46.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:45 smithi003 ceph-mon[103344]: pgmap v628: 89 pgs: 89 active+clean; 131 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 21 MiB/s wr, 336 op/s 2024-08-17T17:28:46.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:45 smithi003 ceph-mon[100615]: pgmap v628: 89 pgs: 89 active+clean; 131 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 21 MiB/s wr, 336 op/s 2024-08-17T17:28:46.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:28:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:28:45] "GET /metrics HTTP/1.1" 200 36435 "" "Prometheus/2.51.0" 2024-08-17T17:28:46.050 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:46.080 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:46.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:46.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:46.181 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:46.213 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:46.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:46.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:46.314 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:46.344 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:46.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:46.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:46.446 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:46.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:46.495 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:46.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:46.552 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 5 16 14369 14353 179.374 161.438 0.00557074 0.00556716 2024-08-17T17:28:46.596 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:46.626 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:46.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:46.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:46.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:46.766 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:46.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:46.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:46.868 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:46.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:46.901 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:46.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:47.002 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:47.029 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:47.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:47.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:47.131 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:47.168 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:47.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:47.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:47.270 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:47.308 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:47.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:47.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:47.410 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:47.441 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:47.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:47.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:47.543 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:47.549 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 6 15 16905 16890 175.903 158.562 0.00479771 0.00567755 2024-08-17T17:28:47.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:47.573 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:47.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:47.674 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:47.718 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:47.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:47.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:47.820 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:47.857 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:28:47.858 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:47.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:47.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:47.901 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:47 smithi003 ceph-mon[103344]: pgmap v629: 89 pgs: 89 active+clean; 352 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 55 MiB/s wr, 873 op/s 2024-08-17T17:28:47.901 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:47 smithi003 ceph-mon[100615]: pgmap v629: 89 pgs: 89 active+clean; 352 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 55 MiB/s wr, 873 op/s 2024-08-17T17:28:47.961 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:47.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:47 smithi053 ceph-mon[89963]: pgmap v629: 89 pgs: 89 active+clean; 352 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 55 MiB/s wr, 873 op/s 2024-08-17T17:28:47.985 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:47.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:47.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:48.087 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:48.110 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:48.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:48.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:48.212 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:48.242 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:48.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:48.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:48.344 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:48.368 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:48.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:48.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:48.469 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:48.499 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:48.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:48.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:48.549 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 7 16 18677 18661 166.586 110.688 0.00874535 0.0059973 2024-08-17T17:28:48.601 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:48.629 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:48.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:48.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:48.731 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:48.759 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:48.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:48.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:48.861 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:48.903 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:48.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:48.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:49.005 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:49.045 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:49.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:49.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:49.147 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:49.179 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:49.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:49.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:49.281 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:49.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:49.311 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:49.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:49.412 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:49.459 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:49.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:49.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:49.549 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 8 16 20638 20622 161.082 122.562 0.0102183 0.00620008 2024-08-17T17:28:49.562 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:49.590 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:49.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:49.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:49.693 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:49.722 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:49.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:49.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:49.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:49 smithi003 ceph-mon[100615]: pgmap v630: 89 pgs: 89 active+clean; 573 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-08-17T17:28:49.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:49 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1902370292' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:28:49.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:49 smithi003 ceph-mon[103344]: pgmap v630: 89 pgs: 89 active+clean; 573 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-08-17T17:28:49.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:49 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1902370292' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:28:49.824 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:49.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:49.847 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:49.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:49.948 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:49.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:49 smithi053 ceph-mon[89963]: pgmap v630: 89 pgs: 89 active+clean; 573 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-08-17T17:28:49.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:49 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1902370292' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:28:49.975 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:49.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:49.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:50.077 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:50.111 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:50.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:50.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:50.212 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:50.250 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:50.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:50.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:50.278 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:28:50.278 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:28:50.279 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:28:50.279 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:28:50.279 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:28:50.279 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:28:50.279 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:28:50.279 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:28:50.279 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:28:50.279 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:28:50.279 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:28:50.279 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:28:50.279 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:28:50.280 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:28:50.280 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:28:50.280 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:28:50.352 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:50.392 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:50.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:50.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:50.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:28:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:28:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:28:50.494 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:50.532 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:50.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:50.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:50.549 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 9 16 22554 22538 156.489 119.75 0.00605323 0.00638402 2024-08-17T17:28:50.634 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:50.662 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:50.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:50.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:50.764 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:50.794 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:50.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:50.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:50.896 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:50.940 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:50.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:50.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:51.041 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:51.080 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:51.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:51.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:51.182 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:51.185 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:28:51.209 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:51.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:51.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:51.312 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:51.342 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:51.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:51.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:51.443 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:51.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:51.495 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:51.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:51.549 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 10 16 24715 24699 154.345 135.062 0.00606653 0.00647278 2024-08-17T17:28:51.597 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:51.627 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:51.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:51.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:51.722 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:51 smithi003 ceph-mon[100615]: pgmap v631: 89 pgs: 89 active+clean; 1.1 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-08-17T17:28:51.723 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:51 smithi003 ceph-mon[103344]: pgmap v631: 89 pgs: 89 active+clean; 1.1 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-08-17T17:28:51.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:51.792 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:51.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:51.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:51.797 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:28:51.797 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-08-17T17:28:51.797 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-08-17T17:28:51.797 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-08-17T17:28:51.797 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:28:51.803 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:28:51.894 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:51.924 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:51.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:51.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:51.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:51 smithi053 ceph-mon[89963]: pgmap v631: 89 pgs: 89 active+clean; 1.1 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 119 MiB/s wr, 1.90k op/s 2024-08-17T17:28:52.026 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:52.057 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:52.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:52.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:52.159 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:52.187 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:52.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:52.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:52.289 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:52.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:52.327 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:52.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:52.428 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:52.456 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:52.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:52.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:52.550 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 11 16 26766 26750 151.966 128.188 0.00549565 0.00657418 2024-08-17T17:28:52.558 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:52.599 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:52.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:52.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:52.701 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:52.728 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:52.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:52.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:52.830 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:52.871 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:52.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:52.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:52.973 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:53.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:53.012 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:53.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:53.113 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:53.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:53.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:53.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:53.191 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:28:53.257 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:53.285 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:53.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:53.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:53.331 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-17T17:28:53.386 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:53.413 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:53.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:53.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:53.515 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:53.541 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:53.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:53.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:53.552 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 12 16 28672 28656 149.228 119.125 0.00492934 0.00669262 2024-08-17T17:28:53.642 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:53.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:53.692 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:53.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:53.793 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:53.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:53.822 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:53.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:53.923 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:53.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:53 smithi053 ceph-mon[89963]: pgmap v632: 89 pgs: 89 active+clean; 1.1 GiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 115 MiB/s wr, 1.85k op/s 2024-08-17T17:28:53.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:28:53.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:53.981 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:53.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:54.000 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:53 smithi003 ceph-mon[100615]: pgmap v632: 89 pgs: 89 active+clean; 1.1 GiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 115 MiB/s wr, 1.85k op/s 2024-08-17T17:28:54.000 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:54.001 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:28:54.001 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:53 smithi003 ceph-mon[103344]: pgmap v632: 89 pgs: 89 active+clean; 1.1 GiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 115 MiB/s wr, 1.85k op/s 2024-08-17T17:28:54.001 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:28:54.001 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:28:54.082 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:54.110 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:54.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:54.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:54.212 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:54.253 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:54.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:54.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:54.355 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:54.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:54.400 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:54.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:54.501 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:54.539 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:54.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:54.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:54.550 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 13 16 30515 30499 146.609 115.188 0.00294157 0.0068141 2024-08-17T17:28:54.641 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:54.660 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:28:54.691 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:54.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:54.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:54.793 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:54.820 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:54.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:54.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:54.922 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:54.951 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:54.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:54.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:55.053 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:55.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:55.096 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:55.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:55.197 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:55.224 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:55.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:55.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:55.326 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:55.348 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:55.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:55.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:55.450 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:55.478 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:55.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:55.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:55.550 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 14 16 32208 32192 143.695 105.812 0.0216558 0.00695426 2024-08-17T17:28:55.580 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:55.609 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:55.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:55.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:55.711 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:55.732 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:55.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:55.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:55.834 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:55.874 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:55.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:55.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:55.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:55 smithi053 ceph-mon[89963]: pgmap v633: 89 pgs: 89 active+clean; 1.5 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-08-17T17:28:55.976 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:55.999 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:55 smithi003 ceph-mon[100615]: pgmap v633: 89 pgs: 89 active+clean; 1.5 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-08-17T17:28:56.000 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:28:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:28:55] "GET /metrics HTTP/1.1" 200 36495 "" "Prometheus/2.51.0" 2024-08-17T17:28:56.000 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:55 smithi003 ceph-mon[103344]: pgmap v633: 89 pgs: 89 active+clean; 1.5 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-08-17T17:28:56.002 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:56.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:56.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:56.104 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:56.126 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:56.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:56.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:56.228 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:56.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:56.265 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:56.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:56.367 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:56.395 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:56.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:56.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:56.497 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:56.528 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:56.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:56.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:56.550 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 15 16 33634 33618 140.056 89.125 0.00955032 0.00713417 2024-08-17T17:28:56.629 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:56.657 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:56.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:56.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:56.760 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:56.788 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:56.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:56.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:56.890 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:56.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:56.929 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:56.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:57.030 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:57.059 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:57.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:57.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:57.161 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:57.198 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:57.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:57.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:57.300 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:57.327 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:57.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:57.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:57.429 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:57.457 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:57.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:57.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:57.550 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 16 16 35404 35388 138.216 110.625 0.00230178 0.00722804 2024-08-17T17:28:57.559 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:28:57.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:57.583 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:57.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:28:57.684 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:28:57.712 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:57.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:57.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:28:57.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:57 smithi003 ceph-mon[103344]: pgmap v634: 89 pgs: 89 active+clean; 1.7 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-08-17T17:28:57.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:57 smithi003 ceph-mon[100615]: pgmap v634: 89 pgs: 89 active+clean; 1.7 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-08-17T17:28:57.813 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:57.853 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:57.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:57.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:57.954 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:57.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:57 smithi053 ceph-mon[89963]: pgmap v634: 89 pgs: 89 active+clean; 1.7 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-08-17T17:28:57.983 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:57.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:57.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:58.084 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:58.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:58.115 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:58.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:58.216 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:58.256 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:58.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:58.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:58.358 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:58.379 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:58.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:58.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:58.481 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:58.510 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:58.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:58.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:58.550 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 17 15 36953 36938 135.784 96.875 0.00801759 0.00735965 2024-08-17T17:28:58.612 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:58.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:58.651 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:58.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:58.752 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:58.777 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:58.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:58.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:58.878 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:28:58.913 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:58.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:58.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:28:59.015 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:28:59.045 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:59.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:59.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:28:59.107 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:28:59.109 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:28:59.146 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:28:59.185 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:59.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:59.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:28:59.287 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:28:59.316 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:28:59.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:59.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:28:59.418 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:59.443 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:59.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:59.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:59.545 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:59.550 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 18 16 38632 38616 134.066 104.875 0.0103321 0.00745289 2024-08-17T17:28:59.588 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:59.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:59.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:59.690 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:28:59.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:59.721 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:59.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:28:59.823 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:59.861 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:28:59.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:28:59.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:28:59.964 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:28:59.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:59 smithi053 ceph-mon[89963]: from='client.45788 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:28:59.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:28:59 smithi053 ceph-mon[89963]: pgmap v635: 89 pgs: 89 active+clean; 1.9 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-08-17T17:29:00.004 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:00.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:00.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:00.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:59 smithi003 ceph-mon[103344]: from='client.45788 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:29:00.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:28:59 smithi003 ceph-mon[103344]: pgmap v635: 89 pgs: 89 active+clean; 1.9 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-08-17T17:29:00.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:59 smithi003 ceph-mon[100615]: from='client.45788 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:29:00.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:28:59 smithi003 ceph-mon[100615]: pgmap v635: 89 pgs: 89 active+clean; 1.9 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 130 MiB/s wr, 2.08k op/s 2024-08-17T17:29:00.106 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:00.137 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:00.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:00.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:00.239 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:00.278 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:00.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:00.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:00.380 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:00.415 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:00.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:00.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:00.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:29:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:29:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:29:00.517 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:00.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:00.547 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:00.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:00.550 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 19 16 40049 40033 131.671 88.5625 0.00374097 0.00758221 2024-08-17T17:29:00.648 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:00.685 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:00.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:00.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:00.787 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:00.815 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:00.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:00.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:00.917 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:00.939 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:00.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:00.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:01.042 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:01.065 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:01.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:01.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:01.167 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:01.197 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:01.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:01.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:01.299 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:01.339 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:01.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:01.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:01.441 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:01.471 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:01.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:01.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:01.550 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-08-17T17:29:01.550829+0000 min lat: 0.00130565 max lat: 0.219368 avg lat: 0.00775103 2024-08-17T17:29:01.551 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:29:01.551 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 20 16 41280 41264 128.934 76.9375 0.0149286 0.00775103 2024-08-17T17:29:01.572 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:01.594 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:01.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:01.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:01.696 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:01.725 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:01.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:01.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:01.827 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:01.867 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:01.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:01.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:01.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:01 smithi053 ceph-mon[89963]: pgmap v636: 89 pgs: 89 active+clean; 2.2 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-08-17T17:29:01.969 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:02.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:02.008 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:02.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:02.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:01 smithi003 ceph-mon[103344]: pgmap v636: 89 pgs: 89 active+clean; 2.2 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-08-17T17:29:02.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:01 smithi003 ceph-mon[100615]: pgmap v636: 89 pgs: 89 active+clean; 2.2 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 141 MiB/s wr, 2.26k op/s 2024-08-17T17:29:02.109 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:02.152 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:02.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:02.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:02.254 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:02.291 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:02.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:02.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:02.392 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:02.431 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:02.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:02.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:02.532 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:02.552 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 21 16 42745 42729 127.146 91.5625 0.0100918 0.00785849 2024-08-17T17:29:02.574 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:02.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:02.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:02.676 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:02.714 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:02.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:02.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:02.816 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:02.858 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:02.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:02.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:02.960 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:02.999 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:03.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:03.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:03.101 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:03.139 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:03.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:03.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:03.178 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:29:03.241 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:03.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:03.296 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:03.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:03.397 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:03.426 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:03.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:03.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:03.528 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:03.552 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 22 16 44352 44336 125.932 100.438 0.0064761 0.00793477 2024-08-17T17:29:03.557 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:03.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:03.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:03.659 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:03.687 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:03.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:03.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:03.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:03 smithi003 ceph-mon[103344]: pgmap v637: 89 pgs: 89 active+clean; 2.2 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-08-17T17:29:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:03 smithi003 ceph-mon[100615]: pgmap v637: 89 pgs: 89 active+clean; 2.2 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-08-17T17:29:03.789 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:03.826 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:03.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:03.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:03.928 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:03.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:03 smithi053 ceph-mon[89963]: pgmap v637: 89 pgs: 89 active+clean; 2.2 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-08-17T17:29:03.967 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:03.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:03.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:04.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:04.113 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:04.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:04.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:04.214 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:04.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:04.243 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:04.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:04.345 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:04.372 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:04.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:04.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:04.474 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:04.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:04.513 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:04.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:04.552 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 23 16 45977 45961 124.872 101.562 0.00790747 0.00800188 2024-08-17T17:29:04.615 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:04.641 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:04.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:04.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:04.685 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":634,"stamp":"2024-08-17T17:28:55.957712+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1862866893,"num_objects":28380,"num_object_clones":0,"num_object_copies":85140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28380,"num_whiteouts":0,"num_read":6376,"num_read_kb":13833,"num_write":29474,"num_write_kb":1829567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":11937158,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29533,"ondisk_log_size":29533,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":270,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":8050540,"kb_used_data":5723380,"kb_used_omap":33,"kb_used_meta":2327070,"kb_avail":648165524,"statfs":{"total":671965249536,"available":663721496576,"internally_reserved":0,"allocated":5860741120,"data_stored":5851533171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34068,"internal_metadata":2382920428},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,7,1,8,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1725759488,"num_objects":26333,"num_object_clones":0,"num_object_copies":78999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26333,"num_write_kb":1685312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003524"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":318,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.617007+0000","last_change":"2024-08-17T17:17:32.495652+0000","last_active":"2024-08-17T17:28:42.617007+0000","last_peered":"2024-08-17T17:28:42.617007+0000","last_clean":"2024-08-17T17:28:42.617007+0000","last_became_active":"2024-08-17T17:17:32.495218+0000","last_became_peered":"2024-08-17T17:17:32.495218+0000","last_unstale":"2024-08-17T17:28:42.617007+0000","last_undegraded":"2024-08-17T17:28:42.617007+0000","last_fullsized":"2024-08-17T17:28:42.617007+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:11:45.821282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":208,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-08-17T17:28:41.398761+0000","last_change":"2024-08-17T17:18:59.424887+0000","last_active":"2024-08-17T17:28:41.398761+0000","last_peered":"2024-08-17T17:28:41.398761+0000","last_clean":"2024-08-17T17:28:41.398761+0000","last_became_active":"2024-08-17T17:18:59.411539+0000","last_became_peered":"2024-08-17T17:18:59.411539+0000","last_unstale":"2024-08-17T17:28:41.398761+0000","last_undegraded":"2024-08-17T17:28:41.398761+0000","last_fullsized":"2024-08-17T17:28:41.398761+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:46:20.082241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":48,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-08-17T17:28:40.591050+0000","last_change":"2024-08-17T17:24:53.460742+0000","last_active":"2024-08-17T17:28:40.591050+0000","last_peered":"2024-08-17T17:28:40.591050+0000","last_clean":"2024-08-17T17:28:40.591050+0000","last_became_active":"2024-08-17T17:24:53.460365+0000","last_became_peered":"2024-08-17T17:24:53.460365+0000","last_unstale":"2024-08-17T17:28:40.591050+0000","last_undegraded":"2024-08-17T17:28:40.591050+0000","last_fullsized":"2024-08-17T17:28:40.591050+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:24:08.116968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":130,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.612620+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:28:42.612620+0000","last_peered":"2024-08-17T17:28:42.612620+0000","last_clean":"2024-08-17T17:28:42.612620+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:28:42.612620+0000","last_undegraded":"2024-08-17T17:28:42.612620+0000","last_fullsized":"2024-08-17T17:28:42.612620+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":129,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.613720+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:28:42.613720+0000","last_peered":"2024-08-17T17:28:42.613720+0000","last_clean":"2024-08-17T17:28:42.613720+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:28:42.613720+0000","last_undegraded":"2024-08-17T17:28:42.613720+0000","last_fullsized":"2024-08-17T17:28:42.613720+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":183,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-08-17T17:28:41.397714+0000","last_change":"2024-08-17T17:20:49.258206+0000","last_active":"2024-08-17T17:28:41.397714+0000","last_peered":"2024-08-17T17:28:41.397714+0000","last_clean":"2024-08-17T17:28:41.397714+0000","last_became_active":"2024-08-17T17:20:49.257761+0000","last_became_peered":"2024-08-17T17:20:49.257761+0000","last_unstale":"2024-08-17T17:28:41.397714+0000","last_undegraded":"2024-08-17T17:28:41.397714+0000","last_fullsized":"2024-08-17T17:28:41.397714+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:38:35.363748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":332,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.617532+0000","last_change":"2024-08-17T17:19:57.514773+0000","last_active":"2024-08-17T17:28:42.617532+0000","last_peered":"2024-08-17T17:28:42.617532+0000","last_clean":"2024-08-17T17:28:42.617532+0000","last_became_active":"2024-08-17T17:19:57.513839+0000","last_became_peered":"2024-08-17T17:19:57.513839+0000","last_unstale":"2024-08-17T17:28:42.617532+0000","last_undegraded":"2024-08-17T17:28:42.617532+0000","last_fullsized":"2024-08-17T17:28:42.617532+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:42:48.063169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":278,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-08-17T17:28:40.780966+0000","last_change":"2024-08-17T17:24:53.465875+0000","last_active":"2024-08-17T17:28:40.780966+0000","last_peered":"2024-08-17T17:28:40.780966+0000","last_clean":"2024-08-17T17:28:40.780966+0000","last_became_active":"2024-08-17T17:24:53.458129+0000","last_became_peered":"2024-08-17T17:24:53.458129+0000","last_unstale":"2024-08-17T17:28:40.780966+0000","last_undegraded":"2024-08-17T17:28:40.780966+0000","last_fullsized":"2024-08-17T17:28:40.780966+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:51:33.584869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"232'2047","reported_seq":2070,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:55.576080+0000","last_change":"2024-08-17T17:28:27.510969+0000","last_active":"2024-08-17T17:28:55.576080+0000","last_peered":"2024-08-17T17:28:55.576080+0000","last_clean":"2024-08-17T17:28:55.576080+0000","last_became_active":"2024-08-17T17:28:27.510664+0000","last_became_peered":"2024-08-17T17:28:27.510664+0000","last_unstale":"2024-08-17T17:28:55.576080+0000","last_undegraded":"2024-08-17T17:28:55.576080+0000","last_fullsized":"2024-08-17T17:28:55.576080+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":2047,"log_dups_size":0,"ondisk_log_size":2047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:16:23.845426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134152192,"num_objects":2047,"num_object_clones":0,"num_object_copies":6141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2047,"num_write_kb":131008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":144,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-08-17T17:28:41.398866+0000","last_change":"2024-08-17T17:24:06.597382+0000","last_active":"2024-08-17T17:28:41.398866+0000","last_peered":"2024-08-17T17:28:41.398866+0000","last_clean":"2024-08-17T17:28:41.398866+0000","last_became_active":"2024-08-17T17:24:06.588881+0000","last_became_peered":"2024-08-17T17:24:06.588881+0000","last_unstale":"2024-08-17T17:28:41.398866+0000","last_undegraded":"2024-08-17T17:28:41.398866+0000","last_fullsized":"2024-08-17T17:28:41.398866+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:27.240751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":58,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-08-17T17:28:40.590399+0000","last_change":"2024-08-17T17:25:04.503780+0000","last_active":"2024-08-17T17:28:40.590399+0000","last_peered":"2024-08-17T17:28:40.590399+0000","last_clean":"2024-08-17T17:28:40.590399+0000","last_became_active":"2024-08-17T17:25:04.503503+0000","last_became_peered":"2024-08-17T17:25:04.503503+0000","last_unstale":"2024-08-17T17:28:40.590399+0000","last_undegraded":"2024-08-17T17:28:40.590399+0000","last_fullsized":"2024-08-17T17:28:40.590399+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:04:48.171216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"232'1646","reported_seq":1669,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:52.045905+0000","last_change":"2024-08-17T17:28:27.509334+0000","last_active":"2024-08-17T17:28:52.045905+0000","last_peered":"2024-08-17T17:28:52.045905+0000","last_clean":"2024-08-17T17:28:52.045905+0000","last_became_active":"2024-08-17T17:28:27.509026+0000","last_became_peered":"2024-08-17T17:28:27.509026+0000","last_unstale":"2024-08-17T17:28:52.045905+0000","last_undegraded":"2024-08-17T17:28:52.045905+0000","last_fullsized":"2024-08-17T17:28:52.045905+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":1646,"log_dups_size":0,"ondisk_log_size":1646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:47:28.720615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107872256,"num_objects":1646,"num_object_clones":0,"num_object_copies":4938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1646,"num_write_kb":105344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":130,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-08-17T17:28:41.398781+0000","last_change":"2024-08-17T17:24:06.599023+0000","last_active":"2024-08-17T17:28:41.398781+0000","last_peered":"2024-08-17T17:28:41.398781+0000","last_clean":"2024-08-17T17:28:41.398781+0000","last_became_active":"2024-08-17T17:24:06.588954+0000","last_became_peered":"2024-08-17T17:24:06.588954+0000","last_unstale":"2024-08-17T17:28:41.398781+0000","last_undegraded":"2024-08-17T17:28:41.398781+0000","last_fullsized":"2024-08-17T17:28:41.398781+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:20:51.445576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":273,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.620465+0000","last_change":"2024-08-17T17:20:49.271540+0000","last_active":"2024-08-17T17:28:42.620465+0000","last_peered":"2024-08-17T17:28:42.620465+0000","last_clean":"2024-08-17T17:28:42.620465+0000","last_became_active":"2024-08-17T17:20:49.271130+0000","last_became_peered":"2024-08-17T17:20:49.271130+0000","last_unstale":"2024-08-17T17:28:42.620465+0000","last_undegraded":"2024-08-17T17:28:42.620465+0000","last_fullsized":"2024-08-17T17:28:42.620465+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:05.386705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.9","version":"232'1715","reported_seq":1738,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:53.103476+0000","last_change":"2024-08-17T17:28:27.508033+0000","last_active":"2024-08-17T17:28:53.103476+0000","last_peered":"2024-08-17T17:28:53.103476+0000","last_clean":"2024-08-17T17:28:53.103476+0000","last_became_active":"2024-08-17T17:28:27.507657+0000","last_became_peered":"2024-08-17T17:28:27.507657+0000","last_unstale":"2024-08-17T17:28:53.103476+0000","last_undegraded":"2024-08-17T17:28:53.103476+0000","last_fullsized":"2024-08-17T17:28:53.103476+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":1715,"log_dups_size":0,"ondisk_log_size":1715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:09:04.908184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112394240,"num_objects":1715,"num_object_clones":0,"num_object_copies":5145,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1715,"num_write_kb":109760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":202,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-08-17T17:28:41.398479+0000","last_change":"2024-08-17T17:24:53.467526+0000","last_active":"2024-08-17T17:28:41.398479+0000","last_peered":"2024-08-17T17:28:41.398479+0000","last_clean":"2024-08-17T17:28:41.398479+0000","last_became_active":"2024-08-17T17:24:53.460631+0000","last_became_peered":"2024-08-17T17:24:53.460631+0000","last_unstale":"2024-08-17T17:28:41.398479+0000","last_undegraded":"2024-08-17T17:28:41.398479+0000","last_fullsized":"2024-08-17T17:28:41.398479+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:05:24.938831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":44,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-08-17T17:28:40.781171+0000","last_change":"2024-08-17T17:25:04.507176+0000","last_active":"2024-08-17T17:28:40.781171+0000","last_peered":"2024-08-17T17:28:40.781171+0000","last_clean":"2024-08-17T17:28:40.781171+0000","last_became_active":"2024-08-17T17:25:04.506788+0000","last_became_peered":"2024-08-17T17:25:04.506788+0000","last_unstale":"2024-08-17T17:28:40.781171+0000","last_undegraded":"2024-08-17T17:28:40.781171+0000","last_fullsized":"2024-08-17T17:28:40.781171+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:22:04.658216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"232'1920","reported_seq":1934,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:54.052341+0000","last_change":"2024-08-17T17:28:27.509743+0000","last_active":"2024-08-17T17:28:54.052341+0000","last_peered":"2024-08-17T17:28:54.052341+0000","last_clean":"2024-08-17T17:28:54.052341+0000","last_became_active":"2024-08-17T17:28:27.509414+0000","last_became_peered":"2024-08-17T17:28:27.509414+0000","last_unstale":"2024-08-17T17:28:54.052341+0000","last_undegraded":"2024-08-17T17:28:54.052341+0000","last_fullsized":"2024-08-17T17:28:54.052341+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":1920,"log_dups_size":0,"ondisk_log_size":1920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:37:42.377123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":1920,"num_object_clones":0,"num_object_copies":5760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1920,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":190,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-08-17T17:28:40.590830+0000","last_change":"2024-08-17T17:17:32.495428+0000","last_active":"2024-08-17T17:28:40.590830+0000","last_peered":"2024-08-17T17:28:40.590830+0000","last_clean":"2024-08-17T17:28:40.590830+0000","last_became_active":"2024-08-17T17:17:32.494978+0000","last_became_peered":"2024-08-17T17:17:32.494978+0000","last_unstale":"2024-08-17T17:28:40.590830+0000","last_undegraded":"2024-08-17T17:28:40.590830+0000","last_fullsized":"2024-08-17T17:28:40.590830+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:42:24.647812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":321,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.613060+0000","last_change":"2024-08-17T17:20:49.272126+0000","last_active":"2024-08-17T17:28:42.613060+0000","last_peered":"2024-08-17T17:28:42.613060+0000","last_clean":"2024-08-17T17:28:42.613060+0000","last_became_active":"2024-08-17T17:20:49.271185+0000","last_became_peered":"2024-08-17T17:20:49.271185+0000","last_unstale":"2024-08-17T17:28:42.613060+0000","last_undegraded":"2024-08-17T17:28:42.613060+0000","last_fullsized":"2024-08-17T17:28:42.613060+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:43:58.184939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"232'1760","reported_seq":1782,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:53.121352+0000","last_change":"2024-08-17T17:28:27.512879+0000","last_active":"2024-08-17T17:28:53.121352+0000","last_peered":"2024-08-17T17:28:53.121352+0000","last_clean":"2024-08-17T17:28:53.121352+0000","last_became_active":"2024-08-17T17:28:27.512586+0000","last_became_peered":"2024-08-17T17:28:27.512586+0000","last_unstale":"2024-08-17T17:28:53.121352+0000","last_undegraded":"2024-08-17T17:28:53.121352+0000","last_fullsized":"2024-08-17T17:28:53.121352+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":1760,"log_dups_size":0,"ondisk_log_size":1760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:12:27.175368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115343360,"num_objects":1760,"num_object_clones":0,"num_object_copies":5280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1760,"num_write_kb":112640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":350,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.614712+0000","last_change":"2024-08-17T17:24:06.595380+0000","last_active":"2024-08-17T17:28:42.614712+0000","last_peered":"2024-08-17T17:28:42.614712+0000","last_clean":"2024-08-17T17:28:42.614712+0000","last_became_active":"2024-08-17T17:24:06.584747+0000","last_became_peered":"2024-08-17T17:24:06.584747+0000","last_unstale":"2024-08-17T17:28:42.614712+0000","last_undegraded":"2024-08-17T17:28:42.614712+0000","last_fullsized":"2024-08-17T17:28:42.614712+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:02:18.473980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":295,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.617541+0000","last_change":"2024-08-17T17:20:49.280596+0000","last_active":"2024-08-17T17:28:42.617541+0000","last_peered":"2024-08-17T17:28:42.617541+0000","last_clean":"2024-08-17T17:28:42.617541+0000","last_became_active":"2024-08-17T17:20:49.279553+0000","last_became_peered":"2024-08-17T17:20:49.279553+0000","last_unstale":"2024-08-17T17:28:42.617541+0000","last_undegraded":"2024-08-17T17:28:42.617541+0000","last_fullsized":"2024-08-17T17:28:42.617541+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:13:24.124392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"232'1842","reported_seq":1865,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:54.526269+0000","last_change":"2024-08-17T17:28:27.506638+0000","last_active":"2024-08-17T17:28:54.526269+0000","last_peered":"2024-08-17T17:28:54.526269+0000","last_clean":"2024-08-17T17:28:54.526269+0000","last_became_active":"2024-08-17T17:28:27.506343+0000","last_became_peered":"2024-08-17T17:28:27.506343+0000","last_unstale":"2024-08-17T17:28:54.526269+0000","last_undegraded":"2024-08-17T17:28:54.526269+0000","last_fullsized":"2024-08-17T17:28:54.526269+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":1842,"log_dups_size":0,"ondisk_log_size":1842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:57:42.320556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120717312,"num_objects":1842,"num_object_clones":0,"num_object_copies":5526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1842,"num_write_kb":117888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":247,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.625147+0000","last_change":"2024-08-17T17:20:49.276769+0000","last_active":"2024-08-17T17:28:42.625147+0000","last_peered":"2024-08-17T17:28:42.625147+0000","last_clean":"2024-08-17T17:28:42.625147+0000","last_became_active":"2024-08-17T17:20:49.276498+0000","last_became_peered":"2024-08-17T17:20:49.276498+0000","last_unstale":"2024-08-17T17:28:42.625147+0000","last_undegraded":"2024-08-17T17:28:42.625147+0000","last_fullsized":"2024-08-17T17:28:42.625147+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:39:05.419440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":211,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-08-17T17:28:41.398278+0000","last_change":"2024-08-17T17:20:14.455738+0000","last_active":"2024-08-17T17:28:41.398278+0000","last_peered":"2024-08-17T17:28:41.398278+0000","last_clean":"2024-08-17T17:28:41.398278+0000","last_became_active":"2024-08-17T17:20:14.455469+0000","last_became_peered":"2024-08-17T17:20:14.455469+0000","last_unstale":"2024-08-17T17:28:41.398278+0000","last_undegraded":"2024-08-17T17:28:41.398278+0000","last_fullsized":"2024-08-17T17:28:41.398278+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:27:28.424336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.d","version":"232'1888","reported_seq":1902,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:53.951058+0000","last_change":"2024-08-17T17:28:27.510620+0000","last_active":"2024-08-17T17:28:53.951058+0000","last_peered":"2024-08-17T17:28:53.951058+0000","last_clean":"2024-08-17T17:28:53.951058+0000","last_became_active":"2024-08-17T17:28:27.510351+0000","last_became_peered":"2024-08-17T17:28:27.510351+0000","last_unstale":"2024-08-17T17:28:53.951058+0000","last_undegraded":"2024-08-17T17:28:53.951058+0000","last_fullsized":"2024-08-17T17:28:53.951058+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":1888,"log_dups_size":0,"ondisk_log_size":1888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:14:19.994370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123731968,"num_objects":1888,"num_object_clones":0,"num_object_copies":5664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1888,"num_write_kb":120832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":165,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-08-17T17:28:41.398166+0000","last_change":"2024-08-17T17:20:14.431140+0000","last_active":"2024-08-17T17:28:41.398166+0000","last_peered":"2024-08-17T17:28:41.398166+0000","last_clean":"2024-08-17T17:28:41.398166+0000","last_became_active":"2024-08-17T17:20:14.430839+0000","last_became_peered":"2024-08-17T17:20:14.430839+0000","last_unstale":"2024-08-17T17:28:41.398166+0000","last_undegraded":"2024-08-17T17:28:41.398166+0000","last_fullsized":"2024-08-17T17:28:41.398166+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:08:16.303616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":44,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-08-17T17:28:40.781424+0000","last_change":"2024-08-17T17:25:04.509325+0000","last_active":"2024-08-17T17:28:40.781424+0000","last_peered":"2024-08-17T17:28:40.781424+0000","last_clean":"2024-08-17T17:28:40.781424+0000","last_became_active":"2024-08-17T17:25:04.508364+0000","last_became_peered":"2024-08-17T17:25:04.508364+0000","last_unstale":"2024-08-17T17:28:40.781424+0000","last_undegraded":"2024-08-17T17:28:40.781424+0000","last_fullsized":"2024-08-17T17:28:40.781424+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:48:56.415763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"232'1797","reported_seq":1811,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:53.914004+0000","last_change":"2024-08-17T17:28:27.512014+0000","last_active":"2024-08-17T17:28:53.914004+0000","last_peered":"2024-08-17T17:28:53.914004+0000","last_clean":"2024-08-17T17:28:53.914004+0000","last_became_active":"2024-08-17T17:28:27.511692+0000","last_became_peered":"2024-08-17T17:28:27.511692+0000","last_unstale":"2024-08-17T17:28:53.914004+0000","last_undegraded":"2024-08-17T17:28:53.914004+0000","last_fullsized":"2024-08-17T17:28:53.914004+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":1797,"log_dups_size":0,"ondisk_log_size":1797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:38:59.458491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117768192,"num_objects":1797,"num_object_clones":0,"num_object_copies":5391,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1797,"num_write_kb":115008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":334,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.615027+0000","last_change":"2024-08-17T17:19:57.520745+0000","last_active":"2024-08-17T17:28:42.615027+0000","last_peered":"2024-08-17T17:28:42.615027+0000","last_clean":"2024-08-17T17:28:42.615027+0000","last_became_active":"2024-08-17T17:19:57.520152+0000","last_became_peered":"2024-08-17T17:19:57.520152+0000","last_unstale":"2024-08-17T17:28:42.615027+0000","last_undegraded":"2024-08-17T17:28:42.615027+0000","last_fullsized":"2024-08-17T17:28:42.615027+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:19:39.495644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":309,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.621703+0000","last_change":"2024-08-17T17:20:49.274372+0000","last_active":"2024-08-17T17:28:42.621703+0000","last_peered":"2024-08-17T17:28:42.621703+0000","last_clean":"2024-08-17T17:28:42.621703+0000","last_became_active":"2024-08-17T17:20:49.274023+0000","last_became_peered":"2024-08-17T17:20:49.274023+0000","last_unstale":"2024-08-17T17:28:42.621703+0000","last_undegraded":"2024-08-17T17:28:42.621703+0000","last_fullsized":"2024-08-17T17:28:42.621703+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:31:02.529766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"232'1752","reported_seq":1775,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:53.115651+0000","last_change":"2024-08-17T17:28:27.511026+0000","last_active":"2024-08-17T17:28:53.115651+0000","last_peered":"2024-08-17T17:28:53.115651+0000","last_clean":"2024-08-17T17:28:53.115651+0000","last_became_active":"2024-08-17T17:28:27.510738+0000","last_became_peered":"2024-08-17T17:28:27.510738+0000","last_unstale":"2024-08-17T17:28:53.115651+0000","last_undegraded":"2024-08-17T17:28:53.115651+0000","last_fullsized":"2024-08-17T17:28:53.115651+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":1752,"log_dups_size":0,"ondisk_log_size":1752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:58:33.553008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114819072,"num_objects":1752,"num_object_clones":0,"num_object_copies":5256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1752,"num_write_kb":112128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":559,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-08-17T17:28:41.397918+0000","last_change":"2024-08-17T17:24:09.599615+0000","last_active":"2024-08-17T17:28:41.397918+0000","last_peered":"2024-08-17T17:28:41.397918+0000","last_clean":"2024-08-17T17:28:41.397918+0000","last_became_active":"2024-08-17T17:24:09.599085+0000","last_became_peered":"2024-08-17T17:24:09.599085+0000","last_unstale":"2024-08-17T17:28:41.397918+0000","last_undegraded":"2024-08-17T17:28:41.397918+0000","last_fullsized":"2024-08-17T17:28:41.397918+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:04:01.141401+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:30:09.569579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":804,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-08-17T17:28:40.590740+0000","last_change":"2024-08-17T17:20:00.434774+0000","last_active":"2024-08-17T17:28:40.590740+0000","last_peered":"2024-08-17T17:28:40.590740+0000","last_clean":"2024-08-17T17:28:40.590740+0000","last_became_active":"2024-08-17T17:20:00.434343+0000","last_became_peered":"2024-08-17T17:20:00.434343+0000","last_unstale":"2024-08-17T17:28:40.590740+0000","last_undegraded":"2024-08-17T17:28:40.590740+0000","last_fullsized":"2024-08-17T17:28:40.590740+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:04.995152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":292,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.614471+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:28:42.614471+0000","last_peered":"2024-08-17T17:28:42.614471+0000","last_clean":"2024-08-17T17:28:42.614471+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:28:42.614471+0000","last_undegraded":"2024-08-17T17:28:42.614471+0000","last_fullsized":"2024-08-17T17:28:42.614471+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":44,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-08-17T17:28:40.780771+0000","last_change":"2024-08-17T17:25:04.579426+0000","last_active":"2024-08-17T17:28:40.780771+0000","last_peered":"2024-08-17T17:28:40.780771+0000","last_clean":"2024-08-17T17:28:40.780771+0000","last_became_active":"2024-08-17T17:25:04.579012+0000","last_became_peered":"2024-08-17T17:25:04.579012+0000","last_unstale":"2024-08-17T17:28:40.780771+0000","last_undegraded":"2024-08-17T17:28:40.780771+0000","last_fullsized":"2024-08-17T17:28:40.780771+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:56:32.241321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"232'1795","reported_seq":1809,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:54.037057+0000","last_change":"2024-08-17T17:28:27.506760+0000","last_active":"2024-08-17T17:28:54.037057+0000","last_peered":"2024-08-17T17:28:54.037057+0000","last_clean":"2024-08-17T17:28:54.037057+0000","last_became_active":"2024-08-17T17:28:27.506306+0000","last_became_peered":"2024-08-17T17:28:27.506306+0000","last_unstale":"2024-08-17T17:28:54.037057+0000","last_undegraded":"2024-08-17T17:28:54.037057+0000","last_fullsized":"2024-08-17T17:28:54.037057+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":1795,"log_dups_size":0,"ondisk_log_size":1795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:41:13.520504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117637120,"num_objects":1795,"num_object_clones":0,"num_object_copies":5385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1795,"num_write_kb":114880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":822,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-08-17T17:28:41.398200+0000","last_change":"2024-08-17T17:16:26.958415+0000","last_active":"2024-08-17T17:28:41.398200+0000","last_peered":"2024-08-17T17:28:41.398200+0000","last_clean":"2024-08-17T17:28:41.398200+0000","last_became_active":"2024-08-17T17:16:26.957956+0000","last_became_peered":"2024-08-17T17:16:26.957956+0000","last_unstale":"2024-08-17T17:28:41.398200+0000","last_undegraded":"2024-08-17T17:28:41.398200+0000","last_fullsized":"2024-08-17T17:28:41.398200+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:22.478077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":321,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.613275+0000","last_change":"2024-08-17T17:18:59.415331+0000","last_active":"2024-08-17T17:28:42.613275+0000","last_peered":"2024-08-17T17:28:42.613275+0000","last_clean":"2024-08-17T17:28:42.613275+0000","last_became_active":"2024-08-17T17:18:59.415008+0000","last_became_peered":"2024-08-17T17:18:59.415008+0000","last_unstale":"2024-08-17T17:28:42.613275+0000","last_undegraded":"2024-08-17T17:28:42.613275+0000","last_fullsized":"2024-08-17T17:28:42.613275+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:22:38.076866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":130,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.613906+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:28:42.613906+0000","last_peered":"2024-08-17T17:28:42.613906+0000","last_clean":"2024-08-17T17:28:42.613906+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:28:42.613906+0000","last_undegraded":"2024-08-17T17:28:42.613906+0000","last_fullsized":"2024-08-17T17:28:42.613906+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"232'1742","reported_seq":1765,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:53.116998+0000","last_change":"2024-08-17T17:28:27.509856+0000","last_active":"2024-08-17T17:28:53.116998+0000","last_peered":"2024-08-17T17:28:53.116998+0000","last_clean":"2024-08-17T17:28:53.116998+0000","last_became_active":"2024-08-17T17:28:27.509524+0000","last_became_peered":"2024-08-17T17:28:27.509524+0000","last_unstale":"2024-08-17T17:28:53.116998+0000","last_undegraded":"2024-08-17T17:28:53.116998+0000","last_fullsized":"2024-08-17T17:28:53.116998+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":1742,"log_dups_size":0,"ondisk_log_size":1742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:02:26.063259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114163712,"num_objects":1742,"num_object_clones":0,"num_object_copies":5226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1742,"num_write_kb":111488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":709,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.614389+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:28:42.614389+0000","last_peered":"2024-08-17T17:28:42.614389+0000","last_clean":"2024-08-17T17:28:42.614389+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:28:42.614389+0000","last_undegraded":"2024-08-17T17:28:42.614389+0000","last_fullsized":"2024-08-17T17:28:42.614389+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":317,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.619904+0000","last_change":"2024-08-17T17:19:57.515564+0000","last_active":"2024-08-17T17:28:42.619904+0000","last_peered":"2024-08-17T17:28:42.619904+0000","last_clean":"2024-08-17T17:28:42.619904+0000","last_became_active":"2024-08-17T17:19:57.515142+0000","last_became_peered":"2024-08-17T17:19:57.515142+0000","last_unstale":"2024-08-17T17:28:42.619904+0000","last_undegraded":"2024-08-17T17:28:42.619904+0000","last_fullsized":"2024-08-17T17:28:42.619904+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:59:33.506176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":347,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.621528+0000","last_change":"2024-08-17T17:24:06.587903+0000","last_active":"2024-08-17T17:28:42.621528+0000","last_peered":"2024-08-17T17:28:42.621528+0000","last_clean":"2024-08-17T17:28:42.621528+0000","last_became_active":"2024-08-17T17:24:06.587503+0000","last_became_peered":"2024-08-17T17:24:06.587503+0000","last_unstale":"2024-08-17T17:28:42.621528+0000","last_undegraded":"2024-08-17T17:28:42.621528+0000","last_fullsized":"2024-08-17T17:28:42.621528+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:05:58.591992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"232'1802","reported_seq":1816,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:53.951769+0000","last_change":"2024-08-17T17:28:27.511940+0000","last_active":"2024-08-17T17:28:53.951769+0000","last_peered":"2024-08-17T17:28:53.951769+0000","last_clean":"2024-08-17T17:28:53.951769+0000","last_became_active":"2024-08-17T17:28:27.511636+0000","last_became_peered":"2024-08-17T17:28:27.511636+0000","last_unstale":"2024-08-17T17:28:53.951769+0000","last_undegraded":"2024-08-17T17:28:53.951769+0000","last_fullsized":"2024-08-17T17:28:53.951769+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":1802,"log_dups_size":0,"ondisk_log_size":1802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:51:38.705815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118095872,"num_objects":1802,"num_object_clones":0,"num_object_copies":5406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1802,"num_write_kb":115328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":1142,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-08-17T17:28:40.590569+0000","last_change":"2024-08-17T17:20:00.433870+0000","last_active":"2024-08-17T17:28:40.590569+0000","last_peered":"2024-08-17T17:28:40.590569+0000","last_clean":"2024-08-17T17:28:40.590569+0000","last_became_active":"2024-08-17T17:20:00.433386+0000","last_became_peered":"2024-08-17T17:20:00.433386+0000","last_unstale":"2024-08-17T17:28:40.590569+0000","last_undegraded":"2024-08-17T17:28:40.590569+0000","last_fullsized":"2024-08-17T17:28:40.590569+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:01:11.848862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1392,"num_read_kb":2863,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":242,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.612900+0000","last_change":"2024-08-17T17:20:49.272407+0000","last_active":"2024-08-17T17:28:42.612900+0000","last_peered":"2024-08-17T17:28:42.612900+0000","last_clean":"2024-08-17T17:28:42.612900+0000","last_became_active":"2024-08-17T17:20:49.271749+0000","last_became_peered":"2024-08-17T17:20:49.271749+0000","last_unstale":"2024-08-17T17:28:42.612900+0000","last_undegraded":"2024-08-17T17:28:42.612900+0000","last_fullsized":"2024-08-17T17:28:42.612900+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:36:01.443457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":314,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.620857+0000","last_change":"2024-08-17T17:17:32.499715+0000","last_active":"2024-08-17T17:28:42.620857+0000","last_peered":"2024-08-17T17:28:42.620857+0000","last_clean":"2024-08-17T17:28:42.620857+0000","last_became_active":"2024-08-17T17:17:32.499412+0000","last_became_peered":"2024-08-17T17:17:32.499412+0000","last_unstale":"2024-08-17T17:28:42.620857+0000","last_undegraded":"2024-08-17T17:28:42.620857+0000","last_fullsized":"2024-08-17T17:28:42.620857+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:39:38.456970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"232'1601","reported_seq":1624,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:52.047084+0000","last_change":"2024-08-17T17:28:27.510443+0000","last_active":"2024-08-17T17:28:52.047084+0000","last_peered":"2024-08-17T17:28:52.047084+0000","last_clean":"2024-08-17T17:28:52.047084+0000","last_became_active":"2024-08-17T17:28:27.510074+0000","last_became_peered":"2024-08-17T17:28:27.510074+0000","last_unstale":"2024-08-17T17:28:52.047084+0000","last_undegraded":"2024-08-17T17:28:52.047084+0000","last_fullsized":"2024-08-17T17:28:52.047084+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":1601,"log_dups_size":0,"ondisk_log_size":1601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:09:30.294320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104923136,"num_objects":1601,"num_object_clones":0,"num_object_copies":4803,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1601,"num_write_kb":102464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":207,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-08-17T17:28:40.591140+0000","last_change":"2024-08-17T17:17:32.495728+0000","last_active":"2024-08-17T17:28:40.591140+0000","last_peered":"2024-08-17T17:28:40.591140+0000","last_clean":"2024-08-17T17:28:40.591140+0000","last_became_active":"2024-08-17T17:17:32.495357+0000","last_became_peered":"2024-08-17T17:17:32.495357+0000","last_unstale":"2024-08-17T17:28:40.591140+0000","last_undegraded":"2024-08-17T17:28:40.591140+0000","last_fullsized":"2024-08-17T17:28:40.591140+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:27:45.561567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":550,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.621074+0000","last_change":"2024-08-17T17:16:26.964621+0000","last_active":"2024-08-17T17:28:42.621074+0000","last_peered":"2024-08-17T17:28:42.621074+0000","last_clean":"2024-08-17T17:28:42.621074+0000","last_became_active":"2024-08-17T17:16:26.964058+0000","last_became_peered":"2024-08-17T17:16:26.964058+0000","last_unstale":"2024-08-17T17:28:42.621074+0000","last_undegraded":"2024-08-17T17:28:42.621074+0000","last_fullsized":"2024-08-17T17:28:42.621074+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:00:04.465644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":320,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.621330+0000","last_change":"2024-08-17T17:24:53.465550+0000","last_active":"2024-08-17T17:28:42.621330+0000","last_peered":"2024-08-17T17:28:42.621330+0000","last_clean":"2024-08-17T17:28:42.621330+0000","last_became_active":"2024-08-17T17:24:53.465228+0000","last_became_peered":"2024-08-17T17:24:53.465228+0000","last_unstale":"2024-08-17T17:28:42.621330+0000","last_undegraded":"2024-08-17T17:28:42.621330+0000","last_fullsized":"2024-08-17T17:28:42.621330+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:21:18.378359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"232'1830","reported_seq":1844,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:53.873731+0000","last_change":"2024-08-17T17:28:27.509315+0000","last_active":"2024-08-17T17:28:53.873731+0000","last_peered":"2024-08-17T17:28:53.873731+0000","last_clean":"2024-08-17T17:28:53.873731+0000","last_became_active":"2024-08-17T17:28:27.509018+0000","last_became_peered":"2024-08-17T17:28:27.509018+0000","last_unstale":"2024-08-17T17:28:53.873731+0000","last_undegraded":"2024-08-17T17:28:53.873731+0000","last_fullsized":"2024-08-17T17:28:53.873731+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":1830,"log_dups_size":0,"ondisk_log_size":1830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:26:22.379668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119930880,"num_objects":1830,"num_object_clones":0,"num_object_copies":5490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1830,"num_write_kb":117120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":345,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.620965+0000","last_change":"2024-08-17T17:24:53.464307+0000","last_active":"2024-08-17T17:28:42.620965+0000","last_peered":"2024-08-17T17:28:42.620965+0000","last_clean":"2024-08-17T17:28:42.620965+0000","last_became_active":"2024-08-17T17:24:53.463996+0000","last_became_peered":"2024-08-17T17:24:53.463996+0000","last_unstale":"2024-08-17T17:28:42.620965+0000","last_undegraded":"2024-08-17T17:28:42.620965+0000","last_fullsized":"2024-08-17T17:28:42.620965+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:04:18.993725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":659,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-08-17T17:28:41.398573+0000","last_change":"2024-08-17T17:24:09.599822+0000","last_active":"2024-08-17T17:28:41.398573+0000","last_peered":"2024-08-17T17:28:41.398573+0000","last_clean":"2024-08-17T17:28:41.398573+0000","last_became_active":"2024-08-17T17:24:09.599189+0000","last_became_peered":"2024-08-17T17:24:09.599189+0000","last_unstale":"2024-08-17T17:28:41.398573+0000","last_undegraded":"2024-08-17T17:28:41.398573+0000","last_fullsized":"2024-08-17T17:28:41.398573+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:04.465604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":304,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.612981+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:28:42.612981+0000","last_peered":"2024-08-17T17:28:42.612981+0000","last_clean":"2024-08-17T17:28:42.612981+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:28:42.612981+0000","last_undegraded":"2024-08-17T17:28:42.612981+0000","last_fullsized":"2024-08-17T17:28:42.612981+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"232'1601","reported_seq":1624,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:52.047409+0000","last_change":"2024-08-17T17:28:27.509838+0000","last_active":"2024-08-17T17:28:52.047409+0000","last_peered":"2024-08-17T17:28:52.047409+0000","last_clean":"2024-08-17T17:28:52.047409+0000","last_became_active":"2024-08-17T17:28:27.509400+0000","last_became_peered":"2024-08-17T17:28:27.509400+0000","last_unstale":"2024-08-17T17:28:52.047409+0000","last_undegraded":"2024-08-17T17:28:52.047409+0000","last_fullsized":"2024-08-17T17:28:52.047409+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":1601,"log_dups_size":0,"ondisk_log_size":1601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:57:28.933136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104923136,"num_objects":1601,"num_object_clones":0,"num_object_copies":4803,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1601,"num_write_kb":102464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":339,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.617750+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:28:42.617750+0000","last_peered":"2024-08-17T17:28:42.617750+0000","last_clean":"2024-08-17T17:28:42.617750+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:28:42.617750+0000","last_undegraded":"2024-08-17T17:28:42.617750+0000","last_fullsized":"2024-08-17T17:28:42.617750+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":858,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.621338+0000","last_change":"2024-08-17T17:23:19.633786+0000","last_active":"2024-08-17T17:28:42.621338+0000","last_peered":"2024-08-17T17:28:42.621338+0000","last_clean":"2024-08-17T17:28:42.621338+0000","last_became_active":"2024-08-17T17:23:19.633230+0000","last_became_peered":"2024-08-17T17:23:19.633230+0000","last_unstale":"2024-08-17T17:28:42.621338+0000","last_undegraded":"2024-08-17T17:28:42.621338+0000","last_fullsized":"2024-08-17T17:28:42.621338+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:30:50.377391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":335,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.614853+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:28:42.614853+0000","last_peered":"2024-08-17T17:28:42.614853+0000","last_clean":"2024-08-17T17:28:42.614853+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:28:42.614853+0000","last_undegraded":"2024-08-17T17:28:42.614853+0000","last_fullsized":"2024-08-17T17:28:42.614853+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"232'1614","reported_seq":1637,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:52.055875+0000","last_change":"2024-08-17T17:28:27.509972+0000","last_active":"2024-08-17T17:28:52.055875+0000","last_peered":"2024-08-17T17:28:52.055875+0000","last_clean":"2024-08-17T17:28:52.055875+0000","last_became_active":"2024-08-17T17:28:27.509668+0000","last_became_peered":"2024-08-17T17:28:27.509668+0000","last_unstale":"2024-08-17T17:28:52.055875+0000","last_undegraded":"2024-08-17T17:28:52.055875+0000","last_fullsized":"2024-08-17T17:28:52.055875+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":1614,"log_dups_size":0,"ondisk_log_size":1614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:54:43.175100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105775104,"num_objects":1614,"num_object_clones":0,"num_object_copies":4842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1614,"num_write_kb":103296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":134,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.613803+0000","last_change":"2024-08-17T17:24:06.593660+0000","last_active":"2024-08-17T17:28:42.613803+0000","last_peered":"2024-08-17T17:28:42.613803+0000","last_clean":"2024-08-17T17:28:42.613803+0000","last_became_active":"2024-08-17T17:24:06.586215+0000","last_became_peered":"2024-08-17T17:24:06.586215+0000","last_unstale":"2024-08-17T17:28:42.613803+0000","last_undegraded":"2024-08-17T17:28:42.613803+0000","last_fullsized":"2024-08-17T17:28:42.613803+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:15:49.475395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":463,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-08-17T17:28:40.781722+0000","last_change":"2024-08-17T17:26:06.855307+0000","last_active":"2024-08-17T17:28:40.781722+0000","last_peered":"2024-08-17T17:28:40.781722+0000","last_clean":"2024-08-17T17:28:40.781722+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:28:40.781722+0000","last_undegraded":"2024-08-17T17:28:40.781722+0000","last_fullsized":"2024-08-17T17:28:40.781722+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":131,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-08-17T17:28:40.590449+0000","last_change":"2024-08-17T17:18:59.417183+0000","last_active":"2024-08-17T17:28:40.590449+0000","last_peered":"2024-08-17T17:28:40.590449+0000","last_clean":"2024-08-17T17:28:40.590449+0000","last_became_active":"2024-08-17T17:18:59.416843+0000","last_became_peered":"2024-08-17T17:18:59.416843+0000","last_unstale":"2024-08-17T17:28:40.590449+0000","last_undegraded":"2024-08-17T17:28:40.590449+0000","last_fullsized":"2024-08-17T17:28:40.590449+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:29:09.336335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":152,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-08-17T17:28:40.781546+0000","last_change":"2024-08-17T17:20:49.258048+0000","last_active":"2024-08-17T17:28:40.781546+0000","last_peered":"2024-08-17T17:28:40.781546+0000","last_clean":"2024-08-17T17:28:40.781546+0000","last_became_active":"2024-08-17T17:20:49.257672+0000","last_became_peered":"2024-08-17T17:20:49.257672+0000","last_unstale":"2024-08-17T17:28:40.781546+0000","last_undegraded":"2024-08-17T17:28:40.781546+0000","last_fullsized":"2024-08-17T17:28:40.781546+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:08:11.734316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":191,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-08-17T17:28:41.398509+0000","last_change":"2024-08-17T17:18:59.430969+0000","last_active":"2024-08-17T17:28:41.398509+0000","last_peered":"2024-08-17T17:28:41.398509+0000","last_clean":"2024-08-17T17:28:41.398509+0000","last_became_active":"2024-08-17T17:18:59.413475+0000","last_became_peered":"2024-08-17T17:18:59.413475+0000","last_unstale":"2024-08-17T17:28:41.398509+0000","last_undegraded":"2024-08-17T17:28:41.398509+0000","last_fullsized":"2024-08-17T17:28:41.398509+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:36:33.732645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":330,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.620649+0000","last_change":"2024-08-17T17:25:04.506413+0000","last_active":"2024-08-17T17:28:42.620649+0000","last_peered":"2024-08-17T17:28:42.620649+0000","last_clean":"2024-08-17T17:28:42.620649+0000","last_became_active":"2024-08-17T17:25:04.506104+0000","last_became_peered":"2024-08-17T17:25:04.506104+0000","last_unstale":"2024-08-17T17:28:42.620649+0000","last_undegraded":"2024-08-17T17:28:42.620649+0000","last_fullsized":"2024-08-17T17:28:42.620649+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:41:53.556223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":134,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-08-17T17:28:41.397739+0000","last_change":"2024-08-17T17:20:49.279458+0000","last_active":"2024-08-17T17:28:41.397739+0000","last_peered":"2024-08-17T17:28:41.397739+0000","last_clean":"2024-08-17T17:28:41.397739+0000","last_became_active":"2024-08-17T17:20:49.260790+0000","last_became_peered":"2024-08-17T17:20:49.260790+0000","last_unstale":"2024-08-17T17:28:41.397739+0000","last_undegraded":"2024-08-17T17:28:41.397739+0000","last_fullsized":"2024-08-17T17:28:41.397739+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:58.092945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":48,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-08-17T17:28:40.590953+0000","last_change":"2024-08-17T17:24:53.463713+0000","last_active":"2024-08-17T17:28:40.590953+0000","last_peered":"2024-08-17T17:28:40.590953+0000","last_clean":"2024-08-17T17:28:40.590953+0000","last_became_active":"2024-08-17T17:24:53.463419+0000","last_became_peered":"2024-08-17T17:24:53.463419+0000","last_unstale":"2024-08-17T17:28:40.590953+0000","last_undegraded":"2024-08-17T17:28:40.590953+0000","last_fullsized":"2024-08-17T17:28:40.590953+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:34:44.545175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":285,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.614101+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:28:42.614101+0000","last_peered":"2024-08-17T17:28:42.614101+0000","last_clean":"2024-08-17T17:28:42.614101+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:28:42.614101+0000","last_undegraded":"2024-08-17T17:28:42.614101+0000","last_fullsized":"2024-08-17T17:28:42.614101+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":313,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.615073+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:28:42.615073+0000","last_peered":"2024-08-17T17:28:42.615073+0000","last_clean":"2024-08-17T17:28:42.615073+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:28:42.615073+0000","last_undegraded":"2024-08-17T17:28:42.615073+0000","last_fullsized":"2024-08-17T17:28:42.615073+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":332,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.614764+0000","last_change":"2024-08-17T17:25:04.509422+0000","last_active":"2024-08-17T17:28:42.614764+0000","last_peered":"2024-08-17T17:28:42.614764+0000","last_clean":"2024-08-17T17:28:42.614764+0000","last_became_active":"2024-08-17T17:25:04.508742+0000","last_became_peered":"2024-08-17T17:25:04.508742+0000","last_unstale":"2024-08-17T17:28:42.614764+0000","last_undegraded":"2024-08-17T17:28:42.614764+0000","last_fullsized":"2024-08-17T17:28:42.614764+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:18:31.976907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":189,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-08-17T17:28:40.781672+0000","last_change":"2024-08-17T17:16:26.963696+0000","last_active":"2024-08-17T17:28:40.781672+0000","last_peered":"2024-08-17T17:28:40.781672+0000","last_clean":"2024-08-17T17:28:40.781672+0000","last_became_active":"2024-08-17T17:16:26.963280+0000","last_became_peered":"2024-08-17T17:16:26.963280+0000","last_unstale":"2024-08-17T17:28:40.781672+0000","last_undegraded":"2024-08-17T17:28:40.781672+0000","last_fullsized":"2024-08-17T17:28:40.781672+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:22:01.474840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":350,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.614934+0000","last_change":"2024-08-17T17:24:06.586886+0000","last_active":"2024-08-17T17:28:42.614934+0000","last_peered":"2024-08-17T17:28:42.614934+0000","last_clean":"2024-08-17T17:28:42.614934+0000","last_became_active":"2024-08-17T17:24:06.586516+0000","last_became_peered":"2024-08-17T17:24:06.586516+0000","last_unstale":"2024-08-17T17:28:42.614934+0000","last_undegraded":"2024-08-17T17:28:42.614934+0000","last_fullsized":"2024-08-17T17:28:42.614934+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:03:30.002726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":130,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.613637+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:28:42.613637+0000","last_peered":"2024-08-17T17:28:42.613637+0000","last_clean":"2024-08-17T17:28:42.613637+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:28:42.613637+0000","last_undegraded":"2024-08-17T17:28:42.613637+0000","last_fullsized":"2024-08-17T17:28:42.613637+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":310,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.620570+0000","last_change":"2024-08-17T17:18:59.412463+0000","last_active":"2024-08-17T17:28:42.620570+0000","last_peered":"2024-08-17T17:28:42.620570+0000","last_clean":"2024-08-17T17:28:42.620570+0000","last_became_active":"2024-08-17T17:18:59.412134+0000","last_became_peered":"2024-08-17T17:18:59.412134+0000","last_unstale":"2024-08-17T17:28:42.620570+0000","last_undegraded":"2024-08-17T17:28:42.620570+0000","last_fullsized":"2024-08-17T17:28:42.620570+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:44.463153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":11,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-08-17T17:28:41.399364+0000","last_change":"2024-08-17T17:28:41.399059+0000","last_active":"2024-08-17T17:28:41.399364+0000","last_peered":"2024-08-17T17:28:41.399364+0000","last_clean":"2024-08-17T17:28:41.399364+0000","last_became_active":"2024-08-17T17:28:41.398560+0000","last_became_peered":"2024-08-17T17:28:41.398560+0000","last_unstale":"2024-08-17T17:28:41.399364+0000","last_undegraded":"2024-08-17T17:28:41.399364+0000","last_fullsized":"2024-08-17T17:28:41.399364+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:31:45.097451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":250,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.624885+0000","last_change":"2024-08-17T17:24:06.591165+0000","last_active":"2024-08-17T17:28:42.624885+0000","last_peered":"2024-08-17T17:28:42.624885+0000","last_clean":"2024-08-17T17:28:42.624885+0000","last_became_active":"2024-08-17T17:24:06.590785+0000","last_became_peered":"2024-08-17T17:24:06.590785+0000","last_unstale":"2024-08-17T17:28:42.624885+0000","last_undegraded":"2024-08-17T17:28:42.624885+0000","last_fullsized":"2024-08-17T17:28:42.624885+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:54:05.775563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":322,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.621590+0000","last_change":"2024-08-17T17:24:32.300560+0000","last_active":"2024-08-17T17:28:42.621590+0000","last_peered":"2024-08-17T17:28:42.621590+0000","last_clean":"2024-08-17T17:28:42.621590+0000","last_became_active":"2024-08-17T17:24:32.300228+0000","last_became_peered":"2024-08-17T17:24:32.300228+0000","last_unstale":"2024-08-17T17:28:42.621590+0000","last_undegraded":"2024-08-17T17:28:42.621590+0000","last_fullsized":"2024-08-17T17:28:42.621590+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:18:05.475449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":129,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.612793+0000","last_change":"2024-08-17T17:24:06.586379+0000","last_active":"2024-08-17T17:28:42.612793+0000","last_peered":"2024-08-17T17:28:42.612793+0000","last_clean":"2024-08-17T17:28:42.612793+0000","last_became_active":"2024-08-17T17:24:06.586023+0000","last_became_peered":"2024-08-17T17:24:06.586023+0000","last_unstale":"2024-08-17T17:28:42.612793+0000","last_undegraded":"2024-08-17T17:28:42.612793+0000","last_fullsized":"2024-08-17T17:28:42.612793+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:45.701637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":339,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.614573+0000","last_change":"2024-08-17T17:21:50.442535+0000","last_active":"2024-08-17T17:28:42.614573+0000","last_peered":"2024-08-17T17:28:42.614573+0000","last_clean":"2024-08-17T17:28:42.614573+0000","last_became_active":"2024-08-17T17:21:50.441891+0000","last_became_peered":"2024-08-17T17:21:50.441891+0000","last_unstale":"2024-08-17T17:28:42.614573+0000","last_undegraded":"2024-08-17T17:28:42.614573+0000","last_fullsized":"2024-08-17T17:28:42.614573+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:46:24.025407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":244,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-08-17T17:28:40.590812+0000","last_change":"2024-08-17T17:25:04.508045+0000","last_active":"2024-08-17T17:28:40.590812+0000","last_peered":"2024-08-17T17:28:40.590812+0000","last_clean":"2024-08-17T17:28:40.590812+0000","last_became_active":"2024-08-17T17:25:04.505083+0000","last_became_peered":"2024-08-17T17:25:04.505083+0000","last_unstale":"2024-08-17T17:28:40.590812+0000","last_undegraded":"2024-08-17T17:28:40.590812+0000","last_fullsized":"2024-08-17T17:28:40.590812+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:23:42.993544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":325,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.615103+0000","last_change":"2024-08-17T17:18:59.418434+0000","last_active":"2024-08-17T17:28:42.615103+0000","last_peered":"2024-08-17T17:28:42.615103+0000","last_clean":"2024-08-17T17:28:42.615103+0000","last_became_active":"2024-08-17T17:18:59.418043+0000","last_became_peered":"2024-08-17T17:18:59.418043+0000","last_unstale":"2024-08-17T17:28:42.615103+0000","last_undegraded":"2024-08-17T17:28:42.615103+0000","last_fullsized":"2024-08-17T17:28:42.615103+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:25:51.371820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":217,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-08-17T17:28:40.591143+0000","last_change":"2024-08-17T17:20:49.291167+0000","last_active":"2024-08-17T17:28:40.591143+0000","last_peered":"2024-08-17T17:28:40.591143+0000","last_clean":"2024-08-17T17:28:40.591143+0000","last_became_active":"2024-08-17T17:20:49.263750+0000","last_became_peered":"2024-08-17T17:20:49.263750+0000","last_unstale":"2024-08-17T17:28:40.591143+0000","last_undegraded":"2024-08-17T17:28:40.591143+0000","last_fullsized":"2024-08-17T17:28:40.591143+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:02.863246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":339,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.614478+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:28:42.614478+0000","last_peered":"2024-08-17T17:28:42.614478+0000","last_clean":"2024-08-17T17:28:42.614478+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:28:42.614478+0000","last_undegraded":"2024-08-17T17:28:42.614478+0000","last_fullsized":"2024-08-17T17:28:42.614478+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":339,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.618146+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:28:42.618146+0000","last_peered":"2024-08-17T17:28:42.618146+0000","last_clean":"2024-08-17T17:28:42.618146+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:28:42.618146+0000","last_undegraded":"2024-08-17T17:28:42.618146+0000","last_fullsized":"2024-08-17T17:28:42.618146+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":44,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-08-17T17:28:40.780918+0000","last_change":"2024-08-17T17:25:04.503834+0000","last_active":"2024-08-17T17:28:40.780918+0000","last_peered":"2024-08-17T17:28:40.780918+0000","last_clean":"2024-08-17T17:28:40.780918+0000","last_became_active":"2024-08-17T17:25:04.502712+0000","last_became_peered":"2024-08-17T17:25:04.502712+0000","last_unstale":"2024-08-17T17:28:40.780918+0000","last_undegraded":"2024-08-17T17:28:40.780918+0000","last_fullsized":"2024-08-17T17:28:40.780918+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:45.389103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":149,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.615048+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:28:42.615048+0000","last_peered":"2024-08-17T17:28:42.615048+0000","last_clean":"2024-08-17T17:28:42.615048+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:28:42.615048+0000","last_undegraded":"2024-08-17T17:28:42.615048+0000","last_fullsized":"2024-08-17T17:28:42.615048+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":1858076672,"num_objects":28352,"num_object_clones":0,"num_object_copies":85056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28352,"num_write_kb":1814528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5694423040,"data_stored":5694423040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28352,"ondisk_log_size":28352,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":6304,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":16702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17971,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234265,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1323256,"kb_used_data":1013424,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":92421896,"statfs":{"total":95995035648,"available":94640021504,"internally_reserved":0,"allocated":1037746176,"data_stored":1036420017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495900,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1129008,"kb_used_data":827496,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":92616144,"statfs":{"total":95995035648,"available":94838931456,"internally_reserved":0,"allocated":847355904,"data_stored":846037637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757537,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":768868,"kb_used_data":463196,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92976284,"statfs":{"total":95995035648,"available":95207714816,"internally_reserved":0,"allocated":474312704,"data_stored":473002180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019175,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435402,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1350304,"kb_used_data":981208,"kb_used_omap":4,"kb_used_meta":369083,"kb_avail":92394848,"statfs":{"total":95995035648,"available":94612324352,"internally_reserved":0,"allocated":1004756992,"data_stored":1003450338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":377941002},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":104,"seq":446676598929,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1090992,"kb_used_data":736232,"kb_used_omap":7,"kb_used_meta":354744,"kb_avail":92654160,"statfs":{"total":95995035648,"available":94877859840,"internally_reserved":0,"allocated":753901568,"data_stored":752583796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":363258018},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762456,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1262884,"kb_used_data":878556,"kb_used_omap":7,"kb_used_meta":384312,"kb_avail":92482268,"statfs":{"total":95995035648,"available":94701842432,"internally_reserved":0,"allocated":899641344,"data_stored":898330561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7945,"internal_metadata":393535735},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726925982,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1125228,"kb_used_data":823268,"kb_used_omap":7,"kb_used_meta":301944,"kb_avail":92619924,"statfs":{"total":95995035648,"available":94842802176,"internally_reserved":0,"allocated":843026432,"data_stored":841708642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":309190977},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6241,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":909115392,"data_stored":909115392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":928251904,"data_stored":928251904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":750452736,"data_stored":750452736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1059717120,"data_stored":1059717120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":481230848,"data_stored":481230848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":840957952,"data_stored":840957952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":724697088,"data_stored":724697088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:29:04.742 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:04.773 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:04.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:04.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:04.875 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:04.926 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:04.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:04.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:05.028 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:05.055 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:05.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:05.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:05.157 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:05.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:05.201 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:05.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:05.302 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:05.429 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:05.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:05.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:05.531 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:05.552 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 24 16 47538 47522 123.734 97.5625 0.0107509 0.0080761 2024-08-17T17:29:05.558 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:05.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:05.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:05.660 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:05.700 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:05.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:05.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:05.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:05 smithi003 ceph-mon[103344]: pgmap v638: 89 pgs: 89 active+clean; 2.5 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-08-17T17:29:05.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:05 smithi003 ceph-mon[100615]: pgmap v638: 89 pgs: 89 active+clean; 2.5 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-08-17T17:29:05.774 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:29:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:29:05] "GET /metrics HTTP/1.1" 200 36495 "" "Prometheus/2.51.0" 2024-08-17T17:29:05.802 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:05.828 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:05.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:05.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:05.930 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:05.958 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:05.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:05.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:05.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:05 smithi053 ceph-mon[89963]: pgmap v638: 89 pgs: 89 active+clean; 2.5 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 117 MiB/s wr, 1.87k op/s 2024-08-17T17:29:06.060 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:06.104 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:06.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:06.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:06.206 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:06.264 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:06.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:06.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:06.366 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:06.414 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:06.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:06.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:06.517 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:06.545 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:06.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:06.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:06.552 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 25 16 48965 48949 122.352 89.1875 0.00528523 0.00816716 2024-08-17T17:29:06.647 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:06.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:06.689 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:06.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:06.692 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-08-17T17:29:06.790 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:06.831 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:06.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:06.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:06.934 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:06.963 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:06.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:06.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:06.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:06 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/247543372' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:29:06.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:06 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:29:07.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:06 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/247543372' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:29:07.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:06 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:29:07.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:06 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/247543372' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:29:07.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:06 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:29:07.065 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:07.092 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:07.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:07.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:07.194 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:07.235 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:07.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:07.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:07.337 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:07.379 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:07.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:07.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:07.481 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:07.507 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:07.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:07.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:07.552 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 26 16 50458 50442 121.235 93.3125 0.0141486 0.00824257 2024-08-17T17:29:07.610 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:07.641 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:07.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:07.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:07.690 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:29:07.737 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:07 smithi003 ceph-mon[103344]: pgmap v639: 89 pgs: 89 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-08-17T17:29:07.737 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:07 smithi003 ceph-mon[103344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:29:07.737 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:07 smithi003 ceph-mon[103344]: Cluster is now healthy 2024-08-17T17:29:07.737 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:07 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:29:07.737 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:07 smithi003 ceph-mon[103344]: osdmap e233: 8 total, 8 up, 7 in 2024-08-17T17:29:07.738 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:07 smithi003 ceph-mon[100615]: pgmap v639: 89 pgs: 89 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-08-17T17:29:07.738 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:07 smithi003 ceph-mon[100615]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:29:07.738 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:07 smithi003 ceph-mon[100615]: Cluster is now healthy 2024-08-17T17:29:07.738 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:07 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:29:07.738 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:07 smithi003 ceph-mon[100615]: osdmap e233: 8 total, 8 up, 7 in 2024-08-17T17:29:07.742 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:07.788 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:07.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:07.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:07.890 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:07.935 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:07.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:07.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:07.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:07 smithi053 ceph-mon[89963]: pgmap v639: 89 pgs: 89 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-08-17T17:29:07.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:07 smithi053 ceph-mon[89963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:29:07.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:07 smithi053 ceph-mon[89963]: Cluster is now healthy 2024-08-17T17:29:07.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:07 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:29:07.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:07 smithi053 ceph-mon[89963]: osdmap e233: 8 total, 8 up, 7 in 2024-08-17T17:29:08.036 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:08.063 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:08.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:08.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:08.165 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:08.208 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:08.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:08.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:08.310 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:08.338 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:08.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:08.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:08.440 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:08.468 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:08.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:08.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:08.498 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:29:08.498 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:29:08.537 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:29:08.538 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:29:08.552 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 27 16 51730 51714 119.689 79.5 0.0102989 0.00834844 2024-08-17T17:29:08.570 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:08.579 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:29:08.580 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:29:08.597 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:08.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:08.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:08.620 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:29:08.620 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:29:08.655 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:29:08.656 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:29:08.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:08.702 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:29:08.702 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:29:08.740 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:29:08.740 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:29:08.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:08.750 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:08.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:08.757 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:29:08.757 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:29:08.776 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:29:08.776 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:29:08.851 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:08.878 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:08.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:08.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:08.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:09.022 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:09.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:09.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:09.124 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:09.151 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:09.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:09.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:09.253 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:09.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:09 smithi003 ceph-mon[103344]: pgmap v641: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 97 MiB/s wr, 1.54k op/s 2024-08-17T17:29:09.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:29:09.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:29:09.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:09 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2234186793' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:29:09.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:09 smithi003 ceph-mon[100615]: pgmap v641: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 97 MiB/s wr, 1.54k op/s 2024-08-17T17:29:09.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:29:09.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:29:09.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:09 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2234186793' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:29:09.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:09.294 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:09.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:09.395 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:09.423 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:09.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:09.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:09 smithi053 ceph-mon[89963]: pgmap v641: 89 pgs: 89 active+clean; 2.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 97 MiB/s wr, 1.54k op/s 2024-08-17T17:29:09.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:29:09.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:29:09.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:09 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2234186793' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:29:09.524 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:09.534 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:29:09.552 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:09.553 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 28 16 52966 52950 118.173 77.25 0.00428882 0.00845555 2024-08-17T17:29:09.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:09.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:09.655 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:09.683 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:09.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:09.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:09.785 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:09.830 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:09.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:09.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:09.932 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:09.969 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:09.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:09.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:10.071 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:10.116 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:10.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:10.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:10.218 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:10.244 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:10.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:10.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:10.346 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:10.371 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:10.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:10.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:10.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:29:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:29:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:29:10.473 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:10.515 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:10.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:10.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:10.552 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 29 16 54191 54175 116.738 76.5625 0.0129518 0.0085596 2024-08-17T17:29:10.617 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:10.623 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-17T17:29:10.642 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:10.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:10.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:10.744 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:10.771 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:10.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:10.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:10.872 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:10.898 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:10.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:10.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:11.001 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:11.047 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:11.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:11.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:11.069 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:29:11.151 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:11.177 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:11.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:11.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:11.279 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:11.306 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:11.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:11.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:11.408 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:11.414 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:11 smithi003 ceph-mon[100615]: pgmap v642: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-08-17T17:29:11.415 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:11 smithi003 ceph-mon[103344]: pgmap v642: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-08-17T17:29:11.456 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:11.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:11.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:11.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:11 smithi053 ceph-mon[89963]: pgmap v642: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 93 MiB/s wr, 1.49k op/s 2024-08-17T17:29:11.553 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 30 16 55378 55362 115.319 74.1875 0.00793312 0.00866541 2024-08-17T17:29:11.558 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:11.603 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:11.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:11.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:11.705 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:11.743 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:11.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:11.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:11.844 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:29:11.845 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:29:11.845 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:11.902 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:11.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:11.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:12.004 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:12.045 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:12.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:12.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:12.147 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:12.185 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:12.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:12.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:12.287 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:12.315 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:12.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:12.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:12.416 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:12.462 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:12.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:12.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:12.553 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 31 16 56677 56661 114.218 81.1875 0.00300356 0.00874908 2024-08-17T17:29:12.564 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:12.602 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:12.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:12.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:12.704 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:12.731 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:12.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:12.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:12.832 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:12.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:12.878 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:12.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:12.979 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:13.005 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:13.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:13.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:13.107 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:13.118 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:29:13.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:13.153 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:13.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:13.228 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:13 smithi003 ceph-mon[103344]: pgmap v643: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-08-17T17:29:13.228 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:13 smithi003 ceph-mon[100615]: pgmap v643: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-08-17T17:29:13.254 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:13.280 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:13.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:13.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:13.382 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:13.434 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:13.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:13.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:13.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:13 smithi053 ceph-mon[89963]: pgmap v643: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-08-17T17:29:13.537 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:13.554 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 32 16 57982 57966 113.192 81.5625 0.00991473 0.0088287 2024-08-17T17:29:13.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:13.581 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:13.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:13.682 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:13.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:13.709 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:13.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:13.810 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:13.859 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:13.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:13.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:13.961 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:13.992 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:13.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:13.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:14.094 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:14.138 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:14.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:14.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:14.240 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:14.268 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:14.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:14.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:14.370 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:14.397 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:14.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:14.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:14.499 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:14.525 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:14.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:14.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:14.554 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 33 16 59310 59294 112.277 83 0.00879454 0.00890048 2024-08-17T17:29:14.627 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:14.669 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:14.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:14.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:14.771 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:14.799 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:14.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:14.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:14.901 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:14.927 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:14.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:14.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:15.029 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:15.057 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:15.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:15.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:15.159 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:15.184 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:15.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:15.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:15.250 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:15 smithi003 ceph-mon[103344]: pgmap v644: 89 pgs: 89 active+clean; 3.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-08-17T17:29:15.251 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:15 smithi003 ceph-mon[100615]: pgmap v644: 89 pgs: 89 active+clean; 3.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-08-17T17:29:15.285 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:15.309 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:15.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:15.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:15.411 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:15.414 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:29:15.415 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:29:15.441 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:15.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:15.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:15.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:15 smithi053 ceph-mon[89963]: pgmap v644: 89 pgs: 89 active+clean; 3.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-08-17T17:29:15.543 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:15.554 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 34 16 60542 60526 111.239 77 0.0122554 0.00898436 2024-08-17T17:29:15.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:15.581 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:15.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:15.682 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:15.705 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:15.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:15.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:15.791 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:29:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:29:15] "GET /metrics HTTP/1.1" 200 36500 "" "Prometheus/2.51.0" 2024-08-17T17:29:15.806 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:15.834 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:15.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:15.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:15.936 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:15.957 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:15.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:15.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:16.059 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:16.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:16.104 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:16.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:16.205 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:16.243 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:16.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:16.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:16.344 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:16.392 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:16.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:16.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:16.399 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:16 smithi003 ceph-mon[103344]: from='client.45806 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:29:16.399 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:16 smithi003 ceph-mon[100615]: from='client.45806 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:29:16.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:16 smithi053 ceph-mon[89963]: from='client.45806 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:29:16.494 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:16.522 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:16.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:16.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:16.555 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 35 16 61667 61651 110.07 70.3125 0.00955867 0.00907897 2024-08-17T17:29:16.624 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:16.664 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:16.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:16.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:16.765 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:16.803 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:16.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:16.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:16.904 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:16.945 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:16.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:16.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:17.047 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:17.074 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:17.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:17.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:17.176 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:17.202 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:17.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:17.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:17.304 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:17.331 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:17.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:17.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:17.433 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:17.461 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:17.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:17.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:17.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:17 smithi053 ceph-mon[89963]: pgmap v645: 89 pgs: 89 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 85 MiB/s wr, 1.37k op/s 2024-08-17T17:29:17.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:17 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/4120407271' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:29:17.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:17 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:29:17.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:17 smithi003 ceph-mon[103344]: pgmap v645: 89 pgs: 89 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 85 MiB/s wr, 1.37k op/s 2024-08-17T17:29:17.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:17 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/4120407271' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:29:17.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:17 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:29:17.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:17 smithi003 ceph-mon[100615]: pgmap v645: 89 pgs: 89 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 85 MiB/s wr, 1.37k op/s 2024-08-17T17:29:17.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:17 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/4120407271' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:29:17.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:17 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:29:17.555 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 36 16 62823 62807 109.019 72.25 0.0134138 0.00916674 2024-08-17T17:29:17.563 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:17.605 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:17.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:17.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:17.607 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-08-17T17:29:17.707 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:17.729 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:17.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:17.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:17.831 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:17.858 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:17.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:17.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:17.960 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:17.987 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:17.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:17.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:18.089 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:18.111 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:18.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:18.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:18.213 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:18.241 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:18.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:18.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:18.342 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:18.364 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:18.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:18.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:18.466 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:18.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:18 smithi053 ceph-mon[89963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:29:18.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:18 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:29:18.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:18 smithi053 ceph-mon[89963]: osdmap e234: 8 total, 8 up, 7 in 2024-08-17T17:29:18.489 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:18.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:18.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:18.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:18 smithi003 ceph-mon[100615]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:29:18.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:18 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:29:18.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:18 smithi003 ceph-mon[100615]: osdmap e234: 8 total, 8 up, 7 in 2024-08-17T17:29:18.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:18 smithi003 ceph-mon[103344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:29:18.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:18 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:29:18.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:18 smithi003 ceph-mon[103344]: osdmap e234: 8 total, 8 up, 7 in 2024-08-17T17:29:18.555 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 37 16 63574 63558 107.341 46.9375 0.00855974 0.00931071 2024-08-17T17:29:18.590 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:18.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:18.633 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:18.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:18.734 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:18.760 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:18.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:18.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:18.862 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:18.890 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:18.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:18.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:18.992 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:19.019 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:19.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:19.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:19.121 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:19.144 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:19.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:19.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:19.247 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:19.275 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:19.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:19.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:19.377 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:19.405 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:19.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:19.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:19.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:19 smithi053 ceph-mon[89963]: pgmap v647: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-08-17T17:29:19.507 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:19.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:19 smithi003 ceph-mon[103344]: pgmap v647: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-08-17T17:29:19.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:19 smithi003 ceph-mon[100615]: pgmap v647: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-08-17T17:29:19.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:19.543 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:19.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:19.555 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 38 16 64789 64773 106.515 75.9375 0.00302011 0.00937922 2024-08-17T17:29:19.645 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:19.690 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:19.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:19.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:19.792 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:19.825 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:19.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:19.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:19.927 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:19.954 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:19.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:19.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:20.056 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:20.082 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:20.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:20.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:20.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:20.223 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:20.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:20.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:20.324 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:20.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:20.366 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:20.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:20.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:29:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:29:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:29:20.467 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:20.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:20.510 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:20.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:20.555 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 39 16 66004 65988 105.731 75.9375 0.00911387 0.00945266 2024-08-17T17:29:20.611 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:20.636 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:20.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:20.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:20.738 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:20.775 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:20.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:20.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:20.877 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:20.905 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:20.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:20.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:21.007 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:21.035 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:21.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:21.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:21.137 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:21.163 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:21.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:21.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:21.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:21.301 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:21.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:21.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:21.403 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:21.438 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:21.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:21.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:21.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:21 smithi053 ceph-mon[89963]: pgmap v648: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-08-17T17:29:21.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:21 smithi003 ceph-mon[103344]: pgmap v648: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-08-17T17:29:21.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:21 smithi003 ceph-mon[100615]: pgmap v648: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-08-17T17:29:21.540 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:21.555 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-08-17T17:29:21.555611+0000 min lat: 0.00130565 max lat: 0.2474 avg lat: 0.00951616 2024-08-17T17:29:21.555 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:29:21.555 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 40 16 67247 67231 105.029 77.6875 0.00683223 0.00951616 2024-08-17T17:29:21.569 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:21.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:21.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:21.671 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:21.698 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:21.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:21.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:21.800 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:21.844 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:21.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:21.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:21.945 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:21.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:21.983 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:21.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:22.084 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:22.112 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:22.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:22.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:22.214 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:22.239 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:22.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:22.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:22.341 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:22.366 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:22.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:22.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:22.368 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":644,"stamp":"2024-08-17T17:29:13.962558+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3613071309,"num_objects":55086,"num_object_clones":0,"num_object_copies":165258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55086,"num_whiteouts":0,"num_read":6376,"num_read_kb":13833,"num_write":56180,"num_write_kb":3538751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":11937158,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56239,"ondisk_log_size":56239,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":270,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":13102720,"kb_used_data":10672776,"kb_used_omap":33,"kb_used_meta":2429790,"kb_avail":643113344,"statfs":{"total":671965249536,"available":658548064256,"internally_reserved":0,"allocated":10928922624,"data_stored":10919687116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34068,"internal_metadata":2488105708},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,0,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":924581888,"num_objects":14108,"num_object_clones":0,"num_object_copies":42324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14108,"num_write_kb":902912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002697"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":321,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.688310+0000","last_change":"2024-08-17T17:17:32.495652+0000","last_active":"2024-08-17T17:29:06.688310+0000","last_peered":"2024-08-17T17:29:06.688310+0000","last_clean":"2024-08-17T17:29:06.688310+0000","last_became_active":"2024-08-17T17:17:32.495218+0000","last_became_peered":"2024-08-17T17:17:32.495218+0000","last_unstale":"2024-08-17T17:29:06.688310+0000","last_undegraded":"2024-08-17T17:29:06.688310+0000","last_fullsized":"2024-08-17T17:29:06.688310+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:11:45.821282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":209,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.605970+0000","last_change":"2024-08-17T17:18:59.424887+0000","last_active":"2024-08-17T17:28:42.605970+0000","last_peered":"2024-08-17T17:28:42.605970+0000","last_clean":"2024-08-17T17:28:42.605970+0000","last_became_active":"2024-08-17T17:18:59.411539+0000","last_became_peered":"2024-08-17T17:18:59.411539+0000","last_unstale":"2024-08-17T17:28:42.605970+0000","last_undegraded":"2024-08-17T17:28:42.605970+0000","last_fullsized":"2024-08-17T17:28:42.605970+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:46:20.082241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":49,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.621304+0000","last_change":"2024-08-17T17:24:53.460742+0000","last_active":"2024-08-17T17:28:42.621304+0000","last_peered":"2024-08-17T17:28:42.621304+0000","last_clean":"2024-08-17T17:28:42.621304+0000","last_became_active":"2024-08-17T17:24:53.460365+0000","last_became_peered":"2024-08-17T17:24:53.460365+0000","last_unstale":"2024-08-17T17:28:42.621304+0000","last_undegraded":"2024-08-17T17:28:42.621304+0000","last_fullsized":"2024-08-17T17:28:42.621304+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:24:08.116968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":132,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.689985+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:29:06.689985+0000","last_peered":"2024-08-17T17:29:06.689985+0000","last_clean":"2024-08-17T17:29:06.689985+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:29:06.689985+0000","last_undegraded":"2024-08-17T17:29:06.689985+0000","last_fullsized":"2024-08-17T17:29:06.689985+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":132,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.695770+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:29:06.695770+0000","last_peered":"2024-08-17T17:29:06.695770+0000","last_clean":"2024-08-17T17:29:06.695770+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:29:06.695770+0000","last_undegraded":"2024-08-17T17:29:06.695770+0000","last_fullsized":"2024-08-17T17:29:06.695770+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":184,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.605880+0000","last_change":"2024-08-17T17:20:49.258206+0000","last_active":"2024-08-17T17:28:42.605880+0000","last_peered":"2024-08-17T17:28:42.605880+0000","last_clean":"2024-08-17T17:28:42.605880+0000","last_became_active":"2024-08-17T17:20:49.257761+0000","last_became_peered":"2024-08-17T17:20:49.257761+0000","last_unstale":"2024-08-17T17:28:42.605880+0000","last_undegraded":"2024-08-17T17:28:42.605880+0000","last_fullsized":"2024-08-17T17:28:42.605880+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:38:35.363748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":334,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.688806+0000","last_change":"2024-08-17T17:19:57.514773+0000","last_active":"2024-08-17T17:29:06.688806+0000","last_peered":"2024-08-17T17:29:06.688806+0000","last_clean":"2024-08-17T17:29:06.688806+0000","last_became_active":"2024-08-17T17:19:57.513839+0000","last_became_peered":"2024-08-17T17:19:57.513839+0000","last_unstale":"2024-08-17T17:29:06.688806+0000","last_undegraded":"2024-08-17T17:29:06.688806+0000","last_fullsized":"2024-08-17T17:29:06.688806+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:42:48.063169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":279,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.614647+0000","last_change":"2024-08-17T17:24:53.465875+0000","last_active":"2024-08-17T17:28:42.614647+0000","last_peered":"2024-08-17T17:28:42.614647+0000","last_clean":"2024-08-17T17:28:42.614647+0000","last_became_active":"2024-08-17T17:24:53.458129+0000","last_became_peered":"2024-08-17T17:24:53.458129+0000","last_unstale":"2024-08-17T17:28:42.614647+0000","last_undegraded":"2024-08-17T17:28:42.614647+0000","last_fullsized":"2024-08-17T17:28:42.614647+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:51:33.584869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"233'3379","reported_seq":3404,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:10.568914+0000","last_change":"2024-08-17T17:28:27.510969+0000","last_active":"2024-08-17T17:29:10.568914+0000","last_peered":"2024-08-17T17:29:10.568914+0000","last_clean":"2024-08-17T17:29:10.568914+0000","last_became_active":"2024-08-17T17:28:27.510664+0000","last_became_peered":"2024-08-17T17:28:27.510664+0000","last_unstale":"2024-08-17T17:29:10.568914+0000","last_undegraded":"2024-08-17T17:29:10.568914+0000","last_fullsized":"2024-08-17T17:29:10.568914+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":3379,"log_dups_size":0,"ondisk_log_size":3379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:16:23.845426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221446144,"num_objects":3379,"num_object_clones":0,"num_object_copies":10137,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3379,"num_write_kb":216256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":145,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.606445+0000","last_change":"2024-08-17T17:24:06.597382+0000","last_active":"2024-08-17T17:28:42.606445+0000","last_peered":"2024-08-17T17:28:42.606445+0000","last_clean":"2024-08-17T17:28:42.606445+0000","last_became_active":"2024-08-17T17:24:06.588881+0000","last_became_peered":"2024-08-17T17:24:06.588881+0000","last_unstale":"2024-08-17T17:28:42.606445+0000","last_undegraded":"2024-08-17T17:28:42.606445+0000","last_fullsized":"2024-08-17T17:28:42.606445+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:27.240751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":59,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.620919+0000","last_change":"2024-08-17T17:25:04.503780+0000","last_active":"2024-08-17T17:28:42.620919+0000","last_peered":"2024-08-17T17:28:42.620919+0000","last_clean":"2024-08-17T17:28:42.620919+0000","last_became_active":"2024-08-17T17:25:04.503503+0000","last_became_peered":"2024-08-17T17:25:04.503503+0000","last_unstale":"2024-08-17T17:28:42.620919+0000","last_undegraded":"2024-08-17T17:28:42.620919+0000","last_fullsized":"2024-08-17T17:28:42.620919+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:04:48.171216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"233'3525","reported_seq":3550,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:12.067953+0000","last_change":"2024-08-17T17:28:27.509334+0000","last_active":"2024-08-17T17:29:12.067953+0000","last_peered":"2024-08-17T17:29:12.067953+0000","last_clean":"2024-08-17T17:29:12.067953+0000","last_became_active":"2024-08-17T17:28:27.509026+0000","last_became_peered":"2024-08-17T17:28:27.509026+0000","last_unstale":"2024-08-17T17:29:12.067953+0000","last_undegraded":"2024-08-17T17:29:12.067953+0000","last_fullsized":"2024-08-17T17:29:12.067953+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":3525,"log_dups_size":0,"ondisk_log_size":3525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:47:28.720615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231014400,"num_objects":3525,"num_object_clones":0,"num_object_copies":10575,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3525,"num_write_kb":225600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":131,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.606463+0000","last_change":"2024-08-17T17:24:06.599023+0000","last_active":"2024-08-17T17:28:42.606463+0000","last_peered":"2024-08-17T17:28:42.606463+0000","last_clean":"2024-08-17T17:28:42.606463+0000","last_became_active":"2024-08-17T17:24:06.588954+0000","last_became_peered":"2024-08-17T17:24:06.588954+0000","last_unstale":"2024-08-17T17:28:42.606463+0000","last_undegraded":"2024-08-17T17:28:42.606463+0000","last_fullsized":"2024-08-17T17:28:42.606463+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:20:51.445576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":275,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.700451+0000","last_change":"2024-08-17T17:20:49.271540+0000","last_active":"2024-08-17T17:29:06.700451+0000","last_peered":"2024-08-17T17:29:06.700451+0000","last_clean":"2024-08-17T17:29:06.700451+0000","last_became_active":"2024-08-17T17:20:49.271130+0000","last_became_peered":"2024-08-17T17:20:49.271130+0000","last_unstale":"2024-08-17T17:29:06.700451+0000","last_undegraded":"2024-08-17T17:29:06.700451+0000","last_fullsized":"2024-08-17T17:29:06.700451+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:05.386705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.9","version":"233'3652","reported_seq":3677,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:13.121188+0000","last_change":"2024-08-17T17:28:27.508033+0000","last_active":"2024-08-17T17:29:13.121188+0000","last_peered":"2024-08-17T17:29:13.121188+0000","last_clean":"2024-08-17T17:29:13.121188+0000","last_became_active":"2024-08-17T17:28:27.507657+0000","last_became_peered":"2024-08-17T17:28:27.507657+0000","last_unstale":"2024-08-17T17:29:13.121188+0000","last_undegraded":"2024-08-17T17:29:13.121188+0000","last_fullsized":"2024-08-17T17:29:13.121188+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":3652,"log_dups_size":0,"ondisk_log_size":3652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:09:04.908184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239337472,"num_objects":3652,"num_object_clones":0,"num_object_copies":10956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3652,"num_write_kb":233728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":203,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.606313+0000","last_change":"2024-08-17T17:24:53.467526+0000","last_active":"2024-08-17T17:28:42.606313+0000","last_peered":"2024-08-17T17:28:42.606313+0000","last_clean":"2024-08-17T17:28:42.606313+0000","last_became_active":"2024-08-17T17:24:53.460631+0000","last_became_peered":"2024-08-17T17:24:53.460631+0000","last_unstale":"2024-08-17T17:28:42.606313+0000","last_undegraded":"2024-08-17T17:28:42.606313+0000","last_fullsized":"2024-08-17T17:28:42.606313+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:05:24.938831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":45,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.614668+0000","last_change":"2024-08-17T17:25:04.507176+0000","last_active":"2024-08-17T17:28:42.614668+0000","last_peered":"2024-08-17T17:28:42.614668+0000","last_clean":"2024-08-17T17:28:42.614668+0000","last_became_active":"2024-08-17T17:25:04.506788+0000","last_became_peered":"2024-08-17T17:25:04.506788+0000","last_unstale":"2024-08-17T17:28:42.614668+0000","last_undegraded":"2024-08-17T17:28:42.614668+0000","last_fullsized":"2024-08-17T17:28:42.614668+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:22:04.658216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"233'3401","reported_seq":3416,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:09.056753+0000","last_change":"2024-08-17T17:28:27.509743+0000","last_active":"2024-08-17T17:29:09.056753+0000","last_peered":"2024-08-17T17:29:09.056753+0000","last_clean":"2024-08-17T17:29:09.056753+0000","last_became_active":"2024-08-17T17:28:27.509414+0000","last_became_peered":"2024-08-17T17:28:27.509414+0000","last_unstale":"2024-08-17T17:29:09.056753+0000","last_undegraded":"2024-08-17T17:29:09.056753+0000","last_fullsized":"2024-08-17T17:29:09.056753+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":3401,"log_dups_size":0,"ondisk_log_size":3401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:37:42.377123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222887936,"num_objects":3401,"num_object_clones":0,"num_object_copies":10203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3401,"num_write_kb":217664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":191,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.621134+0000","last_change":"2024-08-17T17:17:32.495428+0000","last_active":"2024-08-17T17:28:42.621134+0000","last_peered":"2024-08-17T17:28:42.621134+0000","last_clean":"2024-08-17T17:28:42.621134+0000","last_became_active":"2024-08-17T17:17:32.494978+0000","last_became_peered":"2024-08-17T17:17:32.494978+0000","last_unstale":"2024-08-17T17:28:42.621134+0000","last_undegraded":"2024-08-17T17:28:42.621134+0000","last_fullsized":"2024-08-17T17:28:42.621134+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:42:24.647812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":323,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.691711+0000","last_change":"2024-08-17T17:20:49.272126+0000","last_active":"2024-08-17T17:29:06.691711+0000","last_peered":"2024-08-17T17:29:06.691711+0000","last_clean":"2024-08-17T17:29:06.691711+0000","last_became_active":"2024-08-17T17:20:49.271185+0000","last_became_peered":"2024-08-17T17:20:49.271185+0000","last_unstale":"2024-08-17T17:29:06.691711+0000","last_undegraded":"2024-08-17T17:29:06.691711+0000","last_fullsized":"2024-08-17T17:29:06.691711+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:43:58.184939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"233'3621","reported_seq":3646,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:13.119637+0000","last_change":"2024-08-17T17:28:27.512879+0000","last_active":"2024-08-17T17:29:13.119637+0000","last_peered":"2024-08-17T17:29:13.119637+0000","last_clean":"2024-08-17T17:29:13.119637+0000","last_became_active":"2024-08-17T17:28:27.512586+0000","last_became_peered":"2024-08-17T17:28:27.512586+0000","last_unstale":"2024-08-17T17:29:13.119637+0000","last_undegraded":"2024-08-17T17:29:13.119637+0000","last_fullsized":"2024-08-17T17:29:13.119637+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":3621,"log_dups_size":0,"ondisk_log_size":3621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:12:27.175368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237305856,"num_objects":3621,"num_object_clones":0,"num_object_copies":10863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3621,"num_write_kb":231744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":352,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.694534+0000","last_change":"2024-08-17T17:24:06.595380+0000","last_active":"2024-08-17T17:29:06.694534+0000","last_peered":"2024-08-17T17:29:06.694534+0000","last_clean":"2024-08-17T17:29:06.694534+0000","last_became_active":"2024-08-17T17:24:06.584747+0000","last_became_peered":"2024-08-17T17:24:06.584747+0000","last_unstale":"2024-08-17T17:29:06.694534+0000","last_undegraded":"2024-08-17T17:29:06.694534+0000","last_fullsized":"2024-08-17T17:29:06.694534+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:02:18.473980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":297,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.688584+0000","last_change":"2024-08-17T17:20:49.280596+0000","last_active":"2024-08-17T17:29:06.688584+0000","last_peered":"2024-08-17T17:29:06.688584+0000","last_clean":"2024-08-17T17:29:06.688584+0000","last_became_active":"2024-08-17T17:20:49.279553+0000","last_became_peered":"2024-08-17T17:20:49.279553+0000","last_unstale":"2024-08-17T17:29:06.688584+0000","last_undegraded":"2024-08-17T17:29:06.688584+0000","last_fullsized":"2024-08-17T17:29:06.688584+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:13:24.124392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"233'3268","reported_seq":3293,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:09.558961+0000","last_change":"2024-08-17T17:28:27.506638+0000","last_active":"2024-08-17T17:29:09.558961+0000","last_peered":"2024-08-17T17:29:09.558961+0000","last_clean":"2024-08-17T17:29:09.558961+0000","last_became_active":"2024-08-17T17:28:27.506343+0000","last_became_peered":"2024-08-17T17:28:27.506343+0000","last_unstale":"2024-08-17T17:29:09.558961+0000","last_undegraded":"2024-08-17T17:29:09.558961+0000","last_fullsized":"2024-08-17T17:29:09.558961+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":3268,"log_dups_size":0,"ondisk_log_size":3268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:57:42.320556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214171648,"num_objects":3268,"num_object_clones":0,"num_object_copies":9804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3268,"num_write_kb":209152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":249,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.700212+0000","last_change":"2024-08-17T17:20:49.276769+0000","last_active":"2024-08-17T17:29:06.700212+0000","last_peered":"2024-08-17T17:29:06.700212+0000","last_clean":"2024-08-17T17:29:06.700212+0000","last_became_active":"2024-08-17T17:20:49.276498+0000","last_became_peered":"2024-08-17T17:20:49.276498+0000","last_unstale":"2024-08-17T17:29:06.700212+0000","last_undegraded":"2024-08-17T17:29:06.700212+0000","last_fullsized":"2024-08-17T17:29:06.700212+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:39:05.419440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":212,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.606117+0000","last_change":"2024-08-17T17:20:14.455738+0000","last_active":"2024-08-17T17:28:42.606117+0000","last_peered":"2024-08-17T17:28:42.606117+0000","last_clean":"2024-08-17T17:28:42.606117+0000","last_became_active":"2024-08-17T17:20:14.455469+0000","last_became_peered":"2024-08-17T17:20:14.455469+0000","last_unstale":"2024-08-17T17:28:42.606117+0000","last_undegraded":"2024-08-17T17:28:42.606117+0000","last_fullsized":"2024-08-17T17:28:42.606117+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:27:28.424336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.d","version":"233'3213","reported_seq":3228,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:08.945628+0000","last_change":"2024-08-17T17:28:27.510620+0000","last_active":"2024-08-17T17:29:08.945628+0000","last_peered":"2024-08-17T17:29:08.945628+0000","last_clean":"2024-08-17T17:29:08.945628+0000","last_became_active":"2024-08-17T17:28:27.510351+0000","last_became_peered":"2024-08-17T17:28:27.510351+0000","last_unstale":"2024-08-17T17:29:08.945628+0000","last_undegraded":"2024-08-17T17:29:08.945628+0000","last_fullsized":"2024-08-17T17:29:08.945628+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":3213,"log_dups_size":0,"ondisk_log_size":3213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:14:19.994370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210567168,"num_objects":3213,"num_object_clones":0,"num_object_copies":9639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3213,"num_write_kb":205632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":166,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.605774+0000","last_change":"2024-08-17T17:20:14.431140+0000","last_active":"2024-08-17T17:28:42.605774+0000","last_peered":"2024-08-17T17:28:42.605774+0000","last_clean":"2024-08-17T17:28:42.605774+0000","last_became_active":"2024-08-17T17:20:14.430839+0000","last_became_peered":"2024-08-17T17:20:14.430839+0000","last_unstale":"2024-08-17T17:28:42.605774+0000","last_undegraded":"2024-08-17T17:28:42.605774+0000","last_fullsized":"2024-08-17T17:28:42.605774+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:08:16.303616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":45,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.615110+0000","last_change":"2024-08-17T17:25:04.509325+0000","last_active":"2024-08-17T17:28:42.615110+0000","last_peered":"2024-08-17T17:28:42.615110+0000","last_clean":"2024-08-17T17:28:42.615110+0000","last_became_active":"2024-08-17T17:25:04.508364+0000","last_became_peered":"2024-08-17T17:25:04.508364+0000","last_unstale":"2024-08-17T17:28:42.615110+0000","last_undegraded":"2024-08-17T17:28:42.615110+0000","last_fullsized":"2024-08-17T17:28:42.615110+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:48:56.415763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"233'3215","reported_seq":3230,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:08.925248+0000","last_change":"2024-08-17T17:28:27.512014+0000","last_active":"2024-08-17T17:29:08.925248+0000","last_peered":"2024-08-17T17:29:08.925248+0000","last_clean":"2024-08-17T17:29:08.925248+0000","last_became_active":"2024-08-17T17:28:27.511692+0000","last_became_peered":"2024-08-17T17:28:27.511692+0000","last_unstale":"2024-08-17T17:29:08.925248+0000","last_undegraded":"2024-08-17T17:29:08.925248+0000","last_fullsized":"2024-08-17T17:29:08.925248+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":3215,"log_dups_size":0,"ondisk_log_size":3215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:38:59.458491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210698240,"num_objects":3215,"num_object_clones":0,"num_object_copies":9645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3215,"num_write_kb":205760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":336,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.695022+0000","last_change":"2024-08-17T17:19:57.520745+0000","last_active":"2024-08-17T17:29:06.695022+0000","last_peered":"2024-08-17T17:29:06.695022+0000","last_clean":"2024-08-17T17:29:06.695022+0000","last_became_active":"2024-08-17T17:19:57.520152+0000","last_became_peered":"2024-08-17T17:19:57.520152+0000","last_unstale":"2024-08-17T17:29:06.695022+0000","last_undegraded":"2024-08-17T17:29:06.695022+0000","last_fullsized":"2024-08-17T17:29:06.695022+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:19:39.495644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":311,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.700338+0000","last_change":"2024-08-17T17:20:49.274372+0000","last_active":"2024-08-17T17:29:06.700338+0000","last_peered":"2024-08-17T17:29:06.700338+0000","last_clean":"2024-08-17T17:29:06.700338+0000","last_became_active":"2024-08-17T17:20:49.274023+0000","last_became_peered":"2024-08-17T17:20:49.274023+0000","last_unstale":"2024-08-17T17:29:06.700338+0000","last_undegraded":"2024-08-17T17:29:06.700338+0000","last_fullsized":"2024-08-17T17:29:06.700338+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:31:02.529766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"233'3579","reported_seq":3604,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:13.103035+0000","last_change":"2024-08-17T17:28:27.511026+0000","last_active":"2024-08-17T17:29:13.103035+0000","last_peered":"2024-08-17T17:29:13.103035+0000","last_clean":"2024-08-17T17:29:13.103035+0000","last_became_active":"2024-08-17T17:28:27.510738+0000","last_became_peered":"2024-08-17T17:28:27.510738+0000","last_unstale":"2024-08-17T17:29:13.103035+0000","last_undegraded":"2024-08-17T17:29:13.103035+0000","last_fullsized":"2024-08-17T17:29:13.103035+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":3579,"log_dups_size":0,"ondisk_log_size":3579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:58:33.553008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234553344,"num_objects":3579,"num_object_clones":0,"num_object_copies":10737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3579,"num_write_kb":229056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":560,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.605703+0000","last_change":"2024-08-17T17:24:09.599615+0000","last_active":"2024-08-17T17:28:42.605703+0000","last_peered":"2024-08-17T17:28:42.605703+0000","last_clean":"2024-08-17T17:28:42.605703+0000","last_became_active":"2024-08-17T17:24:09.599085+0000","last_became_peered":"2024-08-17T17:24:09.599085+0000","last_unstale":"2024-08-17T17:28:42.605703+0000","last_undegraded":"2024-08-17T17:28:42.605703+0000","last_fullsized":"2024-08-17T17:28:42.605703+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:04:01.141401+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:30:09.569579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":805,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.621294+0000","last_change":"2024-08-17T17:20:00.434774+0000","last_active":"2024-08-17T17:28:42.621294+0000","last_peered":"2024-08-17T17:28:42.621294+0000","last_clean":"2024-08-17T17:28:42.621294+0000","last_became_active":"2024-08-17T17:20:00.434343+0000","last_became_peered":"2024-08-17T17:20:00.434343+0000","last_unstale":"2024-08-17T17:28:42.621294+0000","last_undegraded":"2024-08-17T17:28:42.621294+0000","last_fullsized":"2024-08-17T17:28:42.621294+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:04.995152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":294,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.689128+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:29:06.689128+0000","last_peered":"2024-08-17T17:29:06.689128+0000","last_clean":"2024-08-17T17:29:06.689128+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:29:06.689128+0000","last_undegraded":"2024-08-17T17:29:06.689128+0000","last_fullsized":"2024-08-17T17:29:06.689128+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":45,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.615053+0000","last_change":"2024-08-17T17:25:04.579426+0000","last_active":"2024-08-17T17:28:42.615053+0000","last_peered":"2024-08-17T17:28:42.615053+0000","last_clean":"2024-08-17T17:28:42.615053+0000","last_became_active":"2024-08-17T17:25:04.579012+0000","last_became_peered":"2024-08-17T17:25:04.579012+0000","last_unstale":"2024-08-17T17:28:42.615053+0000","last_undegraded":"2024-08-17T17:28:42.615053+0000","last_fullsized":"2024-08-17T17:28:42.615053+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:56:32.241321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"233'3247","reported_seq":3262,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:09.061471+0000","last_change":"2024-08-17T17:28:27.506760+0000","last_active":"2024-08-17T17:29:09.061471+0000","last_peered":"2024-08-17T17:29:09.061471+0000","last_clean":"2024-08-17T17:29:09.061471+0000","last_became_active":"2024-08-17T17:28:27.506306+0000","last_became_peered":"2024-08-17T17:28:27.506306+0000","last_unstale":"2024-08-17T17:29:09.061471+0000","last_undegraded":"2024-08-17T17:29:09.061471+0000","last_fullsized":"2024-08-17T17:29:09.061471+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":3247,"log_dups_size":0,"ondisk_log_size":3247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:41:13.520504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212795392,"num_objects":3247,"num_object_clones":0,"num_object_copies":9741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3247,"num_write_kb":207808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":823,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.606196+0000","last_change":"2024-08-17T17:16:26.958415+0000","last_active":"2024-08-17T17:28:42.606196+0000","last_peered":"2024-08-17T17:28:42.606196+0000","last_clean":"2024-08-17T17:28:42.606196+0000","last_became_active":"2024-08-17T17:16:26.957956+0000","last_became_peered":"2024-08-17T17:16:26.957956+0000","last_unstale":"2024-08-17T17:28:42.606196+0000","last_undegraded":"2024-08-17T17:28:42.606196+0000","last_fullsized":"2024-08-17T17:28:42.606196+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:22.478077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":323,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.690788+0000","last_change":"2024-08-17T17:18:59.415331+0000","last_active":"2024-08-17T17:29:06.690788+0000","last_peered":"2024-08-17T17:29:06.690788+0000","last_clean":"2024-08-17T17:29:06.690788+0000","last_became_active":"2024-08-17T17:18:59.415008+0000","last_became_peered":"2024-08-17T17:18:59.415008+0000","last_unstale":"2024-08-17T17:29:06.690788+0000","last_undegraded":"2024-08-17T17:29:06.690788+0000","last_fullsized":"2024-08-17T17:29:06.690788+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:22:38.076866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":132,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.696015+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:29:06.696015+0000","last_peered":"2024-08-17T17:29:06.696015+0000","last_clean":"2024-08-17T17:29:06.696015+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:29:06.696015+0000","last_undegraded":"2024-08-17T17:29:06.696015+0000","last_fullsized":"2024-08-17T17:29:06.696015+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"233'3606","reported_seq":3631,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:13.140561+0000","last_change":"2024-08-17T17:28:27.509856+0000","last_active":"2024-08-17T17:29:13.140561+0000","last_peered":"2024-08-17T17:29:13.140561+0000","last_clean":"2024-08-17T17:29:13.140561+0000","last_became_active":"2024-08-17T17:28:27.509524+0000","last_became_peered":"2024-08-17T17:28:27.509524+0000","last_unstale":"2024-08-17T17:29:13.140561+0000","last_undegraded":"2024-08-17T17:29:13.140561+0000","last_fullsized":"2024-08-17T17:29:13.140561+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":3606,"log_dups_size":0,"ondisk_log_size":3606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:02:26.063259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236322816,"num_objects":3606,"num_object_clones":0,"num_object_copies":10818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3606,"num_write_kb":230784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":711,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.693457+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:29:06.693457+0000","last_peered":"2024-08-17T17:29:06.693457+0000","last_clean":"2024-08-17T17:29:06.693457+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:29:06.693457+0000","last_undegraded":"2024-08-17T17:29:06.693457+0000","last_fullsized":"2024-08-17T17:29:06.693457+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":319,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.699870+0000","last_change":"2024-08-17T17:19:57.515564+0000","last_active":"2024-08-17T17:29:06.699870+0000","last_peered":"2024-08-17T17:29:06.699870+0000","last_clean":"2024-08-17T17:29:06.699870+0000","last_became_active":"2024-08-17T17:19:57.515142+0000","last_became_peered":"2024-08-17T17:19:57.515142+0000","last_unstale":"2024-08-17T17:29:06.699870+0000","last_undegraded":"2024-08-17T17:29:06.699870+0000","last_fullsized":"2024-08-17T17:29:06.699870+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:59:33.506176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":349,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.700114+0000","last_change":"2024-08-17T17:24:06.587903+0000","last_active":"2024-08-17T17:29:06.700114+0000","last_peered":"2024-08-17T17:29:06.700114+0000","last_clean":"2024-08-17T17:29:06.700114+0000","last_became_active":"2024-08-17T17:24:06.587503+0000","last_became_peered":"2024-08-17T17:24:06.587503+0000","last_unstale":"2024-08-17T17:29:06.700114+0000","last_undegraded":"2024-08-17T17:29:06.700114+0000","last_fullsized":"2024-08-17T17:29:06.700114+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:05:58.591992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"233'3165","reported_seq":3180,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:08.949061+0000","last_change":"2024-08-17T17:28:27.511940+0000","last_active":"2024-08-17T17:29:08.949061+0000","last_peered":"2024-08-17T17:29:08.949061+0000","last_clean":"2024-08-17T17:29:08.949061+0000","last_became_active":"2024-08-17T17:28:27.511636+0000","last_became_peered":"2024-08-17T17:28:27.511636+0000","last_unstale":"2024-08-17T17:29:08.949061+0000","last_undegraded":"2024-08-17T17:29:08.949061+0000","last_fullsized":"2024-08-17T17:29:08.949061+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":3165,"log_dups_size":0,"ondisk_log_size":3165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:51:38.705815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207421440,"num_objects":3165,"num_object_clones":0,"num_object_copies":9495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3165,"num_write_kb":202560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":1143,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.621235+0000","last_change":"2024-08-17T17:20:00.433870+0000","last_active":"2024-08-17T17:28:42.621235+0000","last_peered":"2024-08-17T17:28:42.621235+0000","last_clean":"2024-08-17T17:28:42.621235+0000","last_became_active":"2024-08-17T17:20:00.433386+0000","last_became_peered":"2024-08-17T17:20:00.433386+0000","last_unstale":"2024-08-17T17:28:42.621235+0000","last_undegraded":"2024-08-17T17:28:42.621235+0000","last_fullsized":"2024-08-17T17:28:42.621235+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:01:11.848862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1392,"num_read_kb":2863,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":245,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.691575+0000","last_change":"2024-08-17T17:20:49.272407+0000","last_active":"2024-08-17T17:29:06.691575+0000","last_peered":"2024-08-17T17:29:06.691575+0000","last_clean":"2024-08-17T17:29:06.691575+0000","last_became_active":"2024-08-17T17:20:49.271749+0000","last_became_peered":"2024-08-17T17:20:49.271749+0000","last_unstale":"2024-08-17T17:29:06.691575+0000","last_undegraded":"2024-08-17T17:29:06.691575+0000","last_fullsized":"2024-08-17T17:29:06.691575+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:36:01.443457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":316,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.700805+0000","last_change":"2024-08-17T17:17:32.499715+0000","last_active":"2024-08-17T17:29:06.700805+0000","last_peered":"2024-08-17T17:29:06.700805+0000","last_clean":"2024-08-17T17:29:06.700805+0000","last_became_active":"2024-08-17T17:17:32.499412+0000","last_became_peered":"2024-08-17T17:17:32.499412+0000","last_unstale":"2024-08-17T17:29:06.700805+0000","last_undegraded":"2024-08-17T17:29:06.700805+0000","last_fullsized":"2024-08-17T17:29:06.700805+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:39:38.456970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"233'3492","reported_seq":3517,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:12.101918+0000","last_change":"2024-08-17T17:28:27.510443+0000","last_active":"2024-08-17T17:29:12.101918+0000","last_peered":"2024-08-17T17:29:12.101918+0000","last_clean":"2024-08-17T17:29:12.101918+0000","last_became_active":"2024-08-17T17:28:27.510074+0000","last_became_peered":"2024-08-17T17:28:27.510074+0000","last_unstale":"2024-08-17T17:29:12.101918+0000","last_undegraded":"2024-08-17T17:29:12.101918+0000","last_fullsized":"2024-08-17T17:29:12.101918+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":3492,"log_dups_size":0,"ondisk_log_size":3492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:09:30.294320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228851712,"num_objects":3492,"num_object_clones":0,"num_object_copies":10476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3492,"num_write_kb":223488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":208,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.621420+0000","last_change":"2024-08-17T17:17:32.495728+0000","last_active":"2024-08-17T17:28:42.621420+0000","last_peered":"2024-08-17T17:28:42.621420+0000","last_clean":"2024-08-17T17:28:42.621420+0000","last_became_active":"2024-08-17T17:17:32.495357+0000","last_became_peered":"2024-08-17T17:17:32.495357+0000","last_unstale":"2024-08-17T17:28:42.621420+0000","last_undegraded":"2024-08-17T17:28:42.621420+0000","last_fullsized":"2024-08-17T17:28:42.621420+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:27:45.561567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":552,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.701032+0000","last_change":"2024-08-17T17:16:26.964621+0000","last_active":"2024-08-17T17:29:06.701032+0000","last_peered":"2024-08-17T17:29:06.701032+0000","last_clean":"2024-08-17T17:29:06.701032+0000","last_became_active":"2024-08-17T17:16:26.964058+0000","last_became_peered":"2024-08-17T17:16:26.964058+0000","last_unstale":"2024-08-17T17:29:06.701032+0000","last_undegraded":"2024-08-17T17:29:06.701032+0000","last_fullsized":"2024-08-17T17:29:06.701032+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:00:04.465644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":322,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.700308+0000","last_change":"2024-08-17T17:24:53.465550+0000","last_active":"2024-08-17T17:29:06.700308+0000","last_peered":"2024-08-17T17:29:06.700308+0000","last_clean":"2024-08-17T17:29:06.700308+0000","last_became_active":"2024-08-17T17:24:53.465228+0000","last_became_peered":"2024-08-17T17:24:53.465228+0000","last_unstale":"2024-08-17T17:29:06.700308+0000","last_undegraded":"2024-08-17T17:29:06.700308+0000","last_fullsized":"2024-08-17T17:29:06.700308+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:21:18.378359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"233'3659","reported_seq":3674,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:13.871139+0000","last_change":"2024-08-17T17:28:27.509315+0000","last_active":"2024-08-17T17:29:13.871139+0000","last_peered":"2024-08-17T17:29:13.871139+0000","last_clean":"2024-08-17T17:29:13.871139+0000","last_became_active":"2024-08-17T17:28:27.509018+0000","last_became_peered":"2024-08-17T17:28:27.509018+0000","last_unstale":"2024-08-17T17:29:13.871139+0000","last_undegraded":"2024-08-17T17:29:13.871139+0000","last_fullsized":"2024-08-17T17:29:13.871139+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":3659,"log_dups_size":0,"ondisk_log_size":3659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:26:22.379668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239796224,"num_objects":3659,"num_object_clones":0,"num_object_copies":10977,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3659,"num_write_kb":234176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":347,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.700943+0000","last_change":"2024-08-17T17:24:53.464307+0000","last_active":"2024-08-17T17:29:06.700943+0000","last_peered":"2024-08-17T17:29:06.700943+0000","last_clean":"2024-08-17T17:29:06.700943+0000","last_became_active":"2024-08-17T17:24:53.463996+0000","last_became_peered":"2024-08-17T17:24:53.463996+0000","last_unstale":"2024-08-17T17:29:06.700943+0000","last_undegraded":"2024-08-17T17:29:06.700943+0000","last_fullsized":"2024-08-17T17:29:06.700943+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:04:18.993725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":660,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.606312+0000","last_change":"2024-08-17T17:24:09.599822+0000","last_active":"2024-08-17T17:28:42.606312+0000","last_peered":"2024-08-17T17:28:42.606312+0000","last_clean":"2024-08-17T17:28:42.606312+0000","last_became_active":"2024-08-17T17:24:09.599189+0000","last_became_peered":"2024-08-17T17:24:09.599189+0000","last_unstale":"2024-08-17T17:28:42.606312+0000","last_undegraded":"2024-08-17T17:28:42.606312+0000","last_fullsized":"2024-08-17T17:28:42.606312+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:04.465604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":306,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.691623+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:29:06.691623+0000","last_peered":"2024-08-17T17:29:06.691623+0000","last_clean":"2024-08-17T17:29:06.691623+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:29:06.691623+0000","last_undegraded":"2024-08-17T17:29:06.691623+0000","last_fullsized":"2024-08-17T17:29:06.691623+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"233'3585","reported_seq":3610,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:12.107190+0000","last_change":"2024-08-17T17:28:27.509838+0000","last_active":"2024-08-17T17:29:12.107190+0000","last_peered":"2024-08-17T17:29:12.107190+0000","last_clean":"2024-08-17T17:29:12.107190+0000","last_became_active":"2024-08-17T17:28:27.509400+0000","last_became_peered":"2024-08-17T17:28:27.509400+0000","last_unstale":"2024-08-17T17:29:12.107190+0000","last_undegraded":"2024-08-17T17:29:12.107190+0000","last_fullsized":"2024-08-17T17:29:12.107190+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":3585,"log_dups_size":0,"ondisk_log_size":3585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:57:28.933136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234946560,"num_objects":3585,"num_object_clones":0,"num_object_copies":10755,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3585,"num_write_kb":229440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":341,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.689038+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:29:06.689038+0000","last_peered":"2024-08-17T17:29:06.689038+0000","last_clean":"2024-08-17T17:29:06.689038+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:29:06.689038+0000","last_undegraded":"2024-08-17T17:29:06.689038+0000","last_fullsized":"2024-08-17T17:29:06.689038+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":860,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.700465+0000","last_change":"2024-08-17T17:23:19.633786+0000","last_active":"2024-08-17T17:29:06.700465+0000","last_peered":"2024-08-17T17:29:06.700465+0000","last_clean":"2024-08-17T17:29:06.700465+0000","last_became_active":"2024-08-17T17:23:19.633230+0000","last_became_peered":"2024-08-17T17:23:19.633230+0000","last_unstale":"2024-08-17T17:29:06.700465+0000","last_undegraded":"2024-08-17T17:29:06.700465+0000","last_fullsized":"2024-08-17T17:29:06.700465+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:30:50.377391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":337,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.693853+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:29:06.693853+0000","last_peered":"2024-08-17T17:29:06.693853+0000","last_clean":"2024-08-17T17:29:06.693853+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:29:06.693853+0000","last_undegraded":"2024-08-17T17:29:06.693853+0000","last_fullsized":"2024-08-17T17:29:06.693853+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"233'3451","reported_seq":3475,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:12.110716+0000","last_change":"2024-08-17T17:28:27.509972+0000","last_active":"2024-08-17T17:29:12.110716+0000","last_peered":"2024-08-17T17:29:12.110716+0000","last_clean":"2024-08-17T17:29:12.110716+0000","last_became_active":"2024-08-17T17:28:27.509668+0000","last_became_peered":"2024-08-17T17:28:27.509668+0000","last_unstale":"2024-08-17T17:29:12.110716+0000","last_undegraded":"2024-08-17T17:29:12.110716+0000","last_fullsized":"2024-08-17T17:29:12.110716+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":3451,"log_dups_size":0,"ondisk_log_size":3451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:54:43.175100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226164736,"num_objects":3451,"num_object_clones":0,"num_object_copies":10353,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3451,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3451,"num_write_kb":220864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":136,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.695864+0000","last_change":"2024-08-17T17:24:06.593660+0000","last_active":"2024-08-17T17:29:06.695864+0000","last_peered":"2024-08-17T17:29:06.695864+0000","last_clean":"2024-08-17T17:29:06.695864+0000","last_became_active":"2024-08-17T17:24:06.586215+0000","last_became_peered":"2024-08-17T17:24:06.586215+0000","last_unstale":"2024-08-17T17:29:06.695864+0000","last_undegraded":"2024-08-17T17:29:06.695864+0000","last_fullsized":"2024-08-17T17:29:06.695864+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:15:49.475395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":464,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.615404+0000","last_change":"2024-08-17T17:26:06.855307+0000","last_active":"2024-08-17T17:28:42.615404+0000","last_peered":"2024-08-17T17:28:42.615404+0000","last_clean":"2024-08-17T17:28:42.615404+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:28:42.615404+0000","last_undegraded":"2024-08-17T17:28:42.615404+0000","last_fullsized":"2024-08-17T17:28:42.615404+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":132,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.620981+0000","last_change":"2024-08-17T17:18:59.417183+0000","last_active":"2024-08-17T17:28:42.620981+0000","last_peered":"2024-08-17T17:28:42.620981+0000","last_clean":"2024-08-17T17:28:42.620981+0000","last_became_active":"2024-08-17T17:18:59.416843+0000","last_became_peered":"2024-08-17T17:18:59.416843+0000","last_unstale":"2024-08-17T17:28:42.620981+0000","last_undegraded":"2024-08-17T17:28:42.620981+0000","last_fullsized":"2024-08-17T17:28:42.620981+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:29:09.336335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":153,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.614836+0000","last_change":"2024-08-17T17:20:49.258048+0000","last_active":"2024-08-17T17:28:42.614836+0000","last_peered":"2024-08-17T17:28:42.614836+0000","last_clean":"2024-08-17T17:28:42.614836+0000","last_became_active":"2024-08-17T17:20:49.257672+0000","last_became_peered":"2024-08-17T17:20:49.257672+0000","last_unstale":"2024-08-17T17:28:42.614836+0000","last_undegraded":"2024-08-17T17:28:42.614836+0000","last_fullsized":"2024-08-17T17:28:42.614836+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:08:11.734316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":192,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.605982+0000","last_change":"2024-08-17T17:18:59.430969+0000","last_active":"2024-08-17T17:28:42.605982+0000","last_peered":"2024-08-17T17:28:42.605982+0000","last_clean":"2024-08-17T17:28:42.605982+0000","last_became_active":"2024-08-17T17:18:59.413475+0000","last_became_peered":"2024-08-17T17:18:59.413475+0000","last_unstale":"2024-08-17T17:28:42.605982+0000","last_undegraded":"2024-08-17T17:28:42.605982+0000","last_fullsized":"2024-08-17T17:28:42.605982+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:36:33.732645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":332,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.700072+0000","last_change":"2024-08-17T17:25:04.506413+0000","last_active":"2024-08-17T17:29:06.700072+0000","last_peered":"2024-08-17T17:29:06.700072+0000","last_clean":"2024-08-17T17:29:06.700072+0000","last_became_active":"2024-08-17T17:25:04.506104+0000","last_became_peered":"2024-08-17T17:25:04.506104+0000","last_unstale":"2024-08-17T17:29:06.700072+0000","last_undegraded":"2024-08-17T17:29:06.700072+0000","last_fullsized":"2024-08-17T17:29:06.700072+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:41:53.556223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":135,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.606255+0000","last_change":"2024-08-17T17:20:49.279458+0000","last_active":"2024-08-17T17:28:42.606255+0000","last_peered":"2024-08-17T17:28:42.606255+0000","last_clean":"2024-08-17T17:28:42.606255+0000","last_became_active":"2024-08-17T17:20:49.260790+0000","last_became_peered":"2024-08-17T17:20:49.260790+0000","last_unstale":"2024-08-17T17:28:42.606255+0000","last_undegraded":"2024-08-17T17:28:42.606255+0000","last_fullsized":"2024-08-17T17:28:42.606255+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:58.092945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":49,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.621164+0000","last_change":"2024-08-17T17:24:53.463713+0000","last_active":"2024-08-17T17:28:42.621164+0000","last_peered":"2024-08-17T17:28:42.621164+0000","last_clean":"2024-08-17T17:28:42.621164+0000","last_became_active":"2024-08-17T17:24:53.463419+0000","last_became_peered":"2024-08-17T17:24:53.463419+0000","last_unstale":"2024-08-17T17:28:42.621164+0000","last_undegraded":"2024-08-17T17:28:42.621164+0000","last_fullsized":"2024-08-17T17:28:42.621164+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:34:44.545175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":288,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.691049+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:29:06.691049+0000","last_peered":"2024-08-17T17:29:06.691049+0000","last_clean":"2024-08-17T17:29:06.691049+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:29:06.691049+0000","last_undegraded":"2024-08-17T17:29:06.691049+0000","last_fullsized":"2024-08-17T17:29:06.691049+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":315,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.690281+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:29:06.690281+0000","last_peered":"2024-08-17T17:29:06.690281+0000","last_clean":"2024-08-17T17:29:06.690281+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:29:06.690281+0000","last_undegraded":"2024-08-17T17:29:06.690281+0000","last_fullsized":"2024-08-17T17:29:06.690281+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":335,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.694779+0000","last_change":"2024-08-17T17:25:04.509422+0000","last_active":"2024-08-17T17:29:06.694779+0000","last_peered":"2024-08-17T17:29:06.694779+0000","last_clean":"2024-08-17T17:29:06.694779+0000","last_became_active":"2024-08-17T17:25:04.508742+0000","last_became_peered":"2024-08-17T17:25:04.508742+0000","last_unstale":"2024-08-17T17:29:06.694779+0000","last_undegraded":"2024-08-17T17:29:06.694779+0000","last_fullsized":"2024-08-17T17:29:06.694779+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:18:31.976907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":190,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.615330+0000","last_change":"2024-08-17T17:16:26.963696+0000","last_active":"2024-08-17T17:28:42.615330+0000","last_peered":"2024-08-17T17:28:42.615330+0000","last_clean":"2024-08-17T17:28:42.615330+0000","last_became_active":"2024-08-17T17:16:26.963280+0000","last_became_peered":"2024-08-17T17:16:26.963280+0000","last_unstale":"2024-08-17T17:28:42.615330+0000","last_undegraded":"2024-08-17T17:28:42.615330+0000","last_fullsized":"2024-08-17T17:28:42.615330+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:22:01.474840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":352,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.694866+0000","last_change":"2024-08-17T17:24:06.586886+0000","last_active":"2024-08-17T17:29:06.694866+0000","last_peered":"2024-08-17T17:29:06.694866+0000","last_clean":"2024-08-17T17:29:06.694866+0000","last_became_active":"2024-08-17T17:24:06.586516+0000","last_became_peered":"2024-08-17T17:24:06.586516+0000","last_unstale":"2024-08-17T17:29:06.694866+0000","last_undegraded":"2024-08-17T17:29:06.694866+0000","last_fullsized":"2024-08-17T17:29:06.694866+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:03:30.002726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":132,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.695682+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:29:06.695682+0000","last_peered":"2024-08-17T17:29:06.695682+0000","last_clean":"2024-08-17T17:29:06.695682+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:29:06.695682+0000","last_undegraded":"2024-08-17T17:29:06.695682+0000","last_fullsized":"2024-08-17T17:29:06.695682+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":312,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.700535+0000","last_change":"2024-08-17T17:18:59.412463+0000","last_active":"2024-08-17T17:29:06.700535+0000","last_peered":"2024-08-17T17:29:06.700535+0000","last_clean":"2024-08-17T17:29:06.700535+0000","last_became_active":"2024-08-17T17:18:59.412134+0000","last_became_peered":"2024-08-17T17:18:59.412134+0000","last_unstale":"2024-08-17T17:29:06.700535+0000","last_undegraded":"2024-08-17T17:29:06.700535+0000","last_fullsized":"2024-08-17T17:29:06.700535+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:44.463153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.621097+0000","last_change":"2024-08-17T17:28:41.399059+0000","last_active":"2024-08-17T17:28:42.621097+0000","last_peered":"2024-08-17T17:28:42.621097+0000","last_clean":"2024-08-17T17:28:42.621097+0000","last_became_active":"2024-08-17T17:28:41.398560+0000","last_became_peered":"2024-08-17T17:28:41.398560+0000","last_unstale":"2024-08-17T17:28:42.621097+0000","last_undegraded":"2024-08-17T17:28:42.621097+0000","last_fullsized":"2024-08-17T17:28:42.621097+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:31:45.097451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":252,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.700393+0000","last_change":"2024-08-17T17:24:06.591165+0000","last_active":"2024-08-17T17:29:06.700393+0000","last_peered":"2024-08-17T17:29:06.700393+0000","last_clean":"2024-08-17T17:29:06.700393+0000","last_became_active":"2024-08-17T17:24:06.590785+0000","last_became_peered":"2024-08-17T17:24:06.590785+0000","last_unstale":"2024-08-17T17:29:06.700393+0000","last_undegraded":"2024-08-17T17:29:06.700393+0000","last_fullsized":"2024-08-17T17:29:06.700393+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:54:05.775563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":324,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.701251+0000","last_change":"2024-08-17T17:24:32.300560+0000","last_active":"2024-08-17T17:29:06.701251+0000","last_peered":"2024-08-17T17:29:06.701251+0000","last_clean":"2024-08-17T17:29:06.701251+0000","last_became_active":"2024-08-17T17:24:32.300228+0000","last_became_peered":"2024-08-17T17:24:32.300228+0000","last_unstale":"2024-08-17T17:29:06.701251+0000","last_undegraded":"2024-08-17T17:29:06.701251+0000","last_fullsized":"2024-08-17T17:29:06.701251+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:18:05.475449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":132,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.691478+0000","last_change":"2024-08-17T17:24:06.586379+0000","last_active":"2024-08-17T17:29:06.691478+0000","last_peered":"2024-08-17T17:29:06.691478+0000","last_clean":"2024-08-17T17:29:06.691478+0000","last_became_active":"2024-08-17T17:24:06.586023+0000","last_became_peered":"2024-08-17T17:24:06.586023+0000","last_unstale":"2024-08-17T17:29:06.691478+0000","last_undegraded":"2024-08-17T17:29:06.691478+0000","last_fullsized":"2024-08-17T17:29:06.691478+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:45.701637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":342,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.694266+0000","last_change":"2024-08-17T17:21:50.442535+0000","last_active":"2024-08-17T17:29:06.694266+0000","last_peered":"2024-08-17T17:29:06.694266+0000","last_clean":"2024-08-17T17:29:06.694266+0000","last_became_active":"2024-08-17T17:21:50.441891+0000","last_became_peered":"2024-08-17T17:21:50.441891+0000","last_unstale":"2024-08-17T17:29:06.694266+0000","last_undegraded":"2024-08-17T17:29:06.694266+0000","last_fullsized":"2024-08-17T17:29:06.694266+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:46:24.025407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":245,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.621099+0000","last_change":"2024-08-17T17:25:04.508045+0000","last_active":"2024-08-17T17:28:42.621099+0000","last_peered":"2024-08-17T17:28:42.621099+0000","last_clean":"2024-08-17T17:28:42.621099+0000","last_became_active":"2024-08-17T17:25:04.505083+0000","last_became_peered":"2024-08-17T17:25:04.505083+0000","last_unstale":"2024-08-17T17:28:42.621099+0000","last_undegraded":"2024-08-17T17:28:42.621099+0000","last_fullsized":"2024-08-17T17:28:42.621099+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:23:42.993544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":328,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.689380+0000","last_change":"2024-08-17T17:18:59.418434+0000","last_active":"2024-08-17T17:29:06.689380+0000","last_peered":"2024-08-17T17:29:06.689380+0000","last_clean":"2024-08-17T17:29:06.689380+0000","last_became_active":"2024-08-17T17:18:59.418043+0000","last_became_peered":"2024-08-17T17:18:59.418043+0000","last_unstale":"2024-08-17T17:29:06.689380+0000","last_undegraded":"2024-08-17T17:29:06.689380+0000","last_fullsized":"2024-08-17T17:29:06.689380+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:25:51.371820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":218,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.621412+0000","last_change":"2024-08-17T17:20:49.291167+0000","last_active":"2024-08-17T17:28:42.621412+0000","last_peered":"2024-08-17T17:28:42.621412+0000","last_clean":"2024-08-17T17:28:42.621412+0000","last_became_active":"2024-08-17T17:20:49.263750+0000","last_became_peered":"2024-08-17T17:20:49.263750+0000","last_unstale":"2024-08-17T17:28:42.621412+0000","last_undegraded":"2024-08-17T17:28:42.621412+0000","last_fullsized":"2024-08-17T17:28:42.621412+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:02.863246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":341,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.693995+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:29:06.693995+0000","last_peered":"2024-08-17T17:29:06.693995+0000","last_clean":"2024-08-17T17:29:06.693995+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:29:06.693995+0000","last_undegraded":"2024-08-17T17:29:06.693995+0000","last_fullsized":"2024-08-17T17:29:06.693995+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":341,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.690873+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:29:06.690873+0000","last_peered":"2024-08-17T17:29:06.690873+0000","last_clean":"2024-08-17T17:29:06.690873+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:29:06.690873+0000","last_undegraded":"2024-08-17T17:29:06.690873+0000","last_fullsized":"2024-08-17T17:29:06.690873+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":45,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-08-17T17:28:42.614532+0000","last_change":"2024-08-17T17:25:04.503834+0000","last_active":"2024-08-17T17:28:42.614532+0000","last_peered":"2024-08-17T17:28:42.614532+0000","last_clean":"2024-08-17T17:28:42.614532+0000","last_became_active":"2024-08-17T17:25:04.502712+0000","last_became_peered":"2024-08-17T17:25:04.502712+0000","last_unstale":"2024-08-17T17:28:42.614532+0000","last_undegraded":"2024-08-17T17:28:42.614532+0000","last_fullsized":"2024-08-17T17:28:42.614532+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:45.389103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":151,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-08-17T17:29:06.690277+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:29:06.690277+0000","last_peered":"2024-08-17T17:29:06.690277+0000","last_clean":"2024-08-17T17:29:06.690277+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:29:06.690277+0000","last_undegraded":"2024-08-17T17:29:06.690277+0000","last_fullsized":"2024-08-17T17:29:06.690277+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3608281088,"num_objects":55058,"num_object_clones":0,"num_object_copies":165174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":55058,"num_write_kb":3523712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10771038208,"data_stored":10771038208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55058,"ondisk_log_size":55058,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":6304,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":16702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17971,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234268,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1789252,"kb_used_data":1479420,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91955900,"statfs":{"total":95995035648,"available":94162841600,"internally_reserved":0,"allocated":1514926080,"data_stored":1513595984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495903,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1769660,"kb_used_data":1459764,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91975492,"statfs":{"total":95995035648,"available":94182903808,"internally_reserved":0,"allocated":1494798336,"data_stored":1493476132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757541,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1225328,"kb_used_data":923816,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":92519824,"statfs":{"total":95995035648,"available":94740299776,"internally_reserved":0,"allocated":945987584,"data_stored":944673123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019179,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435406,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2590188,"kb_used_data":2211364,"kb_used_omap":4,"kb_used_meta":378811,"kb_avail":91154964,"statfs":{"total":95995035648,"available":93342683136,"internally_reserved":0,"allocated":2264436736,"data_stored":2263126145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":387902474},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":104,"seq":446676598932,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1660732,"kb_used_data":1275444,"kb_used_omap":7,"kb_used_meta":385272,"kb_avail":92084420,"statfs":{"total":95995035648,"available":94294446080,"internally_reserved":0,"allocated":1306054656,"data_stored":1304732947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":394518690},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762459,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1896752,"kb_used_data":1494376,"kb_used_omap":7,"kb_used_meta":402360,"kb_avail":91848400,"statfs":{"total":95995035648,"available":94052761600,"internally_reserved":0,"allocated":1530241024,"data_stored":1528926304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7945,"internal_metadata":412016887},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":94,"seq":403726925986,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2170808,"kb_used_data":1828592,"kb_used_omap":7,"kb_used_meta":342200,"kb_avail":91574344,"statfs":{"total":95995035648,"available":93772128256,"internally_reserved":0,"allocated":1872478208,"data_stored":1871156481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":350413121},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6241,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1873412096,"data_stored":1873412096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1551958016,"data_stored":1551958016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1308819456,"data_stored":1308819456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2300182528,"data_stored":2300182528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":952303616,"data_stored":952303616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1490157568,"data_stored":1490157568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1294204928,"data_stored":1294204928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:29:22.373 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-08-17T17:29:22.373 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:29:22.373 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-08-17T17:29:22.373 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-17T17:29:22.373 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-17T17:29:22.374 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-08-17T17:29:22.468 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:22.492 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:22.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:22.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:22.555 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 41 16 68471 68455 104.333 76.5 0.00935879 0.00957919 2024-08-17T17:29:22.594 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:22.639 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:22.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:22.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:22.741 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:22.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:22.793 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:22.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:22.894 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:22.919 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:22.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:22.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:23.021 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:23.049 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:23.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:23.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:23.151 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:23.179 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:23.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:23.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:23.281 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:23.309 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:23.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:23.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:23.411 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:23.437 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:23.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:23.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:23.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:23 smithi053 ceph-mon[89963]: pgmap v649: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-08-17T17:29:23.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:29:23.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:29:23.511 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:23 smithi003 ceph-mon[103344]: pgmap v649: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-08-17T17:29:23.511 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:29:23.511 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:29:23.511 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:23 smithi003 ceph-mon[100615]: pgmap v649: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-08-17T17:29:23.511 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:29:23.511 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:29:23.540 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:23.555 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 42 16 69641 69625 103.59 73.125 0.0129722 0.00964725 2024-08-17T17:29:23.564 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:29:23.565 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:23.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:23.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:23.667 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:23.696 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:23.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:23.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:23.798 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:23.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:23.825 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:23.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:23.926 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:23.975 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:23.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:23.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:24.078 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:24.107 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:24.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:24.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:24.208 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:24.234 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:24.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:24.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:24.336 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:24.363 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:24.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:24.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:24.464 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:24.505 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:24.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:24.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:24.555 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 43 16 70813 70797 102.885 73.25 0.00681487 0.00971431 2024-08-17T17:29:24.607 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:24.651 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:24.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:24.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:24.753 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:24.780 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:24.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:24.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:24.882 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:24.899 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:29:24.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:24.933 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:24.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:24.964 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:29:24.964 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:29:25.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:25.080 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:25.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:25.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:25.182 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:25.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:25.210 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:25.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:25.312 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:25.337 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:25.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:25.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:25.393 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:25 smithi003 ceph-mon[100615]: pgmap v650: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-08-17T17:29:25.395 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:25 smithi003 ceph-mon[103344]: pgmap v650: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-08-17T17:29:25.439 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:25.490 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:25.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:25.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:25.554 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:29:25.557 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 44 16 71944 71928 102.15 70.6875 0.011088 0.00978404 2024-08-17T17:29:25.592 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:25.652 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:29:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:29:25] "GET /metrics HTTP/1.1" 200 36499 "" "Prometheus/2.51.0" 2024-08-17T17:29:25.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:25.659 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:25.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:25.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:25 smithi053 ceph-mon[89963]: pgmap v650: 89 pgs: 89 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-08-17T17:29:25.760 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:25.786 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:25.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:25.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:25.888 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:25.914 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:25.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:25.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:26.016 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:26.038 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:26.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:26.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:26.139 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:26.171 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:26.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:26.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:26.273 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:26.295 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:26.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:26.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:26.397 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:26.424 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:26.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:26.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:26.526 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:26.553 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:26.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:26.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:26.557 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 45 16 72883 72867 101.184 58.6875 0.0100069 0.00987753 2024-08-17T17:29:26.655 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:26.692 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:26.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:26.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:26.794 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:26.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:26.838 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:26.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:26.941 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:26.969 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:26.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:26.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:27.071 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:27.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:27.098 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:27.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:27.199 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:27.241 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:27.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:27.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:27.343 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:27.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:27.370 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:27.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:27.471 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:27.483 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:27 smithi003 ceph-mon[100615]: pgmap v651: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-08-17T17:29:27.484 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:27 smithi003 ceph-mon[103344]: pgmap v651: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-08-17T17:29:27.485 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-17T17:29:27.495 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:27.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:27.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:27.557 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 46 16 74051 74035 100.571 73 0.0114589 0.00993701 2024-08-17T17:29:27.596 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:27.622 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:27.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:27.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:27.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:27 smithi053 ceph-mon[89963]: pgmap v651: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-08-17T17:29:27.724 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:27.782 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:27.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:27.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:27.883 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:27.909 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:27.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:27.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:28.010 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:28.036 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:28.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:28.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:28.138 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:28.165 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:28.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:28.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:28.170 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:29:28.170 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:29:28.170 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:29:28.170 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:29:28.171 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:29:28.171 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:29:28.171 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:29:28.171 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:29:28.171 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:29:28.171 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:29:28.171 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:29:28.171 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:29:28.171 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:29:28.171 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:29:28.171 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:29:28.171 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:29:28.267 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:28.295 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:28.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:28.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:28.396 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:28.449 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:28.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:28.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:28.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:28 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3755685217' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:29:28.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:28 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3755685217' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:29:28.552 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:28.557 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 47 16 75099 75083 99.8252 65.5 0.00724029 0.0100127 2024-08-17T17:29:28.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:28.582 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:28.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:28.683 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:28.709 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:28.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:28.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:28.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:28 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3755685217' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:29:28.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:28.857 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:28.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:28.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:28.959 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:29.007 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:29.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:29.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:29.109 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:29.135 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:29.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:29.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:29.237 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:29.282 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:29.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:29.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:29.384 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:29.427 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:29.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:29.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:29.479 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:29:29.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:29 smithi003 ceph-mon[100615]: pgmap v652: 89 pgs: 89 active+clean; 4.3 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 67 MiB/s wr, 1.06k op/s 2024-08-17T17:29:29.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:29 smithi003 ceph-mon[103344]: pgmap v652: 89 pgs: 89 active+clean; 4.3 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 67 MiB/s wr, 1.06k op/s 2024-08-17T17:29:29.530 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:29.559 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 48 16 76149 76133 99.1127 65.625 0.038132 0.0100833 2024-08-17T17:29:29.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:29.593 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:29.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:29.693 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:29:29.693 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:29.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:29 smithi053 ceph-mon[89963]: pgmap v652: 89 pgs: 89 active+clean; 4.3 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 67 MiB/s wr, 1.06k op/s 2024-08-17T17:29:29.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:29.753 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:29.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:29.854 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:29.881 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:29.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:29.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:29.982 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:30.009 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:30.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:30.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:30.111 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:30.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:30.170 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:30.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:30.271 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:30.320 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:30.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:30.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:30.422 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:30.448 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:30.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:30.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:30.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:29:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:29:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:29:30.551 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:30.557 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 49 16 77219 77203 98.4544 66.875 0.00622168 0.0101505 2024-08-17T17:29:30.593 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:30.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:30.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:30.695 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:30.732 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:30.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:30.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:30.834 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:30.876 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:30.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:30.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:30.978 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:31.002 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:29:31.017 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:31.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:31.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:31.119 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:31.170 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:31.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:31.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:31.271 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:31.297 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:31.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:31.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:31.399 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:31.425 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:31.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:31.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:31.527 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:31.554 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:31.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:31.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:31.557 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 50 16 78237 78221 97.7578 63.625 0.0126607 0.0102235 2024-08-17T17:29:31.620 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:29:31.620 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-17T17:29:31.621 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:29:31.655 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:31.712 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:31.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:31.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:31.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:31 smithi003 ceph-mon[103344]: pgmap v653: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-08-17T17:29:31.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:31 smithi003 ceph-mon[100615]: pgmap v653: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-08-17T17:29:31.814 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:31.842 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:31.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:31.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:31.944 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:31.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:31 smithi053 ceph-mon[89963]: pgmap v653: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-08-17T17:29:31.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:31.989 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:31.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:32.090 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:32.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:32.117 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:32.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:32.218 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:32.257 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:32.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:32.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:32.360 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:32.409 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:32.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:32.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:32.511 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:32.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:32.535 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:32.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:32.557 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 51 16 79211 79195 97.0345 60.875 0.0102644 0.0102853 2024-08-17T17:29:32.636 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:32.679 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:32.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:32.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:32.781 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:32.823 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:32.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:32.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:32.925 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:32.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:32.973 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:32.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:33.075 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:33.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:33.099 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:33.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:33.200 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:33.228 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:33.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:33.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:33.330 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:33.356 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:33.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:33.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:33.457 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:33.479 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:33.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:33.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:33.558 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 52 16 80123 80107 96.2646 57 0.00318419 0.0103506 2024-08-17T17:29:33.581 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:33.611 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:33.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:33.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:33.712 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:33.739 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:33.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:33.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:33.841 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:33.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:33.868 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:33.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:33.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:33 smithi053 ceph-mon[89963]: pgmap v654: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 61 MiB/s wr, 972 op/s 2024-08-17T17:29:33.969 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:33.993 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:33.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:33.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:34.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:33 smithi003 ceph-mon[100615]: pgmap v654: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 61 MiB/s wr, 972 op/s 2024-08-17T17:29:34.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:33 smithi003 ceph-mon[103344]: pgmap v654: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 61 MiB/s wr, 972 op/s 2024-08-17T17:29:34.095 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:34.142 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:34.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:34.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:34.244 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:34.288 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:34.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:34.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:34.390 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:34.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:34.438 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:34.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:34.539 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:34.559 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 53 16 81150 81134 95.6556 64.1875 0.0102896 0.0104474 2024-08-17T17:29:34.578 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:34.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:34.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:34.681 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:34.709 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:34.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:34.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:34.811 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:34.839 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:34.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:34.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:34.941 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:34.986 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:34.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:34.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:35.089 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:35.116 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:35.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:35.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:35.218 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:35.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:35.248 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:35.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:35.349 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:35.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:35.374 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:35.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:35.475 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:35.502 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:35.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:35.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:35.560 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 54 16 82231 82215 95.1353 67.5625 0.00571211 0.0105054 2024-08-17T17:29:35.603 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:35.629 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:35.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:35.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:35.731 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:35.773 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:35.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:35.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:35.875 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:35.902 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:35.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:35.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:35.922 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:35 smithi003 ceph-mon[100615]: pgmap v655: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-08-17T17:29:35.922 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:29:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:29:35] "GET /metrics HTTP/1.1" 200 36499 "" "Prometheus/2.51.0" 2024-08-17T17:29:35.922 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:35 smithi003 ceph-mon[103344]: pgmap v655: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-08-17T17:29:35.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:35 smithi053 ceph-mon[89963]: pgmap v655: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-08-17T17:29:36.004 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:36.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:36.148 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:36.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:36.249 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:36.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:36.290 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:36.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:36.391 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:36.427 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:36.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:36.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:36.520 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:29:36.520 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-08-17T17:29:36.520 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-08-17T17:29:36.520 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-08-17T17:29:36.520 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:29:36.529 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:36.556 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:36.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:36.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:36.560 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 55 16 83247 83231 94.5601 63.5 0.0113882 0.0105687 2024-08-17T17:29:36.657 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:36.700 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:36.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:36.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:36.802 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:36.844 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:36.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:36.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:36.946 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:36.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:36.985 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:36.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:37.086 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:37.117 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:37.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:37.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:37.219 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:37.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:37.255 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:37.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:37.356 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:37.383 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:37.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:37.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:37.485 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:37.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:37.551 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:37.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:37.553 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-17T17:29:37.560 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 56 16 84307 84291 94.0545 66.25 0.0156727 0.0106262 2024-08-17T17:29:37.652 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:37.676 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:37.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:37.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:37.693 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:29:37.778 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:37.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:37.855 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:37.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:37.956 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:37.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:37 smithi053 ceph-mon[89963]: pgmap v656: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-08-17T17:29:38.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:38.011 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:38.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:38.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:37 smithi003 ceph-mon[100615]: pgmap v656: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-08-17T17:29:38.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:37 smithi003 ceph-mon[103344]: pgmap v656: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-08-17T17:29:38.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:38.167 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:38.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:38.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:38.269 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:38.292 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:38.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:38.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:38.394 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:38.436 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:38.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:38.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:38.537 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:38.560 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 57 16 85182 85166 93.3638 54.6875 0.0103765 0.0107055 2024-08-17T17:29:38.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:38.576 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:38.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:38.677 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:38.706 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:38.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:38.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:38.708 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-08-17T17:29:38.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:38 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/159892236' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:29:38.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:38 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:29:38.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:29:38.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:38 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/159892236' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:29:38.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:38 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:29:38.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:29:38.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:38.846 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:38.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:38.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:38.948 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:38.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:38 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/159892236' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:29:38.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:38 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:29:38.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:29:38.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:38.991 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:38.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:39.092 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:39.132 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:39.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:39.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:39.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:39.255 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:39.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:39.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:39.357 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:39.385 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:39.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:39.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:39.487 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:39.525 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:39.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:39.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:39.560 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 58 16 86055 86039 92.6948 54.5625 0.00444201 0.0107828 2024-08-17T17:29:39.627 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:39.650 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:39.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:39.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:39.752 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:39.779 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:39.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:39.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:39.880 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:39.907 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:39.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:39.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:39.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:39 smithi053 ceph-mon[89963]: pgmap v657: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 62 MiB/s wr, 989 op/s 2024-08-17T17:29:39.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:39 smithi053 ceph-mon[89963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:29:39.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:39 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:29:39.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:39 smithi053 ceph-mon[89963]: osdmap e235: 8 total, 8 up, 7 in 2024-08-17T17:29:40.009 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:40.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:39 smithi003 ceph-mon[103344]: pgmap v657: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 62 MiB/s wr, 989 op/s 2024-08-17T17:29:40.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:39 smithi003 ceph-mon[103344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:29:40.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:39 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:29:40.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:39 smithi003 ceph-mon[103344]: osdmap e235: 8 total, 8 up, 7 in 2024-08-17T17:29:40.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:39 smithi003 ceph-mon[100615]: pgmap v657: 89 pgs: 89 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 62 MiB/s wr, 989 op/s 2024-08-17T17:29:40.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:39 smithi003 ceph-mon[100615]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:29:40.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:39 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:29:40.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:39 smithi003 ceph-mon[100615]: osdmap e235: 8 total, 8 up, 7 in 2024-08-17T17:29:40.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:40.056 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:40.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:40.157 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:40.185 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:40.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:40.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:40.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:40.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:40.322 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:40.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:40.423 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:40.450 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:40.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:40.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:40.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:29:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:29:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:29:40.551 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:40.561 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 59 16 87041 87025 92.1674 61.625 0.00883295 0.0108434 2024-08-17T17:29:40.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:40.594 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:40.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:40.695 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:40.735 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:40.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:40.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:40.825 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-17T17:29:40.825 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 0 1 2024-08-17T17:29:40.837 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:40.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:40.868 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:40.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:40.969 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:41.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:41.021 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:41.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:41.122 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:41.177 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:41.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:41.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:41.279 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:41.339 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:41.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:41.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:41.441 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:41.470 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:41.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:41.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:41.562 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-08-17T17:29:41.562829+0000 min lat: 0.00130565 max lat: 0.2474 avg lat: 0.0109156 2024-08-17T17:29:41.563 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:29:41.563 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 60 16 87931 87915 91.556 55.625 0.00449646 0.0109156 2024-08-17T17:29:41.571 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:41.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:41.607 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:41.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:41.708 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:41.747 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:41.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:41.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:41.849 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:41.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:41.905 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:41.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:41.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:41 smithi053 ceph-mon[89963]: pgmap v659: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-08-17T17:29:41.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:41 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:29:41.970 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:41 smithi003 ceph-mon[103344]: pgmap v659: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-08-17T17:29:41.970 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:41 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:29:41.970 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:41 smithi003 ceph-mon[100615]: pgmap v659: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-08-17T17:29:41.970 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:41 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:29:42.006 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:42.055 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:42.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:42.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:42.157 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:42.203 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:42.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:42.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:42.305 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:42.339 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:42.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:42.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:42.441 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:42.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:42.472 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:42.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:42.564 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 61 16 88932 88916 91.0791 62.5625 0.0152835 0.0109737 2024-08-17T17:29:42.573 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:42.601 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:42.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:42.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:42.702 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:42.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:42.764 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:42.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:42.866 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:42.918 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:42.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:42.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:42.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:42 smithi053 ceph-mon[89963]: pgmap v660: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-08-17T17:29:42.972 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:42 smithi003 ceph-mon[100615]: pgmap v660: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-08-17T17:29:42.972 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:42 smithi003 ceph-mon[103344]: pgmap v660: 89 pgs: 89 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-08-17T17:29:42.984 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:29:42.988 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:29:43.020 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:43.045 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:43.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:43.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:43.072 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:29:43.072 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:29:43.108 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:29:43.109 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:29:43.146 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:43.154 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:29:43.154 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:29:43.178 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:43.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:43.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:43.189 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:29:43.189 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:29:43.221 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:29:43.222 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:29:43.254 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:29:43.254 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:29:43.280 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:43.288 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:29:43.288 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:29:43.321 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:29:43.321 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:29:43.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:43.333 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:43.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:43.434 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:43.474 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:43.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:43.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:43.564 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 62 16 89823 89807 90.5083 55.6875 0.0118169 0.0110433 2024-08-17T17:29:43.576 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:43.605 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:43.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:43.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:43.706 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:43.736 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:43.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:43.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:43.838 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:43.866 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:43.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:43.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:43.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:43 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/706144242' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:29:43.968 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:43.996 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:43.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:43.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:44.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:43 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/706144242' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:29:44.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:43 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/706144242' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:29:44.097 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:44.124 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:44.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:44.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:44.225 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:44.260 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:44.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:44.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:44.362 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:44.399 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:44.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:44.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:44.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:29:44.504 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:44.529 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:44.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:44.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:44.564 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 63 16 90705 90689 89.9466 55.125 0.0134141 0.011111 2024-08-17T17:29:44.630 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:44.655 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:44.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:44.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:44.757 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:44.779 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:44.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:44.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:44.880 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:44.907 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:44.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:44.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:45.009 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:45.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:44 smithi003 ceph-mon[100615]: pgmap v661: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 61 MiB/s wr, 976 op/s 2024-08-17T17:29:45.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:44 smithi003 ceph-mon[103344]: pgmap v661: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 61 MiB/s wr, 976 op/s 2024-08-17T17:29:45.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:45.040 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:45.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:45.142 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:45.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:45.178 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:45.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:45.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:44 smithi053 ceph-mon[89963]: pgmap v661: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 61 MiB/s wr, 976 op/s 2024-08-17T17:29:45.279 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:45.306 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:45.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:45.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:45.408 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:45.432 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:45.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:45.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:45.534 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:45.556 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:45.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:45.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:45.564 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 64 16 91382 91366 89.2024 42.3125 0.00320857 0.0111903 2024-08-17T17:29:45.658 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:45.681 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:45.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:45.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:45.722 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:29:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:29:45] "GET /metrics HTTP/1.1" 200 36502 "" "Prometheus/2.51.0" 2024-08-17T17:29:45.783 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:45.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:45.824 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:45.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:45.925 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:45.958 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:45.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:45.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:46.059 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:46.088 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:46.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:46.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:46.190 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:46.221 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:46.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:46.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:46.322 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:46.351 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:46.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:46.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:46.453 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:46.479 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:46.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:46.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:46.564 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 65 16 92197 92181 88.6137 50.9375 0.0140813 0.0112788 2024-08-17T17:29:46.580 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:46.606 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:46.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:46.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:46.708 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:46.733 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:46.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:46.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:46.835 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:46.862 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:46.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:46.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:46.963 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:47.009 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:47.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:47.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:47.048 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:46 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:29:47.048 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:46 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:29:47.048 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:46 smithi003 ceph-mon[100615]: pgmap v662: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 60 MiB/s wr, 960 op/s 2024-08-17T17:29:47.048 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:46 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:29:47.048 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:46 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:29:47.048 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:46 smithi003 ceph-mon[103344]: pgmap v662: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 60 MiB/s wr, 960 op/s 2024-08-17T17:29:47.111 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:47.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:47.139 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:47.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:47.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:46 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:29:47.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:46 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:29:47.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:46 smithi053 ceph-mon[89963]: pgmap v662: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 60 MiB/s wr, 960 op/s 2024-08-17T17:29:47.240 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:47.297 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:47.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:47.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:47.399 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:47.442 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:47.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:47.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:47.544 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:47.564 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 66 16 93081 93065 88.1082 55.25 0.0265645 0.0113441 2024-08-17T17:29:47.568 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:47.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:47.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:47.670 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:47.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:47.708 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:47.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:47.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:47.852 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:47.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:47.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:47.954 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:47.979 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:47.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:47.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:48.080 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:48.093 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:29:48.093 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:29:48.126 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:48.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:48.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:48.228 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:48.255 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:48.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:48.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:48.357 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:48.394 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:48.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:48.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:48.496 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:48.524 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:48.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:48.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:48.542 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:29:48.565 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 67 16 93940 93924 87.5941 53.6875 0.0242418 0.0114106 2024-08-17T17:29:48.626 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:48.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:48.682 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:48.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:48.783 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:48.810 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:48.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:48.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:48.912 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:48.958 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:48.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:48.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:49.062 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:49.087 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:49.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:49.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:49.189 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:49.234 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:49.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:49.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:49.269 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:49 smithi003 ceph-mon[100615]: pgmap v663: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 59 MiB/s wr, 937 op/s 2024-08-17T17:29:49.270 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:49 smithi003 ceph-mon[103344]: pgmap v663: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 59 MiB/s wr, 937 op/s 2024-08-17T17:29:49.336 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:49.363 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:49.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:49.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:49.465 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:49.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:49 smithi053 ceph-mon[89963]: pgmap v663: 89 pgs: 89 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 59 MiB/s wr, 937 op/s 2024-08-17T17:29:49.509 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:49.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:49.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:49.564 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 68 16 94840 94824 87.1332 56.25 0.012295 0.0114707 2024-08-17T17:29:49.610 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:49.637 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:49.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:49.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:49.654 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-17T17:29:49.739 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:49.782 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:49.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:49.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:49.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:49.949 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:49.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:49.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:50.051 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:50.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:50.101 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:50.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:50.202 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:50.250 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:50.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:50.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:50.352 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:50.399 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:50.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:50.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:50.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:29:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:29:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:29:50.501 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:50.542 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:50.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:50.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:50.565 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 69 16 95684 95668 86.6349 52.75 0.018393 0.0115368 2024-08-17T17:29:50.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:50.717 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:29:50.719 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:29:50.722 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:29:50.753 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:50.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:50.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:50.855 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:50.883 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:50.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:50.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:50.985 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:51.031 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:51.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:51.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:51.055 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.0 to 1 (10000) 2024-08-17T17:29:51.133 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:51.149 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:51 smithi003 ceph-mon[103344]: pgmap v664: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 61 MiB/s wr, 973 op/s 2024-08-17T17:29:51.149 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:51 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1572314825' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T17:29:51.149 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:51 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T17:29:51.151 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:51 smithi003 ceph-mon[100615]: pgmap v664: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 61 MiB/s wr, 973 op/s 2024-08-17T17:29:51.151 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:51 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1572314825' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T17:29:51.151 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:51 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T17:29:51.185 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:51.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:51.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:51.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:51.333 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:51.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:51.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:51.435 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:51.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:51 smithi053 ceph-mon[89963]: pgmap v664: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 61 MiB/s wr, 973 op/s 2024-08-17T17:29:51.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:51 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1572314825' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T17:29:51.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:51 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T17:29:51.477 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:51.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:51.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:51.565 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 70 16 96527 96511 86.1499 52.6875 0.0176355 0.0116026 2024-08-17T17:29:51.579 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:51.617 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:51.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:51.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:51.719 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:51.766 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:51.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:51.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:51.868 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:51.911 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:51.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:51.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:52.013 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:52.054 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:52.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:52.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:52.156 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:52.181 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:52.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:52.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:52.282 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:52.308 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:52.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:52.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:52.403 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:52 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-17T17:29:52.403 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:52 smithi003 ceph-mon[103344]: osdmap e236: 8 total, 8 up, 7 in 2024-08-17T17:29:52.404 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:52 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-17T17:29:52.404 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:52 smithi003 ceph-mon[100615]: osdmap e236: 8 total, 8 up, 7 in 2024-08-17T17:29:52.410 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:52.454 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:52.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:52.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:52.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:52 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-17T17:29:52.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:52 smithi053 ceph-mon[89963]: osdmap e236: 8 total, 8 up, 7 in 2024-08-17T17:29:52.556 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:52.565 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 71 16 97473 97457 85.7692 59.125 0.0123479 0.0116536 2024-08-17T17:29:52.583 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:52.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:52.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:52.685 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:52.728 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:52.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:52.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:52.830 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:52.856 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:52.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:52.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:52.958 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:52.985 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:52.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:52.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:53.086 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:53.136 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:53.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:53.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:53.238 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:53.264 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:53.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:53.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:53.366 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:53.386 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:53 smithi003 ceph-mon[103344]: pgmap v666: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 55 MiB/s wr, 877 op/s 2024-08-17T17:29:53.386 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:53 smithi003 ceph-mon[100615]: pgmap v666: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 55 MiB/s wr, 877 op/s 2024-08-17T17:29:53.414 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:53.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:53.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:53.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:53 smithi053 ceph-mon[89963]: pgmap v666: 89 pgs: 89 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 55 MiB/s wr, 877 op/s 2024-08-17T17:29:53.516 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:53.541 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:53.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:53.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:53.565 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 72 16 98410 98394 85.3914 58.5625 0.00592172 0.0117046 2024-08-17T17:29:53.642 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:53.687 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:53.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:53.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:53.789 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:53.826 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:53.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:53.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:53.928 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:53.968 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:53.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:53.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:54.070 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:54.116 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:54.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:54.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:54.146 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 1 1 2024-08-17T17:29:54.218 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:54.243 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:54.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:54.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:54.344 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:54.372 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:54.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:54.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:54.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:54 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:29:54.474 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:54.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:54 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:29:54.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:54 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:29:54.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:54.532 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:54.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:54.567 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 73 16 99277 99261 84.9639 54.1875 0.0232058 0.0117641 2024-08-17T17:29:54.633 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:54.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:54.661 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:54.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:54.763 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:54.789 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:54.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:54.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:54.891 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:54.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:54.945 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:54.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:55.046 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:55.072 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:55.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:55.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:55.174 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:55.229 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:55.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:55.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:55.331 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:55.349 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:55 smithi003 ceph-mon[100615]: pgmap v667: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 54 MiB/s wr, 868 op/s 2024-08-17T17:29:55.350 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:55 smithi003 ceph-mon[103344]: pgmap v667: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 54 MiB/s wr, 868 op/s 2024-08-17T17:29:55.356 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:55.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:55.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:55.458 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:55.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:55 smithi053 ceph-mon[89963]: pgmap v667: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 54 MiB/s wr, 868 op/s 2024-08-17T17:29:55.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:55.516 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:55.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:55.567 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 74 16 100156 100140 84.5561 54.9375 0.00689425 0.0118202 2024-08-17T17:29:55.616 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:55.644 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:55.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:55.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:55.746 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:55.798 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:55.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:55.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:55.900 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:55.906 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:29:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:29:55] "GET /metrics HTTP/1.1" 200 36502 "" "Prometheus/2.51.0" 2024-08-17T17:29:55.924 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:55.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:55.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:56.026 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:56.060 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:56.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:56.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:56.162 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:56.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:56.205 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:56.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:56.306 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:56.332 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:56.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:56.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:56.434 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:56.481 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:56.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:56.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:56.567 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 75 16 100988 100972 84.122 52 0.0337202 0.0118813 2024-08-17T17:29:56.583 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:56.611 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:56.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:56.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:56.713 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:56.740 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:56.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:56.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:56.842 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:56.872 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:56.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:56.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:56.974 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:57.002 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:57.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:57.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:57.104 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:57.131 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:57.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:57.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:57.232 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:57.259 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:57.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:57.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:57.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:57.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:57.402 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:57.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:57.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:57 smithi053 ceph-mon[89963]: pgmap v668: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 54 MiB/s wr, 860 op/s 2024-08-17T17:29:57.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:57 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3099947948' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:29:57.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:57 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:29:57.483 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-08-17T17:29:57.484 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:57 smithi003 ceph-mon[100615]: pgmap v668: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 54 MiB/s wr, 860 op/s 2024-08-17T17:29:57.484 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:57 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3099947948' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:29:57.484 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:57 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:29:57.484 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:57 smithi003 ceph-mon[103344]: pgmap v668: 89 pgs: 89 active+clean; 6.0 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 54 MiB/s wr, 860 op/s 2024-08-17T17:29:57.484 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:57 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3099947948' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:29:57.484 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:57 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:29:57.504 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:57.532 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:57.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:57.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:57.567 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 76 16 101903 101887 83.7676 57.1875 0.0165548 0.0119315 2024-08-17T17:29:57.633 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:57.677 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:57.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:57.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:57.779 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:57.803 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:57.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:57.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:57.905 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:57.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:57.952 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:57.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:58.053 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:29:58.099 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:58.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:58.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:29:58.201 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:58.228 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:58.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:58.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:58.330 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:58.357 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:58.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:58.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:58.459 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:58.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:58 smithi053 ceph-mon[89963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:29:58.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:58 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:29:58.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:58 smithi053 ceph-mon[89963]: osdmap e237: 8 total, 8 up, 7 in 2024-08-17T17:29:58.484 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:58.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:58.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:58.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:58 smithi003 ceph-mon[100615]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:29:58.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:58 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:29:58.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:58 smithi003 ceph-mon[100615]: osdmap e237: 8 total, 8 up, 7 in 2024-08-17T17:29:58.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:58 smithi003 ceph-mon[103344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:29:58.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:58 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:29:58.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:58 smithi003 ceph-mon[103344]: osdmap e237: 8 total, 8 up, 7 in 2024-08-17T17:29:58.567 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 77 16 102739 102723 83.3582 52.25 0.0276614 0.011991 2024-08-17T17:29:58.585 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:29:58.625 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:58.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:58.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:29:58.727 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:29:58.754 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:58.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:58.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:29:58.836 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:29:58.836 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:29:58.856 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:58.884 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:58.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:58.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:58.986 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:29:59.015 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:59.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:59.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:29:59.117 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:59.166 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:59.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:59.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:59.268 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:29:59.304 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:59.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:59.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:29:59.406 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:59.449 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:59.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:59.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:59.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:29:59 smithi053 ceph-mon[89963]: pgmap v670: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 46 MiB/s wr, 740 op/s 2024-08-17T17:29:59.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:29:59 smithi003 ceph-mon[100615]: pgmap v670: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 46 MiB/s wr, 740 op/s 2024-08-17T17:29:59.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:29:59 smithi003 ceph-mon[103344]: pgmap v670: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 46 MiB/s wr, 740 op/s 2024-08-17T17:29:59.551 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:29:59.568 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 78 16 103649 103633 83.0187 56.875 0.0356304 0.0120394 2024-08-17T17:29:59.614 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-17T17:29:59.622 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:29:59.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:59.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:29:59.725 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:29:59.750 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:59.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:59.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:29:59.852 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:29:59.879 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:29:59.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:29:59.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:29:59.981 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:00.035 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:00.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:00.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:00.092 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:30:00.092 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:30:00.136 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:00.161 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:30:00.161 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:30:00.190 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:00.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:00.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:00.208 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:30:00.209 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:30:00.245 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:30:00.245 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:30:00.275 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:30:00.275 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:30:00.293 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:00.310 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:30:00.310 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:30:00.340 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:30:00.341 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:30:00.343 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:00.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:00.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:00.376 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:30:00.376 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:30:00.446 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:00.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:00 smithi053 ceph-mon[89963]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-08-17T17:30:00.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:00 smithi053 ceph-mon[89963]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-08-17T17:30:00.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:00 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2957627029' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:30:00.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:30:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:30:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:30:00.472 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:00.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:00.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:00.478 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:00 smithi003 ceph-mon[103344]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-08-17T17:30:00.479 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:00 smithi003 ceph-mon[103344]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-08-17T17:30:00.479 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:00 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2957627029' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:30:00.480 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:00 smithi003 ceph-mon[100615]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-08-17T17:30:00.480 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:00 smithi003 ceph-mon[100615]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-08-17T17:30:00.480 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:00 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2957627029' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:30:00.567 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 79 16 104411 104395 82.5707 47.625 0.0196911 0.0121055 2024-08-17T17:30:00.575 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:00.615 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:00.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:00.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:00.716 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:00.744 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:00.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:00.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:00.845 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:00.874 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:00.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:00.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:00.976 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:01.003 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:01.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:01.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:01.105 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:01.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:01.134 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:01.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:01.235 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:01.261 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:01.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:01.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:01.272 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:01 smithi003 ceph-mon[100615]: pgmap v671: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 60 MiB/s wr, 965 op/s 2024-08-17T17:30:01.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:01 smithi003 ceph-mon[103344]: pgmap v671: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 60 MiB/s wr, 965 op/s 2024-08-17T17:30:01.363 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:01.406 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:01.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:01.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:01.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:01 smithi053 ceph-mon[89963]: pgmap v671: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 60 MiB/s wr, 965 op/s 2024-08-17T17:30:01.508 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:01.545 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:01.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:01.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:01.568 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-08-17T17:30:01.567828+0000 min lat: 0.00130565 max lat: 0.281731 avg lat: 0.0121523 2024-08-17T17:30:01.568 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:30:01.568 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 80 16 105320 105304 82.2487 56.8125 0.0106725 0.0121523 2024-08-17T17:30:01.647 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:01.687 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:01.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:01.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:01.789 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:01.802 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:30:01.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:01.841 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:01.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:01.889 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:30:01.942 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:02.001 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:02.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:02.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:02.103 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:02.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:02.143 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:02.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:02.244 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:02.267 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:02.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:02.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:02.369 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:02.402 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:02.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:02.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:02.504 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:02.530 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:02.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:02.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:02.567 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 81 16 106036 106020 81.7858 44.75 0.0155363 0.0122218 2024-08-17T17:30:02.632 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:02.659 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:02.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:02.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:02.761 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:02.805 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:02.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:02.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:02.907 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:02.955 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:02.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:02.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:02.985 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:30:03.057 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:03.105 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:03.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:03.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:03.207 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:03.232 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:03.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:03.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:03.334 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:03.364 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:03.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:03.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:03.425 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:03 smithi003 ceph-mon[100615]: pgmap v672: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 55 MiB/s wr, 887 op/s 2024-08-17T17:30:03.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:03 smithi003 ceph-mon[103344]: pgmap v672: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 55 MiB/s wr, 887 op/s 2024-08-17T17:30:03.466 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:03.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:03 smithi053 ceph-mon[89963]: pgmap v672: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 55 MiB/s wr, 887 op/s 2024-08-17T17:30:03.504 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:03.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:03.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:03.567 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 82 16 106842 106826 81.4027 50.375 0.0176214 0.0122791 2024-08-17T17:30:03.606 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:03.640 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:03.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:03.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:03.742 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:03.753 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:30:03.783 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:03.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:03.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:03.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:03.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:03.942 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:03.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:04.043 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:04.075 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:04.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:04.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:04.177 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:04.206 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:04.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:04.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:04.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:04.353 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:04.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:04.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:04.455 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:04.481 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:04.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:04.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:04.568 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 83 16 107659 107643 81.0372 51.0625 0.0233205 0.0123343 2024-08-17T17:30:04.583 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:04.639 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:04.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:04.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:04.741 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:04.769 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:04.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:04.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:04.873 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:04.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:04.923 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:04.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:05.024 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:05.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:05.053 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:05.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:05.154 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:05.180 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:05.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:05.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:05.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:05 smithi053 ceph-mon[89963]: pgmap v673: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 59 MiB/s wr, 941 op/s 2024-08-17T17:30:05.282 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:05.327 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:05.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:05.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:05.421 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:05 smithi003 ceph-mon[103344]: pgmap v673: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 59 MiB/s wr, 941 op/s 2024-08-17T17:30:05.422 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:05 smithi003 ceph-mon[100615]: pgmap v673: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 59 MiB/s wr, 941 op/s 2024-08-17T17:30:05.429 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:05.456 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:05.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:05.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:05.558 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:05.568 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 84 16 108492 108476 80.6922 52.0625 0.0173246 0.0123866 2024-08-17T17:30:05.612 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:05.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:05.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:05.715 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:05.760 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:05.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:05.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:05.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:30:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:30:05] "GET /metrics HTTP/1.1" 200 36502 "" "Prometheus/2.51.0" 2024-08-17T17:30:05.861 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:05.886 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:05.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:05.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:05.987 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:06.016 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:06.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:06.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:06.117 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:06.143 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:06.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:06.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:06.165 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:30:06.165 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-08-17T17:30:06.165 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-08-17T17:30:06.165 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-08-17T17:30:06.165 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:30:06.245 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:06.289 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:06.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:06.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:06.391 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:06.438 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:06.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:06.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:06.540 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:06.566 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:06.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:06.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:06.568 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 85 16 109331 109315 80.3598 52.4375 0.018024 0.0124385 2024-08-17T17:30:06.668 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:06.694 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:06.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:06.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:06.927 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:06.941 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-17T17:30:06.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:06.979 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:06.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:07.080 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:07.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:07.127 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:07.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:07.228 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:07.235 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:30:07.239 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:30:07.251 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:07.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:07.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:07.352 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:07.401 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:07.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:07.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:07.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:07 smithi053 ceph-mon[89963]: pgmap v674: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 53 MiB/s wr, 844 op/s 2024-08-17T17:30:07.480 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:07 smithi003 ceph-mon[103344]: pgmap v674: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 53 MiB/s wr, 844 op/s 2024-08-17T17:30:07.480 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:07 smithi003 ceph-mon[100615]: pgmap v674: 89 pgs: 89 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 53 MiB/s wr, 844 op/s 2024-08-17T17:30:07.503 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:07.526 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:07.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:07.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:07.568 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 86 16 110126 110110 80.003 49.6875 0.0266891 0.0124934 2024-08-17T17:30:07.628 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:07.654 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:07.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:07.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:07.756 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:07.780 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:07.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:07.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:07.882 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:07.929 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:07.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:07.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:08.034 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:08.058 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:08.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:08.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:08.160 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:08.185 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:08.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:08.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:08.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:08.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:08.331 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:08.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:08.366 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:30:08.366 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:30:08.433 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:08.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:30:08.478 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:08.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:08.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:08.568 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 87 16 110906 110890 79.6438 48.75 0.0152349 0.01255 2024-08-17T17:30:08.580 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:08.606 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:08.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:08.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:08.707 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:08.769 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:08.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:08.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:08.871 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:08.898 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:08.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:08.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:09.000 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:09.027 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:09.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:09.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:09.129 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:09.155 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:09.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:09.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:09.257 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:09.285 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:09.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:09.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:09.387 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:09.414 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:09.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:09.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:09 smithi053 ceph-mon[89963]: pgmap v675: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 54 MiB/s wr, 870 op/s 2024-08-17T17:30:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:09 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2051814966' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:30:09.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:09 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:30:09.516 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:09.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:09 smithi003 ceph-mon[100615]: pgmap v675: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 54 MiB/s wr, 870 op/s 2024-08-17T17:30:09.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:09 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2051814966' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:30:09.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:09 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:30:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:09 smithi003 ceph-mon[103344]: pgmap v675: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 54 MiB/s wr, 870 op/s 2024-08-17T17:30:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:09 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2051814966' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:30:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:09 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:30:09.560 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:09.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:09.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:09.568 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 88 16 111741 111725 79.3318 52.1875 0.0263071 0.0126002 2024-08-17T17:30:09.663 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:09.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:09.703 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:09.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:09.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:09.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:09.852 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:09.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:09.953 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:09.981 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:09.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:09.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:10.083 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:10.123 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:10.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:10.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:10.225 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:10.247 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:10.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:10.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:10.349 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:10.378 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:10.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:10.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:10.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:10 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-17T17:30:10.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:10 smithi053 ceph-mon[89963]: osdmap e238: 8 total, 8 up, 7 in 2024-08-17T17:30:10.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:30:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:30:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:30:10.480 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:10.508 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:10.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:10.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:10.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:10 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-17T17:30:10.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:10 smithi003 ceph-mon[103344]: osdmap e238: 8 total, 8 up, 7 in 2024-08-17T17:30:10.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:10 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-17T17:30:10.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:10 smithi003 ceph-mon[100615]: osdmap e238: 8 total, 8 up, 7 in 2024-08-17T17:30:10.568 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 89 16 112528 112512 78.9931 49.1875 0.0225819 0.0126533 2024-08-17T17:30:10.610 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:10.652 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:10.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:10.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:10.753 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:10.788 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:10.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:10.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:10.891 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:10.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:10.913 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:10.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:11.014 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:11.041 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.1 to 1 (10000) 2024-08-17T17:30:11.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:11.049 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:11.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:11.150 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:11.194 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:11.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:11.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:11.296 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:11.334 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:11.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:11.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:11.427 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:11 smithi003 ceph-mon[103344]: pgmap v677: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 51 MiB/s wr, 815 op/s 2024-08-17T17:30:11.427 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:11 smithi003 ceph-mon[100615]: pgmap v677: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 51 MiB/s wr, 815 op/s 2024-08-17T17:30:11.437 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:11.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:11 smithi053 ceph-mon[89963]: pgmap v677: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 51 MiB/s wr, 815 op/s 2024-08-17T17:30:11.484 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:11.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:11.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:11.568 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total time run: 90.0096 2024-08-17T17:30:11.569 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total writes made: 113219 2024-08-17T17:30:11.569 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Write size: 65536 2024-08-17T17:30:11.569 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object size: 65536 2024-08-17T17:30:11.569 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Bandwidth (MB/sec): 78.6159 2024-08-17T17:30:11.569 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Bandwidth: 33.9443 2024-08-17T17:30:11.569 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max bandwidth (MB/sec): 191.75 2024-08-17T17:30:11.569 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min bandwidth (MB/sec): 42.3125 2024-08-17T17:30:11.569 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average IOPS: 1257 2024-08-17T17:30:11.569 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev IOPS: 543.109 2024-08-17T17:30:11.569 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max IOPS: 3068 2024-08-17T17:30:11.569 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min IOPS: 677 2024-08-17T17:30:11.569 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average Latency(s): 0.0127153 2024-08-17T17:30:11.569 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Latency(s): 0.0110273 2024-08-17T17:30:11.569 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max latency(s): 0.281731 2024-08-17T17:30:11.570 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min latency(s): 0.00130565 2024-08-17T17:30:11.573 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Cleaning up (deleting benchmark objects) 2024-08-17T17:30:11.587 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:11.614 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:11.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:11.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:11.716 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:11.759 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:11.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:11.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:11.861 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:11.887 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:11.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:11.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:11.989 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:12.015 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:12.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:12.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:12.116 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:12.142 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:12.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:12.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:12.244 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:12.272 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:12.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:12.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:12.374 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:12.401 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:12.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:12.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:12.503 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:12.528 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:12.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:12.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:12.630 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:12.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:12.656 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:12.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:12.757 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:12.781 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:12.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:12.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:12.883 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:12.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:12.928 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:12.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:13.030 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:13.055 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:13.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:13.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:13.078 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 2 1 2024-08-17T17:30:13.158 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:13.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:13.228 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:13.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:13.329 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:13.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:13.369 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:13.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:13.470 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:13 smithi003 ceph-mon[100615]: pgmap v678: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 51 MiB/s wr, 817 op/s 2024-08-17T17:30:13.471 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:13 smithi003 ceph-mon[103344]: pgmap v678: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 51 MiB/s wr, 817 op/s 2024-08-17T17:30:13.471 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:13.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:13.522 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:13.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:13.584 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:30:13.623 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:13.651 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:13.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:13.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:13.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:13 smithi053 ceph-mon[89963]: pgmap v678: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 51 MiB/s wr, 817 op/s 2024-08-17T17:30:13.753 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:13.777 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:13.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:13.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:13.879 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:13.904 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:13.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:13.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:14.005 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:14.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:14.031 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:14.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:14.132 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:14.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:14.188 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:14.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:14.289 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:14.336 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:14.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:14.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:14.438 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:14.464 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:14.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:14.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:14.565 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:14.591 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:14.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:14.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:14.693 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:14.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:14.733 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:14.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:14.834 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:14.872 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:14.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:14.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:14.973 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:15.001 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:15.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:15.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:15.103 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:15.149 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:15.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:15.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:15.251 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:15.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:15.276 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:15.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:15.354 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:15 smithi003 ceph-mon[103344]: pgmap v679: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 102 B/s rd, 44 MiB/s wr, 811 op/s 2024-08-17T17:30:15.354 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:15 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2606008522' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:30:15.354 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:15 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:30:15.354 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:15 smithi003 ceph-mon[103344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:30:15.355 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:15 smithi003 ceph-mon[103344]: Cluster is now healthy 2024-08-17T17:30:15.355 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:15 smithi003 ceph-mon[100615]: pgmap v679: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 102 B/s rd, 44 MiB/s wr, 811 op/s 2024-08-17T17:30:15.355 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:15 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2606008522' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:30:15.355 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:15 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:30:15.355 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:15 smithi003 ceph-mon[100615]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:30:15.355 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:15 smithi003 ceph-mon[100615]: Cluster is now healthy 2024-08-17T17:30:15.378 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:15.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:15.423 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:15.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:15.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:15 smithi053 ceph-mon[89963]: pgmap v679: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 102 B/s rd, 44 MiB/s wr, 811 op/s 2024-08-17T17:30:15.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:15 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2606008522' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:30:15.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:15 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:30:15.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:15 smithi053 ceph-mon[89963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:30:15.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:15 smithi053 ceph-mon[89963]: Cluster is now healthy 2024-08-17T17:30:15.523 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:15.551 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:15.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:15.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:15.631 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:30:15.631 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:30:15.632 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:30:15.632 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:30:15.632 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:30:15.632 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:30:15.632 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:30:15.632 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:30:15.632 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:30:15.633 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:30:15.633 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:30:15.633 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:30:15.633 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:30:15.633 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:30:15.633 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:30:15.633 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:30:15.653 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:15.692 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:15.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:15.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:15.794 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:15.836 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:15.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:15.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:15.902 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:30:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:30:15] "GET /metrics HTTP/1.1" 200 36498 "" "Prometheus/2.51.0" 2024-08-17T17:30:15.938 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:15.962 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:15.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:15.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:16.064 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:16.093 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:16.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:16.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:16.194 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:16.234 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:16.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:16.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:16.336 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:16.373 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:16.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:16.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:16.477 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:16.504 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:16.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:16.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:16.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:16 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:30:16.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:16 smithi003 ceph-mon[103344]: osdmap e239: 8 total, 8 up, 7 in 2024-08-17T17:30:16.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:16 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2869890846' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:30:16.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:16 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:30:16.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:16 smithi003 ceph-mon[100615]: osdmap e239: 8 total, 8 up, 7 in 2024-08-17T17:30:16.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:16 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2869890846' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:30:16.606 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:16.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:16.633 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:16.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:16.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:16 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:30:16.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:16 smithi053 ceph-mon[89963]: osdmap e239: 8 total, 8 up, 7 in 2024-08-17T17:30:16.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:16 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2869890846' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:30:16.734 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:16.760 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:16.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:16.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:16.861 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:16.902 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:16.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:16.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:16.992 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-08-17T17:30:17.004 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:17.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:17.041 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:17.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:17.142 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:17.183 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:17.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:17.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:17.257 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:17 smithi003 ceph-mon[100615]: pgmap v681: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 127 B/s rd, 42 MiB/s wr, 881 op/s 2024-08-17T17:30:17.285 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:17.310 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:17.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:17.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:17.412 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:17.450 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:17.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:17.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:17.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:17 smithi003 ceph-mon[103344]: pgmap v681: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 127 B/s rd, 42 MiB/s wr, 881 op/s 2024-08-17T17:30:17.553 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:17.580 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:17.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:17.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:17.682 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:17.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:17 smithi053 ceph-mon[89963]: pgmap v681: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 127 B/s rd, 42 MiB/s wr, 881 op/s 2024-08-17T17:30:17.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:17.717 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:17.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:17.819 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:17.844 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:17.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:17.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:17.946 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:17.972 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:17.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:17.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:18.074 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:18.120 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:18.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:18.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:18.222 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:18.249 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:18.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:18.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:18.351 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:18.409 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:18.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:18.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:18.511 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:18.535 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:18.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:18.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:18.637 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:18.662 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:18.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:18.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:18.763 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:18.818 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:18.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:18.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:18.920 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:18.947 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:18.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:18.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:19.049 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:19.101 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:19.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:19.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:19.203 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:19.226 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:19.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:19.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:19.328 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:19.354 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:19.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:19.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:19.456 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:19.509 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:19.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:19.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:19.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:19 smithi003 ceph-mon[103344]: pgmap v682: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 116 B/s rd, 20 MiB/s wr, 621 op/s 2024-08-17T17:30:19.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:19 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:30:19.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:19 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:30:19.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:19 smithi003 ceph-mon[100615]: pgmap v682: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 116 B/s rd, 20 MiB/s wr, 621 op/s 2024-08-17T17:30:19.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:19 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:30:19.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:19 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:30:19.526 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:30:19.611 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:19.637 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:19.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:19.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:19.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:19 smithi053 ceph-mon[89963]: pgmap v682: 89 pgs: 89 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 116 B/s rd, 20 MiB/s wr, 621 op/s 2024-08-17T17:30:19.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:19 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:30:19.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:19 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:30:19.739 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:19.781 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:19.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:19.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:19.883 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:19.887 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:30:19.926 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:19.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:19.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:20.028 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:20.051 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:20.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:20.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:20.153 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:20.177 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:20.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:20.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:20.279 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:20.302 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:20.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:20.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:20.352 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:30:20.404 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:20.427 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:20.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:20.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:20.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:30:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:30:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:30:20.529 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:20.554 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:20.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:20.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:20.656 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:20.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:20.702 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:20.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:20.803 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:20.829 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:20.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:20.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:20.931 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:20.992 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:20.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:20.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:21.094 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:21.132 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:21.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:21.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:21.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:21.276 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:21.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:21.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:21.378 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:21.417 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:21.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:21.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:21.519 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:21.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:21 smithi003 ceph-mon[103344]: pgmap v683: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 16 MiB/s wr, 792 op/s 2024-08-17T17:30:21.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:21 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:30:21.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:21 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:30:21.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:21 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:30:21.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:21 smithi003 ceph-mon[100615]: pgmap v683: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 16 MiB/s wr, 792 op/s 2024-08-17T17:30:21.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:21 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:30:21.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:21 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:30:21.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:21 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:30:21.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:21.548 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:21.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:21.649 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:21.711 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:21.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:21.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:21.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:21 smithi053 ceph-mon[89963]: pgmap v683: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 16 MiB/s wr, 792 op/s 2024-08-17T17:30:21.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:21 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:30:21.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:21 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:30:21.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:21 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:30:21.813 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:21.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:21.839 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:21.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:21.940 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:21.950 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:30:21.950 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:30:22.010 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:22.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:22.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:22.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:22.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:22.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:22.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:22.256 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:22.284 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:22.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:22.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:22.386 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:22.423 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:22.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:22.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:22.525 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:22.563 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:22.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:22.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:22.665 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:22.692 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:22.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:22.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:22.793 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:22.819 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:22.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:22.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:22.921 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:22.965 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:22.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:22.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:23.067 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:23.106 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:23.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:23.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:23.160 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-17T17:30:23.209 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:23.233 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:23.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:23.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:23.271 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:23 smithi003 ceph-mon[103344]: pgmap v684: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 15 MiB/s wr, 788 op/s 2024-08-17T17:30:23.334 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:23.355 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:30:23.357 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:23.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:23.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:23.362 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:30:23.459 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:23.484 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:23.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:23.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:23.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:23 smithi003 ceph-mon[100615]: pgmap v684: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 15 MiB/s wr, 788 op/s 2024-08-17T17:30:23.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:30:23.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:30:23.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:30:23.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:30:23.586 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:23.610 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:23.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:23.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:23.712 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:23 smithi053 ceph-mon[89963]: pgmap v684: 89 pgs: 89 active+clean; 6.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 102 B/s rd, 15 MiB/s wr, 788 op/s 2024-08-17T17:30:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:30:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:30:23.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:23.762 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:23.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:23.863 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:23.906 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:23.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:23.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:24.008 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:24.049 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:24.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:24.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:24.151 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:24.191 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:24.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:24.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:24.293 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:24.298 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.2 to 1 (10000) 2024-08-17T17:30:24.318 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:24.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:24.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:24.420 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:24.471 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:24.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:24.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:24.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:24 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/132528063' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:30:24.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:24 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:30:24.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:24 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/132528063' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:30:24.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:24 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:30:24.573 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:24.600 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:24.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:24.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:24.702 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:24.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:24 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/132528063' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:30:24.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:24 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:30:24.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:24.744 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:24.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:24.845 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:24.870 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:24.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:24.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:24.972 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:24.996 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:24.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:24.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:25.098 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:25.140 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:25.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:25.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:25.242 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:25.278 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:25.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:25.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:25.380 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:25.406 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:25.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:25.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:25.508 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:25.547 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:25.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:25.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:25.649 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:25.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:25 smithi003 ceph-mon[100615]: pgmap v685: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 2.1 MiB/s wr, 784 op/s 2024-08-17T17:30:25.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:25 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-17T17:30:25.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:25 smithi003 ceph-mon[100615]: osdmap e240: 8 total, 8 up, 7 in 2024-08-17T17:30:25.653 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:25 smithi003 ceph-mon[103344]: pgmap v685: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 2.1 MiB/s wr, 784 op/s 2024-08-17T17:30:25.653 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:25 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-17T17:30:25.653 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:25 smithi003 ceph-mon[103344]: osdmap e240: 8 total, 8 up, 7 in 2024-08-17T17:30:25.701 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:25.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:25.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:25.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:25 smithi053 ceph-mon[89963]: pgmap v685: 89 pgs: 89 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 2.1 MiB/s wr, 784 op/s 2024-08-17T17:30:25.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:25 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-17T17:30:25.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:25 smithi053 ceph-mon[89963]: osdmap e240: 8 total, 8 up, 7 in 2024-08-17T17:30:25.803 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:25.845 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:25.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:25.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:25.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:25.988 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:25.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:25.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:25.997 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:30:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:30:25] "GET /metrics HTTP/1.1" 200 36506 "" "Prometheus/2.51.0" 2024-08-17T17:30:26.045 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 3 1 2024-08-17T17:30:26.090 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:26.182 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:26.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:26.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:26.284 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:26.310 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:26.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:26.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:26.412 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:26.453 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:30:26.484 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:26.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:26.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:26.586 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:26.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:26.638 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:26.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:26.739 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:26.817 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:26.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:26.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:26.919 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:26.946 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:26.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:26.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:27.048 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:27.074 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:27.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:27.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:27.176 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:27.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:27.211 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:27.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:27.311 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:27.348 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:27.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:27.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:27.450 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:27.494 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:27.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:27.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:27.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:27 smithi003 ceph-mon[103344]: pgmap v687: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 783 op/s 2024-08-17T17:30:27.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:27 smithi003 ceph-mon[100615]: pgmap v687: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 783 op/s 2024-08-17T17:30:27.596 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:27.622 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:27.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:27.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:27.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:27 smithi053 ceph-mon[89963]: pgmap v687: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 783 op/s 2024-08-17T17:30:27.724 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:27.749 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:27.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:27.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:27.850 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:27.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:27.876 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:27.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:27.977 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:28.004 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:28.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:28.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:28.106 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:28.148 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:28.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:28.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:28.250 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:28.276 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:28.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:28.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:28.317 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-08-17T17:30:28.378 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:28.416 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:28.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:28.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:28.517 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:28.563 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:28.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:28.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:28.665 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:28.701 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:28.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:28.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:28.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:28 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1155241921' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:30:28.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:28 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3043374079' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:30:28.727 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:28 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1155241921' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:30:28.727 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:28 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3043374079' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:30:28.727 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:28 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1155241921' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:30:28.727 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:28 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3043374079' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:30:28.804 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:28.828 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:28.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:28.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:28.931 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:28.956 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:28.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:28.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:29.058 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:29.085 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:29.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:29.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:29.086 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:30:29.087 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:30:29.087 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:30:29.087 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:30:29.087 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:30:29.087 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:30:29.087 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:30:29.087 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:30:29.087 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:30:29.087 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:30:29.087 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:30:29.087 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:30:29.088 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:30:29.088 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:30:29.088 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:30:29.088 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:30:29.186 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:29.230 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:29.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:29.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:29.332 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:29.358 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:29.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:29.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:29.410 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:29 smithi003 ceph-mon[103344]: pgmap v688: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 789 op/s 2024-08-17T17:30:29.410 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:29 smithi003 ceph-mon[103344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:30:29.411 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:29 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3043374079' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:30:29.411 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:29 smithi003 ceph-mon[103344]: osdmap e241: 8 total, 8 up, 7 in 2024-08-17T17:30:29.411 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:29 smithi003 ceph-mon[100615]: pgmap v688: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 789 op/s 2024-08-17T17:30:29.411 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:29 smithi003 ceph-mon[100615]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:30:29.411 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:29 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3043374079' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:30:29.411 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:29 smithi003 ceph-mon[100615]: osdmap e241: 8 total, 8 up, 7 in 2024-08-17T17:30:29.460 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:29.487 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:29.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:29.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:29.590 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:29.628 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:29.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:29.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:29.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:29 smithi053 ceph-mon[89963]: pgmap v688: 89 pgs: 89 active+clean; 6.3 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 789 op/s 2024-08-17T17:30:29.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:29 smithi053 ceph-mon[89963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:30:29.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:29 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3043374079' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:30:29.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:29 smithi053 ceph-mon[89963]: osdmap e241: 8 total, 8 up, 7 in 2024-08-17T17:30:29.730 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:29.767 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:29.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:29.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:29.868 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:29.927 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:29.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:29.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:30.029 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:30.056 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:30.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:30.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:30.157 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:30.199 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:30.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:30.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:30.301 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:30.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:30.352 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:30.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:30.365 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:30:30.453 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:30.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:30:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:30:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:30:30.498 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:30.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:30.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:30.600 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:30.647 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:30.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:30.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:30.749 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:30.786 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:30.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:30.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:30.887 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:30.913 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:30.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:30.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:31.015 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:31.041 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:31.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:31.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:31.140 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:30:31.141 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:30:31.141 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.141 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45767.0:128790 7.7 e241/226)", 2024-08-17T17:30:31.141 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:30:31.078167+0000", 2024-08-17T17:30:31.141 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.022368967999999999, 2024-08-17T17:30:31.141 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.022401909000000001, 2024-08-17T17:30:31.141 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:31.141 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:31.141 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:30:31.141 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:31.141 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.141 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:31.142 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.078167+0000", 2024-08-17T17:30:31.142 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:31.142 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.142 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.142 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:31.142 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.078167+0000", 2024-08-17T17:30:31.142 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:31.142 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.142 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.142 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:31.142 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.078168+0000", 2024-08-17T17:30:31.142 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1999999999999998e-07 2024-08-17T17:30:31.143 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.143 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.143 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:31.143 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.078173+0000", 2024-08-17T17:30:31.143 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8910000000000003e-06 2024-08-17T17:30:31.143 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.143 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.143 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:31.143 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.078182+0000", 2024-08-17T17:30:31.143 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6430000000000001e-06 2024-08-17T17:30:31.143 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.143 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.144 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:31.144 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.078188+0000", 2024-08-17T17:30:31.144 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2779999999999999e-06 2024-08-17T17:30:31.144 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.144 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.144 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:31.144 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.100563+0000", 2024-08-17T17:30:31.144 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.022374773000000001 2024-08-17T17:30:31.144 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:31.144 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:31.144 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:31.145 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.145 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.145 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:128795 7.2 7:4ebba5fc:::benchmark_data_smithi003_201798_object15573:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-08-17T17:30:31.145 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:30:31.086486+0000", 2024-08-17T17:30:31.145 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.014050333, 2024-08-17T17:30:31.145 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.014148734, 2024-08-17T17:30:31.145 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:31.145 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:31.145 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:30:31.145 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:30:31.145 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:30:31.145 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:30:31.145 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 128795 2024-08-17T17:30:31.145 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.146 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:31.146 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.146 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:31.146 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.086486+0000", 2024-08-17T17:30:31.146 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:31.146 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.146 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.146 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:31.146 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.086486+0000", 2024-08-17T17:30:31.146 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:31.146 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.146 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.147 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:31.147 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.086488+0000", 2024-08-17T17:30:31.147 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.671e-06 2024-08-17T17:30:31.147 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.147 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.147 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:31.147 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.086492+0000", 2024-08-17T17:30:31.147 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4830000000000001e-06 2024-08-17T17:30:31.147 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.147 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.147 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:31.147 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.086504+0000", 2024-08-17T17:30:31.147 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2038999999999999e-05 2024-08-17T17:30:31.148 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.148 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.148 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:31.148 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.086527+0000", 2024-08-17T17:30:31.148 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2433e-05 2024-08-17T17:30:31.148 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.148 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.148 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:31.148 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.086665+0000", 2024-08-17T17:30:31.148 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000138078 2024-08-17T17:30:31.148 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.148 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.148 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:30:31.149 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.086783+0000", 2024-08-17T17:30:31.149 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000117905 2024-08-17T17:30:31.149 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.149 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.149 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:30:31.149 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.086915+0000", 2024-08-17T17:30:31.149 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013233 2024-08-17T17:30:31.149 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.149 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.149 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:31.149 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.099134+0000", 2024-08-17T17:30:31.149 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012219077 2024-08-17T17:30:31.149 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.149 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.150 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:30:31.150 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.099717+0000", 2024-08-17T17:30:31.150 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00058299800000000004 2024-08-17T17:30:31.150 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:31.150 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:31.150 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:31.150 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.150 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.150 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:128797 7.2 7:47439879:::benchmark_data_smithi003_201798_object15575:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-08-17T17:30:31.150 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:30:31.089728+0000", 2024-08-17T17:30:31.150 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.010808188999999999, 2024-08-17T17:30:31.150 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010987629, 2024-08-17T17:30:31.150 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:31.151 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:31.151 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:30:31.151 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:30:31.151 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:30:31.151 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:30:31.151 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 128797 2024-08-17T17:30:31.151 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.151 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:31.151 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.151 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:31.151 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.089728+0000", 2024-08-17T17:30:31.151 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:31.151 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.152 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.152 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:31.152 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.089728+0000", 2024-08-17T17:30:31.152 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:31.152 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.152 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.152 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:31.152 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.089730+0000", 2024-08-17T17:30:31.152 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.753e-06 2024-08-17T17:30:31.152 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.152 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.152 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:31.152 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.089736+0000", 2024-08-17T17:30:31.153 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.896e-06 2024-08-17T17:30:31.153 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.153 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.153 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:31.153 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.089748+0000", 2024-08-17T17:30:31.153 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2311e-05 2024-08-17T17:30:31.153 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.153 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.153 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:31.153 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.089756+0000", 2024-08-17T17:30:31.153 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9580000000000007e-06 2024-08-17T17:30:31.153 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.153 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.153 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:31.154 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.092396+0000", 2024-08-17T17:30:31.154 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026402700000000001 2024-08-17T17:30:31.154 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.154 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.154 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:30:31.154 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.092514+0000", 2024-08-17T17:30:31.154 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000118066 2024-08-17T17:30:31.154 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.154 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.154 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:30:31.154 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.092627+0000", 2024-08-17T17:30:31.154 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000112756 2024-08-17T17:30:31.154 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:31.155 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:31.155 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:31.155 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.155 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.155 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:128798 7.3 7:c4e28c7b:::benchmark_data_smithi003_201798_object15576:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-08-17T17:30:31.155 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:30:31.089931+0000", 2024-08-17T17:30:31.155 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.010605570999999999, 2024-08-17T17:30:31.155 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010854196, 2024-08-17T17:30:31.155 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:31.155 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:31.155 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:30:31.155 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:30:31.155 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:30:31.155 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:30:31.155 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 128798 2024-08-17T17:30:31.156 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.156 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:31.156 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.156 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:31.156 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.089931+0000", 2024-08-17T17:30:31.156 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:31.156 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.156 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.156 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:31.156 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.089931+0000", 2024-08-17T17:30:31.156 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:31.156 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.156 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.156 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:31.157 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.089932+0000", 2024-08-17T17:30:31.157 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3799999999999996e-07 2024-08-17T17:30:31.157 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.157 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.157 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:31.157 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.089935+0000", 2024-08-17T17:30:31.157 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9219999999999999e-06 2024-08-17T17:30:31.157 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.157 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.157 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:31.157 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.089946+0000", 2024-08-17T17:30:31.157 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0484e-05 2024-08-17T17:30:31.157 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.157 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.157 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:31.158 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.089953+0000", 2024-08-17T17:30:31.158 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6789999999999999e-06 2024-08-17T17:30:31.158 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.158 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.158 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:31.158 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.090004+0000", 2024-08-17T17:30:31.158 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1371e-05 2024-08-17T17:30:31.158 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.158 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.158 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:30:31.158 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.090105+0000", 2024-08-17T17:30:31.158 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000100696 2024-08-17T17:30:31.158 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.158 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.159 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,6", 2024-08-17T17:30:31.159 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.090204+0000", 2024-08-17T17:30:31.159 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9678000000000003e-05 2024-08-17T17:30:31.159 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.159 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.159 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:30:31.159 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.094198+0000", 2024-08-17T17:30:31.159 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039940870000000003 2024-08-17T17:30:31.159 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.159 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.159 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:31.159 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.094842+0000", 2024-08-17T17:30:31.159 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000643309 2024-08-17T17:30:31.159 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:31.160 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:31.160 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:31.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.160 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.160 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:128801 7.3 7:c207d771:::benchmark_data_smithi003_201798_object15579:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-08-17T17:30:31.160 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:30:31.095473+0000", 2024-08-17T17:30:31.160 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.005063466, 2024-08-17T17:30:31.160 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.005390901, 2024-08-17T17:30:31.160 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:31.160 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:31.160 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:30:31.160 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:30:31.160 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:30:31.160 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:30:31.161 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 128801 2024-08-17T17:30:31.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.161 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:31.161 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.161 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:31.161 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.095473+0000", 2024-08-17T17:30:31.161 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:31.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.161 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.161 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:31.161 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.095473+0000", 2024-08-17T17:30:31.161 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:31.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.162 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.162 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:31.162 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.095474+0000", 2024-08-17T17:30:31.162 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.452e-06 2024-08-17T17:30:31.162 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.162 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.162 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:31.162 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.095480+0000", 2024-08-17T17:30:31.162 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4600000000000002e-06 2024-08-17T17:30:31.162 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.162 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.162 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:31.162 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.095491+0000", 2024-08-17T17:30:31.163 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1201e-05 2024-08-17T17:30:31.163 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.163 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.163 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:31.163 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.095499+0000", 2024-08-17T17:30:31.163 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6569999999999997e-06 2024-08-17T17:30:31.163 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.163 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.163 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:31.163 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.099381+0000", 2024-08-17T17:30:31.163 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038824179999999999 2024-08-17T17:30:31.163 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.163 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.164 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:30:31.164 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.099507+0000", 2024-08-17T17:30:31.164 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012637600000000001 2024-08-17T17:30:31.164 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.164 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.164 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,6", 2024-08-17T17:30:31.164 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.099630+0000", 2024-08-17T17:30:31.164 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000122743 2024-08-17T17:30:31.164 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:31.164 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:31.164 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:31.164 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.165 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.165 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:128802 7.1 7:83c651b6:::benchmark_data_smithi003_201798_object15580:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-08-17T17:30:31.165 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:30:31.095517+0000", 2024-08-17T17:30:31.165 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0050188669999999998, 2024-08-17T17:30:31.165 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0054049409999999999, 2024-08-17T17:30:31.165 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:31.165 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:31.165 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:30:31.165 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:30:31.165 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:30:31.165 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:30:31.165 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 128802 2024-08-17T17:30:31.165 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.165 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:31.165 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.166 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:31.166 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.095517+0000", 2024-08-17T17:30:31.166 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:31.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.166 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.166 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:31.166 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.095517+0000", 2024-08-17T17:30:31.166 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:31.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.166 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.166 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:31.166 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.095518+0000", 2024-08-17T17:30:31.166 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6199999999999998e-07 2024-08-17T17:30:31.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.167 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.167 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:31.167 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.095520+0000", 2024-08-17T17:30:31.167 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.469e-06 2024-08-17T17:30:31.167 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.167 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.167 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:31.167 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.095527+0000", 2024-08-17T17:30:31.167 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9759999999999998e-06 2024-08-17T17:30:31.167 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.167 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.167 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:31.167 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.095533+0000", 2024-08-17T17:30:31.167 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3349999999999997e-06 2024-08-17T17:30:31.167 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.168 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.168 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:31.168 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.095581+0000", 2024-08-17T17:30:31.168 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8511999999999999e-05 2024-08-17T17:30:31.168 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.168 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.168 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:30:31.168 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.095692+0000", 2024-08-17T17:30:31.168 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011065000000000001 2024-08-17T17:30:31.168 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.168 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.168 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:30:31.168 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.095794+0000", 2024-08-17T17:30:31.168 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010259600000000001 2024-08-17T17:30:31.168 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.169 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.169 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:30:31.169 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.100522+0000", 2024-08-17T17:30:31.169 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0047270630000000001 2024-08-17T17:30:31.169 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:31.169 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:31.169 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:31.169 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.169 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.169 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:128803 7.2 7:4202efa7:::benchmark_data_smithi003_201798_object15581:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-08-17T17:30:31.169 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:30:31.099257+0000", 2024-08-17T17:30:31.169 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0012792089999999999, 2024-08-17T17:30:31.169 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017287310000000001, 2024-08-17T17:30:31.169 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:31.169 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:31.170 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:30:31.170 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:30:31.170 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:30:31.170 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:30:31.170 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 128803 2024-08-17T17:30:31.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.170 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:31.170 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.170 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:31.170 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.099257+0000", 2024-08-17T17:30:31.170 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:31.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.171 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.171 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:31.171 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.099257+0000", 2024-08-17T17:30:31.171 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:31.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.171 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.171 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:31.171 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.099258+0000", 2024-08-17T17:30:31.171 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.04e-06 2024-08-17T17:30:31.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.171 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.171 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:31.171 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.099263+0000", 2024-08-17T17:30:31.171 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7790000000000003e-06 2024-08-17T17:30:31.172 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.172 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:31.172 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.099275+0000", 2024-08-17T17:30:31.172 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1776999999999999e-05 2024-08-17T17:30:31.172 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.172 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:31.172 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.099282+0000", 2024-08-17T17:30:31.172 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6969999999999993e-06 2024-08-17T17:30:31.172 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.172 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:31.172 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.099744+0000", 2024-08-17T17:30:31.173 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00046125199999999999 2024-08-17T17:30:31.173 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.173 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.173 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:30:31.173 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.099857+0000", 2024-08-17T17:30:31.173 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011301199999999999 2024-08-17T17:30:31.173 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.173 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.173 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:30:31.173 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.099965+0000", 2024-08-17T17:30:31.173 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000107884 2024-08-17T17:30:31.174 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:31.174 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:31.174 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:31.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.174 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45767.0:128804 7.8 e241/226)", 2024-08-17T17:30:31.174 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:30:31.099744+0000", 2024-08-17T17:30:31.174 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.00079186999999999997, 2024-08-17T17:30:31.174 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001294283, 2024-08-17T17:30:31.174 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:31.174 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:31.174 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-08-17T17:30:31.174 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:31.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.175 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:31.175 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.099744+0000", 2024-08-17T17:30:31.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:31.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.175 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:31.175 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.099744+0000", 2024-08-17T17:30:31.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:31.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.175 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:31.175 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.099745+0000", 2024-08-17T17:30:31.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.06e-07 2024-08-17T17:30:31.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.176 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:31.176 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.099750+0000", 2024-08-17T17:30:31.176 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0560000000000004e-06 2024-08-17T17:30:31.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.176 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:31.176 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.099757+0000", 2024-08-17T17:30:31.176 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2880000000000004e-06 2024-08-17T17:30:31.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.176 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:31.176 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.099763+0000", 2024-08-17T17:30:31.176 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9390000000000002e-06 2024-08-17T17:30:31.177 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:31.177 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:31.177 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:31.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.177 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.177 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45767.0:128777 7.f e241/226 7:f1fec24c:::benchmark_data_smithi003_201798_object15555:head v 241'8069, mlcod=241'8067)", 2024-08-17T17:30:31.177 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:30:31.064804+0000", 2024-08-17T17:30:31.177 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.035732631000000001, 2024-08-17T17:30:31.177 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.036273013999999999, 2024-08-17T17:30:31.177 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:31.177 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:31.177 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:30:31.177 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:31.178 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.178 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:31.178 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.064804+0000", 2024-08-17T17:30:31.178 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:31.178 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.178 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.178 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:31.178 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.064804+0000", 2024-08-17T17:30:31.178 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:31.178 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.178 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.178 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:31.178 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.064804+0000", 2024-08-17T17:30:31.178 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3399999999999998e-07 2024-08-17T17:30:31.178 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.179 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.179 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:31.179 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.064841+0000", 2024-08-17T17:30:31.179 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6494e-05 2024-08-17T17:30:31.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.179 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.179 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:31.179 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.064852+0000", 2024-08-17T17:30:31.179 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1479e-05 2024-08-17T17:30:31.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.179 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.179 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:31.179 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.064860+0000", 2024-08-17T17:30:31.179 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7389999999999999e-06 2024-08-17T17:30:31.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.180 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:31.180 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.064993+0000", 2024-08-17T17:30:31.180 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013254099999999999 2024-08-17T17:30:31.180 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.180 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:30:31.180 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.065015+0000", 2024-08-17T17:30:31.180 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2563000000000001e-05 2024-08-17T17:30:31.180 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:31.180 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:31.180 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:31.180 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.180 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.45767.0:128795 7.2 e241/226)", 2024-08-17T17:30:31.181 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:30:31.099806+0000", 2024-08-17T17:30:31.181 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.00073028199999999996, 2024-08-17T17:30:31.181 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0013130559999999999, 2024-08-17T17:30:31.181 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:31.181 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:31.181 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-08-17T17:30:31.181 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:31.181 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.181 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:31.181 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.099806+0000", 2024-08-17T17:30:31.181 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:31.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.181 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.181 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:31.181 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.099806+0000", 2024-08-17T17:30:31.182 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:31.182 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.182 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:31.182 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.099807+0000", 2024-08-17T17:30:31.182 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4899999999999995e-07 2024-08-17T17:30:31.182 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.182 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:31.182 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.099810+0000", 2024-08-17T17:30:31.182 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0309999999999999e-06 2024-08-17T17:30:31.182 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.182 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:31.183 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.099824+0000", 2024-08-17T17:30:31.183 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3963e-05 2024-08-17T17:30:31.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.183 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:31.183 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.099829+0000", 2024-08-17T17:30:31.183 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7060000000000001e-06 2024-08-17T17:30:31.183 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:31.183 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:31.183 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:31.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.183 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.45767.0:128797 7.2 e241/226)", 2024-08-17T17:30:31.183 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:30:31.100736+0000", 2024-08-17T17:30:31.183 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 4294967295.9998007, 2024-08-17T17:30:31.184 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00041855400000000002, 2024-08-17T17:30:31.184 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:31.184 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:31.184 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-08-17T17:30:31.184 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:31.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.184 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:31.184 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.100736+0000", 2024-08-17T17:30:31.184 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:31.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.184 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:31.184 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.100736+0000", 2024-08-17T17:30:31.184 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:31.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.185 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.185 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:31.185 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.100736+0000", 2024-08-17T17:30:31.185 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7299999999999995e-07 2024-08-17T17:30:31.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.185 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.185 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:31.185 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.100740+0000", 2024-08-17T17:30:31.185 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8970000000000001e-06 2024-08-17T17:30:31.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.185 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.185 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:31.185 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.100750+0000", 2024-08-17T17:30:31.185 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7929999999999992e-06 2024-08-17T17:30:31.186 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.186 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.186 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:31.186 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.100756+0000", 2024-08-17T17:30:31.186 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.353e-06 2024-08-17T17:30:31.186 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:31.186 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:31.186 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:31.186 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.186 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.186 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45767.0:128778 7.7 e241/226 7:ec29c320:::benchmark_data_smithi003_201798_object15556:head v 241'7986, mlcod=241'7984)", 2024-08-17T17:30:31.186 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:30:31.064913+0000", 2024-08-17T17:30:31.187 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.035623509999999997, 2024-08-17T17:30:31.187 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.036279413000000003, 2024-08-17T17:30:31.187 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:31.187 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:31.187 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:30:31.187 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:31.187 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.187 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:31.187 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.064913+0000", 2024-08-17T17:30:31.187 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:31.187 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.187 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.187 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:31.187 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.064913+0000", 2024-08-17T17:30:31.188 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:31.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.188 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:31.188 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.064913+0000", 2024-08-17T17:30:31.188 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0399999999999996e-07 2024-08-17T17:30:31.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.188 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:31.188 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.064918+0000", 2024-08-17T17:30:31.188 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4900000000000002e-06 2024-08-17T17:30:31.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.188 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:31.188 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.064927+0000", 2024-08-17T17:30:31.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6719999999999993e-06 2024-08-17T17:30:31.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:31.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.064933+0000", 2024-08-17T17:30:31.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7009999999999999e-06 2024-08-17T17:30:31.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:31.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.066126+0000", 2024-08-17T17:30:31.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001192987 2024-08-17T17:30:31.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:30:31.190 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.066147+0000", 2024-08-17T17:30:31.190 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0245999999999999e-05 2024-08-17T17:30:31.190 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:31.190 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:31.190 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:31.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.190 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:128788 7.1 7:8ffd8fb7:::benchmark_data_smithi003_201798_object15566:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-08-17T17:30:31.190 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:30:31.067983+0000", 2024-08-17T17:30:31.190 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.032553440000000003, 2024-08-17T17:30:31.190 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.03325782, 2024-08-17T17:30:31.190 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:31.190 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:31.190 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:30:31.191 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:30:31.191 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:30:31.191 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:30:31.191 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 128788 2024-08-17T17:30:31.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.191 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:31.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.191 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:31.191 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.067983+0000", 2024-08-17T17:30:31.191 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:31.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.191 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:31.191 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.067983+0000", 2024-08-17T17:30:31.192 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:31.192 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.192 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.192 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:31.192 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.067984+0000", 2024-08-17T17:30:31.192 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8400000000000003e-07 2024-08-17T17:30:31.192 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.192 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.192 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:31.192 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.067988+0000", 2024-08-17T17:30:31.192 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.065e-06 2024-08-17T17:30:31.192 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.192 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:31.193 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.067997+0000", 2024-08-17T17:30:31.193 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7159999999999998e-06 2024-08-17T17:30:31.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:31.193 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.068005+0000", 2024-08-17T17:30:31.193 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.537e-06 2024-08-17T17:30:31.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:31.193 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.077434+0000", 2024-08-17T17:30:31.193 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0094294200000000009 2024-08-17T17:30:31.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:30:31.194 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.077567+0000", 2024-08-17T17:30:31.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013269900000000001 2024-08-17T17:30:31.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.194 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:30:31.194 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.077690+0000", 2024-08-17T17:30:31.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000123084 2024-08-17T17:30:31.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.194 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:31.194 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.082701+0000", 2024-08-17T17:30:31.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0050105510000000002 2024-08-17T17:30:31.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.195 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:30:31.195 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.082716+0000", 2024-08-17T17:30:31.195 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5037e-05 2024-08-17T17:30:31.195 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:31.195 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:31.195 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:31.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.195 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45767.0:128789 7.7 e241/226 7:eac5d2ec:::benchmark_data_smithi003_201798_object15567:head v 241'7987, mlcod=241'7984)", 2024-08-17T17:30:31.195 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:30:31.071440+0000", 2024-08-17T17:30:31.195 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.029096227999999998, 2024-08-17T17:30:31.195 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.029868305000000001, 2024-08-17T17:30:31.195 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:31.196 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:31.196 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:30:31.196 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:31.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.196 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:31.196 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.071440+0000", 2024-08-17T17:30:31.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:31.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.196 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:31.196 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.071440+0000", 2024-08-17T17:30:31.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:31.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.197 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.197 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:31.197 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.071441+0000", 2024-08-17T17:30:31.197 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6300000000000004e-07 2024-08-17T17:30:31.197 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.197 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.197 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:31.197 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.071447+0000", 2024-08-17T17:30:31.198 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5890000000000002e-06 2024-08-17T17:30:31.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.198 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.198 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:31.198 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.071457+0000", 2024-08-17T17:30:31.198 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0739e-05 2024-08-17T17:30:31.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.198 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.198 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:31.198 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.071464+0000", 2024-08-17T17:30:31.198 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6409999999999996e-06 2024-08-17T17:30:31.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.199 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.199 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:31.199 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.074009+0000", 2024-08-17T17:30:31.199 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00254507 2024-08-17T17:30:31.199 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.199 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.199 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:30:31.199 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.074029+0000", 2024-08-17T17:30:31.199 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0310999999999999e-05 2024-08-17T17:30:31.199 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:31.199 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:31.199 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:31.199 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.199 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.199 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45767.0:128786 7.f e241/226 7:f41f2b7b:::benchmark_data_smithi003_201798_object15564:head v 241'8070, mlcod=241'8067)", 2024-08-17T17:30:31.200 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:30:31.074556+0000", 2024-08-17T17:30:31.200 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.025980012, 2024-08-17T17:30:31.200 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.026803500000000001, 2024-08-17T17:30:31.200 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:31.200 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:31.200 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:30:31.200 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:31.200 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.200 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:31.200 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.074556+0000", 2024-08-17T17:30:31.200 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:31.200 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.200 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.200 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:31.201 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.074556+0000", 2024-08-17T17:30:31.201 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:31.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.201 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:31.201 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.074558+0000", 2024-08-17T17:30:31.201 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.221e-06 2024-08-17T17:30:31.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.201 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:31.201 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.074565+0000", 2024-08-17T17:30:31.201 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6790000000000005e-06 2024-08-17T17:30:31.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.201 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:31.202 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.074577+0000", 2024-08-17T17:30:31.202 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1972e-05 2024-08-17T17:30:31.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.202 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.202 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:31.202 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.074585+0000", 2024-08-17T17:30:31.202 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6699999999999994e-06 2024-08-17T17:30:31.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.202 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.202 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:31.202 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.075920+0000", 2024-08-17T17:30:31.202 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001335464 2024-08-17T17:30:31.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:31.203 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:31.203 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:30:31.203 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:31.075939+0000", 2024-08-17T17:30:31.203 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8448999999999999e-05 2024-08-17T17:30:31.203 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:31.203 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:31.203 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:31.203 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:31.203 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-08-17T17:30:31.203 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 15 2024-08-17T17:30:31.203 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:30:31.204 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:31.248 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:31.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:31.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:31.350 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:31.377 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:31.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:31.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:31.479 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:31.520 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:31.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:31 smithi003 ceph-mon[103344]: pgmap v690: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 958 op/s 2024-08-17T17:30:31.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:31.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:31.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:31 smithi003 ceph-mon[100615]: pgmap v690: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 958 op/s 2024-08-17T17:30:31.624 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:31.663 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:31.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:31.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:31.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:31 smithi053 ceph-mon[89963]: pgmap v690: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 958 op/s 2024-08-17T17:30:31.765 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:31.805 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:31.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:31.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:31.867 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:30:31.907 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:31.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:31.966 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:31.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:32.067 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:32.109 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:32.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:32.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:32.211 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:32.265 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:32.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:32.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:32.339 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:30:32.366 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:32.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:32.466 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:32.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:32.567 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:32.637 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:32.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:32.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:32.739 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:32.766 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:32.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:32.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:32.868 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:32.892 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:32.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:32.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:32.925 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-17T17:30:32.994 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:33.059 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:33.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:33.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:33.160 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:33.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:33.186 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:33.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:33.287 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:33.315 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:33.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:33.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:33.417 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:33.461 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:33.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:33.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:33.523 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:30:33.563 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:33.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:33.652 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:33.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:33.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:33 smithi003 ceph-mon[100615]: pgmap v691: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 615 op/s 2024-08-17T17:30:33.691 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:33 smithi003 ceph-mon[103344]: pgmap v691: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 615 op/s 2024-08-17T17:30:33.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:33 smithi053 ceph-mon[89963]: pgmap v691: 89 pgs: 89 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 615 op/s 2024-08-17T17:30:33.753 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:33.781 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:33.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:33.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:33.883 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:33.942 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:33.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:33.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:34.044 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:34.090 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:34.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:34.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:34.192 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:34.217 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:34.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:34.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:34.308 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:30:34.319 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:34.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:34.375 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:34.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:34.476 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:34.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:34.506 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:34.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:34.607 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:34.634 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:34.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:34.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:34.735 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:34.783 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:34.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:34.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:34.884 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:34.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:34.915 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:34.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:35.016 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:35.047 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:35.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:35.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:35.149 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:35.175 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:35.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:35.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:35.277 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:35.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:35.339 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:35.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:35.440 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:35.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:35 smithi053 ceph-mon[89963]: pgmap v692: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 772 op/s 2024-08-17T17:30:35.468 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:35.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:35.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:35.570 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:35.616 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:35.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:35.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:35.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:35 smithi003 ceph-mon[100615]: pgmap v692: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 772 op/s 2024-08-17T17:30:35.633 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:35 smithi003 ceph-mon[103344]: pgmap v692: 89 pgs: 89 active+clean; 5.9 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 772 op/s 2024-08-17T17:30:35.718 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:35.751 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:35.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:35.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:35.853 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:35.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:35.895 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:35.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:35.906 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:30:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:30:35] "GET /metrics HTTP/1.1" 200 36506 "" "Prometheus/2.51.0" 2024-08-17T17:30:35.996 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:36.022 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:36.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:36.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:36.124 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:36.164 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:36.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:36.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:36.266 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:36.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:36.310 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:36.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:36.411 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:36.436 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:36.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:36.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:36.438 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:30:36.438 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:30:36.478 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:30:36.479 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:30:36.529 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:30:36.529 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:30:36.538 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:36.561 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:36.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:36.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:36.583 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:30:36.584 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:30:36.619 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:30:36.619 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:30:36.656 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:30:36.656 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:30:36.663 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:36.698 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:30:36.698 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:30:36.709 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:36.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:36.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:36.740 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:30:36.741 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:30:36.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:36.860 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:36.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:36.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:36.962 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:36.987 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:36.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:36.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:37.089 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:37.114 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:37.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:37.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:37.216 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:37.242 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:37.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:37.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:37.333 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.3 to 1 (10000) 2024-08-17T17:30:37.344 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:37.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:37 smithi003 ceph-mon[100615]: pgmap v693: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 802 op/s 2024-08-17T17:30:37.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:37 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/4104584843' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:30:37.354 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:37 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/207700808' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:30:37.354 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:37 smithi003 ceph-mon[103344]: pgmap v693: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 802 op/s 2024-08-17T17:30:37.354 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:37 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/4104584843' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:30:37.354 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:37 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/207700808' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:30:37.367 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:37.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:37.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:37.469 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:37.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:37.510 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:37.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:37.611 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:37.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:37.637 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:37.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:37.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:37 smithi053 ceph-mon[89963]: pgmap v693: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 802 op/s 2024-08-17T17:30:37.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:37 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/4104584843' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:30:37.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:37 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/207700808' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:30:37.739 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:37.773 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:37.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:37.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:37.876 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:37.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:37.921 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:37.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:38.023 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:38.057 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:38.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:38.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:38.159 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:38.186 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:38.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:38.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:38.288 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:38.332 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:38.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:38.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:38.435 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:38.478 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:38.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:38.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:38.579 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:38.619 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:38.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:38.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:38.686 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:38 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/207700808' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-17T17:30:38.687 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:38 smithi003 ceph-mon[103344]: osdmap e242: 8 total, 8 up, 7 in 2024-08-17T17:30:38.687 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:30:38.687 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:30:38.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:38 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/207700808' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-17T17:30:38.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:38 smithi003 ceph-mon[100615]: osdmap e242: 8 total, 8 up, 7 in 2024-08-17T17:30:38.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:30:38.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:30:38.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:38 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/207700808' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-17T17:30:38.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:38 smithi053 ceph-mon[89963]: osdmap e242: 8 total, 8 up, 7 in 2024-08-17T17:30:38.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:30:38.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:30:38.721 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:38.760 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:38.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:38.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:38.862 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:38.909 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:38.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:38.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:38.919 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:30:39.012 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:39.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:39.068 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:39.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:39.170 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:39.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:39.213 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:39.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:39.314 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:39.339 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:39.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:39.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:39.441 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:39.465 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:39.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:39.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:39.567 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:39.596 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 5 1 2024-08-17T17:30:39.614 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:39.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:39.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:39.651 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:39 smithi003 ceph-mon[103344]: pgmap v695: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 825 op/s 2024-08-17T17:30:39.651 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:39 smithi003 ceph-mon[100615]: pgmap v695: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 825 op/s 2024-08-17T17:30:39.716 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:39.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:39 smithi053 ceph-mon[89963]: pgmap v695: 89 pgs: 89 active+clean; 5.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 825 op/s 2024-08-17T17:30:39.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:39.777 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:39.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:39.878 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:39.904 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:39.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:39.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:40.005 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:40.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:40.048 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:40.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:40.149 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:40.174 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:40.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:40.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:40.276 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:40.332 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:40.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:40.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:40.434 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:40.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:30:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:30:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:30:40.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:40.474 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:40.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:40.575 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:40.637 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:40.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:40.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:40.739 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:40.765 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:40.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:40.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:40.866 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:40.904 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:40.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:40.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:41.009 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:41.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:41.049 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:41.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:41.150 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:41.175 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:41.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:41.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:41.277 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:41.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:41.305 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:41.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:41.406 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:41.433 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:41.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:41.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:41.534 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:41.558 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:41.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:41.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:41.660 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:41.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:41.686 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:41.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:41.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:41 smithi053 ceph-mon[89963]: pgmap v696: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 805 op/s 2024-08-17T17:30:41.762 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:30:41.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:41 smithi003 ceph-mon[100615]: pgmap v696: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 805 op/s 2024-08-17T17:30:41.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:41 smithi003 ceph-mon[103344]: pgmap v696: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 805 op/s 2024-08-17T17:30:41.787 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:41.811 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:41.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:41.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:41.913 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:41.959 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:41.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:41.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:42.061 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:42.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:42.107 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:42.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:42.208 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:42.247 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:42.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:42.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:42.349 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:42.375 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:42.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:42.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:42.394 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-08-17T17:30:42.476 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:42.521 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:42.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:42.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:42.522 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:42 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2251646024' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:30:42.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:42 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:30:42.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:42 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2251646024' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:30:42.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:42 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:30:42.586 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:30:42.623 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:42.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:42.671 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:42.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:42.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:42 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2251646024' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:30:42.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:42 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:30:42.772 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:42.814 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:42.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:42.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:42.916 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:42.940 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:42.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:42.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:43.042 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:43.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:43.077 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:43.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:43.178 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:43.203 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:43.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:43.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:43.305 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:43.351 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:43.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:43.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:43.453 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:43.492 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:43.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:43.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:43.594 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:43.628 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:43.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:43.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:43.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:43 smithi053 ceph-mon[89963]: pgmap v697: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 809 op/s 2024-08-17T17:30:43.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:43 smithi053 ceph-mon[89963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:30:43.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:43 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:30:43.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:43 smithi053 ceph-mon[89963]: osdmap e243: 8 total, 8 up, 7 in 2024-08-17T17:30:43.729 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:43.762 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:43.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:43.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:43.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:43 smithi003 ceph-mon[103344]: pgmap v697: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 809 op/s 2024-08-17T17:30:43.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:43 smithi003 ceph-mon[103344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:30:43.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:43 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:30:43.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:43 smithi003 ceph-mon[103344]: osdmap e243: 8 total, 8 up, 7 in 2024-08-17T17:30:43.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:43 smithi003 ceph-mon[100615]: pgmap v697: 89 pgs: 89 active+clean; 5.6 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 809 op/s 2024-08-17T17:30:43.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:43 smithi003 ceph-mon[100615]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:30:43.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:43 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:30:43.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:43 smithi003 ceph-mon[100615]: osdmap e243: 8 total, 8 up, 7 in 2024-08-17T17:30:43.865 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:43.889 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:43.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:43.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:43.991 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:44.016 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:44.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:44.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:44.118 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:44.145 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:44.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:44.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:44.248 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:44.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:44.294 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:44.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:44.395 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:44.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:44.419 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:44.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:44.520 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:44.566 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:44.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:44.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:44.669 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:44.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:44.694 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:44.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:44.796 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:44.822 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:44.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:44.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:44.924 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:44.946 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:44.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:44.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:45.048 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:45.091 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:45.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:45.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:45.193 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:45.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:45.234 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:45.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:45.334 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:45.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:45.374 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:45.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:45.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:45 smithi053 ceph-mon[89963]: pgmap v699: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 879 op/s 2024-08-17T17:30:45.475 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:45.501 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:45.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:45.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:45.603 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:45.637 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:45 smithi003 ceph-mon[103344]: pgmap v699: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 879 op/s 2024-08-17T17:30:45.637 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:45 smithi003 ceph-mon[100615]: pgmap v699: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 879 op/s 2024-08-17T17:30:45.641 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:45.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:45.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:45.743 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:45.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:45.787 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:45.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:45.888 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:45.910 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:45.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:45.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:46.012 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:46.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:30:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:30:45] "GET /metrics HTTP/1.1" 200 36509 "" "Prometheus/2.51.0" 2024-08-17T17:30:46.056 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:46.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:46.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:46.158 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:46.182 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:46.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:46.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:46.284 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:46.309 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:46.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:46.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:46.411 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:46.436 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:46.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:46.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:46.538 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:46.564 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:46.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:46.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:46.666 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:46.690 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:46.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:46.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:46.792 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:46.833 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:46.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:46.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:46.934 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:46.985 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:46.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:46.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:47.087 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:47.113 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:47.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:47.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:47.216 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:47.257 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:47.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:47.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:47.358 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:47.408 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:47.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:47.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:47.510 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:47.536 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:47.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:47.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:47.638 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:47.679 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:47 smithi003 ceph-mon[100615]: pgmap v700: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 925 op/s 2024-08-17T17:30:47.679 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:47 smithi003 ceph-mon[103344]: pgmap v700: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 925 op/s 2024-08-17T17:30:47.693 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:47.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:47.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:47.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:47 smithi053 ceph-mon[89963]: pgmap v700: 89 pgs: 89 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 925 op/s 2024-08-17T17:30:47.795 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:47.818 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:47.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:47.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:47.920 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:47.944 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:47.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:47.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:48.045 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:48.069 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:48.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:48.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:48.171 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:48.195 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:48.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:48.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:48.297 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:48.321 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:48.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:48.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:48.423 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:48.449 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:48.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:48.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:48.550 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:48.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:48.637 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:48.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:48.738 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:48.763 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:48.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:48.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:48.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:48.900 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:48.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:48.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:49.002 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:49.028 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:49.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:49.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:49.130 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:49.156 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:49.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:49.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:49.159 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:30:49.159 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:30:49.259 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:49.283 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:49.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:49.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:49.384 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:49.410 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:49.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:49.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:49.512 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:49.540 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:49.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:49.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:49.641 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:49.681 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:49.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:49.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:49.706 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:49 smithi003 ceph-mon[103344]: pgmap v701: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 804 op/s 2024-08-17T17:30:49.706 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:49 smithi003 ceph-mon[100615]: pgmap v701: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 804 op/s 2024-08-17T17:30:49.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:49 smithi053 ceph-mon[89963]: pgmap v701: 89 pgs: 89 active+clean; 5.3 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 804 op/s 2024-08-17T17:30:49.783 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:49.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:49.831 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:49.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:49.848 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-17T17:30:49.932 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:49.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:49.977 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:49.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:50.078 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:50.102 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:50.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:50.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:50.204 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:50.211 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:30:50.283 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:50.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:50.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:50.386 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:50.411 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:50.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:50.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:50.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:30:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:30:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:30:50.512 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:50.554 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:30:50.574 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:50.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:50.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:50.676 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:50.744 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:50.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:50.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:50.846 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:50.872 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:50.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:50.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:50.974 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:50.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:50.998 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:50.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:51.055 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:30:51.099 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:51.140 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:51.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:51.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:51.242 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:51.267 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:51.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:51.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:51.368 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:51.395 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:51.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:51.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:51.404 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.5 to 1 (10000) 2024-08-17T17:30:51.497 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:51.533 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:51.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:51.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:51.635 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:51.682 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:51.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:51.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:51.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:51 smithi053 ceph-mon[89963]: pgmap v702: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 787 op/s 2024-08-17T17:30:51.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:51 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3041892709' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:30:51.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:51 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:30:51.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:51 smithi003 ceph-mon[100615]: pgmap v702: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 787 op/s 2024-08-17T17:30:51.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:51 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3041892709' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:30:51.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:51 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:30:51.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:51 smithi003 ceph-mon[103344]: pgmap v702: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 787 op/s 2024-08-17T17:30:51.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:51 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3041892709' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:30:51.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:51 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:30:51.784 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:51.846 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:51.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:51.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:51.948 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:51.974 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:51.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:51.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:52.076 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:52.113 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:52.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:52.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:52.215 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:52.253 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:52.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:52.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:52.355 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:52.380 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:52.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:52.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:52.482 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:52.507 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:52.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:52.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:52.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:52.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:52.650 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:52.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:52.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:52 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-17T17:30:52.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:52 smithi053 ceph-mon[89963]: osdmap e244: 8 total, 8 up, 7 in 2024-08-17T17:30:52.751 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:52.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:52 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-17T17:30:52.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:52 smithi003 ceph-mon[103344]: osdmap e244: 8 total, 8 up, 7 in 2024-08-17T17:30:52.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:52 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-17T17:30:52.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:52 smithi003 ceph-mon[100615]: osdmap e244: 8 total, 8 up, 7 in 2024-08-17T17:30:52.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:52.780 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:52.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:52.881 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:52.907 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:52.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:52.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:53.009 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:53.045 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:53.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:53.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:53.147 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:53.174 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:53.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:53.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:53.273 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 6 1 2024-08-17T17:30:53.275 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:53.300 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:53.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:53.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:53.402 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:53.445 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:53.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:53.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:53.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:53 smithi003 ceph-mon[100615]: pgmap v704: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 710 op/s 2024-08-17T17:30:53.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:30:53.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:53 smithi003 ceph-mon[103344]: pgmap v704: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 710 op/s 2024-08-17T17:30:53.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:30:53.547 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:53.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:53.607 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:53.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:53.707 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:53.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:53 smithi053 ceph-mon[89963]: pgmap v704: 89 pgs: 89 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 710 op/s 2024-08-17T17:30:53.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:30:53.752 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:53.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:53.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:53.854 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:53.900 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:53.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:53.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:54.002 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:54.027 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:54.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:54.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:54.129 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:54.165 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:30:54.165 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:30:54.174 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:54.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:54.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:54.210 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:30:54.211 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:30:54.257 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:30:54.257 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:30:54.276 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:54.298 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:30:54.298 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:30:54.325 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:54.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:54.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:54.331 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:30:54.331 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:30:54.364 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:30:54.365 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:30:54.397 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:30:54.397 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:30:54.427 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:54.430 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:30:54.430 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:30:54.452 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:54.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:54.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:54.553 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:54.580 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:54.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:54.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:54.682 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:54.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:54 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1234119362' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:30:54.727 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:54.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:54.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:54.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:54 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1234119362' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:30:54.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:54 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1234119362' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:30:54.830 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:54.870 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:54.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:54.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:54.974 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:54.999 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:55.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:55.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:55.101 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:55.144 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:55.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:55.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:55.246 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:55.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:55.271 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:55.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:55.372 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:55.397 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:55.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:55.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:55.499 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:55.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:55 smithi003 ceph-mon[103344]: pgmap v705: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 768 op/s 2024-08-17T17:30:55.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:55 smithi003 ceph-mon[100615]: pgmap v705: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 768 op/s 2024-08-17T17:30:55.525 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:55.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:55.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:55.627 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:55.655 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:55.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:55.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:55.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:55 smithi053 ceph-mon[89963]: pgmap v705: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 768 op/s 2024-08-17T17:30:55.757 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:55.797 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:55.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:55.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:55.876 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:30:55.899 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:30:55.923 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:55.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:55.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:30:55.967 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:30:56.025 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:56.026 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:30:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:30:55] "GET /metrics HTTP/1.1" 200 36505 "" "Prometheus/2.51.0" 2024-08-17T17:30:56.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:56.072 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:56.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:56.173 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:56.222 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:56.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:56.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:56.324 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:56.350 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:56.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:56.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:56.453 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:30:56.514 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:56.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:56.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:30:56.616 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:56.643 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:56.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:56.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:56.745 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:56.772 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:56.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:56.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:56.874 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:56.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:56.913 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:56.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:57.015 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:57.041 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:57.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:57.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:57.143 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:57.187 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:57.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:57.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:57.288 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:57.294 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:30:57.315 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:57.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:57.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:57.417 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:57.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:57.454 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:57.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:57.517 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:30:57.517 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-08-17T17:30:57.518 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-08-17T17:30:57.518 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:30:57.518 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.518 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45491.0:51014 6.c 6:3b1e47a1:::benchmark_data_smithi003_175043_object51013:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-08-17T17:30:57.518 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:25:03.970975+0000", 2024-08-17T17:30:57.518 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 353.53769906899998, 2024-08-17T17:30:57.518 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.707060298, 2024-08-17T17:30:57.518 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:57.519 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:57.519 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:30:57.519 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:30:57.519 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45491", 2024-08-17T17:30:57.519 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:30:57.519 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 51014 2024-08-17T17:30:57.519 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.519 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:57.519 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.519 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:57.519 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.970975+0000", 2024-08-17T17:30:57.519 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.520 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.520 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.520 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:57.520 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.970975+0000", 2024-08-17T17:30:57.520 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.520 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.520 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.520 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:57.520 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.970976+0000", 2024-08-17T17:30:57.520 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.443e-06 2024-08-17T17:30:57.520 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.520 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.521 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:57.521 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.971002+0000", 2024-08-17T17:30:57.521 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5477999999999999e-05 2024-08-17T17:30:57.521 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.521 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.521 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:57.521 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.971015+0000", 2024-08-17T17:30:57.521 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3555e-05 2024-08-17T17:30:57.521 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.521 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.521 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:57.521 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.971023+0000", 2024-08-17T17:30:57.521 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3900000000000004e-06 2024-08-17T17:30:57.522 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.522 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.522 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:57.522 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.975076+0000", 2024-08-17T17:30:57.522 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040535240000000002 2024-08-17T17:30:57.522 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.522 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.522 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:30:57.522 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.975207+0000", 2024-08-17T17:30:57.522 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000130992 2024-08-17T17:30:57.522 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.523 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.523 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-08-17T17:30:57.523 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.975338+0000", 2024-08-17T17:30:57.523 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000130987 2024-08-17T17:30:57.523 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.523 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.523 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.523 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.990507+0000", 2024-08-17T17:30:57.523 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.015168921 2024-08-17T17:30:57.523 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.523 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.523 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:30:57.523 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.990522+0000", 2024-08-17T17:30:57.524 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5148e-05 2024-08-17T17:30:57.524 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.524 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.524 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:57.524 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.995576+0000", 2024-08-17T17:30:57.524 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0050539399999999998 2024-08-17T17:30:57.524 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.524 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.524 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:30:57.524 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.995590+0000", 2024-08-17T17:30:57.524 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3723000000000001e-05 2024-08-17T17:30:57.524 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.524 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.524 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:57.524 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:04.505851+0000", 2024-08-17T17:30:57.524 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.51026153500000004 2024-08-17T17:30:57.525 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.525 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.525 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:30:57.525 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:04.505866+0000", 2024-08-17T17:30:57.525 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4335000000000001e-05 2024-08-17T17:30:57.525 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.525 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.525 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:57.525 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:05.677960+0000", 2024-08-17T17:30:57.525 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.172094752 2024-08-17T17:30:57.525 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.525 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.525 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:30:57.525 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:05.678035+0000", 2024-08-17T17:30:57.526 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4574999999999997e-05 2024-08-17T17:30:57.526 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.526 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:57.526 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.526 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.526 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.526 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45491.0:51030 6.c 6:39c4a7e1:::benchmark_data_smithi003_175043_object51029:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-08-17T17:30:57.526 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:25:03.982807+0000", 2024-08-17T17:30:57.526 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 353.52586711700002, 2024-08-17T17:30:57.526 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7154847520000001, 2024-08-17T17:30:57.526 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:57.526 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:57.526 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:30:57.527 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:30:57.527 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45491", 2024-08-17T17:30:57.527 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:30:57.527 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 51030 2024-08-17T17:30:57.527 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.527 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:57.527 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.527 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:57.527 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.982807+0000", 2024-08-17T17:30:57.527 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.527 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.527 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.527 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:57.527 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.982807+0000", 2024-08-17T17:30:57.527 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.528 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.528 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.528 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:57.528 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.982807+0000", 2024-08-17T17:30:57.528 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3500000000000002e-07 2024-08-17T17:30:57.528 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.528 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.528 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:57.528 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.982837+0000", 2024-08-17T17:30:57.528 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9655999999999999e-05 2024-08-17T17:30:57.528 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.528 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.528 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:57.528 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.982844+0000", 2024-08-17T17:30:57.528 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1029999999999996e-06 2024-08-17T17:30:57.529 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.529 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.529 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:57.529 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.982849+0000", 2024-08-17T17:30:57.529 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8670000000000002e-06 2024-08-17T17:30:57.529 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.529 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.529 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:57.529 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.996164+0000", 2024-08-17T17:30:57.529 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013315268 2024-08-17T17:30:57.529 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.529 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.529 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:30:57.529 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.996178+0000", 2024-08-17T17:30:57.529 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3876999999999999e-05 2024-08-17T17:30:57.530 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.530 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.530 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:57.530 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:04.506236+0000", 2024-08-17T17:30:57.530 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.51005823800000005 2024-08-17T17:30:57.530 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.530 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.530 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:30:57.530 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:04.506250+0000", 2024-08-17T17:30:57.530 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3875e-05 2024-08-17T17:30:57.530 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.530 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.530 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:57.531 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:05.680158+0000", 2024-08-17T17:30:57.531 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1739081920000001 2024-08-17T17:30:57.531 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.531 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.531 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:30:57.531 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:05.680328+0000", 2024-08-17T17:30:57.531 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016995000000000001 2024-08-17T17:30:57.531 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.531 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.531 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,6", 2024-08-17T17:30:57.531 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:05.680581+0000", 2024-08-17T17:30:57.531 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00025260199999999999 2024-08-17T17:30:57.531 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.531 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.532 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:30:57.532 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:05.686170+0000", 2024-08-17T17:30:57.532 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0055897220000000001 2024-08-17T17:30:57.532 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.532 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.532 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.532 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:05.695445+0000", 2024-08-17T17:30:57.532 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0092747639999999996 2024-08-17T17:30:57.532 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.532 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.532 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.532 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:05.695511+0000", 2024-08-17T17:30:57.532 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6025000000000006e-05 2024-08-17T17:30:57.533 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.533 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.533 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:30:57.533 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:05.695579+0000", 2024-08-17T17:30:57.533 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7936999999999997e-05 2024-08-17T17:30:57.533 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.533 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.533 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:30:57.533 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:05.698291+0000", 2024-08-17T17:30:57.533 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027122409999999998 2024-08-17T17:30:57.533 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.533 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:57.533 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.533 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.533 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.534 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45767.0:79206 7.f e234/226 7:f23fe75d:::benchmark_data_smithi003_201798_object79205:head v 234'5014, mlcod=234'5012)", 2024-08-17T17:30:57.534 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:29:32.482333+0000", 2024-08-17T17:30:57.534 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 85.026340540999996, 2024-08-17T17:30:57.534 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.101453185, 2024-08-17T17:30:57.534 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:57.534 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:57.534 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:30:57.534 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:57.534 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.534 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:57.534 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:32.482333+0000", 2024-08-17T17:30:57.534 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.534 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.535 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.535 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:57.535 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:32.482333+0000", 2024-08-17T17:30:57.535 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.535 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.535 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:57.535 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:32.482334+0000", 2024-08-17T17:30:57.535 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0900000000000002e-07 2024-08-17T17:30:57.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.535 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.535 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:57.535 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:32.482359+0000", 2024-08-17T17:30:57.535 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.514e-05 2024-08-17T17:30:57.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.536 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.536 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:57.536 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:32.482367+0000", 2024-08-17T17:30:57.536 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5720000000000003e-06 2024-08-17T17:30:57.536 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.536 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.536 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:57.536 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:32.482373+0000", 2024-08-17T17:30:57.536 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9859999999999999e-06 2024-08-17T17:30:57.536 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.536 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.536 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:57.536 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:32.566105+0000", 2024-08-17T17:30:57.537 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.083731360000000005 2024-08-17T17:30:57.537 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.537 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.537 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:30:57.537 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:32.566125+0000", 2024-08-17T17:30:57.537 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0098999999999998e-05 2024-08-17T17:30:57.537 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.537 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.537 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:30:57.537 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:32.583761+0000", 2024-08-17T17:30:57.537 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017635911000000001 2024-08-17T17:30:57.537 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.537 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.537 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:30:57.538 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:32.583786+0000", 2024-08-17T17:30:57.538 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5707999999999998e-05 2024-08-17T17:30:57.538 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.538 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:57.538 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.538 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.538 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.538 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:80092 7.a 7:5f458f91:::benchmark_data_smithi003_201798_object80091:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-17T17:30:57.538 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:29:33.380788+0000", 2024-08-17T17:30:57.538 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 84.127886028000006, 2024-08-17T17:30:57.538 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.19190290800000001, 2024-08-17T17:30:57.538 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:57.539 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:57.539 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:30:57.539 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:30:57.539 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:30:57.539 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:30:57.539 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 80092 2024-08-17T17:30:57.539 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.539 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:57.539 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.539 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:57.539 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.380788+0000", 2024-08-17T17:30:57.539 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.539 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.539 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.539 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:57.540 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.380788+0000", 2024-08-17T17:30:57.540 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.540 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.540 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.540 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:57.540 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.380789+0000", 2024-08-17T17:30:57.540 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.376e-06 2024-08-17T17:30:57.540 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.540 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.540 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:57.540 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.380820+0000", 2024-08-17T17:30:57.540 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0960000000000002e-05 2024-08-17T17:30:57.540 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.540 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.540 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:57.541 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.380832+0000", 2024-08-17T17:30:57.541 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2342e-05 2024-08-17T17:30:57.541 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.541 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.541 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:57.541 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.380839+0000", 2024-08-17T17:30:57.541 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9460000000000001e-06 2024-08-17T17:30:57.541 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.541 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.541 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:57.541 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.385442+0000", 2024-08-17T17:30:57.541 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004602826 2024-08-17T17:30:57.541 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.541 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.542 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:30:57.542 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.385923+0000", 2024-08-17T17:30:57.542 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00048045500000000002 2024-08-17T17:30:57.542 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.542 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.542 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:30:57.542 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.386043+0000", 2024-08-17T17:30:57.542 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012019999999999999 2024-08-17T17:30:57.542 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.542 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.542 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.542 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.392281+0000", 2024-08-17T17:30:57.542 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006238548 2024-08-17T17:30:57.543 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.543 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.543 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:30:57.543 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.392305+0000", 2024-08-17T17:30:57.543 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3204e-05 2024-08-17T17:30:57.543 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.543 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.543 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.543 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.572586+0000", 2024-08-17T17:30:57.543 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18028166300000001 2024-08-17T17:30:57.543 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.543 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.543 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:30:57.544 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.572644+0000", 2024-08-17T17:30:57.544 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7756000000000001e-05 2024-08-17T17:30:57.544 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.544 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.544 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:30:57.544 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.572691+0000", 2024-08-17T17:30:57.544 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6631999999999997e-05 2024-08-17T17:30:57.544 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.544 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:57.544 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.544 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.544 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.544 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:80095 7.3 7:c369a586:::benchmark_data_smithi003_201798_object80094:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-17T17:30:57.544 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:29:33.381510+0000", 2024-08-17T17:30:57.545 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 84.127163666000001, 2024-08-17T17:30:57.545 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18715829, 2024-08-17T17:30:57.545 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:57.545 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:57.545 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:30:57.545 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:30:57.545 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:30:57.545 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:30:57.545 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 80095 2024-08-17T17:30:57.545 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.545 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:57.545 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.545 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:57.545 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.381510+0000", 2024-08-17T17:30:57.545 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.546 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.546 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.546 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:57.546 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.381510+0000", 2024-08-17T17:30:57.546 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.546 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.546 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.546 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:57.546 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.381511+0000", 2024-08-17T17:30:57.546 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9400000000000004e-07 2024-08-17T17:30:57.546 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.546 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.546 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:57.547 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.381542+0000", 2024-08-17T17:30:57.547 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0762000000000001e-05 2024-08-17T17:30:57.547 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.547 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.547 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:57.547 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.381553+0000", 2024-08-17T17:30:57.547 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0811e-05 2024-08-17T17:30:57.547 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.547 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.547 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:57.547 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.381559+0000", 2024-08-17T17:30:57.547 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3980000000000004e-06 2024-08-17T17:30:57.547 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.548 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.548 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:57.548 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.381659+0000", 2024-08-17T17:30:57.548 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9894000000000003e-05 2024-08-17T17:30:57.548 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.548 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.548 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:30:57.548 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.381804+0000", 2024-08-17T17:30:57.548 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014564800000000001 2024-08-17T17:30:57.548 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.548 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.548 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,6", 2024-08-17T17:30:57.548 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.381953+0000", 2024-08-17T17:30:57.548 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000148305 2024-08-17T17:30:57.549 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.549 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.549 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.549 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389248+0000", 2024-08-17T17:30:57.549 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072953180000000003 2024-08-17T17:30:57.549 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.549 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.549 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:30:57.549 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389260+0000", 2024-08-17T17:30:57.549 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1849e-05 2024-08-17T17:30:57.549 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.550 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.550 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.550 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.568485+0000", 2024-08-17T17:30:57.550 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17922486300000001 2024-08-17T17:30:57.550 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.550 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.550 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:30:57.550 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.568619+0000", 2024-08-17T17:30:57.550 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000133855 2024-08-17T17:30:57.550 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.550 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.550 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:30:57.550 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.568668+0000", 2024-08-17T17:30:57.551 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9793000000000003e-05 2024-08-17T17:30:57.551 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.551 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:57.551 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.551 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.551 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.551 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:80096 7.a 7:54a4ec2f:::benchmark_data_smithi003_201798_object80095:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-17T17:30:57.551 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:29:33.381575+0000", 2024-08-17T17:30:57.551 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 84.127099103000006, 2024-08-17T17:30:57.551 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.19489894499999999, 2024-08-17T17:30:57.551 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:57.551 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:57.551 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:30:57.551 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:30:57.552 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:30:57.552 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:30:57.552 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 80096 2024-08-17T17:30:57.552 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.552 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:57.552 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.552 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:57.552 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.381575+0000", 2024-08-17T17:30:57.552 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.552 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.552 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.552 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:57.552 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.381575+0000", 2024-08-17T17:30:57.553 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.553 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.553 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.553 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:57.553 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.381575+0000", 2024-08-17T17:30:57.553 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0100000000000005e-07 2024-08-17T17:30:57.553 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.553 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.553 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:57.553 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.381594+0000", 2024-08-17T17:30:57.553 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8445e-05 2024-08-17T17:30:57.553 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.554 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.554 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:57.554 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.381600+0000", 2024-08-17T17:30:57.554 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7349999999999999e-06 2024-08-17T17:30:57.554 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.554 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.554 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:57.554 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.381607+0000", 2024-08-17T17:30:57.554 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6429999999999998e-06 2024-08-17T17:30:57.554 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.554 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.554 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:57.554 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389348+0000", 2024-08-17T17:30:57.554 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0077407489999999999 2024-08-17T17:30:57.555 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.555 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.555 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:30:57.555 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389470+0000", 2024-08-17T17:30:57.555 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012184900000000001 2024-08-17T17:30:57.555 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.555 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.555 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:30:57.555 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389642+0000", 2024-08-17T17:30:57.555 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017195000000000001 2024-08-17T17:30:57.555 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.555 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.555 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.556 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.392624+0000", 2024-08-17T17:30:57.556 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029826520000000001 2024-08-17T17:30:57.556 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.556 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.556 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:30:57.556 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.395467+0000", 2024-08-17T17:30:57.556 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002842883 2024-08-17T17:30:57.556 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.556 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.556 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.556 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.576369+0000", 2024-08-17T17:30:57.556 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18090183200000001 2024-08-17T17:30:57.556 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.557 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.557 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:30:57.557 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.576427+0000", 2024-08-17T17:30:57.557 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8245999999999998e-05 2024-08-17T17:30:57.557 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.557 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.557 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:30:57.557 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.576474+0000", 2024-08-17T17:30:57.557 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6459999999999999e-05 2024-08-17T17:30:57.557 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.557 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:57.557 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.558 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.558 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.558 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:80097 7.3 7:cd0150f4:::benchmark_data_smithi003_201798_object80096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-17T17:30:57.558 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:29:33.382635+0000", 2024-08-17T17:30:57.558 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 84.126038993999998, 2024-08-17T17:30:57.558 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18676747299999999, 2024-08-17T17:30:57.558 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:57.558 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:57.558 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:30:57.558 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:30:57.558 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:30:57.558 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:30:57.558 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 80097 2024-08-17T17:30:57.558 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.559 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:57.559 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.559 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:57.559 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.382635+0000", 2024-08-17T17:30:57.559 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.559 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.559 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.559 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:57.559 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.382635+0000", 2024-08-17T17:30:57.559 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.559 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.559 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.559 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:57.559 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.382636+0000", 2024-08-17T17:30:57.560 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3400000000000001e-06 2024-08-17T17:30:57.560 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.560 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.560 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:57.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:57.561 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.382657+0000", 2024-08-17T17:30:57.561 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0624000000000001e-05 2024-08-17T17:30:57.561 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.561 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.561 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:57.561 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.382668+0000", 2024-08-17T17:30:57.562 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1354e-05 2024-08-17T17:30:57.562 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.562 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.562 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:57.562 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.382675+0000", 2024-08-17T17:30:57.562 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1559999999999998e-06 2024-08-17T17:30:57.562 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.562 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.562 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:57.562 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.385528+0000", 2024-08-17T17:30:57.562 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028528249999999998 2024-08-17T17:30:57.562 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.562 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.563 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:30:57.563 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.385862+0000", 2024-08-17T17:30:57.563 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00033433600000000002 2024-08-17T17:30:57.563 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.563 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.563 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,6", 2024-08-17T17:30:57.563 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.386007+0000", 2024-08-17T17:30:57.563 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014439800000000001 2024-08-17T17:30:57.563 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.563 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.563 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.563 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.392256+0000", 2024-08-17T17:30:57.563 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0062494899999999999 2024-08-17T17:30:57.563 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.564 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.564 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:30:57.564 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.392290+0000", 2024-08-17T17:30:57.564 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3670999999999997e-05 2024-08-17T17:30:57.564 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.564 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.564 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.564 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.569320+0000", 2024-08-17T17:30:57.564 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.177029614 2024-08-17T17:30:57.564 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.564 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.564 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:30:57.564 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.569367+0000", 2024-08-17T17:30:57.564 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7874999999999998e-05 2024-08-17T17:30:57.565 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.565 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.565 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:30:57.565 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.569402+0000", 2024-08-17T17:30:57.565 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4789999999999997e-05 2024-08-17T17:30:57.565 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.565 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:57.565 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.565 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.565 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.565 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:80100 7.a 7:50c869f6:::benchmark_data_smithi003_201798_object80099:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-17T17:30:57.565 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:29:33.382998+0000", 2024-08-17T17:30:57.565 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 84.125675606000001, 2024-08-17T17:30:57.566 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.193590702, 2024-08-17T17:30:57.566 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:57.566 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:57.566 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:30:57.566 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:30:57.566 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:30:57.566 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:30:57.566 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 80100 2024-08-17T17:30:57.566 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.566 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:57.566 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.566 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:57.566 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.382998+0000", 2024-08-17T17:30:57.567 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.567 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.567 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.567 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:57.567 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.382998+0000", 2024-08-17T17:30:57.567 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.567 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.567 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.567 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:57.567 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.382999+0000", 2024-08-17T17:30:57.568 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4900000000000005e-07 2024-08-17T17:30:57.568 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.568 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:57.568 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.383020+0000", 2024-08-17T17:30:57.568 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0601000000000001e-05 2024-08-17T17:30:57.568 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.568 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:57.568 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.383029+0000", 2024-08-17T17:30:57.568 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4919999999999999e-06 2024-08-17T17:30:57.568 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.569 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:57.569 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.383035+0000", 2024-08-17T17:30:57.569 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1469999999999998e-06 2024-08-17T17:30:57.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.569 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.569 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:57.569 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.392708+0000", 2024-08-17T17:30:57.569 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0096729090000000004 2024-08-17T17:30:57.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.569 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.569 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:30:57.569 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.392839+0000", 2024-08-17T17:30:57.569 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013080899999999999 2024-08-17T17:30:57.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.570 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:30:57.570 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.392938+0000", 2024-08-17T17:30:57.570 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.912e-05 2024-08-17T17:30:57.570 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.570 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.570 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.395527+0000", 2024-08-17T17:30:57.570 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00258859 2024-08-17T17:30:57.570 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.570 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:30:57.570 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.398822+0000", 2024-08-17T17:30:57.570 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032956299999999999 2024-08-17T17:30:57.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.571 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.571 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.576509+0000", 2024-08-17T17:30:57.571 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17768704900000001 2024-08-17T17:30:57.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.571 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:30:57.571 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.576557+0000", 2024-08-17T17:30:57.571 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7812000000000002e-05 2024-08-17T17:30:57.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.571 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:30:57.571 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.576589+0000", 2024-08-17T17:30:57.572 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1794000000000001e-05 2024-08-17T17:30:57.572 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.572 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:57.572 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.572 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.572 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:80101 7.2 7:42bf2a73:::benchmark_data_smithi003_201798_object80100:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-17T17:30:57.572 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:29:33.386150+0000", 2024-08-17T17:30:57.572 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 84.122523373000007, 2024-08-17T17:30:57.572 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.19212204899999999, 2024-08-17T17:30:57.572 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:57.572 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:57.572 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:30:57.572 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:30:57.572 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:30:57.573 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:30:57.573 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 80101 2024-08-17T17:30:57.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.573 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:57.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:57.573 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.386150+0000", 2024-08-17T17:30:57.573 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:57.573 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.386150+0000", 2024-08-17T17:30:57.573 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.574 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:57.574 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.386152+0000", 2024-08-17T17:30:57.574 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3200000000000001e-06 2024-08-17T17:30:57.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.574 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:57.574 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.386175+0000", 2024-08-17T17:30:57.574 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2830999999999999e-05 2024-08-17T17:30:57.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.574 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:57.574 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.386185+0000", 2024-08-17T17:30:57.574 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0236e-05 2024-08-17T17:30:57.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.575 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.575 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:57.575 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.386192+0000", 2024-08-17T17:30:57.575 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8809999999999998e-06 2024-08-17T17:30:57.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.575 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.575 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:57.575 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.392702+0000", 2024-08-17T17:30:57.575 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006510058 2024-08-17T17:30:57.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.575 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.575 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:30:57.576 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.392903+0000", 2024-08-17T17:30:57.576 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000201709 2024-08-17T17:30:57.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.576 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.576 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:30:57.576 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.393027+0000", 2024-08-17T17:30:57.576 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012381 2024-08-17T17:30:57.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.576 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.576 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.576 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.398525+0000", 2024-08-17T17:30:57.576 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0054977450000000001 2024-08-17T17:30:57.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.577 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:30:57.577 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.398808+0000", 2024-08-17T17:30:57.577 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00028265400000000002 2024-08-17T17:30:57.577 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.577 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.577 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.578131+0000", 2024-08-17T17:30:57.577 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.179323492 2024-08-17T17:30:57.577 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.578 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:30:57.578 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.578229+0000", 2024-08-17T17:30:57.578 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7421000000000001e-05 2024-08-17T17:30:57.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.578 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:30:57.578 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.578272+0000", 2024-08-17T17:30:57.578 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3891999999999999e-05 2024-08-17T17:30:57.578 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.579 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:57.579 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.579 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:80102 7.3 7:c2ad9bad:::benchmark_data_smithi003_201798_object80101:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-17T17:30:57.579 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:29:33.386331+0000", 2024-08-17T17:30:57.579 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 84.122342351, 2024-08-17T17:30:57.579 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18324026700000001, 2024-08-17T17:30:57.579 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:57.579 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:57.579 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:30:57.579 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:30:57.579 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:30:57.580 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:30:57.580 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 80102 2024-08-17T17:30:57.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.580 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:57.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.580 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:57.580 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.386331+0000", 2024-08-17T17:30:57.580 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.580 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:57.580 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.386331+0000", 2024-08-17T17:30:57.580 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.581 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.581 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:57.581 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.386332+0000", 2024-08-17T17:30:57.581 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4799999999999998e-07 2024-08-17T17:30:57.581 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.581 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.581 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:57.581 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.386356+0000", 2024-08-17T17:30:57.581 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4087000000000002e-05 2024-08-17T17:30:57.581 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.581 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.581 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:57.581 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.386371+0000", 2024-08-17T17:30:57.581 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4348999999999999e-05 2024-08-17T17:30:57.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.582 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:57.582 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.386377+0000", 2024-08-17T17:30:57.582 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6590000000000001e-06 2024-08-17T17:30:57.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.582 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:57.582 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389327+0000", 2024-08-17T17:30:57.582 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029500289999999998 2024-08-17T17:30:57.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.582 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:30:57.582 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389469+0000", 2024-08-17T17:30:57.583 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000141378 2024-08-17T17:30:57.583 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.583 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.583 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,6", 2024-08-17T17:30:57.583 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389621+0000", 2024-08-17T17:30:57.583 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015251500000000001 2024-08-17T17:30:57.583 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.583 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.583 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:30:57.583 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.392436+0000", 2024-08-17T17:30:57.583 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028147950000000001 2024-08-17T17:30:57.583 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.583 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.584 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.584 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.392864+0000", 2024-08-17T17:30:57.584 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00042817499999999999 2024-08-17T17:30:57.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.584 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.584 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.584 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.569490+0000", 2024-08-17T17:30:57.584 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17662562800000001 2024-08-17T17:30:57.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.584 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.584 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:30:57.584 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.569536+0000", 2024-08-17T17:30:57.584 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6548000000000002e-05 2024-08-17T17:30:57.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.585 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.585 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:30:57.585 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.569572+0000", 2024-08-17T17:30:57.585 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5456000000000001e-05 2024-08-17T17:30:57.585 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.585 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:57.585 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.585 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.586 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:80107 7.a 7:51d4ace1:::benchmark_data_smithi003_201798_object80106:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-17T17:30:57.586 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:29:33.389761+0000", 2024-08-17T17:30:57.586 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 84.118913206000002, 2024-08-17T17:30:57.586 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.19082833599999999, 2024-08-17T17:30:57.586 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:57.586 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:57.587 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:30:57.587 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:30:57.587 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:30:57.587 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:30:57.587 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 80107 2024-08-17T17:30:57.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.587 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:57.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.588 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:57.588 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389761+0000", 2024-08-17T17:30:57.588 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.588 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.588 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:57.588 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389761+0000", 2024-08-17T17:30:57.588 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.588 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.589 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:57.589 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389762+0000", 2024-08-17T17:30:57.589 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.091e-06 2024-08-17T17:30:57.589 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.589 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.589 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:57.589 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389797+0000", 2024-08-17T17:30:57.589 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5756999999999997e-05 2024-08-17T17:30:57.589 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.590 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:57.590 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389806+0000", 2024-08-17T17:30:57.590 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8519999999999993e-06 2024-08-17T17:30:57.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.591 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:57.591 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389812+0000", 2024-08-17T17:30:57.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4219999999999999e-06 2024-08-17T17:30:57.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.591 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:57.591 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.395577+0000", 2024-08-17T17:30:57.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0057653940000000001 2024-08-17T17:30:57.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.591 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:30:57.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.395713+0000", 2024-08-17T17:30:57.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000135766 2024-08-17T17:30:57.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:30:57.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.395864+0000", 2024-08-17T17:30:57.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015110699999999999 2024-08-17T17:30:57.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:30:57.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.399024+0000", 2024-08-17T17:30:57.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031597499999999998 2024-08-17T17:30:57.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.593 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.593 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.399362+0000", 2024-08-17T17:30:57.593 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000338687 2024-08-17T17:30:57.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.593 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.593 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.580492+0000", 2024-08-17T17:30:57.593 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18112983099999999 2024-08-17T17:30:57.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.593 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:30:57.593 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.580540+0000", 2024-08-17T17:30:57.593 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7556999999999999e-05 2024-08-17T17:30:57.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.594 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:30:57.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.580589+0000", 2024-08-17T17:30:57.594 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9122000000000002e-05 2024-08-17T17:30:57.594 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.594 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:57.594 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.595 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:80108 7.2 7:4c7c8085:::benchmark_data_smithi003_201798_object80107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-17T17:30:57.595 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:29:33.389832+0000", 2024-08-17T17:30:57.595 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 84.118841682999999, 2024-08-17T17:30:57.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.19049521, 2024-08-17T17:30:57.595 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:57.596 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:57.596 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:30:57.596 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:30:57.596 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:30:57.596 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:30:57.596 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 80108 2024-08-17T17:30:57.596 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.596 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:57.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.596 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:57.596 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389832+0000", 2024-08-17T17:30:57.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:57.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389832+0000", 2024-08-17T17:30:57.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:57.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389833+0000", 2024-08-17T17:30:57.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9699999999999996e-07 2024-08-17T17:30:57.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:57.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389856+0000", 2024-08-17T17:30:57.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3558999999999999e-05 2024-08-17T17:30:57.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:57.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389864+0000", 2024-08-17T17:30:57.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.542e-06 2024-08-17T17:30:57.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:57.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389870+0000", 2024-08-17T17:30:57.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9599999999999997e-06 2024-08-17T17:30:57.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:57.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.395563+0000", 2024-08-17T17:30:57.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0056929060000000002 2024-08-17T17:30:57.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.600 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:30:57.600 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.395710+0000", 2024-08-17T17:30:57.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014744700000000001 2024-08-17T17:30:57.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.600 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:30:57.600 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.395862+0000", 2024-08-17T17:30:57.601 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000152001 2024-08-17T17:30:57.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:30:57.601 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.398827+0000", 2024-08-17T17:30:57.601 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029646820000000002 2024-08-17T17:30:57.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.602 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.399994+0000", 2024-08-17T17:30:57.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00116719 2024-08-17T17:30:57.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.602 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.580221+0000", 2024-08-17T17:30:57.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18022680699999999 2024-08-17T17:30:57.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.603 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:30:57.603 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.580272+0000", 2024-08-17T17:30:57.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0918999999999998e-05 2024-08-17T17:30:57.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.603 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:30:57.603 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.580327+0000", 2024-08-17T17:30:57.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5699999999999999e-05 2024-08-17T17:30:57.604 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.604 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:57.604 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.604 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:80117 7.2 7:412925cb:::benchmark_data_smithi003_201798_object80116:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-17T17:30:57.604 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:29:33.402561+0000", 2024-08-17T17:30:57.604 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 84.106112789999997, 2024-08-17T17:30:57.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.177879548, 2024-08-17T17:30:57.604 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:57.604 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:57.604 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:30:57.604 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:30:57.604 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:30:57.605 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:30:57.605 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 80117 2024-08-17T17:30:57.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.605 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:57.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.605 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:57.605 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.402561+0000", 2024-08-17T17:30:57.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.605 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:57.605 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.402561+0000", 2024-08-17T17:30:57.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:57.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.402562+0000", 2024-08-17T17:30:57.606 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.026e-06 2024-08-17T17:30:57.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:57.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.402581+0000", 2024-08-17T17:30:57.606 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9431000000000001e-05 2024-08-17T17:30:57.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:57.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.402592+0000", 2024-08-17T17:30:57.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0492000000000001e-05 2024-08-17T17:30:57.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.607 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:57.607 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.402597+0000", 2024-08-17T17:30:57.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.293e-06 2024-08-17T17:30:57.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.607 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:57.607 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.402641+0000", 2024-08-17T17:30:57.608 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3943000000000003e-05 2024-08-17T17:30:57.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.608 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:30:57.608 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.402740+0000", 2024-08-17T17:30:57.608 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8305000000000002e-05 2024-08-17T17:30:57.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.608 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:30:57.608 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.402846+0000", 2024-08-17T17:30:57.608 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010678799999999999 2024-08-17T17:30:57.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.608 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:30:57.609 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.405210+0000", 2024-08-17T17:30:57.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0023640739999999999 2024-08-17T17:30:57.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.609 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.609 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.405832+0000", 2024-08-17T17:30:57.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00062142600000000001 2024-08-17T17:30:57.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.609 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.609 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.580363+0000", 2024-08-17T17:30:57.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17453089499999999 2024-08-17T17:30:57.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:30:57.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.580409+0000", 2024-08-17T17:30:57.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6294000000000001e-05 2024-08-17T17:30:57.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:30:57.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.580441+0000", 2024-08-17T17:30:57.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1581000000000001e-05 2024-08-17T17:30:57.611 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.611 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:57.611 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.612 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:85766 7.1 7:8a3fb768:::benchmark_data_smithi003_201798_object85765:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e235)", 2024-08-17T17:30:57.612 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:29:39.181565+0000", 2024-08-17T17:30:57.612 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 78.327109057000001, 2024-08-17T17:30:57.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.14816328400000001, 2024-08-17T17:30:57.612 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:57.612 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:57.612 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:30:57.612 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:30:57.612 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:30:57.612 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:30:57.612 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 85766 2024-08-17T17:30:57.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.612 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:57.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:57.613 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.181565+0000", 2024-08-17T17:30:57.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.613 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:57.613 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.181565+0000", 2024-08-17T17:30:57.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.613 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:57.613 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.181566+0000", 2024-08-17T17:30:57.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3850000000000001e-06 2024-08-17T17:30:57.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.614 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:57.614 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.181591+0000", 2024-08-17T17:30:57.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5038999999999999e-05 2024-08-17T17:30:57.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.614 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:57.614 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.181605+0000", 2024-08-17T17:30:57.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3733999999999999e-05 2024-08-17T17:30:57.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.614 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:57.614 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.181613+0000", 2024-08-17T17:30:57.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.551e-06 2024-08-17T17:30:57.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:57.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.181685+0000", 2024-08-17T17:30:57.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1976000000000003e-05 2024-08-17T17:30:57.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:30:57.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.181842+0000", 2024-08-17T17:30:57.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000156212 2024-08-17T17:30:57.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:30:57.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.181983+0000", 2024-08-17T17:30:57.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000141454 2024-08-17T17:30:57.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.616 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:30:57.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.185207+0000", 2024-08-17T17:30:57.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032240749999999999 2024-08-17T17:30:57.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.616 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.187016+0000", 2024-08-17T17:30:57.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001809029 2024-08-17T17:30:57.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.616 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.329510+0000", 2024-08-17T17:30:57.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.14249383700000001 2024-08-17T17:30:57.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:30:57.617 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.329680+0000", 2024-08-17T17:30:57.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016958699999999999 2024-08-17T17:30:57.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:30:57.617 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.329728+0000", 2024-08-17T17:30:57.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8405000000000002e-05 2024-08-17T17:30:57.617 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.618 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:57.618 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.618 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:85792 7.1 7:8addff64:::benchmark_data_smithi003_201798_object85791:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e235)", 2024-08-17T17:30:57.618 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:29:39.207795+0000", 2024-08-17T17:30:57.618 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 78.300878460999996, 2024-08-17T17:30:57.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.12533697999999999, 2024-08-17T17:30:57.618 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:57.618 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:57.618 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:30:57.619 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:30:57.619 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:30:57.619 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:30:57.619 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 85792 2024-08-17T17:30:57.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.619 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:57.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:57.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.207795+0000", 2024-08-17T17:30:57.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:57.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.207795+0000", 2024-08-17T17:30:57.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:57.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.207796+0000", 2024-08-17T17:30:57.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1699999999999997e-07 2024-08-17T17:30:57.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:57.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.207903+0000", 2024-08-17T17:30:57.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010669 2024-08-17T17:30:57.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:57.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.207930+0000", 2024-08-17T17:30:57.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6789000000000001e-05 2024-08-17T17:30:57.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:57.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.207941+0000", 2024-08-17T17:30:57.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1004e-05 2024-08-17T17:30:57.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.622 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:57.622 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.208055+0000", 2024-08-17T17:30:57.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011385 2024-08-17T17:30:57.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.622 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:30:57.622 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.208190+0000", 2024-08-17T17:30:57.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013561500000000001 2024-08-17T17:30:57.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.622 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:30:57.622 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.208326+0000", 2024-08-17T17:30:57.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013564 2024-08-17T17:30:57.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.623 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:30:57.623 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.211276+0000", 2024-08-17T17:30:57.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002950224 2024-08-17T17:30:57.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.623 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.623 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.212856+0000", 2024-08-17T17:30:57.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001579696 2024-08-17T17:30:57.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.623 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.333038+0000", 2024-08-17T17:30:57.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.120182447 2024-08-17T17:30:57.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:30:57.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.333092+0000", 2024-08-17T17:30:57.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4268000000000003e-05 2024-08-17T17:30:57.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:30:57.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.333132+0000", 2024-08-17T17:30:57.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9839999999999998e-05 2024-08-17T17:30:57.624 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.624 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:57.624 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.625 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:85807 7.1 7:88377cf5:::benchmark_data_smithi003_201798_object85806:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e235)", 2024-08-17T17:30:57.625 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:29:39.224148+0000", 2024-08-17T17:30:57.625 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 78.284526212000003, 2024-08-17T17:30:57.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.109268138, 2024-08-17T17:30:57.625 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:57.625 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:57.625 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:30:57.625 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:30:57.625 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:30:57.625 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:30:57.625 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 85807 2024-08-17T17:30:57.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.626 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:57.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:57.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.224148+0000", 2024-08-17T17:30:57.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:57.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.224148+0000", 2024-08-17T17:30:57.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:57.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.224149+0000", 2024-08-17T17:30:57.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.714e-06 2024-08-17T17:30:57.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:57.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.224168+0000", 2024-08-17T17:30:57.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8868e-05 2024-08-17T17:30:57.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:57.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.224181+0000", 2024-08-17T17:30:57.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3269e-05 2024-08-17T17:30:57.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:57.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.224189+0000", 2024-08-17T17:30:57.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0350000000000001e-06 2024-08-17T17:30:57.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:57.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.224252+0000", 2024-08-17T17:30:57.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2275999999999998e-05 2024-08-17T17:30:57.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:30:57.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.224417+0000", 2024-08-17T17:30:57.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016498700000000001 2024-08-17T17:30:57.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:30:57.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.224584+0000", 2024-08-17T17:30:57.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016719599999999999 2024-08-17T17:30:57.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:30:57.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.226700+0000", 2024-08-17T17:30:57.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002116503 2024-08-17T17:30:57.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.227003+0000", 2024-08-17T17:30:57.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00030261099999999999 2024-08-17T17:30:57.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.333275+0000", 2024-08-17T17:30:57.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.106272069 2024-08-17T17:30:57.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:30:57.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.333374+0000", 2024-08-17T17:30:57.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9141000000000005e-05 2024-08-17T17:30:57.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:30:57.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:39.333416+0000", 2024-08-17T17:30:57.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1468999999999999e-05 2024-08-17T17:30:57.631 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.631 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:57.631 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.631 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:105826 7.2 7:44e463c6:::benchmark_data_smithi003_201798_object105825:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-08-17T17:30:57.631 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:30:02.212971+0000", 2024-08-17T17:30:57.631 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 55.295703185000001, 2024-08-17T17:30:57.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.176643831, 2024-08-17T17:30:57.631 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:57.631 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:57.631 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:30:57.632 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:30:57.632 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:30:57.632 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:30:57.632 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 105826 2024-08-17T17:30:57.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.632 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:57.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:57.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.212971+0000", 2024-08-17T17:30:57.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:57.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.212971+0000", 2024-08-17T17:30:57.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:57.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.212972+0000", 2024-08-17T17:30:57.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.102e-06 2024-08-17T17:30:57.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:57.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.212994+0000", 2024-08-17T17:30:57.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2269999999999999e-05 2024-08-17T17:30:57.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:57.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.213006+0000", 2024-08-17T17:30:57.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1756999999999999e-05 2024-08-17T17:30:57.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:57.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.213013+0000", 2024-08-17T17:30:57.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0609999999999998e-06 2024-08-17T17:30:57.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:57.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.221324+0000", 2024-08-17T17:30:57.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0083109840000000004 2024-08-17T17:30:57.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:30:57.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.221479+0000", 2024-08-17T17:30:57.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015485299999999999 2024-08-17T17:30:57.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:30:57.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.221630+0000", 2024-08-17T17:30:57.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015138599999999999 2024-08-17T17:30:57.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.235066+0000", 2024-08-17T17:30:57.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013436116 2024-08-17T17:30:57.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:30:57.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.241302+0000", 2024-08-17T17:30:57.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0062359440000000002 2024-08-17T17:30:57.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.389463+0000", 2024-08-17T17:30:57.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.14816108 2024-08-17T17:30:57.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:30:57.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.389562+0000", 2024-08-17T17:30:57.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8717999999999999e-05 2024-08-17T17:30:57.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:30:57.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.389614+0000", 2024-08-17T17:30:57.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2559999999999998e-05 2024-08-17T17:30:57.640 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.640 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:57.641 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.641 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:105835 7.2 7:4c0b5c90:::benchmark_data_smithi003_201798_object105834:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-08-17T17:30:57.641 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:30:02.222027+0000", 2024-08-17T17:30:57.641 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 55.286646457000003, 2024-08-17T17:30:57.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17165864, 2024-08-17T17:30:57.641 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:57.641 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:57.641 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:30:57.641 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:30:57.641 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:30:57.641 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:30:57.641 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 105835 2024-08-17T17:30:57.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.642 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:57.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:57.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.222027+0000", 2024-08-17T17:30:57.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:57.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.222027+0000", 2024-08-17T17:30:57.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:57.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.222029+0000", 2024-08-17T17:30:57.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4079999999999999e-06 2024-08-17T17:30:57.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:57.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.222050+0000", 2024-08-17T17:30:57.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1472999999999999e-05 2024-08-17T17:30:57.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:57.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.222065+0000", 2024-08-17T17:30:57.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5159e-05 2024-08-17T17:30:57.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:57.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.222072+0000", 2024-08-17T17:30:57.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0380000000000002e-06 2024-08-17T17:30:57.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:57.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.226515+0000", 2024-08-17T17:30:57.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0044424670000000003 2024-08-17T17:30:57.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:30:57.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.227504+0000", 2024-08-17T17:30:57.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0009895119999999999 2024-08-17T17:30:57.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:30:57.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.228432+0000", 2024-08-17T17:30:57.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00092713799999999999 2024-08-17T17:30:57.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.240230+0000", 2024-08-17T17:30:57.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011798332999999999 2024-08-17T17:30:57.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:30:57.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.241311+0000", 2024-08-17T17:30:57.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00108084 2024-08-17T17:30:57.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.393583+0000", 2024-08-17T17:30:57.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.15227199899999999 2024-08-17T17:30:57.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:30:57.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.393644+0000", 2024-08-17T17:30:57.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1489999999999996e-05 2024-08-17T17:30:57.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:30:57.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.393686+0000", 2024-08-17T17:30:57.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1783000000000003e-05 2024-08-17T17:30:57.647 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.647 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:57.647 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.647 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:105842 7.2 7:479c7ffd:::benchmark_data_smithi003_201798_object105841:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-08-17T17:30:57.647 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:30:02.231607+0000", 2024-08-17T17:30:57.647 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 55.277066666000003, 2024-08-17T17:30:57.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.16226211900000001, 2024-08-17T17:30:57.647 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:57.647 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:57.648 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:30:57.648 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:30:57.648 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:30:57.648 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:30:57.648 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 105842 2024-08-17T17:30:57.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.648 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:57.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:57.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.231607+0000", 2024-08-17T17:30:57.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:57.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.231607+0000", 2024-08-17T17:30:57.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:57.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.231609+0000", 2024-08-17T17:30:57.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4279999999999999e-06 2024-08-17T17:30:57.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:57.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.231631+0000", 2024-08-17T17:30:57.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2175999999999999e-05 2024-08-17T17:30:57.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:57.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.231646+0000", 2024-08-17T17:30:57.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5207e-05 2024-08-17T17:30:57.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:57.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.231653+0000", 2024-08-17T17:30:57.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7429999999999996e-06 2024-08-17T17:30:57.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:57.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.235208+0000", 2024-08-17T17:30:57.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0035553149999999999 2024-08-17T17:30:57.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:30:57.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.235540+0000", 2024-08-17T17:30:57.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00033183399999999998 2024-08-17T17:30:57.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:30:57.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.236297+0000", 2024-08-17T17:30:57.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00075741299999999997 2024-08-17T17:30:57.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.241188+0000", 2024-08-17T17:30:57.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00489048 2024-08-17T17:30:57.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:30:57.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.241498+0000", 2024-08-17T17:30:57.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00031072899999999999 2024-08-17T17:30:57.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.393769+0000", 2024-08-17T17:30:57.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.15227086100000001 2024-08-17T17:30:57.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:30:57.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.393835+0000", 2024-08-17T17:30:57.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5189000000000002e-05 2024-08-17T17:30:57.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:30:57.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.393869+0000", 2024-08-17T17:30:57.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4743999999999998e-05 2024-08-17T17:30:57.653 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.653 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:57.654 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.654 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:105855 7.1 7:8c2a3dcd:::benchmark_data_smithi003_201798_object105854:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-08-17T17:30:57.654 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:30:02.243843+0000", 2024-08-17T17:30:57.654 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 55.264830605999997, 2024-08-17T17:30:57.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.14212388500000001, 2024-08-17T17:30:57.654 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:30:57.654 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:30:57.654 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:30:57.654 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:30:57.654 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:30:57.654 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:30:57.655 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 105855 2024-08-17T17:30:57.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.655 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:30:57.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:30:57.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.243843+0000", 2024-08-17T17:30:57.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:30:57.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.243843+0000", 2024-08-17T17:30:57.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:30:57.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:30:57.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.243844+0000", 2024-08-17T17:30:57.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.111e-06 2024-08-17T17:30:57.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:30:57.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.243896+0000", 2024-08-17T17:30:57.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1357999999999999e-05 2024-08-17T17:30:57.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:30:57.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.243907+0000", 2024-08-17T17:30:57.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1646e-05 2024-08-17T17:30:57.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:30:57.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.243914+0000", 2024-08-17T17:30:57.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9079999999999998e-06 2024-08-17T17:30:57.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:30:57.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.243963+0000", 2024-08-17T17:30:57.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8377000000000001e-05 2024-08-17T17:30:57.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:30:57.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.244046+0000", 2024-08-17T17:30:57.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3222999999999999e-05 2024-08-17T17:30:57.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:30:57.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.244122+0000", 2024-08-17T17:30:57.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5945000000000006e-05 2024-08-17T17:30:57.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:30:57.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.247051+0000", 2024-08-17T17:30:57.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029291149999999999 2024-08-17T17:30:57.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.247380+0000", 2024-08-17T17:30:57.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000329222 2024-08-17T17:30:57.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:30:57.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.385857+0000", 2024-08-17T17:30:57.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.138476816 2024-08-17T17:30:57.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:30:57.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.385917+0000", 2024-08-17T17:30:57.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0288999999999998e-05 2024-08-17T17:30:57.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:30:57.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:30:57.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:30:57.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.385967+0000", 2024-08-17T17:30:57.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9874999999999999e-05 2024-08-17T17:30:57.660 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.660 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:57.660 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.660 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:30:57.660 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:30:57.660 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:30:57.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:57.662 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:57.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:57.712 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:57 smithi003 ceph-mon[100615]: pgmap v706: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 771 op/s 2024-08-17T17:30:57.713 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:57 smithi003 ceph-mon[103344]: pgmap v706: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 771 op/s 2024-08-17T17:30:57.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:57 smithi053 ceph-mon[89963]: pgmap v706: 89 pgs: 89 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 771 op/s 2024-08-17T17:30:57.763 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:57.811 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:57.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:57.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:57.912 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:57.953 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:57.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:57.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:58.055 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:58.082 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:58.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:58.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:58.184 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:58.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:58.210 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:58.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:58.311 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:30:58.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:58.348 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:58.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:30:58.449 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:58.473 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:58.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:58.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:58.575 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:58.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:58.600 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:58.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:58.701 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:58.728 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:58.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:58.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:30:58.830 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:58.867 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-17T17:30:58.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:58.897 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:58.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:58.999 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:59.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:59.046 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:59.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:59.147 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:59.175 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:59.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:59.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:59.215 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:30:59.277 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:59.304 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:59.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:59.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:59.406 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:30:59.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:59.463 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:59.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:30:59.564 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:30:59.590 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:59.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:59.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:30:59.597 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:30:59 smithi053 ceph-mon[89963]: pgmap v707: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 775 op/s 2024-08-17T17:30:59.691 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:30:59.731 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:30:59.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:59.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:30:59.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:30:59 smithi003 ceph-mon[103344]: pgmap v707: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 775 op/s 2024-08-17T17:30:59.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:30:59 smithi003 ceph-mon[100615]: pgmap v707: 89 pgs: 89 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 775 op/s 2024-08-17T17:30:59.834 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:30:59.858 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:59.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:59.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:30:59.960 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:30:59.986 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:30:59.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:30:59.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:00.088 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:00.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:00.136 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:00.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:00.237 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:00.261 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:00.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:00.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:00.363 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:00.390 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:00.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:00.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:00.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:31:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:31:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:31:00.492 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:00.535 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:00.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:00.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:00.623 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:31:00.637 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:00.702 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:00.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:00.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:00.803 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:00.834 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:00.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:00.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:00.935 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:00.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:00.965 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:00.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:01.060 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:31:01.061 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2024-08-17T17:31:01.061 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 0 2024-08-17T17:31:01.061 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:31:01.066 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:01.093 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:01.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:01.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:01.195 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:01.223 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:01.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:01.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:01.324 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:01.349 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:01.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:01.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:01.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:01.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:01.508 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:01.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:01.609 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:01.633 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:01.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:01.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:01.657 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:01 smithi003 ceph-mon[103344]: pgmap v708: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 781 op/s 2024-08-17T17:31:01.657 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:01 smithi003 ceph-mon[100615]: pgmap v708: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 781 op/s 2024-08-17T17:31:01.692 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:01 smithi053 ceph-mon[89963]: pgmap v708: 89 pgs: 89 active+clean; 4.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 781 op/s 2024-08-17T17:31:01.721 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-17T17:31:01.734 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:01.792 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:01.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:01.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:01.893 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:01.921 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:01.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:01.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:02.023 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:02.061 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:31:02.073 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:02.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:02.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:02.175 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:02.218 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:02.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:02.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:02.319 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:02.348 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:02.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:02.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:02.450 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:02.497 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:02.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:02.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:02.499 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-08-17T17:31:02.510 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.6 to 1 (10000) 2024-08-17T17:31:02.599 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:02.640 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:02.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:02.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:02.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:02 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/226634454' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:31:02.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:02 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:31:02.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:02 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1320357654' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:31:02.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:02 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:31:02.742 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:02.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:02 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/226634454' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:31:02.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:02 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:31:02.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:02 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1320357654' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:31:02.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:02 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:31:02.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:02 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/226634454' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:31:02.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:02 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:31:02.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:02 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1320357654' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:31:02.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:02 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:31:02.783 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:02.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:02.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:02.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:02.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:02.943 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:02.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:03.044 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:03.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:03.097 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:03.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:03.198 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:03.226 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:03.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:03.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:03.328 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:03.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:03.369 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:03.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:03.470 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:03.515 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:03.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:03.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:03.617 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:03.647 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:03.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:03.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:03.749 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:03.772 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:03 smithi053 ceph-mon[89963]: pgmap v709: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 738 op/s 2024-08-17T17:31:03.773 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:03 smithi053 ceph-mon[89963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:31:03.773 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:03 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-17T17:31:03.773 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:03 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:31:03.773 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:03 smithi053 ceph-mon[89963]: osdmap e245: 8 total, 8 up, 7 in 2024-08-17T17:31:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:03 smithi003 ceph-mon[100615]: pgmap v709: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 738 op/s 2024-08-17T17:31:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:03 smithi003 ceph-mon[100615]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:31:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:03 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-17T17:31:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:03 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:31:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:03 smithi003 ceph-mon[100615]: osdmap e245: 8 total, 8 up, 7 in 2024-08-17T17:31:03.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:03 smithi003 ceph-mon[103344]: pgmap v709: 89 pgs: 89 active+clean; 4.6 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 738 op/s 2024-08-17T17:31:03.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:03 smithi003 ceph-mon[103344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:31:03.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:03 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-17T17:31:03.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:03 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:31:03.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:03 smithi003 ceph-mon[103344]: osdmap e245: 8 total, 8 up, 7 in 2024-08-17T17:31:03.785 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:03.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:03.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:03.803 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:31:03.803 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2024-08-17T17:31:03.804 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2024-08-17T17:31:03.804 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2024-08-17T17:31:03.804 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:31:03.887 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:03.932 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:03.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:03.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:04.034 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:04.062 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:04.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:04.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:04.164 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:04.188 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:04.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:04.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:04.214 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 7 1 2024-08-17T17:31:04.290 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:04.381 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:04.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:04.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:04.477 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:31:04.477 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:31:04.479 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-17T17:31:04.483 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:04.522 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:31:04.522 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:31:04.546 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:04.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:04.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:04.566 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:31:04.566 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:31:04.604 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:31:04.604 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:31:04.639 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:31:04.640 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:31:04.648 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:04.679 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:31:04.679 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:31:04.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:04.696 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:04.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:04.706 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:31:04.707 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:31:04.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:04 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3393813912' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:31:04.735 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:04 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3393813912' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:31:04.736 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:04 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3393813912' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:31:04.738 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:31:04.739 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:31:04.797 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:04.810 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:31:04.825 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:04.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:04.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:04.927 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:04.967 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:04.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:04.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:05.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:05.116 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:05.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:05.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:05.218 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:05.244 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:05.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:05.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:05.346 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:05.371 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:05.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:05.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:05.473 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:05.526 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:31:05.531 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:05.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:05.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:05.633 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:05.681 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:05 smithi003 ceph-mon[103344]: pgmap v711: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 724 op/s 2024-08-17T17:31:05.681 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:05 smithi003 ceph-mon[100615]: pgmap v711: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 724 op/s 2024-08-17T17:31:05.681 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:31:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:31:05] "GET /metrics HTTP/1.1" 200 36505 "" "Prometheus/2.51.0" 2024-08-17T17:31:05.685 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:05.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:05.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:05.787 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:05.826 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:05.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:05.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:05.928 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:05.954 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:05 smithi053 ceph-mon[89963]: pgmap v711: 89 pgs: 89 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 724 op/s 2024-08-17T17:31:05.968 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:05.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:05.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:06.070 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:06.118 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:06.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:06.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:06.220 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:06.248 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:06.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:06.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:06.350 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:06.389 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:06.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:06.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:06.491 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:06.533 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:06.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:06.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:06.583 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:31:06.583 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:31:06.635 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:06.661 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:06.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:06.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:06.762 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:06.786 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:31:06.799 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:06.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:06.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:06.901 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:06.924 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:06.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:06.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:07.026 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:07.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:07.078 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:07.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:07.179 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:07.207 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:07.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:07.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:07.251 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-17T17:31:07.309 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:07.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:07.365 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:07.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:07.459 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:31:07.466 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:07.490 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:07.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:07.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:07.578 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:31:07.592 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:07.629 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:07.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:07.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:07.731 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:07.775 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:07 smithi003 ceph-mon[100615]: pgmap v712: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 793 op/s 2024-08-17T17:31:07.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:07 smithi003 ceph-mon[103344]: pgmap v712: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 793 op/s 2024-08-17T17:31:07.783 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:07.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:07.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:07.839 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:07 smithi053 ceph-mon[89963]: pgmap v712: 89 pgs: 89 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 793 op/s 2024-08-17T17:31:07.886 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:07.940 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:07.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:07.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:08.043 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:08.055 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:31:08.094 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:31:08.123 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:08.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:08.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:08.227 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:08.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:08.286 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:08.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:08.387 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:08.412 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:08.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:08.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:08.514 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:08.541 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:08.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:08.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:08.563 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.7 to 1 (10000) 2024-08-17T17:31:08.644 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:08.682 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:08.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:08.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:08.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:31:08.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:08 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/256907335' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:31:08.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:08 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:31:08.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:31:08.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:08 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/256907335' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:31:08.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:08 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:31:08.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:31:08.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:08 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/256907335' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:31:08.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:08 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:31:08.784 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:08.817 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:08.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:08.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:08.918 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:08.947 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:08.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:08.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:09.048 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:09.090 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:09.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:09.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:09.192 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:09.232 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:09.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:09.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:09.334 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:31:09.335 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:31:09.335 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:09.376 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:09.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:09.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:09.478 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:09.517 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:09.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:09.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:09.619 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:09.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:09.655 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:09.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:09.657 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:09 smithi053 ceph-mon[89963]: pgmap v713: 89 pgs: 89 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 786 op/s 2024-08-17T17:31:09.658 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:09 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-17T17:31:09.658 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:09 smithi053 ceph-mon[89963]: osdmap e246: 8 total, 8 up, 7 in 2024-08-17T17:31:09.756 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:09.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:09 smithi003 ceph-mon[100615]: pgmap v713: 89 pgs: 89 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 786 op/s 2024-08-17T17:31:09.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:09 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-17T17:31:09.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:09 smithi003 ceph-mon[100615]: osdmap e246: 8 total, 8 up, 7 in 2024-08-17T17:31:09.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:09 smithi003 ceph-mon[103344]: pgmap v713: 89 pgs: 89 active+clean; 4.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 786 op/s 2024-08-17T17:31:09.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:09 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-17T17:31:09.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:09 smithi003 ceph-mon[103344]: osdmap e246: 8 total, 8 up, 7 in 2024-08-17T17:31:09.800 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:09.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:09.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:09.902 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:09.929 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:09.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:09.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:10.025 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-17T17:31:10.031 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:10.097 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:10.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:10.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:10.199 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:10.242 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:10.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:10.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:10.344 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:10.372 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:31:10.386 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:10.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:10.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:10.466 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:31:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=cleanup t=2024-08-17T17:31:10.095833564Z level=info msg="Completed cleanup jobs" duration=13.388764ms 2024-08-17T17:31:10.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:31:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:31:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:31:10.488 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:10.537 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:10.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:10.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:10.639 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:10.685 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:10.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:10.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:10.755 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-08-17T17:31:10.755 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:31:10.787 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:10.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:10.814 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:10.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:10.915 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:10.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:10.982 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:10.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:11.083 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:11.108 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:11.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:11.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:11.210 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:11.235 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:11.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:11.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:11.337 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:11.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:11.387 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:11.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:11.487 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:11.537 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:11.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:11.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:11.639 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:11.682 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:11.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:11.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:11.784 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:11.818 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:11 smithi053 ceph-mon[89963]: pgmap v715: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 944 op/s 2024-08-17T17:31:11.823 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:11.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:11.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:11.925 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:11.945 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:11 smithi003 ceph-mon[100615]: pgmap v715: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 944 op/s 2024-08-17T17:31:11.945 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:11 smithi003 ceph-mon[103344]: pgmap v715: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 944 op/s 2024-08-17T17:31:11.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:11.983 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:11.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:12.084 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:12.095 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:31:12.095 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:31:12.119 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:12.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:12.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:12.220 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:12.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:12.275 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:12.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:12.376 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:12.403 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:12.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:12.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:12.505 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:12.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:12.545 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:12.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:12.646 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:12.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:12.685 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:12.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:12.742 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-17T17:31:12.778 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:31:12.786 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:12.813 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:12.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:12.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:12.832 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:31:12.832 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:31:12.868 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:31:12.868 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:31:12.906 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:31:12.906 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:31:12.914 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:12.955 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:31:12.955 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:31:12.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:12.956 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:12.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:12.988 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:31:12.988 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:31:13.019 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:31:13.019 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:31:13.053 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:31:13.054 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:31:13.057 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:13.088 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:13.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:13.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:13.091 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:31:13.091 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:31:13.093 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:31:13.190 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:13.232 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:13.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:13.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:13.334 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:13.367 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:13.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:13.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:13.469 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:13.515 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:13.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:13.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:13.597 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-08-17T17:31:13.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:13.656 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:13.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:13.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:13.758 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:13.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:13 smithi003 ceph-mon[100615]: pgmap v716: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 713 op/s 2024-08-17T17:31:13.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:13 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1105015493' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:31:13.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:13 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:31:13.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:13 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3261754468' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:31:13.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:13 smithi003 ceph-mon[103344]: pgmap v716: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 713 op/s 2024-08-17T17:31:13.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:13 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1105015493' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:31:13.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:13 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:31:13.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:13 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3261754468' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:31:13.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:13.798 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:13.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:13.899 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:13.946 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:13.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:13.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:13.960 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:13 smithi053 ceph-mon[89963]: pgmap v716: 89 pgs: 89 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 713 op/s 2024-08-17T17:31:13.961 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:13 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1105015493' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:31:13.961 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:13 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:31:13.961 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:13 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3261754468' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:31:14.048 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:14.076 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:14.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:14.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:14.178 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:14.200 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:14.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:14.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:14.302 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:14.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:14.340 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:14.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:14.441 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:14.468 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:14.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:14.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:14.570 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:14.597 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:14.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:14.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:14.680 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:14 smithi053 ceph-mon[89963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:31:14.681 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:14 smithi053 ceph-mon[89963]: Cluster is now healthy 2024-08-17T17:31:14.681 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:14 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:31:14.681 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:14 smithi053 ceph-mon[89963]: osdmap e247: 8 total, 8 up, 7 in 2024-08-17T17:31:14.698 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:14.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:14.725 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:14.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:14.826 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:14.863 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:14.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:14.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:14.946 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:31:14.946 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2024-08-17T17:31:14.946 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2024-08-17T17:31:14.946 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2024-08-17T17:31:14.946 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.946 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82573 6.d 6:bd9981c1:::benchmark_data_smithi003_175043_object82572:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:31:14.946 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.116209+0000", 2024-08-17T17:31:14.946 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 335.82732790699998, 2024-08-17T17:31:14.946 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6778493189999999, 2024-08-17T17:31:14.946 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:14.947 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2024-08-17T17:31:14.947 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:14.947 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:31:14.947 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:31:14.947 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82573 2024-08-17T17:31:14.947 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.947 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:14.947 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.947 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:14.947 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.116209+0000", 2024-08-17T17:31:14.947 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:14.947 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.948 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.948 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:14.948 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.116209+0000", 2024-08-17T17:31:14.948 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:14.948 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.948 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.948 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:14.948 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.116207+0000", 2024-08-17T17:31:14.948 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:31:14.948 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.948 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.948 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:14.948 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.116249+0000", 2024-08-17T17:31:14.948 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1340000000000001e-05 2024-08-17T17:31:14.949 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.949 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.949 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:14.949 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.116266+0000", 2024-08-17T17:31:14.949 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7655e-05 2024-08-17T17:31:14.949 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.949 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.949 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:14.949 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.116273+0000", 2024-08-17T17:31:14.949 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0840000000000003e-06 2024-08-17T17:31:14.949 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.949 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.949 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:14.949 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.116355+0000", 2024-08-17T17:31:14.949 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1224999999999996e-05 2024-08-17T17:31:14.950 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.950 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.950 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:31:14.950 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.116366+0000", 2024-08-17T17:31:14.950 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1092e-05 2024-08-17T17:31:14.950 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.950 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.950 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:14.950 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.775203+0000", 2024-08-17T17:31:14.950 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.65883687 2024-08-17T17:31:14.950 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.950 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.951 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for degraded object", 2024-08-17T17:31:14.951 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.775490+0000", 2024-08-17T17:31:14.951 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00028773399999999999 2024-08-17T17:31:14.951 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.951 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.951 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:14.951 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794011+0000", 2024-08-17T17:31:14.951 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.018521115000000001 2024-08-17T17:31:14.951 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.951 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.951 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:14.951 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794059+0000", 2024-08-17T17:31:14.951 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7213999999999999e-05 2024-08-17T17:31:14.951 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:14.951 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:14.952 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:14.952 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.952 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.952 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82600 6.d 6:b9e24bf2:::benchmark_data_smithi003_175043_object82599:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:31:14.952 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.137681+0000", 2024-08-17T17:31:14.952 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 335.80585614, 2024-08-17T17:31:14.952 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.656490799, 2024-08-17T17:31:14.952 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:14.952 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:14.952 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:14.952 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:31:14.952 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:31:14.952 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82600 2024-08-17T17:31:14.952 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.953 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:14.953 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.953 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:14.953 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.137681+0000", 2024-08-17T17:31:14.953 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:14.953 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.953 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.953 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:14.953 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.137681+0000", 2024-08-17T17:31:14.953 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:14.953 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.953 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.953 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:14.953 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.137680+0000", 2024-08-17T17:31:14.954 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:31:14.954 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.954 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.954 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:14.954 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.137739+0000", 2024-08-17T17:31:14.954 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9840999999999998e-05 2024-08-17T17:31:14.954 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.954 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.954 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:14.954 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.137752+0000", 2024-08-17T17:31:14.954 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2431e-05 2024-08-17T17:31:14.954 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.955 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.955 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:14.955 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.137757+0000", 2024-08-17T17:31:14.955 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.023e-06 2024-08-17T17:31:14.955 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.955 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.955 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:14.955 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.137822+0000", 2024-08-17T17:31:14.955 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4703999999999996e-05 2024-08-17T17:31:14.955 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.955 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.955 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:31:14.955 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.137832+0000", 2024-08-17T17:31:14.955 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0244000000000001e-05 2024-08-17T17:31:14.956 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.956 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.956 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:14.956 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.775512+0000", 2024-08-17T17:31:14.956 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.637680322 2024-08-17T17:31:14.956 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.956 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.956 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:14.956 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.775598+0000", 2024-08-17T17:31:14.956 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.6287999999999998e-05 2024-08-17T17:31:14.956 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.956 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.956 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:31:14.956 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.775699+0000", 2024-08-17T17:31:14.956 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000100445 2024-08-17T17:31:14.956 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.957 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.957 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:14.957 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.785737+0000", 2024-08-17T17:31:14.957 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010038126 2024-08-17T17:31:14.957 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.957 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.957 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:14.957 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.793769+0000", 2024-08-17T17:31:14.957 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0080316069999999996 2024-08-17T17:31:14.957 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.957 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.957 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:14.957 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794078+0000", 2024-08-17T17:31:14.957 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00030971299999999998 2024-08-17T17:31:14.957 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.958 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.958 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:14.958 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794109+0000", 2024-08-17T17:31:14.958 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0568000000000003e-05 2024-08-17T17:31:14.958 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.958 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.958 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:14.958 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794172+0000", 2024-08-17T17:31:14.958 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2927000000000002e-05 2024-08-17T17:31:14.958 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:14.958 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:14.958 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:14.958 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.958 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.959 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82640 6.d 6:b46cd03d:::benchmark_data_smithi003_175043_object82639:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:31:14.959 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.177833+0000", 2024-08-17T17:31:14.959 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 335.76570403199997, 2024-08-17T17:31:14.959 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6164523790000001, 2024-08-17T17:31:14.959 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:14.959 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:14.959 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:14.959 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:31:14.959 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:31:14.960 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82640 2024-08-17T17:31:14.960 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.960 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:14.960 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.960 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:14.960 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.177833+0000", 2024-08-17T17:31:14.960 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:14.961 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.961 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.961 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:14.961 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.177833+0000", 2024-08-17T17:31:14.961 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:14.961 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.961 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.961 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:14.961 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.177831+0000", 2024-08-17T17:31:14.961 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:31:14.961 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.961 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.961 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:14.961 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.177866+0000", 2024-08-17T17:31:14.961 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5014000000000001e-05 2024-08-17T17:31:14.961 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.962 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.962 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:14.962 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.177883+0000", 2024-08-17T17:31:14.962 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6385999999999999e-05 2024-08-17T17:31:14.962 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.962 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.962 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:14.962 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.177891+0000", 2024-08-17T17:31:14.962 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.2199999999999992e-06 2024-08-17T17:31:14.962 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.962 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.962 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:14.962 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.177968+0000", 2024-08-17T17:31:14.962 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.6953999999999995e-05 2024-08-17T17:31:14.963 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.963 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.963 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:31:14.963 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.177977+0000", 2024-08-17T17:31:14.963 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.4830000000000005e-06 2024-08-17T17:31:14.963 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.963 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.963 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:14.963 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.777368+0000", 2024-08-17T17:31:14.963 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.599390358 2024-08-17T17:31:14.963 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.963 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.963 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:14.963 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.777492+0000", 2024-08-17T17:31:14.964 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012466300000000001 2024-08-17T17:31:14.964 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.964 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.964 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:31:14.964 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.777652+0000", 2024-08-17T17:31:14.964 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015963500000000001 2024-08-17T17:31:14.964 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.964 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.964 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:14.964 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.785741+0000", 2024-08-17T17:31:14.964 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0080892350000000002 2024-08-17T17:31:14.964 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.964 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.965 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:14.965 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.793812+0000", 2024-08-17T17:31:14.965 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0080707850000000005 2024-08-17T17:31:14.965 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.965 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.965 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:14.965 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794202+0000", 2024-08-17T17:31:14.965 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00038980599999999997 2024-08-17T17:31:14.965 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.965 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.965 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:14.965 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794229+0000", 2024-08-17T17:31:14.965 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7572999999999998e-05 2024-08-17T17:31:14.965 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.966 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.966 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:14.966 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794286+0000", 2024-08-17T17:31:14.966 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6013000000000001e-05 2024-08-17T17:31:14.966 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:14.966 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:14.966 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:14.966 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.966 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.966 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82642 6.d 6:b3782226:::benchmark_data_smithi003_175043_object82641:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:31:14.966 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.178629+0000", 2024-08-17T17:31:14.966 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 335.76490781199999, 2024-08-17T17:31:14.966 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.615915269, 2024-08-17T17:31:14.966 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:14.966 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:14.967 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:14.967 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:31:14.967 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:31:14.967 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82642 2024-08-17T17:31:14.967 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.967 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:14.967 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.967 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:14.967 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.178629+0000", 2024-08-17T17:31:14.967 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:14.967 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.967 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.967 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:14.968 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.178629+0000", 2024-08-17T17:31:14.968 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:14.968 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.968 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.968 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:14.968 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.178628+0000", 2024-08-17T17:31:14.968 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:31:14.968 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.968 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.968 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:14.968 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.178663+0000", 2024-08-17T17:31:14.968 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4839999999999998e-05 2024-08-17T17:31:14.968 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.968 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.969 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:14.969 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.178679+0000", 2024-08-17T17:31:14.969 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6617e-05 2024-08-17T17:31:14.969 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.969 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.969 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:14.969 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.178686+0000", 2024-08-17T17:31:14.969 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.066e-06 2024-08-17T17:31:14.969 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.969 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.969 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:14.969 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.178737+0000", 2024-08-17T17:31:14.969 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1802999999999999e-05 2024-08-17T17:31:14.969 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.970 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.970 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:31:14.970 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.178746+0000", 2024-08-17T17:31:14.970 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.6240000000000008e-06 2024-08-17T17:31:14.970 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.970 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.970 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:14.970 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.778800+0000", 2024-08-17T17:31:14.970 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6000541150000001 2024-08-17T17:31:14.970 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.970 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.970 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:14.970 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.778870+0000", 2024-08-17T17:31:14.970 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9701000000000002e-05 2024-08-17T17:31:14.971 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.971 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.971 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:31:14.971 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.778965+0000", 2024-08-17T17:31:14.971 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.4919000000000004e-05 2024-08-17T17:31:14.971 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.971 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.971 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:14.971 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.785745+0000", 2024-08-17T17:31:14.971 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0067802690000000002 2024-08-17T17:31:14.971 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.971 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.971 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:14.971 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794312+0000", 2024-08-17T17:31:14.971 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0085670390000000003 2024-08-17T17:31:14.972 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.972 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.972 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:14.972 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794453+0000", 2024-08-17T17:31:14.972 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000140548 2024-08-17T17:31:14.972 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.972 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.972 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:14.972 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794479+0000", 2024-08-17T17:31:14.972 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6236000000000001e-05 2024-08-17T17:31:14.973 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.973 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.973 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:14.973 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794545+0000", 2024-08-17T17:31:14.973 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5801000000000002e-05 2024-08-17T17:31:14.973 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:14.973 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:14.973 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:14.973 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.973 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.973 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82693 6.d 6:b633a1e2:::benchmark_data_smithi003_175043_object82692:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:31:14.973 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.226745+0000", 2024-08-17T17:31:14.974 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 335.716792199, 2024-08-17T17:31:14.974 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5679927600000001, 2024-08-17T17:31:14.974 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:14.974 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:14.974 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:14.974 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:31:14.974 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:31:14.974 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82693 2024-08-17T17:31:14.974 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.974 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:14.974 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.974 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:14.974 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.226745+0000", 2024-08-17T17:31:14.975 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:14.975 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.975 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.975 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:14.975 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.226745+0000", 2024-08-17T17:31:14.975 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:14.975 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.975 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.975 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:14.975 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.226743+0000", 2024-08-17T17:31:14.975 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:31:14.975 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.975 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.975 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:14.975 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.226800+0000", 2024-08-17T17:31:14.976 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6427e-05 2024-08-17T17:31:14.976 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.976 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.976 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:14.976 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.226813+0000", 2024-08-17T17:31:14.976 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2911999999999999e-05 2024-08-17T17:31:14.976 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.976 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.976 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:14.977 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.226819+0000", 2024-08-17T17:31:14.977 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2219999999999999e-06 2024-08-17T17:31:14.977 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.977 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.977 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:14.977 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.226876+0000", 2024-08-17T17:31:14.977 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7105999999999998e-05 2024-08-17T17:31:14.977 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.977 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.977 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:31:14.977 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.226886+0000", 2024-08-17T17:31:14.977 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0468e-05 2024-08-17T17:31:14.977 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.977 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.978 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:14.978 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.779914+0000", 2024-08-17T17:31:14.978 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.55302731 2024-08-17T17:31:14.978 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.978 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.978 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:14.978 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.780041+0000", 2024-08-17T17:31:14.978 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000127298 2024-08-17T17:31:14.978 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.978 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.978 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:31:14.978 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.780202+0000", 2024-08-17T17:31:14.978 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016124100000000001 2024-08-17T17:31:14.979 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.979 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.979 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:14.979 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.791392+0000", 2024-08-17T17:31:14.979 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011189486 2024-08-17T17:31:14.979 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.979 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.979 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:14.979 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794335+0000", 2024-08-17T17:31:14.979 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002943219 2024-08-17T17:31:14.979 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.979 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.979 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:14.979 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794589+0000", 2024-08-17T17:31:14.980 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00025462199999999998 2024-08-17T17:31:14.980 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.980 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.980 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:14.980 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794641+0000", 2024-08-17T17:31:14.980 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1773e-05 2024-08-17T17:31:14.980 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.980 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.980 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:14.980 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794738+0000", 2024-08-17T17:31:14.980 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6477000000000005e-05 2024-08-17T17:31:14.980 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:14.980 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:14.980 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:14.980 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.981 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.981 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82697 6.d 6:b974a054:::benchmark_data_smithi003_175043_object82696:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:31:14.981 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.231888+0000", 2024-08-17T17:31:14.981 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 335.71164929399998, 2024-08-17T17:31:14.981 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.563409968, 2024-08-17T17:31:14.981 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:14.981 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:14.981 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:14.981 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:31:14.981 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:31:14.981 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82697 2024-08-17T17:31:14.981 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.981 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:14.981 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.982 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:14.982 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.231888+0000", 2024-08-17T17:31:14.982 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:14.982 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.982 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.982 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:14.982 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.231888+0000", 2024-08-17T17:31:14.982 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:14.982 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.982 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.982 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:14.982 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.231887+0000", 2024-08-17T17:31:14.982 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:31:14.982 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.983 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.983 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:14.983 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.231937+0000", 2024-08-17T17:31:14.983 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0117000000000001e-05 2024-08-17T17:31:14.983 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.983 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.983 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:14.983 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.231949+0000", 2024-08-17T17:31:14.983 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2431e-05 2024-08-17T17:31:14.983 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.983 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.983 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:14.983 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.231954+0000", 2024-08-17T17:31:14.983 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7029999999999997e-06 2024-08-17T17:31:14.983 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.984 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.984 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:14.984 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.232014+0000", 2024-08-17T17:31:14.984 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0628999999999998e-05 2024-08-17T17:31:14.984 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.984 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.984 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:31:14.984 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.232025+0000", 2024-08-17T17:31:14.984 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.062e-05 2024-08-17T17:31:14.984 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.984 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.984 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:14.984 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.781376+0000", 2024-08-17T17:31:14.984 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.549351269 2024-08-17T17:31:14.984 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.984 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.985 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:14.985 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.781451+0000", 2024-08-17T17:31:14.985 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.4862000000000003e-05 2024-08-17T17:31:14.985 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.985 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.985 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:31:14.985 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.781534+0000", 2024-08-17T17:31:14.985 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.2558999999999993e-05 2024-08-17T17:31:14.985 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.985 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.985 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:14.985 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.791395+0000", 2024-08-17T17:31:14.985 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0098611889999999994 2024-08-17T17:31:14.985 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.985 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.986 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:14.986 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794362+0000", 2024-08-17T17:31:14.986 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0029670030000000002 2024-08-17T17:31:14.986 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.986 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.986 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:14.986 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.795163+0000", 2024-08-17T17:31:14.986 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00080066100000000004 2024-08-17T17:31:14.986 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.986 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.986 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:14.986 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.795204+0000", 2024-08-17T17:31:14.986 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1603999999999997e-05 2024-08-17T17:31:14.986 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.987 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.987 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:14.987 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.795298+0000", 2024-08-17T17:31:14.987 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.3583000000000002e-05 2024-08-17T17:31:14.987 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:14.987 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:14.987 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:14.987 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.987 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.987 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82701 6.d 6:bdcfc51a:::benchmark_data_smithi003_175043_object82700:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:31:14.987 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.234530+0000", 2024-08-17T17:31:14.987 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 335.709006665, 2024-08-17T17:31:14.987 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5609469009999999, 2024-08-17T17:31:14.987 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:14.988 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:14.988 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:14.988 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:31:14.988 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:31:14.988 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82701 2024-08-17T17:31:14.988 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.988 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:14.988 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.988 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:14.988 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.234530+0000", 2024-08-17T17:31:14.988 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:14.988 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.988 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.988 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:14.989 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.234530+0000", 2024-08-17T17:31:14.989 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:14.989 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.989 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.989 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:14.989 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.234529+0000", 2024-08-17T17:31:14.989 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:31:14.989 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.989 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.989 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:14.989 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.234620+0000", 2024-08-17T17:31:14.989 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1897999999999993e-05 2024-08-17T17:31:14.989 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.989 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.989 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:14.989 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.234636+0000", 2024-08-17T17:31:14.990 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5603999999999998e-05 2024-08-17T17:31:14.990 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.990 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.990 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:14.990 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.234642+0000", 2024-08-17T17:31:14.990 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7060000000000001e-06 2024-08-17T17:31:14.990 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.990 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.990 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:14.990 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.234711+0000", 2024-08-17T17:31:14.990 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9542999999999995e-05 2024-08-17T17:31:14.990 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.990 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.990 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:31:14.991 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.234722+0000", 2024-08-17T17:31:14.991 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0593e-05 2024-08-17T17:31:14.991 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.991 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.991 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:14.991 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.782419+0000", 2024-08-17T17:31:14.991 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.547697522 2024-08-17T17:31:14.991 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.991 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.992 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:14.992 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.782488+0000", 2024-08-17T17:31:14.992 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.8922000000000002e-05 2024-08-17T17:31:14.992 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.992 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.992 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:31:14.992 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.782573+0000", 2024-08-17T17:31:14.992 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.4179999999999997e-05 2024-08-17T17:31:14.992 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.992 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.992 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:14.992 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.791405+0000", 2024-08-17T17:31:14.993 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0088323220000000001 2024-08-17T17:31:14.993 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.993 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.993 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:14.993 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794389+0000", 2024-08-17T17:31:14.993 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0029838109999999998 2024-08-17T17:31:14.993 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.993 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.993 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:14.993 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.795355+0000", 2024-08-17T17:31:14.993 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00096604799999999995 2024-08-17T17:31:14.993 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.993 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.993 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:14.993 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.795398+0000", 2024-08-17T17:31:14.994 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3146999999999999e-05 2024-08-17T17:31:14.994 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.994 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.994 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:14.994 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.795477+0000", 2024-08-17T17:31:14.994 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.9569999999999999e-05 2024-08-17T17:31:14.994 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:14.994 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:14.994 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:14.994 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.994 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.994 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82722 6.d 6:b3e932d8:::benchmark_data_smithi003_175043_object82721:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:31:14.994 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.264293+0000", 2024-08-17T17:31:14.995 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 335.67924429599998, 2024-08-17T17:31:14.995 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.533356661, 2024-08-17T17:31:14.995 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:14.995 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:14.995 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:14.995 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:31:14.995 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:31:14.995 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82722 2024-08-17T17:31:14.995 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.995 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:14.995 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.995 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:14.995 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.264293+0000", 2024-08-17T17:31:14.995 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:14.995 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.996 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.996 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:14.996 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.264293+0000", 2024-08-17T17:31:14.996 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:14.996 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.996 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.996 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:14.996 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.264291+0000", 2024-08-17T17:31:14.996 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:31:14.996 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.996 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.996 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:14.996 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.264329+0000", 2024-08-17T17:31:14.996 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.7656e-05 2024-08-17T17:31:14.996 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.997 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.997 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:14.997 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.264344+0000", 2024-08-17T17:31:14.997 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5147000000000001e-05 2024-08-17T17:31:14.997 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.997 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.997 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:14.997 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.264349+0000", 2024-08-17T17:31:14.997 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.746e-06 2024-08-17T17:31:14.997 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.997 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.997 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:14.997 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.264416+0000", 2024-08-17T17:31:14.997 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.7662000000000004e-05 2024-08-17T17:31:14.997 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.998 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.998 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:31:14.998 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.264423+0000", 2024-08-17T17:31:14.998 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3450000000000002e-06 2024-08-17T17:31:14.998 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.998 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.998 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:14.998 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.783524+0000", 2024-08-17T17:31:14.998 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5191010029999998 2024-08-17T17:31:14.998 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.998 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.998 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:14.998 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.783598+0000", 2024-08-17T17:31:14.998 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3861000000000003e-05 2024-08-17T17:31:14.998 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.998 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.999 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:31:14.999 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.783677+0000", 2024-08-17T17:31:14.999 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.9592000000000006e-05 2024-08-17T17:31:14.999 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.999 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.999 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:14.999 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.791409+0000", 2024-08-17T17:31:14.999 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.007731435 2024-08-17T17:31:14.999 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.999 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:14.999 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:14.999 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794410+0000", 2024-08-17T17:31:14.999 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003000955 2024-08-17T17:31:14.999 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:14.999 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.000 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:15.000 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.797490+0000", 2024-08-17T17:31:15.000 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0030798610000000001 2024-08-17T17:31:15.000 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.000 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.000 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:15.000 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.797545+0000", 2024-08-17T17:31:15.000 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5507999999999998e-05 2024-08-17T17:31:15.000 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.000 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.000 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:15.000 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.797650+0000", 2024-08-17T17:31:15.000 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000104498 2024-08-17T17:31:15.000 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:15.000 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:15.000 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:15.001 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.001 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.001 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:15.002 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82729 6.d 6:b1d8abf5:::benchmark_data_smithi003_175043_object82728:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:31:15.002 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.271858+0000", 2024-08-17T17:31:15.002 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 335.67167869799999, 2024-08-17T17:31:15.002 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.525969804, 2024-08-17T17:31:15.003 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:15.003 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:15.003 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:15.003 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:31:15.003 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:31:15.003 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82729 2024-08-17T17:31:15.003 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.003 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:15.003 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.003 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:15.003 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.271858+0000", 2024-08-17T17:31:15.003 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:15.003 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.004 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.004 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:15.004 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.271858+0000", 2024-08-17T17:31:15.004 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:15.004 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.004 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.004 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:15.004 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.271857+0000", 2024-08-17T17:31:15.004 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:31:15.004 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.004 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.004 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:15.004 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.271917+0000", 2024-08-17T17:31:15.005 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.003e-05 2024-08-17T17:31:15.005 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.005 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.005 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:15.005 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.271933+0000", 2024-08-17T17:31:15.005 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6038000000000001e-05 2024-08-17T17:31:15.005 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.005 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.005 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:15.005 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.271939+0000", 2024-08-17T17:31:15.005 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4570000000000002e-06 2024-08-17T17:31:15.005 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.005 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.006 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:15.006 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.272016+0000", 2024-08-17T17:31:15.006 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7077999999999996e-05 2024-08-17T17:31:15.006 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.006 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.006 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:31:15.006 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.272027+0000", 2024-08-17T17:31:15.006 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0865e-05 2024-08-17T17:31:15.006 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.006 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.006 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:15.006 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.784655+0000", 2024-08-17T17:31:15.007 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5126276320000001 2024-08-17T17:31:15.007 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.007 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.007 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:15.007 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.784747+0000", 2024-08-17T17:31:15.007 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1725999999999996e-05 2024-08-17T17:31:15.007 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.007 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.007 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:31:15.007 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.784837+0000", 2024-08-17T17:31:15.007 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9955000000000002e-05 2024-08-17T17:31:15.007 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.007 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.008 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:15.008 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.793626+0000", 2024-08-17T17:31:15.008 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0087892579999999994 2024-08-17T17:31:15.008 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.008 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.008 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:15.008 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794430+0000", 2024-08-17T17:31:15.008 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00080423300000000001 2024-08-17T17:31:15.008 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.008 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.008 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:15.008 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.797697+0000", 2024-08-17T17:31:15.008 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0032663890000000002 2024-08-17T17:31:15.008 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.008 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.008 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:15.009 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.797741+0000", 2024-08-17T17:31:15.009 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4150999999999998e-05 2024-08-17T17:31:15.009 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.009 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.009 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:15.009 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.797828+0000", 2024-08-17T17:31:15.009 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.7589000000000005e-05 2024-08-17T17:31:15.009 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:15.009 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:15.009 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:15.009 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.009 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.009 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82733 6.d 6:b4deca26:::benchmark_data_smithi003_175043_object82732:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:31:15.009 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.278865+0000", 2024-08-17T17:31:15.009 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 335.664672397, 2024-08-17T17:31:15.010 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.525599463, 2024-08-17T17:31:15.010 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:15.010 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:15.010 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:15.010 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:31:15.010 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:31:15.010 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82733 2024-08-17T17:31:15.010 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.010 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:15.010 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.010 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:15.010 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.278865+0000", 2024-08-17T17:31:15.010 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:15.010 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.010 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.011 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:15.011 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.278865+0000", 2024-08-17T17:31:15.011 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:15.011 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.011 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.011 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:15.011 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.278863+0000", 2024-08-17T17:31:15.011 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:31:15.011 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.011 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.011 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:15.011 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.278920+0000", 2024-08-17T17:31:15.011 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6860000000000001e-05 2024-08-17T17:31:15.012 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.012 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.012 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:15.012 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.278933+0000", 2024-08-17T17:31:15.012 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3052e-05 2024-08-17T17:31:15.012 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.012 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.012 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:15.012 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.278938+0000", 2024-08-17T17:31:15.012 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9280000000000001e-06 2024-08-17T17:31:15.012 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.012 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.012 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:15.012 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.278995+0000", 2024-08-17T17:31:15.012 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7260999999999999e-05 2024-08-17T17:31:15.013 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.013 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.013 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:31:15.013 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.279003+0000", 2024-08-17T17:31:15.013 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1810000000000001e-06 2024-08-17T17:31:15.013 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.013 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.013 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:15.013 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.785779+0000", 2024-08-17T17:31:15.013 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5067753750000001 2024-08-17T17:31:15.013 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.013 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.013 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:15.013 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.785862+0000", 2024-08-17T17:31:15.014 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.3327999999999999e-05 2024-08-17T17:31:15.014 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.014 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.014 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:31:15.014 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.785964+0000", 2024-08-17T17:31:15.014 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000101952 2024-08-17T17:31:15.014 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.014 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.014 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:15.014 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.793630+0000", 2024-08-17T17:31:15.014 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0076662450000000004 2024-08-17T17:31:15.014 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.014 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.015 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:15.015 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.802127+0000", 2024-08-17T17:31:15.015 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0084964489999999997 2024-08-17T17:31:15.015 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.015 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.015 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:15.015 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.804270+0000", 2024-08-17T17:31:15.015 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0021435270000000001 2024-08-17T17:31:15.015 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.015 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.015 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:15.015 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.804336+0000", 2024-08-17T17:31:15.015 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5512999999999994e-05 2024-08-17T17:31:15.015 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.015 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.016 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:15.016 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.804464+0000", 2024-08-17T17:31:15.016 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012867299999999999 2024-08-17T17:31:15.016 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:15.016 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:15.016 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:15.016 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.016 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.017 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82738 6.d 6:b5082b22:::benchmark_data_smithi003_175043_object82737:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:31:15.017 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.285187+0000", 2024-08-17T17:31:15.017 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 335.658350482, 2024-08-17T17:31:15.017 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5194669009999999, 2024-08-17T17:31:15.017 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:15.017 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:15.017 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:15.018 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:31:15.018 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:31:15.018 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82738 2024-08-17T17:31:15.018 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.018 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:15.018 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.018 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:15.018 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.285187+0000", 2024-08-17T17:31:15.018 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:15.018 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.018 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.018 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:15.018 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.285187+0000", 2024-08-17T17:31:15.018 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:15.019 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.019 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.019 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:15.019 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.285185+0000", 2024-08-17T17:31:15.019 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:31:15.019 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.019 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.019 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:15.019 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.285242+0000", 2024-08-17T17:31:15.019 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7432999999999997e-05 2024-08-17T17:31:15.019 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.019 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.019 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:15.019 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.285256+0000", 2024-08-17T17:31:15.019 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3944000000000001e-05 2024-08-17T17:31:15.020 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.020 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.020 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:15.020 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.285262+0000", 2024-08-17T17:31:15.020 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.152e-06 2024-08-17T17:31:15.020 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.020 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.020 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:15.020 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.285305+0000", 2024-08-17T17:31:15.020 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3881999999999998e-05 2024-08-17T17:31:15.020 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.020 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.020 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:31:15.020 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.285313+0000", 2024-08-17T17:31:15.020 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.6320000000000008e-06 2024-08-17T17:31:15.021 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.021 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.021 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:15.021 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.786864+0000", 2024-08-17T17:31:15.021 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.501550613 2024-08-17T17:31:15.021 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.021 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.021 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:15.021 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.786943+0000", 2024-08-17T17:31:15.021 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.9118999999999999e-05 2024-08-17T17:31:15.021 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.021 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.021 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:31:15.021 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.787026+0000", 2024-08-17T17:31:15.022 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.3361000000000004e-05 2024-08-17T17:31:15.022 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.022 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.022 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:15.022 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.793634+0000", 2024-08-17T17:31:15.022 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0066076249999999998 2024-08-17T17:31:15.022 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.022 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.022 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:15.022 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.802156+0000", 2024-08-17T17:31:15.022 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0085225700000000001 2024-08-17T17:31:15.022 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.022 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.022 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:15.023 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.804510+0000", 2024-08-17T17:31:15.023 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0023532700000000002 2024-08-17T17:31:15.023 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.023 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.023 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:15.023 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.804545+0000", 2024-08-17T17:31:15.023 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5040999999999999e-05 2024-08-17T17:31:15.023 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.023 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.023 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:15.023 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.804654+0000", 2024-08-17T17:31:15.023 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000108884 2024-08-17T17:31:15.023 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:15.023 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:15.023 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:15.024 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.024 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.024 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82775 6.d 6:b9e7b196:::benchmark_data_smithi003_175043_object82774:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:31:15.024 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.335574+0000", 2024-08-17T17:31:15.024 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 335.60796284499997, 2024-08-17T17:31:15.024 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.469304108, 2024-08-17T17:31:15.024 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:15.024 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:15.024 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:15.024 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:31:15.024 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:31:15.024 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82775 2024-08-17T17:31:15.024 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.024 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:15.024 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.025 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:15.025 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.335574+0000", 2024-08-17T17:31:15.025 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:15.025 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.025 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.025 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:15.025 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.335574+0000", 2024-08-17T17:31:15.025 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:15.025 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.025 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.025 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:15.025 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.335573+0000", 2024-08-17T17:31:15.025 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:31:15.026 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.026 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.026 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:15.026 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.335620+0000", 2024-08-17T17:31:15.026 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7202e-05 2024-08-17T17:31:15.026 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.026 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.026 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:15.026 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.335632+0000", 2024-08-17T17:31:15.026 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1447999999999999e-05 2024-08-17T17:31:15.026 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.026 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.026 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:15.026 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.335636+0000", 2024-08-17T17:31:15.027 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.33e-06 2024-08-17T17:31:15.027 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.027 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.027 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:15.027 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.335695+0000", 2024-08-17T17:31:15.027 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8607000000000003e-05 2024-08-17T17:31:15.027 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.027 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.027 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:31:15.027 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.335707+0000", 2024-08-17T17:31:15.027 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2574e-05 2024-08-17T17:31:15.027 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.027 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.027 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:15.028 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.787914+0000", 2024-08-17T17:31:15.028 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.452207059 2024-08-17T17:31:15.028 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.028 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.028 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:15.028 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.787999+0000", 2024-08-17T17:31:15.028 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.4839999999999994e-05 2024-08-17T17:31:15.028 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.028 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.028 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:31:15.028 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.788083+0000", 2024-08-17T17:31:15.028 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.4035000000000004e-05 2024-08-17T17:31:15.028 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.028 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.028 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:15.029 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.793637+0000", 2024-08-17T17:31:15.029 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0055542530000000003 2024-08-17T17:31:15.029 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.029 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.029 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:15.029 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.803058+0000", 2024-08-17T17:31:15.029 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0094209629999999992 2024-08-17T17:31:15.029 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.029 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.029 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:15.029 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.804717+0000", 2024-08-17T17:31:15.029 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0016589689999999999 2024-08-17T17:31:15.029 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.029 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.029 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:15.030 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.804771+0000", 2024-08-17T17:31:15.030 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3931999999999998e-05 2024-08-17T17:31:15.030 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.030 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.030 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:15.030 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.804878+0000", 2024-08-17T17:31:15.030 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000107215 2024-08-17T17:31:15.030 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:15.030 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:15.030 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:15.030 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.030 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.030 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82786 6.d 6:b4168c45:::benchmark_data_smithi003_175043_object82785:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:31:15.030 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.351300+0000", 2024-08-17T17:31:15.031 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 335.59223699900002, 2024-08-17T17:31:15.031 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.454956208, 2024-08-17T17:31:15.031 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:15.031 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:15.031 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:15.031 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:31:15.031 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:31:15.031 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82786 2024-08-17T17:31:15.031 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.031 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:15.031 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.031 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:15.031 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.351300+0000", 2024-08-17T17:31:15.031 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:15.031 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.032 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.032 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:15.032 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.351300+0000", 2024-08-17T17:31:15.032 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:15.032 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.032 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.032 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:15.032 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.351298+0000", 2024-08-17T17:31:15.032 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:31:15.032 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.032 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.032 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:15.032 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.351330+0000", 2024-08-17T17:31:15.033 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1772e-05 2024-08-17T17:31:15.033 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.033 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.033 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:15.033 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.351347+0000", 2024-08-17T17:31:15.033 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6606e-05 2024-08-17T17:31:15.033 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.033 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.033 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:15.033 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.351353+0000", 2024-08-17T17:31:15.033 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4590000000000003e-06 2024-08-17T17:31:15.033 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.034 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.034 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:15.034 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.351412+0000", 2024-08-17T17:31:15.034 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8669999999999999e-05 2024-08-17T17:31:15.034 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.034 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.034 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:31:15.034 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.351421+0000", 2024-08-17T17:31:15.034 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.7309999999999999e-06 2024-08-17T17:31:15.034 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.034 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.035 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:15.035 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.788983+0000", 2024-08-17T17:31:15.035 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.437562008 2024-08-17T17:31:15.035 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.035 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.035 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:15.035 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.789094+0000", 2024-08-17T17:31:15.035 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0001115 2024-08-17T17:31:15.035 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.035 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.035 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:31:15.035 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.789196+0000", 2024-08-17T17:31:15.035 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000101718 2024-08-17T17:31:15.035 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.036 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.036 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:15.036 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.793641+0000", 2024-08-17T17:31:15.036 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0044450870000000003 2024-08-17T17:31:15.036 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.036 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.036 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:15.036 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.804993+0000", 2024-08-17T17:31:15.036 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011352258 2024-08-17T17:31:15.036 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.036 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.036 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:15.036 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.806095+0000", 2024-08-17T17:31:15.036 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0011017240000000001 2024-08-17T17:31:15.037 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.037 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.037 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:15.037 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.806146+0000", 2024-08-17T17:31:15.037 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1461000000000001e-05 2024-08-17T17:31:15.037 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.037 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.037 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:15.037 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.806256+0000", 2024-08-17T17:31:15.037 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000109967 2024-08-17T17:31:15.037 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:15.037 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:15.037 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:15.037 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.037 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.038 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82793 6.d 6:bca5da5d:::benchmark_data_smithi003_175043_object82792:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:31:15.038 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.361189+0000", 2024-08-17T17:31:15.038 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 335.58234767900001, 2024-08-17T17:31:15.038 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.446922399, 2024-08-17T17:31:15.038 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:15.038 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:15.038 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:15.038 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:31:15.038 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:31:15.038 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82793 2024-08-17T17:31:15.038 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.038 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:15.038 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.039 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:15.039 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.361189+0000", 2024-08-17T17:31:15.039 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:15.039 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.039 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.039 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:15.039 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.361189+0000", 2024-08-17T17:31:15.039 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:15.039 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.039 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.039 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:15.039 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.361188+0000", 2024-08-17T17:31:15.040 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:31:15.040 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.040 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.040 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:15.040 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.361225+0000", 2024-08-17T17:31:15.040 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.7224000000000001e-05 2024-08-17T17:31:15.040 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.040 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.040 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:15.040 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.361238+0000", 2024-08-17T17:31:15.040 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2315e-05 2024-08-17T17:31:15.040 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.040 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.041 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:15.041 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.361244+0000", 2024-08-17T17:31:15.041 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1040000000000003e-06 2024-08-17T17:31:15.041 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.041 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.041 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:15.041 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.361304+0000", 2024-08-17T17:31:15.041 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9828999999999999e-05 2024-08-17T17:31:15.041 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.041 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.041 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:31:15.041 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.361315+0000", 2024-08-17T17:31:15.041 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0871e-05 2024-08-17T17:31:15.041 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.042 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.042 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:15.042 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.790129+0000", 2024-08-17T17:31:15.042 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.428814585 2024-08-17T17:31:15.042 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.042 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.042 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:15.042 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.790209+0000", 2024-08-17T17:31:15.042 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0228999999999994e-05 2024-08-17T17:31:15.043 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.043 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.043 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:31:15.043 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.790308+0000", 2024-08-17T17:31:15.043 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.9118999999999997e-05 2024-08-17T17:31:15.043 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.043 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.043 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:15.043 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.793774+0000", 2024-08-17T17:31:15.043 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003465138 2024-08-17T17:31:15.043 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.043 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.043 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:15.043 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.805044+0000", 2024-08-17T17:31:15.043 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011270820000000001 2024-08-17T17:31:15.044 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.044 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.044 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:15.044 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.807955+0000", 2024-08-17T17:31:15.044 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0029106829999999998 2024-08-17T17:31:15.044 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.044 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.044 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:15.044 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.808009+0000", 2024-08-17T17:31:15.044 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3912999999999997e-05 2024-08-17T17:31:15.044 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.044 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.044 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:15.044 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.808112+0000", 2024-08-17T17:31:15.045 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000102864 2024-08-17T17:31:15.045 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:15.045 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:15.045 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:15.045 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.045 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.045 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82802 6.d 6:b0f72434:::benchmark_data_smithi003_175043_object82801:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:31:15.045 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.380777+0000", 2024-08-17T17:31:15.045 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 335.562760411, 2024-08-17T17:31:15.045 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.42952853, 2024-08-17T17:31:15.045 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:15.045 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:15.045 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:15.045 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:31:15.046 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:31:15.046 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82802 2024-08-17T17:31:15.046 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.046 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:15.046 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.046 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:15.046 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.380777+0000", 2024-08-17T17:31:15.046 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:15.046 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.046 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.046 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:15.046 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.380777+0000", 2024-08-17T17:31:15.046 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:15.047 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.047 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.047 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:15.047 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.380775+0000", 2024-08-17T17:31:15.047 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:31:15.047 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.047 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.047 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:15.047 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.380803+0000", 2024-08-17T17:31:15.047 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7353999999999999e-05 2024-08-17T17:31:15.047 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.047 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.047 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:15.047 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.380817+0000", 2024-08-17T17:31:15.047 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3946e-05 2024-08-17T17:31:15.048 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.048 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.048 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:15.048 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.380821+0000", 2024-08-17T17:31:15.048 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9490000000000004e-06 2024-08-17T17:31:15.048 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.048 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.048 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:15.048 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.380880+0000", 2024-08-17T17:31:15.048 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8761000000000002e-05 2024-08-17T17:31:15.048 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.048 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.048 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:31:15.049 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.380891+0000", 2024-08-17T17:31:15.049 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0876999999999999e-05 2024-08-17T17:31:15.049 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.049 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.049 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:15.049 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.791428+0000", 2024-08-17T17:31:15.049 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4105370399999999 2024-08-17T17:31:15.049 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.049 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.049 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:15.049 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.791499+0000", 2024-08-17T17:31:15.049 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.1247000000000004e-05 2024-08-17T17:31:15.049 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.050 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.050 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:31:15.050 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.791582+0000", 2024-08-17T17:31:15.050 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.2639000000000006e-05 2024-08-17T17:31:15.050 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.050 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.050 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:15.050 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.793778+0000", 2024-08-17T17:31:15.050 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0021961469999999999 2024-08-17T17:31:15.050 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.050 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.050 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:15.050 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.805082+0000", 2024-08-17T17:31:15.051 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011303863000000001 2024-08-17T17:31:15.051 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.051 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.051 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:15.051 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.810149+0000", 2024-08-17T17:31:15.051 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0050670180000000004 2024-08-17T17:31:15.051 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.051 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.051 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:15.051 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.810200+0000", 2024-08-17T17:31:15.051 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1430000000000001e-05 2024-08-17T17:31:15.051 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.051 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.051 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:15.052 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.810305+0000", 2024-08-17T17:31:15.052 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000104789 2024-08-17T17:31:15.052 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:15.052 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:15.052 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:15.052 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.052 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.052 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82805 6.d 6:b91bb233:::benchmark_data_smithi003_175043_object82804:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:31:15.052 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.389839+0000", 2024-08-17T17:31:15.052 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 335.553697806, 2024-08-17T17:31:15.052 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4206285620000001, 2024-08-17T17:31:15.052 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:15.052 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:15.052 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:15.053 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:31:15.053 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:31:15.053 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82805 2024-08-17T17:31:15.053 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.053 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:15.053 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.053 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:15.053 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.389839+0000", 2024-08-17T17:31:15.053 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:15.053 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.053 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.053 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:15.053 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.389839+0000", 2024-08-17T17:31:15.053 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:15.053 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.053 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.054 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:15.054 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.389838+0000", 2024-08-17T17:31:15.054 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:31:15.054 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.054 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.054 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:15.054 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.389863+0000", 2024-08-17T17:31:15.054 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4992000000000001e-05 2024-08-17T17:31:15.054 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.054 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.054 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:15.054 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.389877+0000", 2024-08-17T17:31:15.054 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3879999999999999e-05 2024-08-17T17:31:15.054 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.055 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.055 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:15.055 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.389882+0000", 2024-08-17T17:31:15.055 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1399999999999999e-06 2024-08-17T17:31:15.055 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.055 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.055 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:15.055 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.389935+0000", 2024-08-17T17:31:15.055 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2324000000000003e-05 2024-08-17T17:31:15.055 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.055 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.055 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:31:15.055 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.389941+0000", 2024-08-17T17:31:15.055 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5259999999999999e-06 2024-08-17T17:31:15.056 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.056 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.056 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:15.056 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.792504+0000", 2024-08-17T17:31:15.056 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4025628750000001 2024-08-17T17:31:15.056 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.056 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.056 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:15.056 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.792583+0000", 2024-08-17T17:31:15.056 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.9419999999999995e-05 2024-08-17T17:31:15.056 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.056 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.056 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:31:15.056 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.792679+0000", 2024-08-17T17:31:15.056 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5731999999999998e-05 2024-08-17T17:31:15.056 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.057 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.057 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:15.057 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.793981+0000", 2024-08-17T17:31:15.057 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001302213 2024-08-17T17:31:15.057 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.057 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.057 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:15.057 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.805220+0000", 2024-08-17T17:31:15.057 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011238685 2024-08-17T17:31:15.057 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.057 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.057 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:15.057 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.810345+0000", 2024-08-17T17:31:15.057 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0051250130000000003 2024-08-17T17:31:15.057 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.058 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.058 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:15.058 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.810387+0000", 2024-08-17T17:31:15.058 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2119000000000001e-05 2024-08-17T17:31:15.058 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.058 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.058 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:15.058 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.810468+0000", 2024-08-17T17:31:15.058 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0734000000000003e-05 2024-08-17T17:31:15.058 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:15.058 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:15.058 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:15.058 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.058 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.059 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:191527 6.d 6:b1fdfb8f:::benchmark_data_smithi003_175043_object92878:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-08-17T17:31:15.059 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:28:00.574230+0000", 2024-08-17T17:31:15.059 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 194.36930697099999, 2024-08-17T17:31:15.059 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.26055499700000001, 2024-08-17T17:31:15.059 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:15.059 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:15.059 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:15.059 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:31:15.059 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:31:15.059 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 191527 2024-08-17T17:31:15.059 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.059 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:15.059 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.059 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:15.059 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.574230+0000", 2024-08-17T17:31:15.060 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:15.060 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.060 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.060 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:15.060 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.574230+0000", 2024-08-17T17:31:15.060 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:15.060 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.060 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.060 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:15.060 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.574229+0000", 2024-08-17T17:31:15.060 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:31:15.060 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.060 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.060 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:15.061 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.574239+0000", 2024-08-17T17:31:15.061 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0087e-05 2024-08-17T17:31:15.061 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.061 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.061 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:15.061 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.574255+0000", 2024-08-17T17:31:15.061 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6018000000000001e-05 2024-08-17T17:31:15.061 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.061 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.061 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:15.061 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.574260+0000", 2024-08-17T17:31:15.061 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2619999999999998e-06 2024-08-17T17:31:15.061 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.062 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.062 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:15.062 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.574310+0000", 2024-08-17T17:31:15.062 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0043999999999997e-05 2024-08-17T17:31:15.062 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.062 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.062 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:15.062 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.574429+0000", 2024-08-17T17:31:15.062 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011841300000000001 2024-08-17T17:31:15.062 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.062 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.062 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:31:15.062 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.574544+0000", 2024-08-17T17:31:15.063 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011545 2024-08-17T17:31:15.063 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.063 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.063 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:15.063 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.579314+0000", 2024-08-17T17:31:15.063 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0047697069999999998 2024-08-17T17:31:15.063 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.063 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.063 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:15.063 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.580033+0000", 2024-08-17T17:31:15.063 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00071897599999999997 2024-08-17T17:31:15.063 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.063 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.063 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:15.064 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.834672+0000", 2024-08-17T17:31:15.064 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.25463945399999999 2024-08-17T17:31:15.064 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.064 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.064 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:15.064 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.834749+0000", 2024-08-17T17:31:15.064 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.6643999999999993e-05 2024-08-17T17:31:15.064 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.064 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.064 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:15.064 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.834785+0000", 2024-08-17T17:31:15.064 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6307999999999999e-05 2024-08-17T17:31:15.064 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:15.064 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:15.064 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:15.065 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.065 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.065 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:191540 6.d 6:bcc10983:::benchmark_data_smithi003_175043_object92891:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-08-17T17:31:15.065 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:28:00.590757+0000", 2024-08-17T17:31:15.065 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 194.352780263, 2024-08-17T17:31:15.065 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.25455507300000002, 2024-08-17T17:31:15.065 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:15.065 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:15.065 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:15.065 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:31:15.065 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:31:15.065 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 191540 2024-08-17T17:31:15.065 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.065 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:15.066 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.066 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:15.066 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.590757+0000", 2024-08-17T17:31:15.066 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:15.066 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.066 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.066 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:15.066 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.590757+0000", 2024-08-17T17:31:15.066 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:15.066 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.066 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.066 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:15.066 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.590756+0000", 2024-08-17T17:31:15.066 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:31:15.066 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.067 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.067 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:15.067 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.590760+0000", 2024-08-17T17:31:15.067 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.8010000000000003e-06 2024-08-17T17:31:15.067 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.067 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.067 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:15.067 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.590770+0000", 2024-08-17T17:31:15.067 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.825e-06 2024-08-17T17:31:15.067 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.067 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.067 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:15.067 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.590774+0000", 2024-08-17T17:31:15.067 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.002e-06 2024-08-17T17:31:15.067 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.068 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.068 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:15.068 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.590814+0000", 2024-08-17T17:31:15.068 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9489000000000001e-05 2024-08-17T17:31:15.068 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.068 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.068 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:15.068 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.590898+0000", 2024-08-17T17:31:15.068 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.4530999999999994e-05 2024-08-17T17:31:15.068 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.068 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.069 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:31:15.069 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.590976+0000", 2024-08-17T17:31:15.069 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7720999999999997e-05 2024-08-17T17:31:15.069 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.069 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.069 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:15.069 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.599290+0000", 2024-08-17T17:31:15.069 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0083138450000000003 2024-08-17T17:31:15.069 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.069 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.069 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:15.069 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.605077+0000", 2024-08-17T17:31:15.069 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0057869009999999997 2024-08-17T17:31:15.069 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.070 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.070 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:15.070 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.845239+0000", 2024-08-17T17:31:15.070 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.24016241699999999 2024-08-17T17:31:15.070 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.070 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.070 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:15.070 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.845274+0000", 2024-08-17T17:31:15.070 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5181000000000001e-05 2024-08-17T17:31:15.070 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.070 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.070 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:15.070 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.845312+0000", 2024-08-17T17:31:15.071 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.7567999999999997e-05 2024-08-17T17:31:15.071 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:15.071 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:15.071 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:15.071 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.071 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.072 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:14 smithi003 ceph-mon[100615]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:31:15.072 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:14 smithi003 ceph-mon[100615]: Cluster is now healthy 2024-08-17T17:31:15.072 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:14 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:31:15.072 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:14 smithi003 ceph-mon[100615]: osdmap e247: 8 total, 8 up, 7 in 2024-08-17T17:31:15.072 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:191543 6.d 6:b38b2e1c:::benchmark_data_smithi003_175043_object92894:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-08-17T17:31:15.072 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:28:00.592200+0000", 2024-08-17T17:31:15.072 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 194.35133676800001, 2024-08-17T17:31:15.072 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.25325235299999999, 2024-08-17T17:31:15.072 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:15.073 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:15.073 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:15.073 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:31:15.073 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:31:15.073 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 191543 2024-08-17T17:31:15.073 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.073 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:15.073 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.073 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:15.074 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.592200+0000", 2024-08-17T17:31:15.074 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:15.074 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.074 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.074 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:15.074 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.592200+0000", 2024-08-17T17:31:15.074 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:15.074 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.074 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.074 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:15.074 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.592199+0000", 2024-08-17T17:31:15.074 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:31:15.074 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.075 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.075 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:15.075 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.592205+0000", 2024-08-17T17:31:15.075 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3879999999999999e-06 2024-08-17T17:31:15.075 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.075 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.075 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:15.075 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.592218+0000", 2024-08-17T17:31:15.075 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3392e-05 2024-08-17T17:31:15.075 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.075 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.075 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:15.075 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.592223+0000", 2024-08-17T17:31:15.075 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5609999999999998e-06 2024-08-17T17:31:15.075 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.075 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.076 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:15.076 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.594600+0000", 2024-08-17T17:31:15.076 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0023767749999999998 2024-08-17T17:31:15.076 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.076 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.076 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:15.077 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.594722+0000", 2024-08-17T17:31:15.077 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000121311 2024-08-17T17:31:15.077 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.077 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.077 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:31:15.077 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.594837+0000", 2024-08-17T17:31:15.077 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011552099999999999 2024-08-17T17:31:15.077 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.077 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.077 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:15.078 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.599437+0000", 2024-08-17T17:31:15.078 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0045999839999999997 2024-08-17T17:31:15.078 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.078 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.078 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:15.078 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.605113+0000", 2024-08-17T17:31:15.078 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0056763769999999998 2024-08-17T17:31:15.078 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.078 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.078 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:15.078 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.845357+0000", 2024-08-17T17:31:15.078 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.24024379800000001 2024-08-17T17:31:15.078 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.079 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.079 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:15.079 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.845408+0000", 2024-08-17T17:31:15.079 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1208000000000002e-05 2024-08-17T17:31:15.079 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.079 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.079 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:15.079 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.845453+0000", 2024-08-17T17:31:15.079 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4279000000000001e-05 2024-08-17T17:31:15.079 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:15.079 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:15.079 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:15.079 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.079 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.080 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45767.0:105837 7.0 7:005b2fb9:::benchmark_data_smithi003_201798_object105836:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-08-17T17:31:15.080 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:30:02.222000+0000", 2024-08-17T17:31:15.080 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 72.721536868000001, 2024-08-17T17:31:15.080 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.17036312200000001, 2024-08-17T17:31:15.080 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:15.080 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:15.080 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:15.080 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45767", 2024-08-17T17:31:15.080 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:31:15.080 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 105837 2024-08-17T17:31:15.080 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.080 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:15.080 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.080 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:15.081 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:02.222000+0000", 2024-08-17T17:31:15.081 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:15.081 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.081 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.081 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:15.081 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:02.222000+0000", 2024-08-17T17:31:15.081 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:15.081 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.081 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.081 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:15.081 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:02.221999+0000", 2024-08-17T17:31:15.081 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:31:15.081 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.082 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.082 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:15.082 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:02.222069+0000", 2024-08-17T17:31:15.082 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0189000000000002e-05 2024-08-17T17:31:15.082 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.082 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.082 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:15.082 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:02.222079+0000", 2024-08-17T17:31:15.082 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0026e-05 2024-08-17T17:31:15.082 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.082 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.082 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:15.082 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:02.222083+0000", 2024-08-17T17:31:15.082 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9750000000000001e-06 2024-08-17T17:31:15.083 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.083 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.083 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:15.083 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:02.222111+0000", 2024-08-17T17:31:15.083 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7342e-05 2024-08-17T17:31:15.083 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.083 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.083 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:15.083 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:02.222193+0000", 2024-08-17T17:31:15.083 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.2625000000000003e-05 2024-08-17T17:31:15.083 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.083 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.083 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,3", 2024-08-17T17:31:15.083 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:02.222273+0000", 2024-08-17T17:31:15.083 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.9767999999999999e-05 2024-08-17T17:31:15.084 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.084 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.084 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:15.084 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:02.225586+0000", 2024-08-17T17:31:15.084 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0033129470000000001 2024-08-17T17:31:15.084 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.084 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.084 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:15.084 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:02.229083+0000", 2024-08-17T17:31:15.084 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0034971910000000002 2024-08-17T17:31:15.084 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.084 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.084 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:15.084 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:02.392298+0000", 2024-08-17T17:31:15.085 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.16321482200000001 2024-08-17T17:31:15.085 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.085 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.085 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:15.085 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:02.392337+0000", 2024-08-17T17:31:15.085 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9329000000000002e-05 2024-08-17T17:31:15.085 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:15.085 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:15.085 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:15.085 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:02.392363+0000", 2024-08-17T17:31:15.085 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5941000000000001e-05 2024-08-17T17:31:15.085 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:15.085 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:15.085 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:15.085 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:15.085 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:15.086 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:31:15.086 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:14 smithi003 ceph-mon[103344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:31:15.086 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:14 smithi003 ceph-mon[103344]: Cluster is now healthy 2024-08-17T17:31:15.086 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:14 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:31:15.086 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:14 smithi003 ceph-mon[103344]: osdmap e247: 8 total, 8 up, 7 in 2024-08-17T17:31:15.097 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:15.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:15.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:15.199 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:15.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:15.249 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:15.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:15.350 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:15.382 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:15.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:15.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:15.484 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:15.511 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:15.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:15.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:15.613 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:15.637 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:15.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:15.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:15.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:15 smithi053 ceph-mon[89963]: pgmap v718: 89 pgs: 89 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 755 op/s 2024-08-17T17:31:15.740 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:15.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:15 smithi003 ceph-mon[103344]: pgmap v718: 89 pgs: 89 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 755 op/s 2024-08-17T17:31:15.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:15 smithi003 ceph-mon[100615]: pgmap v718: 89 pgs: 89 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 755 op/s 2024-08-17T17:31:15.755 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:31:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:31:15] "GET /metrics HTTP/1.1" 200 36502 "" "Prometheus/2.51.0" 2024-08-17T17:31:15.784 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:15.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:15.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:15.866 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-17T17:31:15.886 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:15.928 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:15.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:15.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:16.029 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:16.074 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:16.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:16.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:16.175 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:16.190 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:31:16.198 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:16.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:16.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:16.300 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:16.322 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:16.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:16.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:16.424 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:16.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:16.473 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:16.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:16.574 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:16.602 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:16.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:16.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:16.703 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:16.731 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:16.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:16.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:16.833 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:16.875 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:16.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:16.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:16.977 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:17.021 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:17.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:17.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:17.123 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:17.170 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:17.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:17.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:17.271 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:17.304 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:17.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:17.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:17.405 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:17.443 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:17.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:17.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:17.545 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:17.572 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:17.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:17.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:17.674 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:17.713 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:17.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:17.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:17.815 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:17.842 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:17.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:17.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:17.938 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:17 smithi053 ceph-mon[89963]: pgmap v719: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 844 op/s 2024-08-17T17:31:17.944 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:17.966 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:31:17.967 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2024-08-17T17:31:17.967 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.967 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45767.0:165240 7.c 7:30b05922:::benchmark_data_smithi003_201798_object52018:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-08-17T17:31:17.967 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:31:17.963764+0000", 2024-08-17T17:31:17.967 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.001053195, 2024-08-17T17:31:17.967 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0010827619999999999, 2024-08-17T17:31:17.967 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:17.967 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:31:17.967 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:17.967 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45767", 2024-08-17T17:31:17.967 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:31:17.968 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 165240 2024-08-17T17:31:17.968 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.968 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:17.968 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.968 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:17.968 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.963764+0000", 2024-08-17T17:31:17.968 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:17.968 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.968 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.968 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:17.968 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.963764+0000", 2024-08-17T17:31:17.968 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:17.969 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.969 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.969 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:17.969 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.963762+0000", 2024-08-17T17:31:17.969 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:31:17.969 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.969 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.969 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:17.969 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.963770+0000", 2024-08-17T17:31:17.969 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.6090000000000003e-06 2024-08-17T17:31:17.969 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.969 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.969 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:17.969 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.963788+0000", 2024-08-17T17:31:17.970 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.8142e-05 2024-08-17T17:31:17.970 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.970 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.970 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:17.970 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.963796+0000", 2024-08-17T17:31:17.970 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.8520000000000004e-06 2024-08-17T17:31:17.970 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.970 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.970 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:17.970 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.963865+0000", 2024-08-17T17:31:17.970 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9329e-05 2024-08-17T17:31:17.970 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.971 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.971 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:17.971 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.963985+0000", 2024-08-17T17:31:17.971 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000120417 2024-08-17T17:31:17.971 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.971 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.971 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,7", 2024-08-17T17:31:17.971 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.964084+0000", 2024-08-17T17:31:17.971 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.8136999999999999e-05 2024-08-17T17:31:17.971 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:17.971 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:17.971 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:17.971 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.971 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.971 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.45767.0:165238 7.4 e247/226 7:26756adf:::benchmark_data_smithi003_201798_object52016:head v 247'10264, mlcod=247'10262)", 2024-08-17T17:31:17.972 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:31:17.963834+0000", 2024-08-17T17:31:17.972 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.000983486, 2024-08-17T17:31:17.972 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001062091, 2024-08-17T17:31:17.972 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:17.972 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2024-08-17T17:31:17.972 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:17.972 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.972 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:17.972 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.963834+0000", 2024-08-17T17:31:17.972 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:17.972 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.972 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.972 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:17.972 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.963834+0000", 2024-08-17T17:31:17.972 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:17.973 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.973 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.973 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:17.973 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.963833+0000", 2024-08-17T17:31:17.973 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:31:17.973 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.973 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.973 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:17.973 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.963839+0000", 2024-08-17T17:31:17.973 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0419999999999999e-06 2024-08-17T17:31:17.973 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.973 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.973 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:17.973 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.963848+0000", 2024-08-17T17:31:17.973 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.5259999999999993e-06 2024-08-17T17:31:17.974 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.974 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.974 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:17.974 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.963853+0000", 2024-08-17T17:31:17.974 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9960000000000001e-06 2024-08-17T17:31:17.974 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.974 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.974 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:17.974 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.963898+0000", 2024-08-17T17:31:17.974 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5287000000000002e-05 2024-08-17T17:31:17.974 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.974 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.974 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:17.974 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.963925+0000", 2024-08-17T17:31:17.975 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7168000000000001e-05 2024-08-17T17:31:17.975 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:17.975 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:17.975 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:17.975 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.975 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.975 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45767.0:165223 7.7 7:e267bd79:::benchmark_data_smithi003_201798_object52001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-08-17T17:31:17.975 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:31:17.946066+0000", 2024-08-17T17:31:17.975 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.018751384999999999, 2024-08-17T17:31:17.975 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.018877781999999999, 2024-08-17T17:31:17.976 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:17.976 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:31:17.976 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:17.976 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45767", 2024-08-17T17:31:17.976 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:31:17.976 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 165223 2024-08-17T17:31:17.976 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.976 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:17.976 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.976 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:17.976 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.946066+0000", 2024-08-17T17:31:17.976 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:17.976 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.976 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.976 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.946066+0000", 2024-08-17T17:31:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.946064+0000", 2024-08-17T17:31:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:31:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.946071+0000", 2024-08-17T17:31:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.8210000000000004e-06 2024-08-17T17:31:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.946087+0000", 2024-08-17T17:31:17.978 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5508000000000001e-05 2024-08-17T17:31:17.978 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.978 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.978 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:17.978 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.946093+0000", 2024-08-17T17:31:17.978 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5259999999999999e-06 2024-08-17T17:31:17.978 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.978 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.978 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:17.978 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.946140+0000", 2024-08-17T17:31:17.978 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6539000000000003e-05 2024-08-17T17:31:17.978 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.978 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.978 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:17.978 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.946280+0000", 2024-08-17T17:31:17.979 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014006000000000001 2024-08-17T17:31:17.979 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.979 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.979 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,3", 2024-08-17T17:31:17.979 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.946382+0000", 2024-08-17T17:31:17.979 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000102515 2024-08-17T17:31:17.979 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.979 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.979 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:17.979 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.952833+0000", 2024-08-17T17:31:17.979 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.006451048 2024-08-17T17:31:17.979 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.979 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.979 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:17.980 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.959728+0000", 2024-08-17T17:31:17.980 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.006895066 2024-08-17T17:31:17.980 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:17.980 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:17.980 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:17.980 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.980 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.980 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45767.0:165228 7.7 7:e61bb742:::benchmark_data_smithi003_201798_object52006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-08-17T17:31:17.980 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:31:17.947871+0000", 2024-08-17T17:31:17.980 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.016946220000000001, 2024-08-17T17:31:17.980 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017119612999999999, 2024-08-17T17:31:17.980 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:17.980 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:31:17.980 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:17.981 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45767", 2024-08-17T17:31:17.981 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:31:17.981 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 165228 2024-08-17T17:31:17.981 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.981 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:17.981 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.981 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:17.981 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.947871+0000", 2024-08-17T17:31:17.981 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:17.981 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.981 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.981 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:17.981 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.947871+0000", 2024-08-17T17:31:17.981 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:17.982 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.982 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.982 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:17.982 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.947870+0000", 2024-08-17T17:31:17.982 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:31:17.982 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.982 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.982 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:17.982 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.947875+0000", 2024-08-17T17:31:17.982 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2599999999999996e-06 2024-08-17T17:31:17.982 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.982 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.982 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:17.982 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.947885+0000", 2024-08-17T17:31:17.983 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0284e-05 2024-08-17T17:31:17.983 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.983 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.983 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:17.983 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.947891+0000", 2024-08-17T17:31:17.983 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7320000000000003e-06 2024-08-17T17:31:17.983 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.983 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.983 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:17.983 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.949736+0000", 2024-08-17T17:31:17.984 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001844856 2024-08-17T17:31:17.984 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.984 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.984 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:17.984 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.949835+0000", 2024-08-17T17:31:17.984 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.9260999999999997e-05 2024-08-17T17:31:17.984 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.984 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.984 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,3", 2024-08-17T17:31:17.984 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.949920+0000", 2024-08-17T17:31:17.984 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.5185000000000005e-05 2024-08-17T17:31:17.984 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.984 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.984 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:17.985 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.952965+0000", 2024-08-17T17:31:17.985 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0030447209999999998 2024-08-17T17:31:17.985 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:17.985 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:17.985 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:17.985 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:17.964417+0000", 2024-08-17T17:31:17.985 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011452189 2024-08-17T17:31:17.985 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:17.985 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:17.985 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:17.985 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:17.985 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-08-17T17:31:17.985 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 4 2024-08-17T17:31:17.985 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:31:17.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:17.986 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:17.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:18.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:17 smithi003 ceph-mon[103344]: pgmap v719: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 844 op/s 2024-08-17T17:31:18.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:17 smithi003 ceph-mon[100615]: pgmap v719: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 844 op/s 2024-08-17T17:31:18.087 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:18.111 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:18.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:18.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:18.213 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:18.242 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:18.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:18.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:18.344 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:18.368 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:18.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:18.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:18.470 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:18.516 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:18.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:18.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:18.618 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:18.656 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:18.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:18.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:18.758 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:18.784 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:18.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:18.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:18.886 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:18.897 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-17T17:31:18.923 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:18.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:18.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:19.025 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:19.062 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:19.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:19.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:19.163 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:19.205 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:19.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:19.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:19.256 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:31:19.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:19.351 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:19.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:19.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:19.453 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:19.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:19.477 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:19.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:19.578 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:19.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:19.623 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:19.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:19.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:19 smithi053 ceph-mon[89963]: pgmap v720: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 702 op/s 2024-08-17T17:31:19.724 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:19.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:19.766 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:19.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:19.867 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:19.882 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:19 smithi003 ceph-mon[103344]: pgmap v720: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 702 op/s 2024-08-17T17:31:19.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:19 smithi003 ceph-mon[100615]: pgmap v720: 89 pgs: 89 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 702 op/s 2024-08-17T17:31:19.915 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:19.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:19.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:20.017 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:20.092 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:20.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:20.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:20.195 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:20.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:20.240 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:20.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:20.341 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:20.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:20.382 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:20.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:20.416 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:31:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:31:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:31:20.482 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:20.511 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:20.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:20.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:20.612 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:20.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:20.656 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:20.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:20.757 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:20.816 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:20.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:20.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:20.918 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:20.947 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:20.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:20.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:21.049 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:21.072 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:21.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:21.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:21.173 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:21.224 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:21.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:21.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:21.326 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:21.350 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:21.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:21.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:21.452 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:21.483 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:21.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:21.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:21.585 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:21.590 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:31:21.592 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:31:21.623 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:21.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:21.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:21.650 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:31:21.707 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:31:21.707 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2024-08-17T17:31:21.707 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2024-08-17T17:31:21.708 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2024-08-17T17:31:21.708 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:31:21.725 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:21.759 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:21.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:21.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:21.861 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:21.873 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:21 smithi053 ceph-mon[89963]: pgmap v721: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 749 op/s 2024-08-17T17:31:21.873 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:21 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:31:21.888 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:21.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:21.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:21.990 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:22.019 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:22.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:22.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:22.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:21 smithi003 ceph-mon[100615]: pgmap v721: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 749 op/s 2024-08-17T17:31:22.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:21 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:31:22.026 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:21 smithi003 ceph-mon[103344]: pgmap v721: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 749 op/s 2024-08-17T17:31:22.026 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:21 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:31:22.122 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:22.151 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:22.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:22.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:22.253 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:22.279 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:22.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:22.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:22.381 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:22.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:22.428 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:22.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:22.529 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:22.552 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:31:22.573 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:22.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:22.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:22.628 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":721,"stamp":"2024-08-17T17:31:20.020695+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3996522469,"num_objects":60938,"num_object_clones":0,"num_object_copies":182814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60938,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":166652,"num_write_kb":7261056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":11937158,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123311,"ondisk_log_size":123311,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":14498024,"kb_used_data":11814828,"kb_used_omap":33,"kb_used_meta":2683038,"kb_avail":641718040,"statfs":{"total":671965249536,"available":657119272960,"internally_reserved":0,"allocated":12098383872,"data_stored":12088750714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34068,"internal_metadata":2747431660},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,6,4,12,9,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-491454464,"num_objects":-7499,"num_object_clones":0,"num_object_copies":-22497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7499,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003839"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":349,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.606742+0000","last_change":"2024-08-17T17:17:32.495652+0000","last_active":"2024-08-17T17:31:13.606742+0000","last_peered":"2024-08-17T17:31:13.606742+0000","last_clean":"2024-08-17T17:31:13.606742+0000","last_became_active":"2024-08-17T17:17:32.495218+0000","last_became_peered":"2024-08-17T17:17:32.495218+0000","last_unstale":"2024-08-17T17:31:13.606742+0000","last_undegraded":"2024-08-17T17:31:13.606742+0000","last_fullsized":"2024-08-17T17:31:13.606742+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:11:45.821282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":223,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.555074+0000","last_change":"2024-08-17T17:18:59.424887+0000","last_active":"2024-08-17T17:31:08.555074+0000","last_peered":"2024-08-17T17:31:08.555074+0000","last_clean":"2024-08-17T17:31:08.555074+0000","last_became_active":"2024-08-17T17:18:59.411539+0000","last_became_peered":"2024-08-17T17:18:59.411539+0000","last_unstale":"2024-08-17T17:31:08.555074+0000","last_undegraded":"2024-08-17T17:31:08.555074+0000","last_fullsized":"2024-08-17T17:31:08.555074+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:46:20.082241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":63,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.553710+0000","last_change":"2024-08-17T17:24:53.460742+0000","last_active":"2024-08-17T17:31:08.553710+0000","last_peered":"2024-08-17T17:31:08.553710+0000","last_clean":"2024-08-17T17:31:08.553710+0000","last_became_active":"2024-08-17T17:24:53.460365+0000","last_became_peered":"2024-08-17T17:24:53.460365+0000","last_unstale":"2024-08-17T17:31:08.553710+0000","last_undegraded":"2024-08-17T17:31:08.553710+0000","last_fullsized":"2024-08-17T17:31:08.553710+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:24:08.116968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":161,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.597684+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:31:13.597684+0000","last_peered":"2024-08-17T17:31:13.597684+0000","last_clean":"2024-08-17T17:31:13.597684+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:31:13.597684+0000","last_undegraded":"2024-08-17T17:31:13.597684+0000","last_fullsized":"2024-08-17T17:31:13.597684+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":160,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.599716+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:31:13.599716+0000","last_peered":"2024-08-17T17:31:13.599716+0000","last_clean":"2024-08-17T17:31:13.599716+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:31:13.599716+0000","last_undegraded":"2024-08-17T17:31:13.599716+0000","last_fullsized":"2024-08-17T17:31:13.599716+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":198,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.554987+0000","last_change":"2024-08-17T17:20:49.258206+0000","last_active":"2024-08-17T17:31:08.554987+0000","last_peered":"2024-08-17T17:31:08.554987+0000","last_clean":"2024-08-17T17:31:08.554987+0000","last_became_active":"2024-08-17T17:20:49.257761+0000","last_became_peered":"2024-08-17T17:20:49.257761+0000","last_unstale":"2024-08-17T17:31:08.554987+0000","last_undegraded":"2024-08-17T17:31:08.554987+0000","last_fullsized":"2024-08-17T17:31:08.554987+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:38:35.363748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":363,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.616727+0000","last_change":"2024-08-17T17:19:57.514773+0000","last_active":"2024-08-17T17:31:13.616727+0000","last_peered":"2024-08-17T17:31:13.616727+0000","last_clean":"2024-08-17T17:31:13.616727+0000","last_became_active":"2024-08-17T17:19:57.513839+0000","last_became_peered":"2024-08-17T17:19:57.513839+0000","last_unstale":"2024-08-17T17:31:13.616727+0000","last_undegraded":"2024-08-17T17:31:13.616727+0000","last_fullsized":"2024-08-17T17:31:13.616727+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:42:48.063169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":293,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.559456+0000","last_change":"2024-08-17T17:24:53.465875+0000","last_active":"2024-08-17T17:31:08.559456+0000","last_peered":"2024-08-17T17:31:08.559456+0000","last_clean":"2024-08-17T17:31:08.559456+0000","last_became_active":"2024-08-17T17:24:53.458129+0000","last_became_peered":"2024-08-17T17:24:53.458129+0000","last_unstale":"2024-08-17T17:31:08.559456+0000","last_undegraded":"2024-08-17T17:31:08.559456+0000","last_fullsized":"2024-08-17T17:31:08.559456+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:51:33.584869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"247'10307","reported_seq":10360,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:15.806625+0000","last_change":"2024-08-17T17:28:27.510969+0000","last_active":"2024-08-17T17:31:15.806625+0000","last_peered":"2024-08-17T17:31:15.806625+0000","last_clean":"2024-08-17T17:31:15.806625+0000","last_became_active":"2024-08-17T17:28:27.510664+0000","last_became_peered":"2024-08-17T17:28:27.510664+0000","last_unstale":"2024-08-17T17:31:15.806625+0000","last_undegraded":"2024-08-17T17:31:15.806625+0000","last_fullsized":"2024-08-17T17:31:15.806625+0000","mapping_epoch":226,"log_start":"233'3600","ondisk_log_start":"233'3600","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":6707,"log_dups_size":0,"ondisk_log_size":6707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:16:23.845426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259719168,"num_objects":3963,"num_object_clones":0,"num_object_copies":11889,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10307,"num_write_kb":456640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":159,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.555557+0000","last_change":"2024-08-17T17:24:06.597382+0000","last_active":"2024-08-17T17:31:08.555557+0000","last_peered":"2024-08-17T17:31:08.555557+0000","last_clean":"2024-08-17T17:31:08.555557+0000","last_became_active":"2024-08-17T17:24:06.588881+0000","last_became_peered":"2024-08-17T17:24:06.588881+0000","last_unstale":"2024-08-17T17:31:08.555557+0000","last_undegraded":"2024-08-17T17:31:08.555557+0000","last_fullsized":"2024-08-17T17:31:08.555557+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:27.240751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":73,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.553306+0000","last_change":"2024-08-17T17:25:04.503780+0000","last_active":"2024-08-17T17:31:08.553306+0000","last_peered":"2024-08-17T17:31:08.553306+0000","last_clean":"2024-08-17T17:31:08.553306+0000","last_became_active":"2024-08-17T17:25:04.503503+0000","last_became_peered":"2024-08-17T17:25:04.503503+0000","last_unstale":"2024-08-17T17:31:08.553306+0000","last_undegraded":"2024-08-17T17:31:08.553306+0000","last_fullsized":"2024-08-17T17:31:08.553306+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:04:48.171216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"247'10291","reported_seq":10345,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:17.458041+0000","last_change":"2024-08-17T17:28:27.509334+0000","last_active":"2024-08-17T17:31:17.458041+0000","last_peered":"2024-08-17T17:31:17.458041+0000","last_clean":"2024-08-17T17:31:17.458041+0000","last_became_active":"2024-08-17T17:28:27.509026+0000","last_became_peered":"2024-08-17T17:28:27.509026+0000","last_unstale":"2024-08-17T17:31:17.458041+0000","last_undegraded":"2024-08-17T17:31:17.458041+0000","last_fullsized":"2024-08-17T17:31:17.458041+0000","mapping_epoch":226,"log_start":"233'3400","ondisk_log_start":"233'3400","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":6891,"log_dups_size":0,"ondisk_log_size":6891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:47:28.720615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246743040,"num_objects":3765,"num_object_clones":0,"num_object_copies":11295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10291,"num_write_kb":449792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":145,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.555398+0000","last_change":"2024-08-17T17:24:06.599023+0000","last_active":"2024-08-17T17:31:08.555398+0000","last_peered":"2024-08-17T17:31:08.555398+0000","last_clean":"2024-08-17T17:31:08.555398+0000","last_became_active":"2024-08-17T17:24:06.588954+0000","last_became_peered":"2024-08-17T17:24:06.588954+0000","last_unstale":"2024-08-17T17:31:08.555398+0000","last_undegraded":"2024-08-17T17:31:08.555398+0000","last_fullsized":"2024-08-17T17:31:08.555398+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:20:51.445576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":304,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.631783+0000","last_change":"2024-08-17T17:20:49.271540+0000","last_active":"2024-08-17T17:31:13.631783+0000","last_peered":"2024-08-17T17:31:13.631783+0000","last_clean":"2024-08-17T17:31:13.631783+0000","last_became_active":"2024-08-17T17:20:49.271130+0000","last_became_peered":"2024-08-17T17:20:49.271130+0000","last_unstale":"2024-08-17T17:31:13.631783+0000","last_undegraded":"2024-08-17T17:31:13.631783+0000","last_fullsized":"2024-08-17T17:31:13.631783+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:05.386705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.9","version":"247'10566","reported_seq":10619,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:18.434092+0000","last_change":"2024-08-17T17:28:27.508033+0000","last_active":"2024-08-17T17:31:18.434092+0000","last_peered":"2024-08-17T17:31:18.434092+0000","last_clean":"2024-08-17T17:31:18.434092+0000","last_became_active":"2024-08-17T17:28:27.507657+0000","last_became_peered":"2024-08-17T17:28:27.507657+0000","last_unstale":"2024-08-17T17:31:18.434092+0000","last_undegraded":"2024-08-17T17:31:18.434092+0000","last_fullsized":"2024-08-17T17:31:18.434092+0000","mapping_epoch":226,"log_start":"233'3200","ondisk_log_start":"233'3200","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":7366,"log_dups_size":0,"ondisk_log_size":7366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:09:04.908184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252575744,"num_objects":3854,"num_object_clones":0,"num_object_copies":11562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10566,"num_write_kb":461440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":217,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.555411+0000","last_change":"2024-08-17T17:24:53.467526+0000","last_active":"2024-08-17T17:31:08.555411+0000","last_peered":"2024-08-17T17:31:08.555411+0000","last_clean":"2024-08-17T17:31:08.555411+0000","last_became_active":"2024-08-17T17:24:53.460631+0000","last_became_peered":"2024-08-17T17:24:53.460631+0000","last_unstale":"2024-08-17T17:31:08.555411+0000","last_undegraded":"2024-08-17T17:31:08.555411+0000","last_fullsized":"2024-08-17T17:31:08.555411+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:05:24.938831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":59,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.559798+0000","last_change":"2024-08-17T17:25:04.507176+0000","last_active":"2024-08-17T17:31:08.559798+0000","last_peered":"2024-08-17T17:31:08.559798+0000","last_clean":"2024-08-17T17:31:08.559798+0000","last_became_active":"2024-08-17T17:25:04.506788+0000","last_became_peered":"2024-08-17T17:25:04.506788+0000","last_unstale":"2024-08-17T17:31:08.559798+0000","last_undegraded":"2024-08-17T17:31:08.559798+0000","last_fullsized":"2024-08-17T17:31:08.559798+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:22:04.658216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"247'10722","reported_seq":10751,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:19.100593+0000","last_change":"2024-08-17T17:28:27.509743+0000","last_active":"2024-08-17T17:31:19.100593+0000","last_peered":"2024-08-17T17:31:19.100593+0000","last_clean":"2024-08-17T17:31:19.100593+0000","last_became_active":"2024-08-17T17:28:27.509414+0000","last_became_peered":"2024-08-17T17:28:27.509414+0000","last_unstale":"2024-08-17T17:31:19.100593+0000","last_undegraded":"2024-08-17T17:31:19.100593+0000","last_fullsized":"2024-08-17T17:31:19.100593+0000","mapping_epoch":226,"log_start":"233'3200","ondisk_log_start":"233'3200","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":7522,"log_dups_size":0,"ondisk_log_size":7522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:37:42.377123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251920384,"num_objects":3844,"num_object_clones":0,"num_object_copies":11532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10722,"num_write_kb":466112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":205,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.553538+0000","last_change":"2024-08-17T17:17:32.495428+0000","last_active":"2024-08-17T17:31:08.553538+0000","last_peered":"2024-08-17T17:31:08.553538+0000","last_clean":"2024-08-17T17:31:08.553538+0000","last_became_active":"2024-08-17T17:17:32.494978+0000","last_became_peered":"2024-08-17T17:17:32.494978+0000","last_unstale":"2024-08-17T17:31:08.553538+0000","last_undegraded":"2024-08-17T17:31:08.553538+0000","last_fullsized":"2024-08-17T17:31:08.553538+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:42:24.647812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":352,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.598177+0000","last_change":"2024-08-17T17:20:49.272126+0000","last_active":"2024-08-17T17:31:13.598177+0000","last_peered":"2024-08-17T17:31:13.598177+0000","last_clean":"2024-08-17T17:31:13.598177+0000","last_became_active":"2024-08-17T17:20:49.271185+0000","last_became_peered":"2024-08-17T17:20:49.271185+0000","last_unstale":"2024-08-17T17:31:13.598177+0000","last_undegraded":"2024-08-17T17:31:13.598177+0000","last_fullsized":"2024-08-17T17:31:13.598177+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:43:58.184939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"247'10401","reported_seq":10455,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:18.386403+0000","last_change":"2024-08-17T17:28:27.512879+0000","last_active":"2024-08-17T17:31:18.386403+0000","last_peered":"2024-08-17T17:31:18.386403+0000","last_clean":"2024-08-17T17:31:18.386403+0000","last_became_active":"2024-08-17T17:28:27.512586+0000","last_became_peered":"2024-08-17T17:28:27.512586+0000","last_unstale":"2024-08-17T17:31:18.386403+0000","last_undegraded":"2024-08-17T17:31:18.386403+0000","last_fullsized":"2024-08-17T17:31:18.386403+0000","mapping_epoch":226,"log_start":"232'3000","ondisk_log_start":"232'3000","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":7401,"log_dups_size":0,"ondisk_log_size":7401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:12:27.175368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251461632,"num_objects":3837,"num_object_clones":0,"num_object_copies":11511,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10401,"num_write_kb":455616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":381,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.612472+0000","last_change":"2024-08-17T17:24:06.595380+0000","last_active":"2024-08-17T17:31:13.612472+0000","last_peered":"2024-08-17T17:31:13.612472+0000","last_clean":"2024-08-17T17:31:13.612472+0000","last_became_active":"2024-08-17T17:24:06.584747+0000","last_became_peered":"2024-08-17T17:24:06.584747+0000","last_unstale":"2024-08-17T17:31:13.612472+0000","last_undegraded":"2024-08-17T17:31:13.612472+0000","last_fullsized":"2024-08-17T17:31:13.612472+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:02:18.473980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":326,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.606986+0000","last_change":"2024-08-17T17:20:49.280596+0000","last_active":"2024-08-17T17:31:13.606986+0000","last_peered":"2024-08-17T17:31:13.606986+0000","last_clean":"2024-08-17T17:31:13.606986+0000","last_became_active":"2024-08-17T17:20:49.279553+0000","last_became_peered":"2024-08-17T17:20:49.279553+0000","last_unstale":"2024-08-17T17:31:13.606986+0000","last_undegraded":"2024-08-17T17:31:13.606986+0000","last_fullsized":"2024-08-17T17:31:13.606986+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:13:24.124392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"247'10211","reported_seq":10265,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:19.733110+0000","last_change":"2024-08-17T17:28:27.506638+0000","last_active":"2024-08-17T17:31:19.733110+0000","last_peered":"2024-08-17T17:31:19.733110+0000","last_clean":"2024-08-17T17:31:19.733110+0000","last_became_active":"2024-08-17T17:28:27.506343+0000","last_became_peered":"2024-08-17T17:28:27.506343+0000","last_unstale":"2024-08-17T17:31:19.733110+0000","last_undegraded":"2024-08-17T17:31:19.733110+0000","last_fullsized":"2024-08-17T17:31:19.733110+0000","mapping_epoch":226,"log_start":"232'200","ondisk_log_start":"232'200","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:57:42.320556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237568000,"num_objects":3625,"num_object_clones":0,"num_object_copies":10875,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10211,"num_write_kb":442752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":278,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.622988+0000","last_change":"2024-08-17T17:20:49.276769+0000","last_active":"2024-08-17T17:31:13.622988+0000","last_peered":"2024-08-17T17:31:13.622988+0000","last_clean":"2024-08-17T17:31:13.622988+0000","last_became_active":"2024-08-17T17:20:49.276498+0000","last_became_peered":"2024-08-17T17:20:49.276498+0000","last_unstale":"2024-08-17T17:31:13.622988+0000","last_undegraded":"2024-08-17T17:31:13.622988+0000","last_fullsized":"2024-08-17T17:31:13.622988+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:39:05.419440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":226,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.555295+0000","last_change":"2024-08-17T17:20:14.455738+0000","last_active":"2024-08-17T17:31:08.555295+0000","last_peered":"2024-08-17T17:31:08.555295+0000","last_clean":"2024-08-17T17:31:08.555295+0000","last_became_active":"2024-08-17T17:20:14.455469+0000","last_became_peered":"2024-08-17T17:20:14.455469+0000","last_unstale":"2024-08-17T17:31:08.555295+0000","last_undegraded":"2024-08-17T17:31:08.555295+0000","last_fullsized":"2024-08-17T17:31:08.555295+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:27:28.424336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.d","version":"247'10213","reported_seq":10243,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:18.986443+0000","last_change":"2024-08-17T17:28:27.510620+0000","last_active":"2024-08-17T17:31:18.986443+0000","last_peered":"2024-08-17T17:31:18.986443+0000","last_clean":"2024-08-17T17:31:18.986443+0000","last_became_active":"2024-08-17T17:28:27.510351+0000","last_became_peered":"2024-08-17T17:28:27.510351+0000","last_unstale":"2024-08-17T17:31:18.986443+0000","last_undegraded":"2024-08-17T17:31:18.986443+0000","last_fullsized":"2024-08-17T17:31:18.986443+0000","mapping_epoch":226,"log_start":"232'2500","ondisk_log_start":"232'2500","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":7713,"log_dups_size":0,"ondisk_log_size":7713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:14:19.994370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243531800,"num_objects":3717,"num_object_clones":0,"num_object_copies":11151,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3717,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10213,"num_write_kb":445697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":180,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.555040+0000","last_change":"2024-08-17T17:20:14.431140+0000","last_active":"2024-08-17T17:31:08.555040+0000","last_peered":"2024-08-17T17:31:08.555040+0000","last_clean":"2024-08-17T17:31:08.555040+0000","last_became_active":"2024-08-17T17:20:14.430839+0000","last_became_peered":"2024-08-17T17:20:14.430839+0000","last_unstale":"2024-08-17T17:31:08.555040+0000","last_undegraded":"2024-08-17T17:31:08.555040+0000","last_fullsized":"2024-08-17T17:31:08.555040+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:08:16.303616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":59,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.560121+0000","last_change":"2024-08-17T17:25:04.509325+0000","last_active":"2024-08-17T17:31:08.560121+0000","last_peered":"2024-08-17T17:31:08.560121+0000","last_clean":"2024-08-17T17:31:08.560121+0000","last_became_active":"2024-08-17T17:25:04.508364+0000","last_became_peered":"2024-08-17T17:25:04.508364+0000","last_unstale":"2024-08-17T17:31:08.560121+0000","last_undegraded":"2024-08-17T17:31:08.560121+0000","last_fullsized":"2024-08-17T17:31:08.560121+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:48:56.415763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"247'10311","reported_seq":10340,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:19.019130+0000","last_change":"2024-08-17T17:28:27.512014+0000","last_active":"2024-08-17T17:31:19.019130+0000","last_peered":"2024-08-17T17:31:19.019130+0000","last_clean":"2024-08-17T17:31:19.019130+0000","last_became_active":"2024-08-17T17:28:27.511692+0000","last_became_peered":"2024-08-17T17:28:27.511692+0000","last_unstale":"2024-08-17T17:31:19.019130+0000","last_undegraded":"2024-08-17T17:31:19.019130+0000","last_fullsized":"2024-08-17T17:31:19.019130+0000","mapping_epoch":226,"log_start":"232'2600","ondisk_log_start":"232'2600","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":7711,"log_dups_size":0,"ondisk_log_size":7711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:38:59.458491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249102336,"num_objects":3801,"num_object_clones":0,"num_object_copies":11403,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10311,"num_write_kb":451584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":365,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.616913+0000","last_change":"2024-08-17T17:19:57.520745+0000","last_active":"2024-08-17T17:31:13.616913+0000","last_peered":"2024-08-17T17:31:13.616913+0000","last_clean":"2024-08-17T17:31:13.616913+0000","last_became_active":"2024-08-17T17:19:57.520152+0000","last_became_peered":"2024-08-17T17:19:57.520152+0000","last_unstale":"2024-08-17T17:31:13.616913+0000","last_undegraded":"2024-08-17T17:31:13.616913+0000","last_fullsized":"2024-08-17T17:31:13.616913+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:19:39.495644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":340,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.605864+0000","last_change":"2024-08-17T17:20:49.274372+0000","last_active":"2024-08-17T17:31:13.605864+0000","last_peered":"2024-08-17T17:31:13.605864+0000","last_clean":"2024-08-17T17:31:13.605864+0000","last_became_active":"2024-08-17T17:20:49.274023+0000","last_became_peered":"2024-08-17T17:20:49.274023+0000","last_unstale":"2024-08-17T17:31:13.605864+0000","last_undegraded":"2024-08-17T17:31:13.605864+0000","last_fullsized":"2024-08-17T17:31:13.605864+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:31:02.529766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"247'10299","reported_seq":10353,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:18.402116+0000","last_change":"2024-08-17T17:28:27.511026+0000","last_active":"2024-08-17T17:31:18.402116+0000","last_peered":"2024-08-17T17:31:18.402116+0000","last_clean":"2024-08-17T17:31:18.402116+0000","last_became_active":"2024-08-17T17:28:27.510738+0000","last_became_peered":"2024-08-17T17:28:27.510738+0000","last_unstale":"2024-08-17T17:31:18.402116+0000","last_undegraded":"2024-08-17T17:31:18.402116+0000","last_fullsized":"2024-08-17T17:31:18.402116+0000","mapping_epoch":226,"log_start":"232'2900","ondisk_log_start":"232'2900","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":7399,"log_dups_size":0,"ondisk_log_size":7399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:58:33.553008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247922688,"num_objects":3783,"num_object_clones":0,"num_object_copies":11349,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10299,"num_write_kb":450624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":574,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.554882+0000","last_change":"2024-08-17T17:24:09.599615+0000","last_active":"2024-08-17T17:31:08.554882+0000","last_peered":"2024-08-17T17:31:08.554882+0000","last_clean":"2024-08-17T17:31:08.554882+0000","last_became_active":"2024-08-17T17:24:09.599085+0000","last_became_peered":"2024-08-17T17:24:09.599085+0000","last_unstale":"2024-08-17T17:31:08.554882+0000","last_undegraded":"2024-08-17T17:31:08.554882+0000","last_fullsized":"2024-08-17T17:31:08.554882+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:04:01.141401+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:30:09.569579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":819,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.553979+0000","last_change":"2024-08-17T17:20:00.434774+0000","last_active":"2024-08-17T17:31:08.553979+0000","last_peered":"2024-08-17T17:31:08.553979+0000","last_clean":"2024-08-17T17:31:08.553979+0000","last_became_active":"2024-08-17T17:20:00.434343+0000","last_became_peered":"2024-08-17T17:20:00.434343+0000","last_unstale":"2024-08-17T17:31:08.553979+0000","last_undegraded":"2024-08-17T17:31:08.553979+0000","last_fullsized":"2024-08-17T17:31:08.553979+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:04.995152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":323,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.606233+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:31:13.606233+0000","last_peered":"2024-08-17T17:31:13.606233+0000","last_clean":"2024-08-17T17:31:13.606233+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:31:13.606233+0000","last_undegraded":"2024-08-17T17:31:13.606233+0000","last_fullsized":"2024-08-17T17:31:13.606233+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":59,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.559844+0000","last_change":"2024-08-17T17:25:04.579426+0000","last_active":"2024-08-17T17:31:08.559844+0000","last_peered":"2024-08-17T17:31:08.559844+0000","last_clean":"2024-08-17T17:31:08.559844+0000","last_became_active":"2024-08-17T17:25:04.579012+0000","last_became_peered":"2024-08-17T17:25:04.579012+0000","last_unstale":"2024-08-17T17:31:08.559844+0000","last_undegraded":"2024-08-17T17:31:08.559844+0000","last_fullsized":"2024-08-17T17:31:08.559844+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:56:32.241321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"247'10370","reported_seq":10399,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:19.081228+0000","last_change":"2024-08-17T17:28:27.506760+0000","last_active":"2024-08-17T17:31:19.081228+0000","last_peered":"2024-08-17T17:31:19.081228+0000","last_clean":"2024-08-17T17:31:19.081228+0000","last_became_active":"2024-08-17T17:28:27.506306+0000","last_became_peered":"2024-08-17T17:28:27.506306+0000","last_unstale":"2024-08-17T17:31:19.081228+0000","last_undegraded":"2024-08-17T17:31:19.081228+0000","last_fullsized":"2024-08-17T17:31:19.081228+0000","mapping_epoch":226,"log_start":"232'2800","ondisk_log_start":"232'2800","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":7570,"log_dups_size":0,"ondisk_log_size":7570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:41:13.520504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249430016,"num_objects":3806,"num_object_clones":0,"num_object_copies":11418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10370,"num_write_kb":453632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":837,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.555094+0000","last_change":"2024-08-17T17:16:26.958415+0000","last_active":"2024-08-17T17:31:08.555094+0000","last_peered":"2024-08-17T17:31:08.555094+0000","last_clean":"2024-08-17T17:31:08.555094+0000","last_became_active":"2024-08-17T17:16:26.957956+0000","last_became_peered":"2024-08-17T17:16:26.957956+0000","last_unstale":"2024-08-17T17:31:08.555094+0000","last_undegraded":"2024-08-17T17:31:08.555094+0000","last_fullsized":"2024-08-17T17:31:08.555094+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:22.478077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":352,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.602441+0000","last_change":"2024-08-17T17:18:59.415331+0000","last_active":"2024-08-17T17:31:13.602441+0000","last_peered":"2024-08-17T17:31:13.602441+0000","last_clean":"2024-08-17T17:31:13.602441+0000","last_became_active":"2024-08-17T17:18:59.415008+0000","last_became_peered":"2024-08-17T17:18:59.415008+0000","last_unstale":"2024-08-17T17:31:13.602441+0000","last_undegraded":"2024-08-17T17:31:13.602441+0000","last_fullsized":"2024-08-17T17:31:13.602441+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:22:38.076866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":161,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.599900+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:31:13.599900+0000","last_peered":"2024-08-17T17:31:13.599900+0000","last_clean":"2024-08-17T17:31:13.599900+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:31:13.599900+0000","last_undegraded":"2024-08-17T17:31:13.599900+0000","last_fullsized":"2024-08-17T17:31:13.599900+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"247'10286","reported_seq":10339,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:18.415539+0000","last_change":"2024-08-17T17:28:27.509856+0000","last_active":"2024-08-17T17:31:18.415539+0000","last_peered":"2024-08-17T17:31:18.415539+0000","last_clean":"2024-08-17T17:31:18.415539+0000","last_became_active":"2024-08-17T17:28:27.509524+0000","last_became_peered":"2024-08-17T17:28:27.509524+0000","last_unstale":"2024-08-17T17:31:18.415539+0000","last_undegraded":"2024-08-17T17:31:18.415539+0000","last_fullsized":"2024-08-17T17:31:18.415539+0000","mapping_epoch":226,"log_start":"232'2900","ondisk_log_start":"232'2900","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":7386,"log_dups_size":0,"ondisk_log_size":7386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:02:26.063259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246153216,"num_objects":3756,"num_object_clones":0,"num_object_copies":11268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10286,"num_write_kb":449344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":740,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.608050+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:31:13.608050+0000","last_peered":"2024-08-17T17:31:13.608050+0000","last_clean":"2024-08-17T17:31:13.608050+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:31:13.608050+0000","last_undegraded":"2024-08-17T17:31:13.608050+0000","last_fullsized":"2024-08-17T17:31:13.608050+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":348,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.605373+0000","last_change":"2024-08-17T17:19:57.515564+0000","last_active":"2024-08-17T17:31:13.605373+0000","last_peered":"2024-08-17T17:31:13.605373+0000","last_clean":"2024-08-17T17:31:13.605373+0000","last_became_active":"2024-08-17T17:19:57.515142+0000","last_became_peered":"2024-08-17T17:19:57.515142+0000","last_unstale":"2024-08-17T17:31:13.605373+0000","last_undegraded":"2024-08-17T17:31:13.605373+0000","last_fullsized":"2024-08-17T17:31:13.605373+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:59:33.506176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":378,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.605662+0000","last_change":"2024-08-17T17:24:06.587903+0000","last_active":"2024-08-17T17:31:13.605662+0000","last_peered":"2024-08-17T17:31:13.605662+0000","last_clean":"2024-08-17T17:31:13.605662+0000","last_became_active":"2024-08-17T17:24:06.587503+0000","last_became_peered":"2024-08-17T17:24:06.587503+0000","last_unstale":"2024-08-17T17:31:13.605662+0000","last_undegraded":"2024-08-17T17:31:13.605662+0000","last_fullsized":"2024-08-17T17:31:13.605662+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:05:58.591992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"247'10242","reported_seq":10271,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:19.019971+0000","last_change":"2024-08-17T17:28:27.511940+0000","last_active":"2024-08-17T17:31:19.019971+0000","last_peered":"2024-08-17T17:31:19.019971+0000","last_clean":"2024-08-17T17:31:19.019971+0000","last_became_active":"2024-08-17T17:28:27.511636+0000","last_became_peered":"2024-08-17T17:28:27.511636+0000","last_unstale":"2024-08-17T17:31:19.019971+0000","last_undegraded":"2024-08-17T17:31:19.019971+0000","last_fullsized":"2024-08-17T17:31:19.019971+0000","mapping_epoch":226,"log_start":"232'2500","ondisk_log_start":"232'2500","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":7742,"log_dups_size":0,"ondisk_log_size":7742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:51:38.705815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251789312,"num_objects":3842,"num_object_clones":0,"num_object_copies":11526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10242,"num_write_kb":450688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":1157,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.553643+0000","last_change":"2024-08-17T17:20:00.433870+0000","last_active":"2024-08-17T17:31:08.553643+0000","last_peered":"2024-08-17T17:31:08.553643+0000","last_clean":"2024-08-17T17:31:08.553643+0000","last_became_active":"2024-08-17T17:20:00.433386+0000","last_became_peered":"2024-08-17T17:20:00.433386+0000","last_unstale":"2024-08-17T17:31:08.553643+0000","last_undegraded":"2024-08-17T17:31:08.553643+0000","last_fullsized":"2024-08-17T17:31:08.553643+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:01:11.848862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1392,"num_read_kb":2863,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":274,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.598055+0000","last_change":"2024-08-17T17:20:49.272407+0000","last_active":"2024-08-17T17:31:13.598055+0000","last_peered":"2024-08-17T17:31:13.598055+0000","last_clean":"2024-08-17T17:31:13.598055+0000","last_became_active":"2024-08-17T17:20:49.271749+0000","last_became_peered":"2024-08-17T17:20:49.271749+0000","last_unstale":"2024-08-17T17:31:13.598055+0000","last_undegraded":"2024-08-17T17:31:13.598055+0000","last_fullsized":"2024-08-17T17:31:13.598055+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:36:01.443457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":345,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.605403+0000","last_change":"2024-08-17T17:17:32.499715+0000","last_active":"2024-08-17T17:31:13.605403+0000","last_peered":"2024-08-17T17:31:13.605403+0000","last_clean":"2024-08-17T17:31:13.605403+0000","last_became_active":"2024-08-17T17:17:32.499412+0000","last_became_peered":"2024-08-17T17:17:32.499412+0000","last_unstale":"2024-08-17T17:31:13.605403+0000","last_undegraded":"2024-08-17T17:31:13.605403+0000","last_fullsized":"2024-08-17T17:31:13.605403+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:39:38.456970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"247'10280","reported_seq":10333,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:17.474109+0000","last_change":"2024-08-17T17:28:27.510443+0000","last_active":"2024-08-17T17:31:17.474109+0000","last_peered":"2024-08-17T17:31:17.474109+0000","last_clean":"2024-08-17T17:31:17.474109+0000","last_became_active":"2024-08-17T17:28:27.510074+0000","last_became_peered":"2024-08-17T17:28:27.510074+0000","last_unstale":"2024-08-17T17:31:17.474109+0000","last_undegraded":"2024-08-17T17:31:17.474109+0000","last_fullsized":"2024-08-17T17:31:17.474109+0000","mapping_epoch":226,"log_start":"233'3400","ondisk_log_start":"233'3400","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":6880,"log_dups_size":0,"ondisk_log_size":6880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:09:30.294320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249954304,"num_objects":3814,"num_object_clones":0,"num_object_copies":11442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10280,"num_write_kb":451008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":222,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.553788+0000","last_change":"2024-08-17T17:17:32.495728+0000","last_active":"2024-08-17T17:31:08.553788+0000","last_peered":"2024-08-17T17:31:08.553788+0000","last_clean":"2024-08-17T17:31:08.553788+0000","last_became_active":"2024-08-17T17:17:32.495357+0000","last_became_peered":"2024-08-17T17:17:32.495357+0000","last_unstale":"2024-08-17T17:31:08.553788+0000","last_undegraded":"2024-08-17T17:31:08.553788+0000","last_fullsized":"2024-08-17T17:31:08.553788+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:27:45.561567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":581,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.605595+0000","last_change":"2024-08-17T17:16:26.964621+0000","last_active":"2024-08-17T17:31:13.605595+0000","last_peered":"2024-08-17T17:31:13.605595+0000","last_clean":"2024-08-17T17:31:13.605595+0000","last_became_active":"2024-08-17T17:16:26.964058+0000","last_became_peered":"2024-08-17T17:16:26.964058+0000","last_unstale":"2024-08-17T17:31:13.605595+0000","last_undegraded":"2024-08-17T17:31:13.605595+0000","last_fullsized":"2024-08-17T17:31:13.605595+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:00:04.465644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":351,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.623105+0000","last_change":"2024-08-17T17:24:53.465550+0000","last_active":"2024-08-17T17:31:13.623105+0000","last_peered":"2024-08-17T17:31:13.623105+0000","last_clean":"2024-08-17T17:31:13.623105+0000","last_became_active":"2024-08-17T17:24:53.465228+0000","last_became_peered":"2024-08-17T17:24:53.465228+0000","last_unstale":"2024-08-17T17:31:13.623105+0000","last_undegraded":"2024-08-17T17:31:13.623105+0000","last_fullsized":"2024-08-17T17:31:13.623105+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:21:18.378359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"247'10377","reported_seq":10406,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:18.911929+0000","last_change":"2024-08-17T17:28:27.509315+0000","last_active":"2024-08-17T17:31:18.911929+0000","last_peered":"2024-08-17T17:31:18.911929+0000","last_clean":"2024-08-17T17:31:18.911929+0000","last_became_active":"2024-08-17T17:28:27.509018+0000","last_became_peered":"2024-08-17T17:28:27.509018+0000","last_unstale":"2024-08-17T17:31:18.911929+0000","last_undegraded":"2024-08-17T17:31:18.911929+0000","last_fullsized":"2024-08-17T17:31:18.911929+0000","mapping_epoch":226,"log_start":"232'300","ondisk_log_start":"232'300","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:26:22.379668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246480896,"num_objects":3761,"num_object_clones":0,"num_object_copies":11283,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10377,"num_write_kb":452416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":376,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.605495+0000","last_change":"2024-08-17T17:24:53.464307+0000","last_active":"2024-08-17T17:31:13.605495+0000","last_peered":"2024-08-17T17:31:13.605495+0000","last_clean":"2024-08-17T17:31:13.605495+0000","last_became_active":"2024-08-17T17:24:53.463996+0000","last_became_peered":"2024-08-17T17:24:53.463996+0000","last_unstale":"2024-08-17T17:31:13.605495+0000","last_undegraded":"2024-08-17T17:31:13.605495+0000","last_fullsized":"2024-08-17T17:31:13.605495+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:04:18.993725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":674,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.555436+0000","last_change":"2024-08-17T17:24:09.599822+0000","last_active":"2024-08-17T17:31:08.555436+0000","last_peered":"2024-08-17T17:31:08.555436+0000","last_clean":"2024-08-17T17:31:08.555436+0000","last_became_active":"2024-08-17T17:24:09.599189+0000","last_became_peered":"2024-08-17T17:24:09.599189+0000","last_unstale":"2024-08-17T17:31:08.555436+0000","last_undegraded":"2024-08-17T17:31:08.555436+0000","last_fullsized":"2024-08-17T17:31:08.555436+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:04.465604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":335,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.598099+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:31:13.598099+0000","last_peered":"2024-08-17T17:31:13.598099+0000","last_clean":"2024-08-17T17:31:13.598099+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:31:13.598099+0000","last_undegraded":"2024-08-17T17:31:13.598099+0000","last_fullsized":"2024-08-17T17:31:13.598099+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"247'10573","reported_seq":10627,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:17.462943+0000","last_change":"2024-08-17T17:28:27.509838+0000","last_active":"2024-08-17T17:31:17.462943+0000","last_peered":"2024-08-17T17:31:17.462943+0000","last_clean":"2024-08-17T17:31:17.462943+0000","last_became_active":"2024-08-17T17:28:27.509400+0000","last_became_peered":"2024-08-17T17:28:27.509400+0000","last_unstale":"2024-08-17T17:31:17.462943+0000","last_undegraded":"2024-08-17T17:31:17.462943+0000","last_fullsized":"2024-08-17T17:31:17.462943+0000","mapping_epoch":226,"log_start":"233'3700","ondisk_log_start":"233'3700","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":6873,"log_dups_size":0,"ondisk_log_size":6873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:57:28.933136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260767744,"num_objects":3979,"num_object_clones":0,"num_object_copies":11937,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10573,"num_write_kb":465664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":370,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.616965+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:31:13.616965+0000","last_peered":"2024-08-17T17:31:13.616965+0000","last_clean":"2024-08-17T17:31:13.616965+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:31:13.616965+0000","last_undegraded":"2024-08-17T17:31:13.616965+0000","last_fullsized":"2024-08-17T17:31:13.616965+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":889,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.606095+0000","last_change":"2024-08-17T17:23:19.633786+0000","last_active":"2024-08-17T17:31:13.606095+0000","last_peered":"2024-08-17T17:31:13.606095+0000","last_clean":"2024-08-17T17:31:13.606095+0000","last_became_active":"2024-08-17T17:23:19.633230+0000","last_became_peered":"2024-08-17T17:23:19.633230+0000","last_unstale":"2024-08-17T17:31:13.606095+0000","last_undegraded":"2024-08-17T17:31:13.606095+0000","last_fullsized":"2024-08-17T17:31:13.606095+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:30:50.377391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":366,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.616706+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:31:13.616706+0000","last_peered":"2024-08-17T17:31:13.616706+0000","last_clean":"2024-08-17T17:31:13.616706+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:31:13.616706+0000","last_undegraded":"2024-08-17T17:31:13.616706+0000","last_fullsized":"2024-08-17T17:31:13.616706+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"247'10081","reported_seq":10135,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:17.457945+0000","last_change":"2024-08-17T17:28:27.509972+0000","last_active":"2024-08-17T17:31:17.457945+0000","last_peered":"2024-08-17T17:31:17.457945+0000","last_clean":"2024-08-17T17:31:17.457945+0000","last_became_active":"2024-08-17T17:28:27.509668+0000","last_became_peered":"2024-08-17T17:28:27.509668+0000","last_unstale":"2024-08-17T17:31:17.457945+0000","last_undegraded":"2024-08-17T17:31:17.457945+0000","last_fullsized":"2024-08-17T17:31:17.457945+0000","mapping_epoch":226,"log_start":"233'3200","ondisk_log_start":"233'3200","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":6881,"log_dups_size":0,"ondisk_log_size":6881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:54:43.175100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246611968,"num_objects":3763,"num_object_clones":0,"num_object_copies":11289,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10081,"num_write_kb":443008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":165,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.599800+0000","last_change":"2024-08-17T17:24:06.593660+0000","last_active":"2024-08-17T17:31:13.599800+0000","last_peered":"2024-08-17T17:31:13.599800+0000","last_clean":"2024-08-17T17:31:13.599800+0000","last_became_active":"2024-08-17T17:24:06.586215+0000","last_became_peered":"2024-08-17T17:24:06.586215+0000","last_unstale":"2024-08-17T17:31:13.599800+0000","last_undegraded":"2024-08-17T17:31:13.599800+0000","last_fullsized":"2024-08-17T17:31:13.599800+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:15:49.475395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":478,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.560155+0000","last_change":"2024-08-17T17:26:06.855307+0000","last_active":"2024-08-17T17:31:08.560155+0000","last_peered":"2024-08-17T17:31:08.560155+0000","last_clean":"2024-08-17T17:31:08.560155+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:31:08.560155+0000","last_undegraded":"2024-08-17T17:31:08.560155+0000","last_fullsized":"2024-08-17T17:31:08.560155+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":146,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.553391+0000","last_change":"2024-08-17T17:18:59.417183+0000","last_active":"2024-08-17T17:31:08.553391+0000","last_peered":"2024-08-17T17:31:08.553391+0000","last_clean":"2024-08-17T17:31:08.553391+0000","last_became_active":"2024-08-17T17:18:59.416843+0000","last_became_peered":"2024-08-17T17:18:59.416843+0000","last_unstale":"2024-08-17T17:31:08.553391+0000","last_undegraded":"2024-08-17T17:31:08.553391+0000","last_fullsized":"2024-08-17T17:31:08.553391+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:29:09.336335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":167,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.561709+0000","last_change":"2024-08-17T17:20:49.258048+0000","last_active":"2024-08-17T17:31:08.561709+0000","last_peered":"2024-08-17T17:31:08.561709+0000","last_clean":"2024-08-17T17:31:08.561709+0000","last_became_active":"2024-08-17T17:20:49.257672+0000","last_became_peered":"2024-08-17T17:20:49.257672+0000","last_unstale":"2024-08-17T17:31:08.561709+0000","last_undegraded":"2024-08-17T17:31:08.561709+0000","last_fullsized":"2024-08-17T17:31:08.561709+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:08:11.734316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":206,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.555165+0000","last_change":"2024-08-17T17:18:59.430969+0000","last_active":"2024-08-17T17:31:08.555165+0000","last_peered":"2024-08-17T17:31:08.555165+0000","last_clean":"2024-08-17T17:31:08.555165+0000","last_became_active":"2024-08-17T17:18:59.413475+0000","last_became_peered":"2024-08-17T17:18:59.413475+0000","last_unstale":"2024-08-17T17:31:08.555165+0000","last_undegraded":"2024-08-17T17:31:08.555165+0000","last_fullsized":"2024-08-17T17:31:08.555165+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:36:33.732645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":361,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.622830+0000","last_change":"2024-08-17T17:25:04.506413+0000","last_active":"2024-08-17T17:31:13.622830+0000","last_peered":"2024-08-17T17:31:13.622830+0000","last_clean":"2024-08-17T17:31:13.622830+0000","last_became_active":"2024-08-17T17:25:04.506104+0000","last_became_peered":"2024-08-17T17:25:04.506104+0000","last_unstale":"2024-08-17T17:31:13.622830+0000","last_undegraded":"2024-08-17T17:31:13.622830+0000","last_fullsized":"2024-08-17T17:31:13.622830+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:41:53.556223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":149,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.555239+0000","last_change":"2024-08-17T17:20:49.279458+0000","last_active":"2024-08-17T17:31:08.555239+0000","last_peered":"2024-08-17T17:31:08.555239+0000","last_clean":"2024-08-17T17:31:08.555239+0000","last_became_active":"2024-08-17T17:20:49.260790+0000","last_became_peered":"2024-08-17T17:20:49.260790+0000","last_unstale":"2024-08-17T17:31:08.555239+0000","last_undegraded":"2024-08-17T17:31:08.555239+0000","last_fullsized":"2024-08-17T17:31:08.555239+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:58.092945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":63,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.553754+0000","last_change":"2024-08-17T17:24:53.463713+0000","last_active":"2024-08-17T17:31:08.553754+0000","last_peered":"2024-08-17T17:31:08.553754+0000","last_clean":"2024-08-17T17:31:08.553754+0000","last_became_active":"2024-08-17T17:24:53.463419+0000","last_became_peered":"2024-08-17T17:24:53.463419+0000","last_unstale":"2024-08-17T17:31:08.553754+0000","last_undegraded":"2024-08-17T17:31:08.553754+0000","last_fullsized":"2024-08-17T17:31:08.553754+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:34:44.545175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":316,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.607802+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:31:13.607802+0000","last_peered":"2024-08-17T17:31:13.607802+0000","last_clean":"2024-08-17T17:31:13.607802+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:31:13.607802+0000","last_undegraded":"2024-08-17T17:31:13.607802+0000","last_fullsized":"2024-08-17T17:31:13.607802+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":344,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.600291+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:31:13.600291+0000","last_peered":"2024-08-17T17:31:13.600291+0000","last_clean":"2024-08-17T17:31:13.600291+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:31:13.600291+0000","last_undegraded":"2024-08-17T17:31:13.600291+0000","last_fullsized":"2024-08-17T17:31:13.600291+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":363,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.608237+0000","last_change":"2024-08-17T17:25:04.509422+0000","last_active":"2024-08-17T17:31:13.608237+0000","last_peered":"2024-08-17T17:31:13.608237+0000","last_clean":"2024-08-17T17:31:13.608237+0000","last_became_active":"2024-08-17T17:25:04.508742+0000","last_became_peered":"2024-08-17T17:25:04.508742+0000","last_unstale":"2024-08-17T17:31:13.608237+0000","last_undegraded":"2024-08-17T17:31:13.608237+0000","last_fullsized":"2024-08-17T17:31:13.608237+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:18:31.976907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":204,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.560129+0000","last_change":"2024-08-17T17:16:26.963696+0000","last_active":"2024-08-17T17:31:08.560129+0000","last_peered":"2024-08-17T17:31:08.560129+0000","last_clean":"2024-08-17T17:31:08.560129+0000","last_became_active":"2024-08-17T17:16:26.963280+0000","last_became_peered":"2024-08-17T17:16:26.963280+0000","last_unstale":"2024-08-17T17:31:08.560129+0000","last_undegraded":"2024-08-17T17:31:08.560129+0000","last_fullsized":"2024-08-17T17:31:08.560129+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:22:01.474840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":381,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.616809+0000","last_change":"2024-08-17T17:24:06.586886+0000","last_active":"2024-08-17T17:31:13.616809+0000","last_peered":"2024-08-17T17:31:13.616809+0000","last_clean":"2024-08-17T17:31:13.616809+0000","last_became_active":"2024-08-17T17:24:06.586516+0000","last_became_peered":"2024-08-17T17:24:06.586516+0000","last_unstale":"2024-08-17T17:31:13.616809+0000","last_undegraded":"2024-08-17T17:31:13.616809+0000","last_fullsized":"2024-08-17T17:31:13.616809+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:03:30.002726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":161,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.599638+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:31:13.599638+0000","last_peered":"2024-08-17T17:31:13.599638+0000","last_clean":"2024-08-17T17:31:13.599638+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:31:13.599638+0000","last_undegraded":"2024-08-17T17:31:13.599638+0000","last_fullsized":"2024-08-17T17:31:13.599638+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":341,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.631882+0000","last_change":"2024-08-17T17:18:59.412463+0000","last_active":"2024-08-17T17:31:13.631882+0000","last_peered":"2024-08-17T17:31:13.631882+0000","last_clean":"2024-08-17T17:31:13.631882+0000","last_became_active":"2024-08-17T17:18:59.412134+0000","last_became_peered":"2024-08-17T17:18:59.412134+0000","last_unstale":"2024-08-17T17:31:13.631882+0000","last_undegraded":"2024-08-17T17:31:13.631882+0000","last_fullsized":"2024-08-17T17:31:13.631882+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:44.463153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":26,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.553535+0000","last_change":"2024-08-17T17:28:41.399059+0000","last_active":"2024-08-17T17:31:08.553535+0000","last_peered":"2024-08-17T17:31:08.553535+0000","last_clean":"2024-08-17T17:31:08.553535+0000","last_became_active":"2024-08-17T17:28:41.398560+0000","last_became_peered":"2024-08-17T17:28:41.398560+0000","last_unstale":"2024-08-17T17:31:08.553535+0000","last_undegraded":"2024-08-17T17:31:08.553535+0000","last_fullsized":"2024-08-17T17:31:08.553535+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:31:45.097451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":281,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.623203+0000","last_change":"2024-08-17T17:24:06.591165+0000","last_active":"2024-08-17T17:31:13.623203+0000","last_peered":"2024-08-17T17:31:13.623203+0000","last_clean":"2024-08-17T17:31:13.623203+0000","last_became_active":"2024-08-17T17:24:06.590785+0000","last_became_peered":"2024-08-17T17:24:06.590785+0000","last_unstale":"2024-08-17T17:31:13.623203+0000","last_undegraded":"2024-08-17T17:31:13.623203+0000","last_fullsized":"2024-08-17T17:31:13.623203+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:54:05.775563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":353,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.605782+0000","last_change":"2024-08-17T17:24:32.300560+0000","last_active":"2024-08-17T17:31:13.605782+0000","last_peered":"2024-08-17T17:31:13.605782+0000","last_clean":"2024-08-17T17:31:13.605782+0000","last_became_active":"2024-08-17T17:24:32.300228+0000","last_became_peered":"2024-08-17T17:24:32.300228+0000","last_unstale":"2024-08-17T17:31:13.605782+0000","last_undegraded":"2024-08-17T17:31:13.605782+0000","last_fullsized":"2024-08-17T17:31:13.605782+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:18:05.475449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":160,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.597921+0000","last_change":"2024-08-17T17:24:06.586379+0000","last_active":"2024-08-17T17:31:13.597921+0000","last_peered":"2024-08-17T17:31:13.597921+0000","last_clean":"2024-08-17T17:31:13.597921+0000","last_became_active":"2024-08-17T17:24:06.586023+0000","last_became_peered":"2024-08-17T17:24:06.586023+0000","last_unstale":"2024-08-17T17:31:13.597921+0000","last_undegraded":"2024-08-17T17:31:13.597921+0000","last_fullsized":"2024-08-17T17:31:13.597921+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:45.701637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":370,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.612380+0000","last_change":"2024-08-17T17:21:50.442535+0000","last_active":"2024-08-17T17:31:13.612380+0000","last_peered":"2024-08-17T17:31:13.612380+0000","last_clean":"2024-08-17T17:31:13.612380+0000","last_became_active":"2024-08-17T17:21:50.441891+0000","last_became_peered":"2024-08-17T17:21:50.441891+0000","last_unstale":"2024-08-17T17:31:13.612380+0000","last_undegraded":"2024-08-17T17:31:13.612380+0000","last_fullsized":"2024-08-17T17:31:13.612380+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:46:24.025407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":259,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.553417+0000","last_change":"2024-08-17T17:25:04.508045+0000","last_active":"2024-08-17T17:31:08.553417+0000","last_peered":"2024-08-17T17:31:08.553417+0000","last_clean":"2024-08-17T17:31:08.553417+0000","last_became_active":"2024-08-17T17:25:04.505083+0000","last_became_peered":"2024-08-17T17:25:04.505083+0000","last_unstale":"2024-08-17T17:31:08.553417+0000","last_undegraded":"2024-08-17T17:31:08.553417+0000","last_fullsized":"2024-08-17T17:31:08.553417+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:23:42.993544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":356,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.606388+0000","last_change":"2024-08-17T17:18:59.418434+0000","last_active":"2024-08-17T17:31:13.606388+0000","last_peered":"2024-08-17T17:31:13.606388+0000","last_clean":"2024-08-17T17:31:13.606388+0000","last_became_active":"2024-08-17T17:18:59.418043+0000","last_became_peered":"2024-08-17T17:18:59.418043+0000","last_unstale":"2024-08-17T17:31:13.606388+0000","last_undegraded":"2024-08-17T17:31:13.606388+0000","last_fullsized":"2024-08-17T17:31:13.606388+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:25:51.371820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":232,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.553713+0000","last_change":"2024-08-17T17:20:49.291167+0000","last_active":"2024-08-17T17:31:08.553713+0000","last_peered":"2024-08-17T17:31:08.553713+0000","last_clean":"2024-08-17T17:31:08.553713+0000","last_became_active":"2024-08-17T17:20:49.263750+0000","last_became_peered":"2024-08-17T17:20:49.263750+0000","last_unstale":"2024-08-17T17:31:08.553713+0000","last_undegraded":"2024-08-17T17:31:08.553713+0000","last_fullsized":"2024-08-17T17:31:08.553713+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:02.863246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":370,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.608137+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:31:13.608137+0000","last_peered":"2024-08-17T17:31:13.608137+0000","last_clean":"2024-08-17T17:31:13.608137+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:31:13.608137+0000","last_undegraded":"2024-08-17T17:31:13.608137+0000","last_fullsized":"2024-08-17T17:31:13.608137+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":370,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.606582+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:31:13.606582+0000","last_peered":"2024-08-17T17:31:13.606582+0000","last_clean":"2024-08-17T17:31:13.606582+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:31:13.606582+0000","last_undegraded":"2024-08-17T17:31:13.606582+0000","last_fullsized":"2024-08-17T17:31:13.606582+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":59,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.559345+0000","last_change":"2024-08-17T17:25:04.503834+0000","last_active":"2024-08-17T17:31:08.559345+0000","last_peered":"2024-08-17T17:31:08.559345+0000","last_clean":"2024-08-17T17:31:08.559345+0000","last_became_active":"2024-08-17T17:25:04.502712+0000","last_became_peered":"2024-08-17T17:25:04.502712+0000","last_unstale":"2024-08-17T17:31:08.559345+0000","last_undegraded":"2024-08-17T17:31:08.559345+0000","last_fullsized":"2024-08-17T17:31:08.559345+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:45.389103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":180,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.600190+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:31:13.600190+0000","last_peered":"2024-08-17T17:31:13.600190+0000","last_clean":"2024-08-17T17:31:13.600190+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:31:13.600190+0000","last_undegraded":"2024-08-17T17:31:13.600190+0000","last_fullsized":"2024-08-17T17:31:13.600190+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3991732248,"num_objects":60910,"num_object_clones":0,"num_object_copies":182730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60910,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165530,"num_write_kb":7246017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11985629184,"data_stored":11985616968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122130,"ondisk_log_size":122130,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":6304,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":16702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17971,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234294,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1785260,"kb_used_data":1483748,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91959892,"statfs":{"total":95995035648,"available":94166929408,"internally_reserved":0,"allocated":1519357952,"data_stored":1517972794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495929,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2034408,"kb_used_data":1724576,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91710744,"statfs":{"total":95995035648,"available":93911801856,"internally_reserved":0,"allocated":1765965824,"data_stored":1764584486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757566,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1276248,"kb_used_data":966416,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":92468904,"statfs":{"total":95995035648,"available":94688157696,"internally_reserved":0,"allocated":989609984,"data_stored":988240461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019204,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435431,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2935252,"kb_used_data":2478988,"kb_used_omap":4,"kb_used_meta":456251,"kb_avail":90809900,"statfs":{"total":95995035648,"available":92989337600,"internally_reserved":0,"allocated":2538483712,"data_stored":2537118059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":467201034},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":104,"seq":446676598958,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1873636,"kb_used_data":1439388,"kb_used_omap":7,"kb_used_meta":434168,"kb_avail":91871516,"statfs":{"total":95995035648,"available":94076432384,"internally_reserved":0,"allocated":1473933312,"data_stored":1472556541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":444588194},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762484,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2231904,"kb_used_data":1770644,"kb_used_omap":7,"kb_used_meta":461240,"kb_avail":91513248,"statfs":{"total":95995035648,"available":93709565952,"internally_reserved":0,"allocated":1813139456,"data_stored":1811765602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7945,"internal_metadata":472310007},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926011,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2361316,"kb_used_data":1951068,"kb_used_omap":7,"kb_used_meta":410232,"kb_avail":91383836,"statfs":{"total":95995035648,"available":93577048064,"internally_reserved":0,"allocated":1997893632,"data_stored":1996512771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":420077889},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,2,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6241,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1989414912,"data_stored":1989410840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1789267968,"data_stored":1789263896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1464336384,"data_stored":1464336384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2529099776,"data_stored":2529099776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":979107840,"data_stored":979107840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1739198464,"data_stored":1739194392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1495203840,"data_stored":1495203840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:31:22.634 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:31:22.675 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:22.699 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:22.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:22.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:22.801 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:22.825 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:22.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:22.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:22.901 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:22 smithi003 ceph-mon[100615]: from='client.55393 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:31:22.901 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:22 smithi003 ceph-mon[103344]: from='client.55393 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:31:22.927 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:22.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:22 smithi053 ceph-mon[89963]: from='client.55393 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:31:22.989 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:22.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:22.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:23.091 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:23.118 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:23.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:23.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:23.220 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:23.246 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:23.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:23.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:23.287 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-17T17:31:23.348 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:23.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:23.404 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:23.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:23.421 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:31:23.421 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:31:23.505 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:23.530 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:31:23.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:23.583 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:23.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:23.684 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:23.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:23.716 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:23.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:23.817 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:23.857 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:23 smithi003 ceph-mon[103344]: pgmap v722: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 746 op/s 2024-08-17T17:31:23.857 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:31:23.857 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:31:23.857 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:23 smithi003 ceph-mon[100615]: pgmap v722: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 746 op/s 2024-08-17T17:31:23.857 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:31:23.858 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:31:23.867 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:23.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:23.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:23.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:23 smithi053 ceph-mon[89963]: pgmap v722: 89 pgs: 89 active+clean; 3.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 746 op/s 2024-08-17T17:31:23.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:31:23.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:31:23.969 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:24.025 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:24.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:24.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:24.127 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:24.139 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:31:24.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:24.169 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:24.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:24.270 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:24.308 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:24.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:24.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:24.410 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:24.440 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:24.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:24.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:24.542 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:24.570 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:24.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:24.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:24.672 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:24.701 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:24.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:24.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:24.803 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:24.847 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:24.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:24.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:24.949 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:24.973 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:24.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:24.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:25.075 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:25.101 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:25.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:25.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:25.203 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:25.228 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:25.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:25.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:25.330 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:25.356 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:25.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:25.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:25.458 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:25.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:25.498 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:25.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:25.598 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:25.622 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:25.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:25.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:25.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:25 smithi053 ceph-mon[89963]: pgmap v723: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 734 op/s 2024-08-17T17:31:25.724 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:25.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:25.755 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:25.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:25.855 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:25.893 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:25.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:25.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:25.910 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:25 smithi003 ceph-mon[103344]: pgmap v723: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 734 op/s 2024-08-17T17:31:25.910 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:25 smithi003 ceph-mon[100615]: pgmap v723: 89 pgs: 89 active+clean; 3.6 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 734 op/s 2024-08-17T17:31:25.910 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:31:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:31:25] "GET /metrics HTTP/1.1" 200 36504 "" "Prometheus/2.51.0" 2024-08-17T17:31:25.995 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:26.022 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:26.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:26.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:26.124 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:26.165 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:26.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:26.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:26.267 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:26.304 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:26.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:26.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:26.346 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:31:26.346 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2024-08-17T17:31:26.347 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2024-08-17T17:31:26.347 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2024-08-17T17:31:26.347 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.347 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:153566 5.2 5:4f012f1e:::benchmark_data_smithi003_140793_object51509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-08-17T17:31:26.347 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:22:02.733431+0000", 2024-08-17T17:31:26.347 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 563.60943959099995, 2024-08-17T17:31:26.347 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.11735413, 2024-08-17T17:31:26.347 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:26.347 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:26.347 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:26.347 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:31:26.348 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:31:26.348 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 153566 2024-08-17T17:31:26.348 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.348 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:26.348 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.348 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:26.348 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.733431+0000", 2024-08-17T17:31:26.348 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.348 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.348 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.348 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:26.348 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.733431+0000", 2024-08-17T17:31:26.349 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.349 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.349 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.349 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:26.349 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.733430+0000", 2024-08-17T17:31:26.349 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:31:26.349 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.349 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.349 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:26.349 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.733436+0000", 2024-08-17T17:31:26.349 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4790000000000001e-06 2024-08-17T17:31:26.349 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.350 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.350 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:26.350 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.733453+0000", 2024-08-17T17:31:26.350 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6509000000000001e-05 2024-08-17T17:31:26.350 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.350 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.350 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:26.350 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.733459+0000", 2024-08-17T17:31:26.350 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0129999999999999e-06 2024-08-17T17:31:26.350 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.350 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.350 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:26.351 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.733505+0000", 2024-08-17T17:31:26.351 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6545999999999998e-05 2024-08-17T17:31:26.351 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.351 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.351 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:26.351 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.733633+0000", 2024-08-17T17:31:26.351 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000127322 2024-08-17T17:31:26.351 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.351 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.351 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:31:26.351 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.733753+0000", 2024-08-17T17:31:26.351 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000119916 2024-08-17T17:31:26.352 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.352 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.352 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.352 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.738221+0000", 2024-08-17T17:31:26.352 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0044680839999999998 2024-08-17T17:31:26.352 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.352 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.352 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:26.352 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.738249+0000", 2024-08-17T17:31:26.352 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8747000000000001e-05 2024-08-17T17:31:26.352 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.352 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.353 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.353 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.850691+0000", 2024-08-17T17:31:26.353 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.11244172199999999 2024-08-17T17:31:26.353 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.353 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.353 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:26.353 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.850745+0000", 2024-08-17T17:31:26.353 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4135000000000003e-05 2024-08-17T17:31:26.353 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.353 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.353 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:26.353 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.850785+0000", 2024-08-17T17:31:26.354 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9805000000000003e-05 2024-08-17T17:31:26.354 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.354 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:26.354 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.354 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.354 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.354 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.35277.0:153570 5.2 5:4f5e89a3:::benchmark_data_smithi003_140793_object51513:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-08-17T17:31:26.354 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:22:02.743390+0000", 2024-08-17T17:31:26.354 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 563.59948089299996, 2024-08-17T17:31:26.355 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.1119482, 2024-08-17T17:31:26.355 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:26.355 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:26.355 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:26.355 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.35277", 2024-08-17T17:31:26.355 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/3488615290", 2024-08-17T17:31:26.355 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 153570 2024-08-17T17:31:26.355 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.355 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:26.355 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.355 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:26.355 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.743390+0000", 2024-08-17T17:31:26.356 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.356 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.356 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.356 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:26.356 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.743390+0000", 2024-08-17T17:31:26.356 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.356 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.356 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.356 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:26.356 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.743389+0000", 2024-08-17T17:31:26.356 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:31:26.356 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.357 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.357 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:26.357 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.743394+0000", 2024-08-17T17:31:26.357 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2900000000000002e-06 2024-08-17T17:31:26.357 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.357 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.357 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:26.357 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.743406+0000", 2024-08-17T17:31:26.357 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1779e-05 2024-08-17T17:31:26.357 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.357 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.357 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:26.357 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.743411+0000", 2024-08-17T17:31:26.358 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9729999999999997e-06 2024-08-17T17:31:26.358 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.358 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.358 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:26.358 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.743464+0000", 2024-08-17T17:31:26.358 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3025000000000001e-05 2024-08-17T17:31:26.358 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.358 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.358 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:26.358 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.743600+0000", 2024-08-17T17:31:26.358 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013659699999999999 2024-08-17T17:31:26.359 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.359 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.359 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:31:26.359 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.743744+0000", 2024-08-17T17:31:26.359 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014353900000000001 2024-08-17T17:31:26.359 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.359 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.359 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.359 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.748190+0000", 2024-08-17T17:31:26.359 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0044460039999999999 2024-08-17T17:31:26.359 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.359 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.360 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:26.360 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.748520+0000", 2024-08-17T17:31:26.360 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00033048799999999998 2024-08-17T17:31:26.360 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.360 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.360 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.360 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.855260+0000", 2024-08-17T17:31:26.360 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.106739537 2024-08-17T17:31:26.360 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.360 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.360 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:26.361 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.855303+0000", 2024-08-17T17:31:26.361 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3640000000000002e-05 2024-08-17T17:31:26.361 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.361 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.361 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:26.361 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:22:02.855338+0000", 2024-08-17T17:31:26.361 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4527999999999999e-05 2024-08-17T17:31:26.361 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.361 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:26.361 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.361 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.361 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.362 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.45491.0:131827 6.6 e212/207 6:68c53fdf:::benchmark_data_smithi003_175043_object33178:head v 212'8072, mlcod=212'8070)", 2024-08-17T17:31:26.362 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:26:42.170765+0000", 2024-08-17T17:31:26.362 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 284.17210512700001, 2024-08-17T17:31:26.362 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.10206145599999999, 2024-08-17T17:31:26.362 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:26.362 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:26.362 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:26.362 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.362 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:26.362 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.170765+0000", 2024-08-17T17:31:26.362 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.363 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.363 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.363 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:26.363 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.170765+0000", 2024-08-17T17:31:26.363 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.363 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.363 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.363 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:26.363 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.170765+0000", 2024-08-17T17:31:26.363 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:31:26.363 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.363 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.364 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:26.364 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.170770+0000", 2024-08-17T17:31:26.364 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.8019999999999999e-06 2024-08-17T17:31:26.364 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.364 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.364 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:26.364 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.265511+0000", 2024-08-17T17:31:26.364 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.094741685000000006 2024-08-17T17:31:26.364 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.364 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.365 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:26.365 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.265531+0000", 2024-08-17T17:31:26.365 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9959e-05 2024-08-17T17:31:26.365 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.365 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.365 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:26.365 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.265763+0000", 2024-08-17T17:31:26.365 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00023125899999999999 2024-08-17T17:31:26.365 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.365 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.365 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:26.366 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.265948+0000", 2024-08-17T17:31:26.366 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000185105 2024-08-17T17:31:26.366 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.366 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.366 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:26.366 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.272764+0000", 2024-08-17T17:31:26.366 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0068160190000000004 2024-08-17T17:31:26.366 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.366 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.366 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:26.366 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.272827+0000", 2024-08-17T17:31:26.366 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3184999999999999e-05 2024-08-17T17:31:26.367 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.367 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:26.367 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.367 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.367 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.367 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.45491.0:131829 6.1 e212/178 6:82d98170:::benchmark_data_smithi003_175043_object33180:head v 212'8326, mlcod=212'8324)", 2024-08-17T17:31:26.367 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:26:42.171012+0000", 2024-08-17T17:31:26.367 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 284.17185830900002, 2024-08-17T17:31:26.367 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.10184240999999999, 2024-08-17T17:31:26.367 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:26.367 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:26.368 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:26.368 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.368 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:26.368 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.171012+0000", 2024-08-17T17:31:26.368 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.368 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.368 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.368 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:26.368 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.171012+0000", 2024-08-17T17:31:26.368 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.368 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.368 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.368 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:26.369 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.171012+0000", 2024-08-17T17:31:26.369 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:31:26.369 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.369 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.369 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:26.369 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.171017+0000", 2024-08-17T17:31:26.369 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8599999999999998e-06 2024-08-17T17:31:26.369 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.369 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.369 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:26.369 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.265874+0000", 2024-08-17T17:31:26.369 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.094856313999999997 2024-08-17T17:31:26.369 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.370 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.370 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:26.370 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.265883+0000", 2024-08-17T17:31:26.370 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.0999999999999993e-06 2024-08-17T17:31:26.370 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.370 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.370 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:26.370 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.265966+0000", 2024-08-17T17:31:26.370 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.2618000000000001e-05 2024-08-17T17:31:26.370 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.370 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.371 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:26.371 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.265987+0000", 2024-08-17T17:31:26.371 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1098999999999999e-05 2024-08-17T17:31:26.371 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.371 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.371 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:26.371 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.272820+0000", 2024-08-17T17:31:26.371 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0068330480000000004 2024-08-17T17:31:26.371 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.371 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.371 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:26.371 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:42.272855+0000", 2024-08-17T17:31:26.371 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5049999999999998e-05 2024-08-17T17:31:26.372 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.372 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:26.372 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.372 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.372 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.372 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45767.0:21209 7.c 7:30ddbba9:::benchmark_data_smithi003_201798_object21208:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-08-17T17:31:26.372 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:28:49.798725+0000", 2024-08-17T17:31:26.372 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 156.54414580900001, 2024-08-17T17:31:26.372 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.12786388300000001, 2024-08-17T17:31:26.372 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:26.372 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:26.373 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:26.373 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45767", 2024-08-17T17:31:26.373 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:31:26.373 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 21209 2024-08-17T17:31:26.373 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.373 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:26.373 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.373 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:26.373 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:49.798725+0000", 2024-08-17T17:31:26.373 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.373 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.373 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.374 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:26.374 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:49.798725+0000", 2024-08-17T17:31:26.374 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.374 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.374 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.374 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:26.374 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:49.798724+0000", 2024-08-17T17:31:26.374 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:31:26.374 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.374 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.374 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:26.375 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:49.798812+0000", 2024-08-17T17:31:26.375 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.7489000000000002e-05 2024-08-17T17:31:26.375 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.375 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.375 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:26.375 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:49.798822+0000", 2024-08-17T17:31:26.375 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0256e-05 2024-08-17T17:31:26.375 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.375 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.375 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:26.376 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:49.798827+0000", 2024-08-17T17:31:26.376 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.8779999999999997e-06 2024-08-17T17:31:26.376 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.376 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.376 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:26.376 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:49.798873+0000", 2024-08-17T17:31:26.376 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6090000000000001e-05 2024-08-17T17:31:26.376 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.376 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.376 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:26.376 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:49.799010+0000", 2024-08-17T17:31:26.376 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013741099999999999 2024-08-17T17:31:26.377 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.377 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.377 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,7", 2024-08-17T17:31:26.377 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:49.799100+0000", 2024-08-17T17:31:26.377 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9709000000000005e-05 2024-08-17T17:31:26.377 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.377 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.377 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:26.377 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:49.800096+0000", 2024-08-17T17:31:26.377 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00099590000000000008 2024-08-17T17:31:26.377 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.377 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.377 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.378 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:49.800251+0000", 2024-08-17T17:31:26.378 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000155057 2024-08-17T17:31:26.378 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.378 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.378 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.378 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:49.926522+0000", 2024-08-17T17:31:26.378 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.12627084399999999 2024-08-17T17:31:26.378 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.378 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.378 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:26.378 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:49.926556+0000", 2024-08-17T17:31:26.378 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4239000000000002e-05 2024-08-17T17:31:26.379 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.379 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.379 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:26.379 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:49.926589+0000", 2024-08-17T17:31:26.379 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2747999999999999e-05 2024-08-17T17:31:26.379 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.380 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:26.380 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.380 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.380 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.380 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45767.0:38882 7.7 7:e3344944:::benchmark_data_smithi003_201798_object38881:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-08-17T17:31:26.380 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:28:59.695232+0000", 2024-08-17T17:31:26.380 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 146.64763856499999, 2024-08-17T17:31:26.380 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.118118741, 2024-08-17T17:31:26.380 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:26.381 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:26.381 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:26.381 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45767", 2024-08-17T17:31:26.381 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:31:26.381 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 38882 2024-08-17T17:31:26.381 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.381 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:26.381 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.381 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:26.381 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.695232+0000", 2024-08-17T17:31:26.381 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.381 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.382 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.382 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:26.382 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.695232+0000", 2024-08-17T17:31:26.382 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.382 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.382 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.382 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:26.382 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.695231+0000", 2024-08-17T17:31:26.382 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:31:26.382 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.382 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.383 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:26.383 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.695319+0000", 2024-08-17T17:31:26.383 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.8147999999999997e-05 2024-08-17T17:31:26.383 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.383 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.383 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:26.383 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.695331+0000", 2024-08-17T17:31:26.383 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2218000000000001e-05 2024-08-17T17:31:26.383 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.383 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.383 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:26.383 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.695336+0000", 2024-08-17T17:31:26.384 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3680000000000001e-06 2024-08-17T17:31:26.384 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.384 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.384 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:26.384 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.695376+0000", 2024-08-17T17:31:26.384 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0064000000000001e-05 2024-08-17T17:31:26.384 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.384 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.384 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:26.384 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.695484+0000", 2024-08-17T17:31:26.384 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010801799999999999 2024-08-17T17:31:26.384 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.385 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.385 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,3", 2024-08-17T17:31:26.385 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.695602+0000", 2024-08-17T17:31:26.385 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000117587 2024-08-17T17:31:26.385 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.385 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.385 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:26.385 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.696658+0000", 2024-08-17T17:31:26.385 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001056422 2024-08-17T17:31:26.385 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.385 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.386 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.386 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.705447+0000", 2024-08-17T17:31:26.386 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0087883820000000008 2024-08-17T17:31:26.386 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.386 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.386 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.386 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.813293+0000", 2024-08-17T17:31:26.386 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.107846518 2024-08-17T17:31:26.386 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.386 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.386 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:26.386 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.813329+0000", 2024-08-17T17:31:26.387 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5264e-05 2024-08-17T17:31:26.387 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.387 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.387 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:26.387 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.813351+0000", 2024-08-17T17:31:26.387 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2127999999999999e-05 2024-08-17T17:31:26.387 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.387 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:26.387 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.387 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.387 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.388 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45767.0:38900 7.7 7:efe41b50:::benchmark_data_smithi003_201798_object38899:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-08-17T17:31:26.388 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:28:59.704197+0000", 2024-08-17T17:31:26.388 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 146.638673896, 2024-08-17T17:31:26.388 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.116557962, 2024-08-17T17:31:26.388 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:26.388 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:26.388 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:26.388 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45767", 2024-08-17T17:31:26.388 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:31:26.388 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 38900 2024-08-17T17:31:26.388 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.388 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:26.389 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.389 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:26.389 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.704197+0000", 2024-08-17T17:31:26.389 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.389 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.389 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.389 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:26.389 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.704197+0000", 2024-08-17T17:31:26.389 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.389 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.389 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.389 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:26.390 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.704196+0000", 2024-08-17T17:31:26.390 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:31:26.390 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.390 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.390 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:26.390 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.704254+0000", 2024-08-17T17:31:26.390 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7887999999999999e-05 2024-08-17T17:31:26.390 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.390 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.390 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:26.390 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.704265+0000", 2024-08-17T17:31:26.390 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1579e-05 2024-08-17T17:31:26.391 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.391 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.391 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:26.391 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.704271+0000", 2024-08-17T17:31:26.391 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0789999999999997e-06 2024-08-17T17:31:26.391 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.391 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.391 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:26.391 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.704310+0000", 2024-08-17T17:31:26.391 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8461999999999999e-05 2024-08-17T17:31:26.391 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.391 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.392 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:26.392 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.704399+0000", 2024-08-17T17:31:26.392 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9308999999999995e-05 2024-08-17T17:31:26.392 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.392 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.392 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,3", 2024-08-17T17:31:26.392 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.704488+0000", 2024-08-17T17:31:26.392 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.899e-05 2024-08-17T17:31:26.392 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.392 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.392 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:26.392 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.705554+0000", 2024-08-17T17:31:26.392 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0010656070000000001 2024-08-17T17:31:26.393 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.393 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.393 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.393 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.802497+0000", 2024-08-17T17:31:26.393 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.096943370000000001 2024-08-17T17:31:26.393 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.393 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.393 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.393 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.820673+0000", 2024-08-17T17:31:26.393 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.018176244000000001 2024-08-17T17:31:26.393 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.393 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.393 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:26.394 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.820719+0000", 2024-08-17T17:31:26.394 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5797000000000002e-05 2024-08-17T17:31:26.394 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.394 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.394 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:26.394 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.820755+0000", 2024-08-17T17:31:26.394 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5680999999999999e-05 2024-08-17T17:31:26.394 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.394 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:26.394 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.394 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.395 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.395 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45767.0:38903 7.d 7:b3217b3e:::benchmark_data_smithi003_201798_object38902:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-08-17T17:31:26.395 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:28:59.707959+0000", 2024-08-17T17:31:26.395 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 146.634911786, 2024-08-17T17:31:26.395 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.112632632, 2024-08-17T17:31:26.395 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:26.395 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:26.395 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:26.395 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45767", 2024-08-17T17:31:26.395 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:31:26.395 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 38903 2024-08-17T17:31:26.396 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.396 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:26.396 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.396 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:26.396 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.707959+0000", 2024-08-17T17:31:26.396 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.396 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.396 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.396 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:26.396 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.707959+0000", 2024-08-17T17:31:26.396 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.397 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.397 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.397 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:26.397 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.707958+0000", 2024-08-17T17:31:26.397 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:31:26.397 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.397 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.397 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:26.397 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.708003+0000", 2024-08-17T17:31:26.397 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4728000000000003e-05 2024-08-17T17:31:26.398 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.398 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.398 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:26.398 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.708015+0000", 2024-08-17T17:31:26.398 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2866e-05 2024-08-17T17:31:26.398 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.398 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.398 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:26.398 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.708021+0000", 2024-08-17T17:31:26.398 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5289999999999999e-06 2024-08-17T17:31:26.399 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.399 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.399 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:26.399 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.708067+0000", 2024-08-17T17:31:26.399 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5796e-05 2024-08-17T17:31:26.399 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.399 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.399 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:26.399 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.708194+0000", 2024-08-17T17:31:26.399 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012734199999999999 2024-08-17T17:31:26.399 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.400 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.400 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:31:26.400 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.708342+0000", 2024-08-17T17:31:26.400 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014774699999999999 2024-08-17T17:31:26.400 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.400 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.400 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:26.400 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.709559+0000", 2024-08-17T17:31:26.400 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0012167429999999999 2024-08-17T17:31:26.400 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.400 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.401 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.401 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.710030+0000", 2024-08-17T17:31:26.401 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00047140000000000002 2024-08-17T17:31:26.401 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.401 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.401 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.401 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.820524+0000", 2024-08-17T17:31:26.401 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.11049360699999999 2024-08-17T17:31:26.401 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.401 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.401 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:26.402 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.820558+0000", 2024-08-17T17:31:26.402 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4755000000000002e-05 2024-08-17T17:31:26.402 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.402 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.402 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:26.402 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:59.820591+0000", 2024-08-17T17:31:26.402 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3076e-05 2024-08-17T17:31:26.402 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.402 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:26.402 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.402 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.403 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.403 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45767.0:40490 7.d 7:bf4f1512:::benchmark_data_smithi003_201798_object40489:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-08-17T17:31:26.403 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:29:00.893011+0000", 2024-08-17T17:31:26.403 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 145.44985982, 2024-08-17T17:31:26.403 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.20164364600000001, 2024-08-17T17:31:26.403 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:26.403 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:26.403 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:26.403 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45767", 2024-08-17T17:31:26.403 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:31:26.403 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 40490 2024-08-17T17:31:26.404 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.404 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:26.404 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.404 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:26.404 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:00.893011+0000", 2024-08-17T17:31:26.404 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.404 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.404 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.404 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:26.404 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:00.893011+0000", 2024-08-17T17:31:26.404 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.404 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.405 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.405 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:26.405 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:00.893005+0000", 2024-08-17T17:31:26.405 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999943 2024-08-17T17:31:26.405 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.405 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.405 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:26.405 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:00.893044+0000", 2024-08-17T17:31:26.405 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9264999999999998e-05 2024-08-17T17:31:26.405 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.405 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.405 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:26.406 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:00.893057+0000", 2024-08-17T17:31:26.406 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2391e-05 2024-08-17T17:31:26.406 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.406 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.406 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:26.406 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:00.893062+0000", 2024-08-17T17:31:26.406 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0679999999999996e-06 2024-08-17T17:31:26.406 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.406 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.406 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:26.406 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:00.893110+0000", 2024-08-17T17:31:26.407 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7880999999999998e-05 2024-08-17T17:31:26.407 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.407 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.407 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:26.407 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:00.893196+0000", 2024-08-17T17:31:26.407 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.6448999999999999e-05 2024-08-17T17:31:26.407 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.407 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.407 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:31:26.407 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:00.893283+0000", 2024-08-17T17:31:26.407 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.7089000000000006e-05 2024-08-17T17:31:26.407 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.408 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.408 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:26.408 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:00.894462+0000", 2024-08-17T17:31:26.408 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0011793179999999999 2024-08-17T17:31:26.408 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.408 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.408 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.408 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:00.895130+0000", 2024-08-17T17:31:26.408 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00066796200000000003 2024-08-17T17:31:26.408 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.408 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.408 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.409 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:01.094540+0000", 2024-08-17T17:31:26.409 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.199409686 2024-08-17T17:31:26.409 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.409 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.409 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:26.409 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:01.094593+0000", 2024-08-17T17:31:26.409 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2811e-05 2024-08-17T17:31:26.409 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.409 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.409 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:26.409 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:01.094654+0000", 2024-08-17T17:31:26.409 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1513000000000005e-05 2024-08-17T17:31:26.410 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.410 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:26.410 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.410 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.410 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.410 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45767.0:80562 7.d 7:bf70ba7e:::benchmark_data_smithi003_201798_object80561:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-17T17:31:26.410 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:29:33.950328+0000", 2024-08-17T17:31:26.410 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 112.392542451, 2024-08-17T17:31:26.410 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.104002503, 2024-08-17T17:31:26.411 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:26.411 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:26.411 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:26.411 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45767", 2024-08-17T17:31:26.411 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:31:26.411 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 80562 2024-08-17T17:31:26.411 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.411 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:26.411 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.411 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:26.411 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:33.950328+0000", 2024-08-17T17:31:26.411 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.412 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.412 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.412 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:26.412 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:33.950328+0000", 2024-08-17T17:31:26.412 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.412 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.412 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.412 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:26.412 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:33.950327+0000", 2024-08-17T17:31:26.412 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:31:26.412 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.412 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.413 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:26.413 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:33.950372+0000", 2024-08-17T17:31:26.413 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4932999999999998e-05 2024-08-17T17:31:26.413 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.413 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.413 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:26.413 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:33.950382+0000", 2024-08-17T17:31:26.413 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0507e-05 2024-08-17T17:31:26.413 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.413 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.413 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:26.414 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:33.950387+0000", 2024-08-17T17:31:26.414 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3769999999999998e-06 2024-08-17T17:31:26.414 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.414 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.414 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:26.414 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:33.950421+0000", 2024-08-17T17:31:26.414 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4443999999999998e-05 2024-08-17T17:31:26.414 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.414 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.414 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:26.415 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:33.950514+0000", 2024-08-17T17:31:26.415 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.2592999999999999e-05 2024-08-17T17:31:26.415 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.415 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.415 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:31:26.415 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:33.950614+0000", 2024-08-17T17:31:26.415 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000100054 2024-08-17T17:31:26.415 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.415 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.415 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:26.415 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:33.953781+0000", 2024-08-17T17:31:26.415 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00316717 2024-08-17T17:31:26.416 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.416 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.416 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.416 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:33.957716+0000", 2024-08-17T17:31:26.416 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0039348880000000001 2024-08-17T17:31:26.416 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.416 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.416 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.416 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:34.054223+0000", 2024-08-17T17:31:26.416 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.096507475999999995 2024-08-17T17:31:26.416 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.416 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.417 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:26.417 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:34.054273+0000", 2024-08-17T17:31:26.417 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9026000000000001e-05 2024-08-17T17:31:26.417 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.417 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.417 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:26.417 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:34.054331+0000", 2024-08-17T17:31:26.417 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8143999999999998e-05 2024-08-17T17:31:26.417 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.417 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:26.417 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.418 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.418 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.418 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45767.0:85760 7.d 7:bdd1a010:::benchmark_data_smithi003_201798_object85759:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e235)", 2024-08-17T17:31:26.418 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:29:39.178970+0000", 2024-08-17T17:31:26.418 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 107.163900762, 2024-08-17T17:31:26.418 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.13338634399999999, 2024-08-17T17:31:26.418 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:26.418 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:26.418 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:26.418 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45767", 2024-08-17T17:31:26.418 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:31:26.419 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 85760 2024-08-17T17:31:26.419 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.419 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:26.419 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.419 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:26.419 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:39.178970+0000", 2024-08-17T17:31:26.419 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.419 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.419 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.419 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:26.419 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:39.178970+0000", 2024-08-17T17:31:26.420 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.420 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.420 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.420 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:26.420 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:39.178969+0000", 2024-08-17T17:31:26.420 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:31:26.420 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.420 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.420 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:26.420 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:39.179041+0000", 2024-08-17T17:31:26.420 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.1538000000000005e-05 2024-08-17T17:31:26.420 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.420 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.421 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:26.421 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:39.179058+0000", 2024-08-17T17:31:26.421 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7465e-05 2024-08-17T17:31:26.421 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.421 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.421 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:26.421 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:39.179064+0000", 2024-08-17T17:31:26.421 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6550000000000001e-06 2024-08-17T17:31:26.421 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.421 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.421 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:26.421 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:39.179122+0000", 2024-08-17T17:31:26.422 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8174000000000002e-05 2024-08-17T17:31:26.422 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.422 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.422 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:26.422 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:39.179225+0000", 2024-08-17T17:31:26.422 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000103463 2024-08-17T17:31:26.422 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.422 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.422 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:31:26.422 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:39.179344+0000", 2024-08-17T17:31:26.422 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011827700000000001 2024-08-17T17:31:26.423 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.423 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.423 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:26.423 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:39.184704+0000", 2024-08-17T17:31:26.423 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.005360671 2024-08-17T17:31:26.423 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.423 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.423 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.423 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:39.188772+0000", 2024-08-17T17:31:26.423 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0040673130000000004 2024-08-17T17:31:26.423 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.424 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.424 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.424 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:39.312289+0000", 2024-08-17T17:31:26.424 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.123517854 2024-08-17T17:31:26.424 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.424 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.424 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:26.424 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:39.312330+0000", 2024-08-17T17:31:26.424 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0129000000000002e-05 2024-08-17T17:31:26.424 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.424 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.424 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:26.425 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:39.312356+0000", 2024-08-17T17:31:26.425 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6633000000000001e-05 2024-08-17T17:31:26.425 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.425 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:26.425 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.425 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.425 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.425 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45767.0:85763 7.d 7:b4eca8a2:::benchmark_data_smithi003_201798_object85762:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e235)", 2024-08-17T17:31:26.425 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:29:39.180545+0000", 2024-08-17T17:31:26.425 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 107.162325692, 2024-08-17T17:31:26.425 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.13322225300000001, 2024-08-17T17:31:26.426 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:26.426 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:26.426 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:26.426 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45767", 2024-08-17T17:31:26.426 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:31:26.426 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 85763 2024-08-17T17:31:26.426 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.426 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:26.426 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.426 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:26.426 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:39.180545+0000", 2024-08-17T17:31:26.426 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.427 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.427 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.427 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:26.427 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:39.180545+0000", 2024-08-17T17:31:26.427 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.427 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.427 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.427 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:26.427 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:39.180544+0000", 2024-08-17T17:31:26.427 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:31:26.427 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.428 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.428 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:26.428 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:39.180619+0000", 2024-08-17T17:31:26.428 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.5115000000000002e-05 2024-08-17T17:31:26.428 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.428 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.428 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:26.428 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:39.180628+0000", 2024-08-17T17:31:26.428 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.2730000000000002e-06 2024-08-17T17:31:26.428 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.428 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.428 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:26.429 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:39.180632+0000", 2024-08-17T17:31:26.429 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2799999999999997e-06 2024-08-17T17:31:26.429 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.429 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.429 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:26.429 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:39.181972+0000", 2024-08-17T17:31:26.429 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001340478 2024-08-17T17:31:26.429 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.429 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.429 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:26.429 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:39.182073+0000", 2024-08-17T17:31:26.430 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000100281 2024-08-17T17:31:26.430 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.430 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.430 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:31:26.430 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:39.182204+0000", 2024-08-17T17:31:26.430 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013080399999999999 2024-08-17T17:31:26.430 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.430 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.430 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:26.431 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:39.184957+0000", 2024-08-17T17:31:26.431 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002753485 2024-08-17T17:31:26.431 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.431 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.431 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.431 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:39.189324+0000", 2024-08-17T17:31:26.431 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0043674040000000001 2024-08-17T17:31:26.431 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.431 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.431 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.431 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:39.313683+0000", 2024-08-17T17:31:26.431 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.12435826899999999 2024-08-17T17:31:26.432 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.432 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.432 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:26.432 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:39.313733+0000", 2024-08-17T17:31:26.432 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0448999999999998e-05 2024-08-17T17:31:26.432 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.432 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.432 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:26.432 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:29:39.313767+0000", 2024-08-17T17:31:26.432 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3994e-05 2024-08-17T17:31:26.432 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.433 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:26.433 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.433 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.433 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.433 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45767.0:105845 7.d 7:ba8085e7:::benchmark_data_smithi003_201798_object105844:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-08-17T17:31:26.433 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:30:02.231936+0000", 2024-08-17T17:31:26.433 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 84.110934579000002, 2024-08-17T17:31:26.433 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.15416881399999999, 2024-08-17T17:31:26.433 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:26.433 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:26.433 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:26.434 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45767", 2024-08-17T17:31:26.434 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:31:26.434 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 105845 2024-08-17T17:31:26.434 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.434 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:26.434 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.434 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:26.434 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:02.231936+0000", 2024-08-17T17:31:26.434 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.434 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.434 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.435 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:26.435 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:02.231936+0000", 2024-08-17T17:31:26.435 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.435 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.435 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.435 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:26.435 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:02.231935+0000", 2024-08-17T17:31:26.435 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:31:26.435 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.435 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.435 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:26.436 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:02.231961+0000", 2024-08-17T17:31:26.436 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5687999999999999e-05 2024-08-17T17:31:26.436 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.436 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.436 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:26.436 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:02.231972+0000", 2024-08-17T17:31:26.436 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1403000000000001e-05 2024-08-17T17:31:26.436 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.436 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.436 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:26.436 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:02.231976+0000", 2024-08-17T17:31:26.436 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4950000000000002e-06 2024-08-17T17:31:26.437 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.437 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.437 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:26.437 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:02.232021+0000", 2024-08-17T17:31:26.437 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4199999999999997e-05 2024-08-17T17:31:26.437 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.437 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.437 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:26.437 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:02.232154+0000", 2024-08-17T17:31:26.437 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013372500000000001 2024-08-17T17:31:26.437 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.437 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.438 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:31:26.438 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:02.232313+0000", 2024-08-17T17:31:26.438 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015845799999999999 2024-08-17T17:31:26.438 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.438 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.438 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:26.438 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:02.235267+0000", 2024-08-17T17:31:26.438 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0029539900000000001 2024-08-17T17:31:26.438 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.438 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.439 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.439 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:02.237754+0000", 2024-08-17T17:31:26.439 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0024872150000000001 2024-08-17T17:31:26.439 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.439 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.439 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.439 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:02.386026+0000", 2024-08-17T17:31:26.439 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.148271507 2024-08-17T17:31:26.439 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.439 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.439 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:26.439 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:02.386063+0000", 2024-08-17T17:31:26.440 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.7771000000000001e-05 2024-08-17T17:31:26.440 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.440 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.440 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:26.440 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:02.386105+0000", 2024-08-17T17:31:26.440 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1492999999999997e-05 2024-08-17T17:31:26.440 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.440 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:26.440 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.440 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.440 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.440 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45767.0:112746 7.7 7:e2173a9d:::benchmark_data_smithi003_201798_object112745:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-08-17T17:31:26.441 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:30:10.870515+0000", 2024-08-17T17:31:26.441 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 75.472355855000004, 2024-08-17T17:31:26.441 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.13205296699999999, 2024-08-17T17:31:26.441 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:26.441 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:26.441 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:26.441 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45767", 2024-08-17T17:31:26.441 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:31:26.441 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 112746 2024-08-17T17:31:26.441 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.441 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:26.442 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.442 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:26.442 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.870515+0000", 2024-08-17T17:31:26.442 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.442 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.442 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.442 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:26.442 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.870515+0000", 2024-08-17T17:31:26.442 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.442 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.442 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.442 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:26.443 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.870513+0000", 2024-08-17T17:31:26.443 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:31:26.443 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.443 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.443 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:26.443 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.870570+0000", 2024-08-17T17:31:26.443 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7132000000000001e-05 2024-08-17T17:31:26.443 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.443 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.443 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:26.443 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.870583+0000", 2024-08-17T17:31:26.443 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2480000000000001e-05 2024-08-17T17:31:26.444 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.444 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.444 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:26.444 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.870587+0000", 2024-08-17T17:31:26.444 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3560000000000003e-06 2024-08-17T17:31:26.444 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.444 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.444 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:26.444 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.870639+0000", 2024-08-17T17:31:26.444 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1829999999999997e-05 2024-08-17T17:31:26.444 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.444 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.444 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:26.445 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.870787+0000", 2024-08-17T17:31:26.445 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000148195 2024-08-17T17:31:26.445 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.445 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.445 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,3", 2024-08-17T17:31:26.445 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.870956+0000", 2024-08-17T17:31:26.445 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000168913 2024-08-17T17:31:26.445 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.445 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.445 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:26.446 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:26.447 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.876564+0000", 2024-08-17T17:31:26.447 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0056076590000000001 2024-08-17T17:31:26.447 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.447 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.447 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.448 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.894715+0000", 2024-08-17T17:31:26.448 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.018151766999999999 2024-08-17T17:31:26.448 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.448 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.448 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.448 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:11.002468+0000", 2024-08-17T17:31:26.448 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.107752153 2024-08-17T17:31:26.448 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.448 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.448 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:26.448 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:11.002523+0000", 2024-08-17T17:31:26.449 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5565000000000001e-05 2024-08-17T17:31:26.449 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.449 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.449 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:26.449 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:11.002568+0000", 2024-08-17T17:31:26.449 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4576999999999997e-05 2024-08-17T17:31:26.449 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.449 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:26.449 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.449 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.449 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.449 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45767.0:112749 7.7 7:efb418c7:::benchmark_data_smithi003_201798_object112748:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-08-17T17:31:26.449 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:30:10.872463+0000", 2024-08-17T17:31:26.450 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 75.470407694000002, 2024-08-17T17:31:26.450 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.130238717, 2024-08-17T17:31:26.450 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:26.450 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:26.450 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:26.450 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45767", 2024-08-17T17:31:26.450 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:31:26.450 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 112749 2024-08-17T17:31:26.450 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.450 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:26.450 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.451 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:26.451 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.872463+0000", 2024-08-17T17:31:26.451 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.451 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.451 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.451 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:26.451 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.872463+0000", 2024-08-17T17:31:26.451 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.451 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.451 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.451 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:26.451 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.872462+0000", 2024-08-17T17:31:26.452 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:31:26.452 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.452 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.452 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:26.452 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.872510+0000", 2024-08-17T17:31:26.452 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7787999999999997e-05 2024-08-17T17:31:26.452 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.452 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.452 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:26.452 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.872522+0000", 2024-08-17T17:31:26.452 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.202e-05 2024-08-17T17:31:26.453 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.453 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.453 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:26.453 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.872526+0000", 2024-08-17T17:31:26.453 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.408e-06 2024-08-17T17:31:26.453 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.453 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.453 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:26.453 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.873815+0000", 2024-08-17T17:31:26.453 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001289117 2024-08-17T17:31:26.453 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.454 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.454 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:26.454 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.873929+0000", 2024-08-17T17:31:26.454 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000113711 2024-08-17T17:31:26.454 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.454 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.454 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,3", 2024-08-17T17:31:26.454 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.874070+0000", 2024-08-17T17:31:26.454 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014101000000000001 2024-08-17T17:31:26.454 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.454 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.455 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:26.455 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.876745+0000", 2024-08-17T17:31:26.455 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002675242 2024-08-17T17:31:26.455 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.455 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.455 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.455 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.894762+0000", 2024-08-17T17:31:26.455 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.018016648 2024-08-17T17:31:26.455 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.455 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.455 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.455 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:11.002617+0000", 2024-08-17T17:31:26.456 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.10785528599999999 2024-08-17T17:31:26.456 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.456 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.456 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:26.456 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:11.002669+0000", 2024-08-17T17:31:26.456 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1836999999999999e-05 2024-08-17T17:31:26.456 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.456 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.456 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:26.456 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:11.002702+0000", 2024-08-17T17:31:26.456 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2561999999999998e-05 2024-08-17T17:31:26.456 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.456 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:26.457 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.457 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.457 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.457 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45767.0:112751 7.7 7:ede8d5d6:::benchmark_data_smithi003_201798_object112750:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-08-17T17:31:26.457 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:30:10.876885+0000", 2024-08-17T17:31:26.457 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 75.465985488000001, 2024-08-17T17:31:26.457 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.138204032, 2024-08-17T17:31:26.457 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:26.457 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:26.457 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:26.457 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45767", 2024-08-17T17:31:26.457 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:31:26.458 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 112751 2024-08-17T17:31:26.458 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.458 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:26.458 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.458 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:26.458 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.876885+0000", 2024-08-17T17:31:26.458 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.458 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.458 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.458 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:26.458 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.876885+0000", 2024-08-17T17:31:26.459 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.459 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.459 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.459 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:26.459 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.876884+0000", 2024-08-17T17:31:26.459 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:31:26.459 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.459 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.459 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:26.459 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.876937+0000", 2024-08-17T17:31:26.459 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3334999999999997e-05 2024-08-17T17:31:26.459 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.459 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.460 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:26.460 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.876948+0000", 2024-08-17T17:31:26.460 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0416e-05 2024-08-17T17:31:26.460 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.460 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.460 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:26.460 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.876954+0000", 2024-08-17T17:31:26.460 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9000000000000003e-06 2024-08-17T17:31:26.460 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.460 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.460 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:26.460 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.876987+0000", 2024-08-17T17:31:26.461 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2793000000000003e-05 2024-08-17T17:31:26.461 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.461 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.461 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:26.461 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.877075+0000", 2024-08-17T17:31:26.461 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.8714000000000005e-05 2024-08-17T17:31:26.461 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.461 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.461 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,3", 2024-08-17T17:31:26.461 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.877161+0000", 2024-08-17T17:31:26.461 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.6007999999999994e-05 2024-08-17T17:31:26.461 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.461 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.462 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:26.462 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.879627+0000", 2024-08-17T17:31:26.462 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0024660519999999998 2024-08-17T17:31:26.462 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.462 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.462 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.462 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.894803+0000", 2024-08-17T17:31:26.462 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.015176105 2024-08-17T17:31:26.462 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.462 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.463 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.463 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:11.015030+0000", 2024-08-17T17:31:26.463 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.120227002 2024-08-17T17:31:26.463 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.463 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.463 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:26.463 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:11.015064+0000", 2024-08-17T17:31:26.463 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3569000000000003e-05 2024-08-17T17:31:26.463 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.463 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.464 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:26.464 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:11.015089+0000", 2024-08-17T17:31:26.464 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5126999999999998e-05 2024-08-17T17:31:26.464 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.464 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:26.464 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.464 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.464 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.464 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45767.0:112753 7.7 7:e9b11c2c:::benchmark_data_smithi003_201798_object112752:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-08-17T17:31:26.464 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:30:10.879674+0000", 2024-08-17T17:31:26.464 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 75.463196534999994, 2024-08-17T17:31:26.465 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.13552082200000001, 2024-08-17T17:31:26.465 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:26.465 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:26.465 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:26.465 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45767", 2024-08-17T17:31:26.465 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:31:26.465 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 112753 2024-08-17T17:31:26.465 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.465 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:26.465 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.465 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:26.466 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.879674+0000", 2024-08-17T17:31:26.466 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.466 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.466 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.466 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:26.466 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.879674+0000", 2024-08-17T17:31:26.466 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.466 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.466 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.466 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:26.467 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.879673+0000", 2024-08-17T17:31:26.467 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:31:26.467 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.467 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.467 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:26.467 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.879726+0000", 2024-08-17T17:31:26.467 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2723000000000003e-05 2024-08-17T17:31:26.467 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.467 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.467 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:26.468 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.879736+0000", 2024-08-17T17:31:26.468 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0124e-05 2024-08-17T17:31:26.468 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.468 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.468 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:26.468 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.879741+0000", 2024-08-17T17:31:26.468 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1710000000000001e-06 2024-08-17T17:31:26.468 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.468 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.468 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:26.468 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.879773+0000", 2024-08-17T17:31:26.469 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2069000000000001e-05 2024-08-17T17:31:26.469 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.469 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.469 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:26.469 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.879857+0000", 2024-08-17T17:31:26.469 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.3208999999999996e-05 2024-08-17T17:31:26.469 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.469 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.469 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,3", 2024-08-17T17:31:26.469 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.879949+0000", 2024-08-17T17:31:26.470 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.2736000000000001e-05 2024-08-17T17:31:26.470 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.470 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.470 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:26.470 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.881950+0000", 2024-08-17T17:31:26.470 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0020004789999999999 2024-08-17T17:31:26.470 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.470 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.470 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.470 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.894893+0000", 2024-08-17T17:31:26.471 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012942877 2024-08-17T17:31:26.471 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.471 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.471 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.471 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:11.015127+0000", 2024-08-17T17:31:26.471 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.120233809 2024-08-17T17:31:26.471 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.471 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.471 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:26.471 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:11.015166+0000", 2024-08-17T17:31:26.472 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9381999999999997e-05 2024-08-17T17:31:26.472 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.472 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.472 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:26.472 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:11.015195+0000", 2024-08-17T17:31:26.472 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8988000000000001e-05 2024-08-17T17:31:26.472 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.472 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:26.472 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.472 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.473 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.473 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45767.0:162880 7.d 7:b7953848:::benchmark_data_smithi003_201798_object49658:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-08-17T17:31:26.473 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:31:14.584038+0000", 2024-08-17T17:31:26.473 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 11.758832685, 2024-08-17T17:31:26.473 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.13005641500000001, 2024-08-17T17:31:26.473 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:26.473 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:26.473 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:26.473 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45767", 2024-08-17T17:31:26.473 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:31:26.473 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 162880 2024-08-17T17:31:26.474 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.474 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:26.474 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.474 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:26.474 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:14.584038+0000", 2024-08-17T17:31:26.474 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.474 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.474 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.474 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:26.474 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:14.584038+0000", 2024-08-17T17:31:26.474 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.475 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.475 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.475 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:26.475 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:14.584037+0000", 2024-08-17T17:31:26.475 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:31:26.475 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.475 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.475 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:26.475 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:14.584042+0000", 2024-08-17T17:31:26.475 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9340000000000003e-06 2024-08-17T17:31:26.476 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.476 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.476 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:26.476 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:14.584055+0000", 2024-08-17T17:31:26.476 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2988e-05 2024-08-17T17:31:26.476 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.476 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.476 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:26.476 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:14.584061+0000", 2024-08-17T17:31:26.476 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.502e-06 2024-08-17T17:31:26.476 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.476 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.477 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:26.477 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:14.584100+0000", 2024-08-17T17:31:26.477 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8804000000000003e-05 2024-08-17T17:31:26.477 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.477 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.477 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:26.477 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:14.584225+0000", 2024-08-17T17:31:26.477 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000124998 2024-08-17T17:31:26.477 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.477 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.477 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:31:26.478 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:14.584338+0000", 2024-08-17T17:31:26.478 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011357399999999999 2024-08-17T17:31:26.478 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.478 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.478 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:26.478 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:14.588962+0000", 2024-08-17T17:31:26.478 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00462335 2024-08-17T17:31:26.478 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.478 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.478 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.478 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:14.593779+0000", 2024-08-17T17:31:26.478 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0048171560000000004 2024-08-17T17:31:26.479 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.479 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.479 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.479 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:14.713827+0000", 2024-08-17T17:31:26.479 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.12004806 2024-08-17T17:31:26.479 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.479 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.479 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:26.479 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:14.714056+0000", 2024-08-17T17:31:26.479 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000229022 2024-08-17T17:31:26.479 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.480 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.480 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:26.480 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:14.714094+0000", 2024-08-17T17:31:26.480 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8392000000000001e-05 2024-08-17T17:31:26.480 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.480 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:26.480 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.480 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.480 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.480 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45767.0:163952 7.d 7:bbefa93f:::benchmark_data_smithi003_201798_object50730:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-08-17T17:31:26.480 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:31:16.144877+0000", 2024-08-17T17:31:26.481 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 10.197993623, 2024-08-17T17:31:26.481 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.19094304000000001, 2024-08-17T17:31:26.481 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:26.481 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:26.481 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:26.481 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45767", 2024-08-17T17:31:26.481 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:31:26.481 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 163952 2024-08-17T17:31:26.481 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.481 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:26.481 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.482 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:26.482 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:16.144877+0000", 2024-08-17T17:31:26.482 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.482 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.482 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.482 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:26.482 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:16.144877+0000", 2024-08-17T17:31:26.482 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.482 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.482 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.482 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:26.483 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:16.144875+0000", 2024-08-17T17:31:26.483 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:31:26.483 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.483 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.483 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:26.483 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:16.144885+0000", 2024-08-17T17:31:26.483 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.4499999999999993e-06 2024-08-17T17:31:26.483 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.483 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.483 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:26.484 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:16.144909+0000", 2024-08-17T17:31:26.484 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4592000000000001e-05 2024-08-17T17:31:26.484 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.484 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.484 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:26.484 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:16.144917+0000", 2024-08-17T17:31:26.484 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.4739999999999999e-06 2024-08-17T17:31:26.484 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.484 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.484 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:26.484 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:16.144978+0000", 2024-08-17T17:31:26.484 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1724999999999996e-05 2024-08-17T17:31:26.485 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.485 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.485 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:26.485 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:16.145134+0000", 2024-08-17T17:31:26.485 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015583 2024-08-17T17:31:26.485 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.485 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.485 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:31:26.485 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:16.145279+0000", 2024-08-17T17:31:26.485 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014515 2024-08-17T17:31:26.486 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.486 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.486 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:26.486 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:16.149303+0000", 2024-08-17T17:31:26.486 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0040235619999999996 2024-08-17T17:31:26.486 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.486 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.486 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.486 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:16.152426+0000", 2024-08-17T17:31:26.486 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0031231520000000001 2024-08-17T17:31:26.486 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.486 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.487 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.487 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:16.335724+0000", 2024-08-17T17:31:26.487 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.183298446 2024-08-17T17:31:26.487 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.487 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.487 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:26.487 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:16.335775+0000", 2024-08-17T17:31:26.487 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0042e-05 2024-08-17T17:31:26.487 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.487 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.487 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:26.487 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:16.335820+0000", 2024-08-17T17:31:26.488 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5482999999999998e-05 2024-08-17T17:31:26.488 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.488 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:26.488 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.488 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.488 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.488 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45767.0:163964 7.d 7:b31d9fe0:::benchmark_data_smithi003_201798_object50742:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-08-17T17:31:26.488 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:31:16.171806+0000", 2024-08-17T17:31:26.488 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 10.171064608, 2024-08-17T17:31:26.488 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.164221492, 2024-08-17T17:31:26.488 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:31:26.489 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:31:26.489 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:31:26.489 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45767", 2024-08-17T17:31:26.489 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:31:26.489 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 163964 2024-08-17T17:31:26.489 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.489 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:31:26.489 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.489 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:31:26.489 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:16.171806+0000", 2024-08-17T17:31:26.489 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.489 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.490 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.490 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:31:26.490 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:16.171806+0000", 2024-08-17T17:31:26.490 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:31:26.490 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.490 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.490 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:31:26.490 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:16.171804+0000", 2024-08-17T17:31:26.490 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999976 2024-08-17T17:31:26.490 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.490 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.491 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:31:26.491 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:16.171814+0000", 2024-08-17T17:31:26.491 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0412e-05 2024-08-17T17:31:26.491 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.491 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.491 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:31:26.491 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:16.171833+0000", 2024-08-17T17:31:26.491 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.8711000000000001e-05 2024-08-17T17:31:26.491 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.491 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.491 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:31:26.492 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:16.171840+0000", 2024-08-17T17:31:26.492 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.1189999999999999e-06 2024-08-17T17:31:26.492 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.492 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.492 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:31:26.492 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:16.171897+0000", 2024-08-17T17:31:26.492 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7463999999999997e-05 2024-08-17T17:31:26.492 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.492 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.492 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:31:26.492 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:16.172054+0000", 2024-08-17T17:31:26.492 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015696199999999999 2024-08-17T17:31:26.493 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.493 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.493 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:31:26.493 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:16.172200+0000", 2024-08-17T17:31:26.493 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000145489 2024-08-17T17:31:26.493 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.493 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.493 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:31:26.493 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:16.175379+0000", 2024-08-17T17:31:26.493 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003178621 2024-08-17T17:31:26.493 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.493 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.494 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.494 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:16.175736+0000", 2024-08-17T17:31:26.494 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00035713299999999998 2024-08-17T17:31:26.494 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.494 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.494 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:26.494 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:16.335976+0000", 2024-08-17T17:31:26.494 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.160240257 2024-08-17T17:31:26.494 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.494 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.494 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:31:26.494 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:16.336006+0000", 2024-08-17T17:31:26.495 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0195000000000001e-05 2024-08-17T17:31:26.495 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:31:26.495 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:31:26.495 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:31:26.495 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:16.336027+0000", 2024-08-17T17:31:26.495 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1358000000000001e-05 2024-08-17T17:31:26.495 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.495 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:26.495 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.495 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:31:26.495 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:31:26.496 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:31:26.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:26.497 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:26.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:26.598 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:26.639 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:26.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:26.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:26.741 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:26.769 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:26.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:26.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:26.871 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:26.896 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:26.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:26.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:26.997 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:27.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:27.066 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:27.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:27.168 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:27.206 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:27.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:27.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:27.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:27.365 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:27.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:27.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:27.467 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:27.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:27.521 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:27.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:27.622 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:27.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:27.669 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:27.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:27.771 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:27.800 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:27.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:27.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:27.901 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:27.928 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:27.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:27.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:27.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:27 smithi053 ceph-mon[89963]: pgmap v724: 89 pgs: 89 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 750 op/s 2024-08-17T17:31:28.003 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:27 smithi003 ceph-mon[100615]: pgmap v724: 89 pgs: 89 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 750 op/s 2024-08-17T17:31:28.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:27 smithi003 ceph-mon[103344]: pgmap v724: 89 pgs: 89 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 750 op/s 2024-08-17T17:31:28.030 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:28.051 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-17T17:31:28.072 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:28.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:28.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:28.174 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:28.232 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:28.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:28.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:28.334 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:28.382 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:28.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:28.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:28.484 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:28.533 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:28.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:28.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:28.634 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:28.661 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:28.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:28.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:28.763 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:28.797 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:28.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:28.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:28.837 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:31:28.839 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:31:28.900 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:28.927 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:28.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:28.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:29.029 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:29.075 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:29.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:29.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:29.126 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:31:29.126 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:31:29.165 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:31:29.165 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:31:29.177 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:29.193 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:31:29.202 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:31:29.202 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:31:29.213 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:29.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:29.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:29.243 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:31:29.244 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:31:29.276 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:31:29.276 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:31:29.314 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:31:29.314 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:31:29.316 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:29.338 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:29.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:29.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:29.347 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:31:29.347 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:31:29.381 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:31:29.381 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:31:29.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:29.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:29.479 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:29.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:29.580 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:29.621 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:29.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:29.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:29.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:29 smithi053 ceph-mon[89963]: pgmap v725: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 688 op/s 2024-08-17T17:31:29.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:29 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/330504943' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:31:29.723 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:29.747 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:29.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:29.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:29.848 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:29.885 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:29.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:29.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:29.987 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:30.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:29 smithi003 ceph-mon[103344]: pgmap v725: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 688 op/s 2024-08-17T17:31:30.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:29 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/330504943' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:31:30.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:29 smithi003 ceph-mon[100615]: pgmap v725: 89 pgs: 89 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 688 op/s 2024-08-17T17:31:30.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:29 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/330504943' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:31:30.026 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:30.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:30.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:30.129 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:30.155 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:30.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:30.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:30.257 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:30.283 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:30.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:30.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:30.298 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:31:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:31:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:31:30.385 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:30.421 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:30.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:30.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:30.523 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:30.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:30.570 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:30.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:30.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:30.711 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:30.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:30.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:30.812 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:30.839 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:30.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:30.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:30.941 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:30.966 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:30.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:30 smithi053 ceph-mon[89963]: from='client.55399 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:31:30.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:30.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:31.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:30 smithi003 ceph-mon[103344]: from='client.55399 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:31:31.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:30 smithi003 ceph-mon[100615]: from='client.55399 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:31:31.067 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:31.094 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:31.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:31.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:31.195 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:31.221 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:31.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:31.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:31.323 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:31.354 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:31.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:31.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:31.456 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:31.499 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:31.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:31.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:31.600 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:31.635 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:31.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:31.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:31.737 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:31.776 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:31.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:31.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:31.878 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:31.915 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:31.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:31.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:31.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:31 smithi053 ceph-mon[89963]: pgmap v726: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 859 op/s 2024-08-17T17:31:31.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:31 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:31:31.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:31 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:31:32.016 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:32.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:31 smithi003 ceph-mon[103344]: pgmap v726: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 859 op/s 2024-08-17T17:31:32.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:31 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:31:32.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:31 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:31:32.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:31 smithi003 ceph-mon[100615]: pgmap v726: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 859 op/s 2024-08-17T17:31:32.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:31 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:31:32.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:31 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:31:32.059 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:32.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:32.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:32.160 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:32.189 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:32.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:32.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:32.292 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:32.305 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:31:32.305 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:31:32.321 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:32.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:32.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:32.423 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:32.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:32.453 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:32.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:32.554 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:32.612 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:32.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:32.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:32.713 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:31:32.759 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:32.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:32.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:31:32.773 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":725,"stamp":"2024-08-17T17:31:28.023687+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3676772325,"num_objects":56059,"num_object_clones":0,"num_object_copies":168177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56059,"num_whiteouts":0,"num_read":6377,"num_read_kb":13834,"num_write":171531,"num_write_kb":7261056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":158,"num_bytes_recovered":11937158,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123090,"ondisk_log_size":123090,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":13526116,"kb_used_data":10855340,"kb_used_omap":33,"kb_used_meta":2670558,"kb_avail":642689948,"statfs":{"total":671965249536,"available":658114506752,"internally_reserved":0,"allocated":11115868160,"data_stored":11106235002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34068,"internal_metadata":2734652140},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,7,4,7,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-541589504,"num_objects":-8264,"num_object_clones":0,"num_object_copies":-24792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8264,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005266"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":349,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.606742+0000","last_change":"2024-08-17T17:17:32.495652+0000","last_active":"2024-08-17T17:31:13.606742+0000","last_peered":"2024-08-17T17:31:13.606742+0000","last_clean":"2024-08-17T17:31:13.606742+0000","last_became_active":"2024-08-17T17:17:32.495218+0000","last_became_peered":"2024-08-17T17:17:32.495218+0000","last_unstale":"2024-08-17T17:31:13.606742+0000","last_undegraded":"2024-08-17T17:31:13.606742+0000","last_fullsized":"2024-08-17T17:31:13.606742+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:11:45.821282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":223,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.555074+0000","last_change":"2024-08-17T17:18:59.424887+0000","last_active":"2024-08-17T17:31:08.555074+0000","last_peered":"2024-08-17T17:31:08.555074+0000","last_clean":"2024-08-17T17:31:08.555074+0000","last_became_active":"2024-08-17T17:18:59.411539+0000","last_became_peered":"2024-08-17T17:18:59.411539+0000","last_unstale":"2024-08-17T17:31:08.555074+0000","last_undegraded":"2024-08-17T17:31:08.555074+0000","last_fullsized":"2024-08-17T17:31:08.555074+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:46:20.082241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":63,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.553710+0000","last_change":"2024-08-17T17:24:53.460742+0000","last_active":"2024-08-17T17:31:08.553710+0000","last_peered":"2024-08-17T17:31:08.553710+0000","last_clean":"2024-08-17T17:31:08.553710+0000","last_became_active":"2024-08-17T17:24:53.460365+0000","last_became_peered":"2024-08-17T17:24:53.460365+0000","last_unstale":"2024-08-17T17:31:08.553710+0000","last_undegraded":"2024-08-17T17:31:08.553710+0000","last_fullsized":"2024-08-17T17:31:08.553710+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:24:08.116968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":161,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.597684+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:31:13.597684+0000","last_peered":"2024-08-17T17:31:13.597684+0000","last_clean":"2024-08-17T17:31:13.597684+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:31:13.597684+0000","last_undegraded":"2024-08-17T17:31:13.597684+0000","last_fullsized":"2024-08-17T17:31:13.597684+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":161,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.599749+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:31:13.599749+0000","last_peered":"2024-08-17T17:31:13.599749+0000","last_clean":"2024-08-17T17:31:13.599749+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:31:13.599749+0000","last_undegraded":"2024-08-17T17:31:13.599749+0000","last_fullsized":"2024-08-17T17:31:13.599749+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":198,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.554987+0000","last_change":"2024-08-17T17:20:49.258206+0000","last_active":"2024-08-17T17:31:08.554987+0000","last_peered":"2024-08-17T17:31:08.554987+0000","last_clean":"2024-08-17T17:31:08.554987+0000","last_became_active":"2024-08-17T17:20:49.257761+0000","last_became_peered":"2024-08-17T17:20:49.257761+0000","last_unstale":"2024-08-17T17:31:08.554987+0000","last_undegraded":"2024-08-17T17:31:08.554987+0000","last_fullsized":"2024-08-17T17:31:08.554987+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:38:35.363748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":363,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.616727+0000","last_change":"2024-08-17T17:19:57.514773+0000","last_active":"2024-08-17T17:31:13.616727+0000","last_peered":"2024-08-17T17:31:13.616727+0000","last_clean":"2024-08-17T17:31:13.616727+0000","last_became_active":"2024-08-17T17:19:57.513839+0000","last_became_peered":"2024-08-17T17:19:57.513839+0000","last_unstale":"2024-08-17T17:31:13.616727+0000","last_undegraded":"2024-08-17T17:31:13.616727+0000","last_fullsized":"2024-08-17T17:31:13.616727+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:42:48.063169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":293,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.559456+0000","last_change":"2024-08-17T17:24:53.465875+0000","last_active":"2024-08-17T17:31:08.559456+0000","last_peered":"2024-08-17T17:31:08.559456+0000","last_clean":"2024-08-17T17:31:08.559456+0000","last_became_active":"2024-08-17T17:24:53.458129+0000","last_became_peered":"2024-08-17T17:24:53.458129+0000","last_unstale":"2024-08-17T17:31:08.559456+0000","last_undegraded":"2024-08-17T17:31:08.559456+0000","last_fullsized":"2024-08-17T17:31:08.559456+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:51:33.584869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.b","version":"247'10678","reported_seq":10732,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:24.417950+0000","last_change":"2024-08-17T17:28:27.510969+0000","last_active":"2024-08-17T17:31:24.417950+0000","last_peered":"2024-08-17T17:31:24.417950+0000","last_clean":"2024-08-17T17:31:24.417950+0000","last_became_active":"2024-08-17T17:28:27.510664+0000","last_became_peered":"2024-08-17T17:28:27.510664+0000","last_unstale":"2024-08-17T17:31:24.417950+0000","last_undegraded":"2024-08-17T17:31:24.417950+0000","last_fullsized":"2024-08-17T17:31:24.417950+0000","mapping_epoch":226,"log_start":"234'4000","ondisk_log_start":"234'4000","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":6678,"log_dups_size":0,"ondisk_log_size":6678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:16:23.845426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235405312,"num_objects":3592,"num_object_clones":0,"num_object_copies":10776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10678,"num_write_kb":456640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":159,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.555557+0000","last_change":"2024-08-17T17:24:06.597382+0000","last_active":"2024-08-17T17:31:08.555557+0000","last_peered":"2024-08-17T17:31:08.555557+0000","last_clean":"2024-08-17T17:31:08.555557+0000","last_became_active":"2024-08-17T17:24:06.588881+0000","last_became_peered":"2024-08-17T17:24:06.588881+0000","last_unstale":"2024-08-17T17:31:08.555557+0000","last_undegraded":"2024-08-17T17:31:08.555557+0000","last_fullsized":"2024-08-17T17:31:08.555557+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:27.240751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":73,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.553306+0000","last_change":"2024-08-17T17:25:04.503780+0000","last_active":"2024-08-17T17:31:08.553306+0000","last_peered":"2024-08-17T17:31:08.553306+0000","last_clean":"2024-08-17T17:31:08.553306+0000","last_became_active":"2024-08-17T17:25:04.503503+0000","last_became_peered":"2024-08-17T17:25:04.503503+0000","last_unstale":"2024-08-17T17:31:08.553306+0000","last_undegraded":"2024-08-17T17:31:08.553306+0000","last_fullsized":"2024-08-17T17:31:08.553306+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:04:48.171216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"247'10742","reported_seq":10796,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:27.456140+0000","last_change":"2024-08-17T17:28:27.509334+0000","last_active":"2024-08-17T17:31:27.456140+0000","last_peered":"2024-08-17T17:31:27.456140+0000","last_clean":"2024-08-17T17:31:27.456140+0000","last_became_active":"2024-08-17T17:28:27.509026+0000","last_became_peered":"2024-08-17T17:28:27.509026+0000","last_unstale":"2024-08-17T17:31:27.456140+0000","last_undegraded":"2024-08-17T17:31:27.456140+0000","last_fullsized":"2024-08-17T17:31:27.456140+0000","mapping_epoch":226,"log_start":"234'3900","ondisk_log_start":"234'3900","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":6842,"log_dups_size":0,"ondisk_log_size":6842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:47:28.720615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217186304,"num_objects":3314,"num_object_clones":0,"num_object_copies":9942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10742,"num_write_kb":449792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":145,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.555398+0000","last_change":"2024-08-17T17:24:06.599023+0000","last_active":"2024-08-17T17:31:08.555398+0000","last_peered":"2024-08-17T17:31:08.555398+0000","last_clean":"2024-08-17T17:31:08.555398+0000","last_became_active":"2024-08-17T17:24:06.588954+0000","last_became_peered":"2024-08-17T17:24:06.588954+0000","last_unstale":"2024-08-17T17:31:08.555398+0000","last_undegraded":"2024-08-17T17:31:08.555398+0000","last_fullsized":"2024-08-17T17:31:08.555398+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:20:51.445576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":304,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.631783+0000","last_change":"2024-08-17T17:20:49.271540+0000","last_active":"2024-08-17T17:31:13.631783+0000","last_peered":"2024-08-17T17:31:13.631783+0000","last_clean":"2024-08-17T17:31:13.631783+0000","last_became_active":"2024-08-17T17:20:49.271130+0000","last_became_peered":"2024-08-17T17:20:49.271130+0000","last_unstale":"2024-08-17T17:31:13.631783+0000","last_undegraded":"2024-08-17T17:31:13.631783+0000","last_fullsized":"2024-08-17T17:31:13.631783+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:05.386705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.9","version":"247'10785","reported_seq":10839,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:23.403076+0000","last_change":"2024-08-17T17:28:27.508033+0000","last_active":"2024-08-17T17:31:23.403076+0000","last_peered":"2024-08-17T17:31:23.403076+0000","last_clean":"2024-08-17T17:31:23.403076+0000","last_became_active":"2024-08-17T17:28:27.507657+0000","last_became_peered":"2024-08-17T17:28:27.507657+0000","last_unstale":"2024-08-17T17:31:23.403076+0000","last_undegraded":"2024-08-17T17:31:23.403076+0000","last_fullsized":"2024-08-17T17:31:23.403076+0000","mapping_epoch":226,"log_start":"233'3400","ondisk_log_start":"233'3400","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":7385,"log_dups_size":0,"ondisk_log_size":7385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:09:04.908184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238223360,"num_objects":3635,"num_object_clones":0,"num_object_copies":10905,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10785,"num_write_kb":461440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":217,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.555411+0000","last_change":"2024-08-17T17:24:53.467526+0000","last_active":"2024-08-17T17:31:08.555411+0000","last_peered":"2024-08-17T17:31:08.555411+0000","last_clean":"2024-08-17T17:31:08.555411+0000","last_became_active":"2024-08-17T17:24:53.460631+0000","last_became_peered":"2024-08-17T17:24:53.460631+0000","last_unstale":"2024-08-17T17:31:08.555411+0000","last_undegraded":"2024-08-17T17:31:08.555411+0000","last_fullsized":"2024-08-17T17:31:08.555411+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:05:24.938831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":59,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.559798+0000","last_change":"2024-08-17T17:25:04.507176+0000","last_active":"2024-08-17T17:31:08.559798+0000","last_peered":"2024-08-17T17:31:08.559798+0000","last_clean":"2024-08-17T17:31:08.559798+0000","last_became_active":"2024-08-17T17:25:04.506788+0000","last_became_peered":"2024-08-17T17:25:04.506788+0000","last_unstale":"2024-08-17T17:31:08.559798+0000","last_undegraded":"2024-08-17T17:31:08.559798+0000","last_fullsized":"2024-08-17T17:31:08.559798+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:22:04.658216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"247'10982","reported_seq":11010,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:24.086011+0000","last_change":"2024-08-17T17:28:27.509743+0000","last_active":"2024-08-17T17:31:24.086011+0000","last_peered":"2024-08-17T17:31:24.086011+0000","last_clean":"2024-08-17T17:31:24.086011+0000","last_became_active":"2024-08-17T17:28:27.509414+0000","last_became_peered":"2024-08-17T17:28:27.509414+0000","last_unstale":"2024-08-17T17:31:24.086011+0000","last_undegraded":"2024-08-17T17:31:24.086011+0000","last_fullsized":"2024-08-17T17:31:24.086011+0000","mapping_epoch":226,"log_start":"233'3400","ondisk_log_start":"233'3400","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":7582,"log_dups_size":0,"ondisk_log_size":7582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:37:42.377123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234881024,"num_objects":3584,"num_object_clones":0,"num_object_copies":10752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10982,"num_write_kb":466112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":205,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.553538+0000","last_change":"2024-08-17T17:17:32.495428+0000","last_active":"2024-08-17T17:31:08.553538+0000","last_peered":"2024-08-17T17:31:08.553538+0000","last_clean":"2024-08-17T17:31:08.553538+0000","last_became_active":"2024-08-17T17:17:32.494978+0000","last_became_peered":"2024-08-17T17:17:32.494978+0000","last_unstale":"2024-08-17T17:31:08.553538+0000","last_undegraded":"2024-08-17T17:31:08.553538+0000","last_fullsized":"2024-08-17T17:31:08.553538+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:42:24.647812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":352,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.598177+0000","last_change":"2024-08-17T17:20:49.272126+0000","last_active":"2024-08-17T17:31:13.598177+0000","last_peered":"2024-08-17T17:31:13.598177+0000","last_clean":"2024-08-17T17:31:13.598177+0000","last_became_active":"2024-08-17T17:20:49.271185+0000","last_became_peered":"2024-08-17T17:20:49.271185+0000","last_unstale":"2024-08-17T17:31:13.598177+0000","last_undegraded":"2024-08-17T17:31:13.598177+0000","last_fullsized":"2024-08-17T17:31:13.598177+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:43:58.184939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.f","version":"247'10651","reported_seq":10705,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:23.413988+0000","last_change":"2024-08-17T17:28:27.512879+0000","last_active":"2024-08-17T17:31:23.413988+0000","last_peered":"2024-08-17T17:31:23.413988+0000","last_clean":"2024-08-17T17:31:23.413988+0000","last_became_active":"2024-08-17T17:28:27.512586+0000","last_became_peered":"2024-08-17T17:28:27.512586+0000","last_unstale":"2024-08-17T17:31:23.413988+0000","last_undegraded":"2024-08-17T17:31:23.413988+0000","last_fullsized":"2024-08-17T17:31:23.413988+0000","mapping_epoch":226,"log_start":"233'3300","ondisk_log_start":"233'3300","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":7351,"log_dups_size":0,"ondisk_log_size":7351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:12:27.175368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235077632,"num_objects":3587,"num_object_clones":0,"num_object_copies":10761,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10651,"num_write_kb":455616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":381,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.612472+0000","last_change":"2024-08-17T17:24:06.595380+0000","last_active":"2024-08-17T17:31:13.612472+0000","last_peered":"2024-08-17T17:31:13.612472+0000","last_clean":"2024-08-17T17:31:13.612472+0000","last_became_active":"2024-08-17T17:24:06.584747+0000","last_became_peered":"2024-08-17T17:24:06.584747+0000","last_unstale":"2024-08-17T17:31:13.612472+0000","last_undegraded":"2024-08-17T17:31:13.612472+0000","last_fullsized":"2024-08-17T17:31:13.612472+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:02:18.473980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":326,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.606986+0000","last_change":"2024-08-17T17:20:49.280596+0000","last_active":"2024-08-17T17:31:13.606986+0000","last_peered":"2024-08-17T17:31:13.606986+0000","last_clean":"2024-08-17T17:31:13.606986+0000","last_became_active":"2024-08-17T17:20:49.279553+0000","last_became_peered":"2024-08-17T17:20:49.279553+0000","last_unstale":"2024-08-17T17:31:13.606986+0000","last_undegraded":"2024-08-17T17:31:13.606986+0000","last_fullsized":"2024-08-17T17:31:13.606986+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:13:24.124392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"247'10433","reported_seq":10487,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:24.748376+0000","last_change":"2024-08-17T17:28:27.506638+0000","last_active":"2024-08-17T17:31:24.748376+0000","last_peered":"2024-08-17T17:31:24.748376+0000","last_clean":"2024-08-17T17:31:24.748376+0000","last_became_active":"2024-08-17T17:28:27.506343+0000","last_became_peered":"2024-08-17T17:28:27.506343+0000","last_unstale":"2024-08-17T17:31:24.748376+0000","last_undegraded":"2024-08-17T17:31:24.748376+0000","last_fullsized":"2024-08-17T17:31:24.748376+0000","mapping_epoch":226,"log_start":"232'400","ondisk_log_start":"232'400","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:57:42.320556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223019008,"num_objects":3403,"num_object_clones":0,"num_object_copies":10209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10433,"num_write_kb":442752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":278,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.622988+0000","last_change":"2024-08-17T17:20:49.276769+0000","last_active":"2024-08-17T17:31:13.622988+0000","last_peered":"2024-08-17T17:31:13.622988+0000","last_clean":"2024-08-17T17:31:13.622988+0000","last_became_active":"2024-08-17T17:20:49.276498+0000","last_became_peered":"2024-08-17T17:20:49.276498+0000","last_unstale":"2024-08-17T17:31:13.622988+0000","last_undegraded":"2024-08-17T17:31:13.622988+0000","last_fullsized":"2024-08-17T17:31:13.622988+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:39:05.419440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":226,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.555295+0000","last_change":"2024-08-17T17:20:14.455738+0000","last_active":"2024-08-17T17:31:08.555295+0000","last_peered":"2024-08-17T17:31:08.555295+0000","last_clean":"2024-08-17T17:31:08.555295+0000","last_became_active":"2024-08-17T17:20:14.455469+0000","last_became_peered":"2024-08-17T17:20:14.455469+0000","last_unstale":"2024-08-17T17:31:08.555295+0000","last_undegraded":"2024-08-17T17:31:08.555295+0000","last_fullsized":"2024-08-17T17:31:08.555295+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:27:28.424336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.d","version":"247'10445","reported_seq":10475,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:23.998302+0000","last_change":"2024-08-17T17:28:27.510620+0000","last_active":"2024-08-17T17:31:23.998302+0000","last_peered":"2024-08-17T17:31:23.998302+0000","last_clean":"2024-08-17T17:31:23.998302+0000","last_became_active":"2024-08-17T17:28:27.510351+0000","last_became_peered":"2024-08-17T17:28:27.510351+0000","last_unstale":"2024-08-17T17:31:23.998302+0000","last_undegraded":"2024-08-17T17:31:23.998302+0000","last_fullsized":"2024-08-17T17:31:23.998302+0000","mapping_epoch":226,"log_start":"232'2700","ondisk_log_start":"232'2700","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":7745,"log_dups_size":0,"ondisk_log_size":7745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:14:19.994370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228327448,"num_objects":3485,"num_object_clones":0,"num_object_copies":10455,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3485,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10445,"num_write_kb":445697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":180,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.555040+0000","last_change":"2024-08-17T17:20:14.431140+0000","last_active":"2024-08-17T17:31:08.555040+0000","last_peered":"2024-08-17T17:31:08.555040+0000","last_clean":"2024-08-17T17:31:08.555040+0000","last_became_active":"2024-08-17T17:20:14.430839+0000","last_became_peered":"2024-08-17T17:20:14.430839+0000","last_unstale":"2024-08-17T17:31:08.555040+0000","last_undegraded":"2024-08-17T17:31:08.555040+0000","last_fullsized":"2024-08-17T17:31:08.555040+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:08:16.303616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":59,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.560121+0000","last_change":"2024-08-17T17:25:04.509325+0000","last_active":"2024-08-17T17:31:08.560121+0000","last_peered":"2024-08-17T17:31:08.560121+0000","last_clean":"2024-08-17T17:31:08.560121+0000","last_became_active":"2024-08-17T17:25:04.508364+0000","last_became_peered":"2024-08-17T17:25:04.508364+0000","last_unstale":"2024-08-17T17:31:08.560121+0000","last_undegraded":"2024-08-17T17:31:08.560121+0000","last_fullsized":"2024-08-17T17:31:08.560121+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:48:56.415763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"247'10551","reported_seq":10579,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:24.014449+0000","last_change":"2024-08-17T17:28:27.512014+0000","last_active":"2024-08-17T17:31:24.014449+0000","last_peered":"2024-08-17T17:31:24.014449+0000","last_clean":"2024-08-17T17:31:24.014449+0000","last_became_active":"2024-08-17T17:28:27.511692+0000","last_became_peered":"2024-08-17T17:28:27.511692+0000","last_unstale":"2024-08-17T17:31:24.014449+0000","last_undegraded":"2024-08-17T17:31:24.014449+0000","last_fullsized":"2024-08-17T17:31:24.014449+0000","mapping_epoch":226,"log_start":"232'2800","ondisk_log_start":"232'2800","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":7751,"log_dups_size":0,"ondisk_log_size":7751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:38:59.458491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233373696,"num_objects":3561,"num_object_clones":0,"num_object_copies":10683,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10551,"num_write_kb":451584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":365,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.616913+0000","last_change":"2024-08-17T17:19:57.520745+0000","last_active":"2024-08-17T17:31:13.616913+0000","last_peered":"2024-08-17T17:31:13.616913+0000","last_clean":"2024-08-17T17:31:13.616913+0000","last_became_active":"2024-08-17T17:19:57.520152+0000","last_became_peered":"2024-08-17T17:19:57.520152+0000","last_unstale":"2024-08-17T17:31:13.616913+0000","last_undegraded":"2024-08-17T17:31:13.616913+0000","last_fullsized":"2024-08-17T17:31:13.616913+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:19:39.495644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":340,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.605864+0000","last_change":"2024-08-17T17:20:49.274372+0000","last_active":"2024-08-17T17:31:13.605864+0000","last_peered":"2024-08-17T17:31:13.605864+0000","last_clean":"2024-08-17T17:31:13.605864+0000","last_became_active":"2024-08-17T17:20:49.274023+0000","last_became_peered":"2024-08-17T17:20:49.274023+0000","last_unstale":"2024-08-17T17:31:13.605864+0000","last_undegraded":"2024-08-17T17:31:13.605864+0000","last_fullsized":"2024-08-17T17:31:13.605864+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:31:02.529766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"247'10529","reported_seq":10582,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:23.447941+0000","last_change":"2024-08-17T17:28:27.511026+0000","last_active":"2024-08-17T17:31:23.447941+0000","last_peered":"2024-08-17T17:31:23.447941+0000","last_clean":"2024-08-17T17:31:23.447941+0000","last_became_active":"2024-08-17T17:28:27.510738+0000","last_became_peered":"2024-08-17T17:28:27.510738+0000","last_unstale":"2024-08-17T17:31:23.447941+0000","last_undegraded":"2024-08-17T17:31:23.447941+0000","last_fullsized":"2024-08-17T17:31:23.447941+0000","mapping_epoch":226,"log_start":"233'3200","ondisk_log_start":"233'3200","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":7329,"log_dups_size":0,"ondisk_log_size":7329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:58:33.553008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232849408,"num_objects":3553,"num_object_clones":0,"num_object_copies":10659,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10529,"num_write_kb":450624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":574,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.554882+0000","last_change":"2024-08-17T17:24:09.599615+0000","last_active":"2024-08-17T17:31:08.554882+0000","last_peered":"2024-08-17T17:31:08.554882+0000","last_clean":"2024-08-17T17:31:08.554882+0000","last_became_active":"2024-08-17T17:24:09.599085+0000","last_became_peered":"2024-08-17T17:24:09.599085+0000","last_unstale":"2024-08-17T17:31:08.554882+0000","last_undegraded":"2024-08-17T17:31:08.554882+0000","last_fullsized":"2024-08-17T17:31:08.554882+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:04:01.141401+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:30:09.569579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":819,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.553979+0000","last_change":"2024-08-17T17:20:00.434774+0000","last_active":"2024-08-17T17:31:08.553979+0000","last_peered":"2024-08-17T17:31:08.553979+0000","last_clean":"2024-08-17T17:31:08.553979+0000","last_became_active":"2024-08-17T17:20:00.434343+0000","last_became_peered":"2024-08-17T17:20:00.434343+0000","last_unstale":"2024-08-17T17:31:08.553979+0000","last_undegraded":"2024-08-17T17:31:08.553979+0000","last_fullsized":"2024-08-17T17:31:08.553979+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:04.995152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":323,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.606233+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:31:13.606233+0000","last_peered":"2024-08-17T17:31:13.606233+0000","last_clean":"2024-08-17T17:31:13.606233+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:31:13.606233+0000","last_undegraded":"2024-08-17T17:31:13.606233+0000","last_fullsized":"2024-08-17T17:31:13.606233+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":59,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.559844+0000","last_change":"2024-08-17T17:25:04.579426+0000","last_active":"2024-08-17T17:31:08.559844+0000","last_peered":"2024-08-17T17:31:08.559844+0000","last_clean":"2024-08-17T17:31:08.559844+0000","last_became_active":"2024-08-17T17:25:04.579012+0000","last_became_peered":"2024-08-17T17:25:04.579012+0000","last_unstale":"2024-08-17T17:31:08.559844+0000","last_undegraded":"2024-08-17T17:31:08.559844+0000","last_fullsized":"2024-08-17T17:31:08.559844+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:56:32.241321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"247'10605","reported_seq":10634,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:24.095005+0000","last_change":"2024-08-17T17:28:27.506760+0000","last_active":"2024-08-17T17:31:24.095005+0000","last_peered":"2024-08-17T17:31:24.095005+0000","last_clean":"2024-08-17T17:31:24.095005+0000","last_became_active":"2024-08-17T17:28:27.506306+0000","last_became_peered":"2024-08-17T17:28:27.506306+0000","last_unstale":"2024-08-17T17:31:24.095005+0000","last_undegraded":"2024-08-17T17:31:24.095005+0000","last_fullsized":"2024-08-17T17:31:24.095005+0000","mapping_epoch":226,"log_start":"233'3100","ondisk_log_start":"233'3100","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":7505,"log_dups_size":0,"ondisk_log_size":7505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:41:13.520504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234029056,"num_objects":3571,"num_object_clones":0,"num_object_copies":10713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10605,"num_write_kb":453632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":837,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.555094+0000","last_change":"2024-08-17T17:16:26.958415+0000","last_active":"2024-08-17T17:31:08.555094+0000","last_peered":"2024-08-17T17:31:08.555094+0000","last_clean":"2024-08-17T17:31:08.555094+0000","last_became_active":"2024-08-17T17:16:26.957956+0000","last_became_peered":"2024-08-17T17:16:26.957956+0000","last_unstale":"2024-08-17T17:31:08.555094+0000","last_undegraded":"2024-08-17T17:31:08.555094+0000","last_fullsized":"2024-08-17T17:31:08.555094+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:30:22.478077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":352,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.602441+0000","last_change":"2024-08-17T17:18:59.415331+0000","last_active":"2024-08-17T17:31:13.602441+0000","last_peered":"2024-08-17T17:31:13.602441+0000","last_clean":"2024-08-17T17:31:13.602441+0000","last_became_active":"2024-08-17T17:18:59.415008+0000","last_became_peered":"2024-08-17T17:18:59.415008+0000","last_unstale":"2024-08-17T17:31:13.602441+0000","last_undegraded":"2024-08-17T17:31:13.602441+0000","last_fullsized":"2024-08-17T17:31:13.602441+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:22:38.076866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":161,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.599900+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:31:13.599900+0000","last_peered":"2024-08-17T17:31:13.599900+0000","last_clean":"2024-08-17T17:31:13.599900+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:31:13.599900+0000","last_undegraded":"2024-08-17T17:31:13.599900+0000","last_fullsized":"2024-08-17T17:31:13.599900+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"247'10531","reported_seq":10584,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:23.450885+0000","last_change":"2024-08-17T17:28:27.509856+0000","last_active":"2024-08-17T17:31:23.450885+0000","last_peered":"2024-08-17T17:31:23.450885+0000","last_clean":"2024-08-17T17:31:23.450885+0000","last_became_active":"2024-08-17T17:28:27.509524+0000","last_became_peered":"2024-08-17T17:28:27.509524+0000","last_unstale":"2024-08-17T17:31:23.450885+0000","last_undegraded":"2024-08-17T17:31:23.450885+0000","last_fullsized":"2024-08-17T17:31:23.450885+0000","mapping_epoch":226,"log_start":"233'3200","ondisk_log_start":"233'3200","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":7331,"log_dups_size":0,"ondisk_log_size":7331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:02:26.063259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230096896,"num_objects":3511,"num_object_clones":0,"num_object_copies":10533,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10531,"num_write_kb":449344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":740,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.608050+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:31:13.608050+0000","last_peered":"2024-08-17T17:31:13.608050+0000","last_clean":"2024-08-17T17:31:13.608050+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:31:13.608050+0000","last_undegraded":"2024-08-17T17:31:13.608050+0000","last_fullsized":"2024-08-17T17:31:13.608050+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":348,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.605373+0000","last_change":"2024-08-17T17:19:57.515564+0000","last_active":"2024-08-17T17:31:13.605373+0000","last_peered":"2024-08-17T17:31:13.605373+0000","last_clean":"2024-08-17T17:31:13.605373+0000","last_became_active":"2024-08-17T17:19:57.515142+0000","last_became_peered":"2024-08-17T17:19:57.515142+0000","last_unstale":"2024-08-17T17:31:13.605373+0000","last_undegraded":"2024-08-17T17:31:13.605373+0000","last_fullsized":"2024-08-17T17:31:13.605373+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:59:33.506176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":378,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.605662+0000","last_change":"2024-08-17T17:24:06.587903+0000","last_active":"2024-08-17T17:31:13.605662+0000","last_peered":"2024-08-17T17:31:13.605662+0000","last_clean":"2024-08-17T17:31:13.605662+0000","last_became_active":"2024-08-17T17:24:06.587503+0000","last_became_peered":"2024-08-17T17:24:06.587503+0000","last_unstale":"2024-08-17T17:31:13.605662+0000","last_undegraded":"2024-08-17T17:31:13.605662+0000","last_fullsized":"2024-08-17T17:31:13.605662+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:05:58.591992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"247'10482","reported_seq":10511,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:23.976036+0000","last_change":"2024-08-17T17:28:27.511940+0000","last_active":"2024-08-17T17:31:23.976036+0000","last_peered":"2024-08-17T17:31:23.976036+0000","last_clean":"2024-08-17T17:31:23.976036+0000","last_became_active":"2024-08-17T17:28:27.511636+0000","last_became_peered":"2024-08-17T17:28:27.511636+0000","last_unstale":"2024-08-17T17:31:23.976036+0000","last_undegraded":"2024-08-17T17:31:23.976036+0000","last_fullsized":"2024-08-17T17:31:23.976036+0000","mapping_epoch":226,"log_start":"232'2700","ondisk_log_start":"232'2700","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":7782,"log_dups_size":0,"ondisk_log_size":7782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:51:38.705815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236060672,"num_objects":3602,"num_object_clones":0,"num_object_copies":10806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10482,"num_write_kb":450688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":1157,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.553643+0000","last_change":"2024-08-17T17:20:00.433870+0000","last_active":"2024-08-17T17:31:08.553643+0000","last_peered":"2024-08-17T17:31:08.553643+0000","last_clean":"2024-08-17T17:31:08.553643+0000","last_became_active":"2024-08-17T17:20:00.433386+0000","last_became_peered":"2024-08-17T17:20:00.433386+0000","last_unstale":"2024-08-17T17:31:08.553643+0000","last_undegraded":"2024-08-17T17:31:08.553643+0000","last_fullsized":"2024-08-17T17:31:08.553643+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:01:11.848862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1392,"num_read_kb":2863,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":274,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.598055+0000","last_change":"2024-08-17T17:20:49.272407+0000","last_active":"2024-08-17T17:31:13.598055+0000","last_peered":"2024-08-17T17:31:13.598055+0000","last_clean":"2024-08-17T17:31:13.598055+0000","last_became_active":"2024-08-17T17:20:49.271749+0000","last_became_peered":"2024-08-17T17:20:49.271749+0000","last_unstale":"2024-08-17T17:31:13.598055+0000","last_undegraded":"2024-08-17T17:31:13.598055+0000","last_fullsized":"2024-08-17T17:31:13.598055+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:36:01.443457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":345,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.605403+0000","last_change":"2024-08-17T17:17:32.499715+0000","last_active":"2024-08-17T17:31:13.605403+0000","last_peered":"2024-08-17T17:31:13.605403+0000","last_clean":"2024-08-17T17:31:13.605403+0000","last_became_active":"2024-08-17T17:17:32.499412+0000","last_became_peered":"2024-08-17T17:17:32.499412+0000","last_unstale":"2024-08-17T17:31:13.605403+0000","last_undegraded":"2024-08-17T17:31:13.605403+0000","last_fullsized":"2024-08-17T17:31:13.605403+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:39:38.456970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"247'10737","reported_seq":10791,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:27.481120+0000","last_change":"2024-08-17T17:28:27.510443+0000","last_active":"2024-08-17T17:31:27.481120+0000","last_peered":"2024-08-17T17:31:27.481120+0000","last_clean":"2024-08-17T17:31:27.481120+0000","last_became_active":"2024-08-17T17:28:27.510074+0000","last_became_peered":"2024-08-17T17:28:27.510074+0000","last_unstale":"2024-08-17T17:31:27.481120+0000","last_undegraded":"2024-08-17T17:31:27.481120+0000","last_fullsized":"2024-08-17T17:31:27.481120+0000","mapping_epoch":226,"log_start":"234'3900","ondisk_log_start":"234'3900","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":6837,"log_dups_size":0,"ondisk_log_size":6837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:09:30.294320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220004352,"num_objects":3357,"num_object_clones":0,"num_object_copies":10071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10737,"num_write_kb":451008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":222,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.553788+0000","last_change":"2024-08-17T17:17:32.495728+0000","last_active":"2024-08-17T17:31:08.553788+0000","last_peered":"2024-08-17T17:31:08.553788+0000","last_clean":"2024-08-17T17:31:08.553788+0000","last_became_active":"2024-08-17T17:17:32.495357+0000","last_became_peered":"2024-08-17T17:17:32.495357+0000","last_unstale":"2024-08-17T17:31:08.553788+0000","last_undegraded":"2024-08-17T17:31:08.553788+0000","last_fullsized":"2024-08-17T17:31:08.553788+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:27:45.561567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":581,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.605595+0000","last_change":"2024-08-17T17:16:26.964621+0000","last_active":"2024-08-17T17:31:13.605595+0000","last_peered":"2024-08-17T17:31:13.605595+0000","last_clean":"2024-08-17T17:31:13.605595+0000","last_became_active":"2024-08-17T17:16:26.964058+0000","last_became_peered":"2024-08-17T17:16:26.964058+0000","last_unstale":"2024-08-17T17:31:13.605595+0000","last_undegraded":"2024-08-17T17:31:13.605595+0000","last_fullsized":"2024-08-17T17:31:13.605595+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:00:04.465644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":351,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.623105+0000","last_change":"2024-08-17T17:24:53.465550+0000","last_active":"2024-08-17T17:31:13.623105+0000","last_peered":"2024-08-17T17:31:13.623105+0000","last_clean":"2024-08-17T17:31:13.623105+0000","last_became_active":"2024-08-17T17:24:53.465228+0000","last_became_peered":"2024-08-17T17:24:53.465228+0000","last_unstale":"2024-08-17T17:31:13.623105+0000","last_undegraded":"2024-08-17T17:31:13.623105+0000","last_fullsized":"2024-08-17T17:31:13.623105+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:21:18.378359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.0","version":"247'10606","reported_seq":10635,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:23.838362+0000","last_change":"2024-08-17T17:28:27.509315+0000","last_active":"2024-08-17T17:31:23.838362+0000","last_peered":"2024-08-17T17:31:23.838362+0000","last_clean":"2024-08-17T17:31:23.838362+0000","last_became_active":"2024-08-17T17:28:27.509018+0000","last_became_peered":"2024-08-17T17:28:27.509018+0000","last_unstale":"2024-08-17T17:31:23.838362+0000","last_undegraded":"2024-08-17T17:31:23.838362+0000","last_fullsized":"2024-08-17T17:31:23.838362+0000","mapping_epoch":226,"log_start":"232'600","ondisk_log_start":"232'600","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:26:22.379668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231473152,"num_objects":3532,"num_object_clones":0,"num_object_copies":10596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10606,"num_write_kb":452416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":376,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.605495+0000","last_change":"2024-08-17T17:24:53.464307+0000","last_active":"2024-08-17T17:31:13.605495+0000","last_peered":"2024-08-17T17:31:13.605495+0000","last_clean":"2024-08-17T17:31:13.605495+0000","last_became_active":"2024-08-17T17:24:53.463996+0000","last_became_peered":"2024-08-17T17:24:53.463996+0000","last_unstale":"2024-08-17T17:31:13.605495+0000","last_undegraded":"2024-08-17T17:31:13.605495+0000","last_fullsized":"2024-08-17T17:31:13.605495+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:04:18.993725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":674,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.555436+0000","last_change":"2024-08-17T17:24:09.599822+0000","last_active":"2024-08-17T17:31:08.555436+0000","last_peered":"2024-08-17T17:31:08.555436+0000","last_clean":"2024-08-17T17:31:08.555436+0000","last_became_active":"2024-08-17T17:24:09.599189+0000","last_became_peered":"2024-08-17T17:24:09.599189+0000","last_unstale":"2024-08-17T17:31:08.555436+0000","last_undegraded":"2024-08-17T17:31:08.555436+0000","last_fullsized":"2024-08-17T17:31:08.555436+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:04.465604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":335,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.598099+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:31:13.598099+0000","last_peered":"2024-08-17T17:31:13.598099+0000","last_clean":"2024-08-17T17:31:13.598099+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:31:13.598099+0000","last_undegraded":"2024-08-17T17:31:13.598099+0000","last_fullsized":"2024-08-17T17:31:13.598099+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"247'11076","reported_seq":11130,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:27.530801+0000","last_change":"2024-08-17T17:28:27.509838+0000","last_active":"2024-08-17T17:31:27.530801+0000","last_peered":"2024-08-17T17:31:27.530801+0000","last_clean":"2024-08-17T17:31:27.530801+0000","last_became_active":"2024-08-17T17:28:27.509400+0000","last_became_peered":"2024-08-17T17:28:27.509400+0000","last_unstale":"2024-08-17T17:31:27.530801+0000","last_undegraded":"2024-08-17T17:31:27.530801+0000","last_fullsized":"2024-08-17T17:31:27.530801+0000","mapping_epoch":226,"log_start":"234'4200","ondisk_log_start":"234'4200","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":6876,"log_dups_size":0,"ondisk_log_size":6876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:57:28.933136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227803136,"num_objects":3476,"num_object_clones":0,"num_object_copies":10428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11076,"num_write_kb":465664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":370,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.616965+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:31:13.616965+0000","last_peered":"2024-08-17T17:31:13.616965+0000","last_clean":"2024-08-17T17:31:13.616965+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:31:13.616965+0000","last_undegraded":"2024-08-17T17:31:13.616965+0000","last_fullsized":"2024-08-17T17:31:13.616965+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":889,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.606095+0000","last_change":"2024-08-17T17:23:19.633786+0000","last_active":"2024-08-17T17:31:13.606095+0000","last_peered":"2024-08-17T17:31:13.606095+0000","last_clean":"2024-08-17T17:31:13.606095+0000","last_became_active":"2024-08-17T17:23:19.633230+0000","last_became_peered":"2024-08-17T17:23:19.633230+0000","last_unstale":"2024-08-17T17:31:13.606095+0000","last_undegraded":"2024-08-17T17:31:13.606095+0000","last_fullsized":"2024-08-17T17:31:13.606095+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:30:50.377391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":366,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.616706+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:31:13.616706+0000","last_peered":"2024-08-17T17:31:13.616706+0000","last_clean":"2024-08-17T17:31:13.616706+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:31:13.616706+0000","last_undegraded":"2024-08-17T17:31:13.616706+0000","last_fullsized":"2024-08-17T17:31:13.616706+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"247'10576","reported_seq":10630,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:27.394165+0000","last_change":"2024-08-17T17:28:27.509972+0000","last_active":"2024-08-17T17:31:27.394165+0000","last_peered":"2024-08-17T17:31:27.394165+0000","last_clean":"2024-08-17T17:31:27.394165+0000","last_became_active":"2024-08-17T17:28:27.509668+0000","last_became_peered":"2024-08-17T17:28:27.509668+0000","last_unstale":"2024-08-17T17:31:27.394165+0000","last_undegraded":"2024-08-17T17:31:27.394165+0000","last_fullsized":"2024-08-17T17:31:27.394165+0000","mapping_epoch":226,"log_start":"233'3700","ondisk_log_start":"233'3700","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:28:26.486401+0000","last_clean_scrub_stamp":"2024-08-17T17:28:26.486401+0000","objects_scrubbed":0,"log_size":6876,"log_dups_size":0,"ondisk_log_size":6876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:54:43.175100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214171648,"num_objects":3268,"num_object_clones":0,"num_object_copies":9804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3268,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10576,"num_write_kb":443008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":165,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.599800+0000","last_change":"2024-08-17T17:24:06.593660+0000","last_active":"2024-08-17T17:31:13.599800+0000","last_peered":"2024-08-17T17:31:13.599800+0000","last_clean":"2024-08-17T17:31:13.599800+0000","last_became_active":"2024-08-17T17:24:06.586215+0000","last_became_peered":"2024-08-17T17:24:06.586215+0000","last_unstale":"2024-08-17T17:31:13.599800+0000","last_undegraded":"2024-08-17T17:31:13.599800+0000","last_fullsized":"2024-08-17T17:31:13.599800+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:15:49.475395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":478,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.560155+0000","last_change":"2024-08-17T17:26:06.855307+0000","last_active":"2024-08-17T17:31:08.560155+0000","last_peered":"2024-08-17T17:31:08.560155+0000","last_clean":"2024-08-17T17:31:08.560155+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:31:08.560155+0000","last_undegraded":"2024-08-17T17:31:08.560155+0000","last_fullsized":"2024-08-17T17:31:08.560155+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":146,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.553391+0000","last_change":"2024-08-17T17:18:59.417183+0000","last_active":"2024-08-17T17:31:08.553391+0000","last_peered":"2024-08-17T17:31:08.553391+0000","last_clean":"2024-08-17T17:31:08.553391+0000","last_became_active":"2024-08-17T17:18:59.416843+0000","last_became_peered":"2024-08-17T17:18:59.416843+0000","last_unstale":"2024-08-17T17:31:08.553391+0000","last_undegraded":"2024-08-17T17:31:08.553391+0000","last_fullsized":"2024-08-17T17:31:08.553391+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:29:09.336335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":167,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.561709+0000","last_change":"2024-08-17T17:20:49.258048+0000","last_active":"2024-08-17T17:31:08.561709+0000","last_peered":"2024-08-17T17:31:08.561709+0000","last_clean":"2024-08-17T17:31:08.561709+0000","last_became_active":"2024-08-17T17:20:49.257672+0000","last_became_peered":"2024-08-17T17:20:49.257672+0000","last_unstale":"2024-08-17T17:31:08.561709+0000","last_undegraded":"2024-08-17T17:31:08.561709+0000","last_fullsized":"2024-08-17T17:31:08.561709+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:08:11.734316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":206,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.555165+0000","last_change":"2024-08-17T17:18:59.430969+0000","last_active":"2024-08-17T17:31:08.555165+0000","last_peered":"2024-08-17T17:31:08.555165+0000","last_clean":"2024-08-17T17:31:08.555165+0000","last_became_active":"2024-08-17T17:18:59.413475+0000","last_became_peered":"2024-08-17T17:18:59.413475+0000","last_unstale":"2024-08-17T17:31:08.555165+0000","last_undegraded":"2024-08-17T17:31:08.555165+0000","last_fullsized":"2024-08-17T17:31:08.555165+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:36:33.732645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":361,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.622830+0000","last_change":"2024-08-17T17:25:04.506413+0000","last_active":"2024-08-17T17:31:13.622830+0000","last_peered":"2024-08-17T17:31:13.622830+0000","last_clean":"2024-08-17T17:31:13.622830+0000","last_became_active":"2024-08-17T17:25:04.506104+0000","last_became_peered":"2024-08-17T17:25:04.506104+0000","last_unstale":"2024-08-17T17:31:13.622830+0000","last_undegraded":"2024-08-17T17:31:13.622830+0000","last_fullsized":"2024-08-17T17:31:13.622830+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:41:53.556223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":149,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.555239+0000","last_change":"2024-08-17T17:20:49.279458+0000","last_active":"2024-08-17T17:31:08.555239+0000","last_peered":"2024-08-17T17:31:08.555239+0000","last_clean":"2024-08-17T17:31:08.555239+0000","last_became_active":"2024-08-17T17:20:49.260790+0000","last_became_peered":"2024-08-17T17:20:49.260790+0000","last_unstale":"2024-08-17T17:31:08.555239+0000","last_undegraded":"2024-08-17T17:31:08.555239+0000","last_fullsized":"2024-08-17T17:31:08.555239+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:58.092945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":63,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.553754+0000","last_change":"2024-08-17T17:24:53.463713+0000","last_active":"2024-08-17T17:31:08.553754+0000","last_peered":"2024-08-17T17:31:08.553754+0000","last_clean":"2024-08-17T17:31:08.553754+0000","last_became_active":"2024-08-17T17:24:53.463419+0000","last_became_peered":"2024-08-17T17:24:53.463419+0000","last_unstale":"2024-08-17T17:31:08.553754+0000","last_undegraded":"2024-08-17T17:31:08.553754+0000","last_fullsized":"2024-08-17T17:31:08.553754+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:34:44.545175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":316,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.607802+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:31:13.607802+0000","last_peered":"2024-08-17T17:31:13.607802+0000","last_clean":"2024-08-17T17:31:13.607802+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:31:13.607802+0000","last_undegraded":"2024-08-17T17:31:13.607802+0000","last_fullsized":"2024-08-17T17:31:13.607802+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":344,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.600291+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:31:13.600291+0000","last_peered":"2024-08-17T17:31:13.600291+0000","last_clean":"2024-08-17T17:31:13.600291+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:31:13.600291+0000","last_undegraded":"2024-08-17T17:31:13.600291+0000","last_fullsized":"2024-08-17T17:31:13.600291+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":363,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.608237+0000","last_change":"2024-08-17T17:25:04.509422+0000","last_active":"2024-08-17T17:31:13.608237+0000","last_peered":"2024-08-17T17:31:13.608237+0000","last_clean":"2024-08-17T17:31:13.608237+0000","last_became_active":"2024-08-17T17:25:04.508742+0000","last_became_peered":"2024-08-17T17:25:04.508742+0000","last_unstale":"2024-08-17T17:31:13.608237+0000","last_undegraded":"2024-08-17T17:31:13.608237+0000","last_fullsized":"2024-08-17T17:31:13.608237+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:18:31.976907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":204,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.560129+0000","last_change":"2024-08-17T17:16:26.963696+0000","last_active":"2024-08-17T17:31:08.560129+0000","last_peered":"2024-08-17T17:31:08.560129+0000","last_clean":"2024-08-17T17:31:08.560129+0000","last_became_active":"2024-08-17T17:16:26.963280+0000","last_became_peered":"2024-08-17T17:16:26.963280+0000","last_unstale":"2024-08-17T17:31:08.560129+0000","last_undegraded":"2024-08-17T17:31:08.560129+0000","last_fullsized":"2024-08-17T17:31:08.560129+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:22:01.474840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":381,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.616809+0000","last_change":"2024-08-17T17:24:06.586886+0000","last_active":"2024-08-17T17:31:13.616809+0000","last_peered":"2024-08-17T17:31:13.616809+0000","last_clean":"2024-08-17T17:31:13.616809+0000","last_became_active":"2024-08-17T17:24:06.586516+0000","last_became_peered":"2024-08-17T17:24:06.586516+0000","last_unstale":"2024-08-17T17:31:13.616809+0000","last_undegraded":"2024-08-17T17:31:13.616809+0000","last_fullsized":"2024-08-17T17:31:13.616809+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:03:30.002726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":161,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.599638+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:31:13.599638+0000","last_peered":"2024-08-17T17:31:13.599638+0000","last_clean":"2024-08-17T17:31:13.599638+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:31:13.599638+0000","last_undegraded":"2024-08-17T17:31:13.599638+0000","last_fullsized":"2024-08-17T17:31:13.599638+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":341,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.631882+0000","last_change":"2024-08-17T17:18:59.412463+0000","last_active":"2024-08-17T17:31:13.631882+0000","last_peered":"2024-08-17T17:31:13.631882+0000","last_clean":"2024-08-17T17:31:13.631882+0000","last_became_active":"2024-08-17T17:18:59.412134+0000","last_became_peered":"2024-08-17T17:18:59.412134+0000","last_unstale":"2024-08-17T17:31:13.631882+0000","last_undegraded":"2024-08-17T17:31:13.631882+0000","last_fullsized":"2024-08-17T17:31:13.631882+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:13:44.463153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":26,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.553535+0000","last_change":"2024-08-17T17:28:41.399059+0000","last_active":"2024-08-17T17:31:08.553535+0000","last_peered":"2024-08-17T17:31:08.553535+0000","last_clean":"2024-08-17T17:31:08.553535+0000","last_became_active":"2024-08-17T17:28:41.398560+0000","last_became_peered":"2024-08-17T17:28:41.398560+0000","last_unstale":"2024-08-17T17:31:08.553535+0000","last_undegraded":"2024-08-17T17:31:08.553535+0000","last_fullsized":"2024-08-17T17:31:08.553535+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:31:45.097451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":281,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.623203+0000","last_change":"2024-08-17T17:24:06.591165+0000","last_active":"2024-08-17T17:31:13.623203+0000","last_peered":"2024-08-17T17:31:13.623203+0000","last_clean":"2024-08-17T17:31:13.623203+0000","last_became_active":"2024-08-17T17:24:06.590785+0000","last_became_peered":"2024-08-17T17:24:06.590785+0000","last_unstale":"2024-08-17T17:31:13.623203+0000","last_undegraded":"2024-08-17T17:31:13.623203+0000","last_fullsized":"2024-08-17T17:31:13.623203+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:54:05.775563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":353,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.605782+0000","last_change":"2024-08-17T17:24:32.300560+0000","last_active":"2024-08-17T17:31:13.605782+0000","last_peered":"2024-08-17T17:31:13.605782+0000","last_clean":"2024-08-17T17:31:13.605782+0000","last_became_active":"2024-08-17T17:24:32.300228+0000","last_became_peered":"2024-08-17T17:24:32.300228+0000","last_unstale":"2024-08-17T17:31:13.605782+0000","last_undegraded":"2024-08-17T17:31:13.605782+0000","last_fullsized":"2024-08-17T17:31:13.605782+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:18:05.475449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":160,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.597921+0000","last_change":"2024-08-17T17:24:06.586379+0000","last_active":"2024-08-17T17:31:13.597921+0000","last_peered":"2024-08-17T17:31:13.597921+0000","last_clean":"2024-08-17T17:31:13.597921+0000","last_became_active":"2024-08-17T17:24:06.586023+0000","last_became_peered":"2024-08-17T17:24:06.586023+0000","last_unstale":"2024-08-17T17:31:13.597921+0000","last_undegraded":"2024-08-17T17:31:13.597921+0000","last_fullsized":"2024-08-17T17:31:13.597921+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:45.701637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":370,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.612380+0000","last_change":"2024-08-17T17:21:50.442535+0000","last_active":"2024-08-17T17:31:13.612380+0000","last_peered":"2024-08-17T17:31:13.612380+0000","last_clean":"2024-08-17T17:31:13.612380+0000","last_became_active":"2024-08-17T17:21:50.441891+0000","last_became_peered":"2024-08-17T17:21:50.441891+0000","last_unstale":"2024-08-17T17:31:13.612380+0000","last_undegraded":"2024-08-17T17:31:13.612380+0000","last_fullsized":"2024-08-17T17:31:13.612380+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:46:24.025407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":259,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.553417+0000","last_change":"2024-08-17T17:25:04.508045+0000","last_active":"2024-08-17T17:31:08.553417+0000","last_peered":"2024-08-17T17:31:08.553417+0000","last_clean":"2024-08-17T17:31:08.553417+0000","last_became_active":"2024-08-17T17:25:04.505083+0000","last_became_peered":"2024-08-17T17:25:04.505083+0000","last_unstale":"2024-08-17T17:31:08.553417+0000","last_undegraded":"2024-08-17T17:31:08.553417+0000","last_fullsized":"2024-08-17T17:31:08.553417+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:23:42.993544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":356,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.606388+0000","last_change":"2024-08-17T17:18:59.418434+0000","last_active":"2024-08-17T17:31:13.606388+0000","last_peered":"2024-08-17T17:31:13.606388+0000","last_clean":"2024-08-17T17:31:13.606388+0000","last_became_active":"2024-08-17T17:18:59.418043+0000","last_became_peered":"2024-08-17T17:18:59.418043+0000","last_unstale":"2024-08-17T17:31:13.606388+0000","last_undegraded":"2024-08-17T17:31:13.606388+0000","last_fullsized":"2024-08-17T17:31:13.606388+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:25:51.371820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":232,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.553713+0000","last_change":"2024-08-17T17:20:49.291167+0000","last_active":"2024-08-17T17:31:08.553713+0000","last_peered":"2024-08-17T17:31:08.553713+0000","last_clean":"2024-08-17T17:31:08.553713+0000","last_became_active":"2024-08-17T17:20:49.263750+0000","last_became_peered":"2024-08-17T17:20:49.263750+0000","last_unstale":"2024-08-17T17:31:08.553713+0000","last_undegraded":"2024-08-17T17:31:08.553713+0000","last_fullsized":"2024-08-17T17:31:08.553713+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:02.863246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":370,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.608137+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:31:13.608137+0000","last_peered":"2024-08-17T17:31:13.608137+0000","last_clean":"2024-08-17T17:31:13.608137+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:31:13.608137+0000","last_undegraded":"2024-08-17T17:31:13.608137+0000","last_fullsized":"2024-08-17T17:31:13.608137+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":370,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.606582+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:31:13.606582+0000","last_peered":"2024-08-17T17:31:13.606582+0000","last_clean":"2024-08-17T17:31:13.606582+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:31:13.606582+0000","last_undegraded":"2024-08-17T17:31:13.606582+0000","last_fullsized":"2024-08-17T17:31:13.606582+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":59,"reported_epoch":246,"state":"active+clean","last_fresh":"2024-08-17T17:31:08.559345+0000","last_change":"2024-08-17T17:25:04.503834+0000","last_active":"2024-08-17T17:31:08.559345+0000","last_peered":"2024-08-17T17:31:08.559345+0000","last_clean":"2024-08-17T17:31:08.559345+0000","last_became_active":"2024-08-17T17:25:04.502712+0000","last_became_peered":"2024-08-17T17:25:04.502712+0000","last_unstale":"2024-08-17T17:31:08.559345+0000","last_undegraded":"2024-08-17T17:31:08.559345+0000","last_fullsized":"2024-08-17T17:31:08.559345+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:45.389103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":180,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-08-17T17:31:13.600190+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:31:13.600190+0000","last_peered":"2024-08-17T17:31:13.600190+0000","last_clean":"2024-08-17T17:31:13.600190+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:31:13.600190+0000","last_undegraded":"2024-08-17T17:31:13.600190+0000","last_fullsized":"2024-08-17T17:31:13.600190+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3671982104,"num_objects":56031,"num_object_clones":0,"num_object_copies":168093,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56031,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170409,"num_write_kb":7246017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11010912256,"data_stored":11010900040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121909,"ondisk_log_size":121909,"up":48,"acting":48,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":6304,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":16702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17971,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234295,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1702636,"kb_used_data":1392804,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":92042516,"statfs":{"total":95995035648,"available":94251536384,"internally_reserved":0,"allocated":1426231296,"data_stored":1424846138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495930,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1919144,"kb_used_data":1621792,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":91826008,"statfs":{"total":95995035648,"available":94029832192,"internally_reserved":0,"allocated":1660715008,"data_stored":1659333670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757567,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1206040,"kb_used_data":908688,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":92539112,"statfs":{"total":95995035648,"available":94760050688,"internally_reserved":0,"allocated":930496512,"data_stored":929126989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019205,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435433,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2622740,"kb_used_data":2166412,"kb_used_omap":4,"kb_used_meta":456251,"kb_avail":91122412,"statfs":{"total":95995035648,"available":93309349888,"internally_reserved":0,"allocated":2218405888,"data_stored":2217040235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":467201034},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,1,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":104,"seq":446676598959,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1788644,"kb_used_data":1350300,"kb_used_omap":7,"kb_used_meta":438328,"kb_avail":91956508,"statfs":{"total":95995035648,"available":94163464192,"internally_reserved":0,"allocated":1382707200,"data_stored":1381330429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":448848034},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":99,"seq":425201762486,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2040732,"kb_used_data":1583636,"kb_used_omap":7,"kb_used_meta":457080,"kb_avail":91704420,"statfs":{"total":95995035648,"available":93905326080,"internally_reserved":0,"allocated":1621643264,"data_stored":1620269410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7945,"internal_metadata":468050167},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926012,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2246180,"kb_used_data":1831708,"kb_used_omap":7,"kb_used_meta":414392,"kb_avail":91498972,"statfs":{"total":95995035648,"available":93694947328,"internally_reserved":0,"allocated":1875668992,"data_stored":1874288131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":424337729},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6241,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1867059200,"data_stored":1867055128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1605832704,"data_stored":1605828632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1372389376,"data_stored":1372389376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2209021952,"data_stored":2209021952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":921829376,"data_stored":921829376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1633226752,"data_stored":1633222680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1401552896,"data_stored":1401552896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:31:32.777 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-08-17T17:31:32.777 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:31:32.778 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-08-17T17:31:32.778 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-08-17T17:31:32.778 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.1 2024-08-17T17:31:32.861 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:32.886 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:32.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:32.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:32.988 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:33.017 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:33.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:33.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:33.038 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-17T17:31:33.119 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:33.170 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:31:32 smithi003 systemd[1]: Stopping Ceph osd.1 for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:31:33.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:33.171 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:33.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:33.272 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:33.302 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:33.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:33.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:33.404 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:33.435 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:33.435 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:31:33.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:33.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:33.536 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:33.568 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:33.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:33.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:33.670 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:33.683 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-08-17T17:31:33.698 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:33.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:33.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:33.783 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:33 smithi053 ceph-mon[89963]: pgmap v727: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 651 op/s 2024-08-17T17:31:33.783 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:33 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1770869660' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:31:33.783 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:33 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:31:33.799 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:33.826 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:33.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:33.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:33.928 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:33.983 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:33.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:33.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:34.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:33 smithi003 ceph-mon[103344]: pgmap v727: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 651 op/s 2024-08-17T17:31:34.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:33 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1770869660' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:31:34.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:33 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:31:34.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:33 smithi003 ceph-mon[100615]: pgmap v727: 89 pgs: 89 active+clean; 3.2 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 651 op/s 2024-08-17T17:31:34.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:33 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1770869660' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:31:34.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:33 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:31:34.024 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:31:33 smithi003 podman[219775]: 2024-08-17 17:31:33.727358876 +0000 UTC m=+0.813599733 container died b972a4bb61062914caa7f27e17a3ff0b2925da0fa9ec41a22772d3dcb10927df (image=quay.ceph.io/ceph-ci/ceph:quincy, name=nice_chandrasekhar, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-f2c39d4, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, ceph=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-08-17T17:31:34.085 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:34.116 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:34.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:34.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:34.217 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:34.261 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:34.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:34.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:34.363 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:34.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:34.417 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:34.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:34.518 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:34.545 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:34.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:34.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:34.647 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:34.698 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:34.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:34.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:34.800 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:34.855 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:34.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:34.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:34.897 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:34 smithi003 ceph-mon[100615]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:31:34.897 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:34 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:31:34.897 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:34 smithi003 ceph-mon[100615]: osdmap e248: 8 total, 8 up, 7 in 2024-08-17T17:31:34.897 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:34 smithi003 ceph-mon[103344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:31:34.898 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:34 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:31:34.898 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:34 smithi003 ceph-mon[103344]: osdmap e248: 8 total, 8 up, 7 in 2024-08-17T17:31:34.957 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:34.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:34 smithi053 ceph-mon[89963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:31:34.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:34 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:31:34.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:34 smithi053 ceph-mon[89963]: osdmap e248: 8 total, 8 up, 7 in 2024-08-17T17:31:34.984 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:34.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:34.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:35.086 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:35.136 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:35.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:35.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:35.238 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:35.264 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:35.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:35.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:35.366 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:35.431 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:35.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:35.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:35.452 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:31:35.452 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:31:35.533 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:35.563 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:35.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:35.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:35.643 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:31:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1[117117]: 2024-08-17T17:31:35.290+0000 7f22fbc8a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T17:31:35.643 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:31:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1[117117]: 2024-08-17T17:31:35.290+0000 7f22fbc8a640 -1 osd.1 248 *** Got signal Terminated *** 2024-08-17T17:31:35.643 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:31:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1[117117]: 2024-08-17T17:31:35.290+0000 7f22fbc8a640 -1 osd.1 248 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-17T17:31:35.665 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:35.723 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:35.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:35.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:35.739 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:31:35.826 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:35.848 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:35.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:35.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:35.950 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:35.972 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:35.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:35.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:36.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:31:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:31:35] "GET /metrics HTTP/1.1" 200 36504 "" "Prometheus/2.51.0" 2024-08-17T17:31:36.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:35 smithi003 ceph-mon[103344]: pgmap v729: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 764 op/s 2024-08-17T17:31:36.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:35 smithi003 ceph-mon[103344]: osd.1 marked itself down and dead 2024-08-17T17:31:36.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:35 smithi003 ceph-mon[100615]: pgmap v729: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 764 op/s 2024-08-17T17:31:36.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:35 smithi003 ceph-mon[100615]: osd.1 marked itself down and dead 2024-08-17T17:31:36.074 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:36.077 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:35 smithi053 ceph-mon[89963]: pgmap v729: 89 pgs: 89 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 764 op/s 2024-08-17T17:31:36.077 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:35 smithi053 ceph-mon[89963]: osd.1 marked itself down and dead 2024-08-17T17:31:36.096 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:36.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:36.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:36.104 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-17T17:31:36.155 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:31:36.199 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:36.221 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:36.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:36.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:36.323 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:36.345 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:36.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:36.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:36.395 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:31:36.409 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:31:36.447 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:36.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:36.471 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:36.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:36.572 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:36.594 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:36.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:36.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:36.696 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:36.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:36.727 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:36.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:36.739 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:31:36 smithi003 podman[219941]: 2024-08-17 17:31:36.389541982 +0000 UTC m=+1.596711521 container died f83a0fcf5c4c6ff21a36865486567ac416fc67b9ae9ae2ce72cdf52eebd9cfc3 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, RELEASE=main-cd8bdbe, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:31:36.828 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:36.837 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:36 smithi053 ceph-mon[89963]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T17:31:36.837 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:36 smithi053 ceph-mon[89963]: osdmap e249: 8 total, 7 up, 7 in 2024-08-17T17:31:36.837 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:36 smithi053 ceph-mon[89963]: osdmap e250: 8 total, 7 up, 7 in 2024-08-17T17:31:36.866 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:36.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:36.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:36.967 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:37.005 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:37.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:37.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:37.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:36 smithi003 ceph-mon[103344]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T17:31:37.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:36 smithi003 ceph-mon[103344]: osdmap e249: 8 total, 7 up, 7 in 2024-08-17T17:31:37.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:36 smithi003 ceph-mon[103344]: osdmap e250: 8 total, 7 up, 7 in 2024-08-17T17:31:37.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:36 smithi003 ceph-mon[100615]: Health check failed: 1 osds down (OSD_DOWN) 2024-08-17T17:31:37.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:36 smithi003 ceph-mon[100615]: osdmap e249: 8 total, 7 up, 7 in 2024-08-17T17:31:37.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:36 smithi003 ceph-mon[100615]: osdmap e250: 8 total, 7 up, 7 in 2024-08-17T17:31:37.107 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:37.135 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:37.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:37.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:37.237 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:37.267 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:37.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:37.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:37.368 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:37.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:37.404 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:37.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:37.441 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:31:37 smithi003 podman[219941]: 2024-08-17 17:31:37.060005572 +0000 UTC m=+2.267175113 container cleanup f83a0fcf5c4c6ff21a36865486567ac416fc67b9ae9ae2ce72cdf52eebd9cfc3 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-cd8bdbe, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-08-17T17:31:37.442 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:31:37 smithi003 bash[219941]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1 2024-08-17T17:31:37.442 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:31:37 smithi003 podman[220236]: 2024-08-17 17:31:37.286943326 +0000 UTC m=+0.883394246 container remove f83a0fcf5c4c6ff21a36865486567ac416fc67b9ae9ae2ce72cdf52eebd9cfc3 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-cd8bdbe, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-08-17T17:31:37.505 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:37.549 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:37.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:37.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:37.651 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:37.687 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:37.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:37.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:37.788 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:37.816 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:37.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:37.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:37.918 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:37.949 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:37 smithi053 ceph-mon[89963]: pgmap v731: 89 pgs: 15 stale+active+clean, 74 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 852 op/s 2024-08-17T17:31:37.955 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:37.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:37.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:38.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:37 smithi003 ceph-mon[100615]: pgmap v731: 89 pgs: 15 stale+active+clean, 74 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 852 op/s 2024-08-17T17:31:38.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:37 smithi003 ceph-mon[103344]: pgmap v731: 89 pgs: 15 stale+active+clean, 74 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 852 op/s 2024-08-17T17:31:38.057 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:38.093 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:38.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:38.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:38.194 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:38.228 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:31:38.228 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:31:38.229 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:38.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:38.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:38.331 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:38.358 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:38.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:38.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:38.461 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:38.489 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:38.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:38.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:38.586 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:31:38.591 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:38.650 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:38.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:38.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:38.752 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:38.803 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:38.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:38.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:38.905 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:38.921 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-17T17:31:38.984 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:38.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:38.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:39.086 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:39.112 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:39.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:39.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:39.214 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:39.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:39 smithi053 ceph-mon[89963]: pgmap v733: 89 pgs: 12 active+undersized, 10 stale+active+clean, 3 active+undersized+degraded, 64 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 782 op/s; 5730/145248 objects degraded (3.945%); 0 B/s, 0 objects/s recovering 2024-08-17T17:31:39.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:31:39.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:31:39.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:31:39.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:31:39.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:39 smithi053 ceph-mon[89963]: Health check failed: Degraded data redundancy: 5730/145248 objects degraded (3.945%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:31:39.241 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:39.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:39.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:39.277 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:39 smithi003 ceph-mon[100615]: pgmap v733: 89 pgs: 12 active+undersized, 10 stale+active+clean, 3 active+undersized+degraded, 64 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 782 op/s; 5730/145248 objects degraded (3.945%); 0 B/s, 0 objects/s recovering 2024-08-17T17:31:39.277 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:31:39.277 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:31:39.277 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:31:39.277 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:31:39.277 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:39 smithi003 ceph-mon[100615]: Health check failed: Degraded data redundancy: 5730/145248 objects degraded (3.945%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:31:39.277 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:39 smithi003 ceph-mon[103344]: pgmap v733: 89 pgs: 12 active+undersized, 10 stale+active+clean, 3 active+undersized+degraded, 64 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 782 op/s; 5730/145248 objects degraded (3.945%); 0 B/s, 0 objects/s recovering 2024-08-17T17:31:39.278 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:31:39.278 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:31:39.278 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:31:39.278 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:31:39.278 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:39 smithi003 ceph-mon[103344]: Health check failed: Degraded data redundancy: 5730/145248 objects degraded (3.945%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:31:39.342 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:39.371 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:39.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:39.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:39.473 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:39.500 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:39.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:39.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:39.602 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:39.636 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:39.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:39.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:39.738 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:39.784 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:39.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:39.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:39.886 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:39.912 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:39.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:39.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:40.014 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:40.045 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:40.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:40.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:40.147 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:40.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:40.186 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:40.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:40.287 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:40.319 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:40.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:40.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:40.421 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:40.450 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:40.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:40.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:40.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:31:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:31:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:31:40.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:40 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:31:40.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:40 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:31:40.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:40 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:31:40.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:40 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:31:40.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:40 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:31:40.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:40 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:31:40.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:40 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:31:40.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:40 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:31:40.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:40 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:31:40.553 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:40.580 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:40.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:40.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:40.681 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:40.711 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:40.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:40.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:40.813 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:40.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:40.865 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:40.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:40.946 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:31:40.946 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:31:40.949 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-17T17:31:40.967 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:40.991 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:31:40.991 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:31:41.010 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:41.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:41.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:41.033 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:31:41.033 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:31:41.066 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:31:41.066 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:31:41.096 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:31:41.096 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:31:41.111 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:41.135 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:31:41.136 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:31:41.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:41.161 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:41.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:41.169 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:31:41.169 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:31:41.262 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:41.291 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:41.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:41.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:41.393 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:41.420 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:41.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:41.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:41.522 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:41.559 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:41.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:41.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:41.661 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:41.688 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:41.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:41.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:41.731 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:41 smithi003 ceph-mon[103344]: pgmap v734: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 2.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 864 op/s; 19988/137139 objects degraded (14.575%) 2024-08-17T17:31:41.731 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:41 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1640582992' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:31:41.731 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:41 smithi003 ceph-mon[100615]: pgmap v734: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 2.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 864 op/s; 19988/137139 objects degraded (14.575%) 2024-08-17T17:31:41.731 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:41 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1640582992' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:31:41.790 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:41.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:41.828 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:41.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:41.929 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:41.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:41 smithi053 ceph-mon[89963]: pgmap v734: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 2.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 864 op/s; 19988/137139 objects degraded (14.575%) 2024-08-17T17:31:41.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:41 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1640582992' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:31:41.976 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:41.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:41.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:42.078 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:42.095 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-17T17:31:42.096 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-17T17:31:42.120 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:31:42.126 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:42.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:42.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:42.228 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:42.258 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:42.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:42.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:42.359 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:42.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:42.406 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:42.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:42.507 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:42.549 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:42.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:42.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:42.576 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-08-17T17:31:42.652 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:42.682 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:42.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:42.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:42.783 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:42.813 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:42.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:42.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:42.915 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:42.927 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:42 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/402987302' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:31:42.928 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:42 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/402987302' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:31:42.959 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:42.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:42.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:42.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:42 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/402987302' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:31:43.061 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:43.088 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:43.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:43.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:43.190 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:43.219 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:43.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:43.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:43.321 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:43.350 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:43.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:43.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:43.451 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:43.478 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:43.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:43.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:43.554 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:31:43 smithi003 podman[221117]: 2024-08-17 17:31:43.191175005 +0000 UTC m=+0.266733566 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:31:43.554 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:31:43 smithi003 podman[221117]: 2024-08-17 17:31:43.414269598 +0000 UTC m=+0.489828143 container create e95f105c89980babd975708416185d21da151e3d5e2fb289d7f6110936d4e35b (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=main-cd8bdbe, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-08-17T17:31:43.580 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:43.615 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:43.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:43.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:43.717 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:43.745 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:43.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:43.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:43.847 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:43.873 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:43.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:43.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:43.956 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:43 smithi003 ceph-mon[100615]: pgmap v735: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 2.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 653 op/s; 19988/137139 objects degraded (14.575%) 2024-08-17T17:31:43.956 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:43 smithi003 ceph-mon[100615]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:31:43.956 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:43 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/402987302' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:31:43.956 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:43 smithi003 ceph-mon[100615]: osdmap e251: 8 total, 7 up, 7 in 2024-08-17T17:31:43.957 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:43 smithi003 ceph-mon[103344]: pgmap v735: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 2.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 653 op/s; 19988/137139 objects degraded (14.575%) 2024-08-17T17:31:43.957 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:43 smithi003 ceph-mon[103344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:31:43.957 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:43 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/402987302' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:31:43.957 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:43 smithi003 ceph-mon[103344]: osdmap e251: 8 total, 7 up, 7 in 2024-08-17T17:31:43.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:43 smithi053 ceph-mon[89963]: pgmap v735: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 2.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 653 op/s; 19988/137139 objects degraded (14.575%) 2024-08-17T17:31:43.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:43 smithi053 ceph-mon[89963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:31:43.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:43 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/402987302' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:31:43.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:43 smithi053 ceph-mon[89963]: osdmap e251: 8 total, 7 up, 7 in 2024-08-17T17:31:43.974 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:44.001 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:44.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:44.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:44.097 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:31:44.103 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:44.181 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:44.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:44.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:44.283 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:44.308 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:44.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:44.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:44.410 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:44.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:44.457 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:44.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:44.558 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:44.617 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:44.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:44.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:44.719 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:44.748 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:44.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:44.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:44.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:44 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 18229/128226 objects degraded (14.216%), 14 pgs degraded (PG_DEGRADED) 2024-08-17T17:31:44.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:44 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 18229/128226 objects degraded (14.216%), 14 pgs degraded (PG_DEGRADED) 2024-08-17T17:31:44.774 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:31:44 smithi003 podman[221117]: 2024-08-17 17:31:44.692752292 +0000 UTC m=+1.768310835 container init e95f105c89980babd975708416185d21da151e3d5e2fb289d7f6110936d4e35b (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, RELEASE=main-cd8bdbe, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-17T17:31:44.774 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:31:44 smithi003 podman[221117]: 2024-08-17 17:31:44.700114568 +0000 UTC m=+1.775673118 container start e95f105c89980babd975708416185d21da151e3d5e2fb289d7f6110936d4e35b (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-deactivate, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-cd8bdbe, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-08-17T17:31:44.850 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:44.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:44.904 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:44.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:44.958 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:31:44.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:44 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 18229/128226 objects degraded (14.216%), 14 pgs degraded (PG_DEGRADED) 2024-08-17T17:31:45.005 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:45.031 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:45.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:45.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:45.127 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:31:44 smithi003 podman[221117]: 2024-08-17 17:31:44.859524119 +0000 UTC m=+1.935082662 container attach e95f105c89980babd975708416185d21da151e3d5e2fb289d7f6110936d4e35b (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-cd8bdbe, org.label-schema.schema-version=1.0) 2024-08-17T17:31:45.133 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:45.178 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:45.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:45.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:45.280 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:45.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:45.323 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:45.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:45.424 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:45.449 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:45.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:45.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:45.551 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:45.581 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:45.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:45.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:45.683 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:45.707 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:45.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:45.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:45.733 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:45 smithi003 ceph-mon[103344]: pgmap v737: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 812 op/s; 18229/128226 objects degraded (14.216%) 2024-08-17T17:31:45.734 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:45 smithi003 ceph-mon[100615]: pgmap v737: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 812 op/s; 18229/128226 objects degraded (14.216%) 2024-08-17T17:31:45.734 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:31:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:31:45] "GET /metrics HTTP/1.1" 200 36506 "" "Prometheus/2.51.0" 2024-08-17T17:31:45.809 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:45.860 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:45.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:45.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:45.961 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:45.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:45 smithi053 ceph-mon[89963]: pgmap v737: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 812 op/s; 18229/128226 objects degraded (14.216%) 2024-08-17T17:31:45.996 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:31:46.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:46.011 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:46.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:46.112 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:46.137 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:46.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:46.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:46.239 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:46.262 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:46.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:46.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:46.363 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:46.388 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:46.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:46.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:46.490 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:46.546 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:46.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:46.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:46.648 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:46.672 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:46.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:46.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:46.774 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:46.804 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:46.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:46.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:46.906 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:46.934 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:46.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:46.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:47.036 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:47.064 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:47.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:47.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:47.166 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:47.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:47.220 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:47.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:47.320 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:47.345 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:47.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:47.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:47.390 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:31:47 smithi003 podman[221117]: 2024-08-17 17:31:47.125740366 +0000 UTC m=+4.201298913 container died e95f105c89980babd975708416185d21da151e3d5e2fb289d7f6110936d4e35b (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, RELEASE=main-cd8bdbe, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-08-17T17:31:47.447 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:47.477 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:47.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:47.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:47.579 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:47.580 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:31:47.580 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:31:47.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.580 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45767.0:188316 7.7 e251/226 7:e18cf181:::benchmark_data_smithi003_201798_object75094:head v 251'11659, mlcod=251'11657)", 2024-08-17T17:31:47.580 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:31:46.597589+0000", 2024-08-17T17:31:47.580 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0058379690000000001, 2024-08-17T17:31:47.580 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0058670989999999998, 2024-08-17T17:31:47.580 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:31:47.581 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:31:47.581 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:31:47.581 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:31:47.581 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.581 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:31:47.581 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.597589+0000", 2024-08-17T17:31:47.581 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:31:47.581 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.581 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.581 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:31:47.581 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.597589+0000", 2024-08-17T17:31:47.581 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:31:47.581 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.582 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:31:47.582 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.597590+0000", 2024-08-17T17:31:47.582 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2100000000000001e-06 2024-08-17T17:31:47.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.582 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:31:47.582 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.597599+0000", 2024-08-17T17:31:47.582 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1659999999999994e-06 2024-08-17T17:31:47.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.582 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:31:47.582 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.597611+0000", 2024-08-17T17:31:47.583 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2033e-05 2024-08-17T17:31:47.583 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.583 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.583 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:31:47.583 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.597618+0000", 2024-08-17T17:31:47.583 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4800000000000004e-06 2024-08-17T17:31:47.583 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.583 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.583 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:31:47.583 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.599027+0000", 2024-08-17T17:31:47.583 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014092009999999999 2024-08-17T17:31:47.583 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.583 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.584 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:31:47.584 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.599137+0000", 2024-08-17T17:31:47.584 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000109527 2024-08-17T17:31:47.584 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:31:47.584 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:31:47.584 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:31:47.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.584 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.584 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:188317 7.4 7:26a21cd5:::benchmark_data_smithi003_201798_object75095:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-08-17T17:31:47.584 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:31:46.599934+0000", 2024-08-17T17:31:47.584 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.003492729, 2024-08-17T17:31:47.584 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0035812320000000002, 2024-08-17T17:31:47.584 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:31:47.585 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:31:47.585 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:31:47.585 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:31:47.585 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:31:47.585 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:31:47.585 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 188317 2024-08-17T17:31:47.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.585 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:31:47.585 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.585 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:31:47.585 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.599934+0000", 2024-08-17T17:31:47.585 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:31:47.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.586 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.586 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:31:47.586 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.599934+0000", 2024-08-17T17:31:47.586 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:31:47.586 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.586 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.586 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:31:47.586 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.599936+0000", 2024-08-17T17:31:47.586 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5680000000000001e-06 2024-08-17T17:31:47.586 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.586 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.586 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:31:47.586 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.599940+0000", 2024-08-17T17:31:47.586 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9820000000000002e-06 2024-08-17T17:31:47.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.587 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:31:47.587 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.599951+0000", 2024-08-17T17:31:47.587 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0509999999999999e-05 2024-08-17T17:31:47.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.587 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:31:47.587 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.599958+0000", 2024-08-17T17:31:47.587 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9689999999999997e-06 2024-08-17T17:31:47.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.587 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:31:47.587 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.600008+0000", 2024-08-17T17:31:47.588 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0992999999999998e-05 2024-08-17T17:31:47.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.588 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.588 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:31:47.588 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.600120+0000", 2024-08-17T17:31:47.588 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000111606 2024-08-17T17:31:47.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.588 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.588 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 6", 2024-08-17T17:31:47.588 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.600231+0000", 2024-08-17T17:31:47.588 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011098 2024-08-17T17:31:47.588 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:31:47.588 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:31:47.588 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:31:47.589 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.589 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.589 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:188318 7.9 7:9a94a631:::benchmark_data_smithi003_201798_object75096:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-08-17T17:31:47.589 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:31:46.599976+0000", 2024-08-17T17:31:47.589 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.003451113, 2024-08-17T17:31:47.589 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0036039589999999999, 2024-08-17T17:31:47.589 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:31:47.589 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:31:47.589 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:31:47.589 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:31:47.589 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:31:47.589 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:31:47.589 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 188318 2024-08-17T17:31:47.589 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.590 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:31:47.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.590 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:31:47.590 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.599976+0000", 2024-08-17T17:31:47.590 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:31:47.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.590 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:31:47.590 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.599976+0000", 2024-08-17T17:31:47.590 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:31:47.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.590 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:31:47.591 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.599977+0000", 2024-08-17T17:31:47.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4499999999999997e-07 2024-08-17T17:31:47.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.591 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:31:47.591 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.599979+0000", 2024-08-17T17:31:47.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6900000000000001e-06 2024-08-17T17:31:47.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.591 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:31:47.591 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.599987+0000", 2024-08-17T17:31:47.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0360000000000006e-06 2024-08-17T17:31:47.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:31:47.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.599993+0000", 2024-08-17T17:31:47.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8660000000000002e-06 2024-08-17T17:31:47.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:31:47.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.600282+0000", 2024-08-17T17:31:47.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00028858099999999999 2024-08-17T17:31:47.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:31:47.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.600393+0000", 2024-08-17T17:31:47.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000111607 2024-08-17T17:31:47.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.593 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 7", 2024-08-17T17:31:47.593 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.600508+0000", 2024-08-17T17:31:47.593 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0001148 2024-08-17T17:31:47.593 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:31:47.593 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:31:47.593 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:31:47.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.593 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:188305 7.f 7:fbc68384:::benchmark_data_smithi003_201798_object75083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-08-17T17:31:47.593 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:31:46.582337+0000", 2024-08-17T17:31:47.593 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.021090614000000001, 2024-08-17T17:31:47.594 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.021305824000000001, 2024-08-17T17:31:47.594 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:31:47.594 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:31:47.594 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:31:47.594 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:31:47.594 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:31:47.594 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:31:47.594 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 188305 2024-08-17T17:31:47.594 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.594 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:31:47.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.594 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:31:47.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.582337+0000", 2024-08-17T17:31:47.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:31:47.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.595 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:31:47.595 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.582337+0000", 2024-08-17T17:31:47.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:31:47.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.595 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:31:47.595 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.582339+0000", 2024-08-17T17:31:47.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0880000000000002e-06 2024-08-17T17:31:47.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.595 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:31:47.596 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.582345+0000", 2024-08-17T17:31:47.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9190000000000004e-06 2024-08-17T17:31:47.596 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.596 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:31:47.596 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.582356+0000", 2024-08-17T17:31:47.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1012000000000001e-05 2024-08-17T17:31:47.596 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.596 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:31:47.596 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.582364+0000", 2024-08-17T17:31:47.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0299999999999994e-06 2024-08-17T17:31:47.596 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:31:47.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.583181+0000", 2024-08-17T17:31:47.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000817777 2024-08-17T17:31:47.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:31:47.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.583488+0000", 2024-08-17T17:31:47.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00030687899999999997 2024-08-17T17:31:47.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-08-17T17:31:47.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.586295+0000", 2024-08-17T17:31:47.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028066699999999998 2024-08-17T17:31:47.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:31:47.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.594965+0000", 2024-08-17T17:31:47.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0086700700000000002 2024-08-17T17:31:47.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:31:47.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.599065+0000", 2024-08-17T17:31:47.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004099503 2024-08-17T17:31:47.598 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:31:47.598 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:31:47.598 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:31:47.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.599 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:188313 7.6 7:6eb4be05:::benchmark_data_smithi003_201798_object75091:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-08-17T17:31:47.599 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:31:46.592172+0000", 2024-08-17T17:31:47.599 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.011255617000000001, 2024-08-17T17:31:47.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011540307, 2024-08-17T17:31:47.599 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:31:47.599 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:31:47.599 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:31:47.599 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:31:47.599 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:31:47.599 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:31:47.599 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 188313 2024-08-17T17:31:47.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.599 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:31:47.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.600 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:31:47.600 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.592172+0000", 2024-08-17T17:31:47.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:31:47.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.600 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:31:47.600 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.592172+0000", 2024-08-17T17:31:47.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:31:47.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.600 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:31:47.600 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.592174+0000", 2024-08-17T17:31:47.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9570000000000001e-06 2024-08-17T17:31:47.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:31:47.601 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.592179+0000", 2024-08-17T17:31:47.601 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8320000000000002e-06 2024-08-17T17:31:47.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:31:47.601 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.592193+0000", 2024-08-17T17:31:47.601 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3183999999999999e-05 2024-08-17T17:31:47.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:31:47.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.592201+0000", 2024-08-17T17:31:47.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4530000000000002e-06 2024-08-17T17:31:47.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.602 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:31:47.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.597335+0000", 2024-08-17T17:31:47.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0051342000000000002 2024-08-17T17:31:47.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.602 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:31:47.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.597747+0000", 2024-08-17T17:31:47.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00041158999999999998 2024-08-17T17:31:47.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:31:47.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:31:47.603 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-08-17T17:31:47.603 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:46.597862+0000", 2024-08-17T17:31:47.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000114732 2024-08-17T17:31:47.603 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:31:47.603 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:31:47.603 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:31:47.603 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:31:47.603 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-08-17T17:31:47.603 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 5 2024-08-17T17:31:47.603 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:31:47.639 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:47.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:47.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:47.741 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:47.769 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:47.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:47.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:47.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:47 smithi003 ceph-mon[103344]: pgmap v738: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 2.5 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 897 op/s; 17949/122640 objects degraded (14.636%) 2024-08-17T17:31:47.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:47 smithi003 ceph-mon[100615]: pgmap v738: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 2.5 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 897 op/s; 17949/122640 objects degraded (14.636%) 2024-08-17T17:31:47.870 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:47.909 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:47.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:47.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:47.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:47 smithi053 ceph-mon[89963]: pgmap v738: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 2.5 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 897 op/s; 17949/122640 objects degraded (14.636%) 2024-08-17T17:31:48.011 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:48.057 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:48.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:48.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:48.159 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:48.186 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:48.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:48.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:48.288 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:48.314 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:48.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:48.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:48.342 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:31:48.342 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:31:48.343 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-17T17:31:48.379 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:31:48.380 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:31:48.413 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:31:48.413 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:31:48.415 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:48.441 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:31:48.441 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:31:48.443 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:48.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:48.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:48.475 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:31:48.475 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:31:48.508 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:31:48.508 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:31:48.542 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:31:48.542 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:31:48.545 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:48.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:48.590 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:48.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:48.691 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:48.707 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:48 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3662982391' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:31:48.707 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:48 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3662982391' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:31:48.718 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:48.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:48.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:48.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.1.service' 2024-08-17T17:31:48.820 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:48.828 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-17T17:31:48.829 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-08-17T17:31:48.849 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:48.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:48.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:48.951 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:48.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:48 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3662982391' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:31:48.996 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:48.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:48.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:49.097 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:49.127 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:49.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:49.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:49.229 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:49.258 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:49.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:49.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:49.359 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:49.400 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:49.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:49.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:49.490 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-17T17:31:49.502 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:49.584 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:49.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:49.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:49.686 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:49.716 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:49.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:49.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:49.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:49 smithi003 ceph-mon[103344]: pgmap v739: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 2.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 873 op/s; 17364/119049 objects degraded (14.586%) 2024-08-17T17:31:49.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:49 smithi003 ceph-mon[100615]: pgmap v739: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 2.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 873 op/s; 17364/119049 objects degraded (14.586%) 2024-08-17T17:31:49.818 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:49.865 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:49.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:49.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:49.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:49 smithi053 ceph-mon[89963]: pgmap v739: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 2.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 873 op/s; 17364/119049 objects degraded (14.586%) 2024-08-17T17:31:49.967 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:49.991 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-17T17:31:49.991 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-17T17:31:49.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:49.992 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:49.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:50.047 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:31:50.093 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:50.098 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:31:50.116 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:50.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:50.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:50.218 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:50.272 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:50.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:50.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:50.374 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:50.401 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:50.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:50.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:50.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:31:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:31:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:31:50.503 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:50.545 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:50.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:50.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:50.647 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:50.688 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:50.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:50.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:50.766 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:50 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 17364/119049 objects degraded (14.586%), 14 pgs degraded (PG_DEGRADED) 2024-08-17T17:31:50.766 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:50 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 17364/119049 objects degraded (14.586%), 14 pgs degraded (PG_DEGRADED) 2024-08-17T17:31:50.790 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:50.817 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:50.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:50.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:50.919 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:50.952 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:50.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:50.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:50.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:50 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 17364/119049 objects degraded (14.586%), 14 pgs degraded (PG_DEGRADED) 2024-08-17T17:31:51.055 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:51.088 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:51.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:51.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:51.189 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:51.226 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:51.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:51.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:51.328 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:51.355 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:51.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:51.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:51.457 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:51.484 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:51.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:51.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:51.586 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:51.623 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:51.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:51.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:51.725 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:51.752 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:51.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:51.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:51.854 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:51.896 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:51.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:51.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:51.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:51 smithi053 ceph-mon[89963]: pgmap v740: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 2.2 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 953 op/s; 15858/108525 objects degraded (14.612%) 2024-08-17T17:31:51.992 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:31:51.998 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:52.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:51 smithi003 ceph-mon[100615]: pgmap v740: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 2.2 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 953 op/s; 15858/108525 objects degraded (14.612%) 2024-08-17T17:31:52.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:51 smithi003 ceph-mon[103344]: pgmap v740: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 2.2 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 953 op/s; 15858/108525 objects degraded (14.612%) 2024-08-17T17:31:52.032 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:52.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:52.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:52.134 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:52.162 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:52.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:52.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:52.264 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:52.287 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:52.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:52.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:52.389 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:52.440 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:52.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:52.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:52.541 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:52.569 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:52.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:52.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:52.671 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:52.696 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:52.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:52.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:52.798 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:52.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:52.827 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:52.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:52.928 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:52.956 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:52.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:52.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:53.058 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:53.085 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:53.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:53.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:53.187 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:53.218 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:53.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:53.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:53.320 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:53.348 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:53.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:53.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:53.449 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:53.497 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:53.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:53.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:53.599 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:53.626 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:53.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:53.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:53.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:53.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:53.763 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:53.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:53.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:53 smithi003 ceph-mon[100615]: pgmap v741: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 2.2 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 953 op/s; 15858/108525 objects degraded (14.612%) 2024-08-17T17:31:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:31:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:31:53.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:53 smithi003 ceph-mon[103344]: pgmap v741: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 2.2 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 953 op/s; 15858/108525 objects degraded (14.612%) 2024-08-17T17:31:53.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:31:53.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:31:53.830 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-08-17T17:31:53.830 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:31:53.830 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-08-17T17:31:53.830 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 1.000000 2024-08-17T17:31:53.830 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 7 1 2024-08-17T17:31:53.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:53.929 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:53.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:53.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:53.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:53 smithi053 ceph-mon[89963]: pgmap v741: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 2.2 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 953 op/s; 15858/108525 objects degraded (14.612%) 2024-08-17T17:31:53.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:31:53.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:31:54.031 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:54.058 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:54.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:54.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:54.160 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:54.185 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:54.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:54.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:54.286 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:54.339 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:54.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:54.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:54.365 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:31:54.365 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:31:54.441 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:54.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:54.470 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:54.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:54.570 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:31:54.571 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:54.597 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:54.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:54.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:54.697 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-08-17T17:31:54.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:54.751 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:54.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:54.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:54.853 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:54.890 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:54.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:54.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:54.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:54 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3975630058' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:31:54.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:54 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:31:54.991 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:55.027 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:54 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3975630058' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:31:55.028 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:54 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:31:55.028 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:54 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3975630058' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:31:55.028 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:54 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:31:55.033 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:55.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:55.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:55.123 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-17T17:31:55.135 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:55.205 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:55.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:55.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:55.307 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:55.357 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:55.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:55.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:55.459 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:55.500 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:55.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:55.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:55.602 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:55.631 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:55.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:55.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:55.733 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:55.769 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:55.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:55.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:55.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:55 smithi003 ceph-mon[103344]: pgmap v742: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 2.0 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 1.10k op/s; 13980/99138 objects degraded (14.102%) 2024-08-17T17:31:55.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:55 smithi003 ceph-mon[103344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:31:55.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:55 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:31:55.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:55 smithi003 ceph-mon[103344]: osdmap e252: 8 total, 7 up, 7 in 2024-08-17T17:31:55.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:55 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 13980/99138 objects degraded (14.102%), 14 pgs degraded (PG_DEGRADED) 2024-08-17T17:31:55.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:55 smithi003 ceph-mon[100615]: pgmap v742: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 2.0 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 1.10k op/s; 13980/99138 objects degraded (14.102%) 2024-08-17T17:31:55.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:55 smithi003 ceph-mon[100615]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:31:55.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:55 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:31:55.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:55 smithi003 ceph-mon[100615]: osdmap e252: 8 total, 7 up, 7 in 2024-08-17T17:31:55.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:55 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 13980/99138 objects degraded (14.102%), 14 pgs degraded (PG_DEGRADED) 2024-08-17T17:31:55.774 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:31:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:31:55] "GET /metrics HTTP/1.1" 200 36507 "" "Prometheus/2.51.0" 2024-08-17T17:31:55.871 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:55.896 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:55.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:55.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:55.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:55 smithi053 ceph-mon[89963]: pgmap v742: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 2.0 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 1.10k op/s; 13980/99138 objects degraded (14.102%) 2024-08-17T17:31:55.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:55 smithi053 ceph-mon[89963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:31:55.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:55 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:31:55.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:55 smithi053 ceph-mon[89963]: osdmap e252: 8 total, 7 up, 7 in 2024-08-17T17:31:55.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:55 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 13980/99138 objects degraded (14.102%), 14 pgs degraded (PG_DEGRADED) 2024-08-17T17:31:55.998 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:56.051 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:56.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:56.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:56.153 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:56.184 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:56.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:56.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:56.285 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:56.313 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:56.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:56.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:56.415 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:56.443 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:56.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:56.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:56.545 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:56.581 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:56.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:56.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:56.623 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:31:56.624 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:31:56.683 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:56.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:56.712 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:56.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:56.813 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:56.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:56.846 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:56.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:56.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:56.982 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:56.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:56.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:57.084 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:57.114 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:57.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:57.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:57.216 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:57.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:57.258 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:57.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:57.359 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:57.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:57.406 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:57.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:57.507 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:57.537 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:57.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:57.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:57.639 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:57.673 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:57.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:57.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:57.775 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:57.806 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:57.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:57.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:57.850 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:31:57.907 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:57.933 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:57.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:57.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:57.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:57 smithi053 ceph-mon[89963]: pgmap v744: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 1.9 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 971 op/s; 13661/93480 objects degraded (14.614%) 2024-08-17T17:31:58.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:57 smithi003 ceph-mon[103344]: pgmap v744: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 1.9 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 971 op/s; 13661/93480 objects degraded (14.614%) 2024-08-17T17:31:58.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:57 smithi003 ceph-mon[100615]: pgmap v744: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 1.9 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 971 op/s; 13661/93480 objects degraded (14.614%) 2024-08-17T17:31:58.035 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:58.061 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:58.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:58.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:58.163 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:58.188 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:58.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:58.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:58.290 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:58.315 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:58.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:58.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:58.417 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:58.446 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:58.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:58.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:58.548 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:58.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:58.597 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:58.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:58.698 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:58.729 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:58.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:58.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:58.831 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:31:58.857 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:58.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:58.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:31:58.959 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:31:58.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:59.000 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:59.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:31:59.101 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:59.139 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:59.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:59.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:59.241 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:59.280 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:59.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:59.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:59.382 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:31:59.407 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:59.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:59.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:31:59.509 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:31:59.545 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:59.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:59.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:31:59.647 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:31:59.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:59.684 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:59.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:31:59.785 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:31:59.815 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:31:59.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:59.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:31:59.917 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:31:59.959 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:31:59.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:31:59.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:31:59.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:31:59 smithi053 ceph-mon[89963]: pgmap v745: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 1.8 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 976 op/s; 13057/89742 objects degraded (14.549%) 2024-08-17T17:32:00.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:31:59 smithi003 ceph-mon[100615]: pgmap v745: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 1.8 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 976 op/s; 13057/89742 objects degraded (14.549%) 2024-08-17T17:32:00.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:31:59 smithi003 ceph-mon[103344]: pgmap v745: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 1.8 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 976 op/s; 13057/89742 objects degraded (14.549%) 2024-08-17T17:32:00.060 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:00.105 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:00.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:00.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:00.207 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:00.234 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:00.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:00.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:00.336 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:00.363 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:00.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:00.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:00.465 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:00.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:32:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:32:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:32:00.507 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:00.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:00.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:00.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:00.646 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:00.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:00.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:00.748 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:00.783 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:00.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:00.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:00.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:00.929 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:00.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:00.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:00.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:00 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 13057/89742 objects degraded (14.549%), 14 pgs degraded (PG_DEGRADED) 2024-08-17T17:32:01.008 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:00 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 13057/89742 objects degraded (14.549%), 14 pgs degraded (PG_DEGRADED) 2024-08-17T17:32:01.009 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:00 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 13057/89742 objects degraded (14.549%), 14 pgs degraded (PG_DEGRADED) 2024-08-17T17:32:01.031 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:01.062 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:01.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:01.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:01.164 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:01.193 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:01.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:01.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:01.295 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:01.320 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:01.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:01.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:01.422 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:01.449 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:01.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:01.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:01.551 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:01.581 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:01.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:01.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:01.683 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:01.709 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:01.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:01.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:01.810 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:01.840 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:01.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:01.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:01.942 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:01.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:01 smithi053 ceph-mon[89963]: pgmap v746: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 1.6 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 996 op/s; 11496/78600 objects degraded (14.626%) 2024-08-17T17:32:01.970 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:01.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:01.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:02.022 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:01 smithi003 ceph-mon[100615]: pgmap v746: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 1.6 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 996 op/s; 11496/78600 objects degraded (14.626%) 2024-08-17T17:32:02.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:01 smithi003 ceph-mon[103344]: pgmap v746: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 1.6 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 996 op/s; 11496/78600 objects degraded (14.626%) 2024-08-17T17:32:02.072 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:02.095 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:02.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:02.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:02.196 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:02.240 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:02.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:02.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:02.342 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:02.369 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:02.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:02.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:02.471 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:02.512 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:02.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:02.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:02.614 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:02.639 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:02.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:02.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:02.742 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:02.768 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:02.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:02.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:02.870 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:02.902 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:02.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:02.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:03.004 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:03.032 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:03.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:03.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:03.133 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:03.161 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:03.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:03.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:03.263 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:03.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:03.305 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:03.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:03.406 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:03.436 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:03.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:03.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:03.537 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:03.560 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:03.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:03.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:03.662 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:03.698 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:03.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:03.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:03.800 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:03.830 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:03.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:03.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:03.932 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:03.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:03 smithi053 ceph-mon[89963]: pgmap v747: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 1.6 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 996 op/s; 11496/78600 objects degraded (14.626%) 2024-08-17T17:32:03.979 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:03.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:03.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:04.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:03 smithi003 ceph-mon[100615]: pgmap v747: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 1.6 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 996 op/s; 11496/78600 objects degraded (14.626%) 2024-08-17T17:32:04.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:03 smithi003 ceph-mon[103344]: pgmap v747: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 1.6 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 996 op/s; 11496/78600 objects degraded (14.626%) 2024-08-17T17:32:04.081 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:04.109 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:04.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:04.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:04.210 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:04.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:04.251 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:04.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:04.352 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:04.385 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:04.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:04.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:04.487 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:04.517 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:04.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:04.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:04.618 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:04.649 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:04.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:04.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:04.751 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:04.789 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:04.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:04.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:04.807 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:04 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 11496/78600 objects degraded (14.626%), 14 pgs degraded (PG_DEGRADED) 2024-08-17T17:32:04.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:04 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 11496/78600 objects degraded (14.626%), 14 pgs degraded (PG_DEGRADED) 2024-08-17T17:32:04.891 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:04.894 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:32:04.895 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:32:04.900 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-17T17:32:04.928 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:04.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:04.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:04.936 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:32:04.936 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:32:04.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:04 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 11496/78600 objects degraded (14.626%), 14 pgs degraded (PG_DEGRADED) 2024-08-17T17:32:04.972 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:32:04.973 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:32:05.004 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:32:05.004 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:32:05.030 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:05.034 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:32:05.035 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:32:05.068 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:32:05.068 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:32:05.080 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:05.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:05.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:05.111 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:32:05.111 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:32:05.182 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:05.214 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:05.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:05.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:05.316 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:05.348 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:05.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:05.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:05.450 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:05.479 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:05.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:05.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:05.581 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:05.626 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:05.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:05.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:05.728 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:05.752 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:05.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:05.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:05.851 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:32:05.851 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-08-17T17:32:05.851 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-08-17T17:32:05.851 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:32:05.851 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.851 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45491.0:160772 6.0 6:0639d6cf:::benchmark_data_smithi003_175043_object62123:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-08-17T17:32:05.851 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:27:20.702861+0000", 2024-08-17T17:32:05.851 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 284.814590211, 2024-08-17T17:32:05.851 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.23765629799999999, 2024-08-17T17:32:05.851 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:32:05.852 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:32:05.852 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:32:05.852 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:32:05.852 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45491", 2024-08-17T17:32:05.852 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:32:05.852 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 160772 2024-08-17T17:32:05.852 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.852 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:32:05.852 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.852 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:32:05.852 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.702861+0000", 2024-08-17T17:32:05.852 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.852 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.853 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.853 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:32:05.853 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.702861+0000", 2024-08-17T17:32:05.853 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.853 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.853 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.853 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:32:05.853 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.702864+0000", 2024-08-17T17:32:05.853 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1390000000000002e-06 2024-08-17T17:32:05.853 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.853 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.853 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:32:05.853 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.702869+0000", 2024-08-17T17:32:05.853 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4280000000000004e-06 2024-08-17T17:32:05.854 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.854 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.854 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:32:05.854 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.702891+0000", 2024-08-17T17:32:05.854 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2172999999999999e-05 2024-08-17T17:32:05.854 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.854 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.854 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:32:05.854 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.702901+0000", 2024-08-17T17:32:05.854 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6010000000000001e-06 2024-08-17T17:32:05.854 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.854 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.854 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:32:05.855 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.709233+0000", 2024-08-17T17:32:05.855 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063318230000000003 2024-08-17T17:32:05.855 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.855 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.855 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:32:05.855 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.709361+0000", 2024-08-17T17:32:05.855 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012887200000000001 2024-08-17T17:32:05.855 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.855 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.855 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3", 2024-08-17T17:32:05.855 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.709485+0000", 2024-08-17T17:32:05.855 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012318600000000001 2024-08-17T17:32:05.856 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.856 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.856 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:32:05.856 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.713843+0000", 2024-08-17T17:32:05.856 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0043586609999999998 2024-08-17T17:32:05.856 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.856 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.856 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.856 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.737302+0000", 2024-08-17T17:32:05.856 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.023458265999999998 2024-08-17T17:32:05.856 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.856 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.857 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.857 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.940421+0000", 2024-08-17T17:32:05.857 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.20311971500000001 2024-08-17T17:32:05.857 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.857 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.857 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:32:05.857 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.940472+0000", 2024-08-17T17:32:05.857 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1215999999999999e-05 2024-08-17T17:32:05.857 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.857 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.857 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:32:05.857 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.940518+0000", 2024-08-17T17:32:05.857 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5218e-05 2024-08-17T17:32:05.858 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.858 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:32:05.858 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.858 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.858 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.858 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45491.0:160779 6.9 6:9d3f1440:::benchmark_data_smithi003_175043_object62130:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-08-17T17:32:05.858 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:27:20.717781+0000", 2024-08-17T17:32:05.858 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 284.799671031, 2024-08-17T17:32:05.858 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.21854267699999999, 2024-08-17T17:32:05.858 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:32:05.858 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:32:05.858 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:32:05.858 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:32:05.859 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45491", 2024-08-17T17:32:05.859 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:32:05.859 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 160779 2024-08-17T17:32:05.859 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.859 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:32:05.859 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.859 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:32:05.859 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.717781+0000", 2024-08-17T17:32:05.859 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.859 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.859 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.859 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:32:05.859 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.717781+0000", 2024-08-17T17:32:05.859 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.859 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.860 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.860 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:32:05.860 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.717783+0000", 2024-08-17T17:32:05.860 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.114e-06 2024-08-17T17:32:05.860 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.860 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.860 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:32:05.860 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.717789+0000", 2024-08-17T17:32:05.860 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6830000000000002e-06 2024-08-17T17:32:05.860 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.860 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.860 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:32:05.860 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.717804+0000", 2024-08-17T17:32:05.860 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4348000000000001e-05 2024-08-17T17:32:05.861 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.861 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.861 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:32:05.861 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.717812+0000", 2024-08-17T17:32:05.861 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5350000000000004e-06 2024-08-17T17:32:05.861 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.861 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.861 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:32:05.861 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.717881+0000", 2024-08-17T17:32:05.861 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8867000000000003e-05 2024-08-17T17:32:05.861 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.861 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.861 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:32:05.861 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.718158+0000", 2024-08-17T17:32:05.861 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00027689199999999999 2024-08-17T17:32:05.862 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.862 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.862 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:32:05.862 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.718571+0000", 2024-08-17T17:32:05.862 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00041280199999999998 2024-08-17T17:32:05.862 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.862 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.862 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.862 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.734795+0000", 2024-08-17T17:32:05.862 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.016223807999999999 2024-08-17T17:32:05.862 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.862 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.862 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:32:05.862 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.739588+0000", 2024-08-17T17:32:05.863 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00479365 2024-08-17T17:32:05.863 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.863 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.863 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.863 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.936222+0000", 2024-08-17T17:32:05.863 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.19663338499999999 2024-08-17T17:32:05.863 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.863 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.863 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:32:05.863 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.936279+0000", 2024-08-17T17:32:05.863 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7426999999999998e-05 2024-08-17T17:32:05.863 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.863 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.863 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:32:05.863 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.936323+0000", 2024-08-17T17:32:05.863 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4165999999999997e-05 2024-08-17T17:32:05.864 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.864 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:32:05.864 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.864 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.864 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.864 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45491.0:160780 6.9 6:9f7045a7:::benchmark_data_smithi003_175043_object62131:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-08-17T17:32:05.864 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:27:20.720442+0000", 2024-08-17T17:32:05.864 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 284.79700955700002, 2024-08-17T17:32:05.864 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.23174434899999999, 2024-08-17T17:32:05.864 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:32:05.864 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:32:05.864 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:32:05.864 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:32:05.864 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45491", 2024-08-17T17:32:05.865 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:32:05.865 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 160780 2024-08-17T17:32:05.865 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.865 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:32:05.865 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.865 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:32:05.865 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.720442+0000", 2024-08-17T17:32:05.865 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.865 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.865 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.865 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:32:05.865 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.720442+0000", 2024-08-17T17:32:05.865 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.866 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.866 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.866 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:32:05.866 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.720444+0000", 2024-08-17T17:32:05.866 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7889999999999999e-06 2024-08-17T17:32:05.866 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.866 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.866 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:32:05.866 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.720451+0000", 2024-08-17T17:32:05.866 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3540000000000002e-06 2024-08-17T17:32:05.866 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.866 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.866 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:32:05.867 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.720471+0000", 2024-08-17T17:32:05.867 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9556999999999999e-05 2024-08-17T17:32:05.867 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.867 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.867 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:32:05.867 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.720481+0000", 2024-08-17T17:32:05.867 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0419e-05 2024-08-17T17:32:05.867 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.867 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.867 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:32:05.867 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.725322+0000", 2024-08-17T17:32:05.867 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0048407249999999997 2024-08-17T17:32:05.867 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.868 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.868 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:32:05.868 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.725672+0000", 2024-08-17T17:32:05.868 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00035019899999999999 2024-08-17T17:32:05.868 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.868 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.868 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:32:05.868 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.726101+0000", 2024-08-17T17:32:05.868 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00042927699999999999 2024-08-17T17:32:05.868 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.868 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.868 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.868 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.734840+0000", 2024-08-17T17:32:05.868 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0087386300000000007 2024-08-17T17:32:05.869 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.869 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.869 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:32:05.869 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.739596+0000", 2024-08-17T17:32:05.869 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0047558419999999997 2024-08-17T17:32:05.869 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.869 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.869 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.869 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.951863+0000", 2024-08-17T17:32:05.869 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.212267492 2024-08-17T17:32:05.869 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.869 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.869 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:32:05.870 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.952036+0000", 2024-08-17T17:32:05.870 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017293500000000001 2024-08-17T17:32:05.870 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.870 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.870 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:32:05.870 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.952186+0000", 2024-08-17T17:32:05.870 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015013 2024-08-17T17:32:05.870 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.870 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:32:05.870 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.870 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.870 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.871 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45491.0:160781 6.0 6:047167c6:::benchmark_data_smithi003_175043_object62132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-08-17T17:32:05.871 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:27:20.721935+0000", 2024-08-17T17:32:05.871 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 284.79551674099997, 2024-08-17T17:32:05.871 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.22536682999999999, 2024-08-17T17:32:05.871 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:32:05.871 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:32:05.871 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:32:05.871 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:32:05.871 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45491", 2024-08-17T17:32:05.871 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:32:05.871 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 160781 2024-08-17T17:32:05.871 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.871 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:32:05.871 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.871 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:32:05.872 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.721935+0000", 2024-08-17T17:32:05.872 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.872 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.872 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.872 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:32:05.872 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.721935+0000", 2024-08-17T17:32:05.872 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.872 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.872 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.872 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:32:05.872 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.721936+0000", 2024-08-17T17:32:05.872 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.47e-07 2024-08-17T17:32:05.872 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.872 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.873 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:32:05.873 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.721940+0000", 2024-08-17T17:32:05.873 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.498e-06 2024-08-17T17:32:05.873 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.873 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.873 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:32:05.873 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.721963+0000", 2024-08-17T17:32:05.873 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2614000000000001e-05 2024-08-17T17:32:05.873 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.873 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.873 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:32:05.873 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.721978+0000", 2024-08-17T17:32:05.873 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4772e-05 2024-08-17T17:32:05.873 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.874 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.874 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:32:05.874 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.722549+0000", 2024-08-17T17:32:05.874 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00057140299999999995 2024-08-17T17:32:05.874 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.874 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.874 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:32:05.874 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.722668+0000", 2024-08-17T17:32:05.874 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000118585 2024-08-17T17:32:05.874 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.874 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.874 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3", 2024-08-17T17:32:05.874 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.722790+0000", 2024-08-17T17:32:05.875 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012234400000000001 2024-08-17T17:32:05.875 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.875 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.875 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:32:05.875 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.735534+0000", 2024-08-17T17:32:05.875 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012744313 2024-08-17T17:32:05.875 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.875 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.875 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.875 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.737405+0000", 2024-08-17T17:32:05.875 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0018705919999999999 2024-08-17T17:32:05.875 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.875 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.875 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.876 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.947003+0000", 2024-08-17T17:32:05.876 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.20959830600000001 2024-08-17T17:32:05.876 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.876 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.876 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:32:05.876 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.947063+0000", 2024-08-17T17:32:05.876 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0235999999999997e-05 2024-08-17T17:32:05.876 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.876 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.876 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:32:05.876 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.947302+0000", 2024-08-17T17:32:05.876 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00023822000000000001 2024-08-17T17:32:05.876 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.876 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:32:05.877 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.877 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.877 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.877 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45491.0:160782 6.0 6:0b386992:::benchmark_data_smithi003_175043_object62133:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-08-17T17:32:05.877 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:27:20.722728+0000", 2024-08-17T17:32:05.877 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 284.79472390900003, 2024-08-17T17:32:05.877 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.24380597800000001, 2024-08-17T17:32:05.877 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:32:05.877 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:32:05.877 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:32:05.877 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:32:05.877 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45491", 2024-08-17T17:32:05.877 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:32:05.877 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 160782 2024-08-17T17:32:05.877 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.878 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:32:05.878 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.878 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:32:05.878 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.722728+0000", 2024-08-17T17:32:05.878 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.878 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.878 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.878 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:32:05.878 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.722728+0000", 2024-08-17T17:32:05.878 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.878 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.878 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.878 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:32:05.878 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.722728+0000", 2024-08-17T17:32:05.879 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2600000000000002e-07 2024-08-17T17:32:05.879 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.879 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.879 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:32:05.879 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.722731+0000", 2024-08-17T17:32:05.879 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0319999999999999e-06 2024-08-17T17:32:05.879 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.879 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.879 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:32:05.879 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.722740+0000", 2024-08-17T17:32:05.879 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2190000000000004e-06 2024-08-17T17:32:05.879 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.879 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.879 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:32:05.879 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.722745+0000", 2024-08-17T17:32:05.880 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7239999999999997e-06 2024-08-17T17:32:05.880 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.880 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.880 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:32:05.880 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.726927+0000", 2024-08-17T17:32:05.880 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0041816099999999997 2024-08-17T17:32:05.880 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.880 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.880 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:32:05.880 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.727233+0000", 2024-08-17T17:32:05.880 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00030585299999999997 2024-08-17T17:32:05.880 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.880 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.880 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3", 2024-08-17T17:32:05.881 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.727611+0000", 2024-08-17T17:32:05.881 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00037850900000000002 2024-08-17T17:32:05.881 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.881 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.881 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:32:05.881 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.735544+0000", 2024-08-17T17:32:05.881 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0079324549999999997 2024-08-17T17:32:05.881 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.881 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.881 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.881 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.737471+0000", 2024-08-17T17:32:05.881 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0019272689999999999 2024-08-17T17:32:05.881 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.881 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.882 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.882 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.966423+0000", 2024-08-17T17:32:05.882 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.228952081 2024-08-17T17:32:05.882 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.882 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.882 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:32:05.882 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.966484+0000", 2024-08-17T17:32:05.882 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0609000000000002e-05 2024-08-17T17:32:05.882 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.882 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.882 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:32:05.882 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.966534+0000", 2024-08-17T17:32:05.882 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9991000000000003e-05 2024-08-17T17:32:05.883 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.883 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:32:05.883 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.883 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.883 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.883 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45491.0:160783 6.9 6:91dbffd2:::benchmark_data_smithi003_175043_object62134:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-08-17T17:32:05.883 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:27:20.722753+0000", 2024-08-17T17:32:05.883 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 284.79469908200002, 2024-08-17T17:32:05.883 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.231241845, 2024-08-17T17:32:05.883 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:32:05.883 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:32:05.883 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:32:05.883 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:32:05.884 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45491", 2024-08-17T17:32:05.884 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:32:05.884 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 160783 2024-08-17T17:32:05.884 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.884 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:32:05.884 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.884 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:32:05.884 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.722753+0000", 2024-08-17T17:32:05.884 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.884 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.884 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.884 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:32:05.884 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.722753+0000", 2024-08-17T17:32:05.885 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.885 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.885 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.885 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:32:05.885 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.722753+0000", 2024-08-17T17:32:05.885 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.01e-07 2024-08-17T17:32:05.885 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.885 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.885 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:32:05.885 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.722755+0000", 2024-08-17T17:32:05.885 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8679999999999999e-06 2024-08-17T17:32:05.885 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.885 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.885 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:32:05.886 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.722761+0000", 2024-08-17T17:32:05.886 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8030000000000002e-06 2024-08-17T17:32:05.886 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.886 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.886 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:32:05.886 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.722765+0000", 2024-08-17T17:32:05.886 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6659999999999999e-06 2024-08-17T17:32:05.886 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.886 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.886 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:32:05.886 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.735169+0000", 2024-08-17T17:32:05.886 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012404188999999999 2024-08-17T17:32:05.886 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.887 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.887 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:32:05.887 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.735342+0000", 2024-08-17T17:32:05.887 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017291999999999999 2024-08-17T17:32:05.887 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.887 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.887 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:32:05.887 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.735502+0000", 2024-08-17T17:32:05.887 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015948899999999999 2024-08-17T17:32:05.887 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.887 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.887 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.887 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.739634+0000", 2024-08-17T17:32:05.888 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0041319909999999998 2024-08-17T17:32:05.888 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.888 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.888 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:32:05.888 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.739714+0000", 2024-08-17T17:32:05.888 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0038999999999997e-05 2024-08-17T17:32:05.888 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.888 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.888 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.888 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.953877+0000", 2024-08-17T17:32:05.888 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.214163459 2024-08-17T17:32:05.888 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.888 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.889 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:32:05.889 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.953943+0000", 2024-08-17T17:32:05.889 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5285999999999998e-05 2024-08-17T17:32:05.889 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.889 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.889 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:32:05.889 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.953994+0000", 2024-08-17T17:32:05.889 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1734000000000003e-05 2024-08-17T17:32:05.889 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.889 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:32:05.889 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.889 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.889 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.890 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45491.0:163200 6.0 6:0577e191:::benchmark_data_smithi003_175043_object64551:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-08-17T17:32:05.890 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:27:23.937400+0000", 2024-08-17T17:32:05.890 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 281.58005167800002, 2024-08-17T17:32:05.890 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17728637699999999, 2024-08-17T17:32:05.890 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:32:05.890 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:32:05.890 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:32:05.890 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:32:05.890 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45491", 2024-08-17T17:32:05.890 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:32:05.890 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 163200 2024-08-17T17:32:05.890 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.891 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:32:05.891 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.891 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:32:05.891 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:23.937400+0000", 2024-08-17T17:32:05.891 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.891 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.891 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.891 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:32:05.891 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:23.937400+0000", 2024-08-17T17:32:05.891 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.891 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.891 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.891 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:32:05.891 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:23.937401+0000", 2024-08-17T17:32:05.891 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3090000000000001e-06 2024-08-17T17:32:05.891 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.892 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.892 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:32:05.892 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:23.937406+0000", 2024-08-17T17:32:05.892 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0050000000000004e-06 2024-08-17T17:32:05.892 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.892 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.892 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:32:05.892 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:23.937418+0000", 2024-08-17T17:32:05.892 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2254e-05 2024-08-17T17:32:05.892 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.892 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.892 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:32:05.892 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:23.937425+0000", 2024-08-17T17:32:05.892 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1430000000000003e-06 2024-08-17T17:32:05.892 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.893 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.893 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:32:05.893 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:23.937478+0000", 2024-08-17T17:32:05.893 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3094999999999999e-05 2024-08-17T17:32:05.893 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.893 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.893 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:32:05.893 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:23.937611+0000", 2024-08-17T17:32:05.893 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000132881 2024-08-17T17:32:05.893 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.893 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.893 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3", 2024-08-17T17:32:05.893 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:23.937743+0000", 2024-08-17T17:32:05.893 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013236300000000001 2024-08-17T17:32:05.894 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.894 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.894 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.894 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:23.941678+0000", 2024-08-17T17:32:05.894 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039347990000000001 2024-08-17T17:32:05.894 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.894 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.894 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:32:05.894 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:23.941771+0000", 2024-08-17T17:32:05.894 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3431999999999996e-05 2024-08-17T17:32:05.894 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.894 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.894 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.894 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:24.114307+0000", 2024-08-17T17:32:05.895 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17253618000000001 2024-08-17T17:32:05.895 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.895 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.895 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:32:05.895 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:24.114536+0000", 2024-08-17T17:32:05.895 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00022906099999999999 2024-08-17T17:32:05.895 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.895 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.895 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:32:05.895 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:24.114686+0000", 2024-08-17T17:32:05.895 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014985500000000001 2024-08-17T17:32:05.895 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.895 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:32:05.895 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.895 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.896 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.896 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45491.0:191504 6.9 6:9fae1e44:::benchmark_data_smithi003_175043_object92855:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-08-17T17:32:05.896 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:28:00.551284+0000", 2024-08-17T17:32:05.896 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 244.966167781, 2024-08-17T17:32:05.896 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.26288574799999997, 2024-08-17T17:32:05.896 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:32:05.896 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:32:05.896 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:32:05.896 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:32:05.896 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45491", 2024-08-17T17:32:05.896 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:32:05.896 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 191504 2024-08-17T17:32:05.896 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.896 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:32:05.896 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.897 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:32:05.897 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.551284+0000", 2024-08-17T17:32:05.897 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.897 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.897 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.897 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:32:05.897 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.551284+0000", 2024-08-17T17:32:05.897 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.897 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.897 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.897 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:32:05.897 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.551286+0000", 2024-08-17T17:32:05.897 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7409999999999999e-06 2024-08-17T17:32:05.897 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.897 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.898 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:32:05.898 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.551292+0000", 2024-08-17T17:32:05.898 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9379999999999997e-06 2024-08-17T17:32:05.898 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.898 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.898 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:32:05.898 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.551306+0000", 2024-08-17T17:32:05.898 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4703e-05 2024-08-17T17:32:05.898 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.898 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.898 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:32:05.898 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.551315+0000", 2024-08-17T17:32:05.898 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4279999999999996e-06 2024-08-17T17:32:05.898 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.899 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.899 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:32:05.899 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.551489+0000", 2024-08-17T17:32:05.899 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017430000000000001 2024-08-17T17:32:05.899 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.899 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.899 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:32:05.899 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.551601+0000", 2024-08-17T17:32:05.899 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000112218 2024-08-17T17:32:05.899 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.899 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.899 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:32:05.899 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.551705+0000", 2024-08-17T17:32:05.899 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000103658 2024-08-17T17:32:05.899 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.899 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.900 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:32:05.900 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.556056+0000", 2024-08-17T17:32:05.900 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0043507700000000003 2024-08-17T17:32:05.900 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.900 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.900 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.900 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.559250+0000", 2024-08-17T17:32:05.900 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031944550000000001 2024-08-17T17:32:05.900 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.900 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.900 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.900 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.814022+0000", 2024-08-17T17:32:05.900 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.25477233999999999 2024-08-17T17:32:05.900 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.900 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.901 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:32:05.901 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.814121+0000", 2024-08-17T17:32:05.901 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8799000000000001e-05 2024-08-17T17:32:05.901 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.901 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.901 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:32:05.901 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.814170+0000", 2024-08-17T17:32:05.901 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8398e-05 2024-08-17T17:32:05.901 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.901 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:32:05.901 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.901 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.901 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.901 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45491.0:191517 6.a e220/214 6:5090b9fe:::benchmark_data_smithi003_175043_object92868:head v 220'11904, mlcod=220'11902)", 2024-08-17T17:32:05.902 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:28:00.567359+0000", 2024-08-17T17:32:05.902 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 244.95009270400001, 2024-08-17T17:32:05.902 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.24201456700000001, 2024-08-17T17:32:05.902 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:32:05.902 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:32:05.902 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:32:05.902 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:32:05.902 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.902 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:32:05.902 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.567359+0000", 2024-08-17T17:32:05.902 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.902 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.902 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.902 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:32:05.902 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.567359+0000", 2024-08-17T17:32:05.903 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.903 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.903 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.903 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:32:05.903 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.567360+0000", 2024-08-17T17:32:05.903 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.474e-06 2024-08-17T17:32:05.903 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.903 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.903 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:32:05.903 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.567371+0000", 2024-08-17T17:32:05.903 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1046999999999999e-05 2024-08-17T17:32:05.903 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.903 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.903 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:32:05.904 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.567383+0000", 2024-08-17T17:32:05.904 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1663e-05 2024-08-17T17:32:05.904 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.904 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.904 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:32:05.904 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.567392+0000", 2024-08-17T17:32:05.904 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6400000000000003e-06 2024-08-17T17:32:05.904 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.904 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.904 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:32:05.904 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.567539+0000", 2024-08-17T17:32:05.904 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014705400000000001 2024-08-17T17:32:05.904 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.904 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.904 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:32:05.904 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.567561+0000", 2024-08-17T17:32:05.905 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2118999999999999e-05 2024-08-17T17:32:05.905 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.905 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.905 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:32:05.905 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.809326+0000", 2024-08-17T17:32:05.905 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.24176547100000001 2024-08-17T17:32:05.905 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.905 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.905 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:32:05.905 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.809373+0000", 2024-08-17T17:32:05.905 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7098999999999997e-05 2024-08-17T17:32:05.905 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.905 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:32:05.905 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.905 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.905 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.906 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:05 smithi003 ceph-mon[103344]: pgmap v748: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 1.4 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s; 9618/69054 objects degraded (13.928%) 2024-08-17T17:32:05.906 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:05 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2648854855' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:32:05.906 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45491.0:191525 6.9 6:990fd26b:::benchmark_data_smithi003_175043_object92876:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-08-17T17:32:05.907 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:28:00.570283+0000", 2024-08-17T17:32:05.907 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 244.94716840199999, 2024-08-17T17:32:05.907 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.244092686, 2024-08-17T17:32:05.907 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:32:05.907 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:32:05.907 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:32:05.907 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:32:05.907 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45491", 2024-08-17T17:32:05.907 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:32:05.907 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 191525 2024-08-17T17:32:05.907 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.907 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:32:05.907 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.907 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:32:05.908 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.570283+0000", 2024-08-17T17:32:05.908 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.908 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.908 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.908 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:32:05.908 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.570283+0000", 2024-08-17T17:32:05.908 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.908 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.908 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.908 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:32:05.908 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.570285+0000", 2024-08-17T17:32:05.908 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.44e-06 2024-08-17T17:32:05.908 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.908 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.908 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:32:05.909 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.570291+0000", 2024-08-17T17:32:05.909 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6120000000000004e-06 2024-08-17T17:32:05.909 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.909 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.909 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:32:05.909 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.570307+0000", 2024-08-17T17:32:05.909 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5922000000000001e-05 2024-08-17T17:32:05.909 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.909 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.909 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:32:05.909 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.570316+0000", 2024-08-17T17:32:05.909 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7979999999999995e-06 2024-08-17T17:32:05.909 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.909 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.909 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:32:05.910 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.570376+0000", 2024-08-17T17:32:05.910 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0198999999999997e-05 2024-08-17T17:32:05.910 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.910 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.910 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:32:05.910 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.570489+0000", 2024-08-17T17:32:05.910 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011271 2024-08-17T17:32:05.910 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.910 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.910 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:32:05.910 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.570607+0000", 2024-08-17T17:32:05.910 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011849900000000001 2024-08-17T17:32:05.910 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.910 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.911 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:32:05.911 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.808963+0000", 2024-08-17T17:32:05.911 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.23835577099999999 2024-08-17T17:32:05.911 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.911 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.911 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.911 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.809227+0000", 2024-08-17T17:32:05.911 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00026338700000000002 2024-08-17T17:32:05.911 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.911 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.911 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.911 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.814257+0000", 2024-08-17T17:32:05.911 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0050304579999999998 2024-08-17T17:32:05.911 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.912 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.912 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:32:05.912 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.814340+0000", 2024-08-17T17:32:05.912 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3177999999999995e-05 2024-08-17T17:32:05.912 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.912 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.912 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:32:05.912 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.814376+0000", 2024-08-17T17:32:05.912 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5712e-05 2024-08-17T17:32:05.912 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.912 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:32:05.912 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.912 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.912 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.912 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45491.0:191524 6.a e220/214 6:5b98c4b4:::benchmark_data_smithi003_175043_object92875:head v 220'11905, mlcod=220'11902)", 2024-08-17T17:32:05.913 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:28:00.572273+0000", 2024-08-17T17:32:05.913 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 244.945178559, 2024-08-17T17:32:05.913 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.237207371, 2024-08-17T17:32:05.913 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:32:05.913 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:32:05.913 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:32:05.913 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:32:05.913 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.913 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:32:05.913 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.572273+0000", 2024-08-17T17:32:05.913 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.913 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.913 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.913 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:32:05.913 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.572273+0000", 2024-08-17T17:32:05.914 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.914 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.914 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.914 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:32:05.914 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.572274+0000", 2024-08-17T17:32:05.914 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.294e-06 2024-08-17T17:32:05.914 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.914 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.914 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:32:05.914 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.572284+0000", 2024-08-17T17:32:05.914 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7100000000000002e-06 2024-08-17T17:32:05.914 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.914 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.914 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:32:05.915 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.572297+0000", 2024-08-17T17:32:05.915 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3122000000000001e-05 2024-08-17T17:32:05.915 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.915 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.915 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:32:05.915 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.572305+0000", 2024-08-17T17:32:05.915 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5120000000000002e-06 2024-08-17T17:32:05.915 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.915 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.915 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:32:05.915 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.572842+0000", 2024-08-17T17:32:05.915 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00053739800000000002 2024-08-17T17:32:05.915 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.915 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.915 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:32:05.916 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.572863+0000", 2024-08-17T17:32:05.916 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1353e-05 2024-08-17T17:32:05.916 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.916 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.916 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:32:05.916 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.809454+0000", 2024-08-17T17:32:05.916 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.23659016799999999 2024-08-17T17:32:05.916 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.916 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.916 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:32:05.916 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.809480+0000", 2024-08-17T17:32:05.916 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6814000000000001e-05 2024-08-17T17:32:05.916 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.916 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:32:05.916 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.917 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.917 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.917 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45491.0:191527 6.d e220/204 6:b1fdfb8f:::benchmark_data_smithi003_175043_object92878:head v 220'11771, mlcod=220'11769)", 2024-08-17T17:32:05.917 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:28:00.575150+0000", 2024-08-17T17:32:05.917 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 244.94230146300001, 2024-08-17T17:32:05.917 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.25536818100000003, 2024-08-17T17:32:05.917 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:32:05.917 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:32:05.917 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:32:05.917 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:32:05.917 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.917 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:32:05.917 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.575150+0000", 2024-08-17T17:32:05.917 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.917 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.918 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.918 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:32:05.918 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.575150+0000", 2024-08-17T17:32:05.918 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.918 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.918 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.918 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:32:05.918 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.575151+0000", 2024-08-17T17:32:05.918 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8899999999999998e-07 2024-08-17T17:32:05.918 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.918 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.918 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:32:05.918 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.575158+0000", 2024-08-17T17:32:05.918 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.798e-06 2024-08-17T17:32:05.918 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.919 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.919 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:32:05.919 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.809006+0000", 2024-08-17T17:32:05.919 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.233848099 2024-08-17T17:32:05.919 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.919 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.919 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:32:05.919 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.809021+0000", 2024-08-17T17:32:05.919 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4999e-05 2024-08-17T17:32:05.919 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.919 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.919 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:32:05.919 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.809081+0000", 2024-08-17T17:32:05.919 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0380000000000001e-05 2024-08-17T17:32:05.919 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.919 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.920 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:32:05.920 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.809105+0000", 2024-08-17T17:32:05.920 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4040999999999999e-05 2024-08-17T17:32:05.920 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.920 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.920 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:32:05.920 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.830479+0000", 2024-08-17T17:32:05.920 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.021373207000000002 2024-08-17T17:32:05.920 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.920 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.920 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:32:05.920 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.830518+0000", 2024-08-17T17:32:05.920 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9668e-05 2024-08-17T17:32:05.920 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.920 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:32:05.921 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.921 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.921 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.921 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.45491.0:191525 6.9 e220/178 ondisk, result = 0)", 2024-08-17T17:32:05.921 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:28:00.576844+0000", 2024-08-17T17:32:05.921 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 244.94060797500001, 2024-08-17T17:32:05.921 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.23239138000000001, 2024-08-17T17:32:05.921 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:32:05.921 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:32:05.921 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:32:05.921 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:32:05.921 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.921 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:32:05.921 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.576844+0000", 2024-08-17T17:32:05.921 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.922 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.922 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.922 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:32:05.922 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.576844+0000", 2024-08-17T17:32:05.922 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.922 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.922 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.922 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:32:05.922 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.576845+0000", 2024-08-17T17:32:05.922 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.288e-06 2024-08-17T17:32:05.922 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.922 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.922 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:32:05.922 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.576849+0000", 2024-08-17T17:32:05.923 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4009999999999999e-06 2024-08-17T17:32:05.923 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.923 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.923 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:32:05.923 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.809135+0000", 2024-08-17T17:32:05.923 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.23228589799999999 2024-08-17T17:32:05.923 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.923 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.923 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:32:05.923 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.809145+0000", 2024-08-17T17:32:05.923 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.023e-05 2024-08-17T17:32:05.923 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.923 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.923 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:32:05.923 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.809209+0000", 2024-08-17T17:32:05.924 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3991000000000004e-05 2024-08-17T17:32:05.924 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.924 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.924 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:32:05.924 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.809219+0000", 2024-08-17T17:32:05.924 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8390000000000002e-06 2024-08-17T17:32:05.924 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.924 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.924 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:32:05.924 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.809235+0000", 2024-08-17T17:32:05.924 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5733e-05 2024-08-17T17:32:05.924 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.924 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:32:05.924 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.924 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.925 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.925 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:40469 7.4 7:2e557b99:::benchmark_data_smithi003_201798_object40468:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-08-17T17:32:05.925 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:29:00.881959+0000", 2024-08-17T17:32:05.925 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 184.63549211700001, 2024-08-17T17:32:05.925 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.210183284, 2024-08-17T17:32:05.925 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:32:05.925 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:32:05.925 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:32:05.925 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:32:05.925 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:32:05.925 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:32:05.925 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 40469 2024-08-17T17:32:05.925 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.925 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:32:05.925 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.925 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:32:05.926 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.881959+0000", 2024-08-17T17:32:05.926 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.926 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.926 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.926 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:32:05.926 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.881959+0000", 2024-08-17T17:32:05.926 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.926 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.926 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.926 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:32:05.926 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.881961+0000", 2024-08-17T17:32:05.926 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.299e-06 2024-08-17T17:32:05.926 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.926 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.927 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:32:05.927 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.881980+0000", 2024-08-17T17:32:05.927 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9432e-05 2024-08-17T17:32:05.927 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.927 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.927 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:32:05.927 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.881992+0000", 2024-08-17T17:32:05.927 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2111e-05 2024-08-17T17:32:05.927 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.927 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.927 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:32:05.927 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.881998+0000", 2024-08-17T17:32:05.927 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8660000000000002e-06 2024-08-17T17:32:05.927 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.927 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.928 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:32:05.928 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.882700+0000", 2024-08-17T17:32:05.928 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00070178199999999997 2024-08-17T17:32:05.928 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.928 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.928 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:32:05.928 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.882835+0000", 2024-08-17T17:32:05.928 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013538100000000001 2024-08-17T17:32:05.928 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.928 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.928 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:32:05.928 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.882968+0000", 2024-08-17T17:32:05.928 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013308699999999999 2024-08-17T17:32:05.928 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.928 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.929 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:32:05.929 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.884691+0000", 2024-08-17T17:32:05.929 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017223609999999999 2024-08-17T17:32:05.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.929 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.929 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.929 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.886183+0000", 2024-08-17T17:32:05.929 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014918139999999999 2024-08-17T17:32:05.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.929 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.929 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.929 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:01.092042+0000", 2024-08-17T17:32:05.929 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.20585922400000001 2024-08-17T17:32:05.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.929 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.929 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:32:05.930 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:01.092098+0000", 2024-08-17T17:32:05.930 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6474999999999997e-05 2024-08-17T17:32:05.930 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.930 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.930 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:32:05.930 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:01.092143+0000", 2024-08-17T17:32:05.930 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4452000000000001e-05 2024-08-17T17:32:05.930 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.930 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:32:05.930 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.930 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.930 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.930 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:40480 7.9 7:97dedd39:::benchmark_data_smithi003_201798_object40479:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-08-17T17:32:05.931 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:29:00.885840+0000", 2024-08-17T17:32:05.931 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 184.63161165100001, 2024-08-17T17:32:05.931 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.21157109700000001, 2024-08-17T17:32:05.931 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:32:05.931 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:32:05.931 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:32:05.931 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:32:05.931 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:32:05.931 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:32:05.931 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 40480 2024-08-17T17:32:05.931 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.931 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:32:05.931 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.931 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:32:05.931 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.885840+0000", 2024-08-17T17:32:05.932 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.932 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.932 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.932 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:32:05.932 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.885840+0000", 2024-08-17T17:32:05.932 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.932 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.932 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.932 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:32:05.932 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.885841+0000", 2024-08-17T17:32:05.932 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.438e-06 2024-08-17T17:32:05.932 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.932 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.933 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:32:05.933 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.885864+0000", 2024-08-17T17:32:05.933 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2421000000000001e-05 2024-08-17T17:32:05.933 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.933 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.933 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:32:05.933 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.886100+0000", 2024-08-17T17:32:05.933 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00023576599999999999 2024-08-17T17:32:05.933 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.933 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.933 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:32:05.933 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.886109+0000", 2024-08-17T17:32:05.933 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9630000000000004e-06 2024-08-17T17:32:05.934 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.934 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.934 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:32:05.934 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.886350+0000", 2024-08-17T17:32:05.934 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00024187300000000001 2024-08-17T17:32:05.934 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.934 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.934 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:32:05.934 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.886483+0000", 2024-08-17T17:32:05.934 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013242500000000001 2024-08-17T17:32:05.934 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.934 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.934 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,7", 2024-08-17T17:32:05.935 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.886637+0000", 2024-08-17T17:32:05.935 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015450699999999999 2024-08-17T17:32:05.935 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.935 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.935 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.935 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.888630+0000", 2024-08-17T17:32:05.935 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0019929959999999999 2024-08-17T17:32:05.935 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.935 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.935 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:32:05.935 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.889350+0000", 2024-08-17T17:32:05.935 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00072008800000000002 2024-08-17T17:32:05.935 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.935 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.936 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.936 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:01.097264+0000", 2024-08-17T17:32:05.936 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.207913083 2024-08-17T17:32:05.936 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.936 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.936 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:32:05.936 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:01.097369+0000", 2024-08-17T17:32:05.936 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010575 2024-08-17T17:32:05.936 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.936 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.936 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:32:05.936 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:01.097411+0000", 2024-08-17T17:32:05.936 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1786999999999998e-05 2024-08-17T17:32:05.936 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.936 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:32:05.937 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.937 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.937 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.937 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:105833 7.4 7:2de12734:::benchmark_data_smithi003_201798_object105832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-08-17T17:32:05.937 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:30:02.222050+0000", 2024-08-17T17:32:05.937 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 123.295401458, 2024-08-17T17:32:05.937 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.167924356, 2024-08-17T17:32:05.937 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:32:05.937 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:32:05.937 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:32:05.937 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:32:05.937 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:32:05.937 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:32:05.937 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 105833 2024-08-17T17:32:05.937 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.937 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:32:05.938 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.938 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:32:05.938 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.222050+0000", 2024-08-17T17:32:05.938 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.938 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.938 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.938 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:32:05.938 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.222050+0000", 2024-08-17T17:32:05.938 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.938 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.938 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.938 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:32:05.938 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.222052+0000", 2024-08-17T17:32:05.938 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3540000000000001e-06 2024-08-17T17:32:05.939 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.939 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.939 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:32:05.939 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.222074+0000", 2024-08-17T17:32:05.939 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2427999999999999e-05 2024-08-17T17:32:05.939 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.939 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.939 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:32:05.939 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.222242+0000", 2024-08-17T17:32:05.939 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016828000000000001 2024-08-17T17:32:05.939 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.939 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.939 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:32:05.939 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.222249+0000", 2024-08-17T17:32:05.939 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2640000000000002e-06 2024-08-17T17:32:05.940 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.940 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.940 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:32:05.940 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.222795+0000", 2024-08-17T17:32:05.940 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00054555800000000002 2024-08-17T17:32:05.940 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.940 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.940 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:32:05.940 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.223241+0000", 2024-08-17T17:32:05.940 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000446217 2024-08-17T17:32:05.940 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.940 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.940 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:32:05.940 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.223627+0000", 2024-08-17T17:32:05.940 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00038565000000000002 2024-08-17T17:32:05.940 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.941 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.941 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.941 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.228474+0000", 2024-08-17T17:32:05.941 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0048469400000000001 2024-08-17T17:32:05.941 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.941 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.941 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:32:05.941 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.228673+0000", 2024-08-17T17:32:05.941 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000199101 2024-08-17T17:32:05.941 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.941 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.941 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.941 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.389872+0000", 2024-08-17T17:32:05.941 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.16119890100000001 2024-08-17T17:32:05.941 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.941 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.942 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:32:05.942 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.389932+0000", 2024-08-17T17:32:05.942 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0241999999999997e-05 2024-08-17T17:32:05.942 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.942 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.942 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:32:05.942 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:30:02.389975+0000", 2024-08-17T17:32:05.942 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2420999999999999e-05 2024-08-17T17:32:05.942 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.942 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:32:05.942 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.942 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.942 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.942 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:162882 7.9 7:999458ea:::benchmark_data_smithi003_201798_object49660:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-08-17T17:32:05.943 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:31:14.584422+0000", 2024-08-17T17:32:05.943 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 50.933029523999998, 2024-08-17T17:32:05.943 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.167048693, 2024-08-17T17:32:05.943 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:32:05.943 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:32:05.943 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:32:05.943 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:32:05.943 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:32:05.943 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:32:05.943 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 162882 2024-08-17T17:32:05.943 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.943 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:32:05.943 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.943 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:32:05.943 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:14.584422+0000", 2024-08-17T17:32:05.944 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.944 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.944 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.944 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:32:05.944 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:14.584422+0000", 2024-08-17T17:32:05.944 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.944 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.944 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.944 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:32:05.944 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:14.584423+0000", 2024-08-17T17:32:05.944 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.75e-07 2024-08-17T17:32:05.944 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.944 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.945 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:32:05.945 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:14.584426+0000", 2024-08-17T17:32:05.945 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6679999999999999e-06 2024-08-17T17:32:05.945 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.945 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.945 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:32:05.945 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:14.584436+0000", 2024-08-17T17:32:05.945 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7340000000000003e-06 2024-08-17T17:32:05.945 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.945 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.945 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:32:05.945 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:14.584442+0000", 2024-08-17T17:32:05.945 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3160000000000002e-06 2024-08-17T17:32:05.945 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.945 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.946 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:32:05.946 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:14.721991+0000", 2024-08-17T17:32:05.946 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.13754866800000001 2024-08-17T17:32:05.946 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.946 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.946 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:32:05.946 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:14.722098+0000", 2024-08-17T17:32:05.946 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010688600000000001 2024-08-17T17:32:05.946 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.946 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.946 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,7", 2024-08-17T17:32:05.946 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:14.722302+0000", 2024-08-17T17:32:05.946 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020361899999999999 2024-08-17T17:32:05.947 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.947 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.947 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.947 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:14.740640+0000", 2024-08-17T17:32:05.947 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.018338579000000001 2024-08-17T17:32:05.947 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.947 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.947 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.947 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:14.740680+0000", 2024-08-17T17:32:05.947 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9816e-05 2024-08-17T17:32:05.947 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.947 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.947 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:32:05.947 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:14.750941+0000", 2024-08-17T17:32:05.948 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010260574999999999 2024-08-17T17:32:05.948 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.948 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.948 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:32:05.948 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:14.751329+0000", 2024-08-17T17:32:05.948 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000388623 2024-08-17T17:32:05.948 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.948 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.948 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:32:05.948 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:14.751471+0000", 2024-08-17T17:32:05.948 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014153399999999999 2024-08-17T17:32:05.948 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.948 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:32:05.949 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.949 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.949 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.949 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:163945 7.4 7:20801a71:::benchmark_data_smithi003_201798_object50723:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-08-17T17:32:05.949 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:31:16.131902+0000", 2024-08-17T17:32:05.949 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 49.385549458, 2024-08-17T17:32:05.949 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.20096719800000001, 2024-08-17T17:32:05.949 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:32:05.949 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:32:05.949 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:32:05.949 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:32:05.949 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:32:05.949 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:32:05.949 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 163945 2024-08-17T17:32:05.949 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.949 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:32:05.950 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.950 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:32:05.950 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.131902+0000", 2024-08-17T17:32:05.950 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.950 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.950 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.950 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:32:05.950 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.131902+0000", 2024-08-17T17:32:05.950 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.950 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.950 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.950 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:32:05.950 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.131904+0000", 2024-08-17T17:32:05.951 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6589999999999999e-06 2024-08-17T17:32:05.951 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.951 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.951 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:32:05.951 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.131910+0000", 2024-08-17T17:32:05.951 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9510000000000003e-06 2024-08-17T17:32:05.951 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.951 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.951 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:32:05.951 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.131932+0000", 2024-08-17T17:32:05.951 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2327000000000002e-05 2024-08-17T17:32:05.951 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.951 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.951 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:32:05.952 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.131941+0000", 2024-08-17T17:32:05.952 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7569999999999993e-06 2024-08-17T17:32:05.952 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.952 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.952 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:32:05.952 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.140353+0000", 2024-08-17T17:32:05.952 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0084117959999999992 2024-08-17T17:32:05.952 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.952 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.952 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:32:05.952 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.140458+0000", 2024-08-17T17:32:05.952 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000105062 2024-08-17T17:32:05.952 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.952 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.953 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:32:05.953 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.140565+0000", 2024-08-17T17:32:05.953 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000106888 2024-08-17T17:32:05.953 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.953 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.953 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:32:05.953 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.145737+0000", 2024-08-17T17:32:05.953 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0051727079999999998 2024-08-17T17:32:05.953 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.953 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.953 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.953 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.147699+0000", 2024-08-17T17:32:05.953 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0019620000000000002 2024-08-17T17:32:05.953 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.954 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.954 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.954 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.332739+0000", 2024-08-17T17:32:05.954 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18503926200000001 2024-08-17T17:32:05.954 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.954 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.954 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:32:05.954 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.332806+0000", 2024-08-17T17:32:05.954 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7860000000000004e-05 2024-08-17T17:32:05.954 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.954 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.954 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:32:05.954 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.332869+0000", 2024-08-17T17:32:05.955 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2928000000000005e-05 2024-08-17T17:32:05.955 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.955 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:32:05.955 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.955 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.955 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.955 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:163949 7.9 7:90abad18:::benchmark_data_smithi003_201798_object50727:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-08-17T17:32:05.955 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:31:16.142530+0000", 2024-08-17T17:32:05.955 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 49.374921645000001, 2024-08-17T17:32:05.955 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.19359621900000001, 2024-08-17T17:32:05.955 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:32:05.955 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:32:05.955 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:32:05.956 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:32:05.956 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:32:05.956 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:32:05.956 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 163949 2024-08-17T17:32:05.956 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.956 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:32:05.956 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.956 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:32:05.956 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.142530+0000", 2024-08-17T17:32:05.956 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.956 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.956 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.956 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:32:05.956 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.142530+0000", 2024-08-17T17:32:05.957 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.957 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.957 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.957 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:32:05.957 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.142532+0000", 2024-08-17T17:32:05.957 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6610000000000001e-06 2024-08-17T17:32:05.957 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.957 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.957 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:32:05.957 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.142538+0000", 2024-08-17T17:32:05.957 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0079999999999999e-06 2024-08-17T17:32:05.957 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.957 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.958 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:32:05.958 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.142551+0000", 2024-08-17T17:32:05.958 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3556e-05 2024-08-17T17:32:05.958 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.958 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.958 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:32:05.958 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.142559+0000", 2024-08-17T17:32:05.958 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9840000000000001e-06 2024-08-17T17:32:05.958 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.958 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.958 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:32:05.958 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.144707+0000", 2024-08-17T17:32:05.958 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0021477689999999999 2024-08-17T17:32:05.958 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.959 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.959 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:32:05.959 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.144831+0000", 2024-08-17T17:32:05.959 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012394700000000001 2024-08-17T17:32:05.959 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.959 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.959 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,7", 2024-08-17T17:32:05.959 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.144940+0000", 2024-08-17T17:32:05.959 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010878999999999999 2024-08-17T17:32:05.959 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.959 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.959 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:32:05.959 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.148944+0000", 2024-08-17T17:32:05.960 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040046910000000003 2024-08-17T17:32:05.960 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.960 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:05 smithi003 ceph-mon[100615]: pgmap v748: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 1.4 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s; 9618/69054 objects degraded (13.928%) 2024-08-17T17:32:05.960 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:05 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2648854855' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:32:05.960 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:32:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:32:05] "GET /metrics HTTP/1.1" 200 36507 "" "Prometheus/2.51.0" 2024-08-17T17:32:05.960 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.961 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.961 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.149743+0000", 2024-08-17T17:32:05.961 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00079879400000000002 2024-08-17T17:32:05.961 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.961 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.961 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.961 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.336019+0000", 2024-08-17T17:32:05.961 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18627561000000001 2024-08-17T17:32:05.961 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.961 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.961 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:32:05.961 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.336077+0000", 2024-08-17T17:32:05.961 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8655e-05 2024-08-17T17:32:05.961 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.962 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.962 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:32:05.962 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.336126+0000", 2024-08-17T17:32:05.962 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8754000000000001e-05 2024-08-17T17:32:05.962 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.962 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:32:05.962 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.962 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.962 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.962 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:163957 7.9 7:9baf8e36:::benchmark_data_smithi003_201798_object50735:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-08-17T17:32:05.962 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:31:16.152637+0000", 2024-08-17T17:32:05.962 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 49.364814518000003, 2024-08-17T17:32:05.962 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18388579999999999, 2024-08-17T17:32:05.963 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:32:05.963 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:32:05.963 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:32:05.963 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:32:05.963 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:32:05.963 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:32:05.963 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 163957 2024-08-17T17:32:05.963 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.963 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:32:05.963 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.963 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:32:05.963 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.152637+0000", 2024-08-17T17:32:05.963 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.963 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.964 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.964 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:32:05.964 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.152637+0000", 2024-08-17T17:32:05.964 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:05.964 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.964 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.964 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:32:05.964 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.152639+0000", 2024-08-17T17:32:05.964 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5689999999999999e-06 2024-08-17T17:32:05.964 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.964 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.964 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:32:05.964 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.152645+0000", 2024-08-17T17:32:05.965 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9880000000000001e-06 2024-08-17T17:32:05.965 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.965 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.965 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:32:05.965 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.152656+0000", 2024-08-17T17:32:05.965 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1764e-05 2024-08-17T17:32:05.965 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.965 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.965 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:32:05.965 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.152664+0000", 2024-08-17T17:32:05.965 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8809999999999996e-06 2024-08-17T17:32:05.965 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.965 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.966 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:32:05.966 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.152744+0000", 2024-08-17T17:32:05.966 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9264000000000005e-05 2024-08-17T17:32:05.966 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.966 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.966 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:32:05.966 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.152896+0000", 2024-08-17T17:32:05.966 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000152614 2024-08-17T17:32:05.966 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.966 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.966 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,7", 2024-08-17T17:32:05.966 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.153041+0000", 2024-08-17T17:32:05.966 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000144413 2024-08-17T17:32:05.966 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.967 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.967 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.967 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.156503+0000", 2024-08-17T17:32:05.967 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034627490000000002 2024-08-17T17:32:05.967 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.967 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.967 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:32:05.967 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.156615+0000", 2024-08-17T17:32:05.967 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011154099999999999 2024-08-17T17:32:05.967 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.967 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.967 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:05.967 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.336416+0000", 2024-08-17T17:32:05.967 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17980094999999999 2024-08-17T17:32:05.967 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.968 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.968 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:32:05.968 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.336480+0000", 2024-08-17T17:32:05.968 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4494999999999998e-05 2024-08-17T17:32:05.968 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:05.968 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:05.968 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:32:05.968 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.336523+0000", 2024-08-17T17:32:05.968 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2571999999999998e-05 2024-08-17T17:32:05.968 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.968 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:32:05.968 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.968 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:05.968 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:32:05.968 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:32:05.969 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:05.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:05 smithi053 ceph-mon[89963]: pgmap v748: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 1.4 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s; 9618/69054 objects degraded (13.928%) 2024-08-17T17:32:05.969 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:05 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2648854855' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:32:06.001 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:06.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:06.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:06.103 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:06.133 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:06.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:06.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:06.235 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:06.267 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:06.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:06.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:06.369 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:06.398 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:06.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:06.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:06.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:06.543 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:06.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:06.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:06.644 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:06.688 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:06.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:06.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:06.790 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:06.818 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:06.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:06.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:06.920 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:06.952 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:06.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:06.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:07.054 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:07.091 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:07.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:07.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:07.122 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-17T17:32:07.194 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:07.264 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:07.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:07.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:07.366 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:07.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:07.420 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:07.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:07.521 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:07.574 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:07.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:07.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:07.594 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-17T17:32:07.595 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-17T17:32:07.626 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:32:07.677 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:07.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:07.730 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:07.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:07.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:07 smithi003 ceph-mon[100615]: pgmap v749: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 1.3 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 914 op/s; 9295/63222 objects degraded (14.702%) 2024-08-17T17:32:07.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:07 smithi003 ceph-mon[103344]: pgmap v749: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 1.3 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 914 op/s; 9295/63222 objects degraded (14.702%) 2024-08-17T17:32:07.831 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:07.855 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:07.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:07.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:07.958 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:07.985 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:07.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:07.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:08.086 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:08.116 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:08.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:08.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:08.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:07 smithi053 ceph-mon[89963]: pgmap v749: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 1.3 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 914 op/s; 9295/63222 objects degraded (14.702%) 2024-08-17T17:32:08.219 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:08.244 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:08.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:08.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:08.345 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:08.371 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:08.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:08.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:08.473 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:08.499 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:08.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:08.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:08.601 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:08.629 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:08.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:08.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:08.731 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:08.758 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:08.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:08.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:08.860 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:08.902 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:08.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:08.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:09.004 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:09.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:32:09.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:32:09.031 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:09.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:09.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:09.097 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:32:09.133 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:09.174 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:09.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:09.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:09.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:32:09.276 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:09.302 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:09.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:09.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:09.404 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:09.430 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:09.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:09.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:09.532 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:09.558 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:09.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:09.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:09.595 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:32:09.659 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:09.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:09.727 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:09.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:09.755 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-08-17T17:32:09.828 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:09.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:09.873 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:09.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:09.957 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:09 smithi003 ceph-mon[100615]: pgmap v750: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 1.2 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 942 op/s; 8717/59547 objects degraded (14.639%) 2024-08-17T17:32:09.958 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:09 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/882096678' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:32:09.958 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:09 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 8717/59547 objects degraded (14.639%), 14 pgs degraded (PG_DEGRADED) 2024-08-17T17:32:09.958 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:09 smithi003 ceph-mon[103344]: pgmap v750: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 1.2 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 942 op/s; 8717/59547 objects degraded (14.639%) 2024-08-17T17:32:09.958 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:09 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/882096678' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:32:09.958 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:09 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 8717/59547 objects degraded (14.639%), 14 pgs degraded (PG_DEGRADED) 2024-08-17T17:32:09.975 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:10.002 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:10.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:10.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:10.104 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:10.130 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:10.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:10.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:10.214 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:09 smithi053 ceph-mon[89963]: pgmap v750: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 1.2 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 942 op/s; 8717/59547 objects degraded (14.639%) 2024-08-17T17:32:10.214 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:09 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/882096678' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:32:10.214 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:09 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 8717/59547 objects degraded (14.639%), 14 pgs degraded (PG_DEGRADED) 2024-08-17T17:32:10.232 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:10.275 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:10.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:10.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:10.377 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:10.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:10.406 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:10.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:10.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:32:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:32:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:32:10.510 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:10.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:10.549 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:10.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:10.650 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:10.687 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:10.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:10.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:10.789 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:10.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:10.837 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:10.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:10.937 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:10.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:10.978 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:10.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:11.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:10 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/882096678' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-08-17T17:32:11.026 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:10 smithi003 ceph-mon[103344]: osdmap e253: 8 total, 7 up, 7 in 2024-08-17T17:32:11.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:10 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/882096678' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-08-17T17:32:11.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:10 smithi003 ceph-mon[100615]: osdmap e253: 8 total, 7 up, 7 in 2024-08-17T17:32:11.080 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:11.120 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:11.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:11.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:11.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:10 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/882096678' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-08-17T17:32:11.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:10 smithi053 ceph-mon[89963]: osdmap e253: 8 total, 7 up, 7 in 2024-08-17T17:32:11.223 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:11.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:11.280 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:11.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:11.381 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:11.409 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:11.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:11.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:11.511 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:11.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:11.552 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:11.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:11.653 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:11.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:11.690 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:11.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:11.791 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:11.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:11.845 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:11.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:11.946 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:11.973 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:11.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:11.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:12.020 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:11 smithi003 ceph-mon[103344]: pgmap v752: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 1013 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s; 7180/48498 objects degraded (14.805%) 2024-08-17T17:32:12.021 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:11 smithi003 ceph-mon[100615]: pgmap v752: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 1013 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s; 7180/48498 objects degraded (14.805%) 2024-08-17T17:32:12.074 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:12.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:12.110 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:12.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:12.211 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:12.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:11 smithi053 ceph-mon[89963]: pgmap v752: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 1013 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s; 7180/48498 objects degraded (14.805%) 2024-08-17T17:32:12.237 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:12.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:12.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:12.339 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:12.367 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:12.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:12.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:12.469 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:12.495 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:12.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:12.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:12.597 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:12.637 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:12.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:12.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:12.738 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:12.768 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:12.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:12.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:12.870 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:12.899 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:12.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:12.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:13.001 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:13.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:12 smithi003 ceph-mon[103344]: pgmap v753: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 1013 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s; 7180/48498 objects degraded (14.805%) 2024-08-17T17:32:13.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:12 smithi003 ceph-mon[100615]: pgmap v753: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 1013 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s; 7180/48498 objects degraded (14.805%) 2024-08-17T17:32:13.032 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:13.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:13.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:13.090 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:32:13.133 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:13.160 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:13.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:13.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:13.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:12 smithi053 ceph-mon[89963]: pgmap v753: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 1013 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s; 7180/48498 objects degraded (14.805%) 2024-08-17T17:32:13.263 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:13.291 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:13.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:13.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:13.393 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:13.402 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:32:13.403 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:32:13.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:13.403 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:214896 7.e 7:71f0f2e5:::benchmark_data_smithi003_201798_object101674:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-08-17T17:32:13.403 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:32:13.388317+0000", 2024-08-17T17:32:13.403 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.012173816, 2024-08-17T17:32:13.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012204522000000001, 2024-08-17T17:32:13.403 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:32:13.403 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:32:13.403 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:32:13.403 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:32:13.403 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:32:13.403 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:32:13.403 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 214896 2024-08-17T17:32:13.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:13.404 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:32:13.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:13.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:32:13.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:32:13.388317+0000", 2024-08-17T17:32:13.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:13.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:13.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:13.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:32:13.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:32:13.388317+0000", 2024-08-17T17:32:13.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:13.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:13.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:13.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:32:13.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:32:13.388319+0000", 2024-08-17T17:32:13.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8369999999999999e-06 2024-08-17T17:32:13.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:13.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:13.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:32:13.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:32:13.388325+0000", 2024-08-17T17:32:13.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1820000000000003e-06 2024-08-17T17:32:13.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:13.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:13.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:32:13.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:32:13.388340+0000", 2024-08-17T17:32:13.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4769999999999999e-05 2024-08-17T17:32:13.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:13.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:13.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:32:13.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:32:13.388349+0000", 2024-08-17T17:32:13.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8400000000000001e-06 2024-08-17T17:32:13.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:13.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:13.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:32:13.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:32:13.388419+0000", 2024-08-17T17:32:13.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9636999999999998e-05 2024-08-17T17:32:13.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:13.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:13.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:32:13.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:32:13.388542+0000", 2024-08-17T17:32:13.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012364099999999999 2024-08-17T17:32:13.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:13.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:13.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:32:13.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:32:13.388660+0000", 2024-08-17T17:32:13.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011784499999999999 2024-08-17T17:32:13.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:13.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:13.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:32:13.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:32:13.394387+0000", 2024-08-17T17:32:13.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0057269909999999999 2024-08-17T17:32:13.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:13.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:13.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:32:13.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:32:13.397657+0000", 2024-08-17T17:32:13.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032695599999999999 2024-08-17T17:32:13.408 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:13.408 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:32:13.408 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:13.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:13.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:13.408 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45767.0:214898 7.8 e253/226 7:150c3f86:::benchmark_data_smithi003_201798_object101676:head v 253'13854, mlcod=253'13851)", 2024-08-17T17:32:13.408 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:32:13.394820+0000", 2024-08-17T17:32:13.408 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0056712610000000004, 2024-08-17T17:32:13.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0057821239999999996, 2024-08-17T17:32:13.409 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:32:13.409 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:32:13.409 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:32:13.409 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:32:13.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:13.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:32:13.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:32:13.394820+0000", 2024-08-17T17:32:13.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:13.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:13.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:13.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:32:13.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:32:13.394820+0000", 2024-08-17T17:32:13.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:13.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:13.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:13.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:32:13.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:32:13.394821+0000", 2024-08-17T17:32:13.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0070000000000001e-06 2024-08-17T17:32:13.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:13.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:13.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:32:13.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:32:13.394827+0000", 2024-08-17T17:32:13.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4230000000000002e-06 2024-08-17T17:32:13.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:13.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:13.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:32:13.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:32:13.394840+0000", 2024-08-17T17:32:13.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3052e-05 2024-08-17T17:32:13.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:13.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:13.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:32:13.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:32:13.394848+0000", 2024-08-17T17:32:13.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8250000000000005e-06 2024-08-17T17:32:13.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:13.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:13.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:32:13.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:32:13.394951+0000", 2024-08-17T17:32:13.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010255900000000001 2024-08-17T17:32:13.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:13.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:13.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:32:13.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:32:13.394973+0000", 2024-08-17T17:32:13.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2189e-05 2024-08-17T17:32:13.412 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:13.412 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:32:13.412 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:13.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:13.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:13.412 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45767.0:214902 7.5 e253/226 7:a8c1ba27:::benchmark_data_smithi003_201798_object101680:head v 253'13480, mlcod=253'13478)", 2024-08-17T17:32:13.413 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:32:13.399735+0000", 2024-08-17T17:32:13.413 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.00075592499999999996, 2024-08-17T17:32:13.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00091588399999999995, 2024-08-17T17:32:13.413 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:32:13.413 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:32:13.413 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:32:13.413 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:32:13.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:13.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:32:13.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:32:13.399735+0000", 2024-08-17T17:32:13.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:13.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:13.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:13.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:32:13.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:32:13.399735+0000", 2024-08-17T17:32:13.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:32:13.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:13.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:13.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:32:13.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:32:13.399737+0000", 2024-08-17T17:32:13.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4160000000000001e-06 2024-08-17T17:32:13.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:13.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:13.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:32:13.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:32:13.399744+0000", 2024-08-17T17:32:13.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9189999999999999e-06 2024-08-17T17:32:13.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:13.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:13.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:32:13.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:32:13.399760+0000", 2024-08-17T17:32:13.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5792000000000001e-05 2024-08-17T17:32:13.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:13.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:13.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:32:13.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:32:13.399772+0000", 2024-08-17T17:32:13.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2098e-05 2024-08-17T17:32:13.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:13.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:13.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:32:13.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:32:13.399839+0000", 2024-08-17T17:32:13.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6180999999999995e-05 2024-08-17T17:32:13.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:32:13.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:32:13.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:32:13.416 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:32:13.399857+0000", 2024-08-17T17:32:13.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8292e-05 2024-08-17T17:32:13.416 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:13.416 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:32:13.416 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:13.416 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:32:13.416 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-08-17T17:32:13.416 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 3 2024-08-17T17:32:13.416 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:32:13.421 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:13.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:13.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:13.523 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:13.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:13.557 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:13.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:13.658 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:13.684 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:13.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:13.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:13.786 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:13.817 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:13.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:13.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:13.843 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-08-17T17:32:13.920 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:13.959 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:13.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:13.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:14.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:13 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3123543699' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:32:14.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:13 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:32:14.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:13 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3123543699' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:32:14.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:13 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:32:14.061 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:14.087 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:14.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:14.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:14.189 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:14.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:13 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3123543699' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:32:14.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:13 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:32:14.230 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:14.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:14.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:14.331 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:14.362 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:14.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:14.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:14.464 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:14.503 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:14.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:14.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:14.605 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:14.646 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:14.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:14.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:14.748 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:14.787 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:14.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:14.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:14.889 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:14.927 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:14.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:14.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:15.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:14 smithi003 ceph-mon[103344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:32:15.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:14 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:32:15.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:14 smithi003 ceph-mon[103344]: osdmap e254: 8 total, 7 up, 7 in 2024-08-17T17:32:15.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:14 smithi003 ceph-mon[103344]: pgmap v755: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 823 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 993 op/s; 5340/39351 objects degraded (13.570%) 2024-08-17T17:32:15.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:14 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 7180/48498 objects degraded (14.805%), 14 pgs degraded (PG_DEGRADED) 2024-08-17T17:32:15.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:14 smithi003 ceph-mon[100615]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:32:15.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:14 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:32:15.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:14 smithi003 ceph-mon[100615]: osdmap e254: 8 total, 7 up, 7 in 2024-08-17T17:32:15.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:14 smithi003 ceph-mon[100615]: pgmap v755: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 823 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 993 op/s; 5340/39351 objects degraded (13.570%) 2024-08-17T17:32:15.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:14 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 7180/48498 objects degraded (14.805%), 14 pgs degraded (PG_DEGRADED) 2024-08-17T17:32:15.029 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:15.055 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:15.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:15.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:15.157 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:15.188 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:15.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:15.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:15.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:14 smithi053 ceph-mon[89963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:32:15.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:14 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:32:15.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:14 smithi053 ceph-mon[89963]: osdmap e254: 8 total, 7 up, 7 in 2024-08-17T17:32:15.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:14 smithi053 ceph-mon[89963]: pgmap v755: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 823 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 993 op/s; 5340/39351 objects degraded (13.570%) 2024-08-17T17:32:15.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:14 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 7180/48498 objects degraded (14.805%), 14 pgs degraded (PG_DEGRADED) 2024-08-17T17:32:15.290 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:15.325 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:15.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:15.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:15.427 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:15.463 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:15.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:15.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:15.565 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:15.594 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:15.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:15.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:15.695 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:15.722 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-08-17T17:32:15.723 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:32:15.723 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-08-17T17:32:15.723 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.458420 2024-08-17T17:32:15.723 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 7 0.45842024308304896 2024-08-17T17:32:15.724 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:15.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:15.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:15.777 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-17T17:32:15.825 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:15.850 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:15.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:15.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:15.935 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:32:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:32:15] "GET /metrics HTTP/1.1" 200 36503 "" "Prometheus/2.51.0" 2024-08-17T17:32:15.952 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:16.020 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:16.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:16.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:16.122 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:16.152 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:16.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:16.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:16.254 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:16.278 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:16.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:16.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:16.380 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:16.408 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:16.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:16.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:16.509 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:16.541 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:16.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:16.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:16.643 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:16.675 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:16.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:16.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:16.776 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:16.802 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:16.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:16.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:16.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:32:16.866 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:32:16.871 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:32:16.904 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:16.927 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:16.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:16.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:17.029 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:17.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:17.094 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:17.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:17.195 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:17.218 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:17.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:17.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:17.320 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:17.337 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:17 smithi003 ceph-mon[100615]: pgmap v756: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 707 MiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 1.07k op/s; 5015/33789 objects degraded (14.842%) 2024-08-17T17:32:17.337 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:17 smithi003 ceph-mon[103344]: pgmap v756: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 707 MiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 1.07k op/s; 5015/33789 objects degraded (14.842%) 2024-08-17T17:32:17.347 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:17.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:17.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:17.448 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:17.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:17 smithi053 ceph-mon[89963]: pgmap v756: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 707 MiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 1.07k op/s; 5015/33789 objects degraded (14.842%) 2024-08-17T17:32:17.491 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:17.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:17.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:17.592 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:17.621 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:17.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:17.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:17.723 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:17.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:17.766 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:17.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:17.867 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:17.912 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:17.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:17.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:18.014 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:18.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:18.057 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:18.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:18.158 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:18.189 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:18.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:18.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:18.291 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:18.318 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:18.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:18.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:18.420 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:18.461 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:18.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:18.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:18.540 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:32:18.540 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:32:18.540 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-17T17:32:18.563 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:18.583 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:32:18.583 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:32:18.589 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:18.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:18.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:18.620 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:32:18.620 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:32:18.638 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:32:18.639 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:32:18.654 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:32:18.655 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:32:18.683 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:32:18.683 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:32:18.690 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:18.715 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:32:18.715 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:32:18.727 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:18.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:18.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:18.829 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:18.858 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:18.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:18.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:18.960 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:18.989 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:18.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:18.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:19.091 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:19.119 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:19.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:19.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:19.221 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:19.250 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:19.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:19.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:19.352 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:19.368 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:19 smithi003 ceph-mon[103344]: pgmap v757: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 630 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 774 op/s; 4395/30087 objects degraded (14.608%) 2024-08-17T17:32:19.368 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:19 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3663908447' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:32:19.368 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:19 smithi003 ceph-mon[100615]: pgmap v757: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 630 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 774 op/s; 4395/30087 objects degraded (14.608%) 2024-08-17T17:32:19.368 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:19 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3663908447' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:32:19.378 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:19.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:19.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:19.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:19 smithi053 ceph-mon[89963]: pgmap v757: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 630 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 774 op/s; 4395/30087 objects degraded (14.608%) 2024-08-17T17:32:19.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:19 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3663908447' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:32:19.479 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:19.518 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:19.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:19.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:19.620 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:19.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:19.647 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:19.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:19.749 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:19.786 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:19.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:19.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:19.888 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:19.924 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:19.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:19.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:20.026 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:20.051 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:20.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:20.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:20.153 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:20.185 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:20.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:20.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:20.197 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:20 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 4395/30087 objects degraded (14.608%), 14 pgs degraded (PG_DEGRADED) 2024-08-17T17:32:20.198 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:20 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 4395/30087 objects degraded (14.608%), 14 pgs degraded (PG_DEGRADED) 2024-08-17T17:32:20.287 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:20.315 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:20.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:20.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:20.417 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:20.446 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:20.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:20.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:20.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:20 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 4395/30087 objects degraded (14.608%), 14 pgs degraded (PG_DEGRADED) 2024-08-17T17:32:20.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:32:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:32:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:32:20.548 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:20.577 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:20.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:20.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:20.679 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:20.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:20.717 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:20.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:20.818 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:20.851 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:20.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:20.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:20.903 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-17T17:32:20.903 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-17T17:32:20.954 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:20.981 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:20.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:20.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:21.083 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:21.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:21.113 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:21.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:21.214 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:21.251 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:21.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:21.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:21.353 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:21.392 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:21.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:21.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:21.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:21 smithi053 ceph-mon[89963]: pgmap v758: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 399 MiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 982 op/s; 2845/19008 objects degraded (14.967%) 2024-08-17T17:32:21.486 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:21 smithi003 ceph-mon[100615]: pgmap v758: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 399 MiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 982 op/s; 2845/19008 objects degraded (14.967%) 2024-08-17T17:32:21.486 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:21 smithi003 ceph-mon[103344]: pgmap v758: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 399 MiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 982 op/s; 2845/19008 objects degraded (14.967%) 2024-08-17T17:32:21.495 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:21.528 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:21.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:21.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:21.630 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:21.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:21.662 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:21.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:21.763 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:21.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:21.804 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:21.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:21.905 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:21.945 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:21.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:21.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:22.047 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:22.076 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:22.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:22.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:22.111 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:32:22.112 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:32:22.178 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:22.206 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:22.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:22.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:22.308 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:22.336 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:22.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:22.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:22.438 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:22.466 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:22.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:22.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:22.568 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:22.610 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:22.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:22.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:22.712 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:22.754 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:22.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:22.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:22.798 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:32:22.833 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-17T17:32:22.856 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:22.883 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:22.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:22.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:22.904 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:32:22.985 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:23.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:23.051 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:23.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:23.152 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:23.176 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:23.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:23.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:23.278 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:23.311 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:23.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:23.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:23.357 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:23 smithi003 ceph-mon[103344]: pgmap v759: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 399 MiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 982 op/s; 2845/19008 objects degraded (14.967%) 2024-08-17T17:32:23.357 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:32:23.357 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:32:23.357 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:23 smithi003 ceph-mon[100615]: pgmap v759: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 399 MiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 982 op/s; 2845/19008 objects degraded (14.967%) 2024-08-17T17:32:23.357 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:32:23.358 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:32:23.413 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:23.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:23.442 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:23.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:23.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:23 smithi053 ceph-mon[89963]: pgmap v759: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 399 MiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 982 op/s; 2845/19008 objects degraded (14.967%) 2024-08-17T17:32:23.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:32:23.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:32:23.543 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:23.568 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:23.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:23.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:23.670 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:23.698 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:23.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:23.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:23.800 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:23.828 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:23.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:23.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:23.930 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:23.970 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:23.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:23.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:24.072 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:24.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:24.138 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:24.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:24.139 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.7 primary-affinity to 0.45842 (8300432) 2024-08-17T17:32:24.152 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:32:24.239 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:24.264 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:24.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:24.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:24.365 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:24.388 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:24.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:24.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:24.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:24 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3134857740' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.45842024308304896} : dispatch 2024-08-17T17:32:24.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:24 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.45842024308304896} : dispatch 2024-08-17T17:32:24.489 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:24.511 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:24.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:24.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:24.522 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:24 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3134857740' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.45842024308304896} : dispatch 2024-08-17T17:32:24.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:24 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.45842024308304896} : dispatch 2024-08-17T17:32:24.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:24 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3134857740' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.45842024308304896} : dispatch 2024-08-17T17:32:24.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:24 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.45842024308304896} : dispatch 2024-08-17T17:32:24.612 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:24.634 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:24.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:24.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:24.736 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:24.757 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:24.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:24.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:24.859 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:24.881 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:24.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:24.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:24.983 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:25.075 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:25.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:25.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:25.176 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:25.208 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:25.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:25.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:25.310 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:25.342 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:25.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:25.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:25.444 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:25.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:25 smithi053 ceph-mon[89963]: pgmap v760: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 200 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s; 932/9468 objects degraded (9.844%) 2024-08-17T17:32:25.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:25 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.45842024308304896}]': finished 2024-08-17T17:32:25.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:25 smithi053 ceph-mon[89963]: osdmap e255: 8 total, 7 up, 7 in 2024-08-17T17:32:25.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:25 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 932/9468 objects degraded (9.844%), 14 pgs degraded (PG_DEGRADED) 2024-08-17T17:32:25.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:25 smithi053 ceph-mon[89963]: osdmap e256: 8 total, 7 up, 7 in 2024-08-17T17:32:25.488 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:25.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:25.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:25.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:25 smithi003 ceph-mon[103344]: pgmap v760: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 200 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s; 932/9468 objects degraded (9.844%) 2024-08-17T17:32:25.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:25 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.45842024308304896}]': finished 2024-08-17T17:32:25.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:25 smithi003 ceph-mon[103344]: osdmap e255: 8 total, 7 up, 7 in 2024-08-17T17:32:25.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:25 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 932/9468 objects degraded (9.844%), 14 pgs degraded (PG_DEGRADED) 2024-08-17T17:32:25.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:25 smithi003 ceph-mon[103344]: osdmap e256: 8 total, 7 up, 7 in 2024-08-17T17:32:25.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:25 smithi003 ceph-mon[100615]: pgmap v760: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 200 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s; 932/9468 objects degraded (9.844%) 2024-08-17T17:32:25.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:25 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.45842024308304896}]': finished 2024-08-17T17:32:25.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:25 smithi003 ceph-mon[100615]: osdmap e255: 8 total, 7 up, 7 in 2024-08-17T17:32:25.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:25 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 932/9468 objects degraded (9.844%), 14 pgs degraded (PG_DEGRADED) 2024-08-17T17:32:25.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:25 smithi003 ceph-mon[100615]: osdmap e256: 8 total, 7 up, 7 in 2024-08-17T17:32:25.590 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:25.617 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:25.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:25.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:25.719 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:25.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:25.747 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:25.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:25.848 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:25.887 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:25.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:25.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:25.988 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:26.016 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:26.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:26.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:26.022 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:32:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:32:25] "GET /metrics HTTP/1.1" 200 36501 "" "Prometheus/2.51.0" 2024-08-17T17:32:26.118 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:26.134 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Removed 113219 objects 2024-08-17T17:32:26.134 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Clean up completed and total clean up time :134.561 2024-08-17T17:32:26.144 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:26.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:26.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:26.245 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:26.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:26.269 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:26.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:26.357 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:32:26.369 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:26.393 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:26.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:26.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:26.495 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:26.517 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:26.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:26.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:26.619 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:26.641 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:26.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:26.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:26.743 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:26.765 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:26.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:26.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:26.867 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:26.893 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:26.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:26.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:26.994 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:27.017 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:27.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:27.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:27.119 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:27.142 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:27.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:27.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:27.231 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-08-17T17:32:27.231 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-08-17T17:32:27.244 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:27.266 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:27.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:27.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:27.333 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:27 smithi003 ceph-mon[103344]: pgmap v763: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 127 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s; 639/5985 objects degraded (10.677%) 2024-08-17T17:32:27.334 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:27 smithi003 ceph-mon[100615]: pgmap v763: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 127 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s; 639/5985 objects degraded (10.677%) 2024-08-17T17:32:27.368 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:27.391 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:27.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:27.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:27.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:27 smithi053 ceph-mon[89963]: pgmap v763: 89 pgs: 31 active+undersized, 14 active+undersized+degraded, 44 active+clean; 127 MiB data, 4.6 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s; 639/5985 objects degraded (10.677%) 2024-08-17T17:32:27.493 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:27.515 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:27.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:27.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:27.616 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:27.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:27.640 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:27.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:27.741 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:27.763 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:27.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:27.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:27.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:27.888 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:27.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:27.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:27.990 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:28.015 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:28.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:28.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:28.117 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:28.139 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:28.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:28.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:28.241 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:28.263 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:28.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:28.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:28.365 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:28.391 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:28.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:28.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:28.493 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:28.519 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:28.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:28.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:28.620 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:28.643 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:28.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:28.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:28.745 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:28.768 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:28.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:28.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:28.870 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:28.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:28.898 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:28.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:28.999 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:29.025 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:29.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:29.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:29.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:29.151 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:29.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:29.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:29.253 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:29.276 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:29.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:29.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:29.279 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:32:29.279 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:32:29.378 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:29.401 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:29.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:29.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:29.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:29 smithi053 ceph-mon[89963]: pgmap v764: 89 pgs: 33 active+undersized, 12 active+undersized+degraded, 44 active+clean; 72 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 653 op/s; 383/3309 objects degraded (11.574%) 2024-08-17T17:32:29.503 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:29.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:29 smithi003 ceph-mon[100615]: pgmap v764: 89 pgs: 33 active+undersized, 12 active+undersized+degraded, 44 active+clean; 72 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 653 op/s; 383/3309 objects degraded (11.574%) 2024-08-17T17:32:29.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:29 smithi003 ceph-mon[103344]: pgmap v764: 89 pgs: 33 active+undersized, 12 active+undersized+degraded, 44 active+clean; 72 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 653 op/s; 383/3309 objects degraded (11.574%) 2024-08-17T17:32:29.529 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:29.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:29.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:29.630 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:29.657 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:29.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:29.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:29.759 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:29.784 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:29.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:29.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:29.886 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:29.911 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:29.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:29.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:30.013 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:30.021 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-17T17:32:30.040 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:30.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:30.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:30.142 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:30.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:30.169 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:30.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:30.270 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:30.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:30 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 383/3309 objects degraded (11.574%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T17:32:30.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:30 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 383/3309 objects degraded (11.574%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T17:32:30.296 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:30.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:30.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:30.398 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:30.420 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:30.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:30.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:30.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:30 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 383/3309 objects degraded (11.574%), 12 pgs degraded (PG_DEGRADED) 2024-08-17T17:32:30.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:32:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:32:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:32:30.522 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:30.544 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:30.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:30.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:30.646 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:30.668 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:30.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:30.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:30.770 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:30.792 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:30.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:30.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:30.894 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:30.917 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:30.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:30.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:31.018 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:31.041 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:31.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:31.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:31.121 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:32:31.143 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:31.150 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:32:31.164 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-08-17T17:32:31.170 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:31.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:31.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:31.271 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:31.295 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:31.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:31.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:31.397 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:31.407 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:31 smithi003 ceph-mon[103344]: pgmap v765: 89 pgs: 33 active+undersized, 12 active+undersized+degraded, 44 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 788 op/s; 7/84 objects degraded (8.333%) 2024-08-17T17:32:31.407 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:31 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3945121711' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:32:31.407 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:31 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:32:31.407 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:31 smithi003 ceph-mon[100615]: pgmap v765: 89 pgs: 33 active+undersized, 12 active+undersized+degraded, 44 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 788 op/s; 7/84 objects degraded (8.333%) 2024-08-17T17:32:31.407 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:31 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3945121711' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:32:31.407 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:31 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:32:31.423 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:31.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:31.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:31.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:31 smithi053 ceph-mon[89963]: pgmap v765: 89 pgs: 33 active+undersized, 12 active+undersized+degraded, 44 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 788 op/s; 7/84 objects degraded (8.333%) 2024-08-17T17:32:31.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:31 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3945121711' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:32:31.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:31 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:32:31.525 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:31.548 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:31.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:31.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:31.649 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:31.671 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:31.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:31.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:31.773 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:31.796 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:31.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:31.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:31.856 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:32:31.857 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:32:31.857 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-17T17:32:31.874 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:32:31.875 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:32:31.891 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:32:31.891 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:32:31.897 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:31.906 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:32:31.907 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:32:31.923 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:32:31.923 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:32:31.924 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:31.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:31.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:31.939 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:32:31.940 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:32:31.956 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:32:31.956 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:32:32.025 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:32.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:32.049 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:32.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:32.149 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:32.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:32.173 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:32.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:32.274 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:32.301 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:32.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:32.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:32.403 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:32.425 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:32.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:32.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:32.461 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:32 smithi003 ceph-mon[103344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:32:32.461 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:32 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:32:32.462 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:32 smithi003 ceph-mon[103344]: osdmap e257: 8 total, 7 up, 7 in 2024-08-17T17:32:32.462 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:32 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1089802160' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:32:32.462 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:32 smithi003 ceph-mon[100615]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:32:32.462 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:32 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:32:32.462 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:32 smithi003 ceph-mon[100615]: osdmap e257: 8 total, 7 up, 7 in 2024-08-17T17:32:32.462 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:32 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1089802160' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:32:32.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:32 smithi053 ceph-mon[89963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:32:32.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:32 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:32:32.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:32 smithi053 ceph-mon[89963]: osdmap e257: 8 total, 7 up, 7 in 2024-08-17T17:32:32.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:32 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1089802160' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:32:32.527 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:32.553 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:32.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:32.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:32.655 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:32.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:32.682 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:32.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:32.783 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:32.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:32.809 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:32.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:32.910 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:32.934 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:32.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:32.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:33.036 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:33.062 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:33.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:33.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:33.164 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:33.191 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:33.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:33.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:33.217 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:33 smithi003 ceph-mon[103344]: pgmap v767: 89 pgs: 33 active+undersized, 12 active+undersized+degraded, 44 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 282 op/s; 7/84 objects degraded (8.333%) 2024-08-17T17:32:33.217 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:33 smithi003 ceph-mon[100615]: pgmap v767: 89 pgs: 33 active+undersized, 12 active+undersized+degraded, 44 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 282 op/s; 7/84 objects degraded (8.333%) 2024-08-17T17:32:33.292 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:33.316 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:33.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:33.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:33.417 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:33.441 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:33.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:33.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:33.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:33 smithi053 ceph-mon[89963]: pgmap v767: 89 pgs: 33 active+undersized, 12 active+undersized+degraded, 44 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 282 op/s; 7/84 objects degraded (8.333%) 2024-08-17T17:32:33.543 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:33.566 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:33.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:33.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:33.668 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:33.691 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:33.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:33.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:33.793 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:33.816 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:33.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:33.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:33.918 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:33.945 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:33.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:33.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:34.047 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:34.074 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:34.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:34.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:34.176 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:34.201 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:34.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:34.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:34.303 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:34.329 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:34.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:34.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:34.377 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-17T17:32:34.378 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-17T17:32:34.431 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:34.458 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:34.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:34.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:34.486 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:32:34.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:34.584 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:34.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:34.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:34.686 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:34.712 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:34.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:34.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:34.814 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:34.836 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:34.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:34.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:34.938 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:34.964 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:34.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:34.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:35.066 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:35.092 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:35.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:35.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:35.194 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:35.221 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:35.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:35.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:35.323 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:35.350 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:35.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:35.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:35.451 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:35.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:35 smithi053 ceph-mon[89963]: pgmap v768: 89 pgs: 37 active+undersized, 8 active+undersized+degraded, 44 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 220 op/s; 7/84 objects degraded (8.333%) 2024-08-17T17:32:35.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:35 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 7/84 objects degraded (8.333%), 8 pgs degraded (PG_DEGRADED) 2024-08-17T17:32:35.477 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:35.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:35.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:35.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:35 smithi003 ceph-mon[100615]: pgmap v768: 89 pgs: 37 active+undersized, 8 active+undersized+degraded, 44 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 220 op/s; 7/84 objects degraded (8.333%) 2024-08-17T17:32:35.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:35 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 7/84 objects degraded (8.333%), 8 pgs degraded (PG_DEGRADED) 2024-08-17T17:32:35.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:35 smithi003 ceph-mon[103344]: pgmap v768: 89 pgs: 37 active+undersized, 8 active+undersized+degraded, 44 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 220 op/s; 7/84 objects degraded (8.333%) 2024-08-17T17:32:35.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:35 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 7/84 objects degraded (8.333%), 8 pgs degraded (PG_DEGRADED) 2024-08-17T17:32:35.580 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:35.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:35.607 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:35.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:35.708 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:35.733 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:35.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:35.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:35.834 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:35.861 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:35.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:35.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:35.963 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:35.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:35.990 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:35.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:36.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:32:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:32:35] "GET /metrics HTTP/1.1" 200 36501 "" "Prometheus/2.51.0" 2024-08-17T17:32:36.091 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:36.098 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-08-17T17:32:36.098 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:32:36.098 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-08-17T17:32:36.099 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd in 4 2024-08-17T17:32:36.118 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:36.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:36.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:36.220 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:36.247 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:36.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:36.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:36.349 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:36.375 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:36.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:36.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:36.379 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:32:36.477 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:36.499 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:36.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:36.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:36.601 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:36.627 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:36.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:36.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:36.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:36.750 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:36.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:36.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:36.852 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:36.878 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:36.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:36.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:36.917 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:32:36.980 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:37.007 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:37.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:37.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:37.108 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:37.126 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:32:37.126 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-08-17T17:32:37.127 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-08-17T17:32:37.127 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:32:37.133 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:37.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:37.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:37.182 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_2' removed 2024-08-17T17:32:37.235 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:37.263 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:37.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:37.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:37.365 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:37.387 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:37.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:37.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:37.448 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:37 smithi003 ceph-mon[103344]: pgmap v769: 89 pgs: 38 active+undersized, 7 active+undersized+degraded, 44 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 196 op/s; 7/84 objects degraded (8.333%) 2024-08-17T17:32:37.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:37 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/172702520' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:32:37.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:37 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:32:37.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:37 smithi003 ceph-mon[100615]: pgmap v769: 89 pgs: 38 active+undersized, 7 active+undersized+degraded, 44 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 196 op/s; 7/84 objects degraded (8.333%) 2024-08-17T17:32:37.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:37 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/172702520' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:32:37.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:37 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:32:37.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:37 smithi053 ceph-mon[89963]: pgmap v769: 89 pgs: 38 active+undersized, 7 active+undersized+degraded, 44 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 196 op/s; 7/84 objects degraded (8.333%) 2024-08-17T17:32:37.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:37 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/172702520' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:32:37.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:37 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:32:37.489 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:37.512 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:37.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:37.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:37.614 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:37.636 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:37.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:37.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:37.737 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:37.761 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:37.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:37.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:37.863 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:37.887 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:37.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:37.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:37.988 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:38.017 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:38.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:38.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:38.119 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:38.141 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:38.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:38.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:38.243 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:38.266 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:38.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:38.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:38.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:38 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T17:32:38.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:38 smithi003 ceph-mon[100615]: osdmap e258: 8 total, 7 up, 7 in 2024-08-17T17:32:38.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:32:38.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:32:38.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:38 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T17:32:38.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:38 smithi003 ceph-mon[103344]: osdmap e258: 8 total, 7 up, 7 in 2024-08-17T17:32:38.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:32:38.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:32:38.368 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:38.392 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:38.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:38.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:38.427 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-08-17T17:32:38.428 INFO:tasks.radosbench:Beginning radosbench... 2024-08-17T17:32:38.428 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-08-17T17:32:38.428 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd pool create unique_pool_3 16 2024-08-17T17:32:38.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:38 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T17:32:38.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:38 smithi053 ceph-mon[89963]: osdmap e258: 8 total, 7 up, 7 in 2024-08-17T17:32:38.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:32:38.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:32:38.494 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:38.516 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:38.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:38.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:38.618 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:38.641 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:38.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:38.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:38.742 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:38.769 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:38.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:38.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:38.871 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:38.897 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:38.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:38.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:38.999 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:39.022 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:39.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:39.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:39.081 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:32:39.124 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:39.146 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:39.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:39.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:39.248 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:39.270 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:39.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:39.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:39.372 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:39.393 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:39.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:39.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:39.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:39 smithi053 ceph-mon[89963]: pgmap v771: 73 pgs: 31 active+undersized, 7 active+undersized+degraded, 35 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 7/84 objects degraded (8.333%) 2024-08-17T17:32:39.470 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:39 smithi003 ceph-mon[103344]: pgmap v771: 73 pgs: 31 active+undersized, 7 active+undersized+degraded, 35 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 7/84 objects degraded (8.333%) 2024-08-17T17:32:39.470 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:39 smithi003 ceph-mon[100615]: pgmap v771: 73 pgs: 31 active+undersized, 7 active+undersized+degraded, 35 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 7/84 objects degraded (8.333%) 2024-08-17T17:32:39.494 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-17T17:32:39.496 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:39.522 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:39.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:39.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:39.542 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:32:39.624 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:39.646 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:39.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:39.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:39.748 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:39.775 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:39.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:39.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:39.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:39.898 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:39.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:39.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:40.000 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:40.023 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:40.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:40.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:40.125 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:40.150 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:40.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:40.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:40.252 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:40.275 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:40.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:40.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:40.376 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:40.399 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:40.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:40.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:40.420 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:40 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 7/84 objects degraded (8.333%), 7 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-08-17T17:32:40.420 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:40 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:32:40.420 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:40 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 7/84 objects degraded (8.333%), 7 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-08-17T17:32:40.420 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:40 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:32:40.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:40 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 7/84 objects degraded (8.333%), 7 pgs degraded, 37 pgs undersized (PG_DEGRADED) 2024-08-17T17:32:40.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:40 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:32:40.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:32:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:32:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:32:40.500 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:40.523 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:40.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:40.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:40.625 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:40.647 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:40.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:40.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:40.749 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:40.771 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:40.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:40.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:40.873 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:40.897 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:40.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:40.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:40.999 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:41.021 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:41.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:41.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:41.123 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:41.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:41.145 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:41.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:41.246 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:41.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:41.270 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:41.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:41.371 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:41.393 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:41.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:41.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:41.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:41 smithi053 ceph-mon[89963]: pgmap v772: 73 pgs: 31 active+undersized, 7 active+undersized+degraded, 35 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 7/84 objects degraded (8.333%) 2024-08-17T17:32:41.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:41.517 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:41.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:41.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:41 smithi003 ceph-mon[100615]: pgmap v772: 73 pgs: 31 active+undersized, 7 active+undersized+degraded, 35 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 7/84 objects degraded (8.333%) 2024-08-17T17:32:41.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:41 smithi003 ceph-mon[103344]: pgmap v772: 73 pgs: 31 active+undersized, 7 active+undersized+degraded, 35 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 7/84 objects degraded (8.333%) 2024-08-17T17:32:41.619 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:41.642 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:41.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:41.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:41.744 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:41.767 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:41.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:41.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:41.868 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:41.886 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:32:41.891 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:41.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:41.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:41.994 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:42.016 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:42.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:42.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:42.118 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:42.142 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:42.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:42.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:42.191 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-08-17T17:32:42.244 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:42.267 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:42.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:42.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:42.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:42 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1641126546' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:32:42.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:42 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1641126546' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:32:42.370 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:42.392 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:42.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:42.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:42.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:42 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1641126546' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:32:42.494 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:42.516 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:42.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:42.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:42.618 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:42.641 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:42.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:42.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:42.742 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:42.764 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:42.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:42.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:42.866 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:42.889 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:42.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:42.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:42.991 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:43.014 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:43.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:43.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:43.116 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:43.140 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:43.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:43.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:43.241 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:43.265 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:43.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:43.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:43.342 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:43 smithi003 ceph-mon[103344]: pgmap v773: 73 pgs: 31 active+undersized, 7 active+undersized+degraded, 35 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 7/84 objects degraded (8.333%) 2024-08-17T17:32:43.342 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:43 smithi003 ceph-mon[103344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:32:43.342 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:43 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1641126546' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:32:43.342 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:43 smithi003 ceph-mon[103344]: osdmap e259: 8 total, 7 up, 7 in 2024-08-17T17:32:43.342 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:43 smithi003 ceph-mon[100615]: pgmap v773: 73 pgs: 31 active+undersized, 7 active+undersized+degraded, 35 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 7/84 objects degraded (8.333%) 2024-08-17T17:32:43.342 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:43 smithi003 ceph-mon[100615]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:32:43.343 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:43 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1641126546' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:32:43.343 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:43 smithi003 ceph-mon[100615]: osdmap e259: 8 total, 7 up, 7 in 2024-08-17T17:32:43.367 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:43.391 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:43.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:43.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:43.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:43 smithi053 ceph-mon[89963]: pgmap v773: 73 pgs: 31 active+undersized, 7 active+undersized+degraded, 35 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 7/84 objects degraded (8.333%) 2024-08-17T17:32:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:43 smithi053 ceph-mon[89963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:32:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:43 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1641126546' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:32:43.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:43 smithi053 ceph-mon[89963]: osdmap e259: 8 total, 7 up, 7 in 2024-08-17T17:32:43.493 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:43.518 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:43.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:43.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:43.620 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:43.642 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:43.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:43.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:43.744 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:43.768 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:43.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:43.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:43.776 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:32:43.777 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:32:43.777 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-17T17:32:43.795 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:32:43.795 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:32:43.811 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:32:43.812 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:32:43.827 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:32:43.828 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:32:43.843 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:32:43.843 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:32:43.859 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:32:43.859 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:32:43.870 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:43.875 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:32:43.875 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:32:43.892 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:43.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:43.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:43.994 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:44.016 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:44.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:44.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:44.118 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:44.140 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:44.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:44.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:44.242 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:44.264 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:44.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:44.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:44.272 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:44 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/411747946' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:32:44.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:44 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/411747946' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:32:44.366 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:44.388 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:44.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:44.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:44.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:44 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/411747946' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:32:44.490 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:44.514 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:44.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:44.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:44.616 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:44.638 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:44.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:44.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:44.740 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:44.763 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:44.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:44.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:44.865 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:44.887 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:44.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:44.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:44.989 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:45.012 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:45.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:45.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:45.114 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:45.136 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:45.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:45.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:45.237 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:45.244 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:44 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-a[100611]: 2024-08-17T17:32:44.960+0000 7fc4d42c5640 -1 mon.a@0(leader).osd e259 definitely_dead 0 2024-08-17T17:32:45.263 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:45.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:45.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:45.360 INFO:teuthology.orchestra.run.smithi003.stderr:marked in osd.4. 2024-08-17T17:32:45.364 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:45.390 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:45.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:45.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:45.492 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:45.514 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:45.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:45.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:45.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:45 smithi003 ceph-mon[100615]: pgmap v775: 73 pgs: 31 active+undersized, 7 active+undersized+degraded, 35 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 7/84 objects degraded (8.333%) 2024-08-17T17:32:45.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:45 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:32:45.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:45 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:32:45.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:45 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2104816167' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-08-17T17:32:45.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:45 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-08-17T17:32:45.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:45 smithi003 ceph-mon[103344]: pgmap v775: 73 pgs: 31 active+undersized, 7 active+undersized+degraded, 35 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 7/84 objects degraded (8.333%) 2024-08-17T17:32:45.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:45 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:32:45.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:45 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:32:45.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:45 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2104816167' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-08-17T17:32:45.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:45 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-08-17T17:32:45.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:45.638 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:45.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:45.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:45.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:45 smithi053 ceph-mon[89963]: pgmap v775: 73 pgs: 31 active+undersized, 7 active+undersized+degraded, 35 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 7/84 objects degraded (8.333%) 2024-08-17T17:32:45.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:45 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:32:45.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:45 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:32:45.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:45 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2104816167' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-08-17T17:32:45.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:45 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-08-17T17:32:45.740 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:45.763 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:45.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:45.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:45.864 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:45.891 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:45.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:45.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:45.958 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:32:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:32:45] "GET /metrics HTTP/1.1" 200 34149 "" "Prometheus/2.51.0" 2024-08-17T17:32:45.993 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:46.015 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:46.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:46.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:46.117 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:46.139 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:46.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:46.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:46.241 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:46.268 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:46.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:46.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:46.370 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:46.393 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:46.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:46.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:46.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:46.516 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:46.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:46.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:46.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:46 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-08-17T17:32:46.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:46 smithi003 ceph-mon[103344]: osdmap e260: 8 total, 7 up, 8 in 2024-08-17T17:32:46.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:46 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-08-17T17:32:46.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:46 smithi003 ceph-mon[100615]: osdmap e260: 8 total, 7 up, 8 in 2024-08-17T17:32:46.619 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:46.646 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:46.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:46.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:46.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:46 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-08-17T17:32:46.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:46 smithi053 ceph-mon[89963]: osdmap e260: 8 total, 7 up, 8 in 2024-08-17T17:32:46.748 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:46.775 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:46.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:46.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:46.876 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:46.903 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:46.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:46.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:47.004 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:47.031 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:47.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:47.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:47.133 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:47.156 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:47.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:47.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:47.258 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:47.266 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:32:47.280 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-17T17:32:47.280 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-17T17:32:47.285 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:47.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:47.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:47.371 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_3' created 2024-08-17T17:32:47.386 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:47.412 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:47.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:47.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:47.514 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:47.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:47 smithi003 ceph-mon[103344]: pgmap v777: 73 pgs: 31 active+undersized, 7 active+undersized+degraded, 35 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 7/84 objects degraded (8.333%) 2024-08-17T17:32:47.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:47 smithi003 ceph-mon[103344]: osdmap e261: 8 total, 7 up, 8 in 2024-08-17T17:32:47.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:47 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1365202993' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-08-17T17:32:47.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:47 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-08-17T17:32:47.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:47 smithi003 ceph-mon[100615]: pgmap v777: 73 pgs: 31 active+undersized, 7 active+undersized+degraded, 35 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 7/84 objects degraded (8.333%) 2024-08-17T17:32:47.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:47 smithi003 ceph-mon[100615]: osdmap e261: 8 total, 7 up, 8 in 2024-08-17T17:32:47.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:47 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1365202993' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-08-17T17:32:47.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:47 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-08-17T17:32:47.539 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:47.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:47.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:47.641 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:47.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:47.666 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:47.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:47.671 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-08-17T17:32:47.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:47 smithi053 ceph-mon[89963]: pgmap v777: 73 pgs: 31 active+undersized, 7 active+undersized+degraded, 35 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 7/84 objects degraded (8.333%) 2024-08-17T17:32:47.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:47 smithi053 ceph-mon[89963]: osdmap e261: 8 total, 7 up, 8 in 2024-08-17T17:32:47.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:47 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1365202993' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-08-17T17:32:47.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:47 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-08-17T17:32:47.767 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:47.794 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:47.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:47.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:47.896 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:47.922 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:47.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:47.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:47.984 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:32:48.024 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:48.050 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:48.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:48.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:48.152 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:48.179 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:48.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:48.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:48.281 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:48.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:48.309 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:48.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:48.410 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:48.435 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:48.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:48.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:48.537 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:48.562 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:48.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:48.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:48.640 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:48 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-08-17T17:32:48.641 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:48 smithi003 ceph-mon[103344]: osdmap e262: 8 total, 7 up, 8 in 2024-08-17T17:32:48.641 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:48 smithi003 ceph-mon[103344]: osdmap e263: 8 total, 7 up, 8 in 2024-08-17T17:32:48.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:48 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-08-17T17:32:48.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:48 smithi003 ceph-mon[100615]: osdmap e262: 8 total, 7 up, 8 in 2024-08-17T17:32:48.641 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:48 smithi003 ceph-mon[100615]: osdmap e263: 8 total, 7 up, 8 in 2024-08-17T17:32:48.642 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-08-17T17:32:48.664 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:48.707 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:48.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:48.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:48.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:48 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-08-17T17:32:48.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:48 smithi053 ceph-mon[89963]: osdmap e262: 8 total, 7 up, 8 in 2024-08-17T17:32:48.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:48 smithi053 ceph-mon[89963]: osdmap e263: 8 total, 7 up, 8 in 2024-08-17T17:32:48.809 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:48.836 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:48.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:48.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:48.938 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:48.965 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:48.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:48.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:49.067 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:49.093 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:49.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:49.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:49.195 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:49.218 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:49.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:49.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:49.281 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:32:49.320 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:49.347 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:49.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:49.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:49.448 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:49.474 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:32:49.477 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:49.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:49.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:49.554 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:49 smithi003 ceph-mon[103344]: pgmap v780: 89 pgs: 4 creating+peering, 1 active+clean+remapped, 12 unknown, 30 active+undersized, 6 active+undersized+degraded, 36 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 6/84 objects degraded (7.143%); 15 B/s, 0 objects/s recovering 2024-08-17T17:32:49.555 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:49 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 6/84 objects degraded (7.143%), 6 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-08-17T17:32:49.555 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:49 smithi003 ceph-mon[103344]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:32:49.555 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:49 smithi003 ceph-mon[103344]: osdmap e264: 8 total, 7 up, 8 in 2024-08-17T17:32:49.555 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:49 smithi003 ceph-mon[100615]: pgmap v780: 89 pgs: 4 creating+peering, 1 active+clean+remapped, 12 unknown, 30 active+undersized, 6 active+undersized+degraded, 36 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 6/84 objects degraded (7.143%); 15 B/s, 0 objects/s recovering 2024-08-17T17:32:49.555 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:49 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 6/84 objects degraded (7.143%), 6 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-08-17T17:32:49.555 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:49 smithi003 ceph-mon[100615]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:32:49.555 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:49 smithi003 ceph-mon[100615]: osdmap e264: 8 total, 7 up, 8 in 2024-08-17T17:32:49.579 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:49.605 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:49.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:49.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:49.707 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:49.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:49 smithi053 ceph-mon[89963]: pgmap v780: 89 pgs: 4 creating+peering, 1 active+clean+remapped, 12 unknown, 30 active+undersized, 6 active+undersized+degraded, 36 active+clean; 4.6 MiB data, 4.3 GiB used, 622 GiB / 626 GiB avail; 6/84 objects degraded (7.143%); 15 B/s, 0 objects/s recovering 2024-08-17T17:32:49.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:49 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 6/84 objects degraded (7.143%), 6 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-08-17T17:32:49.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:49 smithi053 ceph-mon[89963]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:32:49.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:49 smithi053 ceph-mon[89963]: osdmap e264: 8 total, 7 up, 8 in 2024-08-17T17:32:49.734 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:49.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:49.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:49.835 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:49.862 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:49.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:49.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:49.964 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:49.990 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:49.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:49.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:50.092 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:50.117 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:50.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:50.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:50.148 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:32:50.148 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:32:50.219 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:50.242 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:50.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:50.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:50.344 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:50.372 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:50.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:50.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:50.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:32:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:32:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:32:50.474 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:50.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:50.499 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:50.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:50.600 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:50.627 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:50.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:50.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:50.729 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:50.751 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:50.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:50.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:50.852 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:50.880 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:50.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:50.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:50.981 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:51.009 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:51.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:51.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:51.111 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:51.137 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:51.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:51.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:51.239 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:51.265 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:51.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:51.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:51.367 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:51.394 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:51.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:51.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:51.496 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:51.523 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:51.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:51.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:51.624 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:51.651 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:51.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:51.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:51.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:51 smithi053 ceph-mon[89963]: pgmap v783: 89 pgs: 1 peering, 4 creating+peering, 7 active+clean+remapped, 1 remapped+peering, 23 active+undersized, 3 active+undersized+degraded, 50 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 3/84 objects degraded (3.571%); 1/84 objects misplaced (1.190%); 27 B/s, 8 objects/s recovering 2024-08-17T17:32:51.729 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:51 smithi003 ceph-mon[103344]: pgmap v783: 89 pgs: 1 peering, 4 creating+peering, 7 active+clean+remapped, 1 remapped+peering, 23 active+undersized, 3 active+undersized+degraded, 50 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 3/84 objects degraded (3.571%); 1/84 objects misplaced (1.190%); 27 B/s, 8 objects/s recovering 2024-08-17T17:32:51.729 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:51 smithi003 ceph-mon[100615]: pgmap v783: 89 pgs: 1 peering, 4 creating+peering, 7 active+clean+remapped, 1 remapped+peering, 23 active+undersized, 3 active+undersized+degraded, 50 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 3/84 objects degraded (3.571%); 1/84 objects misplaced (1.190%); 27 B/s, 8 objects/s recovering 2024-08-17T17:32:51.753 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:51.754 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-17T17:32:51.780 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:51.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:51.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:51.882 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:51.909 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:51.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:51.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:52.010 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:52.034 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:32:52.037 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:52.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:52.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:52.139 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:52.164 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:52.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:52.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:52.266 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:52.293 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:52.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:52.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:52.395 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:52.422 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:52.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:52.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:52.495 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:32:52.524 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:52.550 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:52.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:52.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:52.652 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:52.672 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-08-17T17:32:52.672 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:32:52.672 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-08-17T17:32:52.673 INFO:tasks.cephadm.osd.1:Restarting osd.1 (starting--it wasn't running)... 2024-08-17T17:32:52.673 DEBUG:teuthology.orchestra.run.smithi003:osd.1> sudo journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.1.service 2024-08-17T17:32:52.675 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl start ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.1 2024-08-17T17:32:52.679 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:52.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:52.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:52.781 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:52.805 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:52.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:52.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:52.907 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:52.917 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:32:52.930 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:52.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:52.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:53.023 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:52 smithi003 systemd[1]: Starting Ceph osd.1 for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:32:53.032 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:53.059 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:53.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:53.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:53.161 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:53.186 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:53.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:53.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:53.288 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:53.311 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:53.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:53.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:53.394 INFO:teuthology.orchestra.run.smithi003.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-08-17T17:32:53.414 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:53.436 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:53.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:53.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:53.538 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:53.565 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:53.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:53.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:53.666 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:53.693 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:53.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:53.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:53.702 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:53 smithi003 ceph-mon[103344]: pgmap v784: 89 pgs: 1 peering, 4 creating+peering, 7 active+clean+remapped, 1 remapped+peering, 23 active+undersized, 3 active+undersized+degraded, 50 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 3/84 objects degraded (3.571%); 1/84 objects misplaced (1.190%); 19 B/s, 5 objects/s recovering 2024-08-17T17:32:53.703 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:53 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2178550971' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:32:53.703 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:53 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:32:53.703 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:32:53.703 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:32:53.704 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:53 smithi003 ceph-mon[100615]: pgmap v784: 89 pgs: 1 peering, 4 creating+peering, 7 active+clean+remapped, 1 remapped+peering, 23 active+undersized, 3 active+undersized+degraded, 50 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 3/84 objects degraded (3.571%); 1/84 objects misplaced (1.190%); 19 B/s, 5 objects/s recovering 2024-08-17T17:32:53.704 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:53 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2178550971' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:32:53.704 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:53 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:32:53.704 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:32:53.704 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:32:53.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:53 smithi053 ceph-mon[89963]: pgmap v784: 89 pgs: 1 peering, 4 creating+peering, 7 active+clean+remapped, 1 remapped+peering, 23 active+undersized, 3 active+undersized+degraded, 50 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 3/84 objects degraded (3.571%); 1/84 objects misplaced (1.190%); 19 B/s, 5 objects/s recovering 2024-08-17T17:32:53.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:53 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2178550971' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:32:53.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:53 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:32:53.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:32:53.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:32:53.796 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:53.819 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:53.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:53.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:53.921 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:53.948 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:53.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:53.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:54.050 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:54.076 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:54.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:54.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:54.178 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:54.204 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:54.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:54.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:54.306 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:54.329 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:54.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:54.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:54.431 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:54.458 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:54.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:54.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:54.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:54.582 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:54.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:54.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:54.669 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:54 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T17:32:54.669 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:54 smithi003 ceph-mon[103344]: osdmap e265: 8 total, 7 up, 8 in 2024-08-17T17:32:54.670 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:54 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T17:32:54.670 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:54 smithi003 ceph-mon[100615]: osdmap e265: 8 total, 7 up, 8 in 2024-08-17T17:32:54.670 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:54 smithi003 podman[228788]: 2024-08-17 17:32:54.540510047 +0000 UTC m=+0.944494989 container create 28215ad53353f20aa32504c3b5b5f75d50ab5f2fc1a5f5fa725978d57517cd77 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=main-cd8bdbe, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-08-17T17:32:54.670 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:54 smithi003 podman[228788]: 2024-08-17 17:32:54.445943827 +0000 UTC m=+0.849928743 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:32:54.684 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:54.711 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:54.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:54.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:54.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:54 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T17:32:54.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:54 smithi053 ceph-mon[89963]: osdmap e265: 8 total, 7 up, 8 in 2024-08-17T17:32:54.813 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:54.840 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:54.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:54.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:54.942 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:54.968 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:54.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:54.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:55.070 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:55.094 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:55.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:55.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:55.196 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:55.218 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:55.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:55.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:55.320 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:55.346 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:55.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:55.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:55.448 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:55.471 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:55.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:55.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:55.573 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:55.595 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:55.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:55.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:55.652 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:55 smithi003 ceph-mon[103344]: pgmap v786: 89 pgs: 1 peering, 7 active+clean+remapped, 1 remapped+peering, 27 active+undersized, 3 active+undersized+degraded, 50 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 3/84 objects degraded (3.571%); 1/84 objects misplaced (1.190%); 3 B/s, 5 objects/s recovering 2024-08-17T17:32:55.700 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:55 smithi003 ceph-mon[103344]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:32:55.700 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:55 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 3/84 objects degraded (3.571%), 3 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-08-17T17:32:55.700 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:55 smithi003 ceph-mon[100615]: pgmap v786: 89 pgs: 1 peering, 7 active+clean+remapped, 1 remapped+peering, 27 active+undersized, 3 active+undersized+degraded, 50 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 3/84 objects degraded (3.571%); 1/84 objects misplaced (1.190%); 3 B/s, 5 objects/s recovering 2024-08-17T17:32:55.700 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:55 smithi003 ceph-mon[100615]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:32:55.700 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:55 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 3/84 objects degraded (3.571%), 3 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-08-17T17:32:55.700 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:55.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:55 smithi053 ceph-mon[89963]: pgmap v786: 89 pgs: 1 peering, 7 active+clean+remapped, 1 remapped+peering, 27 active+undersized, 3 active+undersized+degraded, 50 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 3/84 objects degraded (3.571%); 1/84 objects misplaced (1.190%); 3 B/s, 5 objects/s recovering 2024-08-17T17:32:55.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:55 smithi053 ceph-mon[89963]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:32:55.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:55 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 3/84 objects degraded (3.571%), 3 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-08-17T17:32:55.724 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:55.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:55.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:55.826 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:55.848 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:55.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:55.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:55.914 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:32:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:32:55] "GET /metrics HTTP/1.1" 200 36450 "" "Prometheus/2.51.0" 2024-08-17T17:32:55.950 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:55.973 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:55.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:55.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:56.075 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:56.102 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:56.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:56.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:56.204 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:56.209 DEBUG:teuthology.orchestra.run.smithi003:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-08-17T17:32:56.211 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-08-17T17:32:56.211 INFO:teuthology.orchestra.run:waiting for 3000 2024-08-17T17:32:56.231 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:56.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:56.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:56.333 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:56.360 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:56.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:56.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:56.361 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:hints = 1 2024-08-17T17:32:56.362 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-08-17T17:32:56.362 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object prefix: benchmark_data_smithi003_229103 2024-08-17T17:32:56.362 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:32:56.362 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 0 0 0 0 0 0 - 0 2024-08-17T17:32:56.461 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:56.479 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:56 smithi003 podman[228788]: 2024-08-17 17:32:56.204644242 +0000 UTC m=+2.608629143 container init 28215ad53353f20aa32504c3b5b5f75d50ab5f2fc1a5f5fa725978d57517cd77 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-cd8bdbe, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-08-17T17:32:56.480 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:56 smithi003 podman[228788]: 2024-08-17 17:32:56.209606902 +0000 UTC m=+2.613591803 container start 28215ad53353f20aa32504c3b5b5f75d50ab5f2fc1a5f5fa725978d57517cd77 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=main-cd8bdbe, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:32:56.480 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:56 smithi003 podman[228788]: 2024-08-17 17:32:56.329474561 +0000 UTC m=+2.733459468 container attach 28215ad53353f20aa32504c3b5b5f75d50ab5f2fc1a5f5fa725978d57517cd77 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate, RELEASE=main-cd8bdbe, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-08-17T17:32:56.480 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:56 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate[228909]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:32:56.480 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:56 smithi003 bash[228788]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:32:56.480 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:56 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate[228909]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:32:56.480 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:56 smithi003 bash[228788]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:32:56.480 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:56 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate[228909]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-17T17:32:56.480 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:56 smithi003 bash[228788]: --> Failed to activate via raw: activate() takes 1 positional argument but 5 were given 2024-08-17T17:32:56.480 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:56 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate[228909]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:32:56.480 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:56 smithi003 bash[228788]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:32:56.481 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:56 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate[228909]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:32:56.481 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:56 smithi003 bash[228788]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-17T17:32:56.500 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:56.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:56.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:56.602 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:56.637 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:56.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:56.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:56.742 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:56.774 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:56 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate[228909]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-08-17T17:32:56.774 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:56 smithi003 bash[228788]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-08-17T17:32:56.774 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:56 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate[228909]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2024-08-17T17:32:56.774 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:56 smithi003 bash[228788]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-1 --no-mon-config 2024-08-17T17:32:56.778 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:56.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:56.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:56.880 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:56.916 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:56.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:56.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:57.017 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:32:57.055 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:57.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:57.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:32:57.157 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:57.195 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:57.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:57.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:57.296 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:57.332 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:57.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:57.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:57.362 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 1 16 4409 4393 274.526 274.562 0.00692531 0.00362968 2024-08-17T17:32:57.434 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:57.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:57.473 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:57.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:57.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:57 smithi003 ceph-mon[103344]: pgmap v787: 89 pgs: 8 active+clean+remapped, 27 active+undersized, 3 active+undersized+degraded, 51 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 3/84 objects degraded (3.571%); 1/84 objects misplaced (1.190%); 2 B/s, 4 objects/s recovering 2024-08-17T17:32:57.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:57 smithi003 ceph-mon[100615]: pgmap v787: 89 pgs: 8 active+clean+remapped, 27 active+undersized, 3 active+undersized+degraded, 51 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 3/84 objects degraded (3.571%); 1/84 objects misplaced (1.190%); 2 B/s, 4 objects/s recovering 2024-08-17T17:32:57.574 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:32:57.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:57.615 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:57.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:32:57.716 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:57.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:57 smithi053 ceph-mon[89963]: pgmap v787: 89 pgs: 8 active+clean+remapped, 27 active+undersized, 3 active+undersized+degraded, 51 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 3/84 objects degraded (3.571%); 1/84 objects misplaced (1.190%); 2 B/s, 4 objects/s recovering 2024-08-17T17:32:57.752 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:57.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:57.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:57.825 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:57 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate[228909]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2024-08-17T17:32:57.825 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:57 smithi003 bash[228788]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-1/block 2024-08-17T17:32:57.825 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:57 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate[228909]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2024-08-17T17:32:57.825 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:57 smithi003 bash[228788]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-1/block 2024-08-17T17:32:57.825 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:57 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate[228909]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-08-17T17:32:57.825 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:57 smithi003 bash[228788]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-08-17T17:32:57.825 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:57 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate[228909]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-08-17T17:32:57.825 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:57 smithi003 bash[228788]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-08-17T17:32:57.825 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:57 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate[228909]: --> ceph-volume lvm activate successful for osd ID: 1 2024-08-17T17:32:57.826 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:57 smithi003 bash[228788]: --> ceph-volume lvm activate successful for osd ID: 1 2024-08-17T17:32:57.854 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:57.911 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:57.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:57.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:58.013 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:58.049 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:58.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:58.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:58.138 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:57 smithi003 podman[228788]: 2024-08-17 17:32:57.84374448 +0000 UTC m=+4.247729398 container died 28215ad53353f20aa32504c3b5b5f75d50ab5f2fc1a5f5fa725978d57517cd77 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-cd8bdbe, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=) 2024-08-17T17:32:58.151 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:58.187 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:58.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:58.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:58.289 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:58.326 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:58.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:58.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:58.362 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 2 16 8510 8494 265.409 256.312 0.00388779 0.00376007 2024-08-17T17:32:58.428 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:58.464 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:58.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:58.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:58.566 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:58.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:58.602 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:58.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:58.703 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:32:58.739 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:58.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:58.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:32:58.762 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:32:58 smithi003 podman[229308]: 2024-08-17 17:32:58.446884084 +0000 UTC m=+0.822178045 container remove 28215ad53353f20aa32504c3b5b5f75d50ab5f2fc1a5f5fa725978d57517cd77 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-activate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-cd8bdbe, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_BRANCH=HEAD) 2024-08-17T17:32:58.841 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:58.871 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:58.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:58.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:58.974 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:59.015 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:59.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:59.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:59.117 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:59.153 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:59.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:59.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:59.255 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:59.287 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:59.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:59.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:59.362 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 3 16 11978 11962 249.184 216.75 0.0014951 0.00394627 2024-08-17T17:32:59.389 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:32:59.413 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:59.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:59.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:32:59.517 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:59.554 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:59.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:59.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:32:59.656 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:32:59.691 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:59.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:59.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:32:59.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:59 smithi053 ceph-mon[89963]: pgmap v788: 89 pgs: 8 active+clean+remapped, 23 active+undersized, 7 active+undersized+degraded, 51 active+clean; 102 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 11 MiB/s wr, 179 op/s; 1565/4770 objects degraded (32.809%); 1/4770 objects misplaced (0.021%) 2024-08-17T17:32:59.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:32:59 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2910054194' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:32:59.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:59 smithi003 ceph-mon[103344]: pgmap v788: 89 pgs: 8 active+clean+remapped, 23 active+undersized, 7 active+undersized+degraded, 51 active+clean; 102 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 11 MiB/s wr, 179 op/s; 1565/4770 objects degraded (32.809%); 1/4770 objects misplaced (0.021%) 2024-08-17T17:32:59.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:32:59 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2910054194' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:32:59.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:59 smithi003 ceph-mon[100615]: pgmap v788: 89 pgs: 8 active+clean+remapped, 23 active+undersized, 7 active+undersized+degraded, 51 active+clean; 102 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 11 MiB/s wr, 179 op/s; 1565/4770 objects degraded (32.809%); 1/4770 objects misplaced (0.021%) 2024-08-17T17:32:59.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:32:59 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2910054194' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:32:59.794 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:32:59.829 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:32:59.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:59.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:32:59.931 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:32:59.969 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:32:59.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:32:59.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:00.071 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:00.074 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:33:00.075 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:33:00.075 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:33:00.075 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:33:00.075 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:33:00.075 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:33:00.075 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:33:00.075 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:33:00.075 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-17T17:33:00.075 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:33:00.075 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:33:00.075 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:33:00.076 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:33:00.076 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:33:00.076 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:33:00.108 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:00.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:00.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:00.210 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:00.246 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:00.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:00.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:00.348 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:00.362 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 4 16 15398 15382 240.321 213.75 0.00354283 0.00415348 2024-08-17T17:33:00.383 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:00.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:00.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:00.420 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-08-17T17:33:00.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:00 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 1565/4770 objects degraded (32.809%), 7 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-08-17T17:33:00.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:00 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/43897581' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:33:00.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:33:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:33:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:33:00.485 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:00.523 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:00.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:00.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:00.625 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:00.663 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:00.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:00.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:00.765 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:00.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:00 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 1565/4770 objects degraded (32.809%), 7 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-08-17T17:33:00.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:00 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/43897581' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:33:00.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:00 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 1565/4770 objects degraded (32.809%), 7 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-08-17T17:33:00.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:00 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/43897581' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:33:00.801 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:00.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:00.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:00.903 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:00.938 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:00.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:00.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:01.041 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:01.078 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:01.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:01.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:01.180 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:01.205 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:01.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:01.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:01.307 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:01.344 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:01.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:01.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:01.362 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 5 16 18345 18329 229.091 184.188 0.0102698 0.0043587 2024-08-17T17:33:01.446 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:01.484 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:01.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:01.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:01.586 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:01.622 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:01.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:01.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:01.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:01 smithi053 ceph-mon[89963]: pgmap v789: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 587 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 58 MiB/s wr, 931 op/s; 3456/28026 objects degraded (12.331%); 1/28026 objects misplaced (0.004%) 2024-08-17T17:33:01.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:01 smithi053 ceph-mon[89963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:33:01.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:01 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/43897581' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:33:01.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:01 smithi053 ceph-mon[89963]: osdmap e266: 8 total, 7 up, 8 in 2024-08-17T17:33:01.724 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:01.746 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:01.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:01.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:01.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:01 smithi003 ceph-mon[103344]: pgmap v789: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 587 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 58 MiB/s wr, 931 op/s; 3456/28026 objects degraded (12.331%); 1/28026 objects misplaced (0.004%) 2024-08-17T17:33:01.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:01 smithi003 ceph-mon[103344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:33:01.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:01 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/43897581' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:33:01.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:01 smithi003 ceph-mon[103344]: osdmap e266: 8 total, 7 up, 8 in 2024-08-17T17:33:01.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:01 smithi003 ceph-mon[100615]: pgmap v789: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 587 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 58 MiB/s wr, 931 op/s; 3456/28026 objects degraded (12.331%); 1/28026 objects misplaced (0.004%) 2024-08-17T17:33:01.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:01 smithi003 ceph-mon[100615]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:33:01.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:01 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/43897581' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:33:01.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:01 smithi003 ceph-mon[100615]: osdmap e266: 8 total, 7 up, 8 in 2024-08-17T17:33:01.848 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:01.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:01.886 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:01.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:01.987 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:02.024 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:02.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:02.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:02.126 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:02.163 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:02.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:02.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:02.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:02.299 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:02.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:02.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:02.362 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 6 16 20964 20948 218.188 163.688 0.00149005 0.00455067 2024-08-17T17:33:02.402 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:02.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:02.436 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:02.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:02.537 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:02.573 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:02.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:02.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:02.675 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:02.709 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:02.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:02.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:02.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:02.849 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:02.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:02.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:02.952 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:02.975 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:02.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:02.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:03.077 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:03.111 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:03.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:03.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:03.213 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:03.249 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:03.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:03.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:03.351 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:03.362 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 7 16 23678 23662 211.248 169.625 0.0105925 0.00472802 2024-08-17T17:33:03.386 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:03.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:03.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:03.488 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:03.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:03.526 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:03.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:03.628 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:03.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:03.669 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:03.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:03.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:03 smithi053 ceph-mon[89963]: pgmap v791: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 587 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 3456/28026 objects degraded (12.331%); 1/28026 objects misplaced (0.004%) 2024-08-17T17:33:03.770 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:03.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:03 smithi003 ceph-mon[103344]: pgmap v791: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 587 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 3456/28026 objects degraded (12.331%); 1/28026 objects misplaced (0.004%) 2024-08-17T17:33:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:03 smithi003 ceph-mon[100615]: pgmap v791: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 587 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s; 3456/28026 objects degraded (12.331%); 1/28026 objects misplaced (0.004%) 2024-08-17T17:33:03.792 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:03.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:03.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:03.894 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:03.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:03.929 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:03.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:04.030 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:04.063 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:04.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:04.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:04.165 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:04.208 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:04.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:04.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:04.310 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:04.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:04.350 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:04.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:04.363 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 8 16 25870 25854 201.965 137 0.00402391 0.00494627 2024-08-17T17:33:04.451 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:04.488 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:04.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:04.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:04.590 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:04.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:04.613 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:04.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:04.714 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:04.751 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:04.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:04.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:04.852 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:04.891 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:04.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:04.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:04.993 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:05.030 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:05.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:05.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:05.132 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:05.168 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:05.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:05.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:05.270 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:05.308 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:05.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:05.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:05.363 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 9 16 27854 27838 193.301 124 0.00933282 0.0051671 2024-08-17T17:33:05.410 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:05.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:05.450 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:05.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:05.551 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:05.588 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:05.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:05.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:05.652 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:05 smithi003 ceph-mon[103344]: pgmap v792: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 1.4 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 140 MiB/s wr, 2.24k op/s; 10302/67443 objects degraded (15.275%); 1/67443 objects misplaced (0.001%) 2024-08-17T17:33:05.652 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:05 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 10302/67443 objects degraded (15.275%), 10 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-08-17T17:33:05.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:05 smithi003 ceph-mon[100615]: pgmap v792: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 1.4 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 140 MiB/s wr, 2.24k op/s; 10302/67443 objects degraded (15.275%); 1/67443 objects misplaced (0.001%) 2024-08-17T17:33:05.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:05 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 10302/67443 objects degraded (15.275%), 10 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-08-17T17:33:05.690 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:05.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:05 smithi053 ceph-mon[89963]: pgmap v792: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 1.4 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 140 MiB/s wr, 2.24k op/s; 10302/67443 objects degraded (15.275%); 1/67443 objects misplaced (0.001%) 2024-08-17T17:33:05.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:05 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 10302/67443 objects degraded (15.275%), 10 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-08-17T17:33:05.731 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:05.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:05.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:05.833 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:05.871 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:05.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:05.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:05.973 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:05.987 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:33:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:33:05] "GET /metrics HTTP/1.1" 200 36450 "" "Prometheus/2.51.0" 2024-08-17T17:33:06.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:06.029 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:06.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:06.130 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:06.164 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:06.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:06.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:06.266 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:06.301 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:06.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:06.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:06.363 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 10 16 30547 30531 190.801 168.312 0.00506845 0.00523601 2024-08-17T17:33:06.403 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:06.440 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:06.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:06.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:06.542 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:06.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:06.580 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:06.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:06.630 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-17T17:33:06.630 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-17T17:33:06.682 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:06.715 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:06.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:06.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:06.817 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:06.852 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:06.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:06.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:06.954 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:06.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:06.988 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:06.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:07.089 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:07.124 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:07.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:07.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:07.227 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:07.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:07.268 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:07.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:07.273 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:33:07 smithi003 podman[230057]: 2024-08-17 17:33:06.958028953 +0000 UTC m=+1.663670285 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:33:07.363 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 11 16 33169 33153 188.352 163.875 0.00622203 0.00530412 2024-08-17T17:33:07.369 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:07.405 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:07.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:07.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:07.509 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:07.545 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:07.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:07.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:07.647 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:07.685 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:07.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:07.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:07.715 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:07 smithi003 ceph-mon[103344]: pgmap v793: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 1.5 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 151 MiB/s wr, 2.42k op/s; 10302/72654 objects degraded (14.180%); 1/72654 objects misplaced (0.001%) 2024-08-17T17:33:07.715 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:07 smithi003 ceph-mon[100615]: pgmap v793: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 1.5 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 151 MiB/s wr, 2.42k op/s; 10302/72654 objects degraded (14.180%); 1/72654 objects misplaced (0.001%) 2024-08-17T17:33:07.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:07 smithi053 ceph-mon[89963]: pgmap v793: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 1.5 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 151 MiB/s wr, 2.42k op/s; 10302/72654 objects degraded (14.180%); 1/72654 objects misplaced (0.001%) 2024-08-17T17:33:07.751 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:33:07.788 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:07.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:07.836 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:07.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:07.936 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:08.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:08.006 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:08.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:08.019 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:33:07 smithi003 podman[230057]: 2024-08-17 17:33:07.714719006 +0000 UTC m=+2.420360304 container create 967184dd1121826dd60baf56e6eb0428f31dfe51713cd0fe44b8e9087a768903 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, RELEASE=main-cd8bdbe, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-08-17T17:33:08.107 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:08.141 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:08.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:08.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:08.243 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:08.269 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:33:08 smithi003 podman[230057]: 2024-08-17 17:33:08.136120517 +0000 UTC m=+2.841761828 container init 967184dd1121826dd60baf56e6eb0428f31dfe51713cd0fe44b8e9087a768903 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-cd8bdbe, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-08-17T17:33:08.269 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:33:08 smithi003 podman[230057]: 2024-08-17 17:33:08.142647162 +0000 UTC m=+2.848288462 container start 967184dd1121826dd60baf56e6eb0428f31dfe51713cd0fe44b8e9087a768903 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-cd8bdbe, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-08-17T17:33:08.277 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:08.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:08.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:08.307 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:33:08.307 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-08-17T17:33:08.307 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-08-17T17:33:08.307 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:33:08.308 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.308 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45491.0:51014 6.c 6:3b1e47a1:::benchmark_data_smithi003_175043_object51013:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-08-17T17:33:08.308 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:25:03.970975+0000", 2024-08-17T17:33:08.308 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 484.22697090899999, 2024-08-17T17:33:08.308 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.707060298, 2024-08-17T17:33:08.308 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:33:08.308 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:33:08.308 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:33:08.308 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:33:08.308 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45491", 2024-08-17T17:33:08.308 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:08.308 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 51014 2024-08-17T17:33:08.308 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.308 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:33:08.309 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.309 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:33:08.309 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.970975+0000", 2024-08-17T17:33:08.309 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.309 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.309 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.309 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:33:08.309 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.970975+0000", 2024-08-17T17:33:08.309 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.309 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.309 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.309 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:33:08.309 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.970976+0000", 2024-08-17T17:33:08.310 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.443e-06 2024-08-17T17:33:08.310 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.310 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.310 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:33:08.310 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.971002+0000", 2024-08-17T17:33:08.310 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5477999999999999e-05 2024-08-17T17:33:08.310 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.310 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.310 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:33:08.310 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.971015+0000", 2024-08-17T17:33:08.310 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3555e-05 2024-08-17T17:33:08.310 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.311 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.311 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:33:08.311 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.971023+0000", 2024-08-17T17:33:08.311 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3900000000000004e-06 2024-08-17T17:33:08.311 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.311 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.311 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:33:08.311 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.975076+0000", 2024-08-17T17:33:08.311 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040535240000000002 2024-08-17T17:33:08.311 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.311 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.311 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:33:08.311 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.975207+0000", 2024-08-17T17:33:08.312 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000130992 2024-08-17T17:33:08.312 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.312 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.312 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-08-17T17:33:08.312 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.975338+0000", 2024-08-17T17:33:08.312 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000130987 2024-08-17T17:33:08.312 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.312 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.312 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.312 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.990507+0000", 2024-08-17T17:33:08.312 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.015168921 2024-08-17T17:33:08.312 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.312 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.313 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:33:08.313 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.990522+0000", 2024-08-17T17:33:08.313 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5148e-05 2024-08-17T17:33:08.313 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.313 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.313 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:33:08.313 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.995576+0000", 2024-08-17T17:33:08.313 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0050539399999999998 2024-08-17T17:33:08.313 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.313 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.313 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:33:08.313 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.995590+0000", 2024-08-17T17:33:08.313 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3723000000000001e-05 2024-08-17T17:33:08.314 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.314 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.314 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:33:08.314 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:04.505851+0000", 2024-08-17T17:33:08.314 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.51026153500000004 2024-08-17T17:33:08.314 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.314 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.314 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:33:08.314 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:04.505866+0000", 2024-08-17T17:33:08.314 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4335000000000001e-05 2024-08-17T17:33:08.314 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.314 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.315 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:33:08.315 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:05.677960+0000", 2024-08-17T17:33:08.315 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.172094752 2024-08-17T17:33:08.315 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.315 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.315 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:33:08.315 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:05.678035+0000", 2024-08-17T17:33:08.315 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4574999999999997e-05 2024-08-17T17:33:08.315 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.315 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:33:08.315 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.315 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.315 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.315 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45491.0:51030 6.c 6:39c4a7e1:::benchmark_data_smithi003_175043_object51029:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-08-17T17:33:08.316 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:25:03.982807+0000", 2024-08-17T17:33:08.316 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 484.21513895700002, 2024-08-17T17:33:08.316 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7154847520000001, 2024-08-17T17:33:08.316 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:33:08.316 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:33:08.316 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:08.316 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:33:08.316 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45491", 2024-08-17T17:33:08.316 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:08.316 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 51030 2024-08-17T17:33:08.316 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.316 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:33:08.317 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.317 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:33:08.317 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.982807+0000", 2024-08-17T17:33:08.317 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.317 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.317 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.317 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:33:08.317 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.982807+0000", 2024-08-17T17:33:08.317 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.317 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.317 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.317 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:33:08.317 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.982807+0000", 2024-08-17T17:33:08.317 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3500000000000002e-07 2024-08-17T17:33:08.318 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.318 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.318 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:33:08.318 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.982837+0000", 2024-08-17T17:33:08.318 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9655999999999999e-05 2024-08-17T17:33:08.318 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.318 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.318 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:33:08.318 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.982844+0000", 2024-08-17T17:33:08.318 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1029999999999996e-06 2024-08-17T17:33:08.318 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.318 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.318 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:33:08.319 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.982849+0000", 2024-08-17T17:33:08.319 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8670000000000002e-06 2024-08-17T17:33:08.319 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.319 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.319 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:33:08.319 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.996164+0000", 2024-08-17T17:33:08.319 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013315268 2024-08-17T17:33:08.319 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.319 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.319 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:33:08.319 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:03.996178+0000", 2024-08-17T17:33:08.319 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3876999999999999e-05 2024-08-17T17:33:08.319 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.319 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.320 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:33:08.320 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:04.506236+0000", 2024-08-17T17:33:08.320 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.51005823800000005 2024-08-17T17:33:08.320 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.320 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.320 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:33:08.320 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:04.506250+0000", 2024-08-17T17:33:08.320 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3875e-05 2024-08-17T17:33:08.320 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.320 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.320 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:33:08.320 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:05.680158+0000", 2024-08-17T17:33:08.320 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1739081920000001 2024-08-17T17:33:08.321 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.321 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.321 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:33:08.321 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:05.680328+0000", 2024-08-17T17:33:08.321 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016995000000000001 2024-08-17T17:33:08.321 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.321 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.321 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,6", 2024-08-17T17:33:08.321 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:05.680581+0000", 2024-08-17T17:33:08.321 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00025260199999999999 2024-08-17T17:33:08.321 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.321 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.322 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:33:08.322 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:05.686170+0000", 2024-08-17T17:33:08.322 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0055897220000000001 2024-08-17T17:33:08.322 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.322 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.322 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.322 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:05.695445+0000", 2024-08-17T17:33:08.325 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0092747639999999996 2024-08-17T17:33:08.325 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.325 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.325 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.325 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:05.695511+0000", 2024-08-17T17:33:08.325 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6025000000000006e-05 2024-08-17T17:33:08.325 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.325 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.326 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:33:08.326 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:05.695579+0000", 2024-08-17T17:33:08.326 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7936999999999997e-05 2024-08-17T17:33:08.326 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.326 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.326 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:33:08.326 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:25:05.698291+0000", 2024-08-17T17:33:08.326 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027122409999999998 2024-08-17T17:33:08.326 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.326 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:33:08.326 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.326 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.327 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.327 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:80092 7.a 7:5f458f91:::benchmark_data_smithi003_201798_object80091:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-17T17:33:08.327 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:29:33.380788+0000", 2024-08-17T17:33:08.327 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 214.81715786800001, 2024-08-17T17:33:08.327 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.19190290800000001, 2024-08-17T17:33:08.327 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:33:08.327 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:33:08.327 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:08.327 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:33:08.327 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:33:08.327 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:33:08.327 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 80092 2024-08-17T17:33:08.327 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.328 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:33:08.328 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.328 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:33:08.328 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.380788+0000", 2024-08-17T17:33:08.328 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.328 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.328 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.328 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:33:08.328 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.380788+0000", 2024-08-17T17:33:08.328 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.328 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.328 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.328 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:33:08.328 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.380789+0000", 2024-08-17T17:33:08.329 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.376e-06 2024-08-17T17:33:08.329 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.329 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.329 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:33:08.329 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.380820+0000", 2024-08-17T17:33:08.329 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0960000000000002e-05 2024-08-17T17:33:08.329 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.329 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.329 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:33:08.329 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.380832+0000", 2024-08-17T17:33:08.329 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2342e-05 2024-08-17T17:33:08.329 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.330 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.330 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:33:08.330 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.380839+0000", 2024-08-17T17:33:08.330 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9460000000000001e-06 2024-08-17T17:33:08.330 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.330 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.330 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:33:08.330 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.385442+0000", 2024-08-17T17:33:08.331 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004602826 2024-08-17T17:33:08.331 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.331 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.331 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:33:08.331 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.385923+0000", 2024-08-17T17:33:08.331 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00048045500000000002 2024-08-17T17:33:08.331 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.331 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.331 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:33:08.331 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.386043+0000", 2024-08-17T17:33:08.331 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012019999999999999 2024-08-17T17:33:08.331 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.331 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.332 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.332 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.392281+0000", 2024-08-17T17:33:08.332 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006238548 2024-08-17T17:33:08.332 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.332 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.332 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:33:08.332 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.392305+0000", 2024-08-17T17:33:08.332 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3204e-05 2024-08-17T17:33:08.332 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.332 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.332 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.332 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.572586+0000", 2024-08-17T17:33:08.332 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18028166300000001 2024-08-17T17:33:08.332 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.333 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.333 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:33:08.333 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.572644+0000", 2024-08-17T17:33:08.333 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7756000000000001e-05 2024-08-17T17:33:08.333 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.333 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.333 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:33:08.333 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.572691+0000", 2024-08-17T17:33:08.333 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6631999999999997e-05 2024-08-17T17:33:08.333 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.333 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:33:08.333 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.333 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.333 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.334 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:80095 7.3 7:c369a586:::benchmark_data_smithi003_201798_object80094:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-17T17:33:08.334 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:29:33.381510+0000", 2024-08-17T17:33:08.334 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 214.816435506, 2024-08-17T17:33:08.334 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18715829, 2024-08-17T17:33:08.334 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:33:08.334 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:33:08.334 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:08.334 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:33:08.334 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:33:08.334 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:33:08.334 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 80095 2024-08-17T17:33:08.334 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.335 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:33:08.335 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.335 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:33:08.335 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.381510+0000", 2024-08-17T17:33:08.335 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.335 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.335 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.335 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:33:08.335 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.381510+0000", 2024-08-17T17:33:08.335 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.335 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.335 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.335 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:33:08.335 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.381511+0000", 2024-08-17T17:33:08.336 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9400000000000004e-07 2024-08-17T17:33:08.336 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.336 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.336 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:33:08.336 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.381542+0000", 2024-08-17T17:33:08.336 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0762000000000001e-05 2024-08-17T17:33:08.336 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.336 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.336 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:33:08.337 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.381553+0000", 2024-08-17T17:33:08.337 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0811e-05 2024-08-17T17:33:08.337 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.337 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.337 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:33:08.337 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.381559+0000", 2024-08-17T17:33:08.337 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3980000000000004e-06 2024-08-17T17:33:08.337 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.337 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.337 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:33:08.337 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.381659+0000", 2024-08-17T17:33:08.337 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9894000000000003e-05 2024-08-17T17:33:08.337 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.337 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.337 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:33:08.338 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.381804+0000", 2024-08-17T17:33:08.338 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014564800000000001 2024-08-17T17:33:08.338 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.338 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.338 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,6", 2024-08-17T17:33:08.338 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.381953+0000", 2024-08-17T17:33:08.338 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000148305 2024-08-17T17:33:08.338 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.338 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.338 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.338 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389248+0000", 2024-08-17T17:33:08.339 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072953180000000003 2024-08-17T17:33:08.339 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.339 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.339 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:33:08.339 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389260+0000", 2024-08-17T17:33:08.339 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1849e-05 2024-08-17T17:33:08.339 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.339 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.339 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.339 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.568485+0000", 2024-08-17T17:33:08.339 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17922486300000001 2024-08-17T17:33:08.339 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.339 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.340 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:33:08.340 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.568619+0000", 2024-08-17T17:33:08.340 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000133855 2024-08-17T17:33:08.340 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.340 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.340 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:33:08.340 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.568668+0000", 2024-08-17T17:33:08.340 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9793000000000003e-05 2024-08-17T17:33:08.340 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.340 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:33:08.340 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.340 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.340 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.341 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:80096 7.a 7:54a4ec2f:::benchmark_data_smithi003_201798_object80095:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-17T17:33:08.341 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:29:33.381575+0000", 2024-08-17T17:33:08.341 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 214.81637094300001, 2024-08-17T17:33:08.341 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.19489894499999999, 2024-08-17T17:33:08.341 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:33:08.341 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:33:08.341 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:08.341 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:33:08.341 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:33:08.341 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:33:08.341 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 80096 2024-08-17T17:33:08.341 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.341 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:33:08.341 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.342 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:33:08.342 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.381575+0000", 2024-08-17T17:33:08.342 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.342 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.342 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.342 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:33:08.342 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.381575+0000", 2024-08-17T17:33:08.342 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.342 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.342 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.342 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:33:08.342 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.381575+0000", 2024-08-17T17:33:08.343 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0100000000000005e-07 2024-08-17T17:33:08.343 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.343 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.343 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:33:08.343 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.381594+0000", 2024-08-17T17:33:08.343 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8445e-05 2024-08-17T17:33:08.343 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.343 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.343 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:33:08.343 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.381600+0000", 2024-08-17T17:33:08.343 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7349999999999999e-06 2024-08-17T17:33:08.344 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.344 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.344 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:33:08.344 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.381607+0000", 2024-08-17T17:33:08.344 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6429999999999998e-06 2024-08-17T17:33:08.344 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.344 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.344 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:33:08.344 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389348+0000", 2024-08-17T17:33:08.344 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0077407489999999999 2024-08-17T17:33:08.344 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.344 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.344 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:33:08.344 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389470+0000", 2024-08-17T17:33:08.345 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012184900000000001 2024-08-17T17:33:08.345 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.345 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.345 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:33:08.345 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389642+0000", 2024-08-17T17:33:08.345 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017195000000000001 2024-08-17T17:33:08.345 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.345 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.345 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.345 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.392624+0000", 2024-08-17T17:33:08.345 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029826520000000001 2024-08-17T17:33:08.345 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.345 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.345 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:33:08.346 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.395467+0000", 2024-08-17T17:33:08.346 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002842883 2024-08-17T17:33:08.346 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.346 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.346 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.346 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.576369+0000", 2024-08-17T17:33:08.346 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18090183200000001 2024-08-17T17:33:08.346 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.347 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.347 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:33:08.347 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.576427+0000", 2024-08-17T17:33:08.347 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8245999999999998e-05 2024-08-17T17:33:08.347 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.347 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.347 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:33:08.347 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.576474+0000", 2024-08-17T17:33:08.347 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6459999999999999e-05 2024-08-17T17:33:08.347 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.348 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:33:08.348 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.348 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.348 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.348 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:80097 7.3 7:cd0150f4:::benchmark_data_smithi003_201798_object80096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-17T17:33:08.348 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:29:33.382635+0000", 2024-08-17T17:33:08.348 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 214.815310834, 2024-08-17T17:33:08.348 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18676747299999999, 2024-08-17T17:33:08.349 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:33:08.349 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:33:08.349 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:08.349 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:33:08.349 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:33:08.349 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:33:08.349 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 80097 2024-08-17T17:33:08.349 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.350 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:33:08.350 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.350 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:33:08.350 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.382635+0000", 2024-08-17T17:33:08.350 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.350 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.350 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.350 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:33:08.351 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.382635+0000", 2024-08-17T17:33:08.351 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.351 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.351 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.351 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:33:08.351 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.382636+0000", 2024-08-17T17:33:08.351 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3400000000000001e-06 2024-08-17T17:33:08.351 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.352 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.352 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:33:08.352 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.382657+0000", 2024-08-17T17:33:08.352 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0624000000000001e-05 2024-08-17T17:33:08.352 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.352 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.352 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:33:08.352 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.382668+0000", 2024-08-17T17:33:08.352 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1354e-05 2024-08-17T17:33:08.353 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.353 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.353 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:33:08.353 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.382675+0000", 2024-08-17T17:33:08.353 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1559999999999998e-06 2024-08-17T17:33:08.353 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.353 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.353 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:33:08.353 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.385528+0000", 2024-08-17T17:33:08.354 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028528249999999998 2024-08-17T17:33:08.354 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.354 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.354 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:33:08.354 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.385862+0000", 2024-08-17T17:33:08.354 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00033433600000000002 2024-08-17T17:33:08.354 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.354 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.355 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,6", 2024-08-17T17:33:08.355 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.386007+0000", 2024-08-17T17:33:08.355 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014439800000000001 2024-08-17T17:33:08.355 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.355 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.355 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.355 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.392256+0000", 2024-08-17T17:33:08.355 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0062494899999999999 2024-08-17T17:33:08.356 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.356 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.356 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:33:08.356 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.392290+0000", 2024-08-17T17:33:08.356 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3670999999999997e-05 2024-08-17T17:33:08.356 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.356 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.356 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.356 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.569320+0000", 2024-08-17T17:33:08.357 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.177029614 2024-08-17T17:33:08.357 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.357 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.357 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:33:08.357 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.569367+0000", 2024-08-17T17:33:08.357 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7874999999999998e-05 2024-08-17T17:33:08.357 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.357 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.358 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:33:08.358 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.569402+0000", 2024-08-17T17:33:08.358 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4789999999999997e-05 2024-08-17T17:33:08.358 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.358 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:33:08.358 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.358 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.358 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.359 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:80100 7.a 7:50c869f6:::benchmark_data_smithi003_201798_object80099:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-17T17:33:08.359 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:29:33.382998+0000", 2024-08-17T17:33:08.359 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 214.81494744599999, 2024-08-17T17:33:08.359 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.193590702, 2024-08-17T17:33:08.359 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:33:08.359 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:33:08.359 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:08.359 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:33:08.359 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:33:08.360 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:33:08.360 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 80100 2024-08-17T17:33:08.360 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.360 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:33:08.360 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.360 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:33:08.361 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.382998+0000", 2024-08-17T17:33:08.361 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.361 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.361 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.361 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:33:08.361 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.382998+0000", 2024-08-17T17:33:08.361 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.362 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.362 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.362 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:33:08.362 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.382999+0000", 2024-08-17T17:33:08.362 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4900000000000005e-07 2024-08-17T17:33:08.362 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.362 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.362 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:33:08.363 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.383020+0000", 2024-08-17T17:33:08.363 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0601000000000001e-05 2024-08-17T17:33:08.363 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.363 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.363 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:33:08.363 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.383029+0000", 2024-08-17T17:33:08.363 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4919999999999999e-06 2024-08-17T17:33:08.363 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.363 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.363 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:33:08.363 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.383035+0000", 2024-08-17T17:33:08.363 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1469999999999998e-06 2024-08-17T17:33:08.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.364 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.364 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:33:08.364 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.392708+0000", 2024-08-17T17:33:08.364 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0096729090000000004 2024-08-17T17:33:08.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.364 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.364 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:33:08.364 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.392839+0000", 2024-08-17T17:33:08.364 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013080899999999999 2024-08-17T17:33:08.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.365 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:33:08.365 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.392938+0000", 2024-08-17T17:33:08.365 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.912e-05 2024-08-17T17:33:08.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.365 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.365 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.395527+0000", 2024-08-17T17:33:08.365 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00258859 2024-08-17T17:33:08.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.365 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:33:08.365 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.398822+0000", 2024-08-17T17:33:08.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032956299999999999 2024-08-17T17:33:08.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.366 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.576509+0000", 2024-08-17T17:33:08.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17768704900000001 2024-08-17T17:33:08.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:33:08.366 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.576557+0000", 2024-08-17T17:33:08.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7812000000000002e-05 2024-08-17T17:33:08.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.367 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:33:08.367 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.576589+0000", 2024-08-17T17:33:08.367 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1794000000000001e-05 2024-08-17T17:33:08.367 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.367 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:33:08.367 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.367 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.367 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:80101 7.2 7:42bf2a73:::benchmark_data_smithi003_201798_object80100:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-17T17:33:08.367 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:29:33.386150+0000", 2024-08-17T17:33:08.367 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 214.81179521300001, 2024-08-17T17:33:08.367 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.19212204899999999, 2024-08-17T17:33:08.367 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:33:08.367 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:33:08.368 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:08.368 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:33:08.368 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:33:08.368 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:33:08.368 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 80101 2024-08-17T17:33:08.368 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.368 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:33:08.368 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.368 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:33:08.368 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.386150+0000", 2024-08-17T17:33:08.368 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.368 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.368 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.369 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:33:08.369 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.386150+0000", 2024-08-17T17:33:08.369 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.369 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.369 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.369 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:33:08.369 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.386152+0000", 2024-08-17T17:33:08.369 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3200000000000001e-06 2024-08-17T17:33:08.369 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.369 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.369 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:33:08.369 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.386175+0000", 2024-08-17T17:33:08.369 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2830999999999999e-05 2024-08-17T17:33:08.369 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.369 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.370 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:33:08.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.386185+0000", 2024-08-17T17:33:08.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0236e-05 2024-08-17T17:33:08.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.370 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.370 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:33:08.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.386192+0000", 2024-08-17T17:33:08.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8809999999999998e-06 2024-08-17T17:33:08.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.370 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.370 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:33:08.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.392702+0000", 2024-08-17T17:33:08.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006510058 2024-08-17T17:33:08.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:33:08.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.392903+0000", 2024-08-17T17:33:08.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000201709 2024-08-17T17:33:08.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:33:08.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.393027+0000", 2024-08-17T17:33:08.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012381 2024-08-17T17:33:08.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.372 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.398525+0000", 2024-08-17T17:33:08.372 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0054977450000000001 2024-08-17T17:33:08.372 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.372 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:33:08.372 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.398808+0000", 2024-08-17T17:33:08.372 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00028265400000000002 2024-08-17T17:33:08.372 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.372 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.372 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.578131+0000", 2024-08-17T17:33:08.372 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.179323492 2024-08-17T17:33:08.372 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.373 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.373 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:33:08.373 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.578229+0000", 2024-08-17T17:33:08.373 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7421000000000001e-05 2024-08-17T17:33:08.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.373 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.373 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:33:08.373 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.578272+0000", 2024-08-17T17:33:08.373 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3891999999999999e-05 2024-08-17T17:33:08.373 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.373 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:33:08.373 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.373 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.374 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:80102 7.3 7:c2ad9bad:::benchmark_data_smithi003_201798_object80101:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-17T17:33:08.374 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:29:33.386331+0000", 2024-08-17T17:33:08.374 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 214.81161419099999, 2024-08-17T17:33:08.374 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18324026700000001, 2024-08-17T17:33:08.374 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:33:08.374 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:33:08.374 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:08.374 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:33:08.375 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:33:08.375 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:33:08.375 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 80102 2024-08-17T17:33:08.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.375 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:33:08.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.375 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:33:08.375 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.386331+0000", 2024-08-17T17:33:08.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.375 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:33:08.375 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.386331+0000", 2024-08-17T17:33:08.376 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:33:08.376 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.386332+0000", 2024-08-17T17:33:08.376 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4799999999999998e-07 2024-08-17T17:33:08.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:33:08.376 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.386356+0000", 2024-08-17T17:33:08.376 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4087000000000002e-05 2024-08-17T17:33:08.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:33:08.377 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.386371+0000", 2024-08-17T17:33:08.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4348999999999999e-05 2024-08-17T17:33:08.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:33:08.377 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.386377+0000", 2024-08-17T17:33:08.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6590000000000001e-06 2024-08-17T17:33:08.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:33:08.377 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389327+0000", 2024-08-17T17:33:08.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029500289999999998 2024-08-17T17:33:08.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:33:08.378 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389469+0000", 2024-08-17T17:33:08.378 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000141378 2024-08-17T17:33:08.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,6", 2024-08-17T17:33:08.378 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389621+0000", 2024-08-17T17:33:08.378 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015251500000000001 2024-08-17T17:33:08.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:33:08.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.392436+0000", 2024-08-17T17:33:08.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028147950000000001 2024-08-17T17:33:08.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.392864+0000", 2024-08-17T17:33:08.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00042817499999999999 2024-08-17T17:33:08.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.569490+0000", 2024-08-17T17:33:08.380 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17662562800000001 2024-08-17T17:33:08.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.380 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.380 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:33:08.380 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.569536+0000", 2024-08-17T17:33:08.380 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6548000000000002e-05 2024-08-17T17:33:08.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.380 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.380 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:33:08.380 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.569572+0000", 2024-08-17T17:33:08.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5456000000000001e-05 2024-08-17T17:33:08.381 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.381 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:33:08.381 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.381 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:80107 7.a 7:51d4ace1:::benchmark_data_smithi003_201798_object80106:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-17T17:33:08.381 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:29:33.389761+0000", 2024-08-17T17:33:08.381 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 214.80818504600001, 2024-08-17T17:33:08.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.19082833599999999, 2024-08-17T17:33:08.381 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:33:08.381 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:33:08.381 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:08.382 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:33:08.382 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:33:08.382 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:33:08.382 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 80107 2024-08-17T17:33:08.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.382 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:33:08.382 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.382 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:33:08.382 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389761+0000", 2024-08-17T17:33:08.382 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.382 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.382 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:33:08.383 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389761+0000", 2024-08-17T17:33:08.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:33:08.383 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389762+0000", 2024-08-17T17:33:08.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.091e-06 2024-08-17T17:33:08.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:33:08.383 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389797+0000", 2024-08-17T17:33:08.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5756999999999997e-05 2024-08-17T17:33:08.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:33:08.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389806+0000", 2024-08-17T17:33:08.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8519999999999993e-06 2024-08-17T17:33:08.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:33:08.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389812+0000", 2024-08-17T17:33:08.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4219999999999999e-06 2024-08-17T17:33:08.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:33:08.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.395577+0000", 2024-08-17T17:33:08.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0057653940000000001 2024-08-17T17:33:08.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.385 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.385 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:33:08.385 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.395713+0000", 2024-08-17T17:33:08.385 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000135766 2024-08-17T17:33:08.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.385 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.385 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:33:08.385 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.395864+0000", 2024-08-17T17:33:08.385 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015110699999999999 2024-08-17T17:33:08.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.385 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.385 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:33:08.385 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.399024+0000", 2024-08-17T17:33:08.385 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031597499999999998 2024-08-17T17:33:08.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.386 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.386 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.399362+0000", 2024-08-17T17:33:08.386 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000338687 2024-08-17T17:33:08.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.387 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.387 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.580492+0000", 2024-08-17T17:33:08.387 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18112983099999999 2024-08-17T17:33:08.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.387 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:33:08.388 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.580540+0000", 2024-08-17T17:33:08.388 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7556999999999999e-05 2024-08-17T17:33:08.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.388 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:33:08.388 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.580589+0000", 2024-08-17T17:33:08.388 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9122000000000002e-05 2024-08-17T17:33:08.388 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.388 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:33:08.388 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.388 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:80108 7.2 7:4c7c8085:::benchmark_data_smithi003_201798_object80107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-17T17:33:08.388 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:29:33.389832+0000", 2024-08-17T17:33:08.388 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 214.808113523, 2024-08-17T17:33:08.389 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.19049521, 2024-08-17T17:33:08.389 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:33:08.389 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:33:08.389 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:08.389 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:33:08.389 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:33:08.389 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:33:08.389 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 80108 2024-08-17T17:33:08.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.389 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:33:08.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.389 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:33:08.389 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389832+0000", 2024-08-17T17:33:08.389 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.390 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:33:08.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389832+0000", 2024-08-17T17:33:08.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.390 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:33:08.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389833+0000", 2024-08-17T17:33:08.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9699999999999996e-07 2024-08-17T17:33:08.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.391 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:33:08.391 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389856+0000", 2024-08-17T17:33:08.391 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3558999999999999e-05 2024-08-17T17:33:08.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.391 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:33:08.391 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389864+0000", 2024-08-17T17:33:08.391 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.542e-06 2024-08-17T17:33:08.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.391 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:33:08.391 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.389870+0000", 2024-08-17T17:33:08.392 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9599999999999997e-06 2024-08-17T17:33:08.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.392 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:33:08.392 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.395563+0000", 2024-08-17T17:33:08.392 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0056929060000000002 2024-08-17T17:33:08.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.392 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:33:08.392 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.395710+0000", 2024-08-17T17:33:08.392 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014744700000000001 2024-08-17T17:33:08.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:33:08.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.395862+0000", 2024-08-17T17:33:08.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000152001 2024-08-17T17:33:08.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:33:08.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.398827+0000", 2024-08-17T17:33:08.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029646820000000002 2024-08-17T17:33:08.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.399994+0000", 2024-08-17T17:33:08.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00116719 2024-08-17T17:33:08.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.580221+0000", 2024-08-17T17:33:08.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18022680699999999 2024-08-17T17:33:08.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:33:08.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.580272+0000", 2024-08-17T17:33:08.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0918999999999998e-05 2024-08-17T17:33:08.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:33:08.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.580327+0000", 2024-08-17T17:33:08.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5699999999999999e-05 2024-08-17T17:33:08.395 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.395 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:33:08.395 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.395 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:80117 7.2 7:412925cb:::benchmark_data_smithi003_201798_object80116:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-08-17T17:33:08.395 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:29:33.402561+0000", 2024-08-17T17:33:08.395 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 214.79538463, 2024-08-17T17:33:08.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.177879548, 2024-08-17T17:33:08.395 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:33:08.395 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:33:08.395 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:08.395 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:33:08.396 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:33:08.396 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:33:08.396 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 80117 2024-08-17T17:33:08.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.396 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:33:08.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.396 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:33:08.396 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.402561+0000", 2024-08-17T17:33:08.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.396 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:33:08.396 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.402561+0000", 2024-08-17T17:33:08.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:33:08.397 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.402562+0000", 2024-08-17T17:33:08.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.026e-06 2024-08-17T17:33:08.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:33:08.397 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.402581+0000", 2024-08-17T17:33:08.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9431000000000001e-05 2024-08-17T17:33:08.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:33:08.397 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.402592+0000", 2024-08-17T17:33:08.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0492000000000001e-05 2024-08-17T17:33:08.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.398 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:33:08.398 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.402597+0000", 2024-08-17T17:33:08.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.293e-06 2024-08-17T17:33:08.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.398 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:33:08.398 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.402641+0000", 2024-08-17T17:33:08.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3943000000000003e-05 2024-08-17T17:33:08.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.398 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:33:08.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.402740+0000", 2024-08-17T17:33:08.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8305000000000002e-05 2024-08-17T17:33:08.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:33:08.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.402846+0000", 2024-08-17T17:33:08.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010678799999999999 2024-08-17T17:33:08.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:33:08.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.405210+0000", 2024-08-17T17:33:08.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0023640739999999999 2024-08-17T17:33:08.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.405832+0000", 2024-08-17T17:33:08.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00062142600000000001 2024-08-17T17:33:08.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.580363+0000", 2024-08-17T17:33:08.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17453089499999999 2024-08-17T17:33:08.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:33:08.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.580409+0000", 2024-08-17T17:33:08.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6294000000000001e-05 2024-08-17T17:33:08.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:33:08.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:33.580441+0000", 2024-08-17T17:33:08.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1581000000000001e-05 2024-08-17T17:33:08.401 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.401 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:33:08.401 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.401 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:163936 7.2 7:4b3b72a7:::benchmark_data_smithi003_201798_object50714:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-08-17T17:33:08.401 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:31:16.126841+0000", 2024-08-17T17:33:08.401 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 112.071104402, 2024-08-17T17:33:08.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.21988460500000001, 2024-08-17T17:33:08.402 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:33:08.402 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:33:08.402 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:08.402 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:33:08.402 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:33:08.402 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:33:08.402 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 163936 2024-08-17T17:33:08.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.402 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:33:08.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:33:08.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.126841+0000", 2024-08-17T17:33:08.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:33:08.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.126841+0000", 2024-08-17T17:33:08.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:33:08.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.126842+0000", 2024-08-17T17:33:08.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6400000000000001e-07 2024-08-17T17:33:08.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:33:08.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.126846+0000", 2024-08-17T17:33:08.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1019999999999999e-06 2024-08-17T17:33:08.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:33:08.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.126857+0000", 2024-08-17T17:33:08.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0611000000000001e-05 2024-08-17T17:33:08.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:33:08.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.126864+0000", 2024-08-17T17:33:08.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8800000000000002e-06 2024-08-17T17:33:08.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:33:08.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.140003+0000", 2024-08-17T17:33:08.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013139486000000001 2024-08-17T17:33:08.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:33:08.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.140168+0000", 2024-08-17T17:33:08.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016468400000000001 2024-08-17T17:33:08.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:33:08.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.140328+0000", 2024-08-17T17:33:08.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016009499999999999 2024-08-17T17:33:08.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.147712+0000", 2024-08-17T17:33:08.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0073838189999999998 2024-08-17T17:33:08.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:33:08.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.157111+0000", 2024-08-17T17:33:08.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0093994309999999998 2024-08-17T17:33:08.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.346623+0000", 2024-08-17T17:33:08.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18951233000000001 2024-08-17T17:33:08.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:33:08.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.346679+0000", 2024-08-17T17:33:08.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5266999999999998e-05 2024-08-17T17:33:08.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:33:08.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.346726+0000", 2024-08-17T17:33:08.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7135999999999997e-05 2024-08-17T17:33:08.407 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.407 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:33:08.408 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.408 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:163937 7.2 7:4324d5af:::benchmark_data_smithi003_201798_object50715:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-08-17T17:33:08.408 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:31:16.128037+0000", 2024-08-17T17:33:08.408 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 112.06990829900001, 2024-08-17T17:33:08.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.218813173, 2024-08-17T17:33:08.408 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:33:08.408 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:33:08.408 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:08.408 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:33:08.408 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:33:08.409 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:33:08.409 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 163937 2024-08-17T17:33:08.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.409 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:33:08.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:33:08.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.128037+0000", 2024-08-17T17:33:08.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:33:08.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.128037+0000", 2024-08-17T17:33:08.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:33:08.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.128038+0000", 2024-08-17T17:33:08.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2999999999999999e-07 2024-08-17T17:33:08.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:33:08.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.128095+0000", 2024-08-17T17:33:08.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6449000000000001e-05 2024-08-17T17:33:08.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:33:08.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.128105+0000", 2024-08-17T17:33:08.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0787e-05 2024-08-17T17:33:08.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:33:08.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.128112+0000", 2024-08-17T17:33:08.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0369999999999997e-06 2024-08-17T17:33:08.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:33:08.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.147738+0000", 2024-08-17T17:33:08.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.019625249000000001 2024-08-17T17:33:08.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:33:08.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.147985+0000", 2024-08-17T17:33:08.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00024751599999999998 2024-08-17T17:33:08.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:33:08.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.148115+0000", 2024-08-17T17:33:08.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012946100000000001 2024-08-17T17:33:08.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.157096+0000", 2024-08-17T17:33:08.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.008981157 2024-08-17T17:33:08.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:33:08.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.160071+0000", 2024-08-17T17:33:08.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029748560000000001 2024-08-17T17:33:08.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.346763+0000", 2024-08-17T17:33:08.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18669261700000001 2024-08-17T17:33:08.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:33:08.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.346806+0000", 2024-08-17T17:33:08.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2661999999999999e-05 2024-08-17T17:33:08.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:33:08.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.346851+0000", 2024-08-17T17:33:08.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4552000000000003e-05 2024-08-17T17:33:08.414 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.414 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:33:08.414 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.414 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:163946 7.2 7:4656d40a:::benchmark_data_smithi003_201798_object50724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-08-17T17:33:08.415 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:31:16.128551+0000", 2024-08-17T17:33:08.415 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 112.069394297, 2024-08-17T17:33:08.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.218413474, 2024-08-17T17:33:08.415 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:33:08.415 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:33:08.415 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:08.415 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:33:08.415 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:33:08.415 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:33:08.415 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 163946 2024-08-17T17:33:08.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.415 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:33:08.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:33:08.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.128551+0000", 2024-08-17T17:33:08.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.416 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:33:08.416 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.128551+0000", 2024-08-17T17:33:08.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.416 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:33:08.416 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.128552+0000", 2024-08-17T17:33:08.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0419999999999999e-06 2024-08-17T17:33:08.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.416 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:33:08.416 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.128557+0000", 2024-08-17T17:33:08.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4309999999999996e-06 2024-08-17T17:33:08.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.417 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:33:08.417 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.128567+0000", 2024-08-17T17:33:08.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0438999999999999e-05 2024-08-17T17:33:08.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.417 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:33:08.417 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.128574+0000", 2024-08-17T17:33:08.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9310000000000002e-06 2024-08-17T17:33:08.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.417 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:33:08.417 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.154052+0000", 2024-08-17T17:33:08.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.025477494999999999 2024-08-17T17:33:08.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:33:08.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.154153+0000", 2024-08-17T17:33:08.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010163800000000001 2024-08-17T17:33:08.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:33:08.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.154238+0000", 2024-08-17T17:33:08.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4763000000000001e-05 2024-08-17T17:33:08.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:33:08.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.164876+0000", 2024-08-17T17:33:08.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010638257 2024-08-17T17:33:08.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.164940+0000", 2024-08-17T17:33:08.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3732999999999994e-05 2024-08-17T17:33:08.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.346886+0000", 2024-08-17T17:33:08.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18194640500000001 2024-08-17T17:33:08.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:33:08.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.346930+0000", 2024-08-17T17:33:08.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3742000000000002e-05 2024-08-17T17:33:08.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:33:08.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.346965+0000", 2024-08-17T17:33:08.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4598000000000003e-05 2024-08-17T17:33:08.420 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.420 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:33:08.420 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.420 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:163947 7.2 7:42f1f72e:::benchmark_data_smithi003_201798_object50725:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-08-17T17:33:08.420 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:31:16.138013+0000", 2024-08-17T17:33:08.421 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 112.05993263000001, 2024-08-17T17:33:08.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.21738790999999999, 2024-08-17T17:33:08.421 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:33:08.421 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:33:08.421 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:08.421 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:33:08.421 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:33:08.421 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:33:08.421 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 163947 2024-08-17T17:33:08.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.421 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:33:08.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.421 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:33:08.421 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.138013+0000", 2024-08-17T17:33:08.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:33:08.422 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.138013+0000", 2024-08-17T17:33:08.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:33:08.422 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.138015+0000", 2024-08-17T17:33:08.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.508e-06 2024-08-17T17:33:08.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.423 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:33:08.423 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.138018+0000", 2024-08-17T17:33:08.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9419999999999997e-06 2024-08-17T17:33:08.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.423 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:33:08.423 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.138035+0000", 2024-08-17T17:33:08.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6259000000000001e-05 2024-08-17T17:33:08.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.423 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:33:08.423 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.138042+0000", 2024-08-17T17:33:08.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1770000000000001e-06 2024-08-17T17:33:08.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.424 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:33:08.424 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.157189+0000", 2024-08-17T17:33:08.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.019147125000000001 2024-08-17T17:33:08.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.424 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:33:08.424 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.157326+0000", 2024-08-17T17:33:08.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000136635 2024-08-17T17:33:08.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.424 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:33:08.424 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.157461+0000", 2024-08-17T17:33:08.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013518499999999999 2024-08-17T17:33:08.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.164980+0000", 2024-08-17T17:33:08.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0075195059999999996 2024-08-17T17:33:08.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:33:08.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.164995+0000", 2024-08-17T17:33:08.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4729000000000001e-05 2024-08-17T17:33:08.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.426 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.354766+0000", 2024-08-17T17:33:08.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18977096099999999 2024-08-17T17:33:08.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.426 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.426 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:33:08.426 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.355060+0000", 2024-08-17T17:33:08.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00029444999999999997 2024-08-17T17:33:08.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.426 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.426 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:33:08.426 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.355401+0000", 2024-08-17T17:33:08.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00034043300000000001 2024-08-17T17:33:08.426 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.427 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:33:08.427 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.427 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:163956 7.2 7:4172d7d1:::benchmark_data_smithi003_201798_object50734:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-08-17T17:33:08.427 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:31:16.152262+0000", 2024-08-17T17:33:08.427 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 112.045683966, 2024-08-17T17:33:08.427 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.20329513799999999, 2024-08-17T17:33:08.427 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:33:08.427 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:33:08.427 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:08.427 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:33:08.427 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:33:08.427 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:33:08.428 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 163956 2024-08-17T17:33:08.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.428 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:33:08.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:33:08.428 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.152262+0000", 2024-08-17T17:33:08.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:33:08.428 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.152262+0000", 2024-08-17T17:33:08.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:33:08.429 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.152263+0000", 2024-08-17T17:33:08.429 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6700000000000001e-06 2024-08-17T17:33:08.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.429 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.429 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:33:08.429 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.152269+0000", 2024-08-17T17:33:08.429 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0619999999999997e-06 2024-08-17T17:33:08.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.429 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.429 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:33:08.429 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.152287+0000", 2024-08-17T17:33:08.429 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7859999999999998e-05 2024-08-17T17:33:08.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.429 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.430 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:33:08.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.152297+0000", 2024-08-17T17:33:08.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2550000000000003e-06 2024-08-17T17:33:08.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.430 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:33:08.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.160213+0000", 2024-08-17T17:33:08.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0079164200000000004 2024-08-17T17:33:08.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.430 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:33:08.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.160856+0000", 2024-08-17T17:33:08.431 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00064312500000000003 2024-08-17T17:33:08.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.431 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:33:08.431 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.161231+0000", 2024-08-17T17:33:08.431 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00037470400000000001 2024-08-17T17:33:08.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.431 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.431 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.165046+0000", 2024-08-17T17:33:08.431 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038153330000000002 2024-08-17T17:33:08.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.431 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:33:08.431 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.167474+0000", 2024-08-17T17:33:08.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024279430000000001 2024-08-17T17:33:08.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.432 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.432 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.355451+0000", 2024-08-17T17:33:08.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.187976737 2024-08-17T17:33:08.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.432 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:33:08.432 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.355521+0000", 2024-08-17T17:33:08.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0652000000000001e-05 2024-08-17T17:33:08.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.432 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:33:08.432 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.355557+0000", 2024-08-17T17:33:08.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5376999999999997e-05 2024-08-17T17:33:08.433 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.433 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:33:08.433 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.433 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:163960 7.1 7:893915d4:::benchmark_data_smithi003_201798_object50738:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-08-17T17:33:08.433 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:31:16.154340+0000", 2024-08-17T17:33:08.433 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 112.043605781, 2024-08-17T17:33:08.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18201192299999999, 2024-08-17T17:33:08.433 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:33:08.433 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:33:08.433 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:08.433 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:33:08.433 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:33:08.434 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:33:08.434 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 163960 2024-08-17T17:33:08.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.434 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:33:08.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:33:08.434 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.154340+0000", 2024-08-17T17:33:08.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:33:08.434 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.154340+0000", 2024-08-17T17:33:08.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.435 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:33:08.435 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.154341+0000", 2024-08-17T17:33:08.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.049e-06 2024-08-17T17:33:08.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.435 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:33:08.435 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.154346+0000", 2024-08-17T17:33:08.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7809999999999996e-06 2024-08-17T17:33:08.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.435 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:33:08.435 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.154356+0000", 2024-08-17T17:33:08.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0539e-05 2024-08-17T17:33:08.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.435 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:33:08.436 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.154364+0000", 2024-08-17T17:33:08.436 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7719999999999994e-06 2024-08-17T17:33:08.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.436 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:33:08.436 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.154430+0000", 2024-08-17T17:33:08.436 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5795000000000003e-05 2024-08-17T17:33:08.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.436 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:33:08.436 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.154537+0000", 2024-08-17T17:33:08.436 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010755300000000001 2024-08-17T17:33:08.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.436 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:33:08.436 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.154637+0000", 2024-08-17T17:33:08.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010009 2024-08-17T17:33:08.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.157967+0000", 2024-08-17T17:33:08.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003329243 2024-08-17T17:33:08.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:33:08.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.158028+0000", 2024-08-17T17:33:08.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1535e-05 2024-08-17T17:33:08.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.336236+0000", 2024-08-17T17:33:08.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17820820300000001 2024-08-17T17:33:08.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:33:08.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.336310+0000", 2024-08-17T17:33:08.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3460999999999993e-05 2024-08-17T17:33:08.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:33:08.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.336352+0000", 2024-08-17T17:33:08.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1902e-05 2024-08-17T17:33:08.438 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.439 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:33:08.439 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.439 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:26242 8.f 8:f368567c:::benchmark_data_smithi003_229103_object26241:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-08-17T17:33:08.439 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:04.505811+0000", 2024-08-17T17:33:08.439 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 3.6921348510000001, 2024-08-17T17:33:08.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.19378362299999999, 2024-08-17T17:33:08.439 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:33:08.439 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:33:08.439 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:08.439 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:33:08.439 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:33:08.439 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:33:08.439 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26242 2024-08-17T17:33:08.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.440 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:33:08.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:33:08.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:04.505811+0000", 2024-08-17T17:33:08.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:33:08.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:04.505811+0000", 2024-08-17T17:33:08.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:33:08.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:04.505811+0000", 2024-08-17T17:33:08.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7899999999999998e-07 2024-08-17T17:33:08.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.441 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:33:08.441 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:04.505839+0000", 2024-08-17T17:33:08.441 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7644000000000002e-05 2024-08-17T17:33:08.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.441 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:33:08.441 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:04.505848+0000", 2024-08-17T17:33:08.441 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4659999999999999e-06 2024-08-17T17:33:08.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.441 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:33:08.441 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:04.505854+0000", 2024-08-17T17:33:08.441 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0990000000000004e-06 2024-08-17T17:33:08.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.442 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:33:08.442 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:04.505923+0000", 2024-08-17T17:33:08.442 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8831000000000006e-05 2024-08-17T17:33:08.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.442 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:33:08.442 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:04.506040+0000", 2024-08-17T17:33:08.442 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000117651 2024-08-17T17:33:08.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.442 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2", 2024-08-17T17:33:08.442 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:04.529931+0000", 2024-08-17T17:33:08.443 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.023891315 2024-08-17T17:33:08.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.443 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.443 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:04.534017+0000", 2024-08-17T17:33:08.443 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004085862 2024-08-17T17:33:08.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.443 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:33:08.443 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:04.534179+0000", 2024-08-17T17:33:08.443 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000161459 2024-08-17T17:33:08.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.443 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:33:08.444 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:04.534234+0000", 2024-08-17T17:33:08.444 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4789e-05 2024-08-17T17:33:08.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.444 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:33:08.444 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:04.699594+0000", 2024-08-17T17:33:08.444 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.16536082799999999 2024-08-17T17:33:08.444 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.444 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:33:08.444 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.444 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:26243 8.f 8:fbf7332b:::benchmark_data_smithi003_229103_object26242:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-08-17T17:33:08.444 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:04.506730+0000", 2024-08-17T17:33:08.445 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 3.6912160030000001, 2024-08-17T17:33:08.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.19308233999999999, 2024-08-17T17:33:08.445 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:33:08.445 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:33:08.445 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:08.445 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:33:08.445 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:33:08.445 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:33:08.445 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 26243 2024-08-17T17:33:08.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.445 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:33:08.445 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.445 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:33:08.445 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:04.506730+0000", 2024-08-17T17:33:08.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.446 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:33:08.446 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:04.506730+0000", 2024-08-17T17:33:08.446 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:33:08.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.446 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:33:08.446 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:04.506731+0000", 2024-08-17T17:33:08.446 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9199999999999999e-07 2024-08-17T17:33:08.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.446 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:33:08.446 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:04.506748+0000", 2024-08-17T17:33:08.447 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7694e-05 2024-08-17T17:33:08.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.447 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:33:08.447 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:04.529911+0000", 2024-08-17T17:33:08.447 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.023162344000000001 2024-08-17T17:33:08.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.447 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:33:08.447 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:04.529920+0000", 2024-08-17T17:33:08.447 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1039999999999996e-06 2024-08-17T17:33:08.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.447 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:33:08.447 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:04.531420+0000", 2024-08-17T17:33:08.448 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0015003029999999999 2024-08-17T17:33:08.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.448 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:33:08.448 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:04.531621+0000", 2024-08-17T17:33:08.448 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020086999999999999 2024-08-17T17:33:08.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.448 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2", 2024-08-17T17:33:08.448 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:04.531855+0000", 2024-08-17T17:33:08.448 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00023398099999999999 2024-08-17T17:33:08.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.449 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:08.449 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:04.699652+0000", 2024-08-17T17:33:08.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.167796906 2024-08-17T17:33:08.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.449 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:33:08.449 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:04.699664+0000", 2024-08-17T17:33:08.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2506e-05 2024-08-17T17:33:08.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.449 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:33:08.449 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:04.699727+0000", 2024-08-17T17:33:08.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3075000000000002e-05 2024-08-17T17:33:08.450 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:33:08.450 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:33:08.450 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:33:08.450 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:04.699812+0000", 2024-08-17T17:33:08.450 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4665000000000003e-05 2024-08-17T17:33:08.450 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.450 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:33:08.450 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.450 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:33:08.450 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:33:08.450 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:33:08.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:08.452 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 12 16 35374 35358 184.139 137.812 0.00677788 0.00542494 2024-08-17T17:33:08.487 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:08.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:08.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:08.523 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:33:08 smithi003 ceph-osd[230415]: -- 172.21.15.3:0/1484769781 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5610a89f7ba0 con 0x5610a8b5a000 2024-08-17T17:33:08.523 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:33:08 smithi003 ceph-osd[230415]: -- 172.21.15.3:0/1484769781 <== mon.1 v2:172.21.15.53:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5610a89b7a40 con 0x5610a8b5a000 2024-08-17T17:33:08.589 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:08.625 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:08.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:08.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:08.631 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:33:08.727 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:08.765 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:08.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:08.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:08.841 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-17T17:33:08.867 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:08.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:08.960 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:08.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:08.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:33:08.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:33:08.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:33:08.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:33:08.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:33:08.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:33:08.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:33:09.023 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:33:08 smithi003 bash[230057]: 967184dd1121826dd60baf56e6eb0428f31dfe51713cd0fe44b8e9087a768903 2024-08-17T17:33:09.023 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:33:08 smithi003 systemd[1]: Started Ceph osd.1 for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:33:09.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:33:09.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:33:09.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:33:09.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:33:09.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:33:09.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:33:09.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:33:09.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:33:09.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:33:09.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:33:09.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:33:09.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:33:09.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:33:09.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:33:09.061 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:09.086 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:09.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:09.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:09.188 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:09.224 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:09.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:09.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:09.326 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:09.363 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 13 16 37462 37446 180.012 130.5 0.00427962 0.00554968 2024-08-17T17:33:09.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:09.364 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:09.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:09.465 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:09.500 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:09.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:09.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:09.602 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:09.624 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:09.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:09.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:09.726 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:09.753 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:09.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:09.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:09.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:09 smithi003 ceph-mon[100615]: pgmap v794: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 1.7 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 160 MiB/s wr, 2.56k op/s; 13284/81600 objects degraded (16.279%); 1/81600 objects misplaced (0.001%) 2024-08-17T17:33:09.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:09 smithi003 ceph-mon[103344]: pgmap v794: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 1.7 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 160 MiB/s wr, 2.56k op/s; 13284/81600 objects degraded (16.279%); 1/81600 objects misplaced (0.001%) 2024-08-17T17:33:09.854 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:09.889 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:09.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:09.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:09.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:09 smithi053 ceph-mon[89963]: pgmap v794: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 1.7 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 160 MiB/s wr, 2.56k op/s; 13284/81600 objects degraded (16.279%); 1/81600 objects misplaced (0.001%) 2024-08-17T17:33:09.991 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:10.028 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:10.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:10.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:10.130 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:10.167 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:10.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:10.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:10.269 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:10.273 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:33:09 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1[230371]: 2024-08-17T17:33:09.969+0000 7f7d56b17640 -1 Falling back to public interface 2024-08-17T17:33:10.305 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:10.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:10.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:10.363 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 14 16 39436 39420 175.966 123.375 0.050371 0.00566914 2024-08-17T17:33:10.406 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:10.442 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:10.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:10.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:10.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:33:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:33:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:33:10.544 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:10.581 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:10.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:10.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:10.683 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:10.719 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:10.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:10.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:10.821 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:10.853 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:10.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:10.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:10.955 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:10.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:10 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 13284/81600 objects degraded (16.279%), 10 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-08-17T17:33:10.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:10.994 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:10.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:11.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:10 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 13284/81600 objects degraded (16.279%), 10 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-08-17T17:33:11.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:10 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 13284/81600 objects degraded (16.279%), 10 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-08-17T17:33:11.096 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:11.118 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:11.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:11.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:11.220 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:11.257 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:11.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:11.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:11.359 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:11.363 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 15 16 41579 41563 173.163 133.938 0.00853147 0.00577036 2024-08-17T17:33:11.396 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:11.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:11.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:11.498 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:11.520 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:11.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:11.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:11.622 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:11.665 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:11.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:11.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:11.766 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:11.793 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:11.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:11.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:11.895 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:11.932 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:11.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:11.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:11.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:11 smithi053 ceph-mon[89963]: pgmap v795: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 167 MiB/s wr, 2.66k op/s; 15486/108003 objects degraded (14.338%); 1/108003 objects misplaced (0.001%) 2024-08-17T17:33:12.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:11 smithi003 ceph-mon[103344]: pgmap v795: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 167 MiB/s wr, 2.66k op/s; 15486/108003 objects degraded (14.338%); 1/108003 objects misplaced (0.001%) 2024-08-17T17:33:12.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:11 smithi003 ceph-mon[100615]: pgmap v795: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 167 MiB/s wr, 2.66k op/s; 15486/108003 objects degraded (14.338%); 1/108003 objects misplaced (0.001%) 2024-08-17T17:33:12.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:12.068 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:12.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:12.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:12.170 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:12.207 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:12.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:12.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:12.309 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:12.343 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:12.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:12.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:12.363 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 16 16 43530 43514 169.961 121.938 0.00685325 0.00587827 2024-08-17T17:33:12.446 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:12.483 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:12.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:12.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:12.527 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-17T17:33:12.586 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:12.623 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:12.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:12.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:12.702 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:33:12.705 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:33:12.711 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:33:12.725 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:12.761 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:12.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:12.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:12.863 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:12.900 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:12.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:12.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:12.963 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:33:13.002 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:13.038 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:13.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:13.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:13.142 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:13.179 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:13.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:13.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:13.281 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:13.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:13.320 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:13.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:13.363 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 17 16 45496 45480 167.191 122.875 0.0111342 0.00597602 2024-08-17T17:33:13.421 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:13.442 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:13.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:13.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:13.544 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:13.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:13.584 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:13.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:13.685 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:13.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:13.718 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:13.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:13.819 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:13.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:13.858 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:13.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:13.959 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:13.995 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:13.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:13.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:14.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:13 smithi003 ceph-mon[103344]: pgmap v796: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 143 MiB/s wr, 2.29k op/s; 15486/108003 objects degraded (14.338%); 1/108003 objects misplaced (0.001%) 2024-08-17T17:33:14.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:13 smithi003 ceph-mon[100615]: pgmap v796: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 143 MiB/s wr, 2.29k op/s; 15486/108003 objects degraded (14.338%); 1/108003 objects misplaced (0.001%) 2024-08-17T17:33:14.035 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:13 smithi053 ceph-mon[89963]: pgmap v796: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 143 MiB/s wr, 2.29k op/s; 15486/108003 objects degraded (14.338%); 1/108003 objects misplaced (0.001%) 2024-08-17T17:33:14.097 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:14.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:14.133 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:14.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:14.234 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:14.268 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:14.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:14.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:14.364 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 18 16 47471 47455 164.759 123.438 0.0056446 0.00606508 2024-08-17T17:33:14.370 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:14.404 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:14.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:14.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:14.506 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:14.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:14.545 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:14.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:14.646 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:14.680 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:14.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:14.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:14.782 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:14.821 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:14.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:14.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:14.923 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:14.960 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:14.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:14.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:14.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:14 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 15486/108003 objects degraded (14.338%), 10 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-08-17T17:33:15.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:14 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 15486/108003 objects degraded (14.338%), 10 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-08-17T17:33:15.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:14 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 15486/108003 objects degraded (14.338%), 10 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-08-17T17:33:15.062 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:15.100 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:15.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:15.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:15.201 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:15.237 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:15.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:15.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:15.340 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:15.364 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 19 16 49449 49433 162.593 123.625 0.0103295 0.00614528 2024-08-17T17:33:15.378 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:15.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:15.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:15.480 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:15.529 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:15.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:15.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:15.631 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:15.654 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:33:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:33:15] "GET /metrics HTTP/1.1" 200 36509 "" "Prometheus/2.51.0" 2024-08-17T17:33:15.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:15.677 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:15.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:15.778 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:15.808 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:15.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:15.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:15.867 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:15 smithi053 ceph-mon[89963]: pgmap v797: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 2.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 191 MiB/s wr, 3.05k op/s; 19831/137817 objects degraded (14.389%); 1/137817 objects misplaced (0.001%) 2024-08-17T17:33:15.910 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:15.946 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:15.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:15.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:16.003 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:15 smithi003 ceph-mon[103344]: pgmap v797: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 2.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 191 MiB/s wr, 3.05k op/s; 19831/137817 objects degraded (14.389%); 1/137817 objects misplaced (0.001%) 2024-08-17T17:33:16.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:15 smithi003 ceph-mon[100615]: pgmap v797: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 2.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 191 MiB/s wr, 3.05k op/s; 19831/137817 objects degraded (14.389%); 1/137817 objects misplaced (0.001%) 2024-08-17T17:33:16.026 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:33:16.026 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2024-08-17T17:33:16.026 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:16.026 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.46025.0:50617 8.4 e266/262 8:2d54020e:::benchmark_data_smithi003_229103_object50616:head v 266'3178, mlcod=266'3176)", 2024-08-17T17:33:16.027 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:16.021160+0000", 2024-08-17T17:33:16.027 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.0038064890000000001, 2024-08-17T17:33:16.027 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0038400169999999998, 2024-08-17T17:33:16.027 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:16.027 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2024-08-17T17:33:16.027 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:16.027 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:16.027 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:16.027 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:16.021160+0000", 2024-08-17T17:33:16.027 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:16.027 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:16.027 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:16.027 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:16.028 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:16.021160+0000", 2024-08-17T17:33:16.028 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:16.028 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:16.028 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:16.028 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:16.028 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:16.021160+0000", 2024-08-17T17:33:16.028 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:33:16.028 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:16.028 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:16.028 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:16.028 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:16.021180+0000", 2024-08-17T17:33:16.028 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0051000000000002e-05 2024-08-17T17:33:16.028 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:16.029 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:16.029 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:16.029 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:16.021187+0000", 2024-08-17T17:33:16.029 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.4830000000000002e-06 2024-08-17T17:33:16.029 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:16.029 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:16.029 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:16.029 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:16.021191+0000", 2024-08-17T17:33:16.029 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4450000000000001e-06 2024-08-17T17:33:16.029 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:16.029 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:16.029 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:16.029 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:16.021232+0000", 2024-08-17T17:33:16.029 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0878999999999999e-05 2024-08-17T17:33:16.029 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:16.030 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:16.030 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:16.030 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:16.021249+0000", 2024-08-17T17:33:16.030 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7476999999999999e-05 2024-08-17T17:33:16.030 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:16.030 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:16.030 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:16.030 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:16.030 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:16.030 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:50619 8.2 8:4fbf8bc3:::benchmark_data_smithi003_229103_object50618:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-08-17T17:33:16.030 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:16.021369+0000", 2024-08-17T17:33:16.030 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.003597762, 2024-08-17T17:33:16.030 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0037032039999999999, 2024-08-17T17:33:16.030 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:16.031 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:33:16.031 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:16.031 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:33:16.031 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:33:16.031 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 50619 2024-08-17T17:33:16.031 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:16.031 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:16.031 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:16.031 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:16.031 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:16.021369+0000", 2024-08-17T17:33:16.031 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:16.031 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:16.031 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:16.031 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:16.031 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:16.021369+0000", 2024-08-17T17:33:16.031 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:16.032 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:16.032 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:16.032 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:16.032 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:16.021368+0000", 2024-08-17T17:33:16.032 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:33:16.032 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:16.032 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:16.032 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:16.032 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:16.021403+0000", 2024-08-17T17:33:16.032 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5142000000000003e-05 2024-08-17T17:33:16.032 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:16.032 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:16.032 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:16.032 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:16.021413+0000", 2024-08-17T17:33:16.032 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0723000000000001e-05 2024-08-17T17:33:16.032 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:16.033 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:16.033 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:16.033 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:16.021419+0000", 2024-08-17T17:33:16.033 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.31e-06 2024-08-17T17:33:16.033 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:16.033 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:16.033 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:16.033 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:16.021467+0000", 2024-08-17T17:33:16.033 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7886000000000002e-05 2024-08-17T17:33:16.033 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:16.033 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:16.033 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:16.033 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:16.021596+0000", 2024-08-17T17:33:16.033 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012890199999999999 2024-08-17T17:33:16.033 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:16.034 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:16.034 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:33:16.034 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:16.021743+0000", 2024-08-17T17:33:16.034 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014782599999999999 2024-08-17T17:33:16.034 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:16.034 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:16.034 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:16.034 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:16.025082+0000", 2024-08-17T17:33:16.034 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0033388609999999998 2024-08-17T17:33:16.034 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:16.034 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:16.034 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:16.034 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:16.034 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:16.035 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:50627 8.2 8:4174b382:::benchmark_data_smithi003_229103_object50626:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-08-17T17:33:16.035 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:16.024979+0000", 2024-08-17T17:33:16.035 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4294967295.9999881, 2024-08-17T17:33:16.035 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000174248, 2024-08-17T17:33:16.035 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:16.035 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2024-08-17T17:33:16.035 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:16.035 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:33:16.035 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:33:16.035 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 50627 2024-08-17T17:33:16.035 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:16.035 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:16.035 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:16.035 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:16.036 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:16.024979+0000", 2024-08-17T17:33:16.036 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:16.036 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:16.036 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:16.036 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:16.036 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:16.024979+0000", 2024-08-17T17:33:16.036 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:16.036 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:16.036 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:16.036 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:16.036 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:16.024977+0000", 2024-08-17T17:33:16.036 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:33:16.036 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:16.036 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:16.037 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:16.037 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:16.025019+0000", 2024-08-17T17:33:16.037 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1681000000000003e-05 2024-08-17T17:33:16.037 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:16.037 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:16.037 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:16.037 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:16.025031+0000", 2024-08-17T17:33:16.037 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1969e-05 2024-08-17T17:33:16.037 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:16.037 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:16.037 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:16.037 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:16.025036+0000", 2024-08-17T17:33:16.037 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6239999999999999e-06 2024-08-17T17:33:16.037 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:16.037 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:16.038 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:16.038 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:16.025106+0000", 2024-08-17T17:33:16.038 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9348000000000001e-05 2024-08-17T17:33:16.038 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:16.038 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:16.038 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:16.038 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:16.038 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-08-17T17:33:16.038 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 3 2024-08-17T17:33:16.038 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:33:16.049 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:16.087 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:16.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:16.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:16.190 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:16.229 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:16.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:16.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:16.330 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:16.364 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-08-17T17:33:16.364226+0000 min lat: 0.00109625 max lat: 0.193186 avg lat: 0.00625076 2024-08-17T17:33:16.364 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:33:16.364 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 20 16 51171 51155 159.844 107.625 0.0140806 0.00625076 2024-08-17T17:33:16.373 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:16.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:16.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:16.475 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:16.511 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:16.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:16.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:16.615 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:16.651 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:16.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:16.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:16.753 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:16.787 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:16.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:16.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:16.888 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-17T17:33:16.889 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:16.928 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:16.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:16.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:16.995 INFO:teuthology.orchestra.run.smithi003.stderr:no valid command found; 10 closest matches: 2024-08-17T17:33:16.996 INFO:teuthology.orchestra.run.smithi003.stderr:0 2024-08-17T17:33:16.996 INFO:teuthology.orchestra.run.smithi003.stderr:1 2024-08-17T17:33:16.996 INFO:teuthology.orchestra.run.smithi003.stderr:2 2024-08-17T17:33:16.996 INFO:teuthology.orchestra.run.smithi003.stderr:abort 2024-08-17T17:33:16.996 INFO:teuthology.orchestra.run.smithi003.stderr:assert 2024-08-17T17:33:16.996 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs debug_inject_read_zeros 2024-08-17T17:33:16.996 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs files list 2024-08-17T17:33:16.996 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs stats 2024-08-17T17:33:16.996 INFO:teuthology.orchestra.run.smithi003.stderr:bluestore allocator dump block 2024-08-17T17:33:16.996 INFO:teuthology.orchestra.run.smithi003.stderr:bluestore allocator fragmentation block 2024-08-17T17:33:16.997 INFO:teuthology.orchestra.run.smithi003.stderr:admin_socket: invalid command 2024-08-17T17:33:17.030 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:17.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:17.071 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:17.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:17.172 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:17.207 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:17.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:17.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:17.309 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:17.360 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:17.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:17.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:17.362 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:33:17.364 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 21 16 52846 52830 157.217 104.688 0.0108891 0.00635601 2024-08-17T17:33:17.462 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:17.501 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:17.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:17.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:17.603 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:17.640 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:17.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:17.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:17.742 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:17.782 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:17.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:17.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:17.813 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:17 smithi053 ceph-mon[89963]: pgmap v798: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 122 MiB/s wr, 1.95k op/s; 19831/137817 objects degraded (14.389%); 1/137817 objects misplaced (0.001%) 2024-08-17T17:33:17.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:17.921 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:17.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:17.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:18.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:17 smithi003 ceph-mon[103344]: pgmap v798: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 122 MiB/s wr, 1.95k op/s; 19831/137817 objects degraded (14.389%); 1/137817 objects misplaced (0.001%) 2024-08-17T17:33:18.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:17 smithi003 ceph-mon[100615]: pgmap v798: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 122 MiB/s wr, 1.95k op/s; 19831/137817 objects degraded (14.389%); 1/137817 objects misplaced (0.001%) 2024-08-17T17:33:18.023 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:18.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:18.060 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:18.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:18.161 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:18.199 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:18.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:18.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:18.301 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:18.335 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:18.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:18.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:18.364 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 22 16 54485 54469 154.727 102.438 0.00423819 0.00645806 2024-08-17T17:33:18.437 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:18.462 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:18.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:18.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:18.564 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:18.599 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:18.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:18.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:18.701 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:18.730 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:18.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:18.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:18.833 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:18.839 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:18 smithi003 ceph-mon[100615]: from='osd.1 [v2:172.21.15.3:6810/5043585,v1:172.21.15.3:6811/5043585]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-08-17T17:33:18.840 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:18 smithi003 ceph-mon[100615]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-08-17T17:33:18.840 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:18 smithi003 ceph-mon[103344]: from='osd.1 [v2:172.21.15.3:6810/5043585,v1:172.21.15.3:6811/5043585]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-08-17T17:33:18.840 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:18 smithi003 ceph-mon[103344]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-08-17T17:33:18.840 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:33:18 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1[230371]: 2024-08-17T17:33:18.525+0000 7f7d56b17640 -1 osd.1 248 log_to_monitors true 2024-08-17T17:33:18.840 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:33:18 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1[230371]: 2024-08-17T17:33:18.833+0000 7f7d4a8b9640 -1 osd.1 267 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-17T17:33:18.860 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:18 smithi053 ceph-mon[89963]: from='osd.1 [v2:172.21.15.3:6810/5043585,v1:172.21.15.3:6811/5043585]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-08-17T17:33:18.861 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:18 smithi053 ceph-mon[89963]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-08-17T17:33:18.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:18.862 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:18.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:18.964 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:18.999 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:19.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:19.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:19.101 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:19.135 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:19.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:19.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:19.237 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:19.275 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:19.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:19.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:19.364 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 23 16 56187 56171 152.624 106.375 0.00713295 0.00654717 2024-08-17T17:33:19.377 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:19.406 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:19.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:19.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:19.508 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:19.541 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:19.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:19.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:19.644 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:19.680 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:19.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:19.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:19.782 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:19.817 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:19.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:19.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:19.919 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:19.941 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:19.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:19.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:20.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:19 smithi003 ceph-mon[103344]: pgmap v799: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 125 MiB/s wr, 2.00k op/s; 22177/144855 objects degraded (15.310%); 1/144855 objects misplaced (0.001%) 2024-08-17T17:33:20.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:19 smithi003 ceph-mon[103344]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-17T17:33:20.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:19 smithi003 ceph-mon[103344]: osdmap e267: 8 total, 7 up, 8 in 2024-08-17T17:33:20.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:19 smithi003 ceph-mon[103344]: from='osd.1 [v2:172.21.15.3:6810/5043585,v1:172.21.15.3:6811/5043585]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-08-17T17:33:20.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:19 smithi003 ceph-mon[103344]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-08-17T17:33:20.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:19 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 22177/144855 objects degraded (15.310%), 10 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-08-17T17:33:20.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:19 smithi003 ceph-mon[100615]: pgmap v799: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 125 MiB/s wr, 2.00k op/s; 22177/144855 objects degraded (15.310%); 1/144855 objects misplaced (0.001%) 2024-08-17T17:33:20.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:19 smithi003 ceph-mon[100615]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-17T17:33:20.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:19 smithi003 ceph-mon[100615]: osdmap e267: 8 total, 7 up, 8 in 2024-08-17T17:33:20.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:19 smithi003 ceph-mon[100615]: from='osd.1 [v2:172.21.15.3:6810/5043585,v1:172.21.15.3:6811/5043585]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-08-17T17:33:20.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:19 smithi003 ceph-mon[100615]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-08-17T17:33:20.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:19 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 22177/144855 objects degraded (15.310%), 10 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-08-17T17:33:20.043 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:20.065 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:20.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:20.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:20.166 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:20.189 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:20.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:20.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:20.214 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:19 smithi053 ceph-mon[89963]: pgmap v799: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 125 MiB/s wr, 2.00k op/s; 22177/144855 objects degraded (15.310%); 1/144855 objects misplaced (0.001%) 2024-08-17T17:33:20.215 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:19 smithi053 ceph-mon[89963]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-17T17:33:20.215 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:19 smithi053 ceph-mon[89963]: osdmap e267: 8 total, 7 up, 8 in 2024-08-17T17:33:20.215 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:19 smithi053 ceph-mon[89963]: from='osd.1 [v2:172.21.15.3:6810/5043585,v1:172.21.15.3:6811/5043585]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-08-17T17:33:20.215 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:19 smithi053 ceph-mon[89963]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]} : dispatch 2024-08-17T17:33:20.215 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:19 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 22177/144855 objects degraded (15.310%), 10 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-08-17T17:33:20.247 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-17T17:33:20.247 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-08-17T17:33:20.291 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:20.313 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:20.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:20.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:20.364 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 24 16 56813 56797 147.895 39.125 0.00282906 0.00659649 2024-08-17T17:33:20.416 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:20.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:20.439 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:20.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:20.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:33:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:33:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:33:20.540 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:20.562 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:20.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:20.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:20.664 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:20.686 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:20.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:20.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:20.788 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:20.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:20.816 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:20.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:20.917 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:20.940 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:20.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:20.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:20.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:20 smithi053 ceph-mon[89963]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T17:33:20.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:20 smithi053 ceph-mon[89963]: osd.1 [v2:172.21.15.3:6810/5043585,v1:172.21.15.3:6811/5043585] boot 2024-08-17T17:33:20.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:20 smithi053 ceph-mon[89963]: osdmap e268: 8 total, 8 up, 8 in 2024-08-17T17:33:20.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:20 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-17T17:33:20.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:20 smithi053 ceph-mon[89963]: osdmap e269: 8 total, 8 up, 8 in 2024-08-17T17:33:21.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:20 smithi003 ceph-mon[103344]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T17:33:21.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:20 smithi003 ceph-mon[103344]: osd.1 [v2:172.21.15.3:6810/5043585,v1:172.21.15.3:6811/5043585] boot 2024-08-17T17:33:21.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:20 smithi003 ceph-mon[103344]: osdmap e268: 8 total, 8 up, 8 in 2024-08-17T17:33:21.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:20 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-17T17:33:21.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:20 smithi003 ceph-mon[103344]: osdmap e269: 8 total, 8 up, 8 in 2024-08-17T17:33:21.039 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:20 smithi003 ceph-mon[100615]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-08-17T17:33:21.039 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:20 smithi003 ceph-mon[100615]: osd.1 [v2:172.21.15.3:6810/5043585,v1:172.21.15.3:6811/5043585] boot 2024-08-17T17:33:21.039 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:20 smithi003 ceph-mon[100615]: osdmap e268: 8 total, 8 up, 8 in 2024-08-17T17:33:21.039 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:20 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-17T17:33:21.039 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:20 smithi003 ceph-mon[100615]: osdmap e269: 8 total, 8 up, 8 in 2024-08-17T17:33:21.042 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:21.065 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:21.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:21.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:21.167 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:21.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:21.192 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:21.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:21.293 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:21.338 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:21.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:21.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:21.364 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 25 16 56813 56797 141.979 0 - 0.00659649 2024-08-17T17:33:21.440 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:21.467 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:21.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:21.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:21.569 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:21.573 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:33:21.573 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2024-08-17T17:33:21.573 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2024-08-17T17:33:21.573 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2024-08-17T17:33:21.573 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:33:21.592 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:21.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:21.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:21.666 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:33:21.666 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:33:21.683 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:33:21.683 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-08-17T17:33:21.694 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:21.699 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:33:21.699 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:33:21.715 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:33:21.715 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:33:21.720 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:21.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:21.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:21.731 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:33:21.732 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:33:21.747 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:33:21.747 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:33:21.759 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-08-17T17:33:21.779 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:33:21.779 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:33:21.816 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:33:21.816 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:33:21.822 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:21.859 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:21.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:21.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:21.961 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:21.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:21 smithi053 ceph-mon[89963]: pgmap v802: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 148 MiB/s wr, 2.37k op/s; 23882/164814 objects degraded (14.490%); 1/164814 objects misplaced (0.001%) 2024-08-17T17:33:21.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:21 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/4192136313' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:33:21.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:21 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:33:21.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:21 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3984425015' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:33:21.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:21 smithi053 ceph-mon[89963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:33:21.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:21 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:33:21.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:21 smithi053 ceph-mon[89963]: osdmap e270: 8 total, 8 up, 8 in 2024-08-17T17:33:21.995 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:21.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:21.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:22.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:21 smithi003 ceph-mon[103344]: pgmap v802: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 148 MiB/s wr, 2.37k op/s; 23882/164814 objects degraded (14.490%); 1/164814 objects misplaced (0.001%) 2024-08-17T17:33:22.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:21 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/4192136313' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:33:22.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:21 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:33:22.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:21 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3984425015' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:33:22.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:21 smithi003 ceph-mon[103344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:33:22.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:21 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:33:22.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:21 smithi003 ceph-mon[103344]: osdmap e270: 8 total, 8 up, 8 in 2024-08-17T17:33:22.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:21 smithi003 ceph-mon[100615]: pgmap v802: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 148 MiB/s wr, 2.37k op/s; 23882/164814 objects degraded (14.490%); 1/164814 objects misplaced (0.001%) 2024-08-17T17:33:22.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:21 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/4192136313' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:33:22.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:21 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:33:22.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:21 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3984425015' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:33:22.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:21 smithi003 ceph-mon[100615]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:33:22.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:21 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:33:22.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:21 smithi003 ceph-mon[100615]: osdmap e270: 8 total, 8 up, 8 in 2024-08-17T17:33:22.097 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:22.125 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:22.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:22.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:22.226 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:22.253 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:22.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:22.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:22.355 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:22.364 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 26 16 57187 57171 137.416 11.6875 0.00397572 0.00727064 2024-08-17T17:33:22.395 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:22.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:22.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:22.497 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:22.523 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:22.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:22.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:22.626 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:22.655 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:22.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:22.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:22.757 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:22.800 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:22.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:22.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:22.902 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:22.934 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:22.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:22.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:22.952 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-17T17:33:23.036 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:23.079 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:23.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:23.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:23.181 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:23.227 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:23.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:23.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:23.329 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:23.349 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:33:23.364 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 27 16 58204 58188 134.681 63.5625 0.0201548 0.00741843 2024-08-17T17:33:23.375 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:23.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:23.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:23.477 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:23.506 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:23.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:23.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:23.608 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:23.635 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:23.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:23.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:23.737 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:23.772 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:23.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:23.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:23.874 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:23.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:23.907 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:23.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:24.008 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:24.033 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:24.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:24.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:24.135 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:24.161 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:24.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:24.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:24.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:23 smithi053 ceph-mon[89963]: pgmap v805: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s; 23882/164814 objects degraded (14.490%); 1/164814 objects misplaced (0.001%) 2024-08-17T17:33:24.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:33:24.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:33:24.263 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:24.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:23 smithi003 ceph-mon[103344]: pgmap v805: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s; 23882/164814 objects degraded (14.490%); 1/164814 objects misplaced (0.001%) 2024-08-17T17:33:24.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:33:24.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:33:24.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:23 smithi003 ceph-mon[100615]: pgmap v805: 89 pgs: 8 active+clean+remapped, 20 active+undersized, 10 active+undersized+degraded, 51 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s; 23882/164814 objects degraded (14.490%); 1/164814 objects misplaced (0.001%) 2024-08-17T17:33:24.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:33:24.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:33:24.292 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:24.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:24.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:24.365 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 28 16 59253 59237 132.212 65.5625 0.0130168 0.00755636 2024-08-17T17:33:24.394 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:24.442 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:24.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:24.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:24.544 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:24.588 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:24.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:24.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:24.690 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:24.720 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:24.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:24.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:24.787 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:33:24.822 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:24.849 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:24.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:24.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:24.951 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:25.003 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:25.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:25.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:25.065 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:24 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 23882/164814 objects degraded (14.490%), 10 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-08-17T17:33:25.105 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:25.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:25.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:25.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:25.175 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:24 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 23882/164814 objects degraded (14.490%), 10 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-08-17T17:33:25.175 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:24 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 23882/164814 objects degraded (14.490%), 10 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2024-08-17T17:33:25.222 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:33:25.248 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-17T17:33:25.255 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:25.283 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:25.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:25.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:25.310 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:33:25.310 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:33:25.321 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:33:25.368 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 29 16 60178 60162 129.634 57.8125 0.0319956 0.00770522 2024-08-17T17:33:25.385 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:25.411 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:25.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:25.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:25.513 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:25.579 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:25.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:25.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:25.681 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:25.752 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:25.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:25.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:25.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:33:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:33:25] "GET /metrics HTTP/1.1" 200 36362 "" "Prometheus/2.51.0" 2024-08-17T17:33:25.855 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:25.884 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:25.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:25.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:25.986 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:26.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:26.035 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:26.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:26.064 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:25 smithi003 ceph-mon[100615]: pgmap v806: 89 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized, 2 active+undersized+degraded, 80 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 110 MiB/s wr, 1.77k op/s; 24845/174753 objects degraded (14.217%); 180 KiB/s, 0 objects/s recovering 2024-08-17T17:33:26.065 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:25 smithi003 ceph-mon[103344]: pgmap v806: 89 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized, 2 active+undersized+degraded, 80 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 110 MiB/s wr, 1.77k op/s; 24845/174753 objects degraded (14.217%); 180 KiB/s, 0 objects/s recovering 2024-08-17T17:33:26.135 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:26.136 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:25 smithi053 ceph-mon[89963]: pgmap v806: 89 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized, 2 active+undersized+degraded, 80 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 110 MiB/s wr, 1.77k op/s; 24845/174753 objects degraded (14.217%); 180 KiB/s, 0 objects/s recovering 2024-08-17T17:33:26.171 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-17T17:33:26.179 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:26.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:26.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:26.281 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:26.320 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:26.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:26.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:26.368 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 30 16 61007 60991 127.04 51.8125 0.0691701 0.00786378 2024-08-17T17:33:26.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:26.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:26.468 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:26.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:26.530 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:33:26.568 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:26.602 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:26.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:26.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:26.704 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:26.731 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:26.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:26.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:26.833 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:26.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:26.866 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:26.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:26.948 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:26 smithi053 ceph-mon[89963]: pgmap v807: 89 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 82 active+clean; 3.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 36 MiB/s wr, 575 op/s; 24936/175713 objects degraded (14.191%); 193 KiB/s, 0 objects/s recovering 2024-08-17T17:33:26.968 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:26.989 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:33:26.997 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:26.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:26.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:27.047 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:33:27.099 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:27.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:27.145 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:27.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:27.155 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:26 smithi003 ceph-mon[100615]: pgmap v807: 89 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 82 active+clean; 3.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 36 MiB/s wr, 575 op/s; 24936/175713 objects degraded (14.191%); 193 KiB/s, 0 objects/s recovering 2024-08-17T17:33:27.158 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:26 smithi003 ceph-mon[103344]: pgmap v807: 89 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 82 active+clean; 3.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 36 MiB/s wr, 575 op/s; 24936/175713 objects degraded (14.191%); 193 KiB/s, 0 objects/s recovering 2024-08-17T17:33:27.246 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:27.276 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:27.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:27.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:27.368 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 31 16 61865 61849 124.672 53.625 0.0181465 0.00801296 2024-08-17T17:33:27.378 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:27.425 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:27.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:27.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:27.527 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:27.554 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:27.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:27.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:27.656 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:27.686 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:27.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:27.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:27.787 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:27.816 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:27.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:27.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:27.918 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:27.957 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:27.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:27.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:28.059 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:28.103 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:28.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:28.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:28.205 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:28.261 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:28.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:28.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:28.363 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:28.368 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 32 16 62771 62755 122.545 56.625 0.021434 0.00815429 2024-08-17T17:33:28.404 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:28.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:28.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:28.418 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:33:28.418 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:33:28.506 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:28.533 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:28.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:28.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:28.635 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:28.662 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:28.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:28.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:28.764 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:28.791 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:28.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:28.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:28.893 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:28.939 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:28.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:28.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:29.041 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:29.077 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:29.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:29.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:29.113 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-17T17:33:29.179 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:29.215 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:29.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:29.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:29.317 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:29.352 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:29.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:29.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:29.368 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 33 16 63692 63676 120.576 57.5625 0.0112901 0.0082873 2024-08-17T17:33:29.453 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:29.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:29 smithi053 ceph-mon[89963]: pgmap v808: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 37 MiB/s wr, 598 op/s; 24838/179187 objects degraded (13.861%); 936 KiB/s, 12 objects/s recovering 2024-08-17T17:33:29.478 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:33:29.493 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:29.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:29.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:29.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:29 smithi003 ceph-mon[103344]: pgmap v808: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 37 MiB/s wr, 598 op/s; 24838/179187 objects degraded (13.861%); 936 KiB/s, 12 objects/s recovering 2024-08-17T17:33:29.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:29 smithi003 ceph-mon[100615]: pgmap v808: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 37 MiB/s wr, 598 op/s; 24838/179187 objects degraded (13.861%); 936 KiB/s, 12 objects/s recovering 2024-08-17T17:33:29.595 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:29.622 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:29.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:29.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:29.724 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:29.753 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:29.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:29.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:29.855 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:29.886 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:29.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:29.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:29.988 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:30.014 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:30.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:30.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:30.116 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:30.142 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:30.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:30.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:30.244 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:30.267 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:30 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 24838/179187 objects degraded (13.861%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:33:30.268 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:30 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 24838/179187 objects degraded (13.861%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:33:30.289 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:30.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:30.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:30.368 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 34 16 64672 64656 118.83 61.25 0.0388714 0.00840932 2024-08-17T17:33:30.391 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:30.414 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:33:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:33:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:33:30.415 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:30 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 24838/179187 objects degraded (13.861%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:33:30.421 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:30.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:30.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:30.523 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:30.560 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:30.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:30.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:30.662 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:30.705 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:30.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:30.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:30.806 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:30.832 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:30.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:30.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:30.933 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:30.957 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:30.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:30.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:30.984 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:33:30.984 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:33:31.059 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:31.098 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:31.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:31.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:31.201 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:31.239 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:31.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:31.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:31.341 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:31.368 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 35 16 65560 65544 117.02 55.5 0.00560422 0.00853812 2024-08-17T17:33:31.382 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:31.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:31.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:31.465 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:31 smithi053 ceph-mon[89963]: pgmap v809: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 54 MiB/s wr, 871 op/s; 24838/189174 objects degraded (13.130%); 804 KiB/s, 10 objects/s recovering 2024-08-17T17:33:31.483 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:31.516 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:31.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:31.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:31.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:31 smithi003 ceph-mon[100615]: pgmap v809: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 54 MiB/s wr, 871 op/s; 24838/189174 objects degraded (13.130%); 804 KiB/s, 10 objects/s recovering 2024-08-17T17:33:31.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:31 smithi003 ceph-mon[103344]: pgmap v809: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 54 MiB/s wr, 871 op/s; 24838/189174 objects degraded (13.130%); 804 KiB/s, 10 objects/s recovering 2024-08-17T17:33:31.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:31.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:31.662 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:31.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:31.762 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:31.795 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:31.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:31.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:31.896 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:31.899 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:33:31.899 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:33:31.934 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:31.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:31.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:32.036 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:32.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:32.077 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:32.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:32.161 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-08-17T17:33:32.178 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:32.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:32.276 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:32.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:32.369 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 36 16 66528 66512 115.45 60.5 0.00728187 0.00865502 2024-08-17T17:33:32.377 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:32.404 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:32.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:32.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:32.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:32 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/930694150' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:33:32.506 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:32.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:32 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/930694150' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:33:32.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:32 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/930694150' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:33:32.532 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:32.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:32.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:32.591 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-17T17:33:32.635 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:32.673 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:32.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:32.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:32.754 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:33:32.755 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:33:32.775 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:32.788 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:33:32.788 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:33:32.804 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:32.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:32.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:32.827 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:33:32.827 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:33:32.884 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:33:32.884 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:33:32.906 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:32.920 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:33:32.920 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:33:32.951 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:33:32.952 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:33:32.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:32.953 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:32.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:32.963 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:33:32.981 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:33:32.981 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:33:33.010 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:33:33.010 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:33:33.054 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:33:33.096 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:33.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:33.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:33:33.198 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:33.238 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:33.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:33.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:33.341 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:33.350 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:33 smithi053 ceph-mon[89963]: pgmap v810: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 49 MiB/s wr, 786 op/s; 24838/189174 objects degraded (13.130%); 726 KiB/s, 9 objects/s recovering 2024-08-17T17:33:33.350 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:33 smithi053 ceph-mon[89963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:33:33.350 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:33 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/930694150' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:33:33.351 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:33 smithi053 ceph-mon[89963]: osdmap e271: 8 total, 8 up, 8 in 2024-08-17T17:33:33.351 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:33 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/900956869' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:33:33.369 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 37 16 67471 67455 113.923 58.9375 0.0245726 0.00877119 2024-08-17T17:33:33.382 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:33.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:33.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:33.484 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:33.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:33 smithi003 ceph-mon[103344]: pgmap v810: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 49 MiB/s wr, 786 op/s; 24838/189174 objects degraded (13.130%); 726 KiB/s, 9 objects/s recovering 2024-08-17T17:33:33.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:33 smithi003 ceph-mon[103344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:33:33.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:33 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/930694150' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:33:33.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:33 smithi003 ceph-mon[103344]: osdmap e271: 8 total, 8 up, 8 in 2024-08-17T17:33:33.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:33 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/900956869' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:33:33.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:33 smithi003 ceph-mon[100615]: pgmap v810: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 49 MiB/s wr, 786 op/s; 24838/189174 objects degraded (13.130%); 726 KiB/s, 9 objects/s recovering 2024-08-17T17:33:33.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:33 smithi003 ceph-mon[100615]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:33:33.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:33 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/930694150' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:33:33.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:33 smithi003 ceph-mon[100615]: osdmap e271: 8 total, 8 up, 8 in 2024-08-17T17:33:33.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:33 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/900956869' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:33:33.525 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:33.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:33.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:33.627 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:33.661 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:33.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:33.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:33.762 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:33.796 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:33.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:33.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:33.898 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:33.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:33.921 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:33.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:34.022 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:34.059 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:34.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:34.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:34.161 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:34.194 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:34.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:34.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:34.296 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:34.330 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:34.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:34.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:34.369 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 38 16 68324 68308 112.328 53.3125 0.0124047 0.00889618 2024-08-17T17:33:34.432 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:34.476 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:34.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:34.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:34.578 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:33:34.617 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:34.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:34.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:33:34.722 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:34.773 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:34.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:34.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:34.875 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:34.898 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:34.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:34.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:34.999 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:35.035 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:35.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:35.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:35.137 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:35.160 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:35.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:35.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:35.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:35 smithi053 ceph-mon[89963]: pgmap v812: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 58 MiB/s wr, 923 op/s; 24731/202476 objects degraded (12.214%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:33:35.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:35 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 24731/202476 objects degraded (12.214%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:33:35.262 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:35.299 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:35.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:35.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:35.337 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:33:35.369 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 39 16 69006 68990 110.541 42.625 0.0114562 0.00904035 2024-08-17T17:33:35.400 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:35.428 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:35.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:35.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:35.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:35 smithi003 ceph-mon[103344]: pgmap v812: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 58 MiB/s wr, 923 op/s; 24731/202476 objects degraded (12.214%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:33:35.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:35 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 24731/202476 objects degraded (12.214%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:33:35.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:35 smithi003 ceph-mon[100615]: pgmap v812: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 58 MiB/s wr, 923 op/s; 24731/202476 objects degraded (12.214%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:33:35.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:35 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 24731/202476 objects degraded (12.214%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:33:35.530 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:35.565 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:33:35.565 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2024-08-17T17:33:35.565 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2024-08-17T17:33:35.565 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2024-08-17T17:33:35.565 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.565 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82573 6.d 6:bd9981c1:::benchmark_data_smithi003_175043_object82572:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:33:35.565 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.116209+0000", 2024-08-17T17:33:35.565 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 476.44500383399998, 2024-08-17T17:33:35.566 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6778493189999999, 2024-08-17T17:33:35.566 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:35.566 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2024-08-17T17:33:35.566 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:35.566 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:35.566 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:35.566 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82573 2024-08-17T17:33:35.566 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.566 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:35.566 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.566 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:35.566 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.116209+0000", 2024-08-17T17:33:35.566 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.567 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.567 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.567 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:35.567 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.116209+0000", 2024-08-17T17:33:35.567 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.567 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.567 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.567 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:35.567 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.116207+0000", 2024-08-17T17:33:35.567 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:33:35.567 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.567 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.567 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:35.568 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.116249+0000", 2024-08-17T17:33:35.568 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1340000000000001e-05 2024-08-17T17:33:35.568 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.568 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.568 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:35.568 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.116266+0000", 2024-08-17T17:33:35.568 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7655e-05 2024-08-17T17:33:35.568 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.568 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.568 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:35.568 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.116273+0000", 2024-08-17T17:33:35.568 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0840000000000003e-06 2024-08-17T17:33:35.568 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.568 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.569 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.569 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.116355+0000", 2024-08-17T17:33:35.569 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1224999999999996e-05 2024-08-17T17:33:35.569 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.569 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.569 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:35.569 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.116366+0000", 2024-08-17T17:33:35.569 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1092e-05 2024-08-17T17:33:35.569 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.569 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.569 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.569 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.775203+0000", 2024-08-17T17:33:35.569 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.65883687 2024-08-17T17:33:35.569 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.570 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.570 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for degraded object", 2024-08-17T17:33:35.570 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.775490+0000", 2024-08-17T17:33:35.570 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00028773399999999999 2024-08-17T17:33:35.570 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.570 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.570 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.570 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794011+0000", 2024-08-17T17:33:35.570 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.018521115000000001 2024-08-17T17:33:35.570 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.570 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.570 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:35.570 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794059+0000", 2024-08-17T17:33:35.570 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7213999999999999e-05 2024-08-17T17:33:35.571 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.571 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:35.571 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.571 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.571 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.571 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82600 6.d 6:b9e24bf2:::benchmark_data_smithi003_175043_object82599:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:33:35.571 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.137681+0000", 2024-08-17T17:33:35.571 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 476.423532067, 2024-08-17T17:33:35.571 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.656490799, 2024-08-17T17:33:35.571 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:35.571 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:35.571 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:35.571 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:35.571 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:35.572 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82600 2024-08-17T17:33:35.572 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.572 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:35.572 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.572 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:35.572 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.137681+0000", 2024-08-17T17:33:35.572 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.572 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.572 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.572 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:35.572 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.137681+0000", 2024-08-17T17:33:35.572 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.572 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.572 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.573 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:35.573 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.137680+0000", 2024-08-17T17:33:35.573 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:33:35.573 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.573 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.573 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:35.573 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.137739+0000", 2024-08-17T17:33:35.573 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9840999999999998e-05 2024-08-17T17:33:35.573 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.573 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.573 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:35.573 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.137752+0000", 2024-08-17T17:33:35.573 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2431e-05 2024-08-17T17:33:35.573 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.573 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.574 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:35.574 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.137757+0000", 2024-08-17T17:33:35.574 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.023e-06 2024-08-17T17:33:35.574 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.574 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.574 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.574 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.137822+0000", 2024-08-17T17:33:35.574 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4703999999999996e-05 2024-08-17T17:33:35.574 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.574 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.574 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:35.574 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.137832+0000", 2024-08-17T17:33:35.574 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0244000000000001e-05 2024-08-17T17:33:35.575 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.575 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.575 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.575 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.775512+0000", 2024-08-17T17:33:35.575 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.637680322 2024-08-17T17:33:35.575 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.575 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.575 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:35.575 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.775598+0000", 2024-08-17T17:33:35.575 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.6287999999999998e-05 2024-08-17T17:33:35.575 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.575 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.575 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:33:35.575 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.775699+0000", 2024-08-17T17:33:35.576 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000100445 2024-08-17T17:33:35.576 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.576 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.576 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:35.576 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.785737+0000", 2024-08-17T17:33:35.576 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010038126 2024-08-17T17:33:35.576 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.576 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.576 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.576 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.793769+0000", 2024-08-17T17:33:35.576 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0080316069999999996 2024-08-17T17:33:35.576 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.576 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.576 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.576 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794078+0000", 2024-08-17T17:33:35.577 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00030971299999999998 2024-08-17T17:33:35.577 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.577 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.577 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:35.577 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794109+0000", 2024-08-17T17:33:35.577 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0568000000000003e-05 2024-08-17T17:33:35.577 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.577 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.577 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:35.577 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794172+0000", 2024-08-17T17:33:35.577 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2927000000000002e-05 2024-08-17T17:33:35.577 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.577 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:35.577 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.578 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.578 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.578 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82640 6.d 6:b46cd03d:::benchmark_data_smithi003_175043_object82639:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:33:35.578 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.177833+0000", 2024-08-17T17:33:35.578 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 476.38337995900002, 2024-08-17T17:33:35.578 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6164523790000001, 2024-08-17T17:33:35.578 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:35.578 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:35.578 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:35.578 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:35.578 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:35.578 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82640 2024-08-17T17:33:35.578 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.579 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:35.579 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.579 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:35.579 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.177833+0000", 2024-08-17T17:33:35.579 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.579 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.579 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.579 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:35.579 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.177833+0000", 2024-08-17T17:33:35.579 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.579 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.579 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.579 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:35.579 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.177831+0000", 2024-08-17T17:33:35.580 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:33:35.580 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.580 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.580 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:35.580 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.177866+0000", 2024-08-17T17:33:35.580 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5014000000000001e-05 2024-08-17T17:33:35.580 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.580 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.580 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:35.580 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.177883+0000", 2024-08-17T17:33:35.580 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6385999999999999e-05 2024-08-17T17:33:35.580 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.580 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.580 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:35.581 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.177891+0000", 2024-08-17T17:33:35.581 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.2199999999999992e-06 2024-08-17T17:33:35.581 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.581 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.581 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.581 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.177968+0000", 2024-08-17T17:33:35.581 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.6953999999999995e-05 2024-08-17T17:33:35.581 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.581 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.581 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:35.581 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.177977+0000", 2024-08-17T17:33:35.581 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.4830000000000005e-06 2024-08-17T17:33:35.581 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.581 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.582 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.582 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.777368+0000", 2024-08-17T17:33:35.582 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.599390358 2024-08-17T17:33:35.582 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.582 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.582 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:35.582 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.777492+0000", 2024-08-17T17:33:35.582 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012466300000000001 2024-08-17T17:33:35.582 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.582 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.582 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:33:35.582 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.777652+0000", 2024-08-17T17:33:35.582 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015963500000000001 2024-08-17T17:33:35.582 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.583 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.583 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:35.583 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.785741+0000", 2024-08-17T17:33:35.583 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0080892350000000002 2024-08-17T17:33:35.583 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.583 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.583 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.583 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.793812+0000", 2024-08-17T17:33:35.583 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0080707850000000005 2024-08-17T17:33:35.583 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.583 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.583 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.583 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794202+0000", 2024-08-17T17:33:35.584 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00038980599999999997 2024-08-17T17:33:35.584 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.584 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.584 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:35.584 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794229+0000", 2024-08-17T17:33:35.584 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7572999999999998e-05 2024-08-17T17:33:35.584 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.584 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.584 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:35.584 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794286+0000", 2024-08-17T17:33:35.584 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6013000000000001e-05 2024-08-17T17:33:35.584 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.584 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:35.584 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.585 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.585 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.585 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82642 6.d 6:b3782226:::benchmark_data_smithi003_175043_object82641:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:33:35.585 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.178629+0000", 2024-08-17T17:33:35.585 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 476.38258373899998, 2024-08-17T17:33:35.585 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.615915269, 2024-08-17T17:33:35.585 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:35.585 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:35.585 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:35.585 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:35.585 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:35.585 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82642 2024-08-17T17:33:35.585 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.585 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:35.586 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.586 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:35.586 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.178629+0000", 2024-08-17T17:33:35.586 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.586 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.586 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.586 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:35.586 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.178629+0000", 2024-08-17T17:33:35.586 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.586 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.586 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.586 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:35.586 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.178628+0000", 2024-08-17T17:33:35.587 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:33:35.587 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.587 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.587 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:35.587 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.178663+0000", 2024-08-17T17:33:35.587 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4839999999999998e-05 2024-08-17T17:33:35.587 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.587 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.587 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:35.587 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.178679+0000", 2024-08-17T17:33:35.587 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6617e-05 2024-08-17T17:33:35.587 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.587 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.588 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:35.588 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.178686+0000", 2024-08-17T17:33:35.588 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.066e-06 2024-08-17T17:33:35.588 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.588 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.588 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.588 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.178737+0000", 2024-08-17T17:33:35.588 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1802999999999999e-05 2024-08-17T17:33:35.588 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.588 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.588 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:35.588 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.178746+0000", 2024-08-17T17:33:35.588 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.6240000000000008e-06 2024-08-17T17:33:35.589 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.589 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.589 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.589 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.778800+0000", 2024-08-17T17:33:35.589 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6000541150000001 2024-08-17T17:33:35.589 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.589 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.589 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:35.589 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.778870+0000", 2024-08-17T17:33:35.589 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9701000000000002e-05 2024-08-17T17:33:35.589 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.589 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.589 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:33:35.590 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.778965+0000", 2024-08-17T17:33:35.590 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.4919000000000004e-05 2024-08-17T17:33:35.590 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.590 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.590 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:35.590 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.785745+0000", 2024-08-17T17:33:35.590 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0067802690000000002 2024-08-17T17:33:35.590 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.590 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.590 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.590 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794312+0000", 2024-08-17T17:33:35.590 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0085670390000000003 2024-08-17T17:33:35.591 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.591 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.591 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.591 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794453+0000", 2024-08-17T17:33:35.591 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000140548 2024-08-17T17:33:35.591 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.591 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.591 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:35.591 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794479+0000", 2024-08-17T17:33:35.591 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6236000000000001e-05 2024-08-17T17:33:35.591 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.591 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.591 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:35.592 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794545+0000", 2024-08-17T17:33:35.592 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5801000000000002e-05 2024-08-17T17:33:35.592 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.592 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:35.592 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.592 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.592 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.592 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82693 6.d 6:b633a1e2:::benchmark_data_smithi003_175043_object82692:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:33:35.592 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.226745+0000", 2024-08-17T17:33:35.592 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 476.33446812599999, 2024-08-17T17:33:35.592 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5679927600000001, 2024-08-17T17:33:35.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:35.594 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:35.594 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:35.594 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:35.594 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:35.594 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:35.595 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82693 2024-08-17T17:33:35.595 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.595 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:35.595 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.595 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:35.595 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.226745+0000", 2024-08-17T17:33:35.595 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.595 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.595 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.596 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:35.596 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.226745+0000", 2024-08-17T17:33:35.596 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.596 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.596 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.596 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:35.596 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.226743+0000", 2024-08-17T17:33:35.597 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:33:35.597 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.597 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.597 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:35.597 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.226800+0000", 2024-08-17T17:33:35.597 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6427e-05 2024-08-17T17:33:35.597 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.597 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.597 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:35.597 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.226813+0000", 2024-08-17T17:33:35.597 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2911999999999999e-05 2024-08-17T17:33:35.597 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.598 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.598 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:35.598 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.226819+0000", 2024-08-17T17:33:35.598 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2219999999999999e-06 2024-08-17T17:33:35.598 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.598 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.598 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.598 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.226876+0000", 2024-08-17T17:33:35.598 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7105999999999998e-05 2024-08-17T17:33:35.598 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.598 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.598 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:35.599 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.226886+0000", 2024-08-17T17:33:35.599 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0468e-05 2024-08-17T17:33:35.599 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.599 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.599 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.599 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.779914+0000", 2024-08-17T17:33:35.599 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.55302731 2024-08-17T17:33:35.599 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.599 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.599 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:35.599 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.780041+0000", 2024-08-17T17:33:35.599 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000127298 2024-08-17T17:33:35.599 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.600 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.600 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:33:35.600 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.780202+0000", 2024-08-17T17:33:35.600 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016124100000000001 2024-08-17T17:33:35.600 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.600 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.601 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:35.601 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.791392+0000", 2024-08-17T17:33:35.601 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011189486 2024-08-17T17:33:35.601 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.601 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.601 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.602 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794335+0000", 2024-08-17T17:33:35.602 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002943219 2024-08-17T17:33:35.602 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.602 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.602 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.602 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794589+0000", 2024-08-17T17:33:35.602 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00025462199999999998 2024-08-17T17:33:35.602 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.602 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.602 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:35.602 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794641+0000", 2024-08-17T17:33:35.603 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1773e-05 2024-08-17T17:33:35.603 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.603 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.603 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:35.603 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794738+0000", 2024-08-17T17:33:35.603 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6477000000000005e-05 2024-08-17T17:33:35.603 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.603 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:35.603 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.603 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.603 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.603 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82697 6.d 6:b974a054:::benchmark_data_smithi003_175043_object82696:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:33:35.604 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.231888+0000", 2024-08-17T17:33:35.604 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 476.32932522099998, 2024-08-17T17:33:35.604 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.563409968, 2024-08-17T17:33:35.604 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:35.604 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:35.604 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:35.604 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:35.604 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:35.604 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82697 2024-08-17T17:33:35.604 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.604 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:35.604 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.604 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:35.604 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.231888+0000", 2024-08-17T17:33:35.605 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.605 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.605 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.605 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:35.605 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.231888+0000", 2024-08-17T17:33:35.605 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.605 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.605 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.605 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:35.605 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.231887+0000", 2024-08-17T17:33:35.605 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:33:35.605 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.605 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.606 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:35.606 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.231937+0000", 2024-08-17T17:33:35.606 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0117000000000001e-05 2024-08-17T17:33:35.606 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.606 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.606 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:35.607 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.231949+0000", 2024-08-17T17:33:35.607 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2431e-05 2024-08-17T17:33:35.607 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.607 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.607 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:35.607 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.231954+0000", 2024-08-17T17:33:35.608 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7029999999999997e-06 2024-08-17T17:33:35.608 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.608 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.608 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.608 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.232014+0000", 2024-08-17T17:33:35.609 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0628999999999998e-05 2024-08-17T17:33:35.609 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.609 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.609 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:35.609 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.232025+0000", 2024-08-17T17:33:35.609 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.062e-05 2024-08-17T17:33:35.609 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.609 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.609 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.609 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.781376+0000", 2024-08-17T17:33:35.610 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.549351269 2024-08-17T17:33:35.610 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.610 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.610 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:35.610 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.781451+0000", 2024-08-17T17:33:35.610 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.4862000000000003e-05 2024-08-17T17:33:35.610 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.610 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.611 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:33:35.611 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.781534+0000", 2024-08-17T17:33:35.611 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.2558999999999993e-05 2024-08-17T17:33:35.611 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.611 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.611 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:35.611 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.791395+0000", 2024-08-17T17:33:35.611 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0098611889999999994 2024-08-17T17:33:35.611 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.611 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.611 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.611 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794362+0000", 2024-08-17T17:33:35.611 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0029670030000000002 2024-08-17T17:33:35.611 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.612 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.612 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.612 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.795163+0000", 2024-08-17T17:33:35.612 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00080066100000000004 2024-08-17T17:33:35.612 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.612 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.612 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:35.612 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.795204+0000", 2024-08-17T17:33:35.612 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1603999999999997e-05 2024-08-17T17:33:35.612 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.612 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.612 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:35.612 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.795298+0000", 2024-08-17T17:33:35.612 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.3583000000000002e-05 2024-08-17T17:33:35.613 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.613 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:35.613 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.613 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.613 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.613 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82701 6.d 6:bdcfc51a:::benchmark_data_smithi003_175043_object82700:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:33:35.613 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.234530+0000", 2024-08-17T17:33:35.613 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 476.326682592, 2024-08-17T17:33:35.613 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5609469009999999, 2024-08-17T17:33:35.613 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:35.613 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:35.613 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:35.614 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:35.614 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:35.614 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82701 2024-08-17T17:33:35.614 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.614 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:35.614 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.614 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:35.614 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.234530+0000", 2024-08-17T17:33:35.614 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.614 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.614 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.614 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:35.614 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.234530+0000", 2024-08-17T17:33:35.615 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.615 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.615 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.615 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:35.615 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.234529+0000", 2024-08-17T17:33:35.615 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:33:35.615 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.615 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.615 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:35.615 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.234620+0000", 2024-08-17T17:33:35.615 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1897999999999993e-05 2024-08-17T17:33:35.615 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.615 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.615 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:35.616 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.234636+0000", 2024-08-17T17:33:35.616 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5603999999999998e-05 2024-08-17T17:33:35.616 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.616 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.616 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:35.616 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.234642+0000", 2024-08-17T17:33:35.616 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7060000000000001e-06 2024-08-17T17:33:35.616 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.616 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.616 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.616 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.234711+0000", 2024-08-17T17:33:35.616 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9542999999999995e-05 2024-08-17T17:33:35.616 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.617 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.617 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:35.617 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.234722+0000", 2024-08-17T17:33:35.617 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0593e-05 2024-08-17T17:33:35.617 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.617 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.617 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.617 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.782419+0000", 2024-08-17T17:33:35.617 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.547697522 2024-08-17T17:33:35.617 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.617 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.617 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:35.618 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.782488+0000", 2024-08-17T17:33:35.618 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.8922000000000002e-05 2024-08-17T17:33:35.618 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.618 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.618 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:33:35.618 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.782573+0000", 2024-08-17T17:33:35.618 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.4179999999999997e-05 2024-08-17T17:33:35.618 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.618 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.618 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:35.618 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.791405+0000", 2024-08-17T17:33:35.618 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0088323220000000001 2024-08-17T17:33:35.618 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.619 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.619 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.619 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794389+0000", 2024-08-17T17:33:35.619 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0029838109999999998 2024-08-17T17:33:35.619 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.619 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.619 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.620 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.795355+0000", 2024-08-17T17:33:35.620 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00096604799999999995 2024-08-17T17:33:35.620 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.620 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.620 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:35.620 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.795398+0000", 2024-08-17T17:33:35.621 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3146999999999999e-05 2024-08-17T17:33:35.621 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.621 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.621 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:35.621 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.795477+0000", 2024-08-17T17:33:35.621 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.9569999999999999e-05 2024-08-17T17:33:35.622 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.622 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:35.622 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.622 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.622 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.622 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82722 6.d 6:b3e932d8:::benchmark_data_smithi003_175043_object82721:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:33:35.623 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.264293+0000", 2024-08-17T17:33:35.623 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 476.29692022299997, 2024-08-17T17:33:35.623 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.533356661, 2024-08-17T17:33:35.623 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:35.623 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:35.623 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:35.623 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:35.624 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:35.624 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82722 2024-08-17T17:33:35.624 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.624 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:35.624 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.624 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:35.624 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.264293+0000", 2024-08-17T17:33:35.625 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.625 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.625 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.625 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:35.625 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.264293+0000", 2024-08-17T17:33:35.625 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.625 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.626 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.626 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:35.626 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.264291+0000", 2024-08-17T17:33:35.626 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:33:35.626 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.626 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.627 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:35.627 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.264329+0000", 2024-08-17T17:33:35.627 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.7656e-05 2024-08-17T17:33:35.627 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.627 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.627 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:35.627 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.264344+0000", 2024-08-17T17:33:35.628 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5147000000000001e-05 2024-08-17T17:33:35.628 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.628 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.628 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:35.628 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.264349+0000", 2024-08-17T17:33:35.628 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.746e-06 2024-08-17T17:33:35.628 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.629 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.629 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.629 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.264416+0000", 2024-08-17T17:33:35.629 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.7662000000000004e-05 2024-08-17T17:33:35.629 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.629 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.629 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:35.629 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.264423+0000", 2024-08-17T17:33:35.630 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3450000000000002e-06 2024-08-17T17:33:35.630 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.630 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.630 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.630 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.783524+0000", 2024-08-17T17:33:35.630 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5191010029999998 2024-08-17T17:33:35.630 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.631 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.631 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:35.631 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.783598+0000", 2024-08-17T17:33:35.631 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3861000000000003e-05 2024-08-17T17:33:35.631 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.631 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.632 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:33:35.632 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.783677+0000", 2024-08-17T17:33:35.632 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.9592000000000006e-05 2024-08-17T17:33:35.632 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.632 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.632 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:35.633 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.791409+0000", 2024-08-17T17:33:35.633 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.007731435 2024-08-17T17:33:35.633 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.633 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.633 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.633 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794410+0000", 2024-08-17T17:33:35.634 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003000955 2024-08-17T17:33:35.634 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.634 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.634 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.635 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.797490+0000", 2024-08-17T17:33:35.635 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0030798610000000001 2024-08-17T17:33:35.635 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.635 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.635 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:35.636 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.797545+0000", 2024-08-17T17:33:35.636 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5507999999999998e-05 2024-08-17T17:33:35.636 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.636 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.636 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:35.636 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.797650+0000", 2024-08-17T17:33:35.637 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000104498 2024-08-17T17:33:35.637 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.637 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:35.637 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.637 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.638 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.638 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82729 6.d 6:b1d8abf5:::benchmark_data_smithi003_175043_object82728:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:33:35.638 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.271858+0000", 2024-08-17T17:33:35.638 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 476.28935462499999, 2024-08-17T17:33:35.638 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.525969804, 2024-08-17T17:33:35.639 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:35.639 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:35.639 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:35.639 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:35.639 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:35.639 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82729 2024-08-17T17:33:35.639 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.639 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:35.639 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.640 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:35.640 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.271858+0000", 2024-08-17T17:33:35.640 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.640 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.640 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.640 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:35.640 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.271858+0000", 2024-08-17T17:33:35.640 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.641 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.641 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.641 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:35.641 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.271857+0000", 2024-08-17T17:33:35.641 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:33:35.641 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.642 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.642 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:35.642 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.271917+0000", 2024-08-17T17:33:35.642 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.003e-05 2024-08-17T17:33:35.642 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.642 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.643 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:35.643 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.271933+0000", 2024-08-17T17:33:35.643 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6038000000000001e-05 2024-08-17T17:33:35.643 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.643 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.643 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:35.644 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.271939+0000", 2024-08-17T17:33:35.644 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4570000000000002e-06 2024-08-17T17:33:35.644 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.644 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.644 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.644 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.272016+0000", 2024-08-17T17:33:35.645 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7077999999999996e-05 2024-08-17T17:33:35.645 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.645 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.645 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:35.645 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.272027+0000", 2024-08-17T17:33:35.645 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0865e-05 2024-08-17T17:33:35.645 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.646 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.646 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.646 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.784655+0000", 2024-08-17T17:33:35.646 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5126276320000001 2024-08-17T17:33:35.647 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.647 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.647 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:35.647 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.784747+0000", 2024-08-17T17:33:35.647 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1725999999999996e-05 2024-08-17T17:33:35.647 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.647 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.647 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:33:35.648 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.784837+0000", 2024-08-17T17:33:35.648 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9955000000000002e-05 2024-08-17T17:33:35.648 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.648 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.648 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:35.648 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.793626+0000", 2024-08-17T17:33:35.648 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0087892579999999994 2024-08-17T17:33:35.648 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.648 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.648 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.648 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.794430+0000", 2024-08-17T17:33:35.648 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00080423300000000001 2024-08-17T17:33:35.648 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.648 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.649 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.649 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.797697+0000", 2024-08-17T17:33:35.649 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0032663890000000002 2024-08-17T17:33:35.649 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.649 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.649 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:35.649 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.797741+0000", 2024-08-17T17:33:35.649 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4150999999999998e-05 2024-08-17T17:33:35.650 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.650 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.650 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:35.650 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.797828+0000", 2024-08-17T17:33:35.650 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.7589000000000005e-05 2024-08-17T17:33:35.650 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.650 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:35.650 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.650 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.650 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.650 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82733 6.d 6:b4deca26:::benchmark_data_smithi003_175043_object82732:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:33:35.650 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.278865+0000", 2024-08-17T17:33:35.651 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 476.282348324, 2024-08-17T17:33:35.651 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.525599463, 2024-08-17T17:33:35.651 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:35.651 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:35.651 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:35.651 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:35.651 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:35.651 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82733 2024-08-17T17:33:35.651 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.651 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:35.651 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.651 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:35.651 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.278865+0000", 2024-08-17T17:33:35.651 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.652 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.652 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.652 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:35.652 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.278865+0000", 2024-08-17T17:33:35.652 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.652 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.652 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.652 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:35.652 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.278863+0000", 2024-08-17T17:33:35.652 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:33:35.652 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.652 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.652 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:35.652 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.278920+0000", 2024-08-17T17:33:35.652 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6860000000000001e-05 2024-08-17T17:33:35.653 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.653 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.653 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:35.653 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.278933+0000", 2024-08-17T17:33:35.653 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3052e-05 2024-08-17T17:33:35.653 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.653 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.653 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:35.653 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.278938+0000", 2024-08-17T17:33:35.653 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9280000000000001e-06 2024-08-17T17:33:35.653 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.653 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.653 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.653 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.278995+0000", 2024-08-17T17:33:35.654 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7260999999999999e-05 2024-08-17T17:33:35.654 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.654 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.654 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:35.654 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.279003+0000", 2024-08-17T17:33:35.654 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1810000000000001e-06 2024-08-17T17:33:35.654 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.654 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.654 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.654 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.785779+0000", 2024-08-17T17:33:35.654 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5067753750000001 2024-08-17T17:33:35.654 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.654 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.654 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:35.655 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.785862+0000", 2024-08-17T17:33:35.655 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.3327999999999999e-05 2024-08-17T17:33:35.655 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.655 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.655 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:33:35.655 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.785964+0000", 2024-08-17T17:33:35.655 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000101952 2024-08-17T17:33:35.655 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.655 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.655 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:35.655 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.793630+0000", 2024-08-17T17:33:35.655 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0076662450000000004 2024-08-17T17:33:35.655 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.655 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.656 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.656 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.802127+0000", 2024-08-17T17:33:35.656 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0084964489999999997 2024-08-17T17:33:35.656 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.656 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.656 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.656 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.804270+0000", 2024-08-17T17:33:35.656 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0021435270000000001 2024-08-17T17:33:35.656 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.656 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.656 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:35.656 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.804336+0000", 2024-08-17T17:33:35.656 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5512999999999994e-05 2024-08-17T17:33:35.656 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.656 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.657 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:35.657 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:35.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:35.659 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.804464+0000", 2024-08-17T17:33:35.659 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012867299999999999 2024-08-17T17:33:35.659 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.660 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:35.660 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.660 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.660 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.660 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82738 6.d 6:b5082b22:::benchmark_data_smithi003_175043_object82737:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:33:35.660 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.285187+0000", 2024-08-17T17:33:35.661 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 476.276026409, 2024-08-17T17:33:35.661 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5194669009999999, 2024-08-17T17:33:35.661 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:35.661 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:35.661 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:35.661 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:35.661 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:35.661 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82738 2024-08-17T17:33:35.662 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.662 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:35.662 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.662 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:35.662 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.285187+0000", 2024-08-17T17:33:35.662 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.662 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.662 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.662 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:35.662 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.285187+0000", 2024-08-17T17:33:35.662 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.662 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.662 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.663 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:35.663 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.285185+0000", 2024-08-17T17:33:35.663 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:33:35.663 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.663 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.663 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:35.663 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.285242+0000", 2024-08-17T17:33:35.663 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7432999999999997e-05 2024-08-17T17:33:35.663 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.663 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.663 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:35.663 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.285256+0000", 2024-08-17T17:33:35.663 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3944000000000001e-05 2024-08-17T17:33:35.664 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.664 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.664 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:35.664 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.285262+0000", 2024-08-17T17:33:35.664 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.152e-06 2024-08-17T17:33:35.664 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.664 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.664 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.664 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.285305+0000", 2024-08-17T17:33:35.664 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3881999999999998e-05 2024-08-17T17:33:35.664 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.664 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.664 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:35.664 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.285313+0000", 2024-08-17T17:33:35.665 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.6320000000000008e-06 2024-08-17T17:33:35.665 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.665 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.665 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.665 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.786864+0000", 2024-08-17T17:33:35.665 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.501550613 2024-08-17T17:33:35.665 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.665 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.665 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:35.665 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.786943+0000", 2024-08-17T17:33:35.665 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.9118999999999999e-05 2024-08-17T17:33:35.665 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.665 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.665 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:33:35.666 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.787026+0000", 2024-08-17T17:33:35.666 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.3361000000000004e-05 2024-08-17T17:33:35.666 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.666 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.666 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:35.666 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.793634+0000", 2024-08-17T17:33:35.666 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0066076249999999998 2024-08-17T17:33:35.666 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.666 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.666 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.666 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.802156+0000", 2024-08-17T17:33:35.666 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0085225700000000001 2024-08-17T17:33:35.666 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.666 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.667 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.667 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.804510+0000", 2024-08-17T17:33:35.667 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0023532700000000002 2024-08-17T17:33:35.667 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.667 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.667 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:35.667 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.804545+0000", 2024-08-17T17:33:35.667 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5040999999999999e-05 2024-08-17T17:33:35.667 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.667 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.667 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:35.667 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.804654+0000", 2024-08-17T17:33:35.667 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000108884 2024-08-17T17:33:35.667 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.668 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:35.668 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.668 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.668 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.668 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82775 6.d 6:b9e7b196:::benchmark_data_smithi003_175043_object82774:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:33:35.668 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.335574+0000", 2024-08-17T17:33:35.668 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 476.22563877200002, 2024-08-17T17:33:35.668 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.469304108, 2024-08-17T17:33:35.668 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:35.668 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:35.668 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:35.668 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:35.668 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:35.668 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82775 2024-08-17T17:33:35.668 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.669 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:35.669 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.669 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:35.669 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.335574+0000", 2024-08-17T17:33:35.669 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.669 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.669 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.669 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:35.669 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.335574+0000", 2024-08-17T17:33:35.669 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.669 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.669 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.669 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:35.669 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.335573+0000", 2024-08-17T17:33:35.670 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:33:35.670 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.670 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.670 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:35.670 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.335620+0000", 2024-08-17T17:33:35.670 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7202e-05 2024-08-17T17:33:35.670 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.670 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.670 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:35.670 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.335632+0000", 2024-08-17T17:33:35.670 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1447999999999999e-05 2024-08-17T17:33:35.670 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.670 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.671 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:35.671 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.335636+0000", 2024-08-17T17:33:35.671 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.33e-06 2024-08-17T17:33:35.671 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.671 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.671 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.671 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.335695+0000", 2024-08-17T17:33:35.671 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8607000000000003e-05 2024-08-17T17:33:35.671 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.671 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.671 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:35.671 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.335707+0000", 2024-08-17T17:33:35.671 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2574e-05 2024-08-17T17:33:35.671 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.671 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.672 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.672 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.787914+0000", 2024-08-17T17:33:35.672 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.452207059 2024-08-17T17:33:35.672 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.672 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.672 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:35.672 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.787999+0000", 2024-08-17T17:33:35.672 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.4839999999999994e-05 2024-08-17T17:33:35.672 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.672 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.672 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:33:35.672 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.788083+0000", 2024-08-17T17:33:35.672 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.4035000000000004e-05 2024-08-17T17:33:35.672 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.673 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.673 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:35.673 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.793637+0000", 2024-08-17T17:33:35.673 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0055542530000000003 2024-08-17T17:33:35.673 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.673 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.673 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.673 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.803058+0000", 2024-08-17T17:33:35.673 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0094209629999999992 2024-08-17T17:33:35.673 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.673 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.673 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.673 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.804717+0000", 2024-08-17T17:33:35.673 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0016589689999999999 2024-08-17T17:33:35.674 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.674 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.674 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:35.674 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.804771+0000", 2024-08-17T17:33:35.674 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3931999999999998e-05 2024-08-17T17:33:35.674 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.674 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.674 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:35.674 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.804878+0000", 2024-08-17T17:33:35.674 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000107215 2024-08-17T17:33:35.674 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.674 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:35.674 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.675 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.675 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.675 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82786 6.d 6:b4168c45:::benchmark_data_smithi003_175043_object82785:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:33:35.675 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.351300+0000", 2024-08-17T17:33:35.675 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 476.20991292600002, 2024-08-17T17:33:35.675 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.454956208, 2024-08-17T17:33:35.675 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:35.675 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:35.675 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:35.675 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:35.675 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:35.675 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82786 2024-08-17T17:33:35.675 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.676 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:35.676 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.676 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:35.676 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.351300+0000", 2024-08-17T17:33:35.676 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.676 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.676 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.676 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:35.676 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.351300+0000", 2024-08-17T17:33:35.676 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.676 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.676 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.676 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:35.676 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.351298+0000", 2024-08-17T17:33:35.676 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:33:35.677 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.677 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.677 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:35.677 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.351330+0000", 2024-08-17T17:33:35.677 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1772e-05 2024-08-17T17:33:35.677 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.677 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.677 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:35.677 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.351347+0000", 2024-08-17T17:33:35.677 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6606e-05 2024-08-17T17:33:35.677 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.677 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.677 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:35.677 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.351353+0000", 2024-08-17T17:33:35.678 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4590000000000003e-06 2024-08-17T17:33:35.678 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.678 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.678 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.678 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.351412+0000", 2024-08-17T17:33:35.678 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8669999999999999e-05 2024-08-17T17:33:35.678 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.678 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.678 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:35.678 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.351421+0000", 2024-08-17T17:33:35.678 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.7309999999999999e-06 2024-08-17T17:33:35.678 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.678 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.679 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.679 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.788983+0000", 2024-08-17T17:33:35.679 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.437562008 2024-08-17T17:33:35.679 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.679 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.679 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:35.679 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.789094+0000", 2024-08-17T17:33:35.679 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0001115 2024-08-17T17:33:35.679 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.679 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.679 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:33:35.679 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.789196+0000", 2024-08-17T17:33:35.679 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000101718 2024-08-17T17:33:35.679 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.680 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.680 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:35.680 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.793641+0000", 2024-08-17T17:33:35.680 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0044450870000000003 2024-08-17T17:33:35.680 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.680 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.680 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.680 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.804993+0000", 2024-08-17T17:33:35.680 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011352258 2024-08-17T17:33:35.680 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.680 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.680 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.680 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.806095+0000", 2024-08-17T17:33:35.680 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0011017240000000001 2024-08-17T17:33:35.681 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.681 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.681 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:35.681 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.806146+0000", 2024-08-17T17:33:35.681 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1461000000000001e-05 2024-08-17T17:33:35.681 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.681 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.681 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:35.681 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.806256+0000", 2024-08-17T17:33:35.681 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000109967 2024-08-17T17:33:35.681 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.681 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:35.681 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.681 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.682 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.682 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82793 6.d 6:bca5da5d:::benchmark_data_smithi003_175043_object82792:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:33:35.682 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.361189+0000", 2024-08-17T17:33:35.682 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 476.200023606, 2024-08-17T17:33:35.682 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.446922399, 2024-08-17T17:33:35.682 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:35.682 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:35.682 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:35.682 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:35.682 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:35.682 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82793 2024-08-17T17:33:35.682 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.682 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:35.683 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.683 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:35.683 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.361189+0000", 2024-08-17T17:33:35.683 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.683 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.683 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.683 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:35.683 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.361189+0000", 2024-08-17T17:33:35.683 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.683 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.683 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.683 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:35.683 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.361188+0000", 2024-08-17T17:33:35.684 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:33:35.684 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.684 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.684 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:35.684 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.361225+0000", 2024-08-17T17:33:35.684 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.7224000000000001e-05 2024-08-17T17:33:35.684 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.684 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.684 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:35.684 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.361238+0000", 2024-08-17T17:33:35.684 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2315e-05 2024-08-17T17:33:35.684 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.684 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.684 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:35.685 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.361244+0000", 2024-08-17T17:33:35.685 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1040000000000003e-06 2024-08-17T17:33:35.685 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.685 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.685 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.685 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.361304+0000", 2024-08-17T17:33:35.685 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9828999999999999e-05 2024-08-17T17:33:35.685 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.685 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.685 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:35.685 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.361315+0000", 2024-08-17T17:33:35.685 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0871e-05 2024-08-17T17:33:35.685 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.685 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.686 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.686 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.790129+0000", 2024-08-17T17:33:35.686 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.428814585 2024-08-17T17:33:35.686 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.686 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.686 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:35.686 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.790209+0000", 2024-08-17T17:33:35.686 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0228999999999994e-05 2024-08-17T17:33:35.686 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.686 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.686 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:33:35.686 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.790308+0000", 2024-08-17T17:33:35.686 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.9118999999999997e-05 2024-08-17T17:33:35.687 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.687 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.687 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:35.687 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.793774+0000", 2024-08-17T17:33:35.687 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003465138 2024-08-17T17:33:35.687 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.687 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.687 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.687 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.805044+0000", 2024-08-17T17:33:35.687 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011270820000000001 2024-08-17T17:33:35.687 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.687 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.687 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.687 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.807955+0000", 2024-08-17T17:33:35.688 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0029106829999999998 2024-08-17T17:33:35.688 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.688 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.688 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:35.688 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.808009+0000", 2024-08-17T17:33:35.688 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3912999999999997e-05 2024-08-17T17:33:35.688 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.688 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.688 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:35.688 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.808112+0000", 2024-08-17T17:33:35.688 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000102864 2024-08-17T17:33:35.688 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.688 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:35.689 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.689 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.689 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.689 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82802 6.d 6:b0f72434:::benchmark_data_smithi003_175043_object82801:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:33:35.689 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.380777+0000", 2024-08-17T17:33:35.689 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 476.18043633799999, 2024-08-17T17:33:35.689 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.42952853, 2024-08-17T17:33:35.689 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:35.689 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:35.689 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:35.689 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:35.689 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:35.689 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82802 2024-08-17T17:33:35.689 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.690 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:35.690 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.690 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:35.690 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.380777+0000", 2024-08-17T17:33:35.690 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.690 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.690 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.690 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:35.690 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.380777+0000", 2024-08-17T17:33:35.690 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.690 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.690 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.690 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:35.690 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.380775+0000", 2024-08-17T17:33:35.691 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:33:35.691 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.691 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.691 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:35.691 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.380803+0000", 2024-08-17T17:33:35.691 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7353999999999999e-05 2024-08-17T17:33:35.691 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.691 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.691 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:35.691 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.380817+0000", 2024-08-17T17:33:35.691 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3946e-05 2024-08-17T17:33:35.691 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.691 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.691 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:35.692 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.380821+0000", 2024-08-17T17:33:35.692 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9490000000000004e-06 2024-08-17T17:33:35.692 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.692 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.692 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.692 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.380880+0000", 2024-08-17T17:33:35.692 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8761000000000002e-05 2024-08-17T17:33:35.692 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.692 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.692 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:35.692 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.380891+0000", 2024-08-17T17:33:35.692 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0876999999999999e-05 2024-08-17T17:33:35.692 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.692 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.693 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.693 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.791428+0000", 2024-08-17T17:33:35.693 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4105370399999999 2024-08-17T17:33:35.693 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.693 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.693 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:35.693 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.791499+0000", 2024-08-17T17:33:35.693 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.1247000000000004e-05 2024-08-17T17:33:35.693 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.693 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.694 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:33:35.694 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.791582+0000", 2024-08-17T17:33:35.694 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.2639000000000006e-05 2024-08-17T17:33:35.694 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.694 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.694 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:35.694 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.793778+0000", 2024-08-17T17:33:35.694 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0021961469999999999 2024-08-17T17:33:35.694 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.695 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.695 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.695 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.805082+0000", 2024-08-17T17:33:35.695 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011303863000000001 2024-08-17T17:33:35.695 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.695 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.695 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.695 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.810149+0000", 2024-08-17T17:33:35.695 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0050670180000000004 2024-08-17T17:33:35.695 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.695 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.695 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:35.695 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.810200+0000", 2024-08-17T17:33:35.695 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1430000000000001e-05 2024-08-17T17:33:35.696 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.696 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.696 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:35.696 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.810305+0000", 2024-08-17T17:33:35.696 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000104789 2024-08-17T17:33:35.696 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.696 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:35.696 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.696 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.696 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.696 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:82805 6.d 6:b91bb233:::benchmark_data_smithi003_175043_object82804:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-08-17T17:33:35.696 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:39.389839+0000", 2024-08-17T17:33:35.696 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 476.171373733, 2024-08-17T17:33:35.696 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4206285620000001, 2024-08-17T17:33:35.697 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:35.697 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:35.697 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:35.697 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:35.697 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:35.697 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 82805 2024-08-17T17:33:35.697 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.697 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:35.697 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.697 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:35.697 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.389839+0000", 2024-08-17T17:33:35.697 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.697 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.697 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.698 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:35.698 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.389839+0000", 2024-08-17T17:33:35.698 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.698 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.698 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.698 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:35.698 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.389838+0000", 2024-08-17T17:33:35.698 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:33:35.698 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.698 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.698 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:35.698 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.389863+0000", 2024-08-17T17:33:35.698 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4992000000000001e-05 2024-08-17T17:33:35.698 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.699 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.699 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:35.699 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.389877+0000", 2024-08-17T17:33:35.699 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3879999999999999e-05 2024-08-17T17:33:35.699 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.699 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.699 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:35.699 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.389882+0000", 2024-08-17T17:33:35.699 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1399999999999999e-06 2024-08-17T17:33:35.699 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.699 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.699 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.699 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.389935+0000", 2024-08-17T17:33:35.699 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2324000000000003e-05 2024-08-17T17:33:35.699 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.700 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.700 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:35.700 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:39.389941+0000", 2024-08-17T17:33:35.700 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5259999999999999e-06 2024-08-17T17:33:35.700 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.700 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.700 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.700 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.792504+0000", 2024-08-17T17:33:35.700 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4025628750000001 2024-08-17T17:33:35.700 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.700 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.700 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:35.700 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.792583+0000", 2024-08-17T17:33:35.700 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.9419999999999995e-05 2024-08-17T17:33:35.701 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.701 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.701 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:33:35.701 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.792679+0000", 2024-08-17T17:33:35.701 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5731999999999998e-05 2024-08-17T17:33:35.701 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.701 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.701 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:35.701 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.793981+0000", 2024-08-17T17:33:35.701 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001302213 2024-08-17T17:33:35.701 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.701 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.701 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.701 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.805220+0000", 2024-08-17T17:33:35.701 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011238685 2024-08-17T17:33:35.702 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.702 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.702 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.702 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.810345+0000", 2024-08-17T17:33:35.702 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0051250130000000003 2024-08-17T17:33:35.702 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.702 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.702 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:35.702 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.810387+0000", 2024-08-17T17:33:35.702 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2119000000000001e-05 2024-08-17T17:33:35.702 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.702 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.702 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:35.703 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:40.810468+0000", 2024-08-17T17:33:35.703 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0734000000000003e-05 2024-08-17T17:33:35.703 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.703 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:35.703 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.703 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.703 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.703 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:191527 6.d 6:b1fdfb8f:::benchmark_data_smithi003_175043_object92878:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-08-17T17:33:35.703 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:28:00.574230+0000", 2024-08-17T17:33:35.703 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 334.98698289800001, 2024-08-17T17:33:35.703 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.26055499700000001, 2024-08-17T17:33:35.703 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:35.703 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:35.703 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:35.704 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:35.704 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:35.704 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 191527 2024-08-17T17:33:35.704 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.704 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:35.704 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.704 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:35.704 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.574230+0000", 2024-08-17T17:33:35.704 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.704 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.704 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.704 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:35.704 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.574230+0000", 2024-08-17T17:33:35.704 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.705 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.705 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.705 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:35.705 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.574229+0000", 2024-08-17T17:33:35.705 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:33:35.705 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.705 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.705 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:35.705 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.574239+0000", 2024-08-17T17:33:35.705 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0087e-05 2024-08-17T17:33:35.705 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.705 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.705 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:35.705 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.574255+0000", 2024-08-17T17:33:35.706 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6018000000000001e-05 2024-08-17T17:33:35.706 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.706 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.706 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:35.706 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.574260+0000", 2024-08-17T17:33:35.706 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2619999999999998e-06 2024-08-17T17:33:35.706 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.706 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.706 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.706 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.574310+0000", 2024-08-17T17:33:35.706 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0043999999999997e-05 2024-08-17T17:33:35.706 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.706 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.706 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:35.707 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.574429+0000", 2024-08-17T17:33:35.707 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011841300000000001 2024-08-17T17:33:35.707 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.707 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.707 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:33:35.707 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.574544+0000", 2024-08-17T17:33:35.707 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011545 2024-08-17T17:33:35.707 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.707 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.707 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:35.707 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.579314+0000", 2024-08-17T17:33:35.707 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0047697069999999998 2024-08-17T17:33:35.707 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.707 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.708 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.708 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.580033+0000", 2024-08-17T17:33:35.708 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00071897599999999997 2024-08-17T17:33:35.708 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.708 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.708 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.708 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.834672+0000", 2024-08-17T17:33:35.708 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.25463945399999999 2024-08-17T17:33:35.708 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.708 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.708 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:35.708 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.834749+0000", 2024-08-17T17:33:35.708 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.6643999999999993e-05 2024-08-17T17:33:35.708 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.709 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.709 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:35.709 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.834785+0000", 2024-08-17T17:33:35.709 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6307999999999999e-05 2024-08-17T17:33:35.709 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.709 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:35.709 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.709 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.709 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.709 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:191540 6.d 6:bcc10983:::benchmark_data_smithi003_175043_object92891:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-08-17T17:33:35.709 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:28:00.590757+0000", 2024-08-17T17:33:35.709 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 334.97045618999999, 2024-08-17T17:33:35.709 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.25455507300000002, 2024-08-17T17:33:35.709 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:35.710 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:35.710 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:35.710 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:35.710 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:35.710 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 191540 2024-08-17T17:33:35.710 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.710 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:35.710 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.710 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:35.710 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.590757+0000", 2024-08-17T17:33:35.710 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.710 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.710 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.711 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:35.711 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.590757+0000", 2024-08-17T17:33:35.711 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.711 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.711 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.711 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:35.711 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.590756+0000", 2024-08-17T17:33:35.711 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:33:35.711 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.711 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.711 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:35.711 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.590760+0000", 2024-08-17T17:33:35.711 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.8010000000000003e-06 2024-08-17T17:33:35.711 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.712 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.712 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:35.712 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.590770+0000", 2024-08-17T17:33:35.712 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.825e-06 2024-08-17T17:33:35.712 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.712 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.712 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:35.712 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.590774+0000", 2024-08-17T17:33:35.712 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.002e-06 2024-08-17T17:33:35.712 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.712 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.712 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.712 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.590814+0000", 2024-08-17T17:33:35.712 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9489000000000001e-05 2024-08-17T17:33:35.712 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.713 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.713 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:35.713 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.590898+0000", 2024-08-17T17:33:35.713 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.4530999999999994e-05 2024-08-17T17:33:35.713 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.713 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.713 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:33:35.713 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.590976+0000", 2024-08-17T17:33:35.713 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7720999999999997e-05 2024-08-17T17:33:35.713 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.713 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.713 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:35.713 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.599290+0000", 2024-08-17T17:33:35.713 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0083138450000000003 2024-08-17T17:33:35.714 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.714 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.714 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.714 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.605077+0000", 2024-08-17T17:33:35.714 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0057869009999999997 2024-08-17T17:33:35.714 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.714 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.714 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.714 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.845239+0000", 2024-08-17T17:33:35.714 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.24016241699999999 2024-08-17T17:33:35.714 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.714 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.714 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:35.714 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.845274+0000", 2024-08-17T17:33:35.715 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5181000000000001e-05 2024-08-17T17:33:35.715 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.715 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.715 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:35.715 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.845312+0000", 2024-08-17T17:33:35.715 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.7567999999999997e-05 2024-08-17T17:33:35.715 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.715 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:35.715 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.715 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.715 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.715 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:191543 6.d 6:b38b2e1c:::benchmark_data_smithi003_175043_object92894:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-08-17T17:33:35.715 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:28:00.592200+0000", 2024-08-17T17:33:35.715 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 334.969012695, 2024-08-17T17:33:35.716 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.25325235299999999, 2024-08-17T17:33:35.716 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:35.716 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:35.716 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:35.716 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:35.716 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:35.716 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 191543 2024-08-17T17:33:35.716 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.716 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:35.716 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.716 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:35.716 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.592200+0000", 2024-08-17T17:33:35.716 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.716 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.717 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.717 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:35.717 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.592200+0000", 2024-08-17T17:33:35.717 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.717 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.717 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.717 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:35.717 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.592199+0000", 2024-08-17T17:33:35.717 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:33:35.717 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.717 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.717 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:35.717 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.592205+0000", 2024-08-17T17:33:35.717 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3879999999999999e-06 2024-08-17T17:33:35.718 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.718 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.718 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:35.718 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.592218+0000", 2024-08-17T17:33:35.718 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3392e-05 2024-08-17T17:33:35.718 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.718 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.718 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:35.718 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.592223+0000", 2024-08-17T17:33:35.718 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5609999999999998e-06 2024-08-17T17:33:35.718 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.718 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.718 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.718 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.594600+0000", 2024-08-17T17:33:35.719 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0023767749999999998 2024-08-17T17:33:35.719 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.719 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.719 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:35.719 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.594722+0000", 2024-08-17T17:33:35.719 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000121311 2024-08-17T17:33:35.719 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.719 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.719 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,1", 2024-08-17T17:33:35.719 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.594837+0000", 2024-08-17T17:33:35.719 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011552099999999999 2024-08-17T17:33:35.719 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.719 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.719 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:35.720 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.599437+0000", 2024-08-17T17:33:35.720 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0045999839999999997 2024-08-17T17:33:35.720 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.720 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.720 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.720 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.605113+0000", 2024-08-17T17:33:35.720 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0056763769999999998 2024-08-17T17:33:35.720 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.720 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.720 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.720 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.845357+0000", 2024-08-17T17:33:35.720 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.24024379800000001 2024-08-17T17:33:35.720 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.721 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.721 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:35.721 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.845408+0000", 2024-08-17T17:33:35.721 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1208000000000002e-05 2024-08-17T17:33:35.721 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.721 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.721 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:35.721 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:28:00.845453+0000", 2024-08-17T17:33:35.721 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4279000000000001e-05 2024-08-17T17:33:35.721 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.721 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:35.721 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.721 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.721 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.721 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:68830 8.d 8:b08b2ca6:::benchmark_data_smithi003_229103_object68829:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-08-17T17:33:35.722 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:35.005197+0000", 2024-08-17T17:33:35.722 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.556016277, 2024-08-17T17:33:35.722 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.21011802199999999, 2024-08-17T17:33:35.722 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:35.722 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:35.722 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:35.722 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:33:35.722 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:33:35.722 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 68830 2024-08-17T17:33:35.722 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.722 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:35.722 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.722 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:35.723 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:35.005197+0000", 2024-08-17T17:33:35.723 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.723 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.723 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.723 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:35.723 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:35.005197+0000", 2024-08-17T17:33:35.723 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:35.723 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.723 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.723 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:35.723 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:35.005196+0000", 2024-08-17T17:33:35.723 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:33:35.723 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.723 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.723 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:35.724 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:35.005225+0000", 2024-08-17T17:33:35.724 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9660000000000001e-05 2024-08-17T17:33:35.724 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.724 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.724 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:35.724 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:35.005237+0000", 2024-08-17T17:33:35.724 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1669e-05 2024-08-17T17:33:35.724 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.724 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.724 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:35.724 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:35.005243+0000", 2024-08-17T17:33:35.724 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7799999999999997e-06 2024-08-17T17:33:35.724 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.724 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.725 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:35.725 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:35.006974+0000", 2024-08-17T17:33:35.725 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001731212 2024-08-17T17:33:35.725 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.725 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.725 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:35.725 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:35.007092+0000", 2024-08-17T17:33:35.725 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000118214 2024-08-17T17:33:35.725 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.725 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.725 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:33:35.725 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:35.007224+0000", 2024-08-17T17:33:35.725 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013161999999999999 2024-08-17T17:33:35.725 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.726 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.726 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:35.726 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:35.009928+0000", 2024-08-17T17:33:35.726 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0027040660000000002 2024-08-17T17:33:35.726 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.726 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.726 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.726 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:35.012584+0000", 2024-08-17T17:33:35.726 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002656149 2024-08-17T17:33:35.726 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.726 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.726 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:35.726 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:35.215235+0000", 2024-08-17T17:33:35.726 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.20265053999999999 2024-08-17T17:33:35.727 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.727 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.727 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:35.727 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:35.215280+0000", 2024-08-17T17:33:35.727 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4768999999999998e-05 2024-08-17T17:33:35.727 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:35.727 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:35.727 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:35.727 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:35.215315+0000", 2024-08-17T17:33:35.727 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5329e-05 2024-08-17T17:33:35.727 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.727 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:35.727 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.727 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:35.728 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:35.728 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:33:35.758 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:35.790 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:35.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:35.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:35.892 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:35.930 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:35.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:35.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:36.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:33:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:33:35] "GET /metrics HTTP/1.1" 200 36362 "" "Prometheus/2.51.0" 2024-08-17T17:33:36.032 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:36.057 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:36.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:36.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:36.159 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:36.189 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:36.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:36.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:36.291 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:36.327 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:36.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:36.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:36.369 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-08-17T17:33:36.369512+0000 min lat: 0.00109625 max lat: 2.31183 avg lat: 0.00914707 2024-08-17T17:33:36.369 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:33:36.369 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 40 16 69922 69906 109.208 57.25 0.0200517 0.00914707 2024-08-17T17:33:36.429 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:36.445 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-17T17:33:36.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:36.463 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:36.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:36.565 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:36.586 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:36.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:36.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:36.689 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:36.725 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:36.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:36.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:36.815 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:33:36.826 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:36.877 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:36.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:36.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:36.879 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-08-17T17:33:36.879 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:33:36.880 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-08-17T17:33:36.880 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.000000 2024-08-17T17:33:36.880 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 5 0 2024-08-17T17:33:36.979 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:37.031 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:37.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:37.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:37.133 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:37.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:37.184 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:37.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:37.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:37 smithi053 ceph-mon[89963]: pgmap v813: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 57 MiB/s wr, 918 op/s; 24731/203292 objects degraded (12.165%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:33:37.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:37 smithi003 ceph-mon[100615]: pgmap v813: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 57 MiB/s wr, 918 op/s; 24731/203292 objects degraded (12.165%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:33:37.239 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:37 smithi003 ceph-mon[103344]: pgmap v813: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 57 MiB/s wr, 918 op/s; 24731/203292 objects degraded (12.165%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:33:37.286 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:37.315 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:37.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:37.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:37.369 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 41 16 70765 70749 107.83 52.6875 0.00715653 0.00926814 2024-08-17T17:33:37.417 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:37.452 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:37.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:37.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:37.554 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:37.603 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:37.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:37.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:37.705 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:37.732 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:37.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:37.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:37.834 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:37.870 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:37.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:37.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:37.972 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:38.005 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:38.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:38.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:38.107 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:38.133 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:38.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:38.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:38.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:38.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:38.278 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:38.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:38.369 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 42 16 71605 71589 106.512 52.5 0.0080102 0.00938184 2024-08-17T17:33:38.379 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:38.417 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:38.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:38.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:38.519 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:38.546 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:38.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:38.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:38.581 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:33:38.581 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2024-08-17T17:33:38.581 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:38.581 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:71795 8.9 8:93fb3beb:::benchmark_data_smithi003_229103_object71794:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-08-17T17:33:38.582 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:38.567329+0000", 2024-08-17T17:33:38.582 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.012837678999999999, 2024-08-17T17:33:38.582 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012863741, 2024-08-17T17:33:38.582 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:38.582 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:33:38.582 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:38.582 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:33:38.582 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:33:38.582 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 71795 2024-08-17T17:33:38.582 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:38.582 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:38.583 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:38.583 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:38.583 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:38.567329+0000", 2024-08-17T17:33:38.583 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:38.583 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:38.583 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:38.583 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:38.583 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:38.567329+0000", 2024-08-17T17:33:38.583 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:38.583 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:38.583 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:38.583 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:38.583 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:38.567328+0000", 2024-08-17T17:33:38.584 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:33:38.584 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:38.584 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:38.584 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:38.584 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:38.567356+0000", 2024-08-17T17:33:38.584 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8251000000000001e-05 2024-08-17T17:33:38.584 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:38.584 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:38.584 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:38.584 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:38.567367+0000", 2024-08-17T17:33:38.584 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0597999999999999e-05 2024-08-17T17:33:38.584 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:38.584 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:38.585 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:38.585 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:38.567374+0000", 2024-08-17T17:33:38.585 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3640000000000001e-06 2024-08-17T17:33:38.585 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:38.585 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:38.585 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:38.585 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:38.569765+0000", 2024-08-17T17:33:38.585 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0023910340000000002 2024-08-17T17:33:38.585 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:38.585 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:38.585 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:38.585 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:38.569869+0000", 2024-08-17T17:33:38.585 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000103717 2024-08-17T17:33:38.585 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:38.586 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:38.586 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:33:38.586 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:38.570000+0000", 2024-08-17T17:33:38.586 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013134299999999999 2024-08-17T17:33:38.586 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:38.586 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:38.586 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:38.586 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:38.573517+0000", 2024-08-17T17:33:38.586 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003516384 2024-08-17T17:33:38.586 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:38.586 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:38.586 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:38.586 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:38.577116+0000", 2024-08-17T17:33:38.587 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0035996090000000001 2024-08-17T17:33:38.587 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:38.587 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:38.587 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:38.587 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:38.587 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-08-17T17:33:38.587 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 1 2024-08-17T17:33:38.587 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:33:38.648 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:33:38.672 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:38.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:38.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:33:38.773 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:38.799 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:38.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:38.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:38.900 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:38.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:38.927 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:38.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:38.928 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:33:38.944 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:33:39.028 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:39.091 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:39.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:39.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:39.193 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:39.221 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:39.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:39.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:39.322 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:33:39.360 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:39.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:39.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:33:39.369 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:39 smithi003 ceph-mon[100615]: pgmap v814: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 56 MiB/s wr, 902 op/s; 24603/206256 objects degraded (11.928%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:33:39.369 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:33:39.369 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:33:39.370 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:39 smithi003 ceph-mon[103344]: pgmap v814: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 56 MiB/s wr, 902 op/s; 24603/206256 objects degraded (11.928%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:33:39.370 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:33:39.370 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:33:39.371 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 43 16 72448 72432 105.259 52.6875 0.00783352 0.00949348 2024-08-17T17:33:39.463 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:33:39.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:39 smithi053 ceph-mon[89963]: pgmap v814: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 56 MiB/s wr, 902 op/s; 24603/206256 objects degraded (11.928%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:33:39.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:33:39.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:33:39.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:39.508 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:39.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:33:39.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:39.654 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:39.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:39.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:39.757 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:39.782 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:39.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:39.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:39.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:39.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:39.930 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:39.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:40.031 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:40.068 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:40.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:40.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:40.170 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:40.200 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:40.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:40.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:40.302 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:40.329 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:40 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 24603/206256 objects degraded (11.928%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:33:40.329 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:40 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 24603/206256 objects degraded (11.928%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:33:40.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:40.353 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:40.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:40.370 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 44 16 73311 73295 104.093 53.9375 0.0102008 0.00960136 2024-08-17T17:33:40.455 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:40.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:40 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 24603/206256 objects degraded (11.928%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:33:40.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:33:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:33:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:33:40.489 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:40.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:40.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:40.591 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:40.617 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:40.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:40.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:40.718 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:40.745 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:40.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:40.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:40.773 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:33:40.848 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:33:40.931 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:40.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:40.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:33:41.024 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-17T17:33:41.033 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:41.091 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:41.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:41.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:41.128 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:41 smithi003 ceph-mon[100615]: pgmap v815: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 55 MiB/s wr, 885 op/s; 24603/215736 objects degraded (11.404%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:33:41.193 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:41.220 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:41.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:41.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:41.322 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:41.371 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 45 16 74163 74147 102.963 53.25 0.00619189 0.00970582 2024-08-17T17:33:41.385 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:33:41.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:41.396 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:41.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:41.454 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:41 smithi003 ceph-mon[103344]: pgmap v815: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 55 MiB/s wr, 885 op/s; 24603/215736 objects degraded (11.404%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:33:41.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:41 smithi053 ceph-mon[89963]: pgmap v815: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 55 MiB/s wr, 885 op/s; 24603/215736 objects degraded (11.404%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:33:41.498 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:41.525 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:41.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:41.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:41.626 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:41.659 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:41.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:41.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:41.761 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:33:41.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:41.831 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:41.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:33:41.932 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:41.960 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:41.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:41.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:42.062 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:42.089 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:42.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:42.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:42.163 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:33:42.191 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:42.248 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:42.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:42.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:42.350 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:42.371 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 46 16 74994 74978 101.852 51.9375 0.00704746 0.0098092 2024-08-17T17:33:42.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:42.389 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:42.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:42.490 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:42.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:42.537 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:42.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:42.637 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:42.667 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:42.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:42.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:42.769 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:42.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:42.816 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:42.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:42.917 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:42.955 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:42.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:42.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:43.056 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:43.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:43.101 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:43.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:43.136 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-08-17T17:33:43.137 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.5 primary-affinity to 0 (802) 2024-08-17T17:33:43.172 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:43 smithi053 ceph-mon[89963]: pgmap v816: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 55 MiB/s wr, 885 op/s; 24603/215736 objects degraded (11.404%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:33:43.172 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:43 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3345944854' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.0} : dispatch 2024-08-17T17:33:43.173 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:43 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.0} : dispatch 2024-08-17T17:33:43.173 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:43 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/706503510' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:33:43.173 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:43 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:33:43.190 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:33:43.190 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2024-08-17T17:33:43.191 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2024-08-17T17:33:43.191 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2024-08-17T17:33:43.191 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:33:43.202 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:43.232 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:43.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:43.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:43.334 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:43.357 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:43.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:43.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:43.371 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 47 16 75614 75598 100.51 38.75 0.0072413 0.00991494 2024-08-17T17:33:43.459 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:33:43.482 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:43.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:43.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:33:43.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:43 smithi003 ceph-mon[103344]: pgmap v816: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 55 MiB/s wr, 885 op/s; 24603/215736 objects degraded (11.404%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:33:43.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:43 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3345944854' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.0} : dispatch 2024-08-17T17:33:43.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:43 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.0} : dispatch 2024-08-17T17:33:43.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:43 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/706503510' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:33:43.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:43 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:33:43.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:43 smithi003 ceph-mon[100615]: pgmap v816: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 55 MiB/s wr, 885 op/s; 24603/215736 objects degraded (11.404%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:33:43.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:43 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3345944854' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.0} : dispatch 2024-08-17T17:33:43.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:43 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 0.0} : dispatch 2024-08-17T17:33:43.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:43 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/706503510' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:33:43.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:43 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:33:43.583 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:43.606 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:43.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:43.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:43.708 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:43.731 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:43.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:43.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:43.802 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-17T17:33:43.833 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:33:43.856 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:43.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:43.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:33:43.958 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:43.981 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:43.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:43.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:44.070 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:33:44.083 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:44.106 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:44.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:44.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:44.208 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:33:44.247 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:44.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:44.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:33:44.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:44 smithi003 ceph-mon[100615]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:33:44.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:44 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-08-17T17:33:44.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:44 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:33:44.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:44 smithi003 ceph-mon[100615]: osdmap e272: 8 total, 8 up, 8 in 2024-08-17T17:33:44.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:44 smithi003 ceph-mon[100615]: osdmap e273: 8 total, 8 up, 8 in 2024-08-17T17:33:44.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:44 smithi003 ceph-mon[103344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:33:44.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:44 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-08-17T17:33:44.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:44 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:33:44.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:44 smithi003 ceph-mon[103344]: osdmap e272: 8 total, 8 up, 8 in 2024-08-17T17:33:44.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:44 smithi003 ceph-mon[103344]: osdmap e273: 8 total, 8 up, 8 in 2024-08-17T17:33:44.349 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:44.371 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 48 16 75789 75773 98.643 10.9375 0.0310348 0.0101309 2024-08-17T17:33:44.392 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:44.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:44.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:44.423 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:44 smithi053 ceph-mon[89963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:33:44.423 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:44 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.0}]': finished 2024-08-17T17:33:44.423 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:44 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:33:44.423 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:44 smithi053 ceph-mon[89963]: osdmap e272: 8 total, 8 up, 8 in 2024-08-17T17:33:44.424 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:44 smithi053 ceph-mon[89963]: osdmap e273: 8 total, 8 up, 8 in 2024-08-17T17:33:44.494 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:44.520 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:44.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:44.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:44.622 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:44.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:44.651 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:44.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:44.753 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:44.780 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:44.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:44.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:44.837 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:33:44.838 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:33:44.874 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:33:44.874 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:33:44.882 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:44.906 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:44.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:44.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:44.909 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:33:44.909 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:33:44.947 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:33:44.947 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:33:44.983 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:33:44.983 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:33:45.008 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:45.022 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:33:45.022 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:33:45.039 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:45.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:45.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:45.054 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:33:45.054 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:33:45.087 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:33:45.088 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:33:45.141 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:45.168 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:45.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:45.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:45.270 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:45.313 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:45.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:45.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:45.330 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:45 smithi053 ceph-mon[89963]: pgmap v818: 89 pgs: 9 peering, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 73 active+clean; 4.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 47 MiB/s wr, 747 op/s; 24476/224901 objects degraded (10.883%); 1.6 MiB/s, 23 objects/s recovering 2024-08-17T17:33:45.330 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:45 smithi053 ceph-mon[89963]: Health check failed: Reduced data availability: 2 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-08-17T17:33:45.330 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:45 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 24476/224901 objects degraded (10.883%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:33:45.330 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:45 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1006134282' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:33:45.373 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 49 16 76641 76625 97.7166 53.25 0.00688629 0.0102265 2024-08-17T17:33:45.415 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:45.442 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:45.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:45.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:45.496 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:45 smithi003 ceph-mon[103344]: pgmap v818: 89 pgs: 9 peering, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 73 active+clean; 4.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 47 MiB/s wr, 747 op/s; 24476/224901 objects degraded (10.883%); 1.6 MiB/s, 23 objects/s recovering 2024-08-17T17:33:45.496 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:45 smithi003 ceph-mon[103344]: Health check failed: Reduced data availability: 2 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-08-17T17:33:45.496 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:45 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 24476/224901 objects degraded (10.883%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:33:45.496 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:45 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1006134282' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:33:45.496 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:45 smithi003 ceph-mon[100615]: pgmap v818: 89 pgs: 9 peering, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 73 active+clean; 4.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 47 MiB/s wr, 747 op/s; 24476/224901 objects degraded (10.883%); 1.6 MiB/s, 23 objects/s recovering 2024-08-17T17:33:45.497 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:45 smithi003 ceph-mon[100615]: Health check failed: Reduced data availability: 2 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-08-17T17:33:45.497 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:45 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 24476/224901 objects degraded (10.883%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:33:45.497 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:45 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1006134282' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:33:45.544 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:45.591 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:45.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:45.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:45.693 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:45.733 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:45.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:45.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:45.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:33:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:33:45] "GET /metrics HTTP/1.1" 200 36531 "" "Prometheus/2.51.0" 2024-08-17T17:33:45.835 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:45.870 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:45.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:45.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:45.950 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:33:45.950 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:33:45.972 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:46.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:46.007 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:46.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:46.108 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:46.143 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:46.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:46.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:46.245 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:46.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:46.293 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:46.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:46.372 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 50 16 77547 77531 96.8948 56.625 0.00478119 0.0103132 2024-08-17T17:33:46.394 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:46.424 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:46.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:46.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:46.526 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:46.560 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:46.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:46.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:46.662 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:46.669 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-17T17:33:46.695 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:46.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:46.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:46.797 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:33:46.835 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:46.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:46.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:33:46.937 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:46.987 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:46.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:46.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:47.089 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:47.117 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:33:47.127 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:47.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:47.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:47.207 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:33:47.229 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:47.296 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:47.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:47.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:47.372 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 51 16 78453 78437 96.1052 56.625 0.0331653 0.0103995 2024-08-17T17:33:47.398 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:47.457 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:47.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:47.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:47.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:47 smithi053 ceph-mon[89963]: pgmap v820: 89 pgs: 9 peering, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 73 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 52 MiB/s wr, 833 op/s; 24476/226260 objects degraded (10.818%); 1016 KiB/s, 11 objects/s recovering 2024-08-17T17:33:47.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:47 smithi003 ceph-mon[100615]: pgmap v820: 89 pgs: 9 peering, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 73 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 52 MiB/s wr, 833 op/s; 24476/226260 objects degraded (10.818%); 1016 KiB/s, 11 objects/s recovering 2024-08-17T17:33:47.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:47 smithi003 ceph-mon[103344]: pgmap v820: 89 pgs: 9 peering, 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 73 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 52 MiB/s wr, 833 op/s; 24476/226260 objects degraded (10.818%); 1016 KiB/s, 11 objects/s recovering 2024-08-17T17:33:47.559 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:47.593 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:47.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:47.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:47.688 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:33:47.695 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:33:47.743 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:47.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:47.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:33:47.845 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:47.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:47.891 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:47.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:47.992 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:48.020 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:48.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:48.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:48.122 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:48.165 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:48.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:48.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:48.185 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:33:48.268 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:48.330 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:48.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:48.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:48.372 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 52 16 79262 79246 95.2294 50.5625 0.0299265 0.0104952 2024-08-17T17:33:48.432 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:48.464 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:48.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:48.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:48.565 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:48.594 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:48.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:48.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:48.696 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:48.724 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:48.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:48.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:48.826 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:33:48.870 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:48.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:48.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:33:48.893 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:33:48.893 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:33:48.972 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:48.996 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:48.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:48.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:49.098 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:33:49.155 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:49.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:49.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:33:49.257 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:49.284 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:49.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:49.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:49.372 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 53 16 80131 80115 94.457 54.3125 0.0271466 0.0105807 2024-08-17T17:33:49.386 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:49.412 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:49.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:49.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:49.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:49 smithi053 ceph-mon[89963]: pgmap v821: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 9 peering, 6 active+recovery_wait+undersized+degraded+remapped, 73 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 38 MiB/s wr, 608 op/s; 24268/230349 objects degraded (10.535%); 2.6 MiB/s, 37 objects/s recovering 2024-08-17T17:33:49.513 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:49.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:49 smithi003 ceph-mon[103344]: pgmap v821: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 9 peering, 6 active+recovery_wait+undersized+degraded+remapped, 73 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 38 MiB/s wr, 608 op/s; 24268/230349 objects degraded (10.535%); 2.6 MiB/s, 37 objects/s recovering 2024-08-17T17:33:49.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:49 smithi003 ceph-mon[100615]: pgmap v821: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 9 peering, 6 active+recovery_wait+undersized+degraded+remapped, 73 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 38 MiB/s wr, 608 op/s; 24268/230349 objects degraded (10.535%); 2.6 MiB/s, 37 objects/s recovering 2024-08-17T17:33:49.540 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:49.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:49.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:49.632 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-17T17:33:49.641 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:49.655 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:33:49.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:49.688 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:49.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:49.789 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:49.821 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:49.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:49.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:49.923 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:49.971 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:49.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:49.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:49.994 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:33:50.073 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:50.099 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:50.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:50.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:50.198 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-08-17T17:33:50.200 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:33:50.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:50.255 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:50.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:33:50.356 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:50.373 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 54 16 80893 80877 93.5887 47.625 0.0108372 0.0106782 2024-08-17T17:33:50.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:50.400 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:50.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:50.401 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:33:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:33:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:33:50.401 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:50 smithi053 ceph-mon[89963]: Health check update: Reduced data availability: 5 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-08-17T17:33:50.401 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:50 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 24268/230349 objects degraded (10.535%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:33:50.401 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:50 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1978207942' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:33:50.401 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:50 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:33:50.501 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:50.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:50 smithi003 ceph-mon[100615]: Health check update: Reduced data availability: 5 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-08-17T17:33:50.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:50 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 24268/230349 objects degraded (10.535%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:33:50.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:50 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1978207942' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:33:50.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:50 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:33:50.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:50 smithi003 ceph-mon[103344]: Health check update: Reduced data availability: 5 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-08-17T17:33:50.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:50 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 24268/230349 objects degraded (10.535%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:33:50.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:50 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1978207942' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:33:50.527 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:50 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:33:50.544 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:50.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:50.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:50.646 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:50.674 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:50.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:50.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:50.776 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:50.805 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:50.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:50.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:50.907 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:50.935 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:50.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:50.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:51.036 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:51.065 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:51.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:51.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:51.168 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:51.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:51.223 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:51.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:51.324 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:51.375 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 55 16 81722 81706 92.8284 51.8125 0.0205953 0.0107658 2024-08-17T17:33:51.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:51.377 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:51.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:51.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:51 smithi053 ceph-mon[89963]: pgmap v822: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 60 MiB/s wr, 958 op/s; 24268/238746 objects degraded (10.165%); 2.6 MiB/s, 37 objects/s recovering 2024-08-17T17:33:51.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:51 smithi053 ceph-mon[89963]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 7 pgs peering) 2024-08-17T17:33:51.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:51 smithi053 ceph-mon[89963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:33:51.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:51 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:33:51.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:51 smithi053 ceph-mon[89963]: osdmap e274: 8 total, 8 up, 8 in 2024-08-17T17:33:51.478 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:51.509 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:51.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:51.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:51.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:51 smithi003 ceph-mon[103344]: pgmap v822: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 60 MiB/s wr, 958 op/s; 24268/238746 objects degraded (10.165%); 2.6 MiB/s, 37 objects/s recovering 2024-08-17T17:33:51.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:51 smithi003 ceph-mon[103344]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 7 pgs peering) 2024-08-17T17:33:51.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:51 smithi003 ceph-mon[103344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:33:51.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:51 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:33:51.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:51 smithi003 ceph-mon[103344]: osdmap e274: 8 total, 8 up, 8 in 2024-08-17T17:33:51.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:51 smithi003 ceph-mon[100615]: pgmap v822: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 60 MiB/s wr, 958 op/s; 24268/238746 objects degraded (10.165%); 2.6 MiB/s, 37 objects/s recovering 2024-08-17T17:33:51.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:51 smithi003 ceph-mon[100615]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 7 pgs peering) 2024-08-17T17:33:51.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:51 smithi003 ceph-mon[100615]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:33:51.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:51 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:33:51.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:51 smithi003 ceph-mon[100615]: osdmap e274: 8 total, 8 up, 8 in 2024-08-17T17:33:51.611 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:51.638 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:51.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:51.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:51.683 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-08-17T17:33:51.683 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:33:51.684 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-08-17T17:33:51.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:33:51.740 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:51.768 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:51.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:51.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:51.828 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:33:51.829 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:33:51.870 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:33:51.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:51.914 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:51.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:33:52.015 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:33:52.087 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:52.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:52.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:33:52.189 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:52.216 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:52.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:52.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:52.318 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:52.348 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:52.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:52.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:52.374 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 56 16 82554 82538 92.0993 52 0.0281555 0.0108502 2024-08-17T17:33:52.450 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:52.482 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:52.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:52.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:52.491 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-17T17:33:52.584 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:52.618 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:52.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:52.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:52.720 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:52.746 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:52.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:52.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:52.848 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:52.888 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:52.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:52.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:52.902 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:33:52.991 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:53.044 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:53.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:53.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:53.067 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:33:53.085 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:33:53.086 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:33:53.133 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:33:53.133 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:33:53.146 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:53.172 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:33:53.173 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:33:53.173 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:53.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:53.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:53.218 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:33:53.218 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:33:53.243 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:53 smithi003 ceph-mon[103344]: pgmap v824: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 36 MiB/s wr, 576 op/s; 24268/238746 objects degraded (10.165%); 1.6 MiB/s, 24 objects/s recovering 2024-08-17T17:33:53.243 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:53 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1267844300' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:33:53.243 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:33:53.244 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:53 smithi003 ceph-mon[100615]: pgmap v824: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 36 MiB/s wr, 576 op/s; 24268/238746 objects degraded (10.165%); 1.6 MiB/s, 24 objects/s recovering 2024-08-17T17:33:53.244 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:53 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1267844300' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:33:53.244 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:33:53.250 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:33:53.250 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:33:53.275 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:53.285 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:33:53.285 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:33:53.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:53.302 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:53.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:53.319 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:33:53.319 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:33:53.351 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:33:53.351 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:33:53.373 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 57 16 83304 83288 91.3059 46.875 0.00724752 0.0109446 2024-08-17T17:33:53.403 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:53.429 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:53.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:53.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:53.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:53 smithi053 ceph-mon[89963]: pgmap v824: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 36 MiB/s wr, 576 op/s; 24268/238746 objects degraded (10.165%); 1.6 MiB/s, 24 objects/s recovering 2024-08-17T17:33:53.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:53 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1267844300' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:33:53.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:33:53.531 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:53.560 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:53.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:53.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:53.662 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:53.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:53.715 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:53.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:53.816 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:53.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:53.863 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:53.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:53.964 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:53.994 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:53.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:53.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:54.098 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:54.141 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:54.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:54.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:54.244 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:54.293 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:54.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:54.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:54.374 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 58 16 84159 84143 90.653 53.4375 0.0277948 0.011025 2024-08-17T17:33:54.395 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:54.419 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:33:54.424 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:54.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:54.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:54.525 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:54.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:54.591 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:54.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:54.692 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:54.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:54.746 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:54.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:54.847 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:33:54.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:54.906 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:54.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:33:55.007 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:33:55.063 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:55.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:55.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:33:55.165 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:55.200 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:55.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:55.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:55.268 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:55 smithi053 ceph-mon[89963]: pgmap v825: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 51 MiB/s wr, 817 op/s; 24148/249297 objects degraded (9.686%); 2.1 MiB/s, 32 objects/s recovering 2024-08-17T17:33:55.268 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:55 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 24148/249297 objects degraded (9.686%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:33:55.302 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:55.365 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:55.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:55.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:55.374 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 59 16 84949 84933 89.9534 49.375 0.0136664 0.01111 2024-08-17T17:33:55.394 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:33:55.394 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2024-08-17T17:33:55.394 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2024-08-17T17:33:55.394 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2024-08-17T17:33:55.394 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.394 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:51027 6.5 6:a62b8487:::benchmark_data_smithi003_175043_object51026:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-08-17T17:33:55.394 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:03.980042+0000", 2024-08-17T17:33:55.395 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 531.41027875099996, 2024-08-17T17:33:55.395 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.629104286, 2024-08-17T17:33:55.395 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:55.395 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2024-08-17T17:33:55.395 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:55.395 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:55.395 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:55.395 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 51027 2024-08-17T17:33:55.395 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.396 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:55.396 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.396 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:55.396 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:03.980042+0000", 2024-08-17T17:33:55.396 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.396 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.396 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.396 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:55.396 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:03.980042+0000", 2024-08-17T17:33:55.396 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.396 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.396 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.397 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:55.397 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:03.980041+0000", 2024-08-17T17:33:55.397 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:33:55.397 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.397 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.397 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:55.397 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:03.980090+0000", 2024-08-17T17:33:55.397 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9597e-05 2024-08-17T17:33:55.397 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.397 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.397 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:55.397 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:03.980101+0000", 2024-08-17T17:33:55.398 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1056000000000001e-05 2024-08-17T17:33:55.398 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.398 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.398 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:55.398 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:03.980106+0000", 2024-08-17T17:33:55.398 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1439999999999996e-06 2024-08-17T17:33:55.398 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.398 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.398 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.398 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:03.980149+0000", 2024-08-17T17:33:55.398 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3813999999999998e-05 2024-08-17T17:33:55.399 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.399 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.399 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:55.399 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:03.980282+0000", 2024-08-17T17:33:55.399 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013306400000000001 2024-08-17T17:33:55.399 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.399 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.399 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,3", 2024-08-17T17:33:55.399 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:03.980413+0000", 2024-08-17T17:33:55.399 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013048399999999999 2024-08-17T17:33:55.399 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.399 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.399 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:55.400 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:03.981965+0000", 2024-08-17T17:33:55.400 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0015519030000000001 2024-08-17T17:33:55.400 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.400 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.400 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.400 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:03.987450+0000", 2024-08-17T17:33:55.400 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0054850009999999998 2024-08-17T17:33:55.400 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.400 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.400 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.400 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:03.994306+0000", 2024-08-17T17:33:55.400 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0068563549999999997 2024-08-17T17:33:55.400 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.400 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.401 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:55.401 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:03.994316+0000", 2024-08-17T17:33:55.401 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.9750000000000002e-06 2024-08-17T17:33:55.401 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.401 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.401 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.401 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.498819+0000", 2024-08-17T17:33:55.401 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.50450318999999999 2024-08-17T17:33:55.401 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.401 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.401 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:55.401 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.498830+0000", 2024-08-17T17:33:55.401 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0706999999999999e-05 2024-08-17T17:33:55.402 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.402 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.402 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.402 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.609045+0000", 2024-08-17T17:33:55.402 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1102146129999999 2024-08-17T17:33:55.402 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.403 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.403 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:55.403 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.609147+0000", 2024-08-17T17:33:55.403 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000101789 2024-08-17T17:33:55.403 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.404 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:55.404 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.404 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.404 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.404 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:51039 6.5 6:acbe4258:::benchmark_data_smithi003_175043_object51038:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-08-17T17:33:55.404 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:04.000857+0000", 2024-08-17T17:33:55.404 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 531.38946362399997, 2024-08-17T17:33:55.404 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6213081819999999, 2024-08-17T17:33:55.405 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:55.405 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:55.405 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:55.405 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:55.405 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:55.405 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 51039 2024-08-17T17:33:55.405 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.405 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:55.405 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.405 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:55.405 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.000857+0000", 2024-08-17T17:33:55.406 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.406 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.406 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.406 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:55.406 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.000857+0000", 2024-08-17T17:33:55.406 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.406 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.406 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.406 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:55.406 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.000857+0000", 2024-08-17T17:33:55.406 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:33:55.406 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.407 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.407 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:55.407 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.000884+0000", 2024-08-17T17:33:55.407 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7793999999999999e-05 2024-08-17T17:33:55.407 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.407 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.407 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:55.407 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.000896+0000", 2024-08-17T17:33:55.407 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1399e-05 2024-08-17T17:33:55.407 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.407 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.407 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:55.407 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.000900+0000", 2024-08-17T17:33:55.408 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1609999999999996e-06 2024-08-17T17:33:55.408 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.408 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.408 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.408 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.000953+0000", 2024-08-17T17:33:55.408 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2871000000000003e-05 2024-08-17T17:33:55.408 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.408 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.408 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:55.408 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.000962+0000", 2024-08-17T17:33:55.408 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.8070000000000006e-06 2024-08-17T17:33:55.408 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.409 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.409 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.409 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.498857+0000", 2024-08-17T17:33:55.409 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.49789539199999999 2024-08-17T17:33:55.409 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.409 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.409 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:55.409 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.498879+0000", 2024-08-17T17:33:55.409 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1815e-05 2024-08-17T17:33:55.409 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.409 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.409 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.410 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.610645+0000", 2024-08-17T17:33:55.410 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.111765892 2024-08-17T17:33:55.410 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.410 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.410 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:55.410 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.610784+0000", 2024-08-17T17:33:55.410 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000138889 2024-08-17T17:33:55.410 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.410 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.410 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,6", 2024-08-17T17:33:55.410 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.610899+0000", 2024-08-17T17:33:55.411 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000115567 2024-08-17T17:33:55.411 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.411 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.411 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:55.411 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.614497+0000", 2024-08-17T17:33:55.411 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003598319 2024-08-17T17:33:55.411 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.411 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.412 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.412 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.618508+0000", 2024-08-17T17:33:55.412 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0040104010000000002 2024-08-17T17:33:55.412 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.412 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.412 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.412 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.619872+0000", 2024-08-17T17:33:55.412 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0013636659999999999 2024-08-17T17:33:55.412 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.413 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.413 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:55.413 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.619939+0000", 2024-08-17T17:33:55.413 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.7108999999999997e-05 2024-08-17T17:33:55.413 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.413 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.413 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:55.413 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.622166+0000", 2024-08-17T17:33:55.413 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0022268740000000002 2024-08-17T17:33:55.413 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.414 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:55.414 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.414 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.414 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.414 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:51079 6.5 6:ac56d4c0:::benchmark_data_smithi003_175043_object51078:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-08-17T17:33:55.414 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:04.620257+0000", 2024-08-17T17:33:55.414 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 530.77006383399998, 2024-08-17T17:33:55.414 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0032013129999999, 2024-08-17T17:33:55.414 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:55.414 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:55.414 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:55.415 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:55.415 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:55.415 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 51079 2024-08-17T17:33:55.415 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.415 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:55.415 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.415 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:55.415 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.620257+0000", 2024-08-17T17:33:55.415 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.415 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.415 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.415 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:55.415 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.620257+0000", 2024-08-17T17:33:55.416 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.416 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.416 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.416 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:55.416 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.620257+0000", 2024-08-17T17:33:55.416 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:33:55.416 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.416 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.416 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:55.416 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.620280+0000", 2024-08-17T17:33:55.416 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3575999999999999e-05 2024-08-17T17:33:55.416 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.416 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.417 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:55.417 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.620286+0000", 2024-08-17T17:33:55.417 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1430000000000003e-06 2024-08-17T17:33:55.417 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.417 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.417 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:55.417 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.620289+0000", 2024-08-17T17:33:55.417 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8820000000000001e-06 2024-08-17T17:33:55.417 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.417 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.417 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.417 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.620343+0000", 2024-08-17T17:33:55.417 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3631999999999997e-05 2024-08-17T17:33:55.418 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.418 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.418 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:55.418 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.620353+0000", 2024-08-17T17:33:55.418 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6690000000000001e-06 2024-08-17T17:33:55.418 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.418 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.418 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.418 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.612654+0000", 2024-08-17T17:33:55.418 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.99230118300000003 2024-08-17T17:33:55.418 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.418 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.419 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:55.419 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.612791+0000", 2024-08-17T17:33:55.419 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013729899999999999 2024-08-17T17:33:55.419 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.419 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.419 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,6", 2024-08-17T17:33:55.419 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.612967+0000", 2024-08-17T17:33:55.419 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00017623500000000001 2024-08-17T17:33:55.419 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.419 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.419 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:55.419 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.616374+0000", 2024-08-17T17:33:55.420 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003406708 2024-08-17T17:33:55.420 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.420 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.420 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.420 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.618561+0000", 2024-08-17T17:33:55.420 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0021864879999999999 2024-08-17T17:33:55.420 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.420 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.420 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.420 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.622299+0000", 2024-08-17T17:33:55.420 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0037389419999999999 2024-08-17T17:33:55.420 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.420 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.421 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:55.421 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.622346+0000", 2024-08-17T17:33:55.421 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6230999999999998e-05 2024-08-17T17:33:55.421 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.421 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.421 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:55.421 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.623458+0000", 2024-08-17T17:33:55.421 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001112785 2024-08-17T17:33:55.421 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.421 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:55.421 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.421 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.421 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.422 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:51103 6.5 6:adee92f5:::benchmark_data_smithi003_175043_object51102:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-08-17T17:33:55.422 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:25:04.672180+0000", 2024-08-17T17:33:55.422 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 530.71814059600001, 2024-08-17T17:33:55.422 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.95579269499999997, 2024-08-17T17:33:55.422 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:55.422 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:55.422 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:55.422 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:55.422 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:55.422 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 51103 2024-08-17T17:33:55.422 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.422 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:55.422 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.423 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:55.423 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.672180+0000", 2024-08-17T17:33:55.423 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.423 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.423 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.423 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:55.423 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.672180+0000", 2024-08-17T17:33:55.423 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.423 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.423 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.423 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:55.423 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.672180+0000", 2024-08-17T17:33:55.423 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:33:55.423 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.424 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.424 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:55.424 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.672245+0000", 2024-08-17T17:33:55.424 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5209000000000005e-05 2024-08-17T17:33:55.424 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.424 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.424 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:55.424 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.672258+0000", 2024-08-17T17:33:55.424 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2855e-05 2024-08-17T17:33:55.424 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.424 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.424 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:55.424 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.672263+0000", 2024-08-17T17:33:55.424 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4079999999999997e-06 2024-08-17T17:33:55.425 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.425 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.425 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.425 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.672307+0000", 2024-08-17T17:33:55.425 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4305999999999999e-05 2024-08-17T17:33:55.425 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.425 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.425 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:55.425 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:04.672315+0000", 2024-08-17T17:33:55.425 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3900000000000004e-06 2024-08-17T17:33:55.425 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.425 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.425 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.426 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.614552+0000", 2024-08-17T17:33:55.426 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.94223699100000002 2024-08-17T17:33:55.426 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.426 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.426 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:55.426 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.614660+0000", 2024-08-17T17:33:55.426 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000107824 2024-08-17T17:33:55.426 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.426 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.426 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,6", 2024-08-17T17:33:55.427 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.614765+0000", 2024-08-17T17:33:55.427 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000105938 2024-08-17T17:33:55.427 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.427 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.427 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:55.427 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.616527+0000", 2024-08-17T17:33:55.427 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0017612249999999999 2024-08-17T17:33:55.427 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.427 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.427 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.427 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.623505+0000", 2024-08-17T17:33:55.428 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0069780909999999996 2024-08-17T17:33:55.428 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.428 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.428 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.428 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.625477+0000", 2024-08-17T17:33:55.428 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0019721510000000001 2024-08-17T17:33:55.428 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.428 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.428 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:55.428 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.625547+0000", 2024-08-17T17:33:55.428 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0289000000000004e-05 2024-08-17T17:33:55.428 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.428 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.429 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:55.429 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:25:05.627973+0000", 2024-08-17T17:33:55.429 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0024258679999999999 2024-08-17T17:33:55.429 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.429 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:55.429 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.429 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.429 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.429 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:130180 6.5 6:aab40d0b:::benchmark_data_smithi003_175043_object31531:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-08-17T17:33:55.429 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:26:39.395823+0000", 2024-08-17T17:33:55.429 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 435.99449796200003, 2024-08-17T17:33:55.429 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0299970549999999, 2024-08-17T17:33:55.429 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:55.429 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2024-08-17T17:33:55.430 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:55.430 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:55.430 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:55.430 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 130180 2024-08-17T17:33:55.430 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.430 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:55.430 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.430 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:55.430 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.395823+0000", 2024-08-17T17:33:55.431 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.431 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.431 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.431 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:55.431 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.395823+0000", 2024-08-17T17:33:55.431 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.431 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.431 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.431 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:55.431 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.395822+0000", 2024-08-17T17:33:55.431 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:33:55.431 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.432 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.432 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:55.432 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.395828+0000", 2024-08-17T17:33:55.432 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6110000000000001e-06 2024-08-17T17:33:55.432 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.432 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.432 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:55.432 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.395839+0000", 2024-08-17T17:33:55.432 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.168e-05 2024-08-17T17:33:55.432 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.432 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.432 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:55.433 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.395846+0000", 2024-08-17T17:33:55.433 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0700000000000001e-06 2024-08-17T17:33:55.433 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.433 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.433 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.433 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.395889+0000", 2024-08-17T17:33:55.433 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2938000000000001e-05 2024-08-17T17:33:55.433 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.433 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.433 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:55.433 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.395999+0000", 2024-08-17T17:33:55.433 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000109954 2024-08-17T17:33:55.434 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.434 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.434 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,6", 2024-08-17T17:33:55.434 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.396086+0000", 2024-08-17T17:33:55.434 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.6395000000000002e-05 2024-08-17T17:33:55.434 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.434 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.434 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:55.434 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.399219+0000", 2024-08-17T17:33:55.435 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0031329470000000001 2024-08-17T17:33:55.435 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.435 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.435 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.435 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.399351+0000", 2024-08-17T17:33:55.435 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000131965 2024-08-17T17:33:55.435 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.435 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.435 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.435 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.412194+0000", 2024-08-17T17:33:55.435 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012843108000000001 2024-08-17T17:33:55.435 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.436 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.436 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:55.436 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.412206+0000", 2024-08-17T17:33:55.436 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2860999999999999e-05 2024-08-17T17:33:55.436 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.436 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.436 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.436 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.425771+0000", 2024-08-17T17:33:55.436 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.013564382 2024-08-17T17:33:55.436 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.436 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.436 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:55.436 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.425820+0000", 2024-08-17T17:33:55.436 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9215000000000002e-05 2024-08-17T17:33:55.437 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.437 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:55.437 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.437 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.437 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.437 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:130209 6.5 6:a6e9550c:::benchmark_data_smithi003_175043_object31560:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-08-17T17:33:55.437 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:26:39.452057+0000", 2024-08-17T17:33:55.437 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 435.938264409, 2024-08-17T17:33:55.437 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.98617804200000003, 2024-08-17T17:33:55.437 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:55.437 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:55.437 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:55.438 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:55.438 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:55.438 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 130209 2024-08-17T17:33:55.438 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.438 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:55.438 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.438 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:55.438 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.452057+0000", 2024-08-17T17:33:55.438 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.438 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.438 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.438 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:55.439 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.452057+0000", 2024-08-17T17:33:55.439 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.439 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.439 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.439 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:55.439 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.452056+0000", 2024-08-17T17:33:55.439 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:33:55.439 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.439 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.439 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:55.439 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.452060+0000", 2024-08-17T17:33:55.439 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1189999999999999e-06 2024-08-17T17:33:55.440 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.440 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.440 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:55.440 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.452069+0000", 2024-08-17T17:33:55.440 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.7190000000000007e-06 2024-08-17T17:33:55.440 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.440 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.440 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:55.440 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.452072+0000", 2024-08-17T17:33:55.440 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2200000000000001e-06 2024-08-17T17:33:55.440 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.440 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.440 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.440 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.452128+0000", 2024-08-17T17:33:55.441 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6128999999999997e-05 2024-08-17T17:33:55.441 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.441 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.441 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:55.441 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.452137+0000", 2024-08-17T17:33:55.441 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1090000000000004e-06 2024-08-17T17:33:55.441 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.441 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.441 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.441 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.426086+0000", 2024-08-17T17:33:55.441 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.973948429 2024-08-17T17:33:55.441 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.441 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.442 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:55.442 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.426212+0000", 2024-08-17T17:33:55.442 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000126489 2024-08-17T17:33:55.442 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.442 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.442 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,6", 2024-08-17T17:33:55.442 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.426345+0000", 2024-08-17T17:33:55.442 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013292100000000001 2024-08-17T17:33:55.442 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.442 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.442 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.442 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.432225+0000", 2024-08-17T17:33:55.442 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0058797520000000002 2024-08-17T17:33:55.443 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.443 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.443 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:55.443 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.434985+0000", 2024-08-17T17:33:55.443 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0027605540000000001 2024-08-17T17:33:55.443 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.443 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.443 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.443 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.438156+0000", 2024-08-17T17:33:55.443 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003170679 2024-08-17T17:33:55.443 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.444 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.444 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:55.444 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.438199+0000", 2024-08-17T17:33:55.444 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2528999999999999e-05 2024-08-17T17:33:55.444 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.444 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.444 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:55.444 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.438235+0000", 2024-08-17T17:33:55.444 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6001999999999999e-05 2024-08-17T17:33:55.444 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.445 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:55.445 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.445 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.445 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.445 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:130221 6.5 6:ab1fee0c:::benchmark_data_smithi003_175043_object31572:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-08-17T17:33:55.445 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:26:39.467209+0000", 2024-08-17T17:33:55.446 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 435.92311195899998, 2024-08-17T17:33:55.446 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.97115574800000004, 2024-08-17T17:33:55.446 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:55.446 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:55.446 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:55.447 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:55.447 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:55.447 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 130221 2024-08-17T17:33:55.447 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.447 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:55.447 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.448 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:55.448 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.467209+0000", 2024-08-17T17:33:55.448 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.448 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.448 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.448 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:55.449 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.467209+0000", 2024-08-17T17:33:55.449 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.449 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.449 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.449 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:55.450 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.467208+0000", 2024-08-17T17:33:55.450 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:33:55.450 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.450 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.450 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:55.451 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.467214+0000", 2024-08-17T17:33:55.451 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6829999999999996e-06 2024-08-17T17:33:55.451 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.451 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.451 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:55.451 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.467228+0000", 2024-08-17T17:33:55.452 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4124000000000001e-05 2024-08-17T17:33:55.452 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.452 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.452 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:55.452 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.467235+0000", 2024-08-17T17:33:55.452 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.7320000000000001e-06 2024-08-17T17:33:55.453 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.453 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.453 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.453 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.467292+0000", 2024-08-17T17:33:55.453 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7442999999999999e-05 2024-08-17T17:33:55.453 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.454 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.454 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:55.454 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.467304+0000", 2024-08-17T17:33:55.454 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.163e-05 2024-08-17T17:33:55.454 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.455 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.455 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.455 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.429166+0000", 2024-08-17T17:33:55.455 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.96186255099999995 2024-08-17T17:33:55.455 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.456 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.456 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:55.456 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.429300+0000", 2024-08-17T17:33:55.456 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000133867 2024-08-17T17:33:55.456 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.457 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.457 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,6", 2024-08-17T17:33:55.457 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.429438+0000", 2024-08-17T17:33:55.457 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000137508 2024-08-17T17:33:55.457 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.458 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.458 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:55.458 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.434988+0000", 2024-08-17T17:33:55.458 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0055507059999999999 2024-08-17T17:33:55.458 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.459 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.459 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.459 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.438099+0000", 2024-08-17T17:33:55.459 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0031103350000000001 2024-08-17T17:33:55.459 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.459 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.459 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.460 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.438298+0000", 2024-08-17T17:33:55.460 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00019952500000000001 2024-08-17T17:33:55.460 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.460 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.460 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:55.461 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.438333+0000", 2024-08-17T17:33:55.461 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5238999999999999e-05 2024-08-17T17:33:55.461 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.461 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.461 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:55.461 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.438365+0000", 2024-08-17T17:33:55.462 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1327e-05 2024-08-17T17:33:55.462 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.462 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:55.462 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.462 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.463 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.463 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:130224 6.5 6:a603362e:::benchmark_data_smithi003_175043_object31575:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-08-17T17:33:55.463 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:26:39.472409+0000", 2024-08-17T17:33:55.463 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 435.91791246100001, 2024-08-17T17:33:55.463 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.97190938100000002, 2024-08-17T17:33:55.464 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:55.464 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:55.464 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:55.464 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:55.464 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:55.464 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 130224 2024-08-17T17:33:55.465 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.465 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:55.465 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.465 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:55.465 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.472409+0000", 2024-08-17T17:33:55.465 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.466 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.466 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.466 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:55.466 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.472409+0000", 2024-08-17T17:33:55.466 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.467 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.467 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.467 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:55.467 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.472407+0000", 2024-08-17T17:33:55.467 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:33:55.467 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.468 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.468 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:55.468 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.472414+0000", 2024-08-17T17:33:55.468 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1879999999999999e-06 2024-08-17T17:33:55.468 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.469 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.469 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:55.469 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.472426+0000", 2024-08-17T17:33:55.469 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2283000000000001e-05 2024-08-17T17:33:55.469 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.469 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.470 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:55.470 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.472432+0000", 2024-08-17T17:33:55.470 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.472e-06 2024-08-17T17:33:55.470 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.470 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.471 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.471 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.472487+0000", 2024-08-17T17:33:55.471 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4193000000000001e-05 2024-08-17T17:33:55.471 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.471 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.471 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:55.471 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.472496+0000", 2024-08-17T17:33:55.472 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1749999999999994e-06 2024-08-17T17:33:55.472 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.472 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.472 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.472 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.432270+0000", 2024-08-17T17:33:55.472 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.95977414999999999 2024-08-17T17:33:55.472 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.472 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.473 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:55.473 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.432362+0000", 2024-08-17T17:33:55.473 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1762999999999995e-05 2024-08-17T17:33:55.473 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.473 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.473 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,6", 2024-08-17T17:33:55.473 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.432432+0000", 2024-08-17T17:33:55.473 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.0352000000000007e-05 2024-08-17T17:33:55.473 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.473 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.474 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.474 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:55.476 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.438269+0000", 2024-08-17T17:33:55.476 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0058365830000000002 2024-08-17T17:33:55.476 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.476 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.476 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:55.476 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.438274+0000", 2024-08-17T17:33:55.476 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1590000000000001e-06 2024-08-17T17:33:55.476 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.477 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.477 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.477 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.444187+0000", 2024-08-17T17:33:55.477 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0059136500000000003 2024-08-17T17:33:55.477 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.477 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.477 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:55.477 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.444280+0000", 2024-08-17T17:33:55.477 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.2147999999999999e-05 2024-08-17T17:33:55.477 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.477 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.477 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:55.477 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.444318+0000", 2024-08-17T17:33:55.477 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8347999999999999e-05 2024-08-17T17:33:55.478 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.478 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:55.478 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.478 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.478 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.478 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:130247 6.5 6:af8bf089:::benchmark_data_smithi003_175043_object31598:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-08-17T17:33:55.478 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:26:39.498424+0000", 2024-08-17T17:33:55.478 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 435.89189685899998, 2024-08-17T17:33:55.478 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.94608363699999998, 2024-08-17T17:33:55.478 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:55.478 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:55.479 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:55.479 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:55.479 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:55.479 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 130247 2024-08-17T17:33:55.479 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.479 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:55.479 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.479 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:55.479 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.498424+0000", 2024-08-17T17:33:55.479 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.479 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.479 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.479 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:55.479 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.498424+0000", 2024-08-17T17:33:55.480 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.480 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.480 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.480 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:55.480 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.498424+0000", 2024-08-17T17:33:55.480 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:33:55.480 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.480 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.480 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:55.480 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.498427+0000", 2024-08-17T17:33:55.480 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.8389999999999998e-06 2024-08-17T17:33:55.480 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.480 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.481 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:55.481 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.498432+0000", 2024-08-17T17:33:55.481 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6030000000000004e-06 2024-08-17T17:33:55.481 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.481 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.481 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:55.481 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.498437+0000", 2024-08-17T17:33:55.481 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2340000000000002e-06 2024-08-17T17:33:55.481 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.481 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.481 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.481 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.498499+0000", 2024-08-17T17:33:55.481 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1822000000000005e-05 2024-08-17T17:33:55.481 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.482 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.482 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:55.482 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.498507+0000", 2024-08-17T17:33:55.482 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0600000000000008e-06 2024-08-17T17:33:55.482 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.482 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.482 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.482 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.435029+0000", 2024-08-17T17:33:55.482 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.93652218499999995 2024-08-17T17:33:55.482 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.482 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.482 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:55.482 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.435151+0000", 2024-08-17T17:33:55.483 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000121217 2024-08-17T17:33:55.483 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.483 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.483 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,6", 2024-08-17T17:33:55.483 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.435282+0000", 2024-08-17T17:33:55.483 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000130918 2024-08-17T17:33:55.483 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.483 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.483 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.483 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.438410+0000", 2024-08-17T17:33:55.483 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0031286439999999999 2024-08-17T17:33:55.483 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.483 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.483 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:55.484 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.438415+0000", 2024-08-17T17:33:55.484 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3460000000000004e-06 2024-08-17T17:33:55.484 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.484 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.484 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.484 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.444399+0000", 2024-08-17T17:33:55.484 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0059842430000000002 2024-08-17T17:33:55.484 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.484 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.484 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:55.485 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.444473+0000", 2024-08-17T17:33:55.485 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3987999999999997e-05 2024-08-17T17:33:55.485 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.485 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.485 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:55.485 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.444508+0000", 2024-08-17T17:33:55.485 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4992e-05 2024-08-17T17:33:55.485 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.485 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:55.485 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.485 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.486 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.486 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:130260 6.5 6:a2aa193e:::benchmark_data_smithi003_175043_object31611:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-08-17T17:33:55.486 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:26:39.514808+0000", 2024-08-17T17:33:55.486 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 435.87551258100001, 2024-08-17T17:33:55.486 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.92982520000000002, 2024-08-17T17:33:55.486 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:55.486 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:55.486 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:55.486 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:55.486 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:55.487 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 130260 2024-08-17T17:33:55.487 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.487 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:55.487 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.487 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:55.487 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.514808+0000", 2024-08-17T17:33:55.487 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.487 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.487 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.487 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:55.487 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.514808+0000", 2024-08-17T17:33:55.487 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.487 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.488 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.488 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:55.488 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.514807+0000", 2024-08-17T17:33:55.488 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:33:55.488 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.488 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.488 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:55.488 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.514813+0000", 2024-08-17T17:33:55.488 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7690000000000002e-06 2024-08-17T17:33:55.488 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.488 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.488 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:55.488 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.514825+0000", 2024-08-17T17:33:55.488 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2503999999999999e-05 2024-08-17T17:33:55.488 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.488 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.489 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:55.489 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.514831+0000", 2024-08-17T17:33:55.489 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3240000000000002e-06 2024-08-17T17:33:55.489 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.489 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.489 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.489 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.514874+0000", 2024-08-17T17:33:55.489 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2984e-05 2024-08-17T17:33:55.489 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.489 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.489 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:55.489 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.514882+0000", 2024-08-17T17:33:55.489 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.8849999999999999e-06 2024-08-17T17:33:55.490 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.490 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.490 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.490 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.438431+0000", 2024-08-17T17:33:55.490 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.92354913000000005 2024-08-17T17:33:55.490 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.490 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.490 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:55.490 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.438527+0000", 2024-08-17T17:33:55.490 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5915999999999994e-05 2024-08-17T17:33:55.490 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.490 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.491 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,6", 2024-08-17T17:33:55.491 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.438631+0000", 2024-08-17T17:33:55.491 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010438999999999999 2024-08-17T17:33:55.491 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.491 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.491 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:55.491 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.444332+0000", 2024-08-17T17:33:55.492 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0057008429999999997 2024-08-17T17:33:55.492 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.492 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.492 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.492 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.444369+0000", 2024-08-17T17:33:55.492 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6724000000000003e-05 2024-08-17T17:33:55.492 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.492 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.492 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.492 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.444564+0000", 2024-08-17T17:33:55.492 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000195582 2024-08-17T17:33:55.493 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.493 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.493 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:55.493 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.444602+0000", 2024-08-17T17:33:55.493 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.7768000000000002e-05 2024-08-17T17:33:55.493 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.493 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.493 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:55.493 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.444634+0000", 2024-08-17T17:33:55.493 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1613999999999999e-05 2024-08-17T17:33:55.493 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.493 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:55.493 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.493 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.494 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.494 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:130267 6.5 6:ad16fc1b:::benchmark_data_smithi003_175043_object31618:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-08-17T17:33:55.494 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:26:39.521044+0000", 2024-08-17T17:33:55.494 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 435.86927662699998, 2024-08-17T17:33:55.494 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.93381886199999997, 2024-08-17T17:33:55.494 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:55.494 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:55.494 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:55.494 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:55.494 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:55.494 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 130267 2024-08-17T17:33:55.494 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.495 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:55.495 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.495 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:55.495 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.521044+0000", 2024-08-17T17:33:55.495 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.495 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.495 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.495 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:55.495 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.521044+0000", 2024-08-17T17:33:55.495 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.495 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.495 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.495 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:55.495 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.521044+0000", 2024-08-17T17:33:55.496 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:33:55.496 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.496 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.496 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:55.496 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.521047+0000", 2024-08-17T17:33:55.496 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9409999999999999e-06 2024-08-17T17:33:55.496 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.496 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.496 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:55.496 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.521054+0000", 2024-08-17T17:33:55.496 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9299999999999997e-06 2024-08-17T17:33:55.496 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.496 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.497 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:55.497 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.521058+0000", 2024-08-17T17:33:55.497 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5900000000000001e-06 2024-08-17T17:33:55.497 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.497 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.497 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.497 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.521117+0000", 2024-08-17T17:33:55.497 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8403999999999999e-05 2024-08-17T17:33:55.497 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.497 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.497 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:55.497 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.521127+0000", 2024-08-17T17:33:55.497 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0023e-05 2024-08-17T17:33:55.497 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.497 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.497 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.498 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.441476+0000", 2024-08-17T17:33:55.498 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.92034908999999998 2024-08-17T17:33:55.498 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.498 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.498 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:55.498 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.441571+0000", 2024-08-17T17:33:55.498 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.4856000000000001e-05 2024-08-17T17:33:55.498 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.498 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.498 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,6", 2024-08-17T17:33:55.498 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.441699+0000", 2024-08-17T17:33:55.498 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012802899999999999 2024-08-17T17:33:55.499 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.499 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.499 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.499 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.448154+0000", 2024-08-17T17:33:55.499 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0064556489999999999 2024-08-17T17:33:55.499 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.499 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.499 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:55.499 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.450964+0000", 2024-08-17T17:33:55.499 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00280949 2024-08-17T17:33:55.499 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.499 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.499 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.499 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.454778+0000", 2024-08-17T17:33:55.500 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003814362 2024-08-17T17:33:55.500 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.500 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.500 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:55.500 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.454825+0000", 2024-08-17T17:33:55.500 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6424000000000001e-05 2024-08-17T17:33:55.500 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.500 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.500 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:55.500 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.454863+0000", 2024-08-17T17:33:55.500 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8622999999999999e-05 2024-08-17T17:33:55.500 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.500 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:55.500 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.501 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.501 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.501 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:130296 6.5 6:a964ae2d:::benchmark_data_smithi003_175043_object31647:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-08-17T17:33:55.501 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:26:39.561033+0000", 2024-08-17T17:33:55.501 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 435.82928784199999, 2024-08-17T17:33:55.501 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.90031448700000005, 2024-08-17T17:33:55.501 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:55.501 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:55.501 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:55.501 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:55.501 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:55.501 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 130296 2024-08-17T17:33:55.501 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.501 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:55.502 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.502 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:55.502 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.561033+0000", 2024-08-17T17:33:55.502 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.502 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.502 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.502 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:55.502 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.561033+0000", 2024-08-17T17:33:55.502 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.502 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.502 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.503 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:55.503 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.561031+0000", 2024-08-17T17:33:55.503 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:33:55.503 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.503 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.503 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:55.503 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.561040+0000", 2024-08-17T17:33:55.503 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.7329999999999992e-06 2024-08-17T17:33:55.503 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.504 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.504 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:55.504 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.561056+0000", 2024-08-17T17:33:55.504 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6164999999999999e-05 2024-08-17T17:33:55.504 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.505 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.505 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:55.505 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.561063+0000", 2024-08-17T17:33:55.505 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.1790000000000002e-06 2024-08-17T17:33:55.505 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.505 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.506 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.506 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.561114+0000", 2024-08-17T17:33:55.506 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0864e-05 2024-08-17T17:33:55.506 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.506 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.506 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:55.506 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.561127+0000", 2024-08-17T17:33:55.506 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3145999999999999e-05 2024-08-17T17:33:55.506 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.506 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.507 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.507 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.444677+0000", 2024-08-17T17:33:55.507 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.88354925299999998 2024-08-17T17:33:55.507 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.507 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.507 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:55.507 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.444774+0000", 2024-08-17T17:33:55.507 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6825999999999998e-05 2024-08-17T17:33:55.507 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.507 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.507 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,6", 2024-08-17T17:33:55.507 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.444893+0000", 2024-08-17T17:33:55.507 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011908099999999999 2024-08-17T17:33:55.508 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.508 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.508 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.508 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.454678+0000", 2024-08-17T17:33:55.508 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0097858660000000007 2024-08-17T17:33:55.508 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.508 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.508 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:55.508 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.454687+0000", 2024-08-17T17:33:55.508 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.5329999999999994e-06 2024-08-17T17:33:55.508 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.508 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.509 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.509 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.461279+0000", 2024-08-17T17:33:55.509 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.006592194 2024-08-17T17:33:55.509 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.509 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.509 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:55.509 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.461312+0000", 2024-08-17T17:33:55.509 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3081999999999999e-05 2024-08-17T17:33:55.509 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.509 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.510 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:55.510 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.461348+0000", 2024-08-17T17:33:55.510 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5355999999999998e-05 2024-08-17T17:33:55.510 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.510 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:55.510 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.510 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.510 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.510 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:130341 6.5 6:aac8c266:::benchmark_data_smithi003_175043_object31692:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-08-17T17:33:55.510 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:26:39.624358+0000", 2024-08-17T17:33:55.510 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 435.76596251000001, 2024-08-17T17:33:55.510 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.84451184400000001, 2024-08-17T17:33:55.511 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:55.511 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:55.511 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:55.511 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:55.511 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:55.511 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 130341 2024-08-17T17:33:55.511 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.511 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:55.511 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.511 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:55.511 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.624358+0000", 2024-08-17T17:33:55.512 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.512 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.512 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.512 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:55.512 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.624358+0000", 2024-08-17T17:33:55.512 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.512 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.512 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.512 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:55.512 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.624358+0000", 2024-08-17T17:33:55.512 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:33:55.512 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.513 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.513 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:55.513 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.624363+0000", 2024-08-17T17:33:55.513 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9420000000000003e-06 2024-08-17T17:33:55.513 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.513 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.513 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:55.513 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.624375+0000", 2024-08-17T17:33:55.513 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1846e-05 2024-08-17T17:33:55.513 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.513 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.514 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:55.514 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.624380+0000", 2024-08-17T17:33:55.514 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2440000000000001e-06 2024-08-17T17:33:55.514 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.514 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.514 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.514 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.624438+0000", 2024-08-17T17:33:55.514 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8575000000000001e-05 2024-08-17T17:33:55.514 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.514 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.514 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:55.515 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.624450+0000", 2024-08-17T17:33:55.515 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1609e-05 2024-08-17T17:33:55.515 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.515 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.515 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.515 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.448181+0000", 2024-08-17T17:33:55.515 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.82373111799999998 2024-08-17T17:33:55.515 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.515 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.515 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:55.515 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.448289+0000", 2024-08-17T17:33:55.516 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010764600000000001 2024-08-17T17:33:55.516 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.516 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.516 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,6", 2024-08-17T17:33:55.516 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.448396+0000", 2024-08-17T17:33:55.516 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000106796 2024-08-17T17:33:55.516 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.516 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.516 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.516 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.454737+0000", 2024-08-17T17:33:55.517 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0063414450000000002 2024-08-17T17:33:55.517 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.517 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.517 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:55.517 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.457949+0000", 2024-08-17T17:33:55.517 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0032123429999999999 2024-08-17T17:33:55.517 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.517 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.517 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.517 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.468776+0000", 2024-08-17T17:33:55.517 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010827041000000001 2024-08-17T17:33:55.517 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.517 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.517 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:55.518 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.468828+0000", 2024-08-17T17:33:55.518 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1746000000000002e-05 2024-08-17T17:33:55.518 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.518 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.518 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:55.518 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.468870+0000", 2024-08-17T17:33:55.518 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2222000000000003e-05 2024-08-17T17:33:55.518 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.518 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:55.518 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.518 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.518 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.519 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45491.0:130373 6.5 6:a027210d:::benchmark_data_smithi003_175043_object31724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-08-17T17:33:55.519 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:26:39.667523+0000", 2024-08-17T17:33:55.519 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 435.72279797300001, 2024-08-17T17:33:55.519 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.81279908999999995, 2024-08-17T17:33:55.519 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:55.519 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:55.519 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:55.519 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45491", 2024-08-17T17:33:55.519 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:33:55.519 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 130373 2024-08-17T17:33:55.519 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.519 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:55.519 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.520 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:55.520 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.667523+0000", 2024-08-17T17:33:55.520 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.520 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.520 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.520 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:55.520 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.667523+0000", 2024-08-17T17:33:55.520 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.520 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.520 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.520 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:55.520 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.667522+0000", 2024-08-17T17:33:55.520 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:33:55.520 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.520 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.521 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:55.521 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.667527+0000", 2024-08-17T17:33:55.521 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.109e-06 2024-08-17T17:33:55.521 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.521 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.521 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:55.521 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.667536+0000", 2024-08-17T17:33:55.521 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1870000000000003e-06 2024-08-17T17:33:55.521 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.521 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.521 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:55.521 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.667542+0000", 2024-08-17T17:33:55.521 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.601e-06 2024-08-17T17:33:55.521 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.521 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.522 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.522 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.667591+0000", 2024-08-17T17:33:55.522 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.8463e-05 2024-08-17T17:33:55.522 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.522 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.522 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:33:55.522 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:39.667600+0000", 2024-08-17T17:33:55.522 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.2520000000000005e-06 2024-08-17T17:33:55.522 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.522 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.522 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.522 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.454925+0000", 2024-08-17T17:33:55.523 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.78732473300000005 2024-08-17T17:33:55.523 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.523 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.523 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:55.523 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.455030+0000", 2024-08-17T17:33:55.523 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010560699999999999 2024-08-17T17:33:55.523 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.523 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.523 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,6", 2024-08-17T17:33:55.523 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.455139+0000", 2024-08-17T17:33:55.523 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000108921 2024-08-17T17:33:55.523 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.523 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.524 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.524 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.461260+0000", 2024-08-17T17:33:55.524 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0061209690000000004 2024-08-17T17:33:55.524 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.524 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.524 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:55.524 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.461395+0000", 2024-08-17T17:33:55.524 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013491600000000001 2024-08-17T17:33:55.524 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.524 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.524 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.524 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.480250+0000", 2024-08-17T17:33:55.524 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.018854427 2024-08-17T17:33:55.524 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.525 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.525 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:55.525 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.480297+0000", 2024-08-17T17:33:55.525 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7583999999999997e-05 2024-08-17T17:33:55.525 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.525 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.525 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:55.525 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:26:40.480322+0000", 2024-08-17T17:33:55.525 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4930999999999999e-05 2024-08-17T17:33:55.525 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.525 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:55.525 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.525 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.526 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.526 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45767.0:112283 7.8 7:1f09ca08:::benchmark_data_smithi003_201798_object112282:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-08-17T17:33:55.526 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:30:10.219446+0000", 2024-08-17T17:33:55.526 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 225.170875056, 2024-08-17T17:33:55.526 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.086479023000000002, 2024-08-17T17:33:55.526 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:55.526 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:55.526 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:55.526 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45767", 2024-08-17T17:33:55.526 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:33:55.526 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 112283 2024-08-17T17:33:55.526 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.527 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:55.527 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.527 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:55.527 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.219446+0000", 2024-08-17T17:33:55.527 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.527 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.527 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.527 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:55.527 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.219446+0000", 2024-08-17T17:33:55.527 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.527 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.527 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.528 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:55.528 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.219444+0000", 2024-08-17T17:33:55.528 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:33:55.528 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.528 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.528 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:55.528 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.219552+0000", 2024-08-17T17:33:55.528 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000107822 2024-08-17T17:33:55.528 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.528 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.528 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:55.529 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.219570+0000", 2024-08-17T17:33:55.529 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7708999999999999e-05 2024-08-17T17:33:55.529 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.529 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.529 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:55.529 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.219576+0000", 2024-08-17T17:33:55.529 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9649999999999996e-06 2024-08-17T17:33:55.529 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.529 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.529 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.529 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.219625+0000", 2024-08-17T17:33:55.530 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9193999999999997e-05 2024-08-17T17:33:55.530 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.530 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.530 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:55.530 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.219754+0000", 2024-08-17T17:33:55.530 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012925799999999999 2024-08-17T17:33:55.530 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.530 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.530 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,3", 2024-08-17T17:33:55.530 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.219840+0000", 2024-08-17T17:33:55.530 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.5409999999999997e-05 2024-08-17T17:33:55.531 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.531 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.531 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:55.531 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.222964+0000", 2024-08-17T17:33:55.531 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00312422 2024-08-17T17:33:55.531 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.531 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.531 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.531 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.225900+0000", 2024-08-17T17:33:55.531 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0029365490000000001 2024-08-17T17:33:55.531 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.532 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.532 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.532 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.305826+0000", 2024-08-17T17:33:55.532 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.079925062000000005 2024-08-17T17:33:55.532 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.532 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.532 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:55.532 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.305878+0000", 2024-08-17T17:33:55.532 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2543999999999997e-05 2024-08-17T17:33:55.532 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.532 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.532 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:55.532 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:10.305925+0000", 2024-08-17T17:33:55.533 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6898999999999999e-05 2024-08-17T17:33:55.533 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.533 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:55.533 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.533 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.533 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.533 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45767.0:130749 7.5 7:ab8bd974:::benchmark_data_smithi003_201798_object17527:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-08-17T17:33:55.533 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:30:33.586393+0000", 2024-08-17T17:33:55.533 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 201.80392839000001, 2024-08-17T17:33:55.533 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.067816888000000006, 2024-08-17T17:33:55.533 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:55.533 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:55.533 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:55.533 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45767", 2024-08-17T17:33:55.534 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:33:55.534 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 130749 2024-08-17T17:33:55.534 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.534 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:55.534 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.534 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:55.534 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:33.586393+0000", 2024-08-17T17:33:55.534 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.534 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.534 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.534 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:55.534 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:33.586393+0000", 2024-08-17T17:33:55.535 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.535 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.535 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.535 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:55.535 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:33.586392+0000", 2024-08-17T17:33:55.535 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:33:55.535 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.535 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.535 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:55.535 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:33.586395+0000", 2024-08-17T17:33:55.535 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9780000000000001e-06 2024-08-17T17:33:55.536 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.536 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.536 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:55.536 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:33.586410+0000", 2024-08-17T17:33:55.536 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5392000000000001e-05 2024-08-17T17:33:55.536 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.536 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.536 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:55.536 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:33.586415+0000", 2024-08-17T17:33:55.536 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3189999999999996e-06 2024-08-17T17:33:55.536 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.536 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.536 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.537 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:33.589537+0000", 2024-08-17T17:33:55.537 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003122484 2024-08-17T17:33:55.537 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.537 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.537 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:55.537 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:33.589616+0000", 2024-08-17T17:33:55.537 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.8892000000000003e-05 2024-08-17T17:33:55.537 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.537 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.537 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:33:55.537 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:33.589696+0000", 2024-08-17T17:33:55.537 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.9587999999999998e-05 2024-08-17T17:33:55.537 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.537 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.538 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:55.538 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:33.593176+0000", 2024-08-17T17:33:55.538 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0034807039999999998 2024-08-17T17:33:55.538 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.538 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.538 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.538 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:33.610208+0000", 2024-08-17T17:33:55.538 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017031121 2024-08-17T17:33:55.538 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.538 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.538 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.539 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:33.654130+0000", 2024-08-17T17:33:55.539 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.043922212000000002 2024-08-17T17:33:55.539 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.539 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.539 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:55.539 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:33.654188+0000", 2024-08-17T17:33:55.539 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7861e-05 2024-08-17T17:33:55.539 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.539 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.539 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:55.539 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:30:33.654209+0000", 2024-08-17T17:33:55.539 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1872999999999999e-05 2024-08-17T17:33:55.539 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.539 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:55.540 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.540 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.540 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.540 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45767.0:162891 7.5 7:afa861ca:::benchmark_data_smithi003_201798_object49669:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-08-17T17:33:55.540 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:31:14.596895+0000", 2024-08-17T17:33:55.540 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 160.79342597900001, 2024-08-17T17:33:55.540 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.12128480799999999, 2024-08-17T17:33:55.540 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:55.540 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:55.540 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:55.540 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45767", 2024-08-17T17:33:55.540 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:33:55.540 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 162891 2024-08-17T17:33:55.541 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.541 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:55.541 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.541 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:55.541 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:14.596895+0000", 2024-08-17T17:33:55.541 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.541 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.541 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.541 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:55.541 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:14.596895+0000", 2024-08-17T17:33:55.541 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.541 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.541 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.541 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:55.542 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:14.596894+0000", 2024-08-17T17:33:55.542 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:33:55.542 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.542 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.542 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:55.542 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:14.596898+0000", 2024-08-17T17:33:55.542 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8650000000000003e-06 2024-08-17T17:33:55.542 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.542 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.542 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:55.542 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:14.596909+0000", 2024-08-17T17:33:55.543 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1146e-05 2024-08-17T17:33:55.543 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.543 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.543 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:55.543 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:14.596913+0000", 2024-08-17T17:33:55.543 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.281e-06 2024-08-17T17:33:55.543 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.543 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.543 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.543 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:14.596964+0000", 2024-08-17T17:33:55.543 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1523000000000001e-05 2024-08-17T17:33:55.543 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.544 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.544 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:55.544 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:14.597079+0000", 2024-08-17T17:33:55.544 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000115087 2024-08-17T17:33:55.544 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.544 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.544 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:33:55.544 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:14.597185+0000", 2024-08-17T17:33:55.544 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000106178 2024-08-17T17:33:55.544 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.544 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.544 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:55.545 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:14.600042+0000", 2024-08-17T17:33:55.545 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0028569569999999998 2024-08-17T17:33:55.545 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.545 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.545 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.545 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:14.601105+0000", 2024-08-17T17:33:55.545 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001062976 2024-08-17T17:33:55.545 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.545 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.545 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.545 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:14.718081+0000", 2024-08-17T17:33:55.545 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.116975727 2024-08-17T17:33:55.545 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.545 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.545 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:55.546 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:14.718125+0000", 2024-08-17T17:33:55.546 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4295000000000002e-05 2024-08-17T17:33:55.546 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.546 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.546 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:55.546 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:31:14.718180+0000", 2024-08-17T17:33:55.546 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4367000000000003e-05 2024-08-17T17:33:55.546 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.546 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:55.546 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.546 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.546 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.546 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45767.0:211181 7.5 7:a6977e40:::benchmark_data_smithi003_201798_object97959:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-08-17T17:33:55.547 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:32:09.558141+0000", 2024-08-17T17:33:55.547 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 105.832179693, 2024-08-17T17:33:55.547 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.090496081000000006, 2024-08-17T17:33:55.547 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:55.547 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:55.547 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:55.547 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45767", 2024-08-17T17:33:55.547 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:33:55.547 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 211181 2024-08-17T17:33:55.547 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.547 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:55.547 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.547 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:55.547 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.558141+0000", 2024-08-17T17:33:55.548 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.548 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.548 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.548 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:55.548 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.558141+0000", 2024-08-17T17:33:55.548 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.548 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.548 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.548 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:55.548 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.558140+0000", 2024-08-17T17:33:55.548 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:33:55.548 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.548 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.548 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:55.549 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.558147+0000", 2024-08-17T17:33:55.549 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7109999999999995e-06 2024-08-17T17:33:55.549 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.549 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.549 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:55.549 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.558162+0000", 2024-08-17T17:33:55.549 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5142e-05 2024-08-17T17:33:55.549 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.549 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.549 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:55.549 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.558169+0000", 2024-08-17T17:33:55.549 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4459999999999998e-06 2024-08-17T17:33:55.549 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.550 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.550 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.550 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.558250+0000", 2024-08-17T17:33:55.550 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1448999999999999e-05 2024-08-17T17:33:55.550 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.550 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.550 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:55.550 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.559822+0000", 2024-08-17T17:33:55.550 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0015721940000000001 2024-08-17T17:33:55.550 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.550 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.551 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:33:55.551 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.559957+0000", 2024-08-17T17:33:55.551 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013464799999999999 2024-08-17T17:33:55.551 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.551 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.551 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:55.551 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.563689+0000", 2024-08-17T17:33:55.551 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0037314179999999998 2024-08-17T17:33:55.551 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.551 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.551 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.551 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.565043+0000", 2024-08-17T17:33:55.551 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001354942 2024-08-17T17:33:55.551 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.551 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.552 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.552 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.648566+0000", 2024-08-17T17:33:55.552 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.083522679000000002 2024-08-17T17:33:55.552 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.552 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.552 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:55.552 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.648598+0000", 2024-08-17T17:33:55.552 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1579999999999999e-05 2024-08-17T17:33:55.552 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.552 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.552 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:55.552 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.648637+0000", 2024-08-17T17:33:55.552 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9641000000000002e-05 2024-08-17T17:33:55.553 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.553 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:55.553 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.553 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.553 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.553 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45767.0:211194 7.5 7:a57e8e5c:::benchmark_data_smithi003_201798_object97972:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-08-17T17:33:55.553 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:32:09.572297+0000", 2024-08-17T17:33:55.553 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 105.818023822, 2024-08-17T17:33:55.553 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.089342749999999999, 2024-08-17T17:33:55.553 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:55.553 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:55.553 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:55.553 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45767", 2024-08-17T17:33:55.554 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:33:55.554 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 211194 2024-08-17T17:33:55.554 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.554 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:55.554 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.554 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:55.554 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.572297+0000", 2024-08-17T17:33:55.554 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.554 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.554 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.554 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:55.554 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.572297+0000", 2024-08-17T17:33:55.555 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.555 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.555 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.555 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:55.555 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.572296+0000", 2024-08-17T17:33:55.555 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:33:55.555 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.555 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.555 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:55.555 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.572305+0000", 2024-08-17T17:33:55.555 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.0310000000000004e-06 2024-08-17T17:33:55.555 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.555 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.555 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:55.556 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.572317+0000", 2024-08-17T17:33:55.556 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2130000000000001e-05 2024-08-17T17:33:55.556 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.556 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.556 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:55.556 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.572322+0000", 2024-08-17T17:33:55.556 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3070000000000002e-06 2024-08-17T17:33:55.556 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.556 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.556 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.556 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.572364+0000", 2024-08-17T17:33:55.556 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1390999999999997e-05 2024-08-17T17:33:55.556 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.556 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.557 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:55.557 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.572459+0000", 2024-08-17T17:33:55.557 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5276000000000001e-05 2024-08-17T17:33:55.557 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.557 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.557 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:33:55.557 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.572529+0000", 2024-08-17T17:33:55.557 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9752999999999994e-05 2024-08-17T17:33:55.557 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.557 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.557 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:55.557 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.577379+0000", 2024-08-17T17:33:55.557 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0048502959999999996 2024-08-17T17:33:55.557 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.558 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.558 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.558 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.577413+0000", 2024-08-17T17:33:55.558 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3637000000000003e-05 2024-08-17T17:33:55.558 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.558 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.558 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.558 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.661547+0000", 2024-08-17T17:33:55.558 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.084134833000000006 2024-08-17T17:33:55.558 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.558 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.558 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:55.559 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.661598+0000", 2024-08-17T17:33:55.559 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0763999999999997e-05 2024-08-17T17:33:55.559 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.559 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.559 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:55.559 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.661640+0000", 2024-08-17T17:33:55.559 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1674000000000001e-05 2024-08-17T17:33:55.559 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.559 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:55.559 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.559 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.559 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.559 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.45767.0:211195 7.5 7:ad73cf22:::benchmark_data_smithi003_201798_object97973:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-08-17T17:33:55.560 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:32:09.572338+0000", 2024-08-17T17:33:55.560 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 105.817983126, 2024-08-17T17:33:55.560 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.089393948000000001, 2024-08-17T17:33:55.560 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:33:55.560 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:33:55.560 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:33:55.560 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.45767", 2024-08-17T17:33:55.560 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:33:55.560 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 211195 2024-08-17T17:33:55.560 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.560 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:33:55.560 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.560 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:33:55.561 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.572338+0000", 2024-08-17T17:33:55.561 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.561 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.561 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.561 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:33:55.561 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.572338+0000", 2024-08-17T17:33:55.561 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:33:55.561 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.561 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.561 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:33:55.561 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.572337+0000", 2024-08-17T17:33:55.561 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:33:55.561 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.561 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.562 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:33:55.562 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.572340+0000", 2024-08-17T17:33:55.562 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.48e-06 2024-08-17T17:33:55.562 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.562 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.562 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:33:55.562 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.572344+0000", 2024-08-17T17:33:55.562 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5199999999999999e-06 2024-08-17T17:33:55.562 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.562 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.562 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:33:55.562 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.572347+0000", 2024-08-17T17:33:55.562 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.886e-06 2024-08-17T17:33:55.563 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.563 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.563 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:33:55.563 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.575389+0000", 2024-08-17T17:33:55.563 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0030414729999999998 2024-08-17T17:33:55.563 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.563 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.563 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:33:55.563 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.575466+0000", 2024-08-17T17:33:55.563 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.6814e-05 2024-08-17T17:33:55.563 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.563 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.563 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,2", 2024-08-17T17:33:55.563 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.575532+0000", 2024-08-17T17:33:55.564 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.6657999999999998e-05 2024-08-17T17:33:55.564 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.564 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.564 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.564 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.577447+0000", 2024-08-17T17:33:55.564 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001914551 2024-08-17T17:33:55.564 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.564 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.564 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:33:55.564 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.577481+0000", 2024-08-17T17:33:55.564 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.464e-05 2024-08-17T17:33:55.564 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.564 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.564 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:33:55.564 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.661676+0000", 2024-08-17T17:33:55.564 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.084194743000000002 2024-08-17T17:33:55.565 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.565 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.565 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:33:55.565 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.661709+0000", 2024-08-17T17:33:55.565 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2765000000000002e-05 2024-08-17T17:33:55.565 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:33:55.565 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:33:55.565 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:33:55.565 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:32:09.661732+0000", 2024-08-17T17:33:55.565 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2819999999999998e-05 2024-08-17T17:33:55.565 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.565 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:55.565 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.565 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:33:55.566 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:33:55.566 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:33:55.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:55.567 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:55.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:55.568 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:55 smithi003 ceph-mon[103344]: pgmap v825: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 51 MiB/s wr, 817 op/s; 24148/249297 objects degraded (9.686%); 2.1 MiB/s, 32 objects/s recovering 2024-08-17T17:33:55.568 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:55 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 24148/249297 objects degraded (9.686%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:33:55.568 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:55 smithi003 ceph-mon[100615]: pgmap v825: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 51 MiB/s wr, 817 op/s; 24148/249297 objects degraded (9.686%); 2.1 MiB/s, 32 objects/s recovering 2024-08-17T17:33:55.568 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:55 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 24148/249297 objects degraded (9.686%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:33:55.669 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:55.698 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:55.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:55.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:55.800 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:55.829 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:55.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:55.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:55.891 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:33:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:33:55] "GET /metrics HTTP/1.1" 200 36530 "" "Prometheus/2.51.0" 2024-08-17T17:33:55.931 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:55.959 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:55.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:55.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:56.008 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:33:56.061 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:56.087 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:56.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:56.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:56.188 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:56.213 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:56.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:56.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:56.316 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:56.329 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:33:56.331 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:33:56.332 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-17T17:33:56.345 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:56.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:56.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:56.374 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-08-17T17:33:56.374216+0000 min lat: 0.00109625 max lat: 2.31183 avg lat: 0.0112003 2024-08-17T17:33:56.374 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:33:56.374 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 60 16 85687 85671 89.2229 46.125 0.016967 0.0112003 2024-08-17T17:33:56.447 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:56.479 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:56.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:56.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:56.581 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:56.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:56.643 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:56.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:56.744 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:56.779 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:56.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:56.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:56.881 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:56.903 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:56.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:56.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:57.005 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:57.052 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:57.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:57.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:57.154 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:57.180 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:57.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:57.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:57.282 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:57.308 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:57.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:57.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:57.353 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:33:57.374 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 61 16 86291 86275 88.3791 37.75 0.0544212 0.0113079 2024-08-17T17:33:57.409 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:57.458 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:57.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:57.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:57.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:57 smithi053 ceph-mon[89963]: pgmap v826: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 51 MiB/s wr, 818 op/s; 24148/250821 objects degraded (9.628%); 2.0 MiB/s, 32 objects/s recovering 2024-08-17T17:33:57.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:57 smithi053 ceph-mon[89963]: from='client.36129 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:33:57.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:57 smithi003 ceph-mon[100615]: pgmap v826: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 51 MiB/s wr, 818 op/s; 24148/250821 objects degraded (9.628%); 2.0 MiB/s, 32 objects/s recovering 2024-08-17T17:33:57.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:57 smithi003 ceph-mon[100615]: from='client.36129 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:33:57.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:57 smithi003 ceph-mon[103344]: pgmap v826: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 51 MiB/s wr, 818 op/s; 24148/250821 objects degraded (9.628%); 2.0 MiB/s, 32 objects/s recovering 2024-08-17T17:33:57.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:57 smithi003 ceph-mon[103344]: from='client.36129 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:33:57.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:57.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:57.608 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:57.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:57.709 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:57.731 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:57.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:57.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:57.833 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:57.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:57.877 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:57.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:57.978 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:58.018 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:58.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:58.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:58.120 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:58.160 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:58.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:58.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:58.262 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:33:58.292 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:58.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:58.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:33:58.374 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 62 16 86977 86961 87.6452 42.875 0.0144351 0.0114028 2024-08-17T17:33:58.394 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:33:58.425 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:58.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:58.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:33:58.527 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:58.572 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:58.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:58.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:33:58.674 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:58.708 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:58.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:58.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:58.810 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:33:58.838 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:58.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:58.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:33:58.939 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:58.968 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:58.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:58.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:59.070 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:59.112 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:59.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:59.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:59.214 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:59.241 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:59.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:59.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:59.342 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:33:59.374 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 63 16 87755 87739 87.0258 48.625 0.00968773 0.0114848 2024-08-17T17:33:59.378 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:59.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:59.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:33:59.480 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:33:59.486 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:33:59 smithi003 ceph-mon[100615]: pgmap v827: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 50 MiB/s wr, 795 op/s; 24051/254223 objects degraded (9.461%); 1.4 MiB/s, 21 objects/s recovering 2024-08-17T17:33:59.486 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:33:59 smithi003 ceph-mon[103344]: pgmap v827: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 50 MiB/s wr, 795 op/s; 24051/254223 objects degraded (9.461%); 1.4 MiB/s, 21 objects/s recovering 2024-08-17T17:33:59.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:59.536 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:59.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:33:59.539 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":826,"stamp":"2024-08-17T17:33:56.082655+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5482223565,"num_objects":83607,"num_object_clones":0,"num_object_copies":250821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83607,"num_whiteouts":0,"num_read":6356,"num_read_kb":13813,"num_write":84701,"num_write_kb":5364095,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":928,"num_bytes_recovered":64234186,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84760,"ondisk_log_size":84760,"up":267,"acting":260,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":18049672,"kb_used_data":14517896,"kb_used_omap":37,"kb_used_meta":3531674,"kb_avail":731911544,"statfs":{"total":767960285184,"available":749477421056,"internally_reserved":0,"allocated":14866325504,"data_stored":14854095791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38729,"internal_metadata":3616434359},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":536543232,"num_objects":8187,"num_object_clones":0,"num_object_copies":24561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-328,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8187,"num_write_kb":523968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":21495808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.006694"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":416,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.213331+0000","last_change":"2024-08-17T17:33:20.797029+0000","last_active":"2024-08-17T17:33:50.213331+0000","last_peered":"2024-08-17T17:33:50.213331+0000","last_clean":"2024-08-17T17:33:50.213331+0000","last_became_active":"2024-08-17T17:33:20.796679+0000","last_became_peered":"2024-08-17T17:33:20.796679+0000","last_unstale":"2024-08-17T17:33:50.213331+0000","last_undegraded":"2024-08-17T17:33:50.213331+0000","last_fullsized":"2024-08-17T17:33:50.213331+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:51.438262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":250,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-08-17T17:33:44.139945+0000","last_change":"2024-08-17T17:18:59.424887+0000","last_active":"2024-08-17T17:33:44.139945+0000","last_peered":"2024-08-17T17:33:44.139945+0000","last_clean":"2024-08-17T17:33:44.139945+0000","last_became_active":"2024-08-17T17:18:59.411539+0000","last_became_peered":"2024-08-17T17:18:59.411539+0000","last_unstale":"2024-08-17T17:33:44.139945+0000","last_undegraded":"2024-08-17T17:33:44.139945+0000","last_fullsized":"2024-08-17T17:33:44.139945+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:46:20.082241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":77,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.212279+0000","last_change":"2024-08-17T17:33:44.156036+0000","last_active":"2024-08-17T17:33:50.212279+0000","last_peered":"2024-08-17T17:33:50.212279+0000","last_clean":"2024-08-17T17:33:50.212279+0000","last_became_active":"2024-08-17T17:33:44.155758+0000","last_became_peered":"2024-08-17T17:33:44.155758+0000","last_unstale":"2024-08-17T17:33:50.212279+0000","last_undegraded":"2024-08-17T17:33:50.212279+0000","last_fullsized":"2024-08-17T17:33:50.212279+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:23.443405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":216,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.196509+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:33:50.196509+0000","last_peered":"2024-08-17T17:33:50.196509+0000","last_clean":"2024-08-17T17:33:50.196509+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:33:50.196509+0000","last_undegraded":"2024-08-17T17:33:50.196509+0000","last_fullsized":"2024-08-17T17:33:50.196509+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":216,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.197622+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:33:50.197622+0000","last_peered":"2024-08-17T17:33:50.197622+0000","last_clean":"2024-08-17T17:33:50.197622+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:33:50.197622+0000","last_undegraded":"2024-08-17T17:33:50.197622+0000","last_fullsized":"2024-08-17T17:33:50.197622+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":23,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-08-17T17:33:44.142465+0000","last_change":"2024-08-17T17:32:46.377651+0000","last_active":"2024-08-17T17:33:44.142465+0000","last_peered":"2024-08-17T17:33:44.142465+0000","last_clean":"2024-08-17T17:33:44.142465+0000","last_became_active":"2024-08-17T17:32:46.377282+0000","last_became_peered":"2024-08-17T17:32:46.377282+0000","last_unstale":"2024-08-17T17:33:44.142465+0000","last_undegraded":"2024-08-17T17:33:44.142465+0000","last_fullsized":"2024-08-17T17:33:44.142465+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:45:27.370287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":429,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.221338+0000","last_change":"2024-08-17T17:33:20.794682+0000","last_active":"2024-08-17T17:33:50.221338+0000","last_peered":"2024-08-17T17:33:50.221338+0000","last_clean":"2024-08-17T17:33:50.221338+0000","last_became_active":"2024-08-17T17:33:20.794238+0000","last_became_peered":"2024-08-17T17:33:20.794238+0000","last_unstale":"2024-08-17T17:33:50.221338+0000","last_undegraded":"2024-08-17T17:33:50.221338+0000","last_fullsized":"2024-08-17T17:33:50.221338+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:15:55.821920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":320,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-08-17T17:33:44.140953+0000","last_change":"2024-08-17T17:24:53.465875+0000","last_active":"2024-08-17T17:33:44.140953+0000","last_peered":"2024-08-17T17:33:44.140953+0000","last_clean":"2024-08-17T17:33:44.140953+0000","last_became_active":"2024-08-17T17:24:53.458129+0000","last_became_peered":"2024-08-17T17:24:53.458129+0000","last_unstale":"2024-08-17T17:33:44.140953+0000","last_undegraded":"2024-08-17T17:33:44.140953+0000","last_fullsized":"2024-08-17T17:33:44.140953+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:51:33.584869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"274'5198","reported_seq":5224,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:53.712686+0000","last_change":"2024-08-17T17:33:44.165390+0000","last_active":"2024-08-17T17:33:53.712686+0000","last_peered":"2024-08-17T17:33:53.712686+0000","last_clean":"2024-08-17T17:33:53.712686+0000","last_became_active":"2024-08-17T17:33:44.144942+0000","last_became_peered":"2024-08-17T17:33:44.144942+0000","last_unstale":"2024-08-17T17:33:53.712686+0000","last_undegraded":"2024-08-17T17:33:53.712686+0000","last_fullsized":"2024-08-17T17:33:53.712686+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":5198,"log_dups_size":0,"ondisk_log_size":5198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:49:11.850677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340656128,"num_objects":5198,"num_object_clones":0,"num_object_copies":15594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5198,"num_write_kb":332672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":207,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-08-17T17:33:44.139784+0000","last_change":"2024-08-17T17:32:46.384376+0000","last_active":"2024-08-17T17:33:44.139784+0000","last_peered":"2024-08-17T17:33:44.139784+0000","last_clean":"2024-08-17T17:33:44.139784+0000","last_became_active":"2024-08-17T17:32:46.374177+0000","last_became_peered":"2024-08-17T17:32:46.374177+0000","last_unstale":"2024-08-17T17:33:44.139784+0000","last_undegraded":"2024-08-17T17:33:44.139784+0000","last_fullsized":"2024-08-17T17:33:44.139784+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:51:58.880388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":29,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-08-17T17:33:44.145974+0000","last_change":"2024-08-17T17:33:44.145598+0000","last_active":"2024-08-17T17:33:44.145974+0000","last_peered":"2024-08-17T17:33:44.145974+0000","last_clean":"2024-08-17T17:33:44.145974+0000","last_became_active":"2024-08-17T17:33:44.145283+0000","last_became_peered":"2024-08-17T17:33:44.145283+0000","last_unstale":"2024-08-17T17:33:44.145974+0000","last_undegraded":"2024-08-17T17:33:44.145974+0000","last_fullsized":"2024-08-17T17:33:44.145974+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:11.965226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"274'5205","reported_seq":5225,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:54.064519+0000","last_change":"2024-08-17T17:32:48.380789+0000","last_active":"2024-08-17T17:33:54.064519+0000","last_peered":"2024-08-17T17:33:54.064519+0000","last_clean":"2024-08-17T17:33:54.064519+0000","last_became_active":"2024-08-17T17:32:48.380173+0000","last_became_peered":"2024-08-17T17:32:48.380173+0000","last_unstale":"2024-08-17T17:33:54.064519+0000","last_undegraded":"2024-08-17T17:33:54.064519+0000","last_fullsized":"2024-08-17T17:33:54.064519+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":5205,"log_dups_size":0,"ondisk_log_size":5205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:25:18.190497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341114880,"num_objects":5205,"num_object_clones":0,"num_object_copies":15615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5205,"num_write_kb":333120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":189,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-08-17T17:33:44.139803+0000","last_change":"2024-08-17T17:33:20.834638+0000","last_active":"2024-08-17T17:33:44.139803+0000","last_peered":"2024-08-17T17:33:44.139803+0000","last_clean":"2024-08-17T17:33:44.139803+0000","last_became_active":"2024-08-17T17:33:20.834262+0000","last_became_peered":"2024-08-17T17:33:20.834262+0000","last_unstale":"2024-08-17T17:33:44.139803+0000","last_undegraded":"2024-08-17T17:33:44.139803+0000","last_fullsized":"2024-08-17T17:33:44.139803+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:43:06.894393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":37,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-08-17T17:33:44.142547+0000","last_change":"2024-08-17T17:33:21.766549+0000","last_active":"2024-08-17T17:33:44.142547+0000","last_peered":"2024-08-17T17:33:44.142547+0000","last_clean":"2024-08-17T17:33:44.142547+0000","last_became_active":"2024-08-17T17:33:21.766152+0000","last_became_peered":"2024-08-17T17:33:21.766152+0000","last_unstale":"2024-08-17T17:33:44.142547+0000","last_undegraded":"2024-08-17T17:33:44.142547+0000","last_fullsized":"2024-08-17T17:33:44.142547+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:56:15.855524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"274'5167","reported_seq":5202,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:53.707408+0000","last_change":"2024-08-17T17:32:48.381458+0000","last_active":"2024-08-17T17:33:53.707408+0000","last_peered":"2024-08-17T17:33:53.707408+0000","last_clean":"2024-08-17T17:33:53.707408+0000","last_became_active":"2024-08-17T17:32:48.381201+0000","last_became_peered":"2024-08-17T17:32:48.381201+0000","last_unstale":"2024-08-17T17:33:53.707408+0000","last_undegraded":"2024-08-17T17:33:53.707408+0000","last_fullsized":"2024-08-17T17:33:53.707408+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":5167,"log_dups_size":0,"ondisk_log_size":5167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:12:18.200526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338624512,"num_objects":5167,"num_object_clones":0,"num_object_copies":15501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5167,"num_write_kb":330688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":25,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-08-17T17:33:44.146870+0000","last_change":"2024-08-17T17:33:44.145399+0000","last_active":"2024-08-17T17:33:44.146870+0000","last_peered":"2024-08-17T17:33:44.146870+0000","last_clean":"2024-08-17T17:33:44.146870+0000","last_became_active":"2024-08-17T17:33:44.144751+0000","last_became_peered":"2024-08-17T17:33:44.144751+0000","last_unstale":"2024-08-17T17:33:44.146870+0000","last_undegraded":"2024-08-17T17:33:44.146870+0000","last_fullsized":"2024-08-17T17:33:44.146870+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:54.276350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":103,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-08-17T17:33:44.140614+0000","last_change":"2024-08-17T17:33:20.795582+0000","last_active":"2024-08-17T17:33:44.140614+0000","last_peered":"2024-08-17T17:33:44.140614+0000","last_clean":"2024-08-17T17:33:44.140614+0000","last_became_active":"2024-08-17T17:33:20.795149+0000","last_became_peered":"2024-08-17T17:33:20.795149+0000","last_unstale":"2024-08-17T17:33:44.140614+0000","last_undegraded":"2024-08-17T17:33:44.140614+0000","last_fullsized":"2024-08-17T17:33:44.140614+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:14.020923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"274'5176","reported_seq":5210,"reported_epoch":274,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:33:52.886871+0000","last_change":"2024-08-17T17:33:22.090332+0000","last_active":"2024-08-17T17:33:52.886871+0000","last_peered":"2024-08-17T17:33:52.886871+0000","last_clean":"2024-08-17T17:32:47.362949+0000","last_became_active":"2024-08-17T17:33:22.080130+0000","last_became_peered":"2024-08-17T17:33:22.080130+0000","last_unstale":"2024-08-17T17:33:52.886871+0000","last_undegraded":"2024-08-17T17:33:21.923200+0000","last_fullsized":"2024-08-17T17:33:21.906251+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":5176,"log_dups_size":0,"ondisk_log_size":5176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339214336,"num_objects":5176,"num_object_clones":0,"num_object_copies":15528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5176,"num_write_kb":331264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":1656}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":74,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.213483+0000","last_change":"2024-08-17T17:33:44.154097+0000","last_active":"2024-08-17T17:33:50.213483+0000","last_peered":"2024-08-17T17:33:50.213483+0000","last_clean":"2024-08-17T17:33:50.213483+0000","last_became_active":"2024-08-17T17:33:44.152972+0000","last_became_peered":"2024-08-17T17:33:44.152972+0000","last_unstale":"2024-08-17T17:33:50.213483+0000","last_undegraded":"2024-08-17T17:33:50.213483+0000","last_fullsized":"2024-08-17T17:33:50.213483+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:42.602017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":424,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.196968+0000","last_change":"2024-08-17T17:33:21.763677+0000","last_active":"2024-08-17T17:33:50.196968+0000","last_peered":"2024-08-17T17:33:50.196968+0000","last_clean":"2024-08-17T17:33:50.196968+0000","last_became_active":"2024-08-17T17:33:21.760564+0000","last_became_peered":"2024-08-17T17:33:21.760564+0000","last_unstale":"2024-08-17T17:33:50.196968+0000","last_undegraded":"2024-08-17T17:33:50.196968+0000","last_fullsized":"2024-08-17T17:33:50.196968+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:01:00.772037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.0","version":"274'5158","reported_seq":5191,"reported_epoch":274,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:33:52.869387+0000","last_change":"2024-08-17T17:33:22.010282+0000","last_active":"2024-08-17T17:33:52.869387+0000","last_peered":"2024-08-17T17:33:52.869387+0000","last_clean":"2024-08-17T17:32:47.362949+0000","last_became_active":"2024-08-17T17:33:21.998714+0000","last_became_peered":"2024-08-17T17:33:21.998714+0000","last_unstale":"2024-08-17T17:33:52.869387+0000","last_undegraded":"2024-08-17T17:33:21.807105+0000","last_fullsized":"2024-08-17T17:33:21.761969+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":5158,"log_dups_size":0,"ondisk_log_size":5158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338034688,"num_objects":5158,"num_object_clones":0,"num_object_copies":15474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3554,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5158,"num_write_kb":330112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[{"shards":"3,5","objects":1604}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":451,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.213960+0000","last_change":"2024-08-17T17:32:46.387810+0000","last_active":"2024-08-17T17:33:50.213960+0000","last_peered":"2024-08-17T17:33:50.213960+0000","last_clean":"2024-08-17T17:33:50.213960+0000","last_became_active":"2024-08-17T17:32:46.374238+0000","last_became_peered":"2024-08-17T17:32:46.374238+0000","last_unstale":"2024-08-17T17:33:50.213960+0000","last_undegraded":"2024-08-17T17:33:50.213960+0000","last_fullsized":"2024-08-17T17:33:50.213960+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:53:29.918005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":23,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-08-17T17:33:44.143419+0000","last_change":"2024-08-17T17:32:46.378495+0000","last_active":"2024-08-17T17:33:44.143419+0000","last_peered":"2024-08-17T17:33:44.143419+0000","last_clean":"2024-08-17T17:33:44.143419+0000","last_became_active":"2024-08-17T17:32:46.377709+0000","last_became_peered":"2024-08-17T17:32:46.377709+0000","last_unstale":"2024-08-17T17:33:44.143419+0000","last_undegraded":"2024-08-17T17:33:44.143419+0000","last_fullsized":"2024-08-17T17:33:44.143419+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:22:52.544851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"274'5305","reported_seq":5339,"reported_epoch":274,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:33:53.743133+0000","last_change":"2024-08-17T17:33:21.986278+0000","last_active":"2024-08-17T17:33:53.743133+0000","last_peered":"2024-08-17T17:33:53.743133+0000","last_clean":"2024-08-17T17:32:47.362949+0000","last_became_active":"2024-08-17T17:33:21.976559+0000","last_became_peered":"2024-08-17T17:33:21.976559+0000","last_unstale":"2024-08-17T17:33:53.743133+0000","last_undegraded":"2024-08-17T17:33:21.786755+0000","last_fullsized":"2024-08-17T17:33:21.754986+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":5305,"log_dups_size":0,"ondisk_log_size":5305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347668480,"num_objects":5305,"num_object_clones":0,"num_object_copies":15915,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3606,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5305,"num_write_kb":339520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":1699}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":37,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-08-17T17:33:44.142623+0000","last_change":"2024-08-17T17:33:21.766756+0000","last_active":"2024-08-17T17:33:44.142623+0000","last_peered":"2024-08-17T17:33:44.142623+0000","last_clean":"2024-08-17T17:33:44.142623+0000","last_became_active":"2024-08-17T17:33:21.766347+0000","last_became_peered":"2024-08-17T17:33:21.766347+0000","last_unstale":"2024-08-17T17:33:44.142623+0000","last_undegraded":"2024-08-17T17:33:44.142623+0000","last_fullsized":"2024-08-17T17:33:44.142623+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:13:14.655176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":137,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-08-17T17:33:44.141060+0000","last_change":"2024-08-17T17:32:25.140415+0000","last_active":"2024-08-17T17:33:44.141060+0000","last_peered":"2024-08-17T17:33:44.141060+0000","last_clean":"2024-08-17T17:33:44.141060+0000","last_became_active":"2024-08-17T17:32:25.140060+0000","last_became_peered":"2024-08-17T17:32:25.140060+0000","last_unstale":"2024-08-17T17:33:44.141060+0000","last_undegraded":"2024-08-17T17:33:44.141060+0000","last_fullsized":"2024-08-17T17:33:44.141060+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:14:52.520189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2","version":"274'5256","reported_seq":5275,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:53.684551+0000","last_change":"2024-08-17T17:32:48.378488+0000","last_active":"2024-08-17T17:33:53.684551+0000","last_peered":"2024-08-17T17:33:53.684551+0000","last_clean":"2024-08-17T17:33:53.684551+0000","last_became_active":"2024-08-17T17:32:48.378189+0000","last_became_peered":"2024-08-17T17:32:48.378189+0000","last_unstale":"2024-08-17T17:33:53.684551+0000","last_undegraded":"2024-08-17T17:33:53.684551+0000","last_fullsized":"2024-08-17T17:33:53.684551+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":5256,"log_dups_size":0,"ondisk_log_size":5256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:38.041536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344457216,"num_objects":5256,"num_object_clones":0,"num_object_copies":15768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5256,"num_write_kb":336384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":224,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-08-17T17:33:44.139215+0000","last_change":"2024-08-17T17:33:20.794217+0000","last_active":"2024-08-17T17:33:44.139215+0000","last_peered":"2024-08-17T17:33:44.139215+0000","last_clean":"2024-08-17T17:33:44.139215+0000","last_became_active":"2024-08-17T17:33:20.793899+0000","last_became_peered":"2024-08-17T17:33:20.793899+0000","last_unstale":"2024-08-17T17:33:44.139215+0000","last_undegraded":"2024-08-17T17:33:44.139215+0000","last_fullsized":"2024-08-17T17:33:44.139215+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:01:31.197983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":37,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-08-17T17:33:44.143809+0000","last_change":"2024-08-17T17:33:21.764934+0000","last_active":"2024-08-17T17:33:44.143809+0000","last_peered":"2024-08-17T17:33:44.143809+0000","last_clean":"2024-08-17T17:33:44.143809+0000","last_became_active":"2024-08-17T17:33:21.764341+0000","last_became_peered":"2024-08-17T17:33:21.764341+0000","last_unstale":"2024-08-17T17:33:44.143809+0000","last_undegraded":"2024-08-17T17:33:44.143809+0000","last_fullsized":"2024-08-17T17:33:44.143809+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:20:25.714104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"274'5188","reported_seq":5211,"reported_epoch":274,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:33:53.956609+0000","last_change":"2024-08-17T17:33:21.990474+0000","last_active":"2024-08-17T17:33:53.956609+0000","last_peered":"2024-08-17T17:33:53.956609+0000","last_clean":"2024-08-17T17:32:47.362949+0000","last_became_active":"2024-08-17T17:33:21.980477+0000","last_became_peered":"2024-08-17T17:33:21.980477+0000","last_unstale":"2024-08-17T17:33:53.956609+0000","last_undegraded":"2024-08-17T17:33:21.778308+0000","last_fullsized":"2024-08-17T17:33:21.762700+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":5188,"log_dups_size":0,"ondisk_log_size":5188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:31:39.084953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340000768,"num_objects":5188,"num_object_clones":0,"num_object_copies":15564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5188,"num_write_kb":332032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":1730}],"blocked_by":[],"up_primary":1,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":431,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.216513+0000","last_change":"2024-08-17T17:33:20.796216+0000","last_active":"2024-08-17T17:33:50.216513+0000","last_peered":"2024-08-17T17:33:50.216513+0000","last_clean":"2024-08-17T17:33:50.216513+0000","last_became_active":"2024-08-17T17:33:20.795762+0000","last_became_peered":"2024-08-17T17:33:20.795762+0000","last_unstale":"2024-08-17T17:33:50.216513+0000","last_undegraded":"2024-08-17T17:33:50.216513+0000","last_fullsized":"2024-08-17T17:33:50.216513+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:47.038509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":313,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.241122+0000","last_change":"2024-08-17T17:33:21.976938+0000","last_active":"2024-08-17T17:33:50.241122+0000","last_peered":"2024-08-17T17:33:50.241122+0000","last_clean":"2024-08-17T17:33:50.241122+0000","last_became_active":"2024-08-17T17:33:21.976585+0000","last_became_peered":"2024-08-17T17:33:21.976585+0000","last_unstale":"2024-08-17T17:33:50.241122+0000","last_undegraded":"2024-08-17T17:33:50.241122+0000","last_fullsized":"2024-08-17T17:33:50.241122+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:23:40.758870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"274'5298","reported_seq":5318,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:54.043738+0000","last_change":"2024-08-17T17:32:48.378105+0000","last_active":"2024-08-17T17:33:54.043738+0000","last_peered":"2024-08-17T17:33:54.043738+0000","last_clean":"2024-08-17T17:33:54.043738+0000","last_became_active":"2024-08-17T17:32:48.377534+0000","last_became_peered":"2024-08-17T17:32:48.377534+0000","last_unstale":"2024-08-17T17:33:54.043738+0000","last_undegraded":"2024-08-17T17:33:54.043738+0000","last_fullsized":"2024-08-17T17:33:54.043738+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":5298,"log_dups_size":0,"ondisk_log_size":5298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:53.372506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347209728,"num_objects":5298,"num_object_clones":0,"num_object_copies":15894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5298,"num_write_kb":339072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":528,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.195510+0000","last_change":"2024-08-17T17:32:25.139748+0000","last_active":"2024-08-17T17:33:50.195510+0000","last_peered":"2024-08-17T17:33:50.195510+0000","last_clean":"2024-08-17T17:33:50.195510+0000","last_became_active":"2024-08-17T17:32:25.139295+0000","last_became_peered":"2024-08-17T17:32:25.139295+0000","last_unstale":"2024-08-17T17:33:50.195510+0000","last_undegraded":"2024-08-17T17:33:50.195510+0000","last_fullsized":"2024-08-17T17:33:50.195510+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:04:01.141401+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:35:48.531703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":680,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-08-17T17:33:44.150771+0000","last_change":"2024-08-17T17:33:44.149928+0000","last_active":"2024-08-17T17:33:44.150771+0000","last_peered":"2024-08-17T17:33:44.150771+0000","last_clean":"2024-08-17T17:33:44.150771+0000","last_became_active":"2024-08-17T17:33:44.149498+0000","last_became_peered":"2024-08-17T17:33:44.149498+0000","last_unstale":"2024-08-17T17:33:44.150771+0000","last_undegraded":"2024-08-17T17:33:44.150771+0000","last_fullsized":"2024-08-17T17:33:44.150771+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:17:04.963202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":378,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.212053+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:33:50.212053+0000","last_peered":"2024-08-17T17:33:50.212053+0000","last_clean":"2024-08-17T17:33:50.212053+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:33:50.212053+0000","last_undegraded":"2024-08-17T17:33:50.212053+0000","last_fullsized":"2024-08-17T17:33:50.212053+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":86,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-08-17T17:33:44.141140+0000","last_change":"2024-08-17T17:25:04.579426+0000","last_active":"2024-08-17T17:33:44.141140+0000","last_peered":"2024-08-17T17:33:44.141140+0000","last_clean":"2024-08-17T17:33:44.141140+0000","last_became_active":"2024-08-17T17:25:04.579012+0000","last_became_peered":"2024-08-17T17:25:04.579012+0000","last_unstale":"2024-08-17T17:33:44.141140+0000","last_undegraded":"2024-08-17T17:33:44.141140+0000","last_fullsized":"2024-08-17T17:33:44.141140+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:56:32.241321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"274'5289","reported_seq":5325,"reported_epoch":274,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:33:54.925456+0000","last_change":"2024-08-17T17:33:22.097156+0000","last_active":"2024-08-17T17:33:54.925456+0000","last_peered":"2024-08-17T17:33:54.925456+0000","last_clean":"2024-08-17T17:32:47.362949+0000","last_became_active":"2024-08-17T17:33:21.961793+0000","last_became_peered":"2024-08-17T17:33:21.961793+0000","last_unstale":"2024-08-17T17:33:54.925456+0000","last_undegraded":"2024-08-17T17:33:21.778949+0000","last_fullsized":"2024-08-17T17:33:21.755681+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":5289,"log_dups_size":0,"ondisk_log_size":5289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346619904,"num_objects":5289,"num_object_clones":0,"num_object_copies":15867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3589,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5289,"num_write_kb":338496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":262144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":1700}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":663,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.233066+0000","last_change":"2024-08-17T17:33:20.797705+0000","last_active":"2024-08-17T17:33:50.233066+0000","last_peered":"2024-08-17T17:33:50.233066+0000","last_clean":"2024-08-17T17:33:50.233066+0000","last_became_active":"2024-08-17T17:33:20.797172+0000","last_became_peered":"2024-08-17T17:33:20.797172+0000","last_unstale":"2024-08-17T17:33:50.233066+0000","last_undegraded":"2024-08-17T17:33:50.233066+0000","last_fullsized":"2024-08-17T17:33:50.233066+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:30.754999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":407,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.209230+0000","last_change":"2024-08-17T17:18:59.415331+0000","last_active":"2024-08-17T17:33:50.209230+0000","last_peered":"2024-08-17T17:33:50.209230+0000","last_clean":"2024-08-17T17:33:50.209230+0000","last_became_active":"2024-08-17T17:18:59.415008+0000","last_became_peered":"2024-08-17T17:18:59.415008+0000","last_unstale":"2024-08-17T17:33:50.209230+0000","last_undegraded":"2024-08-17T17:33:50.209230+0000","last_fullsized":"2024-08-17T17:33:50.209230+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:22:38.076866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":216,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.197790+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:33:50.197790+0000","last_peered":"2024-08-17T17:33:50.197790+0000","last_clean":"2024-08-17T17:33:50.197790+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:33:50.197790+0000","last_undegraded":"2024-08-17T17:33:50.197790+0000","last_fullsized":"2024-08-17T17:33:50.197790+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"274'5211","reported_seq":5242,"reported_epoch":274,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:33:52.909485+0000","last_change":"2024-08-17T17:33:21.983064+0000","last_active":"2024-08-17T17:33:52.909485+0000","last_peered":"2024-08-17T17:33:52.909485+0000","last_clean":"2024-08-17T17:32:47.362949+0000","last_became_active":"2024-08-17T17:33:21.969537+0000","last_became_peered":"2024-08-17T17:33:21.969537+0000","last_unstale":"2024-08-17T17:33:52.909485+0000","last_undegraded":"2024-08-17T17:33:21.804745+0000","last_fullsized":"2024-08-17T17:33:21.762111+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":5211,"log_dups_size":0,"ondisk_log_size":5211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341508096,"num_objects":5211,"num_object_clones":0,"num_object_copies":15633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3634,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5211,"num_write_kb":333504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":1577}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":795,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.211832+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:33:50.211832+0000","last_peered":"2024-08-17T17:33:50.211832+0000","last_clean":"2024-08-17T17:33:50.211832+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:33:50.211832+0000","last_undegraded":"2024-08-17T17:33:50.211832+0000","last_fullsized":"2024-08-17T17:33:50.211832+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":301,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.233160+0000","last_change":"2024-08-17T17:33:20.793468+0000","last_active":"2024-08-17T17:33:50.233160+0000","last_peered":"2024-08-17T17:33:50.233160+0000","last_clean":"2024-08-17T17:33:50.233160+0000","last_became_active":"2024-08-17T17:33:20.793166+0000","last_became_peered":"2024-08-17T17:33:20.793166+0000","last_unstale":"2024-08-17T17:33:50.233160+0000","last_undegraded":"2024-08-17T17:33:50.233160+0000","last_fullsized":"2024-08-17T17:33:50.233160+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:12:52.121050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":337,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.241028+0000","last_change":"2024-08-17T17:33:20.835784+0000","last_active":"2024-08-17T17:33:50.241028+0000","last_peered":"2024-08-17T17:33:50.241028+0000","last_clean":"2024-08-17T17:33:50.241028+0000","last_became_active":"2024-08-17T17:33:20.835371+0000","last_became_peered":"2024-08-17T17:33:20.835371+0000","last_unstale":"2024-08-17T17:33:50.241028+0000","last_undegraded":"2024-08-17T17:33:50.241028+0000","last_fullsized":"2024-08-17T17:33:50.241028+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:35:43.369055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.8","version":"274'5123","reported_seq":5153,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:52.912749+0000","last_change":"2024-08-17T17:33:44.165133+0000","last_active":"2024-08-17T17:33:52.912749+0000","last_peered":"2024-08-17T17:33:52.912749+0000","last_clean":"2024-08-17T17:33:52.912749+0000","last_became_active":"2024-08-17T17:33:44.156220+0000","last_became_peered":"2024-08-17T17:33:44.156220+0000","last_unstale":"2024-08-17T17:33:52.912749+0000","last_undegraded":"2024-08-17T17:33:52.912749+0000","last_fullsized":"2024-08-17T17:33:52.912749+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":5123,"log_dups_size":0,"ondisk_log_size":5123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:57:21.141437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335740928,"num_objects":5123,"num_object_clones":0,"num_object_copies":15369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5123,"num_write_kb":327872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":886,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.231950+0000","last_change":"2024-08-17T17:33:44.148227+0000","last_active":"2024-08-17T17:33:50.231950+0000","last_peered":"2024-08-17T17:33:50.231950+0000","last_clean":"2024-08-17T17:33:50.231950+0000","last_became_active":"2024-08-17T17:33:44.147627+0000","last_became_peered":"2024-08-17T17:33:44.147627+0000","last_unstale":"2024-08-17T17:33:50.231950+0000","last_undegraded":"2024-08-17T17:33:50.231950+0000","last_fullsized":"2024-08-17T17:33:50.231950+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:29:42.898460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1372,"num_read_kb":2843,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-08-17T17:33:44.143037+0000","last_change":"2024-08-17T17:32:46.379200+0000","last_active":"2024-08-17T17:33:44.143037+0000","last_peered":"2024-08-17T17:33:44.143037+0000","last_clean":"2024-08-17T17:33:44.143037+0000","last_became_active":"2024-08-17T17:32:46.378870+0000","last_became_peered":"2024-08-17T17:32:46.378870+0000","last_unstale":"2024-08-17T17:33:44.143037+0000","last_undegraded":"2024-08-17T17:33:44.143037+0000","last_fullsized":"2024-08-17T17:33:44.143037+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:55:26.885966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":341,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.232805+0000","last_change":"2024-08-17T17:33:20.772454+0000","last_active":"2024-08-17T17:33:50.232805+0000","last_peered":"2024-08-17T17:33:50.232805+0000","last_clean":"2024-08-17T17:33:50.232805+0000","last_became_active":"2024-08-17T17:33:20.771848+0000","last_became_peered":"2024-08-17T17:33:20.771848+0000","last_unstale":"2024-08-17T17:33:50.232805+0000","last_undegraded":"2024-08-17T17:33:50.232805+0000","last_fullsized":"2024-08-17T17:33:50.232805+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:46:42.300744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.e","version":"274'5238","reported_seq":5273,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:53.773611+0000","last_change":"2024-08-17T17:32:48.460706+0000","last_active":"2024-08-17T17:33:53.773611+0000","last_peered":"2024-08-17T17:33:53.773611+0000","last_clean":"2024-08-17T17:33:53.773611+0000","last_became_active":"2024-08-17T17:32:48.460309+0000","last_became_peered":"2024-08-17T17:32:48.460309+0000","last_unstale":"2024-08-17T17:33:53.773611+0000","last_undegraded":"2024-08-17T17:33:53.773611+0000","last_fullsized":"2024-08-17T17:33:53.773611+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":5238,"log_dups_size":0,"ondisk_log_size":5238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:33:48.006665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343277568,"num_objects":5238,"num_object_clones":0,"num_object_copies":15714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5238,"num_write_kb":335232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":91,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.212448+0000","last_change":"2024-08-17T17:33:44.163867+0000","last_active":"2024-08-17T17:33:50.212448+0000","last_peered":"2024-08-17T17:33:50.212448+0000","last_clean":"2024-08-17T17:33:50.212448+0000","last_became_active":"2024-08-17T17:33:44.163503+0000","last_became_peered":"2024-08-17T17:33:44.163503+0000","last_unstale":"2024-08-17T17:33:50.212448+0000","last_undegraded":"2024-08-17T17:33:50.212448+0000","last_fullsized":"2024-08-17T17:33:50.212448+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:31:53.573596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":556,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.232578+0000","last_change":"2024-08-17T17:33:20.799350+0000","last_active":"2024-08-17T17:33:50.232578+0000","last_peered":"2024-08-17T17:33:50.232578+0000","last_clean":"2024-08-17T17:33:50.232578+0000","last_became_active":"2024-08-17T17:33:20.798925+0000","last_became_peered":"2024-08-17T17:33:20.798925+0000","last_unstale":"2024-08-17T17:33:50.232578+0000","last_undegraded":"2024-08-17T17:33:50.232578+0000","last_fullsized":"2024-08-17T17:33:50.232578+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:16:36.121295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":351,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.233888+0000","last_change":"2024-08-17T17:33:20.835812+0000","last_active":"2024-08-17T17:33:50.233888+0000","last_peered":"2024-08-17T17:33:50.233888+0000","last_clean":"2024-08-17T17:33:50.233888+0000","last_became_active":"2024-08-17T17:33:20.835465+0000","last_became_peered":"2024-08-17T17:33:20.835465+0000","last_unstale":"2024-08-17T17:33:50.233888+0000","last_undegraded":"2024-08-17T17:33:50.233888+0000","last_fullsized":"2024-08-17T17:33:50.233888+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:35:47.750812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.f","version":"274'5263","reported_seq":6906,"reported_epoch":274,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-17T17:33:52.918885+0000","last_change":"2024-08-17T17:33:22.100856+0000","last_active":"2024-08-17T17:33:52.918885+0000","last_peered":"2024-08-17T17:33:52.918885+0000","last_clean":"2024-08-17T17:32:47.362949+0000","last_became_active":"2024-08-17T17:33:22.067826+0000","last_became_peered":"2024-08-17T17:33:22.067826+0000","last_unstale":"2024-08-17T17:33:52.918885+0000","last_undegraded":"2024-08-17T17:33:21.903889+0000","last_fullsized":"2024-08-17T17:33:21.767223+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":5263,"log_dups_size":0,"ondisk_log_size":5263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344915968,"num_objects":5263,"num_object_clones":0,"num_object_copies":15789,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2787,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5263,"num_write_kb":336832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":803,"num_bytes_recovered":52625408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2476}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":376,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.232340+0000","last_change":"2024-08-17T17:33:20.798613+0000","last_active":"2024-08-17T17:33:50.232340+0000","last_peered":"2024-08-17T17:33:50.232340+0000","last_clean":"2024-08-17T17:33:50.232340+0000","last_became_active":"2024-08-17T17:33:20.798260+0000","last_became_peered":"2024-08-17T17:33:20.798260+0000","last_unstale":"2024-08-17T17:33:50.232340+0000","last_undegraded":"2024-08-17T17:33:50.232340+0000","last_fullsized":"2024-08-17T17:33:50.232340+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:18:00.822681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":750,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-08-17T17:33:44.139750+0000","last_change":"2024-08-17T17:32:49.379365+0000","last_active":"2024-08-17T17:33:44.139750+0000","last_peered":"2024-08-17T17:33:44.139750+0000","last_clean":"2024-08-17T17:33:44.139750+0000","last_became_active":"2024-08-17T17:32:49.378742+0000","last_became_peered":"2024-08-17T17:32:49.378742+0000","last_unstale":"2024-08-17T17:33:44.139750+0000","last_undegraded":"2024-08-17T17:33:44.139750+0000","last_fullsized":"2024-08-17T17:33:44.139750+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:32:06.957948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":390,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.196860+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:33:50.196860+0000","last_peered":"2024-08-17T17:33:50.196860+0000","last_clean":"2024-08-17T17:33:50.196860+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:33:50.196860+0000","last_undegraded":"2024-08-17T17:33:50.196860+0000","last_fullsized":"2024-08-17T17:33:50.196860+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"274'5295","reported_seq":5330,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:53.772991+0000","last_change":"2024-08-17T17:32:48.381312+0000","last_active":"2024-08-17T17:33:53.772991+0000","last_peered":"2024-08-17T17:33:53.772991+0000","last_clean":"2024-08-17T17:33:53.772991+0000","last_became_active":"2024-08-17T17:32:48.380990+0000","last_became_peered":"2024-08-17T17:32:48.380990+0000","last_unstale":"2024-08-17T17:33:53.772991+0000","last_undegraded":"2024-08-17T17:33:53.772991+0000","last_fullsized":"2024-08-17T17:33:53.772991+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":5295,"log_dups_size":0,"ondisk_log_size":5295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:39:12.341886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347013120,"num_objects":5295,"num_object_clones":0,"num_object_copies":15885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5295,"num_write_kb":338880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":425,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.222570+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:33:50.222570+0000","last_peered":"2024-08-17T17:33:50.222570+0000","last_clean":"2024-08-17T17:33:50.222570+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:33:50.222570+0000","last_undegraded":"2024-08-17T17:33:50.222570+0000","last_fullsized":"2024-08-17T17:33:50.222570+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":909,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.232803+0000","last_change":"2024-08-17T17:33:21.763805+0000","last_active":"2024-08-17T17:33:50.232803+0000","last_peered":"2024-08-17T17:33:50.232803+0000","last_clean":"2024-08-17T17:33:50.232803+0000","last_became_active":"2024-08-17T17:33:21.763232+0000","last_became_peered":"2024-08-17T17:33:21.763232+0000","last_unstale":"2024-08-17T17:33:50.232803+0000","last_undegraded":"2024-08-17T17:33:50.232803+0000","last_fullsized":"2024-08-17T17:33:50.232803+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:46:08.994820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":422,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.216358+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:33:50.216358+0000","last_peered":"2024-08-17T17:33:50.216358+0000","last_clean":"2024-08-17T17:33:50.216358+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:33:50.216358+0000","last_undegraded":"2024-08-17T17:33:50.216358+0000","last_fullsized":"2024-08-17T17:33:50.216358+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"274'5209","reported_seq":5239,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:54.936737+0000","last_change":"2024-08-17T17:33:44.160183+0000","last_active":"2024-08-17T17:33:54.936737+0000","last_peered":"2024-08-17T17:33:54.936737+0000","last_clean":"2024-08-17T17:33:54.936737+0000","last_became_active":"2024-08-17T17:33:44.144600+0000","last_became_peered":"2024-08-17T17:33:44.144600+0000","last_unstale":"2024-08-17T17:33:54.936737+0000","last_undegraded":"2024-08-17T17:33:54.936737+0000","last_fullsized":"2024-08-17T17:33:54.936737+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":5209,"log_dups_size":0,"ondisk_log_size":5209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:23:53.175287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341377024,"num_objects":5209,"num_object_clones":0,"num_object_copies":15627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5209,"num_write_kb":333376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":240,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.197699+0000","last_change":"2024-08-17T17:33:21.765925+0000","last_active":"2024-08-17T17:33:50.197699+0000","last_peered":"2024-08-17T17:33:50.197699+0000","last_clean":"2024-08-17T17:33:50.197699+0000","last_became_active":"2024-08-17T17:33:21.765372+0000","last_became_peered":"2024-08-17T17:33:21.765372+0000","last_unstale":"2024-08-17T17:33:50.197699+0000","last_undegraded":"2024-08-17T17:33:50.197699+0000","last_fullsized":"2024-08-17T17:33:50.197699+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:14:47.737369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":505,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-08-17T17:33:44.141097+0000","last_change":"2024-08-17T17:26:06.855307+0000","last_active":"2024-08-17T17:33:44.141097+0000","last_peered":"2024-08-17T17:33:44.141097+0000","last_clean":"2024-08-17T17:33:44.141097+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:33:44.141097+0000","last_undegraded":"2024-08-17T17:33:44.141097+0000","last_fullsized":"2024-08-17T17:33:44.141097+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-08-17T17:33:44.149770+0000","last_change":"2024-08-17T17:33:44.149552+0000","last_active":"2024-08-17T17:33:44.149770+0000","last_peered":"2024-08-17T17:33:44.149770+0000","last_clean":"2024-08-17T17:33:44.149770+0000","last_became_active":"2024-08-17T17:33:44.149221+0000","last_became_peered":"2024-08-17T17:33:44.149221+0000","last_unstale":"2024-08-17T17:33:44.149770+0000","last_undegraded":"2024-08-17T17:33:44.149770+0000","last_fullsized":"2024-08-17T17:33:44.149770+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:33:42.937211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":37,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-08-17T17:33:44.143109+0000","last_change":"2024-08-17T17:33:21.763655+0000","last_active":"2024-08-17T17:33:44.143109+0000","last_peered":"2024-08-17T17:33:44.143109+0000","last_clean":"2024-08-17T17:33:44.143109+0000","last_became_active":"2024-08-17T17:33:21.763088+0000","last_became_peered":"2024-08-17T17:33:21.763088+0000","last_unstale":"2024-08-17T17:33:44.143109+0000","last_undegraded":"2024-08-17T17:33:44.143109+0000","last_fullsized":"2024-08-17T17:33:44.143109+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:40:40.126898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":233,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-08-17T17:33:44.139879+0000","last_change":"2024-08-17T17:18:59.430969+0000","last_active":"2024-08-17T17:33:44.139879+0000","last_peered":"2024-08-17T17:33:44.139879+0000","last_clean":"2024-08-17T17:33:44.139879+0000","last_became_active":"2024-08-17T17:18:59.413475+0000","last_became_peered":"2024-08-17T17:18:59.413475+0000","last_unstale":"2024-08-17T17:33:44.139879+0000","last_undegraded":"2024-08-17T17:33:44.139879+0000","last_fullsized":"2024-08-17T17:33:44.139879+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:36:33.732645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":369,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.232065+0000","last_change":"2024-08-17T17:33:20.799402+0000","last_active":"2024-08-17T17:33:50.232065+0000","last_peered":"2024-08-17T17:33:50.232065+0000","last_clean":"2024-08-17T17:33:50.232065+0000","last_became_active":"2024-08-17T17:33:20.798832+0000","last_became_peered":"2024-08-17T17:33:20.798832+0000","last_unstale":"2024-08-17T17:33:50.232065+0000","last_undegraded":"2024-08-17T17:33:50.232065+0000","last_fullsized":"2024-08-17T17:33:50.232065+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:21:05.001134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":117,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.212353+0000","last_change":"2024-08-17T17:32:46.381141+0000","last_active":"2024-08-17T17:33:50.212353+0000","last_peered":"2024-08-17T17:33:50.212353+0000","last_clean":"2024-08-17T17:33:50.212353+0000","last_became_active":"2024-08-17T17:32:46.374259+0000","last_became_peered":"2024-08-17T17:32:46.374259+0000","last_unstale":"2024-08-17T17:33:50.212353+0000","last_undegraded":"2024-08-17T17:33:50.212353+0000","last_fullsized":"2024-08-17T17:33:50.212353+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:19:10.836889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":108,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.232873+0000","last_change":"2024-08-17T17:33:44.146014+0000","last_active":"2024-08-17T17:33:50.232873+0000","last_peered":"2024-08-17T17:33:50.232873+0000","last_clean":"2024-08-17T17:33:50.232873+0000","last_became_active":"2024-08-17T17:33:44.145668+0000","last_became_peered":"2024-08-17T17:33:44.145668+0000","last_unstale":"2024-08-17T17:33:50.232873+0000","last_undegraded":"2024-08-17T17:33:50.232873+0000","last_fullsized":"2024-08-17T17:33:50.232873+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:53:41.679237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":372,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.211620+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:33:50.211620+0000","last_peered":"2024-08-17T17:33:50.211620+0000","last_clean":"2024-08-17T17:33:50.211620+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:33:50.211620+0000","last_undegraded":"2024-08-17T17:33:50.211620+0000","last_fullsized":"2024-08-17T17:33:50.211620+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":399,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.197372+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:33:50.197372+0000","last_peered":"2024-08-17T17:33:50.197372+0000","last_clean":"2024-08-17T17:33:50.197372+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:33:50.197372+0000","last_undegraded":"2024-08-17T17:33:50.197372+0000","last_fullsized":"2024-08-17T17:33:50.197372+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":430,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.216310+0000","last_change":"2024-08-17T17:33:20.790687+0000","last_active":"2024-08-17T17:33:50.216310+0000","last_peered":"2024-08-17T17:33:50.216310+0000","last_clean":"2024-08-17T17:33:50.216310+0000","last_became_active":"2024-08-17T17:33:20.790399+0000","last_became_peered":"2024-08-17T17:33:20.790399+0000","last_unstale":"2024-08-17T17:33:50.216310+0000","last_undegraded":"2024-08-17T17:33:50.216310+0000","last_fullsized":"2024-08-17T17:33:50.216310+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:38:06.849989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":248,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-08-17T17:33:44.141059+0000","last_change":"2024-08-17T17:33:20.835760+0000","last_active":"2024-08-17T17:33:44.141059+0000","last_peered":"2024-08-17T17:33:44.141059+0000","last_clean":"2024-08-17T17:33:44.141059+0000","last_became_active":"2024-08-17T17:33:20.835410+0000","last_became_peered":"2024-08-17T17:33:20.835410+0000","last_unstale":"2024-08-17T17:33:44.141059+0000","last_undegraded":"2024-08-17T17:33:44.141059+0000","last_fullsized":"2024-08-17T17:33:44.141059+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:45.448022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":447,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.216430+0000","last_change":"2024-08-17T17:33:20.771279+0000","last_active":"2024-08-17T17:33:50.216430+0000","last_peered":"2024-08-17T17:33:50.216430+0000","last_clean":"2024-08-17T17:33:50.216430+0000","last_became_active":"2024-08-17T17:33:20.770728+0000","last_became_peered":"2024-08-17T17:33:20.770728+0000","last_unstale":"2024-08-17T17:33:50.216430+0000","last_undegraded":"2024-08-17T17:33:50.216430+0000","last_fullsized":"2024-08-17T17:33:50.216430+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:52:31.540397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":216,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.197545+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:33:50.197545+0000","last_peered":"2024-08-17T17:33:50.197545+0000","last_clean":"2024-08-17T17:33:50.197545+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:33:50.197545+0000","last_undegraded":"2024-08-17T17:33:50.197545+0000","last_fullsized":"2024-08-17T17:33:50.197545+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":347,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.232083+0000","last_change":"2024-08-17T17:33:20.797460+0000","last_active":"2024-08-17T17:33:50.232083+0000","last_peered":"2024-08-17T17:33:50.232083+0000","last_clean":"2024-08-17T17:33:50.232083+0000","last_became_active":"2024-08-17T17:33:20.797154+0000","last_became_peered":"2024-08-17T17:33:20.797154+0000","last_unstale":"2024-08-17T17:33:50.232083+0000","last_undegraded":"2024-08-17T17:33:50.232083+0000","last_fullsized":"2024-08-17T17:33:50.232083+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:02:11.149053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":55,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-08-17T17:33:44.161340+0000","last_change":"2024-08-17T17:33:44.160733+0000","last_active":"2024-08-17T17:33:44.161340+0000","last_peered":"2024-08-17T17:33:44.161340+0000","last_clean":"2024-08-17T17:33:44.161340+0000","last_became_active":"2024-08-17T17:33:44.160368+0000","last_became_peered":"2024-08-17T17:33:44.160368+0000","last_unstale":"2024-08-17T17:33:44.161340+0000","last_undegraded":"2024-08-17T17:33:44.161340+0000","last_fullsized":"2024-08-17T17:33:44.161340+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:37:22.855642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":241,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.232523+0000","last_change":"2024-08-17T17:33:20.774002+0000","last_active":"2024-08-17T17:33:50.232523+0000","last_peered":"2024-08-17T17:33:50.232523+0000","last_clean":"2024-08-17T17:33:50.232523+0000","last_became_active":"2024-08-17T17:33:20.773662+0000","last_became_peered":"2024-08-17T17:33:20.773662+0000","last_unstale":"2024-08-17T17:33:50.232523+0000","last_undegraded":"2024-08-17T17:33:50.232523+0000","last_fullsized":"2024-08-17T17:33:50.232523+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:15:53.475828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":335,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.232725+0000","last_change":"2024-08-17T17:33:20.773207+0000","last_active":"2024-08-17T17:33:50.232725+0000","last_peered":"2024-08-17T17:33:50.232725+0000","last_clean":"2024-08-17T17:33:50.232725+0000","last_became_active":"2024-08-17T17:33:20.772457+0000","last_became_peered":"2024-08-17T17:33:20.772457+0000","last_unstale":"2024-08-17T17:33:50.232725+0000","last_undegraded":"2024-08-17T17:33:50.232725+0000","last_fullsized":"2024-08-17T17:33:50.232725+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:57:09.610084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":226,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.196716+0000","last_change":"2024-08-17T17:32:46.375464+0000","last_active":"2024-08-17T17:33:50.196716+0000","last_peered":"2024-08-17T17:33:50.196716+0000","last_clean":"2024-08-17T17:33:50.196716+0000","last_became_active":"2024-08-17T17:32:46.375161+0000","last_became_peered":"2024-08-17T17:32:46.375161+0000","last_unstale":"2024-08-17T17:33:50.196716+0000","last_undegraded":"2024-08-17T17:33:50.196716+0000","last_fullsized":"2024-08-17T17:33:50.196716+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:27:47.091836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":436,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.213876+0000","last_change":"2024-08-17T17:33:20.774789+0000","last_active":"2024-08-17T17:33:50.213876+0000","last_peered":"2024-08-17T17:33:50.213876+0000","last_clean":"2024-08-17T17:33:50.213876+0000","last_became_active":"2024-08-17T17:33:20.772750+0000","last_became_peered":"2024-08-17T17:33:20.772750+0000","last_unstale":"2024-08-17T17:33:50.213876+0000","last_undegraded":"2024-08-17T17:33:50.213876+0000","last_fullsized":"2024-08-17T17:33:50.213876+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:21:43.839988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":288,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-08-17T17:33:44.165545+0000","last_change":"2024-08-17T17:33:44.165175+0000","last_active":"2024-08-17T17:33:44.165545+0000","last_peered":"2024-08-17T17:33:44.165545+0000","last_clean":"2024-08-17T17:33:44.165545+0000","last_became_active":"2024-08-17T17:33:44.146198+0000","last_became_peered":"2024-08-17T17:33:44.146198+0000","last_unstale":"2024-08-17T17:33:44.165545+0000","last_undegraded":"2024-08-17T17:33:44.165545+0000","last_fullsized":"2024-08-17T17:33:44.165545+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:14:56.291775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":411,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.212182+0000","last_change":"2024-08-17T17:18:59.418434+0000","last_active":"2024-08-17T17:33:50.212182+0000","last_peered":"2024-08-17T17:33:50.212182+0000","last_clean":"2024-08-17T17:33:50.212182+0000","last_became_active":"2024-08-17T17:18:59.418043+0000","last_became_peered":"2024-08-17T17:18:59.418043+0000","last_unstale":"2024-08-17T17:33:50.212182+0000","last_undegraded":"2024-08-17T17:33:50.212182+0000","last_fullsized":"2024-08-17T17:33:50.212182+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:25:51.371820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":268,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-08-17T17:33:44.149034+0000","last_change":"2024-08-17T17:33:44.148090+0000","last_active":"2024-08-17T17:33:44.149034+0000","last_peered":"2024-08-17T17:33:44.149034+0000","last_clean":"2024-08-17T17:33:44.149034+0000","last_became_active":"2024-08-17T17:33:44.147539+0000","last_became_peered":"2024-08-17T17:33:44.147539+0000","last_unstale":"2024-08-17T17:33:44.149034+0000","last_undegraded":"2024-08-17T17:33:44.149034+0000","last_fullsized":"2024-08-17T17:33:44.149034+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:52:02.311332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":425,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.211916+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:33:50.211916+0000","last_peered":"2024-08-17T17:33:50.211916+0000","last_clean":"2024-08-17T17:33:50.211916+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:33:50.211916+0000","last_undegraded":"2024-08-17T17:33:50.211916+0000","last_fullsized":"2024-08-17T17:33:50.211916+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":426,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.212647+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:33:50.212647+0000","last_peered":"2024-08-17T17:33:50.212647+0000","last_clean":"2024-08-17T17:33:50.212647+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:33:50.212647+0000","last_undegraded":"2024-08-17T17:33:50.212647+0000","last_fullsized":"2024-08-17T17:33:50.212647+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":23,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-08-17T17:33:44.142363+0000","last_change":"2024-08-17T17:32:46.376844+0000","last_active":"2024-08-17T17:33:44.142363+0000","last_peered":"2024-08-17T17:33:44.142363+0000","last_clean":"2024-08-17T17:33:44.142363+0000","last_became_active":"2024-08-17T17:32:46.376483+0000","last_became_peered":"2024-08-17T17:32:46.376483+0000","last_unstale":"2024-08-17T17:33:44.142363+0000","last_undegraded":"2024-08-17T17:33:44.142363+0000","last_fullsized":"2024-08-17T17:33:44.142363+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:44:24.358662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":235,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-08-17T17:33:50.197284+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:33:50.197284+0000","last_peered":"2024-08-17T17:33:50.197284+0000","last_clean":"2024-08-17T17:33:50.197284+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:33:50.197284+0000","last_undegraded":"2024-08-17T17:33:50.197284+0000","last_fullsized":"2024-08-17T17:33:50.197284+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":5477433344,"num_objects":83579,"num_object_clones":0,"num_object_copies":250737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24148,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83579,"num_write_kb":5349056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":807,"num_bytes_recovered":52887552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14890696704,"data_stored":14890696704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83579,"ondisk_log_size":83579,"up":48,"acting":41,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":6186,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19280,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5978,"num_read_kb":12991,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":378,"num_read_kb":822,"num_write":215,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":149,"ondisk_log_size":149,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234325,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2253992,"kb_used_data":1668256,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":91491160,"statfs":{"total":95995035648,"available":93686947840,"internally_reserved":0,"allocated":1708294144,"data_stored":1706753094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495960,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2252000,"kb_used_data":1666264,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":91493152,"statfs":{"total":95995035648,"available":93688987648,"internally_reserved":0,"allocated":1706254336,"data_stored":1704721186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757597,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2971092,"kb_used_data":2669580,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":90774060,"statfs":{"total":95995035648,"available":92952637440,"internally_reserved":0,"allocated":2733649920,"data_stored":2732124505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019235,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1970004,"kb_used_data":1664332,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91775148,"statfs":{"total":95995035648,"available":93977751552,"internally_reserved":0,"allocated":1704275968,"data_stored":1702750337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435462,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2584784,"kb_used_data":1993864,"kb_used_omap":4,"kb_used_meta":590907,"kb_avail":91160368,"statfs":{"total":95995035648,"available":93348216832,"internally_reserved":0,"allocated":2041716736,"data_stored":2040195199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":605088778},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":104,"seq":446676598989,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2583200,"kb_used_data":2040728,"kb_used_omap":7,"kb_used_meta":542456,"kb_avail":91161952,"statfs":{"total":95995035648,"available":93349838848,"internally_reserved":0,"allocated":2089705472,"data_stored":2088172809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":555475106},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":268,"seq":1151051235336,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":900368,"kb_used_data":804616,"kb_used_omap":10,"kb_used_meta":95733,"kb_avail":92844784,"statfs":{"total":95995035648,"available":95073058816,"internally_reserved":0,"allocated":823926784,"data_stored":822405192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10902,"internal_metadata":98030954},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926042,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2534232,"kb_used_data":2010256,"kb_used_omap":7,"kb_used_meta":523960,"kb_avail":91210920,"statfs":{"total":95995035648,"available":93399982080,"internally_reserved":0,"allocated":2058502144,"data_stored":2056973469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":536535361},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7550,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2067922944,"data_stored":2067922944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":825819136,"data_stored":825819136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2083782656,"data_stored":2083782656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2044198912,"data_stored":2044198912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1712979968,"data_stored":1712979968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2735603712,"data_stored":2735603712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1711538176,"data_stored":1711538176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1708851200,"data_stored":1708851200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:33:59.543 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-08-17T17:33:59.543 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-08-17T17:33:59.637 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:33:59.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:59.700 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:59.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:33:59.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:33:59 smithi053 ceph-mon[89963]: pgmap v827: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 50 MiB/s wr, 795 op/s; 24051/254223 objects degraded (9.461%); 1.4 MiB/s, 21 objects/s recovering 2024-08-17T17:33:59.801 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:33:59.825 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:33:59.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:59.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:33:59.927 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:33:59.984 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:33:59.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:33:59.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:00.086 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:00.113 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:00.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:00.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:00.215 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:00.259 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:00.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:00.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:00.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:00.374 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 64 16 88509 88493 86.4023 47.125 0.0568763 0.011567 2024-08-17T17:34:00.404 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:00.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:00.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:00.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:00 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 24051/254223 objects degraded (9.461%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:34:00.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:34:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:34:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:34:00.505 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:00.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:00 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 24051/254223 objects degraded (9.461%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:34:00.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:00 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 24051/254223 objects degraded (9.461%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:34:00.544 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:00.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:00.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:00.646 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:00.652 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:34:00.692 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:34:00.727 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:34:00.730 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:00.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:00.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:00.834 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:00.863 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:00.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:00.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:00.965 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:00.990 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:00.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:00.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:01.092 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:01.125 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:01.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:01.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:01.227 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:01.267 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:01.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:01.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:01.369 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:01.374 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 65 16 89284 89268 85.8182 48.4375 0.0282744 0.011646 2024-08-17T17:34:01.397 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:01.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:01.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:01.499 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:01.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:01.544 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:01.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:01.645 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:01.672 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:01.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:01.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:01.774 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:01.801 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:01.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:01.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:01.882 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:01 smithi003 ceph-mon[103344]: pgmap v828: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 48 MiB/s wr, 766 op/s; 24051/261744 objects degraded (9.189%); 1.4 MiB/s, 21 objects/s recovering 2024-08-17T17:34:01.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:01 smithi003 ceph-mon[100615]: pgmap v828: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 48 MiB/s wr, 766 op/s; 24051/261744 objects degraded (9.189%); 1.4 MiB/s, 21 objects/s recovering 2024-08-17T17:34:01.904 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:01.941 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:01.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:01.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:02.044 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:02.077 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:02.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:02.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:02.179 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:02.207 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:02.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:02.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:02.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:01 smithi053 ceph-mon[89963]: pgmap v828: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 48 MiB/s wr, 766 op/s; 24051/261744 objects degraded (9.189%); 1.4 MiB/s, 21 objects/s recovering 2024-08-17T17:34:02.310 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:02.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:02.354 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:02.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:02.374 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 66 16 90131 90115 85.32 52.9375 0.0117532 0.0117144 2024-08-17T17:34:02.455 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:02.512 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:02.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:02.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:02.615 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:02.655 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:02.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:02.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:02.757 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:02.795 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:02.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:02.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:02.897 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:02.938 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:02.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:02.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:03.040 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:03.066 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:03.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:03.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:03.168 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:03.194 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:03.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:03.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:03.296 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:03.321 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:03.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:03.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:03.374 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 67 16 90923 90907 84.7854 49.5 0.0125328 0.0117879 2024-08-17T17:34:03.423 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:03.447 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:03.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:03.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:03.549 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:03.578 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:03.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:03.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:03.680 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:03.727 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:03.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:03.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:03.829 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:03.878 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:03.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:03.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:03.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:04.002 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:04.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:04.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:04.018 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:03 smithi053 ceph-mon[89963]: pgmap v829: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 40 MiB/s wr, 644 op/s; 24051/261744 objects degraded (9.189%); 1.1 MiB/s, 18 objects/s recovering 2024-08-17T17:34:04.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:03 smithi003 ceph-mon[103344]: pgmap v829: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 40 MiB/s wr, 644 op/s; 24051/261744 objects degraded (9.189%); 1.1 MiB/s, 18 objects/s recovering 2024-08-17T17:34:04.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:03 smithi003 ceph-mon[100615]: pgmap v829: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 40 MiB/s wr, 644 op/s; 24051/261744 objects degraded (9.189%); 1.1 MiB/s, 18 objects/s recovering 2024-08-17T17:34:04.103 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:04.148 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:04.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:04.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:04.250 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:04.275 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:04.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:04.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:04.374 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 68 16 91749 91733 84.2978 51.625 0.0325191 0.0118565 2024-08-17T17:34:04.377 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:04.406 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:04.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:04.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:04.507 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:04.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:04.550 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:04.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:04.651 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:04.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:04.685 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:04.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:04.786 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:04.820 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:04.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:04.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:04.921 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:04.952 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:04.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:04.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:04.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:04 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 24051/261744 objects degraded (9.189%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:34:05.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:04 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 24051/261744 objects degraded (9.189%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:34:05.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:04 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 24051/261744 objects degraded (9.189%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:34:05.054 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:05.087 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:05.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:05.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:05.189 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:05.220 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:05.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:05.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:05.322 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:05.370 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:05.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:05.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:05.374 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 69 16 92577 92561 83.826 51.75 0.0067948 0.011924 2024-08-17T17:34:05.472 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:05.507 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:05.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:05.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:05.608 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:05.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:05.637 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:05.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:05.738 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:05.779 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:05.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:05.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:05.880 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:05.915 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:05.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:05.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:06.017 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:06.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:05 smithi003 ceph-mon[103344]: pgmap v830: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 58 MiB/s wr, 927 op/s; 23938/272163 objects degraded (8.795%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T17:34:06.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:05 smithi003 ceph-mon[100615]: pgmap v830: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 58 MiB/s wr, 927 op/s; 23938/272163 objects degraded (8.795%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T17:34:06.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:34:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:34:05] "GET /metrics HTTP/1.1" 200 36530 "" "Prometheus/2.51.0" 2024-08-17T17:34:06.043 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:06.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:06.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:06.145 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:06.190 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:06.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:06.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:06.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:05 smithi053 ceph-mon[89963]: pgmap v830: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 58 MiB/s wr, 927 op/s; 23938/272163 objects degraded (8.795%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T17:34:06.292 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:06.321 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:06.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:06.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:06.375 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 70 16 93316 93300 83.2883 46.1875 0.00889264 0.0119984 2024-08-17T17:34:06.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:06.469 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:06.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:06.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:06.571 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:06.598 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:06.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:06.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:06.700 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:06.728 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:06.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:06.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:06.829 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:06.854 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:06.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:06.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:06.956 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:07.017 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:07.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:07.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:07.119 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:07.145 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:07.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:07.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:07.247 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:07.274 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:07.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:07.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:07.375 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 71 16 94075 94059 82.7834 47.4375 0.0118017 0.0120713 2024-08-17T17:34:07.376 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:07.426 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:07.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:07.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:07.528 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:07.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:07.576 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:07.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:07.677 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:07.709 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:07.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:07.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:07.777 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-08-17T17:34:07.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:07.855 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:07.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:07.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:07.957 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:07.986 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:07.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:07.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:08.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:07 smithi003 ceph-mon[103344]: pgmap v831: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 42 MiB/s wr, 677 op/s; 23938/273693 objects degraded (8.746%); 1.1 MiB/s, 17 objects/s recovering 2024-08-17T17:34:08.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:07 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/743845964' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:34:08.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:07 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:34:08.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:07 smithi003 ceph-mon[100615]: pgmap v831: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 42 MiB/s wr, 677 op/s; 23938/273693 objects degraded (8.746%); 1.1 MiB/s, 17 objects/s recovering 2024-08-17T17:34:08.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:07 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/743845964' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:34:08.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:07 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:34:08.088 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:08.138 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:08.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:08.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:08.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:07 smithi053 ceph-mon[89963]: pgmap v831: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 42 MiB/s wr, 677 op/s; 23938/273693 objects degraded (8.746%); 1.1 MiB/s, 17 objects/s recovering 2024-08-17T17:34:08.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:07 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/743845964' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:34:08.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:07 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:34:08.240 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:08.266 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:08.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:08.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:08.368 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:08.375 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 72 16 94849 94833 82.3055 48.375 0.0127679 0.0121436 2024-08-17T17:34:08.396 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:08.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:08.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:08.498 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:08.526 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:08.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:08.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:08.628 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:08.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:08.684 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:08.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:08.785 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:08.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:08.838 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:08.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:08.932 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:34:08.932 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:34:08.932 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.933 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:95097 8.3 e275/269 8:cdb42420:::benchmark_data_smithi003_229103_object95096:head v 275'5922, mlcod=0'0)", 2024-08-17T17:34:08.933 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:34:08.739096+0000", 2024-08-17T17:34:08.933 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.029383543000000002, 2024-08-17T17:34:08.933 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.029414760000000002, 2024-08-17T17:34:08.933 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:34:08.933 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:34:08.933 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:34:08.933 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:34:08.933 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.933 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:34:08.933 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.739096+0000", 2024-08-17T17:34:08.934 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:08.934 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.934 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.934 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:34:08.934 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.739096+0000", 2024-08-17T17:34:08.934 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:08.934 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.934 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.934 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:34:08.934 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.739096+0000", 2024-08-17T17:34:08.934 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1600000000000001e-07 2024-08-17T17:34:08.934 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.935 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.935 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:34:08.935 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.739135+0000", 2024-08-17T17:34:08.935 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8819999999999998e-05 2024-08-17T17:34:08.935 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.935 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.935 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:34:08.935 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.739149+0000", 2024-08-17T17:34:08.935 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3932e-05 2024-08-17T17:34:08.935 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.935 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.935 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:34:08.936 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.739156+0000", 2024-08-17T17:34:08.936 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8199999999999999e-06 2024-08-17T17:34:08.936 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.936 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.936 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:34:08.936 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.745124+0000", 2024-08-17T17:34:08.936 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.005967656 2024-08-17T17:34:08.936 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.936 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.936 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:34:08.936 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.745150+0000", 2024-08-17T17:34:08.936 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6611e-05 2024-08-17T17:34:08.936 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:08.937 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:34:08.937 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:08.937 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.937 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.937 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:95101 8.b e275/269 8:d47d3ab6:::benchmark_data_smithi003_229103_object95100:head v 275'5973, mlcod=0'0)", 2024-08-17T17:34:08.937 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:34:08.748038+0000", 2024-08-17T17:34:08.937 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.020441055, 2024-08-17T17:34:08.937 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.020533454, 2024-08-17T17:34:08.937 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:34:08.937 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:34:08.937 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:34:08.937 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:34:08.937 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.938 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:34:08.938 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.748038+0000", 2024-08-17T17:34:08.938 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:08.938 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.938 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.938 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:34:08.938 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.748038+0000", 2024-08-17T17:34:08.938 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:08.938 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.938 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.938 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:34:08.938 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.748039+0000", 2024-08-17T17:34:08.939 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6400000000000003e-07 2024-08-17T17:34:08.939 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.939 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.939 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:34:08.939 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.748064+0000", 2024-08-17T17:34:08.939 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5772999999999999e-05 2024-08-17T17:34:08.939 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.939 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.939 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:34:08.939 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.748072+0000", 2024-08-17T17:34:08.939 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7360000000000001e-06 2024-08-17T17:34:08.939 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.939 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.939 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:34:08.940 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.748078+0000", 2024-08-17T17:34:08.940 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.502e-06 2024-08-17T17:34:08.940 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.940 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.940 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:34:08.940 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.749507+0000", 2024-08-17T17:34:08.940 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014293839999999999 2024-08-17T17:34:08.940 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.940 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.940 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:34:08.940 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.749573+0000", 2024-08-17T17:34:08.940 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.635e-05 2024-08-17T17:34:08.940 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:08.941 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:34:08.941 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:08.941 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.941 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.941 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:95107 8.a e275/269 8:5de6b335:::benchmark_data_smithi003_229103_object95106:head v 275'5910, mlcod=265'4)", 2024-08-17T17:34:08.941 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:34:08.757589+0000", 2024-08-17T17:34:08.941 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.010889896, 2024-08-17T17:34:08.941 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011038745000000001, 2024-08-17T17:34:08.941 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:34:08.941 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:34:08.941 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:34:08.941 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:34:08.941 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.941 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:34:08.942 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.757589+0000", 2024-08-17T17:34:08.942 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:08.942 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.942 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.942 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:34:08.942 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.757589+0000", 2024-08-17T17:34:08.942 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:08.942 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.942 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.942 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:34:08.942 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.757590+0000", 2024-08-17T17:34:08.942 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.95e-07 2024-08-17T17:34:08.942 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.943 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.943 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:34:08.943 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.757665+0000", 2024-08-17T17:34:08.943 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4612999999999999e-05 2024-08-17T17:34:08.943 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.943 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.943 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:34:08.943 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.757686+0000", 2024-08-17T17:34:08.943 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0752000000000001e-05 2024-08-17T17:34:08.943 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.943 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.943 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:34:08.943 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.757694+0000", 2024-08-17T17:34:08.943 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4870000000000002e-06 2024-08-17T17:34:08.943 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.943 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.944 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:34:08.944 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.758268+0000", 2024-08-17T17:34:08.944 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00057430199999999995 2024-08-17T17:34:08.944 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.944 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.944 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:34:08.944 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.758308+0000", 2024-08-17T17:34:08.944 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9551999999999997e-05 2024-08-17T17:34:08.944 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:08.944 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:34:08.944 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:08.944 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.944 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.945 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:95104 8.7 e275/269 8:e97ec5fb:::benchmark_data_smithi003_229103_object95103:head v 275'5936, mlcod=0'0)", 2024-08-17T17:34:08.945 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:34:08.757761+0000", 2024-08-17T17:34:08.945 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.010718238999999999, 2024-08-17T17:34:08.945 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010921861999999999, 2024-08-17T17:34:08.945 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:34:08.945 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:34:08.945 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:34:08.945 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:34:08.945 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.945 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:34:08.945 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.757761+0000", 2024-08-17T17:34:08.945 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:08.946 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.946 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.946 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:34:08.946 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.757761+0000", 2024-08-17T17:34:08.946 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:08.946 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.946 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.946 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:34:08.946 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.757762+0000", 2024-08-17T17:34:08.946 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.37e-07 2024-08-17T17:34:08.946 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.946 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.946 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:34:08.947 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.757787+0000", 2024-08-17T17:34:08.947 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5265e-05 2024-08-17T17:34:08.947 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.947 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.947 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:34:08.947 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.757795+0000", 2024-08-17T17:34:08.947 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6780000000000006e-06 2024-08-17T17:34:08.947 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.947 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.947 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:34:08.947 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.757802+0000", 2024-08-17T17:34:08.947 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6590000000000001e-06 2024-08-17T17:34:08.948 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.948 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.948 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:34:08.948 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.759703+0000", 2024-08-17T17:34:08.948 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001900498 2024-08-17T17:34:08.948 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.948 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.948 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:34:08.948 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.759767+0000", 2024-08-17T17:34:08.948 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4766999999999998e-05 2024-08-17T17:34:08.948 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:08.948 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:34:08.948 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:08.948 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.949 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.949 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:95106 8.3 e275/269 8:c5a686e6:::benchmark_data_smithi003_229103_object95105:head v 275'5923, mlcod=0'0)", 2024-08-17T17:34:08.949 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:34:08.758037+0000", 2024-08-17T17:34:08.949 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.010442702999999999, 2024-08-17T17:34:08.949 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010699426999999999, 2024-08-17T17:34:08.949 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:34:08.949 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:34:08.949 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:34:08.949 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:34:08.949 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.949 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:34:08.949 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.758037+0000", 2024-08-17T17:34:08.949 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:08.949 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.950 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.950 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:34:08.950 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.758037+0000", 2024-08-17T17:34:08.950 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:08.950 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.950 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.950 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:34:08.950 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.758037+0000", 2024-08-17T17:34:08.950 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2900000000000003e-07 2024-08-17T17:34:08.950 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.950 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.950 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:34:08.950 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.758091+0000", 2024-08-17T17:34:08.950 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3474000000000003e-05 2024-08-17T17:34:08.951 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.951 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.951 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:34:08.951 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.758105+0000", 2024-08-17T17:34:08.951 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4635e-05 2024-08-17T17:34:08.951 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.951 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.951 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:34:08.951 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.758112+0000", 2024-08-17T17:34:08.951 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.545e-06 2024-08-17T17:34:08.951 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.951 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.952 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:34:08.952 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.758605+0000", 2024-08-17T17:34:08.952 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00049328399999999998 2024-08-17T17:34:08.952 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:08.952 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:08.952 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:34:08.952 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:08.758706+0000", 2024-08-17T17:34:08.952 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010110700000000001 2024-08-17T17:34:08.952 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:08.952 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:34:08.952 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:08.952 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:08.952 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-08-17T17:34:08.953 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 5 2024-08-17T17:34:08.953 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:34:08.953 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:08.953 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:08 smithi003 ceph-mon[103344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:34:08.954 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:08 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:34:08.954 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:08 smithi003 ceph-mon[103344]: osdmap e275: 8 total, 8 up, 8 in 2024-08-17T17:34:08.954 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:08.954 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:34:08.954 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:34:08.954 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:08 smithi003 ceph-mon[100615]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:34:08.954 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:08 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:34:08.954 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:08 smithi003 ceph-mon[100615]: osdmap e275: 8 total, 8 up, 8 in 2024-08-17T17:34:08.955 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:08.955 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:34:08.955 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:34:08.955 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:34:08.955 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:34:08.998 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:34:08.998 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:34:09.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:09.018 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:09.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:09.044 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:34:09.044 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:34:09.081 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:34:09.081 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:34:09.113 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:34:09.114 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:34:09.119 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:09.149 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:34:09.149 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:34:09.150 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:09.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:09.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:09.183 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:34:09.183 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:34:09.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:08 smithi053 ceph-mon[89963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:34:09.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:08 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:34:09.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:08 smithi053 ceph-mon[89963]: osdmap e275: 8 total, 8 up, 8 in 2024-08-17T17:34:09.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:09.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:34:09.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:34:09.217 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:34:09.218 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:34:09.252 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:09.279 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:09.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:09.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:09.375 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 73 16 95515 95499 81.7482 41.625 0.0318236 0.0122267 2024-08-17T17:34:09.381 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:09.408 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:09.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:09.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:09.511 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:09.541 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:09.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:09.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:09.644 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:09.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:09.689 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:09.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:09.790 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:09.819 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:09.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:09.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:09.921 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:09.946 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:09.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:09.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:10.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:09 smithi003 ceph-mon[103344]: pgmap v833: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 48 MiB/s wr, 770 op/s; 23821/277344 objects degraded (8.589%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:34:10.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:09 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/757381220' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:34:10.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:10.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:10.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:10.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:10.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:09 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 23821/277344 objects degraded (8.589%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:34:10.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:09 smithi003 ceph-mon[100615]: pgmap v833: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 48 MiB/s wr, 770 op/s; 23821/277344 objects degraded (8.589%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:34:10.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:09 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/757381220' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:34:10.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:10.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:10.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:10.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:10.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:09 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 23821/277344 objects degraded (8.589%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:34:10.049 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:10.072 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:10.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:10.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:10.173 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:10.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:10.212 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:10.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:10.214 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:09 smithi053 ceph-mon[89963]: pgmap v833: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 48 MiB/s wr, 770 op/s; 23821/277344 objects degraded (8.589%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:34:10.214 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:09 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/757381220' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:34:10.214 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:10.215 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:10.215 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:10.215 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:10.215 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:09 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 23821/277344 objects degraded (8.589%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:34:10.313 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:10.341 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:10.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:10.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:10.375 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 74 16 96232 96216 81.2491 44.8125 0.0194678 0.0123016 2024-08-17T17:34:10.444 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:10.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:34:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:34:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:34:10.469 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:10.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:10.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:10.571 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:10.600 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:10.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:10.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:10.702 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:10.735 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:10.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:10.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:10.837 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:10.865 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:10.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:10.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:10.967 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:11.020 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:11.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:11.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:11.122 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:11.167 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:11.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:11.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:11.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:11.315 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:11.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:11.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:11.375 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 75 16 97040 97024 80.8391 50.5 0.0617179 0.0123631 2024-08-17T17:34:11.417 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:11.454 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:11.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:11.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:11.556 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:11.583 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:11.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:11.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:11.685 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:11.712 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:11.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:11.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:11.815 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:11.839 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:11.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:11.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:11.941 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:11.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:11.982 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:11.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:12.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:11 smithi003 ceph-mon[103344]: pgmap v834: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 49 MiB/s wr, 786 op/s; 23821/285345 objects degraded (8.348%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:34:12.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:11 smithi003 ceph-mon[100615]: pgmap v834: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 49 MiB/s wr, 786 op/s; 23821/285345 objects degraded (8.348%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:34:12.083 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:12.111 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:12.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:12.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:12.213 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:12.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:11 smithi053 ceph-mon[89963]: pgmap v834: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 49 MiB/s wr, 786 op/s; 23821/285345 objects degraded (8.348%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:34:12.259 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:12.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:12.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:12.361 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:12.375 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 76 16 97871 97855 80.4587 51.9375 0.00705615 0.0124227 2024-08-17T17:34:12.391 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:12.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:12.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:12.493 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:12.549 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:12.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:12.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:12.650 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:34:12.651 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:12.675 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:12.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:12.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:12.777 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:12.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:12.848 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:12.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:12.949 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:12.978 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:12.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:12.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:13.080 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:13.111 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:13.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:13.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:13.213 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:13.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-17T17:34:13.278 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:13.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:13.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:13.375 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 77 16 98592 98576 79.9989 45.0625 0.0170385 0.0124944 2024-08-17T17:34:13.382 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:13.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:13.411 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:13.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:13.512 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:13.536 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:34:13.537 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:13.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:13.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:13.640 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:13.669 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:13.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:13.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:13.771 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:13.795 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:13.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:13.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:13.842 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:34:13.897 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:13.923 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:13.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:13.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:13.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:13 smithi053 ceph-mon[89963]: pgmap v835: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 49 MiB/s wr, 786 op/s; 23821/285345 objects degraded (8.348%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:34:14.018 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:13 smithi003 ceph-mon[100615]: pgmap v835: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 49 MiB/s wr, 786 op/s; 23821/285345 objects degraded (8.348%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:34:14.019 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:13 smithi003 ceph-mon[103344]: pgmap v835: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 49 MiB/s wr, 786 op/s; 23821/285345 objects degraded (8.348%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:34:14.024 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:14.054 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:14.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:14.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:14.156 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:14.183 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:14.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:14.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:14.285 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:14.347 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:14.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:14.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:14.375 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 78 16 99180 99164 79.4443 36.75 0.00748179 0.0125812 2024-08-17T17:34:14.449 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:14.479 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:14.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:14.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:14.581 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:14.632 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:14.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:14.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:14.682 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:34:14.734 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:14.791 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:14.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:14.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:14.838 INFO:teuthology.orchestra.run.smithi003.stderr:set pool 1 pg_num to 11 2024-08-17T17:34:14.893 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:14.939 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:14.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:14.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:14.971 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:14 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3035098646' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-08-17T17:34:14.971 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:14 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-08-17T17:34:14.971 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:14 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 23821/285345 objects degraded (8.348%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:34:14.971 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:14 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3035098646' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-08-17T17:34:14.971 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:14 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-08-17T17:34:14.972 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:14 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 23821/285345 objects degraded (8.348%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:34:15.041 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:15.072 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:15.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:15.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:15.173 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:15.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:14 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3035098646' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-08-17T17:34:15.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:14 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-08-17T17:34:15.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:14 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 23821/285345 objects degraded (8.348%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:34:15.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:15.219 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:15.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:15.320 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:15.376 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 79 16 99902 99886 79.0098 45.125 0.0154297 0.0126501 2024-08-17T17:34:15.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:15.380 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:15.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:15.481 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:15.505 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:15.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:15.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:15.607 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:15.640 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:15.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:15.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:15.743 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:15.773 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:15.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:15.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:15.875 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:15.922 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:15.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:15.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:16.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:15 smithi003 ceph-mon[100615]: pgmap v836: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 48 MiB/s wr, 765 op/s; 23690/295143 objects degraded (8.027%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:34:16.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:15 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-08-17T17:34:16.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:15 smithi003 ceph-mon[100615]: osdmap e276: 8 total, 8 up, 8 in 2024-08-17T17:34:16.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:15 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:16.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:15 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:16.024 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:34:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:34:15] "GET /metrics HTTP/1.1" 200 36532 "" "Prometheus/2.51.0" 2024-08-17T17:34:16.025 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:15 smithi003 ceph-mon[103344]: pgmap v836: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 48 MiB/s wr, 765 op/s; 23690/295143 objects degraded (8.027%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:34:16.025 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:15 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-08-17T17:34:16.025 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:15 smithi003 ceph-mon[103344]: osdmap e276: 8 total, 8 up, 8 in 2024-08-17T17:34:16.025 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:15 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:16.025 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:15 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:16.025 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:16.069 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:15 smithi053 ceph-mon[89963]: pgmap v836: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 48 MiB/s wr, 765 op/s; 23690/295143 objects degraded (8.027%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:34:16.069 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:15 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-08-17T17:34:16.069 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:15 smithi053 ceph-mon[89963]: osdmap e276: 8 total, 8 up, 8 in 2024-08-17T17:34:16.069 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:15 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:16.069 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:15 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:16.081 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:16.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:16.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:16.183 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:16.208 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:16.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:16.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:16.309 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:16.347 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:16.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:16.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:16.376 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-08-17T17:34:16.376325+0000 min lat: 0.00109625 max lat: 2.31183 avg lat: 0.0127167 2024-08-17T17:34:16.376 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:34:16.376 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 80 16 100639 100623 78.5979 46.0625 0.00988061 0.0127167 2024-08-17T17:34:16.449 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:16.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:16.496 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:16.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:16.596 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:16.645 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:16.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:16.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:16.746 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:16.776 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:16.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:16.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:16.878 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:16.904 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:16.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:16.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:17.006 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:17.045 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:17.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:17.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:17.066 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:16 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-08-17T17:34:17.067 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:16 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-08-17T17:34:17.148 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:17.170 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:17.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:17.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:17.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:16 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-08-17T17:34:17.273 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:17.295 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:17.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:17.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:17.376 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 81 16 101185 101169 78.0489 34.125 0.0215726 0.0128066 2024-08-17T17:34:17.397 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:17.442 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:17.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:17.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:17.544 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:17.580 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:17.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:17.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:17.682 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:17.711 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:17.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:17.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:17.813 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:17.838 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:17.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:17.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:17.940 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:17.967 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:17.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:17.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:18.047 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:17 smithi003 ceph-mon[100615]: pgmap v838: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 57 MiB/s wr, 910 op/s; 23690/296427 objects degraded (7.992%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T17:34:18.047 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:17 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-08-17T17:34:18.047 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:17 smithi003 ceph-mon[100615]: osdmap e277: 8 total, 8 up, 8 in 2024-08-17T17:34:18.048 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:17 smithi003 ceph-mon[103344]: pgmap v838: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 57 MiB/s wr, 910 op/s; 23690/296427 objects degraded (7.992%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T17:34:18.048 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:17 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-08-17T17:34:18.048 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:17 smithi003 ceph-mon[103344]: osdmap e277: 8 total, 8 up, 8 in 2024-08-17T17:34:18.069 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:18.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:18.095 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:18.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:18.196 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:18.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:17 smithi053 ceph-mon[89963]: pgmap v838: 89 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 57 MiB/s wr, 910 op/s; 23690/296427 objects degraded (7.992%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T17:34:18.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:17 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-08-17T17:34:18.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:17 smithi053 ceph-mon[89963]: osdmap e277: 8 total, 8 up, 8 in 2024-08-17T17:34:18.246 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:18.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:18.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:18.348 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:18.374 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:18.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:18.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:18.376 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 82 16 101950 101934 77.6801 47.8125 0.0153278 0.0128673 2024-08-17T17:34:18.476 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:18.503 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:18.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:18.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:18.605 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:18.662 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:18.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:18.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:18.766 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:18.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:18.806 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:18.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:18.907 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:18.962 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:18.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:18.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:19.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:18 smithi003 ceph-mon[103344]: osdmap e278: 8 total, 8 up, 8 in 2024-08-17T17:34:19.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:18 smithi003 ceph-mon[103344]: pgmap v841: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 10 unknown, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 50 MiB/s wr, 800 op/s; 23546/299760 objects degraded (7.855%); 2.9 MiB/s, 45 objects/s recovering 2024-08-17T17:34:19.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:18 smithi003 ceph-mon[100615]: osdmap e278: 8 total, 8 up, 8 in 2024-08-17T17:34:19.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:18 smithi003 ceph-mon[100615]: pgmap v841: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 10 unknown, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 50 MiB/s wr, 800 op/s; 23546/299760 objects degraded (7.855%); 2.9 MiB/s, 45 objects/s recovering 2024-08-17T17:34:19.065 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:19.091 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:19.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:19.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:19.140 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:34:19.193 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:19.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:18 smithi053 ceph-mon[89963]: osdmap e278: 8 total, 8 up, 8 in 2024-08-17T17:34:19.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:18 smithi053 ceph-mon[89963]: pgmap v841: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 10 unknown, 6 active+recovery_wait+undersized+degraded+remapped, 82 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 50 MiB/s wr, 800 op/s; 23546/299760 objects degraded (7.855%); 2.9 MiB/s, 45 objects/s recovering 2024-08-17T17:34:19.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:19.245 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:19.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:19.255 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:34:19.255 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:34:19.346 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:19.376 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:19.376 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 83 16 102623 102607 77.251 42.0625 0.00906009 0.012939 2024-08-17T17:34:19.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:19.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:19.478 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:19.508 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:19.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:19.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:19.610 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:19.652 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:19.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:19.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:19.754 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:19.800 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:19.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:19.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:19.865 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-08-17T17:34:19.902 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:19.927 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:19.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:19.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:19.953 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-17T17:34:19.953 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:19 smithi003 ceph-mon[103344]: 1.0 scrub starts 2024-08-17T17:34:19.953 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:19 smithi003 ceph-mon[103344]: 1.0 scrub ok 2024-08-17T17:34:19.953 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:19 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1053321296' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:34:19.953 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:19 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:34:19.953 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:19 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 23546/299760 objects degraded (7.855%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:34:19.954 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:19 smithi003 ceph-mon[100615]: 1.0 scrub starts 2024-08-17T17:34:19.954 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:19 smithi003 ceph-mon[100615]: 1.0 scrub ok 2024-08-17T17:34:19.954 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:19 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1053321296' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:34:19.954 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:19 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:34:19.955 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:19 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 23546/299760 objects degraded (7.855%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:34:20.029 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:20.072 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:20.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:20.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:20.174 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:20.214 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:19 smithi053 ceph-mon[89963]: 1.0 scrub starts 2024-08-17T17:34:20.215 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:19 smithi053 ceph-mon[89963]: 1.0 scrub ok 2024-08-17T17:34:20.215 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:19 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1053321296' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:34:20.215 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:19 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:34:20.215 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:19 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 23546/299760 objects degraded (7.855%), 7 pgs degraded (PG_DEGRADED) 2024-08-17T17:34:20.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:20.244 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:20.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:20.345 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:20.377 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 84 16 103339 103323 76.8641 44.75 0.034892 0.0130044 2024-08-17T17:34:20.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:20.405 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:20.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:20.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:34:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:34:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:34:20.505 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:20.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:20.552 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:20.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:20.653 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:20.700 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:20.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:20.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:20.802 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:20.851 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:20.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:20.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:20.953 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:20.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:20.999 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:20.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:21.100 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:21.138 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:21.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:21.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:21.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:20 smithi053 ceph-mon[89963]: 1.4 scrub starts 2024-08-17T17:34:21.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:20 smithi053 ceph-mon[89963]: 1.4 scrub ok 2024-08-17T17:34:21.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:20 smithi053 ceph-mon[89963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:34:21.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:20 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:34:21.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:20 smithi053 ceph-mon[89963]: osdmap e279: 8 total, 8 up, 8 in 2024-08-17T17:34:21.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:20 smithi053 ceph-mon[89963]: pgmap v843: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 45 MiB/s wr, 723 op/s; 23546/306651 objects degraded (7.678%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T17:34:21.240 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:21.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:20 smithi003 ceph-mon[103344]: 1.4 scrub starts 2024-08-17T17:34:21.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:20 smithi003 ceph-mon[103344]: 1.4 scrub ok 2024-08-17T17:34:21.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:20 smithi003 ceph-mon[103344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:34:21.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:20 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:34:21.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:20 smithi003 ceph-mon[103344]: osdmap e279: 8 total, 8 up, 8 in 2024-08-17T17:34:21.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:20 smithi003 ceph-mon[103344]: pgmap v843: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 45 MiB/s wr, 723 op/s; 23546/306651 objects degraded (7.678%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T17:34:21.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:20 smithi003 ceph-mon[100615]: 1.4 scrub starts 2024-08-17T17:34:21.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:20 smithi003 ceph-mon[100615]: 1.4 scrub ok 2024-08-17T17:34:21.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:20 smithi003 ceph-mon[100615]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:34:21.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:20 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:34:21.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:20 smithi003 ceph-mon[100615]: osdmap e279: 8 total, 8 up, 8 in 2024-08-17T17:34:21.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:20 smithi003 ceph-mon[100615]: pgmap v843: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 45 MiB/s wr, 723 op/s; 23546/306651 objects degraded (7.678%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T17:34:21.279 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:21.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:21.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:21.376 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 85 16 104079 104063 76.5039 46.25 0.0305851 0.0130655 2024-08-17T17:34:21.382 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:21.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:21.420 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:21.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:21.521 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:21.539 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-08-17T17:34:21.539 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:34:21.539 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-08-17T17:34:21.539 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 1.000000 2024-08-17T17:34:21.539 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 4 1 2024-08-17T17:34:21.544 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:21.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:21.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:21.646 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:21.675 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:21.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:21.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:21.777 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:21.807 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:21.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:21.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:21.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:21.955 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:21.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:21.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:22.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:21 smithi003 ceph-mon[103344]: 1.7 scrub starts 2024-08-17T17:34:22.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:21 smithi003 ceph-mon[100615]: 1.7 scrub starts 2024-08-17T17:34:22.057 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:22.087 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:22.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:22.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:22.189 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:22.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:21 smithi053 ceph-mon[89963]: 1.7 scrub starts 2024-08-17T17:34:22.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:22.236 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:22.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:22.337 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:22.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:22.364 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:22.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:22.376 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 86 16 104769 104753 76.1158 43.125 0.0287912 0.013132 2024-08-17T17:34:22.465 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:22.489 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:22.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:22.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:22.591 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:22.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:22.622 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:22.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:22.723 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:22.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:22.769 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:22.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:22.870 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:22.913 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:22.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:22.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:23.014 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:23.038 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:23.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:23.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:23.140 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:23.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:23.167 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:23.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:23.209 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:22 smithi003 ceph-mon[100615]: 1.5 scrub starts 2024-08-17T17:34:23.209 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:22 smithi003 ceph-mon[100615]: pgmap v844: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 35 MiB/s wr, 564 op/s; 23546/306651 objects degraded (7.678%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:34:23.210 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:22 smithi003 ceph-mon[103344]: 1.5 scrub starts 2024-08-17T17:34:23.210 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:22 smithi003 ceph-mon[103344]: pgmap v844: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 35 MiB/s wr, 564 op/s; 23546/306651 objects degraded (7.678%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:34:23.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:22 smithi053 ceph-mon[89963]: 1.5 scrub starts 2024-08-17T17:34:23.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:22 smithi053 ceph-mon[89963]: pgmap v844: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 35 MiB/s wr, 564 op/s; 23546/306651 objects degraded (7.678%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:34:23.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:23.312 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:34:23.312 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:34:23.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:23.316 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:23.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:23.352 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:34:23.352 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:34:23.376 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 87 16 105516 105500 75.7775 46.6875 0.0293008 0.0131901 2024-08-17T17:34:23.388 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:34:23.388 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:34:23.416 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:23.426 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:34:23.427 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:34:23.459 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:34:23.459 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:34:23.462 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:23.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:23.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:23.493 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:34:23.493 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:34:23.530 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:34:23.531 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:34:23.562 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:34:23.562 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:34:23.565 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:23.619 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:23.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:23.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:23.720 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:23.759 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:23.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:23.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:23.861 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:23.889 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:23.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:23.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:23.991 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:24.020 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:24.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:24.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:24.121 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:24.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:24.182 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:24.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:24.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:23 smithi053 ceph-mon[89963]: 1.3 scrub starts 2024-08-17T17:34:24.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:24.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:34:24.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:23 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/303100' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:34:24.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:23 smithi003 ceph-mon[100615]: 1.3 scrub starts 2024-08-17T17:34:24.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:24.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:34:24.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:23 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/303100' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:34:24.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:23 smithi003 ceph-mon[103344]: 1.3 scrub starts 2024-08-17T17:34:24.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:24.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:34:24.275 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:23 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/303100' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:34:24.283 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:24.309 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:24.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:24.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:24.377 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 88 16 106254 106238 75.4405 46.125 0.00738981 0.0132498 2024-08-17T17:34:24.411 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:24.440 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:24.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:24.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:24.542 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:24.584 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:24.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:24.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:24.688 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:24.715 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:24.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:24.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:24.816 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:24.890 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:34:24.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:24.914 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:24.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:24.919 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:34:25.015 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:25.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:24 smithi003 ceph-mon[100615]: 1.6 scrub starts 2024-08-17T17:34:25.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:24 smithi003 ceph-mon[100615]: pgmap v845: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 57 MiB/s wr, 905 op/s; 23437/316245 objects degraded (7.411%); 2.1 MiB/s, 34 objects/s recovering 2024-08-17T17:34:25.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:24 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 23546/306651 objects degraded (7.678%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:34:25.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:24 smithi003 ceph-mon[103344]: 1.6 scrub starts 2024-08-17T17:34:25.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:24 smithi003 ceph-mon[103344]: pgmap v845: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 57 MiB/s wr, 905 op/s; 23437/316245 objects degraded (7.411%); 2.1 MiB/s, 34 objects/s recovering 2024-08-17T17:34:25.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:24 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 23546/306651 objects degraded (7.678%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:34:25.033 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:34:25.071 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:25.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:25.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:25.172 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:25.197 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:25.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:25.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:25.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:24 smithi053 ceph-mon[89963]: 1.6 scrub starts 2024-08-17T17:34:25.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:24 smithi053 ceph-mon[89963]: pgmap v845: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 57 MiB/s wr, 905 op/s; 23437/316245 objects degraded (7.411%); 2.1 MiB/s, 34 objects/s recovering 2024-08-17T17:34:25.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:24 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 23546/306651 objects degraded (7.678%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:34:25.299 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:25.324 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:25.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:25.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:25.382 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 89 16 106918 106902 75.0583 41.5 0.0277919 0.013316 2024-08-17T17:34:25.425 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:25.449 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:25.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:25.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:25.551 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:25.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:25.605 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:25.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:25.706 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:25.732 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:25.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:25.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:25.834 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:25.881 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:25.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:25.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:25.944 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:34:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:34:25] "GET /metrics HTTP/1.1" 200 36537 "" "Prometheus/2.51.0" 2024-08-17T17:34:25.983 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:26.091 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:26.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:26.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:26.193 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:26.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:25 smithi053 ceph-mon[89963]: 1.a scrub starts 2024-08-17T17:34:26.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:25 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:26.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:25 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:26.218 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:26.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:26.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:26.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:25 smithi003 ceph-mon[100615]: 1.a scrub starts 2024-08-17T17:34:26.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:25 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:26.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:25 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:26.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:25 smithi003 ceph-mon[103344]: 1.a scrub starts 2024-08-17T17:34:26.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:25 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:26.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:25 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:26.320 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:26.366 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:26.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:26.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:26.378 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 90 10 107472 107462 74.6132 35 0.0137415 0.013395 2024-08-17T17:34:26.469 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:26.491 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:26.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:26.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:26.593 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:26.616 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:26.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:26.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:26.718 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:26.745 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:26.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:26.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:26.765 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:34:26.847 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:26.870 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:26.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:26.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:26.972 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:26.996 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:26.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:26.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:27.098 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:27.124 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:27.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:27.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:27.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:26 smithi053 ceph-mon[89963]: 1.2 scrub starts 2024-08-17T17:34:27.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:26 smithi053 ceph-mon[89963]: pgmap v846: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 53 MiB/s wr, 846 op/s; 23437/317499 objects degraded (7.382%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T17:34:27.226 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:27.249 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:27.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:27.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:27.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:26 smithi003 ceph-mon[103344]: 1.2 scrub starts 2024-08-17T17:34:27.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:26 smithi003 ceph-mon[103344]: pgmap v846: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 53 MiB/s wr, 846 op/s; 23437/317499 objects degraded (7.382%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T17:34:27.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:26 smithi003 ceph-mon[100615]: 1.2 scrub starts 2024-08-17T17:34:27.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:26 smithi003 ceph-mon[100615]: pgmap v846: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 53 MiB/s wr, 846 op/s; 23437/317499 objects degraded (7.382%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T17:34:27.352 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:27.378 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total time run: 90.0702 2024-08-17T17:34:27.378 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total writes made: 107472 2024-08-17T17:34:27.378 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Write size: 65536 2024-08-17T17:34:27.378 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object size: 65536 2024-08-17T17:34:27.378 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Bandwidth (MB/sec): 74.5752 2024-08-17T17:34:27.378 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Bandwidth: 52.72 2024-08-17T17:34:27.378 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max bandwidth (MB/sec): 274.562 2024-08-17T17:34:27.378 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min bandwidth (MB/sec): 0 2024-08-17T17:34:27.379 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average IOPS: 1193 2024-08-17T17:34:27.379 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev IOPS: 843.52 2024-08-17T17:34:27.379 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max IOPS: 4393 2024-08-17T17:34:27.379 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min IOPS: 0 2024-08-17T17:34:27.379 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average Latency(s): 0.0134007 2024-08-17T17:34:27.379 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Latency(s): 0.0324263 2024-08-17T17:34:27.379 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max latency(s): 2.31183 2024-08-17T17:34:27.379 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min latency(s): 0.00109625 2024-08-17T17:34:27.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:27.380 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:27.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:27.384 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Cleaning up (deleting benchmark objects) 2024-08-17T17:34:27.481 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:27.513 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:27.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:27.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:27.615 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:27.659 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:27.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:27.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:27.761 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:27.785 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:27.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:27.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:27.887 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:27.933 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:27.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:27.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:28.035 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:28.060 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:28.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:28.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:28.161 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:28.204 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:28.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:28.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:28.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:27 smithi053 ceph-mon[89963]: 1.8 scrub starts 2024-08-17T17:34:28.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:27 smithi003 ceph-mon[103344]: 1.8 scrub starts 2024-08-17T17:34:28.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:27 smithi003 ceph-mon[100615]: 1.8 scrub starts 2024-08-17T17:34:28.306 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:28.335 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:28.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:28.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:28.436 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:28.472 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:28.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:28.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:28.574 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:28.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:28.608 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:28.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:28.709 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:28.736 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:28.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:28.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:28.837 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:28.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:28.875 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:28.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:28.976 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:29.007 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:29.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:29.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:29.109 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:29.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:29.155 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:29.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:29.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:28 smithi053 ceph-mon[89963]: 1.1 scrub starts 2024-08-17T17:34:29.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:28 smithi053 ceph-mon[89963]: pgmap v847: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 37 MiB/s wr, 586 op/s; 23437/317499 objects degraded (7.382%); 645 KiB/s, 10 objects/s recovering 2024-08-17T17:34:29.256 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:29.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:28 smithi003 ceph-mon[100615]: 1.1 scrub starts 2024-08-17T17:34:29.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:28 smithi003 ceph-mon[100615]: pgmap v847: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 37 MiB/s wr, 586 op/s; 23437/317499 objects degraded (7.382%); 645 KiB/s, 10 objects/s recovering 2024-08-17T17:34:29.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:28 smithi003 ceph-mon[103344]: 1.1 scrub starts 2024-08-17T17:34:29.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:28 smithi003 ceph-mon[103344]: pgmap v847: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 37 MiB/s wr, 586 op/s; 23437/317499 objects degraded (7.382%); 645 KiB/s, 10 objects/s recovering 2024-08-17T17:34:29.284 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:29.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:29.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:29.385 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:29.433 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:29.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:29.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:29.535 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:29.578 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:29.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:29.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:29.680 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:29.720 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:29.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:29.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:29.821 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:29.847 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:29.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:29.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:29.949 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:29.983 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:29.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:29.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:30.084 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:30.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:30.130 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:30.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:30.214 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:29 smithi053 ceph-mon[89963]: 1.9 scrub starts 2024-08-17T17:34:30.214 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:29 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 23437/317499 objects degraded (7.382%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:34:30.231 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:30.255 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:30.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:30.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:30.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:29 smithi003 ceph-mon[103344]: 1.9 scrub starts 2024-08-17T17:34:30.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:29 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 23437/317499 objects degraded (7.382%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:34:30.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:29 smithi003 ceph-mon[100615]: 1.9 scrub starts 2024-08-17T17:34:30.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:29 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 23437/317499 objects degraded (7.382%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:34:30.357 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:30.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:30.390 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:30.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:30.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:34:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:34:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:34:30.491 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:30.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:30.538 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:30.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:30.639 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:30.680 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:30.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:30.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:30.782 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:30.806 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:30.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:30.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:30.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:30.944 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:30.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:30.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:31.046 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:31.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:31.099 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:31.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:31.200 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:31.224 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:31.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:31.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:31.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:30 smithi003 ceph-mon[103344]: 1.7 scrub starts 2024-08-17T17:34:31.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:30 smithi003 ceph-mon[103344]: pgmap v848: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 35 MiB/s wr, 660 op/s; 23208/319344 objects degraded (7.267%); 2.0 MiB/s, 32 objects/s recovering 2024-08-17T17:34:31.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:30 smithi003 ceph-mon[100615]: 1.7 scrub starts 2024-08-17T17:34:31.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:30 smithi003 ceph-mon[100615]: pgmap v848: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 35 MiB/s wr, 660 op/s; 23208/319344 objects degraded (7.267%); 2.0 MiB/s, 32 objects/s recovering 2024-08-17T17:34:31.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:31.371 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:31.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:31.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:31.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:30 smithi053 ceph-mon[89963]: 1.7 scrub starts 2024-08-17T17:34:31.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:30 smithi053 ceph-mon[89963]: pgmap v848: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 35 MiB/s wr, 660 op/s; 23208/319344 objects degraded (7.267%); 2.0 MiB/s, 32 objects/s recovering 2024-08-17T17:34:31.474 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:31.502 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:31.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:31.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:31.604 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:31.659 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:31.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:31.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:31.761 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:31.812 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:31.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:31.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:31.914 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:31.943 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:31.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:31.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:32.045 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:32.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:32.085 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:32.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:32.186 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:32.247 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:32.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:32.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:32.266 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:31 smithi003 ceph-mon[103344]: 1.5 scrub starts 2024-08-17T17:34:32.266 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:31 smithi003 ceph-mon[103344]: 1.3 scrub starts 2024-08-17T17:34:32.266 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:31 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:34:32.266 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:31 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:34:32.266 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:31 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:32.266 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:31 smithi003 ceph-mon[100615]: 1.5 scrub starts 2024-08-17T17:34:32.266 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:31 smithi003 ceph-mon[100615]: 1.3 scrub starts 2024-08-17T17:34:32.267 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:31 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:34:32.267 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:31 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:34:32.267 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:31 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:32.349 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:32.379 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:32.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:32.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:32.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:31 smithi053 ceph-mon[89963]: 1.5 scrub starts 2024-08-17T17:34:32.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:31 smithi053 ceph-mon[89963]: 1.3 scrub starts 2024-08-17T17:34:32.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:31 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:34:32.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:31 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:34:32.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:31 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:34:32.481 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:32.526 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:32.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:32.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:32.628 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:32.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:32.671 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:32.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:32.772 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:32.795 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:32.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:32.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:32.897 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:32.938 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:32.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:32.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:33.071 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:33.109 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:33.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:33.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:33.211 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:33.233 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:32 smithi003 ceph-mon[103344]: pgmap v849: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 27 MiB/s wr, 526 op/s; 23208/319344 objects degraded (7.267%); 1.7 MiB/s, 28 objects/s recovering 2024-08-17T17:34:33.234 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:32 smithi003 ceph-mon[100615]: pgmap v849: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 27 MiB/s wr, 526 op/s; 23208/319344 objects degraded (7.267%); 1.7 MiB/s, 28 objects/s recovering 2024-08-17T17:34:33.235 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:33.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:33.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:33.337 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:33.377 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:33.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:33.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:33.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:32 smithi053 ceph-mon[89963]: pgmap v849: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 27 MiB/s wr, 526 op/s; 23208/319344 objects degraded (7.267%); 1.7 MiB/s, 28 objects/s recovering 2024-08-17T17:34:33.479 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:33.526 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:33.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:33.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:33.628 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:33.657 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:33.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:33.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:33.759 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:33.784 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:33.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:33.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:33.886 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:33.925 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:33.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:33.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:34.027 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:34.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:34.058 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:34.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:34.092 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:33 smithi003 ceph-mon[103344]: 1.6 scrub starts 2024-08-17T17:34:34.092 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:33 smithi003 ceph-mon[103344]: 1.a scrub starts 2024-08-17T17:34:34.092 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:33 smithi003 ceph-mon[100615]: 1.6 scrub starts 2024-08-17T17:34:34.092 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:33 smithi003 ceph-mon[100615]: 1.a scrub starts 2024-08-17T17:34:34.159 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:34.203 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:34.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:34.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:34.305 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:34.331 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:34.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:34.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:34.367 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:33 smithi053 ceph-mon[89963]: 1.6 scrub starts 2024-08-17T17:34:34.367 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:33 smithi053 ceph-mon[89963]: 1.a scrub starts 2024-08-17T17:34:34.369 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:34:34.369 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-08-17T17:34:34.369 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-08-17T17:34:34.369 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:34:34.370 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.370 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:56781 8.3 8.4bf49503 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e268)", 2024-08-17T17:34:34.370 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:19.807785+0000", 2024-08-17T17:34:34.370 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 74.558202178000002, 2024-08-17T17:34:34.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96137050199999996, 2024-08-17T17:34:34.370 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:34:34.370 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:34:34.370 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:34:34.370 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:34:34.370 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:34:34.370 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:34:34.370 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 56781 2024-08-17T17:34:34.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.371 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:34:34.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:34:34.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807785+0000", 2024-08-17T17:34:34.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:34:34.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807785+0000", 2024-08-17T17:34:34.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.372 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:34:34.372 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807786+0000", 2024-08-17T17:34:34.372 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4050000000000001e-06 2024-08-17T17:34:34.372 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.372 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:34:34.372 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807854+0000", 2024-08-17T17:34:34.372 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7267000000000005e-05 2024-08-17T17:34:34.372 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.372 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:34:34.372 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807865+0000", 2024-08-17T17:34:34.373 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1606999999999999e-05 2024-08-17T17:34:34.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.373 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.373 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:34:34.373 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807874+0000", 2024-08-17T17:34:34.373 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4829999999999999e-06 2024-08-17T17:34:34.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.373 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.373 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:34:34.373 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807937+0000", 2024-08-17T17:34:34.373 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3040999999999995e-05 2024-08-17T17:34:34.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.373 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.374 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:34:34.374 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807947+0000", 2024-08-17T17:34:34.374 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8379999999999997e-06 2024-08-17T17:34:34.374 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.374 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:34:34.374 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.769146+0000", 2024-08-17T17:34:34.374 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96119933899999999 2024-08-17T17:34:34.374 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.374 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:34:34.374 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.769155+0000", 2024-08-17T17:34:34.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5219999999999996e-06 2024-08-17T17:34:34.375 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.375 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:34:34.375 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.375 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:56793 8.3 8.83fbcd3 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e268)", 2024-08-17T17:34:34.375 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:19.807896+0000", 2024-08-17T17:34:34.375 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 74.558090768, 2024-08-17T17:34:34.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96128810799999997, 2024-08-17T17:34:34.375 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:34:34.375 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:34:34.376 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:34:34.376 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:34:34.376 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:34:34.376 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:34:34.376 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 56793 2024-08-17T17:34:34.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.376 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:34:34.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:34:34.376 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807896+0000", 2024-08-17T17:34:34.376 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:34:34.377 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807896+0000", 2024-08-17T17:34:34.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:34:34.377 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807897+0000", 2024-08-17T17:34:34.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8500000000000002e-07 2024-08-17T17:34:34.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:34:34.377 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807942+0000", 2024-08-17T17:34:34.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4977e-05 2024-08-17T17:34:34.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:34:34.378 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807950+0000", 2024-08-17T17:34:34.378 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9559999999999997e-06 2024-08-17T17:34:34.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:34:34.378 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807955+0000", 2024-08-17T17:34:34.378 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5509999999999999e-06 2024-08-17T17:34:34.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:34:34.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807997+0000", 2024-08-17T17:34:34.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2179999999999999e-05 2024-08-17T17:34:34.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:34:34.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808002+0000", 2024-08-17T17:34:34.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6639999999999997e-06 2024-08-17T17:34:34.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:34:34.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.769177+0000", 2024-08-17T17:34:34.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96117454300000005 2024-08-17T17:34:34.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.380 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:34:34.380 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.769184+0000", 2024-08-17T17:34:34.380 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7519999999999996e-06 2024-08-17T17:34:34.380 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.380 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:34:34.380 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.380 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.380 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:56797 8.a 8.229444aa (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e268)", 2024-08-17T17:34:34.380 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:19.807968+0000", 2024-08-17T17:34:34.380 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 74.558019145000003, 2024-08-17T17:34:34.380 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.95976723200000003, 2024-08-17T17:34:34.381 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:34:34.381 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:34:34.381 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:34:34.381 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:34:34.381 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:34:34.381 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:34:34.381 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 56797 2024-08-17T17:34:34.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.381 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:34:34.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.381 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:34:34.381 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807968+0000", 2024-08-17T17:34:34.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.382 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.382 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:34:34.382 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807968+0000", 2024-08-17T17:34:34.382 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.382 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.382 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:34:34.382 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807968+0000", 2024-08-17T17:34:34.382 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0900000000000002e-07 2024-08-17T17:34:34.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.382 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.382 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:34:34.382 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808012+0000", 2024-08-17T17:34:34.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3455000000000003e-05 2024-08-17T17:34:34.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:34:34.383 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808020+0000", 2024-08-17T17:34:34.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4409999999999993e-06 2024-08-17T17:34:34.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:34:34.383 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808025+0000", 2024-08-17T17:34:34.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9629999999999997e-06 2024-08-17T17:34:34.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:34:34.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808063+0000", 2024-08-17T17:34:34.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7626000000000002e-05 2024-08-17T17:34:34.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:34:34.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808067+0000", 2024-08-17T17:34:34.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0960000000000003e-06 2024-08-17T17:34:34.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:34:34.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.767725+0000", 2024-08-17T17:34:34.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.95965830900000004 2024-08-17T17:34:34.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.385 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.385 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:34:34.385 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.767735+0000", 2024-08-17T17:34:34.385 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9329999999999996e-06 2024-08-17T17:34:34.385 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.385 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:34:34.385 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.386 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:56804 8.3 8.fd779f13 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-08-17T17:34:34.386 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:19.808037+0000", 2024-08-17T17:34:34.386 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 74.557949788000002, 2024-08-17T17:34:34.386 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96117662699999995, 2024-08-17T17:34:34.386 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:34:34.386 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:34:34.386 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:34:34.386 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:34:34.386 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:34:34.386 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:34:34.386 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 56804 2024-08-17T17:34:34.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.387 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:34:34.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.387 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:34:34.387 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808037+0000", 2024-08-17T17:34:34.387 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.387 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:34:34.387 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808037+0000", 2024-08-17T17:34:34.387 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.388 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:34:34.388 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808038+0000", 2024-08-17T17:34:34.388 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2300000000000002e-07 2024-08-17T17:34:34.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.388 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:34:34.388 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808086+0000", 2024-08-17T17:34:34.388 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8529000000000002e-05 2024-08-17T17:34:34.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.388 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:34:34.388 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808096+0000", 2024-08-17T17:34:34.388 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3999999999999998e-06 2024-08-17T17:34:34.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.389 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:34:34.389 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808101+0000", 2024-08-17T17:34:34.389 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.575e-06 2024-08-17T17:34:34.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.389 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:34:34.389 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808148+0000", 2024-08-17T17:34:34.389 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6981000000000003e-05 2024-08-17T17:34:34.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.389 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:34:34.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808155+0000", 2024-08-17T17:34:34.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8340000000000001e-06 2024-08-17T17:34:34.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.390 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:34:34.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.769207+0000", 2024-08-17T17:34:34.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96105154500000001 2024-08-17T17:34:34.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.390 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:34:34.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.769214+0000", 2024-08-17T17:34:34.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.34e-06 2024-08-17T17:34:34.391 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.391 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:34:34.391 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.391 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:56808 8.3 8.634b5e43 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-08-17T17:34:34.391 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:19.808791+0000", 2024-08-17T17:34:34.391 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 74.557196466999997, 2024-08-17T17:34:34.391 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96046335800000004, 2024-08-17T17:34:34.391 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:34:34.391 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:34:34.391 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:34:34.391 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:34:34.392 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:34:34.392 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:34:34.392 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 56808 2024-08-17T17:34:34.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.392 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:34:34.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.392 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:34:34.392 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808791+0000", 2024-08-17T17:34:34.392 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.392 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:34:34.392 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808791+0000", 2024-08-17T17:34:34.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:34:34.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808791+0000", 2024-08-17T17:34:34.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7199999999999998e-07 2024-08-17T17:34:34.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:34:34.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808851+0000", 2024-08-17T17:34:34.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0016999999999998e-05 2024-08-17T17:34:34.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:34:34.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808862+0000", 2024-08-17T17:34:34.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0604e-05 2024-08-17T17:34:34.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:34:34.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808867+0000", 2024-08-17T17:34:34.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1959999999999999e-06 2024-08-17T17:34:34.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:34:34.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808909+0000", 2024-08-17T17:34:34.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1519e-05 2024-08-17T17:34:34.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:34:34.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808914+0000", 2024-08-17T17:34:34.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6930000000000004e-06 2024-08-17T17:34:34.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:34:34.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.769247+0000", 2024-08-17T17:34:34.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.960332356 2024-08-17T17:34:34.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:34:34.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.769254+0000", 2024-08-17T17:34:34.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2010000000000002e-06 2024-08-17T17:34:34.396 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.396 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:34:34.396 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.396 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:56812 8.7 8.b683b077 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-08-17T17:34:34.396 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:19.812057+0000", 2024-08-17T17:34:34.396 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 74.553930441000006, 2024-08-17T17:34:34.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.950741383, 2024-08-17T17:34:34.396 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:34:34.396 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:34:34.397 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:34:34.397 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:34:34.397 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:34:34.397 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:34:34.397 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 56812 2024-08-17T17:34:34.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.397 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:34:34.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:34:34.397 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.812057+0000", 2024-08-17T17:34:34.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:34:34.398 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.812057+0000", 2024-08-17T17:34:34.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.398 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:34:34.398 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.812058+0000", 2024-08-17T17:34:34.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0899999999999999e-06 2024-08-17T17:34:34.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.398 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:34:34.398 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.812100+0000", 2024-08-17T17:34:34.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2030000000000002e-05 2024-08-17T17:34:34.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:34:34.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.812112+0000", 2024-08-17T17:34:34.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.224e-05 2024-08-17T17:34:34.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:34:34.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.812120+0000", 2024-08-17T17:34:34.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0039999999999999e-06 2024-08-17T17:34:34.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:34:34.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.812179+0000", 2024-08-17T17:34:34.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9104000000000002e-05 2024-08-17T17:34:34.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:34:34.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.812184+0000", 2024-08-17T17:34:34.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0799999999999996e-06 2024-08-17T17:34:34.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:34:34.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.762788+0000", 2024-08-17T17:34:34.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.95060427300000006 2024-08-17T17:34:34.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:34:34.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.762798+0000", 2024-08-17T17:34:34.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5619999999999993e-06 2024-08-17T17:34:34.401 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.401 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:34:34.401 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.401 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:85958 8.b e274/269 8:d8a5509b:::benchmark_data_smithi003_229103_object85957:head v 274'5400, mlcod=0'0)", 2024-08-17T17:34:34.401 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:56.768913+0000", 2024-08-17T17:34:34.401 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 37.597073573000003, 2024-08-17T17:34:34.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.15166750000000001, 2024-08-17T17:34:34.402 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:34:34.402 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:34:34.402 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:34:34.402 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:34:34.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:34:34.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.768913+0000", 2024-08-17T17:34:34.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:34:34.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.768913+0000", 2024-08-17T17:34:34.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:34:34.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.768914+0000", 2024-08-17T17:34:34.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0200000000000001e-07 2024-08-17T17:34:34.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:34:34.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.768938+0000", 2024-08-17T17:34:34.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4090000000000001e-05 2024-08-17T17:34:34.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:34:34.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.768949+0000", 2024-08-17T17:34:34.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1197e-05 2024-08-17T17:34:34.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:34:34.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.768957+0000", 2024-08-17T17:34:34.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4610000000000002e-06 2024-08-17T17:34:34.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:34:34.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.769037+0000", 2024-08-17T17:34:34.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9574999999999996e-05 2024-08-17T17:34:34.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:34:34.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.769059+0000", 2024-08-17T17:34:34.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2779e-05 2024-08-17T17:34:34.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:34:34.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.920541+0000", 2024-08-17T17:34:34.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.15148128399999999 2024-08-17T17:34:34.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:34:34.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.920581+0000", 2024-08-17T17:34:34.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0411999999999999e-05 2024-08-17T17:34:34.406 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.406 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:34:34.406 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.406 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:85957 8.3 e274/269 8:c0f64a24:::benchmark_data_smithi003_229103_object85956:head v 274'5331, mlcod=0'0)", 2024-08-17T17:34:34.406 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:56.769588+0000", 2024-08-17T17:34:34.406 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 37.596398843999999, 2024-08-17T17:34:34.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.151026718, 2024-08-17T17:34:34.406 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:34:34.406 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:34:34.407 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:34:34.407 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:34:34.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:34:34.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.769588+0000", 2024-08-17T17:34:34.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:34:34.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.769588+0000", 2024-08-17T17:34:34.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:34:34.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.769589+0000", 2024-08-17T17:34:34.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8999999999999996e-07 2024-08-17T17:34:34.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:34:34.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.769620+0000", 2024-08-17T17:34:34.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1173000000000001e-05 2024-08-17T17:34:34.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:34:34.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.769631+0000", 2024-08-17T17:34:34.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.083e-05 2024-08-17T17:34:34.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:34:34.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.769638+0000", 2024-08-17T17:34:34.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8539999999999999e-06 2024-08-17T17:34:34.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:34:34.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.769703+0000", 2024-08-17T17:34:34.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5326000000000004e-05 2024-08-17T17:34:34.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:34:34.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.769724+0000", 2024-08-17T17:34:34.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0763000000000001e-05 2024-08-17T17:34:34.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:34:34.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.920586+0000", 2024-08-17T17:34:34.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.150861674 2024-08-17T17:34:34.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:34:34.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.920615+0000", 2024-08-17T17:34:34.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9408e-05 2024-08-17T17:34:34.411 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.411 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:34:34.411 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.411 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:85960 8.1 e274/269 8:8a84ea3f:::benchmark_data_smithi003_229103_object85959:head v 274'5435, mlcod=0'0)", 2024-08-17T17:34:34.411 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:56.771588+0000", 2024-08-17T17:34:34.411 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 37.594398781000002, 2024-08-17T17:34:34.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.148770771, 2024-08-17T17:34:34.411 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:34:34.411 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:34:34.411 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:34:34.411 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:34:34.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:34:34.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.771588+0000", 2024-08-17T17:34:34.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:34:34.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.771588+0000", 2024-08-17T17:34:34.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:34:34.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.771589+0000", 2024-08-17T17:34:34.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.217e-06 2024-08-17T17:34:34.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:34:34.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.771620+0000", 2024-08-17T17:34:34.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0923999999999997e-05 2024-08-17T17:34:34.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:34:34.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.771633+0000", 2024-08-17T17:34:34.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2510999999999999e-05 2024-08-17T17:34:34.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:34:34.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.771641+0000", 2024-08-17T17:34:34.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8210000000000002e-06 2024-08-17T17:34:34.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:34:34.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.771705+0000", 2024-08-17T17:34:34.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4269000000000004e-05 2024-08-17T17:34:34.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:34:34.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.771727+0000", 2024-08-17T17:34:34.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1857000000000001e-05 2024-08-17T17:34:34.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:34:34.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.776415+0000", 2024-08-17T17:34:34.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0046877150000000003 2024-08-17T17:34:34.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:34:34.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.920359+0000", 2024-08-17T17:34:34.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.143944457 2024-08-17T17:34:34.415 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.416 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:34:34.416 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.416 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:85959 8.b e274/269 8:d0920273:::benchmark_data_smithi003_229103_object85958:head v 274'5401, mlcod=0'0)", 2024-08-17T17:34:34.416 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:56.772299+0000", 2024-08-17T17:34:34.416 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 37.593687617999997, 2024-08-17T17:34:34.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.14833513600000001, 2024-08-17T17:34:34.416 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:34:34.416 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:34:34.417 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:34:34.417 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:34:34.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.417 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:34:34.417 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.772299+0000", 2024-08-17T17:34:34.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.417 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:34:34.417 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.772299+0000", 2024-08-17T17:34:34.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:34:34.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.772300+0000", 2024-08-17T17:34:34.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9800000000000003e-07 2024-08-17T17:34:34.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:34:34.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.772322+0000", 2024-08-17T17:34:34.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1888000000000001e-05 2024-08-17T17:34:34.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:34:34.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.772333+0000", 2024-08-17T17:34:34.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0702e-05 2024-08-17T17:34:34.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:34:34.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.772340+0000", 2024-08-17T17:34:34.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.844e-06 2024-08-17T17:34:34.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:34:34.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.774536+0000", 2024-08-17T17:34:34.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0021964490000000001 2024-08-17T17:34:34.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:34:34.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.774560+0000", 2024-08-17T17:34:34.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4065000000000001e-05 2024-08-17T17:34:34.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:34:34.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.920619+0000", 2024-08-17T17:34:34.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.14605860100000001 2024-08-17T17:34:34.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.421 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:34:34.421 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.920635+0000", 2024-08-17T17:34:34.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5889e-05 2024-08-17T17:34:34.421 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.421 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:34:34.421 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.421 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDPGPush(8.f 274/269 [PushOp(8:faa628b5:::benchmark_data_smithi003_229103_object13698:head, version: 265'884, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:faa628b5:::benchmark_data_smithi003_229103_object13698:head@265'884, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-17T17:34:34.421 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:56.779068+0000", 2024-08-17T17:34:34.422 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 37.586919518000002, 2024-08-17T17:34:34.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.14299867799999999, 2024-08-17T17:34:34.422 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:34:34.422 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:34:34.422 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:34:34.422 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:34:34.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:34:34.422 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.779068+0000", 2024-08-17T17:34:34.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.423 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:34:34.423 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.779068+0000", 2024-08-17T17:34:34.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.423 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:34:34.423 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.779069+0000", 2024-08-17T17:34:34.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.189e-06 2024-08-17T17:34:34.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.423 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:34:34.423 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.779097+0000", 2024-08-17T17:34:34.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8402999999999999e-05 2024-08-17T17:34:34.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.424 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:34:34.424 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.920331+0000", 2024-08-17T17:34:34.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.14123345800000001 2024-08-17T17:34:34.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.424 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:34:34.424 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.920367+0000", 2024-08-17T17:34:34.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6538000000000002e-05 2024-08-17T17:34:34.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.424 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:34:34.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.920454+0000", 2024-08-17T17:34:34.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6441999999999997e-05 2024-08-17T17:34:34.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:34:34.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.920468+0000", 2024-08-17T17:34:34.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4449e-05 2024-08-17T17:34:34.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:34:34.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.922066+0000", 2024-08-17T17:34:34.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001598199 2024-08-17T17:34:34.425 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.426 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:34:34.426 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.426 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.426 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:107138 8.1 e279/269 8:82567057:::benchmark_data_smithi003_229103_object107137:head v 279'6810, mlcod=0'0)", 2024-08-17T17:34:34.426 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:34:25.726290+0000", 2024-08-17T17:34:34.426 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 8.6396972680000008, 2024-08-17T17:34:34.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.099167660000000005, 2024-08-17T17:34:34.426 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:34:34.426 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:34:34.426 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:34:34.426 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:34:34.426 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.427 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:34:34.427 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.726290+0000", 2024-08-17T17:34:34.427 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.427 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:34:34.427 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.726290+0000", 2024-08-17T17:34:34.427 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.427 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:34:34.427 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.726291+0000", 2024-08-17T17:34:34.427 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4610000000000001e-06 2024-08-17T17:34:34.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:34:34.428 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.726321+0000", 2024-08-17T17:34:34.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0150000000000001e-05 2024-08-17T17:34:34.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:34:34.428 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.726335+0000", 2024-08-17T17:34:34.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3699e-05 2024-08-17T17:34:34.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:34:34.428 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.726344+0000", 2024-08-17T17:34:34.429 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8780000000000004e-06 2024-08-17T17:34:34.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.429 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.429 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:34:34.429 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.731169+0000", 2024-08-17T17:34:34.429 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0048248639999999999 2024-08-17T17:34:34.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.429 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.429 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:34:34.429 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.731224+0000", 2024-08-17T17:34:34.429 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5084000000000003e-05 2024-08-17T17:34:34.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.429 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.430 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:34:34.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.825216+0000", 2024-08-17T17:34:34.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.093991746000000001 2024-08-17T17:34:34.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.430 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:34:34.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.825457+0000", 2024-08-17T17:34:34.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000241778 2024-08-17T17:34:34.430 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.430 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:34:34.430 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.431 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:107139 8.1 e279/269 8:82a9d69d:::benchmark_data_smithi003_229103_object107138:head v 279'6811, mlcod=0'0)", 2024-08-17T17:34:34.431 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:34:25.730949+0000", 2024-08-17T17:34:34.431 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 8.6350384919999996, 2024-08-17T17:34:34.431 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.094726930000000001, 2024-08-17T17:34:34.431 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:34:34.431 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:34:34.431 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:34:34.431 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:34:34.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.431 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:34:34.431 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.730949+0000", 2024-08-17T17:34:34.431 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.432 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:34:34.432 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.730949+0000", 2024-08-17T17:34:34.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.432 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:34:34.432 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.730950+0000", 2024-08-17T17:34:34.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.003e-06 2024-08-17T17:34:34.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.432 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:34:34.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.730973+0000", 2024-08-17T17:34:34.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3734999999999999e-05 2024-08-17T17:34:34.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.433 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:34:34.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.730988+0000", 2024-08-17T17:34:34.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4384e-05 2024-08-17T17:34:34.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.433 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:34:34.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.730995+0000", 2024-08-17T17:34:34.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4189999999999996e-06 2024-08-17T17:34:34.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:34:34.434 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.759195+0000", 2024-08-17T17:34:34.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.028199607000000002 2024-08-17T17:34:34.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:34:34.434 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.759226+0000", 2024-08-17T17:34:34.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1127000000000002e-05 2024-08-17T17:34:34.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:34:34.435 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.825466+0000", 2024-08-17T17:34:34.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.066239929000000003 2024-08-17T17:34:34.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.435 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:34:34.435 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.825675+0000", 2024-08-17T17:34:34.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000209726 2024-08-17T17:34:34.435 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.435 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:34:34.435 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.435 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:107314 8.1 e279/269 8:83c95973:::benchmark_data_smithi003_229103_object107313:head v 279'6829, mlcod=0'0)", 2024-08-17T17:34:34.436 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:34:26.092005+0000", 2024-08-17T17:34:34.436 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 8.273982535, 2024-08-17T17:34:34.436 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.104778882, 2024-08-17T17:34:34.436 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:34:34.436 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:34:34.436 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:34:34.436 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:34:34.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.436 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:34:34.436 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.092005+0000", 2024-08-17T17:34:34.436 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:34:34.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.092005+0000", 2024-08-17T17:34:34.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:34:34.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.092006+0000", 2024-08-17T17:34:34.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.699e-06 2024-08-17T17:34:34.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:34:34.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.092426+0000", 2024-08-17T17:34:34.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00041939899999999999 2024-08-17T17:34:34.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:34:34.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.092442+0000", 2024-08-17T17:34:34.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6694e-05 2024-08-17T17:34:34.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:34:34.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.092452+0000", 2024-08-17T17:34:34.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3330000000000003e-06 2024-08-17T17:34:34.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:34:34.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.092656+0000", 2024-08-17T17:34:34.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020404599999999999 2024-08-17T17:34:34.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:34:34.439 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.092681+0000", 2024-08-17T17:34:34.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4941000000000001e-05 2024-08-17T17:34:34.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:34:34.439 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.196688+0000", 2024-08-17T17:34:34.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.104007219 2024-08-17T17:34:34.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:34:34.439 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.196783+0000", 2024-08-17T17:34:34.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5550999999999994e-05 2024-08-17T17:34:34.440 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.440 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:34:34.440 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.440 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:107324 8.1 e279/269 8:8addbb73:::benchmark_data_smithi003_229103_object107323:head v 279'6830, mlcod=0'0)", 2024-08-17T17:34:34.440 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:34:26.105580+0000", 2024-08-17T17:34:34.440 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 8.2604066500000002, 2024-08-17T17:34:34.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.091344203999999998, 2024-08-17T17:34:34.440 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:34:34.440 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:34:34.440 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:34:34.440 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:34:34.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:34:34.441 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.105580+0000", 2024-08-17T17:34:34.441 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.441 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:34:34.441 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.105580+0000", 2024-08-17T17:34:34.441 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.441 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:34:34.441 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.105582+0000", 2024-08-17T17:34:34.441 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6300000000000001e-06 2024-08-17T17:34:34.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.442 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:34:34.442 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.105612+0000", 2024-08-17T17:34:34.442 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0139000000000001e-05 2024-08-17T17:34:34.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.442 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:34:34.442 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.105628+0000", 2024-08-17T17:34:34.442 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6024000000000001e-05 2024-08-17T17:34:34.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.442 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:34:34.442 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.105638+0000", 2024-08-17T17:34:34.443 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9380000000000004e-06 2024-08-17T17:34:34.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.443 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:34:34.443 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.107772+0000", 2024-08-17T17:34:34.443 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0021340439999999999 2024-08-17T17:34:34.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.443 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:34:34.443 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.107794+0000", 2024-08-17T17:34:34.443 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2217000000000001e-05 2024-08-17T17:34:34.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.443 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:34:34.444 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.196788+0000", 2024-08-17T17:34:34.444 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.088993823 2024-08-17T17:34:34.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.444 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:34:34.444 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.196925+0000", 2024-08-17T17:34:34.444 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000136389 2024-08-17T17:34:34.444 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.444 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:34:34.444 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.444 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:107329 8.7 e279/269 8:e473a848:::benchmark_data_smithi003_229103_object107328:head v 279'6697, mlcod=0'0)", 2024-08-17T17:34:34.444 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:34:26.114751+0000", 2024-08-17T17:34:34.444 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 8.2512361139999992, 2024-08-17T17:34:34.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.079469463000000004, 2024-08-17T17:34:34.445 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:34:34.445 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:34:34.445 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:34.446 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:34:34.446 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:34:34.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.446 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:34:34.446 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.114751+0000", 2024-08-17T17:34:34.446 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.446 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:34:34.446 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.114751+0000", 2024-08-17T17:34:34.446 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.447 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:34:34.447 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.114752+0000", 2024-08-17T17:34:34.447 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0610000000000001e-06 2024-08-17T17:34:34.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.447 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:34:34.447 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.114773+0000", 2024-08-17T17:34:34.447 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1322e-05 2024-08-17T17:34:34.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.447 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:34:34.447 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.114786+0000", 2024-08-17T17:34:34.448 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2298e-05 2024-08-17T17:34:34.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.448 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:34:34.448 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.114793+0000", 2024-08-17T17:34:34.448 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5379999999999997e-06 2024-08-17T17:34:34.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.448 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:34:34.448 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.115009+0000", 2024-08-17T17:34:34.448 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000215909 2024-08-17T17:34:34.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.448 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:34:34.449 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.115031+0000", 2024-08-17T17:34:34.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1946999999999998e-05 2024-08-17T17:34:34.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.449 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:34:34.449 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.194026+0000", 2024-08-17T17:34:34.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.078995413 2024-08-17T17:34:34.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.449 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:34:34.449 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.194220+0000", 2024-08-17T17:34:34.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019397500000000001 2024-08-17T17:34:34.449 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.449 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:34:34.450 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.450 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.450 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.450 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:107322 8.f e279/269 8:ffaee6ac:::benchmark_data_smithi003_229103_object107321:head v 279'6792, mlcod=266'1578)", 2024-08-17T17:34:34.450 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:34:26.114822+0000", 2024-08-17T17:34:34.450 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 8.2511655220000009, 2024-08-17T17:34:34.450 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.078418020000000005, 2024-08-17T17:34:34.450 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:34:34.450 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:34:34.450 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:34:34.450 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:34:34.450 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.451 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:34:34.451 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.114822+0000", 2024-08-17T17:34:34.451 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.451 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:34:34.451 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.114822+0000", 2024-08-17T17:34:34.451 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.451 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:34:34.451 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.114822+0000", 2024-08-17T17:34:34.451 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1600000000000002e-07 2024-08-17T17:34:34.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.452 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:34:34.452 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.114844+0000", 2024-08-17T17:34:34.452 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2052000000000002e-05 2024-08-17T17:34:34.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.452 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:34:34.452 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.114851+0000", 2024-08-17T17:34:34.452 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2289999999999997e-06 2024-08-17T17:34:34.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.452 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:34:34.452 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.114856+0000", 2024-08-17T17:34:34.453 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6999999999999999e-06 2024-08-17T17:34:34.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.453 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:34:34.453 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.115698+0000", 2024-08-17T17:34:34.453 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00084218300000000004 2024-08-17T17:34:34.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.453 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:34:34.453 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.115717+0000", 2024-08-17T17:34:34.453 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9153000000000001e-05 2024-08-17T17:34:34.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.454 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:34:34.454 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.193064+0000", 2024-08-17T17:34:34.454 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.077347309000000003 2024-08-17T17:34:34.454 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.454 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.454 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:34:34.454 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.193240+0000", 2024-08-17T17:34:34.454 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017507800000000001 2024-08-17T17:34:34.454 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.454 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:34:34.454 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.454 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.454 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.455 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:107332 8.3 e279/269 8:c64c32db:::benchmark_data_smithi003_229103_object107331:head v 279'6693, mlcod=0'0)", 2024-08-17T17:34:34.455 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:34:26.114866+0000", 2024-08-17T17:34:34.455 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 8.2511206520000009, 2024-08-17T17:34:34.455 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.105011047, 2024-08-17T17:34:34.455 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:34:34.455 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:34:34.455 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:34:34.455 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:34:34.455 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.455 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:34:34.455 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.114866+0000", 2024-08-17T17:34:34.455 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.455 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.456 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:34:34.456 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.114866+0000", 2024-08-17T17:34:34.456 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.456 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:34:34.456 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.114867+0000", 2024-08-17T17:34:34.456 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5600000000000004e-07 2024-08-17T17:34:34.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.456 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:34:34.456 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.114909+0000", 2024-08-17T17:34:34.457 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1702999999999997e-05 2024-08-17T17:34:34.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.457 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:34:34.457 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.114922+0000", 2024-08-17T17:34:34.457 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3390000000000001e-05 2024-08-17T17:34:34.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.457 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:34:34.457 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.114931+0000", 2024-08-17T17:34:34.457 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8529999999999998e-06 2024-08-17T17:34:34.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.458 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:34:34.458 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.115011+0000", 2024-08-17T17:34:34.458 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9843000000000001e-05 2024-08-17T17:34:34.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.458 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:34:34.458 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.115033+0000", 2024-08-17T17:34:34.458 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2078000000000001e-05 2024-08-17T17:34:34.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.458 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:34:34.458 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.219809+0000", 2024-08-17T17:34:34.459 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.10477537200000001 2024-08-17T17:34:34.459 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.459 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:34:34.459 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.219877+0000", 2024-08-17T17:34:34.459 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8851999999999997e-05 2024-08-17T17:34:34.459 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.459 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:34:34.459 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.459 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.459 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:107326 8.1 e279/269 8:80c92084:::benchmark_data_smithi003_229103_object107325:head v 279'6831, mlcod=0'0)", 2024-08-17T17:34:34.459 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:34:26.114888+0000", 2024-08-17T17:34:34.460 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 8.2510987369999995, 2024-08-17T17:34:34.460 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.082164046000000004, 2024-08-17T17:34:34.460 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:34:34.460 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:34:34.460 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:34:34.460 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:34:34.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.460 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:34:34.460 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.114888+0000", 2024-08-17T17:34:34.460 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.460 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:34:34.460 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.114888+0000", 2024-08-17T17:34:34.461 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.461 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:34:34.461 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.114889+0000", 2024-08-17T17:34:34.461 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.58e-07 2024-08-17T17:34:34.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.461 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:34:34.461 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.114912+0000", 2024-08-17T17:34:34.461 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3421000000000002e-05 2024-08-17T17:34:34.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.461 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:34:34.462 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.114919+0000", 2024-08-17T17:34:34.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7920000000000004e-06 2024-08-17T17:34:34.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.462 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.462 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:34:34.462 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.114924+0000", 2024-08-17T17:34:34.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1170000000000003e-06 2024-08-17T17:34:34.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.462 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.462 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:34:34.462 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.180754+0000", 2024-08-17T17:34:34.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.065829848999999996 2024-08-17T17:34:34.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.463 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.463 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:34:34.463 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.180784+0000", 2024-08-17T17:34:34.463 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0139999999999999e-05 2024-08-17T17:34:34.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.463 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.463 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:34:34.463 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.196929+0000", 2024-08-17T17:34:34.463 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.016144901 2024-08-17T17:34:34.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.463 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.463 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:34:34.464 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.197052+0000", 2024-08-17T17:34:34.464 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000123368 2024-08-17T17:34:34.464 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.464 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:34:34.464 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.464 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:107343 8.3 e279/269 8:cb7c9311:::benchmark_data_smithi003_229103_object107342:head v 279'6694, mlcod=0'0)", 2024-08-17T17:34:34.464 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:34:26.129103+0000", 2024-08-17T17:34:34.464 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 8.2368841469999996, 2024-08-17T17:34:34.464 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.096487893000000005, 2024-08-17T17:34:34.464 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:34:34.464 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:34:34.465 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:34:34.465 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:34:34.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.465 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:34:34.465 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.129103+0000", 2024-08-17T17:34:34.465 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.465 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:34:34.465 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.129103+0000", 2024-08-17T17:34:34.465 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:34:34.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.466 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:34:34.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.129104+0000", 2024-08-17T17:34:34.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0020000000000001e-06 2024-08-17T17:34:34.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.466 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:34:34.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.129138+0000", 2024-08-17T17:34:34.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4545999999999998e-05 2024-08-17T17:34:34.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.466 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:34:34.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.182785+0000", 2024-08-17T17:34:34.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.053646294999999997 2024-08-17T17:34:34.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.467 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:34:34.467 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.182805+0000", 2024-08-17T17:34:34.467 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0024e-05 2024-08-17T17:34:34.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.467 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:34:34.467 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.184752+0000", 2024-08-17T17:34:34.467 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00194754 2024-08-17T17:34:34.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.467 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:34:34.468 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.184778+0000", 2024-08-17T17:34:34.468 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5476000000000001e-05 2024-08-17T17:34:34.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.468 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:34:34.468 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.225423+0000", 2024-08-17T17:34:34.468 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.040645674999999999 2024-08-17T17:34:34.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:34:34.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:34:34.468 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:34:34.468 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.225591+0000", 2024-08-17T17:34:34.468 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016733500000000001 2024-08-17T17:34:34.468 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.469 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:34:34.469 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.469 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:34:34.469 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:34:34.469 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:34:34.496 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:34.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:34.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:34.598 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:34.645 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:34.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:34.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:34.746 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:34.784 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:34.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:34.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:34.886 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:34.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:34.921 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:34.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:35.009 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-08-17T17:34:35.021 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:35.049 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:35.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:35.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:35.151 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:35.187 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:35.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:35.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:35.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:34 smithi053 ceph-mon[89963]: 1.2 scrub starts 2024-08-17T17:34:35.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:34 smithi053 ceph-mon[89963]: pgmap v850: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 27 MiB/s wr, 808 op/s; 23092/309177 objects degraded (7.469%); 2.3 MiB/s, 37 objects/s recovering 2024-08-17T17:34:35.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:34 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 23208/319344 objects degraded (7.267%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:34:35.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:34 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2166962519' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:34:35.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:34 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:34:35.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:35 smithi003 ceph-mon[103344]: 1.2 scrub starts 2024-08-17T17:34:35.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:35 smithi003 ceph-mon[103344]: pgmap v850: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 27 MiB/s wr, 808 op/s; 23092/309177 objects degraded (7.469%); 2.3 MiB/s, 37 objects/s recovering 2024-08-17T17:34:35.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:35 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 23208/319344 objects degraded (7.267%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:34:35.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:35 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2166962519' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:34:35.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:35 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:34:35.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:35 smithi003 ceph-mon[100615]: 1.2 scrub starts 2024-08-17T17:34:35.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:35 smithi003 ceph-mon[100615]: pgmap v850: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 27 MiB/s wr, 808 op/s; 23092/309177 objects degraded (7.469%); 2.3 MiB/s, 37 objects/s recovering 2024-08-17T17:34:35.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:35 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 23208/319344 objects degraded (7.267%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:34:35.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:35 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2166962519' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:34:35.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:35 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:34:35.289 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:35.321 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:35.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:35.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:35.423 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:35.457 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:35.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:35.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:35.559 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:35.597 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:35.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:35.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:35.681 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-17T17:34:35.699 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:35.731 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:35.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:35.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:35.833 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:35.862 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:35.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:35.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:35.913 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:34:35.964 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:35.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:35.992 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:35.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:36.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:36 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-08-17T17:34:36.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:36 smithi003 ceph-mon[100615]: osdmap e280: 8 total, 8 up, 8 in 2024-08-17T17:34:36.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:36 smithi003 ceph-mon[100615]: 1.8 scrub starts 2024-08-17T17:34:36.007 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:34:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:34:35] "GET /metrics HTTP/1.1" 200 36537 "" "Prometheus/2.51.0" 2024-08-17T17:34:36.093 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:36.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:36.151 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:36.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:36.252 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:36.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:36 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-08-17T17:34:36.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:36 smithi003 ceph-mon[103344]: osdmap e280: 8 total, 8 up, 8 in 2024-08-17T17:34:36.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:36 smithi003 ceph-mon[103344]: 1.8 scrub starts 2024-08-17T17:34:36.298 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:36.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:36.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:36.399 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:36.427 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:36.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:36.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:36.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:36 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-08-17T17:34:36.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:36 smithi053 ceph-mon[89963]: osdmap e280: 8 total, 8 up, 8 in 2024-08-17T17:34:36.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:36 smithi053 ceph-mon[89963]: 1.8 scrub starts 2024-08-17T17:34:36.528 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:36.553 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:36.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:36.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:36.655 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:36.700 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:36.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:36.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:36.778 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:34:36.802 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:36.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:36.870 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:36.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:36.971 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:36.997 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:36.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:36.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:37.039 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:37 smithi003 ceph-mon[103344]: pgmap v852: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 102 B/s rd, 10 MiB/s wr, 656 op/s; 23092/307809 objects degraded (7.502%); 2.2 MiB/s, 34 objects/s recovering 2024-08-17T17:34:37.039 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:37 smithi003 ceph-mon[100615]: pgmap v852: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 102 B/s rd, 10 MiB/s wr, 656 op/s; 23092/307809 objects degraded (7.502%); 2.2 MiB/s, 34 objects/s recovering 2024-08-17T17:34:37.099 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:37.123 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:37.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:37.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:37.225 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:37.274 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:37.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:37.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:37.375 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:37.424 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:37.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:37.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:37.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:37 smithi053 ceph-mon[89963]: pgmap v852: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 102 B/s rd, 10 MiB/s wr, 656 op/s; 23092/307809 objects degraded (7.502%); 2.2 MiB/s, 34 objects/s recovering 2024-08-17T17:34:37.526 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:37.548 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:37.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:37.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:37.650 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:37.677 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:37.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:37.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:37.779 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:37.828 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:37.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:37.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:37.930 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:37.971 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:37.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:37.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:37.989 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:34:38.074 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:38.101 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:38.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:38.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:38.203 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:38.249 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:38.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:38.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:38.272 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:38 smithi003 ceph-mon[100615]: 1.1 scrub starts 2024-08-17T17:34:38.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:38 smithi003 ceph-mon[103344]: 1.1 scrub starts 2024-08-17T17:34:38.351 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:38.378 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:38.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:38.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:38.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:38 smithi053 ceph-mon[89963]: 1.1 scrub starts 2024-08-17T17:34:38.480 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:38.526 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:38.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:38.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:38.628 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:38.655 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:38.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:38.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:38.757 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:38.793 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:38.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:38.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:38.895 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:38.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:38.927 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:38.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:39.028 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:39.068 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:39.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:39.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:39.170 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:39.222 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:39.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:39.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:39.293 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:39 smithi003 ceph-mon[103344]: 1.9 scrub starts 2024-08-17T17:34:39.293 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:34:39.293 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:39 smithi003 ceph-mon[103344]: pgmap v853: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 102 B/s rd, 10 MiB/s wr, 656 op/s; 23092/307809 objects degraded (7.502%); 2.2 MiB/s, 34 objects/s recovering 2024-08-17T17:34:39.293 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:39 smithi003 ceph-mon[100615]: 1.9 scrub starts 2024-08-17T17:34:39.293 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:34:39.293 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:39 smithi003 ceph-mon[100615]: pgmap v853: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 102 B/s rd, 10 MiB/s wr, 656 op/s; 23092/307809 objects degraded (7.502%); 2.2 MiB/s, 34 objects/s recovering 2024-08-17T17:34:39.327 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:39.349 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:39.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:39.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:39.451 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:39.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:39 smithi053 ceph-mon[89963]: 1.9 scrub starts 2024-08-17T17:34:39.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:34:39.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:39 smithi053 ceph-mon[89963]: pgmap v853: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 102 B/s rd, 10 MiB/s wr, 656 op/s; 23092/307809 objects degraded (7.502%); 2.2 MiB/s, 34 objects/s recovering 2024-08-17T17:34:39.478 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:39.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:39.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:39.579 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:39.610 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:39.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:39.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:39.711 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:39.737 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:39.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:39.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:39.839 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:39.886 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:39.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:39.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:39.988 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:40.050 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:40.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:40.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:40.155 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:40.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:40 smithi003 ceph-mon[100615]: 1.7 scrub starts 2024-08-17T17:34:40.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:40 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 23092/307809 objects degraded (7.502%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:34:40.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:40 smithi003 ceph-mon[103344]: 1.7 scrub starts 2024-08-17T17:34:40.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:40 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 23092/307809 objects degraded (7.502%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:34:40.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:40.194 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:40.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:40.295 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:40.322 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:40.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:40.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:40.424 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:40.450 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:40.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:40.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:40.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:34:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:34:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:34:40.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:40 smithi053 ceph-mon[89963]: 1.7 scrub starts 2024-08-17T17:34:40.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:40 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 23092/307809 objects degraded (7.502%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:34:40.553 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:40.593 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:40.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:40.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:40.694 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:40.741 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:40.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:40.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:40.843 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:40.889 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:40.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:40.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:40.991 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:41.031 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:41.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:41.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:41.058 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-08-17T17:34:41.133 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:41.197 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:41.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:41.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:41.299 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:41.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:41.331 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:41.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:41.414 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:41 smithi003 ceph-mon[100615]: 1.5 scrub starts 2024-08-17T17:34:41.414 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:41 smithi003 ceph-mon[100615]: pgmap v854: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 753 op/s; 22966/296715 objects degraded (7.740%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:34:41.415 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:41 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1717836172' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:34:41.415 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:41 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:34:41.415 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:41 smithi003 ceph-mon[103344]: 1.5 scrub starts 2024-08-17T17:34:41.415 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:41 smithi003 ceph-mon[103344]: pgmap v854: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 753 op/s; 22966/296715 objects degraded (7.740%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:34:41.415 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:41 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1717836172' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:34:41.415 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:41 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:34:41.432 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:41.461 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:41.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:41.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:41.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:41 smithi053 ceph-mon[89963]: 1.5 scrub starts 2024-08-17T17:34:41.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:41 smithi053 ceph-mon[89963]: pgmap v854: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 753 op/s; 22966/296715 objects degraded (7.740%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:34:41.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:41 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1717836172' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:34:41.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:41 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:34:41.550 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-08-17T17:34:41.550 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:34:41.551 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-08-17T17:34:41.551 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-17T17:34:41.551 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-17T17:34:41.551 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-08-17T17:34:41.563 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:41.655 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:41.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:41.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:41.757 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:41.804 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:41.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:41.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:41.906 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:41.932 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:41.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:41.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:42.033 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:42.071 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:42.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:42.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:42.173 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:42.215 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:42.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:42.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:42.317 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:42.356 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:42.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:42.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:42.413 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:42 smithi003 ceph-mon[103344]: 1.3 scrub starts 2024-08-17T17:34:42.413 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:42 smithi003 ceph-mon[103344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:34:42.413 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:42 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:34:42.413 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:42 smithi003 ceph-mon[103344]: osdmap e281: 8 total, 8 up, 8 in 2024-08-17T17:34:42.414 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:42 smithi003 ceph-mon[100615]: 1.3 scrub starts 2024-08-17T17:34:42.414 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:42 smithi003 ceph-mon[100615]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:34:42.414 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:42 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:34:42.414 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:42 smithi003 ceph-mon[100615]: osdmap e281: 8 total, 8 up, 8 in 2024-08-17T17:34:42.459 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:42.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:42 smithi053 ceph-mon[89963]: 1.3 scrub starts 2024-08-17T17:34:42.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:42 smithi053 ceph-mon[89963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:34:42.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:42 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:34:42.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:42 smithi053 ceph-mon[89963]: osdmap e281: 8 total, 8 up, 8 in 2024-08-17T17:34:42.510 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:42.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:42.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:42.612 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:42.641 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:42.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:42.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:42.742 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:42.782 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:42.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:42.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:42.884 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:42.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:42.913 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:42.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:43.014 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:43.056 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:43.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:43.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:43.158 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:43.189 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:43.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:43.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:43.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:43 smithi003 ceph-mon[103344]: 1.6 scrub starts 2024-08-17T17:34:43.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:43 smithi003 ceph-mon[103344]: pgmap v856: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 519 op/s; 22966/296715 objects degraded (7.740%); 1008 KiB/s, 15 objects/s recovering 2024-08-17T17:34:43.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:43 smithi003 ceph-mon[100615]: 1.6 scrub starts 2024-08-17T17:34:43.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:43 smithi003 ceph-mon[100615]: pgmap v856: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 519 op/s; 22966/296715 objects degraded (7.740%); 1008 KiB/s, 15 objects/s recovering 2024-08-17T17:34:43.291 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:43.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:43.331 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:43.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:43.432 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:43.455 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:43.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:43.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:43.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:43 smithi053 ceph-mon[89963]: 1.6 scrub starts 2024-08-17T17:34:43.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:43 smithi053 ceph-mon[89963]: pgmap v856: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 88 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 519 op/s; 22966/296715 objects degraded (7.740%); 1008 KiB/s, 15 objects/s recovering 2024-08-17T17:34:43.558 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:43.603 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:43.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:43.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:43.705 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:43.729 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:43.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:43.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:43.831 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:43.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:43.860 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:43.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:43.961 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:43.987 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:43.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:43.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:44.089 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:44.120 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:44.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:44.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:44.221 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:44.249 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:44.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:44.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:44.351 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:44.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:44.383 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:44.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:44.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:44 smithi053 ceph-mon[89963]: 1.a scrub starts 2024-08-17T17:34:44.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:44 smithi053 ceph-mon[89963]: 1.2 scrub starts 2024-08-17T17:34:44.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:44 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3325298878' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:34:44.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:44 smithi053 ceph-mon[89963]: 1.8 scrub starts 2024-08-17T17:34:44.484 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:44.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:44 smithi003 ceph-mon[100615]: 1.a scrub starts 2024-08-17T17:34:44.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:44 smithi003 ceph-mon[100615]: 1.2 scrub starts 2024-08-17T17:34:44.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:44 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3325298878' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:34:44.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:44 smithi003 ceph-mon[100615]: 1.8 scrub starts 2024-08-17T17:34:44.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:44 smithi003 ceph-mon[103344]: 1.a scrub starts 2024-08-17T17:34:44.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:44 smithi003 ceph-mon[103344]: 1.2 scrub starts 2024-08-17T17:34:44.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:44 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3325298878' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:34:44.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:44 smithi003 ceph-mon[103344]: 1.8 scrub starts 2024-08-17T17:34:44.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:44.527 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:44.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:44.566 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:34:44.566 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:34:44.566 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:34:44.566 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:34:44.566 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:34:44.566 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:34:44.566 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:34:44.566 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:34:44.567 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:34:44.567 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:34:44.567 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:34:44.567 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:34:44.567 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:34:44.568 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:34:44.568 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:34:44.568 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:34:44.628 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:44.668 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:44.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:44.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:44.770 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:44.799 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:44.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:44.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:44.901 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:44.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:44.930 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:44.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:45.031 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:45.059 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:45.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:45.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:45.161 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:45.185 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:45.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:45.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:45.286 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:45.316 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:45.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:45.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:45.418 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:45.448 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:45.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:45.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:45.451 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:45 smithi053 ceph-mon[89963]: pgmap v857: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 747 op/s; 22847/287379 objects degraded (7.950%); 1.7 MiB/s, 26 objects/s recovering 2024-08-17T17:34:45.451 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:45 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 22966/296715 objects degraded (7.740%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:34:45.452 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:45 smithi053 ceph-mon[89963]: 1.1 scrub starts 2024-08-17T17:34:45.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:45 smithi003 ceph-mon[100615]: pgmap v857: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 747 op/s; 22847/287379 objects degraded (7.950%); 1.7 MiB/s, 26 objects/s recovering 2024-08-17T17:34:45.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:45 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 22966/296715 objects degraded (7.740%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:34:45.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:45 smithi003 ceph-mon[100615]: 1.1 scrub starts 2024-08-17T17:34:45.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:45 smithi003 ceph-mon[103344]: pgmap v857: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 747 op/s; 22847/287379 objects degraded (7.950%); 1.7 MiB/s, 26 objects/s recovering 2024-08-17T17:34:45.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:45 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 22966/296715 objects degraded (7.740%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:34:45.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:45 smithi003 ceph-mon[103344]: 1.1 scrub starts 2024-08-17T17:34:45.550 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:45.578 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:45.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:45.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:45.680 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:45.713 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:45.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:45.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:45.814 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:45.840 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:45.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:45.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:45.913 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:34:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:34:45] "GET /metrics HTTP/1.1" 200 36535 "" "Prometheus/2.51.0" 2024-08-17T17:34:45.942 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:45.970 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:45.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:45.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:46.072 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:46.098 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:46.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:46.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:46.191 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:46 smithi003 ceph-mon[100615]: 1.9 scrub starts 2024-08-17T17:34:46.192 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:34:46.192 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:34:46.192 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:46 smithi003 ceph-mon[103344]: 1.9 scrub starts 2024-08-17T17:34:46.199 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:46.239 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:46.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:46.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:46.341 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:46.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:46.367 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:46.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:46.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:46 smithi053 ceph-mon[89963]: 1.9 scrub starts 2024-08-17T17:34:46.469 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:46.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:46.516 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:46.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:46.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:46.653 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:46.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:46.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:46.755 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:46.798 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:46.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:46.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:46.900 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:46.929 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:46.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:46.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:47.031 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:47.072 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:47.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:47.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:47.174 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:47.218 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:47.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:47.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:47.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:47 smithi003 ceph-mon[103344]: pgmap v858: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 724 op/s; 22847/286071 objects degraded (7.986%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:34:47.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:47 smithi003 ceph-mon[103344]: 1.7 scrub starts 2024-08-17T17:34:47.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:47 smithi003 ceph-mon[100615]: pgmap v858: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 724 op/s; 22847/286071 objects degraded (7.986%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:34:47.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:47 smithi003 ceph-mon[100615]: 1.7 scrub starts 2024-08-17T17:34:47.320 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:47.349 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:47.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:47.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:47.451 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:47.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:47 smithi053 ceph-mon[89963]: pgmap v858: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 724 op/s; 22847/286071 objects degraded (7.986%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:34:47.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:47 smithi053 ceph-mon[89963]: 1.7 scrub starts 2024-08-17T17:34:47.478 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:47.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:47.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:47.580 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:47.624 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:47.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:47.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:47.726 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:47.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:47.760 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:47.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:47.861 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:47.903 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:47.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:47.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:48.005 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:48.034 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:48.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:48.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:48.074 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-17T17:34:48.127 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:34:48.136 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:48.162 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:48.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:48.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:48.264 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:48.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:48 smithi003 ceph-mon[100615]: 1.5 scrub starts 2024-08-17T17:34:48.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:48 smithi003 ceph-mon[103344]: 1.5 scrub starts 2024-08-17T17:34:48.288 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:48.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:48.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:48.390 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:48.418 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:48.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:48.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:48.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:48 smithi053 ceph-mon[89963]: 1.5 scrub starts 2024-08-17T17:34:48.520 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:48.567 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:48.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:48.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:48.669 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:48.696 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:48.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:48.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:48.798 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:48.837 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:48.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:48.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:48.939 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:48.964 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:48.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:48.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:48.986 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:34:49.066 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:49.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:49.116 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:49.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:49.217 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:49.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:49.259 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:49.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:49.328 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:49 smithi003 ceph-mon[100615]: pgmap v859: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 724 op/s; 22847/286071 objects degraded (7.986%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:34:49.328 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:49 smithi003 ceph-mon[100615]: 1.3 scrub starts 2024-08-17T17:34:49.330 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:49 smithi003 ceph-mon[103344]: pgmap v859: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 724 op/s; 22847/286071 objects degraded (7.986%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:34:49.330 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:49 smithi003 ceph-mon[103344]: 1.3 scrub starts 2024-08-17T17:34:49.359 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:49.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:49.406 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:49.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:49.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:49 smithi053 ceph-mon[89963]: pgmap v859: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 89 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 724 op/s; 22847/286071 objects degraded (7.986%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:34:49.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:49 smithi053 ceph-mon[89963]: 1.3 scrub starts 2024-08-17T17:34:49.507 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:49.536 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:49.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:49.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:49.607 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:34:49.640 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:49.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:49.666 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:49.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:49.708 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:34:49.767 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:49.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:49.813 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:49.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:49.914 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:49.944 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:49.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:49.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:49.997 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:34:50.046 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:50.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:50.117 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:50.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:50.219 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:50.243 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:50.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:50.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:50.252 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:50 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 22847/286071 objects degraded (7.986%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:34:50.252 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:50 smithi003 ceph-mon[100615]: 1.6 scrub starts 2024-08-17T17:34:50.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:50 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 22847/286071 objects degraded (7.986%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:34:50.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:50 smithi003 ceph-mon[103344]: 1.6 scrub starts 2024-08-17T17:34:50.346 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:50.390 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:50.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:50.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:50.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:50 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 22847/286071 objects degraded (7.986%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:34:50.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:50 smithi053 ceph-mon[89963]: 1.6 scrub starts 2024-08-17T17:34:50.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:34:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:34:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:34:50.492 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:50.520 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:50.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:50.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:50.621 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:50.672 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:50.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:50.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:50.774 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:50.794 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:34:50.794 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-17T17:34:50.794 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:34:50.822 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:50.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:50.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:50.923 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:50.951 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:50.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:50.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:51.053 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:51.077 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:51.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:51.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:51.179 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:51.207 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:51.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:51.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:51.309 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:51.351 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:51.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:51.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:51.453 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:51.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:51 smithi053 ceph-mon[89963]: pgmap v860: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 728 op/s; 22714/274860 objects degraded (8.264%); 1.6 MiB/s, 25 objects/s recovering 2024-08-17T17:34:51.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:51 smithi053 ceph-mon[89963]: 1.a scrub starts 2024-08-17T17:34:51.477 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:51.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:51.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:51.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:51 smithi003 ceph-mon[103344]: pgmap v860: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 728 op/s; 22714/274860 objects degraded (8.264%); 1.6 MiB/s, 25 objects/s recovering 2024-08-17T17:34:51.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:51 smithi003 ceph-mon[103344]: 1.a scrub starts 2024-08-17T17:34:51.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:51 smithi003 ceph-mon[100615]: pgmap v860: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 728 op/s; 22714/274860 objects degraded (8.264%); 1.6 MiB/s, 25 objects/s recovering 2024-08-17T17:34:51.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:51 smithi003 ceph-mon[100615]: 1.a scrub starts 2024-08-17T17:34:51.580 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:51.612 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:51.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:51.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:51.713 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:51.742 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:51.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:51.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:51.844 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:51.885 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:51.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:51.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:51.987 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:52.010 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:52.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:52.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:52.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:52.156 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:52.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:52.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:52.258 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:52.286 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:52.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:52.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:52.388 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:52.410 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:52.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:52.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:52.512 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:52.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:52.564 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:52.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:52.665 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:52.694 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:52.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:52.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:52.796 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:52.824 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:52.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:52.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:52.927 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:52.951 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:52.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:52.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:53.054 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:53.082 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:53.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:53.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:53.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:53.229 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:53.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:53.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:53.330 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:53.376 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:53.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:53.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:53.424 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:53 smithi003 ceph-mon[100615]: 1.2 scrub starts 2024-08-17T17:34:53.425 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:53 smithi003 ceph-mon[100615]: pgmap v861: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 658 op/s; 22714/274860 objects degraded (8.264%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:34:53.425 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:34:53.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:53 smithi003 ceph-mon[103344]: 1.2 scrub starts 2024-08-17T17:34:53.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:53 smithi003 ceph-mon[103344]: pgmap v861: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 658 op/s; 22714/274860 objects degraded (8.264%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:34:53.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:34:53.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:53 smithi053 ceph-mon[89963]: 1.2 scrub starts 2024-08-17T17:34:53.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:53 smithi053 ceph-mon[89963]: pgmap v861: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 5.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 658 op/s; 22714/274860 objects degraded (8.264%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:34:53.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:34:53.478 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:53.510 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:53.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:53.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:53.612 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:53.640 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:53.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:53.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:53.742 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:53.767 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:53.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:53.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:53.869 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:53.909 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:53.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:53.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:54.011 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:54.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:54.065 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:54.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:54.166 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:54.192 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:54.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:54.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:54.294 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:54.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:54.346 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:54.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:54.447 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:54.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:54 smithi053 ceph-mon[89963]: 1.8 scrub starts 2024-08-17T17:34:54.474 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:54.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:54.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:54.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:54 smithi003 ceph-mon[103344]: 1.8 scrub starts 2024-08-17T17:34:54.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:54 smithi003 ceph-mon[100615]: 1.8 scrub starts 2024-08-17T17:34:54.577 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:54.616 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:54.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:54.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:54.718 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:54.761 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:54.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:54.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:54.862 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:54.889 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:54.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:54.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:54.991 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:55.017 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:55.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:55.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:55.119 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:55.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:55.200 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:55.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:55.237 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:34:55.301 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:55.318 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:55 smithi003 ceph-mon[103344]: pgmap v862: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 851 op/s; 22564/266040 objects degraded (8.481%); 2.1 MiB/s, 33 objects/s recovering 2024-08-17T17:34:55.318 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:55 smithi003 ceph-mon[103344]: 1.1 scrub starts 2024-08-17T17:34:55.318 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:55 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 22564/266040 objects degraded (8.481%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:34:55.319 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:55 smithi003 ceph-mon[100615]: pgmap v862: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 851 op/s; 22564/266040 objects degraded (8.481%); 2.1 MiB/s, 33 objects/s recovering 2024-08-17T17:34:55.320 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:55 smithi003 ceph-mon[100615]: 1.1 scrub starts 2024-08-17T17:34:55.320 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:55 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 22564/266040 objects degraded (8.481%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:34:55.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:55.348 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:55.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:55.449 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:55.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:55 smithi053 ceph-mon[89963]: pgmap v862: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 851 op/s; 22564/266040 objects degraded (8.481%); 2.1 MiB/s, 33 objects/s recovering 2024-08-17T17:34:55.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:55 smithi053 ceph-mon[89963]: 1.1 scrub starts 2024-08-17T17:34:55.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:55 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 22564/266040 objects degraded (8.481%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:34:55.479 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:55.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:55.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:55.580 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:55.609 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:55.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:55.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:55.711 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:55.763 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:55.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:55.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:55.865 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:55.890 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:55.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:55.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:55.952 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:34:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:34:55] "GET /metrics HTTP/1.1" 200 36535 "" "Prometheus/2.51.0" 2024-08-17T17:34:55.992 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:56.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:56.045 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:56.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:56.052 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:34:56.052 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:34:56.146 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:56.173 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:56.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:56.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:56.213 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:56 smithi003 ceph-mon[103344]: 1.9 scrub starts 2024-08-17T17:34:56.213 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:56 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3527767749' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:34:56.214 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:56 smithi003 ceph-mon[100615]: 1.9 scrub starts 2024-08-17T17:34:56.214 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:56 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3527767749' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:34:56.274 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:56.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:56.323 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:56.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:56.424 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:56.449 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:56.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:56.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:56.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:56 smithi053 ceph-mon[89963]: 1.9 scrub starts 2024-08-17T17:34:56.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:56 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3527767749' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:34:56.552 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:56.582 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:56.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:56.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:56.684 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:56.713 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:56.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:56.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:56.815 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:56.832 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-17T17:34:56.842 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:56.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:56.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:56.884 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:34:56.884 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:34:56.884 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:34:56.884 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:34:56.884 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:34:56.884 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:34:56.885 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:34:56.885 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:34:56.885 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:34:56.885 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:34:56.885 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:34:56.885 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:34:56.885 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:34:56.885 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:34:56.885 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:34:56.885 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:34:56.944 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:56.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:56.975 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:56.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:57.076 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:57.102 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:57.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:57.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:57.179 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:57 smithi003 ceph-mon[100615]: pgmap v863: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 628 op/s; 22564/264735 objects degraded (8.523%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:34:57.180 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:57 smithi003 ceph-mon[100615]: 1.7 scrub starts 2024-08-17T17:34:57.203 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:57.227 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:57.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:57.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:57.329 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:57.356 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:57.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:57.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:57.458 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:57.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:57 smithi053 ceph-mon[89963]: pgmap v863: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 628 op/s; 22564/264735 objects degraded (8.523%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:34:57.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:57 smithi053 ceph-mon[89963]: 1.7 scrub starts 2024-08-17T17:34:57.486 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:57.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:57.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:57.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:57 smithi003 ceph-mon[103344]: pgmap v863: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 628 op/s; 22564/264735 objects degraded (8.523%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:34:57.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:57 smithi003 ceph-mon[103344]: 1.7 scrub starts 2024-08-17T17:34:57.588 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:57.615 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:57.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:57.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:57.716 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:57.761 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:57.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:57.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:57.863 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:57.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:57.918 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:57.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:58.018 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:34:58.069 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:58.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:58.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:34:58.170 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:58.215 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:58.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:58.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:58.317 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:58.349 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:58.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:58.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:58.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:58.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:58 smithi053 ceph-mon[89963]: 1.5 scrub starts 2024-08-17T17:34:58.482 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:58 smithi003 ceph-mon[100615]: 1.5 scrub starts 2024-08-17T17:34:58.482 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:58 smithi003 ceph-mon[103344]: 1.5 scrub starts 2024-08-17T17:34:58.496 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:58.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:58.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:58.599 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:34:58.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:58.640 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:58.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:34:58.741 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:58.768 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:58.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:58.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:58.870 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:58.897 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:58.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:58.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:58.999 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:34:59.024 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:59.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:59.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:34:59.107 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:34:59.126 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:34:59.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:59.192 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:59.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:34:59.223 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-08-17T17:34:59.223 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:34:59.223 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-08-17T17:34:59.223 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:34:59.260 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:59 smithi003 ceph-mon[103344]: pgmap v864: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 592 op/s; 22564/264735 objects degraded (8.523%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:34:59.261 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:34:59 smithi003 ceph-mon[103344]: 1.3 scrub starts 2024-08-17T17:34:59.261 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:59 smithi003 ceph-mon[100615]: pgmap v864: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 592 op/s; 22564/264735 objects degraded (8.523%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:34:59.261 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:34:59 smithi003 ceph-mon[100615]: 1.3 scrub starts 2024-08-17T17:34:59.293 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:34:59.321 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:59.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:59.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:34:59.423 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:59.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:59 smithi053 ceph-mon[89963]: pgmap v864: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 592 op/s; 22564/264735 objects degraded (8.523%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:34:59.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:34:59 smithi053 ceph-mon[89963]: 1.3 scrub starts 2024-08-17T17:34:59.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:59.483 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:59.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:34:59.584 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:34:59.614 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:59.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:59.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:34:59.716 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:34:59.746 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:34:59.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:59.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:34:59.847 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:34:59.902 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:34:59.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:34:59.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:00.004 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:00.054 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:00.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:00.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:00.156 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:00.183 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:00.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:00.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:00.219 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-08-17T17:35:00.284 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:00.312 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:00.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:00.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:00.413 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:00.444 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:00.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:00.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:00.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:35:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:35:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:35:00.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:00 smithi053 ceph-mon[89963]: 1.6 scrub starts 2024-08-17T17:35:00.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:00 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 22564/264735 objects degraded (8.523%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:00.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:00 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/932701202' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:35:00.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:00 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:35:00.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:00 smithi003 ceph-mon[103344]: 1.6 scrub starts 2024-08-17T17:35:00.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:00 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 22564/264735 objects degraded (8.523%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:00.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:00 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/932701202' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:35:00.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:00 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:35:00.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:00 smithi003 ceph-mon[100615]: 1.6 scrub starts 2024-08-17T17:35:00.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:00 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 22564/264735 objects degraded (8.523%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:00.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:00 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/932701202' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:35:00.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:00 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:35:00.527 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:35:00.547 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:00.627 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:00.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:00.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:00.729 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:00.753 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:00.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:00.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:00.854 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:00.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:00.879 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:00.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:00.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:01.036 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:01.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:01.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:01.138 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:01.170 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:01.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:01.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:01.272 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:01.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:01.301 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:01.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:01.402 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:01.441 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:01.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:01.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:01.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:01 smithi003 ceph-mon[103344]: pgmap v865: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.2 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 884 op/s; 22437/254208 objects degraded (8.826%); 2.1 MiB/s, 34 objects/s recovering 2024-08-17T17:35:01.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:01 smithi003 ceph-mon[103344]: 1.a scrub starts 2024-08-17T17:35:01.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:01 smithi003 ceph-mon[103344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:35:01.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:01 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:35:01.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:01 smithi003 ceph-mon[103344]: osdmap e282: 8 total, 8 up, 8 in 2024-08-17T17:35:01.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:01 smithi003 ceph-mon[100615]: pgmap v865: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.2 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 884 op/s; 22437/254208 objects degraded (8.826%); 2.1 MiB/s, 34 objects/s recovering 2024-08-17T17:35:01.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:01 smithi003 ceph-mon[100615]: 1.a scrub starts 2024-08-17T17:35:01.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:01 smithi003 ceph-mon[100615]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:35:01.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:01 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:35:01.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:01 smithi003 ceph-mon[100615]: osdmap e282: 8 total, 8 up, 8 in 2024-08-17T17:35:01.543 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:01.583 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:01 smithi053 ceph-mon[89963]: pgmap v865: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.2 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 884 op/s; 22437/254208 objects degraded (8.826%); 2.1 MiB/s, 34 objects/s recovering 2024-08-17T17:35:01.583 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:01 smithi053 ceph-mon[89963]: 1.a scrub starts 2024-08-17T17:35:01.583 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:01 smithi053 ceph-mon[89963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:35:01.583 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:01 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:35:01.583 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:01 smithi053 ceph-mon[89963]: osdmap e282: 8 total, 8 up, 8 in 2024-08-17T17:35:01.603 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:01.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:01.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:01.705 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:01.733 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:01.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:01.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:01.835 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:01.860 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:01.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:01.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:01.961 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:01.991 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:01.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:01.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:02.092 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:02.117 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:35:02.119 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:02.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:02.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:02.221 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:02.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:02.273 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:02.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:02.374 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:02.401 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:02.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:02.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:02.483 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:02 smithi003 ceph-mon[103344]: 1.2 scrub starts 2024-08-17T17:35:02.483 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:02 smithi003 ceph-mon[103344]: 1.2 scrub ok 2024-08-17T17:35:02.484 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:02 smithi003 ceph-mon[100615]: 1.2 scrub starts 2024-08-17T17:35:02.484 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:02 smithi003 ceph-mon[100615]: 1.2 scrub ok 2024-08-17T17:35:02.502 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:02.539 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:02.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:02.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:02.641 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:02.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:02.696 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:02.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:02.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:02 smithi053 ceph-mon[89963]: 1.2 scrub starts 2024-08-17T17:35:02.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:02 smithi053 ceph-mon[89963]: 1.2 scrub ok 2024-08-17T17:35:02.797 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:02.826 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:02.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:02.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:02.928 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:02.977 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:02.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:02.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:03.079 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:03.130 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:03.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:03.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:03.232 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:03.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:03.260 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:03.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:03.361 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:03.385 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:03.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:03.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:03.387 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:35:03.487 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:03.516 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:03.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:03.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:03.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:03 smithi003 ceph-mon[100615]: pgmap v867: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.2 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 688 op/s; 22437/254208 objects degraded (8.826%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T17:35:03.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:03 smithi003 ceph-mon[100615]: 1.8 scrub starts 2024-08-17T17:35:03.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:03 smithi003 ceph-mon[100615]: 1.8 scrub ok 2024-08-17T17:35:03.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:03 smithi003 ceph-mon[103344]: pgmap v867: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.2 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 688 op/s; 22437/254208 objects degraded (8.826%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T17:35:03.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:03 smithi003 ceph-mon[103344]: 1.8 scrub starts 2024-08-17T17:35:03.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:03 smithi003 ceph-mon[103344]: 1.8 scrub ok 2024-08-17T17:35:03.618 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:03.650 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:03.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:03.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:03.748 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:03 smithi053 ceph-mon[89963]: pgmap v867: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 9 active+clean+scrubbing, 6 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 5.2 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 688 op/s; 22437/254208 objects degraded (8.826%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T17:35:03.749 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:03 smithi053 ceph-mon[89963]: 1.8 scrub starts 2024-08-17T17:35:03.749 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:03 smithi053 ceph-mon[89963]: 1.8 scrub ok 2024-08-17T17:35:03.849 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:03.876 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:03.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:03.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:03.978 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:04.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:04.029 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:04.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:04.082 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:35:04.131 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:04.187 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:04.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:04.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:04.213 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:35:04.213 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-08-17T17:35:04.213 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-08-17T17:35:04.213 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:35:04.213 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.213 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45491.0:160772 6.0 6:0639d6cf:::benchmark_data_smithi003_175043_object62123:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-08-17T17:35:04.213 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:27:20.702861+0000", 2024-08-17T17:35:04.214 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 463.50658963799998, 2024-08-17T17:35:04.214 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.23765629799999999, 2024-08-17T17:35:04.214 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:35:04.214 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:35:04.214 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:35:04.214 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:35:04.214 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45491", 2024-08-17T17:35:04.214 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:35:04.214 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 160772 2024-08-17T17:35:04.214 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.214 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:35:04.214 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.214 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:35:04.215 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.702861+0000", 2024-08-17T17:35:04.215 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.215 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.215 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:35:04.215 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.702861+0000", 2024-08-17T17:35:04.215 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.215 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.215 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:35:04.215 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.702864+0000", 2024-08-17T17:35:04.215 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1390000000000002e-06 2024-08-17T17:35:04.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.215 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.215 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:35:04.216 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.702869+0000", 2024-08-17T17:35:04.216 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4280000000000004e-06 2024-08-17T17:35:04.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.216 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:35:04.216 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.702891+0000", 2024-08-17T17:35:04.216 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2172999999999999e-05 2024-08-17T17:35:04.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.216 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:35:04.216 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.702901+0000", 2024-08-17T17:35:04.216 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6010000000000001e-06 2024-08-17T17:35:04.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.217 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:35:04.217 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.709233+0000", 2024-08-17T17:35:04.217 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063318230000000003 2024-08-17T17:35:04.217 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.217 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.217 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:35:04.217 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.709361+0000", 2024-08-17T17:35:04.217 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012887200000000001 2024-08-17T17:35:04.217 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.217 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.217 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3", 2024-08-17T17:35:04.217 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.709485+0000", 2024-08-17T17:35:04.217 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012318600000000001 2024-08-17T17:35:04.218 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.218 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.218 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:35:04.218 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.713843+0000", 2024-08-17T17:35:04.218 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0043586609999999998 2024-08-17T17:35:04.218 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.218 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.218 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.218 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.737302+0000", 2024-08-17T17:35:04.218 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.023458265999999998 2024-08-17T17:35:04.218 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.218 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.218 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.218 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.940421+0000", 2024-08-17T17:35:04.219 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.20311971500000001 2024-08-17T17:35:04.219 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.219 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.219 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:35:04.219 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.940472+0000", 2024-08-17T17:35:04.219 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1215999999999999e-05 2024-08-17T17:35:04.219 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.219 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.219 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:35:04.220 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.940518+0000", 2024-08-17T17:35:04.220 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5218e-05 2024-08-17T17:35:04.220 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.220 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:35:04.220 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.221 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.221 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.221 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45491.0:160779 6.9 6:9d3f1440:::benchmark_data_smithi003_175043_object62130:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-08-17T17:35:04.221 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:27:20.717781+0000", 2024-08-17T17:35:04.221 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 463.49167045799999, 2024-08-17T17:35:04.221 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.21854267699999999, 2024-08-17T17:35:04.222 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:35:04.222 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:35:04.222 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:35:04.222 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:35:04.222 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45491", 2024-08-17T17:35:04.222 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:35:04.222 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 160779 2024-08-17T17:35:04.223 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.223 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:35:04.223 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.223 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:35:04.223 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.717781+0000", 2024-08-17T17:35:04.223 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.223 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.223 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.224 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:35:04.224 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.717781+0000", 2024-08-17T17:35:04.224 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.224 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.224 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.224 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:35:04.224 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.717783+0000", 2024-08-17T17:35:04.225 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.114e-06 2024-08-17T17:35:04.225 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.225 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.225 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:35:04.225 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.717789+0000", 2024-08-17T17:35:04.225 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6830000000000002e-06 2024-08-17T17:35:04.225 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.225 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.225 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:35:04.225 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.717804+0000", 2024-08-17T17:35:04.226 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4348000000000001e-05 2024-08-17T17:35:04.226 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.226 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.226 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:35:04.226 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.717812+0000", 2024-08-17T17:35:04.226 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5350000000000004e-06 2024-08-17T17:35:04.226 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.226 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.226 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:35:04.226 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.717881+0000", 2024-08-17T17:35:04.226 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8867000000000003e-05 2024-08-17T17:35:04.226 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.226 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.227 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:35:04.227 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.718158+0000", 2024-08-17T17:35:04.227 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00027689199999999999 2024-08-17T17:35:04.227 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.227 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.227 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:35:04.227 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.718571+0000", 2024-08-17T17:35:04.227 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00041280199999999998 2024-08-17T17:35:04.227 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.227 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.227 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.228 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.734795+0000", 2024-08-17T17:35:04.228 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.016223807999999999 2024-08-17T17:35:04.228 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.228 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.228 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:35:04.228 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.739588+0000", 2024-08-17T17:35:04.228 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00479365 2024-08-17T17:35:04.228 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.228 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.228 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.228 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.936222+0000", 2024-08-17T17:35:04.228 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.19663338499999999 2024-08-17T17:35:04.228 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.228 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.229 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:35:04.229 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.936279+0000", 2024-08-17T17:35:04.229 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7426999999999998e-05 2024-08-17T17:35:04.229 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.229 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.229 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:35:04.229 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.936323+0000", 2024-08-17T17:35:04.229 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4165999999999997e-05 2024-08-17T17:35:04.229 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.229 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:35:04.229 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.229 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.229 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.229 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45491.0:160780 6.9 6:9f7045a7:::benchmark_data_smithi003_175043_object62131:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-08-17T17:35:04.230 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:27:20.720442+0000", 2024-08-17T17:35:04.230 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 463.48900898400001, 2024-08-17T17:35:04.230 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.23174434899999999, 2024-08-17T17:35:04.230 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:35:04.230 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:35:04.230 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:35:04.231 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:35:04.231 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45491", 2024-08-17T17:35:04.231 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:35:04.232 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 160780 2024-08-17T17:35:04.232 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.233 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:35:04.233 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.233 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:35:04.233 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.720442+0000", 2024-08-17T17:35:04.233 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.233 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.233 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.233 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:35:04.234 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.720442+0000", 2024-08-17T17:35:04.234 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.234 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.234 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.234 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:35:04.235 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.720444+0000", 2024-08-17T17:35:04.235 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7889999999999999e-06 2024-08-17T17:35:04.235 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.235 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.235 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:35:04.235 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.720451+0000", 2024-08-17T17:35:04.235 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3540000000000002e-06 2024-08-17T17:35:04.235 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.235 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.235 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:35:04.235 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.720471+0000", 2024-08-17T17:35:04.235 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9556999999999999e-05 2024-08-17T17:35:04.235 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.236 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.236 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:35:04.236 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.720481+0000", 2024-08-17T17:35:04.236 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0419e-05 2024-08-17T17:35:04.236 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.236 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.236 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:35:04.236 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.725322+0000", 2024-08-17T17:35:04.236 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0048407249999999997 2024-08-17T17:35:04.236 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.236 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.236 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:35:04.236 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.725672+0000", 2024-08-17T17:35:04.237 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00035019899999999999 2024-08-17T17:35:04.237 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.237 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.237 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:35:04.237 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.726101+0000", 2024-08-17T17:35:04.237 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00042927699999999999 2024-08-17T17:35:04.237 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.237 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.237 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.237 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.734840+0000", 2024-08-17T17:35:04.237 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0087386300000000007 2024-08-17T17:35:04.237 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.237 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.237 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:35:04.238 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.739596+0000", 2024-08-17T17:35:04.238 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0047558419999999997 2024-08-17T17:35:04.238 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.238 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.238 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.238 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.951863+0000", 2024-08-17T17:35:04.238 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.212267492 2024-08-17T17:35:04.238 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.238 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.238 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:35:04.238 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.952036+0000", 2024-08-17T17:35:04.238 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017293500000000001 2024-08-17T17:35:04.238 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.239 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.239 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:35:04.239 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.952186+0000", 2024-08-17T17:35:04.239 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015013 2024-08-17T17:35:04.239 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.239 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:35:04.239 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.239 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.239 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.239 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45491.0:160781 6.0 6:047167c6:::benchmark_data_smithi003_175043_object62132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-08-17T17:35:04.239 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:27:20.721935+0000", 2024-08-17T17:35:04.240 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 463.48751616800001, 2024-08-17T17:35:04.240 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.22536682999999999, 2024-08-17T17:35:04.240 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:35:04.240 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:35:04.240 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:35:04.240 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:35:04.240 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45491", 2024-08-17T17:35:04.240 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:35:04.240 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 160781 2024-08-17T17:35:04.240 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.241 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:35:04.241 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.241 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:35:04.241 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.721935+0000", 2024-08-17T17:35:04.241 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.241 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.241 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.241 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:35:04.241 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.721935+0000", 2024-08-17T17:35:04.241 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.241 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.241 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.241 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:35:04.241 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.721936+0000", 2024-08-17T17:35:04.242 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.47e-07 2024-08-17T17:35:04.242 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.242 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.242 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:35:04.242 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.721940+0000", 2024-08-17T17:35:04.242 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.498e-06 2024-08-17T17:35:04.242 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.243 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.243 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:35:04.243 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.721963+0000", 2024-08-17T17:35:04.243 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2614000000000001e-05 2024-08-17T17:35:04.243 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.243 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.243 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:35:04.243 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.721978+0000", 2024-08-17T17:35:04.243 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4772e-05 2024-08-17T17:35:04.243 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.243 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.243 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:35:04.243 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.722549+0000", 2024-08-17T17:35:04.243 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00057140299999999995 2024-08-17T17:35:04.243 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.244 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.244 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:35:04.244 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.722668+0000", 2024-08-17T17:35:04.244 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000118585 2024-08-17T17:35:04.244 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.244 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.244 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3", 2024-08-17T17:35:04.244 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.722790+0000", 2024-08-17T17:35:04.244 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012234400000000001 2024-08-17T17:35:04.244 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.244 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.244 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:35:04.244 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.735534+0000", 2024-08-17T17:35:04.244 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012744313 2024-08-17T17:35:04.244 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.244 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.245 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.245 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.737405+0000", 2024-08-17T17:35:04.245 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0018705919999999999 2024-08-17T17:35:04.245 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.245 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.245 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.245 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.947003+0000", 2024-08-17T17:35:04.245 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.20959830600000001 2024-08-17T17:35:04.245 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.245 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.245 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:35:04.245 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.947063+0000", 2024-08-17T17:35:04.246 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0235999999999997e-05 2024-08-17T17:35:04.246 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.246 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.246 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:35:04.246 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.947302+0000", 2024-08-17T17:35:04.246 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00023822000000000001 2024-08-17T17:35:04.246 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.246 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:35:04.246 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.246 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.246 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.246 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45491.0:160782 6.0 6:0b386992:::benchmark_data_smithi003_175043_object62133:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-08-17T17:35:04.246 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:27:20.722728+0000", 2024-08-17T17:35:04.247 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 463.48672333600001, 2024-08-17T17:35:04.247 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.24380597800000001, 2024-08-17T17:35:04.247 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:35:04.247 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:35:04.247 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:35:04.247 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:35:04.247 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45491", 2024-08-17T17:35:04.247 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:35:04.247 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 160782 2024-08-17T17:35:04.247 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.247 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:35:04.247 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.248 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:35:04.248 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.722728+0000", 2024-08-17T17:35:04.248 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.248 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.248 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.248 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:35:04.248 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.722728+0000", 2024-08-17T17:35:04.248 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.248 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.248 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.248 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:35:04.248 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.722728+0000", 2024-08-17T17:35:04.248 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2600000000000002e-07 2024-08-17T17:35:04.248 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.248 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.249 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:35:04.249 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.722731+0000", 2024-08-17T17:35:04.249 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0319999999999999e-06 2024-08-17T17:35:04.249 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.249 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.249 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:35:04.249 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.722740+0000", 2024-08-17T17:35:04.249 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2190000000000004e-06 2024-08-17T17:35:04.249 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.249 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.249 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:35:04.249 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.722745+0000", 2024-08-17T17:35:04.249 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7239999999999997e-06 2024-08-17T17:35:04.249 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.249 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.250 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:35:04.250 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.726927+0000", 2024-08-17T17:35:04.250 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0041816099999999997 2024-08-17T17:35:04.250 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.250 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.250 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:35:04.250 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.727233+0000", 2024-08-17T17:35:04.250 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00030585299999999997 2024-08-17T17:35:04.250 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.250 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.250 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3", 2024-08-17T17:35:04.250 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.727611+0000", 2024-08-17T17:35:04.250 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00037850900000000002 2024-08-17T17:35:04.251 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.251 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.251 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:35:04.251 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.735544+0000", 2024-08-17T17:35:04.251 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0079324549999999997 2024-08-17T17:35:04.251 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.251 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.251 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.251 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.737471+0000", 2024-08-17T17:35:04.251 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0019272689999999999 2024-08-17T17:35:04.251 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.251 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.251 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.251 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.966423+0000", 2024-08-17T17:35:04.252 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.228952081 2024-08-17T17:35:04.252 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.252 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.252 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:35:04.252 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.966484+0000", 2024-08-17T17:35:04.252 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0609000000000002e-05 2024-08-17T17:35:04.252 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.252 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.252 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:35:04.252 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.966534+0000", 2024-08-17T17:35:04.252 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9991000000000003e-05 2024-08-17T17:35:04.252 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.252 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:35:04.253 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.253 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.253 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.253 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45491.0:160783 6.9 6:91dbffd2:::benchmark_data_smithi003_175043_object62134:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-08-17T17:35:04.253 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:27:20.722753+0000", 2024-08-17T17:35:04.253 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 463.48669850900001, 2024-08-17T17:35:04.253 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.231241845, 2024-08-17T17:35:04.253 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:35:04.253 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:35:04.253 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:35:04.253 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:35:04.253 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45491", 2024-08-17T17:35:04.253 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:35:04.253 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 160783 2024-08-17T17:35:04.253 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.254 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:35:04.254 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.254 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:35:04.254 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.722753+0000", 2024-08-17T17:35:04.254 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.254 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.254 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.254 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:35:04.254 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.722753+0000", 2024-08-17T17:35:04.254 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.254 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.254 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.254 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:35:04.254 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.722753+0000", 2024-08-17T17:35:04.255 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.01e-07 2024-08-17T17:35:04.255 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.255 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.255 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:35:04.255 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.722755+0000", 2024-08-17T17:35:04.255 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8679999999999999e-06 2024-08-17T17:35:04.255 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.255 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.255 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:35:04.255 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.722761+0000", 2024-08-17T17:35:04.255 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8030000000000002e-06 2024-08-17T17:35:04.255 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.255 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.255 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:35:04.255 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.722765+0000", 2024-08-17T17:35:04.256 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6659999999999999e-06 2024-08-17T17:35:04.256 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.256 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.256 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:35:04.256 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.735169+0000", 2024-08-17T17:35:04.256 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012404188999999999 2024-08-17T17:35:04.256 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.256 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.256 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:35:04.256 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.735342+0000", 2024-08-17T17:35:04.257 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017291999999999999 2024-08-17T17:35:04.257 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.257 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.257 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:35:04.257 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.735502+0000", 2024-08-17T17:35:04.257 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015948899999999999 2024-08-17T17:35:04.257 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.257 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.257 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.257 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.739634+0000", 2024-08-17T17:35:04.257 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0041319909999999998 2024-08-17T17:35:04.258 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.258 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.258 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:35:04.258 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.739714+0000", 2024-08-17T17:35:04.258 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0038999999999997e-05 2024-08-17T17:35:04.259 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.259 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.259 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.259 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.953877+0000", 2024-08-17T17:35:04.259 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.214163459 2024-08-17T17:35:04.260 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.260 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.260 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:35:04.260 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.953943+0000", 2024-08-17T17:35:04.260 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5285999999999998e-05 2024-08-17T17:35:04.260 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.260 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.261 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:35:04.261 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:27:20.953994+0000", 2024-08-17T17:35:04.261 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1734000000000003e-05 2024-08-17T17:35:04.261 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.261 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:35:04.261 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.262 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.262 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.262 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45491.0:191504 6.9 6:9fae1e44:::benchmark_data_smithi003_175043_object92855:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-08-17T17:35:04.263 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:28:00.551284+0000", 2024-08-17T17:35:04.263 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 423.65816720800001, 2024-08-17T17:35:04.263 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.26288574799999997, 2024-08-17T17:35:04.263 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:35:04.263 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:35:04.263 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:35:04.263 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:35:04.263 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45491", 2024-08-17T17:35:04.263 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:35:04.263 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 191504 2024-08-17T17:35:04.263 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.263 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:35:04.263 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.264 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:35:04.264 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.551284+0000", 2024-08-17T17:35:04.264 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.264 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.264 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.264 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:35:04.264 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.551284+0000", 2024-08-17T17:35:04.264 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.264 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.264 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.264 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:35:04.264 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.551286+0000", 2024-08-17T17:35:04.264 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7409999999999999e-06 2024-08-17T17:35:04.264 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.265 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.265 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:35:04.265 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.551292+0000", 2024-08-17T17:35:04.265 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9379999999999997e-06 2024-08-17T17:35:04.265 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.265 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.265 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:35:04.265 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.551306+0000", 2024-08-17T17:35:04.265 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4703e-05 2024-08-17T17:35:04.265 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.265 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.265 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:35:04.265 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.551315+0000", 2024-08-17T17:35:04.265 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4279999999999996e-06 2024-08-17T17:35:04.265 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.266 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.266 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:35:04.266 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.551489+0000", 2024-08-17T17:35:04.266 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017430000000000001 2024-08-17T17:35:04.266 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.266 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.266 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:35:04.266 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.551601+0000", 2024-08-17T17:35:04.266 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000112218 2024-08-17T17:35:04.266 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.266 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.266 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:35:04.267 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.551705+0000", 2024-08-17T17:35:04.267 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000103658 2024-08-17T17:35:04.267 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.267 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.267 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:35:04.267 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.556056+0000", 2024-08-17T17:35:04.267 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0043507700000000003 2024-08-17T17:35:04.267 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.267 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.267 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.267 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.559250+0000", 2024-08-17T17:35:04.267 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031944550000000001 2024-08-17T17:35:04.267 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.268 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.268 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.268 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.814022+0000", 2024-08-17T17:35:04.268 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.25477233999999999 2024-08-17T17:35:04.268 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.268 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.268 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:35:04.268 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.814121+0000", 2024-08-17T17:35:04.268 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8799000000000001e-05 2024-08-17T17:35:04.268 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.268 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.268 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:35:04.268 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.814170+0000", 2024-08-17T17:35:04.268 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8398e-05 2024-08-17T17:35:04.269 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.269 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:35:04.269 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.269 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.269 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.269 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45491.0:191517 6.a e220/214 6:5090b9fe:::benchmark_data_smithi003_175043_object92868:head v 220'11904, mlcod=220'11902)", 2024-08-17T17:35:04.269 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:28:00.567359+0000", 2024-08-17T17:35:04.269 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 423.64209213100003, 2024-08-17T17:35:04.269 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.24201456700000001, 2024-08-17T17:35:04.269 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:35:04.269 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:35:04.270 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:35:04.270 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:35:04.270 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.270 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:35:04.270 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.567359+0000", 2024-08-17T17:35:04.270 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.270 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.270 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.270 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:35:04.270 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.567359+0000", 2024-08-17T17:35:04.270 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.270 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.270 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.271 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:35:04.271 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.567360+0000", 2024-08-17T17:35:04.271 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.474e-06 2024-08-17T17:35:04.271 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.271 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.271 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:35:04.271 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.567371+0000", 2024-08-17T17:35:04.271 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1046999999999999e-05 2024-08-17T17:35:04.271 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.271 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.271 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:35:04.271 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.567383+0000", 2024-08-17T17:35:04.271 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1663e-05 2024-08-17T17:35:04.271 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.271 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.272 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:35:04.272 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.567392+0000", 2024-08-17T17:35:04.272 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6400000000000003e-06 2024-08-17T17:35:04.272 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.272 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.272 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:35:04.272 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.567539+0000", 2024-08-17T17:35:04.272 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014705400000000001 2024-08-17T17:35:04.272 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.272 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.272 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:35:04.272 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.567561+0000", 2024-08-17T17:35:04.272 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2118999999999999e-05 2024-08-17T17:35:04.272 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.272 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.273 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:35:04.273 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.809326+0000", 2024-08-17T17:35:04.273 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.24176547100000001 2024-08-17T17:35:04.273 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.273 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.273 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:35:04.273 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.809373+0000", 2024-08-17T17:35:04.273 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7098999999999997e-05 2024-08-17T17:35:04.273 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.273 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:35:04.273 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.273 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.273 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.273 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45491.0:191525 6.9 6:990fd26b:::benchmark_data_smithi003_175043_object92876:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-08-17T17:35:04.273 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:28:00.570283+0000", 2024-08-17T17:35:04.274 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 423.63916782899997, 2024-08-17T17:35:04.274 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.244092686, 2024-08-17T17:35:04.274 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:35:04.274 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:35:04.274 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:35:04.274 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:35:04.274 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45491", 2024-08-17T17:35:04.274 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2645673232", 2024-08-17T17:35:04.274 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 191525 2024-08-17T17:35:04.274 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.274 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:35:04.274 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.275 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:35:04.275 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.570283+0000", 2024-08-17T17:35:04.275 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.275 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.275 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.275 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:35:04.275 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.570283+0000", 2024-08-17T17:35:04.275 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.275 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.275 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.275 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:35:04.275 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.570285+0000", 2024-08-17T17:35:04.276 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.44e-06 2024-08-17T17:35:04.276 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.276 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.276 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:35:04.276 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.570291+0000", 2024-08-17T17:35:04.276 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6120000000000004e-06 2024-08-17T17:35:04.276 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.276 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.276 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:35:04.276 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.570307+0000", 2024-08-17T17:35:04.276 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5922000000000001e-05 2024-08-17T17:35:04.276 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.276 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.276 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:35:04.277 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.570316+0000", 2024-08-17T17:35:04.277 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7979999999999995e-06 2024-08-17T17:35:04.277 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.277 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.277 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:35:04.277 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.570376+0000", 2024-08-17T17:35:04.277 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0198999999999997e-05 2024-08-17T17:35:04.277 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.277 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.277 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:35:04.277 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.570489+0000", 2024-08-17T17:35:04.277 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011271 2024-08-17T17:35:04.277 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.277 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.277 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:35:04.278 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.570607+0000", 2024-08-17T17:35:04.278 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011849900000000001 2024-08-17T17:35:04.278 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.278 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.278 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:35:04.278 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.808963+0000", 2024-08-17T17:35:04.278 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.23835577099999999 2024-08-17T17:35:04.278 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.278 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.278 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.278 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.809227+0000", 2024-08-17T17:35:04.278 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00026338700000000002 2024-08-17T17:35:04.278 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.278 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.279 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.279 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.814257+0000", 2024-08-17T17:35:04.279 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0050304579999999998 2024-08-17T17:35:04.279 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.279 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.279 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:35:04.279 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.814340+0000", 2024-08-17T17:35:04.279 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3177999999999995e-05 2024-08-17T17:35:04.279 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.279 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.279 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:35:04.279 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.814376+0000", 2024-08-17T17:35:04.279 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5712e-05 2024-08-17T17:35:04.279 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.279 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:35:04.280 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.280 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.280 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.281 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45491.0:191524 6.a e220/214 6:5b98c4b4:::benchmark_data_smithi003_175043_object92875:head v 220'11905, mlcod=220'11902)", 2024-08-17T17:35:04.281 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:28:00.572273+0000", 2024-08-17T17:35:04.281 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 423.63717798599998, 2024-08-17T17:35:04.281 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.237207371, 2024-08-17T17:35:04.281 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:35:04.281 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:35:04.281 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:35:04.281 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:35:04.281 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.281 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:35:04.281 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.572273+0000", 2024-08-17T17:35:04.281 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.281 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.281 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.282 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:35:04.282 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.572273+0000", 2024-08-17T17:35:04.282 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.282 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.282 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.282 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:35:04.282 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.572274+0000", 2024-08-17T17:35:04.282 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.294e-06 2024-08-17T17:35:04.282 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.282 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.282 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:35:04.282 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.572284+0000", 2024-08-17T17:35:04.282 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7100000000000002e-06 2024-08-17T17:35:04.283 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.283 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.283 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:35:04.283 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.572297+0000", 2024-08-17T17:35:04.283 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3122000000000001e-05 2024-08-17T17:35:04.283 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.283 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.283 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:35:04.283 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.572305+0000", 2024-08-17T17:35:04.283 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5120000000000002e-06 2024-08-17T17:35:04.283 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.283 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.283 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:35:04.283 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.572842+0000", 2024-08-17T17:35:04.284 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00053739800000000002 2024-08-17T17:35:04.284 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.284 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.284 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:35:04.284 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.572863+0000", 2024-08-17T17:35:04.284 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1353e-05 2024-08-17T17:35:04.284 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.284 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.284 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:35:04.284 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.809454+0000", 2024-08-17T17:35:04.284 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.23659016799999999 2024-08-17T17:35:04.284 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.284 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.284 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:35:04.285 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.809480+0000", 2024-08-17T17:35:04.285 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6814000000000001e-05 2024-08-17T17:35:04.285 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.285 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:35:04.285 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.285 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.285 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.285 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.45491.0:191527 6.d e220/204 6:b1fdfb8f:::benchmark_data_smithi003_175043_object92878:head v 220'11771, mlcod=220'11769)", 2024-08-17T17:35:04.285 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:28:00.575150+0000", 2024-08-17T17:35:04.285 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 423.63430089000002, 2024-08-17T17:35:04.285 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.25536818100000003, 2024-08-17T17:35:04.285 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:35:04.285 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:35:04.286 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:35:04.286 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:35:04.286 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.286 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:35:04.286 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.575150+0000", 2024-08-17T17:35:04.286 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.286 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.286 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.286 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:35:04.286 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.575150+0000", 2024-08-17T17:35:04.287 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.287 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.287 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.287 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:35:04.287 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.575151+0000", 2024-08-17T17:35:04.287 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8899999999999998e-07 2024-08-17T17:35:04.287 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.287 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.287 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:35:04.287 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.575158+0000", 2024-08-17T17:35:04.287 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.798e-06 2024-08-17T17:35:04.287 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.287 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.288 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:35:04.288 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.809006+0000", 2024-08-17T17:35:04.288 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.233848099 2024-08-17T17:35:04.288 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.288 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.288 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:35:04.288 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.809021+0000", 2024-08-17T17:35:04.288 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4999e-05 2024-08-17T17:35:04.288 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.288 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.288 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:35:04.288 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.809081+0000", 2024-08-17T17:35:04.289 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0380000000000001e-05 2024-08-17T17:35:04.289 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.289 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.289 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:35:04.289 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.809105+0000", 2024-08-17T17:35:04.289 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4040999999999999e-05 2024-08-17T17:35:04.289 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.289 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.289 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:35:04.289 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.830479+0000", 2024-08-17T17:35:04.289 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.021373207000000002 2024-08-17T17:35:04.289 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.290 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.290 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:35:04.290 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.830518+0000", 2024-08-17T17:35:04.290 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9668e-05 2024-08-17T17:35:04.290 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.290 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:35:04.290 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.290 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.290 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.290 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.45491.0:191525 6.9 e220/178 ondisk, result = 0)", 2024-08-17T17:35:04.290 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:28:00.576844+0000", 2024-08-17T17:35:04.291 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 423.63260740200002, 2024-08-17T17:35:04.291 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.23239138000000001, 2024-08-17T17:35:04.291 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:35:04.291 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:35:04.291 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:35:04.291 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:35:04.291 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.291 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:35:04.291 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.576844+0000", 2024-08-17T17:35:04.291 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.291 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.291 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.291 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:35:04.291 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.576844+0000", 2024-08-17T17:35:04.291 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.292 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.292 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.292 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:35:04.292 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.576845+0000", 2024-08-17T17:35:04.292 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.288e-06 2024-08-17T17:35:04.292 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.292 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.292 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:35:04.292 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.576849+0000", 2024-08-17T17:35:04.292 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4009999999999999e-06 2024-08-17T17:35:04.292 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.292 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.293 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:35:04.293 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.809135+0000", 2024-08-17T17:35:04.293 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.23228589799999999 2024-08-17T17:35:04.293 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.293 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.293 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:35:04.293 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.809145+0000", 2024-08-17T17:35:04.293 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.023e-05 2024-08-17T17:35:04.293 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.293 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.293 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:35:04.293 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.809209+0000", 2024-08-17T17:35:04.293 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3991000000000004e-05 2024-08-17T17:35:04.293 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.293 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.294 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:35:04.294 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.809219+0000", 2024-08-17T17:35:04.294 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8390000000000002e-06 2024-08-17T17:35:04.294 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.294 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.294 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:35:04.294 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:28:00.809235+0000", 2024-08-17T17:35:04.294 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5733e-05 2024-08-17T17:35:04.294 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.294 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:35:04.294 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.294 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.294 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.295 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:40469 7.4 7:2e557b99:::benchmark_data_smithi003_201798_object40468:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-08-17T17:35:04.295 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:29:00.881959+0000", 2024-08-17T17:35:04.295 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 363.327491544, 2024-08-17T17:35:04.295 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.210183284, 2024-08-17T17:35:04.295 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:35:04.295 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:35:04.295 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:35:04.295 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:35:04.295 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:35:04.295 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:35:04.295 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 40469 2024-08-17T17:35:04.295 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.295 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:35:04.295 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.295 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:35:04.296 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.881959+0000", 2024-08-17T17:35:04.296 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.296 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.296 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.296 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:35:04.296 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.881959+0000", 2024-08-17T17:35:04.296 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.296 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.296 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.296 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:35:04.296 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.881961+0000", 2024-08-17T17:35:04.296 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.299e-06 2024-08-17T17:35:04.296 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.296 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.297 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:35:04.297 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.881980+0000", 2024-08-17T17:35:04.297 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9432e-05 2024-08-17T17:35:04.297 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.297 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.297 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:35:04.297 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.881992+0000", 2024-08-17T17:35:04.297 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2111e-05 2024-08-17T17:35:04.297 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.297 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.297 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:35:04.297 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.881998+0000", 2024-08-17T17:35:04.297 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8660000000000002e-06 2024-08-17T17:35:04.297 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.298 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.298 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:35:04.298 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.882700+0000", 2024-08-17T17:35:04.298 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00070178199999999997 2024-08-17T17:35:04.298 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.298 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.298 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:35:04.298 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.882835+0000", 2024-08-17T17:35:04.298 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013538100000000001 2024-08-17T17:35:04.298 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.298 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.298 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:35:04.298 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.882968+0000", 2024-08-17T17:35:04.299 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013308699999999999 2024-08-17T17:35:04.299 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.299 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.299 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:35:04.299 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.884691+0000", 2024-08-17T17:35:04.299 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017223609999999999 2024-08-17T17:35:04.299 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.299 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.299 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.299 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.886183+0000", 2024-08-17T17:35:04.299 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014918139999999999 2024-08-17T17:35:04.299 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.299 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.299 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.300 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:01.092042+0000", 2024-08-17T17:35:04.300 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.20585922400000001 2024-08-17T17:35:04.300 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.300 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.300 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:35:04.300 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:01.092098+0000", 2024-08-17T17:35:04.300 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6474999999999997e-05 2024-08-17T17:35:04.300 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.300 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.300 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:35:04.300 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:01.092143+0000", 2024-08-17T17:35:04.300 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4452000000000001e-05 2024-08-17T17:35:04.300 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.300 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:35:04.300 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.301 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.301 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.301 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:40480 7.9 7:97dedd39:::benchmark_data_smithi003_201798_object40479:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-08-17T17:35:04.301 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:29:00.885840+0000", 2024-08-17T17:35:04.301 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 363.323611078, 2024-08-17T17:35:04.301 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.21157109700000001, 2024-08-17T17:35:04.301 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:35:04.301 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:35:04.301 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:35:04.301 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:35:04.301 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:35:04.301 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:35:04.301 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 40480 2024-08-17T17:35:04.301 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.301 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:35:04.302 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.302 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:35:04.302 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.885840+0000", 2024-08-17T17:35:04.302 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.302 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.302 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.302 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:35:04.302 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.885840+0000", 2024-08-17T17:35:04.302 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.302 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.302 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.303 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:35:04.303 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.885841+0000", 2024-08-17T17:35:04.303 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.438e-06 2024-08-17T17:35:04.303 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.303 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.303 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:35:04.303 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.885864+0000", 2024-08-17T17:35:04.303 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2421000000000001e-05 2024-08-17T17:35:04.303 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.303 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.303 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:35:04.303 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.886100+0000", 2024-08-17T17:35:04.303 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00023576599999999999 2024-08-17T17:35:04.303 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.303 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.304 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:35:04.304 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.886109+0000", 2024-08-17T17:35:04.304 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9630000000000004e-06 2024-08-17T17:35:04.304 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.304 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.304 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:35:04.304 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.886350+0000", 2024-08-17T17:35:04.304 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00024187300000000001 2024-08-17T17:35:04.304 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.304 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.304 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:35:04.304 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.886483+0000", 2024-08-17T17:35:04.304 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013242500000000001 2024-08-17T17:35:04.304 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.304 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.305 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,7", 2024-08-17T17:35:04.305 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.886637+0000", 2024-08-17T17:35:04.305 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015450699999999999 2024-08-17T17:35:04.305 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.305 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.305 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.305 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.888630+0000", 2024-08-17T17:35:04.305 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0019929959999999999 2024-08-17T17:35:04.305 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.305 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.305 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:35:04.305 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:00.889350+0000", 2024-08-17T17:35:04.305 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00072008800000000002 2024-08-17T17:35:04.305 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.305 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.306 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.306 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:01.097264+0000", 2024-08-17T17:35:04.306 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.207913083 2024-08-17T17:35:04.306 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.306 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.306 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:35:04.306 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:01.097369+0000", 2024-08-17T17:35:04.306 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010575 2024-08-17T17:35:04.306 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.306 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.306 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:35:04.306 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:29:01.097411+0000", 2024-08-17T17:35:04.306 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1786999999999998e-05 2024-08-17T17:35:04.306 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.307 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:35:04.307 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.307 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.307 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.307 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:163945 7.4 7:20801a71:::benchmark_data_smithi003_201798_object50723:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-08-17T17:35:04.307 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:31:16.131902+0000", 2024-08-17T17:35:04.307 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 228.077548885, 2024-08-17T17:35:04.307 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.20096719800000001, 2024-08-17T17:35:04.307 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:35:04.307 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:35:04.307 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:35:04.307 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:35:04.308 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:35:04.308 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:35:04.308 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 163945 2024-08-17T17:35:04.308 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.308 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:35:04.308 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.308 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:35:04.308 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.131902+0000", 2024-08-17T17:35:04.308 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.308 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.308 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.308 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:35:04.308 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.131902+0000", 2024-08-17T17:35:04.308 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.309 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.309 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.309 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:35:04.309 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.131904+0000", 2024-08-17T17:35:04.309 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6589999999999999e-06 2024-08-17T17:35:04.309 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.309 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.309 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:35:04.309 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.131910+0000", 2024-08-17T17:35:04.309 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9510000000000003e-06 2024-08-17T17:35:04.309 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.309 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.309 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:35:04.309 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.131932+0000", 2024-08-17T17:35:04.309 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2327000000000002e-05 2024-08-17T17:35:04.310 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.310 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.310 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:35:04.310 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.131941+0000", 2024-08-17T17:35:04.310 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7569999999999993e-06 2024-08-17T17:35:04.310 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.310 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.310 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:35:04.310 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.140353+0000", 2024-08-17T17:35:04.310 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0084117959999999992 2024-08-17T17:35:04.310 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.310 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.310 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:35:04.310 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.140458+0000", 2024-08-17T17:35:04.311 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000105062 2024-08-17T17:35:04.311 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.311 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.311 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:35:04.311 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.140565+0000", 2024-08-17T17:35:04.311 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000106888 2024-08-17T17:35:04.311 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.311 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.311 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:35:04.312 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.145737+0000", 2024-08-17T17:35:04.312 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0051727079999999998 2024-08-17T17:35:04.312 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.312 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.312 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.312 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.147699+0000", 2024-08-17T17:35:04.312 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0019620000000000002 2024-08-17T17:35:04.312 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.312 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.312 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.312 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.332739+0000", 2024-08-17T17:35:04.312 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18503926200000001 2024-08-17T17:35:04.313 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.313 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.313 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:35:04.313 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.332806+0000", 2024-08-17T17:35:04.313 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7860000000000004e-05 2024-08-17T17:35:04.313 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.313 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.313 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:35:04.313 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.332869+0000", 2024-08-17T17:35:04.313 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2928000000000005e-05 2024-08-17T17:35:04.313 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.313 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:35:04.313 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.314 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.314 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.314 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.45767.0:163949 7.9 7:90abad18:::benchmark_data_smithi003_201798_object50727:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-08-17T17:35:04.314 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:31:16.142530+0000", 2024-08-17T17:35:04.314 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 228.06692107200001, 2024-08-17T17:35:04.314 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.19359621900000001, 2024-08-17T17:35:04.314 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:35:04.314 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:35:04.315 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:35:04.315 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:35:04.315 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.45767", 2024-08-17T17:35:04.315 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2240569880", 2024-08-17T17:35:04.315 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 163949 2024-08-17T17:35:04.315 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.315 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:35:04.315 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.315 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:35:04.315 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.142530+0000", 2024-08-17T17:35:04.315 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.315 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.315 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.315 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:35:04.316 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.142530+0000", 2024-08-17T17:35:04.316 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.316 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.316 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.316 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:35:04.316 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.142532+0000", 2024-08-17T17:35:04.316 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6610000000000001e-06 2024-08-17T17:35:04.316 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.316 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.316 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:35:04.316 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.142538+0000", 2024-08-17T17:35:04.316 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0079999999999999e-06 2024-08-17T17:35:04.316 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.317 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.317 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:35:04.317 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.142551+0000", 2024-08-17T17:35:04.317 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3556e-05 2024-08-17T17:35:04.317 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.317 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.317 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:35:04.317 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.142559+0000", 2024-08-17T17:35:04.317 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9840000000000001e-06 2024-08-17T17:35:04.317 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.317 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.317 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:35:04.317 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.144707+0000", 2024-08-17T17:35:04.318 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0021477689999999999 2024-08-17T17:35:04.318 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.318 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.318 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:35:04.318 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.144831+0000", 2024-08-17T17:35:04.318 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012394700000000001 2024-08-17T17:35:04.318 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.318 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.318 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,7", 2024-08-17T17:35:04.318 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.144940+0000", 2024-08-17T17:35:04.318 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010878999999999999 2024-08-17T17:35:04.318 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.318 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.318 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:35:04.319 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.148944+0000", 2024-08-17T17:35:04.319 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040046910000000003 2024-08-17T17:35:04.319 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.319 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.319 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.319 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.149743+0000", 2024-08-17T17:35:04.319 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00079879400000000002 2024-08-17T17:35:04.319 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.319 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.319 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.319 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.336019+0000", 2024-08-17T17:35:04.320 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18627561000000001 2024-08-17T17:35:04.320 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.320 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.320 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:35:04.320 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.336077+0000", 2024-08-17T17:35:04.320 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8655e-05 2024-08-17T17:35:04.320 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.320 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.320 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:35:04.320 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:31:16.336126+0000", 2024-08-17T17:35:04.320 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8754000000000001e-05 2024-08-17T17:35:04.320 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.320 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:35:04.321 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.321 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.321 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.321 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:68824 8.e 8:7b9f8998:::benchmark_data_smithi003_229103_object68823:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-08-17T17:35:04.321 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:35.002608+0000", 2024-08-17T17:35:04.321 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 89.206843508000006, 2024-08-17T17:35:04.321 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.20794159600000001, 2024-08-17T17:35:04.321 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:35:04.321 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:35:04.321 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:35:04.321 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:35:04.321 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:35:04.321 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:35:04.322 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68824 2024-08-17T17:35:04.322 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.322 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:35:04.322 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.322 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:35:04.322 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.002608+0000", 2024-08-17T17:35:04.322 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.322 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.322 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.322 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:35:04.323 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.002608+0000", 2024-08-17T17:35:04.323 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.323 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.323 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.323 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:35:04.323 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.002610+0000", 2024-08-17T17:35:04.323 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2019999999999998e-06 2024-08-17T17:35:04.323 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.323 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.324 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:35:04.324 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.002634+0000", 2024-08-17T17:35:04.324 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4655e-05 2024-08-17T17:35:04.324 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.324 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.324 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:35:04.324 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.002651+0000", 2024-08-17T17:35:04.324 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6263999999999999e-05 2024-08-17T17:35:04.324 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.324 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.324 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:35:04.325 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.002658+0000", 2024-08-17T17:35:04.325 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4379999999999998e-06 2024-08-17T17:35:04.325 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.325 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.325 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:35:04.325 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.003672+0000", 2024-08-17T17:35:04.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:04.326 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010134689999999999 2024-08-17T17:35:04.326 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.326 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.326 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:35:04.327 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.003991+0000", 2024-08-17T17:35:04.327 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00031916099999999999 2024-08-17T17:35:04.327 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.327 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.327 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,4", 2024-08-17T17:35:04.327 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.004415+0000", 2024-08-17T17:35:04.327 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00042393999999999998 2024-08-17T17:35:04.327 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.327 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.327 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:35:04.327 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.008823+0000", 2024-08-17T17:35:04.327 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0044082519999999997 2024-08-17T17:35:04.327 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.328 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.328 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.328 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.008939+0000", 2024-08-17T17:35:04.328 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011628099999999999 2024-08-17T17:35:04.328 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.328 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.328 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.328 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.210387+0000", 2024-08-17T17:35:04.328 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.201447337 2024-08-17T17:35:04.328 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.328 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.329 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:35:04.329 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.210500+0000", 2024-08-17T17:35:04.329 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000113132 2024-08-17T17:35:04.329 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.329 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.329 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:35:04.329 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.210549+0000", 2024-08-17T17:35:04.329 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9465000000000002e-05 2024-08-17T17:35:04.329 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.329 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:35:04.329 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.329 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.329 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.329 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:68837 8.e 8:75f2384d:::benchmark_data_smithi003_229103_object68836:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-08-17T17:35:04.330 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:35.016337+0000", 2024-08-17T17:35:04.330 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 89.193113562999997, 2024-08-17T17:35:04.330 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.201609117, 2024-08-17T17:35:04.330 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:35:04.330 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:35:04.330 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:35:04.330 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:35:04.330 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:35:04.330 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:35:04.330 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68837 2024-08-17T17:35:04.330 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.330 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:35:04.330 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.331 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:35:04.331 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.016337+0000", 2024-08-17T17:35:04.331 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.331 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.331 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.331 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:35:04.331 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.016337+0000", 2024-08-17T17:35:04.331 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.331 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.331 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.331 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:35:04.331 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.016339+0000", 2024-08-17T17:35:04.331 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8169999999999999e-06 2024-08-17T17:35:04.332 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.332 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.332 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:35:04.332 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.016364+0000", 2024-08-17T17:35:04.332 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4511999999999998e-05 2024-08-17T17:35:04.332 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.332 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.332 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:35:04.332 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.016381+0000", 2024-08-17T17:35:04.332 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7178999999999999e-05 2024-08-17T17:35:04.332 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.332 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.333 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:35:04.333 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.016390+0000", 2024-08-17T17:35:04.333 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6379999999999993e-06 2024-08-17T17:35:04.333 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.333 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.333 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:35:04.333 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.016583+0000", 2024-08-17T17:35:04.333 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000193492 2024-08-17T17:35:04.333 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.333 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.334 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:35:04.334 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.016951+0000", 2024-08-17T17:35:04.334 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00036739500000000002 2024-08-17T17:35:04.334 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.334 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.334 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,4", 2024-08-17T17:35:04.334 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.017382+0000", 2024-08-17T17:35:04.334 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00043126500000000001 2024-08-17T17:35:04.334 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.334 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.334 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.334 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.020333+0000", 2024-08-17T17:35:04.335 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029511870000000001 2024-08-17T17:35:04.335 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.335 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.335 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:35:04.335 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.022872+0000", 2024-08-17T17:35:04.335 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025392710000000001 2024-08-17T17:35:04.335 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.335 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.335 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.335 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.217453+0000", 2024-08-17T17:35:04.335 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.19458077100000001 2024-08-17T17:35:04.335 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.335 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.336 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:35:04.336 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.217844+0000", 2024-08-17T17:35:04.336 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00039106600000000003 2024-08-17T17:35:04.336 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.336 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.336 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:35:04.336 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.217947+0000", 2024-08-17T17:35:04.336 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000102524 2024-08-17T17:35:04.336 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.336 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:35:04.336 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.336 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.336 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.336 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:68841 8.1 8:8211cd02:::benchmark_data_smithi003_229103_object68840:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-08-17T17:35:04.337 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:35.022604+0000", 2024-08-17T17:35:04.337 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 89.186847090000001, 2024-08-17T17:35:04.337 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.19225920299999999, 2024-08-17T17:35:04.337 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:35:04.337 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:35:04.337 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:35:04.337 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:35:04.337 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:35:04.337 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:35:04.337 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68841 2024-08-17T17:35:04.337 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.337 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:35:04.337 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.338 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:35:04.338 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.022604+0000", 2024-08-17T17:35:04.338 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.338 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.338 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.338 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:35:04.338 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.022604+0000", 2024-08-17T17:35:04.338 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.338 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.338 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.338 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:35:04.338 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.022606+0000", 2024-08-17T17:35:04.338 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7069999999999999e-06 2024-08-17T17:35:04.339 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.339 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.339 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:35:04.339 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.022631+0000", 2024-08-17T17:35:04.339 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5182999999999999e-05 2024-08-17T17:35:04.339 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.340 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.340 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:35:04.340 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.022648+0000", 2024-08-17T17:35:04.340 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7595999999999999e-05 2024-08-17T17:35:04.340 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.340 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.341 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:35:04.341 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.022658+0000", 2024-08-17T17:35:04.341 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6609999999999995e-06 2024-08-17T17:35:04.341 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.341 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.341 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:35:04.341 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.022739+0000", 2024-08-17T17:35:04.342 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0934999999999997e-05 2024-08-17T17:35:04.342 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.342 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.342 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:35:04.342 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.022946+0000", 2024-08-17T17:35:04.343 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020672799999999999 2024-08-17T17:35:04.343 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.343 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.343 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:35:04.343 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.023091+0000", 2024-08-17T17:35:04.343 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000145076 2024-08-17T17:35:04.344 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.344 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.344 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:35:04.344 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.028542+0000", 2024-08-17T17:35:04.344 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0054511430000000003 2024-08-17T17:35:04.344 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.344 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.345 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.345 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.028761+0000", 2024-08-17T17:35:04.345 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00021915199999999999 2024-08-17T17:35:04.345 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.345 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.345 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.345 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.212014+0000", 2024-08-17T17:35:04.346 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18325253999999999 2024-08-17T17:35:04.346 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.346 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.346 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:35:04.346 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.212076+0000", 2024-08-17T17:35:04.347 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2249999999999995e-05 2024-08-17T17:35:04.347 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.347 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.347 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:35:04.347 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.214863+0000", 2024-08-17T17:35:04.347 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027872320000000002 2024-08-17T17:35:04.347 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.347 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:35:04.347 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.348 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.348 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.348 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:68845 8.e 8:74722ba7:::benchmark_data_smithi003_229103_object68844:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-08-17T17:35:04.348 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:35.025985+0000", 2024-08-17T17:35:04.348 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 89.183465636999998, 2024-08-17T17:35:04.348 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.19305075899999999, 2024-08-17T17:35:04.348 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:35:04.348 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:35:04.348 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:35:04.348 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:35:04.348 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:35:04.349 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:35:04.349 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68845 2024-08-17T17:35:04.349 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.349 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:35:04.349 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.349 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:35:04.349 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.025985+0000", 2024-08-17T17:35:04.349 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.349 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.349 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.349 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:35:04.349 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.025985+0000", 2024-08-17T17:35:04.349 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:35:04.349 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.350 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.350 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:35:04.350 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.025987+0000", 2024-08-17T17:35:04.350 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.852e-06 2024-08-17T17:35:04.350 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.350 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.350 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:35:04.350 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.026013+0000", 2024-08-17T17:35:04.350 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5378e-05 2024-08-17T17:35:04.350 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.350 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.350 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:35:04.350 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.026024+0000", 2024-08-17T17:35:04.351 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1513e-05 2024-08-17T17:35:04.351 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.351 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.351 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:35:04.351 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.026032+0000", 2024-08-17T17:35:04.351 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8970000000000008e-06 2024-08-17T17:35:04.351 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.351 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.351 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:35:04.351 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.026238+0000", 2024-08-17T17:35:04.351 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020603900000000001 2024-08-17T17:35:04.351 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.351 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.352 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:35:04.352 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.026586+0000", 2024-08-17T17:35:04.352 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00034832800000000002 2024-08-17T17:35:04.352 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.352 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.352 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,4", 2024-08-17T17:35:04.352 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.027014+0000", 2024-08-17T17:35:04.352 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000427911 2024-08-17T17:35:04.352 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.352 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.352 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.352 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.030291+0000", 2024-08-17T17:35:04.352 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003276532 2024-08-17T17:35:04.352 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.353 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.353 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:35:04.353 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.030460+0000", 2024-08-17T17:35:04.353 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000169502 2024-08-17T17:35:04.353 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.353 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.353 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:35:04.353 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.217987+0000", 2024-08-17T17:35:04.353 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18752703900000001 2024-08-17T17:35:04.353 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.353 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.353 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:35:04.353 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.218983+0000", 2024-08-17T17:35:04.353 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00099567300000000009 2024-08-17T17:35:04.354 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:35:04.354 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:35:04.354 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:35:04.354 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.219036+0000", 2024-08-17T17:35:04.354 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3094999999999999e-05 2024-08-17T17:35:04.354 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.354 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:35:04.354 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.354 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:35:04.354 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:35:04.354 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:35:04.421 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:04.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:04.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:04.522 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:04.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:04 smithi003 ceph-mon[103344]: 1.1 scrub starts 2024-08-17T17:35:04.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:04 smithi003 ceph-mon[103344]: 1.1 scrub ok 2024-08-17T17:35:04.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:04 smithi003 ceph-mon[100615]: 1.1 scrub starts 2024-08-17T17:35:04.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:04 smithi003 ceph-mon[100615]: 1.1 scrub ok 2024-08-17T17:35:04.554 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:04.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:04.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:04.617 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:04 smithi053 ceph-mon[89963]: 1.1 scrub starts 2024-08-17T17:35:04.617 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:04 smithi053 ceph-mon[89963]: 1.1 scrub ok 2024-08-17T17:35:04.657 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:04.689 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:04.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:04.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:04.791 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:04.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:04.833 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:04.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:04.934 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:04.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:04.992 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:04.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:05.093 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:05.122 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:05.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:05.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:05.225 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:05.247 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:05 smithi003 ceph-mon[100615]: pgmap v868: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 697 op/s; 22295/244929 objects degraded (9.103%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T17:35:05.271 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:05.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:05.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:05.373 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:05.413 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:05.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:05.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:05.515 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:05.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:05 smithi003 ceph-mon[100615]: 1.9 scrub starts 2024-08-17T17:35:05.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:05 smithi003 ceph-mon[100615]: 1.9 scrub ok 2024-08-17T17:35:05.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:05 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 22295/244929 objects degraded (9.103%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:05.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:05 smithi003 ceph-mon[103344]: pgmap v868: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 697 op/s; 22295/244929 objects degraded (9.103%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T17:35:05.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:05 smithi003 ceph-mon[103344]: 1.9 scrub starts 2024-08-17T17:35:05.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:05 smithi003 ceph-mon[103344]: 1.9 scrub ok 2024-08-17T17:35:05.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:05 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 22295/244929 objects degraded (9.103%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:05.540 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:05.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:05.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:05.642 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:05.668 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:05.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:05.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:05.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:05 smithi053 ceph-mon[89963]: pgmap v868: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 697 op/s; 22295/244929 objects degraded (9.103%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T17:35:05.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:05 smithi053 ceph-mon[89963]: 1.9 scrub starts 2024-08-17T17:35:05.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:05 smithi053 ceph-mon[89963]: 1.9 scrub ok 2024-08-17T17:35:05.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:05 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 22295/244929 objects degraded (9.103%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:05.770 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:05.801 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:05.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:05.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:05.902 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:05.927 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:35:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:35:05] "GET /metrics HTTP/1.1" 200 36535 "" "Prometheus/2.51.0" 2024-08-17T17:35:05.929 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:05.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:05.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:06.031 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:06.066 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:06.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:06.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:06.168 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:06.218 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:06.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:06.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:06.320 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:06.350 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:06.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:06.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:06.452 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:06.478 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:06.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:06.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:06.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:06 smithi003 ceph-mon[103344]: 1.7 scrub starts 2024-08-17T17:35:06.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:06 smithi003 ceph-mon[103344]: 1.7 scrub ok 2024-08-17T17:35:06.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:06 smithi003 ceph-mon[100615]: 1.7 scrub starts 2024-08-17T17:35:06.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:06 smithi003 ceph-mon[100615]: 1.7 scrub ok 2024-08-17T17:35:06.580 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:06.607 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:06.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:06.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:06.709 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:06.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:06 smithi053 ceph-mon[89963]: 1.7 scrub starts 2024-08-17T17:35:06.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:06 smithi053 ceph-mon[89963]: 1.7 scrub ok 2024-08-17T17:35:06.739 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:06.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:06.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:06.841 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:06.889 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:06.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:06.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:06.991 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:07.017 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:07.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:07.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:07.118 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:07.154 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:07.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:07.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:07.256 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:07.284 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:07.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:07.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:07.386 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:07.412 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:07.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:07.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:07.514 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:07.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:07 smithi003 ceph-mon[103344]: pgmap v869: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 696 op/s; 22295/243636 objects degraded (9.151%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T17:35:07.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:07 smithi003 ceph-mon[103344]: 1.5 scrub starts 2024-08-17T17:35:07.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:07 smithi003 ceph-mon[103344]: 1.5 scrub ok 2024-08-17T17:35:07.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:07 smithi003 ceph-mon[100615]: pgmap v869: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 696 op/s; 22295/243636 objects degraded (9.151%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T17:35:07.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:07 smithi003 ceph-mon[100615]: 1.5 scrub starts 2024-08-17T17:35:07.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:07 smithi003 ceph-mon[100615]: 1.5 scrub ok 2024-08-17T17:35:07.540 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:07.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:07.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:07.642 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:07.673 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:07.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:07.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:07.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:07 smithi053 ceph-mon[89963]: pgmap v869: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 696 op/s; 22295/243636 objects degraded (9.151%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T17:35:07.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:07 smithi053 ceph-mon[89963]: 1.5 scrub starts 2024-08-17T17:35:07.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:07 smithi053 ceph-mon[89963]: 1.5 scrub ok 2024-08-17T17:35:07.774 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:07.807 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:07.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:07.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:07.909 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:07.941 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:07.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:07.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:08.043 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:08.078 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:08.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:08.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:08.180 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:08.204 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:08.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:08.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:08.305 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:08.354 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:08.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:08.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:08.456 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:08.490 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:08.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:08.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:08.592 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:08.621 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:08.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:08.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:08.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:08 smithi053 ceph-mon[89963]: 1.3 scrub starts 2024-08-17T17:35:08.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:08 smithi053 ceph-mon[89963]: 1.3 scrub ok 2024-08-17T17:35:08.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:35:08.723 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:08.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:08.748 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:08.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:08.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:08 smithi003 ceph-mon[100615]: 1.3 scrub starts 2024-08-17T17:35:08.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:08 smithi003 ceph-mon[100615]: 1.3 scrub ok 2024-08-17T17:35:08.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:35:08.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:08 smithi003 ceph-mon[103344]: 1.3 scrub starts 2024-08-17T17:35:08.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:08 smithi003 ceph-mon[103344]: 1.3 scrub ok 2024-08-17T17:35:08.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:35:08.849 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:08.878 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:08.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:08.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:08.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:09.018 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:09.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:09.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:09.120 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:09.156 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:09.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:09.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:09.258 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:09.287 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:09.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:09.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:09.389 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:09.427 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:09 smithi003 ceph-mon[103344]: pgmap v870: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 696 op/s; 22295/243636 objects degraded (9.151%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T17:35:09.427 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:09 smithi003 ceph-mon[103344]: 1.6 scrub starts 2024-08-17T17:35:09.427 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:09 smithi003 ceph-mon[103344]: 1.6 scrub ok 2024-08-17T17:35:09.427 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:09 smithi003 ceph-mon[100615]: pgmap v870: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 696 op/s; 22295/243636 objects degraded (9.151%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T17:35:09.427 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:09 smithi003 ceph-mon[100615]: 1.6 scrub starts 2024-08-17T17:35:09.427 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:09 smithi003 ceph-mon[100615]: 1.6 scrub ok 2024-08-17T17:35:09.432 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:09.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:09.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:09.534 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:09.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:09.573 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:09.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:09.674 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:09.695 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:35:09.696 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:35:09.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:09 smithi053 ceph-mon[89963]: pgmap v870: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 696 op/s; 22295/243636 objects degraded (9.151%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T17:35:09.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:09 smithi053 ceph-mon[89963]: 1.6 scrub starts 2024-08-17T17:35:09.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:09 smithi053 ceph-mon[89963]: 1.6 scrub ok 2024-08-17T17:35:09.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:09.718 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:09.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:09.819 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:09.860 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:09.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:09.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:09.963 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:10.009 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:10.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:10.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:10.111 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:10.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:10.137 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:10.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:10.238 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:10.279 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:10.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:10.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:10.380 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:10.415 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:10.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:10.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:10.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:35:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:35:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:35:10.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:10 smithi053 ceph-mon[89963]: 1.a scrub starts 2024-08-17T17:35:10.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:10 smithi053 ceph-mon[89963]: 1.a scrub ok 2024-08-17T17:35:10.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:10 smithi053 ceph-mon[89963]: from='client.36177 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:35:10.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:10 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 22295/243636 objects degraded (9.151%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:10.486 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-17T17:35:10.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:10.575 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:10.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:10.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:10.634 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:10 smithi003 ceph-mon[103344]: 1.a scrub starts 2024-08-17T17:35:10.634 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:10 smithi003 ceph-mon[103344]: 1.a scrub ok 2024-08-17T17:35:10.634 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:10 smithi003 ceph-mon[103344]: from='client.36177 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:35:10.635 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:10 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 22295/243636 objects degraded (9.151%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:10.635 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:10 smithi003 ceph-mon[100615]: 1.a scrub starts 2024-08-17T17:35:10.635 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:10 smithi003 ceph-mon[100615]: 1.a scrub ok 2024-08-17T17:35:10.635 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:10 smithi003 ceph-mon[100615]: from='client.36177 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:35:10.635 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:10 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 22295/243636 objects degraded (9.151%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:10.677 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:10.712 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:10.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:10.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:10.814 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:10.867 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:10.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:10.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:10.969 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:10.996 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:10.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:10.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:11.098 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":870,"stamp":"2024-08-17T17:35:08.122370+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5325521352,"num_objects":81212,"num_object_clones":0,"num_object_copies":243636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81212,"num_whiteouts":0,"num_read":6184,"num_read_kb":13440,"num_write":134786,"num_write_kb":6892150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2777,"num_bytes_recovered":184330648,"num_keys_recovered":10,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":125533,"ondisk_log_size":125533,"up":297,"acting":290,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":297,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":18861908,"kb_used_data":14877652,"kb_used_omap":37,"kb_used_meta":3984026,"kb_avail":731099308,"statfs":{"total":767960285184,"available":748645691392,"internally_reserved":0,"allocated":15234715648,"data_stored":15221978199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38729,"internal_metadata":4079642807},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,8,2,7,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-460747474,"num_objects":-7033,"num_object_clones":0,"num_object_copies":-21099,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7033,"num_whiteouts":0,"num_read":-104,"num_read_kb":-225,"num_write":6973,"num_write_kb":-659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":266,"num_bytes_recovered":16824136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004590"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":432,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.226200+0000","last_change":"2024-08-17T17:33:20.797029+0000","last_active":"2024-08-17T17:35:00.226200+0000","last_peered":"2024-08-17T17:35:00.226200+0000","last_clean":"2024-08-17T17:35:00.226200+0000","last_became_active":"2024-08-17T17:33:20.796679+0000","last_became_peered":"2024-08-17T17:33:20.796679+0000","last_unstale":"2024-08-17T17:35:00.226200+0000","last_undegraded":"2024-08-17T17:35:00.226200+0000","last_fullsized":"2024-08-17T17:35:00.226200+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:51.438262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":258,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-08-17T17:34:41.049595+0000","last_change":"2024-08-17T17:18:59.424887+0000","last_active":"2024-08-17T17:34:41.049595+0000","last_peered":"2024-08-17T17:34:41.049595+0000","last_clean":"2024-08-17T17:34:41.049595+0000","last_became_active":"2024-08-17T17:18:59.411539+0000","last_became_peered":"2024-08-17T17:18:59.411539+0000","last_unstale":"2024-08-17T17:34:41.049595+0000","last_undegraded":"2024-08-17T17:34:41.049595+0000","last_fullsized":"2024-08-17T17:34:41.049595+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:46:20.082241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":93,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.225605+0000","last_change":"2024-08-17T17:33:44.156036+0000","last_active":"2024-08-17T17:35:00.225605+0000","last_peered":"2024-08-17T17:35:00.225605+0000","last_clean":"2024-08-17T17:35:00.225605+0000","last_became_active":"2024-08-17T17:33:44.155758+0000","last_became_peered":"2024-08-17T17:33:44.155758+0000","last_unstale":"2024-08-17T17:35:00.225605+0000","last_undegraded":"2024-08-17T17:35:00.225605+0000","last_fullsized":"2024-08-17T17:35:00.225605+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:23.443405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":232,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.237630+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:35:00.237630+0000","last_peered":"2024-08-17T17:35:00.237630+0000","last_clean":"2024-08-17T17:35:00.237630+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:35:00.237630+0000","last_undegraded":"2024-08-17T17:35:00.237630+0000","last_fullsized":"2024-08-17T17:35:00.237630+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":232,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.242148+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:35:00.242148+0000","last_peered":"2024-08-17T17:35:00.242148+0000","last_clean":"2024-08-17T17:35:00.242148+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:35:00.242148+0000","last_undegraded":"2024-08-17T17:35:00.242148+0000","last_fullsized":"2024-08-17T17:35:00.242148+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":31,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-08-17T17:34:41.071476+0000","last_change":"2024-08-17T17:32:46.377651+0000","last_active":"2024-08-17T17:34:41.071476+0000","last_peered":"2024-08-17T17:34:41.071476+0000","last_clean":"2024-08-17T17:34:41.071476+0000","last_became_active":"2024-08-17T17:32:46.377282+0000","last_became_peered":"2024-08-17T17:32:46.377282+0000","last_unstale":"2024-08-17T17:34:41.071476+0000","last_undegraded":"2024-08-17T17:34:41.071476+0000","last_fullsized":"2024-08-17T17:34:41.071476+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:45:27.370287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":446,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.227733+0000","last_change":"2024-08-17T17:33:20.794682+0000","last_active":"2024-08-17T17:35:00.227733+0000","last_peered":"2024-08-17T17:35:00.227733+0000","last_clean":"2024-08-17T17:35:00.227733+0000","last_became_active":"2024-08-17T17:33:20.794238+0000","last_became_peered":"2024-08-17T17:33:20.794238+0000","last_unstale":"2024-08-17T17:35:00.227733+0000","last_undegraded":"2024-08-17T17:35:00.227733+0000","last_fullsized":"2024-08-17T17:35:00.227733+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:15:55.821920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":328,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-08-17T17:34:41.051215+0000","last_change":"2024-08-17T17:24:53.465875+0000","last_active":"2024-08-17T17:34:41.051215+0000","last_peered":"2024-08-17T17:34:41.051215+0000","last_clean":"2024-08-17T17:34:41.051215+0000","last_became_active":"2024-08-17T17:24:53.458129+0000","last_became_peered":"2024-08-17T17:24:53.458129+0000","last_unstale":"2024-08-17T17:34:41.051215+0000","last_undegraded":"2024-08-17T17:34:41.051215+0000","last_fullsized":"2024-08-17T17:34:41.051215+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:51:33.584869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"282'8317","reported_seq":8354,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:03.743814+0000","last_change":"2024-08-17T17:33:44.165390+0000","last_active":"2024-08-17T17:35:03.743814+0000","last_peered":"2024-08-17T17:35:03.743814+0000","last_clean":"2024-08-17T17:35:03.743814+0000","last_became_active":"2024-08-17T17:33:44.144942+0000","last_became_peered":"2024-08-17T17:33:44.144942+0000","last_unstale":"2024-08-17T17:35:03.743814+0000","last_undegraded":"2024-08-17T17:35:03.743814+0000","last_fullsized":"2024-08-17T17:35:03.743814+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":8317,"log_dups_size":0,"ondisk_log_size":8317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:49:11.850677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326696960,"num_objects":4985,"num_object_clones":0,"num_object_copies":14955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8317,"num_write_kb":425664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":215,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-08-17T17:34:41.049763+0000","last_change":"2024-08-17T17:32:46.384376+0000","last_active":"2024-08-17T17:34:41.049763+0000","last_peered":"2024-08-17T17:34:41.049763+0000","last_clean":"2024-08-17T17:34:41.049763+0000","last_became_active":"2024-08-17T17:32:46.374177+0000","last_became_peered":"2024-08-17T17:32:46.374177+0000","last_unstale":"2024-08-17T17:34:41.049763+0000","last_undegraded":"2024-08-17T17:34:41.049763+0000","last_fullsized":"2024-08-17T17:34:41.049763+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:51:58.880388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":37,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-08-17T17:34:41.051113+0000","last_change":"2024-08-17T17:33:44.145598+0000","last_active":"2024-08-17T17:34:41.051113+0000","last_peered":"2024-08-17T17:34:41.051113+0000","last_clean":"2024-08-17T17:34:41.051113+0000","last_became_active":"2024-08-17T17:33:44.145283+0000","last_became_peered":"2024-08-17T17:33:44.145283+0000","last_unstale":"2024-08-17T17:34:41.051113+0000","last_undegraded":"2024-08-17T17:34:41.051113+0000","last_fullsized":"2024-08-17T17:34:41.051113+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:11.965226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"282'8216","reported_seq":8244,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:04.043245+0000","last_change":"2024-08-17T17:32:48.380789+0000","last_active":"2024-08-17T17:35:04.043245+0000","last_peered":"2024-08-17T17:35:04.043245+0000","last_clean":"2024-08-17T17:35:04.043245+0000","last_became_active":"2024-08-17T17:32:48.380173+0000","last_became_peered":"2024-08-17T17:32:48.380173+0000","last_unstale":"2024-08-17T17:35:04.043245+0000","last_undegraded":"2024-08-17T17:35:04.043245+0000","last_fullsized":"2024-08-17T17:35:04.043245+0000","mapping_epoch":262,"log_start":"266'1300","ondisk_log_start":"266'1300","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":6916,"log_dups_size":0,"ondisk_log_size":6916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:25:18.190497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327811072,"num_objects":5002,"num_object_clones":0,"num_object_copies":15006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8216,"num_write_kb":422976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":197,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-08-17T17:34:41.049932+0000","last_change":"2024-08-17T17:33:20.834638+0000","last_active":"2024-08-17T17:34:41.049932+0000","last_peered":"2024-08-17T17:34:41.049932+0000","last_clean":"2024-08-17T17:34:41.049932+0000","last_became_active":"2024-08-17T17:33:20.834262+0000","last_became_peered":"2024-08-17T17:33:20.834262+0000","last_unstale":"2024-08-17T17:34:41.049932+0000","last_undegraded":"2024-08-17T17:34:41.049932+0000","last_fullsized":"2024-08-17T17:34:41.049932+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:43:06.894393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":45,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-08-17T17:34:41.071463+0000","last_change":"2024-08-17T17:33:21.766549+0000","last_active":"2024-08-17T17:34:41.071463+0000","last_peered":"2024-08-17T17:34:41.071463+0000","last_clean":"2024-08-17T17:34:41.071463+0000","last_became_active":"2024-08-17T17:33:21.766152+0000","last_became_peered":"2024-08-17T17:33:21.766152+0000","last_unstale":"2024-08-17T17:34:41.071463+0000","last_undegraded":"2024-08-17T17:34:41.071463+0000","last_fullsized":"2024-08-17T17:34:41.071463+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:56:15.855524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"282'8259","reported_seq":8311,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:03.851106+0000","last_change":"2024-08-17T17:32:48.381458+0000","last_active":"2024-08-17T17:35:03.851106+0000","last_peered":"2024-08-17T17:35:03.851106+0000","last_clean":"2024-08-17T17:35:03.851106+0000","last_became_active":"2024-08-17T17:32:48.381201+0000","last_became_peered":"2024-08-17T17:32:48.381201+0000","last_unstale":"2024-08-17T17:35:03.851106+0000","last_undegraded":"2024-08-17T17:35:03.851106+0000","last_fullsized":"2024-08-17T17:35:03.851106+0000","mapping_epoch":262,"log_start":"266'1800","ondisk_log_start":"266'1800","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":6459,"log_dups_size":0,"ondisk_log_size":6459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:12:18.200526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329187328,"num_objects":5023,"num_object_clones":0,"num_object_copies":15069,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8259,"num_write_kb":425024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":33,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-08-17T17:34:41.051628+0000","last_change":"2024-08-17T17:33:44.145399+0000","last_active":"2024-08-17T17:34:41.051628+0000","last_peered":"2024-08-17T17:34:41.051628+0000","last_clean":"2024-08-17T17:34:41.051628+0000","last_became_active":"2024-08-17T17:33:44.144751+0000","last_became_peered":"2024-08-17T17:33:44.144751+0000","last_unstale":"2024-08-17T17:34:41.051628+0000","last_undegraded":"2024-08-17T17:34:41.051628+0000","last_fullsized":"2024-08-17T17:34:41.051628+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:54.276350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":111,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-08-17T17:34:41.051323+0000","last_change":"2024-08-17T17:33:20.795582+0000","last_active":"2024-08-17T17:34:41.051323+0000","last_peered":"2024-08-17T17:34:41.051323+0000","last_clean":"2024-08-17T17:34:41.051323+0000","last_became_active":"2024-08-17T17:33:20.795149+0000","last_became_peered":"2024-08-17T17:33:20.795149+0000","last_unstale":"2024-08-17T17:34:41.051323+0000","last_undegraded":"2024-08-17T17:34:41.051323+0000","last_fullsized":"2024-08-17T17:34:41.051323+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:14.020923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"282'8303","reported_seq":8354,"reported_epoch":282,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:35:03.349554+0000","last_change":"2024-08-17T17:33:22.090332+0000","last_active":"2024-08-17T17:35:03.349554+0000","last_peered":"2024-08-17T17:35:03.349554+0000","last_clean":"2024-08-17T17:32:47.362949+0000","last_became_active":"2024-08-17T17:33:22.080130+0000","last_became_peered":"2024-08-17T17:33:22.080130+0000","last_unstale":"2024-08-17T17:35:03.349554+0000","last_undegraded":"2024-08-17T17:33:21.923200+0000","last_fullsized":"2024-08-17T17:33:21.906251+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":8303,"log_dups_size":0,"ondisk_log_size":8303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335085568,"num_objects":5113,"num_object_clones":0,"num_object_copies":15339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8303,"num_write_kb":429312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":1593}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":90,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.226330+0000","last_change":"2024-08-17T17:33:44.154097+0000","last_active":"2024-08-17T17:35:00.226330+0000","last_peered":"2024-08-17T17:35:00.226330+0000","last_clean":"2024-08-17T17:35:00.226330+0000","last_became_active":"2024-08-17T17:33:44.152972+0000","last_became_peered":"2024-08-17T17:33:44.152972+0000","last_unstale":"2024-08-17T17:35:00.226330+0000","last_undegraded":"2024-08-17T17:35:00.226330+0000","last_fullsized":"2024-08-17T17:35:00.226330+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:42.602017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":440,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.237942+0000","last_change":"2024-08-17T17:33:21.763677+0000","last_active":"2024-08-17T17:35:00.237942+0000","last_peered":"2024-08-17T17:35:00.237942+0000","last_clean":"2024-08-17T17:35:00.237942+0000","last_became_active":"2024-08-17T17:33:21.760564+0000","last_became_peered":"2024-08-17T17:33:21.760564+0000","last_unstale":"2024-08-17T17:35:00.237942+0000","last_undegraded":"2024-08-17T17:35:00.237942+0000","last_fullsized":"2024-08-17T17:35:00.237942+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:01:00.772037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":565,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.239640+0000","last_change":"2024-08-17T17:35:00.239640+0000","last_active":"2024-08-17T17:35:00.239640+0000","last_peered":"2024-08-17T17:35:00.239640+0000","last_clean":"2024-08-17T17:35:00.239640+0000","last_became_active":"2024-08-17T17:34:17.898604+0000","last_became_peered":"2024-08-17T17:34:17.898604+0000","last_unstale":"2024-08-17T17:35:00.239640+0000","last_undegraded":"2024-08-17T17:35:00.239640+0000","last_fullsized":"2024-08-17T17:35:00.239640+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:04:01.141401+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":74,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.0","version":"282'8332","reported_seq":8382,"reported_epoch":282,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:35:03.302400+0000","last_change":"2024-08-17T17:33:22.010282+0000","last_active":"2024-08-17T17:35:03.302400+0000","last_peered":"2024-08-17T17:35:03.302400+0000","last_clean":"2024-08-17T17:32:47.362949+0000","last_became_active":"2024-08-17T17:33:21.998714+0000","last_became_peered":"2024-08-17T17:33:21.998714+0000","last_unstale":"2024-08-17T17:35:03.302400+0000","last_undegraded":"2024-08-17T17:33:21.807105+0000","last_fullsized":"2024-08-17T17:33:21.761969+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":8332,"log_dups_size":0,"ondisk_log_size":8332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330170368,"num_objects":5038,"num_object_clones":0,"num_object_copies":15114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3554,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8332,"num_write_kb":427840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[{"shards":"3,5","objects":1484}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":467,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.226623+0000","last_change":"2024-08-17T17:32:46.387810+0000","last_active":"2024-08-17T17:35:00.226623+0000","last_peered":"2024-08-17T17:35:00.226623+0000","last_clean":"2024-08-17T17:35:00.226623+0000","last_became_active":"2024-08-17T17:32:46.374238+0000","last_became_peered":"2024-08-17T17:32:46.374238+0000","last_unstale":"2024-08-17T17:35:00.226623+0000","last_undegraded":"2024-08-17T17:35:00.226623+0000","last_fullsized":"2024-08-17T17:35:00.226623+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:53:29.918005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":31,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-08-17T17:34:41.071646+0000","last_change":"2024-08-17T17:32:46.378495+0000","last_active":"2024-08-17T17:34:41.071646+0000","last_peered":"2024-08-17T17:34:41.071646+0000","last_clean":"2024-08-17T17:34:41.071646+0000","last_became_active":"2024-08-17T17:32:46.377709+0000","last_became_peered":"2024-08-17T17:32:46.377709+0000","last_unstale":"2024-08-17T17:34:41.071646+0000","last_undegraded":"2024-08-17T17:34:41.071646+0000","last_fullsized":"2024-08-17T17:34:41.071646+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:22:52.544851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":573,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:02.239789+0000","last_change":"2024-08-17T17:35:02.239838+0000","last_active":"2024-08-17T17:35:02.239789+0000","last_peered":"2024-08-17T17:35:02.239789+0000","last_clean":"2024-08-17T17:35:02.239789+0000","last_became_active":"2024-08-17T17:34:17.888626+0000","last_became_peered":"2024-08-17T17:34:17.888626+0000","last_unstale":"2024-08-17T17:35:02.239789+0000","last_undegraded":"2024-08-17T17:35:02.239789+0000","last_fullsized":"2024-08-17T17:35:02.239789+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:35:02.239705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:35:02.239705+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:28:47.050841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"282'8508","reported_seq":8558,"reported_epoch":282,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:35:03.894350+0000","last_change":"2024-08-17T17:33:21.986278+0000","last_active":"2024-08-17T17:35:03.894350+0000","last_peered":"2024-08-17T17:35:03.894350+0000","last_clean":"2024-08-17T17:32:47.362949+0000","last_became_active":"2024-08-17T17:33:21.976559+0000","last_became_peered":"2024-08-17T17:33:21.976559+0000","last_unstale":"2024-08-17T17:35:03.894350+0000","last_undegraded":"2024-08-17T17:33:21.786755+0000","last_fullsized":"2024-08-17T17:33:21.754986+0000","mapping_epoch":269,"log_start":"266'2100","ondisk_log_start":"266'2100","created":262,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":6408,"log_dups_size":0,"ondisk_log_size":6408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338952192,"num_objects":5172,"num_object_clones":0,"num_object_copies":15516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3606,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8508,"num_write_kb":437760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":1566}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":45,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-08-17T17:34:41.071534+0000","last_change":"2024-08-17T17:33:21.766756+0000","last_active":"2024-08-17T17:34:41.071534+0000","last_peered":"2024-08-17T17:34:41.071534+0000","last_clean":"2024-08-17T17:34:41.071534+0000","last_became_active":"2024-08-17T17:33:21.766347+0000","last_became_peered":"2024-08-17T17:33:21.766347+0000","last_unstale":"2024-08-17T17:34:41.071534+0000","last_undegraded":"2024-08-17T17:34:41.071534+0000","last_fullsized":"2024-08-17T17:34:41.071534+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:13:14.655176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":145,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-08-17T17:34:41.051531+0000","last_change":"2024-08-17T17:32:25.140415+0000","last_active":"2024-08-17T17:34:41.051531+0000","last_peered":"2024-08-17T17:34:41.051531+0000","last_clean":"2024-08-17T17:34:41.051531+0000","last_became_active":"2024-08-17T17:32:25.140060+0000","last_became_peered":"2024-08-17T17:32:25.140060+0000","last_unstale":"2024-08-17T17:34:41.051531+0000","last_undegraded":"2024-08-17T17:34:41.051531+0000","last_fullsized":"2024-08-17T17:34:41.051531+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:14:52.520189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2","version":"282'8347","reported_seq":8375,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:03.673747+0000","last_change":"2024-08-17T17:32:48.378488+0000","last_active":"2024-08-17T17:35:03.673747+0000","last_peered":"2024-08-17T17:35:03.673747+0000","last_clean":"2024-08-17T17:35:03.673747+0000","last_became_active":"2024-08-17T17:32:48.378189+0000","last_became_peered":"2024-08-17T17:32:48.378189+0000","last_unstale":"2024-08-17T17:35:03.673747+0000","last_undegraded":"2024-08-17T17:35:03.673747+0000","last_fullsized":"2024-08-17T17:35:03.673747+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":8347,"log_dups_size":0,"ondisk_log_size":8347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:38.041536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331808768,"num_objects":5063,"num_object_clones":0,"num_object_copies":15189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8347,"num_write_kb":429120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":232,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-08-17T17:34:41.049136+0000","last_change":"2024-08-17T17:33:20.794217+0000","last_active":"2024-08-17T17:34:41.049136+0000","last_peered":"2024-08-17T17:34:41.049136+0000","last_clean":"2024-08-17T17:34:41.049136+0000","last_became_active":"2024-08-17T17:33:20.793899+0000","last_became_peered":"2024-08-17T17:33:20.793899+0000","last_unstale":"2024-08-17T17:34:41.049136+0000","last_undegraded":"2024-08-17T17:34:41.049136+0000","last_fullsized":"2024-08-17T17:34:41.049136+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:01:31.197983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":45,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-08-17T17:34:41.071783+0000","last_change":"2024-08-17T17:33:21.764934+0000","last_active":"2024-08-17T17:34:41.071783+0000","last_peered":"2024-08-17T17:34:41.071783+0000","last_clean":"2024-08-17T17:34:41.071783+0000","last_became_active":"2024-08-17T17:33:21.764341+0000","last_became_peered":"2024-08-17T17:33:21.764341+0000","last_unstale":"2024-08-17T17:34:41.071783+0000","last_undegraded":"2024-08-17T17:34:41.071783+0000","last_fullsized":"2024-08-17T17:34:41.071783+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:20:25.714104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":568,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.237301+0000","last_change":"2024-08-17T17:35:00.237301+0000","last_active":"2024-08-17T17:35:00.237301+0000","last_peered":"2024-08-17T17:35:00.237301+0000","last_clean":"2024-08-17T17:35:00.237301+0000","last_became_active":"2024-08-17T17:34:17.889755+0000","last_became_peered":"2024-08-17T17:34:17.889755+0000","last_unstale":"2024-08-17T17:35:00.237301+0000","last_undegraded":"2024-08-17T17:35:00.237301+0000","last_fullsized":"2024-08-17T17:35:00.237301+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:04:01.141401+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-17T17:35:05.142319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":74,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3","version":"282'8293","reported_seq":8324,"reported_epoch":282,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:35:03.948430+0000","last_change":"2024-08-17T17:33:21.990474+0000","last_active":"2024-08-17T17:35:03.948430+0000","last_peered":"2024-08-17T17:35:03.948430+0000","last_clean":"2024-08-17T17:32:47.362949+0000","last_became_active":"2024-08-17T17:33:21.980477+0000","last_became_peered":"2024-08-17T17:33:21.980477+0000","last_unstale":"2024-08-17T17:35:03.948430+0000","last_undegraded":"2024-08-17T17:33:21.778308+0000","last_fullsized":"2024-08-17T17:33:21.762700+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":8293,"log_dups_size":0,"ondisk_log_size":8293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:31:39.084953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334823424,"num_objects":5109,"num_object_clones":0,"num_object_copies":15327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8293,"num_write_kb":428864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":1651}],"blocked_by":[],"up_primary":1,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":448,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.238560+0000","last_change":"2024-08-17T17:33:20.796216+0000","last_active":"2024-08-17T17:35:00.238560+0000","last_peered":"2024-08-17T17:35:00.238560+0000","last_clean":"2024-08-17T17:35:00.238560+0000","last_became_active":"2024-08-17T17:33:20.795762+0000","last_became_peered":"2024-08-17T17:33:20.795762+0000","last_unstale":"2024-08-17T17:35:00.238560+0000","last_undegraded":"2024-08-17T17:35:00.238560+0000","last_fullsized":"2024-08-17T17:35:00.238560+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:47.038509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":329,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.240996+0000","last_change":"2024-08-17T17:33:21.976938+0000","last_active":"2024-08-17T17:35:00.240996+0000","last_peered":"2024-08-17T17:35:00.240996+0000","last_clean":"2024-08-17T17:35:00.240996+0000","last_became_active":"2024-08-17T17:33:21.976585+0000","last_became_peered":"2024-08-17T17:33:21.976585+0000","last_unstale":"2024-08-17T17:35:00.240996+0000","last_undegraded":"2024-08-17T17:35:00.240996+0000","last_fullsized":"2024-08-17T17:35:00.240996+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:23:40.758870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"282'8423","reported_seq":8449,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:04.076634+0000","last_change":"2024-08-17T17:32:48.378105+0000","last_active":"2024-08-17T17:35:04.076634+0000","last_peered":"2024-08-17T17:35:04.076634+0000","last_clean":"2024-08-17T17:35:04.076634+0000","last_became_active":"2024-08-17T17:32:48.377534+0000","last_became_peered":"2024-08-17T17:32:48.377534+0000","last_unstale":"2024-08-17T17:35:04.076634+0000","last_undegraded":"2024-08-17T17:35:04.076634+0000","last_fullsized":"2024-08-17T17:35:04.076634+0000","mapping_epoch":262,"log_start":"266'1600","ondisk_log_start":"266'1600","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":6823,"log_dups_size":0,"ondisk_log_size":6823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:53.372506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336134144,"num_objects":5129,"num_object_clones":0,"num_object_copies":15387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8423,"num_write_kb":433664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":562,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.237366+0000","last_change":"2024-08-17T17:34:18.238306+0000","last_active":"2024-08-17T17:35:00.237366+0000","last_peered":"2024-08-17T17:35:00.237366+0000","last_clean":"2024-08-17T17:35:00.237366+0000","last_became_active":"2024-08-17T17:34:17.878280+0000","last_became_peered":"2024-08-17T17:34:17.878280+0000","last_unstale":"2024-08-17T17:35:00.237366+0000","last_undegraded":"2024-08-17T17:35:00.237366+0000","last_fullsized":"2024-08-17T17:35:00.237366+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:34:18.238171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:34:18.238171+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:27:18.000874+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":688,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-08-17T17:34:41.049392+0000","last_change":"2024-08-17T17:33:44.149928+0000","last_active":"2024-08-17T17:34:41.049392+0000","last_peered":"2024-08-17T17:34:41.049392+0000","last_clean":"2024-08-17T17:34:41.049392+0000","last_became_active":"2024-08-17T17:33:44.149498+0000","last_became_peered":"2024-08-17T17:33:44.149498+0000","last_unstale":"2024-08-17T17:34:41.049392+0000","last_undegraded":"2024-08-17T17:34:41.049392+0000","last_fullsized":"2024-08-17T17:34:41.049392+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:17:04.963202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":395,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.225622+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:35:00.225622+0000","last_peered":"2024-08-17T17:35:00.225622+0000","last_clean":"2024-08-17T17:35:00.225622+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:35:00.225622+0000","last_undegraded":"2024-08-17T17:35:00.225622+0000","last_fullsized":"2024-08-17T17:35:00.225622+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":94,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-08-17T17:34:41.051559+0000","last_change":"2024-08-17T17:25:04.579426+0000","last_active":"2024-08-17T17:34:41.051559+0000","last_peered":"2024-08-17T17:34:41.051559+0000","last_clean":"2024-08-17T17:34:41.051559+0000","last_became_active":"2024-08-17T17:25:04.579012+0000","last_became_peered":"2024-08-17T17:25:04.579012+0000","last_unstale":"2024-08-17T17:34:41.051559+0000","last_undegraded":"2024-08-17T17:34:41.051559+0000","last_fullsized":"2024-08-17T17:34:41.051559+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:56:32.241321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":568,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.237853+0000","last_change":"2024-08-17T17:35:00.237853+0000","last_active":"2024-08-17T17:35:00.237853+0000","last_peered":"2024-08-17T17:35:00.237853+0000","last_clean":"2024-08-17T17:35:00.237853+0000","last_became_active":"2024-08-17T17:34:17.886791+0000","last_became_peered":"2024-08-17T17:34:17.886791+0000","last_unstale":"2024-08-17T17:35:00.237853+0000","last_undegraded":"2024-08-17T17:35:00.237853+0000","last_fullsized":"2024-08-17T17:35:00.237853+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:04:01.141401+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-17T17:35:03.136646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"282'8387","reported_seq":8439,"reported_epoch":282,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:35:05.005096+0000","last_change":"2024-08-17T17:33:22.097156+0000","last_active":"2024-08-17T17:35:05.005096+0000","last_peered":"2024-08-17T17:35:05.005096+0000","last_clean":"2024-08-17T17:32:47.362949+0000","last_became_active":"2024-08-17T17:33:21.961793+0000","last_became_peered":"2024-08-17T17:33:21.961793+0000","last_unstale":"2024-08-17T17:35:05.005096+0000","last_undegraded":"2024-08-17T17:33:21.778949+0000","last_fullsized":"2024-08-17T17:33:21.755681+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":8387,"log_dups_size":0,"ondisk_log_size":8387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324993024,"num_objects":4959,"num_object_clones":0,"num_object_copies":14877,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3589,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8387,"num_write_kb":427072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":262144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":1370}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":679,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.248036+0000","last_change":"2024-08-17T17:33:20.797705+0000","last_active":"2024-08-17T17:35:00.248036+0000","last_peered":"2024-08-17T17:35:00.248036+0000","last_clean":"2024-08-17T17:35:00.248036+0000","last_became_active":"2024-08-17T17:33:20.797172+0000","last_became_peered":"2024-08-17T17:33:20.797172+0000","last_unstale":"2024-08-17T17:35:00.248036+0000","last_undegraded":"2024-08-17T17:35:00.248036+0000","last_fullsized":"2024-08-17T17:35:00.248036+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:30.754999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":424,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.250256+0000","last_change":"2024-08-17T17:18:59.415331+0000","last_active":"2024-08-17T17:35:00.250256+0000","last_peered":"2024-08-17T17:35:00.250256+0000","last_clean":"2024-08-17T17:35:00.250256+0000","last_became_active":"2024-08-17T17:18:59.415008+0000","last_became_peered":"2024-08-17T17:18:59.415008+0000","last_unstale":"2024-08-17T17:35:00.250256+0000","last_undegraded":"2024-08-17T17:35:00.250256+0000","last_fullsized":"2024-08-17T17:35:00.250256+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:22:38.076866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":233,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.242314+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:35:00.242314+0000","last_peered":"2024-08-17T17:35:00.242314+0000","last_clean":"2024-08-17T17:35:00.242314+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:35:00.242314+0000","last_undegraded":"2024-08-17T17:35:00.242314+0000","last_fullsized":"2024-08-17T17:35:00.242314+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":573,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:01.201134+0000","last_change":"2024-08-17T17:35:01.201171+0000","last_active":"2024-08-17T17:35:01.201134+0000","last_peered":"2024-08-17T17:35:01.201134+0000","last_clean":"2024-08-17T17:35:01.201134+0000","last_became_active":"2024-08-17T17:34:17.890103+0000","last_became_peered":"2024-08-17T17:34:17.890103+0000","last_unstale":"2024-08-17T17:35:01.201134+0000","last_undegraded":"2024-08-17T17:35:01.201134+0000","last_fullsized":"2024-08-17T17:35:01.201134+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:35:01.201049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:35:01.201049+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:43:37.917040+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"282'8423","reported_seq":8475,"reported_epoch":282,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:35:03.337772+0000","last_change":"2024-08-17T17:33:21.983064+0000","last_active":"2024-08-17T17:35:03.337772+0000","last_peered":"2024-08-17T17:35:03.337772+0000","last_clean":"2024-08-17T17:32:47.362949+0000","last_became_active":"2024-08-17T17:33:21.969537+0000","last_became_peered":"2024-08-17T17:33:21.969537+0000","last_unstale":"2024-08-17T17:35:03.337772+0000","last_undegraded":"2024-08-17T17:33:21.804745+0000","last_fullsized":"2024-08-17T17:33:21.762111+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":8423,"log_dups_size":0,"ondisk_log_size":8423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333512704,"num_objects":5089,"num_object_clones":0,"num_object_copies":15267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3634,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8423,"num_write_kb":432384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":1455}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":812,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.225412+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:35:00.225412+0000","last_peered":"2024-08-17T17:35:00.225412+0000","last_clean":"2024-08-17T17:35:00.225412+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:35:00.225412+0000","last_undegraded":"2024-08-17T17:35:00.225412+0000","last_fullsized":"2024-08-17T17:35:00.225412+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":317,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.248455+0000","last_change":"2024-08-17T17:33:20.793468+0000","last_active":"2024-08-17T17:35:00.248455+0000","last_peered":"2024-08-17T17:35:00.248455+0000","last_clean":"2024-08-17T17:35:00.248455+0000","last_became_active":"2024-08-17T17:33:20.793166+0000","last_became_peered":"2024-08-17T17:33:20.793166+0000","last_unstale":"2024-08-17T17:35:00.248455+0000","last_undegraded":"2024-08-17T17:35:00.248455+0000","last_fullsized":"2024-08-17T17:35:00.248455+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:12:52.121050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":353,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.240920+0000","last_change":"2024-08-17T17:33:20.835784+0000","last_active":"2024-08-17T17:35:00.240920+0000","last_peered":"2024-08-17T17:35:00.240920+0000","last_clean":"2024-08-17T17:35:00.240920+0000","last_became_active":"2024-08-17T17:33:20.835371+0000","last_became_peered":"2024-08-17T17:33:20.835371+0000","last_unstale":"2024-08-17T17:35:00.240920+0000","last_undegraded":"2024-08-17T17:35:00.240920+0000","last_fullsized":"2024-08-17T17:35:00.240920+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:35:43.369055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":572,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:03.210384+0000","last_change":"2024-08-17T17:35:03.210422+0000","last_active":"2024-08-17T17:35:03.210384+0000","last_peered":"2024-08-17T17:35:03.210384+0000","last_clean":"2024-08-17T17:35:03.210384+0000","last_became_active":"2024-08-17T17:34:17.898012+0000","last_became_peered":"2024-08-17T17:34:17.898012+0000","last_unstale":"2024-08-17T17:35:03.210384+0000","last_undegraded":"2024-08-17T17:35:03.210384+0000","last_fullsized":"2024-08-17T17:35:03.210384+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:35:03.210298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:35:03.210298+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:56:27.154880+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"282'8293","reported_seq":8339,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:03.297793+0000","last_change":"2024-08-17T17:33:44.165133+0000","last_active":"2024-08-17T17:35:03.297793+0000","last_peered":"2024-08-17T17:35:03.297793+0000","last_clean":"2024-08-17T17:35:03.297793+0000","last_became_active":"2024-08-17T17:33:44.156220+0000","last_became_peered":"2024-08-17T17:33:44.156220+0000","last_unstale":"2024-08-17T17:35:03.297793+0000","last_undegraded":"2024-08-17T17:35:03.297793+0000","last_fullsized":"2024-08-17T17:35:03.297793+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":8293,"log_dups_size":0,"ondisk_log_size":8293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:57:21.141437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334430208,"num_objects":5103,"num_object_clones":0,"num_object_copies":15309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8293,"num_write_kb":428672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":902,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.239809+0000","last_change":"2024-08-17T17:33:44.148227+0000","last_active":"2024-08-17T17:35:00.239809+0000","last_peered":"2024-08-17T17:35:00.239809+0000","last_clean":"2024-08-17T17:35:00.239809+0000","last_became_active":"2024-08-17T17:33:44.147627+0000","last_became_peered":"2024-08-17T17:33:44.147627+0000","last_unstale":"2024-08-17T17:35:00.239809+0000","last_undegraded":"2024-08-17T17:35:00.239809+0000","last_fullsized":"2024-08-17T17:35:00.239809+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:29:42.898460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1372,"num_read_kb":2843,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":31,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-08-17T17:34:41.071665+0000","last_change":"2024-08-17T17:32:46.379200+0000","last_active":"2024-08-17T17:34:41.071665+0000","last_peered":"2024-08-17T17:34:41.071665+0000","last_clean":"2024-08-17T17:34:41.071665+0000","last_became_active":"2024-08-17T17:32:46.378870+0000","last_became_peered":"2024-08-17T17:32:46.378870+0000","last_unstale":"2024-08-17T17:34:41.071665+0000","last_undegraded":"2024-08-17T17:34:41.071665+0000","last_fullsized":"2024-08-17T17:34:41.071665+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:55:26.885966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":357,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.240008+0000","last_change":"2024-08-17T17:33:20.772454+0000","last_active":"2024-08-17T17:35:00.240008+0000","last_peered":"2024-08-17T17:35:00.240008+0000","last_clean":"2024-08-17T17:35:00.240008+0000","last_became_active":"2024-08-17T17:33:20.771848+0000","last_became_peered":"2024-08-17T17:33:20.771848+0000","last_unstale":"2024-08-17T17:35:00.240008+0000","last_undegraded":"2024-08-17T17:35:00.240008+0000","last_fullsized":"2024-08-17T17:35:00.240008+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:46:42.300744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":568,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.238027+0000","last_change":"2024-08-17T17:35:00.238027+0000","last_active":"2024-08-17T17:35:00.238027+0000","last_peered":"2024-08-17T17:35:00.238027+0000","last_clean":"2024-08-17T17:35:00.238027+0000","last_became_active":"2024-08-17T17:34:17.885483+0000","last_became_peered":"2024-08-17T17:34:17.885483+0000","last_unstale":"2024-08-17T17:35:00.238027+0000","last_undegraded":"2024-08-17T17:35:00.238027+0000","last_fullsized":"2024-08-17T17:35:00.238027+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:04:01.141401+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-17T17:35:01.152918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":19,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"282'8413","reported_seq":8465,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:03.905558+0000","last_change":"2024-08-17T17:32:48.460706+0000","last_active":"2024-08-17T17:35:03.905558+0000","last_peered":"2024-08-17T17:35:03.905558+0000","last_clean":"2024-08-17T17:35:03.905558+0000","last_became_active":"2024-08-17T17:32:48.460309+0000","last_became_peered":"2024-08-17T17:32:48.460309+0000","last_unstale":"2024-08-17T17:35:03.905558+0000","last_undegraded":"2024-08-17T17:35:03.905558+0000","last_fullsized":"2024-08-17T17:35:03.905558+0000","mapping_epoch":262,"log_start":"266'2000","ondisk_log_start":"266'2000","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":6413,"log_dups_size":0,"ondisk_log_size":6413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:33:48.006665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336003072,"num_objects":5127,"num_object_clones":0,"num_object_copies":15381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8413,"num_write_kb":433280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":107,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.225822+0000","last_change":"2024-08-17T17:33:44.163867+0000","last_active":"2024-08-17T17:35:00.225822+0000","last_peered":"2024-08-17T17:35:00.225822+0000","last_clean":"2024-08-17T17:35:00.225822+0000","last_became_active":"2024-08-17T17:33:44.163503+0000","last_became_peered":"2024-08-17T17:33:44.163503+0000","last_unstale":"2024-08-17T17:35:00.225822+0000","last_undegraded":"2024-08-17T17:35:00.225822+0000","last_fullsized":"2024-08-17T17:35:00.225822+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:31:53.573596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":572,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.239753+0000","last_change":"2024-08-17T17:33:20.799350+0000","last_active":"2024-08-17T17:35:00.239753+0000","last_peered":"2024-08-17T17:35:00.239753+0000","last_clean":"2024-08-17T17:35:00.239753+0000","last_became_active":"2024-08-17T17:33:20.798925+0000","last_became_peered":"2024-08-17T17:33:20.798925+0000","last_unstale":"2024-08-17T17:35:00.239753+0000","last_undegraded":"2024-08-17T17:35:00.239753+0000","last_fullsized":"2024-08-17T17:35:00.239753+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:16:36.121295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":367,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.240183+0000","last_change":"2024-08-17T17:33:20.835812+0000","last_active":"2024-08-17T17:35:00.240183+0000","last_peered":"2024-08-17T17:35:00.240183+0000","last_clean":"2024-08-17T17:35:00.240183+0000","last_became_active":"2024-08-17T17:33:20.835465+0000","last_became_peered":"2024-08-17T17:33:20.835465+0000","last_unstale":"2024-08-17T17:35:00.240183+0000","last_undegraded":"2024-08-17T17:35:00.240183+0000","last_fullsized":"2024-08-17T17:35:00.240183+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:35:47.750812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":568,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.239065+0000","last_change":"2024-08-17T17:35:00.239065+0000","last_active":"2024-08-17T17:35:00.239065+0000","last_peered":"2024-08-17T17:35:00.239065+0000","last_clean":"2024-08-17T17:35:00.239065+0000","last_became_active":"2024-08-17T17:34:17.887618+0000","last_became_peered":"2024-08-17T17:34:17.887618+0000","last_unstale":"2024-08-17T17:35:00.239065+0000","last_undegraded":"2024-08-17T17:35:00.239065+0000","last_fullsized":"2024-08-17T17:35:00.239065+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:04:01.141401+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-17T17:35:04.144924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"282'8486","reported_seq":13852,"reported_epoch":282,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-17T17:35:03.351741+0000","last_change":"2024-08-17T17:33:22.100856+0000","last_active":"2024-08-17T17:35:03.351741+0000","last_peered":"2024-08-17T17:35:03.351741+0000","last_clean":"2024-08-17T17:32:47.362949+0000","last_became_active":"2024-08-17T17:33:22.067826+0000","last_became_peered":"2024-08-17T17:33:22.067826+0000","last_unstale":"2024-08-17T17:35:03.351741+0000","last_undegraded":"2024-08-17T17:33:21.903889+0000","last_fullsized":"2024-08-17T17:33:21.767223+0000","mapping_epoch":269,"log_start":"265'100","ondisk_log_start":"265'100","created":262,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":8386,"log_dups_size":0,"ondisk_log_size":8386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335544320,"num_objects":5120,"num_object_clones":0,"num_object_copies":15360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8486,"num_write_kb":435392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2656,"num_bytes_recovered":174063616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4186}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":392,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.239643+0000","last_change":"2024-08-17T17:33:20.798613+0000","last_active":"2024-08-17T17:35:00.239643+0000","last_peered":"2024-08-17T17:35:00.239643+0000","last_clean":"2024-08-17T17:35:00.239643+0000","last_became_active":"2024-08-17T17:33:20.798260+0000","last_became_peered":"2024-08-17T17:33:20.798260+0000","last_unstale":"2024-08-17T17:35:00.239643+0000","last_undegraded":"2024-08-17T17:35:00.239643+0000","last_fullsized":"2024-08-17T17:35:00.239643+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:18:00.822681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":758,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-08-17T17:34:41.050144+0000","last_change":"2024-08-17T17:32:49.379365+0000","last_active":"2024-08-17T17:34:41.050144+0000","last_peered":"2024-08-17T17:34:41.050144+0000","last_clean":"2024-08-17T17:34:41.050144+0000","last_became_active":"2024-08-17T17:32:49.378742+0000","last_became_peered":"2024-08-17T17:32:49.378742+0000","last_unstale":"2024-08-17T17:34:41.050144+0000","last_undegraded":"2024-08-17T17:34:41.050144+0000","last_fullsized":"2024-08-17T17:34:41.050144+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:32:06.957948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":407,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.237756+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:35:00.237756+0000","last_peered":"2024-08-17T17:35:00.237756+0000","last_clean":"2024-08-17T17:35:00.237756+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:35:00.237756+0000","last_undegraded":"2024-08-17T17:35:00.237756+0000","last_fullsized":"2024-08-17T17:35:00.237756+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":568,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.237197+0000","last_change":"2024-08-17T17:35:00.237197+0000","last_active":"2024-08-17T17:35:00.237197+0000","last_peered":"2024-08-17T17:35:00.237197+0000","last_clean":"2024-08-17T17:35:00.237197+0000","last_became_active":"2024-08-17T17:34:17.886175+0000","last_became_peered":"2024-08-17T17:34:17.886175+0000","last_unstale":"2024-08-17T17:35:00.237197+0000","last_undegraded":"2024-08-17T17:35:00.237197+0000","last_fullsized":"2024-08-17T17:35:00.237197+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:04:01.141401+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-17T17:35:02.161265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":20,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"282'8451","reported_seq":8502,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:03.872161+0000","last_change":"2024-08-17T17:32:48.381312+0000","last_active":"2024-08-17T17:35:03.872161+0000","last_peered":"2024-08-17T17:35:03.872161+0000","last_clean":"2024-08-17T17:35:03.872161+0000","last_became_active":"2024-08-17T17:32:48.380990+0000","last_became_peered":"2024-08-17T17:32:48.380990+0000","last_unstale":"2024-08-17T17:35:03.872161+0000","last_undegraded":"2024-08-17T17:35:03.872161+0000","last_fullsized":"2024-08-17T17:35:03.872161+0000","mapping_epoch":262,"log_start":"266'2000","ondisk_log_start":"266'2000","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":6451,"log_dups_size":0,"ondisk_log_size":6451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:39:12.341886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335740928,"num_objects":5123,"num_object_clones":0,"num_object_copies":15369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8451,"num_write_kb":434368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":442,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.230267+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:35:00.230267+0000","last_peered":"2024-08-17T17:35:00.230267+0000","last_clean":"2024-08-17T17:35:00.230267+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:35:00.230267+0000","last_undegraded":"2024-08-17T17:35:00.230267+0000","last_fullsized":"2024-08-17T17:35:00.230267+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":925,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.240285+0000","last_change":"2024-08-17T17:33:21.763805+0000","last_active":"2024-08-17T17:35:00.240285+0000","last_peered":"2024-08-17T17:35:00.240285+0000","last_clean":"2024-08-17T17:35:00.240285+0000","last_became_active":"2024-08-17T17:33:21.763232+0000","last_became_peered":"2024-08-17T17:33:21.763232+0000","last_unstale":"2024-08-17T17:35:00.240285+0000","last_undegraded":"2024-08-17T17:35:00.240285+0000","last_fullsized":"2024-08-17T17:35:00.240285+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:46:08.994820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":438,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.238409+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:35:00.238409+0000","last_peered":"2024-08-17T17:35:00.238409+0000","last_clean":"2024-08-17T17:35:00.238409+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:35:00.238409+0000","last_undegraded":"2024-08-17T17:35:00.238409+0000","last_fullsized":"2024-08-17T17:35:00.238409+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":561,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.238040+0000","last_change":"2024-08-17T17:34:19.224000+0000","last_active":"2024-08-17T17:35:00.238040+0000","last_peered":"2024-08-17T17:35:00.238040+0000","last_clean":"2024-08-17T17:35:00.238040+0000","last_became_active":"2024-08-17T17:34:17.885462+0000","last_became_peered":"2024-08-17T17:34:17.885462+0000","last_unstale":"2024-08-17T17:35:00.238040+0000","last_undegraded":"2024-08-17T17:35:00.238040+0000","last_fullsized":"2024-08-17T17:35:00.238040+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:34:19.223644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:34:19.223644+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:49:17.164779+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"282'8311","reported_seq":8358,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:05.004549+0000","last_change":"2024-08-17T17:33:44.160183+0000","last_active":"2024-08-17T17:35:05.004549+0000","last_peered":"2024-08-17T17:35:05.004549+0000","last_clean":"2024-08-17T17:35:05.004549+0000","last_became_active":"2024-08-17T17:33:44.144600+0000","last_became_peered":"2024-08-17T17:33:44.144600+0000","last_unstale":"2024-08-17T17:35:05.004549+0000","last_undegraded":"2024-08-17T17:35:05.004549+0000","last_fullsized":"2024-08-17T17:35:05.004549+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":8311,"log_dups_size":0,"ondisk_log_size":8311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:23:53.175287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329515032,"num_objects":5029,"num_object_clones":0,"num_object_copies":15087,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5029,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8311,"num_write_kb":426817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":257,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.242231+0000","last_change":"2024-08-17T17:33:21.765925+0000","last_active":"2024-08-17T17:35:00.242231+0000","last_peered":"2024-08-17T17:35:00.242231+0000","last_clean":"2024-08-17T17:35:00.242231+0000","last_became_active":"2024-08-17T17:33:21.765372+0000","last_became_peered":"2024-08-17T17:33:21.765372+0000","last_unstale":"2024-08-17T17:35:00.242231+0000","last_undegraded":"2024-08-17T17:35:00.242231+0000","last_fullsized":"2024-08-17T17:35:00.242231+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:14:47.737369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":513,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-08-17T17:34:41.051707+0000","last_change":"2024-08-17T17:26:06.855307+0000","last_active":"2024-08-17T17:34:41.051707+0000","last_peered":"2024-08-17T17:34:41.051707+0000","last_clean":"2024-08-17T17:34:41.051707+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:34:41.051707+0000","last_undegraded":"2024-08-17T17:34:41.051707+0000","last_fullsized":"2024-08-17T17:34:41.051707+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":27,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-08-17T17:34:41.051366+0000","last_change":"2024-08-17T17:33:44.149552+0000","last_active":"2024-08-17T17:34:41.051366+0000","last_peered":"2024-08-17T17:34:41.051366+0000","last_clean":"2024-08-17T17:34:41.051366+0000","last_became_active":"2024-08-17T17:33:44.149221+0000","last_became_peered":"2024-08-17T17:33:44.149221+0000","last_unstale":"2024-08-17T17:34:41.051366+0000","last_undegraded":"2024-08-17T17:34:41.051366+0000","last_fullsized":"2024-08-17T17:34:41.051366+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:33:42.937211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":45,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-08-17T17:34:41.071668+0000","last_change":"2024-08-17T17:33:21.763655+0000","last_active":"2024-08-17T17:34:41.071668+0000","last_peered":"2024-08-17T17:34:41.071668+0000","last_clean":"2024-08-17T17:34:41.071668+0000","last_became_active":"2024-08-17T17:33:21.763088+0000","last_became_peered":"2024-08-17T17:33:21.763088+0000","last_unstale":"2024-08-17T17:34:41.071668+0000","last_undegraded":"2024-08-17T17:34:41.071668+0000","last_fullsized":"2024-08-17T17:34:41.071668+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:40:40.126898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":241,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-08-17T17:34:41.049541+0000","last_change":"2024-08-17T17:18:59.430969+0000","last_active":"2024-08-17T17:34:41.049541+0000","last_peered":"2024-08-17T17:34:41.049541+0000","last_clean":"2024-08-17T17:34:41.049541+0000","last_became_active":"2024-08-17T17:18:59.413475+0000","last_became_peered":"2024-08-17T17:18:59.413475+0000","last_unstale":"2024-08-17T17:34:41.049541+0000","last_undegraded":"2024-08-17T17:34:41.049541+0000","last_fullsized":"2024-08-17T17:34:41.049541+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:36:33.732645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":385,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.239931+0000","last_change":"2024-08-17T17:33:20.799402+0000","last_active":"2024-08-17T17:35:00.239931+0000","last_peered":"2024-08-17T17:35:00.239931+0000","last_clean":"2024-08-17T17:35:00.239931+0000","last_became_active":"2024-08-17T17:33:20.798832+0000","last_became_peered":"2024-08-17T17:33:20.798832+0000","last_unstale":"2024-08-17T17:35:00.239931+0000","last_undegraded":"2024-08-17T17:35:00.239931+0000","last_fullsized":"2024-08-17T17:35:00.239931+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:21:05.001134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":134,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.225708+0000","last_change":"2024-08-17T17:32:46.381141+0000","last_active":"2024-08-17T17:35:00.225708+0000","last_peered":"2024-08-17T17:35:00.225708+0000","last_clean":"2024-08-17T17:35:00.225708+0000","last_became_active":"2024-08-17T17:32:46.374259+0000","last_became_peered":"2024-08-17T17:32:46.374259+0000","last_unstale":"2024-08-17T17:35:00.225708+0000","last_undegraded":"2024-08-17T17:35:00.225708+0000","last_fullsized":"2024-08-17T17:35:00.225708+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:19:10.836889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":124,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.240367+0000","last_change":"2024-08-17T17:33:44.146014+0000","last_active":"2024-08-17T17:35:00.240367+0000","last_peered":"2024-08-17T17:35:00.240367+0000","last_clean":"2024-08-17T17:35:00.240367+0000","last_became_active":"2024-08-17T17:33:44.145668+0000","last_became_peered":"2024-08-17T17:33:44.145668+0000","last_unstale":"2024-08-17T17:35:00.240367+0000","last_undegraded":"2024-08-17T17:35:00.240367+0000","last_fullsized":"2024-08-17T17:35:00.240367+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:53:41.679237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":388,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.225122+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:35:00.225122+0000","last_peered":"2024-08-17T17:35:00.225122+0000","last_clean":"2024-08-17T17:35:00.225122+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:35:00.225122+0000","last_undegraded":"2024-08-17T17:35:00.225122+0000","last_fullsized":"2024-08-17T17:35:00.225122+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":415,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.237884+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:35:00.237884+0000","last_peered":"2024-08-17T17:35:00.237884+0000","last_clean":"2024-08-17T17:35:00.237884+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:35:00.237884+0000","last_undegraded":"2024-08-17T17:35:00.237884+0000","last_fullsized":"2024-08-17T17:35:00.237884+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":446,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.238326+0000","last_change":"2024-08-17T17:33:20.790687+0000","last_active":"2024-08-17T17:35:00.238326+0000","last_peered":"2024-08-17T17:35:00.238326+0000","last_clean":"2024-08-17T17:35:00.238326+0000","last_became_active":"2024-08-17T17:33:20.790399+0000","last_became_peered":"2024-08-17T17:33:20.790399+0000","last_unstale":"2024-08-17T17:35:00.238326+0000","last_undegraded":"2024-08-17T17:35:00.238326+0000","last_fullsized":"2024-08-17T17:35:00.238326+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:38:06.849989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":256,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-08-17T17:34:41.051672+0000","last_change":"2024-08-17T17:33:20.835760+0000","last_active":"2024-08-17T17:34:41.051672+0000","last_peered":"2024-08-17T17:34:41.051672+0000","last_clean":"2024-08-17T17:34:41.051672+0000","last_became_active":"2024-08-17T17:33:20.835410+0000","last_became_peered":"2024-08-17T17:33:20.835410+0000","last_unstale":"2024-08-17T17:34:41.051672+0000","last_undegraded":"2024-08-17T17:34:41.051672+0000","last_fullsized":"2024-08-17T17:34:41.051672+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:45.448022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":464,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.238480+0000","last_change":"2024-08-17T17:33:20.771279+0000","last_active":"2024-08-17T17:35:00.238480+0000","last_peered":"2024-08-17T17:35:00.238480+0000","last_clean":"2024-08-17T17:35:00.238480+0000","last_became_active":"2024-08-17T17:33:20.770728+0000","last_became_peered":"2024-08-17T17:33:20.770728+0000","last_unstale":"2024-08-17T17:35:00.238480+0000","last_undegraded":"2024-08-17T17:35:00.238480+0000","last_fullsized":"2024-08-17T17:35:00.238480+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:52:31.540397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":233,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.242065+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:35:00.242065+0000","last_peered":"2024-08-17T17:35:00.242065+0000","last_clean":"2024-08-17T17:35:00.242065+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:35:00.242065+0000","last_undegraded":"2024-08-17T17:35:00.242065+0000","last_fullsized":"2024-08-17T17:35:00.242065+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":363,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.240019+0000","last_change":"2024-08-17T17:33:20.797460+0000","last_active":"2024-08-17T17:35:00.240019+0000","last_peered":"2024-08-17T17:35:00.240019+0000","last_clean":"2024-08-17T17:35:00.240019+0000","last_became_active":"2024-08-17T17:33:20.797154+0000","last_became_peered":"2024-08-17T17:33:20.797154+0000","last_unstale":"2024-08-17T17:35:00.240019+0000","last_undegraded":"2024-08-17T17:35:00.240019+0000","last_fullsized":"2024-08-17T17:35:00.240019+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:02:11.149053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":63,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-08-17T17:34:41.071695+0000","last_change":"2024-08-17T17:33:44.160733+0000","last_active":"2024-08-17T17:34:41.071695+0000","last_peered":"2024-08-17T17:34:41.071695+0000","last_clean":"2024-08-17T17:34:41.071695+0000","last_became_active":"2024-08-17T17:33:44.160368+0000","last_became_peered":"2024-08-17T17:33:44.160368+0000","last_unstale":"2024-08-17T17:34:41.071695+0000","last_undegraded":"2024-08-17T17:34:41.071695+0000","last_fullsized":"2024-08-17T17:34:41.071695+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:37:22.855642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":257,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.240094+0000","last_change":"2024-08-17T17:33:20.774002+0000","last_active":"2024-08-17T17:35:00.240094+0000","last_peered":"2024-08-17T17:35:00.240094+0000","last_clean":"2024-08-17T17:35:00.240094+0000","last_became_active":"2024-08-17T17:33:20.773662+0000","last_became_peered":"2024-08-17T17:33:20.773662+0000","last_unstale":"2024-08-17T17:35:00.240094+0000","last_undegraded":"2024-08-17T17:35:00.240094+0000","last_fullsized":"2024-08-17T17:35:00.240094+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:15:53.475828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":351,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.239919+0000","last_change":"2024-08-17T17:33:20.773207+0000","last_active":"2024-08-17T17:35:00.239919+0000","last_peered":"2024-08-17T17:35:00.239919+0000","last_clean":"2024-08-17T17:35:00.239919+0000","last_became_active":"2024-08-17T17:33:20.772457+0000","last_became_peered":"2024-08-17T17:33:20.772457+0000","last_unstale":"2024-08-17T17:35:00.239919+0000","last_undegraded":"2024-08-17T17:35:00.239919+0000","last_fullsized":"2024-08-17T17:35:00.239919+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:57:09.610084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":243,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.237600+0000","last_change":"2024-08-17T17:32:46.375464+0000","last_active":"2024-08-17T17:35:00.237600+0000","last_peered":"2024-08-17T17:35:00.237600+0000","last_clean":"2024-08-17T17:35:00.237600+0000","last_became_active":"2024-08-17T17:32:46.375161+0000","last_became_peered":"2024-08-17T17:32:46.375161+0000","last_unstale":"2024-08-17T17:35:00.237600+0000","last_undegraded":"2024-08-17T17:35:00.237600+0000","last_fullsized":"2024-08-17T17:35:00.237600+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:27:47.091836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":453,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.226538+0000","last_change":"2024-08-17T17:33:20.774789+0000","last_active":"2024-08-17T17:35:00.226538+0000","last_peered":"2024-08-17T17:35:00.226538+0000","last_clean":"2024-08-17T17:35:00.226538+0000","last_became_active":"2024-08-17T17:33:20.772750+0000","last_became_peered":"2024-08-17T17:33:20.772750+0000","last_unstale":"2024-08-17T17:35:00.226538+0000","last_undegraded":"2024-08-17T17:35:00.226538+0000","last_fullsized":"2024-08-17T17:35:00.226538+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:21:43.839988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":296,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-08-17T17:34:41.071589+0000","last_change":"2024-08-17T17:33:44.165175+0000","last_active":"2024-08-17T17:34:41.071589+0000","last_peered":"2024-08-17T17:34:41.071589+0000","last_clean":"2024-08-17T17:34:41.071589+0000","last_became_active":"2024-08-17T17:33:44.146198+0000","last_became_peered":"2024-08-17T17:33:44.146198+0000","last_unstale":"2024-08-17T17:34:41.071589+0000","last_undegraded":"2024-08-17T17:34:41.071589+0000","last_fullsized":"2024-08-17T17:34:41.071589+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:14:56.291775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":428,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.225776+0000","last_change":"2024-08-17T17:18:59.418434+0000","last_active":"2024-08-17T17:35:00.225776+0000","last_peered":"2024-08-17T17:35:00.225776+0000","last_clean":"2024-08-17T17:35:00.225776+0000","last_became_active":"2024-08-17T17:18:59.418043+0000","last_became_peered":"2024-08-17T17:18:59.418043+0000","last_unstale":"2024-08-17T17:35:00.225776+0000","last_undegraded":"2024-08-17T17:35:00.225776+0000","last_fullsized":"2024-08-17T17:35:00.225776+0000","mapping_epoch":114,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":115,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:25:51.371820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":276,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-08-17T17:34:41.071670+0000","last_change":"2024-08-17T17:33:44.148090+0000","last_active":"2024-08-17T17:34:41.071670+0000","last_peered":"2024-08-17T17:34:41.071670+0000","last_clean":"2024-08-17T17:34:41.071670+0000","last_became_active":"2024-08-17T17:33:44.147539+0000","last_became_peered":"2024-08-17T17:33:44.147539+0000","last_unstale":"2024-08-17T17:34:41.071670+0000","last_undegraded":"2024-08-17T17:34:41.071670+0000","last_fullsized":"2024-08-17T17:34:41.071670+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:52:02.311332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":442,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.225513+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:35:00.225513+0000","last_peered":"2024-08-17T17:35:00.225513+0000","last_clean":"2024-08-17T17:35:00.225513+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:35:00.225513+0000","last_undegraded":"2024-08-17T17:35:00.225513+0000","last_fullsized":"2024-08-17T17:35:00.225513+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":442,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.225988+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:35:00.225988+0000","last_peered":"2024-08-17T17:35:00.225988+0000","last_clean":"2024-08-17T17:35:00.225988+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:35:00.225988+0000","last_undegraded":"2024-08-17T17:35:00.225988+0000","last_fullsized":"2024-08-17T17:35:00.225988+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":31,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-08-17T17:34:41.071432+0000","last_change":"2024-08-17T17:32:46.376844+0000","last_active":"2024-08-17T17:34:41.071432+0000","last_peered":"2024-08-17T17:34:41.071432+0000","last_clean":"2024-08-17T17:34:41.071432+0000","last_became_active":"2024-08-17T17:32:46.376483+0000","last_became_peered":"2024-08-17T17:32:46.376483+0000","last_unstale":"2024-08-17T17:34:41.071432+0000","last_undegraded":"2024-08-17T17:34:41.071432+0000","last_fullsized":"2024-08-17T17:34:41.071432+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:44:24.358662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":252,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-08-17T17:35:00.237772+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:35:00.237772+0000","last_peered":"2024-08-17T17:35:00.237772+0000","last_clean":"2024-08-17T17:35:00.237772+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:35:00.237772+0000","last_undegraded":"2024-08-17T17:35:00.237772+0000","last_fullsized":"2024-08-17T17:35:00.237772+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":5320409112,"num_objects":81184,"num_object_clones":0,"num_object_copies":243552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22295,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81184,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133762,"num_write_kb":6878209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2660,"num_bytes_recovered":174325760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15131553792,"data_stored":15131541576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122862,"ondisk_log_size":122862,"up":48,"acting":41,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":6186,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19280,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5978,"num_read_kb":12991,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":912387,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":205,"num_read_kb":448,"num_write":117,"num_write_kb":1318,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1610094,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":33,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234339,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2231724,"kb_used_data":1633508,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":91513428,"statfs":{"total":95995035648,"available":93709750272,"internally_reserved":0,"allocated":1672712192,"data_stored":1671105130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495974,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2537504,"kb_used_data":1638232,"kb_used_omap":1,"kb_used_meta":899262,"kb_avail":91207648,"statfs":{"total":95995035648,"available":93396631552,"internally_reserved":0,"allocated":1677549568,"data_stored":1675954478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":920844632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757611,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2902104,"kb_used_data":2600592,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":90843048,"statfs":{"total":95995035648,"available":93023281152,"internally_reserved":0,"allocated":2663006208,"data_stored":2661414781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019249,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1932116,"kb_used_data":1626444,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91813036,"statfs":{"total":95995035648,"available":94016548864,"internally_reserved":0,"allocated":1665478656,"data_stored":1663891085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435476,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2598992,"kb_used_data":1971912,"kb_used_omap":4,"kb_used_meta":627003,"kb_avail":91146160,"statfs":{"total":95995035648,"available":93333667840,"internally_reserved":0,"allocated":2019237888,"data_stored":2017654411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":642051082},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599003,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2502500,"kb_used_data":1941980,"kb_used_omap":7,"kb_used_meta":560504,"kb_avail":91242652,"statfs":{"total":95995035648,"available":93432475648,"internally_reserved":0,"allocated":1988587520,"data_stored":1986988845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":573956258},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":268,"seq":1151051235350,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1636688,"kb_used_data":1486792,"kb_used_omap":10,"kb_used_meta":149877,"kb_avail":92108464,"statfs":{"total":95995035648,"available":94319067136,"internally_reserved":0,"allocated":1522475008,"data_stored":1520891476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10902,"internal_metadata":153474410},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926056,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2520280,"kb_used_data":1978192,"kb_used_omap":7,"kb_used_meta":542008,"kb_avail":91224872,"statfs":{"total":95995035648,"available":93414268928,"internally_reserved":0,"allocated":2025668608,"data_stored":2024077993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":555016513},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7550,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2008285184,"data_stored":2008285184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1518010368,"data_stored":1518010368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1977159680,"data_stored":1977155608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2006056960,"data_stored":2006056960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1656160256,"data_stored":1656160256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2649886720,"data_stored":2649882648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1660485632,"data_stored":1660485632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1655508992,"data_stored":1655504920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:35:11.103 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd pool set .mgr pgp_num 11 2024-08-17T17:35:11.104 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:11.130 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:11.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:11.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:11.231 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:11.254 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:11.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:11.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:11.349 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:35:11.356 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:11.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:11.382 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:11.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:11.453 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:35:11.454 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:35:11.483 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:11.498 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:35:11.498 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:35:11.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:11.510 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:11.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:11.544 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:35:11.544 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:35:11.579 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:35:11.579 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:35:11.580 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:11 smithi003 ceph-mon[100615]: pgmap v871: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 4.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 690 op/s; 22131/233022 objects degraded (9.497%); 1.7 MiB/s, 29 objects/s recovering 2024-08-17T17:35:11.580 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:11 smithi003 ceph-mon[103344]: pgmap v871: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 4.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 690 op/s; 22131/233022 objects degraded (9.497%); 1.7 MiB/s, 29 objects/s recovering 2024-08-17T17:35:11.611 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:11.612 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:35:11.612 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:35:11.647 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:35:11.648 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:35:11.663 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:11.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:11.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:11.677 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:35:11.677 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:35:11.711 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:35:11.712 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:35:11.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:11 smithi053 ceph-mon[89963]: pgmap v871: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 4.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 690 op/s; 22131/233022 objects degraded (9.497%); 1.7 MiB/s, 29 objects/s recovering 2024-08-17T17:35:11.766 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:11.789 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:11.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:11.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:11.891 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:11.927 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:11.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:11.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:12.029 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:12.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:12.075 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:12.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:12.176 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:12.206 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:12.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:12.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:12.307 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:12.334 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:12.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:12.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:12.436 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:12.470 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:12.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:12.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:12.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:12 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3696701090' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:35:12.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:12 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3696701090' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:35:12.573 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:12.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:12.615 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:12.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:12.715 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:12.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:12 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3696701090' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:35:12.742 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:12.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:12.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:12.844 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:12.892 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:12.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:12.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:12.994 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:13.043 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:35:13.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:13.055 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:13.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:13.156 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:13.207 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:13.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:13.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:13.309 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:13.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:13.352 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:13.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:13.453 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:13.476 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:13.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:13.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:13.577 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:13.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:13.622 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:13.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:13.623 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:13 smithi003 ceph-mon[100615]: pgmap v872: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 4.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 579 op/s; 22131/233022 objects degraded (9.497%); 1.4 MiB/s, 25 objects/s recovering 2024-08-17T17:35:13.623 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:13 smithi003 ceph-mon[103344]: pgmap v872: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 4.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 579 op/s; 22131/233022 objects degraded (9.497%); 1.4 MiB/s, 25 objects/s recovering 2024-08-17T17:35:13.676 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:35:13.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:13 smithi053 ceph-mon[89963]: pgmap v872: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 4.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 579 op/s; 22131/233022 objects degraded (9.497%); 1.4 MiB/s, 25 objects/s recovering 2024-08-17T17:35:13.724 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:13.753 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:13.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:13.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:13.855 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:13.893 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:13.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:13.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:13.995 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:14.023 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:14.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:14.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:14.124 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:14.155 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:14.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:14.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:14.256 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:14.293 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:14.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:14.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:14.395 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:14.437 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:14.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:14.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:14.539 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:14.581 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:35:14.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:14.594 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:14.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:14.695 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:14.720 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:14.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:14.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:14.822 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:14.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:14.879 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:14.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:14.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:15.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:15.038 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:15.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:15.139 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:15.200 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:15.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:15.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:15.302 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:15.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:15.353 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:15.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:15.454 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:15.479 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:15.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:15.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:15.581 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:15.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:15.614 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:15.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:15.652 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:15 smithi003 ceph-mon[103344]: pgmap v873: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 812 op/s; 21978/224421 objects degraded (9.793%); 2.2 MiB/s, 37 objects/s recovering 2024-08-17T17:35:15.652 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:15 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 21978/224421 objects degraded (9.793%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:15.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:15 smithi003 ceph-mon[100615]: pgmap v873: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 812 op/s; 21978/224421 objects degraded (9.793%); 2.2 MiB/s, 37 objects/s recovering 2024-08-17T17:35:15.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:15 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 21978/224421 objects degraded (9.793%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:15.715 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:15.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:15 smithi053 ceph-mon[89963]: pgmap v873: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 812 op/s; 21978/224421 objects degraded (9.793%); 2.2 MiB/s, 37 objects/s recovering 2024-08-17T17:35:15.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:15 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 21978/224421 objects degraded (9.793%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:15.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:15.766 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:15.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:15.867 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:15.902 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:15.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:15.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:16.004 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:16.021 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:35:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:35:15] "GET /metrics HTTP/1.1" 200 36536 "" "Prometheus/2.51.0" 2024-08-17T17:35:16.055 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:16.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:16.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:16.158 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:16.187 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:16.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:16.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:16.290 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:16.335 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-08-17T17:35:16.336 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:16.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:16.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:16.438 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:16.489 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:16.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:16.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:16.591 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:16.594 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:16 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3079155449' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:35:16.594 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:16 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:35:16.595 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:16 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3079155449' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:35:16.595 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:16 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:35:16.618 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:16.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:16.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:16.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:16 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3079155449' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:35:16.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:16 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:35:16.720 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:16.758 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:16.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:16.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:16.861 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:16.890 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:16.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:16.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:16.944 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:35:16.992 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:17.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:17.041 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:17.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:17.142 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:17.174 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:17.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:17.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:17.276 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:17.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:17.323 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:17.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:17.424 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:17.465 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:17.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:17.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:17.567 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:17.594 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:17.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:17.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:17.677 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:17 smithi003 ceph-mon[100615]: pgmap v874: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 597 op/s; 21978/223056 objects degraded (9.853%); 1.5 MiB/s, 25 objects/s recovering 2024-08-17T17:35:17.677 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:17 smithi003 ceph-mon[100615]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:35:17.677 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:17 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:35:17.677 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:17 smithi003 ceph-mon[100615]: osdmap e283: 8 total, 8 up, 8 in 2024-08-17T17:35:17.677 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:17 smithi003 ceph-mon[103344]: pgmap v874: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 597 op/s; 21978/223056 objects degraded (9.853%); 1.5 MiB/s, 25 objects/s recovering 2024-08-17T17:35:17.677 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:17 smithi003 ceph-mon[103344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:35:17.677 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:17 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:35:17.677 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:17 smithi003 ceph-mon[103344]: osdmap e283: 8 total, 8 up, 8 in 2024-08-17T17:35:17.696 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:17.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:17 smithi053 ceph-mon[89963]: pgmap v874: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 597 op/s; 21978/223056 objects degraded (9.853%); 1.5 MiB/s, 25 objects/s recovering 2024-08-17T17:35:17.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:17 smithi053 ceph-mon[89963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:35:17.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:17 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:35:17.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:17 smithi053 ceph-mon[89963]: osdmap e283: 8 total, 8 up, 8 in 2024-08-17T17:35:17.723 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:17.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:17.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:17.825 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:17.851 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:17.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:17.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:17.953 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:18.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:18.011 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:18.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:18.112 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:18.137 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:18.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:18.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:18.239 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:18.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:18.287 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:18.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:18.388 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:18.416 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:18.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:18.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:18.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:18.555 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:18.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:18.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:18.657 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:18.683 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:18.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:18.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:18.715 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:35:18.715 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:35:18.785 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:18.819 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:18.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:18.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:18.921 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:18.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:18.972 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:18.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:19.073 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:19.119 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:19.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:19.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:19.221 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:19.249 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:19.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:19.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:19.333 INFO:teuthology.orchestra.run.smithi003.stderr:set pool 1 pgp_num to 11 2024-08-17T17:35:19.351 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:19.376 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:19.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:19.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:19.429 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:35:19.478 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:19.549 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:19.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:19.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:19.654 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:19.714 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:19.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:19.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:19.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:19 smithi053 ceph-mon[89963]: pgmap v876: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 674 op/s; 21978/223056 objects degraded (9.853%); 1.8 MiB/s, 31 objects/s recovering 2024-08-17T17:35:19.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:19 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1023912988' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-08-17T17:35:19.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:19 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-08-17T17:35:19.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:19 smithi003 ceph-mon[100615]: pgmap v876: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 674 op/s; 21978/223056 objects degraded (9.853%); 1.8 MiB/s, 31 objects/s recovering 2024-08-17T17:35:19.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:19 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1023912988' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-08-17T17:35:19.775 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:19 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-08-17T17:35:19.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:19 smithi003 ceph-mon[103344]: pgmap v876: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 674 op/s; 21978/223056 objects degraded (9.853%); 1.8 MiB/s, 31 objects/s recovering 2024-08-17T17:35:19.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:19 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1023912988' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-08-17T17:35:19.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:19 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-08-17T17:35:19.815 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:19.843 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:19.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:19.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:19.945 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:19.984 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:19.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:19.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:20.086 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:20.119 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:20.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:20.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:20.221 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:20.257 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:20.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:20.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:20.359 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:20.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:20.400 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:20.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:20.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:35:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:35:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:35:20.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:20 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-08-17T17:35:20.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:20 smithi053 ceph-mon[89963]: osdmap e284: 8 total, 8 up, 8 in 2024-08-17T17:35:20.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:20 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 21978/223056 objects degraded (9.853%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:20.501 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:20.524 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:20.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:20.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:20.626 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:20.667 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:20.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:20.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:20.769 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:20.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:20 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-08-17T17:35:20.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:20 smithi003 ceph-mon[103344]: osdmap e284: 8 total, 8 up, 8 in 2024-08-17T17:35:20.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:20 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 21978/223056 objects degraded (9.853%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:20.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:20 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-08-17T17:35:20.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:20 smithi003 ceph-mon[100615]: osdmap e284: 8 total, 8 up, 8 in 2024-08-17T17:35:20.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:20 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 21978/223056 objects degraded (9.853%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:20.804 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:20.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:20.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:20.907 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:20.959 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:20.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:20.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:21.061 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:21.093 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:21.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:21.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:21.195 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:21.232 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:21.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:21.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:21.334 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:21.367 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:21.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:21.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:21.469 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:21.508 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:21.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:21.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:21.610 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:21.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:21.676 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:21.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:21.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:21 smithi053 ceph-mon[89963]: pgmap v878: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 859 op/s; 21813/212328 objects degraded (10.273%); 2.5 MiB/s, 39 objects/s recovering 2024-08-17T17:35:21.736 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:21 smithi003 ceph-mon[103344]: pgmap v878: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 859 op/s; 21813/212328 objects degraded (10.273%); 2.5 MiB/s, 39 objects/s recovering 2024-08-17T17:35:21.736 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:21 smithi003 ceph-mon[100615]: pgmap v878: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 859 op/s; 21813/212328 objects degraded (10.273%); 2.5 MiB/s, 39 objects/s recovering 2024-08-17T17:35:21.777 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:21.819 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:21.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:21.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:21.921 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:21.961 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:21.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:21.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:22.063 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:22.097 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:22.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:22.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:22.198 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:22.222 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-17T17:35:22.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:22.251 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:22.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:22.352 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:22.382 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:22.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:22.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:22.484 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:22.507 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:22.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:22.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:22.609 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:22.635 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:22.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:22.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:22.737 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:22.795 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:22.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:22.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:22.896 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:22.937 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:22.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:22.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:23.038 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:23.082 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:23.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:23.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:23.137 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:35:23.184 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:23.210 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:23.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:23.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:23.312 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:23.362 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:23.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:23.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:23.464 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:23.488 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:23.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:23.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:23.590 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:23.614 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:23 smithi003 ceph-mon[103344]: pgmap v879: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 503 op/s; 21813/212328 objects degraded (10.273%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:35:23.614 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:35:23.614 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:35:23.614 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:23 smithi003 ceph-mon[100615]: pgmap v879: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 503 op/s; 21813/212328 objects degraded (10.273%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:35:23.614 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:35:23.614 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:35:23.622 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:23.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:23.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:23 smithi053 ceph-mon[89963]: pgmap v879: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 503 op/s; 21813/212328 objects degraded (10.273%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:35:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:35:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:35:23.724 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:23.773 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:23.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:23.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:23.875 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:23.911 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:23.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:23.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:24.013 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:24.042 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:24.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:24.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:24.144 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:24.244 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:24.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:24.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:24.346 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:24.390 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:24.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:24.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:24.442 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:35:24.443 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:35:24.487 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:35:24.487 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:35:24.492 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:24.523 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:24.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:24.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:24.525 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:35:24.525 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:35:24.557 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:35:24.557 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:35:24.592 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:35:24.592 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:35:24.621 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:35:24.621 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:35:24.625 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:24.659 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:35:24.659 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:35:24.667 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:24.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:24.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:24.698 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:35:24.698 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:35:24.769 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:24.799 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:24.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:24.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:24.901 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:24.943 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:24.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:24.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:25.045 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:25.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:25.087 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:25.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:25.188 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:25.219 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:25.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:25.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:25.321 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:25.351 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:25.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:25.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:25.428 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:35:25.453 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:25.499 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:25.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:25.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:25.601 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:25.629 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:25.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:25.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:25.651 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:25 smithi003 ceph-mon[103344]: pgmap v880: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 833 op/s; 21634/203046 objects degraded (10.655%); 2.7 MiB/s, 42 objects/s recovering 2024-08-17T17:35:25.651 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:25 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2111177677' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:35:25.651 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:25 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 21634/203046 objects degraded (10.655%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:25.651 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:25 smithi003 ceph-mon[100615]: pgmap v880: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 833 op/s; 21634/203046 objects degraded (10.655%); 2.7 MiB/s, 42 objects/s recovering 2024-08-17T17:35:25.651 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:25 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2111177677' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:35:25.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:25 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 21634/203046 objects degraded (10.655%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:25.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:25 smithi053 ceph-mon[89963]: pgmap v880: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 833 op/s; 21634/203046 objects degraded (10.655%); 2.7 MiB/s, 42 objects/s recovering 2024-08-17T17:35:25.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:25 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2111177677' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:35:25.718 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:25 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 21634/203046 objects degraded (10.655%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:25.731 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:25.762 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:25.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:25.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:25.865 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:25.893 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:25.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:25.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:25.995 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:26.020 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:26.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:26.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:26.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:35:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:35:25] "GET /metrics HTTP/1.1" 200 36522 "" "Prometheus/2.51.0" 2024-08-17T17:35:26.121 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:26.166 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:26.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:26.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:26.268 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:26.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:26.311 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:26.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:26.412 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:26.441 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:26.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:26.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:26.543 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:26.566 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:26.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:26.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:26.668 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:26.714 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:26.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:26.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:26.816 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:26.841 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:26.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:26.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:26.943 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:26.987 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:26.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:26.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:27.089 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:27.131 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:27.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:27.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:27.232 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:27.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:27.261 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:27.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:27.362 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:27.404 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:27.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:27.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:27.506 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:27.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:27.555 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:27.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:27.655 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-08-17T17:35:27.656 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 0 1 2024-08-17T17:35:27.656 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:27.663 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:27 smithi003 ceph-mon[100615]: pgmap v881: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 721 op/s; 21634/201822 objects degraded (10.719%); 2.2 MiB/s, 35 objects/s recovering 2024-08-17T17:35:27.663 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:27 smithi003 ceph-mon[103344]: pgmap v881: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 721 op/s; 21634/201822 objects degraded (10.719%); 2.2 MiB/s, 35 objects/s recovering 2024-08-17T17:35:27.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:27.716 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:27.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:27.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:27 smithi053 ceph-mon[89963]: pgmap v881: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 721 op/s; 21634/201822 objects degraded (10.719%); 2.2 MiB/s, 35 objects/s recovering 2024-08-17T17:35:27.817 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:27.845 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:27.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:27.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:27.948 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:27.999 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:28.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:28.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:28.101 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:28.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:28.161 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:28.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:28.262 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:28.289 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:28.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:28.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:28.391 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:28.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:28.444 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:28.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:28.545 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:28.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:28.598 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:28.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:28.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:28.744 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:28.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:28.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:28.846 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:28.883 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:28.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:28.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:28.986 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:28.995 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:35:29.009 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:29.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:29.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:29.111 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:29.175 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:29.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:29.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:29.276 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:29.302 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:29.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:29.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:29.329 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:35:29.398 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-08-17T17:35:29.404 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:29.425 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:29 smithi003 ceph-mon[100615]: pgmap v882: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 706 op/s; 21634/201822 objects degraded (10.719%); 2.1 MiB/s, 34 objects/s recovering 2024-08-17T17:35:29.426 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:29 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/365665770' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:35:29.426 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:29 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:35:29.426 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:29 smithi003 ceph-mon[103344]: pgmap v882: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 706 op/s; 21634/201822 objects degraded (10.719%); 2.1 MiB/s, 34 objects/s recovering 2024-08-17T17:35:29.426 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:29 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/365665770' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:35:29.426 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:29 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:35:29.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:29.484 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:29.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:29.585 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:29.612 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:29.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:29.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:29.714 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:29.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:29 smithi053 ceph-mon[89963]: pgmap v882: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 706 op/s; 21634/201822 objects degraded (10.719%); 2.1 MiB/s, 34 objects/s recovering 2024-08-17T17:35:29.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:29 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/365665770' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:35:29.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:29 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:35:29.757 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:29.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:29.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:29.859 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:29.886 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:29.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:29.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:29.988 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:30.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:30.029 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:30.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:30.130 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:30.168 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:30.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:30.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:30.270 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:30.320 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:30.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:30.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:30.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:30.459 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:30.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:30.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:30.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:30 smithi053 ceph-mon[89963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:35:30.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:30 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:35:30.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:30 smithi053 ceph-mon[89963]: osdmap e285: 8 total, 8 up, 8 in 2024-08-17T17:35:30.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:30 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 21634/201822 objects degraded (10.719%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:30.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:35:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:35:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:35:30.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:30 smithi003 ceph-mon[103344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:35:30.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:30 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:35:30.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:30 smithi003 ceph-mon[103344]: osdmap e285: 8 total, 8 up, 8 in 2024-08-17T17:35:30.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:30 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 21634/201822 objects degraded (10.719%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:30.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:30 smithi003 ceph-mon[100615]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:35:30.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:30 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:35:30.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:30 smithi003 ceph-mon[100615]: osdmap e285: 8 total, 8 up, 8 in 2024-08-17T17:35:30.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:30 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 21634/201822 objects degraded (10.719%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:30.561 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:30.589 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:30.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:30.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:30.692 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:30.718 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:30.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:30.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:30.820 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:30.870 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:30.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:30.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:30.972 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:31.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:31.058 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:31.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:31.159 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:31.192 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:31.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:31.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:31.294 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:35:31.294 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-08-17T17:35:31.294 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-08-17T17:35:31.294 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-08-17T17:35:31.295 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:35:31.295 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:31.318 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:31.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:31.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:31.420 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:31.446 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:31.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:31.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:31.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:31 smithi003 ceph-mon[103344]: pgmap v884: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 3.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 685 op/s; 21481/191727 objects degraded (11.204%); 2.1 MiB/s, 33 objects/s recovering 2024-08-17T17:35:31.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:31 smithi003 ceph-mon[100615]: pgmap v884: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 3.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 685 op/s; 21481/191727 objects degraded (11.204%); 2.1 MiB/s, 33 objects/s recovering 2024-08-17T17:35:31.547 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:31.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:31.598 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:31.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:31.699 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:31.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:31 smithi053 ceph-mon[89963]: pgmap v884: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 3.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 685 op/s; 21481/191727 objects degraded (11.204%); 2.1 MiB/s, 33 objects/s recovering 2024-08-17T17:35:31.731 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:31.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:31.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:31.833 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:31.892 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:31.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:31.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:31.995 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:32.031 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:32.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:32.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:32.133 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:32.195 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:32.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:32.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:32.297 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:32.337 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:32.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:32.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:32.439 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:32.486 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:32.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:32.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:32.588 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:32.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:32.617 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:32.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:32.650 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:32 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:35:32.715 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:35:32.718 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:32.742 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:32 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:35:32.743 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:32 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:35:32.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:32.775 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:32.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:32.876 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:32.905 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:32.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:32.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:32.971 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-17T17:35:33.007 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:33.037 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:33.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:33.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:33.080 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:35:33.139 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:33.166 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:33.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:33.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:33.268 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:33.291 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:33.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:33.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:33.393 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:33.416 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:33.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:33.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:33.435 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.0 to 1 (10000) 2024-08-17T17:35:33.518 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:33.545 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:33.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:33.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:33.647 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:33.696 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:33.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:33.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:33.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:33 smithi053 ceph-mon[89963]: pgmap v885: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 3.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 685 op/s; 21481/191727 objects degraded (11.204%); 2.1 MiB/s, 33 objects/s recovering 2024-08-17T17:35:33.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:33 smithi053 ceph-mon[89963]: osdmap e286: 8 total, 8 up, 8 in 2024-08-17T17:35:33.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:33 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2404177057' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T17:35:33.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:33 smithi003 ceph-mon[103344]: pgmap v885: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 3.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 685 op/s; 21481/191727 objects degraded (11.204%); 2.1 MiB/s, 33 objects/s recovering 2024-08-17T17:35:33.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:33 smithi003 ceph-mon[103344]: osdmap e286: 8 total, 8 up, 8 in 2024-08-17T17:35:33.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:33 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2404177057' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T17:35:33.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:33 smithi003 ceph-mon[100615]: pgmap v885: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 3.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 685 op/s; 21481/191727 objects degraded (11.204%); 2.1 MiB/s, 33 objects/s recovering 2024-08-17T17:35:33.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:33 smithi003 ceph-mon[100615]: osdmap e286: 8 total, 8 up, 8 in 2024-08-17T17:35:33.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:33 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2404177057' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T17:35:33.798 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:33.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:33.837 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:33.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:33.938 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:33.991 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:33.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:33.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:34.093 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:34.145 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:34.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:34.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:34.247 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:34.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:34.273 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:34.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:34.374 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:34.399 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:34.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:34.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:34.501 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:34.539 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:34.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:34.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:34.641 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:34.679 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:34.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:34.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:34.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:34 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2404177057' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-17T17:35:34.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:34 smithi053 ceph-mon[89963]: osdmap e287: 8 total, 8 up, 8 in 2024-08-17T17:35:34.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:34 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2404177057' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-17T17:35:34.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:34 smithi003 ceph-mon[103344]: osdmap e287: 8 total, 8 up, 8 in 2024-08-17T17:35:34.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:34 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2404177057' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-17T17:35:34.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:34 smithi003 ceph-mon[100615]: osdmap e287: 8 total, 8 up, 8 in 2024-08-17T17:35:34.781 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:34.816 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:34.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:34.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:34.918 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:34.953 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:34.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:34.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:35.055 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:35.097 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:35.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:35.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:35.199 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:35.239 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:35.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:35.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:35.341 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:35.387 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:35.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:35.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:35.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 1 1 2024-08-17T17:35:35.491 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:35.528 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:35.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:35.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:35.630 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:35.658 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:35.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:35.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:35.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:35 smithi053 ceph-mon[89963]: pgmap v888: 99 pgs: 1 activating, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 988 op/s; 21272/184032 objects degraded (11.559%); 3.8 MiB/s, 60 objects/s recovering 2024-08-17T17:35:35.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:35 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 21272/184032 objects degraded (11.559%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:35.760 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:35.777 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:35 smithi003 ceph-mon[100615]: pgmap v888: 99 pgs: 1 activating, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 988 op/s; 21272/184032 objects degraded (11.559%); 3.8 MiB/s, 60 objects/s recovering 2024-08-17T17:35:35.777 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:35 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 21272/184032 objects degraded (11.559%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:35.777 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:35:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:35:35] "GET /metrics HTTP/1.1" 200 36522 "" "Prometheus/2.51.0" 2024-08-17T17:35:35.777 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:35 smithi003 ceph-mon[103344]: pgmap v888: 99 pgs: 1 activating, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 988 op/s; 21272/184032 objects degraded (11.559%); 3.8 MiB/s, 60 objects/s recovering 2024-08-17T17:35:35.778 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:35 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 21272/184032 objects degraded (11.559%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:35.823 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:35.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:35.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:35.924 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:35.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:35.959 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:35.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:36.060 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:36.103 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:36.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:36.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:36.205 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:36.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:36.250 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:36.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:36.351 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:36.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:36.399 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:36.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:36.500 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:36.530 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:36.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:36.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:36.631 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:36.676 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:36.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:36.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:36.779 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:36.803 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:35:36.811 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:36.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:36.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:36.865 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:35:36.913 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:36.941 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:36.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:36.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:37.043 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:37.083 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:37.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:37.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:37.185 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:37.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:37.220 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:37.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:37.321 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:37.372 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:37.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:37.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:37.474 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:37.507 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:37.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:37.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:37.610 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:37.635 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:37.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:37.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:37.737 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:37.782 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:37.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:37.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:37.793 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:35:37.793 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:35:37.846 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:35:37.846 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:35:37.882 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:35:37.882 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:35:37.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:37.918 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:35:37.918 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:35:37.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:37.926 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:37.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:37.955 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:35:37.956 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:35:37.994 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:35:37.994 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:35:38.023 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:35:38.023 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:35:38.027 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:38.056 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:35:38.056 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:35:38.056 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:37 smithi003 ceph-mon[103344]: pgmap v889: 99 pgs: 1 activating, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 497 op/s; 21272/182877 objects degraded (11.632%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T17:35:38.057 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:37 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:35:38.057 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:37 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:35:38.057 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:37 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/800932754' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:35:38.057 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:37 smithi003 ceph-mon[100615]: pgmap v889: 99 pgs: 1 activating, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 497 op/s; 21272/182877 objects degraded (11.632%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T17:35:38.057 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:37 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:35:38.057 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:37 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:35:38.058 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:37 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/800932754' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:35:38.077 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:38.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:38.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:38.179 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:38.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:38.210 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:38.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:38.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:37 smithi053 ceph-mon[89963]: pgmap v889: 99 pgs: 1 activating, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 497 op/s; 21272/182877 objects degraded (11.632%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T17:35:38.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:37 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:35:38.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:37 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:35:38.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:37 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/800932754' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:35:38.311 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:38.348 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:38.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:38.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:38.450 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:38.477 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:38.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:38.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:38.579 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:38.620 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:38.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:38.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:38.721 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:38.760 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:38.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:38.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:38.862 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:38.903 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:38.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:38.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:39.005 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:39.031 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:39.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:39.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:39.133 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:39.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:39.168 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:39.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:39.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:39.291 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:39.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:39.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:39.393 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:39.439 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:39.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:39.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:39.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:35:39.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:35:39.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:39 smithi053 ceph-mon[89963]: pgmap v890: 99 pgs: 1 activating, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 368 op/s; 21272/182877 objects degraded (11.632%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T17:35:39.510 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:35:39.510 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:35:39.510 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:39 smithi003 ceph-mon[103344]: pgmap v890: 99 pgs: 1 activating, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 368 op/s; 21272/182877 objects degraded (11.632%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T17:35:39.510 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:35:39.510 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:35:39.510 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:39 smithi003 ceph-mon[100615]: pgmap v890: 99 pgs: 1 activating, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 92 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 368 op/s; 21272/182877 objects degraded (11.632%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T17:35:39.541 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:39.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:39.569 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:39.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:39.670 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:39.695 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:39.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:39.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:39.797 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:39.844 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:39.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:39.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:39.946 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:39.985 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:39.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:39.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:40.087 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:40.130 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:40.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:40.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:40.232 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:40.258 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:40.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:40.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:40.360 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:40.393 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:40.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:40.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:40.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:40 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 21272/182877 objects degraded (11.632%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:40.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:35:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:35:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:35:40.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:40.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:40 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 21272/182877 objects degraded (11.632%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:40.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:40 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 21272/182877 objects degraded (11.632%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:40.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:40.527 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:40.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:40.628 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:40.660 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:40.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:40.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:40.762 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:40.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:40.801 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:40.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:40.902 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:40.947 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:40.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:40.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:41.049 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:41.084 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:41.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:41.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:41.186 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:41.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:41.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:41.334 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:41.365 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:41.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:41.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:41.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:41 smithi053 ceph-mon[89963]: pgmap v891: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 799 op/s; 21161/172536 objects degraded (12.265%); 2.5 MiB/s, 39 objects/s recovering 2024-08-17T17:35:41.467 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:41.506 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:41.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:41.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:41.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:41 smithi003 ceph-mon[103344]: pgmap v891: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 799 op/s; 21161/172536 objects degraded (12.265%); 2.5 MiB/s, 39 objects/s recovering 2024-08-17T17:35:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:41 smithi003 ceph-mon[100615]: pgmap v891: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 799 op/s; 21161/172536 objects degraded (12.265%); 2.5 MiB/s, 39 objects/s recovering 2024-08-17T17:35:41.607 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:41.650 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:41.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:41.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:41.752 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:41.777 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:41.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:41.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:41.879 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:41.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:41.907 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:41.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:42.008 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:42.042 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:42.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:42.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:42.144 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:42.176 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:42.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:42.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:42.278 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:42.304 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:42.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:42.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:42.406 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:42.429 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:42.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:42.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:42.531 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:42.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:42.585 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:42.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:42.686 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:42.716 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:42.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:42.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:42.817 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:42.844 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:42.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:42.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:42.945 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:42.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:42.975 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:42.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:43.076 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:43.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:43.102 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:43.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:43.203 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:43.229 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:43.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:43.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:43.331 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:43.371 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:43.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:43.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:43.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:43 smithi053 ceph-mon[89963]: pgmap v892: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 658 op/s; 21161/172536 objects degraded (12.265%); 2.1 MiB/s, 32 objects/s recovering 2024-08-17T17:35:43.473 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:43.514 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:43.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:43.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:43.522 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:43 smithi003 ceph-mon[100615]: pgmap v892: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 658 op/s; 21161/172536 objects degraded (12.265%); 2.1 MiB/s, 32 objects/s recovering 2024-08-17T17:35:43.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:43 smithi003 ceph-mon[103344]: pgmap v892: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 658 op/s; 21161/172536 objects degraded (12.265%); 2.1 MiB/s, 32 objects/s recovering 2024-08-17T17:35:43.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:43.653 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:43.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:43.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:43.754 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:43.782 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:43.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:43.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:43.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:43.925 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:43.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:43.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:44.026 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:44.054 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:44.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:44.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:44.156 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:44.187 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:44.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:44.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:44.289 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:44.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:44.331 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:44.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:44.432 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:44.476 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:44.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:44.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:44.537 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:35:44.579 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:44.604 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:44.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:44.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:44.705 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:44.769 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:44.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:44.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:44.871 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:44.927 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:44.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:44.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:45.030 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:45.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:45.073 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:45.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:45.174 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:45.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:45.226 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:45.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:45.327 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:45.381 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:45.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:45.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:45.398 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:45 smithi003 ceph-mon[100615]: pgmap v893: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 884 op/s; 21062/163308 objects degraded (12.897%); 2.4 MiB/s, 39 objects/s recovering 2024-08-17T17:35:45.398 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:45 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 21062/163308 objects degraded (12.897%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:45.399 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:45 smithi003 ceph-mon[103344]: pgmap v893: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 884 op/s; 21062/163308 objects degraded (12.897%); 2.4 MiB/s, 39 objects/s recovering 2024-08-17T17:35:45.399 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:45 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 21062/163308 objects degraded (12.897%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:45.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:45 smithi053 ceph-mon[89963]: pgmap v893: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 884 op/s; 21062/163308 objects degraded (12.897%); 2.4 MiB/s, 39 objects/s recovering 2024-08-17T17:35:45.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:45 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 21062/163308 objects degraded (12.897%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:45.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:45.532 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:45.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:45.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:45.634 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:45.664 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:45.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:45.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:45.766 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:45.813 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:45.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:45.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:45.915 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:45.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:45.943 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:45.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:45.986 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:35:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:35:45] "GET /metrics HTTP/1.1" 200 36521 "" "Prometheus/2.51.0" 2024-08-17T17:35:46.044 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:46.089 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:46.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:46.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:46.152 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:35:46.191 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:46.229 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-08-17T17:35:46.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:46.263 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:46.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:46.365 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:46.389 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:46.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:46.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:46.491 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:46.522 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:46.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:46.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:46.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:46 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1653827585' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:35:46.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:46 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1653827585' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:35:46.554 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:46 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1653827585' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:35:46.624 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:46.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:46.656 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:46.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:46.757 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:46.782 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:46.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:46.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:46.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:46.925 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:46.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:46.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:47.027 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:47.057 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:47.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:47.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:47.159 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:47.185 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:47.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:47.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:47.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:47.333 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:47.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:47.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:47.379 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:35:47.379 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:35:47.435 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:47.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:47.470 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:47.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:47.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:47 smithi003 ceph-mon[100615]: pgmap v894: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 610 op/s; 21062/162027 objects degraded (12.999%); 1.1 MiB/s, 17 objects/s recovering 2024-08-17T17:35:47.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:47 smithi003 ceph-mon[100615]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:35:47.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:47 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1653827585' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:35:47.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:47 smithi003 ceph-mon[100615]: osdmap e288: 8 total, 8 up, 8 in 2024-08-17T17:35:47.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:47 smithi003 ceph-mon[103344]: pgmap v894: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 610 op/s; 21062/162027 objects degraded (12.999%); 1.1 MiB/s, 17 objects/s recovering 2024-08-17T17:35:47.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:47 smithi003 ceph-mon[103344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:35:47.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:47 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1653827585' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:35:47.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:47 smithi003 ceph-mon[103344]: osdmap e288: 8 total, 8 up, 8 in 2024-08-17T17:35:47.571 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:47.599 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:47.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:47.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:47.701 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:47.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:47 smithi053 ceph-mon[89963]: pgmap v894: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 610 op/s; 21062/162027 objects degraded (12.999%); 1.1 MiB/s, 17 objects/s recovering 2024-08-17T17:35:47.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:47 smithi053 ceph-mon[89963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:35:47.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:47 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1653827585' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:35:47.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:47 smithi053 ceph-mon[89963]: osdmap e288: 8 total, 8 up, 8 in 2024-08-17T17:35:47.741 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:47.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:47.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:47.843 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:47.869 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:47.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:47.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:47.970 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:48.022 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:48.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:48.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:48.058 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-17T17:35:48.125 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:48.151 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:48.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:48.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:48.253 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:48.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:48.318 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:48.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:48.419 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:48.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:48.472 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:48.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:48.573 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:48.637 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:48.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:48.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:48.738 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:48.790 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:48.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:48.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:48.892 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:48.964 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:48.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:48.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:49.067 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:49.094 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:49.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:49.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:49.196 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:49.251 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:49.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:49.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:49.353 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:49.378 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:49.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:49.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:49.426 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:35:49.451 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:49 smithi003 ceph-mon[103344]: pgmap v896: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 694 op/s; 21062/162027 objects degraded (12.999%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:35:49.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:49 smithi003 ceph-mon[100615]: pgmap v896: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 694 op/s; 21062/162027 objects degraded (12.999%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:35:49.480 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:49.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:49.548 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:49.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:49.649 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:49.676 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:49.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:49.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:49.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:49 smithi053 ceph-mon[89963]: pgmap v896: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 694 op/s; 21062/162027 objects degraded (12.999%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:35:49.778 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:49.811 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:49.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:49.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:49.858 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:35:49.913 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:49.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:49.971 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:49.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:50.072 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:50.125 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:50.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:50.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:50.226 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:50.252 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.1 to 1 (10000) 2024-08-17T17:35:50.256 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:50.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:50.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:50.358 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:50.388 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:50.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:50.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:50.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:50 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1976312762' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:35:50.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:50 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:35:50.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:50 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 21062/162027 objects degraded (12.999%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:50.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:35:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:35:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:35:50.489 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:50.520 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:50.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:50.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:50.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:50 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1976312762' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:35:50.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:50 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:35:50.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:50 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 21062/162027 objects degraded (12.999%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:50.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:50 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1976312762' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:35:50.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:50 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:35:50.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:50 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 21062/162027 objects degraded (12.999%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:50.587 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:35:50.622 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:50.649 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:50.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:50.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:50.750 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:50.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:50.774 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:50.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:50.875 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:50.901 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:50.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:50.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:51.002 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:51.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:51.040 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:51.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:51.141 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:51.178 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:51.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:51.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:51.280 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:51.323 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:51.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:51.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:51.425 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:51.451 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:51.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:51.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:51.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:51 smithi003 ceph-mon[103344]: pgmap v897: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 630 op/s; 20957/153603 objects degraded (13.644%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:35:51.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:51 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-17T17:35:51.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:51 smithi003 ceph-mon[103344]: osdmap e289: 8 total, 8 up, 8 in 2024-08-17T17:35:51.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:51 smithi003 ceph-mon[100615]: pgmap v897: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 630 op/s; 20957/153603 objects degraded (13.644%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:35:51.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:51 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-17T17:35:51.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:51 smithi003 ceph-mon[100615]: osdmap e289: 8 total, 8 up, 8 in 2024-08-17T17:35:51.553 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:51.582 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:51.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:51.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:51.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:51.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:51 smithi053 ceph-mon[89963]: pgmap v897: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 630 op/s; 20957/153603 objects degraded (13.644%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:35:51.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:51 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-17T17:35:51.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:51 smithi053 ceph-mon[89963]: osdmap e289: 8 total, 8 up, 8 in 2024-08-17T17:35:51.732 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:51.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:51.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:51.834 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:51.875 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:51.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:51.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:51.977 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:52.029 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:52.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:52.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:52.130 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:52.153 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:52.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:52.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:52.254 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:52.280 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:52.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:52.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:52.382 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:52.418 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:52.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:52.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:52.520 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:52.545 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:52.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:52.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:52.647 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:52.689 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:52.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:52.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:52.790 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:52.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:52.836 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:52.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:52.937 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:52.966 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:52.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:52.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:53.068 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:53.120 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:53.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:53.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:53.222 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:53.250 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:53.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:53.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:53.352 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:53.413 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:53.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:53.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:53.462 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:53 smithi003 ceph-mon[103344]: pgmap v899: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 456 op/s; 20957/152343 objects degraded (13.756%); 840 KiB/s, 13 objects/s recovering 2024-08-17T17:35:53.462 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:35:53.462 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:53 smithi003 ceph-mon[100615]: pgmap v899: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 456 op/s; 20957/152343 objects degraded (13.756%); 840 KiB/s, 13 objects/s recovering 2024-08-17T17:35:53.462 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:35:53.515 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:53.549 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:53.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:53.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:53.651 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:53.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:53.691 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:53.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:53.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:53 smithi053 ceph-mon[89963]: pgmap v899: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 3.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 456 op/s; 20957/152343 objects degraded (13.756%); 840 KiB/s, 13 objects/s recovering 2024-08-17T17:35:53.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:35:53.792 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:53.842 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:53.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:53.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:53.916 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 2 1 2024-08-17T17:35:53.944 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:54.000 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:54.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:54.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:54.102 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:54.127 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:54.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:54.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:54.229 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:54.256 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:54.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:54.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:54.358 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:54.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:54.390 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:54.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:54.491 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:54.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:54.542 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:54.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:54.643 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:54.674 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:54.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:54.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:54.776 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:54.802 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:54.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:54.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:54.904 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:54.975 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:54.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:54.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:55.077 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:55.107 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:55.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:55.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:55.209 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:55.249 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:55.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:55.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:55.351 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:55.383 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:55.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:55.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:55.449 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:35:55.484 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:55.490 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:35:55.512 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:55.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:55.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:55.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:55 smithi003 ceph-mon[100615]: pgmap v900: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 772 op/s; 20833/143487 objects degraded (14.519%); 952 KiB/s, 28 objects/s recovering 2024-08-17T17:35:55.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:55 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 20833/143487 objects degraded (14.519%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:55.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:55 smithi003 ceph-mon[103344]: pgmap v900: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 772 op/s; 20833/143487 objects degraded (14.519%); 952 KiB/s, 28 objects/s recovering 2024-08-17T17:35:55.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:55 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 20833/143487 objects degraded (14.519%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:55.613 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:55.643 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:55.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:55.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:55.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:55 smithi053 ceph-mon[89963]: pgmap v900: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 772 op/s; 20833/143487 objects degraded (14.519%); 952 KiB/s, 28 objects/s recovering 2024-08-17T17:35:55.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:55 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 20833/143487 objects degraded (14.519%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:35:55.745 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:55.801 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:55.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:55.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:55.821 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:35:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:35:55] "GET /metrics HTTP/1.1" 200 36519 "" "Prometheus/2.51.0" 2024-08-17T17:35:55.903 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:55.932 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:55.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:55.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:56.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:56.083 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:56.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:56.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:56.185 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:56.229 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:56.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:56.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:56.331 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:56.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:56.359 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:56.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:56.460 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:56.487 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:56.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:56.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:56.588 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:56.616 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:56.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:56.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:56.717 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:56.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:56.766 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:56.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:56.867 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:56.918 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:56.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:56.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:56.966 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:35:56.966 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:35:57.009 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:35:57.009 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:35:57.019 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:57.045 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:57.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:57.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:57.050 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:35:57.051 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:35:57.101 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:35:57.102 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:35:57.142 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:35:57.142 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:35:57.147 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:57.179 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:57.180 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:35:57.180 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:35:57.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:57.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:57.209 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:35:57.209 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:35:57.245 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:35:57.245 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:35:57.281 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:57.325 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:57.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:57.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:57.427 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:57.460 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:57.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:57.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:57.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:57 smithi003 ceph-mon[103344]: pgmap v901: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 667 op/s; 20833/142158 objects degraded (14.655%); 768 KiB/s, 23 objects/s recovering 2024-08-17T17:35:57.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:57 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:35:57.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:57 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:35:57.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:57 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3163453368' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:35:57.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:57 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:35:57.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:57 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:35:57.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:57 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:35:57.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:57 smithi003 ceph-mon[100615]: pgmap v901: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 667 op/s; 20833/142158 objects degraded (14.655%); 768 KiB/s, 23 objects/s recovering 2024-08-17T17:35:57.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:57 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:35:57.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:57 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:35:57.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:57 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3163453368' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:35:57.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:57 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:35:57.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:57 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:35:57.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:57 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:35:57.563 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:57.587 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:57.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:57.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:57.689 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:57.714 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:57.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:57.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:57.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:57 smithi053 ceph-mon[89963]: pgmap v901: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 667 op/s; 20833/142158 objects degraded (14.655%); 768 KiB/s, 23 objects/s recovering 2024-08-17T17:35:57.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:57 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:35:57.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:57 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:35:57.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:57 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3163453368' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:35:57.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:57 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:35:57.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:57 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:35:57.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:57 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:35:57.816 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:57.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:57.843 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:57.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:35:57.944 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:57.983 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:57.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:57.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:58.085 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:58.126 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:58.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:58.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:58.228 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:58.272 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:58.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:58.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:58.374 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:35:58.403 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:58.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:58.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:35:58.505 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:35:58.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:58.528 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:58.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:35:58.629 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:58.666 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:58.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:58.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:58.768 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:58.814 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:58.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:58.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:58.916 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:35:58.951 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:58.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:58.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:35:59.053 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:59.089 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:59.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:59.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:59.191 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:59.228 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:59.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:59.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:59.330 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:35:59.356 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:59.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:59.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:35:59.458 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:35:59.503 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:59.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:59.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:35:59.605 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:35:59.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:59.634 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:59.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:35:59.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:35:59 smithi053 ceph-mon[89963]: pgmap v902: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 662 op/s; 20833/142158 objects degraded (14.655%); 761 KiB/s, 22 objects/s recovering 2024-08-17T17:35:59.736 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:35:59.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:35:59 smithi003 ceph-mon[100615]: pgmap v902: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 662 op/s; 20833/142158 objects degraded (14.655%); 761 KiB/s, 22 objects/s recovering 2024-08-17T17:35:59.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:35:59 smithi003 ceph-mon[103344]: pgmap v902: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 662 op/s; 20833/142158 objects degraded (14.655%); 761 KiB/s, 22 objects/s recovering 2024-08-17T17:35:59.786 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:35:59.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:59.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:35:59.888 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:35:59.925 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:35:59.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:35:59.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:00.027 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:00.055 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:00.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:00.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:00.157 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:00.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:00.193 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:00.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:00.294 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:00.331 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:00.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:00.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:00.432 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:00.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:00 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 20833/142158 objects degraded (14.655%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:00.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:36:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:36:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:36:00.473 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:00.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:00.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:00.575 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:00.611 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:00.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:00.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:00.713 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:00.736 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:00.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:00.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:00.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:00 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 20833/142158 objects degraded (14.655%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:00.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:00 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 20833/142158 objects degraded (14.655%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:00.837 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:00.871 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:00.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:00.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:00.973 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:01.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:01.014 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:01.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:01.115 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:01.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:01.144 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:01.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:01.245 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:01.283 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:01.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:01.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:01.385 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:01.411 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:01.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:01.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:01.513 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:01.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:01.559 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:01.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:01.660 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:01.700 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:01.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:01.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:01.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:01 smithi053 ceph-mon[89963]: pgmap v903: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 679 op/s; 20725/133200 objects degraded (15.559%); 90 KiB/s, 23 objects/s recovering 2024-08-17T17:36:01.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:01 smithi003 ceph-mon[100615]: pgmap v903: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 679 op/s; 20725/133200 objects degraded (15.559%); 90 KiB/s, 23 objects/s recovering 2024-08-17T17:36:01.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:01 smithi003 ceph-mon[103344]: pgmap v903: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 679 op/s; 20725/133200 objects degraded (15.559%); 90 KiB/s, 23 objects/s recovering 2024-08-17T17:36:01.802 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:01.865 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:01.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:01.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:01.969 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:02.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:02.021 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:02.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:02.122 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:02.149 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:36:02.149 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:36:02.149 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.149 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:173272 8.b 8:dcc13dc5:::benchmark_data_smithi003_229103_object65797:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-08-17T17:36:02.149 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:36:02.099112+0000", 2024-08-17T17:36:02.149 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.046115570000000002, 2024-08-17T17:36:02.149 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.046156862999999999, 2024-08-17T17:36:02.149 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:36:02.149 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:36:02.150 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:36:02.150 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:36:02.150 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:36:02.150 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:02.150 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 173272 2024-08-17T17:36:02.150 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.150 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:36:02.150 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.150 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:36:02.150 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.099112+0000", 2024-08-17T17:36:02.150 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:36:02.150 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.151 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.151 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:36:02.151 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.099112+0000", 2024-08-17T17:36:02.153 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:36:02.153 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.153 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.153 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:36:02.153 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.099113+0000", 2024-08-17T17:36:02.153 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2890000000000001e-06 2024-08-17T17:36:02.153 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.154 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.154 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:36:02.154 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.099118+0000", 2024-08-17T17:36:02.154 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2300000000000002e-06 2024-08-17T17:36:02.154 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.154 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.154 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:36:02.154 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.099129+0000", 2024-08-17T17:36:02.154 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1467999999999999e-05 2024-08-17T17:36:02.154 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.154 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.154 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:36:02.154 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.099136+0000", 2024-08-17T17:36:02.154 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.793e-06 2024-08-17T17:36:02.155 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.155 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.155 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:36:02.155 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.108692+0000", 2024-08-17T17:36:02.155 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0095565530000000006 2024-08-17T17:36:02.155 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.155 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.155 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:36:02.155 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.108851+0000", 2024-08-17T17:36:02.155 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000158393 2024-08-17T17:36:02.155 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.155 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.155 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,4", 2024-08-17T17:36:02.156 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.108990+0000", 2024-08-17T17:36:02.156 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013892700000000001 2024-08-17T17:36:02.156 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.156 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.156 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:02.156 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.118792+0000", 2024-08-17T17:36:02.156 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0098019600000000002 2024-08-17T17:36:02.156 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.156 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.156 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:36:02.156 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.118806+0000", 2024-08-17T17:36:02.156 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4732999999999999e-05 2024-08-17T17:36:02.156 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:36:02.156 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:36:02.157 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:36:02.157 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.157 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.157 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:173276 8.0 8:0ae33e44:::benchmark_data_smithi003_229103_object65801:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-08-17T17:36:02.157 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:36:02.109928+0000", 2024-08-17T17:36:02.157 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.035299898000000003, 2024-08-17T17:36:02.157 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.035420803000000001, 2024-08-17T17:36:02.157 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:36:02.157 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:36:02.157 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:36:02.157 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:36:02.157 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:36:02.157 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:02.157 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 173276 2024-08-17T17:36:02.158 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.158 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:36:02.158 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.158 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:36:02.158 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.109928+0000", 2024-08-17T17:36:02.158 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:36:02.158 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.158 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.158 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:36:02.158 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.109928+0000", 2024-08-17T17:36:02.158 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:36:02.158 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.158 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.159 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:36:02.159 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.109930+0000", 2024-08-17T17:36:02.159 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.542e-06 2024-08-17T17:36:02.159 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.159 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.159 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:36:02.159 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.109939+0000", 2024-08-17T17:36:02.159 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4430000000000003e-06 2024-08-17T17:36:02.159 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.159 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.159 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:36:02.159 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.109956+0000", 2024-08-17T17:36:02.159 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6867e-05 2024-08-17T17:36:02.159 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.160 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.160 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:36:02.160 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.109966+0000", 2024-08-17T17:36:02.160 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0028e-05 2024-08-17T17:36:02.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.160 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.160 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:36:02.160 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.110064+0000", 2024-08-17T17:36:02.160 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7992000000000006e-05 2024-08-17T17:36:02.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.160 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.160 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:36:02.160 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.110198+0000", 2024-08-17T17:36:02.160 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013398699999999999 2024-08-17T17:36:02.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.161 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.161 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,5", 2024-08-17T17:36:02.161 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.110334+0000", 2024-08-17T17:36:02.161 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013601299999999999 2024-08-17T17:36:02.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.161 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.161 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:36:02.161 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.115372+0000", 2024-08-17T17:36:02.161 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0050388129999999996 2024-08-17T17:36:02.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.161 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.161 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:02.161 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.123236+0000", 2024-08-17T17:36:02.161 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0078639959999999998 2024-08-17T17:36:02.162 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:36:02.162 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:36:02.162 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:36:02.162 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.162 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.162 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:173277 8.b 8:df220ae8:::benchmark_data_smithi003_229103_object65802:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-08-17T17:36:02.162 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:36:02.109989+0000", 2024-08-17T17:36:02.162 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.035238844999999998, 2024-08-17T17:36:02.162 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.035429845000000001, 2024-08-17T17:36:02.162 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:36:02.163 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:36:02.163 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:36:02.163 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:36:02.163 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:36:02.163 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:02.163 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 173277 2024-08-17T17:36:02.163 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.163 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:36:02.163 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.163 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:36:02.163 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.109989+0000", 2024-08-17T17:36:02.163 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:36:02.163 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.164 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.164 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:36:02.164 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.109989+0000", 2024-08-17T17:36:02.164 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:36:02.164 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.164 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.164 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:36:02.164 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.109989+0000", 2024-08-17T17:36:02.164 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1900000000000003e-07 2024-08-17T17:36:02.164 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.164 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.164 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:36:02.164 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.109992+0000", 2024-08-17T17:36:02.164 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0690000000000002e-06 2024-08-17T17:36:02.165 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.165 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.165 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:36:02.165 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.110000+0000", 2024-08-17T17:36:02.165 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2899999999999997e-06 2024-08-17T17:36:02.165 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.165 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.165 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:36:02.165 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.110005+0000", 2024-08-17T17:36:02.165 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7219999999999996e-06 2024-08-17T17:36:02.165 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.165 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.165 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:36:02.166 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.113856+0000", 2024-08-17T17:36:02.166 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038508930000000002 2024-08-17T17:36:02.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.166 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.166 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:36:02.166 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.114005+0000", 2024-08-17T17:36:02.166 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014892799999999999 2024-08-17T17:36:02.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.166 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.166 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,4", 2024-08-17T17:36:02.166 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.114152+0000", 2024-08-17T17:36:02.166 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014711100000000001 2024-08-17T17:36:02.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.167 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.167 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:02.167 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.123555+0000", 2024-08-17T17:36:02.167 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0094028050000000002 2024-08-17T17:36:02.167 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.167 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.167 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:36:02.167 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.124131+0000", 2024-08-17T17:36:02.167 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00057625500000000002 2024-08-17T17:36:02.167 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:36:02.167 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:36:02.167 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:36:02.167 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.167 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.168 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:173282 8.0 8:00cf010d:::benchmark_data_smithi003_229103_object65807:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-08-17T17:36:02.168 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:36:02.123370+0000", 2024-08-17T17:36:02.168 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.021857607000000001, 2024-08-17T17:36:02.168 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.022142155, 2024-08-17T17:36:02.168 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:36:02.168 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:36:02.168 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:36:02.168 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:36:02.168 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:36:02.168 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:02.168 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 173282 2024-08-17T17:36:02.168 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.168 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:36:02.168 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.169 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:36:02.169 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.123370+0000", 2024-08-17T17:36:02.169 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:36:02.169 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.169 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.169 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:36:02.169 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.123370+0000", 2024-08-17T17:36:02.169 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:36:02.169 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.169 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.169 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:36:02.169 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.123371+0000", 2024-08-17T17:36:02.169 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3e-06 2024-08-17T17:36:02.169 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.169 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.170 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:36:02.170 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.123376+0000", 2024-08-17T17:36:02.170 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5079999999999999e-06 2024-08-17T17:36:02.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.170 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.170 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:36:02.170 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.123388+0000", 2024-08-17T17:36:02.170 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2063e-05 2024-08-17T17:36:02.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.170 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.170 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:36:02.170 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.123395+0000", 2024-08-17T17:36:02.170 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0949999999999998e-06 2024-08-17T17:36:02.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.171 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.171 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:36:02.171 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.123777+0000", 2024-08-17T17:36:02.171 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00038202199999999998 2024-08-17T17:36:02.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.171 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.171 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:36:02.171 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.123931+0000", 2024-08-17T17:36:02.171 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015363599999999999 2024-08-17T17:36:02.172 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.172 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,5", 2024-08-17T17:36:02.172 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.124049+0000", 2024-08-17T17:36:02.172 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011811699999999999 2024-08-17T17:36:02.172 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.172 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:02.172 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.129811+0000", 2024-08-17T17:36:02.172 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.005762713 2024-08-17T17:36:02.172 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.172 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:36:02.172 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.129853+0000", 2024-08-17T17:36:02.173 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1823999999999998e-05 2024-08-17T17:36:02.173 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:36:02.173 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:36:02.173 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:36:02.173 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.173 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.173 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:173290 8.0 8:0091b6f5:::benchmark_data_smithi003_229103_object65815:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-08-17T17:36:02.173 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:36:02.132966+0000", 2024-08-17T17:36:02.173 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.012261831000000001, 2024-08-17T17:36:02.173 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012616087999999999, 2024-08-17T17:36:02.173 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:36:02.173 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:36:02.173 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:36:02.173 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:36:02.174 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:36:02.174 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:02.174 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 173290 2024-08-17T17:36:02.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.174 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:36:02.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.174 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:36:02.174 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.132966+0000", 2024-08-17T17:36:02.174 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:36:02.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.174 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:36:02.174 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.132966+0000", 2024-08-17T17:36:02.174 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:36:02.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.175 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:36:02.175 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.132968+0000", 2024-08-17T17:36:02.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9999999999999999e-06 2024-08-17T17:36:02.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.175 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:36:02.175 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.132975+0000", 2024-08-17T17:36:02.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1219999999999997e-06 2024-08-17T17:36:02.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.175 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:36:02.175 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.132997+0000", 2024-08-17T17:36:02.176 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1653000000000001e-05 2024-08-17T17:36:02.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.176 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:36:02.176 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.133006+0000", 2024-08-17T17:36:02.176 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7990000000000005e-06 2024-08-17T17:36:02.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.176 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:36:02.176 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.134506+0000", 2024-08-17T17:36:02.176 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001499706 2024-08-17T17:36:02.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.176 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:36:02.177 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.134876+0000", 2024-08-17T17:36:02.177 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00036945599999999999 2024-08-17T17:36:02.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.177 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.177 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,5", 2024-08-17T17:36:02.177 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.135317+0000", 2024-08-17T17:36:02.177 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00044136300000000001 2024-08-17T17:36:02.177 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:36:02.177 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:36:02.177 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:36:02.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.177 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.177 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:173292 8.8 8:1b30ad4a:::benchmark_data_smithi003_229103_object65817:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-08-17T17:36:02.178 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:36:02.138128+0000", 2024-08-17T17:36:02.178 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0070995279999999999, 2024-08-17T17:36:02.178 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.007512693, 2024-08-17T17:36:02.178 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:36:02.178 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:36:02.178 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:36:02.178 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:36:02.178 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:36:02.178 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:02.178 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 173292 2024-08-17T17:36:02.178 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.178 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:36:02.179 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.179 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:36:02.179 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.138128+0000", 2024-08-17T17:36:02.179 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:36:02.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.179 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.179 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:36:02.179 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.138128+0000", 2024-08-17T17:36:02.179 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:36:02.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.179 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.179 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:36:02.179 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.138130+0000", 2024-08-17T17:36:02.180 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3010000000000001e-06 2024-08-17T17:36:02.180 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.180 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:36:02.180 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.138138+0000", 2024-08-17T17:36:02.181 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7419999999999997e-06 2024-08-17T17:36:02.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.181 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.181 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:36:02.181 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.138160+0000", 2024-08-17T17:36:02.181 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2098000000000001e-05 2024-08-17T17:36:02.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.181 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.181 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:36:02.181 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.138171+0000", 2024-08-17T17:36:02.181 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0794e-05 2024-08-17T17:36:02.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.181 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.182 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:36:02.182 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.138521+0000", 2024-08-17T17:36:02.182 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00035018000000000001 2024-08-17T17:36:02.182 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.182 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:36:02.182 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.138620+0000", 2024-08-17T17:36:02.182 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8951999999999997e-05 2024-08-17T17:36:02.182 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.182 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-08-17T17:36:02.182 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.138718+0000", 2024-08-17T17:36:02.182 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7647999999999997e-05 2024-08-17T17:36:02.182 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:36:02.183 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:36:02.183 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:36:02.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.183 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:173294 8.0 8.1becc7f0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-08-17T17:36:02.183 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:36:02.138194+0000", 2024-08-17T17:36:02.183 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0070336189999999996, 2024-08-17T17:36:02.183 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0075049840000000001, 2024-08-17T17:36:02.183 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:36:02.183 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:36:02.183 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-08-17T17:36:02.183 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:36:02.183 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:36:02.184 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:02.184 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 173294 2024-08-17T17:36:02.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.184 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:36:02.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.184 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:36:02.184 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.138194+0000", 2024-08-17T17:36:02.184 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:36:02.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.184 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:36:02.184 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.138194+0000", 2024-08-17T17:36:02.184 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:36:02.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.185 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.185 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:36:02.185 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.138194+0000", 2024-08-17T17:36:02.185 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.82e-07 2024-08-17T17:36:02.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.185 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.185 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:36:02.185 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.138197+0000", 2024-08-17T17:36:02.185 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7700000000000002e-06 2024-08-17T17:36:02.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.185 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.185 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:36:02.185 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.138203+0000", 2024-08-17T17:36:02.186 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1120000000000001e-06 2024-08-17T17:36:02.186 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.186 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.186 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:36:02.186 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.138208+0000", 2024-08-17T17:36:02.186 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7360000000000001e-06 2024-08-17T17:36:02.186 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:36:02.186 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:36:02.186 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:36:02.186 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.186 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.186 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:173296 8.b 8:db1647bd:::benchmark_data_smithi003_229103_object65821:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-08-17T17:36:02.186 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:36:02.138214+0000", 2024-08-17T17:36:02.187 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0070132900000000001, 2024-08-17T17:36:02.187 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0075305449999999996, 2024-08-17T17:36:02.187 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:36:02.187 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:36:02.187 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:36:02.187 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:36:02.187 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:36:02.187 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:02.187 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 173296 2024-08-17T17:36:02.187 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.187 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:36:02.187 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.187 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:36:02.188 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.138214+0000", 2024-08-17T17:36:02.188 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:36:02.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.188 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:36:02.188 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.138214+0000", 2024-08-17T17:36:02.188 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:36:02.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.188 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:36:02.188 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.138215+0000", 2024-08-17T17:36:02.188 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9999999999999999e-07 2024-08-17T17:36:02.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:36:02.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.138216+0000", 2024-08-17T17:36:02.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8449999999999999e-06 2024-08-17T17:36:02.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:36:02.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.138221+0000", 2024-08-17T17:36:02.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5789999999999997e-06 2024-08-17T17:36:02.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:36:02.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.138225+0000", 2024-08-17T17:36:02.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9509999999999999e-06 2024-08-17T17:36:02.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.190 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:36:02.190 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.138326+0000", 2024-08-17T17:36:02.190 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010112500000000001 2024-08-17T17:36:02.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.190 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:36:02.190 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.138468+0000", 2024-08-17T17:36:02.190 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014163899999999999 2024-08-17T17:36:02.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.190 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,4", 2024-08-17T17:36:02.191 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.138599+0000", 2024-08-17T17:36:02.191 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013167300000000001 2024-08-17T17:36:02.191 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:36:02.191 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:36:02.191 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:36:02.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.191 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.46025.0:173290 8.0 e289/269)", 2024-08-17T17:36:02.191 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:36:02.140857+0000", 2024-08-17T17:36:02.191 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0043705380000000002, 2024-08-17T17:36:02.191 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049430680000000001, 2024-08-17T17:36:02.191 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:36:02.191 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:36:02.191 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-08-17T17:36:02.192 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:36:02.192 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.192 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:36:02.192 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.140857+0000", 2024-08-17T17:36:02.192 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:36:02.192 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.192 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.192 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:36:02.192 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.140857+0000", 2024-08-17T17:36:02.192 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:36:02.192 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.192 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.192 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:36:02.192 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.140858+0000", 2024-08-17T17:36:02.192 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0780000000000001e-06 2024-08-17T17:36:02.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:36:02.193 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.140867+0000", 2024-08-17T17:36:02.193 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1020000000000003e-06 2024-08-17T17:36:02.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:36:02.193 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.140881+0000", 2024-08-17T17:36:02.193 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3638000000000001e-05 2024-08-17T17:36:02.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:36:02.193 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.140891+0000", 2024-08-17T17:36:02.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7699999999999996e-06 2024-08-17T17:36:02.194 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:36:02.194 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:36:02.194 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:36:02.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.194 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.46025.0:173292 8.8 e289/272)", 2024-08-17T17:36:02.194 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:36:02.142690+0000", 2024-08-17T17:36:02.194 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0025380009999999998, 2024-08-17T17:36:02.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031539350000000001, 2024-08-17T17:36:02.194 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:36:02.194 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:36:02.194 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-08-17T17:36:02.194 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:36:02.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.195 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:36:02.195 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.142690+0000", 2024-08-17T17:36:02.195 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:36:02.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.195 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:36:02.195 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.142690+0000", 2024-08-17T17:36:02.195 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:36:02.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.195 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:36:02.195 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.142691+0000", 2024-08-17T17:36:02.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.006e-06 2024-08-17T17:36:02.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.196 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:36:02.196 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.142696+0000", 2024-08-17T17:36:02.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1939999999999997e-06 2024-08-17T17:36:02.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.196 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:36:02.196 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.142715+0000", 2024-08-17T17:36:02.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9211999999999998e-05 2024-08-17T17:36:02.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.197 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:36:02.197 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.142723+0000", 2024-08-17T17:36:02.197 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2930000000000001e-06 2024-08-17T17:36:02.197 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:36:02.197 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:36:02.197 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:36:02.197 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.197 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.197 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.46025.0:173296 8.b e289/269)", 2024-08-17T17:36:02.197 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:36:02.142752+0000", 2024-08-17T17:36:02.197 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0024754920000000001, 2024-08-17T17:36:02.197 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031269309999999999, 2024-08-17T17:36:02.197 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:36:02.198 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:36:02.198 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-08-17T17:36:02.198 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:36:02.198 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.198 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:36:02.198 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.142752+0000", 2024-08-17T17:36:02.198 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:36:02.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.198 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.198 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:36:02.198 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.142752+0000", 2024-08-17T17:36:02.198 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:36:02.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.199 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.199 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:36:02.199 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.142753+0000", 2024-08-17T17:36:02.199 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5600000000000005e-07 2024-08-17T17:36:02.199 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.199 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.199 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:36:02.199 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.142760+0000", 2024-08-17T17:36:02.199 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3839999999999999e-06 2024-08-17T17:36:02.199 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.199 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.199 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:36:02.199 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.142771+0000", 2024-08-17T17:36:02.200 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1124999999999999e-05 2024-08-17T17:36:02.200 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.200 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.200 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:36:02.200 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.142779+0000", 2024-08-17T17:36:02.200 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6979999999999998e-06 2024-08-17T17:36:02.200 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:36:02.200 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:36:02.200 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:36:02.200 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.200 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.200 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.46025.0:173292 8.8 e289/272)", 2024-08-17T17:36:02.200 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:36:02.143133+0000", 2024-08-17T17:36:02.200 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0020946599999999999, 2024-08-17T17:36:02.201 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027808569999999999, 2024-08-17T17:36:02.201 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:36:02.201 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:36:02.201 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-08-17T17:36:02.201 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:36:02.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.201 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:36:02.201 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.143133+0000", 2024-08-17T17:36:02.201 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:36:02.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.201 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:36:02.201 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.143133+0000", 2024-08-17T17:36:02.202 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:36:02.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.202 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.202 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:36:02.202 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.143134+0000", 2024-08-17T17:36:02.202 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8400000000000004e-07 2024-08-17T17:36:02.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.202 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.202 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:36:02.202 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.143141+0000", 2024-08-17T17:36:02.202 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5259999999999996e-06 2024-08-17T17:36:02.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.202 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.202 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:36:02.203 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.143154+0000", 2024-08-17T17:36:02.203 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3086e-05 2024-08-17T17:36:02.203 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:36:02.203 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:36:02.203 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:36:02.203 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:02.143161+0000", 2024-08-17T17:36:02.203 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0890000000000002e-06 2024-08-17T17:36:02.203 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:36:02.203 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:36:02.203 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:36:02.203 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:36:02.203 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-08-17T17:36:02.203 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 12 2024-08-17T17:36:02.203 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:36:02.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:02.204 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:02.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:02.305 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:02.343 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:02.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:02.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:02.445 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:02.483 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:02.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:02.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:02.586 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:02.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:02.624 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:02.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:02.725 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:02.751 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:02.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:02.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:02.853 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:02.878 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:02.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:02.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:02.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:03.015 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:03.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:03.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:03.117 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:03.152 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:03.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:03.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:03.253 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:03.279 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:03.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:03.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:03.381 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:03.408 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:03.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:03.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:03.510 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:03.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:03 smithi003 ceph-mon[103344]: pgmap v904: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 570 op/s; 20725/131961 objects degraded (15.705%); 75 KiB/s, 19 objects/s recovering 2024-08-17T17:36:03.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:03 smithi003 ceph-mon[100615]: pgmap v904: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 570 op/s; 20725/131961 objects degraded (15.705%); 75 KiB/s, 19 objects/s recovering 2024-08-17T17:36:03.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:03.556 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:03.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:03.657 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:03.698 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:03.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:03.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:03.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:03 smithi053 ceph-mon[89963]: pgmap v904: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 570 op/s; 20725/131961 objects degraded (15.705%); 75 KiB/s, 19 objects/s recovering 2024-08-17T17:36:03.802 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:03.852 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-17T17:36:03.855 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:03.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:03.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:03.957 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:04.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:04.020 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:04.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:04.121 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:04.144 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:04.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:04.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:04.246 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:04.273 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:04.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:04.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:04.375 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:04.400 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:04.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:04.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:04.502 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:04.550 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:04.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:04.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:04.652 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:04.677 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:04.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:04.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:04.779 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:04.805 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:04.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:04.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:04.907 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:04.930 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:04.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:04.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:05.032 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:05.063 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:05.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:05.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:05.165 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:05.308 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:36:05.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:05.405 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:05.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:05.453 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:05 smithi003 ceph-mon[103344]: pgmap v905: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 837 op/s; 20604/122172 objects degraded (16.865%); 75 KiB/s, 29 objects/s recovering 2024-08-17T17:36:05.453 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:05 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 20604/122172 objects degraded (16.865%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:05.453 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:05 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/402033640' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:36:05.453 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:05 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1039219089' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:36:05.454 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:05 smithi003 ceph-mon[100615]: pgmap v905: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 837 op/s; 20604/122172 objects degraded (16.865%); 75 KiB/s, 29 objects/s recovering 2024-08-17T17:36:05.454 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:05 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 20604/122172 objects degraded (16.865%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:05.454 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:05 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/402033640' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:36:05.454 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:05 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1039219089' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:36:05.506 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:05.543 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:05.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:05.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:05.645 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:05.677 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:05.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:05.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:05.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:05 smithi053 ceph-mon[89963]: pgmap v905: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 837 op/s; 20604/122172 objects degraded (16.865%); 75 KiB/s, 29 objects/s recovering 2024-08-17T17:36:05.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:05 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 20604/122172 objects degraded (16.865%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:05.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:05 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/402033640' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:36:05.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:05 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1039219089' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:36:05.768 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:36:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:36:05] "GET /metrics HTTP/1.1" 200 36519 "" "Prometheus/2.51.0" 2024-08-17T17:36:05.777 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-08-17T17:36:05.779 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:05.780 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.2 to 1 (10000) 2024-08-17T17:36:05.802 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:05.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:05.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:05.904 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:05.930 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:05.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:05.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:06.032 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:06.065 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:06.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:06.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:06.167 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:06.208 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:06.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:06.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:06.310 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:06.358 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:06.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:06.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:06.460 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:06.497 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:06.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:06.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:06.599 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:06.633 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:06.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:06.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:06.735 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:06.773 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:06.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:06.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:06.784 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:06 smithi003 ceph-mon[103344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:36:06.784 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:06 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/402033640' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:36:06.785 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:06 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1039219089' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-17T17:36:06.785 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:06 smithi003 ceph-mon[103344]: osdmap e290: 8 total, 8 up, 8 in 2024-08-17T17:36:06.785 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:06 smithi003 ceph-mon[100615]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:36:06.876 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:06.914 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:06.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:06.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:06.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:06 smithi053 ceph-mon[89963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:36:06.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:06 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/402033640' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:36:06.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:06 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1039219089' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-17T17:36:06.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:06 smithi053 ceph-mon[89963]: osdmap e290: 8 total, 8 up, 8 in 2024-08-17T17:36:07.016 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:07.071 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:07.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:07.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:07.173 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:07.222 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:07.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:07.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:07.262 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:06 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/402033640' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:36:07.262 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:06 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1039219089' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-17T17:36:07.262 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:06 smithi003 ceph-mon[100615]: osdmap e290: 8 total, 8 up, 8 in 2024-08-17T17:36:07.309 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:36:07.325 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:07.353 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:07.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:07.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:07.455 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:07.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:07.512 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:07.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:07.611 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 3 1 2024-08-17T17:36:07.613 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:07.641 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:07.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:07.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:07.743 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:07.800 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:07.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:07.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:07.901 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:07.930 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:07.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:07.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:08.015 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:07 smithi003 ceph-mon[100615]: pgmap v907: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 710 op/s; 20604/120849 objects degraded (17.049%); 0 B/s, 22 objects/s recovering 2024-08-17T17:36:08.015 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:07 smithi003 ceph-mon[103344]: pgmap v907: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 710 op/s; 20604/120849 objects degraded (17.049%); 0 B/s, 22 objects/s recovering 2024-08-17T17:36:08.031 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:08.056 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:08.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:08.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:08.158 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:08.184 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:08.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:08.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:08.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:07 smithi053 ceph-mon[89963]: pgmap v907: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 710 op/s; 20604/120849 objects degraded (17.049%); 0 B/s, 22 objects/s recovering 2024-08-17T17:36:08.286 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:08.328 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:36:08.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:08.345 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:08.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:08.438 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:36:08.446 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:08.506 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:08.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:08.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:08.608 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:08.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:08.655 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:08.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:08.756 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:08.809 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:08.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:08.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:08.911 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:08.915 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:36:08.916 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:36:08.958 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:08.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:08.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:08.986 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:36:09.060 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:09.082 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:36:09.082 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-08-17T17:36:09.082 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-08-17T17:36:09.082 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-08-17T17:36:09.082 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:36:09.090 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:09.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:09.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:09.192 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:09.216 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:09.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:36:09.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:09.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:09.318 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:09.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:09.345 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:09.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:09.445 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:09.469 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:09.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:09.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:09.570 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:09.597 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:09.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:09.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:09.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:09.742 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:09.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:09.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:09.844 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:09.880 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-17T17:36:09.903 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:09.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:09.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:10.007 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:10.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:10.059 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:10.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:10.062 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:09 smithi003 ceph-mon[100615]: pgmap v908: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 710 op/s; 20604/120849 objects degraded (17.049%); 0 B/s, 22 objects/s recovering 2024-08-17T17:36:10.063 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:09 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 20604/120849 objects degraded (17.049%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:10.063 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:09 smithi003 ceph-mon[103344]: pgmap v908: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 710 op/s; 20604/120849 objects degraded (17.049%); 0 B/s, 22 objects/s recovering 2024-08-17T17:36:10.063 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:09 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 20604/120849 objects degraded (17.049%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:10.161 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:10.213 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:09 smithi053 ceph-mon[89963]: pgmap v908: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 710 op/s; 20604/120849 objects degraded (17.049%); 0 B/s, 22 objects/s recovering 2024-08-17T17:36:10.213 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:09 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 20604/120849 objects degraded (17.049%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:10.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:10.219 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:10.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:10.320 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:10.379 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:10.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:10.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:10.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:36:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:36:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:36:10.481 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:10.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:10.513 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:10.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:10.614 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:10.646 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:10.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:10.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:10.748 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:10.775 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:10.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:10.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:10.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:10.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:10.921 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:10.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:11.022 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:11.049 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:36:11.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:11.077 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:11.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:11.178 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:11.205 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:11.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:11.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:11.307 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:11.353 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:11.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:11.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:11.455 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:11.481 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:11.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:11.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:11.584 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:11.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:11.654 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:11.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:11.755 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:11.810 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:11.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:11.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:11.912 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:11.937 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:11.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:11.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:12.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:11 smithi003 ceph-mon[100615]: pgmap v909: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 704 op/s; 20477/112065 objects degraded (18.272%); 0 B/s, 24 objects/s recovering 2024-08-17T17:36:12.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:11 smithi003 ceph-mon[103344]: pgmap v909: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 704 op/s; 20477/112065 objects degraded (18.272%); 0 B/s, 24 objects/s recovering 2024-08-17T17:36:12.039 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:12.072 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:12.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:12.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:12.074 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:36:12.175 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:12.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:11 smithi053 ceph-mon[89963]: pgmap v909: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 704 op/s; 20477/112065 objects degraded (18.272%); 0 B/s, 24 objects/s recovering 2024-08-17T17:36:12.224 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:12.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:12.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:12.326 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:12.381 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:12.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:12.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:12.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:12.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:12.527 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:12.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:12.628 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:12.673 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:12.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:12.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:12.774 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:12.818 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:12.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:12.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:12.919 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:12.958 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:12.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:12.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:13.060 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:13.102 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:13.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:13.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:13.203 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:13.234 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:13.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:13.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:13.336 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:13.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:13.378 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:13.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:13.453 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:36:13.454 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:36:13.479 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:13.502 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:36:13.502 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:36:13.510 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:13.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:13.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:13.537 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:36:13.538 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:36:13.582 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:36:13.583 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:36:13.612 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:36:13.612 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:36:13.613 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:13.638 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:13.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:13.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:13.646 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:36:13.646 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:36:13.678 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:36:13.679 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:36:13.713 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:36:13.713 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:36:13.739 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:13.765 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:13.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:13.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:13.867 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:13.889 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:13.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:13.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:13.991 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:14.021 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:14.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:14.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:14.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:13 smithi003 ceph-mon[100615]: pgmap v910: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 706 op/s; 20477/110757 objects degraded (18.488%); 0 B/s, 24 objects/s recovering 2024-08-17T17:36:14.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:13 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/245704741' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:36:14.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:13 smithi003 ceph-mon[103344]: pgmap v910: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 706 op/s; 20477/110757 objects degraded (18.488%); 0 B/s, 24 objects/s recovering 2024-08-17T17:36:14.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:13 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/245704741' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:36:14.123 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:14.158 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:14.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:14.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:14.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:13 smithi053 ceph-mon[89963]: pgmap v910: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 706 op/s; 20477/110757 objects degraded (18.488%); 0 B/s, 24 objects/s recovering 2024-08-17T17:36:14.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:13 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/245704741' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:36:14.260 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:14.289 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:14.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:14.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:14.391 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:14.434 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:14.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:14.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:14.536 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:14.595 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:14.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:14.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:14.697 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:14.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:14.737 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:14.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:14.838 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:14.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:14.869 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:14.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:14.970 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:15.008 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:15.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:15.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:15.110 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:15.143 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:15.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:15.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:15.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:14 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 20477/110757 objects degraded (18.488%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:15.246 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:15.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:14 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 20477/110757 objects degraded (18.488%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:15.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:14 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 20477/110757 objects degraded (18.488%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:15.291 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:15.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:15.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:15.393 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:15.421 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:15.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:15.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:15.522 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:15.548 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:15.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:15.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:15.650 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:15.675 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:15.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:15.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:15.777 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:15.803 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:15.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:15.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:15.905 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:15.931 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:15.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:15.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:15.983 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:15 smithi003 ceph-mon[103344]: pgmap v911: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 695 op/s; 20359/101301 objects degraded (20.098%); 0 B/s, 24 objects/s recovering 2024-08-17T17:36:15.983 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:15 smithi003 ceph-mon[100615]: pgmap v911: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 695 op/s; 20359/101301 objects degraded (20.098%); 0 B/s, 24 objects/s recovering 2024-08-17T17:36:15.984 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:36:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:36:15] "GET /metrics HTTP/1.1" 200 36510 "" "Prometheus/2.51.0" 2024-08-17T17:36:16.033 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:16.058 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:16.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:16.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:16.160 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:16.187 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:16.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:16.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:16.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:15 smithi053 ceph-mon[89963]: pgmap v911: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 695 op/s; 20359/101301 objects degraded (20.098%); 0 B/s, 24 objects/s recovering 2024-08-17T17:36:16.290 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:16.312 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:16.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:16.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:16.414 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:16.449 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:16.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:16.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:16.551 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:16.592 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:16.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:16.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:16.694 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:16.738 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:16.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:16.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:16.840 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:16.846 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.3 to 1 (10000) 2024-08-17T17:36:16.872 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:16.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:16.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:16.974 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:17.027 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:17.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:17.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:17.129 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:17.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:17.186 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:17.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:17.214 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:16 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3365093015' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:36:17.215 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:16 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:36:17.216 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:16 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3365093015' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:36:17.216 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:16 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:36:17.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:16 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3365093015' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:36:17.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:16 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:36:17.288 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:17.320 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:17.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:17.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:17.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:17.470 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:17.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:17.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:17.570 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 5 1 2024-08-17T17:36:17.572 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:17.624 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:17.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:17.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:17.726 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:17.764 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:17.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:17.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:17.866 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:17.898 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:17.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:17.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:18.000 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:18.040 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:18.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:18.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:18.142 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:18.187 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:18.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:18.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:18.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:17 smithi053 ceph-mon[89963]: pgmap v912: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.0 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 669 op/s; 20359/100005 objects degraded (20.358%); 0 B/s, 23 objects/s recovering 2024-08-17T17:36:18.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:17 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-17T17:36:18.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:17 smithi053 ceph-mon[89963]: osdmap e291: 8 total, 8 up, 8 in 2024-08-17T17:36:18.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:17 smithi003 ceph-mon[103344]: pgmap v912: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.0 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 669 op/s; 20359/100005 objects degraded (20.358%); 0 B/s, 23 objects/s recovering 2024-08-17T17:36:18.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:17 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-17T17:36:18.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:17 smithi003 ceph-mon[103344]: osdmap e291: 8 total, 8 up, 8 in 2024-08-17T17:36:18.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:17 smithi003 ceph-mon[100615]: pgmap v912: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.0 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 669 op/s; 20359/100005 objects degraded (20.358%); 0 B/s, 23 objects/s recovering 2024-08-17T17:36:18.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:17 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-17T17:36:18.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:17 smithi003 ceph-mon[100615]: osdmap e291: 8 total, 8 up, 8 in 2024-08-17T17:36:18.289 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:18.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:18.326 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:18.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:18.427 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:18.465 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:18.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:18.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:18.567 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:18.599 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:18.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:18.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:18.700 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:18.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:18.749 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:18.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:18.850 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:18.891 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:18.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:18.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:18.993 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:19.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:19.027 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:19.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:19.128 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:19.152 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:19.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:19.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:19.254 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:19.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:19.281 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:19.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:19.382 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:19.432 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:19.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:19.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:19.534 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:19.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:19.563 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:19.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:19.664 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:19.708 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:19.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:19.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:19.810 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:19.838 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:19.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:19.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:19.876 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-08-17T17:36:19.939 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:19.965 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:19.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:19.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:20.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:19 smithi003 ceph-mon[103344]: pgmap v914: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.0 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 694 op/s; 20359/100005 objects degraded (20.358%); 0 B/s, 24 objects/s recovering 2024-08-17T17:36:20.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:19 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1783057479' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:36:20.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:19 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 20359/100005 objects degraded (20.358%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:20.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:19 smithi003 ceph-mon[100615]: pgmap v914: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.0 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 694 op/s; 20359/100005 objects degraded (20.358%); 0 B/s, 24 objects/s recovering 2024-08-17T17:36:20.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:19 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1783057479' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:36:20.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:19 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 20359/100005 objects degraded (20.358%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:20.067 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:20.111 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:20.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:20.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:20.213 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:20.214 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:19 smithi053 ceph-mon[89963]: pgmap v914: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 2.0 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 694 op/s; 20359/100005 objects degraded (20.358%); 0 B/s, 24 objects/s recovering 2024-08-17T17:36:20.214 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:19 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1783057479' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:36:20.214 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:19 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 20359/100005 objects degraded (20.358%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:20.256 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:36:20.256 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:36:20.271 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:20.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:20.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:20.374 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:20.376 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:36:20.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:20.419 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:20.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:20.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:36:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:36:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:36:20.520 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:20.543 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:20.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:20.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:20.645 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:20.670 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:20.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:20.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:20.772 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:20.798 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:20.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:20.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:20.900 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:20.937 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:20.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:20.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:21.039 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:21.058 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-17T17:36:21.065 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:21.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:21.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:21.167 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:21.201 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:21.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:21.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:21.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:20 smithi053 ceph-mon[89963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:36:21.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:20 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1783057479' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:36:21.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:20 smithi053 ceph-mon[89963]: osdmap e292: 8 total, 8 up, 8 in 2024-08-17T17:36:21.225 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:20 smithi003 ceph-mon[100615]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:36:21.225 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:20 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1783057479' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:36:21.225 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:20 smithi003 ceph-mon[100615]: osdmap e292: 8 total, 8 up, 8 in 2024-08-17T17:36:21.225 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:20 smithi003 ceph-mon[103344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:36:21.225 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:20 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1783057479' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:36:21.226 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:20 smithi003 ceph-mon[103344]: osdmap e292: 8 total, 8 up, 8 in 2024-08-17T17:36:21.303 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:21.335 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:21.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:21.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:21.438 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:21.461 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:36:21.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:21.475 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:21.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:21.576 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:21.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:21.606 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:21.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:21.689 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:36:21.708 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:21.737 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:21.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:21.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:21.839 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:21.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:21.867 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:21.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:21.968 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:21.991 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:21.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:21.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:22.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:21 smithi003 ceph-mon[103344]: pgmap v916: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 828 op/s; 20230/90870 objects degraded (22.263%); 0 B/s, 30 objects/s recovering 2024-08-17T17:36:22.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:21 smithi003 ceph-mon[100615]: pgmap v916: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 828 op/s; 20230/90870 objects degraded (22.263%); 0 B/s, 30 objects/s recovering 2024-08-17T17:36:22.094 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:22.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:22.119 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:22.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:22.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:21 smithi053 ceph-mon[89963]: pgmap v916: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 828 op/s; 20230/90870 objects degraded (22.263%); 0 B/s, 30 objects/s recovering 2024-08-17T17:36:22.221 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:22.253 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:22.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:22.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:22.355 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:22.389 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:22.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:22.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:22.490 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:22.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:22.526 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:22.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:22.627 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:22.662 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:22.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:22.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:22.764 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:22.791 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:22.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:22.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:22.892 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:22.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:22.938 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:22.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:23.039 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:23.068 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:23.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:23.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:23.170 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:23.195 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:23.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:23.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:23.297 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:23.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:23.328 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:23.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:23.429 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:23.466 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:23.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:23.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:23.497 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:36:23.497 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:36:23.568 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:23.612 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:23.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:23.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:23.714 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:23.726 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:36:23.759 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:23.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:23.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:23.862 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:23.899 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:23.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:23.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:24.001 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:24.030 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:24.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:24.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:24.132 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:24.168 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:24.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:24.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:24.187 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:23 smithi053 ceph-mon[89963]: pgmap v917: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 482 op/s; 20230/89709 objects degraded (22.551%); 0 B/s, 16 objects/s recovering 2024-08-17T17:36:24.188 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:36:24.188 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:36:24.222 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-17T17:36:24.270 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:24.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:23 smithi003 ceph-mon[100615]: pgmap v917: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 482 op/s; 20230/89709 objects degraded (22.551%); 0 B/s, 16 objects/s recovering 2024-08-17T17:36:24.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:36:24.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:36:24.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:23 smithi003 ceph-mon[103344]: pgmap v917: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 482 op/s; 20230/89709 objects degraded (22.551%); 0 B/s, 16 objects/s recovering 2024-08-17T17:36:24.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:36:24.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:36:24.313 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:24.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:24.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:24.415 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:24.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:24.455 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:24.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:24.557 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:24.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:24.603 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:24.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:24.647 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:36:24.704 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:24.744 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:24.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:24.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:24.847 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:24.877 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:24.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:24.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:24.978 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:25.021 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:25.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:25.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:25.050 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:24 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 20230/89709 objects degraded (22.551%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:25.123 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:25.155 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:25.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:25.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:25.174 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:24 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 20230/89709 objects degraded (22.551%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:25.174 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:24 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 20230/89709 objects degraded (22.551%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:25.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:25.307 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:25.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:25.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:25.409 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:25.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:25.445 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:25.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:25.547 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:25.595 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:25.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:25.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:25.697 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:25.730 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:25.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:25.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:25.832 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:25.857 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:25.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:25.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:25.959 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:25.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:25.991 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:25.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:26.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:25 smithi003 ceph-mon[103344]: pgmap v918: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.6 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 798 op/s; 20105/80838 objects degraded (24.871%); 0 B/s, 31 objects/s recovering 2024-08-17T17:36:26.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:25 smithi003 ceph-mon[100615]: pgmap v918: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.6 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 798 op/s; 20105/80838 objects degraded (24.871%); 0 B/s, 31 objects/s recovering 2024-08-17T17:36:26.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:36:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:36:25] "GET /metrics HTTP/1.1" 200 36506 "" "Prometheus/2.51.0" 2024-08-17T17:36:26.092 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:26.116 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:26.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:26.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:26.172 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:36:26.179 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:25 smithi053 ceph-mon[89963]: pgmap v918: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.6 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 798 op/s; 20105/80838 objects degraded (24.871%); 0 B/s, 31 objects/s recovering 2024-08-17T17:36:26.218 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:26.242 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:26.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:26.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:26.344 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:26.391 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:26.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:26.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:26.493 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:26.535 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:26.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:26.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:26.630 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:36:26.630 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:36:26.637 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:26.668 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:26.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:26.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:26.770 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:26.797 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:26.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:26.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:26.899 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:26.937 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:26.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:26.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:27.039 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:27.064 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:27.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:27.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:27.166 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:27.206 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:27.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:27.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:27.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:26 smithi053 ceph-mon[89963]: pgmap v919: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.6 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 730 op/s; 20105/79596 objects degraded (25.259%); 0 B/s, 27 objects/s recovering 2024-08-17T17:36:27.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:26 smithi003 ceph-mon[100615]: pgmap v919: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.6 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 730 op/s; 20105/79596 objects degraded (25.259%); 0 B/s, 27 objects/s recovering 2024-08-17T17:36:27.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:26 smithi003 ceph-mon[103344]: pgmap v919: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.6 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 730 op/s; 20105/79596 objects degraded (25.259%); 0 B/s, 27 objects/s recovering 2024-08-17T17:36:27.309 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:27.348 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:27.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:27.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:27.450 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:27.469 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:36:27.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:27.492 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:27.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:27.593 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:27.652 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:27.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:27.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:27.754 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:27.799 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:27.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:27.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:27.906 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:27.933 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:27.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:27.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:28.035 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:28.090 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:28.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:28.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:28.192 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:28.226 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:28.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:28.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:28.276 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-17T17:36:28.328 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:28.357 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:28.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:28.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:28.459 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:28.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:28.519 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:28.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:28.620 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:28.645 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:36:28.651 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:28.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:28.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:28.753 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:28.778 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:28.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:28.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:28.880 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:28.906 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:28.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:28.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:29.008 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:29.039 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:29.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:29.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:29.050 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:36:29.050 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:36:29.097 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:36:29.097 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:36:29.134 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:36:29.134 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:36:29.142 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:29.175 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:36:29.175 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:36:29.183 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:29.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:29.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:29.209 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:36:29.209 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:36:29.223 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.5 to 1 (10000) 2024-08-17T17:36:29.260 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:36:29.261 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:36:29.285 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:29.305 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:36:29.305 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:36:29.333 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:29.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:29.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:29.352 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:36:29.353 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:36:29.435 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:29.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:29 smithi053 ceph-mon[89963]: pgmap v920: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.6 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 680 op/s; 20105/79596 objects degraded (25.259%); 0 B/s, 25 objects/s recovering 2024-08-17T17:36:29.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:29 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/704025907' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:36:29.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:29 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2476029479' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:36:29.468 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:29 smithi003 ceph-mon[100615]: pgmap v920: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.6 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 680 op/s; 20105/79596 objects degraded (25.259%); 0 B/s, 25 objects/s recovering 2024-08-17T17:36:29.468 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:29 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/704025907' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:36:29.468 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:29 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2476029479' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:36:29.468 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:29 smithi003 ceph-mon[103344]: pgmap v920: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.6 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 680 op/s; 20105/79596 objects degraded (25.259%); 0 B/s, 25 objects/s recovering 2024-08-17T17:36:29.468 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:29 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/704025907' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:36:29.468 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:29 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2476029479' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:36:29.478 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:29.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:29.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:29.580 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:29.613 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:29.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:29.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:29.715 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:29.745 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:29.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:29.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:29.847 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:29.882 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:29.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:29.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:29.984 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:30.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:30.026 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:30.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:30.127 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:30.157 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:30.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:30.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:30.259 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:30.298 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:30.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:30.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:30.399 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:30.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:30.433 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:30.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:30.440 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:36:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:36:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:36:30.440 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:30 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/704025907' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-17T17:36:30.440 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:30 smithi053 ceph-mon[89963]: osdmap e293: 8 total, 8 up, 8 in 2024-08-17T17:36:30.441 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:30 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 20105/79596 objects degraded (25.259%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:30.483 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:36:30.484 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2024-08-17T17:36:30.484 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2024-08-17T17:36:30.484 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2024-08-17T17:36:30.484 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.484 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:17827 8.2 8:4b5aa46c:::benchmark_data_smithi003_229103_object17826:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-08-17T17:36:30.484 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:01.102709+0000", 2024-08-17T17:36:30.484 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 209.37818711200001, 2024-08-17T17:36:30.484 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.10004417, 2024-08-17T17:36:30.484 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:30.484 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:36:30.484 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:36:30.484 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:36:30.484 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:30.485 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 17827 2024-08-17T17:36:30.485 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.485 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:30.485 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.485 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:30.485 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:01.102709+0000", 2024-08-17T17:36:30.485 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.485 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.485 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.485 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:30.485 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:01.102709+0000", 2024-08-17T17:36:30.485 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.485 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.486 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.486 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:30.486 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:01.102708+0000", 2024-08-17T17:36:30.486 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:36:30.486 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.486 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.486 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:30.486 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:01.102785+0000", 2024-08-17T17:36:30.486 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7650000000000004e-05 2024-08-17T17:36:30.486 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.486 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.486 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:30.486 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:01.102801+0000", 2024-08-17T17:36:30.486 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5270000000000001e-05 2024-08-17T17:36:30.487 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.487 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.487 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:30.487 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:01.102807+0000", 2024-08-17T17:36:30.487 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1800000000000001e-06 2024-08-17T17:36:30.487 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.487 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.487 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:30.487 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:01.102853+0000", 2024-08-17T17:36:30.487 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6476e-05 2024-08-17T17:36:30.487 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.487 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.487 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:36:30.487 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:01.103040+0000", 2024-08-17T17:36:30.488 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00018695100000000001 2024-08-17T17:36:30.488 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.488 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.488 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:36:30.488 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:01.103183+0000", 2024-08-17T17:36:30.488 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000143094 2024-08-17T17:36:30.488 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.488 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.488 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:36:30.488 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:01.104339+0000", 2024-08-17T17:36:30.488 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001155702 2024-08-17T17:36:30.488 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.488 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.488 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.489 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:01.104806+0000", 2024-08-17T17:36:30.489 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00046730799999999999 2024-08-17T17:36:30.489 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.489 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.489 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.489 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:01.202658+0000", 2024-08-17T17:36:30.489 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.097852013000000002 2024-08-17T17:36:30.489 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.489 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.489 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:36:30.489 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:01.202712+0000", 2024-08-17T17:36:30.489 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3328000000000002e-05 2024-08-17T17:36:30.489 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.489 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.489 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:36:30.490 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:01.202753+0000", 2024-08-17T17:36:30.490 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1219e-05 2024-08-17T17:36:30.490 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.490 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:30.490 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.490 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.490 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.490 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:24716 8.2 8:48d30eab:::benchmark_data_smithi003_229103_object24715:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-08-17T17:36:30.490 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:03.748727+0000", 2024-08-17T17:36:30.490 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 206.73216938600001, 2024-08-17T17:36:30.490 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.15246537299999999, 2024-08-17T17:36:30.490 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:30.490 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:36:30.490 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:36:30.491 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:36:30.491 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:30.491 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 24716 2024-08-17T17:36:30.491 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.491 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:30.491 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.491 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:30.491 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.748727+0000", 2024-08-17T17:36:30.491 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.491 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.491 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.491 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:30.491 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.748727+0000", 2024-08-17T17:36:30.491 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.492 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.492 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.492 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:30.492 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.748707+0000", 2024-08-17T17:36:30.492 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999804 2024-08-17T17:36:30.492 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.492 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.492 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:30.492 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.748819+0000", 2024-08-17T17:36:30.492 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000112417 2024-08-17T17:36:30.492 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.492 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.492 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:30.492 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.748831+0000", 2024-08-17T17:36:30.493 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1411e-05 2024-08-17T17:36:30.493 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.493 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.493 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:30.493 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.748837+0000", 2024-08-17T17:36:30.493 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1650000000000003e-06 2024-08-17T17:36:30.493 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.493 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.493 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:30.493 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.748883+0000", 2024-08-17T17:36:30.493 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5904e-05 2024-08-17T17:36:30.493 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.493 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.493 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:36:30.493 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.749118+0000", 2024-08-17T17:36:30.494 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00023476500000000001 2024-08-17T17:36:30.494 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.494 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.494 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:36:30.494 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.749261+0000", 2024-08-17T17:36:30.494 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014338299999999999 2024-08-17T17:36:30.494 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.494 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.494 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:36:30.494 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.751689+0000", 2024-08-17T17:36:30.494 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0024283690000000001 2024-08-17T17:36:30.494 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.494 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.494 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.495 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.755002+0000", 2024-08-17T17:36:30.495 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0033125759999999998 2024-08-17T17:36:30.495 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.495 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.495 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.495 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.901091+0000", 2024-08-17T17:36:30.495 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.146088676 2024-08-17T17:36:30.495 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.495 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.495 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:36:30.495 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.901150+0000", 2024-08-17T17:36:30.495 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9098999999999998e-05 2024-08-17T17:36:30.495 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.496 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.496 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:36:30.496 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.901192+0000", 2024-08-17T17:36:30.496 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2212000000000002e-05 2024-08-17T17:36:30.496 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.496 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:30.496 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.496 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.496 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.496 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:24721 8.2 8:42725ccf:::benchmark_data_smithi003_229103_object24720:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-08-17T17:36:30.496 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:03.749342+0000", 2024-08-17T17:36:30.496 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 206.731553561, 2024-08-17T17:36:30.496 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.151983322, 2024-08-17T17:36:30.496 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:30.496 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:36:30.497 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:36:30.497 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:36:30.497 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:30.497 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 24721 2024-08-17T17:36:30.498 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.498 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:30.498 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.498 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:30.498 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.749342+0000", 2024-08-17T17:36:30.498 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.498 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.498 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.498 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:30.498 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.749342+0000", 2024-08-17T17:36:30.498 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.498 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.498 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.499 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:30.499 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.749341+0000", 2024-08-17T17:36:30.499 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:36:30.499 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.499 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.499 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:30.499 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.749372+0000", 2024-08-17T17:36:30.499 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0431000000000001e-05 2024-08-17T17:36:30.499 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.499 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.499 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:30.499 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.749383+0000", 2024-08-17T17:36:30.499 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0606e-05 2024-08-17T17:36:30.500 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.500 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.500 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:30.500 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.749388+0000", 2024-08-17T17:36:30.500 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.75e-06 2024-08-17T17:36:30.500 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.500 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.500 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:30.500 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.750461+0000", 2024-08-17T17:36:30.500 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0010728910000000001 2024-08-17T17:36:30.500 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.500 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.500 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:36:30.500 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.750571+0000", 2024-08-17T17:36:30.501 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010960500000000001 2024-08-17T17:36:30.501 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.501 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.501 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:36:30.501 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.750725+0000", 2024-08-17T17:36:30.501 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000153843 2024-08-17T17:36:30.501 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.501 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.501 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:36:30.501 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.753106+0000", 2024-08-17T17:36:30.501 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0023815120000000001 2024-08-17T17:36:30.501 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.501 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.501 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.502 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.755039+0000", 2024-08-17T17:36:30.502 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001932556 2024-08-17T17:36:30.502 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.502 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.502 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.502 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.901240+0000", 2024-08-17T17:36:30.502 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.14620155800000001 2024-08-17T17:36:30.502 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.502 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.502 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:36:30.502 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.901292+0000", 2024-08-17T17:36:30.502 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1804000000000001e-05 2024-08-17T17:36:30.502 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.502 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.503 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:36:30.503 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.901326+0000", 2024-08-17T17:36:30.503 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3634999999999999e-05 2024-08-17T17:36:30.503 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.503 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:30.503 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.503 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.503 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.503 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:24723 8.2 8:421a075d:::benchmark_data_smithi003_229103_object24722:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-08-17T17:36:30.503 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:03.749395+0000", 2024-08-17T17:36:30.503 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 206.73150055100001, 2024-08-17T17:36:30.503 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.15213708100000001, 2024-08-17T17:36:30.503 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:30.503 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:36:30.504 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:36:30.504 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:36:30.504 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:30.504 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 24723 2024-08-17T17:36:30.504 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.504 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:30.504 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.504 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:30.504 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.749395+0000", 2024-08-17T17:36:30.504 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.504 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.504 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.504 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:30.504 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.749395+0000", 2024-08-17T17:36:30.504 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.505 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.505 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.505 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:30.505 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.749395+0000", 2024-08-17T17:36:30.505 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:36:30.505 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.505 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.505 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:30.505 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.749425+0000", 2024-08-17T17:36:30.505 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0185e-05 2024-08-17T17:36:30.505 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.505 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.505 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:30.505 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.749433+0000", 2024-08-17T17:36:30.506 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1049999999999995e-06 2024-08-17T17:36:30.506 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.506 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.506 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:30.506 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.749438+0000", 2024-08-17T17:36:30.506 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5480000000000003e-06 2024-08-17T17:36:30.506 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.506 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.506 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:30.506 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.751950+0000", 2024-08-17T17:36:30.506 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002512298 2024-08-17T17:36:30.506 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.506 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.506 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:36:30.507 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.752061+0000", 2024-08-17T17:36:30.507 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000111057 2024-08-17T17:36:30.507 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.507 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.507 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:36:30.507 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.752182+0000", 2024-08-17T17:36:30.507 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000121279 2024-08-17T17:36:30.507 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.507 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.507 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:36:30.507 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.753507+0000", 2024-08-17T17:36:30.507 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0013248750000000001 2024-08-17T17:36:30.507 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.508 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.508 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.508 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.755068+0000", 2024-08-17T17:36:30.508 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0015612410000000001 2024-08-17T17:36:30.508 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.508 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.508 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.508 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.901382+0000", 2024-08-17T17:36:30.508 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.14631343899999999 2024-08-17T17:36:30.508 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.508 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.508 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:36:30.508 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.901499+0000", 2024-08-17T17:36:30.509 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011757899999999999 2024-08-17T17:36:30.509 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.509 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.509 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:36:30.509 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.901532+0000", 2024-08-17T17:36:30.509 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3061000000000001e-05 2024-08-17T17:36:30.509 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.509 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:30.509 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.509 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.509 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.509 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:41874 8.2 8:4a3ae660:::benchmark_data_smithi003_229103_object41873:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-08-17T17:36:30.509 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:11.491486+0000", 2024-08-17T17:36:30.509 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 198.98940985499999, 2024-08-17T17:36:30.509 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.094758932000000004, 2024-08-17T17:36:30.510 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:30.510 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:36:30.510 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:36:30.510 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:36:30.510 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:30.510 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 41874 2024-08-17T17:36:30.510 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.510 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:30.510 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.510 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:30.510 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:11.491486+0000", 2024-08-17T17:36:30.510 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.510 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.511 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.511 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:30.511 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:11.491486+0000", 2024-08-17T17:36:30.511 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.511 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.511 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.511 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:30.511 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:11.491484+0000", 2024-08-17T17:36:30.511 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:36:30.511 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.511 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.511 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:30.512 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:11.491541+0000", 2024-08-17T17:36:30.512 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6184999999999998e-05 2024-08-17T17:36:30.512 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.512 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.512 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:30.512 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:11.491555+0000", 2024-08-17T17:36:30.512 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4153e-05 2024-08-17T17:36:30.512 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.512 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.512 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:30.512 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:11.491561+0000", 2024-08-17T17:36:30.513 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5359999999999998e-06 2024-08-17T17:36:30.513 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.513 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.513 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:30.513 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:11.491611+0000", 2024-08-17T17:36:30.513 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9393e-05 2024-08-17T17:36:30.513 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.513 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.513 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:36:30.513 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:11.491695+0000", 2024-08-17T17:36:30.513 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.4523000000000004e-05 2024-08-17T17:36:30.513 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.513 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.514 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:36:30.514 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:11.491782+0000", 2024-08-17T17:36:30.514 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.6409000000000006e-05 2024-08-17T17:36:30.514 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.514 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.514 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:36:30.514 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:11.493272+0000", 2024-08-17T17:36:30.514 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001489841 2024-08-17T17:36:30.514 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.514 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.514 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.514 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:11.493517+0000", 2024-08-17T17:36:30.514 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000245176 2024-08-17T17:36:30.515 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.515 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.515 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.515 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:11.586145+0000", 2024-08-17T17:36:30.515 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.092628586999999998 2024-08-17T17:36:30.515 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.515 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.515 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:36:30.515 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:11.586207+0000", 2024-08-17T17:36:30.515 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1554000000000001e-05 2024-08-17T17:36:30.515 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.515 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.515 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:36:30.515 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:11.586245+0000", 2024-08-17T17:36:30.516 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8149999999999999e-05 2024-08-17T17:36:30.516 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.516 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:30.516 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.516 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.516 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.516 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:75561 8.4 8:2c426cbf:::benchmark_data_smithi003_229103_object75560:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-08-17T17:36:30.516 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:43.185837+0000", 2024-08-17T17:36:30.516 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 167.29505933199999, 2024-08-17T17:36:30.516 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.98011287700000005, 2024-08-17T17:36:30.516 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:30.516 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:36:30.516 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:36:30.516 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:36:30.516 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:30.517 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 75561 2024-08-17T17:36:30.517 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.517 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:30.517 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.517 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:30.517 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.185837+0000", 2024-08-17T17:36:30.517 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.517 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.517 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.517 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:30.517 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.185837+0000", 2024-08-17T17:36:30.517 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.517 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.517 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.518 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:30.518 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.185836+0000", 2024-08-17T17:36:30.518 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:36:30.518 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.518 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.518 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:30.518 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.185885+0000", 2024-08-17T17:36:30.518 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9588e-05 2024-08-17T17:36:30.518 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.518 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.518 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:30.518 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.185896+0000", 2024-08-17T17:36:30.518 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0937e-05 2024-08-17T17:36:30.519 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.519 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.519 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:30.519 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.185900+0000", 2024-08-17T17:36:30.519 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8909999999999997e-06 2024-08-17T17:36:30.519 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.519 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.519 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:30.519 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.185957+0000", 2024-08-17T17:36:30.519 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7135000000000001e-05 2024-08-17T17:36:30.519 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.519 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.519 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:36:30.519 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.185963+0000", 2024-08-17T17:36:30.520 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6250000000000004e-06 2024-08-17T17:36:30.520 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.520 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.520 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:30.520 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.150376+0000", 2024-08-17T17:36:30.520 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.96441359400000004 2024-08-17T17:36:30.520 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.520 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.520 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:36:30.520 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.151738+0000", 2024-08-17T17:36:30.520 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001362032 2024-08-17T17:36:30.520 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.520 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.520 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:36:30.521 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.151916+0000", 2024-08-17T17:36:30.521 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00017807399999999999 2024-08-17T17:36:30.521 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.521 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.521 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:36:30.521 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.158973+0000", 2024-08-17T17:36:30.521 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0070567449999999997 2024-08-17T17:36:30.521 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.521 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.521 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.521 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165494+0000", 2024-08-17T17:36:30.521 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0065203179999999998 2024-08-17T17:36:30.521 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.521 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.522 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.522 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165827+0000", 2024-08-17T17:36:30.522 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00033381300000000002 2024-08-17T17:36:30.522 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.522 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.522 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:36:30.522 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165897+0000", 2024-08-17T17:36:30.522 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9472000000000002e-05 2024-08-17T17:36:30.522 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.522 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.522 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:36:30.522 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165949+0000", 2024-08-17T17:36:30.522 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2636999999999998e-05 2024-08-17T17:36:30.523 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.523 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:30.523 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.523 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.523 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.523 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:75575 8.4 8:241d3c27:::benchmark_data_smithi003_229103_object75574:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-08-17T17:36:30.523 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:43.209432+0000", 2024-08-17T17:36:30.523 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 167.27146407800001, 2024-08-17T17:36:30.523 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.95669896099999996, 2024-08-17T17:36:30.523 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:30.523 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:36:30.523 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:36:30.523 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:36:30.523 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:30.524 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 75575 2024-08-17T17:36:30.524 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.524 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:30.524 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.524 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:30.524 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.209432+0000", 2024-08-17T17:36:30.524 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.524 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.524 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.524 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:30.524 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.209432+0000", 2024-08-17T17:36:30.524 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.524 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.524 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.525 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:30.525 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.209430+0000", 2024-08-17T17:36:30.525 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:36:30.525 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.525 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.525 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:30.525 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.209491+0000", 2024-08-17T17:36:30.525 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1189e-05 2024-08-17T17:36:30.525 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.525 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.525 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:30.525 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.209532+0000", 2024-08-17T17:36:30.525 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0274000000000001e-05 2024-08-17T17:36:30.526 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.526 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.526 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:30.526 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.209542+0000", 2024-08-17T17:36:30.526 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0556e-05 2024-08-17T17:36:30.526 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.526 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.526 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:30.526 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.209612+0000", 2024-08-17T17:36:30.526 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9669e-05 2024-08-17T17:36:30.526 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.526 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.526 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:36:30.527 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.209623+0000", 2024-08-17T17:36:30.527 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1402e-05 2024-08-17T17:36:30.527 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.527 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.527 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:30.527 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.153453+0000", 2024-08-17T17:36:30.527 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.94382987200000001 2024-08-17T17:36:30.527 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.527 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.527 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:36:30.527 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.153538+0000", 2024-08-17T17:36:30.527 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.4448000000000002e-05 2024-08-17T17:36:30.527 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.527 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.528 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:36:30.528 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.153641+0000", 2024-08-17T17:36:30.528 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010312200000000001 2024-08-17T17:36:30.528 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.528 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.528 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:36:30.528 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.164927+0000", 2024-08-17T17:36:30.528 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011286503999999999 2024-08-17T17:36:30.528 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.528 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.528 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.528 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165658+0000", 2024-08-17T17:36:30.528 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00073055899999999998 2024-08-17T17:36:30.528 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.529 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.529 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.529 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.166034+0000", 2024-08-17T17:36:30.529 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000375891 2024-08-17T17:36:30.529 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.529 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.529 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:36:30.529 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.166087+0000", 2024-08-17T17:36:30.529 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3570999999999999e-05 2024-08-17T17:36:30.529 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.529 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.529 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:36:30.529 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.166131+0000", 2024-08-17T17:36:30.529 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3636e-05 2024-08-17T17:36:30.530 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.530 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:30.530 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.530 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.530 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.530 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:75590 8.4 8:2d80d3ce:::benchmark_data_smithi003_229103_object75589:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-08-17T17:36:30.530 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:43.225574+0000", 2024-08-17T17:36:30.530 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 167.25532194100001, 2024-08-17T17:36:30.530 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.94686139400000002, 2024-08-17T17:36:30.530 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:30.530 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:36:30.530 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:36:30.530 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:36:30.531 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:30.531 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 75590 2024-08-17T17:36:30.531 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.531 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:30.531 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.531 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:30.531 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.225574+0000", 2024-08-17T17:36:30.531 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.531 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.531 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.531 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:30.531 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.225574+0000", 2024-08-17T17:36:30.531 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.531 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.532 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.532 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:30.532 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.225573+0000", 2024-08-17T17:36:30.532 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:36:30.532 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.532 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.532 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:30.532 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.225645+0000", 2024-08-17T17:36:30.532 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.1625999999999995e-05 2024-08-17T17:36:30.532 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.532 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.532 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:30.532 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.225655+0000", 2024-08-17T17:36:30.532 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.043e-05 2024-08-17T17:36:30.533 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.533 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.533 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:30.533 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.225662+0000", 2024-08-17T17:36:30.533 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4420000000000003e-06 2024-08-17T17:36:30.533 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.533 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.533 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:30.533 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.225718+0000", 2024-08-17T17:36:30.533 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6232e-05 2024-08-17T17:36:30.533 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.533 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.533 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:36:30.533 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.225727+0000", 2024-08-17T17:36:30.534 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1949999999999992e-06 2024-08-17T17:36:30.534 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.534 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.534 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:30.534 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.155232+0000", 2024-08-17T17:36:30.534 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.92950502499999998 2024-08-17T17:36:30.534 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.534 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.534 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:36:30.534 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.155349+0000", 2024-08-17T17:36:30.534 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000117036 2024-08-17T17:36:30.534 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.535 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.535 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:36:30.535 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.155535+0000", 2024-08-17T17:36:30.535 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000185374 2024-08-17T17:36:30.535 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.535 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.535 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:36:30.535 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.164936+0000", 2024-08-17T17:36:30.535 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0094012939999999993 2024-08-17T17:36:30.535 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.535 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.535 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.535 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165743+0000", 2024-08-17T17:36:30.535 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00080753499999999996 2024-08-17T17:36:30.536 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.536 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.536 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.536 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.172337+0000", 2024-08-17T17:36:30.536 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0065936959999999996 2024-08-17T17:36:30.536 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.536 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.536 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:36:30.536 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.172389+0000", 2024-08-17T17:36:30.536 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1940000000000001e-05 2024-08-17T17:36:30.537 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.537 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.537 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:36:30.537 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.172435+0000", 2024-08-17T17:36:30.537 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6322999999999997e-05 2024-08-17T17:36:30.537 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.537 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:30.537 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.537 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.537 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.537 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:75606 8.4 8:2b96e39f:::benchmark_data_smithi003_229103_object75605:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-08-17T17:36:30.537 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:43.250288+0000", 2024-08-17T17:36:30.537 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 167.23060773099999, 2024-08-17T17:36:30.538 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.925042578, 2024-08-17T17:36:30.538 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:30.538 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:36:30.538 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:36:30.538 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:36:30.538 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:30.538 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 75606 2024-08-17T17:36:30.538 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.538 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:30.538 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.538 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:30.538 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.250288+0000", 2024-08-17T17:36:30.538 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.538 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.539 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.539 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:30.539 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.250288+0000", 2024-08-17T17:36:30.539 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.539 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.539 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.539 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:30.539 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.250287+0000", 2024-08-17T17:36:30.539 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:36:30.539 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.539 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.539 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:30.540 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:30.541 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:30 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/704025907' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-17T17:36:30.541 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:30 smithi003 ceph-mon[100615]: osdmap e293: 8 total, 8 up, 8 in 2024-08-17T17:36:30.541 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:30 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 20105/79596 objects degraded (25.259%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:30.541 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:30 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/704025907' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-17T17:36:30.541 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:30 smithi003 ceph-mon[103344]: osdmap e293: 8 total, 8 up, 8 in 2024-08-17T17:36:30.542 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:30 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 20105/79596 objects degraded (25.259%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:30.542 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.250335+0000", 2024-08-17T17:36:30.543 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.8593e-05 2024-08-17T17:36:30.543 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.543 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.543 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:30.543 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.250348+0000", 2024-08-17T17:36:30.543 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3128e-05 2024-08-17T17:36:30.543 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.543 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.543 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:30.543 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.250353+0000", 2024-08-17T17:36:30.543 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6589999999999998e-06 2024-08-17T17:36:30.543 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.544 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.544 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:30.544 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.250400+0000", 2024-08-17T17:36:30.544 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7262999999999998e-05 2024-08-17T17:36:30.544 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.544 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.544 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:36:30.544 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.250407+0000", 2024-08-17T17:36:30.544 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.6880000000000002e-06 2024-08-17T17:36:30.544 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.544 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.544 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:30.544 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.157095+0000", 2024-08-17T17:36:30.544 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.90668845600000003 2024-08-17T17:36:30.545 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.545 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.545 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:36:30.545 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.157173+0000", 2024-08-17T17:36:30.545 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7803e-05 2024-08-17T17:36:30.545 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.545 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.545 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:36:30.545 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.157268+0000", 2024-08-17T17:36:30.545 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.4374000000000001e-05 2024-08-17T17:36:30.545 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.545 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.546 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:36:30.546 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.164993+0000", 2024-08-17T17:36:30.546 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0077249240000000002 2024-08-17T17:36:30.546 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.546 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.546 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.546 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165789+0000", 2024-08-17T17:36:30.546 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000796486 2024-08-17T17:36:30.546 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.546 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.546 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.546 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175203+0000", 2024-08-17T17:36:30.546 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0094138179999999991 2024-08-17T17:36:30.547 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.547 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.547 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:36:30.547 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175253+0000", 2024-08-17T17:36:30.547 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9712000000000001e-05 2024-08-17T17:36:30.547 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.547 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.547 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:36:30.547 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175331+0000", 2024-08-17T17:36:30.547 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.8245999999999996e-05 2024-08-17T17:36:30.547 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.547 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:30.547 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.547 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.548 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.548 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:75612 8.4 8:2055fef1:::benchmark_data_smithi003_229103_object75611:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-08-17T17:36:30.548 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:43.260677+0000", 2024-08-17T17:36:30.548 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 167.22021892800001, 2024-08-17T17:36:30.548 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.91479233199999999, 2024-08-17T17:36:30.548 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:30.548 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:36:30.548 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:36:30.548 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:36:30.548 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:30.548 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 75612 2024-08-17T17:36:30.548 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.548 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:30.549 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.549 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:30.549 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.260677+0000", 2024-08-17T17:36:30.549 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.549 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.549 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.549 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:30.549 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.260677+0000", 2024-08-17T17:36:30.549 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.549 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.549 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.549 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:30.549 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.260676+0000", 2024-08-17T17:36:30.549 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:36:30.550 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.550 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.550 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:30.550 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.260728+0000", 2024-08-17T17:36:30.550 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1887e-05 2024-08-17T17:36:30.550 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.550 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.550 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:30.550 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.260737+0000", 2024-08-17T17:36:30.550 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.516e-06 2024-08-17T17:36:30.550 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.550 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.550 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:30.550 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.260741+0000", 2024-08-17T17:36:30.551 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5039999999999998e-06 2024-08-17T17:36:30.551 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.551 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.551 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:30.551 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.260794+0000", 2024-08-17T17:36:30.551 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3078000000000002e-05 2024-08-17T17:36:30.551 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.551 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.551 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:36:30.551 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.260805+0000", 2024-08-17T17:36:30.551 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1459e-05 2024-08-17T17:36:30.551 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.551 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.552 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:30.552 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.159008+0000", 2024-08-17T17:36:30.552 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.89820229900000004 2024-08-17T17:36:30.552 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.552 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.552 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:36:30.552 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.159094+0000", 2024-08-17T17:36:30.552 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.6119000000000007e-05 2024-08-17T17:36:30.552 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.552 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.552 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:36:30.552 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.159193+0000", 2024-08-17T17:36:30.552 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.9285999999999998e-05 2024-08-17T17:36:30.552 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.553 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.553 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:36:30.553 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165000+0000", 2024-08-17T17:36:30.553 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.005806833 2024-08-17T17:36:30.553 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.553 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.553 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.553 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165990+0000", 2024-08-17T17:36:30.553 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000990152 2024-08-17T17:36:30.553 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.553 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.553 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.553 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175382+0000", 2024-08-17T17:36:30.553 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0093919659999999999 2024-08-17T17:36:30.554 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.554 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.554 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:36:30.554 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175431+0000", 2024-08-17T17:36:30.554 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9101999999999998e-05 2024-08-17T17:36:30.554 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.554 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.554 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:36:30.554 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175469+0000", 2024-08-17T17:36:30.554 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8189000000000003e-05 2024-08-17T17:36:30.554 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.554 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:30.554 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.555 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.555 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.555 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:75613 8.4 8:2044481c:::benchmark_data_smithi003_229103_object75612:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-08-17T17:36:30.555 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:43.262848+0000", 2024-08-17T17:36:30.555 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 167.21804782999999, 2024-08-17T17:36:30.555 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.91278752399999996, 2024-08-17T17:36:30.555 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:30.555 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:36:30.555 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:36:30.555 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:36:30.555 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:30.555 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 75613 2024-08-17T17:36:30.555 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.555 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:30.556 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.556 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:30.556 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.262848+0000", 2024-08-17T17:36:30.556 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.556 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.556 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.556 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:30.556 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.262848+0000", 2024-08-17T17:36:30.556 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.556 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.556 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.556 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:30.556 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.262847+0000", 2024-08-17T17:36:30.557 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:36:30.557 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.557 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.557 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:30.557 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.262896+0000", 2024-08-17T17:36:30.557 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9047e-05 2024-08-17T17:36:30.557 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.557 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.557 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:30.557 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.262914+0000", 2024-08-17T17:36:30.557 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7093000000000001e-05 2024-08-17T17:36:30.557 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.557 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.558 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:30.558 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.262918+0000", 2024-08-17T17:36:30.558 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.561e-06 2024-08-17T17:36:30.558 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.558 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.558 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:30.558 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.262960+0000", 2024-08-17T17:36:30.558 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1659000000000002e-05 2024-08-17T17:36:30.558 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.558 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.558 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:36:30.558 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.262967+0000", 2024-08-17T17:36:30.558 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7279999999999995e-06 2024-08-17T17:36:30.559 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.559 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.559 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:30.559 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.160829+0000", 2024-08-17T17:36:30.559 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.89786185900000004 2024-08-17T17:36:30.559 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.559 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.559 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:36:30.559 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.160952+0000", 2024-08-17T17:36:30.559 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012245499999999999 2024-08-17T17:36:30.559 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.559 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.559 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:36:30.559 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.161116+0000", 2024-08-17T17:36:30.560 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000164208 2024-08-17T17:36:30.560 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.560 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.560 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:36:30.560 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165004+0000", 2024-08-17T17:36:30.560 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0038882230000000001 2024-08-17T17:36:30.560 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.560 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.560 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.560 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.167386+0000", 2024-08-17T17:36:30.560 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0023818960000000001 2024-08-17T17:36:30.560 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.560 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.560 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.561 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175558+0000", 2024-08-17T17:36:30.561 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0081719789999999994 2024-08-17T17:36:30.561 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.561 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.561 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:36:30.561 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175602+0000", 2024-08-17T17:36:30.561 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4008999999999998e-05 2024-08-17T17:36:30.561 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.561 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.561 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:36:30.561 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175636+0000", 2024-08-17T17:36:30.561 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3494999999999997e-05 2024-08-17T17:36:30.561 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.561 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:30.562 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.562 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.562 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.562 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:75614 8.4 8:2ac96a7e:::benchmark_data_smithi003_229103_object75613:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-08-17T17:36:30.562 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:43.265251+0000", 2024-08-17T17:36:30.562 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 167.215645029, 2024-08-17T17:36:30.562 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.91049424000000001, 2024-08-17T17:36:30.562 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:30.562 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:36:30.562 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:36:30.562 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:36:30.562 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:30.562 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 75614 2024-08-17T17:36:30.563 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.563 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:30.563 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.563 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:30.563 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.265251+0000", 2024-08-17T17:36:30.563 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.563 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.563 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.563 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:30.563 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.265251+0000", 2024-08-17T17:36:30.563 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.563 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.563 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.563 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:30.563 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.265250+0000", 2024-08-17T17:36:30.564 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:36:30.564 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.564 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.564 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:30.564 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.265297+0000", 2024-08-17T17:36:30.564 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7088e-05 2024-08-17T17:36:30.564 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.564 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.564 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:30.564 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.265306+0000", 2024-08-17T17:36:30.564 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9819999999999997e-06 2024-08-17T17:36:30.564 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.564 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.565 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:30.565 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.265310+0000", 2024-08-17T17:36:30.565 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4189999999999998e-06 2024-08-17T17:36:30.565 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.565 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.565 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:30.565 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.265345+0000", 2024-08-17T17:36:30.565 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5271000000000002e-05 2024-08-17T17:36:30.565 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.565 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.565 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:36:30.565 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.265354+0000", 2024-08-17T17:36:30.565 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.5150000000000006e-06 2024-08-17T17:36:30.565 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.566 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.566 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:30.566 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.162842+0000", 2024-08-17T17:36:30.566 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.89748799899999998 2024-08-17T17:36:30.566 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.566 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.566 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:36:30.566 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.163010+0000", 2024-08-17T17:36:30.566 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016868699999999999 2024-08-17T17:36:30.566 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.567 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.567 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:36:30.567 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.163147+0000", 2024-08-17T17:36:30.567 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000136821 2024-08-17T17:36:30.567 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.567 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.567 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:36:30.567 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165452+0000", 2024-08-17T17:36:30.567 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0023047039999999999 2024-08-17T17:36:30.567 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.567 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.567 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.567 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.167852+0000", 2024-08-17T17:36:30.568 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002400648 2024-08-17T17:36:30.568 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.568 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.568 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.568 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175678+0000", 2024-08-17T17:36:30.568 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0078257429999999996 2024-08-17T17:36:30.568 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.568 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.568 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:36:30.568 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175710+0000", 2024-08-17T17:36:30.568 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2128999999999997e-05 2024-08-17T17:36:30.568 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.568 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.568 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:36:30.569 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175745+0000", 2024-08-17T17:36:30.569 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4854000000000002e-05 2024-08-17T17:36:30.569 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.569 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:30.569 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.569 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.569 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.569 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:85945 8.4 8:2a9d539f:::benchmark_data_smithi003_229103_object85944:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-08-17T17:36:30.569 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:56.753525+0000", 2024-08-17T17:36:30.569 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 153.72737119499999, 2024-08-17T17:36:30.569 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.19019344299999999, 2024-08-17T17:36:30.569 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:30.569 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:36:30.570 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:36:30.570 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:36:30.570 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:30.570 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 85945 2024-08-17T17:36:30.570 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.570 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:30.570 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.570 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:30.570 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.753525+0000", 2024-08-17T17:36:30.570 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.570 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.570 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.570 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:30.571 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.753525+0000", 2024-08-17T17:36:30.571 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.571 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.571 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.571 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:30.571 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.753523+0000", 2024-08-17T17:36:30.571 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:36:30.571 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.571 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.571 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:30.571 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.753569+0000", 2024-08-17T17:36:30.571 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5286e-05 2024-08-17T17:36:30.571 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.571 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.572 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:30.572 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.753590+0000", 2024-08-17T17:36:30.572 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1053999999999999e-05 2024-08-17T17:36:30.572 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.572 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.572 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:30.572 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.753594+0000", 2024-08-17T17:36:30.572 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9189999999999998e-06 2024-08-17T17:36:30.572 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.572 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.572 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:30.572 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.753651+0000", 2024-08-17T17:36:30.573 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6291e-05 2024-08-17T17:36:30.573 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.573 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.573 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:36:30.573 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.753776+0000", 2024-08-17T17:36:30.573 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012524199999999999 2024-08-17T17:36:30.573 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.573 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.573 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:36:30.573 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.753920+0000", 2024-08-17T17:36:30.573 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000143688 2024-08-17T17:36:30.573 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.573 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.573 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:36:30.574 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.756430+0000", 2024-08-17T17:36:30.574 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002510721 2024-08-17T17:36:30.574 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.574 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.574 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.574 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.756580+0000", 2024-08-17T17:36:30.574 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014915600000000001 2024-08-17T17:36:30.574 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.574 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.574 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.574 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.943621+0000", 2024-08-17T17:36:30.574 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.18704188599999999 2024-08-17T17:36:30.574 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.574 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.575 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:36:30.575 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.943674+0000", 2024-08-17T17:36:30.575 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3020999999999999e-05 2024-08-17T17:36:30.575 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.575 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.575 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:36:30.575 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.943718+0000", 2024-08-17T17:36:30.575 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3674999999999998e-05 2024-08-17T17:36:30.575 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.575 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:30.575 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.575 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.575 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.576 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:85950 8.4 8:2480217a:::benchmark_data_smithi003_229103_object85949:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-08-17T17:36:30.576 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:56.759541+0000", 2024-08-17T17:36:30.576 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 153.72135500799999, 2024-08-17T17:36:30.576 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.19582751800000001, 2024-08-17T17:36:30.576 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:30.576 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:36:30.576 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:36:30.576 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:36:30.576 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:30.576 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 85950 2024-08-17T17:36:30.576 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.576 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:30.576 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.576 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:30.577 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.759541+0000", 2024-08-17T17:36:30.577 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.577 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.577 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.577 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:30.577 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.759541+0000", 2024-08-17T17:36:30.577 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.577 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.577 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.577 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:30.577 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.759539+0000", 2024-08-17T17:36:30.577 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:36:30.577 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.577 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.578 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:30.578 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.759623+0000", 2024-08-17T17:36:30.578 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.3555999999999998e-05 2024-08-17T17:36:30.578 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.578 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.578 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:30.578 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.759634+0000", 2024-08-17T17:36:30.578 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1618e-05 2024-08-17T17:36:30.578 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.578 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.578 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:30.578 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.759640+0000", 2024-08-17T17:36:30.579 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0630000000000002e-06 2024-08-17T17:36:30.579 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.579 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.579 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:30.579 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.759677+0000", 2024-08-17T17:36:30.579 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6434999999999999e-05 2024-08-17T17:36:30.579 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.579 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.579 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:36:30.579 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.759777+0000", 2024-08-17T17:36:30.579 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010024199999999999 2024-08-17T17:36:30.579 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.579 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.579 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:36:30.580 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.759863+0000", 2024-08-17T17:36:30.580 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.5628000000000001e-05 2024-08-17T17:36:30.580 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.580 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.580 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:36:30.580 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.762723+0000", 2024-08-17T17:36:30.580 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0028608790000000002 2024-08-17T17:36:30.580 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.580 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.580 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.580 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.764164+0000", 2024-08-17T17:36:30.580 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0014406239999999999 2024-08-17T17:36:30.580 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.580 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.581 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.581 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.955281+0000", 2024-08-17T17:36:30.581 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.19111708599999999 2024-08-17T17:36:30.581 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.581 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.581 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:36:30.581 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.955332+0000", 2024-08-17T17:36:30.581 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0785000000000003e-05 2024-08-17T17:36:30.581 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.581 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.581 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:36:30.581 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.955368+0000", 2024-08-17T17:36:30.581 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6528e-05 2024-08-17T17:36:30.581 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.582 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:30.582 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.582 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.582 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.582 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:131591 8.2 8:484be707:::benchmark_data_smithi003_229103_object24116:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-08-17T17:36:30.582 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:35:00.828656+0000", 2024-08-17T17:36:30.582 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 89.652239774999998, 2024-08-17T17:36:30.582 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.13269879100000001, 2024-08-17T17:36:30.582 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:30.582 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:36:30.582 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:36:30.582 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:36:30.582 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:30.583 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 131591 2024-08-17T17:36:30.583 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.583 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:30.583 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.583 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:30.583 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.828656+0000", 2024-08-17T17:36:30.583 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.583 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.583 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.583 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:30.583 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.828656+0000", 2024-08-17T17:36:30.583 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.583 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.583 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.584 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:30.584 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.828647+0000", 2024-08-17T17:36:30.584 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999909 2024-08-17T17:36:30.584 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.584 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.584 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:30.584 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.828666+0000", 2024-08-17T17:36:30.584 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.8836000000000001e-05 2024-08-17T17:36:30.584 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.584 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.584 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:30.584 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.828699+0000", 2024-08-17T17:36:30.584 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3535999999999999e-05 2024-08-17T17:36:30.584 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.585 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.585 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:30.585 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.828714+0000", 2024-08-17T17:36:30.585 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4779e-05 2024-08-17T17:36:30.585 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.585 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.585 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:30.585 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.828837+0000", 2024-08-17T17:36:30.585 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000122268 2024-08-17T17:36:30.585 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.585 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.585 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:36:30.585 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.829019+0000", 2024-08-17T17:36:30.586 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00018226200000000001 2024-08-17T17:36:30.586 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.586 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.586 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:36:30.586 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.829143+0000", 2024-08-17T17:36:30.586 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000124308 2024-08-17T17:36:30.586 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.586 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.586 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:36:30.586 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.835723+0000", 2024-08-17T17:36:30.586 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0065798590000000004 2024-08-17T17:36:30.586 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.586 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.587 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.587 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.836985+0000", 2024-08-17T17:36:30.587 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0012621900000000001 2024-08-17T17:36:30.587 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.587 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.587 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.587 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.961251+0000", 2024-08-17T17:36:30.587 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.124266183 2024-08-17T17:36:30.587 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.587 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.587 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:36:30.587 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.961306+0000", 2024-08-17T17:36:30.587 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5061e-05 2024-08-17T17:36:30.588 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.588 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.588 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:36:30.588 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.961355+0000", 2024-08-17T17:36:30.588 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.8547999999999997e-05 2024-08-17T17:36:30.588 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.588 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:30.588 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.588 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.588 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.588 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:131598 8.2 8:4c61b07f:::benchmark_data_smithi003_229103_object24123:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-08-17T17:36:30.588 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:35:00.830343+0000", 2024-08-17T17:36:30.588 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 89.650552911999995, 2024-08-17T17:36:30.589 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.13114300000000001, 2024-08-17T17:36:30.589 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:30.589 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:36:30.589 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:36:30.589 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:36:30.589 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:30.589 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 131598 2024-08-17T17:36:30.589 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.589 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:30.589 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.589 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:30.589 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.830343+0000", 2024-08-17T17:36:30.589 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.589 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.590 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.590 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:30.590 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.830343+0000", 2024-08-17T17:36:30.590 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.590 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.590 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.590 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:30.590 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.830342+0000", 2024-08-17T17:36:30.590 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:36:30.590 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.590 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.590 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:30.590 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.830348+0000", 2024-08-17T17:36:30.591 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6869999999999999e-06 2024-08-17T17:36:30.591 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.591 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.591 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:30.591 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.830361+0000", 2024-08-17T17:36:30.591 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3200000000000001e-05 2024-08-17T17:36:30.591 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.591 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.591 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:30.591 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.830366+0000", 2024-08-17T17:36:30.591 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5620000000000003e-06 2024-08-17T17:36:30.591 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.591 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.591 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:30.592 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.832067+0000", 2024-08-17T17:36:30.592 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0017004380000000001 2024-08-17T17:36:30.592 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.592 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.592 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:36:30.592 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.832204+0000", 2024-08-17T17:36:30.592 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013674299999999999 2024-08-17T17:36:30.592 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.592 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.592 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:36:30.592 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.832327+0000", 2024-08-17T17:36:30.592 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000123576 2024-08-17T17:36:30.592 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.592 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.593 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:36:30.593 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.835962+0000", 2024-08-17T17:36:30.593 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0036344350000000001 2024-08-17T17:36:30.593 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.593 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.593 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.593 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.837168+0000", 2024-08-17T17:36:30.593 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0012062119999999999 2024-08-17T17:36:30.593 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.593 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.593 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.593 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.961398+0000", 2024-08-17T17:36:30.593 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.124230363 2024-08-17T17:36:30.594 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.594 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.594 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:36:30.594 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.961445+0000", 2024-08-17T17:36:30.594 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7236e-05 2024-08-17T17:36:30.594 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.594 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.594 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:36:30.594 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.961486+0000", 2024-08-17T17:36:30.594 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0608999999999997e-05 2024-08-17T17:36:30.594 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.594 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:30.594 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.595 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.595 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.595 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:136792 8.2 8:496a5b24:::benchmark_data_smithi003_229103_object29317:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-08-17T17:36:30.595 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:35:08.110793+0000", 2024-08-17T17:36:30.595 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 82.370102877999997, 2024-08-17T17:36:30.595 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.089844666000000004, 2024-08-17T17:36:30.595 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:30.595 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:36:30.595 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:36:30.595 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:36:30.595 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:30.595 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 136792 2024-08-17T17:36:30.595 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.595 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:30.596 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.596 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:30.596 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:08.110793+0000", 2024-08-17T17:36:30.596 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.596 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.596 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.596 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:30.596 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:08.110793+0000", 2024-08-17T17:36:30.596 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.596 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.596 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.596 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:30.596 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:08.110792+0000", 2024-08-17T17:36:30.596 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:36:30.597 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.597 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.597 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:30.597 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:08.110798+0000", 2024-08-17T17:36:30.597 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0959999999999997e-06 2024-08-17T17:36:30.597 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.597 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.597 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:30.597 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:08.110812+0000", 2024-08-17T17:36:30.597 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4379000000000001e-05 2024-08-17T17:36:30.597 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.597 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.597 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:30.597 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:08.110818+0000", 2024-08-17T17:36:30.598 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7899999999999996e-06 2024-08-17T17:36:30.598 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.598 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.598 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:30.598 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:08.110869+0000", 2024-08-17T17:36:30.598 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0673000000000001e-05 2024-08-17T17:36:30.598 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.598 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.598 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:36:30.598 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:08.111016+0000", 2024-08-17T17:36:30.598 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014711800000000001 2024-08-17T17:36:30.598 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.598 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.599 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:36:30.599 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:08.111128+0000", 2024-08-17T17:36:30.599 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000112453 2024-08-17T17:36:30.599 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.599 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.599 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:36:30.599 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:08.114982+0000", 2024-08-17T17:36:30.599 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0038537549999999999 2024-08-17T17:36:30.599 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.599 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.599 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.600 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:08.115700+0000", 2024-08-17T17:36:30.600 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00071780200000000002 2024-08-17T17:36:30.600 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.600 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.600 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.600 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:08.200541+0000", 2024-08-17T17:36:30.600 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.084841666999999996 2024-08-17T17:36:30.600 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.600 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.600 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:36:30.600 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:08.200596+0000", 2024-08-17T17:36:30.600 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4877000000000003e-05 2024-08-17T17:36:30.600 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.600 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.600 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:36:30.601 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:08.200638+0000", 2024-08-17T17:36:30.601 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1511999999999998e-05 2024-08-17T17:36:30.601 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.601 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:30.601 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.601 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.601 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.601 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:148933 8.2 8:40c1af93:::benchmark_data_smithi003_229103_object41458:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-08-17T17:36:30.601 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:35:25.618517+0000", 2024-08-17T17:36:30.601 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 64.862378796000002, 2024-08-17T17:36:30.601 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.113172405, 2024-08-17T17:36:30.601 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:30.601 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:36:30.602 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:36:30.602 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:36:30.602 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:30.602 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 148933 2024-08-17T17:36:30.602 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.602 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:30.602 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.602 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:30.602 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.618517+0000", 2024-08-17T17:36:30.602 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.602 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.602 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.602 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:30.602 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.618517+0000", 2024-08-17T17:36:30.603 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.603 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.603 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.603 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:30.603 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.618516+0000", 2024-08-17T17:36:30.603 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:36:30.603 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.603 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.603 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:30.603 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.618522+0000", 2024-08-17T17:36:30.603 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8449999999999999e-06 2024-08-17T17:36:30.603 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.603 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.603 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:30.604 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.618533+0000", 2024-08-17T17:36:30.604 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1015999999999999e-05 2024-08-17T17:36:30.604 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.604 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.604 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:30.604 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.618539+0000", 2024-08-17T17:36:30.604 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0299999999999999e-06 2024-08-17T17:36:30.604 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.604 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.604 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:30.604 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.622725+0000", 2024-08-17T17:36:30.604 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0041859269999999999 2024-08-17T17:36:30.604 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.604 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.605 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:36:30.605 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.622845+0000", 2024-08-17T17:36:30.605 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000120127 2024-08-17T17:36:30.605 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.605 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.605 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:36:30.605 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.622962+0000", 2024-08-17T17:36:30.605 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000116393 2024-08-17T17:36:30.605 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.605 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.605 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:36:30.605 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.635482+0000", 2024-08-17T17:36:30.605 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.012520619 2024-08-17T17:36:30.605 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.606 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.606 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.606 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.635526+0000", 2024-08-17T17:36:30.606 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4060999999999997e-05 2024-08-17T17:36:30.606 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.606 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.606 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.606 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.731596+0000", 2024-08-17T17:36:30.606 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.096069885999999993 2024-08-17T17:36:30.606 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.606 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.606 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:36:30.606 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.731648+0000", 2024-08-17T17:36:30.606 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2123999999999998e-05 2024-08-17T17:36:30.607 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.607 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.607 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:36:30.607 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.731690+0000", 2024-08-17T17:36:30.607 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1257999999999997e-05 2024-08-17T17:36:30.607 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.607 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:30.607 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.607 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.607 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.607 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:148936 8.2 8:4c6ec7d8:::benchmark_data_smithi003_229103_object41461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-08-17T17:36:30.607 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:35:25.624795+0000", 2024-08-17T17:36:30.607 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 64.856101279000001, 2024-08-17T17:36:30.607 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.107005407, 2024-08-17T17:36:30.608 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:30.608 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:36:30.608 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:36:30.608 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:36:30.608 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:30.608 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 148936 2024-08-17T17:36:30.608 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.608 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:30.608 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.608 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:30.608 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.624795+0000", 2024-08-17T17:36:30.608 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.608 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.608 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.609 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:30.609 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.624795+0000", 2024-08-17T17:36:30.609 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.609 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.609 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.609 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:30.609 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.624793+0000", 2024-08-17T17:36:30.609 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:36:30.609 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.609 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.609 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:30.609 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.624799+0000", 2024-08-17T17:36:30.609 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8479999999999997e-06 2024-08-17T17:36:30.609 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.610 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.610 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:30.610 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.624810+0000", 2024-08-17T17:36:30.610 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0743e-05 2024-08-17T17:36:30.610 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.610 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.610 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:30.610 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.624815+0000", 2024-08-17T17:36:30.610 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5750000000000002e-06 2024-08-17T17:36:30.610 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.610 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.610 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:30.610 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.627514+0000", 2024-08-17T17:36:30.611 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0026989449999999999 2024-08-17T17:36:30.611 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.611 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.611 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:36:30.611 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.627640+0000", 2024-08-17T17:36:30.611 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012653400000000001 2024-08-17T17:36:30.611 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.611 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.611 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:36:30.611 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.627820+0000", 2024-08-17T17:36:30.611 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00018041699999999999 2024-08-17T17:36:30.611 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.611 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.611 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:36:30.612 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.635489+0000", 2024-08-17T17:36:30.612 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0076683080000000004 2024-08-17T17:36:30.612 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.612 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.612 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.612 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.636781+0000", 2024-08-17T17:36:30.612 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0012921479999999999 2024-08-17T17:36:30.612 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.612 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.612 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.612 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.731734+0000", 2024-08-17T17:36:30.612 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.094953234999999997 2024-08-17T17:36:30.612 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.612 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.613 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:36:30.613 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.731764+0000", 2024-08-17T17:36:30.613 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9689999999999999e-05 2024-08-17T17:36:30.613 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.613 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.613 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:36:30.613 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.731800+0000", 2024-08-17T17:36:30.613 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6210000000000001e-05 2024-08-17T17:36:30.613 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.613 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:30.613 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.613 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.613 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.613 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:148940 8.2 8:499a33a4:::benchmark_data_smithi003_229103_object41465:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-08-17T17:36:30.614 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:35:25.627862+0000", 2024-08-17T17:36:30.614 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 64.853033687999996, 2024-08-17T17:36:30.614 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.10406536600000001, 2024-08-17T17:36:30.614 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:30.614 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:36:30.614 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:36:30.614 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:36:30.614 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:30.614 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 148940 2024-08-17T17:36:30.614 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.614 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:30.614 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.614 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:30.615 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.627862+0000", 2024-08-17T17:36:30.615 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.615 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.615 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.615 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:30.615 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.627862+0000", 2024-08-17T17:36:30.615 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:30.615 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.615 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.615 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:30.615 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.627861+0000", 2024-08-17T17:36:30.615 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:36:30.615 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.615 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.616 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:30.616 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.627870+0000", 2024-08-17T17:36:30.616 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1700000000000003e-06 2024-08-17T17:36:30.616 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.616 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.616 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:30.616 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.627883+0000", 2024-08-17T17:36:30.616 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2564e-05 2024-08-17T17:36:30.616 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.616 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.616 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:30.616 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.627889+0000", 2024-08-17T17:36:30.616 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8869999999999997e-06 2024-08-17T17:36:30.617 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.617 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.617 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:30.617 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.631689+0000", 2024-08-17T17:36:30.617 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0038005880000000001 2024-08-17T17:36:30.617 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.617 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.617 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:36:30.617 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.631814+0000", 2024-08-17T17:36:30.617 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012496 2024-08-17T17:36:30.617 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.617 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.617 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:36:30.617 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.631939+0000", 2024-08-17T17:36:30.618 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000124912 2024-08-17T17:36:30.618 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.618 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.618 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:36:30.618 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.635584+0000", 2024-08-17T17:36:30.618 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003644521 2024-08-17T17:36:30.618 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.618 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.618 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.618 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.636857+0000", 2024-08-17T17:36:30.618 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001273649 2024-08-17T17:36:30.618 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.618 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.618 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:30.619 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.731839+0000", 2024-08-17T17:36:30.619 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.094981306000000001 2024-08-17T17:36:30.619 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.619 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.619 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:36:30.619 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.731882+0000", 2024-08-17T17:36:30.619 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3563000000000003e-05 2024-08-17T17:36:30.619 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:30.619 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:30.619 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:36:30.619 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:25.731928+0000", 2024-08-17T17:36:30.619 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5445999999999998e-05 2024-08-17T17:36:30.619 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.619 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:30.620 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.620 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:30.620 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:30.620 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:36:30.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:30.622 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:30.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:30.723 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:30.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:30.765 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:30.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:30.866 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:30.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:30.908 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:30.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:31.009 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:31.043 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:31.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:31.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:31.144 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:31.186 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:31.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:31.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:31.288 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:31.334 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:31.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:31.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:31.436 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:31.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:31 smithi053 ceph-mon[89963]: pgmap v922: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.4 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 682 op/s; 19983/70374 objects degraded (28.395%); 0 B/s, 24 objects/s recovering 2024-08-17T17:36:31.486 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:31.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:31.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:31.559 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:31 smithi003 ceph-mon[103344]: pgmap v922: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.4 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 682 op/s; 19983/70374 objects degraded (28.395%); 0 B/s, 24 objects/s recovering 2024-08-17T17:36:31.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:31 smithi003 ceph-mon[100615]: pgmap v922: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.4 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 682 op/s; 19983/70374 objects degraded (28.395%); 0 B/s, 24 objects/s recovering 2024-08-17T17:36:31.587 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:31.600 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 6 1 2024-08-17T17:36:31.615 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:31.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:31.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:31.716 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:31.753 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:31.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:31.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:31.768 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-17T17:36:31.855 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:31.881 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:31.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:31.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:31.983 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:32.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:32.038 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:32.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:32.139 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:32.178 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:36:32.190 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:32.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:32.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:32.292 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:32.334 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:32.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:32.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:32.436 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:32.471 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:32.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:32.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:32.573 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:32.601 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:32.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:32.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:32.703 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:32.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:32.780 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:32.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:32.878 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:36:32.881 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:32.929 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:32.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:32.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:33.032 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:33.068 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:33.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:33.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:33.170 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:33.197 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:33.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:33.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:33.299 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:33.300 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-08-17T17:36:33.324 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:33.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:33.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:33.426 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:33.471 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:33.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:33.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:33.573 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:33.609 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:33.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:33.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:33.711 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:33.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:33 smithi053 ceph-mon[89963]: pgmap v923: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.4 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 688 op/s; 19983/69036 objects degraded (28.946%); 0 B/s, 24 objects/s recovering 2024-08-17T17:36:33.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:33 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3809644801' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:36:33.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:33 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:36:33.751 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:33.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:33.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:33.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:33 smithi003 ceph-mon[100615]: pgmap v923: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.4 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 688 op/s; 19983/69036 objects degraded (28.946%); 0 B/s, 24 objects/s recovering 2024-08-17T17:36:33.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:33 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3809644801' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:36:33.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:33 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:36:33.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:33 smithi003 ceph-mon[103344]: pgmap v923: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.4 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 688 op/s; 19983/69036 objects degraded (28.946%); 0 B/s, 24 objects/s recovering 2024-08-17T17:36:33.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:33 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3809644801' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:36:33.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:33 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:36:33.853 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:33.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:33.900 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:33.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:34.002 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:34.032 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:34.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:34.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:34.134 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:34.171 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:34.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:34.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:34.272 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:34.316 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:34.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:34.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:34.418 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:34.435 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:36:34.447 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:34.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:34.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:34.476 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:34 smithi003 ceph-mon[103344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:36:34.477 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:34 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:36:34.477 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:34 smithi003 ceph-mon[103344]: osdmap e294: 8 total, 8 up, 8 in 2024-08-17T17:36:34.477 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:34 smithi003 ceph-mon[100615]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:36:34.477 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:34 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:36:34.477 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:34 smithi003 ceph-mon[100615]: osdmap e294: 8 total, 8 up, 8 in 2024-08-17T17:36:34.549 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:34.579 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:34.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:34.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:34.681 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:34.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:34 smithi053 ceph-mon[89963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:36:34.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:34 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:36:34.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:34 smithi053 ceph-mon[89963]: osdmap e294: 8 total, 8 up, 8 in 2024-08-17T17:36:34.762 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:34.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:34.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:34.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:34.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:34.917 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:34.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:35.018 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:35.047 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:35.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:35.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:35.098 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:36:35.148 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:35.178 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:35.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:35.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:35.280 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:35.311 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:35.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:35.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:35.413 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:35.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:35.454 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:35.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:35.516 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:35 smithi003 ceph-mon[103344]: pgmap v925: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.2 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 826 op/s; 19846/59760 objects degraded (33.210%); 0 B/s, 32 objects/s recovering 2024-08-17T17:36:35.516 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:35 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 19846/59760 objects degraded (33.210%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:35.516 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:35 smithi003 ceph-mon[100615]: pgmap v925: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.2 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 826 op/s; 19846/59760 objects degraded (33.210%); 0 B/s, 32 objects/s recovering 2024-08-17T17:36:35.516 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:35 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 19846/59760 objects degraded (33.210%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:35.555 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:35.593 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:35.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:35.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:35.668 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:35 smithi053 ceph-mon[89963]: pgmap v925: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.2 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 826 op/s; 19846/59760 objects degraded (33.210%); 0 B/s, 32 objects/s recovering 2024-08-17T17:36:35.668 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:35 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 19846/59760 objects degraded (33.210%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:35.695 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:35.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:35.732 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:35.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:35.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:36:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:36:35] "GET /metrics HTTP/1.1" 200 36506 "" "Prometheus/2.51.0" 2024-08-17T17:36:35.833 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:35.935 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:35.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:35.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:36.037 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:36.080 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:36.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:36.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:36.169 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:36:36.169 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2024-08-17T17:36:36.170 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.170 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:196843 8.3 8:cea74927:::benchmark_data_smithi003_229103_object89368:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-08-17T17:36:36.170 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:36.126164+0000", 2024-08-17T17:36:36.170 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.040699845999999998, 2024-08-17T17:36:36.170 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.040730032999999999, 2024-08-17T17:36:36.170 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:36.170 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:36:36.170 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:36:36.170 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:36:36.170 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:36.170 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 196843 2024-08-17T17:36:36.171 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.171 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:36.171 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.171 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:36.171 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.126164+0000", 2024-08-17T17:36:36.171 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:36.171 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.171 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.171 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:36.171 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.126164+0000", 2024-08-17T17:36:36.171 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:36.171 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.171 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.172 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:36.172 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.126162+0000", 2024-08-17T17:36:36.172 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999976 2024-08-17T17:36:36.172 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.172 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.172 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:36.172 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.126171+0000", 2024-08-17T17:36:36.172 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9490000000000002e-06 2024-08-17T17:36:36.172 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.172 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.172 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:36.172 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.126192+0000", 2024-08-17T17:36:36.172 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0511999999999999e-05 2024-08-17T17:36:36.173 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.173 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.173 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:36.173 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.126200+0000", 2024-08-17T17:36:36.173 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1839999999999999e-06 2024-08-17T17:36:36.173 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.173 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.173 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:36.173 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.126252+0000", 2024-08-17T17:36:36.173 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2744999999999997e-05 2024-08-17T17:36:36.173 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.173 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.173 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:36:36.173 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.126400+0000", 2024-08-17T17:36:36.173 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014729399999999999 2024-08-17T17:36:36.174 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.174 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.174 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:36:36.174 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.126566+0000", 2024-08-17T17:36:36.174 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000166367 2024-08-17T17:36:36.174 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.174 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.174 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:36.174 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.146059+0000", 2024-08-17T17:36:36.174 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.019492637 2024-08-17T17:36:36.174 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.174 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.175 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:36:36.175 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.146070+0000", 2024-08-17T17:36:36.175 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1369000000000001e-05 2024-08-17T17:36:36.175 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:36.175 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:36.175 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:36.175 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.175 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.175 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:196849 8.3 8:cc078342:::benchmark_data_smithi003_229103_object89374:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-08-17T17:36:36.175 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:36.128930+0000", 2024-08-17T17:36:36.175 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.037934178999999998, 2024-08-17T17:36:36.175 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.038059976000000002, 2024-08-17T17:36:36.175 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:36.175 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:36:36.176 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:36:36.176 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:36:36.176 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:36.176 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 196849 2024-08-17T17:36:36.176 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.176 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:36.176 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.176 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:36.176 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.128930+0000", 2024-08-17T17:36:36.176 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:36.176 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.176 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.176 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:36.176 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.128930+0000", 2024-08-17T17:36:36.176 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:36.177 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.177 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.177 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:36.177 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.128928+0000", 2024-08-17T17:36:36.177 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:36:36.177 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.177 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.177 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:36.177 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.128936+0000", 2024-08-17T17:36:36.177 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1680000000000004e-06 2024-08-17T17:36:36.177 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.177 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.177 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:36.177 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.128952+0000", 2024-08-17T17:36:36.178 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6084999999999999e-05 2024-08-17T17:36:36.178 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.178 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.178 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:36.178 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.128960+0000", 2024-08-17T17:36:36.178 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.2529999999999998e-06 2024-08-17T17:36:36.178 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.178 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.179 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:36.179 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.136935+0000", 2024-08-17T17:36:36.179 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0079758940000000007 2024-08-17T17:36:36.179 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.179 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.179 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:36:36.179 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.137075+0000", 2024-08-17T17:36:36.179 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013993600000000001 2024-08-17T17:36:36.179 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.179 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.179 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:36:36.179 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.137212+0000", 2024-08-17T17:36:36.179 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013705100000000001 2024-08-17T17:36:36.179 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.180 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.180 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:36:36.180 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.151205+0000", 2024-08-17T17:36:36.180 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.013992292 2024-08-17T17:36:36.180 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.180 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.180 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:36:36.180 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.151735+0000", 2024-08-17T17:36:36.180 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00053023899999999995 2024-08-17T17:36:36.180 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:36.181 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:36.181 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:36.181 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.181 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.181 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:196855 8.3 8:ca6b9c31:::benchmark_data_smithi003_229103_object89380:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-08-17T17:36:36.181 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:36.142193+0000", 2024-08-17T17:36:36.181 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.024670652000000001, 2024-08-17T17:36:36.181 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.024877211, 2024-08-17T17:36:36.181 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:36.181 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:36:36.181 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:36:36.181 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:36:36.181 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:36.181 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 196855 2024-08-17T17:36:36.181 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.182 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:36.182 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.182 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:36.182 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.142193+0000", 2024-08-17T17:36:36.182 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:36.182 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.182 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.182 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:36.182 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.142193+0000", 2024-08-17T17:36:36.182 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:36.182 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.182 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.183 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:36.183 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.142192+0000", 2024-08-17T17:36:36.183 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:36:36.183 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.183 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.183 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:36.183 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.142199+0000", 2024-08-17T17:36:36.183 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.114e-06 2024-08-17T17:36:36.183 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.183 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.183 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:36.184 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.142213+0000", 2024-08-17T17:36:36.184 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4059999999999999e-05 2024-08-17T17:36:36.184 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.184 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.184 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:36.184 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.142220+0000", 2024-08-17T17:36:36.184 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9190000000000002e-06 2024-08-17T17:36:36.184 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.184 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.184 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:36.184 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.146096+0000", 2024-08-17T17:36:36.185 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:36.186 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0038763589999999998 2024-08-17T17:36:36.186 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.186 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.186 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:36:36.186 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.146245+0000", 2024-08-17T17:36:36.186 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000148553 2024-08-17T17:36:36.186 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.186 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.187 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:36:36.187 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.146382+0000", 2024-08-17T17:36:36.187 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000136768 2024-08-17T17:36:36.187 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.187 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.187 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:36:36.187 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.152624+0000", 2024-08-17T17:36:36.187 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0062420870000000003 2024-08-17T17:36:36.187 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:36.187 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:36.187 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:36.187 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.187 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.188 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.46025.0:196864 8.2 e294/262 8:40c3f3ab:::benchmark_data_smithi003_229103_object89389:head v 294'12299, mlcod=294'12296)", 2024-08-17T17:36:36.188 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:36.158546+0000", 2024-08-17T17:36:36.188 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.0083184369999999997, 2024-08-17T17:36:36.188 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0086001949999999997, 2024-08-17T17:36:36.188 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:36.188 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2024-08-17T17:36:36.188 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:36.188 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.188 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:36.188 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.158546+0000", 2024-08-17T17:36:36.188 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:36.188 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.188 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.189 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:36.189 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.158546+0000", 2024-08-17T17:36:36.189 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:36.189 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.189 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.189 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:36.189 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.158545+0000", 2024-08-17T17:36:36.189 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:36:36.189 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.189 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.189 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:36.190 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.158553+0000", 2024-08-17T17:36:36.190 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.4800000000000001e-06 2024-08-17T17:36:36.190 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.190 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.190 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:36.190 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.158573+0000", 2024-08-17T17:36:36.190 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9799000000000001e-05 2024-08-17T17:36:36.190 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.190 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.191 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:36.191 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.158580+0000", 2024-08-17T17:36:36.191 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5989999999999998e-06 2024-08-17T17:36:36.191 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.191 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.191 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:36.191 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.158750+0000", 2024-08-17T17:36:36.191 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016995199999999999 2024-08-17T17:36:36.191 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.191 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.191 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:36:36.191 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.158771+0000", 2024-08-17T17:36:36.191 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1188000000000001e-05 2024-08-17T17:36:36.191 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:36.192 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:36.192 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:36.192 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.192 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.192 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.46025.0:196866 8.0 e294/269 8:09d9c14a:::benchmark_data_smithi003_229103_object89391:head v 294'12256, mlcod=0'0)", 2024-08-17T17:36:36.192 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:36.158651+0000", 2024-08-17T17:36:36.192 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.0082128400000000008, 2024-08-17T17:36:36.192 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0085498359999999999, 2024-08-17T17:36:36.192 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:36.192 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2024-08-17T17:36:36.192 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:36.192 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.192 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:36.193 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.158651+0000", 2024-08-17T17:36:36.193 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:36.193 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.193 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.193 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:36.193 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.158651+0000", 2024-08-17T17:36:36.193 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:36.193 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.193 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.193 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:36.193 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.158651+0000", 2024-08-17T17:36:36.194 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:36:36.194 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.194 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.194 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:36.194 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.158655+0000", 2024-08-17T17:36:36.194 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1559999999999997e-06 2024-08-17T17:36:36.194 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.194 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.195 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:36.195 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.158662+0000", 2024-08-17T17:36:36.195 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.8920000000000003e-06 2024-08-17T17:36:36.195 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.195 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.195 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:36.195 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.158666+0000", 2024-08-17T17:36:36.195 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4309999999999996e-06 2024-08-17T17:36:36.196 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.196 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.196 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:36.196 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.158714+0000", 2024-08-17T17:36:36.196 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7333000000000002e-05 2024-08-17T17:36:36.196 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.196 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.196 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:36:36.197 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.158738+0000", 2024-08-17T17:36:36.197 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4273999999999998e-05 2024-08-17T17:36:36.197 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:36.197 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:36.197 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:36.197 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.197 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.197 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.46025.0:196867 8.4 e294/272 8:26b78344:::benchmark_data_smithi003_229103_object89392:head v 294'12206, mlcod=294'12204)", 2024-08-17T17:36:36.198 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:36.160618+0000", 2024-08-17T17:36:36.198 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.0062465039999999999, 2024-08-17T17:36:36.198 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0066382389999999998, 2024-08-17T17:36:36.198 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:36.198 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2024-08-17T17:36:36.198 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:36.198 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.199 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:36.199 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.160618+0000", 2024-08-17T17:36:36.199 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:36.199 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.199 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.199 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:36.199 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.160618+0000", 2024-08-17T17:36:36.199 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:36.199 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.200 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.200 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:36.200 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.160617+0000", 2024-08-17T17:36:36.200 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:36:36.200 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.200 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.200 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:36.200 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.160624+0000", 2024-08-17T17:36:36.201 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.7630000000000003e-06 2024-08-17T17:36:36.201 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.201 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.201 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:36.201 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.160639+0000", 2024-08-17T17:36:36.201 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5858999999999998e-05 2024-08-17T17:36:36.201 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.201 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.202 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:36.202 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.160646+0000", 2024-08-17T17:36:36.202 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1850000000000001e-06 2024-08-17T17:36:36.202 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.202 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.202 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:36.202 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.160711+0000", 2024-08-17T17:36:36.202 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5135999999999994e-05 2024-08-17T17:36:36.203 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.203 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.203 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:36:36.203 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.160730+0000", 2024-08-17T17:36:36.203 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9219999999999999e-05 2024-08-17T17:36:36.203 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:36.203 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:36.204 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:36.204 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.204 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.204 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:196871 8.3 8:c191d033:::benchmark_data_smithi003_229103_object89396:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-08-17T17:36:36.204 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:36.161278+0000", 2024-08-17T17:36:36.204 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.005585802, 2024-08-17T17:36:36.204 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0060337749999999999, 2024-08-17T17:36:36.204 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:36.205 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:36:36.205 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:36:36.205 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:36:36.205 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:36:36.205 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 196871 2024-08-17T17:36:36.205 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.205 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:36.205 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.205 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:36.206 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.161278+0000", 2024-08-17T17:36:36.206 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:36.206 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.206 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.206 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:36.206 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.161278+0000", 2024-08-17T17:36:36.206 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:36.206 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.207 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.207 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:36.207 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.161277+0000", 2024-08-17T17:36:36.207 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:36:36.207 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.207 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.207 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:36.207 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.161287+0000", 2024-08-17T17:36:36.208 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0383999999999999e-05 2024-08-17T17:36:36.208 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.208 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.208 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:36.208 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.161301+0000", 2024-08-17T17:36:36.208 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3994e-05 2024-08-17T17:36:36.209 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.209 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.209 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:36.209 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.161308+0000", 2024-08-17T17:36:36.209 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5039999999999999e-06 2024-08-17T17:36:36.210 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.210 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.210 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:36.210 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.161364+0000", 2024-08-17T17:36:36.210 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6613999999999997e-05 2024-08-17T17:36:36.211 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.211 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.211 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:36:36.211 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.161498+0000", 2024-08-17T17:36:36.211 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000133528 2024-08-17T17:36:36.211 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.211 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.211 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:36:36.211 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.161633+0000", 2024-08-17T17:36:36.211 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013543100000000001 2024-08-17T17:36:36.212 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:36.212 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:36.212 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:36.212 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.212 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.212 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.46025.0:196870 8.0 e294/269)", 2024-08-17T17:36:36.212 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:36.167179+0000", 2024-08-17T17:36:36.212 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4294967295.9996848, 2024-08-17T17:36:36.212 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00022247099999999999, 2024-08-17T17:36:36.212 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:36.212 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "queued for pg", 2024-08-17T17:36:36.212 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:36.213 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.213 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:36.213 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167179+0000", 2024-08-17T17:36:36.213 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:36.213 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.213 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.213 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:36.213 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167179+0000", 2024-08-17T17:36:36.213 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:36.213 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.213 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.213 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:36.213 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167178+0000", 2024-08-17T17:36:36.214 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:36:36.214 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.214 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.214 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:36.214 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167186+0000", 2024-08-17T17:36:36.214 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1939999999999998e-06 2024-08-17T17:36:36.214 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.214 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.214 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:36.214 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167203+0000", 2024-08-17T17:36:36.214 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6940999999999999e-05 2024-08-17T17:36:36.214 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.215 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.215 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:36.215 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167209+0000", 2024-08-17T17:36:36.215 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9390000000000002e-06 2024-08-17T17:36:36.215 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:36.215 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:36.215 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:36.215 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.215 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.215 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.46025.0:196869 8.2 e294/262)", 2024-08-17T17:36:36.215 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:36.167229+0000", 2024-08-17T17:36:36.216 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4294967295.9996352, 2024-08-17T17:36:36.216 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000220232, 2024-08-17T17:36:36.216 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:36.216 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2024-08-17T17:36:36.216 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:36.216 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.216 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:36.216 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167229+0000", 2024-08-17T17:36:36.216 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:36.216 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.216 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.216 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:36.216 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167229+0000", 2024-08-17T17:36:36.216 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:36.216 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.217 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.217 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:36.217 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167228+0000", 2024-08-17T17:36:36.217 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:36:36.217 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.217 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.217 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:36.217 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167233+0000", 2024-08-17T17:36:36.217 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1910000000000002e-06 2024-08-17T17:36:36.217 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.217 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.217 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:36.217 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167239+0000", 2024-08-17T17:36:36.217 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4749999999999998e-06 2024-08-17T17:36:36.218 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.218 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.218 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:36.218 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167243+0000", 2024-08-17T17:36:36.218 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4209999999999999e-06 2024-08-17T17:36:36.218 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.218 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.218 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:36:36.218 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167431+0000", 2024-08-17T17:36:36.218 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00018876900000000001 2024-08-17T17:36:36.218 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.218 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.218 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:36:36.218 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167451+0000", 2024-08-17T17:36:36.219 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9272000000000001e-05 2024-08-17T17:36:36.219 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:36.219 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:36.219 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:36.219 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.219 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.219 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop_reply(client.46025.0:196843 8.3 e294/269)", 2024-08-17T17:36:36.219 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:36.167265+0000", 2024-08-17T17:36:36.219 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4294967295.9995995, 2024-08-17T17:36:36.219 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00024409599999999999, 2024-08-17T17:36:36.220 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:36.220 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "queued for pg", 2024-08-17T17:36:36.220 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:36.220 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.220 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:36.220 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167265+0000", 2024-08-17T17:36:36.220 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:36.220 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.220 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.220 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:36.220 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167265+0000", 2024-08-17T17:36:36.220 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:36.220 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.221 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.221 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:36.221 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167264+0000", 2024-08-17T17:36:36.221 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:36:36.221 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.221 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.221 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:36.221 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167268+0000", 2024-08-17T17:36:36.221 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2930000000000001e-06 2024-08-17T17:36:36.221 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.221 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.221 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:36.222 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167274+0000", 2024-08-17T17:36:36.222 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.6889999999999998e-06 2024-08-17T17:36:36.222 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.222 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.222 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:36.222 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167278+0000", 2024-08-17T17:36:36.222 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.41e-06 2024-08-17T17:36:36.222 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:36.222 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:36.222 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:36.222 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.222 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.222 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop_reply(client.46025.0:196849 8.3 e294/269)", 2024-08-17T17:36:36.223 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:36.167286+0000", 2024-08-17T17:36:36.223 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4294967295.9995775, 2024-08-17T17:36:36.223 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000264133, 2024-08-17T17:36:36.223 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:36.223 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "queued for pg", 2024-08-17T17:36:36.223 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:36.223 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.223 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:36.223 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167286+0000", 2024-08-17T17:36:36.223 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:36.223 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.223 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.223 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:36.224 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167286+0000", 2024-08-17T17:36:36.224 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:36.224 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.224 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.224 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:36.224 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167286+0000", 2024-08-17T17:36:36.224 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967296 2024-08-17T17:36:36.224 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.224 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.224 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:36.224 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167288+0000", 2024-08-17T17:36:36.224 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9259999999999999e-06 2024-08-17T17:36:36.224 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.224 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.224 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:36.225 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167293+0000", 2024-08-17T17:36:36.225 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5329999999999996e-06 2024-08-17T17:36:36.225 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.225 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.225 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:36.225 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167296+0000", 2024-08-17T17:36:36.225 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9239999999999999e-06 2024-08-17T17:36:36.225 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:36.225 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:36.225 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:36.225 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.226 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.226 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop_reply(client.46025.0:196855 8.3 e294/269)", 2024-08-17T17:36:36.226 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:36.167299+0000", 2024-08-17T17:36:36.226 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4294967295.9995651, 2024-08-17T17:36:36.226 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00029196099999999998, 2024-08-17T17:36:36.226 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:36.226 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "queued for pg", 2024-08-17T17:36:36.226 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:36.226 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.226 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:36.226 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167299+0000", 2024-08-17T17:36:36.226 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:36.227 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.227 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.227 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:36.227 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167299+0000", 2024-08-17T17:36:36.227 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:36.227 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.227 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.227 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:36.227 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167299+0000", 2024-08-17T17:36:36.227 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967296 2024-08-17T17:36:36.227 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.227 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.227 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:36.227 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167301+0000", 2024-08-17T17:36:36.227 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7460000000000001e-06 2024-08-17T17:36:36.228 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.228 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.228 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:36.228 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167305+0000", 2024-08-17T17:36:36.228 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5549999999999996e-06 2024-08-17T17:36:36.228 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.228 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.228 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:36.228 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167308+0000", 2024-08-17T17:36:36.228 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7709999999999998e-06 2024-08-17T17:36:36.228 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:36.228 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:36.228 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:36.228 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.228 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.228 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop_reply(client.46025.0:196855 8.3 e294/269)", 2024-08-17T17:36:36.229 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:36.167449+0000", 2024-08-17T17:36:36.229 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 4294967295.9994149, 2024-08-17T17:36:36.229 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00018260600000000001, 2024-08-17T17:36:36.229 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:36:36.229 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "queued for pg", 2024-08-17T17:36:36.229 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:36:36.229 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.229 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:36:36.229 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167449+0000", 2024-08-17T17:36:36.229 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:36.229 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.229 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.229 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:36:36.230 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167449+0000", 2024-08-17T17:36:36.230 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:36:36.230 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.230 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.230 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:36:36.230 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167449+0000", 2024-08-17T17:36:36.230 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:36:36.230 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.230 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.230 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:36:36.230 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167454+0000", 2024-08-17T17:36:36.230 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4709999999999998e-06 2024-08-17T17:36:36.230 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.230 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.231 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:36:36.231 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167464+0000", 2024-08-17T17:36:36.231 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.4499999999999993e-06 2024-08-17T17:36:36.231 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:36:36.231 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:36:36.231 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:36:36.231 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:36.167468+0000", 2024-08-17T17:36:36.231 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.729e-06 2024-08-17T17:36:36.231 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:36.231 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:36:36.231 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:36.232 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:36:36.232 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-08-17T17:36:36.232 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 13 2024-08-17T17:36:36.232 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:36:36.232 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:36:36.266 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:36.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:36.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:36.346 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.6 to 1 (10000) 2024-08-17T17:36:36.368 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:36.392 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:36.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:36.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:36.494 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:36.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:36 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/792298966' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:36:36.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:36 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:36:36.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:36 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/792298966' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:36:36.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:36 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:36:36.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:36.527 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:36.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:36.628 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:36.650 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:36 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/792298966' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:36:36.650 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:36 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:36:36.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:36.676 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:36.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:36.777 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:36.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:36.813 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:36.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:36.914 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:36.953 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:36.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:36.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:37.055 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:37.093 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:37.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:37.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:37.131 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-17T17:36:37.196 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:37.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:37.238 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:37.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:37.339 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:37.379 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:37.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:37.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:37.481 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:37.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:37.521 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:37.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:37.583 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:36:37.621 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:37.668 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:37.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:37.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:37.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:37 smithi053 ceph-mon[89963]: pgmap v926: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.2 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 880 op/s; 19846/58446 objects degraded (33.956%); 0 B/s, 32 objects/s recovering 2024-08-17T17:36:37.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:37 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-17T17:36:37.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:37 smithi053 ceph-mon[89963]: osdmap e295: 8 total, 8 up, 8 in 2024-08-17T17:36:37.770 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:37.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:37 smithi003 ceph-mon[100615]: pgmap v926: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.2 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 880 op/s; 19846/58446 objects degraded (33.956%); 0 B/s, 32 objects/s recovering 2024-08-17T17:36:37.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:37 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-17T17:36:37.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:37 smithi003 ceph-mon[100615]: osdmap e295: 8 total, 8 up, 8 in 2024-08-17T17:36:37.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:37 smithi003 ceph-mon[103344]: pgmap v926: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.2 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 880 op/s; 19846/58446 objects degraded (33.956%); 0 B/s, 32 objects/s recovering 2024-08-17T17:36:37.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:37 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-17T17:36:37.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:37 smithi003 ceph-mon[103344]: osdmap e295: 8 total, 8 up, 8 in 2024-08-17T17:36:37.800 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:37.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:37.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:37.902 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:37.944 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:37.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:37.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:38.046 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:38.092 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:38.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:38.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:38.194 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:38.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:38.245 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:38.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:38.346 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:38.374 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:38.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:38.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:38.476 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:38.511 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:38.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:38.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:38.614 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:38.641 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:38.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:38.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:38.743 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:38.768 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:38.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:38.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:38.870 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:38.900 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:38.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:38.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:39.001 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:39.043 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:39.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:39.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:39.145 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:39.191 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:39.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:39.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:39.293 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:39.339 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:39.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:39.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:39.357 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:36:39.382 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:36:39.383 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:36:39.383 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:39 smithi053 ceph-mon[89963]: pgmap v928: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.2 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 496 op/s; 19846/58446 objects degraded (33.956%); 0 B/s, 17 objects/s recovering 2024-08-17T17:36:39.388 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:36:39.388 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:36:39.388 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:39 smithi003 ceph-mon[103344]: pgmap v928: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.2 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 496 op/s; 19846/58446 objects degraded (33.956%); 0 B/s, 17 objects/s recovering 2024-08-17T17:36:39.389 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:36:39.389 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:36:39.389 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:39 smithi003 ceph-mon[100615]: pgmap v928: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.2 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 496 op/s; 19846/58446 objects degraded (33.956%); 0 B/s, 17 objects/s recovering 2024-08-17T17:36:39.391 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 7 1 2024-08-17T17:36:39.442 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:39.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:39.534 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:39.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:39.635 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:39.668 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:36:39.669 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2024-08-17T17:36:39.669 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2024-08-17T17:36:39.669 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2024-08-17T17:36:39.669 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:36:39.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:39.695 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:39.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:39.796 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:39.852 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:39.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:39.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:39.953 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:40.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:40.009 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:40.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:40.111 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:40.155 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:40.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:40.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:40.238 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:36:40.238 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:36:40.238 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:40 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 19846/58446 objects degraded (33.956%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:40.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:40 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 19846/58446 objects degraded (33.956%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:40.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:40.286 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:36:40.286 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:36:40.309 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:40.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:40.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:40.337 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:36:40.337 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:36:40.355 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:36:40.356 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:36:40.411 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:40.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:40.437 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:40.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:40.462 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:36:40.462 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:36:40.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:40 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 19846/58446 objects degraded (33.956%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:40.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:36:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:36:40.216Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:36:40.493 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:36:40.494 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:36:40.526 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:36:40.526 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:36:40.538 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:40.564 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:36:40.564 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:36:40.585 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:40.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:40.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:40.686 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:40.730 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:40.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:40.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:40.831 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:40.863 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:40.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:40.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:40.967 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:41.005 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:36:41.019 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:41.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:41.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:41.121 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:41.147 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:41.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:41.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:41.236 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-17T17:36:41.249 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:41.310 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:41.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:41.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:41.411 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:41.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:41.459 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:41.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:41.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:41 smithi053 ceph-mon[89963]: pgmap v929: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.0 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 814 op/s; 19707/49482 objects degraded (39.827%); 0 B/s, 34 objects/s recovering 2024-08-17T17:36:41.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:41 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3366550230' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:36:41.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:41 smithi003 ceph-mon[103344]: pgmap v929: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.0 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 814 op/s; 19707/49482 objects degraded (39.827%); 0 B/s, 34 objects/s recovering 2024-08-17T17:36:41.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:41 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3366550230' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:36:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:41 smithi003 ceph-mon[100615]: pgmap v929: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1.0 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 814 op/s; 19707/49482 objects degraded (39.827%); 0 B/s, 34 objects/s recovering 2024-08-17T17:36:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:41 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3366550230' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:36:41.559 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:41.592 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:41.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:41.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:41.669 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:36:41.696 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:41.733 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:41.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:41.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:41.835 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:41.863 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:41.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:41.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:41.965 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:42.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:42.006 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:42.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:42.108 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:42.142 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:42.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:42.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:42.244 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:42.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:42.279 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:42.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:42.380 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:42.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:42.433 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:42.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:42.534 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:42.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:42.581 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:42.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:42.682 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:42.727 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:42.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:42.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:42.829 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:42.879 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:42.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:42.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:42.981 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:43.021 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:43.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:43.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:43.123 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:43.161 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:43.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:43.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:43.263 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:43.300 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:43.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:43.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:43.402 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:43.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:43.442 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:43.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:43.494 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:43 smithi053 ceph-mon[89963]: pgmap v930: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1007 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 782 op/s; 19707/48201 objects degraded (40.885%); 0 B/s, 31 objects/s recovering 2024-08-17T17:36:43.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:43 smithi003 ceph-mon[100615]: pgmap v930: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1007 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 782 op/s; 19707/48201 objects degraded (40.885%); 0 B/s, 31 objects/s recovering 2024-08-17T17:36:43.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:43 smithi003 ceph-mon[103344]: pgmap v930: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 1007 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 782 op/s; 19707/48201 objects degraded (40.885%); 0 B/s, 31 objects/s recovering 2024-08-17T17:36:43.523 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:36:43.524 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:36:43.543 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:43.586 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:43.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:43.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:43.687 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:43.742 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:43.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:43.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:43.844 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:43.884 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:43.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:43.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:43.986 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:44.022 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:44.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:44.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:44.123 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:44.148 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:44.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:44.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:44.211 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-17T17:36:44.239 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-08-17T17:36:44.250 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:44.282 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:44.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:44.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:44.377 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:36:44.383 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:44.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:44 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2708601266' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:36:44.470 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:44 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:36:44.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:44.480 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:44.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:44.499 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:44 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2708601266' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:36:44.499 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:44 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:36:44.499 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:44 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2708601266' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:36:44.499 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:44 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:36:44.577 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:36:44.582 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:44.607 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:44.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:44.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:44.709 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:44.735 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:44.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:44.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:44.837 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:44.888 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:44.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:44.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:44.990 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:45.027 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:45.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:45.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:45.129 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:45.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:45.174 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:45.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:45.275 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:45.304 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:45.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:45.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:45.406 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:45.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:45 smithi053 ceph-mon[89963]: pgmap v931: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 816 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 689 op/s; 19559/39057 objects degraded (50.078%); 0 B/s, 28 objects/s recovering 2024-08-17T17:36:45.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:45 smithi053 ceph-mon[89963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:36:45.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:45 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:36:45.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:45 smithi053 ceph-mon[89963]: osdmap e296: 8 total, 8 up, 8 in 2024-08-17T17:36:45.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:45 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 19559/39057 objects degraded (50.078%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:45.477 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:45.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:45.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:45.506 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:36:45.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:45 smithi003 ceph-mon[100615]: pgmap v931: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 816 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 689 op/s; 19559/39057 objects degraded (50.078%); 0 B/s, 28 objects/s recovering 2024-08-17T17:36:45.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:45 smithi003 ceph-mon[100615]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:36:45.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:45 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:36:45.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:45 smithi003 ceph-mon[100615]: osdmap e296: 8 total, 8 up, 8 in 2024-08-17T17:36:45.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:45 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 19559/39057 objects degraded (50.078%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:45.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:45 smithi003 ceph-mon[103344]: pgmap v931: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 816 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 689 op/s; 19559/39057 objects degraded (50.078%); 0 B/s, 28 objects/s recovering 2024-08-17T17:36:45.525 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:45 smithi003 ceph-mon[103344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:36:45.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:45 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:36:45.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:45 smithi003 ceph-mon[103344]: osdmap e296: 8 total, 8 up, 8 in 2024-08-17T17:36:45.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:45 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 19559/39057 objects degraded (50.078%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:45.578 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:45.604 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:45.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:45.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:45.706 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:45.748 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:45.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:45.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:45.850 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:45.892 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:45.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:45.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:45.994 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:46.024 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:36:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:36:45] "GET /metrics HTTP/1.1" 200 36506 "" "Prometheus/2.51.0" 2024-08-17T17:36:46.035 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:46.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:46.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:46.137 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:46.176 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:46.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:46.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:46.246 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.7 to 1 (10000) 2024-08-17T17:36:46.278 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:46.326 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:46.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:46.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:46.431 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:46.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:46.470 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:46.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:46.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:46 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/391610451' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:36:46.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:46 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/391610451' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:36:46.571 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:46.573 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:46 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/391610451' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:36:46.610 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:36:46.610 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:36:46.614 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:46.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:46.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:46.715 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:46.752 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:46.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:46.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:46.854 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:46.889 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:46.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:46.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:46.991 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:47.016 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:47.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:47.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:47.118 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:47.165 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:47.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:47.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:47.200 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:36:47.267 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:47.306 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-17T17:36:47.309 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:47.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:47.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:47.411 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:47.456 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:47.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:47.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:47.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:47 smithi003 ceph-mon[103344]: pgmap v933: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 788 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 704 op/s; 19559/37668 objects degraded (51.925%); 0 B/s, 29 objects/s recovering 2024-08-17T17:36:47.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:47 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/391610451' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-17T17:36:47.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:47 smithi003 ceph-mon[103344]: osdmap e297: 8 total, 8 up, 8 in 2024-08-17T17:36:47.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:47 smithi003 ceph-mon[100615]: pgmap v933: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 788 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 704 op/s; 19559/37668 objects degraded (51.925%); 0 B/s, 29 objects/s recovering 2024-08-17T17:36:47.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:47 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/391610451' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-17T17:36:47.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:47 smithi003 ceph-mon[100615]: osdmap e297: 8 total, 8 up, 8 in 2024-08-17T17:36:47.559 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:47.572 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:47 smithi053 ceph-mon[89963]: pgmap v933: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 788 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 704 op/s; 19559/37668 objects degraded (51.925%); 0 B/s, 29 objects/s recovering 2024-08-17T17:36:47.572 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:47 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/391610451' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-17T17:36:47.573 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:47 smithi053 ceph-mon[89963]: osdmap e297: 8 total, 8 up, 8 in 2024-08-17T17:36:47.591 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:47.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:47.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:47.681 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:36:47.693 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:47.742 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:47.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:47.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:47.844 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:47.879 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:47.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:47.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:47.981 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:48.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:48.023 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:48.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:48.124 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:48.159 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:48.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:48.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:48.261 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:48.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:48.306 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:48.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:48.407 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:48.439 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:48.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:48.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:48.541 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:48.584 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:48.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:48.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:48.686 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:48.711 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:48.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:48.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:48.813 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:48.839 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:48.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:48.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:48.941 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:48.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:48.978 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:48.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:49.079 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:49.108 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:49.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:49.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:49.210 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:49.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:49.264 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:49.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:49.365 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:49.389 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:49.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:49.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:49.392 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:49 smithi053 ceph-mon[89963]: pgmap v935: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 788 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 492 op/s; 19559/37668 objects degraded (51.925%); 0 B/s, 18 objects/s recovering 2024-08-17T17:36:49.491 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:49.518 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:36:49.518 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:36:49.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:49 smithi003 ceph-mon[103344]: pgmap v935: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 788 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 492 op/s; 19559/37668 objects degraded (51.925%); 0 B/s, 18 objects/s recovering 2024-08-17T17:36:49.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:49 smithi003 ceph-mon[100615]: pgmap v935: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 788 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 492 op/s; 19559/37668 objects degraded (51.925%); 0 B/s, 18 objects/s recovering 2024-08-17T17:36:49.529 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:49.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:49.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:49.631 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:49.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:49.661 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:49.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:49.762 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:49.790 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:49.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:49.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:49.892 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:49.920 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:49.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:49.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:50.022 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:50.054 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:50.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:50.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:50.155 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:50.189 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-17T17:36:50.189 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:50.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:50.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:50.292 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:50.326 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:50.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:50.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:50.428 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:50.457 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:50.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:50.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:50.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:50 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 19559/37668 objects degraded (51.925%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:50.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:36:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:36:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:36:50.559 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:50.590 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:50.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:50.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:50.640 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:36:50.692 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:50.722 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:50.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:50.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:50.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:50 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 19559/37668 objects degraded (51.925%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:50.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:50 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 19559/37668 objects degraded (51.925%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:50.824 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:50.851 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:50.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:50.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:50.953 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:50.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:50.984 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:50.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:51.085 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:51.120 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:51.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:51.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:51.221 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:51.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:51.250 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:51.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:51.351 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:51.386 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:51.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:51.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:51.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:51 smithi053 ceph-mon[89963]: pgmap v936: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 596 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 820 op/s; 19410/28491 objects degraded (68.127%); 0 B/s, 37 objects/s recovering 2024-08-17T17:36:51.488 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:51.517 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:51.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:51.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:51.620 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:51.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:51.665 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:51.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:51.766 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:51.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:51 smithi003 ceph-mon[100615]: pgmap v936: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 596 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 820 op/s; 19410/28491 objects degraded (68.127%); 0 B/s, 37 objects/s recovering 2024-08-17T17:36:51.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:51 smithi003 ceph-mon[103344]: pgmap v936: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 596 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 820 op/s; 19410/28491 objects degraded (68.127%); 0 B/s, 37 objects/s recovering 2024-08-17T17:36:51.812 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:51.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:51.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:51.913 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:51.952 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:51.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:51.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:52.054 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:52.083 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:52.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:52.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:52.185 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:52.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:52.229 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:52.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:52.330 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:52.363 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:52.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:52.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:52.368 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 4 1 2024-08-17T17:36:52.461 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:36:52.462 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:36:52.465 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:52.498 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:52.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:52.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:52.600 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:52.657 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:36:52.675 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:52.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:52.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:52.777 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:52.803 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:52.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:52.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:52.905 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:52.991 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:52.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:52.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:53.093 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:53.133 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:53.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:53.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:53.169 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-17T17:36:53.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:53.278 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:53.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:53.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:53.379 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:53.434 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:53.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:53.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:53.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:53 smithi053 ceph-mon[89963]: pgmap v937: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 570 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 492 op/s; 19410/27243 objects degraded (71.248%); 0 B/s, 18 objects/s recovering 2024-08-17T17:36:53.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:36:53.528 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:36:53.536 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:53.555 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:53 smithi003 ceph-mon[103344]: pgmap v937: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 570 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 492 op/s; 19410/27243 objects degraded (71.248%); 0 B/s, 18 objects/s recovering 2024-08-17T17:36:53.555 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:36:53.556 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:53 smithi003 ceph-mon[100615]: pgmap v937: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 570 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 492 op/s; 19410/27243 objects degraded (71.248%); 0 B/s, 18 objects/s recovering 2024-08-17T17:36:53.556 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:36:53.557 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:36:53.557 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:36:53.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:53.585 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:53.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:53.604 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:36:53.605 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:36:53.648 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:36:53.648 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:36:53.686 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:53.687 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:36:53.687 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:36:53.723 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:36:53.723 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:36:53.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:53.751 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:53.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:53.754 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:36:53.754 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:36:53.786 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:36:53.787 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:36:53.817 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:36:53.818 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:36:53.851 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:53.895 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:53.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:53.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:53.997 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:54.044 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:54.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:54.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:54.146 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:54.173 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:54.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:54.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:54.275 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:54.318 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:54.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:54.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:54.420 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:54.445 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:54.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:54.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:54.547 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:54.604 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:54.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:54.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:54.686 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:54 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2876194938' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:36:54.691 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:36:54.706 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:54.738 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:54.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:54.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:54.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:54 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2876194938' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:36:54.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:54 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2876194938' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:36:54.841 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:54.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:54.883 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:54.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:54.985 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:55.027 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:55.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:55.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:55.129 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:55.176 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:55.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:55.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:55.278 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:55.307 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:36:55.307 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2024-08-17T17:36:55.308 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 0 2024-08-17T17:36:55.308 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:36:55.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:55.310 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:55.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:55.411 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:55.443 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:55.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:55.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:55.463 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:55 smithi003 ceph-mon[103344]: pgmap v938: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 435 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 614 op/s; 19266/20748 objects degraded (92.857%); 0 B/s, 29 objects/s recovering 2024-08-17T17:36:55.463 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:55 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 19266/20748 objects degraded (92.857%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:55.463 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:55 smithi003 ceph-mon[100615]: pgmap v938: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 435 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 614 op/s; 19266/20748 objects degraded (92.857%); 0 B/s, 29 objects/s recovering 2024-08-17T17:36:55.463 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:55 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 19266/20748 objects degraded (92.857%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:55.545 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:55.588 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:55.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:55.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:55.685 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:55 smithi053 ceph-mon[89963]: pgmap v938: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 435 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 614 op/s; 19266/20748 objects degraded (92.857%); 0 B/s, 29 objects/s recovering 2024-08-17T17:36:55.685 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:55 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 19266/20748 objects degraded (92.857%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:36:55.690 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:36:55.727 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:36:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:36:55] "GET /metrics HTTP/1.1" 200 36506 "" "Prometheus/2.51.0" 2024-08-17T17:36:55.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:55.730 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:55.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:36:55.831 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:55.876 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:55.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:55.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:55.978 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:56.010 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:56.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:56.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:56.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:56.156 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:56.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:56.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:56.258 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:56.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:56.299 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:56.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:56.400 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:56.438 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:56.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:56.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:56.540 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:56.574 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:56.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:56.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:56.677 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:56.711 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:56.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:56.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:56.813 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:56.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:56.866 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:56.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:56.967 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:57.003 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:57.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:57.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:57.104 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:57.148 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:57.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:57.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:57.165 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:36:57.250 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:57.317 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:57.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:57.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:57.419 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:57.447 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:57.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:57.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:57.549 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:57.614 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:57.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:57.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:57.716 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:57.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:57 smithi053 ceph-mon[89963]: pgmap v939: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 356 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 690 op/s; 19266/16944 objects degraded (113.704%); 0 B/s, 29 objects/s recovering 2024-08-17T17:36:57.744 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:57.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:57.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:57.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:57 smithi003 ceph-mon[103344]: pgmap v939: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 356 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 690 op/s; 19266/16944 objects degraded (113.704%); 0 B/s, 29 objects/s recovering 2024-08-17T17:36:57.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:57 smithi003 ceph-mon[100615]: pgmap v939: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 356 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 690 op/s; 19266/16944 objects degraded (113.704%); 0 B/s, 29 objects/s recovering 2024-08-17T17:36:57.846 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:57.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:57.918 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:57.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:58.019 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:36:58.045 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:58.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:58.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:36:58.147 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:58.205 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:58.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:58.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:58.251 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-17T17:36:58.307 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:58.333 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-08-17T17:36:58.365 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:58.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:58.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:58.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:58 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:36:58.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:58 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/540098140' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:36:58.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:58 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:36:58.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:58 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/540098140' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:36:58.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:58 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:36:58.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:58 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/540098140' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:36:58.468 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:36:58.511 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:58.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:58.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:36:58.613 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:36:58.658 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:58.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:58.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:36:58.760 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:58.820 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:58.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:58.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:58.921 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:58.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:58.958 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:58.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:59.059 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:59.097 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:59.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:59.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:59.200 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:59.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:59.309 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:59.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:59.325 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:36:59.327 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:36:59.361 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.4 to 1 (10000) 2024-08-17T17:36:59.409 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:36:59.455 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:59.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:59.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:36:59.557 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:36:59.580 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:59 smithi053 ceph-mon[89963]: pgmap v940: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 356 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 579 op/s; 19266/16944 objects degraded (113.704%); 0 B/s, 24 objects/s recovering 2024-08-17T17:36:59.580 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:59 smithi053 ceph-mon[89963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:36:59.580 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:59 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/540098140' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:36:59.580 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:59 smithi053 ceph-mon[89963]: osdmap e298: 8 total, 8 up, 8 in 2024-08-17T17:36:59.580 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:59 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1135135502' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:36:59.580 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:36:59 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:36:59.596 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:59.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:59.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:36:59.698 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:36:59.733 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:59 smithi003 ceph-mon[100615]: pgmap v940: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 356 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 579 op/s; 19266/16944 objects degraded (113.704%); 0 B/s, 24 objects/s recovering 2024-08-17T17:36:59.733 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:59 smithi003 ceph-mon[100615]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:36:59.733 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:59 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/540098140' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:36:59.733 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:59 smithi003 ceph-mon[100615]: osdmap e298: 8 total, 8 up, 8 in 2024-08-17T17:36:59.733 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:59 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1135135502' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:36:59.734 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:36:59 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:36:59.734 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:59 smithi003 ceph-mon[103344]: pgmap v940: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 356 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 579 op/s; 19266/16944 objects degraded (113.704%); 0 B/s, 24 objects/s recovering 2024-08-17T17:36:59.734 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:59 smithi003 ceph-mon[103344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:36:59.734 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:59 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/540098140' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:36:59.734 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:59 smithi003 ceph-mon[103344]: osdmap e298: 8 total, 8 up, 8 in 2024-08-17T17:36:59.734 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:59 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1135135502' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:36:59.734 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:36:59 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:36:59.745 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:36:59.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:59.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:36:59.847 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:36:59.872 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:36:59.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:36:59.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:36:59.974 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:00.014 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:00.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:00.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:00.117 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:00.155 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:00.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:00.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:00.256 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:00.291 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:00.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:00.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:00.393 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:00.441 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:00.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:00.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:00.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:37:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:37:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:37:00.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:00 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-17T17:37:00.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:00 smithi053 ceph-mon[89963]: osdmap e299: 8 total, 8 up, 8 in 2024-08-17T17:37:00.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:00 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 19266/16944 objects degraded (113.704%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:37:00.543 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:00.583 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:00.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:00.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:00.685 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:00.710 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:00.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:00.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:00.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:00 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-17T17:37:00.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:00 smithi003 ceph-mon[103344]: osdmap e299: 8 total, 8 up, 8 in 2024-08-17T17:37:00.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:00 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 19266/16944 objects degraded (113.704%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:37:00.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:00 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-17T17:37:00.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:00 smithi003 ceph-mon[100615]: osdmap e299: 8 total, 8 up, 8 in 2024-08-17T17:37:00.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:00 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 19266/16944 objects degraded (113.704%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:37:00.812 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:00.849 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:00.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:00.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:00.874 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-08-17T17:37:00.874 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:37:00.951 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:00.975 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:00.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:00.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:01.077 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:01.136 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:01.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:01.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:01.202 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:37:01.238 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:01.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:01.292 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:01.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:01.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:01 smithi003 ceph-mon[100615]: pgmap v943: 99 pgs: 2 peering, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 180 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 780 op/s; 19111/8505 objects degraded (224.703%); 0 B/s, 37 objects/s recovering 2024-08-17T17:37:01.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:01 smithi003 ceph-mon[100615]: osdmap e300: 8 total, 8 up, 8 in 2024-08-17T17:37:01.393 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:01.422 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:01.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:01.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:01.494 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:01 smithi053 ceph-mon[89963]: pgmap v943: 99 pgs: 2 peering, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 180 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 780 op/s; 19111/8505 objects degraded (224.703%); 0 B/s, 37 objects/s recovering 2024-08-17T17:37:01.494 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:01 smithi053 ceph-mon[89963]: osdmap e300: 8 total, 8 up, 8 in 2024-08-17T17:37:01.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:01.573 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:01.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:01.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:01.675 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:01.690 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:01 smithi003 ceph-mon[103344]: pgmap v943: 99 pgs: 2 peering, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 180 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 780 op/s; 19111/8505 objects degraded (224.703%); 0 B/s, 37 objects/s recovering 2024-08-17T17:37:01.690 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:01 smithi003 ceph-mon[103344]: osdmap e300: 8 total, 8 up, 8 in 2024-08-17T17:37:01.726 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:01.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:01.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:01.828 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:01.878 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:01.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:01.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:01.979 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:02.024 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:02.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:02.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:02.126 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:02.184 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:02.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:02.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:02.286 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:02.336 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:02.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:02.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:02.439 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:02.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:02.481 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:02.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:02.505 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:37:02.505 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2024-08-17T17:37:02.506 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2024-08-17T17:37:02.506 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2024-08-17T17:37:02.506 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:37:02.581 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:02.634 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:02.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:02.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:02.716 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:37:02.716 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:37:02.729 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:37:02.736 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:02.760 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:37:02.760 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:37:02.762 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:02.762 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:37:02.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:02.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:02.768 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Removed 107472 objects 2024-08-17T17:37:02.769 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Clean up completed and total clean up time :155.385 2024-08-17T17:37:02.790 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:37:02.790 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:37:02.811 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:37:02.811 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:37:02.830 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:37:02.830 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:37:02.850 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:37:02.851 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:37:02.864 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:02.870 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:37:02.871 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:37:02.886 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:02.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:02.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:02.888 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:37:02.888 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:37:02.988 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:03.012 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:03.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:03.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:03.114 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:03.136 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:03.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:03.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:03.159 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-17T17:37:03.238 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:03.262 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:03.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:03.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:03.364 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:03.388 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:03.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:03.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:03.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:03 smithi053 ceph-mon[89963]: pgmap v945: 99 pgs: 2 peering, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 155 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 534 op/s; 19111/7314 objects degraded (261.293%); 0 B/s, 25 objects/s recovering 2024-08-17T17:37:03.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:03 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2502323969' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:37:03.490 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:03.513 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:03.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:03.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:03.536 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:37:03.615 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:03.638 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:03.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:03.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:03.740 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:03.763 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:03.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:03.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:03.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:03 smithi003 ceph-mon[103344]: pgmap v945: 99 pgs: 2 peering, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 155 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 534 op/s; 19111/7314 objects degraded (261.293%); 0 B/s, 25 objects/s recovering 2024-08-17T17:37:03.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:03 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2502323969' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:37:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:03 smithi003 ceph-mon[100615]: pgmap v945: 99 pgs: 2 peering, 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 155 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 534 op/s; 19111/7314 objects degraded (261.293%); 0 B/s, 25 objects/s recovering 2024-08-17T17:37:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:03 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2502323969' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:37:03.864 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:03.888 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:03.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:03.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:03.990 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:04.013 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:04.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:04.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:04.114 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:04.137 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:04.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:04.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:04.238 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:04.261 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:04.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:04.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:04.363 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:04.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:04.387 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:04.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:04.488 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:04.510 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:04.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:04.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:04.612 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:04.635 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:04.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:04.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:04.736 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:04.759 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:04.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:04.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:04.861 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:04.884 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:04.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:04.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:04.986 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:05.009 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:05.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:05.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:05.111 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:05.136 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:05.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:05.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:05.237 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:05.260 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:05.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:05.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:05.362 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:05.386 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:05.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:05.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:05.487 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:05.510 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:05.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:05.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:05.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:05 smithi003 ceph-mon[103344]: pgmap v946: 99 pgs: 2 peering, 6 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 55 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 802 op/s; 19065/2499 objects degraded (762.905%); 0 B/s, 33 objects/s recovering 2024-08-17T17:37:05.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:05 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 19065/2499 objects degraded (762.905%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:37:05.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:05 smithi003 ceph-mon[100615]: pgmap v946: 99 pgs: 2 peering, 6 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 55 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 802 op/s; 19065/2499 objects degraded (762.905%); 0 B/s, 33 objects/s recovering 2024-08-17T17:37:05.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:05 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 19065/2499 objects degraded (762.905%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:37:05.611 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:05.635 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:05.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:05.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:05.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:05 smithi053 ceph-mon[89963]: pgmap v946: 99 pgs: 2 peering, 6 active+recovery_wait+undersized+degraded+remapped, 91 active+clean; 55 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 802 op/s; 19065/2499 objects degraded (762.905%); 0 B/s, 33 objects/s recovering 2024-08-17T17:37:05.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:05 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 19065/2499 objects degraded (762.905%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:37:05.737 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:05.759 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:05.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:05.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:05.861 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:05.883 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:05.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:05.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:05.985 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:06.009 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:06.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:06.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:06.022 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:37:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:37:05] "GET /metrics HTTP/1.1" 200 36506 "" "Prometheus/2.51.0" 2024-08-17T17:37:06.101 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:37:06.101 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:37:06.111 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:06.135 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:06.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:06.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:06.237 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:06.258 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:06.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:06.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:06.362 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:06.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:06.386 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:06.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:06.404 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-08-17T17:37:06.404 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-08-17T17:37:06.488 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:06.510 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:06.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:06.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:06.611 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:06.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:06.634 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:06.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:06.705 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-17T17:37:06.730 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:06 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:37:06.730 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:06 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:37:06.730 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:06 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:37:06.730 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:06 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:37:06.735 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:06.756 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:06.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:06.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:06.858 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:06.881 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:06.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:06.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:06.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:06 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:37:06.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:06 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:37:06.983 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:07.007 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:07.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:07.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:07.109 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:07.132 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:07.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:07.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:07.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:07.256 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:07.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:07.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:07.358 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:07.381 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:07.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:07.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:07.483 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:07.509 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:07.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:07.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:07.611 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:07.633 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:07.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:07.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:07.735 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:07.760 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:07.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:07.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:07.862 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:07.885 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:07.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:07.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:07.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:07 smithi053 ceph-mon[89963]: pgmap v947: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 716 op/s; 18803/84 objects degraded (22384.524%); 4 B/s, 1 keys/s, 59 objects/s recovering 2024-08-17T17:37:07.987 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:08.013 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:08.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:08.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:08.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:07 smithi003 ceph-mon[103344]: pgmap v947: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 716 op/s; 18803/84 objects degraded (22384.524%); 4 B/s, 1 keys/s, 59 objects/s recovering 2024-08-17T17:37:08.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:07 smithi003 ceph-mon[100615]: pgmap v947: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 716 op/s; 18803/84 objects degraded (22384.524%); 4 B/s, 1 keys/s, 59 objects/s recovering 2024-08-17T17:37:08.115 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:08.140 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:08.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:08.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:08.242 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:08.269 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:08.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:08.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:08.370 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:08.393 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:08.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:08.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:08.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:08.519 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:08.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:08.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:08.621 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:08.645 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:08.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:08.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:08.747 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:08.769 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:08.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:08.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:08.871 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:08.898 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:08.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:08.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:08.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:37:09.000 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:09.002 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:37:09.003 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:37:09.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:09.027 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:09.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:09.128 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:09.156 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:09.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:09.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:09.258 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:09.281 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:09.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:09.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:09.383 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:09.406 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:09.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:09.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:09.508 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:09.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:09.538 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:09.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:09.639 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:09.662 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:09.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:09.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:09.764 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:09.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:09 smithi003 ceph-mon[100615]: pgmap v948: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 318 op/s; 18803/84 objects degraded (22384.524%); 3 B/s, 1 keys/s, 35 objects/s recovering 2024-08-17T17:37:09.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:09 smithi003 ceph-mon[103344]: pgmap v948: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 318 op/s; 18803/84 objects degraded (22384.524%); 3 B/s, 1 keys/s, 35 objects/s recovering 2024-08-17T17:37:09.791 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:09.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:09.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:09.893 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:09.920 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:09.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:09.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:09.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:09 smithi053 ceph-mon[89963]: pgmap v948: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 318 op/s; 18803/84 objects degraded (22384.524%); 3 B/s, 1 keys/s, 35 objects/s recovering 2024-08-17T17:37:10.022 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:10.045 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:10.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:10.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:10.147 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:10.170 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:10.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:10.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:10.272 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:10.300 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:10.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:10.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:10.401 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:10.427 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:10.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:10.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:10.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:37:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:37:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:37:10.529 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:10.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:10.557 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:10.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:10.658 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:10.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:10.682 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:10.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:10.783 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:10.811 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:10.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:10.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:10.913 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:10.940 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:10.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:10.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:10.950 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:10 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 18803/84 objects degraded (22384.524%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:37:10.950 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:10 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 18803/84 objects degraded (22384.524%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:37:10.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:10 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 18803/84 objects degraded (22384.524%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:37:11.042 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:11.068 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:11.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:11.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:11.169 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:11.197 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:11.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:11.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:11.299 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:11.323 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:11.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:11.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:11.426 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:11.448 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:11.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:11.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:11.550 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:11.573 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:11.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:11.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:11.675 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:11.702 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:11.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:11.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:11.804 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:11.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:11.831 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:11.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:11.932 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:11.958 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:11.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:11.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:11.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:11 smithi053 ceph-mon[89963]: pgmap v949: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 280 op/s; 18027/84 objects degraded (21460.714%); 3 B/s, 0 keys/s, 108 objects/s recovering 2024-08-17T17:37:12.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:11 smithi003 ceph-mon[103344]: pgmap v949: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 280 op/s; 18027/84 objects degraded (21460.714%); 3 B/s, 0 keys/s, 108 objects/s recovering 2024-08-17T17:37:12.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:11 smithi003 ceph-mon[100615]: pgmap v949: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 280 op/s; 18027/84 objects degraded (21460.714%); 3 B/s, 0 keys/s, 108 objects/s recovering 2024-08-17T17:37:12.060 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:12.084 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:12.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:12.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:12.186 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:12.211 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:12.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:12.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:12.243 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:37:12.244 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:37:12.313 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:12.336 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:12.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:12.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:12.399 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:37:12.438 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:12.465 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:12.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:12.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:12.567 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:12.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:12.595 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:12.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:12.695 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:12.723 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:12.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:12.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:12.825 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:12.851 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:12.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:12.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:12.953 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:12.980 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:12.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:12.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:13.081 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:13.108 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:13.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:13.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:13.210 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:13.235 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:13.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:13.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:13.337 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:13.360 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:13.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:13.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:13.382 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":950,"stamp":"2024-08-17T17:37:12.176336+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790221,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":5979,"num_read_kb":12992,"num_write":215853,"num_write_kb":6890832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7042,"num_bytes_recovered":271325260,"num_keys_recovered":20,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130182,"ondisk_log_size":130182,"up":297,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":306,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":4814740,"kb_used_data":45140,"kb_used_omap":37,"kb_used_meta":4769498,"kb_avail":745146476,"statfs":{"total":767960285184,"available":763029991424,"internally_reserved":0,"allocated":46223360,"data_stored":32364703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38729,"internal_metadata":4883966135},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-157876248,"num_objects":-2410,"num_object_clones":0,"num_object_copies":-7230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2410,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1087,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.810653"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":469,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.378322+0000","last_change":"2024-08-17T17:33:20.797029+0000","last_active":"2024-08-17T17:37:00.378322+0000","last_peered":"2024-08-17T17:37:00.378322+0000","last_clean":"2024-08-17T17:37:00.378322+0000","last_became_active":"2024-08-17T17:33:20.796679+0000","last_became_peered":"2024-08-17T17:33:20.796679+0000","last_unstale":"2024-08-17T17:37:00.378322+0000","last_undegraded":"2024-08-17T17:37:00.378322+0000","last_fullsized":"2024-08-17T17:37:00.378322+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:51.438262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":292,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.420764+0000","last_change":"2024-08-17T17:37:00.420764+0000","last_active":"2024-08-17T17:37:00.420764+0000","last_peered":"2024-08-17T17:37:00.420764+0000","last_clean":"2024-08-17T17:37:00.420764+0000","last_became_active":"2024-08-17T17:37:00.387450+0000","last_became_peered":"2024-08-17T17:37:00.387450+0000","last_unstale":"2024-08-17T17:37:00.420764+0000","last_undegraded":"2024-08-17T17:37:00.420764+0000","last_fullsized":"2024-08-17T17:37:00.420764+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:09:59.462151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":130,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.377613+0000","last_change":"2024-08-17T17:33:44.156036+0000","last_active":"2024-08-17T17:37:00.377613+0000","last_peered":"2024-08-17T17:37:00.377613+0000","last_clean":"2024-08-17T17:37:00.377613+0000","last_became_active":"2024-08-17T17:33:44.155758+0000","last_became_peered":"2024-08-17T17:33:44.155758+0000","last_unstale":"2024-08-17T17:37:00.377613+0000","last_undegraded":"2024-08-17T17:37:00.377613+0000","last_fullsized":"2024-08-17T17:37:00.377613+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:23.443405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":269,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.390756+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:37:00.390756+0000","last_peered":"2024-08-17T17:37:00.390756+0000","last_clean":"2024-08-17T17:37:00.390756+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:37:00.390756+0000","last_undegraded":"2024-08-17T17:37:00.390756+0000","last_fullsized":"2024-08-17T17:37:00.390756+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":269,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.388436+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:37:00.388436+0000","last_peered":"2024-08-17T17:37:00.388436+0000","last_clean":"2024-08-17T17:37:00.388436+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:37:00.388436+0000","last_undegraded":"2024-08-17T17:37:00.388436+0000","last_fullsized":"2024-08-17T17:37:00.388436+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":49,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-17T17:36:59.355356+0000","last_change":"2024-08-17T17:32:46.377651+0000","last_active":"2024-08-17T17:36:59.355356+0000","last_peered":"2024-08-17T17:36:59.355356+0000","last_clean":"2024-08-17T17:36:59.355356+0000","last_became_active":"2024-08-17T17:32:46.377282+0000","last_became_peered":"2024-08-17T17:32:46.377282+0000","last_unstale":"2024-08-17T17:36:59.355356+0000","last_undegraded":"2024-08-17T17:36:59.355356+0000","last_fullsized":"2024-08-17T17:36:59.355356+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:45:27.370287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":483,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.381763+0000","last_change":"2024-08-17T17:33:20.794682+0000","last_active":"2024-08-17T17:37:00.381763+0000","last_peered":"2024-08-17T17:37:00.381763+0000","last_clean":"2024-08-17T17:37:00.381763+0000","last_became_active":"2024-08-17T17:33:20.794238+0000","last_became_peered":"2024-08-17T17:33:20.794238+0000","last_unstale":"2024-08-17T17:37:00.381763+0000","last_undegraded":"2024-08-17T17:37:00.381763+0000","last_fullsized":"2024-08-17T17:37:00.381763+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:15:55.821920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":346,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-17T17:36:59.359277+0000","last_change":"2024-08-17T17:24:53.465875+0000","last_active":"2024-08-17T17:36:59.359277+0000","last_peered":"2024-08-17T17:36:59.359277+0000","last_clean":"2024-08-17T17:36:59.359277+0000","last_became_active":"2024-08-17T17:24:53.458129+0000","last_became_peered":"2024-08-17T17:24:53.458129+0000","last_unstale":"2024-08-17T17:36:59.359277+0000","last_undegraded":"2024-08-17T17:36:59.359277+0000","last_fullsized":"2024-08-17T17:36:59.359277+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:51:33.584869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"300'13302","reported_seq":13357,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:02.750749+0000","last_change":"2024-08-17T17:33:44.165390+0000","last_active":"2024-08-17T17:37:02.750749+0000","last_peered":"2024-08-17T17:37:02.750749+0000","last_clean":"2024-08-17T17:37:02.750749+0000","last_became_active":"2024-08-17T17:33:44.144942+0000","last_became_peered":"2024-08-17T17:33:44.144942+0000","last_unstale":"2024-08-17T17:37:02.750749+0000","last_undegraded":"2024-08-17T17:37:02.750749+0000","last_fullsized":"2024-08-17T17:37:02.750749+0000","mapping_epoch":272,"log_start":"274'5309","ondisk_log_start":"274'5309","created":262,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":7993,"log_dups_size":0,"ondisk_log_size":7993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:49:11.850677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13302,"num_write_kb":425664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":233,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-17T17:36:59.365603+0000","last_change":"2024-08-17T17:32:46.384376+0000","last_active":"2024-08-17T17:36:59.365603+0000","last_peered":"2024-08-17T17:36:59.365603+0000","last_clean":"2024-08-17T17:36:59.365603+0000","last_became_active":"2024-08-17T17:32:46.374177+0000","last_became_peered":"2024-08-17T17:32:46.374177+0000","last_unstale":"2024-08-17T17:36:59.365603+0000","last_undegraded":"2024-08-17T17:36:59.365603+0000","last_fullsized":"2024-08-17T17:36:59.365603+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:51:58.880388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":55,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-17T17:36:59.359069+0000","last_change":"2024-08-17T17:33:44.145598+0000","last_active":"2024-08-17T17:36:59.359069+0000","last_peered":"2024-08-17T17:36:59.359069+0000","last_clean":"2024-08-17T17:36:59.359069+0000","last_became_active":"2024-08-17T17:33:44.145283+0000","last_became_peered":"2024-08-17T17:33:44.145283+0000","last_unstale":"2024-08-17T17:36:59.359069+0000","last_undegraded":"2024-08-17T17:36:59.359069+0000","last_fullsized":"2024-08-17T17:36:59.359069+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:11.965226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.5","version":"300'13218","reported_seq":13264,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:02.583954+0000","last_change":"2024-08-17T17:32:48.380789+0000","last_active":"2024-08-17T17:37:02.583954+0000","last_peered":"2024-08-17T17:37:02.583954+0000","last_clean":"2024-08-17T17:37:02.583954+0000","last_became_active":"2024-08-17T17:32:48.380173+0000","last_became_peered":"2024-08-17T17:32:48.380173+0000","last_unstale":"2024-08-17T17:37:02.583954+0000","last_undegraded":"2024-08-17T17:37:02.583954+0000","last_fullsized":"2024-08-17T17:37:02.583954+0000","mapping_epoch":262,"log_start":"278'6300","ondisk_log_start":"278'6300","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":6918,"log_dups_size":0,"ondisk_log_size":6918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:25:18.190497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13218,"num_write_kb":422976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":229,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.416059+0000","last_change":"2024-08-17T17:37:00.416059+0000","last_active":"2024-08-17T17:37:00.416059+0000","last_peered":"2024-08-17T17:37:00.416059+0000","last_clean":"2024-08-17T17:37:00.416059+0000","last_became_active":"2024-08-17T17:37:00.385913+0000","last_became_peered":"2024-08-17T17:37:00.385913+0000","last_unstale":"2024-08-17T17:37:00.416059+0000","last_undegraded":"2024-08-17T17:37:00.416059+0000","last_fullsized":"2024-08-17T17:37:00.416059+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:11:24.884146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":63,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-17T17:36:59.355434+0000","last_change":"2024-08-17T17:33:21.766549+0000","last_active":"2024-08-17T17:36:59.355434+0000","last_peered":"2024-08-17T17:36:59.355434+0000","last_clean":"2024-08-17T17:36:59.355434+0000","last_became_active":"2024-08-17T17:33:21.766152+0000","last_became_peered":"2024-08-17T17:33:21.766152+0000","last_unstale":"2024-08-17T17:36:59.355434+0000","last_undegraded":"2024-08-17T17:36:59.355434+0000","last_fullsized":"2024-08-17T17:36:59.355434+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:56:15.855524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"300'13282","reported_seq":13371,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:02.762254+0000","last_change":"2024-08-17T17:32:48.381458+0000","last_active":"2024-08-17T17:37:02.762254+0000","last_peered":"2024-08-17T17:37:02.762254+0000","last_clean":"2024-08-17T17:37:02.762254+0000","last_became_active":"2024-08-17T17:32:48.381201+0000","last_became_peered":"2024-08-17T17:32:48.381201+0000","last_unstale":"2024-08-17T17:37:02.762254+0000","last_undegraded":"2024-08-17T17:37:02.762254+0000","last_fullsized":"2024-08-17T17:37:02.762254+0000","mapping_epoch":262,"log_start":"279'6800","ondisk_log_start":"279'6800","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":6482,"log_dups_size":0,"ondisk_log_size":6482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:12:18.200526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13282,"num_write_kb":425024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":51,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-17T17:36:59.361426+0000","last_change":"2024-08-17T17:33:44.145399+0000","last_active":"2024-08-17T17:36:59.361426+0000","last_peered":"2024-08-17T17:36:59.361426+0000","last_clean":"2024-08-17T17:36:59.361426+0000","last_became_active":"2024-08-17T17:33:44.144751+0000","last_became_peered":"2024-08-17T17:33:44.144751+0000","last_unstale":"2024-08-17T17:36:59.361426+0000","last_undegraded":"2024-08-17T17:36:59.361426+0000","last_fullsized":"2024-08-17T17:36:59.361426+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:54.276350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":129,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-17T17:36:59.359401+0000","last_change":"2024-08-17T17:33:20.795582+0000","last_active":"2024-08-17T17:36:59.359401+0000","last_peered":"2024-08-17T17:36:59.359401+0000","last_clean":"2024-08-17T17:36:59.359401+0000","last_became_active":"2024-08-17T17:33:20.795149+0000","last_became_peered":"2024-08-17T17:33:20.795149+0000","last_unstale":"2024-08-17T17:36:59.359401+0000","last_undegraded":"2024-08-17T17:36:59.359401+0000","last_fullsized":"2024-08-17T17:36:59.359401+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:14.020923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"300'13416","reported_seq":13504,"reported_epoch":300,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:37:02.768379+0000","last_change":"2024-08-17T17:33:22.090332+0000","last_active":"2024-08-17T17:37:02.768379+0000","last_peered":"2024-08-17T17:37:02.768379+0000","last_clean":"2024-08-17T17:32:47.362949+0000","last_became_active":"2024-08-17T17:33:22.080130+0000","last_became_peered":"2024-08-17T17:33:22.080130+0000","last_unstale":"2024-08-17T17:37:02.768379+0000","last_undegraded":"2024-08-17T17:33:21.923200+0000","last_fullsized":"2024-08-17T17:33:21.906251+0000","mapping_epoch":269,"log_start":"273'5000","ondisk_log_start":"273'5000","created":262,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":8416,"log_dups_size":0,"ondisk_log_size":8416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3520,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13416,"num_write_kb":429312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":-3520}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":127,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.378450+0000","last_change":"2024-08-17T17:33:44.154097+0000","last_active":"2024-08-17T17:37:00.378450+0000","last_peered":"2024-08-17T17:37:00.378450+0000","last_clean":"2024-08-17T17:37:00.378450+0000","last_became_active":"2024-08-17T17:33:44.152972+0000","last_became_peered":"2024-08-17T17:33:44.152972+0000","last_unstale":"2024-08-17T17:37:00.378450+0000","last_undegraded":"2024-08-17T17:37:00.378450+0000","last_fullsized":"2024-08-17T17:37:00.378450+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:42.602017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":477,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.385414+0000","last_change":"2024-08-17T17:33:21.763677+0000","last_active":"2024-08-17T17:37:00.385414+0000","last_peered":"2024-08-17T17:37:00.385414+0000","last_clean":"2024-08-17T17:37:00.385414+0000","last_became_active":"2024-08-17T17:33:21.760564+0000","last_became_peered":"2024-08-17T17:33:21.760564+0000","last_unstale":"2024-08-17T17:37:00.385414+0000","last_undegraded":"2024-08-17T17:37:00.385414+0000","last_fullsized":"2024-08-17T17:37:00.385414+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:01:00.772037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":611,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.389021+0000","last_change":"2024-08-17T17:35:04.273750+0000","last_active":"2024-08-17T17:37:00.389021+0000","last_peered":"2024-08-17T17:37:00.389021+0000","last_clean":"2024-08-17T17:37:00.389021+0000","last_became_active":"2024-08-17T17:34:17.898604+0000","last_became_peered":"2024-08-17T17:34:17.898604+0000","last_unstale":"2024-08-17T17:37:00.389021+0000","last_undegraded":"2024-08-17T17:37:00.389021+0000","last_fullsized":"2024-08-17T17:37:00.389021+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:35:04.265381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:35:04.265381+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:32:27.625196+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.0","version":"300'13370","reported_seq":13457,"reported_epoch":300,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:37:02.619430+0000","last_change":"2024-08-17T17:33:22.010282+0000","last_active":"2024-08-17T17:37:02.619430+0000","last_peered":"2024-08-17T17:37:02.619430+0000","last_clean":"2024-08-17T17:32:47.362949+0000","last_became_active":"2024-08-17T17:33:21.998714+0000","last_became_peered":"2024-08-17T17:33:21.998714+0000","last_unstale":"2024-08-17T17:37:02.619430+0000","last_undegraded":"2024-08-17T17:33:21.807105+0000","last_fullsized":"2024-08-17T17:33:21.761969+0000","mapping_epoch":269,"log_start":"273'5000","ondisk_log_start":"273'5000","created":262,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":8370,"log_dups_size":0,"ondisk_log_size":8370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3554,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13370,"num_write_kb":427840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[{"shards":"3,5","objects":-3554}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":504,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.380051+0000","last_change":"2024-08-17T17:32:46.387810+0000","last_active":"2024-08-17T17:37:00.380051+0000","last_peered":"2024-08-17T17:37:00.380051+0000","last_clean":"2024-08-17T17:37:00.380051+0000","last_became_active":"2024-08-17T17:32:46.374238+0000","last_became_peered":"2024-08-17T17:32:46.374238+0000","last_unstale":"2024-08-17T17:37:00.380051+0000","last_undegraded":"2024-08-17T17:37:00.380051+0000","last_fullsized":"2024-08-17T17:37:00.380051+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:53:29.918005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":49,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-17T17:36:59.356376+0000","last_change":"2024-08-17T17:32:46.378495+0000","last_active":"2024-08-17T17:36:59.356376+0000","last_peered":"2024-08-17T17:36:59.356376+0000","last_clean":"2024-08-17T17:36:59.356376+0000","last_became_active":"2024-08-17T17:32:46.377709+0000","last_became_peered":"2024-08-17T17:32:46.377709+0000","last_unstale":"2024-08-17T17:36:59.356376+0000","last_undegraded":"2024-08-17T17:36:59.356376+0000","last_fullsized":"2024-08-17T17:36:59.356376+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:22:52.544851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":611,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.382325+0000","last_change":"2024-08-17T17:35:02.239838+0000","last_active":"2024-08-17T17:37:00.382325+0000","last_peered":"2024-08-17T17:37:00.382325+0000","last_clean":"2024-08-17T17:37:00.382325+0000","last_became_active":"2024-08-17T17:34:17.888626+0000","last_became_peered":"2024-08-17T17:34:17.888626+0000","last_unstale":"2024-08-17T17:37:00.382325+0000","last_undegraded":"2024-08-17T17:37:00.382325+0000","last_fullsized":"2024-08-17T17:37:00.382325+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:35:02.239705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:35:02.239705+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:43:42.111630+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"300'13680","reported_seq":15845,"reported_epoch":300,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-17T17:37:09.189475+0000","last_change":"2024-08-17T17:37:00.421932+0000","last_active":"2024-08-17T17:37:09.189475+0000","last_peered":"2024-08-17T17:37:09.189475+0000","last_clean":"2024-08-17T17:32:47.362949+0000","last_became_active":"2024-08-17T17:33:21.976559+0000","last_became_peered":"2024-08-17T17:33:21.976559+0000","last_unstale":"2024-08-17T17:37:09.189475+0000","last_undegraded":"2024-08-17T17:33:21.786755+0000","last_fullsized":"2024-08-17T17:33:21.754986+0000","mapping_epoch":269,"log_start":"279'7200","ondisk_log_start":"279'7200","created":262,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":6480,"log_dups_size":0,"ondisk_log_size":6480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2568,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13680,"num_write_kb":437760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1038,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":-2568}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":63,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-17T17:36:59.355511+0000","last_change":"2024-08-17T17:33:21.766756+0000","last_active":"2024-08-17T17:36:59.355511+0000","last_peered":"2024-08-17T17:36:59.355511+0000","last_clean":"2024-08-17T17:36:59.355511+0000","last_became_active":"2024-08-17T17:33:21.766347+0000","last_became_peered":"2024-08-17T17:33:21.766347+0000","last_unstale":"2024-08-17T17:36:59.355511+0000","last_undegraded":"2024-08-17T17:36:59.355511+0000","last_fullsized":"2024-08-17T17:36:59.355511+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:13:14.655176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":10,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.386391+0000","last_change":"2024-08-17T17:37:00.386391+0000","last_active":"2024-08-17T17:37:00.386391+0000","last_peered":"2024-08-17T17:37:00.386391+0000","last_clean":"2024-08-17T17:37:00.386391+0000","last_became_active":"2024-08-17T17:37:00.385803+0000","last_became_peered":"2024-08-17T17:37:00.385803+0000","last_unstale":"2024-08-17T17:37:00.386391+0000","last_undegraded":"2024-08-17T17:37:00.386391+0000","last_fullsized":"2024-08-17T17:37:00.386391+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:59:49.051157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.2","version":"300'13410","reported_seq":13456,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:02.706959+0000","last_change":"2024-08-17T17:32:48.378488+0000","last_active":"2024-08-17T17:37:02.706959+0000","last_peered":"2024-08-17T17:37:02.706959+0000","last_clean":"2024-08-17T17:37:02.706959+0000","last_became_active":"2024-08-17T17:32:48.378189+0000","last_became_peered":"2024-08-17T17:32:48.378189+0000","last_unstale":"2024-08-17T17:37:02.706959+0000","last_undegraded":"2024-08-17T17:37:02.706959+0000","last_fullsized":"2024-08-17T17:37:02.706959+0000","mapping_epoch":262,"log_start":"274'5426","ondisk_log_start":"274'5426","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":7984,"log_dups_size":0,"ondisk_log_size":7984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:38.041536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13410,"num_write_kb":429120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.376047+0000","last_change":"2024-08-17T17:37:00.376047+0000","last_active":"2024-08-17T17:37:00.376047+0000","last_peered":"2024-08-17T17:37:00.376047+0000","last_clean":"2024-08-17T17:37:00.376047+0000","last_became_active":"2024-08-17T17:37:00.375687+0000","last_became_peered":"2024-08-17T17:37:00.375687+0000","last_unstale":"2024-08-17T17:37:00.376047+0000","last_undegraded":"2024-08-17T17:37:00.376047+0000","last_fullsized":"2024-08-17T17:37:00.376047+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:57:17.605170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":63,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-17T17:36:59.356242+0000","last_change":"2024-08-17T17:33:21.764934+0000","last_active":"2024-08-17T17:36:59.356242+0000","last_peered":"2024-08-17T17:36:59.356242+0000","last_clean":"2024-08-17T17:36:59.356242+0000","last_became_active":"2024-08-17T17:33:21.764341+0000","last_became_peered":"2024-08-17T17:33:21.764341+0000","last_unstale":"2024-08-17T17:36:59.356242+0000","last_undegraded":"2024-08-17T17:36:59.356242+0000","last_fullsized":"2024-08-17T17:36:59.356242+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:20:25.714104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":614,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.387725+0000","last_change":"2024-08-17T17:35:09.302863+0000","last_active":"2024-08-17T17:37:00.387725+0000","last_peered":"2024-08-17T17:37:00.387725+0000","last_clean":"2024-08-17T17:37:00.387725+0000","last_became_active":"2024-08-17T17:34:17.889755+0000","last_became_peered":"2024-08-17T17:34:17.889755+0000","last_unstale":"2024-08-17T17:37:00.387725+0000","last_undegraded":"2024-08-17T17:37:00.387725+0000","last_fullsized":"2024-08-17T17:37:00.387725+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:35:09.302394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:35:09.302394+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:17.433313+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3","version":"300'13402","reported_seq":13451,"reported_epoch":300,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:37:02.746585+0000","last_change":"2024-08-17T17:33:21.990474+0000","last_active":"2024-08-17T17:37:02.746585+0000","last_peered":"2024-08-17T17:37:02.746585+0000","last_clean":"2024-08-17T17:32:47.362949+0000","last_became_active":"2024-08-17T17:33:21.980477+0000","last_became_peered":"2024-08-17T17:33:21.980477+0000","last_unstale":"2024-08-17T17:37:02.746585+0000","last_undegraded":"2024-08-17T17:33:21.778308+0000","last_fullsized":"2024-08-17T17:33:21.762700+0000","mapping_epoch":269,"log_start":"270'3700","ondisk_log_start":"270'3700","created":262,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":9702,"log_dups_size":0,"ondisk_log_size":9702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:31:39.084953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3458,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13402,"num_write_kb":428864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":-3458}],"blocked_by":[],"up_primary":1,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":485,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.382811+0000","last_change":"2024-08-17T17:33:20.796216+0000","last_active":"2024-08-17T17:37:00.382811+0000","last_peered":"2024-08-17T17:37:00.382811+0000","last_clean":"2024-08-17T17:37:00.382811+0000","last_became_active":"2024-08-17T17:33:20.795762+0000","last_became_peered":"2024-08-17T17:33:20.795762+0000","last_unstale":"2024-08-17T17:37:00.382811+0000","last_undegraded":"2024-08-17T17:37:00.382811+0000","last_fullsized":"2024-08-17T17:37:00.382811+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:47.038509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":366,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.407020+0000","last_change":"2024-08-17T17:33:21.976938+0000","last_active":"2024-08-17T17:37:00.407020+0000","last_peered":"2024-08-17T17:37:00.407020+0000","last_clean":"2024-08-17T17:37:00.407020+0000","last_became_active":"2024-08-17T17:33:21.976585+0000","last_became_peered":"2024-08-17T17:33:21.976585+0000","last_unstale":"2024-08-17T17:37:00.407020+0000","last_undegraded":"2024-08-17T17:37:00.407020+0000","last_fullsized":"2024-08-17T17:37:00.407020+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:23:40.758870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"300'13552","reported_seq":13598,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:02.722353+0000","last_change":"2024-08-17T17:32:48.378105+0000","last_active":"2024-08-17T17:37:02.722353+0000","last_peered":"2024-08-17T17:37:02.722353+0000","last_clean":"2024-08-17T17:37:02.722353+0000","last_became_active":"2024-08-17T17:32:48.377534+0000","last_became_peered":"2024-08-17T17:32:48.377534+0000","last_unstale":"2024-08-17T17:37:02.722353+0000","last_undegraded":"2024-08-17T17:37:02.722353+0000","last_fullsized":"2024-08-17T17:37:02.722353+0000","mapping_epoch":262,"log_start":"279'6600","ondisk_log_start":"279'6600","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":6952,"log_dups_size":0,"ondisk_log_size":6952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:53.372506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13552,"num_write_kb":433664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":600,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.382573+0000","last_change":"2024-08-17T17:34:18.238306+0000","last_active":"2024-08-17T17:37:00.382573+0000","last_peered":"2024-08-17T17:37:00.382573+0000","last_clean":"2024-08-17T17:37:00.382573+0000","last_became_active":"2024-08-17T17:34:17.878280+0000","last_became_peered":"2024-08-17T17:34:17.878280+0000","last_unstale":"2024-08-17T17:37:00.382573+0000","last_undegraded":"2024-08-17T17:37:00.382573+0000","last_fullsized":"2024-08-17T17:37:00.382573+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:34:18.238171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:34:18.238171+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:35:36.485730+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":706,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-17T17:36:59.363459+0000","last_change":"2024-08-17T17:33:44.149928+0000","last_active":"2024-08-17T17:36:59.363459+0000","last_peered":"2024-08-17T17:36:59.363459+0000","last_clean":"2024-08-17T17:36:59.363459+0000","last_became_active":"2024-08-17T17:33:44.149498+0000","last_became_peered":"2024-08-17T17:33:44.149498+0000","last_unstale":"2024-08-17T17:36:59.363459+0000","last_undegraded":"2024-08-17T17:36:59.363459+0000","last_fullsized":"2024-08-17T17:36:59.363459+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:17:04.963202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":432,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.377357+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:37:00.377357+0000","last_peered":"2024-08-17T17:37:00.377357+0000","last_clean":"2024-08-17T17:37:00.377357+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:37:00.377357+0000","last_undegraded":"2024-08-17T17:37:00.377357+0000","last_fullsized":"2024-08-17T17:37:00.377357+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":122,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.384064+0000","last_change":"2024-08-17T17:37:00.384064+0000","last_active":"2024-08-17T17:37:00.384064+0000","last_peered":"2024-08-17T17:37:00.384064+0000","last_clean":"2024-08-17T17:37:00.384064+0000","last_became_active":"2024-08-17T17:37:00.383681+0000","last_became_peered":"2024-08-17T17:37:00.383681+0000","last_unstale":"2024-08-17T17:37:00.384064+0000","last_undegraded":"2024-08-17T17:37:00.384064+0000","last_fullsized":"2024-08-17T17:37:00.384064+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:10:22.569694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":614,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.386302+0000","last_change":"2024-08-17T17:35:07.285757+0000","last_active":"2024-08-17T17:37:00.386302+0000","last_peered":"2024-08-17T17:37:00.386302+0000","last_clean":"2024-08-17T17:37:00.386302+0000","last_became_active":"2024-08-17T17:34:17.886791+0000","last_became_peered":"2024-08-17T17:34:17.886791+0000","last_unstale":"2024-08-17T17:37:00.386302+0000","last_undegraded":"2024-08-17T17:37:00.386302+0000","last_fullsized":"2024-08-17T17:37:00.386302+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:35:07.285633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:35:07.285633+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:32:46.956526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"300'13346","reported_seq":13435,"reported_epoch":300,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:37:02.719604+0000","last_change":"2024-08-17T17:33:22.097156+0000","last_active":"2024-08-17T17:37:02.719604+0000","last_peered":"2024-08-17T17:37:02.719604+0000","last_clean":"2024-08-17T17:32:47.362949+0000","last_became_active":"2024-08-17T17:33:21.961793+0000","last_became_peered":"2024-08-17T17:33:21.961793+0000","last_unstale":"2024-08-17T17:37:02.719604+0000","last_undegraded":"2024-08-17T17:33:21.778949+0000","last_fullsized":"2024-08-17T17:33:21.755681+0000","mapping_epoch":269,"log_start":"266'3300","ondisk_log_start":"266'3300","created":262,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3589,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13346,"num_write_kb":427072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":262144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[{"shards":"2,5","objects":-3589}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":716,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.380739+0000","last_change":"2024-08-17T17:33:20.797705+0000","last_active":"2024-08-17T17:37:00.380739+0000","last_peered":"2024-08-17T17:37:00.380739+0000","last_clean":"2024-08-17T17:37:00.380739+0000","last_became_active":"2024-08-17T17:33:20.797172+0000","last_became_peered":"2024-08-17T17:33:20.797172+0000","last_unstale":"2024-08-17T17:37:00.380739+0000","last_undegraded":"2024-08-17T17:37:00.380739+0000","last_fullsized":"2024-08-17T17:37:00.380739+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:30.754999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":468,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.411126+0000","last_change":"2024-08-17T17:37:00.411061+0000","last_active":"2024-08-17T17:37:00.411126+0000","last_peered":"2024-08-17T17:37:00.411126+0000","last_clean":"2024-08-17T17:37:00.411126+0000","last_became_active":"2024-08-17T17:37:00.410374+0000","last_became_peered":"2024-08-17T17:37:00.410374+0000","last_unstale":"2024-08-17T17:37:00.411126+0000","last_undegraded":"2024-08-17T17:37:00.411126+0000","last_fullsized":"2024-08-17T17:37:00.411126+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:09:04.878864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":270,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.388683+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:37:00.388683+0000","last_peered":"2024-08-17T17:37:00.388683+0000","last_clean":"2024-08-17T17:37:00.388683+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:37:00.388683+0000","last_undegraded":"2024-08-17T17:37:00.388683+0000","last_fullsized":"2024-08-17T17:37:00.388683+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":611,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.387996+0000","last_change":"2024-08-17T17:35:01.201171+0000","last_active":"2024-08-17T17:37:00.387996+0000","last_peered":"2024-08-17T17:37:00.387996+0000","last_clean":"2024-08-17T17:37:00.387996+0000","last_became_active":"2024-08-17T17:34:17.890103+0000","last_became_peered":"2024-08-17T17:34:17.890103+0000","last_unstale":"2024-08-17T17:37:00.387996+0000","last_undegraded":"2024-08-17T17:37:00.387996+0000","last_fullsized":"2024-08-17T17:37:00.387996+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:35:01.201049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:35:01.201049+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:01:32.456637+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"300'13512","reported_seq":18195,"reported_epoch":300,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:37:02.697157+0000","last_change":"2024-08-17T17:37:00.415226+0000","last_active":"2024-08-17T17:37:02.697157+0000","last_peered":"2024-08-17T17:37:02.697157+0000","last_clean":"2024-08-17T17:32:47.362949+0000","last_became_active":"2024-08-17T17:33:21.969537+0000","last_became_peered":"2024-08-17T17:33:21.969537+0000","last_unstale":"2024-08-17T17:37:02.697157+0000","last_undegraded":"2024-08-17T17:33:21.804745+0000","last_fullsized":"2024-08-17T17:33:21.762111+0000","mapping_epoch":269,"log_start":"274'5100","ondisk_log_start":"274'5100","created":262,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":8412,"log_dups_size":0,"ondisk_log_size":8412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1338,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13512,"num_write_kb":432384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2296,"num_bytes_recovered":27394048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":-1338}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":849,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.377116+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:37:00.377116+0000","last_peered":"2024-08-17T17:37:00.377116+0000","last_clean":"2024-08-17T17:37:00.377116+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:37:00.377116+0000","last_undegraded":"2024-08-17T17:37:00.377116+0000","last_fullsized":"2024-08-17T17:37:00.377116+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":354,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.380850+0000","last_change":"2024-08-17T17:33:20.793468+0000","last_active":"2024-08-17T17:37:00.380850+0000","last_peered":"2024-08-17T17:37:00.380850+0000","last_clean":"2024-08-17T17:37:00.380850+0000","last_became_active":"2024-08-17T17:33:20.793166+0000","last_became_peered":"2024-08-17T17:33:20.793166+0000","last_unstale":"2024-08-17T17:37:00.380850+0000","last_undegraded":"2024-08-17T17:37:00.380850+0000","last_fullsized":"2024-08-17T17:37:00.380850+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:12:52.121050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":390,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.406886+0000","last_change":"2024-08-17T17:33:20.835784+0000","last_active":"2024-08-17T17:37:00.406886+0000","last_peered":"2024-08-17T17:37:00.406886+0000","last_clean":"2024-08-17T17:37:00.406886+0000","last_became_active":"2024-08-17T17:33:20.835371+0000","last_became_peered":"2024-08-17T17:33:20.835371+0000","last_unstale":"2024-08-17T17:37:00.406886+0000","last_undegraded":"2024-08-17T17:37:00.406886+0000","last_fullsized":"2024-08-17T17:37:00.406886+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:35:43.369055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":610,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.389113+0000","last_change":"2024-08-17T17:35:03.210422+0000","last_active":"2024-08-17T17:37:00.389113+0000","last_peered":"2024-08-17T17:37:00.389113+0000","last_clean":"2024-08-17T17:37:00.389113+0000","last_became_active":"2024-08-17T17:34:17.898012+0000","last_became_peered":"2024-08-17T17:34:17.898012+0000","last_unstale":"2024-08-17T17:37:00.389113+0000","last_undegraded":"2024-08-17T17:37:00.389113+0000","last_fullsized":"2024-08-17T17:37:00.389113+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:35:03.210298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:35:03.210298+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:16:50.801783+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"300'13396","reported_seq":13479,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:02.671912+0000","last_change":"2024-08-17T17:33:44.165133+0000","last_active":"2024-08-17T17:37:02.671912+0000","last_peered":"2024-08-17T17:37:02.671912+0000","last_clean":"2024-08-17T17:37:02.671912+0000","last_became_active":"2024-08-17T17:33:44.156220+0000","last_became_peered":"2024-08-17T17:33:44.156220+0000","last_unstale":"2024-08-17T17:37:02.671912+0000","last_undegraded":"2024-08-17T17:37:02.671912+0000","last_fullsized":"2024-08-17T17:37:02.671912+0000","mapping_epoch":272,"log_start":"273'5000","ondisk_log_start":"273'5000","created":262,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":8396,"log_dups_size":0,"ondisk_log_size":8396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:57:21.141437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13396,"num_write_kb":428672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":939,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.387332+0000","last_change":"2024-08-17T17:33:44.148227+0000","last_active":"2024-08-17T17:37:00.387332+0000","last_peered":"2024-08-17T17:37:00.387332+0000","last_clean":"2024-08-17T17:37:00.387332+0000","last_became_active":"2024-08-17T17:33:44.147627+0000","last_became_peered":"2024-08-17T17:33:44.147627+0000","last_unstale":"2024-08-17T17:37:00.387332+0000","last_undegraded":"2024-08-17T17:37:00.387332+0000","last_fullsized":"2024-08-17T17:37:00.387332+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:29:42.898460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1372,"num_read_kb":2843,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":49,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-17T17:36:59.355760+0000","last_change":"2024-08-17T17:32:46.379200+0000","last_active":"2024-08-17T17:36:59.355760+0000","last_peered":"2024-08-17T17:36:59.355760+0000","last_clean":"2024-08-17T17:36:59.355760+0000","last_became_active":"2024-08-17T17:32:46.378870+0000","last_became_peered":"2024-08-17T17:32:46.378870+0000","last_unstale":"2024-08-17T17:36:59.355760+0000","last_undegraded":"2024-08-17T17:36:59.355760+0000","last_fullsized":"2024-08-17T17:36:59.355760+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:55:26.885966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":394,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.398993+0000","last_change":"2024-08-17T17:33:20.772454+0000","last_active":"2024-08-17T17:37:00.398993+0000","last_peered":"2024-08-17T17:37:00.398993+0000","last_clean":"2024-08-17T17:37:00.398993+0000","last_became_active":"2024-08-17T17:33:20.771848+0000","last_became_peered":"2024-08-17T17:33:20.771848+0000","last_unstale":"2024-08-17T17:37:00.398993+0000","last_undegraded":"2024-08-17T17:37:00.398993+0000","last_fullsized":"2024-08-17T17:37:00.398993+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:46:42.300744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":614,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.385639+0000","last_change":"2024-08-17T17:35:05.269962+0000","last_active":"2024-08-17T17:37:00.385639+0000","last_peered":"2024-08-17T17:37:00.385639+0000","last_clean":"2024-08-17T17:37:00.385639+0000","last_became_active":"2024-08-17T17:34:17.885483+0000","last_became_peered":"2024-08-17T17:34:17.885483+0000","last_unstale":"2024-08-17T17:37:00.385639+0000","last_undegraded":"2024-08-17T17:37:00.385639+0000","last_fullsized":"2024-08-17T17:37:00.385639+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:35:05.269844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:35:05.269844+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:09:31.270296+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"300'13540","reported_seq":13628,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:02.650410+0000","last_change":"2024-08-17T17:32:48.460706+0000","last_active":"2024-08-17T17:37:02.650410+0000","last_peered":"2024-08-17T17:37:02.650410+0000","last_clean":"2024-08-17T17:37:02.650410+0000","last_became_active":"2024-08-17T17:32:48.460309+0000","last_became_peered":"2024-08-17T17:32:48.460309+0000","last_unstale":"2024-08-17T17:37:02.650410+0000","last_undegraded":"2024-08-17T17:37:02.650410+0000","last_fullsized":"2024-08-17T17:37:02.650410+0000","mapping_epoch":262,"log_start":"279'7100","ondisk_log_start":"279'7100","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":6440,"log_dups_size":0,"ondisk_log_size":6440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:33:48.006665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13540,"num_write_kb":433280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":144,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.377887+0000","last_change":"2024-08-17T17:33:44.163867+0000","last_active":"2024-08-17T17:37:00.377887+0000","last_peered":"2024-08-17T17:37:00.377887+0000","last_clean":"2024-08-17T17:37:00.377887+0000","last_became_active":"2024-08-17T17:33:44.163503+0000","last_became_peered":"2024-08-17T17:33:44.163503+0000","last_unstale":"2024-08-17T17:37:00.377887+0000","last_undegraded":"2024-08-17T17:37:00.377887+0000","last_fullsized":"2024-08-17T17:37:00.377887+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:31:53.573596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":609,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.398780+0000","last_change":"2024-08-17T17:33:20.799350+0000","last_active":"2024-08-17T17:37:00.398780+0000","last_peered":"2024-08-17T17:37:00.398780+0000","last_clean":"2024-08-17T17:37:00.398780+0000","last_became_active":"2024-08-17T17:33:20.798925+0000","last_became_peered":"2024-08-17T17:33:20.798925+0000","last_unstale":"2024-08-17T17:37:00.398780+0000","last_undegraded":"2024-08-17T17:37:00.398780+0000","last_fullsized":"2024-08-17T17:37:00.398780+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:16:36.121295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":411,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.397164+0000","last_change":"2024-08-17T17:37:00.397108+0000","last_active":"2024-08-17T17:37:00.397164+0000","last_peered":"2024-08-17T17:37:00.397164+0000","last_clean":"2024-08-17T17:37:00.397164+0000","last_became_active":"2024-08-17T17:37:00.395931+0000","last_became_peered":"2024-08-17T17:37:00.395931+0000","last_unstale":"2024-08-17T17:37:00.397164+0000","last_undegraded":"2024-08-17T17:37:00.397164+0000","last_fullsized":"2024-08-17T17:37:00.397164+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:54:31.517826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":614,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.384905+0000","last_change":"2024-08-17T17:35:08.269771+0000","last_active":"2024-08-17T17:37:00.384905+0000","last_peered":"2024-08-17T17:37:00.384905+0000","last_clean":"2024-08-17T17:37:00.384905+0000","last_became_active":"2024-08-17T17:34:17.887618+0000","last_became_peered":"2024-08-17T17:34:17.887618+0000","last_unstale":"2024-08-17T17:37:00.384905+0000","last_undegraded":"2024-08-17T17:37:00.384905+0000","last_fullsized":"2024-08-17T17:37:00.384905+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:35:08.269476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:35:08.269476+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:10:44.708147+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.f","version":"300'13606","reported_seq":20883,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:02.677361+0000","last_change":"2024-08-17T17:35:33.473629+0000","last_active":"2024-08-17T17:37:02.677361+0000","last_peered":"2024-08-17T17:37:02.677361+0000","last_clean":"2024-08-17T17:37:02.677361+0000","last_became_active":"2024-08-17T17:35:33.447013+0000","last_became_peered":"2024-08-17T17:35:33.447013+0000","last_unstale":"2024-08-17T17:37:02.677361+0000","last_undegraded":"2024-08-17T17:37:02.677361+0000","last_fullsized":"2024-08-17T17:37:02.677361+0000","mapping_epoch":286,"log_start":"274'5200","ondisk_log_start":"274'5200","created":262,"last_epoch_clean":287,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":8406,"log_dups_size":0,"ondisk_log_size":8406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:35:36.377891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13606,"num_write_kb":435392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3590,"num_bytes_recovered":235274240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":429,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.398672+0000","last_change":"2024-08-17T17:33:20.798613+0000","last_active":"2024-08-17T17:37:00.398672+0000","last_peered":"2024-08-17T17:37:00.398672+0000","last_clean":"2024-08-17T17:37:00.398672+0000","last_became_active":"2024-08-17T17:33:20.798260+0000","last_became_peered":"2024-08-17T17:33:20.798260+0000","last_unstale":"2024-08-17T17:37:00.398672+0000","last_undegraded":"2024-08-17T17:37:00.398672+0000","last_fullsized":"2024-08-17T17:37:00.398672+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:18:00.822681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":776,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-17T17:36:59.365584+0000","last_change":"2024-08-17T17:32:49.379365+0000","last_active":"2024-08-17T17:36:59.365584+0000","last_peered":"2024-08-17T17:36:59.365584+0000","last_clean":"2024-08-17T17:36:59.365584+0000","last_became_active":"2024-08-17T17:32:49.378742+0000","last_became_peered":"2024-08-17T17:32:49.378742+0000","last_unstale":"2024-08-17T17:36:59.365584+0000","last_undegraded":"2024-08-17T17:36:59.365584+0000","last_fullsized":"2024-08-17T17:36:59.365584+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:32:06.957948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":444,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.383070+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:37:00.383070+0000","last_peered":"2024-08-17T17:37:00.383070+0000","last_clean":"2024-08-17T17:37:00.383070+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:37:00.383070+0000","last_undegraded":"2024-08-17T17:37:00.383070+0000","last_fullsized":"2024-08-17T17:37:00.383070+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":614,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.385473+0000","last_change":"2024-08-17T17:35:06.281979+0000","last_active":"2024-08-17T17:37:00.385473+0000","last_peered":"2024-08-17T17:37:00.385473+0000","last_clean":"2024-08-17T17:37:00.385473+0000","last_became_active":"2024-08-17T17:34:17.886175+0000","last_became_peered":"2024-08-17T17:34:17.886175+0000","last_unstale":"2024-08-17T17:37:00.385473+0000","last_undegraded":"2024-08-17T17:37:00.385473+0000","last_fullsized":"2024-08-17T17:37:00.385473+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:35:06.281848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:35:06.281848+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:50:28.599262+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"300'13574","reported_seq":13662,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:02.761173+0000","last_change":"2024-08-17T17:32:48.381312+0000","last_active":"2024-08-17T17:37:02.761173+0000","last_peered":"2024-08-17T17:37:02.761173+0000","last_clean":"2024-08-17T17:37:02.761173+0000","last_became_active":"2024-08-17T17:32:48.380990+0000","last_became_peered":"2024-08-17T17:32:48.380990+0000","last_unstale":"2024-08-17T17:37:02.761173+0000","last_undegraded":"2024-08-17T17:37:02.761173+0000","last_fullsized":"2024-08-17T17:37:02.761173+0000","mapping_epoch":262,"log_start":"279'7100","ondisk_log_start":"279'7100","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":6474,"log_dups_size":0,"ondisk_log_size":6474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:39:12.341886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13574,"num_write_kb":434368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":479,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.382116+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:37:00.382116+0000","last_peered":"2024-08-17T17:37:00.382116+0000","last_clean":"2024-08-17T17:37:00.382116+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:37:00.382116+0000","last_undegraded":"2024-08-17T17:37:00.382116+0000","last_fullsized":"2024-08-17T17:37:00.382116+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":962,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.399211+0000","last_change":"2024-08-17T17:33:21.763805+0000","last_active":"2024-08-17T17:37:00.399211+0000","last_peered":"2024-08-17T17:37:00.399211+0000","last_clean":"2024-08-17T17:37:00.399211+0000","last_became_active":"2024-08-17T17:33:21.763232+0000","last_became_peered":"2024-08-17T17:33:21.763232+0000","last_unstale":"2024-08-17T17:37:00.399211+0000","last_undegraded":"2024-08-17T17:37:00.399211+0000","last_fullsized":"2024-08-17T17:37:00.399211+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:46:08.994820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":475,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.382617+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:37:00.382617+0000","last_peered":"2024-08-17T17:37:00.382617+0000","last_clean":"2024-08-17T17:37:00.382617+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:37:00.382617+0000","last_undegraded":"2024-08-17T17:37:00.382617+0000","last_fullsized":"2024-08-17T17:37:00.382617+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":599,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.382493+0000","last_change":"2024-08-17T17:34:19.224000+0000","last_active":"2024-08-17T17:37:00.382493+0000","last_peered":"2024-08-17T17:37:00.382493+0000","last_clean":"2024-08-17T17:37:00.382493+0000","last_became_active":"2024-08-17T17:34:17.885462+0000","last_became_peered":"2024-08-17T17:34:17.885462+0000","last_unstale":"2024-08-17T17:37:00.382493+0000","last_undegraded":"2024-08-17T17:37:00.382493+0000","last_fullsized":"2024-08-17T17:37:00.382493+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:34:19.223644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:34:19.223644+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:19:57.910956+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"300'13340","reported_seq":13424,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:02.760051+0000","last_change":"2024-08-17T17:33:44.160183+0000","last_active":"2024-08-17T17:37:02.760051+0000","last_peered":"2024-08-17T17:37:02.760051+0000","last_clean":"2024-08-17T17:37:02.760051+0000","last_became_active":"2024-08-17T17:33:44.144600+0000","last_became_peered":"2024-08-17T17:33:44.144600+0000","last_unstale":"2024-08-17T17:37:02.760051+0000","last_undegraded":"2024-08-17T17:37:02.760051+0000","last_fullsized":"2024-08-17T17:37:02.760051+0000","mapping_epoch":272,"log_start":"266'3300","ondisk_log_start":"266'3300","created":262,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:32:47.362949+0000","last_clean_scrub_stamp":"2024-08-17T17:32:47.362949+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:23:53.175287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13340,"num_write_kb":426817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":294,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.388601+0000","last_change":"2024-08-17T17:33:21.765925+0000","last_active":"2024-08-17T17:37:00.388601+0000","last_peered":"2024-08-17T17:37:00.388601+0000","last_clean":"2024-08-17T17:37:00.388601+0000","last_became_active":"2024-08-17T17:33:21.765372+0000","last_became_peered":"2024-08-17T17:33:21.765372+0000","last_unstale":"2024-08-17T17:37:00.388601+0000","last_undegraded":"2024-08-17T17:37:00.388601+0000","last_fullsized":"2024-08-17T17:37:00.388601+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:14:47.737369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":531,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-17T17:36:59.362269+0000","last_change":"2024-08-17T17:26:06.855307+0000","last_active":"2024-08-17T17:36:59.362269+0000","last_peered":"2024-08-17T17:36:59.362269+0000","last_clean":"2024-08-17T17:36:59.362269+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:36:59.362269+0000","last_undegraded":"2024-08-17T17:36:59.362269+0000","last_fullsized":"2024-08-17T17:36:59.362269+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.382745+0000","last_change":"2024-08-17T17:37:00.382745+0000","last_active":"2024-08-17T17:37:00.382745+0000","last_peered":"2024-08-17T17:37:00.382745+0000","last_clean":"2024-08-17T17:37:00.382745+0000","last_became_active":"2024-08-17T17:37:00.382032+0000","last_became_peered":"2024-08-17T17:37:00.382032+0000","last_unstale":"2024-08-17T17:37:00.382745+0000","last_undegraded":"2024-08-17T17:37:00.382745+0000","last_fullsized":"2024-08-17T17:37:00.382745+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:24:51.920975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":63,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-17T17:36:59.355810+0000","last_change":"2024-08-17T17:33:21.763655+0000","last_active":"2024-08-17T17:36:59.355810+0000","last_peered":"2024-08-17T17:36:59.355810+0000","last_clean":"2024-08-17T17:36:59.355810+0000","last_became_active":"2024-08-17T17:33:21.763088+0000","last_became_peered":"2024-08-17T17:33:21.763088+0000","last_unstale":"2024-08-17T17:36:59.355810+0000","last_undegraded":"2024-08-17T17:36:59.355810+0000","last_fullsized":"2024-08-17T17:36:59.355810+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:40:40.126898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":20,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.413709+0000","last_change":"2024-08-17T17:37:00.413709+0000","last_active":"2024-08-17T17:37:00.413709+0000","last_peered":"2024-08-17T17:37:00.413709+0000","last_clean":"2024-08-17T17:37:00.413709+0000","last_became_active":"2024-08-17T17:37:00.383355+0000","last_became_peered":"2024-08-17T17:37:00.383355+0000","last_unstale":"2024-08-17T17:37:00.413709+0000","last_undegraded":"2024-08-17T17:37:00.413709+0000","last_fullsized":"2024-08-17T17:37:00.413709+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:47:24.501358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":422,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.387321+0000","last_change":"2024-08-17T17:33:20.799402+0000","last_active":"2024-08-17T17:37:00.387321+0000","last_peered":"2024-08-17T17:37:00.387321+0000","last_clean":"2024-08-17T17:37:00.387321+0000","last_became_active":"2024-08-17T17:33:20.798832+0000","last_became_peered":"2024-08-17T17:33:20.798832+0000","last_unstale":"2024-08-17T17:37:00.387321+0000","last_undegraded":"2024-08-17T17:37:00.387321+0000","last_fullsized":"2024-08-17T17:37:00.387321+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:21:05.001134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":170,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.377699+0000","last_change":"2024-08-17T17:32:46.381141+0000","last_active":"2024-08-17T17:37:00.377699+0000","last_peered":"2024-08-17T17:37:00.377699+0000","last_clean":"2024-08-17T17:37:00.377699+0000","last_became_active":"2024-08-17T17:32:46.374259+0000","last_became_peered":"2024-08-17T17:32:46.374259+0000","last_unstale":"2024-08-17T17:37:00.377699+0000","last_undegraded":"2024-08-17T17:37:00.377699+0000","last_fullsized":"2024-08-17T17:37:00.377699+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:19:10.836889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":161,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.399298+0000","last_change":"2024-08-17T17:33:44.146014+0000","last_active":"2024-08-17T17:37:00.399298+0000","last_peered":"2024-08-17T17:37:00.399298+0000","last_clean":"2024-08-17T17:37:00.399298+0000","last_became_active":"2024-08-17T17:33:44.145668+0000","last_became_peered":"2024-08-17T17:33:44.145668+0000","last_unstale":"2024-08-17T17:37:00.399298+0000","last_undegraded":"2024-08-17T17:37:00.399298+0000","last_fullsized":"2024-08-17T17:37:00.399298+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:53:41.679237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":425,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.376690+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:37:00.376690+0000","last_peered":"2024-08-17T17:37:00.376690+0000","last_clean":"2024-08-17T17:37:00.376690+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:37:00.376690+0000","last_undegraded":"2024-08-17T17:37:00.376690+0000","last_fullsized":"2024-08-17T17:37:00.376690+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":452,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.385875+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:37:00.385875+0000","last_peered":"2024-08-17T17:37:00.385875+0000","last_clean":"2024-08-17T17:37:00.385875+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:37:00.385875+0000","last_undegraded":"2024-08-17T17:37:00.385875+0000","last_fullsized":"2024-08-17T17:37:00.385875+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":483,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.382536+0000","last_change":"2024-08-17T17:33:20.790687+0000","last_active":"2024-08-17T17:37:00.382536+0000","last_peered":"2024-08-17T17:37:00.382536+0000","last_clean":"2024-08-17T17:37:00.382536+0000","last_became_active":"2024-08-17T17:33:20.790399+0000","last_became_peered":"2024-08-17T17:33:20.790399+0000","last_unstale":"2024-08-17T17:37:00.382536+0000","last_undegraded":"2024-08-17T17:37:00.382536+0000","last_fullsized":"2024-08-17T17:37:00.382536+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:38:06.849989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":274,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-17T17:36:59.362183+0000","last_change":"2024-08-17T17:33:20.835760+0000","last_active":"2024-08-17T17:36:59.362183+0000","last_peered":"2024-08-17T17:36:59.362183+0000","last_clean":"2024-08-17T17:36:59.362183+0000","last_became_active":"2024-08-17T17:33:20.835410+0000","last_became_peered":"2024-08-17T17:33:20.835410+0000","last_unstale":"2024-08-17T17:36:59.362183+0000","last_undegraded":"2024-08-17T17:36:59.362183+0000","last_fullsized":"2024-08-17T17:36:59.362183+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:45.448022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":501,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.382693+0000","last_change":"2024-08-17T17:33:20.771279+0000","last_active":"2024-08-17T17:37:00.382693+0000","last_peered":"2024-08-17T17:37:00.382693+0000","last_clean":"2024-08-17T17:37:00.382693+0000","last_became_active":"2024-08-17T17:33:20.770728+0000","last_became_peered":"2024-08-17T17:33:20.770728+0000","last_unstale":"2024-08-17T17:37:00.382693+0000","last_undegraded":"2024-08-17T17:37:00.382693+0000","last_fullsized":"2024-08-17T17:37:00.382693+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:52:31.540397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":270,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.388334+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:37:00.388334+0000","last_peered":"2024-08-17T17:37:00.388334+0000","last_clean":"2024-08-17T17:37:00.388334+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:37:00.388334+0000","last_undegraded":"2024-08-17T17:37:00.388334+0000","last_fullsized":"2024-08-17T17:37:00.388334+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":407,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.403360+0000","last_change":"2024-08-17T17:37:00.403306+0000","last_active":"2024-08-17T17:37:00.403360+0000","last_peered":"2024-08-17T17:37:00.403360+0000","last_clean":"2024-08-17T17:37:00.403360+0000","last_became_active":"2024-08-17T17:37:00.403030+0000","last_became_peered":"2024-08-17T17:37:00.403030+0000","last_unstale":"2024-08-17T17:37:00.403360+0000","last_undegraded":"2024-08-17T17:37:00.403360+0000","last_fullsized":"2024-08-17T17:37:00.403360+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:19:07.078877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":81,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-17T17:36:59.356091+0000","last_change":"2024-08-17T17:33:44.160733+0000","last_active":"2024-08-17T17:36:59.356091+0000","last_peered":"2024-08-17T17:36:59.356091+0000","last_clean":"2024-08-17T17:36:59.356091+0000","last_became_active":"2024-08-17T17:33:44.160368+0000","last_became_peered":"2024-08-17T17:33:44.160368+0000","last_unstale":"2024-08-17T17:36:59.356091+0000","last_undegraded":"2024-08-17T17:36:59.356091+0000","last_fullsized":"2024-08-17T17:36:59.356091+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:37:22.855642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":10,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.405506+0000","last_change":"2024-08-17T17:37:00.405506+0000","last_active":"2024-08-17T17:37:00.405506+0000","last_peered":"2024-08-17T17:37:00.405506+0000","last_clean":"2024-08-17T17:37:00.405506+0000","last_became_active":"2024-08-17T17:37:00.400982+0000","last_became_peered":"2024-08-17T17:37:00.400982+0000","last_unstale":"2024-08-17T17:37:00.405506+0000","last_undegraded":"2024-08-17T17:37:00.405506+0000","last_fullsized":"2024-08-17T17:37:00.405506+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:47:17.134917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":388,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.398919+0000","last_change":"2024-08-17T17:33:20.773207+0000","last_active":"2024-08-17T17:37:00.398919+0000","last_peered":"2024-08-17T17:37:00.398919+0000","last_clean":"2024-08-17T17:37:00.398919+0000","last_became_active":"2024-08-17T17:33:20.772457+0000","last_became_peered":"2024-08-17T17:33:20.772457+0000","last_unstale":"2024-08-17T17:37:00.398919+0000","last_undegraded":"2024-08-17T17:37:00.398919+0000","last_fullsized":"2024-08-17T17:37:00.398919+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:57:09.610084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":280,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.382907+0000","last_change":"2024-08-17T17:32:46.375464+0000","last_active":"2024-08-17T17:37:00.382907+0000","last_peered":"2024-08-17T17:37:00.382907+0000","last_clean":"2024-08-17T17:37:00.382907+0000","last_became_active":"2024-08-17T17:32:46.375161+0000","last_became_peered":"2024-08-17T17:32:46.375161+0000","last_unstale":"2024-08-17T17:37:00.382907+0000","last_undegraded":"2024-08-17T17:37:00.382907+0000","last_fullsized":"2024-08-17T17:37:00.382907+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:27:47.091836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":10,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.447160+0000","last_change":"2024-08-17T17:37:00.447160+0000","last_active":"2024-08-17T17:37:00.447160+0000","last_peered":"2024-08-17T17:37:00.447160+0000","last_clean":"2024-08-17T17:37:00.447160+0000","last_became_active":"2024-08-17T17:37:00.446710+0000","last_became_peered":"2024-08-17T17:37:00.446710+0000","last_unstale":"2024-08-17T17:37:00.447160+0000","last_undegraded":"2024-08-17T17:37:00.447160+0000","last_fullsized":"2024-08-17T17:37:00.447160+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:51:43.200559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":314,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-17T17:36:59.356307+0000","last_change":"2024-08-17T17:33:44.165175+0000","last_active":"2024-08-17T17:36:59.356307+0000","last_peered":"2024-08-17T17:36:59.356307+0000","last_clean":"2024-08-17T17:36:59.356307+0000","last_became_active":"2024-08-17T17:33:44.146198+0000","last_became_peered":"2024-08-17T17:33:44.146198+0000","last_unstale":"2024-08-17T17:36:59.356307+0000","last_undegraded":"2024-08-17T17:36:59.356307+0000","last_fullsized":"2024-08-17T17:36:59.356307+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:14:56.291775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":10,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.405789+0000","last_change":"2024-08-17T17:37:00.405789+0000","last_active":"2024-08-17T17:37:00.405789+0000","last_peered":"2024-08-17T17:37:00.405789+0000","last_clean":"2024-08-17T17:37:00.405789+0000","last_became_active":"2024-08-17T17:37:00.405044+0000","last_became_peered":"2024-08-17T17:37:00.405044+0000","last_unstale":"2024-08-17T17:37:00.405789+0000","last_undegraded":"2024-08-17T17:37:00.405789+0000","last_fullsized":"2024-08-17T17:37:00.405789+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:15:57.228668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":294,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-17T17:36:59.356055+0000","last_change":"2024-08-17T17:33:44.148090+0000","last_active":"2024-08-17T17:36:59.356055+0000","last_peered":"2024-08-17T17:36:59.356055+0000","last_clean":"2024-08-17T17:36:59.356055+0000","last_became_active":"2024-08-17T17:33:44.147539+0000","last_became_peered":"2024-08-17T17:33:44.147539+0000","last_unstale":"2024-08-17T17:36:59.356055+0000","last_undegraded":"2024-08-17T17:36:59.356055+0000","last_fullsized":"2024-08-17T17:36:59.356055+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:52:02.311332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":479,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.377244+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:37:00.377244+0000","last_peered":"2024-08-17T17:37:00.377244+0000","last_clean":"2024-08-17T17:37:00.377244+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:37:00.377244+0000","last_undegraded":"2024-08-17T17:37:00.377244+0000","last_fullsized":"2024-08-17T17:37:00.377244+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":479,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.378090+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:37:00.378090+0000","last_peered":"2024-08-17T17:37:00.378090+0000","last_clean":"2024-08-17T17:37:00.378090+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:37:00.378090+0000","last_undegraded":"2024-08-17T17:37:00.378090+0000","last_fullsized":"2024-08-17T17:37:00.378090+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":49,"reported_epoch":299,"state":"active+clean","last_fresh":"2024-08-17T17:36:59.355250+0000","last_change":"2024-08-17T17:32:46.376844+0000","last_active":"2024-08-17T17:36:59.355250+0000","last_peered":"2024-08-17T17:36:59.355250+0000","last_clean":"2024-08-17T17:36:59.355250+0000","last_became_active":"2024-08-17T17:32:46.376483+0000","last_became_peered":"2024-08-17T17:32:46.376483+0000","last_unstale":"2024-08-17T17:36:59.355250+0000","last_undegraded":"2024-08-17T17:36:59.355250+0000","last_fullsized":"2024-08-17T17:36:59.355250+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:44:24.358662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":289,"reported_epoch":300,"state":"active+clean","last_fresh":"2024-08-17T17:37:00.386194+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:37:00.386194+0000","last_peered":"2024-08-17T17:37:00.386194+0000","last_clean":"2024-08-17T17:37:00.386194+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:37:00.386194+0000","last_undegraded":"2024-08-17T17:37:00.386194+0000","last_fullsized":"2024-08-17T17:37:00.386194+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18027,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":214946,"num_write_kb":6878209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6928,"num_bytes_recovered":262930432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127511,"ondisk_log_size":127511,"up":48,"acting":42,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":6220,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":21334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19280,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5978,"num_read_kb":12991,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":33,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234364,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":914952,"kb_used_data":8576,"kb_used_omap":1,"kb_used_meta":906366,"kb_avail":92830200,"statfs":{"total":95995035648,"available":95058124800,"internally_reserved":0,"allocated":8781824,"data_stored":7038099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":928119128},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495999,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":895356,"kb_used_data":8564,"kb_used_omap":1,"kb_used_meta":886782,"kb_avail":92849796,"statfs":{"total":95995035648,"available":95078191104,"internally_reserved":0,"allocated":8769536,"data_stored":7037773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":908065112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757636,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":589620,"kb_used_data":3884,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":93155532,"statfs":{"total":95995035648,"available":95391264768,"internally_reserved":0,"allocated":3977216,"data_stored":2249126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019274,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309572,"kb_used_data":3900,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93435580,"statfs":{"total":95995035648,"available":95678033920,"internally_reserved":0,"allocated":3993600,"data_stored":2253688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435501,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":676784,"kb_used_data":3880,"kb_used_omap":4,"kb_used_meta":672891,"kb_avail":93068368,"statfs":{"total":95995035648,"available":95302008832,"internally_reserved":0,"allocated":3973120,"data_stored":2248908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":689040394},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599028,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":622656,"kb_used_data":7992,"kb_used_omap":7,"kb_used_meta":614648,"kb_avail":93122496,"statfs":{"total":95995035648,"available":95357435904,"internally_reserved":0,"allocated":8183808,"data_stored":6448470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":629399714},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":268,"seq":1151051235375,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":210736,"kb_used_data":3880,"kb_used_omap":10,"kb_used_meta":206837,"kb_avail":93534416,"statfs":{"total":95995035648,"available":95779241984,"internally_reserved":0,"allocated":3973120,"data_stored":2248853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10902,"internal_metadata":211801450},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926081,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":595064,"kb_used_data":4464,"kb_used_omap":7,"kb_used_meta":590584,"kb_avail":93150088,"statfs":{"total":95995035648,"available":95385690112,"internally_reserved":0,"allocated":4571136,"data_stored":2839786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":604758337},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7550,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:37:13.387 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:37:13.400 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:37:13.400 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:37:13.462 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:13.487 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:13.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:13.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:13.588 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:13.613 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:13.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:13.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:13.716 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:13.743 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:13.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:13.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:13.844 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:13.868 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:13.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:13.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:13.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:13 smithi053 ceph-mon[89963]: pgmap v950: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 204 op/s; 18027/84 objects degraded (21460.714%); 2 B/s, 0 keys/s, 92 objects/s recovering 2024-08-17T17:37:13.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:13 smithi053 ceph-mon[89963]: from='client.46229 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:37:13.969 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:13.987 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:13 smithi003 ceph-mon[103344]: pgmap v950: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 204 op/s; 18027/84 objects degraded (21460.714%); 2 B/s, 0 keys/s, 92 objects/s recovering 2024-08-17T17:37:13.987 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:13 smithi003 ceph-mon[103344]: from='client.46229 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:37:13.987 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:13 smithi003 ceph-mon[100615]: pgmap v950: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 204 op/s; 18027/84 objects degraded (21460.714%); 2 B/s, 0 keys/s, 92 objects/s recovering 2024-08-17T17:37:13.987 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:13 smithi003 ceph-mon[100615]: from='client.46229 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:37:13.994 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:13.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:13.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:14.096 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:14.120 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:14.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:14.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:14.222 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:14.248 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:14.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:14.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:14.350 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:14.377 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:14.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:14.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:14.479 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:14.505 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:14.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:14.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:14.607 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:14.632 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:14.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:14.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:14.734 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:14.758 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:14.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:14.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:14.860 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:14.887 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:14.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:14.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:14.988 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:15.012 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:15.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:15.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:15.114 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:15.141 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:15.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:15.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:15.243 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:15.269 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:15.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:15.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:15.371 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:15.393 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:15.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:15.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:15.495 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:15.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:15.522 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:15.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:15.623 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:15.650 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:15.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:15.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:15.709 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-08-17T17:37:15.751 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:15.775 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:15.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:15.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:15.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:15.902 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:15.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:15.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:15.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:15 smithi053 ceph-mon[89963]: pgmap v951: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 200 op/s; 18027/84 objects degraded (21460.714%); 2 B/s, 0 keys/s, 90 objects/s recovering 2024-08-17T17:37:15.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:15 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3504158590' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:37:15.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:15 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 18027/84 objects degraded (21460.714%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:37:16.003 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:16.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:15 smithi003 ceph-mon[103344]: pgmap v951: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 200 op/s; 18027/84 objects degraded (21460.714%); 2 B/s, 0 keys/s, 90 objects/s recovering 2024-08-17T17:37:16.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:15 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3504158590' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:37:16.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:15 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 18027/84 objects degraded (21460.714%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:37:16.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:15 smithi003 ceph-mon[100615]: pgmap v951: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 200 op/s; 18027/84 objects degraded (21460.714%); 2 B/s, 0 keys/s, 90 objects/s recovering 2024-08-17T17:37:16.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:15 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3504158590' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:37:16.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:15 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 18027/84 objects degraded (21460.714%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:37:16.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:37:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:37:15] "GET /metrics HTTP/1.1" 200 36445 "" "Prometheus/2.51.0" 2024-08-17T17:37:16.028 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:16.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:16.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:16.130 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:16.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:16.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:16.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:16.256 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:16.281 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:16.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:16.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:16.383 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:16.407 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:16.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:16.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:16.509 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:16.535 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:16.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:16.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:16.637 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:16.661 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:16.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:16.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:16.762 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:16.786 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:16.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:16.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:16.888 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:16.910 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:16.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:16.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:16.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:16 smithi053 ceph-mon[89963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:37:16.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:16 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3504158590' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:37:16.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:16 smithi053 ceph-mon[89963]: osdmap e301: 8 total, 8 up, 8 in 2024-08-17T17:37:17.012 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:17.015 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:16 smithi003 ceph-mon[103344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:37:17.015 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:16 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3504158590' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:37:17.015 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:16 smithi003 ceph-mon[103344]: osdmap e301: 8 total, 8 up, 8 in 2024-08-17T17:37:17.016 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:16 smithi003 ceph-mon[100615]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:37:17.016 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:16 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3504158590' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:37:17.016 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:16 smithi003 ceph-mon[100615]: osdmap e301: 8 total, 8 up, 8 in 2024-08-17T17:37:17.036 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:17.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:17.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:17.138 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:17.161 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:17.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:17.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:17.263 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:17.285 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:17.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:17.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:17.387 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:17.410 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:17.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:17.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:17.512 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:17.539 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:17.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:17.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:17.641 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:17.664 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:17.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:17.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:17.766 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:17.788 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:17.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:17.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:17.890 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:17.917 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:17.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:17.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:17.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:17 smithi053 ceph-mon[89963]: pgmap v953: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 17029/84 objects degraded (20272.619%); 0 B/s, 177 objects/s recovering 2024-08-17T17:37:18.019 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:18.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:17 smithi003 ceph-mon[103344]: pgmap v953: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 17029/84 objects degraded (20272.619%); 0 B/s, 177 objects/s recovering 2024-08-17T17:37:18.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:17 smithi003 ceph-mon[100615]: pgmap v953: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 17029/84 objects degraded (20272.619%); 0 B/s, 177 objects/s recovering 2024-08-17T17:37:18.043 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:18.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:18.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:18.144 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:18.171 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:18.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:18.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:18.273 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:18.299 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:18.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:18.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:18.400 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:18.422 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:18.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:18.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:18.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:18.547 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:18.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:18.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:18.648 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:18.670 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:18.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:18.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:18.772 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:18.797 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:18.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:18.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:18.899 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:18.922 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:18.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:18.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:19.024 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:19.047 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:19.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:19.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:19.149 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:19.172 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:19.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:19.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:19.274 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:19.297 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:19.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:19.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:19.399 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:19.422 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:19.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:19.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:19.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:19.546 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:19.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:19.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:19.648 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:19.671 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:19.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:19.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:19.767 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:37:19.772 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:19.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:19 smithi003 ceph-mon[103344]: pgmap v954: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 17029/84 objects degraded (20272.619%); 0 B/s, 177 objects/s recovering 2024-08-17T17:37:19.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:19 smithi003 ceph-mon[103344]: osdmap e302: 8 total, 8 up, 8 in 2024-08-17T17:37:19.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:19 smithi003 ceph-mon[100615]: pgmap v954: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 17029/84 objects degraded (20272.619%); 0 B/s, 177 objects/s recovering 2024-08-17T17:37:19.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:19 smithi003 ceph-mon[100615]: osdmap e302: 8 total, 8 up, 8 in 2024-08-17T17:37:19.800 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:19.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:19.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:19.902 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:19.925 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:19.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:19.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:20.026 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:20.049 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:20.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:20.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:20.151 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:20.173 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:20.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:20.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:20.214 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:19 smithi053 ceph-mon[89963]: pgmap v954: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 93 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 17029/84 objects degraded (20272.619%); 0 B/s, 177 objects/s recovering 2024-08-17T17:37:20.214 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:19 smithi053 ceph-mon[89963]: osdmap e302: 8 total, 8 up, 8 in 2024-08-17T17:37:20.274 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:20.297 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:20.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:20.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:20.399 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:20.421 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:20.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:20.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:20.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:37:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:37:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:37:20.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:20.546 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:20.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:20.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:20.648 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:20.671 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:20.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:20.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:20.773 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:20.795 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:20.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:20.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:20.897 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:20.920 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:20.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:20.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:20.984 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:20 smithi003 ceph-mon[103344]: osdmap e303: 8 total, 8 up, 8 in 2024-08-17T17:37:20.984 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:20 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 17029/84 objects degraded (20272.619%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:37:20.984 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:20 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:37:20.984 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:20 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:37:20.985 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:20 smithi003 ceph-mon[100615]: osdmap e303: 8 total, 8 up, 8 in 2024-08-17T17:37:20.985 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:20 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 17029/84 objects degraded (20272.619%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:37:20.985 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:20 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:37:20.985 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:20 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:37:21.021 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:21.045 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:21.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:21.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:21.146 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:21.171 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:21.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:21.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:21.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:20 smithi053 ceph-mon[89963]: osdmap e303: 8 total, 8 up, 8 in 2024-08-17T17:37:21.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:20 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 17029/84 objects degraded (20272.619%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-08-17T17:37:21.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:20 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:37:21.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:20 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:37:21.251 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:37:21.252 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:37:21.273 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:21.297 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:21.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:21.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:21.399 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:21.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:21.422 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:21.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:21.523 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:21.545 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:21.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:21.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:21.647 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:21.671 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:21.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:21.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:21.773 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:21.795 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:21.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:21.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:21.897 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:21.920 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:21.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:21.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:22.021 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:21 smithi003 ceph-mon[103344]: pgmap v957: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 15270/84 objects degraded (18178.571%); 0 B/s, 459 objects/s recovering 2024-08-17T17:37:22.021 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:21 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:37:22.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:21 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:37:22.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:21 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:37:22.022 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:21 smithi003 ceph-mon[100615]: pgmap v957: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 15270/84 objects degraded (18178.571%); 0 B/s, 459 objects/s recovering 2024-08-17T17:37:22.022 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:21 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:37:22.022 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:21 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:37:22.022 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:21 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:37:22.022 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:22.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:22.045 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:22.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:22.146 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:22.168 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:22.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:22.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:22.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:21 smithi053 ceph-mon[89963]: pgmap v957: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 15270/84 objects degraded (18178.571%); 0 B/s, 459 objects/s recovering 2024-08-17T17:37:22.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:21 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:37:22.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:21 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:37:22.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:21 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:37:22.270 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:22.293 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:22.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:22.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:22.395 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:22.418 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:22.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:22.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:22.447 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-17T17:37:22.520 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:22.556 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:22.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:22.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:22.658 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:22.680 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:22.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:22.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:22.782 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:22.804 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:22.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:22.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:22.857 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:37:22.907 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:22.928 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:22.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:22.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:23.030 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:23.052 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:23.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:23.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:23.154 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:23.176 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:23.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:23.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:23.278 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:23.300 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:23.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:23.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:23.402 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:23.424 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:23.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:23.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:23.525 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:23.548 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:23.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:23.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:23.650 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:23.672 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:23.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:23.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:23.774 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:23.799 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:23.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:23.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:23.901 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:23.924 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:23.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:23.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:24.022 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:23 smithi003 ceph-mon[100615]: pgmap v958: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 15270/84 objects degraded (18178.571%); 0 B/s, 271 objects/s recovering 2024-08-17T17:37:24.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:37:24.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:37:24.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:23 smithi003 ceph-mon[103344]: pgmap v958: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 15270/84 objects degraded (18178.571%); 0 B/s, 271 objects/s recovering 2024-08-17T17:37:24.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:37:24.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:37:24.025 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:24.048 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:24.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:24.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:24.149 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:24.151 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:37:24.172 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:24.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:24.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:24.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:23 smithi053 ceph-mon[89963]: pgmap v958: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 15270/84 objects degraded (18178.571%); 0 B/s, 271 objects/s recovering 2024-08-17T17:37:24.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:37:24.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:37:24.274 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:24.297 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:24.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:24.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:24.398 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:24.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:24.422 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:24.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:24.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:24.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:24.548 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:24.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:24.648 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:24.672 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:24.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:24.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:24.774 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:24.796 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:24.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:24.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:24.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:37:24.898 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:24.950 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:24.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:24.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:25.052 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:25.074 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:25.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:25.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:25.176 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:25.198 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:25.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:25.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:25.300 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:25.325 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:25.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:25.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:25.427 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:25.449 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:25.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:25.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:25.551 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:25.574 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:25.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:25.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:25.676 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:25.698 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:25.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:25.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:25.800 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:25.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:25.824 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:25.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:25.908 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:25 smithi003 ceph-mon[103344]: pgmap v959: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 14168/84 objects degraded (16866.667%); 0 B/s, 357 objects/s recovering 2024-08-17T17:37:25.908 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:25 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 14168/84 objects degraded (16866.667%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:37:25.908 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:37:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:37:25] "GET /metrics HTTP/1.1" 200 36460 "" "Prometheus/2.51.0" 2024-08-17T17:37:25.908 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:25 smithi003 ceph-mon[100615]: pgmap v959: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 14168/84 objects degraded (16866.667%); 0 B/s, 357 objects/s recovering 2024-08-17T17:37:25.908 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:25 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 14168/84 objects degraded (16866.667%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:37:25.926 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:25.949 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:25.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:25.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:26.051 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:26.074 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:26.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:26.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:26.176 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:26.200 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:26.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:26.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:26.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:25 smithi053 ceph-mon[89963]: pgmap v959: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 14168/84 objects degraded (16866.667%); 0 B/s, 357 objects/s recovering 2024-08-17T17:37:26.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:25 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 14168/84 objects degraded (16866.667%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-08-17T17:37:26.302 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:26.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:26.326 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:26.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:26.427 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:26.450 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:26.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:26.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:26.551 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:26.576 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:26.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:26.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:26.678 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:26.700 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:26.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:26.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:26.743 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_3' removed 2024-08-17T17:37:26.802 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:26.825 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:26.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:26.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:26.927 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:26.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:26 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1077920062' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:37:26.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:26 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1077920062' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:37:26.951 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:26.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:26.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:27.053 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:27.075 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:27.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:27.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:27.177 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:27.204 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:27.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:27.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:27.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:26 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1077920062' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:37:27.306 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:27.330 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:27.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:27.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:27.432 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:27.456 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:27.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:27.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:27.558 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:27.582 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:27.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:27.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:27.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:27.708 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:27.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:27.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:27.773 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-08-17T17:37:27.774 INFO:tasks.radosbench:Beginning radosbench... 2024-08-17T17:37:27.774 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-08-17T17:37:27.774 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd pool create unique_pool_4 16 2024-08-17T17:37:27.810 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:27.839 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:27.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:27.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:27.941 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:27.964 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:27.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:27.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:27.988 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:27 smithi003 ceph-mon[103344]: pgmap v960: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 14168/84 objects degraded (16866.667%); 0 B/s, 357 objects/s recovering 2024-08-17T17:37:27.988 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:27 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1077920062' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T17:37:27.989 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:27 smithi003 ceph-mon[103344]: osdmap e304: 8 total, 8 up, 8 in 2024-08-17T17:37:27.989 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:27 smithi003 ceph-mon[100615]: pgmap v960: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 14168/84 objects degraded (16866.667%); 0 B/s, 357 objects/s recovering 2024-08-17T17:37:27.989 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:27 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1077920062' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T17:37:27.989 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:27 smithi003 ceph-mon[100615]: osdmap e304: 8 total, 8 up, 8 in 2024-08-17T17:37:28.065 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:28.088 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:28.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:28.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:28.190 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:28.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:27 smithi053 ceph-mon[89963]: pgmap v960: 99 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 14168/84 objects degraded (16866.667%); 0 B/s, 357 objects/s recovering 2024-08-17T17:37:28.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:27 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1077920062' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T17:37:28.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:27 smithi053 ceph-mon[89963]: osdmap e304: 8 total, 8 up, 8 in 2024-08-17T17:37:28.217 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:28.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:28.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:28.275 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:37:28.275 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:37:28.320 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:28.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:28.349 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:28.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:28.450 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:28.475 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:28.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:28.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:28.476 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:37:28.476 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:37:28.493 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:37:28.493 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:37:28.510 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:37:28.510 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:37:28.526 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:37:28.526 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:37:28.544 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:37:28.544 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:37:28.561 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:37:28.561 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:37:28.577 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:28.578 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:37:28.579 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:37:28.593 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:37:28.594 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:37:28.605 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:28.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:28.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:28.706 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:28.729 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:28.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:28.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:28.831 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:28.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:28.854 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:28.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:28.914 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:28 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-08-17T17:37:28.915 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:28 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2601553110' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:37:28.915 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:28 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-08-17T17:37:28.915 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:28 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2601553110' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:37:28.955 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:28.982 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:28.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:28.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:29.084 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:29.107 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:29.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:29.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:29.209 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:29.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:28 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-08-17T17:37:29.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:28 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2601553110' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:37:29.232 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:29.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:29.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:29.334 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:29.356 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:29.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:29.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:29.458 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:29.481 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:29.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:29.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:29.583 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:29.610 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:29.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:29.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:29.712 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:29.738 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:29.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:29.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:29.821 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":962,"stamp":"2024-08-17T17:37:28.185101+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790221,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":5978,"num_read_kb":12991,"num_write":907,"num_write_kb":12623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":8394828,"num_keys_recovered":20,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2671,"ondisk_log_size":2671,"up":249,"acting":249,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":301,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":4798524,"kb_used_data":45524,"kb_used_omap":37,"kb_used_meta":4752858,"kb_avail":745162692,"statfs":{"total":767960285184,"available":763046596608,"internally_reserved":0,"allocated":46616576,"data_stored":32568943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38729,"internal_metadata":4866926775},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-16840,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-214946,"num_write_kb":-6878209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8115,"num_bytes_recovered":-262930432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.465783"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":475,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:20.846169+0000","last_change":"2024-08-17T17:33:20.797029+0000","last_active":"2024-08-17T17:37:20.846169+0000","last_peered":"2024-08-17T17:37:20.846169+0000","last_clean":"2024-08-17T17:37:20.846169+0000","last_became_active":"2024-08-17T17:33:20.796679+0000","last_became_peered":"2024-08-17T17:33:20.796679+0000","last_unstale":"2024-08-17T17:37:20.846169+0000","last_undegraded":"2024-08-17T17:37:20.846169+0000","last_fullsized":"2024-08-17T17:37:20.846169+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:51.438262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":295,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-08-17T17:37:18.732122+0000","last_change":"2024-08-17T17:37:00.420764+0000","last_active":"2024-08-17T17:37:18.732122+0000","last_peered":"2024-08-17T17:37:18.732122+0000","last_clean":"2024-08-17T17:37:18.732122+0000","last_became_active":"2024-08-17T17:37:00.387450+0000","last_became_peered":"2024-08-17T17:37:00.387450+0000","last_unstale":"2024-08-17T17:37:18.732122+0000","last_undegraded":"2024-08-17T17:37:18.732122+0000","last_fullsized":"2024-08-17T17:37:18.732122+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:09:59.462151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":136,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:20.845597+0000","last_change":"2024-08-17T17:33:44.156036+0000","last_active":"2024-08-17T17:37:20.845597+0000","last_peered":"2024-08-17T17:37:20.845597+0000","last_clean":"2024-08-17T17:37:20.845597+0000","last_became_active":"2024-08-17T17:33:44.155758+0000","last_became_peered":"2024-08-17T17:33:44.155758+0000","last_unstale":"2024-08-17T17:37:20.845597+0000","last_undegraded":"2024-08-17T17:37:20.845597+0000","last_fullsized":"2024-08-17T17:37:20.845597+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:23.443405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":275,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.723394+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:37:19.723394+0000","last_peered":"2024-08-17T17:37:19.723394+0000","last_clean":"2024-08-17T17:37:19.723394+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:37:19.723394+0000","last_undegraded":"2024-08-17T17:37:19.723394+0000","last_fullsized":"2024-08-17T17:37:19.723394+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":275,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.724734+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:37:19.724734+0000","last_peered":"2024-08-17T17:37:19.724734+0000","last_clean":"2024-08-17T17:37:19.724734+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:37:19.724734+0000","last_undegraded":"2024-08-17T17:37:19.724734+0000","last_fullsized":"2024-08-17T17:37:19.724734+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":52,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-08-17T17:37:18.717655+0000","last_change":"2024-08-17T17:32:46.377651+0000","last_active":"2024-08-17T17:37:18.717655+0000","last_peered":"2024-08-17T17:37:18.717655+0000","last_clean":"2024-08-17T17:37:18.717655+0000","last_became_active":"2024-08-17T17:32:46.377282+0000","last_became_peered":"2024-08-17T17:32:46.377282+0000","last_unstale":"2024-08-17T17:37:18.717655+0000","last_undegraded":"2024-08-17T17:37:18.717655+0000","last_fullsized":"2024-08-17T17:37:18.717655+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:45:27.370287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":489,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:20.846237+0000","last_change":"2024-08-17T17:33:20.794682+0000","last_active":"2024-08-17T17:37:20.846237+0000","last_peered":"2024-08-17T17:37:20.846237+0000","last_clean":"2024-08-17T17:37:20.846237+0000","last_became_active":"2024-08-17T17:33:20.794238+0000","last_became_peered":"2024-08-17T17:33:20.794238+0000","last_unstale":"2024-08-17T17:37:20.846237+0000","last_undegraded":"2024-08-17T17:37:20.846237+0000","last_fullsized":"2024-08-17T17:37:20.846237+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:15:55.821920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":349,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-08-17T17:37:18.731571+0000","last_change":"2024-08-17T17:24:53.465875+0000","last_active":"2024-08-17T17:37:18.731571+0000","last_peered":"2024-08-17T17:37:18.731571+0000","last_clean":"2024-08-17T17:37:18.731571+0000","last_became_active":"2024-08-17T17:24:53.458129+0000","last_became_peered":"2024-08-17T17:24:53.458129+0000","last_unstale":"2024-08-17T17:37:18.731571+0000","last_undegraded":"2024-08-17T17:37:18.731571+0000","last_fullsized":"2024-08-17T17:37:18.731571+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:51:33.584869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":236,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-08-17T17:37:18.732218+0000","last_change":"2024-08-17T17:32:46.384376+0000","last_active":"2024-08-17T17:37:18.732218+0000","last_peered":"2024-08-17T17:37:18.732218+0000","last_clean":"2024-08-17T17:37:18.732218+0000","last_became_active":"2024-08-17T17:32:46.374177+0000","last_became_peered":"2024-08-17T17:32:46.374177+0000","last_unstale":"2024-08-17T17:37:18.732218+0000","last_undegraded":"2024-08-17T17:37:18.732218+0000","last_fullsized":"2024-08-17T17:37:18.732218+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:51:58.880388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":58,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-08-17T17:37:18.731373+0000","last_change":"2024-08-17T17:33:44.145598+0000","last_active":"2024-08-17T17:37:18.731373+0000","last_peered":"2024-08-17T17:37:18.731373+0000","last_clean":"2024-08-17T17:37:18.731373+0000","last_became_active":"2024-08-17T17:33:44.145283+0000","last_became_peered":"2024-08-17T17:33:44.145283+0000","last_unstale":"2024-08-17T17:37:18.731373+0000","last_undegraded":"2024-08-17T17:37:18.731373+0000","last_fullsized":"2024-08-17T17:37:18.731373+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:11.965226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":232,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-08-17T17:37:18.732226+0000","last_change":"2024-08-17T17:37:00.416059+0000","last_active":"2024-08-17T17:37:18.732226+0000","last_peered":"2024-08-17T17:37:18.732226+0000","last_clean":"2024-08-17T17:37:18.732226+0000","last_became_active":"2024-08-17T17:37:00.385913+0000","last_became_peered":"2024-08-17T17:37:00.385913+0000","last_unstale":"2024-08-17T17:37:18.732226+0000","last_undegraded":"2024-08-17T17:37:18.732226+0000","last_fullsized":"2024-08-17T17:37:18.732226+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:11:24.884146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":66,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-08-17T17:37:18.717789+0000","last_change":"2024-08-17T17:33:21.766549+0000","last_active":"2024-08-17T17:37:18.717789+0000","last_peered":"2024-08-17T17:37:18.717789+0000","last_clean":"2024-08-17T17:37:18.717789+0000","last_became_active":"2024-08-17T17:33:21.766152+0000","last_became_peered":"2024-08-17T17:33:21.766152+0000","last_unstale":"2024-08-17T17:37:18.717789+0000","last_undegraded":"2024-08-17T17:37:18.717789+0000","last_fullsized":"2024-08-17T17:37:18.717789+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:56:15.855524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":54,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-08-17T17:37:18.731747+0000","last_change":"2024-08-17T17:33:44.145399+0000","last_active":"2024-08-17T17:37:18.731747+0000","last_peered":"2024-08-17T17:37:18.731747+0000","last_clean":"2024-08-17T17:37:18.731747+0000","last_became_active":"2024-08-17T17:33:44.144751+0000","last_became_peered":"2024-08-17T17:33:44.144751+0000","last_unstale":"2024-08-17T17:37:18.731747+0000","last_undegraded":"2024-08-17T17:37:18.731747+0000","last_fullsized":"2024-08-17T17:37:18.731747+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:54.276350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":132,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-08-17T17:37:18.732167+0000","last_change":"2024-08-17T17:33:20.795582+0000","last_active":"2024-08-17T17:37:18.732167+0000","last_peered":"2024-08-17T17:37:18.732167+0000","last_clean":"2024-08-17T17:37:18.732167+0000","last_became_active":"2024-08-17T17:33:20.795149+0000","last_became_peered":"2024-08-17T17:33:20.795149+0000","last_unstale":"2024-08-17T17:37:18.732167+0000","last_undegraded":"2024-08-17T17:37:18.732167+0000","last_fullsized":"2024-08-17T17:37:18.732167+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:14.020923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":133,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:20.845621+0000","last_change":"2024-08-17T17:33:44.154097+0000","last_active":"2024-08-17T17:37:20.845621+0000","last_peered":"2024-08-17T17:37:20.845621+0000","last_clean":"2024-08-17T17:37:20.845621+0000","last_became_active":"2024-08-17T17:33:44.152972+0000","last_became_peered":"2024-08-17T17:33:44.152972+0000","last_unstale":"2024-08-17T17:37:20.845621+0000","last_undegraded":"2024-08-17T17:37:20.845621+0000","last_fullsized":"2024-08-17T17:37:20.845621+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:42.602017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":483,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.723667+0000","last_change":"2024-08-17T17:33:21.763677+0000","last_active":"2024-08-17T17:37:19.723667+0000","last_peered":"2024-08-17T17:37:19.723667+0000","last_clean":"2024-08-17T17:37:19.723667+0000","last_became_active":"2024-08-17T17:33:21.760564+0000","last_became_peered":"2024-08-17T17:33:21.760564+0000","last_unstale":"2024-08-17T17:37:19.723667+0000","last_undegraded":"2024-08-17T17:37:19.723667+0000","last_fullsized":"2024-08-17T17:37:19.723667+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:01:00.772037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":617,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.723294+0000","last_change":"2024-08-17T17:35:04.273750+0000","last_active":"2024-08-17T17:37:19.723294+0000","last_peered":"2024-08-17T17:37:19.723294+0000","last_clean":"2024-08-17T17:37:19.723294+0000","last_became_active":"2024-08-17T17:34:17.898604+0000","last_became_peered":"2024-08-17T17:34:17.898604+0000","last_unstale":"2024-08-17T17:37:19.723294+0000","last_undegraded":"2024-08-17T17:37:19.723294+0000","last_fullsized":"2024-08-17T17:37:19.723294+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:35:04.265381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:35:04.265381+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:32:27.625196+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":510,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:20.846455+0000","last_change":"2024-08-17T17:32:46.387810+0000","last_active":"2024-08-17T17:37:20.846455+0000","last_peered":"2024-08-17T17:37:20.846455+0000","last_clean":"2024-08-17T17:37:20.846455+0000","last_became_active":"2024-08-17T17:32:46.374238+0000","last_became_peered":"2024-08-17T17:32:46.374238+0000","last_unstale":"2024-08-17T17:37:20.846455+0000","last_undegraded":"2024-08-17T17:37:20.846455+0000","last_fullsized":"2024-08-17T17:37:20.846455+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:53:29.918005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":52,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-08-17T17:37:18.717978+0000","last_change":"2024-08-17T17:32:46.378495+0000","last_active":"2024-08-17T17:37:18.717978+0000","last_peered":"2024-08-17T17:37:18.717978+0000","last_clean":"2024-08-17T17:37:18.717978+0000","last_became_active":"2024-08-17T17:32:46.377709+0000","last_became_peered":"2024-08-17T17:32:46.377709+0000","last_unstale":"2024-08-17T17:37:18.717978+0000","last_undegraded":"2024-08-17T17:37:18.717978+0000","last_fullsized":"2024-08-17T17:37:18.717978+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:22:52.544851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":617,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.723148+0000","last_change":"2024-08-17T17:35:02.239838+0000","last_active":"2024-08-17T17:37:19.723148+0000","last_peered":"2024-08-17T17:37:19.723148+0000","last_clean":"2024-08-17T17:37:19.723148+0000","last_became_active":"2024-08-17T17:34:17.888626+0000","last_became_peered":"2024-08-17T17:34:17.888626+0000","last_unstale":"2024-08-17T17:37:19.723148+0000","last_undegraded":"2024-08-17T17:37:19.723148+0000","last_fullsized":"2024-08-17T17:37:19.723148+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:35:02.239705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:35:02.239705+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:43:42.111630+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":66,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-08-17T17:37:18.718026+0000","last_change":"2024-08-17T17:33:21.766756+0000","last_active":"2024-08-17T17:37:18.718026+0000","last_peered":"2024-08-17T17:37:18.718026+0000","last_clean":"2024-08-17T17:37:18.718026+0000","last_became_active":"2024-08-17T17:33:21.766347+0000","last_became_peered":"2024-08-17T17:33:21.766347+0000","last_unstale":"2024-08-17T17:37:18.718026+0000","last_undegraded":"2024-08-17T17:37:18.718026+0000","last_fullsized":"2024-08-17T17:37:18.718026+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:13:14.655176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-08-17T17:37:18.718112+0000","last_change":"2024-08-17T17:37:00.386391+0000","last_active":"2024-08-17T17:37:18.718112+0000","last_peered":"2024-08-17T17:37:18.718112+0000","last_clean":"2024-08-17T17:37:18.718112+0000","last_became_active":"2024-08-17T17:37:00.385803+0000","last_became_peered":"2024-08-17T17:37:00.385803+0000","last_unstale":"2024-08-17T17:37:18.718112+0000","last_undegraded":"2024-08-17T17:37:18.718112+0000","last_fullsized":"2024-08-17T17:37:18.718112+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:59:49.051157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-08-17T17:37:18.717574+0000","last_change":"2024-08-17T17:37:00.376047+0000","last_active":"2024-08-17T17:37:18.717574+0000","last_peered":"2024-08-17T17:37:18.717574+0000","last_clean":"2024-08-17T17:37:18.717574+0000","last_became_active":"2024-08-17T17:37:00.375687+0000","last_became_peered":"2024-08-17T17:37:00.375687+0000","last_unstale":"2024-08-17T17:37:18.717574+0000","last_undegraded":"2024-08-17T17:37:18.717574+0000","last_fullsized":"2024-08-17T17:37:18.717574+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:57:17.605170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":66,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-08-17T17:37:18.717979+0000","last_change":"2024-08-17T17:33:21.764934+0000","last_active":"2024-08-17T17:37:18.717979+0000","last_peered":"2024-08-17T17:37:18.717979+0000","last_clean":"2024-08-17T17:37:18.717979+0000","last_became_active":"2024-08-17T17:33:21.764341+0000","last_became_peered":"2024-08-17T17:33:21.764341+0000","last_unstale":"2024-08-17T17:37:18.717979+0000","last_undegraded":"2024-08-17T17:37:18.717979+0000","last_fullsized":"2024-08-17T17:37:18.717979+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:20:25.714104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":620,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.723242+0000","last_change":"2024-08-17T17:35:09.302863+0000","last_active":"2024-08-17T17:37:19.723242+0000","last_peered":"2024-08-17T17:37:19.723242+0000","last_clean":"2024-08-17T17:37:19.723242+0000","last_became_active":"2024-08-17T17:34:17.889755+0000","last_became_peered":"2024-08-17T17:34:17.889755+0000","last_unstale":"2024-08-17T17:37:19.723242+0000","last_undegraded":"2024-08-17T17:37:19.723242+0000","last_fullsized":"2024-08-17T17:37:19.723242+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:35:09.302394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:35:09.302394+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:16:17.433313+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":491,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:20.846181+0000","last_change":"2024-08-17T17:33:20.796216+0000","last_active":"2024-08-17T17:37:20.846181+0000","last_peered":"2024-08-17T17:37:20.846181+0000","last_clean":"2024-08-17T17:37:20.846181+0000","last_became_active":"2024-08-17T17:33:20.795762+0000","last_became_peered":"2024-08-17T17:33:20.795762+0000","last_unstale":"2024-08-17T17:37:20.846181+0000","last_undegraded":"2024-08-17T17:37:20.846181+0000","last_fullsized":"2024-08-17T17:37:20.846181+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:47.038509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":372,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.722724+0000","last_change":"2024-08-17T17:33:21.976938+0000","last_active":"2024-08-17T17:37:19.722724+0000","last_peered":"2024-08-17T17:37:19.722724+0000","last_clean":"2024-08-17T17:37:19.722724+0000","last_became_active":"2024-08-17T17:33:21.976585+0000","last_became_peered":"2024-08-17T17:33:21.976585+0000","last_unstale":"2024-08-17T17:37:19.722724+0000","last_undegraded":"2024-08-17T17:37:19.722724+0000","last_fullsized":"2024-08-17T17:37:19.722724+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:23:40.758870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":606,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.723257+0000","last_change":"2024-08-17T17:34:18.238306+0000","last_active":"2024-08-17T17:37:19.723257+0000","last_peered":"2024-08-17T17:37:19.723257+0000","last_clean":"2024-08-17T17:37:19.723257+0000","last_became_active":"2024-08-17T17:34:17.878280+0000","last_became_peered":"2024-08-17T17:34:17.878280+0000","last_unstale":"2024-08-17T17:37:19.723257+0000","last_undegraded":"2024-08-17T17:37:19.723257+0000","last_fullsized":"2024-08-17T17:37:19.723257+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:34:18.238171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:34:18.238171+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:35:36.485730+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":709,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-08-17T17:37:18.731875+0000","last_change":"2024-08-17T17:33:44.149928+0000","last_active":"2024-08-17T17:37:18.731875+0000","last_peered":"2024-08-17T17:37:18.731875+0000","last_clean":"2024-08-17T17:37:18.731875+0000","last_became_active":"2024-08-17T17:33:44.149498+0000","last_became_peered":"2024-08-17T17:33:44.149498+0000","last_unstale":"2024-08-17T17:37:18.731875+0000","last_undegraded":"2024-08-17T17:37:18.731875+0000","last_fullsized":"2024-08-17T17:37:18.731875+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:17:04.963202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":438,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:20.845798+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:37:20.845798+0000","last_peered":"2024-08-17T17:37:20.845798+0000","last_clean":"2024-08-17T17:37:20.845798+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:37:20.845798+0000","last_undegraded":"2024-08-17T17:37:20.845798+0000","last_fullsized":"2024-08-17T17:37:20.845798+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":125,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-08-17T17:37:18.731665+0000","last_change":"2024-08-17T17:37:00.384064+0000","last_active":"2024-08-17T17:37:18.731665+0000","last_peered":"2024-08-17T17:37:18.731665+0000","last_clean":"2024-08-17T17:37:18.731665+0000","last_became_active":"2024-08-17T17:37:00.383681+0000","last_became_peered":"2024-08-17T17:37:00.383681+0000","last_unstale":"2024-08-17T17:37:18.731665+0000","last_undegraded":"2024-08-17T17:37:18.731665+0000","last_fullsized":"2024-08-17T17:37:18.731665+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:10:22.569694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":620,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.723607+0000","last_change":"2024-08-17T17:35:07.285757+0000","last_active":"2024-08-17T17:37:19.723607+0000","last_peered":"2024-08-17T17:37:19.723607+0000","last_clean":"2024-08-17T17:37:19.723607+0000","last_became_active":"2024-08-17T17:34:17.886791+0000","last_became_peered":"2024-08-17T17:34:17.886791+0000","last_unstale":"2024-08-17T17:37:19.723607+0000","last_undegraded":"2024-08-17T17:37:19.723607+0000","last_fullsized":"2024-08-17T17:37:19.723607+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:35:07.285633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:35:07.285633+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:32:46.956526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":723,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.721275+0000","last_change":"2024-08-17T17:33:20.797705+0000","last_active":"2024-08-17T17:37:19.721275+0000","last_peered":"2024-08-17T17:37:19.721275+0000","last_clean":"2024-08-17T17:37:19.721275+0000","last_became_active":"2024-08-17T17:33:20.797172+0000","last_became_peered":"2024-08-17T17:33:20.797172+0000","last_unstale":"2024-08-17T17:37:19.721275+0000","last_undegraded":"2024-08-17T17:37:19.721275+0000","last_fullsized":"2024-08-17T17:37:19.721275+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:30.754999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":474,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.723594+0000","last_change":"2024-08-17T17:37:00.411061+0000","last_active":"2024-08-17T17:37:19.723594+0000","last_peered":"2024-08-17T17:37:19.723594+0000","last_clean":"2024-08-17T17:37:19.723594+0000","last_became_active":"2024-08-17T17:37:00.410374+0000","last_became_peered":"2024-08-17T17:37:00.410374+0000","last_unstale":"2024-08-17T17:37:19.723594+0000","last_undegraded":"2024-08-17T17:37:19.723594+0000","last_fullsized":"2024-08-17T17:37:19.723594+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:09:04.878864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":276,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.724856+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:37:19.724856+0000","last_peered":"2024-08-17T17:37:19.724856+0000","last_clean":"2024-08-17T17:37:19.724856+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:37:19.724856+0000","last_undegraded":"2024-08-17T17:37:19.724856+0000","last_fullsized":"2024-08-17T17:37:19.724856+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":617,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.723285+0000","last_change":"2024-08-17T17:35:01.201171+0000","last_active":"2024-08-17T17:37:19.723285+0000","last_peered":"2024-08-17T17:37:19.723285+0000","last_clean":"2024-08-17T17:37:19.723285+0000","last_became_active":"2024-08-17T17:34:17.890103+0000","last_became_peered":"2024-08-17T17:34:17.890103+0000","last_unstale":"2024-08-17T17:37:19.723285+0000","last_undegraded":"2024-08-17T17:37:19.723285+0000","last_fullsized":"2024-08-17T17:37:19.723285+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:35:01.201049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:35:01.201049+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:01:32.456637+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":855,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:20.845926+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:37:20.845926+0000","last_peered":"2024-08-17T17:37:20.845926+0000","last_clean":"2024-08-17T17:37:20.845926+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:37:20.845926+0000","last_undegraded":"2024-08-17T17:37:20.845926+0000","last_fullsized":"2024-08-17T17:37:20.845926+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":361,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.721313+0000","last_change":"2024-08-17T17:33:20.793468+0000","last_active":"2024-08-17T17:37:19.721313+0000","last_peered":"2024-08-17T17:37:19.721313+0000","last_clean":"2024-08-17T17:37:19.721313+0000","last_became_active":"2024-08-17T17:33:20.793166+0000","last_became_peered":"2024-08-17T17:33:20.793166+0000","last_unstale":"2024-08-17T17:37:19.721313+0000","last_undegraded":"2024-08-17T17:37:19.721313+0000","last_fullsized":"2024-08-17T17:37:19.721313+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:12:52.121050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":396,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.722635+0000","last_change":"2024-08-17T17:33:20.835784+0000","last_active":"2024-08-17T17:37:19.722635+0000","last_peered":"2024-08-17T17:37:19.722635+0000","last_clean":"2024-08-17T17:37:19.722635+0000","last_became_active":"2024-08-17T17:33:20.835371+0000","last_became_peered":"2024-08-17T17:33:20.835371+0000","last_unstale":"2024-08-17T17:37:19.722635+0000","last_undegraded":"2024-08-17T17:37:19.722635+0000","last_fullsized":"2024-08-17T17:37:19.722635+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:35:43.369055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":616,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.723389+0000","last_change":"2024-08-17T17:35:03.210422+0000","last_active":"2024-08-17T17:37:19.723389+0000","last_peered":"2024-08-17T17:37:19.723389+0000","last_clean":"2024-08-17T17:37:19.723389+0000","last_became_active":"2024-08-17T17:34:17.898012+0000","last_became_peered":"2024-08-17T17:34:17.898012+0000","last_unstale":"2024-08-17T17:37:19.723389+0000","last_undegraded":"2024-08-17T17:37:19.723389+0000","last_fullsized":"2024-08-17T17:37:19.723389+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:35:03.210298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:35:03.210298+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:16:50.801783+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":945,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.721547+0000","last_change":"2024-08-17T17:33:44.148227+0000","last_active":"2024-08-17T17:37:19.721547+0000","last_peered":"2024-08-17T17:37:19.721547+0000","last_clean":"2024-08-17T17:37:19.721547+0000","last_became_active":"2024-08-17T17:33:44.147627+0000","last_became_peered":"2024-08-17T17:33:44.147627+0000","last_unstale":"2024-08-17T17:37:19.721547+0000","last_undegraded":"2024-08-17T17:37:19.721547+0000","last_fullsized":"2024-08-17T17:37:19.721547+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:29:42.898460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1372,"num_read_kb":2843,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":52,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-08-17T17:37:18.718334+0000","last_change":"2024-08-17T17:32:46.379200+0000","last_active":"2024-08-17T17:37:18.718334+0000","last_peered":"2024-08-17T17:37:18.718334+0000","last_clean":"2024-08-17T17:37:18.718334+0000","last_became_active":"2024-08-17T17:32:46.378870+0000","last_became_peered":"2024-08-17T17:32:46.378870+0000","last_unstale":"2024-08-17T17:37:18.718334+0000","last_undegraded":"2024-08-17T17:37:18.718334+0000","last_fullsized":"2024-08-17T17:37:18.718334+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:55:26.885966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":400,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.722493+0000","last_change":"2024-08-17T17:33:20.772454+0000","last_active":"2024-08-17T17:37:19.722493+0000","last_peered":"2024-08-17T17:37:19.722493+0000","last_clean":"2024-08-17T17:37:19.722493+0000","last_became_active":"2024-08-17T17:33:20.771848+0000","last_became_peered":"2024-08-17T17:33:20.771848+0000","last_unstale":"2024-08-17T17:37:19.722493+0000","last_undegraded":"2024-08-17T17:37:19.722493+0000","last_fullsized":"2024-08-17T17:37:19.722493+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:46:42.300744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":620,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.724056+0000","last_change":"2024-08-17T17:35:05.269962+0000","last_active":"2024-08-17T17:37:19.724056+0000","last_peered":"2024-08-17T17:37:19.724056+0000","last_clean":"2024-08-17T17:37:19.724056+0000","last_became_active":"2024-08-17T17:34:17.885483+0000","last_became_peered":"2024-08-17T17:34:17.885483+0000","last_unstale":"2024-08-17T17:37:19.724056+0000","last_undegraded":"2024-08-17T17:37:19.724056+0000","last_fullsized":"2024-08-17T17:37:19.724056+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:35:05.269844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:35:05.269844+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:09:31.270296+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":150,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:20.845472+0000","last_change":"2024-08-17T17:33:44.163867+0000","last_active":"2024-08-17T17:37:20.845472+0000","last_peered":"2024-08-17T17:37:20.845472+0000","last_clean":"2024-08-17T17:37:20.845472+0000","last_became_active":"2024-08-17T17:33:44.163503+0000","last_became_peered":"2024-08-17T17:33:44.163503+0000","last_unstale":"2024-08-17T17:37:20.845472+0000","last_undegraded":"2024-08-17T17:37:20.845472+0000","last_fullsized":"2024-08-17T17:37:20.845472+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:31:53.573596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":615,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.721614+0000","last_change":"2024-08-17T17:33:20.799350+0000","last_active":"2024-08-17T17:37:19.721614+0000","last_peered":"2024-08-17T17:37:19.721614+0000","last_clean":"2024-08-17T17:37:19.721614+0000","last_became_active":"2024-08-17T17:33:20.798925+0000","last_became_peered":"2024-08-17T17:33:20.798925+0000","last_unstale":"2024-08-17T17:37:19.721614+0000","last_undegraded":"2024-08-17T17:37:19.721614+0000","last_fullsized":"2024-08-17T17:37:19.721614+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:16:36.121295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":417,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.721789+0000","last_change":"2024-08-17T17:37:00.397108+0000","last_active":"2024-08-17T17:37:19.721789+0000","last_peered":"2024-08-17T17:37:19.721789+0000","last_clean":"2024-08-17T17:37:19.721789+0000","last_became_active":"2024-08-17T17:37:00.395931+0000","last_became_peered":"2024-08-17T17:37:00.395931+0000","last_unstale":"2024-08-17T17:37:19.721789+0000","last_undegraded":"2024-08-17T17:37:19.721789+0000","last_fullsized":"2024-08-17T17:37:19.721789+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:54:31.517826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":620,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.724543+0000","last_change":"2024-08-17T17:35:08.269771+0000","last_active":"2024-08-17T17:37:19.724543+0000","last_peered":"2024-08-17T17:37:19.724543+0000","last_clean":"2024-08-17T17:37:19.724543+0000","last_became_active":"2024-08-17T17:34:17.887618+0000","last_became_peered":"2024-08-17T17:34:17.887618+0000","last_unstale":"2024-08-17T17:37:19.724543+0000","last_undegraded":"2024-08-17T17:37:19.724543+0000","last_fullsized":"2024-08-17T17:37:19.724543+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:35:08.269476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:35:08.269476+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:10:44.708147+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":435,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.721526+0000","last_change":"2024-08-17T17:33:20.798613+0000","last_active":"2024-08-17T17:37:19.721526+0000","last_peered":"2024-08-17T17:37:19.721526+0000","last_clean":"2024-08-17T17:37:19.721526+0000","last_became_active":"2024-08-17T17:33:20.798260+0000","last_became_peered":"2024-08-17T17:33:20.798260+0000","last_unstale":"2024-08-17T17:37:19.721526+0000","last_undegraded":"2024-08-17T17:37:19.721526+0000","last_fullsized":"2024-08-17T17:37:19.721526+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:18:00.822681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":779,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-08-17T17:37:18.732445+0000","last_change":"2024-08-17T17:32:49.379365+0000","last_active":"2024-08-17T17:37:18.732445+0000","last_peered":"2024-08-17T17:37:18.732445+0000","last_clean":"2024-08-17T17:37:18.732445+0000","last_became_active":"2024-08-17T17:32:49.378742+0000","last_became_peered":"2024-08-17T17:32:49.378742+0000","last_unstale":"2024-08-17T17:37:18.732445+0000","last_undegraded":"2024-08-17T17:37:18.732445+0000","last_fullsized":"2024-08-17T17:37:18.732445+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:32:06.957948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":450,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.723522+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:37:19.723522+0000","last_peered":"2024-08-17T17:37:19.723522+0000","last_clean":"2024-08-17T17:37:19.723522+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:37:19.723522+0000","last_undegraded":"2024-08-17T17:37:19.723522+0000","last_fullsized":"2024-08-17T17:37:19.723522+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":620,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.723806+0000","last_change":"2024-08-17T17:35:06.281979+0000","last_active":"2024-08-17T17:37:19.723806+0000","last_peered":"2024-08-17T17:37:19.723806+0000","last_clean":"2024-08-17T17:37:19.723806+0000","last_became_active":"2024-08-17T17:34:17.886175+0000","last_became_peered":"2024-08-17T17:34:17.886175+0000","last_unstale":"2024-08-17T17:37:19.723806+0000","last_undegraded":"2024-08-17T17:37:19.723806+0000","last_fullsized":"2024-08-17T17:37:19.723806+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:35:06.281848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:35:06.281848+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:50:28.599262+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":485,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:20.846524+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:37:20.846524+0000","last_peered":"2024-08-17T17:37:20.846524+0000","last_clean":"2024-08-17T17:37:20.846524+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:37:20.846524+0000","last_undegraded":"2024-08-17T17:37:20.846524+0000","last_fullsized":"2024-08-17T17:37:20.846524+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":968,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.722082+0000","last_change":"2024-08-17T17:33:21.763805+0000","last_active":"2024-08-17T17:37:19.722082+0000","last_peered":"2024-08-17T17:37:19.722082+0000","last_clean":"2024-08-17T17:37:19.722082+0000","last_became_active":"2024-08-17T17:33:21.763232+0000","last_became_peered":"2024-08-17T17:33:21.763232+0000","last_unstale":"2024-08-17T17:37:19.722082+0000","last_undegraded":"2024-08-17T17:37:19.722082+0000","last_fullsized":"2024-08-17T17:37:19.722082+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:46:08.994820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":481,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:20.845734+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:37:20.845734+0000","last_peered":"2024-08-17T17:37:20.845734+0000","last_clean":"2024-08-17T17:37:20.845734+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:37:20.845734+0000","last_undegraded":"2024-08-17T17:37:20.845734+0000","last_fullsized":"2024-08-17T17:37:20.845734+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":605,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.723563+0000","last_change":"2024-08-17T17:34:19.224000+0000","last_active":"2024-08-17T17:37:19.723563+0000","last_peered":"2024-08-17T17:37:19.723563+0000","last_clean":"2024-08-17T17:37:19.723563+0000","last_became_active":"2024-08-17T17:34:17.885462+0000","last_became_peered":"2024-08-17T17:34:17.885462+0000","last_unstale":"2024-08-17T17:37:19.723563+0000","last_undegraded":"2024-08-17T17:37:19.723563+0000","last_fullsized":"2024-08-17T17:37:19.723563+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:34:19.223644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:34:19.223644+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:19:57.910956+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":300,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.724750+0000","last_change":"2024-08-17T17:33:21.765925+0000","last_active":"2024-08-17T17:37:19.724750+0000","last_peered":"2024-08-17T17:37:19.724750+0000","last_clean":"2024-08-17T17:37:19.724750+0000","last_became_active":"2024-08-17T17:33:21.765372+0000","last_became_peered":"2024-08-17T17:33:21.765372+0000","last_unstale":"2024-08-17T17:37:19.724750+0000","last_undegraded":"2024-08-17T17:37:19.724750+0000","last_fullsized":"2024-08-17T17:37:19.724750+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:14:47.737369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":534,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-08-17T17:37:18.732751+0000","last_change":"2024-08-17T17:26:06.855307+0000","last_active":"2024-08-17T17:37:18.732751+0000","last_peered":"2024-08-17T17:37:18.732751+0000","last_clean":"2024-08-17T17:37:18.732751+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:37:18.732751+0000","last_undegraded":"2024-08-17T17:37:18.732751+0000","last_fullsized":"2024-08-17T17:37:18.732751+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-08-17T17:37:18.717635+0000","last_change":"2024-08-17T17:37:00.382745+0000","last_active":"2024-08-17T17:37:18.717635+0000","last_peered":"2024-08-17T17:37:18.717635+0000","last_clean":"2024-08-17T17:37:18.717635+0000","last_became_active":"2024-08-17T17:37:00.382032+0000","last_became_peered":"2024-08-17T17:37:00.382032+0000","last_unstale":"2024-08-17T17:37:18.717635+0000","last_undegraded":"2024-08-17T17:37:18.717635+0000","last_fullsized":"2024-08-17T17:37:18.717635+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:24:51.920975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":66,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-08-17T17:37:18.718358+0000","last_change":"2024-08-17T17:33:21.763655+0000","last_active":"2024-08-17T17:37:18.718358+0000","last_peered":"2024-08-17T17:37:18.718358+0000","last_clean":"2024-08-17T17:37:18.718358+0000","last_became_active":"2024-08-17T17:33:21.763088+0000","last_became_peered":"2024-08-17T17:33:21.763088+0000","last_unstale":"2024-08-17T17:37:18.718358+0000","last_undegraded":"2024-08-17T17:37:18.718358+0000","last_fullsized":"2024-08-17T17:37:18.718358+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:40:40.126898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":23,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-08-17T17:37:18.718069+0000","last_change":"2024-08-17T17:37:00.413709+0000","last_active":"2024-08-17T17:37:18.718069+0000","last_peered":"2024-08-17T17:37:18.718069+0000","last_clean":"2024-08-17T17:37:18.718069+0000","last_became_active":"2024-08-17T17:37:00.383355+0000","last_became_peered":"2024-08-17T17:37:00.383355+0000","last_unstale":"2024-08-17T17:37:18.718069+0000","last_undegraded":"2024-08-17T17:37:18.718069+0000","last_fullsized":"2024-08-17T17:37:18.718069+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:47:24.501358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":428,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.721646+0000","last_change":"2024-08-17T17:33:20.799402+0000","last_active":"2024-08-17T17:37:19.721646+0000","last_peered":"2024-08-17T17:37:19.721646+0000","last_clean":"2024-08-17T17:37:19.721646+0000","last_became_active":"2024-08-17T17:33:20.798832+0000","last_became_peered":"2024-08-17T17:33:20.798832+0000","last_unstale":"2024-08-17T17:37:19.721646+0000","last_undegraded":"2024-08-17T17:37:19.721646+0000","last_fullsized":"2024-08-17T17:37:19.721646+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:21:05.001134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":177,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:20.846090+0000","last_change":"2024-08-17T17:32:46.381141+0000","last_active":"2024-08-17T17:37:20.846090+0000","last_peered":"2024-08-17T17:37:20.846090+0000","last_clean":"2024-08-17T17:37:20.846090+0000","last_became_active":"2024-08-17T17:32:46.374259+0000","last_became_peered":"2024-08-17T17:32:46.374259+0000","last_unstale":"2024-08-17T17:37:20.846090+0000","last_undegraded":"2024-08-17T17:37:20.846090+0000","last_fullsized":"2024-08-17T17:37:20.846090+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:19:10.836889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":167,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.722410+0000","last_change":"2024-08-17T17:33:44.146014+0000","last_active":"2024-08-17T17:37:19.722410+0000","last_peered":"2024-08-17T17:37:19.722410+0000","last_clean":"2024-08-17T17:37:19.722410+0000","last_became_active":"2024-08-17T17:33:44.145668+0000","last_became_peered":"2024-08-17T17:33:44.145668+0000","last_unstale":"2024-08-17T17:37:19.722410+0000","last_undegraded":"2024-08-17T17:37:19.722410+0000","last_fullsized":"2024-08-17T17:37:19.722410+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:53:41.679237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":431,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:20.845695+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:37:20.845695+0000","last_peered":"2024-08-17T17:37:20.845695+0000","last_clean":"2024-08-17T17:37:20.845695+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:37:20.845695+0000","last_undegraded":"2024-08-17T17:37:20.845695+0000","last_fullsized":"2024-08-17T17:37:20.845695+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":458,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.724355+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:37:19.724355+0000","last_peered":"2024-08-17T17:37:19.724355+0000","last_clean":"2024-08-17T17:37:19.724355+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:37:19.724355+0000","last_undegraded":"2024-08-17T17:37:19.724355+0000","last_fullsized":"2024-08-17T17:37:19.724355+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":489,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:20.845636+0000","last_change":"2024-08-17T17:33:20.790687+0000","last_active":"2024-08-17T17:37:20.845636+0000","last_peered":"2024-08-17T17:37:20.845636+0000","last_clean":"2024-08-17T17:37:20.845636+0000","last_became_active":"2024-08-17T17:33:20.790399+0000","last_became_peered":"2024-08-17T17:33:20.790399+0000","last_unstale":"2024-08-17T17:37:20.845636+0000","last_undegraded":"2024-08-17T17:37:20.845636+0000","last_fullsized":"2024-08-17T17:37:20.845636+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:38:06.849989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":277,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-08-17T17:37:18.732703+0000","last_change":"2024-08-17T17:33:20.835760+0000","last_active":"2024-08-17T17:37:18.732703+0000","last_peered":"2024-08-17T17:37:18.732703+0000","last_clean":"2024-08-17T17:37:18.732703+0000","last_became_active":"2024-08-17T17:33:20.835410+0000","last_became_peered":"2024-08-17T17:33:20.835410+0000","last_unstale":"2024-08-17T17:37:18.732703+0000","last_undegraded":"2024-08-17T17:37:18.732703+0000","last_fullsized":"2024-08-17T17:37:18.732703+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:45.448022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":507,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:20.846092+0000","last_change":"2024-08-17T17:33:20.771279+0000","last_active":"2024-08-17T17:37:20.846092+0000","last_peered":"2024-08-17T17:37:20.846092+0000","last_clean":"2024-08-17T17:37:20.846092+0000","last_became_active":"2024-08-17T17:33:20.770728+0000","last_became_peered":"2024-08-17T17:33:20.770728+0000","last_unstale":"2024-08-17T17:37:20.846092+0000","last_undegraded":"2024-08-17T17:37:20.846092+0000","last_fullsized":"2024-08-17T17:37:20.846092+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:52:31.540397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":276,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.724655+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:37:19.724655+0000","last_peered":"2024-08-17T17:37:19.724655+0000","last_clean":"2024-08-17T17:37:19.724655+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:37:19.724655+0000","last_undegraded":"2024-08-17T17:37:19.724655+0000","last_fullsized":"2024-08-17T17:37:19.724655+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":413,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.722387+0000","last_change":"2024-08-17T17:37:00.403306+0000","last_active":"2024-08-17T17:37:19.722387+0000","last_peered":"2024-08-17T17:37:19.722387+0000","last_clean":"2024-08-17T17:37:19.722387+0000","last_became_active":"2024-08-17T17:37:00.403030+0000","last_became_peered":"2024-08-17T17:37:00.403030+0000","last_unstale":"2024-08-17T17:37:19.722387+0000","last_undegraded":"2024-08-17T17:37:19.722387+0000","last_fullsized":"2024-08-17T17:37:19.722387+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:19:07.078877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":84,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-08-17T17:37:18.718035+0000","last_change":"2024-08-17T17:33:44.160733+0000","last_active":"2024-08-17T17:37:18.718035+0000","last_peered":"2024-08-17T17:37:18.718035+0000","last_clean":"2024-08-17T17:37:18.718035+0000","last_became_active":"2024-08-17T17:33:44.160368+0000","last_became_peered":"2024-08-17T17:33:44.160368+0000","last_unstale":"2024-08-17T17:37:18.718035+0000","last_undegraded":"2024-08-17T17:37:18.718035+0000","last_fullsized":"2024-08-17T17:37:18.718035+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:37:22.855642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":13,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-08-17T17:37:18.717832+0000","last_change":"2024-08-17T17:37:00.405506+0000","last_active":"2024-08-17T17:37:18.717832+0000","last_peered":"2024-08-17T17:37:18.717832+0000","last_clean":"2024-08-17T17:37:18.717832+0000","last_became_active":"2024-08-17T17:37:00.400982+0000","last_became_peered":"2024-08-17T17:37:00.400982+0000","last_unstale":"2024-08-17T17:37:18.717832+0000","last_undegraded":"2024-08-17T17:37:18.717832+0000","last_fullsized":"2024-08-17T17:37:18.717832+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:47:17.134917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":394,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.721670+0000","last_change":"2024-08-17T17:33:20.773207+0000","last_active":"2024-08-17T17:37:19.721670+0000","last_peered":"2024-08-17T17:37:19.721670+0000","last_clean":"2024-08-17T17:37:19.721670+0000","last_became_active":"2024-08-17T17:33:20.772457+0000","last_became_peered":"2024-08-17T17:33:20.772457+0000","last_unstale":"2024-08-17T17:37:19.721670+0000","last_undegraded":"2024-08-17T17:37:19.721670+0000","last_fullsized":"2024-08-17T17:37:19.721670+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:57:09.610084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":286,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.723390+0000","last_change":"2024-08-17T17:32:46.375464+0000","last_active":"2024-08-17T17:37:19.723390+0000","last_peered":"2024-08-17T17:37:19.723390+0000","last_clean":"2024-08-17T17:37:19.723390+0000","last_became_active":"2024-08-17T17:32:46.375161+0000","last_became_peered":"2024-08-17T17:32:46.375161+0000","last_unstale":"2024-08-17T17:37:19.723390+0000","last_undegraded":"2024-08-17T17:37:19.723390+0000","last_fullsized":"2024-08-17T17:37:19.723390+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:27:47.091836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-08-17T17:37:18.718167+0000","last_change":"2024-08-17T17:37:00.447160+0000","last_active":"2024-08-17T17:37:18.718167+0000","last_peered":"2024-08-17T17:37:18.718167+0000","last_clean":"2024-08-17T17:37:18.718167+0000","last_became_active":"2024-08-17T17:37:00.446710+0000","last_became_peered":"2024-08-17T17:37:00.446710+0000","last_unstale":"2024-08-17T17:37:18.718167+0000","last_undegraded":"2024-08-17T17:37:18.718167+0000","last_fullsized":"2024-08-17T17:37:18.718167+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:51:43.200559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":317,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-08-17T17:37:18.717914+0000","last_change":"2024-08-17T17:33:44.165175+0000","last_active":"2024-08-17T17:37:18.717914+0000","last_peered":"2024-08-17T17:37:18.717914+0000","last_clean":"2024-08-17T17:37:18.717914+0000","last_became_active":"2024-08-17T17:33:44.146198+0000","last_became_peered":"2024-08-17T17:33:44.146198+0000","last_unstale":"2024-08-17T17:37:18.717914+0000","last_undegraded":"2024-08-17T17:37:18.717914+0000","last_fullsized":"2024-08-17T17:37:18.717914+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:14:56.291775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-08-17T17:37:18.718241+0000","last_change":"2024-08-17T17:37:00.405789+0000","last_active":"2024-08-17T17:37:18.718241+0000","last_peered":"2024-08-17T17:37:18.718241+0000","last_clean":"2024-08-17T17:37:18.718241+0000","last_became_active":"2024-08-17T17:37:00.405044+0000","last_became_peered":"2024-08-17T17:37:00.405044+0000","last_unstale":"2024-08-17T17:37:18.718241+0000","last_undegraded":"2024-08-17T17:37:18.718241+0000","last_fullsized":"2024-08-17T17:37:18.718241+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:15:57.228668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":297,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-08-17T17:37:18.717954+0000","last_change":"2024-08-17T17:33:44.148090+0000","last_active":"2024-08-17T17:37:18.717954+0000","last_peered":"2024-08-17T17:37:18.717954+0000","last_clean":"2024-08-17T17:37:18.717954+0000","last_became_active":"2024-08-17T17:33:44.147539+0000","last_became_peered":"2024-08-17T17:33:44.147539+0000","last_unstale":"2024-08-17T17:37:18.717954+0000","last_undegraded":"2024-08-17T17:37:18.717954+0000","last_fullsized":"2024-08-17T17:37:18.717954+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:52:02.311332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":485,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:20.846009+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:37:20.846009+0000","last_peered":"2024-08-17T17:37:20.846009+0000","last_clean":"2024-08-17T17:37:20.846009+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:37:20.846009+0000","last_undegraded":"2024-08-17T17:37:20.846009+0000","last_fullsized":"2024-08-17T17:37:20.846009+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":485,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:20.845468+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:37:20.845468+0000","last_peered":"2024-08-17T17:37:20.845468+0000","last_clean":"2024-08-17T17:37:20.845468+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:37:20.845468+0000","last_undegraded":"2024-08-17T17:37:20.845468+0000","last_fullsized":"2024-08-17T17:37:20.845468+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":52,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-08-17T17:37:18.717563+0000","last_change":"2024-08-17T17:32:46.376844+0000","last_active":"2024-08-17T17:37:18.717563+0000","last_peered":"2024-08-17T17:37:18.717563+0000","last_clean":"2024-08-17T17:37:18.717563+0000","last_became_active":"2024-08-17T17:32:46.376483+0000","last_became_peered":"2024-08-17T17:32:46.376483+0000","last_unstale":"2024-08-17T17:37:18.717563+0000","last_undegraded":"2024-08-17T17:37:18.717563+0000","last_fullsized":"2024-08-17T17:37:18.717563+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:44:24.358662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":295,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-08-17T17:37:19.724262+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:37:19.724262+0000","last_peered":"2024-08-17T17:37:19.724262+0000","last_clean":"2024-08-17T17:37:19.724262+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:37:19.724262+0000","last_undegraded":"2024-08-17T17:37:19.724262+0000","last_fullsized":"2024-08-17T17:37:19.724262+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":6220,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":21334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19280,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5978,"num_read_kb":12991,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":33,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234367,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":898364,"kb_used_data":8624,"kb_used_omap":1,"kb_used_meta":889726,"kb_avail":92846788,"statfs":{"total":95995035648,"available":95075110912,"internally_reserved":0,"allocated":8830976,"data_stored":7063629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":911079768},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496002,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":895412,"kb_used_data":8612,"kb_used_omap":1,"kb_used_meta":886782,"kb_avail":92849740,"statfs":{"total":95995035648,"available":95078133760,"internally_reserved":0,"allocated":8818688,"data_stored":7063303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":908065112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757639,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":589676,"kb_used_data":3932,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":93155476,"statfs":{"total":95995035648,"available":95391207424,"internally_reserved":0,"allocated":4026368,"data_stored":2274656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019277,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309624,"kb_used_data":3948,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93435528,"statfs":{"total":95995035648,"available":95677980672,"internally_reserved":0,"allocated":4042752,"data_stored":2279218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435504,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":676836,"kb_used_data":3928,"kb_used_omap":4,"kb_used_meta":672891,"kb_avail":93068316,"statfs":{"total":95995035648,"available":95301955584,"internally_reserved":0,"allocated":4022272,"data_stored":2274438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":689040394},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599031,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":622708,"kb_used_data":8040,"kb_used_omap":7,"kb_used_meta":614648,"kb_avail":93122444,"statfs":{"total":95995035648,"available":95357382656,"internally_reserved":0,"allocated":8232960,"data_stored":6474000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":629399714},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":268,"seq":1151051235378,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":210784,"kb_used_data":3928,"kb_used_omap":10,"kb_used_meta":206837,"kb_avail":93534368,"statfs":{"total":95995035648,"available":95779192832,"internally_reserved":0,"allocated":4022272,"data_stored":2274383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10902,"internal_metadata":211801450},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926084,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":595120,"kb_used_data":4512,"kb_used_omap":7,"kb_used_meta":590584,"kb_avail":93150032,"statfs":{"total":95995035648,"available":95385632768,"internally_reserved":0,"allocated":4620288,"data_stored":2865316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":604758337},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7550,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:37:29.825 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-08-17T17:37:29.825 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:37:29.825 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-08-17T17:37:29.825 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.480950 2024-08-17T17:37:29.826 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 7 0.48095017357254866 2024-08-17T17:37:29.840 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:29.891 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:29.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:29.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:29.992 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:30.015 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:30.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:30.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:30.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:29 smithi003 ceph-mon[103344]: pgmap v962: 83 pgs: 83 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-08-17T17:37:30.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:29 smithi003 ceph-mon[103344]: from='client.46247 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:37:30.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:29 smithi003 ceph-mon[103344]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14168/84 objects degraded (16866.667%), 5 pgs degraded, 5 pgs undersized) 2024-08-17T17:37:30.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:29 smithi003 ceph-mon[103344]: Cluster is now healthy 2024-08-17T17:37:30.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:29 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-08-17T17:37:30.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:29 smithi003 ceph-mon[103344]: osdmap e305: 8 total, 8 up, 8 in 2024-08-17T17:37:30.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:29 smithi003 ceph-mon[100615]: pgmap v962: 83 pgs: 83 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-08-17T17:37:30.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:29 smithi003 ceph-mon[100615]: from='client.46247 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:37:30.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:29 smithi003 ceph-mon[100615]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14168/84 objects degraded (16866.667%), 5 pgs degraded, 5 pgs undersized) 2024-08-17T17:37:30.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:29 smithi003 ceph-mon[100615]: Cluster is now healthy 2024-08-17T17:37:30.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:29 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-08-17T17:37:30.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:29 smithi003 ceph-mon[100615]: osdmap e305: 8 total, 8 up, 8 in 2024-08-17T17:37:30.117 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:30.144 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:30.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:30.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:30.213 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:29 smithi053 ceph-mon[89963]: pgmap v962: 83 pgs: 83 active+clean; 4.6 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-08-17T17:37:30.213 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:29 smithi053 ceph-mon[89963]: from='client.46247 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:37:30.214 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:29 smithi053 ceph-mon[89963]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14168/84 objects degraded (16866.667%), 5 pgs degraded, 5 pgs undersized) 2024-08-17T17:37:30.214 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:29 smithi053 ceph-mon[89963]: Cluster is now healthy 2024-08-17T17:37:30.214 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:29 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-08-17T17:37:30.214 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:29 smithi053 ceph-mon[89963]: osdmap e305: 8 total, 8 up, 8 in 2024-08-17T17:37:30.247 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:30.272 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:30.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:30.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:30.374 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:30.400 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:30.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:30.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:30.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:37:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:37:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:37:30.503 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:30.530 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:37:30.531 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:37:30.542 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:30.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:30.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:30.571 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:37:30.644 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:30.670 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:30.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:30.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:30.772 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:30.798 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:30.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:30.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:30.900 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:30.926 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:30.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:30.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:31.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:30 smithi003 ceph-mon[100615]: osdmap e306: 8 total, 8 up, 8 in 2024-08-17T17:37:31.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:30 smithi003 ceph-mon[103344]: osdmap e306: 8 total, 8 up, 8 in 2024-08-17T17:37:31.027 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:31.053 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:31.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:31.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:31.155 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:31.179 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:31.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:31.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:31.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:30 smithi053 ceph-mon[89963]: osdmap e306: 8 total, 8 up, 8 in 2024-08-17T17:37:31.281 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:31.308 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:31.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:31.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:31.410 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:31.439 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:31.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:31.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:31.541 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:31.568 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:31.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:31.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:31.670 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:31.696 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:31.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:31.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:31.798 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:31.823 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:31.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:31.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:31.925 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:31.949 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:31.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:31.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:32.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:31 smithi003 ceph-mon[100615]: pgmap v965: 83 pgs: 3 unknown, 5 peering, 75 active+clean; 4.2 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 37 KiB/s rd, 16 op/s 2024-08-17T17:37:32.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:31 smithi003 ceph-mon[100615]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-08-17T17:37:32.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:31 smithi003 ceph-mon[100615]: osdmap e307: 8 total, 8 up, 8 in 2024-08-17T17:37:32.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:31 smithi003 ceph-mon[103344]: pgmap v965: 83 pgs: 3 unknown, 5 peering, 75 active+clean; 4.2 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 37 KiB/s rd, 16 op/s 2024-08-17T17:37:32.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:31 smithi003 ceph-mon[103344]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-08-17T17:37:32.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:31 smithi003 ceph-mon[103344]: osdmap e307: 8 total, 8 up, 8 in 2024-08-17T17:37:32.051 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:32.076 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:32.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:32.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:32.178 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:32.204 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:32.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:32.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:32.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:31 smithi053 ceph-mon[89963]: pgmap v965: 83 pgs: 3 unknown, 5 peering, 75 active+clean; 4.2 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 37 KiB/s rd, 16 op/s 2024-08-17T17:37:32.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:31 smithi053 ceph-mon[89963]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-08-17T17:37:32.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:31 smithi053 ceph-mon[89963]: osdmap e307: 8 total, 8 up, 8 in 2024-08-17T17:37:32.306 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:32.333 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:32.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:32.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:32.434 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:32.460 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:32.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:32.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:32.477 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:37:32.562 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:32.588 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:32.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:32.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:32.689 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:32.715 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:32.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:32.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:32.817 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:32.843 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:32.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:32.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:32.944 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:32 smithi003 ceph-mon[103344]: osdmap e308: 8 total, 8 up, 8 in 2024-08-17T17:37:32.944 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:32 smithi003 ceph-mon[100615]: osdmap e308: 8 total, 8 up, 8 in 2024-08-17T17:37:32.945 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:32.970 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:32.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:32.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:33.072 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:33.098 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:33.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:33.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:33.200 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:33.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:32 smithi053 ceph-mon[89963]: osdmap e308: 8 total, 8 up, 8 in 2024-08-17T17:37:33.223 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:33.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:33.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:33.325 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:33.352 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:33.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:33.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:33.454 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:33.477 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:33.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:33.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:33.579 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:33.607 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:33.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:33.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:33.709 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:33.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:33.734 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:33.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:33.835 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:33.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:33.859 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:33.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:33.960 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:33.986 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:33.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:33.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:34.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:33 smithi003 ceph-mon[103344]: pgmap v968: 83 pgs: 3 unknown, 5 peering, 75 active+clean; 4.2 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 56 KiB/s rd, 165 KiB/s wr, 39 op/s; 118 KiB/s, 0 objects/s recovering 2024-08-17T17:37:34.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:33 smithi003 ceph-mon[103344]: osdmap e309: 8 total, 8 up, 8 in 2024-08-17T17:37:34.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:33 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:37:34.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:33 smithi003 ceph-mon[100615]: pgmap v968: 83 pgs: 3 unknown, 5 peering, 75 active+clean; 4.2 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 56 KiB/s rd, 165 KiB/s wr, 39 op/s; 118 KiB/s, 0 objects/s recovering 2024-08-17T17:37:34.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:33 smithi003 ceph-mon[100615]: osdmap e309: 8 total, 8 up, 8 in 2024-08-17T17:37:34.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:33 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:37:34.067 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:37:34.068 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-08-17T17:37:34.068 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-08-17T17:37:34.068 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-08-17T17:37:34.068 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:37:34.088 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:34.114 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:34.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:34.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:34.216 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:34.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:33 smithi053 ceph-mon[89963]: pgmap v968: 83 pgs: 3 unknown, 5 peering, 75 active+clean; 4.2 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 56 KiB/s rd, 165 KiB/s wr, 39 op/s; 118 KiB/s, 0 objects/s recovering 2024-08-17T17:37:34.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:33 smithi053 ceph-mon[89963]: osdmap e309: 8 total, 8 up, 8 in 2024-08-17T17:37:34.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:33 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:37:34.239 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:34.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:34.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:34.341 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:34.368 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:34.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:34.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:34.470 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:34.493 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:34.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:34.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:34.595 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:34.622 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:34.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:34.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:34.727 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:34.733 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-17T17:37:34.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:34.751 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:34.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:34.853 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:34.876 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:34.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:34.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:34.956 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:34 smithi003 ceph-mon[103344]: 1.6 scrub starts 2024-08-17T17:37:34.956 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:34 smithi003 ceph-mon[103344]: 1.6 scrub ok 2024-08-17T17:37:34.956 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:34 smithi003 ceph-mon[100615]: 1.6 scrub starts 2024-08-17T17:37:34.956 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:34 smithi003 ceph-mon[100615]: 1.6 scrub ok 2024-08-17T17:37:34.978 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:35.005 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:35.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:35.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:35.106 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:35.117 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:34 smithi053 ceph-mon[89963]: 1.6 scrub starts 2024-08-17T17:37:35.117 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:34 smithi053 ceph-mon[89963]: 1.6 scrub ok 2024-08-17T17:37:35.133 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:35.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:35.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:35.235 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:35.261 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:35.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:35.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:35.362 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:35.389 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:35.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:35.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:35.491 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:35.514 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:35.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:35.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:35.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:35.638 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:35.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:35.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:35.739 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:35.761 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:35.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:35.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:35.780 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-08-17T17:37:35.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:35.886 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:35.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:35.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:35.988 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:36.010 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:36.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:36.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:36.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:35 smithi003 ceph-mon[103344]: pgmap v970: 83 pgs: 2 peering, 81 active+clean; 4.7 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-08-17T17:37:36.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:35 smithi003 ceph-mon[103344]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-08-17T17:37:36.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:35 smithi003 ceph-mon[103344]: Cluster is now healthy 2024-08-17T17:37:36.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:35 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3826270221' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:37:36.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:35 smithi003 ceph-mon[100615]: pgmap v970: 83 pgs: 2 peering, 81 active+clean; 4.7 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-08-17T17:37:36.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:35 smithi003 ceph-mon[100615]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-08-17T17:37:36.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:35 smithi003 ceph-mon[100615]: Cluster is now healthy 2024-08-17T17:37:36.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:35 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3826270221' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:37:36.024 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:37:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:37:35] "GET /metrics HTTP/1.1" 200 36460 "" "Prometheus/2.51.0" 2024-08-17T17:37:36.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:36.135 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:36.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:36.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:36.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:35 smithi053 ceph-mon[89963]: pgmap v970: 83 pgs: 2 peering, 81 active+clean; 4.7 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-08-17T17:37:36.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:35 smithi053 ceph-mon[89963]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-08-17T17:37:36.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:35 smithi053 ceph-mon[89963]: Cluster is now healthy 2024-08-17T17:37:36.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:35 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3826270221' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:37:36.237 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:36.259 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:36.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:36.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:36.361 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:36.388 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:36.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:36.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:36.490 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:36.516 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:36.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:36.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:36.618 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:36.642 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:36.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:36.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:36.744 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:36.766 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:36.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:36.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:36.868 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:36.891 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:36.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:36.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:36.993 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:36.999 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:36 smithi003 ceph-mon[103344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:37:36.999 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:36 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3826270221' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:37:36.999 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:36 smithi003 ceph-mon[103344]: osdmap e310: 8 total, 8 up, 8 in 2024-08-17T17:37:36.999 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:36 smithi003 ceph-mon[100615]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:37:36.999 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:36 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3826270221' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:37:36.999 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:36 smithi003 ceph-mon[100615]: osdmap e310: 8 total, 8 up, 8 in 2024-08-17T17:37:37.021 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:37.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:37.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:37.123 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:37.151 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:37.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:37.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:37.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:36 smithi053 ceph-mon[89963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:37:37.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:36 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3826270221' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:37:37.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:36 smithi053 ceph-mon[89963]: osdmap e310: 8 total, 8 up, 8 in 2024-08-17T17:37:37.253 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:37.280 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:37.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:37.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:37.382 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:37.404 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:37.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:37.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:37.506 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:37.529 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:37.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:37.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:37.631 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:37.658 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:37.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:37.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:37.725 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:37:37.725 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:37:37.760 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:37.787 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:37.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:37.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:37.794 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_4' created 2024-08-17T17:37:37.795 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:37 smithi003 ceph-mon[103344]: pgmap v972: 83 pgs: 83 active+clean; 4.7 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-08-17T17:37:37.795 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:37 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/889933763' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-08-17T17:37:37.795 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:37 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-08-17T17:37:37.795 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:37 smithi003 ceph-mon[100615]: pgmap v972: 83 pgs: 83 active+clean; 4.7 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-08-17T17:37:37.795 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:37 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/889933763' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-08-17T17:37:37.795 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:37 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-08-17T17:37:37.889 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:37.914 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:37.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:37.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:38.016 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:38.043 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:38.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:38.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:38.145 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:38.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:38.172 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:38.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:38.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:37 smithi053 ceph-mon[89963]: pgmap v972: 83 pgs: 83 active+clean; 4.7 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-08-17T17:37:38.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:37 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/889933763' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-08-17T17:37:38.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:37 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-08-17T17:37:38.274 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:38.297 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:38.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:38.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:38.398 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:38.425 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:38.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:38.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:38.527 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:38.553 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:38.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:38.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:38.655 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:38.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:38.683 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:38.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:38.750 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:37:38.784 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:38.797 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.7 primary-affinity to 0.48095 (8315192) 2024-08-17T17:37:38.815 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:38.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:38.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:38.840 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-08-17T17:37:38.916 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:38.941 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:38.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:38.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:39.043 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:39.067 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:39.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:39.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:39.151 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:38 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-08-17T17:37:39.152 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:38 smithi003 ceph-mon[103344]: osdmap e311: 8 total, 8 up, 8 in 2024-08-17T17:37:39.152 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:38 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3566598177' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.48095017357254866} : dispatch 2024-08-17T17:37:39.152 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:38 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.48095017357254866} : dispatch 2024-08-17T17:37:39.152 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:37:39.152 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:37:39.152 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:38 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-08-17T17:37:39.152 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:38 smithi003 ceph-mon[100615]: osdmap e311: 8 total, 8 up, 8 in 2024-08-17T17:37:39.152 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:38 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3566598177' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.48095017357254866} : dispatch 2024-08-17T17:37:39.152 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:38 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.48095017357254866} : dispatch 2024-08-17T17:37:39.152 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:37:39.152 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:37:39.169 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:39.195 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:39.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:39.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:39.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:38 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-08-17T17:37:39.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:38 smithi053 ceph-mon[89963]: osdmap e311: 8 total, 8 up, 8 in 2024-08-17T17:37:39.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:38 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3566598177' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.48095017357254866} : dispatch 2024-08-17T17:37:39.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:38 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.48095017357254866} : dispatch 2024-08-17T17:37:39.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:37:39.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:37:39.297 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:39.320 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:39.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:39.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:39.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:39.445 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:39.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:39.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:39.547 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:39.574 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:39.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:39.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:39.676 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:39.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:39.700 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:39.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:39.801 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:39.828 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:39.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:39.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:39.930 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:39.957 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:39.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:39.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:40.016 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:39 smithi003 ceph-mon[100615]: pgmap v974: 99 pgs: 16 unknown, 83 active+clean; 4.7 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-08-17T17:37:40.016 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:39 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.48095017357254866}]': finished 2024-08-17T17:37:40.016 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:39 smithi003 ceph-mon[100615]: osdmap e312: 8 total, 8 up, 8 in 2024-08-17T17:37:40.016 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:39 smithi003 ceph-mon[100615]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:37:40.016 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:39 smithi003 ceph-mon[103344]: pgmap v974: 99 pgs: 16 unknown, 83 active+clean; 4.7 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-08-17T17:37:40.017 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:39 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.48095017357254866}]': finished 2024-08-17T17:37:40.017 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:39 smithi003 ceph-mon[103344]: osdmap e312: 8 total, 8 up, 8 in 2024-08-17T17:37:40.017 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:39 smithi003 ceph-mon[103344]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:37:40.058 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:40.085 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:40.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:40.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:40.187 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:40.209 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:40.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:40.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:40.213 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:39 smithi053 ceph-mon[89963]: pgmap v974: 99 pgs: 16 unknown, 83 active+clean; 4.7 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-08-17T17:37:40.213 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:39 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.48095017357254866}]': finished 2024-08-17T17:37:40.213 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:39 smithi053 ceph-mon[89963]: osdmap e312: 8 total, 8 up, 8 in 2024-08-17T17:37:40.213 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:39 smithi053 ceph-mon[89963]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:37:40.311 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:40.339 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:40.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:40.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:40.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:40.463 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:40.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:40.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:40.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:37:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:37:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:37:40.565 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:40.592 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:40.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:40.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:40.693 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:40.717 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:40.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:40.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:40.819 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:40.845 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:40.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:40.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:40.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:40.970 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:40.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:40.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:41.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:40 smithi003 ceph-mon[103344]: osdmap e313: 8 total, 8 up, 8 in 2024-08-17T17:37:41.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:40 smithi003 ceph-mon[100615]: osdmap e313: 8 total, 8 up, 8 in 2024-08-17T17:37:41.072 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:41.099 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:41.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:41.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:41.201 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:41.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:40 smithi053 ceph-mon[89963]: osdmap e313: 8 total, 8 up, 8 in 2024-08-17T17:37:41.227 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:41.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:41.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:41.329 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:41.355 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:41.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:41.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:41.457 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:41.481 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:41.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:41.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:41.567 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:37:41.569 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:37:41.583 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:41.607 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:41.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:41.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:41.709 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:41.736 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:41.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:41.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:41.837 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:41.865 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:41.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:41.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:41.966 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:41.994 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:41.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:41.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:42.095 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:42.122 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:42.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:42.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:42.149 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:41 smithi003 ceph-mon[100615]: pgmap v977: 99 pgs: 6 creating+activating, 93 active+clean; 4.7 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-08-17T17:37:42.149 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:41 smithi003 ceph-mon[103344]: pgmap v977: 99 pgs: 6 creating+activating, 93 active+clean; 4.7 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-08-17T17:37:42.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:41 smithi053 ceph-mon[89963]: pgmap v977: 99 pgs: 6 creating+activating, 93 active+clean; 4.7 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-08-17T17:37:42.225 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:42.248 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:37:42.248 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-08-17T17:37:42.248 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-08-17T17:37:42.248 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:37:42.248 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.248 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:56781 8.3 8.4bf49503 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e268)", 2024-08-17T17:37:42.248 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:19.807785+0000", 2024-08-17T17:37:42.248 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 262.438557344, 2024-08-17T17:37:42.248 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96137050199999996, 2024-08-17T17:37:42.249 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:37:42.249 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:37:42.249 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:37:42.249 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:37:42.249 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:37:42.249 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:37:42.249 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 56781 2024-08-17T17:37:42.249 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.249 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:37:42.249 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.249 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:37:42.249 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807785+0000", 2024-08-17T17:37:42.249 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.249 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.250 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.250 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:37:42.250 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807785+0000", 2024-08-17T17:37:42.250 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.250 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.250 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.250 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:37:42.250 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807786+0000", 2024-08-17T17:37:42.250 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4050000000000001e-06 2024-08-17T17:37:42.250 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.250 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.250 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:37:42.250 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807854+0000", 2024-08-17T17:37:42.251 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7267000000000005e-05 2024-08-17T17:37:42.251 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.251 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.251 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:37:42.251 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807865+0000", 2024-08-17T17:37:42.251 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1606999999999999e-05 2024-08-17T17:37:42.251 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.251 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.251 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:37:42.251 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807874+0000", 2024-08-17T17:37:42.251 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4829999999999999e-06 2024-08-17T17:37:42.251 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.251 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.251 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:37:42.251 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807937+0000", 2024-08-17T17:37:42.251 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3040999999999995e-05 2024-08-17T17:37:42.252 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.252 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.252 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:37:42.252 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807947+0000", 2024-08-17T17:37:42.252 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8379999999999997e-06 2024-08-17T17:37:42.252 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.252 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.252 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:37:42.252 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.769146+0000", 2024-08-17T17:37:42.252 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96119933899999999 2024-08-17T17:37:42.252 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.252 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.252 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:37:42.252 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.769155+0000", 2024-08-17T17:37:42.252 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5219999999999996e-06 2024-08-17T17:37:42.253 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.253 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:37:42.253 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.253 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.253 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.253 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:56793 8.3 8.83fbcd3 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e268)", 2024-08-17T17:37:42.253 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:19.807896+0000", 2024-08-17T17:37:42.253 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 262.43844593400001, 2024-08-17T17:37:42.253 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96128810799999997, 2024-08-17T17:37:42.253 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:37:42.253 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:37:42.253 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:37:42.253 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:37:42.253 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:37:42.253 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:37:42.253 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 56793 2024-08-17T17:37:42.254 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.254 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:37:42.254 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.254 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:37:42.254 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807896+0000", 2024-08-17T17:37:42.254 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.254 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.254 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.254 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:37:42.254 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807896+0000", 2024-08-17T17:37:42.254 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.254 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.254 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.254 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:37:42.255 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807897+0000", 2024-08-17T17:37:42.255 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8500000000000002e-07 2024-08-17T17:37:42.255 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.255 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.255 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:37:42.255 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807942+0000", 2024-08-17T17:37:42.255 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4977e-05 2024-08-17T17:37:42.255 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.255 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.255 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:37:42.255 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807950+0000", 2024-08-17T17:37:42.255 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9559999999999997e-06 2024-08-17T17:37:42.255 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.255 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.255 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:37:42.256 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807955+0000", 2024-08-17T17:37:42.256 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5509999999999999e-06 2024-08-17T17:37:42.256 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.256 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.256 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:37:42.256 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807997+0000", 2024-08-17T17:37:42.256 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2179999999999999e-05 2024-08-17T17:37:42.256 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.256 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.256 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:37:42.256 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808002+0000", 2024-08-17T17:37:42.256 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6639999999999997e-06 2024-08-17T17:37:42.256 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.256 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.256 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:37:42.256 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.769177+0000", 2024-08-17T17:37:42.257 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96117454300000005 2024-08-17T17:37:42.257 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.257 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.257 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:37:42.257 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.769184+0000", 2024-08-17T17:37:42.257 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7519999999999996e-06 2024-08-17T17:37:42.257 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.257 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:37:42.257 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.257 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.257 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.257 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:56797 8.a 8.229444aa (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e268)", 2024-08-17T17:37:42.257 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:19.807968+0000", 2024-08-17T17:37:42.257 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 262.43837431100002, 2024-08-17T17:37:42.257 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.95976723200000003, 2024-08-17T17:37:42.257 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:37:42.258 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:37:42.258 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:37:42.258 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:37:42.258 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:37:42.258 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:37:42.258 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 56797 2024-08-17T17:37:42.258 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.258 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:37:42.258 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.258 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:37:42.258 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807968+0000", 2024-08-17T17:37:42.258 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.258 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.258 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.259 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:37:42.259 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807968+0000", 2024-08-17T17:37:42.259 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.259 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.259 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.259 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:37:42.259 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807968+0000", 2024-08-17T17:37:42.259 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0900000000000002e-07 2024-08-17T17:37:42.259 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.259 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.259 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:37:42.259 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808012+0000", 2024-08-17T17:37:42.259 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3455000000000003e-05 2024-08-17T17:37:42.259 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.259 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.259 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:37:42.260 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808020+0000", 2024-08-17T17:37:42.260 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4409999999999993e-06 2024-08-17T17:37:42.260 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.260 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.260 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:37:42.260 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808025+0000", 2024-08-17T17:37:42.260 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9629999999999997e-06 2024-08-17T17:37:42.260 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.260 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.260 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:37:42.260 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808063+0000", 2024-08-17T17:37:42.260 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7626000000000002e-05 2024-08-17T17:37:42.260 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.260 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.260 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:37:42.261 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808067+0000", 2024-08-17T17:37:42.261 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0960000000000003e-06 2024-08-17T17:37:42.261 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.261 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.261 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:37:42.261 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.767725+0000", 2024-08-17T17:37:42.261 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.95965830900000004 2024-08-17T17:37:42.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:42.262 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.262 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.262 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:37:42.262 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.767735+0000", 2024-08-17T17:37:42.262 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9329999999999996e-06 2024-08-17T17:37:42.262 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.263 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:37:42.263 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.263 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.263 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.263 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:56804 8.3 8.fd779f13 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-08-17T17:37:42.263 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:19.808037+0000", 2024-08-17T17:37:42.263 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 262.43830495399999, 2024-08-17T17:37:42.263 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96117662699999995, 2024-08-17T17:37:42.263 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:37:42.263 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:37:42.263 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:37:42.263 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:37:42.263 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:37:42.263 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:37:42.263 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 56804 2024-08-17T17:37:42.264 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.264 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:37:42.264 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.264 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:37:42.264 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808037+0000", 2024-08-17T17:37:42.264 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.264 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.264 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.264 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:37:42.264 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808037+0000", 2024-08-17T17:37:42.264 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.264 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.264 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.264 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:37:42.264 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808038+0000", 2024-08-17T17:37:42.265 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2300000000000002e-07 2024-08-17T17:37:42.265 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.265 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.265 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:37:42.265 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808086+0000", 2024-08-17T17:37:42.265 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8529000000000002e-05 2024-08-17T17:37:42.265 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.265 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.265 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:37:42.265 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808096+0000", 2024-08-17T17:37:42.265 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3999999999999998e-06 2024-08-17T17:37:42.265 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.265 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.265 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:37:42.266 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808101+0000", 2024-08-17T17:37:42.266 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.575e-06 2024-08-17T17:37:42.266 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.266 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.266 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:37:42.266 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808148+0000", 2024-08-17T17:37:42.266 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6981000000000003e-05 2024-08-17T17:37:42.266 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.266 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.266 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:37:42.266 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808155+0000", 2024-08-17T17:37:42.266 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8340000000000001e-06 2024-08-17T17:37:42.266 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.267 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.267 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:37:42.267 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.769207+0000", 2024-08-17T17:37:42.267 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96105154500000001 2024-08-17T17:37:42.267 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.267 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.267 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:37:42.267 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.769214+0000", 2024-08-17T17:37:42.267 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.34e-06 2024-08-17T17:37:42.267 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.267 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:37:42.267 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.267 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.267 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.268 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:56808 8.3 8.634b5e43 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-08-17T17:37:42.268 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:19.808791+0000", 2024-08-17T17:37:42.268 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 262.437551633, 2024-08-17T17:37:42.268 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96046335800000004, 2024-08-17T17:37:42.268 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:37:42.268 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:37:42.268 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:37:42.268 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:37:42.268 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:37:42.268 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:37:42.268 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 56808 2024-08-17T17:37:42.268 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.268 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:37:42.268 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.269 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:37:42.269 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808791+0000", 2024-08-17T17:37:42.269 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.269 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.269 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.269 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:37:42.269 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808791+0000", 2024-08-17T17:37:42.269 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.269 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.269 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.269 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:37:42.269 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808791+0000", 2024-08-17T17:37:42.269 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7199999999999998e-07 2024-08-17T17:37:42.269 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.270 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.270 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:37:42.270 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808851+0000", 2024-08-17T17:37:42.270 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0016999999999998e-05 2024-08-17T17:37:42.270 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.270 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.270 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:37:42.270 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808862+0000", 2024-08-17T17:37:42.270 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0604e-05 2024-08-17T17:37:42.270 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.270 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.270 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:37:42.270 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808867+0000", 2024-08-17T17:37:42.271 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1959999999999999e-06 2024-08-17T17:37:42.271 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.271 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.271 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:37:42.271 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808909+0000", 2024-08-17T17:37:42.271 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1519e-05 2024-08-17T17:37:42.271 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.271 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.271 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:37:42.271 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808914+0000", 2024-08-17T17:37:42.271 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6930000000000004e-06 2024-08-17T17:37:42.271 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.271 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.271 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:37:42.271 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.769247+0000", 2024-08-17T17:37:42.272 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.960332356 2024-08-17T17:37:42.272 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.272 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.272 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:37:42.272 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.769254+0000", 2024-08-17T17:37:42.272 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2010000000000002e-06 2024-08-17T17:37:42.272 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.272 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:37:42.272 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.272 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.272 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.272 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:56812 8.7 8.b683b077 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-08-17T17:37:42.272 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:19.812057+0000", 2024-08-17T17:37:42.272 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 262.43428560699999, 2024-08-17T17:37:42.273 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.950741383, 2024-08-17T17:37:42.273 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:37:42.273 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:37:42.273 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:37:42.273 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:37:42.273 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:37:42.273 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:37:42.273 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 56812 2024-08-17T17:37:42.273 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.273 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:37:42.273 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.273 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:37:42.273 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.812057+0000", 2024-08-17T17:37:42.273 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.274 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.274 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.274 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:37:42.274 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.812057+0000", 2024-08-17T17:37:42.274 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.274 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.274 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.274 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:37:42.274 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.812058+0000", 2024-08-17T17:37:42.274 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0899999999999999e-06 2024-08-17T17:37:42.274 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.274 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.274 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:37:42.274 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.812100+0000", 2024-08-17T17:37:42.275 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2030000000000002e-05 2024-08-17T17:37:42.275 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.275 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.275 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:37:42.275 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.812112+0000", 2024-08-17T17:37:42.275 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.224e-05 2024-08-17T17:37:42.275 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.275 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.275 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:37:42.275 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.812120+0000", 2024-08-17T17:37:42.275 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0039999999999999e-06 2024-08-17T17:37:42.275 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.275 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.275 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:37:42.276 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.812179+0000", 2024-08-17T17:37:42.276 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9104000000000002e-05 2024-08-17T17:37:42.276 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.276 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.276 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:37:42.276 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.812184+0000", 2024-08-17T17:37:42.276 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0799999999999996e-06 2024-08-17T17:37:42.276 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.276 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.276 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:37:42.276 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.762788+0000", 2024-08-17T17:37:42.276 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.95060427300000006 2024-08-17T17:37:42.276 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.276 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.276 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:37:42.277 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.762798+0000", 2024-08-17T17:37:42.277 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5619999999999993e-06 2024-08-17T17:37:42.277 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.277 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:37:42.277 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.277 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.277 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.277 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:85958 8.b e274/269 8:d8a5509b:::benchmark_data_smithi003_229103_object85957:head v 274'5400, mlcod=0'0)", 2024-08-17T17:37:42.277 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:56.768913+0000", 2024-08-17T17:37:42.277 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 225.477428739, 2024-08-17T17:37:42.277 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.15166750000000001, 2024-08-17T17:37:42.277 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:37:42.277 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:37:42.277 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:37:42.277 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:37:42.278 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.278 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:37:42.278 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.768913+0000", 2024-08-17T17:37:42.278 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.278 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.278 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.278 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:37:42.278 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.768913+0000", 2024-08-17T17:37:42.278 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.278 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.278 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.278 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:37:42.278 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.768914+0000", 2024-08-17T17:37:42.279 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0200000000000001e-07 2024-08-17T17:37:42.279 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.279 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.279 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:37:42.279 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.768938+0000", 2024-08-17T17:37:42.279 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4090000000000001e-05 2024-08-17T17:37:42.279 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.279 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.279 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:37:42.279 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.768949+0000", 2024-08-17T17:37:42.279 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1197e-05 2024-08-17T17:37:42.279 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.279 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.279 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:37:42.279 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.768957+0000", 2024-08-17T17:37:42.280 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4610000000000002e-06 2024-08-17T17:37:42.280 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.280 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.280 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:37:42.280 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.769037+0000", 2024-08-17T17:37:42.280 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9574999999999996e-05 2024-08-17T17:37:42.280 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.280 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.280 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:37:42.280 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.769059+0000", 2024-08-17T17:37:42.280 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2779e-05 2024-08-17T17:37:42.280 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.280 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.280 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:37:42.281 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.920541+0000", 2024-08-17T17:37:42.281 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.15148128399999999 2024-08-17T17:37:42.281 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.281 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.281 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:37:42.281 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.920581+0000", 2024-08-17T17:37:42.281 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0411999999999999e-05 2024-08-17T17:37:42.281 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.281 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:37:42.281 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.281 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.281 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.281 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:85957 8.3 e274/269 8:c0f64a24:::benchmark_data_smithi003_229103_object85956:head v 274'5331, mlcod=0'0)", 2024-08-17T17:37:42.281 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:56.769588+0000", 2024-08-17T17:37:42.282 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 225.47675401000001, 2024-08-17T17:37:42.282 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.151026718, 2024-08-17T17:37:42.282 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:37:42.282 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:37:42.282 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:37:42.282 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:37:42.282 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.282 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:37:42.282 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.769588+0000", 2024-08-17T17:37:42.282 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.282 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.282 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.282 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:37:42.282 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.769588+0000", 2024-08-17T17:37:42.283 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.283 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.283 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.283 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:37:42.283 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.769589+0000", 2024-08-17T17:37:42.283 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8999999999999996e-07 2024-08-17T17:37:42.283 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.283 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.283 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:37:42.283 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.769620+0000", 2024-08-17T17:37:42.283 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1173000000000001e-05 2024-08-17T17:37:42.283 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.283 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.283 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:37:42.284 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.769631+0000", 2024-08-17T17:37:42.284 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.083e-05 2024-08-17T17:37:42.284 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.284 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.284 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:37:42.284 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.769638+0000", 2024-08-17T17:37:42.284 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8539999999999999e-06 2024-08-17T17:37:42.284 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.284 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.284 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:37:42.284 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.769703+0000", 2024-08-17T17:37:42.284 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5326000000000004e-05 2024-08-17T17:37:42.284 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.284 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.284 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:37:42.285 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.769724+0000", 2024-08-17T17:37:42.285 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0763000000000001e-05 2024-08-17T17:37:42.285 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.285 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.285 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:37:42.285 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.920586+0000", 2024-08-17T17:37:42.285 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.150861674 2024-08-17T17:37:42.285 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.285 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.285 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:37:42.285 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.920615+0000", 2024-08-17T17:37:42.285 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9408e-05 2024-08-17T17:37:42.285 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.285 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:37:42.285 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.286 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.286 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.286 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:85960 8.1 e274/269 8:8a84ea3f:::benchmark_data_smithi003_229103_object85959:head v 274'5435, mlcod=0'0)", 2024-08-17T17:37:42.286 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:56.771588+0000", 2024-08-17T17:37:42.286 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 225.47475394700001, 2024-08-17T17:37:42.286 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.148770771, 2024-08-17T17:37:42.286 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:37:42.286 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:37:42.286 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:37:42.286 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:37:42.286 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.286 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:37:42.286 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.771588+0000", 2024-08-17T17:37:42.287 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.287 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.287 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.287 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:37:42.287 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.771588+0000", 2024-08-17T17:37:42.287 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.287 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.287 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.287 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:37:42.287 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.771589+0000", 2024-08-17T17:37:42.287 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.217e-06 2024-08-17T17:37:42.287 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.287 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.287 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:37:42.287 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.771620+0000", 2024-08-17T17:37:42.287 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0923999999999997e-05 2024-08-17T17:37:42.288 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.288 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.288 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:37:42.288 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.771633+0000", 2024-08-17T17:37:42.288 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2510999999999999e-05 2024-08-17T17:37:42.288 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.288 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.288 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:37:42.288 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.771641+0000", 2024-08-17T17:37:42.288 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8210000000000002e-06 2024-08-17T17:37:42.288 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.288 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.288 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:37:42.288 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.771705+0000", 2024-08-17T17:37:42.288 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4269000000000004e-05 2024-08-17T17:37:42.289 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.289 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.289 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:37:42.289 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.771727+0000", 2024-08-17T17:37:42.289 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1857000000000001e-05 2024-08-17T17:37:42.289 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.289 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.289 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:37:42.289 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.776415+0000", 2024-08-17T17:37:42.289 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0046877150000000003 2024-08-17T17:37:42.289 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.289 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.289 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:37:42.289 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.920359+0000", 2024-08-17T17:37:42.289 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.143944457 2024-08-17T17:37:42.290 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.290 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:37:42.290 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.290 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.290 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.290 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:85959 8.b e274/269 8:d0920273:::benchmark_data_smithi003_229103_object85958:head v 274'5401, mlcod=0'0)", 2024-08-17T17:37:42.290 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:56.772299+0000", 2024-08-17T17:37:42.290 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 225.47404278400001, 2024-08-17T17:37:42.290 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.14833513600000001, 2024-08-17T17:37:42.290 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:37:42.290 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:37:42.290 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:37:42.291 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:37:42.291 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.291 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:37:42.291 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.772299+0000", 2024-08-17T17:37:42.291 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.291 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.291 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.291 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:37:42.291 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.772299+0000", 2024-08-17T17:37:42.291 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.291 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.291 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.291 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:37:42.291 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.772300+0000", 2024-08-17T17:37:42.291 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9800000000000003e-07 2024-08-17T17:37:42.291 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.292 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.292 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:37:42.292 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.772322+0000", 2024-08-17T17:37:42.292 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1888000000000001e-05 2024-08-17T17:37:42.292 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.292 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.292 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:37:42.292 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.772333+0000", 2024-08-17T17:37:42.292 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0702e-05 2024-08-17T17:37:42.292 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.292 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.292 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:37:42.292 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.772340+0000", 2024-08-17T17:37:42.293 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.844e-06 2024-08-17T17:37:42.293 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.293 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.293 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:37:42.293 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.774536+0000", 2024-08-17T17:37:42.293 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0021964490000000001 2024-08-17T17:37:42.293 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.293 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.293 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:37:42.293 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.774560+0000", 2024-08-17T17:37:42.293 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4065000000000001e-05 2024-08-17T17:37:42.293 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.293 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.293 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:37:42.294 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.920619+0000", 2024-08-17T17:37:42.294 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.14605860100000001 2024-08-17T17:37:42.294 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.294 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.294 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:37:42.294 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.920635+0000", 2024-08-17T17:37:42.294 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5889e-05 2024-08-17T17:37:42.294 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.294 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:37:42.294 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.294 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.294 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.294 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDPGPush(8.f 274/269 [PushOp(8:faa628b5:::benchmark_data_smithi003_229103_object13698:head, version: 265'884, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:faa628b5:::benchmark_data_smithi003_229103_object13698:head@265'884, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-17T17:37:42.295 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:56.779068+0000", 2024-08-17T17:37:42.295 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 225.46727468399999, 2024-08-17T17:37:42.295 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.14299867799999999, 2024-08-17T17:37:42.295 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:37:42.295 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:37:42.295 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:37:42.295 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:37:42.295 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.295 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:37:42.295 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.779068+0000", 2024-08-17T17:37:42.295 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.295 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.295 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.295 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:37:42.295 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.779068+0000", 2024-08-17T17:37:42.296 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.296 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.296 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.296 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:37:42.296 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.779069+0000", 2024-08-17T17:37:42.296 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.189e-06 2024-08-17T17:37:42.296 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.296 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.296 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:37:42.296 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.779097+0000", 2024-08-17T17:37:42.296 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8402999999999999e-05 2024-08-17T17:37:42.296 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.296 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.296 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:37:42.296 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.920331+0000", 2024-08-17T17:37:42.296 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.14123345800000001 2024-08-17T17:37:42.297 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.297 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.297 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:37:42.297 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.920367+0000", 2024-08-17T17:37:42.297 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6538000000000002e-05 2024-08-17T17:37:42.297 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.297 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.297 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:37:42.297 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.920454+0000", 2024-08-17T17:37:42.297 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6441999999999997e-05 2024-08-17T17:37:42.297 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.297 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.297 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:37:42.297 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.920468+0000", 2024-08-17T17:37:42.297 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4449e-05 2024-08-17T17:37:42.298 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.298 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.298 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:37:42.298 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.922066+0000", 2024-08-17T17:37:42.298 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001598199 2024-08-17T17:37:42.298 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.298 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:37:42.298 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.298 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.298 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.298 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:107138 8.1 e279/269 8:82567057:::benchmark_data_smithi003_229103_object107137:head v 279'6810, mlcod=0'0)", 2024-08-17T17:37:42.298 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:34:25.726290+0000", 2024-08-17T17:37:42.298 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 196.52005243400001, 2024-08-17T17:37:42.299 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.099167660000000005, 2024-08-17T17:37:42.299 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:37:42.299 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:37:42.299 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:37:42.299 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:37:42.299 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.299 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:37:42.299 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.726290+0000", 2024-08-17T17:37:42.299 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.299 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.299 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.299 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:37:42.299 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.726290+0000", 2024-08-17T17:37:42.299 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.299 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.299 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.300 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:37:42.300 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.726291+0000", 2024-08-17T17:37:42.300 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4610000000000001e-06 2024-08-17T17:37:42.300 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.300 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.300 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:37:42.300 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.726321+0000", 2024-08-17T17:37:42.300 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0150000000000001e-05 2024-08-17T17:37:42.300 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.300 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.300 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:37:42.300 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.726335+0000", 2024-08-17T17:37:42.300 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3699e-05 2024-08-17T17:37:42.300 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.300 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.301 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:37:42.301 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.726344+0000", 2024-08-17T17:37:42.301 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8780000000000004e-06 2024-08-17T17:37:42.301 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.301 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.301 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:37:42.301 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.731169+0000", 2024-08-17T17:37:42.301 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0048248639999999999 2024-08-17T17:37:42.301 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.301 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.301 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:37:42.301 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.731224+0000", 2024-08-17T17:37:42.301 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5084000000000003e-05 2024-08-17T17:37:42.301 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.302 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.302 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:37:42.302 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.825216+0000", 2024-08-17T17:37:42.302 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.093991746000000001 2024-08-17T17:37:42.302 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.302 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.302 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:37:42.302 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.825457+0000", 2024-08-17T17:37:42.302 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000241778 2024-08-17T17:37:42.302 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.302 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:37:42.302 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.302 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.302 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.303 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:42.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:42.304 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:107139 8.1 e279/269 8:82a9d69d:::benchmark_data_smithi003_229103_object107138:head v 279'6811, mlcod=0'0)", 2024-08-17T17:37:42.304 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:34:25.730949+0000", 2024-08-17T17:37:42.304 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 196.51539365799999, 2024-08-17T17:37:42.304 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.094726930000000001, 2024-08-17T17:37:42.304 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:37:42.304 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:37:42.304 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:37:42.304 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:37:42.304 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.304 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:37:42.304 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.730949+0000", 2024-08-17T17:37:42.305 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.305 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.305 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.305 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:37:42.305 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.730949+0000", 2024-08-17T17:37:42.305 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.305 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.305 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.305 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:37:42.305 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.730950+0000", 2024-08-17T17:37:42.305 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.003e-06 2024-08-17T17:37:42.305 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.305 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.306 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:37:42.306 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.730973+0000", 2024-08-17T17:37:42.306 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3734999999999999e-05 2024-08-17T17:37:42.306 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.306 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.306 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:37:42.306 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.730988+0000", 2024-08-17T17:37:42.306 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4384e-05 2024-08-17T17:37:42.306 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.306 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.306 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:37:42.306 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.730995+0000", 2024-08-17T17:37:42.306 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4189999999999996e-06 2024-08-17T17:37:42.306 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.307 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.307 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:37:42.307 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.759195+0000", 2024-08-17T17:37:42.307 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.028199607000000002 2024-08-17T17:37:42.307 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.307 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.307 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:37:42.307 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.759226+0000", 2024-08-17T17:37:42.307 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1127000000000002e-05 2024-08-17T17:37:42.307 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.307 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.307 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:37:42.307 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.825466+0000", 2024-08-17T17:37:42.307 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.066239929000000003 2024-08-17T17:37:42.308 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.308 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.308 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:37:42.308 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:25.825675+0000", 2024-08-17T17:37:42.308 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000209726 2024-08-17T17:37:42.308 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.308 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:37:42.308 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.308 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.308 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.308 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:107314 8.1 e279/269 8:83c95973:::benchmark_data_smithi003_229103_object107313:head v 279'6829, mlcod=0'0)", 2024-08-17T17:37:42.308 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:34:26.092005+0000", 2024-08-17T17:37:42.308 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 196.154337701, 2024-08-17T17:37:42.309 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.104778882, 2024-08-17T17:37:42.309 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:37:42.309 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:37:42.309 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:37:42.309 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:37:42.309 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.309 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:37:42.309 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.092005+0000", 2024-08-17T17:37:42.309 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.309 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.309 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.309 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:37:42.309 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.092005+0000", 2024-08-17T17:37:42.309 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.310 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.310 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.310 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:37:42.310 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.092006+0000", 2024-08-17T17:37:42.310 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.699e-06 2024-08-17T17:37:42.310 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.310 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.310 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:37:42.310 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.092426+0000", 2024-08-17T17:37:42.310 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00041939899999999999 2024-08-17T17:37:42.310 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.310 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.310 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:37:42.311 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.092442+0000", 2024-08-17T17:37:42.311 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6694e-05 2024-08-17T17:37:42.311 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.311 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.311 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:37:42.311 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.092452+0000", 2024-08-17T17:37:42.311 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3330000000000003e-06 2024-08-17T17:37:42.311 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.311 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.311 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:37:42.311 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.092656+0000", 2024-08-17T17:37:42.311 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020404599999999999 2024-08-17T17:37:42.311 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.311 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.312 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:37:42.312 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.092681+0000", 2024-08-17T17:37:42.312 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4941000000000001e-05 2024-08-17T17:37:42.312 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.312 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.312 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:37:42.312 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.196688+0000", 2024-08-17T17:37:42.312 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.104007219 2024-08-17T17:37:42.312 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.312 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.312 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:37:42.312 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.196783+0000", 2024-08-17T17:37:42.312 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5550999999999994e-05 2024-08-17T17:37:42.312 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.313 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:37:42.313 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.313 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.313 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.313 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:107332 8.3 e279/269 8:c64c32db:::benchmark_data_smithi003_229103_object107331:head v 279'6693, mlcod=0'0)", 2024-08-17T17:37:42.313 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:34:26.114866+0000", 2024-08-17T17:37:42.313 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 196.13147581800001, 2024-08-17T17:37:42.313 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.105011047, 2024-08-17T17:37:42.313 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:37:42.313 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:37:42.313 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:37:42.313 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:37:42.313 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.313 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:37:42.313 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.114866+0000", 2024-08-17T17:37:42.314 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.314 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.314 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.314 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:37:42.314 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.114866+0000", 2024-08-17T17:37:42.314 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.314 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.314 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.314 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:37:42.314 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.114867+0000", 2024-08-17T17:37:42.314 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5600000000000004e-07 2024-08-17T17:37:42.314 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.314 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.315 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:37:42.315 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.114909+0000", 2024-08-17T17:37:42.315 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1702999999999997e-05 2024-08-17T17:37:42.315 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.315 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.315 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:37:42.315 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.114922+0000", 2024-08-17T17:37:42.315 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3390000000000001e-05 2024-08-17T17:37:42.315 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.315 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.315 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:37:42.315 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.114931+0000", 2024-08-17T17:37:42.315 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8529999999999998e-06 2024-08-17T17:37:42.315 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.316 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.316 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:37:42.316 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.115011+0000", 2024-08-17T17:37:42.316 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9843000000000001e-05 2024-08-17T17:37:42.316 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.316 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.316 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:37:42.316 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.115033+0000", 2024-08-17T17:37:42.316 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2078000000000001e-05 2024-08-17T17:37:42.316 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.316 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.316 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:37:42.316 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.219809+0000", 2024-08-17T17:37:42.316 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.10477537200000001 2024-08-17T17:37:42.317 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.317 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.317 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:37:42.317 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.219877+0000", 2024-08-17T17:37:42.317 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8851999999999997e-05 2024-08-17T17:37:42.317 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.317 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:37:42.317 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.317 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.317 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.317 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:107343 8.3 e279/269 8:cb7c9311:::benchmark_data_smithi003_229103_object107342:head v 279'6694, mlcod=0'0)", 2024-08-17T17:37:42.317 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:34:26.129103+0000", 2024-08-17T17:37:42.317 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 196.117239313, 2024-08-17T17:37:42.317 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.096487893000000005, 2024-08-17T17:37:42.318 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:37:42.318 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:37:42.318 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:37:42.318 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:37:42.318 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.318 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:37:42.318 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.129103+0000", 2024-08-17T17:37:42.318 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.318 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.318 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.318 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:37:42.318 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.129103+0000", 2024-08-17T17:37:42.318 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.319 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.319 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.319 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:37:42.319 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.129104+0000", 2024-08-17T17:37:42.319 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0020000000000001e-06 2024-08-17T17:37:42.319 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.319 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.319 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:37:42.319 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.129138+0000", 2024-08-17T17:37:42.319 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4545999999999998e-05 2024-08-17T17:37:42.319 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.319 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.319 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:37:42.320 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.182785+0000", 2024-08-17T17:37:42.320 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.053646294999999997 2024-08-17T17:37:42.320 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.320 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.320 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:37:42.320 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.182805+0000", 2024-08-17T17:37:42.320 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0024e-05 2024-08-17T17:37:42.320 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.320 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.320 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:37:42.320 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.184752+0000", 2024-08-17T17:37:42.320 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00194754 2024-08-17T17:37:42.320 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.320 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.321 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:37:42.321 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.184778+0000", 2024-08-17T17:37:42.321 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5476000000000001e-05 2024-08-17T17:37:42.321 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.321 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.321 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:37:42.321 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.225423+0000", 2024-08-17T17:37:42.321 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.040645674999999999 2024-08-17T17:37:42.321 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.321 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.321 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:37:42.321 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:34:26.225591+0000", 2024-08-17T17:37:42.321 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016733500000000001 2024-08-17T17:37:42.321 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.322 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:37:42.322 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.322 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.322 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.322 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:150054 8.b e284/269 8:d8f88633:::benchmark_data_smithi003_229103_object42579:head v 284'9480, mlcod=0'0)", 2024-08-17T17:37:42.322 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:35:27.276461+0000", 2024-08-17T17:37:42.322 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 134.969881531, 2024-08-17T17:37:42.322 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.097565896999999999, 2024-08-17T17:37:42.322 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:37:42.322 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:37:42.322 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:37:42.323 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:37:42.323 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.323 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:37:42.323 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:27.276461+0000", 2024-08-17T17:37:42.323 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.323 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.323 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.323 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:37:42.323 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:27.276461+0000", 2024-08-17T17:37:42.323 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.323 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.323 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.324 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:37:42.324 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:27.276461+0000", 2024-08-17T17:37:42.324 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5300000000000003e-07 2024-08-17T17:37:42.324 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.324 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.324 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:37:42.324 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:27.276467+0000", 2024-08-17T17:37:42.324 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9950000000000002e-06 2024-08-17T17:37:42.324 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.324 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.324 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:37:42.324 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:27.276483+0000", 2024-08-17T17:37:42.324 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5982e-05 2024-08-17T17:37:42.324 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.324 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.324 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:37:42.325 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:27.276491+0000", 2024-08-17T17:37:42.325 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.622e-06 2024-08-17T17:37:42.325 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.325 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.325 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:37:42.325 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:27.319082+0000", 2024-08-17T17:37:42.325 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.042591361000000001 2024-08-17T17:37:42.325 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.325 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.325 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:37:42.325 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:27.319108+0000", 2024-08-17T17:37:42.325 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5140999999999999e-05 2024-08-17T17:37:42.325 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.325 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.326 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:37:42.326 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:27.373830+0000", 2024-08-17T17:37:42.326 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.054722123999999997 2024-08-17T17:37:42.326 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.326 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.326 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:37:42.326 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:27.374027+0000", 2024-08-17T17:37:42.326 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019691899999999999 2024-08-17T17:37:42.326 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.326 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:37:42.326 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.326 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.326 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.326 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:170728 8.7 e289/269 8:e235288c:::benchmark_data_smithi003_229103_object63253:head v 289'10634, mlcod=0'0)", 2024-08-17T17:37:42.327 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:35:58.649459+0000", 2024-08-17T17:37:42.327 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 103.59688278500001, 2024-08-17T17:37:42.327 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.099480357000000005, 2024-08-17T17:37:42.327 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:37:42.327 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:37:42.327 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:37:42.327 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:37:42.327 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.327 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:37:42.327 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:58.649459+0000", 2024-08-17T17:37:42.327 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.327 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.327 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.327 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:37:42.328 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:58.649459+0000", 2024-08-17T17:37:42.328 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.328 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.328 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.328 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:37:42.328 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:58.649460+0000", 2024-08-17T17:37:42.328 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3799999999999996e-07 2024-08-17T17:37:42.328 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.328 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.328 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:37:42.328 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:58.649468+0000", 2024-08-17T17:37:42.328 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5100000000000001e-06 2024-08-17T17:37:42.328 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.328 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.329 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:37:42.329 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:58.649479+0000", 2024-08-17T17:37:42.329 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0719e-05 2024-08-17T17:37:42.329 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.329 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.329 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:37:42.329 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:58.649485+0000", 2024-08-17T17:37:42.329 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9380000000000003e-06 2024-08-17T17:37:42.329 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.329 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.329 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:37:42.329 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:58.649601+0000", 2024-08-17T17:37:42.329 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000115532 2024-08-17T17:37:42.329 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.329 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.329 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:37:42.330 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:58.649624+0000", 2024-08-17T17:37:42.330 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2711e-05 2024-08-17T17:37:42.330 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.330 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.330 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:37:42.330 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:58.748897+0000", 2024-08-17T17:37:42.330 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.099273506999999997 2024-08-17T17:37:42.330 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.330 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.330 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:37:42.330 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:58.748940+0000", 2024-08-17T17:37:42.330 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2602000000000003e-05 2024-08-17T17:37:42.330 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.330 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:37:42.331 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.331 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.331 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.331 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:209905 8.7 e297/269 8:e59c3040:::benchmark_data_smithi003_229103_object102430:head v 297'13089, mlcod=0'0)", 2024-08-17T17:37:42.331 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:36:55.106870+0000", 2024-08-17T17:37:42.331 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 47.139472093999998, 2024-08-17T17:37:42.331 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.10400959999999999, 2024-08-17T17:37:42.331 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:37:42.331 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:37:42.331 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:37:42.331 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:37:42.331 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.331 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:37:42.331 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:55.106870+0000", 2024-08-17T17:37:42.332 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.332 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.332 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.332 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:37:42.332 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:55.106870+0000", 2024-08-17T17:37:42.332 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.332 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.332 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.332 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:37:42.332 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:55.106870+0000", 2024-08-17T17:37:42.332 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8599999999999999e-07 2024-08-17T17:37:42.332 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.332 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.332 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:37:42.332 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:55.106878+0000", 2024-08-17T17:37:42.333 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4279999999999999e-06 2024-08-17T17:37:42.333 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.333 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.333 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:37:42.333 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:55.106887+0000", 2024-08-17T17:37:42.333 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6250000000000002e-06 2024-08-17T17:37:42.333 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.333 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.333 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:37:42.333 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:55.106892+0000", 2024-08-17T17:37:42.333 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9069999999999998e-06 2024-08-17T17:37:42.333 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.333 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.333 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:37:42.333 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:55.109159+0000", 2024-08-17T17:37:42.334 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0022668440000000001 2024-08-17T17:37:42.334 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.334 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.334 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:37:42.334 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:55.121748+0000", 2024-08-17T17:37:42.334 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012589145 2024-08-17T17:37:42.334 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.334 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.334 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:37:42.334 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:55.210846+0000", 2024-08-17T17:37:42.334 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.089097935000000003 2024-08-17T17:37:42.334 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.334 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.334 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:37:42.335 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:55.210880+0000", 2024-08-17T17:37:42.335 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3429999999999997e-05 2024-08-17T17:37:42.335 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.335 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:37:42.335 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.335 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.335 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.335 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:209904 8.f e297/286 8:f7977543:::benchmark_data_smithi003_229103_object102429:head v 297'13285, mlcod=297'13282)", 2024-08-17T17:37:42.335 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:36:55.108340+0000", 2024-08-17T17:37:42.335 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 47.138002071000003, 2024-08-17T17:37:42.335 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.102502882, 2024-08-17T17:37:42.335 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:37:42.335 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:37:42.336 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:37:42.336 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:37:42.336 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.336 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:37:42.336 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:55.108340+0000", 2024-08-17T17:37:42.336 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.336 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.336 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.336 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:37:42.336 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:55.108340+0000", 2024-08-17T17:37:42.336 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:37:42.336 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.336 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.336 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:37:42.336 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:55.108340+0000", 2024-08-17T17:37:42.337 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.41e-07 2024-08-17T17:37:42.337 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.337 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.337 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:37:42.337 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:55.108344+0000", 2024-08-17T17:37:42.337 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5020000000000001e-06 2024-08-17T17:37:42.337 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.337 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.337 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:37:42.337 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:55.108350+0000", 2024-08-17T17:37:42.337 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0619999999999997e-06 2024-08-17T17:37:42.337 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.337 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.337 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:37:42.337 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:55.108354+0000", 2024-08-17T17:37:42.337 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5379999999999996e-06 2024-08-17T17:37:42.338 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.338 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.338 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:37:42.338 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:55.117198+0000", 2024-08-17T17:37:42.338 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0088430720000000004 2024-08-17T17:37:42.338 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.338 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.338 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:37:42.338 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:55.117254+0000", 2024-08-17T17:37:42.338 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6444999999999999e-05 2024-08-17T17:37:42.338 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.338 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.338 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:37:42.339 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:55.210774+0000", 2024-08-17T17:37:42.339 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.093519788000000006 2024-08-17T17:37:42.339 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:37:42.339 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:37:42.339 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:37:42.339 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:36:55.210843+0000", 2024-08-17T17:37:42.339 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9233999999999995e-05 2024-08-17T17:37:42.339 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.339 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:37:42.339 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.339 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:37:42.339 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:37:42.339 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:37:42.404 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:42.447 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:42.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:42.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:42.549 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:42.576 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:42.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:42.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:42.678 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:42.705 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:42.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:42.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:42.806 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:42.834 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:42.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:42.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:42.936 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:42.962 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:42.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:42.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:43.064 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:43.088 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:43.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:43.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:43.189 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:43.211 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:43.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:43.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:43.313 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:43.338 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:43.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:43.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:43.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:43.462 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:43.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:43.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:43.526 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:37:43.526 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:37:43.537 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:37:43.538 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:37:43.554 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:37:43.554 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:37:43.564 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:43.570 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:37:43.570 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:37:43.586 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:37:43.586 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:37:43.591 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:43.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:43.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:43.602 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:37:43.603 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:37:43.618 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:37:43.619 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:37:43.634 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:37:43.635 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:37:43.692 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:43.719 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:43.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:43.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:43.821 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:43.844 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:43.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:43.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:43.946 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:43.976 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:43.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:43.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:44.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:43 smithi003 ceph-mon[100615]: pgmap v978: 99 pgs: 6 creating+activating, 93 active+clean; 4.7 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-08-17T17:37:44.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:43 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:37:44.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:43 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3980802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:37:44.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:43 smithi003 ceph-mon[103344]: pgmap v978: 99 pgs: 6 creating+activating, 93 active+clean; 4.7 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-08-17T17:37:44.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:43 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:37:44.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:43 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3980802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:37:44.078 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:44.100 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:44.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:44.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:44.202 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:44.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:43 smithi053 ceph-mon[89963]: pgmap v978: 99 pgs: 6 creating+activating, 93 active+clean; 4.7 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-08-17T17:37:44.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:43 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:37:44.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:43 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3980802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:37:44.225 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:44.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:44.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:44.327 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:44.354 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:44.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:44.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:44.459 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:44.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:44.487 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:44.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:44.589 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:44.613 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:44.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:44.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:44.714 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:44.741 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:44.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:44.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:44.843 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:44.868 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:44.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:44.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:44.891 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-17T17:37:44.970 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:44.998 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:44.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:44.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:45.100 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:45.127 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:45.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:45.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:45.229 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:45.252 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:45.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:45.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:45.354 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:45.376 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:45.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:45.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:45.478 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:45.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:45.505 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:45.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:45.606 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:45.633 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:45.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:45.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:45.734 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:45.760 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:45.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:45.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:45.827 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:37:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:37:45] "GET /metrics HTTP/1.1" 200 36454 "" "Prometheus/2.51.0" 2024-08-17T17:37:45.862 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:45.886 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:45.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:45.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:45.988 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:46.013 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:46.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:46.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:46.115 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:46.140 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:46.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:46.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:46.179 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:45 smithi003 ceph-mon[103344]: pgmap v979: 99 pgs: 3 creating+activating, 96 active+clean; 4.7 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-08-17T17:37:46.179 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:45 smithi003 ceph-mon[100615]: pgmap v979: 99 pgs: 3 creating+activating, 96 active+clean; 4.7 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-08-17T17:37:46.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:45 smithi053 ceph-mon[89963]: pgmap v979: 99 pgs: 3 creating+activating, 96 active+clean; 4.7 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-08-17T17:37:46.242 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:46.268 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:46.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:46.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:46.369 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:46.396 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:46.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:46.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:46.498 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:46.520 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:46.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:46.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:46.559 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-08-17T17:37:46.560 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:37:46.621 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:46.648 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:46.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:46.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:46.750 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:46.772 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:46.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:46.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:46.874 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:46.897 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:46.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:46.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:46.998 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:47.025 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:47.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:47.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:47.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:47.149 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:47.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:47.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:47.251 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:47.279 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:47.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:47.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:47.380 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:47.403 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:47.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:47.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:47.505 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:47.532 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:47.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:47.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:47.563 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:37:47.633 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:47.656 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:47.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:47.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:47.758 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:47.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:47.782 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:47.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:47.835 INFO:teuthology.orchestra.run.smithi003.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-08-17T17:37:47.841 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-08-17T17:37:47.883 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:47.911 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:47.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:47.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:48.012 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:48.035 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:48.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:48.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:48.137 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:48.164 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:48.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:48.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:48.210 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:47 smithi003 ceph-mon[103344]: pgmap v980: 99 pgs: 99 active+clean; 4.7 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-08-17T17:37:48.211 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:47 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1097364810' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:37:48.211 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:47 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2468390051' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:37:48.211 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:47 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:37:48.211 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:47 smithi003 ceph-mon[100615]: pgmap v980: 99 pgs: 99 active+clean; 4.7 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-08-17T17:37:48.211 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:47 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1097364810' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:37:48.211 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:47 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2468390051' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:37:48.211 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:47 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:37:48.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:47 smithi053 ceph-mon[89963]: pgmap v980: 99 pgs: 99 active+clean; 4.7 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-08-17T17:37:48.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:47 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1097364810' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:37:48.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:47 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2468390051' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:37:48.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:47 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:37:48.266 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:48.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:48.290 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:48.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:48.392 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:48.418 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:48.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:48.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:48.521 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:48.543 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:48.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:48.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:48.645 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:48.667 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:48.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:48.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:48.769 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:48.792 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:48.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:48.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:48.894 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:48.921 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:48.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:48.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:48.966 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:48 smithi003 ceph-mon[103344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:37:48.966 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:48 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1097364810' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T17:37:48.966 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:48 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:37:48.966 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:48 smithi003 ceph-mon[103344]: osdmap e314: 8 total, 8 up, 8 in 2024-08-17T17:37:48.966 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:48 smithi003 ceph-mon[100615]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:37:48.966 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:48 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1097364810' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T17:37:48.966 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:48 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:37:48.966 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:48 smithi003 ceph-mon[100615]: osdmap e314: 8 total, 8 up, 8 in 2024-08-17T17:37:49.022 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:49.048 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:49.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:49.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:49.150 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:49.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:49.173 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:49.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:49.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:48 smithi053 ceph-mon[89963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:37:49.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:48 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1097364810' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T17:37:49.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:48 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:37:49.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:48 smithi053 ceph-mon[89963]: osdmap e314: 8 total, 8 up, 8 in 2024-08-17T17:37:49.275 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:49.301 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:49.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:49.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:49.403 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:49.425 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:49.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:49.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:49.527 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:49.549 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:49.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:49.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:49.551 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:37:49.652 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:49.678 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:49.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:49.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:49.780 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:49.804 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:49.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:49.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:49.906 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:49.928 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:49.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:49.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:49.945 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:49 smithi003 ceph-mon[103344]: pgmap v982: 99 pgs: 99 active+clean; 4.7 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-08-17T17:37:49.945 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:49 smithi003 ceph-mon[103344]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:37:49.946 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:49 smithi003 ceph-mon[100615]: pgmap v982: 99 pgs: 99 active+clean; 4.7 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-08-17T17:37:49.946 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:49 smithi003 ceph-mon[100615]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:37:50.029 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:50.051 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:50.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:50.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:50.153 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:50.180 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:50.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:50.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:50.213 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:49 smithi053 ceph-mon[89963]: pgmap v982: 99 pgs: 99 active+clean; 4.7 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-08-17T17:37:50.213 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:49 smithi053 ceph-mon[89963]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:37:50.266 DEBUG:teuthology.orchestra.run.smithi003:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-08-17T17:37:50.267 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-08-17T17:37:50.267 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:37:50.269 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-08-17T17:37:50.269 INFO:teuthology.orchestra.run:waiting for 3000 2024-08-17T17:37:50.282 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:50.309 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:50.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:50.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:50.318 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:hints = 1 2024-08-17T17:37:50.318 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-08-17T17:37:50.318 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object prefix: benchmark_data_smithi003_261051 2024-08-17T17:37:50.319 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:37:50.319 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 0 0 0 0 0 0 - 0 2024-08-17T17:37:50.411 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:50.454 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:50.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:50.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:50.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:37:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:37:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:37:50.468 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:37:50.468 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:37:50.556 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:50.587 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:50.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:50.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:50.689 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:50.718 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:50.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:50.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:50.820 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:50.862 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:50.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:50.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:50.964 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:50.997 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:50.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:50.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:51.099 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:51.140 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:51.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:51.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:51.241 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:51.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:51.271 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:51.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:51.301 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-17T17:37:51.319 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 1 16 3296 3280 204.865 205 0.00404104 0.00485288 2024-08-17T17:37:51.336 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:37:51.372 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:51.381 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:37:51.399 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:51.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:51.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:51.501 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:51.532 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:51.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:51.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:51.633 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:51.663 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:51.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:51.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:51.765 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:51.795 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:51.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:51.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:51.897 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:51.929 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:51.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:51.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:51.971 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:37:52.031 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:52.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:52.080 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:52.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:52.181 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:52.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:51 smithi053 ceph-mon[89963]: pgmap v983: 99 pgs: 99 active+clean; 4.7 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-08-17T17:37:52.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:52.222 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:52.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:52.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:51 smithi003 ceph-mon[100615]: pgmap v983: 99 pgs: 99 active+clean; 4.7 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-08-17T17:37:52.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:51 smithi003 ceph-mon[103344]: pgmap v983: 99 pgs: 99 active+clean; 4.7 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-08-17T17:37:52.319 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 2 16 6363 6347 198.27 191.688 0.00236671 0.00502105 2024-08-17T17:37:52.323 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:52.355 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:52.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:52.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:52.458 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:52.487 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:52.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:52.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:52.589 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:52.629 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:52.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:52.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:52.731 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:52.769 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:52.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:52.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:52.871 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:52.907 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:52.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:52.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:53.009 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:53.051 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:53.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:53.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:53.153 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:53.185 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:53.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:53.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:53.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:53.320 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 3 16 9434 9418 196.154 191.938 0.00222797 0.00508787 2024-08-17T17:37:53.326 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:53.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:53.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:53.428 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:53.455 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:53.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:53.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:53.557 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:53.590 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:53.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:53.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:53.691 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:53.723 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:53.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:53.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:53.825 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:53.857 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:53.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:53.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:53.905 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:53 smithi003 ceph-mon[100615]: pgmap v984: 99 pgs: 99 active+clean; 4.7 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-08-17T17:37:53.905 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:37:53.905 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:37:53.906 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:53 smithi003 ceph-mon[103344]: pgmap v984: 99 pgs: 99 active+clean; 4.7 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-08-17T17:37:53.906 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:37:53.906 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:37:53.959 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:53.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:53.994 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:53.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:54.045 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:37:54.095 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:54.125 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:54.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:54.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:54.135 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:37:54.135 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:37:54.176 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:37:54.177 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:37:54.213 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:37:54.214 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:37:54.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:53 smithi053 ceph-mon[89963]: pgmap v984: 99 pgs: 99 active+clean; 4.7 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-08-17T17:37:54.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:37:54.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:37:54.227 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:54.252 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:37:54.252 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:37:54.272 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:54.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:54.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:54.285 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:37:54.285 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:37:54.319 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:37:54.319 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:37:54.320 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 4 15 12170 12155 189.878 171.062 0.00198929 0.00525924 2024-08-17T17:37:54.355 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:37:54.355 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:37:54.375 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:54.393 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:37:54.393 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:37:54.403 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:54.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:54.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:54.505 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:54.544 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:54.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:54.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:54.646 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:54.684 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:54.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:54.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:54.786 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:54.820 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:54.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:54.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:54.922 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:54.962 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:54.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:54.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:55.064 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:55.089 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:55.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:55.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:55.191 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:55.211 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:54 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2390968027' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:37:55.212 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:54 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2390968027' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:37:55.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:54 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2390968027' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:37:55.228 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:55.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:55.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:55.320 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 5 16 14840 14824 185.262 166.812 0.00666841 0.00539139 2024-08-17T17:37:55.330 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:55.370 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:55.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:55.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:55.472 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:55.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:55.510 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:55.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:55.611 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:55.649 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:55.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:55.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:55.751 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:55.778 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:55.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:55.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:55.879 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:55.912 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:55.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:55.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:56.014 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:56.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:55 smithi003 ceph-mon[103344]: pgmap v985: 99 pgs: 99 active+clean; 397 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 39 MiB/s wr, 625 op/s 2024-08-17T17:37:56.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:55 smithi003 ceph-mon[100615]: pgmap v985: 99 pgs: 99 active+clean; 397 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 39 MiB/s wr, 625 op/s 2024-08-17T17:37:56.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:37:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:37:55] "GET /metrics HTTP/1.1" 200 36481 "" "Prometheus/2.51.0" 2024-08-17T17:37:56.055 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:56.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:56.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:56.157 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:56.195 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:56.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:56.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:56.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:55 smithi053 ceph-mon[89963]: pgmap v985: 99 pgs: 99 active+clean; 397 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 39 MiB/s wr, 625 op/s 2024-08-17T17:37:56.298 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:56.320 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 6 16 17561 17545 182.727 170.062 0.00601074 0.00546605 2024-08-17T17:37:56.329 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:56.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:56.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:56.432 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:56.464 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:56.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:56.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:56.566 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:56.595 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:56.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:56.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:56.697 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:37:56.730 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:56.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:56.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:37:56.832 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:56.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:56.873 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:56.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:56.974 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:57.002 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:57.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:57.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:57.104 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:57.147 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:57.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:57.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:57.249 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:57.283 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:57.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:57.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:57.320 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 7 16 19887 19871 177.389 145.375 0.00410076 0.00563098 2024-08-17T17:37:57.350 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:37:57.384 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:57.407 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:57.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:57.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:57.509 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:57.552 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:57.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:57.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:57.654 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:57.692 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:57.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:57.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:57.795 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:57.822 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:57.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:57.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:57.923 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:57.960 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:57.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:57.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:58.062 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:58.093 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:58.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:58.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:58.195 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:58.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:57 smithi053 ceph-mon[89963]: pgmap v986: 99 pgs: 99 active+clean; 763 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-08-17T17:37:58.227 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:58.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:58.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:58.271 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:57 smithi003 ceph-mon[103344]: pgmap v986: 99 pgs: 99 active+clean; 763 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-08-17T17:37:58.271 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:57 smithi003 ceph-mon[100615]: pgmap v986: 99 pgs: 99 active+clean; 763 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-08-17T17:37:58.320 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 8 16 21950 21934 171.332 128.938 0.00965943 0.00583026 2024-08-17T17:37:58.329 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:37:58.362 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:58.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:58.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:37:58.464 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:58.505 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:58.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:58.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:58.606 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:58.643 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:58.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:58.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:58.745 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:58.771 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:58.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:58.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:58.873 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:37:58.913 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:58.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:58.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:37:59.015 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:59.056 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:59.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:59.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:59.159 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:37:59.194 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:59.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:59.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:37:59.296 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:37:59.320 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 9 16 24030 24014 166.737 130 0.00586782 0.0059905 2024-08-17T17:37:59.341 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:59.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:59.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:37:59.443 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:37:59.475 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:37:59.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:59.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:37:59.577 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:37:59.619 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:59.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:59.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:37:59.721 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:59.762 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:59.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:59.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:37:59.880 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:37:59.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:37:59.922 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:37:59.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:00.023 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:00.060 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:00.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:00.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:00.130 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:37:59 smithi003 ceph-mon[103344]: pgmap v987: 99 pgs: 99 active+clean; 763 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-08-17T17:38:00.130 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:37:59 smithi003 ceph-mon[100615]: pgmap v987: 99 pgs: 99 active+clean; 763 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-08-17T17:38:00.162 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:00.202 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:00.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:00.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:00.214 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:37:59 smithi053 ceph-mon[89963]: pgmap v987: 99 pgs: 99 active+clean; 763 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-08-17T17:38:00.304 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:00.321 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 10 16 26124 26108 163.144 130.875 0.0098991 0.0061238 2024-08-17T17:38:00.346 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:00.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:00.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:00.368 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:38:00.368 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-08-17T17:38:00.368 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-08-17T17:38:00.369 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-08-17T17:38:00.369 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:38:00.448 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:00.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:38:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:38:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:38:00.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:00.485 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:00.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:00.586 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:00.628 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:00.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:00.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:00.729 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:00.768 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:00.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:00.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:00.870 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:00.911 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:00.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:00.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:01.013 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:01.051 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:01.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:01.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:01.152 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:01.190 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:01.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:01.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:01.282 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:38:01.283 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:38:01.292 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:01.321 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 11 16 28125 28109 159.681 125.062 0.00664317 0.00625557 2024-08-17T17:38:01.322 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:01.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:01.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:01.421 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-17T17:38:01.424 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:01.457 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:01.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:01.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:01.560 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:01.589 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:01.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:01.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:01.691 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:01.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:01.735 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:01.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:01.836 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:01.878 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:01.878 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:01 smithi003 ceph-mon[100615]: pgmap v988: 99 pgs: 99 active+clean; 1.5 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 125 MiB/s wr, 1.99k op/s 2024-08-17T17:38:01.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:01.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:01.895 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:38:01.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:02.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:02.029 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:02.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:02.130 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:02.162 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:02.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:02.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:02.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:01 smithi053 ceph-mon[89963]: pgmap v988: 99 pgs: 99 active+clean; 1.5 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 125 MiB/s wr, 1.99k op/s 2024-08-17T17:38:02.220 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:01 smithi003 ceph-mon[103344]: pgmap v988: 99 pgs: 99 active+clean; 1.5 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 125 MiB/s wr, 1.99k op/s 2024-08-17T17:38:02.264 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:02.278 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":988,"stamp":"2024-08-17T17:38:00.208707+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1574222616,"num_objects":23974,"num_object_clones":0,"num_object_copies":71922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23974,"num_whiteouts":0,"num_read":6046,"num_read_kb":13140,"num_write":24891,"num_write_kb":1545606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":9307214,"num_keys_recovered":20,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":26617,"ondisk_log_size":26617,"up":297,"acting":297,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":297,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":9352840,"kb_used_data":4236680,"kb_used_omap":37,"kb_used_meta":5116058,"kb_avail":740608376,"statfs":{"total":767960285184,"available":758382977024,"internally_reserved":0,"allocated":4338360320,"data_stored":4323944567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38729,"internal_metadata":5238843575},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,9,5,6],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1569325056,"num_objects":23946,"num_object_clones":0,"num_object_copies":71838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23946,"num_write_kb":1532544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.019181"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":497,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.860418+0000","last_change":"2024-08-17T17:33:20.797029+0000","last_active":"2024-08-17T17:37:48.860418+0000","last_peered":"2024-08-17T17:37:48.860418+0000","last_clean":"2024-08-17T17:37:48.860418+0000","last_became_active":"2024-08-17T17:33:20.796679+0000","last_became_peered":"2024-08-17T17:33:20.796679+0000","last_unstale":"2024-08-17T17:37:48.860418+0000","last_undegraded":"2024-08-17T17:37:48.860418+0000","last_fullsized":"2024-08-17T17:37:48.860418+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:51.438262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":306,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:39.818396+0000","last_change":"2024-08-17T17:37:00.420764+0000","last_active":"2024-08-17T17:37:39.818396+0000","last_peered":"2024-08-17T17:37:39.818396+0000","last_clean":"2024-08-17T17:37:39.818396+0000","last_became_active":"2024-08-17T17:37:00.387450+0000","last_became_peered":"2024-08-17T17:37:00.387450+0000","last_unstale":"2024-08-17T17:37:39.818396+0000","last_undegraded":"2024-08-17T17:37:39.818396+0000","last_fullsized":"2024-08-17T17:37:39.818396+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:09:59.462151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":158,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.859903+0000","last_change":"2024-08-17T17:33:44.156036+0000","last_active":"2024-08-17T17:37:48.859903+0000","last_peered":"2024-08-17T17:37:48.859903+0000","last_clean":"2024-08-17T17:37:48.859903+0000","last_became_active":"2024-08-17T17:33:44.155758+0000","last_became_peered":"2024-08-17T17:33:44.155758+0000","last_unstale":"2024-08-17T17:37:48.859903+0000","last_undegraded":"2024-08-17T17:37:48.859903+0000","last_fullsized":"2024-08-17T17:37:48.859903+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:23.443405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":297,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.860959+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:37:48.860959+0000","last_peered":"2024-08-17T17:37:48.860959+0000","last_clean":"2024-08-17T17:37:48.860959+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:37:48.860959+0000","last_undegraded":"2024-08-17T17:37:48.860959+0000","last_fullsized":"2024-08-17T17:37:48.860959+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":297,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.861399+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:37:48.861399+0000","last_peered":"2024-08-17T17:37:48.861399+0000","last_clean":"2024-08-17T17:37:48.861399+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:37:48.861399+0000","last_undegraded":"2024-08-17T17:37:48.861399+0000","last_fullsized":"2024-08-17T17:37:48.861399+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":63,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:40.030252+0000","last_change":"2024-08-17T17:32:46.377651+0000","last_active":"2024-08-17T17:37:40.030252+0000","last_peered":"2024-08-17T17:37:40.030252+0000","last_clean":"2024-08-17T17:37:40.030252+0000","last_became_active":"2024-08-17T17:32:46.377282+0000","last_became_peered":"2024-08-17T17:32:46.377282+0000","last_unstale":"2024-08-17T17:37:40.030252+0000","last_undegraded":"2024-08-17T17:37:40.030252+0000","last_fullsized":"2024-08-17T17:37:40.030252+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:45:27.370287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":511,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.860409+0000","last_change":"2024-08-17T17:33:20.794682+0000","last_active":"2024-08-17T17:37:48.860409+0000","last_peered":"2024-08-17T17:37:48.860409+0000","last_clean":"2024-08-17T17:37:48.860409+0000","last_became_active":"2024-08-17T17:33:20.794238+0000","last_became_peered":"2024-08-17T17:33:20.794238+0000","last_unstale":"2024-08-17T17:37:48.860409+0000","last_undegraded":"2024-08-17T17:37:48.860409+0000","last_fullsized":"2024-08-17T17:37:48.860409+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:15:55.821920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":360,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:40.030570+0000","last_change":"2024-08-17T17:24:53.465875+0000","last_active":"2024-08-17T17:37:40.030570+0000","last_peered":"2024-08-17T17:37:40.030570+0000","last_clean":"2024-08-17T17:37:40.030570+0000","last_became_active":"2024-08-17T17:24:53.458129+0000","last_became_peered":"2024-08-17T17:24:53.458129+0000","last_unstale":"2024-08-17T17:37:40.030570+0000","last_undegraded":"2024-08-17T17:37:40.030570+0000","last_fullsized":"2024-08-17T17:37:40.030570+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:51:33.584869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"314'1454","reported_seq":1470,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:59.200904+0000","last_change":"2024-08-17T17:37:39.458750+0000","last_active":"2024-08-17T17:37:59.200904+0000","last_peered":"2024-08-17T17:37:59.200904+0000","last_clean":"2024-08-17T17:37:59.200904+0000","last_became_active":"2024-08-17T17:37:39.458436+0000","last_became_peered":"2024-08-17T17:37:39.458436+0000","last_unstale":"2024-08-17T17:37:59.200904+0000","last_undegraded":"2024-08-17T17:37:59.200904+0000","last_fullsized":"2024-08-17T17:37:59.200904+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":1454,"log_dups_size":0,"ondisk_log_size":1454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:12:25.533220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95289344,"num_objects":1454,"num_object_clones":0,"num_object_copies":4362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1454,"num_write_kb":93056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":247,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:39.819031+0000","last_change":"2024-08-17T17:32:46.384376+0000","last_active":"2024-08-17T17:37:39.819031+0000","last_peered":"2024-08-17T17:37:39.819031+0000","last_clean":"2024-08-17T17:37:39.819031+0000","last_became_active":"2024-08-17T17:32:46.374177+0000","last_became_peered":"2024-08-17T17:32:46.374177+0000","last_unstale":"2024-08-17T17:37:39.819031+0000","last_undegraded":"2024-08-17T17:37:39.819031+0000","last_fullsized":"2024-08-17T17:37:39.819031+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:51:58.880388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":69,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:40.030048+0000","last_change":"2024-08-17T17:33:44.145598+0000","last_active":"2024-08-17T17:37:40.030048+0000","last_peered":"2024-08-17T17:37:40.030048+0000","last_clean":"2024-08-17T17:37:40.030048+0000","last_became_active":"2024-08-17T17:33:44.145283+0000","last_became_peered":"2024-08-17T17:33:44.145283+0000","last_unstale":"2024-08-17T17:37:40.030048+0000","last_undegraded":"2024-08-17T17:37:40.030048+0000","last_fullsized":"2024-08-17T17:37:40.030048+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:11.965226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"314'1546","reported_seq":1563,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:59.744181+0000","last_change":"2024-08-17T17:37:38.806674+0000","last_active":"2024-08-17T17:37:59.744181+0000","last_peered":"2024-08-17T17:37:59.744181+0000","last_clean":"2024-08-17T17:37:59.744181+0000","last_became_active":"2024-08-17T17:37:38.806352+0000","last_became_peered":"2024-08-17T17:37:38.806352+0000","last_unstale":"2024-08-17T17:37:59.744181+0000","last_undegraded":"2024-08-17T17:37:59.744181+0000","last_fullsized":"2024-08-17T17:37:59.744181+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":1546,"log_dups_size":0,"ondisk_log_size":1546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:27:58.888880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101318656,"num_objects":1546,"num_object_clones":0,"num_object_copies":4638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1546,"num_write_kb":98944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":243,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:39.818830+0000","last_change":"2024-08-17T17:37:00.416059+0000","last_active":"2024-08-17T17:37:39.818830+0000","last_peered":"2024-08-17T17:37:39.818830+0000","last_clean":"2024-08-17T17:37:39.818830+0000","last_became_active":"2024-08-17T17:37:00.385913+0000","last_became_peered":"2024-08-17T17:37:00.385913+0000","last_unstale":"2024-08-17T17:37:39.818830+0000","last_undegraded":"2024-08-17T17:37:39.818830+0000","last_fullsized":"2024-08-17T17:37:39.818830+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:11:24.884146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":77,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:40.030165+0000","last_change":"2024-08-17T17:33:21.766549+0000","last_active":"2024-08-17T17:37:40.030165+0000","last_peered":"2024-08-17T17:37:40.030165+0000","last_clean":"2024-08-17T17:37:40.030165+0000","last_became_active":"2024-08-17T17:33:21.766152+0000","last_became_peered":"2024-08-17T17:33:21.766152+0000","last_unstale":"2024-08-17T17:37:40.030165+0000","last_undegraded":"2024-08-17T17:37:40.030165+0000","last_fullsized":"2024-08-17T17:37:40.030165+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:56:15.855524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"314'1410","reported_seq":1427,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:58.913072+0000","last_change":"2024-08-17T17:37:39.459385+0000","last_active":"2024-08-17T17:37:58.913072+0000","last_peered":"2024-08-17T17:37:58.913072+0000","last_clean":"2024-08-17T17:37:58.913072+0000","last_became_active":"2024-08-17T17:37:39.458883+0000","last_became_peered":"2024-08-17T17:37:39.458883+0000","last_unstale":"2024-08-17T17:37:58.913072+0000","last_undegraded":"2024-08-17T17:37:58.913072+0000","last_fullsized":"2024-08-17T17:37:58.913072+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":1410,"log_dups_size":0,"ondisk_log_size":1410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:02:52.635742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92405760,"num_objects":1410,"num_object_clones":0,"num_object_copies":4230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1410,"num_write_kb":90240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":65,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:40.031060+0000","last_change":"2024-08-17T17:33:44.145399+0000","last_active":"2024-08-17T17:37:40.031060+0000","last_peered":"2024-08-17T17:37:40.031060+0000","last_clean":"2024-08-17T17:37:40.031060+0000","last_became_active":"2024-08-17T17:33:44.144751+0000","last_became_peered":"2024-08-17T17:33:44.144751+0000","last_unstale":"2024-08-17T17:37:40.031060+0000","last_undegraded":"2024-08-17T17:37:40.031060+0000","last_fullsized":"2024-08-17T17:37:40.031060+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:54.276350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":143,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:40.031560+0000","last_change":"2024-08-17T17:33:20.795582+0000","last_active":"2024-08-17T17:37:40.031560+0000","last_peered":"2024-08-17T17:37:40.031560+0000","last_clean":"2024-08-17T17:37:40.031560+0000","last_became_active":"2024-08-17T17:33:20.795149+0000","last_became_peered":"2024-08-17T17:33:20.795149+0000","last_unstale":"2024-08-17T17:37:40.031560+0000","last_undegraded":"2024-08-17T17:37:40.031560+0000","last_fullsized":"2024-08-17T17:37:40.031560+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:14.020923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"314'1578","reported_seq":1595,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:59.758000+0000","last_change":"2024-08-17T17:37:39.458059+0000","last_active":"2024-08-17T17:37:59.758000+0000","last_peered":"2024-08-17T17:37:59.758000+0000","last_clean":"2024-08-17T17:37:59.758000+0000","last_became_active":"2024-08-17T17:37:39.457731+0000","last_became_peered":"2024-08-17T17:37:39.457731+0000","last_unstale":"2024-08-17T17:37:59.758000+0000","last_undegraded":"2024-08-17T17:37:59.758000+0000","last_fullsized":"2024-08-17T17:37:59.758000+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:08:04.029927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103415808,"num_objects":1578,"num_object_clones":0,"num_object_copies":4734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1578,"num_write_kb":100992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":155,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.860550+0000","last_change":"2024-08-17T17:33:44.154097+0000","last_active":"2024-08-17T17:37:48.860550+0000","last_peered":"2024-08-17T17:37:48.860550+0000","last_clean":"2024-08-17T17:37:48.860550+0000","last_became_active":"2024-08-17T17:33:44.152972+0000","last_became_peered":"2024-08-17T17:33:44.152972+0000","last_unstale":"2024-08-17T17:37:48.860550+0000","last_undegraded":"2024-08-17T17:37:48.860550+0000","last_fullsized":"2024-08-17T17:37:48.860550+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:42.602017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":505,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.861220+0000","last_change":"2024-08-17T17:33:21.763677+0000","last_active":"2024-08-17T17:37:48.861220+0000","last_peered":"2024-08-17T17:37:48.861220+0000","last_clean":"2024-08-17T17:37:48.861220+0000","last_became_active":"2024-08-17T17:33:21.760564+0000","last_became_peered":"2024-08-17T17:33:21.760564+0000","last_unstale":"2024-08-17T17:37:48.861220+0000","last_undegraded":"2024-08-17T17:37:48.861220+0000","last_fullsized":"2024-08-17T17:37:48.861220+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:01:00.772037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1","version":"314'1426","reported_seq":1443,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:58.920161+0000","last_change":"2024-08-17T17:37:38.803662+0000","last_active":"2024-08-17T17:37:58.920161+0000","last_peered":"2024-08-17T17:37:58.920161+0000","last_clean":"2024-08-17T17:37:58.920161+0000","last_became_active":"2024-08-17T17:37:38.803355+0000","last_became_peered":"2024-08-17T17:37:38.803355+0000","last_unstale":"2024-08-17T17:37:58.920161+0000","last_undegraded":"2024-08-17T17:37:58.920161+0000","last_fullsized":"2024-08-17T17:37:58.920161+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":1426,"log_dups_size":0,"ondisk_log_size":1426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:16:15.411410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93454336,"num_objects":1426,"num_object_clones":0,"num_object_copies":4278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1426,"num_write_kb":91264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":643,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.859520+0000","last_change":"2024-08-17T17:37:32.778073+0000","last_active":"2024-08-17T17:37:48.859520+0000","last_peered":"2024-08-17T17:37:48.859520+0000","last_clean":"2024-08-17T17:37:48.859520+0000","last_became_active":"2024-08-17T17:37:32.777646+0000","last_became_peered":"2024-08-17T17:37:32.777646+0000","last_unstale":"2024-08-17T17:37:48.859520+0000","last_undegraded":"2024-08-17T17:37:48.859520+0000","last_fullsized":"2024-08-17T17:37:48.859520+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:35:04.265381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:35:04.265381+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:24:55.329104+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":533,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.860064+0000","last_change":"2024-08-17T17:32:46.387810+0000","last_active":"2024-08-17T17:37:48.860064+0000","last_peered":"2024-08-17T17:37:48.860064+0000","last_clean":"2024-08-17T17:37:48.860064+0000","last_became_active":"2024-08-17T17:32:46.374238+0000","last_became_peered":"2024-08-17T17:32:46.374238+0000","last_unstale":"2024-08-17T17:37:48.860064+0000","last_undegraded":"2024-08-17T17:37:48.860064+0000","last_fullsized":"2024-08-17T17:37:48.860064+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:53:29.918005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":63,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:40.030933+0000","last_change":"2024-08-17T17:32:46.378495+0000","last_active":"2024-08-17T17:37:40.030933+0000","last_peered":"2024-08-17T17:37:40.030933+0000","last_clean":"2024-08-17T17:37:40.030933+0000","last_became_active":"2024-08-17T17:32:46.377709+0000","last_became_peered":"2024-08-17T17:32:46.377709+0000","last_unstale":"2024-08-17T17:37:40.030933+0000","last_undegraded":"2024-08-17T17:37:40.030933+0000","last_fullsized":"2024-08-17T17:37:40.030933+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:22:52.544851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"314'1523","reported_seq":1540,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:59.760247+0000","last_change":"2024-08-17T17:37:38.939023+0000","last_active":"2024-08-17T17:37:59.760247+0000","last_peered":"2024-08-17T17:37:59.760247+0000","last_clean":"2024-08-17T17:37:59.760247+0000","last_became_active":"2024-08-17T17:37:38.938675+0000","last_became_peered":"2024-08-17T17:37:38.938675+0000","last_unstale":"2024-08-17T17:37:59.760247+0000","last_undegraded":"2024-08-17T17:37:59.760247+0000","last_fullsized":"2024-08-17T17:37:59.760247+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":1523,"log_dups_size":0,"ondisk_log_size":1523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:59:47.176803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99811328,"num_objects":1523,"num_object_clones":0,"num_object_copies":4569,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1523,"num_write_kb":97472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":643,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.859466+0000","last_change":"2024-08-17T17:37:32.781009+0000","last_active":"2024-08-17T17:37:48.859466+0000","last_peered":"2024-08-17T17:37:48.859466+0000","last_clean":"2024-08-17T17:37:48.859466+0000","last_became_active":"2024-08-17T17:37:32.780627+0000","last_became_peered":"2024-08-17T17:37:32.780627+0000","last_unstale":"2024-08-17T17:37:48.859466+0000","last_undegraded":"2024-08-17T17:37:48.859466+0000","last_fullsized":"2024-08-17T17:37:48.859466+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:35:02.239705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:35:02.239705+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:19:13.130232+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":77,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:40.030150+0000","last_change":"2024-08-17T17:33:21.766756+0000","last_active":"2024-08-17T17:37:40.030150+0000","last_peered":"2024-08-17T17:37:40.030150+0000","last_clean":"2024-08-17T17:37:40.030150+0000","last_became_active":"2024-08-17T17:33:21.766347+0000","last_became_peered":"2024-08-17T17:33:21.766347+0000","last_unstale":"2024-08-17T17:37:40.030150+0000","last_undegraded":"2024-08-17T17:37:40.030150+0000","last_fullsized":"2024-08-17T17:37:40.030150+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:13:14.655176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":24,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:40.031400+0000","last_change":"2024-08-17T17:37:00.386391+0000","last_active":"2024-08-17T17:37:40.031400+0000","last_peered":"2024-08-17T17:37:40.031400+0000","last_clean":"2024-08-17T17:37:40.031400+0000","last_became_active":"2024-08-17T17:37:00.385803+0000","last_became_peered":"2024-08-17T17:37:00.385803+0000","last_unstale":"2024-08-17T17:37:40.031400+0000","last_undegraded":"2024-08-17T17:37:40.031400+0000","last_fullsized":"2024-08-17T17:37:40.031400+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:59:49.051157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.3","version":"314'1463","reported_seq":1474,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:59.209484+0000","last_change":"2024-08-17T17:37:38.804823+0000","last_active":"2024-08-17T17:37:59.209484+0000","last_peered":"2024-08-17T17:37:59.209484+0000","last_clean":"2024-08-17T17:37:59.209484+0000","last_became_active":"2024-08-17T17:37:38.804431+0000","last_became_peered":"2024-08-17T17:37:38.804431+0000","last_unstale":"2024-08-17T17:37:59.209484+0000","last_undegraded":"2024-08-17T17:37:59.209484+0000","last_fullsized":"2024-08-17T17:37:59.209484+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":1463,"log_dups_size":0,"ondisk_log_size":1463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:03:53.415193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95879168,"num_objects":1463,"num_object_clones":0,"num_object_copies":4389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1463,"num_write_kb":93632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:40.029934+0000","last_change":"2024-08-17T17:37:00.376047+0000","last_active":"2024-08-17T17:37:40.029934+0000","last_peered":"2024-08-17T17:37:40.029934+0000","last_clean":"2024-08-17T17:37:40.029934+0000","last_became_active":"2024-08-17T17:37:00.375687+0000","last_became_peered":"2024-08-17T17:37:00.375687+0000","last_unstale":"2024-08-17T17:37:40.029934+0000","last_undegraded":"2024-08-17T17:37:40.029934+0000","last_fullsized":"2024-08-17T17:37:40.029934+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:57:17.605170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":77,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:40.031630+0000","last_change":"2024-08-17T17:33:21.764934+0000","last_active":"2024-08-17T17:37:40.031630+0000","last_peered":"2024-08-17T17:37:40.031630+0000","last_clean":"2024-08-17T17:37:40.031630+0000","last_became_active":"2024-08-17T17:33:21.764341+0000","last_became_peered":"2024-08-17T17:33:21.764341+0000","last_unstale":"2024-08-17T17:37:40.031630+0000","last_undegraded":"2024-08-17T17:37:40.031630+0000","last_fullsized":"2024-08-17T17:37:40.031630+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:20:25.714104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"314'1469","reported_seq":1486,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:58.918855+0000","last_change":"2024-08-17T17:37:38.939685+0000","last_active":"2024-08-17T17:37:58.918855+0000","last_peered":"2024-08-17T17:37:58.918855+0000","last_clean":"2024-08-17T17:37:58.918855+0000","last_became_active":"2024-08-17T17:37:38.939345+0000","last_became_peered":"2024-08-17T17:37:38.939345+0000","last_unstale":"2024-08-17T17:37:58.918855+0000","last_undegraded":"2024-08-17T17:37:58.918855+0000","last_fullsized":"2024-08-17T17:37:58.918855+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":1469,"log_dups_size":0,"ondisk_log_size":1469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:07:23.306893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96272384,"num_objects":1469,"num_object_clones":0,"num_object_copies":4407,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1469,"num_write_kb":94016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":402,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:40.031415+0000","last_change":"2024-08-17T17:37:32.776163+0000","last_active":"2024-08-17T17:37:40.031415+0000","last_peered":"2024-08-17T17:37:40.031415+0000","last_clean":"2024-08-17T17:37:40.031415+0000","last_became_active":"2024-08-17T17:37:32.775713+0000","last_became_peered":"2024-08-17T17:37:32.775713+0000","last_unstale":"2024-08-17T17:37:40.031415+0000","last_undegraded":"2024-08-17T17:37:40.031415+0000","last_fullsized":"2024-08-17T17:37:40.031415+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:35:09.302394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:35:09.302394+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":74,"num_write":19,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":513,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.860777+0000","last_change":"2024-08-17T17:33:20.796216+0000","last_active":"2024-08-17T17:37:48.860777+0000","last_peered":"2024-08-17T17:37:48.860777+0000","last_clean":"2024-08-17T17:37:48.860777+0000","last_became_active":"2024-08-17T17:33:20.795762+0000","last_became_peered":"2024-08-17T17:33:20.795762+0000","last_unstale":"2024-08-17T17:37:48.860777+0000","last_undegraded":"2024-08-17T17:37:48.860777+0000","last_fullsized":"2024-08-17T17:37:48.860777+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:47.038509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":395,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:47.841057+0000","last_change":"2024-08-17T17:33:21.976938+0000","last_active":"2024-08-17T17:37:47.841057+0000","last_peered":"2024-08-17T17:37:47.841057+0000","last_clean":"2024-08-17T17:37:47.841057+0000","last_became_active":"2024-08-17T17:33:21.976585+0000","last_became_peered":"2024-08-17T17:33:21.976585+0000","last_unstale":"2024-08-17T17:37:47.841057+0000","last_undegraded":"2024-08-17T17:37:47.841057+0000","last_fullsized":"2024-08-17T17:37:47.841057+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:23:40.758870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"314'1423","reported_seq":1440,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:58.899460+0000","last_change":"2024-08-17T17:37:39.459686+0000","last_active":"2024-08-17T17:37:58.899460+0000","last_peered":"2024-08-17T17:37:58.899460+0000","last_clean":"2024-08-17T17:37:58.899460+0000","last_became_active":"2024-08-17T17:37:39.459260+0000","last_became_peered":"2024-08-17T17:37:39.459260+0000","last_unstale":"2024-08-17T17:37:58.899460+0000","last_undegraded":"2024-08-17T17:37:58.899460+0000","last_fullsized":"2024-08-17T17:37:58.899460+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":1423,"log_dups_size":0,"ondisk_log_size":1423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:54:14.542106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93257728,"num_objects":1423,"num_object_clones":0,"num_object_copies":4269,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1423,"num_write_kb":91072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":629,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.860047+0000","last_change":"2024-08-17T17:34:18.238306+0000","last_active":"2024-08-17T17:37:48.860047+0000","last_peered":"2024-08-17T17:37:48.860047+0000","last_clean":"2024-08-17T17:37:48.860047+0000","last_became_active":"2024-08-17T17:34:17.878280+0000","last_became_peered":"2024-08-17T17:34:17.878280+0000","last_unstale":"2024-08-17T17:37:48.860047+0000","last_undegraded":"2024-08-17T17:37:48.860047+0000","last_fullsized":"2024-08-17T17:37:48.860047+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:34:18.238171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:34:18.238171+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:23:10.219823+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":720,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:39.818209+0000","last_change":"2024-08-17T17:33:44.149928+0000","last_active":"2024-08-17T17:37:39.818209+0000","last_peered":"2024-08-17T17:37:39.818209+0000","last_clean":"2024-08-17T17:37:39.818209+0000","last_became_active":"2024-08-17T17:33:44.149498+0000","last_became_peered":"2024-08-17T17:33:44.149498+0000","last_unstale":"2024-08-17T17:37:39.818209+0000","last_undegraded":"2024-08-17T17:37:39.818209+0000","last_fullsized":"2024-08-17T17:37:39.818209+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:17:04.963202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":460,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.859739+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:37:48.859739+0000","last_peered":"2024-08-17T17:37:48.859739+0000","last_clean":"2024-08-17T17:37:48.859739+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:37:48.859739+0000","last_undegraded":"2024-08-17T17:37:48.859739+0000","last_fullsized":"2024-08-17T17:37:48.859739+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":136,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:40.031326+0000","last_change":"2024-08-17T17:37:00.384064+0000","last_active":"2024-08-17T17:37:40.031326+0000","last_peered":"2024-08-17T17:37:40.031326+0000","last_clean":"2024-08-17T17:37:40.031326+0000","last_became_active":"2024-08-17T17:37:00.383681+0000","last_became_peered":"2024-08-17T17:37:00.383681+0000","last_unstale":"2024-08-17T17:37:40.031326+0000","last_undegraded":"2024-08-17T17:37:40.031326+0000","last_fullsized":"2024-08-17T17:37:40.031326+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:10:22.569694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"314'1483","reported_seq":1500,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:59.189480+0000","last_change":"2024-08-17T17:37:39.458006+0000","last_active":"2024-08-17T17:37:59.189480+0000","last_peered":"2024-08-17T17:37:59.189480+0000","last_clean":"2024-08-17T17:37:59.189480+0000","last_became_active":"2024-08-17T17:37:39.457681+0000","last_became_peered":"2024-08-17T17:37:39.457681+0000","last_unstale":"2024-08-17T17:37:59.189480+0000","last_undegraded":"2024-08-17T17:37:59.189480+0000","last_fullsized":"2024-08-17T17:37:59.189480+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":1483,"log_dups_size":0,"ondisk_log_size":1483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:09:17.360518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97189888,"num_objects":1483,"num_object_clones":0,"num_object_copies":4449,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1483,"num_write_kb":94912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":647,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:47.840209+0000","last_change":"2024-08-17T17:37:32.776749+0000","last_active":"2024-08-17T17:37:47.840209+0000","last_peered":"2024-08-17T17:37:47.840209+0000","last_clean":"2024-08-17T17:37:47.840209+0000","last_became_active":"2024-08-17T17:37:32.776334+0000","last_became_peered":"2024-08-17T17:37:32.776334+0000","last_unstale":"2024-08-17T17:37:47.840209+0000","last_undegraded":"2024-08-17T17:37:47.840209+0000","last_fullsized":"2024-08-17T17:37:47.840209+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:35:07.285633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:35:07.285633+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:39:17.743649+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":745,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:47.839775+0000","last_change":"2024-08-17T17:33:20.797705+0000","last_active":"2024-08-17T17:37:47.839775+0000","last_peered":"2024-08-17T17:37:47.839775+0000","last_clean":"2024-08-17T17:37:47.839775+0000","last_became_active":"2024-08-17T17:33:20.797172+0000","last_became_peered":"2024-08-17T17:33:20.797172+0000","last_unstale":"2024-08-17T17:37:47.839775+0000","last_undegraded":"2024-08-17T17:37:47.839775+0000","last_fullsized":"2024-08-17T17:37:47.839775+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:30.754999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":496,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.860921+0000","last_change":"2024-08-17T17:37:00.411061+0000","last_active":"2024-08-17T17:37:48.860921+0000","last_peered":"2024-08-17T17:37:48.860921+0000","last_clean":"2024-08-17T17:37:48.860921+0000","last_became_active":"2024-08-17T17:37:00.410374+0000","last_became_peered":"2024-08-17T17:37:00.410374+0000","last_unstale":"2024-08-17T17:37:48.860921+0000","last_undegraded":"2024-08-17T17:37:48.860921+0000","last_fullsized":"2024-08-17T17:37:48.860921+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:09:04.878864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":298,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.861488+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:37:48.861488+0000","last_peered":"2024-08-17T17:37:48.861488+0000","last_clean":"2024-08-17T17:37:48.861488+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:37:48.861488+0000","last_undegraded":"2024-08-17T17:37:48.861488+0000","last_fullsized":"2024-08-17T17:37:48.861488+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"314'1561","reported_seq":1578,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:59.766432+0000","last_change":"2024-08-17T17:37:39.458235+0000","last_active":"2024-08-17T17:37:59.766432+0000","last_peered":"2024-08-17T17:37:59.766432+0000","last_clean":"2024-08-17T17:37:59.766432+0000","last_became_active":"2024-08-17T17:37:39.457773+0000","last_became_peered":"2024-08-17T17:37:39.457773+0000","last_unstale":"2024-08-17T17:37:59.766432+0000","last_undegraded":"2024-08-17T17:37:59.766432+0000","last_fullsized":"2024-08-17T17:37:59.766432+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":1561,"log_dups_size":0,"ondisk_log_size":1561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:54:08.694457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102301696,"num_objects":1561,"num_object_clones":0,"num_object_copies":4683,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1561,"num_write_kb":99904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":649,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.859563+0000","last_change":"2024-08-17T17:37:32.778666+0000","last_active":"2024-08-17T17:37:48.859563+0000","last_peered":"2024-08-17T17:37:48.859563+0000","last_clean":"2024-08-17T17:37:48.859563+0000","last_became_active":"2024-08-17T17:37:32.778265+0000","last_became_peered":"2024-08-17T17:37:32.778265+0000","last_unstale":"2024-08-17T17:37:48.859563+0000","last_undegraded":"2024-08-17T17:37:48.859563+0000","last_fullsized":"2024-08-17T17:37:48.859563+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:35:01.201049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:35:01.201049+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:06:24.348791+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":877,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.859926+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:37:48.859926+0000","last_peered":"2024-08-17T17:37:48.859926+0000","last_clean":"2024-08-17T17:37:48.859926+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:37:48.859926+0000","last_undegraded":"2024-08-17T17:37:48.859926+0000","last_fullsized":"2024-08-17T17:37:48.859926+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":383,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:47.839929+0000","last_change":"2024-08-17T17:33:20.793468+0000","last_active":"2024-08-17T17:37:47.839929+0000","last_peered":"2024-08-17T17:37:47.839929+0000","last_clean":"2024-08-17T17:37:47.839929+0000","last_became_active":"2024-08-17T17:33:20.793166+0000","last_became_peered":"2024-08-17T17:33:20.793166+0000","last_unstale":"2024-08-17T17:37:47.839929+0000","last_undegraded":"2024-08-17T17:37:47.839929+0000","last_fullsized":"2024-08-17T17:37:47.839929+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:12:52.121050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":419,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:47.840940+0000","last_change":"2024-08-17T17:33:20.835784+0000","last_active":"2024-08-17T17:37:47.840940+0000","last_peered":"2024-08-17T17:37:47.840940+0000","last_clean":"2024-08-17T17:37:47.840940+0000","last_became_active":"2024-08-17T17:33:20.835371+0000","last_became_peered":"2024-08-17T17:33:20.835371+0000","last_unstale":"2024-08-17T17:37:47.840940+0000","last_undegraded":"2024-08-17T17:37:47.840940+0000","last_fullsized":"2024-08-17T17:37:47.840940+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:35:43.369055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"314'1480","reported_seq":1497,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:59.188975+0000","last_change":"2024-08-17T17:37:39.458043+0000","last_active":"2024-08-17T17:37:59.188975+0000","last_peered":"2024-08-17T17:37:59.188975+0000","last_clean":"2024-08-17T17:37:59.188975+0000","last_became_active":"2024-08-17T17:37:39.457730+0000","last_became_peered":"2024-08-17T17:37:39.457730+0000","last_unstale":"2024-08-17T17:37:59.188975+0000","last_undegraded":"2024-08-17T17:37:59.188975+0000","last_fullsized":"2024-08-17T17:37:59.188975+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":1480,"log_dups_size":0,"ondisk_log_size":1480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:14:23.151646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96993280,"num_objects":1480,"num_object_clones":0,"num_object_copies":4440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1480,"num_write_kb":94720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":648,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.860409+0000","last_change":"2024-08-17T17:37:32.776954+0000","last_active":"2024-08-17T17:37:48.860409+0000","last_peered":"2024-08-17T17:37:48.860409+0000","last_clean":"2024-08-17T17:37:48.860409+0000","last_became_active":"2024-08-17T17:37:32.776572+0000","last_became_peered":"2024-08-17T17:37:32.776572+0000","last_unstale":"2024-08-17T17:37:48.860409+0000","last_undegraded":"2024-08-17T17:37:48.860409+0000","last_fullsized":"2024-08-17T17:37:48.860409+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:35:03.210298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:35:03.210298+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:08:58.261234+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":968,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:47.840086+0000","last_change":"2024-08-17T17:33:44.148227+0000","last_active":"2024-08-17T17:37:47.840086+0000","last_peered":"2024-08-17T17:37:47.840086+0000","last_clean":"2024-08-17T17:37:47.840086+0000","last_became_active":"2024-08-17T17:33:44.147627+0000","last_became_peered":"2024-08-17T17:33:44.147627+0000","last_unstale":"2024-08-17T17:37:47.840086+0000","last_undegraded":"2024-08-17T17:37:47.840086+0000","last_fullsized":"2024-08-17T17:37:47.840086+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:29:42.898460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1372,"num_read_kb":2843,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":63,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:40.030356+0000","last_change":"2024-08-17T17:32:46.379200+0000","last_active":"2024-08-17T17:37:40.030356+0000","last_peered":"2024-08-17T17:37:40.030356+0000","last_clean":"2024-08-17T17:37:40.030356+0000","last_became_active":"2024-08-17T17:32:46.378870+0000","last_became_peered":"2024-08-17T17:32:46.378870+0000","last_unstale":"2024-08-17T17:37:40.030356+0000","last_undegraded":"2024-08-17T17:37:40.030356+0000","last_fullsized":"2024-08-17T17:37:40.030356+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:55:26.885966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":423,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:47.840483+0000","last_change":"2024-08-17T17:33:20.772454+0000","last_active":"2024-08-17T17:37:47.840483+0000","last_peered":"2024-08-17T17:37:47.840483+0000","last_clean":"2024-08-17T17:37:47.840483+0000","last_became_active":"2024-08-17T17:33:20.771848+0000","last_became_peered":"2024-08-17T17:33:20.771848+0000","last_unstale":"2024-08-17T17:37:47.840483+0000","last_undegraded":"2024-08-17T17:37:47.840483+0000","last_fullsized":"2024-08-17T17:37:47.840483+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:46:42.300744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.f","version":"314'1578","reported_seq":1595,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:59.762080+0000","last_change":"2024-08-17T17:37:39.458090+0000","last_active":"2024-08-17T17:37:59.762080+0000","last_peered":"2024-08-17T17:37:59.762080+0000","last_clean":"2024-08-17T17:37:59.762080+0000","last_became_active":"2024-08-17T17:37:39.457805+0000","last_became_peered":"2024-08-17T17:37:39.457805+0000","last_unstale":"2024-08-17T17:37:59.762080+0000","last_undegraded":"2024-08-17T17:37:59.762080+0000","last_fullsized":"2024-08-17T17:37:59.762080+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":1578,"log_dups_size":0,"ondisk_log_size":1578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:02:18.222490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103415808,"num_objects":1578,"num_object_clones":0,"num_object_copies":4734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1578,"num_write_kb":100992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":402,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:40.031145+0000","last_change":"2024-08-17T17:37:32.780006+0000","last_active":"2024-08-17T17:37:40.031145+0000","last_peered":"2024-08-17T17:37:40.031145+0000","last_clean":"2024-08-17T17:37:40.031145+0000","last_became_active":"2024-08-17T17:37:32.778851+0000","last_became_peered":"2024-08-17T17:37:32.778851+0000","last_unstale":"2024-08-17T17:37:40.031145+0000","last_undegraded":"2024-08-17T17:37:40.031145+0000","last_fullsized":"2024-08-17T17:37:40.031145+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:35:05.269844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:35:05.269844+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":34,"num_read_kb":75,"num_write":19,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":173,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.860324+0000","last_change":"2024-08-17T17:33:44.163867+0000","last_active":"2024-08-17T17:37:48.860324+0000","last_peered":"2024-08-17T17:37:48.860324+0000","last_clean":"2024-08-17T17:37:48.860324+0000","last_became_active":"2024-08-17T17:33:44.163503+0000","last_became_peered":"2024-08-17T17:33:44.163503+0000","last_unstale":"2024-08-17T17:37:48.860324+0000","last_undegraded":"2024-08-17T17:37:48.860324+0000","last_fullsized":"2024-08-17T17:37:48.860324+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:31:53.573596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":638,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:47.840324+0000","last_change":"2024-08-17T17:33:20.799350+0000","last_active":"2024-08-17T17:37:47.840324+0000","last_peered":"2024-08-17T17:37:47.840324+0000","last_clean":"2024-08-17T17:37:47.840324+0000","last_became_active":"2024-08-17T17:33:20.798925+0000","last_became_peered":"2024-08-17T17:33:20.798925+0000","last_unstale":"2024-08-17T17:37:47.840324+0000","last_undegraded":"2024-08-17T17:37:47.840324+0000","last_fullsized":"2024-08-17T17:37:47.840324+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:16:36.121295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":440,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:47.840395+0000","last_change":"2024-08-17T17:37:00.397108+0000","last_active":"2024-08-17T17:37:47.840395+0000","last_peered":"2024-08-17T17:37:47.840395+0000","last_clean":"2024-08-17T17:37:47.840395+0000","last_became_active":"2024-08-17T17:37:00.395931+0000","last_became_peered":"2024-08-17T17:37:00.395931+0000","last_unstale":"2024-08-17T17:37:47.840395+0000","last_undegraded":"2024-08-17T17:37:47.840395+0000","last_fullsized":"2024-08-17T17:37:47.840395+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:54:31.517826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.e","version":"314'1449","reported_seq":1460,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:58.767940+0000","last_change":"2024-08-17T17:37:38.801415+0000","last_active":"2024-08-17T17:37:58.767940+0000","last_peered":"2024-08-17T17:37:58.767940+0000","last_clean":"2024-08-17T17:37:58.767940+0000","last_became_active":"2024-08-17T17:37:38.800618+0000","last_became_peered":"2024-08-17T17:37:38.800618+0000","last_unstale":"2024-08-17T17:37:58.767940+0000","last_undegraded":"2024-08-17T17:37:58.767940+0000","last_fullsized":"2024-08-17T17:37:58.767940+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":1449,"log_dups_size":0,"ondisk_log_size":1449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:29:57.226272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94961664,"num_objects":1449,"num_object_clones":0,"num_object_copies":4347,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1449,"num_write_kb":92736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":412,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:40.031174+0000","last_change":"2024-08-17T17:37:33.743367+0000","last_active":"2024-08-17T17:37:40.031174+0000","last_peered":"2024-08-17T17:37:40.031174+0000","last_clean":"2024-08-17T17:37:40.031174+0000","last_became_active":"2024-08-17T17:37:32.778778+0000","last_became_peered":"2024-08-17T17:37:32.778778+0000","last_unstale":"2024-08-17T17:37:40.031174+0000","last_undegraded":"2024-08-17T17:37:40.031174+0000","last_fullsized":"2024-08-17T17:37:40.031174+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:37:33.743295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:37:33.743295+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:47:51.392703+0000","scrub_duration":0.001909006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":458,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:47.840152+0000","last_change":"2024-08-17T17:33:20.798613+0000","last_active":"2024-08-17T17:37:47.840152+0000","last_peered":"2024-08-17T17:37:47.840152+0000","last_clean":"2024-08-17T17:37:47.840152+0000","last_became_active":"2024-08-17T17:33:20.798260+0000","last_became_peered":"2024-08-17T17:33:20.798260+0000","last_unstale":"2024-08-17T17:37:47.840152+0000","last_undegraded":"2024-08-17T17:37:47.840152+0000","last_fullsized":"2024-08-17T17:37:47.840152+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:18:00.822681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":790,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:39.818756+0000","last_change":"2024-08-17T17:32:49.379365+0000","last_active":"2024-08-17T17:37:39.818756+0000","last_peered":"2024-08-17T17:37:39.818756+0000","last_clean":"2024-08-17T17:37:39.818756+0000","last_became_active":"2024-08-17T17:32:49.378742+0000","last_became_peered":"2024-08-17T17:32:49.378742+0000","last_unstale":"2024-08-17T17:37:39.818756+0000","last_undegraded":"2024-08-17T17:37:39.818756+0000","last_fullsized":"2024-08-17T17:37:39.818756+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:32:06.957948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":472,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.861130+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:37:48.861130+0000","last_peered":"2024-08-17T17:37:48.861130+0000","last_clean":"2024-08-17T17:37:48.861130+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:37:48.861130+0000","last_undegraded":"2024-08-17T17:37:48.861130+0000","last_fullsized":"2024-08-17T17:37:48.861130+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"314'1635","reported_seq":1652,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:59.762472+0000","last_change":"2024-08-17T17:37:39.460626+0000","last_active":"2024-08-17T17:37:59.762472+0000","last_peered":"2024-08-17T17:37:59.762472+0000","last_clean":"2024-08-17T17:37:59.762472+0000","last_became_active":"2024-08-17T17:37:39.459631+0000","last_became_peered":"2024-08-17T17:37:39.459631+0000","last_unstale":"2024-08-17T17:37:59.762472+0000","last_undegraded":"2024-08-17T17:37:59.762472+0000","last_fullsized":"2024-08-17T17:37:59.762472+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":1635,"log_dups_size":0,"ondisk_log_size":1635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:51:35.697161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107151360,"num_objects":1635,"num_object_clones":0,"num_object_copies":4905,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1635,"num_write_kb":104640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":642,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:40.031124+0000","last_change":"2024-08-17T17:37:32.777579+0000","last_active":"2024-08-17T17:37:40.031124+0000","last_peered":"2024-08-17T17:37:40.031124+0000","last_clean":"2024-08-17T17:37:40.031124+0000","last_became_active":"2024-08-17T17:37:32.777142+0000","last_became_peered":"2024-08-17T17:37:32.777142+0000","last_unstale":"2024-08-17T17:37:40.031124+0000","last_undegraded":"2024-08-17T17:37:40.031124+0000","last_fullsized":"2024-08-17T17:37:40.031124+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:35:06.281848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:35:06.281848+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:20:50.701748+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":507,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.860597+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:37:48.860597+0000","last_peered":"2024-08-17T17:37:48.860597+0000","last_clean":"2024-08-17T17:37:48.860597+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:37:48.860597+0000","last_undegraded":"2024-08-17T17:37:48.860597+0000","last_fullsized":"2024-08-17T17:37:48.860597+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":991,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:47.840727+0000","last_change":"2024-08-17T17:33:21.763805+0000","last_active":"2024-08-17T17:37:47.840727+0000","last_peered":"2024-08-17T17:37:47.840727+0000","last_clean":"2024-08-17T17:37:47.840727+0000","last_became_active":"2024-08-17T17:33:21.763232+0000","last_became_peered":"2024-08-17T17:33:21.763232+0000","last_unstale":"2024-08-17T17:37:47.840727+0000","last_undegraded":"2024-08-17T17:37:47.840727+0000","last_fullsized":"2024-08-17T17:37:47.840727+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:46:08.994820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":504,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.860588+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:37:48.860588+0000","last_peered":"2024-08-17T17:37:48.860588+0000","last_clean":"2024-08-17T17:37:48.860588+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:37:48.860588+0000","last_undegraded":"2024-08-17T17:37:48.860588+0000","last_fullsized":"2024-08-17T17:37:48.860588+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"314'1468","reported_seq":1485,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:58.917013+0000","last_change":"2024-08-17T17:37:38.805976+0000","last_active":"2024-08-17T17:37:58.917013+0000","last_peered":"2024-08-17T17:37:58.917013+0000","last_clean":"2024-08-17T17:37:58.917013+0000","last_became_active":"2024-08-17T17:37:38.805541+0000","last_became_peered":"2024-08-17T17:37:38.805541+0000","last_unstale":"2024-08-17T17:37:58.917013+0000","last_undegraded":"2024-08-17T17:37:58.917013+0000","last_fullsized":"2024-08-17T17:37:58.917013+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":1468,"log_dups_size":0,"ondisk_log_size":1468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:40:37.333614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96206848,"num_objects":1468,"num_object_clones":0,"num_object_copies":4404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1468,"num_write_kb":93952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":631,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.859513+0000","last_change":"2024-08-17T17:37:32.778597+0000","last_active":"2024-08-17T17:37:48.859513+0000","last_peered":"2024-08-17T17:37:48.859513+0000","last_clean":"2024-08-17T17:37:48.859513+0000","last_became_active":"2024-08-17T17:37:32.778167+0000","last_became_peered":"2024-08-17T17:37:32.778167+0000","last_unstale":"2024-08-17T17:37:48.859513+0000","last_undegraded":"2024-08-17T17:37:48.859513+0000","last_fullsized":"2024-08-17T17:37:48.859513+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:34:19.223644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:34:19.223644+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:06:31.553942+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":322,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.861434+0000","last_change":"2024-08-17T17:33:21.765925+0000","last_active":"2024-08-17T17:37:48.861434+0000","last_peered":"2024-08-17T17:37:48.861434+0000","last_clean":"2024-08-17T17:37:48.861434+0000","last_became_active":"2024-08-17T17:33:21.765372+0000","last_became_peered":"2024-08-17T17:33:21.765372+0000","last_unstale":"2024-08-17T17:37:48.861434+0000","last_undegraded":"2024-08-17T17:37:48.861434+0000","last_fullsized":"2024-08-17T17:37:48.861434+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:14:47.737369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":545,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:40.031368+0000","last_change":"2024-08-17T17:26:06.855307+0000","last_active":"2024-08-17T17:37:40.031368+0000","last_peered":"2024-08-17T17:37:40.031368+0000","last_clean":"2024-08-17T17:37:40.031368+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:37:40.031368+0000","last_undegraded":"2024-08-17T17:37:40.031368+0000","last_fullsized":"2024-08-17T17:37:40.031368+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":24,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:40.029929+0000","last_change":"2024-08-17T17:37:00.382745+0000","last_active":"2024-08-17T17:37:40.029929+0000","last_peered":"2024-08-17T17:37:40.029929+0000","last_clean":"2024-08-17T17:37:40.029929+0000","last_became_active":"2024-08-17T17:37:00.382032+0000","last_became_peered":"2024-08-17T17:37:00.382032+0000","last_unstale":"2024-08-17T17:37:40.029929+0000","last_undegraded":"2024-08-17T17:37:40.029929+0000","last_fullsized":"2024-08-17T17:37:40.029929+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:24:51.920975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":77,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:40.030442+0000","last_change":"2024-08-17T17:33:21.763655+0000","last_active":"2024-08-17T17:37:40.030442+0000","last_peered":"2024-08-17T17:37:40.030442+0000","last_clean":"2024-08-17T17:37:40.030442+0000","last_became_active":"2024-08-17T17:33:21.763088+0000","last_became_peered":"2024-08-17T17:33:21.763088+0000","last_unstale":"2024-08-17T17:37:40.030442+0000","last_undegraded":"2024-08-17T17:37:40.030442+0000","last_fullsized":"2024-08-17T17:37:40.030442+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:40:40.126898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":34,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:40.030759+0000","last_change":"2024-08-17T17:37:00.413709+0000","last_active":"2024-08-17T17:37:40.030759+0000","last_peered":"2024-08-17T17:37:40.030759+0000","last_clean":"2024-08-17T17:37:40.030759+0000","last_became_active":"2024-08-17T17:37:00.383355+0000","last_became_peered":"2024-08-17T17:37:00.383355+0000","last_unstale":"2024-08-17T17:37:40.030759+0000","last_undegraded":"2024-08-17T17:37:40.030759+0000","last_fullsized":"2024-08-17T17:37:40.030759+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:47:24.501358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":451,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:47.840531+0000","last_change":"2024-08-17T17:33:20.799402+0000","last_active":"2024-08-17T17:37:47.840531+0000","last_peered":"2024-08-17T17:37:47.840531+0000","last_clean":"2024-08-17T17:37:47.840531+0000","last_became_active":"2024-08-17T17:33:20.798832+0000","last_became_peered":"2024-08-17T17:33:20.798832+0000","last_unstale":"2024-08-17T17:37:47.840531+0000","last_undegraded":"2024-08-17T17:37:47.840531+0000","last_fullsized":"2024-08-17T17:37:47.840531+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:21:05.001134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":199,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.860049+0000","last_change":"2024-08-17T17:32:46.381141+0000","last_active":"2024-08-17T17:37:48.860049+0000","last_peered":"2024-08-17T17:37:48.860049+0000","last_clean":"2024-08-17T17:37:48.860049+0000","last_became_active":"2024-08-17T17:32:46.374259+0000","last_became_peered":"2024-08-17T17:32:46.374259+0000","last_unstale":"2024-08-17T17:37:48.860049+0000","last_undegraded":"2024-08-17T17:37:48.860049+0000","last_fullsized":"2024-08-17T17:37:48.860049+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:19:10.836889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":189,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:47.840737+0000","last_change":"2024-08-17T17:33:44.146014+0000","last_active":"2024-08-17T17:37:47.840737+0000","last_peered":"2024-08-17T17:37:47.840737+0000","last_clean":"2024-08-17T17:37:47.840737+0000","last_became_active":"2024-08-17T17:33:44.145668+0000","last_became_peered":"2024-08-17T17:33:44.145668+0000","last_unstale":"2024-08-17T17:37:47.840737+0000","last_undegraded":"2024-08-17T17:37:47.840737+0000","last_fullsized":"2024-08-17T17:37:47.840737+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:53:41.679237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":453,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.859694+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:37:48.859694+0000","last_peered":"2024-08-17T17:37:48.859694+0000","last_clean":"2024-08-17T17:37:48.859694+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:37:48.859694+0000","last_undegraded":"2024-08-17T17:37:48.859694+0000","last_fullsized":"2024-08-17T17:37:48.859694+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":481,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.860277+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:37:48.860277+0000","last_peered":"2024-08-17T17:37:48.860277+0000","last_clean":"2024-08-17T17:37:48.860277+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:37:48.860277+0000","last_undegraded":"2024-08-17T17:37:48.860277+0000","last_fullsized":"2024-08-17T17:37:48.860277+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":511,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.860579+0000","last_change":"2024-08-17T17:33:20.790687+0000","last_active":"2024-08-17T17:37:48.860579+0000","last_peered":"2024-08-17T17:37:48.860579+0000","last_clean":"2024-08-17T17:37:48.860579+0000","last_became_active":"2024-08-17T17:33:20.790399+0000","last_became_peered":"2024-08-17T17:33:20.790399+0000","last_unstale":"2024-08-17T17:37:48.860579+0000","last_undegraded":"2024-08-17T17:37:48.860579+0000","last_fullsized":"2024-08-17T17:37:48.860579+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:38:06.849989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":288,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:40.031258+0000","last_change":"2024-08-17T17:33:20.835760+0000","last_active":"2024-08-17T17:37:40.031258+0000","last_peered":"2024-08-17T17:37:40.031258+0000","last_clean":"2024-08-17T17:37:40.031258+0000","last_became_active":"2024-08-17T17:33:20.835410+0000","last_became_peered":"2024-08-17T17:33:20.835410+0000","last_unstale":"2024-08-17T17:37:40.031258+0000","last_undegraded":"2024-08-17T17:37:40.031258+0000","last_fullsized":"2024-08-17T17:37:40.031258+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:45.448022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":529,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.860681+0000","last_change":"2024-08-17T17:33:20.771279+0000","last_active":"2024-08-17T17:37:48.860681+0000","last_peered":"2024-08-17T17:37:48.860681+0000","last_clean":"2024-08-17T17:37:48.860681+0000","last_became_active":"2024-08-17T17:33:20.770728+0000","last_became_peered":"2024-08-17T17:33:20.770728+0000","last_unstale":"2024-08-17T17:37:48.860681+0000","last_undegraded":"2024-08-17T17:37:48.860681+0000","last_fullsized":"2024-08-17T17:37:48.860681+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:52:31.540397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":298,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.861354+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:37:48.861354+0000","last_peered":"2024-08-17T17:37:48.861354+0000","last_clean":"2024-08-17T17:37:48.861354+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:37:48.861354+0000","last_undegraded":"2024-08-17T17:37:48.861354+0000","last_fullsized":"2024-08-17T17:37:48.861354+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":435,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:47.839837+0000","last_change":"2024-08-17T17:37:00.403306+0000","last_active":"2024-08-17T17:37:47.839837+0000","last_peered":"2024-08-17T17:37:47.839837+0000","last_clean":"2024-08-17T17:37:47.839837+0000","last_became_active":"2024-08-17T17:37:00.403030+0000","last_became_peered":"2024-08-17T17:37:00.403030+0000","last_unstale":"2024-08-17T17:37:47.839837+0000","last_undegraded":"2024-08-17T17:37:47.839837+0000","last_fullsized":"2024-08-17T17:37:47.839837+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:19:07.078877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":95,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:40.031319+0000","last_change":"2024-08-17T17:33:44.160733+0000","last_active":"2024-08-17T17:37:40.031319+0000","last_peered":"2024-08-17T17:37:40.031319+0000","last_clean":"2024-08-17T17:37:40.031319+0000","last_became_active":"2024-08-17T17:33:44.160368+0000","last_became_peered":"2024-08-17T17:33:44.160368+0000","last_unstale":"2024-08-17T17:37:40.031319+0000","last_undegraded":"2024-08-17T17:37:40.031319+0000","last_fullsized":"2024-08-17T17:37:40.031319+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:37:22.855642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":24,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:40.030043+0000","last_change":"2024-08-17T17:37:00.405506+0000","last_active":"2024-08-17T17:37:40.030043+0000","last_peered":"2024-08-17T17:37:40.030043+0000","last_clean":"2024-08-17T17:37:40.030043+0000","last_became_active":"2024-08-17T17:37:00.400982+0000","last_became_peered":"2024-08-17T17:37:00.400982+0000","last_unstale":"2024-08-17T17:37:40.030043+0000","last_undegraded":"2024-08-17T17:37:40.030043+0000","last_fullsized":"2024-08-17T17:37:40.030043+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:47:17.134917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":416,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:47.840388+0000","last_change":"2024-08-17T17:33:20.773207+0000","last_active":"2024-08-17T17:37:47.840388+0000","last_peered":"2024-08-17T17:37:47.840388+0000","last_clean":"2024-08-17T17:37:47.840388+0000","last_became_active":"2024-08-17T17:33:20.772457+0000","last_became_peered":"2024-08-17T17:33:20.772457+0000","last_unstale":"2024-08-17T17:37:47.840388+0000","last_undegraded":"2024-08-17T17:37:47.840388+0000","last_fullsized":"2024-08-17T17:37:47.840388+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:57:09.610084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":308,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.861043+0000","last_change":"2024-08-17T17:32:46.375464+0000","last_active":"2024-08-17T17:37:48.861043+0000","last_peered":"2024-08-17T17:37:48.861043+0000","last_clean":"2024-08-17T17:37:48.861043+0000","last_became_active":"2024-08-17T17:32:46.375161+0000","last_became_peered":"2024-08-17T17:32:46.375161+0000","last_unstale":"2024-08-17T17:37:48.861043+0000","last_undegraded":"2024-08-17T17:37:48.861043+0000","last_fullsized":"2024-08-17T17:37:48.861043+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:27:47.091836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:40.030011+0000","last_change":"2024-08-17T17:37:00.447160+0000","last_active":"2024-08-17T17:37:40.030011+0000","last_peered":"2024-08-17T17:37:40.030011+0000","last_clean":"2024-08-17T17:37:40.030011+0000","last_became_active":"2024-08-17T17:37:00.446710+0000","last_became_peered":"2024-08-17T17:37:00.446710+0000","last_unstale":"2024-08-17T17:37:40.030011+0000","last_undegraded":"2024-08-17T17:37:40.030011+0000","last_fullsized":"2024-08-17T17:37:40.030011+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:51:43.200559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":328,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:40.030924+0000","last_change":"2024-08-17T17:33:44.165175+0000","last_active":"2024-08-17T17:37:40.030924+0000","last_peered":"2024-08-17T17:37:40.030924+0000","last_clean":"2024-08-17T17:37:40.030924+0000","last_became_active":"2024-08-17T17:33:44.146198+0000","last_became_peered":"2024-08-17T17:33:44.146198+0000","last_unstale":"2024-08-17T17:37:40.030924+0000","last_undegraded":"2024-08-17T17:37:40.030924+0000","last_fullsized":"2024-08-17T17:37:40.030924+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:14:56.291775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":24,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:40.030066+0000","last_change":"2024-08-17T17:37:00.405789+0000","last_active":"2024-08-17T17:37:40.030066+0000","last_peered":"2024-08-17T17:37:40.030066+0000","last_clean":"2024-08-17T17:37:40.030066+0000","last_became_active":"2024-08-17T17:37:00.405044+0000","last_became_peered":"2024-08-17T17:37:00.405044+0000","last_unstale":"2024-08-17T17:37:40.030066+0000","last_undegraded":"2024-08-17T17:37:40.030066+0000","last_fullsized":"2024-08-17T17:37:40.030066+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:15:57.228668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":308,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:40.031229+0000","last_change":"2024-08-17T17:33:44.148090+0000","last_active":"2024-08-17T17:37:40.031229+0000","last_peered":"2024-08-17T17:37:40.031229+0000","last_clean":"2024-08-17T17:37:40.031229+0000","last_became_active":"2024-08-17T17:33:44.147539+0000","last_became_peered":"2024-08-17T17:33:44.147539+0000","last_unstale":"2024-08-17T17:37:40.031229+0000","last_undegraded":"2024-08-17T17:37:40.031229+0000","last_fullsized":"2024-08-17T17:37:40.031229+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:52:02.311332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":507,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.860026+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:37:48.860026+0000","last_peered":"2024-08-17T17:37:48.860026+0000","last_clean":"2024-08-17T17:37:48.860026+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:37:48.860026+0000","last_undegraded":"2024-08-17T17:37:48.860026+0000","last_fullsized":"2024-08-17T17:37:48.860026+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":508,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.860097+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:37:48.860097+0000","last_peered":"2024-08-17T17:37:48.860097+0000","last_clean":"2024-08-17T17:37:48.860097+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:37:48.860097+0000","last_undegraded":"2024-08-17T17:37:48.860097+0000","last_fullsized":"2024-08-17T17:37:48.860097+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":63,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-08-17T17:37:40.029788+0000","last_change":"2024-08-17T17:32:46.376844+0000","last_active":"2024-08-17T17:37:40.029788+0000","last_peered":"2024-08-17T17:37:40.029788+0000","last_clean":"2024-08-17T17:37:40.029788+0000","last_became_active":"2024-08-17T17:32:46.376483+0000","last_became_peered":"2024-08-17T17:32:46.376483+0000","last_unstale":"2024-08-17T17:37:40.029788+0000","last_undegraded":"2024-08-17T17:37:40.029788+0000","last_fullsized":"2024-08-17T17:37:40.029788+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:44:24.358662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":317,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-08-17T17:37:48.861482+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:37:48.861482+0000","last_peered":"2024-08-17T17:37:48.861482+0000","last_clean":"2024-08-17T17:37:48.861482+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:37:48.861482+0000","last_undegraded":"2024-08-17T17:37:48.861482+0000","last_fullsized":"2024-08-17T17:37:48.861482+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":1569325056,"num_objects":23946,"num_object_clones":0,"num_object_copies":71838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23946,"num_write_kb":1532544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4571529216,"data_stored":4571529216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23946,"ondisk_log_size":23946,"up":48,"acting":48,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":6220,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19280,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5978,"num_read_kb":12991,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":697707,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":149,"num_write":38,"num_write_kb":439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":912386,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1639,"ondisk_log_size":1639,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234374,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1690620,"kb_used_data":788404,"kb_used_omap":1,"kb_used_meta":902206,"kb_avail":92054532,"statfs":{"total":95995035648,"available":94263840768,"internally_reserved":0,"allocated":807325696,"data_stored":805514943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":923859288},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496009,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1165424,"kb_used_data":278632,"kb_used_omap":1,"kb_used_meta":886782,"kb_avail":92579728,"statfs":{"total":95995035648,"available":94801641472,"internally_reserved":0,"allocated":285319168,"data_stored":283520377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":908065112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757646,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1710384,"kb_used_data":815592,"kb_used_omap":1,"kb_used_meta":894782,"kb_avail":92034768,"statfs":{"total":95995035648,"available":94243602432,"internally_reserved":0,"allocated":835166208,"data_stored":833363986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":916257112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019284,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":581180,"kb_used_data":275508,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93163972,"statfs":{"total":95995035648,"available":95399907328,"internally_reserved":0,"allocated":282120192,"data_stored":280309700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435511,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1369260,"kb_used_data":678308,"kb_used_omap":4,"kb_used_meta":690939,"kb_avail":92375892,"statfs":{"total":95995035648,"available":94592913408,"internally_reserved":0,"allocated":694587392,"data_stored":692789048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":707521546},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599037,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":793840,"kb_used_data":170856,"kb_used_omap":7,"kb_used_meta":622968,"kb_avail":92951312,"statfs":{"total":95995035648,"available":95182143488,"internally_reserved":0,"allocated":174956544,"data_stored":173154618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":637919394},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":268,"seq":1151051235385,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1094632,"kb_used_data":878048,"kb_used_omap":10,"kb_used_meta":216565,"kb_avail":92650520,"statfs":{"total":95995035648,"available":94874132480,"internally_reserved":0,"allocated":899121152,"data_stored":897326305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10902,"internal_metadata":221762922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,3,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926091,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":947500,"kb_used_data":351332,"kb_used_omap":7,"kb_used_meta":596152,"kb_avail":92797652,"statfs":{"total":95995035648,"available":95024795648,"internally_reserved":0,"allocated":359763968,"data_stored":357965590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":610459969},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7550,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":384827392,"data_stored":384827392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":926351360,"data_stored":926351360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":184745984,"data_stored":184745984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":756744192,"data_stored":756744192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":283181056,"data_stored":283181056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":872087552,"data_stored":872087552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":285802496,"data_stored":285802496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":877789184,"data_stored":877789184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:38:02.283 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-08-17T17:38:02.283 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd pool set .mgr pg_num 21 2024-08-17T17:38:02.307 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:02.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:02.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:02.321 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 12 16 30130 30114 156.816 125.312 0.0182079 0.00636986 2024-08-17T17:38:02.409 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:02.454 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:02.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:02.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:02.556 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:02.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:02.621 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:02.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:02.722 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:02.745 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:02.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:02.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:02.846 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:02.885 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:02.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:02.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:02.987 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:03.023 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:03.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:03.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:03.125 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:03.153 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:03.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:03.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:03.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:02 smithi053 ceph-mon[89963]: from='client.55744 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:38:03.254 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:03.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:02 smithi003 ceph-mon[103344]: from='client.55744 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:38:03.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:02 smithi003 ceph-mon[100615]: from='client.55744 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:38:03.292 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:03.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:03.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:03.321 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 13 16 31732 31716 152.455 100.125 0.00788698 0.00655314 2024-08-17T17:38:03.395 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:03.430 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:03.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:03.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:03.532 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:03.554 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:03.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:03.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:03.656 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:03.692 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:03.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:03.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:03.794 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:03.837 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:03.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:03.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:03.939 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:03.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:03.977 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:03.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:04.078 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:04.110 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:04.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:04.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:04.212 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:04.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:03 smithi053 ceph-mon[89963]: pgmap v989: 99 pgs: 99 active+clean; 1.5 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 125 MiB/s wr, 1.99k op/s 2024-08-17T17:38:04.234 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:04.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:04.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:04.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:03 smithi003 ceph-mon[103344]: pgmap v989: 99 pgs: 99 active+clean; 1.5 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 125 MiB/s wr, 1.99k op/s 2024-08-17T17:38:04.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:03 smithi003 ceph-mon[100615]: pgmap v989: 99 pgs: 99 active+clean; 1.5 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 125 MiB/s wr, 1.99k op/s 2024-08-17T17:38:04.321 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 14 16 33119 33103 147.757 86.6875 0.00166514 0.00669903 2024-08-17T17:38:04.337 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:04.361 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:04.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:04.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:04.463 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:04.503 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:04.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:04.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:04.606 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:04.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:04.648 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:04.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:04.749 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:04.772 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:04.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:04.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:04.874 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:04.916 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:04.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:04.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:05.018 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:05.040 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:05.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:05.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:05.142 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:05.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:05.179 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:05.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:05.280 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:05.320 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:05.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:05.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:05.321 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 15 16 34623 34607 144.173 94 0.0104342 0.00693018 2024-08-17T17:38:05.422 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:05.457 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:05.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:05.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:05.559 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:05.592 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:05.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:05.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:05.694 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:05.730 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:05.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:05.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:05.832 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:05.873 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:05.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:05.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:05.974 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:06.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:06.011 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:06.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:06.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:05 smithi003 ceph-mon[103344]: pgmap v990: 99 pgs: 99 active+clean; 1.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 152 MiB/s wr, 2.43k op/s 2024-08-17T17:38:06.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:05 smithi003 ceph-mon[100615]: pgmap v990: 99 pgs: 99 active+clean; 1.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 152 MiB/s wr, 2.43k op/s 2024-08-17T17:38:06.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:38:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:38:05] "GET /metrics HTTP/1.1" 200 36481 "" "Prometheus/2.51.0" 2024-08-17T17:38:06.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:06.151 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:06.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:06.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:06.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:05 smithi053 ceph-mon[89963]: pgmap v990: 99 pgs: 99 active+clean; 1.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 152 MiB/s wr, 2.43k op/s 2024-08-17T17:38:06.253 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:06.287 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:06.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:06.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:06.322 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 16 16 36166 36150 141.189 96.4375 0.0137638 0.00707703 2024-08-17T17:38:06.389 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:06.429 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:06.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:06.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:06.531 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:06.570 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:06.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:06.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:06.671 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:06.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:06.700 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:06.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:06.801 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:06.838 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:06.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:06.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:06.940 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:06.965 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:06.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:06.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:07.067 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:07.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:07.101 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:07.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:07.202 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:07.237 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:07.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:07.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:07.322 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 17 16 37944 37928 139.42 111.125 0.00322289 0.00716315 2024-08-17T17:38:07.339 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:07.361 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:07.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:07.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:07.463 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:07.495 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:07.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:07.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:07.597 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:07.635 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:07.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:07.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:07.737 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:07.775 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:07.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:07.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:07.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:07.919 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:07.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:07.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:08.021 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:08.060 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:08.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:08.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:08.162 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:08.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:08.189 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:08.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:08.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:07 smithi053 ceph-mon[89963]: pgmap v991: 99 pgs: 99 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 140 MiB/s wr, 2.24k op/s 2024-08-17T17:38:08.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:07 smithi003 ceph-mon[103344]: pgmap v991: 99 pgs: 99 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 140 MiB/s wr, 2.24k op/s 2024-08-17T17:38:08.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:07 smithi003 ceph-mon[100615]: pgmap v991: 99 pgs: 99 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 140 MiB/s wr, 2.24k op/s 2024-08-17T17:38:08.290 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:08.322 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 18 16 39366 39350 136.611 88.875 0.00712376 0.00731457 2024-08-17T17:38:08.329 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:08.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:08.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:08.431 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:08.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:08.454 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:08.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:08.555 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:08.595 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:08.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:08.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:08.697 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:08.738 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:08.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:08.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:08.840 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:08.876 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:08.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:08.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:08.978 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:09.012 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:09.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:09.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:09.113 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:09.143 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:09.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:09.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:09.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:38:09.245 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:09.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:38:09.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:38:09.279 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:09.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:09.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:09.322 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 19 16 41259 41243 135.648 118.312 0.00460386 0.0073665 2024-08-17T17:38:09.382 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:09.423 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:09.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:09.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:09.525 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:09.556 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:09.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:09.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:09.658 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:09.683 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:09.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:09.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:09.785 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:09.817 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:09.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:09.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:09.919 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:09.956 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:09.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:09.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:10.055 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:38:10.059 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:10.102 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:38:10.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:10.107 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:10.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:10.208 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:10.209 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:09 smithi003 ceph-mon[103344]: pgmap v992: 99 pgs: 99 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-08-17T17:38:10.212 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:09 smithi003 ceph-mon[100615]: pgmap v992: 99 pgs: 99 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-08-17T17:38:10.213 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:09 smithi053 ceph-mon[89963]: pgmap v992: 99 pgs: 99 active+clean; 2.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-08-17T17:38:10.242 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:10.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:10.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:10.279 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:38:10.322 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-08-17T17:38:10.322231+0000 min lat: 0.001429 max lat: 0.17992 avg lat: 0.00746243 2024-08-17T17:38:10.322 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:38:10.322 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 20 16 42864 42848 133.881 100.312 0.00176641 0.00746243 2024-08-17T17:38:10.343 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:10.369 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:10.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:10.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:10.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:38:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:38:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:38:10.471 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:10.512 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:10.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:10.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:10.614 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:10.651 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:10.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:10.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:10.753 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:10.784 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:10.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:10.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:10.886 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:10.909 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:10.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:10.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:11.011 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:11.042 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:11.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:11.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:11.144 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:11.185 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:11.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:11.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:11.287 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:11.313 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:11.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:11.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:11.322 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 21 16 44212 44196 131.517 84.25 0.00493379 0.00758984 2024-08-17T17:38:11.414 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:11.452 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:11.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:11.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:11.554 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:11.576 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:11.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:11.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:11.678 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:11.716 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:11.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:11.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:11.818 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:11.856 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:11.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:11.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:11.958 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:11.998 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:12.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:12.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:12.103 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:12.137 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:12.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:12.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:12.199 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:11 smithi003 ceph-mon[103344]: pgmap v993: 99 pgs: 99 active+clean; 2.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 151 MiB/s wr, 2.42k op/s 2024-08-17T17:38:12.199 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:11 smithi003 ceph-mon[100615]: pgmap v993: 99 pgs: 99 active+clean; 2.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 151 MiB/s wr, 2.42k op/s 2024-08-17T17:38:12.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:11 smithi053 ceph-mon[89963]: pgmap v993: 99 pgs: 99 active+clean; 2.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 151 MiB/s wr, 2.42k op/s 2024-08-17T17:38:12.239 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:12.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:12.279 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:12.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:12.323 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 22 16 45829 45813 130.129 101.062 0.0083192 0.00767875 2024-08-17T17:38:12.380 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:12.415 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:12.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:12.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:12.517 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:12.555 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:12.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:12.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:12.657 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:12.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:12.693 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:12.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:12.794 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:12.833 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:12.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:12.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:12.935 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:12.971 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:12.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:12.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:13.073 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:13.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:13.125 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:13.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:13.227 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:13.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:13.269 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:13.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:13.322 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 23 16 47466 47450 128.92 102.312 0.00671844 0.00775184 2024-08-17T17:38:13.370 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:13.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:13.394 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:13.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:13.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:13.532 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:13.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:13.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:13.633 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:13.667 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:13.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:13.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:13.769 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:13.805 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:13.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:13.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:13.907 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:13.944 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:13.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:13.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:14.046 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:14.077 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:14.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:14.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:14.179 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:14.216 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:14.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:13 smithi053 ceph-mon[89963]: pgmap v994: 99 pgs: 99 active+clean; 2.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-08-17T17:38:14.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:14.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:14.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:13 smithi003 ceph-mon[100615]: pgmap v994: 99 pgs: 99 active+clean; 2.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-08-17T17:38:14.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:13 smithi003 ceph-mon[103344]: pgmap v994: 99 pgs: 99 active+clean; 2.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-08-17T17:38:14.318 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:14.322 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 24 16 49162 49146 127.964 106 0.00764512 0.00780979 2024-08-17T17:38:14.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:14.353 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:14.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:14.454 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:14.488 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:14.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:14.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:14.590 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:14.627 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:14.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:14.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:14.729 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:14.759 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:14.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:14.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:14.861 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:14.914 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:14.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:14.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:15.016 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:15.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:15.056 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:15.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:15.157 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:15.202 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:15.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:15.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:15.304 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:15.323 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 25 16 50806 50790 126.956 102.75 0.0117234 0.00787171 2024-08-17T17:38:15.338 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:15.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:15.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:15.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:15.482 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:15.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:15.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:15.584 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:15.616 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:15.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:15.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:15.718 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:15.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:15.770 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:15.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:15.826 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:38:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:38:15] "GET /metrics HTTP/1.1" 200 36511 "" "Prometheus/2.51.0" 2024-08-17T17:38:15.871 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:15.920 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:15.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:15.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:16.022 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:16.058 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:16.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:16.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:16.160 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:16.198 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:16.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:16.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:16.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:15 smithi053 ceph-mon[89963]: pgmap v995: 99 pgs: 99 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-08-17T17:38:16.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:15 smithi003 ceph-mon[103344]: pgmap v995: 99 pgs: 99 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-08-17T17:38:16.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:15 smithi003 ceph-mon[100615]: pgmap v995: 99 pgs: 99 active+clean; 2.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-08-17T17:38:16.300 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:16.323 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 26 16 52435 52419 125.986 101.812 0.0191986 0.00793204 2024-08-17T17:38:16.338 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:16.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:16.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:16.440 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:16.472 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:16.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:16.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:16.574 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:16.604 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:16.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:16.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:16.706 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:16.739 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:16.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:16.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:16.840 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:16.863 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:16.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:16.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:16.964 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:16.997 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:16.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:16.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:17.099 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:17.127 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:17.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:17.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:17.229 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:17.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:17.267 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:17.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:17.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:16 smithi003 ceph-mon[103344]: pgmap v996: 99 pgs: 99 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-08-17T17:38:17.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:16 smithi003 ceph-mon[100615]: pgmap v996: 99 pgs: 99 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-08-17T17:38:17.323 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 27 16 53828 53812 124.545 87.0625 0.00959267 0.0080239 2024-08-17T17:38:17.368 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:17.406 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:17.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:17.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:17.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:16 smithi053 ceph-mon[89963]: pgmap v996: 99 pgs: 99 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s 2024-08-17T17:38:17.507 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:17.544 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:17.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:17.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:17.646 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:17.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:17.671 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:17.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:17.772 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:17.804 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:17.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:17.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:17.906 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:17.931 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:17.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:17.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:18.033 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:18.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:18.078 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:18.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:18.179 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:18.211 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:18.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:18.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:18.238 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-08-17T17:38:18.314 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:18.326 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 28 15 54983 54968 122.677 72.25 0.0115404 0.00814656 2024-08-17T17:38:18.348 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:18.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:18.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:18.450 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:18.477 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:18.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:18.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:18.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:18 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3776641949' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:38:18.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:18 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:38:18.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:18 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3776641949' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:38:18.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:18 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:38:18.579 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:18.609 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:18.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:18.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:18.711 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:18.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:18 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3776641949' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:38:18.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:18 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:38:18.760 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:18.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:18.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:18.862 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:18.885 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:18.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:18.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:18.986 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:19.034 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:19.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:19.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:19.135 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:19.160 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:19.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:19.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:19.262 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:19.286 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:19.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:19.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:19.323 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 29 16 56139 56123 120.936 72.1875 0.0139611 0.00826385 2024-08-17T17:38:19.389 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:19.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:19.425 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:19.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:19.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:19 smithi003 ceph-mon[100615]: pgmap v997: 99 pgs: 99 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-08-17T17:38:19.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:19 smithi003 ceph-mon[100615]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:38:19.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:19 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:38:19.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:19 smithi003 ceph-mon[100615]: osdmap e315: 8 total, 8 up, 8 in 2024-08-17T17:38:19.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:19 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2510389473' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-08-17T17:38:19.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:19 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-08-17T17:38:19.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:19 smithi003 ceph-mon[103344]: pgmap v997: 99 pgs: 99 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-08-17T17:38:19.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:19 smithi003 ceph-mon[103344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:38:19.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:19 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:38:19.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:19 smithi003 ceph-mon[103344]: osdmap e315: 8 total, 8 up, 8 in 2024-08-17T17:38:19.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:19 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2510389473' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-08-17T17:38:19.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:19 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-08-17T17:38:19.526 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:19.557 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:19.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:19.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:19.626 INFO:teuthology.orchestra.run.smithi003.stderr:set pool 1 pg_num to 21 2024-08-17T17:38:19.659 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:19.693 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:19.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:19.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:19.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:19 smithi053 ceph-mon[89963]: pgmap v997: 99 pgs: 99 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-08-17T17:38:19.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:19 smithi053 ceph-mon[89963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:38:19.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:19 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:38:19.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:19 smithi053 ceph-mon[89963]: osdmap e315: 8 total, 8 up, 8 in 2024-08-17T17:38:19.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:19 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2510389473' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-08-17T17:38:19.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:19 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-08-17T17:38:19.794 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:19.827 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:19.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:19.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:19.929 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:19.968 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:19.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:19.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:20.070 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:20.109 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:20.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:20.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:20.212 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:20.254 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:20.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:20.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:20.324 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 30 16 57784 57768 120.331 102.812 0.0110865 0.00830503 2024-08-17T17:38:20.356 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:20.398 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:20.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:20.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:20.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:20 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-08-17T17:38:20.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:20 smithi053 ceph-mon[89963]: osdmap e316: 8 total, 8 up, 8 in 2024-08-17T17:38:20.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:20 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-08-17T17:38:20.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:38:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:38:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:38:20.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:20.538 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:20.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:20.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:20.602 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:20 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-08-17T17:38:20.603 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:20 smithi003 ceph-mon[103344]: osdmap e316: 8 total, 8 up, 8 in 2024-08-17T17:38:20.603 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:20 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-08-17T17:38:20.603 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:20 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-08-17T17:38:20.603 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:20 smithi003 ceph-mon[100615]: osdmap e316: 8 total, 8 up, 8 in 2024-08-17T17:38:20.603 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:20 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-08-17T17:38:20.641 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:20.675 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:20.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:20.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:20.778 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:20.801 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:20.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:20.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:20.903 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:20.942 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:20.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:20.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:21.043 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:21.082 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:21.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:21.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:21.184 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:21.218 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:21.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:21.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:21.320 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:21.326 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 31 16 59256 59240 119.409 92 0.0204083 0.00836803 2024-08-17T17:38:21.354 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:21.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:21.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:21.456 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:21.497 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:21.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:21.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:21.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:21 smithi003 ceph-mon[103344]: 1.7 scrub starts 2024-08-17T17:38:21.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:21 smithi003 ceph-mon[103344]: 1.7 scrub ok 2024-08-17T17:38:21.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:21 smithi003 ceph-mon[103344]: pgmap v1000: 99 pgs: 99 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-08-17T17:38:21.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:21 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-08-17T17:38:21.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:21 smithi003 ceph-mon[103344]: osdmap e317: 8 total, 8 up, 8 in 2024-08-17T17:38:21.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:21 smithi003 ceph-mon[100615]: 1.7 scrub starts 2024-08-17T17:38:21.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:21 smithi003 ceph-mon[100615]: 1.7 scrub ok 2024-08-17T17:38:21.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:21 smithi003 ceph-mon[100615]: pgmap v1000: 99 pgs: 99 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-08-17T17:38:21.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:21 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-08-17T17:38:21.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:21 smithi003 ceph-mon[100615]: osdmap e317: 8 total, 8 up, 8 in 2024-08-17T17:38:21.598 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:21.630 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:21.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:21.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:21.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:21 smithi053 ceph-mon[89963]: 1.7 scrub starts 2024-08-17T17:38:21.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:21 smithi053 ceph-mon[89963]: 1.7 scrub ok 2024-08-17T17:38:21.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:21 smithi053 ceph-mon[89963]: pgmap v1000: 99 pgs: 99 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-08-17T17:38:21.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:21 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-08-17T17:38:21.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:21 smithi053 ceph-mon[89963]: osdmap e317: 8 total, 8 up, 8 in 2024-08-17T17:38:21.732 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:21.787 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:21.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:21.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:21.888 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:21.935 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:21.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:21.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:21.991 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:38:21.992 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:38:22.034 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:38:22.034 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:38:22.037 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:22.068 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:38:22.068 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:38:22.081 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:22.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:22.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:22.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:22.188 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:38:22.189 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:38:22.223 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:38:22.223 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:38:22.227 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:22.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:22.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:22.254 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:38:22.255 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:38:22.285 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:38:22.285 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:38:22.319 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:38:22.319 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:38:22.326 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 32 16 60516 60500 118.139 78.75 0.00472941 0.00845938 2024-08-17T17:38:22.329 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:22.364 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:22.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:22.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:22.466 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:22.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:22.505 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:22.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:22.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:22 smithi003 ceph-mon[103344]: osdmap e318: 8 total, 8 up, 8 in 2024-08-17T17:38:22.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:22 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:38:22.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:22 smithi003 ceph-mon[103344]: 1.3 scrub starts 2024-08-17T17:38:22.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:22 smithi003 ceph-mon[103344]: 1.3 scrub ok 2024-08-17T17:38:22.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:22 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/219657985' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:38:22.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:22 smithi003 ceph-mon[100615]: osdmap e318: 8 total, 8 up, 8 in 2024-08-17T17:38:22.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:22 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:38:22.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:22 smithi003 ceph-mon[100615]: 1.3 scrub starts 2024-08-17T17:38:22.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:22 smithi003 ceph-mon[100615]: 1.3 scrub ok 2024-08-17T17:38:22.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:22 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/219657985' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:38:22.606 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:22.629 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:22.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:22.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:22.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:22 smithi053 ceph-mon[89963]: osdmap e318: 8 total, 8 up, 8 in 2024-08-17T17:38:22.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:22 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:38:22.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:22 smithi053 ceph-mon[89963]: 1.3 scrub starts 2024-08-17T17:38:22.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:22 smithi053 ceph-mon[89963]: 1.3 scrub ok 2024-08-17T17:38:22.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:22 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/219657985' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:38:22.730 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:22.770 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:22.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:22.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:22.872 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:22.915 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:22.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:22.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:23.017 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:23.051 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:23.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:23.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:23.153 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:23.178 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:23.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:23.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:23.280 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:23.310 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:23.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:23.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:23.326 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 33 16 61642 61626 116.691 70.375 0.0132267 0.00856427 2024-08-17T17:38:23.412 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:23.442 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:23.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:23.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:23.544 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:23.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:23.582 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:23.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:23.683 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:23.686 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:38:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:23 smithi053 ceph-mon[89963]: 1.14 scrub starts 2024-08-17T17:38:23.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:23 smithi053 ceph-mon[89963]: 1.14 scrub ok 2024-08-17T17:38:23.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:23 smithi053 ceph-mon[89963]: pgmap v1003: 109 pgs: 10 unknown, 99 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 112 MiB/s wr, 1.78k op/s 2024-08-17T17:38:23.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:23 smithi053 ceph-mon[89963]: 1.1 scrub starts 2024-08-17T17:38:23.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:23 smithi053 ceph-mon[89963]: 1.1 scrub ok 2024-08-17T17:38:23.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:23 smithi053 ceph-mon[89963]: 1.b scrub starts 2024-08-17T17:38:23.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:23 smithi053 ceph-mon[89963]: 1.b scrub ok 2024-08-17T17:38:23.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:38:23.718 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:23.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:23.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:23.722 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:23 smithi003 ceph-mon[103344]: 1.14 scrub starts 2024-08-17T17:38:23.723 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:23 smithi003 ceph-mon[103344]: 1.14 scrub ok 2024-08-17T17:38:23.723 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:23 smithi003 ceph-mon[103344]: pgmap v1003: 109 pgs: 10 unknown, 99 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 112 MiB/s wr, 1.78k op/s 2024-08-17T17:38:23.723 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:23 smithi003 ceph-mon[103344]: 1.1 scrub starts 2024-08-17T17:38:23.723 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:23 smithi003 ceph-mon[103344]: 1.1 scrub ok 2024-08-17T17:38:23.723 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:23 smithi003 ceph-mon[103344]: 1.b scrub starts 2024-08-17T17:38:23.723 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:23 smithi003 ceph-mon[103344]: 1.b scrub ok 2024-08-17T17:38:23.723 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:38:23.723 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:23 smithi003 ceph-mon[100615]: 1.14 scrub starts 2024-08-17T17:38:23.723 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:23 smithi003 ceph-mon[100615]: 1.14 scrub ok 2024-08-17T17:38:23.723 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:23 smithi003 ceph-mon[100615]: pgmap v1003: 109 pgs: 10 unknown, 99 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 112 MiB/s wr, 1.78k op/s 2024-08-17T17:38:23.723 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:23 smithi003 ceph-mon[100615]: 1.1 scrub starts 2024-08-17T17:38:23.723 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:23 smithi003 ceph-mon[100615]: 1.1 scrub ok 2024-08-17T17:38:23.724 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:23 smithi003 ceph-mon[100615]: 1.b scrub starts 2024-08-17T17:38:23.724 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:23 smithi003 ceph-mon[100615]: 1.b scrub ok 2024-08-17T17:38:23.724 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:38:23.820 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:23.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:23.870 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:23.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:23.971 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:24.018 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:24.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:24.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:24.120 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:24.156 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:24.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:24.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:24.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:24.296 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:24.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:24.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:24.326 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 34 16 62981 62965 115.721 83.6875 0.0149915 0.00863628 2024-08-17T17:38:24.397 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:24.424 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:24.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:24.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:24.526 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:24.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:24.563 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:24.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:24.649 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:24 smithi053 ceph-mon[89963]: 1.4 scrub starts 2024-08-17T17:38:24.650 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:24 smithi053 ceph-mon[89963]: 1.4 scrub ok 2024-08-17T17:38:24.650 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:24 smithi053 ceph-mon[89963]: 1.2 scrub starts 2024-08-17T17:38:24.650 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:24 smithi053 ceph-mon[89963]: 1.2 scrub ok 2024-08-17T17:38:24.650 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:24 smithi053 ceph-mon[89963]: 1.5 scrub starts 2024-08-17T17:38:24.650 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:24 smithi053 ceph-mon[89963]: 1.5 scrub ok 2024-08-17T17:38:24.650 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:24 smithi053 ceph-mon[89963]: 1.13 scrub starts 2024-08-17T17:38:24.650 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:24 smithi053 ceph-mon[89963]: 1.13 scrub ok 2024-08-17T17:38:24.664 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:24.702 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:24.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:24.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:24.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:24 smithi003 ceph-mon[103344]: 1.4 scrub starts 2024-08-17T17:38:24.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:24 smithi003 ceph-mon[103344]: 1.4 scrub ok 2024-08-17T17:38:24.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:24 smithi003 ceph-mon[103344]: 1.2 scrub starts 2024-08-17T17:38:24.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:24 smithi003 ceph-mon[103344]: 1.2 scrub ok 2024-08-17T17:38:24.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:24 smithi003 ceph-mon[103344]: 1.5 scrub starts 2024-08-17T17:38:24.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:24 smithi003 ceph-mon[103344]: 1.5 scrub ok 2024-08-17T17:38:24.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:24 smithi003 ceph-mon[103344]: 1.13 scrub starts 2024-08-17T17:38:24.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:24 smithi003 ceph-mon[103344]: 1.13 scrub ok 2024-08-17T17:38:24.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:24 smithi003 ceph-mon[100615]: 1.4 scrub starts 2024-08-17T17:38:24.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:24 smithi003 ceph-mon[100615]: 1.4 scrub ok 2024-08-17T17:38:24.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:24 smithi003 ceph-mon[100615]: 1.2 scrub starts 2024-08-17T17:38:24.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:24 smithi003 ceph-mon[100615]: 1.2 scrub ok 2024-08-17T17:38:24.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:24 smithi003 ceph-mon[100615]: 1.5 scrub starts 2024-08-17T17:38:24.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:24 smithi003 ceph-mon[100615]: 1.5 scrub ok 2024-08-17T17:38:24.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:24 smithi003 ceph-mon[100615]: 1.13 scrub starts 2024-08-17T17:38:24.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:24 smithi003 ceph-mon[100615]: 1.13 scrub ok 2024-08-17T17:38:24.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:24.849 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:24.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:24.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:24.951 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:24.989 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:24.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:24.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:25.091 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:25.127 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:25.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:25.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:25.229 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:25.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:25.296 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:25.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:25.326 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 35 16 64347 64331 114.853 85.375 0.0139887 0.00870179 2024-08-17T17:38:25.365 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-08-17T17:38:25.366 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:38:25.366 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-08-17T17:38:25.366 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 1.000000 2024-08-17T17:38:25.366 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 4 1 2024-08-17T17:38:25.397 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:25.459 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:25.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:25.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:25.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:25 smithi003 ceph-mon[103344]: 1.c scrub starts 2024-08-17T17:38:25.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:25 smithi003 ceph-mon[103344]: 1.c scrub ok 2024-08-17T17:38:25.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:25 smithi003 ceph-mon[103344]: pgmap v1004: 109 pgs: 2 unknown, 107 active+clean; 3.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-08-17T17:38:25.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:25 smithi003 ceph-mon[103344]: 1.12 scrub starts 2024-08-17T17:38:25.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:25 smithi003 ceph-mon[103344]: 1.12 scrub ok 2024-08-17T17:38:25.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:25 smithi003 ceph-mon[100615]: 1.c scrub starts 2024-08-17T17:38:25.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:25 smithi003 ceph-mon[100615]: 1.c scrub ok 2024-08-17T17:38:25.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:25 smithi003 ceph-mon[100615]: pgmap v1004: 109 pgs: 2 unknown, 107 active+clean; 3.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-08-17T17:38:25.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:25 smithi003 ceph-mon[100615]: 1.12 scrub starts 2024-08-17T17:38:25.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:25 smithi003 ceph-mon[100615]: 1.12 scrub ok 2024-08-17T17:38:25.561 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:25.590 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:25.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:25.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:25.626 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:38:25.626 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:38:25.658 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:25 smithi053 ceph-mon[89963]: 1.c scrub starts 2024-08-17T17:38:25.658 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:25 smithi053 ceph-mon[89963]: 1.c scrub ok 2024-08-17T17:38:25.658 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:25 smithi053 ceph-mon[89963]: pgmap v1004: 109 pgs: 2 unknown, 107 active+clean; 3.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-08-17T17:38:25.659 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:25 smithi053 ceph-mon[89963]: 1.12 scrub starts 2024-08-17T17:38:25.659 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:25 smithi053 ceph-mon[89963]: 1.12 scrub ok 2024-08-17T17:38:25.692 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:25.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:25.748 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:25.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:25.849 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:25.897 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:25.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:25.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:25.936 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:38:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:38:25] "GET /metrics HTTP/1.1" 200 36517 "" "Prometheus/2.51.0" 2024-08-17T17:38:25.999 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:26.044 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:26.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:26.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:26.146 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:26.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:26.183 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:26.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:26.284 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:26.322 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:26.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:26.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:26.326 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 36 16 65529 65513 113.715 73.875 0.0144688 0.00878851 2024-08-17T17:38:26.426 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:26.452 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:26.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:26.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:26.456 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:38:26.464 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-17T17:38:26.513 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:26 smithi003 ceph-mon[100615]: 1.0 scrub starts 2024-08-17T17:38:26.514 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:26 smithi003 ceph-mon[100615]: 1.0 scrub ok 2024-08-17T17:38:26.514 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:26 smithi003 ceph-mon[100615]: 1.d scrub starts 2024-08-17T17:38:26.514 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:26 smithi003 ceph-mon[100615]: 1.d scrub ok 2024-08-17T17:38:26.514 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:26 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-08-17T17:38:26.514 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:26 smithi003 ceph-mon[103344]: 1.0 scrub starts 2024-08-17T17:38:26.514 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:26 smithi003 ceph-mon[103344]: 1.0 scrub ok 2024-08-17T17:38:26.514 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:26 smithi003 ceph-mon[103344]: 1.d scrub starts 2024-08-17T17:38:26.514 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:26 smithi003 ceph-mon[103344]: 1.d scrub ok 2024-08-17T17:38:26.514 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:26 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-08-17T17:38:26.554 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:26.556 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:38:26.608 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:26.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:26.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:26.710 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:26.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:26 smithi053 ceph-mon[89963]: 1.0 scrub starts 2024-08-17T17:38:26.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:26 smithi053 ceph-mon[89963]: 1.0 scrub ok 2024-08-17T17:38:26.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:26 smithi053 ceph-mon[89963]: 1.d scrub starts 2024-08-17T17:38:26.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:26 smithi053 ceph-mon[89963]: 1.d scrub ok 2024-08-17T17:38:26.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:26 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-08-17T17:38:26.741 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:26.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:26.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:26.843 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:26.873 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:26.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:26.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:26.976 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:27.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:27.026 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:27.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:27.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:27.172 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:27.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:27.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:27.270 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:38:27.274 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:27.303 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:27.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:27.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:27.326 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 37 16 66606 66590 112.46 67.3125 0.00480142 0.00888466 2024-08-17T17:38:27.405 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:27.440 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:27.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:27.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:27.542 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:27.563 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:27.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:27.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:27.665 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:27.667 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:27 smithi053 ceph-mon[89963]: 1.a scrub starts 2024-08-17T17:38:27.667 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:27 smithi053 ceph-mon[89963]: 1.a scrub ok 2024-08-17T17:38:27.667 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:27 smithi053 ceph-mon[89963]: pgmap v1005: 109 pgs: 109 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-08-17T17:38:27.667 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:27 smithi053 ceph-mon[89963]: 1.10 scrub starts 2024-08-17T17:38:27.667 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:27 smithi053 ceph-mon[89963]: 1.10 scrub ok 2024-08-17T17:38:27.667 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:27 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-08-17T17:38:27.667 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:27 smithi053 ceph-mon[89963]: osdmap e319: 8 total, 8 up, 8 in 2024-08-17T17:38:27.704 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:27.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:27.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:27.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:27 smithi003 ceph-mon[103344]: 1.a scrub starts 2024-08-17T17:38:27.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:27 smithi003 ceph-mon[103344]: 1.a scrub ok 2024-08-17T17:38:27.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:27 smithi003 ceph-mon[103344]: pgmap v1005: 109 pgs: 109 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-08-17T17:38:27.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:27 smithi003 ceph-mon[103344]: 1.10 scrub starts 2024-08-17T17:38:27.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:27 smithi003 ceph-mon[103344]: 1.10 scrub ok 2024-08-17T17:38:27.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:27 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-08-17T17:38:27.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:27 smithi003 ceph-mon[103344]: osdmap e319: 8 total, 8 up, 8 in 2024-08-17T17:38:27.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:27 smithi003 ceph-mon[100615]: 1.a scrub starts 2024-08-17T17:38:27.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:27 smithi003 ceph-mon[100615]: 1.a scrub ok 2024-08-17T17:38:27.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:27 smithi003 ceph-mon[100615]: pgmap v1005: 109 pgs: 109 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-08-17T17:38:27.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:27 smithi003 ceph-mon[100615]: 1.10 scrub starts 2024-08-17T17:38:27.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:27 smithi003 ceph-mon[100615]: 1.10 scrub ok 2024-08-17T17:38:27.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:27 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-08-17T17:38:27.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:27 smithi003 ceph-mon[100615]: osdmap e319: 8 total, 8 up, 8 in 2024-08-17T17:38:27.807 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:27.846 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:27.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:27.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:27.949 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:27.988 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:27.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:27.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:28.090 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:28.128 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:28.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:28.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:28.230 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:28.268 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:28.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:28.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:28.327 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 38 16 67824 67808 111.504 76.125 0.00706523 0.00896022 2024-08-17T17:38:28.370 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:28.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:28.409 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:28.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:28.510 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:28.550 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:28.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:28.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:28.652 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:28.686 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:28.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:28.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:28.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:28 smithi053 ceph-mon[89963]: osdmap e320: 8 total, 8 up, 8 in 2024-08-17T17:38:28.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:28 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:38:28.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:28 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:38:28.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:28 smithi053 ceph-mon[89963]: osdmap e321: 8 total, 8 up, 8 in 2024-08-17T17:38:28.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:28 smithi003 ceph-mon[103344]: osdmap e320: 8 total, 8 up, 8 in 2024-08-17T17:38:28.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:28 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:38:28.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:28 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:38:28.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:28 smithi003 ceph-mon[103344]: osdmap e321: 8 total, 8 up, 8 in 2024-08-17T17:38:28.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:28 smithi003 ceph-mon[100615]: osdmap e320: 8 total, 8 up, 8 in 2024-08-17T17:38:28.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:28 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:38:28.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:28 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:38:28.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:28 smithi003 ceph-mon[100615]: osdmap e321: 8 total, 8 up, 8 in 2024-08-17T17:38:28.788 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:28.818 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:28.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:28.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:28.920 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:28.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:28.955 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:28.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:29.056 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:29.085 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:29.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:29.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:29.187 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:29.225 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:29.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:29.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:29.326 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:29.327 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 39 16 69192 69176 110.837 85.5 0.00401096 0.00901703 2024-08-17T17:38:29.372 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:29.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:29.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:29.474 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:29.513 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:29.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:29.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:29.614 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:29.651 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:29.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:29.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:29.665 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:29 smithi003 ceph-mon[103344]: 1.7 scrub starts 2024-08-17T17:38:29.665 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:29 smithi003 ceph-mon[103344]: 1.7 scrub ok 2024-08-17T17:38:29.665 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:29 smithi003 ceph-mon[103344]: pgmap v1008: 109 pgs: 109 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 60 MiB/s wr, 957 op/s 2024-08-17T17:38:29.665 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:29 smithi003 ceph-mon[103344]: osdmap e322: 8 total, 8 up, 8 in 2024-08-17T17:38:29.665 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:29 smithi003 ceph-mon[100615]: 1.7 scrub starts 2024-08-17T17:38:29.666 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:29 smithi003 ceph-mon[100615]: 1.7 scrub ok 2024-08-17T17:38:29.666 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:29 smithi003 ceph-mon[100615]: pgmap v1008: 109 pgs: 109 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 60 MiB/s wr, 957 op/s 2024-08-17T17:38:29.666 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:29 smithi003 ceph-mon[100615]: osdmap e322: 8 total, 8 up, 8 in 2024-08-17T17:38:29.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:29 smithi053 ceph-mon[89963]: 1.7 scrub starts 2024-08-17T17:38:29.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:29 smithi053 ceph-mon[89963]: 1.7 scrub ok 2024-08-17T17:38:29.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:29 smithi053 ceph-mon[89963]: pgmap v1008: 109 pgs: 109 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 60 MiB/s wr, 957 op/s 2024-08-17T17:38:29.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:29 smithi053 ceph-mon[89963]: osdmap e322: 8 total, 8 up, 8 in 2024-08-17T17:38:29.753 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:29.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:29.783 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:29.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:29.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:29.923 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:29.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:29.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:30.025 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:30.061 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:30.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:30.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:30.163 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:30.194 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:30.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:30.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:30.296 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:30.327 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-08-17T17:38:30.327094+0000 min lat: 0.001429 max lat: 0.203649 avg lat: 0.00907589 2024-08-17T17:38:30.327 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:38:30.327 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 40 16 70508 70492 110.122 82.25 0.00664639 0.00907589 2024-08-17T17:38:30.335 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:30.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:30.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:30.437 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:30.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:38:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:38:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:38:30.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:30 smithi053 ceph-mon[89963]: 1.6 scrub starts 2024-08-17T17:38:30.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:30 smithi053 ceph-mon[89963]: 1.6 scrub ok 2024-08-17T17:38:30.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:30 smithi053 ceph-mon[89963]: osdmap e323: 8 total, 8 up, 8 in 2024-08-17T17:38:30.472 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:30.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:30.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:30.574 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:30.610 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:30.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:30.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:30.712 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:30.750 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:30.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:30.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:30.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:30 smithi003 ceph-mon[103344]: 1.6 scrub starts 2024-08-17T17:38:30.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:30 smithi003 ceph-mon[103344]: 1.6 scrub ok 2024-08-17T17:38:30.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:30 smithi003 ceph-mon[103344]: osdmap e323: 8 total, 8 up, 8 in 2024-08-17T17:38:30.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:30 smithi003 ceph-mon[100615]: 1.6 scrub starts 2024-08-17T17:38:30.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:30 smithi003 ceph-mon[100615]: 1.6 scrub ok 2024-08-17T17:38:30.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:30 smithi003 ceph-mon[100615]: osdmap e323: 8 total, 8 up, 8 in 2024-08-17T17:38:30.853 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:30.922 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:30.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:30.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:31.023 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:31.059 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:31.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:31.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:31.161 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:31.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:31.195 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:31.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:31.296 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:31.327 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 41 16 71711 71695 109.268 75.1875 0.0293884 0.00914556 2024-08-17T17:38:31.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:31.354 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:31.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:31.455 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:31.502 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:31.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:31.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:31.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:31 smithi003 ceph-mon[103344]: pgmap v1011: 109 pgs: 3 peering, 6 active+undersized+remapped, 100 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 97 MiB/s wr, 1.56k op/s; 34 KiB/s, 0 objects/s recovering 2024-08-17T17:38:31.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:31 smithi003 ceph-mon[103344]: 1.11 scrub starts 2024-08-17T17:38:31.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:31 smithi003 ceph-mon[103344]: 1.11 scrub ok 2024-08-17T17:38:31.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:31 smithi003 ceph-mon[100615]: pgmap v1011: 109 pgs: 3 peering, 6 active+undersized+remapped, 100 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 97 MiB/s wr, 1.56k op/s; 34 KiB/s, 0 objects/s recovering 2024-08-17T17:38:31.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:31 smithi003 ceph-mon[100615]: 1.11 scrub starts 2024-08-17T17:38:31.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:31 smithi003 ceph-mon[100615]: 1.11 scrub ok 2024-08-17T17:38:31.604 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:31.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:31.640 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:31.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:31.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:31 smithi053 ceph-mon[89963]: pgmap v1011: 109 pgs: 3 peering, 6 active+undersized+remapped, 100 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 97 MiB/s wr, 1.56k op/s; 34 KiB/s, 0 objects/s recovering 2024-08-17T17:38:31.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:31 smithi053 ceph-mon[89963]: 1.11 scrub starts 2024-08-17T17:38:31.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:31 smithi053 ceph-mon[89963]: 1.11 scrub ok 2024-08-17T17:38:31.741 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:31.788 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:31.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:31.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:31.890 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:31.918 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:31.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:31.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:32.019 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:32.058 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:32.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:32.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:32.160 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:32.188 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:32.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:32.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:32.291 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:32.330 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 42 16 72938 72922 108.49 76.6875 0.0164461 0.00921022 2024-08-17T17:38:32.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:32.336 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:32.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:32.343 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:38:32.437 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:32.465 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:32.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:32.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:32.566 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:32.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:32.611 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:32.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:32.712 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:32.761 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:32.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:32.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:32.863 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:32.894 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:32.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:32.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:32.996 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:33.042 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:33.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:33.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:33.144 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:33.156 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:38:33.174 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:33.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:33.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:33.276 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:33.328 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 43 16 74113 74097 107.675 73.4375 0.0171971 0.00928191 2024-08-17T17:38:33.340 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:33.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:33.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:33.400 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-08-17T17:38:33.401 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-08-17T17:38:33.443 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:33.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:33.490 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:33.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:33.591 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:33.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:33.628 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:33.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:33.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:33 smithi053 ceph-mon[89963]: 1.10 scrub starts 2024-08-17T17:38:33.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:33 smithi053 ceph-mon[89963]: 1.10 scrub ok 2024-08-17T17:38:33.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:33 smithi053 ceph-mon[89963]: pgmap v1013: 109 pgs: 3 peering, 5 active+undersized+remapped, 101 active+clean; 4.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 80 MiB/s wr, 1.27k op/s; 28 KiB/s, 0 objects/s recovering 2024-08-17T17:38:33.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:33 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2349786447' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:38:33.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:33 smithi053 ceph-mon[89963]: 1.e scrub starts 2024-08-17T17:38:33.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:33 smithi053 ceph-mon[89963]: 1.e scrub ok 2024-08-17T17:38:33.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:33 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1725355541' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:38:33.729 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:33.740 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:33 smithi003 ceph-mon[103344]: 1.10 scrub starts 2024-08-17T17:38:33.740 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:33 smithi003 ceph-mon[103344]: 1.10 scrub ok 2024-08-17T17:38:33.740 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:33 smithi003 ceph-mon[103344]: pgmap v1013: 109 pgs: 3 peering, 5 active+undersized+remapped, 101 active+clean; 4.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 80 MiB/s wr, 1.27k op/s; 28 KiB/s, 0 objects/s recovering 2024-08-17T17:38:33.740 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:33 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2349786447' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:38:33.740 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:33 smithi003 ceph-mon[103344]: 1.e scrub starts 2024-08-17T17:38:33.740 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:33 smithi003 ceph-mon[103344]: 1.e scrub ok 2024-08-17T17:38:33.740 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:33 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1725355541' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:38:33.740 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:33 smithi003 ceph-mon[100615]: 1.10 scrub starts 2024-08-17T17:38:33.740 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:33 smithi003 ceph-mon[100615]: 1.10 scrub ok 2024-08-17T17:38:33.740 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:33 smithi003 ceph-mon[100615]: pgmap v1013: 109 pgs: 3 peering, 5 active+undersized+remapped, 101 active+clean; 4.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 80 MiB/s wr, 1.27k op/s; 28 KiB/s, 0 objects/s recovering 2024-08-17T17:38:33.741 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:33 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2349786447' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:38:33.741 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:33 smithi003 ceph-mon[100615]: 1.e scrub starts 2024-08-17T17:38:33.741 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:33 smithi003 ceph-mon[100615]: 1.e scrub ok 2024-08-17T17:38:33.741 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:33 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1725355541' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:38:33.778 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:33.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:33.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:33.880 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:33.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:33.915 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:33.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:34.016 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:34.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:34.057 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:34.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:34.158 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:34.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:34.212 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:34.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:34.313 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:34.330 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 44 16 75392 75376 107.041 79.9375 0.00970201 0.00933643 2024-08-17T17:38:34.362 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:34.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:34.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:34.464 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:34.502 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:34.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:34.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:34.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:34 smithi003 ceph-mon[103344]: 1.f scrub starts 2024-08-17T17:38:34.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:34 smithi003 ceph-mon[103344]: 1.f scrub ok 2024-08-17T17:38:34.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:34 smithi003 ceph-mon[103344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:38:34.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:34 smithi003 ceph-mon[103344]: Cluster is now healthy 2024-08-17T17:38:34.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:34 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2349786447' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:38:34.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:34 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1725355541' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-08-17T17:38:34.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:34 smithi003 ceph-mon[103344]: osdmap e324: 8 total, 8 up, 8 in 2024-08-17T17:38:34.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:34 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:38:34.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:34 smithi003 ceph-mon[100615]: 1.f scrub starts 2024-08-17T17:38:34.506 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:34 smithi003 ceph-mon[100615]: 1.f scrub ok 2024-08-17T17:38:34.506 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:34 smithi003 ceph-mon[100615]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:38:34.506 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:34 smithi003 ceph-mon[100615]: Cluster is now healthy 2024-08-17T17:38:34.506 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:34 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2349786447' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:38:34.506 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:34 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1725355541' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-08-17T17:38:34.506 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:34 smithi003 ceph-mon[100615]: osdmap e324: 8 total, 8 up, 8 in 2024-08-17T17:38:34.506 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:34 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:38:34.604 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:34.646 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:34.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:34.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:34.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:34 smithi053 ceph-mon[89963]: 1.f scrub starts 2024-08-17T17:38:34.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:34 smithi053 ceph-mon[89963]: 1.f scrub ok 2024-08-17T17:38:34.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:34 smithi053 ceph-mon[89963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:38:34.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:34 smithi053 ceph-mon[89963]: Cluster is now healthy 2024-08-17T17:38:34.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:34 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2349786447' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:38:34.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:34 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1725355541' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-08-17T17:38:34.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:34 smithi053 ceph-mon[89963]: osdmap e324: 8 total, 8 up, 8 in 2024-08-17T17:38:34.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:34 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:38:34.748 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:34.778 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:34.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:34.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:34.880 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:34.912 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:34.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:34.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:35.014 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:35.052 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:35.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:35.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:35.154 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:35.195 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:35.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:35.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:35.297 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:35.331 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 45 16 76723 76707 106.511 83.1875 0.0105095 0.00938372 2024-08-17T17:38:35.343 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:35.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:35.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:35.445 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:35.475 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:35.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:35.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:35.535 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:35 smithi003 ceph-mon[103344]: pgmap v1015: 109 pgs: 3 peering, 106 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s; 9.5 KiB/s, 0 objects/s recovering 2024-08-17T17:38:35.536 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:35 smithi003 ceph-mon[100615]: pgmap v1015: 109 pgs: 3 peering, 106 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s; 9.5 KiB/s, 0 objects/s recovering 2024-08-17T17:38:35.577 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:35.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:35.635 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:35.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:35.736 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:35.772 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:35.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:35.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:35.874 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:35.921 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:35.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:35.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:35.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:35 smithi053 ceph-mon[89963]: pgmap v1015: 109 pgs: 3 peering, 106 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s; 9.5 KiB/s, 0 objects/s recovering 2024-08-17T17:38:36.005 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:38:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:38:35] "GET /metrics HTTP/1.1" 200 36517 "" "Prometheus/2.51.0" 2024-08-17T17:38:36.023 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:36.070 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:36.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:36.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:36.172 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:36.207 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:36.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:36.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:36.309 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:36.330 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 46 16 78014 77998 105.949 80.6875 0.00932901 0.00943374 2024-08-17T17:38:36.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:36.346 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:36.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:36.447 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:36.470 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:36.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:36.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:36.572 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:36.593 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:36 smithi003 ceph-mon[103344]: 1.d scrub starts 2024-08-17T17:38:36.593 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:36 smithi003 ceph-mon[103344]: 1.d scrub ok 2024-08-17T17:38:36.593 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:36 smithi003 ceph-mon[100615]: 1.d scrub starts 2024-08-17T17:38:36.594 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:36 smithi003 ceph-mon[100615]: 1.d scrub ok 2024-08-17T17:38:36.594 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:36.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:36.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:36.696 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:36.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:36.726 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:36.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:36.827 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:36.849 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:36.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:36.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:36.867 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:36 smithi053 ceph-mon[89963]: 1.d scrub starts 2024-08-17T17:38:36.867 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:36 smithi053 ceph-mon[89963]: 1.d scrub ok 2024-08-17T17:38:36.951 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:37.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:37.029 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:37.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:37.130 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:37.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:37.168 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:37.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:37.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:37.311 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:37.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:37.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:37.332 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 47 16 78668 78652 104.561 40.875 0.0133896 0.00955841 2024-08-17T17:38:37.413 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:37.452 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:37.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:37.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:37.541 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:37 smithi053 ceph-mon[89963]: pgmap v1016: 109 pgs: 109 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-08-17T17:38:37.554 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:37.582 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:38:37.590 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:37.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:37.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:37.693 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:37.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:37.740 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:37.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:37.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:37 smithi003 ceph-mon[103344]: pgmap v1016: 109 pgs: 109 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-08-17T17:38:37.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:37 smithi003 ceph-mon[100615]: pgmap v1016: 109 pgs: 109 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-08-17T17:38:37.777 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:38:37.777 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:38:37.842 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:37.885 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:37.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:37.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:37.987 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:38.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:38.043 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:38.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:38.051 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:38:38.051 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:38:38.090 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:38:38.091 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:38:38.131 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:38:38.131 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:38:38.144 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:38.174 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:38:38.177 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:38:38.178 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:38.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:38.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:38.210 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:38:38.210 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:38:38.243 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:38:38.243 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:38:38.279 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:38:38.279 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:38:38.280 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:38.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:38.315 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:38.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:38.316 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:38:38.317 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:38:38.332 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 48 16 79764 79748 103.81 68.5 0.0146552 0.00962729 2024-08-17T17:38:38.416 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:38.458 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:38.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:38.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:38.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:38.597 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:38.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:38.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:38.613 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:38 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2628382546' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:38:38.613 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:38:38.613 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:38:38.613 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:38 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2628382546' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:38:38.613 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:38:38.613 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:38:38.658 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-17T17:38:38.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:38.796 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:38.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:38.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:38.898 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:38.929 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:38.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:38.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:38.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:38 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2628382546' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:38:38.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:38:38.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:38:39.031 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:39.059 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:39.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:39.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:39.161 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:39.192 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:39.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:39.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:39.294 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:39.328 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:39.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:39.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:39.332 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 49 16 80969 80953 103.228 75.3125 0.00834547 0.00968225 2024-08-17T17:38:39.430 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:39.466 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:39.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:39.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:39.568 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:39.603 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:39.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:39.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:39.705 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:39.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:39.742 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:39.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:39.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:39 smithi003 ceph-mon[100615]: pgmap v1017: 109 pgs: 109 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 49 MiB/s wr, 781 op/s 2024-08-17T17:38:39.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:39 smithi003 ceph-mon[103344]: pgmap v1017: 109 pgs: 109 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 49 MiB/s wr, 781 op/s 2024-08-17T17:38:39.843 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:39.871 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:39.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:39.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:39.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:39 smithi053 ceph-mon[89963]: pgmap v1017: 109 pgs: 109 active+clean; 4.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 49 MiB/s wr, 781 op/s 2024-08-17T17:38:39.973 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:40.004 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:40.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:40.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:40.106 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:40.142 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:40.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:40.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:40.244 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:40.295 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:40.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:40.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:40.336 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 50 16 82236 82220 102.745 79.1875 0.0166215 0.00972766 2024-08-17T17:38:40.346 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:38:40.396 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:40.427 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:38:40.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:40.428 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:40.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:40.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:38:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:38:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:38:40.529 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:40.576 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:40.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:40.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:40.678 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:40.729 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:40.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:40.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:40.832 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:40.865 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:40.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:40.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:40.967 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:41.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:41.006 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:41.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:41.055 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-08-17T17:38:41.055 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:38:41.055 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 3, 5, 6, 7, 4] 2024-08-17T17:38:41.055 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd out 4 2024-08-17T17:38:41.107 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:41.160 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:41.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:41.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:41.262 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:41.297 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:41.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:41.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:41.334 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 51 16 83397 83381 102.153 72.5625 0.00577938 0.00978313 2024-08-17T17:38:41.398 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:41.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:41.441 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:41.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:41.543 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:41.581 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:41.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:41.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:41.683 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:41.721 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:41.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:41.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:41.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:41 smithi003 ceph-mon[103344]: pgmap v1018: 109 pgs: 109 active+clean; 5.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-08-17T17:38:41.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:41 smithi003 ceph-mon[100615]: pgmap v1018: 109 pgs: 109 active+clean; 5.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-08-17T17:38:41.823 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:41.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:41.866 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:41.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:41.888 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:38:41.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:41 smithi053 ceph-mon[89963]: pgmap v1018: 109 pgs: 109 active+clean; 5.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-08-17T17:38:41.968 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:42.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:42.015 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:42.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:42.116 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:42.180 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:42.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:42.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:42.282 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:42.313 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:42.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:42.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:42.334 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 52 16 84581 84565 101.611 74 0.0130962 0.00983653 2024-08-17T17:38:42.415 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:42.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:42.461 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:42.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:42.562 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:42.599 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:42.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:42.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:42.702 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:42.732 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:42.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:42.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:42.834 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:42.866 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:42.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:42.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:42.968 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:43.017 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:43.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:43.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:43.119 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:43.161 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:43.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:43.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:43.262 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:43.288 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:43.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:43.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:43.334 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 53 16 85772 85756 101.098 74.4375 0.00668672 0.00988636 2024-08-17T17:38:43.390 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:43.433 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:43.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:43.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:43.535 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:43.566 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:43.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:43.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:43.668 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:43.710 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:43.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:43.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:43.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:43 smithi003 ceph-mon[103344]: pgmap v1019: 109 pgs: 109 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-08-17T17:38:43.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:43 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:38:43.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:43 smithi003 ceph-mon[100615]: pgmap v1019: 109 pgs: 109 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-08-17T17:38:43.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:43 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:38:43.812 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:43.862 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:43.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:43.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:43.964 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:43.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:43 smithi053 ceph-mon[89963]: pgmap v1019: 109 pgs: 109 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-08-17T17:38:43.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:43 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:38:44.015 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:44.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:44.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:44.117 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:44.168 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:44.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:44.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:44.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:44.312 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:44.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:44.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:44.334 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 54 16 86854 86838 100.478 67.625 0.00574095 0.00994495 2024-08-17T17:38:44.414 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:44.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:44.462 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:44.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:44.563 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:44.571 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-08-17T17:38:44.588 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:44.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:44.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:44.601 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:38:44.602 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-08-17T17:38:44.602 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-08-17T17:38:44.602 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-08-17T17:38:44.602 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:38:44.689 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:44.726 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:44.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:44.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:44.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:44 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/4283457649' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:38:44.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:44 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:38:44.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:44 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/4283457649' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:38:44.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:44 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:38:44.833 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:44.870 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:44.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:44.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:44.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:44 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/4283457649' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:38:44.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:44 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:38:44.972 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:45.008 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:45.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:45.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:45.110 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:45.145 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:45.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:45.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:45.247 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:45.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:45.288 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:45.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:45.334 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 55 16 87965 87949 99.9137 69.4375 0.0144552 0.0100032 2024-08-17T17:38:45.389 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:45.432 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:45.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:45.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:45.534 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:45.566 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:45.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:45.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:45.668 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:45.707 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:45.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:45.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:45.809 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:45.846 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:45.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:45.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:45.925 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:45 smithi003 ceph-mon[103344]: pgmap v1020: 109 pgs: 109 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-08-17T17:38:45.926 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:45 smithi003 ceph-mon[103344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:38:45.926 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:45 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:38:45.926 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:45 smithi003 ceph-mon[103344]: osdmap e325: 8 total, 8 up, 8 in 2024-08-17T17:38:45.926 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:45 smithi003 ceph-mon[100615]: pgmap v1020: 109 pgs: 109 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-08-17T17:38:45.926 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:45 smithi003 ceph-mon[100615]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:38:45.926 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:45 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:38:45.926 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:45 smithi003 ceph-mon[100615]: osdmap e325: 8 total, 8 up, 8 in 2024-08-17T17:38:45.927 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:38:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:38:45] "GET /metrics HTTP/1.1" 200 36498 "" "Prometheus/2.51.0" 2024-08-17T17:38:45.948 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:45.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:45 smithi053 ceph-mon[89963]: pgmap v1020: 109 pgs: 109 active+clean; 5.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-08-17T17:38:45.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:45 smithi053 ceph-mon[89963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:38:45.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:45 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:38:45.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:45 smithi053 ceph-mon[89963]: osdmap e325: 8 total, 8 up, 8 in 2024-08-17T17:38:45.990 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:45.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:45.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:46.094 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:46.132 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:46.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:46.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:46.233 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:46.256 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:46.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:46.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:46.335 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 56 16 89097 89081 99.3929 70.75 0.00843694 0.0100561 2024-08-17T17:38:46.357 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:46.401 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:46.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:46.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:46.503 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:46.539 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:38:46.548 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:46.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:46.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:46.651 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:46.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:46.709 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:46.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:46.810 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:46.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:46.860 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:46.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:46.961 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:46.991 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:46.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:46.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:46.997 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-17T17:38:47.092 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:47.174 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:47.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:47.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:47.276 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:47.307 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:47.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:47.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:47.337 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 57 16 90141 90125 98.7905 65.25 0.017961 0.0101172 2024-08-17T17:38:47.408 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:47.455 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:47.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:47.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:47.527 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:38:47.557 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:47.586 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:47.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:47.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:47.690 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:47.736 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:47.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:47.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:47.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:47 smithi003 ceph-mon[103344]: pgmap v1022: 109 pgs: 109 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-08-17T17:38:47.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:47 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:38:47.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:47 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:38:47.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:47 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:38:47.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:47 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:38:47.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:47 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:38:47.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:47 smithi003 ceph-mon[100615]: pgmap v1022: 109 pgs: 109 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-08-17T17:38:47.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:47 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:38:47.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:47 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:38:47.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:47 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:38:47.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:47 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:38:47.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:47 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:38:47.838 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:47.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:47.891 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:47.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:47.919 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:47 smithi053 ceph-mon[89963]: pgmap v1022: 109 pgs: 109 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-08-17T17:38:47.919 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:47 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:38:47.919 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:47 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:38:47.919 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:47 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:38:47.919 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:47 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:38:47.919 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:47 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:38:47.991 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:48.034 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:48.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:48.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:48.136 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:48.172 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:48.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:48.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:48.274 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:48.308 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:48.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:48.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:48.337 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 58 16 91245 91229 98.2769 69 0.00883777 0.0101697 2024-08-17T17:38:48.409 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:48.443 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:48.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:48.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:48.544 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:48.580 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:48.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:48.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:48.682 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:48.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:48.723 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:48.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:48.824 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:48.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:48.862 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:48.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:48.963 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:49.004 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:49.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:49.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:49.012 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:38:49.107 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:49.139 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:49.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:49.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:49.241 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:49.276 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:49.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:49.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:49.337 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 59 16 92434 92418 97.8707 74.3125 0.00954448 0.0102129 2024-08-17T17:38:49.378 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:49.414 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:49.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:49.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:49.515 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:49.557 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:49.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:49.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:49.568 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:38:49.659 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:49.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:49.720 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:49.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:49.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:49 smithi003 ceph-mon[103344]: pgmap v1023: 109 pgs: 109 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-08-17T17:38:49.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:49 smithi003 ceph-mon[100615]: pgmap v1023: 109 pgs: 109 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-08-17T17:38:49.821 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:49.850 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:49.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:49.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:49.952 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:49.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:49 smithi053 ceph-mon[89963]: pgmap v1023: 109 pgs: 109 active+clean; 5.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-08-17T17:38:49.989 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:49.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:49.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:50.091 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:50.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:50.126 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:50.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:50.227 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:50.268 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:50.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:50.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:50.337 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-08-17T17:38:50.337358+0000 min lat: 0.001429 max lat: 0.376468 avg lat: 0.01026 2024-08-17T17:38:50.337 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:38:50.337 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 60 16 93565 93549 97.4176 70.6875 0.00700038 0.01026 2024-08-17T17:38:50.369 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:50.413 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:50.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:50.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:50.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:38:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:38:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:38:50.515 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:50.545 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:50.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:50.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:50.648 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:50.675 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:50.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:50.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:50.777 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:50.819 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:50.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:50.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:50.921 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:50.951 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:50.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:50.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:51.053 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:51.091 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:51.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:51.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:51.193 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:51.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:51.236 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:51.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:51.337 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 61 16 94709 94693 96.9927 71.5 0.0125312 0.0103051 2024-08-17T17:38:51.338 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:51.380 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:51.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:51.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:51.483 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:51.513 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:51.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:51.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:51.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:51.650 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:51.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:51.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:51.752 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:51.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:51 smithi003 ceph-mon[103344]: pgmap v1024: 109 pgs: 109 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-08-17T17:38:51.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:51 smithi003 ceph-mon[100615]: pgmap v1024: 109 pgs: 109 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-08-17T17:38:51.791 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:51.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:51.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:51.894 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:51.923 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:51.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:51.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:51.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:51 smithi053 ceph-mon[89963]: pgmap v1024: 109 pgs: 109 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-08-17T17:38:52.025 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:52.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:52.066 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:52.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:52.167 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:52.207 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:52.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:52.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:52.309 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:52.338 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 62 16 95746 95730 96.4717 64.8125 0.0169484 0.0103613 2024-08-17T17:38:52.352 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:52.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:52.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:52.454 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:52.476 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:52.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:52.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:52.578 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:52.615 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:52.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:52.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:52.717 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:52.751 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:52.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:52.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:52.853 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:52.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:52.890 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:52.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:52.991 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:53.030 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:53.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:53.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:53.132 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:53.166 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:53.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:53.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:53.268 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:53.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:53.304 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:53.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:53.338 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 63 16 96727 96711 95.9137 61.3125 0.00663914 0.0104211 2024-08-17T17:38:53.405 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:53.445 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:53.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:53.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:53.547 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:53.588 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:53.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:53.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:53.690 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:53.722 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:53.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:53.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:53 smithi003 ceph-mon[100615]: pgmap v1025: 109 pgs: 109 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-08-17T17:38:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:38:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:38:53.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:53 smithi003 ceph-mon[103344]: pgmap v1025: 109 pgs: 109 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-08-17T17:38:53.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:38:53.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:38:53.824 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:53.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:53.865 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:53.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:53.966 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:53 smithi053 ceph-mon[89963]: pgmap v1025: 109 pgs: 109 active+clean; 5.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-08-17T17:38:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:38:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:38:53.999 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:54.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:54.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:54.101 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:54.144 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:54.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:54.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:54.246 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:54.288 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:54.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:54.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:54.339 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 64 16 97802 97786 95.4648 67.1875 0.0173197 0.0104696 2024-08-17T17:38:54.390 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:54.425 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:54.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:54.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:54.527 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:54.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:54.571 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:54.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:54.673 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:54.711 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:54.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:54.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:54.813 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:54.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:54.835 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:54.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:54.936 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:54.978 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:54.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:54.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:55.080 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:55.101 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:55.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:55.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:55.203 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:55.243 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:55.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:55.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:55.339 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 65 16 98817 98801 94.9721 63.4375 0.007576 0.0105246 2024-08-17T17:38:55.344 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:55.366 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:55.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:55.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:55.468 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:55.499 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:55.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:55.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:55.600 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:55.622 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:55.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:55.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:55.724 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:55.756 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:55.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:55.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:55.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:55 smithi003 ceph-mon[100615]: pgmap v1026: 109 pgs: 109 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 70 MiB/s wr, 1.13k op/s 2024-08-17T17:38:55.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:55 smithi003 ceph-mon[103344]: pgmap v1026: 109 pgs: 109 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 70 MiB/s wr, 1.13k op/s 2024-08-17T17:38:55.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:38:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:38:55] "GET /metrics HTTP/1.1" 200 36498 "" "Prometheus/2.51.0" 2024-08-17T17:38:55.859 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:55.891 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:55.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:55.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:55.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:55 smithi053 ceph-mon[89963]: pgmap v1026: 109 pgs: 109 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 70 MiB/s wr, 1.13k op/s 2024-08-17T17:38:55.992 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:56.014 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:56.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:56.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:56.116 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:56.158 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:56.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:56.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:56.260 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:56.309 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:56.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:56.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:56.339 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 66 16 99558 99542 94.2349 46.3125 0.016495 0.0106064 2024-08-17T17:38:56.411 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:56.443 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:56.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:56.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:56.545 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:56.577 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:56.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:56.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:56.679 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:56.713 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:56.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:56.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:56.815 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:56.854 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:56.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:56.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:56.956 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:56.988 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:56.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:56.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:57.090 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:57.121 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:57.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:57.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:57.223 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:57.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:57.270 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:57.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:57.339 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 67 16 100513 100497 93.7193 59.6875 0.00509999 0.010665 2024-08-17T17:38:57.370 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:57.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:57.402 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:57.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:57.503 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:57.540 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:57.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:57.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:57.642 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:57.661 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:57 smithi003 ceph-mon[103344]: pgmap v1027: 109 pgs: 109 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-08-17T17:38:57.661 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:57 smithi003 ceph-mon[100615]: pgmap v1027: 109 pgs: 109 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-08-17T17:38:57.670 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:57.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:57.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:57.772 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:38:57.800 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:57.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:57.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:38:57.902 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:57.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:57.948 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:57.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:57.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:57 smithi053 ceph-mon[89963]: pgmap v1027: 109 pgs: 109 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-08-17T17:38:58.050 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:58.078 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:58.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:58.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:58.180 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:38:58.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:58.242 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:58.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:38:58.339 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 68 16 101431 101415 93.1849 57.375 0.00995536 0.0107258 2024-08-17T17:38:58.342 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:58.374 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:58.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:58.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:58.476 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:58.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:58.523 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:58.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:58.624 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:38:58.667 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:58.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:58.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:38:58.769 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:58.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:58.796 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:58.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:58.896 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:38:58.925 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:58.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:58.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:38:59.027 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:38:59.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:59.085 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:59.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:38:59.186 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:59.216 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:59.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:59.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:59.318 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:59.339 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 69 16 102292 102276 92.6143 53.8125 0.026464 0.0107912 2024-08-17T17:38:59.367 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:59.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:59.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:59.393 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:38:59.394 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:38:59.394 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:38:59.394 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:38:59.394 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:38:59.394 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:38:59.394 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:38:59.394 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:38:59.394 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:38:59.395 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:38:59.395 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:38:59.395 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:38:59.395 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:38:59.395 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:38:59.395 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:38:59.395 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:38:59.471 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:38:59.499 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:59.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:59.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:38:59.601 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:59.605 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:38:59.631 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:59.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:59.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:59.729 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:38:59.729 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:38:59.729 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:59 smithi003 ceph-mon[100615]: pgmap v1028: 109 pgs: 109 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 57 MiB/s wr, 911 op/s 2024-08-17T17:38:59.729 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:59 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2863509303' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:38:59.729 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:38:59 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-a[100611]: 2024-08-17T17:38:59.562+0000 7fc4d42c5640 -1 mon.a@0(leader).osd e325 definitely_dead 0 2024-08-17T17:38:59.730 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:59 smithi003 ceph-mon[103344]: pgmap v1028: 109 pgs: 109 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 57 MiB/s wr, 911 op/s 2024-08-17T17:38:59.730 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:38:59 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2863509303' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:38:59.732 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:38:59.769 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:38:59.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:59.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:38:59.871 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:38:59.910 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:38:59.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:38:59.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:38:59.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:59 smithi053 ceph-mon[89963]: pgmap v1028: 109 pgs: 109 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 57 MiB/s wr, 911 op/s 2024-08-17T17:38:59.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:38:59 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2863509303' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:39:00.012 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:00.043 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:00.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:00.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:00.145 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:00.174 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:00.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:00.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:00.276 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:00.306 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:00.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:00.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:00.339 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 70 16 103332 103316 92.2198 65 0.00852702 0.0108386 2024-08-17T17:39:00.407 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:00.440 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:00.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:00.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:00.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:39:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:39:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:39:00.541 INFO:teuthology.orchestra.run.smithi003.stderr:marked out osd.4. 2024-08-17T17:39:00.543 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:00.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:00.593 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:00.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:00.694 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:00.717 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:00.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:00.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:00.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-17T17:39:00.819 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:00.841 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:00.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:00.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:00.942 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:00.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:00 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2801867113' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-08-17T17:39:00.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:00 smithi053 ceph-mon[89963]: Client client.admin marked osd.4 out, while it was still marked up 2024-08-17T17:39:00.966 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:00.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:00.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:01.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:00 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2801867113' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-08-17T17:39:01.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:00 smithi003 ceph-mon[103344]: Client client.admin marked osd.4 out, while it was still marked up 2024-08-17T17:39:01.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:00 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2801867113' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-08-17T17:39:01.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:00 smithi003 ceph-mon[100615]: Client client.admin marked osd.4 out, while it was still marked up 2024-08-17T17:39:01.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:01.092 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:01.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:01.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:01.194 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:01.219 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:01.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:01.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:01.321 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:01.339 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 71 16 103665 103649 91.2142 20.8125 0.00408471 0.0108483 2024-08-17T17:39:01.343 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:01.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:01.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:01.445 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:01.469 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:01.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:01.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:01.571 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:01.599 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:01.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:01.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:01.701 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:01.726 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:01.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:01.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:01.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:01 smithi003 ceph-mon[103344]: pgmap v1029: 109 pgs: 109 active+clean; 6.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-08-17T17:39:01.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:01 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2801867113' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-08-17T17:39:01.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:01 smithi003 ceph-mon[103344]: osdmap e326: 8 total, 8 up, 7 in 2024-08-17T17:39:01.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:01 smithi003 ceph-mon[100615]: pgmap v1029: 109 pgs: 109 active+clean; 6.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-08-17T17:39:01.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:01 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2801867113' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-08-17T17:39:01.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:01 smithi003 ceph-mon[100615]: osdmap e326: 8 total, 8 up, 7 in 2024-08-17T17:39:01.828 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:01.852 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:01.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:01.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:01.954 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:01.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:01 smithi053 ceph-mon[89963]: pgmap v1029: 109 pgs: 109 active+clean; 6.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-08-17T17:39:01.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:01 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2801867113' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-08-17T17:39:01.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:01 smithi053 ceph-mon[89963]: osdmap e326: 8 total, 8 up, 7 in 2024-08-17T17:39:01.979 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:01.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:01.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:02.081 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:02.107 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:02.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:02.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:02.208 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:02.235 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:02.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:02.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:02.337 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:02.341 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 72 16 103665 103649 89.9476 0 - 0.0108483 2024-08-17T17:39:02.365 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:02.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:02.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:02.467 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:02.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:02.495 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:02.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:02.596 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:02.632 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:02.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:02.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:02.734 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:02.766 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:02.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:02.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:02.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:02 smithi003 ceph-mon[100615]: osdmap e327: 8 total, 8 up, 7 in 2024-08-17T17:39:02.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:02 smithi003 ceph-mon[100615]: osdmap e328: 8 total, 8 up, 7 in 2024-08-17T17:39:02.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:02 smithi003 ceph-mon[103344]: osdmap e327: 8 total, 8 up, 7 in 2024-08-17T17:39:02.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:02 smithi003 ceph-mon[103344]: osdmap e328: 8 total, 8 up, 7 in 2024-08-17T17:39:02.869 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:02.893 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:02.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:02.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:02.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:02 smithi053 ceph-mon[89963]: osdmap e327: 8 total, 8 up, 7 in 2024-08-17T17:39:02.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:02 smithi053 ceph-mon[89963]: osdmap e328: 8 total, 8 up, 7 in 2024-08-17T17:39:02.995 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:03.021 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:03.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:03.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:03.123 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:03.159 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:03.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:03.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:03.262 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:03.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:03.294 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:03.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:03.340 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 73 16 103809 103793 88.839 4.5 0.00730981 0.0112504 2024-08-17T17:39:03.395 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:03.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:03.422 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:03.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:03.524 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:03.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:03.551 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:03.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:03.652 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:03.693 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:03.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:03.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:03.795 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:03.832 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:03.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:03.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:03.842 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:03 smithi003 ceph-mon[100615]: pgmap v1032: 109 pgs: 109 active+clean; 6.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 62 MiB/s wr, 987 op/s 2024-08-17T17:39:03.842 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:03 smithi003 ceph-mon[103344]: pgmap v1032: 109 pgs: 109 active+clean; 6.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 62 MiB/s wr, 987 op/s 2024-08-17T17:39:03.934 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:03.958 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:03.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:03.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:03.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:03 smithi053 ceph-mon[89963]: pgmap v1032: 109 pgs: 109 active+clean; 6.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 62 MiB/s wr, 987 op/s 2024-08-17T17:39:04.060 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:04.122 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:04.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:04.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:04.224 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:04.265 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:04.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:04.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:04.339 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 74 16 104536 104520 88.2525 45.4375 0.00811559 0.0113203 2024-08-17T17:39:04.368 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:04.403 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:39:04.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:04.408 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:04.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:04.509 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:04.533 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:04.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:04.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:04.591 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:39:04.635 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:04.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:04.707 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:04.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:04.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:04 smithi003 ceph-mon[103344]: osdmap e329: 8 total, 8 up, 7 in 2024-08-17T17:39:04.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:04 smithi003 ceph-mon[103344]: osdmap e330: 8 total, 8 up, 7 in 2024-08-17T17:39:04.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:04 smithi003 ceph-mon[100615]: osdmap e329: 8 total, 8 up, 7 in 2024-08-17T17:39:04.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:04 smithi003 ceph-mon[100615]: osdmap e330: 8 total, 8 up, 7 in 2024-08-17T17:39:04.808 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:04.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:04.834 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:04.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:04.935 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:04.958 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:04.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:04.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:04.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:04 smithi053 ceph-mon[89963]: osdmap e329: 8 total, 8 up, 7 in 2024-08-17T17:39:04.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:04 smithi053 ceph-mon[89963]: osdmap e330: 8 total, 8 up, 7 in 2024-08-17T17:39:05.060 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:05.085 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:05.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:05.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:05.187 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:05.213 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:05.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:05.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:05.315 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:05.339 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 75 16 105195 105179 87.625 41.1875 0.0395231 0.011406 2024-08-17T17:39:05.365 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:05.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:05.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:05.467 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:05.510 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:05.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:05.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:05.612 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:05.624 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:05 smithi003 ceph-mon[103344]: pgmap v1035: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 107 active+clean; 6.3 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 11 MiB/s wr, 183 op/s; 6477/309459 objects degraded (2.093%); 22 B/s, 0 objects/s recovering 2024-08-17T17:39:05.624 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:05 smithi003 ceph-mon[103344]: Health check failed: Degraded data redundancy: 6477/309459 objects degraded (2.093%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:39:05.624 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:05 smithi003 ceph-mon[103344]: 1.5 scrub starts 2024-08-17T17:39:05.624 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:05 smithi003 ceph-mon[100615]: pgmap v1035: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 107 active+clean; 6.3 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 11 MiB/s wr, 183 op/s; 6477/309459 objects degraded (2.093%); 22 B/s, 0 objects/s recovering 2024-08-17T17:39:05.624 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:05 smithi003 ceph-mon[100615]: Health check failed: Degraded data redundancy: 6477/309459 objects degraded (2.093%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:39:05.624 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:05 smithi003 ceph-mon[100615]: 1.5 scrub starts 2024-08-17T17:39:05.636 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:05.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:05.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:05.738 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:05.779 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:05.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:05.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:05.881 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:05.898 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:39:05.906 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:05.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:05.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:05.952 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:39:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:39:05] "GET /metrics HTTP/1.1" 200 36498 "" "Prometheus/2.51.0" 2024-08-17T17:39:05.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:05 smithi053 ceph-mon[89963]: pgmap v1035: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 peering, 107 active+clean; 6.3 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 11 MiB/s wr, 183 op/s; 6477/309459 objects degraded (2.093%); 22 B/s, 0 objects/s recovering 2024-08-17T17:39:05.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:05 smithi053 ceph-mon[89963]: Health check failed: Degraded data redundancy: 6477/309459 objects degraded (2.093%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:39:05.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:05 smithi053 ceph-mon[89963]: 1.5 scrub starts 2024-08-17T17:39:06.009 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:06.034 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:06.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:06.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:06.136 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:06.165 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:06.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:06.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:06.267 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:06.309 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:06.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:06.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:06.340 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 76 16 105923 105907 87.0708 45.5 0.0484262 0.0114788 2024-08-17T17:39:06.411 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:06.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:06.463 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:06.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:06.482 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-08-17T17:39:06.482 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:39:06.482 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-08-17T17:39:06.482 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-08-17T17:39:06.565 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:06.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:06.617 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:06.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:06.624 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-08-17T17:39:06.717 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:06.735 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:06 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3516391169' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:39:06.735 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:06 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:39:06.735 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:06 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3516391169' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:39:06.735 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:06 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:39:06.741 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:06.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:06.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:06.843 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:06.867 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:06 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3516391169' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:39:06.867 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:06 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:39:06.868 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:06.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:06.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:06.970 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:06.992 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:06.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:06.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:07.094 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:07.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:07.126 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:07.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:07.226 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:07.270 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:07.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:07.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:07.340 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 77 16 106578 106562 86.4717 40.9375 0.0120142 0.0115578 2024-08-17T17:39:07.372 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:07.409 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:07.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:07.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:07.510 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:07.541 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:07.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:07.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:07.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:07.679 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:07.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:07.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:07.781 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:07.808 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:07.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:07.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:07.910 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:07.936 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:07.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:07.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:07.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:07 smithi053 ceph-mon[89963]: pgmap v1037: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 peering, 101 active+clean; 6.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 30 MiB/s wr, 484 op/s; 12969/314082 objects degraded (4.129%); 6492/314082 objects misplaced (2.067%); 697 KiB/s, 2 keys/s, 18 objects/s recovering 2024-08-17T17:39:07.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:07 smithi053 ceph-mon[89963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:39:07.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:07 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:39:07.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:07 smithi053 ceph-mon[89963]: osdmap e331: 8 total, 8 up, 7 in 2024-08-17T17:39:08.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:07 smithi003 ceph-mon[103344]: pgmap v1037: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 peering, 101 active+clean; 6.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 30 MiB/s wr, 484 op/s; 12969/314082 objects degraded (4.129%); 6492/314082 objects misplaced (2.067%); 697 KiB/s, 2 keys/s, 18 objects/s recovering 2024-08-17T17:39:08.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:07 smithi003 ceph-mon[103344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:39:08.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:07 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:39:08.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:07 smithi003 ceph-mon[103344]: osdmap e331: 8 total, 8 up, 7 in 2024-08-17T17:39:08.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:07 smithi003 ceph-mon[100615]: pgmap v1037: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 peering, 101 active+clean; 6.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 30 MiB/s wr, 484 op/s; 12969/314082 objects degraded (4.129%); 6492/314082 objects misplaced (2.067%); 697 KiB/s, 2 keys/s, 18 objects/s recovering 2024-08-17T17:39:08.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:07 smithi003 ceph-mon[100615]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:39:08.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:07 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:39:08.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:07 smithi003 ceph-mon[100615]: osdmap e331: 8 total, 8 up, 7 in 2024-08-17T17:39:08.038 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:08.060 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:08.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:08.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:08.162 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:08.209 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:08.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:08.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:08.311 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:08.334 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:08.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:08.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:08.342 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 78 16 107278 107262 85.9211 43.75 0.0491478 0.0116332 2024-08-17T17:39:08.436 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:08.464 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:08.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:08.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:08.566 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:08.595 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:08.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:08.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:08.697 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:08.736 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:08.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:08.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:08.838 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:08.878 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:08.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:08.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:08.980 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:09.006 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:09.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:09.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:09.108 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:09.152 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:09.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:09.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:09.254 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:09.305 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:09.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:09.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:09.342 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 79 16 108002 107986 85.4063 45.25 0.00702462 0.0117012 2024-08-17T17:39:09.407 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:39:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:39:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:09 smithi053 ceph-mon[89963]: pgmap v1039: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 peering, 101 active+clean; 6.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 25 MiB/s wr, 399 op/s; 12969/314082 objects degraded (4.129%); 6492/314082 objects misplaced (2.067%); 574 KiB/s, 1 keys/s, 15 objects/s recovering 2024-08-17T17:39:09.472 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:09.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:09.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:39:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:39:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:09 smithi003 ceph-mon[103344]: pgmap v1039: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 peering, 101 active+clean; 6.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 25 MiB/s wr, 399 op/s; 12969/314082 objects degraded (4.129%); 6492/314082 objects misplaced (2.067%); 574 KiB/s, 1 keys/s, 15 objects/s recovering 2024-08-17T17:39:09.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:39:09.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:39:09.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:09 smithi003 ceph-mon[100615]: pgmap v1039: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 5 peering, 101 active+clean; 6.4 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 25 MiB/s wr, 399 op/s; 12969/314082 objects degraded (4.129%); 6492/314082 objects misplaced (2.067%); 574 KiB/s, 1 keys/s, 15 objects/s recovering 2024-08-17T17:39:09.574 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:09.597 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:09.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:09.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:09.699 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:09.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:09.729 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:09.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:09.830 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:09.859 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:09.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:09.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:09.961 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:09.994 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:09.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:09.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:10.097 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:10.141 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:10.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:10.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:10.243 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:10.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:10 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 12969/314082 objects degraded (4.129%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:39:10.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:10 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 12969/314082 objects degraded (4.129%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:39:10.274 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:10.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:10.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:10.343 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-08-17T17:39:10.343189+0000 min lat: 0.001429 max lat: 2.55533 avg lat: 0.0117716 2024-08-17T17:39:10.343 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:39:10.343 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 80 16 108727 108711 84.9052 45.3125 0.01148 0.0117716 2024-08-17T17:39:10.376 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:10.415 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:10.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:10.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:10.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:10 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 12969/314082 objects degraded (4.129%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:39:10.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:39:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:39:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:39:10.517 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:10.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:10.573 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:10.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:10.674 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:10.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:10.700 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:10.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:10.801 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:10.826 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:10.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:10.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:10.927 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:10.952 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:10.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:10.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:11.054 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:11.078 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:11.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:11.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:11.180 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:11.208 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:11.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:11.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:11.310 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:11.335 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:11.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:11.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:11.343 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 81 16 109469 109453 84.4295 46.375 0.0130297 0.0118371 2024-08-17T17:39:11.394 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:11 smithi003 ceph-mon[103344]: 1.5 scrub starts 2024-08-17T17:39:11.394 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:11 smithi003 ceph-mon[103344]: pgmap v1040: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.6 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 54 MiB/s wr, 859 op/s; 12969/324495 objects degraded (3.997%); 6394/324495 objects misplaced (1.970%); 1.4 MiB/s, 1 keys/s, 27 objects/s recovering 2024-08-17T17:39:11.394 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:11 smithi003 ceph-mon[100615]: 1.5 scrub starts 2024-08-17T17:39:11.395 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:11 smithi003 ceph-mon[100615]: pgmap v1040: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.6 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 54 MiB/s wr, 859 op/s; 12969/324495 objects degraded (3.997%); 6394/324495 objects misplaced (1.970%); 1.4 MiB/s, 1 keys/s, 27 objects/s recovering 2024-08-17T17:39:11.437 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:11.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:11 smithi053 ceph-mon[89963]: 1.5 scrub starts 2024-08-17T17:39:11.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:11 smithi053 ceph-mon[89963]: pgmap v1040: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.6 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 54 MiB/s wr, 859 op/s; 12969/324495 objects degraded (3.997%); 6394/324495 objects misplaced (1.970%); 1.4 MiB/s, 1 keys/s, 27 objects/s recovering 2024-08-17T17:39:11.475 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:11.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:11.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:11.576 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:11.603 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:11.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:11.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:11.705 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:11.731 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:11.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:11.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:11.833 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:11.870 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:11.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:11.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:11.972 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:11.997 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:11.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:11.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:12.100 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:12.123 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:12.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:12.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:12.183 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:39:12.225 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:12.244 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:39:12.244 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:39:12.251 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:12.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:12.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:12.343 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 82 16 110184 110168 83.9449 44.6875 0.0181531 0.0119064 2024-08-17T17:39:12.352 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:12.396 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:12.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:12.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:12.498 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:12.535 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:12.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:12.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:12.636 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:12.674 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:12.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:12.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:12.776 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:12.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:12.822 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:12.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:12.923 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:12.964 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:12.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:12.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:13.066 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:13.091 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:13.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:13.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:13.168 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:39:13.193 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:13.268 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:13.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:13.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:13.343 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 83 16 110851 110835 83.4358 41.6875 0.0215382 0.0119783 2024-08-17T17:39:13.370 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:13.396 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:13.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:13.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:13.459 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:13 smithi003 ceph-mon[103344]: pgmap v1041: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.6 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 39 MiB/s wr, 626 op/s; 12969/324495 objects degraded (3.997%); 6394/324495 objects misplaced (1.970%); 1.2 MiB/s, 1 keys/s, 23 objects/s recovering 2024-08-17T17:39:13.459 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:13 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1088022828' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:39:13.459 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:13 smithi003 ceph-mon[100615]: pgmap v1041: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.6 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 39 MiB/s wr, 626 op/s; 12969/324495 objects degraded (3.997%); 6394/324495 objects misplaced (1.970%); 1.2 MiB/s, 1 keys/s, 23 objects/s recovering 2024-08-17T17:39:13.459 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:13 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1088022828' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:39:13.498 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:13.521 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:13.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:13.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:13.552 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:39:13.552 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:39:13.552 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:39:13.552 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:39:13.553 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:39:13.553 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:39:13.553 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:39:13.553 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:39:13.553 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:39:13.553 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:39:13.553 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:39:13.553 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:39:13.553 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:39:13.553 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:39:13.554 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:39:13.554 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:39:13.622 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:13.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:13.668 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:13.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:13.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:13 smithi053 ceph-mon[89963]: pgmap v1041: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.6 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 39 MiB/s wr, 626 op/s; 12969/324495 objects degraded (3.997%); 6394/324495 objects misplaced (1.970%); 1.2 MiB/s, 1 keys/s, 23 objects/s recovering 2024-08-17T17:39:13.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:13 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1088022828' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:39:13.769 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:13.797 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:13.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:13.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:13.899 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:13.952 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:13.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:13.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:14.053 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:14.080 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:14.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:14.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:14.182 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:14.223 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:14.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:14.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:14.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:14.344 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 84 16 111581 111565 82.9849 45.625 0.00941735 0.0120446 2024-08-17T17:39:14.363 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:14.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:14.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:14.465 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:14.491 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:14.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:14.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:14.593 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:14.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:14.635 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:14.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:14.736 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:14.780 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:14.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:14.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:14.882 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:14.905 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-17T17:39:14.906 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:14.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:14.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:15.008 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:15.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:15.055 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:15.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:15.156 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:15.179 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:15.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:15.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:15.281 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:15.306 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:15.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:15.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:15.344 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 85 15 112303 112288 82.5403 45.1875 0.0522895 0.012108 2024-08-17T17:39:15.407 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:15.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:15 smithi003 ceph-mon[100615]: pgmap v1042: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.7 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 36 MiB/s wr, 577 op/s; 12969/327897 objects degraded (3.955%); 6394/327897 objects misplaced (1.950%); 985 KiB/s, 17 objects/s recovering 2024-08-17T17:39:15.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:15 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 12969/327897 objects degraded (3.955%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:39:15.437 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:15 smithi003 ceph-mon[103344]: pgmap v1042: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.7 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 36 MiB/s wr, 577 op/s; 12969/327897 objects degraded (3.955%); 6394/327897 objects misplaced (1.950%); 985 KiB/s, 17 objects/s recovering 2024-08-17T17:39:15.437 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:15 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 12969/327897 objects degraded (3.955%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:39:15.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:15.452 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:15.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:15.554 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:15.570 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:39:15.578 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:15.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:15.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:15.679 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:15.702 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:15.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:15.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:15.705 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:39:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:39:15] "GET /metrics HTTP/1.1" 200 36524 "" "Prometheus/2.51.0" 2024-08-17T17:39:15.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:15 smithi053 ceph-mon[89963]: pgmap v1042: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.7 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 36 MiB/s wr, 577 op/s; 12969/327897 objects degraded (3.955%); 6394/327897 objects misplaced (1.950%); 985 KiB/s, 17 objects/s recovering 2024-08-17T17:39:15.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:15 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 12969/327897 objects degraded (3.955%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:39:15.805 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:15.863 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:15.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:15.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:15.965 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:15.993 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:15.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:15.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:16.094 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:16.120 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:16.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:16.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:16.222 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:16.265 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:16.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:16.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:16.346 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 86 16 113010 112994 82.0937 44.125 0.00910716 0.0121744 2024-08-17T17:39:16.353 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:39:16.367 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:16.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:16.421 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:16.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:16.522 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:16.547 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:16.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:16.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:16.649 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:16.689 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:16.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:16.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:16.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:16 smithi053 ceph-mon[89963]: 1.5 scrub starts 2024-08-17T17:39:16.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:16 smithi003 ceph-mon[103344]: 1.5 scrub starts 2024-08-17T17:39:16.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:16 smithi003 ceph-mon[100615]: 1.5 scrub starts 2024-08-17T17:39:16.792 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:16.831 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:16.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:16.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:16.933 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:16.969 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:16.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:16.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:17.071 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:17.116 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:17.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:17.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:17.160 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-08-17T17:39:17.217 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:17.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:17.270 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:17.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:17.344 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 87 16 113740 113724 81.6745 45.625 0.0109432 0.0122377 2024-08-17T17:39:17.372 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:17.399 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:17.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:17.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:17.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:17.513 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:39:17.562 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:17.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:17.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:17.570 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:17 smithi003 ceph-mon[100615]: pgmap v1043: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 44 MiB/s wr, 708 op/s; 12969/335358 objects degraded (3.867%); 6294/335358 objects misplaced (1.877%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:39:17.573 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:17 smithi003 ceph-mon[103344]: pgmap v1043: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 44 MiB/s wr, 708 op/s; 12969/335358 objects degraded (3.867%); 6294/335358 objects misplaced (1.877%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:39:17.664 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:17.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:17.713 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:17.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:17.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:17 smithi053 ceph-mon[89963]: pgmap v1043: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 44 MiB/s wr, 708 op/s; 12969/335358 objects degraded (3.867%); 6294/335358 objects misplaced (1.877%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:39:17.814 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:17.841 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:17.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:17.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:17.942 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:17.966 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:17.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:17.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:18.068 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:18.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:18.095 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:18.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:18.196 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:18.221 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:18.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:18.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:18.323 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:18.344 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 88 16 114474 114458 81.2677 45.875 0.079684 0.0122989 2024-08-17T17:39:18.348 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:18.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:18.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:18.450 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:18.478 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:18.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:18.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:18.579 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:18.629 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:18.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:18.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:18.731 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:18.757 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:18.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:18.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:18.858 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:18.938 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:18.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:18.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:19.040 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:19.099 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:19.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:19.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:19.201 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:19.281 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:19.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:19.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:19.344 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 89 16 115192 115176 80.8589 44.875 0.0394827 0.0123584 2024-08-17T17:39:19.345 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:39:19.383 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:19.426 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:19.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:19.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:19.528 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:19.575 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:19.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:19.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:19.606 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:19 smithi003 ceph-mon[103344]: pgmap v1044: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 38 MiB/s wr, 609 op/s; 12969/335358 objects degraded (3.867%); 6294/335358 objects misplaced (1.877%); 1.1 MiB/s, 17 objects/s recovering 2024-08-17T17:39:19.607 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:19 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:39:19.607 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:19 smithi003 ceph-mon[100615]: pgmap v1044: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 38 MiB/s wr, 609 op/s; 12969/335358 objects degraded (3.867%); 6294/335358 objects misplaced (1.877%); 1.1 MiB/s, 17 objects/s recovering 2024-08-17T17:39:19.607 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:19 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:39:19.678 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:19.704 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:19.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:19.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:19.806 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:19.831 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:19.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:19.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:19.933 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:19.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:19 smithi053 ceph-mon[89963]: pgmap v1044: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 38 MiB/s wr, 609 op/s; 12969/335358 objects degraded (3.867%); 6294/335358 objects misplaced (1.877%); 1.1 MiB/s, 17 objects/s recovering 2024-08-17T17:39:19.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:19 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:39:19.974 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:19.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:19.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:20.076 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:20.102 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:20.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:20.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:20.204 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:20.248 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:20.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:20.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:20.345 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total time run: 90.0121 2024-08-17T17:39:20.345 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total writes made: 115882 2024-08-17T17:39:20.345 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Write size: 65536 2024-08-17T17:39:20.345 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object size: 65536 2024-08-17T17:39:20.345 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Bandwidth (MB/sec): 80.4628 2024-08-17T17:39:20.345 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Bandwidth: 38.5324 2024-08-17T17:39:20.345 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max bandwidth (MB/sec): 205 2024-08-17T17:39:20.345 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min bandwidth (MB/sec): 0 2024-08-17T17:39:20.345 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average IOPS: 1287 2024-08-17T17:39:20.345 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev IOPS: 616.519 2024-08-17T17:39:20.345 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max IOPS: 3280 2024-08-17T17:39:20.346 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min IOPS: 0 2024-08-17T17:39:20.346 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average Latency(s): 0.0124231 2024-08-17T17:39:20.346 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Latency(s): 0.0312838 2024-08-17T17:39:20.346 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max latency(s): 2.55533 2024-08-17T17:39:20.346 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min latency(s): 0.001429 2024-08-17T17:39:20.350 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:20.351 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Cleaning up (deleting benchmark objects) 2024-08-17T17:39:20.376 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:20.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:20.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:20.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:39:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:39:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:39:20.478 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:20.519 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:20.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:20.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:20.550 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-08-17T17:39:20.621 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:20.659 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:20.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:20.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:20.760 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:20.813 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:20.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:20.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:20.903 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:39:20.915 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:20.938 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:20.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:20.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:20.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:20 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 12969/335358 objects degraded (3.867%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:39:20.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:20 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2851655046' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:39:20.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:20 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:39:21.012 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:20 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 12969/335358 objects degraded (3.867%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:39:21.012 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:20 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2851655046' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:39:21.012 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:20 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:39:21.012 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:20 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 12969/335358 objects degraded (3.867%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:39:21.012 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:20 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2851655046' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:39:21.013 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:20 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:39:21.041 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:21.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:21.064 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:21.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:21.165 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:21.189 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:21.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:21.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:21.291 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:21.334 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:21.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:21.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:21.436 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:21.480 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:21.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:21.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:21.582 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:21.610 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:21.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:21.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:21.712 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:21.735 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:21.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:21.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:21.837 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:21.852 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:21 smithi003 ceph-mon[100615]: pgmap v1045: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 56 MiB/s wr, 894 op/s; 12969/346284 objects degraded (3.745%); 6197/346284 objects misplaced (1.790%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:39:21.852 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:21 smithi003 ceph-mon[100615]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:39:21.853 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:21 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:39:21.853 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:21 smithi003 ceph-mon[100615]: osdmap e332: 8 total, 8 up, 7 in 2024-08-17T17:39:21.853 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:21 smithi003 ceph-mon[100615]: 1.5 scrub starts 2024-08-17T17:39:21.853 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:21 smithi003 ceph-mon[100615]: 1.5 scrub ok 2024-08-17T17:39:21.853 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:21 smithi003 ceph-mon[103344]: pgmap v1045: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 56 MiB/s wr, 894 op/s; 12969/346284 objects degraded (3.745%); 6197/346284 objects misplaced (1.790%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:39:21.853 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:21 smithi003 ceph-mon[103344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:39:21.853 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:21 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:39:21.853 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:21 smithi003 ceph-mon[103344]: osdmap e332: 8 total, 8 up, 7 in 2024-08-17T17:39:21.853 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:21 smithi003 ceph-mon[103344]: 1.5 scrub starts 2024-08-17T17:39:21.853 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:21 smithi003 ceph-mon[103344]: 1.5 scrub ok 2024-08-17T17:39:21.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:21.887 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:21.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:21.915 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:39:21.915 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-08-17T17:39:21.915 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-08-17T17:39:21.915 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-08-17T17:39:21.915 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:39:21.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:21 smithi053 ceph-mon[89963]: pgmap v1045: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 56 MiB/s wr, 894 op/s; 12969/346284 objects degraded (3.745%); 6197/346284 objects misplaced (1.790%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:39:21.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:21 smithi053 ceph-mon[89963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:39:21.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:21 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:39:21.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:21 smithi053 ceph-mon[89963]: osdmap e332: 8 total, 8 up, 7 in 2024-08-17T17:39:21.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:21 smithi053 ceph-mon[89963]: 1.5 scrub starts 2024-08-17T17:39:21.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:21 smithi053 ceph-mon[89963]: 1.5 scrub ok 2024-08-17T17:39:21.988 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:22.010 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:22.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:22.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:22.112 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:22.141 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:22.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:22.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:22.242 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:22.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:22.283 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:22.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:22.384 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:22.422 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:22.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:22.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:22.524 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:22.550 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:22.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:22.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:22.651 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:22.695 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:22.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:22.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:22.797 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:22.841 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:22.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:22.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:22.943 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:23.009 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:23.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:23.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:23.111 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:23.162 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:23.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:23.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:23.263 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:23.300 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:23.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:23.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:23.402 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:23.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:23.430 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:23.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:23.531 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:23.560 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:23.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:23.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:23.662 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:23.709 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:23.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:23.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:23.807 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:23 smithi003 ceph-mon[103344]: pgmap v1047: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 45 MiB/s wr, 725 op/s; 12969/346284 objects degraded (3.745%); 6197/346284 objects misplaced (1.790%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:39:23.807 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:39:23.807 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:23 smithi003 ceph-mon[100615]: pgmap v1047: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 45 MiB/s wr, 725 op/s; 12969/346284 objects degraded (3.745%); 6197/346284 objects misplaced (1.790%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:39:23.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:39:23.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:23.851 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:23.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:23.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:23.953 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:23.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:23 smithi053 ceph-mon[89963]: pgmap v1047: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 45 MiB/s wr, 725 op/s; 12969/346284 objects degraded (3.745%); 6197/346284 objects misplaced (1.790%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:39:23.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:39:23.982 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:23.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:23.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:24.013 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:39:24.013 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:39:24.050 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:39:24.050 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:39:24.083 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:24.086 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:39:24.087 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:39:24.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:24.127 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:24.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:24.132 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:39:24.132 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:39:24.167 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:39:24.168 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:39:24.199 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:39:24.199 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:39:24.228 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:24.236 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:39:24.236 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:39:24.269 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:39:24.270 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:39:24.272 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:24.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:24.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:24.374 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:24.399 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:24.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:24.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:24.501 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:24.525 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:24.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:24.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:24.627 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:24.666 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:24.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:24.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:24.767 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:24.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:24 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1799615452' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:39:24.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:24 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1799615452' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:39:24.792 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:24.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:24.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:24.894 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:24.929 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:24.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:24.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:24.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:24 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1799615452' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:39:25.021 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-17T17:39:25.032 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:25.057 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:25.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:25.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:25.159 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:25.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:25.206 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:25.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:25.307 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:25.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:25.334 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:25.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:25.438 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:25.482 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:25.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:25.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:25.584 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:25.622 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:25.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:25.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:25.654 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:25 smithi003 ceph-mon[103344]: pgmap v1048: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 40 MiB/s wr, 726 op/s; 12969/344523 objects degraded (3.764%); 6197/344523 objects misplaced (1.799%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:39:25.654 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:25 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 12969/344523 objects degraded (3.764%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:39:25.655 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:25 smithi003 ceph-mon[100615]: pgmap v1048: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 40 MiB/s wr, 726 op/s; 12969/344523 objects degraded (3.764%); 6197/344523 objects misplaced (1.799%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:39:25.655 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:25 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 12969/344523 objects degraded (3.764%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:39:25.724 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:25.735 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:39:25.759 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:25.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:25.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:25.861 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:25.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:25.923 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:25.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:25.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:25 smithi053 ceph-mon[89963]: pgmap v1048: 109 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 40 MiB/s wr, 726 op/s; 12969/344523 objects degraded (3.764%); 6197/344523 objects misplaced (1.799%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:39:25.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:25 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 12969/344523 objects degraded (3.764%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:39:25.997 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:39:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:39:25] "GET /metrics HTTP/1.1" 200 36522 "" "Prometheus/2.51.0" 2024-08-17T17:39:26.024 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:26.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:26.069 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:26.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:26.170 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:26.194 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:26.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:26.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:26.296 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:26.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:26.336 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:26.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:26.436 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:39:26.438 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:26.461 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:26.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:26.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:26.520 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:39:26.562 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:26.632 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:26.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:26.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:26.733 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:26.761 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:26.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:26.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:26.863 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:26.913 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:26.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:26.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:27.015 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:27.038 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:27.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:27.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:27.140 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:27.163 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:27.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:27.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:27.264 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:27.306 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:27.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:27.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:27.410 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:27.443 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:27.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:27.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:27.545 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:27.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:27.567 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:27.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:27.668 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:27.692 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:27.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:27.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:27.793 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:27.817 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:27.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:27.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:27.919 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:27.951 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:27.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:27.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:27.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:27 smithi053 ceph-mon[89963]: pgmap v1049: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.9 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 102 B/s rd, 26 MiB/s wr, 735 op/s; 12969/338037 objects degraded (3.837%); 6092/338037 objects misplaced (1.802%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:39:28.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:27 smithi003 ceph-mon[103344]: pgmap v1049: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.9 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 102 B/s rd, 26 MiB/s wr, 735 op/s; 12969/338037 objects degraded (3.837%); 6092/338037 objects misplaced (1.802%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:39:28.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:27 smithi003 ceph-mon[100615]: pgmap v1049: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.9 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 102 B/s rd, 26 MiB/s wr, 735 op/s; 12969/338037 objects degraded (3.837%); 6092/338037 objects misplaced (1.802%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:39:28.053 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:28.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:28.098 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:28.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:28.199 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:28.223 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:28.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:28.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:28.325 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:28.352 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:28.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:28.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:28.383 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:39:28.454 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:28.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:28.494 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:28.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:28.595 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:28.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:28.618 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:28.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:28.718 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:28.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:28.754 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:28.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:28.855 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:28.877 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:28.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:28.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:28.979 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:29.020 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:29.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:29.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:29.122 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:29.153 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:29.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:29.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:29.255 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:29.280 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:29.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:29.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:29.381 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:29.405 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:29.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:29.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:29.507 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:29.546 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:29.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:29.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:29.648 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:29.672 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:29.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:29.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:29.774 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:29.799 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:29.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:29.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:29.845 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:29 smithi003 ceph-mon[103344]: pgmap v1050: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.9 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 102 B/s rd, 26 MiB/s wr, 735 op/s; 12969/338037 objects degraded (3.837%); 6092/338037 objects misplaced (1.802%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:39:29.846 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:29 smithi003 ceph-mon[100615]: pgmap v1050: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.9 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 102 B/s rd, 26 MiB/s wr, 735 op/s; 12969/338037 objects degraded (3.837%); 6092/338037 objects misplaced (1.802%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:39:29.901 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:29.947 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:29.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:29.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:29.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:29 smithi053 ceph-mon[89963]: pgmap v1050: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.9 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 102 B/s rd, 26 MiB/s wr, 735 op/s; 12969/338037 objects degraded (3.837%); 6092/338037 objects misplaced (1.802%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:39:30.048 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:30.073 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:30.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:30.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:30.175 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:30.211 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:30.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:30.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:30.312 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:30.354 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:30.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:30.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:30.456 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:30.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:39:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:39:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:39:30.481 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:30.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:30.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:30.583 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:30.608 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:30.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:30.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:30.710 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:30.747 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:30.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:30.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:30.849 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:30.874 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:30.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:30.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:30.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:30 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 12969/338037 objects degraded (3.837%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:39:30.977 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:31.001 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:31.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:31.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:31.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:30 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 12969/338037 objects degraded (3.837%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:39:31.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:30 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 12969/338037 objects degraded (3.837%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:39:31.103 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:31.130 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:31.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:31.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:31.231 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:31.267 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:31.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:31.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:31.369 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:31.395 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:31.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:31.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:31.498 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:31.519 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:31.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:31.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:31.621 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:31.647 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:31.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:31.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:31.749 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:31.783 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:31.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:31.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:31.885 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:31.911 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:31.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:31.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:31.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:31 smithi053 ceph-mon[89963]: pgmap v1051: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 102 B/s rd, 3.0 MiB/s wr, 555 op/s; 12969/332502 objects degraded (3.900%); 6092/332502 objects misplaced (1.832%); 672 KiB/s, 10 objects/s recovering 2024-08-17T17:39:32.013 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:32.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:31 smithi003 ceph-mon[100615]: pgmap v1051: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 102 B/s rd, 3.0 MiB/s wr, 555 op/s; 12969/332502 objects degraded (3.900%); 6092/332502 objects misplaced (1.832%); 672 KiB/s, 10 objects/s recovering 2024-08-17T17:39:32.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:31 smithi003 ceph-mon[103344]: pgmap v1051: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 102 B/s rd, 3.0 MiB/s wr, 555 op/s; 12969/332502 objects degraded (3.900%); 6092/332502 objects misplaced (1.832%); 672 KiB/s, 10 objects/s recovering 2024-08-17T17:39:32.048 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:32.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:32.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:32.149 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:32.173 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:32.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:32.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:32.274 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:32.309 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:32.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:32.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:32.411 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:32.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:32.452 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:32.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:32.553 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:32.578 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:32.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:32.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:32.680 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:32.719 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:32.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:32.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:32.821 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:32.855 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:32.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:32.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:32.956 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:32.983 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:32.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:32.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:33.085 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:33.111 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:33.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:33.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:33.214 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:33.250 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:33.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:33.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:33.352 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:33.378 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:33.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:33.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:33.480 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:33.506 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:33.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:33.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:33.607 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:33.633 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:33.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:33.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:33.735 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:33.771 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:33.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:33.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:33.873 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:33.898 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:33.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:33.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:33.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:33 smithi053 ceph-mon[89963]: pgmap v1052: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.7 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 87 B/s rd, 2.6 MiB/s wr, 604 op/s; 12969/327990 objects degraded (3.954%); 5990/327990 objects misplaced (1.826%); 1.1 MiB/s, 17 objects/s recovering 2024-08-17T17:39:33.992 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:33 smithi003 ceph-mon[103344]: pgmap v1052: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.7 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 87 B/s rd, 2.6 MiB/s wr, 604 op/s; 12969/327990 objects degraded (3.954%); 5990/327990 objects misplaced (1.826%); 1.1 MiB/s, 17 objects/s recovering 2024-08-17T17:39:33.992 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:33 smithi003 ceph-mon[100615]: pgmap v1052: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.7 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 87 B/s rd, 2.6 MiB/s wr, 604 op/s; 12969/327990 objects degraded (3.954%); 5990/327990 objects misplaced (1.826%); 1.1 MiB/s, 17 objects/s recovering 2024-08-17T17:39:34.000 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:34.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:34.044 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:34.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:34.145 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:34.171 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:34.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:34.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:34.273 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:34.295 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:34.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:34.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:34.397 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:34.422 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:34.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:34.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:34.523 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:34.562 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:34.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:34.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:34.665 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:34.691 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:34.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:34.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:34.793 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:34.816 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:34.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:34.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:34.918 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:34.955 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:34.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:34.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:35.057 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:35.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:35.096 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:35.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:35.197 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:35.222 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:35.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:35.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:35.323 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:35.369 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:35.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:35.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:35.471 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:35.493 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:35.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:35.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:35.595 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:35.616 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:35.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:35.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:35.718 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:35.754 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:35.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:35.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:35.856 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:35.880 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:35.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:35.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:35.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:35 smithi053 ceph-mon[89963]: pgmap v1053: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 85 B/s rd, 2.5 MiB/s wr, 682 op/s; 12969/324603 objects degraded (3.995%); 5990/324603 objects misplaced (1.845%); 1.1 MiB/s, 17 objects/s recovering 2024-08-17T17:39:35.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:35 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 12969/324603 objects degraded (3.995%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:39:35.982 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:36.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:35 smithi003 ceph-mon[103344]: pgmap v1053: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 85 B/s rd, 2.5 MiB/s wr, 682 op/s; 12969/324603 objects degraded (3.995%); 5990/324603 objects misplaced (1.845%); 1.1 MiB/s, 17 objects/s recovering 2024-08-17T17:39:36.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:35 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 12969/324603 objects degraded (3.995%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:39:36.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:35 smithi003 ceph-mon[100615]: pgmap v1053: 109 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 85 B/s rd, 2.5 MiB/s wr, 682 op/s; 12969/324603 objects degraded (3.995%); 5990/324603 objects misplaced (1.845%); 1.1 MiB/s, 17 objects/s recovering 2024-08-17T17:39:36.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:35 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 12969/324603 objects degraded (3.995%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:39:36.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:39:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:39:35] "GET /metrics HTTP/1.1" 200 36522 "" "Prometheus/2.51.0" 2024-08-17T17:39:36.025 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:36.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:36.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:36.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:36.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:36.169 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:36.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:36.270 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:36.296 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:36.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:36.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:36.398 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:36.424 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:36.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:36.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:36.526 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:36.552 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:36.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:36.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:36.654 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:36.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:36.682 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:36.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:36.783 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:36.808 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:36.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:36.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:36.910 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:36.950 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:36.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:36.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:37.052 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:37.092 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:37.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:37.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:37.194 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:37.221 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:37.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:37.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:37.322 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:37.349 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:37.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:37.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:37.450 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:37.475 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:37.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:37.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:37.577 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:37.602 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:37.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:37.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:37.704 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:37.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:37.740 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:37.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:37.841 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:37.862 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:37 smithi003 ceph-mon[103344]: pgmap v1054: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 85 B/s rd, 1.1 MiB/s wr, 798 op/s; 12973/317031 objects degraded (4.092%); 5886/317031 objects misplaced (1.857%); 1.6 MiB/s, 25 objects/s recovering 2024-08-17T17:39:37.863 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:37 smithi003 ceph-mon[100615]: pgmap v1054: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 85 B/s rd, 1.1 MiB/s wr, 798 op/s; 12973/317031 objects degraded (4.092%); 5886/317031 objects misplaced (1.857%); 1.6 MiB/s, 25 objects/s recovering 2024-08-17T17:39:37.883 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:37.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:37.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:37.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:37 smithi053 ceph-mon[89963]: pgmap v1054: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 85 B/s rd, 1.1 MiB/s wr, 798 op/s; 12973/317031 objects degraded (4.092%); 5886/317031 objects misplaced (1.857%); 1.6 MiB/s, 25 objects/s recovering 2024-08-17T17:39:37.984 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:38.041 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:38.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:38.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:38.143 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:38.169 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:38.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:38.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:38.270 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:38.297 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:38.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:38.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:38.399 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:38.421 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:38.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:38.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:38.523 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:38.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:38.578 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:38.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:38.677 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:39:38.678 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:39:38.680 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:38.733 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:38.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:38.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:38.835 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:38.858 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:38.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:38.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:38.960 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:38.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:39:39.010 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:39.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:39.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:39.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:39.151 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:39.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:39.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:39.253 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:39.289 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:39.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:39.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:39.391 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:39.431 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:39.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:39.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:39.533 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:39.556 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:39.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:39.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:39.631 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:39:39.643 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-08-17T17:39:39.658 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:39.681 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:39.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:39.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:39.782 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:39.811 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:39.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:39.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:39.912 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:39.938 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:39.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:39.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:39.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:39 smithi053 ceph-mon[89963]: pgmap v1055: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 583 op/s; 12973/317031 objects degraded (4.092%); 5886/317031 objects misplaced (1.857%); 1.1 MiB/s, 16 objects/s recovering 2024-08-17T17:39:39.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:39 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2953080118' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:39:40.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:39 smithi003 ceph-mon[103344]: pgmap v1055: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 583 op/s; 12973/317031 objects degraded (4.092%); 5886/317031 objects misplaced (1.857%); 1.1 MiB/s, 16 objects/s recovering 2024-08-17T17:39:40.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:39 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2953080118' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:39:40.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:39 smithi003 ceph-mon[100615]: pgmap v1055: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 583 op/s; 12973/317031 objects degraded (4.092%); 5886/317031 objects misplaced (1.857%); 1.1 MiB/s, 16 objects/s recovering 2024-08-17T17:39:40.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:39 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2953080118' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:39:40.040 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:40.063 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:40.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:40.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:40.165 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:40.190 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:40.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:40.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:40.292 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:40.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:40.331 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:40.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:40.432 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:40.458 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:40.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:40.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:40.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:39:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:39:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:39:40.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:40.594 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:40.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:40.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:40.696 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:40.721 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:40.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:40.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:40.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:40 smithi003 ceph-mon[103344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:39:40.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:40 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2953080118' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:39:40.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:40 smithi003 ceph-mon[103344]: osdmap e333: 8 total, 8 up, 7 in 2024-08-17T17:39:40.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:40 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 12973/317031 objects degraded (4.092%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:39:40.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:40 smithi003 ceph-mon[100615]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:39:40.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:40 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2953080118' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:39:40.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:40 smithi003 ceph-mon[100615]: osdmap e333: 8 total, 8 up, 7 in 2024-08-17T17:39:40.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:40 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 12973/317031 objects degraded (4.092%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:39:40.823 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:40.863 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:40.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:40.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:40.965 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:40.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:40 smithi053 ceph-mon[89963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:39:40.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:40 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2953080118' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:39:40.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:40 smithi053 ceph-mon[89963]: osdmap e333: 8 total, 8 up, 7 in 2024-08-17T17:39:40.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:40 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 12973/317031 objects degraded (4.092%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:39:41.004 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:41.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:41.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:41.106 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:41.151 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:41.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:41.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:41.253 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:41.295 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:41.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:41.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:41.397 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:41.441 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:41.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:41.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:41.543 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:41.569 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:41.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:41.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:41.671 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:41.696 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:41.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:41.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:41.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:41 smithi003 ceph-mon[103344]: pgmap v1057: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 713 op/s; 12973/311100 objects degraded (4.170%); 5886/311100 objects misplaced (1.892%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:39:41.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:41 smithi003 ceph-mon[100615]: pgmap v1057: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 713 op/s; 12973/311100 objects degraded (4.170%); 5886/311100 objects misplaced (1.892%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:39:41.797 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:41.840 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:41.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:41.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:41.907 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:39:41.908 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:39:41.942 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:41.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:41 smithi053 ceph-mon[89963]: pgmap v1057: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 713 op/s; 12973/311100 objects degraded (4.170%); 5886/311100 objects misplaced (1.892%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:39:41.986 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:41.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:41.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:42.087 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:42.113 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:42.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:42.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:42.215 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:42.254 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:42.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:42.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:42.356 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:42.393 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:42.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:42.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:42.496 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:42.521 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:42.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:42.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:42.623 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:42.633 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-08-17T17:39:42.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:42.652 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:42.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:42.753 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:42.778 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:42.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:42.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:42.880 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:42.905 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:42.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:42.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:43.007 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:43.061 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:43.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:43.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:43.163 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:43.219 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:43.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:43.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:43.321 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:43.371 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:43.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:43.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:43.472 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:43.514 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:43.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:43.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:43.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:43.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:43.664 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:43.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:43.701 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-17T17:39:43.765 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:43.801 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:43.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:43.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:43.903 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:43.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:43.945 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:43.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:43.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:43 smithi053 ceph-mon[89963]: pgmap v1058: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 719 op/s; 13087/306414 objects degraded (4.271%); 5671/306414 objects misplaced (1.851%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:39:44.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:43 smithi003 ceph-mon[103344]: pgmap v1058: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 719 op/s; 13087/306414 objects degraded (4.271%); 5671/306414 objects misplaced (1.851%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:39:44.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:43 smithi003 ceph-mon[100615]: pgmap v1058: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 719 op/s; 13087/306414 objects degraded (4.271%); 5671/306414 objects misplaced (1.851%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:39:44.034 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:39:44.046 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:44.071 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:44.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:44.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:44.174 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:44.204 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:44.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:44.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:44.306 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:44.377 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:44.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:44.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:44.479 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:44.503 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:44.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:44.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:44.547 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:39:44.605 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:44.618 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:39:44.618 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:39:44.627 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:44.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:44.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:44.640 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:39:44.640 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:39:44.661 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:39:44.662 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:39:44.685 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:39:44.685 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:39:44.701 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:39:44.701 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:39:44.717 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:39:44.718 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:39:44.729 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:44.733 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:39:44.733 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:39:44.751 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:44.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:44.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:44.785 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:39:44.785 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:39:44.852 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:44.871 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:44 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3530180040' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:39:44.872 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:44 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3530180040' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:39:44.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:44.878 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:44.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:44.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:44 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3530180040' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:39:44.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:45.028 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:45.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:45.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:45.130 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:45.155 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:45.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:45.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:45.256 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:45.299 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:45.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:45.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:45.401 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:45.434 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:45.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:45.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:45.536 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:45.575 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:45.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:45.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:45.677 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:45.716 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:45.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:45.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:45.817 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:45.860 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:45.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:45.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:45.962 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:45.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:45 smithi053 ceph-mon[89963]: pgmap v1059: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.2 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 727 op/s; 13087/302781 objects degraded (4.322%); 5671/302781 objects misplaced (1.873%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:39:45.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:45 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 13087/302781 objects degraded (4.322%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:39:45.993 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:45.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:45.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:46.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:45 smithi003 ceph-mon[103344]: pgmap v1059: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.2 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 727 op/s; 13087/302781 objects degraded (4.322%); 5671/302781 objects misplaced (1.873%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:39:46.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:45 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 13087/302781 objects degraded (4.322%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:39:46.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:45 smithi003 ceph-mon[100615]: pgmap v1059: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.2 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 727 op/s; 13087/302781 objects degraded (4.322%); 5671/302781 objects misplaced (1.873%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:39:46.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:45 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 13087/302781 objects degraded (4.322%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:39:46.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:39:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:39:45] "GET /metrics HTTP/1.1" 200 36528 "" "Prometheus/2.51.0" 2024-08-17T17:39:46.080 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:39:46.094 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:46.119 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:46.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:46.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:46.222 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:46.260 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:39:46.261 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2024-08-17T17:39:46.261 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 0 2024-08-17T17:39:46.261 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:39:46.275 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:46.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:46.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:46.377 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:46.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:46.406 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:46.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:46.508 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:46.550 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:46.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:46.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:46.651 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:46.691 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:46.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:46.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:46.792 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:46.818 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:46.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:46.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:46.920 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:46.929 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-17T17:39:46.945 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:46.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:46.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:47.047 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:47.073 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:47.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:47.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:47.175 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:47.213 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:47.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:47.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:47.289 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:39:47.316 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:47.357 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:47.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:47.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:47.459 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:47.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:47.501 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:47.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:47.602 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:47.647 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:47.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:47.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:47.748 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:47.787 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:47.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:47.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:47.889 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:47.937 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:47.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:47.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:47.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:47 smithi053 ceph-mon[89963]: pgmap v1060: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 738 op/s; 13242/294879 objects degraded (4.491%); 5409/294879 objects misplaced (1.834%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:39:47.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:47 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:39:48.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:47 smithi003 ceph-mon[103344]: pgmap v1060: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 738 op/s; 13242/294879 objects degraded (4.491%); 5409/294879 objects misplaced (1.834%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:39:48.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:47 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:39:48.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:47 smithi003 ceph-mon[100615]: pgmap v1060: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 738 op/s; 13242/294879 objects degraded (4.491%); 5409/294879 objects misplaced (1.834%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:39:48.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:47 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:39:48.039 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:48.078 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:48.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:48.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:48.091 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:39:48.180 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:48.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:48.207 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:48.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:48.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:48.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:48.358 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:48.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:48.459 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:48.516 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:48.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:48.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:48.618 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:48.645 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:48.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:48.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:48.746 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:48.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:48.769 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:48.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:48.870 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:48.895 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:48.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:48.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:48.997 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:48.998 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:39:48.998 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2024-08-17T17:39:48.999 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2024-08-17T17:39:48.999 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2024-08-17T17:39:48.999 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:39:49.042 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:49.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:49.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:49.144 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:49.169 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:49.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:49.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:49.271 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:49.298 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:49.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:49.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:49.400 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:49.426 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:49.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:49.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:49.528 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:49.571 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:49.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:49.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:49.673 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:49.715 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:49.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:49.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:49.718 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-17T17:39:49.817 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:49.869 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:49.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:49.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:49.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:49 smithi053 ceph-mon[89963]: pgmap v1061: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 738 op/s; 13242/294879 objects degraded (4.491%); 5409/294879 objects misplaced (1.834%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:39:49.972 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:49.998 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:49.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:49.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:50.021 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:49 smithi003 ceph-mon[103344]: pgmap v1061: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 738 op/s; 13242/294879 objects degraded (4.491%); 5409/294879 objects misplaced (1.834%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:39:50.021 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:49 smithi003 ceph-mon[100615]: pgmap v1061: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 738 op/s; 13242/294879 objects degraded (4.491%); 5409/294879 objects misplaced (1.834%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:39:50.100 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:50.144 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:50.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:50.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:50.246 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:50.273 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:50.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:50.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:50.375 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:50.431 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:50.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:50.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:50.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:39:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:39:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:39:50.533 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:50.575 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:50.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:50.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:50.676 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:50.708 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:50.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:50.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:50.755 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:39:50.809 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:50.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:50.846 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:50.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:50.889 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:50 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 13242/294879 objects degraded (4.491%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:39:50.947 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:50.972 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:50.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:50.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:51.026 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:50 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 13242/294879 objects degraded (4.491%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:39:51.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:50 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 13242/294879 objects degraded (4.491%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:39:51.074 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:51.116 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:51.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:51.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:51.218 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:51.243 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:51.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:51.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:51.345 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:51.385 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:51.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:51.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:51.486 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:51.513 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:51.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:51.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:51.615 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:51.640 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:51.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:51.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:51.742 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:51.775 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:51.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:51.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:51.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:51.907 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:51.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:51.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:51.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:51 smithi053 ceph-mon[89963]: pgmap v1062: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 691 op/s; 13242/289104 objects degraded (4.580%); 5409/289104 objects misplaced (1.871%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:39:52.009 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:52.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:51 smithi003 ceph-mon[100615]: pgmap v1062: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 691 op/s; 13242/289104 objects degraded (4.580%); 5409/289104 objects misplaced (1.871%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:39:52.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:51 smithi003 ceph-mon[103344]: pgmap v1062: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 691 op/s; 13242/289104 objects degraded (4.580%); 5409/289104 objects misplaced (1.871%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:39:52.045 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:52.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:52.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:52.146 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:52.169 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:52.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:52.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:52.271 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:52.293 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:52.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:52.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:52.395 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:52.420 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:52.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:52.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:52.522 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:52.561 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:52.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:52.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:52.663 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:52.687 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:52.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:52.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:52.727 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:39:52.789 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:52.815 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:52.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:52.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:52.917 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:52.938 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:52.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:52.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:53.040 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:53.083 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:53.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:53.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:53.184 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:53.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:53.207 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:53.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:53.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:53.331 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:53.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:53.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:53.432 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:53.469 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:53.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:53.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:53.501 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:39:53.501 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2024-08-17T17:39:53.501 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2024-08-17T17:39:53.502 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2024-08-17T17:39:53.502 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.502 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:24716 8.2 8:48d30eab:::benchmark_data_smithi003_229103_object24715:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-08-17T17:39:53.502 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:03.748727+0000", 2024-08-17T17:39:53.502 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 409.74902764400002, 2024-08-17T17:39:53.502 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.15246537299999999, 2024-08-17T17:39:53.502 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:39:53.502 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:39:53.502 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:39:53.502 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:39:53.502 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:39:53.503 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 24716 2024-08-17T17:39:53.503 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.503 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:39:53.503 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.503 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:39:53.503 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.748727+0000", 2024-08-17T17:39:53.503 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.503 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.503 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.503 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:39:53.503 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.748727+0000", 2024-08-17T17:39:53.503 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.503 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.504 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.504 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:39:53.504 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.748707+0000", 2024-08-17T17:39:53.504 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999804 2024-08-17T17:39:53.504 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.504 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.504 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:39:53.504 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.748819+0000", 2024-08-17T17:39:53.504 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000112417 2024-08-17T17:39:53.504 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.504 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.504 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:39:53.504 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.748831+0000", 2024-08-17T17:39:53.505 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1411e-05 2024-08-17T17:39:53.505 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.505 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.505 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:39:53.505 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.748837+0000", 2024-08-17T17:39:53.505 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1650000000000003e-06 2024-08-17T17:39:53.505 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.505 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.505 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:39:53.505 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.748883+0000", 2024-08-17T17:39:53.505 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5904e-05 2024-08-17T17:39:53.505 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.505 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.506 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:39:53.506 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.749118+0000", 2024-08-17T17:39:53.506 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00023476500000000001 2024-08-17T17:39:53.506 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.506 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.506 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:39:53.506 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.749261+0000", 2024-08-17T17:39:53.506 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014338299999999999 2024-08-17T17:39:53.506 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.506 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.506 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:39:53.506 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.751689+0000", 2024-08-17T17:39:53.507 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0024283690000000001 2024-08-17T17:39:53.507 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.507 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.507 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.507 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.755002+0000", 2024-08-17T17:39:53.507 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0033125759999999998 2024-08-17T17:39:53.507 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.507 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.507 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.507 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.901091+0000", 2024-08-17T17:39:53.507 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.146088676 2024-08-17T17:39:53.507 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.507 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.508 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:39:53.508 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.901150+0000", 2024-08-17T17:39:53.508 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9098999999999998e-05 2024-08-17T17:39:53.508 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.508 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.508 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:39:53.508 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.901192+0000", 2024-08-17T17:39:53.508 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2212000000000002e-05 2024-08-17T17:39:53.508 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.508 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:39:53.508 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.508 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.509 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.509 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:24721 8.2 8:42725ccf:::benchmark_data_smithi003_229103_object24720:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-08-17T17:39:53.509 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:03.749342+0000", 2024-08-17T17:39:53.509 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 409.74841181900001, 2024-08-17T17:39:53.509 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.151983322, 2024-08-17T17:39:53.509 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:39:53.509 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:39:53.509 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:39:53.509 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:39:53.509 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:39:53.509 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 24721 2024-08-17T17:39:53.509 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.509 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:39:53.510 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.510 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:39:53.510 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.749342+0000", 2024-08-17T17:39:53.510 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.510 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.510 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.510 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:39:53.510 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.749342+0000", 2024-08-17T17:39:53.510 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.510 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.510 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.510 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:39:53.510 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.749341+0000", 2024-08-17T17:39:53.511 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:39:53.511 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.511 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.511 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:39:53.511 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.749372+0000", 2024-08-17T17:39:53.511 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0431000000000001e-05 2024-08-17T17:39:53.511 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.511 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.511 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:39:53.511 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.749383+0000", 2024-08-17T17:39:53.511 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0606e-05 2024-08-17T17:39:53.511 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.511 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.512 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:39:53.512 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.749388+0000", 2024-08-17T17:39:53.512 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.75e-06 2024-08-17T17:39:53.512 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.512 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.512 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:39:53.512 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.750461+0000", 2024-08-17T17:39:53.512 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0010728910000000001 2024-08-17T17:39:53.512 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.512 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.512 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:39:53.512 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.750571+0000", 2024-08-17T17:39:53.512 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010960500000000001 2024-08-17T17:39:53.513 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.513 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.513 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:39:53.513 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.750725+0000", 2024-08-17T17:39:53.513 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000153843 2024-08-17T17:39:53.513 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.513 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.513 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:39:53.513 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.753106+0000", 2024-08-17T17:39:53.513 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0023815120000000001 2024-08-17T17:39:53.513 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.513 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.513 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.514 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.755039+0000", 2024-08-17T17:39:53.514 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001932556 2024-08-17T17:39:53.514 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.514 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.514 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.514 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.901240+0000", 2024-08-17T17:39:53.514 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.14620155800000001 2024-08-17T17:39:53.514 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.514 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.514 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:39:53.514 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.901292+0000", 2024-08-17T17:39:53.514 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1804000000000001e-05 2024-08-17T17:39:53.514 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.515 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.515 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:39:53.515 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.901326+0000", 2024-08-17T17:39:53.515 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3634999999999999e-05 2024-08-17T17:39:53.515 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.515 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:39:53.515 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.515 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.515 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.515 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:24723 8.2 8:421a075d:::benchmark_data_smithi003_229103_object24722:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-08-17T17:39:53.515 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:03.749395+0000", 2024-08-17T17:39:53.515 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 409.74835880900002, 2024-08-17T17:39:53.515 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.15213708100000001, 2024-08-17T17:39:53.516 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:39:53.516 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:39:53.516 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:39:53.516 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:39:53.516 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:39:53.517 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 24723 2024-08-17T17:39:53.517 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.517 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:39:53.517 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.517 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:39:53.517 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.749395+0000", 2024-08-17T17:39:53.517 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.517 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.517 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.517 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:39:53.517 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.749395+0000", 2024-08-17T17:39:53.517 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.518 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.518 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.518 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:39:53.518 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.749395+0000", 2024-08-17T17:39:53.518 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:39:53.518 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.518 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.518 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:39:53.518 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.749425+0000", 2024-08-17T17:39:53.518 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0185e-05 2024-08-17T17:39:53.518 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.518 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.518 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:39:53.519 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.749433+0000", 2024-08-17T17:39:53.519 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1049999999999995e-06 2024-08-17T17:39:53.519 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.519 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.519 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:39:53.519 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.749438+0000", 2024-08-17T17:39:53.519 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5480000000000003e-06 2024-08-17T17:39:53.519 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.519 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.519 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:39:53.519 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.751950+0000", 2024-08-17T17:39:53.519 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002512298 2024-08-17T17:39:53.519 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.519 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.520 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:39:53.520 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.752061+0000", 2024-08-17T17:39:53.520 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000111057 2024-08-17T17:39:53.520 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.520 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.520 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:39:53.520 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.752182+0000", 2024-08-17T17:39:53.520 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000121279 2024-08-17T17:39:53.520 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.520 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.520 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:39:53.520 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.753507+0000", 2024-08-17T17:39:53.520 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0013248750000000001 2024-08-17T17:39:53.520 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.521 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.521 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.521 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.755068+0000", 2024-08-17T17:39:53.521 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0015612410000000001 2024-08-17T17:39:53.521 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.521 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.521 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.521 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.901382+0000", 2024-08-17T17:39:53.521 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.14631343899999999 2024-08-17T17:39:53.521 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.521 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.521 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:39:53.521 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.901499+0000", 2024-08-17T17:39:53.521 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011757899999999999 2024-08-17T17:39:53.522 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.522 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.522 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:39:53.522 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:03.901532+0000", 2024-08-17T17:39:53.522 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3061000000000001e-05 2024-08-17T17:39:53.522 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.522 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:39:53.523 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.523 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.523 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.523 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:75561 8.4 8:2c426cbf:::benchmark_data_smithi003_229103_object75560:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-08-17T17:39:53.523 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:43.185837+0000", 2024-08-17T17:39:53.523 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 370.31191759000001, 2024-08-17T17:39:53.523 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.98011287700000005, 2024-08-17T17:39:53.523 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:39:53.523 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:39:53.523 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:39:53.523 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:39:53.523 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:39:53.523 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 75561 2024-08-17T17:39:53.524 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.524 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:39:53.524 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.524 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:39:53.524 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.185837+0000", 2024-08-17T17:39:53.524 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.524 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.524 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.524 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:39:53.524 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.185837+0000", 2024-08-17T17:39:53.524 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.524 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.524 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.525 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:39:53.525 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.185836+0000", 2024-08-17T17:39:53.525 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:39:53.525 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.525 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.525 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:39:53.525 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.185885+0000", 2024-08-17T17:39:53.525 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9588e-05 2024-08-17T17:39:53.525 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.525 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.525 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:39:53.525 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.185896+0000", 2024-08-17T17:39:53.525 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0937e-05 2024-08-17T17:39:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:39:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.185900+0000", 2024-08-17T17:39:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8909999999999997e-06 2024-08-17T17:39:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:39:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.185957+0000", 2024-08-17T17:39:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7135000000000001e-05 2024-08-17T17:39:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:39:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.185963+0000", 2024-08-17T17:39:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6250000000000004e-06 2024-08-17T17:39:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.527 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.527 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:39:53.527 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.150376+0000", 2024-08-17T17:39:53.527 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.96441359400000004 2024-08-17T17:39:53.527 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.527 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.527 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:39:53.527 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.151738+0000", 2024-08-17T17:39:53.527 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001362032 2024-08-17T17:39:53.527 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.527 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.527 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:39:53.527 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.151916+0000", 2024-08-17T17:39:53.527 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00017807399999999999 2024-08-17T17:39:53.528 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.528 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.528 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:39:53.528 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.158973+0000", 2024-08-17T17:39:53.528 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0070567449999999997 2024-08-17T17:39:53.528 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.528 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.528 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.528 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165494+0000", 2024-08-17T17:39:53.528 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0065203179999999998 2024-08-17T17:39:53.528 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.528 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.528 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.528 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165827+0000", 2024-08-17T17:39:53.529 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00033381300000000002 2024-08-17T17:39:53.529 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.529 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.529 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:39:53.529 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165897+0000", 2024-08-17T17:39:53.529 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9472000000000002e-05 2024-08-17T17:39:53.529 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.529 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.529 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:39:53.529 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165949+0000", 2024-08-17T17:39:53.529 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2636999999999998e-05 2024-08-17T17:39:53.529 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.529 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:39:53.529 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.529 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.530 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.530 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:75575 8.4 8:241d3c27:::benchmark_data_smithi003_229103_object75574:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-08-17T17:39:53.530 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:43.209432+0000", 2024-08-17T17:39:53.530 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 370.28832233600002, 2024-08-17T17:39:53.530 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.95669896099999996, 2024-08-17T17:39:53.530 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:39:53.530 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:39:53.530 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:39:53.530 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:39:53.530 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:39:53.530 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 75575 2024-08-17T17:39:53.530 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.530 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:39:53.530 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.530 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:39:53.531 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.209432+0000", 2024-08-17T17:39:53.531 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.531 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.531 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.531 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:39:53.531 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.209432+0000", 2024-08-17T17:39:53.531 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.531 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.531 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.531 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:39:53.531 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.209430+0000", 2024-08-17T17:39:53.531 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:39:53.531 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.531 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.531 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:39:53.532 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.209491+0000", 2024-08-17T17:39:53.532 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1189e-05 2024-08-17T17:39:53.532 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.532 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.532 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:39:53.532 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.209532+0000", 2024-08-17T17:39:53.532 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0274000000000001e-05 2024-08-17T17:39:53.532 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.532 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.532 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:39:53.532 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.209542+0000", 2024-08-17T17:39:53.533 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0556e-05 2024-08-17T17:39:53.533 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.533 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.533 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:39:53.533 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.209612+0000", 2024-08-17T17:39:53.533 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9669e-05 2024-08-17T17:39:53.533 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.533 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.533 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:39:53.533 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.209623+0000", 2024-08-17T17:39:53.533 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1402e-05 2024-08-17T17:39:53.533 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.533 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.534 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:39:53.534 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.153453+0000", 2024-08-17T17:39:53.534 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.94382987200000001 2024-08-17T17:39:53.534 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.534 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.534 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:39:53.534 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.153538+0000", 2024-08-17T17:39:53.534 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.4448000000000002e-05 2024-08-17T17:39:53.534 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.534 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.534 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:39:53.534 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.153641+0000", 2024-08-17T17:39:53.534 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010312200000000001 2024-08-17T17:39:53.535 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.535 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.535 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:39:53.535 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.164927+0000", 2024-08-17T17:39:53.535 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011286503999999999 2024-08-17T17:39:53.535 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.535 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.535 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.535 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165658+0000", 2024-08-17T17:39:53.535 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00073055899999999998 2024-08-17T17:39:53.535 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.536 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.536 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.536 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.166034+0000", 2024-08-17T17:39:53.536 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000375891 2024-08-17T17:39:53.536 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.536 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.536 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:39:53.536 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.166087+0000", 2024-08-17T17:39:53.536 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3570999999999999e-05 2024-08-17T17:39:53.536 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.536 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.536 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:39:53.537 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.166131+0000", 2024-08-17T17:39:53.537 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3636e-05 2024-08-17T17:39:53.537 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.537 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:39:53.537 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.537 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.537 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.537 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:75590 8.4 8:2d80d3ce:::benchmark_data_smithi003_229103_object75589:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-08-17T17:39:53.537 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:43.225574+0000", 2024-08-17T17:39:53.537 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 370.27218019899999, 2024-08-17T17:39:53.537 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.94686139400000002, 2024-08-17T17:39:53.537 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:39:53.537 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:39:53.537 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:39:53.537 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:39:53.538 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:39:53.538 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 75590 2024-08-17T17:39:53.538 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.538 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:39:53.538 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.538 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:39:53.538 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.225574+0000", 2024-08-17T17:39:53.538 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.538 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.538 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.538 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:39:53.538 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.225574+0000", 2024-08-17T17:39:53.538 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.539 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.539 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.539 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:39:53.539 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.225573+0000", 2024-08-17T17:39:53.539 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:39:53.539 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.539 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.539 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:39:53.539 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.225645+0000", 2024-08-17T17:39:53.539 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.1625999999999995e-05 2024-08-17T17:39:53.539 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.539 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.539 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:39:53.539 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.225655+0000", 2024-08-17T17:39:53.539 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.043e-05 2024-08-17T17:39:53.540 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.540 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.540 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:39:53.540 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.225662+0000", 2024-08-17T17:39:53.540 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4420000000000003e-06 2024-08-17T17:39:53.540 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.540 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.540 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:39:53.540 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.225718+0000", 2024-08-17T17:39:53.540 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6232e-05 2024-08-17T17:39:53.540 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.540 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.540 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:39:53.540 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.225727+0000", 2024-08-17T17:39:53.541 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1949999999999992e-06 2024-08-17T17:39:53.541 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.541 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.541 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:39:53.541 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.155232+0000", 2024-08-17T17:39:53.541 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.92950502499999998 2024-08-17T17:39:53.541 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.541 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.541 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:39:53.541 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.155349+0000", 2024-08-17T17:39:53.541 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000117036 2024-08-17T17:39:53.541 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.541 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.541 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:39:53.541 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.155535+0000", 2024-08-17T17:39:53.542 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000185374 2024-08-17T17:39:53.542 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.542 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.542 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:39:53.542 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.164936+0000", 2024-08-17T17:39:53.542 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0094012939999999993 2024-08-17T17:39:53.542 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.542 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.542 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.542 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165743+0000", 2024-08-17T17:39:53.542 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00080753499999999996 2024-08-17T17:39:53.542 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.543 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.543 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.543 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.172337+0000", 2024-08-17T17:39:53.543 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0065936959999999996 2024-08-17T17:39:53.543 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.543 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.543 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:39:53.543 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.172389+0000", 2024-08-17T17:39:53.543 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1940000000000001e-05 2024-08-17T17:39:53.543 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.543 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.543 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:39:53.543 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.172435+0000", 2024-08-17T17:39:53.543 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6322999999999997e-05 2024-08-17T17:39:53.544 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.544 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:39:53.544 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.544 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.544 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.544 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:75606 8.4 8:2b96e39f:::benchmark_data_smithi003_229103_object75605:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-08-17T17:39:53.544 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:43.250288+0000", 2024-08-17T17:39:53.544 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 370.24746598899998, 2024-08-17T17:39:53.544 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.925042578, 2024-08-17T17:39:53.544 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:39:53.544 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:39:53.544 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:39:53.545 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:39:53.545 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:39:53.545 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 75606 2024-08-17T17:39:53.545 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.545 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:39:53.545 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.545 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:39:53.545 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.250288+0000", 2024-08-17T17:39:53.545 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.545 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.545 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.545 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:39:53.545 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.250288+0000", 2024-08-17T17:39:53.546 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.546 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.546 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.546 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:39:53.546 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.250287+0000", 2024-08-17T17:39:53.546 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:39:53.546 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.546 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.546 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:39:53.546 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.250335+0000", 2024-08-17T17:39:53.546 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.8593e-05 2024-08-17T17:39:53.546 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.546 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.546 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:39:53.547 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.250348+0000", 2024-08-17T17:39:53.547 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3128e-05 2024-08-17T17:39:53.547 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.547 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.547 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:39:53.547 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.250353+0000", 2024-08-17T17:39:53.547 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6589999999999998e-06 2024-08-17T17:39:53.547 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.547 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.547 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:39:53.547 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.250400+0000", 2024-08-17T17:39:53.547 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7262999999999998e-05 2024-08-17T17:39:53.547 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.548 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.548 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:39:53.548 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.250407+0000", 2024-08-17T17:39:53.548 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.6880000000000002e-06 2024-08-17T17:39:53.548 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.548 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.548 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:39:53.548 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.157095+0000", 2024-08-17T17:39:53.548 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.90668845600000003 2024-08-17T17:39:53.548 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.548 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.548 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:39:53.548 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.157173+0000", 2024-08-17T17:39:53.548 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7803e-05 2024-08-17T17:39:53.549 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.549 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.549 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:39:53.549 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.157268+0000", 2024-08-17T17:39:53.549 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.4374000000000001e-05 2024-08-17T17:39:53.549 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.549 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.549 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:39:53.549 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.164993+0000", 2024-08-17T17:39:53.549 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0077249240000000002 2024-08-17T17:39:53.549 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.549 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.549 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.549 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165789+0000", 2024-08-17T17:39:53.549 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000796486 2024-08-17T17:39:53.550 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.550 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.550 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.550 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175203+0000", 2024-08-17T17:39:53.550 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0094138179999999991 2024-08-17T17:39:53.550 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.550 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.550 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:39:53.550 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175253+0000", 2024-08-17T17:39:53.550 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9712000000000001e-05 2024-08-17T17:39:53.550 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.550 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.550 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:39:53.551 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175331+0000", 2024-08-17T17:39:53.551 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.8245999999999996e-05 2024-08-17T17:39:53.551 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.551 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:39:53.551 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.551 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.551 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.551 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:75612 8.4 8:2055fef1:::benchmark_data_smithi003_229103_object75611:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-08-17T17:39:53.551 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:43.260677+0000", 2024-08-17T17:39:53.551 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 370.23707718600002, 2024-08-17T17:39:53.551 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.91479233199999999, 2024-08-17T17:39:53.551 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:39:53.551 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:39:53.551 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:39:53.552 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:39:53.552 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:39:53.552 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 75612 2024-08-17T17:39:53.552 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.552 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:39:53.552 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.552 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:39:53.552 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.260677+0000", 2024-08-17T17:39:53.552 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.552 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.552 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.552 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:39:53.552 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.260677+0000", 2024-08-17T17:39:53.552 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.552 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.553 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.553 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:39:53.553 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.260676+0000", 2024-08-17T17:39:53.553 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:39:53.553 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.553 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.553 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:39:53.553 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.260728+0000", 2024-08-17T17:39:53.553 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1887e-05 2024-08-17T17:39:53.553 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.553 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.553 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:39:53.553 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.260737+0000", 2024-08-17T17:39:53.553 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.516e-06 2024-08-17T17:39:53.554 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.554 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.554 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:39:53.554 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.260741+0000", 2024-08-17T17:39:53.554 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5039999999999998e-06 2024-08-17T17:39:53.554 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.554 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.554 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:39:53.554 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.260794+0000", 2024-08-17T17:39:53.554 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3078000000000002e-05 2024-08-17T17:39:53.554 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.555 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.555 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:39:53.555 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.260805+0000", 2024-08-17T17:39:53.555 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1459e-05 2024-08-17T17:39:53.555 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.555 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.555 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:39:53.555 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.159008+0000", 2024-08-17T17:39:53.555 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.89820229900000004 2024-08-17T17:39:53.555 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.555 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.555 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:39:53.556 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.159094+0000", 2024-08-17T17:39:53.556 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.6119000000000007e-05 2024-08-17T17:39:53.556 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.556 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.556 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:39:53.556 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.159193+0000", 2024-08-17T17:39:53.556 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.9285999999999998e-05 2024-08-17T17:39:53.556 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.556 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.556 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:39:53.556 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165000+0000", 2024-08-17T17:39:53.556 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.005806833 2024-08-17T17:39:53.556 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.556 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.557 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.557 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165990+0000", 2024-08-17T17:39:53.557 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000990152 2024-08-17T17:39:53.557 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.557 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.557 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.557 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175382+0000", 2024-08-17T17:39:53.557 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0093919659999999999 2024-08-17T17:39:53.557 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.557 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.557 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:39:53.557 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175431+0000", 2024-08-17T17:39:53.557 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9101999999999998e-05 2024-08-17T17:39:53.557 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.558 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.558 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:39:53.558 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175469+0000", 2024-08-17T17:39:53.558 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8189000000000003e-05 2024-08-17T17:39:53.558 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.558 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:39:53.558 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.558 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.558 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.558 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:75613 8.4 8:2044481c:::benchmark_data_smithi003_229103_object75612:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-08-17T17:39:53.558 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:43.262848+0000", 2024-08-17T17:39:53.559 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 370.234906088, 2024-08-17T17:39:53.559 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.91278752399999996, 2024-08-17T17:39:53.559 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:39:53.559 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:39:53.559 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:39:53.559 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:39:53.560 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:39:53.560 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 75613 2024-08-17T17:39:53.560 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.560 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:39:53.560 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.560 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:39:53.560 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.262848+0000", 2024-08-17T17:39:53.560 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.560 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.560 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.561 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:39:53.561 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.262848+0000", 2024-08-17T17:39:53.561 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.561 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.561 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.561 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:39:53.561 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.262847+0000", 2024-08-17T17:39:53.561 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:39:53.561 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.561 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.561 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:39:53.561 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.262896+0000", 2024-08-17T17:39:53.561 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9047e-05 2024-08-17T17:39:53.562 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.562 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.562 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:39:53.562 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.262914+0000", 2024-08-17T17:39:53.562 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7093000000000001e-05 2024-08-17T17:39:53.562 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.562 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.562 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:39:53.562 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.262918+0000", 2024-08-17T17:39:53.562 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.561e-06 2024-08-17T17:39:53.562 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.562 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.562 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:39:53.563 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.262960+0000", 2024-08-17T17:39:53.563 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1659000000000002e-05 2024-08-17T17:39:53.563 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.563 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.563 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:39:53.563 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.262967+0000", 2024-08-17T17:39:53.563 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7279999999999995e-06 2024-08-17T17:39:53.563 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.563 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.563 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:39:53.563 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.160829+0000", 2024-08-17T17:39:53.563 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.89786185900000004 2024-08-17T17:39:53.564 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.564 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.564 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:39:53.564 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.160952+0000", 2024-08-17T17:39:53.564 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012245499999999999 2024-08-17T17:39:53.564 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.564 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.564 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:39:53.564 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.161116+0000", 2024-08-17T17:39:53.564 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000164208 2024-08-17T17:39:53.564 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.564 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.564 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:39:53.565 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165004+0000", 2024-08-17T17:39:53.565 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0038882230000000001 2024-08-17T17:39:53.565 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.565 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.565 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.565 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.167386+0000", 2024-08-17T17:39:53.565 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0023818960000000001 2024-08-17T17:39:53.565 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.565 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.565 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.565 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175558+0000", 2024-08-17T17:39:53.565 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0081719789999999994 2024-08-17T17:39:53.566 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.566 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.566 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:39:53.566 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175602+0000", 2024-08-17T17:39:53.566 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4008999999999998e-05 2024-08-17T17:39:53.566 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.566 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.566 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:39:53.566 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175636+0000", 2024-08-17T17:39:53.566 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3494999999999997e-05 2024-08-17T17:39:53.566 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.566 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:39:53.566 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.567 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.567 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.567 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:75614 8.4 8:2ac96a7e:::benchmark_data_smithi003_229103_object75613:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-08-17T17:39:53.567 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:43.265251+0000", 2024-08-17T17:39:53.567 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 370.23250328699999, 2024-08-17T17:39:53.567 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.91049424000000001, 2024-08-17T17:39:53.567 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:39:53.567 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:39:53.567 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:39:53.567 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:39:53.567 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:39:53.567 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 75614 2024-08-17T17:39:53.567 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.568 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:39:53.568 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.568 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:39:53.568 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.265251+0000", 2024-08-17T17:39:53.568 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.568 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.568 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.568 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:39:53.568 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.265251+0000", 2024-08-17T17:39:53.568 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.568 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.568 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.568 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:39:53.568 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.265250+0000", 2024-08-17T17:39:53.568 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:39:53.569 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.569 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.569 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:39:53.569 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.265297+0000", 2024-08-17T17:39:53.569 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7088e-05 2024-08-17T17:39:53.569 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.569 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.569 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:39:53.569 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.265306+0000", 2024-08-17T17:39:53.569 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9819999999999997e-06 2024-08-17T17:39:53.569 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.569 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.569 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:39:53.569 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.265310+0000", 2024-08-17T17:39:53.569 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4189999999999998e-06 2024-08-17T17:39:53.569 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.570 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.570 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:39:53.570 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.265345+0000", 2024-08-17T17:39:53.570 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5271000000000002e-05 2024-08-17T17:39:53.570 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.570 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.570 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:39:53.570 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.265354+0000", 2024-08-17T17:39:53.570 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.5150000000000006e-06 2024-08-17T17:39:53.570 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.570 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.570 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:39:53.570 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.162842+0000", 2024-08-17T17:39:53.571 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.89748799899999998 2024-08-17T17:39:53.571 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.571 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.571 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:39:53.571 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.163010+0000", 2024-08-17T17:39:53.571 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016868699999999999 2024-08-17T17:39:53.571 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.571 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.571 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:39:53.571 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.163147+0000", 2024-08-17T17:39:53.571 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000136821 2024-08-17T17:39:53.571 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.571 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.571 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:39:53.571 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165452+0000", 2024-08-17T17:39:53.572 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0023047039999999999 2024-08-17T17:39:53.572 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.572 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.572 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.572 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.167852+0000", 2024-08-17T17:39:53.572 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002400648 2024-08-17T17:39:53.572 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.572 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.572 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.572 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175678+0000", 2024-08-17T17:39:53.572 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0078257429999999996 2024-08-17T17:39:53.572 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.572 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.572 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:39:53.572 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175710+0000", 2024-08-17T17:39:53.573 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2128999999999997e-05 2024-08-17T17:39:53.573 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.573 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.573 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:39:53.573 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175745+0000", 2024-08-17T17:39:53.573 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4854000000000002e-05 2024-08-17T17:39:53.573 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.573 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:39:53.573 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.573 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.573 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.573 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:85945 8.4 8:2a9d539f:::benchmark_data_smithi003_229103_object85944:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-08-17T17:39:53.573 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:56.753525+0000", 2024-08-17T17:39:53.573 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 356.744229453, 2024-08-17T17:39:53.573 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.19019344299999999, 2024-08-17T17:39:53.574 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:39:53.574 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:39:53.574 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:39:53.574 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:39:53.574 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:39:53.574 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 85945 2024-08-17T17:39:53.574 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.574 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:39:53.574 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.574 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:39:53.574 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.753525+0000", 2024-08-17T17:39:53.574 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.574 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.575 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.575 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:39:53.575 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.753525+0000", 2024-08-17T17:39:53.575 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.575 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.575 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.575 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:39:53.575 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.753523+0000", 2024-08-17T17:39:53.575 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:39:53.575 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.575 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.575 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:39:53.575 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.753569+0000", 2024-08-17T17:39:53.576 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5286e-05 2024-08-17T17:39:53.576 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.576 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.576 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:39:53.576 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.753590+0000", 2024-08-17T17:39:53.576 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1053999999999999e-05 2024-08-17T17:39:53.576 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.576 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.576 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:39:53.576 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.753594+0000", 2024-08-17T17:39:53.576 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9189999999999998e-06 2024-08-17T17:39:53.576 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.576 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.576 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:39:53.577 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.753651+0000", 2024-08-17T17:39:53.577 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6291e-05 2024-08-17T17:39:53.577 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.577 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.577 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:39:53.577 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.753776+0000", 2024-08-17T17:39:53.577 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012524199999999999 2024-08-17T17:39:53.577 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.577 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.577 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:39:53.577 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.753920+0000", 2024-08-17T17:39:53.577 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000143688 2024-08-17T17:39:53.577 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.577 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.578 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:39:53.578 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.756430+0000", 2024-08-17T17:39:53.578 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002510721 2024-08-17T17:39:53.578 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.578 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.578 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.578 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.756580+0000", 2024-08-17T17:39:53.578 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014915600000000001 2024-08-17T17:39:53.578 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.578 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.578 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.578 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.943621+0000", 2024-08-17T17:39:53.578 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.18704188599999999 2024-08-17T17:39:53.579 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.579 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.579 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:39:53.579 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.943674+0000", 2024-08-17T17:39:53.579 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3020999999999999e-05 2024-08-17T17:39:53.579 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.579 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.579 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:39:53.579 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.943718+0000", 2024-08-17T17:39:53.579 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3674999999999998e-05 2024-08-17T17:39:53.579 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.579 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:39:53.579 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.579 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.580 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.580 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:85950 8.4 8:2480217a:::benchmark_data_smithi003_229103_object85949:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-08-17T17:39:53.580 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:56.759541+0000", 2024-08-17T17:39:53.580 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 356.738213266, 2024-08-17T17:39:53.580 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.19582751800000001, 2024-08-17T17:39:53.580 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:39:53.580 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:39:53.580 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:39:53.580 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:39:53.580 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:39:53.580 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 85950 2024-08-17T17:39:53.580 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.580 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:39:53.580 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.581 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:39:53.581 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.759541+0000", 2024-08-17T17:39:53.581 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.581 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.581 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.581 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:39:53.581 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.759541+0000", 2024-08-17T17:39:53.581 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.581 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.581 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.581 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:39:53.581 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.759539+0000", 2024-08-17T17:39:53.581 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:39:53.581 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.582 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.582 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:39:53.582 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.759623+0000", 2024-08-17T17:39:53.582 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.3555999999999998e-05 2024-08-17T17:39:53.582 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.582 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.582 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:39:53.582 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.759634+0000", 2024-08-17T17:39:53.582 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1618e-05 2024-08-17T17:39:53.582 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.582 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.582 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:39:53.582 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.759640+0000", 2024-08-17T17:39:53.582 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0630000000000002e-06 2024-08-17T17:39:53.583 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.583 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.583 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:39:53.583 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.759677+0000", 2024-08-17T17:39:53.583 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6434999999999999e-05 2024-08-17T17:39:53.583 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.583 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.583 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:39:53.583 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.759777+0000", 2024-08-17T17:39:53.583 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010024199999999999 2024-08-17T17:39:53.583 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.583 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.583 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:39:53.583 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.759863+0000", 2024-08-17T17:39:53.584 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.5628000000000001e-05 2024-08-17T17:39:53.584 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.584 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.584 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:39:53.584 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.762723+0000", 2024-08-17T17:39:53.584 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0028608790000000002 2024-08-17T17:39:53.584 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.584 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.584 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.584 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.764164+0000", 2024-08-17T17:39:53.584 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0014406239999999999 2024-08-17T17:39:53.584 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.584 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.585 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.585 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.955281+0000", 2024-08-17T17:39:53.585 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.19111708599999999 2024-08-17T17:39:53.585 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.585 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.585 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:39:53.585 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.955332+0000", 2024-08-17T17:39:53.585 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0785000000000003e-05 2024-08-17T17:39:53.585 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.585 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.585 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:39:53.585 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.955368+0000", 2024-08-17T17:39:53.585 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6528e-05 2024-08-17T17:39:53.586 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.586 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:39:53.586 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.586 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.586 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.586 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:131591 8.2 8:484be707:::benchmark_data_smithi003_229103_object24116:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-08-17T17:39:53.586 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:35:00.828656+0000", 2024-08-17T17:39:53.586 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 292.66909803300001, 2024-08-17T17:39:53.586 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.13269879100000001, 2024-08-17T17:39:53.586 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:39:53.586 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:39:53.586 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:39:53.586 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:39:53.586 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:39:53.587 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 131591 2024-08-17T17:39:53.587 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.587 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:39:53.587 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.587 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:39:53.587 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.828656+0000", 2024-08-17T17:39:53.587 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.587 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.587 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.587 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:39:53.587 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.828656+0000", 2024-08-17T17:39:53.587 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.587 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.587 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.588 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:39:53.588 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.828647+0000", 2024-08-17T17:39:53.588 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999909 2024-08-17T17:39:53.588 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.588 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.588 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:39:53.588 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.828666+0000", 2024-08-17T17:39:53.588 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.8836000000000001e-05 2024-08-17T17:39:53.588 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.588 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.588 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:39:53.588 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.828699+0000", 2024-08-17T17:39:53.588 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3535999999999999e-05 2024-08-17T17:39:53.589 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.589 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.589 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:39:53.589 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.828714+0000", 2024-08-17T17:39:53.589 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4779e-05 2024-08-17T17:39:53.589 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.589 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.589 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:39:53.589 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.828837+0000", 2024-08-17T17:39:53.589 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000122268 2024-08-17T17:39:53.589 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.589 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.589 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:39:53.589 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.829019+0000", 2024-08-17T17:39:53.590 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00018226200000000001 2024-08-17T17:39:53.590 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.590 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.590 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:39:53.590 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.829143+0000", 2024-08-17T17:39:53.590 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000124308 2024-08-17T17:39:53.590 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.590 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.590 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:39:53.590 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.835723+0000", 2024-08-17T17:39:53.590 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0065798590000000004 2024-08-17T17:39:53.590 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.590 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.590 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.591 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.836985+0000", 2024-08-17T17:39:53.591 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0012621900000000001 2024-08-17T17:39:53.591 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.591 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.591 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.591 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.961251+0000", 2024-08-17T17:39:53.591 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.124266183 2024-08-17T17:39:53.591 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.591 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.591 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:39:53.591 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.961306+0000", 2024-08-17T17:39:53.591 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5061e-05 2024-08-17T17:39:53.591 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.591 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.592 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:39:53.592 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.961355+0000", 2024-08-17T17:39:53.592 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.8547999999999997e-05 2024-08-17T17:39:53.592 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.592 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:39:53.592 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.592 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.592 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.592 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:131598 8.2 8:4c61b07f:::benchmark_data_smithi003_229103_object24123:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-08-17T17:39:53.592 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:35:00.830343+0000", 2024-08-17T17:39:53.592 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 292.66741116999998, 2024-08-17T17:39:53.592 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.13114300000000001, 2024-08-17T17:39:53.592 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:39:53.592 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:39:53.593 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:39:53.593 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:39:53.593 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:39:53.593 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 131598 2024-08-17T17:39:53.593 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.593 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:39:53.593 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.593 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:39:53.593 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.830343+0000", 2024-08-17T17:39:53.593 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.593 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.593 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.593 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:39:53.593 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.830343+0000", 2024-08-17T17:39:53.593 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.594 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.594 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.594 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:39:53.594 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.830342+0000", 2024-08-17T17:39:53.594 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:39:53.594 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.594 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.594 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:39:53.594 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.830348+0000", 2024-08-17T17:39:53.594 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6869999999999999e-06 2024-08-17T17:39:53.594 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.594 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.594 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:39:53.594 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.830361+0000", 2024-08-17T17:39:53.594 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3200000000000001e-05 2024-08-17T17:39:53.595 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.595 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.595 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:39:53.595 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.830366+0000", 2024-08-17T17:39:53.595 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5620000000000003e-06 2024-08-17T17:39:53.595 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.595 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.595 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:39:53.595 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.832067+0000", 2024-08-17T17:39:53.595 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0017004380000000001 2024-08-17T17:39:53.595 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.595 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.595 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:39:53.595 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.832204+0000", 2024-08-17T17:39:53.595 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013674299999999999 2024-08-17T17:39:53.596 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.596 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.596 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:39:53.596 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.832327+0000", 2024-08-17T17:39:53.596 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000123576 2024-08-17T17:39:53.596 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.596 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.596 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:39:53.596 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.835962+0000", 2024-08-17T17:39:53.596 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0036344350000000001 2024-08-17T17:39:53.596 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.596 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.596 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.596 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.837168+0000", 2024-08-17T17:39:53.596 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0012062119999999999 2024-08-17T17:39:53.597 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.597 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.597 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.597 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.961398+0000", 2024-08-17T17:39:53.597 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.124230363 2024-08-17T17:39:53.597 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.597 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.597 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:39:53.597 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.961445+0000", 2024-08-17T17:39:53.597 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7236e-05 2024-08-17T17:39:53.597 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.597 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.597 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:39:53.597 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.961486+0000", 2024-08-17T17:39:53.598 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0608999999999997e-05 2024-08-17T17:39:53.598 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.598 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:39:53.598 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.598 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.598 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.598 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:199665 8.4 8:2fc9da80:::benchmark_data_smithi003_229103_object92190:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e295)", 2024-08-17T17:39:53.598 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:40.260406+0000", 2024-08-17T17:39:53.598 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 193.23734865200001, 2024-08-17T17:39:53.598 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.17482935799999999, 2024-08-17T17:39:53.598 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:39:53.598 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:39:53.598 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:39:53.599 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:39:53.599 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:39:53.599 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 199665 2024-08-17T17:39:53.599 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.599 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:39:53.599 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.599 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:39:53.599 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.260406+0000", 2024-08-17T17:39:53.599 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.599 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.599 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.599 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:39:53.599 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.260406+0000", 2024-08-17T17:39:53.599 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.600 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.600 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.600 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:39:53.600 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.260405+0000", 2024-08-17T17:39:53.600 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:39:53.600 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.600 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.600 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:39:53.600 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.260409+0000", 2024-08-17T17:39:53.600 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4549999999999997e-06 2024-08-17T17:39:53.600 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.600 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.600 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:39:53.600 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.260418+0000", 2024-08-17T17:39:53.600 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.8720000000000008e-06 2024-08-17T17:39:53.601 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.601 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.601 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:39:53.601 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.260422+0000", 2024-08-17T17:39:53.601 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2590000000000002e-06 2024-08-17T17:39:53.601 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.601 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.601 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:39:53.601 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.264708+0000", 2024-08-17T17:39:53.601 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0042857590000000001 2024-08-17T17:39:53.601 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.601 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.601 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:39:53.601 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.264787+0000", 2024-08-17T17:39:53.601 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.9277999999999995e-05 2024-08-17T17:39:53.602 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.602 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.602 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:39:53.602 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.264861+0000", 2024-08-17T17:39:53.602 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3536999999999997e-05 2024-08-17T17:39:53.602 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.602 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.602 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:39:53.602 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.269373+0000", 2024-08-17T17:39:53.602 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0045125080000000001 2024-08-17T17:39:53.602 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.602 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.602 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.603 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.270503+0000", 2024-08-17T17:39:53.603 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001130065 2024-08-17T17:39:53.603 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.603 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.603 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.603 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435126+0000", 2024-08-17T17:39:53.603 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.16462236399999999 2024-08-17T17:39:53.603 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.603 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.603 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:39:53.603 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435185+0000", 2024-08-17T17:39:53.603 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9175999999999997e-05 2024-08-17T17:39:53.603 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.603 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.603 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:39:53.604 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435235+0000", 2024-08-17T17:39:53.604 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9889000000000003e-05 2024-08-17T17:39:53.604 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.604 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:39:53.604 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.604 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.604 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.604 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:199682 8.2 8:459cfaea:::benchmark_data_smithi003_229103_object92207:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e295)", 2024-08-17T17:39:53.604 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:40.289885+0000", 2024-08-17T17:39:53.604 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 193.20786904799999, 2024-08-17T17:39:53.604 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.14626383300000001, 2024-08-17T17:39:53.604 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:39:53.604 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:39:53.604 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:39:53.604 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:39:53.605 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:39:53.605 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 199682 2024-08-17T17:39:53.605 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.605 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:39:53.605 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.605 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:39:53.605 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289885+0000", 2024-08-17T17:39:53.605 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.605 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.605 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.605 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:39:53.605 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289885+0000", 2024-08-17T17:39:53.605 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.606 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.606 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.606 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:39:53.606 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289884+0000", 2024-08-17T17:39:53.606 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:39:53.606 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.606 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.606 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:39:53.606 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289890+0000", 2024-08-17T17:39:53.606 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5989999999999998e-06 2024-08-17T17:39:53.606 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.606 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.606 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:39:53.607 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289902+0000", 2024-08-17T17:39:53.607 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2007999999999999e-05 2024-08-17T17:39:53.607 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.607 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.607 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:39:53.607 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289919+0000", 2024-08-17T17:39:53.607 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.702e-05 2024-08-17T17:39:53.607 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.607 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.607 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:39:53.607 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289967+0000", 2024-08-17T17:39:53.607 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7367000000000002e-05 2024-08-17T17:39:53.607 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.607 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.607 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:39:53.608 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.290071+0000", 2024-08-17T17:39:53.608 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000103952 2024-08-17T17:39:53.608 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.608 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.608 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:39:53.608 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.290175+0000", 2024-08-17T17:39:53.608 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000104299 2024-08-17T17:39:53.608 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.608 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.608 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:39:53.608 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435038+0000", 2024-08-17T17:39:53.608 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.144863348 2024-08-17T17:39:53.608 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.608 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.609 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.609 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435989+0000", 2024-08-17T17:39:53.609 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00095057200000000001 2024-08-17T17:39:53.609 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.609 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.609 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.609 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.436062+0000", 2024-08-17T17:39:53.609 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3015999999999993e-05 2024-08-17T17:39:53.609 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.609 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.609 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:39:53.609 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.436109+0000", 2024-08-17T17:39:53.609 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7234000000000002e-05 2024-08-17T17:39:53.610 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.610 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.610 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:39:53.610 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.436149+0000", 2024-08-17T17:39:53.610 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9816e-05 2024-08-17T17:39:53.610 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.610 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:39:53.611 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.611 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.611 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.611 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop_reply(client.46025.0:199665 8.4 e295/272 ondisk, result = 0)", 2024-08-17T17:39:53.611 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:40.291703+0000", 2024-08-17T17:39:53.612 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 193.20605092299999, 2024-08-17T17:39:53.612 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.14354199400000001, 2024-08-17T17:39:53.612 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:39:53.612 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2024-08-17T17:39:53.612 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:39:53.612 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.612 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:39:53.612 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.291703+0000", 2024-08-17T17:39:53.612 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.612 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.612 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.612 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:39:53.613 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.291703+0000", 2024-08-17T17:39:53.613 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.613 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.613 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.613 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:39:53.613 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.291702+0000", 2024-08-17T17:39:53.614 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:39:53.614 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.614 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.614 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:39:53.614 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.291708+0000", 2024-08-17T17:39:53.614 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7710000000000003e-06 2024-08-17T17:39:53.614 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.614 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.614 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:39:53.614 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.430310+0000", 2024-08-17T17:39:53.614 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.13860172500000001 2024-08-17T17:39:53.614 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.615 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.615 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:39:53.615 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.430318+0000", 2024-08-17T17:39:53.615 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.4009999999999997e-06 2024-08-17T17:39:53.615 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.615 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.615 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:39:53.615 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435110+0000", 2024-08-17T17:39:53.615 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0047917469999999998 2024-08-17T17:39:53.615 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.616 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.616 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:39:53.616 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435119+0000", 2024-08-17T17:39:53.616 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.2280000000000004e-06 2024-08-17T17:39:53.616 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.616 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.616 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:39:53.616 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435245+0000", 2024-08-17T17:39:53.617 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012596800000000001 2024-08-17T17:39:53.617 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.617 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:39:53.617 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.617 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.617 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.617 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:199687 8.4 8:2561a798:::benchmark_data_smithi003_229103_object92212:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e295)", 2024-08-17T17:39:53.617 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:40.300569+0000", 2024-08-17T17:39:53.617 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 193.19718476099999, 2024-08-17T17:39:53.618 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.13988004500000001, 2024-08-17T17:39:53.618 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:53.620 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:39:53.620 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:39:53.620 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:39:53.620 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:39:53.620 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:39:53.620 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 199687 2024-08-17T17:39:53.620 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.621 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:39:53.621 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.621 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:39:53.621 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.300569+0000", 2024-08-17T17:39:53.621 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.621 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.621 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.621 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:39:53.621 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.300569+0000", 2024-08-17T17:39:53.621 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.621 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.621 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.622 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:39:53.622 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.300567+0000", 2024-08-17T17:39:53.622 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:39:53.622 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.622 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.622 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:39:53.622 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.300578+0000", 2024-08-17T17:39:53.622 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1070000000000001e-05 2024-08-17T17:39:53.622 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.622 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.623 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:39:53.623 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.430060+0000", 2024-08-17T17:39:53.623 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.129482028 2024-08-17T17:39:53.623 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.623 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.623 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:39:53.623 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.430078+0000", 2024-08-17T17:39:53.623 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7278999999999998e-05 2024-08-17T17:39:53.623 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.623 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.623 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:39:53.624 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.430135+0000", 2024-08-17T17:39:53.624 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7362999999999999e-05 2024-08-17T17:39:53.624 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.624 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.624 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:39:53.624 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.430275+0000", 2024-08-17T17:39:53.624 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014024400000000001 2024-08-17T17:39:53.624 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.624 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.624 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:39:53.625 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.430407+0000", 2024-08-17T17:39:53.625 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000131309 2024-08-17T17:39:53.625 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.625 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.625 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:39:53.625 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435256+0000", 2024-08-17T17:39:53.625 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0048498359999999997 2024-08-17T17:39:53.625 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.625 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.625 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.625 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435299+0000", 2024-08-17T17:39:53.625 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2561999999999997e-05 2024-08-17T17:39:53.625 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.625 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.626 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.626 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.440368+0000", 2024-08-17T17:39:53.626 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0050688039999999997 2024-08-17T17:39:53.626 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.626 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.626 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:39:53.626 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.440412+0000", 2024-08-17T17:39:53.626 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4582000000000001e-05 2024-08-17T17:39:53.626 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.626 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.626 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:39:53.626 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.440449+0000", 2024-08-17T17:39:53.626 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.7024999999999999e-05 2024-08-17T17:39:53.627 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.627 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:39:53.627 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.627 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.627 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.627 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36354.0:54190 9.e 9:7ae4152c:::benchmark_data_smithi003_261051_object54189:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-08-17T17:39:53.627 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:38:17.551910+0000", 2024-08-17T17:39:53.627 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 95.945844511999994, 2024-08-17T17:39:53.627 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.11876806299999999, 2024-08-17T17:39:53.627 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:39:53.627 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:39:53.627 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:39:53.627 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36354", 2024-08-17T17:39:53.627 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/1330158990", 2024-08-17T17:39:53.627 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 54190 2024-08-17T17:39:53.628 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.628 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:39:53.628 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.628 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:39:53.628 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:38:17.551910+0000", 2024-08-17T17:39:53.628 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.628 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.628 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.628 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:39:53.628 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:38:17.551910+0000", 2024-08-17T17:39:53.628 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.628 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.628 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.628 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:39:53.628 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:38:17.551908+0000", 2024-08-17T17:39:53.629 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:39:53.629 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.629 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.629 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:39:53.629 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:38:17.551962+0000", 2024-08-17T17:39:53.629 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4076999999999997e-05 2024-08-17T17:39:53.629 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.629 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.629 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:39:53.629 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:38:17.551974+0000", 2024-08-17T17:39:53.629 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2069000000000001e-05 2024-08-17T17:39:53.630 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.630 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.630 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:39:53.630 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:38:17.551979+0000", 2024-08-17T17:39:53.630 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.437e-06 2024-08-17T17:39:53.630 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.630 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.630 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:39:53.630 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:38:17.552032+0000", 2024-08-17T17:39:53.630 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3640000000000001e-05 2024-08-17T17:39:53.630 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.630 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.631 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:39:53.631 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:38:17.552180+0000", 2024-08-17T17:39:53.631 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014772799999999999 2024-08-17T17:39:53.631 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.631 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.631 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,7", 2024-08-17T17:39:53.631 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:38:17.552346+0000", 2024-08-17T17:39:53.631 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016576699999999999 2024-08-17T17:39:53.631 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.631 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.631 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:39:53.632 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:38:17.553918+0000", 2024-08-17T17:39:53.632 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0015721699999999999 2024-08-17T17:39:53.632 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.632 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.632 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.632 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:38:17.554117+0000", 2024-08-17T17:39:53.632 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000198691 2024-08-17T17:39:53.632 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.632 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.632 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.632 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:38:17.670582+0000", 2024-08-17T17:39:53.632 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.116465289 2024-08-17T17:39:53.632 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.632 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.632 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:39:53.633 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:38:17.670637+0000", 2024-08-17T17:39:53.633 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4781000000000003e-05 2024-08-17T17:39:53.633 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.633 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.633 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:39:53.633 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:38:17.670678+0000", 2024-08-17T17:39:53.633 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1124000000000001e-05 2024-08-17T17:39:53.633 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.633 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:39:53.633 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.633 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.633 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.633 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36354.0:56076 9.e 9:7ded9cee:::benchmark_data_smithi003_261051_object56075:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-08-17T17:39:53.633 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:38:19.179354+0000", 2024-08-17T17:39:53.634 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 94.318399779999993, 2024-08-17T17:39:53.634 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.11623832100000001, 2024-08-17T17:39:53.634 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:39:53.634 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:39:53.634 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:39:53.634 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36354", 2024-08-17T17:39:53.634 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/1330158990", 2024-08-17T17:39:53.634 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 56076 2024-08-17T17:39:53.634 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.634 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:39:53.634 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.634 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:39:53.634 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:38:19.179354+0000", 2024-08-17T17:39:53.634 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.635 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.635 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.635 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:39:53.635 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:38:19.179354+0000", 2024-08-17T17:39:53.635 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:39:53.635 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.635 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.635 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:39:53.635 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:38:19.179353+0000", 2024-08-17T17:39:53.635 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:39:53.635 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.635 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.635 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:39:53.635 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:38:19.179449+0000", 2024-08-17T17:39:53.635 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6311999999999995e-05 2024-08-17T17:39:53.636 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.636 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.636 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:39:53.636 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:38:19.179465+0000", 2024-08-17T17:39:53.636 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6198e-05 2024-08-17T17:39:53.636 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.636 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.636 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:39:53.636 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:38:19.179471+0000", 2024-08-17T17:39:53.636 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9800000000000003e-06 2024-08-17T17:39:53.636 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.636 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.636 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:39:53.636 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:38:19.179532+0000", 2024-08-17T17:39:53.636 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0952999999999997e-05 2024-08-17T17:39:53.637 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.637 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.637 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:39:53.637 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:38:19.179681+0000", 2024-08-17T17:39:53.637 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014834399999999999 2024-08-17T17:39:53.637 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.637 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.637 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,7", 2024-08-17T17:39:53.637 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:38:19.179839+0000", 2024-08-17T17:39:53.637 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000158517 2024-08-17T17:39:53.637 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.637 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.637 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:39:53.638 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:38:19.183700+0000", 2024-08-17T17:39:53.638 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0038613509999999998 2024-08-17T17:39:53.638 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.638 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.638 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.638 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:38:19.183739+0000", 2024-08-17T17:39:53.638 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8379e-05 2024-08-17T17:39:53.638 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.638 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.638 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:39:53.638 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:38:19.295514+0000", 2024-08-17T17:39:53.638 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.11177476 2024-08-17T17:39:53.638 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.638 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.639 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:39:53.639 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:38:19.295559+0000", 2024-08-17T17:39:53.639 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5185999999999997e-05 2024-08-17T17:39:53.639 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:39:53.639 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:39:53.639 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:39:53.639 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:38:19.295593+0000", 2024-08-17T17:39:53.639 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3992000000000003e-05 2024-08-17T17:39:53.639 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.639 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:39:53.639 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.639 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:39:53.639 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:39:53.639 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:39:53.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:53.666 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:53.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:53.767 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:53.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:53 smithi003 ceph-mon[100615]: pgmap v1063: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.8 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 738 op/s; 13376/284490 objects degraded (4.702%); 5177/284490 objects misplaced (1.820%); 1.6 MiB/s, 25 objects/s recovering 2024-08-17T17:39:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:39:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:39:53.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:53 smithi003 ceph-mon[103344]: pgmap v1063: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.8 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 738 op/s; 13376/284490 objects degraded (4.702%); 5177/284490 objects misplaced (1.820%); 1.6 MiB/s, 25 objects/s recovering 2024-08-17T17:39:53.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:39:53.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:39:53.792 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:53.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:53.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:53.894 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:53.924 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:53.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:53.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:53.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:53 smithi053 ceph-mon[89963]: pgmap v1063: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.8 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 738 op/s; 13376/284490 objects degraded (4.702%); 5177/284490 objects misplaced (1.820%); 1.6 MiB/s, 25 objects/s recovering 2024-08-17T17:39:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:39:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:39:53.978 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-08-17T17:39:54.026 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:54.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:54.085 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:54.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:54.186 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:54.214 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:54.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:54.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:54.316 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:54.364 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:54.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:54.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:54.466 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:54.492 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:54.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:54.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:54.541 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-17T17:39:54.594 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:54.636 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:54.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:54.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:54.718 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:39:54.738 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:54.763 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:54.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:54.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:54.866 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:54.889 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:54.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:54.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:54.991 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:55.017 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:55.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:55.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:55.122 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:55.152 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:55.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:55.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:55.254 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:39:55.279 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:55.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:55.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:39:55.381 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:55.439 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:39:55.444 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:55.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:55.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:55.546 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:55.581 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:55.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:55.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:55.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:55.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:55.730 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:55.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:55.734 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:55 smithi003 ceph-mon[103344]: pgmap v1064: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 699 op/s; 13376/281205 objects degraded (4.757%); 5177/281205 objects misplaced (1.841%); 1.1 MiB/s, 17 objects/s recovering 2024-08-17T17:39:55.735 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:55 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 13376/281205 objects degraded (4.757%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:39:55.735 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:55 smithi003 ceph-mon[100615]: pgmap v1064: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 699 op/s; 13376/281205 objects degraded (4.757%); 5177/281205 objects misplaced (1.841%); 1.1 MiB/s, 17 objects/s recovering 2024-08-17T17:39:55.735 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:55 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 13376/281205 objects degraded (4.757%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:39:55.735 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:39:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:39:55] "GET /metrics HTTP/1.1" 200 36523 "" "Prometheus/2.51.0" 2024-08-17T17:39:55.835 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:55.880 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:55 smithi053 ceph-mon[89963]: pgmap v1064: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 699 op/s; 13376/281205 objects degraded (4.757%); 5177/281205 objects misplaced (1.841%); 1.1 MiB/s, 17 objects/s recovering 2024-08-17T17:39:55.880 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:55 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 13376/281205 objects degraded (4.757%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:39:55.885 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:55.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:55.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:55.986 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:56.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:56.024 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:56.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:56.125 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:56.179 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:56.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:56.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:56.281 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:56.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:56.321 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:56.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:56.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:56.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:56.467 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:56.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:56.568 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:39:56.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:56.598 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:56.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:39:56.625 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:39:56.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:56.776 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:56.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:56.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:56.878 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:56.920 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:56.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:56.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:56.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:56 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1427506022' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:39:56.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:56 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:39:57.022 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:39:57.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:56 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1427506022' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:39:57.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:56 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:39:57.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:56 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1427506022' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:39:57.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:56 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:39:57.047 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:57.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:57.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:39:57.149 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:57.193 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:57.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:57.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:57.295 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:57.351 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:57.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:57.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:57.426 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-08-17T17:39:57.453 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:57.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:57.502 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:57.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:57.603 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:57.650 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:57.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:57.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:57.752 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:57.776 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:57.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:57.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:57.878 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:57.924 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:57.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:57.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:57.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:57 smithi053 ceph-mon[89963]: pgmap v1065: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 799 op/s; 13471/273996 objects degraded (4.916%); 4965/273996 objects misplaced (1.812%); 1.7 MiB/s, 26 objects/s recovering 2024-08-17T17:39:57.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:57 smithi053 ceph-mon[89963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:39:57.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:57 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:39:57.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:57 smithi053 ceph-mon[89963]: osdmap e334: 8 total, 8 up, 7 in 2024-08-17T17:39:58.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:57 smithi003 ceph-mon[103344]: pgmap v1065: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 799 op/s; 13471/273996 objects degraded (4.916%); 4965/273996 objects misplaced (1.812%); 1.7 MiB/s, 26 objects/s recovering 2024-08-17T17:39:58.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:57 smithi003 ceph-mon[103344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:39:58.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:57 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:39:58.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:57 smithi003 ceph-mon[103344]: osdmap e334: 8 total, 8 up, 7 in 2024-08-17T17:39:58.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:57 smithi003 ceph-mon[100615]: pgmap v1065: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 799 op/s; 13471/273996 objects degraded (4.916%); 4965/273996 objects misplaced (1.812%); 1.7 MiB/s, 26 objects/s recovering 2024-08-17T17:39:58.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:57 smithi003 ceph-mon[100615]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:39:58.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:57 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:39:58.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:57 smithi003 ceph-mon[100615]: osdmap e334: 8 total, 8 up, 7 in 2024-08-17T17:39:58.026 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:58.065 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:58.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:58.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:58.167 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:58.223 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:58.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:58.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:58.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:58.362 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:58.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:58.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:58.464 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:39:58.499 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:58.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:58.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:39:58.601 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:58.630 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:58.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:58.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:58.657 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:39:58.658 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2024-08-17T17:39:58.658 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 0 2024-08-17T17:39:58.658 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:39:58.732 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:39:58.778 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:58.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:58.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:39:58.880 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:39:58.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:58.922 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:58.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:39:59.023 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:59.053 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:59.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:59.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:59.155 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:59.199 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:59.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:59.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:59.294 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-17T17:39:59.301 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:59.343 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:59.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:59.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:59.445 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:59.478 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:59.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:59.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:59.580 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:59.617 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:59.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:59.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:59.665 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:39:59.719 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:39:59.750 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:39:59.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:59.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:39:59.852 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:39:59.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:39:59.886 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:39:59.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:39:59.987 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:00.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:59 smithi003 ceph-mon[100615]: pgmap v1067: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 695 op/s; 13471/273996 objects degraded (4.916%); 4965/273996 objects misplaced (1.812%); 1.3 MiB/s, 21 objects/s recovering 2024-08-17T17:40:00.027 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:59 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:40:00.027 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:39:59 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:40:00.027 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:59 smithi003 ceph-mon[103344]: pgmap v1067: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 695 op/s; 13471/273996 objects degraded (4.916%); 4965/273996 objects misplaced (1.812%); 1.3 MiB/s, 21 objects/s recovering 2024-08-17T17:40:00.027 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:59 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:40:00.027 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:39:59 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:40:00.059 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:59 smithi053 ceph-mon[89963]: pgmap v1067: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 695 op/s; 13471/273996 objects degraded (4.916%); 4965/273996 objects misplaced (1.812%); 1.3 MiB/s, 21 objects/s recovering 2024-08-17T17:40:00.059 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:59 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:40:00.059 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:39:59 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:40:00.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:00.066 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:00.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:00.167 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:00.224 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:40:00.228 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:00.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:00.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:00.331 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:00.357 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:00.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:00.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:00.458 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:00.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:40:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:40:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:40:00.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:00.504 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:00.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:00.605 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:00.630 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:00.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:00.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:00.725 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:40:00.725 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:40:00.731 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:00.776 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:40:00.776 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:40:00.797 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:00.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:00.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:00.808 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:40:00.808 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:40:00.864 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:40:00.864 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:40:00.898 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:00.900 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:40:00.900 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:40:00.925 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:00.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:00.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:00.953 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:40:00.953 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:40:00.986 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:40:00.986 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:40:00.988 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:00 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 13471/273996 objects degraded (4.916%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:40:00.988 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:00 smithi003 ceph-mon[100615]: Health detail: HEALTH_WARN noscrub flag(s) set; Degraded data redundancy: 13471/273996 objects degraded (4.916%), 3 pgs degraded 2024-08-17T17:40:00.988 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:00 smithi003 ceph-mon[100615]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-08-17T17:40:00.988 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:00 smithi003 ceph-mon[100615]: [WRN] PG_DEGRADED: Degraded data redundancy: 13471/273996 objects degraded (4.916%), 3 pgs degraded 2024-08-17T17:40:00.989 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:00 smithi003 ceph-mon[100615]: pg 9.4 is active+recovery_wait+undersized+degraded+remapped, acting [1,3] 2024-08-17T17:40:00.989 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:00 smithi003 ceph-mon[100615]: pg 9.c is active+recovery_wait+undersized+degraded+remapped, acting [3,7] 2024-08-17T17:40:00.989 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:00 smithi003 ceph-mon[100615]: pg 9.e is active+recovering+undersized+degraded+remapped, acting [1,7] 2024-08-17T17:40:00.989 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:00 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/4066395288' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:40:00.989 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:00 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 13471/273996 objects degraded (4.916%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:40:00.989 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:00 smithi003 ceph-mon[103344]: Health detail: HEALTH_WARN noscrub flag(s) set; Degraded data redundancy: 13471/273996 objects degraded (4.916%), 3 pgs degraded 2024-08-17T17:40:00.989 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:00 smithi003 ceph-mon[103344]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-08-17T17:40:00.989 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:00 smithi003 ceph-mon[103344]: [WRN] PG_DEGRADED: Degraded data redundancy: 13471/273996 objects degraded (4.916%), 3 pgs degraded 2024-08-17T17:40:00.989 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:00 smithi003 ceph-mon[103344]: pg 9.4 is active+recovery_wait+undersized+degraded+remapped, acting [1,3] 2024-08-17T17:40:00.989 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:00 smithi003 ceph-mon[103344]: pg 9.c is active+recovery_wait+undersized+degraded+remapped, acting [3,7] 2024-08-17T17:40:00.989 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:00 smithi003 ceph-mon[103344]: pg 9.e is active+recovering+undersized+degraded+remapped, acting [1,7] 2024-08-17T17:40:00.989 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:00 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/4066395288' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:40:01.016 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:00 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 13471/273996 objects degraded (4.916%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:40:01.016 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:00 smithi053 ceph-mon[89963]: Health detail: HEALTH_WARN noscrub flag(s) set; Degraded data redundancy: 13471/273996 objects degraded (4.916%), 3 pgs degraded 2024-08-17T17:40:01.016 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:00 smithi053 ceph-mon[89963]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-08-17T17:40:01.016 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:00 smithi053 ceph-mon[89963]: [WRN] PG_DEGRADED: Degraded data redundancy: 13471/273996 objects degraded (4.916%), 3 pgs degraded 2024-08-17T17:40:01.016 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:00 smithi053 ceph-mon[89963]: pg 9.4 is active+recovery_wait+undersized+degraded+remapped, acting [1,3] 2024-08-17T17:40:01.016 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:00 smithi053 ceph-mon[89963]: pg 9.c is active+recovery_wait+undersized+degraded+remapped, acting [3,7] 2024-08-17T17:40:01.016 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:00 smithi053 ceph-mon[89963]: pg 9.e is active+recovering+undersized+degraded+remapped, acting [1,7] 2024-08-17T17:40:01.016 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:00 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/4066395288' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:40:01.018 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:40:01.018 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:40:01.028 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:01.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:01.078 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:01.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:01.179 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:40:01.206 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:01.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:01.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:40:01.308 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:01.335 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:01.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:01.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:01.422 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:40:01.422 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:40:01.437 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:01.462 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:01.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:01.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:01.564 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:01.608 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:01.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:01.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:01.710 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:40:01.737 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:01.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:01.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:40:01.839 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:01.862 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:01.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:01.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:01.964 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:01.988 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:01.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:01.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:02.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:01 smithi003 ceph-mon[103344]: pgmap v1068: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.5 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 708 op/s; 13471/267831 objects degraded (5.030%); 4965/267831 objects misplaced (1.854%); 1.3 MiB/s, 21 objects/s recovering 2024-08-17T17:40:02.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:01 smithi003 ceph-mon[100615]: pgmap v1068: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.5 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 708 op/s; 13471/267831 objects degraded (5.030%); 4965/267831 objects misplaced (1.854%); 1.3 MiB/s, 21 objects/s recovering 2024-08-17T17:40:02.089 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:02.124 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:01 smithi053 ceph-mon[89963]: pgmap v1068: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.5 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 708 op/s; 13471/267831 objects degraded (5.030%); 4965/267831 objects misplaced (1.854%); 1.3 MiB/s, 21 objects/s recovering 2024-08-17T17:40:02.127 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:02.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:02.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:02.161 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-17T17:40:02.228 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:02.257 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:02.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:02.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:02.358 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:02.388 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:02.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:02.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:02.490 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:02.506 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:40:02.525 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:02.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:02.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:02.627 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:02.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:02.654 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:02.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:02.755 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:02.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:02.796 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:02.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:02.897 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:02.924 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:02.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:02.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:03.026 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:03.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:03.051 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:03.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:03.070 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-08-17T17:40:03.141 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:40:03.153 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:03.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:03.207 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:03.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:03.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:03.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:03.356 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:03.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:03.457 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:03.502 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:03.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:03.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:03.604 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:03.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:03.651 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:03.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:03.752 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:03.801 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:03.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:03.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:03.882 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:03 smithi053 ceph-mon[89963]: pgmap v1069: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 708 op/s; 13571/263205 objects degraded (5.156%); 4755/263205 objects misplaced (1.807%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:03.903 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:03.928 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:03.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:03.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:04.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:03 smithi003 ceph-mon[100615]: pgmap v1069: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 708 op/s; 13571/263205 objects degraded (5.156%); 4755/263205 objects misplaced (1.807%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:04.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:03 smithi003 ceph-mon[103344]: pgmap v1069: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 708 op/s; 13571/263205 objects degraded (5.156%); 4755/263205 objects misplaced (1.807%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:04.030 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:04.105 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:04.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:04.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:04.211 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:04.257 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:04.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:04.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:04.359 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:04.408 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:04.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:04.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:04.471 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:40:04.471 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:40:04.510 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:04.550 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:04.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:04.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:04.583 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:40:04.624 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:40:04.652 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:04.703 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:04.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:04.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:04.805 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:04.829 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:04.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:04.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:04.934 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:05.003 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:05.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:05.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:05.105 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:05.133 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-17T17:40:05.158 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:05.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:05.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:05.261 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:05.292 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:05.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:05.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:05.393 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:05.429 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:05.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:05.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:05.468 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:40:05.530 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:05.557 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:05.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:05.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:05.659 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:05.685 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:05.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:05.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:05.788 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:05.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:05.817 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:05.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:05.861 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:05 smithi053 ceph-mon[89963]: pgmap v1070: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 711 op/s; 13571/259839 objects degraded (5.223%); 4755/259839 objects misplaced (1.830%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:05.861 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:05 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 13571/259839 objects degraded (5.223%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:05.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:05 smithi003 ceph-mon[103344]: pgmap v1070: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 711 op/s; 13571/259839 objects degraded (5.223%); 4755/259839 objects misplaced (1.830%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:05.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:05 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 13571/259839 objects degraded (5.223%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:05.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:05 smithi003 ceph-mon[100615]: pgmap v1070: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 711 op/s; 13571/259839 objects degraded (5.223%); 4755/259839 objects misplaced (1.830%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:05.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:05 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 13571/259839 objects degraded (5.223%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:05.875 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:40:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:40:05] "GET /metrics HTTP/1.1" 200 36523 "" "Prometheus/2.51.0" 2024-08-17T17:40:05.918 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:05.942 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:05.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:05.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:06.043 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:06.090 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:06.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:06.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:06.192 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:06.254 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:06.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:06.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:06.355 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:06.392 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:06.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:06.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:06.494 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:06.535 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:06.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:06.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:06.636 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:06.659 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:06.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:06.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:06.761 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:06.783 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-08-17T17:40:06.788 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:06.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:06.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:06.890 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:06.930 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:06.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:06.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:06.944 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:06 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2003506482' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:40:07.032 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:07.056 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:07.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:07.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:07.158 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:07.185 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:07.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:07.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:07.235 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:40:07.235 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:40:07.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:06 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2003506482' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:40:07.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:06 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2003506482' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:40:07.287 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:07.318 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:07.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:07.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:07.420 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:07.447 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:07.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:07.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:07.549 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:07.581 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:07.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:07.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:07.683 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:07.724 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:07.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:07.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:07.826 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:07.846 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:07 smithi053 ceph-mon[89963]: pgmap v1071: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 720 op/s; 13692/252369 objects degraded (5.425%); 4526/252369 objects misplaced (1.793%); 1.4 MiB/s, 21 objects/s recovering 2024-08-17T17:40:07.846 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:07 smithi053 ceph-mon[89963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:40:07.846 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:07 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2003506482' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:40:07.846 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:07 smithi053 ceph-mon[89963]: osdmap e335: 8 total, 8 up, 7 in 2024-08-17T17:40:07.866 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:07.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:07.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:07.910 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-17T17:40:07.968 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:08.003 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:08.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:08.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:08.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:07 smithi003 ceph-mon[103344]: pgmap v1071: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 720 op/s; 13692/252369 objects degraded (5.425%); 4526/252369 objects misplaced (1.793%); 1.4 MiB/s, 21 objects/s recovering 2024-08-17T17:40:08.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:07 smithi003 ceph-mon[103344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:40:08.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:07 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2003506482' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:40:08.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:07 smithi003 ceph-mon[103344]: osdmap e335: 8 total, 8 up, 7 in 2024-08-17T17:40:08.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:07 smithi003 ceph-mon[100615]: pgmap v1071: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 720 op/s; 13692/252369 objects degraded (5.425%); 4526/252369 objects misplaced (1.793%); 1.4 MiB/s, 21 objects/s recovering 2024-08-17T17:40:08.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:07 smithi003 ceph-mon[100615]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:40:08.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:07 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2003506482' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:40:08.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:07 smithi003 ceph-mon[100615]: osdmap e335: 8 total, 8 up, 7 in 2024-08-17T17:40:08.105 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:08.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:08.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:08.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:08.249 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:40:08.256 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:08.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:08.293 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:08.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:08.395 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:08.422 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:08.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:08.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:08.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:08.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:08.566 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:08.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:08.667 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:08.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:08.709 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:08.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:08.810 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:08.849 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:08.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:08.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:08.951 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:08.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:08.987 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:08.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:09.088 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:09.114 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:09.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:09.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:09.216 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:09.246 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:09.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:09.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:09.348 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:09.384 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:09.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:09.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:09.424 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:40:09.424 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:40:09.424 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:09 smithi053 ceph-mon[89963]: pgmap v1073: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 720 op/s; 13692/252369 objects degraded (5.425%); 4526/252369 objects misplaced (1.793%); 1.4 MiB/s, 21 objects/s recovering 2024-08-17T17:40:09.485 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:09.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:40:09.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:40:09.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:09 smithi003 ceph-mon[100615]: pgmap v1073: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 720 op/s; 13692/252369 objects degraded (5.425%); 4526/252369 objects misplaced (1.793%); 1.4 MiB/s, 21 objects/s recovering 2024-08-17T17:40:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:40:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:40:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:09 smithi003 ceph-mon[103344]: pgmap v1073: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 720 op/s; 13692/252369 objects degraded (5.425%); 4526/252369 objects misplaced (1.793%); 1.4 MiB/s, 21 objects/s recovering 2024-08-17T17:40:09.526 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:09.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:09.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:09.628 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:09.664 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:09.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:09.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:09.766 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:09.795 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:09.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:09.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:09.897 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:09.932 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:09.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:09.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:10.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:10.067 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:40:10.067 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:40:10.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:10.069 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:10.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:10.170 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:10.195 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:10.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:10.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:10.214 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:10 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 13692/252369 objects degraded (5.425%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:10.297 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:40:10.325 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:10.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:10.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:40:10.426 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:10.452 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:10.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:10.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:10.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:40:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:40:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:40:10.522 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:10 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 13692/252369 objects degraded (5.425%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:10.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:10 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 13692/252369 objects degraded (5.425%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:10.554 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:10.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:10.591 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:10.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:10.693 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:10.718 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:10.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:10.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:10.721 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-17T17:40:10.820 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:10.856 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:10.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:10.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:10.958 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:10.991 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:10.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:10.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:11.093 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:11.099 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:40:11.125 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:11.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:11.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:11.227 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:11.266 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:11.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:11.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:11.368 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:11.396 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:11.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:11.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:11.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:11 smithi053 ceph-mon[89963]: pgmap v1074: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 715 op/s; 13692/246363 objects degraded (5.558%); 4526/246363 objects misplaced (1.837%); 1.4 MiB/s, 21 objects/s recovering 2024-08-17T17:40:11.498 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:11.522 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:11 smithi003 ceph-mon[100615]: pgmap v1074: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 715 op/s; 13692/246363 objects degraded (5.558%); 4526/246363 objects misplaced (1.837%); 1.4 MiB/s, 21 objects/s recovering 2024-08-17T17:40:11.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:11 smithi003 ceph-mon[103344]: pgmap v1074: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 715 op/s; 13692/246363 objects degraded (5.558%); 4526/246363 objects misplaced (1.837%); 1.4 MiB/s, 21 objects/s recovering 2024-08-17T17:40:11.526 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:11.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:11.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:11.628 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:11.664 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:11.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:11.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:11.766 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:11.802 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:11.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:11.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:11.904 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:11.939 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:11.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:11.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:12.041 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:12.082 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:12.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:12.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:12.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:12.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:12.227 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:12.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:12.328 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:12.356 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:12.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:12.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:12.458 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:12.484 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:12.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:12.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:12.586 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:12.607 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:12.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:12.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:12.709 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:12.748 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:12.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:12.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:12.838 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:40:12.839 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:40:12.850 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:12.881 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:12.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:12.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:12.983 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:13.025 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:13.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:13.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:13.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:13.164 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:13.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:13.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:13.266 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:13.310 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:13.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:13.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:13.412 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:13.471 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:13.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:13.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:13.573 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:13.609 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:13 smithi053 ceph-mon[89963]: pgmap v1075: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 719 op/s; 13835/241620 objects degraded (5.726%); 4277/241620 objects misplaced (1.770%); 1.3 MiB/s, 21 objects/s recovering 2024-08-17T17:40:13.612 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:13.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:13.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:13.708 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:13 smithi003 ceph-mon[100615]: pgmap v1075: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 719 op/s; 13835/241620 objects degraded (5.726%); 4277/241620 objects misplaced (1.770%); 1.3 MiB/s, 21 objects/s recovering 2024-08-17T17:40:13.708 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:13 smithi003 ceph-mon[103344]: pgmap v1075: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 719 op/s; 13835/241620 objects degraded (5.726%); 4277/241620 objects misplaced (1.770%); 1.3 MiB/s, 21 objects/s recovering 2024-08-17T17:40:13.714 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:13.757 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:13.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:13.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:13.859 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:13.902 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:13.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:13.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:14.005 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:40:14.031 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:14.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:14.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:40:14.107 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-17T17:40:14.133 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:14.182 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:14.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:14.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:14.284 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:14.319 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:14.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:14.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:14.421 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:14.450 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:14.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:14.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:14.460 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:40:14.553 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:14.590 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:14.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:14.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:14.692 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:14.745 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:14.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:14.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:14.847 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:14.892 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:14.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:14.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:14.994 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:15.020 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:15.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:15.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:15.122 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:15.145 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:15.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:15.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:15.247 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:15.288 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:15.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:15.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:15.391 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:15.416 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:15.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:15.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:15.427 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:15 smithi003 ceph-mon[103344]: pgmap v1076: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 722 op/s; 13835/238161 objects degraded (5.809%); 4277/238161 objects misplaced (1.796%); 1.3 MiB/s, 21 objects/s recovering 2024-08-17T17:40:15.427 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:15 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 13835/238161 objects degraded (5.809%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:15.428 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:15 smithi003 ceph-mon[100615]: pgmap v1076: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 722 op/s; 13835/238161 objects degraded (5.809%); 4277/238161 objects misplaced (1.796%); 1.3 MiB/s, 21 objects/s recovering 2024-08-17T17:40:15.428 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:15 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 13835/238161 objects degraded (5.809%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:15.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:15.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:15.553 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:15.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:15.654 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:15.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:15.705 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:15.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:15.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:15 smithi053 ceph-mon[89963]: pgmap v1076: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 722 op/s; 13835/238161 objects degraded (5.809%); 4277/238161 objects misplaced (1.796%); 1.3 MiB/s, 21 objects/s recovering 2024-08-17T17:40:15.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:15 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 13835/238161 objects degraded (5.809%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:15.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:40:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:40:15] "GET /metrics HTTP/1.1" 200 36524 "" "Prometheus/2.51.0" 2024-08-17T17:40:15.806 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:15.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:15.854 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:15.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:15.955 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:15.980 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:15.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:15.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:16.082 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:16.106 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:16.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:16.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:16.207 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:16.231 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:16.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:16.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:16.333 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:16.375 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:16.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:16.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:16.396 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:40:16.477 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:16.502 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:16.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:16.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:16.603 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:16.606 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:40:16.628 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:16.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:16.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:16.639 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-08-17T17:40:16.730 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:16.752 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:16.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:16.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:16.854 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:16.868 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:40:16.868 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:40:16.888 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:16.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:16.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:16.990 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:17.014 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:17.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:17.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:17.116 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:17.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:17.157 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:17.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:17.258 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:17.286 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:17.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:17.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:17.388 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:17.410 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:17.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:17.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:17.512 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:17.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:17.552 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:17.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:17.555 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:17 smithi053 ceph-mon[89963]: pgmap v1077: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.7 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 738 op/s; 13942/230220 objects degraded (6.056%); 4055/230220 objects misplaced (1.761%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:17.589 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-17T17:40:17.653 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:17.690 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:17.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:17.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:17.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:17 smithi003 ceph-mon[103344]: pgmap v1077: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.7 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 738 op/s; 13942/230220 objects degraded (6.056%); 4055/230220 objects misplaced (1.761%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:17.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:17 smithi003 ceph-mon[100615]: pgmap v1077: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.7 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 738 op/s; 13942/230220 objects degraded (6.056%); 4055/230220 objects misplaced (1.761%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:17.792 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:17.824 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:17.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:17.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:17.921 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:40:17.926 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:17.958 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:17.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:17.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:18.060 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:18.099 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:18.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:18.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:18.201 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:18.223 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:18.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:18.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:18.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:18.362 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:18.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:18.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:18.463 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:18.507 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:18.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:18.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:18.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:18.647 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:18.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:18.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:18.749 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:40:18.774 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:18.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:18.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:40:18.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:18.929 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:18.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:18.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:19.031 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:19.079 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:19.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:19.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:19.181 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:19.211 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:19.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:19.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:19.313 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:40:19.338 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:19.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:19.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:40:19.440 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:19.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:19.470 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:19.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:19.571 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:40:19.600 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:19.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:19.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:40:19.698 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:40:19.698 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:40:19.702 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:19.740 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:19.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:19.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:19.842 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:19.887 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:40:19.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:19.915 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:19.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:19.951 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:40:19.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:19 smithi053 ceph-mon[89963]: pgmap v1078: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.7 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 643 op/s; 13942/230220 objects degraded (6.056%); 4055/230220 objects misplaced (1.761%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:40:19.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:19 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:40:19.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:19 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:40:19.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:19 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:40:19.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:19 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:40:19.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:19 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:40:20.016 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:20.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:20.055 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:20.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:20.059 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:19 smithi003 ceph-mon[103344]: pgmap v1078: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.7 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 643 op/s; 13942/230220 objects degraded (6.056%); 4055/230220 objects misplaced (1.761%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:40:20.059 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:19 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:40:20.059 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:19 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:40:20.059 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:19 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:40:20.059 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:19 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:40:20.059 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:19 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:40:20.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:19 smithi003 ceph-mon[100615]: pgmap v1078: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.7 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 643 op/s; 13942/230220 objects degraded (6.056%); 4055/230220 objects misplaced (1.761%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:40:20.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:19 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:40:20.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:19 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:40:20.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:19 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:40:20.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:19 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:40:20.060 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:19 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:40:20.156 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:20.179 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:20.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:20.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:20.281 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:20.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:20.307 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:20.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:20.352 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-17T17:40:20.353 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:40:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:40:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:40:20.408 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:20.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:20.446 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:20.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:20.547 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:20.580 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:20.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:20.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:20.682 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:20.693 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:40:20.724 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:20.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:20.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:20.825 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:20.849 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:20.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:20.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:20.951 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:20.984 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:20.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:20.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:21.064 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:20 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 13942/230220 objects degraded (6.056%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:21.064 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:20 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 13942/230220 objects degraded (6.056%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:21.087 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:21.175 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:21.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:21.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:21.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:20 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 13942/230220 objects degraded (6.056%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:21.277 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:40:21.301 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:21.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:21.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:40:21.403 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:40:21.428 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:21.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:21.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:40:21.530 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:21.556 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:21.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:21.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:21.658 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:21.683 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:21.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:21.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:21.785 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:21.819 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:21.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:21.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:21.921 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:21.958 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:21.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:21.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:21.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:21 smithi053 ceph-mon[89963]: pgmap v1079: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 781 op/s; 13942/224223 objects degraded (6.218%); 4055/224223 objects misplaced (1.808%); 1.2 MiB/s, 18 objects/s recovering 2024-08-17T17:40:22.060 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:22.095 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:22.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:22.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:22.104 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:21 smithi003 ceph-mon[103344]: pgmap v1079: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 781 op/s; 13942/224223 objects degraded (6.218%); 4055/224223 objects misplaced (1.808%); 1.2 MiB/s, 18 objects/s recovering 2024-08-17T17:40:22.104 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:21 smithi003 ceph-mon[100615]: pgmap v1079: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 781 op/s; 13942/224223 objects degraded (6.218%); 4055/224223 objects misplaced (1.808%); 1.2 MiB/s, 18 objects/s recovering 2024-08-17T17:40:22.197 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:22.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:22.222 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:22.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:22.323 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:22.350 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:22.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:22.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:22.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:22.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:22.488 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:22.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:22.589 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:22.626 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:22.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:22.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:22.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:22.774 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:22.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:22.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:22.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:22.916 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:22.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:22.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:23.018 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:23.055 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:23.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:23.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:23.157 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:23.203 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:23.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:23.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:23.248 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:40:23.248 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:40:23.306 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:23.334 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:23.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:23.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:23.439 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:23.474 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:23.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:23.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:23.576 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:40:23.601 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:23.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:23.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:40:23.703 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:23.733 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:23.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:23.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:23.835 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:23.878 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:23.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:23.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:23.903 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-17T17:40:23.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:23 smithi053 ceph-mon[89963]: pgmap v1080: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 742 op/s; 14054/219618 objects degraded (6.399%); 3829/219618 objects misplaced (1.743%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T17:40:23.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:40:23.981 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:24.004 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:24.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:24.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:24.106 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:24.111 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:40:24.111 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:40:24.112 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:23 smithi003 ceph-mon[100615]: pgmap v1080: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 742 op/s; 14054/219618 objects degraded (6.399%); 3829/219618 objects misplaced (1.743%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T17:40:24.112 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:40:24.112 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:23 smithi003 ceph-mon[103344]: pgmap v1080: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 742 op/s; 14054/219618 objects degraded (6.399%); 3829/219618 objects misplaced (1.743%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T17:40:24.112 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:40:24.157 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:24.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:24.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:24.161 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:40:24.162 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:40:24.199 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:40:24.199 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:40:24.245 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:40:24.246 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:40:24.259 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:24.285 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:40:24.286 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:40:24.316 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:24.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:24.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:24.318 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:40:24.319 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:40:24.355 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:40:24.355 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:40:24.388 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:40:24.388 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:40:24.417 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:24.442 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:24.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:24.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:24.543 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:24.569 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:24.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:24.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:24.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:24.709 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:24.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:24.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:24.811 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:40:24.834 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:24.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:24.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:40:24.936 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:24.978 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:24.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:24.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:25.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:24 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1608625005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:40:25.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:24 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1608625005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:40:25.080 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:25.121 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:25.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:25.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:25.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:24 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1608625005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:40:25.222 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:25.252 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:25.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:25.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:25.353 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:25.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:25.426 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:25.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:25.527 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:25.564 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:25.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:25.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:25.666 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:25.692 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:25.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:25.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:25.794 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:25.818 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:25.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:25.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:25.902 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:25 smithi003 ceph-mon[103344]: pgmap v1081: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 610 op/s; 14054/219618 objects degraded (6.399%); 3829/219618 objects misplaced (1.743%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:40:25.902 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:25 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 14054/219618 objects degraded (6.399%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:25.903 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:25 smithi003 ceph-mon[100615]: pgmap v1081: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 610 op/s; 14054/219618 objects degraded (6.399%); 3829/219618 objects misplaced (1.743%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:40:25.903 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:25 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 14054/219618 objects degraded (6.399%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:25.904 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:40:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:40:25] "GET /metrics HTTP/1.1" 200 36523 "" "Prometheus/2.51.0" 2024-08-17T17:40:25.920 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:25.946 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:25.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:25.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:26.048 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:26.103 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:26.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:26.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:26.205 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:26.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:25 smithi053 ceph-mon[89963]: pgmap v1081: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 610 op/s; 14054/219618 objects degraded (6.399%); 3829/219618 objects misplaced (1.743%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:40:26.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:25 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 14054/219618 objects degraded (6.399%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:26.229 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:26.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:26.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:26.331 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:26.373 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:26.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:26.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:26.475 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:26.516 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:26.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:26.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:26.618 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:26.658 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:26.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:26.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:26.760 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:26.784 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:26.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:26.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:26.885 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:40:26.910 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:26.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:26.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:40:27.012 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:27.047 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:27.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:27.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:27.149 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:27.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:27.173 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:27.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:27.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:26 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2299206717' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:40:27.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:26 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:40:27.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:26 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2299206717' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:40:27.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:26 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:40:27.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:26 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2299206717' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:40:27.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:26 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:40:27.275 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:27.300 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:27.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:27.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:27.402 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:27.438 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:27.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:27.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:27.541 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:27.581 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:27.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:27.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:27.683 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:27.722 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:27.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:27.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:27.824 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:27.863 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:27.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:27.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:27.965 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:28.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:28.006 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:28.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:28.107 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:28.132 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:28.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:28.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:28.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:27 smithi053 ceph-mon[89963]: pgmap v1082: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 832 op/s; 14177/208179 objects degraded (6.810%); 3590/208179 objects misplaced (1.724%); 1.8 MiB/s, 28 objects/s recovering 2024-08-17T17:40:28.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:27 smithi053 ceph-mon[89963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:40:28.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:27 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:40:28.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:27 smithi053 ceph-mon[89963]: osdmap e336: 8 total, 8 up, 7 in 2024-08-17T17:40:28.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:28.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:27 smithi003 ceph-mon[103344]: pgmap v1082: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 832 op/s; 14177/208179 objects degraded (6.810%); 3590/208179 objects misplaced (1.724%); 1.8 MiB/s, 28 objects/s recovering 2024-08-17T17:40:28.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:27 smithi003 ceph-mon[103344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:40:28.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:27 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:40:28.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:27 smithi003 ceph-mon[103344]: osdmap e336: 8 total, 8 up, 7 in 2024-08-17T17:40:28.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:27 smithi003 ceph-mon[100615]: pgmap v1082: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 832 op/s; 14177/208179 objects degraded (6.810%); 3590/208179 objects misplaced (1.724%); 1.8 MiB/s, 28 objects/s recovering 2024-08-17T17:40:28.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:27 smithi003 ceph-mon[100615]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:40:28.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:27 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:40:28.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:27 smithi003 ceph-mon[100615]: osdmap e336: 8 total, 8 up, 7 in 2024-08-17T17:40:28.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:28.275 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:28.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:28.376 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:28.416 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:28.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:28.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:28.517 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:40:28.545 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:28.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:28.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:40:28.647 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:28.696 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:28.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:28.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:28.769 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-08-17T17:40:28.799 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:28.839 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:28.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:28.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:28.942 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:28.983 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:28.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:28.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:29.087 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:29.126 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:29.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:29.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:29.228 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:29.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:29.270 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:29.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:29.371 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:29.411 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:29.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:29.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:29.513 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:29.558 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:29.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:29.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:29.661 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:29.703 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:29.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:29.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:29.805 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:29.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:29.849 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:29.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:29.950 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:29.977 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:29.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:29.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:30.079 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:30.087 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:29 smithi003 ceph-mon[100615]: pgmap v1084: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 734 op/s; 14177/208179 objects degraded (6.810%); 3590/208179 objects misplaced (1.724%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:30.087 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:29 smithi003 ceph-mon[103344]: pgmap v1084: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 734 op/s; 14177/208179 objects degraded (6.810%); 3590/208179 objects misplaced (1.724%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:30.131 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:30.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:30.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:30.214 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:29 smithi053 ceph-mon[89963]: pgmap v1084: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 734 op/s; 14177/208179 objects degraded (6.810%); 3590/208179 objects misplaced (1.724%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:30.233 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:30.261 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:30.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:30.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:30.363 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:30.403 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:30.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:30.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:30.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:40:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:40:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:40:30.505 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:30.543 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-08-17T17:40:30.562 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:30.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:30.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:30.575 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:40:30.664 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:30.716 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:30.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:30.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:30.817 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:30.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:30.872 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:30.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:30.973 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:31.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:30 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 14177/208179 objects degraded (6.810%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:31.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:30 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 14177/208179 objects degraded (6.810%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:31.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:31.034 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:31.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:31.135 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:31.163 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:31.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:31.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:31.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:30 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 14177/208179 objects degraded (6.810%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:31.265 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:31.290 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:31.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:31.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:31.391 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:31.445 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:31.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:31.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:31.547 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:31.586 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:31.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:31.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:31.599 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:40:31.688 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:31.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:31.748 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:31.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:31.849 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:31.875 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:31.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:31.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:31.976 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:40:31.992 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:31 smithi003 ceph-mon[103344]: pgmap v1085: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 742 op/s; 14177/201948 objects degraded (7.020%); 3590/201948 objects misplaced (1.778%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:31.992 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:31 smithi003 ceph-mon[100615]: pgmap v1085: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 742 op/s; 14177/201948 objects degraded (7.020%); 3590/201948 objects misplaced (1.778%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:32.000 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:32.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:32.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:40:32.089 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:40:32.090 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:40:32.102 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:32.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:31 smithi053 ceph-mon[89963]: pgmap v1085: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 742 op/s; 14177/201948 objects degraded (7.020%); 3590/201948 objects misplaced (1.778%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:32.227 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:32.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:32.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:32.330 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:32.384 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:32.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:32.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:32.485 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:32.545 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:32.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:32.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:32.647 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:32.670 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:32.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:32.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:32.772 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:32.797 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:32.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:32.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:32.899 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:32.914 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:40:32.914 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:40:32.915 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.915 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36354.0:168499 9.d 9:bc817a10:::benchmark_data_smithi003_261051_object52614:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-08-17T17:40:32.915 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:40:32.903742+0000", 2024-08-17T17:40:32.915 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0088743390000000002, 2024-08-17T17:40:32.915 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0089034590000000007, 2024-08-17T17:40:32.915 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:40:32.915 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:40:32.915 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:40:32.915 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:40:32.915 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36354", 2024-08-17T17:40:32.915 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1330158990", 2024-08-17T17:40:32.915 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 168499 2024-08-17T17:40:32.916 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.916 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:40:32.916 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.916 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:40:32.916 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.903742+0000", 2024-08-17T17:40:32.916 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:40:32.916 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.916 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.916 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:40:32.916 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.903742+0000", 2024-08-17T17:40:32.916 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:40:32.916 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.916 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.916 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:40:32.917 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.903743+0000", 2024-08-17T17:40:32.917 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.203e-06 2024-08-17T17:40:32.917 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.917 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.917 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:40:32.917 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.903748+0000", 2024-08-17T17:40:32.917 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8980000000000004e-06 2024-08-17T17:40:32.917 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.917 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.917 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:40:32.917 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.903767+0000", 2024-08-17T17:40:32.917 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9607e-05 2024-08-17T17:40:32.917 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.917 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.917 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:40:32.918 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.903775+0000", 2024-08-17T17:40:32.918 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4220000000000003e-06 2024-08-17T17:40:32.918 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.918 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.918 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:40:32.918 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.903859+0000", 2024-08-17T17:40:32.918 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4672000000000005e-05 2024-08-17T17:40:32.918 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.918 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.918 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:40:32.918 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.903979+0000", 2024-08-17T17:40:32.918 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000119336 2024-08-17T17:40:32.918 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.919 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.919 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-08-17T17:40:32.919 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.904102+0000", 2024-08-17T17:40:32.919 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012337099999999999 2024-08-17T17:40:32.919 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.919 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.919 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:40:32.919 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.908012+0000", 2024-08-17T17:40:32.919 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039100250000000001 2024-08-17T17:40:32.919 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.919 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.919 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:40:32.919 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.909825+0000", 2024-08-17T17:40:32.919 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001812595 2024-08-17T17:40:32.920 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:40:32.920 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:40:32.920 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:40:32.920 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.920 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.920 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36354.0:168475 9.e 9:733901f2:::benchmark_data_smithi003_261051_object52590:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-08-17T17:40:32.920 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:40:32.874300+0000", 2024-08-17T17:40:32.920 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.038316171000000003, 2024-08-17T17:40:32.920 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.038426080000000001, 2024-08-17T17:40:32.920 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:40:32.920 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:40:32.920 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:40:32.920 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:40:32.920 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36354", 2024-08-17T17:40:32.921 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1330158990", 2024-08-17T17:40:32.921 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 168475 2024-08-17T17:40:32.921 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.921 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:40:32.921 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.921 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:40:32.921 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.874300+0000", 2024-08-17T17:40:32.921 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:40:32.921 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.921 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.921 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:40:32.921 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.874300+0000", 2024-08-17T17:40:32.921 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:40:32.921 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.921 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.922 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:40:32.922 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.874300+0000", 2024-08-17T17:40:32.922 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9699999999999996e-07 2024-08-17T17:40:32.922 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.922 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.922 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:40:32.922 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.874303+0000", 2024-08-17T17:40:32.922 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8590000000000001e-06 2024-08-17T17:40:32.922 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.922 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.922 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:40:32.922 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.874310+0000", 2024-08-17T17:40:32.922 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8700000000000003e-06 2024-08-17T17:40:32.922 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.923 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.923 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:40:32.923 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.874315+0000", 2024-08-17T17:40:32.923 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2889999999999997e-06 2024-08-17T17:40:32.923 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.923 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.923 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:40:32.923 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.884132+0000", 2024-08-17T17:40:32.923 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0098164510000000003 2024-08-17T17:40:32.923 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.923 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.923 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:40:32.923 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.884234+0000", 2024-08-17T17:40:32.923 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010261599999999999 2024-08-17T17:40:32.923 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.924 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.924 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,7", 2024-08-17T17:40:32.924 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.884374+0000", 2024-08-17T17:40:32.924 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013995199999999999 2024-08-17T17:40:32.924 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.924 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.924 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:40:32.924 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.887806+0000", 2024-08-17T17:40:32.924 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034320269999999998 2024-08-17T17:40:32.924 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.924 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.924 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:40:32.924 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.895386+0000", 2024-08-17T17:40:32.924 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0075802439999999999 2024-08-17T17:40:32.924 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:40:32.924 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:40:32.925 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:40:32.925 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.925 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.925 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36354.0:168502 9.0 9:04fbeca0:::benchmark_data_smithi003_261051_object52617:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-08-17T17:40:32.925 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:40:32.903931+0000", 2024-08-17T17:40:32.925 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0086845610000000004, 2024-08-17T17:40:32.925 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0088658869999999994, 2024-08-17T17:40:32.925 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:40:32.925 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:40:32.925 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:40:32.925 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:40:32.925 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36354", 2024-08-17T17:40:32.925 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1330158990", 2024-08-17T17:40:32.925 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 168502 2024-08-17T17:40:32.925 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.926 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:40:32.926 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.926 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:40:32.926 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.903931+0000", 2024-08-17T17:40:32.926 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:40:32.926 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.926 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.926 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:40:32.926 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.903931+0000", 2024-08-17T17:40:32.926 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:40:32.926 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.926 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.926 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:40:32.927 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.903932+0000", 2024-08-17T17:40:32.927 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9900000000000009e-07 2024-08-17T17:40:32.927 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.927 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.927 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:40:32.927 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.903937+0000", 2024-08-17T17:40:32.927 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2520000000000001e-06 2024-08-17T17:40:32.927 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.927 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.927 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:40:32.927 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.903948+0000", 2024-08-17T17:40:32.927 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1578e-05 2024-08-17T17:40:32.927 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.927 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.927 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:40:32.928 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.903957+0000", 2024-08-17T17:40:32.928 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0990000000000005e-06 2024-08-17T17:40:32.928 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.928 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:32.929 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:40:32.929 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.904016+0000", 2024-08-17T17:40:32.929 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8761000000000002e-05 2024-08-17T17:40:32.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.929 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.929 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:40:32.929 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.904125+0000", 2024-08-17T17:40:32.929 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000108993 2024-08-17T17:40:32.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.929 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.930 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,6", 2024-08-17T17:40:32.930 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.904230+0000", 2024-08-17T17:40:32.930 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010500700000000001 2024-08-17T17:40:32.930 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:40:32.930 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:40:32.930 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:40:32.930 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.930 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.930 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36354.0:168503 9.6 9:6da89fc5:::benchmark_data_smithi003_261051_object52618:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-08-17T17:40:32.930 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:40:32.906969+0000", 2024-08-17T17:40:32.930 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0056471079999999996, 2024-08-17T17:40:32.930 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0058983280000000004, 2024-08-17T17:40:32.930 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:40:32.930 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:40:32.931 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:40:32.931 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:40:32.931 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36354", 2024-08-17T17:40:32.931 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1330158990", 2024-08-17T17:40:32.931 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 168503 2024-08-17T17:40:32.931 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.931 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:40:32.931 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.931 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:40:32.931 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.906969+0000", 2024-08-17T17:40:32.931 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:40:32.931 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.931 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.932 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:40:32.932 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.906969+0000", 2024-08-17T17:40:32.932 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:40:32.932 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.932 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.932 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:40:32.932 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.906971+0000", 2024-08-17T17:40:32.932 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1050000000000002e-06 2024-08-17T17:40:32.932 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.932 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.932 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:40:32.932 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.906979+0000", 2024-08-17T17:40:32.932 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8939999999999993e-06 2024-08-17T17:40:32.932 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.933 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.933 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:40:32.933 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.906994+0000", 2024-08-17T17:40:32.933 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5115e-05 2024-08-17T17:40:32.933 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.933 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.933 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:40:32.933 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.907004+0000", 2024-08-17T17:40:32.933 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0138e-05 2024-08-17T17:40:32.933 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.933 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.933 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:40:32.933 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.907064+0000", 2024-08-17T17:40:32.933 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0093000000000002e-05 2024-08-17T17:40:32.933 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.934 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.934 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:40:32.934 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.907196+0000", 2024-08-17T17:40:32.934 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013205199999999999 2024-08-17T17:40:32.934 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.934 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.934 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:40:32.934 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.907326+0000", 2024-08-17T17:40:32.934 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000130335 2024-08-17T17:40:32.934 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:40:32.934 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:40:32.934 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:40:32.934 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.934 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.935 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.36354.0:168475 9.e e336/327)", 2024-08-17T17:40:32.935 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:40:32.907045+0000", 2024-08-17T17:40:32.935 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0055708880000000004, 2024-08-17T17:40:32.935 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0058826160000000002, 2024-08-17T17:40:32.935 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:40:32.935 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:40:32.935 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-08-17T17:40:32.935 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:40:32.935 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.935 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:40:32.935 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.907045+0000", 2024-08-17T17:40:32.935 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:40:32.935 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.935 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.935 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:40:32.936 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.907045+0000", 2024-08-17T17:40:32.936 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:40:32.936 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.936 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.936 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:40:32.936 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.907046+0000", 2024-08-17T17:40:32.936 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4399999999999999e-07 2024-08-17T17:40:32.936 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.936 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.936 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:40:32.936 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.907052+0000", 2024-08-17T17:40:32.936 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0109999999999997e-06 2024-08-17T17:40:32.936 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.936 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.936 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:40:32.936 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.907063+0000", 2024-08-17T17:40:32.937 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1105e-05 2024-08-17T17:40:32.937 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.937 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.937 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:40:32.937 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.907070+0000", 2024-08-17T17:40:32.937 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0140000000000001e-06 2024-08-17T17:40:32.937 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:40:32.937 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:40:32.937 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:40:32.937 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.937 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.937 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36354.0:168481 9.4 9:225d359b:::benchmark_data_smithi003_261051_object52596:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-08-17T17:40:32.937 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:40:32.874902+0000", 2024-08-17T17:40:32.937 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.037714171999999997, 2024-08-17T17:40:32.937 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.038073058999999999, 2024-08-17T17:40:32.938 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:40:32.938 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:40:32.938 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:40:32.938 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:40:32.938 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36354", 2024-08-17T17:40:32.938 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1330158990", 2024-08-17T17:40:32.938 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 168481 2024-08-17T17:40:32.938 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.938 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:40:32.938 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.938 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:40:32.938 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.874902+0000", 2024-08-17T17:40:32.938 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:40:32.938 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.939 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.939 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:40:32.939 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.874902+0000", 2024-08-17T17:40:32.939 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:40:32.939 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.939 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.939 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:40:32.939 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.874903+0000", 2024-08-17T17:40:32.939 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1999999999999998e-07 2024-08-17T17:40:32.939 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.939 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.939 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:40:32.939 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.874906+0000", 2024-08-17T17:40:32.939 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9539999999999998e-06 2024-08-17T17:40:32.940 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.940 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.940 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:40:32.940 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.874918+0000", 2024-08-17T17:40:32.940 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1776999999999999e-05 2024-08-17T17:40:32.940 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.940 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.940 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:40:32.940 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.874925+0000", 2024-08-17T17:40:32.940 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9290000000000001e-06 2024-08-17T17:40:32.940 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.940 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.940 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:40:32.940 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.886093+0000", 2024-08-17T17:40:32.941 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011167493000000001 2024-08-17T17:40:32.941 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.941 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.941 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:40:32.941 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.886258+0000", 2024-08-17T17:40:32.941 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016488900000000001 2024-08-17T17:40:32.941 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.941 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.941 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,3", 2024-08-17T17:40:32.941 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.886423+0000", 2024-08-17T17:40:32.941 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016564999999999999 2024-08-17T17:40:32.941 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.941 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.942 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:40:32.942 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.902839+0000", 2024-08-17T17:40:32.942 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.016415984000000002 2024-08-17T17:40:32.942 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.942 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.942 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:40:32.942 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.905121+0000", 2024-08-17T17:40:32.942 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0022815639999999998 2024-08-17T17:40:32.942 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:40:32.942 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:40:32.942 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:40:32.942 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.942 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.943 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.36354.0:168502 9.0 e336/311)", 2024-08-17T17:40:32.943 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:40:32.908269+0000", 2024-08-17T17:40:32.943 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0043466779999999997, 2024-08-17T17:40:32.943 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0047732, 2024-08-17T17:40:32.943 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:40:32.943 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:40:32.943 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-08-17T17:40:32.943 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:40:32.943 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.943 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:40:32.943 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.908269+0000", 2024-08-17T17:40:32.943 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:40:32.943 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.943 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.944 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:40:32.944 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.908269+0000", 2024-08-17T17:40:32.944 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:40:32.944 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.944 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.944 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:40:32.944 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.908270+0000", 2024-08-17T17:40:32.944 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0200000000000001e-07 2024-08-17T17:40:32.944 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.944 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.944 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:40:32.944 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.908274+0000", 2024-08-17T17:40:32.944 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0890000000000002e-06 2024-08-17T17:40:32.944 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.944 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.945 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:40:32.945 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.908289+0000", 2024-08-17T17:40:32.945 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5336e-05 2024-08-17T17:40:32.945 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.945 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.945 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:40:32.945 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.908296+0000", 2024-08-17T17:40:32.945 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0230000000000004e-06 2024-08-17T17:40:32.945 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:40:32.945 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:40:32.945 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:40:32.945 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.945 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.945 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36354.0:168486 9.4 9:2fd14c56:::benchmark_data_smithi003_261051_object52601:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-08-17T17:40:32.946 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:40:32.879721+0000", 2024-08-17T17:40:32.946 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.032895143000000002, 2024-08-17T17:40:32.946 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.033362589999999998, 2024-08-17T17:40:32.946 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:40:32.946 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:40:32.946 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:40:32.946 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:40:32.946 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36354", 2024-08-17T17:40:32.946 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1330158990", 2024-08-17T17:40:32.946 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 168486 2024-08-17T17:40:32.946 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.946 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:40:32.946 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.946 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:40:32.947 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.879721+0000", 2024-08-17T17:40:32.947 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:40:32.947 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.947 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.947 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:40:32.947 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.879721+0000", 2024-08-17T17:40:32.947 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:40:32.947 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.947 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.947 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:40:32.947 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.879721+0000", 2024-08-17T17:40:32.947 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4099999999999998e-07 2024-08-17T17:40:32.947 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.948 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.948 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:40:32.948 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.879725+0000", 2024-08-17T17:40:32.948 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0429999999999999e-06 2024-08-17T17:40:32.948 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.948 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.948 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:40:32.948 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.879735+0000", 2024-08-17T17:40:32.948 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0017000000000001e-05 2024-08-17T17:40:32.948 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.948 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.948 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:40:32.948 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.879741+0000", 2024-08-17T17:40:32.949 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.144e-06 2024-08-17T17:40:32.949 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.949 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.949 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:40:32.949 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.895421+0000", 2024-08-17T17:40:32.949 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.015680316 2024-08-17T17:40:32.949 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.949 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.949 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:40:32.949 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.895590+0000", 2024-08-17T17:40:32.949 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000169445 2024-08-17T17:40:32.949 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.949 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.949 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,3", 2024-08-17T17:40:32.950 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.895752+0000", 2024-08-17T17:40:32.950 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000161423 2024-08-17T17:40:32.950 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.950 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.950 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:40:32.950 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.902851+0000", 2024-08-17T17:40:32.950 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0070986570000000004 2024-08-17T17:40:32.950 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:40:32.950 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:40:32.950 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:40:32.950 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.950 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.950 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.36354.0:168499 9.d e336/311)", 2024-08-17T17:40:32.950 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:40:32.911860+0000", 2024-08-17T17:40:32.951 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.00075568200000000003, 2024-08-17T17:40:32.951 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001282577, 2024-08-17T17:40:32.951 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:40:32.951 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:40:32.951 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-08-17T17:40:32.951 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:40:32.951 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.951 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:40:32.951 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.911860+0000", 2024-08-17T17:40:32.951 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:40:32.951 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.951 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.951 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:40:32.951 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.911860+0000", 2024-08-17T17:40:32.951 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:40:32.952 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.952 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.952 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:40:32.952 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.911861+0000", 2024-08-17T17:40:32.952 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8400000000000004e-07 2024-08-17T17:40:32.952 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.952 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.952 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:40:32.952 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.911866+0000", 2024-08-17T17:40:32.952 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7709999999999997e-06 2024-08-17T17:40:32.952 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.952 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.952 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:40:32.952 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.911879+0000", 2024-08-17T17:40:32.953 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.377e-05 2024-08-17T17:40:32.953 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.953 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.953 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:40:32.953 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.911886+0000", 2024-08-17T17:40:32.953 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0859999999999996e-06 2024-08-17T17:40:32.953 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:40:32.953 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:40:32.953 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:40:32.953 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.953 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.954 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.36354.0:168503 9.6 e336/311)", 2024-08-17T17:40:32.954 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:40:32.911903+0000", 2024-08-17T17:40:32.954 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.00071264899999999999, 2024-08-17T17:40:32.954 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001275062, 2024-08-17T17:40:32.954 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:40:32.954 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:40:32.954 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-08-17T17:40:32.954 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:40:32.954 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.954 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:40:32.954 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.911903+0000", 2024-08-17T17:40:32.954 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:40:32.955 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.955 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.955 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:40:32.955 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.911903+0000", 2024-08-17T17:40:32.955 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:40:32.955 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.955 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.955 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:40:32.955 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.911903+0000", 2024-08-17T17:40:32.955 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2700000000000001e-07 2024-08-17T17:40:32.955 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.955 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.955 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:40:32.955 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.911906+0000", 2024-08-17T17:40:32.956 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0980000000000001e-06 2024-08-17T17:40:32.956 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.956 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.956 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:40:32.956 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.911911+0000", 2024-08-17T17:40:32.956 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6849999999999998e-06 2024-08-17T17:40:32.956 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.956 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.956 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:40:32.956 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.911915+0000", 2024-08-17T17:40:32.956 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2359999999999998e-06 2024-08-17T17:40:32.956 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:40:32.956 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:40:32.956 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:40:32.957 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.957 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.957 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36354.0:168506 9.2 e336/311 9:42bef706:::benchmark_data_smithi003_261051_object52621:head v 336'10591, mlcod=336'10589)", 2024-08-17T17:40:32.957 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:40:32.911957+0000", 2024-08-17T17:40:32.957 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.00065918400000000005, 2024-08-17T17:40:32.957 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0012602570000000001, 2024-08-17T17:40:32.957 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:40:32.957 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:40:32.957 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:40:32.957 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:40:32.957 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.957 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:40:32.957 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.911957+0000", 2024-08-17T17:40:32.958 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:40:32.958 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.958 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.958 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:40:32.958 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.911957+0000", 2024-08-17T17:40:32.958 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:40:32.958 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.958 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.958 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:40:32.958 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.911957+0000", 2024-08-17T17:40:32.958 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5499999999999999e-07 2024-08-17T17:40:32.958 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.958 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.959 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:40:32.959 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.911962+0000", 2024-08-17T17:40:32.959 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2059999999999998e-06 2024-08-17T17:40:32.959 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.959 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.959 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:40:32.959 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.911970+0000", 2024-08-17T17:40:32.959 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7009999999999996e-06 2024-08-17T17:40:32.959 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.959 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.959 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:40:32.959 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.911975+0000", 2024-08-17T17:40:32.959 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1619999999999999e-06 2024-08-17T17:40:32.959 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.959 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.960 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:40:32.960 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.912043+0000", 2024-08-17T17:40:32.960 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8367000000000005e-05 2024-08-17T17:40:32.960 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:40:32.960 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:40:32.960 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:40:32.960 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:32.912062+0000", 2024-08-17T17:40:32.960 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8272000000000001e-05 2024-08-17T17:40:32.960 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:40:32.960 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:40:32.960 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:40:32.960 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:40:32.960 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-08-17T17:40:32.960 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 11 2024-08-17T17:40:32.960 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:40:32.961 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:32.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:33.062 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:33.087 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:33.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:33.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:33.188 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:33.228 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:33.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:33.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:33.330 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:33.372 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:33.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:33.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:33.474 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:33.516 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:33.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:33.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:33.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:33.656 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:33.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:33.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:33.758 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:40:33.783 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:33.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:33.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:40:33.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:33.912 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:33.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:33.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:34.014 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:34.041 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:34.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:34.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:34.142 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:34.181 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:34.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:34.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:34.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:33 smithi053 ceph-mon[89963]: pgmap v1086: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 759 op/s; 14304/196839 objects degraded (7.267%); 3349/196839 objects misplaced (1.701%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:34.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:33 smithi003 ceph-mon[103344]: pgmap v1086: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 759 op/s; 14304/196839 objects degraded (7.267%); 3349/196839 objects misplaced (1.701%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:34.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:33 smithi003 ceph-mon[100615]: pgmap v1086: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 759 op/s; 14304/196839 objects degraded (7.267%); 3349/196839 objects misplaced (1.701%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:34.282 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:40:34.307 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:34.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:34.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:40:34.409 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:34.449 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:34.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:34.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:34.551 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:40:34.574 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:34.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:34.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:40:34.676 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:34.700 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:34.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:34.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:34.802 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:34.814 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:40:34.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:34.866 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:34.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:34.967 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:34.995 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:34.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:34.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:35.098 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:35.131 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:35.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:35.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:35.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:34 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 14304/196839 objects degraded (7.267%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:35.233 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:35.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:35.271 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:35.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:35.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:34 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 14304/196839 objects degraded (7.267%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:35.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:34 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 14304/196839 objects degraded (7.267%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:35.372 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:35.395 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:35.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:35.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:35.496 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:35.521 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:35.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:35.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:35.623 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:35.645 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:35.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:35.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:35.747 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:35.773 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:35.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:35.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:35.875 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:35.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:35.917 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:35.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:36.018 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:36.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:35 smithi003 ceph-mon[103344]: pgmap v1087: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.0 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 759 op/s; 14304/196839 objects degraded (7.267%); 3349/196839 objects misplaced (1.701%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:36.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:35 smithi003 ceph-mon[100615]: pgmap v1087: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.0 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 759 op/s; 14304/196839 objects degraded (7.267%); 3349/196839 objects misplaced (1.701%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:36.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:40:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:40:35] "GET /metrics HTTP/1.1" 200 36523 "" "Prometheus/2.51.0" 2024-08-17T17:40:36.041 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:36.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:36.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:36.143 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:36.167 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:36.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:36.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:36.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:35 smithi053 ceph-mon[89963]: pgmap v1087: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 4.0 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 759 op/s; 14304/196839 objects degraded (7.267%); 3349/196839 objects misplaced (1.701%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:36.270 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:36.304 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:36.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:36.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:36.405 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:36.452 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:36.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:36.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:36.554 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:36.580 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:36.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:36.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:36.682 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:36.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:36.743 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:36.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:36.844 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:36.869 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:36.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:36.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:36.971 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:40:36.996 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:36.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:36.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:40:37.098 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:37.144 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:37.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:37.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:37.246 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:37.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:37.288 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:37.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:37.348 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-17T17:40:37.389 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:40:37.411 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:37.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:37.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:40:37.513 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:37.537 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:37.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:37.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:37.639 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:40:37.667 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:37.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:37.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:40:37.769 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:37.825 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:37.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:37.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:37.927 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:37.965 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:37.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:37.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:38.067 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:38.117 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:38.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:38.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:38.151 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:37 smithi003 ceph-mon[103344]: pgmap v1088: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 771 op/s; 14431/185031 objects degraded (7.799%); 3110/185031 objects misplaced (1.681%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:38.151 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:37 smithi003 ceph-mon[100615]: pgmap v1088: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 771 op/s; 14431/185031 objects degraded (7.799%); 3110/185031 objects misplaced (1.681%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:38.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:37 smithi053 ceph-mon[89963]: pgmap v1088: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 771 op/s; 14431/185031 objects degraded (7.799%); 3110/185031 objects misplaced (1.681%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:38.219 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:38.245 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:38.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:38.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:38.347 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:38.374 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:38.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:38.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:38.476 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:38.517 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:38.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:38.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:38.618 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:38.652 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:38.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:38.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:38.754 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:38.791 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:38.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:38.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:38.893 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:38.926 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:38.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:38.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:39.028 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:39.059 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:39.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:39.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:39.090 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:40:39.161 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:39.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:39.195 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:39.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:39.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:40:39.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:40:39.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:40:39.296 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:39.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:39.331 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:39.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:39.432 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:39.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:39.463 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:39.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:39.564 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:39.603 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:39.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:39.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:39.705 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:39.758 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:39.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:39.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:39.860 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:39.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:39.905 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:39.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:40.005 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:40.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:39 smithi003 ceph-mon[100615]: pgmap v1089: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 675 op/s; 14431/185031 objects degraded (7.799%); 3110/185031 objects misplaced (1.681%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:40:40.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:39 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 14431/185031 objects degraded (7.799%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:40.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:39 smithi003 ceph-mon[103344]: pgmap v1089: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 675 op/s; 14431/185031 objects degraded (7.799%); 3110/185031 objects misplaced (1.681%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:40:40.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:39 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 14431/185031 objects degraded (7.799%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:40.029 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:40.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:40.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:40.131 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:40.159 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:40.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:40.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:40.214 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:39 smithi053 ceph-mon[89963]: pgmap v1089: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 675 op/s; 14431/185031 objects degraded (7.799%); 3110/185031 objects misplaced (1.681%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:40:40.215 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:39 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 14431/185031 objects degraded (7.799%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:40.261 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:40:40.285 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:40.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:40.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:40:40.387 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:40.429 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:40.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:40.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:40.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:40:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:40:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:40:40.530 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:40.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:40.570 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:40.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:40.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:40.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:40.713 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:40.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:40.814 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:40.863 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:40.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:40.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:40.964 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:40.989 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:40.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:40.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:41.090 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:41.126 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:41.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:41.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:41.228 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:41.265 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:41.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:41.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:41.367 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:41.406 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:41.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:41.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:41.508 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:41.531 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:41.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:41.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:41.633 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:41.659 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:41.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:41.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:41.761 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:41.827 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:41.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:41.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:41.929 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:41.978 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:41.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:41.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:41.992 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:40:41.992 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:40:42.030 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:40:42.030 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:40:42.069 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:40:42.070 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:40:42.080 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:42.122 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:40:42.123 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:40:42.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:42.131 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:42.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:42.153 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:41 smithi003 ceph-mon[103344]: pgmap v1090: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.6 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 821 op/s; 14431/178563 objects degraded (8.082%); 3110/178563 objects misplaced (1.742%); 1.2 MiB/s, 18 objects/s recovering 2024-08-17T17:40:42.153 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:40:42.153 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:40:42.154 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:41 smithi003 ceph-mon[100615]: pgmap v1090: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.6 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 821 op/s; 14431/178563 objects degraded (8.082%); 3110/178563 objects misplaced (1.742%); 1.2 MiB/s, 18 objects/s recovering 2024-08-17T17:40:42.186 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:40:42.187 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:40:42.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:41 smithi053 ceph-mon[89963]: pgmap v1090: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.6 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 821 op/s; 14431/178563 objects degraded (8.082%); 3110/178563 objects misplaced (1.742%); 1.2 MiB/s, 18 objects/s recovering 2024-08-17T17:40:42.221 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:40:42.221 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:40:42.232 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:42.254 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:42.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:42.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:42.257 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:40:42.257 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:40:42.356 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:42.398 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:42.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:42.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:42.499 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:42.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:42.543 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:42.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:42.644 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:42.682 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:40:42.686 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:42.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:42.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:42.788 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:42.813 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:42.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:42.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:42.906 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-08-17T17:40:42.915 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:40:42.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:42 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2738935185' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:40:42.933 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:42 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3415258622' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:40:42.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:42 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2738935185' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:40:42.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:42 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3415258622' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:40:42.938 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:42.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:42.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:40:43.040 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:43.076 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:43.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:43.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:43.178 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:43.205 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:43.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:43.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:43.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:42 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2738935185' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:40:43.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:42 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3415258622' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:40:43.308 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:43.335 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:43.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:43.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:43.437 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:43.484 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:43.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:43.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:43.586 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:43.629 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:43.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:43.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:43.730 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:40:43.756 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:43.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:43.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:40:43.858 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:43.901 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:43.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:43.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:44.003 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:44.046 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:44.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:44.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:44.147 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:44.184 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:44.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:44.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:44.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:43 smithi053 ceph-mon[89963]: pgmap v1091: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 784 op/s; 14571/173646 objects degraded (8.391%); 2857/173646 objects misplaced (1.645%); 1.8 MiB/s, 28 objects/s recovering 2024-08-17T17:40:44.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:43 smithi053 ceph-mon[89963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:40:44.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:43 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3415258622' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:40:44.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:43 smithi053 ceph-mon[89963]: osdmap e337: 8 total, 8 up, 7 in 2024-08-17T17:40:44.241 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:43 smithi003 ceph-mon[103344]: pgmap v1091: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 784 op/s; 14571/173646 objects degraded (8.391%); 2857/173646 objects misplaced (1.645%); 1.8 MiB/s, 28 objects/s recovering 2024-08-17T17:40:44.241 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:43 smithi003 ceph-mon[103344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:40:44.241 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:43 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3415258622' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:40:44.241 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:43 smithi003 ceph-mon[103344]: osdmap e337: 8 total, 8 up, 7 in 2024-08-17T17:40:44.242 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:43 smithi003 ceph-mon[100615]: pgmap v1091: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 784 op/s; 14571/173646 objects degraded (8.391%); 2857/173646 objects misplaced (1.645%); 1.8 MiB/s, 28 objects/s recovering 2024-08-17T17:40:44.242 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:43 smithi003 ceph-mon[100615]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:40:44.242 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:43 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3415258622' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:40:44.242 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:43 smithi003 ceph-mon[100615]: osdmap e337: 8 total, 8 up, 7 in 2024-08-17T17:40:44.286 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:44.320 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:44.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:44.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:44.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:44.468 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:44.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:44.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:44.570 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:44.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:44.614 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:44.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:44.715 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:44.739 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:44.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:44.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:44.841 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:44.864 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:44.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:44.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:44.966 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:44.990 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:44.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:44.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:45.091 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:45.118 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:45.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:45.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:45.176 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:40:45.215 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:44 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 14571/173646 objects degraded (8.391%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:45.216 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:44 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 14571/173646 objects degraded (8.391%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:45.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:44 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 14571/173646 objects degraded (8.391%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:45.219 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:45.242 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:45.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:45.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:45.283 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-08-17T17:40:45.283 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:40:45.284 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-08-17T17:40:45.284 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:40:45.344 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:45.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:45.401 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:45.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:45.502 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:45.560 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:45.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:45.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:45.662 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:45.686 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:45.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:45.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:45.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:40:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:40:45] "GET /metrics HTTP/1.1" 200 36526 "" "Prometheus/2.51.0" 2024-08-17T17:40:45.787 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:45.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:45.828 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:45.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:45.929 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:45.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:45.966 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:45.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:46.031 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:45 smithi003 ceph-mon[103344]: pgmap v1093: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 794 op/s; 14571/172941 objects degraded (8.425%); 2857/172941 objects misplaced (1.652%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:46.032 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:45 smithi003 ceph-mon[100615]: pgmap v1093: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 794 op/s; 14571/172941 objects degraded (8.425%); 2857/172941 objects misplaced (1.652%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:46.034 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:40:46.034 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:40:46.047 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:40:46.067 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:46.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:46.125 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:46.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:46.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:45 smithi053 ceph-mon[89963]: pgmap v1093: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 794 op/s; 14571/172941 objects degraded (8.425%); 2857/172941 objects misplaced (1.652%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:46.226 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:46.272 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:46.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:46.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:46.283 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:40:46.374 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:46.416 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:46.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:46.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:46.518 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:46.540 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:46.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:46.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:46.643 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:46.666 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:46.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:46.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:46.767 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:46.814 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:46.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:46.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:46.916 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:46.958 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:46.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:46.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:47.060 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:47.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:47.109 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:47.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:47.210 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:40:47.239 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:47.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:47.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:40:47.341 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:47.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:47.368 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:47.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:47.469 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:47.512 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:47.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:47.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:47.614 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:47.664 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:47.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:47.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:47.766 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:47.791 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:47.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:47.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:47.892 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:47.930 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:47.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:47.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:48.032 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:48.070 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:48.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:48.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:48.172 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:48.195 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:48.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:48.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:48.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:47 smithi053 ceph-mon[89963]: pgmap v1094: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 758 op/s; 14702/162207 objects degraded (9.064%); 2609/162207 objects misplaced (1.608%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:48.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:47 smithi003 ceph-mon[100615]: pgmap v1094: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 758 op/s; 14702/162207 objects degraded (9.064%); 2609/162207 objects misplaced (1.608%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:48.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:47 smithi003 ceph-mon[103344]: pgmap v1094: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 758 op/s; 14702/162207 objects degraded (9.064%); 2609/162207 objects misplaced (1.608%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:48.296 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:48.321 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:48.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:48.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:48.342 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:40:48.389 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-17T17:40:48.423 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:48.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:48.495 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:48.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:48.596 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:48.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:48.644 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:48.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:48.745 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:48.768 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:48.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:48.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:48.870 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:48.895 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:48.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:48.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:48.997 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:49.024 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:49.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:49.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:49.126 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:49.151 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:49.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:49.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:49.253 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:49.278 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:49.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:49.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:49.380 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:49.407 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:49.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:49.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:49.509 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:49.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:49.535 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:49.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:49.636 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:49.658 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:49.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:49.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:49.760 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:49.799 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:49.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:49.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:49.901 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:49.928 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:49.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:49.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:50.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:49 smithi003 ceph-mon[103344]: pgmap v1095: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 758 op/s; 14702/162207 objects degraded (9.064%); 2609/162207 objects misplaced (1.608%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:50.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:49 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 14702/162207 objects degraded (9.064%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:50.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:49 smithi003 ceph-mon[100615]: pgmap v1095: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 758 op/s; 14702/162207 objects degraded (9.064%); 2609/162207 objects misplaced (1.608%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:50.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:49 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 14702/162207 objects degraded (9.064%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:50.029 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:50.055 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:50.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:50.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:50.157 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:50.183 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:50.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:50.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:50.214 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:49 smithi053 ceph-mon[89963]: pgmap v1095: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 758 op/s; 14702/162207 objects degraded (9.064%); 2609/162207 objects misplaced (1.608%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:50.214 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:49 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 14702/162207 objects degraded (9.064%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:50.285 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:50.311 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:50.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:50.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:50.413 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:50.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:50.454 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:50.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:50.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:40:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:40:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:40:50.555 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:50.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:50.583 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:50.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:50.685 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:50.708 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:50.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:50.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:50.810 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:40:50.833 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:50.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:50.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:40:50.935 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:50.965 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:50.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:50.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:51.067 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:51.091 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:51.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:51.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:51.193 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:51.242 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:51.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:51.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:51.344 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:51.367 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:51.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:51.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:51.468 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:51.505 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:51.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:51.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:51.606 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:51.630 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:51.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:51.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:51.732 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:51.746 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:40:51.746 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:40:51.746 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:40:51.746 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:40:51.746 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:40:51.746 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:40:51.747 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:40:51.747 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:40:51.747 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:40:51.747 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:40:51.747 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:40:51.747 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:40:51.747 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:40:51.747 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:40:51.747 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:40:51.747 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:40:51.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:51.792 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:51.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:51.893 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:51.917 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:51.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:51.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:52.020 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:52.059 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:52.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:52.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:52.161 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:52.187 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:52.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:52.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:52.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:51 smithi053 ceph-mon[89963]: pgmap v1096: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 743 op/s; 14702/156234 objects degraded (9.410%); 2609/156234 objects misplaced (1.670%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:52.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:51 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1225700173' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:40:52.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:51 smithi003 ceph-mon[103344]: pgmap v1096: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 743 op/s; 14702/156234 objects degraded (9.410%); 2609/156234 objects misplaced (1.670%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:52.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:51 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1225700173' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:40:52.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:51 smithi003 ceph-mon[100615]: pgmap v1096: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 743 op/s; 14702/156234 objects degraded (9.410%); 2609/156234 objects misplaced (1.670%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:40:52.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:51 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1225700173' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:40:52.289 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:52.314 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:52.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:52.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:52.416 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:52.455 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:52.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:52.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:52.557 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:52.606 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:52.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:52.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:52.708 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:52.735 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:52.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:52.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:52.837 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:52.875 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:52.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:52.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:52.977 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:53.016 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:53.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:53.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:53.022 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:40:53.022 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:40:53.118 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:40:53.142 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:53.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:53.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:40:53.244 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:53.285 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:53.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:53.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:53.387 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:53.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:53.439 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:53.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:53.540 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:53.582 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:53.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:53.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:53.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:53.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:53.724 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:53.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:53.825 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:53.871 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:53.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:53.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:53.973 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:54.004 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:54.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:54.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:54.106 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:54.147 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:54.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:54.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:54.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:53 smithi053 ceph-mon[89963]: pgmap v1097: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 737 op/s; 14820/151503 objects degraded (9.782%); 2372/151503 objects misplaced (1.566%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:40:54.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:40:54.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:40:54.249 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:54.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:53 smithi003 ceph-mon[103344]: pgmap v1097: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 737 op/s; 14820/151503 objects degraded (9.782%); 2372/151503 objects misplaced (1.566%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:40:54.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:40:54.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:40:54.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:53 smithi003 ceph-mon[100615]: pgmap v1097: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 737 op/s; 14820/151503 objects degraded (9.782%); 2372/151503 objects misplaced (1.566%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:40:54.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:40:54.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:40:54.292 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:54.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:54.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:54.394 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:54.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:54.436 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:54.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:54.537 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:54.564 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:54.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:54.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:54.666 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:54.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:54.714 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:54.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:54.815 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:54.838 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:54.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:54.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:54.940 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:54.966 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:54.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:54.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:55.064 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:40:55.068 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:55.157 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:55.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:55.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:55.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:54 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 14820/151503 objects degraded (9.782%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:55.258 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:55.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:54 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 14820/151503 objects degraded (9.782%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:55.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:54 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 14820/151503 objects degraded (9.782%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:40:55.306 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:55.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:55.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:55.408 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:55.432 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:55.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:55.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:55.534 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:55.581 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:55.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:55.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:55.683 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:55.708 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:55.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:55.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:55.809 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:55.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:55.856 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:55.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:55.957 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:55.968 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:55 smithi003 ceph-mon[100615]: pgmap v1098: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 668 op/s; 14820/150810 objects degraded (9.827%); 2372/150810 objects misplaced (1.573%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:40:55.969 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:40:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:40:55] "GET /metrics HTTP/1.1" 200 36525 "" "Prometheus/2.51.0" 2024-08-17T17:40:55.991 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:55.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:55.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:56.093 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:56.115 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:56.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:56.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:56.217 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:56.237 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:55 smithi003 ceph-mon[103344]: pgmap v1098: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 668 op/s; 14820/150810 objects degraded (9.827%); 2372/150810 objects misplaced (1.573%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:40:56.241 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:56.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:56.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:56.343 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:56.382 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:56.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:56.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:56.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:55 smithi053 ceph-mon[89963]: pgmap v1098: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 3.1 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 668 op/s; 14820/150810 objects degraded (9.827%); 2372/150810 objects misplaced (1.573%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:40:56.484 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:40:56.507 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:56.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:56.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:40:56.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:56.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:56.668 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:56.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:56.769 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:56.793 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:56.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:56.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:56.895 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:56.919 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:56.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:56.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:57.021 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:57.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:57.067 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:57.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:57.168 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:57.205 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:57.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:57.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:57.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:57 smithi003 ceph-mon[100615]: pgmap v1099: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 913 op/s; 14932/140025 objects degraded (10.664%); 2125/140025 objects misplaced (1.518%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T17:40:57.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:57 smithi003 ceph-mon[100615]: from='client.36513 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:40:57.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:57 smithi003 ceph-mon[103344]: pgmap v1099: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 913 op/s; 14932/140025 objects degraded (10.664%); 2125/140025 objects misplaced (1.518%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T17:40:57.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:57 smithi003 ceph-mon[103344]: from='client.36513 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:40:57.307 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:40:57.332 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:57.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:57.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:40:57.373 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:40:57.378 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:40:57.434 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:57.458 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:57.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:57.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:57.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:57 smithi053 ceph-mon[89963]: pgmap v1099: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 913 op/s; 14932/140025 objects degraded (10.664%); 2125/140025 objects misplaced (1.518%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T17:40:57.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:57 smithi053 ceph-mon[89963]: from='client.36513 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:40:57.561 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:57.600 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:57.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:57.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:57.702 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:57.741 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:57.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:57.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:57.843 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:40:57.868 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:57.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:57.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:40:57.970 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:58.015 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:58.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:58.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:58.117 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:40:58.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:58.155 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:58.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:40:58.256 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:40:58.278 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:58.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:58.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:40:58.380 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:58.417 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:58.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:58.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:58.519 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:58.562 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:58.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:58.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:58.664 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:58.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:58.705 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:58.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:58.806 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:58.842 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:58.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:58.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:58.944 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:58.987 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:58.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:58.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:59.089 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:59.117 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:59.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:59.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:59.218 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:40:59.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:59.255 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:59.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:40:59.356 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:59.391 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:59.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:59.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:40:59.493 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:59.519 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:59.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:59.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:59.620 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:40:59.659 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:59.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:59.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:40:59.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:40:59 smithi053 ceph-mon[89963]: pgmap v1100: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 615 op/s; 14932/140025 objects degraded (10.664%); 2125/140025 objects misplaced (1.518%); 1.3 MiB/s, 21 objects/s recovering 2024-08-17T17:40:59.761 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:40:59.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:40:59 smithi003 ceph-mon[100615]: pgmap v1100: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 615 op/s; 14932/140025 objects degraded (10.664%); 2125/140025 objects misplaced (1.518%); 1.3 MiB/s, 21 objects/s recovering 2024-08-17T17:40:59.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:40:59 smithi003 ceph-mon[103344]: pgmap v1100: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 615 op/s; 14932/140025 objects degraded (10.664%); 2125/140025 objects misplaced (1.518%); 1.3 MiB/s, 21 objects/s recovering 2024-08-17T17:40:59.785 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:40:59.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:59.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:40:59.887 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:40:59.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:40:59.926 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:40:59.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:00.027 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:00.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:00.064 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:00.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:00.165 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:00.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:00.210 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:00.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:00.310 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:00.337 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:00.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:00.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:00.439 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:00.464 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:00.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:00.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:00.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:00 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 14932/140025 objects degraded (10.664%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:41:00.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:41:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:41:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:41:00.566 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:00.590 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:00.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:00.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:00.692 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:00.723 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:00.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:00.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:00.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:00 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 14932/140025 objects degraded (10.664%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:41:00.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:00 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 14932/140025 objects degraded (10.664%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:41:00.826 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:00.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:00.867 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:00.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:00.968 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:01.004 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:01.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:01.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:01.106 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:01.137 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:01.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:01.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:01.239 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:01.280 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:01.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:01.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:01.384 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:01.409 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:01.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:01.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:01.511 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:01.554 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:01.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:01.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:01.656 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:01.682 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:01.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:01.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:01.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:01 smithi053 ceph-mon[89963]: pgmap v1101: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 780 op/s; 14932/134100 objects degraded (11.135%); 2125/134100 objects misplaced (1.585%); 1.3 MiB/s, 21 objects/s recovering 2024-08-17T17:41:01.757 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:01 smithi003 ceph-mon[103344]: pgmap v1101: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 780 op/s; 14932/134100 objects degraded (11.135%); 2125/134100 objects misplaced (1.585%); 1.3 MiB/s, 21 objects/s recovering 2024-08-17T17:41:01.757 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:01 smithi003 ceph-mon[100615]: pgmap v1101: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 780 op/s; 14932/134100 objects degraded (11.135%); 2125/134100 objects misplaced (1.585%); 1.3 MiB/s, 21 objects/s recovering 2024-08-17T17:41:01.785 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:01.819 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:01.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:01.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:01.921 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:01.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:01.960 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:01.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:02.061 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:02.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:02.103 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:02.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:02.204 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:02.242 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:02.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:02.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:02.344 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:02.366 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:02.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:02.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:02.467 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:02.489 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:02.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:02.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:02.591 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:02.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:02.619 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:02.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:02.720 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:02.768 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:02.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:02.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:02.820 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1099,"stamp":"2024-08-17T17:40:56.301483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3061782501,"num_objects":46675,"num_object_clones":0,"num_object_copies":140025,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14932,"num_objects_misplaced":2125,"num_objects_unfound":0,"num_objects_dirty":46675,"num_whiteouts":0,"num_read":5979,"num_read_kb":12992,"num_write":186026,"num_write_kb":7429072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2608,"num_bytes_recovered":169880966,"num_keys_recovered":30,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97940,"ondisk_log_size":97940,"up":327,"acting":324,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":327,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":13925480,"kb_used_data":8692976,"kb_used_omap":36,"kb_used_meta":5232347,"kb_avail":642290584,"statfs":{"total":671965249536,"available":657705558016,"internally_reserved":0,"allocated":8901607424,"data_stored":8888002420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37025,"internal_metadata":5357924191},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1,5,5,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-719060992,"num_objects":-10972,"num_object_clones":0,"num_object_copies":-32916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":361,"num_objects_misplaced":-732,"num_objects_unfound":0,"num_objects_dirty":-10972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10972,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":371,"num_bytes_recovered":24313856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011753"},"pg_stats":[{"pgid":"1.11","version":"90'149","reported_seq":721,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.924848+0000","last_change":"2024-08-17T17:38:30.378664+0000","last_active":"2024-08-17T17:40:42.924848+0000","last_peered":"2024-08-17T17:40:42.924848+0000","last_clean":"2024-08-17T17:40:42.924848+0000","last_became_active":"2024-08-17T17:38:30.342287+0000","last_became_peered":"2024-08-17T17:38:30.342287+0000","last_unstale":"2024-08-17T17:40:42.924848+0000","last_undegraded":"2024-08-17T17:40:42.924848+0000","last_fullsized":"2024-08-17T17:40:42.924848+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:30.378592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:30.378592+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:33:06.891167+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":545,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.925137+0000","last_change":"2024-08-17T17:33:20.797029+0000","last_active":"2024-08-17T17:40:42.925137+0000","last_peered":"2024-08-17T17:40:42.925137+0000","last_clean":"2024-08-17T17:40:42.925137+0000","last_became_active":"2024-08-17T17:33:20.796679+0000","last_became_peered":"2024-08-17T17:33:20.796679+0000","last_unstale":"2024-08-17T17:40:42.925137+0000","last_undegraded":"2024-08-17T17:40:42.925137+0000","last_fullsized":"2024-08-17T17:40:42.925137+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:51.438262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":345,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-08-17T17:40:26.846812+0000","last_change":"2024-08-17T17:39:01.579531+0000","last_active":"2024-08-17T17:40:26.846812+0000","last_peered":"2024-08-17T17:40:26.846812+0000","last_clean":"2024-08-17T17:40:26.846812+0000","last_became_active":"2024-08-17T17:39:01.558717+0000","last_became_peered":"2024-08-17T17:39:01.558717+0000","last_unstale":"2024-08-17T17:40:26.846812+0000","last_undegraded":"2024-08-17T17:40:26.846812+0000","last_fullsized":"2024-08-17T17:40:26.846812+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:55:34.087601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":102,"num_keys_recovered":30,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"90'149","reported_seq":421,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-08-17T17:40:26.855403+0000","last_change":"2024-08-17T17:38:31.874623+0000","last_active":"2024-08-17T17:40:26.855403+0000","last_peered":"2024-08-17T17:40:26.855403+0000","last_clean":"2024-08-17T17:40:26.855403+0000","last_became_active":"2024-08-17T17:38:30.339313+0000","last_became_peered":"2024-08-17T17:38:30.339313+0000","last_unstale":"2024-08-17T17:40:26.855403+0000","last_undegraded":"2024-08-17T17:40:26.855403+0000","last_fullsized":"2024-08-17T17:40:26.855403+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:31.874423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:31.874423+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:28:17.980339+0000","scrub_duration":0.0017696579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":205,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.917285+0000","last_change":"2024-08-17T17:33:44.156036+0000","last_active":"2024-08-17T17:40:42.917285+0000","last_peered":"2024-08-17T17:40:42.917285+0000","last_clean":"2024-08-17T17:40:42.917285+0000","last_became_active":"2024-08-17T17:33:44.155758+0000","last_became_peered":"2024-08-17T17:33:44.155758+0000","last_unstale":"2024-08-17T17:40:42.917285+0000","last_undegraded":"2024-08-17T17:40:42.917285+0000","last_fullsized":"2024-08-17T17:40:42.917285+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:23.443405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":345,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.924077+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:40:42.924077+0000","last_peered":"2024-08-17T17:40:42.924077+0000","last_clean":"2024-08-17T17:40:42.924077+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:40:42.924077+0000","last_undegraded":"2024-08-17T17:40:42.924077+0000","last_fullsized":"2024-08-17T17:40:42.924077+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"90'149","reported_seq":730,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.914661+0000","last_change":"2024-08-17T17:39:04.605229+0000","last_active":"2024-08-17T17:40:42.914661+0000","last_peered":"2024-08-17T17:40:42.914661+0000","last_clean":"2024-08-17T17:40:42.914661+0000","last_became_active":"2024-08-17T17:39:04.604491+0000","last_became_peered":"2024-08-17T17:39:04.604491+0000","last_unstale":"2024-08-17T17:40:42.914661+0000","last_undegraded":"2024-08-17T17:40:42.914661+0000","last_fullsized":"2024-08-17T17:40:42.914661+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.844252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.844252+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:42:31.463485+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":344,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.924217+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:40:42.924217+0000","last_peered":"2024-08-17T17:40:42.924217+0000","last_clean":"2024-08-17T17:40:42.924217+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:40:42.924217+0000","last_undegraded":"2024-08-17T17:40:42.924217+0000","last_fullsized":"2024-08-17T17:40:42.924217+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":121,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-08-17T17:40:26.855678+0000","last_change":"2024-08-17T17:39:01.561623+0000","last_active":"2024-08-17T17:40:26.855678+0000","last_peered":"2024-08-17T17:40:26.855678+0000","last_clean":"2024-08-17T17:40:26.855678+0000","last_became_active":"2024-08-17T17:39:01.560618+0000","last_became_peered":"2024-08-17T17:39:01.560618+0000","last_unstale":"2024-08-17T17:40:26.855678+0000","last_undegraded":"2024-08-17T17:40:26.855678+0000","last_fullsized":"2024-08-17T17:40:26.855678+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:23:14.839861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.12","version":"90'149","reported_seq":716,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.925549+0000","last_change":"2024-08-17T17:38:30.446165+0000","last_active":"2024-08-17T17:40:42.925549+0000","last_peered":"2024-08-17T17:40:42.925549+0000","last_clean":"2024-08-17T17:40:42.925549+0000","last_became_active":"2024-08-17T17:38:30.445048+0000","last_became_peered":"2024-08-17T17:38:30.445048+0000","last_unstale":"2024-08-17T17:40:42.925549+0000","last_undegraded":"2024-08-17T17:40:42.925549+0000","last_fullsized":"2024-08-17T17:40:42.925549+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:24.291592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:24.291592+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:38:58.855193+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":558,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.920272+0000","last_change":"2024-08-17T17:33:20.794682+0000","last_active":"2024-08-17T17:40:42.920272+0000","last_peered":"2024-08-17T17:40:42.920272+0000","last_clean":"2024-08-17T17:40:42.920272+0000","last_became_active":"2024-08-17T17:33:20.794238+0000","last_became_peered":"2024-08-17T17:33:20.794238+0000","last_unstale":"2024-08-17T17:40:42.920272+0000","last_undegraded":"2024-08-17T17:40:42.920272+0000","last_fullsized":"2024-08-17T17:40:42.920272+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:15:55.821920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":383,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-08-17T17:40:26.855622+0000","last_change":"2024-08-17T17:24:53.465875+0000","last_active":"2024-08-17T17:40:26.855622+0000","last_peered":"2024-08-17T17:40:26.855622+0000","last_clean":"2024-08-17T17:40:26.855622+0000","last_became_active":"2024-08-17T17:24:53.458129+0000","last_became_peered":"2024-08-17T17:24:53.458129+0000","last_unstale":"2024-08-17T17:40:26.855622+0000","last_undegraded":"2024-08-17T17:40:26.855622+0000","last_fullsized":"2024-08-17T17:40:26.855622+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:51:33.584869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"90'149","reported_seq":683,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-08-17T17:40:26.846792+0000","last_change":"2024-08-17T17:39:04.576278+0000","last_active":"2024-08-17T17:40:26.846792+0000","last_peered":"2024-08-17T17:40:26.846792+0000","last_clean":"2024-08-17T17:40:26.846792+0000","last_became_active":"2024-08-17T17:39:04.576038+0000","last_became_peered":"2024-08-17T17:39:04.576038+0000","last_unstale":"2024-08-17T17:40:26.846792+0000","last_undegraded":"2024-08-17T17:40:26.846792+0000","last_fullsized":"2024-08-17T17:40:26.846792+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:35.370936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:35.370936+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:15:01.712930+0000","scrub_duration":0.0033549180000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"337'11546","reported_seq":11610,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:54.786232+0000","last_change":"2024-08-17T17:37:39.458750+0000","last_active":"2024-08-17T17:40:54.786232+0000","last_peered":"2024-08-17T17:40:54.786232+0000","last_clean":"2024-08-17T17:40:54.786232+0000","last_became_active":"2024-08-17T17:37:39.458436+0000","last_became_peered":"2024-08-17T17:37:39.458436+0000","last_unstale":"2024-08-17T17:40:54.786232+0000","last_undegraded":"2024-08-17T17:40:54.786232+0000","last_fullsized":"2024-08-17T17:40:54.786232+0000","mapping_epoch":311,"log_start":"324'5116","ondisk_log_start":"324'5116","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":6430,"log_dups_size":0,"ondisk_log_size":6430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:12:25.533220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195428352,"num_objects":2982,"num_object_clones":0,"num_object_copies":8946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11546,"num_write_kb":464896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":284,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-08-17T17:40:26.847107+0000","last_change":"2024-08-17T17:39:01.581318+0000","last_active":"2024-08-17T17:40:26.847107+0000","last_peered":"2024-08-17T17:40:26.847107+0000","last_clean":"2024-08-17T17:40:26.847107+0000","last_became_active":"2024-08-17T17:39:01.562753+0000","last_became_peered":"2024-08-17T17:39:01.562753+0000","last_unstale":"2024-08-17T17:40:26.847107+0000","last_undegraded":"2024-08-17T17:40:26.847107+0000","last_fullsized":"2024-08-17T17:40:26.847107+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:42:31.442637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":92,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-08-17T17:40:26.855481+0000","last_change":"2024-08-17T17:33:44.145598+0000","last_active":"2024-08-17T17:40:26.855481+0000","last_peered":"2024-08-17T17:40:26.855481+0000","last_clean":"2024-08-17T17:40:26.855481+0000","last_became_active":"2024-08-17T17:33:44.145283+0000","last_became_peered":"2024-08-17T17:33:44.145283+0000","last_unstale":"2024-08-17T17:40:26.855481+0000","last_undegraded":"2024-08-17T17:40:26.855481+0000","last_fullsized":"2024-08-17T17:40:26.855481+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:11.965226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"90'149","reported_seq":710,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.923971+0000","last_change":"2024-08-17T17:38:30.341631+0000","last_active":"2024-08-17T17:40:42.923971+0000","last_peered":"2024-08-17T17:40:42.923971+0000","last_clean":"2024-08-17T17:40:42.923971+0000","last_became_active":"2024-08-17T17:38:30.341195+0000","last_became_peered":"2024-08-17T17:38:30.341195+0000","last_unstale":"2024-08-17T17:40:42.923971+0000","last_undegraded":"2024-08-17T17:40:42.923971+0000","last_fullsized":"2024-08-17T17:40:42.923971+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:24.016647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:24.016647+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:45:56.818984+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.4","version":"337'11627","reported_seq":11690,"reported_epoch":337,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:40:55.623878+0000","last_change":"2024-08-17T17:39:03.007264+0000","last_active":"2024-08-17T17:40:55.623878+0000","last_peered":"2024-08-17T17:40:55.623878+0000","last_clean":"2024-08-17T17:39:00.556544+0000","last_became_active":"2024-08-17T17:39:02.996076+0000","last_became_peered":"2024-08-17T17:39:02.996076+0000","last_unstale":"2024-08-17T17:40:55.623878+0000","last_undegraded":"2024-08-17T17:39:02.598342+0000","last_fullsized":"2024-08-17T17:39:02.551968+0000","mapping_epoch":327,"log_start":"325'5932","ondisk_log_start":"325'5932","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":5695,"log_dups_size":0,"ondisk_log_size":5695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193134592,"num_objects":2947,"num_object_clones":0,"num_object_copies":8841,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6492,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11627,"num_write_kb":466368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3],"avail_no_missing":["1","3","4"],"object_location_counts":[{"shards":"1,3","objects":-3545},{"shards":"1,3,4","objects":2152}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":280,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-08-17T17:40:26.848647+0000","last_change":"2024-08-17T17:39:01.583216+0000","last_active":"2024-08-17T17:40:26.848647+0000","last_peered":"2024-08-17T17:40:26.848647+0000","last_clean":"2024-08-17T17:40:26.848647+0000","last_became_active":"2024-08-17T17:39:01.564671+0000","last_became_peered":"2024-08-17T17:39:01.564671+0000","last_unstale":"2024-08-17T17:40:26.848647+0000","last_undegraded":"2024-08-17T17:40:26.848647+0000","last_fullsized":"2024-08-17T17:40:26.848647+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:45:06.383901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":289,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.921800+0000","last_change":"2024-08-17T17:39:01.566676+0000","last_active":"2024-08-17T17:40:42.921800+0000","last_peered":"2024-08-17T17:40:42.921800+0000","last_clean":"2024-08-17T17:40:42.921800+0000","last_became_active":"2024-08-17T17:39:01.566148+0000","last_became_peered":"2024-08-17T17:39:01.566148+0000","last_unstale":"2024-08-17T17:40:42.921800+0000","last_undegraded":"2024-08-17T17:40:42.921800+0000","last_fullsized":"2024-08-17T17:40:42.921800+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:25:50.810755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.f","version":"90'149","reported_seq":494,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-08-17T17:40:26.856068+0000","last_change":"2024-08-17T17:39:04.581626+0000","last_active":"2024-08-17T17:40:26.856068+0000","last_peered":"2024-08-17T17:40:26.856068+0000","last_clean":"2024-08-17T17:40:26.856068+0000","last_became_active":"2024-08-17T17:39:04.581244+0000","last_became_peered":"2024-08-17T17:39:04.581244+0000","last_unstale":"2024-08-17T17:40:26.856068+0000","last_undegraded":"2024-08-17T17:40:26.856068+0000","last_fullsized":"2024-08-17T17:40:26.856068+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:32.843802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:32.843802+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:16:31.263947+0000","scrub_duration":0.0040803669999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.7","version":"337'11310","reported_seq":11374,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:54.296440+0000","last_change":"2024-08-17T17:37:39.459385+0000","last_active":"2024-08-17T17:40:54.296440+0000","last_peered":"2024-08-17T17:40:54.296440+0000","last_clean":"2024-08-17T17:40:54.296440+0000","last_became_active":"2024-08-17T17:37:39.458883+0000","last_became_peered":"2024-08-17T17:37:39.458883+0000","last_unstale":"2024-08-17T17:40:54.296440+0000","last_undegraded":"2024-08-17T17:40:54.296440+0000","last_fullsized":"2024-08-17T17:40:54.296440+0000","mapping_epoch":311,"log_start":"325'5595","ondisk_log_start":"325'5595","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":5715,"log_dups_size":0,"ondisk_log_size":5715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:02:52.635742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190447616,"num_objects":2906,"num_object_clones":0,"num_object_copies":8718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11310,"num_write_kb":454912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":88,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-08-17T17:40:26.856033+0000","last_change":"2024-08-17T17:33:44.145399+0000","last_active":"2024-08-17T17:40:26.856033+0000","last_peered":"2024-08-17T17:40:26.856033+0000","last_clean":"2024-08-17T17:40:26.856033+0000","last_became_active":"2024-08-17T17:33:44.144751+0000","last_became_peered":"2024-08-17T17:33:44.144751+0000","last_unstale":"2024-08-17T17:40:26.856033+0000","last_undegraded":"2024-08-17T17:40:26.856033+0000","last_fullsized":"2024-08-17T17:40:26.856033+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:54.276350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":166,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-08-17T17:40:26.855677+0000","last_change":"2024-08-17T17:33:20.795582+0000","last_active":"2024-08-17T17:40:26.855677+0000","last_peered":"2024-08-17T17:40:26.855677+0000","last_clean":"2024-08-17T17:40:26.855677+0000","last_became_active":"2024-08-17T17:33:20.795149+0000","last_became_peered":"2024-08-17T17:33:20.795149+0000","last_unstale":"2024-08-17T17:40:26.855677+0000","last_undegraded":"2024-08-17T17:40:26.855677+0000","last_fullsized":"2024-08-17T17:40:26.855677+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:14.020923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"90'149","reported_seq":656,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-08-17T17:40:26.846973+0000","last_change":"2024-08-17T17:39:04.581597+0000","last_active":"2024-08-17T17:40:26.846973+0000","last_peered":"2024-08-17T17:40:26.846973+0000","last_clean":"2024-08-17T17:40:26.846973+0000","last_became_active":"2024-08-17T17:39:04.581219+0000","last_became_peered":"2024-08-17T17:39:04.581219+0000","last_unstale":"2024-08-17T17:40:26.846973+0000","last_undegraded":"2024-08-17T17:40:26.846973+0000","last_fullsized":"2024-08-17T17:40:26.846973+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:32.597252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:32.597252+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"337'11478","reported_seq":11542,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:55.614368+0000","last_change":"2024-08-17T17:37:39.458059+0000","last_active":"2024-08-17T17:40:55.614368+0000","last_peered":"2024-08-17T17:40:55.614368+0000","last_clean":"2024-08-17T17:40:55.614368+0000","last_became_active":"2024-08-17T17:37:39.457731+0000","last_became_peered":"2024-08-17T17:37:39.457731+0000","last_unstale":"2024-08-17T17:40:55.614368+0000","last_undegraded":"2024-08-17T17:40:55.614368+0000","last_fullsized":"2024-08-17T17:40:55.614368+0000","mapping_epoch":311,"log_start":"325'5774","ondisk_log_start":"325'5774","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":5704,"log_dups_size":0,"ondisk_log_size":5704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:08:04.029927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193593344,"num_objects":2954,"num_object_clones":0,"num_object_copies":8862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11478,"num_write_kb":461824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":203,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.951528+0000","last_change":"2024-08-17T17:33:44.154097+0000","last_active":"2024-08-17T17:40:42.951528+0000","last_peered":"2024-08-17T17:40:42.951528+0000","last_clean":"2024-08-17T17:40:42.951528+0000","last_became_active":"2024-08-17T17:33:44.152972+0000","last_became_peered":"2024-08-17T17:33:44.152972+0000","last_unstale":"2024-08-17T17:40:42.951528+0000","last_undegraded":"2024-08-17T17:40:42.951528+0000","last_fullsized":"2024-08-17T17:40:42.951528+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:42.602017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":560,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.925698+0000","last_change":"2024-08-17T17:39:01.566491+0000","last_active":"2024-08-17T17:40:42.925698+0000","last_peered":"2024-08-17T17:40:42.925698+0000","last_clean":"2024-08-17T17:40:42.925698+0000","last_became_active":"2024-08-17T17:39:01.564362+0000","last_became_peered":"2024-08-17T17:39:01.564362+0000","last_unstale":"2024-08-17T17:40:42.925698+0000","last_undegraded":"2024-08-17T17:40:42.925698+0000","last_fullsized":"2024-08-17T17:40:42.925698+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:16:01.369342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1","version":"337'11472","reported_seq":11536,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:54.350309+0000","last_change":"2024-08-17T17:37:38.803662+0000","last_active":"2024-08-17T17:40:54.350309+0000","last_peered":"2024-08-17T17:40:54.350309+0000","last_clean":"2024-08-17T17:40:54.350309+0000","last_became_active":"2024-08-17T17:37:38.803355+0000","last_became_peered":"2024-08-17T17:37:38.803355+0000","last_unstale":"2024-08-17T17:40:54.350309+0000","last_undegraded":"2024-08-17T17:40:54.350309+0000","last_fullsized":"2024-08-17T17:40:54.350309+0000","mapping_epoch":311,"log_start":"325'5794","ondisk_log_start":"325'5794","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":5678,"log_dups_size":0,"ondisk_log_size":5678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:16:15.411410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191758336,"num_objects":2926,"num_object_clones":0,"num_object_copies":8778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2926,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11472,"num_write_kb":460736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":693,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.917131+0000","last_change":"2024-08-17T17:37:32.778073+0000","last_active":"2024-08-17T17:40:42.917131+0000","last_peered":"2024-08-17T17:40:42.917131+0000","last_clean":"2024-08-17T17:40:42.917131+0000","last_became_active":"2024-08-17T17:37:32.777646+0000","last_became_peered":"2024-08-17T17:37:32.777646+0000","last_unstale":"2024-08-17T17:40:42.917131+0000","last_undegraded":"2024-08-17T17:40:42.917131+0000","last_fullsized":"2024-08-17T17:40:42.917131+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:35:04.265381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:35:04.265381+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:08:23.389860+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":591,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.920296+0000","last_change":"2024-08-17T17:39:01.575154+0000","last_active":"2024-08-17T17:40:42.920296+0000","last_peered":"2024-08-17T17:40:42.920296+0000","last_clean":"2024-08-17T17:40:42.920296+0000","last_became_active":"2024-08-17T17:39:01.568605+0000","last_became_peered":"2024-08-17T17:39:01.568605+0000","last_unstale":"2024-08-17T17:40:42.920296+0000","last_undegraded":"2024-08-17T17:40:42.920296+0000","last_fullsized":"2024-08-17T17:40:42.920296+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:19:38.025059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":387,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.951538+0000","last_change":"2024-08-17T17:39:01.567511+0000","last_active":"2024-08-17T17:40:42.951538+0000","last_peered":"2024-08-17T17:40:42.951538+0000","last_clean":"2024-08-17T17:40:42.951538+0000","last_became_active":"2024-08-17T17:39:01.566979+0000","last_became_peered":"2024-08-17T17:39:01.566979+0000","last_unstale":"2024-08-17T17:40:42.951538+0000","last_undegraded":"2024-08-17T17:40:42.951538+0000","last_fullsized":"2024-08-17T17:40:42.951538+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:55:09.441541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.0","version":"337'11580","reported_seq":11644,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:55.617133+0000","last_change":"2024-08-17T17:37:38.939023+0000","last_active":"2024-08-17T17:40:55.617133+0000","last_peered":"2024-08-17T17:40:55.617133+0000","last_clean":"2024-08-17T17:40:55.617133+0000","last_became_active":"2024-08-17T17:37:38.938675+0000","last_became_peered":"2024-08-17T17:37:38.938675+0000","last_unstale":"2024-08-17T17:40:55.617133+0000","last_undegraded":"2024-08-17T17:40:55.617133+0000","last_fullsized":"2024-08-17T17:40:55.617133+0000","mapping_epoch":311,"log_start":"325'5874","ondisk_log_start":"325'5874","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":5706,"log_dups_size":0,"ondisk_log_size":5706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:59:47.176803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187301888,"num_objects":2858,"num_object_clones":0,"num_object_copies":8574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11580,"num_write_kb":462016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":693,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.917161+0000","last_change":"2024-08-17T17:37:32.781009+0000","last_active":"2024-08-17T17:40:42.917161+0000","last_peered":"2024-08-17T17:40:42.917161+0000","last_clean":"2024-08-17T17:40:42.917161+0000","last_became_active":"2024-08-17T17:37:32.780627+0000","last_became_peered":"2024-08-17T17:37:32.780627+0000","last_unstale":"2024-08-17T17:40:42.917161+0000","last_undegraded":"2024-08-17T17:40:42.917161+0000","last_fullsized":"2024-08-17T17:40:42.917161+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:35:02.239705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:35:02.239705+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:57:17.418142+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":263,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.925282+0000","last_change":"2024-08-17T17:39:01.567663+0000","last_active":"2024-08-17T17:40:42.925282+0000","last_peered":"2024-08-17T17:40:42.925282+0000","last_clean":"2024-08-17T17:40:42.925282+0000","last_became_active":"2024-08-17T17:39:01.567355+0000","last_became_peered":"2024-08-17T17:39:01.567355+0000","last_unstale":"2024-08-17T17:40:42.925282+0000","last_undegraded":"2024-08-17T17:40:42.925282+0000","last_fullsized":"2024-08-17T17:40:42.925282+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:39:40.654857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":20,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-08-17T17:40:26.855926+0000","last_change":"2024-08-17T17:39:01.561695+0000","last_active":"2024-08-17T17:40:26.855926+0000","last_peered":"2024-08-17T17:40:26.855926+0000","last_clean":"2024-08-17T17:40:26.855926+0000","last_became_active":"2024-08-17T17:39:01.561258+0000","last_became_peered":"2024-08-17T17:39:01.561258+0000","last_unstale":"2024-08-17T17:40:26.855926+0000","last_undegraded":"2024-08-17T17:40:26.855926+0000","last_fullsized":"2024-08-17T17:40:26.855926+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:59:32.336318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"90'149","reported_seq":720,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.914756+0000","last_change":"2024-08-17T17:38:30.440514+0000","last_active":"2024-08-17T17:40:42.914756+0000","last_peered":"2024-08-17T17:40:42.914756+0000","last_clean":"2024-08-17T17:40:42.914756+0000","last_became_active":"2024-08-17T17:38:30.439649+0000","last_became_peered":"2024-08-17T17:38:30.439649+0000","last_unstale":"2024-08-17T17:40:42.914756+0000","last_undegraded":"2024-08-17T17:40:42.914756+0000","last_fullsized":"2024-08-17T17:40:42.914756+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:22.806597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:22.806597+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:12:11.905510+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"337'11505","reported_seq":11539,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:54.257287+0000","last_change":"2024-08-17T17:37:38.804823+0000","last_active":"2024-08-17T17:40:54.257287+0000","last_peered":"2024-08-17T17:40:54.257287+0000","last_clean":"2024-08-17T17:40:54.257287+0000","last_became_active":"2024-08-17T17:37:38.804431+0000","last_became_peered":"2024-08-17T17:37:38.804431+0000","last_unstale":"2024-08-17T17:40:54.257287+0000","last_undegraded":"2024-08-17T17:40:54.257287+0000","last_fullsized":"2024-08-17T17:40:54.257287+0000","mapping_epoch":311,"log_start":"325'5496","ondisk_log_start":"325'5496","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":6009,"log_dups_size":0,"ondisk_log_size":6009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:03:53.415193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193789952,"num_objects":2957,"num_object_clones":0,"num_object_copies":8871,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11505,"num_write_kb":462784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":270,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-08-17T17:40:26.846604+0000","last_change":"2024-08-17T17:39:01.559671+0000","last_active":"2024-08-17T17:40:26.846604+0000","last_peered":"2024-08-17T17:40:26.846604+0000","last_clean":"2024-08-17T17:40:26.846604+0000","last_became_active":"2024-08-17T17:39:01.557716+0000","last_became_peered":"2024-08-17T17:39:01.557716+0000","last_unstale":"2024-08-17T17:40:26.846604+0000","last_undegraded":"2024-08-17T17:40:26.846604+0000","last_fullsized":"2024-08-17T17:40:26.846604+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:58:55.377710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":100,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-08-17T17:40:26.856073+0000","last_change":"2024-08-17T17:39:01.567132+0000","last_active":"2024-08-17T17:40:26.856073+0000","last_peered":"2024-08-17T17:40:26.856073+0000","last_clean":"2024-08-17T17:40:26.856073+0000","last_became_active":"2024-08-17T17:39:01.565828+0000","last_became_peered":"2024-08-17T17:39:01.565828+0000","last_unstale":"2024-08-17T17:40:26.856073+0000","last_undegraded":"2024-08-17T17:40:26.856073+0000","last_fullsized":"2024-08-17T17:40:26.856073+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:14:51.546222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.2","version":"337'11604","reported_seq":11668,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:54.334803+0000","last_change":"2024-08-17T17:37:38.939685+0000","last_active":"2024-08-17T17:40:54.334803+0000","last_peered":"2024-08-17T17:40:54.334803+0000","last_clean":"2024-08-17T17:40:54.334803+0000","last_became_active":"2024-08-17T17:37:38.939345+0000","last_became_peered":"2024-08-17T17:37:38.939345+0000","last_unstale":"2024-08-17T17:40:54.334803+0000","last_undegraded":"2024-08-17T17:40:54.334803+0000","last_fullsized":"2024-08-17T17:40:54.334803+0000","mapping_epoch":311,"log_start":"325'5933","ondisk_log_start":"325'5933","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":5671,"log_dups_size":0,"ondisk_log_size":5671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:07:23.306893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193462272,"num_objects":2952,"num_object_clones":0,"num_object_copies":8856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11604,"num_write_kb":465792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":435,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-08-17T17:40:26.855558+0000","last_change":"2024-08-17T17:38:25.844844+0000","last_active":"2024-08-17T17:40:26.855558+0000","last_peered":"2024-08-17T17:40:26.855558+0000","last_clean":"2024-08-17T17:40:26.855558+0000","last_became_active":"2024-08-17T17:37:32.775713+0000","last_became_peered":"2024-08-17T17:37:32.775713+0000","last_unstale":"2024-08-17T17:40:26.855558+0000","last_undegraded":"2024-08-17T17:40:26.855558+0000","last_fullsized":"2024-08-17T17:40:26.855558+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:25.844747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:25.844747+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:38:00.170417+0000","scrub_duration":0.0032978740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":560,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.917624+0000","last_change":"2024-08-17T17:33:20.796216+0000","last_active":"2024-08-17T17:40:42.917624+0000","last_peered":"2024-08-17T17:40:42.917624+0000","last_clean":"2024-08-17T17:40:42.917624+0000","last_became_active":"2024-08-17T17:33:20.795762+0000","last_became_peered":"2024-08-17T17:33:20.795762+0000","last_unstale":"2024-08-17T17:40:42.917624+0000","last_undegraded":"2024-08-17T17:40:42.917624+0000","last_fullsized":"2024-08-17T17:40:42.917624+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:47.038509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":449,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.915891+0000","last_change":"2024-08-17T17:39:01.567695+0000","last_active":"2024-08-17T17:40:42.915891+0000","last_peered":"2024-08-17T17:40:42.915891+0000","last_clean":"2024-08-17T17:40:42.915891+0000","last_became_active":"2024-08-17T17:39:01.566677+0000","last_became_peered":"2024-08-17T17:39:01.566677+0000","last_unstale":"2024-08-17T17:40:42.915891+0000","last_undegraded":"2024-08-17T17:40:42.915891+0000","last_fullsized":"2024-08-17T17:40:42.915891+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:48:22.014206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.8","version":"337'11548","reported_seq":11612,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:54.362357+0000","last_change":"2024-08-17T17:37:39.459686+0000","last_active":"2024-08-17T17:40:54.362357+0000","last_peered":"2024-08-17T17:40:54.362357+0000","last_clean":"2024-08-17T17:40:54.362357+0000","last_became_active":"2024-08-17T17:37:39.459260+0000","last_became_peered":"2024-08-17T17:37:39.459260+0000","last_unstale":"2024-08-17T17:40:54.362357+0000","last_undegraded":"2024-08-17T17:40:54.362357+0000","last_fullsized":"2024-08-17T17:40:54.362357+0000","mapping_epoch":311,"log_start":"325'5790","ondisk_log_start":"325'5790","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":5758,"log_dups_size":0,"ondisk_log_size":5758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:54:14.542106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186253312,"num_objects":2842,"num_object_clones":0,"num_object_copies":8526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11548,"num_write_kb":460480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":694,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.923465+0000","last_change":"2024-08-17T17:38:25.247159+0000","last_active":"2024-08-17T17:40:42.923465+0000","last_peered":"2024-08-17T17:40:42.923465+0000","last_clean":"2024-08-17T17:40:42.923465+0000","last_became_active":"2024-08-17T17:38:21.311361+0000","last_became_peered":"2024-08-17T17:38:21.311361+0000","last_unstale":"2024-08-17T17:40:42.923465+0000","last_undegraded":"2024-08-17T17:40:42.923465+0000","last_fullsized":"2024-08-17T17:40:42.923465+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:25.247027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:25.247027+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:48:51.235302+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":743,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-08-17T17:40:26.846524+0000","last_change":"2024-08-17T17:33:44.149928+0000","last_active":"2024-08-17T17:40:26.846524+0000","last_peered":"2024-08-17T17:40:26.846524+0000","last_clean":"2024-08-17T17:40:26.846524+0000","last_became_active":"2024-08-17T17:33:44.149498+0000","last_became_peered":"2024-08-17T17:33:44.149498+0000","last_unstale":"2024-08-17T17:40:26.846524+0000","last_undegraded":"2024-08-17T17:40:26.846524+0000","last_fullsized":"2024-08-17T17:40:26.846524+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:17:04.963202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":508,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.919997+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:40:42.919997+0000","last_peered":"2024-08-17T17:40:42.919997+0000","last_clean":"2024-08-17T17:40:42.919997+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:40:42.919997+0000","last_undegraded":"2024-08-17T17:40:42.919997+0000","last_fullsized":"2024-08-17T17:40:42.919997+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":169,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-08-17T17:40:26.856013+0000","last_change":"2024-08-17T17:39:01.562404+0000","last_active":"2024-08-17T17:40:26.856013+0000","last_peered":"2024-08-17T17:40:26.856013+0000","last_clean":"2024-08-17T17:40:26.856013+0000","last_became_active":"2024-08-17T17:39:01.562044+0000","last_became_peered":"2024-08-17T17:39:01.562044+0000","last_unstale":"2024-08-17T17:40:26.856013+0000","last_undegraded":"2024-08-17T17:40:26.856013+0000","last_fullsized":"2024-08-17T17:40:26.856013+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:10:28.532320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"337'11571","reported_seq":11635,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:54.718226+0000","last_change":"2024-08-17T17:37:39.458006+0000","last_active":"2024-08-17T17:40:54.718226+0000","last_peered":"2024-08-17T17:40:54.718226+0000","last_clean":"2024-08-17T17:40:54.718226+0000","last_became_active":"2024-08-17T17:37:39.457681+0000","last_became_peered":"2024-08-17T17:37:39.457681+0000","last_unstale":"2024-08-17T17:40:54.718226+0000","last_undegraded":"2024-08-17T17:40:54.718226+0000","last_fullsized":"2024-08-17T17:40:54.718226+0000","mapping_epoch":311,"log_start":"324'5100","ondisk_log_start":"324'5100","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":6471,"log_dups_size":0,"ondisk_log_size":6471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:09:17.360518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190119936,"num_objects":2901,"num_object_clones":0,"num_object_copies":8703,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11571,"num_write_kb":463104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":721,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.914957+0000","last_change":"2024-08-17T17:39:04.591366+0000","last_active":"2024-08-17T17:40:42.914957+0000","last_peered":"2024-08-17T17:40:42.914957+0000","last_clean":"2024-08-17T17:40:42.914957+0000","last_became_active":"2024-08-17T17:39:04.590890+0000","last_became_peered":"2024-08-17T17:39:04.590890+0000","last_unstale":"2024-08-17T17:40:42.914957+0000","last_undegraded":"2024-08-17T17:40:42.914957+0000","last_fullsized":"2024-08-17T17:40:42.914957+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:21.814077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:21.814077+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:15:26.176604+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":792,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.914195+0000","last_change":"2024-08-17T17:33:20.797705+0000","last_active":"2024-08-17T17:40:42.914195+0000","last_peered":"2024-08-17T17:40:42.914195+0000","last_clean":"2024-08-17T17:40:42.914195+0000","last_became_active":"2024-08-17T17:33:20.797172+0000","last_became_peered":"2024-08-17T17:33:20.797172+0000","last_unstale":"2024-08-17T17:40:42.914195+0000","last_undegraded":"2024-08-17T17:40:42.914195+0000","last_fullsized":"2024-08-17T17:40:42.914195+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:30.754999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":551,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.924749+0000","last_change":"2024-08-17T17:39:01.563813+0000","last_active":"2024-08-17T17:40:42.924749+0000","last_peered":"2024-08-17T17:40:42.924749+0000","last_clean":"2024-08-17T17:40:42.924749+0000","last_became_active":"2024-08-17T17:39:01.563407+0000","last_became_peered":"2024-08-17T17:39:01.563407+0000","last_unstale":"2024-08-17T17:40:42.924749+0000","last_undegraded":"2024-08-17T17:40:42.924749+0000","last_fullsized":"2024-08-17T17:40:42.924749+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:14:09.230542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":345,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.924148+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:40:42.924148+0000","last_peered":"2024-08-17T17:40:42.924148+0000","last_clean":"2024-08-17T17:40:42.924148+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:40:42.924148+0000","last_undegraded":"2024-08-17T17:40:42.924148+0000","last_fullsized":"2024-08-17T17:40:42.924148+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"337'11601","reported_seq":11665,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:55.616251+0000","last_change":"2024-08-17T17:37:39.458235+0000","last_active":"2024-08-17T17:40:55.616251+0000","last_peered":"2024-08-17T17:40:55.616251+0000","last_clean":"2024-08-17T17:40:55.616251+0000","last_became_active":"2024-08-17T17:37:39.457773+0000","last_became_peered":"2024-08-17T17:37:39.457773+0000","last_unstale":"2024-08-17T17:40:55.616251+0000","last_undegraded":"2024-08-17T17:40:55.616251+0000","last_fullsized":"2024-08-17T17:40:55.616251+0000","mapping_epoch":311,"log_start":"325'5927","ondisk_log_start":"325'5927","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":5674,"log_dups_size":0,"ondisk_log_size":5674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:54:08.694457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187498496,"num_objects":2861,"num_object_clones":0,"num_object_copies":8583,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11601,"num_write_kb":462784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":724,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.923973+0000","last_change":"2024-08-17T17:39:04.581550+0000","last_active":"2024-08-17T17:40:42.923973+0000","last_peered":"2024-08-17T17:40:42.923973+0000","last_clean":"2024-08-17T17:40:42.923973+0000","last_became_active":"2024-08-17T17:39:04.581155+0000","last_became_peered":"2024-08-17T17:39:04.581155+0000","last_unstale":"2024-08-17T17:40:42.923973+0000","last_undegraded":"2024-08-17T17:40:42.923973+0000","last_fullsized":"2024-08-17T17:40:42.923973+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.289528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.289528+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:36:42.607547+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":924,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.917581+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:40:42.917581+0000","last_peered":"2024-08-17T17:40:42.917581+0000","last_clean":"2024-08-17T17:40:42.917581+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:40:42.917581+0000","last_undegraded":"2024-08-17T17:40:42.917581+0000","last_fullsized":"2024-08-17T17:40:42.917581+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":430,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.914282+0000","last_change":"2024-08-17T17:33:20.793468+0000","last_active":"2024-08-17T17:40:42.914282+0000","last_peered":"2024-08-17T17:40:42.914282+0000","last_clean":"2024-08-17T17:40:42.914282+0000","last_became_active":"2024-08-17T17:33:20.793166+0000","last_became_peered":"2024-08-17T17:33:20.793166+0000","last_unstale":"2024-08-17T17:40:42.914282+0000","last_undegraded":"2024-08-17T17:40:42.914282+0000","last_fullsized":"2024-08-17T17:40:42.914282+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:12:52.121050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":466,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.915798+0000","last_change":"2024-08-17T17:33:20.835784+0000","last_active":"2024-08-17T17:40:42.915798+0000","last_peered":"2024-08-17T17:40:42.915798+0000","last_clean":"2024-08-17T17:40:42.915798+0000","last_became_active":"2024-08-17T17:33:20.835371+0000","last_became_peered":"2024-08-17T17:33:20.835371+0000","last_unstale":"2024-08-17T17:40:42.915798+0000","last_undegraded":"2024-08-17T17:40:42.915798+0000","last_fullsized":"2024-08-17T17:40:42.915798+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:35:43.369055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"337'11682","reported_seq":11746,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:54.778639+0000","last_change":"2024-08-17T17:37:39.458043+0000","last_active":"2024-08-17T17:40:54.778639+0000","last_peered":"2024-08-17T17:40:54.778639+0000","last_clean":"2024-08-17T17:40:54.778639+0000","last_became_active":"2024-08-17T17:37:39.457730+0000","last_became_peered":"2024-08-17T17:37:39.457730+0000","last_unstale":"2024-08-17T17:40:54.778639+0000","last_undegraded":"2024-08-17T17:40:54.778639+0000","last_fullsized":"2024-08-17T17:40:54.778639+0000","mapping_epoch":311,"log_start":"324'5261","ondisk_log_start":"324'5261","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":6421,"log_dups_size":0,"ondisk_log_size":6421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:14:23.151646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193069056,"num_objects":2946,"num_object_clones":0,"num_object_copies":8838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11682,"num_write_kb":468096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":713,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.924619+0000","last_change":"2024-08-17T17:38:22.254841+0000","last_active":"2024-08-17T17:40:42.924619+0000","last_peered":"2024-08-17T17:40:42.924619+0000","last_clean":"2024-08-17T17:40:42.924619+0000","last_became_active":"2024-08-17T17:38:21.306619+0000","last_became_peered":"2024-08-17T17:38:21.306619+0000","last_unstale":"2024-08-17T17:40:42.924619+0000","last_undegraded":"2024-08-17T17:40:42.924619+0000","last_fullsized":"2024-08-17T17:40:42.924619+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:22.254705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:22.254705+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:04:31.055328+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":1015,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.925079+0000","last_change":"2024-08-17T17:33:44.148227+0000","last_active":"2024-08-17T17:40:42.925079+0000","last_peered":"2024-08-17T17:40:42.925079+0000","last_clean":"2024-08-17T17:40:42.925079+0000","last_became_active":"2024-08-17T17:33:44.147627+0000","last_became_peered":"2024-08-17T17:33:44.147627+0000","last_unstale":"2024-08-17T17:40:42.925079+0000","last_undegraded":"2024-08-17T17:40:42.925079+0000","last_fullsized":"2024-08-17T17:40:42.925079+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:29:42.898460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1372,"num_read_kb":2843,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":334,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.925415+0000","last_change":"2024-08-17T17:39:01.558829+0000","last_active":"2024-08-17T17:40:42.925415+0000","last_peered":"2024-08-17T17:40:42.925415+0000","last_clean":"2024-08-17T17:40:42.925415+0000","last_became_active":"2024-08-17T17:39:01.558487+0000","last_became_peered":"2024-08-17T17:39:01.558487+0000","last_unstale":"2024-08-17T17:40:42.925415+0000","last_undegraded":"2024-08-17T17:40:42.925415+0000","last_fullsized":"2024-08-17T17:40:42.925415+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:10:21.209195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":470,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.931549+0000","last_change":"2024-08-17T17:33:20.772454+0000","last_active":"2024-08-17T17:40:42.931549+0000","last_peered":"2024-08-17T17:40:42.931549+0000","last_clean":"2024-08-17T17:40:42.931549+0000","last_became_active":"2024-08-17T17:33:20.771848+0000","last_became_peered":"2024-08-17T17:33:20.771848+0000","last_unstale":"2024-08-17T17:40:42.931549+0000","last_undegraded":"2024-08-17T17:40:42.931549+0000","last_fullsized":"2024-08-17T17:40:42.931549+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:46:42.300744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.f","version":"337'11629","reported_seq":11693,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:55.568314+0000","last_change":"2024-08-17T17:37:39.458090+0000","last_active":"2024-08-17T17:40:55.568314+0000","last_peered":"2024-08-17T17:40:55.568314+0000","last_clean":"2024-08-17T17:40:55.568314+0000","last_became_active":"2024-08-17T17:37:39.457805+0000","last_became_peered":"2024-08-17T17:37:39.457805+0000","last_unstale":"2024-08-17T17:40:55.568314+0000","last_undegraded":"2024-08-17T17:40:55.568314+0000","last_fullsized":"2024-08-17T17:40:55.568314+0000","mapping_epoch":311,"log_start":"325'5953","ondisk_log_start":"325'5953","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":5676,"log_dups_size":0,"ondisk_log_size":5676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:02:18.222490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190513152,"num_objects":2907,"num_object_clones":0,"num_object_copies":8721,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11629,"num_write_kb":465152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":455,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-08-17T17:40:26.856035+0000","last_change":"2024-08-17T17:38:27.878557+0000","last_active":"2024-08-17T17:40:26.856035+0000","last_peered":"2024-08-17T17:40:26.856035+0000","last_clean":"2024-08-17T17:40:26.856035+0000","last_became_active":"2024-08-17T17:38:21.306565+0000","last_became_peered":"2024-08-17T17:38:21.306565+0000","last_unstale":"2024-08-17T17:40:26.856035+0000","last_undegraded":"2024-08-17T17:40:26.856035+0000","last_fullsized":"2024-08-17T17:40:26.856035+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:27.878444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:27.878444+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:49:42.179687+0000","scrub_duration":0.0041727099999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":220,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.917510+0000","last_change":"2024-08-17T17:33:44.163867+0000","last_active":"2024-08-17T17:40:42.917510+0000","last_peered":"2024-08-17T17:40:42.917510+0000","last_clean":"2024-08-17T17:40:42.917510+0000","last_became_active":"2024-08-17T17:33:44.163503+0000","last_became_peered":"2024-08-17T17:33:44.163503+0000","last_unstale":"2024-08-17T17:40:42.917510+0000","last_undegraded":"2024-08-17T17:40:42.917510+0000","last_fullsized":"2024-08-17T17:40:42.917510+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:31:53.573596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":685,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.927118+0000","last_change":"2024-08-17T17:33:20.799350+0000","last_active":"2024-08-17T17:40:42.927118+0000","last_peered":"2024-08-17T17:40:42.927118+0000","last_clean":"2024-08-17T17:40:42.927118+0000","last_became_active":"2024-08-17T17:33:20.798925+0000","last_became_peered":"2024-08-17T17:33:20.798925+0000","last_unstale":"2024-08-17T17:40:42.927118+0000","last_undegraded":"2024-08-17T17:40:42.927118+0000","last_fullsized":"2024-08-17T17:40:42.927118+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:16:36.121295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":494,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.925661+0000","last_change":"2024-08-17T17:39:01.564610+0000","last_active":"2024-08-17T17:40:42.925661+0000","last_peered":"2024-08-17T17:40:42.925661+0000","last_clean":"2024-08-17T17:40:42.925661+0000","last_became_active":"2024-08-17T17:39:01.564047+0000","last_became_peered":"2024-08-17T17:39:01.564047+0000","last_unstale":"2024-08-17T17:40:42.925661+0000","last_undegraded":"2024-08-17T17:40:42.925661+0000","last_fullsized":"2024-08-17T17:40:42.925661+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:11:50.269346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.e","version":"337'11692","reported_seq":16652,"reported_epoch":337,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-17T17:40:55.637889+0000","last_change":"2024-08-17T17:39:35.232601+0000","last_active":"2024-08-17T17:40:55.637889+0000","last_peered":"2024-08-17T17:40:55.637889+0000","last_clean":"2024-08-17T17:39:00.487737+0000","last_became_active":"2024-08-17T17:39:02.844956+0000","last_became_peered":"2024-08-17T17:39:02.844956+0000","last_unstale":"2024-08-17T17:40:55.637889+0000","last_undegraded":"2024-08-17T17:39:35.230572+0000","last_fullsized":"2024-08-17T17:39:02.561039+0000","mapping_epoch":327,"log_start":"325'5983","ondisk_log_start":"325'5983","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":5709,"log_dups_size":0,"ondisk_log_size":5709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187432960,"num_objects":2860,"num_object_clones":0,"num_object_copies":8580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1963,"num_objects_misplaced":2125,"num_objects_unfound":0,"num_objects_dirty":2860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11692,"num_write_kb":465664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2455,"num_bytes_recovered":160890880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7],"avail_no_missing":["1","4","7"],"object_location_counts":[{"shards":"1,4,7","objects":2125},{"shards":"1,7","objects":-1228}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":455,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-08-17T17:40:26.855985+0000","last_change":"2024-08-17T17:38:29.833878+0000","last_active":"2024-08-17T17:40:26.855985+0000","last_peered":"2024-08-17T17:40:26.855985+0000","last_clean":"2024-08-17T17:40:26.855985+0000","last_became_active":"2024-08-17T17:38:21.306189+0000","last_became_peered":"2024-08-17T17:38:21.306189+0000","last_unstale":"2024-08-17T17:40:26.855985+0000","last_undegraded":"2024-08-17T17:40:26.855985+0000","last_fullsized":"2024-08-17T17:40:26.855985+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:29.833791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:29.833791+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:08:59.035531+0000","scrub_duration":0.0038975160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":505,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.927033+0000","last_change":"2024-08-17T17:33:20.798613+0000","last_active":"2024-08-17T17:40:42.927033+0000","last_peered":"2024-08-17T17:40:42.927033+0000","last_clean":"2024-08-17T17:40:42.927033+0000","last_became_active":"2024-08-17T17:33:20.798260+0000","last_became_peered":"2024-08-17T17:33:20.798260+0000","last_unstale":"2024-08-17T17:40:42.927033+0000","last_undegraded":"2024-08-17T17:40:42.927033+0000","last_fullsized":"2024-08-17T17:40:42.927033+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:18:00.822681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":855,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-08-17T17:40:26.847093+0000","last_change":"2024-08-17T17:39:04.576291+0000","last_active":"2024-08-17T17:40:26.847093+0000","last_peered":"2024-08-17T17:40:26.847093+0000","last_clean":"2024-08-17T17:40:26.847093+0000","last_became_active":"2024-08-17T17:39:04.576027+0000","last_became_peered":"2024-08-17T17:39:04.576027+0000","last_unstale":"2024-08-17T17:40:26.847093+0000","last_undegraded":"2024-08-17T17:40:26.847093+0000","last_fullsized":"2024-08-17T17:40:26.847093+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:02:04.267251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":520,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.925544+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:40:42.925544+0000","last_peered":"2024-08-17T17:40:42.925544+0000","last_clean":"2024-08-17T17:40:42.925544+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:40:42.925544+0000","last_undegraded":"2024-08-17T17:40:42.925544+0000","last_fullsized":"2024-08-17T17:40:42.925544+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.d","version":"337'11711","reported_seq":11776,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:55.621355+0000","last_change":"2024-08-17T17:37:39.460626+0000","last_active":"2024-08-17T17:40:55.621355+0000","last_peered":"2024-08-17T17:40:55.621355+0000","last_clean":"2024-08-17T17:40:55.621355+0000","last_became_active":"2024-08-17T17:37:39.459631+0000","last_became_peered":"2024-08-17T17:37:39.459631+0000","last_unstale":"2024-08-17T17:40:55.621355+0000","last_undegraded":"2024-08-17T17:40:55.621355+0000","last_fullsized":"2024-08-17T17:40:55.621355+0000","mapping_epoch":311,"log_start":"325'5995","ondisk_log_start":"325'5995","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":5716,"log_dups_size":0,"ondisk_log_size":5716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:51:35.697161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188612632,"num_objects":2879,"num_object_clones":0,"num_object_copies":8637,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2879,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11711,"num_write_kb":466817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":676,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.933662+0000","last_change":"2024-08-17T17:39:20.822196+0000","last_active":"2024-08-17T17:40:42.933662+0000","last_peered":"2024-08-17T17:40:42.933662+0000","last_clean":"2024-08-17T17:40:42.933662+0000","last_became_active":"2024-08-17T17:39:04.578879+0000","last_became_peered":"2024-08-17T17:39:04.578879+0000","last_unstale":"2024-08-17T17:40:42.933662+0000","last_undegraded":"2024-08-17T17:40:42.933662+0000","last_fullsized":"2024-08-17T17:40:42.933662+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:39:20.822072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:39:20.822072+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:00:53.735025+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":554,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.920467+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:40:42.920467+0000","last_peered":"2024-08-17T17:40:42.920467+0000","last_clean":"2024-08-17T17:40:42.920467+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:40:42.920467+0000","last_undegraded":"2024-08-17T17:40:42.920467+0000","last_fullsized":"2024-08-17T17:40:42.920467+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":1084,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.929734+0000","last_change":"2024-08-17T17:39:04.593062+0000","last_active":"2024-08-17T17:40:42.929734+0000","last_peered":"2024-08-17T17:40:42.929734+0000","last_clean":"2024-08-17T17:40:42.929734+0000","last_became_active":"2024-08-17T17:39:04.592291+0000","last_became_peered":"2024-08-17T17:39:04.592291+0000","last_unstale":"2024-08-17T17:40:42.929734+0000","last_undegraded":"2024-08-17T17:40:42.929734+0000","last_fullsized":"2024-08-17T17:40:42.929734+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:21:47.935115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":551,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.917329+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:40:42.917329+0000","last_peered":"2024-08-17T17:40:42.917329+0000","last_clean":"2024-08-17T17:40:42.917329+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:40:42.917329+0000","last_undegraded":"2024-08-17T17:40:42.917329+0000","last_fullsized":"2024-08-17T17:40:42.917329+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"337'11563","reported_seq":11630,"reported_epoch":337,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-08-17T17:40:54.340184+0000","last_change":"2024-08-17T17:39:02.849620+0000","last_active":"2024-08-17T17:40:54.340184+0000","last_peered":"2024-08-17T17:40:54.340184+0000","last_clean":"2024-08-17T17:39:00.544064+0000","last_became_active":"2024-08-17T17:39:02.829606+0000","last_became_peered":"2024-08-17T17:39:02.829606+0000","last_unstale":"2024-08-17T17:40:54.340184+0000","last_undegraded":"2024-08-17T17:39:02.597728+0000","last_fullsized":"2024-08-17T17:39:02.552281+0000","mapping_epoch":327,"log_start":"325'5817","ondisk_log_start":"325'5817","created":311,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:37:37.783085+0000","last_clean_scrub_stamp":"2024-08-17T17:37:37.783085+0000","objects_scrubbed":0,"log_size":5746,"log_dups_size":0,"ondisk_log_size":5746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194576384,"num_objects":2969,"num_object_clones":0,"num_object_copies":8907,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6477,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11563,"num_write_kb":465024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"3,4,7","objects":2180},{"shards":"3,7","objects":-3508}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":695,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.923450+0000","last_change":"2024-08-17T17:38:23.113397+0000","last_active":"2024-08-17T17:40:42.923450+0000","last_peered":"2024-08-17T17:40:42.923450+0000","last_clean":"2024-08-17T17:40:42.923450+0000","last_became_active":"2024-08-17T17:38:21.305580+0000","last_became_peered":"2024-08-17T17:38:21.305580+0000","last_unstale":"2024-08-17T17:40:42.923450+0000","last_undegraded":"2024-08-17T17:40:42.923450+0000","last_fullsized":"2024-08-17T17:40:42.923450+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.113279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.113279+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:26:25.711926+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":380,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.924022+0000","last_change":"2024-08-17T17:39:01.571007+0000","last_active":"2024-08-17T17:40:42.924022+0000","last_peered":"2024-08-17T17:40:42.924022+0000","last_clean":"2024-08-17T17:40:42.924022+0000","last_became_active":"2024-08-17T17:39:01.556339+0000","last_became_peered":"2024-08-17T17:39:01.556339+0000","last_unstale":"2024-08-17T17:40:42.924022+0000","last_undegraded":"2024-08-17T17:40:42.924022+0000","last_fullsized":"2024-08-17T17:40:42.924022+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:15:21.130193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":568,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-08-17T17:40:26.856140+0000","last_change":"2024-08-17T17:26:06.855307+0000","last_active":"2024-08-17T17:40:26.856140+0000","last_peered":"2024-08-17T17:40:26.856140+0000","last_clean":"2024-08-17T17:40:26.856140+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:40:26.856140+0000","last_undegraded":"2024-08-17T17:40:26.856140+0000","last_fullsized":"2024-08-17T17:40:26.856140+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":65,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-08-17T17:40:26.855634+0000","last_change":"2024-08-17T17:39:01.568614+0000","last_active":"2024-08-17T17:40:26.855634+0000","last_peered":"2024-08-17T17:40:26.855634+0000","last_clean":"2024-08-17T17:40:26.855634+0000","last_became_active":"2024-08-17T17:39:01.567827+0000","last_became_peered":"2024-08-17T17:39:01.567827+0000","last_unstale":"2024-08-17T17:40:26.855634+0000","last_undegraded":"2024-08-17T17:40:26.855634+0000","last_fullsized":"2024-08-17T17:40:26.855634+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:21:47.166997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":208,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-08-17T17:40:26.855865+0000","last_change":"2024-08-17T17:39:01.567870+0000","last_active":"2024-08-17T17:40:26.855865+0000","last_peered":"2024-08-17T17:40:26.855865+0000","last_clean":"2024-08-17T17:40:26.855865+0000","last_became_active":"2024-08-17T17:39:01.567200+0000","last_became_peered":"2024-08-17T17:39:01.567200+0000","last_unstale":"2024-08-17T17:40:26.855865+0000","last_undegraded":"2024-08-17T17:40:26.855865+0000","last_fullsized":"2024-08-17T17:40:26.855865+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:25:31.161816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":289,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-08-17T17:40:26.847006+0000","last_change":"2024-08-17T17:39:01.573670+0000","last_active":"2024-08-17T17:40:26.847006+0000","last_peered":"2024-08-17T17:40:26.847006+0000","last_clean":"2024-08-17T17:40:26.847006+0000","last_became_active":"2024-08-17T17:39:01.558648+0000","last_became_peered":"2024-08-17T17:39:01.558648+0000","last_unstale":"2024-08-17T17:40:26.847006+0000","last_undegraded":"2024-08-17T17:40:26.847006+0000","last_fullsized":"2024-08-17T17:40:26.847006+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:19:50.905619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":498,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.925191+0000","last_change":"2024-08-17T17:33:20.799402+0000","last_active":"2024-08-17T17:40:42.925191+0000","last_peered":"2024-08-17T17:40:42.925191+0000","last_clean":"2024-08-17T17:40:42.925191+0000","last_became_active":"2024-08-17T17:33:20.798832+0000","last_became_peered":"2024-08-17T17:33:20.798832+0000","last_unstale":"2024-08-17T17:40:42.925191+0000","last_undegraded":"2024-08-17T17:40:42.925191+0000","last_fullsized":"2024-08-17T17:40:42.925191+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:21:05.001134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":259,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.917398+0000","last_change":"2024-08-17T17:39:01.573596+0000","last_active":"2024-08-17T17:40:42.917398+0000","last_peered":"2024-08-17T17:40:42.917398+0000","last_clean":"2024-08-17T17:40:42.917398+0000","last_became_active":"2024-08-17T17:39:01.567912+0000","last_became_peered":"2024-08-17T17:39:01.567912+0000","last_unstale":"2024-08-17T17:40:42.917398+0000","last_undegraded":"2024-08-17T17:40:42.917398+0000","last_fullsized":"2024-08-17T17:40:42.917398+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:35.510974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":270,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.14","version":"90'149","reported_seq":709,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.934379+0000","last_change":"2024-08-17T17:39:04.652001+0000","last_active":"2024-08-17T17:40:42.934379+0000","last_peered":"2024-08-17T17:40:42.934379+0000","last_clean":"2024-08-17T17:40:42.934379+0000","last_became_active":"2024-08-17T17:39:04.651655+0000","last_became_peered":"2024-08-17T17:39:04.651655+0000","last_unstale":"2024-08-17T17:40:42.934379+0000","last_undegraded":"2024-08-17T17:40:42.934379+0000","last_fullsized":"2024-08-17T17:40:42.934379+0000","mapping_epoch":329,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:22.030514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:22.030514+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:48:20.005932+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":236,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.929797+0000","last_change":"2024-08-17T17:33:44.146014+0000","last_active":"2024-08-17T17:40:42.929797+0000","last_peered":"2024-08-17T17:40:42.929797+0000","last_clean":"2024-08-17T17:40:42.929797+0000","last_became_active":"2024-08-17T17:33:44.145668+0000","last_became_peered":"2024-08-17T17:33:44.145668+0000","last_unstale":"2024-08-17T17:40:42.929797+0000","last_undegraded":"2024-08-17T17:40:42.929797+0000","last_fullsized":"2024-08-17T17:40:42.929797+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:53:41.679237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":501,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.917680+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:40:42.917680+0000","last_peered":"2024-08-17T17:40:42.917680+0000","last_clean":"2024-08-17T17:40:42.917680+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:40:42.917680+0000","last_undegraded":"2024-08-17T17:40:42.917680+0000","last_fullsized":"2024-08-17T17:40:42.917680+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":528,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.923747+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:40:42.923747+0000","last_peered":"2024-08-17T17:40:42.923747+0000","last_clean":"2024-08-17T17:40:42.923747+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:40:42.923747+0000","last_undegraded":"2024-08-17T17:40:42.923747+0000","last_fullsized":"2024-08-17T17:40:42.923747+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":559,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.917286+0000","last_change":"2024-08-17T17:33:20.790687+0000","last_active":"2024-08-17T17:40:42.917286+0000","last_peered":"2024-08-17T17:40:42.917286+0000","last_clean":"2024-08-17T17:40:42.917286+0000","last_became_active":"2024-08-17T17:33:20.790399+0000","last_became_peered":"2024-08-17T17:33:20.790399+0000","last_unstale":"2024-08-17T17:40:42.917286+0000","last_undegraded":"2024-08-17T17:40:42.917286+0000","last_fullsized":"2024-08-17T17:40:42.917286+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:38:06.849989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":311,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-08-17T17:40:26.856119+0000","last_change":"2024-08-17T17:33:20.835760+0000","last_active":"2024-08-17T17:40:26.856119+0000","last_peered":"2024-08-17T17:40:26.856119+0000","last_clean":"2024-08-17T17:40:26.856119+0000","last_became_active":"2024-08-17T17:33:20.835410+0000","last_became_peered":"2024-08-17T17:33:20.835410+0000","last_unstale":"2024-08-17T17:40:26.856119+0000","last_undegraded":"2024-08-17T17:40:26.856119+0000","last_fullsized":"2024-08-17T17:40:26.856119+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:45.448022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":576,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.917537+0000","last_change":"2024-08-17T17:33:20.771279+0000","last_active":"2024-08-17T17:40:42.917537+0000","last_peered":"2024-08-17T17:40:42.917537+0000","last_clean":"2024-08-17T17:40:42.917537+0000","last_became_active":"2024-08-17T17:33:20.770728+0000","last_became_peered":"2024-08-17T17:33:20.770728+0000","last_unstale":"2024-08-17T17:40:42.917537+0000","last_undegraded":"2024-08-17T17:40:42.917537+0000","last_fullsized":"2024-08-17T17:40:42.917537+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:52:31.540397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":346,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.923702+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:40:42.923702+0000","last_peered":"2024-08-17T17:40:42.923702+0000","last_clean":"2024-08-17T17:40:42.923702+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:40:42.923702+0000","last_undegraded":"2024-08-17T17:40:42.923702+0000","last_fullsized":"2024-08-17T17:40:42.923702+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":490,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.921709+0000","last_change":"2024-08-17T17:39:01.567704+0000","last_active":"2024-08-17T17:40:42.921709+0000","last_peered":"2024-08-17T17:40:42.921709+0000","last_clean":"2024-08-17T17:40:42.921709+0000","last_became_active":"2024-08-17T17:39:01.567339+0000","last_became_peered":"2024-08-17T17:39:01.567339+0000","last_unstale":"2024-08-17T17:40:42.921709+0000","last_undegraded":"2024-08-17T17:40:42.921709+0000","last_fullsized":"2024-08-17T17:40:42.921709+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:52:21.825174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":20,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-08-17T17:40:26.855915+0000","last_change":"2024-08-17T17:39:01.564893+0000","last_active":"2024-08-17T17:40:26.855915+0000","last_peered":"2024-08-17T17:40:26.855915+0000","last_clean":"2024-08-17T17:40:26.855915+0000","last_became_active":"2024-08-17T17:39:01.564553+0000","last_became_peered":"2024-08-17T17:39:01.564553+0000","last_unstale":"2024-08-17T17:40:26.855915+0000","last_undegraded":"2024-08-17T17:40:26.855915+0000","last_fullsized":"2024-08-17T17:40:26.855915+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:51:34.874281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":322,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.925386+0000","last_change":"2024-08-17T17:39:01.562674+0000","last_active":"2024-08-17T17:40:42.925386+0000","last_peered":"2024-08-17T17:40:42.925386+0000","last_clean":"2024-08-17T17:40:42.925386+0000","last_became_active":"2024-08-17T17:39:01.559291+0000","last_became_peered":"2024-08-17T17:39:01.559291+0000","last_unstale":"2024-08-17T17:40:42.925386+0000","last_undegraded":"2024-08-17T17:40:42.925386+0000","last_fullsized":"2024-08-17T17:40:42.925386+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:51:26.461542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":464,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.927255+0000","last_change":"2024-08-17T17:33:20.773207+0000","last_active":"2024-08-17T17:40:42.927255+0000","last_peered":"2024-08-17T17:40:42.927255+0000","last_clean":"2024-08-17T17:40:42.927255+0000","last_became_active":"2024-08-17T17:33:20.772457+0000","last_became_peered":"2024-08-17T17:33:20.772457+0000","last_unstale":"2024-08-17T17:40:42.927255+0000","last_undegraded":"2024-08-17T17:40:42.927255+0000","last_fullsized":"2024-08-17T17:40:42.927255+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:57:09.610084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":362,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.925308+0000","last_change":"2024-08-17T17:39:01.558304+0000","last_active":"2024-08-17T17:40:42.925308+0000","last_peered":"2024-08-17T17:40:42.925308+0000","last_clean":"2024-08-17T17:40:42.925308+0000","last_became_active":"2024-08-17T17:39:01.557715+0000","last_became_peered":"2024-08-17T17:39:01.557715+0000","last_unstale":"2024-08-17T17:40:42.925308+0000","last_undegraded":"2024-08-17T17:40:42.925308+0000","last_fullsized":"2024-08-17T17:40:42.925308+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:37:58.224844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":518,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.920201+0000","last_change":"2024-08-17T17:39:01.569074+0000","last_active":"2024-08-17T17:40:42.920201+0000","last_peered":"2024-08-17T17:40:42.920201+0000","last_clean":"2024-08-17T17:40:42.920201+0000","last_became_active":"2024-08-17T17:39:01.568641+0000","last_became_peered":"2024-08-17T17:39:01.568641+0000","last_unstale":"2024-08-17T17:40:42.920201+0000","last_undegraded":"2024-08-17T17:40:42.920201+0000","last_fullsized":"2024-08-17T17:40:42.920201+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:33:02.727543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":24,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-08-17T17:40:26.855817+0000","last_change":"2024-08-17T17:39:01.575978+0000","last_active":"2024-08-17T17:40:26.855817+0000","last_peered":"2024-08-17T17:40:26.855817+0000","last_clean":"2024-08-17T17:40:26.855817+0000","last_became_active":"2024-08-17T17:39:01.564487+0000","last_became_peered":"2024-08-17T17:39:01.564487+0000","last_unstale":"2024-08-17T17:40:26.855817+0000","last_undegraded":"2024-08-17T17:40:26.855817+0000","last_fullsized":"2024-08-17T17:40:26.855817+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:19:17.427639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":493,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.919855+0000","last_change":"2024-08-17T17:39:01.565849+0000","last_active":"2024-08-17T17:40:42.919855+0000","last_peered":"2024-08-17T17:40:42.919855+0000","last_clean":"2024-08-17T17:40:42.919855+0000","last_became_active":"2024-08-17T17:39:01.565059+0000","last_became_peered":"2024-08-17T17:39:01.565059+0000","last_unstale":"2024-08-17T17:40:42.919855+0000","last_undegraded":"2024-08-17T17:40:42.919855+0000","last_fullsized":"2024-08-17T17:40:42.919855+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:52:37.878402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":115,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-08-17T17:40:26.856132+0000","last_change":"2024-08-17T17:39:01.578208+0000","last_active":"2024-08-17T17:40:26.856132+0000","last_peered":"2024-08-17T17:40:26.856132+0000","last_clean":"2024-08-17T17:40:26.856132+0000","last_became_active":"2024-08-17T17:39:01.566765+0000","last_became_peered":"2024-08-17T17:39:01.566765+0000","last_unstale":"2024-08-17T17:40:26.856132+0000","last_undegraded":"2024-08-17T17:40:26.856132+0000","last_fullsized":"2024-08-17T17:40:26.856132+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:05:41.620419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":554,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.917727+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:40:42.917727+0000","last_peered":"2024-08-17T17:40:42.917727+0000","last_clean":"2024-08-17T17:40:42.917727+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:40:42.917727+0000","last_undegraded":"2024-08-17T17:40:42.917727+0000","last_fullsized":"2024-08-17T17:40:42.917727+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":555,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.918151+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:40:42.918151+0000","last_peered":"2024-08-17T17:40:42.918151+0000","last_clean":"2024-08-17T17:40:42.918151+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:40:42.918151+0000","last_undegraded":"2024-08-17T17:40:42.918151+0000","last_fullsized":"2024-08-17T17:40:42.918151+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":93,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-08-17T17:40:26.855512+0000","last_change":"2024-08-17T17:39:01.569403+0000","last_active":"2024-08-17T17:40:26.855512+0000","last_peered":"2024-08-17T17:40:26.855512+0000","last_clean":"2024-08-17T17:40:26.855512+0000","last_became_active":"2024-08-17T17:39:01.568071+0000","last_became_peered":"2024-08-17T17:39:01.568071+0000","last_unstale":"2024-08-17T17:40:26.855512+0000","last_undegraded":"2024-08-17T17:40:26.855512+0000","last_fullsized":"2024-08-17T17:40:26.855512+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:05:58.801566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":365,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-08-17T17:40:42.923693+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:40:42.923693+0000","last_peered":"2024-08-17T17:40:42.923693+0000","last_clean":"2024-08-17T17:40:42.923693+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:40:42.923693+0000","last_undegraded":"2024-08-17T17:40:42.923693+0000","last_fullsized":"2024-08-17T17:40:42.923693+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":3056992280,"num_objects":46647,"num_object_clones":0,"num_object_copies":139941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14932,"num_objects_misplaced":2125,"num_objects_unfound":0,"num_objects_dirty":46647,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":185119,"num_write_kb":7416449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2455,"num_bytes_recovered":160890880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10067652608,"data_stored":10067640392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93779,"ondisk_log_size":93779,"up":48,"acting":45,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":11110,"num_keys_recovered":30,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":21480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19280,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5978,"num_read_kb":12991,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3129,"ondisk_log_size":3129,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234409,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2616328,"kb_used_data":1726592,"kb_used_omap":1,"kb_used_meta":889726,"kb_avail":91128824,"statfs":{"total":95995035648,"available":93315915776,"internally_reserved":0,"allocated":1768030208,"data_stored":1766077120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":911079768},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496044,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1475260,"kb_used_data":580148,"kb_used_omap":1,"kb_used_meta":895102,"kb_avail":92269892,"statfs":{"total":95995035648,"available":94484369408,"internally_reserved":0,"allocated":594071552,"data_stored":592130436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":916584792},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757681,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2608316,"kb_used_data":1721844,"kb_used_omap":1,"kb_used_meta":886462,"kb_avail":91136836,"statfs":{"total":95995035648,"available":93324120064,"internally_reserved":0,"allocated":1763168256,"data_stored":1761223699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":907737432},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":128849019319,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435546,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2350836,"kb_used_data":1515500,"kb_used_omap":4,"kb_used_meta":835323,"kb_avail":91394316,"statfs":{"total":95995035648,"available":93587779584,"internally_reserved":0,"allocated":1551872000,"data_stored":1549935393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":855370762},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599073,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1225412,"kb_used_data":570108,"kb_used_omap":7,"kb_used_meta":655288,"kb_avail":92519740,"statfs":{"total":95995035648,"available":94740213760,"internally_reserved":0,"allocated":583790592,"data_stored":581842859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":671015074},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":268,"seq":1151051235420,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2062072,"kb_used_data":1674736,"kb_used_omap":10,"kb_used_meta":387317,"kb_avail":91683080,"statfs":{"total":95995035648,"available":93883473920,"internally_reserved":0,"allocated":1714929664,"data_stored":1712988408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10902,"internal_metadata":396612970},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,5,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926126,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1587256,"kb_used_data":904048,"kb_used_omap":7,"kb_used_meta":683128,"kb_avail":92157896,"statfs":{"total":95995035648,"available":94369685504,"internally_reserved":0,"allocated":925745152,"data_stored":923804505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":699523393},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7550,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":915210240,"data_stored":915210240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1706627072,"data_stored":1706623000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":571998208,"data_stored":571998208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1541799936,"data_stored":1541799936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1278738432,"data_stored":1278738432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1738412032,"data_stored":1738407960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":575930368,"data_stored":575930368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1738936320,"data_stored":1738932248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:41:02.825 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-08-17T17:41:02.825 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd pool set .mgr pg_num 31 2024-08-17T17:41:02.870 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:02.923 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:02.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:02.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:03.025 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:03.068 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:03.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:03.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:03.169 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:03.193 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:03.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:03.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:03.295 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:03.334 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:03.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:03.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:03.436 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:03.491 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:03.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:03.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:03.593 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:03.618 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:03.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:03.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:03.703 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:03 smithi003 ceph-mon[103344]: pgmap v1102: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 745 op/s; 15046/129390 objects degraded (11.628%); 1895/129390 objects misplaced (1.465%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T17:41:03.704 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:03 smithi003 ceph-mon[100615]: pgmap v1102: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 745 op/s; 15046/129390 objects degraded (11.628%); 1895/129390 objects misplaced (1.465%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T17:41:03.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:03 smithi053 ceph-mon[89963]: pgmap v1102: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 745 op/s; 15046/129390 objects degraded (11.628%); 1895/129390 objects misplaced (1.465%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T17:41:03.720 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:03.744 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:03.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:03.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:03.846 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:03.908 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:03.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:03.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:04.010 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:04.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:04.079 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:04.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:04.180 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:04.206 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:04.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:04.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:04.308 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:04.331 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:04.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:04.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:04.369 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:41:04.433 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:04.460 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:04.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:04.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:04.562 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:04.587 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:04.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:04.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:04.689 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:04.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:04.728 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:04.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:04.829 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:04.852 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:04.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:04.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:04.954 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:04.987 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:41:04.987 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-08-17T17:41:04.987 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-08-17T17:41:04.987 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:41:04.988 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:04.988 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:56781 8.3 8.4bf49503 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e268)", 2024-08-17T17:41:04.988 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:19.807785+0000", 2024-08-17T17:41:04.988 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 465.17582810099998, 2024-08-17T17:41:04.988 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96137050199999996, 2024-08-17T17:41:04.988 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:04.988 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:04.988 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:41:04.988 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:41:04.988 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:41:04.988 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:41:04.988 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 56781 2024-08-17T17:41:04.988 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:04.988 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:04.989 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:04.989 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:04.989 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807785+0000", 2024-08-17T17:41:04.989 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:04.989 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:04.989 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:04.989 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:04.989 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807785+0000", 2024-08-17T17:41:04.989 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:04.989 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:04.989 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:04.989 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:04.990 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807786+0000", 2024-08-17T17:41:04.990 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4050000000000001e-06 2024-08-17T17:41:04.990 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:04.990 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:04.990 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:04.990 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807854+0000", 2024-08-17T17:41:04.990 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7267000000000005e-05 2024-08-17T17:41:04.990 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:04.990 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:04.990 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:04.990 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807865+0000", 2024-08-17T17:41:04.990 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1606999999999999e-05 2024-08-17T17:41:04.990 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:04.991 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:04.991 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:04.991 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807874+0000", 2024-08-17T17:41:04.991 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4829999999999999e-06 2024-08-17T17:41:04.991 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:04.991 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:04.991 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:04.991 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807937+0000", 2024-08-17T17:41:04.991 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3040999999999995e-05 2024-08-17T17:41:04.991 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:04.991 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:04.991 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:41:04.991 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807947+0000", 2024-08-17T17:41:04.991 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8379999999999997e-06 2024-08-17T17:41:04.992 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:04.992 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:04.992 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:04.992 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.769146+0000", 2024-08-17T17:41:04.992 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96119933899999999 2024-08-17T17:41:04.992 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:04.992 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:04.992 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:04.992 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.769155+0000", 2024-08-17T17:41:04.992 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5219999999999996e-06 2024-08-17T17:41:04.992 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:04.992 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:04.992 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:04.993 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:04.993 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:04.993 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:56793 8.3 8.83fbcd3 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e268)", 2024-08-17T17:41:04.993 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:19.807896+0000", 2024-08-17T17:41:04.993 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 465.17571669099999, 2024-08-17T17:41:04.993 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96128810799999997, 2024-08-17T17:41:04.993 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:04.993 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:04.993 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:41:04.993 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:41:04.993 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:41:04.993 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:41:04.993 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 56793 2024-08-17T17:41:04.993 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:04.994 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:04.994 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:04.994 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:04.994 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807896+0000", 2024-08-17T17:41:04.994 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:04.994 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:04.994 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:04.994 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:04.994 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807896+0000", 2024-08-17T17:41:04.994 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:04.994 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:04.994 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:04.994 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:04.995 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807897+0000", 2024-08-17T17:41:04.995 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8500000000000002e-07 2024-08-17T17:41:04.995 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:04.995 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:04.995 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:04.995 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807942+0000", 2024-08-17T17:41:04.995 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4977e-05 2024-08-17T17:41:04.995 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:04.995 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:04.995 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:04.995 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807950+0000", 2024-08-17T17:41:04.995 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9559999999999997e-06 2024-08-17T17:41:04.995 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:04.995 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:04.996 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:04.996 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807955+0000", 2024-08-17T17:41:04.996 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5509999999999999e-06 2024-08-17T17:41:04.996 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:04.996 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:04.996 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:04.996 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807997+0000", 2024-08-17T17:41:04.996 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2179999999999999e-05 2024-08-17T17:41:04.996 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:04.996 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:04.996 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:41:04.996 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808002+0000", 2024-08-17T17:41:04.996 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6639999999999997e-06 2024-08-17T17:41:04.996 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:04.997 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:04.997 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:04.997 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.769177+0000", 2024-08-17T17:41:04.997 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96117454300000005 2024-08-17T17:41:04.997 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:04.997 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:04.997 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:04.997 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.769184+0000", 2024-08-17T17:41:04.997 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7519999999999996e-06 2024-08-17T17:41:04.997 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:04.997 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:04.997 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:04.997 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:04.998 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:04.998 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:56797 8.a 8.229444aa (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e268)", 2024-08-17T17:41:04.998 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:19.807968+0000", 2024-08-17T17:41:04.998 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 465.17564506799999, 2024-08-17T17:41:04.998 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.95976723200000003, 2024-08-17T17:41:04.998 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:04.998 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:04.998 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:41:04.998 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:41:04.998 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:41:04.998 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:41:04.998 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 56797 2024-08-17T17:41:04.999 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:04.999 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:04.999 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:04.999 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:04.999 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807968+0000", 2024-08-17T17:41:04.999 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:04.999 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:04.999 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:04.999 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:04.999 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807968+0000", 2024-08-17T17:41:04.999 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:04.999 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:04.999 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:04.999 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:04.999 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.807968+0000", 2024-08-17T17:41:05.000 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0900000000000002e-07 2024-08-17T17:41:05.000 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.000 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.000 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:05.000 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808012+0000", 2024-08-17T17:41:05.000 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3455000000000003e-05 2024-08-17T17:41:05.000 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.000 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.000 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:05.000 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808020+0000", 2024-08-17T17:41:05.000 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4409999999999993e-06 2024-08-17T17:41:05.000 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.000 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.000 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:05.001 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808025+0000", 2024-08-17T17:41:05.001 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9629999999999997e-06 2024-08-17T17:41:05.001 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.001 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.001 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:05.001 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808063+0000", 2024-08-17T17:41:05.001 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7626000000000002e-05 2024-08-17T17:41:05.001 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.001 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.001 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:41:05.001 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808067+0000", 2024-08-17T17:41:05.001 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0960000000000003e-06 2024-08-17T17:41:05.001 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.001 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.002 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:05.002 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.767725+0000", 2024-08-17T17:41:05.002 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.95965830900000004 2024-08-17T17:41:05.002 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.002 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.003 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:05.003 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.767735+0000", 2024-08-17T17:41:05.003 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9329999999999996e-06 2024-08-17T17:41:05.003 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.003 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:05.003 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.003 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.003 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.003 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:56804 8.3 8.fd779f13 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-08-17T17:41:05.003 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:19.808037+0000", 2024-08-17T17:41:05.003 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 465.17557571100002, 2024-08-17T17:41:05.003 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96117662699999995, 2024-08-17T17:41:05.003 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:05.004 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:05.004 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:41:05.004 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:41:05.004 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:41:05.004 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:41:05.004 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 56804 2024-08-17T17:41:05.004 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.004 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:05.004 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.004 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:05.004 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808037+0000", 2024-08-17T17:41:05.004 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.004 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.005 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.005 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:05.005 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808037+0000", 2024-08-17T17:41:05.005 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.005 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.005 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.005 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:05.005 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808038+0000", 2024-08-17T17:41:05.005 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2300000000000002e-07 2024-08-17T17:41:05.005 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.005 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.005 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:05.005 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808086+0000", 2024-08-17T17:41:05.006 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8529000000000002e-05 2024-08-17T17:41:05.006 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.006 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.006 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:05.006 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808096+0000", 2024-08-17T17:41:05.006 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3999999999999998e-06 2024-08-17T17:41:05.006 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.006 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.006 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:05.006 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808101+0000", 2024-08-17T17:41:05.006 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.575e-06 2024-08-17T17:41:05.006 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.006 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.006 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:05.007 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808148+0000", 2024-08-17T17:41:05.007 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6981000000000003e-05 2024-08-17T17:41:05.007 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.007 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.007 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:41:05.007 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808155+0000", 2024-08-17T17:41:05.007 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8340000000000001e-06 2024-08-17T17:41:05.007 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.007 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.007 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:05.007 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.769207+0000", 2024-08-17T17:41:05.007 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96105154500000001 2024-08-17T17:41:05.007 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.007 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.008 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:05.008 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.769214+0000", 2024-08-17T17:41:05.008 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.34e-06 2024-08-17T17:41:05.008 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.008 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:05.008 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.008 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.008 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.008 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:56808 8.3 8.634b5e43 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-08-17T17:41:05.008 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:19.808791+0000", 2024-08-17T17:41:05.008 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 465.17482238999997, 2024-08-17T17:41:05.008 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96046335800000004, 2024-08-17T17:41:05.008 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:05.008 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:05.008 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:41:05.009 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:41:05.009 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:41:05.009 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:41:05.009 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 56808 2024-08-17T17:41:05.009 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.009 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:05.009 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.009 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:05.009 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808791+0000", 2024-08-17T17:41:05.009 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.009 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.009 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.009 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:05.010 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808791+0000", 2024-08-17T17:41:05.010 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.010 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.010 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.010 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:05.010 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808791+0000", 2024-08-17T17:41:05.010 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7199999999999998e-07 2024-08-17T17:41:05.010 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.010 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.010 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:05.010 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808851+0000", 2024-08-17T17:41:05.010 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0016999999999998e-05 2024-08-17T17:41:05.010 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.011 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.011 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:05.011 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808862+0000", 2024-08-17T17:41:05.011 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0604e-05 2024-08-17T17:41:05.011 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.011 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.011 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:05.011 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808867+0000", 2024-08-17T17:41:05.011 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1959999999999999e-06 2024-08-17T17:41:05.011 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.011 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.011 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:05.011 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808909+0000", 2024-08-17T17:41:05.011 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1519e-05 2024-08-17T17:41:05.011 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.011 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.012 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:41:05.012 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.808914+0000", 2024-08-17T17:41:05.012 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6930000000000004e-06 2024-08-17T17:41:05.012 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.012 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.012 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:05.012 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.769247+0000", 2024-08-17T17:41:05.012 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.960332356 2024-08-17T17:41:05.012 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.012 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.012 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:05.012 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.769254+0000", 2024-08-17T17:41:05.012 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2010000000000002e-06 2024-08-17T17:41:05.012 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.012 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:05.013 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.013 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.013 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.013 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:56812 8.7 8.b683b077 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-08-17T17:41:05.013 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:19.812057+0000", 2024-08-17T17:41:05.013 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 465.17155636400003, 2024-08-17T17:41:05.013 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.950741383, 2024-08-17T17:41:05.013 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:05.013 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:05.013 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:41:05.013 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:41:05.013 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:41:05.013 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:41:05.013 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 56812 2024-08-17T17:41:05.013 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.013 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:05.014 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.014 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:05.014 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.812057+0000", 2024-08-17T17:41:05.014 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.014 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.014 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.014 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:05.014 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.812057+0000", 2024-08-17T17:41:05.014 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.014 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.014 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.014 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:05.014 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.812058+0000", 2024-08-17T17:41:05.014 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0899999999999999e-06 2024-08-17T17:41:05.015 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.015 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.015 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:05.015 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.812100+0000", 2024-08-17T17:41:05.015 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2030000000000002e-05 2024-08-17T17:41:05.015 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.015 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.015 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:05.015 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.812112+0000", 2024-08-17T17:41:05.015 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.224e-05 2024-08-17T17:41:05.015 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.015 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.015 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:05.015 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.812120+0000", 2024-08-17T17:41:05.016 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0039999999999999e-06 2024-08-17T17:41:05.016 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.016 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.016 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:05.016 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.812179+0000", 2024-08-17T17:41:05.016 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9104000000000002e-05 2024-08-17T17:41:05.016 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.016 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.016 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:41:05.016 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:19.812184+0000", 2024-08-17T17:41:05.016 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0799999999999996e-06 2024-08-17T17:41:05.016 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.016 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.016 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:05.017 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.762788+0000", 2024-08-17T17:41:05.017 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.95060427300000006 2024-08-17T17:41:05.017 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.017 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.017 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:05.017 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:20.762798+0000", 2024-08-17T17:41:05.017 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5619999999999993e-06 2024-08-17T17:41:05.017 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.017 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:05.017 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.017 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.017 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.017 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36354.0:136272 9.e 9:75d7b4d9:::benchmark_data_smithi003_261051_object20387:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-08-17T17:41:05.017 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:39:48.640319+0000", 2024-08-17T17:41:05.018 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 76.343293863, 2024-08-17T17:41:05.018 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.27006159299999999, 2024-08-17T17:41:05.018 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:05.018 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:05.018 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:41:05.018 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:41:05.018 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36354", 2024-08-17T17:41:05.018 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1330158990", 2024-08-17T17:41:05.018 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 136272 2024-08-17T17:41:05.018 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.018 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:05.018 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.018 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:05.018 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.640319+0000", 2024-08-17T17:41:05.019 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.019 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.019 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.019 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:05.019 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.640319+0000", 2024-08-17T17:41:05.019 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.019 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.019 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.019 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:05.019 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.640321+0000", 2024-08-17T17:41:05.019 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6789999999999999e-06 2024-08-17T17:41:05.019 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.019 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.019 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:05.020 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.640327+0000", 2024-08-17T17:41:05.020 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1249999999999998e-06 2024-08-17T17:41:05.020 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.020 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.020 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:05.020 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.640343+0000", 2024-08-17T17:41:05.020 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6288000000000001e-05 2024-08-17T17:41:05.020 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.020 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.020 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:05.020 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.640352+0000", 2024-08-17T17:41:05.020 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3089999999999996e-06 2024-08-17T17:41:05.020 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.020 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.021 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:05.021 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.657779+0000", 2024-08-17T17:41:05.021 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017427345 2024-08-17T17:41:05.021 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.021 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.021 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for rw locks", 2024-08-17T17:41:05.021 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.657885+0000", 2024-08-17T17:41:05.021 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000106134 2024-08-17T17:41:05.021 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.021 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.021 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:05.021 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.716241+0000", 2024-08-17T17:41:05.021 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.058355957999999999 2024-08-17T17:41:05.021 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.022 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.022 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:05.022 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.716355+0000", 2024-08-17T17:41:05.022 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011435200000000001 2024-08-17T17:41:05.022 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.022 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.022 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,7", 2024-08-17T17:41:05.022 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.716513+0000", 2024-08-17T17:41:05.022 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000157448 2024-08-17T17:41:05.022 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.022 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.022 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:05.022 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.721485+0000", 2024-08-17T17:41:05.023 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049726839999999998 2024-08-17T17:41:05.023 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.023 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.023 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:41:05.023 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.721508+0000", 2024-08-17T17:41:05.023 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2058000000000001e-05 2024-08-17T17:41:05.023 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.023 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.023 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:05.023 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.908091+0000", 2024-08-17T17:41:05.023 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18658300799999999 2024-08-17T17:41:05.023 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.023 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.023 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:41:05.024 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.908144+0000", 2024-08-17T17:41:05.024 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3699000000000002e-05 2024-08-17T17:41:05.024 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.024 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.024 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:05.024 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.910381+0000", 2024-08-17T17:41:05.024 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0022365060000000001 2024-08-17T17:41:05.024 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.024 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:05.024 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.024 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.024 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.024 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36354.0:136300 9.4 9:26d8721f:::benchmark_data_smithi003_261051_object20415:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-08-17T17:41:05.024 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:39:48.680294+0000", 2024-08-17T17:41:05.025 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 76.303318953000002, 2024-08-17T17:41:05.025 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.23265981699999999, 2024-08-17T17:41:05.025 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:05.025 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:05.025 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:41:05.025 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:41:05.025 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36354", 2024-08-17T17:41:05.025 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1330158990", 2024-08-17T17:41:05.025 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 136300 2024-08-17T17:41:05.025 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.025 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:05.025 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.025 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:05.025 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.680294+0000", 2024-08-17T17:41:05.026 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.026 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.026 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.026 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:05.026 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.680294+0000", 2024-08-17T17:41:05.026 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.026 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.026 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.026 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:05.026 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.680295+0000", 2024-08-17T17:41:05.026 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2869999999999999e-06 2024-08-17T17:41:05.026 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.026 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.026 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:05.027 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.680301+0000", 2024-08-17T17:41:05.027 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1950000000000002e-06 2024-08-17T17:41:05.027 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.027 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.027 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:05.027 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.680311+0000", 2024-08-17T17:41:05.027 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0784999999999999e-05 2024-08-17T17:41:05.027 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.027 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.027 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:05.027 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.680318+0000", 2024-08-17T17:41:05.027 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9739999999999996e-06 2024-08-17T17:41:05.027 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.027 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.028 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:05.028 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.680406+0000", 2024-08-17T17:41:05.028 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8036999999999998e-05 2024-08-17T17:41:05.028 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.028 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.028 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:05.028 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.680572+0000", 2024-08-17T17:41:05.028 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000165506 2024-08-17T17:41:05.028 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.028 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.028 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,3", 2024-08-17T17:41:05.028 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.680804+0000", 2024-08-17T17:41:05.028 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000231735 2024-08-17T17:41:05.028 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.029 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.029 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:41:05.029 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.687715+0000", 2024-08-17T17:41:05.029 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0069117639999999999 2024-08-17T17:41:05.029 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.029 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.029 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:05.029 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.696429+0000", 2024-08-17T17:41:05.029 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0087134729999999994 2024-08-17T17:41:05.029 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.029 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.029 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:05.029 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.907695+0000", 2024-08-17T17:41:05.029 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.21126613899999999 2024-08-17T17:41:05.030 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.030 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.030 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:41:05.030 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.907845+0000", 2024-08-17T17:41:05.030 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015011000000000001 2024-08-17T17:41:05.030 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.030 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.030 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:05.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:05.031 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:05.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:05.032 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.912954+0000", 2024-08-17T17:41:05.032 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0051088119999999999 2024-08-17T17:41:05.032 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.032 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:05.032 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.032 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.033 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.033 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36354.0:136311 9.4 9:22840a96:::benchmark_data_smithi003_261051_object20426:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-08-17T17:41:05.033 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:39:48.690438+0000", 2024-08-17T17:41:05.033 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 76.293174659000002, 2024-08-17T17:41:05.033 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.239921211, 2024-08-17T17:41:05.033 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:05.033 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:05.033 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:41:05.033 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:41:05.033 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36354", 2024-08-17T17:41:05.033 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1330158990", 2024-08-17T17:41:05.033 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 136311 2024-08-17T17:41:05.033 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.033 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:05.034 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.034 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:05.034 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.690438+0000", 2024-08-17T17:41:05.034 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.034 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.034 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.034 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:05.034 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.690438+0000", 2024-08-17T17:41:05.034 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.034 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.034 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.034 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:05.034 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.690440+0000", 2024-08-17T17:41:05.034 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4479999999999999e-06 2024-08-17T17:41:05.035 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.035 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.035 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:05.035 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.690446+0000", 2024-08-17T17:41:05.035 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9529999999999996e-06 2024-08-17T17:41:05.035 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.035 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.035 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:05.035 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.690456+0000", 2024-08-17T17:41:05.035 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0288e-05 2024-08-17T17:41:05.035 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.035 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.035 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:05.035 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.690463+0000", 2024-08-17T17:41:05.036 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0199999999999997e-06 2024-08-17T17:41:05.036 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.036 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.036 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:05.036 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.696466+0000", 2024-08-17T17:41:05.036 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0060025590000000002 2024-08-17T17:41:05.036 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.036 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.036 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:05.036 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.696610+0000", 2024-08-17T17:41:05.036 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014404000000000001 2024-08-17T17:41:05.036 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.036 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.037 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,3", 2024-08-17T17:41:05.037 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.696754+0000", 2024-08-17T17:41:05.037 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014437599999999999 2024-08-17T17:41:05.037 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.037 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.037 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:41:05.037 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.700011+0000", 2024-08-17T17:41:05.037 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003257069 2024-08-17T17:41:05.037 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.037 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.037 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:05.037 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.717109+0000", 2024-08-17T17:41:05.037 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017097788999999999 2024-08-17T17:41:05.037 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.038 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.038 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:05.038 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.925693+0000", 2024-08-17T17:41:05.038 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.20858369199999999 2024-08-17T17:41:05.038 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.038 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.038 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:41:05.038 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.925866+0000", 2024-08-17T17:41:05.038 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017299700000000001 2024-08-17T17:41:05.038 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.038 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.038 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:05.038 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.930360+0000", 2024-08-17T17:41:05.038 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0044939799999999998 2024-08-17T17:41:05.039 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.039 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:05.039 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.039 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.039 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.039 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36354.0:136326 9.4 9:223c4187:::benchmark_data_smithi003_261051_object20441:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-08-17T17:41:05.039 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:39:48.710034+0000", 2024-08-17T17:41:05.039 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 76.273579479000006, 2024-08-17T17:41:05.039 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.22490632699999999, 2024-08-17T17:41:05.039 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:05.039 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:05.039 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:41:05.039 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:41:05.039 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36354", 2024-08-17T17:41:05.040 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1330158990", 2024-08-17T17:41:05.040 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 136326 2024-08-17T17:41:05.040 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.040 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:05.040 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.040 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:05.040 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.710034+0000", 2024-08-17T17:41:05.040 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.040 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.040 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.040 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:05.040 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.710034+0000", 2024-08-17T17:41:05.040 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.040 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.040 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.041 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:05.041 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.710035+0000", 2024-08-17T17:41:05.041 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0389999999999999e-06 2024-08-17T17:41:05.041 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.041 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.041 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:05.041 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.710039+0000", 2024-08-17T17:41:05.041 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6299999999999997e-06 2024-08-17T17:41:05.041 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.041 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.041 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:05.041 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.710049+0000", 2024-08-17T17:41:05.041 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7100000000000002e-06 2024-08-17T17:41:05.041 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.041 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.042 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:05.042 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.710055+0000", 2024-08-17T17:41:05.042 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1929999999999998e-06 2024-08-17T17:41:05.042 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.042 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.042 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:05.042 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.710433+0000", 2024-08-17T17:41:05.042 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00037831600000000001 2024-08-17T17:41:05.042 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.042 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.042 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:05.042 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.710540+0000", 2024-08-17T17:41:05.042 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000106315 2024-08-17T17:41:05.042 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.043 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.043 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,3", 2024-08-17T17:41:05.043 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.710637+0000", 2024-08-17T17:41:05.043 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7258999999999998e-05 2024-08-17T17:41:05.043 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.043 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.043 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:41:05.043 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.717071+0000", 2024-08-17T17:41:05.043 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0064343610000000004 2024-08-17T17:41:05.043 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.043 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.043 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:05.043 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.721458+0000", 2024-08-17T17:41:05.043 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0043865060000000001 2024-08-17T17:41:05.043 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.044 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.044 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:05.044 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.930432+0000", 2024-08-17T17:41:05.044 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.20897447399999999 2024-08-17T17:41:05.044 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.044 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.044 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:41:05.044 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.930487+0000", 2024-08-17T17:41:05.044 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4740000000000001e-05 2024-08-17T17:41:05.044 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.044 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.044 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:05.044 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.934940+0000", 2024-08-17T17:41:05.044 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0044527840000000004 2024-08-17T17:41:05.044 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.045 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:05.045 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.045 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.045 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.045 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36354.0:136327 9.4 9:20672cfd:::benchmark_data_smithi003_261051_object20442:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-08-17T17:41:05.045 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:39:48.710073+0000", 2024-08-17T17:41:05.045 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 76.273539825, 2024-08-17T17:41:05.045 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.22927357500000001, 2024-08-17T17:41:05.045 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:05.045 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:05.045 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:41:05.045 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:41:05.045 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36354", 2024-08-17T17:41:05.045 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1330158990", 2024-08-17T17:41:05.045 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 136327 2024-08-17T17:41:05.046 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.046 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:05.046 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.046 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:05.046 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.710073+0000", 2024-08-17T17:41:05.046 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.046 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.046 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.046 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:05.046 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.710073+0000", 2024-08-17T17:41:05.046 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.046 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.046 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.046 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:05.046 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.710073+0000", 2024-08-17T17:41:05.047 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1e-07 2024-08-17T17:41:05.047 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.047 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.047 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:05.047 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.710075+0000", 2024-08-17T17:41:05.047 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.578e-06 2024-08-17T17:41:05.047 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.047 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.047 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:05.047 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.710079+0000", 2024-08-17T17:41:05.047 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0029999999999996e-06 2024-08-17T17:41:05.047 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.047 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.047 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:05.048 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.710082+0000", 2024-08-17T17:41:05.048 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.061e-06 2024-08-17T17:41:05.048 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.048 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.048 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:05.048 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.713650+0000", 2024-08-17T17:41:05.048 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003567976 2024-08-17T17:41:05.048 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.048 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.048 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:05.048 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.713801+0000", 2024-08-17T17:41:05.048 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015055900000000001 2024-08-17T17:41:05.048 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.049 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.049 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,3", 2024-08-17T17:41:05.049 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.714220+0000", 2024-08-17T17:41:05.049 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00041978799999999999 2024-08-17T17:41:05.049 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.049 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.049 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:41:05.049 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.717194+0000", 2024-08-17T17:41:05.049 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029736839999999999 2024-08-17T17:41:05.049 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.049 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.049 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:05.049 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.723679+0000", 2024-08-17T17:41:05.049 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0064852520000000004 2024-08-17T17:41:05.049 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.050 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.050 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:05.050 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.935002+0000", 2024-08-17T17:41:05.050 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.211322916 2024-08-17T17:41:05.050 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.050 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.050 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:41:05.050 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.935062+0000", 2024-08-17T17:41:05.050 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0096999999999997e-05 2024-08-17T17:41:05.050 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.050 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.050 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:05.050 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.939347+0000", 2024-08-17T17:41:05.050 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042843509999999996 2024-08-17T17:41:05.051 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.051 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:05.051 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.051 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.051 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.051 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36354.0:136330 9.e 9:7656f25a:::benchmark_data_smithi003_261051_object20445:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-08-17T17:41:05.051 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:39:48.713904+0000", 2024-08-17T17:41:05.051 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 76.269709378000002, 2024-08-17T17:41:05.051 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.198543054, 2024-08-17T17:41:05.051 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:05.051 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:05.051 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:41:05.051 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:41:05.051 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36354", 2024-08-17T17:41:05.052 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1330158990", 2024-08-17T17:41:05.052 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 136330 2024-08-17T17:41:05.052 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.052 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:05.052 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.052 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:05.052 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.713904+0000", 2024-08-17T17:41:05.052 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.052 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.052 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.052 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:05.052 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.713904+0000", 2024-08-17T17:41:05.052 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.052 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.053 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.053 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:05.053 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.713904+0000", 2024-08-17T17:41:05.053 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3300000000000002e-07 2024-08-17T17:41:05.053 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.053 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.053 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:05.053 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.713908+0000", 2024-08-17T17:41:05.053 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6160000000000002e-06 2024-08-17T17:41:05.053 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.053 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.053 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:05.053 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.713917+0000", 2024-08-17T17:41:05.053 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8810000000000002e-06 2024-08-17T17:41:05.053 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.054 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.054 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:05.054 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.713934+0000", 2024-08-17T17:41:05.054 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6946999999999999e-05 2024-08-17T17:41:05.054 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.054 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.054 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:05.054 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.718929+0000", 2024-08-17T17:41:05.054 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049957370000000001 2024-08-17T17:41:05.054 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.054 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.054 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:05.054 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.719019+0000", 2024-08-17T17:41:05.054 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9469999999999995e-05 2024-08-17T17:41:05.055 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.055 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.055 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,7", 2024-08-17T17:41:05.055 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.719110+0000", 2024-08-17T17:41:05.055 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1016999999999999e-05 2024-08-17T17:41:05.055 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.055 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.055 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:41:05.055 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.728923+0000", 2024-08-17T17:41:05.055 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0098129480000000002 2024-08-17T17:41:05.055 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.055 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.055 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:05.055 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.728965+0000", 2024-08-17T17:41:05.055 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2107999999999997e-05 2024-08-17T17:41:05.055 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.056 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.056 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:05.056 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.910460+0000", 2024-08-17T17:41:05.056 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18149542900000001 2024-08-17T17:41:05.056 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.056 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.056 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:41:05.056 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.910506+0000", 2024-08-17T17:41:05.056 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5562000000000002e-05 2024-08-17T17:41:05.056 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.056 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.056 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:05.056 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.912447+0000", 2024-08-17T17:41:05.056 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0019407059999999999 2024-08-17T17:41:05.057 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.057 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:05.057 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.057 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.057 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.057 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36354.0:138791 9.d 9:b1398701:::benchmark_data_smithi003_261051_object22906:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-08-17T17:41:05.057 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:39:52.113857+0000", 2024-08-17T17:41:05.057 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 72.869755827999995, 2024-08-17T17:41:05.057 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.20165855199999999, 2024-08-17T17:41:05.057 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:05.057 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:05.057 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:41:05.057 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:41:05.057 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36354", 2024-08-17T17:41:05.057 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1330158990", 2024-08-17T17:41:05.057 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 138791 2024-08-17T17:41:05.058 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.058 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:05.058 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.058 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:05.058 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.113857+0000", 2024-08-17T17:41:05.058 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.058 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.058 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.058 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:05.058 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.113857+0000", 2024-08-17T17:41:05.058 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.058 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.058 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.058 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:05.058 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.113858+0000", 2024-08-17T17:41:05.059 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3499999999999996e-07 2024-08-17T17:41:05.059 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.059 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.060 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:05.060 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.113861+0000", 2024-08-17T17:41:05.060 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.129e-06 2024-08-17T17:41:05.060 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.060 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.060 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:05.060 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.113872+0000", 2024-08-17T17:41:05.060 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.163e-05 2024-08-17T17:41:05.060 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.060 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.060 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:05.060 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.113891+0000", 2024-08-17T17:41:05.060 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.853e-05 2024-08-17T17:41:05.060 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.061 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.061 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:05.061 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.120396+0000", 2024-08-17T17:41:05.061 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0065048090000000003 2024-08-17T17:41:05.061 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.061 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.061 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:05.061 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.120513+0000", 2024-08-17T17:41:05.061 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000117483 2024-08-17T17:41:05.061 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.061 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.061 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-08-17T17:41:05.061 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.121591+0000", 2024-08-17T17:41:05.061 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010772690000000001 2024-08-17T17:41:05.061 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.062 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.062 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:41:05.062 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.125100+0000", 2024-08-17T17:41:05.062 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003509917 2024-08-17T17:41:05.062 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.062 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.062 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:05.062 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.125589+0000", 2024-08-17T17:41:05.062 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000488672 2024-08-17T17:41:05.062 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.062 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.062 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:05.062 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.315405+0000", 2024-08-17T17:41:05.063 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.189816241 2024-08-17T17:41:05.063 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.063 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.063 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:41:05.063 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.315467+0000", 2024-08-17T17:41:05.063 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1488999999999994e-05 2024-08-17T17:41:05.063 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.063 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.063 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:05.063 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.315516+0000", 2024-08-17T17:41:05.063 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8847999999999997e-05 2024-08-17T17:41:05.063 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.063 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:05.063 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.064 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.064 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.064 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36354.0:138801 9.d 9:bbbaf5a9:::benchmark_data_smithi003_261051_object22916:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-08-17T17:41:05.064 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:39:52.126934+0000", 2024-08-17T17:41:05.064 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 72.856679229999997, 2024-08-17T17:41:05.064 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.196312915, 2024-08-17T17:41:05.064 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:05.064 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:05.064 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:41:05.064 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:41:05.064 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36354", 2024-08-17T17:41:05.064 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1330158990", 2024-08-17T17:41:05.064 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 138801 2024-08-17T17:41:05.064 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.065 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:05.065 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.065 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:05.065 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.126934+0000", 2024-08-17T17:41:05.065 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.065 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.065 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.065 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:05.065 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.126934+0000", 2024-08-17T17:41:05.065 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.065 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.065 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.065 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:05.065 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.126934+0000", 2024-08-17T17:41:05.066 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3000000000000001e-07 2024-08-17T17:41:05.066 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.066 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.066 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:05.066 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.126938+0000", 2024-08-17T17:41:05.066 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3249999999999999e-06 2024-08-17T17:41:05.066 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.066 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.066 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:05.066 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.126945+0000", 2024-08-17T17:41:05.066 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7249999999999997e-06 2024-08-17T17:41:05.066 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.066 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.067 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:05.067 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.126951+0000", 2024-08-17T17:41:05.067 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7830000000000004e-06 2024-08-17T17:41:05.067 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.067 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.067 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:05.067 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.130552+0000", 2024-08-17T17:41:05.067 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003600795 2024-08-17T17:41:05.067 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.067 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.067 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:05.067 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.130671+0000", 2024-08-17T17:41:05.067 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000119417 2024-08-17T17:41:05.067 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.067 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.068 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-08-17T17:41:05.068 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.130791+0000", 2024-08-17T17:41:05.068 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000119709 2024-08-17T17:41:05.068 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.068 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.068 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:41:05.068 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.136466+0000", 2024-08-17T17:41:05.068 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0056751220000000003 2024-08-17T17:41:05.068 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.068 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.068 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:05.068 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.312347+0000", 2024-08-17T17:41:05.068 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.175880652 2024-08-17T17:41:05.068 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.069 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.069 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:05.069 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.323165+0000", 2024-08-17T17:41:05.069 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010818458 2024-08-17T17:41:05.069 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.069 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.069 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:41:05.069 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.323215+0000", 2024-08-17T17:41:05.069 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9910999999999997e-05 2024-08-17T17:41:05.069 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.069 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.069 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:05.069 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.323247+0000", 2024-08-17T17:41:05.069 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1488000000000001e-05 2024-08-17T17:41:05.070 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.070 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:05.070 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.070 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.070 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.070 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36354.0:185821 9.e 9:7ad4282c:::benchmark_data_smithi003_261051_object69936:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-17T17:41:05.070 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:40:55.947792+0000", 2024-08-17T17:41:05.070 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 9.0358207529999994, 2024-08-17T17:41:05.070 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.27158397499999998, 2024-08-17T17:41:05.070 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:05.070 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:05.070 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:41:05.070 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:41:05.070 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36354", 2024-08-17T17:41:05.071 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1330158990", 2024-08-17T17:41:05.071 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 185821 2024-08-17T17:41:05.071 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.071 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:05.071 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.071 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:05.071 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:55.947792+0000", 2024-08-17T17:41:05.071 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.071 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.071 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.071 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:05.071 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:55.947792+0000", 2024-08-17T17:41:05.071 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.071 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.072 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.072 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:05.072 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:55.947793+0000", 2024-08-17T17:41:05.072 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.33e-07 2024-08-17T17:41:05.072 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.072 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.072 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:05.072 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:55.947797+0000", 2024-08-17T17:41:05.072 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.296e-06 2024-08-17T17:41:05.072 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.072 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.072 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:05.072 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:55.947808+0000", 2024-08-17T17:41:05.072 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0672e-05 2024-08-17T17:41:05.072 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.073 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.073 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:05.073 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:55.947825+0000", 2024-08-17T17:41:05.073 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7345000000000001e-05 2024-08-17T17:41:05.073 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.073 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.073 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:05.073 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:55.963134+0000", 2024-08-17T17:41:05.073 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.015308175 2024-08-17T17:41:05.073 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.073 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.073 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for rw locks", 2024-08-17T17:41:05.073 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:55.963254+0000", 2024-08-17T17:41:05.073 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000120568 2024-08-17T17:41:05.074 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.074 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.074 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:05.074 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.043292+0000", 2024-08-17T17:41:05.074 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.080037547000000001 2024-08-17T17:41:05.074 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.074 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.074 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:05.074 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.043423+0000", 2024-08-17T17:41:05.074 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013141499999999999 2024-08-17T17:41:05.074 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.074 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.074 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,7", 2024-08-17T17:41:05.074 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.043593+0000", 2024-08-17T17:41:05.075 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000169755 2024-08-17T17:41:05.075 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.075 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.075 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:41:05.075 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.060619+0000", 2024-08-17T17:41:05.075 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017026039999999999 2024-08-17T17:41:05.075 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.075 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.075 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:05.075 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.063932+0000", 2024-08-17T17:41:05.075 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003313164 2024-08-17T17:41:05.075 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.075 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.075 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:05.075 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.217484+0000", 2024-08-17T17:41:05.076 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.15355152499999999 2024-08-17T17:41:05.076 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.076 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.076 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:41:05.076 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.217571+0000", 2024-08-17T17:41:05.076 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7198000000000001e-05 2024-08-17T17:41:05.076 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.076 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.076 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:05.076 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.219376+0000", 2024-08-17T17:41:05.076 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0018053419999999999 2024-08-17T17:41:05.076 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.076 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:05.076 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.076 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.077 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.077 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36354.0:185849 9.e 9:73fbc905:::benchmark_data_smithi003_261051_object69964:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-17T17:41:05.077 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:40:56.044199+0000", 2024-08-17T17:41:05.077 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 8.9394138400000003, 2024-08-17T17:41:05.077 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.24577770500000001, 2024-08-17T17:41:05.077 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:05.077 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:05.077 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:41:05.077 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:41:05.077 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36354", 2024-08-17T17:41:05.077 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1330158990", 2024-08-17T17:41:05.077 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 185849 2024-08-17T17:41:05.077 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.077 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:05.077 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.078 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:05.078 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.044199+0000", 2024-08-17T17:41:05.078 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.078 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.078 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.078 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:05.078 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.044199+0000", 2024-08-17T17:41:05.078 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.078 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.078 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.078 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:05.078 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.044200+0000", 2024-08-17T17:41:05.078 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0400000000000005e-07 2024-08-17T17:41:05.078 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.079 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.079 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:05.079 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.044204+0000", 2024-08-17T17:41:05.079 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1330000000000001e-06 2024-08-17T17:41:05.079 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.079 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.079 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:05.079 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.044215+0000", 2024-08-17T17:41:05.079 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1209000000000001e-05 2024-08-17T17:41:05.079 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.079 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.079 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:05.079 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.044222+0000", 2024-08-17T17:41:05.079 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9800000000000001e-06 2024-08-17T17:41:05.079 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.080 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.080 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:05.080 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.063970+0000", 2024-08-17T17:41:05.080 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.01974795 2024-08-17T17:41:05.080 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.080 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.080 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for rw locks", 2024-08-17T17:41:05.080 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.064076+0000", 2024-08-17T17:41:05.080 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000106189 2024-08-17T17:41:05.080 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.080 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.080 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:05.080 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.232114+0000", 2024-08-17T17:41:05.080 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.168037619 2024-08-17T17:41:05.080 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.081 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.081 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:05.081 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.232257+0000", 2024-08-17T17:41:05.081 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014321200000000001 2024-08-17T17:41:05.081 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.081 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.081 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,7", 2024-08-17T17:41:05.081 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.232402+0000", 2024-08-17T17:41:05.081 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014444300000000001 2024-08-17T17:41:05.081 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.081 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.081 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:05.081 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.255634+0000", 2024-08-17T17:41:05.081 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.023231892000000001 2024-08-17T17:41:05.081 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.082 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.082 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:41:05.082 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.255654+0000", 2024-08-17T17:41:05.082 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0707e-05 2024-08-17T17:41:05.082 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.082 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.082 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:05.082 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.288100+0000", 2024-08-17T17:41:05.082 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.032445430999999997 2024-08-17T17:41:05.082 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.082 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.082 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:41:05.082 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.288169+0000", 2024-08-17T17:41:05.083 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.915e-05 2024-08-17T17:41:05.083 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.083 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.083 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:05.083 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.289977+0000", 2024-08-17T17:41:05.083 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0018079859999999999 2024-08-17T17:41:05.083 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.083 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:05.083 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.083 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.083 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.083 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36354.0:185864 9.4 9:2d723ac4:::benchmark_data_smithi003_261051_object69979:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-17T17:41:05.083 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:40:56.065354+0000", 2024-08-17T17:41:05.083 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 8.9182593069999996, 2024-08-17T17:41:05.083 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.238942406, 2024-08-17T17:41:05.084 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:05.084 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:05.084 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:41:05.084 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:41:05.084 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36354", 2024-08-17T17:41:05.084 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1330158990", 2024-08-17T17:41:05.084 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 185864 2024-08-17T17:41:05.084 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.084 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:05.084 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.084 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:05.084 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.065354+0000", 2024-08-17T17:41:05.084 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.084 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.084 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.084 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:05.085 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.065354+0000", 2024-08-17T17:41:05.085 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.085 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.085 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.085 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:05.085 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.065355+0000", 2024-08-17T17:41:05.085 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5579999999999999e-06 2024-08-17T17:41:05.085 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.085 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.085 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:05.085 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.065362+0000", 2024-08-17T17:41:05.085 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5209999999999999e-06 2024-08-17T17:41:05.085 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.085 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.085 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:05.086 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.065378+0000", 2024-08-17T17:41:05.086 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5838999999999999e-05 2024-08-17T17:41:05.086 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.086 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.086 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:05.086 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.065386+0000", 2024-08-17T17:41:05.086 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1349999999999992e-06 2024-08-17T17:41:05.087 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.087 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.087 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:05.087 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.065910+0000", 2024-08-17T17:41:05.087 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00052377399999999996 2024-08-17T17:41:05.087 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.087 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.087 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:05.087 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.066299+0000", 2024-08-17T17:41:05.087 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000389732 2024-08-17T17:41:05.087 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.087 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.088 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,3", 2024-08-17T17:41:05.088 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.066399+0000", 2024-08-17T17:41:05.088 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9796999999999993e-05 2024-08-17T17:41:05.088 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.088 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.088 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:41:05.088 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.075517+0000", 2024-08-17T17:41:05.088 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0091178490000000008 2024-08-17T17:41:05.088 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.088 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.088 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:05.088 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.075615+0000", 2024-08-17T17:41:05.088 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8408999999999999e-05 2024-08-17T17:41:05.088 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.089 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.089 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:05.089 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.300178+0000", 2024-08-17T17:41:05.089 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.224563075 2024-08-17T17:41:05.089 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.089 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.089 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:41:05.089 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.300233+0000", 2024-08-17T17:41:05.089 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4206999999999998e-05 2024-08-17T17:41:05.089 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.089 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.089 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:05.089 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.304296+0000", 2024-08-17T17:41:05.089 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040635100000000002 2024-08-17T17:41:05.090 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.090 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:05.090 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.090 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.090 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.090 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36354.0:185869 9.4 9:2baf2493:::benchmark_data_smithi003_261051_object69984:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-17T17:41:05.090 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:40:56.070007+0000", 2024-08-17T17:41:05.090 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 8.9136055919999997, 2024-08-17T17:41:05.090 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.245844707, 2024-08-17T17:41:05.090 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:05.090 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:05.090 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:41:05.090 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:41:05.090 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36354", 2024-08-17T17:41:05.091 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1330158990", 2024-08-17T17:41:05.091 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 185869 2024-08-17T17:41:05.091 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.091 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:05.091 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.091 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:05.091 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.070007+0000", 2024-08-17T17:41:05.091 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.091 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.091 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.091 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:05.091 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.070007+0000", 2024-08-17T17:41:05.091 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.092 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.092 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.092 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:05.092 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.070008+0000", 2024-08-17T17:41:05.092 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5199999999999995e-07 2024-08-17T17:41:05.092 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.092 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.092 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:05.092 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.070067+0000", 2024-08-17T17:41:05.092 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8480999999999998e-05 2024-08-17T17:41:05.092 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.092 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.092 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:05.092 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.070078+0000", 2024-08-17T17:41:05.093 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0900999999999999e-05 2024-08-17T17:41:05.093 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.093 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.093 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:05.093 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.070084+0000", 2024-08-17T17:41:05.093 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7440000000000001e-06 2024-08-17T17:41:05.093 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.093 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.093 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:05.093 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.072768+0000", 2024-08-17T17:41:05.093 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026833650000000001 2024-08-17T17:41:05.093 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.093 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.093 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:05.093 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.072941+0000", 2024-08-17T17:41:05.094 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017289399999999999 2024-08-17T17:41:05.094 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.094 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.094 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,3", 2024-08-17T17:41:05.094 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.073117+0000", 2024-08-17T17:41:05.094 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017597700000000001 2024-08-17T17:41:05.094 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.094 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.094 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:41:05.094 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.079228+0000", 2024-08-17T17:41:05.094 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0061115020000000004 2024-08-17T17:41:05.094 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.094 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.095 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:05.095 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.079279+0000", 2024-08-17T17:41:05.095 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0958000000000002e-05 2024-08-17T17:41:05.095 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.095 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.095 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:05.095 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.312207+0000", 2024-08-17T17:41:05.095 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.232927682 2024-08-17T17:41:05.095 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.095 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.095 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:41:05.095 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.312280+0000", 2024-08-17T17:41:05.095 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3357999999999998e-05 2024-08-17T17:41:05.095 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.095 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.095 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:05.096 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.315852+0000", 2024-08-17T17:41:05.096 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003571893 2024-08-17T17:41:05.096 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.096 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:05.096 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.096 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.096 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.096 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36354.0:185870 9.4 9:2cbbf017:::benchmark_data_smithi003_261051_object69985:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-17T17:41:05.096 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:40:56.070538+0000", 2024-08-17T17:41:05.096 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 8.9130751779999997, 2024-08-17T17:41:05.096 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.248458659, 2024-08-17T17:41:05.096 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:05.096 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:05.096 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:41:05.097 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:41:05.097 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36354", 2024-08-17T17:41:05.097 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1330158990", 2024-08-17T17:41:05.097 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 185870 2024-08-17T17:41:05.097 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.097 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:05.097 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.097 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:05.097 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.070538+0000", 2024-08-17T17:41:05.097 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.097 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.097 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.097 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:05.097 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.070538+0000", 2024-08-17T17:41:05.098 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.098 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.098 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.098 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:05.098 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.070539+0000", 2024-08-17T17:41:05.098 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8299999999999995e-07 2024-08-17T17:41:05.098 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.098 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.098 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:05.098 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.070543+0000", 2024-08-17T17:41:05.098 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.189e-06 2024-08-17T17:41:05.098 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.098 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.099 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:05.099 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.070554+0000", 2024-08-17T17:41:05.099 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0893000000000001e-05 2024-08-17T17:41:05.099 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.099 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.099 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:05.099 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.070561+0000", 2024-08-17T17:41:05.099 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9249999999999998e-06 2024-08-17T17:41:05.099 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.099 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.099 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:05.099 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.075668+0000", 2024-08-17T17:41:05.099 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.005107576 2024-08-17T17:41:05.099 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.100 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.100 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:05.100 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.075779+0000", 2024-08-17T17:41:05.100 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011114499999999999 2024-08-17T17:41:05.100 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.100 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.100 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,3", 2024-08-17T17:41:05.100 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.075888+0000", 2024-08-17T17:41:05.100 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000108521 2024-08-17T17:41:05.100 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.100 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.100 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:41:05.100 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.079233+0000", 2024-08-17T17:41:05.100 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0033449640000000002 2024-08-17T17:41:05.100 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.101 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.101 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:05.101 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.079679+0000", 2024-08-17T17:41:05.101 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00044570299999999998 2024-08-17T17:41:05.101 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.101 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.101 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:05.101 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.315911+0000", 2024-08-17T17:41:05.101 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.23623243899999999 2024-08-17T17:41:05.101 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.101 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.101 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:41:05.101 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.315999+0000", 2024-08-17T17:41:05.102 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7714999999999996e-05 2024-08-17T17:41:05.102 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.102 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.102 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:05.102 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.318996+0000", 2024-08-17T17:41:05.102 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029977060000000002 2024-08-17T17:41:05.102 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.102 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:05.102 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.102 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.102 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.102 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36354.0:185871 9.4 9:2f5ae608:::benchmark_data_smithi003_261051_object69986:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-08-17T17:41:05.102 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:40:56.073820+0000", 2024-08-17T17:41:05.102 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 8.9097927109999997, 2024-08-17T17:41:05.103 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.25858441500000001, 2024-08-17T17:41:05.103 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:05.103 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:05.103 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:41:05.103 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:41:05.103 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36354", 2024-08-17T17:41:05.103 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1330158990", 2024-08-17T17:41:05.103 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 185871 2024-08-17T17:41:05.103 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.103 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:05.103 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.103 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:05.103 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.073820+0000", 2024-08-17T17:41:05.103 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.103 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.104 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.104 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:05.104 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.073820+0000", 2024-08-17T17:41:05.104 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:05.104 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.104 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.104 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:05.104 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.073822+0000", 2024-08-17T17:41:05.104 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.294e-06 2024-08-17T17:41:05.104 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.104 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.104 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:05.104 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.073826+0000", 2024-08-17T17:41:05.104 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4610000000000001e-06 2024-08-17T17:41:05.105 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.105 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.105 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:05.105 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.073836+0000", 2024-08-17T17:41:05.105 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7550000000000006e-06 2024-08-17T17:41:05.105 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.105 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.105 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:05.105 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.073843+0000", 2024-08-17T17:41:05.105 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3440000000000003e-06 2024-08-17T17:41:05.105 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.105 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.105 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:05.105 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.077800+0000", 2024-08-17T17:41:05.106 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039566410000000003 2024-08-17T17:41:05.106 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.106 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.106 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:05.106 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.077904+0000", 2024-08-17T17:41:05.106 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000104342 2024-08-17T17:41:05.106 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.106 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.106 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,3", 2024-08-17T17:41:05.106 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.078013+0000", 2024-08-17T17:41:05.106 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000109121 2024-08-17T17:41:05.106 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.106 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.106 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:41:05.107 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.079389+0000", 2024-08-17T17:41:05.107 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001375338 2024-08-17T17:41:05.107 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.107 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.107 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:05.107 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.079801+0000", 2024-08-17T17:41:05.107 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00041275600000000002 2024-08-17T17:41:05.107 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.107 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.107 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:05.107 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.328906+0000", 2024-08-17T17:41:05.107 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.24910434100000001 2024-08-17T17:41:05.107 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.108 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.108 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:41:05.108 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.328957+0000", 2024-08-17T17:41:05.108 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1363999999999998e-05 2024-08-17T17:41:05.108 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:05.108 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:05.108 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:05.108 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:40:56.332405+0000", 2024-08-17T17:41:05.108 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034476580000000001 2024-08-17T17:41:05.108 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.108 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:05.108 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.108 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:05.108 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:05.109 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:41:05.133 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:05.175 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:05.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:05.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:05.277 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:05.301 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:05.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:05.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:05.402 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:05.414 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-08-17T17:41:05.427 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:05.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:05.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:05.529 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:05.566 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:05.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:05.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:05.652 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:05 smithi003 ceph-mon[103344]: pgmap v1103: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 632 op/s; 15046/128709 objects degraded (11.690%); 1895/128709 objects misplaced (1.472%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:41:05.652 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:05 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3976576379' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:41:05.652 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:05 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:41:05.652 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:05 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 15046/128709 objects degraded (11.690%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:41:05.653 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:05 smithi003 ceph-mon[100615]: pgmap v1103: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 632 op/s; 15046/128709 objects degraded (11.690%); 1895/128709 objects misplaced (1.472%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:41:05.653 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:05 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3976576379' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:41:05.653 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:05 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:41:05.653 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:05 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 15046/128709 objects degraded (11.690%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:41:05.667 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:05.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:05.692 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:05.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:05.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:05 smithi053 ceph-mon[89963]: pgmap v1103: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 632 op/s; 15046/128709 objects degraded (11.690%); 1895/128709 objects misplaced (1.472%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:41:05.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:05 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3976576379' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:41:05.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:05 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:41:05.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:05 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 15046/128709 objects degraded (11.690%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:41:05.794 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:05.817 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:05.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:05.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:05.919 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:05.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:05.959 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:05.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:05.975 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:41:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:41:05] "GET /metrics HTTP/1.1" 200 36525 "" "Prometheus/2.51.0" 2024-08-17T17:41:06.060 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:06.085 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:06.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:06.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:06.187 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:06.213 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:06.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:06.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:06.315 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:06.366 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:06.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:06.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:06.468 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:06.510 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:06.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:06.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:06.612 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:06.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:06.653 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:06.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:06.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:06 smithi053 ceph-mon[89963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:41:06.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:06 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:41:06.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:06 smithi053 ceph-mon[89963]: osdmap e338: 8 total, 8 up, 7 in 2024-08-17T17:41:06.754 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:06.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:06 smithi003 ceph-mon[100615]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:41:06.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:06 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:41:06.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:06 smithi003 ceph-mon[100615]: osdmap e338: 8 total, 8 up, 7 in 2024-08-17T17:41:06.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:06 smithi003 ceph-mon[103344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:41:06.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:06 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:41:06.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:06 smithi003 ceph-mon[103344]: osdmap e338: 8 total, 8 up, 7 in 2024-08-17T17:41:06.777 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:06.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:06.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:06.879 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:06.932 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:06.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:06.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:06.941 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-17T17:41:07.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:07.084 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:07.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:07.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:07.186 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:07.233 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:07.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:07.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:07.335 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:07.364 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:07.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:07.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:07.466 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:07.490 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:07.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:07.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:07.592 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:07.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:07.619 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:07.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:07.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:07 smithi053 ceph-mon[89963]: pgmap v1105: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 721 op/s; 15140/118377 objects degraded (12.790%); 1673/118377 objects misplaced (1.413%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:41:07.720 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:07.766 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:07.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:07.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:07.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:07 smithi003 ceph-mon[100615]: pgmap v1105: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 721 op/s; 15140/118377 objects degraded (12.790%); 1673/118377 objects misplaced (1.413%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:41:07.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:07 smithi003 ceph-mon[103344]: pgmap v1105: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 721 op/s; 15140/118377 objects degraded (12.790%); 1673/118377 objects misplaced (1.413%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:41:07.868 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:07.892 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:07.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:07.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:07.964 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:41:07.994 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:08.017 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:08.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:08.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:08.119 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:08.146 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:08.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:08.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:08.248 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:08.277 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:08.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:08.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:08.379 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:08.424 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:08.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:08.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:08.526 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:08.583 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:08.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:08.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:08.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:08.728 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:08.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:08.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:08.830 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:08.855 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:08.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:08.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:08.957 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:08.997 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:08.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:08.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:09.099 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:09.123 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:09.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:09.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:09.218 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:41:09.224 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:09.248 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:09.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:09.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:09.349 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:09.392 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:09.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:09.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:41:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:41:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:09 smithi053 ceph-mon[89963]: pgmap v1106: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 721 op/s; 15140/118377 objects degraded (12.790%); 1673/118377 objects misplaced (1.413%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:41:09.494 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:09.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:41:09.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:41:09.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:09 smithi003 ceph-mon[103344]: pgmap v1106: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 721 op/s; 15140/118377 objects degraded (12.790%); 1673/118377 objects misplaced (1.413%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:41:09.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:41:09.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:41:09.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:09 smithi003 ceph-mon[100615]: pgmap v1106: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 721 op/s; 15140/118377 objects degraded (12.790%); 1673/118377 objects misplaced (1.413%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:41:09.516 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:41:09.528 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:09.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:09.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:09.630 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:09.656 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:09.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:09.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:09.758 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:09.801 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:09.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:09.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:09.902 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:09.929 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:09.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:09.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:10.033 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:10.070 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:10.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:10.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:10.154 INFO:teuthology.orchestra.run.smithi003.stderr:set pool 1 pg_num to 31 2024-08-17T17:41:10.172 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:10.200 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:10.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:10.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:10.214 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:41:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a[76692]: logger=cleanup t=2024-08-17T17:41:10.128697826Z level=info msg="Completed cleanup jobs" duration=46.350702ms 2024-08-17T17:41:10.214 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:10 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/132738099' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-08-17T17:41:10.214 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:10 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-08-17T17:41:10.214 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:10 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 15140/118377 objects degraded (12.790%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:41:10.302 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:10.329 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:10.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:10.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:10.431 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:10.459 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:10.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:10.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:10.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:41:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:41:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:41:10.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:10 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/132738099' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-08-17T17:41:10.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:10 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-08-17T17:41:10.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:10 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 15140/118377 objects degraded (12.790%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:41:10.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:10 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/132738099' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-08-17T17:41:10.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:10 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-08-17T17:41:10.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:10 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 15140/118377 objects degraded (12.790%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:41:10.561 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:10.586 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:10.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:10.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:10.687 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:10.712 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:10.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:10.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:10.814 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:10.838 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:10.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:10.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:10.940 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:10.982 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:10.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:10.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:11.084 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:11.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:11.121 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:11.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:11.222 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:11.261 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:11.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:11.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:11.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:11 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-08-17T17:41:11.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:11 smithi003 ceph-mon[103344]: osdmap e339: 8 total, 8 up, 7 in 2024-08-17T17:41:11.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:11 smithi003 ceph-mon[103344]: pgmap v1108: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 724 op/s; 15140/111999 objects degraded (13.518%); 1673/111999 objects misplaced (1.494%); 1024 KiB/s, 15 objects/s recovering 2024-08-17T17:41:11.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:11 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-08-17T17:41:11.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:11 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-08-17T17:41:11.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:11 smithi003 ceph-mon[100615]: osdmap e339: 8 total, 8 up, 7 in 2024-08-17T17:41:11.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:11 smithi003 ceph-mon[100615]: pgmap v1108: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 724 op/s; 15140/111999 objects degraded (13.518%); 1673/111999 objects misplaced (1.494%); 1024 KiB/s, 15 objects/s recovering 2024-08-17T17:41:11.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:11 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-08-17T17:41:11.363 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:11.387 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:11.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:11.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:11.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:11 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-08-17T17:41:11.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:11 smithi053 ceph-mon[89963]: osdmap e339: 8 total, 8 up, 7 in 2024-08-17T17:41:11.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:11 smithi053 ceph-mon[89963]: pgmap v1108: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 724 op/s; 15140/111999 objects degraded (13.518%); 1673/111999 objects misplaced (1.494%); 1024 KiB/s, 15 objects/s recovering 2024-08-17T17:41:11.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:11 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-08-17T17:41:11.489 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:11.514 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:11.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:11.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:11.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:11.662 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:11.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:11.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:11.764 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:11.790 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:11.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:11.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:11.892 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:11.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:11.974 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:11.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:12.075 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:12.099 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:12.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:12.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:12.201 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:12.243 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:12.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:12.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:12.345 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:12.373 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:41:12.373 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:41:12.373 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:41:12.374 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:41:12.374 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:41:12.374 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:41:12.374 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:41:12.374 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:41:12.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:12.383 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:12.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:12.398 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:41:12.399 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:41:12.428 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:41:12.429 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:41:12.461 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:12 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-08-17T17:41:12.461 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:12 smithi003 ceph-mon[103344]: osdmap e340: 8 total, 8 up, 7 in 2024-08-17T17:41:12.461 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:12 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1251198301' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:41:12.462 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:41:12.462 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:41:12.462 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:12 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-08-17T17:41:12.462 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:12 smithi003 ceph-mon[100615]: osdmap e340: 8 total, 8 up, 7 in 2024-08-17T17:41:12.462 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:12 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1251198301' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:41:12.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:12 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-08-17T17:41:12.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:12 smithi053 ceph-mon[89963]: osdmap e340: 8 total, 8 up, 7 in 2024-08-17T17:41:12.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:12 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1251198301' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:41:12.484 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:12.494 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:41:12.494 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:41:12.527 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:12.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:12.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:12.629 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:12.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:12.669 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:12.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:12.769 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:12.795 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:12.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:12.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:12.897 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:12.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:12.924 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:12.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:13.025 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:13.050 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:13.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:13.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:13.152 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:13.189 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:13.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:13.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:13.290 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:13.326 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:13.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:13.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:13.428 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:13.453 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:13.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:13.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:13.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:13 smithi053 ceph-mon[89963]: osdmap e341: 8 total, 8 up, 7 in 2024-08-17T17:41:13.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:13 smithi053 ceph-mon[89963]: pgmap v1111: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 10 unknown, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.2 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 649 op/s; 15255/106686 objects degraded (14.299%); 1428/106686 objects misplaced (1.339%); 1.4 MiB/s, 21 objects/s recovering 2024-08-17T17:41:13.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:13 smithi053 ceph-mon[89963]: 1.c scrub starts 2024-08-17T17:41:13.525 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:13 smithi003 ceph-mon[100615]: osdmap e341: 8 total, 8 up, 7 in 2024-08-17T17:41:13.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:13 smithi003 ceph-mon[100615]: pgmap v1111: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 10 unknown, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.2 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 649 op/s; 15255/106686 objects degraded (14.299%); 1428/106686 objects misplaced (1.339%); 1.4 MiB/s, 21 objects/s recovering 2024-08-17T17:41:13.526 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:13 smithi003 ceph-mon[100615]: 1.c scrub starts 2024-08-17T17:41:13.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:13 smithi003 ceph-mon[103344]: osdmap e341: 8 total, 8 up, 7 in 2024-08-17T17:41:13.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:13 smithi003 ceph-mon[103344]: pgmap v1111: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 10 unknown, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 2.2 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 649 op/s; 15255/106686 objects degraded (14.299%); 1428/106686 objects misplaced (1.339%); 1.4 MiB/s, 21 objects/s recovering 2024-08-17T17:41:13.526 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:13 smithi003 ceph-mon[103344]: 1.c scrub starts 2024-08-17T17:41:13.554 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:13.578 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:13.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:13.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:13.680 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:13.708 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:13.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:13.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:13.762 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:41:13.810 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:13.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:13.847 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:13.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:13.948 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:13.986 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:13.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:13.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:14.087 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:14.125 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:14.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:14.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:14.227 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:14.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:14.272 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:14.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:14.373 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:14.398 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:14.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:14.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:14.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:14 smithi053 ceph-mon[89963]: 1.1b scrub starts 2024-08-17T17:41:14.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:14 smithi053 ceph-mon[89963]: 1.5 scrub starts 2024-08-17T17:41:14.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:14 smithi053 ceph-mon[89963]: 1.1c scrub starts 2024-08-17T17:41:14.501 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:14.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:14 smithi003 ceph-mon[103344]: 1.1b scrub starts 2024-08-17T17:41:14.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:14 smithi003 ceph-mon[103344]: 1.5 scrub starts 2024-08-17T17:41:14.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:14 smithi003 ceph-mon[103344]: 1.1c scrub starts 2024-08-17T17:41:14.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:14 smithi003 ceph-mon[100615]: 1.1b scrub starts 2024-08-17T17:41:14.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:14 smithi003 ceph-mon[100615]: 1.5 scrub starts 2024-08-17T17:41:14.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:14 smithi003 ceph-mon[100615]: 1.1c scrub starts 2024-08-17T17:41:14.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:14.539 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:14.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:14.640 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:14.666 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:14.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:14.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:14.768 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:14.791 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:14.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:14.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:14.893 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:14.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:14.934 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:14.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:15.035 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:15.060 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:15.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:15.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:15.162 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:15.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:15.187 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:15.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:15.288 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:15.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:15.340 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:15.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:15.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:15 smithi003 ceph-mon[103344]: 1.b scrub starts 2024-08-17T17:41:15.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:15 smithi003 ceph-mon[103344]: 1.15 scrub starts 2024-08-17T17:41:15.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:15 smithi003 ceph-mon[103344]: pgmap v1112: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 5 unknown, 2 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 686 op/s; 15255/106020 objects degraded (14.389%); 1428/106020 objects misplaced (1.347%); 1.4 MiB/s, 21 objects/s recovering 2024-08-17T17:41:15.433 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:15 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 15255/106686 objects degraded (14.299%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:41:15.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:15 smithi003 ceph-mon[100615]: 1.b scrub starts 2024-08-17T17:41:15.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:15 smithi003 ceph-mon[100615]: 1.15 scrub starts 2024-08-17T17:41:15.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:15 smithi003 ceph-mon[100615]: pgmap v1112: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 5 unknown, 2 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 686 op/s; 15255/106020 objects degraded (14.389%); 1428/106020 objects misplaced (1.347%); 1.4 MiB/s, 21 objects/s recovering 2024-08-17T17:41:15.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:15 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 15255/106686 objects degraded (14.299%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:41:15.441 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:15.486 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:15.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:15.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:15.588 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:15.612 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:15.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:15.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:15.714 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:15.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:15 smithi053 ceph-mon[89963]: 1.b scrub starts 2024-08-17T17:41:15.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:15 smithi053 ceph-mon[89963]: 1.15 scrub starts 2024-08-17T17:41:15.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:15 smithi053 ceph-mon[89963]: pgmap v1112: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 5 unknown, 2 active+recovery_wait+undersized+degraded+remapped, 111 active+clean; 2.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 686 op/s; 15255/106020 objects degraded (14.389%); 1428/106020 objects misplaced (1.347%); 1.4 MiB/s, 21 objects/s recovering 2024-08-17T17:41:15.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:15 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 15255/106686 objects degraded (14.299%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:41:15.739 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:15.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:15.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:15.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:41:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:41:15] "GET /metrics HTTP/1.1" 200 36524 "" "Prometheus/2.51.0" 2024-08-17T17:41:15.840 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:15.867 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:15.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:15.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:15.969 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:16.014 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:16.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:16.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:16.116 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:16.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:16.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:16.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:16.256 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:16.283 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:16.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:16.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:16.360 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:41:16.385 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:16.409 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:16.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:16.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:16.510 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:16.535 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:16.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:16.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:16.637 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:16.638 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-08-17T17:41:16.638 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:41:16.639 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-08-17T17:41:16.639 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd in 4 2024-08-17T17:41:16.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:16.708 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:16.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:16.715 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:41:16.715 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:41:16.810 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:16.839 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:16.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:16.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:16.941 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:16.967 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:16.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:16.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:17.069 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:17.094 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:17.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:17.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:17.195 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:17.219 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:17.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:17.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:17.321 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:17.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:17.379 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:17.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:17.480 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:17.512 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:17.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:17.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:17.592 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:17 smithi003 ceph-mon[103344]: 1.8 scrub starts 2024-08-17T17:41:17.592 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:17 smithi003 ceph-mon[103344]: pgmap v1113: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 112 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 922 op/s; 15349/94959 objects degraded (16.164%); 1169/94959 objects misplaced (1.231%); 3.0 MiB/s, 47 objects/s recovering 2024-08-17T17:41:17.592 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:17 smithi003 ceph-mon[100615]: 1.8 scrub starts 2024-08-17T17:41:17.592 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:17 smithi003 ceph-mon[100615]: pgmap v1113: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 112 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 922 op/s; 15349/94959 objects degraded (16.164%); 1169/94959 objects misplaced (1.231%); 3.0 MiB/s, 47 objects/s recovering 2024-08-17T17:41:17.614 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:17.659 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:17.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:17.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:17.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:17 smithi053 ceph-mon[89963]: 1.8 scrub starts 2024-08-17T17:41:17.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:17 smithi053 ceph-mon[89963]: pgmap v1113: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 112 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 922 op/s; 15349/94959 objects degraded (16.164%); 1169/94959 objects misplaced (1.231%); 3.0 MiB/s, 47 objects/s recovering 2024-08-17T17:41:17.761 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:17.788 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:17.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:17.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:17.890 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:17.917 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:17.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:17.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:18.019 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:18.043 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:18.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:18.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:18.145 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:18.174 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:18.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:18.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:18.276 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:18.319 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:18.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:18.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:18.421 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:18.445 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:18.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:18.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:18.547 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:18.570 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:18.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:18.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:18.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:18.714 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:18.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:18.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:18.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:18 smithi053 ceph-mon[89963]: 1.18 scrub starts 2024-08-17T17:41:18.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:18 smithi053 ceph-mon[89963]: 1.5 scrub starts 2024-08-17T17:41:18.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:18 smithi003 ceph-mon[103344]: 1.18 scrub starts 2024-08-17T17:41:18.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:18 smithi003 ceph-mon[103344]: 1.5 scrub starts 2024-08-17T17:41:18.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:18 smithi003 ceph-mon[100615]: 1.18 scrub starts 2024-08-17T17:41:18.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:18 smithi003 ceph-mon[100615]: 1.5 scrub starts 2024-08-17T17:41:18.817 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:18.844 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:18.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:18.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:18.946 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:18.985 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:18.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:18.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:19.087 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:19.125 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:19.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:19.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:19.227 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:19.251 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:19.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:19.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:19.353 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:19.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:19.393 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:19.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:19.493 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:19.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:19 smithi003 ceph-mon[103344]: 1.c scrub starts 2024-08-17T17:41:19.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:19 smithi003 ceph-mon[103344]: pgmap v1114: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 112 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 709 op/s; 15349/94959 objects degraded (16.164%); 1169/94959 objects misplaced (1.231%); 2.3 MiB/s, 36 objects/s recovering 2024-08-17T17:41:19.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:19 smithi003 ceph-mon[103344]: 1.1b scrub starts 2024-08-17T17:41:19.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:19 smithi003 ceph-mon[103344]: 1.15 scrub starts 2024-08-17T17:41:19.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:19 smithi003 ceph-mon[100615]: 1.c scrub starts 2024-08-17T17:41:19.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:19 smithi003 ceph-mon[100615]: pgmap v1114: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 112 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 709 op/s; 15349/94959 objects degraded (16.164%); 1169/94959 objects misplaced (1.231%); 2.3 MiB/s, 36 objects/s recovering 2024-08-17T17:41:19.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:19 smithi003 ceph-mon[100615]: 1.1b scrub starts 2024-08-17T17:41:19.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:19 smithi003 ceph-mon[100615]: 1.15 scrub starts 2024-08-17T17:41:19.530 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:19.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:19.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:19.632 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:19.674 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:19.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:19.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:19.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:19 smithi053 ceph-mon[89963]: 1.c scrub starts 2024-08-17T17:41:19.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:19 smithi053 ceph-mon[89963]: pgmap v1114: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 112 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 709 op/s; 15349/94959 objects degraded (16.164%); 1169/94959 objects misplaced (1.231%); 2.3 MiB/s, 36 objects/s recovering 2024-08-17T17:41:19.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:19 smithi053 ceph-mon[89963]: 1.1b scrub starts 2024-08-17T17:41:19.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:19 smithi053 ceph-mon[89963]: 1.15 scrub starts 2024-08-17T17:41:19.776 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:19.801 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:19.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:19.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:19.903 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:19.926 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:19.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:19.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:20.028 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:20.056 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:20.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:20.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:20.158 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:20.188 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:20.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:20.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:20.290 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:20.335 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:20.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:20.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:20.399 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-08-17T17:41:20.437 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:20.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:20.462 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:20.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:20.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:41:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:41:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:41:20.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:20 smithi053 ceph-mon[89963]: 1.1c scrub starts 2024-08-17T17:41:20.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:20 smithi053 ceph-mon[89963]: 1.b scrub starts 2024-08-17T17:41:20.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:20 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:41:20.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:20 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3102836138' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:41:20.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:20 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 15349/94959 objects degraded (16.164%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:41:20.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:20 smithi003 ceph-mon[103344]: 1.1c scrub starts 2024-08-17T17:41:20.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:20 smithi003 ceph-mon[103344]: 1.b scrub starts 2024-08-17T17:41:20.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:20 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:41:20.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:20 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3102836138' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:41:20.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:20 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 15349/94959 objects degraded (16.164%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:41:20.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:20 smithi003 ceph-mon[100615]: 1.1c scrub starts 2024-08-17T17:41:20.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:20 smithi003 ceph-mon[100615]: 1.b scrub starts 2024-08-17T17:41:20.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:20 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:41:20.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:20 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3102836138' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:41:20.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:20 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 15349/94959 objects degraded (16.164%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:41:20.563 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:20.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:20.594 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:20.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:20.695 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:20.732 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:20.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:20.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:20.834 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:20.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:20.861 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:20.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:20.962 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:20.992 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:20.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:20.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:21.094 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:21.115 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:21.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:21.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:21.217 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:21.240 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:21.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:21.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:21.341 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:21.363 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:21.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:21.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:21.465 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:21.503 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:21.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:21.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:21.576 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:21 smithi053 ceph-mon[89963]: pgmap v1115: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.8 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 661 op/s; 15349/88524 objects degraded (17.339%); 1169/88524 objects misplaced (1.321%); 1.1 MiB/s, 18 objects/s recovering 2024-08-17T17:41:21.576 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:21 smithi053 ceph-mon[89963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:41:21.576 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:21 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3102836138' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:41:21.576 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:21 smithi053 ceph-mon[89963]: osdmap e342: 8 total, 8 up, 7 in 2024-08-17T17:41:21.605 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:21.629 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:21.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:21.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:21.731 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:21.757 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:21.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:21.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:21.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:21 smithi003 ceph-mon[103344]: pgmap v1115: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.8 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 661 op/s; 15349/88524 objects degraded (17.339%); 1169/88524 objects misplaced (1.321%); 1.1 MiB/s, 18 objects/s recovering 2024-08-17T17:41:21.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:21 smithi003 ceph-mon[103344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:41:21.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:21 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3102836138' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:41:21.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:21 smithi003 ceph-mon[103344]: osdmap e342: 8 total, 8 up, 7 in 2024-08-17T17:41:21.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:21 smithi003 ceph-mon[100615]: pgmap v1115: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.8 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 661 op/s; 15349/88524 objects degraded (17.339%); 1169/88524 objects misplaced (1.321%); 1.1 MiB/s, 18 objects/s recovering 2024-08-17T17:41:21.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:21 smithi003 ceph-mon[100615]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:41:21.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:21 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3102836138' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:41:21.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:21 smithi003 ceph-mon[100615]: osdmap e342: 8 total, 8 up, 7 in 2024-08-17T17:41:21.859 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:21.897 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:21.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:21.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:21.999 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:22.045 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:22.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:22.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:22.147 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:22.174 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:22.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:22.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:22.275 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:22.301 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:22.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:22.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:22.402 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:22.443 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:22.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:22.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:22.544 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:22.570 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:22.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:22.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:22.672 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:22.709 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:22.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:22.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:22.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:22.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:22.833 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:22.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:22.934 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:22.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:22.975 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:22.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:23.076 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:23.113 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:23.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:23.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:23.215 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:23.262 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:23.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:23.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:23.364 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:23.396 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:23.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:23.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:23.497 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:23.532 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:23.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:23.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:23.634 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:23.660 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:23.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:23.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:23 smithi053 ceph-mon[89963]: pgmap v1117: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 1.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 773 op/s; 15440/83463 objects degraded (18.499%); 933/83463 objects misplaced (1.118%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T17:41:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:23 smithi053 ceph-mon[89963]: 1.8 scrub starts 2024-08-17T17:41:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:41:23.762 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:23.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:23 smithi003 ceph-mon[103344]: pgmap v1117: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 1.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 773 op/s; 15440/83463 objects degraded (18.499%); 933/83463 objects misplaced (1.118%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T17:41:23.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:23 smithi003 ceph-mon[103344]: 1.8 scrub starts 2024-08-17T17:41:23.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:41:23.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:23 smithi003 ceph-mon[100615]: pgmap v1117: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 1.7 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 773 op/s; 15440/83463 objects degraded (18.499%); 933/83463 objects misplaced (1.118%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T17:41:23.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:23 smithi003 ceph-mon[100615]: 1.8 scrub starts 2024-08-17T17:41:23.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:41:23.788 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:23.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:23.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:23.890 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:23.936 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:23.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:23.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:24.038 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:24.080 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:24.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:24.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:24.182 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:24.208 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:24.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:24.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:24.310 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:24.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:24.355 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:24.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:24.456 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:24.491 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:24.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:24.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:24.593 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:24.626 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:24.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:24.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:24.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:24 smithi053 ceph-mon[89963]: 1.5 scrub starts 2024-08-17T17:41:24.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:24 smithi053 ceph-mon[89963]: 1.18 scrub starts 2024-08-17T17:41:24.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:24 smithi053 ceph-mon[89963]: 1.1b scrub starts 2024-08-17T17:41:24.728 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:24.745 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:24 smithi003 ceph-mon[103344]: 1.5 scrub starts 2024-08-17T17:41:24.745 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:24 smithi003 ceph-mon[103344]: 1.18 scrub starts 2024-08-17T17:41:24.745 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:24 smithi003 ceph-mon[103344]: 1.1b scrub starts 2024-08-17T17:41:24.745 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:24 smithi003 ceph-mon[100615]: 1.5 scrub starts 2024-08-17T17:41:24.745 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:24 smithi003 ceph-mon[100615]: 1.18 scrub starts 2024-08-17T17:41:24.745 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:24 smithi003 ceph-mon[100615]: 1.1b scrub starts 2024-08-17T17:41:24.754 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:24.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:24.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:24.788 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-17T17:41:24.856 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:24.926 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:24.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:24.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:25.028 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:25.103 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:25.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:25.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:25.205 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:25.229 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:25.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:25.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:25.331 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:25.372 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:25.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:25.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:25.474 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:25.498 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:25.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:25.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:25.600 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:25.605 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:41:25.606 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:41:25.658 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:41:25.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:25.686 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:25.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:25.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:25 smithi003 ceph-mon[100615]: pgmap v1118: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 1.7 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 772 op/s; 15440/82815 objects degraded (18.644%); 933/82815 objects misplaced (1.127%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T17:41:25.775 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:25 smithi003 ceph-mon[100615]: 1.15 scrub starts 2024-08-17T17:41:25.775 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:25 smithi003 ceph-mon[100615]: 1.c scrub starts 2024-08-17T17:41:25.775 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:25 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:41:25.775 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:25 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:41:25.775 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:25 smithi003 ceph-mon[100615]: 1.b scrub starts 2024-08-17T17:41:25.775 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:25 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 15440/82815 objects degraded (18.644%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:41:25.775 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:41:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:41:25] "GET /metrics HTTP/1.1" 200 36523 "" "Prometheus/2.51.0" 2024-08-17T17:41:25.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:25 smithi003 ceph-mon[103344]: pgmap v1118: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 1.7 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 772 op/s; 15440/82815 objects degraded (18.644%); 933/82815 objects misplaced (1.127%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T17:41:25.776 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:25 smithi003 ceph-mon[103344]: 1.15 scrub starts 2024-08-17T17:41:25.776 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:25 smithi003 ceph-mon[103344]: 1.c scrub starts 2024-08-17T17:41:25.776 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:25 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:41:25.776 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:25 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:41:25.776 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:25 smithi003 ceph-mon[103344]: 1.b scrub starts 2024-08-17T17:41:25.776 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:25 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 15440/82815 objects degraded (18.644%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:41:25.787 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:25.814 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:25.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:25.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:25.918 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:25.955 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:25.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:25.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:25.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:25 smithi053 ceph-mon[89963]: pgmap v1118: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 6 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 1.7 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 772 op/s; 15440/82815 objects degraded (18.644%); 933/82815 objects misplaced (1.127%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T17:41:25.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:25 smithi053 ceph-mon[89963]: 1.15 scrub starts 2024-08-17T17:41:25.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:25 smithi053 ceph-mon[89963]: 1.c scrub starts 2024-08-17T17:41:25.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:25 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:41:25.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:25 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:41:25.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:25 smithi053 ceph-mon[89963]: 1.b scrub starts 2024-08-17T17:41:25.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:25 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 15440/82815 objects degraded (18.644%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:41:26.057 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:26.114 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:26.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:26.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:26.216 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:26.249 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:26.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:26.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:26.351 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:26.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:26.391 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:26.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:26.492 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:26.519 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:26.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:26.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:26.621 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:26.656 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:26.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:26.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:26.758 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:26.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:26.803 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:26.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:26.819 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:26 smithi003 ceph-mon[103344]: 1.1c scrub starts 2024-08-17T17:41:26.819 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:26 smithi003 ceph-mon[100615]: 1.1c scrub starts 2024-08-17T17:41:26.904 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:26.947 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:26.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:26.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:26.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:26 smithi053 ceph-mon[89963]: 1.1c scrub starts 2024-08-17T17:41:27.048 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:27.073 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:27.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:27.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:27.175 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:27.213 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:27.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:27.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:27.316 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:27.361 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:27.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:27.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:27.464 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:27.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:27.530 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:27.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:27.561 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:41:27.632 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:27.656 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:27.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:27.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:27.758 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:27.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:27 smithi003 ceph-mon[100615]: pgmap v1119: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.5 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 741 op/s; 15548/72714 objects degraded (21.382%); 691/72714 objects misplaced (0.950%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T17:41:27.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:27 smithi003 ceph-mon[103344]: pgmap v1119: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.5 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 741 op/s; 15548/72714 objects degraded (21.382%); 691/72714 objects misplaced (0.950%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T17:41:27.815 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:27.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:27.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:27.917 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:27.942 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:27.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:27.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:27.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:27 smithi053 ceph-mon[89963]: pgmap v1119: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.5 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 741 op/s; 15548/72714 objects degraded (21.382%); 691/72714 objects misplaced (0.950%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T17:41:28.044 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:28.076 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:28.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:28.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:28.177 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:28.213 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:28.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:28.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:28.315 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:28.363 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:28.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:28.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:28.464 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:28.491 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:28.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:28.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:28.593 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:28.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:28.623 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:28.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:28.724 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:28.764 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:28.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:28.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:28.867 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:28.890 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:28.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:28.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:28.992 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:29.027 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:29.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:29.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:29.129 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:29.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:29.171 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:29.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:29.272 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:29.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:29.299 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:29.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:29.400 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:29.455 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:29.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:29.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:29.557 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:29.583 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:29.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:29.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:29.685 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:29.711 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:29.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:29.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:29.812 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:29.842 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:29.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:29.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:29.944 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:29.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:29 smithi053 ceph-mon[89963]: pgmap v1120: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.5 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 741 op/s; 15548/72714 objects degraded (21.382%); 691/72714 objects misplaced (0.950%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T17:41:29.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:29 smithi053 ceph-mon[89963]: 1.8 scrub starts 2024-08-17T17:41:29.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:29 smithi053 ceph-mon[89963]: 1.1b scrub starts 2024-08-17T17:41:29.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:29 smithi053 ceph-mon[89963]: 1.5 scrub starts 2024-08-17T17:41:29.970 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:29.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:29.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:30.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:29 smithi003 ceph-mon[100615]: pgmap v1120: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.5 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 741 op/s; 15548/72714 objects degraded (21.382%); 691/72714 objects misplaced (0.950%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T17:41:30.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:29 smithi003 ceph-mon[100615]: 1.8 scrub starts 2024-08-17T17:41:30.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:29 smithi003 ceph-mon[100615]: 1.1b scrub starts 2024-08-17T17:41:30.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:29 smithi003 ceph-mon[100615]: 1.5 scrub starts 2024-08-17T17:41:30.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:29 smithi003 ceph-mon[103344]: pgmap v1120: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 8 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 108 active+clean; 1.5 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 741 op/s; 15548/72714 objects degraded (21.382%); 691/72714 objects misplaced (0.950%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T17:41:30.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:29 smithi003 ceph-mon[103344]: 1.8 scrub starts 2024-08-17T17:41:30.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:29 smithi003 ceph-mon[103344]: 1.1b scrub starts 2024-08-17T17:41:30.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:29 smithi003 ceph-mon[103344]: 1.5 scrub starts 2024-08-17T17:41:30.072 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:30.099 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:30.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:30.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:30.201 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:30.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:30.230 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:30.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:30.331 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:30.355 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:30.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:30.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:30.456 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:30.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:41:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:41:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:41:30.489 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:30.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:30.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:30.591 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:30.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:30.632 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:30.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:30.733 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:30.768 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:30.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:30.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:30.870 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:30.896 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:30.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:30.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:30.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:30 smithi053 ceph-mon[89963]: 1.18 scrub starts 2024-08-17T17:41:30.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:30 smithi053 ceph-mon[89963]: 1.b scrub starts 2024-08-17T17:41:30.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:30 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 15548/72714 objects degraded (21.382%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:41:30.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:30 smithi053 ceph-mon[89963]: 1.15 scrub starts 2024-08-17T17:41:30.998 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:31.022 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:30 smithi003 ceph-mon[100615]: 1.18 scrub starts 2024-08-17T17:41:31.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:30 smithi003 ceph-mon[100615]: 1.b scrub starts 2024-08-17T17:41:31.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:30 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 15548/72714 objects degraded (21.382%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:41:31.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:30 smithi003 ceph-mon[100615]: 1.15 scrub starts 2024-08-17T17:41:31.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:30 smithi003 ceph-mon[103344]: 1.18 scrub starts 2024-08-17T17:41:31.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:30 smithi003 ceph-mon[103344]: 1.b scrub starts 2024-08-17T17:41:31.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:30 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 15548/72714 objects degraded (21.382%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:41:31.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:30 smithi003 ceph-mon[103344]: 1.15 scrub starts 2024-08-17T17:41:31.025 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:31.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:31.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:31.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:31.167 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:31.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:31.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:31.269 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:31.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:31.297 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:31.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:31.398 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:31.423 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:31.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:31.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:31.525 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:31.551 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:31.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:31.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:31.653 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:31.679 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:31.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:31.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:31.781 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:31.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:31.831 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:31.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:31.858 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:31 smithi003 ceph-mon[100615]: pgmap v1121: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 10 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.3 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 755 op/s; 15548/65832 objects degraded (23.618%); 691/65832 objects misplaced (1.050%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T17:41:31.858 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:31 smithi003 ceph-mon[100615]: 1.c scrub starts 2024-08-17T17:41:31.858 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:31 smithi003 ceph-mon[103344]: pgmap v1121: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 10 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.3 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 755 op/s; 15548/65832 objects degraded (23.618%); 691/65832 objects misplaced (1.050%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T17:41:31.858 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:31 smithi003 ceph-mon[103344]: 1.c scrub starts 2024-08-17T17:41:31.932 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:31.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:31 smithi053 ceph-mon[89963]: pgmap v1121: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 10 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.3 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 755 op/s; 15548/65832 objects degraded (23.618%); 691/65832 objects misplaced (1.050%); 1.7 MiB/s, 27 objects/s recovering 2024-08-17T17:41:31.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:31 smithi053 ceph-mon[89963]: 1.c scrub starts 2024-08-17T17:41:31.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:31.980 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:31.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:32.081 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:32.121 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:32.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:32.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:32.223 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:32.249 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:32.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:32.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:32.351 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:32.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:32.394 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:32.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:32.495 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:32.518 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:32.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:32.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:32.620 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:32.666 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:32.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:32.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:32.702 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:32 smithi003 ceph-mon[103344]: 1.1c scrub starts 2024-08-17T17:41:32.703 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:32 smithi003 ceph-mon[100615]: 1.1c scrub starts 2024-08-17T17:41:32.768 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:32.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:32.807 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:32.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:32.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:32.961 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:32.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:32.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:32.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:32 smithi053 ceph-mon[89963]: 1.1c scrub starts 2024-08-17T17:41:33.063 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:33.087 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:33.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:33.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:33.189 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:33.237 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:33.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:33.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:33.339 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:33.402 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:33.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:33.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:33.504 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:33.528 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:33.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:33.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:33.585 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:33 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-a[100611]: 2024-08-17T17:41:33.461+0000 7fc4d42c5640 -1 mon.a@0(leader).osd e342 definitely_dead 0 2024-08-17T17:41:33.593 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:41:33.619 INFO:teuthology.orchestra.run.smithi003.stderr:marked in osd.4. 2024-08-17T17:41:33.630 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:33.653 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:33.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:33.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:33.654 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:41:33.654 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:41:33.754 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:33.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:33.777 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:33.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:33.878 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:33.898 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:33 smithi003 ceph-mon[103344]: pgmap v1122: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 10 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.2 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 789 op/s; 15642/60285 objects degraded (25.947%); 424/60285 objects misplaced (0.703%); 2.4 MiB/s, 37 objects/s recovering 2024-08-17T17:41:33.898 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:33 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1861851927' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-08-17T17:41:33.898 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:33 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-08-17T17:41:33.898 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:33 smithi003 ceph-mon[100615]: pgmap v1122: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 10 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.2 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 789 op/s; 15642/60285 objects degraded (25.947%); 424/60285 objects misplaced (0.703%); 2.4 MiB/s, 37 objects/s recovering 2024-08-17T17:41:33.898 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:33 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1861851927' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-08-17T17:41:33.898 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:33 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-08-17T17:41:33.906 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:33.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:33.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:33.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:33 smithi053 ceph-mon[89963]: pgmap v1122: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 10 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 106 active+clean; 1.2 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 789 op/s; 15642/60285 objects degraded (25.947%); 424/60285 objects misplaced (0.703%); 2.4 MiB/s, 37 objects/s recovering 2024-08-17T17:41:33.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:33 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1861851927' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-08-17T17:41:33.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:33 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-08-17T17:41:34.008 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:34.031 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:34.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:34.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:34.132 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:34.155 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:34.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:34.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:34.257 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:34.279 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:34.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:34.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:34.380 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:34.403 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:34.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:34.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:34.505 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:34.527 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:34.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:34.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:34.630 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:34.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:34.668 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:34.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:34.769 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:34.769 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:41:34.770 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:41:34.786 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:41:34.786 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:41:34.791 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:34.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:34.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:34.803 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:41:34.803 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:41:34.819 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:41:34.820 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:41:34.836 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:41:34.836 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:41:34.852 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:41:34.852 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:41:34.868 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:41:34.868 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:41:34.884 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:41:34.884 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:41:34.893 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:34.893 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:34 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-08-17T17:41:34.894 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:34 smithi003 ceph-mon[103344]: osdmap e343: 8 total, 8 up, 8 in 2024-08-17T17:41:34.894 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:34 smithi003 ceph-mon[103344]: osdmap e344: 8 total, 8 up, 8 in 2024-08-17T17:41:34.894 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:34 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-08-17T17:41:34.894 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:34 smithi003 ceph-mon[100615]: osdmap e343: 8 total, 8 up, 8 in 2024-08-17T17:41:34.894 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:34 smithi003 ceph-mon[100615]: osdmap e344: 8 total, 8 up, 8 in 2024-08-17T17:41:34.916 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:34.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:34.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:34.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:34 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-08-17T17:41:34.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:34 smithi053 ceph-mon[89963]: osdmap e343: 8 total, 8 up, 8 in 2024-08-17T17:41:34.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:34 smithi053 ceph-mon[89963]: osdmap e344: 8 total, 8 up, 8 in 2024-08-17T17:41:35.018 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:35.041 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:35.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:35.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:35.143 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:35.166 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:35.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:35.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:35.268 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:35.290 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:35.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:35.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:35.392 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:35.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:35.416 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:35.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:35.517 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:35.540 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:35.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:35.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:35.642 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:35.669 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:35.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:35.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:35.771 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:35.793 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:35.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:35.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:35.895 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:35.921 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:35.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:35.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:35.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:35 smithi053 ceph-mon[89963]: pgmap v1124: 119 pgs: 26 peering, 1 active+recovering+undersized+degraded+remapped, 10 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 771 op/s; 15642/59553 objects degraded (26.266%); 424/59553 objects misplaced (0.712%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T17:41:35.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:35 smithi053 ceph-mon[89963]: 1.8 scrub starts 2024-08-17T17:41:35.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:35 smithi053 ceph-mon[89963]: 1.1b scrub starts 2024-08-17T17:41:35.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:35 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1849054143' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:41:35.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:35 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 15642/59553 objects degraded (26.266%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:41:35.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:35 smithi053 ceph-mon[89963]: osdmap e345: 8 total, 8 up, 8 in 2024-08-17T17:41:36.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:35 smithi003 ceph-mon[103344]: pgmap v1124: 119 pgs: 26 peering, 1 active+recovering+undersized+degraded+remapped, 10 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 771 op/s; 15642/59553 objects degraded (26.266%); 424/59553 objects misplaced (0.712%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T17:41:36.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:35 smithi003 ceph-mon[103344]: 1.8 scrub starts 2024-08-17T17:41:36.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:35 smithi003 ceph-mon[103344]: 1.1b scrub starts 2024-08-17T17:41:36.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:35 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1849054143' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:41:36.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:35 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 15642/59553 objects degraded (26.266%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:41:36.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:35 smithi003 ceph-mon[103344]: osdmap e345: 8 total, 8 up, 8 in 2024-08-17T17:41:36.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:35 smithi003 ceph-mon[100615]: pgmap v1124: 119 pgs: 26 peering, 1 active+recovering+undersized+degraded+remapped, 10 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 80 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 771 op/s; 15642/59553 objects degraded (26.266%); 424/59553 objects misplaced (0.712%); 1.9 MiB/s, 29 objects/s recovering 2024-08-17T17:41:36.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:35 smithi003 ceph-mon[100615]: 1.8 scrub starts 2024-08-17T17:41:36.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:35 smithi003 ceph-mon[100615]: 1.1b scrub starts 2024-08-17T17:41:36.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:35 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1849054143' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:41:36.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:35 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 15642/59553 objects degraded (26.266%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-08-17T17:41:36.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:35 smithi003 ceph-mon[100615]: osdmap e345: 8 total, 8 up, 8 in 2024-08-17T17:41:36.024 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:41:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:41:35] "GET /metrics HTTP/1.1" 200 36523 "" "Prometheus/2.51.0" 2024-08-17T17:41:36.024 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:36.051 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:36.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:36.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:36.153 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:36.181 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:36.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:36.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:36.283 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:36.309 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:36.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:36.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:36.411 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:36.445 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:36.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:36.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:36.547 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:36.588 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:36.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:36.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:36.690 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:36.732 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:36.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:36.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:36.834 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:36.856 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:36.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:36.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:36.958 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:36.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:36 smithi053 ceph-mon[89963]: 1.18 scrub starts 2024-08-17T17:41:36.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:36 smithi053 ceph-mon[89963]: 9.e starting backfill to osd.4 from (0'0,325'6543] MAX to 342'13541 2024-08-17T17:41:36.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:36 smithi053 ceph-mon[89963]: 9.c starting backfill to osd.4 from (0'0,325'6477] MAX to 342'13418 2024-08-17T17:41:36.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:36 smithi053 ceph-mon[89963]: 9.4 starting backfill to osd.4 from (0'0,325'6492] MAX to 342'13447 2024-08-17T17:41:36.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:36 smithi053 ceph-mon[89963]: 1.b scrub starts 2024-08-17T17:41:36.994 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:36.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:36.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:37.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:36 smithi003 ceph-mon[103344]: 1.18 scrub starts 2024-08-17T17:41:37.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:36 smithi003 ceph-mon[103344]: 9.e starting backfill to osd.4 from (0'0,325'6543] MAX to 342'13541 2024-08-17T17:41:37.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:36 smithi003 ceph-mon[103344]: 9.c starting backfill to osd.4 from (0'0,325'6477] MAX to 342'13418 2024-08-17T17:41:37.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:36 smithi003 ceph-mon[103344]: 9.4 starting backfill to osd.4 from (0'0,325'6492] MAX to 342'13447 2024-08-17T17:41:37.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:36 smithi003 ceph-mon[103344]: 1.b scrub starts 2024-08-17T17:41:37.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:36 smithi003 ceph-mon[100615]: 1.18 scrub starts 2024-08-17T17:41:37.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:36 smithi003 ceph-mon[100615]: 9.e starting backfill to osd.4 from (0'0,325'6543] MAX to 342'13541 2024-08-17T17:41:37.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:36 smithi003 ceph-mon[100615]: 9.c starting backfill to osd.4 from (0'0,325'6477] MAX to 342'13418 2024-08-17T17:41:37.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:36 smithi003 ceph-mon[100615]: 9.4 starting backfill to osd.4 from (0'0,325'6492] MAX to 342'13447 2024-08-17T17:41:37.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:36 smithi003 ceph-mon[100615]: 1.b scrub starts 2024-08-17T17:41:37.096 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:37.123 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:37.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:37.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:37.224 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:37.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:37.270 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:37.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:37.307 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-08-17T17:41:37.371 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:37.412 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:37.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:37.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:37.514 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:37.542 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:37.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:37.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:37.644 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:37.673 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:37.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:37.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:37.775 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:37.801 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:37.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:37.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:37.903 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:37.932 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:37.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:37.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:37.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:37 smithi053 ceph-mon[89963]: pgmap v1127: 119 pgs: 1 activating+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 3 active+undersized+remapped, 7 remapped+peering, 28 peering, 2 active+clean+scrubbing, 77 active+clean; 1.1 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 783 op/s; 2135/51621 objects degraded (4.136%); 2.7 MiB/s, 44 objects/s recovering 2024-08-17T17:41:37.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:37 smithi053 ceph-mon[89963]: 1.c scrub starts 2024-08-17T17:41:37.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:37 smithi053 ceph-mon[89963]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-08-17T17:41:37.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:37 smithi053 ceph-mon[89963]: osdmap e346: 8 total, 8 up, 8 in 2024-08-17T17:41:38.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:37 smithi003 ceph-mon[103344]: pgmap v1127: 119 pgs: 1 activating+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 3 active+undersized+remapped, 7 remapped+peering, 28 peering, 2 active+clean+scrubbing, 77 active+clean; 1.1 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 783 op/s; 2135/51621 objects degraded (4.136%); 2.7 MiB/s, 44 objects/s recovering 2024-08-17T17:41:38.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:37 smithi003 ceph-mon[103344]: 1.c scrub starts 2024-08-17T17:41:38.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:37 smithi003 ceph-mon[103344]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-08-17T17:41:38.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:37 smithi003 ceph-mon[103344]: osdmap e346: 8 total, 8 up, 8 in 2024-08-17T17:41:38.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:37 smithi003 ceph-mon[100615]: pgmap v1127: 119 pgs: 1 activating+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 3 active+undersized+remapped, 7 remapped+peering, 28 peering, 2 active+clean+scrubbing, 77 active+clean; 1.1 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 783 op/s; 2135/51621 objects degraded (4.136%); 2.7 MiB/s, 44 objects/s recovering 2024-08-17T17:41:38.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:37 smithi003 ceph-mon[100615]: 1.c scrub starts 2024-08-17T17:41:38.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:37 smithi003 ceph-mon[100615]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-08-17T17:41:38.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:37 smithi003 ceph-mon[100615]: osdmap e346: 8 total, 8 up, 8 in 2024-08-17T17:41:38.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:38.071 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:38.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:38.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:38.172 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:38.211 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:38.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:38.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:38.313 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:38.338 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:38.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:38.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:38.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:38.480 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:38.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:38.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:38.581 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:38.617 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:38.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:38.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:38.718 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:38.758 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:38.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:38.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:38.769 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-17T17:41:38.861 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:38.885 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:38.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:38.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:38.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:38 smithi053 ceph-mon[89963]: 1.1c scrub starts 2024-08-17T17:41:38.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:38 smithi053 ceph-mon[89963]: osdmap e347: 8 total, 8 up, 8 in 2024-08-17T17:41:38.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:41:38.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:41:38.987 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:39.010 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:39.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:39.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:39.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:38 smithi003 ceph-mon[103344]: 1.1c scrub starts 2024-08-17T17:41:39.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:38 smithi003 ceph-mon[103344]: osdmap e347: 8 total, 8 up, 8 in 2024-08-17T17:41:39.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:41:39.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:41:39.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:38 smithi003 ceph-mon[100615]: 1.1c scrub starts 2024-08-17T17:41:39.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:38 smithi003 ceph-mon[100615]: osdmap e347: 8 total, 8 up, 8 in 2024-08-17T17:41:39.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:41:39.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:41:39.112 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:39.137 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:39.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:39.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:39.239 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:39.286 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:39.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:39.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:39.388 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:39.424 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:39.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:39.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:39.526 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:39.550 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:39.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:39.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:39.652 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:39.677 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:39.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:39.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:39.722 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:39 smithi003 ceph-mon[103344]: pgmap v1130: 119 pgs: 1 activating+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 3 active+undersized+remapped, 7 remapped+peering, 28 peering, 2 active+clean+scrubbing, 77 active+clean; 1.1 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 660 op/s; 2135/51621 objects degraded (4.136%); 1.4 MiB/s, 26 objects/s recovering 2024-08-17T17:41:39.722 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:39 smithi003 ceph-mon[103344]: 1.9 scrub starts 2024-08-17T17:41:39.723 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:39 smithi003 ceph-mon[100615]: pgmap v1130: 119 pgs: 1 activating+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 3 active+undersized+remapped, 7 remapped+peering, 28 peering, 2 active+clean+scrubbing, 77 active+clean; 1.1 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 660 op/s; 2135/51621 objects degraded (4.136%); 1.4 MiB/s, 26 objects/s recovering 2024-08-17T17:41:39.723 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:39 smithi003 ceph-mon[100615]: 1.9 scrub starts 2024-08-17T17:41:39.779 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:39.803 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:39.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:39.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:39.806 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:41:39.904 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:39.930 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:39.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:39.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:39.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:39 smithi053 ceph-mon[89963]: pgmap v1130: 119 pgs: 1 activating+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 3 active+undersized+remapped, 7 remapped+peering, 28 peering, 2 active+clean+scrubbing, 77 active+clean; 1.1 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 660 op/s; 2135/51621 objects degraded (4.136%); 1.4 MiB/s, 26 objects/s recovering 2024-08-17T17:41:39.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:39 smithi053 ceph-mon[89963]: 1.9 scrub starts 2024-08-17T17:41:40.032 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:40.070 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:40.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:40.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:40.172 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:40.222 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:40.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:40.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:40.323 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:40.367 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:40.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:40.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:40.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:41:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:41:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:41:40.468 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:40.496 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:40.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:40.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:40.598 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:40.625 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:40.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:40.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:40.684 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-08-17T17:41:40.726 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:40.754 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:40.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:40.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:40.855 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:40.881 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:40.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:40.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:40.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:40 smithi053 ceph-mon[89963]: 1.19 scrub starts 2024-08-17T17:41:40.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:40 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 2135/51621 objects degraded (4.136%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:41:40.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:40 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2328242091' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:41:40.983 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:41.008 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:40 smithi003 ceph-mon[103344]: 1.19 scrub starts 2024-08-17T17:41:41.008 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:40 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 2135/51621 objects degraded (4.136%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:41:41.008 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:40 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2328242091' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:41:41.008 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:40 smithi003 ceph-mon[100615]: 1.19 scrub starts 2024-08-17T17:41:41.008 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:40 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 2135/51621 objects degraded (4.136%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:41:41.009 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:40 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2328242091' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:41:41.009 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:41.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:41.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:41.112 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:41.139 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:41.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:41.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:41.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:41:41.241 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:41.264 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:41.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:41.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:41.366 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:41.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:41.415 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:41.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:41.516 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:41.541 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:41.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:41.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:41.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:41.691 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:41.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:41.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:41.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:41 smithi053 ceph-mon[89963]: pgmap v1131: 119 pgs: 1 activating+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 3 active+undersized+remapped, 1 remapped+peering, 3 active+clean+scrubbing, 109 active+clean; 990 MiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 3.3 KiB/s wr, 532 op/s; 3087/47391 objects degraded (6.514%); 2.8 MiB/s, 1 keys/s, 52 objects/s recovering 2024-08-17T17:41:41.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:41 smithi053 ceph-mon[89963]: 1.8 scrub starts 2024-08-17T17:41:41.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:41 smithi053 ceph-mon[89963]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-08-17T17:41:41.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:41 smithi053 ceph-mon[89963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:41:41.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:41 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2328242091' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:41:41.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:41 smithi053 ceph-mon[89963]: osdmap e348: 8 total, 8 up, 8 in 2024-08-17T17:41:41.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:41 smithi053 ceph-mon[89963]: 1.1b scrub starts 2024-08-17T17:41:41.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:41 smithi053 ceph-mon[89963]: 1.1b scrub ok 2024-08-17T17:41:41.792 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:41.824 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:41.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:41.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:41.866 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:41 smithi003 ceph-mon[103344]: pgmap v1131: 119 pgs: 1 activating+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 3 active+undersized+remapped, 1 remapped+peering, 3 active+clean+scrubbing, 109 active+clean; 990 MiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 3.3 KiB/s wr, 532 op/s; 3087/47391 objects degraded (6.514%); 2.8 MiB/s, 1 keys/s, 52 objects/s recovering 2024-08-17T17:41:41.867 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:41 smithi003 ceph-mon[103344]: 1.8 scrub starts 2024-08-17T17:41:41.867 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:41 smithi003 ceph-mon[103344]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-08-17T17:41:41.867 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:41 smithi003 ceph-mon[103344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:41:41.867 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:41 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2328242091' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:41:41.867 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:41 smithi003 ceph-mon[103344]: osdmap e348: 8 total, 8 up, 8 in 2024-08-17T17:41:41.867 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:41 smithi003 ceph-mon[103344]: 1.1b scrub starts 2024-08-17T17:41:41.867 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:41 smithi003 ceph-mon[103344]: 1.1b scrub ok 2024-08-17T17:41:41.867 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:41 smithi003 ceph-mon[100615]: pgmap v1131: 119 pgs: 1 activating+degraded+remapped, 2 active+recovery_wait+degraded+remapped, 3 active+undersized+remapped, 1 remapped+peering, 3 active+clean+scrubbing, 109 active+clean; 990 MiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 3.3 KiB/s wr, 532 op/s; 3087/47391 objects degraded (6.514%); 2.8 MiB/s, 1 keys/s, 52 objects/s recovering 2024-08-17T17:41:41.867 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:41 smithi003 ceph-mon[100615]: 1.8 scrub starts 2024-08-17T17:41:41.868 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:41 smithi003 ceph-mon[100615]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-08-17T17:41:41.868 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:41 smithi003 ceph-mon[100615]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:41:41.868 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:41 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2328242091' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:41:41.868 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:41 smithi003 ceph-mon[100615]: osdmap e348: 8 total, 8 up, 8 in 2024-08-17T17:41:41.868 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:41 smithi003 ceph-mon[100615]: 1.1b scrub starts 2024-08-17T17:41:41.868 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:41 smithi003 ceph-mon[100615]: 1.1b scrub ok 2024-08-17T17:41:41.927 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:41.952 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:41.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:41.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:42.053 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:42.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:42.091 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:42.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:42.193 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:42.219 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:42.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:42.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:42.308 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-08-17T17:41:42.308 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:41:42.308 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-08-17T17:41:42.308 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-17T17:41:42.308 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-17T17:41:42.309 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-08-17T17:41:42.320 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:42.342 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:42.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:42.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:42.444 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:42.494 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:42.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:42.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:42.597 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:42.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:42.638 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:42.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:42.739 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:42.779 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:42.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:42.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:42.881 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:42.906 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:42.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:42.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:42.960 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:42 smithi003 ceph-mon[103344]: 1.a scrub starts 2024-08-17T17:41:42.960 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:42 smithi003 ceph-mon[103344]: 1.a scrub ok 2024-08-17T17:41:42.960 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:42 smithi003 ceph-mon[103344]: 1.18 scrub starts 2024-08-17T17:41:42.960 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:42 smithi003 ceph-mon[103344]: 1.18 scrub ok 2024-08-17T17:41:42.960 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:42 smithi003 ceph-mon[103344]: 1.b scrub starts 2024-08-17T17:41:42.960 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:42 smithi003 ceph-mon[103344]: 1.b scrub ok 2024-08-17T17:41:42.961 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:42 smithi003 ceph-mon[100615]: 1.a scrub starts 2024-08-17T17:41:42.961 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:42 smithi003 ceph-mon[100615]: 1.a scrub ok 2024-08-17T17:41:42.961 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:42 smithi003 ceph-mon[100615]: 1.18 scrub starts 2024-08-17T17:41:42.961 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:42 smithi003 ceph-mon[100615]: 1.18 scrub ok 2024-08-17T17:41:42.961 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:42 smithi003 ceph-mon[100615]: 1.b scrub starts 2024-08-17T17:41:42.961 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:42 smithi003 ceph-mon[100615]: 1.b scrub ok 2024-08-17T17:41:42.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:42 smithi053 ceph-mon[89963]: 1.a scrub starts 2024-08-17T17:41:42.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:42 smithi053 ceph-mon[89963]: 1.a scrub ok 2024-08-17T17:41:42.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:42 smithi053 ceph-mon[89963]: 1.18 scrub starts 2024-08-17T17:41:42.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:42 smithi053 ceph-mon[89963]: 1.18 scrub ok 2024-08-17T17:41:42.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:42 smithi053 ceph-mon[89963]: 1.b scrub starts 2024-08-17T17:41:42.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:42 smithi053 ceph-mon[89963]: 1.b scrub ok 2024-08-17T17:41:43.008 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:43.034 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:43.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:43.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:43.135 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:43.161 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:43.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:43.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:43.263 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:43.341 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:43.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:43.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:43.447 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:43.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:43.498 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:43.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:43.589 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:41:43.589 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-08-17T17:41:43.589 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-08-17T17:41:43.589 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:41:43.589 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.590 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:68824 8.e 8:7b9f8998:::benchmark_data_smithi003_229103_object68823:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-08-17T17:41:43.590 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:35.002608+0000", 2024-08-17T17:41:43.590 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 488.58170084400001, 2024-08-17T17:41:43.590 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.20794159600000001, 2024-08-17T17:41:43.590 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:43.590 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:43.590 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:41:43.590 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:41:43.590 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:41:43.590 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:41:43.590 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68824 2024-08-17T17:41:43.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.590 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:43.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.591 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:43.591 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.002608+0000", 2024-08-17T17:41:43.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.591 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:43.591 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.002608+0000", 2024-08-17T17:41:43.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.591 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:43.591 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.002610+0000", 2024-08-17T17:41:43.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2019999999999998e-06 2024-08-17T17:41:43.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:43.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.002634+0000", 2024-08-17T17:41:43.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4655e-05 2024-08-17T17:41:43.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:43.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.002651+0000", 2024-08-17T17:41:43.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6263999999999999e-05 2024-08-17T17:41:43.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:43.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.002658+0000", 2024-08-17T17:41:43.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4379999999999998e-06 2024-08-17T17:41:43.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.593 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:43.593 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.003672+0000", 2024-08-17T17:41:43.593 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010134689999999999 2024-08-17T17:41:43.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.593 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:43.593 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.003991+0000", 2024-08-17T17:41:43.593 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00031916099999999999 2024-08-17T17:41:43.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.594 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,4", 2024-08-17T17:41:43.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.004415+0000", 2024-08-17T17:41:43.594 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00042393999999999998 2024-08-17T17:41:43.594 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.594 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:41:43.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.008823+0000", 2024-08-17T17:41:43.594 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0044082519999999997 2024-08-17T17:41:43.594 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.594 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:43.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.008939+0000", 2024-08-17T17:41:43.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011628099999999999 2024-08-17T17:41:43.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.595 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:43.595 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.210387+0000", 2024-08-17T17:41:43.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.201447337 2024-08-17T17:41:43.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.595 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:41:43.595 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.210500+0000", 2024-08-17T17:41:43.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000113132 2024-08-17T17:41:43.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.595 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:43.596 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.210549+0000", 2024-08-17T17:41:43.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9465000000000002e-05 2024-08-17T17:41:43.596 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.596 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:43.596 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.596 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.596 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:68837 8.e 8:75f2384d:::benchmark_data_smithi003_229103_object68836:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-08-17T17:41:43.596 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:35.016337+0000", 2024-08-17T17:41:43.596 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 488.56797089899999, 2024-08-17T17:41:43.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.201609117, 2024-08-17T17:41:43.596 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:43.596 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:43.596 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:41:43.597 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:41:43.597 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:41:43.597 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:41:43.597 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68837 2024-08-17T17:41:43.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.597 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:43.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:43.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.016337+0000", 2024-08-17T17:41:43.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:43.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.016337+0000", 2024-08-17T17:41:43.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:43.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.016339+0000", 2024-08-17T17:41:43.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8169999999999999e-06 2024-08-17T17:41:43.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:43.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.016364+0000", 2024-08-17T17:41:43.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4511999999999998e-05 2024-08-17T17:41:43.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:43.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.016381+0000", 2024-08-17T17:41:43.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7178999999999999e-05 2024-08-17T17:41:43.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:43.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.016390+0000", 2024-08-17T17:41:43.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6379999999999993e-06 2024-08-17T17:41:43.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:43.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.016583+0000", 2024-08-17T17:41:43.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000193492 2024-08-17T17:41:43.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:43.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.016951+0000", 2024-08-17T17:41:43.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00036739500000000002 2024-08-17T17:41:43.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.600 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,4", 2024-08-17T17:41:43.600 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.017382+0000", 2024-08-17T17:41:43.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00043126500000000001 2024-08-17T17:41:43.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.600 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:43.600 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.020333+0000", 2024-08-17T17:41:43.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029511870000000001 2024-08-17T17:41:43.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.600 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:41:43.600 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.022872+0000", 2024-08-17T17:41:43.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025392710000000001 2024-08-17T17:41:43.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:43.601 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.217453+0000", 2024-08-17T17:41:43.601 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.19458077100000001 2024-08-17T17:41:43.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:41:43.601 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.217844+0000", 2024-08-17T17:41:43.601 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00039106600000000003 2024-08-17T17:41:43.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:43.601 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.217947+0000", 2024-08-17T17:41:43.601 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000102524 2024-08-17T17:41:43.602 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.602 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:43.602 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.602 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:68841 8.1 8:8211cd02:::benchmark_data_smithi003_229103_object68840:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-08-17T17:41:43.602 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:35.022604+0000", 2024-08-17T17:41:43.602 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 488.56170442600001, 2024-08-17T17:41:43.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.19225920299999999, 2024-08-17T17:41:43.602 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:43.602 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:43.602 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:41:43.602 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:41:43.602 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:41:43.602 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:41:43.603 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68841 2024-08-17T17:41:43.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.603 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:43.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.603 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:43.603 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.022604+0000", 2024-08-17T17:41:43.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.603 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:43.603 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.022604+0000", 2024-08-17T17:41:43.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:43.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.022606+0000", 2024-08-17T17:41:43.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7069999999999999e-06 2024-08-17T17:41:43.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:43.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.022631+0000", 2024-08-17T17:41:43.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5182999999999999e-05 2024-08-17T17:41:43.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:43.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.022648+0000", 2024-08-17T17:41:43.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7595999999999999e-05 2024-08-17T17:41:43.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.605 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:43.605 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.022658+0000", 2024-08-17T17:41:43.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6609999999999995e-06 2024-08-17T17:41:43.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.605 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:43.605 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.022739+0000", 2024-08-17T17:41:43.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0934999999999997e-05 2024-08-17T17:41:43.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.605 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:43.605 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.022946+0000", 2024-08-17T17:41:43.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020672799999999999 2024-08-17T17:41:43.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:41:43.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.023091+0000", 2024-08-17T17:41:43.606 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000145076 2024-08-17T17:41:43.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:41:43.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.028542+0000", 2024-08-17T17:41:43.606 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0054511430000000003 2024-08-17T17:41:43.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:43.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.028761+0000", 2024-08-17T17:41:43.606 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00021915199999999999 2024-08-17T17:41:43.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.607 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:43.607 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.212014+0000", 2024-08-17T17:41:43.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18325253999999999 2024-08-17T17:41:43.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.607 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:41:43.607 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.212076+0000", 2024-08-17T17:41:43.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2249999999999995e-05 2024-08-17T17:41:43.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.607 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:43.607 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.214863+0000", 2024-08-17T17:41:43.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027872320000000002 2024-08-17T17:41:43.608 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.608 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:43.608 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.608 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:68845 8.e 8:74722ba7:::benchmark_data_smithi003_229103_object68844:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-08-17T17:41:43.608 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:35.025985+0000", 2024-08-17T17:41:43.608 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 488.55832297299997, 2024-08-17T17:41:43.608 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.19305075899999999, 2024-08-17T17:41:43.608 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:43.608 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:43.608 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:41:43.608 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:41:43.608 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:41:43.608 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:41:43.609 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68845 2024-08-17T17:41:43.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.609 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:43.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.609 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:43.609 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.025985+0000", 2024-08-17T17:41:43.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.609 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:43.609 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.025985+0000", 2024-08-17T17:41:43.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.609 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:43.610 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.025987+0000", 2024-08-17T17:41:43.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.852e-06 2024-08-17T17:41:43.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.610 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:43.610 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.026013+0000", 2024-08-17T17:41:43.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5378e-05 2024-08-17T17:41:43.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.610 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:43.610 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.026024+0000", 2024-08-17T17:41:43.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1513e-05 2024-08-17T17:41:43.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:43.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.026032+0000", 2024-08-17T17:41:43.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8970000000000008e-06 2024-08-17T17:41:43.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:43.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.026238+0000", 2024-08-17T17:41:43.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020603900000000001 2024-08-17T17:41:43.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:43.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.026586+0000", 2024-08-17T17:41:43.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00034832800000000002 2024-08-17T17:41:43.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,4", 2024-08-17T17:41:43.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.027014+0000", 2024-08-17T17:41:43.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000427911 2024-08-17T17:41:43.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:43.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.030291+0000", 2024-08-17T17:41:43.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003276532 2024-08-17T17:41:43.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:41:43.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.030460+0000", 2024-08-17T17:41:43.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000169502 2024-08-17T17:41:43.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.613 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:43.613 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.217987+0000", 2024-08-17T17:41:43.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18752703900000001 2024-08-17T17:41:43.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.613 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:41:43.613 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.218983+0000", 2024-08-17T17:41:43.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00099567300000000009 2024-08-17T17:41:43.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.613 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:43.613 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.219036+0000", 2024-08-17T17:41:43.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3094999999999999e-05 2024-08-17T17:41:43.613 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.613 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:43.614 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.614 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36354.0:136294 9.c e333/327 9:30c5d0fb:::benchmark_data_smithi003_261051_object20409:head v 333'8570, mlcod=0'0)", 2024-08-17T17:41:43.614 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:39:48.671514+0000", 2024-08-17T17:41:43.614 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 114.91279476, 2024-08-17T17:41:43.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.226340771, 2024-08-17T17:41:43.614 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:43.614 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:43.614 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:41:43.614 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:43.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.614 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:43.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.671514+0000", 2024-08-17T17:41:43.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:43.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.671514+0000", 2024-08-17T17:41:43.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:43.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.671515+0000", 2024-08-17T17:41:43.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1340000000000001e-06 2024-08-17T17:41:43.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:43.616 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:43.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.671520+0000", 2024-08-17T17:41:43.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5659999999999997e-06 2024-08-17T17:41:43.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:43.617 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.671534+0000", 2024-08-17T17:41:43.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4073999999999999e-05 2024-08-17T17:41:43.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:43.617 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.671541+0000", 2024-08-17T17:41:43.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7329999999999997e-06 2024-08-17T17:41:43.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:43.618 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.685805+0000", 2024-08-17T17:41:43.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.014263868000000001 2024-08-17T17:41:43.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.618 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:43.618 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.685834+0000", 2024-08-17T17:41:43.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9402000000000001e-05 2024-08-17T17:41:43.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.618 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:41:43.618 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.897792+0000", 2024-08-17T17:41:43.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.21195795000000001 2024-08-17T17:41:43.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:43.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.897854+0000", 2024-08-17T17:41:43.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2044000000000004e-05 2024-08-17T17:41:43.619 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.619 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:43.619 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.619 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36354.0:136300 9.4 e333/327 9:26d8721f:::benchmark_data_smithi003_261051_object20415:head v 333'8561, mlcod=0'0)", 2024-08-17T17:41:43.619 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:39:48.683855+0000", 2024-08-17T17:41:43.619 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 114.900453453, 2024-08-17T17:41:43.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.22357099899999999, 2024-08-17T17:41:43.619 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:43.619 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:43.619 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:41:43.620 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:43.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:43.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.683855+0000", 2024-08-17T17:41:43.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:43.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.683855+0000", 2024-08-17T17:41:43.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:43.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.683856+0000", 2024-08-17T17:41:43.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5460000000000001e-06 2024-08-17T17:41:43.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:43.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.683867+0000", 2024-08-17T17:41:43.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0241000000000001e-05 2024-08-17T17:41:43.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:43.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.683884+0000", 2024-08-17T17:41:43.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7085e-05 2024-08-17T17:41:43.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:43.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.683893+0000", 2024-08-17T17:41:43.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7059999999999999e-06 2024-08-17T17:41:43.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.622 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:43.622 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.687919+0000", 2024-08-17T17:41:43.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040253349999999997 2024-08-17T17:41:43.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.622 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:43.622 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.687940+0000", 2024-08-17T17:41:43.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1529999999999999e-05 2024-08-17T17:41:43.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.622 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:41:43.623 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.906782+0000", 2024-08-17T17:41:43.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.218841594 2024-08-17T17:41:43.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.623 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:43.623 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.907426+0000", 2024-08-17T17:41:43.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00064396199999999998 2024-08-17T17:41:43.623 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.623 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:43.623 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.623 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36354.0:136310 9.c e333/327 9:3700bf26:::benchmark_data_smithi003_261051_object20425:head v 333'8571, mlcod=0'0)", 2024-08-17T17:41:43.623 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:39:48.691749+0000", 2024-08-17T17:41:43.624 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 114.892559362, 2024-08-17T17:41:43.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.22337359300000001, 2024-08-17T17:41:43.624 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:43.624 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:43.624 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:41:43.624 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:43.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:43.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.691749+0000", 2024-08-17T17:41:43.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:43.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.691749+0000", 2024-08-17T17:41:43.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.625 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:43.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.691750+0000", 2024-08-17T17:41:43.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5799999999999999e-07 2024-08-17T17:41:43.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.625 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:43.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.691754+0000", 2024-08-17T17:41:43.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5140000000000003e-06 2024-08-17T17:41:43.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.625 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:43.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.691763+0000", 2024-08-17T17:41:43.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0860000000000008e-06 2024-08-17T17:41:43.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:43.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.691769+0000", 2024-08-17T17:41:43.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1020000000000002e-06 2024-08-17T17:41:43.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:43.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.696302+0000", 2024-08-17T17:41:43.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0045328649999999996 2024-08-17T17:41:43.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:43.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.696324+0000", 2024-08-17T17:41:43.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1804999999999999e-05 2024-08-17T17:41:43.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:41:43.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.915080+0000", 2024-08-17T17:41:43.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.21875614299999999 2024-08-17T17:41:43.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:43.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.915123+0000", 2024-08-17T17:41:43.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2419999999999997e-05 2024-08-17T17:41:43.627 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.627 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:43.627 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.627 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36354.0:136315 9.c e333/327 9:39cb2799:::benchmark_data_smithi003_261051_object20430:head v 333'8572, mlcod=0'0)", 2024-08-17T17:41:43.627 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:39:48.695981+0000", 2024-08-17T17:41:43.628 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 114.888327172, 2024-08-17T17:41:43.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.22913499700000001, 2024-08-17T17:41:43.628 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:43.628 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:43.628 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:41:43.628 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:43.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:43.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.695981+0000", 2024-08-17T17:41:43.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:43.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.695981+0000", 2024-08-17T17:41:43.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:43.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.695982+0000", 2024-08-17T17:41:43.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2799999999999995e-07 2024-08-17T17:41:43.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:43.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.695999+0000", 2024-08-17T17:41:43.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6631999999999999e-05 2024-08-17T17:41:43.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:43.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.696008+0000", 2024-08-17T17:41:43.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0739999999999999e-06 2024-08-17T17:41:43.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:43.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.696015+0000", 2024-08-17T17:41:43.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9e-06 2024-08-17T17:41:43.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:43.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.704162+0000", 2024-08-17T17:41:43.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0081470169999999995 2024-08-17T17:41:43.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:43.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.704184+0000", 2024-08-17T17:41:43.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2317e-05 2024-08-17T17:41:43.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.631 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:41:43.631 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.925069+0000", 2024-08-17T17:41:43.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.22088473 2024-08-17T17:41:43.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.631 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:43.631 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.925116+0000", 2024-08-17T17:41:43.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7499e-05 2024-08-17T17:41:43.631 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.631 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:43.631 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.632 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36354.0:136311 9.4 e333/327 9:22840a96:::benchmark_data_smithi003_261051_object20426:head v 333'8562, mlcod=0'0)", 2024-08-17T17:41:43.632 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:39:48.700078+0000", 2024-08-17T17:41:43.632 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 114.884230084, 2024-08-17T17:41:43.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.22510350000000001, 2024-08-17T17:41:43.632 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:43.632 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:43.632 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:41:43.632 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:43.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:43.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.700078+0000", 2024-08-17T17:41:43.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:43.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.700078+0000", 2024-08-17T17:41:43.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:43.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.700079+0000", 2024-08-17T17:41:43.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0299999999999998e-07 2024-08-17T17:41:43.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:43.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.700086+0000", 2024-08-17T17:41:43.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2949999999999996e-06 2024-08-17T17:41:43.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:43.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.700096+0000", 2024-08-17T17:41:43.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4579999999999999e-06 2024-08-17T17:41:43.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:43.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.700102+0000", 2024-08-17T17:41:43.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3369999999999996e-06 2024-08-17T17:41:43.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:43.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.704156+0000", 2024-08-17T17:41:43.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040539 2024-08-17T17:41:43.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:43.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.704183+0000", 2024-08-17T17:41:43.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7250000000000002e-05 2024-08-17T17:41:43.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:41:43.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.925119+0000", 2024-08-17T17:41:43.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.22093612700000001 2024-08-17T17:41:43.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:43.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.925182+0000", 2024-08-17T17:41:43.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2329999999999994e-05 2024-08-17T17:41:43.635 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.636 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:43.636 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.636 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36354.0:136316 9.c e333/327 9:32cd8c01:::benchmark_data_smithi003_261051_object20431:head v 333'8573, mlcod=0'0)", 2024-08-17T17:41:43.636 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:39:48.700417+0000", 2024-08-17T17:41:43.636 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 114.88389149, 2024-08-17T17:41:43.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.224797726, 2024-08-17T17:41:43.636 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:43.636 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:43.636 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:41:43.636 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:43.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.636 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:43.636 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.700417+0000", 2024-08-17T17:41:43.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:43.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.700417+0000", 2024-08-17T17:41:43.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:43.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.700418+0000", 2024-08-17T17:41:43.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8700000000000005e-07 2024-08-17T17:41:43.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:43.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.700423+0000", 2024-08-17T17:41:43.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8289999999999996e-06 2024-08-17T17:41:43.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:43.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.700434+0000", 2024-08-17T17:41:43.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0366e-05 2024-08-17T17:41:43.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:43.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.700440+0000", 2024-08-17T17:41:43.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5459999999999997e-06 2024-08-17T17:41:43.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:43.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.891166+0000", 2024-08-17T17:41:43.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.190725849 2024-08-17T17:41:43.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:43.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.891185+0000", 2024-08-17T17:41:43.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8700000000000001e-05 2024-08-17T17:41:43.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:41:43.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.925184+0000", 2024-08-17T17:41:43.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.033999569 2024-08-17T17:41:43.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:43.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.925215+0000", 2024-08-17T17:41:43.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0179999999999999e-05 2024-08-17T17:41:43.640 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.640 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:43.640 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.640 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36354.0:136321 9.5 9:a66fd1dd:::benchmark_data_smithi003_261051_object20436:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-08-17T17:41:43.640 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:39:48.704258+0000", 2024-08-17T17:41:43.640 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 114.880050525, 2024-08-17T17:41:43.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.190828579, 2024-08-17T17:41:43.640 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:43.640 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:43.640 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:41:43.640 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:41:43.641 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36354", 2024-08-17T17:41:43.641 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1330158990", 2024-08-17T17:41:43.641 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 136321 2024-08-17T17:41:43.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.641 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:43.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:43.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.704258+0000", 2024-08-17T17:41:43.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:43.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.704258+0000", 2024-08-17T17:41:43.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:43.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.704260+0000", 2024-08-17T17:41:43.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8819999999999999e-06 2024-08-17T17:41:43.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:43.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.704265+0000", 2024-08-17T17:41:43.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6559999999999997e-06 2024-08-17T17:41:43.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:43.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.704276+0000", 2024-08-17T17:41:43.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0801999999999999e-05 2024-08-17T17:41:43.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:43.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.704283+0000", 2024-08-17T17:41:43.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9199999999999998e-06 2024-08-17T17:41:43.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:43.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.704337+0000", 2024-08-17T17:41:43.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3464999999999997e-05 2024-08-17T17:41:43.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:43.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.704408+0000", 2024-08-17T17:41:43.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1685000000000002e-05 2024-08-17T17:41:43.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-08-17T17:41:43.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.704477+0000", 2024-08-17T17:41:43.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8891000000000002e-05 2024-08-17T17:41:43.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:43.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.708432+0000", 2024-08-17T17:41:43.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039552900000000002 2024-08-17T17:41:43.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:43.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.708466+0000", 2024-08-17T17:41:43.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3099999999999998e-05 2024-08-17T17:41:43.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:41:43.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.894991+0000", 2024-08-17T17:41:43.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.186525153 2024-08-17T17:41:43.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:41:43.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.895042+0000", 2024-08-17T17:41:43.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1063999999999998e-05 2024-08-17T17:41:43.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:43.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.895086+0000", 2024-08-17T17:41:43.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4671e-05 2024-08-17T17:41:43.647 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.647 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:43.647 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.647 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36354.0:136324 9.5 9:a1e7fb12:::benchmark_data_smithi003_261051_object20439:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-08-17T17:41:43.647 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:39:48.707704+0000", 2024-08-17T17:41:43.647 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 114.876604728, 2024-08-17T17:41:43.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.187465886, 2024-08-17T17:41:43.648 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:43.648 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:43.648 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:41:43.648 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:41:43.648 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36354", 2024-08-17T17:41:43.648 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1330158990", 2024-08-17T17:41:43.648 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 136324 2024-08-17T17:41:43.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.648 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:43.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:43.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.707704+0000", 2024-08-17T17:41:43.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:43.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.707704+0000", 2024-08-17T17:41:43.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:43.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.707705+0000", 2024-08-17T17:41:43.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8169999999999999e-06 2024-08-17T17:41:43.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:43.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.707711+0000", 2024-08-17T17:41:43.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4639999999999997e-06 2024-08-17T17:41:43.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:43.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.707724+0000", 2024-08-17T17:41:43.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2826000000000001e-05 2024-08-17T17:41:43.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:43.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.707731+0000", 2024-08-17T17:41:43.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2570000000000001e-06 2024-08-17T17:41:43.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:43.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.708487+0000", 2024-08-17T17:41:43.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00075555000000000001 2024-08-17T17:41:43.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:43.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.708580+0000", 2024-08-17T17:41:43.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3756000000000002e-05 2024-08-17T17:41:43.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-08-17T17:41:43.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.708679+0000", 2024-08-17T17:41:43.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8552000000000001e-05 2024-08-17T17:41:43.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:43.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.888048+0000", 2024-08-17T17:41:43.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.179369631 2024-08-17T17:41:43.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:43.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.891984+0000", 2024-08-17T17:41:43.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039350979999999997 2024-08-17T17:41:43.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:41:43.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.895095+0000", 2024-08-17T17:41:43.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031110930000000001 2024-08-17T17:41:43.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:41:43.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.895137+0000", 2024-08-17T17:41:43.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2271999999999998e-05 2024-08-17T17:41:43.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:43.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.895170+0000", 2024-08-17T17:41:43.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2570000000000002e-05 2024-08-17T17:41:43.653 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.653 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:43.653 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.654 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36354.0:136326 9.4 e333/327 9:223c4187:::benchmark_data_smithi003_261051_object20441:head v 333'8563, mlcod=0'0)", 2024-08-17T17:41:43.654 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:39:48.710952+0000", 2024-08-17T17:41:43.654 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 114.87335684999999, 2024-08-17T17:41:43.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.21429687, 2024-08-17T17:41:43.654 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:43.654 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:43.654 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:41:43.654 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:43.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:43.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.710952+0000", 2024-08-17T17:41:43.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:43.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.710952+0000", 2024-08-17T17:41:43.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:43.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.710952+0000", 2024-08-17T17:41:43.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7000000000000003e-07 2024-08-17T17:41:43.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:43.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.710959+0000", 2024-08-17T17:41:43.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9179999999999997e-06 2024-08-17T17:41:43.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:43.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.887890+0000", 2024-08-17T17:41:43.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17693062000000001 2024-08-17T17:41:43.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:43.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.887904+0000", 2024-08-17T17:41:43.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3791999999999999e-05 2024-08-17T17:41:43.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:43.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.897824+0000", 2024-08-17T17:41:43.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0099197620000000004 2024-08-17T17:41:43.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:43.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.897852+0000", 2024-08-17T17:41:43.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8708999999999999e-05 2024-08-17T17:41:43.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:41:43.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.925217+0000", 2024-08-17T17:41:43.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.027365074999999999 2024-08-17T17:41:43.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:43.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.925248+0000", 2024-08-17T17:41:43.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1024e-05 2024-08-17T17:41:43.658 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.658 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:43.658 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.658 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.36354.0:136324 9.5 e333/311 ondisk, result = 0)", 2024-08-17T17:41:43.658 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:39:48.712804+0000", 2024-08-17T17:41:43.658 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 114.871504221, 2024-08-17T17:41:43.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17919397200000001, 2024-08-17T17:41:43.658 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:43.658 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:43.659 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:41:43.659 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:43.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:43.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.712804+0000", 2024-08-17T17:41:43.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:43.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.712804+0000", 2024-08-17T17:41:43.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:43.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.712805+0000", 2024-08-17T17:41:43.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3459999999999999e-06 2024-08-17T17:41:43.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:43.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.712813+0000", 2024-08-17T17:41:43.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3300000000000001e-06 2024-08-17T17:41:43.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:43.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.888224+0000", 2024-08-17T17:41:43.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.175410699 2024-08-17T17:41:43.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:43.661 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.888231+0000", 2024-08-17T17:41:43.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0400000000000004e-06 2024-08-17T17:41:43.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:43.661 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.891966+0000", 2024-08-17T17:41:43.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0037355890000000001 2024-08-17T17:41:43.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:43.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.891975+0000", 2024-08-17T17:41:43.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5799999999999992e-06 2024-08-17T17:41:43.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:43.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:48.891998+0000", 2024-08-17T17:41:43.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3388e-05 2024-08-17T17:41:43.662 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.662 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:43.662 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.663 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36354.0:138804 9.5 9:aec0763c:::benchmark_data_smithi003_261051_object22919:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-08-17T17:41:43.663 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:39:52.128013+0000", 2024-08-17T17:41:43.663 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 111.45629567100001, 2024-08-17T17:41:43.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18760734800000001, 2024-08-17T17:41:43.663 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:43.663 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:43.663 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:41:43.663 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:41:43.663 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36354", 2024-08-17T17:41:43.663 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1330158990", 2024-08-17T17:41:43.663 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 138804 2024-08-17T17:41:43.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.664 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:43.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:43.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.128013+0000", 2024-08-17T17:41:43.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:43.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.128013+0000", 2024-08-17T17:41:43.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:43.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.128014+0000", 2024-08-17T17:41:43.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.48e-06 2024-08-17T17:41:43.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:43.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.128018+0000", 2024-08-17T17:41:43.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.087e-06 2024-08-17T17:41:43.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:43.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.128030+0000", 2024-08-17T17:41:43.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1983e-05 2024-08-17T17:41:43.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:43.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.128037+0000", 2024-08-17T17:41:43.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7750000000000004e-06 2024-08-17T17:41:43.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:43.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.128099+0000", 2024-08-17T17:41:43.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2347999999999993e-05 2024-08-17T17:41:43.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:43.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.128205+0000", 2024-08-17T17:41:43.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000105732 2024-08-17T17:41:43.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-08-17T17:41:43.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.128306+0000", 2024-08-17T17:41:43.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000101143 2024-08-17T17:41:43.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:41:43.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.131705+0000", 2024-08-17T17:41:43.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0033984480000000001 2024-08-17T17:41:43.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:43.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.303353+0000", 2024-08-17T17:41:43.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17164818000000001 2024-08-17T17:41:43.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:43.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.315458+0000", 2024-08-17T17:41:43.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012105273 2024-08-17T17:41:43.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:41:43.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.315555+0000", 2024-08-17T17:41:43.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6929000000000007e-05 2024-08-17T17:41:43.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:43.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.315620+0000", 2024-08-17T17:41:43.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4969999999999996e-05 2024-08-17T17:41:43.668 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.668 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:43.668 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.669 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36354.0:138809 9.5 9:a429ad7c:::benchmark_data_smithi003_261051_object22924:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-08-17T17:41:43.669 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:39:52.135863+0000", 2024-08-17T17:41:43.669 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 111.44844537199999, 2024-08-17T17:41:43.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18420662199999999, 2024-08-17T17:41:43.669 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:43.669 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:43.669 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:41:43.669 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:41:43.669 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36354", 2024-08-17T17:41:43.669 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1330158990", 2024-08-17T17:41:43.669 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 138809 2024-08-17T17:41:43.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.670 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:43.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:43.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.135863+0000", 2024-08-17T17:41:43.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:43.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.135863+0000", 2024-08-17T17:41:43.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:43.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.135865+0000", 2024-08-17T17:41:43.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9180000000000001e-06 2024-08-17T17:41:43.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:43.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.135872+0000", 2024-08-17T17:41:43.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8809999999999998e-06 2024-08-17T17:41:43.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:43.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.135888+0000", 2024-08-17T17:41:43.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5767999999999999e-05 2024-08-17T17:41:43.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.672 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:43.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.135896+0000", 2024-08-17T17:41:43.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8999999999999995e-06 2024-08-17T17:41:43.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.672 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:43.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.136050+0000", 2024-08-17T17:41:43.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000153153 2024-08-17T17:41:43.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:43.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.136169+0000", 2024-08-17T17:41:43.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000119858 2024-08-17T17:41:43.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-08-17T17:41:43.674 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.136282+0000", 2024-08-17T17:41:43.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011288799999999999 2024-08-17T17:41:43.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:41:43.674 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.143505+0000", 2024-08-17T17:41:43.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072225370000000002 2024-08-17T17:41:43.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:43.674 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.315029+0000", 2024-08-17T17:41:43.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17152447300000001 2024-08-17T17:41:43.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:43.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.319979+0000", 2024-08-17T17:41:43.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049492169999999997 2024-08-17T17:41:43.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:41:43.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.320035+0000", 2024-08-17T17:41:43.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6350000000000001e-05 2024-08-17T17:41:43.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:43.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.320070+0000", 2024-08-17T17:41:43.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4678999999999998e-05 2024-08-17T17:41:43.676 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.676 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:43.676 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.676 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36354.0:138812 9.5 9:a33d4886:::benchmark_data_smithi003_261051_object22927:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-08-17T17:41:43.676 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:39:52.139826+0000", 2024-08-17T17:41:43.676 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 111.444482021, 2024-08-17T17:41:43.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18066331299999999, 2024-08-17T17:41:43.676 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:43.676 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:43.676 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:41:43.676 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:41:43.676 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36354", 2024-08-17T17:41:43.676 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1330158990", 2024-08-17T17:41:43.677 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 138812 2024-08-17T17:41:43.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.677 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:43.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.677 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:43.677 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.139826+0000", 2024-08-17T17:41:43.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.677 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:43.677 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.139826+0000", 2024-08-17T17:41:43.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:43.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.139828+0000", 2024-08-17T17:41:43.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4130000000000001e-06 2024-08-17T17:41:43.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:43.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.139833+0000", 2024-08-17T17:41:43.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.614e-06 2024-08-17T17:41:43.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:43.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.139846+0000", 2024-08-17T17:41:43.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2136999999999999e-05 2024-08-17T17:41:43.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:43.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.139853+0000", 2024-08-17T17:41:43.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7589999999999997e-06 2024-08-17T17:41:43.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:43.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.139906+0000", 2024-08-17T17:41:43.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2252e-05 2024-08-17T17:41:43.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:43.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.140033+0000", 2024-08-17T17:41:43.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000127949 2024-08-17T17:41:43.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-08-17T17:41:43.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.140132+0000", 2024-08-17T17:41:43.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8228999999999998e-05 2024-08-17T17:41:43.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:41:43.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.146948+0000", 2024-08-17T17:41:43.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0068162800000000001 2024-08-17T17:41:43.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:43.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.315076+0000", 2024-08-17T17:41:43.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.16812785799999999 2024-08-17T17:41:43.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:43.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.320142+0000", 2024-08-17T17:41:43.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0050661100000000004 2024-08-17T17:41:43.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:41:43.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.320290+0000", 2024-08-17T17:41:43.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014792200000000001 2024-08-17T17:41:43.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:43.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.320490+0000", 2024-08-17T17:41:43.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019979000000000001 2024-08-17T17:41:43.682 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.682 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:43.682 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.682 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36354.0:138814 9.5 9:aa962cc3:::benchmark_data_smithi003_261051_object22929:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-08-17T17:41:43.682 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:39:52.141874+0000", 2024-08-17T17:41:43.682 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 111.442434013, 2024-08-17T17:41:43.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18161983900000001, 2024-08-17T17:41:43.682 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:43.683 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:43.683 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:41:43.683 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:41:43.683 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36354", 2024-08-17T17:41:43.683 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1330158990", 2024-08-17T17:41:43.683 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 138814 2024-08-17T17:41:43.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.683 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:43.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:43.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.141874+0000", 2024-08-17T17:41:43.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:43.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.141874+0000", 2024-08-17T17:41:43.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:43.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.141876+0000", 2024-08-17T17:41:43.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.793e-06 2024-08-17T17:41:43.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:43.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.141882+0000", 2024-08-17T17:41:43.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.733e-06 2024-08-17T17:41:43.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:43.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.141896+0000", 2024-08-17T17:41:43.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4236999999999999e-05 2024-08-17T17:41:43.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:43.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.141905+0000", 2024-08-17T17:41:43.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5399999999999996e-06 2024-08-17T17:41:43.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:43.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.143528+0000", 2024-08-17T17:41:43.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00162352 2024-08-17T17:41:43.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:43.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.143650+0000", 2024-08-17T17:41:43.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000121455 2024-08-17T17:41:43.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-08-17T17:41:43.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.143749+0000", 2024-08-17T17:41:43.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9104999999999994e-05 2024-08-17T17:41:43.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:41:43.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.147192+0000", 2024-08-17T17:41:43.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034432619999999999 2024-08-17T17:41:43.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:43.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.315105+0000", 2024-08-17T17:41:43.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.16791325900000001 2024-08-17T17:41:43.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:41:43.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.323403+0000", 2024-08-17T17:41:43.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0082974959999999997 2024-08-17T17:41:43.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:41:43.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.323454+0000", 2024-08-17T17:41:43.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1036999999999999e-05 2024-08-17T17:41:43.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:43.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:39:52.323494+0000", 2024-08-17T17:41:43.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0401999999999997e-05 2024-08-17T17:41:43.688 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.688 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:43.688 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.688 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDPGPush(9.4 347/344 [PushOp(9:2ff9121a:::benchmark_data_smithi003_261051_object100767:head, version: 325'6305, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:2ff9121a:::benchmark_data_smithi003_261051_object100767:head@325'6305, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-17T17:41:43.689 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:41:38.776718+0000", 2024-08-17T17:41:43.689 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 4.80759063, 2024-08-17T17:41:43.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.21963379299999999, 2024-08-17T17:41:43.689 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:43.689 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:43.689 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:41:43.689 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:43.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:43.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:38.776718+0000", 2024-08-17T17:41:43.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:43.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:38.776718+0000", 2024-08-17T17:41:43.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:43.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:38.776718+0000", 2024-08-17T17:41:43.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4099999999999998e-07 2024-08-17T17:41:43.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:43.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:38.776748+0000", 2024-08-17T17:41:43.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9382000000000001e-05 2024-08-17T17:41:43.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.691 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:43.691 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:38.776766+0000", 2024-08-17T17:41:43.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8119999999999999e-05 2024-08-17T17:41:43.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.691 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:43.691 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:38.776785+0000", 2024-08-17T17:41:43.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8770000000000002e-05 2024-08-17T17:41:43.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.691 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:43.691 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:38.993324+0000", 2024-08-17T17:41:43.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.21653958100000001 2024-08-17T17:41:43.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:43.692 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:38.993337+0000", 2024-08-17T17:41:43.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2982e-05 2024-08-17T17:41:43.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:43.692 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:38.996352+0000", 2024-08-17T17:41:43.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003014217 2024-08-17T17:41:43.692 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.692 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:43.692 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.693 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDPGPush(9.4 347/344 [PushOp(9:2e77b8cf:::benchmark_data_smithi003_261051_object100778:head, version: 325'6306, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:2e77b8cf:::benchmark_data_smithi003_261051_object100778:head@325'6306, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-17T17:41:43.693 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:41:38.802450+0000", 2024-08-17T17:41:43.693 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 4.781858207, 2024-08-17T17:41:43.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.196904839, 2024-08-17T17:41:43.693 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:41:43.693 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:41:43.693 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:41:43.693 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:41:43.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:41:43.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:38.802450+0000", 2024-08-17T17:41:43.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:41:43.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:38.802450+0000", 2024-08-17T17:41:43.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:41:43.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:41:43.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:38.802451+0000", 2024-08-17T17:41:43.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.063e-06 2024-08-17T17:41:43.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:41:43.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:38.802487+0000", 2024-08-17T17:41:43.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5636000000000002e-05 2024-08-17T17:41:43.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:41:43.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:38.802513+0000", 2024-08-17T17:41:43.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6194999999999999e-05 2024-08-17T17:41:43.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:41:43.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:38.802530+0000", 2024-08-17T17:41:43.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7244e-05 2024-08-17T17:41:43.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:41:43.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:38.996401+0000", 2024-08-17T17:41:43.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.19387051 2024-08-17T17:41:43.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:41:43.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:38.996412+0000", 2024-08-17T17:41:43.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1418e-05 2024-08-17T17:41:43.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:41:43.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:41:43.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:41:43.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:38.999355+0000", 2024-08-17T17:41:43.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029427730000000001 2024-08-17T17:41:43.696 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.696 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:43.696 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.697 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:41:43.697 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:41:43.697 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:41:43.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:43.698 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:43.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:43.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:43 smithi003 ceph-mon[100615]: pgmap v1133: 119 pgs: 2 active+recovery_wait+degraded+remapped, 1 active+recovering+degraded+remapped, 2 active+clean+scrubbing, 114 active+clean; 903 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 3.2 KiB/s wr, 473 op/s; 1871/43191 objects degraded (4.332%); 6.1 MiB/s, 1 keys/s, 102 objects/s recovering 2024-08-17T17:41:43.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:43 smithi003 ceph-mon[100615]: 1.c scrub starts 2024-08-17T17:41:43.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:43 smithi003 ceph-mon[100615]: 1.c scrub ok 2024-08-17T17:41:43.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:43 smithi003 ceph-mon[103344]: pgmap v1133: 119 pgs: 2 active+recovery_wait+degraded+remapped, 1 active+recovering+degraded+remapped, 2 active+clean+scrubbing, 114 active+clean; 903 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 3.2 KiB/s wr, 473 op/s; 1871/43191 objects degraded (4.332%); 6.1 MiB/s, 1 keys/s, 102 objects/s recovering 2024-08-17T17:41:43.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:43 smithi003 ceph-mon[103344]: 1.c scrub starts 2024-08-17T17:41:43.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:43 smithi003 ceph-mon[103344]: 1.c scrub ok 2024-08-17T17:41:43.799 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:43.824 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:43.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:43.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:43.926 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:43.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:43 smithi053 ceph-mon[89963]: pgmap v1133: 119 pgs: 2 active+recovery_wait+degraded+remapped, 1 active+recovering+degraded+remapped, 2 active+clean+scrubbing, 114 active+clean; 903 MiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 3.2 KiB/s wr, 473 op/s; 1871/43191 objects degraded (4.332%); 6.1 MiB/s, 1 keys/s, 102 objects/s recovering 2024-08-17T17:41:43.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:43 smithi053 ceph-mon[89963]: 1.c scrub starts 2024-08-17T17:41:43.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:43 smithi053 ceph-mon[89963]: 1.c scrub ok 2024-08-17T17:41:43.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:43.968 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:43.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:44.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:44.106 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:44.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:44.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:44.208 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:44.232 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:44.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:44.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:44.334 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:44.375 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:44.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:44.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:44.477 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:44.515 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:44.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:44.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:44.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:44.658 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:44.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:44.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:44.760 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:44.800 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:44.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:44.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:44.902 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:44.918 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:44 smithi003 ceph-mon[100615]: 1.7 scrub starts 2024-08-17T17:41:44.919 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:44 smithi003 ceph-mon[100615]: 1.7 scrub ok 2024-08-17T17:41:44.919 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:44 smithi003 ceph-mon[100615]: 1.1c scrub starts 2024-08-17T17:41:44.919 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:44 smithi003 ceph-mon[100615]: 1.1c scrub ok 2024-08-17T17:41:44.919 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:44 smithi003 ceph-mon[103344]: 1.7 scrub starts 2024-08-17T17:41:44.919 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:44 smithi003 ceph-mon[103344]: 1.7 scrub ok 2024-08-17T17:41:44.919 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:44 smithi003 ceph-mon[103344]: 1.1c scrub starts 2024-08-17T17:41:44.919 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:44 smithi003 ceph-mon[103344]: 1.1c scrub ok 2024-08-17T17:41:44.944 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:44.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:44.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:44.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:44 smithi053 ceph-mon[89963]: 1.7 scrub starts 2024-08-17T17:41:44.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:44 smithi053 ceph-mon[89963]: 1.7 scrub ok 2024-08-17T17:41:44.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:44 smithi053 ceph-mon[89963]: 1.1c scrub starts 2024-08-17T17:41:44.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:44 smithi053 ceph-mon[89963]: 1.1c scrub ok 2024-08-17T17:41:45.046 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:45.057 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:41:45.096 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:45.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:45.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:45.198 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:45.225 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:45.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:45.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:45.327 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:45.352 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:45.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:45.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:45.361 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:41:45.367 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:41:45.391 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-17T17:41:45.454 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:45.509 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:45.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:45.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:45.611 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:45.663 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:45.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:45.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:45.765 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:45.792 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:45.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:45.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:45.894 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:45.940 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:45.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:45.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:45.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:45 smithi053 ceph-mon[89963]: 1.6 scrub starts 2024-08-17T17:41:45.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:45 smithi053 ceph-mon[89963]: 1.6 scrub ok 2024-08-17T17:41:45.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:45 smithi053 ceph-mon[89963]: pgmap v1134: 119 pgs: 2 active+recovery_wait+degraded+remapped, 1 active+recovering+degraded+remapped, 2 active+clean+scrubbing, 114 active+clean; 887 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 2.5 KiB/s wr, 402 op/s; 1871/42465 objects degraded (4.406%); 4.7 MiB/s, 1 keys/s, 80 objects/s recovering 2024-08-17T17:41:45.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:45 smithi053 ceph-mon[89963]: 1.d scrub starts 2024-08-17T17:41:45.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:45 smithi053 ceph-mon[89963]: 1.9 scrub starts 2024-08-17T17:41:45.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:45 smithi053 ceph-mon[89963]: 1.9 scrub ok 2024-08-17T17:41:45.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:45 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 1871/42465 objects degraded (4.406%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:41:45.984 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:45 smithi003 ceph-mon[103344]: 1.6 scrub starts 2024-08-17T17:41:45.984 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:45 smithi003 ceph-mon[103344]: 1.6 scrub ok 2024-08-17T17:41:45.984 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:45 smithi003 ceph-mon[103344]: pgmap v1134: 119 pgs: 2 active+recovery_wait+degraded+remapped, 1 active+recovering+degraded+remapped, 2 active+clean+scrubbing, 114 active+clean; 887 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 2.5 KiB/s wr, 402 op/s; 1871/42465 objects degraded (4.406%); 4.7 MiB/s, 1 keys/s, 80 objects/s recovering 2024-08-17T17:41:45.984 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:45 smithi003 ceph-mon[103344]: 1.d scrub starts 2024-08-17T17:41:45.984 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:45 smithi003 ceph-mon[103344]: 1.9 scrub starts 2024-08-17T17:41:45.985 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:45 smithi003 ceph-mon[103344]: 1.9 scrub ok 2024-08-17T17:41:45.985 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:45 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 1871/42465 objects degraded (4.406%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:41:45.985 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:41:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:41:45] "GET /metrics HTTP/1.1" 200 36518 "" "Prometheus/2.51.0" 2024-08-17T17:41:45.985 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:45 smithi003 ceph-mon[100615]: 1.6 scrub starts 2024-08-17T17:41:45.985 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:45 smithi003 ceph-mon[100615]: 1.6 scrub ok 2024-08-17T17:41:45.985 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:45 smithi003 ceph-mon[100615]: pgmap v1134: 119 pgs: 2 active+recovery_wait+degraded+remapped, 1 active+recovering+degraded+remapped, 2 active+clean+scrubbing, 114 active+clean; 887 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 2.5 KiB/s wr, 402 op/s; 1871/42465 objects degraded (4.406%); 4.7 MiB/s, 1 keys/s, 80 objects/s recovering 2024-08-17T17:41:45.985 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:45 smithi003 ceph-mon[100615]: 1.d scrub starts 2024-08-17T17:41:45.985 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:45 smithi003 ceph-mon[100615]: 1.9 scrub starts 2024-08-17T17:41:45.985 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:45 smithi003 ceph-mon[100615]: 1.9 scrub ok 2024-08-17T17:41:45.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:45 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 1871/42465 objects degraded (4.406%), 3 pgs degraded (PG_DEGRADED) 2024-08-17T17:41:46.042 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:46.078 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:46.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:46.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:46.180 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:46.217 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:46.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:46.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:46.319 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:46.365 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:46.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:46.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:46.467 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:46.492 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:46.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:46.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:46.594 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:46.622 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:46.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:46.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:46.724 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:46.761 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:46.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:46.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:46.864 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:46.889 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:46.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:46.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:46.991 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:47.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:46 smithi003 ceph-mon[103344]: 1.19 scrub starts 2024-08-17T17:41:47.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:46 smithi003 ceph-mon[103344]: 1.19 scrub ok 2024-08-17T17:41:47.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:46 smithi003 ceph-mon[100615]: 1.19 scrub starts 2024-08-17T17:41:47.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:46 smithi003 ceph-mon[100615]: 1.19 scrub ok 2024-08-17T17:41:47.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:47.024 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:47.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:47.125 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:47.168 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:47.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:47.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:47.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:46 smithi053 ceph-mon[89963]: 1.19 scrub starts 2024-08-17T17:41:47.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:46 smithi053 ceph-mon[89963]: 1.19 scrub ok 2024-08-17T17:41:47.270 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:47.295 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:47.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:47.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:47.397 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:47.439 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:47.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:47.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:47.540 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:47.566 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:47.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:47.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:47.667 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:47.693 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:47.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:47.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:47.795 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:47.832 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:47.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:47.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:47.934 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:47.973 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:47.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:47.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:48.026 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:47 smithi003 ceph-mon[103344]: pgmap v1135: 119 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 116 active+clean; 666 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 2.2 KiB/s wr, 765 op/s; 1326/31818 objects degraded (4.167%); 6.7 MiB/s, 1 keys/s, 113 objects/s recovering 2024-08-17T17:41:48.026 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:47 smithi003 ceph-mon[103344]: 1.8 scrub starts 2024-08-17T17:41:48.026 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:47 smithi003 ceph-mon[103344]: 1.8 scrub ok 2024-08-17T17:41:48.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:47 smithi003 ceph-mon[100615]: pgmap v1135: 119 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 116 active+clean; 666 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 2.2 KiB/s wr, 765 op/s; 1326/31818 objects degraded (4.167%); 6.7 MiB/s, 1 keys/s, 113 objects/s recovering 2024-08-17T17:41:48.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:47 smithi003 ceph-mon[100615]: 1.8 scrub starts 2024-08-17T17:41:48.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:47 smithi003 ceph-mon[100615]: 1.8 scrub ok 2024-08-17T17:41:48.076 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:48.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:48.119 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:48.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:48.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:47 smithi053 ceph-mon[89963]: pgmap v1135: 119 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 116 active+clean; 666 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 2.2 KiB/s wr, 765 op/s; 1326/31818 objects degraded (4.167%); 6.7 MiB/s, 1 keys/s, 113 objects/s recovering 2024-08-17T17:41:48.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:47 smithi053 ceph-mon[89963]: 1.8 scrub starts 2024-08-17T17:41:48.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:47 smithi053 ceph-mon[89963]: 1.8 scrub ok 2024-08-17T17:41:48.220 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:48.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:48.280 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:48.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:48.381 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:48.410 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:48.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:48.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:48.512 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:48.539 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:48.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:48.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:48.641 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:48.688 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:48.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:48.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:48.789 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:48.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:48.829 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:48.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:48.930 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:48.959 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:48.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:48.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:49.061 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:49.085 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:49.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:49.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:49.187 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:49.210 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:49.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:49.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:49.312 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:49.344 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:49.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:49.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:49.446 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:49.475 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:49.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:49.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:49.577 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:49.618 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:49.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:49.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:49.720 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:49.752 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:49.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:49.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:49.854 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:49.879 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:49.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:49.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:49.980 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:50.008 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:50.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:50.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:50.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:49 smithi003 ceph-mon[103344]: pgmap v1136: 119 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 116 active+clean; 666 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.9 KiB/s wr, 663 op/s; 1326/31818 objects degraded (4.167%); 5.8 MiB/s, 0 keys/s, 98 objects/s recovering 2024-08-17T17:41:50.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:49 smithi003 ceph-mon[100615]: pgmap v1136: 119 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 116 active+clean; 666 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.9 KiB/s wr, 663 op/s; 1326/31818 objects degraded (4.167%); 5.8 MiB/s, 0 keys/s, 98 objects/s recovering 2024-08-17T17:41:50.110 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:50.145 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:50.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:50.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:50.213 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:49 smithi053 ceph-mon[89963]: pgmap v1136: 119 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 116 active+clean; 666 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.9 KiB/s wr, 663 op/s; 1326/31818 objects degraded (4.167%); 5.8 MiB/s, 0 keys/s, 98 objects/s recovering 2024-08-17T17:41:50.247 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:50.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:50.302 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:50.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:50.403 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:50.429 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:50.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:50.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:50.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:41:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:41:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:41:50.532 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:50.560 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:50.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:50.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:50.662 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:50.694 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:50.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:50.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:50.796 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:50.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:50.825 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:50.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:50.926 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:50.963 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:50.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:50.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:51.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:50 smithi003 ceph-mon[103344]: 1.16 scrub starts 2024-08-17T17:41:51.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:50 smithi003 ceph-mon[103344]: 1.16 scrub ok 2024-08-17T17:41:51.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:50 smithi003 ceph-mon[103344]: 1.1d scrub starts 2024-08-17T17:41:51.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:50 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 1326/31818 objects degraded (4.167%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:41:51.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:50 smithi003 ceph-mon[100615]: 1.16 scrub starts 2024-08-17T17:41:51.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:50 smithi003 ceph-mon[100615]: 1.16 scrub ok 2024-08-17T17:41:51.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:50 smithi003 ceph-mon[100615]: 1.1d scrub starts 2024-08-17T17:41:51.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:50 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 1326/31818 objects degraded (4.167%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:41:51.066 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:51.091 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:51.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:51.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:51.193 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:51.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:50 smithi053 ceph-mon[89963]: 1.16 scrub starts 2024-08-17T17:41:51.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:50 smithi053 ceph-mon[89963]: 1.16 scrub ok 2024-08-17T17:41:51.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:50 smithi053 ceph-mon[89963]: 1.1d scrub starts 2024-08-17T17:41:51.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:50 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 1326/31818 objects degraded (4.167%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:41:51.238 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:51.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:51.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:51.340 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:51.382 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:51.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:51.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:51.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:51.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:51.525 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:51.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:51.626 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:51.651 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:51.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:51.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:51.753 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:51.779 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:51.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:51.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:51.881 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:51.923 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:51.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:51.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:51.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:51 smithi053 ceph-mon[89963]: pgmap v1137: 119 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 116 active+clean; 522 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 748 op/s; 1059/24939 objects degraded (4.246%); 4.7 MiB/s, 79 objects/s recovering 2024-08-17T17:41:52.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:51 smithi003 ceph-mon[103344]: pgmap v1137: 119 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 116 active+clean; 522 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 748 op/s; 1059/24939 objects degraded (4.246%); 4.7 MiB/s, 79 objects/s recovering 2024-08-17T17:41:52.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:51 smithi003 ceph-mon[100615]: pgmap v1137: 119 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 116 active+clean; 522 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 748 op/s; 1059/24939 objects degraded (4.246%); 4.7 MiB/s, 79 objects/s recovering 2024-08-17T17:41:52.025 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:52.054 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:52.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:52.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:52.156 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:52.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:52.193 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:52.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:52.294 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:52.329 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:52.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:52.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:52.431 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:52.459 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:52.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:52.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:52.561 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:52.589 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:52.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:52.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:52.691 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:52.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:52.724 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:52.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:52.825 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:52.850 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:52.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:52.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:52.952 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:52.981 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:52.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:52.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:53.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:52 smithi003 ceph-mon[103344]: 1.17 scrub starts 2024-08-17T17:41:53.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:52 smithi003 ceph-mon[103344]: 1.17 scrub ok 2024-08-17T17:41:53.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:52 smithi003 ceph-mon[103344]: 1.d scrub starts 2024-08-17T17:41:53.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:52 smithi003 ceph-mon[100615]: 1.17 scrub starts 2024-08-17T17:41:53.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:52 smithi003 ceph-mon[100615]: 1.17 scrub ok 2024-08-17T17:41:53.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:52 smithi003 ceph-mon[100615]: 1.d scrub starts 2024-08-17T17:41:53.084 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:53.114 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:53.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:53.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:53.216 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:53.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:52 smithi053 ceph-mon[89963]: 1.17 scrub starts 2024-08-17T17:41:53.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:52 smithi053 ceph-mon[89963]: 1.17 scrub ok 2024-08-17T17:41:53.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:52 smithi053 ceph-mon[89963]: 1.d scrub starts 2024-08-17T17:41:53.243 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:53.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:53.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:53.344 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:53.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:53.408 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:53.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:53.509 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:53.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:53.542 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:53.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:53.643 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:53.669 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:53.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:53.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:53.771 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:53.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:53 smithi003 ceph-mon[103344]: pgmap v1138: 119 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 116 active+clean; 412 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 794 op/s; 809/19653 objects degraded (4.116%); 4.1 MiB/s, 72 objects/s recovering 2024-08-17T17:41:53.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:41:53.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:41:53.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:41:53.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:41:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:53 smithi003 ceph-mon[100615]: pgmap v1138: 119 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 116 active+clean; 412 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 794 op/s; 809/19653 objects degraded (4.116%); 4.1 MiB/s, 72 objects/s recovering 2024-08-17T17:41:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:41:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:41:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:41:53.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:41:53.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:53.795 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:53.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:53.896 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:53.959 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:53.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:53.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:54.061 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:54.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:54.117 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:54.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:54.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:53 smithi053 ceph-mon[89963]: pgmap v1138: 119 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 116 active+clean; 412 MiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 794 op/s; 809/19653 objects degraded (4.116%); 4.1 MiB/s, 72 objects/s recovering 2024-08-17T17:41:54.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:41:54.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:41:54.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:41:54.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:41:54.218 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:54.257 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:54.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:54.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:54.359 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:54.388 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:54.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:54.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:54.490 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:54.519 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:54.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:54.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:54.621 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:54.651 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:54.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:54.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:54.738 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:41:54.747 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:41:54.753 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:54.827 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:54.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:54.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:54.929 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:54.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:54.959 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:54.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:55.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:54 smithi003 ceph-mon[100615]: 1.1a scrub starts 2024-08-17T17:41:55.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:54 smithi003 ceph-mon[100615]: 1.1a scrub ok 2024-08-17T17:41:55.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:54 smithi003 ceph-mon[100615]: 1.1d scrub starts 2024-08-17T17:41:55.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:54 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:41:55.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:54 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:41:55.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:54 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:41:55.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:54 smithi003 ceph-mon[103344]: 1.1a scrub starts 2024-08-17T17:41:55.025 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:54 smithi003 ceph-mon[103344]: 1.1a scrub ok 2024-08-17T17:41:55.025 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:54 smithi003 ceph-mon[103344]: 1.1d scrub starts 2024-08-17T17:41:55.025 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:54 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:41:55.025 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:54 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:41:55.025 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:54 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:41:55.060 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:55.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:55.110 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:55.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:55.211 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:55.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:54 smithi053 ceph-mon[89963]: 1.1a scrub starts 2024-08-17T17:41:55.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:54 smithi053 ceph-mon[89963]: 1.1a scrub ok 2024-08-17T17:41:55.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:54 smithi053 ceph-mon[89963]: 1.1d scrub starts 2024-08-17T17:41:55.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:54 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:41:55.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:54 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:41:55.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:54 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:41:55.238 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:55.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:55.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:55.340 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:55.368 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:55.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:55.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:55.470 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:55.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:55.509 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:55.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:55.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:55.651 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:55.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:55.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:55.754 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:55.794 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:55.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:55.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:55.850 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:55 smithi003 ceph-mon[103344]: pgmap v1139: 119 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 116 active+clean; 396 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 674 op/s; 809/18894 objects degraded (4.282%); 1.8 MiB/s, 35 objects/s recovering 2024-08-17T17:41:55.850 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:55 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 809/19653 objects degraded (4.116%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:41:55.851 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:55 smithi003 ceph-mon[100615]: pgmap v1139: 119 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 116 active+clean; 396 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 674 op/s; 809/18894 objects degraded (4.282%); 1.8 MiB/s, 35 objects/s recovering 2024-08-17T17:41:55.852 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:55 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 809/19653 objects degraded (4.116%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:41:55.852 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:41:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:41:55] "GET /metrics HTTP/1.1" 200 36511 "" "Prometheus/2.51.0" 2024-08-17T17:41:55.895 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:55.945 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:55.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:55.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:56.047 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:56.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:56.078 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:56.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:56.179 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:56.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:55 smithi053 ceph-mon[89963]: pgmap v1139: 119 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 116 active+clean; 396 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 674 op/s; 809/18894 objects degraded (4.282%); 1.8 MiB/s, 35 objects/s recovering 2024-08-17T17:41:56.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:55 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 809/19653 objects degraded (4.116%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:41:56.235 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:56.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:56.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:56.339 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:56.364 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:56.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:56.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:56.466 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:56.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:56.533 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:56.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:56.634 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:56.679 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:56.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:56.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:56.701 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:41:56.702 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:41:56.704 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:41:56.705 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:41:56.742 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:41:56.743 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:41:56.781 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:56.786 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:41:56.786 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:41:56.818 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:41:56.818 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:41:56.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:56.836 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:56.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:56.856 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:41:56.856 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:41:56.884 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:41:56.885 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:41:56.912 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:41:56.913 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:41:56.937 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:56.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:56.962 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:56.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:57.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:56 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1100973316' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:41:57.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:56 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1100973316' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:41:57.063 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:57.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:57.109 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:57.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:57.210 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:57.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:56 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1100973316' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:41:57.237 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:57.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:57.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:57.339 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:57.340 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:41:57.340 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "" 2024-08-17T17:41:57.340 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:41:57.373 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:57.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:57.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:57.474 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:57.508 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:57.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:57.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:57.611 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:57.641 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:57.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:57.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:57.742 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:57.772 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:57.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:57.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:57.874 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:57.903 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:57.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:57.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:58.005 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:58.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:57 smithi003 ceph-mon[100615]: pgmap v1140: 119 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 116 active+clean; 156 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 975 op/s; 297/7344 objects degraded (4.044%); 1.8 MiB/s, 40 objects/s recovering 2024-08-17T17:41:58.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:57 smithi003 ceph-mon[103344]: pgmap v1140: 119 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 116 active+clean; 156 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 975 op/s; 297/7344 objects degraded (4.044%); 1.8 MiB/s, 40 objects/s recovering 2024-08-17T17:41:58.039 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:58.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:58.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:58.141 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:58.197 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:58.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:58.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:41:58.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:57 smithi053 ceph-mon[89963]: pgmap v1140: 119 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 116 active+clean; 156 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 975 op/s; 297/7344 objects degraded (4.044%); 1.8 MiB/s, 40 objects/s recovering 2024-08-17T17:41:58.299 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:58.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:58.361 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:58.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:58.369 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Removed 115882 objects 2024-08-17T17:41:58.369 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Clean up completed and total clean up time :158.018 2024-08-17T17:41:58.462 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:58.484 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:58.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:58.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:58.586 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:41:58.609 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:58.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:58.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:41:58.711 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:41:58.735 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:58.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:58.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:41:58.836 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:58.860 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:58.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:58.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:58.962 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:58.984 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:58.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:58.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:59.086 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:41:59.109 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:59.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:59.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:41:59.211 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:41:59.235 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:59.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:59.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:41:59.336 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:59.361 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:59.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:59.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:59.463 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:41:59.489 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:59.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:59.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:41:59.591 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:59.599 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:41:59.599 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:41:59.615 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:59.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:59.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:59.717 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:41:59.742 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:59.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:59.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:41:59.843 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:41:59.869 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:41:59.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:59.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:41:59.971 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:41:59.994 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:41:59.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:41:59.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:00.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:41:59 smithi003 ceph-mon[100615]: pgmap v1141: 119 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 116 active+clean; 156 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 679 op/s; 297/7344 objects degraded (4.044%); 0 B/s, 9 objects/s recovering 2024-08-17T17:42:00.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:41:59 smithi003 ceph-mon[103344]: pgmap v1141: 119 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 116 active+clean; 156 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 679 op/s; 297/7344 objects degraded (4.044%); 0 B/s, 9 objects/s recovering 2024-08-17T17:42:00.096 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:00.119 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:00.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:00.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:00.213 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:41:59 smithi053 ceph-mon[89963]: pgmap v1141: 119 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+degraded+remapped, 116 active+clean; 156 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 679 op/s; 297/7344 objects degraded (4.044%); 0 B/s, 9 objects/s recovering 2024-08-17T17:42:00.221 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:00.247 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:00.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:00.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:00.349 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:00.372 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:00.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:00.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:00.424 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-08-17T17:42:00.424 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-08-17T17:42:00.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:42:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:42:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:42:00.474 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:00.497 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:00.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:00.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:00.599 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:00.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-17T17:42:00.625 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:00.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:00.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:00.727 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:00.754 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:00.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:00.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:00.856 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:00.881 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:00.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:00.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:00.982 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:01.007 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:01.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:01.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:01.022 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:00 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 297/7344 objects degraded (4.044%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:42:01.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:00 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 297/7344 objects degraded (4.044%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:42:01.108 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:01.132 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:01.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:01.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:01.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:00 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 297/7344 objects degraded (4.044%), 2 pgs degraded (PG_DEGRADED) 2024-08-17T17:42:01.234 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:01.259 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:01.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:01.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:01.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:01.388 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:01.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:01.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:01.420 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:42:01.425 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:42:01.490 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:01.516 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:01.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:01.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:01.618 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:01.641 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:01.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:01.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:01.743 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:01.766 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:01.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:01.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:01.776 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-08-17T17:42:01.868 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:01.891 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:01.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:01.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:01.993 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:02.016 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:02.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:02.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:02.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:01 smithi003 ceph-mon[103344]: pgmap v1142: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 1 active+recovery_wait+degraded+remapped, 116 active+clean; 54 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 814 op/s; 125/2472 objects degraded (5.057%); 0 B/s, 9 objects/s recovering 2024-08-17T17:42:02.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:01 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1684958736' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:42:02.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:01 smithi003 ceph-mon[100615]: pgmap v1142: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 1 active+recovery_wait+degraded+remapped, 116 active+clean; 54 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 814 op/s; 125/2472 objects degraded (5.057%); 0 B/s, 9 objects/s recovering 2024-08-17T17:42:02.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:01 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1684958736' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:42:02.118 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:02.142 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:02.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:02.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:02.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:01 smithi053 ceph-mon[89963]: pgmap v1142: 119 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 1 active+recovery_wait+degraded+remapped, 116 active+clean; 54 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 814 op/s; 125/2472 objects degraded (5.057%); 0 B/s, 9 objects/s recovering 2024-08-17T17:42:02.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:01 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1684958736' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:42:02.244 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:02.268 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:02.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:02.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:02.370 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:02.393 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:02.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:02.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:02.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:02.518 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:02.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:02.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:02.620 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:02.642 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:02.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:02.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:02.744 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:02.766 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:02.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:02.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:02.868 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:02.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:02.895 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:02.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:02.996 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:03.021 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:03.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:03.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:03.117 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:02 smithi003 ceph-mon[103344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:42:03.117 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:02 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1684958736' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:42:03.117 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:02 smithi003 ceph-mon[103344]: osdmap e349: 8 total, 8 up, 8 in 2024-08-17T17:42:03.117 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:02 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-08-17T17:42:03.117 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:02 smithi003 ceph-mon[103344]: 1.d deep-scrub starts 2024-08-17T17:42:03.118 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:02 smithi003 ceph-mon[100615]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:42:03.118 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:02 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1684958736' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:42:03.118 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:02 smithi003 ceph-mon[100615]: osdmap e349: 8 total, 8 up, 8 in 2024-08-17T17:42:03.118 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:02 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-08-17T17:42:03.118 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:02 smithi003 ceph-mon[100615]: 1.d deep-scrub starts 2024-08-17T17:42:03.123 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:03.149 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:03.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:03.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:03.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:02 smithi053 ceph-mon[89963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:42:03.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:02 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1684958736' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:42:03.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:02 smithi053 ceph-mon[89963]: osdmap e349: 8 total, 8 up, 8 in 2024-08-17T17:42:03.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:02 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-08-17T17:42:03.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:02 smithi053 ceph-mon[89963]: 1.d deep-scrub starts 2024-08-17T17:42:03.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:03.275 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:03.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:03.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:03.376 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:03.403 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:03.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:03.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:03.409 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:42:03.505 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:03.530 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:03.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:03.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:03.632 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:03.655 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:03.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:03.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:03.757 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:03.784 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:03.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:03.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:03.886 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:03.912 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:03.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:03.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:04.013 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:04.040 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:04.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:04.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:04.114 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-08-17T17:42:04.114 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:42:04.114 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-08-17T17:42:04.114 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-17T17:42:04.114 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-17T17:42:04.114 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-08-17T17:42:04.121 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:03 smithi003 ceph-mon[103344]: pgmap v1144: 119 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+remapped, 116 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 652 op/s; 0 B/s, 36 objects/s recovering 2024-08-17T17:42:04.121 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:03 smithi003 ceph-mon[103344]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 125/2472 objects degraded (5.057%), 1 pg degraded) 2024-08-17T17:42:04.121 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:03 smithi003 ceph-mon[103344]: Cluster is now healthy 2024-08-17T17:42:04.121 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:03 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-08-17T17:42:04.121 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:03 smithi003 ceph-mon[103344]: osdmap e350: 8 total, 8 up, 8 in 2024-08-17T17:42:04.122 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:03 smithi003 ceph-mon[100615]: pgmap v1144: 119 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+remapped, 116 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 652 op/s; 0 B/s, 36 objects/s recovering 2024-08-17T17:42:04.122 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:03 smithi003 ceph-mon[100615]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 125/2472 objects degraded (5.057%), 1 pg degraded) 2024-08-17T17:42:04.122 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:03 smithi003 ceph-mon[100615]: Cluster is now healthy 2024-08-17T17:42:04.122 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:03 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-08-17T17:42:04.122 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:03 smithi003 ceph-mon[100615]: osdmap e350: 8 total, 8 up, 8 in 2024-08-17T17:42:04.142 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:04.167 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:04.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:04.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:04.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:03 smithi053 ceph-mon[89963]: pgmap v1144: 119 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+remapped, 116 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 652 op/s; 0 B/s, 36 objects/s recovering 2024-08-17T17:42:04.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:03 smithi053 ceph-mon[89963]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 125/2472 objects degraded (5.057%), 1 pg degraded) 2024-08-17T17:42:04.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:03 smithi053 ceph-mon[89963]: Cluster is now healthy 2024-08-17T17:42:04.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:03 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-08-17T17:42:04.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:03 smithi053 ceph-mon[89963]: osdmap e350: 8 total, 8 up, 8 in 2024-08-17T17:42:04.268 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:04.292 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:04.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:04.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:04.394 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:04.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:04.421 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:04.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:04.522 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:04.549 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:04.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:04.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:04.650 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:04.675 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:04.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:04.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:04.777 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:04.810 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:04.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:04.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:04.912 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:04.936 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:04.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:04.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:05.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:04 smithi003 ceph-mon[103344]: osdmap e351: 8 total, 8 up, 8 in 2024-08-17T17:42:05.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:04 smithi003 ceph-mon[103344]: 1.d scrub starts 2024-08-17T17:42:05.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:04 smithi003 ceph-mon[103344]: osdmap e352: 8 total, 8 up, 8 in 2024-08-17T17:42:05.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:04 smithi003 ceph-mon[100615]: osdmap e351: 8 total, 8 up, 8 in 2024-08-17T17:42:05.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:04 smithi003 ceph-mon[100615]: 1.d scrub starts 2024-08-17T17:42:05.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:04 smithi003 ceph-mon[100615]: osdmap e352: 8 total, 8 up, 8 in 2024-08-17T17:42:05.038 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:05.062 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:05.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:05.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:05.165 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:05.188 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:05.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:05.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:05.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:04 smithi053 ceph-mon[89963]: osdmap e351: 8 total, 8 up, 8 in 2024-08-17T17:42:05.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:04 smithi053 ceph-mon[89963]: 1.d scrub starts 2024-08-17T17:42:05.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:04 smithi053 ceph-mon[89963]: osdmap e352: 8 total, 8 up, 8 in 2024-08-17T17:42:05.290 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:05.316 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:05.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:05.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:05.418 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:05.442 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:05.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:05.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:05.519 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:42:05.544 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:05.570 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:05.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:05.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:05.673 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:05.698 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:05.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:05.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:05.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:42:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:42:05] "GET /metrics HTTP/1.1" 200 36511 "" "Prometheus/2.51.0" 2024-08-17T17:42:05.800 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:05.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:05.829 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:05.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:05.915 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:42:05.930 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:05.956 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:05.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:05.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:06.058 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:06.066 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:05 smithi003 ceph-mon[103344]: pgmap v1147: 119 pgs: 2 remapped+peering, 1 active+recovering+remapped, 2 active+recovery_wait+remapped, 114 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 403 op/s; 0 B/s, 51 objects/s recovering 2024-08-17T17:42:06.066 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:05 smithi003 ceph-mon[103344]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-08-17T17:42:06.066 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:05 smithi003 ceph-mon[103344]: osdmap e353: 8 total, 8 up, 8 in 2024-08-17T17:42:06.066 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:05 smithi003 ceph-mon[100615]: pgmap v1147: 119 pgs: 2 remapped+peering, 1 active+recovering+remapped, 2 active+recovery_wait+remapped, 114 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 403 op/s; 0 B/s, 51 objects/s recovering 2024-08-17T17:42:06.066 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:05 smithi003 ceph-mon[100615]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-08-17T17:42:06.067 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:05 smithi003 ceph-mon[100615]: osdmap e353: 8 total, 8 up, 8 in 2024-08-17T17:42:06.083 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:06.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:06.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:06.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:06.208 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:06.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:06.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:06.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:05 smithi053 ceph-mon[89963]: pgmap v1147: 119 pgs: 2 remapped+peering, 1 active+recovering+remapped, 2 active+recovery_wait+remapped, 114 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 403 op/s; 0 B/s, 51 objects/s recovering 2024-08-17T17:42:06.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:05 smithi053 ceph-mon[89963]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-08-17T17:42:06.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:05 smithi053 ceph-mon[89963]: osdmap e353: 8 total, 8 up, 8 in 2024-08-17T17:42:06.310 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:06.333 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:06.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:06.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:06.435 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:06.462 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:06.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:06.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:06.564 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:06.591 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:06.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:06.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:06.692 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:06.717 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:06.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:06.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:06.819 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:06.845 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:06.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:06.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:06.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:06.972 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:06.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:06.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:07.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:06 smithi003 ceph-mon[103344]: osdmap e354: 8 total, 8 up, 8 in 2024-08-17T17:42:07.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:06 smithi003 ceph-mon[103344]: 1.15 scrub starts 2024-08-17T17:42:07.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:06 smithi003 ceph-mon[103344]: 1.15 scrub ok 2024-08-17T17:42:07.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:06 smithi003 ceph-mon[100615]: osdmap e354: 8 total, 8 up, 8 in 2024-08-17T17:42:07.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:06 smithi003 ceph-mon[100615]: 1.15 scrub starts 2024-08-17T17:42:07.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:06 smithi003 ceph-mon[100615]: 1.15 scrub ok 2024-08-17T17:42:07.075 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:07.098 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:07.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:07.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:07.200 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:07.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:06 smithi053 ceph-mon[89963]: osdmap e354: 8 total, 8 up, 8 in 2024-08-17T17:42:07.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:06 smithi053 ceph-mon[89963]: 1.15 scrub starts 2024-08-17T17:42:07.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:06 smithi053 ceph-mon[89963]: 1.15 scrub ok 2024-08-17T17:42:07.230 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:07.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:07.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:07.331 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:07.358 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:07.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:07.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:07.460 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:07.483 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:07.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:07.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:07.585 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:07.611 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:07.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:07.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:07.713 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:07.738 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:07.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:07.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:07.772 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:42:07.772 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-08-17T17:42:07.772 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:42:07.840 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:07.864 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:07.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:07.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:07.966 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:07.993 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:07.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:07.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:08.022 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:07 smithi003 ceph-mon[100615]: pgmap v1151: 119 pgs: 1 active+recovering+remapped, 1 peering, 4 remapped+peering, 2 active+recovery_wait+remapped, 111 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 11 KiB/s rd, 31 KiB/s wr, 7 op/s; 22 KiB/s, 225 objects/s recovering 2024-08-17T17:42:08.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:07 smithi003 ceph-mon[100615]: 1.17 deep-scrub starts 2024-08-17T17:42:08.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:07 smithi003 ceph-mon[100615]: 1.17 deep-scrub ok 2024-08-17T17:42:08.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:07 smithi003 ceph-mon[100615]: 1.1e scrub starts 2024-08-17T17:42:08.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:07 smithi003 ceph-mon[100615]: 1.1e scrub ok 2024-08-17T17:42:08.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:07 smithi003 ceph-mon[103344]: pgmap v1151: 119 pgs: 1 active+recovering+remapped, 1 peering, 4 remapped+peering, 2 active+recovery_wait+remapped, 111 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 11 KiB/s rd, 31 KiB/s wr, 7 op/s; 22 KiB/s, 225 objects/s recovering 2024-08-17T17:42:08.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:07 smithi003 ceph-mon[103344]: 1.17 deep-scrub starts 2024-08-17T17:42:08.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:07 smithi003 ceph-mon[103344]: 1.17 deep-scrub ok 2024-08-17T17:42:08.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:07 smithi003 ceph-mon[103344]: 1.1e scrub starts 2024-08-17T17:42:08.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:07 smithi003 ceph-mon[103344]: 1.1e scrub ok 2024-08-17T17:42:08.095 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:08.121 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:08.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:08.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:08.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:07 smithi053 ceph-mon[89963]: pgmap v1151: 119 pgs: 1 active+recovering+remapped, 1 peering, 4 remapped+peering, 2 active+recovery_wait+remapped, 111 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 11 KiB/s rd, 31 KiB/s wr, 7 op/s; 22 KiB/s, 225 objects/s recovering 2024-08-17T17:42:08.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:07 smithi053 ceph-mon[89963]: 1.17 deep-scrub starts 2024-08-17T17:42:08.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:07 smithi053 ceph-mon[89963]: 1.17 deep-scrub ok 2024-08-17T17:42:08.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:07 smithi053 ceph-mon[89963]: 1.1e scrub starts 2024-08-17T17:42:08.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:07 smithi053 ceph-mon[89963]: 1.1e scrub ok 2024-08-17T17:42:08.223 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:08.248 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:08.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:08.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:08.350 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:08.373 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:08.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:08.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:08.475 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:08.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:08.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:08.604 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:08.627 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:08.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:08.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:08.754 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:08.777 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:08.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:08.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:08.878 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:08.905 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:08.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:08.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:09.007 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:09.030 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:09.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:09.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:09.131 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:09.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:09.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:09.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:09.255 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:09.278 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:09.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:09.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:09.359 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:42:09.360 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:42:09.380 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:09.407 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:09.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:09.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:09 smithi053 ceph-mon[89963]: 1.16 scrub starts 2024-08-17T17:42:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:09 smithi053 ceph-mon[89963]: 1.16 scrub ok 2024-08-17T17:42:09.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:42:09.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:42:09.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:09 smithi053 ceph-mon[89963]: pgmap v1152: 119 pgs: 1 active+recovering+remapped, 1 peering, 4 remapped+peering, 2 active+recovery_wait+remapped, 111 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 8.4 KiB/s rd, 24 KiB/s wr, 6 op/s; 17 KiB/s, 176 objects/s recovering 2024-08-17T17:42:09.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:09 smithi053 ceph-mon[89963]: 1.18 scrub starts 2024-08-17T17:42:09.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:09 smithi053 ceph-mon[89963]: 1.18 scrub ok 2024-08-17T17:42:09.509 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:09 smithi003 ceph-mon[103344]: 1.16 scrub starts 2024-08-17T17:42:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:09 smithi003 ceph-mon[103344]: 1.16 scrub ok 2024-08-17T17:42:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:42:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:42:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:09 smithi003 ceph-mon[103344]: pgmap v1152: 119 pgs: 1 active+recovering+remapped, 1 peering, 4 remapped+peering, 2 active+recovery_wait+remapped, 111 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 8.4 KiB/s rd, 24 KiB/s wr, 6 op/s; 17 KiB/s, 176 objects/s recovering 2024-08-17T17:42:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:09 smithi003 ceph-mon[103344]: 1.18 scrub starts 2024-08-17T17:42:09.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:09 smithi003 ceph-mon[103344]: 1.18 scrub ok 2024-08-17T17:42:09.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:09 smithi003 ceph-mon[100615]: 1.16 scrub starts 2024-08-17T17:42:09.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:09 smithi003 ceph-mon[100615]: 1.16 scrub ok 2024-08-17T17:42:09.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:42:09.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:42:09.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:09 smithi003 ceph-mon[100615]: pgmap v1152: 119 pgs: 1 active+recovering+remapped, 1 peering, 4 remapped+peering, 2 active+recovery_wait+remapped, 111 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 8.4 KiB/s rd, 24 KiB/s wr, 6 op/s; 17 KiB/s, 176 objects/s recovering 2024-08-17T17:42:09.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:09 smithi003 ceph-mon[100615]: 1.18 scrub starts 2024-08-17T17:42:09.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:09 smithi003 ceph-mon[100615]: 1.18 scrub ok 2024-08-17T17:42:09.536 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:09.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:09.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:09.638 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:09.663 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:09.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:09.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:09.764 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:09.787 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:09.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:09.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:09.888 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:09.915 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:09.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:09.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:10.017 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:10.039 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:10.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:10.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:10.141 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:10.164 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:10.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:10.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:10.266 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:10.290 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:10.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:10.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:10.392 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:10.418 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:10.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:10.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:10.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:10 smithi053 ceph-mon[89963]: Health check update: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-08-17T17:42:10.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:42:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:42:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:42:10.520 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:10.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:10 smithi003 ceph-mon[100615]: Health check update: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-08-17T17:42:10.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:10 smithi003 ceph-mon[103344]: Health check update: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-08-17T17:42:10.544 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:10.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:10.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:10.645 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:10.672 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:10.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:10.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:10.774 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:10.796 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:10.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:10.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:10.898 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:10.925 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:10.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:10.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:11.027 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:11.045 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:42:11.053 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:11.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:11.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:11.155 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:11.182 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:11.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:11.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:11.283 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:11.310 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:11.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:11.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:11.412 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:11.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:11.439 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:11.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:11.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:11 smithi053 ceph-mon[89963]: pgmap v1153: 119 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+remapped, 116 active+clean; 4.8 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 6.2 KiB/s rd, 18 KiB/s wr, 4 op/s; 13 KiB/s, 132 objects/s recovering 2024-08-17T17:42:11.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:11 smithi053 ceph-mon[89963]: 1.d scrub starts 2024-08-17T17:42:11.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:11 smithi003 ceph-mon[103344]: pgmap v1153: 119 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+remapped, 116 active+clean; 4.8 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 6.2 KiB/s rd, 18 KiB/s wr, 4 op/s; 13 KiB/s, 132 objects/s recovering 2024-08-17T17:42:11.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:11 smithi003 ceph-mon[103344]: 1.d scrub starts 2024-08-17T17:42:11.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:11 smithi003 ceph-mon[100615]: pgmap v1153: 119 pgs: 1 active+recovering+remapped, 2 active+recovery_wait+remapped, 116 active+clean; 4.8 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 6.2 KiB/s rd, 18 KiB/s wr, 4 op/s; 13 KiB/s, 132 objects/s recovering 2024-08-17T17:42:11.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:11 smithi003 ceph-mon[100615]: 1.d scrub starts 2024-08-17T17:42:11.540 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:11.564 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:11.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:11.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:11.666 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:11.693 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:11.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:11.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:11.795 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:11.821 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:11.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:11.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:11.923 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:11.949 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:11.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:11.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:12.051 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:12.078 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:12.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:12.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:12.179 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:12.206 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:12.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:12.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:12.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:12.330 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:12.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:12.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:12.432 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:12.455 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:12.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:12.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:12.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:12 smithi053 ceph-mon[89963]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-08-17T17:42:12.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:12 smithi053 ceph-mon[89963]: Cluster is now healthy 2024-08-17T17:42:12.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:12 smithi053 ceph-mon[89963]: 1.d scrub starts 2024-08-17T17:42:12.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:12 smithi053 ceph-mon[89963]: 1.1c scrub starts 2024-08-17T17:42:12.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:12 smithi053 ceph-mon[89963]: 1.1c scrub ok 2024-08-17T17:42:12.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:12 smithi003 ceph-mon[103344]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-08-17T17:42:12.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:12 smithi003 ceph-mon[103344]: Cluster is now healthy 2024-08-17T17:42:12.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:12 smithi003 ceph-mon[103344]: 1.d scrub starts 2024-08-17T17:42:12.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:12 smithi003 ceph-mon[103344]: 1.1c scrub starts 2024-08-17T17:42:12.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:12 smithi003 ceph-mon[103344]: 1.1c scrub ok 2024-08-17T17:42:12.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:12 smithi003 ceph-mon[100615]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-08-17T17:42:12.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:12 smithi003 ceph-mon[100615]: Cluster is now healthy 2024-08-17T17:42:12.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:12 smithi003 ceph-mon[100615]: 1.d scrub starts 2024-08-17T17:42:12.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:12 smithi003 ceph-mon[100615]: 1.1c scrub starts 2024-08-17T17:42:12.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:12 smithi003 ceph-mon[100615]: 1.1c scrub ok 2024-08-17T17:42:12.557 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:12.580 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:12.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:12.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:12.682 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:12.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:12.706 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:12.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:12.807 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:12.830 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:12.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:12.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:12.932 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:12.956 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:12.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:12.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:13.057 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:13.080 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:13.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:13.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:13.182 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:13.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:13.210 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:13.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:13.311 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:13.333 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:13.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:13.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:13.435 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:13.457 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:13.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:13.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:13.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:13 smithi053 ceph-mon[89963]: pgmap v1154: 119 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+remapped, 115 active+clean; 4.8 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 219 objects/s recovering 2024-08-17T17:42:13.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:13 smithi003 ceph-mon[103344]: pgmap v1154: 119 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+remapped, 115 active+clean; 4.8 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 219 objects/s recovering 2024-08-17T17:42:13.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:13 smithi003 ceph-mon[100615]: pgmap v1154: 119 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+remapped, 115 active+clean; 4.8 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 219 objects/s recovering 2024-08-17T17:42:13.559 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:13.581 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:13.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:13.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:13.683 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:13.706 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:13.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:13.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:13.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:13.830 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:13.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:13.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:13.932 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:13.953 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:13.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:13.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:14.055 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:14.078 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:14.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:14.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:14.180 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:14.202 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:14.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:14.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:14.304 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:14.327 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:14.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:14.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:14.428 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:14.455 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:14.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:14.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:14.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:14 smithi053 ceph-mon[89963]: 1.d scrub starts 2024-08-17T17:42:14.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:14 smithi003 ceph-mon[103344]: 1.d scrub starts 2024-08-17T17:42:14.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:14 smithi003 ceph-mon[100615]: 1.d scrub starts 2024-08-17T17:42:14.557 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:14.584 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:14.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:14.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:14.685 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:14.708 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:14.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:14.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:14.809 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:14.836 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:14.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:14.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:14.938 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:14.965 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:14.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:14.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:15.067 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:15.090 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:15.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:15.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:15.192 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:15.215 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:15.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:15.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:15.316 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:15.340 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:15.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:15.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:15.441 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:15.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:15 smithi053 ceph-mon[89963]: pgmap v1155: 119 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+remapped, 115 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 175 objects/s recovering 2024-08-17T17:42:15.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:15 smithi053 ceph-mon[89963]: 1.d deep-scrub starts 2024-08-17T17:42:15.468 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:15.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:15.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:15.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:15 smithi003 ceph-mon[103344]: pgmap v1155: 119 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+remapped, 115 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 175 objects/s recovering 2024-08-17T17:42:15.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:15 smithi003 ceph-mon[103344]: 1.d deep-scrub starts 2024-08-17T17:42:15.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:15 smithi003 ceph-mon[100615]: pgmap v1155: 119 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+remapped, 115 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 175 objects/s recovering 2024-08-17T17:42:15.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:15 smithi003 ceph-mon[100615]: 1.d deep-scrub starts 2024-08-17T17:42:15.569 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:15.596 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:15.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:15.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:15.697 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:15.724 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:15.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:15.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:15.825 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:15.848 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:15.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:15.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:15.950 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:15.972 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:15.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:15.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:16.022 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:42:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:42:15] "GET /metrics HTTP/1.1" 200 36515 "" "Prometheus/2.51.0" 2024-08-17T17:42:16.074 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:16.097 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:16.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:16.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:16.149 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_4' removed 2024-08-17T17:42:16.198 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:16.224 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:16.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:16.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:16.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:16.348 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:16.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:16.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:16.450 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:16.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:16 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1545196419' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:42:16.472 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:16.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:16.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:16.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:16 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1545196419' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:42:16.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:16 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1545196419' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:42:16.575 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:16.603 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:16.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:16.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:16.705 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:16.728 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:16.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:16.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:16.829 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:16.853 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:16.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:16.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:16.955 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:16.977 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:16.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:16.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:17.079 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:17.102 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:17.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:17.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:17.204 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:17.229 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:17.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:17.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:17.331 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:17.357 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:17.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:17.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:17.459 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:17.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:17 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1545196419' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T17:42:17.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:17 smithi053 ceph-mon[89963]: osdmap e355: 8 total, 8 up, 8 in 2024-08-17T17:42:17.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:17 smithi053 ceph-mon[89963]: pgmap v1157: 103 pgs: 1 active+clean+scrubbing, 102 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:17.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:17 smithi053 ceph-mon[89963]: 1.d scrub starts 2024-08-17T17:42:17.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:17 smithi053 ceph-mon[89963]: 1.d scrub ok 2024-08-17T17:42:17.483 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:17.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:17.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:17.522 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:17 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1545196419' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T17:42:17.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:17 smithi003 ceph-mon[100615]: osdmap e355: 8 total, 8 up, 8 in 2024-08-17T17:42:17.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:17 smithi003 ceph-mon[100615]: pgmap v1157: 103 pgs: 1 active+clean+scrubbing, 102 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:17.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:17 smithi003 ceph-mon[100615]: 1.d scrub starts 2024-08-17T17:42:17.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:17 smithi003 ceph-mon[100615]: 1.d scrub ok 2024-08-17T17:42:17.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:17 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1545196419' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T17:42:17.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:17 smithi003 ceph-mon[103344]: osdmap e355: 8 total, 8 up, 8 in 2024-08-17T17:42:17.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:17 smithi003 ceph-mon[103344]: pgmap v1157: 103 pgs: 1 active+clean+scrubbing, 102 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:17.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:17 smithi003 ceph-mon[103344]: 1.d scrub starts 2024-08-17T17:42:17.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:17 smithi003 ceph-mon[103344]: 1.d scrub ok 2024-08-17T17:42:17.585 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:17.609 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:17.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:17.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:17.710 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:17.737 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:17.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:17.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:17.839 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:17.864 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:17.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:17.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:17.966 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:17.994 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:17.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:17.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:18.096 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:18.122 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:18.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:18.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:18.224 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:18.250 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:18.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:18.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:18.352 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:18.378 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:18.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:18.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:18.480 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:18.506 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:18.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:18.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:18.608 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:18.614 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-17T17:42:18.632 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:18.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:18.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:18.734 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:18.756 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:18.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:18.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:18.858 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:18.885 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:18.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:18.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:18.986 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:19.010 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:19.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:19.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:19.111 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:19.134 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:19.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:19.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:19.236 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:19.259 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:19.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:19.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:19.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:19.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:19.385 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:19.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:19.486 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:19.510 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:19.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:19.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:19.615 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:19.639 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:19.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:19.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:19.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:19 smithi053 ceph-mon[89963]: pgmap v1158: 103 pgs: 1 active+clean+scrubbing, 102 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:19.741 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:19.768 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:19.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:19.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:19.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:19 smithi003 ceph-mon[103344]: pgmap v1158: 103 pgs: 1 active+clean+scrubbing, 102 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:19.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:19 smithi003 ceph-mon[100615]: pgmap v1158: 103 pgs: 1 active+clean+scrubbing, 102 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:19.869 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:19.896 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:19.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:19.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:19.998 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:20.025 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:20.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:20.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:20.127 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:20.153 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:20.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:20.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:20.255 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:20.282 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:20.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:20.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:20.383 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:20.410 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:20.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:20.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:20.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:20 smithi053 ceph-mon[89963]: 1.1d deep-scrub starts 2024-08-17T17:42:20.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:20 smithi053 ceph-mon[89963]: 1.1d deep-scrub ok 2024-08-17T17:42:20.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:20 smithi053 ceph-mon[89963]: 1.5 deep-scrub starts 2024-08-17T17:42:20.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:20 smithi053 ceph-mon[89963]: 1.5 deep-scrub ok 2024-08-17T17:42:20.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:42:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:42:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:42:20.511 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:20.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:20.540 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:20.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:20.641 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:20.667 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:20.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:20.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:20.769 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:20.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:20 smithi003 ceph-mon[100615]: 1.1d deep-scrub starts 2024-08-17T17:42:20.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:20 smithi003 ceph-mon[100615]: 1.1d deep-scrub ok 2024-08-17T17:42:20.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:20 smithi003 ceph-mon[100615]: 1.5 deep-scrub starts 2024-08-17T17:42:20.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:20 smithi003 ceph-mon[100615]: 1.5 deep-scrub ok 2024-08-17T17:42:20.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:20 smithi003 ceph-mon[103344]: 1.1d deep-scrub starts 2024-08-17T17:42:20.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:20 smithi003 ceph-mon[103344]: 1.1d deep-scrub ok 2024-08-17T17:42:20.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:20 smithi003 ceph-mon[103344]: 1.5 deep-scrub starts 2024-08-17T17:42:20.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:20 smithi003 ceph-mon[103344]: 1.5 deep-scrub ok 2024-08-17T17:42:20.795 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:20.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:20.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:20.897 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:20.924 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:20.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:20.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:21.026 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:21.053 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:21.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:21.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:21.155 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:21.182 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:21.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:21.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:21.283 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:21.310 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:21.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:21.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:21.412 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:21.438 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:21.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:21.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:21.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:21 smithi003 ceph-mon[100615]: pgmap v1159: 103 pgs: 1 active+clean+scrubbing, 102 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:21.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:21 smithi003 ceph-mon[100615]: 1.e scrub starts 2024-08-17T17:42:21.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:21 smithi003 ceph-mon[100615]: 1.e scrub ok 2024-08-17T17:42:21.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:21 smithi003 ceph-mon[103344]: pgmap v1159: 103 pgs: 1 active+clean+scrubbing, 102 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:21.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:21 smithi003 ceph-mon[103344]: 1.e scrub starts 2024-08-17T17:42:21.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:21 smithi003 ceph-mon[103344]: 1.e scrub ok 2024-08-17T17:42:21.540 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:21.567 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:21.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:21.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:21.669 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:21.696 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:21.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:21.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:21.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:21 smithi053 ceph-mon[89963]: pgmap v1159: 103 pgs: 1 active+clean+scrubbing, 102 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:21.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:21 smithi053 ceph-mon[89963]: 1.e scrub starts 2024-08-17T17:42:21.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:21 smithi053 ceph-mon[89963]: 1.e scrub ok 2024-08-17T17:42:21.798 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:21.825 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:21.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:21.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:21.926 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:21.953 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:21.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:21.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:22.054 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:22.081 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:22.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:22.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:22.183 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:22.210 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:22.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:22.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:22.311 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:22.338 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:22.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:22.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:22.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:22.466 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:22.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:22.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:22.568 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:22.594 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:22.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:22.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:22.696 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:22.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:22 smithi053 ceph-mon[89963]: 1.1a deep-scrub starts 2024-08-17T17:42:22.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:22 smithi053 ceph-mon[89963]: 1.1a deep-scrub ok 2024-08-17T17:42:22.722 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:22.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:22.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:22.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:22 smithi003 ceph-mon[100615]: 1.1a deep-scrub starts 2024-08-17T17:42:22.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:22 smithi003 ceph-mon[100615]: 1.1a deep-scrub ok 2024-08-17T17:42:22.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:22 smithi003 ceph-mon[103344]: 1.1a deep-scrub starts 2024-08-17T17:42:22.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:22 smithi003 ceph-mon[103344]: 1.1a deep-scrub ok 2024-08-17T17:42:22.823 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:22.848 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:22.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:22.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:22.950 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:22.976 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:22.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:22.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:23.078 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:23.105 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:23.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:23.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:23.207 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:23.233 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:23.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:23.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:23.335 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:23.362 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:23.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:23.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:23.464 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:23.491 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:23.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:23.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:23.592 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:23.620 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:23.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:23.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:23 smithi053 ceph-mon[89963]: pgmap v1160: 103 pgs: 103 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:42:23.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:42:23.721 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:23.748 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:23.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:23.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:23.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:23 smithi003 ceph-mon[103344]: pgmap v1160: 103 pgs: 103 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:23.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:42:23.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:42:23.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:23 smithi003 ceph-mon[100615]: pgmap v1160: 103 pgs: 103 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:23.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:42:23.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:42:23.850 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:23.877 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:23.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:23.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:23.979 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:24.005 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:24.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:24.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:24.107 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:24.134 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:24.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:24.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:24.236 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:24.263 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:24.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:24.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:24.364 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:24.391 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:24.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:24.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:24.493 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:24.520 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:24.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:24.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:24.622 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:24.649 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:24.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:24.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:24.750 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:24.777 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:24.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:24.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:24.878 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:24.888 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:24 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:42:24.888 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:24 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:42:24.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:24.906 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:24.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:24.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:24 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:42:25.008 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:25.035 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:25.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:25.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:25.136 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:25.163 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:25.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:25.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:25.265 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:25.292 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:25.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:25.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:25.394 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:25.421 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:25.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:25.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:25.523 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:25.550 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:25.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:25.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:25.652 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:25.675 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:25.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:25.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:25.777 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:25.804 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:25.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:25.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:25.868 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:25 smithi003 ceph-mon[103344]: pgmap v1161: 103 pgs: 103 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:25.868 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:25 smithi003 ceph-mon[100615]: pgmap v1161: 103 pgs: 103 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:25.869 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:42:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:42:25] "GET /metrics HTTP/1.1" 200 34158 "" "Prometheus/2.51.0" 2024-08-17T17:42:25.906 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:25.933 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:25.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:25.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:25.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:25 smithi053 ceph-mon[89963]: pgmap v1161: 103 pgs: 103 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:26.035 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:26.060 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:26.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:26.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:26.162 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:26.189 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:26.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:26.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:26.290 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:26.317 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:26.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:26.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:26.320 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-08-17T17:42:26.321 INFO:tasks.radosbench:Beginning radosbench... 2024-08-17T17:42:26.321 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_5 2024-08-17T17:42:26.321 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd pool create unique_pool_5 16 2024-08-17T17:42:26.418 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:26.443 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:26.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:26.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:26.545 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:26.571 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:26.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:26.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:26.673 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:26.696 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:26.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:26.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:26.778 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:42:26.798 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:26.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:26.821 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:26.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:26.922 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:26.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:26.946 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:26.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:27.047 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:27.074 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:27.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:27.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:27.176 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:27.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:27.199 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:27.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:27.300 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:27.323 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:27.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:27.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:27.345 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:42:27.424 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:27.451 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:27.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:27.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:27.552 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:27.578 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:27.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:27.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:27.680 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:27.702 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:27.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:27.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:27.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:27.826 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:27.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:27.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:27.928 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:27.955 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:27.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:27.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:27.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:27 smithi053 ceph-mon[89963]: pgmap v1162: 103 pgs: 103 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:27.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:27 smithi053 ceph-mon[89963]: 1.1b scrub starts 2024-08-17T17:42:27.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:27 smithi053 ceph-mon[89963]: 1.1b scrub ok 2024-08-17T17:42:28.000 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:42:28.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:27 smithi003 ceph-mon[103344]: pgmap v1162: 103 pgs: 103 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:28.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:27 smithi003 ceph-mon[103344]: 1.1b scrub starts 2024-08-17T17:42:28.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:27 smithi003 ceph-mon[103344]: 1.1b scrub ok 2024-08-17T17:42:28.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:27 smithi003 ceph-mon[100615]: pgmap v1162: 103 pgs: 103 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:28.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:27 smithi003 ceph-mon[100615]: 1.1b scrub starts 2024-08-17T17:42:28.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:27 smithi003 ceph-mon[100615]: 1.1b scrub ok 2024-08-17T17:42:28.056 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:28.083 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:28.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:28.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:28.184 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:28.211 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:28.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:28.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:28.313 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:28.339 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:28.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:28.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:28.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:28.463 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:28.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:28.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:28.564 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:28.591 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:28.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:28.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:28.693 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:28.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:28.717 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:28.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:28.818 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:28.845 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:28.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:28.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:28.947 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:28.974 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:28.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:28.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:29.076 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:29.099 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:29.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:29.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:29.213 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:29.237 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:29.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:29.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:29.338 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:29.360 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:29.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:29.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:29.462 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:29.489 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:29.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:29.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:29.591 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:29.616 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:29.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:29.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:29.718 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:29.744 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:29.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:29.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:29.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:29 smithi003 ceph-mon[100615]: pgmap v1163: 103 pgs: 103 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:29.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:29 smithi003 ceph-mon[103344]: pgmap v1163: 103 pgs: 103 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:29.845 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:29.868 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:29.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:29.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:29.918 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-08-17T17:42:29.918 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:42:29.918 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-08-17T17:42:29.918 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 1.000000 2024-08-17T17:42:29.918 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 4 1 2024-08-17T17:42:29.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:29 smithi053 ceph-mon[89963]: pgmap v1163: 103 pgs: 103 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:29.970 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:29.997 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:29.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:29.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:30.099 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:30.126 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:30.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:30.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:30.228 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:30.255 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:30.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:30.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:30.356 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:30.383 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:30.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:30.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:30.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:42:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:42:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:42:30.485 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:30.510 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:30.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:30.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:30.595 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-08-17T17:42:30.612 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:30.638 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:30.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:30.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:30.740 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:30.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:30.766 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:30.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:30.867 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:30.894 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:30.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:30.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:30.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:30 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/734925808' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:42:30.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:30 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:42:30.996 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:31.021 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:31.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:31.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:31.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:30 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/734925808' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:42:31.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:30 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:42:31.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:30 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/734925808' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:42:31.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:30 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:42:31.123 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:31.151 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:31.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:31.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:31.253 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:31.279 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:31.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:31.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:31.381 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:31.407 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:31.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:31.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:31.509 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:31.536 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:31.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:31.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:31.638 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:31.665 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:31.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:31.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:31.766 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:31.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:31 smithi003 ceph-mon[100615]: pgmap v1164: 103 pgs: 103 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:31.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:31 smithi003 ceph-mon[100615]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:42:31.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:31 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:42:31.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:31 smithi003 ceph-mon[100615]: osdmap e356: 8 total, 8 up, 8 in 2024-08-17T17:42:31.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:31 smithi003 ceph-mon[103344]: pgmap v1164: 103 pgs: 103 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:31.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:31 smithi003 ceph-mon[103344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:42:31.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:31 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:42:31.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:31 smithi003 ceph-mon[103344]: osdmap e356: 8 total, 8 up, 8 in 2024-08-17T17:42:31.791 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:31.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:31.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:31.893 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:31.918 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:31.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:31.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:31.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:31 smithi053 ceph-mon[89963]: pgmap v1164: 103 pgs: 103 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:31.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:31 smithi053 ceph-mon[89963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:42:31.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:31 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:42:31.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:31 smithi053 ceph-mon[89963]: osdmap e356: 8 total, 8 up, 8 in 2024-08-17T17:42:32.020 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:32.047 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:32.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:32.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:32.148 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:32.176 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:32.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:32.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:32.278 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:32.301 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:32.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:32.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:32.403 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:32.430 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:32.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:32.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:32.532 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:32.558 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:32.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:32.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:32.660 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:32.686 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:32.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:32.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:32.788 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:32.813 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:32.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:32.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:32.914 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:32.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:32.938 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:32.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:32.994 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:42:32.994 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:42:33.010 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:42:33.010 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:42:33.027 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:42:33.027 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:42:33.039 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:33.043 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:42:33.044 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:42:33.060 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:42:33.060 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:42:33.061 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:33.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:33.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:33.077 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:42:33.077 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:42:33.093 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:42:33.094 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:42:33.110 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:42:33.110 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:42:33.164 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:33.186 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:33.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:33.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:33.288 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:33.309 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:33.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:33.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:33.411 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:33.438 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:33.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:33.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:33.540 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:33.566 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:33.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:33.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:33.667 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:33.693 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:33.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:33.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:33.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:33 smithi003 ceph-mon[100615]: pgmap v1166: 103 pgs: 103 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:33.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:33 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3675691955' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:42:33.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:33 smithi003 ceph-mon[103344]: pgmap v1166: 103 pgs: 103 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:33.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:33 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3675691955' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:42:33.794 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:33.820 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:33.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:33.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:33.922 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:33.945 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:33.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:33.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:33.957 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:42:33.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:33 smithi053 ceph-mon[89963]: pgmap v1166: 103 pgs: 103 active+clean; 4.6 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:33.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:33 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3675691955' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:42:34.046 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:34.071 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:34.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:34.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:34.173 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:34.197 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:34.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:34.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:34.296 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:42:34.296 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-08-17T17:42:34.296 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-08-17T17:42:34.296 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:42:34.296 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.296 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:68819 8.a 8:553c3286:::benchmark_data_smithi003_229103_object68818:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-08-17T17:42:34.296 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:34.995320+0000", 2024-08-17T17:42:34.296 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 538.60873356599996, 2024-08-17T17:42:34.297 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.21313156799999999, 2024-08-17T17:42:34.297 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:42:34.297 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:42:34.297 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:42:34.297 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:42:34.297 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:42:34.297 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:42:34.297 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68819 2024-08-17T17:42:34.297 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.297 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:42:34.297 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.297 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:42:34.297 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:34.995320+0000", 2024-08-17T17:42:34.297 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.298 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.298 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.298 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:42:34.298 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:34.995320+0000", 2024-08-17T17:42:34.298 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.298 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.298 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.298 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:42:34.298 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:34.995321+0000", 2024-08-17T17:42:34.298 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.781e-06 2024-08-17T17:42:34.298 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.298 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.298 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:42:34.299 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:34.995343+0000", 2024-08-17T17:42:34.299 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1611e-05 2024-08-17T17:42:34.299 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.299 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.299 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:42:34.299 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:34.995356+0000", 2024-08-17T17:42:34.299 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2924e-05 2024-08-17T17:42:34.299 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.299 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.299 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:42:34.299 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:34.995364+0000", 2024-08-17T17:42:34.299 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9030000000000004e-06 2024-08-17T17:42:34.300 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.300 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.300 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:42:34.300 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:34.996273+0000", 2024-08-17T17:42:34.300 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00090877299999999998 2024-08-17T17:42:34.300 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.300 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.300 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:42:34.300 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:34.996422+0000", 2024-08-17T17:42:34.301 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000149438 2024-08-17T17:42:34.301 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.301 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.301 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,5", 2024-08-17T17:42:34.301 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:34.996577+0000", 2024-08-17T17:42:34.301 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015460699999999999 2024-08-17T17:42:34.301 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.301 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.301 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:42:34.301 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.002635+0000", 2024-08-17T17:42:34.301 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0060584330000000002 2024-08-17T17:42:34.301 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.301 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.301 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:42:34.302 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.002900+0000", 2024-08-17T17:42:34.302 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00026447299999999999 2024-08-17T17:42:34.302 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.302 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.302 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:42:34.302 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.004421+0000", 2024-08-17T17:42:34.302 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0015214580000000001 2024-08-17T17:42:34.302 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.302 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.302 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:42:34.302 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.004672+0000", 2024-08-17T17:42:34.302 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00025048099999999999 2024-08-17T17:42:34.303 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.303 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.303 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:42:34.303 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.208451+0000", 2024-08-17T17:42:34.303 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.20377968599999999 2024-08-17T17:42:34.303 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.303 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:42:34.303 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.303 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.303 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.303 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:68829 8.a 8:5948bb26:::benchmark_data_smithi003_229103_object68828:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-08-17T17:42:34.303 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:35.004588+0000", 2024-08-17T17:42:34.303 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 538.59946537300004, 2024-08-17T17:42:34.303 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.243142357, 2024-08-17T17:42:34.304 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:42:34.304 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:42:34.304 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:42:34.304 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:42:34.304 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:42:34.304 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:42:34.304 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68829 2024-08-17T17:42:34.304 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.304 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:42:34.304 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.304 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:42:34.304 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.004588+0000", 2024-08-17T17:42:34.305 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.305 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.305 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.305 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:42:34.305 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.004588+0000", 2024-08-17T17:42:34.305 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.305 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.305 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.305 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:42:34.305 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.004589+0000", 2024-08-17T17:42:34.305 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5719999999999999e-06 2024-08-17T17:42:34.305 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.305 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.306 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:42:34.306 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.004615+0000", 2024-08-17T17:42:34.306 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5463e-05 2024-08-17T17:42:34.306 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.306 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.306 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:42:34.306 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.004625+0000", 2024-08-17T17:42:34.306 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.047e-05 2024-08-17T17:42:34.306 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.306 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.306 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:42:34.306 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.004632+0000", 2024-08-17T17:42:34.306 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4749999999999998e-06 2024-08-17T17:42:34.307 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.307 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.307 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:42:34.307 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.208513+0000", 2024-08-17T17:42:34.307 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.203881596 2024-08-17T17:42:34.307 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.307 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.307 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:42:34.307 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.208687+0000", 2024-08-17T17:42:34.307 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000173497 2024-08-17T17:42:34.307 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.307 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.307 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,5", 2024-08-17T17:42:34.307 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.208855+0000", 2024-08-17T17:42:34.307 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016800800000000001 2024-08-17T17:42:34.308 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.308 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.308 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:42:34.308 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.216609+0000", 2024-08-17T17:42:34.308 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0077542399999999999 2024-08-17T17:42:34.308 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.308 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.308 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:42:34.308 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.242427+0000", 2024-08-17T17:42:34.308 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.025817659 2024-08-17T17:42:34.308 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.308 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.308 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:42:34.309 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.243230+0000", 2024-08-17T17:42:34.309 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00080306699999999997 2024-08-17T17:42:34.309 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.309 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.309 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:42:34.309 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.243279+0000", 2024-08-17T17:42:34.309 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9490999999999998e-05 2024-08-17T17:42:34.309 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.309 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.309 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:42:34.309 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.247730+0000", 2024-08-17T17:42:34.309 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004450819 2024-08-17T17:42:34.310 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.310 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:42:34.310 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.310 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.310 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.310 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:68831 8.a 8:5ce5df0e:::benchmark_data_smithi003_229103_object68830:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e271)", 2024-08-17T17:42:34.310 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:35.008904+0000", 2024-08-17T17:42:34.310 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 538.59514874800004, 2024-08-17T17:42:34.310 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.24321522100000001, 2024-08-17T17:42:34.310 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:42:34.310 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:42:34.310 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:42:34.311 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:42:34.311 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:42:34.311 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:42:34.311 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 68831 2024-08-17T17:42:34.311 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.311 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:42:34.311 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.311 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:42:34.311 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.008904+0000", 2024-08-17T17:42:34.311 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.311 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.311 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.312 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:42:34.312 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.008904+0000", 2024-08-17T17:42:34.312 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.312 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.312 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.312 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:42:34.312 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.008906+0000", 2024-08-17T17:42:34.312 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8479999999999999e-06 2024-08-17T17:42:34.312 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.312 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.312 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:42:34.312 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.008931+0000", 2024-08-17T17:42:34.312 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4671000000000001e-05 2024-08-17T17:42:34.312 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.313 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.313 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:42:34.313 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.208380+0000", 2024-08-17T17:42:34.313 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.19944935799999999 2024-08-17T17:42:34.313 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.313 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.313 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:42:34.313 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.208400+0000", 2024-08-17T17:42:34.313 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9639999999999998e-05 2024-08-17T17:42:34.313 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.313 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.313 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:42:34.313 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.211549+0000", 2024-08-17T17:42:34.313 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031490720000000002 2024-08-17T17:42:34.314 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.314 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.314 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:42:34.314 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.211682+0000", 2024-08-17T17:42:34.314 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000133149 2024-08-17T17:42:34.314 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.314 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.314 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,5", 2024-08-17T17:42:34.314 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.212110+0000", 2024-08-17T17:42:34.314 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00042811999999999999 2024-08-17T17:42:34.314 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.314 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.314 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:42:34.315 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.216656+0000", 2024-08-17T17:42:34.315 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0045462020000000001 2024-08-17T17:42:34.315 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.315 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.315 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:42:34.315 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.242702+0000", 2024-08-17T17:42:34.315 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.026045216999999999 2024-08-17T17:42:34.315 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.315 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.315 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:42:34.315 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.247751+0000", 2024-08-17T17:42:34.315 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0050492649999999998 2024-08-17T17:42:34.315 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.315 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.316 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:42:34.316 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.249840+0000", 2024-08-17T17:42:34.316 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020893779999999998 2024-08-17T17:42:34.316 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.316 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.316 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:42:34.316 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:35.252120+0000", 2024-08-17T17:42:34.316 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0022793010000000001 2024-08-17T17:42:34.316 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.316 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:42:34.316 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.316 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.316 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.316 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:75541 8.d 8:b645ac60:::benchmark_data_smithi003_229103_object75540:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e272)", 2024-08-17T17:42:34.316 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:43.151663+0000", 2024-08-17T17:42:34.317 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 530.45239022600003, 2024-08-17T17:42:34.317 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0162890959999999, 2024-08-17T17:42:34.317 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:42:34.317 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:42:34.317 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:42:34.317 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:42:34.317 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:42:34.317 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:42:34.317 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 75541 2024-08-17T17:42:34.317 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.317 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:42:34.317 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.317 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:42:34.318 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.151663+0000", 2024-08-17T17:42:34.318 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.318 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.318 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.318 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:42:34.318 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.151663+0000", 2024-08-17T17:42:34.318 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.318 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.318 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.318 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:42:34.318 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.151665+0000", 2024-08-17T17:42:34.318 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5680000000000001e-06 2024-08-17T17:42:34.318 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.319 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.319 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:42:34.319 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.151691+0000", 2024-08-17T17:42:34.319 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6721e-05 2024-08-17T17:42:34.319 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.319 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.319 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:42:34.319 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.151703+0000", 2024-08-17T17:42:34.319 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2123000000000001e-05 2024-08-17T17:42:34.319 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.319 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.319 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:42:34.319 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.151712+0000", 2024-08-17T17:42:34.319 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1069999999999994e-06 2024-08-17T17:42:34.320 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.320 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.320 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:42:34.320 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.153978+0000", 2024-08-17T17:42:34.320 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0022650869999999998 2024-08-17T17:42:34.320 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.320 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.320 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:42:34.320 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.153987+0000", 2024-08-17T17:42:34.320 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4590000000000004e-06 2024-08-17T17:42:34.320 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.320 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.320 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:42:34.321 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.151869+0000", 2024-08-17T17:42:34.321 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99788173599999996 2024-08-17T17:42:34.321 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.321 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.321 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:42:34.321 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.157147+0000", 2024-08-17T17:42:34.321 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0052777579999999996 2024-08-17T17:42:34.321 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.321 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.321 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-08-17T17:42:34.321 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.157302+0000", 2024-08-17T17:42:34.321 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015505099999999999 2024-08-17T17:42:34.321 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.322 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.322 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:42:34.322 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.160327+0000", 2024-08-17T17:42:34.322 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00302521 2024-08-17T17:42:34.322 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.322 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.322 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:42:34.322 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.162728+0000", 2024-08-17T17:42:34.322 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024007960000000002 2024-08-17T17:42:34.322 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.322 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.322 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:42:34.322 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.167852+0000", 2024-08-17T17:42:34.323 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0051244209999999997 2024-08-17T17:42:34.323 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.323 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.323 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:42:34.323 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.167906+0000", 2024-08-17T17:42:34.323 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3971999999999998e-05 2024-08-17T17:42:34.323 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.323 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.323 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:42:34.323 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.167952+0000", 2024-08-17T17:42:34.323 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6087000000000001e-05 2024-08-17T17:42:34.323 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.324 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:42:34.324 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.324 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.324 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.324 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:75571 8.d 8:bd5a5eb6:::benchmark_data_smithi003_229103_object75570:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-08-17T17:42:34.324 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:43.205950+0000", 2024-08-17T17:42:34.324 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 530.39810292699997, 2024-08-17T17:42:34.324 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96226860400000003, 2024-08-17T17:42:34.324 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:42:34.324 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:42:34.324 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:42:34.324 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:42:34.324 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:42:34.324 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:42:34.324 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 75571 2024-08-17T17:42:34.325 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.325 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:42:34.325 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.325 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:42:34.325 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.205950+0000", 2024-08-17T17:42:34.325 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.325 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.325 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.325 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:42:34.325 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.205950+0000", 2024-08-17T17:42:34.325 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.325 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.325 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.326 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:42:34.326 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.205951+0000", 2024-08-17T17:42:34.326 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.82e-07 2024-08-17T17:42:34.326 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.326 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.326 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:42:34.326 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.205973+0000", 2024-08-17T17:42:34.326 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2419999999999999e-05 2024-08-17T17:42:34.326 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.326 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.326 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:42:34.326 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.205981+0000", 2024-08-17T17:42:34.326 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8460000000000008e-06 2024-08-17T17:42:34.327 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.327 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.327 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:42:34.327 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.205987+0000", 2024-08-17T17:42:34.327 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.66e-06 2024-08-17T17:42:34.327 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.327 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.327 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:42:34.327 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.206052+0000", 2024-08-17T17:42:34.327 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5110000000000005e-05 2024-08-17T17:42:34.327 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.327 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.327 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:42:34.327 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.206062+0000", 2024-08-17T17:42:34.328 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0213e-05 2024-08-17T17:42:34.328 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.328 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.328 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:42:34.328 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.160368+0000", 2024-08-17T17:42:34.328 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.95430568500000001 2024-08-17T17:42:34.328 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.328 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.328 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:42:34.328 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.160533+0000", 2024-08-17T17:42:34.328 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000165425 2024-08-17T17:42:34.328 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.328 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.328 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-08-17T17:42:34.329 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.160717+0000", 2024-08-17T17:42:34.329 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018439899999999999 2024-08-17T17:42:34.329 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.329 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.329 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:42:34.329 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.168035+0000", 2024-08-17T17:42:34.329 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0073178050000000001 2024-08-17T17:42:34.329 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.329 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.329 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:42:34.329 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.168089+0000", 2024-08-17T17:42:34.329 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3866000000000002e-05 2024-08-17T17:42:34.329 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.329 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.330 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:42:34.330 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.168141+0000", 2024-08-17T17:42:34.330 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2296999999999998e-05 2024-08-17T17:42:34.330 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.330 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.330 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:42:34.330 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.168181+0000", 2024-08-17T17:42:34.330 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9823000000000001e-05 2024-08-17T17:42:34.331 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.331 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.331 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:42:34.331 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.168219+0000", 2024-08-17T17:42:34.331 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7573000000000001e-05 2024-08-17T17:42:34.331 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.331 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:42:34.331 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.331 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.331 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.331 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:75591 8.d 8:b181f172:::benchmark_data_smithi003_229103_object75590:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-08-17T17:42:34.331 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:43.228856+0000", 2024-08-17T17:42:34.331 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 530.37519669799997, 2024-08-17T17:42:34.331 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.94263202599999996, 2024-08-17T17:42:34.331 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:42:34.332 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:42:34.332 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:42:34.332 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:42:34.332 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:42:34.332 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:42:34.332 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 75591 2024-08-17T17:42:34.332 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.332 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:42:34.332 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.332 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:42:34.332 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.228856+0000", 2024-08-17T17:42:34.332 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.332 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.332 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.332 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:42:34.333 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.228856+0000", 2024-08-17T17:42:34.333 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.333 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.333 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.333 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:42:34.333 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.228858+0000", 2024-08-17T17:42:34.333 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.875e-06 2024-08-17T17:42:34.333 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.333 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.333 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:42:34.333 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.228887+0000", 2024-08-17T17:42:34.333 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8532999999999999e-05 2024-08-17T17:42:34.333 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.333 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.334 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:42:34.334 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.228907+0000", 2024-08-17T17:42:34.334 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9675000000000001e-05 2024-08-17T17:42:34.334 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.334 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.334 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:42:34.334 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.228916+0000", 2024-08-17T17:42:34.334 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0839999999999998e-06 2024-08-17T17:42:34.334 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.334 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.334 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:42:34.334 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.228993+0000", 2024-08-17T17:42:34.334 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7296e-05 2024-08-17T17:42:34.335 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.335 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.335 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:42:34.335 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.229004+0000", 2024-08-17T17:42:34.335 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0998999999999999e-05 2024-08-17T17:42:34.335 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.335 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.335 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:42:34.335 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.162917+0000", 2024-08-17T17:42:34.335 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.93391301000000004 2024-08-17T17:42:34.335 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.335 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.335 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:42:34.336 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.163058+0000", 2024-08-17T17:42:34.336 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000141061 2024-08-17T17:42:34.336 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.336 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.336 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-08-17T17:42:34.336 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.163148+0000", 2024-08-17T17:42:34.336 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9523000000000003e-05 2024-08-17T17:42:34.336 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.336 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.336 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:42:34.336 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.168043+0000", 2024-08-17T17:42:34.336 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0048952730000000003 2024-08-17T17:42:34.336 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.336 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.336 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:42:34.337 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.171308+0000", 2024-08-17T17:42:34.337 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032651469999999999 2024-08-17T17:42:34.337 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.337 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.337 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:42:34.337 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.171426+0000", 2024-08-17T17:42:34.337 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011787500000000001 2024-08-17T17:42:34.337 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.337 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.338 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:42:34.338 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.171461+0000", 2024-08-17T17:42:34.338 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4678000000000002e-05 2024-08-17T17:42:34.338 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.338 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.338 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:42:34.338 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.171489+0000", 2024-08-17T17:42:34.338 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7997e-05 2024-08-17T17:42:34.339 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.339 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:42:34.339 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.339 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.339 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.339 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:75599 8.d 8:ba9bb339:::benchmark_data_smithi003_229103_object75598:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-08-17T17:42:34.339 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:43.238881+0000", 2024-08-17T17:42:34.339 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 530.36517252299996, 2024-08-17T17:42:34.339 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.93273590900000003, 2024-08-17T17:42:34.340 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:42:34.340 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:42:34.340 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:42:34.340 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:42:34.340 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:42:34.340 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:42:34.340 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 75599 2024-08-17T17:42:34.340 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.341 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:42:34.341 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.341 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:42:34.341 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.238881+0000", 2024-08-17T17:42:34.341 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.341 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.341 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.341 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:42:34.341 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.238881+0000", 2024-08-17T17:42:34.342 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.342 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.342 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.342 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:42:34.342 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.238883+0000", 2024-08-17T17:42:34.342 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.39e-06 2024-08-17T17:42:34.342 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.342 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.343 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:42:34.343 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.238915+0000", 2024-08-17T17:42:34.343 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1903000000000003e-05 2024-08-17T17:42:34.343 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.343 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.343 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:42:34.343 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.238938+0000", 2024-08-17T17:42:34.343 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3431e-05 2024-08-17T17:42:34.344 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.344 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.344 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:42:34.344 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.238948+0000", 2024-08-17T17:42:34.344 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8460000000000003e-06 2024-08-17T17:42:34.344 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.344 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.344 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:42:34.345 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.239035+0000", 2024-08-17T17:42:34.345 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6732999999999998e-05 2024-08-17T17:42:34.345 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.345 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.345 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:42:34.345 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.239049+0000", 2024-08-17T17:42:34.345 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4251e-05 2024-08-17T17:42:34.345 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.346 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.346 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:42:34.346 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.165247+0000", 2024-08-17T17:42:34.346 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.92619812199999996 2024-08-17T17:42:34.346 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.346 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.346 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:42:34.346 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.165384+0000", 2024-08-17T17:42:34.347 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013682899999999999 2024-08-17T17:42:34.347 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.347 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.347 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-08-17T17:42:34.347 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.165520+0000", 2024-08-17T17:42:34.347 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000135353 2024-08-17T17:42:34.347 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.347 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.348 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:42:34.348 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.171360+0000", 2024-08-17T17:42:34.348 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0058409029999999997 2024-08-17T17:42:34.348 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.348 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.348 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:42:34.348 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.171373+0000", 2024-08-17T17:42:34.348 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2265e-05 2024-08-17T17:42:34.348 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.349 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.349 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:42:34.349 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.171528+0000", 2024-08-17T17:42:34.349 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015536900000000001 2024-08-17T17:42:34.349 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.349 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.349 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:42:34.349 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.171578+0000", 2024-08-17T17:42:34.350 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9505999999999997e-05 2024-08-17T17:42:34.350 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.350 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.350 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:42:34.350 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.171617+0000", 2024-08-17T17:42:34.350 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9008000000000003e-05 2024-08-17T17:42:34.350 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.350 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:42:34.351 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.351 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.351 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.351 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:75607 8.d 8:b7147ca5:::benchmark_data_smithi003_229103_object75606:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-08-17T17:42:34.351 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:43.251892+0000", 2024-08-17T17:42:34.351 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 530.35216147200003, 2024-08-17T17:42:34.351 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.923633064, 2024-08-17T17:42:34.351 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:42:34.352 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:42:34.352 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:42:34.352 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:42:34.352 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:42:34.352 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:42:34.352 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 75607 2024-08-17T17:42:34.352 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.352 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:42:34.352 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.352 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:42:34.352 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.251892+0000", 2024-08-17T17:42:34.352 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.353 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.353 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.353 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:42:34.353 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.251892+0000", 2024-08-17T17:42:34.353 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.353 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.353 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.353 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:42:34.353 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.251894+0000", 2024-08-17T17:42:34.353 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3149999999999999e-06 2024-08-17T17:42:34.353 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.353 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.353 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:42:34.353 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.251919+0000", 2024-08-17T17:42:34.354 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5338999999999999e-05 2024-08-17T17:42:34.354 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.354 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.354 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:42:34.354 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.251941+0000", 2024-08-17T17:42:34.354 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1563e-05 2024-08-17T17:42:34.354 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.354 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.354 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:42:34.354 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.251950+0000", 2024-08-17T17:42:34.354 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1230000000000006e-06 2024-08-17T17:42:34.354 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.354 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.354 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:42:34.355 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.252090+0000", 2024-08-17T17:42:34.355 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000139593 2024-08-17T17:42:34.355 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.355 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.355 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:42:34.355 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.252100+0000", 2024-08-17T17:42:34.355 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0844e-05 2024-08-17T17:42:34.355 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.355 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.355 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:42:34.355 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.168279+0000", 2024-08-17T17:42:34.355 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.91617848099999999 2024-08-17T17:42:34.355 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.355 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.356 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:42:34.356 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.168489+0000", 2024-08-17T17:42:34.356 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00021019800000000001 2024-08-17T17:42:34.356 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.356 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.356 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-08-17T17:42:34.356 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.168628+0000", 2024-08-17T17:42:34.356 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013881399999999999 2024-08-17T17:42:34.356 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.356 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.356 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:42:34.356 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.173609+0000", 2024-08-17T17:42:34.356 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049808059999999999 2024-08-17T17:42:34.356 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.357 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.357 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:42:34.357 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.174679+0000", 2024-08-17T17:42:34.357 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010701370000000001 2024-08-17T17:42:34.357 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.357 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.357 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:42:34.357 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.175372+0000", 2024-08-17T17:42:34.357 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00069280000000000003 2024-08-17T17:42:34.357 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.357 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.357 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:42:34.357 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.175469+0000", 2024-08-17T17:42:34.358 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7516000000000006e-05 2024-08-17T17:42:34.358 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.358 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.358 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:42:34.358 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.175525+0000", 2024-08-17T17:42:34.358 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5535000000000003e-05 2024-08-17T17:42:34.358 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.358 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:42:34.358 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.358 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.358 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.358 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:85906 8.a 8:5389820b:::benchmark_data_smithi003_229103_object85905:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-08-17T17:42:34.358 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:56.705679+0000", 2024-08-17T17:42:34.359 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 516.89837456999999, 2024-08-17T17:42:34.359 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.23050036400000001, 2024-08-17T17:42:34.359 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:42:34.359 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:42:34.359 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:42:34.359 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:42:34.359 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:42:34.359 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:42:34.359 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 85906 2024-08-17T17:42:34.359 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.359 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:42:34.359 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.359 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:42:34.360 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.705679+0000", 2024-08-17T17:42:34.360 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.360 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.360 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.360 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:42:34.360 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.705679+0000", 2024-08-17T17:42:34.360 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.360 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.360 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.360 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:42:34.360 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.705679+0000", 2024-08-17T17:42:34.360 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9599999999999998e-07 2024-08-17T17:42:34.360 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.360 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.361 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:42:34.361 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.705705+0000", 2024-08-17T17:42:34.361 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5551e-05 2024-08-17T17:42:34.361 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.361 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.361 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:42:34.361 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.705715+0000", 2024-08-17T17:42:34.361 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.012e-05 2024-08-17T17:42:34.361 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.361 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.361 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:42:34.361 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.705723+0000", 2024-08-17T17:42:34.361 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.011e-06 2024-08-17T17:42:34.361 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.362 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.362 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:42:34.362 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.708726+0000", 2024-08-17T17:42:34.362 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0030030669999999999 2024-08-17T17:42:34.362 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.362 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.362 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:42:34.362 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.708983+0000", 2024-08-17T17:42:34.362 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00025700400000000002 2024-08-17T17:42:34.362 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.362 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.362 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,5", 2024-08-17T17:42:34.362 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.709177+0000", 2024-08-17T17:42:34.363 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019346599999999999 2024-08-17T17:42:34.363 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.363 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.363 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:42:34.363 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.712947+0000", 2024-08-17T17:42:34.363 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003770018 2024-08-17T17:42:34.363 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.363 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.363 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:42:34.363 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.712964+0000", 2024-08-17T17:42:34.363 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7527999999999999e-05 2024-08-17T17:42:34.363 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.363 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.363 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:42:34.364 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.740443+0000", 2024-08-17T17:42:34.364 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.027478529000000002 2024-08-17T17:42:34.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.364 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.364 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:42:34.364 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.933705+0000", 2024-08-17T17:42:34.364 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.19326238000000001 2024-08-17T17:42:34.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.364 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.364 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:42:34.364 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.936179+0000", 2024-08-17T17:42:34.364 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024738939999999999 2024-08-17T17:42:34.364 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.364 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:42:34.365 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.365 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:85913 8.a 8:56788f29:::benchmark_data_smithi003_229103_object85912:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-08-17T17:42:34.365 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:56.714103+0000", 2024-08-17T17:42:34.365 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 516.88994979100005, 2024-08-17T17:42:34.365 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.22400641600000001, 2024-08-17T17:42:34.365 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:42:34.365 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:42:34.365 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:42:34.365 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:42:34.365 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:42:34.365 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:42:34.365 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 85913 2024-08-17T17:42:34.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.366 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:42:34.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:42:34.366 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.714103+0000", 2024-08-17T17:42:34.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:42:34.366 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.714103+0000", 2024-08-17T17:42:34.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.367 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.367 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:42:34.367 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.714104+0000", 2024-08-17T17:42:34.367 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.13e-07 2024-08-17T17:42:34.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.367 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.367 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:42:34.367 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.714150+0000", 2024-08-17T17:42:34.367 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6300000000000001e-05 2024-08-17T17:42:34.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.367 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.367 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:42:34.367 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.714158+0000", 2024-08-17T17:42:34.368 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5449999999999998e-06 2024-08-17T17:42:34.368 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.368 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.368 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:42:34.368 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.714163+0000", 2024-08-17T17:42:34.368 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1440000000000002e-06 2024-08-17T17:42:34.368 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.368 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.368 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:42:34.368 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.715797+0000", 2024-08-17T17:42:34.368 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001633921 2024-08-17T17:42:34.368 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.368 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.368 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:42:34.368 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.715976+0000", 2024-08-17T17:42:34.369 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017951800000000001 2024-08-17T17:42:34.369 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.369 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.369 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,5", 2024-08-17T17:42:34.369 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.716139+0000", 2024-08-17T17:42:34.369 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016219699999999999 2024-08-17T17:42:34.369 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.369 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.369 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:42:34.369 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.719311+0000", 2024-08-17T17:42:34.369 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031727859999999999 2024-08-17T17:42:34.369 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.369 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.369 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:42:34.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.722660+0000", 2024-08-17T17:42:34.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0033487080000000002 2024-08-17T17:42:34.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.370 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.370 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:42:34.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.936261+0000", 2024-08-17T17:42:34.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.21360137200000001 2024-08-17T17:42:34.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.370 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.370 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:42:34.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.936362+0000", 2024-08-17T17:42:34.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000101028 2024-08-17T17:42:34.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:42:34.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.938110+0000", 2024-08-17T17:42:34.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001747384 2024-08-17T17:42:34.371 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.371 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:42:34.371 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.371 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:85917 8.a 8:5dc02d90:::benchmark_data_smithi003_229103_object85916:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-08-17T17:42:34.371 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:56.718628+0000", 2024-08-17T17:42:34.371 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 516.88542514799997, 2024-08-17T17:42:34.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.22129468199999999, 2024-08-17T17:42:34.372 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:42:34.372 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:42:34.372 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:42:34.372 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:42:34.372 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:42:34.372 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:42:34.372 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 85917 2024-08-17T17:42:34.372 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.372 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:42:34.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.372 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:42:34.372 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.718628+0000", 2024-08-17T17:42:34.372 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.373 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.373 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:42:34.373 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.718628+0000", 2024-08-17T17:42:34.373 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.373 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.373 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:42:34.373 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.718630+0000", 2024-08-17T17:42:34.373 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.714e-06 2024-08-17T17:42:34.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.373 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.373 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:42:34.373 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.718680+0000", 2024-08-17T17:42:34.374 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0294999999999999e-05 2024-08-17T17:42:34.374 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.374 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:42:34.374 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.718690+0000", 2024-08-17T17:42:34.374 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0417999999999999e-05 2024-08-17T17:42:34.374 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.374 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:42:34.374 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.718697+0000", 2024-08-17T17:42:34.374 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3180000000000003e-06 2024-08-17T17:42:34.374 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.375 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:42:34.375 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.719346+0000", 2024-08-17T17:42:34.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00064915700000000003 2024-08-17T17:42:34.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.375 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:42:34.375 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.719496+0000", 2024-08-17T17:42:34.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014981400000000001 2024-08-17T17:42:34.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.375 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,5", 2024-08-17T17:42:34.375 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.719640+0000", 2024-08-17T17:42:34.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014463599999999999 2024-08-17T17:42:34.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:42:34.376 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.722634+0000", 2024-08-17T17:42:34.376 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029938370000000001 2024-08-17T17:42:34.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:42:34.376 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.727944+0000", 2024-08-17T17:42:34.376 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0053097810000000004 2024-08-17T17:42:34.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:42:34.377 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.938193+0000", 2024-08-17T17:42:34.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.21024947799999999 2024-08-17T17:42:34.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:42:34.377 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.938269+0000", 2024-08-17T17:42:34.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5630000000000006e-05 2024-08-17T17:42:34.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:42:34.378 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.939923+0000", 2024-08-17T17:42:34.378 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016536039999999999 2024-08-17T17:42:34.378 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.378 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:42:34.378 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.378 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.46025.0:85917 8.a e274/269 ondisk, result = 0)", 2024-08-17T17:42:34.378 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:56.732543+0000", 2024-08-17T17:42:34.378 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 516.87151031099995, 2024-08-17T17:42:34.378 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.20739052099999999, 2024-08-17T17:42:34.378 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:42:34.378 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:42:34.379 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:42:34.379 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:42:34.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:42:34.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.732543+0000", 2024-08-17T17:42:34.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:42:34.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.732543+0000", 2024-08-17T17:42:34.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:42:34.380 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.732543+0000", 2024-08-17T17:42:34.380 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.58e-07 2024-08-17T17:42:34.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.380 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.380 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:42:34.380 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.732545+0000", 2024-08-17T17:42:34.380 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2069999999999998e-06 2024-08-17T17:42:34.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.380 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.380 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:42:34.380 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.732552+0000", 2024-08-17T17:42:34.380 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3249999999999996e-06 2024-08-17T17:42:34.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.381 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:42:34.381 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.732556+0000", 2024-08-17T17:42:34.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7230000000000004e-06 2024-08-17T17:42:34.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.381 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:42:34.381 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.938175+0000", 2024-08-17T17:42:34.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.205618942 2024-08-17T17:42:34.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.381 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:42:34.382 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.938186+0000", 2024-08-17T17:42:34.382 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0769e-05 2024-08-17T17:42:34.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.382 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.382 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:42:34.382 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.939933+0000", 2024-08-17T17:42:34.382 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001747197 2024-08-17T17:42:34.382 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.383 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:42:34.383 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.383 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.46025.0:85932 8.f e274/269 8:f9889587:::benchmark_data_smithi003_229103_object85931:head v 274'5457, mlcod=265'880)", 2024-08-17T17:42:34.383 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:56.744608+0000", 2024-08-17T17:42:34.383 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 516.85944483599997, 2024-08-17T17:42:34.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.210705691, 2024-08-17T17:42:34.383 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:42:34.383 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:42:34.383 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:42:34.383 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:42:34.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:42:34.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.744608+0000", 2024-08-17T17:42:34.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:42:34.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.744608+0000", 2024-08-17T17:42:34.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:42:34.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.744610+0000", 2024-08-17T17:42:34.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.767e-06 2024-08-17T17:42:34.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.385 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.385 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:42:34.385 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.744641+0000", 2024-08-17T17:42:34.385 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1337999999999997e-05 2024-08-17T17:42:34.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.385 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.385 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:42:34.385 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.933303+0000", 2024-08-17T17:42:34.385 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.188661099 2024-08-17T17:42:34.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.385 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.385 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:42:34.385 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.933316+0000", 2024-08-17T17:42:34.385 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3536000000000001e-05 2024-08-17T17:42:34.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.386 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:42:34.386 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.935987+0000", 2024-08-17T17:42:34.386 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002670614 2024-08-17T17:42:34.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.386 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:42:34.386 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.936006+0000", 2024-08-17T17:42:34.386 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9281999999999999e-05 2024-08-17T17:42:34.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.386 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:42:34.386 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.955229+0000", 2024-08-17T17:42:34.387 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.019222586999999999 2024-08-17T17:42:34.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.387 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:42:34.387 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:56.955314+0000", 2024-08-17T17:42:34.387 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5468000000000002e-05 2024-08-17T17:42:34.387 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.387 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:42:34.387 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.387 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36354.0:22720 9.6 e314/311 9:6a7b00cf:::benchmark_data_smithi003_261051_object22719:head v 314'1422, mlcod=314'1420)", 2024-08-17T17:42:34.387 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:37:58.657489+0000", 2024-08-17T17:42:34.388 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 274.94656376099999, 2024-08-17T17:42:34.388 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.095732504999999996, 2024-08-17T17:42:34.388 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:42:34.388 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:42:34.388 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:42:34.388 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:42:34.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.388 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:42:34.388 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.657489+0000", 2024-08-17T17:42:34.388 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.389 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:42:34.389 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.657489+0000", 2024-08-17T17:42:34.389 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.389 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:42:34.389 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.657491+0000", 2024-08-17T17:42:34.389 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3039999999999999e-06 2024-08-17T17:42:34.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.389 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:42:34.389 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.657516+0000", 2024-08-17T17:42:34.389 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5686000000000001e-05 2024-08-17T17:42:34.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.390 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:42:34.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.657533+0000", 2024-08-17T17:42:34.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6889e-05 2024-08-17T17:42:34.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.390 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:42:34.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.657542+0000", 2024-08-17T17:42:34.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5229999999999995e-06 2024-08-17T17:42:34.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.391 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:42:34.391 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.657627+0000", 2024-08-17T17:42:34.391 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4790999999999995e-05 2024-08-17T17:42:34.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.391 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:42:34.391 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.657648+0000", 2024-08-17T17:42:34.391 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1158999999999999e-05 2024-08-17T17:42:34.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.391 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:42:34.391 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.660228+0000", 2024-08-17T17:42:34.391 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025802329999999999 2024-08-17T17:42:34.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.392 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:42:34.392 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.753222+0000", 2024-08-17T17:42:34.392 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.092993919999999994 2024-08-17T17:42:34.392 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.392 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:42:34.392 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.392 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36354.0:22723 9.6 e314/311 9:6a56a808:::benchmark_data_smithi003_261051_object22722:head v 314'1423, mlcod=314'1420)", 2024-08-17T17:42:34.392 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:37:58.658821+0000", 2024-08-17T17:42:34.393 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 274.945232304, 2024-08-17T17:42:34.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.094450671999999999, 2024-08-17T17:42:34.393 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:42:34.393 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:42:34.393 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:42:34.393 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:42:34.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:42:34.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.658821+0000", 2024-08-17T17:42:34.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:42:34.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.658821+0000", 2024-08-17T17:42:34.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:42:34.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.658822+0000", 2024-08-17T17:42:34.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0279999999999999e-06 2024-08-17T17:42:34.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:42:34.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.658845+0000", 2024-08-17T17:42:34.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3519e-05 2024-08-17T17:42:34.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:42:34.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.658856+0000", 2024-08-17T17:42:34.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0686999999999999e-05 2024-08-17T17:42:34.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:42:34.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.658862+0000", 2024-08-17T17:42:34.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2949999999999999e-06 2024-08-17T17:42:34.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.396 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:42:34.396 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.659339+0000", 2024-08-17T17:42:34.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00047612499999999999 2024-08-17T17:42:34.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.396 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:42:34.396 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.659358+0000", 2024-08-17T17:42:34.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9692999999999999e-05 2024-08-17T17:42:34.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.396 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:42:34.396 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.753229+0000", 2024-08-17T17:42:34.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.093871011000000004 2024-08-17T17:42:34.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:42:34.397 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.753272+0000", 2024-08-17T17:42:34.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2314000000000002e-05 2024-08-17T17:42:34.397 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.397 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:42:34.397 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.397 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36354.0:22740 9.2 e314/311 9:48d0b398:::benchmark_data_smithi003_261051_object22739:head v 314'1437, mlcod=314'1435)", 2024-08-17T17:42:34.397 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:37:58.664609+0000", 2024-08-17T17:42:34.397 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 274.939444477, 2024-08-17T17:42:34.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.091359095000000001, 2024-08-17T17:42:34.397 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:42:34.397 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:42:34.398 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:42:34.398 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:42:34.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.398 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:42:34.398 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.664609+0000", 2024-08-17T17:42:34.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.398 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:42:34.398 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.664609+0000", 2024-08-17T17:42:34.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:42:34.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.664610+0000", 2024-08-17T17:42:34.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.291e-06 2024-08-17T17:42:34.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:42:34.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.664635+0000", 2024-08-17T17:42:34.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5326000000000001e-05 2024-08-17T17:42:34.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:42:34.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.753200+0000", 2024-08-17T17:42:34.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.088564697999999997 2024-08-17T17:42:34.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:42:34.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.753215+0000", 2024-08-17T17:42:34.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4684e-05 2024-08-17T17:42:34.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:42:34.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.753270+0000", 2024-08-17T17:42:34.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4922e-05 2024-08-17T17:42:34.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:42:34.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.753290+0000", 2024-08-17T17:42:34.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0832999999999999e-05 2024-08-17T17:42:34.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:42:34.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.755893+0000", 2024-08-17T17:42:34.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026026769999999999 2024-08-17T17:42:34.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:42:34.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.755968+0000", 2024-08-17T17:42:34.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4664000000000002e-05 2024-08-17T17:42:34.402 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.402 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:42:34.402 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.402 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36354.0:55396 9.6 e315/311 9:6d7d6566:::benchmark_data_smithi003_261051_object55395:head v 315'3425, mlcod=315'3421)", 2024-08-17T17:42:34.402 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:38:18.615557+0000", 2024-08-17T17:42:34.402 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 254.988496583, 2024-08-17T17:42:34.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.033345782999999997, 2024-08-17T17:42:34.402 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:42:34.402 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:42:34.402 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:42:34.403 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:42:34.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:42:34.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:18.615557+0000", 2024-08-17T17:42:34.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:42:34.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:18.615557+0000", 2024-08-17T17:42:34.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:42:34.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:18.615558+0000", 2024-08-17T17:42:34.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.469e-06 2024-08-17T17:42:34.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:42:34.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:18.615777+0000", 2024-08-17T17:42:34.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00021862499999999999 2024-08-17T17:42:34.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:42:34.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:18.615794+0000", 2024-08-17T17:42:34.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7204999999999999e-05 2024-08-17T17:42:34.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:42:34.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:18.615803+0000", 2024-08-17T17:42:34.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9190000000000005e-06 2024-08-17T17:42:34.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:42:34.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:18.615871+0000", 2024-08-17T17:42:34.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7995000000000002e-05 2024-08-17T17:42:34.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:42:34.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:18.615890+0000", 2024-08-17T17:42:34.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9400000000000001e-05 2024-08-17T17:42:34.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:42:34.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:18.648826+0000", 2024-08-17T17:42:34.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.032935706000000002 2024-08-17T17:42:34.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:42:34.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:18.648902+0000", 2024-08-17T17:42:34.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6464000000000005e-05 2024-08-17T17:42:34.406 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.406 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:42:34.406 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.407 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36354.0:97677 9.6 e325/311 9:6a9b1633:::benchmark_data_smithi003_261051_object97676:head v 325'6053, mlcod=325'6050)", 2024-08-17T17:42:34.407 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:38:54.226274+0000", 2024-08-17T17:42:34.407 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 219.37777934100001, 2024-08-17T17:42:34.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.032037085, 2024-08-17T17:42:34.407 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:42:34.407 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:42:34.407 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:42:34.407 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:42:34.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:42:34.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:54.226274+0000", 2024-08-17T17:42:34.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:42:34.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:54.226274+0000", 2024-08-17T17:42:34.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:42:34.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:54.226274+0000", 2024-08-17T17:42:34.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5499999999999998e-07 2024-08-17T17:42:34.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:42:34.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:54.226296+0000", 2024-08-17T17:42:34.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1852e-05 2024-08-17T17:42:34.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:42:34.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:54.226304+0000", 2024-08-17T17:42:34.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7139999999999993e-06 2024-08-17T17:42:34.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:42:34.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:54.226309+0000", 2024-08-17T17:42:34.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.237e-06 2024-08-17T17:42:34.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:42:34.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:54.230273+0000", 2024-08-17T17:42:34.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039643120000000002 2024-08-17T17:42:34.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:42:34.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:54.230324+0000", 2024-08-17T17:42:34.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0624000000000002e-05 2024-08-17T17:42:34.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:42:34.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:54.258270+0000", 2024-08-17T17:42:34.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.027945817000000001 2024-08-17T17:42:34.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:42:34.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:54.258311+0000", 2024-08-17T17:42:34.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1074e-05 2024-08-17T17:42:34.411 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.411 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:42:34.411 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.411 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36354.0:197939 9.2 e341/311 9:4e7d31d2:::benchmark_data_smithi003_261051_object82054:head v 341'12476, mlcod=340'12469)", 2024-08-17T17:42:34.412 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:41:12.304803+0000", 2024-08-17T17:42:34.412 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 81.299250435000005, 2024-08-17T17:42:34.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.028314441999999999, 2024-08-17T17:42:34.412 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:42:34.412 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:42:34.412 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:42:34.412 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:42:34.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:42:34.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:12.304803+0000", 2024-08-17T17:42:34.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:42:34.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:12.304803+0000", 2024-08-17T17:42:34.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:42:34.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:12.304803+0000", 2024-08-17T17:42:34.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.05e-07 2024-08-17T17:42:34.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:42:34.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:12.304812+0000", 2024-08-17T17:42:34.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6940000000000001e-06 2024-08-17T17:42:34.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:42:34.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:12.305254+0000", 2024-08-17T17:42:34.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000442051 2024-08-17T17:42:34.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:42:34.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:12.305260+0000", 2024-08-17T17:42:34.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8499999999999999e-06 2024-08-17T17:42:34.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:42:34.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:12.320609+0000", 2024-08-17T17:42:34.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.015349339 2024-08-17T17:42:34.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:42:34.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:12.320649+0000", 2024-08-17T17:42:34.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9848999999999997e-05 2024-08-17T17:42:34.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:42:34.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:12.330980+0000", 2024-08-17T17:42:34.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010331028000000001 2024-08-17T17:42:34.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:42:34.416 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:12.333117+0000", 2024-08-17T17:42:34.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002137426 2024-08-17T17:42:34.416 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.416 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:42:34.416 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.416 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36354.0:197946 9.2 e341/311 9:4f5106d0:::benchmark_data_smithi003_261051_object82061:head v 341'12477, mlcod=340'12469)", 2024-08-17T17:42:34.416 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:41:12.305271+0000", 2024-08-17T17:42:34.416 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 81.298782359, 2024-08-17T17:42:34.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.028884229000000001, 2024-08-17T17:42:34.416 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:42:34.416 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:42:34.416 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:42:34.417 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:42:34.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.417 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:42:34.417 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:12.305271+0000", 2024-08-17T17:42:34.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.417 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:42:34.417 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:12.305271+0000", 2024-08-17T17:42:34.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:42:34.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.417 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:42:34.417 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:12.305271+0000", 2024-08-17T17:42:34.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8400000000000005e-07 2024-08-17T17:42:34.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:42:34.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:12.305276+0000", 2024-08-17T17:42:34.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.758e-06 2024-08-17T17:42:34.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:42:34.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:12.305283+0000", 2024-08-17T17:42:34.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5849999999999996e-06 2024-08-17T17:42:34.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:42:34.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:12.305287+0000", 2024-08-17T17:42:34.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8160000000000001e-06 2024-08-17T17:42:34.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:42:34.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:12.325474+0000", 2024-08-17T17:42:34.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.020186681000000001 2024-08-17T17:42:34.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:42:34.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:12.325494+0000", 2024-08-17T17:42:34.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9868e-05 2024-08-17T17:42:34.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:42:34.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:12.333973+0000", 2024-08-17T17:42:34.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0084785409999999992 2024-08-17T17:42:34.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:42:34.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:42:34.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:42:34.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:41:12.334155+0000", 2024-08-17T17:42:34.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018249600000000001 2024-08-17T17:42:34.420 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.420 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:42:34.420 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.420 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:42:34.421 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:42:34.421 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:42:34.421 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:34.444 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:34.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:34.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:34.546 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:34.568 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:34.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:34.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:34.670 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:34.697 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:34.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:34.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:34.799 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:34.824 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:34.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:34.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:34.926 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:34.949 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:34.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:34.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:34.985 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:42:35.050 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:35.072 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:35.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:35.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:35.174 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:35.196 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:35.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:35.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:35.298 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:35.325 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:35.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:35.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:35.426 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:35.450 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:35.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:35.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:35.470 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-17T17:42:35.552 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:35.576 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:35.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:35.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:35.606 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_5' created 2024-08-17T17:42:35.678 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:35.700 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:35.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:35.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:35.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:35 smithi003 ceph-mon[103344]: pgmap v1167: 103 pgs: 103 active+clean; 4.6 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:35.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:35 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1749461462' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-08-17T17:42:35.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:35 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-08-17T17:42:35.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:35 smithi003 ceph-mon[100615]: pgmap v1167: 103 pgs: 103 active+clean; 4.6 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:35.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:35 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1749461462' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-08-17T17:42:35.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:35 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-08-17T17:42:35.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:42:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:42:35] "GET /metrics HTTP/1.1" 200 34158 "" "Prometheus/2.51.0" 2024-08-17T17:42:35.802 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:35.829 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:35.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:35.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:35.931 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:35.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:35.955 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:35.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:35.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:35 smithi053 ceph-mon[89963]: pgmap v1167: 103 pgs: 103 active+clean; 4.6 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:35.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:35 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1749461462' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-08-17T17:42:35.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:35 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-08-17T17:42:36.056 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:36.080 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:36.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:36.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:36.169 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:42:36.181 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:36.208 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:36.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:36.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:36.310 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:36.336 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:36.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:36.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:36.438 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:36.465 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:36.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:36.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:36.566 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:36.592 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:36.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:36.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:36.694 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:36.716 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:36.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:36.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:36.818 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:36.845 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:36.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:36.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:36.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:36.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:36 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-08-17T17:42:36.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:36 smithi053 ceph-mon[89963]: osdmap e357: 8 total, 8 up, 8 in 2024-08-17T17:42:36.969 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:36.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:36.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:37.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:36 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-08-17T17:42:37.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:36 smithi003 ceph-mon[103344]: osdmap e357: 8 total, 8 up, 8 in 2024-08-17T17:42:37.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:36 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-08-17T17:42:37.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:36 smithi003 ceph-mon[100615]: osdmap e357: 8 total, 8 up, 8 in 2024-08-17T17:42:37.071 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:37.084 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd pool application enable unique_pool_5 rados --yes-i-really-mean-it || true 2024-08-17T17:42:37.093 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:37.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:37.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:37.194 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:37.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:37.217 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:37.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:37.318 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:37.341 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:37.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:37.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:37.442 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:37.469 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:37.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:37.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:37.571 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:37.597 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:37.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:37.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:37.640 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:42:37.640 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:42:37.673 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:42:37.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:37.722 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:37.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:37.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:37.824 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:37.846 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:37.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:37.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:37.898 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:37 smithi003 ceph-mon[103344]: pgmap v1169: 119 pgs: 1 creating+peering, 15 unknown, 103 active+clean; 4.6 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:37.898 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:37 smithi003 ceph-mon[103344]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:42:37.898 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:37 smithi003 ceph-mon[103344]: osdmap e358: 8 total, 8 up, 8 in 2024-08-17T17:42:37.898 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:37 smithi003 ceph-mon[100615]: pgmap v1169: 119 pgs: 1 creating+peering, 15 unknown, 103 active+clean; 4.6 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:37.898 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:37 smithi003 ceph-mon[100615]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:42:37.898 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:37 smithi003 ceph-mon[100615]: osdmap e358: 8 total, 8 up, 8 in 2024-08-17T17:42:37.948 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:37.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:37 smithi053 ceph-mon[89963]: pgmap v1169: 119 pgs: 1 creating+peering, 15 unknown, 103 active+clean; 4.6 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:37.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:37 smithi053 ceph-mon[89963]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:42:37.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:37 smithi053 ceph-mon[89963]: osdmap e358: 8 total, 8 up, 8 in 2024-08-17T17:42:37.970 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:37.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:37.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:38.072 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:38.099 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:38.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:38.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:38.200 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:38.227 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:38.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:38.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:38.329 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:38.340 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:42:38.354 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:38.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:38.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:38.456 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:38.482 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:38.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:38.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:38.584 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:38.611 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:38.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:38.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:38.713 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:38.739 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:38.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:38.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:38.840 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:38.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:38 smithi003 ceph-mon[103344]: osdmap e359: 8 total, 8 up, 8 in 2024-08-17T17:42:38.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:42:38.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:42:38.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:38 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2954361183' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:42:38.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:38 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:42:38.854 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:38 smithi003 ceph-mon[100615]: osdmap e359: 8 total, 8 up, 8 in 2024-08-17T17:42:38.854 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:42:38.854 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:42:38.854 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:38 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2954361183' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:42:38.855 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:38 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:42:38.863 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:38.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:38.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:38.965 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:38.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:38 smithi053 ceph-mon[89963]: osdmap e359: 8 total, 8 up, 8 in 2024-08-17T17:42:38.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:42:38.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:42:38.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:38 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2954361183' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:42:38.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:38 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:42:38.992 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:38.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:38.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:39.093 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:39.118 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-08-17T17:42:39.122 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:39.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:39.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:39.224 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:39.252 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:39.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:39.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:39.354 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:39.377 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:39.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:39.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:39.479 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:39.505 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:39.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:39.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:39.608 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:39.634 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:39.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:39.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:39.736 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:39.759 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:39.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:39.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:39.861 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:39.867 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:39 smithi053 ceph-mon[89963]: pgmap v1172: 119 pgs: 1 creating+peering, 15 unknown, 103 active+clean; 4.6 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:39.867 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:39 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-08-17T17:42:39.867 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:39 smithi053 ceph-mon[89963]: osdmap e360: 8 total, 8 up, 8 in 2024-08-17T17:42:39.887 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:39.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:39.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:39.961 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:39 smithi003 ceph-mon[103344]: pgmap v1172: 119 pgs: 1 creating+peering, 15 unknown, 103 active+clean; 4.6 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:39.962 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:39 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-08-17T17:42:39.962 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:39 smithi003 ceph-mon[103344]: osdmap e360: 8 total, 8 up, 8 in 2024-08-17T17:42:39.962 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:39 smithi003 ceph-mon[100615]: pgmap v1172: 119 pgs: 1 creating+peering, 15 unknown, 103 active+clean; 4.6 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:39.962 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:39 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-08-17T17:42:39.962 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:39 smithi003 ceph-mon[100615]: osdmap e360: 8 total, 8 up, 8 in 2024-08-17T17:42:39.989 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:40.015 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:40.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:40.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:40.117 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:40.143 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:40.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:40.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:40.245 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:40.269 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:40.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:40.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:40.371 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:40.394 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:40.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:40.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:40.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:42:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:42:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:42:40.496 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:40.522 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:40.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:40.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:40.623 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:40.649 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:40.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:40.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:40.751 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:40.776 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:40.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:40.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:40.878 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:40.902 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:40.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:40.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:41.004 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:41.031 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:41.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:41.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:41.133 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:41.156 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:41.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:41.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:41.258 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:41.284 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:41.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:41.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:41.386 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:41.413 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:41.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:41.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:41.514 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:41.541 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:41.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:41.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:41.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:41.671 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:41.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:41.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:41.772 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:41.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:41 smithi003 ceph-mon[103344]: pgmap v1174: 119 pgs: 1 creating+peering, 118 active+clean; 4.6 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:41.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:41 smithi003 ceph-mon[100615]: pgmap v1174: 119 pgs: 1 creating+peering, 118 active+clean; 4.6 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:41.800 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:41.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:41.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:41.901 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:41.927 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:41.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:41.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:41.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:41 smithi053 ceph-mon[89963]: pgmap v1174: 119 pgs: 1 creating+peering, 118 active+clean; 4.6 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:42.029 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:42.056 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:42.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:42.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:42.157 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:42.185 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:42.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:42.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:42.286 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:42.312 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:42.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:42.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:42.414 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:42.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:42.442 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:42.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:42.542 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:42.569 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:42.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:42.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:42.671 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:42.698 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:42.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:42.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:42.799 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:42.827 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:42.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:42.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:42.928 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:42.955 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:42.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:42.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:43.057 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:43.085 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:43.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:43.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:43.187 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:43.212 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:43.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:43.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:43.314 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:43.340 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:43.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:43.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:43.442 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:43.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:43.467 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:43.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:43.568 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:43.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:43.591 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:43.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:43.692 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:43.719 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:43.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:43.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:43.820 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:43.843 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:43.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:43.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:43.871 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:43 smithi003 ceph-mon[103344]: pgmap v1175: 119 pgs: 119 active+clean; 4.6 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:43.872 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:43 smithi003 ceph-mon[100615]: pgmap v1175: 119 pgs: 119 active+clean; 4.6 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:43.944 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:43.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:43 smithi053 ceph-mon[89963]: pgmap v1175: 119 pgs: 119 active+clean; 4.6 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:43.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:43.967 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:43.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:44.068 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:44.090 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:44.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:44.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:44.191 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:44.214 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:44.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:44.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:44.316 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:44.339 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:44.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:44.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:44.441 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:44.463 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:44.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:44.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:44.565 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:44.587 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:44.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:44.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:44.636 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-08-17T17:42:44.689 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:44.711 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:44.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:44.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:44.730 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:44 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2540916143' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:42:44.730 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:44 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:42:44.731 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:44 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2540916143' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:42:44.731 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:44 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:42:44.741 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:42:44.741 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-08-17T17:42:44.741 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-08-17T17:42:44.741 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:42:44.813 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:44.841 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:44.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:44.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:44.942 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:44.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:44 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2540916143' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:42:44.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:44 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:42:44.968 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:44.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:44.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:45.070 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:45.096 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:45.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:45.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:45.198 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:45.221 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:45.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:45.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:45.323 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:45.350 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:45.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:45.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:45.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:45.474 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:45.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:45.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:45.575 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:45.598 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:45.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:45.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:45.636 INFO:teuthology.orchestra.run.smithi003.stderr:enabled application 'rados' on pool 'unique_pool_5' 2024-08-17T17:42:45.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:45.723 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:45.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:45.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:45.770 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-08-17T17:42:45.770 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:42:45.770 INFO:tasks.thrashosds.thrasher:Reweighting osd 5 to 0.9198237420013917 2024-08-17T17:42:45.770 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 5 0.9198237420013917 2024-08-17T17:42:45.825 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:45.848 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:45.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:45.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:45.949 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:45.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:45 smithi053 ceph-mon[89963]: pgmap v1176: 119 pgs: 119 active+clean; 4.6 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:45.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:45 smithi053 ceph-mon[89963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:42:45.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:45 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:42:45.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:45 smithi053 ceph-mon[89963]: osdmap e361: 8 total, 8 up, 8 in 2024-08-17T17:42:45.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:45 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3496971581' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:42:45.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:45 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:42:45.976 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:45.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:45.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:46.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:45 smithi003 ceph-mon[103344]: pgmap v1176: 119 pgs: 119 active+clean; 4.6 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:46.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:45 smithi003 ceph-mon[103344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:42:46.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:45 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:42:46.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:45 smithi003 ceph-mon[103344]: osdmap e361: 8 total, 8 up, 8 in 2024-08-17T17:42:46.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:45 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3496971581' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:42:46.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:45 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:42:46.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:45 smithi003 ceph-mon[100615]: pgmap v1176: 119 pgs: 119 active+clean; 4.6 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:46.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:45 smithi003 ceph-mon[100615]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:42:46.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:45 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:42:46.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:45 smithi003 ceph-mon[100615]: osdmap e361: 8 total, 8 up, 8 in 2024-08-17T17:42:46.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:45 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3496971581' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:42:46.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:45 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-08-17T17:42:46.024 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:42:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:42:45] "GET /metrics HTTP/1.1" 200 36491 "" "Prometheus/2.51.0" 2024-08-17T17:42:46.078 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:46.104 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:46.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:46.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:46.125 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:42:46.125 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:42:46.143 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:42:46.143 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:42:46.159 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:42:46.159 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:42:46.175 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:42:46.176 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:42:46.191 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:42:46.192 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:42:46.206 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:46.210 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:42:46.210 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:42:46.227 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:42:46.228 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:42:46.232 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:46.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:46.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:46.243 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:42:46.244 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:42:46.334 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:46.357 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:46.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:46.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:46.459 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:46.486 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:46.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:46.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:46.588 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:46.615 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:46.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:46.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:46.678 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-17T17:42:46.716 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:46.742 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:46.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:46.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:46.844 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:46.866 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:46.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:46.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:46.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:46 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T17:42:46.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:46 smithi053 ceph-mon[89963]: osdmap e362: 8 total, 8 up, 8 in 2024-08-17T17:42:46.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:46 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3832884923' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:42:46.968 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:46.995 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:46.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:46.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:47.016 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:46 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T17:42:47.016 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:46 smithi003 ceph-mon[103344]: osdmap e362: 8 total, 8 up, 8 in 2024-08-17T17:42:47.016 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:46 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3832884923' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:42:47.016 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:46 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-08-17T17:42:47.016 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:46 smithi003 ceph-mon[100615]: osdmap e362: 8 total, 8 up, 8 in 2024-08-17T17:42:47.017 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:46 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3832884923' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:42:47.096 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:47.123 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:47.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:47.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:47.225 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:47.248 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:47.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:47.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:47.350 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:47.377 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:47.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:47.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:47.479 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:47.493 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:42:47.495 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:42:47.503 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:47.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:47.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:47.606 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:47.627 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:47.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:47.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:47.729 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:47.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:47.759 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:47.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:47.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:47 smithi003 ceph-mon[100615]: pgmap v1179: 119 pgs: 119 active+clean; 4.6 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:47.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:47 smithi003 ceph-mon[100615]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:42:47.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:47 smithi003 ceph-mon[103344]: pgmap v1179: 119 pgs: 119 active+clean; 4.6 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:47.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:47 smithi003 ceph-mon[103344]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:42:47.860 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:47.883 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:47.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:47.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:47.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:47 smithi053 ceph-mon[89963]: pgmap v1179: 119 pgs: 119 active+clean; 4.6 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:47.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:47 smithi053 ceph-mon[89963]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:42:47.985 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:48.011 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:48.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:48.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:48.113 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:48.140 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:48.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:48.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:48.241 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:48.264 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:48.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:48.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:48.365 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:48.396 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:48.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:48.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:48.498 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:48.520 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:48.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:48.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:48.622 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:48.649 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:48.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:48.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:48.751 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:48.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:48.779 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:48.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:48.880 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:48.906 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:48.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:48.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:49.008 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:49.035 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:49.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:49.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:49.136 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:49.163 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:49.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:49.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:49.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:49.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:49.289 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:49.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:49.390 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:49.413 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:49.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:49.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:49.416 DEBUG:teuthology.orchestra.run.smithi003:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_5 bench 90 write' 2024-08-17T17:42:49.458 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-08-17T17:42:49.458 INFO:teuthology.orchestra.run:waiting for 3000 2024-08-17T17:42:49.465 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:42:49.515 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:49.533 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:hints = 1 2024-08-17T17:42:49.533 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-08-17T17:42:49.533 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object prefix: benchmark_data_smithi003_292749 2024-08-17T17:42:49.534 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:42:49.534 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 0 0 0 0 0 0 - 0 2024-08-17T17:42:49.550 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:49.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:49.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:49.652 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:49.689 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:49.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:49.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:49.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:49 smithi003 ceph-mon[100615]: pgmap v1180: 119 pgs: 119 active+clean; 4.6 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:49.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:49 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:42:49.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:49 smithi003 ceph-mon[103344]: pgmap v1180: 119 pgs: 119 active+clean; 4.6 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:49.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:49 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:42:49.791 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:49.831 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:49.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:49.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:49.928 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:42:49.933 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:49.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:49 smithi053 ceph-mon[89963]: pgmap v1180: 119 pgs: 119 active+clean; 4.6 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:42:49.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:49 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:42:50.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:50.020 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:50.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:50.122 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:50.157 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:50.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:50.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:50.259 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:50.302 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:50.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:50.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:50.403 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:50.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:50.445 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:50.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:50.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:42:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:42:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:42:50.534 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 1 16 3709 3693 230.798 230.812 0.00521295 0.00431499 2024-08-17T17:42:50.546 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:50.589 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:50.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:50.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:50.691 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:50.727 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:50.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:50.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:50.828 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:50.863 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:50.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:50.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:50.965 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:51.002 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:51.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:51.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:51.104 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:51.142 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:51.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:51.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:51.244 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:51.282 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:51.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:51.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:51.384 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:51.396 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:42:51.416 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:51.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:51.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:51.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:51.534 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 2 16 7103 7087 221.449 212.125 0.00324222 0.00450568 2024-08-17T17:42:51.557 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:51.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:51.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:51.658 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:51.694 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:51.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:51.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:51.796 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:51.831 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:51.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:51.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:51.847 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:51 smithi003 ceph-mon[103344]: pgmap v1181: 119 pgs: 119 active+clean; 33 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 3.5 MiB/s wr, 56 op/s 2024-08-17T17:42:51.847 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:51 smithi003 ceph-mon[100615]: pgmap v1181: 119 pgs: 119 active+clean; 33 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 3.5 MiB/s wr, 56 op/s 2024-08-17T17:42:51.850 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:42:51.933 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:51.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:51 smithi053 ceph-mon[89963]: pgmap v1181: 119 pgs: 119 active+clean; 33 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 3.5 MiB/s wr, 56 op/s 2024-08-17T17:42:51.973 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:51.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:51.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:52.075 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:52.115 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:52.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:52.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:52.217 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:52.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:52.255 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:52.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:52.356 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:52.379 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:52.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:52.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:52.481 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:52.516 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:52.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:52.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:52.534 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 3 16 10145 10129 211.002 190.125 0.00445399 0.00473196 2024-08-17T17:42:52.617 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:52.651 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:52.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:52.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:52.718 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:42:52.718 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:42:52.753 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:52.788 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:52.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:52.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:52.890 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:52.924 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:52.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:52.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:53.025 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:53.060 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:53.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:53.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:53.163 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:53.203 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:53.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:53.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:53.304 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:53.339 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:53.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:53.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:53.441 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:53.477 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:53.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:53.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:53.534 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 4 16 13217 13201 206.247 192 0.00476333 0.00484076 2024-08-17T17:42:53.579 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:53.614 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:53.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:53.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:53.669 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-17T17:42:53.715 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:53.737 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:53.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:53.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:53.839 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:53.847 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.5 to 0.919824 (eb79) 2024-08-17T17:42:53.847 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:53 smithi003 ceph-mon[103344]: pgmap v1182: 119 pgs: 119 active+clean; 53 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 6.0 MiB/s wr, 96 op/s 2024-08-17T17:42:53.847 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:53 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/781057085' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 0.9198237420013917} : dispatch 2024-08-17T17:42:53.847 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:42:53.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:42:53.848 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:53 smithi003 ceph-mon[100615]: pgmap v1182: 119 pgs: 119 active+clean; 53 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 6.0 MiB/s wr, 96 op/s 2024-08-17T17:42:53.848 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:53 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/781057085' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 0.9198237420013917} : dispatch 2024-08-17T17:42:53.848 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:42:53.848 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:42:53.861 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:53.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:53.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:53.963 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:53 smithi053 ceph-mon[89963]: pgmap v1182: 119 pgs: 119 active+clean; 53 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 6.0 MiB/s wr, 96 op/s 2024-08-17T17:42:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:53 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/781057085' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 0.9198237420013917} : dispatch 2024-08-17T17:42:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:42:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:42:53.986 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:53.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:53.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:54.088 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:54.110 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:54.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:54.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:54.212 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:54.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:54.235 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:54.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:54.336 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:54.362 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:54.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:54.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:54.463 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:54.487 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:54.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:54.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:54.534 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 5 16 13673 13657 170.697 28.5 0.00177013 0.00483769 2024-08-17T17:42:54.588 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:54.611 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:54.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:54.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:54.713 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:54.734 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:54.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:54.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:54.836 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:54.859 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:54.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:54.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:54.961 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:54.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:54 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/781057085' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 0.9198237420013917}]': finished 2024-08-17T17:42:54.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:54 smithi053 ceph-mon[89963]: osdmap e363: 8 total, 8 up, 8 in 2024-08-17T17:42:54.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:54 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:42:54.971 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:54 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/781057085' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 0.9198237420013917}]': finished 2024-08-17T17:42:54.972 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:54 smithi003 ceph-mon[103344]: osdmap e363: 8 total, 8 up, 8 in 2024-08-17T17:42:54.972 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:54 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:42:54.972 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:54 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/781057085' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 0.9198237420013917}]': finished 2024-08-17T17:42:54.972 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:54 smithi003 ceph-mon[100615]: osdmap e363: 8 total, 8 up, 8 in 2024-08-17T17:42:54.972 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:54 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:42:54.984 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:54.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:54.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:55.086 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:55.107 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:55.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:55.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:55.209 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:55.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:55.233 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:55.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:55.334 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:55.358 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:55.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:55.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:55.460 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:55.485 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:55.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:55.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:55.535 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 6 16 13673 13657 142.246 0 - 0.00483769 2024-08-17T17:42:55.586 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:55.610 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:55.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:55.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:55.711 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:55.746 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:55.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:55.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:55.848 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:55.890 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:55.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:55.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:55.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:55 smithi053 ceph-mon[89963]: pgmap v1184: 119 pgs: 2 peering, 117 active+clean; 583 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-08-17T17:42:55.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:55 smithi053 ceph-mon[89963]: osdmap e364: 8 total, 8 up, 8 in 2024-08-17T17:42:55.972 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:55 smithi003 ceph-mon[103344]: pgmap v1184: 119 pgs: 2 peering, 117 active+clean; 583 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-08-17T17:42:55.972 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:55 smithi003 ceph-mon[103344]: osdmap e364: 8 total, 8 up, 8 in 2024-08-17T17:42:55.972 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:55 smithi003 ceph-mon[100615]: pgmap v1184: 119 pgs: 2 peering, 117 active+clean; 583 MiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-08-17T17:42:55.972 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:55 smithi003 ceph-mon[100615]: osdmap e364: 8 total, 8 up, 8 in 2024-08-17T17:42:55.973 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:42:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:42:55] "GET /metrics HTTP/1.1" 200 36543 "" "Prometheus/2.51.0" 2024-08-17T17:42:55.992 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:56.027 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:56.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:56.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:56.130 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:56.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:56.168 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:56.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:56.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:56.310 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:56.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:56.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:56.413 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:56.453 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:56.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:56.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:56.535 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 7 16 15975 15959 142.477 71.9375 0.00772508 0.00701365 2024-08-17T17:42:56.555 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:42:56.592 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:56.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:56.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:42:56.694 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:56.732 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:56 smithi003 ceph-mon[103344]: osdmap e365: 8 total, 8 up, 8 in 2024-08-17T17:42:56.732 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:56 smithi003 ceph-mon[100615]: osdmap e365: 8 total, 8 up, 8 in 2024-08-17T17:42:56.738 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:56.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:56.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:56.840 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:56.881 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:56.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:56.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:56.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:56 smithi053 ceph-mon[89963]: osdmap e365: 8 total, 8 up, 8 in 2024-08-17T17:42:56.985 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:57.022 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:57.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:57.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:57.124 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:57.166 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:57.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:57.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:57.268 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:57.306 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:57.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:57.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:57.408 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:57.447 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:57.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:57.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:57.535 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 8 16 18281 18265 142.681 144.125 0.00639353 0.00700117 2024-08-17T17:42:57.549 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:57.583 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:57.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:57.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:57.685 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:42:57.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:57 smithi053 ceph-mon[89963]: pgmap v1187: 119 pgs: 3 peering, 116 active+clean; 852 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 136 MiB/s wr, 2.18k op/s 2024-08-17T17:42:57.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:57 smithi053 ceph-mon[89963]: osdmap e366: 8 total, 8 up, 8 in 2024-08-17T17:42:57.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:57.718 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:57.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:42:57.819 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:57.855 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:57.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:57.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:57.956 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:57.996 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:57.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:57.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:58.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:57 smithi003 ceph-mon[103344]: pgmap v1187: 119 pgs: 3 peering, 116 active+clean; 852 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 136 MiB/s wr, 2.18k op/s 2024-08-17T17:42:58.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:57 smithi003 ceph-mon[103344]: osdmap e366: 8 total, 8 up, 8 in 2024-08-17T17:42:58.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:57 smithi003 ceph-mon[100615]: pgmap v1187: 119 pgs: 3 peering, 116 active+clean; 852 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 136 MiB/s wr, 2.18k op/s 2024-08-17T17:42:58.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:57 smithi003 ceph-mon[100615]: osdmap e366: 8 total, 8 up, 8 in 2024-08-17T17:42:58.100 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:58.122 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:58.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:58.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:58.223 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:58.260 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:58.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:58.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:58.362 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:58.387 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:58.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:58.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:58.489 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:58.511 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:58.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:58.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:58.535 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 9 16 19885 19869 137.966 100.25 0.00199774 0.00715233 2024-08-17T17:42:58.612 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:58.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:58.642 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:58.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:58.743 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:58.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:58 smithi003 ceph-mon[103344]: osdmap e367: 8 total, 8 up, 8 in 2024-08-17T17:42:58.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:58 smithi003 ceph-mon[100615]: osdmap e367: 8 total, 8 up, 8 in 2024-08-17T17:42:58.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:58.786 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:58.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:58.887 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:58.922 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:58.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:58.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:58.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:58 smithi053 ceph-mon[89963]: osdmap e367: 8 total, 8 up, 8 in 2024-08-17T17:42:59.024 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:59.060 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:59.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:59.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:59.161 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:42:59.199 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:59.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:59.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:42:59.303 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:42:59.325 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:59.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:59.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:42:59.427 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:42:59.463 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:59.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:59.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:42:59.535 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 10 15 21929 21914 136.949 127.812 0.00541299 0.00729643 2024-08-17T17:42:59.564 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:42:59.597 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:59.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:59.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:42:59.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:42:59.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:59.738 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:42:59.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:42:59.839 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:42:59.880 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:42:59.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:42:59.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:42:59.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:42:59 smithi053 ceph-mon[89963]: pgmap v1190: 119 pgs: 3 peering, 116 active+clean; 852 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-08-17T17:42:59.983 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:00.019 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:00.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:00.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:00.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:42:59 smithi003 ceph-mon[103344]: pgmap v1190: 119 pgs: 3 peering, 116 active+clean; 852 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-08-17T17:43:00.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:42:59 smithi003 ceph-mon[100615]: pgmap v1190: 119 pgs: 3 peering, 116 active+clean; 852 MiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-08-17T17:43:00.121 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:00.159 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:00.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:00.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:00.261 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:00.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:00.293 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:00.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:00.394 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:00.428 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:00.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:00.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:00.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:43:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:43:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:43:00.531 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:00.535 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 11 16 23982 23966 136.157 128.25 0.00661382 0.00733751 2024-08-17T17:43:00.566 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:00.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:00.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:00.668 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:00.695 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:00.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:00.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:00.709 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-08-17T17:43:00.709 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 0 1 2024-08-17T17:43:00.796 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:00.838 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:00.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:00.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:00.939 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:00.976 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:00.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:00.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:01.078 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:01.117 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:01.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:01.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:01.219 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:01.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:01.255 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:01.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:43:01.356 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:01.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:01.404 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:01.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:01.505 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:01.535 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 12 16 26067 26051 135.668 130.312 0.00473355 0.00736524 2024-08-17T17:43:01.539 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:01.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:01.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:01.642 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:01.681 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:01.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:01.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:01.784 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:01.820 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:01.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:01.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:01.918 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:01 smithi003 ceph-mon[103344]: pgmap v1191: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 1.3 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 113 MiB/s wr, 1.81k op/s; 531/61626 objects misplaced (0.862%); 4.1 MiB/s, 66 objects/s recovering 2024-08-17T17:43:01.923 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:01.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:01 smithi003 ceph-mon[100615]: pgmap v1191: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 1.3 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 113 MiB/s wr, 1.81k op/s; 531/61626 objects misplaced (0.862%); 4.1 MiB/s, 66 objects/s recovering 2024-08-17T17:43:01.955 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:01.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:01.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:01.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:01 smithi053 ceph-mon[89963]: pgmap v1191: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 1.3 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 113 MiB/s wr, 1.81k op/s; 531/61626 objects misplaced (0.862%); 4.1 MiB/s, 66 objects/s recovering 2024-08-17T17:43:02.057 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:02.091 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:02.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:02.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:02.193 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:02.217 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:02.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:02.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:02.319 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:02.341 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:02.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:02.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:02.443 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:02.466 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:02.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:02.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:02.535 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 13 16 27552 27536 132.37 92.8125 0.00426278 0.00754806 2024-08-17T17:43:02.568 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:02.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:02.606 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:02.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:02.702 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-08-17T17:43:02.706 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:02.747 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:02.747 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:02 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1464918318' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:43:02.747 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:02 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1015050837' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:43:02.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:02.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:02.748 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:02 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1464918318' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:43:02.749 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:02 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1015050837' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:43:02.849 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:02.876 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:02.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:02.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:02.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:02 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1464918318' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:43:02.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:02 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1015050837' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:43:02.977 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:02.999 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:03.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:03.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:03.101 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:03.122 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:03.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:03.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:03.195 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:43:03.195 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:43:03.195 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:43:03.196 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:43:03.196 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:43:03.196 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:43:03.196 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:43:03.196 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:43:03.196 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:43:03.196 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:43:03.196 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:43:03.197 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:43:03.197 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:43:03.197 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:43:03.197 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:43:03.197 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:43:03.224 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:03.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:03.247 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:03.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:03.348 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:03.370 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:03.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:03.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:03.472 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:03.494 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:03.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:03.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:03.535 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 14 16 28172 28156 125.682 38.75 0.00182181 0.00753584 2024-08-17T17:43:03.596 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:03.619 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:03.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:03.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:03.720 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:03.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:03.758 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:03.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:03 smithi003 ceph-mon[100615]: pgmap v1192: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s; 531/63585 objects misplaced (0.835%); 3.5 MiB/s, 56 objects/s recovering 2024-08-17T17:43:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:03 smithi003 ceph-mon[100615]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:43:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:03 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1464918318' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:43:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:03 smithi003 ceph-mon[100615]: osdmap e368: 8 total, 8 up, 8 in 2024-08-17T17:43:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:03 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:43:03.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:03 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:43:03.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:03 smithi003 ceph-mon[103344]: pgmap v1192: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s; 531/63585 objects misplaced (0.835%); 3.5 MiB/s, 56 objects/s recovering 2024-08-17T17:43:03.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:03 smithi003 ceph-mon[103344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:43:03.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:03 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1464918318' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:43:03.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:03 smithi003 ceph-mon[103344]: osdmap e368: 8 total, 8 up, 8 in 2024-08-17T17:43:03.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:03 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:43:03.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:03 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:43:03.859 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:03.897 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:03.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:03.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:03.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:03 smithi053 ceph-mon[89963]: pgmap v1192: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s; 531/63585 objects misplaced (0.835%); 3.5 MiB/s, 56 objects/s recovering 2024-08-17T17:43:03.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:03 smithi053 ceph-mon[89963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:43:03.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:03 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1464918318' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:43:03.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:03 smithi053 ceph-mon[89963]: osdmap e368: 8 total, 8 up, 8 in 2024-08-17T17:43:03.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:03 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:43:03.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:03 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:43:03.999 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:04.033 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:04.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:04.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:04.134 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:04.176 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:04.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:04.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:04.278 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:04.326 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:04.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:04.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:04.428 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:04.463 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:04.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:04.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:04.535 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 15 16 30082 30066 125.261 119.375 0.00493647 0.00797809 2024-08-17T17:43:04.565 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:04.597 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:04.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:04.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:04.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:04.743 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:04.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:04.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:04.766 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:04 smithi003 ceph-mon[103344]: osdmap e369: 8 total, 8 up, 8 in 2024-08-17T17:43:04.767 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:04 smithi003 ceph-mon[100615]: osdmap e369: 8 total, 8 up, 8 in 2024-08-17T17:43:04.845 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:04.896 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:04.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:04.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:04.998 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:05.033 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:05.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:05.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:05.135 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:05.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:05.178 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:05.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:05.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:04 smithi053 ceph-mon[89963]: osdmap e369: 8 total, 8 up, 8 in 2024-08-17T17:43:05.279 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:05.315 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:05.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:05.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:05.417 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:05.453 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:05.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:05.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:05.536 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 16 16 32381 32365 126.412 143.688 0.00586126 0.00790538 2024-08-17T17:43:05.555 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:05.590 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:05.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:05.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:05.692 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:05.721 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:43:05.722 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:05.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:05.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:05.824 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:05.866 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:05.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:05.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:05.927 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:05 smithi003 ceph-mon[103344]: pgmap v1195: 119 pgs: 1 peering, 118 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 119 MiB/s wr, 1.91k op/s; 8.5 MiB/s, 136 objects/s recovering 2024-08-17T17:43:05.928 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:05 smithi003 ceph-mon[100615]: pgmap v1195: 119 pgs: 1 peering, 118 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 119 MiB/s wr, 1.91k op/s; 8.5 MiB/s, 136 objects/s recovering 2024-08-17T17:43:05.928 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:43:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:43:05] "GET /metrics HTTP/1.1" 200 36543 "" "Prometheus/2.51.0" 2024-08-17T17:43:05.967 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:06.007 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:06.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:06.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:06.109 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:06.147 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:06.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:06.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:06.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:05 smithi053 ceph-mon[89963]: pgmap v1195: 119 pgs: 1 peering, 118 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 119 MiB/s wr, 1.91k op/s; 8.5 MiB/s, 136 objects/s recovering 2024-08-17T17:43:06.249 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:06.285 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:06.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:06.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:06.387 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:06.420 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:06.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:06.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:06.522 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:06.536 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 17 16 34547 34531 126.934 135.375 0.0108766 0.00787187 2024-08-17T17:43:06.558 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:06.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:06.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:06.660 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:06.698 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:06.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:06.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:06.800 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:06.817 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:43:06.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:06.845 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:06.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:06.946 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:06.996 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:06.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:06.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:07.098 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:07.134 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:43:07.135 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-08-17T17:43:07.135 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-08-17T17:43:07.135 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:43:07.135 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.135 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:75588 8.8 8:17b78267:::benchmark_data_smithi003_229103_object75587:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-08-17T17:43:07.135 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:43.225006+0000", 2024-08-17T17:43:07.135 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 563.90440212399994, 2024-08-17T17:43:07.135 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.94613346600000003, 2024-08-17T17:43:07.135 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:43:07.135 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:43:07.135 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:07.135 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:43:07.135 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:43:07.136 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:07.136 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 75588 2024-08-17T17:43:07.136 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.136 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:43:07.136 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.136 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:43:07.136 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.225006+0000", 2024-08-17T17:43:07.136 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.136 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.136 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.136 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:43:07.136 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.225006+0000", 2024-08-17T17:43:07.136 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.136 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.137 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.137 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:43:07.137 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.225008+0000", 2024-08-17T17:43:07.137 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.762e-06 2024-08-17T17:43:07.137 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.137 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.137 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:43:07.137 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.225044+0000", 2024-08-17T17:43:07.137 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6461999999999998e-05 2024-08-17T17:43:07.137 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.137 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.137 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:43:07.137 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.225062+0000", 2024-08-17T17:43:07.137 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8130000000000001e-05 2024-08-17T17:43:07.137 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.137 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.138 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:43:07.138 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.225075+0000", 2024-08-17T17:43:07.138 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2332e-05 2024-08-17T17:43:07.138 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.138 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.138 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.138 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.225151+0000", 2024-08-17T17:43:07.138 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6749000000000007e-05 2024-08-17T17:43:07.138 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.138 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.138 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:43:07.138 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.225165+0000", 2024-08-17T17:43:07.138 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3286e-05 2024-08-17T17:43:07.139 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.139 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.139 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.139 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.160730+0000", 2024-08-17T17:43:07.139 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.93556483899999998 2024-08-17T17:43:07.139 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.139 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.139 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:43:07.139 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.162515+0000", 2024-08-17T17:43:07.139 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017854640000000001 2024-08-17T17:43:07.139 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.139 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.139 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-08-17T17:43:07.139 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.162676+0000", 2024-08-17T17:43:07.140 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016062499999999999 2024-08-17T17:43:07.140 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.140 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.140 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:43:07.140 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.167639+0000", 2024-08-17T17:43:07.140 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049631270000000003 2024-08-17T17:43:07.140 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.140 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.140 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.140 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.170933+0000", 2024-08-17T17:43:07.140 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032941379999999998 2024-08-17T17:43:07.140 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.140 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.140 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.141 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.171024+0000", 2024-08-17T17:43:07.141 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1162999999999994e-05 2024-08-17T17:43:07.141 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.141 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.141 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:43:07.141 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.171084+0000", 2024-08-17T17:43:07.141 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0068999999999997e-05 2024-08-17T17:43:07.141 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.141 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.141 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:43:07.141 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.171139+0000", 2024-08-17T17:43:07.141 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5319999999999999e-05 2024-08-17T17:43:07.141 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.142 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:43:07.142 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.142 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.142 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.142 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:75593 8.8 8:1a34edc5:::benchmark_data_smithi003_229103_object75592:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-08-17T17:43:07.142 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:43.229517+0000", 2024-08-17T17:43:07.142 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 563.89989090500001, 2024-08-17T17:43:07.142 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.95146924799999999, 2024-08-17T17:43:07.142 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:43:07.142 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:43:07.142 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:07.142 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:43:07.142 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:43:07.142 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:07.143 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 75593 2024-08-17T17:43:07.143 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.143 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:43:07.143 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.143 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:43:07.143 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.229517+0000", 2024-08-17T17:43:07.143 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.143 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.143 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.143 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:43:07.143 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.229517+0000", 2024-08-17T17:43:07.143 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.143 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.144 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.144 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:43:07.144 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.229518+0000", 2024-08-17T17:43:07.144 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3200000000000005e-07 2024-08-17T17:43:07.144 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.144 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.144 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:43:07.144 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.229549+0000", 2024-08-17T17:43:07.144 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1182000000000001e-05 2024-08-17T17:43:07.144 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.144 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.144 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:43:07.144 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.229558+0000", 2024-08-17T17:43:07.144 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0049999999999993e-06 2024-08-17T17:43:07.144 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.145 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.145 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:43:07.145 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.229564+0000", 2024-08-17T17:43:07.145 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0109999999999997e-06 2024-08-17T17:43:07.145 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.145 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.145 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.145 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.230327+0000", 2024-08-17T17:43:07.145 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00076271200000000003 2024-08-17T17:43:07.145 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.145 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.145 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:43:07.145 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.230333+0000", 2024-08-17T17:43:07.145 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3489999999999997e-06 2024-08-17T17:43:07.146 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.146 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.146 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.146 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.165320+0000", 2024-08-17T17:43:07.146 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.93498701900000003 2024-08-17T17:43:07.146 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.146 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.146 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:43:07.146 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.165446+0000", 2024-08-17T17:43:07.146 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012568599999999999 2024-08-17T17:43:07.146 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.146 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.146 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-08-17T17:43:07.146 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.165596+0000", 2024-08-17T17:43:07.147 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015010800000000001 2024-08-17T17:43:07.147 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.147 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.147 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:43:07.147 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.171153+0000", 2024-08-17T17:43:07.147 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0055573760000000002 2024-08-17T17:43:07.147 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.147 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.147 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.147 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.180402+0000", 2024-08-17T17:43:07.147 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0092493209999999996 2024-08-17T17:43:07.147 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.147 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.148 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.148 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.180868+0000", 2024-08-17T17:43:07.148 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00046594799999999999 2024-08-17T17:43:07.148 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.148 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.148 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:43:07.148 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.180935+0000", 2024-08-17T17:43:07.148 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6484999999999998e-05 2024-08-17T17:43:07.148 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.148 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.148 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:43:07.148 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.180986+0000", 2024-08-17T17:43:07.148 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1514000000000002e-05 2024-08-17T17:43:07.149 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.149 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:43:07.149 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.149 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.149 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.149 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:75594 8.8 8:1144a47c:::benchmark_data_smithi003_229103_object75593:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-08-17T17:43:07.149 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:43.229585+0000", 2024-08-17T17:43:07.149 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 563.89982301600003, 2024-08-17T17:43:07.149 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.95152271399999999, 2024-08-17T17:43:07.149 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:43:07.149 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:43:07.149 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:07.149 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:43:07.149 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:43:07.150 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:07.150 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 75594 2024-08-17T17:43:07.150 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.150 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:43:07.150 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.150 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:43:07.150 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.229585+0000", 2024-08-17T17:43:07.150 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.150 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.150 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.150 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:43:07.150 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.229585+0000", 2024-08-17T17:43:07.150 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.150 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.151 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.151 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:43:07.151 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.229586+0000", 2024-08-17T17:43:07.151 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8100000000000003e-07 2024-08-17T17:43:07.151 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.151 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.151 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:43:07.151 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.229634+0000", 2024-08-17T17:43:07.151 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8489999999999998e-05 2024-08-17T17:43:07.151 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.151 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.151 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:43:07.151 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.229643+0000", 2024-08-17T17:43:07.151 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4980000000000006e-06 2024-08-17T17:43:07.152 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.152 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.152 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:43:07.152 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.229648+0000", 2024-08-17T17:43:07.152 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8900000000000004e-06 2024-08-17T17:43:07.152 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.152 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.152 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.152 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.230346+0000", 2024-08-17T17:43:07.152 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00069730200000000001 2024-08-17T17:43:07.152 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.152 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.152 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:43:07.152 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.230350+0000", 2024-08-17T17:43:07.153 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.805e-06 2024-08-17T17:43:07.153 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.153 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.153 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.153 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.167725+0000", 2024-08-17T17:43:07.153 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.93737582100000005 2024-08-17T17:43:07.153 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.153 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.153 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:43:07.153 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.167862+0000", 2024-08-17T17:43:07.153 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013633300000000001 2024-08-17T17:43:07.153 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.153 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.153 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-08-17T17:43:07.154 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.168045+0000", 2024-08-17T17:43:07.154 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018327099999999999 2024-08-17T17:43:07.154 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.154 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.154 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:43:07.154 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.171158+0000", 2024-08-17T17:43:07.154 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031133829999999999 2024-08-17T17:43:07.154 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.154 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.154 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.154 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.180442+0000", 2024-08-17T17:43:07.154 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0092836410000000005 2024-08-17T17:43:07.154 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.155 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.155 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.155 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.181030+0000", 2024-08-17T17:43:07.155 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00058834600000000001 2024-08-17T17:43:07.155 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.155 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.155 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:43:07.155 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.181075+0000", 2024-08-17T17:43:07.155 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4728999999999998e-05 2024-08-17T17:43:07.155 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.155 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.155 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:43:07.155 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.181108+0000", 2024-08-17T17:43:07.155 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2724e-05 2024-08-17T17:43:07.156 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.156 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:43:07.156 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.156 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.156 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.156 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:75597 8.8 8:150f69d5:::benchmark_data_smithi003_229103_object75596:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-08-17T17:43:07.156 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:33:43.232559+0000", 2024-08-17T17:43:07.156 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 563.89684859900001, 2024-08-17T17:43:07.156 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.95297027099999998, 2024-08-17T17:43:07.156 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:43:07.156 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:43:07.156 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:07.156 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:43:07.156 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:43:07.157 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:07.157 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 75597 2024-08-17T17:43:07.157 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.157 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:43:07.157 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.157 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:43:07.157 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.232559+0000", 2024-08-17T17:43:07.157 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.157 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.157 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.157 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:43:07.157 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.232559+0000", 2024-08-17T17:43:07.157 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.157 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.157 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.158 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:43:07.158 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.232561+0000", 2024-08-17T17:43:07.158 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0109999999999999e-06 2024-08-17T17:43:07.158 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.158 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.158 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:43:07.158 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.232600+0000", 2024-08-17T17:43:07.158 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9369000000000002e-05 2024-08-17T17:43:07.158 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.158 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.158 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:43:07.158 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.232609+0000", 2024-08-17T17:43:07.159 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3419999999999997e-06 2024-08-17T17:43:07.159 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.159 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.159 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:43:07.159 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.232614+0000", 2024-08-17T17:43:07.159 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9320000000000004e-06 2024-08-17T17:43:07.159 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.159 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.159 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.159 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.232669+0000", 2024-08-17T17:43:07.159 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5016000000000003e-05 2024-08-17T17:43:07.159 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.159 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.159 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:43:07.159 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:43.232679+0000", 2024-08-17T17:43:07.159 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4620000000000002e-06 2024-08-17T17:43:07.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.160 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.160 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.160 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.171188+0000", 2024-08-17T17:43:07.160 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.938509276 2024-08-17T17:43:07.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.160 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.160 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:43:07.160 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.174563+0000", 2024-08-17T17:43:07.160 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0033746399999999999 2024-08-17T17:43:07.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.160 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.160 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-08-17T17:43:07.160 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.174712+0000", 2024-08-17T17:43:07.160 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014937199999999999 2024-08-17T17:43:07.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.161 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.161 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.161 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.181147+0000", 2024-08-17T17:43:07.161 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0064351269999999997 2024-08-17T17:43:07.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.161 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.161 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.161 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.181576+0000", 2024-08-17T17:43:07.161 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00042896999999999998 2024-08-17T17:43:07.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.162 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.162 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:43:07.162 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.185428+0000", 2024-08-17T17:43:07.162 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038522560000000001 2024-08-17T17:43:07.162 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.162 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.162 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:43:07.162 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.185482+0000", 2024-08-17T17:43:07.162 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3616999999999998e-05 2024-08-17T17:43:07.162 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.162 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.162 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:43:07.163 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:33:44.185530+0000", 2024-08-17T17:43:07.163 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7880999999999998e-05 2024-08-17T17:43:07.163 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.163 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:43:07.163 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.163 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.163 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.163 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:153490 8.f 8:f953e3c7:::benchmark_data_smithi003_229103_object46015:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-08-17T17:43:07.163 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:35:32.413685+0000", 2024-08-17T17:43:07.163 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 454.71572354599999, 2024-08-17T17:43:07.163 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.059624836, 2024-08-17T17:43:07.163 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:43:07.163 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:43:07.163 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-08-17T17:43:07.163 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:43:07.164 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:43:07.164 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:07.164 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 153490 2024-08-17T17:43:07.164 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.164 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:43:07.164 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.164 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:43:07.164 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.413685+0000", 2024-08-17T17:43:07.164 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.164 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.164 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.164 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:43:07.164 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.413685+0000", 2024-08-17T17:43:07.164 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.165 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.165 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.165 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:43:07.165 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.413686+0000", 2024-08-17T17:43:07.165 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2440000000000001e-06 2024-08-17T17:43:07.165 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.165 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.165 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:43:07.165 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.413691+0000", 2024-08-17T17:43:07.165 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8339999999999998e-06 2024-08-17T17:43:07.165 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.165 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.165 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:43:07.165 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.413705+0000", 2024-08-17T17:43:07.166 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4414e-05 2024-08-17T17:43:07.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.166 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.166 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:43:07.166 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.413712+0000", 2024-08-17T17:43:07.166 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1369999999999996e-06 2024-08-17T17:43:07.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.166 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.166 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.166 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.425036+0000", 2024-08-17T17:43:07.166 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011323878000000001 2024-08-17T17:43:07.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.166 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.167 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:43:07.167 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.425196+0000", 2024-08-17T17:43:07.167 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000159932 2024-08-17T17:43:07.167 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.167 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.167 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:43:07.167 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.425356+0000", 2024-08-17T17:43:07.167 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016027000000000001 2024-08-17T17:43:07.167 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.167 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.167 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:43:07.167 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.452454+0000", 2024-08-17T17:43:07.167 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.027097959000000001 2024-08-17T17:43:07.168 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.168 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.168 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.168 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.458256+0000", 2024-08-17T17:43:07.168 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0058021030000000003 2024-08-17T17:43:07.168 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.168 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.168 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:43:07.168 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.458269+0000", 2024-08-17T17:43:07.168 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2588e-05 2024-08-17T17:43:07.168 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.168 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.168 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.168 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.473227+0000", 2024-08-17T17:43:07.168 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0149584279999999 2024-08-17T17:43:07.169 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.169 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.169 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:43:07.169 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.473309+0000", 2024-08-17T17:43:07.169 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2049e-05 2024-08-17T17:43:07.169 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.169 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:43:07.169 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.169 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.169 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.170 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:153507 8.f 8:f8a73db7:::benchmark_data_smithi003_229103_object46032:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-08-17T17:43:07.170 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:35:32.458106+0000", 2024-08-17T17:43:07.170 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 454.67130204, 2024-08-17T17:43:07.170 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0289008230000001, 2024-08-17T17:43:07.170 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:43:07.170 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:43:07.170 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:07.170 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:43:07.170 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:43:07.170 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:07.170 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 153507 2024-08-17T17:43:07.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.170 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:43:07.170 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.171 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:43:07.171 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.458106+0000", 2024-08-17T17:43:07.171 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.171 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.171 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:43:07.171 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.458106+0000", 2024-08-17T17:43:07.171 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.171 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.171 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:43:07.171 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.458108+0000", 2024-08-17T17:43:07.171 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2050000000000001e-06 2024-08-17T17:43:07.172 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.172 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:43:07.172 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.458118+0000", 2024-08-17T17:43:07.172 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2730000000000008e-06 2024-08-17T17:43:07.172 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.172 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:43:07.172 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.458135+0000", 2024-08-17T17:43:07.172 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7034999999999999e-05 2024-08-17T17:43:07.172 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.172 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:43:07.173 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.458150+0000", 2024-08-17T17:43:07.173 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5931999999999999e-05 2024-08-17T17:43:07.173 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.173 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.173 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.173 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.460027+0000", 2024-08-17T17:43:07.173 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0018768369999999999 2024-08-17T17:43:07.173 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.173 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.173 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:43:07.173 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.460038+0000", 2024-08-17T17:43:07.173 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0893000000000001e-05 2024-08-17T17:43:07.173 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.173 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.173 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.174 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.473849+0000", 2024-08-17T17:43:07.174 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.013810498 2024-08-17T17:43:07.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.174 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:43:07.174 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.473942+0000", 2024-08-17T17:43:07.174 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2850999999999996e-05 2024-08-17T17:43:07.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.174 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:43:07.174 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.474045+0000", 2024-08-17T17:43:07.174 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000103921 2024-08-17T17:43:07.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.175 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.175 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.486749+0000", 2024-08-17T17:43:07.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012703325 2024-08-17T17:43:07.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.175 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.175 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.486860+0000", 2024-08-17T17:43:07.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000111018 2024-08-17T17:43:07.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.175 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:43:07.175 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.486947+0000", 2024-08-17T17:43:07.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7536999999999999e-05 2024-08-17T17:43:07.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.176 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:43:07.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:07.177 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:07.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:07.177 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.486982+0000", 2024-08-17T17:43:07.177 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4474999999999998e-05 2024-08-17T17:43:07.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.177 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.178 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:43:07.178 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.487007+0000", 2024-08-17T17:43:07.178 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5023000000000001e-05 2024-08-17T17:43:07.178 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.178 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:43:07.178 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.178 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.178 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.178 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:153526 8.f 8:f4c83a6d:::benchmark_data_smithi003_229103_object46051:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-08-17T17:43:07.178 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:35:32.479609+0000", 2024-08-17T17:43:07.178 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 454.64979946, 2024-08-17T17:43:07.178 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.007457512, 2024-08-17T17:43:07.178 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:43:07.179 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:43:07.179 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:07.179 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:43:07.179 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:43:07.179 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:07.179 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 153526 2024-08-17T17:43:07.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.179 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:43:07.179 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.179 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:43:07.179 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.479609+0000", 2024-08-17T17:43:07.179 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.179 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.180 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:43:07.180 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.479609+0000", 2024-08-17T17:43:07.180 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.180 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.180 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:43:07.180 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.479610+0000", 2024-08-17T17:43:07.180 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.624e-06 2024-08-17T17:43:07.180 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.180 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:43:07.180 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.479617+0000", 2024-08-17T17:43:07.180 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2779999999999999e-06 2024-08-17T17:43:07.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.181 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.181 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:43:07.181 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.479629+0000", 2024-08-17T17:43:07.181 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2458e-05 2024-08-17T17:43:07.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.181 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.181 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:43:07.181 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.479638+0000", 2024-08-17T17:43:07.181 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4979999999999995e-06 2024-08-17T17:43:07.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.181 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.181 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.181 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.484973+0000", 2024-08-17T17:43:07.182 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0053346649999999997 2024-08-17T17:43:07.182 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.182 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:43:07.182 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.484984+0000", 2024-08-17T17:43:07.182 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0614e-05 2024-08-17T17:43:07.182 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.182 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.182 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.477685+0000", 2024-08-17T17:43:07.182 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99270100900000002 2024-08-17T17:43:07.182 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.183 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:43:07.183 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.477805+0000", 2024-08-17T17:43:07.183 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000120597 2024-08-17T17:43:07.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.183 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:43:07.183 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.477987+0000", 2024-08-17T17:43:07.183 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000181516 2024-08-17T17:43:07.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.183 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.183 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.486825+0000", 2024-08-17T17:43:07.183 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.008838149 2024-08-17T17:43:07.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.184 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.184 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.486915+0000", 2024-08-17T17:43:07.184 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9970999999999996e-05 2024-08-17T17:43:07.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.184 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:43:07.184 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.487017+0000", 2024-08-17T17:43:07.184 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000101554 2024-08-17T17:43:07.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.184 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:43:07.184 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.487045+0000", 2024-08-17T17:43:07.184 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8793999999999999e-05 2024-08-17T17:43:07.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.185 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.185 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:43:07.185 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.487066+0000", 2024-08-17T17:43:07.185 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0784999999999998e-05 2024-08-17T17:43:07.185 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.185 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:43:07.185 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.185 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.185 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:153534 8.f 8:f321a12c:::benchmark_data_smithi003_229103_object46059:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-08-17T17:43:07.185 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:35:32.498655+0000", 2024-08-17T17:43:07.185 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 454.63075282900002, 2024-08-17T17:43:07.186 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.98856764399999997, 2024-08-17T17:43:07.186 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:43:07.186 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:43:07.186 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:07.186 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:43:07.186 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:43:07.186 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:07.186 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 153534 2024-08-17T17:43:07.186 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.186 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:43:07.186 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.186 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:43:07.186 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.498655+0000", 2024-08-17T17:43:07.187 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.187 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.187 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.187 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:43:07.187 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.498655+0000", 2024-08-17T17:43:07.187 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.187 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.187 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.187 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:43:07.187 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.498657+0000", 2024-08-17T17:43:07.187 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.733e-06 2024-08-17T17:43:07.187 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.187 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.187 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:43:07.187 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.498663+0000", 2024-08-17T17:43:07.188 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4060000000000002e-06 2024-08-17T17:43:07.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.188 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:43:07.188 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.498680+0000", 2024-08-17T17:43:07.188 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6932999999999999e-05 2024-08-17T17:43:07.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.188 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:43:07.188 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.498688+0000", 2024-08-17T17:43:07.188 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2579999999999998e-06 2024-08-17T17:43:07.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.188 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.498745+0000", 2024-08-17T17:43:07.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7040000000000003e-05 2024-08-17T17:43:07.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:43:07.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.498755+0000", 2024-08-17T17:43:07.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0227999999999999e-05 2024-08-17T17:43:07.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.481451+0000", 2024-08-17T17:43:07.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.98269635 2024-08-17T17:43:07.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.190 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:43:07.190 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.481581+0000", 2024-08-17T17:43:07.190 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013028000000000001 2024-08-17T17:43:07.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.190 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:43:07.190 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.481705+0000", 2024-08-17T17:43:07.190 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012376600000000001 2024-08-17T17:43:07.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.190 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.190 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.486894+0000", 2024-08-17T17:43:07.191 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.005189234 2024-08-17T17:43:07.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.191 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:43:07.191 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.487074+0000", 2024-08-17T17:43:07.191 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017963300000000001 2024-08-17T17:43:07.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.191 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.191 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.487141+0000", 2024-08-17T17:43:07.191 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7108999999999997e-05 2024-08-17T17:43:07.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.191 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:43:07.192 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.487187+0000", 2024-08-17T17:43:07.192 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5361000000000002e-05 2024-08-17T17:43:07.192 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.192 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.192 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:43:07.192 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.487223+0000", 2024-08-17T17:43:07.192 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6313000000000003e-05 2024-08-17T17:43:07.192 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.192 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:43:07.192 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.192 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.192 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.192 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:153548 8.f 8:ff1ace37:::benchmark_data_smithi003_229103_object46073:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-08-17T17:43:07.192 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:35:32.512022+0000", 2024-08-17T17:43:07.192 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 454.61738635199998, 2024-08-17T17:43:07.193 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.98511574199999996, 2024-08-17T17:43:07.193 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:43:07.193 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:43:07.193 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:07.193 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:43:07.193 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:43:07.193 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:07.193 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 153548 2024-08-17T17:43:07.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.193 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:43:07.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:43:07.194 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.512022+0000", 2024-08-17T17:43:07.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.194 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:43:07.194 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.512022+0000", 2024-08-17T17:43:07.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.194 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:43:07.194 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.512023+0000", 2024-08-17T17:43:07.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7060000000000001e-06 2024-08-17T17:43:07.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.195 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:43:07.195 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.512028+0000", 2024-08-17T17:43:07.195 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.809e-06 2024-08-17T17:43:07.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.195 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:43:07.195 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.512044+0000", 2024-08-17T17:43:07.195 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5546e-05 2024-08-17T17:43:07.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.195 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:43:07.196 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.512051+0000", 2024-08-17T17:43:07.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3039999999999999e-06 2024-08-17T17:43:07.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.196 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.196 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.512130+0000", 2024-08-17T17:43:07.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9289999999999995e-05 2024-08-17T17:43:07.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.196 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:43:07.196 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.512140+0000", 2024-08-17T17:43:07.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3060000000000003e-06 2024-08-17T17:43:07.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.196 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.197 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.487261+0000", 2024-08-17T17:43:07.197 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97512163500000004 2024-08-17T17:43:07.197 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.197 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.197 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:43:07.197 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.487384+0000", 2024-08-17T17:43:07.197 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000122538 2024-08-17T17:43:07.197 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.197 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.197 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:43:07.197 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.487518+0000", 2024-08-17T17:43:07.197 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013437 2024-08-17T17:43:07.197 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.198 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.198 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.198 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.496981+0000", 2024-08-17T17:43:07.198 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0094629880000000003 2024-08-17T17:43:07.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.198 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.198 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:43:07.198 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.496994+0000", 2024-08-17T17:43:07.198 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3011e-05 2024-08-17T17:43:07.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.198 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.198 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.198 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.497032+0000", 2024-08-17T17:43:07.199 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7283999999999997e-05 2024-08-17T17:43:07.199 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.199 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.199 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:43:07.199 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.497092+0000", 2024-08-17T17:43:07.199 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0225e-05 2024-08-17T17:43:07.199 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.199 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.199 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:43:07.199 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.497137+0000", 2024-08-17T17:43:07.199 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5729999999999998e-05 2024-08-17T17:43:07.199 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.199 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:43:07.200 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.200 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.200 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.200 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:153553 8.f 8:f4d40d08:::benchmark_data_smithi003_229103_object46078:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-08-17T17:43:07.200 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:35:32.521019+0000", 2024-08-17T17:43:07.200 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 454.60838887800003, 2024-08-17T17:43:07.200 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.98537688499999998, 2024-08-17T17:43:07.200 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:43:07.200 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:43:07.200 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:07.200 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:43:07.200 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:43:07.200 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:07.200 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 153553 2024-08-17T17:43:07.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.201 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:43:07.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.201 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:43:07.201 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.521019+0000", 2024-08-17T17:43:07.201 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.201 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:43:07.201 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.521019+0000", 2024-08-17T17:43:07.201 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.201 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:43:07.201 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.521021+0000", 2024-08-17T17:43:07.202 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.906e-06 2024-08-17T17:43:07.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.202 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.202 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:43:07.202 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.521027+0000", 2024-08-17T17:43:07.202 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3230000000000003e-06 2024-08-17T17:43:07.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.202 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.202 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:43:07.202 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.521046+0000", 2024-08-17T17:43:07.202 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8168999999999998e-05 2024-08-17T17:43:07.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.202 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.203 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:43:07.203 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.521055+0000", 2024-08-17T17:43:07.203 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8560000000000002e-06 2024-08-17T17:43:07.203 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.203 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.203 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.203 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.521163+0000", 2024-08-17T17:43:07.203 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000107642 2024-08-17T17:43:07.203 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.203 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.203 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:43:07.203 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.521180+0000", 2024-08-17T17:43:07.203 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7155000000000001e-05 2024-08-17T17:43:07.203 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.204 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.204 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.204 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.492958+0000", 2024-08-17T17:43:07.204 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97177762700000003 2024-08-17T17:43:07.204 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.204 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.204 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:43:07.204 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.493091+0000", 2024-08-17T17:43:07.204 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013330699999999999 2024-08-17T17:43:07.204 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.204 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.204 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:43:07.204 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.493214+0000", 2024-08-17T17:43:07.205 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012319800000000001 2024-08-17T17:43:07.205 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.205 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.205 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:43:07.205 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.506183+0000", 2024-08-17T17:43:07.205 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012968169999999999 2024-08-17T17:43:07.205 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.205 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.205 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.205 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.506223+0000", 2024-08-17T17:43:07.205 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0000000000000003e-05 2024-08-17T17:43:07.205 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.205 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.205 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.206 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.506299+0000", 2024-08-17T17:43:07.206 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6779000000000005e-05 2024-08-17T17:43:07.206 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.206 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.206 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:43:07.207 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.506351+0000", 2024-08-17T17:43:07.207 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1817999999999998e-05 2024-08-17T17:43:07.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.207 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:43:07.207 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.506396+0000", 2024-08-17T17:43:07.207 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4935000000000003e-05 2024-08-17T17:43:07.207 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.207 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:43:07.207 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.207 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:153557 8.f 8:fa54677e:::benchmark_data_smithi003_229103_object46082:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-08-17T17:43:07.208 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:35:32.527589+0000", 2024-08-17T17:43:07.208 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 454.60181884399998, 2024-08-17T17:43:07.208 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97973583900000005, 2024-08-17T17:43:07.208 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:43:07.208 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:43:07.208 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:07.208 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:43:07.208 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:43:07.208 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:07.208 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 153557 2024-08-17T17:43:07.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.208 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:43:07.208 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.209 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:43:07.209 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.527589+0000", 2024-08-17T17:43:07.209 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.209 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.209 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.209 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:43:07.209 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.527589+0000", 2024-08-17T17:43:07.210 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.210 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.210 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.210 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:43:07.210 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.527591+0000", 2024-08-17T17:43:07.210 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7e-06 2024-08-17T17:43:07.210 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.210 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.210 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:43:07.210 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.527597+0000", 2024-08-17T17:43:07.210 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4439999999999996e-06 2024-08-17T17:43:07.210 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.210 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.211 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:43:07.211 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.527617+0000", 2024-08-17T17:43:07.211 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9460999999999999e-05 2024-08-17T17:43:07.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.211 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.211 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:43:07.211 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.527626+0000", 2024-08-17T17:43:07.211 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0890000000000006e-06 2024-08-17T17:43:07.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.213 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.213 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.528901+0000", 2024-08-17T17:43:07.213 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001275017 2024-08-17T17:43:07.213 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.213 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.213 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:43:07.213 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.528913+0000", 2024-08-17T17:43:07.213 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1943999999999999e-05 2024-08-17T17:43:07.213 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.213 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.213 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.214 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.497164+0000", 2024-08-17T17:43:07.214 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96825112700000004 2024-08-17T17:43:07.214 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.214 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.214 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:43:07.214 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.497272+0000", 2024-08-17T17:43:07.215 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000107587 2024-08-17T17:43:07.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.215 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.215 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:43:07.215 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.497393+0000", 2024-08-17T17:43:07.215 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012113899999999999 2024-08-17T17:43:07.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.215 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.215 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:43:07.215 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.506233+0000", 2024-08-17T17:43:07.215 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0088400530000000005 2024-08-17T17:43:07.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.215 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.215 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.216 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.506265+0000", 2024-08-17T17:43:07.216 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1770000000000002e-05 2024-08-17T17:43:07.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.216 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.216 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.506500+0000", 2024-08-17T17:43:07.216 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00023526299999999999 2024-08-17T17:43:07.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.216 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:43:07.216 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.507266+0000", 2024-08-17T17:43:07.216 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00076610700000000003 2024-08-17T17:43:07.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.217 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:43:07.217 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.507325+0000", 2024-08-17T17:43:07.217 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9138000000000002e-05 2024-08-17T17:43:07.217 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.217 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:43:07.217 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.217 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.217 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.217 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:153564 8.f 8:f9b8d161:::benchmark_data_smithi003_229103_object46089:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-08-17T17:43:07.217 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:35:32.537158+0000", 2024-08-17T17:43:07.217 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 454.592250337, 2024-08-17T17:43:07.217 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.98091856799999999, 2024-08-17T17:43:07.218 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:43:07.218 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:43:07.218 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:07.218 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:43:07.218 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:43:07.218 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:07.218 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 153564 2024-08-17T17:43:07.218 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.218 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:43:07.218 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.218 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:43:07.218 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.537158+0000", 2024-08-17T17:43:07.219 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.219 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.219 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.219 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:43:07.219 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.537158+0000", 2024-08-17T17:43:07.219 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.219 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.219 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.219 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:43:07.219 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.537160+0000", 2024-08-17T17:43:07.219 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0880000000000002e-06 2024-08-17T17:43:07.219 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.219 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.219 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:43:07.220 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.537166+0000", 2024-08-17T17:43:07.220 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8919999999999997e-06 2024-08-17T17:43:07.220 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.220 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.220 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:43:07.220 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.537274+0000", 2024-08-17T17:43:07.220 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000108523 2024-08-17T17:43:07.220 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.220 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.220 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:43:07.220 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.537284+0000", 2024-08-17T17:43:07.220 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6970000000000005e-06 2024-08-17T17:43:07.220 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.220 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.221 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.221 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.537380+0000", 2024-08-17T17:43:07.221 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5917999999999999e-05 2024-08-17T17:43:07.221 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.221 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.221 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:43:07.221 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.537390+0000", 2024-08-17T17:43:07.221 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6979999999999993e-06 2024-08-17T17:43:07.221 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.221 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.221 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.221 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.501688+0000", 2024-08-17T17:43:07.221 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.964298027 2024-08-17T17:43:07.221 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.222 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.222 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:43:07.222 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.501798+0000", 2024-08-17T17:43:07.222 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110828 2024-08-17T17:43:07.222 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.222 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.222 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:43:07.222 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.501921+0000", 2024-08-17T17:43:07.222 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000123075 2024-08-17T17:43:07.222 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.222 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.222 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.222 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.506437+0000", 2024-08-17T17:43:07.223 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0045151870000000004 2024-08-17T17:43:07.223 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.223 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.223 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.223 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.513175+0000", 2024-08-17T17:43:07.223 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006738478 2024-08-17T17:43:07.223 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.223 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.223 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:43:07.223 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.517934+0000", 2024-08-17T17:43:07.223 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0047588309999999998 2024-08-17T17:43:07.223 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.223 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.224 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:43:07.224 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.518030+0000", 2024-08-17T17:43:07.224 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6447999999999995e-05 2024-08-17T17:43:07.224 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.224 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.224 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:43:07.224 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.518076+0000", 2024-08-17T17:43:07.224 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5877999999999997e-05 2024-08-17T17:43:07.224 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.224 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:43:07.224 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.224 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.224 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.224 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:153576 8.f 8:fd72e7d0:::benchmark_data_smithi003_229103_object46101:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-08-17T17:43:07.225 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:35:32.557477+0000", 2024-08-17T17:43:07.225 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 454.57193099099999, 2024-08-17T17:43:07.225 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96976243500000003, 2024-08-17T17:43:07.225 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:43:07.225 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:43:07.225 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:07.225 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:43:07.225 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:43:07.225 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:07.225 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 153576 2024-08-17T17:43:07.225 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.225 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:43:07.225 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.226 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:43:07.226 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.557477+0000", 2024-08-17T17:43:07.226 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.226 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.226 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.226 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:43:07.226 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.557477+0000", 2024-08-17T17:43:07.226 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.226 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.226 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.226 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:43:07.226 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.557479+0000", 2024-08-17T17:43:07.226 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3530000000000002e-06 2024-08-17T17:43:07.226 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.227 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.227 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:43:07.227 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.557488+0000", 2024-08-17T17:43:07.227 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9320000000000002e-06 2024-08-17T17:43:07.227 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.227 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.227 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:43:07.227 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.557610+0000", 2024-08-17T17:43:07.227 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000121136 2024-08-17T17:43:07.227 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.227 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.227 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:43:07.227 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.557628+0000", 2024-08-17T17:43:07.228 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8034e-05 2024-08-17T17:43:07.228 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.228 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.228 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.228 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.557745+0000", 2024-08-17T17:43:07.228 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000117514 2024-08-17T17:43:07.228 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.228 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.228 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:43:07.228 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.557761+0000", 2024-08-17T17:43:07.228 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6194000000000001e-05 2024-08-17T17:43:07.228 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.228 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.228 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.229 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.507373+0000", 2024-08-17T17:43:07.229 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.94961181299999997 2024-08-17T17:43:07.229 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.229 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.229 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:43:07.229 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.507472+0000", 2024-08-17T17:43:07.229 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8960000000000001e-05 2024-08-17T17:43:07.229 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.229 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.229 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:43:07.229 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.507590+0000", 2024-08-17T17:43:07.229 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011822199999999999 2024-08-17T17:43:07.229 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.229 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.230 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:43:07.230 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.522203+0000", 2024-08-17T17:43:07.230 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.014613050000000001 2024-08-17T17:43:07.230 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.230 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.230 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.230 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.527088+0000", 2024-08-17T17:43:07.230 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0048849999999999996 2024-08-17T17:43:07.230 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.230 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.230 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.230 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.527128+0000", 2024-08-17T17:43:07.230 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9572e-05 2024-08-17T17:43:07.230 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.231 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.231 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:43:07.231 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.527190+0000", 2024-08-17T17:43:07.231 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1716000000000003e-05 2024-08-17T17:43:07.231 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.231 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.231 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:43:07.231 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.527240+0000", 2024-08-17T17:43:07.231 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9938999999999997e-05 2024-08-17T17:43:07.231 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.231 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:43:07.231 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.231 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.231 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.231 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:153583 8.f 8:f57d2655:::benchmark_data_smithi003_229103_object46108:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-08-17T17:43:07.232 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:35:32.566062+0000", 2024-08-17T17:43:07.232 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 454.563345884, 2024-08-17T17:43:07.232 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96134504700000001, 2024-08-17T17:43:07.232 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:43:07.232 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:43:07.232 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:07.232 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:43:07.232 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:43:07.232 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:07.232 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 153583 2024-08-17T17:43:07.232 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.232 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:43:07.232 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.232 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:43:07.233 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.566062+0000", 2024-08-17T17:43:07.233 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.233 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.233 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.233 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:43:07.233 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.566062+0000", 2024-08-17T17:43:07.233 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.233 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.233 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.233 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:43:07.233 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.566064+0000", 2024-08-17T17:43:07.233 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6050000000000001e-06 2024-08-17T17:43:07.233 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.234 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.234 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:43:07.234 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.566069+0000", 2024-08-17T17:43:07.234 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9350000000000002e-06 2024-08-17T17:43:07.234 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.234 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.234 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:43:07.234 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.566079+0000", 2024-08-17T17:43:07.234 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0368e-05 2024-08-17T17:43:07.234 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.234 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.234 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:43:07.234 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.566087+0000", 2024-08-17T17:43:07.234 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4410000000000004e-06 2024-08-17T17:43:07.235 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.235 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.235 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.235 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.567489+0000", 2024-08-17T17:43:07.235 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001402695 2024-08-17T17:43:07.235 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.235 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.235 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:43:07.235 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.567501+0000", 2024-08-17T17:43:07.235 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2031000000000001e-05 2024-08-17T17:43:07.235 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.235 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.235 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.235 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.513307+0000", 2024-08-17T17:43:07.236 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.94580583399999996 2024-08-17T17:43:07.236 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.236 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.236 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:43:07.236 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.513407+0000", 2024-08-17T17:43:07.236 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9908000000000006e-05 2024-08-17T17:43:07.236 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.236 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.236 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:43:07.236 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.513512+0000", 2024-08-17T17:43:07.236 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000104793 2024-08-17T17:43:07.236 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.236 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.236 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.237 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.527282+0000", 2024-08-17T17:43:07.237 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013770677 2024-08-17T17:43:07.237 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.237 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.238 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.238 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.527315+0000", 2024-08-17T17:43:07.238 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2342000000000003e-05 2024-08-17T17:43:07.238 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.238 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.238 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:43:07.238 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.527329+0000", 2024-08-17T17:43:07.238 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4402e-05 2024-08-17T17:43:07.238 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.238 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.238 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:43:07.238 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.527370+0000", 2024-08-17T17:43:07.239 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1189000000000002e-05 2024-08-17T17:43:07.239 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.239 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.239 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:43:07.239 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.527407+0000", 2024-08-17T17:43:07.239 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6826999999999998e-05 2024-08-17T17:43:07.239 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.239 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:43:07.239 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.239 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.239 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.239 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:153584 8.f 8:f00a2922:::benchmark_data_smithi003_229103_object46109:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-08-17T17:43:07.239 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:35:32.566418+0000", 2024-08-17T17:43:07.239 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 454.562990007, 2024-08-17T17:43:07.240 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96577511599999999, 2024-08-17T17:43:07.240 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:43:07.240 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:43:07.240 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:07.240 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:43:07.240 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:43:07.240 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:07.240 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 153584 2024-08-17T17:43:07.240 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.240 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:43:07.240 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.240 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:43:07.240 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.566418+0000", 2024-08-17T17:43:07.241 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.241 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.241 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.241 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:43:07.241 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.566418+0000", 2024-08-17T17:43:07.241 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.241 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.241 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.241 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:43:07.241 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.566419+0000", 2024-08-17T17:43:07.241 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0640000000000001e-06 2024-08-17T17:43:07.241 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.241 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.241 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:43:07.241 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.566425+0000", 2024-08-17T17:43:07.242 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5350000000000004e-06 2024-08-17T17:43:07.242 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.242 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.242 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:43:07.242 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.566437+0000", 2024-08-17T17:43:07.242 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1888999999999999e-05 2024-08-17T17:43:07.242 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.242 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.242 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:43:07.242 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.566445+0000", 2024-08-17T17:43:07.242 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2889999999999998e-06 2024-08-17T17:43:07.242 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.242 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.242 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.243 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.567531+0000", 2024-08-17T17:43:07.243 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001085946 2024-08-17T17:43:07.243 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.243 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.243 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:43:07.243 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.567541+0000", 2024-08-17T17:43:07.243 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0288e-05 2024-08-17T17:43:07.243 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.243 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.243 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.243 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.518109+0000", 2024-08-17T17:43:07.243 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.95056744299999996 2024-08-17T17:43:07.243 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.243 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.244 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:43:07.244 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.518231+0000", 2024-08-17T17:43:07.244 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000122951 2024-08-17T17:43:07.244 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.244 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.244 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:43:07.244 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.518359+0000", 2024-08-17T17:43:07.244 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012795800000000001 2024-08-17T17:43:07.244 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.244 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.244 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.244 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.527446+0000", 2024-08-17T17:43:07.244 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0090865900000000003 2024-08-17T17:43:07.244 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.245 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.245 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:43:07.245 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.532052+0000", 2024-08-17T17:43:07.245 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0046056430000000004 2024-08-17T17:43:07.245 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.245 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.245 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.245 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.532097+0000", 2024-08-17T17:43:07.245 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4909e-05 2024-08-17T17:43:07.245 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.245 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.245 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:43:07.245 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.532151+0000", 2024-08-17T17:43:07.245 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4109e-05 2024-08-17T17:43:07.245 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.246 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.246 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:43:07.246 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.532193+0000", 2024-08-17T17:43:07.246 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2502000000000001e-05 2024-08-17T17:43:07.246 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.246 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:43:07.246 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.246 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.246 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.246 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:153587 8.f 8:f8989161:::benchmark_data_smithi003_229103_object46112:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-08-17T17:43:07.246 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:35:32.571419+0000", 2024-08-17T17:43:07.246 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 454.55798917099997, 2024-08-17T17:43:07.246 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97013380699999996, 2024-08-17T17:43:07.247 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:43:07.247 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:43:07.247 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:07.247 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:43:07.247 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:43:07.247 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:07.247 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 153587 2024-08-17T17:43:07.247 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.247 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:43:07.247 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.247 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:43:07.247 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.571419+0000", 2024-08-17T17:43:07.247 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.247 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.248 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.248 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:43:07.248 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.571419+0000", 2024-08-17T17:43:07.248 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.248 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.248 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.248 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:43:07.248 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.571420+0000", 2024-08-17T17:43:07.248 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.71e-07 2024-08-17T17:43:07.248 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.248 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.248 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:43:07.248 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.571424+0000", 2024-08-17T17:43:07.249 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2640000000000002e-06 2024-08-17T17:43:07.249 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.249 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.249 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:43:07.249 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.571435+0000", 2024-08-17T17:43:07.249 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0742e-05 2024-08-17T17:43:07.249 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.249 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.249 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:43:07.249 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.571442+0000", 2024-08-17T17:43:07.249 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1450000000000002e-06 2024-08-17T17:43:07.249 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.249 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.249 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.249 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.571515+0000", 2024-08-17T17:43:07.250 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3009000000000005e-05 2024-08-17T17:43:07.250 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.250 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.250 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:43:07.250 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.571524+0000", 2024-08-17T17:43:07.250 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5210000000000008e-06 2024-08-17T17:43:07.250 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.250 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.250 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.250 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.522223+0000", 2024-08-17T17:43:07.250 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.95069885300000001 2024-08-17T17:43:07.250 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.250 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.250 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:43:07.251 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.522350+0000", 2024-08-17T17:43:07.251 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012620900000000001 2024-08-17T17:43:07.251 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.251 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.251 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:43:07.251 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.522603+0000", 2024-08-17T17:43:07.251 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00025304599999999999 2024-08-17T17:43:07.251 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.251 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.251 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:43:07.251 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.532203+0000", 2024-08-17T17:43:07.251 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0096008710000000004 2024-08-17T17:43:07.251 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.251 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.252 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.252 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.532242+0000", 2024-08-17T17:43:07.252 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8469e-05 2024-08-17T17:43:07.252 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.252 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.252 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.252 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.541455+0000", 2024-08-17T17:43:07.252 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0092134939999999992 2024-08-17T17:43:07.252 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.252 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.252 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:43:07.252 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.541510+0000", 2024-08-17T17:43:07.252 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4567000000000001e-05 2024-08-17T17:43:07.252 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.253 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.253 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:43:07.253 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.541553+0000", 2024-08-17T17:43:07.253 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2746e-05 2024-08-17T17:43:07.253 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.253 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:43:07.253 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.253 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.253 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.253 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:153596 8.f 8:f444e16f:::benchmark_data_smithi003_229103_object46121:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-08-17T17:43:07.253 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:35:32.591109+0000", 2024-08-17T17:43:07.253 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 454.53829946100001, 2024-08-17T17:43:07.253 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.95572653399999996, 2024-08-17T17:43:07.253 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:43:07.254 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:43:07.254 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:07.254 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:43:07.254 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:43:07.254 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:07.254 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 153596 2024-08-17T17:43:07.254 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.254 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:43:07.254 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.254 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:43:07.254 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.591109+0000", 2024-08-17T17:43:07.254 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.254 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.255 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.255 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:43:07.255 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.591109+0000", 2024-08-17T17:43:07.255 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.255 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.255 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.255 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:43:07.255 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.591110+0000", 2024-08-17T17:43:07.255 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.562e-06 2024-08-17T17:43:07.255 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.255 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.255 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:43:07.255 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.591116+0000", 2024-08-17T17:43:07.255 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8189999999999997e-06 2024-08-17T17:43:07.256 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.256 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.256 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:43:07.256 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.591131+0000", 2024-08-17T17:43:07.256 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5464999999999998e-05 2024-08-17T17:43:07.256 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.256 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.256 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:43:07.256 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.591140+0000", 2024-08-17T17:43:07.256 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3340000000000002e-06 2024-08-17T17:43:07.256 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.256 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.256 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.257 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.591297+0000", 2024-08-17T17:43:07.257 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015688900000000001 2024-08-17T17:43:07.257 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.257 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.257 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:43:07.257 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.591310+0000", 2024-08-17T17:43:07.257 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.325e-05 2024-08-17T17:43:07.257 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.257 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.257 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.257 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.527481+0000", 2024-08-17T17:43:07.257 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.93617078499999995 2024-08-17T17:43:07.257 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.257 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.258 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:43:07.258 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.527599+0000", 2024-08-17T17:43:07.258 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000118682 2024-08-17T17:43:07.258 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.258 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.258 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:43:07.258 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.527715+0000", 2024-08-17T17:43:07.258 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000115234 2024-08-17T17:43:07.258 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.258 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.258 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:43:07.258 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.541411+0000", 2024-08-17T17:43:07.259 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013695964 2024-08-17T17:43:07.259 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.259 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.259 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.259 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.541595+0000", 2024-08-17T17:43:07.259 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018422800000000001 2024-08-17T17:43:07.259 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.259 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.259 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.259 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.546723+0000", 2024-08-17T17:43:07.259 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0051277400000000004 2024-08-17T17:43:07.259 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.259 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.259 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:43:07.259 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.546776+0000", 2024-08-17T17:43:07.259 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2954999999999997e-05 2024-08-17T17:43:07.260 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.260 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.260 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:43:07.260 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.546835+0000", 2024-08-17T17:43:07.260 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9626999999999997e-05 2024-08-17T17:43:07.260 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.260 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:43:07.260 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.260 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.260 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.260 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:153652 8.f 8:f25ef829:::benchmark_data_smithi003_229103_object46177:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-08-17T17:43:07.260 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:35:32.737604+0000", 2024-08-17T17:43:07.260 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 454.39180409699998, 2024-08-17T17:43:07.261 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.816668693, 2024-08-17T17:43:07.261 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:43:07.261 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:43:07.261 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:07.261 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:43:07.261 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:43:07.261 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:07.261 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 153652 2024-08-17T17:43:07.261 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.261 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:43:07.261 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.261 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:43:07.261 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.737604+0000", 2024-08-17T17:43:07.261 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.262 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.262 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.262 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:43:07.262 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.737604+0000", 2024-08-17T17:43:07.262 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.262 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.262 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.262 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:43:07.262 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.737606+0000", 2024-08-17T17:43:07.262 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3520000000000001e-06 2024-08-17T17:43:07.262 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.262 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.262 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:43:07.262 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.737613+0000", 2024-08-17T17:43:07.263 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2319999999999998e-06 2024-08-17T17:43:07.263 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.263 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.263 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:43:07.263 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.737631+0000", 2024-08-17T17:43:07.263 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8606000000000001e-05 2024-08-17T17:43:07.263 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.263 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.263 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:43:07.263 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.737683+0000", 2024-08-17T17:43:07.263 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1922000000000002e-05 2024-08-17T17:43:07.263 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.263 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.263 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.264 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.737791+0000", 2024-08-17T17:43:07.264 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010830700000000001 2024-08-17T17:43:07.264 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.265 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.265 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:43:07.265 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.737807+0000", 2024-08-17T17:43:07.265 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5721000000000001e-05 2024-08-17T17:43:07.265 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.265 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.266 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.266 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.532273+0000", 2024-08-17T17:43:07.266 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.79446544299999999 2024-08-17T17:43:07.266 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.266 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.266 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:43:07.266 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.532368+0000", 2024-08-17T17:43:07.266 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5742999999999995e-05 2024-08-17T17:43:07.266 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.266 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.266 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:43:07.266 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.532472+0000", 2024-08-17T17:43:07.266 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000103229 2024-08-17T17:43:07.267 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.267 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.267 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.267 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.541635+0000", 2024-08-17T17:43:07.267 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0091629699999999994 2024-08-17T17:43:07.267 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.267 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.267 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.267 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.546878+0000", 2024-08-17T17:43:07.267 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0052432859999999998 2024-08-17T17:43:07.267 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.267 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.267 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:43:07.268 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.554179+0000", 2024-08-17T17:43:07.268 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0073012620000000002 2024-08-17T17:43:07.268 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.268 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.268 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:43:07.268 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.554223+0000", 2024-08-17T17:43:07.268 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3795999999999999e-05 2024-08-17T17:43:07.268 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.268 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.268 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:43:07.268 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.554273+0000", 2024-08-17T17:43:07.268 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9824000000000003e-05 2024-08-17T17:43:07.268 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.268 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:43:07.269 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.269 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.269 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.269 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:153665 8.f 8:f5927b09:::benchmark_data_smithi003_229103_object46190:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-08-17T17:43:07.269 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:35:32.787078+0000", 2024-08-17T17:43:07.269 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 454.34233038399998, 2024-08-17T17:43:07.269 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.78623647500000005, 2024-08-17T17:43:07.269 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:43:07.269 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:43:07.269 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:07.269 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:43:07.269 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:43:07.269 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:07.270 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 153665 2024-08-17T17:43:07.270 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.270 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:43:07.270 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.270 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:43:07.270 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.787078+0000", 2024-08-17T17:43:07.270 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.270 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.270 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.270 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:43:07.270 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.787078+0000", 2024-08-17T17:43:07.270 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.270 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.271 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.271 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:43:07.271 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.787079+0000", 2024-08-17T17:43:07.271 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7060000000000001e-06 2024-08-17T17:43:07.271 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.271 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.271 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:43:07.271 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.787085+0000", 2024-08-17T17:43:07.271 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7200000000000003e-06 2024-08-17T17:43:07.271 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.271 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.271 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:43:07.271 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.787101+0000", 2024-08-17T17:43:07.272 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5750999999999999e-05 2024-08-17T17:43:07.272 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.272 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.272 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:43:07.272 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.787108+0000", 2024-08-17T17:43:07.272 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3239999999999997e-06 2024-08-17T17:43:07.272 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.272 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.272 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.272 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.787182+0000", 2024-08-17T17:43:07.272 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3431999999999998e-05 2024-08-17T17:43:07.272 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.272 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.273 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:43:07.273 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.787197+0000", 2024-08-17T17:43:07.273 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4912999999999999e-05 2024-08-17T17:43:07.273 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.273 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.273 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.273 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.536851+0000", 2024-08-17T17:43:07.273 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.74965494799999999 2024-08-17T17:43:07.273 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.273 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.273 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:43:07.273 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.536955+0000", 2024-08-17T17:43:07.273 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000103529 2024-08-17T17:43:07.274 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.274 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.274 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:43:07.274 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.537062+0000", 2024-08-17T17:43:07.274 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000107254 2024-08-17T17:43:07.274 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.274 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.274 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:43:07.274 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.554280+0000", 2024-08-17T17:43:07.274 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017217525000000001 2024-08-17T17:43:07.274 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.274 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.274 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.275 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.559289+0000", 2024-08-17T17:43:07.275 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0050087760000000004 2024-08-17T17:43:07.275 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.275 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.275 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.275 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.572947+0000", 2024-08-17T17:43:07.275 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013658541 2024-08-17T17:43:07.275 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.275 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.275 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:43:07.275 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.573183+0000", 2024-08-17T17:43:07.275 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00023583600000000001 2024-08-17T17:43:07.275 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.275 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.276 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:43:07.276 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.573314+0000", 2024-08-17T17:43:07.276 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013122000000000001 2024-08-17T17:43:07.276 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.276 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:43:07.276 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.276 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.276 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.276 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.46025.0:153684 8.f 8:f512e9d1:::benchmark_data_smithi003_229103_object46209:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-08-17T17:43:07.276 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:35:32.903058+0000", 2024-08-17T17:43:07.276 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 454.22635003200003, 2024-08-17T17:43:07.276 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.67573724700000004, 2024-08-17T17:43:07.276 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:43:07.276 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:43:07.277 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:07.277 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:43:07.277 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.46025", 2024-08-17T17:43:07.277 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:07.277 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 153684 2024-08-17T17:43:07.277 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.277 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:43:07.277 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.277 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:43:07.277 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.903058+0000", 2024-08-17T17:43:07.277 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.277 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.277 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.277 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:43:07.278 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.903058+0000", 2024-08-17T17:43:07.278 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:43:07.278 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.278 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.278 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:43:07.278 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.903060+0000", 2024-08-17T17:43:07.278 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0190000000000001e-06 2024-08-17T17:43:07.278 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.278 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.278 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:43:07.278 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.903068+0000", 2024-08-17T17:43:07.278 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1669999999999999e-06 2024-08-17T17:43:07.278 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.279 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.279 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:43:07.279 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.903091+0000", 2024-08-17T17:43:07.279 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2503000000000001e-05 2024-08-17T17:43:07.279 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.279 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.279 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:43:07.279 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.903101+0000", 2024-08-17T17:43:07.279 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0749e-05 2024-08-17T17:43:07.279 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.279 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.279 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.279 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.903165+0000", 2024-08-17T17:43:07.279 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3631000000000001e-05 2024-08-17T17:43:07.280 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.280 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.280 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-08-17T17:43:07.280 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:32.903176+0000", 2024-08-17T17:43:07.280 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0736e-05 2024-08-17T17:43:07.280 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.280 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.280 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:43:07.280 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.541855+0000", 2024-08-17T17:43:07.280 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.638679569 2024-08-17T17:43:07.280 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.280 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.280 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:43:07.280 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.541956+0000", 2024-08-17T17:43:07.281 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000100659 2024-08-17T17:43:07.281 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.281 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:07.281 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:43:07.282 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.282 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-08-17T17:43:07.282 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.542061+0000", 2024-08-17T17:43:07.282 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000105096 2024-08-17T17:43:07.282 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.282 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.283 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.283 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.564428+0000", 2024-08-17T17:43:07.283 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.022367070999999999 2024-08-17T17:43:07.283 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.283 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.283 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:07.283 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.573356+0000", 2024-08-17T17:43:07.283 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0089273730000000006 2024-08-17T17:43:07.283 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.283 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.283 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-08-17T17:43:07.283 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.573521+0000", 2024-08-17T17:43:07.283 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016540700000000001 2024-08-17T17:43:07.283 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.284 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.284 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:43:07.284 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.578740+0000", 2024-08-17T17:43:07.284 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0052193170000000002 2024-08-17T17:43:07.284 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:43:07.284 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:43:07.284 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:43:07.284 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:35:33.578795+0000", 2024-08-17T17:43:07.284 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4950000000000001e-05 2024-08-17T17:43:07.284 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.284 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:43:07.284 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.284 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:43:07.284 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:43:07.284 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:43:07.326 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:07.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:07.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:07.428 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:07.478 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:07.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:07.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:07.537 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 18 16 36331 36315 126.073 111.5 0.0110202 0.00792673 2024-08-17T17:43:07.580 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:07.632 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:07.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:07.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:07.734 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:07.775 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:07.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:07.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:07.878 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:07.919 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:07.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:07.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:08.021 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:08.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:07 smithi003 ceph-mon[103344]: pgmap v1196: 119 pgs: 1 peering, 118 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 123 MiB/s wr, 1.97k op/s; 7.1 MiB/s, 113 objects/s recovering 2024-08-17T17:43:08.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:07 smithi003 ceph-mon[100615]: pgmap v1196: 119 pgs: 1 peering, 118 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 123 MiB/s wr, 1.97k op/s; 7.1 MiB/s, 113 objects/s recovering 2024-08-17T17:43:08.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:08.062 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:08.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:08.163 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:08.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:08.197 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:08.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:08.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:07 smithi053 ceph-mon[89963]: pgmap v1196: 119 pgs: 1 peering, 118 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 123 MiB/s wr, 1.97k op/s; 7.1 MiB/s, 113 objects/s recovering 2024-08-17T17:43:08.299 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:08.335 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:08.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:08.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:08.437 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:08.472 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:08.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:08.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:08.537 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 19 16 38349 38333 126.075 126.125 0.00580052 0.00792681 2024-08-17T17:43:08.573 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:08.605 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:08.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:08.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:08.706 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:08.744 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:08.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:08.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:08.845 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:08.882 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:08.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:08.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:08.984 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:09.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:43:09.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:43:09.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:43:09.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:43:09.048 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:09.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:09.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:09.117 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:43:09.117 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:43:09.150 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:09.187 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:09.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:09.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:09.289 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:09.327 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:09.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:09.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:09.429 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:09.465 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:09.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:09.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:09.502 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-17T17:43:09.537 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-08-17T17:43:09.537426+0000 min lat: 0.00107156 max lat: 2.04224 avg lat: 0.00793733 2024-08-17T17:43:09.537 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:43:09.537 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 20 16 40309 40293 125.896 122.5 0.00845537 0.00793733 2024-08-17T17:43:09.567 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:09.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:09.613 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:09.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:09.714 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:09.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:09.757 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:09.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:09.858 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:09.906 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:09 smithi053 ceph-mon[89963]: pgmap v1197: 119 pgs: 1 peering, 118 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s; 4.1 MiB/s, 66 objects/s recovering 2024-08-17T17:43:09.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:09.907 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:09.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:09.988 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:43:10.012 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:10.013 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:43:10.022 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:09 smithi003 ceph-mon[100615]: pgmap v1197: 119 pgs: 1 peering, 118 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s; 4.1 MiB/s, 66 objects/s recovering 2024-08-17T17:43:10.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:09 smithi003 ceph-mon[103344]: pgmap v1197: 119 pgs: 1 peering, 118 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s; 4.1 MiB/s, 66 objects/s recovering 2024-08-17T17:43:10.051 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:10.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:10.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:10.153 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:10.197 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:10.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:10.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:10.298 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:10.350 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:10.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:10.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:10.452 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:10.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:43:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:43:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:43:10.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:10.492 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:10.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:10.537 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 21 16 41936 41920 124.743 101.688 0.00657989 0.0080117 2024-08-17T17:43:10.593 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:10.630 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:10.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:10.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:10.732 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:10.769 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:10.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:10.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:10.871 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:10.909 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:10.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:10.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:11.010 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:11.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:11.048 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:11.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:11.149 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:11.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:11.185 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:11.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:11.286 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:11.308 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:11.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:11.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:11.410 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:11.446 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:11.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:11.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:11.537 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 22 16 43407 43391 123.251 91.9375 0.0084731 0.0081077 2024-08-17T17:43:11.549 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:11.584 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:11.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:11.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:11.663 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:43:11.686 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:11.719 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:11.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:11.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:11.755 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.0 to 1 (10000) 2024-08-17T17:43:11.821 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:11.853 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:11.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:11.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:11.955 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:11.956 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:11 smithi003 ceph-mon[100615]: pgmap v1198: 119 pgs: 119 active+clean; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 139 MiB/s wr, 2.22k op/s; 4.1 MiB/s, 66 objects/s recovering 2024-08-17T17:43:11.956 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:11 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2483673117' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T17:43:11.956 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:11 smithi003 ceph-mon[103344]: pgmap v1198: 119 pgs: 119 active+clean; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 139 MiB/s wr, 2.22k op/s; 4.1 MiB/s, 66 objects/s recovering 2024-08-17T17:43:11.956 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:11 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2483673117' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T17:43:12.007 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:12.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:12.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:12.109 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:12.145 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:12.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:12.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:12.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:11 smithi053 ceph-mon[89963]: pgmap v1198: 119 pgs: 119 active+clean; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 139 MiB/s wr, 2.22k op/s; 4.1 MiB/s, 66 objects/s recovering 2024-08-17T17:43:12.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:11 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2483673117' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T17:43:12.247 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:12.273 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:12.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:12.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:12.374 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:12.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:12.416 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:12.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:12.518 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:12.542 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 23 16 45118 45102 122.515 106.938 0.00452383 0.00815668 2024-08-17T17:43:12.549 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:12.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:12.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:12.651 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:12.689 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:12.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:12.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:12.791 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:12.827 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:12.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:12.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:12.929 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:12.964 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:12.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:12.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:13.041 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:12 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2483673117' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-17T17:43:13.041 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:12 smithi003 ceph-mon[100615]: osdmap e370: 8 total, 8 up, 8 in 2024-08-17T17:43:13.041 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:12 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2483673117' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-17T17:43:13.041 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:12 smithi003 ceph-mon[103344]: osdmap e370: 8 total, 8 up, 8 in 2024-08-17T17:43:13.067 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:13.088 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 1 1 2024-08-17T17:43:13.106 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:12 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2483673117' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-17T17:43:13.106 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:12 smithi053 ceph-mon[89963]: osdmap e370: 8 total, 8 up, 8 in 2024-08-17T17:43:13.108 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:13.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:13.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:13.210 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:13.252 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:13.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:13.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:13.354 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:13.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:13.404 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:13.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:13.505 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:13.537 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:13.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:13.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:13.542 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 24 16 46725 46709 121.595 100.438 0.00307084 0.00821637 2024-08-17T17:43:13.639 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:13.670 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:13.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:13.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:13.772 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:13.806 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:13.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:13.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:13.898 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:43:13.899 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2024-08-17T17:43:13.899 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 0 2024-08-17T17:43:13.899 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:43:13.899 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:13 smithi053 ceph-mon[89963]: pgmap v1200: 119 pgs: 119 active+clean; 2.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-08-17T17:43:13.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:13.944 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:13.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:13.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:13.981 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:13 smithi003 ceph-mon[103344]: pgmap v1200: 119 pgs: 119 active+clean; 2.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-08-17T17:43:13.981 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:13 smithi003 ceph-mon[100615]: pgmap v1200: 119 pgs: 119 active+clean; 2.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 96 MiB/s wr, 1.54k op/s 2024-08-17T17:43:14.046 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:14.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:14.084 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:14.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:14.185 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:14.210 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:14.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:14.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:14.312 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:14.334 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:14.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:14.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:14.424 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:43:14.425 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:43:14.431 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:43:14.431 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:43:14.436 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:14.459 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:43:14.459 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:43:14.474 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:14.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:14.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:14.492 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:43:14.492 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:43:14.518 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:43:14.518 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:43:14.542 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 25 16 47862 47846 119.574 71.0625 0.0128198 0.0083583 2024-08-17T17:43:14.552 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:43:14.552 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:43:14.576 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:14.585 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:43:14.585 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:43:14.612 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:14.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:14.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:14.621 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:43:14.621 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:43:14.714 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:14.750 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:14.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:14.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:14.852 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:14.896 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:14.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:14.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:14.998 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:15.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:14 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2461799129' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:43:15.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:14 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2461799129' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:43:15.029 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:15.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:15.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:15.131 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:15.168 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:15.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:15.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:15.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:14 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2461799129' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:43:15.270 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:15.307 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:15.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:15.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:15.409 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:15.447 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:15.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:15.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:15.543 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 26 16 49418 49402 118.715 97.25 0.00487662 0.00841756 2024-08-17T17:43:15.549 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:15.610 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:15.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:15.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:15.712 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:15.749 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:15.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:15.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:15.772 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:43:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:43:15] "GET /metrics HTTP/1.1" 200 36584 "" "Prometheus/2.51.0" 2024-08-17T17:43:15.796 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-17T17:43:15.851 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:15.895 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:15.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:15.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:15.997 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:16.043 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:16.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:16.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:16.145 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:16.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:16.196 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:16.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:16.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:15 smithi053 ceph-mon[89963]: pgmap v1201: 119 pgs: 119 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-08-17T17:43:16.231 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:15 smithi003 ceph-mon[103344]: pgmap v1201: 119 pgs: 119 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-08-17T17:43:16.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:15 smithi003 ceph-mon[100615]: pgmap v1201: 119 pgs: 119 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 115 MiB/s wr, 1.84k op/s 2024-08-17T17:43:16.275 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:43:16.297 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:16.342 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:16.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:16.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:16.444 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:16.483 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:16.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:16.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:16.543 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 27 16 51084 51068 118.175 104.125 0.0108934 0.00845687 2024-08-17T17:43:16.584 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:16.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:16.625 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:16.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:16.726 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:16.761 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:16.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:16.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:16.863 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:16.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:16.901 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:16.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:17.002 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:17.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:17.051 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:17.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:17.152 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:17.174 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:17.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:17.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:17.213 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:43:17.276 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:17.301 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:17.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:17.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:17.403 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:17.424 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:17.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:17.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:17.502 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:43:17.526 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:17.543 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 28 16 52129 52113 116.287 65.3125 0.00243735 0.00851279 2024-08-17T17:43:17.564 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:17.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:17.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:17.665 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:17.725 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:17.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:17.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:17.818 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-08-17T17:43:17.828 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:17.855 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:17.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:17.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:17.957 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:17.992 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:17.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:17.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:18.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:17 smithi003 ceph-mon[100615]: pgmap v1202: 119 pgs: 119 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-08-17T17:43:18.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:17 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/224305069' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:43:18.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:17 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:43:18.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:17 smithi003 ceph-mon[103344]: pgmap v1202: 119 pgs: 119 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-08-17T17:43:18.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:17 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/224305069' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:43:18.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:17 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:43:18.094 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:18.126 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:18.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:18.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:18.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:17 smithi053 ceph-mon[89963]: pgmap v1202: 119 pgs: 119 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-08-17T17:43:18.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:17 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/224305069' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:43:18.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:17 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:43:18.228 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:18.262 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:18.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:18.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:18.364 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:18.404 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:18.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:18.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:18.506 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:18.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:18.528 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:18.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:18.543 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 29 16 53469 53453 115.165 83.75 0.0648667 0.00867293 2024-08-17T17:43:18.629 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:18.663 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:18.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:18.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:18.765 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:18.806 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:18.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:18.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:18.908 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:18.944 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:18.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:18.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:19.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:18 smithi003 ceph-mon[100615]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:43:19.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:18 smithi003 ceph-mon[100615]: Cluster is now healthy 2024-08-17T17:43:19.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:18 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:43:19.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:18 smithi003 ceph-mon[100615]: osdmap e371: 8 total, 8 up, 8 in 2024-08-17T17:43:19.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:18 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:43:19.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:18 smithi003 ceph-mon[103344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:43:19.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:18 smithi003 ceph-mon[103344]: Cluster is now healthy 2024-08-17T17:43:19.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:18 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:43:19.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:18 smithi003 ceph-mon[103344]: osdmap e371: 8 total, 8 up, 8 in 2024-08-17T17:43:19.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:18 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:43:19.046 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:19.080 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:19.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:19.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:19.182 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:19.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:18 smithi053 ceph-mon[89963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:43:19.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:18 smithi053 ceph-mon[89963]: Cluster is now healthy 2024-08-17T17:43:19.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:18 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:43:19.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:18 smithi053 ceph-mon[89963]: osdmap e371: 8 total, 8 up, 8 in 2024-08-17T17:43:19.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:18 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:43:19.220 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:19.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:19.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:19.322 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:19.347 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:19.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:19.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:19.449 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:19.473 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:19.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:19.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:19.543 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 30 16 54980 54964 114.474 94.4375 0.00545963 0.00873045 2024-08-17T17:43:19.574 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:19.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:19.610 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:19.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:19.711 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:19.747 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:19.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:19.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:19.849 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:19.871 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:19.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:19.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:19.973 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:20.006 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:20.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:20.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:20.108 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:20.146 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:20.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:20.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:20.215 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:19 smithi053 ceph-mon[89963]: pgmap v1204: 119 pgs: 119 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-08-17T17:43:20.248 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:20.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:19 smithi003 ceph-mon[103344]: pgmap v1204: 119 pgs: 119 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-08-17T17:43:20.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:19 smithi003 ceph-mon[100615]: pgmap v1204: 119 pgs: 119 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-08-17T17:43:20.280 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:20.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:20.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:20.382 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:20.432 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:20.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:20.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:20.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:43:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:43:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:43:20.534 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:20.545 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 31 16 56398 56382 113.64 88.625 0.0113805 0.00879423 2024-08-17T17:43:20.566 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:20.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:20.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:20.668 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:20.706 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:20.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:20.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:20.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:20.830 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:20.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:20.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:20.932 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:20.972 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:20.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:20.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:21.074 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:21.091 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:20 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:43:21.091 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:20 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:43:21.091 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:20 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:43:21.091 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:20 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:43:21.091 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:20 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:43:21.091 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:20 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:43:21.091 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:20 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:43:21.091 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:20 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:43:21.092 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:20 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:43:21.092 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:20 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:43:21.099 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:21.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:21.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:21.105 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:43:21.201 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:21.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:20 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:43:21.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:20 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:43:21.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:20 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:43:21.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:20 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:43:21.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:20 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:43:21.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:21.243 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:21.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:21.344 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:21.392 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:21.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:21.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:21.493 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:21.532 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:21.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:21.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:21.543 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 32 16 57532 57516 112.303 70.875 0.0120847 0.00889868 2024-08-17T17:43:21.634 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:21.682 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:21.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:21.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:21.784 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:21.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:21.826 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:21.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:21.901 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.1 to 1 (10000) 2024-08-17T17:43:21.928 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:21.974 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:21.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:21.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:22.067 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:43:22.067 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:43:22.075 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:22.080 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:21 smithi053 ceph-mon[89963]: pgmap v1205: 119 pgs: 119 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-08-17T17:43:22.080 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:21 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/920206603' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:43:22.080 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:21 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:43:22.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:22.111 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:22.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:22.211 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:22.250 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:22.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:22.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:22.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:21 smithi003 ceph-mon[103344]: pgmap v1205: 119 pgs: 119 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-08-17T17:43:22.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:21 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/920206603' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:43:22.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:21 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:43:22.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:21 smithi003 ceph-mon[100615]: pgmap v1205: 119 pgs: 119 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-08-17T17:43:22.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:21 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/920206603' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:43:22.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:21 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:43:22.353 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:22.362 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:43:22.385 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:22.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:22.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:22.487 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:22.524 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:22.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:22.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:22.543 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 33 16 59016 59000 111.71 92.75 0.0112643 0.00894627 2024-08-17T17:43:22.626 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:22.660 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:22.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:22.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:22.749 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-17T17:43:22.761 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:22.800 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:22.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:22.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:22.902 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:22.949 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:22.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:22.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:23.052 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:23.080 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:23.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:23.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:23.174 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:43:23.182 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:23.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:22 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-17T17:43:23.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:22 smithi053 ceph-mon[89963]: osdmap e372: 8 total, 8 up, 8 in 2024-08-17T17:43:23.224 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:23.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:23.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:23.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:22 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-17T17:43:23.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:22 smithi003 ceph-mon[103344]: osdmap e372: 8 total, 8 up, 8 in 2024-08-17T17:43:23.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:22 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-17T17:43:23.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:22 smithi003 ceph-mon[100615]: osdmap e372: 8 total, 8 up, 8 in 2024-08-17T17:43:23.326 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:23.364 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 2 1 2024-08-17T17:43:23.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:23.395 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:23.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:23.496 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:23.540 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:23.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:23.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:23.543 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 34 15 60420 60405 111.007 87.8125 0.0109763 0.00900285 2024-08-17T17:43:23.642 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:23.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:23.696 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:23.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:23.797 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:23.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:23.840 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:23.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:23.918 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:23 smithi053 ceph-mon[89963]: pgmap v1207: 119 pgs: 119 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-08-17T17:43:23.918 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:43:23.919 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:43:23.941 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:23.974 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:23.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:23.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:24.076 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:24.114 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:24.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:24.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:24.216 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:24.247 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:23 smithi003 ceph-mon[103344]: pgmap v1207: 119 pgs: 119 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-08-17T17:43:24.247 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:43:24.247 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:43:24.248 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:23 smithi003 ceph-mon[100615]: pgmap v1207: 119 pgs: 119 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-08-17T17:43:24.248 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:43:24.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:43:24.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:24.260 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:24.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:24.360 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:24.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:24.420 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:24.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:24.521 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:24.543 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 35 16 61877 61861 110.435 91 0.0130444 0.00904914 2024-08-17T17:43:24.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:24.572 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:24.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:24.673 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:24.709 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:24.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:24.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:24.811 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:24.854 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:24.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:24.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:24.926 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:43:24.929 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:43:24.929 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:43:24.947 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:43:24.956 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:24.959 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:24 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1890060764' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:43:24.967 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:43:24.968 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:43:25.006 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:43:25.007 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:43:25.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:25.014 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:25.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:25.051 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:24 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1890060764' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:43:25.051 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:24 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1890060764' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:43:25.052 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:43:25.052 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:43:25.086 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:43:25.086 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:43:25.115 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:25.115 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:43:25.115 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2024-08-17T17:43:25.115 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2024-08-17T17:43:25.116 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2024-08-17T17:43:25.116 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.116 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:75561 8.4 8:2c426cbf:::benchmark_data_smithi003_229103_object75560:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-08-17T17:43:25.116 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:43.185837+0000", 2024-08-17T17:43:25.116 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 581.92563517400004, 2024-08-17T17:43:25.116 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.98011287700000005, 2024-08-17T17:43:25.116 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:25.116 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:25.116 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:43:25.116 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:43:25.116 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:25.116 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 75561 2024-08-17T17:43:25.116 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.117 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:25.117 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.117 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:25.117 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.185837+0000", 2024-08-17T17:43:25.117 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.117 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.117 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.117 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:25.117 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.185837+0000", 2024-08-17T17:43:25.117 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.117 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.117 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.117 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:25.117 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.185836+0000", 2024-08-17T17:43:25.118 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:43:25.118 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.118 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.118 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:25.118 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.185885+0000", 2024-08-17T17:43:25.118 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9588e-05 2024-08-17T17:43:25.118 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.118 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.118 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:25.118 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.185896+0000", 2024-08-17T17:43:25.118 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0937e-05 2024-08-17T17:43:25.118 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.119 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.119 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:25.119 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.185900+0000", 2024-08-17T17:43:25.119 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8909999999999997e-06 2024-08-17T17:43:25.119 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.119 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.119 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:25.119 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.185957+0000", 2024-08-17T17:43:25.119 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7135000000000001e-05 2024-08-17T17:43:25.119 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.119 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.120 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:43:25.120 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.185963+0000", 2024-08-17T17:43:25.120 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6250000000000004e-06 2024-08-17T17:43:25.120 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.120 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.120 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:25.120 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.150376+0000", 2024-08-17T17:43:25.120 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.96441359400000004 2024-08-17T17:43:25.120 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.120 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.120 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:25.121 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.151738+0000", 2024-08-17T17:43:25.121 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001362032 2024-08-17T17:43:25.121 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.121 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.121 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:43:25.122 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.151916+0000", 2024-08-17T17:43:25.122 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00017807399999999999 2024-08-17T17:43:25.122 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.122 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.122 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:43:25.122 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.158973+0000", 2024-08-17T17:43:25.122 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0070567449999999997 2024-08-17T17:43:25.123 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.123 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.123 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.123 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165494+0000", 2024-08-17T17:43:25.123 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0065203179999999998 2024-08-17T17:43:25.123 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.123 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.123 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.123 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165827+0000", 2024-08-17T17:43:25.123 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00033381300000000002 2024-08-17T17:43:25.123 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.123 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.123 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:25.123 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165897+0000", 2024-08-17T17:43:25.124 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9472000000000002e-05 2024-08-17T17:43:25.124 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.124 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.124 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:25.124 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165949+0000", 2024-08-17T17:43:25.124 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2636999999999998e-05 2024-08-17T17:43:25.124 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.124 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:25.124 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.124 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.124 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.124 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:75575 8.4 8:241d3c27:::benchmark_data_smithi003_229103_object75574:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-08-17T17:43:25.125 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:43.209432+0000", 2024-08-17T17:43:25.125 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 581.90203991999999, 2024-08-17T17:43:25.125 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.95669896099999996, 2024-08-17T17:43:25.125 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:25.125 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:25.125 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:43:25.125 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:43:25.125 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:25.125 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 75575 2024-08-17T17:43:25.125 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.125 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:25.125 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.125 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:25.126 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.209432+0000", 2024-08-17T17:43:25.126 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.126 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.126 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.126 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:25.126 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.209432+0000", 2024-08-17T17:43:25.126 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.126 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.126 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.126 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:25.126 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.209430+0000", 2024-08-17T17:43:25.126 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:43:25.126 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.126 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.127 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:25.127 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.209491+0000", 2024-08-17T17:43:25.127 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1189e-05 2024-08-17T17:43:25.127 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.127 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.127 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:25.127 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.209532+0000", 2024-08-17T17:43:25.127 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0274000000000001e-05 2024-08-17T17:43:25.127 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.127 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.127 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:25.127 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.209542+0000", 2024-08-17T17:43:25.128 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0556e-05 2024-08-17T17:43:25.128 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.128 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.128 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:25.128 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.209612+0000", 2024-08-17T17:43:25.128 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9669e-05 2024-08-17T17:43:25.128 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.128 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.128 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:43:25.128 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.209623+0000", 2024-08-17T17:43:25.128 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1402e-05 2024-08-17T17:43:25.128 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.128 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.128 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:25.129 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.153453+0000", 2024-08-17T17:43:25.129 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.94382987200000001 2024-08-17T17:43:25.129 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.129 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.129 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:25.129 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.153538+0000", 2024-08-17T17:43:25.129 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.4448000000000002e-05 2024-08-17T17:43:25.129 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.129 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.129 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:43:25.129 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.153641+0000", 2024-08-17T17:43:25.129 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010312200000000001 2024-08-17T17:43:25.129 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.129 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.129 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:43:25.130 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.164927+0000", 2024-08-17T17:43:25.130 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011286503999999999 2024-08-17T17:43:25.130 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.130 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.130 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.130 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165658+0000", 2024-08-17T17:43:25.130 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00073055899999999998 2024-08-17T17:43:25.130 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.130 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.130 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.130 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.166034+0000", 2024-08-17T17:43:25.130 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000375891 2024-08-17T17:43:25.130 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.130 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.131 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:25.131 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.166087+0000", 2024-08-17T17:43:25.131 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3570999999999999e-05 2024-08-17T17:43:25.131 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.132 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:43:25.132 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:43:25.132 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.132 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:25.133 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.166131+0000", 2024-08-17T17:43:25.133 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3636e-05 2024-08-17T17:43:25.133 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.133 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:25.133 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.133 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.133 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.133 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:75590 8.4 8:2d80d3ce:::benchmark_data_smithi003_229103_object75589:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-08-17T17:43:25.134 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:43.225574+0000", 2024-08-17T17:43:25.134 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 581.88589778300002, 2024-08-17T17:43:25.134 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.94686139400000002, 2024-08-17T17:43:25.134 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:25.134 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:25.134 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:43:25.134 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:43:25.134 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:25.134 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 75590 2024-08-17T17:43:25.134 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.134 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:25.135 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.135 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:25.135 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.225574+0000", 2024-08-17T17:43:25.135 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.135 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.135 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.135 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:25.135 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.225574+0000", 2024-08-17T17:43:25.135 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.135 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.135 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.135 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:25.136 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.225573+0000", 2024-08-17T17:43:25.136 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:43:25.136 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.136 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.136 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:25.136 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.225645+0000", 2024-08-17T17:43:25.136 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.1625999999999995e-05 2024-08-17T17:43:25.136 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.136 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.136 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:25.136 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.225655+0000", 2024-08-17T17:43:25.137 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.043e-05 2024-08-17T17:43:25.137 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.137 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.137 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:25.137 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.225662+0000", 2024-08-17T17:43:25.137 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4420000000000003e-06 2024-08-17T17:43:25.137 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.137 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.137 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:25.137 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.225718+0000", 2024-08-17T17:43:25.137 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6232e-05 2024-08-17T17:43:25.137 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.138 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.138 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:43:25.138 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.225727+0000", 2024-08-17T17:43:25.138 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1949999999999992e-06 2024-08-17T17:43:25.138 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.138 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.138 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:25.138 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.155232+0000", 2024-08-17T17:43:25.138 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.92950502499999998 2024-08-17T17:43:25.138 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.138 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.138 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:25.139 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.155349+0000", 2024-08-17T17:43:25.139 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000117036 2024-08-17T17:43:25.139 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.139 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.139 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:43:25.139 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.155535+0000", 2024-08-17T17:43:25.139 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000185374 2024-08-17T17:43:25.139 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.139 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.139 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:43:25.139 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.164936+0000", 2024-08-17T17:43:25.139 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0094012939999999993 2024-08-17T17:43:25.139 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.139 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.140 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.140 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165743+0000", 2024-08-17T17:43:25.140 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00080753499999999996 2024-08-17T17:43:25.140 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.140 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.140 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.140 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.172337+0000", 2024-08-17T17:43:25.140 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0065936959999999996 2024-08-17T17:43:25.140 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.140 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.141 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:25.141 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.172389+0000", 2024-08-17T17:43:25.141 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1940000000000001e-05 2024-08-17T17:43:25.141 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.141 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.141 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:25.141 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.172435+0000", 2024-08-17T17:43:25.141 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6322999999999997e-05 2024-08-17T17:43:25.141 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.141 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:25.141 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.142 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.142 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.142 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:75606 8.4 8:2b96e39f:::benchmark_data_smithi003_229103_object75605:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-08-17T17:43:25.142 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:43.250288+0000", 2024-08-17T17:43:25.142 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 581.86118357299995, 2024-08-17T17:43:25.142 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.925042578, 2024-08-17T17:43:25.142 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:25.142 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:25.142 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:43:25.142 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:43:25.142 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:25.142 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 75606 2024-08-17T17:43:25.142 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.143 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:25.143 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.143 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:25.143 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.250288+0000", 2024-08-17T17:43:25.143 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.143 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.143 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.143 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:25.143 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.250288+0000", 2024-08-17T17:43:25.143 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.143 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.144 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.144 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:25.144 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.250287+0000", 2024-08-17T17:43:25.144 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:43:25.144 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.144 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.144 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:25.144 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.250335+0000", 2024-08-17T17:43:25.144 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.8593e-05 2024-08-17T17:43:25.145 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.145 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.145 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:25.145 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.250348+0000", 2024-08-17T17:43:25.145 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3128e-05 2024-08-17T17:43:25.145 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.145 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.145 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:25.145 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.250353+0000", 2024-08-17T17:43:25.145 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6589999999999998e-06 2024-08-17T17:43:25.145 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.145 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.146 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:25.146 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.250400+0000", 2024-08-17T17:43:25.146 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7262999999999998e-05 2024-08-17T17:43:25.146 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.146 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.146 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:43:25.146 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.250407+0000", 2024-08-17T17:43:25.146 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.6880000000000002e-06 2024-08-17T17:43:25.146 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.146 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.147 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:25.147 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.157095+0000", 2024-08-17T17:43:25.147 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.90668845600000003 2024-08-17T17:43:25.147 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.147 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.147 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:25.147 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.157173+0000", 2024-08-17T17:43:25.147 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7803e-05 2024-08-17T17:43:25.147 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.147 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.147 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:43:25.147 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.157268+0000", 2024-08-17T17:43:25.147 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.4374000000000001e-05 2024-08-17T17:43:25.148 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.148 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.148 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:43:25.148 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.164993+0000", 2024-08-17T17:43:25.148 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0077249240000000002 2024-08-17T17:43:25.148 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.148 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.148 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.148 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165789+0000", 2024-08-17T17:43:25.148 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000796486 2024-08-17T17:43:25.148 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.148 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.148 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.148 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175203+0000", 2024-08-17T17:43:25.149 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0094138179999999991 2024-08-17T17:43:25.149 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.149 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.149 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:25.149 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175253+0000", 2024-08-17T17:43:25.149 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9712000000000001e-05 2024-08-17T17:43:25.149 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.149 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.149 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:25.149 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175331+0000", 2024-08-17T17:43:25.149 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.8245999999999996e-05 2024-08-17T17:43:25.149 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.149 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:25.149 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.150 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.150 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.150 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:75612 8.4 8:2055fef1:::benchmark_data_smithi003_229103_object75611:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-08-17T17:43:25.150 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:43.260677+0000", 2024-08-17T17:43:25.150 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 581.85079476999999, 2024-08-17T17:43:25.150 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.91479233199999999, 2024-08-17T17:43:25.150 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:25.150 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:25.150 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:43:25.150 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:43:25.150 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:25.150 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 75612 2024-08-17T17:43:25.150 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.150 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:25.151 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.151 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:25.151 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.260677+0000", 2024-08-17T17:43:25.151 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.151 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.151 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.151 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:25.151 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.260677+0000", 2024-08-17T17:43:25.151 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.151 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.151 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.151 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:25.151 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.260676+0000", 2024-08-17T17:43:25.151 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:43:25.152 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.152 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.152 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:25.152 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.260728+0000", 2024-08-17T17:43:25.152 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1887e-05 2024-08-17T17:43:25.152 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.152 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.152 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:25.152 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.260737+0000", 2024-08-17T17:43:25.152 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.516e-06 2024-08-17T17:43:25.152 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.152 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.152 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:25.152 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.260741+0000", 2024-08-17T17:43:25.153 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5039999999999998e-06 2024-08-17T17:43:25.153 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.153 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.153 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:25.153 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.260794+0000", 2024-08-17T17:43:25.153 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3078000000000002e-05 2024-08-17T17:43:25.153 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.153 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.153 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:43:25.153 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.260805+0000", 2024-08-17T17:43:25.153 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1459e-05 2024-08-17T17:43:25.153 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.153 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.153 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:25.154 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.159008+0000", 2024-08-17T17:43:25.154 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.89820229900000004 2024-08-17T17:43:25.154 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.154 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.154 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:25.154 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.159094+0000", 2024-08-17T17:43:25.154 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.6119000000000007e-05 2024-08-17T17:43:25.154 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.154 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.154 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:43:25.154 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.159193+0000", 2024-08-17T17:43:25.155 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.9285999999999998e-05 2024-08-17T17:43:25.155 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.155 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.155 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:43:25.155 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165000+0000", 2024-08-17T17:43:25.155 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.005806833 2024-08-17T17:43:25.155 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.155 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.155 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.155 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165990+0000", 2024-08-17T17:43:25.155 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000990152 2024-08-17T17:43:25.155 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.155 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.155 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.155 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175382+0000", 2024-08-17T17:43:25.156 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0093919659999999999 2024-08-17T17:43:25.156 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.156 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.156 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:25.156 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175431+0000", 2024-08-17T17:43:25.156 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9101999999999998e-05 2024-08-17T17:43:25.156 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.156 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.156 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:25.156 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175469+0000", 2024-08-17T17:43:25.156 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8189000000000003e-05 2024-08-17T17:43:25.156 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.156 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:25.156 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.156 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.157 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.157 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:75613 8.4 8:2044481c:::benchmark_data_smithi003_229103_object75612:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-08-17T17:43:25.157 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:43.262848+0000", 2024-08-17T17:43:25.157 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 581.84862367200003, 2024-08-17T17:43:25.157 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.91278752399999996, 2024-08-17T17:43:25.157 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:25.157 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:25.157 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:43:25.157 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:43:25.157 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:25.157 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 75613 2024-08-17T17:43:25.157 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.157 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:25.157 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.157 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:25.157 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.262848+0000", 2024-08-17T17:43:25.158 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.158 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.158 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.158 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:25.158 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.262848+0000", 2024-08-17T17:43:25.158 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.158 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.158 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.158 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:25.158 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.262847+0000", 2024-08-17T17:43:25.158 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:43:25.158 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.158 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.158 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:25.158 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.262896+0000", 2024-08-17T17:43:25.159 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9047e-05 2024-08-17T17:43:25.159 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.159 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.159 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:25.159 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.262914+0000", 2024-08-17T17:43:25.159 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7093000000000001e-05 2024-08-17T17:43:25.159 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.159 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.159 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:25.159 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.262918+0000", 2024-08-17T17:43:25.159 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.561e-06 2024-08-17T17:43:25.159 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.160 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.160 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:25.160 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.262960+0000", 2024-08-17T17:43:25.160 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1659000000000002e-05 2024-08-17T17:43:25.160 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.160 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.160 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:43:25.160 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.262967+0000", 2024-08-17T17:43:25.160 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7279999999999995e-06 2024-08-17T17:43:25.160 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.160 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.160 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:25.160 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.160829+0000", 2024-08-17T17:43:25.160 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.89786185900000004 2024-08-17T17:43:25.161 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.161 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.161 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:25.161 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.160952+0000", 2024-08-17T17:43:25.161 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012245499999999999 2024-08-17T17:43:25.161 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.161 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.161 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:43:25.161 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.161116+0000", 2024-08-17T17:43:25.161 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000164208 2024-08-17T17:43:25.161 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.161 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.161 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:43:25.161 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165004+0000", 2024-08-17T17:43:25.162 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0038882230000000001 2024-08-17T17:43:25.162 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.162 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.162 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.162 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.167386+0000", 2024-08-17T17:43:25.162 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0023818960000000001 2024-08-17T17:43:25.162 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.162 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.162 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.162 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175558+0000", 2024-08-17T17:43:25.162 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0081719789999999994 2024-08-17T17:43:25.162 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.162 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.162 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:25.163 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175602+0000", 2024-08-17T17:43:25.163 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4008999999999998e-05 2024-08-17T17:43:25.163 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.163 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.163 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:25.163 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175636+0000", 2024-08-17T17:43:25.163 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3494999999999997e-05 2024-08-17T17:43:25.163 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.163 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:25.163 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.163 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.163 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.163 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:75614 8.4 8:2ac96a7e:::benchmark_data_smithi003_229103_object75613:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e272)", 2024-08-17T17:43:25.163 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:43.265251+0000", 2024-08-17T17:43:25.164 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 581.84622087100001, 2024-08-17T17:43:25.164 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.91049424000000001, 2024-08-17T17:43:25.164 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:25.164 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:25.164 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:43:25.164 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:43:25.164 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:25.164 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 75614 2024-08-17T17:43:25.164 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.164 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:25.164 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.164 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:25.164 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.265251+0000", 2024-08-17T17:43:25.164 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.165 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.165 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.165 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:25.165 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.265251+0000", 2024-08-17T17:43:25.165 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.165 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.165 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.165 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:25.166 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.265250+0000", 2024-08-17T17:43:25.166 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:43:25.166 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.166 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.166 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:25.166 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.265297+0000", 2024-08-17T17:43:25.166 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7088e-05 2024-08-17T17:43:25.166 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.166 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.166 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:25.166 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.265306+0000", 2024-08-17T17:43:25.166 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9819999999999997e-06 2024-08-17T17:43:25.166 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.167 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.167 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:25.167 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.265310+0000", 2024-08-17T17:43:25.167 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4189999999999998e-06 2024-08-17T17:43:25.167 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.167 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.167 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:25.167 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.265345+0000", 2024-08-17T17:43:25.167 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5271000000000002e-05 2024-08-17T17:43:25.167 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.167 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.167 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:43:25.167 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:43.265354+0000", 2024-08-17T17:43:25.167 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.5150000000000006e-06 2024-08-17T17:43:25.168 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.168 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.168 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:25.168 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.162842+0000", 2024-08-17T17:43:25.168 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.89748799899999998 2024-08-17T17:43:25.168 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.168 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.168 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:25.168 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.163010+0000", 2024-08-17T17:43:25.168 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016868699999999999 2024-08-17T17:43:25.168 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.168 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.168 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:43:25.168 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.163147+0000", 2024-08-17T17:43:25.169 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000136821 2024-08-17T17:43:25.169 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.169 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.169 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:43:25.169 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.165452+0000", 2024-08-17T17:43:25.169 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0023047039999999999 2024-08-17T17:43:25.169 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.169 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.169 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.169 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.167852+0000", 2024-08-17T17:43:25.169 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002400648 2024-08-17T17:43:25.169 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.169 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.169 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.169 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175678+0000", 2024-08-17T17:43:25.169 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0078257429999999996 2024-08-17T17:43:25.170 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.170 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.170 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:25.170 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175710+0000", 2024-08-17T17:43:25.170 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2128999999999997e-05 2024-08-17T17:43:25.170 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.170 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.170 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:25.170 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:44.175745+0000", 2024-08-17T17:43:25.170 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4854000000000002e-05 2024-08-17T17:43:25.170 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.170 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:25.170 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.170 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.171 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.171 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:85945 8.4 8:2a9d539f:::benchmark_data_smithi003_229103_object85944:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-08-17T17:43:25.171 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:56.753525+0000", 2024-08-17T17:43:25.171 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 568.35794703700003, 2024-08-17T17:43:25.171 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.19019344299999999, 2024-08-17T17:43:25.171 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:25.171 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:25.171 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:43:25.171 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:43:25.171 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:25.171 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 85945 2024-08-17T17:43:25.171 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.171 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:25.172 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.172 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:25.172 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.753525+0000", 2024-08-17T17:43:25.172 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.172 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.172 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.172 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:25.172 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.753525+0000", 2024-08-17T17:43:25.172 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.172 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.172 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.172 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:25.172 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.753523+0000", 2024-08-17T17:43:25.172 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:43:25.172 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.172 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.173 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:25.173 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.753569+0000", 2024-08-17T17:43:25.173 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5286e-05 2024-08-17T17:43:25.173 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.173 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.173 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:25.173 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.753590+0000", 2024-08-17T17:43:25.173 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1053999999999999e-05 2024-08-17T17:43:25.173 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.173 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.173 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:25.173 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.753594+0000", 2024-08-17T17:43:25.173 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9189999999999998e-06 2024-08-17T17:43:25.173 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.173 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.174 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:25.174 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.753651+0000", 2024-08-17T17:43:25.174 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6291e-05 2024-08-17T17:43:25.174 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.174 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.174 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:25.174 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.753776+0000", 2024-08-17T17:43:25.174 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012524199999999999 2024-08-17T17:43:25.174 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.174 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.174 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:43:25.174 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.753920+0000", 2024-08-17T17:43:25.174 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000143688 2024-08-17T17:43:25.174 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.175 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.175 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:43:25.175 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.756430+0000", 2024-08-17T17:43:25.175 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002510721 2024-08-17T17:43:25.175 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.175 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.175 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.175 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.756580+0000", 2024-08-17T17:43:25.175 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014915600000000001 2024-08-17T17:43:25.175 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.175 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.175 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.175 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.943621+0000", 2024-08-17T17:43:25.175 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.18704188599999999 2024-08-17T17:43:25.175 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.175 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.176 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:25.176 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.943674+0000", 2024-08-17T17:43:25.177 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:43:25.177 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:43:25.178 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3020999999999999e-05 2024-08-17T17:43:25.178 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.178 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.178 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:25.178 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.943718+0000", 2024-08-17T17:43:25.178 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3674999999999998e-05 2024-08-17T17:43:25.178 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.178 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:25.178 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.178 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.178 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.179 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:85950 8.4 8:2480217a:::benchmark_data_smithi003_229103_object85949:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e274)", 2024-08-17T17:43:25.179 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:33:56.759541+0000", 2024-08-17T17:43:25.179 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 568.35193085000003, 2024-08-17T17:43:25.179 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.19582751800000001, 2024-08-17T17:43:25.179 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:25.179 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:25.179 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:43:25.179 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:43:25.179 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:25.179 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 85950 2024-08-17T17:43:25.179 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.179 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:25.179 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.179 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:25.180 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.759541+0000", 2024-08-17T17:43:25.180 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.180 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.180 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.180 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:25.180 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.759541+0000", 2024-08-17T17:43:25.180 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.180 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.180 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.180 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:25.180 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.759539+0000", 2024-08-17T17:43:25.180 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:43:25.180 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.180 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.180 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:25.181 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.759623+0000", 2024-08-17T17:43:25.181 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.3555999999999998e-05 2024-08-17T17:43:25.181 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.181 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.181 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:25.181 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.759634+0000", 2024-08-17T17:43:25.181 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1618e-05 2024-08-17T17:43:25.181 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.181 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.181 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:25.181 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.759640+0000", 2024-08-17T17:43:25.181 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0630000000000002e-06 2024-08-17T17:43:25.181 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.181 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.182 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:25.182 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.759677+0000", 2024-08-17T17:43:25.182 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6434999999999999e-05 2024-08-17T17:43:25.182 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.182 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.182 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:25.182 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.759777+0000", 2024-08-17T17:43:25.182 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010024199999999999 2024-08-17T17:43:25.183 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.183 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.183 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:43:25.183 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.759863+0000", 2024-08-17T17:43:25.183 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.5628000000000001e-05 2024-08-17T17:43:25.184 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.184 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.184 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:43:25.184 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.762723+0000", 2024-08-17T17:43:25.184 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0028608790000000002 2024-08-17T17:43:25.184 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.184 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.184 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.184 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.764164+0000", 2024-08-17T17:43:25.184 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0014406239999999999 2024-08-17T17:43:25.184 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.184 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.184 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.185 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.955281+0000", 2024-08-17T17:43:25.185 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.19111708599999999 2024-08-17T17:43:25.185 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.185 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.185 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:25.185 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.955332+0000", 2024-08-17T17:43:25.185 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0785000000000003e-05 2024-08-17T17:43:25.185 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.185 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.185 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:25.185 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:33:56.955368+0000", 2024-08-17T17:43:25.185 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6528e-05 2024-08-17T17:43:25.185 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.185 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:25.185 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.186 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.186 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.186 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:131591 8.2 8:484be707:::benchmark_data_smithi003_229103_object24116:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-08-17T17:43:25.186 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:35:00.828656+0000", 2024-08-17T17:43:25.186 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 504.28281561699998, 2024-08-17T17:43:25.186 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.13269879100000001, 2024-08-17T17:43:25.186 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:25.186 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:25.186 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:43:25.186 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:43:25.187 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:25.187 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 131591 2024-08-17T17:43:25.187 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.187 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:25.187 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.187 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:25.187 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.828656+0000", 2024-08-17T17:43:25.187 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.187 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.187 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.188 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:25.188 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.828656+0000", 2024-08-17T17:43:25.188 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.188 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.188 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.188 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:25.188 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.828647+0000", 2024-08-17T17:43:25.188 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999909 2024-08-17T17:43:25.188 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.188 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.188 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:25.188 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.828666+0000", 2024-08-17T17:43:25.188 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.8836000000000001e-05 2024-08-17T17:43:25.188 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.189 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.189 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:25.189 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.828699+0000", 2024-08-17T17:43:25.189 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3535999999999999e-05 2024-08-17T17:43:25.189 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.189 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.189 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:25.189 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.828714+0000", 2024-08-17T17:43:25.189 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4779e-05 2024-08-17T17:43:25.189 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.189 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.189 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:25.189 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.828837+0000", 2024-08-17T17:43:25.189 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000122268 2024-08-17T17:43:25.189 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.190 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.190 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:25.190 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.829019+0000", 2024-08-17T17:43:25.190 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00018226200000000001 2024-08-17T17:43:25.190 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.190 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.190 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:43:25.190 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.829143+0000", 2024-08-17T17:43:25.190 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000124308 2024-08-17T17:43:25.190 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.190 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.190 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:43:25.190 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.835723+0000", 2024-08-17T17:43:25.190 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0065798590000000004 2024-08-17T17:43:25.190 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.191 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.191 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.191 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.836985+0000", 2024-08-17T17:43:25.191 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0012621900000000001 2024-08-17T17:43:25.191 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.191 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.191 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.191 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.961251+0000", 2024-08-17T17:43:25.191 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.124266183 2024-08-17T17:43:25.191 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.191 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.191 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:25.191 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.961306+0000", 2024-08-17T17:43:25.191 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5061e-05 2024-08-17T17:43:25.192 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.192 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.192 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:25.192 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.961355+0000", 2024-08-17T17:43:25.192 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.8547999999999997e-05 2024-08-17T17:43:25.192 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.192 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:25.192 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.192 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.192 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.192 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:131598 8.2 8:4c61b07f:::benchmark_data_smithi003_229103_object24123:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-08-17T17:43:25.192 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:35:00.830343+0000", 2024-08-17T17:43:25.192 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 504.28112875400001, 2024-08-17T17:43:25.192 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.13114300000000001, 2024-08-17T17:43:25.192 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:25.193 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:25.193 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:43:25.193 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:43:25.193 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:25.193 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 131598 2024-08-17T17:43:25.193 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.193 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:25.193 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.193 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:25.193 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.830343+0000", 2024-08-17T17:43:25.194 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.194 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.194 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.194 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:25.194 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.830343+0000", 2024-08-17T17:43:25.194 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.194 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.194 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.194 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:25.194 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.830342+0000", 2024-08-17T17:43:25.194 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:43:25.194 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.194 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.195 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:25.195 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.830348+0000", 2024-08-17T17:43:25.195 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6869999999999999e-06 2024-08-17T17:43:25.195 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.195 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.195 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:25.195 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.830361+0000", 2024-08-17T17:43:25.195 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3200000000000001e-05 2024-08-17T17:43:25.195 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.195 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.195 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:25.195 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.830366+0000", 2024-08-17T17:43:25.195 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5620000000000003e-06 2024-08-17T17:43:25.195 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.195 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.196 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:25.196 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.832067+0000", 2024-08-17T17:43:25.196 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0017004380000000001 2024-08-17T17:43:25.196 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.196 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.196 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:25.196 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.832204+0000", 2024-08-17T17:43:25.196 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013674299999999999 2024-08-17T17:43:25.196 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.196 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.196 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:43:25.196 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.832327+0000", 2024-08-17T17:43:25.196 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000123576 2024-08-17T17:43:25.196 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.197 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.197 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:43:25.197 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.835962+0000", 2024-08-17T17:43:25.197 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0036344350000000001 2024-08-17T17:43:25.197 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.197 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.197 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.197 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.837168+0000", 2024-08-17T17:43:25.197 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0012062119999999999 2024-08-17T17:43:25.197 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.198 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.198 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.198 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.961398+0000", 2024-08-17T17:43:25.198 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.124230363 2024-08-17T17:43:25.198 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.198 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.198 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:25.198 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.961445+0000", 2024-08-17T17:43:25.198 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7236e-05 2024-08-17T17:43:25.198 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.198 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.198 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:25.198 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:00.961486+0000", 2024-08-17T17:43:25.198 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0608999999999997e-05 2024-08-17T17:43:25.199 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.199 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:25.199 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.199 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.199 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.199 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:199665 8.4 8:2fc9da80:::benchmark_data_smithi003_229103_object92190:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e295)", 2024-08-17T17:43:25.199 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:40.260406+0000", 2024-08-17T17:43:25.199 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 404.85106623600001, 2024-08-17T17:43:25.199 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.17482935799999999, 2024-08-17T17:43:25.199 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:25.199 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:25.199 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:43:25.199 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:43:25.200 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:25.200 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 199665 2024-08-17T17:43:25.200 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.200 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:25.200 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.200 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:25.200 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.260406+0000", 2024-08-17T17:43:25.200 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.200 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.200 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.200 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:25.200 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.260406+0000", 2024-08-17T17:43:25.200 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.201 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.201 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.201 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:25.201 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.260405+0000", 2024-08-17T17:43:25.201 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:43:25.201 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.201 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.201 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:25.201 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.260409+0000", 2024-08-17T17:43:25.201 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4549999999999997e-06 2024-08-17T17:43:25.202 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.202 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.202 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:25.202 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.260418+0000", 2024-08-17T17:43:25.202 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.8720000000000008e-06 2024-08-17T17:43:25.202 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.202 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.202 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:25.202 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.260422+0000", 2024-08-17T17:43:25.202 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2590000000000002e-06 2024-08-17T17:43:25.202 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.202 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.203 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:25.203 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.264708+0000", 2024-08-17T17:43:25.203 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0042857590000000001 2024-08-17T17:43:25.203 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.203 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.203 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:25.203 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.264787+0000", 2024-08-17T17:43:25.203 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.9277999999999995e-05 2024-08-17T17:43:25.203 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.203 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.203 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:43:25.203 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.264861+0000", 2024-08-17T17:43:25.203 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3536999999999997e-05 2024-08-17T17:43:25.203 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.203 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.204 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:43:25.204 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.269373+0000", 2024-08-17T17:43:25.204 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0045125080000000001 2024-08-17T17:43:25.204 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.204 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.204 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.204 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.270503+0000", 2024-08-17T17:43:25.204 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001130065 2024-08-17T17:43:25.204 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.204 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.204 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.204 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435126+0000", 2024-08-17T17:43:25.204 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.16462236399999999 2024-08-17T17:43:25.204 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.204 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.205 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:25.205 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435185+0000", 2024-08-17T17:43:25.205 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9175999999999997e-05 2024-08-17T17:43:25.205 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.205 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.205 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:25.205 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435235+0000", 2024-08-17T17:43:25.205 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9889000000000003e-05 2024-08-17T17:43:25.205 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.205 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:25.205 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.205 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.205 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.205 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:199682 8.2 8:459cfaea:::benchmark_data_smithi003_229103_object92207:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e295)", 2024-08-17T17:43:25.205 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:40.289885+0000", 2024-08-17T17:43:25.206 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 404.82158663199999, 2024-08-17T17:43:25.206 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.14626383300000001, 2024-08-17T17:43:25.206 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:25.206 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:25.206 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:43:25.206 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:43:25.206 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:25.206 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 199682 2024-08-17T17:43:25.206 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.206 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:25.206 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.206 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:25.206 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289885+0000", 2024-08-17T17:43:25.207 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.207 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.207 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.207 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:25.207 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289885+0000", 2024-08-17T17:43:25.207 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.207 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.207 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.207 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:25.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:25.208 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:25.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:25.209 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289884+0000", 2024-08-17T17:43:25.209 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:43:25.209 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.209 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.209 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:25.209 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289890+0000", 2024-08-17T17:43:25.210 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5989999999999998e-06 2024-08-17T17:43:25.210 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.210 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.210 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:25.210 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289902+0000", 2024-08-17T17:43:25.210 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2007999999999999e-05 2024-08-17T17:43:25.210 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.210 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.210 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:25.210 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289919+0000", 2024-08-17T17:43:25.210 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.702e-05 2024-08-17T17:43:25.211 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.211 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.211 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:25.211 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289967+0000", 2024-08-17T17:43:25.211 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7367000000000002e-05 2024-08-17T17:43:25.211 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.211 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.211 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:25.211 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.290071+0000", 2024-08-17T17:43:25.211 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000103952 2024-08-17T17:43:25.211 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.211 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.211 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:43:25.211 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.290175+0000", 2024-08-17T17:43:25.212 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000104299 2024-08-17T17:43:25.212 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.212 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.212 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:43:25.212 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435038+0000", 2024-08-17T17:43:25.212 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.144863348 2024-08-17T17:43:25.212 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.212 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.212 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.212 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435989+0000", 2024-08-17T17:43:25.212 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00095057200000000001 2024-08-17T17:43:25.212 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.212 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.212 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.212 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.436062+0000", 2024-08-17T17:43:25.213 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3015999999999993e-05 2024-08-17T17:43:25.213 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.213 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.213 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:25.213 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.436109+0000", 2024-08-17T17:43:25.213 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7234000000000002e-05 2024-08-17T17:43:25.213 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.213 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.213 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:25.213 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.436149+0000", 2024-08-17T17:43:25.213 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9816e-05 2024-08-17T17:43:25.213 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.213 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:25.213 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.213 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.213 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.214 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop_reply(client.46025.0:199665 8.4 e295/272 ondisk, result = 0)", 2024-08-17T17:43:25.214 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:40.291703+0000", 2024-08-17T17:43:25.214 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 404.81976850699999, 2024-08-17T17:43:25.214 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.14354199400000001, 2024-08-17T17:43:25.214 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:25.214 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2024-08-17T17:43:25.214 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:25.214 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.214 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:25.214 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.291703+0000", 2024-08-17T17:43:25.214 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.214 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.214 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.214 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:25.214 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.291703+0000", 2024-08-17T17:43:25.215 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.215 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.215 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.215 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:25.215 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.291702+0000", 2024-08-17T17:43:25.215 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:43:25.215 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.215 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.215 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:25.215 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.291708+0000", 2024-08-17T17:43:25.215 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7710000000000003e-06 2024-08-17T17:43:25.215 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.215 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.215 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:25.215 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.430310+0000", 2024-08-17T17:43:25.216 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.13860172500000001 2024-08-17T17:43:25.216 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.216 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.216 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:25.216 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.430318+0000", 2024-08-17T17:43:25.216 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.4009999999999997e-06 2024-08-17T17:43:25.216 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.216 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.216 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:25.216 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435110+0000", 2024-08-17T17:43:25.216 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0047917469999999998 2024-08-17T17:43:25.216 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.216 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.216 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:25.216 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435119+0000", 2024-08-17T17:43:25.216 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.2280000000000004e-06 2024-08-17T17:43:25.217 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.217 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.217 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:25.217 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435245+0000", 2024-08-17T17:43:25.217 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012596800000000001 2024-08-17T17:43:25.217 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.217 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:25.217 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.217 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.217 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.217 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:199687 8.4 8:2561a798:::benchmark_data_smithi003_229103_object92212:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e295)", 2024-08-17T17:43:25.217 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:40.300569+0000", 2024-08-17T17:43:25.217 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 404.81090234499999, 2024-08-17T17:43:25.218 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.13988004500000001, 2024-08-17T17:43:25.218 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:25.218 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:25.218 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:43:25.218 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:43:25.218 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:25.218 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 199687 2024-08-17T17:43:25.218 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.218 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:25.218 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.218 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:25.219 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.300569+0000", 2024-08-17T17:43:25.219 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.219 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.219 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.219 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:25.219 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.300569+0000", 2024-08-17T17:43:25.219 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.219 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.219 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.219 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:25.219 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.300567+0000", 2024-08-17T17:43:25.219 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:43:25.220 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.220 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.220 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:25.220 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.300578+0000", 2024-08-17T17:43:25.220 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1070000000000001e-05 2024-08-17T17:43:25.220 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.220 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.220 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:25.220 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.430060+0000", 2024-08-17T17:43:25.220 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.129482028 2024-08-17T17:43:25.220 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.220 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.220 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:25.220 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.430078+0000", 2024-08-17T17:43:25.221 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7278999999999998e-05 2024-08-17T17:43:25.221 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.221 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.221 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:25.221 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.430135+0000", 2024-08-17T17:43:25.221 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7362999999999999e-05 2024-08-17T17:43:25.221 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.221 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.221 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:25.221 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.430275+0000", 2024-08-17T17:43:25.221 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014024400000000001 2024-08-17T17:43:25.221 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.222 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.222 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:43:25.222 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.430407+0000", 2024-08-17T17:43:25.222 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000131309 2024-08-17T17:43:25.222 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.223 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.223 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:43:25.223 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435256+0000", 2024-08-17T17:43:25.223 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0048498359999999997 2024-08-17T17:43:25.223 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.223 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.223 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.223 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435299+0000", 2024-08-17T17:43:25.224 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2561999999999997e-05 2024-08-17T17:43:25.224 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.224 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.224 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.224 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.440368+0000", 2024-08-17T17:43:25.224 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0050688039999999997 2024-08-17T17:43:25.224 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.224 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.224 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:25.224 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.440412+0000", 2024-08-17T17:43:25.224 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4582000000000001e-05 2024-08-17T17:43:25.224 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.224 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.224 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:25.224 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.440449+0000", 2024-08-17T17:43:25.225 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.7024999999999999e-05 2024-08-17T17:43:25.225 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.225 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:25.225 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.225 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.225 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.225 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:47655 10.1 10:8699caaa:::benchmark_data_smithi003_292749_object47654:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-08-17T17:43:25.225 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:14.182384+0000", 2024-08-17T17:43:25.225 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 10.929087778, 2024-08-17T17:43:25.225 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.25415833900000001, 2024-08-17T17:43:25.225 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:25.225 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:25.225 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:43:25.225 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:43:25.225 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:43:25.226 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 47655 2024-08-17T17:43:25.226 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.226 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:25.226 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.226 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:25.226 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182384+0000", 2024-08-17T17:43:25.226 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.226 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.226 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.226 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:25.226 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182384+0000", 2024-08-17T17:43:25.226 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.226 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.226 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.226 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:25.227 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182383+0000", 2024-08-17T17:43:25.227 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:43:25.227 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.227 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.227 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:25.227 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182543+0000", 2024-08-17T17:43:25.227 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016050299999999999 2024-08-17T17:43:25.227 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.227 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.227 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:25.227 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182569+0000", 2024-08-17T17:43:25.227 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5814000000000001e-05 2024-08-17T17:43:25.227 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.227 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.228 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:25.228 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182576+0000", 2024-08-17T17:43:25.228 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.6579999999999996e-06 2024-08-17T17:43:25.228 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.228 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.228 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:25.228 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182785+0000", 2024-08-17T17:43:25.228 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00020952300000000001 2024-08-17T17:43:25.228 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.229 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.229 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:25.229 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182968+0000", 2024-08-17T17:43:25.229 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00018305700000000001 2024-08-17T17:43:25.229 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.229 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.229 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,7", 2024-08-17T17:43:25.229 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.183099+0000", 2024-08-17T17:43:25.229 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013062999999999999 2024-08-17T17:43:25.229 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.229 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.229 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.229 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.436288+0000", 2024-08-17T17:43:25.229 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.25318881399999998 2024-08-17T17:43:25.230 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.230 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.230 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:43:25.230 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.436326+0000", 2024-08-17T17:43:25.230 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8656999999999999e-05 2024-08-17T17:43:25.230 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.230 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.230 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.230 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.436399+0000", 2024-08-17T17:43:25.230 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.2798000000000003e-05 2024-08-17T17:43:25.230 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.230 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.230 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:25.230 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.436501+0000", 2024-08-17T17:43:25.231 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000102064 2024-08-17T17:43:25.231 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.231 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.231 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:25.231 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.436542+0000", 2024-08-17T17:43:25.231 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1038000000000002e-05 2024-08-17T17:43:25.231 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.231 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:25.231 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.231 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.231 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.231 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:47658 10.e 10:71d71bc3:::benchmark_data_smithi003_292749_object47657:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-08-17T17:43:25.231 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:14.182625+0000", 2024-08-17T17:43:25.231 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 10.928846824000001, 2024-08-17T17:43:25.232 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.25454966400000001, 2024-08-17T17:43:25.232 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:25.232 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:25.232 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:43:25.232 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:43:25.232 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:43:25.232 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 47658 2024-08-17T17:43:25.232 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.232 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:25.232 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.232 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:25.232 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182625+0000", 2024-08-17T17:43:25.232 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.232 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.232 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.232 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:25.233 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182625+0000", 2024-08-17T17:43:25.233 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.233 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.233 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.233 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:25.233 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182624+0000", 2024-08-17T17:43:25.233 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:43:25.233 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.233 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.233 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:25.233 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182672+0000", 2024-08-17T17:43:25.233 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.8316000000000003e-05 2024-08-17T17:43:25.233 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.233 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.234 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:25.234 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182683+0000", 2024-08-17T17:43:25.234 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0292000000000001e-05 2024-08-17T17:43:25.234 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.234 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.234 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:25.234 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182688+0000", 2024-08-17T17:43:25.234 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3530000000000002e-06 2024-08-17T17:43:25.234 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.234 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.234 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:25.234 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182736+0000", 2024-08-17T17:43:25.234 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7615999999999999e-05 2024-08-17T17:43:25.234 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.234 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.235 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:25.235 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182854+0000", 2024-08-17T17:43:25.235 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000118405 2024-08-17T17:43:25.235 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.235 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.235 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,6", 2024-08-17T17:43:25.235 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182999+0000", 2024-08-17T17:43:25.235 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014474900000000001 2024-08-17T17:43:25.235 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.235 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.235 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.235 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.434748+0000", 2024-08-17T17:43:25.235 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.25174945700000001 2024-08-17T17:43:25.235 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.236 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.236 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:43:25.236 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.434786+0000", 2024-08-17T17:43:25.236 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.7866e-05 2024-08-17T17:43:25.236 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.236 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.236 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.236 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.437084+0000", 2024-08-17T17:43:25.236 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0022976289999999998 2024-08-17T17:43:25.236 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.236 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.237 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:25.237 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.437136+0000", 2024-08-17T17:43:25.237 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2423999999999998e-05 2024-08-17T17:43:25.237 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.237 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.237 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:25.237 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.437175+0000", 2024-08-17T17:43:25.237 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8452999999999999e-05 2024-08-17T17:43:25.237 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.237 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:25.237 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.237 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.237 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.238 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:47660 10.1 10:803ae287:::benchmark_data_smithi003_292749_object47659:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-08-17T17:43:25.238 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:14.182705+0000", 2024-08-17T17:43:25.238 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 10.928767194000001, 2024-08-17T17:43:25.238 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.25615803199999998, 2024-08-17T17:43:25.238 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:25.238 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:25.238 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:43:25.238 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:43:25.238 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:43:25.238 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 47660 2024-08-17T17:43:25.238 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.238 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:25.238 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.238 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:25.239 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182705+0000", 2024-08-17T17:43:25.239 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.239 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.239 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.239 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:25.239 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182705+0000", 2024-08-17T17:43:25.239 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.239 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.239 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.239 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:25.240 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182704+0000", 2024-08-17T17:43:25.240 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:43:25.240 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.240 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.240 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:25.240 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182731+0000", 2024-08-17T17:43:25.240 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7056e-05 2024-08-17T17:43:25.240 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.240 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.240 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:25.240 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182740+0000", 2024-08-17T17:43:25.240 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.4530000000000002e-06 2024-08-17T17:43:25.240 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.241 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.241 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:25.241 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182746+0000", 2024-08-17T17:43:25.241 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.81e-06 2024-08-17T17:43:25.241 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.241 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.241 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:25.241 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.432151+0000", 2024-08-17T17:43:25.241 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.24940480800000001 2024-08-17T17:43:25.241 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.241 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.241 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:25.241 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.432280+0000", 2024-08-17T17:43:25.242 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000128756 2024-08-17T17:43:25.242 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.242 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.242 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,7", 2024-08-17T17:43:25.242 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.432444+0000", 2024-08-17T17:43:25.242 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000164344 2024-08-17T17:43:25.242 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.242 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.242 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.242 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.436320+0000", 2024-08-17T17:43:25.242 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0038760370000000001 2024-08-17T17:43:25.242 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.243 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.243 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:43:25.243 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.436331+0000", 2024-08-17T17:43:25.243 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0494e-05 2024-08-17T17:43:25.243 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.243 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.243 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.243 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.438739+0000", 2024-08-17T17:43:25.243 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002408243 2024-08-17T17:43:25.243 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.243 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.243 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:25.243 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.438826+0000", 2024-08-17T17:43:25.243 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.6760000000000003e-05 2024-08-17T17:43:25.243 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.243 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.244 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:25.244 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.438863+0000", 2024-08-17T17:43:25.244 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6829000000000003e-05 2024-08-17T17:43:25.244 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.244 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:25.244 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.244 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.244 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.244 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.36612.0:47656 10.0 e370/357 10:02151d15:::benchmark_data_smithi003_292749_object47655:head v 370'2871, mlcod=370'2869)", 2024-08-17T17:43:25.244 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:14.183177+0000", 2024-08-17T17:43:25.244 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 10.928295137999999, 2024-08-17T17:43:25.244 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.25087112900000003, 2024-08-17T17:43:25.244 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:25.244 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:25.245 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:25.245 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.245 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:25.245 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.183177+0000", 2024-08-17T17:43:25.245 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.245 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.245 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.245 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:25.245 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.183177+0000", 2024-08-17T17:43:25.245 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.245 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.245 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.245 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:25.245 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.183176+0000", 2024-08-17T17:43:25.245 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:43:25.245 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.246 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.246 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:25.246 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.183217+0000", 2024-08-17T17:43:25.246 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0825999999999998e-05 2024-08-17T17:43:25.246 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.246 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.246 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:25.246 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.430423+0000", 2024-08-17T17:43:25.246 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.247206178 2024-08-17T17:43:25.246 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.246 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.246 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:25.249 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.430437+0000", 2024-08-17T17:43:25.249 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4673e-05 2024-08-17T17:43:25.250 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.250 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.250 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:25.250 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.430489+0000", 2024-08-17T17:43:25.250 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1541e-05 2024-08-17T17:43:25.250 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.250 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.250 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:25.250 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.430516+0000", 2024-08-17T17:43:25.250 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6764e-05 2024-08-17T17:43:25.251 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.251 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.251 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:25.251 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.433951+0000", 2024-08-17T17:43:25.251 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0034349010000000002 2024-08-17T17:43:25.251 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.251 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.251 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:25.251 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.434048+0000", 2024-08-17T17:43:25.251 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.7137000000000002e-05 2024-08-17T17:43:25.251 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.251 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:25.251 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.251 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.252 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.252 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:47666 10.e 10:7e345cce:::benchmark_data_smithi003_292749_object47665:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-08-17T17:43:25.252 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:14.187962+0000", 2024-08-17T17:43:25.252 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 10.923509287, 2024-08-17T17:43:25.252 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.25411109900000001, 2024-08-17T17:43:25.252 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:25.252 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:25.252 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:43:25.252 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:43:25.252 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:43:25.252 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 47666 2024-08-17T17:43:25.252 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.252 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:25.252 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.252 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:25.253 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.187962+0000", 2024-08-17T17:43:25.253 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.253 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.253 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.253 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:25.253 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.187962+0000", 2024-08-17T17:43:25.253 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:25.253 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.253 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.253 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:25.253 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.187961+0000", 2024-08-17T17:43:25.253 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:43:25.253 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.253 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.254 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:25.254 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.187997+0000", 2024-08-17T17:43:25.254 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6801999999999998e-05 2024-08-17T17:43:25.254 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.254 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.254 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:25.254 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.430420+0000", 2024-08-17T17:43:25.254 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.24242243999999999 2024-08-17T17:43:25.254 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.254 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.254 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:25.254 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.430438+0000", 2024-08-17T17:43:25.254 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.8215000000000001e-05 2024-08-17T17:43:25.255 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.255 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.255 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:25.255 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.432267+0000", 2024-08-17T17:43:25.255 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0018294559999999999 2024-08-17T17:43:25.255 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.255 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.255 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:25.255 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.432399+0000", 2024-08-17T17:43:25.255 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013195500000000001 2024-08-17T17:43:25.255 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.255 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.255 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,6", 2024-08-17T17:43:25.255 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.432528+0000", 2024-08-17T17:43:25.256 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012893799999999999 2024-08-17T17:43:25.256 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.256 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.256 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.256 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.434780+0000", 2024-08-17T17:43:25.256 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002251795 2024-08-17T17:43:25.256 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.256 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.256 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:43:25.256 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.434791+0000", 2024-08-17T17:43:25.256 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0694e-05 2024-08-17T17:43:25.256 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.256 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.256 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:25.256 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.441939+0000", 2024-08-17T17:43:25.257 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0071477600000000004 2024-08-17T17:43:25.257 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.257 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.257 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:25.257 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.442033+0000", 2024-08-17T17:43:25.257 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.4889999999999994e-05 2024-08-17T17:43:25.257 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:25.257 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:25.257 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:25.257 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.442074+0000", 2024-08-17T17:43:25.257 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0085e-05 2024-08-17T17:43:25.257 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.257 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:25.257 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.257 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:25.258 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:25.258 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:43:25.258 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:43:25.258 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:43:25.310 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:25.348 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:25.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:25.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:25.450 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:25.484 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:25.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:25.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:25.544 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 36 16 63220 63204 109.699 83.9375 0.0060041 0.00911048 2024-08-17T17:43:25.586 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:25.624 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:25.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:25.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:25.725 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:25.773 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:25.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:25.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:25.875 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:25.913 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:25.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:25.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:25.948 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:43:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:43:25] "GET /metrics HTTP/1.1" 200 36572 "" "Prometheus/2.51.0" 2024-08-17T17:43:26.016 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:26.051 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:26.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:26.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:26.153 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:26.162 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:25 smithi053 ceph-mon[89963]: pgmap v1208: 119 pgs: 119 active+clean; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-08-17T17:43:26.200 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:26.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:26.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:26.213 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-17T17:43:26.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:25 smithi003 ceph-mon[103344]: pgmap v1208: 119 pgs: 119 active+clean; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-08-17T17:43:26.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:25 smithi003 ceph-mon[100615]: pgmap v1208: 119 pgs: 119 active+clean; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 87 MiB/s wr, 1.40k op/s 2024-08-17T17:43:26.302 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:26.345 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:26.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:26.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:26.447 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:26.493 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:26.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:26.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:26.545 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 37 16 64703 64687 109.235 92.6875 0.0076715 0.00914852 2024-08-17T17:43:26.595 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:26.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:26.642 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:26.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:26.716 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:43:26.743 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:26.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:26.781 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:26.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:26.882 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:26.924 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:26.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:26.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:27.025 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:27.070 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:27.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:27.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:27.172 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:27.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:27.213 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:27.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:27.314 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:27.350 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:27.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:27.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:27.452 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:27.486 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:27.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:27.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:27.546 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 38 16 66074 66058 108.615 85.6875 0.0129445 0.00920184 2024-08-17T17:43:27.588 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:27.625 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:27.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:27.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:27.726 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:27.762 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:27.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:27.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:27.864 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:27.895 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:27.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:27.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:27.997 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:28.034 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:28.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:28.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:28.136 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:28.144 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:27 smithi003 ceph-mon[103344]: pgmap v1209: 119 pgs: 119 active+clean; 3.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-08-17T17:43:28.144 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:27 smithi003 ceph-mon[100615]: pgmap v1209: 119 pgs: 119 active+clean; 3.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-08-17T17:43:28.173 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:27 smithi053 ceph-mon[89963]: pgmap v1209: 119 pgs: 119 active+clean; 3.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 101 MiB/s wr, 1.61k op/s 2024-08-17T17:43:28.177 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:28.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:28.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:28.279 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:28.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:28.313 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:28.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:28.414 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:28.444 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:28.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:28.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:28.546 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 39 16 67347 67331 107.87 79.5625 0.00374362 0.00926339 2024-08-17T17:43:28.546 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:28.582 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:28.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:28.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:28.598 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:43:28.599 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2024-08-17T17:43:28.599 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 0 2024-08-17T17:43:28.599 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:43:28.683 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:28.722 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:28.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:28.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:28.824 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:28.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:28.868 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:28.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:28.931 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:43:28.969 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:28.971 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.2 to 1 (10000) 2024-08-17T17:43:28.994 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:28.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:28.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:29.096 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:29.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:29.120 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:29.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:29.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:28 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/881402027' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:43:29.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:28 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:43:29.221 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:29.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:28 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/881402027' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:43:29.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:28 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:43:29.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:28 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/881402027' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:43:29.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:28 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:43:29.275 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:29.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:29.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:29.284 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-17T17:43:29.377 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:29.418 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:29.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:29.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:29.520 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:29.545 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-08-17T17:43:29.545924+0000 min lat: 0.00107156 max lat: 2.04224 avg lat: 0.00934925 2024-08-17T17:43:29.546 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:43:29.546 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 40 16 68451 68435 106.899 69 0.0148182 0.00934925 2024-08-17T17:43:29.556 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:29.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:29.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:29.658 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:29.701 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:29.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:29.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:29.731 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:43:29.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:29.841 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:29.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:29.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:29.943 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:29.966 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:29.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:29.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:30.068 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:30.090 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:30.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:30.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:30.162 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:29 smithi053 ceph-mon[89963]: pgmap v1210: 119 pgs: 119 active+clean; 3.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-08-17T17:43:30.162 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:29 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-17T17:43:30.162 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:29 smithi053 ceph-mon[89963]: osdmap e373: 8 total, 8 up, 8 in 2024-08-17T17:43:30.192 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:30.230 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:30.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:30.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:30.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:29 smithi003 ceph-mon[100615]: pgmap v1210: 119 pgs: 119 active+clean; 3.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-08-17T17:43:30.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:30 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-17T17:43:30.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:30 smithi003 ceph-mon[100615]: osdmap e373: 8 total, 8 up, 8 in 2024-08-17T17:43:30.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:29 smithi003 ceph-mon[103344]: pgmap v1210: 119 pgs: 119 active+clean; 3.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-08-17T17:43:30.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:30 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-17T17:43:30.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:30 smithi003 ceph-mon[103344]: osdmap e373: 8 total, 8 up, 8 in 2024-08-17T17:43:30.332 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:30.371 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:30.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:30.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:30.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:43:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:43:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:43:30.473 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:30.508 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:30.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:30.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:30.545 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 41 16 69534 69518 105.942 67.6875 0.0149287 0.00943453 2024-08-17T17:43:30.611 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:30.648 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:30.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:30.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:30.750 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:30.786 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:30.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:30.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:30.888 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:30.931 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:30.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:30.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:31.033 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:31.070 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:31.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:31.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:31.171 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:31.208 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:31.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:31.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:31.310 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:31.347 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:31.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:31.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:31.448 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:31.489 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:31.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:31.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:31.545 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 42 16 70918 70902 105.479 86.5 0.0120039 0.00947561 2024-08-17T17:43:31.591 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:31.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:31.622 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:31.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:31.723 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:31.756 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:31.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:31.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:31.858 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:31.896 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:31.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:31.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:31.998 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:32.039 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:32.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:32.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:32.141 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:32.177 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:32.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:32.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:32.272 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:31 smithi003 ceph-mon[100615]: pgmap v1212: 119 pgs: 119 active+clean; 4.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-08-17T17:43:32.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:31 smithi003 ceph-mon[103344]: pgmap v1212: 119 pgs: 119 active+clean; 4.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-08-17T17:43:32.278 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:32.314 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:32.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:32.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:32.399 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:31 smithi053 ceph-mon[89963]: pgmap v1212: 119 pgs: 119 active+clean; 4.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-08-17T17:43:32.416 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:32.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:32.449 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:32.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:32.546 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 43 16 72191 72175 104.877 79.5625 0.0123929 0.00952943 2024-08-17T17:43:32.549 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:32.586 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:32.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:32.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:32.688 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:32.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:32.722 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:32.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:32.823 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:32.857 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:32.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:32.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:32.959 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:32.993 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:32.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:32.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:33.083 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:43:33.083 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2024-08-17T17:43:33.083 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2024-08-17T17:43:33.083 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2024-08-17T17:43:33.083 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:43:33.095 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:33.142 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:33.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:33.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:33.244 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:33.280 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:33.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:33.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:33.382 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:33.417 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:33.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:33.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:33.519 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:33.546 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 44 16 73447 73431 104.277 78.5 0.0107428 0.00958387 2024-08-17T17:43:33.557 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:33.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:33.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:33.659 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:33.687 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:33.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:33.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:33.788 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:33.826 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:33.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:33.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:33.928 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:33.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:33.969 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:33.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:34.070 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:34.098 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:33 smithi053 ceph-mon[89963]: pgmap v1213: 119 pgs: 119 active+clean; 4.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-08-17T17:43:34.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:34.113 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:34.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:34.143 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-17T17:43:34.214 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:34.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:33 smithi003 ceph-mon[103344]: pgmap v1213: 119 pgs: 119 active+clean; 4.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-08-17T17:43:34.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:33 smithi003 ceph-mon[100615]: pgmap v1213: 119 pgs: 119 active+clean; 4.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-08-17T17:43:34.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:34.274 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:34.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:34.375 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:34.398 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:34.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:34.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:34.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:34.534 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:34.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:34.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:34.546 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 45 16 74685 74669 103.679 77.375 0.0169599 0.00963958 2024-08-17T17:43:34.636 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:34.637 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:43:34.673 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:34.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:34.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:34.775 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:34.807 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:34.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:34.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:34.909 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:34.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:34.950 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:34.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:35.051 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:35.085 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:35.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:35.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:35.187 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:35.228 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:35.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:35.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:35.329 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:35.369 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:35.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:35.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:35.470 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:35.511 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:35.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:35.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:35.546 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 46 16 76038 76022 103.263 84.5625 0.00863337 0.00967931 2024-08-17T17:43:35.614 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:35.653 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 3 1 2024-08-17T17:43:35.671 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:43:35.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:35.678 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:35.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:35.779 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:35.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:35.835 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:35.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:35.925 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:43:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:43:35] "GET /metrics HTTP/1.1" 200 36572 "" "Prometheus/2.51.0" 2024-08-17T17:43:35.936 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:35.998 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:35.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:35.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:36.094 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:35 smithi053 ceph-mon[89963]: pgmap v1214: 119 pgs: 119 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-08-17T17:43:36.100 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:36.149 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:36.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:36.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:36.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:35 smithi003 ceph-mon[100615]: pgmap v1214: 119 pgs: 119 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-08-17T17:43:36.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:36 smithi003 ceph-mon[103344]: pgmap v1214: 119 pgs: 119 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-08-17T17:43:36.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:36.279 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:36.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:36.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:36.381 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:36.420 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:36.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:36.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:36.522 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:36.529 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:43:36.529 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2024-08-17T17:43:36.529 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2024-08-17T17:43:36.529 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2024-08-17T17:43:36.529 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.529 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:170851 8.3 8:ca2380e7:::benchmark_data_smithi003_229103_object63376:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e289)", 2024-08-17T17:43:36.529 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:35:58.844092+0000", 2024-08-17T17:43:36.529 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 457.67977965199998, 2024-08-17T17:43:36.529 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.087490153000000001, 2024-08-17T17:43:36.529 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:36.529 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:36.529 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:43:36.530 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:43:36.530 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:36.530 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 170851 2024-08-17T17:43:36.530 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.530 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:36.530 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.530 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:36.530 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:58.844092+0000", 2024-08-17T17:43:36.530 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.530 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.530 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.530 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:36.531 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:58.844092+0000", 2024-08-17T17:43:36.531 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.531 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.531 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.531 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:36.531 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:58.844091+0000", 2024-08-17T17:43:36.531 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:43:36.531 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.531 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.531 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:36.531 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:58.844096+0000", 2024-08-17T17:43:36.531 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0950000000000003e-06 2024-08-17T17:43:36.531 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.531 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.531 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:36.532 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:58.844110+0000", 2024-08-17T17:43:36.532 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3652000000000001e-05 2024-08-17T17:43:36.532 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.532 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.532 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:36.532 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:58.844117+0000", 2024-08-17T17:43:36.532 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.2119999999999997e-06 2024-08-17T17:43:36.532 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.532 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.532 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:36.532 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:58.854843+0000", 2024-08-17T17:43:36.532 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.010726009999999999 2024-08-17T17:43:36.532 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.533 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.533 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:36.533 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:58.854991+0000", 2024-08-17T17:43:36.533 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014785300000000001 2024-08-17T17:43:36.533 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.533 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.533 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:43:36.533 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:58.855131+0000", 2024-08-17T17:43:36.533 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000139686 2024-08-17T17:43:36.533 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.533 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.533 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:43:36.533 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:58.862648+0000", 2024-08-17T17:43:36.533 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0075177969999999997 2024-08-17T17:43:36.533 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.534 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.534 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:36.534 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:58.862694+0000", 2024-08-17T17:43:36.534 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6068e-05 2024-08-17T17:43:36.534 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.534 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.534 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:36.534 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:58.930568+0000", 2024-08-17T17:43:36.534 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.067873304999999995 2024-08-17T17:43:36.534 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.534 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.534 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:36.534 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:58.930606+0000", 2024-08-17T17:43:36.534 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8549e-05 2024-08-17T17:43:36.535 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.535 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.535 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:36.535 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:35:58.931582+0000", 2024-08-17T17:43:36.535 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00097578599999999997 2024-08-17T17:43:36.535 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.535 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:36.535 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.535 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.535 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.535 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:176586 8.3 8:ca2246a8:::benchmark_data_smithi003_229103_object69111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-08-17T17:43:36.535 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:06.846318+0000", 2024-08-17T17:43:36.535 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 449.677553209, 2024-08-17T17:43:36.535 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.085138861999999996, 2024-08-17T17:43:36.536 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:36.536 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:36.536 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:43:36.536 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:43:36.536 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:36.536 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 176586 2024-08-17T17:43:36.536 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.536 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:36.536 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.536 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:36.536 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:06.846318+0000", 2024-08-17T17:43:36.536 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.536 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.536 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.536 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:36.537 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:06.846318+0000", 2024-08-17T17:43:36.537 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.537 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.537 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.537 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:36.537 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:06.846317+0000", 2024-08-17T17:43:36.537 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:43:36.537 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.537 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.537 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:36.537 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:06.846325+0000", 2024-08-17T17:43:36.537 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.5169999999999999e-06 2024-08-17T17:43:36.537 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.537 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.538 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:36.538 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:06.846347+0000", 2024-08-17T17:43:36.538 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1501e-05 2024-08-17T17:43:36.538 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.538 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.538 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:36.538 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:06.846354+0000", 2024-08-17T17:43:36.538 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.6340000000000001e-06 2024-08-17T17:43:36.538 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.538 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.538 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:36.538 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:06.853741+0000", 2024-08-17T17:43:36.538 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0073862529999999997 2024-08-17T17:43:36.538 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.539 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.539 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:36.539 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:06.853859+0000", 2024-08-17T17:43:36.539 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000118572 2024-08-17T17:43:36.539 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.539 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.539 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:43:36.539 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:06.853991+0000", 2024-08-17T17:43:36.539 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000132187 2024-08-17T17:43:36.539 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.539 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.539 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:43:36.539 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:06.859119+0000", 2024-08-17T17:43:36.540 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0051280759999999996 2024-08-17T17:43:36.540 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.540 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.540 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:36.540 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:06.859791+0000", 2024-08-17T17:43:36.540 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00067184799999999995 2024-08-17T17:43:36.540 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.540 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.540 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:36.540 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:06.930115+0000", 2024-08-17T17:43:36.540 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.070324052999999997 2024-08-17T17:43:36.540 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.540 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.541 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:36.541 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:06.930215+0000", 2024-08-17T17:43:36.541 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.9716999999999994e-05 2024-08-17T17:43:36.541 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.541 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.541 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:36.541 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:06.931457+0000", 2024-08-17T17:43:36.541 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001242188 2024-08-17T17:43:36.541 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.541 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:36.541 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.541 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.541 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.542 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:196068 8.3 8:c0a6037d:::benchmark_data_smithi003_229103_object88593:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-08-17T17:43:36.542 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:34.989805+0000", 2024-08-17T17:43:36.542 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 421.53406682600001, 2024-08-17T17:43:36.542 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.086223106999999993, 2024-08-17T17:43:36.542 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:36.542 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:36.542 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:43:36.542 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:43:36.542 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:36.542 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 196068 2024-08-17T17:43:36.542 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.542 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:36.542 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.543 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:36.543 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:34.989805+0000", 2024-08-17T17:43:36.543 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.543 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.543 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.543 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:36.543 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:34.989805+0000", 2024-08-17T17:43:36.543 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.543 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.543 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.543 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:36.543 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:34.989803+0000", 2024-08-17T17:43:36.543 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:43:36.544 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.544 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.544 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:36.544 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:34.989812+0000", 2024-08-17T17:43:36.544 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.7430000000000008e-06 2024-08-17T17:43:36.544 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.544 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.544 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:36.544 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:34.989831+0000", 2024-08-17T17:43:36.544 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.8896e-05 2024-08-17T17:43:36.544 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.544 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.544 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:36.544 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:34.989838+0000", 2024-08-17T17:43:36.545 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.1860000000000003e-06 2024-08-17T17:43:36.545 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.545 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.545 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:36.545 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:34.989891+0000", 2024-08-17T17:43:36.545 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3409000000000003e-05 2024-08-17T17:43:36.545 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.545 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.545 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:36.545 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:34.990047+0000", 2024-08-17T17:43:36.545 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015573 2024-08-17T17:43:36.545 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.545 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.545 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:43:36.545 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:34.990197+0000", 2024-08-17T17:43:36.546 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014992900000000001 2024-08-17T17:43:36.546 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.546 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.546 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:36.546 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:34.996524+0000", 2024-08-17T17:43:36.546 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0063272900000000002 2024-08-17T17:43:36.546 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.546 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.546 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:43:36.546 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:34.996541+0000", 2024-08-17T17:43:36.546 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7283e-05 2024-08-17T17:43:36.546 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.546 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.547 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:36.547 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:35.075030+0000", 2024-08-17T17:43:36.547 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.078488675999999993 2024-08-17T17:43:36.547 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.547 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.547 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:36.547 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:35.075073+0000", 2024-08-17T17:43:36.547 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3139999999999997e-05 2024-08-17T17:43:36.547 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.547 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.547 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:36.547 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:35.076028+0000", 2024-08-17T17:43:36.547 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00095467800000000004 2024-08-17T17:43:36.548 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.548 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:36.548 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.548 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.548 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.548 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:199676 8.3 8:cc22ccb8:::benchmark_data_smithi003_229103_object92201:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e295)", 2024-08-17T17:43:36.548 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:40.279393+0000", 2024-08-17T17:43:36.548 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 416.24447808600002, 2024-08-17T17:43:36.548 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.16695074400000001, 2024-08-17T17:43:36.548 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:36.548 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:36.548 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:43:36.548 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:43:36.549 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:36.549 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 199676 2024-08-17T17:43:36.549 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.549 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:36.549 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.549 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:36.549 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.279393+0000", 2024-08-17T17:43:36.549 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.549 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.549 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.549 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:36.549 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.279393+0000", 2024-08-17T17:43:36.549 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.549 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.549 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.550 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:36.550 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.279392+0000", 2024-08-17T17:43:36.550 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:43:36.550 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.550 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.550 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:36.550 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.279398+0000", 2024-08-17T17:43:36.550 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5110000000000003e-06 2024-08-17T17:43:36.550 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.550 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.550 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:36.550 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.279410+0000", 2024-08-17T17:43:36.550 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2189e-05 2024-08-17T17:43:36.551 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.551 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.551 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:36.551 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.279415+0000", 2024-08-17T17:43:36.551 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4079999999999997e-06 2024-08-17T17:43:36.551 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.551 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.551 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:36.551 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.279463+0000", 2024-08-17T17:43:36.551 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.8162999999999999e-05 2024-08-17T17:43:36.551 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.551 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.551 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:36.552 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.279551+0000", 2024-08-17T17:43:36.552 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.7639999999999994e-05 2024-08-17T17:43:36.552 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.552 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.552 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:43:36.552 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.279630+0000", 2024-08-17T17:43:36.552 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.9404000000000001e-05 2024-08-17T17:43:36.552 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.552 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.552 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:43:36.552 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.284840+0000", 2024-08-17T17:43:36.552 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0052092420000000002 2024-08-17T17:43:36.552 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.552 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.552 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:36.552 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.285008+0000", 2024-08-17T17:43:36.553 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016781000000000001 2024-08-17T17:43:36.553 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.553 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.553 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:36.553 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.445095+0000", 2024-08-17T17:43:36.553 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.160087962 2024-08-17T17:43:36.553 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.553 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.553 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:36.553 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.445149+0000", 2024-08-17T17:43:36.553 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3649000000000001e-05 2024-08-17T17:43:36.553 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.553 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.553 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:36.554 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.446344+0000", 2024-08-17T17:43:36.554 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00119509 2024-08-17T17:43:36.554 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.554 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:36.554 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.554 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.554 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.554 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.46025.0:199677 8.0 e295/269 8:06071170:::benchmark_data_smithi003_229103_object92202:head v 295'12428, mlcod=0'0)", 2024-08-17T17:43:36.554 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:40.284892+0000", 2024-08-17T17:43:36.554 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 416.238979261, 2024-08-17T17:43:36.554 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.145367366, 2024-08-17T17:43:36.554 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:36.554 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:36.554 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:36.555 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.555 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:36.555 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.284892+0000", 2024-08-17T17:43:36.555 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.555 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.555 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.555 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:36.555 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.284892+0000", 2024-08-17T17:43:36.555 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.555 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.555 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.555 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:36.555 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.284891+0000", 2024-08-17T17:43:36.555 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:43:36.556 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.556 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.556 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:36.556 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.284899+0000", 2024-08-17T17:43:36.556 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.6559999999999992e-06 2024-08-17T17:43:36.556 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.556 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.556 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:36.556 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.284919+0000", 2024-08-17T17:43:36.556 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0423000000000001e-05 2024-08-17T17:43:36.556 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.556 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.556 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:36.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:36.558 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:36.558 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.284926+0000", 2024-08-17T17:43:36.558 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.7460000000000003e-06 2024-08-17T17:43:36.559 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.559 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.559 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:36.559 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.286181+0000", 2024-08-17T17:43:36.559 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0012550129999999999 2024-08-17T17:43:36.559 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.559 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.559 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:36.559 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.286196+0000", 2024-08-17T17:43:36.559 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4731e-05 2024-08-17T17:43:36.559 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.559 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.559 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:36.559 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.293234+0000", 2024-08-17T17:43:36.560 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0070382680000000003 2024-08-17T17:43:36.560 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.560 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.560 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:36.560 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.430260+0000", 2024-08-17T17:43:36.560 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.137025444 2024-08-17T17:43:36.560 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.560 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:36.560 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.560 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.560 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.560 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:199681 8.3 8:c67e547d:::benchmark_data_smithi003_229103_object92206:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e295)", 2024-08-17T17:43:36.560 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:40.286985+0000", 2024-08-17T17:43:36.560 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 416.236886637, 2024-08-17T17:43:36.561 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.16041424000000001, 2024-08-17T17:43:36.561 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:36.561 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:36.561 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:43:36.561 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:43:36.561 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:36.561 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 199681 2024-08-17T17:43:36.561 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.562 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:36.562 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.562 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:36.562 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.286985+0000", 2024-08-17T17:43:36.562 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.562 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.562 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.562 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:36.562 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.286985+0000", 2024-08-17T17:43:36.562 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.562 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.562 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.563 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:36.563 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.286983+0000", 2024-08-17T17:43:36.563 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:43:36.563 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.563 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.563 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:36.563 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.286990+0000", 2024-08-17T17:43:36.563 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.55e-06 2024-08-17T17:43:36.563 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.563 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.563 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:36.563 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.287003+0000", 2024-08-17T17:43:36.564 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2738999999999999e-05 2024-08-17T17:43:36.564 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.564 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.564 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:36.564 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.287009+0000", 2024-08-17T17:43:36.564 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1720000000000004e-06 2024-08-17T17:43:36.564 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.564 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.564 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:36.564 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.287055+0000", 2024-08-17T17:43:36.564 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5735000000000002e-05 2024-08-17T17:43:36.564 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.564 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.564 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:36.565 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.287190+0000", 2024-08-17T17:43:36.565 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013504200000000001 2024-08-17T17:43:36.565 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.565 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.565 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:43:36.565 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.287326+0000", 2024-08-17T17:43:36.565 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013652200000000001 2024-08-17T17:43:36.565 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.565 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.565 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:36.565 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.292091+0000", 2024-08-17T17:43:36.565 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0047653069999999999 2024-08-17T17:43:36.566 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.566 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.566 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:43:36.566 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.292147+0000", 2024-08-17T17:43:36.566 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5853999999999998e-05 2024-08-17T17:43:36.566 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.566 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.566 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:36.566 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.446392+0000", 2024-08-17T17:43:36.566 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.15424431599999999 2024-08-17T17:43:36.566 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.566 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.566 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:36.567 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.446444+0000", 2024-08-17T17:43:36.567 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1971000000000001e-05 2024-08-17T17:43:36.567 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.567 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.567 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:36.567 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.447399+0000", 2024-08-17T17:43:36.567 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00095564200000000004 2024-08-17T17:43:36.567 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.567 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:36.567 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.567 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.567 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.567 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.46025.0:199680 8.a e295/269 8:5c51b8a8:::benchmark_data_smithi003_229103_object92205:head v 295'12415, mlcod=265'4)", 2024-08-17T17:43:36.567 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:40.289342+0000", 2024-08-17T17:43:36.568 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 416.23452916399998, 2024-08-17T17:43:36.568 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.14536207700000001, 2024-08-17T17:43:36.568 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:36.568 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:36.568 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:36.568 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.568 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:36.568 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289342+0000", 2024-08-17T17:43:36.568 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.568 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.568 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.568 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:36.568 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289342+0000", 2024-08-17T17:43:36.568 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.569 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.569 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.569 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:36.569 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289342+0000", 2024-08-17T17:43:36.569 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:43:36.569 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.569 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.569 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:36.569 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289348+0000", 2024-08-17T17:43:36.569 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.246e-06 2024-08-17T17:43:36.569 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.569 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.569 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:36.569 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289359+0000", 2024-08-17T17:43:36.570 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0954e-05 2024-08-17T17:43:36.570 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.570 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.570 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:36.570 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289365+0000", 2024-08-17T17:43:36.570 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7740000000000001e-06 2024-08-17T17:43:36.570 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.570 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.570 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:36.570 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289407+0000", 2024-08-17T17:43:36.570 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2178000000000002e-05 2024-08-17T17:43:36.570 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.570 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.570 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:36.571 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289420+0000", 2024-08-17T17:43:36.571 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3473e-05 2024-08-17T17:43:36.571 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.571 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.571 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:36.571 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.434661+0000", 2024-08-17T17:43:36.571 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.14524047200000001 2024-08-17T17:43:36.571 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.571 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.571 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:36.571 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.434704+0000", 2024-08-17T17:43:36.571 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3822999999999997e-05 2024-08-17T17:43:36.571 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.571 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:36.572 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.572 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.572 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.572 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.46025.0:199679 8.0 e295/269 8:013198ab:::benchmark_data_smithi003_229103_object92204:head v 295'12429, mlcod=0'0)", 2024-08-17T17:43:36.572 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:40.289543+0000", 2024-08-17T17:43:36.572 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 416.23432868499998, 2024-08-17T17:43:36.572 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.14077357500000001, 2024-08-17T17:43:36.572 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:36.572 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:36.572 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:36.572 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.572 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:36.572 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289543+0000", 2024-08-17T17:43:36.572 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.573 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.573 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.573 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:36.573 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289543+0000", 2024-08-17T17:43:36.573 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.573 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.573 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.573 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:36.573 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289542+0000", 2024-08-17T17:43:36.573 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:43:36.573 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.573 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.573 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:36.574 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289548+0000", 2024-08-17T17:43:36.574 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2700000000000004e-06 2024-08-17T17:43:36.574 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.574 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.574 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:36.574 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289556+0000", 2024-08-17T17:43:36.574 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.4440000000000008e-06 2024-08-17T17:43:36.574 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.574 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.574 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:36.574 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289560+0000", 2024-08-17T17:43:36.574 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.63e-06 2024-08-17T17:43:36.574 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.575 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.575 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:36.575 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289821+0000", 2024-08-17T17:43:36.575 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00026115399999999999 2024-08-17T17:43:36.575 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.575 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.575 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:36.575 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289839+0000", 2024-08-17T17:43:36.575 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.8009999999999999e-05 2024-08-17T17:43:36.575 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.575 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.575 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:36.575 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.430267+0000", 2024-08-17T17:43:36.575 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.14042780299999999 2024-08-17T17:43:36.576 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.576 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.576 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:36.576 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.430316+0000", 2024-08-17T17:43:36.576 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9873999999999997e-05 2024-08-17T17:43:36.576 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.576 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:36.576 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.576 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.576 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.576 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.46025.0:199682 8.2 e295/262 8:459cfaea:::benchmark_data_smithi003_229103_object92207:head v 295'12477, mlcod=295'12475)", 2024-08-17T17:43:36.576 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:40.290267+0000", 2024-08-17T17:43:36.576 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 416.23360483499999, 2024-08-17T17:43:36.576 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.14446336700000001, 2024-08-17T17:43:36.577 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:36.577 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:36.577 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:36.577 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.577 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:36.577 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.290267+0000", 2024-08-17T17:43:36.577 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.577 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.577 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.577 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:36.577 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.290267+0000", 2024-08-17T17:43:36.577 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.577 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.578 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.578 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:36.578 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.290266+0000", 2024-08-17T17:43:36.578 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:43:36.578 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.578 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.578 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:36.578 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.290270+0000", 2024-08-17T17:43:36.578 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0090000000000001e-06 2024-08-17T17:43:36.578 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.578 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.578 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:36.578 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.290277+0000", 2024-08-17T17:43:36.579 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.2060000000000001e-06 2024-08-17T17:43:36.579 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.579 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.579 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:36.579 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.290281+0000", 2024-08-17T17:43:36.579 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3330000000000001e-06 2024-08-17T17:43:36.579 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.579 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.579 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:36.579 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.290315+0000", 2024-08-17T17:43:36.579 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4536000000000003e-05 2024-08-17T17:43:36.579 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.579 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.580 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:36.580 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.290332+0000", 2024-08-17T17:43:36.580 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7133e-05 2024-08-17T17:43:36.580 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.580 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.580 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:36.580 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.434709+0000", 2024-08-17T17:43:36.580 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.144376162 2024-08-17T17:43:36.580 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.580 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.580 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:36.580 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.434730+0000", 2024-08-17T17:43:36.580 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1537000000000001e-05 2024-08-17T17:43:36.580 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.581 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:36.581 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.581 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.581 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.581 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.46025.0:199683 8.0 e295/269 8:0617af90:::benchmark_data_smithi003_229103_object92208:head v 295'12430, mlcod=0'0)", 2024-08-17T17:43:36.581 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:40.301053+0000", 2024-08-17T17:43:36.581 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 416.222818733, 2024-08-17T17:43:36.581 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.138813256, 2024-08-17T17:43:36.581 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:36.581 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:36.581 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:36.581 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.581 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:36.581 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.301053+0000", 2024-08-17T17:43:36.582 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.582 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.582 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.582 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:36.582 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.301053+0000", 2024-08-17T17:43:36.582 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.582 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.582 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.582 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:36.582 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.301052+0000", 2024-08-17T17:43:36.582 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:43:36.582 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.582 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.582 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:36.583 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.301058+0000", 2024-08-17T17:43:36.583 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2530000000000001e-06 2024-08-17T17:43:36.583 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.583 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.583 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:36.583 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.430126+0000", 2024-08-17T17:43:36.583 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.12906774200000001 2024-08-17T17:43:36.583 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.583 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.583 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:36.583 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.430136+0000", 2024-08-17T17:43:36.583 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0127e-05 2024-08-17T17:43:36.583 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.583 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.584 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:36.584 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435393+0000", 2024-08-17T17:43:36.584 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0052568839999999999 2024-08-17T17:43:36.584 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.584 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.584 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:36.584 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435417+0000", 2024-08-17T17:43:36.584 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3934000000000002e-05 2024-08-17T17:43:36.584 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.584 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.584 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:36.584 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.439826+0000", 2024-08-17T17:43:36.584 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0044093489999999999 2024-08-17T17:43:36.584 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.584 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.584 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:36.585 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.439866+0000", 2024-08-17T17:43:36.585 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9737999999999998e-05 2024-08-17T17:43:36.585 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.585 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:36.585 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.585 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.585 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.585 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:199689 8.3 8:c0375079:::benchmark_data_smithi003_229103_object92214:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e295)", 2024-08-17T17:43:36.585 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:40.314370+0000", 2024-08-17T17:43:36.585 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 416.20950187099999, 2024-08-17T17:43:36.585 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.14278712099999999, 2024-08-17T17:43:36.585 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:36.585 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:36.585 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:43:36.585 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:43:36.585 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:36.586 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 199689 2024-08-17T17:43:36.586 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.586 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:36.586 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.586 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:36.586 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.314370+0000", 2024-08-17T17:43:36.586 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.586 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.586 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.586 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:36.586 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.314370+0000", 2024-08-17T17:43:36.586 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.586 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.586 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.587 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:36.587 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.314368+0000", 2024-08-17T17:43:36.587 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:43:36.587 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.587 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.587 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:36.587 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.314374+0000", 2024-08-17T17:43:36.587 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8309999999999997e-06 2024-08-17T17:43:36.587 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.587 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.587 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:36.587 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.430112+0000", 2024-08-17T17:43:36.587 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.11573762 2024-08-17T17:43:36.588 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.588 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.588 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:36.588 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.430129+0000", 2024-08-17T17:43:36.588 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6838e-05 2024-08-17T17:43:36.588 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.588 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.588 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:36.588 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.430236+0000", 2024-08-17T17:43:36.588 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010734300000000001 2024-08-17T17:43:36.588 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.588 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.588 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:36.589 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.434681+0000", 2024-08-17T17:43:36.589 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0044449579999999997 2024-08-17T17:43:36.589 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.589 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.589 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:43:36.589 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435137+0000", 2024-08-17T17:43:36.589 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000456377 2024-08-17T17:43:36.589 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.589 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.589 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:43:36.589 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.446355+0000", 2024-08-17T17:43:36.589 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011217789000000001 2024-08-17T17:43:36.589 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.590 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.590 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:36.590 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.447437+0000", 2024-08-17T17:43:36.590 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0010820350000000001 2024-08-17T17:43:36.590 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.590 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.590 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:36.590 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.455940+0000", 2024-08-17T17:43:36.590 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0085026900000000002 2024-08-17T17:43:36.590 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.590 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.590 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:36.590 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.455997+0000", 2024-08-17T17:43:36.590 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7160999999999997e-05 2024-08-17T17:43:36.591 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.591 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.591 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:36.591 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.457157+0000", 2024-08-17T17:43:36.591 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0011597249999999999 2024-08-17T17:43:36.591 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.591 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:36.591 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.591 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.591 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.591 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:209392 8.3 8:c1f63b84:::benchmark_data_smithi003_229103_object101917:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-08-17T17:43:36.591 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:54.274481+0000", 2024-08-17T17:43:36.591 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 402.24939072000001, 2024-08-17T17:43:36.592 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.090519438999999993, 2024-08-17T17:43:36.592 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:36.592 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:36.592 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:43:36.592 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:43:36.592 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:36.592 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 209392 2024-08-17T17:43:36.592 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.592 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:36.592 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.592 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:36.592 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:54.274481+0000", 2024-08-17T17:43:36.592 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.592 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.593 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.593 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:36.593 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:54.274481+0000", 2024-08-17T17:43:36.593 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.593 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.593 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.593 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:36.593 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:54.274480+0000", 2024-08-17T17:43:36.593 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:43:36.593 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.593 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.593 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:36.593 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:54.274486+0000", 2024-08-17T17:43:36.593 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0560000000000001e-06 2024-08-17T17:43:36.593 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.594 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.594 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:36.594 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:54.274499+0000", 2024-08-17T17:43:36.594 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3363999999999999e-05 2024-08-17T17:43:36.594 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.594 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.594 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:36.594 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:54.274505+0000", 2024-08-17T17:43:36.594 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8699999999999997e-06 2024-08-17T17:43:36.594 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.594 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.594 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:36.594 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:54.277817+0000", 2024-08-17T17:43:36.595 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0033122910000000002 2024-08-17T17:43:36.595 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.595 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.595 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:36.595 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:54.277982+0000", 2024-08-17T17:43:36.595 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000164714 2024-08-17T17:43:36.595 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.595 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.595 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:43:36.595 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:54.278142+0000", 2024-08-17T17:43:36.595 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016009800000000001 2024-08-17T17:43:36.595 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.595 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.595 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:43:36.596 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:54.282003+0000", 2024-08-17T17:43:36.596 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0038607509999999999 2024-08-17T17:43:36.596 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.596 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.596 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:36.596 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:54.283405+0000", 2024-08-17T17:43:36.596 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0014020009999999999 2024-08-17T17:43:36.596 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.596 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.596 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:36.596 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:54.364009+0000", 2024-08-17T17:43:36.596 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.080604008000000005 2024-08-17T17:43:36.596 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.596 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.597 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:36.597 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:54.364046+0000", 2024-08-17T17:43:36.597 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6763e-05 2024-08-17T17:43:36.597 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.597 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.597 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:36.597 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:54.365000+0000", 2024-08-17T17:43:36.597 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00095474399999999999 2024-08-17T17:43:36.597 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.597 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:36.597 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.597 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.597 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.597 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:211758 8.3 8:c8ce4a73:::benchmark_data_smithi003_229103_object104283:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-08-17T17:43:36.598 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:57.957839+0000", 2024-08-17T17:43:36.598 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 398.56603285900002, 2024-08-17T17:43:36.598 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.094953488000000003, 2024-08-17T17:43:36.598 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:36.598 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:36.598 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:43:36.598 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:43:36.598 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:43:36.598 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 211758 2024-08-17T17:43:36.598 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.598 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:36.598 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.598 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:36.598 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:57.957839+0000", 2024-08-17T17:43:36.598 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.599 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.599 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.599 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:36.599 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:57.957839+0000", 2024-08-17T17:43:36.599 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.599 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.599 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.599 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:36.599 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:57.957837+0000", 2024-08-17T17:43:36.599 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:43:36.599 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.599 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.599 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:36.600 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:57.957844+0000", 2024-08-17T17:43:36.600 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7400000000000004e-06 2024-08-17T17:43:36.600 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.600 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.600 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:36.600 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:57.957861+0000", 2024-08-17T17:43:36.600 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6232999999999999e-05 2024-08-17T17:43:36.600 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.600 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.600 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:36.600 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:57.957867+0000", 2024-08-17T17:43:36.600 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.8199999999999999e-06 2024-08-17T17:43:36.600 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.600 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.600 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:36.600 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:57.966596+0000", 2024-08-17T17:43:36.601 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0087287130000000008 2024-08-17T17:43:36.601 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.601 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.601 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:36.601 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:57.966724+0000", 2024-08-17T17:43:36.601 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0001282 2024-08-17T17:43:36.601 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.601 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.601 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,6", 2024-08-17T17:43:36.601 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:57.966852+0000", 2024-08-17T17:43:36.601 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012788599999999999 2024-08-17T17:43:36.601 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.601 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.601 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:43:36.602 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:57.970730+0000", 2024-08-17T17:43:36.602 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0038780899999999998 2024-08-17T17:43:36.602 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.602 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.602 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:36.602 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:57.971988+0000", 2024-08-17T17:43:36.602 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0012575469999999999 2024-08-17T17:43:36.602 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.602 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.602 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:43:36.602 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:58.051745+0000", 2024-08-17T17:43:36.602 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.079757148999999999 2024-08-17T17:43:36.602 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.603 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.603 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:36.603 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:58.051796+0000", 2024-08-17T17:43:36.603 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1316000000000001e-05 2024-08-17T17:43:36.603 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.603 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.603 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:36.603 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:58.052792+0000", 2024-08-17T17:43:36.603 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00099588699999999999 2024-08-17T17:43:36.603 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.603 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:36.603 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.603 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.603 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.603 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.36354.0:138793 9.a e333/311 9:565da231:::benchmark_data_smithi003_261051_object22908:head v 333'8662, mlcod=333'8658)", 2024-08-17T17:43:36.603 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:39:52.123715+0000", 2024-08-17T17:43:36.604 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 224.40015696, 2024-08-17T17:43:36.604 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.183263708, 2024-08-17T17:43:36.604 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:36.604 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:36.604 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:36.604 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.604 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:36.604 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.123715+0000", 2024-08-17T17:43:36.604 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.604 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.604 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.604 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:36.604 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.123715+0000", 2024-08-17T17:43:36.604 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.604 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.605 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.605 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:36.605 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.123714+0000", 2024-08-17T17:43:36.605 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:43:36.605 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.605 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.605 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:36.605 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.123724+0000", 2024-08-17T17:43:36.605 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0536999999999999e-05 2024-08-17T17:43:36.605 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.605 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.605 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:36.606 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.123732+0000", 2024-08-17T17:43:36.606 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7819999999999994e-06 2024-08-17T17:43:36.606 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.606 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.606 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:36.606 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.123736+0000", 2024-08-17T17:43:36.606 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.523e-06 2024-08-17T17:43:36.606 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.606 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.606 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:36.606 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.123765+0000", 2024-08-17T17:43:36.606 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9468e-05 2024-08-17T17:43:36.606 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.607 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.607 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:36.607 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.123784+0000", 2024-08-17T17:43:36.607 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9131e-05 2024-08-17T17:43:36.607 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.607 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.607 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:36.607 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.306941+0000", 2024-08-17T17:43:36.607 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.18315631499999999 2024-08-17T17:43:36.607 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.607 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.607 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:36.607 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.306978+0000", 2024-08-17T17:43:36.607 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.7796000000000002e-05 2024-08-17T17:43:36.608 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.608 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:36.608 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.608 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.608 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.608 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.36354.0:138791 9.d e333/311 9:b1398701:::benchmark_data_smithi003_261051_object22906:head v 333'8813, mlcod=333'8810)", 2024-08-17T17:43:36.608 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:39:52.123745+0000", 2024-08-17T17:43:36.608 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 224.40012607200001, 2024-08-17T17:43:36.608 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.191437196, 2024-08-17T17:43:36.608 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:36.608 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:36.608 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:36.608 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.608 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:36.608 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.123745+0000", 2024-08-17T17:43:36.609 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.609 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.609 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.609 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:36.609 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.123745+0000", 2024-08-17T17:43:36.609 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.609 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.609 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.609 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:36.609 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.123745+0000", 2024-08-17T17:43:36.609 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:43:36.609 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.609 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.609 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:36.610 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.123749+0000", 2024-08-17T17:43:36.610 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6940000000000001e-06 2024-08-17T17:43:36.610 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.610 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.610 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:36.610 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.123754+0000", 2024-08-17T17:43:36.610 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1959999999999999e-06 2024-08-17T17:43:36.610 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.610 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.610 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:36.610 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.123758+0000", 2024-08-17T17:43:36.610 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.7249999999999999e-06 2024-08-17T17:43:36.610 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.611 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.611 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:36.611 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.123796+0000", 2024-08-17T17:43:36.611 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8476000000000002e-05 2024-08-17T17:43:36.611 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.611 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.611 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:36.611 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.123812+0000", 2024-08-17T17:43:36.611 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5977999999999998e-05 2024-08-17T17:43:36.611 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.611 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.611 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:36.611 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.315158+0000", 2024-08-17T17:43:36.611 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.19134570200000001 2024-08-17T17:43:36.611 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.612 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.612 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:36.612 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.315183+0000", 2024-08-17T17:43:36.612 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4678999999999999e-05 2024-08-17T17:43:36.612 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.612 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:36.612 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.612 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.612 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.612 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.36354.0:138802 9.8 e333/311 9:1c1d2ede:::benchmark_data_smithi003_261051_object22917:head v 333'8604, mlcod=333'8600)", 2024-08-17T17:43:36.612 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:39:52.126240+0000", 2024-08-17T17:43:36.612 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 224.39763152399999, 2024-08-17T17:43:36.612 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.18037687999999999, 2024-08-17T17:43:36.613 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:36.613 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:36.613 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:36.613 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.613 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:36.613 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.126240+0000", 2024-08-17T17:43:36.613 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.613 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.613 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.613 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:36.613 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.126240+0000", 2024-08-17T17:43:36.613 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.613 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.613 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.614 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:36.614 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.126239+0000", 2024-08-17T17:43:36.614 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:43:36.614 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.614 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.614 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:36.614 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.126245+0000", 2024-08-17T17:43:36.614 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2109999999999997e-06 2024-08-17T17:43:36.614 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.614 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.614 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:36.614 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.300989+0000", 2024-08-17T17:43:36.614 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.174743917 2024-08-17T17:43:36.615 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.615 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.615 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:36.615 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.300999+0000", 2024-08-17T17:43:36.615 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0621e-05 2024-08-17T17:43:36.615 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.615 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.615 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:36.615 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.301109+0000", 2024-08-17T17:43:36.615 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011025 2024-08-17T17:43:36.615 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.615 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.615 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:36.615 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.301133+0000", 2024-08-17T17:43:36.616 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.3278999999999999e-05 2024-08-17T17:43:36.616 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.616 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.616 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:36.616 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.306543+0000", 2024-08-17T17:43:36.616 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0054107210000000003 2024-08-17T17:43:36.616 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.616 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.616 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:36.616 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.306617+0000", 2024-08-17T17:43:36.616 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3542999999999997e-05 2024-08-17T17:43:36.616 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.616 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:36.616 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.617 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.617 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.617 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.36354.0:138804 9.5 e333/311 9:aec0763c:::benchmark_data_smithi003_261051_object22919:head v 333'8669, mlcod=333'8667)", 2024-08-17T17:43:36.617 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:39:52.131393+0000", 2024-08-17T17:43:36.617 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 224.39247820899999, 2024-08-17T17:43:36.617 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.18381320500000001, 2024-08-17T17:43:36.617 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:36.617 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:36.617 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:36.617 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.617 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:36.617 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.131393+0000", 2024-08-17T17:43:36.617 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.617 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.617 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.618 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:36.618 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.131393+0000", 2024-08-17T17:43:36.618 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.618 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.618 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.618 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:36.618 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.131393+0000", 2024-08-17T17:43:36.618 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:43:36.618 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.618 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.618 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:36.618 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.131399+0000", 2024-08-17T17:43:36.618 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.8299999999999998e-06 2024-08-17T17:43:36.618 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.619 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.619 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:36.619 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.300988+0000", 2024-08-17T17:43:36.619 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.16958846599999999 2024-08-17T17:43:36.619 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.619 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.619 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:36.619 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.300998+0000", 2024-08-17T17:43:36.619 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0538e-05 2024-08-17T17:43:36.619 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.619 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.619 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:36.619 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.301136+0000", 2024-08-17T17:43:36.619 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013740700000000001 2024-08-17T17:43:36.620 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.620 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.620 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:36.620 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.301155+0000", 2024-08-17T17:43:36.620 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9199999999999999e-05 2024-08-17T17:43:36.620 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.620 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.620 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:36.620 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.315187+0000", 2024-08-17T17:43:36.620 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.014031609 2024-08-17T17:43:36.620 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.620 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.620 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:36.620 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:39:52.315207+0000", 2024-08-17T17:43:36.621 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9979e-05 2024-08-17T17:43:36.621 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.621 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:36.621 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.621 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.621 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.621 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.36612.0:47553 10.a e370/357 10:5747a865:::benchmark_data_smithi003_292749_object47552:head v 370'3000, mlcod=370'2998)", 2024-08-17T17:43:36.621 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:14.066993+0000", 2024-08-17T17:43:36.621 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 22.456878631999999, 2024-08-17T17:43:36.621 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.095881411, 2024-08-17T17:43:36.621 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:36.621 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:36.621 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:36.621 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.621 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:36.622 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.066993+0000", 2024-08-17T17:43:36.622 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.622 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.622 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.622 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:36.622 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.066993+0000", 2024-08-17T17:43:36.622 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.622 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.622 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.622 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:36.622 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.066992+0000", 2024-08-17T17:43:36.622 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:43:36.622 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.622 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.623 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:36.623 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.067016+0000", 2024-08-17T17:43:36.623 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.4278999999999999e-05 2024-08-17T17:43:36.623 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.623 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.623 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:36.623 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.067028+0000", 2024-08-17T17:43:36.623 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1539000000000001e-05 2024-08-17T17:43:36.623 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.623 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.623 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:36.623 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.067034+0000", 2024-08-17T17:43:36.623 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9359999999999996e-06 2024-08-17T17:43:36.623 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.623 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.624 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:36.624 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.067065+0000", 2024-08-17T17:43:36.624 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1251000000000003e-05 2024-08-17T17:43:36.624 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.624 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.624 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:36.624 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.067084+0000", 2024-08-17T17:43:36.624 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9018e-05 2024-08-17T17:43:36.624 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.624 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.624 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:36.624 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.162757+0000", 2024-08-17T17:43:36.624 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.095672835999999997 2024-08-17T17:43:36.624 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.625 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.625 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:36.625 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.162874+0000", 2024-08-17T17:43:36.625 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000117438 2024-08-17T17:43:36.625 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.625 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:36.625 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.625 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.625 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.625 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.36612.0:47560 10.a e370/357 10:550c7c8d:::benchmark_data_smithi003_292749_object47559:head v 370'3001, mlcod=370'2998)", 2024-08-17T17:43:36.625 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:14.072693+0000", 2024-08-17T17:43:36.625 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 22.451178967000001, 2024-08-17T17:43:36.625 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.090282564999999995, 2024-08-17T17:43:36.625 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:36.626 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:36.626 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:36.626 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.626 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:36.626 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.072693+0000", 2024-08-17T17:43:36.626 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.626 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.626 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.626 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:36.626 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.072693+0000", 2024-08-17T17:43:36.626 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.626 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.626 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.626 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:36.627 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.072692+0000", 2024-08-17T17:43:36.627 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:43:36.627 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.627 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.627 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:36.627 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.072874+0000", 2024-08-17T17:43:36.627 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00018266100000000001 2024-08-17T17:43:36.627 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.627 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.627 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:36.627 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.156798+0000", 2024-08-17T17:43:36.627 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.083923442000000001 2024-08-17T17:43:36.627 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.627 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.628 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:36.628 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.156804+0000", 2024-08-17T17:43:36.628 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3199999999999996e-06 2024-08-17T17:43:36.628 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.628 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.628 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:36.628 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.157333+0000", 2024-08-17T17:43:36.628 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00052913799999999996 2024-08-17T17:43:36.628 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.628 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.628 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:36.628 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.157351+0000", 2024-08-17T17:43:36.628 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7835000000000001e-05 2024-08-17T17:43:36.629 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.629 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.629 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:36.629 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.162880+0000", 2024-08-17T17:43:36.629 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.005528542 2024-08-17T17:43:36.629 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.629 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.629 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:36.629 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.162975+0000", 2024-08-17T17:43:36.629 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5628e-05 2024-08-17T17:43:36.629 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.629 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:36.629 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.629 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.630 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.630 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.36612.0:76540 10.6 e373/357 10:64476808:::benchmark_data_smithi003_292749_object76539:head v 373'4759, mlcod=373'4755)", 2024-08-17T17:43:36.630 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:35.986831+0000", 2024-08-17T17:43:36.630 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.53704030599999997, 2024-08-17T17:43:36.630 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.018928904, 2024-08-17T17:43:36.630 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:36.630 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:43:36.630 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:36.630 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.630 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:36.630 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:35.986831+0000", 2024-08-17T17:43:36.630 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.630 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.630 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.631 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:36.631 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:35.986831+0000", 2024-08-17T17:43:36.631 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:36.631 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.631 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.631 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:36.631 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:35.986830+0000", 2024-08-17T17:43:36.631 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:43:36.631 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.631 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.631 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:36.631 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:36.003521+0000", 2024-08-17T17:43:36.631 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016690607999999999 2024-08-17T17:43:36.632 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.632 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.632 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:36.632 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:36.003537+0000", 2024-08-17T17:43:36.632 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5899000000000001e-05 2024-08-17T17:43:36.632 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.632 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.632 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:36.632 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:36.003542+0000", 2024-08-17T17:43:36.632 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.366e-06 2024-08-17T17:43:36.632 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.632 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.632 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:36.632 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:36.003584+0000", 2024-08-17T17:43:36.632 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1897000000000002e-05 2024-08-17T17:43:36.633 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.633 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.633 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:36.633 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:36.003603+0000", 2024-08-17T17:43:36.633 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9015999999999999e-05 2024-08-17T17:43:36.633 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.633 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.633 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:43:36.633 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:36.005730+0000", 2024-08-17T17:43:36.633 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0021270429999999999 2024-08-17T17:43:36.633 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:36.633 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:36.633 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:43:36.633 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:36.005760+0000", 2024-08-17T17:43:36.634 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9932000000000001e-05 2024-08-17T17:43:36.634 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.634 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:36.634 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.634 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:36.634 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:36.634 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:43:36.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:36.634 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 47 16 77172 77156 102.574 70.875 0.0102408 0.00974034 2024-08-17T17:43:36.735 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:36.773 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:36.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:36.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:36.875 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:36.910 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:36.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:36.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:37.011 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:37.018 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-08-17T17:43:37.045 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:37.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:37.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:37.147 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:37.185 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:37.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:37.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:37.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:37 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3466575591' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:43:37.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:37 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:43:37.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:37 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3466575591' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:43:37.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:37 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:43:37.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:37.330 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:37.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:37.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:37.432 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:37.465 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:37 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3466575591' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:43:37.465 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:37 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:43:37.475 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:37.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:37.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:37.509 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-17T17:43:37.546 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 48 16 78452 78436 102.104 80 0.00971064 0.00978909 2024-08-17T17:43:37.577 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:37.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:37.638 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:37.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:37.739 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:37.782 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:37.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:37.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:37.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:37.926 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:37.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:37.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:38.022 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:43:38.028 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:38.068 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:38.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:38.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:38.170 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:38.215 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:38.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:38.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:38.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:38 smithi003 ceph-mon[100615]: pgmap v1215: 119 pgs: 119 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-08-17T17:43:38.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:38 smithi003 ceph-mon[100615]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:43:38.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:38 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:43:38.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:38 smithi003 ceph-mon[100615]: osdmap e374: 8 total, 8 up, 8 in 2024-08-17T17:43:38.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:38 smithi003 ceph-mon[103344]: pgmap v1215: 119 pgs: 119 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-08-17T17:43:38.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:38 smithi003 ceph-mon[103344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:43:38.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:38 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:43:38.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:38 smithi003 ceph-mon[103344]: osdmap e374: 8 total, 8 up, 8 in 2024-08-17T17:43:38.317 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:38.345 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:38 smithi053 ceph-mon[89963]: pgmap v1215: 119 pgs: 119 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-08-17T17:43:38.345 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:38 smithi053 ceph-mon[89963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:43:38.345 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:38 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:43:38.345 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:38 smithi053 ceph-mon[89963]: osdmap e374: 8 total, 8 up, 8 in 2024-08-17T17:43:38.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:38.354 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:38.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:38.411 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:43:38.455 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:38.497 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:38.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:38.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:38.547 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 49 16 79672 79656 101.575 76.25 0.00372491 0.00983961 2024-08-17T17:43:38.599 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:38.621 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:38.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:38.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:38.723 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:38.775 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:38.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:38.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:38.877 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:38.913 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:38.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:38.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:39.015 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:39.043 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:39.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:39.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:39.145 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:39.187 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:39.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:39.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:39.289 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:39.328 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:39.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:39.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:39.430 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:39.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:43:39.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:43:39.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:39 smithi053 ceph-mon[89963]: pgmap v1217: 119 pgs: 119 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 57 MiB/s wr, 915 op/s 2024-08-17T17:43:39.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:39.470 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:39.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:39.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:43:39.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:43:39.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:39 smithi003 ceph-mon[103344]: pgmap v1217: 119 pgs: 119 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 57 MiB/s wr, 915 op/s 2024-08-17T17:43:39.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:43:39.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:43:39.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:39 smithi003 ceph-mon[100615]: pgmap v1217: 119 pgs: 119 active+clean; 4.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 57 MiB/s wr, 915 op/s 2024-08-17T17:43:39.547 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 50 16 80780 80764 100.929 69.25 0.00590452 0.0099022 2024-08-17T17:43:39.571 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:39.613 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:39.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:39.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:39.715 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:39.745 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:39.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:39.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:39.847 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:39.887 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:39.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:39.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:39.897 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:43:39.932 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:43:39.932 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:43:39.959 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:43:39.959 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:43:39.976 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:43:39.976 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:43:39.988 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:39.992 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:43:39.993 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:43:40.020 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:40.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:40.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:40.032 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:43:40.033 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:43:40.063 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:43:40.064 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:43:40.096 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:43:40.096 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:43:40.123 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:40.131 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:43:40.131 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:43:40.180 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:40.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:40.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:40.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:40 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2169183457' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:43:40.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:40 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2169183457' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:43:40.282 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:40.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:40.324 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:40.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:40.425 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:40.468 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:43:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:43:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:43:40.468 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:40.468 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:40 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2169183457' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:43:40.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:40.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:40.547 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 51 16 81800 81784 100.2 63.75 0.0177716 0.00997482 2024-08-17T17:43:40.570 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:40.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:40.609 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:40.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:40.710 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:40.739 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:40.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:40.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:40.841 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:40.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:40.881 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:40.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:40.982 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:41.022 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:41.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:41.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:41.124 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:41.146 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:41.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:41.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:41.227 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:41 smithi053 ceph-mon[89963]: pgmap v1218: 119 pgs: 119 active+clean; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-08-17T17:43:41.248 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:41.295 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:41.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:41.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:41.397 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:41.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:41.433 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:41.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:41.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:41 smithi003 ceph-mon[103344]: pgmap v1218: 119 pgs: 119 active+clean; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-08-17T17:43:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:41 smithi003 ceph-mon[100615]: pgmap v1218: 119 pgs: 119 active+clean; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-08-17T17:43:41.534 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:41.547 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 52 16 82917 82901 99.6155 69.8125 0.0185951 0.0100337 2024-08-17T17:43:41.585 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:41.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:41.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:41.674 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:43:41.674 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:43:41.687 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:41.724 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:41.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:41.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:41.825 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:41.862 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:41.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:41.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:41.964 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:42.003 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:42.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:42.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:42.105 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:42.142 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:42.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:42.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:42.244 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:42.277 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:42.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:42.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:42.379 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:42.416 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-17T17:43:42.459 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:42.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:42.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:42.547 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 53 16 84071 84055 99.0967 72.125 0.00966448 0.0100855 2024-08-17T17:43:42.562 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:42.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:42.605 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:42.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:42.671 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:43:42.706 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:42.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:42.751 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:42.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:42.822 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:43:42.852 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:42.898 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:42.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:42.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:43.000 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:43.030 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:43.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:43.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:43.132 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:43.175 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:43.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:43.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:43.276 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:43.313 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:43.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:43.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:43.415 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:43.437 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.3 to 1 (10000) 2024-08-17T17:43:43.458 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:43.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:43.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:43.547 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 54 16 85171 85155 98.5347 68.75 0.00901103 0.0101434 2024-08-17T17:43:43.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:43.599 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:43.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:43.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:43.700 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:43.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:43 smithi053 ceph-mon[89963]: pgmap v1219: 119 pgs: 119 active+clean; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-08-17T17:43:43.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:43 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1422687112' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:43:43.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:43 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:43:43.738 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:43.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:43.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:43.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:43 smithi003 ceph-mon[103344]: pgmap v1219: 119 pgs: 119 active+clean; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-08-17T17:43:43.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:43 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1422687112' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:43:43.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:43 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:43:43.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:43 smithi003 ceph-mon[100615]: pgmap v1219: 119 pgs: 119 active+clean; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-08-17T17:43:43.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:43 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1422687112' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:43:43.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:43 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:43:43.840 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:43.881 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:43.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:43.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:43.983 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:44.025 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:44.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:44.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:44.094 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:43:44.127 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:44.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:44.169 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:44.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:44.231 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 5 1 2024-08-17T17:43:44.270 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:44.338 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:44.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:44.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:44.441 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:44.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:44.506 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:44.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:44.547 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 55 15 86311 86296 98.0397 71.3125 0.0171231 0.0101944 2024-08-17T17:43:44.607 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:44.623 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:44 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-17T17:43:44.623 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:44 smithi053 ceph-mon[89963]: osdmap e375: 8 total, 8 up, 8 in 2024-08-17T17:43:44.650 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:44.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:44.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:44.753 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:44.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:44 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-17T17:43:44.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:44 smithi003 ceph-mon[100615]: osdmap e375: 8 total, 8 up, 8 in 2024-08-17T17:43:44.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:44 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-17T17:43:44.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:44 smithi003 ceph-mon[103344]: osdmap e375: 8 total, 8 up, 8 in 2024-08-17T17:43:44.783 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:43:44.783 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:43:44.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:44.792 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:44.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:44.893 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:44.926 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:44.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:44.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:45.028 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:45.081 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:45.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:45.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:45.183 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:45.236 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:45.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:45.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:45.338 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:45.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:45.380 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:45.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:45.481 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:45.514 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:45.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:45.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:45.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:45 smithi003 ceph-mon[103344]: pgmap v1221: 119 pgs: 119 active+clean; 5.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-08-17T17:43:45.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:45 smithi003 ceph-mon[100615]: pgmap v1221: 119 pgs: 119 active+clean; 5.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-08-17T17:43:45.547 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 56 16 87431 87415 97.5379 69.9375 0.0103682 0.0102466 2024-08-17T17:43:45.550 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:45 smithi053 ceph-mon[89963]: pgmap v1221: 119 pgs: 119 active+clean; 5.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-08-17T17:43:45.592 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-17T17:43:45.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:45.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:45.660 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:45.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:45.761 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:45.805 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:45.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:45.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:45.877 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:43:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:43:45] "GET /metrics HTTP/1.1" 200 36569 "" "Prometheus/2.51.0" 2024-08-17T17:43:45.886 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:43:45.886 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:43:45.910 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:45.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:45.983 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:45.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:46.008 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:43:46.084 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:46.121 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:46.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:46.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:46.223 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:46.254 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:46.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:46.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:46.356 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:46.390 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:46.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:46.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:46.484 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-08-17T17:43:46.492 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:46.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:46.526 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:46.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:46.547 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 57 16 88521 88505 97.0218 68.125 0.0173162 0.0103011 2024-08-17T17:43:46.627 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:46.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:46.669 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:46.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:46.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:46 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1130387927' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:43:46.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:46 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:43:46.770 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:46.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:46 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1130387927' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:43:46.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:46 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:43:46.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:46 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1130387927' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:43:46.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:46 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:43:46.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:46.810 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:46.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:46.911 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:46.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:46.948 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:46.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:47.049 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:47.089 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:47.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:47.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:47.190 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:47.238 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:47.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:47.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:47.340 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:47.377 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:47.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:47.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:47.479 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:47.511 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:47.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:47.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:47.547 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 58 16 89728 89712 96.6496 75.4375 0.00754918 0.0103419 2024-08-17T17:43:47.613 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:47.646 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:47.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:47.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:47.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:47 smithi053 ceph-mon[89963]: pgmap v1222: 119 pgs: 119 active+clean; 5.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-08-17T17:43:47.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:47 smithi053 ceph-mon[89963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:43:47.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:47 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:43:47.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:47 smithi053 ceph-mon[89963]: osdmap e376: 8 total, 8 up, 8 in 2024-08-17T17:43:47.748 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:47.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:47 smithi003 ceph-mon[103344]: pgmap v1222: 119 pgs: 119 active+clean; 5.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-08-17T17:43:47.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:47 smithi003 ceph-mon[103344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:43:47.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:47 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:43:47.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:47 smithi003 ceph-mon[103344]: osdmap e376: 8 total, 8 up, 8 in 2024-08-17T17:43:47.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:47 smithi003 ceph-mon[100615]: pgmap v1222: 119 pgs: 119 active+clean; 5.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-08-17T17:43:47.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:47 smithi003 ceph-mon[100615]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:43:47.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:47 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:43:47.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:47 smithi003 ceph-mon[100615]: osdmap e376: 8 total, 8 up, 8 in 2024-08-17T17:43:47.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:47.784 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:47.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:47.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:47.905 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:43:47.905 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:43:47.923 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:47.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:47.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:48.025 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:48.057 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:48.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:48.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:48.159 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:48.200 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:48.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:48.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:48.302 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:48.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:48.337 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:48.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:48.438 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:48.472 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:48.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:48.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:48.548 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 59 16 90813 90797 96.1608 67.8125 0.0118098 0.010394 2024-08-17T17:43:48.574 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:48.607 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:48.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:48.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:48.709 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:48.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:48.749 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:48.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:48.850 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:48.885 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:48.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:48.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:48.987 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:49.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:49.028 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:49.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:49.114 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-17T17:43:49.130 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:49.185 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:49.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:49.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:49.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:49.325 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:49.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:49.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:49.427 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:49.466 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:49.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:49.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:49.548 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-08-17T17:43:49.548201+0000 min lat: 0.00107156 max lat: 2.04224 avg lat: 0.0104375 2024-08-17T17:43:49.548 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:43:49.548 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 60 16 91964 91948 95.757 71.9375 0.0117947 0.0104375 2024-08-17T17:43:49.567 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:49.576 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:43:49.611 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:49.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:49.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:49.713 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:49.755 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:49 smithi053 ceph-mon[89963]: pgmap v1224: 119 pgs: 119 active+clean; 5.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 46 MiB/s wr, 739 op/s 2024-08-17T17:43:49.758 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:49.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:49.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:49.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:49 smithi003 ceph-mon[103344]: pgmap v1224: 119 pgs: 119 active+clean; 5.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 46 MiB/s wr, 739 op/s 2024-08-17T17:43:49.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:49 smithi003 ceph-mon[100615]: pgmap v1224: 119 pgs: 119 active+clean; 5.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 46 MiB/s wr, 739 op/s 2024-08-17T17:43:49.859 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:49.901 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:49.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:49.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:50.004 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:50.020 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:43:50.020 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:43:50.020 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:43:50.020 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:43:50.020 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:43:50.021 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:43:50.021 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:43:50.021 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:43:50.021 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:43:50.021 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:43:50.042 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:43:50.042 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:43:50.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:50.050 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:50.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:50.076 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:43:50.077 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:43:50.108 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:43:50.108 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:43:50.151 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:50.189 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:50.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:50.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:50.291 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:50.332 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:50.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:50.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:50.434 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:50.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:43:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:43:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:43:50.472 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:50.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:50.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:50.548 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 61 16 93066 93050 95.3163 68.875 0.0156892 0.0104863 2024-08-17T17:43:50.574 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:50.614 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:50.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:50.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:50.716 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:50.744 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:50 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2976068167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:43:50.745 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:50 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2976068167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:43:50.752 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:50.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:50.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:50.760 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:50 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2976068167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:43:50.854 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:50.868 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:43:50.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:50.894 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:50.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:50.996 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:51.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:51.090 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:51.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:51.191 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:51.219 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:51.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:51.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:51.320 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:51.367 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:51.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:51.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:51.438 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:43:51.469 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:51.492 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.5 to 1 (10000) 2024-08-17T17:43:51.509 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:51.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:51.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:51.548 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 62 16 94039 94023 94.7591 60.8125 0.0115275 0.0105474 2024-08-17T17:43:51.554 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:51 smithi053 ceph-mon[89963]: pgmap v1225: 119 pgs: 119 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-08-17T17:43:51.555 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:51 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/108057563' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:43:51.555 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:51 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:43:51.611 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:51.643 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:51.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:51.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:51.693 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:43:51.693 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2024-08-17T17:43:51.693 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:51.693 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.36612.0:94168 10.8 e377/357 10:1a0b98eb:::benchmark_data_smithi003_292749_object94167:head v 377'5830, mlcod=377'5828)", 2024-08-17T17:43:51.693 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:51.688291+0000", 2024-08-17T17:43:51.693 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.003325724, 2024-08-17T17:43:51.693 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0033517429999999999, 2024-08-17T17:43:51.693 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:51.693 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2024-08-17T17:43:51.693 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:51.693 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:51.694 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:51.694 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.688291+0000", 2024-08-17T17:43:51.694 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:51.694 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:51.694 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:51.694 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:51.694 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.688291+0000", 2024-08-17T17:43:51.694 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:51.694 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:51.694 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:51.694 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:51.694 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.688290+0000", 2024-08-17T17:43:51.694 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:43:51.695 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:51.695 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:51.695 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:51.695 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.688321+0000", 2024-08-17T17:43:51.695 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1220999999999998e-05 2024-08-17T17:43:51.695 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:51.695 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:51.695 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:51.695 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.688335+0000", 2024-08-17T17:43:51.695 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3652000000000001e-05 2024-08-17T17:43:51.695 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:51.695 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:51.696 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:51.696 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.688341+0000", 2024-08-17T17:43:51.696 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2149999999999998e-06 2024-08-17T17:43:51.696 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:51.696 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:51.696 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:51.696 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.688381+0000", 2024-08-17T17:43:51.696 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9301999999999997e-05 2024-08-17T17:43:51.696 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:51.696 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:51.696 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:51.696 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.688400+0000", 2024-08-17T17:43:51.696 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9607999999999999e-05 2024-08-17T17:43:51.697 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:51.697 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:51.697 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:51.697 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:51.697 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:51.697 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:94170 10.9 10:93e701b9:::benchmark_data_smithi003_292749_object94169:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-08-17T17:43:51.697 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:51.688966+0000", 2024-08-17T17:43:51.697 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.0026509260000000001, 2024-08-17T17:43:51.697 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0027453830000000001, 2024-08-17T17:43:51.697 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:51.697 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:43:51.697 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:43:51.697 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:43:51.697 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:43:51.698 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 94170 2024-08-17T17:43:51.698 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:51.698 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:51.698 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:51.698 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:51.698 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.688966+0000", 2024-08-17T17:43:51.698 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:51.698 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:51.698 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:51.698 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:51.698 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.688966+0000", 2024-08-17T17:43:51.698 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:51.699 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:51.699 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:51.699 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:51.699 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.688964+0000", 2024-08-17T17:43:51.699 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:43:51.699 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:51.699 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:51.699 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:51.699 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.689012+0000", 2024-08-17T17:43:51.699 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7555999999999997e-05 2024-08-17T17:43:51.699 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:51.699 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:51.699 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:51.699 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.689023+0000", 2024-08-17T17:43:51.699 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1353e-05 2024-08-17T17:43:51.700 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:51.700 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:51.700 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:51.700 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.689027+0000", 2024-08-17T17:43:51.700 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1640000000000003e-06 2024-08-17T17:43:51.700 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:51.700 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:51.700 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:51.700 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.689069+0000", 2024-08-17T17:43:51.700 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2045000000000001e-05 2024-08-17T17:43:51.700 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:51.700 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:51.700 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:51.701 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.689169+0000", 2024-08-17T17:43:51.701 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.9874e-05 2024-08-17T17:43:51.701 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:51.701 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:51.701 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:43:51.701 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.689262+0000", 2024-08-17T17:43:51.701 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.2686e-05 2024-08-17T17:43:51.701 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:51.701 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:51.701 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:51.701 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:51.701 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:51.701 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.36612.0:94171 10.8 e377/357 10:17f3a47a:::benchmark_data_smithi003_292749_object94170:head v 377'5831, mlcod=377'5828)", 2024-08-17T17:43:51.701 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:51.690573+0000", 2024-08-17T17:43:51.702 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 0.001043806, 2024-08-17T17:43:51.702 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0012071219999999999, 2024-08-17T17:43:51.702 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:43:51.702 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2024-08-17T17:43:51.702 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:43:51.702 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:51.702 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:43:51.702 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.690573+0000", 2024-08-17T17:43:51.702 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:51.702 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:51.702 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:51.702 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:43:51.702 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.690573+0000", 2024-08-17T17:43:51.703 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:43:51.703 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:51.703 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:51.703 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:43:51.703 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.690572+0000", 2024-08-17T17:43:51.703 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:43:51.703 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:51.703 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:51.703 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:43:51.703 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.690635+0000", 2024-08-17T17:43:51.703 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3095999999999994e-05 2024-08-17T17:43:51.704 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:51.704 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:51.704 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:43:51.704 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.690647+0000", 2024-08-17T17:43:51.704 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1472e-05 2024-08-17T17:43:51.704 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:51.704 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:51.704 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:43:51.704 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.690651+0000", 2024-08-17T17:43:51.704 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5210000000000004e-06 2024-08-17T17:43:51.704 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:51.704 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:51.704 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:43:51.705 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.690901+0000", 2024-08-17T17:43:51.705 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00024945399999999997 2024-08-17T17:43:51.705 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:43:51.705 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:43:51.705 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:43:51.705 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.690926+0000", 2024-08-17T17:43:51.705 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5102000000000001e-05 2024-08-17T17:43:51.705 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:51.705 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:43:51.705 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:51.705 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:43:51.705 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-08-17T17:43:51.705 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 3 2024-08-17T17:43:51.705 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:43:51.745 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:51.771 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:51.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:51.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:51.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:51 smithi003 ceph-mon[100615]: pgmap v1225: 119 pgs: 119 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-08-17T17:43:51.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:51 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/108057563' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:43:51.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:51 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:43:51.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:51 smithi003 ceph-mon[103344]: pgmap v1225: 119 pgs: 119 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s 2024-08-17T17:43:51.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:51 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/108057563' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:43:51.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:51 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:43:51.873 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:51.897 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:51.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:51.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:51.999 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:52.021 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:52.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:52.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:52.123 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:52.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:52.146 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:52.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:52.247 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:52.269 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:52.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:52.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:52.371 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:52.394 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:52.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:52.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:52.496 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:52.523 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:52.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:52.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:52.548 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 63 16 94262 94246 93.4764 13.9375 0.00237507 0.01054 2024-08-17T17:43:52.624 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:52.647 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:52.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:52.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:52.749 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:52.772 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:52.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:52 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-17T17:43:52.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:52 smithi003 ceph-mon[103344]: osdmap e377: 8 total, 8 up, 8 in 2024-08-17T17:43:52.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:52 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-17T17:43:52.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:52 smithi003 ceph-mon[100615]: osdmap e377: 8 total, 8 up, 8 in 2024-08-17T17:43:52.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:52.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:52.874 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:52.899 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:52.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:52.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:52.950 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:52 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-17T17:43:52.950 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:52 smithi053 ceph-mon[89963]: osdmap e377: 8 total, 8 up, 8 in 2024-08-17T17:43:53.001 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:53.023 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:53.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:53.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:53.125 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:53.150 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:53.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:53.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:53.226 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:43:53.252 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:53.277 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:53.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:53.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:53.280 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-17T17:43:53.301 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 6 1 2024-08-17T17:43:53.378 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:53.401 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:53.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:53.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:53.503 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:53.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:53 smithi003 ceph-mon[100615]: pgmap v1227: 119 pgs: 119 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 61 MiB/s wr, 974 op/s 2024-08-17T17:43:53.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:53 smithi003 ceph-mon[100615]: osdmap e378: 8 total, 8 up, 8 in 2024-08-17T17:43:53.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:43:53.532 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:53.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:53.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:53.548 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 64 16 94262 94246 92.016 0 - 0.01054 2024-08-17T17:43:53.622 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:43:53.634 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:53.658 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:53.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:53.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:53.760 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:53.761 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:53 smithi053 ceph-mon[89963]: pgmap v1227: 119 pgs: 119 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 61 MiB/s wr, 974 op/s 2024-08-17T17:43:53.761 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:53 smithi053 ceph-mon[89963]: osdmap e378: 8 total, 8 up, 8 in 2024-08-17T17:43:53.761 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:43:53.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:53 smithi003 ceph-mon[103344]: pgmap v1227: 119 pgs: 119 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 61 MiB/s wr, 974 op/s 2024-08-17T17:43:53.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:53 smithi003 ceph-mon[103344]: osdmap e378: 8 total, 8 up, 8 in 2024-08-17T17:43:53.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:43:53.791 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:53.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:53.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:53.866 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:43:53.882 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:43:53.893 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:53.930 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:53.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:53.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:54.032 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:54.069 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:54.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:54.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:54.171 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:54.206 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:54.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:54.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:54.308 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:54.347 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:54.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:54.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:54.449 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:54.485 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:54.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:54.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:54.517 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-08-17T17:43:54.548 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 65 16 95060 95044 91.3677 24.9375 0.00826801 0.0109364 2024-08-17T17:43:54.587 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:54.624 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:54.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:54.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:54.726 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:54.730 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:54 smithi003 ceph-mon[100615]: osdmap e379: 8 total, 8 up, 8 in 2024-08-17T17:43:54.730 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:54 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/4106449412' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:43:54.730 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:54 smithi003 ceph-mon[103344]: osdmap e379: 8 total, 8 up, 8 in 2024-08-17T17:43:54.730 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:54 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/4106449412' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:43:54.767 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:54.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:54.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:54.803 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:54 smithi053 ceph-mon[89963]: osdmap e379: 8 total, 8 up, 8 in 2024-08-17T17:43:54.804 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:54 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/4106449412' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:43:54.869 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:54.907 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:54.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:54.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:55.009 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:55.052 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:55.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:55.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:55.154 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:55.182 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:55.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:55.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:55.284 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:55.322 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:55.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:55.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:55.424 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:55.454 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:55.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:55.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:55.477 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:43:55.477 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2024-08-17T17:43:55.477 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2024-08-17T17:43:55.477 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2024-08-17T17:43:55.477 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:43:55.549 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 66 16 96019 96003 90.8914 59.9375 0.0274186 0.0109916 2024-08-17T17:43:55.555 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:55.578 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:55.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:55.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:55.679 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:55.713 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:55.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:55.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:55.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:55 smithi053 ceph-mon[89963]: pgmap v1230: 119 pgs: 1 activating+undersized+degraded+remapped, 118 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s; 5892/281007 objects degraded (2.097%); 0 B/s, 0 objects/s recovering 2024-08-17T17:43:55.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:55 smithi053 ceph-mon[89963]: Health check failed: Degraded data redundancy: 5892/281007 objects degraded (2.097%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:43:55.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:55 smithi053 ceph-mon[89963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:43:55.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:55 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/4106449412' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:43:55.717 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:55 smithi053 ceph-mon[89963]: osdmap e380: 8 total, 8 up, 8 in 2024-08-17T17:43:55.815 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:55.854 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:55.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:55.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:55.956 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:55.987 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:55.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:55.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:56.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:55 smithi003 ceph-mon[103344]: pgmap v1230: 119 pgs: 1 activating+undersized+degraded+remapped, 118 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s; 5892/281007 objects degraded (2.097%); 0 B/s, 0 objects/s recovering 2024-08-17T17:43:56.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:55 smithi003 ceph-mon[103344]: Health check failed: Degraded data redundancy: 5892/281007 objects degraded (2.097%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:43:56.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:55 smithi003 ceph-mon[103344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:43:56.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:55 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/4106449412' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:43:56.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:55 smithi003 ceph-mon[103344]: osdmap e380: 8 total, 8 up, 8 in 2024-08-17T17:43:56.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:55 smithi003 ceph-mon[100615]: pgmap v1230: 119 pgs: 1 activating+undersized+degraded+remapped, 118 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s; 5892/281007 objects degraded (2.097%); 0 B/s, 0 objects/s recovering 2024-08-17T17:43:56.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:55 smithi003 ceph-mon[100615]: Health check failed: Degraded data redundancy: 5892/281007 objects degraded (2.097%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:43:56.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:55 smithi003 ceph-mon[100615]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:43:56.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:55 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/4106449412' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:43:56.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:55 smithi003 ceph-mon[100615]: osdmap e380: 8 total, 8 up, 8 in 2024-08-17T17:43:56.024 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:43:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:43:55] "GET /metrics HTTP/1.1" 200 36553 "" "Prometheus/2.51.0" 2024-08-17T17:43:56.089 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:56.122 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:56.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:56.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:56.202 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-17T17:43:56.224 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:56.256 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:56.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:56.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:56.357 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:56.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:56.397 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:56.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:56.498 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:56.549 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 67 16 96974 96958 90.4257 59.6875 0.0190512 0.011053 2024-08-17T17:43:56.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:56.553 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:56.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:56.654 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:56.677 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:43:56.705 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:56.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:56.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:56.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:56 smithi003 ceph-mon[100615]: osdmap e381: 8 total, 8 up, 8 in 2024-08-17T17:43:56.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:56 smithi003 ceph-mon[103344]: osdmap e381: 8 total, 8 up, 8 in 2024-08-17T17:43:56.807 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:43:56.825 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:56 smithi053 ceph-mon[89963]: osdmap e381: 8 total, 8 up, 8 in 2024-08-17T17:43:56.845 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:56.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:56.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:43:56.947 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:56.990 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:56.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:56.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:57.092 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:57.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:57.132 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:57.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:57.233 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:43:57.292 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:57.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:57.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:43:57.393 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:57.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:57.436 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:57.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:57.537 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:57.549 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 68 16 97929 97913 89.9737 59.6875 0.00883906 0.0111076 2024-08-17T17:43:57.566 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:57.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:57.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:57.668 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:57.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:57.708 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:57.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:57.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:57 smithi003 ceph-mon[103344]: pgmap v1233: 119 pgs: 1 peering, 1 activating+undersized+degraded+remapped, 117 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 50 MiB/s wr, 807 op/s; 5892/284475 objects degraded (2.071%); 0 B/s, 0 objects/s recovering 2024-08-17T17:43:57.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:57 smithi003 ceph-mon[100615]: pgmap v1233: 119 pgs: 1 peering, 1 activating+undersized+degraded+remapped, 117 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 50 MiB/s wr, 807 op/s; 5892/284475 objects degraded (2.071%); 0 B/s, 0 objects/s recovering 2024-08-17T17:43:57.810 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:57.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:57.849 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:57.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:57.950 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:43:57.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:57 smithi053 ceph-mon[89963]: pgmap v1233: 119 pgs: 1 peering, 1 activating+undersized+degraded+remapped, 117 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 50 MiB/s wr, 807 op/s; 5892/284475 objects degraded (2.071%); 0 B/s, 0 objects/s recovering 2024-08-17T17:43:57.972 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:57.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:57.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:43:58.074 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:58.108 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:58.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:58.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:58.149 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:43:58.150 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:43:58.150 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:43:58.150 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:43:58.150 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:43:58.150 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:43:58.150 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:43:58.150 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:43:58.150 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:43:58.150 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:43:58.150 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:43:58.151 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:43:58.151 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:43:58.151 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:43:58.151 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:43:58.151 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:43:58.210 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:58.246 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:58.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:58.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:58.348 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:58.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:58.391 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:58.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:58.492 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:58.523 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:58.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:58.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:58.549 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 69 15 98802 98787 89.4613 54.625 0.0046441 0.011171 2024-08-17T17:43:58.626 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:58.640 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:58 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2490172792' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:43:58.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:58 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2490172792' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:43:58.665 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:58.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:58.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:58.767 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:43:58.799 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:58.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:58.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:43:58.886 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:43:58.901 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:58.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:58.929 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:58.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:58.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:58 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2490172792' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:43:59.030 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:43:59.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:59.069 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:59.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:43:59.169 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:43:59.201 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:59.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:59.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:43:59.303 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:59.340 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:59.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:59.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:59.435 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:43:59.442 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:59.478 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:59.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:59.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:59.549 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 70 16 99687 99671 88.9724 55.25 0.00687259 0.0112333 2024-08-17T17:43:59.580 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:59.623 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:59.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:59.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:59.725 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:43:59.762 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:43:59.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:59.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:43:59.864 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:43:59.867 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:43:59 smithi003 ceph-mon[103344]: pgmap v1234: 119 pgs: 1 peering, 1 activating+undersized+degraded+remapped, 117 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 34 MiB/s wr, 549 op/s; 5892/284475 objects degraded (2.071%); 0 B/s, 0 objects/s recovering 2024-08-17T17:43:59.868 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:43:59 smithi003 ceph-mon[100615]: pgmap v1234: 119 pgs: 1 peering, 1 activating+undersized+degraded+remapped, 117 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 34 MiB/s wr, 549 op/s; 5892/284475 objects degraded (2.071%); 0 B/s, 0 objects/s recovering 2024-08-17T17:43:59.901 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:43:59.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:43:59.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:43:59.951 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:43:59 smithi053 ceph-mon[89963]: pgmap v1234: 119 pgs: 1 peering, 1 activating+undersized+degraded+remapped, 117 active+clean; 5.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 34 MiB/s wr, 549 op/s; 5892/284475 objects degraded (2.071%); 0 B/s, 0 objects/s recovering 2024-08-17T17:44:00.003 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:00.038 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:00.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:00.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:00.139 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:00.174 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:00.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:00.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:00.276 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:00.311 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:00.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:00.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:00.413 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:00.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:00.453 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:00.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:00.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:44:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:44:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:44:00.549 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 71 16 100654 100638 88.5705 60.4375 0.00819993 0.0112849 2024-08-17T17:44:00.554 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:00.588 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:00.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:00.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:00.646 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:00 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 5892/284475 objects degraded (2.071%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:44:00.646 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:00 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 5892/284475 objects degraded (2.071%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:44:00.691 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:00.727 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:00.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:00.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:00.829 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:00.873 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:00.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:00.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:00.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:00 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 5892/284475 objects degraded (2.071%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:44:00.975 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:00.996 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:00.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:00.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:01.098 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:01.132 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:01.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:01.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:01.226 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:44:01.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:01.315 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:01.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:01.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:01.416 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:01.446 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:01.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:01.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:01.548 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:01.552 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 72 15 101554 101539 88.1224 56.3125 0.00551751 0.0113409 2024-08-17T17:44:01.593 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:01.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:01.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:01.600 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.6 to 1 (10000) 2024-08-17T17:44:01.695 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:01.725 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:01.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:01.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:01.826 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:01.828 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:01 smithi053 ceph-mon[89963]: pgmap v1235: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s; 5822/297549 objects misplaced (1.957%); 652 KiB/s, 10 objects/s recovering 2024-08-17T17:44:01.828 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:01 smithi053 ceph-mon[89963]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5892/284475 objects degraded (2.071%), 1 pg degraded) 2024-08-17T17:44:01.828 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:01 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/996875061' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:44:01.828 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:01 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:44:01.863 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:01.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:01.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:01.965 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:02.018 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:02.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:02.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:02.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:01 smithi003 ceph-mon[103344]: pgmap v1235: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s; 5822/297549 objects misplaced (1.957%); 652 KiB/s, 10 objects/s recovering 2024-08-17T17:44:02.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:01 smithi003 ceph-mon[103344]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5892/284475 objects degraded (2.071%), 1 pg degraded) 2024-08-17T17:44:02.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:01 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/996875061' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:44:02.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:01 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:44:02.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:01 smithi003 ceph-mon[100615]: pgmap v1235: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 69 MiB/s wr, 1.10k op/s; 5822/297549 objects misplaced (1.957%); 652 KiB/s, 10 objects/s recovering 2024-08-17T17:44:02.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:01 smithi003 ceph-mon[100615]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5892/284475 objects degraded (2.071%), 1 pg degraded) 2024-08-17T17:44:02.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:01 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/996875061' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:44:02.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:01 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:44:02.120 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:02.147 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:02.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:02.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:02.249 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:02.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:02.286 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:02.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:02.387 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:02.438 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:02.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:02.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:02.498 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:44:02.498 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:44:02.541 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:02.549 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 73 16 102431 102415 87.6653 54.75 0.0172531 0.0114019 2024-08-17T17:44:02.581 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:02.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:02.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:02.683 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:02.719 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:02.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:02.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:02.821 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:02.871 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:02.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:02.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:02.876 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:02 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-17T17:44:02.876 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:02 smithi053 ceph-mon[89963]: osdmap e382: 8 total, 8 up, 8 in 2024-08-17T17:44:02.973 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:03.010 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:03.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:03.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:03.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:02 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-17T17:44:03.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:02 smithi003 ceph-mon[103344]: osdmap e382: 8 total, 8 up, 8 in 2024-08-17T17:44:03.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:02 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-17T17:44:03.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:02 smithi003 ceph-mon[100615]: osdmap e382: 8 total, 8 up, 8 in 2024-08-17T17:44:03.111 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:03.136 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:03.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:03.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:03.215 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-17T17:44:03.238 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:03.304 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:03.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:03.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:03.405 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:03.456 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:03.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:03.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:03.550 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 74 16 103383 103367 87.2845 59.5 0.02498 0.0114508 2024-08-17T17:44:03.558 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:03.597 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:03.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:03.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:03.699 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:03.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:03.736 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:03.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:03.837 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:03.876 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:03.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:03.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:03.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:03 smithi053 ceph-mon[89963]: pgmap v1237: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 46 MiB/s wr, 734 op/s; 5822/298335 objects misplaced (1.951%); 570 KiB/s, 8 objects/s recovering 2024-08-17T17:44:03.978 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:04.011 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:04.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:04.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:04.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:03 smithi003 ceph-mon[103344]: pgmap v1237: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 46 MiB/s wr, 734 op/s; 5822/298335 objects misplaced (1.951%); 570 KiB/s, 8 objects/s recovering 2024-08-17T17:44:04.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:03 smithi003 ceph-mon[100615]: pgmap v1237: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 46 MiB/s wr, 734 op/s; 5822/298335 objects misplaced (1.951%); 570 KiB/s, 8 objects/s recovering 2024-08-17T17:44:04.113 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:04.150 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:04.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:04.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:04.252 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:04.274 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:04.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:04.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:04.375 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:04.411 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:04.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:04.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:04.513 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:04.548 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:04.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:04.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:04.550 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 75 16 104257 104241 86.849 54.625 0.00696131 0.0115083 2024-08-17T17:44:04.650 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:04.706 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:04.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:04.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:04.808 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:04.846 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:04.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:04.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:04.948 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:04.984 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:04.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:04.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:05.086 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:05.121 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:05.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:05.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:05.223 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:05.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:05.260 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:05.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:05.279 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:44:05.285 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 7 1 2024-08-17T17:44:05.361 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:05.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:05.400 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:05.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:05.501 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:05.531 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:05.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:05.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:05.550 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 76 16 105230 105214 86.5063 60.8125 0.0440001 0.0115534 2024-08-17T17:44:05.633 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:05.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:05.671 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:05.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:05.772 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:05.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:05 smithi003 ceph-mon[100615]: pgmap v1238: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 54 MiB/s wr, 869 op/s; 5750/306993 objects misplaced (1.873%); 1.0 MiB/s, 16 objects/s recovering 2024-08-17T17:44:05.808 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:05 smithi003 ceph-mon[103344]: pgmap v1238: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 54 MiB/s wr, 869 op/s; 5750/306993 objects misplaced (1.873%); 1.0 MiB/s, 16 objects/s recovering 2024-08-17T17:44:05.808 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:44:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:44:05] "GET /metrics HTTP/1.1" 200 36553 "" "Prometheus/2.51.0" 2024-08-17T17:44:05.817 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:05.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:05.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:05.919 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:05.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:05 smithi053 ceph-mon[89963]: pgmap v1238: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 54 MiB/s wr, 869 op/s; 5750/306993 objects misplaced (1.873%); 1.0 MiB/s, 16 objects/s recovering 2024-08-17T17:44:05.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:05.971 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:05.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:06.072 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:06.119 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:06.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:06.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:06.221 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:06.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:06.258 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:06.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:06.359 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:06.396 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:06.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:06.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:06.498 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:06.530 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:06.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:06.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:06.550 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 77 16 106175 106159 86.1499 59.0625 0.0091968 0.0116021 2024-08-17T17:44:06.632 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:06.633 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-08-17T17:44:06.670 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:06.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:06.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:06.772 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:06.806 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:06.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:06.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:06.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:06.913 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:06 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3516916788' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:44:06.914 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:06 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:44:06.914 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:06 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3516916788' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:44:06.914 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:06 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:44:06.941 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:06.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:06.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:06.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:06 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3516916788' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:44:06.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:06 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:44:07.043 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:07.080 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:07.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:07.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:07.181 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:07.213 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:07.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:07.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:07.315 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:07.348 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:07.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:07.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:07.450 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:07.482 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:07.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:07.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:07.550 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 78 16 107169 107153 85.8418 62.125 0.00624241 0.0116427 2024-08-17T17:44:07.584 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:07.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:07.620 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:07.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:07.721 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:07.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:07.764 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:07.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:07.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:07 smithi003 ceph-mon[103344]: pgmap v1239: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 58 MiB/s wr, 921 op/s; 5750/312126 objects misplaced (1.842%); 908 KiB/s, 14 objects/s recovering 2024-08-17T17:44:07.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:07 smithi003 ceph-mon[103344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:44:07.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:07 smithi003 ceph-mon[103344]: Cluster is now healthy 2024-08-17T17:44:07.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:07 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:44:07.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:07 smithi003 ceph-mon[103344]: osdmap e383: 8 total, 8 up, 8 in 2024-08-17T17:44:07.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:07 smithi003 ceph-mon[100615]: pgmap v1239: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 58 MiB/s wr, 921 op/s; 5750/312126 objects misplaced (1.842%); 908 KiB/s, 14 objects/s recovering 2024-08-17T17:44:07.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:07 smithi003 ceph-mon[100615]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:44:07.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:07 smithi003 ceph-mon[100615]: Cluster is now healthy 2024-08-17T17:44:07.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:07 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:44:07.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:07 smithi003 ceph-mon[100615]: osdmap e383: 8 total, 8 up, 8 in 2024-08-17T17:44:07.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:07.897 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:07.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:07.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:07.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:07 smithi053 ceph-mon[89963]: pgmap v1239: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 58 MiB/s wr, 921 op/s; 5750/312126 objects misplaced (1.842%); 908 KiB/s, 14 objects/s recovering 2024-08-17T17:44:07.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:07 smithi053 ceph-mon[89963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:44:07.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:07 smithi053 ceph-mon[89963]: Cluster is now healthy 2024-08-17T17:44:07.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:07 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:44:07.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:07 smithi053 ceph-mon[89963]: osdmap e383: 8 total, 8 up, 8 in 2024-08-17T17:44:07.998 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:08.033 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:08.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:08.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:08.135 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:08.176 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:08.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:08.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:08.278 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:08.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:08.318 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:08.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:08.419 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:08.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:08.455 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:08.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:08.550 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 79 16 108140 108124 85.5234 60.6875 0.0127965 0.0116871 2024-08-17T17:44:08.557 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:08.588 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:08.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:08.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:08.662 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:44:08.691 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:08.704 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:44:08.704 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:44:08.721 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:08.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:08.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:08.728 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:44:08.823 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:08.857 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:08.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:08.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:08.960 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:08.995 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:08.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:08.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:09.097 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:09.139 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:09.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:09.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:09.240 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:09.285 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:09.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:09.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:09.387 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:09.420 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:09.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:09.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:44:09.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:09 smithi053 ceph-mon[89963]: pgmap v1241: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 38 MiB/s wr, 607 op/s; 5750/312126 objects misplaced (1.842%); 576 KiB/s, 8 objects/s recovering 2024-08-17T17:44:09.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:09.474 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:09.474 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:44:09.475 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:09 smithi003 ceph-mon[103344]: pgmap v1241: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 38 MiB/s wr, 607 op/s; 5750/312126 objects misplaced (1.842%); 576 KiB/s, 8 objects/s recovering 2024-08-17T17:44:09.475 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:09.475 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:09.475 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:44:09.475 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:09 smithi003 ceph-mon[100615]: pgmap v1241: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 38 MiB/s wr, 607 op/s; 5750/312126 objects misplaced (1.842%); 576 KiB/s, 8 objects/s recovering 2024-08-17T17:44:09.475 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:09.522 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:09.550 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-08-17T17:44:09.550829+0000 min lat: 0.00107156 max lat: 2.20534 avg lat: 0.0117357 2024-08-17T17:44:09.551 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-17T17:44:09.551 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 80 16 109050 109034 85.1653 56.875 0.0242917 0.0117357 2024-08-17T17:44:09.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:09.566 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:09.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:09.667 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:09.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:09.703 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:09.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:09.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:09.837 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:09.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:09.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:09.938 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:09.976 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:09.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:09.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:10.078 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:10.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:10.111 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:10.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:10.212 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:10.253 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:10.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:10.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:10.355 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:10.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:10.395 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:10.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:10.461 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:44:10.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:44:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:44:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:44:10.495 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:10.528 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:10.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:10.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:10.550 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 81 16 109941 109925 84.8013 55.6875 0.00686724 0.0117871 2024-08-17T17:44:10.630 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:10.665 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:10.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:10.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:10.767 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:10.811 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:10.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:10.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:10.913 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:10.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:10.950 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:10.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:11.051 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:11.082 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:11.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:11.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:11.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:11.218 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:11.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:11.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:11.320 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:11.356 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:11.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:11.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:11.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-17T17:44:11.458 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:11.493 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:11.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:11.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:11.550 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 82 16 110811 110795 84.4302 54.375 0.00768158 0.011838 2024-08-17T17:44:11.595 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:11.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:11.635 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:11.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:11.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:11 smithi053 ceph-mon[89963]: pgmap v1242: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s; 5664/325542 objects misplaced (1.740%); 1.1 MiB/s, 17 objects/s recovering 2024-08-17T17:44:11.736 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:11.771 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:11.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:11.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:11.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:11 smithi003 ceph-mon[100615]: pgmap v1242: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s; 5664/325542 objects misplaced (1.740%); 1.1 MiB/s, 17 objects/s recovering 2024-08-17T17:44:11.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:11 smithi003 ceph-mon[103344]: pgmap v1242: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s; 5664/325542 objects misplaced (1.740%); 1.1 MiB/s, 17 objects/s recovering 2024-08-17T17:44:11.874 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:11.913 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:11.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:11.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:12.014 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:12.049 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:12.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:12.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:12.151 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:12.193 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:12.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:12.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:12.295 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:12.337 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:12.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:12.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:12.439 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:12.471 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:12.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:12.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:12.551 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 83 16 111785 111769 84.1464 60.875 0.0301906 0.0118784 2024-08-17T17:44:12.573 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:12.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:12.612 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:12.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:12.713 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:12.743 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:12.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:12.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:12.845 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:12.881 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:12.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:12.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:12.983 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:13.028 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:13.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:13.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:13.131 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:13.171 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:13.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:13.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:13.251 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:44:13.251 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:44:13.251 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:44:13.251 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:44:13.251 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:44:13.251 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:44:13.251 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:44:13.252 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:44:13.252 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:44:13.252 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:44:13.252 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:44:13.252 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:44:13.252 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:44:13.252 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:44:13.253 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:44:13.253 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:44:13.273 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:13.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:13.328 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:13.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:13.429 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:13.461 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:13.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:13.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:13.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:13 smithi003 ceph-mon[103344]: pgmap v1243: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 58 MiB/s wr, 931 op/s; 5664/326301 objects misplaced (1.736%); 1011 KiB/s, 15 objects/s recovering 2024-08-17T17:44:13.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:13 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2161189225' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:44:13.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:13 smithi003 ceph-mon[100615]: pgmap v1243: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 58 MiB/s wr, 931 op/s; 5664/326301 objects misplaced (1.736%); 1011 KiB/s, 15 objects/s recovering 2024-08-17T17:44:13.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:13 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2161189225' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:44:13.551 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 84 16 112706 112690 83.8299 57.5625 0.00950585 0.0119237 2024-08-17T17:44:13.563 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:13.583 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:44:13.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:13.616 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:13.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:13.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:13 smithi053 ceph-mon[89963]: pgmap v1243: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 58 MiB/s wr, 931 op/s; 5664/326301 objects misplaced (1.736%); 1011 KiB/s, 15 objects/s recovering 2024-08-17T17:44:13.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:13 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2161189225' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:44:13.717 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:13.763 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:13.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:13.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:13.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:13.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:13.903 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:13.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:14.005 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:14.045 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:14.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:14.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:14.144 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:44:14.146 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:14.196 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:14.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:14.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:14.298 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:14.338 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:14.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:14.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:14.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:14.460 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.7 to 1 (10000) 2024-08-17T17:44:14.473 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:14.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:14.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:14.551 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 85 16 113669 113653 83.5517 60.1875 0.00805977 0.0119626 2024-08-17T17:44:14.574 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:14.611 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:14.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:14.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:14.713 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:14.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:14 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2098996457' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:44:14.749 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:14.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:14.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:14.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:14 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2098996457' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:44:14.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:14 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2098996457' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:44:14.851 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:14.892 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:14.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:14.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:14.994 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:15.029 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:15.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:15.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:15.131 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:15.173 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:15.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:15.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:15.275 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:15.310 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:15.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:15.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:15.412 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:15.448 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:15.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:15.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:15.550 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:15.551 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 86 16 114697 114681 83.3272 64.25 0.011106 0.0119944 2024-08-17T17:44:15.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:15.602 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:15.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:15.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:15 smithi003 ceph-mon[100615]: pgmap v1244: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.8 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 58 MiB/s wr, 930 op/s; 5586/334932 objects misplaced (1.668%); 1.0 MiB/s, 16 objects/s recovering 2024-08-17T17:44:15.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:15 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2098996457' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-17T17:44:15.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:15 smithi003 ceph-mon[100615]: osdmap e384: 8 total, 8 up, 8 in 2024-08-17T17:44:15.652 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:15 smithi003 ceph-mon[103344]: pgmap v1244: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.8 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 58 MiB/s wr, 930 op/s; 5586/334932 objects misplaced (1.668%); 1.0 MiB/s, 16 objects/s recovering 2024-08-17T17:44:15.652 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:15 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2098996457' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-17T17:44:15.652 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:15 smithi003 ceph-mon[103344]: osdmap e384: 8 total, 8 up, 8 in 2024-08-17T17:44:15.703 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:15.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:15 smithi053 ceph-mon[89963]: pgmap v1244: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.8 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 58 MiB/s wr, 930 op/s; 5586/334932 objects misplaced (1.668%); 1.0 MiB/s, 16 objects/s recovering 2024-08-17T17:44:15.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:15 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2098996457' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-17T17:44:15.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:15 smithi053 ceph-mon[89963]: osdmap e384: 8 total, 8 up, 8 in 2024-08-17T17:44:15.740 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:15.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:15.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:15.842 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:15.875 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:15.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:15.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:15.946 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:44:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:44:15] "GET /metrics HTTP/1.1" 200 36570 "" "Prometheus/2.51.0" 2024-08-17T17:44:15.977 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:16.016 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:16.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:16.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:16.118 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:16.156 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:16.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:16.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:16.178 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:44:16.258 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:16.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:16.291 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:16.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:16.392 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:16.429 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:16.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:16.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:16.531 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:16.551 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 87 16 115658 115642 83.0597 60.0625 0.0279631 0.0120338 2024-08-17T17:44:16.580 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:16.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:16.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:16.587 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 4 1 2024-08-17T17:44:16.682 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:16.712 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:16.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:16.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:16.814 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:16.852 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:16.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:16.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:16.954 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:16.995 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:16.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:16.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:17.097 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:17.141 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:17.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:17.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:17.206 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:44:17.206 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-08-17T17:44:17.206 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-08-17T17:44:17.206 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-08-17T17:44:17.206 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:44:17.244 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:17.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:17.283 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:17.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:17.384 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:17.417 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:17.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:17.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:17.519 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:17.551 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 88 16 116607 116591 82.7898 59.3125 0.010005 0.0120712 2024-08-17T17:44:17.556 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:17.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:17.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:17.658 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:17.697 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:17.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:17.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:17.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:17 smithi053 ceph-mon[89963]: pgmap v1246: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 60 MiB/s wr, 957 op/s; 5586/340164 objects misplaced (1.642%); 1.0 MiB/s, 16 objects/s recovering 2024-08-17T17:44:17.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:17 smithi003 ceph-mon[103344]: pgmap v1246: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 60 MiB/s wr, 957 op/s; 5586/340164 objects misplaced (1.642%); 1.0 MiB/s, 16 objects/s recovering 2024-08-17T17:44:17.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:17 smithi003 ceph-mon[100615]: pgmap v1246: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 60 MiB/s wr, 957 op/s; 5586/340164 objects misplaced (1.642%); 1.0 MiB/s, 16 objects/s recovering 2024-08-17T17:44:17.799 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:17.838 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:17.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:17.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:17.940 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:17.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:17.977 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:17.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:18.078 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:18.118 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:18.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:18.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:18.220 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:18.259 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:18.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:18.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:18.361 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:18.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:18.400 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:18.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:18.501 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:18.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:18.539 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:18.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:18.551 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 89 16 117624 117608 82.5738 63.5625 0.00784217 0.0121045 2024-08-17T17:44:18.641 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:18.675 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:18.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:18.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:18.777 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:18.804 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:18.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:18.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:18.906 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:18.944 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:18.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:18.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:18.984 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-17T17:44:19.046 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:19.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:19.097 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:19.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:19.198 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:19.238 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:19.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:19.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:19.340 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:19.381 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:19.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:19.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:19.482 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:19.519 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:19.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:19.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:19.551 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 90 1 118543 118542 82.3048 58.375 0.040452 0.0121442 2024-08-17T17:44:19.621 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:19.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:19.650 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:19.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:19.688 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:44:19.689 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:44:19.725 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:19 smithi003 ceph-mon[103344]: pgmap v1247: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 58 MiB/s wr, 934 op/s; 5586/340164 objects misplaced (1.642%); 1.0 MiB/s, 16 objects/s recovering 2024-08-17T17:44:19.726 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:19 smithi003 ceph-mon[100615]: pgmap v1247: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 58 MiB/s wr, 934 op/s; 5586/340164 objects misplaced (1.642%); 1.0 MiB/s, 16 objects/s recovering 2024-08-17T17:44:19.751 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:19.774 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:19.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:19.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:19.877 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:19.899 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:19.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:19.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:19.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:19 smithi053 ceph-mon[89963]: pgmap v1247: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 58 MiB/s wr, 934 op/s; 5586/340164 objects misplaced (1.642%); 1.0 MiB/s, 16 objects/s recovering 2024-08-17T17:44:20.002 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:20.025 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:20.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:20.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:20.127 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:20.151 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:20.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:20.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:20.253 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:20.276 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:20.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:20.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:20.378 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:20.401 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:20.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:20.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:20.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:44:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:44:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:44:20.503 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:20.527 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:20.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:20.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:20.551 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total time run: 90.0186 2024-08-17T17:44:20.552 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total writes made: 118543 2024-08-17T17:44:20.552 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Write size: 65536 2024-08-17T17:44:20.552 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object size: 65536 2024-08-17T17:44:20.552 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Bandwidth (MB/sec): 82.3045 2024-08-17T17:44:20.552 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Bandwidth: 38.5535 2024-08-17T17:44:20.552 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max bandwidth (MB/sec): 230.812 2024-08-17T17:44:20.552 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min bandwidth (MB/sec): 0 2024-08-17T17:44:20.552 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average IOPS: 1316 2024-08-17T17:44:20.552 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev IOPS: 616.855 2024-08-17T17:44:20.552 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max IOPS: 3693 2024-08-17T17:44:20.552 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min IOPS: 0 2024-08-17T17:44:20.552 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average Latency(s): 0.0121443 2024-08-17T17:44:20.552 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Latency(s): 0.0371897 2024-08-17T17:44:20.552 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max latency(s): 2.20534 2024-08-17T17:44:20.553 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min latency(s): 0.00107156 2024-08-17T17:44:20.555 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Cleaning up (deleting benchmark objects) 2024-08-17T17:44:20.628 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:20.680 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:20.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:20.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:20.782 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:20.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:20.832 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:20.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:20.933 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:20.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:20.976 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:20.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:21.077 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:21.113 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:21.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:21.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:21.215 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:21.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:21.256 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:21.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:21.358 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:21.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:21.398 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:21.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:21.485 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-08-17T17:44:21.499 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:21.543 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:21.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:21.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:21.645 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:21.686 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:21.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:21.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:21.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:21 smithi003 ceph-mon[103344]: pgmap v1248: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 7.2 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 58 MiB/s wr, 932 op/s; 5514/353541 objects misplaced (1.560%); 960 KiB/s, 14 objects/s recovering 2024-08-17T17:44:21.795 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:21 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:44:21.795 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:21 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1633858532' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:44:21.795 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:21 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:44:21.795 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:21 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:21.795 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:21 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:21.795 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:21 smithi003 ceph-mon[100615]: pgmap v1248: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 7.2 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 58 MiB/s wr, 932 op/s; 5514/353541 objects misplaced (1.560%); 960 KiB/s, 14 objects/s recovering 2024-08-17T17:44:21.796 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:21 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:44:21.796 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:21 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1633858532' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:44:21.796 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:21 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:44:21.796 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:21 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:21.796 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:21 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:21.796 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:21.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:21.844 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:21.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:21.945 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:21.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:21 smithi053 ceph-mon[89963]: pgmap v1248: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 7.2 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 58 MiB/s wr, 932 op/s; 5514/353541 objects misplaced (1.560%); 960 KiB/s, 14 objects/s recovering 2024-08-17T17:44:21.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:21 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:44:21.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:21 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1633858532' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:44:21.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:21 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:44:21.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:21 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:21.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:21 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:21.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:21.986 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:21.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:22.088 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:22.121 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:22.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:22.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:22.223 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:22.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:22.267 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:22.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:22.368 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:22.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:22.406 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:22.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:22.507 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:22.548 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:22.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:22.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:22.556 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:22 smithi053 ceph-mon[89963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:44:22.556 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:22 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:44:22.556 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:22 smithi053 ceph-mon[89963]: osdmap e385: 8 total, 8 up, 8 in 2024-08-17T17:44:22.650 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:22.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:22.689 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:22.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:22.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:22 smithi003 ceph-mon[103344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:44:22.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:22 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:44:22.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:22 smithi003 ceph-mon[103344]: osdmap e385: 8 total, 8 up, 8 in 2024-08-17T17:44:22.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:22 smithi003 ceph-mon[100615]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:44:22.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:22 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:44:22.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:22 smithi003 ceph-mon[100615]: osdmap e385: 8 total, 8 up, 8 in 2024-08-17T17:44:22.790 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:22.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:22.821 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:22.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:22.922 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:22.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:22.974 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:22.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:23.075 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:23.117 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:23.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:23.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:23.218 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:23.232 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:44:23.268 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:23.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:23.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:23.369 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:23.405 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:23.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:23.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:23.506 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:23.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:23.544 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:23.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:23.645 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:23.678 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:23.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:23.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:23 smithi053 ceph-mon[89963]: pgmap v1250: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 7.2 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 50 MiB/s wr, 808 op/s; 5514/354189 objects misplaced (1.557%); 576 KiB/s, 8 objects/s recovering 2024-08-17T17:44:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:23.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:44:23.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:23 smithi003 ceph-mon[103344]: pgmap v1250: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 7.2 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 50 MiB/s wr, 808 op/s; 5514/354189 objects misplaced (1.557%); 576 KiB/s, 8 objects/s recovering 2024-08-17T17:44:23.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:23.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:23.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:23.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:44:23.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:23 smithi003 ceph-mon[100615]: pgmap v1250: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 7.2 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 50 MiB/s wr, 808 op/s; 5514/354189 objects misplaced (1.557%); 576 KiB/s, 8 objects/s recovering 2024-08-17T17:44:23.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:23.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:23.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:23.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:44:23.780 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:23.813 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:23.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:23.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:23.915 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:23.951 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:23.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:23.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:24.053 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:24.084 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:24.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:24.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:24.185 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:24.241 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:24.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:24.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:24.343 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:24.376 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:24.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:24.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:24.478 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:24.517 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:24.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:24.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:24.619 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:24.650 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:24.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:24.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:24.751 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:24.792 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:24.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:24.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:24.894 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:24.947 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:24.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:24.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:25.049 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:25.096 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:25.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:25.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:25.198 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:25.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:25.239 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:25.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:25.340 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:25.374 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:25.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:25.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:25.476 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:25.515 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:25.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:25.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:25.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:25.655 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:25.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:25.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:25.757 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:25.780 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:25.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:25.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:25.785 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:44:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:44:25] "GET /metrics HTTP/1.1" 200 36568 "" "Prometheus/2.51.0" 2024-08-17T17:44:25.883 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:25.924 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:25.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:25.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:25.939 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:44:25.939 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:44:25.939 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:44:25.939 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:44:25.940 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:44:25.940 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:44:25.940 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:44:25.940 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:44:25.940 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:44:25.940 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:44:25.940 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:44:25.940 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:44:25.941 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:44:25.941 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:44:25.941 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:44:25.941 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:44:25.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:25 smithi053 ceph-mon[89963]: pgmap v1251: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 7.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 103 B/s rd, 44 MiB/s wr, 887 op/s; 5377/350064 objects misplaced (1.536%); 1.3 MiB/s, 21 objects/s recovering 2024-08-17T17:44:25.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:25 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/522448122' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:44:26.026 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:26.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:26.080 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:26.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:26.181 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:26.207 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:26.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:26.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:26.230 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:44:26.271 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:25 smithi003 ceph-mon[103344]: pgmap v1251: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 7.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 103 B/s rd, 44 MiB/s wr, 887 op/s; 5377/350064 objects misplaced (1.536%); 1.3 MiB/s, 21 objects/s recovering 2024-08-17T17:44:26.271 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:25 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/522448122' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:44:26.272 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:25 smithi003 ceph-mon[100615]: pgmap v1251: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 7.1 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 103 B/s rd, 44 MiB/s wr, 887 op/s; 5377/350064 objects misplaced (1.536%); 1.3 MiB/s, 21 objects/s recovering 2024-08-17T17:44:26.272 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:25 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/522448122' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:44:26.309 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:26.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:26.354 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:26.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:26.455 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:26.499 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:26.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:26.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:26.600 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:26.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:26.644 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:26.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:26.745 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:26.784 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:26.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:26.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:26.886 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:26.917 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:26.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:26.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:27.019 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:27.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:27.066 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:27.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:27.167 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:27.202 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:27.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:27.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:27.304 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:27.332 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:27.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:27.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:27.434 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:27.486 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:27.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:27.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:27.588 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:27.629 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:27.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:27.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:27.731 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:27.776 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:27.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:27.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:27.878 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:27.924 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:27.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:27.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:28.026 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:28.055 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:28.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:28.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:28.157 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:28.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:28.190 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:28.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:28.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:27 smithi053 ceph-mon[89963]: pgmap v1252: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 32 MiB/s wr, 863 op/s; 5377/345348 objects misplaced (1.557%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:44:28.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:27 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:28.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:27 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:28.261 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:27 smithi003 ceph-mon[100615]: pgmap v1252: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 32 MiB/s wr, 863 op/s; 5377/345348 objects misplaced (1.557%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:44:28.262 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:27 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:28.262 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:27 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:28.262 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:27 smithi003 ceph-mon[103344]: pgmap v1252: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 32 MiB/s wr, 863 op/s; 5377/345348 objects misplaced (1.557%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:44:28.262 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:27 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:28.262 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:27 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:28.291 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:28.335 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:28.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:28.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:28.436 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:28.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:28.473 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:28.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:28.574 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:28.627 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:28.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:28.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:28.729 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:28.766 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:28.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:28.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:28.868 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:28.913 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:28.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:28.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:29.015 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:29.054 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:29.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:29.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:29.156 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:29.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:29.189 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:29.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:29.205 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.4 to 1 (10000) 2024-08-17T17:44:29.290 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:29.329 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:29.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:29.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:29.431 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:29.465 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:29.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:29.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:29.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:29 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:29.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:29 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:29.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:29 smithi053 ceph-mon[89963]: pgmap v1253: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 32 MiB/s wr, 863 op/s; 5377/345348 objects misplaced (1.557%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:44:29.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:29 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2528867212' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:44:29.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:29 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:44:29.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:29 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:29.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:29 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:29.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:29 smithi003 ceph-mon[103344]: pgmap v1253: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 32 MiB/s wr, 863 op/s; 5377/345348 objects misplaced (1.557%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:44:29.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:29 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2528867212' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:44:29.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:29 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:44:29.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:29 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:29.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:29 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:29.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:29 smithi003 ceph-mon[100615]: pgmap v1253: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 102 B/s rd, 32 MiB/s wr, 863 op/s; 5377/345348 objects misplaced (1.557%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:44:29.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:29 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2528867212' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:44:29.524 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:29 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:44:29.567 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:29.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:29.612 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:29.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:29.713 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:29.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:29.754 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:29.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:29.855 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:29.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:29.909 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:29.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:30.010 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:30.053 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:30.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:30.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:30.156 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:30.199 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:30.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:30.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:30.300 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:30.334 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:30.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:30.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:30.436 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:30.464 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:44:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:44:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:44:30.464 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:30 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-17T17:44:30.464 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:30 smithi053 ceph-mon[89963]: osdmap e386: 8 total, 8 up, 8 in 2024-08-17T17:44:30.469 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:30.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:30.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:30.516 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:30 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-17T17:44:30.516 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:30 smithi003 ceph-mon[100615]: osdmap e386: 8 total, 8 up, 8 in 2024-08-17T17:44:30.516 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:30 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-17T17:44:30.516 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:30 smithi003 ceph-mon[103344]: osdmap e386: 8 total, 8 up, 8 in 2024-08-17T17:44:30.557 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-08-17T17:44:30.557 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:44:30.571 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:30.605 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:30.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:30.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:30.706 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:30.744 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:30.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:30.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:30.793 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:44:30.845 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:30.851 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:44:30.851 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:44:30.890 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:30.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:30.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:30.992 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:31.047 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:31.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:31.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:31.148 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:31.187 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:31.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:31.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:31.289 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:31.328 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:31.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:31.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:31.429 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:31.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:31 smithi053 ceph-mon[89963]: pgmap v1255: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 114 B/s rd, 3.4 MiB/s wr, 935 op/s; 5306/332082 objects misplaced (1.598%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:44:31.477 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:31.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:31.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:31.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:31 smithi003 ceph-mon[103344]: pgmap v1255: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 114 B/s rd, 3.4 MiB/s wr, 935 op/s; 5306/332082 objects misplaced (1.598%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:44:31.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:31 smithi003 ceph-mon[100615]: pgmap v1255: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 114 B/s rd, 3.4 MiB/s wr, 935 op/s; 5306/332082 objects misplaced (1.598%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:44:31.578 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:31.613 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:31.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:31.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:31.628 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:44:31.715 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:31.747 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:31.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:31.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:31.849 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:31.857 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-17T17:44:31.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:31.887 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:31.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:31.988 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:32.029 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:32.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:32.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:32.131 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:32.174 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:32.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:32.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:32.241 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:44:32.276 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:32.313 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:32.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:32.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:32.415 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:32.445 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:32.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:32.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:32.547 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:32.569 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:32.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:32.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:32.671 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:32.710 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:32.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:32.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:32.812 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:32.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:32.844 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:32.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:32.946 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:32.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:32.984 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:32.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:33.085 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:33.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:33.121 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:33.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:33.222 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:33.257 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:33.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:33.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:33.359 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:33.394 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:33.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:33.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:33.496 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:33.529 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:33.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:33.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:33.577 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:33 smithi003 ceph-mon[103344]: pgmap v1256: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.7 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 3.0 MiB/s wr, 859 op/s; 5306/331302 objects misplaced (1.602%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:44:33.577 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:33 smithi003 ceph-mon[100615]: pgmap v1256: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.7 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 3.0 MiB/s wr, 859 op/s; 5306/331302 objects misplaced (1.602%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:44:33.631 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:33.661 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:33.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:33.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:33.715 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:33 smithi053 ceph-mon[89963]: pgmap v1256: 119 pgs: 1 active+recovering+undersized+remapped, 118 active+clean; 6.7 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 102 B/s rd, 3.0 MiB/s wr, 859 op/s; 5306/331302 objects misplaced (1.602%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:44:33.763 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:33.799 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:33.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:33.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:33.901 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:33.930 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:44:33.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:33.963 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:33.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:34.064 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:34.107 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:34.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:34.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:34.209 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:34.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:34.243 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:34.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:34.346 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:34.385 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:34.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:34.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:34.487 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:34.522 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:34.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:34.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:34.624 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:34.662 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:44:34.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:34.663 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:34.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:34.764 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:34.799 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:34.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:34.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:34.901 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:34.932 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:34.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:34.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:35.034 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:35.072 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:35.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:35.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:35.174 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:35.214 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:35.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:35.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:35.316 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:35.328 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:35 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:35.329 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:35 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:35.329 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:35 smithi003 ceph-mon[100615]: pgmap v1257: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 902 op/s; 147/322971 objects degraded (0.046%); 5089/322971 objects misplaced (1.576%); 902 KiB/s, 14 objects/s recovering 2024-08-17T17:44:35.329 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:35 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:35.329 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:35 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:35.329 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:35 smithi003 ceph-mon[103344]: pgmap v1257: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 902 op/s; 147/322971 objects degraded (0.046%); 5089/322971 objects misplaced (1.576%); 902 KiB/s, 14 objects/s recovering 2024-08-17T17:44:35.356 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:35.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:35.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:35.458 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:35.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:35 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:35.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:35 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:35.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:35 smithi053 ceph-mon[89963]: pgmap v1257: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 6.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 902 op/s; 147/322971 objects degraded (0.046%); 5089/322971 objects misplaced (1.576%); 902 KiB/s, 14 objects/s recovering 2024-08-17T17:44:35.502 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:35.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:35.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:35.604 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:35.645 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:35.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:35.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:35.747 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:35.789 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:35.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:35.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:35.873 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:44:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:44:35] "GET /metrics HTTP/1.1" 200 36568 "" "Prometheus/2.51.0" 2024-08-17T17:44:35.890 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:35.925 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:35.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:35.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:36.026 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:36.061 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:36.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:36.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:36.163 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:36.198 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:36.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:36.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:36.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:36 smithi003 ceph-mon[103344]: Health check failed: Degraded data redundancy: 147/322971 objects degraded (0.046%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:44:36.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:36 smithi003 ceph-mon[100615]: Health check failed: Degraded data redundancy: 147/322971 objects degraded (0.046%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:44:36.301 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:36.335 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:36.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:36.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:36.437 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:36.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:36.464 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:36.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:36.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:36 smithi053 ceph-mon[89963]: Health check failed: Degraded data redundancy: 147/322971 objects degraded (0.046%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:44:36.566 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:36.604 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:36.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:36.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:36.706 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:36.739 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:36.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:36.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:36.841 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:36.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:36.880 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:36.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:36.981 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:37.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:37.018 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:37.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:37.119 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:37.152 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:37.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:37.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:37.243 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:37 smithi003 ceph-mon[103344]: pgmap v1258: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 910 op/s; 147/318030 objects degraded (0.046%); 5089/318030 objects misplaced (1.600%); 902 KiB/s, 14 objects/s recovering 2024-08-17T17:44:37.243 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:37 smithi003 ceph-mon[100615]: pgmap v1258: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 910 op/s; 147/318030 objects degraded (0.046%); 5089/318030 objects misplaced (1.600%); 902 KiB/s, 14 objects/s recovering 2024-08-17T17:44:37.254 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:37.293 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:37.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:37.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:37.395 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:37.435 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:37.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:37.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:37.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:37 smithi053 ceph-mon[89963]: pgmap v1258: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 910 op/s; 147/318030 objects degraded (0.046%); 5089/318030 objects misplaced (1.600%); 902 KiB/s, 14 objects/s recovering 2024-08-17T17:44:37.537 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:37.567 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:37.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:37.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:37.669 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:37.709 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:37.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:37.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:37.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:37.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:37.853 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:37.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:37.954 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:37.995 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:37.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:37.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:38.097 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:38.111 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-08-17T17:44:38.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:38.168 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:38.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:38.193 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:38 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1192166337' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:44:38.193 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:38 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:44:38.194 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:38 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1192166337' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:44:38.194 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:38 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:44:38.269 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:38.315 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:38.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:38.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:38.417 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:38.451 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:38.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:38.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:38.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:38 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1192166337' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:44:38.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:38 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:44:38.553 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:38.594 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:38.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:38.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:38.696 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:38.725 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:38.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:38.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:38.827 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:38.860 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:38.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:38.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:38.962 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:38.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:38.993 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:38.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:39.094 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:39.129 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:39.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:39.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:39.224 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:39 smithi003 ceph-mon[103344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:44:39.224 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:39 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:44:39.224 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:39 smithi003 ceph-mon[103344]: osdmap e387: 8 total, 8 up, 8 in 2024-08-17T17:44:39.225 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:39.225 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:44:39.225 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:39 smithi003 ceph-mon[103344]: pgmap v1260: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 668 op/s; 147/318030 objects degraded (0.046%); 5089/318030 objects misplaced (1.600%); 487 KiB/s, 7 objects/s recovering 2024-08-17T17:44:39.225 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:39 smithi003 ceph-mon[100615]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:44:39.225 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:39 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:44:39.225 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:39 smithi003 ceph-mon[100615]: osdmap e387: 8 total, 8 up, 8 in 2024-08-17T17:44:39.225 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:39.225 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:44:39.225 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:39 smithi003 ceph-mon[100615]: pgmap v1260: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 668 op/s; 147/318030 objects degraded (0.046%); 5089/318030 objects misplaced (1.600%); 487 KiB/s, 7 objects/s recovering 2024-08-17T17:44:39.231 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:39.269 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:39.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:39.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:39.367 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:39 smithi053 ceph-mon[89963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:44:39.367 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:39 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:44:39.368 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:39 smithi053 ceph-mon[89963]: osdmap e387: 8 total, 8 up, 8 in 2024-08-17T17:44:39.368 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:39.368 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:44:39.368 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:39 smithi053 ceph-mon[89963]: pgmap v1260: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 6.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 668 op/s; 147/318030 objects degraded (0.046%); 5089/318030 objects misplaced (1.600%); 487 KiB/s, 7 objects/s recovering 2024-08-17T17:44:39.371 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:39.421 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:39.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:39.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:39.523 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:39.557 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:39.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:39.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:39.659 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:39.694 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:39.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:39.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:39.796 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:39.831 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:39.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:39.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:39.933 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:39.968 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:39.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:39.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:40.070 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:40.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:40.115 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:40.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:40.216 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:40.251 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:40.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:40.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:40.353 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:40.360 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:44:40.361 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:44:40.399 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:40.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:40.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:40.407 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:44:40.407 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:44:40.441 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:44:40.441 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:44:40.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:40 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3223149709' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:44:40.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:44:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:44:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:44:40.478 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:44:40.478 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:44:40.501 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:40.512 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:44:40.513 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:44:40.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:40.543 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:40.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:40.552 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:44:40.552 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:44:40.583 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:44:40.583 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:44:40.615 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:40 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3223149709' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:44:40.616 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:40 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3223149709' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:44:40.620 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:44:40.620 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:44:40.644 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:40.709 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:40.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:40.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:40.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:40.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:40.856 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:40.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:40.957 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:40.994 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:40.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:40.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:41.096 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:41.129 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:41.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:41.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:41.231 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:41.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:41.270 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:41.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:41.371 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:41.414 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:41.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:41.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:41.516 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:41.552 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:41.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:41.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:41.653 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:41.687 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:41.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:41.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:41.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:41 smithi053 ceph-mon[89963]: pgmap v1261: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 935 op/s; 393/304017 objects degraded (0.129%); 4754/304017 objects misplaced (1.564%); 1017 KiB/s, 15 objects/s recovering 2024-08-17T17:44:41.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:41 smithi003 ceph-mon[100615]: pgmap v1261: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 935 op/s; 393/304017 objects degraded (0.129%); 4754/304017 objects misplaced (1.564%); 1017 KiB/s, 15 objects/s recovering 2024-08-17T17:44:41.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:41 smithi003 ceph-mon[103344]: pgmap v1261: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 935 op/s; 393/304017 objects degraded (0.129%); 4754/304017 objects misplaced (1.564%); 1017 KiB/s, 15 objects/s recovering 2024-08-17T17:44:41.789 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:41.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:41.822 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:41.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:41.923 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:41.974 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:41.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:41.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:42.076 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:42.095 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:44:42.106 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:42.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:42.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:42.207 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:42.263 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:42.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:42.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:42.365 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:42.411 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:42.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:42.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:42.513 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:42.546 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:42.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:42.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:42.648 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:42.681 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:42.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:42.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:42.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:42 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 393/304017 objects degraded (0.129%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:44:42.771 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:42 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 393/304017 objects degraded (0.129%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:44:42.771 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:42 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 393/304017 objects degraded (0.129%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:44:42.783 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:42.826 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:42.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:42.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:42.928 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:42.965 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:42.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:42.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:43.067 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:43.103 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:43.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:43.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:43.205 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:43.237 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:43.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:43.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:43.339 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:43.388 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:43.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:43.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:43.490 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:43.523 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:43.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:43.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:43.625 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:43.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:43.661 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:43.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:43.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:43 smithi053 ceph-mon[89963]: pgmap v1262: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 938 op/s; 393/303147 objects degraded (0.130%); 4754/303147 objects misplaced (1.568%); 1017 KiB/s, 15 objects/s recovering 2024-08-17T17:44:43.762 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:43.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:43 smithi003 ceph-mon[100615]: pgmap v1262: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 938 op/s; 393/303147 objects degraded (0.130%); 4754/303147 objects misplaced (1.568%); 1017 KiB/s, 15 objects/s recovering 2024-08-17T17:44:43.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:43 smithi003 ceph-mon[103344]: pgmap v1262: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 6.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 938 op/s; 393/303147 objects degraded (0.130%); 4754/303147 objects misplaced (1.568%); 1017 KiB/s, 15 objects/s recovering 2024-08-17T17:44:43.808 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:43.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:43.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:43.910 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:43.933 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:43.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:43.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:43.956 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:44:43.995 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:44:43.997 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:44:44.035 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:44.092 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:44.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:44.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:44.194 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:44.254 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:44.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:44.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:44.356 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:44.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:44.395 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:44.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:44.434 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:44:44.434 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:44:44.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.434 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36612.0:139645 10.a e387/357 10:55e30b1e:::benchmark_data_smithi003_292749_object21099:head v 387'8685, mlcod=387'8683)", 2024-08-17T17:44:44.435 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:44:42.933922+0000", 2024-08-17T17:44:44.435 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0041668180000000001, 2024-08-17T17:44:44.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042035960000000004, 2024-08-17T17:44:44.435 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:44:44.435 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:44:44.435 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:44:44.435 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:44:44.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.435 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:44:44.435 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.933922+0000", 2024-08-17T17:44:44.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:44:44.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.436 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:44:44.436 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.933922+0000", 2024-08-17T17:44:44.436 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:44:44.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.436 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:44:44.436 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.933923+0000", 2024-08-17T17:44:44.436 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.75e-07 2024-08-17T17:44:44.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.436 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:44:44.436 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.933930+0000", 2024-08-17T17:44:44.436 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8800000000000002e-06 2024-08-17T17:44:44.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:44:44.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.933940+0000", 2024-08-17T17:44:44.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.043e-05 2024-08-17T17:44:44.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:44:44.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.933947+0000", 2024-08-17T17:44:44.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4359999999999999e-06 2024-08-17T17:44:44.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:44:44.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.936511+0000", 2024-08-17T17:44:44.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025646900000000001 2024-08-17T17:44:44.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:44:44.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.936532+0000", 2024-08-17T17:44:44.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0265999999999999e-05 2024-08-17T17:44:44.438 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:44:44.438 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:44:44.438 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:44:44.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.438 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36612.0:139646 10.8 10:1aaa1409:::benchmark_data_smithi003_292749_object21100:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-08-17T17:44:44.438 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:44:42.935137+0000", 2024-08-17T17:44:44.439 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0029518180000000002, 2024-08-17T17:44:44.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0030545070000000001, 2024-08-17T17:44:44.439 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:44:44.439 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:44:44.439 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:44:44.439 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:44:44.439 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36612", 2024-08-17T17:44:44.439 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:44:44.439 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 139646 2024-08-17T17:44:44.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.439 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:44:44.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:44:44.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.935137+0000", 2024-08-17T17:44:44.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:44:44.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:44:44.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.935137+0000", 2024-08-17T17:44:44.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:44:44.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:44:44.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.935139+0000", 2024-08-17T17:44:44.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6920000000000001e-06 2024-08-17T17:44:44.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.441 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:44:44.441 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.935146+0000", 2024-08-17T17:44:44.441 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6200000000000001e-06 2024-08-17T17:44:44.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.441 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:44:44.441 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.935162+0000", 2024-08-17T17:44:44.441 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6384000000000001e-05 2024-08-17T17:44:44.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.441 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:44:44.441 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.935171+0000", 2024-08-17T17:44:44.441 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6459999999999999e-06 2024-08-17T17:44:44.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.442 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:44:44.442 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.935228+0000", 2024-08-17T17:44:44.442 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7513999999999999e-05 2024-08-17T17:44:44.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.442 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:44:44.442 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.935354+0000", 2024-08-17T17:44:44.442 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012621799999999999 2024-08-17T17:44:44.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.442 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,7", 2024-08-17T17:44:44.442 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.935490+0000", 2024-08-17T17:44:44.442 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000135792 2024-08-17T17:44:44.443 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:44:44.443 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:44:44.443 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:44:44.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.443 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36612.0:139647 10.a e387/357)", 2024-08-17T17:44:44.443 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:44:42.938154+0000", 2024-08-17T17:44:44.443 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 4294967295.9999352, 2024-08-17T17:44:44.443 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9783000000000003e-05, 2024-08-17T17:44:44.443 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:44:44.443 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:44:44.443 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-08-17T17:44:44.443 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:44:44.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.444 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:44:44.444 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.938154+0000", 2024-08-17T17:44:44.444 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:44:44.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.444 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:44:44.444 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.938154+0000", 2024-08-17T17:44:44.444 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:44:44.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.444 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:44:44.444 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.938155+0000", 2024-08-17T17:44:44.444 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0419999999999999e-06 2024-08-17T17:44:44.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.445 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:44:44.445 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.938163+0000", 2024-08-17T17:44:44.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6529999999999994e-06 2024-08-17T17:44:44.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.445 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.445 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:44:44.445 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.938174+0000", 2024-08-17T17:44:44.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1134999999999999e-05 2024-08-17T17:44:44.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.445 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.445 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:44:44.445 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.938181+0000", 2024-08-17T17:44:44.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9110000000000004e-06 2024-08-17T17:44:44.446 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:44:44.446 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:44:44.446 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:44:44.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.446 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36612.0:139624 10.2 e387/357 10:46f6f4a3:::benchmark_data_smithi003_292749_object21078:head v 387'8540, mlcod=387'8536)", 2024-08-17T17:44:44.446 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:44:42.911801+0000", 2024-08-17T17:44:44.446 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.026287822999999998, 2024-08-17T17:44:44.446 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.026492326, 2024-08-17T17:44:44.446 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:44:44.446 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:44:44.446 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:44:44.446 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:44:44.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.447 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:44:44.447 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.911801+0000", 2024-08-17T17:44:44.447 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:44:44.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.447 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:44:44.447 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.911801+0000", 2024-08-17T17:44:44.447 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:44:44.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.447 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:44:44.447 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.911802+0000", 2024-08-17T17:44:44.447 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1299999999999999e-07 2024-08-17T17:44:44.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.448 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:44:44.448 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.911808+0000", 2024-08-17T17:44:44.448 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6760000000000004e-06 2024-08-17T17:44:44.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.448 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:44:44.448 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.911828+0000", 2024-08-17T17:44:44.448 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0635000000000001e-05 2024-08-17T17:44:44.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.448 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:44:44.448 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.911835+0000", 2024-08-17T17:44:44.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9360000000000002e-06 2024-08-17T17:44:44.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.449 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:44:44.449 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.919493+0000", 2024-08-17T17:44:44.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0076577420000000004 2024-08-17T17:44:44.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.449 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:44:44.449 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.919511+0000", 2024-08-17T17:44:44.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8382000000000001e-05 2024-08-17T17:44:44.449 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:44:44.449 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:44:44.450 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:44:44.450 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.450 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.450 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36612.0:139636 10.4 10:26e650a4:::benchmark_data_smithi003_292749_object21090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-08-17T17:44:44.450 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:44:42.920561+0000", 2024-08-17T17:44:44.450 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.017527728999999999, 2024-08-17T17:44:44.450 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017785103999999999, 2024-08-17T17:44:44.450 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:44:44.450 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:44:44.450 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:44:44.450 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:44:44.450 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36612", 2024-08-17T17:44:44.450 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:44:44.450 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 139636 2024-08-17T17:44:44.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.451 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:44:44.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.451 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:44:44.451 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.920561+0000", 2024-08-17T17:44:44.451 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:44:44.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.451 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:44:44.451 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.920561+0000", 2024-08-17T17:44:44.451 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:44:44.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.451 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:44:44.452 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.920562+0000", 2024-08-17T17:44:44.452 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0999999999999997e-07 2024-08-17T17:44:44.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.452 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:44:44.452 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.920566+0000", 2024-08-17T17:44:44.452 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0960000000000003e-06 2024-08-17T17:44:44.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.452 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:44:44.452 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.920577+0000", 2024-08-17T17:44:44.452 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0749e-05 2024-08-17T17:44:44.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.453 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:44:44.453 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.920584+0000", 2024-08-17T17:44:44.453 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7089999999999996e-06 2024-08-17T17:44:44.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.453 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:44:44.453 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.920748+0000", 2024-08-17T17:44:44.453 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000163962 2024-08-17T17:44:44.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.453 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:44:44.453 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.920935+0000", 2024-08-17T17:44:44.453 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000186787 2024-08-17T17:44:44.454 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.454 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.454 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,4", 2024-08-17T17:44:44.454 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.921059+0000", 2024-08-17T17:44:44.454 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012398500000000001 2024-08-17T17:44:44.454 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.454 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.454 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:44:44.454 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.933569+0000", 2024-08-17T17:44:44.454 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012510571999999999 2024-08-17T17:44:44.454 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.454 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.454 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:44:44.454 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.933624+0000", 2024-08-17T17:44:44.455 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4904999999999997e-05 2024-08-17T17:44:44.455 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:44:44.455 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:44:44.455 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:44:44.455 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.455 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.455 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36612.0:139638 10.2 e387/357 10:4ff6c046:::benchmark_data_smithi003_292749_object21092:head v 387'8541, mlcod=387'8538)", 2024-08-17T17:44:44.455 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:44:42.926719+0000", 2024-08-17T17:44:44.455 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.011370521, 2024-08-17T17:44:44.455 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011705724000000001, 2024-08-17T17:44:44.455 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:44:44.455 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:44:44.455 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:44:44.455 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:44:44.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.456 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:44:44.456 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.926719+0000", 2024-08-17T17:44:44.456 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:44:44.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.456 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:44:44.456 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.926719+0000", 2024-08-17T17:44:44.456 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:44:44.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.456 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:44:44.456 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.926719+0000", 2024-08-17T17:44:44.456 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7100000000000001e-07 2024-08-17T17:44:44.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.457 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:44:44.457 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.926726+0000", 2024-08-17T17:44:44.457 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4470000000000003e-06 2024-08-17T17:44:44.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.457 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:44:44.457 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.926738+0000", 2024-08-17T17:44:44.457 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1793e-05 2024-08-17T17:44:44.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.457 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:44:44.457 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.926745+0000", 2024-08-17T17:44:44.457 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0450000000000003e-06 2024-08-17T17:44:44.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.458 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:44:44.458 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.931887+0000", 2024-08-17T17:44:44.458 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.005142204 2024-08-17T17:44:44.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.458 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:44:44.458 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.931911+0000", 2024-08-17T17:44:44.458 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4358999999999998e-05 2024-08-17T17:44:44.458 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:44:44.458 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:44:44.458 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:44:44.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.459 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.36612.0:139642 10.4 10:2ac83463:::benchmark_data_smithi003_292749_object21096:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-08-17T17:44:44.459 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:44:42.926760+0000", 2024-08-17T17:44:44.459 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.011329304, 2024-08-17T17:44:44.459 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011720685, 2024-08-17T17:44:44.459 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:44:44.459 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:44:44.459 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-08-17T17:44:44.459 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-08-17T17:44:44.459 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.36612", 2024-08-17T17:44:44.459 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:44:44.459 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 139642 2024-08-17T17:44:44.459 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.459 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:44:44.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.460 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:44:44.460 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.926760+0000", 2024-08-17T17:44:44.460 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:44:44.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.460 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:44:44.460 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.926760+0000", 2024-08-17T17:44:44.460 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:44:44.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.460 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:44:44.460 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.926761+0000", 2024-08-17T17:44:44.460 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9400000000000004e-07 2024-08-17T17:44:44.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.461 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:44:44.461 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.926778+0000", 2024-08-17T17:44:44.461 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7110000000000001e-05 2024-08-17T17:44:44.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.461 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:44:44.461 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.926788+0000", 2024-08-17T17:44:44.461 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9329999999999996e-06 2024-08-17T17:44:44.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.461 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:44:44.461 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.926794+0000", 2024-08-17T17:44:44.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3239999999999999e-06 2024-08-17T17:44:44.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.462 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.462 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:44:44.462 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.934579+0000", 2024-08-17T17:44:44.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0077853669999999996 2024-08-17T17:44:44.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.462 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.462 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:44:44.462 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.934696+0000", 2024-08-17T17:44:44.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000116935 2024-08-17T17:44:44.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.462 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.463 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,4", 2024-08-17T17:44:44.463 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.934806+0000", 2024-08-17T17:44:44.463 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110092 2024-08-17T17:44:44.463 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:44:44.463 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:44:44.463 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:44:44.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.463 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.463 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36612.0:139641 10.2 e387/357 10:440cde1f:::benchmark_data_smithi003_292749_object21095:head v 387'8542, mlcod=387'8538)", 2024-08-17T17:44:44.463 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:44:42.931569+0000", 2024-08-17T17:44:44.463 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0065201449999999998, 2024-08-17T17:44:44.463 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0069783800000000002, 2024-08-17T17:44:44.463 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:44:44.463 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:44:44.464 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:44:44.464 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:44:44.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.464 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:44:44.464 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.931569+0000", 2024-08-17T17:44:44.464 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:44:44.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.464 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:44:44.464 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.931569+0000", 2024-08-17T17:44:44.464 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:44:44.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.464 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:44:44.465 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.931570+0000", 2024-08-17T17:44:44.465 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1599999999999999e-06 2024-08-17T17:44:44.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.465 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:44:44.465 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.931578+0000", 2024-08-17T17:44:44.465 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2430000000000005e-06 2024-08-17T17:44:44.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.465 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:44:44.465 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.931599+0000", 2024-08-17T17:44:44.465 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0290999999999999e-05 2024-08-17T17:44:44.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.466 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:44:44.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.931608+0000", 2024-08-17T17:44:44.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9320000000000002e-06 2024-08-17T17:44:44.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.466 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:44:44.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.937590+0000", 2024-08-17T17:44:44.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0059821029999999999 2024-08-17T17:44:44.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:44:44.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:44:44.466 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:44:44.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:42.937609+0000", 2024-08-17T17:44:44.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9619999999999998e-05 2024-08-17T17:44:44.467 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:44:44.467 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:44:44.467 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:44:44.467 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:44:44.467 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-08-17T17:44:44.467 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 8 2024-08-17T17:44:44.467 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:44:44.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:44.531 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:44.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:44.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:44.633 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:44.635 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:44 smithi003 ceph-mon[100615]: from='client.56044 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:44:44.635 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:44 smithi003 ceph-mon[103344]: from='client.56044 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:44:44.669 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:44.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:44.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:44.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:44 smithi053 ceph-mon[89963]: from='client.56044 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:44:44.771 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:44.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:44.806 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:44.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:44.907 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:44.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:44.967 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:44.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:45.068 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:45.106 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:45.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:45.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:45.208 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:45.248 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:45.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:45.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:45.350 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:45.389 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:45.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:45.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:45.491 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:45.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:45.522 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:45.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:45.524 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1262,"stamp":"2024-08-17T17:44:42.399527+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6625236965,"num_objects":101049,"num_object_clones":0,"num_object_copies":303147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":393,"num_objects_misplaced":4754,"num_objects_unfound":0,"num_objects_dirty":101049,"num_whiteouts":0,"num_read":5979,"num_read_kb":12992,"num_write":136974,"num_write_kb":7599376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1847,"num_bytes_recovered":116730872,"num_keys_recovered":40,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107518,"ondisk_log_size":107518,"up":357,"acting":356,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":362,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":26076112,"kb_used_data":20036476,"kb_used_omap":37,"kb_used_meta":6039450,"kb_avail":723885104,"statfs":{"total":767960285184,"available":741258346496,"internally_reserved":0,"allocated":20517351424,"data_stored":20499725399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38729,"internal_metadata":6184396983},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-615055360,"num_objects":-9385,"num_object_clones":0,"num_object_copies":-28155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":393,"num_objects_misplaced":-552,"num_objects_unfound":0,"num_objects_dirty":-9385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9385,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":159,"num_bytes_recovered":10420224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004026"},"pg_stats":[{"pgid":"1.11","version":"90'149","reported_seq":825,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.126283+0000","last_change":"2024-08-17T17:38:30.378664+0000","last_active":"2024-08-17T17:44:38.126283+0000","last_peered":"2024-08-17T17:44:38.126283+0000","last_clean":"2024-08-17T17:44:38.126283+0000","last_became_active":"2024-08-17T17:38:30.342287+0000","last_became_peered":"2024-08-17T17:38:30.342287+0000","last_unstale":"2024-08-17T17:44:38.126283+0000","last_undegraded":"2024-08-17T17:44:38.126283+0000","last_fullsized":"2024-08-17T17:44:38.126283+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:30.378592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:30.378592+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:04:28.933354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":646,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.123276+0000","last_change":"2024-08-17T17:33:20.797029+0000","last_active":"2024-08-17T17:44:38.123276+0000","last_peered":"2024-08-17T17:44:38.123276+0000","last_clean":"2024-08-17T17:44:38.123276+0000","last_became_active":"2024-08-17T17:33:20.796679+0000","last_became_peered":"2024-08-17T17:33:20.796679+0000","last_unstale":"2024-08-17T17:44:38.123276+0000","last_undegraded":"2024-08-17T17:44:38.123276+0000","last_fullsized":"2024-08-17T17:44:38.123276+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:51.438262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":411,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.206373+0000","last_change":"2024-08-17T17:41:34.624198+0000","last_active":"2024-08-17T17:44:29.206373+0000","last_peered":"2024-08-17T17:44:29.206373+0000","last_clean":"2024-08-17T17:44:29.206373+0000","last_became_active":"2024-08-17T17:41:34.616071+0000","last_became_peered":"2024-08-17T17:41:34.616071+0000","last_unstale":"2024-08-17T17:44:29.206373+0000","last_undegraded":"2024-08-17T17:44:29.206373+0000","last_fullsized":"2024-08-17T17:44:29.206373+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:34:48.805354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"90'149","reported_seq":470,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.210656+0000","last_change":"2024-08-17T17:38:31.874623+0000","last_active":"2024-08-17T17:44:29.210656+0000","last_peered":"2024-08-17T17:44:29.210656+0000","last_clean":"2024-08-17T17:44:29.210656+0000","last_became_active":"2024-08-17T17:38:30.339313+0000","last_became_peered":"2024-08-17T17:38:30.339313+0000","last_unstale":"2024-08-17T17:44:29.210656+0000","last_undegraded":"2024-08-17T17:44:29.210656+0000","last_fullsized":"2024-08-17T17:44:29.210656+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:31.874423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:31.874423+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:28:17.980339+0000","scrub_duration":0.0017696579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":307,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.124783+0000","last_change":"2024-08-17T17:33:44.156036+0000","last_active":"2024-08-17T17:44:38.124783+0000","last_peered":"2024-08-17T17:44:38.124783+0000","last_clean":"2024-08-17T17:44:38.124783+0000","last_became_active":"2024-08-17T17:33:44.155758+0000","last_became_peered":"2024-08-17T17:33:44.155758+0000","last_unstale":"2024-08-17T17:44:38.124783+0000","last_undegraded":"2024-08-17T17:44:38.124783+0000","last_fullsized":"2024-08-17T17:44:38.124783+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:23.443405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":446,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.126589+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:44:38.126589+0000","last_peered":"2024-08-17T17:44:38.126589+0000","last_clean":"2024-08-17T17:44:38.126589+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:44:38.126589+0000","last_undegraded":"2024-08-17T17:44:38.126589+0000","last_fullsized":"2024-08-17T17:44:38.126589+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"90'149","reported_seq":843,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.114034+0000","last_change":"2024-08-17T17:41:37.681187+0000","last_active":"2024-08-17T17:44:38.114034+0000","last_peered":"2024-08-17T17:44:38.114034+0000","last_clean":"2024-08-17T17:44:38.114034+0000","last_became_active":"2024-08-17T17:41:37.680605+0000","last_became_peered":"2024-08-17T17:41:37.680605+0000","last_unstale":"2024-08-17T17:44:38.114034+0000","last_undegraded":"2024-08-17T17:44:38.114034+0000","last_fullsized":"2024-08-17T17:44:38.114034+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.844252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.844252+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:13:31.682357+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":446,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.136613+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:44:38.136613+0000","last_peered":"2024-08-17T17:44:38.136613+0000","last_clean":"2024-08-17T17:44:38.136613+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:44:38.136613+0000","last_undegraded":"2024-08-17T17:44:38.136613+0000","last_fullsized":"2024-08-17T17:44:38.136613+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":51,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.208673+0000","last_change":"2024-08-17T17:41:34.611693+0000","last_active":"2024-08-17T17:44:29.208673+0000","last_peered":"2024-08-17T17:44:29.208673+0000","last_clean":"2024-08-17T17:44:29.208673+0000","last_became_active":"2024-08-17T17:41:34.611374+0000","last_became_peered":"2024-08-17T17:41:34.611374+0000","last_unstale":"2024-08-17T17:44:29.208673+0000","last_undegraded":"2024-08-17T17:44:29.208673+0000","last_fullsized":"2024-08-17T17:44:29.208673+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:17:09.897446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"90'149","reported_seq":819,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.115316+0000","last_change":"2024-08-17T17:38:30.446165+0000","last_active":"2024-08-17T17:44:38.115316+0000","last_peered":"2024-08-17T17:44:38.115316+0000","last_clean":"2024-08-17T17:44:38.115316+0000","last_became_active":"2024-08-17T17:38:30.445048+0000","last_became_peered":"2024-08-17T17:38:30.445048+0000","last_unstale":"2024-08-17T17:44:38.115316+0000","last_undegraded":"2024-08-17T17:44:38.115316+0000","last_fullsized":"2024-08-17T17:44:38.115316+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:24.291592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:24.291592+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:17.100154+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":660,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.123478+0000","last_change":"2024-08-17T17:33:20.794682+0000","last_active":"2024-08-17T17:44:38.123478+0000","last_peered":"2024-08-17T17:44:38.123478+0000","last_clean":"2024-08-17T17:44:38.123478+0000","last_became_active":"2024-08-17T17:33:20.794238+0000","last_became_peered":"2024-08-17T17:33:20.794238+0000","last_unstale":"2024-08-17T17:44:38.123478+0000","last_undegraded":"2024-08-17T17:44:38.123478+0000","last_fullsized":"2024-08-17T17:44:38.123478+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:15:55.821920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":472,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.210931+0000","last_change":"2024-08-17T17:43:52.520659+0000","last_active":"2024-08-17T17:44:29.210931+0000","last_peered":"2024-08-17T17:44:29.210931+0000","last_clean":"2024-08-17T17:44:29.210931+0000","last_became_active":"2024-08-17T17:43:52.516799+0000","last_became_peered":"2024-08-17T17:43:52.516799+0000","last_unstale":"2024-08-17T17:44:29.210931+0000","last_undegraded":"2024-08-17T17:44:29.210931+0000","last_fullsized":"2024-08-17T17:44:29.210931+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:25:59.408383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"387'8547","reported_seq":8618,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:40.177424+0000","last_change":"2024-08-17T17:42:36.946078+0000","last_active":"2024-08-17T17:44:40.177424+0000","last_peered":"2024-08-17T17:44:40.177424+0000","last_clean":"2024-08-17T17:44:40.177424+0000","last_became_active":"2024-08-17T17:42:36.945602+0000","last_became_peered":"2024-08-17T17:42:36.945602+0000","last_unstale":"2024-08-17T17:44:40.177424+0000","last_undegraded":"2024-08-17T17:44:40.177424+0000","last_fullsized":"2024-08-17T17:44:40.177424+0000","mapping_epoch":357,"log_start":"369'2000","ondisk_log_start":"369'2000","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6547,"log_dups_size":0,"ondisk_log_size":6547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:18:43.716968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411893760,"num_objects":6285,"num_object_clones":0,"num_object_copies":18855,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8547,"num_write_kb":474624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"90'149","reported_seq":797,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.206474+0000","last_change":"2024-08-17T17:42:16.610940+0000","last_active":"2024-08-17T17:44:29.206474+0000","last_peered":"2024-08-17T17:44:29.206474+0000","last_clean":"2024-08-17T17:44:29.206474+0000","last_became_active":"2024-08-17T17:41:37.670855+0000","last_became_peered":"2024-08-17T17:41:37.670855+0000","last_unstale":"2024-08-17T17:44:29.206474+0000","last_undegraded":"2024-08-17T17:44:29.206474+0000","last_fullsized":"2024-08-17T17:44:29.206474+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:16.610828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:16.610828+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:48:25.918610+0000","scrub_duration":0.086892487000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":348,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.208328+0000","last_change":"2024-08-17T17:41:34.635980+0000","last_active":"2024-08-17T17:44:29.208328+0000","last_peered":"2024-08-17T17:44:29.208328+0000","last_clean":"2024-08-17T17:44:29.208328+0000","last_became_active":"2024-08-17T17:41:34.620596+0000","last_became_peered":"2024-08-17T17:41:34.620596+0000","last_unstale":"2024-08-17T17:44:29.208328+0000","last_undegraded":"2024-08-17T17:44:29.208328+0000","last_fullsized":"2024-08-17T17:44:29.208328+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:19:15.792577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":141,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.210781+0000","last_change":"2024-08-17T17:33:44.145598+0000","last_active":"2024-08-17T17:44:29.210781+0000","last_peered":"2024-08-17T17:44:29.210781+0000","last_clean":"2024-08-17T17:44:29.210781+0000","last_became_active":"2024-08-17T17:33:44.145283+0000","last_became_peered":"2024-08-17T17:33:44.145283+0000","last_unstale":"2024-08-17T17:44:29.210781+0000","last_undegraded":"2024-08-17T17:44:29.210781+0000","last_fullsized":"2024-08-17T17:44:29.210781+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:11.965226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.7","version":"387'8557","reported_seq":11932,"reported_epoch":387,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-17T17:44:38.788642+0000","last_change":"2024-08-17T17:44:30.783759+0000","last_active":"2024-08-17T17:44:38.788642+0000","last_peered":"2024-08-17T17:44:38.788642+0000","last_clean":"2024-08-17T17:43:51.465744+0000","last_became_active":"2024-08-17T17:43:53.682178+0000","last_became_peered":"2024-08-17T17:43:53.682178+0000","last_unstale":"2024-08-17T17:44:38.788642+0000","last_undegraded":"2024-08-17T17:44:30.779387+0000","last_fullsized":"2024-08-17T17:43:53.513318+0000","mapping_epoch":378,"log_start":"369'2100","ondisk_log_start":"369'2100","created":357,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6457,"log_dups_size":0,"ondisk_log_size":6457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:42:59.608872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411631616,"num_objects":6281,"num_object_clones":0,"num_object_copies":18843,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":393,"num_objects_misplaced":4754,"num_objects_unfound":0,"num_objects_dirty":6281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8557,"num_write_kb":474816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1653,"num_bytes_recovered":108331008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3],"avail_no_missing":["6","2","3"],"object_location_counts":[{"shards":"2,3,6","objects":4754},{"shards":"3,6","objects":1134}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"90'149","reported_seq":844,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.123145+0000","last_change":"2024-08-17T17:41:42.603841+0000","last_active":"2024-08-17T17:44:38.123145+0000","last_peered":"2024-08-17T17:44:38.123145+0000","last_clean":"2024-08-17T17:44:38.123145+0000","last_became_active":"2024-08-17T17:41:12.203400+0000","last_became_peered":"2024-08-17T17:41:12.203400+0000","last_unstale":"2024-08-17T17:44:38.123145+0000","last_undegraded":"2024-08-17T17:44:38.123145+0000","last_fullsized":"2024-08-17T17:44:38.123145+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:42.603561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:42.603561+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:50:12.843193+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":344,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.208437+0000","last_change":"2024-08-17T17:41:34.637696+0000","last_active":"2024-08-17T17:44:29.208437+0000","last_peered":"2024-08-17T17:44:29.208437+0000","last_clean":"2024-08-17T17:44:29.208437+0000","last_became_active":"2024-08-17T17:41:34.621388+0000","last_became_peered":"2024-08-17T17:41:34.621388+0000","last_unstale":"2024-08-17T17:44:29.208437+0000","last_undegraded":"2024-08-17T17:44:29.208437+0000","last_fullsized":"2024-08-17T17:44:29.208437+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:13:04.607036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":51,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.208494+0000","last_change":"2024-08-17T17:41:34.621078+0000","last_active":"2024-08-17T17:44:29.208494+0000","last_peered":"2024-08-17T17:44:29.208494+0000","last_clean":"2024-08-17T17:44:29.208494+0000","last_became_active":"2024-08-17T17:41:34.620730+0000","last_became_peered":"2024-08-17T17:41:34.620730+0000","last_unstale":"2024-08-17T17:44:29.208494+0000","last_undegraded":"2024-08-17T17:44:29.208494+0000","last_fullsized":"2024-08-17T17:44:29.208494+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:55:01.312283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"387'8599","reported_seq":8670,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:40.184327+0000","last_change":"2024-08-17T17:42:36.620130+0000","last_active":"2024-08-17T17:44:40.184327+0000","last_peered":"2024-08-17T17:44:40.184327+0000","last_clean":"2024-08-17T17:44:40.184327+0000","last_became_active":"2024-08-17T17:42:36.619813+0000","last_became_peered":"2024-08-17T17:42:36.619813+0000","last_unstale":"2024-08-17T17:44:40.184327+0000","last_undegraded":"2024-08-17T17:44:40.184327+0000","last_fullsized":"2024-08-17T17:44:40.184327+0000","mapping_epoch":357,"log_start":"369'2000","ondisk_log_start":"369'2000","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6599,"log_dups_size":0,"ondisk_log_size":6599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:39:22.366905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414908416,"num_objects":6331,"num_object_clones":0,"num_object_copies":18993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8599,"num_write_kb":477760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"90'149","reported_seq":563,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.215651+0000","last_change":"2024-08-17T17:41:37.668050+0000","last_active":"2024-08-17T17:44:29.215651+0000","last_peered":"2024-08-17T17:44:29.215651+0000","last_clean":"2024-08-17T17:44:29.215651+0000","last_became_active":"2024-08-17T17:41:37.667655+0000","last_became_peered":"2024-08-17T17:41:37.667655+0000","last_unstale":"2024-08-17T17:44:29.215651+0000","last_undegraded":"2024-08-17T17:44:29.215651+0000","last_fullsized":"2024-08-17T17:44:29.215651+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:32.843802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:32.843802+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:39:46.766280+0000","scrub_duration":0.0040803669999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":137,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.211570+0000","last_change":"2024-08-17T17:33:44.145399+0000","last_active":"2024-08-17T17:44:29.211570+0000","last_peered":"2024-08-17T17:44:29.211570+0000","last_clean":"2024-08-17T17:44:29.211570+0000","last_became_active":"2024-08-17T17:33:44.144751+0000","last_became_peered":"2024-08-17T17:33:44.144751+0000","last_unstale":"2024-08-17T17:44:29.211570+0000","last_undegraded":"2024-08-17T17:44:29.211570+0000","last_fullsized":"2024-08-17T17:44:29.211570+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:54.276350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":215,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.211620+0000","last_change":"2024-08-17T17:33:20.795582+0000","last_active":"2024-08-17T17:44:29.211620+0000","last_peered":"2024-08-17T17:44:29.211620+0000","last_clean":"2024-08-17T17:44:29.211620+0000","last_became_active":"2024-08-17T17:33:20.795149+0000","last_became_peered":"2024-08-17T17:33:20.795149+0000","last_unstale":"2024-08-17T17:44:29.211620+0000","last_undegraded":"2024-08-17T17:44:29.211620+0000","last_fullsized":"2024-08-17T17:44:29.211620+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:14.020923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"387'8531","reported_seq":8601,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:40.204797+0000","last_change":"2024-08-17T17:42:36.929599+0000","last_active":"2024-08-17T17:44:40.204797+0000","last_peered":"2024-08-17T17:44:40.204797+0000","last_clean":"2024-08-17T17:44:40.204797+0000","last_became_active":"2024-08-17T17:42:36.929274+0000","last_became_peered":"2024-08-17T17:42:36.929274+0000","last_unstale":"2024-08-17T17:44:40.204797+0000","last_undegraded":"2024-08-17T17:44:40.204797+0000","last_fullsized":"2024-08-17T17:44:40.204797+0000","mapping_epoch":357,"log_start":"369'2000","ondisk_log_start":"369'2000","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6531,"log_dups_size":0,"ondisk_log_size":6531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:29:35.955743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414515200,"num_objects":6325,"num_object_clones":0,"num_object_copies":18975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8531,"num_write_kb":475392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"90'149","reported_seq":695,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.209582+0000","last_change":"2024-08-17T17:42:20.927007+0000","last_active":"2024-08-17T17:44:29.209582+0000","last_peered":"2024-08-17T17:44:29.209582+0000","last_clean":"2024-08-17T17:44:29.209582+0000","last_became_active":"2024-08-17T17:41:37.670914+0000","last_became_peered":"2024-08-17T17:41:37.670914+0000","last_unstale":"2024-08-17T17:44:29.209582+0000","last_undegraded":"2024-08-17T17:44:29.209582+0000","last_fullsized":"2024-08-17T17:44:29.209582+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:20.926924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:20.926924+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:57:44.156600+0000","scrub_duration":0.0018485109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":304,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.123432+0000","last_change":"2024-08-17T17:33:44.154097+0000","last_active":"2024-08-17T17:44:38.123432+0000","last_peered":"2024-08-17T17:44:38.123432+0000","last_clean":"2024-08-17T17:44:38.123432+0000","last_became_active":"2024-08-17T17:33:44.152972+0000","last_became_peered":"2024-08-17T17:33:44.152972+0000","last_unstale":"2024-08-17T17:44:38.123432+0000","last_undegraded":"2024-08-17T17:44:38.123432+0000","last_fullsized":"2024-08-17T17:44:38.123432+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:42.602017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":668,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.131133+0000","last_change":"2024-08-17T17:41:34.615197+0000","last_active":"2024-08-17T17:44:38.131133+0000","last_peered":"2024-08-17T17:44:38.131133+0000","last_clean":"2024-08-17T17:44:38.131133+0000","last_became_active":"2024-08-17T17:41:34.614873+0000","last_became_peered":"2024-08-17T17:41:34.614873+0000","last_unstale":"2024-08-17T17:44:38.131133+0000","last_undegraded":"2024-08-17T17:44:38.131133+0000","last_fullsized":"2024-08-17T17:44:38.131133+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:21:57.391953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2","version":"387'8402","reported_seq":8473,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:40.961489+0000","last_change":"2024-08-17T17:42:36.618260+0000","last_active":"2024-08-17T17:44:40.961489+0000","last_peered":"2024-08-17T17:44:40.961489+0000","last_clean":"2024-08-17T17:44:40.961489+0000","last_became_active":"2024-08-17T17:42:36.617875+0000","last_became_peered":"2024-08-17T17:42:36.617875+0000","last_unstale":"2024-08-17T17:44:40.961489+0000","last_undegraded":"2024-08-17T17:44:40.961489+0000","last_fullsized":"2024-08-17T17:44:40.961489+0000","mapping_epoch":357,"log_start":"367'1400","ondisk_log_start":"367'1400","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":7002,"log_dups_size":0,"ondisk_log_size":7002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:09:07.102359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399245312,"num_objects":6092,"num_object_clones":0,"num_object_copies":18276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8402,"num_write_kb":463808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":819,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.120619+0000","last_change":"2024-08-17T17:41:44.585386+0000","last_active":"2024-08-17T17:44:38.120619+0000","last_peered":"2024-08-17T17:44:38.120619+0000","last_clean":"2024-08-17T17:44:38.120619+0000","last_became_active":"2024-08-17T17:41:12.307398+0000","last_became_peered":"2024-08-17T17:41:12.307398+0000","last_unstale":"2024-08-17T17:44:38.120619+0000","last_undegraded":"2024-08-17T17:44:38.120619+0000","last_fullsized":"2024-08-17T17:44:38.120619+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:44.583653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:44.583653+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:27:44.756681+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":704,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.120847+0000","last_change":"2024-08-17T17:41:34.630434+0000","last_active":"2024-08-17T17:44:38.120847+0000","last_peered":"2024-08-17T17:44:38.120847+0000","last_clean":"2024-08-17T17:44:38.120847+0000","last_became_active":"2024-08-17T17:41:34.616406+0000","last_became_peered":"2024-08-17T17:41:34.616406+0000","last_unstale":"2024-08-17T17:44:38.120847+0000","last_undegraded":"2024-08-17T17:44:38.120847+0000","last_fullsized":"2024-08-17T17:44:38.120847+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:30:44.817203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":51,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.208711+0000","last_change":"2024-08-17T17:41:34.611049+0000","last_active":"2024-08-17T17:44:29.208711+0000","last_peered":"2024-08-17T17:44:29.208711+0000","last_clean":"2024-08-17T17:44:29.208711+0000","last_became_active":"2024-08-17T17:41:34.610727+0000","last_became_peered":"2024-08-17T17:41:34.610727+0000","last_unstale":"2024-08-17T17:44:29.208711+0000","last_undegraded":"2024-08-17T17:44:29.208711+0000","last_fullsized":"2024-08-17T17:44:29.208711+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:04:05.477660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"387'8642","reported_seq":8713,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:39.572711+0000","last_change":"2024-08-17T17:42:36.946226+0000","last_active":"2024-08-17T17:44:39.572711+0000","last_peered":"2024-08-17T17:44:39.572711+0000","last_clean":"2024-08-17T17:44:39.572711+0000","last_became_active":"2024-08-17T17:42:36.945809+0000","last_became_peered":"2024-08-17T17:42:36.945809+0000","last_unstale":"2024-08-17T17:44:39.572711+0000","last_undegraded":"2024-08-17T17:44:39.572711+0000","last_fullsized":"2024-08-17T17:44:39.572711+0000","mapping_epoch":357,"log_start":"369'2300","ondisk_log_start":"369'2300","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6342,"log_dups_size":0,"ondisk_log_size":6342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:04:58.633957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418512896,"num_objects":6386,"num_object_clones":0,"num_object_copies":19158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8642,"num_write_kb":480896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":847,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.120298+0000","last_change":"2024-08-17T17:43:55.595647+0000","last_active":"2024-08-17T17:44:38.120298+0000","last_peered":"2024-08-17T17:44:38.120298+0000","last_clean":"2024-08-17T17:44:38.120298+0000","last_became_active":"2024-08-17T17:43:55.595256+0000","last_became_peered":"2024-08-17T17:43:55.595256+0000","last_unstale":"2024-08-17T17:44:38.120298+0000","last_undegraded":"2024-08-17T17:44:38.120298+0000","last_fullsized":"2024-08-17T17:44:38.120298+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:46.587712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:46.587712+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:05:01.927968+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":51,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.208149+0000","last_change":"2024-08-17T17:41:34.622803+0000","last_active":"2024-08-17T17:44:29.208149+0000","last_peered":"2024-08-17T17:44:29.208149+0000","last_clean":"2024-08-17T17:44:29.208149+0000","last_became_active":"2024-08-17T17:41:34.622525+0000","last_became_peered":"2024-08-17T17:41:34.622525+0000","last_unstale":"2024-08-17T17:44:29.208149+0000","last_undegraded":"2024-08-17T17:44:29.208149+0000","last_fullsized":"2024-08-17T17:44:29.208149+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:42:39.728377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":51,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.209129+0000","last_change":"2024-08-17T17:41:34.619268+0000","last_active":"2024-08-17T17:44:29.209129+0000","last_peered":"2024-08-17T17:44:29.209129+0000","last_clean":"2024-08-17T17:44:29.209129+0000","last_became_active":"2024-08-17T17:41:34.618914+0000","last_became_peered":"2024-08-17T17:41:34.618914+0000","last_unstale":"2024-08-17T17:44:29.209129+0000","last_undegraded":"2024-08-17T17:44:29.209129+0000","last_fullsized":"2024-08-17T17:44:29.209129+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:08:54.248386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.0","version":"387'8218","reported_seq":8256,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.734883+0000","last_change":"2024-08-17T17:42:36.617537+0000","last_active":"2024-08-17T17:44:38.734883+0000","last_peered":"2024-08-17T17:44:38.734883+0000","last_clean":"2024-08-17T17:44:38.734883+0000","last_became_active":"2024-08-17T17:42:36.617201+0000","last_became_peered":"2024-08-17T17:42:36.617201+0000","last_unstale":"2024-08-17T17:44:38.734883+0000","last_undegraded":"2024-08-17T17:44:38.734883+0000","last_fullsized":"2024-08-17T17:44:38.734883+0000","mapping_epoch":357,"log_start":"369'1800","ondisk_log_start":"369'1800","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6418,"log_dups_size":0,"ondisk_log_size":6418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:51:58.927214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402915328,"num_objects":6148,"num_object_clones":0,"num_object_copies":18444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8218,"num_write_kb":459712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"90'149","reported_seq":852,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.114263+0000","last_change":"2024-08-17T17:41:41.743931+0000","last_active":"2024-08-17T17:44:38.114263+0000","last_peered":"2024-08-17T17:44:38.114263+0000","last_clean":"2024-08-17T17:44:38.114263+0000","last_became_active":"2024-08-17T17:41:12.203771+0000","last_became_peered":"2024-08-17T17:41:12.203771+0000","last_unstale":"2024-08-17T17:44:38.114263+0000","last_undegraded":"2024-08-17T17:44:38.114263+0000","last_fullsized":"2024-08-17T17:44:38.114263+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:41.743570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:41.743570+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:21:03.865738+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":51,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.208724+0000","last_change":"2024-08-17T17:41:34.607602+0000","last_active":"2024-08-17T17:44:29.208724+0000","last_peered":"2024-08-17T17:44:29.208724+0000","last_clean":"2024-08-17T17:44:29.208724+0000","last_became_active":"2024-08-17T17:41:34.607300+0000","last_became_peered":"2024-08-17T17:41:34.607300+0000","last_unstale":"2024-08-17T17:44:29.208724+0000","last_undegraded":"2024-08-17T17:44:29.208724+0000","last_fullsized":"2024-08-17T17:44:29.208724+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:20:31.486698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":51,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.209447+0000","last_change":"2024-08-17T17:41:34.622102+0000","last_active":"2024-08-17T17:44:29.209447+0000","last_peered":"2024-08-17T17:44:29.209447+0000","last_clean":"2024-08-17T17:44:29.209447+0000","last_became_active":"2024-08-17T17:41:34.621733+0000","last_became_peered":"2024-08-17T17:41:34.621733+0000","last_unstale":"2024-08-17T17:44:29.209447+0000","last_undegraded":"2024-08-17T17:44:29.209447+0000","last_fullsized":"2024-08-17T17:44:29.209447+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:39:41.668709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"387'8624","reported_seq":8660,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.690147+0000","last_change":"2024-08-17T17:42:36.620155+0000","last_active":"2024-08-17T17:44:38.690147+0000","last_peered":"2024-08-17T17:44:38.690147+0000","last_clean":"2024-08-17T17:44:38.690147+0000","last_became_active":"2024-08-17T17:42:36.619813+0000","last_became_peered":"2024-08-17T17:42:36.619813+0000","last_unstale":"2024-08-17T17:44:38.690147+0000","last_undegraded":"2024-08-17T17:44:38.690147+0000","last_fullsized":"2024-08-17T17:44:38.690147+0000","mapping_epoch":357,"log_start":"370'3100","ondisk_log_start":"370'3100","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":5524,"log_dups_size":0,"ondisk_log_size":5524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:16:27.534439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":424542208,"num_objects":6478,"num_object_clones":0,"num_object_copies":19434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8624,"num_write_kb":483264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":504,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.211155+0000","last_change":"2024-08-17T17:41:41.036615+0000","last_active":"2024-08-17T17:44:29.211155+0000","last_peered":"2024-08-17T17:44:29.211155+0000","last_clean":"2024-08-17T17:44:29.211155+0000","last_became_active":"2024-08-17T17:41:12.329214+0000","last_became_peered":"2024-08-17T17:41:12.329214+0000","last_unstale":"2024-08-17T17:44:29.211155+0000","last_undegraded":"2024-08-17T17:44:29.211155+0000","last_fullsized":"2024-08-17T17:44:29.211155+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:41.036524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:41.036524+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:59:58.695900+0000","scrub_duration":0.001820803,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":662,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.120416+0000","last_change":"2024-08-17T17:33:20.796216+0000","last_active":"2024-08-17T17:44:38.120416+0000","last_peered":"2024-08-17T17:44:38.120416+0000","last_clean":"2024-08-17T17:44:38.120416+0000","last_became_active":"2024-08-17T17:33:20.795762+0000","last_became_peered":"2024-08-17T17:33:20.795762+0000","last_unstale":"2024-08-17T17:44:38.120416+0000","last_undegraded":"2024-08-17T17:44:38.120416+0000","last_fullsized":"2024-08-17T17:44:38.120416+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:47.038509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":557,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.109316+0000","last_change":"2024-08-17T17:41:34.625551+0000","last_active":"2024-08-17T17:44:38.109316+0000","last_peered":"2024-08-17T17:44:38.109316+0000","last_clean":"2024-08-17T17:44:38.109316+0000","last_became_active":"2024-08-17T17:41:34.625251+0000","last_became_peered":"2024-08-17T17:41:34.625251+0000","last_unstale":"2024-08-17T17:44:38.109316+0000","last_undegraded":"2024-08-17T17:44:38.109316+0000","last_fullsized":"2024-08-17T17:44:38.109316+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:28:22.354213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"387'8468","reported_seq":8506,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.795570+0000","last_change":"2024-08-17T17:42:36.619525+0000","last_active":"2024-08-17T17:44:38.795570+0000","last_peered":"2024-08-17T17:44:38.795570+0000","last_clean":"2024-08-17T17:44:38.795570+0000","last_became_active":"2024-08-17T17:42:36.619074+0000","last_became_peered":"2024-08-17T17:42:36.619074+0000","last_unstale":"2024-08-17T17:44:38.795570+0000","last_undegraded":"2024-08-17T17:44:38.795570+0000","last_fullsized":"2024-08-17T17:44:38.795570+0000","mapping_epoch":357,"log_start":"369'2000","ondisk_log_start":"369'2000","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6468,"log_dups_size":0,"ondisk_log_size":6468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:11:32.849616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417857536,"num_objects":6376,"num_object_clones":0,"num_object_copies":19128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8468,"num_write_kb":475008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":799,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.126027+0000","last_change":"2024-08-17T17:38:25.247159+0000","last_active":"2024-08-17T17:44:38.126027+0000","last_peered":"2024-08-17T17:44:38.126027+0000","last_clean":"2024-08-17T17:44:38.126027+0000","last_became_active":"2024-08-17T17:38:21.311361+0000","last_became_peered":"2024-08-17T17:38:21.311361+0000","last_unstale":"2024-08-17T17:44:38.126027+0000","last_undegraded":"2024-08-17T17:44:38.126027+0000","last_fullsized":"2024-08-17T17:44:38.126027+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:25.247027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:25.247027+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:46:21.734712+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":793,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.206166+0000","last_change":"2024-08-17T17:33:44.149928+0000","last_active":"2024-08-17T17:44:29.206166+0000","last_peered":"2024-08-17T17:44:29.206166+0000","last_clean":"2024-08-17T17:44:29.206166+0000","last_became_active":"2024-08-17T17:33:44.149498+0000","last_became_peered":"2024-08-17T17:33:44.149498+0000","last_unstale":"2024-08-17T17:44:29.206166+0000","last_undegraded":"2024-08-17T17:44:29.206166+0000","last_fullsized":"2024-08-17T17:44:29.206166+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:17:04.963202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":610,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.121186+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:44:38.121186+0000","last_peered":"2024-08-17T17:44:38.121186+0000","last_clean":"2024-08-17T17:44:38.121186+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:44:38.121186+0000","last_undegraded":"2024-08-17T17:44:38.121186+0000","last_fullsized":"2024-08-17T17:44:38.121186+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":228,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.211418+0000","last_change":"2024-08-17T17:41:34.618631+0000","last_active":"2024-08-17T17:44:29.211418+0000","last_peered":"2024-08-17T17:44:29.211418+0000","last_clean":"2024-08-17T17:44:29.211418+0000","last_became_active":"2024-08-17T17:41:34.618313+0000","last_became_peered":"2024-08-17T17:41:34.618313+0000","last_unstale":"2024-08-17T17:44:29.211418+0000","last_undegraded":"2024-08-17T17:44:29.211418+0000","last_fullsized":"2024-08-17T17:44:29.211418+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:07:13.915088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"387'8489","reported_seq":8560,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:40.141811+0000","last_change":"2024-08-17T17:42:36.929502+0000","last_active":"2024-08-17T17:44:40.141811+0000","last_peered":"2024-08-17T17:44:40.141811+0000","last_clean":"2024-08-17T17:44:40.141811+0000","last_became_active":"2024-08-17T17:42:36.929113+0000","last_became_peered":"2024-08-17T17:42:36.929113+0000","last_unstale":"2024-08-17T17:44:40.141811+0000","last_undegraded":"2024-08-17T17:44:40.141811+0000","last_fullsized":"2024-08-17T17:44:40.141811+0000","mapping_epoch":357,"log_start":"369'1900","ondisk_log_start":"369'1900","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6589,"log_dups_size":0,"ondisk_log_size":6589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:31:25.879221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403636224,"num_objects":6159,"num_object_clones":0,"num_object_copies":18477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8489,"num_write_kb":468736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":835,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.114439+0000","last_change":"2024-08-17T17:41:37.676347+0000","last_active":"2024-08-17T17:44:38.114439+0000","last_peered":"2024-08-17T17:44:38.114439+0000","last_clean":"2024-08-17T17:44:38.114439+0000","last_became_active":"2024-08-17T17:41:37.675869+0000","last_became_peered":"2024-08-17T17:41:37.675869+0000","last_unstale":"2024-08-17T17:44:38.114439+0000","last_undegraded":"2024-08-17T17:44:38.114439+0000","last_fullsized":"2024-08-17T17:44:38.114439+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:21.814077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:21.814077+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:04:16.240754+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":893,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.107641+0000","last_change":"2024-08-17T17:33:20.797705+0000","last_active":"2024-08-17T17:44:38.107641+0000","last_peered":"2024-08-17T17:44:38.107641+0000","last_clean":"2024-08-17T17:44:38.107641+0000","last_became_active":"2024-08-17T17:33:20.797172+0000","last_became_peered":"2024-08-17T17:33:20.797172+0000","last_unstale":"2024-08-17T17:44:38.107641+0000","last_undegraded":"2024-08-17T17:44:38.107641+0000","last_fullsized":"2024-08-17T17:44:38.107641+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:30.754999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":659,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.126178+0000","last_change":"2024-08-17T17:41:34.614108+0000","last_active":"2024-08-17T17:44:38.126178+0000","last_peered":"2024-08-17T17:44:38.126178+0000","last_clean":"2024-08-17T17:44:38.126178+0000","last_became_active":"2024-08-17T17:41:34.613481+0000","last_became_peered":"2024-08-17T17:41:34.613481+0000","last_unstale":"2024-08-17T17:44:38.126178+0000","last_undegraded":"2024-08-17T17:44:38.126178+0000","last_fullsized":"2024-08-17T17:44:38.126178+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:21:02.346115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":447,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.137281+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:44:38.137281+0000","last_peered":"2024-08-17T17:44:38.137281+0000","last_clean":"2024-08-17T17:44:38.137281+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:44:38.137281+0000","last_undegraded":"2024-08-17T17:44:38.137281+0000","last_fullsized":"2024-08-17T17:44:38.137281+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"387'8377","reported_seq":8415,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.639247+0000","last_change":"2024-08-17T17:42:36.946382+0000","last_active":"2024-08-17T17:44:38.639247+0000","last_peered":"2024-08-17T17:44:38.639247+0000","last_clean":"2024-08-17T17:44:38.639247+0000","last_became_active":"2024-08-17T17:42:36.945591+0000","last_became_peered":"2024-08-17T17:42:36.945591+0000","last_unstale":"2024-08-17T17:44:38.639247+0000","last_undegraded":"2024-08-17T17:44:38.639247+0000","last_fullsized":"2024-08-17T17:44:38.639247+0000","mapping_epoch":357,"log_start":"369'2100","ondisk_log_start":"369'2100","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6277,"log_dups_size":0,"ondisk_log_size":6277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:08:17.912891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418447360,"num_objects":6385,"num_object_clones":0,"num_object_copies":19155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8377,"num_write_kb":472384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":838,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.126491+0000","last_change":"2024-08-17T17:41:37.683450+0000","last_active":"2024-08-17T17:44:38.126491+0000","last_peered":"2024-08-17T17:44:38.126491+0000","last_clean":"2024-08-17T17:44:38.126491+0000","last_became_active":"2024-08-17T17:41:37.683103+0000","last_became_peered":"2024-08-17T17:41:37.683103+0000","last_unstale":"2024-08-17T17:44:38.126491+0000","last_undegraded":"2024-08-17T17:44:38.126491+0000","last_fullsized":"2024-08-17T17:44:38.126491+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.289528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.289528+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:39:17.141469+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":1026,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.120986+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:44:38.120986+0000","last_peered":"2024-08-17T17:44:38.120986+0000","last_clean":"2024-08-17T17:44:38.120986+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:44:38.120986+0000","last_undegraded":"2024-08-17T17:44:38.120986+0000","last_fullsized":"2024-08-17T17:44:38.120986+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":531,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.107951+0000","last_change":"2024-08-17T17:33:20.793468+0000","last_active":"2024-08-17T17:44:38.107951+0000","last_peered":"2024-08-17T17:44:38.107951+0000","last_clean":"2024-08-17T17:44:38.107951+0000","last_became_active":"2024-08-17T17:33:20.793166+0000","last_became_peered":"2024-08-17T17:33:20.793166+0000","last_unstale":"2024-08-17T17:44:38.107951+0000","last_undegraded":"2024-08-17T17:44:38.107951+0000","last_fullsized":"2024-08-17T17:44:38.107951+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:12:52.121050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":567,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.109238+0000","last_change":"2024-08-17T17:33:20.835784+0000","last_active":"2024-08-17T17:44:38.109238+0000","last_peered":"2024-08-17T17:44:38.109238+0000","last_clean":"2024-08-17T17:44:38.109238+0000","last_became_active":"2024-08-17T17:33:20.835371+0000","last_became_peered":"2024-08-17T17:33:20.835371+0000","last_unstale":"2024-08-17T17:44:38.109238+0000","last_undegraded":"2024-08-17T17:44:38.109238+0000","last_fullsized":"2024-08-17T17:44:38.109238+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:35:43.369055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"387'8448","reported_seq":8486,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.730708+0000","last_change":"2024-08-17T17:42:36.946026+0000","last_active":"2024-08-17T17:44:38.730708+0000","last_peered":"2024-08-17T17:44:38.730708+0000","last_clean":"2024-08-17T17:44:38.730708+0000","last_became_active":"2024-08-17T17:42:36.945638+0000","last_became_peered":"2024-08-17T17:42:36.945638+0000","last_unstale":"2024-08-17T17:44:38.730708+0000","last_undegraded":"2024-08-17T17:44:38.730708+0000","last_fullsized":"2024-08-17T17:44:38.730708+0000","mapping_epoch":357,"log_start":"369'2000","ondisk_log_start":"369'2000","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6448,"log_dups_size":0,"ondisk_log_size":6448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:07:34.864262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411959296,"num_objects":6286,"num_object_clones":0,"num_object_copies":18858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8448,"num_write_kb":471488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":817,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.126075+0000","last_change":"2024-08-17T17:38:22.254841+0000","last_active":"2024-08-17T17:44:38.126075+0000","last_peered":"2024-08-17T17:44:38.126075+0000","last_clean":"2024-08-17T17:44:38.126075+0000","last_became_active":"2024-08-17T17:38:21.306619+0000","last_became_peered":"2024-08-17T17:38:21.306619+0000","last_unstale":"2024-08-17T17:44:38.126075+0000","last_undegraded":"2024-08-17T17:44:38.126075+0000","last_fullsized":"2024-08-17T17:44:38.126075+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:22.254705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:22.254705+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:47:38.184089+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":1116,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.114974+0000","last_change":"2024-08-17T17:33:44.148227+0000","last_active":"2024-08-17T17:44:38.114974+0000","last_peered":"2024-08-17T17:44:38.114974+0000","last_clean":"2024-08-17T17:44:38.114974+0000","last_became_active":"2024-08-17T17:33:44.147627+0000","last_became_peered":"2024-08-17T17:33:44.147627+0000","last_unstale":"2024-08-17T17:44:38.114974+0000","last_undegraded":"2024-08-17T17:44:38.114974+0000","last_fullsized":"2024-08-17T17:44:38.114974+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:29:42.898460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1372,"num_read_kb":2843,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":51,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.208418+0000","last_change":"2024-08-17T17:41:34.611109+0000","last_active":"2024-08-17T17:44:29.208418+0000","last_peered":"2024-08-17T17:44:29.208418+0000","last_clean":"2024-08-17T17:44:29.208418+0000","last_became_active":"2024-08-17T17:41:34.610170+0000","last_became_peered":"2024-08-17T17:41:34.610170+0000","last_unstale":"2024-08-17T17:44:29.208418+0000","last_undegraded":"2024-08-17T17:44:29.208418+0000","last_fullsized":"2024-08-17T17:44:29.208418+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:39:05.301260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":571,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.108578+0000","last_change":"2024-08-17T17:33:20.772454+0000","last_active":"2024-08-17T17:44:38.108578+0000","last_peered":"2024-08-17T17:44:38.108578+0000","last_clean":"2024-08-17T17:44:38.108578+0000","last_became_active":"2024-08-17T17:33:20.771848+0000","last_became_peered":"2024-08-17T17:33:20.771848+0000","last_unstale":"2024-08-17T17:44:38.108578+0000","last_undegraded":"2024-08-17T17:44:38.108578+0000","last_fullsized":"2024-08-17T17:44:38.108578+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:46:42.300744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"387'8684","reported_seq":8722,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.637263+0000","last_change":"2024-08-17T17:42:36.618414+0000","last_active":"2024-08-17T17:44:38.637263+0000","last_peered":"2024-08-17T17:44:38.637263+0000","last_clean":"2024-08-17T17:44:38.637263+0000","last_became_active":"2024-08-17T17:42:36.618086+0000","last_became_peered":"2024-08-17T17:42:36.618086+0000","last_unstale":"2024-08-17T17:44:38.637263+0000","last_undegraded":"2024-08-17T17:44:38.637263+0000","last_fullsized":"2024-08-17T17:44:38.637263+0000","mapping_epoch":357,"log_start":"369'2400","ondisk_log_start":"369'2400","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6284,"log_dups_size":0,"ondisk_log_size":6284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:26:27.772894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426115072,"num_objects":6502,"num_object_clones":0,"num_object_copies":19506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8684,"num_write_kb":485952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":524,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.215570+0000","last_change":"2024-08-17T17:41:43.035215+0000","last_active":"2024-08-17T17:44:29.215570+0000","last_peered":"2024-08-17T17:44:29.215570+0000","last_clean":"2024-08-17T17:44:29.215570+0000","last_became_active":"2024-08-17T17:41:12.201158+0000","last_became_peered":"2024-08-17T17:41:12.201158+0000","last_unstale":"2024-08-17T17:44:29.215570+0000","last_undegraded":"2024-08-17T17:44:29.215570+0000","last_fullsized":"2024-08-17T17:44:29.215570+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:43.035108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:43.035108+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:43:27.108994+0000","scrub_duration":0.0028877400000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":322,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.124991+0000","last_change":"2024-08-17T17:33:44.163867+0000","last_active":"2024-08-17T17:44:38.124991+0000","last_peered":"2024-08-17T17:44:38.124991+0000","last_clean":"2024-08-17T17:44:38.124991+0000","last_became_active":"2024-08-17T17:33:44.163503+0000","last_became_peered":"2024-08-17T17:33:44.163503+0000","last_unstale":"2024-08-17T17:44:38.124991+0000","last_undegraded":"2024-08-17T17:44:38.124991+0000","last_fullsized":"2024-08-17T17:44:38.124991+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:31:53.573596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":786,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.108326+0000","last_change":"2024-08-17T17:33:20.799350+0000","last_active":"2024-08-17T17:44:38.108326+0000","last_peered":"2024-08-17T17:44:38.108326+0000","last_clean":"2024-08-17T17:44:38.108326+0000","last_became_active":"2024-08-17T17:33:20.798925+0000","last_became_peered":"2024-08-17T17:33:20.798925+0000","last_unstale":"2024-08-17T17:44:38.108326+0000","last_undegraded":"2024-08-17T17:44:38.108326+0000","last_fullsized":"2024-08-17T17:44:38.108326+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:16:36.121295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":602,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.115432+0000","last_change":"2024-08-17T17:41:34.627064+0000","last_active":"2024-08-17T17:44:38.115432+0000","last_peered":"2024-08-17T17:44:38.115432+0000","last_clean":"2024-08-17T17:44:38.115432+0000","last_became_active":"2024-08-17T17:41:34.626453+0000","last_became_peered":"2024-08-17T17:41:34.626453+0000","last_unstale":"2024-08-17T17:44:38.115432+0000","last_undegraded":"2024-08-17T17:44:38.115432+0000","last_fullsized":"2024-08-17T17:44:38.115432+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:02:11.894042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"387'8526","reported_seq":8565,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.787093+0000","last_change":"2024-08-17T17:42:36.620669+0000","last_active":"2024-08-17T17:44:38.787093+0000","last_peered":"2024-08-17T17:44:38.787093+0000","last_clean":"2024-08-17T17:44:38.787093+0000","last_became_active":"2024-08-17T17:42:36.619876+0000","last_became_peered":"2024-08-17T17:42:36.619876+0000","last_unstale":"2024-08-17T17:44:38.787093+0000","last_undegraded":"2024-08-17T17:44:38.787093+0000","last_fullsized":"2024-08-17T17:44:38.787093+0000","mapping_epoch":357,"log_start":"369'2100","ondisk_log_start":"369'2100","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6426,"log_dups_size":0,"ondisk_log_size":6426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:59:34.661001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415957016,"num_objects":6348,"num_object_clones":0,"num_object_copies":19044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6348,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8526,"num_write_kb":475905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":524,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.211710+0000","last_change":"2024-08-17T17:41:44.058291+0000","last_active":"2024-08-17T17:44:29.211710+0000","last_peered":"2024-08-17T17:44:29.211710+0000","last_clean":"2024-08-17T17:44:29.211710+0000","last_became_active":"2024-08-17T17:41:12.200077+0000","last_became_peered":"2024-08-17T17:41:12.200077+0000","last_unstale":"2024-08-17T17:44:29.211710+0000","last_undegraded":"2024-08-17T17:44:29.211710+0000","last_fullsized":"2024-08-17T17:44:29.211710+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:44.058233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:44.058233+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:42:55.911623+0000","scrub_duration":0.0074268499999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":606,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.108219+0000","last_change":"2024-08-17T17:33:20.798613+0000","last_active":"2024-08-17T17:44:38.108219+0000","last_peered":"2024-08-17T17:44:38.108219+0000","last_clean":"2024-08-17T17:44:38.108219+0000","last_became_active":"2024-08-17T17:33:20.798260+0000","last_became_peered":"2024-08-17T17:33:20.798260+0000","last_unstale":"2024-08-17T17:44:38.108219+0000","last_undegraded":"2024-08-17T17:44:38.108219+0000","last_fullsized":"2024-08-17T17:44:38.108219+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:18:00.822681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":947,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.206856+0000","last_change":"2024-08-17T17:41:37.683212+0000","last_active":"2024-08-17T17:44:29.206856+0000","last_peered":"2024-08-17T17:44:29.206856+0000","last_clean":"2024-08-17T17:44:29.206856+0000","last_became_active":"2024-08-17T17:41:37.682730+0000","last_became_peered":"2024-08-17T17:41:37.682730+0000","last_unstale":"2024-08-17T17:44:29.206856+0000","last_undegraded":"2024-08-17T17:44:29.206856+0000","last_fullsized":"2024-08-17T17:44:29.206856+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:51:35.444433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":621,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.131050+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:44:38.131050+0000","last_peered":"2024-08-17T17:44:38.131050+0000","last_clean":"2024-08-17T17:44:38.131050+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:44:38.131050+0000","last_undegraded":"2024-08-17T17:44:38.131050+0000","last_fullsized":"2024-08-17T17:44:38.131050+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"387'8530","reported_seq":8566,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.693721+0000","last_change":"2024-08-17T17:42:36.929709+0000","last_active":"2024-08-17T17:44:38.693721+0000","last_peered":"2024-08-17T17:44:38.693721+0000","last_clean":"2024-08-17T17:44:38.693721+0000","last_became_active":"2024-08-17T17:42:36.929285+0000","last_became_peered":"2024-08-17T17:42:36.929285+0000","last_unstale":"2024-08-17T17:44:38.693721+0000","last_undegraded":"2024-08-17T17:44:38.693721+0000","last_fullsized":"2024-08-17T17:44:38.693721+0000","mapping_epoch":357,"log_start":"370'3000","ondisk_log_start":"370'3000","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":5530,"log_dups_size":0,"ondisk_log_size":5530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:47:39.760414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415760384,"num_objects":6344,"num_object_clones":0,"num_object_copies":19032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8530,"num_write_kb":475968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":770,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.208912+0000","last_change":"2024-08-17T17:42:19.960161+0000","last_active":"2024-08-17T17:44:29.208912+0000","last_peered":"2024-08-17T17:44:29.208912+0000","last_clean":"2024-08-17T17:44:29.208912+0000","last_became_active":"2024-08-17T17:41:37.665223+0000","last_became_peered":"2024-08-17T17:41:37.665223+0000","last_unstale":"2024-08-17T17:44:29.208912+0000","last_undegraded":"2024-08-17T17:44:29.208912+0000","last_fullsized":"2024-08-17T17:44:29.208912+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:19.960086+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:19.960086+0000","last_clean_scrub_stamp":"2024-08-17T17:42:19.960086+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:57.861661+0000","scrub_duration":0.0022612000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":656,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.123672+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:44:38.123672+0000","last_peered":"2024-08-17T17:44:38.123672+0000","last_clean":"2024-08-17T17:44:38.123672+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:44:38.123672+0000","last_undegraded":"2024-08-17T17:44:38.123672+0000","last_fullsized":"2024-08-17T17:44:38.123672+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":1231,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.108950+0000","last_change":"2024-08-17T17:41:37.677524+0000","last_active":"2024-08-17T17:44:38.108950+0000","last_peered":"2024-08-17T17:44:38.108950+0000","last_clean":"2024-08-17T17:44:38.108950+0000","last_became_active":"2024-08-17T17:41:37.676764+0000","last_became_peered":"2024-08-17T17:41:37.676764+0000","last_unstale":"2024-08-17T17:44:38.108950+0000","last_undegraded":"2024-08-17T17:44:38.108950+0000","last_fullsized":"2024-08-17T17:44:38.108950+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:14:49.341080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":653,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.120323+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:44:38.120323+0000","last_peered":"2024-08-17T17:44:38.120323+0000","last_clean":"2024-08-17T17:44:38.120323+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:44:38.120323+0000","last_undegraded":"2024-08-17T17:44:38.120323+0000","last_fullsized":"2024-08-17T17:44:38.120323+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"387'8425","reported_seq":8463,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.799229+0000","last_change":"2024-08-17T17:42:36.929613+0000","last_active":"2024-08-17T17:44:38.799229+0000","last_peered":"2024-08-17T17:44:38.799229+0000","last_clean":"2024-08-17T17:44:38.799229+0000","last_became_active":"2024-08-17T17:42:36.929191+0000","last_became_peered":"2024-08-17T17:42:36.929191+0000","last_unstale":"2024-08-17T17:44:38.799229+0000","last_undegraded":"2024-08-17T17:44:38.799229+0000","last_fullsized":"2024-08-17T17:44:38.799229+0000","mapping_epoch":357,"log_start":"369'2000","ondisk_log_start":"369'2000","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6425,"log_dups_size":0,"ondisk_log_size":6425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:21:40.881779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412549120,"num_objects":6295,"num_object_clones":0,"num_object_copies":18885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8425,"num_write_kb":471040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":800,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.123033+0000","last_change":"2024-08-17T17:38:23.113397+0000","last_active":"2024-08-17T17:44:38.123033+0000","last_peered":"2024-08-17T17:44:38.123033+0000","last_clean":"2024-08-17T17:44:38.123033+0000","last_became_active":"2024-08-17T17:38:21.305580+0000","last_became_peered":"2024-08-17T17:38:21.305580+0000","last_unstale":"2024-08-17T17:44:38.123033+0000","last_undegraded":"2024-08-17T17:44:38.123033+0000","last_fullsized":"2024-08-17T17:44:38.123033+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.113279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.113279+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:38:07.015721+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":493,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.136704+0000","last_change":"2024-08-17T17:41:34.629522+0000","last_active":"2024-08-17T17:44:38.136704+0000","last_peered":"2024-08-17T17:44:38.136704+0000","last_clean":"2024-08-17T17:44:38.136704+0000","last_became_active":"2024-08-17T17:41:34.619340+0000","last_became_peered":"2024-08-17T17:41:34.619340+0000","last_unstale":"2024-08-17T17:44:38.136704+0000","last_undegraded":"2024-08-17T17:44:38.136704+0000","last_fullsized":"2024-08-17T17:44:38.136704+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:30:03.902956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":617,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.215898+0000","last_change":"2024-08-17T17:26:06.855307+0000","last_active":"2024-08-17T17:44:29.215898+0000","last_peered":"2024-08-17T17:44:29.215898+0000","last_clean":"2024-08-17T17:44:29.215898+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:44:29.215898+0000","last_undegraded":"2024-08-17T17:44:29.215898+0000","last_fullsized":"2024-08-17T17:44:29.215898+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":51,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.208597+0000","last_change":"2024-08-17T17:41:34.614028+0000","last_active":"2024-08-17T17:44:29.208597+0000","last_peered":"2024-08-17T17:44:29.208597+0000","last_clean":"2024-08-17T17:44:29.208597+0000","last_became_active":"2024-08-17T17:41:34.613706+0000","last_became_peered":"2024-08-17T17:41:34.613706+0000","last_unstale":"2024-08-17T17:44:29.208597+0000","last_undegraded":"2024-08-17T17:44:29.208597+0000","last_fullsized":"2024-08-17T17:44:29.208597+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:19:55.254902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"90'149","reported_seq":711,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.108793+0000","last_change":"2024-08-17T17:42:07.681204+0000","last_active":"2024-08-17T17:44:38.108793+0000","last_peered":"2024-08-17T17:44:38.108793+0000","last_clean":"2024-08-17T17:44:38.108793+0000","last_became_active":"2024-08-17T17:42:05.878769+0000","last_became_peered":"2024-08-17T17:42:05.878769+0000","last_unstale":"2024-08-17T17:44:38.108793+0000","last_undegraded":"2024-08-17T17:44:38.108793+0000","last_fullsized":"2024-08-17T17:44:38.108793+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:07.681064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:07.681064+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:12:12.835715+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":51,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.208588+0000","last_change":"2024-08-17T17:41:34.622180+0000","last_active":"2024-08-17T17:44:29.208588+0000","last_peered":"2024-08-17T17:44:29.208588+0000","last_clean":"2024-08-17T17:44:29.208588+0000","last_became_active":"2024-08-17T17:41:34.621724+0000","last_became_peered":"2024-08-17T17:41:34.621724+0000","last_unstale":"2024-08-17T17:44:29.208588+0000","last_undegraded":"2024-08-17T17:44:29.208588+0000","last_fullsized":"2024-08-17T17:44:29.208588+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:15.807235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":61,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.208790+0000","last_change":"2024-08-17T17:41:34.626584+0000","last_active":"2024-08-17T17:44:29.208790+0000","last_peered":"2024-08-17T17:44:29.208790+0000","last_clean":"2024-08-17T17:44:29.208790+0000","last_became_active":"2024-08-17T17:41:34.609878+0000","last_became_peered":"2024-08-17T17:41:34.609878+0000","last_unstale":"2024-08-17T17:44:29.208790+0000","last_undegraded":"2024-08-17T17:44:29.208790+0000","last_fullsized":"2024-08-17T17:44:29.208790+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:08:12.810347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.17","version":"90'149","reported_seq":470,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.129578+0000","last_change":"2024-08-17T17:42:06.699082+0000","last_active":"2024-08-17T17:44:38.129578+0000","last_peered":"2024-08-17T17:44:38.129578+0000","last_clean":"2024-08-17T17:44:38.129578+0000","last_became_active":"2024-08-17T17:42:05.876218+0000","last_became_peered":"2024-08-17T17:42:05.876218+0000","last_unstale":"2024-08-17T17:44:38.129578+0000","last_undegraded":"2024-08-17T17:44:38.129578+0000","last_fullsized":"2024-08-17T17:44:38.129578+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.698998+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:06.698998+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.698998+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:22:16.510150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":598,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.115066+0000","last_change":"2024-08-17T17:33:20.799402+0000","last_active":"2024-08-17T17:44:38.115066+0000","last_peered":"2024-08-17T17:44:38.115066+0000","last_clean":"2024-08-17T17:44:38.115066+0000","last_became_active":"2024-08-17T17:33:20.798832+0000","last_became_peered":"2024-08-17T17:33:20.798832+0000","last_unstale":"2024-08-17T17:44:38.115066+0000","last_undegraded":"2024-08-17T17:44:38.115066+0000","last_fullsized":"2024-08-17T17:44:38.115066+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:21:05.001134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":374,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.124885+0000","last_change":"2024-08-17T17:41:34.634301+0000","last_active":"2024-08-17T17:44:38.124885+0000","last_peered":"2024-08-17T17:44:38.124885+0000","last_clean":"2024-08-17T17:44:38.124885+0000","last_became_active":"2024-08-17T17:41:34.620508+0000","last_became_peered":"2024-08-17T17:41:34.620508+0000","last_unstale":"2024-08-17T17:44:38.124885+0000","last_undegraded":"2024-08-17T17:44:38.124885+0000","last_fullsized":"2024-08-17T17:44:38.124885+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:10:54.576812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.14","version":"90'149","reported_seq":823,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.127163+0000","last_change":"2024-08-17T17:41:37.664471+0000","last_active":"2024-08-17T17:44:38.127163+0000","last_peered":"2024-08-17T17:44:38.127163+0000","last_clean":"2024-08-17T17:44:38.127163+0000","last_became_active":"2024-08-17T17:41:37.664108+0000","last_became_peered":"2024-08-17T17:41:37.664108+0000","last_unstale":"2024-08-17T17:44:38.127163+0000","last_undegraded":"2024-08-17T17:44:38.127163+0000","last_fullsized":"2024-08-17T17:44:38.127163+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:22.030514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:22.030514+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:15:52.325163+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":337,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.109038+0000","last_change":"2024-08-17T17:33:44.146014+0000","last_active":"2024-08-17T17:44:38.109038+0000","last_peered":"2024-08-17T17:44:38.109038+0000","last_clean":"2024-08-17T17:44:38.109038+0000","last_became_active":"2024-08-17T17:33:44.145668+0000","last_became_peered":"2024-08-17T17:33:44.145668+0000","last_unstale":"2024-08-17T17:44:38.109038+0000","last_undegraded":"2024-08-17T17:44:38.109038+0000","last_fullsized":"2024-08-17T17:44:38.109038+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:53:41.679237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":602,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.120711+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:44:38.120711+0000","last_peered":"2024-08-17T17:44:38.120711+0000","last_clean":"2024-08-17T17:44:38.120711+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:44:38.120711+0000","last_undegraded":"2024-08-17T17:44:38.120711+0000","last_fullsized":"2024-08-17T17:44:38.120711+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"90'149","reported_seq":773,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.206786+0000","last_change":"2024-08-17T17:42:06.456562+0000","last_active":"2024-08-17T17:44:29.206786+0000","last_peered":"2024-08-17T17:44:29.206786+0000","last_clean":"2024-08-17T17:44:29.206786+0000","last_became_active":"2024-08-17T17:42:05.876577+0000","last_became_peered":"2024-08-17T17:42:05.876577+0000","last_unstale":"2024-08-17T17:44:29.206786+0000","last_undegraded":"2024-08-17T17:44:29.206786+0000","last_fullsized":"2024-08-17T17:44:29.206786+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.456490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.456490+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:10:03.654953+0000","scrub_duration":0.001537285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":629,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.129918+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:44:38.129918+0000","last_peered":"2024-08-17T17:44:38.129918+0000","last_clean":"2024-08-17T17:44:38.129918+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:44:38.129918+0000","last_undegraded":"2024-08-17T17:44:38.129918+0000","last_fullsized":"2024-08-17T17:44:38.129918+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":660,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.120229+0000","last_change":"2024-08-17T17:33:20.790687+0000","last_active":"2024-08-17T17:44:38.120229+0000","last_peered":"2024-08-17T17:44:38.120229+0000","last_clean":"2024-08-17T17:44:38.120229+0000","last_became_active":"2024-08-17T17:33:20.790399+0000","last_became_peered":"2024-08-17T17:33:20.790399+0000","last_unstale":"2024-08-17T17:44:38.120229+0000","last_undegraded":"2024-08-17T17:44:38.120229+0000","last_fullsized":"2024-08-17T17:44:38.120229+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:38:06.849989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1a","version":"90'149","reported_seq":435,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.207881+0000","last_change":"2024-08-17T17:42:21.932830+0000","last_active":"2024-08-17T17:44:29.207881+0000","last_peered":"2024-08-17T17:44:29.207881+0000","last_clean":"2024-08-17T17:44:29.207881+0000","last_became_active":"2024-08-17T17:42:05.876835+0000","last_became_peered":"2024-08-17T17:42:05.876835+0000","last_unstale":"2024-08-17T17:44:29.207881+0000","last_undegraded":"2024-08-17T17:44:29.207881+0000","last_fullsized":"2024-08-17T17:44:29.207881+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:21.932757+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:21.932757+0000","last_clean_scrub_stamp":"2024-08-17T17:42:21.932757+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:36:39.848570+0000","scrub_duration":0.0020218810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":360,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.215728+0000","last_change":"2024-08-17T17:33:20.835760+0000","last_active":"2024-08-17T17:44:29.215728+0000","last_peered":"2024-08-17T17:44:29.215728+0000","last_clean":"2024-08-17T17:44:29.215728+0000","last_became_active":"2024-08-17T17:33:20.835410+0000","last_became_peered":"2024-08-17T17:33:20.835410+0000","last_unstale":"2024-08-17T17:44:29.215728+0000","last_undegraded":"2024-08-17T17:44:29.215728+0000","last_fullsized":"2024-08-17T17:44:29.215728+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:45.448022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":678,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.121341+0000","last_change":"2024-08-17T17:33:20.771279+0000","last_active":"2024-08-17T17:44:38.121341+0000","last_peered":"2024-08-17T17:44:38.121341+0000","last_clean":"2024-08-17T17:44:38.121341+0000","last_became_active":"2024-08-17T17:33:20.770728+0000","last_became_peered":"2024-08-17T17:33:20.770728+0000","last_unstale":"2024-08-17T17:44:38.121341+0000","last_undegraded":"2024-08-17T17:44:38.121341+0000","last_fullsized":"2024-08-17T17:44:38.121341+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:52:31.540397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"90'149","reported_seq":730,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.208346+0000","last_change":"2024-08-17T17:42:26.967686+0000","last_active":"2024-08-17T17:44:29.208346+0000","last_peered":"2024-08-17T17:44:29.208346+0000","last_clean":"2024-08-17T17:44:29.208346+0000","last_became_active":"2024-08-17T17:42:05.873657+0000","last_became_peered":"2024-08-17T17:42:05.873657+0000","last_unstale":"2024-08-17T17:44:29.208346+0000","last_undegraded":"2024-08-17T17:44:29.208346+0000","last_fullsized":"2024-08-17T17:44:29.208346+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:26.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:26.967611+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:31:03.344332+0000","scrub_duration":0.0015978279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":448,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.136508+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:44:38.136508+0000","last_peered":"2024-08-17T17:44:38.136508+0000","last_clean":"2024-08-17T17:44:38.136508+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:44:38.136508+0000","last_undegraded":"2024-08-17T17:44:38.136508+0000","last_fullsized":"2024-08-17T17:44:38.136508+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":597,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.108396+0000","last_change":"2024-08-17T17:41:34.625011+0000","last_active":"2024-08-17T17:44:38.108396+0000","last_peered":"2024-08-17T17:44:38.108396+0000","last_clean":"2024-08-17T17:44:38.108396+0000","last_became_active":"2024-08-17T17:41:34.624631+0000","last_became_peered":"2024-08-17T17:41:34.624631+0000","last_unstale":"2024-08-17T17:44:38.108396+0000","last_undegraded":"2024-08-17T17:44:38.108396+0000","last_fullsized":"2024-08-17T17:44:38.108396+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:03:05.565435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"90'149","reported_seq":693,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.215981+0000","last_change":"2024-08-17T17:42:08.482975+0000","last_active":"2024-08-17T17:44:29.215981+0000","last_peered":"2024-08-17T17:44:29.215981+0000","last_clean":"2024-08-17T17:44:29.215981+0000","last_became_active":"2024-08-17T17:42:05.873523+0000","last_became_peered":"2024-08-17T17:42:05.873523+0000","last_unstale":"2024-08-17T17:44:29.215981+0000","last_undegraded":"2024-08-17T17:44:29.215981+0000","last_fullsized":"2024-08-17T17:44:29.215981+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:08.482892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:08.482892+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:41.477341+0000","scrub_duration":0.0017925479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":51,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.209253+0000","last_change":"2024-08-17T17:41:34.616482+0000","last_active":"2024-08-17T17:44:29.209253+0000","last_peered":"2024-08-17T17:44:29.209253+0000","last_clean":"2024-08-17T17:44:29.209253+0000","last_became_active":"2024-08-17T17:41:34.616040+0000","last_became_peered":"2024-08-17T17:41:34.616040+0000","last_unstale":"2024-08-17T17:44:29.209253+0000","last_undegraded":"2024-08-17T17:44:29.209253+0000","last_fullsized":"2024-08-17T17:44:29.209253+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:53:08.502264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":51,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.208078+0000","last_change":"2024-08-17T17:41:34.622184+0000","last_active":"2024-08-17T17:44:29.208078+0000","last_peered":"2024-08-17T17:44:29.208078+0000","last_clean":"2024-08-17T17:44:29.208078+0000","last_became_active":"2024-08-17T17:41:34.621831+0000","last_became_peered":"2024-08-17T17:41:34.621831+0000","last_unstale":"2024-08-17T17:44:29.208078+0000","last_undegraded":"2024-08-17T17:44:29.208078+0000","last_fullsized":"2024-08-17T17:44:29.208078+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:59.826667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"90'149","reported_seq":827,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.120526+0000","last_change":"2024-08-17T17:42:05.876463+0000","last_active":"2024-08-17T17:44:38.120526+0000","last_peered":"2024-08-17T17:44:38.120526+0000","last_clean":"2024-08-17T17:44:38.120526+0000","last_became_active":"2024-08-17T17:42:05.876069+0000","last_became_peered":"2024-08-17T17:42:05.876069+0000","last_unstale":"2024-08-17T17:44:38.120526+0000","last_undegraded":"2024-08-17T17:44:38.120526+0000","last_fullsized":"2024-08-17T17:44:38.120526+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:45.552985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:45.552985+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:55:06.027278+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":564,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.108549+0000","last_change":"2024-08-17T17:33:20.773207+0000","last_active":"2024-08-17T17:44:38.108549+0000","last_peered":"2024-08-17T17:44:38.108549+0000","last_clean":"2024-08-17T17:44:38.108549+0000","last_became_active":"2024-08-17T17:33:20.772457+0000","last_became_peered":"2024-08-17T17:33:20.772457+0000","last_unstale":"2024-08-17T17:44:38.108549+0000","last_undegraded":"2024-08-17T17:44:38.108549+0000","last_fullsized":"2024-08-17T17:44:38.108549+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:57:09.610084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":471,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.130848+0000","last_change":"2024-08-17T17:41:34.618968+0000","last_active":"2024-08-17T17:44:38.130848+0000","last_peered":"2024-08-17T17:44:38.130848+0000","last_clean":"2024-08-17T17:44:38.130848+0000","last_became_active":"2024-08-17T17:41:34.618576+0000","last_became_peered":"2024-08-17T17:41:34.618576+0000","last_unstale":"2024-08-17T17:44:38.130848+0000","last_undegraded":"2024-08-17T17:44:38.130848+0000","last_fullsized":"2024-08-17T17:44:38.130848+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:47:45.888590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"90'149","reported_seq":475,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.121638+0000","last_change":"2024-08-17T17:42:06.742801+0000","last_active":"2024-08-17T17:44:38.121638+0000","last_peered":"2024-08-17T17:44:38.121638+0000","last_clean":"2024-08-17T17:44:38.121638+0000","last_became_active":"2024-08-17T17:42:05.876895+0000","last_became_peered":"2024-08-17T17:42:05.876895+0000","last_unstale":"2024-08-17T17:44:38.121638+0000","last_undegraded":"2024-08-17T17:44:38.121638+0000","last_fullsized":"2024-08-17T17:44:38.121638+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.742723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.742723+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:36:46.651953+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":51,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.208273+0000","last_change":"2024-08-17T17:41:34.622094+0000","last_active":"2024-08-17T17:44:29.208273+0000","last_peered":"2024-08-17T17:44:29.208273+0000","last_clean":"2024-08-17T17:44:29.208273+0000","last_became_active":"2024-08-17T17:41:34.621773+0000","last_became_peered":"2024-08-17T17:41:34.621773+0000","last_unstale":"2024-08-17T17:44:29.208273+0000","last_undegraded":"2024-08-17T17:44:29.208273+0000","last_fullsized":"2024-08-17T17:44:29.208273+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:34:30.540886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":55,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.209434+0000","last_change":"2024-08-17T17:41:34.629135+0000","last_active":"2024-08-17T17:44:29.209434+0000","last_peered":"2024-08-17T17:44:29.209434+0000","last_clean":"2024-08-17T17:44:29.209434+0000","last_became_active":"2024-08-17T17:41:34.613095+0000","last_became_peered":"2024-08-17T17:41:34.613095+0000","last_unstale":"2024-08-17T17:44:29.209434+0000","last_undegraded":"2024-08-17T17:44:29.209434+0000","last_fullsized":"2024-08-17T17:44:29.209434+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:27:06.542416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":51,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.208212+0000","last_change":"2024-08-17T17:41:34.616251+0000","last_active":"2024-08-17T17:44:29.208212+0000","last_peered":"2024-08-17T17:44:29.208212+0000","last_clean":"2024-08-17T17:44:29.208212+0000","last_became_active":"2024-08-17T17:41:34.615952+0000","last_became_peered":"2024-08-17T17:41:34.615952+0000","last_unstale":"2024-08-17T17:44:29.208212+0000","last_undegraded":"2024-08-17T17:44:29.208212+0000","last_fullsized":"2024-08-17T17:44:29.208212+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:12:24.892508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":55,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.208979+0000","last_change":"2024-08-17T17:41:34.631152+0000","last_active":"2024-08-17T17:44:29.208979+0000","last_peered":"2024-08-17T17:44:29.208979+0000","last_clean":"2024-08-17T17:44:29.208979+0000","last_became_active":"2024-08-17T17:41:34.615617+0000","last_became_peered":"2024-08-17T17:41:34.615617+0000","last_unstale":"2024-08-17T17:44:29.208979+0000","last_undegraded":"2024-08-17T17:44:29.208979+0000","last_fullsized":"2024-08-17T17:44:29.208979+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:43:08.257084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"90'149","reported_seq":727,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.211281+0000","last_change":"2024-08-17T17:42:11.809047+0000","last_active":"2024-08-17T17:44:29.211281+0000","last_peered":"2024-08-17T17:44:29.211281+0000","last_clean":"2024-08-17T17:44:29.211281+0000","last_became_active":"2024-08-17T17:42:05.883770+0000","last_became_peered":"2024-08-17T17:42:05.883770+0000","last_unstale":"2024-08-17T17:44:29.211281+0000","last_undegraded":"2024-08-17T17:44:29.211281+0000","last_fullsized":"2024-08-17T17:44:29.211281+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:11.808971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:11.808971+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:48:15.989437+0000","scrub_duration":0.0018167420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":656,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.121063+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:44:38.121063+0000","last_peered":"2024-08-17T17:44:38.121063+0000","last_clean":"2024-08-17T17:44:38.121063+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:44:38.121063+0000","last_undegraded":"2024-08-17T17:44:38.121063+0000","last_fullsized":"2024-08-17T17:44:38.121063+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":657,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.121766+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:44:38.121766+0000","last_peered":"2024-08-17T17:44:38.121766+0000","last_clean":"2024-08-17T17:44:38.121766+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:44:38.121766+0000","last_undegraded":"2024-08-17T17:44:38.121766+0000","last_fullsized":"2024-08-17T17:44:38.121766+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1d","version":"90'149","reported_seq":723,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.108687+0000","last_change":"2024-08-17T17:42:18.668058+0000","last_active":"2024-08-17T17:44:38.108687+0000","last_peered":"2024-08-17T17:44:38.108687+0000","last_clean":"2024-08-17T17:44:38.108687+0000","last_became_active":"2024-08-17T17:42:05.876721+0000","last_became_peered":"2024-08-17T17:42:05.876721+0000","last_unstale":"2024-08-17T17:44:38.108687+0000","last_undegraded":"2024-08-17T17:44:38.108687+0000","last_fullsized":"2024-08-17T17:44:38.108687+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:18.667957+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:18.667957+0000","last_clean_scrub_stamp":"2024-08-17T17:42:18.667957+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:38.170146+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":51,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-08-17T17:44:29.208735+0000","last_change":"2024-08-17T17:41:34.611711+0000","last_active":"2024-08-17T17:44:29.208735+0000","last_peered":"2024-08-17T17:44:29.208735+0000","last_clean":"2024-08-17T17:44:29.208735+0000","last_became_active":"2024-08-17T17:41:34.611407+0000","last_became_peered":"2024-08-17T17:41:34.611407+0000","last_unstale":"2024-08-17T17:44:29.208735+0000","last_undegraded":"2024-08-17T17:44:29.208735+0000","last_fullsized":"2024-08-17T17:44:29.208735+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:42:41.582320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":466,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.129820+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:44:38.129820+0000","last_peered":"2024-08-17T17:44:38.129820+0000","last_clean":"2024-08-17T17:44:38.129820+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:44:38.129820+0000","last_undegraded":"2024-08-17T17:44:38.129820+0000","last_fullsized":"2024-08-17T17:44:38.129820+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":6620446744,"num_objects":101021,"num_object_clones":0,"num_object_copies":303063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":393,"num_objects_misplaced":4754,"num_objects_unfound":0,"num_objects_dirty":101021,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":136067,"num_write_kb":7586753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1653,"num_bytes_recovered":108331008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19897921536,"data_stored":19897909320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101867,"ondisk_log_size":101867,"up":48,"acting":47,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":11256,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19280,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5978,"num_read_kb":12991,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4619,"ondisk_log_size":4619,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234454,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2558428,"kb_used_data":1660372,"kb_used_omap":1,"kb_used_meta":898046,"kb_avail":91186724,"statfs":{"total":95995035648,"available":93375205376,"internally_reserved":0,"allocated":1700220928,"data_stored":1698009803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":919599448},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496089,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3739360,"kb_used_data":2844244,"kb_used_omap":1,"kb_used_meta":895102,"kb_avail":90005792,"statfs":{"total":95995035648,"available":92165931008,"internally_reserved":0,"allocated":2912505856,"data_stored":2910302257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":916584792},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757726,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2633756,"kb_used_data":1743120,"kb_used_omap":1,"kb_used_meta":890622,"kb_avail":91111396,"statfs":{"total":95995035648,"available":93298069504,"internally_reserved":0,"allocated":1784954880,"data_stored":1782751922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":911997272},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019364,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4028892,"kb_used_data":3719060,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":89716260,"statfs":{"total":95995035648,"available":91869450240,"internally_reserved":0,"allocated":3808317440,"data_stored":3806106056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435591,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2957652,"kb_used_data":2043336,"kb_used_omap":4,"kb_used_meta":914299,"kb_avail":90787500,"statfs":{"total":95995035648,"available":92966400000,"internally_reserved":0,"allocated":2092376064,"data_stored":2090180536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":936242186},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599118,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3594652,"kb_used_data":2803280,"kb_used_omap":7,"kb_used_meta":791352,"kb_avail":90150500,"statfs":{"total":95995035648,"available":92314112000,"internally_reserved":0,"allocated":2870558720,"data_stored":2868356058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":810344610},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":268,"seq":1151051235465,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1634192,"kb_used_data":1183944,"kb_used_omap":10,"kb_used_meta":450229,"kb_avail":92110960,"statfs":{"total":95995035648,"available":94321623040,"internally_reserved":0,"allocated":1212358656,"data_stored":1210163073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10902,"internal_metadata":461034858},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926171,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4929180,"kb_used_data":4039120,"kb_used_omap":7,"kb_used_meta":889976,"kb_avail":88815972,"statfs":{"total":95995035648,"available":90947555328,"internally_reserved":0,"allocated":4136058880,"data_stored":4133855694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":911335745},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7550,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4113436672,"data_stored":4113432600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1203437568,"data_stored":1203437568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2841116672,"data_stored":2841116672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2069626880,"data_stored":2069626880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3332509696,"data_stored":3332505624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1770848256,"data_stored":1770848256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2888110080,"data_stored":2888106008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1678835712,"data_stored":1678835712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:44:45.529 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:44:45.623 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:45.668 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:45.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:45.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:45.683 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:45 smithi003 ceph-mon[100615]: pgmap v1263: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 971 op/s; 589/293802 objects degraded (0.200%); 4455/293802 objects misplaced (1.516%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:44:45.683 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:45 smithi003 ceph-mon[103344]: pgmap v1263: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 971 op/s; 589/293802 objects degraded (0.200%); 4455/293802 objects misplaced (1.516%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:44:45.684 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:44:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:44:45] "GET /metrics HTTP/1.1" 200 36579 "" "Prometheus/2.51.0" 2024-08-17T17:44:45.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:45 smithi053 ceph-mon[89963]: pgmap v1263: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 971 op/s; 589/293802 objects degraded (0.200%); 4455/293802 objects misplaced (1.516%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:44:45.771 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:45.812 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:45.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:45.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:45.914 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:45.951 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:45.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:45.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:46.053 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:46.058 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-17T17:44:46.084 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:46.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:46.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:46.098 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:44:46.112 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:44:46.186 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:46.236 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:46.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:46.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:46.338 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:46.381 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:46.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:46.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:46.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:46.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:46 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 589/288420 objects degraded (0.204%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:44:46.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:46 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 589/288420 objects degraded (0.204%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:44:46.534 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:46.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:46.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:46.636 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:46.672 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:46.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:46.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:46.774 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:46.810 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:46.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:46.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:46.912 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:46.957 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:46.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:46.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:46.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:46 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 589/288420 objects degraded (0.204%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:44:47.059 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:47.088 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:47.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:47.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:47.189 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:47.231 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:47.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:47.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:47.333 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:47.376 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:47.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:47.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:47.478 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:47.512 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:47.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:47.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:47.528 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:47 smithi003 ceph-mon[100615]: pgmap v1264: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 986 op/s; 589/288420 objects degraded (0.204%); 4455/288420 objects misplaced (1.545%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:44:47.529 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:47 smithi003 ceph-mon[103344]: pgmap v1264: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 986 op/s; 589/288420 objects degraded (0.204%); 4455/288420 objects misplaced (1.545%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:44:47.615 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:47.641 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:47.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:47.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:47.743 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:47.781 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:47.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:47.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:47.883 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:47.921 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:47.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:47.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:47.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:47 smithi053 ceph-mon[89963]: pgmap v1264: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 986 op/s; 589/288420 objects degraded (0.204%); 4455/288420 objects misplaced (1.545%); 1.2 MiB/s, 19 objects/s recovering 2024-08-17T17:44:48.022 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:48.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:48.068 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:48.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:48.170 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:48.206 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:48.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:48.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:48.308 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:48.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:48.342 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:48.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:48.443 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:48.479 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:48.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:48.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:48.581 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:48.622 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:48.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:48.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:48.724 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:48.761 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:48.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:48.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:48.863 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:48.903 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:48.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:48.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:49.005 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:49.009 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:44:49.057 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:49.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:49.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:49.160 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:49.194 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:49.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:49.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:49.296 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:49.328 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:49.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:49.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:49.430 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:49.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:49.471 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:49.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:49.572 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:49.624 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:49.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:49.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:49.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:49 smithi053 ceph-mon[89963]: pgmap v1265: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 958 op/s; 589/288420 objects degraded (0.204%); 4455/288420 objects misplaced (1.545%); 1.2 MiB/s, 18 objects/s recovering 2024-08-17T17:44:49.727 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:49.766 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:49.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:49.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:49.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:49 smithi003 ceph-mon[103344]: pgmap v1265: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 958 op/s; 589/288420 objects degraded (0.204%); 4455/288420 objects misplaced (1.545%); 1.2 MiB/s, 18 objects/s recovering 2024-08-17T17:44:49.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:49 smithi003 ceph-mon[100615]: pgmap v1265: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 958 op/s; 589/288420 objects degraded (0.204%); 4455/288420 objects misplaced (1.545%); 1.2 MiB/s, 18 objects/s recovering 2024-08-17T17:44:49.867 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:49.909 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:49.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:49.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:50.011 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:50.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:50.076 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:50.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:50.177 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:50.216 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:50.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:50.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:50.317 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:50.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:50.351 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:50.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:50.452 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:50.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:44:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:44:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:44:50.488 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:50.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:50.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:50.590 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:50.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:50.630 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:50.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:50.731 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:50.769 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:50.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:50.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:50.787 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:44:50.788 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:44:50.819 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:44:50.819 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:44:50.858 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:44:50.858 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:44:50.870 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:50.884 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:44:50.885 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:44:50.906 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:50.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:50.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:50.915 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:44:50.915 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:44:50.947 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:44:50.947 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:44:50.982 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:44:50.982 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:44:51.008 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:51.013 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:44:51.014 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:44:51.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:51.059 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:51.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:51.160 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:51.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:51.197 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:51.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:51.298 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:51.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:51.339 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:51.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:51.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:51.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:51.487 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:51.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:51.589 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:51.637 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:51.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:51.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:51.739 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:51.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:51 smithi003 ceph-mon[103344]: pgmap v1266: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s; 797/274182 objects degraded (0.291%); 4146/274182 objects misplaced (1.512%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:44:51.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:51 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/849061203' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:44:51.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:51 smithi003 ceph-mon[100615]: pgmap v1266: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s; 797/274182 objects degraded (0.291%); 4146/274182 objects misplaced (1.512%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:44:51.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:51 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/849061203' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:44:51.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:51.780 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:51.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:51.881 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:51.922 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:51.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:51.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:51.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:51 smithi053 ceph-mon[89963]: pgmap v1266: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.22k op/s; 797/274182 objects degraded (0.291%); 4146/274182 objects misplaced (1.512%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:44:51.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:51 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/849061203' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:44:52.024 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:52.064 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:52.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:52.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:52.166 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:52.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:52.201 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:52.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:52.302 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:52.331 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:52.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:52.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:52.433 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:52.475 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:52.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:52.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:52.600 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:52.638 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:52.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:52.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:52.738 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:52 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1476564057' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:44:52.738 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:52 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1476564057' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:44:52.740 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:52.772 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:52.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:52.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:52.874 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:52.905 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:52.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:52.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:52.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:52 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1476564057' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:44:53.008 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:53.043 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:53.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:53.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:53.144 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:53.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:53.182 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:53.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:53.283 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:53.318 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-08-17T17:44:53.319 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:53.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:53.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:53.421 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:53.455 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:53.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:53.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:53.533 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:44:53.556 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:53.611 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:53.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:53.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:53.712 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:53.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:53.754 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:53.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:53.855 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:53.892 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:53.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:53.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:53.924 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:53 smithi003 ceph-mon[103344]: pgmap v1267: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 852 op/s; 797/273324 objects degraded (0.292%); 4146/273324 objects misplaced (1.517%); 1.1 MiB/s, 16 objects/s recovering 2024-08-17T17:44:53.924 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:53 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 797/273324 objects degraded (0.292%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:44:53.924 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:53 smithi003 ceph-mon[103344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:44:53.924 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:53 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1476564057' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:44:53.924 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:53 smithi003 ceph-mon[103344]: osdmap e388: 8 total, 8 up, 8 in 2024-08-17T17:44:53.925 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:44:53.925 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:53 smithi003 ceph-mon[100615]: pgmap v1267: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 852 op/s; 797/273324 objects degraded (0.292%); 4146/273324 objects misplaced (1.517%); 1.1 MiB/s, 16 objects/s recovering 2024-08-17T17:44:53.925 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:53 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 797/273324 objects degraded (0.292%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:44:53.925 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:53 smithi003 ceph-mon[100615]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:44:53.925 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:53 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1476564057' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:44:53.925 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:53 smithi003 ceph-mon[100615]: osdmap e388: 8 total, 8 up, 8 in 2024-08-17T17:44:53.925 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:44:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:53 smithi053 ceph-mon[89963]: pgmap v1267: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 852 op/s; 797/273324 objects degraded (0.292%); 4146/273324 objects misplaced (1.517%); 1.1 MiB/s, 16 objects/s recovering 2024-08-17T17:44:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:53 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 797/273324 objects degraded (0.292%), 1 pg degraded (PG_DEGRADED) 2024-08-17T17:44:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:53 smithi053 ceph-mon[89963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:44:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:53 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1476564057' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:44:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:53 smithi053 ceph-mon[89963]: osdmap e388: 8 total, 8 up, 8 in 2024-08-17T17:44:53.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:44:53.994 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:54.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:54.029 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:54.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:54.130 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:54.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:54.176 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:54.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:54.277 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:54.314 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:54.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:54.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:54.416 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:54.450 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:54.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:54.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:54.552 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:54.597 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:54.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:54.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:54.702 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:54.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:54.739 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:54.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:54.781 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:44:54.840 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:54.878 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:54.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:54.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:54.980 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:55.019 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:55.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:55.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:55.121 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:55.152 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:55.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:55.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:55.254 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:55.326 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:55.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:55.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:55.428 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:55.494 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:55.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:55.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:55.596 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:55.652 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:55.652 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:55 smithi003 ceph-mon[103344]: pgmap v1269: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 5.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 1026/263670 objects degraded (0.389%); 3816/263670 objects misplaced (1.447%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:44:55.652 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:55 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:55.652 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:55 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:55.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:55.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:55.654 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:55 smithi003 ceph-mon[100615]: pgmap v1269: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 5.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 1026/263670 objects degraded (0.389%); 3816/263670 objects misplaced (1.447%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:44:55.654 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:55 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:55.654 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:55 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:55.664 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:44:55.666 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:44:55.754 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:55.788 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:55.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:55.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:55.889 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:55.923 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:55.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:55.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:55.925 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:44:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:44:55] "GET /metrics HTTP/1.1" 200 36578 "" "Prometheus/2.51.0" 2024-08-17T17:44:55.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:55 smithi053 ceph-mon[89963]: pgmap v1269: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 5.4 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 1026/263670 objects degraded (0.389%); 3816/263670 objects misplaced (1.447%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:44:55.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:55 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:55.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:55 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:55.967 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:44:56.025 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:56.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:56.063 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:56.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:56.164 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:56.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:56.199 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:56.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:56.299 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:44:56.300 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:56.349 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:56.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:56.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:56.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:56.486 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:56.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:56.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:56.588 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:56.626 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:56.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:56.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:56.728 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:56.765 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:56.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:56.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:56.792 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:56 smithi003 ceph-mon[103344]: from='client.46700 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:44:56.792 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:56 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:44:56.792 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:56 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:44:56.792 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:56 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:56.792 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:56 smithi003 ceph-mon[100615]: from='client.46700 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:44:56.792 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:56 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:44:56.792 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:56 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:44:56.792 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:56 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:56.868 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:56.902 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:56.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:56.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:56.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:56 smithi053 ceph-mon[89963]: from='client.46700 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:44:56.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:56 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:44:56.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:56 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:44:56.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:56 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:44:57.004 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:57.048 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:57.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:57.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:57.150 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:57.186 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:57.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:57.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:57.288 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:57.326 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:57.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:57.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:57.429 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1269,"stamp":"2024-08-17T17:44:54.403112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5762848741,"num_objects":87890,"num_object_clones":0,"num_object_copies":263670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1026,"num_objects_misplaced":3816,"num_objects_unfound":0,"num_objects_dirty":87890,"num_whiteouts":0,"num_read":5979,"num_read_kb":12992,"num_write":150133,"num_write_kb":7599376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2152,"num_bytes_recovered":136719352,"num_keys_recovered":40,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107277,"ondisk_log_size":107277,"up":357,"acting":356,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":362,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":23655628,"kb_used_data":17584004,"kb_used_omap":37,"kb_used_meta":6071386,"kb_avail":726305588,"statfs":{"total":767960285184,"available":743736922112,"internally_reserved":0,"allocated":18006020096,"data_stored":17988370575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38729,"internal_metadata":6217099447},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,8,4,6,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-658243584,"num_objects":-10044,"num_object_clones":0,"num_object_copies":-30132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":437,"num_objects_misplaced":-639,"num_objects_unfound":0,"num_objects_dirty":-10044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10044,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":202,"num_bytes_recovered":13238272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003047"},"pg_stats":[{"pgid":"1.11","version":"90'149","reported_seq":825,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.126283+0000","last_change":"2024-08-17T17:38:30.378664+0000","last_active":"2024-08-17T17:44:38.126283+0000","last_peered":"2024-08-17T17:44:38.126283+0000","last_clean":"2024-08-17T17:44:38.126283+0000","last_became_active":"2024-08-17T17:38:30.342287+0000","last_became_peered":"2024-08-17T17:38:30.342287+0000","last_unstale":"2024-08-17T17:44:38.126283+0000","last_undegraded":"2024-08-17T17:44:38.126283+0000","last_fullsized":"2024-08-17T17:44:38.126283+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:30.378592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:30.378592+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:04:28.933354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":646,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.123276+0000","last_change":"2024-08-17T17:33:20.797029+0000","last_active":"2024-08-17T17:44:38.123276+0000","last_peered":"2024-08-17T17:44:38.123276+0000","last_clean":"2024-08-17T17:44:38.123276+0000","last_became_active":"2024-08-17T17:33:20.796679+0000","last_became_peered":"2024-08-17T17:33:20.796679+0000","last_unstale":"2024-08-17T17:44:38.123276+0000","last_undegraded":"2024-08-17T17:44:38.123276+0000","last_fullsized":"2024-08-17T17:44:38.123276+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:51.438262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":412,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.108462+0000","last_change":"2024-08-17T17:41:34.624198+0000","last_active":"2024-08-17T17:44:38.108462+0000","last_peered":"2024-08-17T17:44:38.108462+0000","last_clean":"2024-08-17T17:44:38.108462+0000","last_became_active":"2024-08-17T17:41:34.616071+0000","last_became_peered":"2024-08-17T17:41:34.616071+0000","last_unstale":"2024-08-17T17:44:38.108462+0000","last_undegraded":"2024-08-17T17:44:38.108462+0000","last_fullsized":"2024-08-17T17:44:38.108462+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:34:48.805354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"90'149","reported_seq":471,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.116193+0000","last_change":"2024-08-17T17:38:31.874623+0000","last_active":"2024-08-17T17:44:38.116193+0000","last_peered":"2024-08-17T17:44:38.116193+0000","last_clean":"2024-08-17T17:44:38.116193+0000","last_became_active":"2024-08-17T17:38:30.339313+0000","last_became_peered":"2024-08-17T17:38:30.339313+0000","last_unstale":"2024-08-17T17:44:38.116193+0000","last_undegraded":"2024-08-17T17:44:38.116193+0000","last_fullsized":"2024-08-17T17:44:38.116193+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:31.874423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:31.874423+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:28:17.980339+0000","scrub_duration":0.0017696579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":307,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.124783+0000","last_change":"2024-08-17T17:33:44.156036+0000","last_active":"2024-08-17T17:44:38.124783+0000","last_peered":"2024-08-17T17:44:38.124783+0000","last_clean":"2024-08-17T17:44:38.124783+0000","last_became_active":"2024-08-17T17:33:44.155758+0000","last_became_peered":"2024-08-17T17:33:44.155758+0000","last_unstale":"2024-08-17T17:44:38.124783+0000","last_undegraded":"2024-08-17T17:44:38.124783+0000","last_fullsized":"2024-08-17T17:44:38.124783+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:23.443405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":446,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.126589+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:44:38.126589+0000","last_peered":"2024-08-17T17:44:38.126589+0000","last_clean":"2024-08-17T17:44:38.126589+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:44:38.126589+0000","last_undegraded":"2024-08-17T17:44:38.126589+0000","last_fullsized":"2024-08-17T17:44:38.126589+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"90'149","reported_seq":843,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.114034+0000","last_change":"2024-08-17T17:41:37.681187+0000","last_active":"2024-08-17T17:44:38.114034+0000","last_peered":"2024-08-17T17:44:38.114034+0000","last_clean":"2024-08-17T17:44:38.114034+0000","last_became_active":"2024-08-17T17:41:37.680605+0000","last_became_peered":"2024-08-17T17:41:37.680605+0000","last_unstale":"2024-08-17T17:44:38.114034+0000","last_undegraded":"2024-08-17T17:44:38.114034+0000","last_fullsized":"2024-08-17T17:44:38.114034+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.844252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.844252+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:13:31.682357+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":446,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.136613+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:44:38.136613+0000","last_peered":"2024-08-17T17:44:38.136613+0000","last_clean":"2024-08-17T17:44:38.136613+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:44:38.136613+0000","last_undegraded":"2024-08-17T17:44:38.136613+0000","last_fullsized":"2024-08-17T17:44:38.136613+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":52,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.103586+0000","last_change":"2024-08-17T17:41:34.611693+0000","last_active":"2024-08-17T17:44:38.103586+0000","last_peered":"2024-08-17T17:44:38.103586+0000","last_clean":"2024-08-17T17:44:38.103586+0000","last_became_active":"2024-08-17T17:41:34.611374+0000","last_became_peered":"2024-08-17T17:41:34.611374+0000","last_unstale":"2024-08-17T17:44:38.103586+0000","last_undegraded":"2024-08-17T17:44:38.103586+0000","last_fullsized":"2024-08-17T17:44:38.103586+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:17:09.897446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"90'149","reported_seq":819,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.115316+0000","last_change":"2024-08-17T17:38:30.446165+0000","last_active":"2024-08-17T17:44:38.115316+0000","last_peered":"2024-08-17T17:44:38.115316+0000","last_clean":"2024-08-17T17:44:38.115316+0000","last_became_active":"2024-08-17T17:38:30.445048+0000","last_became_peered":"2024-08-17T17:38:30.445048+0000","last_unstale":"2024-08-17T17:44:38.115316+0000","last_undegraded":"2024-08-17T17:44:38.115316+0000","last_fullsized":"2024-08-17T17:44:38.115316+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:24.291592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:24.291592+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:17.100154+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":660,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.123478+0000","last_change":"2024-08-17T17:33:20.794682+0000","last_active":"2024-08-17T17:44:38.123478+0000","last_peered":"2024-08-17T17:44:38.123478+0000","last_clean":"2024-08-17T17:44:38.123478+0000","last_became_active":"2024-08-17T17:33:20.794238+0000","last_became_peered":"2024-08-17T17:33:20.794238+0000","last_unstale":"2024-08-17T17:44:38.123478+0000","last_undegraded":"2024-08-17T17:44:38.123478+0000","last_fullsized":"2024-08-17T17:44:38.123478+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:15:55.821920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":473,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.116376+0000","last_change":"2024-08-17T17:43:52.520659+0000","last_active":"2024-08-17T17:44:38.116376+0000","last_peered":"2024-08-17T17:44:38.116376+0000","last_clean":"2024-08-17T17:44:38.116376+0000","last_became_active":"2024-08-17T17:43:52.516799+0000","last_became_peered":"2024-08-17T17:43:52.516799+0000","last_unstale":"2024-08-17T17:44:38.116376+0000","last_undegraded":"2024-08-17T17:44:38.116376+0000","last_fullsized":"2024-08-17T17:44:38.116376+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:25:59.408383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"387'9186","reported_seq":9255,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:50.242105+0000","last_change":"2024-08-17T17:42:36.946078+0000","last_active":"2024-08-17T17:44:50.242105+0000","last_peered":"2024-08-17T17:44:50.242105+0000","last_clean":"2024-08-17T17:44:50.242105+0000","last_became_active":"2024-08-17T17:42:36.945602+0000","last_became_peered":"2024-08-17T17:42:36.945602+0000","last_unstale":"2024-08-17T17:44:50.242105+0000","last_undegraded":"2024-08-17T17:44:50.242105+0000","last_fullsized":"2024-08-17T17:44:50.242105+0000","mapping_epoch":357,"log_start":"369'2600","ondisk_log_start":"369'2600","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6586,"log_dups_size":0,"ondisk_log_size":6586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:18:43.716968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370016256,"num_objects":5646,"num_object_clones":0,"num_object_copies":16938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9186,"num_write_kb":474624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"90'149","reported_seq":798,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.108777+0000","last_change":"2024-08-17T17:42:16.610940+0000","last_active":"2024-08-17T17:44:38.108777+0000","last_peered":"2024-08-17T17:44:38.108777+0000","last_clean":"2024-08-17T17:44:38.108777+0000","last_became_active":"2024-08-17T17:41:37.670855+0000","last_became_peered":"2024-08-17T17:41:37.670855+0000","last_unstale":"2024-08-17T17:44:38.108777+0000","last_undegraded":"2024-08-17T17:44:38.108777+0000","last_fullsized":"2024-08-17T17:44:38.108777+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:16.610828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:16.610828+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:48:25.918610+0000","scrub_duration":0.086892487000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":349,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.109622+0000","last_change":"2024-08-17T17:41:34.635980+0000","last_active":"2024-08-17T17:44:38.109622+0000","last_peered":"2024-08-17T17:44:38.109622+0000","last_clean":"2024-08-17T17:44:38.109622+0000","last_became_active":"2024-08-17T17:41:34.620596+0000","last_became_peered":"2024-08-17T17:41:34.620596+0000","last_unstale":"2024-08-17T17:44:38.109622+0000","last_undegraded":"2024-08-17T17:44:38.109622+0000","last_fullsized":"2024-08-17T17:44:38.109622+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:19:15.792577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":142,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.116268+0000","last_change":"2024-08-17T17:33:44.145598+0000","last_active":"2024-08-17T17:44:38.116268+0000","last_peered":"2024-08-17T17:44:38.116268+0000","last_clean":"2024-08-17T17:44:38.116268+0000","last_became_active":"2024-08-17T17:33:44.145283+0000","last_became_peered":"2024-08-17T17:33:44.145283+0000","last_unstale":"2024-08-17T17:44:38.116268+0000","last_undegraded":"2024-08-17T17:44:38.116268+0000","last_fullsized":"2024-08-17T17:44:38.116268+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:11.965226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.7","version":"388'9495","reported_seq":13482,"reported_epoch":388,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-17T17:44:53.805036+0000","last_change":"2024-08-17T17:44:30.783759+0000","last_active":"2024-08-17T17:44:53.805036+0000","last_peered":"2024-08-17T17:44:53.805036+0000","last_clean":"2024-08-17T17:43:51.465744+0000","last_became_active":"2024-08-17T17:43:53.682178+0000","last_became_peered":"2024-08-17T17:43:53.682178+0000","last_unstale":"2024-08-17T17:44:53.805036+0000","last_undegraded":"2024-08-17T17:44:30.779387+0000","last_fullsized":"2024-08-17T17:43:53.513318+0000","mapping_epoch":378,"log_start":"370'3100","ondisk_log_start":"370'3100","created":357,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6395,"log_dups_size":0,"ondisk_log_size":6395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:42:59.608872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350158848,"num_objects":5343,"num_object_clones":0,"num_object_copies":16029,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1026,"num_objects_misplaced":3816,"num_objects_unfound":0,"num_objects_dirty":5343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9495,"num_write_kb":474816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1958,"num_bytes_recovered":128319488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3],"avail_no_missing":["6","2","3"],"object_location_counts":[{"shards":"2,3,6","objects":3816},{"shards":"3,6","objects":501}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"90'149","reported_seq":844,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.123145+0000","last_change":"2024-08-17T17:41:42.603841+0000","last_active":"2024-08-17T17:44:38.123145+0000","last_peered":"2024-08-17T17:44:38.123145+0000","last_clean":"2024-08-17T17:44:38.123145+0000","last_became_active":"2024-08-17T17:41:12.203400+0000","last_became_peered":"2024-08-17T17:41:12.203400+0000","last_unstale":"2024-08-17T17:44:38.123145+0000","last_undegraded":"2024-08-17T17:44:38.123145+0000","last_fullsized":"2024-08-17T17:44:38.123145+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:42.603561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:42.603561+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:50:12.843193+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":345,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.109347+0000","last_change":"2024-08-17T17:41:34.637696+0000","last_active":"2024-08-17T17:44:38.109347+0000","last_peered":"2024-08-17T17:44:38.109347+0000","last_clean":"2024-08-17T17:44:38.109347+0000","last_became_active":"2024-08-17T17:41:34.621388+0000","last_became_peered":"2024-08-17T17:41:34.621388+0000","last_unstale":"2024-08-17T17:44:38.109347+0000","last_undegraded":"2024-08-17T17:44:38.109347+0000","last_fullsized":"2024-08-17T17:44:38.109347+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:13:04.607036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":52,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.103466+0000","last_change":"2024-08-17T17:41:34.621078+0000","last_active":"2024-08-17T17:44:38.103466+0000","last_peered":"2024-08-17T17:44:38.103466+0000","last_clean":"2024-08-17T17:44:38.103466+0000","last_became_active":"2024-08-17T17:41:34.620730+0000","last_became_peered":"2024-08-17T17:41:34.620730+0000","last_unstale":"2024-08-17T17:44:38.103466+0000","last_undegraded":"2024-08-17T17:44:38.103466+0000","last_fullsized":"2024-08-17T17:44:38.103466+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:55:01.312283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"387'9222","reported_seq":9294,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:50.229441+0000","last_change":"2024-08-17T17:42:36.620130+0000","last_active":"2024-08-17T17:44:50.229441+0000","last_peered":"2024-08-17T17:44:50.229441+0000","last_clean":"2024-08-17T17:44:50.229441+0000","last_became_active":"2024-08-17T17:42:36.619813+0000","last_became_peered":"2024-08-17T17:42:36.619813+0000","last_unstale":"2024-08-17T17:44:50.229441+0000","last_undegraded":"2024-08-17T17:44:50.229441+0000","last_fullsized":"2024-08-17T17:44:50.229441+0000","mapping_epoch":357,"log_start":"369'2700","ondisk_log_start":"369'2700","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6522,"log_dups_size":0,"ondisk_log_size":6522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:39:22.366905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374079488,"num_objects":5708,"num_object_clones":0,"num_object_copies":17124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9222,"num_write_kb":477760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"90'149","reported_seq":564,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.119516+0000","last_change":"2024-08-17T17:41:37.668050+0000","last_active":"2024-08-17T17:44:38.119516+0000","last_peered":"2024-08-17T17:44:38.119516+0000","last_clean":"2024-08-17T17:44:38.119516+0000","last_became_active":"2024-08-17T17:41:37.667655+0000","last_became_peered":"2024-08-17T17:41:37.667655+0000","last_unstale":"2024-08-17T17:44:38.119516+0000","last_undegraded":"2024-08-17T17:44:38.119516+0000","last_fullsized":"2024-08-17T17:44:38.119516+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:32.843802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:32.843802+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:39:46.766280+0000","scrub_duration":0.0040803669999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":138,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.116700+0000","last_change":"2024-08-17T17:33:44.145399+0000","last_active":"2024-08-17T17:44:38.116700+0000","last_peered":"2024-08-17T17:44:38.116700+0000","last_clean":"2024-08-17T17:44:38.116700+0000","last_became_active":"2024-08-17T17:33:44.144751+0000","last_became_peered":"2024-08-17T17:33:44.144751+0000","last_unstale":"2024-08-17T17:44:38.116700+0000","last_undegraded":"2024-08-17T17:44:38.116700+0000","last_fullsized":"2024-08-17T17:44:38.116700+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:54.276350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":216,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.116535+0000","last_change":"2024-08-17T17:33:20.795582+0000","last_active":"2024-08-17T17:44:38.116535+0000","last_peered":"2024-08-17T17:44:38.116535+0000","last_clean":"2024-08-17T17:44:38.116535+0000","last_became_active":"2024-08-17T17:33:20.795149+0000","last_became_peered":"2024-08-17T17:33:20.795149+0000","last_unstale":"2024-08-17T17:44:38.116535+0000","last_undegraded":"2024-08-17T17:44:38.116535+0000","last_fullsized":"2024-08-17T17:44:38.116535+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:14.020923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"387'9142","reported_seq":9214,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:50.225574+0000","last_change":"2024-08-17T17:42:36.929599+0000","last_active":"2024-08-17T17:44:50.225574+0000","last_peered":"2024-08-17T17:44:50.225574+0000","last_clean":"2024-08-17T17:44:50.225574+0000","last_became_active":"2024-08-17T17:42:36.929274+0000","last_became_peered":"2024-08-17T17:42:36.929274+0000","last_unstale":"2024-08-17T17:44:50.225574+0000","last_undegraded":"2024-08-17T17:44:50.225574+0000","last_fullsized":"2024-08-17T17:44:50.225574+0000","mapping_epoch":357,"log_start":"369'2600","ondisk_log_start":"369'2600","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6542,"log_dups_size":0,"ondisk_log_size":6542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:29:35.955743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374472704,"num_objects":5714,"num_object_clones":0,"num_object_copies":17142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9142,"num_write_kb":475392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"90'149","reported_seq":696,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.104817+0000","last_change":"2024-08-17T17:42:20.927007+0000","last_active":"2024-08-17T17:44:38.104817+0000","last_peered":"2024-08-17T17:44:38.104817+0000","last_clean":"2024-08-17T17:44:38.104817+0000","last_became_active":"2024-08-17T17:41:37.670914+0000","last_became_peered":"2024-08-17T17:41:37.670914+0000","last_unstale":"2024-08-17T17:44:38.104817+0000","last_undegraded":"2024-08-17T17:44:38.104817+0000","last_fullsized":"2024-08-17T17:44:38.104817+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:20.926924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:20.926924+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:57:44.156600+0000","scrub_duration":0.0018485109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":304,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.123432+0000","last_change":"2024-08-17T17:33:44.154097+0000","last_active":"2024-08-17T17:44:38.123432+0000","last_peered":"2024-08-17T17:44:38.123432+0000","last_clean":"2024-08-17T17:44:38.123432+0000","last_became_active":"2024-08-17T17:33:44.152972+0000","last_became_peered":"2024-08-17T17:33:44.152972+0000","last_unstale":"2024-08-17T17:44:38.123432+0000","last_undegraded":"2024-08-17T17:44:38.123432+0000","last_fullsized":"2024-08-17T17:44:38.123432+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:42.602017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":668,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.131133+0000","last_change":"2024-08-17T17:41:34.615197+0000","last_active":"2024-08-17T17:44:38.131133+0000","last_peered":"2024-08-17T17:44:38.131133+0000","last_clean":"2024-08-17T17:44:38.131133+0000","last_became_active":"2024-08-17T17:41:34.614873+0000","last_became_peered":"2024-08-17T17:41:34.614873+0000","last_unstale":"2024-08-17T17:44:38.131133+0000","last_undegraded":"2024-08-17T17:44:38.131133+0000","last_fullsized":"2024-08-17T17:44:38.131133+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:21:57.391953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2","version":"387'9017","reported_seq":9088,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:50.944921+0000","last_change":"2024-08-17T17:42:36.618260+0000","last_active":"2024-08-17T17:44:50.944921+0000","last_peered":"2024-08-17T17:44:50.944921+0000","last_clean":"2024-08-17T17:44:50.944921+0000","last_became_active":"2024-08-17T17:42:36.617875+0000","last_became_peered":"2024-08-17T17:42:36.617875+0000","last_unstale":"2024-08-17T17:44:50.944921+0000","last_undegraded":"2024-08-17T17:44:50.944921+0000","last_fullsized":"2024-08-17T17:44:50.944921+0000","mapping_epoch":357,"log_start":"369'2000","ondisk_log_start":"369'2000","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":7017,"log_dups_size":0,"ondisk_log_size":7017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:09:07.102359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358940672,"num_objects":5477,"num_object_clones":0,"num_object_copies":16431,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9017,"num_write_kb":463808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":819,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.120619+0000","last_change":"2024-08-17T17:41:44.585386+0000","last_active":"2024-08-17T17:44:38.120619+0000","last_peered":"2024-08-17T17:44:38.120619+0000","last_clean":"2024-08-17T17:44:38.120619+0000","last_became_active":"2024-08-17T17:41:12.307398+0000","last_became_peered":"2024-08-17T17:41:12.307398+0000","last_unstale":"2024-08-17T17:44:38.120619+0000","last_undegraded":"2024-08-17T17:44:38.120619+0000","last_fullsized":"2024-08-17T17:44:38.120619+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:44.583653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:44.583653+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:27:44.756681+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":704,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.120847+0000","last_change":"2024-08-17T17:41:34.630434+0000","last_active":"2024-08-17T17:44:38.120847+0000","last_peered":"2024-08-17T17:44:38.120847+0000","last_clean":"2024-08-17T17:44:38.120847+0000","last_became_active":"2024-08-17T17:41:34.616406+0000","last_became_peered":"2024-08-17T17:41:34.616406+0000","last_unstale":"2024-08-17T17:44:38.120847+0000","last_undegraded":"2024-08-17T17:44:38.120847+0000","last_fullsized":"2024-08-17T17:44:38.120847+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:30:44.817203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":52,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.105189+0000","last_change":"2024-08-17T17:41:34.611049+0000","last_active":"2024-08-17T17:44:38.105189+0000","last_peered":"2024-08-17T17:44:38.105189+0000","last_clean":"2024-08-17T17:44:38.105189+0000","last_became_active":"2024-08-17T17:41:34.610727+0000","last_became_peered":"2024-08-17T17:41:34.610727+0000","last_unstale":"2024-08-17T17:44:38.105189+0000","last_undegraded":"2024-08-17T17:44:38.105189+0000","last_fullsized":"2024-08-17T17:44:38.105189+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:04:05.477660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"387'9263","reported_seq":9335,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:49.588128+0000","last_change":"2024-08-17T17:42:36.946226+0000","last_active":"2024-08-17T17:44:49.588128+0000","last_peered":"2024-08-17T17:44:49.588128+0000","last_clean":"2024-08-17T17:44:49.588128+0000","last_became_active":"2024-08-17T17:42:36.945809+0000","last_became_peered":"2024-08-17T17:42:36.945809+0000","last_unstale":"2024-08-17T17:44:49.588128+0000","last_undegraded":"2024-08-17T17:44:49.588128+0000","last_fullsized":"2024-08-17T17:44:49.588128+0000","mapping_epoch":357,"log_start":"370'3000","ondisk_log_start":"370'3000","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6263,"log_dups_size":0,"ondisk_log_size":6263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:04:58.633957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":377815040,"num_objects":5765,"num_object_clones":0,"num_object_copies":17295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9263,"num_write_kb":480896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":847,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.120298+0000","last_change":"2024-08-17T17:43:55.595647+0000","last_active":"2024-08-17T17:44:38.120298+0000","last_peered":"2024-08-17T17:44:38.120298+0000","last_clean":"2024-08-17T17:44:38.120298+0000","last_became_active":"2024-08-17T17:43:55.595256+0000","last_became_peered":"2024-08-17T17:43:55.595256+0000","last_unstale":"2024-08-17T17:44:38.120298+0000","last_undegraded":"2024-08-17T17:44:38.120298+0000","last_fullsized":"2024-08-17T17:44:38.120298+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:46.587712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:46.587712+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:05:01.927968+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":52,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.106422+0000","last_change":"2024-08-17T17:41:34.622803+0000","last_active":"2024-08-17T17:44:38.106422+0000","last_peered":"2024-08-17T17:44:38.106422+0000","last_clean":"2024-08-17T17:44:38.106422+0000","last_became_active":"2024-08-17T17:41:34.622525+0000","last_became_peered":"2024-08-17T17:41:34.622525+0000","last_unstale":"2024-08-17T17:44:38.106422+0000","last_undegraded":"2024-08-17T17:44:38.106422+0000","last_fullsized":"2024-08-17T17:44:38.106422+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:42:39.728377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":52,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.104790+0000","last_change":"2024-08-17T17:41:34.619268+0000","last_active":"2024-08-17T17:44:38.104790+0000","last_peered":"2024-08-17T17:44:38.104790+0000","last_clean":"2024-08-17T17:44:38.104790+0000","last_became_active":"2024-08-17T17:41:34.618914+0000","last_became_peered":"2024-08-17T17:41:34.618914+0000","last_unstale":"2024-08-17T17:44:38.104790+0000","last_undegraded":"2024-08-17T17:44:38.104790+0000","last_fullsized":"2024-08-17T17:44:38.104790+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:08:54.248386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.0","version":"388'9134","reported_seq":9173,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-08-17T17:44:53.827752+0000","last_change":"2024-08-17T17:42:36.617537+0000","last_active":"2024-08-17T17:44:53.827752+0000","last_peered":"2024-08-17T17:44:53.827752+0000","last_clean":"2024-08-17T17:44:53.827752+0000","last_became_active":"2024-08-17T17:42:36.617201+0000","last_became_peered":"2024-08-17T17:42:36.617201+0000","last_unstale":"2024-08-17T17:44:53.827752+0000","last_undegraded":"2024-08-17T17:44:53.827752+0000","last_fullsized":"2024-08-17T17:44:53.827752+0000","mapping_epoch":357,"log_start":"370'2700","ondisk_log_start":"370'2700","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6434,"log_dups_size":0,"ondisk_log_size":6434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:51:58.927214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342884352,"num_objects":5232,"num_object_clones":0,"num_object_copies":15696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9134,"num_write_kb":459712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"90'149","reported_seq":852,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.114263+0000","last_change":"2024-08-17T17:41:41.743931+0000","last_active":"2024-08-17T17:44:38.114263+0000","last_peered":"2024-08-17T17:44:38.114263+0000","last_clean":"2024-08-17T17:44:38.114263+0000","last_became_active":"2024-08-17T17:41:12.203771+0000","last_became_peered":"2024-08-17T17:41:12.203771+0000","last_unstale":"2024-08-17T17:44:38.114263+0000","last_undegraded":"2024-08-17T17:44:38.114263+0000","last_fullsized":"2024-08-17T17:44:38.114263+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:41.743570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:41.743570+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:21:03.865738+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":52,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.103651+0000","last_change":"2024-08-17T17:41:34.607602+0000","last_active":"2024-08-17T17:44:38.103651+0000","last_peered":"2024-08-17T17:44:38.103651+0000","last_clean":"2024-08-17T17:44:38.103651+0000","last_became_active":"2024-08-17T17:41:34.607300+0000","last_became_peered":"2024-08-17T17:41:34.607300+0000","last_unstale":"2024-08-17T17:44:38.103651+0000","last_undegraded":"2024-08-17T17:44:38.103651+0000","last_fullsized":"2024-08-17T17:44:38.103651+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:20:31.486698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":52,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.104682+0000","last_change":"2024-08-17T17:41:34.622102+0000","last_active":"2024-08-17T17:44:38.104682+0000","last_peered":"2024-08-17T17:44:38.104682+0000","last_clean":"2024-08-17T17:44:38.104682+0000","last_became_active":"2024-08-17T17:41:34.621733+0000","last_became_peered":"2024-08-17T17:41:34.621733+0000","last_unstale":"2024-08-17T17:44:38.104682+0000","last_undegraded":"2024-08-17T17:44:38.104682+0000","last_fullsized":"2024-08-17T17:44:38.104682+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:39:41.668709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"388'9578","reported_seq":9615,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-08-17T17:44:53.702893+0000","last_change":"2024-08-17T17:42:36.620155+0000","last_active":"2024-08-17T17:44:53.702893+0000","last_peered":"2024-08-17T17:44:53.702893+0000","last_clean":"2024-08-17T17:44:53.702893+0000","last_became_active":"2024-08-17T17:42:36.619813+0000","last_became_peered":"2024-08-17T17:42:36.619813+0000","last_unstale":"2024-08-17T17:44:53.702893+0000","last_undegraded":"2024-08-17T17:44:53.702893+0000","last_fullsized":"2024-08-17T17:44:53.702893+0000","mapping_epoch":357,"log_start":"372'4100","ondisk_log_start":"372'4100","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":5478,"log_dups_size":0,"ondisk_log_size":5478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:16:27.534439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362020864,"num_objects":5524,"num_object_clones":0,"num_object_copies":16572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9578,"num_write_kb":483264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":505,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.116312+0000","last_change":"2024-08-17T17:41:41.036615+0000","last_active":"2024-08-17T17:44:38.116312+0000","last_peered":"2024-08-17T17:44:38.116312+0000","last_clean":"2024-08-17T17:44:38.116312+0000","last_became_active":"2024-08-17T17:41:12.329214+0000","last_became_peered":"2024-08-17T17:41:12.329214+0000","last_unstale":"2024-08-17T17:44:38.116312+0000","last_undegraded":"2024-08-17T17:44:38.116312+0000","last_fullsized":"2024-08-17T17:44:38.116312+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:41.036524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:41.036524+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:59:58.695900+0000","scrub_duration":0.001820803,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":662,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.120416+0000","last_change":"2024-08-17T17:33:20.796216+0000","last_active":"2024-08-17T17:44:38.120416+0000","last_peered":"2024-08-17T17:44:38.120416+0000","last_clean":"2024-08-17T17:44:38.120416+0000","last_became_active":"2024-08-17T17:33:20.795762+0000","last_became_peered":"2024-08-17T17:33:20.795762+0000","last_unstale":"2024-08-17T17:44:38.120416+0000","last_undegraded":"2024-08-17T17:44:38.120416+0000","last_fullsized":"2024-08-17T17:44:38.120416+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:47.038509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":557,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.109316+0000","last_change":"2024-08-17T17:41:34.625551+0000","last_active":"2024-08-17T17:44:38.109316+0000","last_peered":"2024-08-17T17:44:38.109316+0000","last_clean":"2024-08-17T17:44:38.109316+0000","last_became_active":"2024-08-17T17:41:34.625251+0000","last_became_peered":"2024-08-17T17:41:34.625251+0000","last_unstale":"2024-08-17T17:44:38.109316+0000","last_undegraded":"2024-08-17T17:44:38.109316+0000","last_fullsized":"2024-08-17T17:44:38.109316+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:28:22.354213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"388'9470","reported_seq":9509,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-08-17T17:44:53.814069+0000","last_change":"2024-08-17T17:42:36.619525+0000","last_active":"2024-08-17T17:44:53.814069+0000","last_peered":"2024-08-17T17:44:53.814069+0000","last_clean":"2024-08-17T17:44:53.814069+0000","last_became_active":"2024-08-17T17:42:36.619074+0000","last_became_peered":"2024-08-17T17:42:36.619074+0000","last_unstale":"2024-08-17T17:44:53.814069+0000","last_undegraded":"2024-08-17T17:44:53.814069+0000","last_fullsized":"2024-08-17T17:44:53.814069+0000","mapping_epoch":357,"log_start":"370'3000","ondisk_log_start":"370'3000","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6470,"log_dups_size":0,"ondisk_log_size":6470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:11:32.849616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352190464,"num_objects":5374,"num_object_clones":0,"num_object_copies":16122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9470,"num_write_kb":475008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":799,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.126027+0000","last_change":"2024-08-17T17:38:25.247159+0000","last_active":"2024-08-17T17:44:38.126027+0000","last_peered":"2024-08-17T17:44:38.126027+0000","last_clean":"2024-08-17T17:44:38.126027+0000","last_became_active":"2024-08-17T17:38:21.311361+0000","last_became_peered":"2024-08-17T17:38:21.311361+0000","last_unstale":"2024-08-17T17:44:38.126027+0000","last_undegraded":"2024-08-17T17:44:38.126027+0000","last_fullsized":"2024-08-17T17:44:38.126027+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:25.247027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:25.247027+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:46:21.734712+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":794,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.108259+0000","last_change":"2024-08-17T17:33:44.149928+0000","last_active":"2024-08-17T17:44:38.108259+0000","last_peered":"2024-08-17T17:44:38.108259+0000","last_clean":"2024-08-17T17:44:38.108259+0000","last_became_active":"2024-08-17T17:33:44.149498+0000","last_became_peered":"2024-08-17T17:33:44.149498+0000","last_unstale":"2024-08-17T17:44:38.108259+0000","last_undegraded":"2024-08-17T17:44:38.108259+0000","last_fullsized":"2024-08-17T17:44:38.108259+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:17:04.963202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":610,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.121186+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:44:38.121186+0000","last_peered":"2024-08-17T17:44:38.121186+0000","last_clean":"2024-08-17T17:44:38.121186+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:44:38.121186+0000","last_undegraded":"2024-08-17T17:44:38.121186+0000","last_fullsized":"2024-08-17T17:44:38.121186+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":229,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.116613+0000","last_change":"2024-08-17T17:41:34.618631+0000","last_active":"2024-08-17T17:44:38.116613+0000","last_peered":"2024-08-17T17:44:38.116613+0000","last_clean":"2024-08-17T17:44:38.116613+0000","last_became_active":"2024-08-17T17:41:34.618313+0000","last_became_peered":"2024-08-17T17:41:34.618313+0000","last_unstale":"2024-08-17T17:44:38.116613+0000","last_undegraded":"2024-08-17T17:44:38.116613+0000","last_fullsized":"2024-08-17T17:44:38.116613+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:07:13.915088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"387'9078","reported_seq":9150,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:50.233945+0000","last_change":"2024-08-17T17:42:36.929502+0000","last_active":"2024-08-17T17:44:50.233945+0000","last_peered":"2024-08-17T17:44:50.233945+0000","last_clean":"2024-08-17T17:44:50.233945+0000","last_became_active":"2024-08-17T17:42:36.929113+0000","last_became_peered":"2024-08-17T17:42:36.929113+0000","last_unstale":"2024-08-17T17:44:50.233945+0000","last_undegraded":"2024-08-17T17:44:50.233945+0000","last_fullsized":"2024-08-17T17:44:50.233945+0000","mapping_epoch":357,"log_start":"369'2500","ondisk_log_start":"369'2500","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6578,"log_dups_size":0,"ondisk_log_size":6578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:31:25.879221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365035520,"num_objects":5570,"num_object_clones":0,"num_object_copies":16710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9078,"num_write_kb":468736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":835,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.114439+0000","last_change":"2024-08-17T17:41:37.676347+0000","last_active":"2024-08-17T17:44:38.114439+0000","last_peered":"2024-08-17T17:44:38.114439+0000","last_clean":"2024-08-17T17:44:38.114439+0000","last_became_active":"2024-08-17T17:41:37.675869+0000","last_became_peered":"2024-08-17T17:41:37.675869+0000","last_unstale":"2024-08-17T17:44:38.114439+0000","last_undegraded":"2024-08-17T17:44:38.114439+0000","last_fullsized":"2024-08-17T17:44:38.114439+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:21.814077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:21.814077+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:04:16.240754+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":893,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.107641+0000","last_change":"2024-08-17T17:33:20.797705+0000","last_active":"2024-08-17T17:44:38.107641+0000","last_peered":"2024-08-17T17:44:38.107641+0000","last_clean":"2024-08-17T17:44:38.107641+0000","last_became_active":"2024-08-17T17:33:20.797172+0000","last_became_peered":"2024-08-17T17:33:20.797172+0000","last_unstale":"2024-08-17T17:44:38.107641+0000","last_undegraded":"2024-08-17T17:44:38.107641+0000","last_fullsized":"2024-08-17T17:44:38.107641+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:30.754999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":659,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.126178+0000","last_change":"2024-08-17T17:41:34.614108+0000","last_active":"2024-08-17T17:44:38.126178+0000","last_peered":"2024-08-17T17:44:38.126178+0000","last_clean":"2024-08-17T17:44:38.126178+0000","last_became_active":"2024-08-17T17:41:34.613481+0000","last_became_peered":"2024-08-17T17:41:34.613481+0000","last_unstale":"2024-08-17T17:44:38.126178+0000","last_undegraded":"2024-08-17T17:44:38.126178+0000","last_fullsized":"2024-08-17T17:44:38.126178+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:21:02.346115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":447,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.137281+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:44:38.137281+0000","last_peered":"2024-08-17T17:44:38.137281+0000","last_clean":"2024-08-17T17:44:38.137281+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:44:38.137281+0000","last_undegraded":"2024-08-17T17:44:38.137281+0000","last_fullsized":"2024-08-17T17:44:38.137281+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"388'9295","reported_seq":9334,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-08-17T17:44:53.694755+0000","last_change":"2024-08-17T17:42:36.946382+0000","last_active":"2024-08-17T17:44:53.694755+0000","last_peered":"2024-08-17T17:44:53.694755+0000","last_clean":"2024-08-17T17:44:53.694755+0000","last_became_active":"2024-08-17T17:42:36.945591+0000","last_became_peered":"2024-08-17T17:42:36.945591+0000","last_unstale":"2024-08-17T17:44:53.694755+0000","last_undegraded":"2024-08-17T17:44:53.694755+0000","last_fullsized":"2024-08-17T17:44:53.694755+0000","mapping_epoch":357,"log_start":"370'3000","ondisk_log_start":"370'3000","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6295,"log_dups_size":0,"ondisk_log_size":6295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:08:17.912891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358285312,"num_objects":5467,"num_object_clones":0,"num_object_copies":16401,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9295,"num_write_kb":472384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":839,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.126537+0000","last_change":"2024-08-17T17:41:37.683450+0000","last_active":"2024-08-17T17:44:38.126537+0000","last_peered":"2024-08-17T17:44:38.126537+0000","last_clean":"2024-08-17T17:44:38.126537+0000","last_became_active":"2024-08-17T17:41:37.683103+0000","last_became_peered":"2024-08-17T17:41:37.683103+0000","last_unstale":"2024-08-17T17:44:38.126537+0000","last_undegraded":"2024-08-17T17:44:38.126537+0000","last_fullsized":"2024-08-17T17:44:38.126537+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.289528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.289528+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:39:17.141469+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":1026,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.120986+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:44:38.120986+0000","last_peered":"2024-08-17T17:44:38.120986+0000","last_clean":"2024-08-17T17:44:38.120986+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:44:38.120986+0000","last_undegraded":"2024-08-17T17:44:38.120986+0000","last_fullsized":"2024-08-17T17:44:38.120986+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":531,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.107951+0000","last_change":"2024-08-17T17:33:20.793468+0000","last_active":"2024-08-17T17:44:38.107951+0000","last_peered":"2024-08-17T17:44:38.107951+0000","last_clean":"2024-08-17T17:44:38.107951+0000","last_became_active":"2024-08-17T17:33:20.793166+0000","last_became_peered":"2024-08-17T17:33:20.793166+0000","last_unstale":"2024-08-17T17:44:38.107951+0000","last_undegraded":"2024-08-17T17:44:38.107951+0000","last_fullsized":"2024-08-17T17:44:38.107951+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:12:52.121050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":567,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.109238+0000","last_change":"2024-08-17T17:33:20.835784+0000","last_active":"2024-08-17T17:44:38.109238+0000","last_peered":"2024-08-17T17:44:38.109238+0000","last_clean":"2024-08-17T17:44:38.109238+0000","last_became_active":"2024-08-17T17:33:20.835371+0000","last_became_peered":"2024-08-17T17:33:20.835371+0000","last_unstale":"2024-08-17T17:44:38.109238+0000","last_undegraded":"2024-08-17T17:44:38.109238+0000","last_fullsized":"2024-08-17T17:44:38.109238+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:35:43.369055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"388'9386","reported_seq":9425,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-08-17T17:44:53.821780+0000","last_change":"2024-08-17T17:42:36.946026+0000","last_active":"2024-08-17T17:44:53.821780+0000","last_peered":"2024-08-17T17:44:53.821780+0000","last_clean":"2024-08-17T17:44:53.821780+0000","last_became_active":"2024-08-17T17:42:36.945638+0000","last_became_peered":"2024-08-17T17:42:36.945638+0000","last_unstale":"2024-08-17T17:44:53.821780+0000","last_undegraded":"2024-08-17T17:44:53.821780+0000","last_fullsized":"2024-08-17T17:44:53.821780+0000","mapping_epoch":357,"log_start":"370'3000","ondisk_log_start":"370'3000","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6386,"log_dups_size":0,"ondisk_log_size":6386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:07:34.864262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350486528,"num_objects":5348,"num_object_clones":0,"num_object_copies":16044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9386,"num_write_kb":471488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":818,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.126121+0000","last_change":"2024-08-17T17:38:22.254841+0000","last_active":"2024-08-17T17:44:38.126121+0000","last_peered":"2024-08-17T17:44:38.126121+0000","last_clean":"2024-08-17T17:44:38.126121+0000","last_became_active":"2024-08-17T17:38:21.306619+0000","last_became_peered":"2024-08-17T17:38:21.306619+0000","last_unstale":"2024-08-17T17:44:38.126121+0000","last_undegraded":"2024-08-17T17:44:38.126121+0000","last_fullsized":"2024-08-17T17:44:38.126121+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:22.254705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:22.254705+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:47:38.184089+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":1116,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.114974+0000","last_change":"2024-08-17T17:33:44.148227+0000","last_active":"2024-08-17T17:44:38.114974+0000","last_peered":"2024-08-17T17:44:38.114974+0000","last_clean":"2024-08-17T17:44:38.114974+0000","last_became_active":"2024-08-17T17:33:44.147627+0000","last_became_peered":"2024-08-17T17:33:44.147627+0000","last_unstale":"2024-08-17T17:44:38.114974+0000","last_undegraded":"2024-08-17T17:44:38.114974+0000","last_fullsized":"2024-08-17T17:44:38.114974+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:29:42.898460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1372,"num_read_kb":2843,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":52,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.103946+0000","last_change":"2024-08-17T17:41:34.611109+0000","last_active":"2024-08-17T17:44:38.103946+0000","last_peered":"2024-08-17T17:44:38.103946+0000","last_clean":"2024-08-17T17:44:38.103946+0000","last_became_active":"2024-08-17T17:41:34.610170+0000","last_became_peered":"2024-08-17T17:41:34.610170+0000","last_unstale":"2024-08-17T17:44:38.103946+0000","last_undegraded":"2024-08-17T17:44:38.103946+0000","last_fullsized":"2024-08-17T17:44:38.103946+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:39:05.301260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":571,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.108578+0000","last_change":"2024-08-17T17:33:20.772454+0000","last_active":"2024-08-17T17:44:38.108578+0000","last_peered":"2024-08-17T17:44:38.108578+0000","last_clean":"2024-08-17T17:44:38.108578+0000","last_became_active":"2024-08-17T17:33:20.771848+0000","last_became_peered":"2024-08-17T17:33:20.771848+0000","last_unstale":"2024-08-17T17:44:38.108578+0000","last_undegraded":"2024-08-17T17:44:38.108578+0000","last_fullsized":"2024-08-17T17:44:38.108578+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:46:42.300744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"388'9675","reported_seq":9712,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-08-17T17:44:53.662751+0000","last_change":"2024-08-17T17:42:36.618414+0000","last_active":"2024-08-17T17:44:53.662751+0000","last_peered":"2024-08-17T17:44:53.662751+0000","last_clean":"2024-08-17T17:44:53.662751+0000","last_became_active":"2024-08-17T17:42:36.618086+0000","last_became_peered":"2024-08-17T17:42:36.618086+0000","last_unstale":"2024-08-17T17:44:53.662751+0000","last_undegraded":"2024-08-17T17:44:53.662751+0000","last_fullsized":"2024-08-17T17:44:53.662751+0000","mapping_epoch":357,"log_start":"371'3400","ondisk_log_start":"371'3400","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6275,"log_dups_size":0,"ondisk_log_size":6275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:26:27.772894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":361168896,"num_objects":5511,"num_object_clones":0,"num_object_copies":16533,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9675,"num_write_kb":485952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":525,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.119443+0000","last_change":"2024-08-17T17:41:43.035215+0000","last_active":"2024-08-17T17:44:38.119443+0000","last_peered":"2024-08-17T17:44:38.119443+0000","last_clean":"2024-08-17T17:44:38.119443+0000","last_became_active":"2024-08-17T17:41:12.201158+0000","last_became_peered":"2024-08-17T17:41:12.201158+0000","last_unstale":"2024-08-17T17:44:38.119443+0000","last_undegraded":"2024-08-17T17:44:38.119443+0000","last_fullsized":"2024-08-17T17:44:38.119443+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:43.035108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:43.035108+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:43:27.108994+0000","scrub_duration":0.0028877400000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":322,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.124991+0000","last_change":"2024-08-17T17:33:44.163867+0000","last_active":"2024-08-17T17:44:38.124991+0000","last_peered":"2024-08-17T17:44:38.124991+0000","last_clean":"2024-08-17T17:44:38.124991+0000","last_became_active":"2024-08-17T17:33:44.163503+0000","last_became_peered":"2024-08-17T17:33:44.163503+0000","last_unstale":"2024-08-17T17:44:38.124991+0000","last_undegraded":"2024-08-17T17:44:38.124991+0000","last_fullsized":"2024-08-17T17:44:38.124991+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:31:53.573596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":786,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.108326+0000","last_change":"2024-08-17T17:33:20.799350+0000","last_active":"2024-08-17T17:44:38.108326+0000","last_peered":"2024-08-17T17:44:38.108326+0000","last_clean":"2024-08-17T17:44:38.108326+0000","last_became_active":"2024-08-17T17:33:20.798925+0000","last_became_peered":"2024-08-17T17:33:20.798925+0000","last_unstale":"2024-08-17T17:44:38.108326+0000","last_undegraded":"2024-08-17T17:44:38.108326+0000","last_fullsized":"2024-08-17T17:44:38.108326+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:16:36.121295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":602,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.115432+0000","last_change":"2024-08-17T17:41:34.627064+0000","last_active":"2024-08-17T17:44:38.115432+0000","last_peered":"2024-08-17T17:44:38.115432+0000","last_clean":"2024-08-17T17:44:38.115432+0000","last_became_active":"2024-08-17T17:41:34.626453+0000","last_became_peered":"2024-08-17T17:41:34.626453+0000","last_unstale":"2024-08-17T17:44:38.115432+0000","last_undegraded":"2024-08-17T17:44:38.115432+0000","last_fullsized":"2024-08-17T17:44:38.115432+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:02:11.894042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"388'9441","reported_seq":9481,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-08-17T17:44:53.799231+0000","last_change":"2024-08-17T17:42:36.620669+0000","last_active":"2024-08-17T17:44:53.799231+0000","last_peered":"2024-08-17T17:44:53.799231+0000","last_clean":"2024-08-17T17:44:53.799231+0000","last_became_active":"2024-08-17T17:42:36.619876+0000","last_became_peered":"2024-08-17T17:42:36.619876+0000","last_unstale":"2024-08-17T17:44:53.799231+0000","last_undegraded":"2024-08-17T17:44:53.799231+0000","last_fullsized":"2024-08-17T17:44:53.799231+0000","mapping_epoch":357,"log_start":"370'3000","ondisk_log_start":"370'3000","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6441,"log_dups_size":0,"ondisk_log_size":6441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:59:34.661001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355991576,"num_objects":5433,"num_object_clones":0,"num_object_copies":16299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5433,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9441,"num_write_kb":475905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":525,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.116680+0000","last_change":"2024-08-17T17:41:44.058291+0000","last_active":"2024-08-17T17:44:38.116680+0000","last_peered":"2024-08-17T17:44:38.116680+0000","last_clean":"2024-08-17T17:44:38.116680+0000","last_became_active":"2024-08-17T17:41:12.200077+0000","last_became_peered":"2024-08-17T17:41:12.200077+0000","last_unstale":"2024-08-17T17:44:38.116680+0000","last_undegraded":"2024-08-17T17:44:38.116680+0000","last_fullsized":"2024-08-17T17:44:38.116680+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:44.058233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:44.058233+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:42:55.911623+0000","scrub_duration":0.0074268499999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":606,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.108219+0000","last_change":"2024-08-17T17:33:20.798613+0000","last_active":"2024-08-17T17:44:38.108219+0000","last_peered":"2024-08-17T17:44:38.108219+0000","last_clean":"2024-08-17T17:44:38.108219+0000","last_became_active":"2024-08-17T17:33:20.798260+0000","last_became_peered":"2024-08-17T17:33:20.798260+0000","last_unstale":"2024-08-17T17:44:38.108219+0000","last_undegraded":"2024-08-17T17:44:38.108219+0000","last_fullsized":"2024-08-17T17:44:38.108219+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:18:00.822681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":948,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.109047+0000","last_change":"2024-08-17T17:41:37.683212+0000","last_active":"2024-08-17T17:44:38.109047+0000","last_peered":"2024-08-17T17:44:38.109047+0000","last_clean":"2024-08-17T17:44:38.109047+0000","last_became_active":"2024-08-17T17:41:37.682730+0000","last_became_peered":"2024-08-17T17:41:37.682730+0000","last_unstale":"2024-08-17T17:44:38.109047+0000","last_undegraded":"2024-08-17T17:44:38.109047+0000","last_fullsized":"2024-08-17T17:44:38.109047+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:51:35.444433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":621,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.131050+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:44:38.131050+0000","last_peered":"2024-08-17T17:44:38.131050+0000","last_clean":"2024-08-17T17:44:38.131050+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:44:38.131050+0000","last_undegraded":"2024-08-17T17:44:38.131050+0000","last_fullsized":"2024-08-17T17:44:38.131050+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"388'9490","reported_seq":9527,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-08-17T17:44:53.701575+0000","last_change":"2024-08-17T17:42:36.929709+0000","last_active":"2024-08-17T17:44:53.701575+0000","last_peered":"2024-08-17T17:44:53.701575+0000","last_clean":"2024-08-17T17:44:53.701575+0000","last_became_active":"2024-08-17T17:42:36.929285+0000","last_became_peered":"2024-08-17T17:42:36.929285+0000","last_unstale":"2024-08-17T17:44:53.701575+0000","last_undegraded":"2024-08-17T17:44:53.701575+0000","last_fullsized":"2024-08-17T17:44:53.701575+0000","mapping_epoch":357,"log_start":"372'4000","ondisk_log_start":"372'4000","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":5490,"log_dups_size":0,"ondisk_log_size":5490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:47:39.760414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352845824,"num_objects":5384,"num_object_clones":0,"num_object_copies":16152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9490,"num_write_kb":475968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":771,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.104488+0000","last_change":"2024-08-17T17:42:19.960161+0000","last_active":"2024-08-17T17:44:38.104488+0000","last_peered":"2024-08-17T17:44:38.104488+0000","last_clean":"2024-08-17T17:44:38.104488+0000","last_became_active":"2024-08-17T17:41:37.665223+0000","last_became_peered":"2024-08-17T17:41:37.665223+0000","last_unstale":"2024-08-17T17:44:38.104488+0000","last_undegraded":"2024-08-17T17:44:38.104488+0000","last_fullsized":"2024-08-17T17:44:38.104488+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:19.960086+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:19.960086+0000","last_clean_scrub_stamp":"2024-08-17T17:42:19.960086+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:57.861661+0000","scrub_duration":0.0022612000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":656,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.123672+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:44:38.123672+0000","last_peered":"2024-08-17T17:44:38.123672+0000","last_clean":"2024-08-17T17:44:38.123672+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:44:38.123672+0000","last_undegraded":"2024-08-17T17:44:38.123672+0000","last_fullsized":"2024-08-17T17:44:38.123672+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":1231,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.108950+0000","last_change":"2024-08-17T17:41:37.677524+0000","last_active":"2024-08-17T17:44:38.108950+0000","last_peered":"2024-08-17T17:44:38.108950+0000","last_clean":"2024-08-17T17:44:38.108950+0000","last_became_active":"2024-08-17T17:41:37.676764+0000","last_became_peered":"2024-08-17T17:41:37.676764+0000","last_unstale":"2024-08-17T17:44:38.108950+0000","last_undegraded":"2024-08-17T17:44:38.108950+0000","last_fullsized":"2024-08-17T17:44:38.108950+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:14:49.341080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":653,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.120323+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:44:38.120323+0000","last_peered":"2024-08-17T17:44:38.120323+0000","last_clean":"2024-08-17T17:44:38.120323+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:44:38.120323+0000","last_undegraded":"2024-08-17T17:44:38.120323+0000","last_fullsized":"2024-08-17T17:44:38.120323+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"388'9354","reported_seq":9393,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-08-17T17:44:53.811331+0000","last_change":"2024-08-17T17:42:36.929613+0000","last_active":"2024-08-17T17:44:53.811331+0000","last_peered":"2024-08-17T17:44:53.811331+0000","last_clean":"2024-08-17T17:44:53.811331+0000","last_became_active":"2024-08-17T17:42:36.929191+0000","last_became_peered":"2024-08-17T17:42:36.929191+0000","last_unstale":"2024-08-17T17:44:53.811331+0000","last_undegraded":"2024-08-17T17:44:53.811331+0000","last_fullsized":"2024-08-17T17:44:53.811331+0000","mapping_epoch":357,"log_start":"370'2900","ondisk_log_start":"370'2900","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6454,"log_dups_size":0,"ondisk_log_size":6454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:21:40.881779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351666176,"num_objects":5366,"num_object_clones":0,"num_object_copies":16098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9354,"num_write_kb":471040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":800,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.123033+0000","last_change":"2024-08-17T17:38:23.113397+0000","last_active":"2024-08-17T17:44:38.123033+0000","last_peered":"2024-08-17T17:44:38.123033+0000","last_clean":"2024-08-17T17:44:38.123033+0000","last_became_active":"2024-08-17T17:38:21.305580+0000","last_became_peered":"2024-08-17T17:38:21.305580+0000","last_unstale":"2024-08-17T17:44:38.123033+0000","last_undegraded":"2024-08-17T17:44:38.123033+0000","last_fullsized":"2024-08-17T17:44:38.123033+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.113279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.113279+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:38:07.015721+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":493,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.136704+0000","last_change":"2024-08-17T17:41:34.629522+0000","last_active":"2024-08-17T17:44:38.136704+0000","last_peered":"2024-08-17T17:44:38.136704+0000","last_clean":"2024-08-17T17:44:38.136704+0000","last_became_active":"2024-08-17T17:41:34.619340+0000","last_became_peered":"2024-08-17T17:41:34.619340+0000","last_unstale":"2024-08-17T17:44:38.136704+0000","last_undegraded":"2024-08-17T17:44:38.136704+0000","last_fullsized":"2024-08-17T17:44:38.136704+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:30:03.902956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":618,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.119724+0000","last_change":"2024-08-17T17:26:06.855307+0000","last_active":"2024-08-17T17:44:38.119724+0000","last_peered":"2024-08-17T17:44:38.119724+0000","last_clean":"2024-08-17T17:44:38.119724+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:44:38.119724+0000","last_undegraded":"2024-08-17T17:44:38.119724+0000","last_fullsized":"2024-08-17T17:44:38.119724+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":52,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.103525+0000","last_change":"2024-08-17T17:41:34.614028+0000","last_active":"2024-08-17T17:44:38.103525+0000","last_peered":"2024-08-17T17:44:38.103525+0000","last_clean":"2024-08-17T17:44:38.103525+0000","last_became_active":"2024-08-17T17:41:34.613706+0000","last_became_peered":"2024-08-17T17:41:34.613706+0000","last_unstale":"2024-08-17T17:44:38.103525+0000","last_undegraded":"2024-08-17T17:44:38.103525+0000","last_fullsized":"2024-08-17T17:44:38.103525+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:19:55.254902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"90'149","reported_seq":711,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.108793+0000","last_change":"2024-08-17T17:42:07.681204+0000","last_active":"2024-08-17T17:44:38.108793+0000","last_peered":"2024-08-17T17:44:38.108793+0000","last_clean":"2024-08-17T17:44:38.108793+0000","last_became_active":"2024-08-17T17:42:05.878769+0000","last_became_peered":"2024-08-17T17:42:05.878769+0000","last_unstale":"2024-08-17T17:44:38.108793+0000","last_undegraded":"2024-08-17T17:44:38.108793+0000","last_fullsized":"2024-08-17T17:44:38.108793+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:07.681064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:07.681064+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:12:12.835715+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":52,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.104011+0000","last_change":"2024-08-17T17:41:34.622180+0000","last_active":"2024-08-17T17:44:38.104011+0000","last_peered":"2024-08-17T17:44:38.104011+0000","last_clean":"2024-08-17T17:44:38.104011+0000","last_became_active":"2024-08-17T17:41:34.621724+0000","last_became_peered":"2024-08-17T17:41:34.621724+0000","last_unstale":"2024-08-17T17:44:38.104011+0000","last_undegraded":"2024-08-17T17:44:38.104011+0000","last_fullsized":"2024-08-17T17:44:38.104011+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:15.807235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":62,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.105266+0000","last_change":"2024-08-17T17:41:34.626584+0000","last_active":"2024-08-17T17:44:38.105266+0000","last_peered":"2024-08-17T17:44:38.105266+0000","last_clean":"2024-08-17T17:44:38.105266+0000","last_became_active":"2024-08-17T17:41:34.609878+0000","last_became_peered":"2024-08-17T17:41:34.609878+0000","last_unstale":"2024-08-17T17:44:38.105266+0000","last_undegraded":"2024-08-17T17:44:38.105266+0000","last_fullsized":"2024-08-17T17:44:38.105266+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:08:12.810347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.17","version":"90'149","reported_seq":470,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.129578+0000","last_change":"2024-08-17T17:42:06.699082+0000","last_active":"2024-08-17T17:44:38.129578+0000","last_peered":"2024-08-17T17:44:38.129578+0000","last_clean":"2024-08-17T17:44:38.129578+0000","last_became_active":"2024-08-17T17:42:05.876218+0000","last_became_peered":"2024-08-17T17:42:05.876218+0000","last_unstale":"2024-08-17T17:44:38.129578+0000","last_undegraded":"2024-08-17T17:44:38.129578+0000","last_fullsized":"2024-08-17T17:44:38.129578+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.698998+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:06.698998+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.698998+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:22:16.510150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":598,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.115066+0000","last_change":"2024-08-17T17:33:20.799402+0000","last_active":"2024-08-17T17:44:38.115066+0000","last_peered":"2024-08-17T17:44:38.115066+0000","last_clean":"2024-08-17T17:44:38.115066+0000","last_became_active":"2024-08-17T17:33:20.798832+0000","last_became_peered":"2024-08-17T17:33:20.798832+0000","last_unstale":"2024-08-17T17:44:38.115066+0000","last_undegraded":"2024-08-17T17:44:38.115066+0000","last_fullsized":"2024-08-17T17:44:38.115066+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:21:05.001134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":374,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.124885+0000","last_change":"2024-08-17T17:41:34.634301+0000","last_active":"2024-08-17T17:44:38.124885+0000","last_peered":"2024-08-17T17:44:38.124885+0000","last_clean":"2024-08-17T17:44:38.124885+0000","last_became_active":"2024-08-17T17:41:34.620508+0000","last_became_peered":"2024-08-17T17:41:34.620508+0000","last_unstale":"2024-08-17T17:44:38.124885+0000","last_undegraded":"2024-08-17T17:44:38.124885+0000","last_fullsized":"2024-08-17T17:44:38.124885+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:10:54.576812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.14","version":"90'149","reported_seq":823,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.127163+0000","last_change":"2024-08-17T17:41:37.664471+0000","last_active":"2024-08-17T17:44:38.127163+0000","last_peered":"2024-08-17T17:44:38.127163+0000","last_clean":"2024-08-17T17:44:38.127163+0000","last_became_active":"2024-08-17T17:41:37.664108+0000","last_became_peered":"2024-08-17T17:41:37.664108+0000","last_unstale":"2024-08-17T17:44:38.127163+0000","last_undegraded":"2024-08-17T17:44:38.127163+0000","last_fullsized":"2024-08-17T17:44:38.127163+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:22.030514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:22.030514+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:15:52.325163+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":337,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.109038+0000","last_change":"2024-08-17T17:33:44.146014+0000","last_active":"2024-08-17T17:44:38.109038+0000","last_peered":"2024-08-17T17:44:38.109038+0000","last_clean":"2024-08-17T17:44:38.109038+0000","last_became_active":"2024-08-17T17:33:44.145668+0000","last_became_peered":"2024-08-17T17:33:44.145668+0000","last_unstale":"2024-08-17T17:44:38.109038+0000","last_undegraded":"2024-08-17T17:44:38.109038+0000","last_fullsized":"2024-08-17T17:44:38.109038+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:53:41.679237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":602,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.120711+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:44:38.120711+0000","last_peered":"2024-08-17T17:44:38.120711+0000","last_clean":"2024-08-17T17:44:38.120711+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:44:38.120711+0000","last_undegraded":"2024-08-17T17:44:38.120711+0000","last_fullsized":"2024-08-17T17:44:38.120711+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"90'149","reported_seq":774,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.109011+0000","last_change":"2024-08-17T17:42:06.456562+0000","last_active":"2024-08-17T17:44:38.109011+0000","last_peered":"2024-08-17T17:44:38.109011+0000","last_clean":"2024-08-17T17:44:38.109011+0000","last_became_active":"2024-08-17T17:42:05.876577+0000","last_became_peered":"2024-08-17T17:42:05.876577+0000","last_unstale":"2024-08-17T17:44:38.109011+0000","last_undegraded":"2024-08-17T17:44:38.109011+0000","last_fullsized":"2024-08-17T17:44:38.109011+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.456490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.456490+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:10:03.654953+0000","scrub_duration":0.001537285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":629,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.129918+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:44:38.129918+0000","last_peered":"2024-08-17T17:44:38.129918+0000","last_clean":"2024-08-17T17:44:38.129918+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:44:38.129918+0000","last_undegraded":"2024-08-17T17:44:38.129918+0000","last_fullsized":"2024-08-17T17:44:38.129918+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":660,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.120229+0000","last_change":"2024-08-17T17:33:20.790687+0000","last_active":"2024-08-17T17:44:38.120229+0000","last_peered":"2024-08-17T17:44:38.120229+0000","last_clean":"2024-08-17T17:44:38.120229+0000","last_became_active":"2024-08-17T17:33:20.790399+0000","last_became_peered":"2024-08-17T17:33:20.790399+0000","last_unstale":"2024-08-17T17:44:38.120229+0000","last_undegraded":"2024-08-17T17:44:38.120229+0000","last_fullsized":"2024-08-17T17:44:38.120229+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:38:06.849989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1a","version":"90'149","reported_seq":436,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.105919+0000","last_change":"2024-08-17T17:42:21.932830+0000","last_active":"2024-08-17T17:44:38.105919+0000","last_peered":"2024-08-17T17:44:38.105919+0000","last_clean":"2024-08-17T17:44:38.105919+0000","last_became_active":"2024-08-17T17:42:05.876835+0000","last_became_peered":"2024-08-17T17:42:05.876835+0000","last_unstale":"2024-08-17T17:44:38.105919+0000","last_undegraded":"2024-08-17T17:44:38.105919+0000","last_fullsized":"2024-08-17T17:44:38.105919+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:21.932757+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:21.932757+0000","last_clean_scrub_stamp":"2024-08-17T17:42:21.932757+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:36:39.848570+0000","scrub_duration":0.0020218810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":361,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.119583+0000","last_change":"2024-08-17T17:33:20.835760+0000","last_active":"2024-08-17T17:44:38.119583+0000","last_peered":"2024-08-17T17:44:38.119583+0000","last_clean":"2024-08-17T17:44:38.119583+0000","last_became_active":"2024-08-17T17:33:20.835410+0000","last_became_peered":"2024-08-17T17:33:20.835410+0000","last_unstale":"2024-08-17T17:44:38.119583+0000","last_undegraded":"2024-08-17T17:44:38.119583+0000","last_fullsized":"2024-08-17T17:44:38.119583+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:45.448022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":678,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.121341+0000","last_change":"2024-08-17T17:33:20.771279+0000","last_active":"2024-08-17T17:44:38.121341+0000","last_peered":"2024-08-17T17:44:38.121341+0000","last_clean":"2024-08-17T17:44:38.121341+0000","last_became_active":"2024-08-17T17:33:20.770728+0000","last_became_peered":"2024-08-17T17:33:20.770728+0000","last_unstale":"2024-08-17T17:44:38.121341+0000","last_undegraded":"2024-08-17T17:44:38.121341+0000","last_fullsized":"2024-08-17T17:44:38.121341+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:52:31.540397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"90'149","reported_seq":731,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.103862+0000","last_change":"2024-08-17T17:42:26.967686+0000","last_active":"2024-08-17T17:44:38.103862+0000","last_peered":"2024-08-17T17:44:38.103862+0000","last_clean":"2024-08-17T17:44:38.103862+0000","last_became_active":"2024-08-17T17:42:05.873657+0000","last_became_peered":"2024-08-17T17:42:05.873657+0000","last_unstale":"2024-08-17T17:44:38.103862+0000","last_undegraded":"2024-08-17T17:44:38.103862+0000","last_fullsized":"2024-08-17T17:44:38.103862+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:26.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:26.967611+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:31:03.344332+0000","scrub_duration":0.0015978279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":448,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.136508+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:44:38.136508+0000","last_peered":"2024-08-17T17:44:38.136508+0000","last_clean":"2024-08-17T17:44:38.136508+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:44:38.136508+0000","last_undegraded":"2024-08-17T17:44:38.136508+0000","last_fullsized":"2024-08-17T17:44:38.136508+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":597,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.108396+0000","last_change":"2024-08-17T17:41:34.625011+0000","last_active":"2024-08-17T17:44:38.108396+0000","last_peered":"2024-08-17T17:44:38.108396+0000","last_clean":"2024-08-17T17:44:38.108396+0000","last_became_active":"2024-08-17T17:41:34.624631+0000","last_became_peered":"2024-08-17T17:41:34.624631+0000","last_unstale":"2024-08-17T17:44:38.108396+0000","last_undegraded":"2024-08-17T17:44:38.108396+0000","last_fullsized":"2024-08-17T17:44:38.108396+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:03:05.565435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"90'149","reported_seq":694,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.109296+0000","last_change":"2024-08-17T17:42:08.482975+0000","last_active":"2024-08-17T17:44:38.109296+0000","last_peered":"2024-08-17T17:44:38.109296+0000","last_clean":"2024-08-17T17:44:38.109296+0000","last_became_active":"2024-08-17T17:42:05.873523+0000","last_became_peered":"2024-08-17T17:42:05.873523+0000","last_unstale":"2024-08-17T17:44:38.109296+0000","last_undegraded":"2024-08-17T17:44:38.109296+0000","last_fullsized":"2024-08-17T17:44:38.109296+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:08.482892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:08.482892+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:41.477341+0000","scrub_duration":0.0017925479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":52,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.104912+0000","last_change":"2024-08-17T17:41:34.616482+0000","last_active":"2024-08-17T17:44:38.104912+0000","last_peered":"2024-08-17T17:44:38.104912+0000","last_clean":"2024-08-17T17:44:38.104912+0000","last_became_active":"2024-08-17T17:41:34.616040+0000","last_became_peered":"2024-08-17T17:41:34.616040+0000","last_unstale":"2024-08-17T17:44:38.104912+0000","last_undegraded":"2024-08-17T17:44:38.104912+0000","last_fullsized":"2024-08-17T17:44:38.104912+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:53:08.502264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":52,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.106394+0000","last_change":"2024-08-17T17:41:34.622184+0000","last_active":"2024-08-17T17:44:38.106394+0000","last_peered":"2024-08-17T17:44:38.106394+0000","last_clean":"2024-08-17T17:44:38.106394+0000","last_became_active":"2024-08-17T17:41:34.621831+0000","last_became_peered":"2024-08-17T17:41:34.621831+0000","last_unstale":"2024-08-17T17:44:38.106394+0000","last_undegraded":"2024-08-17T17:44:38.106394+0000","last_fullsized":"2024-08-17T17:44:38.106394+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:59.826667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"90'149","reported_seq":827,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.120526+0000","last_change":"2024-08-17T17:42:05.876463+0000","last_active":"2024-08-17T17:44:38.120526+0000","last_peered":"2024-08-17T17:44:38.120526+0000","last_clean":"2024-08-17T17:44:38.120526+0000","last_became_active":"2024-08-17T17:42:05.876069+0000","last_became_peered":"2024-08-17T17:42:05.876069+0000","last_unstale":"2024-08-17T17:44:38.120526+0000","last_undegraded":"2024-08-17T17:44:38.120526+0000","last_fullsized":"2024-08-17T17:44:38.120526+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:45.552985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:45.552985+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:55:06.027278+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":564,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.108549+0000","last_change":"2024-08-17T17:33:20.773207+0000","last_active":"2024-08-17T17:44:38.108549+0000","last_peered":"2024-08-17T17:44:38.108549+0000","last_clean":"2024-08-17T17:44:38.108549+0000","last_became_active":"2024-08-17T17:33:20.772457+0000","last_became_peered":"2024-08-17T17:33:20.772457+0000","last_unstale":"2024-08-17T17:44:38.108549+0000","last_undegraded":"2024-08-17T17:44:38.108549+0000","last_fullsized":"2024-08-17T17:44:38.108549+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:57:09.610084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":471,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.130848+0000","last_change":"2024-08-17T17:41:34.618968+0000","last_active":"2024-08-17T17:44:38.130848+0000","last_peered":"2024-08-17T17:44:38.130848+0000","last_clean":"2024-08-17T17:44:38.130848+0000","last_became_active":"2024-08-17T17:41:34.618576+0000","last_became_peered":"2024-08-17T17:41:34.618576+0000","last_unstale":"2024-08-17T17:44:38.130848+0000","last_undegraded":"2024-08-17T17:44:38.130848+0000","last_fullsized":"2024-08-17T17:44:38.130848+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:47:45.888590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"90'149","reported_seq":475,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.121638+0000","last_change":"2024-08-17T17:42:06.742801+0000","last_active":"2024-08-17T17:44:38.121638+0000","last_peered":"2024-08-17T17:44:38.121638+0000","last_clean":"2024-08-17T17:44:38.121638+0000","last_became_active":"2024-08-17T17:42:05.876895+0000","last_became_peered":"2024-08-17T17:42:05.876895+0000","last_unstale":"2024-08-17T17:44:38.121638+0000","last_undegraded":"2024-08-17T17:44:38.121638+0000","last_fullsized":"2024-08-17T17:44:38.121638+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.742723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.742723+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:36:46.651953+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":52,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.103776+0000","last_change":"2024-08-17T17:41:34.622094+0000","last_active":"2024-08-17T17:44:38.103776+0000","last_peered":"2024-08-17T17:44:38.103776+0000","last_clean":"2024-08-17T17:44:38.103776+0000","last_became_active":"2024-08-17T17:41:34.621773+0000","last_became_peered":"2024-08-17T17:41:34.621773+0000","last_unstale":"2024-08-17T17:44:38.103776+0000","last_undegraded":"2024-08-17T17:44:38.103776+0000","last_fullsized":"2024-08-17T17:44:38.103776+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:34:30.540886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":56,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.105388+0000","last_change":"2024-08-17T17:41:34.629135+0000","last_active":"2024-08-17T17:44:38.105388+0000","last_peered":"2024-08-17T17:44:38.105388+0000","last_clean":"2024-08-17T17:44:38.105388+0000","last_became_active":"2024-08-17T17:41:34.613095+0000","last_became_peered":"2024-08-17T17:41:34.613095+0000","last_unstale":"2024-08-17T17:44:38.105388+0000","last_undegraded":"2024-08-17T17:44:38.105388+0000","last_fullsized":"2024-08-17T17:44:38.105388+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:27:06.542416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":52,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.106477+0000","last_change":"2024-08-17T17:41:34.616251+0000","last_active":"2024-08-17T17:44:38.106477+0000","last_peered":"2024-08-17T17:44:38.106477+0000","last_clean":"2024-08-17T17:44:38.106477+0000","last_became_active":"2024-08-17T17:41:34.615952+0000","last_became_peered":"2024-08-17T17:41:34.615952+0000","last_unstale":"2024-08-17T17:44:38.106477+0000","last_undegraded":"2024-08-17T17:44:38.106477+0000","last_fullsized":"2024-08-17T17:44:38.106477+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:12:24.892508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":56,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.104566+0000","last_change":"2024-08-17T17:41:34.631152+0000","last_active":"2024-08-17T17:44:38.104566+0000","last_peered":"2024-08-17T17:44:38.104566+0000","last_clean":"2024-08-17T17:44:38.104566+0000","last_became_active":"2024-08-17T17:41:34.615617+0000","last_became_peered":"2024-08-17T17:41:34.615617+0000","last_unstale":"2024-08-17T17:44:38.104566+0000","last_undegraded":"2024-08-17T17:44:38.104566+0000","last_fullsized":"2024-08-17T17:44:38.104566+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:43:08.257084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"90'149","reported_seq":728,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.116440+0000","last_change":"2024-08-17T17:42:11.809047+0000","last_active":"2024-08-17T17:44:38.116440+0000","last_peered":"2024-08-17T17:44:38.116440+0000","last_clean":"2024-08-17T17:44:38.116440+0000","last_became_active":"2024-08-17T17:42:05.883770+0000","last_became_peered":"2024-08-17T17:42:05.883770+0000","last_unstale":"2024-08-17T17:44:38.116440+0000","last_undegraded":"2024-08-17T17:44:38.116440+0000","last_fullsized":"2024-08-17T17:44:38.116440+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:11.808971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:11.808971+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:48:15.989437+0000","scrub_duration":0.0018167420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":656,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.121063+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:44:38.121063+0000","last_peered":"2024-08-17T17:44:38.121063+0000","last_clean":"2024-08-17T17:44:38.121063+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:44:38.121063+0000","last_undegraded":"2024-08-17T17:44:38.121063+0000","last_fullsized":"2024-08-17T17:44:38.121063+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":657,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.121766+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:44:38.121766+0000","last_peered":"2024-08-17T17:44:38.121766+0000","last_clean":"2024-08-17T17:44:38.121766+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:44:38.121766+0000","last_undegraded":"2024-08-17T17:44:38.121766+0000","last_fullsized":"2024-08-17T17:44:38.121766+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1d","version":"90'149","reported_seq":723,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.108687+0000","last_change":"2024-08-17T17:42:18.668058+0000","last_active":"2024-08-17T17:44:38.108687+0000","last_peered":"2024-08-17T17:44:38.108687+0000","last_clean":"2024-08-17T17:44:38.108687+0000","last_became_active":"2024-08-17T17:42:05.876721+0000","last_became_peered":"2024-08-17T17:42:05.876721+0000","last_unstale":"2024-08-17T17:44:38.108687+0000","last_undegraded":"2024-08-17T17:44:38.108687+0000","last_fullsized":"2024-08-17T17:44:38.108687+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:18.667957+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:18.667957+0000","last_clean_scrub_stamp":"2024-08-17T17:42:18.667957+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:38.170146+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":52,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.103342+0000","last_change":"2024-08-17T17:41:34.611711+0000","last_active":"2024-08-17T17:44:38.103342+0000","last_peered":"2024-08-17T17:44:38.103342+0000","last_clean":"2024-08-17T17:44:38.103342+0000","last_became_active":"2024-08-17T17:41:34.611407+0000","last_became_peered":"2024-08-17T17:41:34.611407+0000","last_unstale":"2024-08-17T17:44:38.103342+0000","last_undegraded":"2024-08-17T17:44:38.103342+0000","last_fullsized":"2024-08-17T17:44:38.103342+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:42:41.582320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":466,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-08-17T17:44:38.129820+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:44:38.129820+0000","last_peered":"2024-08-17T17:44:38.129820+0000","last_clean":"2024-08-17T17:44:38.129820+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:44:38.129820+0000","last_undegraded":"2024-08-17T17:44:38.129820+0000","last_fullsized":"2024-08-17T17:44:38.129820+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":5758058520,"num_objects":87862,"num_object_clones":0,"num_object_copies":263586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1026,"num_objects_misplaced":3816,"num_objects_unfound":0,"num_objects_dirty":87862,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149226,"num_write_kb":7586753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1958,"num_bytes_recovered":128319488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17505464320,"data_stored":17505452104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101626,"ondisk_log_size":101626,"up":48,"acting":47,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":11256,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19280,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5978,"num_read_kb":12991,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4619,"ondisk_log_size":4619,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234457,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2321144,"kb_used_data":1423084,"kb_used_omap":1,"kb_used_meta":898046,"kb_avail":91424008,"statfs":{"total":95995035648,"available":93618184192,"internally_reserved":0,"allocated":1457238016,"data_stored":1455019059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":919599448},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496092,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3317416,"kb_used_data":2418080,"kb_used_omap":1,"kb_used_meta":899262,"kb_avail":90427736,"statfs":{"total":95995035648,"available":92598001664,"internally_reserved":0,"allocated":2476113920,"data_stored":2473906405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":920844632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757729,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2403304,"kb_used_data":1508508,"kb_used_omap":1,"kb_used_meta":894782,"kb_avail":91341848,"statfs":{"total":95995035648,"available":93534052352,"internally_reserved":0,"allocated":1544712192,"data_stored":1542505318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":916257112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019367,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3447348,"kb_used_data":3141676,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":90297804,"statfs":{"total":95995035648,"available":92464951296,"internally_reserved":0,"allocated":3217076224,"data_stored":3214857008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435593,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2759440,"kb_used_data":1831176,"kb_used_omap":4,"kb_used_meta":928187,"kb_avail":90985712,"statfs":{"total":95995035648,"available":93169369088,"internally_reserved":0,"allocated":1875124224,"data_stored":1872928696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":950463498},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599120,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3373080,"kb_used_data":2563664,"kb_used_omap":7,"kb_used_meta":809400,"kb_avail":90372072,"statfs":{"total":95995035648,"available":92541001728,"internally_reserved":0,"allocated":2625191936,"data_stored":2622989274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":828825762},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":268,"seq":1151051235467,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1519056,"kb_used_data":1064648,"kb_used_omap":10,"kb_used_meta":454389,"kb_avail":92226096,"statfs":{"total":95995035648,"available":94439522304,"internally_reserved":0,"allocated":1090199552,"data_stored":1088003969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10902,"internal_metadata":465294698},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926173,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4514840,"kb_used_data":3633168,"kb_used_omap":7,"kb_used_meta":881656,"kb_avail":89230312,"statfs":{"total":95995035648,"available":91371839488,"internally_reserved":0,"allocated":3720364032,"data_stored":3718160846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":902816065},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,4,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7550,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3701084160,"data_stored":3701080088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1080426496,"data_stored":1080426496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2598895616,"data_stored":2598895616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1869545472,"data_stored":1869545472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2840989696,"data_stored":2840985624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1527185408,"data_stored":1527185408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2455506944,"data_stored":2455502872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1431830528,"data_stored":1431830528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:44:57.434 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph status --format=json 2024-08-17T17:44:57.435 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:57.480 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:57.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:57.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:57.582 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:44:57.624 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:57.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:57.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:44:57.695 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:57 smithi003 ceph-mon[103344]: pgmap v1270: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 1026/257976 objects degraded (0.398%); 3816/257976 objects misplaced (1.479%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:44:57.696 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:57 smithi003 ceph-mon[100615]: pgmap v1270: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 1026/257976 objects degraded (0.398%); 3816/257976 objects misplaced (1.479%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:44:57.727 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:44:57.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:57.779 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:57.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:44:57.881 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:57.908 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:57.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:57.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:57.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:57 smithi053 ceph-mon[89963]: pgmap v1270: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 1026/257976 objects degraded (0.398%); 3816/257976 objects misplaced (1.479%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:44:58.010 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:58.051 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:58.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:58.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:58.153 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:44:58.191 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:58.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:58.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:44:58.292 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:58.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:58.328 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:58.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:58.429 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:58.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:58.465 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:58.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:58.566 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:44:58.604 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:58.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:58.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:44:58.706 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:58.741 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:58.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:58.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:58.814 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:44:58.843 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:58.918 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:58.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:58.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:59.020 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:44:59.057 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:59.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:59.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:44:59.139 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:44:59.159 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:59.203 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:59.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:59.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:59.305 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:59.346 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:59.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:59.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:59.448 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:44:59.476 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:59.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:59.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:44:59.578 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:44:59.614 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:44:59.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:59.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:44:59.715 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:59.753 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:59.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:59.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:59.855 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:44:59.892 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:44:59.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:44:59.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:44:59.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:44:59 smithi053 ceph-mon[89963]: pgmap v1271: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 1026/257976 objects degraded (0.398%); 3816/257976 objects misplaced (1.479%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:44:59.994 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:00.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:44:59 smithi003 ceph-mon[103344]: pgmap v1271: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 1026/257976 objects degraded (0.398%); 3816/257976 objects misplaced (1.479%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:45:00.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:44:59 smithi003 ceph-mon[100615]: pgmap v1271: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 1026/257976 objects degraded (0.398%); 3816/257976 objects misplaced (1.479%); 1.3 MiB/s, 20 objects/s recovering 2024-08-17T17:45:00.029 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:00.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:00.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:00.130 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:00.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:00.169 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:00.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:00.270 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:00.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:00.313 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:00.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:00.414 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:00.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:00.451 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:00.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:00.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:45:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:45:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:45:00.552 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:00.574 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:00.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:00.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:00.676 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:00.709 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:00.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:00.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:00.810 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:00.846 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:00.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:00.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:00.948 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:00.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:00 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 1026/257976 objects degraded (0.398%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:00.980 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:00.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:00.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:01.022 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:00 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 1026/257976 objects degraded (0.398%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:01.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:00 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 1026/257976 objects degraded (0.398%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:01.082 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:01.116 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:01.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:01.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:01.218 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:01.252 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:01.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:01.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:01.353 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:01.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:01.385 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:01.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:01.486 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:01.520 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:01.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:01.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:01.622 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:01.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:01.657 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:01.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:01.684 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:45:01.684 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:45:01.759 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:01.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:01.797 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:01.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:01.898 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:01.941 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:01.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:01.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:02.043 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:02.080 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:02.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:02.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:02.182 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:02.195 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:01 smithi003 ceph-mon[103344]: pgmap v1272: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 989 op/s; 1204/244470 objects degraded (0.492%); 3510/244470 objects misplaced (1.436%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:45:02.196 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:01 smithi003 ceph-mon[100615]: pgmap v1272: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 989 op/s; 1204/244470 objects degraded (0.492%); 3510/244470 objects misplaced (1.436%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:45:02.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:01 smithi053 ceph-mon[89963]: pgmap v1272: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 989 op/s; 1204/244470 objects degraded (0.492%); 3510/244470 objects misplaced (1.436%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:45:02.224 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:02.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:02.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:02.326 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:02.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:02.358 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:02.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:02.460 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:02.492 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:02.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:02.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:02.566 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:45:02.567 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:45:02.567 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:45:02.567 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:45:02.567 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:45:02.569 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:45:02.569 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:45:02.569 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:45:02.569 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:45:02.569 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:45:02.569 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:45:02.570 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:45:02.570 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:45:02.570 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:45:02.570 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:45:02.570 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:45:02.595 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:02.630 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:02.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:02.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:02.733 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:02.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:02.769 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:02.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:02.869 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:02.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:02.910 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:02.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:02.928 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-08-17T17:45:03.011 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:03.048 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:03.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:03.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:03.150 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:03.188 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:03.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:03.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:03.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:02 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1871637634' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:45:03.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:02 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/4073282392' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:45:03.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:02 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:45:03.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:02 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1871637634' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:45:03.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:02 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/4073282392' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:45:03.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:02 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:45:03.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:02 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1871637634' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:45:03.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:02 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/4073282392' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:45:03.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:02 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:45:03.290 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:03.330 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:03.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:03.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:03.432 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:03.465 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:03.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:03.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:03.567 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:03.605 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:03.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:03.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:03.707 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:03.745 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:03.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:03.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:03.847 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:03.883 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:03.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:03.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:03.985 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:04.020 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:04.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:04.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:04.122 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:04.155 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:04.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:04.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:04.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:03 smithi053 ceph-mon[89963]: pgmap v1273: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 992 op/s; 1204/243522 objects degraded (0.494%); 3510/243522 objects misplaced (1.441%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:45:04.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:03 smithi053 ceph-mon[89963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:45:04.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:03 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:45:04.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:03 smithi053 ceph-mon[89963]: osdmap e389: 8 total, 8 up, 8 in 2024-08-17T17:45:04.231 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:03 smithi003 ceph-mon[103344]: pgmap v1273: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 992 op/s; 1204/243522 objects degraded (0.494%); 3510/243522 objects misplaced (1.441%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:45:04.231 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:03 smithi003 ceph-mon[103344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:45:04.231 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:03 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:45:04.231 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:03 smithi003 ceph-mon[103344]: osdmap e389: 8 total, 8 up, 8 in 2024-08-17T17:45:04.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:03 smithi003 ceph-mon[100615]: pgmap v1273: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 992 op/s; 1204/243522 objects degraded (0.494%); 3510/243522 objects misplaced (1.441%); 1.4 MiB/s, 22 objects/s recovering 2024-08-17T17:45:04.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:03 smithi003 ceph-mon[100615]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:45:04.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:03 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:45:04.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:03 smithi003 ceph-mon[100615]: osdmap e389: 8 total, 8 up, 8 in 2024-08-17T17:45:04.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:04.274 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-17T17:45:04.298 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:04.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:04.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:04.401 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:04.431 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:04.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:04.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:04.533 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:04.567 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:04.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:04.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:04.669 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:04.699 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:04.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:04.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:04.801 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:04.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:04.858 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:04.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:04.959 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:04.996 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:04.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:04.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:05.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:04 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 1204/243522 objects degraded (0.494%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:05.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:04 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 1204/243522 objects degraded (0.494%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:05.098 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:05.128 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:05.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:05.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:05.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:04 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 1204/243522 objects degraded (0.494%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:05.231 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:05.268 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:05.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:05.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:05.370 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:05.406 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:05.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:05.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:05.508 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:05.544 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:05.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:05.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:05.645 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:05.681 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:05.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:05.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:05.783 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:05.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:05.822 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:05.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:05.923 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:05.935 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:45:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:45:05] "GET /metrics HTTP/1.1" 200 36578 "" "Prometheus/2.51.0" 2024-08-17T17:45:05.959 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:05.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:05.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:06.061 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:06.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:06.101 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:06.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:06.201 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:06.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:05 smithi053 ceph-mon[89963]: pgmap v1275: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 978 op/s; 1392/234285 objects degraded (0.594%); 3202/234285 objects misplaced (1.367%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:45:06.224 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:06.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:06.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:06.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:05 smithi003 ceph-mon[100615]: pgmap v1275: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 978 op/s; 1392/234285 objects degraded (0.594%); 3202/234285 objects misplaced (1.367%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:45:06.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:05 smithi003 ceph-mon[103344]: pgmap v1275: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 978 op/s; 1392/234285 objects degraded (0.594%); 3202/234285 objects misplaced (1.367%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:45:06.325 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:06.362 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:06.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:06.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:06.464 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:06.499 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:06.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:06.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:06.600 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:06.635 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:06.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:06.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:06.737 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:06.777 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:06.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:06.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:06.790 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:45:06.879 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:06.917 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:06.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:06.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:07.018 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:07.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:07.057 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:07.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:07.158 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:07.191 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:07.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:07.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:07.293 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:07.327 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:07.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:07.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:07.429 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:07.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:07.473 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:07.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:07.574 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:07.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:07.609 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:07.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:07.710 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:07.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:07.750 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:07.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:07.851 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:07.885 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:07.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:07.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:07.987 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:08.021 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:08.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:08.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:08.123 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:08.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:08.165 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:08.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:08.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:07 smithi053 ceph-mon[89963]: pgmap v1276: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 979 op/s; 1392/228573 objects degraded (0.609%); 3202/228573 objects misplaced (1.401%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:45:08.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:08.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:07 smithi003 ceph-mon[100615]: pgmap v1276: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 979 op/s; 1392/228573 objects degraded (0.609%); 3202/228573 objects misplaced (1.401%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:45:08.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:07 smithi003 ceph-mon[103344]: pgmap v1276: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 979 op/s; 1392/228573 objects degraded (0.609%); 3202/228573 objects misplaced (1.401%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:45:08.300 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:08.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:08.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:08.402 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:08.442 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:08.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:08.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:08.544 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:08.576 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:08.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:08.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:08.678 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:08.713 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:08.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:08.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:08.815 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:08.850 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:08.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:08.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:08.952 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:08.983 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:08.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:08.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:09.061 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:45:09.085 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:09.115 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:45:09.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:09.141 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:09.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:09.242 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:09.278 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:09.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:09.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:09.380 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:09.404 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:45:09.404 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:45:09.404 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:09 smithi003 ceph-mon[103344]: pgmap v1277: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 979 op/s; 1392/228573 objects degraded (0.609%); 3202/228573 objects misplaced (1.401%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:45:09.405 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:45:09.405 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:45:09.405 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:09 smithi003 ceph-mon[100615]: pgmap v1277: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 979 op/s; 1392/228573 objects degraded (0.609%); 3202/228573 objects misplaced (1.401%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:45:09.423 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:09.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:09.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:45:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:45:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:09 smithi053 ceph-mon[89963]: pgmap v1277: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 979 op/s; 1392/228573 objects degraded (0.609%); 3202/228573 objects misplaced (1.401%); 1.5 MiB/s, 24 objects/s recovering 2024-08-17T17:45:09.476 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:45:09.525 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:09.561 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:09.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:09.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:09.663 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:09.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:09.707 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:09.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:09.808 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:09.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:09.849 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:09.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:09.950 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:09.991 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:09.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:09.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:10.092 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:10.130 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:10.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:10.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:10.232 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:10.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:10.269 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:10.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:10.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:10 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 1392/228573 objects degraded (0.609%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:10.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:10 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 1392/228573 objects degraded (0.609%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:10.370 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:10.408 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:10.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:10.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:10.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:10 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 1392/228573 objects degraded (0.609%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:10.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:45:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:45:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:45:10.510 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:10.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:10.550 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:10.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:10.651 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:10.685 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:10.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:10.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:10.786 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:10.823 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:10.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:10.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:10.925 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:10.961 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:10.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:10.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:11.063 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:11.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:11.104 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:11.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:11.204 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:11.242 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:11.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:11.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:11.272 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:11 smithi003 ceph-mon[103344]: pgmap v1278: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 1600/214197 objects degraded (0.747%); 2877/214197 objects misplaced (1.343%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:45:11.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:11 smithi003 ceph-mon[100615]: pgmap v1278: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 1600/214197 objects degraded (0.747%); 2877/214197 objects misplaced (1.343%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:45:11.344 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:11.378 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:11.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:11.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:11.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:11 smithi053 ceph-mon[89963]: pgmap v1278: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 1600/214197 objects degraded (0.747%); 2877/214197 objects misplaced (1.343%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:45:11.480 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:11.512 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:11.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:11.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:11.614 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:11.645 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:11.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:11.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:11.747 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:11.780 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:11.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:11.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:11.882 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:11.922 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:11.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:11.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:12.024 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:12.073 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:12.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:12.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:12.175 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:12.213 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:12.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:12.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:12.262 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:45:12.262 INFO:teuthology.orchestra.run.smithi003.stdout:{"fsid":"2b5b662e-5cba-11ef-bccf-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1600/214197 objects degraded (0.747%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1853,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":389,"num_osds":8,"num_up_osds":8,"osd_up_since":1723915999,"num_in_osds":8,"osd_in_since":1723916493,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":118},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":119,"num_pools":5,"num_objects":71399,"data_bytes":4682094565,"bytes_used":20943949824,"bytes_avail":747016335360,"bytes_total":767960285184,"degraded_objects":1600,"degraded_total":214197,"degraded_ratio":0.0074697591469535052,"misplaced_objects":2877,"misplaced_total":214197,"misplaced_ratio":0.013431560666115772,"recovering_objects_per_sec":23,"recovering_bytes_per_sec":1552721,"recovering_keys_per_sec":0,"num_objects_recovered":237,"num_bytes_recovered":15532032,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":1008},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi003.civgvf","status":"up:active","gid":14664}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.3:8443/","prometheus":"http://172.21.15.3:9283/"}},"servicemap":{"epoch":29,"modified":"2024-08-17T17:43:40.367726+0000","services":{}},"progress_events":{"a4a400b3-2a31-493f-9d26-837737f27ab1":{"message":"Global Recovery Event (0s)\n [............................] ","progress":0,"add_to_ceph_s":true},"fcbeb9b7-7baf-488b-9fe4-c01aca441f9c":{"message":"Upgrade to 19.0.0-5836-gcd8bdbeb (4m)\n [============................] (remaining: 5m)","progress":0.44999998807907104,"add_to_ceph_s":true}}} 2024-08-17T17:45:12.315 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:12.350 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:12.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:12.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:12.452 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:12.493 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:12.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:12.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:12.596 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:12.630 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:12.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:12.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:12.689 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:12 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3936635713' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T17:45:12.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:12 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3936635713' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T17:45:12.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:12 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3936635713' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T17:45:12.732 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:12.765 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:12.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:12.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:12.843 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:45:12.843 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:45:12.867 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:12.878 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:45:12.879 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:45:12.902 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:12.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:12.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:12.910 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:45:12.910 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:45:12.945 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:45:12.945 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:45:12.981 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:45:12.981 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:45:13.004 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:13.018 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:45:13.018 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:45:13.039 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:13.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:13.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:13.060 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:45:13.060 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:45:13.091 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:45:13.091 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:45:13.141 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:13.187 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:13.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:13.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:13.289 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:13.299 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-08-17T17:45:13.299 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:45:13.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:13.327 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:13.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:13.428 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:13.460 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:13.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:13.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:13.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:13 smithi003 ceph-mon[103344]: pgmap v1279: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 1600/213324 objects degraded (0.750%); 2877/213324 objects misplaced (1.349%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:45:13.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:13 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/147238355' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:45:13.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:13 smithi003 ceph-mon[100615]: pgmap v1279: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 1600/213324 objects degraded (0.750%); 2877/213324 objects misplaced (1.349%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:45:13.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:13 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/147238355' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:45:13.562 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:13.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:13.608 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:13.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:13.708 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:13.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:13 smithi053 ceph-mon[89963]: pgmap v1279: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 1600/213324 objects degraded (0.750%); 2877/213324 objects misplaced (1.349%); 1.5 MiB/s, 23 objects/s recovering 2024-08-17T17:45:13.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:13 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/147238355' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:45:13.744 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:13.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:13.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:13.846 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:13.881 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:13.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:13.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:13.983 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:14.023 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:14.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:14.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:14.125 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:14.164 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:14.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:14.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:14.267 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:14.303 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:14.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:14.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:14.405 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:14.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:14.441 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:14.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:14.542 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:14.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:14.578 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:14.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:14.679 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:14.715 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:14.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:14.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:14.776 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:45:14.783 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:45:14.817 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:14.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:14.878 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:14.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:14.979 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:15.022 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:15.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:15.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:15.124 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:15.156 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:15.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:15.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:15.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:15.295 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:15.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:15.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:15.397 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:15.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:15.441 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:15.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:15.542 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:15.575 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:15.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:15.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:15.677 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:15.701 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:45:15.701 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:45:15.712 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:15 smithi003 ceph-mon[103344]: pgmap v1280: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s; 1770/204057 objects degraded (0.867%); 2570/204057 objects misplaced (1.259%); 2.0 MiB/s, 32 objects/s recovering 2024-08-17T17:45:15.712 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:15 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 1770/204057 objects degraded (0.867%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:15.713 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:15 smithi003 ceph-mon[100615]: pgmap v1280: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s; 1770/204057 objects degraded (0.867%); 2570/204057 objects misplaced (1.259%); 2.0 MiB/s, 32 objects/s recovering 2024-08-17T17:45:15.713 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:15 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 1770/204057 objects degraded (0.867%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:15.714 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:45:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:45:15] "GET /metrics HTTP/1.1" 200 36578 "" "Prometheus/2.51.0" 2024-08-17T17:45:15.720 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:15.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:15.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:15.822 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:15.852 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:15.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:15.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:15.954 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:15.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:15 smithi053 ceph-mon[89963]: pgmap v1280: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s; 1770/204057 objects degraded (0.867%); 2570/204057 objects misplaced (1.259%); 2.0 MiB/s, 32 objects/s recovering 2024-08-17T17:45:15.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:15 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 1770/204057 objects degraded (0.867%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:15.989 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:45:15.994 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:15.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:15.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:16.096 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:16.129 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:16.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:16.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:16.231 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:16.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:16.266 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:16.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:16.367 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:16.401 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:16.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:16.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:16.503 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:16.536 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:16.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:16.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:16.638 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:16.670 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:16.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:16.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:16.692 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-17T17:45:16.772 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:16.804 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:16.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:16.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:16.906 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:16.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:16.949 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:16.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:17.050 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:17.081 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:17.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:17.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:17.183 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:17.222 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:17.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:17.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:17.324 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:17.359 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:17.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:17.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:17.461 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:17.491 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:17.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:17.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:17.593 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:17.628 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:17.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:17.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:17.730 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:17.764 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:17.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:17.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:17.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:17 smithi003 ceph-mon[103344]: pgmap v1281: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 994 op/s; 1770/198459 objects degraded (0.892%); 2570/198459 objects misplaced (1.295%); 1.3 MiB/s, 21 objects/s recovering 2024-08-17T17:45:17.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:17 smithi003 ceph-mon[100615]: pgmap v1281: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 994 op/s; 1770/198459 objects degraded (0.892%); 2570/198459 objects misplaced (1.295%); 1.3 MiB/s, 21 objects/s recovering 2024-08-17T17:45:17.865 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:17.906 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:17.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:17.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:17.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:17 smithi053 ceph-mon[89963]: pgmap v1281: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 994 op/s; 1770/198459 objects degraded (0.892%); 2570/198459 objects misplaced (1.295%); 1.3 MiB/s, 21 objects/s recovering 2024-08-17T17:45:18.008 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:18.048 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:18.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:18.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:18.150 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:18.197 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:18.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:18.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:18.299 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:18.338 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:18.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:18.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:18.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:18.474 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:18.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:18.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:18.576 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:18.619 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:18.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:18.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:18.722 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:18.768 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:18.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:18.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:18.870 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:18.911 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:18.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:18.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:19.013 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:19.053 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:19.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:19.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:19.155 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:19.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:19.191 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:19.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:19.292 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:19.324 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:19.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:19.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:19.426 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:19.460 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:19.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:19.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:19.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:19 smithi003 ceph-mon[100615]: pgmap v1282: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 836 op/s; 1770/198459 objects degraded (0.892%); 2570/198459 objects misplaced (1.295%); 1.3 MiB/s, 21 objects/s recovering 2024-08-17T17:45:19.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:19 smithi003 ceph-mon[103344]: pgmap v1282: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 836 op/s; 1770/198459 objects degraded (0.892%); 2570/198459 objects misplaced (1.295%); 1.3 MiB/s, 21 objects/s recovering 2024-08-17T17:45:19.562 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:19.594 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:19.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:19.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:19.696 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:19.733 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:19.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:19.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:19.835 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:19.857 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:19.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:19.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:19.958 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:19.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:19 smithi053 ceph-mon[89963]: pgmap v1282: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 836 op/s; 1770/198459 objects degraded (0.892%); 2570/198459 objects misplaced (1.295%); 1.3 MiB/s, 21 objects/s recovering 2024-08-17T17:45:19.999 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:20.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:20.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:20.101 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:20.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:20.146 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:20.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:20.151 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:45:20.247 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:20.283 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:20.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:20.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:20.385 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:20.421 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:20.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:20.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:20.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:45:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:45:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:45:20.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:20 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 1770/198459 objects degraded (0.892%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:20.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:20 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 1770/198459 objects degraded (0.892%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:20.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:20.561 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:20.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:20.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:20.663 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:20.700 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:20.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:20.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:20.802 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:20.841 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:20.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:20.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:20.943 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:20.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:20 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 1770/198459 objects degraded (0.892%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:20.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:20.979 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:20.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:21.080 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:21.115 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:21.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:21.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:21.216 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:21.255 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:21.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:21.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:21.356 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:21.398 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:21.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:21.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:21.488 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-08-17T17:45:21.499 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:21.536 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:21.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:21.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:21.638 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:21.676 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:21.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:21.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:21.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:21 smithi003 ceph-mon[103344]: pgmap v1283: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.24k op/s; 1950/184002 objects degraded (1.060%); 2265/184002 objects misplaced (1.231%); 2.0 MiB/s, 31 objects/s recovering 2024-08-17T17:45:21.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:21 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1436294339' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:45:21.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:21 smithi003 ceph-mon[100615]: pgmap v1283: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.24k op/s; 1950/184002 objects degraded (1.060%); 2265/184002 objects misplaced (1.231%); 2.0 MiB/s, 31 objects/s recovering 2024-08-17T17:45:21.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:21 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1436294339' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:45:21.777 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:21.809 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:21.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:21.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:21.911 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:21.947 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:21.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:21.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:21.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:21 smithi053 ceph-mon[89963]: pgmap v1283: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 3.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.24k op/s; 1950/184002 objects degraded (1.060%); 2265/184002 objects misplaced (1.231%); 2.0 MiB/s, 31 objects/s recovering 2024-08-17T17:45:21.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:21 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1436294339' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:45:22.049 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:22.085 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:22.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:22.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:22.187 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:22.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:22.223 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:22.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:22.324 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:22.362 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:22.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:22.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:22.463 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:22.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:22.496 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:22.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:22.598 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:22.628 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:22.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:22.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:22.640 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:45:22.730 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:22.748 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:22 smithi003 ceph-mon[103344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:45:22.748 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:22 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1436294339' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:45:22.749 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:22 smithi003 ceph-mon[103344]: osdmap e390: 8 total, 8 up, 8 in 2024-08-17T17:45:22.749 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:22 smithi003 ceph-mon[100615]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:45:22.749 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:22 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1436294339' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:45:22.749 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:22 smithi003 ceph-mon[100615]: osdmap e390: 8 total, 8 up, 8 in 2024-08-17T17:45:22.770 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:22.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:22.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:22.872 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:22.918 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:22.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:22.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:22.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:22 smithi053 ceph-mon[89963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:45:22.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:22 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1436294339' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:45:22.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:22 smithi053 ceph-mon[89963]: osdmap e390: 8 total, 8 up, 8 in 2024-08-17T17:45:23.021 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:23.055 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:23.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:23.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:23.157 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:23.191 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:23.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:23.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:23.339 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:23.382 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:23.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:23.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:23.484 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:23.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:23.537 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:23.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:23.638 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:23.673 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:23.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:23.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:23.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:23 smithi003 ceph-mon[100615]: pgmap v1285: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 3.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 1950/182994 objects degraded (1.066%); 2265/182994 objects misplaced (1.238%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T17:45:23.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:45:23.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:45:23.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:23 smithi003 ceph-mon[103344]: pgmap v1285: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 3.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 1950/182994 objects degraded (1.066%); 2265/182994 objects misplaced (1.238%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T17:45:23.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:45:23.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:45:23.776 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:23.813 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:23.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:23.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:23.916 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:23.962 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:23.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:23.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:23.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:23 smithi053 ceph-mon[89963]: pgmap v1285: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 3.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 1950/182994 objects degraded (1.066%); 2265/182994 objects misplaced (1.238%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T17:45:23.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:45:23.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:45:24.064 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:24.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:24.104 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:24.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:24.207 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:24.240 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:24.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:24.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:24.342 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:24.386 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:24.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:24.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:24.488 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:24.523 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:24.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:24.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:24.624 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:24.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:24.662 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:24.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:24.763 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:24.769 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:45:24.769 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:45:24.806 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:24.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:24.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:24.813 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:45:24.813 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:45:24.859 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:45:24.861 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:45:24.863 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:45:24.863 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:45:24.897 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:45:24.897 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:45:24.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:24.934 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:45:24.935 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:45:24.948 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:24.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:24.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:24.969 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:45:24.969 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:45:25.001 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:45:25.001 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:45:25.028 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:45:25.028 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:45:25.050 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:25.088 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:25.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:25.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:25.190 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:25.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:25.226 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:25.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:25.327 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:25.364 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:25.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:25.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:25.466 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:25.499 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:25.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:25.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:25.600 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:25.640 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:25.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:25.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:25.744 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:25.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:25 smithi003 ceph-mon[103344]: pgmap v1286: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2119/173502 objects degraded (1.221%); 1957/173502 objects misplaced (1.128%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T17:45:25.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:25 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2278830350' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:45:25.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:25 smithi003 ceph-mon[103344]: from='client.36798 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:45:25.775 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:25 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 2119/173502 objects degraded (1.221%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:25.776 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:25 smithi003 ceph-mon[100615]: pgmap v1286: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2119/173502 objects degraded (1.221%); 1957/173502 objects misplaced (1.128%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T17:45:25.776 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:25 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2278830350' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:45:25.776 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:25 smithi003 ceph-mon[100615]: from='client.36798 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:45:25.777 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:25 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 2119/173502 objects degraded (1.221%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:25.777 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:45:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:45:25] "GET /metrics HTTP/1.1" 200 36581 "" "Prometheus/2.51.0" 2024-08-17T17:45:25.782 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:25.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:25.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:25.884 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:25.917 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:25.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:25.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:25.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:25 smithi053 ceph-mon[89963]: pgmap v1286: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 3.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2119/173502 objects degraded (1.221%); 1957/173502 objects misplaced (1.128%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T17:45:25.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:25 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2278830350' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:45:25.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:25 smithi053 ceph-mon[89963]: from='client.36798 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:45:25.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:25 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 2119/173502 objects degraded (1.221%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:25.969 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:45:26.019 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:26.053 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:26.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:26.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:26.155 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:26.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:26.195 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:26.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:26.296 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:26.331 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:26.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:26.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:26.433 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:26.472 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:26.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:26.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:26.574 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:26.617 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:26.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:26.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:26.719 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:26.760 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:26.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:26.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:26.862 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:26.901 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:26.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:26.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:27.003 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:27.042 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:27.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:27.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:27.143 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:27.179 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:27.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:27.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:27.281 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:27.313 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:27.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:27.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:27.415 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:27.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:27.454 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:27.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:27.555 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:27.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:27.599 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:27.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:27.700 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:27.734 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:27.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:27.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:27.836 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:27.881 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:27.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:27.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:27.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:27 smithi053 ceph-mon[89963]: pgmap v1287: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 3.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2119/167694 objects degraded (1.264%); 1957/167694 objects misplaced (1.167%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T17:45:27.967 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:27 smithi003 ceph-mon[103344]: pgmap v1287: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 3.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2119/167694 objects degraded (1.264%); 1957/167694 objects misplaced (1.167%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T17:45:27.967 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:27 smithi003 ceph-mon[100615]: pgmap v1287: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 3.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2119/167694 objects degraded (1.264%); 1957/167694 objects misplaced (1.167%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T17:45:27.984 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:28.022 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:28.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:28.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:28.124 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:28.161 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:28.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:28.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:28.263 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:28.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:28.294 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:28.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:28.395 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:28.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:28.430 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:28.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:28.531 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:28.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:28.573 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:28.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:28.673 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:28.710 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:28.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:28.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:28.801 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1286,"stamp":"2024-08-17T17:45:24.415470+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3793098725,"num_objects":57834,"num_object_clones":0,"num_object_copies":173502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":1957,"num_objects_unfound":0,"num_objects_dirty":57834,"num_whiteouts":0,"num_read":5979,"num_read_kb":12992,"num_write":180189,"num_write_kb":7599376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2918,"num_bytes_recovered":186919928,"num_keys_recovered":40,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107433,"ondisk_log_size":107433,"up":357,"acting":356,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":362,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":17901320,"kb_used_data":11810308,"kb_used_omap":37,"kb_used_meta":6090906,"kb_avail":732059896,"statfs":{"total":767960285184,"available":749629333504,"internally_reserved":0,"allocated":12093755392,"data_stored":12076043215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38729,"internal_metadata":6237087927},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-667484160,"num_objects":-10185,"num_object_clones":0,"num_object_copies":-30555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":349,"num_objects_misplaced":-613,"num_objects_unfound":0,"num_objects_dirty":-10185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10185,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":17301504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003694"},"pg_stats":[{"pgid":"1.11","version":"90'149","reported_seq":829,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.932786+0000","last_change":"2024-08-17T17:38:30.378664+0000","last_active":"2024-08-17T17:45:02.932786+0000","last_peered":"2024-08-17T17:45:02.932786+0000","last_clean":"2024-08-17T17:45:02.932786+0000","last_became_active":"2024-08-17T17:38:30.342287+0000","last_became_peered":"2024-08-17T17:38:30.342287+0000","last_unstale":"2024-08-17T17:45:02.932786+0000","last_undegraded":"2024-08-17T17:45:02.932786+0000","last_fullsized":"2024-08-17T17:45:02.932786+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:30.378592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:30.378592+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:04:28.933354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":650,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.930797+0000","last_change":"2024-08-17T17:33:20.797029+0000","last_active":"2024-08-17T17:45:02.930797+0000","last_peered":"2024-08-17T17:45:02.930797+0000","last_clean":"2024-08-17T17:45:02.930797+0000","last_became_active":"2024-08-17T17:33:20.796679+0000","last_became_peered":"2024-08-17T17:33:20.796679+0000","last_unstale":"2024-08-17T17:45:02.930797+0000","last_undegraded":"2024-08-17T17:45:02.930797+0000","last_fullsized":"2024-08-17T17:45:02.930797+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:51.438262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":414,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.925979+0000","last_change":"2024-08-17T17:41:34.624198+0000","last_active":"2024-08-17T17:45:02.925979+0000","last_peered":"2024-08-17T17:45:02.925979+0000","last_clean":"2024-08-17T17:45:02.925979+0000","last_became_active":"2024-08-17T17:41:34.616071+0000","last_became_peered":"2024-08-17T17:41:34.616071+0000","last_unstale":"2024-08-17T17:45:02.925979+0000","last_undegraded":"2024-08-17T17:45:02.925979+0000","last_fullsized":"2024-08-17T17:45:02.925979+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:34:48.805354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"90'149","reported_seq":473,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.936271+0000","last_change":"2024-08-17T17:38:31.874623+0000","last_active":"2024-08-17T17:45:02.936271+0000","last_peered":"2024-08-17T17:45:02.936271+0000","last_clean":"2024-08-17T17:45:02.936271+0000","last_became_active":"2024-08-17T17:38:30.339313+0000","last_became_peered":"2024-08-17T17:38:30.339313+0000","last_unstale":"2024-08-17T17:45:02.936271+0000","last_undegraded":"2024-08-17T17:45:02.936271+0000","last_fullsized":"2024-08-17T17:45:02.936271+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:31.874423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:31.874423+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:28:17.980339+0000","scrub_duration":0.0017696579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":311,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.929968+0000","last_change":"2024-08-17T17:33:44.156036+0000","last_active":"2024-08-17T17:45:02.929968+0000","last_peered":"2024-08-17T17:45:02.929968+0000","last_clean":"2024-08-17T17:45:02.929968+0000","last_became_active":"2024-08-17T17:33:44.155758+0000","last_became_peered":"2024-08-17T17:33:44.155758+0000","last_unstale":"2024-08-17T17:45:02.929968+0000","last_undegraded":"2024-08-17T17:45:02.929968+0000","last_fullsized":"2024-08-17T17:45:02.929968+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:23.443405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":451,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.929138+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:45:02.929138+0000","last_peered":"2024-08-17T17:45:02.929138+0000","last_clean":"2024-08-17T17:45:02.929138+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:45:02.929138+0000","last_undegraded":"2024-08-17T17:45:02.929138+0000","last_fullsized":"2024-08-17T17:45:02.929138+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"90'149","reported_seq":847,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.944973+0000","last_change":"2024-08-17T17:41:37.681187+0000","last_active":"2024-08-17T17:45:02.944973+0000","last_peered":"2024-08-17T17:45:02.944973+0000","last_clean":"2024-08-17T17:45:02.944973+0000","last_became_active":"2024-08-17T17:41:37.680605+0000","last_became_peered":"2024-08-17T17:41:37.680605+0000","last_unstale":"2024-08-17T17:45:02.944973+0000","last_undegraded":"2024-08-17T17:45:02.944973+0000","last_fullsized":"2024-08-17T17:45:02.944973+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.844252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.844252+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:13:31.682357+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":450,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.934713+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:45:02.934713+0000","last_peered":"2024-08-17T17:45:02.934713+0000","last_clean":"2024-08-17T17:45:02.934713+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:45:02.934713+0000","last_undegraded":"2024-08-17T17:45:02.934713+0000","last_fullsized":"2024-08-17T17:45:02.934713+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":54,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.929491+0000","last_change":"2024-08-17T17:41:34.611693+0000","last_active":"2024-08-17T17:45:02.929491+0000","last_peered":"2024-08-17T17:45:02.929491+0000","last_clean":"2024-08-17T17:45:02.929491+0000","last_became_active":"2024-08-17T17:41:34.611374+0000","last_became_peered":"2024-08-17T17:41:34.611374+0000","last_unstale":"2024-08-17T17:45:02.929491+0000","last_undegraded":"2024-08-17T17:45:02.929491+0000","last_fullsized":"2024-08-17T17:45:02.929491+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:17:09.897446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"90'149","reported_seq":823,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.943018+0000","last_change":"2024-08-17T17:38:30.446165+0000","last_active":"2024-08-17T17:45:02.943018+0000","last_peered":"2024-08-17T17:45:02.943018+0000","last_clean":"2024-08-17T17:45:02.943018+0000","last_became_active":"2024-08-17T17:38:30.445048+0000","last_became_peered":"2024-08-17T17:38:30.445048+0000","last_unstale":"2024-08-17T17:45:02.943018+0000","last_undegraded":"2024-08-17T17:45:02.943018+0000","last_fullsized":"2024-08-17T17:45:02.943018+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:24.291592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:24.291592+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:17.100154+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":664,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.933068+0000","last_change":"2024-08-17T17:33:20.794682+0000","last_active":"2024-08-17T17:45:02.933068+0000","last_peered":"2024-08-17T17:45:02.933068+0000","last_clean":"2024-08-17T17:45:02.933068+0000","last_became_active":"2024-08-17T17:33:20.794238+0000","last_became_peered":"2024-08-17T17:33:20.794238+0000","last_unstale":"2024-08-17T17:45:02.933068+0000","last_undegraded":"2024-08-17T17:45:02.933068+0000","last_fullsized":"2024-08-17T17:45:02.933068+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:15:55.821920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":475,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.936536+0000","last_change":"2024-08-17T17:43:52.520659+0000","last_active":"2024-08-17T17:45:02.936536+0000","last_peered":"2024-08-17T17:45:02.936536+0000","last_clean":"2024-08-17T17:45:02.936536+0000","last_became_active":"2024-08-17T17:43:52.516799+0000","last_became_peered":"2024-08-17T17:43:52.516799+0000","last_unstale":"2024-08-17T17:45:02.936536+0000","last_undegraded":"2024-08-17T17:45:02.936536+0000","last_fullsized":"2024-08-17T17:45:02.936536+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:25:59.408383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"389'11063","reported_seq":11138,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:20.287164+0000","last_change":"2024-08-17T17:42:36.946078+0000","last_active":"2024-08-17T17:45:20.287164+0000","last_peered":"2024-08-17T17:45:20.287164+0000","last_clean":"2024-08-17T17:45:20.287164+0000","last_became_active":"2024-08-17T17:42:36.945602+0000","last_became_peered":"2024-08-17T17:42:36.945602+0000","last_unstale":"2024-08-17T17:45:20.287164+0000","last_undegraded":"2024-08-17T17:45:20.287164+0000","last_fullsized":"2024-08-17T17:45:20.287164+0000","mapping_epoch":357,"log_start":"373'4500","ondisk_log_start":"373'4500","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6563,"log_dups_size":0,"ondisk_log_size":6563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:18:43.716968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247005184,"num_objects":3769,"num_object_clones":0,"num_object_copies":11307,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11063,"num_write_kb":474624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"90'149","reported_seq":800,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.925002+0000","last_change":"2024-08-17T17:42:16.610940+0000","last_active":"2024-08-17T17:45:02.925002+0000","last_peered":"2024-08-17T17:45:02.925002+0000","last_clean":"2024-08-17T17:45:02.925002+0000","last_became_active":"2024-08-17T17:41:37.670855+0000","last_became_peered":"2024-08-17T17:41:37.670855+0000","last_unstale":"2024-08-17T17:45:02.925002+0000","last_undegraded":"2024-08-17T17:45:02.925002+0000","last_fullsized":"2024-08-17T17:45:02.925002+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:16.610828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:16.610828+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:48:25.918610+0000","scrub_duration":0.086892487000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":351,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.927032+0000","last_change":"2024-08-17T17:41:34.635980+0000","last_active":"2024-08-17T17:45:02.927032+0000","last_peered":"2024-08-17T17:45:02.927032+0000","last_clean":"2024-08-17T17:45:02.927032+0000","last_became_active":"2024-08-17T17:41:34.620596+0000","last_became_peered":"2024-08-17T17:41:34.620596+0000","last_unstale":"2024-08-17T17:45:02.927032+0000","last_undegraded":"2024-08-17T17:45:02.927032+0000","last_fullsized":"2024-08-17T17:45:02.927032+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:19:15.792577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":144,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.936378+0000","last_change":"2024-08-17T17:33:44.145598+0000","last_active":"2024-08-17T17:45:02.936378+0000","last_peered":"2024-08-17T17:45:02.936378+0000","last_clean":"2024-08-17T17:45:02.936378+0000","last_became_active":"2024-08-17T17:33:44.145283+0000","last_became_peered":"2024-08-17T17:33:44.145283+0000","last_unstale":"2024-08-17T17:45:02.936378+0000","last_undegraded":"2024-08-17T17:45:02.936378+0000","last_fullsized":"2024-08-17T17:45:02.936378+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:11.965226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.7","version":"390'11354","reported_seq":16874,"reported_epoch":390,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-17T17:45:23.921944+0000","last_change":"2024-08-17T17:44:30.783759+0000","last_active":"2024-08-17T17:45:23.921944+0000","last_peered":"2024-08-17T17:45:23.921944+0000","last_clean":"2024-08-17T17:43:51.465744+0000","last_became_active":"2024-08-17T17:43:53.682178+0000","last_became_peered":"2024-08-17T17:43:53.682178+0000","last_unstale":"2024-08-17T17:45:23.921944+0000","last_undegraded":"2024-08-17T17:44:30.779387+0000","last_fullsized":"2024-08-17T17:43:53.513318+0000","mapping_epoch":378,"log_start":"374'4900","ondisk_log_start":"374'4900","created":357,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6454,"log_dups_size":0,"ondisk_log_size":6454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:42:59.608872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228327424,"num_objects":3484,"num_object_clones":0,"num_object_copies":10452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":1957,"num_objects_unfound":0,"num_objects_dirty":3484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11354,"num_write_kb":474816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2724,"num_bytes_recovered":178520064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3],"avail_no_missing":["6","2","3"],"object_location_counts":[{"shards":"2,3,6","objects":1957},{"shards":"3,6","objects":-592}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"90'149","reported_seq":849,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.930658+0000","last_change":"2024-08-17T17:41:42.603841+0000","last_active":"2024-08-17T17:45:02.930658+0000","last_peered":"2024-08-17T17:45:02.930658+0000","last_clean":"2024-08-17T17:45:02.930658+0000","last_became_active":"2024-08-17T17:41:12.203400+0000","last_became_peered":"2024-08-17T17:41:12.203400+0000","last_unstale":"2024-08-17T17:45:02.930658+0000","last_undegraded":"2024-08-17T17:45:02.930658+0000","last_fullsized":"2024-08-17T17:45:02.930658+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:42.603561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:42.603561+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:50:12.843193+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":347,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.926987+0000","last_change":"2024-08-17T17:41:34.637696+0000","last_active":"2024-08-17T17:45:02.926987+0000","last_peered":"2024-08-17T17:45:02.926987+0000","last_clean":"2024-08-17T17:45:02.926987+0000","last_became_active":"2024-08-17T17:41:34.621388+0000","last_became_peered":"2024-08-17T17:41:34.621388+0000","last_unstale":"2024-08-17T17:45:02.926987+0000","last_undegraded":"2024-08-17T17:45:02.926987+0000","last_fullsized":"2024-08-17T17:45:02.926987+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:13:04.607036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":54,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.929356+0000","last_change":"2024-08-17T17:41:34.621078+0000","last_active":"2024-08-17T17:45:02.929356+0000","last_peered":"2024-08-17T17:45:02.929356+0000","last_clean":"2024-08-17T17:45:02.929356+0000","last_became_active":"2024-08-17T17:41:34.620730+0000","last_became_peered":"2024-08-17T17:41:34.620730+0000","last_unstale":"2024-08-17T17:45:02.929356+0000","last_undegraded":"2024-08-17T17:45:02.929356+0000","last_fullsized":"2024-08-17T17:45:02.929356+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:55:01.312283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"389'11152","reported_seq":11228,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:20.275936+0000","last_change":"2024-08-17T17:42:36.620130+0000","last_active":"2024-08-17T17:45:20.275936+0000","last_peered":"2024-08-17T17:45:20.275936+0000","last_clean":"2024-08-17T17:45:20.275936+0000","last_became_active":"2024-08-17T17:42:36.619813+0000","last_became_peered":"2024-08-17T17:42:36.619813+0000","last_unstale":"2024-08-17T17:45:20.275936+0000","last_undegraded":"2024-08-17T17:45:20.275936+0000","last_fullsized":"2024-08-17T17:45:20.275936+0000","mapping_epoch":357,"log_start":"373'4600","ondisk_log_start":"373'4600","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6552,"log_dups_size":0,"ondisk_log_size":6552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:39:22.366905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247595008,"num_objects":3778,"num_object_clones":0,"num_object_copies":11334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11152,"num_write_kb":477760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"90'149","reported_seq":566,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.937647+0000","last_change":"2024-08-17T17:41:37.668050+0000","last_active":"2024-08-17T17:45:02.937647+0000","last_peered":"2024-08-17T17:45:02.937647+0000","last_clean":"2024-08-17T17:45:02.937647+0000","last_became_active":"2024-08-17T17:41:37.667655+0000","last_became_peered":"2024-08-17T17:41:37.667655+0000","last_unstale":"2024-08-17T17:45:02.937647+0000","last_undegraded":"2024-08-17T17:45:02.937647+0000","last_fullsized":"2024-08-17T17:45:02.937647+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:32.843802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:32.843802+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:39:46.766280+0000","scrub_duration":0.0040803669999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":140,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.936073+0000","last_change":"2024-08-17T17:33:44.145399+0000","last_active":"2024-08-17T17:45:02.936073+0000","last_peered":"2024-08-17T17:45:02.936073+0000","last_clean":"2024-08-17T17:45:02.936073+0000","last_became_active":"2024-08-17T17:33:44.144751+0000","last_became_peered":"2024-08-17T17:33:44.144751+0000","last_unstale":"2024-08-17T17:45:02.936073+0000","last_undegraded":"2024-08-17T17:45:02.936073+0000","last_fullsized":"2024-08-17T17:45:02.936073+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:54.276350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":218,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.937613+0000","last_change":"2024-08-17T17:33:20.795582+0000","last_active":"2024-08-17T17:45:02.937613+0000","last_peered":"2024-08-17T17:45:02.937613+0000","last_clean":"2024-08-17T17:45:02.937613+0000","last_became_active":"2024-08-17T17:33:20.795149+0000","last_became_peered":"2024-08-17T17:33:20.795149+0000","last_unstale":"2024-08-17T17:45:02.937613+0000","last_undegraded":"2024-08-17T17:45:02.937613+0000","last_fullsized":"2024-08-17T17:45:02.937613+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:14.020923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"389'11046","reported_seq":11122,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:20.289928+0000","last_change":"2024-08-17T17:42:36.929599+0000","last_active":"2024-08-17T17:45:20.289928+0000","last_peered":"2024-08-17T17:45:20.289928+0000","last_clean":"2024-08-17T17:45:20.289928+0000","last_became_active":"2024-08-17T17:42:36.929274+0000","last_became_peered":"2024-08-17T17:42:36.929274+0000","last_unstale":"2024-08-17T17:45:20.289928+0000","last_undegraded":"2024-08-17T17:45:20.289928+0000","last_fullsized":"2024-08-17T17:45:20.289928+0000","mapping_epoch":357,"log_start":"373'4500","ondisk_log_start":"373'4500","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6546,"log_dups_size":0,"ondisk_log_size":6546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:29:35.955743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249692160,"num_objects":3810,"num_object_clones":0,"num_object_copies":11430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11046,"num_write_kb":475392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"90'149","reported_seq":698,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.930440+0000","last_change":"2024-08-17T17:42:20.927007+0000","last_active":"2024-08-17T17:45:02.930440+0000","last_peered":"2024-08-17T17:45:02.930440+0000","last_clean":"2024-08-17T17:45:02.930440+0000","last_became_active":"2024-08-17T17:41:37.670914+0000","last_became_peered":"2024-08-17T17:41:37.670914+0000","last_unstale":"2024-08-17T17:45:02.930440+0000","last_undegraded":"2024-08-17T17:45:02.930440+0000","last_fullsized":"2024-08-17T17:45:02.930440+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:20.926924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:20.926924+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:57:44.156600+0000","scrub_duration":0.0018485109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":309,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.930989+0000","last_change":"2024-08-17T17:33:44.154097+0000","last_active":"2024-08-17T17:45:02.930989+0000","last_peered":"2024-08-17T17:45:02.930989+0000","last_clean":"2024-08-17T17:45:02.930989+0000","last_became_active":"2024-08-17T17:33:44.152972+0000","last_became_peered":"2024-08-17T17:33:44.152972+0000","last_unstale":"2024-08-17T17:45:02.930989+0000","last_undegraded":"2024-08-17T17:45:02.930989+0000","last_fullsized":"2024-08-17T17:45:02.930989+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:42.602017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":673,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.931621+0000","last_change":"2024-08-17T17:41:34.615197+0000","last_active":"2024-08-17T17:45:02.931621+0000","last_peered":"2024-08-17T17:45:02.931621+0000","last_clean":"2024-08-17T17:45:02.931621+0000","last_became_active":"2024-08-17T17:41:34.614873+0000","last_became_peered":"2024-08-17T17:41:34.614873+0000","last_unstale":"2024-08-17T17:45:02.931621+0000","last_undegraded":"2024-08-17T17:45:02.931621+0000","last_fullsized":"2024-08-17T17:45:02.931621+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:21:57.391953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2","version":"389'10882","reported_seq":10957,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:20.979293+0000","last_change":"2024-08-17T17:42:36.618260+0000","last_active":"2024-08-17T17:45:20.979293+0000","last_peered":"2024-08-17T17:45:20.979293+0000","last_clean":"2024-08-17T17:45:20.979293+0000","last_became_active":"2024-08-17T17:42:36.617875+0000","last_became_peered":"2024-08-17T17:42:36.617875+0000","last_unstale":"2024-08-17T17:45:20.979293+0000","last_undegraded":"2024-08-17T17:45:20.979293+0000","last_fullsized":"2024-08-17T17:45:20.979293+0000","mapping_epoch":357,"log_start":"372'3900","ondisk_log_start":"372'3900","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6982,"log_dups_size":0,"ondisk_log_size":6982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:09:07.102359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236716032,"num_objects":3612,"num_object_clones":0,"num_object_copies":10836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10882,"num_write_kb":463808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":823,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.929563+0000","last_change":"2024-08-17T17:41:44.585386+0000","last_active":"2024-08-17T17:45:02.929563+0000","last_peered":"2024-08-17T17:45:02.929563+0000","last_clean":"2024-08-17T17:45:02.929563+0000","last_became_active":"2024-08-17T17:41:12.307398+0000","last_became_peered":"2024-08-17T17:41:12.307398+0000","last_unstale":"2024-08-17T17:45:02.929563+0000","last_undegraded":"2024-08-17T17:45:02.929563+0000","last_fullsized":"2024-08-17T17:45:02.929563+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:44.583653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:44.583653+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:27:44.756681+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":708,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.930132+0000","last_change":"2024-08-17T17:41:34.630434+0000","last_active":"2024-08-17T17:45:02.930132+0000","last_peered":"2024-08-17T17:45:02.930132+0000","last_clean":"2024-08-17T17:45:02.930132+0000","last_became_active":"2024-08-17T17:41:34.616406+0000","last_became_peered":"2024-08-17T17:41:34.616406+0000","last_unstale":"2024-08-17T17:45:02.930132+0000","last_undegraded":"2024-08-17T17:45:02.930132+0000","last_fullsized":"2024-08-17T17:45:02.930132+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:30:44.817203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":54,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.932216+0000","last_change":"2024-08-17T17:41:34.611049+0000","last_active":"2024-08-17T17:45:02.932216+0000","last_peered":"2024-08-17T17:45:02.932216+0000","last_clean":"2024-08-17T17:45:02.932216+0000","last_became_active":"2024-08-17T17:41:34.610727+0000","last_became_peered":"2024-08-17T17:41:34.610727+0000","last_unstale":"2024-08-17T17:45:02.932216+0000","last_undegraded":"2024-08-17T17:45:02.932216+0000","last_fullsized":"2024-08-17T17:45:02.932216+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:04:05.477660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"389'11145","reported_seq":11221,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:19.608206+0000","last_change":"2024-08-17T17:42:36.946226+0000","last_active":"2024-08-17T17:45:19.608206+0000","last_peered":"2024-08-17T17:45:19.608206+0000","last_clean":"2024-08-17T17:45:19.608206+0000","last_became_active":"2024-08-17T17:42:36.945809+0000","last_became_peered":"2024-08-17T17:42:36.945809+0000","last_unstale":"2024-08-17T17:45:19.608206+0000","last_undegraded":"2024-08-17T17:45:19.608206+0000","last_fullsized":"2024-08-17T17:45:19.608206+0000","mapping_epoch":357,"log_start":"373'4800","ondisk_log_start":"373'4800","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6345,"log_dups_size":0,"ondisk_log_size":6345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:04:58.633957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254476288,"num_objects":3883,"num_object_clones":0,"num_object_copies":11649,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11145,"num_write_kb":480896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":851,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.929092+0000","last_change":"2024-08-17T17:43:55.595647+0000","last_active":"2024-08-17T17:45:02.929092+0000","last_peered":"2024-08-17T17:45:02.929092+0000","last_clean":"2024-08-17T17:45:02.929092+0000","last_became_active":"2024-08-17T17:43:55.595256+0000","last_became_peered":"2024-08-17T17:43:55.595256+0000","last_unstale":"2024-08-17T17:45:02.929092+0000","last_undegraded":"2024-08-17T17:45:02.929092+0000","last_fullsized":"2024-08-17T17:45:02.929092+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:46.587712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:46.587712+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:05:01.927968+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":54,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.930394+0000","last_change":"2024-08-17T17:41:34.622803+0000","last_active":"2024-08-17T17:45:02.930394+0000","last_peered":"2024-08-17T17:45:02.930394+0000","last_clean":"2024-08-17T17:45:02.930394+0000","last_became_active":"2024-08-17T17:41:34.622525+0000","last_became_peered":"2024-08-17T17:41:34.622525+0000","last_unstale":"2024-08-17T17:45:02.930394+0000","last_undegraded":"2024-08-17T17:45:02.930394+0000","last_fullsized":"2024-08-17T17:45:02.930394+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:42:39.728377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":54,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.930011+0000","last_change":"2024-08-17T17:41:34.619268+0000","last_active":"2024-08-17T17:45:02.930011+0000","last_peered":"2024-08-17T17:45:02.930011+0000","last_clean":"2024-08-17T17:45:02.930011+0000","last_became_active":"2024-08-17T17:41:34.618914+0000","last_became_peered":"2024-08-17T17:41:34.618914+0000","last_unstale":"2024-08-17T17:45:02.930011+0000","last_undegraded":"2024-08-17T17:45:02.930011+0000","last_fullsized":"2024-08-17T17:45:02.930011+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:08:54.248386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.0","version":"390'10942","reported_seq":10983,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:23.912060+0000","last_change":"2024-08-17T17:42:36.617537+0000","last_active":"2024-08-17T17:45:23.912060+0000","last_peered":"2024-08-17T17:45:23.912060+0000","last_clean":"2024-08-17T17:45:23.912060+0000","last_became_active":"2024-08-17T17:42:36.617201+0000","last_became_peered":"2024-08-17T17:42:36.617201+0000","last_unstale":"2024-08-17T17:45:23.912060+0000","last_undegraded":"2024-08-17T17:45:23.912060+0000","last_fullsized":"2024-08-17T17:45:23.912060+0000","mapping_epoch":357,"log_start":"373'4500","ondisk_log_start":"373'4500","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6442,"log_dups_size":0,"ondisk_log_size":6442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:51:58.927214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224395264,"num_objects":3424,"num_object_clones":0,"num_object_copies":10272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10942,"num_write_kb":459712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"90'149","reported_seq":857,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.945233+0000","last_change":"2024-08-17T17:41:41.743931+0000","last_active":"2024-08-17T17:45:02.945233+0000","last_peered":"2024-08-17T17:45:02.945233+0000","last_clean":"2024-08-17T17:45:02.945233+0000","last_became_active":"2024-08-17T17:41:12.203771+0000","last_became_peered":"2024-08-17T17:41:12.203771+0000","last_unstale":"2024-08-17T17:45:02.945233+0000","last_undegraded":"2024-08-17T17:45:02.945233+0000","last_fullsized":"2024-08-17T17:45:02.945233+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:41.743570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:41.743570+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:21:03.865738+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":54,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.929569+0000","last_change":"2024-08-17T17:41:34.607602+0000","last_active":"2024-08-17T17:45:02.929569+0000","last_peered":"2024-08-17T17:45:02.929569+0000","last_clean":"2024-08-17T17:45:02.929569+0000","last_became_active":"2024-08-17T17:41:34.607300+0000","last_became_peered":"2024-08-17T17:41:34.607300+0000","last_unstale":"2024-08-17T17:45:02.929569+0000","last_undegraded":"2024-08-17T17:45:02.929569+0000","last_fullsized":"2024-08-17T17:45:02.929569+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:20:31.486698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":54,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.930311+0000","last_change":"2024-08-17T17:41:34.622102+0000","last_active":"2024-08-17T17:45:02.930311+0000","last_peered":"2024-08-17T17:45:02.930311+0000","last_clean":"2024-08-17T17:45:02.930311+0000","last_became_active":"2024-08-17T17:41:34.621733+0000","last_became_peered":"2024-08-17T17:41:34.621733+0000","last_unstale":"2024-08-17T17:45:02.930311+0000","last_undegraded":"2024-08-17T17:45:02.930311+0000","last_fullsized":"2024-08-17T17:45:02.930311+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:39:41.668709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"390'11515","reported_seq":11554,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:23.753663+0000","last_change":"2024-08-17T17:42:36.620155+0000","last_active":"2024-08-17T17:45:23.753663+0000","last_peered":"2024-08-17T17:45:23.753663+0000","last_clean":"2024-08-17T17:45:23.753663+0000","last_became_active":"2024-08-17T17:42:36.619813+0000","last_became_peered":"2024-08-17T17:42:36.619813+0000","last_unstale":"2024-08-17T17:45:23.753663+0000","last_undegraded":"2024-08-17T17:45:23.753663+0000","last_fullsized":"2024-08-17T17:45:23.753663+0000","mapping_epoch":357,"log_start":"379'6000","ondisk_log_start":"379'6000","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":5515,"log_dups_size":0,"ondisk_log_size":5515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:16:27.534439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235077632,"num_objects":3587,"num_object_clones":0,"num_object_copies":10761,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11515,"num_write_kb":483264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":507,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.936732+0000","last_change":"2024-08-17T17:41:41.036615+0000","last_active":"2024-08-17T17:45:02.936732+0000","last_peered":"2024-08-17T17:45:02.936732+0000","last_clean":"2024-08-17T17:45:02.936732+0000","last_became_active":"2024-08-17T17:41:12.329214+0000","last_became_peered":"2024-08-17T17:41:12.329214+0000","last_unstale":"2024-08-17T17:45:02.936732+0000","last_undegraded":"2024-08-17T17:45:02.936732+0000","last_fullsized":"2024-08-17T17:45:02.936732+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:41.036524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:41.036524+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:59:58.695900+0000","scrub_duration":0.001820803,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":666,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.929458+0000","last_change":"2024-08-17T17:33:20.796216+0000","last_active":"2024-08-17T17:45:02.929458+0000","last_peered":"2024-08-17T17:45:02.929458+0000","last_clean":"2024-08-17T17:45:02.929458+0000","last_became_active":"2024-08-17T17:33:20.795762+0000","last_became_peered":"2024-08-17T17:33:20.795762+0000","last_unstale":"2024-08-17T17:45:02.929458+0000","last_undegraded":"2024-08-17T17:45:02.929458+0000","last_fullsized":"2024-08-17T17:45:02.929458+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:47.038509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":561,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.940063+0000","last_change":"2024-08-17T17:41:34.625551+0000","last_active":"2024-08-17T17:45:02.940063+0000","last_peered":"2024-08-17T17:45:02.940063+0000","last_clean":"2024-08-17T17:45:02.940063+0000","last_became_active":"2024-08-17T17:41:34.625251+0000","last_became_peered":"2024-08-17T17:41:34.625251+0000","last_unstale":"2024-08-17T17:45:02.940063+0000","last_undegraded":"2024-08-17T17:45:02.940063+0000","last_fullsized":"2024-08-17T17:45:02.940063+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:28:22.354213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"390'11373","reported_seq":11414,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:23.913972+0000","last_change":"2024-08-17T17:42:36.619525+0000","last_active":"2024-08-17T17:45:23.913972+0000","last_peered":"2024-08-17T17:45:23.913972+0000","last_clean":"2024-08-17T17:45:23.913972+0000","last_became_active":"2024-08-17T17:42:36.619074+0000","last_became_peered":"2024-08-17T17:42:36.619074+0000","last_unstale":"2024-08-17T17:45:23.913972+0000","last_undegraded":"2024-08-17T17:45:23.913972+0000","last_fullsized":"2024-08-17T17:45:23.913972+0000","mapping_epoch":357,"log_start":"373'4900","ondisk_log_start":"373'4900","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6473,"log_dups_size":0,"ondisk_log_size":6473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:11:32.849616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227475456,"num_objects":3471,"num_object_clones":0,"num_object_copies":10413,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11373,"num_write_kb":475008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":803,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.932603+0000","last_change":"2024-08-17T17:38:25.247159+0000","last_active":"2024-08-17T17:45:02.932603+0000","last_peered":"2024-08-17T17:45:02.932603+0000","last_clean":"2024-08-17T17:45:02.932603+0000","last_became_active":"2024-08-17T17:38:21.311361+0000","last_became_peered":"2024-08-17T17:38:21.311361+0000","last_unstale":"2024-08-17T17:45:02.932603+0000","last_undegraded":"2024-08-17T17:45:02.932603+0000","last_fullsized":"2024-08-17T17:45:02.932603+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:25.247027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:25.247027+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:46:21.734712+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":796,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.925792+0000","last_change":"2024-08-17T17:33:44.149928+0000","last_active":"2024-08-17T17:45:02.925792+0000","last_peered":"2024-08-17T17:45:02.925792+0000","last_clean":"2024-08-17T17:45:02.925792+0000","last_became_active":"2024-08-17T17:33:44.149498+0000","last_became_peered":"2024-08-17T17:33:44.149498+0000","last_unstale":"2024-08-17T17:45:02.925792+0000","last_undegraded":"2024-08-17T17:45:02.925792+0000","last_fullsized":"2024-08-17T17:45:02.925792+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:17:04.963202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":614,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.929229+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:45:02.929229+0000","last_peered":"2024-08-17T17:45:02.929229+0000","last_clean":"2024-08-17T17:45:02.929229+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:45:02.929229+0000","last_undegraded":"2024-08-17T17:45:02.929229+0000","last_fullsized":"2024-08-17T17:45:02.929229+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":231,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.935984+0000","last_change":"2024-08-17T17:41:34.618631+0000","last_active":"2024-08-17T17:45:02.935984+0000","last_peered":"2024-08-17T17:45:02.935984+0000","last_clean":"2024-08-17T17:45:02.935984+0000","last_became_active":"2024-08-17T17:41:34.618313+0000","last_became_peered":"2024-08-17T17:41:34.618313+0000","last_unstale":"2024-08-17T17:45:02.935984+0000","last_undegraded":"2024-08-17T17:45:02.935984+0000","last_fullsized":"2024-08-17T17:45:02.935984+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:07:13.915088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"389'10909","reported_seq":10985,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:20.267703+0000","last_change":"2024-08-17T17:42:36.929502+0000","last_active":"2024-08-17T17:45:20.267703+0000","last_peered":"2024-08-17T17:45:20.267703+0000","last_clean":"2024-08-17T17:45:20.267703+0000","last_became_active":"2024-08-17T17:42:36.929113+0000","last_became_peered":"2024-08-17T17:42:36.929113+0000","last_unstale":"2024-08-17T17:45:20.267703+0000","last_undegraded":"2024-08-17T17:45:20.267703+0000","last_fullsized":"2024-08-17T17:45:20.267703+0000","mapping_epoch":357,"log_start":"373'4300","ondisk_log_start":"373'4300","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6609,"log_dups_size":0,"ondisk_log_size":6609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:31:25.879221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245039104,"num_objects":3739,"num_object_clones":0,"num_object_copies":11217,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10909,"num_write_kb":468736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":839,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.945418+0000","last_change":"2024-08-17T17:41:37.676347+0000","last_active":"2024-08-17T17:45:02.945418+0000","last_peered":"2024-08-17T17:45:02.945418+0000","last_clean":"2024-08-17T17:45:02.945418+0000","last_became_active":"2024-08-17T17:41:37.675869+0000","last_became_peered":"2024-08-17T17:41:37.675869+0000","last_unstale":"2024-08-17T17:45:02.945418+0000","last_undegraded":"2024-08-17T17:45:02.945418+0000","last_fullsized":"2024-08-17T17:45:02.945418+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:21.814077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:21.814077+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:04:16.240754+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":897,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.941334+0000","last_change":"2024-08-17T17:33:20.797705+0000","last_active":"2024-08-17T17:45:02.941334+0000","last_peered":"2024-08-17T17:45:02.941334+0000","last_clean":"2024-08-17T17:45:02.941334+0000","last_became_active":"2024-08-17T17:33:20.797172+0000","last_became_peered":"2024-08-17T17:33:20.797172+0000","last_unstale":"2024-08-17T17:45:02.941334+0000","last_undegraded":"2024-08-17T17:45:02.941334+0000","last_fullsized":"2024-08-17T17:45:02.941334+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:30.754999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":664,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.932637+0000","last_change":"2024-08-17T17:41:34.614108+0000","last_active":"2024-08-17T17:45:02.932637+0000","last_peered":"2024-08-17T17:45:02.932637+0000","last_clean":"2024-08-17T17:45:02.932637+0000","last_became_active":"2024-08-17T17:41:34.613481+0000","last_became_peered":"2024-08-17T17:41:34.613481+0000","last_unstale":"2024-08-17T17:45:02.932637+0000","last_undegraded":"2024-08-17T17:45:02.932637+0000","last_fullsized":"2024-08-17T17:45:02.932637+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:21:02.346115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":451,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.934852+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:45:02.934852+0000","last_peered":"2024-08-17T17:45:02.934852+0000","last_clean":"2024-08-17T17:45:02.934852+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:45:02.934852+0000","last_undegraded":"2024-08-17T17:45:02.934852+0000","last_fullsized":"2024-08-17T17:45:02.934852+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"390'11155","reported_seq":11195,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:23.707932+0000","last_change":"2024-08-17T17:42:36.946382+0000","last_active":"2024-08-17T17:45:23.707932+0000","last_peered":"2024-08-17T17:45:23.707932+0000","last_clean":"2024-08-17T17:45:23.707932+0000","last_became_active":"2024-08-17T17:42:36.945591+0000","last_became_peered":"2024-08-17T17:42:36.945591+0000","last_unstale":"2024-08-17T17:45:23.707932+0000","last_undegraded":"2024-08-17T17:45:23.707932+0000","last_fullsized":"2024-08-17T17:45:23.707932+0000","mapping_epoch":357,"log_start":"374'4900","ondisk_log_start":"374'4900","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6255,"log_dups_size":0,"ondisk_log_size":6255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:08:17.912891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236388352,"num_objects":3607,"num_object_clones":0,"num_object_copies":10821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11155,"num_write_kb":472384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":843,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.928786+0000","last_change":"2024-08-17T17:41:37.683450+0000","last_active":"2024-08-17T17:45:02.928786+0000","last_peered":"2024-08-17T17:45:02.928786+0000","last_clean":"2024-08-17T17:45:02.928786+0000","last_became_active":"2024-08-17T17:41:37.683103+0000","last_became_peered":"2024-08-17T17:41:37.683103+0000","last_unstale":"2024-08-17T17:45:02.928786+0000","last_undegraded":"2024-08-17T17:45:02.928786+0000","last_fullsized":"2024-08-17T17:45:02.928786+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.289528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.289528+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:39:17.141469+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":1031,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.930081+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:45:02.930081+0000","last_peered":"2024-08-17T17:45:02.930081+0000","last_clean":"2024-08-17T17:45:02.930081+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:45:02.930081+0000","last_undegraded":"2024-08-17T17:45:02.930081+0000","last_fullsized":"2024-08-17T17:45:02.930081+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":536,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.942107+0000","last_change":"2024-08-17T17:33:20.793468+0000","last_active":"2024-08-17T17:45:02.942107+0000","last_peered":"2024-08-17T17:45:02.942107+0000","last_clean":"2024-08-17T17:45:02.942107+0000","last_became_active":"2024-08-17T17:33:20.793166+0000","last_became_peered":"2024-08-17T17:33:20.793166+0000","last_unstale":"2024-08-17T17:45:02.942107+0000","last_undegraded":"2024-08-17T17:45:02.942107+0000","last_fullsized":"2024-08-17T17:45:02.942107+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:12:52.121050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":571,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.939975+0000","last_change":"2024-08-17T17:33:20.835784+0000","last_active":"2024-08-17T17:45:02.939975+0000","last_peered":"2024-08-17T17:45:02.939975+0000","last_clean":"2024-08-17T17:45:02.939975+0000","last_became_active":"2024-08-17T17:33:20.835371+0000","last_became_peered":"2024-08-17T17:33:20.835371+0000","last_unstale":"2024-08-17T17:45:02.939975+0000","last_undegraded":"2024-08-17T17:45:02.939975+0000","last_fullsized":"2024-08-17T17:45:02.939975+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:35:43.369055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"390'11248","reported_seq":11289,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:23.889508+0000","last_change":"2024-08-17T17:42:36.946026+0000","last_active":"2024-08-17T17:45:23.889508+0000","last_peered":"2024-08-17T17:45:23.889508+0000","last_clean":"2024-08-17T17:45:23.889508+0000","last_became_active":"2024-08-17T17:42:36.945638+0000","last_became_peered":"2024-08-17T17:42:36.945638+0000","last_unstale":"2024-08-17T17:45:23.889508+0000","last_undegraded":"2024-08-17T17:45:23.889508+0000","last_fullsized":"2024-08-17T17:45:23.889508+0000","mapping_epoch":357,"log_start":"373'4800","ondisk_log_start":"373'4800","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6448,"log_dups_size":0,"ondisk_log_size":6448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:07:34.864262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228458496,"num_objects":3486,"num_object_clones":0,"num_object_copies":10458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11248,"num_write_kb":471488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":822,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.929789+0000","last_change":"2024-08-17T17:38:22.254841+0000","last_active":"2024-08-17T17:45:02.929789+0000","last_peered":"2024-08-17T17:45:02.929789+0000","last_clean":"2024-08-17T17:45:02.929789+0000","last_became_active":"2024-08-17T17:38:21.306619+0000","last_became_peered":"2024-08-17T17:38:21.306619+0000","last_unstale":"2024-08-17T17:45:02.929789+0000","last_undegraded":"2024-08-17T17:45:02.929789+0000","last_fullsized":"2024-08-17T17:45:02.929789+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:22.254705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:22.254705+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:47:38.184089+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":1120,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.942361+0000","last_change":"2024-08-17T17:33:44.148227+0000","last_active":"2024-08-17T17:45:02.942361+0000","last_peered":"2024-08-17T17:45:02.942361+0000","last_clean":"2024-08-17T17:45:02.942361+0000","last_became_active":"2024-08-17T17:33:44.147627+0000","last_became_peered":"2024-08-17T17:33:44.147627+0000","last_unstale":"2024-08-17T17:45:02.942361+0000","last_undegraded":"2024-08-17T17:45:02.942361+0000","last_fullsized":"2024-08-17T17:45:02.942361+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:29:42.898460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1372,"num_read_kb":2843,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":54,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.929836+0000","last_change":"2024-08-17T17:41:34.611109+0000","last_active":"2024-08-17T17:45:02.929836+0000","last_peered":"2024-08-17T17:45:02.929836+0000","last_clean":"2024-08-17T17:45:02.929836+0000","last_became_active":"2024-08-17T17:41:34.610170+0000","last_became_peered":"2024-08-17T17:41:34.610170+0000","last_unstale":"2024-08-17T17:45:02.929836+0000","last_undegraded":"2024-08-17T17:45:02.929836+0000","last_fullsized":"2024-08-17T17:45:02.929836+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:39:05.301260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":575,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.939422+0000","last_change":"2024-08-17T17:33:20.772454+0000","last_active":"2024-08-17T17:45:02.939422+0000","last_peered":"2024-08-17T17:45:02.939422+0000","last_clean":"2024-08-17T17:45:02.939422+0000","last_became_active":"2024-08-17T17:33:20.771848+0000","last_became_peered":"2024-08-17T17:33:20.771848+0000","last_unstale":"2024-08-17T17:45:02.939422+0000","last_undegraded":"2024-08-17T17:45:02.939422+0000","last_fullsized":"2024-08-17T17:45:02.939422+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:46:42.300744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"390'11556","reported_seq":11597,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:23.695207+0000","last_change":"2024-08-17T17:42:36.618414+0000","last_active":"2024-08-17T17:45:23.695207+0000","last_peered":"2024-08-17T17:45:23.695207+0000","last_clean":"2024-08-17T17:45:23.695207+0000","last_became_active":"2024-08-17T17:42:36.618086+0000","last_became_peered":"2024-08-17T17:42:36.618086+0000","last_unstale":"2024-08-17T17:45:23.695207+0000","last_undegraded":"2024-08-17T17:45:23.695207+0000","last_fullsized":"2024-08-17T17:45:23.695207+0000","mapping_epoch":357,"log_start":"374'5300","ondisk_log_start":"374'5300","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6256,"log_dups_size":0,"ondisk_log_size":6256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:26:27.772894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237895680,"num_objects":3630,"num_object_clones":0,"num_object_copies":10890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11556,"num_write_kb":485952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":527,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.937522+0000","last_change":"2024-08-17T17:41:43.035215+0000","last_active":"2024-08-17T17:45:02.937522+0000","last_peered":"2024-08-17T17:45:02.937522+0000","last_clean":"2024-08-17T17:45:02.937522+0000","last_became_active":"2024-08-17T17:41:12.201158+0000","last_became_peered":"2024-08-17T17:41:12.201158+0000","last_unstale":"2024-08-17T17:45:02.937522+0000","last_undegraded":"2024-08-17T17:45:02.937522+0000","last_fullsized":"2024-08-17T17:45:02.937522+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:43.035108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:43.035108+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:43:27.108994+0000","scrub_duration":0.0028877400000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":326,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.930177+0000","last_change":"2024-08-17T17:33:44.163867+0000","last_active":"2024-08-17T17:45:02.930177+0000","last_peered":"2024-08-17T17:45:02.930177+0000","last_clean":"2024-08-17T17:45:02.930177+0000","last_became_active":"2024-08-17T17:33:44.163503+0000","last_became_peered":"2024-08-17T17:33:44.163503+0000","last_unstale":"2024-08-17T17:45:02.930177+0000","last_undegraded":"2024-08-17T17:45:02.930177+0000","last_fullsized":"2024-08-17T17:45:02.930177+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:31:53.573596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":790,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.939137+0000","last_change":"2024-08-17T17:33:20.799350+0000","last_active":"2024-08-17T17:45:02.939137+0000","last_peered":"2024-08-17T17:45:02.939137+0000","last_clean":"2024-08-17T17:45:02.939137+0000","last_became_active":"2024-08-17T17:33:20.798925+0000","last_became_peered":"2024-08-17T17:33:20.798925+0000","last_unstale":"2024-08-17T17:45:02.939137+0000","last_undegraded":"2024-08-17T17:45:02.939137+0000","last_fullsized":"2024-08-17T17:45:02.939137+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:16:36.121295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":606,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.943148+0000","last_change":"2024-08-17T17:41:34.627064+0000","last_active":"2024-08-17T17:45:02.943148+0000","last_peered":"2024-08-17T17:45:02.943148+0000","last_clean":"2024-08-17T17:45:02.943148+0000","last_became_active":"2024-08-17T17:41:34.626453+0000","last_became_peered":"2024-08-17T17:41:34.626453+0000","last_unstale":"2024-08-17T17:45:02.943148+0000","last_undegraded":"2024-08-17T17:45:02.943148+0000","last_fullsized":"2024-08-17T17:45:02.943148+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:02:11.894042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"390'11348","reported_seq":11390,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:23.912741+0000","last_change":"2024-08-17T17:42:36.620669+0000","last_active":"2024-08-17T17:45:23.912741+0000","last_peered":"2024-08-17T17:45:23.912741+0000","last_clean":"2024-08-17T17:45:23.912741+0000","last_became_active":"2024-08-17T17:42:36.619876+0000","last_became_peered":"2024-08-17T17:42:36.619876+0000","last_unstale":"2024-08-17T17:45:23.912741+0000","last_undegraded":"2024-08-17T17:45:23.912741+0000","last_fullsized":"2024-08-17T17:45:23.912741+0000","mapping_epoch":357,"log_start":"373'4900","ondisk_log_start":"373'4900","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6448,"log_dups_size":0,"ondisk_log_size":6448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:59:34.661001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231014424,"num_objects":3526,"num_object_clones":0,"num_object_copies":10578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3526,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11348,"num_write_kb":475905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":527,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.935635+0000","last_change":"2024-08-17T17:41:44.058291+0000","last_active":"2024-08-17T17:45:02.935635+0000","last_peered":"2024-08-17T17:45:02.935635+0000","last_clean":"2024-08-17T17:45:02.935635+0000","last_became_active":"2024-08-17T17:41:12.200077+0000","last_became_peered":"2024-08-17T17:41:12.200077+0000","last_unstale":"2024-08-17T17:45:02.935635+0000","last_undegraded":"2024-08-17T17:45:02.935635+0000","last_fullsized":"2024-08-17T17:45:02.935635+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:44.058233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:44.058233+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:42:55.911623+0000","scrub_duration":0.0074268499999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":610,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.939033+0000","last_change":"2024-08-17T17:33:20.798613+0000","last_active":"2024-08-17T17:45:02.939033+0000","last_peered":"2024-08-17T17:45:02.939033+0000","last_clean":"2024-08-17T17:45:02.939033+0000","last_became_active":"2024-08-17T17:33:20.798260+0000","last_became_peered":"2024-08-17T17:33:20.798260+0000","last_unstale":"2024-08-17T17:45:02.939033+0000","last_undegraded":"2024-08-17T17:45:02.939033+0000","last_fullsized":"2024-08-17T17:45:02.939033+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:18:00.822681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":950,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.927403+0000","last_change":"2024-08-17T17:41:37.683212+0000","last_active":"2024-08-17T17:45:02.927403+0000","last_peered":"2024-08-17T17:45:02.927403+0000","last_clean":"2024-08-17T17:45:02.927403+0000","last_became_active":"2024-08-17T17:41:37.682730+0000","last_became_peered":"2024-08-17T17:41:37.682730+0000","last_unstale":"2024-08-17T17:45:02.927403+0000","last_undegraded":"2024-08-17T17:45:02.927403+0000","last_fullsized":"2024-08-17T17:45:02.927403+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:51:35.444433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":626,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.931544+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:45:02.931544+0000","last_peered":"2024-08-17T17:45:02.931544+0000","last_clean":"2024-08-17T17:45:02.931544+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:45:02.931544+0000","last_undegraded":"2024-08-17T17:45:02.931544+0000","last_fullsized":"2024-08-17T17:45:02.931544+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"390'11359","reported_seq":11398,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:23.769456+0000","last_change":"2024-08-17T17:42:36.929709+0000","last_active":"2024-08-17T17:45:23.769456+0000","last_peered":"2024-08-17T17:45:23.769456+0000","last_clean":"2024-08-17T17:45:23.769456+0000","last_became_active":"2024-08-17T17:42:36.929285+0000","last_became_peered":"2024-08-17T17:42:36.929285+0000","last_unstale":"2024-08-17T17:45:23.769456+0000","last_undegraded":"2024-08-17T17:45:23.769456+0000","last_fullsized":"2024-08-17T17:45:23.769456+0000","mapping_epoch":357,"log_start":"376'5900","ondisk_log_start":"376'5900","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":5459,"log_dups_size":0,"ondisk_log_size":5459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:47:39.760414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230359040,"num_objects":3515,"num_object_clones":0,"num_object_copies":10545,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11359,"num_write_kb":475968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":773,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.929700+0000","last_change":"2024-08-17T17:42:19.960161+0000","last_active":"2024-08-17T17:45:02.929700+0000","last_peered":"2024-08-17T17:45:02.929700+0000","last_clean":"2024-08-17T17:45:02.929700+0000","last_became_active":"2024-08-17T17:41:37.665223+0000","last_became_peered":"2024-08-17T17:41:37.665223+0000","last_unstale":"2024-08-17T17:45:02.929700+0000","last_undegraded":"2024-08-17T17:45:02.929700+0000","last_fullsized":"2024-08-17T17:45:02.929700+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:19.960086+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:19.960086+0000","last_clean_scrub_stamp":"2024-08-17T17:42:19.960086+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:57.861661+0000","scrub_duration":0.0022612000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":660,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.933215+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:45:02.933215+0000","last_peered":"2024-08-17T17:45:02.933215+0000","last_clean":"2024-08-17T17:45:02.933215+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:45:02.933215+0000","last_undegraded":"2024-08-17T17:45:02.933215+0000","last_fullsized":"2024-08-17T17:45:02.933215+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":1235,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.939667+0000","last_change":"2024-08-17T17:41:37.677524+0000","last_active":"2024-08-17T17:45:02.939667+0000","last_peered":"2024-08-17T17:45:02.939667+0000","last_clean":"2024-08-17T17:45:02.939667+0000","last_became_active":"2024-08-17T17:41:37.676764+0000","last_became_peered":"2024-08-17T17:41:37.676764+0000","last_unstale":"2024-08-17T17:45:02.939667+0000","last_undegraded":"2024-08-17T17:45:02.939667+0000","last_fullsized":"2024-08-17T17:45:02.939667+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:14:49.341080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":657,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.929286+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:45:02.929286+0000","last_peered":"2024-08-17T17:45:02.929286+0000","last_clean":"2024-08-17T17:45:02.929286+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:45:02.929286+0000","last_undegraded":"2024-08-17T17:45:02.929286+0000","last_fullsized":"2024-08-17T17:45:02.929286+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"390'11235","reported_seq":11276,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:23.885874+0000","last_change":"2024-08-17T17:42:36.929613+0000","last_active":"2024-08-17T17:45:23.885874+0000","last_peered":"2024-08-17T17:45:23.885874+0000","last_clean":"2024-08-17T17:45:23.885874+0000","last_became_active":"2024-08-17T17:42:36.929191+0000","last_became_peered":"2024-08-17T17:42:36.929191+0000","last_unstale":"2024-08-17T17:45:23.885874+0000","last_undegraded":"2024-08-17T17:45:23.885874+0000","last_fullsized":"2024-08-17T17:45:23.885874+0000","mapping_epoch":357,"log_start":"373'4800","ondisk_log_start":"373'4800","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6435,"log_dups_size":0,"ondisk_log_size":6435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:21:40.881779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228392960,"num_objects":3485,"num_object_clones":0,"num_object_copies":10455,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11235,"num_write_kb":471040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":804,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.930523+0000","last_change":"2024-08-17T17:38:23.113397+0000","last_active":"2024-08-17T17:45:02.930523+0000","last_peered":"2024-08-17T17:45:02.930523+0000","last_clean":"2024-08-17T17:45:02.930523+0000","last_became_active":"2024-08-17T17:38:21.305580+0000","last_became_peered":"2024-08-17T17:38:21.305580+0000","last_unstale":"2024-08-17T17:45:02.930523+0000","last_undegraded":"2024-08-17T17:45:02.930523+0000","last_fullsized":"2024-08-17T17:45:02.930523+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.113279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.113279+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:38:07.015721+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":497,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.934885+0000","last_change":"2024-08-17T17:41:34.629522+0000","last_active":"2024-08-17T17:45:02.934885+0000","last_peered":"2024-08-17T17:45:02.934885+0000","last_clean":"2024-08-17T17:45:02.934885+0000","last_became_active":"2024-08-17T17:41:34.619340+0000","last_became_peered":"2024-08-17T17:41:34.619340+0000","last_unstale":"2024-08-17T17:45:02.934885+0000","last_undegraded":"2024-08-17T17:45:02.934885+0000","last_fullsized":"2024-08-17T17:45:02.934885+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:30:03.902956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":620,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.937884+0000","last_change":"2024-08-17T17:26:06.855307+0000","last_active":"2024-08-17T17:45:02.937884+0000","last_peered":"2024-08-17T17:45:02.937884+0000","last_clean":"2024-08-17T17:45:02.937884+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:45:02.937884+0000","last_undegraded":"2024-08-17T17:45:02.937884+0000","last_fullsized":"2024-08-17T17:45:02.937884+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":54,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.929422+0000","last_change":"2024-08-17T17:41:34.614028+0000","last_active":"2024-08-17T17:45:02.929422+0000","last_peered":"2024-08-17T17:45:02.929422+0000","last_clean":"2024-08-17T17:45:02.929422+0000","last_became_active":"2024-08-17T17:41:34.613706+0000","last_became_peered":"2024-08-17T17:41:34.613706+0000","last_unstale":"2024-08-17T17:45:02.929422+0000","last_undegraded":"2024-08-17T17:45:02.929422+0000","last_fullsized":"2024-08-17T17:45:02.929422+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:19:55.254902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"90'149","reported_seq":715,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.939519+0000","last_change":"2024-08-17T17:42:07.681204+0000","last_active":"2024-08-17T17:45:02.939519+0000","last_peered":"2024-08-17T17:45:02.939519+0000","last_clean":"2024-08-17T17:45:02.939519+0000","last_became_active":"2024-08-17T17:42:05.878769+0000","last_became_peered":"2024-08-17T17:42:05.878769+0000","last_unstale":"2024-08-17T17:45:02.939519+0000","last_undegraded":"2024-08-17T17:45:02.939519+0000","last_fullsized":"2024-08-17T17:45:02.939519+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:07.681064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:07.681064+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:12:12.835715+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":54,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.929927+0000","last_change":"2024-08-17T17:41:34.622180+0000","last_active":"2024-08-17T17:45:02.929927+0000","last_peered":"2024-08-17T17:45:02.929927+0000","last_clean":"2024-08-17T17:45:02.929927+0000","last_became_active":"2024-08-17T17:41:34.621724+0000","last_became_peered":"2024-08-17T17:41:34.621724+0000","last_unstale":"2024-08-17T17:45:02.929927+0000","last_undegraded":"2024-08-17T17:45:02.929927+0000","last_fullsized":"2024-08-17T17:45:02.929927+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:15.807235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":64,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.932237+0000","last_change":"2024-08-17T17:41:34.626584+0000","last_active":"2024-08-17T17:45:02.932237+0000","last_peered":"2024-08-17T17:45:02.932237+0000","last_clean":"2024-08-17T17:45:02.932237+0000","last_became_active":"2024-08-17T17:41:34.609878+0000","last_became_peered":"2024-08-17T17:41:34.609878+0000","last_unstale":"2024-08-17T17:45:02.932237+0000","last_undegraded":"2024-08-17T17:45:02.932237+0000","last_fullsized":"2024-08-17T17:45:02.932237+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:08:12.810347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.17","version":"90'149","reported_seq":474,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.933462+0000","last_change":"2024-08-17T17:42:06.699082+0000","last_active":"2024-08-17T17:45:02.933462+0000","last_peered":"2024-08-17T17:45:02.933462+0000","last_clean":"2024-08-17T17:45:02.933462+0000","last_became_active":"2024-08-17T17:42:05.876218+0000","last_became_peered":"2024-08-17T17:42:05.876218+0000","last_unstale":"2024-08-17T17:45:02.933462+0000","last_undegraded":"2024-08-17T17:45:02.933462+0000","last_fullsized":"2024-08-17T17:45:02.933462+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.698998+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:06.698998+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.698998+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:22:16.510150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":603,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.942540+0000","last_change":"2024-08-17T17:33:20.799402+0000","last_active":"2024-08-17T17:45:02.942540+0000","last_peered":"2024-08-17T17:45:02.942540+0000","last_clean":"2024-08-17T17:45:02.942540+0000","last_became_active":"2024-08-17T17:33:20.798832+0000","last_became_peered":"2024-08-17T17:33:20.798832+0000","last_unstale":"2024-08-17T17:45:02.942540+0000","last_undegraded":"2024-08-17T17:45:02.942540+0000","last_fullsized":"2024-08-17T17:45:02.942540+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:21:05.001134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":378,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.930071+0000","last_change":"2024-08-17T17:41:34.634301+0000","last_active":"2024-08-17T17:45:02.930071+0000","last_peered":"2024-08-17T17:45:02.930071+0000","last_clean":"2024-08-17T17:45:02.930071+0000","last_became_active":"2024-08-17T17:41:34.620508+0000","last_became_peered":"2024-08-17T17:41:34.620508+0000","last_unstale":"2024-08-17T17:45:02.930071+0000","last_undegraded":"2024-08-17T17:45:02.930071+0000","last_fullsized":"2024-08-17T17:45:02.930071+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:10:54.576812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.14","version":"90'149","reported_seq":827,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.934548+0000","last_change":"2024-08-17T17:41:37.664471+0000","last_active":"2024-08-17T17:45:02.934548+0000","last_peered":"2024-08-17T17:45:02.934548+0000","last_clean":"2024-08-17T17:45:02.934548+0000","last_became_active":"2024-08-17T17:41:37.664108+0000","last_became_peered":"2024-08-17T17:41:37.664108+0000","last_unstale":"2024-08-17T17:45:02.934548+0000","last_undegraded":"2024-08-17T17:45:02.934548+0000","last_fullsized":"2024-08-17T17:45:02.934548+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:22.030514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:22.030514+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:15:52.325163+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":341,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.939757+0000","last_change":"2024-08-17T17:33:44.146014+0000","last_active":"2024-08-17T17:45:02.939757+0000","last_peered":"2024-08-17T17:45:02.939757+0000","last_clean":"2024-08-17T17:45:02.939757+0000","last_became_active":"2024-08-17T17:33:44.145668+0000","last_became_peered":"2024-08-17T17:33:44.145668+0000","last_unstale":"2024-08-17T17:45:02.939757+0000","last_undegraded":"2024-08-17T17:45:02.939757+0000","last_fullsized":"2024-08-17T17:45:02.939757+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:53:41.679237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":606,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.929705+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:45:02.929705+0000","last_peered":"2024-08-17T17:45:02.929705+0000","last_clean":"2024-08-17T17:45:02.929705+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:45:02.929705+0000","last_undegraded":"2024-08-17T17:45:02.929705+0000","last_fullsized":"2024-08-17T17:45:02.929705+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"90'149","reported_seq":776,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.927337+0000","last_change":"2024-08-17T17:42:06.456562+0000","last_active":"2024-08-17T17:45:02.927337+0000","last_peered":"2024-08-17T17:45:02.927337+0000","last_clean":"2024-08-17T17:45:02.927337+0000","last_became_active":"2024-08-17T17:42:05.876577+0000","last_became_peered":"2024-08-17T17:42:05.876577+0000","last_unstale":"2024-08-17T17:45:02.927337+0000","last_undegraded":"2024-08-17T17:45:02.927337+0000","last_fullsized":"2024-08-17T17:45:02.927337+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.456490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.456490+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:10:03.654953+0000","scrub_duration":0.001537285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":634,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.933765+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:45:02.933765+0000","last_peered":"2024-08-17T17:45:02.933765+0000","last_clean":"2024-08-17T17:45:02.933765+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:45:02.933765+0000","last_undegraded":"2024-08-17T17:45:02.933765+0000","last_fullsized":"2024-08-17T17:45:02.933765+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":664,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.929203+0000","last_change":"2024-08-17T17:33:20.790687+0000","last_active":"2024-08-17T17:45:02.929203+0000","last_peered":"2024-08-17T17:45:02.929203+0000","last_clean":"2024-08-17T17:45:02.929203+0000","last_became_active":"2024-08-17T17:33:20.790399+0000","last_became_peered":"2024-08-17T17:33:20.790399+0000","last_unstale":"2024-08-17T17:45:02.929203+0000","last_undegraded":"2024-08-17T17:45:02.929203+0000","last_fullsized":"2024-08-17T17:45:02.929203+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:38:06.849989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1a","version":"90'149","reported_seq":438,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.930107+0000","last_change":"2024-08-17T17:42:21.932830+0000","last_active":"2024-08-17T17:45:02.930107+0000","last_peered":"2024-08-17T17:45:02.930107+0000","last_clean":"2024-08-17T17:45:02.930107+0000","last_became_active":"2024-08-17T17:42:05.876835+0000","last_became_peered":"2024-08-17T17:42:05.876835+0000","last_unstale":"2024-08-17T17:45:02.930107+0000","last_undegraded":"2024-08-17T17:45:02.930107+0000","last_fullsized":"2024-08-17T17:45:02.930107+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:21.932757+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:21.932757+0000","last_clean_scrub_stamp":"2024-08-17T17:42:21.932757+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:36:39.848570+0000","scrub_duration":0.0020218810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":363,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.937730+0000","last_change":"2024-08-17T17:33:20.835760+0000","last_active":"2024-08-17T17:45:02.937730+0000","last_peered":"2024-08-17T17:45:02.937730+0000","last_clean":"2024-08-17T17:45:02.937730+0000","last_became_active":"2024-08-17T17:33:20.835410+0000","last_became_peered":"2024-08-17T17:33:20.835410+0000","last_unstale":"2024-08-17T17:45:02.937730+0000","last_undegraded":"2024-08-17T17:45:02.937730+0000","last_fullsized":"2024-08-17T17:45:02.937730+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:45.448022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":682,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.929364+0000","last_change":"2024-08-17T17:33:20.771279+0000","last_active":"2024-08-17T17:45:02.929364+0000","last_peered":"2024-08-17T17:45:02.929364+0000","last_clean":"2024-08-17T17:45:02.929364+0000","last_became_active":"2024-08-17T17:33:20.770728+0000","last_became_peered":"2024-08-17T17:33:20.770728+0000","last_unstale":"2024-08-17T17:45:02.929364+0000","last_undegraded":"2024-08-17T17:45:02.929364+0000","last_fullsized":"2024-08-17T17:45:02.929364+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:52:31.540397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"90'149","reported_seq":733,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.929750+0000","last_change":"2024-08-17T17:42:26.967686+0000","last_active":"2024-08-17T17:45:02.929750+0000","last_peered":"2024-08-17T17:45:02.929750+0000","last_clean":"2024-08-17T17:45:02.929750+0000","last_became_active":"2024-08-17T17:42:05.873657+0000","last_became_peered":"2024-08-17T17:42:05.873657+0000","last_unstale":"2024-08-17T17:45:02.929750+0000","last_undegraded":"2024-08-17T17:45:02.929750+0000","last_fullsized":"2024-08-17T17:45:02.929750+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:26.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:26.967611+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:31:03.344332+0000","scrub_duration":0.0015978279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":452,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.934719+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:45:02.934719+0000","last_peered":"2024-08-17T17:45:02.934719+0000","last_clean":"2024-08-17T17:45:02.934719+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:45:02.934719+0000","last_undegraded":"2024-08-17T17:45:02.934719+0000","last_fullsized":"2024-08-17T17:45:02.934719+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":601,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.939217+0000","last_change":"2024-08-17T17:41:34.625011+0000","last_active":"2024-08-17T17:45:02.939217+0000","last_peered":"2024-08-17T17:45:02.939217+0000","last_clean":"2024-08-17T17:45:02.939217+0000","last_became_active":"2024-08-17T17:41:34.624631+0000","last_became_peered":"2024-08-17T17:41:34.624631+0000","last_unstale":"2024-08-17T17:45:02.939217+0000","last_undegraded":"2024-08-17T17:45:02.939217+0000","last_fullsized":"2024-08-17T17:45:02.939217+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:03:05.565435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"90'149","reported_seq":696,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.930846+0000","last_change":"2024-08-17T17:42:08.482975+0000","last_active":"2024-08-17T17:45:02.930846+0000","last_peered":"2024-08-17T17:45:02.930846+0000","last_clean":"2024-08-17T17:45:02.930846+0000","last_became_active":"2024-08-17T17:42:05.873523+0000","last_became_peered":"2024-08-17T17:42:05.873523+0000","last_unstale":"2024-08-17T17:45:02.930846+0000","last_undegraded":"2024-08-17T17:45:02.930846+0000","last_fullsized":"2024-08-17T17:45:02.930846+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:08.482892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:08.482892+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:41.477341+0000","scrub_duration":0.0017925479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":54,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.930125+0000","last_change":"2024-08-17T17:41:34.616482+0000","last_active":"2024-08-17T17:45:02.930125+0000","last_peered":"2024-08-17T17:45:02.930125+0000","last_clean":"2024-08-17T17:45:02.930125+0000","last_became_active":"2024-08-17T17:41:34.616040+0000","last_became_peered":"2024-08-17T17:41:34.616040+0000","last_unstale":"2024-08-17T17:45:02.930125+0000","last_undegraded":"2024-08-17T17:45:02.930125+0000","last_fullsized":"2024-08-17T17:45:02.930125+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:53:08.502264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":54,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.930312+0000","last_change":"2024-08-17T17:41:34.622184+0000","last_active":"2024-08-17T17:45:02.930312+0000","last_peered":"2024-08-17T17:45:02.930312+0000","last_clean":"2024-08-17T17:45:02.930312+0000","last_became_active":"2024-08-17T17:41:34.621831+0000","last_became_peered":"2024-08-17T17:41:34.621831+0000","last_unstale":"2024-08-17T17:45:02.930312+0000","last_undegraded":"2024-08-17T17:45:02.930312+0000","last_fullsized":"2024-08-17T17:45:02.930312+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:59.826667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"90'149","reported_seq":831,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.929092+0000","last_change":"2024-08-17T17:42:05.876463+0000","last_active":"2024-08-17T17:45:02.929092+0000","last_peered":"2024-08-17T17:45:02.929092+0000","last_clean":"2024-08-17T17:45:02.929092+0000","last_became_active":"2024-08-17T17:42:05.876069+0000","last_became_peered":"2024-08-17T17:42:05.876069+0000","last_unstale":"2024-08-17T17:45:02.929092+0000","last_undegraded":"2024-08-17T17:45:02.929092+0000","last_fullsized":"2024-08-17T17:45:02.929092+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:45.552985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:45.552985+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:55:06.027278+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":568,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.939341+0000","last_change":"2024-08-17T17:33:20.773207+0000","last_active":"2024-08-17T17:45:02.939341+0000","last_peered":"2024-08-17T17:45:02.939341+0000","last_clean":"2024-08-17T17:45:02.939341+0000","last_became_active":"2024-08-17T17:33:20.772457+0000","last_became_peered":"2024-08-17T17:33:20.772457+0000","last_unstale":"2024-08-17T17:45:02.939341+0000","last_undegraded":"2024-08-17T17:45:02.939341+0000","last_fullsized":"2024-08-17T17:45:02.939341+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:57:09.610084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":475,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.931298+0000","last_change":"2024-08-17T17:41:34.618968+0000","last_active":"2024-08-17T17:45:02.931298+0000","last_peered":"2024-08-17T17:45:02.931298+0000","last_clean":"2024-08-17T17:45:02.931298+0000","last_became_active":"2024-08-17T17:41:34.618576+0000","last_became_peered":"2024-08-17T17:41:34.618576+0000","last_unstale":"2024-08-17T17:45:02.931298+0000","last_undegraded":"2024-08-17T17:45:02.931298+0000","last_fullsized":"2024-08-17T17:45:02.931298+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:47:45.888590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"90'149","reported_seq":479,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.929935+0000","last_change":"2024-08-17T17:42:06.742801+0000","last_active":"2024-08-17T17:45:02.929935+0000","last_peered":"2024-08-17T17:45:02.929935+0000","last_clean":"2024-08-17T17:45:02.929935+0000","last_became_active":"2024-08-17T17:42:05.876895+0000","last_became_peered":"2024-08-17T17:42:05.876895+0000","last_unstale":"2024-08-17T17:45:02.929935+0000","last_undegraded":"2024-08-17T17:45:02.929935+0000","last_fullsized":"2024-08-17T17:45:02.929935+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.742723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.742723+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:36:46.651953+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":54,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.929655+0000","last_change":"2024-08-17T17:41:34.622094+0000","last_active":"2024-08-17T17:45:02.929655+0000","last_peered":"2024-08-17T17:45:02.929655+0000","last_clean":"2024-08-17T17:45:02.929655+0000","last_became_active":"2024-08-17T17:41:34.621773+0000","last_became_peered":"2024-08-17T17:41:34.621773+0000","last_unstale":"2024-08-17T17:45:02.929655+0000","last_undegraded":"2024-08-17T17:45:02.929655+0000","last_fullsized":"2024-08-17T17:45:02.929655+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:34:30.540886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":58,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.932290+0000","last_change":"2024-08-17T17:41:34.629135+0000","last_active":"2024-08-17T17:45:02.932290+0000","last_peered":"2024-08-17T17:45:02.932290+0000","last_clean":"2024-08-17T17:45:02.932290+0000","last_became_active":"2024-08-17T17:41:34.613095+0000","last_became_peered":"2024-08-17T17:41:34.613095+0000","last_unstale":"2024-08-17T17:45:02.932290+0000","last_undegraded":"2024-08-17T17:45:02.932290+0000","last_fullsized":"2024-08-17T17:45:02.932290+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:27:06.542416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":54,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.930465+0000","last_change":"2024-08-17T17:41:34.616251+0000","last_active":"2024-08-17T17:45:02.930465+0000","last_peered":"2024-08-17T17:45:02.930465+0000","last_clean":"2024-08-17T17:45:02.930465+0000","last_became_active":"2024-08-17T17:41:34.615952+0000","last_became_peered":"2024-08-17T17:41:34.615952+0000","last_unstale":"2024-08-17T17:45:02.930465+0000","last_undegraded":"2024-08-17T17:45:02.930465+0000","last_fullsized":"2024-08-17T17:45:02.930465+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:12:24.892508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":58,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.929779+0000","last_change":"2024-08-17T17:41:34.631152+0000","last_active":"2024-08-17T17:45:02.929779+0000","last_peered":"2024-08-17T17:45:02.929779+0000","last_clean":"2024-08-17T17:45:02.929779+0000","last_became_active":"2024-08-17T17:41:34.615617+0000","last_became_peered":"2024-08-17T17:41:34.615617+0000","last_unstale":"2024-08-17T17:45:02.929779+0000","last_undegraded":"2024-08-17T17:45:02.929779+0000","last_fullsized":"2024-08-17T17:45:02.929779+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:43:08.257084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"90'149","reported_seq":730,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.935234+0000","last_change":"2024-08-17T17:42:11.809047+0000","last_active":"2024-08-17T17:45:02.935234+0000","last_peered":"2024-08-17T17:45:02.935234+0000","last_clean":"2024-08-17T17:45:02.935234+0000","last_became_active":"2024-08-17T17:42:05.883770+0000","last_became_peered":"2024-08-17T17:42:05.883770+0000","last_unstale":"2024-08-17T17:45:02.935234+0000","last_undegraded":"2024-08-17T17:45:02.935234+0000","last_fullsized":"2024-08-17T17:45:02.935234+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:11.808971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:11.808971+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:48:15.989437+0000","scrub_duration":0.0018167420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":660,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.930129+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:45:02.930129+0000","last_peered":"2024-08-17T17:45:02.930129+0000","last_clean":"2024-08-17T17:45:02.930129+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:45:02.930129+0000","last_undegraded":"2024-08-17T17:45:02.930129+0000","last_fullsized":"2024-08-17T17:45:02.930129+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":661,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.930042+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:45:02.930042+0000","last_peered":"2024-08-17T17:45:02.930042+0000","last_clean":"2024-08-17T17:45:02.930042+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:45:02.930042+0000","last_undegraded":"2024-08-17T17:45:02.930042+0000","last_fullsized":"2024-08-17T17:45:02.930042+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1d","version":"90'149","reported_seq":727,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.939521+0000","last_change":"2024-08-17T17:42:18.668058+0000","last_active":"2024-08-17T17:45:02.939521+0000","last_peered":"2024-08-17T17:45:02.939521+0000","last_clean":"2024-08-17T17:45:02.939521+0000","last_became_active":"2024-08-17T17:42:05.876721+0000","last_became_peered":"2024-08-17T17:42:05.876721+0000","last_unstale":"2024-08-17T17:45:02.939521+0000","last_undegraded":"2024-08-17T17:45:02.939521+0000","last_fullsized":"2024-08-17T17:45:02.939521+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:18.667957+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:18.667957+0000","last_clean_scrub_stamp":"2024-08-17T17:42:18.667957+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:38.170146+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":54,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.930199+0000","last_change":"2024-08-17T17:41:34.611711+0000","last_active":"2024-08-17T17:45:02.930199+0000","last_peered":"2024-08-17T17:45:02.930199+0000","last_clean":"2024-08-17T17:45:02.930199+0000","last_became_active":"2024-08-17T17:41:34.611407+0000","last_became_peered":"2024-08-17T17:41:34.611407+0000","last_unstale":"2024-08-17T17:45:02.930199+0000","last_undegraded":"2024-08-17T17:45:02.930199+0000","last_fullsized":"2024-08-17T17:45:02.930199+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:42:41.582320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":471,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-08-17T17:45:02.933669+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:45:02.933669+0000","last_peered":"2024-08-17T17:45:02.933669+0000","last_clean":"2024-08-17T17:45:02.933669+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:45:02.933669+0000","last_undegraded":"2024-08-17T17:45:02.933669+0000","last_fullsized":"2024-08-17T17:45:02.933669+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":3788308504,"num_objects":57806,"num_object_clones":0,"num_object_copies":173418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":1957,"num_objects_unfound":0,"num_objects_dirty":57806,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":179282,"num_write_kb":7586753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2724,"num_bytes_recovered":178520064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11712868352,"data_stored":11712856136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101782,"ondisk_log_size":101782,"up":48,"acting":47,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":11256,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19280,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5978,"num_read_kb":12991,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4619,"ondisk_log_size":4619,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234463,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1821836,"kb_used_data":927940,"kb_used_omap":1,"kb_used_meta":893886,"kb_avail":91923316,"statfs":{"total":95995035648,"available":94129475584,"internally_reserved":0,"allocated":950210560,"data_stored":947983771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":915339608},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496098,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2483776,"kb_used_data":1584504,"kb_used_omap":1,"kb_used_meta":899262,"kb_avail":91261376,"statfs":{"total":95995035648,"available":93451649024,"internally_reserved":0,"allocated":1622532096,"data_stored":1620316749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":920844632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757735,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1912828,"kb_used_data":1030516,"kb_used_omap":1,"kb_used_meta":882302,"kb_avail":91832324,"statfs":{"total":95995035648,"available":94036299776,"internally_reserved":0,"allocated":1055248384,"data_stored":1053033678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":903477592},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019373,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2314000,"kb_used_data":2016644,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":91431152,"statfs":{"total":95995035648,"available":93625499648,"internally_reserved":0,"allocated":2065043456,"data_stored":2062816408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435599,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2196008,"kb_used_data":1245600,"kb_used_omap":4,"kb_used_meta":950395,"kb_avail":91549144,"statfs":{"total":95995035648,"available":93746323456,"internally_reserved":0,"allocated":1275494400,"data_stored":1273291040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":973204490},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599126,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2666608,"kb_used_data":1848872,"kb_used_omap":7,"kb_used_meta":817720,"kb_avail":91078544,"statfs":{"total":95995035648,"available":93264429056,"internally_reserved":0,"allocated":1893244928,"data_stored":1891034434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":837345442},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":268,"seq":1151051235473,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1153832,"kb_used_data":707744,"kb_used_omap":10,"kb_used_meta":446069,"kb_avail":92591320,"statfs":{"total":95995035648,"available":94813511680,"internally_reserved":0,"allocated":724729856,"data_stored":722526441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10902,"internal_metadata":456775018},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926179,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3352432,"kb_used_data":2448488,"kb_used_omap":7,"kb_used_meta":903928,"kb_avail":90392720,"statfs":{"total":95995035648,"available":92562145280,"internally_reserved":0,"allocated":2507251712,"data_stored":2505040694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":925622593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7550,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2463502336,"data_stored":2463498264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":711786496,"data_stored":711786496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1866858496,"data_stored":1866858496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1249509376,"data_stored":1249509376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1849888768,"data_stored":1849884696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1034747904,"data_stored":1034747904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1597902848,"data_stored":1597898776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":938672128,"data_stored":938672128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:45:28.812 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:28.841 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:28.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:28.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:28.943 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:28.973 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:28.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:28.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:29.075 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:29.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:29.119 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:29.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:29.162 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:45:29.220 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:29.255 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:29.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:29.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:29.357 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:29.388 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:29.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:29.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:29.490 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:29.535 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:29.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:29.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:29.636 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:29.674 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:29.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:29.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:29.775 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:29.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:29.811 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:29.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:29.860 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:29 smithi003 ceph-mon[100615]: pgmap v1288: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 3.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2119/167694 objects degraded (1.264%); 1957/167694 objects misplaced (1.167%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T17:45:29.861 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:29 smithi003 ceph-mon[103344]: pgmap v1288: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 3.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2119/167694 objects degraded (1.264%); 1957/167694 objects misplaced (1.167%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T17:45:29.912 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:29.955 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:29.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:29.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:29.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:29 smithi053 ceph-mon[89963]: pgmap v1288: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 3.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2119/167694 objects degraded (1.264%); 1957/167694 objects misplaced (1.167%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T17:45:29.971 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:45:30.042 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:45:30.042 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:45:30.042 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:30.042 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36612.0:186878 10.f e390/357 10:f3d002bd:::benchmark_data_smithi003_292749_object68332:head v 390'11634, mlcod=390'11631)", 2024-08-17T17:45:30.042 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:45:30.028478+0000", 2024-08-17T17:45:30.042 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.00140105, 2024-08-17T17:45:30.042 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014275430000000001, 2024-08-17T17:45:30.042 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:45:30.042 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:45:30.042 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:45:30.043 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:45:30.043 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:30.043 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:45:30.043 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:30.028478+0000", 2024-08-17T17:45:30.043 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:30.043 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:30.043 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:30.043 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:45:30.043 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:30.028478+0000", 2024-08-17T17:45:30.043 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:30.043 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:30.043 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:30.044 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:45:30.044 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:30.028479+0000", 2024-08-17T17:45:30.044 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.271e-06 2024-08-17T17:45:30.044 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:30.044 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:30.044 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:45:30.044 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:30.028488+0000", 2024-08-17T17:45:30.044 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1349999999999998e-06 2024-08-17T17:45:30.044 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:30.044 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:30.044 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:45:30.044 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:30.028509+0000", 2024-08-17T17:45:30.044 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0316e-05 2024-08-17T17:45:30.045 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:30.045 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:30.045 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:45:30.045 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:30.028517+0000", 2024-08-17T17:45:30.045 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3680000000000002e-06 2024-08-17T17:45:30.045 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:30.045 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:30.045 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:45:30.045 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:30.028912+0000", 2024-08-17T17:45:30.045 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00039464200000000001 2024-08-17T17:45:30.045 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:30.045 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:30.045 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:45:30.046 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:30.029023+0000", 2024-08-17T17:45:30.046 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000111629 2024-08-17T17:45:30.046 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:30.046 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:45:30.046 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:30.046 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:30.046 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:30.046 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36612.0:186876 10.9 e390/357)", 2024-08-17T17:45:30.046 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:45:30.028579+0000", 2024-08-17T17:45:30.046 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.001300084, 2024-08-17T17:45:30.046 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00137685, 2024-08-17T17:45:30.046 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:45:30.046 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:45:30.047 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-08-17T17:45:30.047 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:45:30.047 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:30.047 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:45:30.047 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:30.028579+0000", 2024-08-17T17:45:30.047 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:30.047 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:30.047 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:30.047 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:45:30.047 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:30.028579+0000", 2024-08-17T17:45:30.047 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:30.047 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:30.047 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:30.047 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:45:30.047 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:30.028580+0000", 2024-08-17T17:45:30.047 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4000000000000002e-07 2024-08-17T17:45:30.048 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:30.048 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:30.048 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:45:30.048 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:30.028585+0000", 2024-08-17T17:45:30.048 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.575e-06 2024-08-17T17:45:30.048 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:30.048 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:30.048 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:45:30.048 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:30.028594+0000", 2024-08-17T17:45:30.048 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8470000000000002e-06 2024-08-17T17:45:30.048 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:30.048 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:30.048 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:45:30.048 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:30.028600+0000", 2024-08-17T17:45:30.048 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6269999999999997e-06 2024-08-17T17:45:30.049 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:30.049 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:45:30.049 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:30.049 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:30.049 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:30.049 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36612.0:186877 10.3 e390/357 10:c55310e7:::benchmark_data_smithi003_292749_object68331:head v 390'11845, mlcod=390'11843)", 2024-08-17T17:45:30.049 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:45:30.028633+0000", 2024-08-17T17:45:30.049 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.001245833, 2024-08-17T17:45:30.049 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0013618180000000001, 2024-08-17T17:45:30.049 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:45:30.049 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:45:30.049 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-08-17T17:45:30.049 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:45:30.049 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:30.049 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:45:30.050 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:30.028633+0000", 2024-08-17T17:45:30.050 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:30.050 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:30.050 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:30.050 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:45:30.050 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:30.028633+0000", 2024-08-17T17:45:30.050 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:30.050 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:30.050 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:30.050 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:45:30.050 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:30.028634+0000", 2024-08-17T17:45:30.050 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3700000000000001e-07 2024-08-17T17:45:30.050 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:30.050 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:30.050 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:45:30.050 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:30.028638+0000", 2024-08-17T17:45:30.051 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.002e-06 2024-08-17T17:45:30.051 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:30.051 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:30.051 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:45:30.051 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:30.028645+0000", 2024-08-17T17:45:30.051 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1450000000000002e-06 2024-08-17T17:45:30.051 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:30.051 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:30.051 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:45:30.051 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:30.028649+0000", 2024-08-17T17:45:30.051 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.78e-06 2024-08-17T17:45:30.051 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:30.051 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:30.051 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:45:30.051 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:30.028843+0000", 2024-08-17T17:45:30.051 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019383399999999999 2024-08-17T17:45:30.052 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:30.052 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:30.052 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:45:30.052 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:30.028880+0000", 2024-08-17T17:45:30.052 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6266999999999997e-05 2024-08-17T17:45:30.052 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:30.052 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:45:30.052 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:30.052 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:30.052 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-08-17T17:45:30.052 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 3 2024-08-17T17:45:30.052 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:45:30.058 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:30.093 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:30.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:30.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:30.195 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:30.235 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:30.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:30.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:30.337 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:30.371 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:30.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:30.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:30.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:45:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:45:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:45:30.473 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:30.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:30.515 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:30.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:30.616 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:30.651 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:30.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:30.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:30.753 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:30.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:30 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 2119/167694 objects degraded (1.264%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:30.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:30 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 2119/167694 objects degraded (1.264%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:30.788 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:30.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:30.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:30.889 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:30.931 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:30.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:30.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:30.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:30 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 2119/167694 objects degraded (1.264%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:31.033 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:31.072 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-17T17:45:31.080 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:31.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:31.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:31.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:31.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:31.224 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:31.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:31.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:31.363 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:31.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:31.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:31.465 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:31.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:31.500 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:31.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:31.577 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:45:31.600 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:31.634 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:31 smithi003 ceph-mon[100615]: pgmap v1289: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2282/153432 objects degraded (1.487%); 1645/153432 objects misplaced (1.072%); 1.8 MiB/s, 28 objects/s recovering 2024-08-17T17:45:31.634 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:31 smithi003 ceph-mon[103344]: pgmap v1289: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2282/153432 objects degraded (1.487%); 1645/153432 objects misplaced (1.072%); 1.8 MiB/s, 28 objects/s recovering 2024-08-17T17:45:31.670 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:31.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:31.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:31.772 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:31.805 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:31.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:31.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:31.807 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:45:31.906 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:31.942 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:31.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:31.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:31.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:31 smithi053 ceph-mon[89963]: pgmap v1289: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 3.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2282/153432 objects degraded (1.487%); 1645/153432 objects misplaced (1.072%); 1.8 MiB/s, 28 objects/s recovering 2024-08-17T17:45:32.044 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:32.078 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:32.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:32.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:32.180 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:32.221 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:32.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:32.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:32.323 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:32.356 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:32.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:32.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:32.458 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:32.494 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:32.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:32.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:32.596 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:32.643 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:32.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:32.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:32.745 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:32.781 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:32.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:32.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:32.882 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:32.914 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:32.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:32.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:33.018 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:33.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:33.055 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:33.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:33.156 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:33.194 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:33.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:33.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:33.296 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:33.328 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:33.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:33.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:33.430 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:33.461 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:33.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:33.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:33.562 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:33.620 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:33.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:33.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:33.721 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:33.760 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:33.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:33.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:33.862 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:33.891 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:33.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:33.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:33.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:33 smithi053 ceph-mon[89963]: pgmap v1290: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 3.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 930 op/s; 2282/152496 objects degraded (1.496%); 1645/152496 objects misplaced (1.079%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T17:45:33.992 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:34.014 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:34.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:34.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:34.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:33 smithi003 ceph-mon[103344]: pgmap v1290: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 3.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 930 op/s; 2282/152496 objects degraded (1.496%); 1645/152496 objects misplaced (1.079%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T17:45:34.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:33 smithi003 ceph-mon[100615]: pgmap v1290: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 3.1 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 930 op/s; 2282/152496 objects degraded (1.496%); 1645/152496 objects misplaced (1.079%); 1.6 MiB/s, 26 objects/s recovering 2024-08-17T17:45:34.116 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:34.151 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:34.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:34.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:34.253 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:34.291 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:34.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:34.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:34.393 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:34.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:34.429 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:34.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:34.530 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:34.561 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:34.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:34.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:34.569 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-08-17T17:45:34.664 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:34.701 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:34.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:34.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:34.802 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:34.838 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:34.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:34.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:34.940 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:34.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:34 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3710639043' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:45:34.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:34 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:45:34.977 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:34.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:34.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:34.981 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:34 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3710639043' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:45:34.981 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:34 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:45:34.984 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:34 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3710639043' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:45:34.984 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:34 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:45:35.079 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:35.112 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:35.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:35.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:35.213 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:35.248 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:35.249 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:45:35.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:35.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:35.351 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:35.387 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:35.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:35.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:35.488 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:35.528 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:35.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:35.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:35.585 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:35 smithi003 ceph-mon[103344]: pgmap v1291: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 2411/143100 objects degraded (1.685%); 1350/143100 objects misplaced (0.943%); 2.4 MiB/s, 37 objects/s recovering 2024-08-17T17:45:35.585 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:35 smithi003 ceph-mon[103344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:45:35.586 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:35 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:45:35.586 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:35 smithi003 ceph-mon[103344]: osdmap e391: 8 total, 8 up, 8 in 2024-08-17T17:45:35.586 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:35 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 2411/143100 objects degraded (1.685%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:35.586 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:35 smithi003 ceph-mon[100615]: pgmap v1291: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 2411/143100 objects degraded (1.685%); 1350/143100 objects misplaced (0.943%); 2.4 MiB/s, 37 objects/s recovering 2024-08-17T17:45:35.586 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:35 smithi003 ceph-mon[100615]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:45:35.586 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:35 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:45:35.586 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:35 smithi003 ceph-mon[100615]: osdmap e391: 8 total, 8 up, 8 in 2024-08-17T17:45:35.586 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:35 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 2411/143100 objects degraded (1.685%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:35.630 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:35.670 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:35.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:35.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:35.771 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:35.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:35.811 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:35.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:35.912 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:35.921 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:45:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:45:35] "GET /metrics HTTP/1.1" 200 36581 "" "Prometheus/2.51.0" 2024-08-17T17:45:35.947 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:35.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:35.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:35.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:35 smithi053 ceph-mon[89963]: pgmap v1291: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.11k op/s; 2411/143100 objects degraded (1.685%); 1350/143100 objects misplaced (0.943%); 2.4 MiB/s, 37 objects/s recovering 2024-08-17T17:45:35.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:35 smithi053 ceph-mon[89963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:45:35.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:35 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:45:35.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:35 smithi053 ceph-mon[89963]: osdmap e391: 8 total, 8 up, 8 in 2024-08-17T17:45:35.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:35 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 2411/143100 objects degraded (1.685%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:36.050 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:36.081 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:45:36.081 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:45:36.085 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:36.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:36.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:36.111 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:45:36.111 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:45:36.142 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:45:36.142 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:45:36.173 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:45:36.174 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:45:36.186 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:36.212 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:45:36.212 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:45:36.217 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:36.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:36.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:36.245 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:45:36.245 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:45:36.283 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:45:36.284 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:45:36.315 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:45:36.315 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:45:36.318 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:36.358 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:36.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:36.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:36.460 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:36.499 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:36.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:36.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:36.600 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:36.638 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:36.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:36.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:36.740 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:36.772 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:36.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:36.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:36.874 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:36.910 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:36.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:36.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:36.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:36 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2993698760' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:45:37.012 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:37.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:36 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2993698760' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:45:37.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:36 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2993698760' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:45:37.054 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:37.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:37.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:37.156 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:37.185 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:37.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:37.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:37.286 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:37.315 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:37.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:37.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:37.417 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:37.459 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:37.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:37.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:37.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:37.601 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:37.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:37.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:37.703 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:37.742 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:37.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:37.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:37.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:37 smithi003 ceph-mon[103344]: pgmap v1293: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 2411/137271 objects degraded (1.756%); 1350/137271 objects misplaced (0.983%); 2.0 MiB/s, 31 objects/s recovering 2024-08-17T17:45:37.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:37 smithi003 ceph-mon[100615]: pgmap v1293: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 2411/137271 objects degraded (1.756%); 1350/137271 objects misplaced (0.983%); 2.0 MiB/s, 31 objects/s recovering 2024-08-17T17:45:37.844 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:37.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:37.888 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:37.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:37.889 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:45:37.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:37 smithi053 ceph-mon[89963]: pgmap v1293: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 2411/137271 objects degraded (1.756%); 1350/137271 objects misplaced (0.983%); 2.0 MiB/s, 31 objects/s recovering 2024-08-17T17:45:37.989 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:38.026 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:38.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:38.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:38.128 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:38.168 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:38.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:38.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:38.271 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:38.304 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:38.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:38.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:38.405 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:38.432 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:38.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:38.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:38.534 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:38.574 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:38.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:38.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:38.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:45:38.676 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:38.713 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:38.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:38.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:38.814 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:38.855 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:38.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:38.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:38.957 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:38.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:45:38.997 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:38.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:38.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:39.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:45:39.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:45:39.098 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:39.135 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:39.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:39.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:39.237 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:39.281 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:39.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:39.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:39.383 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:39.422 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:39.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:39.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:39.486 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:45:39.523 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:39.564 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:39.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:39.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:39.666 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:39.704 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:39.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:39.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:39.746 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:39 smithi003 ceph-mon[103344]: pgmap v1294: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 2411/137271 objects degraded (1.756%); 1350/137271 objects misplaced (0.983%); 2.0 MiB/s, 31 objects/s recovering 2024-08-17T17:45:39.747 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:39 smithi003 ceph-mon[100615]: pgmap v1294: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 2411/137271 objects degraded (1.756%); 1350/137271 objects misplaced (0.983%); 2.0 MiB/s, 31 objects/s recovering 2024-08-17T17:45:39.806 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:39.848 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:39.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:39.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:39.867 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:39 smithi053 ceph-mon[89963]: pgmap v1294: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 2411/137271 objects degraded (1.756%); 1350/137271 objects misplaced (0.983%); 2.0 MiB/s, 31 objects/s recovering 2024-08-17T17:45:39.951 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:39.987 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:39.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:39.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:40.089 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:40.122 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:40.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:40.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:40.223 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:40.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:40.259 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:40.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:40.360 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:40.396 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:40.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:40.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:40.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:45:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:45:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:45:40.499 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:40.533 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:40.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:40.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:40.635 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:40.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:40.678 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:40.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:40.779 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:40.811 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:40.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:40.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:40.913 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:40.964 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:40.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:40.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:40.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:40 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 2411/137271 objects degraded (1.756%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:41.020 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:40 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 2411/137271 objects degraded (1.756%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:41.020 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:40 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 2411/137271 objects degraded (1.756%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:41.066 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:41.104 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:41.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:41.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:41.206 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:41.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:41.245 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:41.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:41.346 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:41.379 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:41.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:41.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:41.481 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:41.518 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:41.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:41.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:41.537 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:45:41.537 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:45:41.620 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:41.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:41.662 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:41.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:41.763 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:41.772 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:45:41.804 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:41.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:41.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:41.902 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:41 smithi003 ceph-mon[103344]: pgmap v1295: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2538/122757 objects degraded (2.067%); 1067/122757 objects misplaced (0.869%); 2.0 MiB/s, 32 objects/s recovering 2024-08-17T17:45:41.902 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:41 smithi003 ceph-mon[100615]: pgmap v1295: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2538/122757 objects degraded (2.067%); 1067/122757 objects misplaced (0.869%); 2.0 MiB/s, 32 objects/s recovering 2024-08-17T17:45:41.906 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:41.937 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:41.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:41.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:41.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:41 smithi053 ceph-mon[89963]: pgmap v1295: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2538/122757 objects degraded (2.067%); 1067/122757 objects misplaced (0.869%); 2.0 MiB/s, 32 objects/s recovering 2024-08-17T17:45:42.040 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:42.082 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:42.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:42.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:42.184 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:42.217 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:42.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:42.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:42.318 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:42.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:42.350 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:42.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:42.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:42.473 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:42.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:42.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:42.574 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:42.615 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:42.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:42.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:42.717 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:42.751 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:42.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:42.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:42.853 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:42.898 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:42.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:42.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:43.000 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:43.021 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-17T17:45:43.038 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:43.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:43.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:43.064 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:45:43.140 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:43.174 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:43.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:43.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:43.275 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:43.307 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:43.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:43.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:43.409 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:43.443 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:43.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:43.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:43.545 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:43.583 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:43.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:43.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:43.661 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:43 smithi003 ceph-mon[103344]: pgmap v1296: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2538/121914 objects degraded (2.082%); 1067/121914 objects misplaced (0.875%); 2.0 MiB/s, 32 objects/s recovering 2024-08-17T17:45:43.662 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:43 smithi003 ceph-mon[100615]: pgmap v1296: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2538/121914 objects degraded (2.082%); 1067/121914 objects misplaced (0.875%); 2.0 MiB/s, 32 objects/s recovering 2024-08-17T17:45:43.685 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:43.719 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:43.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:43.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:43.821 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:43.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:43.859 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:43.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:43.960 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:43.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:43 smithi053 ceph-mon[89963]: pgmap v1296: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 2538/121914 objects degraded (2.082%); 1067/121914 objects misplaced (0.875%); 2.0 MiB/s, 32 objects/s recovering 2024-08-17T17:45:43.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:43.998 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:43.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:44.099 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:44.134 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:44.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:44.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:44.207 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:45:44.209 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:45:44.236 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:44.277 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:44.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:44.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:44.379 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:44.414 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:44.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:44.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:44.516 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:44.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:44.557 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:44.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:44.657 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:44.693 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:44.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:44.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:44.794 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:44.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:44.831 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:44.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:44.932 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:44.966 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:44.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:44.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:45.067 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:45.099 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:45.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:45.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:45.201 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:45.232 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:45.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:45.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:45.334 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:45.368 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:45.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:45.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:45.470 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:45.505 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:45.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:45.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:45.607 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:45.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:45.650 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:45.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:45.751 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:45.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:45 smithi003 ceph-mon[100615]: from='client.36819 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:45:45.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:45 smithi003 ceph-mon[100615]: pgmap v1297: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 2666/113046 objects degraded (2.358%); 734/113046 objects misplaced (0.649%); 2.3 MiB/s, 36 objects/s recovering 2024-08-17T17:45:45.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:45 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 2666/113046 objects degraded (2.358%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:45.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:45:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:45:45] "GET /metrics HTTP/1.1" 200 36579 "" "Prometheus/2.51.0" 2024-08-17T17:45:45.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:45 smithi003 ceph-mon[103344]: from='client.36819 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:45:45.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:45 smithi003 ceph-mon[103344]: pgmap v1297: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 2666/113046 objects degraded (2.358%); 734/113046 objects misplaced (0.649%); 2.3 MiB/s, 36 objects/s recovering 2024-08-17T17:45:45.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:45 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 2666/113046 objects degraded (2.358%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:45.792 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:45.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:45.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:45.894 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:45.928 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:45.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:45.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:45.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:45 smithi053 ceph-mon[89963]: from='client.36819 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:45:45.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:45 smithi053 ceph-mon[89963]: pgmap v1297: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 1.00k op/s; 2666/113046 objects degraded (2.358%); 734/113046 objects misplaced (0.649%); 2.3 MiB/s, 36 objects/s recovering 2024-08-17T17:45:45.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:45 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 2666/113046 objects degraded (2.358%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:46.030 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:46.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:46.068 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:46.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:46.141 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1296,"stamp":"2024-08-17T17:45:42.421045+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2666141669,"num_objects":40638,"num_object_clones":0,"num_object_copies":121914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2538,"num_objects_misplaced":1067,"num_objects_unfound":0,"num_objects_dirty":40638,"num_whiteouts":0,"num_read":5979,"num_read_kb":12992,"num_write":197385,"num_write_kb":7599376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3389,"num_bytes_recovered":217787384,"num_keys_recovered":40,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107429,"ondisk_log_size":107429,"up":357,"acting":356,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":362,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":14440724,"kb_used_data":8333012,"kb_used_omap":37,"kb_used_meta":6107546,"kb_avail":735520492,"statfs":{"total":767960285184,"available":753172983808,"internally_reserved":0,"allocated":8533004288,"data_stored":8515245119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38729,"internal_metadata":6254127287},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,7,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-668073984,"num_objects":-10194,"num_object_clones":0,"num_object_copies":-30582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":256,"num_objects_misplaced":-578,"num_objects_unfound":0,"num_objects_dirty":-10194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10194,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":322,"num_bytes_recovered":21102592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003053"},"pg_stats":[{"pgid":"1.11","version":"90'149","reported_seq":834,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.572470+0000","last_change":"2024-08-17T17:38:30.378664+0000","last_active":"2024-08-17T17:45:34.572470+0000","last_peered":"2024-08-17T17:45:34.572470+0000","last_clean":"2024-08-17T17:45:34.572470+0000","last_became_active":"2024-08-17T17:38:30.342287+0000","last_became_peered":"2024-08-17T17:38:30.342287+0000","last_unstale":"2024-08-17T17:45:34.572470+0000","last_undegraded":"2024-08-17T17:45:34.572470+0000","last_fullsized":"2024-08-17T17:45:34.572470+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:30.378592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:30.378592+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:04:28.933354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":655,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.565326+0000","last_change":"2024-08-17T17:33:20.797029+0000","last_active":"2024-08-17T17:45:34.565326+0000","last_peered":"2024-08-17T17:45:34.565326+0000","last_clean":"2024-08-17T17:45:34.565326+0000","last_became_active":"2024-08-17T17:33:20.796679+0000","last_became_peered":"2024-08-17T17:33:20.796679+0000","last_unstale":"2024-08-17T17:45:34.565326+0000","last_undegraded":"2024-08-17T17:45:34.565326+0000","last_fullsized":"2024-08-17T17:45:34.565326+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:51.438262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":415,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.490571+0000","last_change":"2024-08-17T17:41:34.624198+0000","last_active":"2024-08-17T17:45:21.490571+0000","last_peered":"2024-08-17T17:45:21.490571+0000","last_clean":"2024-08-17T17:45:21.490571+0000","last_became_active":"2024-08-17T17:41:34.616071+0000","last_became_peered":"2024-08-17T17:41:34.616071+0000","last_unstale":"2024-08-17T17:45:21.490571+0000","last_undegraded":"2024-08-17T17:45:21.490571+0000","last_fullsized":"2024-08-17T17:45:21.490571+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:34:48.805354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"90'149","reported_seq":474,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.497521+0000","last_change":"2024-08-17T17:38:31.874623+0000","last_active":"2024-08-17T17:45:21.497521+0000","last_peered":"2024-08-17T17:45:21.497521+0000","last_clean":"2024-08-17T17:45:21.497521+0000","last_became_active":"2024-08-17T17:38:30.339313+0000","last_became_peered":"2024-08-17T17:38:30.339313+0000","last_unstale":"2024-08-17T17:45:21.497521+0000","last_undegraded":"2024-08-17T17:45:21.497521+0000","last_fullsized":"2024-08-17T17:45:21.497521+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:31.874423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:31.874423+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:28:17.980339+0000","scrub_duration":0.0017696579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":315,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.568594+0000","last_change":"2024-08-17T17:33:44.156036+0000","last_active":"2024-08-17T17:45:34.568594+0000","last_peered":"2024-08-17T17:45:34.568594+0000","last_clean":"2024-08-17T17:45:34.568594+0000","last_became_active":"2024-08-17T17:33:44.155758+0000","last_became_peered":"2024-08-17T17:33:44.155758+0000","last_unstale":"2024-08-17T17:45:34.568594+0000","last_undegraded":"2024-08-17T17:45:34.568594+0000","last_fullsized":"2024-08-17T17:45:34.568594+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:23.443405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":455,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.581952+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:45:34.581952+0000","last_peered":"2024-08-17T17:45:34.581952+0000","last_clean":"2024-08-17T17:45:34.581952+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:45:34.581952+0000","last_undegraded":"2024-08-17T17:45:34.581952+0000","last_fullsized":"2024-08-17T17:45:34.581952+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"90'149","reported_seq":852,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.573321+0000","last_change":"2024-08-17T17:41:37.681187+0000","last_active":"2024-08-17T17:45:34.573321+0000","last_peered":"2024-08-17T17:45:34.573321+0000","last_clean":"2024-08-17T17:45:34.573321+0000","last_became_active":"2024-08-17T17:41:37.680605+0000","last_became_peered":"2024-08-17T17:41:37.680605+0000","last_unstale":"2024-08-17T17:45:34.573321+0000","last_undegraded":"2024-08-17T17:45:34.573321+0000","last_fullsized":"2024-08-17T17:45:34.573321+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.844252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.844252+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:13:31.682357+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":454,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.578402+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:45:34.578402+0000","last_peered":"2024-08-17T17:45:34.578402+0000","last_clean":"2024-08-17T17:45:34.578402+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:45:34.578402+0000","last_undegraded":"2024-08-17T17:45:34.578402+0000","last_fullsized":"2024-08-17T17:45:34.578402+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":55,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.499816+0000","last_change":"2024-08-17T17:41:34.611693+0000","last_active":"2024-08-17T17:45:21.499816+0000","last_peered":"2024-08-17T17:45:21.499816+0000","last_clean":"2024-08-17T17:45:21.499816+0000","last_became_active":"2024-08-17T17:41:34.611374+0000","last_became_peered":"2024-08-17T17:41:34.611374+0000","last_unstale":"2024-08-17T17:45:21.499816+0000","last_undegraded":"2024-08-17T17:45:21.499816+0000","last_fullsized":"2024-08-17T17:45:21.499816+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:17:09.897446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"90'149","reported_seq":828,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.569858+0000","last_change":"2024-08-17T17:38:30.446165+0000","last_active":"2024-08-17T17:45:34.569858+0000","last_peered":"2024-08-17T17:45:34.569858+0000","last_clean":"2024-08-17T17:45:34.569858+0000","last_became_active":"2024-08-17T17:38:30.445048+0000","last_became_peered":"2024-08-17T17:38:30.445048+0000","last_unstale":"2024-08-17T17:45:34.569858+0000","last_undegraded":"2024-08-17T17:45:34.569858+0000","last_fullsized":"2024-08-17T17:45:34.569858+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:24.291592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:24.291592+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:17.100154+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":668,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.568089+0000","last_change":"2024-08-17T17:33:20.794682+0000","last_active":"2024-08-17T17:45:34.568089+0000","last_peered":"2024-08-17T17:45:34.568089+0000","last_clean":"2024-08-17T17:45:34.568089+0000","last_became_active":"2024-08-17T17:33:20.794238+0000","last_became_peered":"2024-08-17T17:33:20.794238+0000","last_unstale":"2024-08-17T17:45:34.568089+0000","last_undegraded":"2024-08-17T17:45:34.568089+0000","last_fullsized":"2024-08-17T17:45:34.568089+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:15:55.821920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":476,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.497926+0000","last_change":"2024-08-17T17:43:52.520659+0000","last_active":"2024-08-17T17:45:21.497926+0000","last_peered":"2024-08-17T17:45:21.497926+0000","last_clean":"2024-08-17T17:45:21.497926+0000","last_became_active":"2024-08-17T17:43:52.516799+0000","last_became_peered":"2024-08-17T17:43:52.516799+0000","last_unstale":"2024-08-17T17:45:21.497926+0000","last_undegraded":"2024-08-17T17:45:21.497926+0000","last_fullsized":"2024-08-17T17:45:21.497926+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:25:59.408383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"391'12352","reported_seq":12432,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:40.271861+0000","last_change":"2024-08-17T17:42:36.946078+0000","last_active":"2024-08-17T17:45:40.271861+0000","last_peered":"2024-08-17T17:45:40.271861+0000","last_clean":"2024-08-17T17:45:40.271861+0000","last_became_active":"2024-08-17T17:42:36.945602+0000","last_became_peered":"2024-08-17T17:42:36.945602+0000","last_unstale":"2024-08-17T17:45:40.271861+0000","last_undegraded":"2024-08-17T17:45:40.271861+0000","last_fullsized":"2024-08-17T17:45:40.271861+0000","mapping_epoch":357,"log_start":"376'5800","ondisk_log_start":"376'5800","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6552,"log_dups_size":0,"ondisk_log_size":6552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:18:43.716968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162529280,"num_objects":2480,"num_object_clones":0,"num_object_copies":7440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12352,"num_write_kb":474624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"90'149","reported_seq":801,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.490652+0000","last_change":"2024-08-17T17:42:16.610940+0000","last_active":"2024-08-17T17:45:21.490652+0000","last_peered":"2024-08-17T17:45:21.490652+0000","last_clean":"2024-08-17T17:45:21.490652+0000","last_became_active":"2024-08-17T17:41:37.670855+0000","last_became_peered":"2024-08-17T17:41:37.670855+0000","last_unstale":"2024-08-17T17:45:21.490652+0000","last_undegraded":"2024-08-17T17:45:21.490652+0000","last_fullsized":"2024-08-17T17:45:21.490652+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:16.610828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:16.610828+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:48:25.918610+0000","scrub_duration":0.086892487000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":352,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.491391+0000","last_change":"2024-08-17T17:41:34.635980+0000","last_active":"2024-08-17T17:45:21.491391+0000","last_peered":"2024-08-17T17:45:21.491391+0000","last_clean":"2024-08-17T17:45:21.491391+0000","last_became_active":"2024-08-17T17:41:34.620596+0000","last_became_peered":"2024-08-17T17:41:34.620596+0000","last_unstale":"2024-08-17T17:45:21.491391+0000","last_undegraded":"2024-08-17T17:45:21.491391+0000","last_fullsized":"2024-08-17T17:45:21.491391+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:19:15.792577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":145,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.497659+0000","last_change":"2024-08-17T17:33:44.145598+0000","last_active":"2024-08-17T17:45:21.497659+0000","last_peered":"2024-08-17T17:45:21.497659+0000","last_clean":"2024-08-17T17:45:21.497659+0000","last_became_active":"2024-08-17T17:33:44.145283+0000","last_became_peered":"2024-08-17T17:33:44.145283+0000","last_unstale":"2024-08-17T17:45:21.497659+0000","last_undegraded":"2024-08-17T17:45:21.497659+0000","last_fullsized":"2024-08-17T17:45:21.497659+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:11.965226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.7","version":"391'12244","reported_seq":18708,"reported_epoch":391,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-17T17:45:38.931933+0000","last_change":"2024-08-17T17:44:30.783759+0000","last_active":"2024-08-17T17:45:38.931933+0000","last_peered":"2024-08-17T17:45:38.931933+0000","last_clean":"2024-08-17T17:43:51.465744+0000","last_became_active":"2024-08-17T17:43:53.682178+0000","last_became_peered":"2024-08-17T17:43:53.682178+0000","last_unstale":"2024-08-17T17:45:38.931933+0000","last_undegraded":"2024-08-17T17:44:30.779387+0000","last_fullsized":"2024-08-17T17:43:53.513318+0000","mapping_epoch":378,"log_start":"376'5800","ondisk_log_start":"376'5800","created":357,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6444,"log_dups_size":0,"ondisk_log_size":6444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:42:59.608872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170000384,"num_objects":2594,"num_object_clones":0,"num_object_copies":7782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2538,"num_objects_misplaced":1067,"num_objects_unfound":0,"num_objects_dirty":2594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12244,"num_write_kb":474816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3195,"num_bytes_recovered":209387520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3],"avail_no_missing":["6","2","3"],"object_location_counts":[{"shards":"2,3,6","objects":1067},{"shards":"3,6","objects":-1011}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"90'149","reported_seq":853,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.565138+0000","last_change":"2024-08-17T17:41:42.603841+0000","last_active":"2024-08-17T17:45:34.565138+0000","last_peered":"2024-08-17T17:45:34.565138+0000","last_clean":"2024-08-17T17:45:34.565138+0000","last_became_active":"2024-08-17T17:41:12.203400+0000","last_became_peered":"2024-08-17T17:41:12.203400+0000","last_unstale":"2024-08-17T17:45:34.565138+0000","last_undegraded":"2024-08-17T17:45:34.565138+0000","last_fullsized":"2024-08-17T17:45:34.565138+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:42.603561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:42.603561+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:50:12.843193+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":348,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.491377+0000","last_change":"2024-08-17T17:41:34.637696+0000","last_active":"2024-08-17T17:45:21.491377+0000","last_peered":"2024-08-17T17:45:21.491377+0000","last_clean":"2024-08-17T17:45:21.491377+0000","last_became_active":"2024-08-17T17:41:34.621388+0000","last_became_peered":"2024-08-17T17:41:34.621388+0000","last_unstale":"2024-08-17T17:45:21.491377+0000","last_undegraded":"2024-08-17T17:45:21.491377+0000","last_fullsized":"2024-08-17T17:45:21.491377+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:13:04.607036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":55,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.499784+0000","last_change":"2024-08-17T17:41:34.621078+0000","last_active":"2024-08-17T17:45:21.499784+0000","last_peered":"2024-08-17T17:45:21.499784+0000","last_clean":"2024-08-17T17:45:21.499784+0000","last_became_active":"2024-08-17T17:41:34.620730+0000","last_became_peered":"2024-08-17T17:41:34.620730+0000","last_unstale":"2024-08-17T17:45:21.499784+0000","last_undegraded":"2024-08-17T17:45:21.499784+0000","last_fullsized":"2024-08-17T17:45:21.499784+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:55:01.312283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"391'12425","reported_seq":12505,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:40.299526+0000","last_change":"2024-08-17T17:42:36.620130+0000","last_active":"2024-08-17T17:45:40.299526+0000","last_peered":"2024-08-17T17:45:40.299526+0000","last_clean":"2024-08-17T17:45:40.299526+0000","last_became_active":"2024-08-17T17:42:36.619813+0000","last_became_peered":"2024-08-17T17:42:36.619813+0000","last_unstale":"2024-08-17T17:45:40.299526+0000","last_undegraded":"2024-08-17T17:45:40.299526+0000","last_fullsized":"2024-08-17T17:45:40.299526+0000","mapping_epoch":357,"log_start":"376'5900","ondisk_log_start":"376'5900","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6525,"log_dups_size":0,"ondisk_log_size":6525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:39:22.366905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164167680,"num_objects":2505,"num_object_clones":0,"num_object_copies":7515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12425,"num_write_kb":477760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"90'149","reported_seq":567,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.502265+0000","last_change":"2024-08-17T17:41:37.668050+0000","last_active":"2024-08-17T17:45:21.502265+0000","last_peered":"2024-08-17T17:45:21.502265+0000","last_clean":"2024-08-17T17:45:21.502265+0000","last_became_active":"2024-08-17T17:41:37.667655+0000","last_became_peered":"2024-08-17T17:41:37.667655+0000","last_unstale":"2024-08-17T17:45:21.502265+0000","last_undegraded":"2024-08-17T17:45:21.502265+0000","last_fullsized":"2024-08-17T17:45:21.502265+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:32.843802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:32.843802+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:39:46.766280+0000","scrub_duration":0.0040803669999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":141,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.497295+0000","last_change":"2024-08-17T17:33:44.145399+0000","last_active":"2024-08-17T17:45:21.497295+0000","last_peered":"2024-08-17T17:45:21.497295+0000","last_clean":"2024-08-17T17:45:21.497295+0000","last_became_active":"2024-08-17T17:33:44.144751+0000","last_became_peered":"2024-08-17T17:33:44.144751+0000","last_unstale":"2024-08-17T17:45:21.497295+0000","last_undegraded":"2024-08-17T17:45:21.497295+0000","last_fullsized":"2024-08-17T17:45:21.497295+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:54.276350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":219,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.496924+0000","last_change":"2024-08-17T17:33:20.795582+0000","last_active":"2024-08-17T17:45:21.496924+0000","last_peered":"2024-08-17T17:45:21.496924+0000","last_clean":"2024-08-17T17:45:21.496924+0000","last_became_active":"2024-08-17T17:33:20.795149+0000","last_became_peered":"2024-08-17T17:33:20.795149+0000","last_unstale":"2024-08-17T17:45:21.496924+0000","last_undegraded":"2024-08-17T17:45:21.496924+0000","last_fullsized":"2024-08-17T17:45:21.496924+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:14.020923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"391'12307","reported_seq":12387,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:40.310857+0000","last_change":"2024-08-17T17:42:36.929599+0000","last_active":"2024-08-17T17:45:40.310857+0000","last_peered":"2024-08-17T17:45:40.310857+0000","last_clean":"2024-08-17T17:45:40.310857+0000","last_became_active":"2024-08-17T17:42:36.929274+0000","last_became_peered":"2024-08-17T17:42:36.929274+0000","last_unstale":"2024-08-17T17:45:40.310857+0000","last_undegraded":"2024-08-17T17:45:40.310857+0000","last_fullsized":"2024-08-17T17:45:40.310857+0000","mapping_epoch":357,"log_start":"376'5700","ondisk_log_start":"376'5700","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6607,"log_dups_size":0,"ondisk_log_size":6607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:29:35.955743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167051264,"num_objects":2549,"num_object_clones":0,"num_object_copies":7647,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12307,"num_write_kb":475392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"90'149","reported_seq":699,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.501622+0000","last_change":"2024-08-17T17:42:20.927007+0000","last_active":"2024-08-17T17:45:21.501622+0000","last_peered":"2024-08-17T17:45:21.501622+0000","last_clean":"2024-08-17T17:45:21.501622+0000","last_became_active":"2024-08-17T17:41:37.670914+0000","last_became_peered":"2024-08-17T17:41:37.670914+0000","last_unstale":"2024-08-17T17:45:21.501622+0000","last_undegraded":"2024-08-17T17:45:21.501622+0000","last_fullsized":"2024-08-17T17:45:21.501622+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:20.926924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:20.926924+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:57:44.156600+0000","scrub_duration":0.0018485109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":313,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.565427+0000","last_change":"2024-08-17T17:33:44.154097+0000","last_active":"2024-08-17T17:45:34.565427+0000","last_peered":"2024-08-17T17:45:34.565427+0000","last_clean":"2024-08-17T17:45:34.565427+0000","last_became_active":"2024-08-17T17:33:44.152972+0000","last_became_peered":"2024-08-17T17:33:44.152972+0000","last_unstale":"2024-08-17T17:45:34.565427+0000","last_undegraded":"2024-08-17T17:45:34.565427+0000","last_fullsized":"2024-08-17T17:45:34.565427+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:42.602017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":677,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.574633+0000","last_change":"2024-08-17T17:41:34.615197+0000","last_active":"2024-08-17T17:45:34.574633+0000","last_peered":"2024-08-17T17:45:34.574633+0000","last_clean":"2024-08-17T17:45:34.574633+0000","last_became_active":"2024-08-17T17:41:34.614873+0000","last_became_peered":"2024-08-17T17:41:34.614873+0000","last_unstale":"2024-08-17T17:45:34.574633+0000","last_undegraded":"2024-08-17T17:45:34.574633+0000","last_fullsized":"2024-08-17T17:45:34.574633+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:21:57.391953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2","version":"391'12139","reported_seq":12218,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:40.991040+0000","last_change":"2024-08-17T17:42:36.618260+0000","last_active":"2024-08-17T17:45:40.991040+0000","last_peered":"2024-08-17T17:45:40.991040+0000","last_clean":"2024-08-17T17:45:40.991040+0000","last_became_active":"2024-08-17T17:42:36.617875+0000","last_became_peered":"2024-08-17T17:42:36.617875+0000","last_unstale":"2024-08-17T17:45:40.991040+0000","last_undegraded":"2024-08-17T17:45:40.991040+0000","last_fullsized":"2024-08-17T17:45:40.991040+0000","mapping_epoch":357,"log_start":"374'5100","ondisk_log_start":"374'5100","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":7039,"log_dups_size":0,"ondisk_log_size":7039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:09:07.102359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154337280,"num_objects":2355,"num_object_clones":0,"num_object_copies":7065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12139,"num_write_kb":463808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":827,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.565043+0000","last_change":"2024-08-17T17:41:44.585386+0000","last_active":"2024-08-17T17:45:34.565043+0000","last_peered":"2024-08-17T17:45:34.565043+0000","last_clean":"2024-08-17T17:45:34.565043+0000","last_became_active":"2024-08-17T17:41:12.307398+0000","last_became_peered":"2024-08-17T17:41:12.307398+0000","last_unstale":"2024-08-17T17:45:34.565043+0000","last_undegraded":"2024-08-17T17:45:34.565043+0000","last_fullsized":"2024-08-17T17:45:34.565043+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:44.583653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:44.583653+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:27:44.756681+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":712,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.567749+0000","last_change":"2024-08-17T17:41:34.630434+0000","last_active":"2024-08-17T17:45:34.567749+0000","last_peered":"2024-08-17T17:45:34.567749+0000","last_clean":"2024-08-17T17:45:34.567749+0000","last_became_active":"2024-08-17T17:41:34.616406+0000","last_became_peered":"2024-08-17T17:41:34.616406+0000","last_unstale":"2024-08-17T17:45:34.567749+0000","last_undegraded":"2024-08-17T17:45:34.567749+0000","last_fullsized":"2024-08-17T17:45:34.567749+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:30:44.817203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":55,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.500000+0000","last_change":"2024-08-17T17:41:34.611049+0000","last_active":"2024-08-17T17:45:21.500000+0000","last_peered":"2024-08-17T17:45:21.500000+0000","last_clean":"2024-08-17T17:45:21.500000+0000","last_became_active":"2024-08-17T17:41:34.610727+0000","last_became_peered":"2024-08-17T17:41:34.610727+0000","last_unstale":"2024-08-17T17:45:21.500000+0000","last_undegraded":"2024-08-17T17:45:21.500000+0000","last_fullsized":"2024-08-17T17:45:21.500000+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:04:05.477660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"391'12473","reported_seq":12553,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:39.635944+0000","last_change":"2024-08-17T17:42:36.946226+0000","last_active":"2024-08-17T17:45:39.635944+0000","last_peered":"2024-08-17T17:45:39.635944+0000","last_clean":"2024-08-17T17:45:39.635944+0000","last_became_active":"2024-08-17T17:42:36.945809+0000","last_became_peered":"2024-08-17T17:42:36.945809+0000","last_unstale":"2024-08-17T17:45:39.635944+0000","last_undegraded":"2024-08-17T17:45:39.635944+0000","last_fullsized":"2024-08-17T17:45:39.635944+0000","mapping_epoch":357,"log_start":"381'6200","ondisk_log_start":"381'6200","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6273,"log_dups_size":0,"ondisk_log_size":6273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:04:58.633957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167444480,"num_objects":2555,"num_object_clones":0,"num_object_copies":7665,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12473,"num_write_kb":480896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":855,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.565933+0000","last_change":"2024-08-17T17:43:55.595647+0000","last_active":"2024-08-17T17:45:34.565933+0000","last_peered":"2024-08-17T17:45:34.565933+0000","last_clean":"2024-08-17T17:45:34.565933+0000","last_became_active":"2024-08-17T17:43:55.595256+0000","last_became_peered":"2024-08-17T17:43:55.595256+0000","last_unstale":"2024-08-17T17:45:34.565933+0000","last_undegraded":"2024-08-17T17:45:34.565933+0000","last_fullsized":"2024-08-17T17:45:34.565933+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:46.587712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:46.587712+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:05:01.927968+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":55,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.500053+0000","last_change":"2024-08-17T17:41:34.622803+0000","last_active":"2024-08-17T17:45:21.500053+0000","last_peered":"2024-08-17T17:45:21.500053+0000","last_clean":"2024-08-17T17:45:21.500053+0000","last_became_active":"2024-08-17T17:41:34.622525+0000","last_became_peered":"2024-08-17T17:41:34.622525+0000","last_unstale":"2024-08-17T17:45:21.500053+0000","last_undegraded":"2024-08-17T17:45:21.500053+0000","last_fullsized":"2024-08-17T17:45:21.500053+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:42:39.728377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":55,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.502302+0000","last_change":"2024-08-17T17:41:34.619268+0000","last_active":"2024-08-17T17:45:21.502302+0000","last_peered":"2024-08-17T17:45:21.502302+0000","last_clean":"2024-08-17T17:45:21.502302+0000","last_became_active":"2024-08-17T17:41:34.618914+0000","last_became_peered":"2024-08-17T17:41:34.618914+0000","last_unstale":"2024-08-17T17:45:21.502302+0000","last_undegraded":"2024-08-17T17:45:21.502302+0000","last_fullsized":"2024-08-17T17:45:21.502302+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:08:54.248386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.0","version":"391'11871","reported_seq":11913,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:38.893685+0000","last_change":"2024-08-17T17:42:36.617537+0000","last_active":"2024-08-17T17:45:38.893685+0000","last_peered":"2024-08-17T17:45:38.893685+0000","last_clean":"2024-08-17T17:45:38.893685+0000","last_became_active":"2024-08-17T17:42:36.617201+0000","last_became_peered":"2024-08-17T17:42:36.617201+0000","last_unstale":"2024-08-17T17:45:38.893685+0000","last_undegraded":"2024-08-17T17:45:38.893685+0000","last_fullsized":"2024-08-17T17:45:38.893685+0000","mapping_epoch":357,"log_start":"376'5400","ondisk_log_start":"376'5400","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6471,"log_dups_size":0,"ondisk_log_size":6471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:51:58.927214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163512320,"num_objects":2495,"num_object_clones":0,"num_object_copies":7485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11871,"num_write_kb":459712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"90'149","reported_seq":861,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.573506+0000","last_change":"2024-08-17T17:41:41.743931+0000","last_active":"2024-08-17T17:45:34.573506+0000","last_peered":"2024-08-17T17:45:34.573506+0000","last_clean":"2024-08-17T17:45:34.573506+0000","last_became_active":"2024-08-17T17:41:12.203771+0000","last_became_peered":"2024-08-17T17:41:12.203771+0000","last_unstale":"2024-08-17T17:45:34.573506+0000","last_undegraded":"2024-08-17T17:45:34.573506+0000","last_fullsized":"2024-08-17T17:45:34.573506+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:41.743570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:41.743570+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:21:03.865738+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":55,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.504061+0000","last_change":"2024-08-17T17:41:34.607602+0000","last_active":"2024-08-17T17:45:21.504061+0000","last_peered":"2024-08-17T17:45:21.504061+0000","last_clean":"2024-08-17T17:45:21.504061+0000","last_became_active":"2024-08-17T17:41:34.607300+0000","last_became_peered":"2024-08-17T17:41:34.607300+0000","last_unstale":"2024-08-17T17:45:21.504061+0000","last_undegraded":"2024-08-17T17:45:21.504061+0000","last_fullsized":"2024-08-17T17:45:21.504061+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:20:31.486698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":55,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.501483+0000","last_change":"2024-08-17T17:41:34.622102+0000","last_active":"2024-08-17T17:45:21.501483+0000","last_peered":"2024-08-17T17:45:21.501483+0000","last_clean":"2024-08-17T17:45:21.501483+0000","last_became_active":"2024-08-17T17:41:34.621733+0000","last_became_peered":"2024-08-17T17:41:34.621733+0000","last_unstale":"2024-08-17T17:45:21.501483+0000","last_undegraded":"2024-08-17T17:45:21.501483+0000","last_fullsized":"2024-08-17T17:45:21.501483+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:39:41.668709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"391'12471","reported_seq":12511,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:38.780140+0000","last_change":"2024-08-17T17:42:36.620155+0000","last_active":"2024-08-17T17:45:38.780140+0000","last_peered":"2024-08-17T17:45:38.780140+0000","last_clean":"2024-08-17T17:45:38.780140+0000","last_became_active":"2024-08-17T17:42:36.619813+0000","last_became_peered":"2024-08-17T17:42:36.619813+0000","last_unstale":"2024-08-17T17:45:38.780140+0000","last_undegraded":"2024-08-17T17:45:38.780140+0000","last_fullsized":"2024-08-17T17:45:38.780140+0000","mapping_epoch":357,"log_start":"383'7000","ondisk_log_start":"383'7000","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":5471,"log_dups_size":0,"ondisk_log_size":5471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:16:27.534439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172425216,"num_objects":2631,"num_object_clones":0,"num_object_copies":7893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12471,"num_write_kb":483264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":508,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.496559+0000","last_change":"2024-08-17T17:41:41.036615+0000","last_active":"2024-08-17T17:45:21.496559+0000","last_peered":"2024-08-17T17:45:21.496559+0000","last_clean":"2024-08-17T17:45:21.496559+0000","last_became_active":"2024-08-17T17:41:12.329214+0000","last_became_peered":"2024-08-17T17:41:12.329214+0000","last_unstale":"2024-08-17T17:45:21.496559+0000","last_undegraded":"2024-08-17T17:45:21.496559+0000","last_fullsized":"2024-08-17T17:45:21.496559+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:41.036524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:41.036524+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:59:58.695900+0000","scrub_duration":0.001820803,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":670,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.566307+0000","last_change":"2024-08-17T17:33:20.796216+0000","last_active":"2024-08-17T17:45:34.566307+0000","last_peered":"2024-08-17T17:45:34.566307+0000","last_clean":"2024-08-17T17:45:34.566307+0000","last_became_active":"2024-08-17T17:33:20.795762+0000","last_became_peered":"2024-08-17T17:33:20.795762+0000","last_unstale":"2024-08-17T17:45:34.566307+0000","last_undegraded":"2024-08-17T17:45:34.566307+0000","last_fullsized":"2024-08-17T17:45:34.566307+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:47.038509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":565,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.571151+0000","last_change":"2024-08-17T17:41:34.625551+0000","last_active":"2024-08-17T17:45:34.571151+0000","last_peered":"2024-08-17T17:45:34.571151+0000","last_clean":"2024-08-17T17:45:34.571151+0000","last_became_active":"2024-08-17T17:41:34.625251+0000","last_became_peered":"2024-08-17T17:41:34.625251+0000","last_unstale":"2024-08-17T17:45:34.571151+0000","last_undegraded":"2024-08-17T17:45:34.571151+0000","last_fullsized":"2024-08-17T17:45:34.571151+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:28:22.354213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"391'12315","reported_seq":12357,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:38.909114+0000","last_change":"2024-08-17T17:42:36.619525+0000","last_active":"2024-08-17T17:45:38.909114+0000","last_peered":"2024-08-17T17:45:38.909114+0000","last_clean":"2024-08-17T17:45:38.909114+0000","last_became_active":"2024-08-17T17:42:36.619074+0000","last_became_peered":"2024-08-17T17:42:36.619074+0000","last_unstale":"2024-08-17T17:45:38.909114+0000","last_undegraded":"2024-08-17T17:45:38.909114+0000","last_fullsized":"2024-08-17T17:45:38.909114+0000","mapping_epoch":357,"log_start":"377'5900","ondisk_log_start":"377'5900","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6415,"log_dups_size":0,"ondisk_log_size":6415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:11:32.849616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165740544,"num_objects":2529,"num_object_clones":0,"num_object_copies":7587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12315,"num_write_kb":475008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":807,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.574035+0000","last_change":"2024-08-17T17:38:25.247159+0000","last_active":"2024-08-17T17:45:34.574035+0000","last_peered":"2024-08-17T17:45:34.574035+0000","last_clean":"2024-08-17T17:45:34.574035+0000","last_became_active":"2024-08-17T17:38:21.311361+0000","last_became_peered":"2024-08-17T17:38:21.311361+0000","last_unstale":"2024-08-17T17:45:34.574035+0000","last_undegraded":"2024-08-17T17:45:34.574035+0000","last_fullsized":"2024-08-17T17:45:34.574035+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:25.247027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:25.247027+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:46:21.734712+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":797,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.490259+0000","last_change":"2024-08-17T17:33:44.149928+0000","last_active":"2024-08-17T17:45:21.490259+0000","last_peered":"2024-08-17T17:45:21.490259+0000","last_clean":"2024-08-17T17:45:21.490259+0000","last_became_active":"2024-08-17T17:33:44.149498+0000","last_became_peered":"2024-08-17T17:33:44.149498+0000","last_unstale":"2024-08-17T17:45:21.490259+0000","last_undegraded":"2024-08-17T17:45:21.490259+0000","last_fullsized":"2024-08-17T17:45:21.490259+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:17:04.963202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":618,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.564770+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:45:34.564770+0000","last_peered":"2024-08-17T17:45:34.564770+0000","last_clean":"2024-08-17T17:45:34.564770+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:45:34.564770+0000","last_undegraded":"2024-08-17T17:45:34.564770+0000","last_fullsized":"2024-08-17T17:45:34.564770+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":232,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.497211+0000","last_change":"2024-08-17T17:41:34.618631+0000","last_active":"2024-08-17T17:45:21.497211+0000","last_peered":"2024-08-17T17:45:21.497211+0000","last_clean":"2024-08-17T17:45:21.497211+0000","last_became_active":"2024-08-17T17:41:34.618313+0000","last_became_peered":"2024-08-17T17:41:34.618313+0000","last_unstale":"2024-08-17T17:45:21.497211+0000","last_undegraded":"2024-08-17T17:45:21.497211+0000","last_fullsized":"2024-08-17T17:45:21.497211+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:07:13.915088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"391'12169","reported_seq":12249,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:40.276422+0000","last_change":"2024-08-17T17:42:36.929502+0000","last_active":"2024-08-17T17:45:40.276422+0000","last_peered":"2024-08-17T17:45:40.276422+0000","last_clean":"2024-08-17T17:45:40.276422+0000","last_became_active":"2024-08-17T17:42:36.929113+0000","last_became_peered":"2024-08-17T17:42:36.929113+0000","last_unstale":"2024-08-17T17:45:40.276422+0000","last_undegraded":"2024-08-17T17:45:40.276422+0000","last_fullsized":"2024-08-17T17:45:40.276422+0000","mapping_epoch":357,"log_start":"376'5600","ondisk_log_start":"376'5600","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6569,"log_dups_size":0,"ondisk_log_size":6569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:31:25.879221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162463744,"num_objects":2479,"num_object_clones":0,"num_object_copies":7437,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12169,"num_write_kb":468736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":843,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.573676+0000","last_change":"2024-08-17T17:41:37.676347+0000","last_active":"2024-08-17T17:45:34.573676+0000","last_peered":"2024-08-17T17:45:34.573676+0000","last_clean":"2024-08-17T17:45:34.573676+0000","last_became_active":"2024-08-17T17:41:37.675869+0000","last_became_peered":"2024-08-17T17:41:37.675869+0000","last_unstale":"2024-08-17T17:45:34.573676+0000","last_undegraded":"2024-08-17T17:45:34.573676+0000","last_fullsized":"2024-08-17T17:45:34.573676+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:21.814077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:21.814077+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:04:16.240754+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":902,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.569460+0000","last_change":"2024-08-17T17:33:20.797705+0000","last_active":"2024-08-17T17:45:34.569460+0000","last_peered":"2024-08-17T17:45:34.569460+0000","last_clean":"2024-08-17T17:45:34.569460+0000","last_became_active":"2024-08-17T17:33:20.797172+0000","last_became_peered":"2024-08-17T17:33:20.797172+0000","last_unstale":"2024-08-17T17:45:34.569460+0000","last_undegraded":"2024-08-17T17:45:34.569460+0000","last_fullsized":"2024-08-17T17:45:34.569460+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:30.754999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":668,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.572385+0000","last_change":"2024-08-17T17:41:34.614108+0000","last_active":"2024-08-17T17:45:34.572385+0000","last_peered":"2024-08-17T17:45:34.572385+0000","last_clean":"2024-08-17T17:45:34.572385+0000","last_became_active":"2024-08-17T17:41:34.613481+0000","last_became_peered":"2024-08-17T17:41:34.613481+0000","last_unstale":"2024-08-17T17:45:34.572385+0000","last_undegraded":"2024-08-17T17:45:34.572385+0000","last_fullsized":"2024-08-17T17:45:34.572385+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:21:02.346115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":455,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.578575+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:45:34.578575+0000","last_peered":"2024-08-17T17:45:34.578575+0000","last_clean":"2024-08-17T17:45:34.578575+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:45:34.578575+0000","last_undegraded":"2024-08-17T17:45:34.578575+0000","last_fullsized":"2024-08-17T17:45:34.578575+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"391'12092","reported_seq":12134,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:38.701257+0000","last_change":"2024-08-17T17:42:36.946382+0000","last_active":"2024-08-17T17:45:38.701257+0000","last_peered":"2024-08-17T17:45:38.701257+0000","last_clean":"2024-08-17T17:45:38.701257+0000","last_became_active":"2024-08-17T17:42:36.945591+0000","last_became_peered":"2024-08-17T17:42:36.945591+0000","last_unstale":"2024-08-17T17:45:38.701257+0000","last_undegraded":"2024-08-17T17:45:38.701257+0000","last_fullsized":"2024-08-17T17:45:38.701257+0000","mapping_epoch":357,"log_start":"377'5800","ondisk_log_start":"377'5800","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6292,"log_dups_size":0,"ondisk_log_size":6292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:08:17.912891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174981120,"num_objects":2670,"num_object_clones":0,"num_object_copies":8010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12092,"num_write_kb":472384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":847,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.581885+0000","last_change":"2024-08-17T17:41:37.683450+0000","last_active":"2024-08-17T17:45:34.581885+0000","last_peered":"2024-08-17T17:45:34.581885+0000","last_clean":"2024-08-17T17:45:34.581885+0000","last_became_active":"2024-08-17T17:41:37.683103+0000","last_became_peered":"2024-08-17T17:41:37.683103+0000","last_unstale":"2024-08-17T17:45:34.581885+0000","last_undegraded":"2024-08-17T17:45:34.581885+0000","last_fullsized":"2024-08-17T17:45:34.581885+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.289528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.289528+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:39:17.141469+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":1035,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.565449+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:45:34.565449+0000","last_peered":"2024-08-17T17:45:34.565449+0000","last_clean":"2024-08-17T17:45:34.565449+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:45:34.565449+0000","last_undegraded":"2024-08-17T17:45:34.565449+0000","last_fullsized":"2024-08-17T17:45:34.565449+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":540,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.569337+0000","last_change":"2024-08-17T17:33:20.793468+0000","last_active":"2024-08-17T17:45:34.569337+0000","last_peered":"2024-08-17T17:45:34.569337+0000","last_clean":"2024-08-17T17:45:34.569337+0000","last_became_active":"2024-08-17T17:33:20.793166+0000","last_became_peered":"2024-08-17T17:33:20.793166+0000","last_unstale":"2024-08-17T17:45:34.569337+0000","last_undegraded":"2024-08-17T17:45:34.569337+0000","last_fullsized":"2024-08-17T17:45:34.569337+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:12:52.121050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":575,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.571068+0000","last_change":"2024-08-17T17:33:20.835784+0000","last_active":"2024-08-17T17:45:34.571068+0000","last_peered":"2024-08-17T17:45:34.571068+0000","last_clean":"2024-08-17T17:45:34.571068+0000","last_became_active":"2024-08-17T17:33:20.835371+0000","last_became_peered":"2024-08-17T17:33:20.835371+0000","last_unstale":"2024-08-17T17:45:34.571068+0000","last_undegraded":"2024-08-17T17:45:34.571068+0000","last_fullsized":"2024-08-17T17:45:34.571068+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:35:43.369055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"391'12239","reported_seq":12279,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:38.944471+0000","last_change":"2024-08-17T17:42:36.946026+0000","last_active":"2024-08-17T17:45:38.944471+0000","last_peered":"2024-08-17T17:45:38.944471+0000","last_clean":"2024-08-17T17:45:38.944471+0000","last_became_active":"2024-08-17T17:42:36.945638+0000","last_became_peered":"2024-08-17T17:42:36.945638+0000","last_unstale":"2024-08-17T17:45:38.944471+0000","last_undegraded":"2024-08-17T17:45:38.944471+0000","last_fullsized":"2024-08-17T17:45:38.944471+0000","mapping_epoch":357,"log_start":"376'5800","ondisk_log_start":"376'5800","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6439,"log_dups_size":0,"ondisk_log_size":6439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:07:34.864262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163512320,"num_objects":2495,"num_object_clones":0,"num_object_copies":7485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12239,"num_write_kb":471488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":826,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.572289+0000","last_change":"2024-08-17T17:38:22.254841+0000","last_active":"2024-08-17T17:45:34.572289+0000","last_peered":"2024-08-17T17:45:34.572289+0000","last_clean":"2024-08-17T17:45:34.572289+0000","last_became_active":"2024-08-17T17:38:21.306619+0000","last_became_peered":"2024-08-17T17:38:21.306619+0000","last_unstale":"2024-08-17T17:45:34.572289+0000","last_undegraded":"2024-08-17T17:45:34.572289+0000","last_fullsized":"2024-08-17T17:45:34.572289+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:22.254705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:22.254705+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:47:38.184089+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":1124,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.569504+0000","last_change":"2024-08-17T17:33:44.148227+0000","last_active":"2024-08-17T17:45:34.569504+0000","last_peered":"2024-08-17T17:45:34.569504+0000","last_clean":"2024-08-17T17:45:34.569504+0000","last_became_active":"2024-08-17T17:33:44.147627+0000","last_became_peered":"2024-08-17T17:33:44.147627+0000","last_unstale":"2024-08-17T17:45:34.569504+0000","last_undegraded":"2024-08-17T17:45:34.569504+0000","last_fullsized":"2024-08-17T17:45:34.569504+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:29:42.898460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1372,"num_read_kb":2843,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":55,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.500105+0000","last_change":"2024-08-17T17:41:34.611109+0000","last_active":"2024-08-17T17:45:21.500105+0000","last_peered":"2024-08-17T17:45:21.500105+0000","last_clean":"2024-08-17T17:45:21.500105+0000","last_became_active":"2024-08-17T17:41:34.610170+0000","last_became_peered":"2024-08-17T17:41:34.610170+0000","last_unstale":"2024-08-17T17:45:21.500105+0000","last_undegraded":"2024-08-17T17:45:21.500105+0000","last_fullsized":"2024-08-17T17:45:21.500105+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:39:05.301260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":579,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.570168+0000","last_change":"2024-08-17T17:33:20.772454+0000","last_active":"2024-08-17T17:45:34.570168+0000","last_peered":"2024-08-17T17:45:34.570168+0000","last_clean":"2024-08-17T17:45:34.570168+0000","last_became_active":"2024-08-17T17:33:20.771848+0000","last_became_peered":"2024-08-17T17:33:20.771848+0000","last_unstale":"2024-08-17T17:45:34.570168+0000","last_undegraded":"2024-08-17T17:45:34.570168+0000","last_fullsized":"2024-08-17T17:45:34.570168+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:46:42.300744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"391'12533","reported_seq":12574,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:38.702633+0000","last_change":"2024-08-17T17:42:36.618414+0000","last_active":"2024-08-17T17:45:38.702633+0000","last_peered":"2024-08-17T17:45:38.702633+0000","last_clean":"2024-08-17T17:45:38.702633+0000","last_became_active":"2024-08-17T17:42:36.618086+0000","last_became_peered":"2024-08-17T17:42:36.618086+0000","last_unstale":"2024-08-17T17:45:38.702633+0000","last_undegraded":"2024-08-17T17:45:38.702633+0000","last_fullsized":"2024-08-17T17:45:38.702633+0000","mapping_epoch":357,"log_start":"381'6200","ondisk_log_start":"381'6200","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6333,"log_dups_size":0,"ondisk_log_size":6333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:26:27.772894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173867008,"num_objects":2653,"num_object_clones":0,"num_object_copies":7959,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12533,"num_write_kb":485952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":528,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.502236+0000","last_change":"2024-08-17T17:41:43.035215+0000","last_active":"2024-08-17T17:45:21.502236+0000","last_peered":"2024-08-17T17:45:21.502236+0000","last_clean":"2024-08-17T17:45:21.502236+0000","last_became_active":"2024-08-17T17:41:12.201158+0000","last_became_peered":"2024-08-17T17:41:12.201158+0000","last_unstale":"2024-08-17T17:45:21.502236+0000","last_undegraded":"2024-08-17T17:45:21.502236+0000","last_fullsized":"2024-08-17T17:45:21.502236+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:43.035108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:43.035108+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:43:27.108994+0000","scrub_duration":0.0028877400000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":330,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.568801+0000","last_change":"2024-08-17T17:33:44.163867+0000","last_active":"2024-08-17T17:45:34.568801+0000","last_peered":"2024-08-17T17:45:34.568801+0000","last_clean":"2024-08-17T17:45:34.568801+0000","last_became_active":"2024-08-17T17:33:44.163503+0000","last_became_peered":"2024-08-17T17:33:44.163503+0000","last_unstale":"2024-08-17T17:45:34.568801+0000","last_undegraded":"2024-08-17T17:45:34.568801+0000","last_fullsized":"2024-08-17T17:45:34.568801+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:31:53.573596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":794,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.569867+0000","last_change":"2024-08-17T17:33:20.799350+0000","last_active":"2024-08-17T17:45:34.569867+0000","last_peered":"2024-08-17T17:45:34.569867+0000","last_clean":"2024-08-17T17:45:34.569867+0000","last_became_active":"2024-08-17T17:33:20.798925+0000","last_became_peered":"2024-08-17T17:33:20.798925+0000","last_unstale":"2024-08-17T17:45:34.569867+0000","last_undegraded":"2024-08-17T17:45:34.569867+0000","last_fullsized":"2024-08-17T17:45:34.569867+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:16:36.121295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":610,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.569959+0000","last_change":"2024-08-17T17:41:34.627064+0000","last_active":"2024-08-17T17:45:34.569959+0000","last_peered":"2024-08-17T17:45:34.569959+0000","last_clean":"2024-08-17T17:45:34.569959+0000","last_became_active":"2024-08-17T17:41:34.626453+0000","last_became_peered":"2024-08-17T17:41:34.626453+0000","last_unstale":"2024-08-17T17:45:34.569959+0000","last_undegraded":"2024-08-17T17:45:34.569959+0000","last_fullsized":"2024-08-17T17:45:34.569959+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:02:11.894042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"391'12327","reported_seq":12370,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:38.950677+0000","last_change":"2024-08-17T17:42:36.620669+0000","last_active":"2024-08-17T17:45:38.950677+0000","last_peered":"2024-08-17T17:45:38.950677+0000","last_clean":"2024-08-17T17:45:38.950677+0000","last_became_active":"2024-08-17T17:42:36.619876+0000","last_became_peered":"2024-08-17T17:42:36.619876+0000","last_unstale":"2024-08-17T17:45:38.950677+0000","last_undegraded":"2024-08-17T17:45:38.950677+0000","last_fullsized":"2024-08-17T17:45:38.950677+0000","mapping_epoch":357,"log_start":"376'5900","ondisk_log_start":"376'5900","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6427,"log_dups_size":0,"ondisk_log_size":6427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:59:34.661001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166854680,"num_objects":2547,"num_object_clones":0,"num_object_copies":7641,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2547,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12327,"num_write_kb":475905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":528,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.497504+0000","last_change":"2024-08-17T17:41:44.058291+0000","last_active":"2024-08-17T17:45:21.497504+0000","last_peered":"2024-08-17T17:45:21.497504+0000","last_clean":"2024-08-17T17:45:21.497504+0000","last_became_active":"2024-08-17T17:41:12.200077+0000","last_became_peered":"2024-08-17T17:41:12.200077+0000","last_unstale":"2024-08-17T17:45:21.497504+0000","last_undegraded":"2024-08-17T17:45:21.497504+0000","last_fullsized":"2024-08-17T17:45:21.497504+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:44.058233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:44.058233+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:42:55.911623+0000","scrub_duration":0.0074268499999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":614,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.571242+0000","last_change":"2024-08-17T17:33:20.798613+0000","last_active":"2024-08-17T17:45:34.571242+0000","last_peered":"2024-08-17T17:45:34.571242+0000","last_clean":"2024-08-17T17:45:34.571242+0000","last_became_active":"2024-08-17T17:33:20.798260+0000","last_became_peered":"2024-08-17T17:33:20.798260+0000","last_unstale":"2024-08-17T17:45:34.571242+0000","last_undegraded":"2024-08-17T17:45:34.571242+0000","last_fullsized":"2024-08-17T17:45:34.571242+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:18:00.822681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":951,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.491034+0000","last_change":"2024-08-17T17:41:37.683212+0000","last_active":"2024-08-17T17:45:21.491034+0000","last_peered":"2024-08-17T17:45:21.491034+0000","last_clean":"2024-08-17T17:45:21.491034+0000","last_became_active":"2024-08-17T17:41:37.682730+0000","last_became_peered":"2024-08-17T17:41:37.682730+0000","last_unstale":"2024-08-17T17:45:21.491034+0000","last_undegraded":"2024-08-17T17:45:21.491034+0000","last_fullsized":"2024-08-17T17:45:21.491034+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:51:35.444433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":630,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.573999+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:45:34.573999+0000","last_peered":"2024-08-17T17:45:34.573999+0000","last_clean":"2024-08-17T17:45:34.573999+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:45:34.573999+0000","last_undegraded":"2024-08-17T17:45:34.573999+0000","last_fullsized":"2024-08-17T17:45:34.573999+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"391'12319","reported_seq":12359,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:38.802331+0000","last_change":"2024-08-17T17:42:36.929709+0000","last_active":"2024-08-17T17:45:38.802331+0000","last_peered":"2024-08-17T17:45:38.802331+0000","last_clean":"2024-08-17T17:45:38.802331+0000","last_became_active":"2024-08-17T17:42:36.929285+0000","last_became_peered":"2024-08-17T17:42:36.929285+0000","last_unstale":"2024-08-17T17:45:38.802331+0000","last_undegraded":"2024-08-17T17:45:38.802331+0000","last_fullsized":"2024-08-17T17:45:38.802331+0000","mapping_epoch":357,"log_start":"383'6800","ondisk_log_start":"383'6800","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":5519,"log_dups_size":0,"ondisk_log_size":5519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:47:39.760414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167444480,"num_objects":2555,"num_object_clones":0,"num_object_copies":7665,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12319,"num_write_kb":475968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":774,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.499837+0000","last_change":"2024-08-17T17:42:19.960161+0000","last_active":"2024-08-17T17:45:21.499837+0000","last_peered":"2024-08-17T17:45:21.499837+0000","last_clean":"2024-08-17T17:45:21.499837+0000","last_became_active":"2024-08-17T17:41:37.665223+0000","last_became_peered":"2024-08-17T17:41:37.665223+0000","last_unstale":"2024-08-17T17:45:21.499837+0000","last_undegraded":"2024-08-17T17:45:21.499837+0000","last_fullsized":"2024-08-17T17:45:21.499837+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:19.960086+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:19.960086+0000","last_clean_scrub_stamp":"2024-08-17T17:42:19.960086+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:57.861661+0000","scrub_duration":0.0022612000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":664,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.568278+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:45:34.568278+0000","last_peered":"2024-08-17T17:45:34.568278+0000","last_clean":"2024-08-17T17:45:34.568278+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:45:34.568278+0000","last_undegraded":"2024-08-17T17:45:34.568278+0000","last_fullsized":"2024-08-17T17:45:34.568278+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":1239,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.570777+0000","last_change":"2024-08-17T17:41:37.677524+0000","last_active":"2024-08-17T17:45:34.570777+0000","last_peered":"2024-08-17T17:45:34.570777+0000","last_clean":"2024-08-17T17:45:34.570777+0000","last_became_active":"2024-08-17T17:41:37.676764+0000","last_became_peered":"2024-08-17T17:41:37.676764+0000","last_unstale":"2024-08-17T17:45:34.570777+0000","last_undegraded":"2024-08-17T17:45:34.570777+0000","last_fullsized":"2024-08-17T17:45:34.570777+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:14:49.341080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":662,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.566160+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:45:34.566160+0000","last_peered":"2024-08-17T17:45:34.566160+0000","last_clean":"2024-08-17T17:45:34.566160+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:45:34.566160+0000","last_undegraded":"2024-08-17T17:45:34.566160+0000","last_fullsized":"2024-08-17T17:45:34.566160+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"391'12202","reported_seq":12243,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:38.931579+0000","last_change":"2024-08-17T17:42:36.929613+0000","last_active":"2024-08-17T17:45:38.931579+0000","last_peered":"2024-08-17T17:45:38.931579+0000","last_clean":"2024-08-17T17:45:38.931579+0000","last_became_active":"2024-08-17T17:42:36.929191+0000","last_became_peered":"2024-08-17T17:42:36.929191+0000","last_unstale":"2024-08-17T17:45:38.931579+0000","last_undegraded":"2024-08-17T17:45:38.931579+0000","last_fullsized":"2024-08-17T17:45:38.931579+0000","mapping_epoch":357,"log_start":"376'5800","ondisk_log_start":"376'5800","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6402,"log_dups_size":0,"ondisk_log_size":6402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:21:40.881779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165019648,"num_objects":2518,"num_object_clones":0,"num_object_copies":7554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12202,"num_write_kb":471040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":808,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.565037+0000","last_change":"2024-08-17T17:38:23.113397+0000","last_active":"2024-08-17T17:45:34.565037+0000","last_peered":"2024-08-17T17:45:34.565037+0000","last_clean":"2024-08-17T17:45:34.565037+0000","last_became_active":"2024-08-17T17:38:21.305580+0000","last_became_peered":"2024-08-17T17:38:21.305580+0000","last_unstale":"2024-08-17T17:45:34.565037+0000","last_undegraded":"2024-08-17T17:45:34.565037+0000","last_fullsized":"2024-08-17T17:45:34.565037+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.113279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.113279+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:38:07.015721+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":501,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.578482+0000","last_change":"2024-08-17T17:41:34.629522+0000","last_active":"2024-08-17T17:45:34.578482+0000","last_peered":"2024-08-17T17:45:34.578482+0000","last_clean":"2024-08-17T17:45:34.578482+0000","last_became_active":"2024-08-17T17:41:34.619340+0000","last_became_peered":"2024-08-17T17:41:34.619340+0000","last_unstale":"2024-08-17T17:45:34.578482+0000","last_undegraded":"2024-08-17T17:45:34.578482+0000","last_fullsized":"2024-08-17T17:45:34.578482+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:30:03.902956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":621,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.502385+0000","last_change":"2024-08-17T17:26:06.855307+0000","last_active":"2024-08-17T17:45:21.502385+0000","last_peered":"2024-08-17T17:45:21.502385+0000","last_clean":"2024-08-17T17:45:21.502385+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:45:21.502385+0000","last_undegraded":"2024-08-17T17:45:21.502385+0000","last_fullsized":"2024-08-17T17:45:21.502385+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":55,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.499727+0000","last_change":"2024-08-17T17:41:34.614028+0000","last_active":"2024-08-17T17:45:21.499727+0000","last_peered":"2024-08-17T17:45:21.499727+0000","last_clean":"2024-08-17T17:45:21.499727+0000","last_became_active":"2024-08-17T17:41:34.613706+0000","last_became_peered":"2024-08-17T17:41:34.613706+0000","last_unstale":"2024-08-17T17:45:21.499727+0000","last_undegraded":"2024-08-17T17:45:21.499727+0000","last_fullsized":"2024-08-17T17:45:21.499727+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:19:55.254902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"90'149","reported_seq":719,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.570624+0000","last_change":"2024-08-17T17:42:07.681204+0000","last_active":"2024-08-17T17:45:34.570624+0000","last_peered":"2024-08-17T17:45:34.570624+0000","last_clean":"2024-08-17T17:45:34.570624+0000","last_became_active":"2024-08-17T17:42:05.878769+0000","last_became_peered":"2024-08-17T17:42:05.878769+0000","last_unstale":"2024-08-17T17:45:34.570624+0000","last_undegraded":"2024-08-17T17:45:34.570624+0000","last_fullsized":"2024-08-17T17:45:34.570624+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:07.681064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:07.681064+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:12:12.835715+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":55,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.500178+0000","last_change":"2024-08-17T17:41:34.622180+0000","last_active":"2024-08-17T17:45:21.500178+0000","last_peered":"2024-08-17T17:45:21.500178+0000","last_clean":"2024-08-17T17:45:21.500178+0000","last_became_active":"2024-08-17T17:41:34.621724+0000","last_became_peered":"2024-08-17T17:41:34.621724+0000","last_unstale":"2024-08-17T17:45:21.500178+0000","last_undegraded":"2024-08-17T17:45:21.500178+0000","last_fullsized":"2024-08-17T17:45:21.500178+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:15.807235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":65,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.500084+0000","last_change":"2024-08-17T17:41:34.626584+0000","last_active":"2024-08-17T17:45:21.500084+0000","last_peered":"2024-08-17T17:45:21.500084+0000","last_clean":"2024-08-17T17:45:21.500084+0000","last_became_active":"2024-08-17T17:41:34.609878+0000","last_became_peered":"2024-08-17T17:41:34.609878+0000","last_unstale":"2024-08-17T17:45:21.500084+0000","last_undegraded":"2024-08-17T17:45:21.500084+0000","last_fullsized":"2024-08-17T17:45:21.500084+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:08:12.810347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.17","version":"90'149","reported_seq":478,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.570648+0000","last_change":"2024-08-17T17:42:06.699082+0000","last_active":"2024-08-17T17:45:34.570648+0000","last_peered":"2024-08-17T17:45:34.570648+0000","last_clean":"2024-08-17T17:45:34.570648+0000","last_became_active":"2024-08-17T17:42:05.876218+0000","last_became_peered":"2024-08-17T17:42:05.876218+0000","last_unstale":"2024-08-17T17:45:34.570648+0000","last_undegraded":"2024-08-17T17:45:34.570648+0000","last_fullsized":"2024-08-17T17:45:34.570648+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.698998+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:06.698998+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.698998+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:22:16.510150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":607,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.569626+0000","last_change":"2024-08-17T17:33:20.799402+0000","last_active":"2024-08-17T17:45:34.569626+0000","last_peered":"2024-08-17T17:45:34.569626+0000","last_clean":"2024-08-17T17:45:34.569626+0000","last_became_active":"2024-08-17T17:33:20.798832+0000","last_became_peered":"2024-08-17T17:33:20.798832+0000","last_unstale":"2024-08-17T17:45:34.569626+0000","last_undegraded":"2024-08-17T17:45:34.569626+0000","last_fullsized":"2024-08-17T17:45:34.569626+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:21:05.001134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":382,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.568706+0000","last_change":"2024-08-17T17:41:34.634301+0000","last_active":"2024-08-17T17:45:34.568706+0000","last_peered":"2024-08-17T17:45:34.568706+0000","last_clean":"2024-08-17T17:45:34.568706+0000","last_became_active":"2024-08-17T17:41:34.620508+0000","last_became_peered":"2024-08-17T17:41:34.620508+0000","last_unstale":"2024-08-17T17:45:34.568706+0000","last_undegraded":"2024-08-17T17:45:34.568706+0000","last_fullsized":"2024-08-17T17:45:34.568706+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:10:54.576812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.14","version":"90'149","reported_seq":832,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.560048+0000","last_change":"2024-08-17T17:41:37.664471+0000","last_active":"2024-08-17T17:45:34.560048+0000","last_peered":"2024-08-17T17:45:34.560048+0000","last_clean":"2024-08-17T17:45:34.560048+0000","last_became_active":"2024-08-17T17:41:37.664108+0000","last_became_peered":"2024-08-17T17:41:37.664108+0000","last_unstale":"2024-08-17T17:45:34.560048+0000","last_undegraded":"2024-08-17T17:45:34.560048+0000","last_fullsized":"2024-08-17T17:45:34.560048+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:22.030514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:22.030514+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:15:52.325163+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":345,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.570870+0000","last_change":"2024-08-17T17:33:44.146014+0000","last_active":"2024-08-17T17:45:34.570870+0000","last_peered":"2024-08-17T17:45:34.570870+0000","last_clean":"2024-08-17T17:45:34.570870+0000","last_became_active":"2024-08-17T17:33:44.145668+0000","last_became_peered":"2024-08-17T17:33:44.145668+0000","last_unstale":"2024-08-17T17:45:34.570870+0000","last_undegraded":"2024-08-17T17:45:34.570870+0000","last_fullsized":"2024-08-17T17:45:34.570870+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:53:41.679237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":611,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.565176+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:45:34.565176+0000","last_peered":"2024-08-17T17:45:34.565176+0000","last_clean":"2024-08-17T17:45:34.565176+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:45:34.565176+0000","last_undegraded":"2024-08-17T17:45:34.565176+0000","last_fullsized":"2024-08-17T17:45:34.565176+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"90'149","reported_seq":777,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.490963+0000","last_change":"2024-08-17T17:42:06.456562+0000","last_active":"2024-08-17T17:45:21.490963+0000","last_peered":"2024-08-17T17:45:21.490963+0000","last_clean":"2024-08-17T17:45:21.490963+0000","last_became_active":"2024-08-17T17:42:05.876577+0000","last_became_peered":"2024-08-17T17:42:05.876577+0000","last_unstale":"2024-08-17T17:45:21.490963+0000","last_undegraded":"2024-08-17T17:45:21.490963+0000","last_fullsized":"2024-08-17T17:45:21.490963+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.456490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.456490+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:10:03.654953+0000","scrub_duration":0.001537285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":638,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.571141+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:45:34.571141+0000","last_peered":"2024-08-17T17:45:34.571141+0000","last_clean":"2024-08-17T17:45:34.571141+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:45:34.571141+0000","last_undegraded":"2024-08-17T17:45:34.571141+0000","last_fullsized":"2024-08-17T17:45:34.571141+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":669,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.566070+0000","last_change":"2024-08-17T17:33:20.790687+0000","last_active":"2024-08-17T17:45:34.566070+0000","last_peered":"2024-08-17T17:45:34.566070+0000","last_clean":"2024-08-17T17:45:34.566070+0000","last_became_active":"2024-08-17T17:33:20.790399+0000","last_became_peered":"2024-08-17T17:33:20.790399+0000","last_unstale":"2024-08-17T17:45:34.566070+0000","last_undegraded":"2024-08-17T17:45:34.566070+0000","last_fullsized":"2024-08-17T17:45:34.566070+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:38:06.849989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1a","version":"90'149","reported_seq":439,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.499658+0000","last_change":"2024-08-17T17:42:21.932830+0000","last_active":"2024-08-17T17:45:21.499658+0000","last_peered":"2024-08-17T17:45:21.499658+0000","last_clean":"2024-08-17T17:45:21.499658+0000","last_became_active":"2024-08-17T17:42:05.876835+0000","last_became_peered":"2024-08-17T17:42:05.876835+0000","last_unstale":"2024-08-17T17:45:21.499658+0000","last_undegraded":"2024-08-17T17:45:21.499658+0000","last_fullsized":"2024-08-17T17:45:21.499658+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:21.932757+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:21.932757+0000","last_clean_scrub_stamp":"2024-08-17T17:42:21.932757+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:36:39.848570+0000","scrub_duration":0.0020218810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":364,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.502312+0000","last_change":"2024-08-17T17:33:20.835760+0000","last_active":"2024-08-17T17:45:21.502312+0000","last_peered":"2024-08-17T17:45:21.502312+0000","last_clean":"2024-08-17T17:45:21.502312+0000","last_became_active":"2024-08-17T17:33:20.835410+0000","last_became_peered":"2024-08-17T17:33:20.835410+0000","last_unstale":"2024-08-17T17:45:21.502312+0000","last_undegraded":"2024-08-17T17:45:21.502312+0000","last_fullsized":"2024-08-17T17:45:21.502312+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:45.448022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":686,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.566210+0000","last_change":"2024-08-17T17:33:20.771279+0000","last_active":"2024-08-17T17:45:34.566210+0000","last_peered":"2024-08-17T17:45:34.566210+0000","last_clean":"2024-08-17T17:45:34.566210+0000","last_became_active":"2024-08-17T17:33:20.770728+0000","last_became_peered":"2024-08-17T17:33:20.770728+0000","last_unstale":"2024-08-17T17:45:34.566210+0000","last_undegraded":"2024-08-17T17:45:34.566210+0000","last_fullsized":"2024-08-17T17:45:34.566210+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:52:31.540397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"90'149","reported_seq":734,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.500023+0000","last_change":"2024-08-17T17:42:26.967686+0000","last_active":"2024-08-17T17:45:21.500023+0000","last_peered":"2024-08-17T17:45:21.500023+0000","last_clean":"2024-08-17T17:45:21.500023+0000","last_became_active":"2024-08-17T17:42:05.873657+0000","last_became_peered":"2024-08-17T17:42:05.873657+0000","last_unstale":"2024-08-17T17:45:21.500023+0000","last_undegraded":"2024-08-17T17:45:21.500023+0000","last_fullsized":"2024-08-17T17:45:21.500023+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:26.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:26.967611+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:31:03.344332+0000","scrub_duration":0.0015978279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":456,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.577215+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:45:34.577215+0000","last_peered":"2024-08-17T17:45:34.577215+0000","last_clean":"2024-08-17T17:45:34.577215+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:45:34.577215+0000","last_undegraded":"2024-08-17T17:45:34.577215+0000","last_fullsized":"2024-08-17T17:45:34.577215+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":606,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.569412+0000","last_change":"2024-08-17T17:41:34.625011+0000","last_active":"2024-08-17T17:45:34.569412+0000","last_peered":"2024-08-17T17:45:34.569412+0000","last_clean":"2024-08-17T17:45:34.569412+0000","last_became_active":"2024-08-17T17:41:34.624631+0000","last_became_peered":"2024-08-17T17:41:34.624631+0000","last_unstale":"2024-08-17T17:45:34.569412+0000","last_undegraded":"2024-08-17T17:45:34.569412+0000","last_fullsized":"2024-08-17T17:45:34.569412+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:03:05.565435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"90'149","reported_seq":697,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.503669+0000","last_change":"2024-08-17T17:42:08.482975+0000","last_active":"2024-08-17T17:45:21.503669+0000","last_peered":"2024-08-17T17:45:21.503669+0000","last_clean":"2024-08-17T17:45:21.503669+0000","last_became_active":"2024-08-17T17:42:05.873523+0000","last_became_peered":"2024-08-17T17:42:05.873523+0000","last_unstale":"2024-08-17T17:45:21.503669+0000","last_undegraded":"2024-08-17T17:45:21.503669+0000","last_fullsized":"2024-08-17T17:45:21.503669+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:08.482892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:08.482892+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:41.477341+0000","scrub_duration":0.0017925479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":55,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.502429+0000","last_change":"2024-08-17T17:41:34.616482+0000","last_active":"2024-08-17T17:45:21.502429+0000","last_peered":"2024-08-17T17:45:21.502429+0000","last_clean":"2024-08-17T17:45:21.502429+0000","last_became_active":"2024-08-17T17:41:34.616040+0000","last_became_peered":"2024-08-17T17:41:34.616040+0000","last_unstale":"2024-08-17T17:45:21.502429+0000","last_undegraded":"2024-08-17T17:45:21.502429+0000","last_fullsized":"2024-08-17T17:45:21.502429+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:53:08.502264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":55,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.499971+0000","last_change":"2024-08-17T17:41:34.622184+0000","last_active":"2024-08-17T17:45:21.499971+0000","last_peered":"2024-08-17T17:45:21.499971+0000","last_clean":"2024-08-17T17:45:21.499971+0000","last_became_active":"2024-08-17T17:41:34.621831+0000","last_became_peered":"2024-08-17T17:41:34.621831+0000","last_unstale":"2024-08-17T17:45:21.499971+0000","last_undegraded":"2024-08-17T17:45:21.499971+0000","last_fullsized":"2024-08-17T17:45:21.499971+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:59.826667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"90'149","reported_seq":835,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.564770+0000","last_change":"2024-08-17T17:42:05.876463+0000","last_active":"2024-08-17T17:45:34.564770+0000","last_peered":"2024-08-17T17:45:34.564770+0000","last_clean":"2024-08-17T17:45:34.564770+0000","last_became_active":"2024-08-17T17:42:05.876069+0000","last_became_peered":"2024-08-17T17:42:05.876069+0000","last_unstale":"2024-08-17T17:45:34.564770+0000","last_undegraded":"2024-08-17T17:45:34.564770+0000","last_fullsized":"2024-08-17T17:45:34.564770+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:45.552985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:45.552985+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:55:06.027278+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":572,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.570077+0000","last_change":"2024-08-17T17:33:20.773207+0000","last_active":"2024-08-17T17:45:34.570077+0000","last_peered":"2024-08-17T17:45:34.570077+0000","last_clean":"2024-08-17T17:45:34.570077+0000","last_became_active":"2024-08-17T17:33:20.772457+0000","last_became_peered":"2024-08-17T17:33:20.772457+0000","last_unstale":"2024-08-17T17:45:34.570077+0000","last_undegraded":"2024-08-17T17:45:34.570077+0000","last_fullsized":"2024-08-17T17:45:34.570077+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:57:09.610084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":479,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.573711+0000","last_change":"2024-08-17T17:41:34.618968+0000","last_active":"2024-08-17T17:45:34.573711+0000","last_peered":"2024-08-17T17:45:34.573711+0000","last_clean":"2024-08-17T17:45:34.573711+0000","last_became_active":"2024-08-17T17:41:34.618576+0000","last_became_peered":"2024-08-17T17:41:34.618576+0000","last_unstale":"2024-08-17T17:45:34.573711+0000","last_undegraded":"2024-08-17T17:45:34.573711+0000","last_fullsized":"2024-08-17T17:45:34.573711+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:47:45.888590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"90'149","reported_seq":483,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.565341+0000","last_change":"2024-08-17T17:42:06.742801+0000","last_active":"2024-08-17T17:45:34.565341+0000","last_peered":"2024-08-17T17:45:34.565341+0000","last_clean":"2024-08-17T17:45:34.565341+0000","last_became_active":"2024-08-17T17:42:05.876895+0000","last_became_peered":"2024-08-17T17:42:05.876895+0000","last_unstale":"2024-08-17T17:45:34.565341+0000","last_undegraded":"2024-08-17T17:45:34.565341+0000","last_fullsized":"2024-08-17T17:45:34.565341+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.742723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.742723+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:36:46.651953+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":55,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.499941+0000","last_change":"2024-08-17T17:41:34.622094+0000","last_active":"2024-08-17T17:45:21.499941+0000","last_peered":"2024-08-17T17:45:21.499941+0000","last_clean":"2024-08-17T17:45:21.499941+0000","last_became_active":"2024-08-17T17:41:34.621773+0000","last_became_peered":"2024-08-17T17:41:34.621773+0000","last_unstale":"2024-08-17T17:45:21.499941+0000","last_undegraded":"2024-08-17T17:45:21.499941+0000","last_fullsized":"2024-08-17T17:45:21.499941+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:34:30.540886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":59,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.500189+0000","last_change":"2024-08-17T17:41:34.629135+0000","last_active":"2024-08-17T17:45:21.500189+0000","last_peered":"2024-08-17T17:45:21.500189+0000","last_clean":"2024-08-17T17:45:21.500189+0000","last_became_active":"2024-08-17T17:41:34.613095+0000","last_became_peered":"2024-08-17T17:41:34.613095+0000","last_unstale":"2024-08-17T17:45:21.500189+0000","last_undegraded":"2024-08-17T17:45:21.500189+0000","last_fullsized":"2024-08-17T17:45:21.500189+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:27:06.542416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":55,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.500099+0000","last_change":"2024-08-17T17:41:34.616251+0000","last_active":"2024-08-17T17:45:21.500099+0000","last_peered":"2024-08-17T17:45:21.500099+0000","last_clean":"2024-08-17T17:45:21.500099+0000","last_became_active":"2024-08-17T17:41:34.615952+0000","last_became_peered":"2024-08-17T17:41:34.615952+0000","last_unstale":"2024-08-17T17:45:21.500099+0000","last_undegraded":"2024-08-17T17:45:21.500099+0000","last_fullsized":"2024-08-17T17:45:21.500099+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:12:24.892508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":59,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.499933+0000","last_change":"2024-08-17T17:41:34.631152+0000","last_active":"2024-08-17T17:45:21.499933+0000","last_peered":"2024-08-17T17:45:21.499933+0000","last_clean":"2024-08-17T17:45:21.499933+0000","last_became_active":"2024-08-17T17:41:34.615617+0000","last_became_peered":"2024-08-17T17:41:34.615617+0000","last_unstale":"2024-08-17T17:45:21.499933+0000","last_undegraded":"2024-08-17T17:45:21.499933+0000","last_fullsized":"2024-08-17T17:45:21.499933+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:43:08.257084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"90'149","reported_seq":731,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.497117+0000","last_change":"2024-08-17T17:42:11.809047+0000","last_active":"2024-08-17T17:45:21.497117+0000","last_peered":"2024-08-17T17:45:21.497117+0000","last_clean":"2024-08-17T17:45:21.497117+0000","last_became_active":"2024-08-17T17:42:05.883770+0000","last_became_peered":"2024-08-17T17:42:05.883770+0000","last_unstale":"2024-08-17T17:45:21.497117+0000","last_undegraded":"2024-08-17T17:45:21.497117+0000","last_fullsized":"2024-08-17T17:45:21.497117+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:11.808971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:11.808971+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:48:15.989437+0000","scrub_duration":0.0018167420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":664,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.567826+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:45:34.567826+0000","last_peered":"2024-08-17T17:45:34.567826+0000","last_clean":"2024-08-17T17:45:34.567826+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:45:34.567826+0000","last_undegraded":"2024-08-17T17:45:34.567826+0000","last_fullsized":"2024-08-17T17:45:34.567826+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":666,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.565533+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:45:34.565533+0000","last_peered":"2024-08-17T17:45:34.565533+0000","last_clean":"2024-08-17T17:45:34.565533+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:45:34.565533+0000","last_undegraded":"2024-08-17T17:45:34.565533+0000","last_fullsized":"2024-08-17T17:45:34.565533+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1d","version":"90'149","reported_seq":731,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.570269+0000","last_change":"2024-08-17T17:42:18.668058+0000","last_active":"2024-08-17T17:45:34.570269+0000","last_peered":"2024-08-17T17:45:34.570269+0000","last_clean":"2024-08-17T17:45:34.570269+0000","last_became_active":"2024-08-17T17:42:05.876721+0000","last_became_peered":"2024-08-17T17:42:05.876721+0000","last_unstale":"2024-08-17T17:45:34.570269+0000","last_undegraded":"2024-08-17T17:45:34.570269+0000","last_fullsized":"2024-08-17T17:45:34.570269+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:18.667957+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:18.667957+0000","last_clean_scrub_stamp":"2024-08-17T17:42:18.667957+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:38.170146+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":55,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-08-17T17:45:21.499578+0000","last_change":"2024-08-17T17:41:34.611711+0000","last_active":"2024-08-17T17:45:21.499578+0000","last_peered":"2024-08-17T17:45:21.499578+0000","last_clean":"2024-08-17T17:45:21.499578+0000","last_became_active":"2024-08-17T17:41:34.611407+0000","last_became_peered":"2024-08-17T17:41:34.611407+0000","last_unstale":"2024-08-17T17:45:21.499578+0000","last_undegraded":"2024-08-17T17:45:21.499578+0000","last_fullsized":"2024-08-17T17:45:21.499578+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:42:41.582320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":475,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-08-17T17:45:34.571036+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:45:34.571036+0000","last_peered":"2024-08-17T17:45:34.571036+0000","last_clean":"2024-08-17T17:45:34.571036+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:45:34.571036+0000","last_undegraded":"2024-08-17T17:45:34.571036+0000","last_fullsized":"2024-08-17T17:45:34.571036+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":2661351448,"num_objects":40610,"num_object_clones":0,"num_object_copies":121830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2538,"num_objects_misplaced":1067,"num_objects_unfound":0,"num_objects_dirty":40610,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196478,"num_write_kb":7586753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3195,"num_bytes_recovered":209387520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8268754944,"data_stored":8268742728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101778,"ondisk_log_size":101778,"up":48,"acting":47,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":11256,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19280,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5978,"num_read_kb":12991,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4619,"ondisk_log_size":4619,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234466,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1573528,"kb_used_data":683792,"kb_used_omap":1,"kb_used_meta":889726,"kb_avail":92171624,"statfs":{"total":95995035648,"available":94383742976,"internally_reserved":0,"allocated":700203008,"data_stored":697972303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":911079768},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,0,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496101,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2048908,"kb_used_data":1157956,"kb_used_omap":1,"kb_used_meta":890942,"kb_avail":91696244,"statfs":{"total":95995035648,"available":93896953856,"internally_reserved":0,"allocated":1185746944,"data_stored":1183527681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":912324952},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757738,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1667592,"kb_used_data":781120,"kb_used_omap":1,"kb_used_meta":886462,"kb_avail":92077560,"statfs":{"total":95995035648,"available":94287421440,"internally_reserved":0,"allocated":799866880,"data_stored":797648258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":907737432},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019376,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1747416,"kb_used_data":1437520,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91997736,"statfs":{"total":95995035648,"available":94205681664,"internally_reserved":0,"allocated":1472020480,"data_stored":1469789516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435603,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1763904,"kb_used_data":821816,"kb_used_omap":4,"kb_used_meta":942075,"kb_avail":91981248,"statfs":{"total":95995035648,"available":94188797952,"internally_reserved":0,"allocated":841539584,"data_stored":839328392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":964684810},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599130,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2168968,"kb_used_data":1355392,"kb_used_omap":7,"kb_used_meta":813560,"kb_avail":91576184,"statfs":{"total":95995035648,"available":93774012416,"internally_reserved":0,"allocated":1387921408,"data_stored":1385703082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":833085602},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":268,"seq":1151051235477,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":927232,"kb_used_data":468664,"kb_used_omap":10,"kb_used_meta":458549,"kb_avail":92817920,"statfs":{"total":95995035648,"available":95045550080,"internally_reserved":0,"allocated":479911936,"data_stored":477700689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10902,"internal_metadata":469554538},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926183,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2543176,"kb_used_data":1626752,"kb_used_omap":7,"kb_used_meta":916408,"kb_avail":91201976,"statfs":{"total":95995035648,"available":93390823424,"internally_reserved":0,"allocated":1665794048,"data_stored":1663575198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":938402113},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7550,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1625362432,"data_stored":1625358360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":468385792,"data_stored":468385792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1362034688,"data_stored":1362034688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":833355776,"data_stored":833355776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1348341760,"data_stored":1348337688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":780337152,"data_stored":780337152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1161039872,"data_stored":1161035800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":689897472,"data_stored":689897472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:45:46.147 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph status --format=json 2024-08-17T17:45:46.170 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:46.202 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:46.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:46.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:46.304 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:46.337 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:46.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:46.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:46.439 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:46.467 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:46.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:46.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:46.569 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:46.602 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:46.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:46.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:46.704 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:46.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:46.751 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:46.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:46.853 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:46.886 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:46.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:46.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:46.988 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:47.024 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:47.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:47.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:47.126 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:47.157 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:47.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:47.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:47.259 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:47.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:47.295 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:47.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:47.396 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:47.434 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:47.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:47.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:47.536 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:47.575 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:47.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:47.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:47.676 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:47.723 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:47.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:47.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:47.731 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:45:47.731 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:45:47.773 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:45:47.773 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:45:47.803 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:45:47.803 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:45:47.825 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:47.838 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:45:47.838 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:45:47.860 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:47.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:47.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:47.869 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:45:47.869 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:45:47.900 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:45:47.900 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:45:47.900 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:47 smithi003 ceph-mon[100615]: pgmap v1298: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 998 op/s; 2666/107550 objects degraded (2.479%); 734/107550 objects misplaced (0.682%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T17:45:47.900 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:47 smithi003 ceph-mon[103344]: pgmap v1298: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 998 op/s; 2666/107550 objects degraded (2.479%); 734/107550 objects misplaced (0.682%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T17:45:47.931 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:45:47.932 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:45:47.962 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:47.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:47 smithi053 ceph-mon[89963]: pgmap v1298: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 998 op/s; 2666/107550 objects degraded (2.479%); 734/107550 objects misplaced (0.682%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T17:45:47.969 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:45:47.970 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:45:48.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:48.013 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:48.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:48.114 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:48.163 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:48.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:48.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:48.265 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:48.295 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:48.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:48.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:48.397 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:48.429 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:48.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:48.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:48.531 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:48.560 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:48.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:48.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:48.630 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-08-17T17:45:48.662 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:48.701 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:48.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:48.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:48.803 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:48.842 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:48.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:48.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:48.946 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:48.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:48 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/911224459' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:45:48.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:48 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/227371476' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:45:48.981 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:48.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:48.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:49.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:48 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/911224459' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:45:49.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:48 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/227371476' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:45:49.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:48 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/911224459' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:45:49.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:48 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/227371476' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:45:49.083 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:49.123 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:49.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:49.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:49.225 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:49.258 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:49.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:49.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:49.361 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:49.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:49.399 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:49.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:49.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:49.541 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:49.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:49.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:49.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:49.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:49.679 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:49.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:49.780 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:49.814 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:49.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:49.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:49.916 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:49.952 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:49.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:49.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:49.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:49 smithi053 ceph-mon[89963]: pgmap v1299: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 825 op/s; 2666/107550 objects degraded (2.479%); 734/107550 objects misplaced (0.682%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T17:45:49.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:49 smithi053 ceph-mon[89963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:45:49.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:49 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/227371476' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:45:49.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:49 smithi053 ceph-mon[89963]: osdmap e392: 8 total, 8 up, 8 in 2024-08-17T17:45:50.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:49 smithi003 ceph-mon[103344]: pgmap v1299: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 825 op/s; 2666/107550 objects degraded (2.479%); 734/107550 objects misplaced (0.682%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T17:45:50.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:49 smithi003 ceph-mon[103344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:45:50.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:49 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/227371476' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:45:50.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:49 smithi003 ceph-mon[103344]: osdmap e392: 8 total, 8 up, 8 in 2024-08-17T17:45:50.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:49 smithi003 ceph-mon[100615]: pgmap v1299: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 825 op/s; 2666/107550 objects degraded (2.479%); 734/107550 objects misplaced (0.682%); 1.9 MiB/s, 30 objects/s recovering 2024-08-17T17:45:50.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:49 smithi003 ceph-mon[100615]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:45:50.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:49 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/227371476' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:45:50.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:49 smithi003 ceph-mon[100615]: osdmap e392: 8 total, 8 up, 8 in 2024-08-17T17:45:50.054 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:50.084 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:50.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:50.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:50.186 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:50.220 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:50.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:50.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:50.321 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:50.355 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:50.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:50.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:50.457 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:50.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:45:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:45:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:45:50.495 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:50.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:50.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:50.597 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:50.637 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:50.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:50.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:50.738 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:50.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:50 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 2666/107550 objects degraded (2.479%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:50.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:50 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 2666/107550 objects degraded (2.479%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:50.777 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:50.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:50.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:50.816 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:45:50.818 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:45:50.879 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:50.916 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:50.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:50.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:50.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:50 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 2666/107550 objects degraded (2.479%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:51.018 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:51.042 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:51.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:51.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:51.144 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:51.184 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:51.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:51.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:51.286 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:51.325 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:51.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:51.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:51.427 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:51.464 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:51.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:51.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:51.566 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:51.607 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:51.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:51.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:51.709 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:51.752 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:51.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:51.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:51.854 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:51.891 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:51.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:51.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:51.898 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:51 smithi003 ceph-mon[103344]: pgmap v1301: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 992 op/s; 2804/92982 objects degraded (3.016%); 407/92982 objects misplaced (0.438%); 2.5 MiB/s, 39 objects/s recovering 2024-08-17T17:45:51.899 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:51 smithi003 ceph-mon[100615]: pgmap v1301: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 992 op/s; 2804/92982 objects degraded (3.016%); 407/92982 objects misplaced (0.438%); 2.5 MiB/s, 39 objects/s recovering 2024-08-17T17:45:51.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:51 smithi053 ceph-mon[89963]: pgmap v1301: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 992 op/s; 2804/92982 objects degraded (3.016%); 407/92982 objects misplaced (0.438%); 2.5 MiB/s, 39 objects/s recovering 2024-08-17T17:45:51.993 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:52.031 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:52.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:52.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:52.132 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:52.167 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:52.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:52.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:52.268 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:52.311 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:52.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:52.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:52.413 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:52.450 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:52.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:52.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:52.547 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:45:52.552 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:52.593 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:52.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:52.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:52.695 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:52.734 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:52.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:52.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:52.836 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:52.869 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:52.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:52.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:52.971 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:53.009 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:53.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:53.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:53.111 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:53.170 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:53.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:53.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:53.272 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:53.305 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:53.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:53.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:53.375 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:45:53.407 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:53.442 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:53.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:53.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:53.544 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:53.583 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:53.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:53.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:53.685 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:53.725 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:53.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:53.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:53.827 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:53.865 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:53.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:53.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:53.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:53 smithi053 ceph-mon[89963]: pgmap v1302: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 2804/90423 objects degraded (3.101%); 407/90423 objects misplaced (0.450%); 2.5 MiB/s, 39 objects/s recovering 2024-08-17T17:45:53.978 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:45:53.978 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:54.011 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:54.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:54.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:54.016 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:53 smithi003 ceph-mon[103344]: pgmap v1302: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 2804/90423 objects degraded (3.101%); 407/90423 objects misplaced (0.450%); 2.5 MiB/s, 39 objects/s recovering 2024-08-17T17:45:54.017 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:45:54.017 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:53 smithi003 ceph-mon[100615]: pgmap v1302: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 2804/90423 objects degraded (3.101%); 407/90423 objects misplaced (0.450%); 2.5 MiB/s, 39 objects/s recovering 2024-08-17T17:45:54.017 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:45:54.113 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:54.144 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:54.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:54.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:54.246 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:54.283 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:54.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:54.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:54.384 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:54.426 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:54.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:54.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:54.527 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:54.559 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:54.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:54.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:54.661 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:54.699 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:54.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:54.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:54.801 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:54.852 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:54.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:54.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:54.857 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:45:54.857 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-08-17T17:45:54.857 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-08-17T17:45:54.857 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-08-17T17:45:54.857 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.857 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36354.0:22720 9.6 e314/311 9:6a7b00cf:::benchmark_data_smithi003_261051_object22719:head v 314'1422, mlcod=314'1420)", 2024-08-17T17:45:54.857 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:37:58.657489+0000", 2024-08-17T17:45:54.857 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 476.193645522, 2024-08-17T17:45:54.857 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.095732504999999996, 2024-08-17T17:45:54.857 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:45:54.857 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:45:54.858 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:45:54.858 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:45:54.858 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.858 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:45:54.858 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.657489+0000", 2024-08-17T17:45:54.858 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.858 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.858 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.858 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:45:54.858 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.657489+0000", 2024-08-17T17:45:54.858 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.858 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.859 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.859 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:45:54.859 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.657491+0000", 2024-08-17T17:45:54.859 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3039999999999999e-06 2024-08-17T17:45:54.859 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.859 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.859 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:45:54.859 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.657516+0000", 2024-08-17T17:45:54.859 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5686000000000001e-05 2024-08-17T17:45:54.859 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.859 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.859 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:45:54.859 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.657533+0000", 2024-08-17T17:45:54.859 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6889e-05 2024-08-17T17:45:54.860 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.860 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.860 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:45:54.860 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.657542+0000", 2024-08-17T17:45:54.860 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5229999999999995e-06 2024-08-17T17:45:54.860 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.860 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.860 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:45:54.860 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.657627+0000", 2024-08-17T17:45:54.860 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4790999999999995e-05 2024-08-17T17:45:54.860 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.860 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.860 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:45:54.861 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.657648+0000", 2024-08-17T17:45:54.861 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1158999999999999e-05 2024-08-17T17:45:54.861 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.861 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.861 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:45:54.861 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.660228+0000", 2024-08-17T17:45:54.861 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025802329999999999 2024-08-17T17:45:54.861 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.861 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.861 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:45:54.861 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.753222+0000", 2024-08-17T17:45:54.861 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.092993919999999994 2024-08-17T17:45:54.861 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.861 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:45:54.862 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.862 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.862 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.862 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36354.0:22723 9.6 e314/311 9:6a56a808:::benchmark_data_smithi003_261051_object22722:head v 314'1423, mlcod=314'1420)", 2024-08-17T17:45:54.862 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:37:58.658821+0000", 2024-08-17T17:45:54.862 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 476.19231406500001, 2024-08-17T17:45:54.862 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.094450671999999999, 2024-08-17T17:45:54.862 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:45:54.862 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:45:54.862 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:45:54.862 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:45:54.862 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.862 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:45:54.862 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.658821+0000", 2024-08-17T17:45:54.863 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.863 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.863 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.863 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:45:54.863 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.658821+0000", 2024-08-17T17:45:54.863 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.863 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.863 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.863 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:45:54.863 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.658822+0000", 2024-08-17T17:45:54.863 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0279999999999999e-06 2024-08-17T17:45:54.863 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.863 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.864 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:45:54.864 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.658845+0000", 2024-08-17T17:45:54.864 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3519e-05 2024-08-17T17:45:54.864 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.864 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.864 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:45:54.864 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.658856+0000", 2024-08-17T17:45:54.864 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0686999999999999e-05 2024-08-17T17:45:54.864 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.864 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.864 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:45:54.864 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.658862+0000", 2024-08-17T17:45:54.864 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2949999999999999e-06 2024-08-17T17:45:54.865 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.865 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.865 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:45:54.865 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.659339+0000", 2024-08-17T17:45:54.865 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00047612499999999999 2024-08-17T17:45:54.865 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.865 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.865 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:45:54.865 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.659358+0000", 2024-08-17T17:45:54.865 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9692999999999999e-05 2024-08-17T17:45:54.865 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.865 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.865 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:45:54.865 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.753229+0000", 2024-08-17T17:45:54.866 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.093871011000000004 2024-08-17T17:45:54.866 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.866 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.866 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:45:54.866 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.753272+0000", 2024-08-17T17:45:54.866 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2314000000000002e-05 2024-08-17T17:45:54.866 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.866 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:45:54.866 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.866 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.866 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.866 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36354.0:22740 9.2 e314/311 9:48d0b398:::benchmark_data_smithi003_261051_object22739:head v 314'1437, mlcod=314'1435)", 2024-08-17T17:45:54.866 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:37:58.664609+0000", 2024-08-17T17:45:54.867 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 476.186526238, 2024-08-17T17:45:54.867 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.091359095000000001, 2024-08-17T17:45:54.867 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:45:54.867 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:45:54.867 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:45:54.867 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:45:54.867 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.867 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:45:54.867 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.664609+0000", 2024-08-17T17:45:54.867 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.867 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.867 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.867 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:45:54.867 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.664609+0000", 2024-08-17T17:45:54.868 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.868 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.868 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.868 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:45:54.868 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.664610+0000", 2024-08-17T17:45:54.868 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.291e-06 2024-08-17T17:45:54.868 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.868 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.868 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:45:54.868 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.664635+0000", 2024-08-17T17:45:54.868 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5326000000000001e-05 2024-08-17T17:45:54.868 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.868 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.868 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:45:54.868 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.753200+0000", 2024-08-17T17:45:54.869 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.088564697999999997 2024-08-17T17:45:54.869 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.869 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.869 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:45:54.869 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.753215+0000", 2024-08-17T17:45:54.869 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4684e-05 2024-08-17T17:45:54.869 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.869 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.869 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:45:54.869 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.753270+0000", 2024-08-17T17:45:54.869 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4922e-05 2024-08-17T17:45:54.869 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.869 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.870 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:45:54.870 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.753290+0000", 2024-08-17T17:45:54.870 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0832999999999999e-05 2024-08-17T17:45:54.870 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.870 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.870 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:45:54.870 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.755893+0000", 2024-08-17T17:45:54.870 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026026769999999999 2024-08-17T17:45:54.870 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.870 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.870 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:45:54.870 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:37:58.755968+0000", 2024-08-17T17:45:54.870 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4664000000000002e-05 2024-08-17T17:45:54.870 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.870 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:45:54.871 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.871 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.871 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.871 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36354.0:55396 9.6 e315/311 9:6d7d6566:::benchmark_data_smithi003_261051_object55395:head v 315'3425, mlcod=315'3421)", 2024-08-17T17:45:54.871 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:38:18.615557+0000", 2024-08-17T17:45:54.871 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 456.23557834399998, 2024-08-17T17:45:54.871 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.033345782999999997, 2024-08-17T17:45:54.871 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:45:54.871 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:45:54.871 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:45:54.871 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:45:54.871 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.871 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:45:54.872 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:18.615557+0000", 2024-08-17T17:45:54.872 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.872 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.872 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.872 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:45:54.872 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:18.615557+0000", 2024-08-17T17:45:54.872 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.872 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.872 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.872 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:45:54.872 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:18.615558+0000", 2024-08-17T17:45:54.872 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.469e-06 2024-08-17T17:45:54.872 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.872 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.872 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:45:54.873 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:18.615777+0000", 2024-08-17T17:45:54.873 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00021862499999999999 2024-08-17T17:45:54.873 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.873 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.873 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:45:54.873 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:18.615794+0000", 2024-08-17T17:45:54.873 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7204999999999999e-05 2024-08-17T17:45:54.873 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.873 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.873 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:45:54.873 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:18.615803+0000", 2024-08-17T17:45:54.873 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9190000000000005e-06 2024-08-17T17:45:54.873 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.874 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.874 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:45:54.874 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:18.615871+0000", 2024-08-17T17:45:54.874 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7995000000000002e-05 2024-08-17T17:45:54.874 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.874 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.874 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:45:54.874 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:18.615890+0000", 2024-08-17T17:45:54.874 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9400000000000001e-05 2024-08-17T17:45:54.874 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.874 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.874 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:45:54.874 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:18.648826+0000", 2024-08-17T17:45:54.875 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.032935706000000002 2024-08-17T17:45:54.875 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.875 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.875 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:45:54.875 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:18.648902+0000", 2024-08-17T17:45:54.875 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6464000000000005e-05 2024-08-17T17:45:54.875 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.875 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:45:54.875 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.875 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.875 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.875 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36354.0:97677 9.6 e325/311 9:6a9b1633:::benchmark_data_smithi003_261051_object97676:head v 325'6053, mlcod=325'6050)", 2024-08-17T17:45:54.875 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:38:54.226274+0000", 2024-08-17T17:45:54.875 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 420.62486110200001, 2024-08-17T17:45:54.876 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.032037085, 2024-08-17T17:45:54.876 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:45:54.876 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:45:54.876 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:45:54.876 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:45:54.876 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.876 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:45:54.876 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:54.226274+0000", 2024-08-17T17:45:54.876 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.876 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.876 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.876 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:45:54.876 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:54.226274+0000", 2024-08-17T17:45:54.877 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.877 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.877 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.877 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:45:54.877 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:54.226274+0000", 2024-08-17T17:45:54.877 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5499999999999998e-07 2024-08-17T17:45:54.877 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.877 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.877 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:45:54.877 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:54.226296+0000", 2024-08-17T17:45:54.877 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1852e-05 2024-08-17T17:45:54.877 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.877 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.877 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:45:54.878 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:54.226304+0000", 2024-08-17T17:45:54.878 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7139999999999993e-06 2024-08-17T17:45:54.878 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.878 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.878 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:45:54.878 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:54.226309+0000", 2024-08-17T17:45:54.878 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.237e-06 2024-08-17T17:45:54.878 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.878 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.878 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:45:54.878 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:54.230273+0000", 2024-08-17T17:45:54.878 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039643120000000002 2024-08-17T17:45:54.878 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.879 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.879 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:45:54.879 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:54.230324+0000", 2024-08-17T17:45:54.879 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0624000000000002e-05 2024-08-17T17:45:54.879 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.879 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.879 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:45:54.879 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:54.258270+0000", 2024-08-17T17:45:54.879 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.027945817000000001 2024-08-17T17:45:54.879 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.879 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.879 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:45:54.879 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:38:54.258311+0000", 2024-08-17T17:45:54.879 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1074e-05 2024-08-17T17:45:54.880 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.880 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:45:54.880 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.880 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.880 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.880 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36612.0:67174 10.5 e372/357 10:a128242e:::benchmark_data_smithi003_292749_object67173:head v 372'4151, mlcod=372'4149)", 2024-08-17T17:45:54.880 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:43:28.396807+0000", 2024-08-17T17:45:54.880 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 146.45432778200001, 2024-08-17T17:45:54.880 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.033948962999999999, 2024-08-17T17:45:54.880 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:45:54.880 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:45:54.880 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:45:54.880 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:45:54.880 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.880 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:45:54.881 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:43:28.396807+0000", 2024-08-17T17:45:54.881 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.881 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.881 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.881 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:45:54.881 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:43:28.396807+0000", 2024-08-17T17:45:54.881 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.881 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.881 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.881 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:45:54.881 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:43:28.396808+0000", 2024-08-17T17:45:54.881 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9699999999999995e-07 2024-08-17T17:45:54.881 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.881 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.882 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:45:54.882 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:43:28.401537+0000", 2024-08-17T17:45:54.882 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0047287639999999999 2024-08-17T17:45:54.882 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.882 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.882 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:45:54.882 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:43:28.401548+0000", 2024-08-17T17:45:54.882 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1476e-05 2024-08-17T17:45:54.882 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.882 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.882 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:45:54.882 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:43:28.401556+0000", 2024-08-17T17:45:54.882 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4680000000000003e-06 2024-08-17T17:45:54.882 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.882 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.883 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:45:54.883 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:43:28.401725+0000", 2024-08-17T17:45:54.883 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000168925 2024-08-17T17:45:54.883 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.883 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.883 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:45:54.883 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:43:28.401746+0000", 2024-08-17T17:45:54.883 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1279000000000001e-05 2024-08-17T17:45:54.883 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.883 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.883 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:45:54.883 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:43:28.430563+0000", 2024-08-17T17:45:54.883 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.028817519999999999 2024-08-17T17:45:54.883 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.884 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.884 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:45:54.884 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:43:28.430756+0000", 2024-08-17T17:45:54.884 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019273399999999999 2024-08-17T17:45:54.884 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.884 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:45:54.884 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.884 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.884 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.884 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36612.0:93606 10.7 e376/368 10:eede7a59:::benchmark_data_smithi003_292749_object93605:head v 376'5873, mlcod=376'5869)", 2024-08-17T17:45:54.884 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:43:51.072105+0000", 2024-08-17T17:45:54.884 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 123.779030365, 2024-08-17T17:45:54.884 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.035921967999999999, 2024-08-17T17:45:54.884 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:45:54.885 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:45:54.885 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:45:54.885 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:45:54.885 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.885 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:45:54.885 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:43:51.072105+0000", 2024-08-17T17:45:54.885 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.885 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.885 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.885 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:45:54.885 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:43:51.072105+0000", 2024-08-17T17:45:54.885 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.885 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.886 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.886 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:45:54.886 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:43:51.072105+0000", 2024-08-17T17:45:54.886 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6899999999999999e-07 2024-08-17T17:45:54.886 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.886 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.886 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:45:54.886 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:43:51.072137+0000", 2024-08-17T17:45:54.886 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2495e-05 2024-08-17T17:45:54.886 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.886 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.886 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:45:54.887 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:43:51.072144+0000", 2024-08-17T17:45:54.887 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9e-06 2024-08-17T17:45:54.887 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.887 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.887 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:45:54.887 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:43:51.072149+0000", 2024-08-17T17:45:54.887 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0370000000000002e-06 2024-08-17T17:45:54.887 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.887 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.887 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:45:54.887 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:43:51.075865+0000", 2024-08-17T17:45:54.887 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0037158769999999998 2024-08-17T17:45:54.887 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.887 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.887 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:45:54.888 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:43:51.075887+0000", 2024-08-17T17:45:54.888 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1733e-05 2024-08-17T17:45:54.888 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.888 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.888 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:45:54.888 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:43:51.107980+0000", 2024-08-17T17:45:54.888 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.032092622000000001 2024-08-17T17:45:54.888 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.888 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.888 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:45:54.888 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:43:51.108027+0000", 2024-08-17T17:45:54.888 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7034999999999999e-05 2024-08-17T17:45:54.888 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.888 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:45:54.889 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.889 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.889 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.889 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36612.0:96017 10.f e381/357 10:f2c035cb:::benchmark_data_smithi003_292749_object96016:head v 381'5950, mlcod=380'5947)", 2024-08-17T17:45:54.889 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:43:55.536566+0000", 2024-08-17T17:45:54.889 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 119.31456892200001, 2024-08-17T17:45:54.889 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.070865821999999995, 2024-08-17T17:45:54.889 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:45:54.889 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:45:54.889 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:45:54.889 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:45:54.889 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.889 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:45:54.890 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:43:55.536566+0000", 2024-08-17T17:45:54.890 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.890 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.890 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.890 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:45:54.890 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:43:55.536566+0000", 2024-08-17T17:45:54.890 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.890 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.890 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.890 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:45:54.890 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:43:55.536567+0000", 2024-08-17T17:45:54.890 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0999999999999999e-07 2024-08-17T17:45:54.890 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.890 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.891 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:45:54.891 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:43:55.536621+0000", 2024-08-17T17:45:54.891 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.435e-05 2024-08-17T17:45:54.891 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.891 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.891 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:45:54.891 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:43:55.593770+0000", 2024-08-17T17:45:54.891 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.057149459 2024-08-17T17:45:54.891 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.891 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.891 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:45:54.891 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:43:55.593787+0000", 2024-08-17T17:45:54.891 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6223e-05 2024-08-17T17:45:54.892 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.892 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.892 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:45:54.892 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:43:55.594554+0000", 2024-08-17T17:45:54.892 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00076705600000000001 2024-08-17T17:45:54.892 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.892 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.892 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:45:54.892 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:43:55.594568+0000", 2024-08-17T17:45:54.892 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4676e-05 2024-08-17T17:45:54.892 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.892 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.892 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:45:54.892 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:43:55.607345+0000", 2024-08-17T17:45:54.892 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012776973 2024-08-17T17:45:54.893 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.893 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.893 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:45:54.893 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:43:55.607432+0000", 2024-08-17T17:45:54.893 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6575000000000004e-05 2024-08-17T17:45:54.893 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.893 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:45:54.893 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.893 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.893 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.893 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36612.0:112444 10.8 e383/357 10:1b67f91e:::benchmark_data_smithi003_292749_object112443:head v 383'6937, mlcod=383'6934)", 2024-08-17T17:45:54.893 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:44:13.264327+0000", 2024-08-17T17:45:54.893 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 101.586807991, 2024-08-17T17:45:54.893 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.034937735999999997, 2024-08-17T17:45:54.894 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:45:54.894 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:45:54.894 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:45:54.894 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:45:54.894 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.894 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:45:54.894 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:13.264327+0000", 2024-08-17T17:45:54.894 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.894 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.894 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.894 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:45:54.894 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:13.264327+0000", 2024-08-17T17:45:54.894 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.894 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.894 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.895 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:45:54.895 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:13.264327+0000", 2024-08-17T17:45:54.895 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.34e-07 2024-08-17T17:45:54.895 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.895 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.895 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:45:54.895 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:13.264350+0000", 2024-08-17T17:45:54.895 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2762999999999999e-05 2024-08-17T17:45:54.895 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.895 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.895 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:45:54.895 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:13.264357+0000", 2024-08-17T17:45:54.895 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4320000000000002e-06 2024-08-17T17:45:54.895 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.896 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.896 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:45:54.896 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:13.264364+0000", 2024-08-17T17:45:54.896 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2979999999999997e-06 2024-08-17T17:45:54.896 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.896 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.896 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:45:54.896 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:13.272082+0000", 2024-08-17T17:45:54.896 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0077181080000000004 2024-08-17T17:45:54.896 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.896 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.896 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:45:54.896 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:13.272363+0000", 2024-08-17T17:45:54.896 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00028091999999999999 2024-08-17T17:45:54.897 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.897 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.897 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:45:54.897 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:13.299094+0000", 2024-08-17T17:45:54.897 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.026731511999999999 2024-08-17T17:45:54.897 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.897 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.897 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:45:54.897 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:13.299265+0000", 2024-08-17T17:45:54.897 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000170469 2024-08-17T17:45:54.897 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.897 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:45:54.898 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.898 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.898 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.898 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36612.0:116023 10.f e384/357 10:fe99c120:::benchmark_data_smithi003_292749_object116022:head v 384'7190, mlcod=384'7188)", 2024-08-17T17:45:54.898 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:44:16.967664+0000", 2024-08-17T17:45:54.898 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 97.883470509000006, 2024-08-17T17:45:54.898 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.039208822999999997, 2024-08-17T17:45:54.898 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:45:54.898 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:45:54.898 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:45:54.898 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:45:54.898 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.898 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:45:54.898 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.967664+0000", 2024-08-17T17:45:54.899 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.899 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.899 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.899 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:45:54.899 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.967664+0000", 2024-08-17T17:45:54.899 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.899 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.899 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.899 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:45:54.899 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.967665+0000", 2024-08-17T17:45:54.900 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3299999999999998e-07 2024-08-17T17:45:54.900 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.900 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.900 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:45:54.900 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.967701+0000", 2024-08-17T17:45:54.900 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5830000000000001e-05 2024-08-17T17:45:54.900 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.901 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.901 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:45:54.901 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.967708+0000", 2024-08-17T17:45:54.901 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5490000000000001e-06 2024-08-17T17:45:54.901 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.901 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.901 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:45:54.901 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.967715+0000", 2024-08-17T17:45:54.901 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9099999999999999e-06 2024-08-17T17:45:54.901 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.901 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.901 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:45:54.901 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.968364+0000", 2024-08-17T17:45:54.902 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00064860000000000004 2024-08-17T17:45:54.902 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.902 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.902 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:45:54.902 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.968415+0000", 2024-08-17T17:45:54.902 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0890999999999998e-05 2024-08-17T17:45:54.902 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.902 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.902 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:45:54.902 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:17.006653+0000", 2024-08-17T17:45:54.902 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.038238286000000003 2024-08-17T17:45:54.902 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.902 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.902 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:45:54.903 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:17.006873+0000", 2024-08-17T17:45:54.903 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00022042399999999999 2024-08-17T17:45:54.903 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.903 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:45:54.903 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.903 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.903 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.903 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36612.0:116024 10.f e384/357 10:fb99b45b:::benchmark_data_smithi003_292749_object116023:head v 384'7191, mlcod=384'7188)", 2024-08-17T17:45:54.903 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:44:16.968142+0000", 2024-08-17T17:45:54.903 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 97.882992637000001, 2024-08-17T17:45:54.903 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.038927207999999998, 2024-08-17T17:45:54.903 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:45:54.903 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:45:54.904 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:45:54.904 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:45:54.904 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.904 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:45:54.904 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.968142+0000", 2024-08-17T17:45:54.904 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.904 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.904 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.904 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:45:54.904 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.968142+0000", 2024-08-17T17:45:54.904 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.904 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.904 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.904 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:45:54.904 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.968143+0000", 2024-08-17T17:45:54.905 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5700000000000002e-07 2024-08-17T17:45:54.905 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.905 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.905 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:45:54.905 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.968185+0000", 2024-08-17T17:45:54.905 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1995e-05 2024-08-17T17:45:54.905 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.905 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.905 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:45:54.906 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.968199+0000", 2024-08-17T17:45:54.906 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4015999999999999e-05 2024-08-17T17:45:54.906 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.906 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.906 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:45:54.906 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.968205+0000", 2024-08-17T17:45:54.906 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1e-06 2024-08-17T17:45:54.906 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.906 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.907 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:45:54.907 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.974126+0000", 2024-08-17T17:45:54.907 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0059213970000000001 2024-08-17T17:45:54.907 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.907 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.907 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:45:54.907 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.974186+0000", 2024-08-17T17:45:54.907 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9215000000000001e-05 2024-08-17T17:45:54.907 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.907 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.907 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:45:54.907 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:17.006879+0000", 2024-08-17T17:45:54.907 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.032693246000000002 2024-08-17T17:45:54.907 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.908 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.908 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:45:54.908 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:17.007070+0000", 2024-08-17T17:45:54.908 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019068199999999999 2024-08-17T17:45:54.908 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.908 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:45:54.908 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.908 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.908 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.908 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36612.0:116025 10.f e384/357 10:fde4dded:::benchmark_data_smithi003_292749_object116024:head v 384'7192, mlcod=384'7188)", 2024-08-17T17:45:54.908 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:44:16.970937+0000", 2024-08-17T17:45:54.908 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 97.880197690000003, 2024-08-17T17:45:54.908 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.036336733000000003, 2024-08-17T17:45:54.908 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:45:54.908 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:45:54.909 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:45:54.909 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:45:54.909 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.909 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:45:54.909 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.970937+0000", 2024-08-17T17:45:54.909 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.909 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.909 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.909 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:45:54.909 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.970937+0000", 2024-08-17T17:45:54.909 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.909 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.909 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.909 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:45:54.909 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.970938+0000", 2024-08-17T17:45:54.910 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0559999999999999e-06 2024-08-17T17:45:54.910 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.910 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.910 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:45:54.910 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.971007+0000", 2024-08-17T17:45:54.910 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9031999999999999e-05 2024-08-17T17:45:54.910 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.910 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.910 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:45:54.910 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.971022+0000", 2024-08-17T17:45:54.910 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4996e-05 2024-08-17T17:45:54.910 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.910 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.911 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:45:54.911 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.971031+0000", 2024-08-17T17:45:54.911 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2500000000000006e-06 2024-08-17T17:45:54.911 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.911 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.911 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:45:54.911 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.978170+0000", 2024-08-17T17:45:54.911 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0071390079999999996 2024-08-17T17:45:54.911 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.911 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.911 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:45:54.911 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.978192+0000", 2024-08-17T17:45:54.911 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2001000000000002e-05 2024-08-17T17:45:54.911 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.911 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.912 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:45:54.912 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:17.007073+0000", 2024-08-17T17:45:54.912 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.028881459000000002 2024-08-17T17:45:54.912 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.912 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.912 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:45:54.912 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:17.007274+0000", 2024-08-17T17:45:54.912 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000200931 2024-08-17T17:45:54.912 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.912 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:45:54.912 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.912 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.912 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.912 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36612.0:116031 10.f e384/357 10:f21aec4e:::benchmark_data_smithi003_292749_object116030:head v 384'7193, mlcod=384'7188)", 2024-08-17T17:45:54.912 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:44:16.973665+0000", 2024-08-17T17:45:54.913 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 97.877470375000001, 2024-08-17T17:45:54.913 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.033728018999999998, 2024-08-17T17:45:54.913 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:45:54.913 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:45:54.913 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:45:54.913 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:45:54.913 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.913 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:45:54.913 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.973665+0000", 2024-08-17T17:45:54.913 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.913 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.913 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.913 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:45:54.913 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.973665+0000", 2024-08-17T17:45:54.913 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.914 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.914 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.914 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:45:54.914 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.973665+0000", 2024-08-17T17:45:54.914 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.09e-07 2024-08-17T17:45:54.914 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.914 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.914 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:45:54.914 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.973740+0000", 2024-08-17T17:45:54.914 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4744000000000001e-05 2024-08-17T17:45:54.914 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.914 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.914 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:45:54.915 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.973759+0000", 2024-08-17T17:45:54.915 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8947e-05 2024-08-17T17:45:54.915 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.915 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.915 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:45:54.915 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.973767+0000", 2024-08-17T17:45:54.915 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8439999999999998e-06 2024-08-17T17:45:54.915 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.915 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.915 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:45:54.915 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.981767+0000", 2024-08-17T17:45:54.915 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0079999119999999996 2024-08-17T17:45:54.915 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.916 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.916 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:45:54.916 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.981788+0000", 2024-08-17T17:45:54.916 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1452000000000001e-05 2024-08-17T17:45:54.916 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.916 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.916 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:45:54.916 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:17.007280+0000", 2024-08-17T17:45:54.916 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.025491796000000001 2024-08-17T17:45:54.916 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.916 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.916 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:45:54.916 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:17.007393+0000", 2024-08-17T17:45:54.917 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000112415 2024-08-17T17:45:54.917 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.917 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:45:54.917 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.917 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.917 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.917 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36612.0:116035 10.f e384/357 10:f704a67c:::benchmark_data_smithi003_292749_object116034:head v 384'7194, mlcod=384'7188)", 2024-08-17T17:45:54.917 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:44:16.975584+0000", 2024-08-17T17:45:54.917 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 97.875551048000005, 2024-08-17T17:45:54.917 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.031903954999999998, 2024-08-17T17:45:54.917 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:45:54.917 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:45:54.918 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:45:54.918 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:45:54.918 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.918 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:45:54.918 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.975584+0000", 2024-08-17T17:45:54.918 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.918 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.918 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.918 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:45:54.918 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.975584+0000", 2024-08-17T17:45:54.918 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.918 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.918 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.919 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:45:54.919 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.975585+0000", 2024-08-17T17:45:54.919 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9100000000000002e-07 2024-08-17T17:45:54.919 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.919 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.919 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:45:54.919 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.975667+0000", 2024-08-17T17:45:54.919 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2003999999999996e-05 2024-08-17T17:45:54.919 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.919 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.919 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:45:54.919 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.975681+0000", 2024-08-17T17:45:54.919 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4293e-05 2024-08-17T17:45:54.919 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.920 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.920 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:45:54.920 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.975688+0000", 2024-08-17T17:45:54.920 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5119999999999997e-06 2024-08-17T17:45:54.920 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.920 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.920 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:45:54.920 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.985271+0000", 2024-08-17T17:45:54.920 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0095838820000000002 2024-08-17T17:45:54.920 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.921 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.921 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:45:54.921 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:16.985297+0000", 2024-08-17T17:45:54.921 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5853000000000001e-05 2024-08-17T17:45:54.921 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.921 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.921 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:45:54.921 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:17.007395+0000", 2024-08-17T17:45:54.921 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.022097327 2024-08-17T17:45:54.921 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.921 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.921 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:45:54.921 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:17.007488+0000", 2024-08-17T17:45:54.922 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3193e-05 2024-08-17T17:45:54.922 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.922 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:45:54.922 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.922 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.922 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.922 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36612.0:147279 10.5 e387/357 10:a7e6adb5:::benchmark_data_smithi003_292749_object28733:head v 387'9164, mlcod=387'9159)", 2024-08-17T17:45:54.922 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:44:50.577633+0000", 2024-08-17T17:45:54.922 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 64.273501956000004, 2024-08-17T17:45:54.922 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.032722017999999999, 2024-08-17T17:45:54.922 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:45:54.922 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:45:54.922 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:45:54.922 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:45:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:45:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:50.577633+0000", 2024-08-17T17:45:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:45:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:50.577633+0000", 2024-08-17T17:45:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:45:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:50.577634+0000", 2024-08-17T17:45:54.923 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7899999999999997e-07 2024-08-17T17:45:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:45:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:50.577640+0000", 2024-08-17T17:45:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9009999999999999e-06 2024-08-17T17:45:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:45:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:50.577656+0000", 2024-08-17T17:45:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6396e-05 2024-08-17T17:45:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.924 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:45:54.925 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:50.577664+0000", 2024-08-17T17:45:54.925 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2360000000000004e-06 2024-08-17T17:45:54.925 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.925 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.925 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:45:54.925 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:50.587090+0000", 2024-08-17T17:45:54.925 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0094251460000000006 2024-08-17T17:45:54.925 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.925 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.925 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:45:54.925 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:50.587109+0000", 2024-08-17T17:45:54.925 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8972999999999999e-05 2024-08-17T17:45:54.925 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:45:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:50.610149+0000", 2024-08-17T17:45:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.023040323000000001 2024-08-17T17:45:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:45:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:44:50.610355+0000", 2024-08-17T17:45:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000206164 2024-08-17T17:45:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:45:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.926 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36612.0:162864 10.e e389/357 10:7f048e0a:::benchmark_data_smithi003_292749_object44318:head v 389'10247, mlcod=389'10245)", 2024-08-17T17:45:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:45:06.198364+0000", 2024-08-17T17:45:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 48.652771055999999, 2024-08-17T17:45:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.055109177000000002, 2024-08-17T17:45:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:45:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:45:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:45:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:45:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:45:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:06.198364+0000", 2024-08-17T17:45:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:45:54.927 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:06.198364+0000", 2024-08-17T17:45:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:45:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:06.198365+0000", 2024-08-17T17:45:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0500000000000003e-07 2024-08-17T17:45:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:45:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:06.198370+0000", 2024-08-17T17:45:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7910000000000001e-06 2024-08-17T17:45:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:45:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:06.198380+0000", 2024-08-17T17:45:54.928 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0030000000000001e-05 2024-08-17T17:45:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:45:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:06.198387+0000", 2024-08-17T17:45:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2330000000000003e-06 2024-08-17T17:45:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:45:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:06.198440+0000", 2024-08-17T17:45:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3825000000000001e-05 2024-08-17T17:45:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:45:54.929 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:06.198459+0000", 2024-08-17T17:45:54.930 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8233e-05 2024-08-17T17:45:54.930 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.930 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.930 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:45:54.930 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:06.201078+0000", 2024-08-17T17:45:54.930 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002619031 2024-08-17T17:45:54.930 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.930 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.930 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:45:54.930 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:06.253473+0000", 2024-08-17T17:45:54.930 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.052395328999999997 2024-08-17T17:45:54.930 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.930 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:45:54.930 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.931 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.931 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.931 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36612.0:162861 10.3 e389/357 10:c19becba:::benchmark_data_smithi003_292749_object44315:head v 389'10273, mlcod=389'10269)", 2024-08-17T17:45:54.931 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:45:06.201182+0000", 2024-08-17T17:45:54.931 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 48.649952493999997, 2024-08-17T17:45:54.931 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.055855744999999998, 2024-08-17T17:45:54.931 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:45:54.931 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:45:54.931 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:45:54.931 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:45:54.931 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.931 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:45:54.931 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:06.201182+0000", 2024-08-17T17:45:54.932 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.932 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.932 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.932 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:45:54.932 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:06.201182+0000", 2024-08-17T17:45:54.932 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.932 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.932 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.932 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:45:54.932 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:06.201183+0000", 2024-08-17T17:45:54.932 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3500000000000005e-07 2024-08-17T17:45:54.932 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.932 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.933 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:45:54.933 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:06.201189+0000", 2024-08-17T17:45:54.933 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5369999999999997e-06 2024-08-17T17:45:54.933 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.933 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.933 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:45:54.933 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:06.253426+0000", 2024-08-17T17:45:54.933 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.052237231000000002 2024-08-17T17:45:54.933 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.933 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.933 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:45:54.933 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:06.253441+0000", 2024-08-17T17:45:54.933 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4968e-05 2024-08-17T17:45:54.934 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.934 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.934 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:45:54.934 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:06.253530+0000", 2024-08-17T17:45:54.934 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8938999999999997e-05 2024-08-17T17:45:54.934 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.934 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.934 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:45:54.934 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:06.253552+0000", 2024-08-17T17:45:54.934 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1736e-05 2024-08-17T17:45:54.934 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.934 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.934 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:45:54.934 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:06.256954+0000", 2024-08-17T17:45:54.934 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034019190000000002 2024-08-17T17:45:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:45:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:06.257038+0000", 2024-08-17T17:45:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4580000000000007e-05 2024-08-17T17:45:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:45:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36612.0:162870 10.e e389/357 10:735439f6:::benchmark_data_smithi003_292749_object44324:head v 389'10248, mlcod=389'10245)", 2024-08-17T17:45:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:45:06.203410+0000", 2024-08-17T17:45:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 48.647725012999999, 2024-08-17T17:45:54.935 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.053775679, 2024-08-17T17:45:54.936 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:45:54.936 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:45:54.936 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:45:54.936 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:45:54.936 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.936 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:45:54.936 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:06.203410+0000", 2024-08-17T17:45:54.936 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.936 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.936 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.936 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:45:54.936 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:06.203410+0000", 2024-08-17T17:45:54.936 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.936 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.937 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.937 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:45:54.937 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:06.203411+0000", 2024-08-17T17:45:54.937 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1299999999999998e-07 2024-08-17T17:45:54.937 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.937 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.937 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:45:54.937 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:06.203418+0000", 2024-08-17T17:45:54.937 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5560000000000002e-06 2024-08-17T17:45:54.937 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.937 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.937 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:45:54.937 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:06.253883+0000", 2024-08-17T17:45:54.938 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.050464330000000002 2024-08-17T17:45:54.938 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.938 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.938 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:45:54.938 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:06.253892+0000", 2024-08-17T17:45:54.938 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1519999999999998e-06 2024-08-17T17:45:54.938 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.938 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.938 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:45:54.938 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:06.253948+0000", 2024-08-17T17:45:54.938 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5974999999999999e-05 2024-08-17T17:45:54.938 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.938 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.938 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:45:54.939 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:06.253967+0000", 2024-08-17T17:45:54.939 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9255000000000001e-05 2024-08-17T17:45:54.939 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.939 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.939 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:45:54.939 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:06.256999+0000", 2024-08-17T17:45:54.939 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0030320199999999999 2024-08-17T17:45:54.939 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.939 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.939 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:45:54.939 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:06.257186+0000", 2024-08-17T17:45:54.939 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018647800000000001 2024-08-17T17:45:54.939 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.939 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:45:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36612.0:168245 10.3 e389/357 10:cc7ca8ff:::benchmark_data_smithi003_292749_object49699:head v 389'10624, mlcod=389'10621)", 2024-08-17T17:45:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:45:11.630920+0000", 2024-08-17T17:45:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 43.220214970000001, 2024-08-17T17:45:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.033589358999999999, 2024-08-17T17:45:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:45:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:45:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:45:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:45:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:45:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:11.630920+0000", 2024-08-17T17:45:54.940 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:45:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:11.630920+0000", 2024-08-17T17:45:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:45:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:11.630921+0000", 2024-08-17T17:45:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3099999999999997e-07 2024-08-17T17:45:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:45:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:11.630926+0000", 2024-08-17T17:45:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4489999999999998e-06 2024-08-17T17:45:54.941 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:45:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:11.630937+0000", 2024-08-17T17:45:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0695e-05 2024-08-17T17:45:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:45:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:11.630944+0000", 2024-08-17T17:45:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6839999999999999e-06 2024-08-17T17:45:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:45:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:11.636346+0000", 2024-08-17T17:45:54.942 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0054023730000000002 2024-08-17T17:45:54.943 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.943 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.943 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:45:54.943 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:11.636380+0000", 2024-08-17T17:45:54.943 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4226000000000001e-05 2024-08-17T17:45:54.943 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.943 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.943 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:45:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:11.664394+0000", 2024-08-17T17:45:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.028013639999999999 2024-08-17T17:45:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:45:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:11.664509+0000", 2024-08-17T17:45:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000115561 2024-08-17T17:45:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:45:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.944 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.36612.0:205145 10.f e391/357 10:fcbf8884:::benchmark_data_smithi003_292749_object86599:head v 391'12757, mlcod=391'12753)", 2024-08-17T17:45:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-08-17T17:45:48.214214+0000", 2024-08-17T17:45:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 6.6369205889999998, 2024-08-17T17:45:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.036890470000000002, 2024-08-17T17:45:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-08-17T17:45:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-08-17T17:45:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:45:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-08-17T17:45:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-08-17T17:45:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:48.214214+0000", 2024-08-17T17:45:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-08-17T17:45:54.945 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:48.214214+0000", 2024-08-17T17:45:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-08-17T17:45:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-08-17T17:45:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:48.214215+0000", 2024-08-17T17:45:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2399999999999996e-07 2024-08-17T17:45:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-08-17T17:45:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:48.214222+0000", 2024-08-17T17:45:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3829999999999997e-06 2024-08-17T17:45:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.946 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-08-17T17:45:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:48.214235+0000", 2024-08-17T17:45:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3192e-05 2024-08-17T17:45:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-08-17T17:45:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:48.214241+0000", 2024-08-17T17:45:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5400000000000001e-06 2024-08-17T17:45:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-08-17T17:45:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:48.221414+0000", 2024-08-17T17:45:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0071723309999999997 2024-08-17T17:45:54.947 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-08-17T17:45:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:48.221442+0000", 2024-08-17T17:45:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7832999999999999e-05 2024-08-17T17:45:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-08-17T17:45:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:48.250914+0000", 2024-08-17T17:45:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.029472206000000001 2024-08-17T17:45:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-08-17T17:45:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-08-17T17:45:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-08-17T17:45:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-08-17T17:45:48.251105+0000", 2024-08-17T17:45:54.948 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019106099999999999 2024-08-17T17:45:54.949 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.949 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:45:54.949 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.949 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-08-17T17:45:54.949 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-08-17T17:45:54.949 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:45:54.954 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:55.003 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:55.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:55.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:55.105 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:55.145 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:55.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:55.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:55.247 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:55.284 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:55.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:55.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:55.385 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:55.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:55.424 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:55.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:55.525 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:55.529 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:45:55.557 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:55.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:55.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:55.568 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:45:55.568 INFO:teuthology.orchestra.run.smithi003.stdout:{"fsid":"2b5b662e-5cba-11ef-bccf-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2948/80607 objects degraded (3.657%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1897,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":392,"num_osds":8,"num_up_osds":8,"osd_up_since":1723915999,"num_in_osds":8,"osd_in_since":1723916493,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":118},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":119,"num_pools":5,"num_objects":26869,"data_bytes":1763776485,"bytes_used":12198404096,"bytes_avail":755761881088,"bytes_total":767960285184,"degraded_objects":2948,"degraded_total":80607,"degraded_ratio":0.036572506109891198,"misplaced_objects":82,"misplaced_total":80607,"misplaced_ratio":0.0010172813775478557,"recovering_objects_per_sec":36,"recovering_bytes_per_sec":2423594,"recovering_keys_per_sec":0,"num_objects_recovered":370,"num_bytes_recovered":24248320,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":1080},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi003.civgvf","status":"up:active","gid":14664}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.3:8443/","prometheus":"http://172.21.15.3:9283/"}},"servicemap":{"epoch":30,"modified":"2024-08-17T17:45:18.413636+0000","services":{}},"progress_events":{"a4a400b3-2a31-493f-9d26-837737f27ab1":{"message":"Global Recovery Event (0s)\n [............................] ","progress":0,"add_to_ceph_s":true},"fcbeb9b7-7baf-488b-9fe4-c01aca441f9c":{"message":"Upgrade to 19.0.0-5836-gcd8bdbeb (4m)\n [============................] (remaining: 5m)","progress":0.44999998807907104,"add_to_ceph_s":true}}} 2024-08-17T17:45:55.659 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:55.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:55.702 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:55.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:55.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:55 smithi003 ceph-mon[103344]: pgmap v1303: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 2948/80607 objects degraded (3.657%); 82/80607 objects misplaced (0.102%); 2.3 MiB/s, 36 objects/s recovering 2024-08-17T17:45:55.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:55 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 2948/80607 objects degraded (3.657%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:55.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:55 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2420776069' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T17:45:55.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:55 smithi003 ceph-mon[100615]: pgmap v1303: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 2948/80607 objects degraded (3.657%); 82/80607 objects misplaced (0.102%); 2.3 MiB/s, 36 objects/s recovering 2024-08-17T17:45:55.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:55 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 2948/80607 objects degraded (3.657%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:55.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:55 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2420776069' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T17:45:55.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:45:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:45:55] "GET /metrics HTTP/1.1" 200 36576 "" "Prometheus/2.51.0" 2024-08-17T17:45:55.804 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:55.850 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:55.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:55.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:55.953 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:55.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:55 smithi053 ceph-mon[89963]: pgmap v1303: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 2948/80607 objects degraded (3.657%); 82/80607 objects misplaced (0.102%); 2.3 MiB/s, 36 objects/s recovering 2024-08-17T17:45:55.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:55 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 2948/80607 objects degraded (3.657%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:45:55.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:55 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2420776069' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T17:45:55.998 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:55.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:55.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:56.100 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:56.140 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:56.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:56.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:56.241 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:56.283 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:56.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:56.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:56.385 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:56.413 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:56.414 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-08-17T17:45:56.414 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:45:56.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:56.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:56.516 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:56.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:56.591 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:56.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:56.691 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:56.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:56.762 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:56.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:56.858 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:56 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:45:56.858 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:56 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:45:56.864 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:45:56.873 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:45:56.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:56.902 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:56.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:45:56.916 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-17T17:45:56.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:56 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:45:57.003 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:57.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:57.063 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:57.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:57.164 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:57.231 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:57.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:57.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:57.332 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:57.392 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:57.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:57.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:57.493 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:57.535 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:57.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:57.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:57.637 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:57.671 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:57.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:57.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:57.772 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:57.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:57 smithi003 ceph-mon[103344]: pgmap v1304: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 960 op/s; 2948/78714 objects degraded (3.745%); 82/78714 objects misplaced (0.104%); 2.3 MiB/s, 36 objects/s recovering 2024-08-17T17:45:57.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:57 smithi003 ceph-mon[100615]: pgmap v1304: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 960 op/s; 2948/78714 objects degraded (3.745%); 82/78714 objects misplaced (0.104%); 2.3 MiB/s, 36 objects/s recovering 2024-08-17T17:45:57.820 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:57.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:57.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:57.922 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:57.952 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:57.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:57.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:57.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:57 smithi053 ceph-mon[89963]: pgmap v1304: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 960 op/s; 2948/78714 objects degraded (3.745%); 82/78714 objects misplaced (0.104%); 2.3 MiB/s, 36 objects/s recovering 2024-08-17T17:45:58.054 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:58.098 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:58.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:58.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:58.200 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:58.239 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:58.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:58.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:58.341 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:58.376 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:58.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:58.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:58.478 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:45:58.513 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:58.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:58.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:45:58.615 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:45:58.659 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:58.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:58.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:45:58.762 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:45:58.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:58.821 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:58.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:45:58.922 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:58.956 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:58.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:58.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:59.058 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:59.114 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:59.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:59.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:59.216 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:59.252 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:59.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:59.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:59.354 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:45:59.387 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:59.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:59.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:45:59.489 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:45:59.512 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:45:59.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:59.567 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:45:59.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:45:59.668 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:45:59.678 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:45:59 smithi053 ceph-mon[89963]: pgmap v1305: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 2948/75012 objects degraded (3.930%); 82/75012 objects misplaced (0.109%); 2.3 MiB/s, 36 objects/s recovering 2024-08-17T17:45:59.709 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:59.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:59.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:45:59.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:45:59 smithi003 ceph-mon[103344]: pgmap v1305: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 2948/75012 objects degraded (3.930%); 82/75012 objects misplaced (0.109%); 2.3 MiB/s, 36 objects/s recovering 2024-08-17T17:45:59.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:45:59 smithi003 ceph-mon[100615]: pgmap v1305: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 2948/75012 objects degraded (3.930%); 82/75012 objects misplaced (0.109%); 2.3 MiB/s, 36 objects/s recovering 2024-08-17T17:45:59.811 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:45:59.855 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:45:59.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:45:59.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:45:59.957 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:00.006 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:00.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:00.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:00.108 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:00.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:00.150 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:00.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:00.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:00.316 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:00.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:00.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:00.418 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:00.455 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:00.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:00.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:00.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:46:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:46:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:46:00.558 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:00.617 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:00.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:00.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:00.719 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:00.757 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:00.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:00.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:00.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:00 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 2948/75012 objects degraded (3.930%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:46:00.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:00 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 2948/75012 objects degraded (3.930%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:46:00.859 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:00.897 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:00.898 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:46:00.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:00.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:00.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:00 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 2948/75012 objects degraded (3.930%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:46:01.001 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:01.049 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:01.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:01.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:01.151 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:01.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:01.191 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:01.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:01.292 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:01.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:01.346 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:01.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:01.447 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:01.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:01.483 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:01.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:01.584 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:01.631 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:01.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:01.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:01.712 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-08-17T17:46:01.733 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:01.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:01 smithi003 ceph-mon[103344]: pgmap v1306: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s; 2789/65160 objects degraded (4.280%); 2.3 MiB/s, 51 objects/s recovering 2024-08-17T17:46:01.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:01 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/953311358' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:46:01.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:01 smithi003 ceph-mon[100615]: pgmap v1306: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s; 2789/65160 objects degraded (4.280%); 2.3 MiB/s, 51 objects/s recovering 2024-08-17T17:46:01.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:01 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/953311358' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:46:01.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:01.782 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:01.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:01.883 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:01.925 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:01.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:01.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:01.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:01 smithi053 ceph-mon[89963]: pgmap v1306: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s; 2789/65160 objects degraded (4.280%); 2.3 MiB/s, 51 objects/s recovering 2024-08-17T17:46:01.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:01 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/953311358' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:46:02.027 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:02.095 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:02.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:02.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:02.168 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:46:02.168 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:46:02.168 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:46:02.168 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:46:02.168 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:46:02.168 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:46:02.168 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:46:02.168 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:46:02.169 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:46:02.169 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:46:02.169 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:46:02.169 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:46:02.169 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:46:02.169 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:46:02.169 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:46:02.169 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:46:02.197 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:02.229 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:02.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:02.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:02.331 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:02.360 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:02.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:02.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:02.462 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:02.494 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:02.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:02.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:02.596 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:02.630 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:02.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:02.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:02.732 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:02.773 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:02.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:02.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:02.875 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:02.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:02.915 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:02.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:02.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:02 smithi053 ceph-mon[89963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:46:02.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:02 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/953311358' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:46:02.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:02 smithi053 ceph-mon[89963]: osdmap e393: 8 total, 8 up, 8 in 2024-08-17T17:46:02.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:02 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/4175262909' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:46:02.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:02 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:46:02.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:02 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:46:03.016 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:03.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:02 smithi003 ceph-mon[103344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:46:03.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:02 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/953311358' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:46:03.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:02 smithi003 ceph-mon[103344]: osdmap e393: 8 total, 8 up, 8 in 2024-08-17T17:46:03.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:02 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/4175262909' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:46:03.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:02 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:46:03.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:02 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:46:03.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:02 smithi003 ceph-mon[100615]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:46:03.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:02 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/953311358' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:46:03.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:02 smithi003 ceph-mon[100615]: osdmap e393: 8 total, 8 up, 8 in 2024-08-17T17:46:03.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:02 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/4175262909' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:46:03.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:02 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:46:03.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:02 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:46:03.052 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:03.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:03.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:03.154 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:03.188 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:03.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:03.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:03.290 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:03.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:03.335 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:03.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:03.436 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:03.486 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:03.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:03.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:03.588 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:03.630 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:03.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:03.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:03.732 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:03.763 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:03.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:03.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:03.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:03.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:03.909 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:03.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:04.010 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:04.051 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:04.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:04.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:04.153 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:04.198 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:04.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:04.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:04.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:03 smithi053 ceph-mon[89963]: pgmap v1308: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 990 op/s; 2789/60684 objects degraded (4.596%); 1.5 MiB/s, 42 objects/s recovering 2024-08-17T17:46:04.272 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:03 smithi003 ceph-mon[100615]: pgmap v1308: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 990 op/s; 2789/60684 objects degraded (4.596%); 1.5 MiB/s, 42 objects/s recovering 2024-08-17T17:46:04.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:03 smithi003 ceph-mon[103344]: pgmap v1308: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 990 op/s; 2789/60684 objects degraded (4.596%); 1.5 MiB/s, 42 objects/s recovering 2024-08-17T17:46:04.299 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:04.332 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:04.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:04.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:04.433 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:04.469 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:04.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:04.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:04.571 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:04.612 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:04.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:04.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:04.714 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:04.739 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:04.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:04.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:04.841 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:04.864 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:04.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:04.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:04.966 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:04.993 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:04.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:04.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:05.095 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:05.133 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:05.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:05.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:05.235 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:05.268 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:05.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:05.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:05.293 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:46:05.370 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:05.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:05.417 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:05.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:05.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:05.559 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:05.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:05.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:05.665 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:05.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:05.701 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:05.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:05.802 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:05.857 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:05.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:05.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:05.916 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:46:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:46:05] "GET /metrics HTTP/1.1" 200 36576 "" "Prometheus/2.51.0" 2024-08-17T17:46:05.919 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:05 smithi003 ceph-mon[100615]: pgmap v1309: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.0 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 974 op/s; 2566/51363 objects degraded (4.996%); 384 KiB/s, 46 objects/s recovering 2024-08-17T17:46:05.919 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:05 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 2566/51363 objects degraded (4.996%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:46:05.959 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:05.995 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:05.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:05.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:06.097 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:06.134 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:06.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:06.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:06.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:05 smithi053 ceph-mon[89963]: pgmap v1309: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.0 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 974 op/s; 2566/51363 objects degraded (4.996%); 384 KiB/s, 46 objects/s recovering 2024-08-17T17:46:06.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:05 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 2566/51363 objects degraded (4.996%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:46:06.236 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:06.252 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:05 smithi003 ceph-mon[103344]: pgmap v1309: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.0 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 974 op/s; 2566/51363 objects degraded (4.996%); 384 KiB/s, 46 objects/s recovering 2024-08-17T17:46:06.253 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:05 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 2566/51363 objects degraded (4.996%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:46:06.271 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:06.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:06.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:06.372 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:06.409 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:06.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:06.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:06.511 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:06.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:06.547 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:06.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:06.648 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:06.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:06.691 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:06.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:06.791 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:06.827 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:06.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:06.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:06.929 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:06.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:06.970 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:06.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:06.972 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:46:07.072 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:07.107 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:07.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:07.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:07.209 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:07.251 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:07.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:07.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:07.353 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:07.383 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:07.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:07.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:07.485 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:07.525 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:07.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:07.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:07.627 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:07.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:07.666 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:07.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:07.767 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:07.803 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:07.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:07.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:07.904 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:07.946 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:07.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:07.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:08.047 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:08.087 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:08.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:08.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:08.094 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:07 smithi003 ceph-mon[103344]: pgmap v1310: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.0 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 971 op/s; 2566/49554 objects degraded (5.178%); 384 KiB/s, 46 objects/s recovering 2024-08-17T17:46:08.095 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:07 smithi003 ceph-mon[100615]: pgmap v1310: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.0 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 971 op/s; 2566/49554 objects degraded (5.178%); 384 KiB/s, 46 objects/s recovering 2024-08-17T17:46:08.189 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:08.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:07 smithi053 ceph-mon[89963]: pgmap v1310: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 1.0 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 971 op/s; 2566/49554 objects degraded (5.178%); 384 KiB/s, 46 objects/s recovering 2024-08-17T17:46:08.226 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:08.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:08.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:08.328 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:08.365 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:08.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:08.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:08.467 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:08.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:08.508 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:08.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:08.609 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:08.646 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:08.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:08.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:08.713 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set noscrub 2024-08-17T17:46:08.748 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:08.806 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:08.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:08.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:08.908 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:08.938 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:46:08.940 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:46:08.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:08.942 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:08.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:09.043 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:09.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:09.093 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:09.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:09.194 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:09.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:09.231 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:09.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:09.332 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:09.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:09.380 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:09.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:09.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:46:09.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:46:09.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:09 smithi003 ceph-mon[100615]: pgmap v1311: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 955 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 976 op/s; 2566/45693 objects degraded (5.616%); 384 KiB/s, 46 objects/s recovering 2024-08-17T17:46:09.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:09 smithi003 ceph-mon[100615]: from='client.46781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:46:09.451 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:46:09.451 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:46:09.452 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:09 smithi003 ceph-mon[103344]: pgmap v1311: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 955 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 976 op/s; 2566/45693 objects degraded (5.616%); 384 KiB/s, 46 objects/s recovering 2024-08-17T17:46:09.452 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:09 smithi003 ceph-mon[103344]: from='client.46781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:46:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:46:09.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:46:09.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:09 smithi053 ceph-mon[89963]: pgmap v1311: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 955 MiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 976 op/s; 2566/45693 objects degraded (5.616%); 384 KiB/s, 46 objects/s recovering 2024-08-17T17:46:09.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:09 smithi053 ceph-mon[89963]: from='client.46781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:46:09.481 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:09.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:09.523 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:09.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:09.624 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:09.656 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:09.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:09.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:09.758 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:09.790 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:09.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:09.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:09.892 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:09.938 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:09.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:09.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:10.040 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:10.083 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:10.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:10.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:10.185 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:10.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:10 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 2566/45693 objects degraded (5.616%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:46:10.224 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:10.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:10.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:10.326 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:10.363 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:10.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:10.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:10.464 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:10.489 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:10 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 2566/45693 objects degraded (5.616%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:46:10.489 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:10 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 2566/45693 objects degraded (5.616%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:46:10.502 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:10.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:10.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:10.604 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:10.636 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:10.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:10.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:10.716 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:46:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:46:10.219Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:46:10.739 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:10.769 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:10.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:10.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:10.871 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:10.902 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:10.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:10.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:10.910 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1311,"stamp":"2024-08-17T17:46:08.431742+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1001068517,"num_objects":15231,"num_object_clones":0,"num_object_copies":45693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2566,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15231,"num_whiteouts":0,"num_read":5979,"num_read_kb":12992,"num_write":222792,"num_write_kb":7599376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4428,"num_bytes_recovered":259402744,"num_keys_recovered":40,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":108036,"ondisk_log_size":108036,"up":357,"acting":356,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":359,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":9591768,"kb_used_data":3460436,"kb_used_omap":37,"kb_used_meta":6131162,"kb_avail":740369448,"statfs":{"total":767960285184,"available":758138314752,"internally_reserved":0,"allocated":3543486464,"data_stored":3525664639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38729,"internal_metadata":6278310071},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,7,6,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-640483328,"num_objects":-9773,"num_object_clones":0,"num_object_copies":-29319,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-382,"num_objects_misplaced":-82,"num_objects_unfound":0,"num_objects_dirty":-9773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9773,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":464,"num_bytes_recovered":3932160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003548"},"pg_stats":[{"pgid":"1.11","version":"90'149","reported_seq":838,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.734617+0000","last_change":"2024-08-17T17:38:30.378664+0000","last_active":"2024-08-17T17:46:01.734617+0000","last_peered":"2024-08-17T17:46:01.734617+0000","last_clean":"2024-08-17T17:46:01.734617+0000","last_became_active":"2024-08-17T17:38:30.342287+0000","last_became_peered":"2024-08-17T17:38:30.342287+0000","last_unstale":"2024-08-17T17:46:01.734617+0000","last_undegraded":"2024-08-17T17:46:01.734617+0000","last_fullsized":"2024-08-17T17:46:01.734617+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:30.378592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:30.378592+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:04:28.933354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":659,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.724535+0000","last_change":"2024-08-17T17:33:20.797029+0000","last_active":"2024-08-17T17:46:01.724535+0000","last_peered":"2024-08-17T17:46:01.724535+0000","last_clean":"2024-08-17T17:46:01.724535+0000","last_became_active":"2024-08-17T17:33:20.796679+0000","last_became_peered":"2024-08-17T17:33:20.796679+0000","last_unstale":"2024-08-17T17:46:01.724535+0000","last_undegraded":"2024-08-17T17:46:01.724535+0000","last_fullsized":"2024-08-17T17:46:01.724535+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:51.438262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":417,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.634699+0000","last_change":"2024-08-17T17:41:34.624198+0000","last_active":"2024-08-17T17:45:48.634699+0000","last_peered":"2024-08-17T17:45:48.634699+0000","last_clean":"2024-08-17T17:45:48.634699+0000","last_became_active":"2024-08-17T17:41:34.616071+0000","last_became_peered":"2024-08-17T17:41:34.616071+0000","last_unstale":"2024-08-17T17:45:48.634699+0000","last_undegraded":"2024-08-17T17:45:48.634699+0000","last_fullsized":"2024-08-17T17:45:48.634699+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:34:48.805354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"90'149","reported_seq":476,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.645548+0000","last_change":"2024-08-17T17:38:31.874623+0000","last_active":"2024-08-17T17:45:48.645548+0000","last_peered":"2024-08-17T17:45:48.645548+0000","last_clean":"2024-08-17T17:45:48.645548+0000","last_became_active":"2024-08-17T17:38:30.339313+0000","last_became_peered":"2024-08-17T17:38:30.339313+0000","last_unstale":"2024-08-17T17:45:48.645548+0000","last_undegraded":"2024-08-17T17:45:48.645548+0000","last_fullsized":"2024-08-17T17:45:48.645548+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:31.874423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:31.874423+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:28:17.980339+0000","scrub_duration":0.0017696579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":320,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.724425+0000","last_change":"2024-08-17T17:33:44.156036+0000","last_active":"2024-08-17T17:46:01.724425+0000","last_peered":"2024-08-17T17:46:01.724425+0000","last_clean":"2024-08-17T17:46:01.724425+0000","last_became_active":"2024-08-17T17:33:44.155758+0000","last_became_peered":"2024-08-17T17:33:44.155758+0000","last_unstale":"2024-08-17T17:46:01.724425+0000","last_undegraded":"2024-08-17T17:46:01.724425+0000","last_fullsized":"2024-08-17T17:46:01.724425+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:23.443405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":459,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.734498+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:46:01.734498+0000","last_peered":"2024-08-17T17:46:01.734498+0000","last_clean":"2024-08-17T17:46:01.734498+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:46:01.734498+0000","last_undegraded":"2024-08-17T17:46:01.734498+0000","last_fullsized":"2024-08-17T17:46:01.734498+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"90'149","reported_seq":856,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.740758+0000","last_change":"2024-08-17T17:41:37.681187+0000","last_active":"2024-08-17T17:46:01.740758+0000","last_peered":"2024-08-17T17:46:01.740758+0000","last_clean":"2024-08-17T17:46:01.740758+0000","last_became_active":"2024-08-17T17:41:37.680605+0000","last_became_peered":"2024-08-17T17:41:37.680605+0000","last_unstale":"2024-08-17T17:46:01.740758+0000","last_undegraded":"2024-08-17T17:46:01.740758+0000","last_fullsized":"2024-08-17T17:46:01.740758+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.844252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.844252+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:13:31.682357+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":459,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.735447+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:46:01.735447+0000","last_peered":"2024-08-17T17:46:01.735447+0000","last_clean":"2024-08-17T17:46:01.735447+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:46:01.735447+0000","last_undegraded":"2024-08-17T17:46:01.735447+0000","last_fullsized":"2024-08-17T17:46:01.735447+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":57,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.643563+0000","last_change":"2024-08-17T17:41:34.611693+0000","last_active":"2024-08-17T17:45:48.643563+0000","last_peered":"2024-08-17T17:45:48.643563+0000","last_clean":"2024-08-17T17:45:48.643563+0000","last_became_active":"2024-08-17T17:41:34.611374+0000","last_became_peered":"2024-08-17T17:41:34.611374+0000","last_unstale":"2024-08-17T17:45:48.643563+0000","last_undegraded":"2024-08-17T17:45:48.643563+0000","last_fullsized":"2024-08-17T17:45:48.643563+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:17:09.897446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"90'149","reported_seq":832,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.741597+0000","last_change":"2024-08-17T17:38:30.446165+0000","last_active":"2024-08-17T17:46:01.741597+0000","last_peered":"2024-08-17T17:46:01.741597+0000","last_clean":"2024-08-17T17:46:01.741597+0000","last_became_active":"2024-08-17T17:38:30.445048+0000","last_became_peered":"2024-08-17T17:38:30.445048+0000","last_unstale":"2024-08-17T17:46:01.741597+0000","last_undegraded":"2024-08-17T17:46:01.741597+0000","last_fullsized":"2024-08-17T17:46:01.741597+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:24.291592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:24.291592+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:17.100154+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":672,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.725995+0000","last_change":"2024-08-17T17:33:20.794682+0000","last_active":"2024-08-17T17:46:01.725995+0000","last_peered":"2024-08-17T17:46:01.725995+0000","last_clean":"2024-08-17T17:46:01.725995+0000","last_became_active":"2024-08-17T17:33:20.794238+0000","last_became_peered":"2024-08-17T17:33:20.794238+0000","last_unstale":"2024-08-17T17:46:01.725995+0000","last_undegraded":"2024-08-17T17:46:01.725995+0000","last_fullsized":"2024-08-17T17:46:01.725995+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:15:55.821920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":478,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.646924+0000","last_change":"2024-08-17T17:43:52.520659+0000","last_active":"2024-08-17T17:45:48.646924+0000","last_peered":"2024-08-17T17:45:48.646924+0000","last_clean":"2024-08-17T17:45:48.646924+0000","last_became_active":"2024-08-17T17:43:52.516799+0000","last_became_peered":"2024-08-17T17:43:52.516799+0000","last_unstale":"2024-08-17T17:45:48.646924+0000","last_undegraded":"2024-08-17T17:45:48.646924+0000","last_fullsized":"2024-08-17T17:45:48.646924+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:25:59.408383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"393'14028","reported_seq":14112,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:07.346525+0000","last_change":"2024-08-17T17:42:36.946078+0000","last_active":"2024-08-17T17:46:07.346525+0000","last_peered":"2024-08-17T17:46:07.346525+0000","last_clean":"2024-08-17T17:46:07.346525+0000","last_became_active":"2024-08-17T17:42:36.945602+0000","last_became_peered":"2024-08-17T17:42:36.945602+0000","last_unstale":"2024-08-17T17:46:07.346525+0000","last_undegraded":"2024-08-17T17:46:07.346525+0000","last_fullsized":"2024-08-17T17:46:07.346525+0000","mapping_epoch":357,"log_start":"385'7500","ondisk_log_start":"385'7500","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6528,"log_dups_size":0,"ondisk_log_size":6528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:18:43.716968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52690944,"num_objects":804,"num_object_clones":0,"num_object_copies":2412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":804,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14028,"num_write_kb":474624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"90'149","reported_seq":803,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.634646+0000","last_change":"2024-08-17T17:42:16.610940+0000","last_active":"2024-08-17T17:45:48.634646+0000","last_peered":"2024-08-17T17:45:48.634646+0000","last_clean":"2024-08-17T17:45:48.634646+0000","last_became_active":"2024-08-17T17:41:37.670855+0000","last_became_peered":"2024-08-17T17:41:37.670855+0000","last_unstale":"2024-08-17T17:45:48.634646+0000","last_undegraded":"2024-08-17T17:45:48.634646+0000","last_fullsized":"2024-08-17T17:45:48.634646+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:16.610828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:16.610828+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:48:25.918610+0000","scrub_duration":0.086892487000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":354,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.636285+0000","last_change":"2024-08-17T17:41:34.635980+0000","last_active":"2024-08-17T17:45:48.636285+0000","last_peered":"2024-08-17T17:45:48.636285+0000","last_clean":"2024-08-17T17:45:48.636285+0000","last_became_active":"2024-08-17T17:41:34.620596+0000","last_became_peered":"2024-08-17T17:41:34.620596+0000","last_unstale":"2024-08-17T17:45:48.636285+0000","last_undegraded":"2024-08-17T17:45:48.636285+0000","last_fullsized":"2024-08-17T17:45:48.636285+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:19:15.792577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":147,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.645952+0000","last_change":"2024-08-17T17:33:44.145598+0000","last_active":"2024-08-17T17:45:48.645952+0000","last_peered":"2024-08-17T17:45:48.645952+0000","last_clean":"2024-08-17T17:45:48.645952+0000","last_became_active":"2024-08-17T17:33:44.145283+0000","last_became_peered":"2024-08-17T17:33:44.145283+0000","last_unstale":"2024-08-17T17:45:48.645952+0000","last_undegraded":"2024-08-17T17:45:48.645952+0000","last_fullsized":"2024-08-17T17:45:48.645952+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:11.965226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.7","version":"393'13836","reported_seq":22380,"reported_epoch":393,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-17T17:46:04.030709+0000","last_change":"2024-08-17T17:44:30.783759+0000","last_active":"2024-08-17T17:46:04.030709+0000","last_peered":"2024-08-17T17:46:04.030709+0000","last_clean":"2024-08-17T17:43:51.465744+0000","last_became_active":"2024-08-17T17:43:53.682178+0000","last_became_peered":"2024-08-17T17:43:53.682178+0000","last_unstale":"2024-08-17T17:46:04.030709+0000","last_undegraded":"2024-08-17T17:44:30.779387+0000","last_fullsized":"2024-08-17T17:43:53.513318+0000","mapping_epoch":378,"log_start":"384'7400","ondisk_log_start":"384'7400","created":357,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6436,"log_dups_size":0,"ondisk_log_size":6436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:42:59.608872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65667072,"num_objects":1002,"num_object_clones":0,"num_object_copies":3006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2566,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13836,"num_write_kb":474816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4234,"num_bytes_recovered":251002880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3],"avail_no_missing":["6","2","3"],"object_location_counts":[{"shards":"3,6","objects":-1564}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"90'149","reported_seq":857,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.724400+0000","last_change":"2024-08-17T17:41:42.603841+0000","last_active":"2024-08-17T17:46:01.724400+0000","last_peered":"2024-08-17T17:46:01.724400+0000","last_clean":"2024-08-17T17:46:01.724400+0000","last_became_active":"2024-08-17T17:41:12.203400+0000","last_became_peered":"2024-08-17T17:41:12.203400+0000","last_unstale":"2024-08-17T17:46:01.724400+0000","last_undegraded":"2024-08-17T17:46:01.724400+0000","last_fullsized":"2024-08-17T17:46:01.724400+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:42.603561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:42.603561+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:50:12.843193+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":350,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.636259+0000","last_change":"2024-08-17T17:41:34.637696+0000","last_active":"2024-08-17T17:45:48.636259+0000","last_peered":"2024-08-17T17:45:48.636259+0000","last_clean":"2024-08-17T17:45:48.636259+0000","last_became_active":"2024-08-17T17:41:34.621388+0000","last_became_peered":"2024-08-17T17:41:34.621388+0000","last_unstale":"2024-08-17T17:45:48.636259+0000","last_undegraded":"2024-08-17T17:45:48.636259+0000","last_fullsized":"2024-08-17T17:45:48.636259+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:13:04.607036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":57,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.643372+0000","last_change":"2024-08-17T17:41:34.621078+0000","last_active":"2024-08-17T17:45:48.643372+0000","last_peered":"2024-08-17T17:45:48.643372+0000","last_clean":"2024-08-17T17:45:48.643372+0000","last_became_active":"2024-08-17T17:41:34.620730+0000","last_became_peered":"2024-08-17T17:41:34.620730+0000","last_unstale":"2024-08-17T17:45:48.643372+0000","last_undegraded":"2024-08-17T17:45:48.643372+0000","last_fullsized":"2024-08-17T17:45:48.643372+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:55:01.312283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"393'14128","reported_seq":14212,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:07.348136+0000","last_change":"2024-08-17T17:42:36.620130+0000","last_active":"2024-08-17T17:46:07.348136+0000","last_peered":"2024-08-17T17:46:07.348136+0000","last_clean":"2024-08-17T17:46:07.348136+0000","last_became_active":"2024-08-17T17:42:36.619813+0000","last_became_peered":"2024-08-17T17:42:36.619813+0000","last_unstale":"2024-08-17T17:46:07.348136+0000","last_undegraded":"2024-08-17T17:46:07.348136+0000","last_fullsized":"2024-08-17T17:46:07.348136+0000","mapping_epoch":357,"log_start":"385'7600","ondisk_log_start":"385'7600","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6528,"log_dups_size":0,"ondisk_log_size":6528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:39:22.366905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52559872,"num_objects":802,"num_object_clones":0,"num_object_copies":2406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":802,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14128,"num_write_kb":477760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"90'149","reported_seq":569,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.653267+0000","last_change":"2024-08-17T17:41:37.668050+0000","last_active":"2024-08-17T17:45:48.653267+0000","last_peered":"2024-08-17T17:45:48.653267+0000","last_clean":"2024-08-17T17:45:48.653267+0000","last_became_active":"2024-08-17T17:41:37.667655+0000","last_became_peered":"2024-08-17T17:41:37.667655+0000","last_unstale":"2024-08-17T17:45:48.653267+0000","last_undegraded":"2024-08-17T17:45:48.653267+0000","last_fullsized":"2024-08-17T17:45:48.653267+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:32.843802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:32.843802+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:39:46.766280+0000","scrub_duration":0.0040803669999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":143,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.646758+0000","last_change":"2024-08-17T17:33:44.145399+0000","last_active":"2024-08-17T17:45:48.646758+0000","last_peered":"2024-08-17T17:45:48.646758+0000","last_clean":"2024-08-17T17:45:48.646758+0000","last_became_active":"2024-08-17T17:33:44.144751+0000","last_became_peered":"2024-08-17T17:33:44.144751+0000","last_unstale":"2024-08-17T17:45:48.646758+0000","last_undegraded":"2024-08-17T17:45:48.646758+0000","last_fullsized":"2024-08-17T17:45:48.646758+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:54.276350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":221,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.646503+0000","last_change":"2024-08-17T17:33:20.795582+0000","last_active":"2024-08-17T17:45:48.646503+0000","last_peered":"2024-08-17T17:45:48.646503+0000","last_clean":"2024-08-17T17:45:48.646503+0000","last_became_active":"2024-08-17T17:33:20.795149+0000","last_became_peered":"2024-08-17T17:33:20.795149+0000","last_unstale":"2024-08-17T17:45:48.646503+0000","last_undegraded":"2024-08-17T17:45:48.646503+0000","last_fullsized":"2024-08-17T17:45:48.646503+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:14.020923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"393'14051","reported_seq":14135,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:07.361894+0000","last_change":"2024-08-17T17:42:36.929599+0000","last_active":"2024-08-17T17:46:07.361894+0000","last_peered":"2024-08-17T17:46:07.361894+0000","last_clean":"2024-08-17T17:46:07.361894+0000","last_became_active":"2024-08-17T17:42:36.929274+0000","last_became_peered":"2024-08-17T17:42:36.929274+0000","last_unstale":"2024-08-17T17:46:07.361894+0000","last_undegraded":"2024-08-17T17:46:07.361894+0000","last_fullsized":"2024-08-17T17:46:07.361894+0000","mapping_epoch":357,"log_start":"385'7500","ondisk_log_start":"385'7500","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6551,"log_dups_size":0,"ondisk_log_size":6551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:29:35.955743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52756480,"num_objects":805,"num_object_clones":0,"num_object_copies":2415,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14051,"num_write_kb":475392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"90'149","reported_seq":701,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.648480+0000","last_change":"2024-08-17T17:42:20.927007+0000","last_active":"2024-08-17T17:45:48.648480+0000","last_peered":"2024-08-17T17:45:48.648480+0000","last_clean":"2024-08-17T17:45:48.648480+0000","last_became_active":"2024-08-17T17:41:37.670914+0000","last_became_peered":"2024-08-17T17:41:37.670914+0000","last_unstale":"2024-08-17T17:45:48.648480+0000","last_undegraded":"2024-08-17T17:45:48.648480+0000","last_fullsized":"2024-08-17T17:45:48.648480+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:20.926924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:20.926924+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:57:44.156600+0000","scrub_duration":0.0018485109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":317,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.724658+0000","last_change":"2024-08-17T17:33:44.154097+0000","last_active":"2024-08-17T17:46:01.724658+0000","last_peered":"2024-08-17T17:46:01.724658+0000","last_clean":"2024-08-17T17:46:01.724658+0000","last_became_active":"2024-08-17T17:33:44.152972+0000","last_became_peered":"2024-08-17T17:33:44.152972+0000","last_unstale":"2024-08-17T17:46:01.724658+0000","last_undegraded":"2024-08-17T17:46:01.724658+0000","last_fullsized":"2024-08-17T17:46:01.724658+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:42.602017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":681,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.736723+0000","last_change":"2024-08-17T17:41:34.615197+0000","last_active":"2024-08-17T17:46:01.736723+0000","last_peered":"2024-08-17T17:46:01.736723+0000","last_clean":"2024-08-17T17:46:01.736723+0000","last_became_active":"2024-08-17T17:41:34.614873+0000","last_became_peered":"2024-08-17T17:41:34.614873+0000","last_unstale":"2024-08-17T17:46:01.736723+0000","last_undegraded":"2024-08-17T17:46:01.736723+0000","last_fullsized":"2024-08-17T17:46:01.736723+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:21:57.391953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2","version":"393'13626","reported_seq":13709,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:06.015570+0000","last_change":"2024-08-17T17:42:36.618260+0000","last_active":"2024-08-17T17:46:06.015570+0000","last_peered":"2024-08-17T17:46:06.015570+0000","last_clean":"2024-08-17T17:46:06.015570+0000","last_became_active":"2024-08-17T17:42:36.617875+0000","last_became_peered":"2024-08-17T17:42:36.617875+0000","last_unstale":"2024-08-17T17:46:06.015570+0000","last_undegraded":"2024-08-17T17:46:06.015570+0000","last_fullsized":"2024-08-17T17:46:06.015570+0000","mapping_epoch":357,"log_start":"383'6600","ondisk_log_start":"383'6600","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":7026,"log_dups_size":0,"ondisk_log_size":7026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:09:07.102359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56885248,"num_objects":868,"num_object_clones":0,"num_object_copies":2604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13626,"num_write_kb":463808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":831,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.722570+0000","last_change":"2024-08-17T17:41:44.585386+0000","last_active":"2024-08-17T17:46:01.722570+0000","last_peered":"2024-08-17T17:46:01.722570+0000","last_clean":"2024-08-17T17:46:01.722570+0000","last_became_active":"2024-08-17T17:41:12.307398+0000","last_became_peered":"2024-08-17T17:41:12.307398+0000","last_unstale":"2024-08-17T17:46:01.722570+0000","last_undegraded":"2024-08-17T17:46:01.722570+0000","last_fullsized":"2024-08-17T17:46:01.722570+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:44.583653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:44.583653+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:27:44.756681+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":716,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.723307+0000","last_change":"2024-08-17T17:41:34.630434+0000","last_active":"2024-08-17T17:46:01.723307+0000","last_peered":"2024-08-17T17:46:01.723307+0000","last_clean":"2024-08-17T17:46:01.723307+0000","last_became_active":"2024-08-17T17:41:34.616406+0000","last_became_peered":"2024-08-17T17:41:34.616406+0000","last_unstale":"2024-08-17T17:46:01.723307+0000","last_undegraded":"2024-08-17T17:46:01.723307+0000","last_fullsized":"2024-08-17T17:46:01.723307+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:30:44.817203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":57,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.648219+0000","last_change":"2024-08-17T17:41:34.611049+0000","last_active":"2024-08-17T17:45:48.648219+0000","last_peered":"2024-08-17T17:45:48.648219+0000","last_clean":"2024-08-17T17:45:48.648219+0000","last_became_active":"2024-08-17T17:41:34.610727+0000","last_became_peered":"2024-08-17T17:41:34.610727+0000","last_unstale":"2024-08-17T17:45:48.648219+0000","last_undegraded":"2024-08-17T17:45:48.648219+0000","last_fullsized":"2024-08-17T17:45:48.648219+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:04:05.477660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"393'14065","reported_seq":14149,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:04.685804+0000","last_change":"2024-08-17T17:42:36.946226+0000","last_active":"2024-08-17T17:46:04.685804+0000","last_peered":"2024-08-17T17:46:04.685804+0000","last_clean":"2024-08-17T17:46:04.685804+0000","last_became_active":"2024-08-17T17:42:36.945809+0000","last_became_peered":"2024-08-17T17:42:36.945809+0000","last_unstale":"2024-08-17T17:46:04.685804+0000","last_undegraded":"2024-08-17T17:46:04.685804+0000","last_fullsized":"2024-08-17T17:46:04.685804+0000","mapping_epoch":357,"log_start":"385'7800","ondisk_log_start":"385'7800","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6265,"log_dups_size":0,"ondisk_log_size":6265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:04:58.633957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63111168,"num_objects":963,"num_object_clones":0,"num_object_copies":2889,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14065,"num_write_kb":480896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":859,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.722570+0000","last_change":"2024-08-17T17:43:55.595647+0000","last_active":"2024-08-17T17:46:01.722570+0000","last_peered":"2024-08-17T17:46:01.722570+0000","last_clean":"2024-08-17T17:46:01.722570+0000","last_became_active":"2024-08-17T17:43:55.595256+0000","last_became_peered":"2024-08-17T17:43:55.595256+0000","last_unstale":"2024-08-17T17:46:01.722570+0000","last_undegraded":"2024-08-17T17:46:01.722570+0000","last_fullsized":"2024-08-17T17:46:01.722570+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:46.587712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:46.587712+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:05:01.927968+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":57,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.644303+0000","last_change":"2024-08-17T17:41:34.622803+0000","last_active":"2024-08-17T17:45:48.644303+0000","last_peered":"2024-08-17T17:45:48.644303+0000","last_clean":"2024-08-17T17:45:48.644303+0000","last_became_active":"2024-08-17T17:41:34.622525+0000","last_became_peered":"2024-08-17T17:41:34.622525+0000","last_unstale":"2024-08-17T17:45:48.644303+0000","last_undegraded":"2024-08-17T17:45:48.644303+0000","last_fullsized":"2024-08-17T17:45:48.644303+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:42:39.728377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":57,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.644155+0000","last_change":"2024-08-17T17:41:34.619268+0000","last_active":"2024-08-17T17:45:48.644155+0000","last_peered":"2024-08-17T17:45:48.644155+0000","last_clean":"2024-08-17T17:45:48.644155+0000","last_became_active":"2024-08-17T17:41:34.618914+0000","last_became_peered":"2024-08-17T17:41:34.618914+0000","last_unstale":"2024-08-17T17:45:48.644155+0000","last_undegraded":"2024-08-17T17:45:48.644155+0000","last_fullsized":"2024-08-17T17:45:48.644155+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:08:54.248386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.0","version":"393'13383","reported_seq":13427,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:04.043259+0000","last_change":"2024-08-17T17:42:36.617537+0000","last_active":"2024-08-17T17:46:04.043259+0000","last_peered":"2024-08-17T17:46:04.043259+0000","last_clean":"2024-08-17T17:46:04.043259+0000","last_became_active":"2024-08-17T17:42:36.617201+0000","last_became_peered":"2024-08-17T17:42:36.617201+0000","last_unstale":"2024-08-17T17:46:04.043259+0000","last_undegraded":"2024-08-17T17:46:04.043259+0000","last_fullsized":"2024-08-17T17:46:04.043259+0000","mapping_epoch":357,"log_start":"384'7000","ondisk_log_start":"384'7000","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6383,"log_dups_size":0,"ondisk_log_size":6383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:51:58.927214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13383,"num_write_kb":459712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"90'149","reported_seq":865,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.741966+0000","last_change":"2024-08-17T17:41:41.743931+0000","last_active":"2024-08-17T17:46:01.741966+0000","last_peered":"2024-08-17T17:46:01.741966+0000","last_clean":"2024-08-17T17:46:01.741966+0000","last_became_active":"2024-08-17T17:41:12.203771+0000","last_became_peered":"2024-08-17T17:41:12.203771+0000","last_unstale":"2024-08-17T17:46:01.741966+0000","last_undegraded":"2024-08-17T17:46:01.741966+0000","last_fullsized":"2024-08-17T17:46:01.741966+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:41.743570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:41.743570+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:21:03.865738+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":57,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.643638+0000","last_change":"2024-08-17T17:41:34.607602+0000","last_active":"2024-08-17T17:45:48.643638+0000","last_peered":"2024-08-17T17:45:48.643638+0000","last_clean":"2024-08-17T17:45:48.643638+0000","last_became_active":"2024-08-17T17:41:34.607300+0000","last_became_peered":"2024-08-17T17:41:34.607300+0000","last_unstale":"2024-08-17T17:45:48.643638+0000","last_undegraded":"2024-08-17T17:45:48.643638+0000","last_fullsized":"2024-08-17T17:45:48.643638+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:20:31.486698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":57,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.648325+0000","last_change":"2024-08-17T17:41:34.622102+0000","last_active":"2024-08-17T17:45:48.648325+0000","last_peered":"2024-08-17T17:45:48.648325+0000","last_clean":"2024-08-17T17:45:48.648325+0000","last_became_active":"2024-08-17T17:41:34.621733+0000","last_became_peered":"2024-08-17T17:41:34.621733+0000","last_unstale":"2024-08-17T17:45:48.648325+0000","last_undegraded":"2024-08-17T17:45:48.648325+0000","last_fullsized":"2024-08-17T17:45:48.648325+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:39:41.668709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"393'14067","reported_seq":14109,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:03.860041+0000","last_change":"2024-08-17T17:42:36.620155+0000","last_active":"2024-08-17T17:46:03.860041+0000","last_peered":"2024-08-17T17:46:03.860041+0000","last_clean":"2024-08-17T17:46:03.860041+0000","last_became_active":"2024-08-17T17:42:36.619813+0000","last_became_peered":"2024-08-17T17:42:36.619813+0000","last_unstale":"2024-08-17T17:46:03.860041+0000","last_undegraded":"2024-08-17T17:46:03.860041+0000","last_fullsized":"2024-08-17T17:46:03.860041+0000","mapping_epoch":357,"log_start":"386'8200","ondisk_log_start":"386'8200","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":5867,"log_dups_size":0,"ondisk_log_size":5867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:16:27.534439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67829760,"num_objects":1035,"num_object_clones":0,"num_object_copies":3105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14067,"num_write_kb":483264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":510,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.645785+0000","last_change":"2024-08-17T17:41:41.036615+0000","last_active":"2024-08-17T17:45:48.645785+0000","last_peered":"2024-08-17T17:45:48.645785+0000","last_clean":"2024-08-17T17:45:48.645785+0000","last_became_active":"2024-08-17T17:41:12.329214+0000","last_became_peered":"2024-08-17T17:41:12.329214+0000","last_unstale":"2024-08-17T17:45:48.645785+0000","last_undegraded":"2024-08-17T17:45:48.645785+0000","last_fullsized":"2024-08-17T17:45:48.645785+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:41.036524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:41.036524+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:59:58.695900+0000","scrub_duration":0.001820803,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":675,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.722879+0000","last_change":"2024-08-17T17:33:20.796216+0000","last_active":"2024-08-17T17:46:01.722879+0000","last_peered":"2024-08-17T17:46:01.722879+0000","last_clean":"2024-08-17T17:46:01.722879+0000","last_became_active":"2024-08-17T17:33:20.795762+0000","last_became_peered":"2024-08-17T17:33:20.795762+0000","last_unstale":"2024-08-17T17:46:01.722879+0000","last_undegraded":"2024-08-17T17:46:01.722879+0000","last_fullsized":"2024-08-17T17:46:01.722879+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:47.038509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":569,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.738442+0000","last_change":"2024-08-17T17:41:34.625551+0000","last_active":"2024-08-17T17:46:01.738442+0000","last_peered":"2024-08-17T17:46:01.738442+0000","last_clean":"2024-08-17T17:46:01.738442+0000","last_became_active":"2024-08-17T17:41:34.625251+0000","last_became_peered":"2024-08-17T17:41:34.625251+0000","last_unstale":"2024-08-17T17:46:01.738442+0000","last_undegraded":"2024-08-17T17:46:01.738442+0000","last_fullsized":"2024-08-17T17:46:01.738442+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:28:22.354213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"393'13819","reported_seq":13863,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:04.025345+0000","last_change":"2024-08-17T17:42:36.619525+0000","last_active":"2024-08-17T17:46:04.025345+0000","last_peered":"2024-08-17T17:46:04.025345+0000","last_clean":"2024-08-17T17:46:04.025345+0000","last_became_active":"2024-08-17T17:42:36.619074+0000","last_became_peered":"2024-08-17T17:42:36.619074+0000","last_unstale":"2024-08-17T17:46:04.025345+0000","last_undegraded":"2024-08-17T17:46:04.025345+0000","last_fullsized":"2024-08-17T17:46:04.025345+0000","mapping_epoch":357,"log_start":"384'7400","ondisk_log_start":"384'7400","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6419,"log_dups_size":0,"ondisk_log_size":6419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:11:32.849616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67174400,"num_objects":1025,"num_object_clones":0,"num_object_copies":3075,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13819,"num_write_kb":475008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":811,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.734464+0000","last_change":"2024-08-17T17:38:25.247159+0000","last_active":"2024-08-17T17:46:01.734464+0000","last_peered":"2024-08-17T17:46:01.734464+0000","last_clean":"2024-08-17T17:46:01.734464+0000","last_became_active":"2024-08-17T17:38:21.311361+0000","last_became_peered":"2024-08-17T17:38:21.311361+0000","last_unstale":"2024-08-17T17:46:01.734464+0000","last_undegraded":"2024-08-17T17:46:01.734464+0000","last_fullsized":"2024-08-17T17:46:01.734464+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:25.247027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:25.247027+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:46:21.734712+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":799,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.634496+0000","last_change":"2024-08-17T17:33:44.149928+0000","last_active":"2024-08-17T17:45:48.634496+0000","last_peered":"2024-08-17T17:45:48.634496+0000","last_clean":"2024-08-17T17:45:48.634496+0000","last_became_active":"2024-08-17T17:33:44.149498+0000","last_became_peered":"2024-08-17T17:33:44.149498+0000","last_unstale":"2024-08-17T17:45:48.634496+0000","last_undegraded":"2024-08-17T17:45:48.634496+0000","last_fullsized":"2024-08-17T17:45:48.634496+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:17:04.963202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":622,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.723099+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:46:01.723099+0000","last_peered":"2024-08-17T17:46:01.723099+0000","last_clean":"2024-08-17T17:46:01.723099+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:46:01.723099+0000","last_undegraded":"2024-08-17T17:46:01.723099+0000","last_fullsized":"2024-08-17T17:46:01.723099+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":234,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.646683+0000","last_change":"2024-08-17T17:41:34.618631+0000","last_active":"2024-08-17T17:45:48.646683+0000","last_peered":"2024-08-17T17:45:48.646683+0000","last_clean":"2024-08-17T17:45:48.646683+0000","last_became_active":"2024-08-17T17:41:34.618313+0000","last_became_peered":"2024-08-17T17:41:34.618313+0000","last_unstale":"2024-08-17T17:45:48.646683+0000","last_undegraded":"2024-08-17T17:45:48.646683+0000","last_fullsized":"2024-08-17T17:45:48.646683+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:07:13.915088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"393'13886","reported_seq":13970,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:07.379940+0000","last_change":"2024-08-17T17:42:36.929502+0000","last_active":"2024-08-17T17:46:07.379940+0000","last_peered":"2024-08-17T17:46:07.379940+0000","last_clean":"2024-08-17T17:46:07.379940+0000","last_became_active":"2024-08-17T17:42:36.929113+0000","last_became_peered":"2024-08-17T17:42:36.929113+0000","last_unstale":"2024-08-17T17:46:07.379940+0000","last_undegraded":"2024-08-17T17:46:07.379940+0000","last_fullsized":"2024-08-17T17:46:07.379940+0000","mapping_epoch":357,"log_start":"384'7300","ondisk_log_start":"384'7300","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6586,"log_dups_size":0,"ondisk_log_size":6586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:31:25.879221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49938432,"num_objects":762,"num_object_clones":0,"num_object_copies":2286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13886,"num_write_kb":468736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":848,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.742141+0000","last_change":"2024-08-17T17:41:37.676347+0000","last_active":"2024-08-17T17:46:01.742141+0000","last_peered":"2024-08-17T17:46:01.742141+0000","last_clean":"2024-08-17T17:46:01.742141+0000","last_became_active":"2024-08-17T17:41:37.675869+0000","last_became_peered":"2024-08-17T17:41:37.675869+0000","last_unstale":"2024-08-17T17:46:01.742141+0000","last_undegraded":"2024-08-17T17:46:01.742141+0000","last_fullsized":"2024-08-17T17:46:01.742141+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:21.814077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:21.814077+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:04:16.240754+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":906,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.741873+0000","last_change":"2024-08-17T17:33:20.797705+0000","last_active":"2024-08-17T17:46:01.741873+0000","last_peered":"2024-08-17T17:46:01.741873+0000","last_clean":"2024-08-17T17:46:01.741873+0000","last_became_active":"2024-08-17T17:33:20.797172+0000","last_became_peered":"2024-08-17T17:33:20.797172+0000","last_unstale":"2024-08-17T17:46:01.741873+0000","last_undegraded":"2024-08-17T17:46:01.741873+0000","last_fullsized":"2024-08-17T17:46:01.741873+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:30.754999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":672,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.734517+0000","last_change":"2024-08-17T17:41:34.614108+0000","last_active":"2024-08-17T17:46:01.734517+0000","last_peered":"2024-08-17T17:46:01.734517+0000","last_clean":"2024-08-17T17:46:01.734517+0000","last_became_active":"2024-08-17T17:41:34.613481+0000","last_became_peered":"2024-08-17T17:41:34.613481+0000","last_unstale":"2024-08-17T17:46:01.734517+0000","last_undegraded":"2024-08-17T17:46:01.734517+0000","last_fullsized":"2024-08-17T17:46:01.734517+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:21:02.346115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":460,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.735608+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:46:01.735608+0000","last_peered":"2024-08-17T17:46:01.735608+0000","last_clean":"2024-08-17T17:46:01.735608+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:46:01.735608+0000","last_undegraded":"2024-08-17T17:46:01.735608+0000","last_fullsized":"2024-08-17T17:46:01.735608+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"393'13710","reported_seq":13754,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:03.704459+0000","last_change":"2024-08-17T17:42:36.946382+0000","last_active":"2024-08-17T17:46:03.704459+0000","last_peered":"2024-08-17T17:46:03.704459+0000","last_clean":"2024-08-17T17:46:03.704459+0000","last_became_active":"2024-08-17T17:42:36.945591+0000","last_became_peered":"2024-08-17T17:42:36.945591+0000","last_unstale":"2024-08-17T17:46:03.704459+0000","last_undegraded":"2024-08-17T17:46:03.704459+0000","last_fullsized":"2024-08-17T17:46:03.704459+0000","mapping_epoch":357,"log_start":"384'7400","ondisk_log_start":"384'7400","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6310,"log_dups_size":0,"ondisk_log_size":6310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:08:17.912891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68943872,"num_objects":1052,"num_object_clones":0,"num_object_copies":3156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13710,"num_write_kb":472384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":851,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.734396+0000","last_change":"2024-08-17T17:41:37.683450+0000","last_active":"2024-08-17T17:46:01.734396+0000","last_peered":"2024-08-17T17:46:01.734396+0000","last_clean":"2024-08-17T17:46:01.734396+0000","last_became_active":"2024-08-17T17:41:37.683103+0000","last_became_peered":"2024-08-17T17:41:37.683103+0000","last_unstale":"2024-08-17T17:46:01.734396+0000","last_undegraded":"2024-08-17T17:46:01.734396+0000","last_fullsized":"2024-08-17T17:46:01.734396+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.289528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.289528+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:39:17.141469+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":1039,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.724023+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:46:01.724023+0000","last_peered":"2024-08-17T17:46:01.724023+0000","last_clean":"2024-08-17T17:46:01.724023+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:46:01.724023+0000","last_undegraded":"2024-08-17T17:46:01.724023+0000","last_fullsized":"2024-08-17T17:46:01.724023+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":544,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.742434+0000","last_change":"2024-08-17T17:33:20.793468+0000","last_active":"2024-08-17T17:46:01.742434+0000","last_peered":"2024-08-17T17:46:01.742434+0000","last_clean":"2024-08-17T17:46:01.742434+0000","last_became_active":"2024-08-17T17:33:20.793166+0000","last_became_peered":"2024-08-17T17:33:20.793166+0000","last_unstale":"2024-08-17T17:46:01.742434+0000","last_undegraded":"2024-08-17T17:46:01.742434+0000","last_fullsized":"2024-08-17T17:46:01.742434+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:12:52.121050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":579,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.738355+0000","last_change":"2024-08-17T17:33:20.835784+0000","last_active":"2024-08-17T17:46:01.738355+0000","last_peered":"2024-08-17T17:46:01.738355+0000","last_clean":"2024-08-17T17:46:01.738355+0000","last_became_active":"2024-08-17T17:33:20.835371+0000","last_became_peered":"2024-08-17T17:33:20.835371+0000","last_unstale":"2024-08-17T17:46:01.738355+0000","last_undegraded":"2024-08-17T17:46:01.738355+0000","last_fullsized":"2024-08-17T17:46:01.738355+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:35:43.369055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"393'13760","reported_seq":13804,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:04.010203+0000","last_change":"2024-08-17T17:42:36.946026+0000","last_active":"2024-08-17T17:46:04.010203+0000","last_peered":"2024-08-17T17:46:04.010203+0000","last_clean":"2024-08-17T17:46:04.010203+0000","last_became_active":"2024-08-17T17:42:36.945638+0000","last_became_peered":"2024-08-17T17:42:36.945638+0000","last_unstale":"2024-08-17T17:46:04.010203+0000","last_undegraded":"2024-08-17T17:46:04.010203+0000","last_fullsized":"2024-08-17T17:46:04.010203+0000","mapping_epoch":357,"log_start":"384'7300","ondisk_log_start":"384'7300","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6460,"log_dups_size":0,"ondisk_log_size":6460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:07:34.864262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63832064,"num_objects":974,"num_object_clones":0,"num_object_copies":2922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13760,"num_write_kb":471488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":830,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.734412+0000","last_change":"2024-08-17T17:38:22.254841+0000","last_active":"2024-08-17T17:46:01.734412+0000","last_peered":"2024-08-17T17:46:01.734412+0000","last_clean":"2024-08-17T17:46:01.734412+0000","last_became_active":"2024-08-17T17:38:21.306619+0000","last_became_peered":"2024-08-17T17:38:21.306619+0000","last_unstale":"2024-08-17T17:46:01.734412+0000","last_undegraded":"2024-08-17T17:46:01.734412+0000","last_fullsized":"2024-08-17T17:46:01.734412+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:22.254705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:22.254705+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:47:38.184089+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":1128,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.740102+0000","last_change":"2024-08-17T17:33:44.148227+0000","last_active":"2024-08-17T17:46:01.740102+0000","last_peered":"2024-08-17T17:46:01.740102+0000","last_clean":"2024-08-17T17:46:01.740102+0000","last_became_active":"2024-08-17T17:33:44.147627+0000","last_became_peered":"2024-08-17T17:33:44.147627+0000","last_unstale":"2024-08-17T17:46:01.740102+0000","last_undegraded":"2024-08-17T17:46:01.740102+0000","last_fullsized":"2024-08-17T17:46:01.740102+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:29:42.898460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1372,"num_read_kb":2843,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":57,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.644699+0000","last_change":"2024-08-17T17:41:34.611109+0000","last_active":"2024-08-17T17:45:48.644699+0000","last_peered":"2024-08-17T17:45:48.644699+0000","last_clean":"2024-08-17T17:45:48.644699+0000","last_became_active":"2024-08-17T17:41:34.610170+0000","last_became_peered":"2024-08-17T17:41:34.610170+0000","last_unstale":"2024-08-17T17:45:48.644699+0000","last_undegraded":"2024-08-17T17:45:48.644699+0000","last_fullsized":"2024-08-17T17:45:48.644699+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:39:05.301260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":584,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.738879+0000","last_change":"2024-08-17T17:33:20.772454+0000","last_active":"2024-08-17T17:46:01.738879+0000","last_peered":"2024-08-17T17:46:01.738879+0000","last_clean":"2024-08-17T17:46:01.738879+0000","last_became_active":"2024-08-17T17:33:20.771848+0000","last_became_peered":"2024-08-17T17:33:20.771848+0000","last_unstale":"2024-08-17T17:46:01.738879+0000","last_undegraded":"2024-08-17T17:46:01.738879+0000","last_fullsized":"2024-08-17T17:46:01.738879+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:46:42.300744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"393'14097","reported_seq":14139,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:03.723218+0000","last_change":"2024-08-17T17:42:36.618414+0000","last_active":"2024-08-17T17:46:03.723218+0000","last_peered":"2024-08-17T17:46:03.723218+0000","last_clean":"2024-08-17T17:46:03.723218+0000","last_became_active":"2024-08-17T17:42:36.618086+0000","last_became_peered":"2024-08-17T17:42:36.618086+0000","last_unstale":"2024-08-17T17:46:03.723218+0000","last_undegraded":"2024-08-17T17:46:03.723218+0000","last_fullsized":"2024-08-17T17:46:03.723218+0000","mapping_epoch":357,"log_start":"385'7800","ondisk_log_start":"385'7800","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6297,"log_dups_size":0,"ondisk_log_size":6297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:26:27.772894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71368704,"num_objects":1089,"num_object_clones":0,"num_object_copies":3267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14097,"num_write_kb":485952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":530,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.653222+0000","last_change":"2024-08-17T17:41:43.035215+0000","last_active":"2024-08-17T17:45:48.653222+0000","last_peered":"2024-08-17T17:45:48.653222+0000","last_clean":"2024-08-17T17:45:48.653222+0000","last_became_active":"2024-08-17T17:41:12.201158+0000","last_became_peered":"2024-08-17T17:41:12.201158+0000","last_unstale":"2024-08-17T17:45:48.653222+0000","last_undegraded":"2024-08-17T17:45:48.653222+0000","last_fullsized":"2024-08-17T17:45:48.653222+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:43.035108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:43.035108+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:43:27.108994+0000","scrub_duration":0.0028877400000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":334,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.724606+0000","last_change":"2024-08-17T17:33:44.163867+0000","last_active":"2024-08-17T17:46:01.724606+0000","last_peered":"2024-08-17T17:46:01.724606+0000","last_clean":"2024-08-17T17:46:01.724606+0000","last_became_active":"2024-08-17T17:33:44.163503+0000","last_became_peered":"2024-08-17T17:33:44.163503+0000","last_unstale":"2024-08-17T17:46:01.724606+0000","last_undegraded":"2024-08-17T17:46:01.724606+0000","last_fullsized":"2024-08-17T17:46:01.724606+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:31:53.573596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":799,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.738648+0000","last_change":"2024-08-17T17:33:20.799350+0000","last_active":"2024-08-17T17:46:01.738648+0000","last_peered":"2024-08-17T17:46:01.738648+0000","last_clean":"2024-08-17T17:46:01.738648+0000","last_became_active":"2024-08-17T17:33:20.798925+0000","last_became_peered":"2024-08-17T17:33:20.798925+0000","last_unstale":"2024-08-17T17:46:01.738648+0000","last_undegraded":"2024-08-17T17:46:01.738648+0000","last_fullsized":"2024-08-17T17:46:01.738648+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:16:36.121295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":615,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.741694+0000","last_change":"2024-08-17T17:41:34.627064+0000","last_active":"2024-08-17T17:46:01.741694+0000","last_peered":"2024-08-17T17:46:01.741694+0000","last_clean":"2024-08-17T17:46:01.741694+0000","last_became_active":"2024-08-17T17:41:34.626453+0000","last_became_peered":"2024-08-17T17:41:34.626453+0000","last_unstale":"2024-08-17T17:46:01.741694+0000","last_undegraded":"2024-08-17T17:46:01.741694+0000","last_fullsized":"2024-08-17T17:46:01.741694+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:02:11.894042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"393'13855","reported_seq":13900,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:04.010105+0000","last_change":"2024-08-17T17:42:36.620669+0000","last_active":"2024-08-17T17:46:04.010105+0000","last_peered":"2024-08-17T17:46:04.010105+0000","last_clean":"2024-08-17T17:46:04.010105+0000","last_became_active":"2024-08-17T17:42:36.619876+0000","last_became_peered":"2024-08-17T17:42:36.619876+0000","last_unstale":"2024-08-17T17:46:04.010105+0000","last_undegraded":"2024-08-17T17:46:04.010105+0000","last_fullsized":"2024-08-17T17:46:04.010105+0000","mapping_epoch":357,"log_start":"384'7400","ondisk_log_start":"384'7400","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6455,"log_dups_size":0,"ondisk_log_size":6455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:59:34.661001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66715672,"num_objects":1019,"num_object_clones":0,"num_object_copies":3057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1019,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13855,"num_write_kb":475905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":530,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.646368+0000","last_change":"2024-08-17T17:41:44.058291+0000","last_active":"2024-08-17T17:45:48.646368+0000","last_peered":"2024-08-17T17:45:48.646368+0000","last_clean":"2024-08-17T17:45:48.646368+0000","last_became_active":"2024-08-17T17:41:12.200077+0000","last_became_peered":"2024-08-17T17:41:12.200077+0000","last_unstale":"2024-08-17T17:45:48.646368+0000","last_undegraded":"2024-08-17T17:45:48.646368+0000","last_fullsized":"2024-08-17T17:45:48.646368+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:44.058233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:44.058233+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:42:55.911623+0000","scrub_duration":0.0074268499999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":619,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.738143+0000","last_change":"2024-08-17T17:33:20.798613+0000","last_active":"2024-08-17T17:46:01.738143+0000","last_peered":"2024-08-17T17:46:01.738143+0000","last_clean":"2024-08-17T17:46:01.738143+0000","last_became_active":"2024-08-17T17:33:20.798260+0000","last_became_peered":"2024-08-17T17:33:20.798260+0000","last_unstale":"2024-08-17T17:46:01.738143+0000","last_undegraded":"2024-08-17T17:46:01.738143+0000","last_fullsized":"2024-08-17T17:46:01.738143+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:18:00.822681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":953,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.635099+0000","last_change":"2024-08-17T17:41:37.683212+0000","last_active":"2024-08-17T17:45:48.635099+0000","last_peered":"2024-08-17T17:45:48.635099+0000","last_clean":"2024-08-17T17:45:48.635099+0000","last_became_active":"2024-08-17T17:41:37.682730+0000","last_became_peered":"2024-08-17T17:41:37.682730+0000","last_unstale":"2024-08-17T17:45:48.635099+0000","last_undegraded":"2024-08-17T17:45:48.635099+0000","last_fullsized":"2024-08-17T17:45:48.635099+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:51:35.444433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":634,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.736643+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:46:01.736643+0000","last_peered":"2024-08-17T17:46:01.736643+0000","last_clean":"2024-08-17T17:46:01.736643+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:46:01.736643+0000","last_undegraded":"2024-08-17T17:46:01.736643+0000","last_fullsized":"2024-08-17T17:46:01.736643+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"393'13857","reported_seq":13899,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:03.790291+0000","last_change":"2024-08-17T17:42:36.929709+0000","last_active":"2024-08-17T17:46:03.790291+0000","last_peered":"2024-08-17T17:46:03.790291+0000","last_clean":"2024-08-17T17:46:03.790291+0000","last_became_active":"2024-08-17T17:42:36.929285+0000","last_became_peered":"2024-08-17T17:42:36.929285+0000","last_unstale":"2024-08-17T17:46:03.790291+0000","last_undegraded":"2024-08-17T17:46:03.790291+0000","last_fullsized":"2024-08-17T17:46:03.790291+0000","mapping_epoch":357,"log_start":"386'8000","ondisk_log_start":"386'8000","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":5857,"log_dups_size":0,"ondisk_log_size":5857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:47:39.760414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66650112,"num_objects":1017,"num_object_clones":0,"num_object_copies":3051,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13857,"num_write_kb":475968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":776,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.643789+0000","last_change":"2024-08-17T17:42:19.960161+0000","last_active":"2024-08-17T17:45:48.643789+0000","last_peered":"2024-08-17T17:45:48.643789+0000","last_clean":"2024-08-17T17:45:48.643789+0000","last_became_active":"2024-08-17T17:41:37.665223+0000","last_became_peered":"2024-08-17T17:41:37.665223+0000","last_unstale":"2024-08-17T17:45:48.643789+0000","last_undegraded":"2024-08-17T17:45:48.643789+0000","last_fullsized":"2024-08-17T17:45:48.643789+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:19.960086+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:19.960086+0000","last_clean_scrub_stamp":"2024-08-17T17:42:19.960086+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:57.861661+0000","scrub_duration":0.0022612000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":669,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.726183+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:46:01.726183+0000","last_peered":"2024-08-17T17:46:01.726183+0000","last_clean":"2024-08-17T17:46:01.726183+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:46:01.726183+0000","last_undegraded":"2024-08-17T17:46:01.726183+0000","last_fullsized":"2024-08-17T17:46:01.726183+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":1243,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.739020+0000","last_change":"2024-08-17T17:41:37.677524+0000","last_active":"2024-08-17T17:46:01.739020+0000","last_peered":"2024-08-17T17:46:01.739020+0000","last_clean":"2024-08-17T17:46:01.739020+0000","last_became_active":"2024-08-17T17:41:37.676764+0000","last_became_peered":"2024-08-17T17:41:37.676764+0000","last_unstale":"2024-08-17T17:46:01.739020+0000","last_undegraded":"2024-08-17T17:46:01.739020+0000","last_fullsized":"2024-08-17T17:46:01.739020+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:14:49.341080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":666,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.722771+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:46:01.722771+0000","last_peered":"2024-08-17T17:46:01.722771+0000","last_clean":"2024-08-17T17:46:01.722771+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:46:01.722771+0000","last_undegraded":"2024-08-17T17:46:01.722771+0000","last_fullsized":"2024-08-17T17:46:01.722771+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"393'13717","reported_seq":13760,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:04.010802+0000","last_change":"2024-08-17T17:42:36.929613+0000","last_active":"2024-08-17T17:46:04.010802+0000","last_peered":"2024-08-17T17:46:04.010802+0000","last_clean":"2024-08-17T17:46:04.010802+0000","last_became_active":"2024-08-17T17:42:36.929191+0000","last_became_peered":"2024-08-17T17:42:36.929191+0000","last_unstale":"2024-08-17T17:46:04.010802+0000","last_undegraded":"2024-08-17T17:46:04.010802+0000","last_fullsized":"2024-08-17T17:46:04.010802+0000","mapping_epoch":357,"log_start":"384'7300","ondisk_log_start":"384'7300","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6417,"log_dups_size":0,"ondisk_log_size":6417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:21:40.881779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65732608,"num_objects":1003,"num_object_clones":0,"num_object_copies":3009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13717,"num_write_kb":471040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":813,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.724305+0000","last_change":"2024-08-17T17:38:23.113397+0000","last_active":"2024-08-17T17:46:01.724305+0000","last_peered":"2024-08-17T17:46:01.724305+0000","last_clean":"2024-08-17T17:46:01.724305+0000","last_became_active":"2024-08-17T17:38:21.305580+0000","last_became_peered":"2024-08-17T17:38:21.305580+0000","last_unstale":"2024-08-17T17:46:01.724305+0000","last_undegraded":"2024-08-17T17:46:01.724305+0000","last_fullsized":"2024-08-17T17:46:01.724305+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.113279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.113279+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:38:07.015721+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":506,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.735521+0000","last_change":"2024-08-17T17:41:34.629522+0000","last_active":"2024-08-17T17:46:01.735521+0000","last_peered":"2024-08-17T17:46:01.735521+0000","last_clean":"2024-08-17T17:46:01.735521+0000","last_became_active":"2024-08-17T17:41:34.619340+0000","last_became_peered":"2024-08-17T17:41:34.619340+0000","last_unstale":"2024-08-17T17:46:01.735521+0000","last_undegraded":"2024-08-17T17:46:01.735521+0000","last_fullsized":"2024-08-17T17:46:01.735521+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:30:03.902956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":623,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.653509+0000","last_change":"2024-08-17T17:26:06.855307+0000","last_active":"2024-08-17T17:45:48.653509+0000","last_peered":"2024-08-17T17:45:48.653509+0000","last_clean":"2024-08-17T17:45:48.653509+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:45:48.653509+0000","last_undegraded":"2024-08-17T17:45:48.653509+0000","last_fullsized":"2024-08-17T17:45:48.653509+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":57,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.643489+0000","last_change":"2024-08-17T17:41:34.614028+0000","last_active":"2024-08-17T17:45:48.643489+0000","last_peered":"2024-08-17T17:45:48.643489+0000","last_clean":"2024-08-17T17:45:48.643489+0000","last_became_active":"2024-08-17T17:41:34.613706+0000","last_became_peered":"2024-08-17T17:41:34.613706+0000","last_unstale":"2024-08-17T17:45:48.643489+0000","last_undegraded":"2024-08-17T17:45:48.643489+0000","last_fullsized":"2024-08-17T17:45:48.643489+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:19:55.254902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"90'149","reported_seq":723,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.738869+0000","last_change":"2024-08-17T17:42:07.681204+0000","last_active":"2024-08-17T17:46:01.738869+0000","last_peered":"2024-08-17T17:46:01.738869+0000","last_clean":"2024-08-17T17:46:01.738869+0000","last_became_active":"2024-08-17T17:42:05.878769+0000","last_became_peered":"2024-08-17T17:42:05.878769+0000","last_unstale":"2024-08-17T17:46:01.738869+0000","last_undegraded":"2024-08-17T17:46:01.738869+0000","last_fullsized":"2024-08-17T17:46:01.738869+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:07.681064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:07.681064+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:12:12.835715+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":57,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.644764+0000","last_change":"2024-08-17T17:41:34.622180+0000","last_active":"2024-08-17T17:45:48.644764+0000","last_peered":"2024-08-17T17:45:48.644764+0000","last_clean":"2024-08-17T17:45:48.644764+0000","last_became_active":"2024-08-17T17:41:34.621724+0000","last_became_peered":"2024-08-17T17:41:34.621724+0000","last_unstale":"2024-08-17T17:45:48.644764+0000","last_undegraded":"2024-08-17T17:45:48.644764+0000","last_fullsized":"2024-08-17T17:45:48.644764+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:15.807235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":67,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.648300+0000","last_change":"2024-08-17T17:41:34.626584+0000","last_active":"2024-08-17T17:45:48.648300+0000","last_peered":"2024-08-17T17:45:48.648300+0000","last_clean":"2024-08-17T17:45:48.648300+0000","last_became_active":"2024-08-17T17:41:34.609878+0000","last_became_peered":"2024-08-17T17:41:34.609878+0000","last_unstale":"2024-08-17T17:45:48.648300+0000","last_undegraded":"2024-08-17T17:45:48.648300+0000","last_fullsized":"2024-08-17T17:45:48.648300+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:08:12.810347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.17","version":"90'149","reported_seq":482,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.734881+0000","last_change":"2024-08-17T17:42:06.699082+0000","last_active":"2024-08-17T17:46:01.734881+0000","last_peered":"2024-08-17T17:46:01.734881+0000","last_clean":"2024-08-17T17:46:01.734881+0000","last_became_active":"2024-08-17T17:42:05.876218+0000","last_became_peered":"2024-08-17T17:42:05.876218+0000","last_unstale":"2024-08-17T17:46:01.734881+0000","last_undegraded":"2024-08-17T17:46:01.734881+0000","last_fullsized":"2024-08-17T17:46:01.734881+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.698998+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:06.698998+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.698998+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:22:16.510150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":611,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.740539+0000","last_change":"2024-08-17T17:33:20.799402+0000","last_active":"2024-08-17T17:46:01.740539+0000","last_peered":"2024-08-17T17:46:01.740539+0000","last_clean":"2024-08-17T17:46:01.740539+0000","last_became_active":"2024-08-17T17:33:20.798832+0000","last_became_peered":"2024-08-17T17:33:20.798832+0000","last_unstale":"2024-08-17T17:46:01.740539+0000","last_undegraded":"2024-08-17T17:46:01.740539+0000","last_fullsized":"2024-08-17T17:46:01.740539+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:21:05.001134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":386,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.724510+0000","last_change":"2024-08-17T17:41:34.634301+0000","last_active":"2024-08-17T17:46:01.724510+0000","last_peered":"2024-08-17T17:46:01.724510+0000","last_clean":"2024-08-17T17:46:01.724510+0000","last_became_active":"2024-08-17T17:41:34.620508+0000","last_became_peered":"2024-08-17T17:41:34.620508+0000","last_unstale":"2024-08-17T17:46:01.724510+0000","last_undegraded":"2024-08-17T17:46:01.724510+0000","last_fullsized":"2024-08-17T17:46:01.724510+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:10:54.576812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.14","version":"90'149","reported_seq":836,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.719643+0000","last_change":"2024-08-17T17:41:37.664471+0000","last_active":"2024-08-17T17:46:01.719643+0000","last_peered":"2024-08-17T17:46:01.719643+0000","last_clean":"2024-08-17T17:46:01.719643+0000","last_became_active":"2024-08-17T17:41:37.664108+0000","last_became_peered":"2024-08-17T17:41:37.664108+0000","last_unstale":"2024-08-17T17:46:01.719643+0000","last_undegraded":"2024-08-17T17:46:01.719643+0000","last_fullsized":"2024-08-17T17:46:01.719643+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:22.030514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:22.030514+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:15:52.325163+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":350,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.739103+0000","last_change":"2024-08-17T17:33:44.146014+0000","last_active":"2024-08-17T17:46:01.739103+0000","last_peered":"2024-08-17T17:46:01.739103+0000","last_clean":"2024-08-17T17:46:01.739103+0000","last_became_active":"2024-08-17T17:33:44.145668+0000","last_became_peered":"2024-08-17T17:33:44.145668+0000","last_unstale":"2024-08-17T17:46:01.739103+0000","last_undegraded":"2024-08-17T17:46:01.739103+0000","last_fullsized":"2024-08-17T17:46:01.739103+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:53:41.679237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":615,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.723748+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:46:01.723748+0000","last_peered":"2024-08-17T17:46:01.723748+0000","last_clean":"2024-08-17T17:46:01.723748+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:46:01.723748+0000","last_undegraded":"2024-08-17T17:46:01.723748+0000","last_fullsized":"2024-08-17T17:46:01.723748+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"90'149","reported_seq":779,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.635022+0000","last_change":"2024-08-17T17:42:06.456562+0000","last_active":"2024-08-17T17:45:48.635022+0000","last_peered":"2024-08-17T17:45:48.635022+0000","last_clean":"2024-08-17T17:45:48.635022+0000","last_became_active":"2024-08-17T17:42:05.876577+0000","last_became_peered":"2024-08-17T17:42:05.876577+0000","last_unstale":"2024-08-17T17:45:48.635022+0000","last_undegraded":"2024-08-17T17:45:48.635022+0000","last_fullsized":"2024-08-17T17:45:48.635022+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.456490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.456490+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:10:03.654953+0000","scrub_duration":0.001537285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":642,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.735184+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:46:01.735184+0000","last_peered":"2024-08-17T17:46:01.735184+0000","last_clean":"2024-08-17T17:46:01.735184+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:46:01.735184+0000","last_undegraded":"2024-08-17T17:46:01.735184+0000","last_fullsized":"2024-08-17T17:46:01.735184+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":673,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.722678+0000","last_change":"2024-08-17T17:33:20.790687+0000","last_active":"2024-08-17T17:46:01.722678+0000","last_peered":"2024-08-17T17:46:01.722678+0000","last_clean":"2024-08-17T17:46:01.722678+0000","last_became_active":"2024-08-17T17:33:20.790399+0000","last_became_peered":"2024-08-17T17:33:20.790399+0000","last_unstale":"2024-08-17T17:46:01.722678+0000","last_undegraded":"2024-08-17T17:46:01.722678+0000","last_fullsized":"2024-08-17T17:46:01.722678+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:38:06.849989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1a","version":"90'149","reported_seq":441,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.644034+0000","last_change":"2024-08-17T17:42:21.932830+0000","last_active":"2024-08-17T17:45:48.644034+0000","last_peered":"2024-08-17T17:45:48.644034+0000","last_clean":"2024-08-17T17:45:48.644034+0000","last_became_active":"2024-08-17T17:42:05.876835+0000","last_became_peered":"2024-08-17T17:42:05.876835+0000","last_unstale":"2024-08-17T17:45:48.644034+0000","last_undegraded":"2024-08-17T17:45:48.644034+0000","last_fullsized":"2024-08-17T17:45:48.644034+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:21.932757+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:21.932757+0000","last_clean_scrub_stamp":"2024-08-17T17:42:21.932757+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:36:39.848570+0000","scrub_duration":0.0020218810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":366,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.653309+0000","last_change":"2024-08-17T17:33:20.835760+0000","last_active":"2024-08-17T17:45:48.653309+0000","last_peered":"2024-08-17T17:45:48.653309+0000","last_clean":"2024-08-17T17:45:48.653309+0000","last_became_active":"2024-08-17T17:33:20.835410+0000","last_became_peered":"2024-08-17T17:33:20.835410+0000","last_unstale":"2024-08-17T17:45:48.653309+0000","last_undegraded":"2024-08-17T17:45:48.653309+0000","last_fullsized":"2024-08-17T17:45:48.653309+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:45.448022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":690,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.724183+0000","last_change":"2024-08-17T17:33:20.771279+0000","last_active":"2024-08-17T17:46:01.724183+0000","last_peered":"2024-08-17T17:46:01.724183+0000","last_clean":"2024-08-17T17:46:01.724183+0000","last_became_active":"2024-08-17T17:33:20.770728+0000","last_became_peered":"2024-08-17T17:33:20.770728+0000","last_unstale":"2024-08-17T17:46:01.724183+0000","last_undegraded":"2024-08-17T17:46:01.724183+0000","last_fullsized":"2024-08-17T17:46:01.724183+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:52:31.540397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"90'149","reported_seq":736,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.644626+0000","last_change":"2024-08-17T17:42:26.967686+0000","last_active":"2024-08-17T17:45:48.644626+0000","last_peered":"2024-08-17T17:45:48.644626+0000","last_clean":"2024-08-17T17:45:48.644626+0000","last_became_active":"2024-08-17T17:42:05.873657+0000","last_became_peered":"2024-08-17T17:42:05.873657+0000","last_unstale":"2024-08-17T17:45:48.644626+0000","last_undegraded":"2024-08-17T17:45:48.644626+0000","last_fullsized":"2024-08-17T17:45:48.644626+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:26.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:26.967611+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:31:03.344332+0000","scrub_duration":0.0015978279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":460,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.735372+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:46:01.735372+0000","last_peered":"2024-08-17T17:46:01.735372+0000","last_clean":"2024-08-17T17:46:01.735372+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:46:01.735372+0000","last_undegraded":"2024-08-17T17:46:01.735372+0000","last_fullsized":"2024-08-17T17:46:01.735372+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":610,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.737916+0000","last_change":"2024-08-17T17:41:34.625011+0000","last_active":"2024-08-17T17:46:01.737916+0000","last_peered":"2024-08-17T17:46:01.737916+0000","last_clean":"2024-08-17T17:46:01.737916+0000","last_became_active":"2024-08-17T17:41:34.624631+0000","last_became_peered":"2024-08-17T17:41:34.624631+0000","last_unstale":"2024-08-17T17:46:01.737916+0000","last_undegraded":"2024-08-17T17:46:01.737916+0000","last_fullsized":"2024-08-17T17:46:01.737916+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:03:05.565435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"90'149","reported_seq":699,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.651922+0000","last_change":"2024-08-17T17:42:08.482975+0000","last_active":"2024-08-17T17:45:48.651922+0000","last_peered":"2024-08-17T17:45:48.651922+0000","last_clean":"2024-08-17T17:45:48.651922+0000","last_became_active":"2024-08-17T17:42:05.873523+0000","last_became_peered":"2024-08-17T17:42:05.873523+0000","last_unstale":"2024-08-17T17:45:48.651922+0000","last_undegraded":"2024-08-17T17:45:48.651922+0000","last_fullsized":"2024-08-17T17:45:48.651922+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:08.482892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:08.482892+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:41.477341+0000","scrub_duration":0.0017925479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":57,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.644291+0000","last_change":"2024-08-17T17:41:34.616482+0000","last_active":"2024-08-17T17:45:48.644291+0000","last_peered":"2024-08-17T17:45:48.644291+0000","last_clean":"2024-08-17T17:45:48.644291+0000","last_became_active":"2024-08-17T17:41:34.616040+0000","last_became_peered":"2024-08-17T17:41:34.616040+0000","last_unstale":"2024-08-17T17:45:48.644291+0000","last_undegraded":"2024-08-17T17:45:48.644291+0000","last_fullsized":"2024-08-17T17:45:48.644291+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:53:08.502264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":57,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.644232+0000","last_change":"2024-08-17T17:41:34.622184+0000","last_active":"2024-08-17T17:45:48.644232+0000","last_peered":"2024-08-17T17:45:48.644232+0000","last_clean":"2024-08-17T17:45:48.644232+0000","last_became_active":"2024-08-17T17:41:34.621831+0000","last_became_peered":"2024-08-17T17:41:34.621831+0000","last_unstale":"2024-08-17T17:45:48.644232+0000","last_undegraded":"2024-08-17T17:45:48.644232+0000","last_fullsized":"2024-08-17T17:45:48.644232+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:59.826667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"90'149","reported_seq":840,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.722738+0000","last_change":"2024-08-17T17:42:05.876463+0000","last_active":"2024-08-17T17:46:01.722738+0000","last_peered":"2024-08-17T17:46:01.722738+0000","last_clean":"2024-08-17T17:46:01.722738+0000","last_became_active":"2024-08-17T17:42:05.876069+0000","last_became_peered":"2024-08-17T17:42:05.876069+0000","last_unstale":"2024-08-17T17:46:01.722738+0000","last_undegraded":"2024-08-17T17:46:01.722738+0000","last_fullsized":"2024-08-17T17:46:01.722738+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:45.552985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:45.552985+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:55:06.027278+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":577,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.738957+0000","last_change":"2024-08-17T17:33:20.773207+0000","last_active":"2024-08-17T17:46:01.738957+0000","last_peered":"2024-08-17T17:46:01.738957+0000","last_clean":"2024-08-17T17:46:01.738957+0000","last_became_active":"2024-08-17T17:33:20.772457+0000","last_became_peered":"2024-08-17T17:33:20.772457+0000","last_unstale":"2024-08-17T17:46:01.738957+0000","last_undegraded":"2024-08-17T17:46:01.738957+0000","last_fullsized":"2024-08-17T17:46:01.738957+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:57:09.610084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":483,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.736528+0000","last_change":"2024-08-17T17:41:34.618968+0000","last_active":"2024-08-17T17:46:01.736528+0000","last_peered":"2024-08-17T17:46:01.736528+0000","last_clean":"2024-08-17T17:46:01.736528+0000","last_became_active":"2024-08-17T17:41:34.618576+0000","last_became_peered":"2024-08-17T17:41:34.618576+0000","last_unstale":"2024-08-17T17:46:01.736528+0000","last_undegraded":"2024-08-17T17:46:01.736528+0000","last_fullsized":"2024-08-17T17:46:01.736528+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:47:45.888590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"90'149","reported_seq":488,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.723546+0000","last_change":"2024-08-17T17:42:06.742801+0000","last_active":"2024-08-17T17:46:01.723546+0000","last_peered":"2024-08-17T17:46:01.723546+0000","last_clean":"2024-08-17T17:46:01.723546+0000","last_became_active":"2024-08-17T17:42:05.876895+0000","last_became_peered":"2024-08-17T17:42:05.876895+0000","last_unstale":"2024-08-17T17:46:01.723546+0000","last_undegraded":"2024-08-17T17:46:01.723546+0000","last_fullsized":"2024-08-17T17:46:01.723546+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.742723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.742723+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:36:46.651953+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":57,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.644556+0000","last_change":"2024-08-17T17:41:34.622094+0000","last_active":"2024-08-17T17:45:48.644556+0000","last_peered":"2024-08-17T17:45:48.644556+0000","last_clean":"2024-08-17T17:45:48.644556+0000","last_became_active":"2024-08-17T17:41:34.621773+0000","last_became_peered":"2024-08-17T17:41:34.621773+0000","last_unstale":"2024-08-17T17:45:48.644556+0000","last_undegraded":"2024-08-17T17:45:48.644556+0000","last_fullsized":"2024-08-17T17:45:48.644556+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:34:30.540886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":61,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.648429+0000","last_change":"2024-08-17T17:41:34.629135+0000","last_active":"2024-08-17T17:45:48.648429+0000","last_peered":"2024-08-17T17:45:48.648429+0000","last_clean":"2024-08-17T17:45:48.648429+0000","last_became_active":"2024-08-17T17:41:34.613095+0000","last_became_peered":"2024-08-17T17:41:34.613095+0000","last_unstale":"2024-08-17T17:45:48.648429+0000","last_undegraded":"2024-08-17T17:45:48.648429+0000","last_fullsized":"2024-08-17T17:45:48.648429+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:27:06.542416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":57,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.644311+0000","last_change":"2024-08-17T17:41:34.616251+0000","last_active":"2024-08-17T17:45:48.644311+0000","last_peered":"2024-08-17T17:45:48.644311+0000","last_clean":"2024-08-17T17:45:48.644311+0000","last_became_active":"2024-08-17T17:41:34.615952+0000","last_became_peered":"2024-08-17T17:41:34.615952+0000","last_unstale":"2024-08-17T17:45:48.644311+0000","last_undegraded":"2024-08-17T17:45:48.644311+0000","last_fullsized":"2024-08-17T17:45:48.644311+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:12:24.892508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":61,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.643892+0000","last_change":"2024-08-17T17:41:34.631152+0000","last_active":"2024-08-17T17:45:48.643892+0000","last_peered":"2024-08-17T17:45:48.643892+0000","last_clean":"2024-08-17T17:45:48.643892+0000","last_became_active":"2024-08-17T17:41:34.615617+0000","last_became_peered":"2024-08-17T17:41:34.615617+0000","last_unstale":"2024-08-17T17:45:48.643892+0000","last_undegraded":"2024-08-17T17:45:48.643892+0000","last_fullsized":"2024-08-17T17:45:48.643892+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:43:08.257084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"90'149","reported_seq":733,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.646016+0000","last_change":"2024-08-17T17:42:11.809047+0000","last_active":"2024-08-17T17:45:48.646016+0000","last_peered":"2024-08-17T17:45:48.646016+0000","last_clean":"2024-08-17T17:45:48.646016+0000","last_became_active":"2024-08-17T17:42:05.883770+0000","last_became_peered":"2024-08-17T17:42:05.883770+0000","last_unstale":"2024-08-17T17:45:48.646016+0000","last_undegraded":"2024-08-17T17:45:48.646016+0000","last_fullsized":"2024-08-17T17:45:48.646016+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:11.808971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:11.808971+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:48:15.989437+0000","scrub_duration":0.0018167420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":668,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.724106+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:46:01.724106+0000","last_peered":"2024-08-17T17:46:01.724106+0000","last_clean":"2024-08-17T17:46:01.724106+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:46:01.724106+0000","last_undegraded":"2024-08-17T17:46:01.724106+0000","last_fullsized":"2024-08-17T17:46:01.724106+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":670,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.723638+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:46:01.723638+0000","last_peered":"2024-08-17T17:46:01.723638+0000","last_clean":"2024-08-17T17:46:01.723638+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:46:01.723638+0000","last_undegraded":"2024-08-17T17:46:01.723638+0000","last_fullsized":"2024-08-17T17:46:01.723638+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1d","version":"90'149","reported_seq":735,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.738086+0000","last_change":"2024-08-17T17:42:18.668058+0000","last_active":"2024-08-17T17:46:01.738086+0000","last_peered":"2024-08-17T17:46:01.738086+0000","last_clean":"2024-08-17T17:46:01.738086+0000","last_became_active":"2024-08-17T17:42:05.876721+0000","last_became_peered":"2024-08-17T17:42:05.876721+0000","last_unstale":"2024-08-17T17:46:01.738086+0000","last_undegraded":"2024-08-17T17:46:01.738086+0000","last_fullsized":"2024-08-17T17:46:01.738086+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:18.667957+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:18.667957+0000","last_clean_scrub_stamp":"2024-08-17T17:42:18.667957+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:38.170146+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":57,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-08-17T17:45:48.643539+0000","last_change":"2024-08-17T17:41:34.611711+0000","last_active":"2024-08-17T17:45:48.643539+0000","last_peered":"2024-08-17T17:45:48.643539+0000","last_clean":"2024-08-17T17:45:48.643539+0000","last_became_active":"2024-08-17T17:41:34.611407+0000","last_became_peered":"2024-08-17T17:41:34.611407+0000","last_unstale":"2024-08-17T17:45:48.643539+0000","last_undegraded":"2024-08-17T17:45:48.643539+0000","last_fullsized":"2024-08-17T17:45:48.643539+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:42:41.582320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":479,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.735084+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:46:01.735084+0000","last_peered":"2024-08-17T17:46:01.735084+0000","last_clean":"2024-08-17T17:46:01.735084+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:46:01.735084+0000","last_undegraded":"2024-08-17T17:46:01.735084+0000","last_fullsized":"2024-08-17T17:46:01.735084+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":996278296,"num_objects":15203,"num_object_clones":0,"num_object_copies":45609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2566,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15203,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":221885,"num_write_kb":7586753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4234,"num_bytes_recovered":251002880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3370790912,"data_stored":3370778696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102385,"ondisk_log_size":102385,"up":48,"acting":47,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":11256,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19280,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5978,"num_read_kb":12991,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4619,"ondisk_log_size":4619,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234471,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1177652,"kb_used_data":279592,"kb_used_omap":1,"kb_used_meta":898046,"kb_avail":92567500,"statfs":{"total":95995035648,"available":94789120000,"internally_reserved":0,"allocated":286302208,"data_stored":284063671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":919599448},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496106,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1366180,"kb_used_data":471068,"kb_used_omap":1,"kb_used_meta":895102,"kb_avail":92378972,"statfs":{"total":95995035648,"available":94596067328,"internally_reserved":0,"allocated":482373632,"data_stored":480146537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":916584792},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757743,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1241056,"kb_used_data":350424,"kb_used_omap":1,"kb_used_meta":890622,"kb_avail":92504096,"statfs":{"total":95995035648,"available":94724194304,"internally_reserved":0,"allocated":358834176,"data_stored":356607722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":911997272},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019381,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":860720,"kb_used_data":559208,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":92884432,"statfs":{"total":95995035648,"available":95113658368,"internally_reserved":0,"allocated":572628992,"data_stored":570390196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435608,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1256600,"kb_used_data":318672,"kb_used_omap":4,"kb_used_meta":937915,"kb_avail":92488552,"statfs":{"total":95995035648,"available":94708277248,"internally_reserved":0,"allocated":326320128,"data_stored":324101104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":960424970},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599135,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1590624,"kb_used_data":754840,"kb_used_omap":7,"kb_used_meta":835768,"kb_avail":92154528,"statfs":{"total":95995035648,"available":94366236672,"internally_reserved":0,"allocated":772956160,"data_stored":770730002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":855826594},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":268,"seq":1151051235482,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":622168,"kb_used_data":176080,"kb_used_omap":10,"kb_used_meta":446069,"kb_avail":93122984,"statfs":{"total":95995035648,"available":95357935616,"internally_reserved":0,"allocated":180305920,"data_stored":178086841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10902,"internal_metadata":456775018},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926189,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1476768,"kb_used_data":550552,"kb_used_omap":7,"kb_used_meta":926136,"kb_avail":92268384,"statfs":{"total":95995035648,"available":94482825216,"internally_reserved":0,"allocated":563765248,"data_stored":561538566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":948363585},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7550,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":527765504,"data_stored":527761432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":173539328,"data_stored":173539328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":745603072,"data_stored":745603072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":316145664,"data_stored":316145664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":538120192,"data_stored":538116120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":335609856,"data_stored":335609856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":459476992,"data_stored":459472920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":274530304,"data_stored":274530304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:46:11.004 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:11.049 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:11.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:11.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:11.153 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:11.207 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:11.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:11.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:11.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:11.337 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:11.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:11.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:11.439 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:11.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:11 smithi053 ceph-mon[89963]: pgmap v1312: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 740 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 992 op/s; 2304/35385 objects degraded (6.511%); 0 B/s, 48 objects/s recovering 2024-08-17T17:46:11.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:11.485 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:11.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:11.522 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:11 smithi003 ceph-mon[103344]: pgmap v1312: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 740 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 992 op/s; 2304/35385 objects degraded (6.511%); 0 B/s, 48 objects/s recovering 2024-08-17T17:46:11.522 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:46:11.522 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:46:11.522 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:11 smithi003 ceph-mon[100615]: pgmap v1312: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 740 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 992 op/s; 2304/35385 objects degraded (6.511%); 0 B/s, 48 objects/s recovering 2024-08-17T17:46:11.586 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:11.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:11.622 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:11.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:11.723 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:11.755 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:11.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:11.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:11.857 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:11.900 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:11.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:11.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:12.002 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:12.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:12.043 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:12.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:12.144 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:12.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:12.183 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:12.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:12.283 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:12.315 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:12.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:12.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:12.417 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:12.455 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:12.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:12.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:12.557 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:12.590 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:12.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:12.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:12.692 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:12.726 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:12.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:12.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:12.828 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:12.869 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:12.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:12.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:12.971 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:13.009 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:13.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:13.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:13.111 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:13.150 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:13.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:13.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:13.252 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:13.285 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:13.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:13.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:13.386 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:13.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:13.431 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:13.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:13.532 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:13.563 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:13.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:13.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:13.665 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:13.699 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:13.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:13.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:13.700 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:13 smithi003 ceph-mon[103344]: pgmap v1313: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 645 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 2304/30816 objects degraded (7.477%); 0 B/s, 45 objects/s recovering 2024-08-17T17:46:13.701 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:13 smithi003 ceph-mon[100615]: pgmap v1313: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 645 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 2304/30816 objects degraded (7.477%); 0 B/s, 45 objects/s recovering 2024-08-17T17:46:13.745 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-17T17:46:13.800 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:13.842 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:13.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:13.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:13.917 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:46:13.944 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:13.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:13 smithi053 ceph-mon[89963]: pgmap v1313: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 645 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 2304/30816 objects degraded (7.477%); 0 B/s, 45 objects/s recovering 2024-08-17T17:46:14.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:14.036 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:14.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:14.137 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:14.163 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:14.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:14.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:14.264 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:14.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:14.317 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:14.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:14.418 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:14.464 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:14.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:14.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:14.566 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:14.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:14.601 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:14.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:14.635 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:46:14.635 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:46:14.679 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:46:14.679 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:46:14.703 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:14.715 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:46:14.715 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:46:14.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:14.746 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:14.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:14.758 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:46:14.759 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:46:14.792 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:46:14.793 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:46:14.826 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:46:14.826 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:46:14.847 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:14.869 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:46:14.869 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:46:14.891 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:14.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:14.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:14.901 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:46:14.901 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:46:14.993 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:15.022 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:15.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:15.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:15.124 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:15.157 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:15.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:15.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:15.259 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:15.303 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:15.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:15.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:15.405 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:15.450 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:15.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:15.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:15.551 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:15.591 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:15.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:15.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:15.693 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:15.742 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:15.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:15.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:15.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:15 smithi003 ceph-mon[103344]: pgmap v1314: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 445 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 2077/21246 objects degraded (9.776%); 0 B/s, 59 objects/s recovering 2024-08-17T17:46:15.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:15 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3974350999' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:46:15.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:15 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 2077/21246 objects degraded (9.776%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:46:15.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:15 smithi003 ceph-mon[100615]: pgmap v1314: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 445 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 2077/21246 objects degraded (9.776%); 0 B/s, 59 objects/s recovering 2024-08-17T17:46:15.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:15 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3974350999' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:46:15.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:15 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 2077/21246 objects degraded (9.776%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:46:15.774 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:46:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:46:15] "GET /metrics HTTP/1.1" 200 36570 "" "Prometheus/2.51.0" 2024-08-17T17:46:15.844 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:15.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:15.884 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:15.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:15.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:15 smithi053 ceph-mon[89963]: pgmap v1314: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 445 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 2077/21246 objects degraded (9.776%); 0 B/s, 59 objects/s recovering 2024-08-17T17:46:15.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:15 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3974350999' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:46:15.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:15 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 2077/21246 objects degraded (9.776%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:46:15.985 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:16.016 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:16.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:16.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:16.118 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:16.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:16.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:16.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:16.255 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:16.297 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:16.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:16.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:16.399 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:16.412 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:46:16.429 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:16.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:16.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:16.434 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:46:16.440 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:46:16.531 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:16.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:16.581 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:16.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:16.682 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:16.716 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:16.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:16.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:16.818 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:16.852 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:16.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:16.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:16.954 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:16.996 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:16.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:16.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:17.099 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:17.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:17.139 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:17.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:17.240 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:17.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:17.287 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:17.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:17.388 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:17.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:17.431 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:17.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:17.532 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:17.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:17.566 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:17.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:17.629 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:17 smithi003 ceph-mon[103344]: pgmap v1315: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 407 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 887 op/s; 2077/19395 objects degraded (10.709%); 0 B/s, 40 objects/s recovering 2024-08-17T17:46:17.629 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:17 smithi003 ceph-mon[100615]: pgmap v1315: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 407 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 887 op/s; 2077/19395 objects degraded (10.709%); 0 B/s, 40 objects/s recovering 2024-08-17T17:46:17.667 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:17.708 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:17.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:17.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:17.810 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:17.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:17.845 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:17.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:17.946 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:17.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:17 smithi053 ceph-mon[89963]: pgmap v1315: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 407 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 887 op/s; 2077/19395 objects degraded (10.709%); 0 B/s, 40 objects/s recovering 2024-08-17T17:46:17.989 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:17.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:17.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:18.020 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:46:18.091 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:18.140 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:18.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:18.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:18.241 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:18.277 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:18.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:18.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:18.378 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:18.425 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:18.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:18.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:18.526 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:18.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:18.568 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:18.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:18.669 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:18.706 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:18.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:18.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:18.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:18.840 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:18.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:18.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:18.942 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:18.976 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:18.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:18.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:19.078 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:19.118 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:19.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:19.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:19.220 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:19.255 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:19.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:19.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:19.357 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:19.387 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:19.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:19.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:19.489 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:19.531 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:19.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:19.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:19.633 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:19.668 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:19.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:19.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:19.770 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:19.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:19 smithi003 ceph-mon[103344]: pgmap v1316: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 329 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 941 op/s; 2077/15654 objects degraded (13.268%); 0 B/s, 40 objects/s recovering 2024-08-17T17:46:19.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:19 smithi003 ceph-mon[100615]: pgmap v1316: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 329 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 941 op/s; 2077/15654 objects degraded (13.268%); 0 B/s, 40 objects/s recovering 2024-08-17T17:46:19.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:19.806 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:19.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:19.907 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:19.952 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:19.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:19.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:19.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:19 smithi053 ceph-mon[89963]: pgmap v1316: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 329 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 941 op/s; 2077/15654 objects degraded (13.268%); 0 B/s, 40 objects/s recovering 2024-08-17T17:46:20.054 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:20.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:20.094 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:20.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:20.115 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Removed 118543 objects 2024-08-17T17:46:20.115 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Clean up completed and total clean up time :119.559 2024-08-17T17:46:20.195 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:20.217 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:20.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:20.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:20.319 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:20.341 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:20.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:20.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:20.443 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:20.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:46:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:46:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:46:20.467 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:20.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:20.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:20.568 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:20.590 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:20.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:20.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:20.692 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:20.714 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:20.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:20.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:20.807 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:20 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 2077/15654 objects degraded (13.268%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:46:20.807 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:20 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 2077/15654 objects degraded (13.268%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:46:20.815 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:20.838 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:20.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:20.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:20.939 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:20.963 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:20.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:20.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:20.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:20 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 2077/15654 objects degraded (13.268%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:46:21.065 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:21.087 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:21.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:21.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:21.189 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:21.211 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:21.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:21.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:21.313 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:21.336 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:21.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:21.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:21.437 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:21.460 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:21.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:21.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:21.562 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:21.584 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:21.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:21.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:21.685 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:21.708 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:21.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:21.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:21.809 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:21.831 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:21.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:21.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:21.933 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:21.949 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:21 smithi003 ceph-mon[100615]: pgmap v1317: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 116 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s; 1736/5412 objects degraded (32.077%); 0 B/s, 69 objects/s recovering 2024-08-17T17:46:21.949 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:21 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:46:21.949 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:21 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:46:21.949 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:21 smithi003 ceph-mon[103344]: pgmap v1317: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 116 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s; 1736/5412 objects degraded (32.077%); 0 B/s, 69 objects/s recovering 2024-08-17T17:46:21.949 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:21 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:46:21.949 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:21 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:46:21.957 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:21.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:21.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:21.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:21 smithi053 ceph-mon[89963]: pgmap v1317: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 116 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 1.12k op/s; 1736/5412 objects degraded (32.077%); 0 B/s, 69 objects/s recovering 2024-08-17T17:46:21.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:21 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:46:21.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:21 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:46:22.059 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:22.081 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:22.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:22.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:22.183 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:22.206 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:22.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:22.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:22.308 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:22.331 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:22.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:22.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:22.401 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:46:22.401 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-08-17T17:46:22.401 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-08-17T17:46:22.401 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-08-17T17:46:22.401 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:46:22.433 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:22.455 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:22.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:22.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:22.557 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:22.579 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:22.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:22.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:22.681 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:22.703 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:22.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:22.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:22.805 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:22.828 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:22.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:22.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:22.930 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:22.931 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:46:22.955 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:22.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:22.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:23.003 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_5 2024-08-17T17:46:23.003 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd pool rm unique_pool_5 unique_pool_5 --yes-i-really-really-mean-it 2024-08-17T17:46:23.057 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:23.080 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:23.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:23.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:23.181 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:23.206 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:23.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:23.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:23.308 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:23.330 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:23.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:23.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:23.432 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:23.454 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:23.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:23.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:23.468 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-17T17:46:23.547 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-08-17T17:46:23.555 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:23.578 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:23.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:23.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:23.680 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:23.704 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:23.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:23.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:23.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:23 smithi003 ceph-mon[100615]: pgmap v1318: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 98 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 855 op/s; 1736/4587 objects degraded (37.846%); 0 B/s, 47 objects/s recovering 2024-08-17T17:46:23.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:23 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/4260614090' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:46:23.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:46:23.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:46:23.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:46:23.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:46:23.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:23 smithi003 ceph-mon[103344]: pgmap v1318: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 98 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 855 op/s; 1736/4587 objects degraded (37.846%); 0 B/s, 47 objects/s recovering 2024-08-17T17:46:23.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:23 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/4260614090' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:46:23.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:46:23.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:46:23.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:46:23.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:46:23.805 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:23.829 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:23.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:23.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:23.931 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:23.953 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:23.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:23.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:23.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:23 smithi053 ceph-mon[89963]: pgmap v1318: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 98 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 855 op/s; 1736/4587 objects degraded (37.846%); 0 B/s, 47 objects/s recovering 2024-08-17T17:46:23.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:23 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/4260614090' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-17T17:46:23.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:46:23.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:46:23.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:46:23.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:46:24.055 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:24.079 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:24.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:24.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:24.180 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:24.203 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:24.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:24.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:24.305 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:24.329 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:24.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:24.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:24.431 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:24.454 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:24.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:24.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:24.556 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:24.578 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:24.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:24.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:24.680 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:24.714 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:24.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:24.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:24.748 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:24 smithi003 ceph-mon[103344]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:46:24.748 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:24 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/4260614090' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:46:24.748 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:24 smithi003 ceph-mon[103344]: osdmap e394: 8 total, 8 up, 8 in 2024-08-17T17:46:24.748 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:24 smithi003 ceph-mon[100615]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:46:24.749 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:24 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/4260614090' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:46:24.749 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:24 smithi003 ceph-mon[100615]: osdmap e394: 8 total, 8 up, 8 in 2024-08-17T17:46:24.816 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:24.838 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:24.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:24.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:24.940 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:24.964 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:24.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:24.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:24.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:24 smithi053 ceph-mon[89963]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:46:24.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:24 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/4260614090' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-17T17:46:24.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:24 smithi053 ceph-mon[89963]: osdmap e394: 8 total, 8 up, 8 in 2024-08-17T17:46:25.066 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:25.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:25.090 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:25.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:25.191 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:25.200 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:46:25.216 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:25.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:25.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:25.317 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:25.341 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:25.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:25.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:25.443 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:25.502 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:25.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:25.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:25.604 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:25.629 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:25.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:25.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:25.731 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:25.753 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:25.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:25.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:25.855 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:25.878 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:25.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:25.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:25.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:25 smithi053 ceph-mon[89963]: pgmap v1320: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 6.0 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 702 op/s; 320/153 objects degraded (209.150%); 0 B/s, 175 objects/s recovering 2024-08-17T17:46:25.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:25 smithi053 ceph-mon[89963]: osdmap e395: 8 total, 8 up, 8 in 2024-08-17T17:46:25.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:25 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 320/153 objects degraded (209.150%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:46:25.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:26.002 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:26.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:26.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:26.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:25 smithi003 ceph-mon[103344]: pgmap v1320: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 6.0 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 702 op/s; 320/153 objects degraded (209.150%); 0 B/s, 175 objects/s recovering 2024-08-17T17:46:26.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:25 smithi003 ceph-mon[103344]: osdmap e395: 8 total, 8 up, 8 in 2024-08-17T17:46:26.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:25 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 320/153 objects degraded (209.150%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:46:26.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:25 smithi003 ceph-mon[100615]: pgmap v1320: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 6.0 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 702 op/s; 320/153 objects degraded (209.150%); 0 B/s, 175 objects/s recovering 2024-08-17T17:46:26.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:25 smithi003 ceph-mon[100615]: osdmap e395: 8 total, 8 up, 8 in 2024-08-17T17:46:26.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:25 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 320/153 objects degraded (209.150%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:46:26.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:46:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:46:25] "GET /metrics HTTP/1.1" 200 36556 "" "Prometheus/2.51.0" 2024-08-17T17:46:26.103 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:26.128 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:26.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:26.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:26.230 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:26.253 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:26.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:26.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:26.355 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:26.378 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:26.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:26.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:26.478 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:46:26.480 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:46:26.481 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:26.505 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:26.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:26.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:26.607 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:26.630 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:26.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:26.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:26.731 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:26.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:26.754 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:26.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:26.855 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:26.877 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:26.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:26.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:26.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:26 smithi003 ceph-mon[103344]: osdmap e396: 8 total, 8 up, 8 in 2024-08-17T17:46:26.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:26 smithi003 ceph-mon[100615]: osdmap e396: 8 total, 8 up, 8 in 2024-08-17T17:46:26.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:26 smithi053 ceph-mon[89963]: osdmap e396: 8 total, 8 up, 8 in 2024-08-17T17:46:26.980 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:27.006 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:27.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:27.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:27.108 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:27.134 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:27.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:27.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:27.165 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:46:27.165 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:46:27.165 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:46:27.165 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:46:27.165 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:46:27.165 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:46:27.165 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:46:27.166 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:46:27.166 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:46:27.166 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:46:27.166 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:46:27.166 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:46:27.166 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:46:27.166 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:46:27.167 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:46:27.167 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:46:27.236 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:27.262 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:27.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:27.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:27.364 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:27.386 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:27.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:27.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:27.488 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:27.514 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:27.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:27.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:27.616 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:27.643 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:27.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:27.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:27.744 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:27.771 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:27.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:27.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:27.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:27 smithi003 ceph-mon[100615]: pgmap v1323: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 295 op/s; 320/84 objects degraded (380.952%); 0 B/s, 235 objects/s recovering 2024-08-17T17:46:27.772 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:27 smithi003 ceph-mon[100615]: from='client.46799 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:46:27.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:27 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3887846654' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:46:27.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:27 smithi003 ceph-mon[103344]: pgmap v1323: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 295 op/s; 320/84 objects degraded (380.952%); 0 B/s, 235 objects/s recovering 2024-08-17T17:46:27.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:27 smithi003 ceph-mon[103344]: from='client.46799 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:46:27.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:27 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3887846654' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:46:27.802 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1323,"stamp":"2024-08-17T17:46:26.437533+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790221,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":5979,"num_read_kb":12992,"num_write":237995,"num_write_kb":7599376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6674,"num_bytes_recovered":259402744,"num_keys_recovered":40,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107939,"ondisk_log_size":107939,"up":357,"acting":356,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":359,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6594152,"kb_used_data":432336,"kb_used_omap":37,"kb_used_meta":6161690,"kb_avail":743367064,"statfs":{"total":767960285184,"available":761207873536,"internally_reserved":0,"allocated":442712064,"data_stored":424878959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38729,"internal_metadata":6309570743},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-116326424,"num_objects":-1776,"num_object_clones":0,"num_object_copies":-5328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1416,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1776,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1416,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001799"},"pg_stats":[{"pgid":"1.11","version":"90'149","reported_seq":838,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.734617+0000","last_change":"2024-08-17T17:38:30.378664+0000","last_active":"2024-08-17T17:46:01.734617+0000","last_peered":"2024-08-17T17:46:01.734617+0000","last_clean":"2024-08-17T17:46:01.734617+0000","last_became_active":"2024-08-17T17:38:30.342287+0000","last_became_peered":"2024-08-17T17:38:30.342287+0000","last_unstale":"2024-08-17T17:46:01.734617+0000","last_undegraded":"2024-08-17T17:46:01.734617+0000","last_fullsized":"2024-08-17T17:46:01.734617+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:30.378592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:30.378592+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:04:28.933354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":663,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.600907+0000","last_change":"2024-08-17T17:33:20.797029+0000","last_active":"2024-08-17T17:46:24.600907+0000","last_peered":"2024-08-17T17:46:24.600907+0000","last_clean":"2024-08-17T17:46:24.600907+0000","last_became_active":"2024-08-17T17:33:20.796679+0000","last_became_peered":"2024-08-17T17:33:20.796679+0000","last_unstale":"2024-08-17T17:46:24.600907+0000","last_undegraded":"2024-08-17T17:46:24.600907+0000","last_fullsized":"2024-08-17T17:46:24.600907+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:51.438262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":418,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.735537+0000","last_change":"2024-08-17T17:41:34.624198+0000","last_active":"2024-08-17T17:46:01.735537+0000","last_peered":"2024-08-17T17:46:01.735537+0000","last_clean":"2024-08-17T17:46:01.735537+0000","last_became_active":"2024-08-17T17:41:34.616071+0000","last_became_peered":"2024-08-17T17:41:34.616071+0000","last_unstale":"2024-08-17T17:46:01.735537+0000","last_undegraded":"2024-08-17T17:46:01.735537+0000","last_fullsized":"2024-08-17T17:46:01.735537+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:34:48.805354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"90'149","reported_seq":477,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.726708+0000","last_change":"2024-08-17T17:38:31.874623+0000","last_active":"2024-08-17T17:46:01.726708+0000","last_peered":"2024-08-17T17:46:01.726708+0000","last_clean":"2024-08-17T17:46:01.726708+0000","last_became_active":"2024-08-17T17:38:30.339313+0000","last_became_peered":"2024-08-17T17:38:30.339313+0000","last_unstale":"2024-08-17T17:46:01.726708+0000","last_undegraded":"2024-08-17T17:46:01.726708+0000","last_fullsized":"2024-08-17T17:46:01.726708+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:31.874423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:31.874423+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:28:17.980339+0000","scrub_duration":0.0017696579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":324,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.600694+0000","last_change":"2024-08-17T17:33:44.156036+0000","last_active":"2024-08-17T17:46:24.600694+0000","last_peered":"2024-08-17T17:46:24.600694+0000","last_clean":"2024-08-17T17:46:24.600694+0000","last_became_active":"2024-08-17T17:33:44.155758+0000","last_became_peered":"2024-08-17T17:33:44.155758+0000","last_unstale":"2024-08-17T17:46:24.600694+0000","last_undegraded":"2024-08-17T17:46:24.600694+0000","last_fullsized":"2024-08-17T17:46:24.600694+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:23.443405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":459,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.734498+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:46:01.734498+0000","last_peered":"2024-08-17T17:46:01.734498+0000","last_clean":"2024-08-17T17:46:01.734498+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:46:01.734498+0000","last_undegraded":"2024-08-17T17:46:01.734498+0000","last_fullsized":"2024-08-17T17:46:01.734498+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"90'149","reported_seq":860,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.872090+0000","last_change":"2024-08-17T17:41:37.681187+0000","last_active":"2024-08-17T17:46:24.872090+0000","last_peered":"2024-08-17T17:46:24.872090+0000","last_clean":"2024-08-17T17:46:24.872090+0000","last_became_active":"2024-08-17T17:41:37.680605+0000","last_became_peered":"2024-08-17T17:41:37.680605+0000","last_unstale":"2024-08-17T17:46:24.872090+0000","last_undegraded":"2024-08-17T17:46:24.872090+0000","last_fullsized":"2024-08-17T17:46:24.872090+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.844252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.844252+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:13:31.682357+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":459,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.735447+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:46:01.735447+0000","last_peered":"2024-08-17T17:46:01.735447+0000","last_clean":"2024-08-17T17:46:01.735447+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:46:01.735447+0000","last_undegraded":"2024-08-17T17:46:01.735447+0000","last_fullsized":"2024-08-17T17:46:01.735447+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":58,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.728176+0000","last_change":"2024-08-17T17:41:34.611693+0000","last_active":"2024-08-17T17:46:01.728176+0000","last_peered":"2024-08-17T17:46:01.728176+0000","last_clean":"2024-08-17T17:46:01.728176+0000","last_became_active":"2024-08-17T17:41:34.611374+0000","last_became_peered":"2024-08-17T17:41:34.611374+0000","last_unstale":"2024-08-17T17:46:01.728176+0000","last_undegraded":"2024-08-17T17:46:01.728176+0000","last_fullsized":"2024-08-17T17:46:01.728176+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:17:09.897446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"90'149","reported_seq":836,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.872207+0000","last_change":"2024-08-17T17:38:30.446165+0000","last_active":"2024-08-17T17:46:24.872207+0000","last_peered":"2024-08-17T17:46:24.872207+0000","last_clean":"2024-08-17T17:46:24.872207+0000","last_became_active":"2024-08-17T17:38:30.445048+0000","last_became_peered":"2024-08-17T17:38:30.445048+0000","last_unstale":"2024-08-17T17:46:24.872207+0000","last_undegraded":"2024-08-17T17:46:24.872207+0000","last_fullsized":"2024-08-17T17:46:24.872207+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:24.291592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:24.291592+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:17.100154+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":676,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.600812+0000","last_change":"2024-08-17T17:33:20.794682+0000","last_active":"2024-08-17T17:46:24.600812+0000","last_peered":"2024-08-17T17:46:24.600812+0000","last_clean":"2024-08-17T17:46:24.600812+0000","last_became_active":"2024-08-17T17:33:20.794238+0000","last_became_peered":"2024-08-17T17:33:20.794238+0000","last_unstale":"2024-08-17T17:46:24.600812+0000","last_undegraded":"2024-08-17T17:46:24.600812+0000","last_fullsized":"2024-08-17T17:46:24.600812+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:15:55.821920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":479,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.727026+0000","last_change":"2024-08-17T17:43:52.520659+0000","last_active":"2024-08-17T17:46:01.727026+0000","last_peered":"2024-08-17T17:46:01.727026+0000","last_clean":"2024-08-17T17:46:01.727026+0000","last_became_active":"2024-08-17T17:43:52.516799+0000","last_became_peered":"2024-08-17T17:43:52.516799+0000","last_unstale":"2024-08-17T17:46:01.727026+0000","last_undegraded":"2024-08-17T17:46:01.727026+0000","last_fullsized":"2024-08-17T17:46:01.727026+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:25:59.408383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.6","version":"393'14832","reported_seq":14916,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:20.067362+0000","last_change":"2024-08-17T17:42:36.946078+0000","last_active":"2024-08-17T17:46:20.067362+0000","last_peered":"2024-08-17T17:46:20.067362+0000","last_clean":"2024-08-17T17:46:20.067362+0000","last_became_active":"2024-08-17T17:42:36.945602+0000","last_became_peered":"2024-08-17T17:42:36.945602+0000","last_unstale":"2024-08-17T17:46:20.067362+0000","last_undegraded":"2024-08-17T17:46:20.067362+0000","last_fullsized":"2024-08-17T17:46:20.067362+0000","mapping_epoch":357,"log_start":"386'8300","ondisk_log_start":"386'8300","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6532,"log_dups_size":0,"ondisk_log_size":6532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:18:43.716968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14832,"num_write_kb":474624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"90'149","reported_seq":804,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.735633+0000","last_change":"2024-08-17T17:42:16.610940+0000","last_active":"2024-08-17T17:46:01.735633+0000","last_peered":"2024-08-17T17:46:01.735633+0000","last_clean":"2024-08-17T17:46:01.735633+0000","last_became_active":"2024-08-17T17:41:37.670855+0000","last_became_peered":"2024-08-17T17:41:37.670855+0000","last_unstale":"2024-08-17T17:46:01.735633+0000","last_undegraded":"2024-08-17T17:46:01.735633+0000","last_fullsized":"2024-08-17T17:46:01.735633+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:16.610828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:16.610828+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:48:25.918610+0000","scrub_duration":0.086892487000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":355,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.736320+0000","last_change":"2024-08-17T17:41:34.635980+0000","last_active":"2024-08-17T17:46:01.736320+0000","last_peered":"2024-08-17T17:46:01.736320+0000","last_clean":"2024-08-17T17:46:01.736320+0000","last_became_active":"2024-08-17T17:41:34.620596+0000","last_became_peered":"2024-08-17T17:41:34.620596+0000","last_unstale":"2024-08-17T17:46:01.736320+0000","last_undegraded":"2024-08-17T17:46:01.736320+0000","last_fullsized":"2024-08-17T17:46:01.736320+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:19:15.792577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":148,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.726833+0000","last_change":"2024-08-17T17:33:44.145598+0000","last_active":"2024-08-17T17:46:01.726833+0000","last_peered":"2024-08-17T17:46:01.726833+0000","last_clean":"2024-08-17T17:46:01.726833+0000","last_became_active":"2024-08-17T17:33:44.145283+0000","last_became_peered":"2024-08-17T17:33:44.145283+0000","last_unstale":"2024-08-17T17:46:01.726833+0000","last_undegraded":"2024-08-17T17:46:01.726833+0000","last_fullsized":"2024-08-17T17:46:01.726833+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:11.965226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.7","version":"393'14838","reported_seq":27875,"reported_epoch":394,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-08-17T17:46:24.052378+0000","last_change":"2024-08-17T17:44:30.783759+0000","last_active":"2024-08-17T17:46:24.052378+0000","last_peered":"2024-08-17T17:46:24.052378+0000","last_clean":"2024-08-17T17:43:51.465744+0000","last_became_active":"2024-08-17T17:43:53.682178+0000","last_became_peered":"2024-08-17T17:43:53.682178+0000","last_unstale":"2024-08-17T17:46:24.052378+0000","last_undegraded":"2024-08-17T17:44:30.779387+0000","last_fullsized":"2024-08-17T17:43:53.513318+0000","mapping_epoch":378,"log_start":"386'8400","ondisk_log_start":"386'8400","created":357,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6438,"log_dups_size":0,"ondisk_log_size":6438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:42:59.608872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14838,"num_write_kb":474816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6480,"num_bytes_recovered":251002880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3],"avail_no_missing":["6","2","3"],"object_location_counts":[{"shards":"3,6","objects":-320}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"90'149","reported_seq":861,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.600493+0000","last_change":"2024-08-17T17:41:42.603841+0000","last_active":"2024-08-17T17:46:24.600493+0000","last_peered":"2024-08-17T17:46:24.600493+0000","last_clean":"2024-08-17T17:46:24.600493+0000","last_became_active":"2024-08-17T17:41:12.203400+0000","last_became_peered":"2024-08-17T17:41:12.203400+0000","last_unstale":"2024-08-17T17:46:24.600493+0000","last_undegraded":"2024-08-17T17:46:24.600493+0000","last_fullsized":"2024-08-17T17:46:24.600493+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:42.603561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:42.603561+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:50:12.843193+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":351,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.735940+0000","last_change":"2024-08-17T17:41:34.637696+0000","last_active":"2024-08-17T17:46:01.735940+0000","last_peered":"2024-08-17T17:46:01.735940+0000","last_clean":"2024-08-17T17:46:01.735940+0000","last_became_active":"2024-08-17T17:41:34.621388+0000","last_became_peered":"2024-08-17T17:41:34.621388+0000","last_unstale":"2024-08-17T17:46:01.735940+0000","last_undegraded":"2024-08-17T17:46:01.735940+0000","last_fullsized":"2024-08-17T17:46:01.735940+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:13:04.607036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":58,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.728099+0000","last_change":"2024-08-17T17:41:34.621078+0000","last_active":"2024-08-17T17:46:01.728099+0000","last_peered":"2024-08-17T17:46:01.728099+0000","last_clean":"2024-08-17T17:46:01.728099+0000","last_became_active":"2024-08-17T17:41:34.620730+0000","last_became_peered":"2024-08-17T17:41:34.620730+0000","last_unstale":"2024-08-17T17:46:01.728099+0000","last_undegraded":"2024-08-17T17:46:01.728099+0000","last_fullsized":"2024-08-17T17:46:01.728099+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:55:01.312283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"393'14930","reported_seq":15014,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:20.097960+0000","last_change":"2024-08-17T17:42:36.620130+0000","last_active":"2024-08-17T17:46:20.097960+0000","last_peered":"2024-08-17T17:46:20.097960+0000","last_clean":"2024-08-17T17:46:20.097960+0000","last_became_active":"2024-08-17T17:42:36.619813+0000","last_became_peered":"2024-08-17T17:42:36.619813+0000","last_unstale":"2024-08-17T17:46:20.097960+0000","last_undegraded":"2024-08-17T17:46:20.097960+0000","last_fullsized":"2024-08-17T17:46:20.097960+0000","mapping_epoch":357,"log_start":"386'8400","ondisk_log_start":"386'8400","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6530,"log_dups_size":0,"ondisk_log_size":6530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:39:22.366905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14930,"num_write_kb":477760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"90'149","reported_seq":570,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.729167+0000","last_change":"2024-08-17T17:41:37.668050+0000","last_active":"2024-08-17T17:46:01.729167+0000","last_peered":"2024-08-17T17:46:01.729167+0000","last_clean":"2024-08-17T17:46:01.729167+0000","last_became_active":"2024-08-17T17:41:37.667655+0000","last_became_peered":"2024-08-17T17:41:37.667655+0000","last_unstale":"2024-08-17T17:46:01.729167+0000","last_undegraded":"2024-08-17T17:46:01.729167+0000","last_fullsized":"2024-08-17T17:46:01.729167+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:32.843802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:32.843802+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:39:46.766280+0000","scrub_duration":0.0040803669999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":144,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.727251+0000","last_change":"2024-08-17T17:33:44.145399+0000","last_active":"2024-08-17T17:46:01.727251+0000","last_peered":"2024-08-17T17:46:01.727251+0000","last_clean":"2024-08-17T17:46:01.727251+0000","last_became_active":"2024-08-17T17:33:44.144751+0000","last_became_peered":"2024-08-17T17:33:44.144751+0000","last_unstale":"2024-08-17T17:46:01.727251+0000","last_undegraded":"2024-08-17T17:46:01.727251+0000","last_fullsized":"2024-08-17T17:46:01.727251+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:54.276350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":222,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.727281+0000","last_change":"2024-08-17T17:33:20.795582+0000","last_active":"2024-08-17T17:46:01.727281+0000","last_peered":"2024-08-17T17:46:01.727281+0000","last_clean":"2024-08-17T17:46:01.727281+0000","last_became_active":"2024-08-17T17:33:20.795149+0000","last_became_peered":"2024-08-17T17:33:20.795149+0000","last_unstale":"2024-08-17T17:46:01.727281+0000","last_undegraded":"2024-08-17T17:46:01.727281+0000","last_fullsized":"2024-08-17T17:46:01.727281+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:14.020923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"393'14856","reported_seq":14940,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:20.095799+0000","last_change":"2024-08-17T17:42:36.929599+0000","last_active":"2024-08-17T17:46:20.095799+0000","last_peered":"2024-08-17T17:46:20.095799+0000","last_clean":"2024-08-17T17:46:20.095799+0000","last_became_active":"2024-08-17T17:42:36.929274+0000","last_became_peered":"2024-08-17T17:42:36.929274+0000","last_unstale":"2024-08-17T17:46:20.095799+0000","last_undegraded":"2024-08-17T17:46:20.095799+0000","last_fullsized":"2024-08-17T17:46:20.095799+0000","mapping_epoch":357,"log_start":"386'8300","ondisk_log_start":"386'8300","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6556,"log_dups_size":0,"ondisk_log_size":6556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:29:35.955743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14856,"num_write_kb":475392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"90'149","reported_seq":702,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.729481+0000","last_change":"2024-08-17T17:42:20.927007+0000","last_active":"2024-08-17T17:46:01.729481+0000","last_peered":"2024-08-17T17:46:01.729481+0000","last_clean":"2024-08-17T17:46:01.729481+0000","last_became_active":"2024-08-17T17:41:37.670914+0000","last_became_peered":"2024-08-17T17:41:37.670914+0000","last_unstale":"2024-08-17T17:46:01.729481+0000","last_undegraded":"2024-08-17T17:46:01.729481+0000","last_fullsized":"2024-08-17T17:46:01.729481+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:20.926924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:20.926924+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:57:44.156600+0000","scrub_duration":0.0018485109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":321,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.600752+0000","last_change":"2024-08-17T17:33:44.154097+0000","last_active":"2024-08-17T17:46:24.600752+0000","last_peered":"2024-08-17T17:46:24.600752+0000","last_clean":"2024-08-17T17:46:24.600752+0000","last_became_active":"2024-08-17T17:33:44.152972+0000","last_became_peered":"2024-08-17T17:33:44.152972+0000","last_unstale":"2024-08-17T17:46:24.600752+0000","last_undegraded":"2024-08-17T17:46:24.600752+0000","last_fullsized":"2024-08-17T17:46:24.600752+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:42.602017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":681,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.736723+0000","last_change":"2024-08-17T17:41:34.615197+0000","last_active":"2024-08-17T17:46:01.736723+0000","last_peered":"2024-08-17T17:46:01.736723+0000","last_clean":"2024-08-17T17:46:01.736723+0000","last_became_active":"2024-08-17T17:41:34.614873+0000","last_became_peered":"2024-08-17T17:41:34.614873+0000","last_unstale":"2024-08-17T17:46:01.736723+0000","last_undegraded":"2024-08-17T17:46:01.736723+0000","last_fullsized":"2024-08-17T17:46:01.736723+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:21:57.391953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.2","version":"393'14494","reported_seq":14581,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.871732+0000","last_change":"2024-08-17T17:42:36.618260+0000","last_active":"2024-08-17T17:46:24.871732+0000","last_peered":"2024-08-17T17:46:24.871732+0000","last_clean":"2024-08-17T17:46:24.871732+0000","last_became_active":"2024-08-17T17:42:36.617875+0000","last_became_peered":"2024-08-17T17:42:36.617875+0000","last_unstale":"2024-08-17T17:46:24.871732+0000","last_undegraded":"2024-08-17T17:46:24.871732+0000","last_fullsized":"2024-08-17T17:46:24.871732+0000","mapping_epoch":357,"log_start":"385'7500","ondisk_log_start":"385'7500","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6994,"log_dups_size":0,"ondisk_log_size":6994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:09:07.102359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14494,"num_write_kb":463808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":836,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.600049+0000","last_change":"2024-08-17T17:41:44.585386+0000","last_active":"2024-08-17T17:46:24.600049+0000","last_peered":"2024-08-17T17:46:24.600049+0000","last_clean":"2024-08-17T17:46:24.600049+0000","last_became_active":"2024-08-17T17:41:12.307398+0000","last_became_peered":"2024-08-17T17:41:12.307398+0000","last_unstale":"2024-08-17T17:46:24.600049+0000","last_undegraded":"2024-08-17T17:46:24.600049+0000","last_fullsized":"2024-08-17T17:46:24.600049+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:44.583653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:44.583653+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:27:44.756681+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":720,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.600401+0000","last_change":"2024-08-17T17:41:34.630434+0000","last_active":"2024-08-17T17:46:24.600401+0000","last_peered":"2024-08-17T17:46:24.600401+0000","last_clean":"2024-08-17T17:46:24.600401+0000","last_became_active":"2024-08-17T17:41:34.616406+0000","last_became_peered":"2024-08-17T17:41:34.616406+0000","last_unstale":"2024-08-17T17:46:24.600401+0000","last_undegraded":"2024-08-17T17:46:24.600401+0000","last_fullsized":"2024-08-17T17:46:24.600401+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:30:44.817203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":58,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.729668+0000","last_change":"2024-08-17T17:41:34.611049+0000","last_active":"2024-08-17T17:46:01.729668+0000","last_peered":"2024-08-17T17:46:01.729668+0000","last_clean":"2024-08-17T17:46:01.729668+0000","last_became_active":"2024-08-17T17:41:34.610727+0000","last_became_peered":"2024-08-17T17:41:34.610727+0000","last_unstale":"2024-08-17T17:46:01.729668+0000","last_undegraded":"2024-08-17T17:46:01.729668+0000","last_fullsized":"2024-08-17T17:46:01.729668+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:04:05.477660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"393'15028","reported_seq":15116,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.600241+0000","last_change":"2024-08-17T17:42:36.946226+0000","last_active":"2024-08-17T17:46:24.600241+0000","last_peered":"2024-08-17T17:46:24.600241+0000","last_clean":"2024-08-17T17:46:24.600241+0000","last_became_active":"2024-08-17T17:42:36.945809+0000","last_became_peered":"2024-08-17T17:42:36.945809+0000","last_unstale":"2024-08-17T17:46:24.600241+0000","last_undegraded":"2024-08-17T17:46:24.600241+0000","last_fullsized":"2024-08-17T17:46:24.600241+0000","mapping_epoch":357,"log_start":"387'8700","ondisk_log_start":"387'8700","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6328,"log_dups_size":0,"ondisk_log_size":6328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:04:58.633957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15028,"num_write_kb":480896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":864,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.599985+0000","last_change":"2024-08-17T17:43:55.595647+0000","last_active":"2024-08-17T17:46:24.599985+0000","last_peered":"2024-08-17T17:46:24.599985+0000","last_clean":"2024-08-17T17:46:24.599985+0000","last_became_active":"2024-08-17T17:43:55.595256+0000","last_became_peered":"2024-08-17T17:43:55.595256+0000","last_unstale":"2024-08-17T17:46:24.599985+0000","last_undegraded":"2024-08-17T17:46:24.599985+0000","last_fullsized":"2024-08-17T17:46:24.599985+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:46.587712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:46.587712+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:05:01.927968+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":58,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.728980+0000","last_change":"2024-08-17T17:41:34.622803+0000","last_active":"2024-08-17T17:46:01.728980+0000","last_peered":"2024-08-17T17:46:01.728980+0000","last_clean":"2024-08-17T17:46:01.728980+0000","last_became_active":"2024-08-17T17:41:34.622525+0000","last_became_peered":"2024-08-17T17:41:34.622525+0000","last_unstale":"2024-08-17T17:46:01.728980+0000","last_undegraded":"2024-08-17T17:46:01.728980+0000","last_fullsized":"2024-08-17T17:46:01.728980+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:42:39.728377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":58,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.728745+0000","last_change":"2024-08-17T17:41:34.619268+0000","last_active":"2024-08-17T17:46:01.728745+0000","last_peered":"2024-08-17T17:46:01.728745+0000","last_clean":"2024-08-17T17:46:01.728745+0000","last_became_active":"2024-08-17T17:41:34.618914+0000","last_became_peered":"2024-08-17T17:41:34.618914+0000","last_unstale":"2024-08-17T17:46:01.728745+0000","last_undegraded":"2024-08-17T17:46:01.728745+0000","last_fullsized":"2024-08-17T17:46:01.728745+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:08:54.248386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.0","version":"393'14366","reported_seq":14411,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:20.095156+0000","last_change":"2024-08-17T17:42:36.617537+0000","last_active":"2024-08-17T17:46:20.095156+0000","last_peered":"2024-08-17T17:46:20.095156+0000","last_clean":"2024-08-17T17:46:20.095156+0000","last_became_active":"2024-08-17T17:42:36.617201+0000","last_became_peered":"2024-08-17T17:42:36.617201+0000","last_unstale":"2024-08-17T17:46:20.095156+0000","last_undegraded":"2024-08-17T17:46:20.095156+0000","last_fullsized":"2024-08-17T17:46:20.095156+0000","mapping_epoch":357,"log_start":"386'7900","ondisk_log_start":"386'7900","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6466,"log_dups_size":0,"ondisk_log_size":6466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:51:58.927214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14366,"num_write_kb":459712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"90'149","reported_seq":869,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.872214+0000","last_change":"2024-08-17T17:41:41.743931+0000","last_active":"2024-08-17T17:46:24.872214+0000","last_peered":"2024-08-17T17:46:24.872214+0000","last_clean":"2024-08-17T17:46:24.872214+0000","last_became_active":"2024-08-17T17:41:12.203771+0000","last_became_peered":"2024-08-17T17:41:12.203771+0000","last_unstale":"2024-08-17T17:46:24.872214+0000","last_undegraded":"2024-08-17T17:46:24.872214+0000","last_fullsized":"2024-08-17T17:46:24.872214+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:41.743570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:41.743570+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:21:03.865738+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":58,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.728216+0000","last_change":"2024-08-17T17:41:34.607602+0000","last_active":"2024-08-17T17:46:01.728216+0000","last_peered":"2024-08-17T17:46:01.728216+0000","last_clean":"2024-08-17T17:46:01.728216+0000","last_became_active":"2024-08-17T17:41:34.607300+0000","last_became_peered":"2024-08-17T17:41:34.607300+0000","last_unstale":"2024-08-17T17:46:01.728216+0000","last_undegraded":"2024-08-17T17:46:01.728216+0000","last_fullsized":"2024-08-17T17:46:01.728216+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:20:31.486698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":58,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.729402+0000","last_change":"2024-08-17T17:41:34.622102+0000","last_active":"2024-08-17T17:46:01.729402+0000","last_peered":"2024-08-17T17:46:01.729402+0000","last_clean":"2024-08-17T17:46:01.729402+0000","last_became_active":"2024-08-17T17:41:34.621733+0000","last_became_peered":"2024-08-17T17:41:34.621733+0000","last_unstale":"2024-08-17T17:46:01.729402+0000","last_undegraded":"2024-08-17T17:46:01.729402+0000","last_fullsized":"2024-08-17T17:46:01.729402+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:39:41.668709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"393'15102","reported_seq":15145,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:20.108045+0000","last_change":"2024-08-17T17:42:36.620155+0000","last_active":"2024-08-17T17:46:20.108045+0000","last_peered":"2024-08-17T17:46:20.108045+0000","last_clean":"2024-08-17T17:46:20.108045+0000","last_became_active":"2024-08-17T17:42:36.619813+0000","last_became_peered":"2024-08-17T17:42:36.619813+0000","last_unstale":"2024-08-17T17:46:20.108045+0000","last_undegraded":"2024-08-17T17:46:20.108045+0000","last_fullsized":"2024-08-17T17:46:20.108045+0000","mapping_epoch":357,"log_start":"387'9300","ondisk_log_start":"387'9300","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":5802,"log_dups_size":0,"ondisk_log_size":5802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:16:27.534439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15102,"num_write_kb":483264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":511,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.727017+0000","last_change":"2024-08-17T17:41:41.036615+0000","last_active":"2024-08-17T17:46:01.727017+0000","last_peered":"2024-08-17T17:46:01.727017+0000","last_clean":"2024-08-17T17:46:01.727017+0000","last_became_active":"2024-08-17T17:41:12.329214+0000","last_became_peered":"2024-08-17T17:41:12.329214+0000","last_unstale":"2024-08-17T17:46:01.727017+0000","last_undegraded":"2024-08-17T17:46:01.727017+0000","last_fullsized":"2024-08-17T17:46:01.727017+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:41.036524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:41.036524+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:59:58.695900+0000","scrub_duration":0.001820803,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":679,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.600238+0000","last_change":"2024-08-17T17:33:20.796216+0000","last_active":"2024-08-17T17:46:24.600238+0000","last_peered":"2024-08-17T17:46:24.600238+0000","last_clean":"2024-08-17T17:46:24.600238+0000","last_became_active":"2024-08-17T17:33:20.795762+0000","last_became_peered":"2024-08-17T17:33:20.795762+0000","last_unstale":"2024-08-17T17:46:24.600238+0000","last_undegraded":"2024-08-17T17:46:24.600238+0000","last_fullsized":"2024-08-17T17:46:24.600238+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:47.038509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":573,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.873012+0000","last_change":"2024-08-17T17:41:34.625551+0000","last_active":"2024-08-17T17:46:24.873012+0000","last_peered":"2024-08-17T17:46:24.873012+0000","last_clean":"2024-08-17T17:46:24.873012+0000","last_became_active":"2024-08-17T17:41:34.625251+0000","last_became_peered":"2024-08-17T17:41:34.625251+0000","last_unstale":"2024-08-17T17:46:24.873012+0000","last_undegraded":"2024-08-17T17:46:24.873012+0000","last_fullsized":"2024-08-17T17:46:24.873012+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:28:22.354213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"393'14844","reported_seq":14889,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:20.109327+0000","last_change":"2024-08-17T17:42:36.619525+0000","last_active":"2024-08-17T17:46:20.109327+0000","last_peered":"2024-08-17T17:46:20.109327+0000","last_clean":"2024-08-17T17:46:20.109327+0000","last_became_active":"2024-08-17T17:42:36.619074+0000","last_became_peered":"2024-08-17T17:42:36.619074+0000","last_unstale":"2024-08-17T17:46:20.109327+0000","last_undegraded":"2024-08-17T17:46:20.109327+0000","last_fullsized":"2024-08-17T17:46:20.109327+0000","mapping_epoch":357,"log_start":"386'8400","ondisk_log_start":"386'8400","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6444,"log_dups_size":0,"ondisk_log_size":6444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:11:32.849616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14844,"num_write_kb":475008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":811,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.734464+0000","last_change":"2024-08-17T17:38:25.247159+0000","last_active":"2024-08-17T17:46:01.734464+0000","last_peered":"2024-08-17T17:46:01.734464+0000","last_clean":"2024-08-17T17:46:01.734464+0000","last_became_active":"2024-08-17T17:38:21.311361+0000","last_became_peered":"2024-08-17T17:38:21.311361+0000","last_unstale":"2024-08-17T17:46:01.734464+0000","last_undegraded":"2024-08-17T17:46:01.734464+0000","last_fullsized":"2024-08-17T17:46:01.734464+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:25.247027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:25.247027+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:46:21.734712+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":800,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.735328+0000","last_change":"2024-08-17T17:33:44.149928+0000","last_active":"2024-08-17T17:46:01.735328+0000","last_peered":"2024-08-17T17:46:01.735328+0000","last_clean":"2024-08-17T17:46:01.735328+0000","last_became_active":"2024-08-17T17:33:44.149498+0000","last_became_peered":"2024-08-17T17:33:44.149498+0000","last_unstale":"2024-08-17T17:46:01.735328+0000","last_undegraded":"2024-08-17T17:46:01.735328+0000","last_fullsized":"2024-08-17T17:46:01.735328+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:17:04.963202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":626,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.600132+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:46:24.600132+0000","last_peered":"2024-08-17T17:46:24.600132+0000","last_clean":"2024-08-17T17:46:24.600132+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:46:24.600132+0000","last_undegraded":"2024-08-17T17:46:24.600132+0000","last_fullsized":"2024-08-17T17:46:24.600132+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":235,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.727165+0000","last_change":"2024-08-17T17:41:34.618631+0000","last_active":"2024-08-17T17:46:01.727165+0000","last_peered":"2024-08-17T17:46:01.727165+0000","last_clean":"2024-08-17T17:46:01.727165+0000","last_became_active":"2024-08-17T17:41:34.618313+0000","last_became_peered":"2024-08-17T17:41:34.618313+0000","last_unstale":"2024-08-17T17:46:01.727165+0000","last_undegraded":"2024-08-17T17:46:01.727165+0000","last_fullsized":"2024-08-17T17:46:01.727165+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:07:13.915088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"393'14648","reported_seq":14732,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:20.112773+0000","last_change":"2024-08-17T17:42:36.929502+0000","last_active":"2024-08-17T17:46:20.112773+0000","last_peered":"2024-08-17T17:46:20.112773+0000","last_clean":"2024-08-17T17:46:20.112773+0000","last_became_active":"2024-08-17T17:42:36.929113+0000","last_became_peered":"2024-08-17T17:42:36.929113+0000","last_unstale":"2024-08-17T17:46:20.112773+0000","last_undegraded":"2024-08-17T17:46:20.112773+0000","last_fullsized":"2024-08-17T17:46:20.112773+0000","mapping_epoch":357,"log_start":"386'8100","ondisk_log_start":"386'8100","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6548,"log_dups_size":0,"ondisk_log_size":6548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:31:25.879221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14648,"num_write_kb":468736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":852,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.872365+0000","last_change":"2024-08-17T17:41:37.676347+0000","last_active":"2024-08-17T17:46:24.872365+0000","last_peered":"2024-08-17T17:46:24.872365+0000","last_clean":"2024-08-17T17:46:24.872365+0000","last_became_active":"2024-08-17T17:41:37.675869+0000","last_became_peered":"2024-08-17T17:41:37.675869+0000","last_unstale":"2024-08-17T17:46:24.872365+0000","last_undegraded":"2024-08-17T17:46:24.872365+0000","last_fullsized":"2024-08-17T17:46:24.872365+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:21.814077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:21.814077+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:04:16.240754+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":910,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.871161+0000","last_change":"2024-08-17T17:33:20.797705+0000","last_active":"2024-08-17T17:46:24.871161+0000","last_peered":"2024-08-17T17:46:24.871161+0000","last_clean":"2024-08-17T17:46:24.871161+0000","last_became_active":"2024-08-17T17:33:20.797172+0000","last_became_peered":"2024-08-17T17:33:20.797172+0000","last_unstale":"2024-08-17T17:46:24.871161+0000","last_undegraded":"2024-08-17T17:46:24.871161+0000","last_fullsized":"2024-08-17T17:46:24.871161+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:30.754999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":672,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.734517+0000","last_change":"2024-08-17T17:41:34.614108+0000","last_active":"2024-08-17T17:46:01.734517+0000","last_peered":"2024-08-17T17:46:01.734517+0000","last_clean":"2024-08-17T17:46:01.734517+0000","last_became_active":"2024-08-17T17:41:34.613481+0000","last_became_peered":"2024-08-17T17:41:34.613481+0000","last_unstale":"2024-08-17T17:46:01.734517+0000","last_undegraded":"2024-08-17T17:46:01.734517+0000","last_fullsized":"2024-08-17T17:46:01.734517+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:21:02.346115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":460,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.735608+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:46:01.735608+0000","last_peered":"2024-08-17T17:46:01.735608+0000","last_clean":"2024-08-17T17:46:01.735608+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:46:01.735608+0000","last_undegraded":"2024-08-17T17:46:01.735608+0000","last_fullsized":"2024-08-17T17:46:01.735608+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"393'14762","reported_seq":14807,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:20.106191+0000","last_change":"2024-08-17T17:42:36.946382+0000","last_active":"2024-08-17T17:46:20.106191+0000","last_peered":"2024-08-17T17:46:20.106191+0000","last_clean":"2024-08-17T17:46:20.106191+0000","last_became_active":"2024-08-17T17:42:36.945591+0000","last_became_peered":"2024-08-17T17:42:36.945591+0000","last_unstale":"2024-08-17T17:46:20.106191+0000","last_undegraded":"2024-08-17T17:46:20.106191+0000","last_fullsized":"2024-08-17T17:46:20.106191+0000","mapping_epoch":357,"log_start":"387'8500","ondisk_log_start":"387'8500","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6262,"log_dups_size":0,"ondisk_log_size":6262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:08:17.912891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14762,"num_write_kb":472384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":851,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.734396+0000","last_change":"2024-08-17T17:41:37.683450+0000","last_active":"2024-08-17T17:46:01.734396+0000","last_peered":"2024-08-17T17:46:01.734396+0000","last_clean":"2024-08-17T17:46:01.734396+0000","last_became_active":"2024-08-17T17:41:37.683103+0000","last_became_peered":"2024-08-17T17:41:37.683103+0000","last_unstale":"2024-08-17T17:46:01.734396+0000","last_undegraded":"2024-08-17T17:46:01.734396+0000","last_fullsized":"2024-08-17T17:46:01.734396+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.289528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.289528+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:39:17.141469+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":1043,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.600476+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:46:24.600476+0000","last_peered":"2024-08-17T17:46:24.600476+0000","last_clean":"2024-08-17T17:46:24.600476+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:46:24.600476+0000","last_undegraded":"2024-08-17T17:46:24.600476+0000","last_fullsized":"2024-08-17T17:46:24.600476+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":548,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.871444+0000","last_change":"2024-08-17T17:33:20.793468+0000","last_active":"2024-08-17T17:46:24.871444+0000","last_peered":"2024-08-17T17:46:24.871444+0000","last_clean":"2024-08-17T17:46:24.871444+0000","last_became_active":"2024-08-17T17:33:20.793166+0000","last_became_peered":"2024-08-17T17:33:20.793166+0000","last_unstale":"2024-08-17T17:46:24.871444+0000","last_undegraded":"2024-08-17T17:46:24.871444+0000","last_fullsized":"2024-08-17T17:46:24.871444+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:12:52.121050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":584,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.873011+0000","last_change":"2024-08-17T17:33:20.835784+0000","last_active":"2024-08-17T17:46:24.873011+0000","last_peered":"2024-08-17T17:46:24.873011+0000","last_clean":"2024-08-17T17:46:24.873011+0000","last_became_active":"2024-08-17T17:33:20.835371+0000","last_became_peered":"2024-08-17T17:33:20.835371+0000","last_unstale":"2024-08-17T17:46:24.873011+0000","last_undegraded":"2024-08-17T17:46:24.873011+0000","last_fullsized":"2024-08-17T17:46:24.873011+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:35:43.369055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.a","version":"393'14734","reported_seq":14779,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:20.115046+0000","last_change":"2024-08-17T17:42:36.946026+0000","last_active":"2024-08-17T17:46:20.115046+0000","last_peered":"2024-08-17T17:46:20.115046+0000","last_clean":"2024-08-17T17:46:20.115046+0000","last_became_active":"2024-08-17T17:42:36.945638+0000","last_became_peered":"2024-08-17T17:42:36.945638+0000","last_unstale":"2024-08-17T17:46:20.115046+0000","last_undegraded":"2024-08-17T17:46:20.115046+0000","last_fullsized":"2024-08-17T17:46:20.115046+0000","mapping_epoch":357,"log_start":"386'8300","ondisk_log_start":"386'8300","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6434,"log_dups_size":0,"ondisk_log_size":6434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:07:34.864262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14734,"num_write_kb":471488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":830,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.734412+0000","last_change":"2024-08-17T17:38:22.254841+0000","last_active":"2024-08-17T17:46:01.734412+0000","last_peered":"2024-08-17T17:46:01.734412+0000","last_clean":"2024-08-17T17:46:01.734412+0000","last_became_active":"2024-08-17T17:38:21.306619+0000","last_became_peered":"2024-08-17T17:38:21.306619+0000","last_unstale":"2024-08-17T17:46:01.734412+0000","last_undegraded":"2024-08-17T17:46:01.734412+0000","last_fullsized":"2024-08-17T17:46:01.734412+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:22.254705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:22.254705+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:47:38.184089+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":1133,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.871852+0000","last_change":"2024-08-17T17:33:44.148227+0000","last_active":"2024-08-17T17:46:24.871852+0000","last_peered":"2024-08-17T17:46:24.871852+0000","last_clean":"2024-08-17T17:46:24.871852+0000","last_became_active":"2024-08-17T17:33:44.147627+0000","last_became_peered":"2024-08-17T17:33:44.147627+0000","last_unstale":"2024-08-17T17:46:24.871852+0000","last_undegraded":"2024-08-17T17:46:24.871852+0000","last_fullsized":"2024-08-17T17:46:24.871852+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:29:42.898460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1372,"num_read_kb":2843,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":58,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.728813+0000","last_change":"2024-08-17T17:41:34.611109+0000","last_active":"2024-08-17T17:46:01.728813+0000","last_peered":"2024-08-17T17:46:01.728813+0000","last_clean":"2024-08-17T17:46:01.728813+0000","last_became_active":"2024-08-17T17:41:34.610170+0000","last_became_peered":"2024-08-17T17:41:34.610170+0000","last_unstale":"2024-08-17T17:46:01.728813+0000","last_undegraded":"2024-08-17T17:46:01.728813+0000","last_fullsized":"2024-08-17T17:46:01.728813+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:39:05.301260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":588,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.872708+0000","last_change":"2024-08-17T17:33:20.772454+0000","last_active":"2024-08-17T17:46:24.872708+0000","last_peered":"2024-08-17T17:46:24.872708+0000","last_clean":"2024-08-17T17:46:24.872708+0000","last_became_active":"2024-08-17T17:33:20.771848+0000","last_became_peered":"2024-08-17T17:33:20.771848+0000","last_unstale":"2024-08-17T17:46:24.872708+0000","last_undegraded":"2024-08-17T17:46:24.872708+0000","last_fullsized":"2024-08-17T17:46:24.872708+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:46:42.300744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"393'15186","reported_seq":15231,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:20.095373+0000","last_change":"2024-08-17T17:42:36.618414+0000","last_active":"2024-08-17T17:46:20.095373+0000","last_peered":"2024-08-17T17:46:20.095373+0000","last_clean":"2024-08-17T17:46:20.095373+0000","last_became_active":"2024-08-17T17:42:36.618086+0000","last_became_peered":"2024-08-17T17:42:36.618086+0000","last_unstale":"2024-08-17T17:46:20.095373+0000","last_undegraded":"2024-08-17T17:46:20.095373+0000","last_fullsized":"2024-08-17T17:46:20.095373+0000","mapping_epoch":357,"log_start":"387'8900","ondisk_log_start":"387'8900","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6286,"log_dups_size":0,"ondisk_log_size":6286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:26:27.772894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15186,"num_write_kb":485952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":531,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.729125+0000","last_change":"2024-08-17T17:41:43.035215+0000","last_active":"2024-08-17T17:46:01.729125+0000","last_peered":"2024-08-17T17:46:01.729125+0000","last_clean":"2024-08-17T17:46:01.729125+0000","last_became_active":"2024-08-17T17:41:12.201158+0000","last_became_peered":"2024-08-17T17:41:12.201158+0000","last_unstale":"2024-08-17T17:46:01.729125+0000","last_undegraded":"2024-08-17T17:46:01.729125+0000","last_fullsized":"2024-08-17T17:46:01.729125+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:43.035108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:43.035108+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:43:27.108994+0000","scrub_duration":0.0028877400000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":339,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.600798+0000","last_change":"2024-08-17T17:33:44.163867+0000","last_active":"2024-08-17T17:46:24.600798+0000","last_peered":"2024-08-17T17:46:24.600798+0000","last_clean":"2024-08-17T17:46:24.600798+0000","last_became_active":"2024-08-17T17:33:44.163503+0000","last_became_peered":"2024-08-17T17:33:44.163503+0000","last_unstale":"2024-08-17T17:46:24.600798+0000","last_undegraded":"2024-08-17T17:46:24.600798+0000","last_fullsized":"2024-08-17T17:46:24.600798+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:31:53.573596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":803,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.872483+0000","last_change":"2024-08-17T17:33:20.799350+0000","last_active":"2024-08-17T17:46:24.872483+0000","last_peered":"2024-08-17T17:46:24.872483+0000","last_clean":"2024-08-17T17:46:24.872483+0000","last_became_active":"2024-08-17T17:33:20.798925+0000","last_became_peered":"2024-08-17T17:33:20.798925+0000","last_unstale":"2024-08-17T17:46:24.872483+0000","last_undegraded":"2024-08-17T17:46:24.872483+0000","last_fullsized":"2024-08-17T17:46:24.872483+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:16:36.121295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":619,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.872315+0000","last_change":"2024-08-17T17:41:34.627064+0000","last_active":"2024-08-17T17:46:24.872315+0000","last_peered":"2024-08-17T17:46:24.872315+0000","last_clean":"2024-08-17T17:46:24.872315+0000","last_became_active":"2024-08-17T17:41:34.626453+0000","last_became_peered":"2024-08-17T17:41:34.626453+0000","last_unstale":"2024-08-17T17:46:24.872315+0000","last_undegraded":"2024-08-17T17:46:24.872315+0000","last_fullsized":"2024-08-17T17:46:24.872315+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:02:11.894042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"393'14874","reported_seq":14920,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:20.119143+0000","last_change":"2024-08-17T17:42:36.620669+0000","last_active":"2024-08-17T17:46:20.119143+0000","last_peered":"2024-08-17T17:46:20.119143+0000","last_clean":"2024-08-17T17:46:20.119143+0000","last_became_active":"2024-08-17T17:42:36.619876+0000","last_became_peered":"2024-08-17T17:42:36.619876+0000","last_unstale":"2024-08-17T17:46:20.119143+0000","last_undegraded":"2024-08-17T17:46:20.119143+0000","last_fullsized":"2024-08-17T17:46:20.119143+0000","mapping_epoch":357,"log_start":"386'8400","ondisk_log_start":"386'8400","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6474,"log_dups_size":0,"ondisk_log_size":6474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:59:34.661001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14874,"num_write_kb":475905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":531,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.728275+0000","last_change":"2024-08-17T17:41:44.058291+0000","last_active":"2024-08-17T17:46:01.728275+0000","last_peered":"2024-08-17T17:46:01.728275+0000","last_clean":"2024-08-17T17:46:01.728275+0000","last_became_active":"2024-08-17T17:41:12.200077+0000","last_became_peered":"2024-08-17T17:41:12.200077+0000","last_unstale":"2024-08-17T17:46:01.728275+0000","last_undegraded":"2024-08-17T17:46:01.728275+0000","last_fullsized":"2024-08-17T17:46:01.728275+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:44.058233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:44.058233+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:42:55.911623+0000","scrub_duration":0.0074268499999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":623,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.872113+0000","last_change":"2024-08-17T17:33:20.798613+0000","last_active":"2024-08-17T17:46:24.872113+0000","last_peered":"2024-08-17T17:46:24.872113+0000","last_clean":"2024-08-17T17:46:24.872113+0000","last_became_active":"2024-08-17T17:33:20.798260+0000","last_became_peered":"2024-08-17T17:33:20.798260+0000","last_unstale":"2024-08-17T17:46:24.872113+0000","last_undegraded":"2024-08-17T17:46:24.872113+0000","last_fullsized":"2024-08-17T17:46:24.872113+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:18:00.822681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":954,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.736101+0000","last_change":"2024-08-17T17:41:37.683212+0000","last_active":"2024-08-17T17:46:01.736101+0000","last_peered":"2024-08-17T17:46:01.736101+0000","last_clean":"2024-08-17T17:46:01.736101+0000","last_became_active":"2024-08-17T17:41:37.682730+0000","last_became_peered":"2024-08-17T17:41:37.682730+0000","last_unstale":"2024-08-17T17:46:01.736101+0000","last_undegraded":"2024-08-17T17:46:01.736101+0000","last_fullsized":"2024-08-17T17:46:01.736101+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:51:35.444433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":634,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.736643+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:46:01.736643+0000","last_peered":"2024-08-17T17:46:01.736643+0000","last_clean":"2024-08-17T17:46:01.736643+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:46:01.736643+0000","last_undegraded":"2024-08-17T17:46:01.736643+0000","last_fullsized":"2024-08-17T17:46:01.736643+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.e","version":"393'14874","reported_seq":14917,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:20.068280+0000","last_change":"2024-08-17T17:42:36.929709+0000","last_active":"2024-08-17T17:46:20.068280+0000","last_peered":"2024-08-17T17:46:20.068280+0000","last_clean":"2024-08-17T17:46:20.068280+0000","last_became_active":"2024-08-17T17:42:36.929285+0000","last_became_peered":"2024-08-17T17:42:36.929285+0000","last_unstale":"2024-08-17T17:46:20.068280+0000","last_undegraded":"2024-08-17T17:46:20.068280+0000","last_fullsized":"2024-08-17T17:46:20.068280+0000","mapping_epoch":357,"log_start":"387'9100","ondisk_log_start":"387'9100","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":5774,"log_dups_size":0,"ondisk_log_size":5774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:47:39.760414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14874,"num_write_kb":475968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":777,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.728412+0000","last_change":"2024-08-17T17:42:19.960161+0000","last_active":"2024-08-17T17:46:01.728412+0000","last_peered":"2024-08-17T17:46:01.728412+0000","last_clean":"2024-08-17T17:46:01.728412+0000","last_became_active":"2024-08-17T17:41:37.665223+0000","last_became_peered":"2024-08-17T17:41:37.665223+0000","last_unstale":"2024-08-17T17:46:01.728412+0000","last_undegraded":"2024-08-17T17:46:01.728412+0000","last_fullsized":"2024-08-17T17:46:01.728412+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:19.960086+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:19.960086+0000","last_clean_scrub_stamp":"2024-08-17T17:42:19.960086+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:57.861661+0000","scrub_duration":0.0022612000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":673,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.600992+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:46:24.600992+0000","last_peered":"2024-08-17T17:46:24.600992+0000","last_clean":"2024-08-17T17:46:24.600992+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:46:24.600992+0000","last_undegraded":"2024-08-17T17:46:24.600992+0000","last_fullsized":"2024-08-17T17:46:24.600992+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":1248,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.872922+0000","last_change":"2024-08-17T17:41:37.677524+0000","last_active":"2024-08-17T17:46:24.872922+0000","last_peered":"2024-08-17T17:46:24.872922+0000","last_clean":"2024-08-17T17:46:24.872922+0000","last_became_active":"2024-08-17T17:41:37.676764+0000","last_became_peered":"2024-08-17T17:41:37.676764+0000","last_unstale":"2024-08-17T17:46:24.872922+0000","last_undegraded":"2024-08-17T17:46:24.872922+0000","last_fullsized":"2024-08-17T17:46:24.872922+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:14:49.341080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":670,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.600116+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:46:24.600116+0000","last_peered":"2024-08-17T17:46:24.600116+0000","last_clean":"2024-08-17T17:46:24.600116+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:46:24.600116+0000","last_undegraded":"2024-08-17T17:46:24.600116+0000","last_fullsized":"2024-08-17T17:46:24.600116+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"393'14720","reported_seq":14765,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:20.067849+0000","last_change":"2024-08-17T17:42:36.929613+0000","last_active":"2024-08-17T17:46:20.067849+0000","last_peered":"2024-08-17T17:46:20.067849+0000","last_clean":"2024-08-17T17:46:20.067849+0000","last_became_active":"2024-08-17T17:42:36.929191+0000","last_became_peered":"2024-08-17T17:42:36.929191+0000","last_unstale":"2024-08-17T17:46:20.067849+0000","last_undegraded":"2024-08-17T17:46:20.067849+0000","last_fullsized":"2024-08-17T17:46:20.067849+0000","mapping_epoch":357,"log_start":"386'8300","ondisk_log_start":"386'8300","created":357,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:42:35.592462+0000","last_clean_scrub_stamp":"2024-08-17T17:42:35.592462+0000","objects_scrubbed":0,"log_size":6420,"log_dups_size":0,"ondisk_log_size":6420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:21:40.881779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14720,"num_write_kb":471040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":817,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.600380+0000","last_change":"2024-08-17T17:38:23.113397+0000","last_active":"2024-08-17T17:46:24.600380+0000","last_peered":"2024-08-17T17:46:24.600380+0000","last_clean":"2024-08-17T17:46:24.600380+0000","last_became_active":"2024-08-17T17:38:21.305580+0000","last_became_peered":"2024-08-17T17:38:21.305580+0000","last_unstale":"2024-08-17T17:46:24.600380+0000","last_undegraded":"2024-08-17T17:46:24.600380+0000","last_fullsized":"2024-08-17T17:46:24.600380+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.113279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.113279+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:38:07.015721+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":506,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.735521+0000","last_change":"2024-08-17T17:41:34.629522+0000","last_active":"2024-08-17T17:46:01.735521+0000","last_peered":"2024-08-17T17:46:01.735521+0000","last_clean":"2024-08-17T17:46:01.735521+0000","last_became_active":"2024-08-17T17:41:34.619340+0000","last_became_peered":"2024-08-17T17:41:34.619340+0000","last_unstale":"2024-08-17T17:46:01.735521+0000","last_undegraded":"2024-08-17T17:46:01.735521+0000","last_fullsized":"2024-08-17T17:46:01.735521+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:30:03.902956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":624,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.729361+0000","last_change":"2024-08-17T17:26:06.855307+0000","last_active":"2024-08-17T17:46:01.729361+0000","last_peered":"2024-08-17T17:46:01.729361+0000","last_clean":"2024-08-17T17:46:01.729361+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:46:01.729361+0000","last_undegraded":"2024-08-17T17:46:01.729361+0000","last_fullsized":"2024-08-17T17:46:01.729361+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":58,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.728135+0000","last_change":"2024-08-17T17:41:34.614028+0000","last_active":"2024-08-17T17:46:01.728135+0000","last_peered":"2024-08-17T17:46:01.728135+0000","last_clean":"2024-08-17T17:46:01.728135+0000","last_became_active":"2024-08-17T17:41:34.613706+0000","last_became_peered":"2024-08-17T17:41:34.613706+0000","last_unstale":"2024-08-17T17:46:01.728135+0000","last_undegraded":"2024-08-17T17:46:01.728135+0000","last_fullsized":"2024-08-17T17:46:01.728135+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:19:55.254902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"90'149","reported_seq":727,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.872710+0000","last_change":"2024-08-17T17:42:07.681204+0000","last_active":"2024-08-17T17:46:24.872710+0000","last_peered":"2024-08-17T17:46:24.872710+0000","last_clean":"2024-08-17T17:46:24.872710+0000","last_became_active":"2024-08-17T17:42:05.878769+0000","last_became_peered":"2024-08-17T17:42:05.878769+0000","last_unstale":"2024-08-17T17:46:24.872710+0000","last_undegraded":"2024-08-17T17:46:24.872710+0000","last_fullsized":"2024-08-17T17:46:24.872710+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:07.681064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:07.681064+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:12:12.835715+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":58,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.728887+0000","last_change":"2024-08-17T17:41:34.622180+0000","last_active":"2024-08-17T17:46:01.728887+0000","last_peered":"2024-08-17T17:46:01.728887+0000","last_clean":"2024-08-17T17:46:01.728887+0000","last_became_active":"2024-08-17T17:41:34.621724+0000","last_became_peered":"2024-08-17T17:41:34.621724+0000","last_unstale":"2024-08-17T17:46:01.728887+0000","last_undegraded":"2024-08-17T17:46:01.728887+0000","last_fullsized":"2024-08-17T17:46:01.728887+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:15.807235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":68,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.729742+0000","last_change":"2024-08-17T17:41:34.626584+0000","last_active":"2024-08-17T17:46:01.729742+0000","last_peered":"2024-08-17T17:46:01.729742+0000","last_clean":"2024-08-17T17:46:01.729742+0000","last_became_active":"2024-08-17T17:41:34.609878+0000","last_became_peered":"2024-08-17T17:41:34.609878+0000","last_unstale":"2024-08-17T17:46:01.729742+0000","last_undegraded":"2024-08-17T17:46:01.729742+0000","last_fullsized":"2024-08-17T17:46:01.729742+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:08:12.810347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.17","version":"90'149","reported_seq":482,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.734881+0000","last_change":"2024-08-17T17:42:06.699082+0000","last_active":"2024-08-17T17:46:01.734881+0000","last_peered":"2024-08-17T17:46:01.734881+0000","last_clean":"2024-08-17T17:46:01.734881+0000","last_became_active":"2024-08-17T17:42:05.876218+0000","last_became_peered":"2024-08-17T17:42:05.876218+0000","last_unstale":"2024-08-17T17:46:01.734881+0000","last_undegraded":"2024-08-17T17:46:01.734881+0000","last_fullsized":"2024-08-17T17:46:01.734881+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.698998+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:06.698998+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.698998+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:22:16.510150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":615,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.871955+0000","last_change":"2024-08-17T17:33:20.799402+0000","last_active":"2024-08-17T17:46:24.871955+0000","last_peered":"2024-08-17T17:46:24.871955+0000","last_clean":"2024-08-17T17:46:24.871955+0000","last_became_active":"2024-08-17T17:33:20.798832+0000","last_became_peered":"2024-08-17T17:33:20.798832+0000","last_unstale":"2024-08-17T17:46:24.871955+0000","last_undegraded":"2024-08-17T17:46:24.871955+0000","last_fullsized":"2024-08-17T17:46:24.871955+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:21:05.001134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":390,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.600753+0000","last_change":"2024-08-17T17:41:34.634301+0000","last_active":"2024-08-17T17:46:24.600753+0000","last_peered":"2024-08-17T17:46:24.600753+0000","last_clean":"2024-08-17T17:46:24.600753+0000","last_became_active":"2024-08-17T17:41:34.620508+0000","last_became_peered":"2024-08-17T17:41:34.620508+0000","last_unstale":"2024-08-17T17:46:24.600753+0000","last_undegraded":"2024-08-17T17:46:24.600753+0000","last_fullsized":"2024-08-17T17:46:24.600753+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:10:54.576812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.14","version":"90'149","reported_seq":840,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.592773+0000","last_change":"2024-08-17T17:41:37.664471+0000","last_active":"2024-08-17T17:46:24.592773+0000","last_peered":"2024-08-17T17:46:24.592773+0000","last_clean":"2024-08-17T17:46:24.592773+0000","last_became_active":"2024-08-17T17:41:37.664108+0000","last_became_peered":"2024-08-17T17:41:37.664108+0000","last_unstale":"2024-08-17T17:46:24.592773+0000","last_undegraded":"2024-08-17T17:46:24.592773+0000","last_fullsized":"2024-08-17T17:46:24.592773+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:22.030514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:22.030514+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:15:52.325163+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":354,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.872927+0000","last_change":"2024-08-17T17:33:44.146014+0000","last_active":"2024-08-17T17:46:24.872927+0000","last_peered":"2024-08-17T17:46:24.872927+0000","last_clean":"2024-08-17T17:46:24.872927+0000","last_became_active":"2024-08-17T17:33:44.145668+0000","last_became_peered":"2024-08-17T17:33:44.145668+0000","last_unstale":"2024-08-17T17:46:24.872927+0000","last_undegraded":"2024-08-17T17:46:24.872927+0000","last_fullsized":"2024-08-17T17:46:24.872927+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:53:41.679237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":619,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.600178+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:46:24.600178+0000","last_peered":"2024-08-17T17:46:24.600178+0000","last_clean":"2024-08-17T17:46:24.600178+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:46:24.600178+0000","last_undegraded":"2024-08-17T17:46:24.600178+0000","last_fullsized":"2024-08-17T17:46:24.600178+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"90'149","reported_seq":780,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.736012+0000","last_change":"2024-08-17T17:42:06.456562+0000","last_active":"2024-08-17T17:46:01.736012+0000","last_peered":"2024-08-17T17:46:01.736012+0000","last_clean":"2024-08-17T17:46:01.736012+0000","last_became_active":"2024-08-17T17:42:05.876577+0000","last_became_peered":"2024-08-17T17:42:05.876577+0000","last_unstale":"2024-08-17T17:46:01.736012+0000","last_undegraded":"2024-08-17T17:46:01.736012+0000","last_fullsized":"2024-08-17T17:46:01.736012+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.456490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.456490+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:10:03.654953+0000","scrub_duration":0.001537285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":642,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.735184+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:46:01.735184+0000","last_peered":"2024-08-17T17:46:01.735184+0000","last_clean":"2024-08-17T17:46:01.735184+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:46:01.735184+0000","last_undegraded":"2024-08-17T17:46:01.735184+0000","last_fullsized":"2024-08-17T17:46:01.735184+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":677,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.599991+0000","last_change":"2024-08-17T17:33:20.790687+0000","last_active":"2024-08-17T17:46:24.599991+0000","last_peered":"2024-08-17T17:46:24.599991+0000","last_clean":"2024-08-17T17:46:24.599991+0000","last_became_active":"2024-08-17T17:33:20.790399+0000","last_became_peered":"2024-08-17T17:33:20.790399+0000","last_unstale":"2024-08-17T17:46:24.599991+0000","last_undegraded":"2024-08-17T17:46:24.599991+0000","last_fullsized":"2024-08-17T17:46:24.599991+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:38:06.849989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1a","version":"90'149","reported_seq":442,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.728688+0000","last_change":"2024-08-17T17:42:21.932830+0000","last_active":"2024-08-17T17:46:01.728688+0000","last_peered":"2024-08-17T17:46:01.728688+0000","last_clean":"2024-08-17T17:46:01.728688+0000","last_became_active":"2024-08-17T17:42:05.876835+0000","last_became_peered":"2024-08-17T17:42:05.876835+0000","last_unstale":"2024-08-17T17:46:01.728688+0000","last_undegraded":"2024-08-17T17:46:01.728688+0000","last_fullsized":"2024-08-17T17:46:01.728688+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:21.932757+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:21.932757+0000","last_clean_scrub_stamp":"2024-08-17T17:42:21.932757+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:36:39.848570+0000","scrub_duration":0.0020218810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":367,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.729206+0000","last_change":"2024-08-17T17:33:20.835760+0000","last_active":"2024-08-17T17:46:01.729206+0000","last_peered":"2024-08-17T17:46:01.729206+0000","last_clean":"2024-08-17T17:46:01.729206+0000","last_became_active":"2024-08-17T17:33:20.835410+0000","last_became_peered":"2024-08-17T17:33:20.835410+0000","last_unstale":"2024-08-17T17:46:01.729206+0000","last_undegraded":"2024-08-17T17:46:01.729206+0000","last_fullsized":"2024-08-17T17:46:01.729206+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:45.448022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":694,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.600124+0000","last_change":"2024-08-17T17:33:20.771279+0000","last_active":"2024-08-17T17:46:24.600124+0000","last_peered":"2024-08-17T17:46:24.600124+0000","last_clean":"2024-08-17T17:46:24.600124+0000","last_became_active":"2024-08-17T17:33:20.770728+0000","last_became_peered":"2024-08-17T17:33:20.770728+0000","last_unstale":"2024-08-17T17:46:24.600124+0000","last_undegraded":"2024-08-17T17:46:24.600124+0000","last_fullsized":"2024-08-17T17:46:24.600124+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:52:31.540397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"90'149","reported_seq":737,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.728727+0000","last_change":"2024-08-17T17:42:26.967686+0000","last_active":"2024-08-17T17:46:01.728727+0000","last_peered":"2024-08-17T17:46:01.728727+0000","last_clean":"2024-08-17T17:46:01.728727+0000","last_became_active":"2024-08-17T17:42:05.873657+0000","last_became_peered":"2024-08-17T17:42:05.873657+0000","last_unstale":"2024-08-17T17:46:01.728727+0000","last_undegraded":"2024-08-17T17:46:01.728727+0000","last_fullsized":"2024-08-17T17:46:01.728727+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:26.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:26.967611+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:31:03.344332+0000","scrub_duration":0.0015978279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":461,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.735406+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:46:01.735406+0000","last_peered":"2024-08-17T17:46:01.735406+0000","last_clean":"2024-08-17T17:46:01.735406+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:46:01.735406+0000","last_undegraded":"2024-08-17T17:46:01.735406+0000","last_fullsized":"2024-08-17T17:46:01.735406+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":614,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.871081+0000","last_change":"2024-08-17T17:41:34.625011+0000","last_active":"2024-08-17T17:46:24.871081+0000","last_peered":"2024-08-17T17:46:24.871081+0000","last_clean":"2024-08-17T17:46:24.871081+0000","last_became_active":"2024-08-17T17:41:34.624631+0000","last_became_peered":"2024-08-17T17:41:34.624631+0000","last_unstale":"2024-08-17T17:46:24.871081+0000","last_undegraded":"2024-08-17T17:46:24.871081+0000","last_fullsized":"2024-08-17T17:46:24.871081+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:03:05.565435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"90'149","reported_seq":700,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.731553+0000","last_change":"2024-08-17T17:42:08.482975+0000","last_active":"2024-08-17T17:46:01.731553+0000","last_peered":"2024-08-17T17:46:01.731553+0000","last_clean":"2024-08-17T17:46:01.731553+0000","last_became_active":"2024-08-17T17:42:05.873523+0000","last_became_peered":"2024-08-17T17:42:05.873523+0000","last_unstale":"2024-08-17T17:46:01.731553+0000","last_undegraded":"2024-08-17T17:46:01.731553+0000","last_fullsized":"2024-08-17T17:46:01.731553+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:08.482892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:08.482892+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:41.477341+0000","scrub_duration":0.0017925479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":58,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.728874+0000","last_change":"2024-08-17T17:41:34.616482+0000","last_active":"2024-08-17T17:46:01.728874+0000","last_peered":"2024-08-17T17:46:01.728874+0000","last_clean":"2024-08-17T17:46:01.728874+0000","last_became_active":"2024-08-17T17:41:34.616040+0000","last_became_peered":"2024-08-17T17:41:34.616040+0000","last_unstale":"2024-08-17T17:46:01.728874+0000","last_undegraded":"2024-08-17T17:46:01.728874+0000","last_fullsized":"2024-08-17T17:46:01.728874+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:53:08.502264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":58,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.728873+0000","last_change":"2024-08-17T17:41:34.622184+0000","last_active":"2024-08-17T17:46:01.728873+0000","last_peered":"2024-08-17T17:46:01.728873+0000","last_clean":"2024-08-17T17:46:01.728873+0000","last_became_active":"2024-08-17T17:41:34.621831+0000","last_became_peered":"2024-08-17T17:41:34.621831+0000","last_unstale":"2024-08-17T17:46:01.728873+0000","last_undegraded":"2024-08-17T17:46:01.728873+0000","last_fullsized":"2024-08-17T17:46:01.728873+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:59.826667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"90'149","reported_seq":844,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.600043+0000","last_change":"2024-08-17T17:42:05.876463+0000","last_active":"2024-08-17T17:46:24.600043+0000","last_peered":"2024-08-17T17:46:24.600043+0000","last_clean":"2024-08-17T17:46:24.600043+0000","last_became_active":"2024-08-17T17:42:05.876069+0000","last_became_peered":"2024-08-17T17:42:05.876069+0000","last_unstale":"2024-08-17T17:46:24.600043+0000","last_undegraded":"2024-08-17T17:46:24.600043+0000","last_fullsized":"2024-08-17T17:46:24.600043+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:45.552985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:45.552985+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:55:06.027278+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":581,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.872741+0000","last_change":"2024-08-17T17:33:20.773207+0000","last_active":"2024-08-17T17:46:24.872741+0000","last_peered":"2024-08-17T17:46:24.872741+0000","last_clean":"2024-08-17T17:46:24.872741+0000","last_became_active":"2024-08-17T17:33:20.772457+0000","last_became_peered":"2024-08-17T17:33:20.772457+0000","last_unstale":"2024-08-17T17:46:24.872741+0000","last_undegraded":"2024-08-17T17:46:24.872741+0000","last_fullsized":"2024-08-17T17:46:24.872741+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:57:09.610084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":483,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.736528+0000","last_change":"2024-08-17T17:41:34.618968+0000","last_active":"2024-08-17T17:46:01.736528+0000","last_peered":"2024-08-17T17:46:01.736528+0000","last_clean":"2024-08-17T17:46:01.736528+0000","last_became_active":"2024-08-17T17:41:34.618576+0000","last_became_peered":"2024-08-17T17:41:34.618576+0000","last_unstale":"2024-08-17T17:46:01.736528+0000","last_undegraded":"2024-08-17T17:46:01.736528+0000","last_fullsized":"2024-08-17T17:46:01.736528+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:47:45.888590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"90'149","reported_seq":492,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.600741+0000","last_change":"2024-08-17T17:42:06.742801+0000","last_active":"2024-08-17T17:46:24.600741+0000","last_peered":"2024-08-17T17:46:24.600741+0000","last_clean":"2024-08-17T17:46:24.600741+0000","last_became_active":"2024-08-17T17:42:05.876895+0000","last_became_peered":"2024-08-17T17:42:05.876895+0000","last_unstale":"2024-08-17T17:46:24.600741+0000","last_undegraded":"2024-08-17T17:46:24.600741+0000","last_fullsized":"2024-08-17T17:46:24.600741+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.742723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.742723+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:36:46.651953+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":58,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.728651+0000","last_change":"2024-08-17T17:41:34.622094+0000","last_active":"2024-08-17T17:46:01.728651+0000","last_peered":"2024-08-17T17:46:01.728651+0000","last_clean":"2024-08-17T17:46:01.728651+0000","last_became_active":"2024-08-17T17:41:34.621773+0000","last_became_peered":"2024-08-17T17:41:34.621773+0000","last_unstale":"2024-08-17T17:46:01.728651+0000","last_undegraded":"2024-08-17T17:46:01.728651+0000","last_fullsized":"2024-08-17T17:46:01.728651+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:34:30.540886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":62,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.729775+0000","last_change":"2024-08-17T17:41:34.629135+0000","last_active":"2024-08-17T17:46:01.729775+0000","last_peered":"2024-08-17T17:46:01.729775+0000","last_clean":"2024-08-17T17:46:01.729775+0000","last_became_active":"2024-08-17T17:41:34.613095+0000","last_became_peered":"2024-08-17T17:41:34.613095+0000","last_unstale":"2024-08-17T17:46:01.729775+0000","last_undegraded":"2024-08-17T17:46:01.729775+0000","last_fullsized":"2024-08-17T17:46:01.729775+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:27:06.542416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":58,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.729003+0000","last_change":"2024-08-17T17:41:34.616251+0000","last_active":"2024-08-17T17:46:01.729003+0000","last_peered":"2024-08-17T17:46:01.729003+0000","last_clean":"2024-08-17T17:46:01.729003+0000","last_became_active":"2024-08-17T17:41:34.615952+0000","last_became_peered":"2024-08-17T17:41:34.615952+0000","last_unstale":"2024-08-17T17:46:01.729003+0000","last_undegraded":"2024-08-17T17:46:01.729003+0000","last_fullsized":"2024-08-17T17:46:01.729003+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:12:24.892508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":62,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.728497+0000","last_change":"2024-08-17T17:41:34.631152+0000","last_active":"2024-08-17T17:46:01.728497+0000","last_peered":"2024-08-17T17:46:01.728497+0000","last_clean":"2024-08-17T17:46:01.728497+0000","last_became_active":"2024-08-17T17:41:34.615617+0000","last_became_peered":"2024-08-17T17:41:34.615617+0000","last_unstale":"2024-08-17T17:46:01.728497+0000","last_undegraded":"2024-08-17T17:46:01.728497+0000","last_fullsized":"2024-08-17T17:46:01.728497+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:43:08.257084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"90'149","reported_seq":734,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.726894+0000","last_change":"2024-08-17T17:42:11.809047+0000","last_active":"2024-08-17T17:46:01.726894+0000","last_peered":"2024-08-17T17:46:01.726894+0000","last_clean":"2024-08-17T17:46:01.726894+0000","last_became_active":"2024-08-17T17:42:05.883770+0000","last_became_peered":"2024-08-17T17:42:05.883770+0000","last_unstale":"2024-08-17T17:46:01.726894+0000","last_undegraded":"2024-08-17T17:46:01.726894+0000","last_fullsized":"2024-08-17T17:46:01.726894+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:11.808971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:11.808971+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:48:15.989437+0000","scrub_duration":0.0018167420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":672,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.600497+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:46:24.600497+0000","last_peered":"2024-08-17T17:46:24.600497+0000","last_clean":"2024-08-17T17:46:24.600497+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:46:24.600497+0000","last_undegraded":"2024-08-17T17:46:24.600497+0000","last_fullsized":"2024-08-17T17:46:24.600497+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":674,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.600842+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:46:24.600842+0000","last_peered":"2024-08-17T17:46:24.600842+0000","last_clean":"2024-08-17T17:46:24.600842+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:46:24.600842+0000","last_undegraded":"2024-08-17T17:46:24.600842+0000","last_fullsized":"2024-08-17T17:46:24.600842+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1d","version":"90'149","reported_seq":739,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-08-17T17:46:24.871965+0000","last_change":"2024-08-17T17:42:18.668058+0000","last_active":"2024-08-17T17:46:24.871965+0000","last_peered":"2024-08-17T17:46:24.871965+0000","last_clean":"2024-08-17T17:46:24.871965+0000","last_became_active":"2024-08-17T17:42:05.876721+0000","last_became_peered":"2024-08-17T17:42:05.876721+0000","last_unstale":"2024-08-17T17:46:24.871965+0000","last_undegraded":"2024-08-17T17:46:24.871965+0000","last_fullsized":"2024-08-17T17:46:24.871965+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:18.667957+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:18.667957+0000","last_clean_scrub_stamp":"2024-08-17T17:42:18.667957+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:38.170146+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":58,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.727979+0000","last_change":"2024-08-17T17:41:34.611711+0000","last_active":"2024-08-17T17:46:01.727979+0000","last_peered":"2024-08-17T17:46:01.727979+0000","last_clean":"2024-08-17T17:46:01.727979+0000","last_became_active":"2024-08-17T17:41:34.611407+0000","last_became_peered":"2024-08-17T17:41:34.611407+0000","last_unstale":"2024-08-17T17:46:01.727979+0000","last_undegraded":"2024-08-17T17:46:01.727979+0000","last_fullsized":"2024-08-17T17:46:01.727979+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:42:41.582320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":479,"reported_epoch":393,"state":"active+clean","last_fresh":"2024-08-17T17:46:01.735084+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:46:01.735084+0000","last_peered":"2024-08-17T17:46:01.735084+0000","last_clean":"2024-08-17T17:46:01.735084+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:46:01.735084+0000","last_undegraded":"2024-08-17T17:46:01.735084+0000","last_fullsized":"2024-08-17T17:46:01.735084+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":320,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":237088,"num_write_kb":7586753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6480,"num_bytes_recovered":251002880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":386138112,"data_stored":386138112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102288,"ondisk_log_size":102288,"up":48,"acting":47,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":11256,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19280,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5978,"num_read_kb":12991,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4619,"ondisk_log_size":4619,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234475,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":907584,"kb_used_data":9524,"kb_used_omap":1,"kb_used_meta":898046,"kb_avail":92837568,"statfs":{"total":95995035648,"available":95065669632,"internally_reserved":0,"allocated":9752576,"data_stored":7510123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":919599448},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496110,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":900512,"kb_used_data":9560,"kb_used_omap":1,"kb_used_meta":890942,"kb_avail":92844640,"statfs":{"total":95995035648,"available":95072911360,"internally_reserved":0,"allocated":9789440,"data_stored":7566417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":912324952},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757747,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":896032,"kb_used_data":5400,"kb_used_omap":1,"kb_used_meta":890622,"kb_avail":92849120,"statfs":{"total":95995035648,"available":95077498880,"internally_reserved":0,"allocated":5529600,"data_stored":3303146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":911997272},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019385,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306668,"kb_used_data":5156,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93438484,"statfs":{"total":95995035648,"available":95681007616,"internally_reserved":0,"allocated":5279744,"data_stored":3045020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435612,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":965608,"kb_used_data":5468,"kb_used_omap":4,"kb_used_meta":960123,"kb_avail":92779544,"statfs":{"total":95995035648,"available":95006253056,"internally_reserved":0,"allocated":5599232,"data_stored":3376292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":983165962},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599139,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1226560,"kb_used_data":386608,"kb_used_omap":7,"kb_used_meta":839928,"kb_avail":92518592,"statfs":{"total":95995035648,"available":94739038208,"internally_reserved":0,"allocated":395886592,"data_stored":393652602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":860086434},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":268,"seq":1151051235486,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":459576,"kb_used_data":5160,"kb_used_omap":10,"kb_used_meta":454389,"kb_avail":93285576,"statfs":{"total":95995035648,"available":95524429824,"internally_reserved":0,"allocated":5283840,"data_stored":3056929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10902,"internal_metadata":465294698},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926192,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":931612,"kb_used_data":5460,"kb_used_omap":7,"kb_used_meta":926136,"kb_avail":92813540,"statfs":{"total":95995035648,"available":95041064960,"internally_reserved":0,"allocated":5591040,"data_stored":3368430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":948363585},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7550,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":386138112,"data_stored":386138112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:46:27.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph status --format=json 2024-08-17T17:46:27.872 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:27.895 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:27.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:27.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:27.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:27 smithi053 ceph-mon[89963]: pgmap v1323: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 295 op/s; 320/84 objects degraded (380.952%); 0 B/s, 235 objects/s recovering 2024-08-17T17:46:27.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:27 smithi053 ceph-mon[89963]: from='client.46799 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:46:27.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:27 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3887846654' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:46:27.997 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:28.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:28.020 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:28.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:28.121 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:28.148 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:28.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:28.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:28.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:28.273 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:28.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:28.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:28.375 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:28.397 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:28.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:28.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:28.499 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:28.526 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:28.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:28.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:28.627 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:28.650 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:28.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:28.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:28.754 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:28.780 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:28.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:28.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:28.882 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:28.909 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:28.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:28.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:29.010 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:29.032 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:29.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:29.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:29.134 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:29.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:29.163 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:29.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:29.264 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:29.287 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:29.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:29.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:29.389 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:29.416 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:29.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:29.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:29.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:29.540 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:29.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:29.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:29.642 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:29.665 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:29.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:29.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:29.766 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:29.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:29 smithi003 ceph-mon[100615]: pgmap v1324: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 250 op/s; 320/84 objects degraded (380.952%); 0 B/s, 235 objects/s recovering 2024-08-17T17:46:29.774 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:29 smithi003 ceph-mon[103344]: pgmap v1324: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 250 op/s; 320/84 objects degraded (380.952%); 0 B/s, 235 objects/s recovering 2024-08-17T17:46:29.793 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:29.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:29.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:29.894 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:29.917 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:29.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:29.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:29.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:29 smithi053 ceph-mon[89963]: pgmap v1324: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 250 op/s; 320/84 objects degraded (380.952%); 0 B/s, 235 objects/s recovering 2024-08-17T17:46:30.019 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:30.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:30.042 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:30.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:30.143 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:30.169 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:30.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:30.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:30.173 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-17T17:46:30.193 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:46:30.193 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:46:30.271 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:30.298 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:30.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:30.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:30.399 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:30.426 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:30.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:30.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:30.445 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:46:30.445 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:46:30.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:46:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:46:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:46:30.527 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:30.554 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:30.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:30.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:30.656 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:30.681 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:30.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:30.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:30.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:30 smithi003 ceph-mon[100615]: Health check update: Degraded data redundancy: 320/84 objects degraded (380.952%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:46:30.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:30 smithi003 ceph-mon[103344]: Health check update: Degraded data redundancy: 320/84 objects degraded (380.952%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:46:30.783 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:30.805 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:30.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:30.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:30.907 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:30.934 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:30.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:30.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:30.958 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-17T17:46:30.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:30 smithi053 ceph-mon[89963]: Health check update: Degraded data redundancy: 320/84 objects degraded (380.952%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-08-17T17:46:31.036 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:31.061 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:31.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:31.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:31.162 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:31.187 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:31.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:31.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:31.249 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:46:31.289 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:31.311 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:31.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:31.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:31.412 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:31.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:31.437 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:31.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:31.538 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:31.561 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:31.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:31.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:31.660 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:31 smithi053 ceph-mon[89963]: pgmap v1325: 119 pgs: 119 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 117 op/s; 0 B/s, 252 objects/s recovering 2024-08-17T17:46:31.660 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:31 smithi053 ceph-mon[89963]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 320/84 objects degraded (380.952%), 1 pg degraded, 1 pg undersized) 2024-08-17T17:46:31.663 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:31.685 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:31.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:31.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:31.787 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:31.809 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:31.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:31.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:31.852 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:46:31.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:31 smithi003 ceph-mon[103344]: pgmap v1325: 119 pgs: 119 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 117 op/s; 0 B/s, 252 objects/s recovering 2024-08-17T17:46:31.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:31 smithi003 ceph-mon[103344]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 320/84 objects degraded (380.952%), 1 pg degraded, 1 pg undersized) 2024-08-17T17:46:31.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:31 smithi003 ceph-mon[100615]: pgmap v1325: 119 pgs: 119 active+clean; 4.6 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 117 op/s; 0 B/s, 252 objects/s recovering 2024-08-17T17:46:31.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:31 smithi003 ceph-mon[100615]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 320/84 objects degraded (380.952%), 1 pg degraded, 1 pg undersized) 2024-08-17T17:46:31.911 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:31.934 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:31.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:31.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:32.036 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:32.061 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:32.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:32.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:32.162 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:32.185 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:32.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:32.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:32.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:32.310 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:32.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:32.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:32.412 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:32.435 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:32.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:32.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:32.537 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:32.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:32.560 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:32.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:32.661 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:32.684 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:32.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:32.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:32.785 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:32.808 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:32.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:32.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:32.819 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:46:32.822 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:46:32.909 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:32.931 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:32.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:32.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:32.954 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:46:32.954 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:46:33.033 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:33.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:33.056 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:33.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:33.157 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:33.183 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:33.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:33.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:33.285 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:33.307 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:33.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:33.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:33.408 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:33.431 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:33.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:33.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:33.533 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:33.560 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:33.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:33.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:33.578 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-17T17:46:33.662 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:33.684 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:33.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:33.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:33.716 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:33 smithi053 ceph-mon[89963]: pgmap v1326: 119 pgs: 119 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 2 op/s; 0 B/s, 39 objects/s recovering 2024-08-17T17:46:33.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:33 smithi003 ceph-mon[100615]: pgmap v1326: 119 pgs: 119 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 2 op/s; 0 B/s, 39 objects/s recovering 2024-08-17T17:46:33.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:33 smithi003 ceph-mon[103344]: pgmap v1326: 119 pgs: 119 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 2 op/s; 0 B/s, 39 objects/s recovering 2024-08-17T17:46:33.786 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:33.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:33.810 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:33.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:33.894 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:46:33.911 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:33.934 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:33.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:33.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:34.036 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:34.057 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:34.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:34.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:34.159 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:34.182 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:34.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:34.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:34.283 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:34.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:34.309 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:34.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:34.410 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:34.438 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:34.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:34.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:34.540 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:34.565 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:34.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:34.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:34.666 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:34.690 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:34.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:34.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:34.792 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:34.814 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:34.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:34.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:34.915 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:34.941 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:34.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:34.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:35.043 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:35.066 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:35.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:35.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:35.168 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:35.191 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:35.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:35.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:35.292 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:35.317 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:35.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:35.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:35.418 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:35.443 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:35.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:35.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:35.513 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:46:35.514 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:46:35.545 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:35.567 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:35.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:35.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:35.669 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:35.691 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:35.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:35.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:35.723 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:35 smithi003 ceph-mon[103344]: pgmap v1327: 119 pgs: 119 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 2 op/s; 0 B/s, 32 objects/s recovering 2024-08-17T17:46:35.723 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:35 smithi003 ceph-mon[100615]: pgmap v1327: 119 pgs: 119 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 2 op/s; 0 B/s, 32 objects/s recovering 2024-08-17T17:46:35.724 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:46:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:46:35] "GET /metrics HTTP/1.1" 200 36556 "" "Prometheus/2.51.0" 2024-08-17T17:46:35.793 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:35.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:35.817 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:35.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:35.918 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:35.940 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:35.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:35.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:35.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:35 smithi053 ceph-mon[89963]: pgmap v1327: 119 pgs: 119 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 2 op/s; 0 B/s, 32 objects/s recovering 2024-08-17T17:46:35.967 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:46:35.967 INFO:teuthology.orchestra.run.smithi003.stdout:{"fsid":"2b5b662e-5cba-11ef-bccf-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false}},"mutes":[]},"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1937,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":396,"num_osds":8,"num_up_osds":8,"osd_up_since":1723915999,"num_in_osds":8,"osd_in_since":1723916493,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":119}],"num_pgs":119,"num_pools":5,"num_objects":28,"data_bytes":4790221,"bytes_used":6362275840,"bytes_avail":761598009344,"bytes_total":767960285184,"recovering_objects_per_sec":32,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":320,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":2},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi003.civgvf","status":"up:active","gid":14664}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.3:8443/","prometheus":"http://172.21.15.3:9283/"}},"servicemap":{"epoch":30,"modified":"2024-08-17T17:45:18.413636+0000","services":{}},"progress_events":{"fcbeb9b7-7baf-488b-9fe4-c01aca441f9c":{"message":"Upgrade to 19.0.0-5836-gcd8bdbeb (4m)\n [============................] (remaining: 5m)","progress":0.44999998807907104,"add_to_ceph_s":true}}} 2024-08-17T17:46:36.041 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:36.063 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:36.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:36.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:36.138 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-17T17:46:36.166 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:36.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:36.189 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:36.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:36.290 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:36.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:36.312 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:36.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:36.403 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:46:36.413 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:36.463 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:36.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:36.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:36.565 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:36.588 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:36.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:36.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:36.689 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:36.712 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:36.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:36.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:36.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:36 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3467904047' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T17:46:36.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:36 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3467904047' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T17:46:36.792 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:36 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3467904047' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T17:46:36.814 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:36.836 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:36.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:36.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:36.938 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:36.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:36.961 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:36.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:37.062 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:37.085 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:37.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:37.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:37.186 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:37.209 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:37.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:37.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:37.311 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:37.333 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:37.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:37.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:37.435 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:37.458 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:37.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:37.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:37.471 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:46:37.471 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:46:37.488 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:46:37.488 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:46:37.505 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:46:37.505 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:46:37.521 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:46:37.521 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:46:37.537 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:46:37.537 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:46:37.553 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:46:37.553 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:46:37.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:37.569 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:46:37.569 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:46:37.581 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:37.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:37.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:37.585 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:46:37.585 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:46:37.634 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-08-17T17:46:37.683 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:37.706 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:37.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:37.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:37.772 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:37 smithi003 ceph-mon[103344]: pgmap v1328: 119 pgs: 119 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 29 objects/s recovering 2024-08-17T17:46:37.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:37 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/899208489' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:46:37.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:37 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:46:37.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:37 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2343172397' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:46:37.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:37 smithi003 ceph-mon[100615]: pgmap v1328: 119 pgs: 119 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 29 objects/s recovering 2024-08-17T17:46:37.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:37 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/899208489' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:46:37.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:37 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:46:37.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:37 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2343172397' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:46:37.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:37.830 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:37.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:37.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:37.833 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:37 smithi053 ceph-mon[89963]: pgmap v1328: 119 pgs: 119 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 29 objects/s recovering 2024-08-17T17:46:37.833 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:37 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/899208489' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:46:37.833 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:37 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:46:37.833 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:37 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2343172397' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:46:37.932 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:37.954 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:37.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:37.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:38.056 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:38.057 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:46:38.057 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2024-08-17T17:46:38.057 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2024-08-17T17:46:38.057 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2024-08-17T17:46:38.057 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.058 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:199665 8.4 8:2fc9da80:::benchmark_data_smithi003_229103_object92190:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e295)", 2024-08-17T17:46:38.058 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:40.260406+0000", 2024-08-17T17:46:38.058 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 597.79413076000003, 2024-08-17T17:46:38.058 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.17482935799999999, 2024-08-17T17:46:38.058 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:38.058 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:38.058 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:38.058 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:46:38.058 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:46:38.058 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 199665 2024-08-17T17:46:38.058 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.058 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:38.058 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.059 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:38.059 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.260406+0000", 2024-08-17T17:46:38.059 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.059 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.059 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.059 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:38.059 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.260406+0000", 2024-08-17T17:46:38.059 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.059 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.059 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.059 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:38.059 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.260405+0000", 2024-08-17T17:46:38.059 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:46:38.059 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.060 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.060 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:38.060 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.260409+0000", 2024-08-17T17:46:38.060 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4549999999999997e-06 2024-08-17T17:46:38.060 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.060 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.060 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:38.060 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.260418+0000", 2024-08-17T17:46:38.060 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.8720000000000008e-06 2024-08-17T17:46:38.060 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.060 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.060 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:38.060 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.260422+0000", 2024-08-17T17:46:38.060 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2590000000000002e-06 2024-08-17T17:46:38.061 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.061 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.061 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:38.061 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.264708+0000", 2024-08-17T17:46:38.061 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0042857590000000001 2024-08-17T17:46:38.061 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.061 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.061 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:38.061 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.264787+0000", 2024-08-17T17:46:38.061 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.9277999999999995e-05 2024-08-17T17:46:38.061 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.061 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.061 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:46:38.062 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.264861+0000", 2024-08-17T17:46:38.062 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3536999999999997e-05 2024-08-17T17:46:38.062 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.062 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.062 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:38.062 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.269373+0000", 2024-08-17T17:46:38.062 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0045125080000000001 2024-08-17T17:46:38.062 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.062 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.062 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.062 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.270503+0000", 2024-08-17T17:46:38.062 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001130065 2024-08-17T17:46:38.063 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.063 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.063 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.063 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435126+0000", 2024-08-17T17:46:38.063 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.16462236399999999 2024-08-17T17:46:38.063 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.063 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.063 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:38.063 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435185+0000", 2024-08-17T17:46:38.063 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9175999999999997e-05 2024-08-17T17:46:38.063 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.063 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.063 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:38.063 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435235+0000", 2024-08-17T17:46:38.064 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9889000000000003e-05 2024-08-17T17:46:38.064 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.064 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:38.064 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.064 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.064 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.064 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:199682 8.2 8:459cfaea:::benchmark_data_smithi003_229103_object92207:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e295)", 2024-08-17T17:46:38.064 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:40.289885+0000", 2024-08-17T17:46:38.064 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 597.76465115600001, 2024-08-17T17:46:38.064 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.14626383300000001, 2024-08-17T17:46:38.064 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:38.064 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:38.064 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:38.064 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:46:38.065 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:46:38.065 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 199682 2024-08-17T17:46:38.065 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.065 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:38.065 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.065 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:38.065 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289885+0000", 2024-08-17T17:46:38.065 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.065 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.065 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.065 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:38.065 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289885+0000", 2024-08-17T17:46:38.065 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.065 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.066 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.066 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:38.066 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289884+0000", 2024-08-17T17:46:38.066 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:46:38.066 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.066 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.066 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:38.066 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289890+0000", 2024-08-17T17:46:38.066 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5989999999999998e-06 2024-08-17T17:46:38.066 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.066 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.066 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:38.066 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289902+0000", 2024-08-17T17:46:38.066 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2007999999999999e-05 2024-08-17T17:46:38.067 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.067 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.067 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:38.067 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289919+0000", 2024-08-17T17:46:38.067 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.702e-05 2024-08-17T17:46:38.067 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.067 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.067 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:38.067 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.289967+0000", 2024-08-17T17:46:38.067 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7367000000000002e-05 2024-08-17T17:46:38.067 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.067 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.067 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:38.067 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.290071+0000", 2024-08-17T17:46:38.067 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000103952 2024-08-17T17:46:38.068 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.068 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.068 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,5", 2024-08-17T17:46:38.068 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.290175+0000", 2024-08-17T17:46:38.068 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000104299 2024-08-17T17:46:38.068 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.068 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.068 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:38.068 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435038+0000", 2024-08-17T17:46:38.068 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.144863348 2024-08-17T17:46:38.068 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.068 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.068 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.068 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435989+0000", 2024-08-17T17:46:38.069 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00095057200000000001 2024-08-17T17:46:38.069 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.069 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.069 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.069 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.436062+0000", 2024-08-17T17:46:38.069 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3015999999999993e-05 2024-08-17T17:46:38.069 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.069 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.069 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:38.069 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.436109+0000", 2024-08-17T17:46:38.069 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7234000000000002e-05 2024-08-17T17:46:38.069 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.069 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.069 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:38.070 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.436149+0000", 2024-08-17T17:46:38.070 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9816e-05 2024-08-17T17:46:38.070 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.070 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:38.070 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.070 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.070 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.070 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop_reply(client.46025.0:199665 8.4 e295/272 ondisk, result = 0)", 2024-08-17T17:46:38.070 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:40.291703+0000", 2024-08-17T17:46:38.070 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 597.76283303100001, 2024-08-17T17:46:38.070 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.14354199400000001, 2024-08-17T17:46:38.070 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:38.071 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "started", 2024-08-17T17:46:38.071 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:38.071 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.071 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:38.071 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.291703+0000", 2024-08-17T17:46:38.071 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.071 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.071 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.071 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:38.071 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.291703+0000", 2024-08-17T17:46:38.071 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.071 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.072 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.072 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:38.072 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.291702+0000", 2024-08-17T17:46:38.072 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:46:38.072 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.072 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.072 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:38.072 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.291708+0000", 2024-08-17T17:46:38.072 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7710000000000003e-06 2024-08-17T17:46:38.072 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.072 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.072 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:38.072 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.430310+0000", 2024-08-17T17:46:38.073 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.13860172500000001 2024-08-17T17:46:38.073 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.073 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.073 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:38.073 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.430318+0000", 2024-08-17T17:46:38.073 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.4009999999999997e-06 2024-08-17T17:46:38.073 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.073 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.073 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:38.073 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435110+0000", 2024-08-17T17:46:38.073 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0047917469999999998 2024-08-17T17:46:38.073 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.073 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.073 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:38.073 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435119+0000", 2024-08-17T17:46:38.074 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.2280000000000004e-06 2024-08-17T17:46:38.074 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.074 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.074 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:38.074 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435245+0000", 2024-08-17T17:46:38.074 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012596800000000001 2024-08-17T17:46:38.074 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.074 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:38.074 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.074 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.074 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.074 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.46025.0:199687 8.4 8:2561a798:::benchmark_data_smithi003_229103_object92212:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e295)", 2024-08-17T17:46:38.074 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:36:40.300569+0000", 2024-08-17T17:46:38.074 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 597.75396686900001, 2024-08-17T17:46:38.075 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.13988004500000001, 2024-08-17T17:46:38.075 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:38.075 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:38.075 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:38.075 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.46025", 2024-08-17T17:46:38.075 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/565297143", 2024-08-17T17:46:38.075 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 199687 2024-08-17T17:46:38.075 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.075 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:38.075 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.075 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:38.075 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.300569+0000", 2024-08-17T17:46:38.076 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.076 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.076 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.076 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:38.076 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.300569+0000", 2024-08-17T17:46:38.076 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.076 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.076 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.076 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:38.076 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.300567+0000", 2024-08-17T17:46:38.076 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:46:38.076 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.076 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.077 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:38.077 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.300578+0000", 2024-08-17T17:46:38.077 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1070000000000001e-05 2024-08-17T17:46:38.077 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.077 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.077 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:38.077 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.430060+0000", 2024-08-17T17:46:38.077 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.129482028 2024-08-17T17:46:38.077 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.077 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.077 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:38.077 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.430078+0000", 2024-08-17T17:46:38.077 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.7278999999999998e-05 2024-08-17T17:46:38.078 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.078 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.078 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:38.078 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.430135+0000", 2024-08-17T17:46:38.078 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7362999999999999e-05 2024-08-17T17:46:38.078 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.078 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.078 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:38.079 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.430275+0000", 2024-08-17T17:46:38.079 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014024400000000001 2024-08-17T17:46:38.079 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.079 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.079 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,5", 2024-08-17T17:46:38.079 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.430407+0000", 2024-08-17T17:46:38.079 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000131309 2024-08-17T17:46:38.079 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.079 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.079 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:38.079 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435256+0000", 2024-08-17T17:46:38.079 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0048498359999999997 2024-08-17T17:46:38.080 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.080 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.080 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.080 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.435299+0000", 2024-08-17T17:46:38.080 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2561999999999997e-05 2024-08-17T17:46:38.080 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.080 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.080 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.080 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.440368+0000", 2024-08-17T17:46:38.080 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0050688039999999997 2024-08-17T17:46:38.080 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.080 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.081 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:38.081 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.440412+0000", 2024-08-17T17:46:38.081 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4582000000000001e-05 2024-08-17T17:46:38.081 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.081 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.081 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:38.081 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:36:40.440449+0000", 2024-08-17T17:46:38.081 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.7024999999999999e-05 2024-08-17T17:46:38.081 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.081 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:38.081 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.081 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.082 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.082 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:47655 10.1 10:8699caaa:::benchmark_data_smithi003_292749_object47654:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-08-17T17:46:38.082 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:14.182384+0000", 2024-08-17T17:46:38.082 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 203.87215230199999, 2024-08-17T17:46:38.082 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.25415833900000001, 2024-08-17T17:46:38.082 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:38.082 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:38.082 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:38.082 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:38.082 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:38.082 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 47655 2024-08-17T17:46:38.082 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.083 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:38.083 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.083 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:38.083 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182384+0000", 2024-08-17T17:46:38.083 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.083 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.083 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.083 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:38.083 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182384+0000", 2024-08-17T17:46:38.083 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.083 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.083 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.083 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:38.084 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182383+0000", 2024-08-17T17:46:38.084 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:46:38.084 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.084 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.084 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:38.084 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182543+0000", 2024-08-17T17:46:38.084 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016050299999999999 2024-08-17T17:46:38.084 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.084 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.084 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:38.084 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182569+0000", 2024-08-17T17:46:38.084 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5814000000000001e-05 2024-08-17T17:46:38.084 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.085 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.085 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:38.085 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182576+0000", 2024-08-17T17:46:38.085 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.6579999999999996e-06 2024-08-17T17:46:38.085 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.085 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.085 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:38.085 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182785+0000", 2024-08-17T17:46:38.085 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00020952300000000001 2024-08-17T17:46:38.085 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.085 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.085 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:38.086 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182968+0000", 2024-08-17T17:46:38.086 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00018305700000000001 2024-08-17T17:46:38.086 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.086 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.086 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,7", 2024-08-17T17:46:38.086 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.183099+0000", 2024-08-17T17:46:38.086 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013062999999999999 2024-08-17T17:46:38.086 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.086 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.086 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.086 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.436288+0000", 2024-08-17T17:46:38.087 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.25318881399999998 2024-08-17T17:46:38.087 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.087 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.088 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:38.088 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.436326+0000", 2024-08-17T17:46:38.088 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8656999999999999e-05 2024-08-17T17:46:38.088 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.089 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.089 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.089 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.436399+0000", 2024-08-17T17:46:38.089 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.2798000000000003e-05 2024-08-17T17:46:38.089 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.089 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.089 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:38.089 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.436501+0000", 2024-08-17T17:46:38.089 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000102064 2024-08-17T17:46:38.089 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.089 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.089 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:38.089 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.436542+0000", 2024-08-17T17:46:38.090 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1038000000000002e-05 2024-08-17T17:46:38.090 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.090 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:38.090 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.090 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.090 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.090 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:47658 10.e 10:71d71bc3:::benchmark_data_smithi003_292749_object47657:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-08-17T17:46:38.090 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:14.182625+0000", 2024-08-17T17:46:38.090 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 203.871911348, 2024-08-17T17:46:38.090 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.25454966400000001, 2024-08-17T17:46:38.091 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:38.091 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:38.091 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:38.091 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:38.091 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:38.091 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 47658 2024-08-17T17:46:38.091 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.091 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:38.091 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.091 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:38.091 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182625+0000", 2024-08-17T17:46:38.091 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.091 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.092 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.092 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:38.092 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182625+0000", 2024-08-17T17:46:38.092 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.092 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.092 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.092 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:38.092 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182624+0000", 2024-08-17T17:46:38.092 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:46:38.092 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.092 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.092 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:38.092 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182672+0000", 2024-08-17T17:46:38.093 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.8316000000000003e-05 2024-08-17T17:46:38.093 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.093 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.093 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:38.093 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182683+0000", 2024-08-17T17:46:38.093 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0292000000000001e-05 2024-08-17T17:46:38.093 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.093 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.093 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:38.093 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182688+0000", 2024-08-17T17:46:38.093 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3530000000000002e-06 2024-08-17T17:46:38.093 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.093 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.093 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:38.094 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182736+0000", 2024-08-17T17:46:38.094 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7615999999999999e-05 2024-08-17T17:46:38.094 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.094 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.094 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:38.094 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182854+0000", 2024-08-17T17:46:38.094 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000118405 2024-08-17T17:46:38.094 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.094 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.094 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,6", 2024-08-17T17:46:38.094 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182999+0000", 2024-08-17T17:46:38.094 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014474900000000001 2024-08-17T17:46:38.094 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.095 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.095 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.095 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.434748+0000", 2024-08-17T17:46:38.095 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.25174945700000001 2024-08-17T17:46:38.095 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.095 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.095 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:38.095 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.434786+0000", 2024-08-17T17:46:38.095 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.7866e-05 2024-08-17T17:46:38.095 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.095 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.095 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.095 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.437084+0000", 2024-08-17T17:46:38.096 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0022976289999999998 2024-08-17T17:46:38.096 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.096 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.096 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:38.096 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.437136+0000", 2024-08-17T17:46:38.096 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2423999999999998e-05 2024-08-17T17:46:38.096 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.096 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.096 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:38.096 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.437175+0000", 2024-08-17T17:46:38.096 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8452999999999999e-05 2024-08-17T17:46:38.096 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.096 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:38.096 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.097 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.097 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.097 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:47660 10.1 10:803ae287:::benchmark_data_smithi003_292749_object47659:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-08-17T17:46:38.097 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:14.182705+0000", 2024-08-17T17:46:38.097 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 203.87183171800001, 2024-08-17T17:46:38.097 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.25615803199999998, 2024-08-17T17:46:38.097 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:38.097 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:38.097 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:38.097 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:38.097 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:38.098 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 47660 2024-08-17T17:46:38.098 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.098 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:38.098 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.098 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:38.098 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182705+0000", 2024-08-17T17:46:38.098 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.098 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.099 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.099 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:38.099 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182705+0000", 2024-08-17T17:46:38.099 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.099 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.099 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.099 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:38.099 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182704+0000", 2024-08-17T17:46:38.099 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:46:38.099 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.099 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.099 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:38.099 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182731+0000", 2024-08-17T17:46:38.099 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.7056e-05 2024-08-17T17:46:38.100 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.100 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.100 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:38.100 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182740+0000", 2024-08-17T17:46:38.100 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.4530000000000002e-06 2024-08-17T17:46:38.100 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.100 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.100 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:38.100 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182746+0000", 2024-08-17T17:46:38.100 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.81e-06 2024-08-17T17:46:38.100 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.100 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.100 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:38.100 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.432151+0000", 2024-08-17T17:46:38.100 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.24940480800000001 2024-08-17T17:46:38.100 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.101 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.101 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:38.101 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.432280+0000", 2024-08-17T17:46:38.101 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000128756 2024-08-17T17:46:38.101 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.101 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.101 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,7", 2024-08-17T17:46:38.101 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.432444+0000", 2024-08-17T17:46:38.101 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000164344 2024-08-17T17:46:38.101 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.101 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.101 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.101 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.436320+0000", 2024-08-17T17:46:38.101 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0038760370000000001 2024-08-17T17:46:38.101 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.102 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.102 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:38.102 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.436331+0000", 2024-08-17T17:46:38.102 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0494e-05 2024-08-17T17:46:38.102 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.102 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.102 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.102 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.438739+0000", 2024-08-17T17:46:38.102 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002408243 2024-08-17T17:46:38.102 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.102 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.103 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:38.103 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.438826+0000", 2024-08-17T17:46:38.103 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.6760000000000003e-05 2024-08-17T17:46:38.103 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.103 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.103 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:38.103 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.438863+0000", 2024-08-17T17:46:38.103 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6829000000000003e-05 2024-08-17T17:46:38.103 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.103 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:38.103 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.104 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.104 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.104 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_repop(client.36612.0:47656 10.0 e370/357 10:02151d15:::benchmark_data_smithi003_292749_object47655:head v 370'2871, mlcod=370'2869)", 2024-08-17T17:46:38.104 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:14.183177+0000", 2024-08-17T17:46:38.104 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 203.871359662, 2024-08-17T17:46:38.104 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.25087112900000003, 2024-08-17T17:46:38.104 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:38.104 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:38.104 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:38.104 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.104 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:38.104 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.183177+0000", 2024-08-17T17:46:38.104 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.105 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.105 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.105 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:38.105 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.183177+0000", 2024-08-17T17:46:38.105 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.105 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.105 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.105 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:38.105 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.183176+0000", 2024-08-17T17:46:38.105 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:46:38.105 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.105 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.105 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:38.106 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.183217+0000", 2024-08-17T17:46:38.106 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0825999999999998e-05 2024-08-17T17:46:38.106 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.106 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.106 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:38.106 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.430423+0000", 2024-08-17T17:46:38.106 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.247206178 2024-08-17T17:46:38.106 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.106 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.107 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:38.107 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.430437+0000", 2024-08-17T17:46:38.107 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4673e-05 2024-08-17T17:46:38.107 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.107 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.107 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:38.107 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.430489+0000", 2024-08-17T17:46:38.107 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1541e-05 2024-08-17T17:46:38.107 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.107 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.107 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:38.107 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.430516+0000", 2024-08-17T17:46:38.107 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6764e-05 2024-08-17T17:46:38.108 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.108 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.108 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:38.108 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.433951+0000", 2024-08-17T17:46:38.108 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0034349010000000002 2024-08-17T17:46:38.108 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.108 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.108 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:38.108 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.434048+0000", 2024-08-17T17:46:38.108 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.7137000000000002e-05 2024-08-17T17:46:38.108 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.108 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:38.108 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.109 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.109 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.109 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:47666 10.e 10:7e345cce:::benchmark_data_smithi003_292749_object47665:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-08-17T17:46:38.109 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:14.187962+0000", 2024-08-17T17:46:38.109 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 203.86657381099999, 2024-08-17T17:46:38.109 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.25411109900000001, 2024-08-17T17:46:38.109 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:38.109 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:38.109 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:38.109 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:38.109 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:38.109 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 47666 2024-08-17T17:46:38.109 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.109 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:38.110 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.110 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:38.110 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.187962+0000", 2024-08-17T17:46:38.110 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.110 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.110 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.110 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:38.110 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.187962+0000", 2024-08-17T17:46:38.110 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.110 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.110 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.110 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:38.111 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.187961+0000", 2024-08-17T17:46:38.111 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:46:38.111 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.111 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.111 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:38.111 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.187997+0000", 2024-08-17T17:46:38.111 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6801999999999998e-05 2024-08-17T17:46:38.111 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.111 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.111 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:38.111 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.430420+0000", 2024-08-17T17:46:38.111 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.24242243999999999 2024-08-17T17:46:38.111 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.111 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.112 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:38.112 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.430438+0000", 2024-08-17T17:46:38.112 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.8215000000000001e-05 2024-08-17T17:46:38.112 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.112 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.112 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:38.112 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.432267+0000", 2024-08-17T17:46:38.112 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0018294559999999999 2024-08-17T17:46:38.112 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.112 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.112 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:38.112 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.432399+0000", 2024-08-17T17:46:38.112 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013195500000000001 2024-08-17T17:46:38.113 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.113 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.113 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,6", 2024-08-17T17:46:38.113 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.432528+0000", 2024-08-17T17:46:38.113 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012893799999999999 2024-08-17T17:46:38.113 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.113 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.113 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.113 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.434780+0000", 2024-08-17T17:46:38.113 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002251795 2024-08-17T17:46:38.113 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.113 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.113 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:38.113 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.434791+0000", 2024-08-17T17:46:38.114 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0694e-05 2024-08-17T17:46:38.114 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.114 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.114 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.114 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.441939+0000", 2024-08-17T17:46:38.114 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0071477600000000004 2024-08-17T17:46:38.114 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.114 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.114 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:38.114 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.442033+0000", 2024-08-17T17:46:38.114 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.4889999999999994e-05 2024-08-17T17:46:38.114 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.114 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.115 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:38.115 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.442074+0000", 2024-08-17T17:46:38.115 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0085e-05 2024-08-17T17:46:38.115 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.115 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:38.115 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.115 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.115 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.115 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:94244 10.1 10:893c945e:::benchmark_data_smithi003_292749_object94243:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-08-17T17:46:38.115 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:51.797718+0000", 2024-08-17T17:46:38.116 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 166.25681861800001, 2024-08-17T17:46:38.116 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.095825684999999994, 2024-08-17T17:46:38.116 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:38.116 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:38.116 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:38.116 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:38.116 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:38.116 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 94244 2024-08-17T17:46:38.116 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.116 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:38.116 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.117 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:38.117 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.797718+0000", 2024-08-17T17:46:38.117 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.117 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.117 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.117 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:38.117 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.797718+0000", 2024-08-17T17:46:38.117 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.117 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.117 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.117 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:38.117 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.797716+0000", 2024-08-17T17:46:38.117 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:46:38.117 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.118 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.118 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:38.118 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.797768+0000", 2024-08-17T17:46:38.118 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2086000000000002e-05 2024-08-17T17:46:38.118 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.118 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.118 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:38.118 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.797781+0000", 2024-08-17T17:46:38.118 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3182e-05 2024-08-17T17:46:38.118 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.118 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.118 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:38.118 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.797788+0000", 2024-08-17T17:46:38.118 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4690000000000002e-06 2024-08-17T17:46:38.119 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.119 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.119 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:38.119 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.797824+0000", 2024-08-17T17:46:38.119 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6636e-05 2024-08-17T17:46:38.119 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.119 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.119 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:38.119 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.797928+0000", 2024-08-17T17:46:38.119 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0001036 2024-08-17T17:46:38.119 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.119 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.119 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,7", 2024-08-17T17:46:38.119 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.798020+0000", 2024-08-17T17:46:38.120 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.2429999999999994e-05 2024-08-17T17:46:38.120 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.120 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.120 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:38.120 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.800065+0000", 2024-08-17T17:46:38.120 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.002044101 2024-08-17T17:46:38.120 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.120 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.120 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.120 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.800529+0000", 2024-08-17T17:46:38.120 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00046412500000000002 2024-08-17T17:46:38.120 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.121 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.121 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.121 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.893450+0000", 2024-08-17T17:46:38.121 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.092921163000000001 2024-08-17T17:46:38.121 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.121 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.121 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:38.121 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.893501+0000", 2024-08-17T17:46:38.121 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0954e-05 2024-08-17T17:46:38.121 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.121 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.121 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:38.121 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.893543+0000", 2024-08-17T17:46:38.121 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2539e-05 2024-08-17T17:46:38.122 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.122 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:38.122 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.122 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.122 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.122 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:140581 10.1 10:8c6c6b1f:::benchmark_data_smithi003_292749_object22035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-08-17T17:46:38.122 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:44:43.853724+0000", 2024-08-17T17:46:38.122 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 114.20081194799999, 2024-08-17T17:46:38.122 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.16983812300000001, 2024-08-17T17:46:38.122 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:38.122 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:38.122 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:38.123 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:38.123 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:38.123 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 140581 2024-08-17T17:46:38.123 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.123 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:38.123 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.123 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:38.123 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.853724+0000", 2024-08-17T17:46:38.123 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.123 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.123 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.123 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:38.123 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.853724+0000", 2024-08-17T17:46:38.124 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.124 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.124 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.124 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:38.124 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.853723+0000", 2024-08-17T17:46:38.124 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:46:38.124 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.124 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.124 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:38.124 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.853731+0000", 2024-08-17T17:46:38.124 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.9580000000000007e-06 2024-08-17T17:46:38.124 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.124 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.124 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:38.125 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.853745+0000", 2024-08-17T17:46:38.125 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4343e-05 2024-08-17T17:46:38.125 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.125 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.125 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:38.125 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.853752+0000", 2024-08-17T17:46:38.125 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.3309999999999998e-06 2024-08-17T17:46:38.125 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.125 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.125 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:38.125 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.853791+0000", 2024-08-17T17:46:38.125 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8495000000000003e-05 2024-08-17T17:46:38.125 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.125 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.125 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:38.126 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.853872+0000", 2024-08-17T17:46:38.126 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.1471000000000007e-05 2024-08-17T17:46:38.126 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.126 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.126 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,7", 2024-08-17T17:46:38.126 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.853959+0000", 2024-08-17T17:46:38.126 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.6775999999999997e-05 2024-08-17T17:46:38.126 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.126 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.126 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.126 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.863045+0000", 2024-08-17T17:46:38.126 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0090858750000000002 2024-08-17T17:46:38.126 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.127 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.127 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:38.127 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.863054+0000", 2024-08-17T17:46:38.127 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.2760000000000006e-06 2024-08-17T17:46:38.127 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.127 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.127 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.127 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:44.023449+0000", 2024-08-17T17:46:38.127 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.16039450999999999 2024-08-17T17:46:38.127 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.127 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.128 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:38.128 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:44.023524+0000", 2024-08-17T17:46:38.128 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.4920999999999996e-05 2024-08-17T17:46:38.128 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.128 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.128 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:38.128 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:44.023562+0000", 2024-08-17T17:46:38.128 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8729000000000001e-05 2024-08-17T17:46:38.128 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.128 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:38.128 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.128 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.128 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.128 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:140582 10.1 10:8d113be8:::benchmark_data_smithi003_292749_object22036:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-08-17T17:46:38.128 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:44:43.853762+0000", 2024-08-17T17:46:38.128 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 114.200774282, 2024-08-17T17:46:38.129 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.16991869200000001, 2024-08-17T17:46:38.129 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:38.129 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:38.129 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:38.129 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:38.129 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:38.129 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 140582 2024-08-17T17:46:38.129 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.129 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:38.129 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.129 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:38.129 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.853762+0000", 2024-08-17T17:46:38.129 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.129 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.129 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.129 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:38.130 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.853762+0000", 2024-08-17T17:46:38.130 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.130 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.130 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.130 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:38.130 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.853762+0000", 2024-08-17T17:46:38.130 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:46:38.130 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.130 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.130 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:38.130 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.853764+0000", 2024-08-17T17:46:38.130 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.514e-06 2024-08-17T17:46:38.130 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.130 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.131 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:38.131 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.853770+0000", 2024-08-17T17:46:38.131 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.587e-06 2024-08-17T17:46:38.131 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.131 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.131 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:38.131 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.853773+0000", 2024-08-17T17:46:38.131 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4819999999999999e-06 2024-08-17T17:46:38.131 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.131 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.131 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:38.131 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.856991+0000", 2024-08-17T17:46:38.132 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0032178139999999998 2024-08-17T17:46:38.132 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.132 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.132 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:38.132 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.857104+0000", 2024-08-17T17:46:38.132 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00011271499999999999 2024-08-17T17:46:38.132 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.132 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.132 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,7", 2024-08-17T17:46:38.132 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.857218+0000", 2024-08-17T17:46:38.132 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000114273 2024-08-17T17:46:38.132 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.132 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.132 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:38.133 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.863062+0000", 2024-08-17T17:46:38.133 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0058435730000000003 2024-08-17T17:46:38.133 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.133 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.133 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.133 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.863672+0000", 2024-08-17T17:46:38.133 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00061087399999999997 2024-08-17T17:46:38.133 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.133 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.133 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.133 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:44.023605+0000", 2024-08-17T17:46:38.133 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.159933095 2024-08-17T17:46:38.133 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.133 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.134 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:38.134 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:44.023649+0000", 2024-08-17T17:46:38.134 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3520999999999999e-05 2024-08-17T17:46:38.134 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.134 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.134 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:38.134 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:44.023681+0000", 2024-08-17T17:46:38.134 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1674999999999997e-05 2024-08-17T17:46:38.134 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.134 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:38.134 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.134 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.134 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.135 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:140586 10.1 10:805ff845:::benchmark_data_smithi003_292749_object22040:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-08-17T17:46:38.135 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:44:43.855796+0000", 2024-08-17T17:46:38.135 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 114.198739976, 2024-08-17T17:46:38.135 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.167987687, 2024-08-17T17:46:38.135 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:38.135 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:38.135 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:38.135 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:38.135 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:38.135 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 140586 2024-08-17T17:46:38.135 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.135 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:38.135 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.135 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:38.136 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.855796+0000", 2024-08-17T17:46:38.136 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.136 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.136 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.136 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:38.136 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.855796+0000", 2024-08-17T17:46:38.136 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.136 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.136 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.136 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:38.136 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.855795+0000", 2024-08-17T17:46:38.136 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:46:38.136 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.136 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.137 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:38.137 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.855801+0000", 2024-08-17T17:46:38.137 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9630000000000003e-06 2024-08-17T17:46:38.137 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.137 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.137 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:38.137 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.855817+0000", 2024-08-17T17:46:38.137 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6008e-05 2024-08-17T17:46:38.137 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.137 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.137 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:38.137 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.855823+0000", 2024-08-17T17:46:38.137 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1099999999999999e-06 2024-08-17T17:46:38.137 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.137 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.138 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:38.138 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.860061+0000", 2024-08-17T17:46:38.138 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0042380999999999999 2024-08-17T17:46:38.138 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.138 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.138 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:38.138 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.860159+0000", 2024-08-17T17:46:38.138 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.8047999999999994e-05 2024-08-17T17:46:38.138 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.138 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.138 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,7", 2024-08-17T17:46:38.138 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.860260+0000", 2024-08-17T17:46:38.138 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000101024 2024-08-17T17:46:38.139 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.139 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.139 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:38.139 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.863149+0000", 2024-08-17T17:46:38.139 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0028888709999999999 2024-08-17T17:46:38.139 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.139 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.139 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.139 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:43.863695+0000", 2024-08-17T17:46:38.139 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00054574199999999999 2024-08-17T17:46:38.139 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.139 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.139 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.139 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:44.023715+0000", 2024-08-17T17:46:38.139 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.16001976400000001 2024-08-17T17:46:38.140 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.140 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.140 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:38.140 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:44.023754+0000", 2024-08-17T17:46:38.140 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9335999999999997e-05 2024-08-17T17:46:38.140 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.140 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.140 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:38.140 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:44:44.023784+0000", 2024-08-17T17:46:38.140 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.0043e-05 2024-08-17T17:46:38.140 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.140 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:38.140 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.140 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.140 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.140 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:162864 10.e 10:7f048e0a:::benchmark_data_smithi003_292749_object44318:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-08-17T17:46:38.141 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:45:06.197889+0000", 2024-08-17T17:46:38.141 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 91.856646815999994, 2024-08-17T17:46:38.141 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.056288874000000003, 2024-08-17T17:46:38.141 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:38.141 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:38.141 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:38.141 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:38.141 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:38.141 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 162864 2024-08-17T17:46:38.141 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.141 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:38.141 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.141 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:38.141 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:06.197889+0000", 2024-08-17T17:46:38.141 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.142 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.142 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.142 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:38.142 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:06.197889+0000", 2024-08-17T17:46:38.142 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.142 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.142 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.142 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:38.142 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:06.197888+0000", 2024-08-17T17:46:38.142 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:46:38.142 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.142 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.142 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:38.143 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:06.197895+0000", 2024-08-17T17:46:38.143 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.4909999999999999e-06 2024-08-17T17:46:38.143 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.143 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.143 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:38.143 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:06.197917+0000", 2024-08-17T17:46:38.143 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2103000000000002e-05 2024-08-17T17:46:38.143 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.143 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.143 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:38.143 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:06.197924+0000", 2024-08-17T17:46:38.143 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.691e-06 2024-08-17T17:46:38.143 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.144 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.144 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:38.144 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:06.197979+0000", 2024-08-17T17:46:38.144 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4784000000000003e-05 2024-08-17T17:46:38.144 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.144 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.144 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:38.144 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:06.198104+0000", 2024-08-17T17:46:38.144 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012540000000000001 2024-08-17T17:46:38.144 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.144 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.144 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,6", 2024-08-17T17:46:38.144 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:06.198217+0000", 2024-08-17T17:46:38.144 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000113322 2024-08-17T17:46:38.145 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.145 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.145 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:38.145 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:06.201461+0000", 2024-08-17T17:46:38.145 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0032432519999999999 2024-08-17T17:46:38.145 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.145 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.145 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.145 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:06.206083+0000", 2024-08-17T17:46:38.145 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00462277 2024-08-17T17:46:38.145 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.145 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.145 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.145 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:06.254086+0000", 2024-08-17T17:46:38.146 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.048002230999999999 2024-08-17T17:46:38.146 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.146 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:38.147 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:38.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:38.147 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:38.148 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:06.254134+0000", 2024-08-17T17:46:38.148 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.8226999999999998e-05 2024-08-17T17:46:38.148 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.148 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.148 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:38.148 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:06.254178+0000", 2024-08-17T17:46:38.148 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4393e-05 2024-08-17T17:46:38.148 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.148 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:38.148 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.148 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.148 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.148 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:162870 10.e 10:735439f6:::benchmark_data_smithi003_292749_object44324:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-08-17T17:46:38.149 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:45:06.202729+0000", 2024-08-17T17:46:38.149 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 91.851807460000003, 2024-08-17T17:46:38.149 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.054628287999999997, 2024-08-17T17:46:38.149 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:38.149 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:38.149 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:38.149 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:38.149 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:38.149 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 162870 2024-08-17T17:46:38.149 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.149 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:38.149 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.149 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:38.149 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:06.202729+0000", 2024-08-17T17:46:38.149 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.150 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.150 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.150 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:38.150 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:06.202729+0000", 2024-08-17T17:46:38.150 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.150 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.150 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.150 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:38.150 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:06.202727+0000", 2024-08-17T17:46:38.150 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:46:38.150 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.150 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.150 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:38.151 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:06.202735+0000", 2024-08-17T17:46:38.151 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7139999999999993e-06 2024-08-17T17:46:38.151 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.151 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.151 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:38.151 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:06.202749+0000", 2024-08-17T17:46:38.151 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4579999999999999e-05 2024-08-17T17:46:38.151 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.151 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.151 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:38.151 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:06.202756+0000", 2024-08-17T17:46:38.151 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.613e-06 2024-08-17T17:46:38.151 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.151 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.151 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:38.151 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:06.202805+0000", 2024-08-17T17:46:38.152 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9055999999999999e-05 2024-08-17T17:46:38.152 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.152 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.152 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:38.152 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:06.202917+0000", 2024-08-17T17:46:38.152 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000112433 2024-08-17T17:46:38.152 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.152 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.152 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,6", 2024-08-17T17:46:38.152 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:06.203025+0000", 2024-08-17T17:46:38.152 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000107117 2024-08-17T17:46:38.152 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.152 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.152 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:38.153 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:06.206467+0000", 2024-08-17T17:46:38.153 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.003442765 2024-08-17T17:46:38.153 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.153 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.153 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.153 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:06.208006+0000", 2024-08-17T17:46:38.153 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0015386359999999999 2024-08-17T17:46:38.153 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.153 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.153 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.153 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:06.257235+0000", 2024-08-17T17:46:38.153 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.049229433000000003 2024-08-17T17:46:38.153 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.153 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.154 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:38.154 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:06.257316+0000", 2024-08-17T17:46:38.154 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0430000000000001e-05 2024-08-17T17:46:38.154 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.154 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.154 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:38.154 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:06.257357+0000", 2024-08-17T17:46:38.154 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1266000000000001e-05 2024-08-17T17:46:38.154 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.154 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:38.154 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.154 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.154 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.155 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:164355 10.1 10:8bfa59d1:::benchmark_data_smithi003_292749_object45809:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-08-17T17:46:38.155 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:45:07.658251+0000", 2024-08-17T17:46:38.155 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 90.396284980999994, 2024-08-17T17:46:38.155 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.053015829, 2024-08-17T17:46:38.155 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:38.155 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:38.155 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:38.155 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:38.155 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:38.155 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 164355 2024-08-17T17:46:38.155 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.155 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:38.155 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.155 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:38.155 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:07.658251+0000", 2024-08-17T17:46:38.156 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.156 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.156 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.156 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:38.156 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:07.658251+0000", 2024-08-17T17:46:38.156 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.156 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.156 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.156 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:38.156 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:07.658250+0000", 2024-08-17T17:46:38.156 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:46:38.156 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.157 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.157 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:38.157 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:07.658256+0000", 2024-08-17T17:46:38.157 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5050000000000004e-06 2024-08-17T17:46:38.157 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.157 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.157 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:38.157 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:07.658270+0000", 2024-08-17T17:46:38.157 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3835999999999999e-05 2024-08-17T17:46:38.157 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.157 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.157 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:38.157 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:07.658277+0000", 2024-08-17T17:46:38.158 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.1520000000000003e-06 2024-08-17T17:46:38.158 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.158 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.158 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:38.158 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:07.658338+0000", 2024-08-17T17:46:38.158 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0406999999999999e-05 2024-08-17T17:46:38.158 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.158 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.158 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:38.158 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:07.658450+0000", 2024-08-17T17:46:38.158 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000112355 2024-08-17T17:46:38.158 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.158 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.159 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,7", 2024-08-17T17:46:38.159 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:07.658573+0000", 2024-08-17T17:46:38.159 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012317900000000001 2024-08-17T17:46:38.159 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.159 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.159 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:38.159 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:07.661793+0000", 2024-08-17T17:46:38.159 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0032199669999999998 2024-08-17T17:46:38.159 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.159 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.159 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.159 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:07.663186+0000", 2024-08-17T17:46:38.159 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0013930909999999999 2024-08-17T17:46:38.159 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.159 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.159 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.160 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:07.711215+0000", 2024-08-17T17:46:38.160 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.048028795999999999 2024-08-17T17:46:38.160 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.160 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.160 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:38.160 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:07.711245+0000", 2024-08-17T17:46:38.160 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.9462e-05 2024-08-17T17:46:38.160 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.160 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.160 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:38.160 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:07.711267+0000", 2024-08-17T17:46:38.160 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.2597999999999999e-05 2024-08-17T17:46:38.160 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.160 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:38.160 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.161 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.161 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.161 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:164700 10.1 10:802fb6d9:::benchmark_data_smithi003_292749_object46154:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-08-17T17:46:38.161 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:45:08.040899+0000", 2024-08-17T17:46:38.161 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 90.013637142999997, 2024-08-17T17:46:38.161 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.049075052000000001, 2024-08-17T17:46:38.161 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:38.161 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:38.161 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:38.161 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:38.161 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:38.161 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 164700 2024-08-17T17:46:38.161 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.161 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:38.162 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.162 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:38.162 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:08.040899+0000", 2024-08-17T17:46:38.162 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.162 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.162 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.162 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:38.162 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:08.040899+0000", 2024-08-17T17:46:38.162 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.162 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.162 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.162 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:38.162 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:08.040898+0000", 2024-08-17T17:46:38.162 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:46:38.163 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.163 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.163 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:38.163 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:08.040909+0000", 2024-08-17T17:46:38.163 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0366999999999999e-05 2024-08-17T17:46:38.163 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.163 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.163 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:38.163 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:08.040920+0000", 2024-08-17T17:46:38.163 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1647e-05 2024-08-17T17:46:38.163 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.163 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.163 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:38.163 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:08.040926+0000", 2024-08-17T17:46:38.163 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2439999999999998e-06 2024-08-17T17:46:38.164 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.164 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.164 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:38.164 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:08.043691+0000", 2024-08-17T17:46:38.164 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0027642090000000001 2024-08-17T17:46:38.164 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.164 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.164 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:38.164 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:08.043794+0000", 2024-08-17T17:46:38.164 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010336 2024-08-17T17:46:38.164 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.164 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.164 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 0,7", 2024-08-17T17:46:38.164 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:08.043898+0000", 2024-08-17T17:46:38.164 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000103869 2024-08-17T17:46:38.165 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.165 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.165 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:38.165 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:08.046876+0000", 2024-08-17T17:46:38.165 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0029783969999999998 2024-08-17T17:46:38.165 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.165 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.165 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.165 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:08.056008+0000", 2024-08-17T17:46:38.165 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0091318779999999995 2024-08-17T17:46:38.165 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.165 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.165 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.165 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:08.089872+0000", 2024-08-17T17:46:38.165 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.033863788999999998 2024-08-17T17:46:38.166 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.166 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.166 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:38.166 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:08.089931+0000", 2024-08-17T17:46:38.166 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8715000000000003e-05 2024-08-17T17:46:38.166 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.166 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.166 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:38.166 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:08.089974+0000", 2024-08-17T17:46:38.166 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3568e-05 2024-08-17T17:46:38.166 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.166 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:38.166 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.166 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.167 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.167 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:168772 10.e 10:7af0033d:::benchmark_data_smithi003_292749_object50226:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-08-17T17:46:38.167 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:45:12.214816+0000", 2024-08-17T17:46:38.167 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 85.839720182999997, 2024-08-17T17:46:38.167 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.048343891999999999, 2024-08-17T17:46:38.167 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:38.167 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:38.167 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:38.167 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:38.167 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:38.167 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 168772 2024-08-17T17:46:38.167 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.168 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:38.168 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.168 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:38.168 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:12.214816+0000", 2024-08-17T17:46:38.168 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.168 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.168 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.168 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:38.168 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:12.214816+0000", 2024-08-17T17:46:38.168 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.168 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.168 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.168 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:38.169 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:12.214815+0000", 2024-08-17T17:46:38.169 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:46:38.169 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.169 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.169 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:38.169 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:12.214822+0000", 2024-08-17T17:46:38.169 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.2690000000000002e-06 2024-08-17T17:46:38.169 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.169 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.169 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:38.169 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:12.214834+0000", 2024-08-17T17:46:38.169 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2463000000000001e-05 2024-08-17T17:46:38.169 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.169 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.170 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:38.170 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:12.214840+0000", 2024-08-17T17:46:38.170 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3879999999999999e-06 2024-08-17T17:46:38.170 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.170 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.170 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:38.170 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:12.217202+0000", 2024-08-17T17:46:38.170 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0023623989999999998 2024-08-17T17:46:38.170 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.170 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.170 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:38.170 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:12.217313+0000", 2024-08-17T17:46:38.170 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0001106 2024-08-17T17:46:38.171 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.171 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.171 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,6", 2024-08-17T17:46:38.171 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:12.217416+0000", 2024-08-17T17:46:38.171 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000102913 2024-08-17T17:46:38.171 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.171 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.171 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:38.171 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:12.220157+0000", 2024-08-17T17:46:38.171 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0027410009999999999 2024-08-17T17:46:38.171 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.171 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.171 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.172 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:12.246312+0000", 2024-08-17T17:46:38.172 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.026155571999999998 2024-08-17T17:46:38.172 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.172 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.172 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.172 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:12.263086+0000", 2024-08-17T17:46:38.172 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.016773349 2024-08-17T17:46:38.172 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.172 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.172 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:38.172 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:12.263128+0000", 2024-08-17T17:46:38.172 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2324000000000003e-05 2024-08-17T17:46:38.172 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.172 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.173 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:38.173 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:12.263160+0000", 2024-08-17T17:46:38.173 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.2023000000000002e-05 2024-08-17T17:46:38.173 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.173 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:38.173 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.173 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.173 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.173 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:188946 10.e 10:729d3c0f:::benchmark_data_smithi003_292749_object70400:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-08-17T17:46:38.173 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:45:32.018101+0000", 2024-08-17T17:46:38.173 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 66.036435374999996, 2024-08-17T17:46:38.173 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.045829697000000003, 2024-08-17T17:46:38.173 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:38.173 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:38.174 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:38.174 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:38.174 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:38.174 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 188946 2024-08-17T17:46:38.174 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.174 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:38.174 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.174 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:38.174 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:32.018101+0000", 2024-08-17T17:46:38.174 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.174 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.174 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.174 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:38.174 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:32.018101+0000", 2024-08-17T17:46:38.175 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.175 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.175 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.175 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:38.175 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:32.018100+0000", 2024-08-17T17:46:38.175 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:46:38.175 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.175 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.175 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:38.175 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:32.018105+0000", 2024-08-17T17:46:38.175 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9239999999999998e-06 2024-08-17T17:46:38.175 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.175 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.175 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:38.175 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:32.018116+0000", 2024-08-17T17:46:38.176 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1375999999999999e-05 2024-08-17T17:46:38.176 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.176 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.176 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:38.176 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:32.018122+0000", 2024-08-17T17:46:38.176 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8880000000000002e-06 2024-08-17T17:46:38.176 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.176 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.176 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:38.176 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:32.032575+0000", 2024-08-17T17:46:38.176 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.014452242000000001 2024-08-17T17:46:38.176 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.176 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.176 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:38.176 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:32.032680+0000", 2024-08-17T17:46:38.177 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000104984 2024-08-17T17:46:38.177 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.177 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.177 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,6", 2024-08-17T17:46:38.177 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:32.032778+0000", 2024-08-17T17:46:38.177 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.8313000000000006e-05 2024-08-17T17:46:38.177 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.177 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.177 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.177 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:32.037253+0000", 2024-08-17T17:46:38.177 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0044753359999999999 2024-08-17T17:46:38.177 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.177 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.177 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:38.177 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:32.062225+0000", 2024-08-17T17:46:38.177 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.024971700999999999 2024-08-17T17:46:38.178 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.178 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.178 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.178 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:32.063835+0000", 2024-08-17T17:46:38.178 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001610516 2024-08-17T17:46:38.178 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.178 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.178 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:38.178 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:32.063886+0000", 2024-08-17T17:46:38.178 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.0763000000000002e-05 2024-08-17T17:46:38.178 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.178 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.178 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:38.179 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:32.063931+0000", 2024-08-17T17:46:38.179 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4404999999999999e-05 2024-08-17T17:46:38.179 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.179 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:38.179 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.179 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.179 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.179 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:188949 10.e 10:776fe166:::benchmark_data_smithi003_292749_object70403:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-08-17T17:46:38.179 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:45:32.019687+0000", 2024-08-17T17:46:38.179 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 66.034849734000005, 2024-08-17T17:46:38.180 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.044359195999999997, 2024-08-17T17:46:38.180 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:38.180 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:38.180 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:38.180 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:38.180 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:38.180 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 188949 2024-08-17T17:46:38.180 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.180 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:38.180 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.180 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:38.180 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:32.019687+0000", 2024-08-17T17:46:38.180 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.180 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.180 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.181 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:38.181 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:32.019687+0000", 2024-08-17T17:46:38.181 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:38.181 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.181 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.181 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:38.181 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:32.019685+0000", 2024-08-17T17:46:38.181 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:46:38.181 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.181 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.181 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:38.181 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:32.019691+0000", 2024-08-17T17:46:38.181 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6069999999999998e-06 2024-08-17T17:46:38.181 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.182 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.182 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:38.182 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:32.019716+0000", 2024-08-17T17:46:38.182 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.5534999999999999e-05 2024-08-17T17:46:38.182 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.182 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.182 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:38.182 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:32.019723+0000", 2024-08-17T17:46:38.182 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.2809999999999997e-06 2024-08-17T17:46:38.182 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.182 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.182 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:38.183 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:32.037276+0000", 2024-08-17T17:46:38.183 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017553749 2024-08-17T17:46:38.183 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.183 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.183 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:38.183 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:32.037374+0000", 2024-08-17T17:46:38.183 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.7757999999999995e-05 2024-08-17T17:46:38.183 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.183 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.183 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,6", 2024-08-17T17:46:38.183 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:32.037478+0000", 2024-08-17T17:46:38.183 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000103402 2024-08-17T17:46:38.184 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.184 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.184 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.184 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:32.046463+0000", 2024-08-17T17:46:38.184 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0089857089999999997 2024-08-17T17:46:38.184 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.184 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.184 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:38.184 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:32.062306+0000", 2024-08-17T17:46:38.184 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.015842722 2024-08-17T17:46:38.184 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.184 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.184 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:38.184 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:32.063969+0000", 2024-08-17T17:46:38.184 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001662763 2024-08-17T17:46:38.184 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.185 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.185 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:38.185 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:32.064008+0000", 2024-08-17T17:46:38.185 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9461e-05 2024-08-17T17:46:38.185 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:38.185 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:38.185 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:38.185 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:45:32.064046+0000", 2024-08-17T17:46:38.185 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.7435000000000003e-05 2024-08-17T17:46:38.185 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.185 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:38.185 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.185 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:38.185 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:38.185 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:46:38.248 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:38.270 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:38.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:38.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:38.343 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-08-17T17:46:38.343 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:46:38.371 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:38.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:38.395 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:38.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:38.496 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:38.518 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:38.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:38.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:38.620 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:38.630 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:38 smithi003 ceph-mon[100615]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:46:38.630 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:38 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:46:38.630 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:38 smithi003 ceph-mon[100615]: osdmap e397: 8 total, 8 up, 8 in 2024-08-17T17:46:38.630 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:46:38.644 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:38.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:38.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:38.745 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:38.767 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:38.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:38.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:38.869 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:38.891 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:38.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:38.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:38.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:38 smithi053 ceph-mon[89963]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:46:38.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:38 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:46:38.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:38 smithi053 ceph-mon[89963]: osdmap e397: 8 total, 8 up, 8 in 2024-08-17T17:46:38.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:46:38.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:46:38.993 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:39.016 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:39.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:39.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:39.022 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:46:39.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:38 smithi003 ceph-mon[103344]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:46:39.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:38 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-17T17:46:39.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:38 smithi003 ceph-mon[103344]: osdmap e397: 8 total, 8 up, 8 in 2024-08-17T17:46:39.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:46:39.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:46:39.118 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:39.142 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:39.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:39.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:39.243 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:39.267 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:39.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:39.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:39.369 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:39.393 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:39.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:39.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:39.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:39.519 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:39.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:39.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:39.611 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-17T17:46:39.620 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:39.678 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:39.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:39.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:39.683 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_5' removed 2024-08-17T17:46:39.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:39 smithi003 ceph-mon[103344]: pgmap v1330: 119 pgs: 119 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 31 objects/s recovering 2024-08-17T17:46:39.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:46:39.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:39 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2950793554' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:46:39.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:39 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:46:39.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:39 smithi003 ceph-mon[100615]: pgmap v1330: 119 pgs: 119 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 31 objects/s recovering 2024-08-17T17:46:39.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:46:39.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:39 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2950793554' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:46:39.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:39 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:46:39.780 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:39.801 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:39.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:39.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:39.903 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:39.904 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:46:39.927 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:39.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:39.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:39.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:39 smithi053 ceph-mon[89963]: pgmap v1330: 119 pgs: 119 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 31 objects/s recovering 2024-08-17T17:46:39.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:46:39.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:39 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2950793554' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:46:39.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:39 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-08-17T17:46:40.029 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:40.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:40.057 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:40.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:40.158 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:40.181 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:40.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:40.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:40.283 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:40.305 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:40.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:40.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:40.323 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:46:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:46:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:46:40.407 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:40.430 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:40.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:40.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:40.532 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:40.554 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:40.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:40.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:40.619 INFO:teuthology.task.sequential:In sequential, running task print... 2024-08-17T17:46:40.620 INFO:teuthology.task.print:**** done end radosbench.yaml 2024-08-17T17:46:40.620 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-08-17T17:46:40.621 INFO:tasks.workunit:Pulling workunits from ref quincy 2024-08-17T17:46:40.621 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-08-17T17:46:40.621 DEBUG:teuthology.orchestra.run.smithi003:> stat -- /home/ubuntu/cephtest/mnt.0 2024-08-17T17:46:40.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:40.654 INFO:teuthology.orchestra.run.smithi003.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-17T17:46:40.654 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-08-17T17:46:40.657 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:40.663 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:46:40.688 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:40.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:40.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:40.700 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-08-17T17:46:40.700 DEBUG:teuthology.orchestra.run.smithi003:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-08-17T17:46:40.756 INFO:tasks.workunit:timeout=3h 2024-08-17T17:46:40.757 INFO:tasks.workunit:cleanup=True 2024-08-17T17:46:40.757 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch quincy https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 2024-08-17T17:46:40.790 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:40.814 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:40.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:40.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:40.826 INFO:tasks.workunit.client.0.smithi003.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-08-17T17:46:40.915 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:40.916 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:40 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T17:46:40.916 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:40 smithi003 ceph-mon[103344]: osdmap e398: 8 total, 8 up, 8 in 2024-08-17T17:46:40.917 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:40 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T17:46:40.917 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:40 smithi003 ceph-mon[100615]: osdmap e398: 8 total, 8 up, 8 in 2024-08-17T17:46:40.938 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:40.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:40.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:40.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:40 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-08-17T17:46:40.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:40 smithi053 ceph-mon[89963]: osdmap e398: 8 total, 8 up, 8 in 2024-08-17T17:46:41.040 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:41.062 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:41.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:41.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:41.141 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:46:41.164 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:41.206 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:41.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:41.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:41.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:41.332 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:41.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:41.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:41.433 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:41.456 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:41.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:41.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:41.550 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:46:41.551 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2024-08-17T17:46:41.551 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 0 2024-08-17T17:46:41.551 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:46:41.557 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:41.579 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:41.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:41.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:41.680 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:41.705 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:41.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:41.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:41.806 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:41.829 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:41.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:41.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:41.844 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:46:41.892 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:41 smithi003 ceph-mon[103344]: pgmap v1332: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:46:41.893 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:41 smithi003 ceph-mon[100615]: pgmap v1332: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:46:41.931 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:41.954 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:41.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:41.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:41.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:41 smithi053 ceph-mon[89963]: pgmap v1332: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:46:41.979 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:46:42.057 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:42.080 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:42.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:42.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:42.154 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-17T17:46:42.181 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:42.207 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:42.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:42.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:42.309 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:42.332 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:42.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:42.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:42.434 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:42.459 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:42.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:42.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:42.462 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:46:42.561 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:42.584 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:42.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:42.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:42.686 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:42.709 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:42.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:42.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:42.812 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:42.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:42.835 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:42.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:42.936 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:42.958 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:42.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:42.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:43.060 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:43.083 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:43.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:43.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:43.185 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:43.208 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:43.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:43.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:43.310 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:43.332 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:43.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:43.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:43.435 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:43.458 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:43.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:43.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:43.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:43.584 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:43.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:43.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:43.685 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:43.710 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:43.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:43.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:43.713 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:43 smithi003 ceph-mon[103344]: pgmap v1333: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:46:43.713 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:43 smithi003 ceph-mon[100615]: pgmap v1333: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:46:43.811 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:43.817 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:46:43.817 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:46:43.835 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:46:43.835 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:46:43.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:43.836 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:43.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:43.852 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:46:43.853 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:46:43.869 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:46:43.870 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:46:43.886 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:46:43.886 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:46:43.902 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:46:43.903 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:46:43.918 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:46:43.919 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:46:43.934 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:46:43.934 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:46:43.937 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:43.960 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:43.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:43.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:43.962 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:43 smithi053 ceph-mon[89963]: pgmap v1333: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:46:44.063 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:44.085 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:44.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:44.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:44.187 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:44.209 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:44.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:44.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:44.311 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:44.333 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:44.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:44.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:44.377 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:46:44.379 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:46:44.435 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:44.457 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:44.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:44.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:44.559 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:44.585 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:44.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:44.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:44.686 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:44.710 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:44.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:44.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:44.812 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:44.834 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:44.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:44.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:44.936 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:44.958 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:44.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:44.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:44.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:44 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1681541588' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:46:45.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:44 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1681541588' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:46:45.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:44 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1681541588' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:46:45.060 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:45.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:45.089 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:45.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:45.190 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:45.213 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:45.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:45.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:45.314 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:45.340 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:45.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:45.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:45.441 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:45.464 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:45.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:45.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:45.565 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:45.587 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:45.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:45.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:45.689 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:45.716 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:45.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:45.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:45.818 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:45.841 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:45.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:45.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:45.943 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:45.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:45 smithi053 ceph-mon[89963]: from='client.46841 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:46:45.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:45 smithi053 ceph-mon[89963]: pgmap v1334: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:46:45.970 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:45.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:45.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:46.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:45 smithi003 ceph-mon[103344]: from='client.46841 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:46:46.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:45 smithi003 ceph-mon[103344]: pgmap v1334: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:46:46.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:45 smithi003 ceph-mon[100615]: from='client.46841 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:46:46.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:45 smithi003 ceph-mon[100615]: pgmap v1334: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:46:46.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:46:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:46:45] "GET /metrics HTTP/1.1" 200 36508 "" "Prometheus/2.51.0" 2024-08-17T17:46:46.051 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:46:46.051 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:46:46.072 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:46.099 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:46.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:46.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:46.201 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:46.223 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:46.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:46.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:46.295 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1333,"stamp":"2024-08-17T17:46:42.441497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790221,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":5978,"num_read_kb":12991,"num_write":907,"num_write_kb":12623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":194,"num_bytes_recovered":8399864,"num_keys_recovered":40,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5651,"ondisk_log_size":5651,"up":309,"acting":309,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":357,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6213228,"kb_used_data":55552,"kb_used_omap":37,"kb_used_meta":6157530,"kb_avail":743747988,"statfs":{"total":767960285184,"available":761597939712,"internally_reserved":0,"allocated":56885248,"data_stored":38949784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38729,"internal_metadata":6305310903},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-237088,"num_write_kb":-7586753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6800,"num_bytes_recovered":-251002880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002012"},"pg_stats":[{"pgid":"1.11","version":"90'149","reported_seq":844,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.118072+0000","last_change":"2024-08-17T17:38:30.378664+0000","last_active":"2024-08-17T17:46:26.118072+0000","last_peered":"2024-08-17T17:46:26.118072+0000","last_clean":"2024-08-17T17:46:26.118072+0000","last_became_active":"2024-08-17T17:38:30.342287+0000","last_became_peered":"2024-08-17T17:38:30.342287+0000","last_unstale":"2024-08-17T17:46:26.118072+0000","last_undegraded":"2024-08-17T17:46:26.118072+0000","last_fullsized":"2024-08-17T17:46:26.118072+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:30.378592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:30.378592+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:04:28.933354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":667,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-08-17T17:46:38.105602+0000","last_change":"2024-08-17T17:33:20.797029+0000","last_active":"2024-08-17T17:46:38.105602+0000","last_peered":"2024-08-17T17:46:38.105602+0000","last_clean":"2024-08-17T17:46:38.105602+0000","last_became_active":"2024-08-17T17:33:20.796679+0000","last_became_peered":"2024-08-17T17:33:20.796679+0000","last_unstale":"2024-08-17T17:46:38.105602+0000","last_undegraded":"2024-08-17T17:46:38.105602+0000","last_fullsized":"2024-08-17T17:46:38.105602+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:51.438262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":421,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:25.594872+0000","last_change":"2024-08-17T17:41:34.624198+0000","last_active":"2024-08-17T17:46:25.594872+0000","last_peered":"2024-08-17T17:46:25.594872+0000","last_clean":"2024-08-17T17:46:25.594872+0000","last_became_active":"2024-08-17T17:41:34.616071+0000","last_became_peered":"2024-08-17T17:41:34.616071+0000","last_unstale":"2024-08-17T17:46:25.594872+0000","last_undegraded":"2024-08-17T17:46:25.594872+0000","last_fullsized":"2024-08-17T17:46:25.594872+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:34:48.805354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"90'149","reported_seq":480,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:25.596235+0000","last_change":"2024-08-17T17:38:31.874623+0000","last_active":"2024-08-17T17:46:25.596235+0000","last_peered":"2024-08-17T17:46:25.596235+0000","last_clean":"2024-08-17T17:46:25.596235+0000","last_became_active":"2024-08-17T17:38:30.339313+0000","last_became_peered":"2024-08-17T17:38:30.339313+0000","last_unstale":"2024-08-17T17:46:25.596235+0000","last_undegraded":"2024-08-17T17:46:25.596235+0000","last_fullsized":"2024-08-17T17:46:25.596235+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:31.874423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:31.874423+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:28:17.980339+0000","scrub_duration":0.0017696579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":328,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-08-17T17:46:38.105430+0000","last_change":"2024-08-17T17:33:44.156036+0000","last_active":"2024-08-17T17:46:38.105430+0000","last_peered":"2024-08-17T17:46:38.105430+0000","last_clean":"2024-08-17T17:46:38.105430+0000","last_became_active":"2024-08-17T17:33:44.155758+0000","last_became_peered":"2024-08-17T17:33:44.155758+0000","last_unstale":"2024-08-17T17:46:38.105430+0000","last_undegraded":"2024-08-17T17:46:38.105430+0000","last_fullsized":"2024-08-17T17:46:38.105430+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:23.443405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":465,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.118337+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:46:26.118337+0000","last_peered":"2024-08-17T17:46:26.118337+0000","last_clean":"2024-08-17T17:46:26.118337+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:46:26.118337+0000","last_undegraded":"2024-08-17T17:46:26.118337+0000","last_fullsized":"2024-08-17T17:46:26.118337+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"90'149","reported_seq":866,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-08-17T17:46:40.276718+0000","last_change":"2024-08-17T17:41:37.681187+0000","last_active":"2024-08-17T17:46:40.276718+0000","last_peered":"2024-08-17T17:46:40.276718+0000","last_clean":"2024-08-17T17:46:40.276718+0000","last_became_active":"2024-08-17T17:41:37.680605+0000","last_became_peered":"2024-08-17T17:41:37.680605+0000","last_unstale":"2024-08-17T17:46:40.276718+0000","last_undegraded":"2024-08-17T17:46:40.276718+0000","last_fullsized":"2024-08-17T17:46:40.276718+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.844252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.844252+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:13:31.682357+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":465,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.119161+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:46:26.119161+0000","last_peered":"2024-08-17T17:46:26.119161+0000","last_clean":"2024-08-17T17:46:26.119161+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:46:26.119161+0000","last_undegraded":"2024-08-17T17:46:26.119161+0000","last_fullsized":"2024-08-17T17:46:26.119161+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":61,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.043702+0000","last_change":"2024-08-17T17:41:34.611693+0000","last_active":"2024-08-17T17:46:26.043702+0000","last_peered":"2024-08-17T17:46:26.043702+0000","last_clean":"2024-08-17T17:46:26.043702+0000","last_became_active":"2024-08-17T17:41:34.611374+0000","last_became_peered":"2024-08-17T17:41:34.611374+0000","last_unstale":"2024-08-17T17:46:26.043702+0000","last_undegraded":"2024-08-17T17:46:26.043702+0000","last_fullsized":"2024-08-17T17:46:26.043702+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:17:09.897446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"90'149","reported_seq":842,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-08-17T17:46:40.277437+0000","last_change":"2024-08-17T17:38:30.446165+0000","last_active":"2024-08-17T17:46:40.277437+0000","last_peered":"2024-08-17T17:46:40.277437+0000","last_clean":"2024-08-17T17:46:40.277437+0000","last_became_active":"2024-08-17T17:38:30.445048+0000","last_became_peered":"2024-08-17T17:38:30.445048+0000","last_unstale":"2024-08-17T17:46:40.277437+0000","last_undegraded":"2024-08-17T17:46:40.277437+0000","last_fullsized":"2024-08-17T17:46:40.277437+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:24.291592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:24.291592+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:17.100154+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":681,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-08-17T17:46:38.105628+0000","last_change":"2024-08-17T17:33:20.794682+0000","last_active":"2024-08-17T17:46:38.105628+0000","last_peered":"2024-08-17T17:46:38.105628+0000","last_clean":"2024-08-17T17:46:38.105628+0000","last_became_active":"2024-08-17T17:33:20.794238+0000","last_became_peered":"2024-08-17T17:33:20.794238+0000","last_unstale":"2024-08-17T17:46:38.105628+0000","last_undegraded":"2024-08-17T17:46:38.105628+0000","last_fullsized":"2024-08-17T17:46:38.105628+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:15:55.821920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":482,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:25.596466+0000","last_change":"2024-08-17T17:43:52.520659+0000","last_active":"2024-08-17T17:46:25.596466+0000","last_peered":"2024-08-17T17:46:25.596466+0000","last_clean":"2024-08-17T17:46:25.596466+0000","last_became_active":"2024-08-17T17:43:52.516799+0000","last_became_peered":"2024-08-17T17:43:52.516799+0000","last_unstale":"2024-08-17T17:46:25.596466+0000","last_undegraded":"2024-08-17T17:46:25.596466+0000","last_fullsized":"2024-08-17T17:46:25.596466+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:25:59.408383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.d","version":"90'149","reported_seq":807,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:25.594818+0000","last_change":"2024-08-17T17:42:16.610940+0000","last_active":"2024-08-17T17:46:25.594818+0000","last_peered":"2024-08-17T17:46:25.594818+0000","last_clean":"2024-08-17T17:46:25.594818+0000","last_became_active":"2024-08-17T17:41:37.670855+0000","last_became_peered":"2024-08-17T17:41:37.670855+0000","last_unstale":"2024-08-17T17:46:25.594818+0000","last_undegraded":"2024-08-17T17:46:25.594818+0000","last_fullsized":"2024-08-17T17:46:25.594818+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:16.610828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:16.610828+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:48:25.918610+0000","scrub_duration":0.086892487000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":358,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:25.595610+0000","last_change":"2024-08-17T17:41:34.635980+0000","last_active":"2024-08-17T17:46:25.595610+0000","last_peered":"2024-08-17T17:46:25.595610+0000","last_clean":"2024-08-17T17:46:25.595610+0000","last_became_active":"2024-08-17T17:41:34.620596+0000","last_became_peered":"2024-08-17T17:41:34.620596+0000","last_unstale":"2024-08-17T17:46:25.595610+0000","last_undegraded":"2024-08-17T17:46:25.595610+0000","last_fullsized":"2024-08-17T17:46:25.595610+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:19:15.792577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":151,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:25.596320+0000","last_change":"2024-08-17T17:33:44.145598+0000","last_active":"2024-08-17T17:46:25.596320+0000","last_peered":"2024-08-17T17:46:25.596320+0000","last_clean":"2024-08-17T17:46:25.596320+0000","last_became_active":"2024-08-17T17:33:44.145283+0000","last_became_peered":"2024-08-17T17:33:44.145283+0000","last_unstale":"2024-08-17T17:46:25.596320+0000","last_undegraded":"2024-08-17T17:46:25.596320+0000","last_fullsized":"2024-08-17T17:46:25.596320+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:11.965226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"90'149","reported_seq":865,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-08-17T17:46:38.105529+0000","last_change":"2024-08-17T17:41:42.603841+0000","last_active":"2024-08-17T17:46:38.105529+0000","last_peered":"2024-08-17T17:46:38.105529+0000","last_clean":"2024-08-17T17:46:38.105529+0000","last_became_active":"2024-08-17T17:41:12.203400+0000","last_became_peered":"2024-08-17T17:41:12.203400+0000","last_unstale":"2024-08-17T17:46:38.105529+0000","last_undegraded":"2024-08-17T17:46:38.105529+0000","last_fullsized":"2024-08-17T17:46:38.105529+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:42.603561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:42.603561+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:50:12.843193+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":354,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:25.596140+0000","last_change":"2024-08-17T17:41:34.637696+0000","last_active":"2024-08-17T17:46:25.596140+0000","last_peered":"2024-08-17T17:46:25.596140+0000","last_clean":"2024-08-17T17:46:25.596140+0000","last_became_active":"2024-08-17T17:41:34.621388+0000","last_became_peered":"2024-08-17T17:41:34.621388+0000","last_unstale":"2024-08-17T17:46:25.596140+0000","last_undegraded":"2024-08-17T17:46:25.596140+0000","last_fullsized":"2024-08-17T17:46:25.596140+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:13:04.607036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":61,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.042942+0000","last_change":"2024-08-17T17:41:34.621078+0000","last_active":"2024-08-17T17:46:26.042942+0000","last_peered":"2024-08-17T17:46:26.042942+0000","last_clean":"2024-08-17T17:46:26.042942+0000","last_became_active":"2024-08-17T17:41:34.620730+0000","last_became_peered":"2024-08-17T17:41:34.620730+0000","last_unstale":"2024-08-17T17:46:26.042942+0000","last_undegraded":"2024-08-17T17:46:26.042942+0000","last_fullsized":"2024-08-17T17:46:26.042942+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:55:01.312283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.f","version":"90'149","reported_seq":573,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:25.596731+0000","last_change":"2024-08-17T17:41:37.668050+0000","last_active":"2024-08-17T17:46:25.596731+0000","last_peered":"2024-08-17T17:46:25.596731+0000","last_clean":"2024-08-17T17:46:25.596731+0000","last_became_active":"2024-08-17T17:41:37.667655+0000","last_became_peered":"2024-08-17T17:41:37.667655+0000","last_unstale":"2024-08-17T17:46:25.596731+0000","last_undegraded":"2024-08-17T17:46:25.596731+0000","last_fullsized":"2024-08-17T17:46:25.596731+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:32.843802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:32.843802+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:39:46.766280+0000","scrub_duration":0.0040803669999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":147,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:25.595084+0000","last_change":"2024-08-17T17:33:44.145399+0000","last_active":"2024-08-17T17:46:25.595084+0000","last_peered":"2024-08-17T17:46:25.595084+0000","last_clean":"2024-08-17T17:46:25.595084+0000","last_became_active":"2024-08-17T17:33:44.144751+0000","last_became_peered":"2024-08-17T17:33:44.144751+0000","last_unstale":"2024-08-17T17:46:25.595084+0000","last_undegraded":"2024-08-17T17:46:25.595084+0000","last_fullsized":"2024-08-17T17:46:25.595084+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:54.276350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":225,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:25.595664+0000","last_change":"2024-08-17T17:33:20.795582+0000","last_active":"2024-08-17T17:46:25.595664+0000","last_peered":"2024-08-17T17:46:25.595664+0000","last_clean":"2024-08-17T17:46:25.595664+0000","last_became_active":"2024-08-17T17:33:20.795149+0000","last_became_peered":"2024-08-17T17:33:20.795149+0000","last_unstale":"2024-08-17T17:46:25.595664+0000","last_undegraded":"2024-08-17T17:46:25.595664+0000","last_fullsized":"2024-08-17T17:46:25.595664+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:14.020923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"90'149","reported_seq":705,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.042573+0000","last_change":"2024-08-17T17:42:20.927007+0000","last_active":"2024-08-17T17:46:26.042573+0000","last_peered":"2024-08-17T17:46:26.042573+0000","last_clean":"2024-08-17T17:46:26.042573+0000","last_became_active":"2024-08-17T17:41:37.670914+0000","last_became_peered":"2024-08-17T17:41:37.670914+0000","last_unstale":"2024-08-17T17:46:26.042573+0000","last_undegraded":"2024-08-17T17:46:26.042573+0000","last_fullsized":"2024-08-17T17:46:26.042573+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:20.926924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:20.926924+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:57:44.156600+0000","scrub_duration":0.0018485109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":325,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-08-17T17:46:38.105717+0000","last_change":"2024-08-17T17:33:44.154097+0000","last_active":"2024-08-17T17:46:38.105717+0000","last_peered":"2024-08-17T17:46:38.105717+0000","last_clean":"2024-08-17T17:46:38.105717+0000","last_became_active":"2024-08-17T17:33:44.152972+0000","last_became_peered":"2024-08-17T17:33:44.152972+0000","last_unstale":"2024-08-17T17:46:38.105717+0000","last_undegraded":"2024-08-17T17:46:38.105717+0000","last_fullsized":"2024-08-17T17:46:38.105717+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:42.602017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":687,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.118941+0000","last_change":"2024-08-17T17:41:34.615197+0000","last_active":"2024-08-17T17:46:26.118941+0000","last_peered":"2024-08-17T17:46:26.118941+0000","last_clean":"2024-08-17T17:46:26.118941+0000","last_became_active":"2024-08-17T17:41:34.614873+0000","last_became_peered":"2024-08-17T17:41:34.614873+0000","last_unstale":"2024-08-17T17:46:26.118941+0000","last_undegraded":"2024-08-17T17:46:26.118941+0000","last_fullsized":"2024-08-17T17:46:26.118941+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:21:57.391953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":840,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-08-17T17:46:38.105074+0000","last_change":"2024-08-17T17:41:44.585386+0000","last_active":"2024-08-17T17:46:38.105074+0000","last_peered":"2024-08-17T17:46:38.105074+0000","last_clean":"2024-08-17T17:46:38.105074+0000","last_became_active":"2024-08-17T17:41:12.307398+0000","last_became_peered":"2024-08-17T17:41:12.307398+0000","last_unstale":"2024-08-17T17:46:38.105074+0000","last_undegraded":"2024-08-17T17:46:38.105074+0000","last_fullsized":"2024-08-17T17:46:38.105074+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:44.583653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:44.583653+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:27:44.756681+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":724,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-08-17T17:46:38.105313+0000","last_change":"2024-08-17T17:41:34.630434+0000","last_active":"2024-08-17T17:46:38.105313+0000","last_peered":"2024-08-17T17:46:38.105313+0000","last_clean":"2024-08-17T17:46:38.105313+0000","last_became_active":"2024-08-17T17:41:34.616406+0000","last_became_peered":"2024-08-17T17:41:34.616406+0000","last_unstale":"2024-08-17T17:46:38.105313+0000","last_undegraded":"2024-08-17T17:46:38.105313+0000","last_fullsized":"2024-08-17T17:46:38.105313+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:30:44.817203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":61,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.043659+0000","last_change":"2024-08-17T17:41:34.611049+0000","last_active":"2024-08-17T17:46:26.043659+0000","last_peered":"2024-08-17T17:46:26.043659+0000","last_clean":"2024-08-17T17:46:26.043659+0000","last_became_active":"2024-08-17T17:41:34.610727+0000","last_became_peered":"2024-08-17T17:41:34.610727+0000","last_unstale":"2024-08-17T17:46:26.043659+0000","last_undegraded":"2024-08-17T17:46:26.043659+0000","last_fullsized":"2024-08-17T17:46:26.043659+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:04:05.477660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":868,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-08-17T17:46:38.105028+0000","last_change":"2024-08-17T17:43:55.595647+0000","last_active":"2024-08-17T17:46:38.105028+0000","last_peered":"2024-08-17T17:46:38.105028+0000","last_clean":"2024-08-17T17:46:38.105028+0000","last_became_active":"2024-08-17T17:43:55.595256+0000","last_became_peered":"2024-08-17T17:43:55.595256+0000","last_unstale":"2024-08-17T17:46:38.105028+0000","last_undegraded":"2024-08-17T17:46:38.105028+0000","last_fullsized":"2024-08-17T17:46:38.105028+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:46.587712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:46.587712+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:05:01.927968+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":61,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.043408+0000","last_change":"2024-08-17T17:41:34.622803+0000","last_active":"2024-08-17T17:46:26.043408+0000","last_peered":"2024-08-17T17:46:26.043408+0000","last_clean":"2024-08-17T17:46:26.043408+0000","last_became_active":"2024-08-17T17:41:34.622525+0000","last_became_peered":"2024-08-17T17:41:34.622525+0000","last_unstale":"2024-08-17T17:46:26.043408+0000","last_undegraded":"2024-08-17T17:46:26.043408+0000","last_fullsized":"2024-08-17T17:46:26.043408+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:42:39.728377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":61,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.043776+0000","last_change":"2024-08-17T17:41:34.619268+0000","last_active":"2024-08-17T17:46:26.043776+0000","last_peered":"2024-08-17T17:46:26.043776+0000","last_clean":"2024-08-17T17:46:26.043776+0000","last_became_active":"2024-08-17T17:41:34.618914+0000","last_became_peered":"2024-08-17T17:41:34.618914+0000","last_unstale":"2024-08-17T17:46:26.043776+0000","last_undegraded":"2024-08-17T17:46:26.043776+0000","last_fullsized":"2024-08-17T17:46:26.043776+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:08:54.248386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"90'149","reported_seq":875,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-08-17T17:46:40.276484+0000","last_change":"2024-08-17T17:41:41.743931+0000","last_active":"2024-08-17T17:46:40.276484+0000","last_peered":"2024-08-17T17:46:40.276484+0000","last_clean":"2024-08-17T17:46:40.276484+0000","last_became_active":"2024-08-17T17:41:12.203771+0000","last_became_peered":"2024-08-17T17:41:12.203771+0000","last_unstale":"2024-08-17T17:46:40.276484+0000","last_undegraded":"2024-08-17T17:46:40.276484+0000","last_fullsized":"2024-08-17T17:46:40.276484+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:41.743570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:41.743570+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:21:03.865738+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":61,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.043212+0000","last_change":"2024-08-17T17:41:34.607602+0000","last_active":"2024-08-17T17:46:26.043212+0000","last_peered":"2024-08-17T17:46:26.043212+0000","last_clean":"2024-08-17T17:46:26.043212+0000","last_became_active":"2024-08-17T17:41:34.607300+0000","last_became_peered":"2024-08-17T17:41:34.607300+0000","last_unstale":"2024-08-17T17:46:26.043212+0000","last_undegraded":"2024-08-17T17:46:26.043212+0000","last_fullsized":"2024-08-17T17:46:26.043212+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:20:31.486698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":61,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.043798+0000","last_change":"2024-08-17T17:41:34.622102+0000","last_active":"2024-08-17T17:46:26.043798+0000","last_peered":"2024-08-17T17:46:26.043798+0000","last_clean":"2024-08-17T17:46:26.043798+0000","last_became_active":"2024-08-17T17:41:34.621733+0000","last_became_peered":"2024-08-17T17:41:34.621733+0000","last_unstale":"2024-08-17T17:46:26.043798+0000","last_undegraded":"2024-08-17T17:46:26.043798+0000","last_fullsized":"2024-08-17T17:46:26.043798+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:39:41.668709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":514,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:25.595292+0000","last_change":"2024-08-17T17:41:41.036615+0000","last_active":"2024-08-17T17:46:25.595292+0000","last_peered":"2024-08-17T17:46:25.595292+0000","last_clean":"2024-08-17T17:46:25.595292+0000","last_became_active":"2024-08-17T17:41:12.329214+0000","last_became_peered":"2024-08-17T17:41:12.329214+0000","last_unstale":"2024-08-17T17:46:25.595292+0000","last_undegraded":"2024-08-17T17:46:25.595292+0000","last_fullsized":"2024-08-17T17:46:25.595292+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:41.036524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:41.036524+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:59:58.695900+0000","scrub_duration":0.001820803,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":683,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-08-17T17:46:38.105310+0000","last_change":"2024-08-17T17:33:20.796216+0000","last_active":"2024-08-17T17:46:38.105310+0000","last_peered":"2024-08-17T17:46:38.105310+0000","last_clean":"2024-08-17T17:46:38.105310+0000","last_became_active":"2024-08-17T17:33:20.795762+0000","last_became_peered":"2024-08-17T17:33:20.795762+0000","last_unstale":"2024-08-17T17:46:38.105310+0000","last_undegraded":"2024-08-17T17:46:38.105310+0000","last_fullsized":"2024-08-17T17:46:38.105310+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:47.038509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":579,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-08-17T17:46:40.277334+0000","last_change":"2024-08-17T17:41:34.625551+0000","last_active":"2024-08-17T17:46:40.277334+0000","last_peered":"2024-08-17T17:46:40.277334+0000","last_clean":"2024-08-17T17:46:40.277334+0000","last_became_active":"2024-08-17T17:41:34.625251+0000","last_became_peered":"2024-08-17T17:41:34.625251+0000","last_unstale":"2024-08-17T17:46:40.277334+0000","last_undegraded":"2024-08-17T17:46:40.277334+0000","last_fullsized":"2024-08-17T17:46:40.277334+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:28:22.354213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":818,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.117894+0000","last_change":"2024-08-17T17:38:25.247159+0000","last_active":"2024-08-17T17:46:26.117894+0000","last_peered":"2024-08-17T17:46:26.117894+0000","last_clean":"2024-08-17T17:46:26.117894+0000","last_became_active":"2024-08-17T17:38:21.311361+0000","last_became_peered":"2024-08-17T17:38:21.311361+0000","last_unstale":"2024-08-17T17:46:26.117894+0000","last_undegraded":"2024-08-17T17:46:26.117894+0000","last_fullsized":"2024-08-17T17:46:26.117894+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:25.247027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:25.247027+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:46:21.734712+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":803,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:25.594758+0000","last_change":"2024-08-17T17:33:44.149928+0000","last_active":"2024-08-17T17:46:25.594758+0000","last_peered":"2024-08-17T17:46:25.594758+0000","last_clean":"2024-08-17T17:46:25.594758+0000","last_became_active":"2024-08-17T17:33:44.149498+0000","last_became_peered":"2024-08-17T17:33:44.149498+0000","last_unstale":"2024-08-17T17:46:25.594758+0000","last_undegraded":"2024-08-17T17:46:25.594758+0000","last_fullsized":"2024-08-17T17:46:25.594758+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:17:04.963202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":630,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-08-17T17:46:38.105122+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:46:38.105122+0000","last_peered":"2024-08-17T17:46:38.105122+0000","last_clean":"2024-08-17T17:46:38.105122+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:46:38.105122+0000","last_undegraded":"2024-08-17T17:46:38.105122+0000","last_fullsized":"2024-08-17T17:46:38.105122+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":238,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:25.594999+0000","last_change":"2024-08-17T17:41:34.618631+0000","last_active":"2024-08-17T17:46:25.594999+0000","last_peered":"2024-08-17T17:46:25.594999+0000","last_clean":"2024-08-17T17:46:25.594999+0000","last_became_active":"2024-08-17T17:41:34.618313+0000","last_became_peered":"2024-08-17T17:41:34.618313+0000","last_unstale":"2024-08-17T17:46:25.594999+0000","last_undegraded":"2024-08-17T17:46:25.594999+0000","last_fullsized":"2024-08-17T17:46:25.594999+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:07:13.915088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":858,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-08-17T17:46:40.277256+0000","last_change":"2024-08-17T17:41:37.676347+0000","last_active":"2024-08-17T17:46:40.277256+0000","last_peered":"2024-08-17T17:46:40.277256+0000","last_clean":"2024-08-17T17:46:40.277256+0000","last_became_active":"2024-08-17T17:41:37.675869+0000","last_became_peered":"2024-08-17T17:41:37.675869+0000","last_unstale":"2024-08-17T17:46:40.277256+0000","last_undegraded":"2024-08-17T17:46:40.277256+0000","last_fullsized":"2024-08-17T17:46:40.277256+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:21.814077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:21.814077+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:04:16.240754+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":916,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-08-17T17:46:40.276519+0000","last_change":"2024-08-17T17:33:20.797705+0000","last_active":"2024-08-17T17:46:40.276519+0000","last_peered":"2024-08-17T17:46:40.276519+0000","last_clean":"2024-08-17T17:46:40.276519+0000","last_became_active":"2024-08-17T17:33:20.797172+0000","last_became_peered":"2024-08-17T17:33:20.797172+0000","last_unstale":"2024-08-17T17:46:40.276519+0000","last_undegraded":"2024-08-17T17:46:40.276519+0000","last_fullsized":"2024-08-17T17:46:40.276519+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:30.754999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":678,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.117967+0000","last_change":"2024-08-17T17:41:34.614108+0000","last_active":"2024-08-17T17:46:26.117967+0000","last_peered":"2024-08-17T17:46:26.117967+0000","last_clean":"2024-08-17T17:46:26.117967+0000","last_became_active":"2024-08-17T17:41:34.613481+0000","last_became_peered":"2024-08-17T17:41:34.613481+0000","last_unstale":"2024-08-17T17:46:26.117967+0000","last_undegraded":"2024-08-17T17:46:26.117967+0000","last_fullsized":"2024-08-17T17:46:26.117967+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:21:02.346115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":466,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.119250+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:46:26.119250+0000","last_peered":"2024-08-17T17:46:26.119250+0000","last_clean":"2024-08-17T17:46:26.119250+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:46:26.119250+0000","last_undegraded":"2024-08-17T17:46:26.119250+0000","last_fullsized":"2024-08-17T17:46:26.119250+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":857,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.117612+0000","last_change":"2024-08-17T17:41:37.683450+0000","last_active":"2024-08-17T17:46:26.117612+0000","last_peered":"2024-08-17T17:46:26.117612+0000","last_clean":"2024-08-17T17:46:26.117612+0000","last_became_active":"2024-08-17T17:41:37.683103+0000","last_became_peered":"2024-08-17T17:41:37.683103+0000","last_unstale":"2024-08-17T17:46:26.117612+0000","last_undegraded":"2024-08-17T17:46:26.117612+0000","last_fullsized":"2024-08-17T17:46:26.117612+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.289528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.289528+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:39:17.141469+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":1047,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-08-17T17:46:38.105377+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:46:38.105377+0000","last_peered":"2024-08-17T17:46:38.105377+0000","last_clean":"2024-08-17T17:46:38.105377+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:46:38.105377+0000","last_undegraded":"2024-08-17T17:46:38.105377+0000","last_fullsized":"2024-08-17T17:46:38.105377+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":554,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-08-17T17:46:40.276633+0000","last_change":"2024-08-17T17:33:20.793468+0000","last_active":"2024-08-17T17:46:40.276633+0000","last_peered":"2024-08-17T17:46:40.276633+0000","last_clean":"2024-08-17T17:46:40.276633+0000","last_became_active":"2024-08-17T17:33:20.793166+0000","last_became_peered":"2024-08-17T17:33:20.793166+0000","last_unstale":"2024-08-17T17:46:40.276633+0000","last_undegraded":"2024-08-17T17:46:40.276633+0000","last_fullsized":"2024-08-17T17:46:40.276633+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:12:52.121050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":590,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-08-17T17:46:40.277235+0000","last_change":"2024-08-17T17:33:20.835784+0000","last_active":"2024-08-17T17:46:40.277235+0000","last_peered":"2024-08-17T17:46:40.277235+0000","last_clean":"2024-08-17T17:46:40.277235+0000","last_became_active":"2024-08-17T17:33:20.835371+0000","last_became_peered":"2024-08-17T17:33:20.835371+0000","last_unstale":"2024-08-17T17:46:40.277235+0000","last_undegraded":"2024-08-17T17:46:40.277235+0000","last_fullsized":"2024-08-17T17:46:40.277235+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:35:43.369055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":836,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.117863+0000","last_change":"2024-08-17T17:38:22.254841+0000","last_active":"2024-08-17T17:46:26.117863+0000","last_peered":"2024-08-17T17:46:26.117863+0000","last_clean":"2024-08-17T17:46:26.117863+0000","last_became_active":"2024-08-17T17:38:21.306619+0000","last_became_peered":"2024-08-17T17:38:21.306619+0000","last_unstale":"2024-08-17T17:46:26.117863+0000","last_undegraded":"2024-08-17T17:46:26.117863+0000","last_fullsized":"2024-08-17T17:46:26.117863+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:22.254705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:22.254705+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:47:38.184089+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":1139,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-08-17T17:46:40.276672+0000","last_change":"2024-08-17T17:33:44.148227+0000","last_active":"2024-08-17T17:46:40.276672+0000","last_peered":"2024-08-17T17:46:40.276672+0000","last_clean":"2024-08-17T17:46:40.276672+0000","last_became_active":"2024-08-17T17:33:44.147627+0000","last_became_peered":"2024-08-17T17:33:44.147627+0000","last_unstale":"2024-08-17T17:46:40.276672+0000","last_undegraded":"2024-08-17T17:46:40.276672+0000","last_fullsized":"2024-08-17T17:46:40.276672+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:29:42.898460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1372,"num_read_kb":2843,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":61,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.043649+0000","last_change":"2024-08-17T17:41:34.611109+0000","last_active":"2024-08-17T17:46:26.043649+0000","last_peered":"2024-08-17T17:46:26.043649+0000","last_clean":"2024-08-17T17:46:26.043649+0000","last_became_active":"2024-08-17T17:41:34.610170+0000","last_became_peered":"2024-08-17T17:41:34.610170+0000","last_unstale":"2024-08-17T17:46:26.043649+0000","last_undegraded":"2024-08-17T17:46:26.043649+0000","last_fullsized":"2024-08-17T17:46:26.043649+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:39:05.301260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":594,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-08-17T17:46:40.277241+0000","last_change":"2024-08-17T17:33:20.772454+0000","last_active":"2024-08-17T17:46:40.277241+0000","last_peered":"2024-08-17T17:46:40.277241+0000","last_clean":"2024-08-17T17:46:40.277241+0000","last_became_active":"2024-08-17T17:33:20.771848+0000","last_became_peered":"2024-08-17T17:33:20.771848+0000","last_unstale":"2024-08-17T17:46:40.277241+0000","last_undegraded":"2024-08-17T17:46:40.277241+0000","last_fullsized":"2024-08-17T17:46:40.277241+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:46:42.300744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":534,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:25.596633+0000","last_change":"2024-08-17T17:41:43.035215+0000","last_active":"2024-08-17T17:46:25.596633+0000","last_peered":"2024-08-17T17:46:25.596633+0000","last_clean":"2024-08-17T17:46:25.596633+0000","last_became_active":"2024-08-17T17:41:12.201158+0000","last_became_peered":"2024-08-17T17:41:12.201158+0000","last_unstale":"2024-08-17T17:46:25.596633+0000","last_undegraded":"2024-08-17T17:46:25.596633+0000","last_fullsized":"2024-08-17T17:46:25.596633+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:43.035108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:43.035108+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:43:27.108994+0000","scrub_duration":0.0028877400000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":343,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-08-17T17:46:38.105603+0000","last_change":"2024-08-17T17:33:44.163867+0000","last_active":"2024-08-17T17:46:38.105603+0000","last_peered":"2024-08-17T17:46:38.105603+0000","last_clean":"2024-08-17T17:46:38.105603+0000","last_became_active":"2024-08-17T17:33:44.163503+0000","last_became_peered":"2024-08-17T17:33:44.163503+0000","last_unstale":"2024-08-17T17:46:38.105603+0000","last_undegraded":"2024-08-17T17:46:38.105603+0000","last_fullsized":"2024-08-17T17:46:38.105603+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:31:53.573596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":809,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-08-17T17:46:40.276840+0000","last_change":"2024-08-17T17:33:20.799350+0000","last_active":"2024-08-17T17:46:40.276840+0000","last_peered":"2024-08-17T17:46:40.276840+0000","last_clean":"2024-08-17T17:46:40.276840+0000","last_became_active":"2024-08-17T17:33:20.798925+0000","last_became_peered":"2024-08-17T17:33:20.798925+0000","last_unstale":"2024-08-17T17:46:40.276840+0000","last_undegraded":"2024-08-17T17:46:40.276840+0000","last_fullsized":"2024-08-17T17:46:40.276840+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:16:36.121295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":625,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-08-17T17:46:40.277507+0000","last_change":"2024-08-17T17:41:34.627064+0000","last_active":"2024-08-17T17:46:40.277507+0000","last_peered":"2024-08-17T17:46:40.277507+0000","last_clean":"2024-08-17T17:46:40.277507+0000","last_became_active":"2024-08-17T17:41:34.626453+0000","last_became_peered":"2024-08-17T17:41:34.626453+0000","last_unstale":"2024-08-17T17:46:40.277507+0000","last_undegraded":"2024-08-17T17:46:40.277507+0000","last_fullsized":"2024-08-17T17:46:40.277507+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:02:11.894042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":534,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:25.595489+0000","last_change":"2024-08-17T17:41:44.058291+0000","last_active":"2024-08-17T17:46:25.595489+0000","last_peered":"2024-08-17T17:46:25.595489+0000","last_clean":"2024-08-17T17:46:25.595489+0000","last_became_active":"2024-08-17T17:41:12.200077+0000","last_became_peered":"2024-08-17T17:41:12.200077+0000","last_unstale":"2024-08-17T17:46:25.595489+0000","last_undegraded":"2024-08-17T17:46:25.595489+0000","last_fullsized":"2024-08-17T17:46:25.595489+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:44.058233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:44.058233+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:42:55.911623+0000","scrub_duration":0.0074268499999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":629,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-08-17T17:46:40.276760+0000","last_change":"2024-08-17T17:33:20.798613+0000","last_active":"2024-08-17T17:46:40.276760+0000","last_peered":"2024-08-17T17:46:40.276760+0000","last_clean":"2024-08-17T17:46:40.276760+0000","last_became_active":"2024-08-17T17:33:20.798260+0000","last_became_peered":"2024-08-17T17:33:20.798260+0000","last_unstale":"2024-08-17T17:46:40.276760+0000","last_undegraded":"2024-08-17T17:46:40.276760+0000","last_fullsized":"2024-08-17T17:46:40.276760+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:18:00.822681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":957,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:25.595998+0000","last_change":"2024-08-17T17:41:37.683212+0000","last_active":"2024-08-17T17:46:25.595998+0000","last_peered":"2024-08-17T17:46:25.595998+0000","last_clean":"2024-08-17T17:46:25.595998+0000","last_became_active":"2024-08-17T17:41:37.682730+0000","last_became_peered":"2024-08-17T17:41:37.682730+0000","last_unstale":"2024-08-17T17:46:25.595998+0000","last_undegraded":"2024-08-17T17:46:25.595998+0000","last_fullsized":"2024-08-17T17:46:25.595998+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:51:35.444433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":640,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.118858+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:46:26.118858+0000","last_peered":"2024-08-17T17:46:26.118858+0000","last_clean":"2024-08-17T17:46:26.118858+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:46:26.118858+0000","last_undegraded":"2024-08-17T17:46:26.118858+0000","last_fullsized":"2024-08-17T17:46:26.118858+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":780,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.042960+0000","last_change":"2024-08-17T17:42:19.960161+0000","last_active":"2024-08-17T17:46:26.042960+0000","last_peered":"2024-08-17T17:46:26.042960+0000","last_clean":"2024-08-17T17:46:26.042960+0000","last_became_active":"2024-08-17T17:41:37.665223+0000","last_became_peered":"2024-08-17T17:41:37.665223+0000","last_unstale":"2024-08-17T17:46:26.042960+0000","last_undegraded":"2024-08-17T17:46:26.042960+0000","last_fullsized":"2024-08-17T17:46:26.042960+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:19.960086+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:19.960086+0000","last_clean_scrub_stamp":"2024-08-17T17:42:19.960086+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:57.861661+0000","scrub_duration":0.0022612000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":677,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-08-17T17:46:38.104947+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:46:38.104947+0000","last_peered":"2024-08-17T17:46:38.104947+0000","last_clean":"2024-08-17T17:46:38.104947+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:46:38.104947+0000","last_undegraded":"2024-08-17T17:46:38.104947+0000","last_fullsized":"2024-08-17T17:46:38.104947+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":1254,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-08-17T17:46:40.277340+0000","last_change":"2024-08-17T17:41:37.677524+0000","last_active":"2024-08-17T17:46:40.277340+0000","last_peered":"2024-08-17T17:46:40.277340+0000","last_clean":"2024-08-17T17:46:40.277340+0000","last_became_active":"2024-08-17T17:41:37.676764+0000","last_became_peered":"2024-08-17T17:41:37.676764+0000","last_unstale":"2024-08-17T17:46:40.277340+0000","last_undegraded":"2024-08-17T17:46:40.277340+0000","last_fullsized":"2024-08-17T17:46:40.277340+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:14:49.341080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":674,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-08-17T17:46:38.105200+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:46:38.105200+0000","last_peered":"2024-08-17T17:46:38.105200+0000","last_clean":"2024-08-17T17:46:38.105200+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:46:38.105200+0000","last_undegraded":"2024-08-17T17:46:38.105200+0000","last_fullsized":"2024-08-17T17:46:38.105200+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":821,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-08-17T17:46:38.105404+0000","last_change":"2024-08-17T17:38:23.113397+0000","last_active":"2024-08-17T17:46:38.105404+0000","last_peered":"2024-08-17T17:46:38.105404+0000","last_clean":"2024-08-17T17:46:38.105404+0000","last_became_active":"2024-08-17T17:38:21.305580+0000","last_became_peered":"2024-08-17T17:38:21.305580+0000","last_unstale":"2024-08-17T17:46:38.105404+0000","last_undegraded":"2024-08-17T17:46:38.105404+0000","last_fullsized":"2024-08-17T17:46:38.105404+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.113279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.113279+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:38:07.015721+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":512,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.119222+0000","last_change":"2024-08-17T17:41:34.629522+0000","last_active":"2024-08-17T17:46:26.119222+0000","last_peered":"2024-08-17T17:46:26.119222+0000","last_clean":"2024-08-17T17:46:26.119222+0000","last_became_active":"2024-08-17T17:41:34.619340+0000","last_became_peered":"2024-08-17T17:41:34.619340+0000","last_unstale":"2024-08-17T17:46:26.119222+0000","last_undegraded":"2024-08-17T17:46:26.119222+0000","last_fullsized":"2024-08-17T17:46:26.119222+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:30:03.902956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":627,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:25.596826+0000","last_change":"2024-08-17T17:26:06.855307+0000","last_active":"2024-08-17T17:46:25.596826+0000","last_peered":"2024-08-17T17:46:25.596826+0000","last_clean":"2024-08-17T17:46:25.596826+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:46:25.596826+0000","last_undegraded":"2024-08-17T17:46:25.596826+0000","last_fullsized":"2024-08-17T17:46:25.596826+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":61,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.043041+0000","last_change":"2024-08-17T17:41:34.614028+0000","last_active":"2024-08-17T17:46:26.043041+0000","last_peered":"2024-08-17T17:46:26.043041+0000","last_clean":"2024-08-17T17:46:26.043041+0000","last_became_active":"2024-08-17T17:41:34.613706+0000","last_became_peered":"2024-08-17T17:41:34.613706+0000","last_unstale":"2024-08-17T17:46:26.043041+0000","last_undegraded":"2024-08-17T17:46:26.043041+0000","last_fullsized":"2024-08-17T17:46:26.043041+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:19:55.254902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"90'149","reported_seq":733,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-08-17T17:46:40.277092+0000","last_change":"2024-08-17T17:42:07.681204+0000","last_active":"2024-08-17T17:46:40.277092+0000","last_peered":"2024-08-17T17:46:40.277092+0000","last_clean":"2024-08-17T17:46:40.277092+0000","last_became_active":"2024-08-17T17:42:05.878769+0000","last_became_peered":"2024-08-17T17:42:05.878769+0000","last_unstale":"2024-08-17T17:46:40.277092+0000","last_undegraded":"2024-08-17T17:46:40.277092+0000","last_fullsized":"2024-08-17T17:46:40.277092+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:07.681064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:07.681064+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:12:12.835715+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":61,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.043503+0000","last_change":"2024-08-17T17:41:34.622180+0000","last_active":"2024-08-17T17:46:26.043503+0000","last_peered":"2024-08-17T17:46:26.043503+0000","last_clean":"2024-08-17T17:46:26.043503+0000","last_became_active":"2024-08-17T17:41:34.621724+0000","last_became_peered":"2024-08-17T17:41:34.621724+0000","last_unstale":"2024-08-17T17:46:26.043503+0000","last_undegraded":"2024-08-17T17:46:26.043503+0000","last_fullsized":"2024-08-17T17:46:26.043503+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:15.807235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":71,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.043836+0000","last_change":"2024-08-17T17:41:34.626584+0000","last_active":"2024-08-17T17:46:26.043836+0000","last_peered":"2024-08-17T17:46:26.043836+0000","last_clean":"2024-08-17T17:46:26.043836+0000","last_became_active":"2024-08-17T17:41:34.609878+0000","last_became_peered":"2024-08-17T17:41:34.609878+0000","last_unstale":"2024-08-17T17:46:26.043836+0000","last_undegraded":"2024-08-17T17:46:26.043836+0000","last_fullsized":"2024-08-17T17:46:26.043836+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:08:12.810347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.17","version":"90'149","reported_seq":488,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.118603+0000","last_change":"2024-08-17T17:42:06.699082+0000","last_active":"2024-08-17T17:46:26.118603+0000","last_peered":"2024-08-17T17:46:26.118603+0000","last_clean":"2024-08-17T17:46:26.118603+0000","last_became_active":"2024-08-17T17:42:05.876218+0000","last_became_peered":"2024-08-17T17:42:05.876218+0000","last_unstale":"2024-08-17T17:46:26.118603+0000","last_undegraded":"2024-08-17T17:46:26.118603+0000","last_fullsized":"2024-08-17T17:46:26.118603+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.698998+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:06.698998+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.698998+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:22:16.510150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":621,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-08-17T17:46:40.276800+0000","last_change":"2024-08-17T17:33:20.799402+0000","last_active":"2024-08-17T17:46:40.276800+0000","last_peered":"2024-08-17T17:46:40.276800+0000","last_clean":"2024-08-17T17:46:40.276800+0000","last_became_active":"2024-08-17T17:33:20.798832+0000","last_became_peered":"2024-08-17T17:33:20.798832+0000","last_unstale":"2024-08-17T17:46:40.276800+0000","last_undegraded":"2024-08-17T17:46:40.276800+0000","last_fullsized":"2024-08-17T17:46:40.276800+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:21:05.001134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":394,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-08-17T17:46:38.105626+0000","last_change":"2024-08-17T17:41:34.634301+0000","last_active":"2024-08-17T17:46:38.105626+0000","last_peered":"2024-08-17T17:46:38.105626+0000","last_clean":"2024-08-17T17:46:38.105626+0000","last_became_active":"2024-08-17T17:41:34.620508+0000","last_became_peered":"2024-08-17T17:41:34.620508+0000","last_unstale":"2024-08-17T17:46:38.105626+0000","last_undegraded":"2024-08-17T17:46:38.105626+0000","last_fullsized":"2024-08-17T17:46:38.105626+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:10:54.576812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.14","version":"90'149","reported_seq":846,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-08-17T17:46:39.994355+0000","last_change":"2024-08-17T17:41:37.664471+0000","last_active":"2024-08-17T17:46:39.994355+0000","last_peered":"2024-08-17T17:46:39.994355+0000","last_clean":"2024-08-17T17:46:39.994355+0000","last_became_active":"2024-08-17T17:41:37.664108+0000","last_became_peered":"2024-08-17T17:41:37.664108+0000","last_unstale":"2024-08-17T17:46:39.994355+0000","last_undegraded":"2024-08-17T17:46:39.994355+0000","last_fullsized":"2024-08-17T17:46:39.994355+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:22.030514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:22.030514+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:15:52.325163+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":360,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-08-17T17:46:40.277248+0000","last_change":"2024-08-17T17:33:44.146014+0000","last_active":"2024-08-17T17:46:40.277248+0000","last_peered":"2024-08-17T17:46:40.277248+0000","last_clean":"2024-08-17T17:46:40.277248+0000","last_became_active":"2024-08-17T17:33:44.145668+0000","last_became_peered":"2024-08-17T17:33:44.145668+0000","last_unstale":"2024-08-17T17:46:40.277248+0000","last_undegraded":"2024-08-17T17:46:40.277248+0000","last_fullsized":"2024-08-17T17:46:40.277248+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:53:41.679237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":623,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-08-17T17:46:38.104925+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:46:38.104925+0000","last_peered":"2024-08-17T17:46:38.104925+0000","last_clean":"2024-08-17T17:46:38.104925+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:46:38.104925+0000","last_undegraded":"2024-08-17T17:46:38.104925+0000","last_fullsized":"2024-08-17T17:46:38.104925+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"90'149","reported_seq":783,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:25.595893+0000","last_change":"2024-08-17T17:42:06.456562+0000","last_active":"2024-08-17T17:46:25.595893+0000","last_peered":"2024-08-17T17:46:25.595893+0000","last_clean":"2024-08-17T17:46:25.595893+0000","last_became_active":"2024-08-17T17:42:05.876577+0000","last_became_peered":"2024-08-17T17:42:05.876577+0000","last_unstale":"2024-08-17T17:46:25.595893+0000","last_undegraded":"2024-08-17T17:46:25.595893+0000","last_fullsized":"2024-08-17T17:46:25.595893+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.456490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.456490+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:10:03.654953+0000","scrub_duration":0.001537285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":648,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.118927+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:46:26.118927+0000","last_peered":"2024-08-17T17:46:26.118927+0000","last_clean":"2024-08-17T17:46:26.118927+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:46:26.118927+0000","last_undegraded":"2024-08-17T17:46:26.118927+0000","last_fullsized":"2024-08-17T17:46:26.118927+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":681,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-08-17T17:46:38.105033+0000","last_change":"2024-08-17T17:33:20.790687+0000","last_active":"2024-08-17T17:46:38.105033+0000","last_peered":"2024-08-17T17:46:38.105033+0000","last_clean":"2024-08-17T17:46:38.105033+0000","last_became_active":"2024-08-17T17:33:20.790399+0000","last_became_peered":"2024-08-17T17:33:20.790399+0000","last_unstale":"2024-08-17T17:46:38.105033+0000","last_undegraded":"2024-08-17T17:46:38.105033+0000","last_fullsized":"2024-08-17T17:46:38.105033+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:38:06.849989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1a","version":"90'149","reported_seq":445,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.042782+0000","last_change":"2024-08-17T17:42:21.932830+0000","last_active":"2024-08-17T17:46:26.042782+0000","last_peered":"2024-08-17T17:46:26.042782+0000","last_clean":"2024-08-17T17:46:26.042782+0000","last_became_active":"2024-08-17T17:42:05.876835+0000","last_became_peered":"2024-08-17T17:42:05.876835+0000","last_unstale":"2024-08-17T17:46:26.042782+0000","last_undegraded":"2024-08-17T17:46:26.042782+0000","last_fullsized":"2024-08-17T17:46:26.042782+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:21.932757+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:21.932757+0000","last_clean_scrub_stamp":"2024-08-17T17:42:21.932757+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:36:39.848570+0000","scrub_duration":0.0020218810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":370,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:25.596728+0000","last_change":"2024-08-17T17:33:20.835760+0000","last_active":"2024-08-17T17:46:25.596728+0000","last_peered":"2024-08-17T17:46:25.596728+0000","last_clean":"2024-08-17T17:46:25.596728+0000","last_became_active":"2024-08-17T17:33:20.835410+0000","last_became_peered":"2024-08-17T17:33:20.835410+0000","last_unstale":"2024-08-17T17:46:25.596728+0000","last_undegraded":"2024-08-17T17:46:25.596728+0000","last_fullsized":"2024-08-17T17:46:25.596728+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:45.448022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":699,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-08-17T17:46:38.105222+0000","last_change":"2024-08-17T17:33:20.771279+0000","last_active":"2024-08-17T17:46:38.105222+0000","last_peered":"2024-08-17T17:46:38.105222+0000","last_clean":"2024-08-17T17:46:38.105222+0000","last_became_active":"2024-08-17T17:33:20.770728+0000","last_became_peered":"2024-08-17T17:33:20.770728+0000","last_unstale":"2024-08-17T17:46:38.105222+0000","last_undegraded":"2024-08-17T17:46:38.105222+0000","last_fullsized":"2024-08-17T17:46:38.105222+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:52:31.540397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"90'149","reported_seq":740,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.043354+0000","last_change":"2024-08-17T17:42:26.967686+0000","last_active":"2024-08-17T17:46:26.043354+0000","last_peered":"2024-08-17T17:46:26.043354+0000","last_clean":"2024-08-17T17:46:26.043354+0000","last_became_active":"2024-08-17T17:42:05.873657+0000","last_became_peered":"2024-08-17T17:42:05.873657+0000","last_unstale":"2024-08-17T17:46:26.043354+0000","last_undegraded":"2024-08-17T17:46:26.043354+0000","last_fullsized":"2024-08-17T17:46:26.043354+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:26.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:26.967611+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:31:03.344332+0000","scrub_duration":0.0015978279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":467,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.119112+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:46:26.119112+0000","last_peered":"2024-08-17T17:46:26.119112+0000","last_clean":"2024-08-17T17:46:26.119112+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:46:26.119112+0000","last_undegraded":"2024-08-17T17:46:26.119112+0000","last_fullsized":"2024-08-17T17:46:26.119112+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":620,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-08-17T17:46:40.276589+0000","last_change":"2024-08-17T17:41:34.625011+0000","last_active":"2024-08-17T17:46:40.276589+0000","last_peered":"2024-08-17T17:46:40.276589+0000","last_clean":"2024-08-17T17:46:40.276589+0000","last_became_active":"2024-08-17T17:41:34.624631+0000","last_became_peered":"2024-08-17T17:41:34.624631+0000","last_unstale":"2024-08-17T17:46:40.276589+0000","last_undegraded":"2024-08-17T17:46:40.276589+0000","last_fullsized":"2024-08-17T17:46:40.276589+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:03:05.565435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"90'149","reported_seq":703,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:25.597632+0000","last_change":"2024-08-17T17:42:08.482975+0000","last_active":"2024-08-17T17:46:25.597632+0000","last_peered":"2024-08-17T17:46:25.597632+0000","last_clean":"2024-08-17T17:46:25.597632+0000","last_became_active":"2024-08-17T17:42:05.873523+0000","last_became_peered":"2024-08-17T17:42:05.873523+0000","last_unstale":"2024-08-17T17:46:25.597632+0000","last_undegraded":"2024-08-17T17:46:25.597632+0000","last_fullsized":"2024-08-17T17:46:25.597632+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:08.482892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:08.482892+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:41.477341+0000","scrub_duration":0.0017925479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":61,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.043815+0000","last_change":"2024-08-17T17:41:34.616482+0000","last_active":"2024-08-17T17:46:26.043815+0000","last_peered":"2024-08-17T17:46:26.043815+0000","last_clean":"2024-08-17T17:46:26.043815+0000","last_became_active":"2024-08-17T17:41:34.616040+0000","last_became_peered":"2024-08-17T17:41:34.616040+0000","last_unstale":"2024-08-17T17:46:26.043815+0000","last_undegraded":"2024-08-17T17:46:26.043815+0000","last_fullsized":"2024-08-17T17:46:26.043815+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:53:08.502264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":61,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.043295+0000","last_change":"2024-08-17T17:41:34.622184+0000","last_active":"2024-08-17T17:46:26.043295+0000","last_peered":"2024-08-17T17:46:26.043295+0000","last_clean":"2024-08-17T17:46:26.043295+0000","last_became_active":"2024-08-17T17:41:34.621831+0000","last_became_peered":"2024-08-17T17:41:34.621831+0000","last_unstale":"2024-08-17T17:46:26.043295+0000","last_undegraded":"2024-08-17T17:46:26.043295+0000","last_fullsized":"2024-08-17T17:46:26.043295+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:59.826667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"90'149","reported_seq":848,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-08-17T17:46:38.105030+0000","last_change":"2024-08-17T17:42:05.876463+0000","last_active":"2024-08-17T17:46:38.105030+0000","last_peered":"2024-08-17T17:46:38.105030+0000","last_clean":"2024-08-17T17:46:38.105030+0000","last_became_active":"2024-08-17T17:42:05.876069+0000","last_became_peered":"2024-08-17T17:42:05.876069+0000","last_unstale":"2024-08-17T17:46:38.105030+0000","last_undegraded":"2024-08-17T17:46:38.105030+0000","last_fullsized":"2024-08-17T17:46:38.105030+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:45.552985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:45.552985+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:55:06.027278+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":587,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-08-17T17:46:40.277179+0000","last_change":"2024-08-17T17:33:20.773207+0000","last_active":"2024-08-17T17:46:40.277179+0000","last_peered":"2024-08-17T17:46:40.277179+0000","last_clean":"2024-08-17T17:46:40.277179+0000","last_became_active":"2024-08-17T17:33:20.772457+0000","last_became_peered":"2024-08-17T17:33:20.772457+0000","last_unstale":"2024-08-17T17:46:40.277179+0000","last_undegraded":"2024-08-17T17:46:40.277179+0000","last_fullsized":"2024-08-17T17:46:40.277179+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:57:09.610084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":490,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.118632+0000","last_change":"2024-08-17T17:41:34.618968+0000","last_active":"2024-08-17T17:46:26.118632+0000","last_peered":"2024-08-17T17:46:26.118632+0000","last_clean":"2024-08-17T17:46:26.118632+0000","last_became_active":"2024-08-17T17:41:34.618576+0000","last_became_peered":"2024-08-17T17:41:34.618576+0000","last_unstale":"2024-08-17T17:46:26.118632+0000","last_undegraded":"2024-08-17T17:46:26.118632+0000","last_fullsized":"2024-08-17T17:46:26.118632+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:47:45.888590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1e","version":"90'149","reported_seq":496,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-08-17T17:46:38.104957+0000","last_change":"2024-08-17T17:42:06.742801+0000","last_active":"2024-08-17T17:46:38.104957+0000","last_peered":"2024-08-17T17:46:38.104957+0000","last_clean":"2024-08-17T17:46:38.104957+0000","last_became_active":"2024-08-17T17:42:05.876895+0000","last_became_peered":"2024-08-17T17:42:05.876895+0000","last_unstale":"2024-08-17T17:46:38.104957+0000","last_undegraded":"2024-08-17T17:46:38.104957+0000","last_fullsized":"2024-08-17T17:46:38.104957+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.742723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.742723+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:36:46.651953+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":61,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.043224+0000","last_change":"2024-08-17T17:41:34.622094+0000","last_active":"2024-08-17T17:46:26.043224+0000","last_peered":"2024-08-17T17:46:26.043224+0000","last_clean":"2024-08-17T17:46:26.043224+0000","last_became_active":"2024-08-17T17:41:34.621773+0000","last_became_peered":"2024-08-17T17:41:34.621773+0000","last_unstale":"2024-08-17T17:46:26.043224+0000","last_undegraded":"2024-08-17T17:46:26.043224+0000","last_fullsized":"2024-08-17T17:46:26.043224+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:34:30.540886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":65,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.043839+0000","last_change":"2024-08-17T17:41:34.629135+0000","last_active":"2024-08-17T17:46:26.043839+0000","last_peered":"2024-08-17T17:46:26.043839+0000","last_clean":"2024-08-17T17:46:26.043839+0000","last_became_active":"2024-08-17T17:41:34.613095+0000","last_became_peered":"2024-08-17T17:41:34.613095+0000","last_unstale":"2024-08-17T17:46:26.043839+0000","last_undegraded":"2024-08-17T17:46:26.043839+0000","last_fullsized":"2024-08-17T17:46:26.043839+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:27:06.542416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":61,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.043505+0000","last_change":"2024-08-17T17:41:34.616251+0000","last_active":"2024-08-17T17:46:26.043505+0000","last_peered":"2024-08-17T17:46:26.043505+0000","last_clean":"2024-08-17T17:46:26.043505+0000","last_became_active":"2024-08-17T17:41:34.615952+0000","last_became_peered":"2024-08-17T17:41:34.615952+0000","last_unstale":"2024-08-17T17:46:26.043505+0000","last_undegraded":"2024-08-17T17:46:26.043505+0000","last_fullsized":"2024-08-17T17:46:26.043505+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:12:24.892508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":65,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.043591+0000","last_change":"2024-08-17T17:41:34.631152+0000","last_active":"2024-08-17T17:46:26.043591+0000","last_peered":"2024-08-17T17:46:26.043591+0000","last_clean":"2024-08-17T17:46:26.043591+0000","last_became_active":"2024-08-17T17:41:34.615617+0000","last_became_peered":"2024-08-17T17:41:34.615617+0000","last_unstale":"2024-08-17T17:46:26.043591+0000","last_undegraded":"2024-08-17T17:46:26.043591+0000","last_fullsized":"2024-08-17T17:46:26.043591+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:43:08.257084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"90'149","reported_seq":737,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:25.596079+0000","last_change":"2024-08-17T17:42:11.809047+0000","last_active":"2024-08-17T17:46:25.596079+0000","last_peered":"2024-08-17T17:46:25.596079+0000","last_clean":"2024-08-17T17:46:25.596079+0000","last_became_active":"2024-08-17T17:42:05.883770+0000","last_became_peered":"2024-08-17T17:42:05.883770+0000","last_unstale":"2024-08-17T17:46:25.596079+0000","last_undegraded":"2024-08-17T17:46:25.596079+0000","last_fullsized":"2024-08-17T17:46:25.596079+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:11.808971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:11.808971+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:48:15.989437+0000","scrub_duration":0.0018167420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":677,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-08-17T17:46:38.105509+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:46:38.105509+0000","last_peered":"2024-08-17T17:46:38.105509+0000","last_clean":"2024-08-17T17:46:38.105509+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:46:38.105509+0000","last_undegraded":"2024-08-17T17:46:38.105509+0000","last_fullsized":"2024-08-17T17:46:38.105509+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":678,"reported_epoch":397,"state":"active+clean","last_fresh":"2024-08-17T17:46:38.105617+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:46:38.105617+0000","last_peered":"2024-08-17T17:46:38.105617+0000","last_clean":"2024-08-17T17:46:38.105617+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:46:38.105617+0000","last_undegraded":"2024-08-17T17:46:38.105617+0000","last_fullsized":"2024-08-17T17:46:38.105617+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1d","version":"90'149","reported_seq":745,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-08-17T17:46:40.276837+0000","last_change":"2024-08-17T17:42:18.668058+0000","last_active":"2024-08-17T17:46:40.276837+0000","last_peered":"2024-08-17T17:46:40.276837+0000","last_clean":"2024-08-17T17:46:40.276837+0000","last_became_active":"2024-08-17T17:42:05.876721+0000","last_became_peered":"2024-08-17T17:42:05.876721+0000","last_unstale":"2024-08-17T17:46:40.276837+0000","last_undegraded":"2024-08-17T17:46:40.276837+0000","last_fullsized":"2024-08-17T17:46:40.276837+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:18.667957+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:18.667957+0000","last_clean_scrub_stamp":"2024-08-17T17:42:18.667957+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:38.170146+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":61,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.042694+0000","last_change":"2024-08-17T17:41:34.611711+0000","last_active":"2024-08-17T17:46:26.042694+0000","last_peered":"2024-08-17T17:46:26.042694+0000","last_clean":"2024-08-17T17:46:26.042694+0000","last_became_active":"2024-08-17T17:41:34.611407+0000","last_became_peered":"2024-08-17T17:41:34.611407+0000","last_unstale":"2024-08-17T17:46:26.042694+0000","last_undegraded":"2024-08-17T17:46:26.042694+0000","last_fullsized":"2024-08-17T17:46:26.042694+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:42:41.582320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"71'4","reported_seq":485,"reported_epoch":396,"state":"active+clean","last_fresh":"2024-08-17T17:46:26.118813+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:46:26.118813+0000","last_peered":"2024-08-17T17:46:26.118813+0000","last_clean":"2024-08-17T17:46:26.118813+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:46:26.118813+0000","last_undegraded":"2024-08-17T17:46:26.118813+0000","last_fullsized":"2024-08-17T17:46:26.118813+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":11256,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19280,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5978,"num_read_kb":12991,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4619,"ondisk_log_size":4619,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234478,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":907620,"kb_used_data":9560,"kb_used_omap":1,"kb_used_meta":898046,"kb_avail":92837532,"statfs":{"total":95995035648,"available":95065632768,"internally_reserved":0,"allocated":9789440,"data_stored":7535189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":919599448},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496113,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":900564,"kb_used_data":9608,"kb_used_omap":1,"kb_used_meta":890942,"kb_avail":92844588,"statfs":{"total":95995035648,"available":95072858112,"internally_reserved":0,"allocated":9838592,"data_stored":7599855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":912324952},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757750,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":887760,"kb_used_data":5436,"kb_used_omap":1,"kb_used_meta":882302,"kb_avail":92857392,"statfs":{"total":95995035648,"available":95085969408,"internally_reserved":0,"allocated":5566464,"data_stored":3328241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":903477592},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019388,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306720,"kb_used_data":5204,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93438432,"statfs":{"total":95995035648,"available":95680954368,"internally_reserved":0,"allocated":5328896,"data_stored":3078458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435615,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":965644,"kb_used_data":5504,"kb_used_omap":4,"kb_used_meta":960123,"kb_avail":92779508,"statfs":{"total":95995035648,"available":95006216192,"internally_reserved":0,"allocated":5636096,"data_stored":3401358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":983165962},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599142,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":853668,"kb_used_data":9560,"kb_used_omap":7,"kb_used_meta":844088,"kb_avail":92891484,"statfs":{"total":95995035648,"available":95120879616,"internally_reserved":0,"allocated":9789440,"data_stored":7539535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":864346274},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":268,"seq":1151051235489,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":459596,"kb_used_data":5184,"kb_used_omap":10,"kb_used_meta":454389,"kb_avail":93285556,"statfs":{"total":95995035648,"available":95524409344,"internally_reserved":0,"allocated":5308416,"data_stored":3073623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10902,"internal_metadata":465294698},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926195,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":931656,"kb_used_data":5496,"kb_used_omap":7,"kb_used_meta":926136,"kb_avail":92813496,"statfs":{"total":95995035648,"available":95041019904,"internally_reserved":0,"allocated":5627904,"data_stored":3393525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":948363585},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7550,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:46:46.305 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:46:46.325 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:46.352 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:46.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:46.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:46.454 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:46.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:46.478 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:46.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:46.579 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:46.606 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:46.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:46.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:46.707 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:46.733 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:46.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:46.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:46.755 DEBUG:teuthology.orchestra.run.smithi003:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-08-17T17:46:46.800 INFO:tasks.workunit.client.0.smithi003.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-08-17T17:46:46.803 INFO:tasks.workunit.client.0.smithi003.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-08-17T17:46:46.803 INFO:tasks.workunit.client.0.smithi003.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-08-17T17:46:46.835 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:46.857 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:46.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:46.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:46.959 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:46.986 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:46.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:46.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:47.088 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:47.089 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-17T17:46:47.090 INFO:tasks.workunit.client.0.smithi003.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-08-17T17:46:47.113 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:47.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:47.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:47.142 INFO:tasks.workunit.client.0.smithi003.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-08-17T17:46:47.186 INFO:tasks.workunit.client.0.smithi003.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-08-17T17:46:47.188 INFO:tasks.workunit.client.0.smithi003.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-08-17T17:46:47.188 INFO:tasks.workunit.client.0.smithi003.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-08-17T17:46:47.215 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:47.236 INFO:tasks.workunit.client.0.smithi003.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-08-17T17:46:47.242 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:47.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:47.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:47.265 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-08-17T17:46:47.265 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-08-17T17:46:47.344 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:47.355 INFO:tasks.workunit:Running workunits matching cls/test_cls_rbd.sh on client.0... 2024-08-17T17:46:47.356 INFO:tasks.workunit:Running workunit cls/test_cls_rbd.sh... 2024-08-17T17:46:47.356 DEBUG:teuthology.orchestra.run.smithi003:workunit test cls/test_cls_rbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh 2024-08-17T17:46:47.367 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:47.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:47.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:47.468 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:47.495 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:47.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:47.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:47.579 INFO:tasks.workunit.client.0.smithi003.stdout:Running main() from gmock_main.cc 2024-08-17T17:46:47.579 INFO:tasks.workunit.client.0.smithi003.stdout:[==========] Running 70 tests from 1 test suite. 2024-08-17T17:46:47.579 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] Global test environment set-up. 2024-08-17T17:46:47.579 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] 70 tests from TestClsRbd 2024-08-17T17:46:47.597 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:47.619 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:47.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:47.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:47.707 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:46:47.721 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:47.744 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:47.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:47.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:47.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:47 smithi003 ceph-mon[103344]: pgmap v1335: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:46:47.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:47 smithi003 ceph-mon[100615]: pgmap v1335: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:46:47.845 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:47.868 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:47.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:47.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:47.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:47 smithi053 ceph-mon[89963]: pgmap v1335: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:46:47.970 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:47.995 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:47.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:47.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:48.005 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-17T17:46:48.097 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:48.119 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:48.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:48.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:48.221 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:48.246 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:48.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:48.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:48.279 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:46:48.348 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:48.371 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:48.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:48.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:48.472 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:48.495 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:48.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:48.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:48.596 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:48.618 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:48.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:48.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:48.720 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:48.723 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.get_all_features 2024-08-17T17:46:48.723 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-08-17T17:46:48.747 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:48.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:48.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:48.778 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.get_all_features (57 ms) 2024-08-17T17:46:48.778 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.copyup 2024-08-17T17:46:48.849 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:48.871 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:48.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:48.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:48.951 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.copyup (172 ms) 2024-08-17T17:46:48.951 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.sparse_copyup 2024-08-17T17:46:48.964 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.sparse_copyup (13 ms) 2024-08-17T17:46:48.964 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.get_and_set_id 2024-08-17T17:46:48.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:48 smithi053 ceph-mon[89963]: osdmap e399: 8 total, 8 up, 8 in 2024-08-17T17:46:48.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:48 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3536984383' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi003-319205-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:46:48.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:48 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1711227584' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:46:48.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:48 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:46:48.971 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.get_and_set_id (8 ms) 2024-08-17T17:46:48.972 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.add_remove_child 2024-08-17T17:46:48.973 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:48.981 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.add_remove_child (10 ms) 2024-08-17T17:46:48.981 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.directory_methods 2024-08-17T17:46:48.997 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:48.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:48.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:49.006 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.directory_methods (25 ms) 2024-08-17T17:46:49.006 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.create 2024-08-17T17:46:49.022 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:48 smithi003 ceph-mon[100615]: osdmap e399: 8 total, 8 up, 8 in 2024-08-17T17:46:49.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:48 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3536984383' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi003-319205-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:46:49.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:48 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1711227584' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:46:49.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:48 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:46:49.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:48 smithi003 ceph-mon[103344]: osdmap e399: 8 total, 8 up, 8 in 2024-08-17T17:46:49.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:48 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3536984383' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi003-319205-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-17T17:46:49.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:48 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1711227584' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:46:49.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:48 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:46:49.029 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.create (22 ms) 2024-08-17T17:46:49.029 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.get_features 2024-08-17T17:46:49.031 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.get_features (2 ms) 2024-08-17T17:46:49.031 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.get_object_prefix 2024-08-17T17:46:49.033 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.get_object_prefix (2 ms) 2024-08-17T17:46:49.033 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.get_create_timestamp 2024-08-17T17:46:49.035 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.get_create_timestamp (2 ms) 2024-08-17T17:46:49.035 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.get_access_timestamp 2024-08-17T17:46:49.036 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.get_access_timestamp (2 ms) 2024-08-17T17:46:49.036 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.get_modify_timestamp 2024-08-17T17:46:49.038 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.get_modify_timestamp (1 ms) 2024-08-17T17:46:49.038 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.get_data_pool 2024-08-17T17:46:49.041 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.get_data_pool (3 ms) 2024-08-17T17:46:49.041 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.get_size 2024-08-17T17:46:49.045 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.get_size (4 ms) 2024-08-17T17:46:49.045 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.set_size 2024-08-17T17:46:49.049 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.set_size (4 ms) 2024-08-17T17:46:49.049 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.protection_status 2024-08-17T17:46:49.067 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.protection_status (18 ms) 2024-08-17T17:46:49.067 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.snapshot_limits 2024-08-17T17:46:49.074 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.snapshot_limits (8 ms) 2024-08-17T17:46:49.075 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.parents_v1 2024-08-17T17:46:49.099 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:49.126 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.parents_v1 (51 ms) 2024-08-17T17:46:49.126 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.parents_v2 2024-08-17T17:46:49.129 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:49.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:49.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:49.155 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.parents_v2 (30 ms) 2024-08-17T17:46:49.156 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.snapshots 2024-08-17T17:46:49.176 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.snapshots (20 ms) 2024-08-17T17:46:49.176 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.snapid_race 2024-08-17T17:46:49.182 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.snapid_race (6 ms) 2024-08-17T17:46:49.182 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.stripingv2 2024-08-17T17:46:49.191 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.stripingv2 (9 ms) 2024-08-17T17:46:49.191 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.object_map_save 2024-08-17T17:46:49.195 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.object_map_save (4 ms) 2024-08-17T17:46:49.195 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.object_map_resize 2024-08-17T17:46:49.204 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.object_map_resize (8 ms) 2024-08-17T17:46:49.204 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.object_map_update 2024-08-17T17:46:49.212 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.object_map_update (9 ms) 2024-08-17T17:46:49.212 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.object_map_load_enoent 2024-08-17T17:46:49.212 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.object_map_load_enoent (0 ms) 2024-08-17T17:46:49.212 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.object_map_snap_add 2024-08-17T17:46:49.219 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.object_map_snap_add (7 ms) 2024-08-17T17:46:49.219 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.object_map_snap_remove 2024-08-17T17:46:49.226 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.object_map_snap_remove (7 ms) 2024-08-17T17:46:49.226 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.flags 2024-08-17T17:46:49.230 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.flags (5 ms) 2024-08-17T17:46:49.231 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.metadata 2024-08-17T17:46:49.231 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:49.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:49.254 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:49.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:49.257 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.metadata (26 ms) 2024-08-17T17:46:49.257 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.set_features 2024-08-17T17:46:49.264 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.set_features (7 ms) 2024-08-17T17:46:49.264 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.mirror 2024-08-17T17:46:49.300 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:46:49.300 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.mirror (36 ms) 2024-08-17T17:46:49.300 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.mirror_image 2024-08-17T17:46:49.317 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.mirror_image (17 ms) 2024-08-17T17:46:49.318 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.mirror_image_status 2024-08-17T17:46:49.355 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:49.384 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:49.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:49.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:49.485 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:49.508 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:49.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:49.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:49.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:49.636 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:49.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:49.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:49.738 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:49.765 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:49.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:49.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:49.867 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:49.889 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:49.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:49.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:49.907 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:46:49.907 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:46:49.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:49 smithi053 ceph-mon[89963]: pgmap v1337: 135 pgs: 32 unknown, 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:46:49.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:49 smithi053 ceph-mon[89963]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:46:49.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:49 smithi053 ceph-mon[89963]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:46:49.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:49 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3536984383' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi003-319205-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:46:49.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:49 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:46:49.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:49 smithi053 ceph-mon[89963]: osdmap e400: 8 total, 8 up, 8 in 2024-08-17T17:46:49.991 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:50.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:50.014 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:50.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:50.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:49 smithi003 ceph-mon[103344]: pgmap v1337: 135 pgs: 32 unknown, 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:46:50.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:49 smithi003 ceph-mon[103344]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:46:50.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:49 smithi003 ceph-mon[103344]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:46:50.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:49 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3536984383' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi003-319205-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:46:50.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:49 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:46:50.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:49 smithi003 ceph-mon[103344]: osdmap e400: 8 total, 8 up, 8 in 2024-08-17T17:46:50.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:49 smithi003 ceph-mon[100615]: pgmap v1337: 135 pgs: 32 unknown, 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:46:50.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:49 smithi003 ceph-mon[100615]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-17T17:46:50.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:49 smithi003 ceph-mon[100615]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-17T17:46:50.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:49 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3536984383' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi003-319205-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-17T17:46:50.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:49 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:46:50.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:49 smithi003 ceph-mon[100615]: osdmap e400: 8 total, 8 up, 8 in 2024-08-17T17:46:50.114 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:50.141 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:50.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:50.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:50.243 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:50.266 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:50.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:50.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:50.368 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:50.390 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:50.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:50.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:50.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:46:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:46:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:46:50.492 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:50.514 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:50.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:50.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:50.581 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-17T17:46:50.615 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:50.648 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:50.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:50.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:50.749 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:50.772 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:50.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:50.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:50.850 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:46:50.874 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:50.901 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:50.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:50.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:50.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:50 smithi053 ceph-mon[89963]: osdmap e401: 8 total, 8 up, 8 in 2024-08-17T17:46:51.000 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:50 smithi003 ceph-mon[100615]: osdmap e401: 8 total, 8 up, 8 in 2024-08-17T17:46:51.000 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:50 smithi003 ceph-mon[103344]: osdmap e401: 8 total, 8 up, 8 in 2024-08-17T17:46:51.003 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:51.025 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:51.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:51.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:51.127 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:51.148 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:51.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:51.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:51.250 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:51.276 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:51.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:51.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:51.378 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:51.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:51.403 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:51.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:51.504 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:51.534 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:51.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:51.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:51.558 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:46:51.559 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:46:51.581 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:46:51.585 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:46:51.585 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:46:51.608 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:46:51.608 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:46:51.633 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:46:51.634 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:46:51.637 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0 2024-08-17T17:46:51.655 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:46:51.655 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:46:51.664 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:51.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:51.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.0' 2024-08-17T17:46:51.676 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:46:51.676 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:46:51.696 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:46:51.697 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:46:51.716 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:46:51.716 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:46:51.766 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:51.789 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:51.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:51.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:51.833 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:51 smithi003 ceph-mon[100615]: pgmap v1340: 135 pgs: 2 creating+activating, 11 unknown, 122 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 41 KiB/s rd, 11 KiB/s wr, 62 op/s 2024-08-17T17:46:51.833 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:51 smithi003 ceph-mon[100615]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:46:51.833 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:51 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2530313484' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:46:51.834 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:51 smithi003 ceph-mon[103344]: pgmap v1340: 135 pgs: 2 creating+activating, 11 unknown, 122 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 41 KiB/s rd, 11 KiB/s wr, 62 op/s 2024-08-17T17:46:51.834 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:51 smithi003 ceph-mon[103344]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:46:51.834 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:51 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2530313484' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:46:51.891 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:51.914 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:51.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:51.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:51.974 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.mirror_image_status (2657 ms) 2024-08-17T17:46:51.974 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.mirror_image_map 2024-08-17T17:46:51.985 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:51 smithi053 ceph-mon[89963]: pgmap v1340: 135 pgs: 2 creating+activating, 11 unknown, 122 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 41 KiB/s rd, 11 KiB/s wr, 62 op/s 2024-08-17T17:46:51.985 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:51 smithi053 ceph-mon[89963]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-17T17:46:51.985 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:51 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2530313484' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:46:52.015 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2 2024-08-17T17:46:52.041 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:52.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:52.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.2' 2024-08-17T17:46:52.065 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.mirror_image_map (91 ms) 2024-08-17T17:46:52.065 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.mirror_instances 2024-08-17T17:46:52.072 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.mirror_instances (7 ms) 2024-08-17T17:46:52.072 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.mirror_snapshot 2024-08-17T17:46:52.086 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.mirror_snapshot (14 ms) 2024-08-17T17:46:52.086 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_dir_list 2024-08-17T17:46:52.089 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_dir_list (3 ms) 2024-08-17T17:46:52.089 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_dir_add 2024-08-17T17:46:52.091 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_dir_add (2 ms) 2024-08-17T17:46:52.091 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.dir_add_already_existing 2024-08-17T17:46:52.095 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.dir_add_already_existing (4 ms) 2024-08-17T17:46:52.095 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_dir_rename 2024-08-17T17:46:52.100 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_dir_rename (5 ms) 2024-08-17T17:46:52.100 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_dir_remove 2024-08-17T17:46:52.103 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_dir_remove (3 ms) 2024-08-17T17:46:52.103 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_dir_remove_missing 2024-08-17T17:46:52.106 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_dir_remove_missing (4 ms) 2024-08-17T17:46:52.107 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_image_add 2024-08-17T17:46:52.109 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_image_add (2 ms) 2024-08-17T17:46:52.109 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_image_remove 2024-08-17T17:46:52.112 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_image_remove (3 ms) 2024-08-17T17:46:52.112 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_image_list 2024-08-17T17:46:52.115 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_image_list (3 ms) 2024-08-17T17:46:52.115 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_image_clean 2024-08-17T17:46:52.119 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_image_clean (4 ms) 2024-08-17T17:46:52.119 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.image_group_add 2024-08-17T17:46:52.121 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.image_group_add (2 ms) 2024-08-17T17:46:52.121 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.image_group_remove 2024-08-17T17:46:52.124 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.image_group_remove (3 ms) 2024-08-17T17:46:52.124 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.image_group_get 2024-08-17T17:46:52.128 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.image_group_get (4 ms) 2024-08-17T17:46:52.128 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_name 2024-08-17T17:46:52.129 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_snap_set_empty_name (2 ms) 2024-08-17T17:46:52.129 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_id 2024-08-17T17:46:52.131 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_snap_set_empty_id (2 ms) 2024-08-17T17:46:52.132 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_id 2024-08-17T17:46:52.134 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_id (3 ms) 2024-08-17T17:46:52.134 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_name 2024-08-17T17:46:52.137 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_name (2 ms) 2024-08-17T17:46:52.137 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_snap_set 2024-08-17T17:46:52.139 INFO:tasks.workunit.client.0.smithi003.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/17.2.7-1404-gf2c39d4d/rpm/el9/BUILD/ceph-17.2.7-1404-gf2c39d4d/src/test/cls_rbd/test_cls_rbd.cc:2681: Failure 2024-08-17T17:46:52.139 INFO:tasks.workunit.client.0.smithi003.stdout:Expected equality of these values: 2024-08-17T17:46:52.139 INFO:tasks.workunit.client.0.smithi003.stdout: 1U 2024-08-17T17:46:52.139 INFO:tasks.workunit.client.0.smithi003.stdout: Which is: 1 2024-08-17T17:46:52.139 INFO:tasks.workunit.client.0.smithi003.stdout: keys.size() 2024-08-17T17:46:52.140 INFO:tasks.workunit.client.0.smithi003.stdout: Which is: 3 2024-08-17T17:46:52.140 INFO:tasks.workunit.client.0.smithi003.stdout:[ FAILED ] TestClsRbd.group_snap_set (2 ms) 2024-08-17T17:46:52.140 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_snap_list 2024-08-17T17:46:52.142 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_snap_list (3 ms) 2024-08-17T17:46:52.142 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_snap_list_max_return 2024-08-17T17:46:52.144 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:52.160 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_snap_list_max_return (19 ms) 2024-08-17T17:46:52.160 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_snap_list_max_read 2024-08-17T17:46:52.168 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:52.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:52.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:52.270 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:52.295 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:52.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:52.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:52.397 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:52.412 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_snap_list_max_read (251 ms) 2024-08-17T17:46:52.412 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_snap_remove 2024-08-17T17:46:52.415 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_snap_remove (3 ms) 2024-08-17T17:46:52.416 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.group_snap_get_by_id 2024-08-17T17:46:52.417 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.group_snap_get_by_id (3 ms) 2024-08-17T17:46:52.417 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.trash_methods 2024-08-17T17:46:52.422 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:52.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:52.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:52.424 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.trash_methods (7 ms) 2024-08-17T17:46:52.425 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.op_features 2024-08-17T17:46:52.432 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.op_features (8 ms) 2024-08-17T17:46:52.432 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.clone_parent 2024-08-17T17:46:52.453 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.clone_parent (20 ms) 2024-08-17T17:46:52.453 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.clone_parent_ns 2024-08-17T17:46:52.462 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.clone_parent_ns (9 ms) 2024-08-17T17:46:52.462 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.clone_child 2024-08-17T17:46:52.473 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.clone_child (11 ms) 2024-08-17T17:46:52.473 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.namespace_methods 2024-08-17T17:46:52.479 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.namespace_methods (6 ms) 2024-08-17T17:46:52.479 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.migration 2024-08-17T17:46:52.490 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.migration (12 ms) 2024-08-17T17:46:52.491 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.migration_v1 2024-08-17T17:46:52.501 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.migration_v1 (10 ms) 2024-08-17T17:46:52.501 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.assert_snapc_seq 2024-08-17T17:46:52.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:52.545 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:52.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:52.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:52.551 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:46:52.551 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2024-08-17T17:46:52.551 INFO:teuthology.orchestra.run.smithi053.stdout: "num_ops": 0 2024-08-17T17:46:52.551 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:46:52.647 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:52.670 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:52.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:52.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:52.772 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:52.794 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:52.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:52.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:52.896 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:52.918 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:52.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:52.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:53.019 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:53.030 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:46:53.041 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:53.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:53.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:53.143 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3 2024-08-17T17:46:53.166 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:53.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:53.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.3' 2024-08-17T17:46:53.171 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-17T17:46:53.268 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:53.291 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:53.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:53.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:53.393 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:53.416 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:53.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:53.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:53.436 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:46:53.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:53.542 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:53.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:53.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:53.643 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:53.667 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:53.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:53.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:53.743 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.assert_snapc_seq (1242 ms) 2024-08-17T17:46:53.744 INFO:tasks.workunit.client.0.smithi003.stdout:[ RUN ] TestClsRbd.sparsify 2024-08-17T17:46:53.768 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:53.771 INFO:tasks.workunit.client.0.smithi003.stdout:[ OK ] TestClsRbd.sparsify (28 ms) 2024-08-17T17:46:53.790 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:53.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:53.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:53.812 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:53 smithi053 ceph-mon[89963]: pgmap v1341: 135 pgs: 2 creating+activating, 4 unknown, 129 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.4 MiB/s rd, 700 KiB/s wr, 106 op/s 2024-08-17T17:46:53.812 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:53 smithi053 ceph-mon[89963]: osdmap e402: 8 total, 8 up, 8 in 2024-08-17T17:46:53.812 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:46:53.890 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:53 smithi003 ceph-mon[103344]: pgmap v1341: 135 pgs: 2 creating+activating, 4 unknown, 129 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.4 MiB/s rd, 700 KiB/s wr, 106 op/s 2024-08-17T17:46:53.890 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:53 smithi003 ceph-mon[103344]: osdmap e402: 8 total, 8 up, 8 in 2024-08-17T17:46:53.890 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:46:53.890 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:53 smithi003 ceph-mon[100615]: pgmap v1341: 135 pgs: 2 creating+activating, 4 unknown, 129 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.4 MiB/s rd, 700 KiB/s wr, 106 op/s 2024-08-17T17:46:53.890 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:53 smithi003 ceph-mon[100615]: osdmap e402: 8 total, 8 up, 8 in 2024-08-17T17:46:53.890 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:46:53.892 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7 2024-08-17T17:46:53.915 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:53.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:53.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.7' 2024-08-17T17:46:54.016 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:54.039 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:54.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:54.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:54.140 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:54.157 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:46:54.164 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:54.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:54.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:54.266 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:54.289 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:54.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:54.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:54.309 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:46:54.310 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:46:54.390 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:54.413 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:54.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:54.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:54.515 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1 2024-08-17T17:46:54.538 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-08-17T17:46:54.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:54.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.1' 2024-08-17T17:46:54.640 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4 2024-08-17T17:46:54.663 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:54.664 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-17T17:46:54.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:54.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.4' 2024-08-17T17:46:54.750 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] 70 tests from TestClsRbd (5049 ms total) 2024-08-17T17:46:54.751 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-08-17T17:46:54.751 INFO:tasks.workunit.client.0.smithi003.stdout:[----------] Global test environment tear-down 2024-08-17T17:46:54.751 INFO:tasks.workunit.client.0.smithi003.stdout:[==========] 70 tests from 1 test suite ran. (7172 ms total) 2024-08-17T17:46:54.751 INFO:tasks.workunit.client.0.smithi003.stdout:[ PASSED ] 69 tests. 2024-08-17T17:46:54.751 INFO:tasks.workunit.client.0.smithi003.stdout:[ FAILED ] 1 test, listed below: 2024-08-17T17:46:54.751 INFO:tasks.workunit.client.0.smithi003.stdout:[ FAILED ] TestClsRbd.group_snap_set 2024-08-17T17:46:54.751 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-08-17T17:46:54.751 INFO:tasks.workunit.client.0.smithi003.stdout: 1 FAILED TEST 2024-08-17T17:46:54.766 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5 2024-08-17T17:46:54.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:54.771 INFO:tasks.workunit:Stopping ['cls/test_cls_rbd.sh'] on client.0... 2024-08-17T17:46:54.772 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-08-17T17:46:54.788 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:54.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:54.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.5' 2024-08-17T17:46:54.810 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:54 smithi053 ceph-mon[89963]: osdmap e403: 8 total, 8 up, 8 in 2024-08-17T17:46:54.890 DEBUG:teuthology.orchestra.run.smithi053:> sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6 2024-08-17T17:46:54.913 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: docker: command not found 2024-08-17T17:46:54.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T17:46:54.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi053 with status 1: 'sudo docker kill -s 1 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd.6' 2024-08-17T17:46:54.983 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:54 smithi003 ceph-mon[103344]: osdmap e403: 8 total, 8 up, 8 in 2024-08-17T17:46:54.983 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:54 smithi003 ceph-mon[100615]: osdmap e403: 8 total, 8 up, 8 in 2024-08-17T17:46:54.984 INFO:tasks.thrashosds:joining thrashosds 2024-08-17T17:46:55.013 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:46:55.013 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [], 2024-08-17T17:46:55.013 INFO:teuthology.orchestra.run.smithi053.stdout: "complaint_time": 30, 2024-08-17T17:46:55.013 INFO:teuthology.orchestra.run.smithi053.stdout: "num_blocked_ops": 0 2024-08-17T17:46:55.013 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:46:55.520 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1343,"stamp":"2024-08-17T17:46:53.740899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790595,"num_objects":56,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":15048,"num_read_kb":30221,"num_write":5246,"num_write_kb":20990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":194,"num_bytes_recovered":8399864,"num_keys_recovered":40,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7997,"ondisk_log_size":7997,"up":405,"acting":405,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":315,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6209424,"kb_used_data":55900,"kb_used_omap":37,"kb_used_meta":6153370,"kb_avail":743751792,"statfs":{"total":767960285184,"available":761601835008,"internally_reserved":0,"allocated":57241600,"data_stored":39147953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38729,"internal_metadata":6301051063},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":374,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":9070,"num_read_kb":17230,"num_write":4339,"num_write_kb":8367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.297942"},"pg_stats":[{"pgid":"3.1f","version":"71'4","reported_seq":495,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.721798+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:46:49.721798+0000","last_peered":"2024-08-17T17:46:49.721798+0000","last_clean":"2024-08-17T17:46:49.721798+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:46:49.721798+0000","last_undegraded":"2024-08-17T17:46:49.721798+0000","last_fullsized":"2024-08-17T17:46:49.721798+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":64,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.714320+0000","last_change":"2024-08-17T17:41:34.611711+0000","last_active":"2024-08-17T17:46:47.714320+0000","last_peered":"2024-08-17T17:46:47.714320+0000","last_clean":"2024-08-17T17:46:47.714320+0000","last_became_active":"2024-08-17T17:41:34.611407+0000","last_became_peered":"2024-08-17T17:41:34.611407+0000","last_unstale":"2024-08-17T17:46:47.714320+0000","last_undegraded":"2024-08-17T17:46:47.714320+0000","last_fullsized":"2024-08-17T17:46:47.714320+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:42:41.582320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1d","version":"90'149","reported_seq":751,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.724980+0000","last_change":"2024-08-17T17:42:18.668058+0000","last_active":"2024-08-17T17:46:49.724980+0000","last_peered":"2024-08-17T17:46:49.724980+0000","last_clean":"2024-08-17T17:46:49.724980+0000","last_became_active":"2024-08-17T17:42:05.876721+0000","last_became_peered":"2024-08-17T17:42:05.876721+0000","last_unstale":"2024-08-17T17:46:49.724980+0000","last_undegraded":"2024-08-17T17:46:49.724980+0000","last_fullsized":"2024-08-17T17:46:49.724980+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:18.667957+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:18.667957+0000","last_clean_scrub_stamp":"2024-08-17T17:42:18.667957+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:38.170146+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":3,"reported_epoch":400,"state":"creating+activating","last_fresh":"2024-08-17T17:46:48.722849+0000","last_change":"2024-08-17T17:46:48.722849+0000","last_active":"2024-08-17T17:46:47.699070+0000","last_peered":"2024-08-17T17:46:47.699070+0000","last_clean":"2024-08-17T17:46:47.699070+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T17:46:48.722849+0000","last_undegraded":"2024-08-17T17:46:48.722849+0000","last_fullsized":"2024-08-17T17:46:48.722849+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:49:35.000271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":["7","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":684,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.729674+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:46:48.729674+0000","last_peered":"2024-08-17T17:46:48.729674+0000","last_clean":"2024-08-17T17:46:48.729674+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:46:48.729674+0000","last_undegraded":"2024-08-17T17:46:48.729674+0000","last_fullsized":"2024-08-17T17:46:48.729674+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":683,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.727879+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:46:48.727879+0000","last_peered":"2024-08-17T17:46:48.727879+0000","last_clean":"2024-08-17T17:46:48.727879+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:46:48.727879+0000","last_undegraded":"2024-08-17T17:46:48.727879+0000","last_fullsized":"2024-08-17T17:46:48.727879+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"90'149","reported_seq":740,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.716171+0000","last_change":"2024-08-17T17:42:11.809047+0000","last_active":"2024-08-17T17:46:47.716171+0000","last_peered":"2024-08-17T17:46:47.716171+0000","last_clean":"2024-08-17T17:46:47.716171+0000","last_became_active":"2024-08-17T17:42:05.883770+0000","last_became_peered":"2024-08-17T17:42:05.883770+0000","last_unstale":"2024-08-17T17:46:47.716171+0000","last_undegraded":"2024-08-17T17:46:47.716171+0000","last_fullsized":"2024-08-17T17:46:47.716171+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:11.808971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:11.808971+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:48:15.989437+0000","scrub_duration":0.0018167420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":3,"reported_epoch":400,"state":"creating+activating","last_fresh":"2024-08-17T17:46:48.722388+0000","last_change":"2024-08-17T17:46:48.722388+0000","last_active":"2024-08-17T17:46:47.699070+0000","last_peered":"2024-08-17T17:46:47.699070+0000","last_clean":"2024-08-17T17:46:47.699070+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-08-17T17:46:48.722388+0000","last_undegraded":"2024-08-17T17:46:48.722388+0000","last_fullsized":"2024-08-17T17:46:48.722388+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:29:48.952287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":["7","3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":68,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.715092+0000","last_change":"2024-08-17T17:41:34.631152+0000","last_active":"2024-08-17T17:46:47.715092+0000","last_peered":"2024-08-17T17:46:47.715092+0000","last_clean":"2024-08-17T17:46:47.715092+0000","last_became_active":"2024-08-17T17:41:34.615617+0000","last_became_peered":"2024-08-17T17:41:34.615617+0000","last_unstale":"2024-08-17T17:46:47.715092+0000","last_undegraded":"2024-08-17T17:46:47.715092+0000","last_fullsized":"2024-08-17T17:46:47.715092+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:43:08.257084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":64,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.714676+0000","last_change":"2024-08-17T17:41:34.616251+0000","last_active":"2024-08-17T17:46:47.714676+0000","last_peered":"2024-08-17T17:46:47.714676+0000","last_clean":"2024-08-17T17:46:47.714676+0000","last_became_active":"2024-08-17T17:41:34.615952+0000","last_became_peered":"2024-08-17T17:41:34.615952+0000","last_unstale":"2024-08-17T17:46:47.714676+0000","last_undegraded":"2024-08-17T17:46:47.714676+0000","last_fullsized":"2024-08-17T17:46:47.714676+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:12:24.892508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.15","version":"400'1","reported_seq":16,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.725038+0000","last_change":"2024-08-17T17:46:48.736579+0000","last_active":"2024-08-17T17:46:49.725038+0000","last_peered":"2024-08-17T17:46:49.725038+0000","last_clean":"2024-08-17T17:46:49.725038+0000","last_became_active":"2024-08-17T17:46:48.733581+0000","last_became_peered":"2024-08-17T17:46:48.733581+0000","last_unstale":"2024-08-17T17:46:49.725038+0000","last_undegraded":"2024-08-17T17:46:49.725038+0000","last_fullsized":"2024-08-17T17:46:49.725038+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:27:10.571236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":68,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.715072+0000","last_change":"2024-08-17T17:41:34.629135+0000","last_active":"2024-08-17T17:46:47.715072+0000","last_peered":"2024-08-17T17:46:47.715072+0000","last_clean":"2024-08-17T17:46:47.715072+0000","last_became_active":"2024-08-17T17:41:34.613095+0000","last_became_peered":"2024-08-17T17:41:34.613095+0000","last_unstale":"2024-08-17T17:46:47.715072+0000","last_undegraded":"2024-08-17T17:46:47.715072+0000","last_fullsized":"2024-08-17T17:46:47.715072+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:27:06.542416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":64,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.715209+0000","last_change":"2024-08-17T17:41:34.622094+0000","last_active":"2024-08-17T17:46:47.715209+0000","last_peered":"2024-08-17T17:46:47.715209+0000","last_clean":"2024-08-17T17:46:47.715209+0000","last_became_active":"2024-08-17T17:41:34.621773+0000","last_became_peered":"2024-08-17T17:41:34.621773+0000","last_unstale":"2024-08-17T17:46:47.715209+0000","last_undegraded":"2024-08-17T17:46:47.715209+0000","last_fullsized":"2024-08-17T17:46:47.715209+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:34:30.540886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"90'149","reported_seq":502,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.729518+0000","last_change":"2024-08-17T17:42:06.742801+0000","last_active":"2024-08-17T17:46:48.729518+0000","last_peered":"2024-08-17T17:46:48.729518+0000","last_clean":"2024-08-17T17:46:48.729518+0000","last_became_active":"2024-08-17T17:42:05.876895+0000","last_became_peered":"2024-08-17T17:42:05.876895+0000","last_unstale":"2024-08-17T17:46:48.729518+0000","last_undegraded":"2024-08-17T17:46:48.729518+0000","last_fullsized":"2024-08-17T17:46:48.729518+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.742723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.742723+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:36:46.651953+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":9,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.740071+0000","last_change":"2024-08-17T17:46:48.740071+0000","last_active":"2024-08-17T17:46:48.740071+0000","last_peered":"2024-08-17T17:46:48.740071+0000","last_clean":"2024-08-17T17:46:48.740071+0000","last_became_active":"2024-08-17T17:46:48.739518+0000","last_became_peered":"2024-08-17T17:46:48.739518+0000","last_unstale":"2024-08-17T17:46:48.740071+0000","last_undegraded":"2024-08-17T17:46:48.740071+0000","last_fullsized":"2024-08-17T17:46:48.740071+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:56:43.909430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":500,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.722212+0000","last_change":"2024-08-17T17:41:34.618968+0000","last_active":"2024-08-17T17:46:49.722212+0000","last_peered":"2024-08-17T17:46:49.722212+0000","last_clean":"2024-08-17T17:46:49.722212+0000","last_became_active":"2024-08-17T17:41:34.618576+0000","last_became_peered":"2024-08-17T17:41:34.618576+0000","last_unstale":"2024-08-17T17:46:49.722212+0000","last_undegraded":"2024-08-17T17:46:49.722212+0000","last_fullsized":"2024-08-17T17:46:49.722212+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:47:45.888590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":593,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.725144+0000","last_change":"2024-08-17T17:33:20.773207+0000","last_active":"2024-08-17T17:46:49.725144+0000","last_peered":"2024-08-17T17:46:49.725144+0000","last_clean":"2024-08-17T17:46:49.725144+0000","last_became_active":"2024-08-17T17:33:20.772457+0000","last_became_peered":"2024-08-17T17:33:20.772457+0000","last_unstale":"2024-08-17T17:46:49.725144+0000","last_undegraded":"2024-08-17T17:46:49.725144+0000","last_fullsized":"2024-08-17T17:46:49.725144+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:57:09.610084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"90'149","reported_seq":854,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.727099+0000","last_change":"2024-08-17T17:42:05.876463+0000","last_active":"2024-08-17T17:46:48.727099+0000","last_peered":"2024-08-17T17:46:48.727099+0000","last_clean":"2024-08-17T17:46:48.727099+0000","last_became_active":"2024-08-17T17:42:05.876069+0000","last_became_peered":"2024-08-17T17:42:05.876069+0000","last_unstale":"2024-08-17T17:46:48.727099+0000","last_undegraded":"2024-08-17T17:46:48.727099+0000","last_fullsized":"2024-08-17T17:46:48.727099+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:45.552985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:45.552985+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:55:06.027278+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":9,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.842607+0000","last_change":"2024-08-17T17:46:48.842607+0000","last_active":"2024-08-17T17:46:48.842607+0000","last_peered":"2024-08-17T17:46:48.842607+0000","last_clean":"2024-08-17T17:46:48.842607+0000","last_became_active":"2024-08-17T17:46:48.842290+0000","last_became_peered":"2024-08-17T17:46:48.842290+0000","last_unstale":"2024-08-17T17:46:48.842607+0000","last_undegraded":"2024-08-17T17:46:48.842607+0000","last_fullsized":"2024-08-17T17:46:48.842607+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:35:55.699493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":64,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.714589+0000","last_change":"2024-08-17T17:41:34.622184+0000","last_active":"2024-08-17T17:46:47.714589+0000","last_peered":"2024-08-17T17:46:47.714589+0000","last_clean":"2024-08-17T17:46:47.714589+0000","last_became_active":"2024-08-17T17:41:34.621831+0000","last_became_peered":"2024-08-17T17:41:34.621831+0000","last_unstale":"2024-08-17T17:46:47.714589+0000","last_undegraded":"2024-08-17T17:46:47.714589+0000","last_fullsized":"2024-08-17T17:46:47.714589+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:59.826667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":64,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.715450+0000","last_change":"2024-08-17T17:41:34.616482+0000","last_active":"2024-08-17T17:46:47.715450+0000","last_peered":"2024-08-17T17:46:47.715450+0000","last_clean":"2024-08-17T17:46:47.715450+0000","last_became_active":"2024-08-17T17:41:34.616040+0000","last_became_peered":"2024-08-17T17:41:34.616040+0000","last_unstale":"2024-08-17T17:46:47.715450+0000","last_undegraded":"2024-08-17T17:46:47.715450+0000","last_fullsized":"2024-08-17T17:46:47.715450+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:53:08.502264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.18","version":"90'149","reported_seq":705,"reported_epoch":398,"state":"active+clean","last_fresh":"2024-08-17T17:46:39.682763+0000","last_change":"2024-08-17T17:42:08.482975+0000","last_active":"2024-08-17T17:46:39.682763+0000","last_peered":"2024-08-17T17:46:39.682763+0000","last_clean":"2024-08-17T17:46:39.682763+0000","last_became_active":"2024-08-17T17:42:05.873523+0000","last_became_peered":"2024-08-17T17:42:05.873523+0000","last_unstale":"2024-08-17T17:46:39.682763+0000","last_undegraded":"2024-08-17T17:46:39.682763+0000","last_fullsized":"2024-08-17T17:46:39.682763+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:08.482892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:08.482892+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:41.477341+0000","scrub_duration":0.0017925479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.12","version":"400'1","reported_seq":16,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.724424+0000","last_change":"2024-08-17T17:46:48.723416+0000","last_active":"2024-08-17T17:46:49.724424+0000","last_peered":"2024-08-17T17:46:49.724424+0000","last_clean":"2024-08-17T17:46:49.724424+0000","last_became_active":"2024-08-17T17:46:48.723119+0000","last_became_peered":"2024-08-17T17:46:48.723119+0000","last_unstale":"2024-08-17T17:46:49.724424+0000","last_undegraded":"2024-08-17T17:46:49.724424+0000","last_fullsized":"2024-08-17T17:46:49.724424+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:24:20.245636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":626,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.723667+0000","last_change":"2024-08-17T17:41:34.625011+0000","last_active":"2024-08-17T17:46:49.723667+0000","last_peered":"2024-08-17T17:46:49.723667+0000","last_clean":"2024-08-17T17:46:49.723667+0000","last_became_active":"2024-08-17T17:41:34.624631+0000","last_became_peered":"2024-08-17T17:41:34.624631+0000","last_unstale":"2024-08-17T17:46:49.723667+0000","last_undegraded":"2024-08-17T17:46:49.723667+0000","last_fullsized":"2024-08-17T17:46:49.723667+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:03:05.565435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":477,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.722856+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:46:49.722856+0000","last_peered":"2024-08-17T17:46:49.722856+0000","last_clean":"2024-08-17T17:46:49.722856+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:46:49.722856+0000","last_undegraded":"2024-08-17T17:46:49.722856+0000","last_fullsized":"2024-08-17T17:46:49.722856+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1b","version":"90'149","reported_seq":743,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.715219+0000","last_change":"2024-08-17T17:42:26.967686+0000","last_active":"2024-08-17T17:46:47.715219+0000","last_peered":"2024-08-17T17:46:47.715219+0000","last_clean":"2024-08-17T17:46:47.715219+0000","last_became_active":"2024-08-17T17:42:05.873657+0000","last_became_peered":"2024-08-17T17:42:05.873657+0000","last_unstale":"2024-08-17T17:46:47.715219+0000","last_undegraded":"2024-08-17T17:46:47.715219+0000","last_fullsized":"2024-08-17T17:46:47.715219+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:26.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:26.967611+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:31:03.344332+0000","scrub_duration":0.0015978279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.11","version":"400'5","reported_seq":18,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.184448+0000","last_change":"2024-08-17T17:46:48.734020+0000","last_active":"2024-08-17T17:46:49.184448+0000","last_peered":"2024-08-17T17:46:49.184448+0000","last_clean":"2024-08-17T17:46:49.184448+0000","last_became_active":"2024-08-17T17:46:48.733714+0000","last_became_peered":"2024-08-17T17:46:48.733714+0000","last_unstale":"2024-08-17T17:46:49.184448+0000","last_undegraded":"2024-08-17T17:46:49.184448+0000","last_fullsized":"2024-08-17T17:46:49.184448+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:21:08.490595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":705,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.725882+0000","last_change":"2024-08-17T17:33:20.771279+0000","last_active":"2024-08-17T17:46:48.725882+0000","last_peered":"2024-08-17T17:46:48.725882+0000","last_clean":"2024-08-17T17:46:48.725882+0000","last_became_active":"2024-08-17T17:33:20.770728+0000","last_became_peered":"2024-08-17T17:33:20.770728+0000","last_unstale":"2024-08-17T17:46:48.725882+0000","last_undegraded":"2024-08-17T17:46:48.725882+0000","last_fullsized":"2024-08-17T17:46:48.725882+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:52:31.540397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":373,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.717761+0000","last_change":"2024-08-17T17:33:20.835760+0000","last_active":"2024-08-17T17:46:47.717761+0000","last_peered":"2024-08-17T17:46:47.717761+0000","last_clean":"2024-08-17T17:46:47.717761+0000","last_became_active":"2024-08-17T17:33:20.835410+0000","last_became_peered":"2024-08-17T17:33:20.835410+0000","last_unstale":"2024-08-17T17:46:47.717761+0000","last_undegraded":"2024-08-17T17:46:47.717761+0000","last_fullsized":"2024-08-17T17:46:47.717761+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:45.448022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1a","version":"90'149","reported_seq":448,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.714448+0000","last_change":"2024-08-17T17:42:21.932830+0000","last_active":"2024-08-17T17:46:47.714448+0000","last_peered":"2024-08-17T17:46:47.714448+0000","last_clean":"2024-08-17T17:46:47.714448+0000","last_became_active":"2024-08-17T17:42:05.876835+0000","last_became_peered":"2024-08-17T17:42:05.876835+0000","last_unstale":"2024-08-17T17:46:47.714448+0000","last_undegraded":"2024-08-17T17:46:47.714448+0000","last_fullsized":"2024-08-17T17:46:47.714448+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:21.932757+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:21.932757+0000","last_clean_scrub_stamp":"2024-08-17T17:42:21.932757+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:36:39.848570+0000","scrub_duration":0.0020218810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.10","version":"400'4","reported_seq":18,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.720809+0000","last_change":"2024-08-17T17:46:48.734679+0000","last_active":"2024-08-17T17:46:49.720809+0000","last_peered":"2024-08-17T17:46:49.720809+0000","last_clean":"2024-08-17T17:46:49.720809+0000","last_became_active":"2024-08-17T17:46:48.734412+0000","last_became_peered":"2024-08-17T17:46:48.734412+0000","last_unstale":"2024-08-17T17:46:49.720809+0000","last_undegraded":"2024-08-17T17:46:49.720809+0000","last_fullsized":"2024-08-17T17:46:49.720809+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:53:49.393988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":11,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.736822+0000","last_change":"2024-08-17T17:46:48.735943+0000","last_active":"2024-08-17T17:46:48.736822+0000","last_peered":"2024-08-17T17:46:48.736822+0000","last_clean":"2024-08-17T17:46:48.736822+0000","last_became_active":"2024-08-17T17:46:48.735535+0000","last_became_peered":"2024-08-17T17:46:48.735535+0000","last_unstale":"2024-08-17T17:46:48.736822+0000","last_undegraded":"2024-08-17T17:46:48.736822+0000","last_fullsized":"2024-08-17T17:46:48.736822+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:02:41.232229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":687,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.725695+0000","last_change":"2024-08-17T17:33:20.790687+0000","last_active":"2024-08-17T17:46:48.725695+0000","last_peered":"2024-08-17T17:46:48.725695+0000","last_clean":"2024-08-17T17:46:48.725695+0000","last_became_active":"2024-08-17T17:33:20.790399+0000","last_became_peered":"2024-08-17T17:33:20.790399+0000","last_unstale":"2024-08-17T17:46:48.725695+0000","last_undegraded":"2024-08-17T17:46:48.725695+0000","last_fullsized":"2024-08-17T17:46:48.725695+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:38:06.849989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":658,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.721897+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:46:49.721897+0000","last_peered":"2024-08-17T17:46:49.721897+0000","last_clean":"2024-08-17T17:46:49.721897+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:46:49.721897+0000","last_undegraded":"2024-08-17T17:46:49.721897+0000","last_fullsized":"2024-08-17T17:46:49.721897+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"90'149","reported_seq":786,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.709035+0000","last_change":"2024-08-17T17:42:06.456562+0000","last_active":"2024-08-17T17:46:47.709035+0000","last_peered":"2024-08-17T17:46:47.709035+0000","last_clean":"2024-08-17T17:46:47.709035+0000","last_became_active":"2024-08-17T17:42:05.876577+0000","last_became_peered":"2024-08-17T17:42:05.876577+0000","last_unstale":"2024-08-17T17:46:47.709035+0000","last_undegraded":"2024-08-17T17:46:47.709035+0000","last_fullsized":"2024-08-17T17:46:47.709035+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.456490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.456490+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:10:03.654953+0000","scrub_duration":0.001537285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1e","version":"400'7","reported_seq":17,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.971721+0000","last_change":"2024-08-17T17:46:48.734636+0000","last_active":"2024-08-17T17:46:48.971721+0000","last_peered":"2024-08-17T17:46:48.971721+0000","last_clean":"2024-08-17T17:46:48.971721+0000","last_became_active":"2024-08-17T17:46:48.734282+0000","last_became_peered":"2024-08-17T17:46:48.734282+0000","last_unstale":"2024-08-17T17:46:48.971721+0000","last_undegraded":"2024-08-17T17:46:48.971721+0000","last_fullsized":"2024-08-17T17:46:48.971721+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:07:19.822469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":629,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.727297+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:46:48.727297+0000","last_peered":"2024-08-17T17:46:48.727297+0000","last_clean":"2024-08-17T17:46:48.727297+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:46:48.727297+0000","last_undegraded":"2024-08-17T17:46:48.727297+0000","last_fullsized":"2024-08-17T17:46:48.727297+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":366,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.725525+0000","last_change":"2024-08-17T17:33:44.146014+0000","last_active":"2024-08-17T17:46:49.725525+0000","last_peered":"2024-08-17T17:46:49.725525+0000","last_clean":"2024-08-17T17:46:49.725525+0000","last_became_active":"2024-08-17T17:33:44.145668+0000","last_became_peered":"2024-08-17T17:33:44.145668+0000","last_unstale":"2024-08-17T17:46:49.725525+0000","last_undegraded":"2024-08-17T17:46:49.725525+0000","last_fullsized":"2024-08-17T17:46:49.725525+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:53:41.679237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.14","version":"90'149","reported_seq":852,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.723903+0000","last_change":"2024-08-17T17:41:37.664471+0000","last_active":"2024-08-17T17:46:49.723903+0000","last_peered":"2024-08-17T17:46:49.723903+0000","last_clean":"2024-08-17T17:46:49.723903+0000","last_became_active":"2024-08-17T17:41:37.664108+0000","last_became_peered":"2024-08-17T17:41:37.664108+0000","last_unstale":"2024-08-17T17:46:49.723903+0000","last_undegraded":"2024-08-17T17:46:49.723903+0000","last_fullsized":"2024-08-17T17:46:49.723903+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:22.030514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:22.030514+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:15:52.325163+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1d","version":"401'9","reported_seq":25,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:52.130056+0000","last_change":"2024-08-17T17:46:48.730468+0000","last_active":"2024-08-17T17:46:52.130056+0000","last_peered":"2024-08-17T17:46:52.130056+0000","last_clean":"2024-08-17T17:46:52.130056+0000","last_became_active":"2024-08-17T17:46:48.730180+0000","last_became_peered":"2024-08-17T17:46:48.730180+0000","last_unstale":"2024-08-17T17:46:52.130056+0000","last_undegraded":"2024-08-17T17:46:52.130056+0000","last_fullsized":"2024-08-17T17:46:52.130056+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:35:57.207252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":13,"num_write":11,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":400,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.729067+0000","last_change":"2024-08-17T17:41:34.634301+0000","last_active":"2024-08-17T17:46:48.729067+0000","last_peered":"2024-08-17T17:46:48.729067+0000","last_clean":"2024-08-17T17:46:48.729067+0000","last_became_active":"2024-08-17T17:41:34.620508+0000","last_became_peered":"2024-08-17T17:41:34.620508+0000","last_unstale":"2024-08-17T17:46:48.729067+0000","last_undegraded":"2024-08-17T17:46:48.729067+0000","last_fullsized":"2024-08-17T17:46:48.729067+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:10:54.576812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":627,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.723772+0000","last_change":"2024-08-17T17:33:20.799402+0000","last_active":"2024-08-17T17:46:49.723772+0000","last_peered":"2024-08-17T17:46:49.723772+0000","last_clean":"2024-08-17T17:46:49.723772+0000","last_became_active":"2024-08-17T17:33:20.798832+0000","last_became_peered":"2024-08-17T17:33:20.798832+0000","last_unstale":"2024-08-17T17:46:49.723772+0000","last_undegraded":"2024-08-17T17:46:49.723772+0000","last_fullsized":"2024-08-17T17:46:49.723772+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:21:05.001134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"90'149","reported_seq":498,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.721611+0000","last_change":"2024-08-17T17:42:06.699082+0000","last_active":"2024-08-17T17:46:49.721611+0000","last_peered":"2024-08-17T17:46:49.721611+0000","last_clean":"2024-08-17T17:46:49.721611+0000","last_became_active":"2024-08-17T17:42:05.876218+0000","last_became_peered":"2024-08-17T17:42:05.876218+0000","last_unstale":"2024-08-17T17:46:49.721611+0000","last_undegraded":"2024-08-17T17:46:49.721611+0000","last_fullsized":"2024-08-17T17:46:49.721611+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.698998+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:06.698998+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.698998+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:22:16.510150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":9,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.737822+0000","last_change":"2024-08-17T17:46:48.737822+0000","last_active":"2024-08-17T17:46:48.737822+0000","last_peered":"2024-08-17T17:46:48.737822+0000","last_clean":"2024-08-17T17:46:48.737822+0000","last_became_active":"2024-08-17T17:46:48.737489+0000","last_became_peered":"2024-08-17T17:46:48.737489+0000","last_unstale":"2024-08-17T17:46:48.737822+0000","last_undegraded":"2024-08-17T17:46:48.737822+0000","last_fullsized":"2024-08-17T17:46:48.737822+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:43:05.585227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":74,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.714961+0000","last_change":"2024-08-17T17:41:34.626584+0000","last_active":"2024-08-17T17:46:47.714961+0000","last_peered":"2024-08-17T17:46:47.714961+0000","last_clean":"2024-08-17T17:46:47.714961+0000","last_became_active":"2024-08-17T17:41:34.609878+0000","last_became_peered":"2024-08-17T17:41:34.609878+0000","last_unstale":"2024-08-17T17:46:47.714961+0000","last_undegraded":"2024-08-17T17:46:47.714961+0000","last_fullsized":"2024-08-17T17:46:47.714961+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:08:12.810347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":64,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.715331+0000","last_change":"2024-08-17T17:41:34.622180+0000","last_active":"2024-08-17T17:46:47.715331+0000","last_peered":"2024-08-17T17:46:47.715331+0000","last_clean":"2024-08-17T17:46:47.715331+0000","last_became_active":"2024-08-17T17:41:34.621724+0000","last_became_peered":"2024-08-17T17:41:34.621724+0000","last_unstale":"2024-08-17T17:46:47.715331+0000","last_undegraded":"2024-08-17T17:46:47.715331+0000","last_fullsized":"2024-08-17T17:46:47.715331+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:15.807235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"90'149","reported_seq":739,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.725291+0000","last_change":"2024-08-17T17:42:07.681204+0000","last_active":"2024-08-17T17:46:49.725291+0000","last_peered":"2024-08-17T17:46:49.725291+0000","last_clean":"2024-08-17T17:46:49.725291+0000","last_became_active":"2024-08-17T17:42:05.878769+0000","last_became_peered":"2024-08-17T17:42:05.878769+0000","last_unstale":"2024-08-17T17:46:49.725291+0000","last_undegraded":"2024-08-17T17:46:49.725291+0000","last_fullsized":"2024-08-17T17:46:49.725291+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:07.681064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:07.681064+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:12:12.835715+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":64,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.714583+0000","last_change":"2024-08-17T17:41:34.614028+0000","last_active":"2024-08-17T17:46:47.714583+0000","last_peered":"2024-08-17T17:46:47.714583+0000","last_clean":"2024-08-17T17:46:47.714583+0000","last_became_active":"2024-08-17T17:41:34.613706+0000","last_became_peered":"2024-08-17T17:41:34.613706+0000","last_unstale":"2024-08-17T17:46:47.714583+0000","last_undegraded":"2024-08-17T17:46:47.714583+0000","last_fullsized":"2024-08-17T17:46:47.714583+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:19:55.254902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":630,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.717745+0000","last_change":"2024-08-17T17:26:06.855307+0000","last_active":"2024-08-17T17:46:47.717745+0000","last_peered":"2024-08-17T17:46:47.717745+0000","last_clean":"2024-08-17T17:46:47.717745+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:46:47.717745+0000","last_undegraded":"2024-08-17T17:46:47.717745+0000","last_fullsized":"2024-08-17T17:46:47.717745+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":522,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.723037+0000","last_change":"2024-08-17T17:41:34.629522+0000","last_active":"2024-08-17T17:46:49.723037+0000","last_peered":"2024-08-17T17:46:49.723037+0000","last_clean":"2024-08-17T17:46:49.723037+0000","last_became_active":"2024-08-17T17:41:34.619340+0000","last_became_peered":"2024-08-17T17:41:34.619340+0000","last_unstale":"2024-08-17T17:46:49.723037+0000","last_undegraded":"2024-08-17T17:46:49.723037+0000","last_fullsized":"2024-08-17T17:46:49.723037+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:30:03.902956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":827,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.726832+0000","last_change":"2024-08-17T17:38:23.113397+0000","last_active":"2024-08-17T17:46:48.726832+0000","last_peered":"2024-08-17T17:46:48.726832+0000","last_clean":"2024-08-17T17:46:48.726832+0000","last_became_active":"2024-08-17T17:38:21.305580+0000","last_became_peered":"2024-08-17T17:38:21.305580+0000","last_unstale":"2024-08-17T17:46:48.726832+0000","last_undegraded":"2024-08-17T17:46:48.726832+0000","last_fullsized":"2024-08-17T17:46:48.726832+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.113279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.113279+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:38:07.015721+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":9,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.739518+0000","last_change":"2024-08-17T17:46:48.739518+0000","last_active":"2024-08-17T17:46:48.739518+0000","last_peered":"2024-08-17T17:46:48.739518+0000","last_clean":"2024-08-17T17:46:48.739518+0000","last_became_active":"2024-08-17T17:46:48.739232+0000","last_became_peered":"2024-08-17T17:46:48.739232+0000","last_unstale":"2024-08-17T17:46:48.739518+0000","last_undegraded":"2024-08-17T17:46:48.739518+0000","last_fullsized":"2024-08-17T17:46:48.739518+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:11:58.410232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":680,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.725774+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:46:48.725774+0000","last_peered":"2024-08-17T17:46:48.725774+0000","last_clean":"2024-08-17T17:46:48.725774+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:46:48.725774+0000","last_undegraded":"2024-08-17T17:46:48.725774+0000","last_fullsized":"2024-08-17T17:46:48.725774+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":1260,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.725518+0000","last_change":"2024-08-17T17:41:37.677524+0000","last_active":"2024-08-17T17:46:49.725518+0000","last_peered":"2024-08-17T17:46:49.725518+0000","last_clean":"2024-08-17T17:46:49.725518+0000","last_became_active":"2024-08-17T17:41:37.676764+0000","last_became_peered":"2024-08-17T17:41:37.676764+0000","last_unstale":"2024-08-17T17:46:49.725518+0000","last_undegraded":"2024-08-17T17:46:49.725518+0000","last_fullsized":"2024-08-17T17:46:49.725518+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:14:49.341080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":683,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.731653+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:46:48.731653+0000","last_peered":"2024-08-17T17:46:48.731653+0000","last_clean":"2024-08-17T17:46:48.731653+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:46:48.731653+0000","last_undegraded":"2024-08-17T17:46:48.731653+0000","last_fullsized":"2024-08-17T17:46:48.731653+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":783,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.714985+0000","last_change":"2024-08-17T17:42:19.960161+0000","last_active":"2024-08-17T17:46:47.714985+0000","last_peered":"2024-08-17T17:46:47.714985+0000","last_clean":"2024-08-17T17:46:47.714985+0000","last_became_active":"2024-08-17T17:41:37.665223+0000","last_became_peered":"2024-08-17T17:41:37.665223+0000","last_unstale":"2024-08-17T17:46:47.714985+0000","last_undegraded":"2024-08-17T17:46:47.714985+0000","last_fullsized":"2024-08-17T17:46:47.714985+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:19.960086+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:19.960086+0000","last_clean_scrub_stamp":"2024-08-17T17:42:19.960086+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:57.861661+0000","scrub_duration":0.0022612000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.f","version":"400'10","reported_seq":29,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.725142+0000","last_change":"2024-08-17T17:46:48.734909+0000","last_active":"2024-08-17T17:46:49.725142+0000","last_peered":"2024-08-17T17:46:49.725142+0000","last_clean":"2024-08-17T17:46:49.725142+0000","last_became_active":"2024-08-17T17:46:48.734547+0000","last_became_peered":"2024-08-17T17:46:48.734547+0000","last_unstale":"2024-08-17T17:46:49.725142+0000","last_undegraded":"2024-08-17T17:46:49.725142+0000","last_fullsized":"2024-08-17T17:46:49.725142+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:41:54.804454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":8199,"num_write":8,"num_write_kb":4100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":650,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.722527+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:46:49.722527+0000","last_peered":"2024-08-17T17:46:49.722527+0000","last_clean":"2024-08-17T17:46:49.722527+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:46:49.722527+0000","last_undegraded":"2024-08-17T17:46:49.722527+0000","last_fullsized":"2024-08-17T17:46:49.722527+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":960,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.709080+0000","last_change":"2024-08-17T17:41:37.683212+0000","last_active":"2024-08-17T17:46:47.709080+0000","last_peered":"2024-08-17T17:46:47.709080+0000","last_clean":"2024-08-17T17:46:47.709080+0000","last_became_active":"2024-08-17T17:41:37.682730+0000","last_became_peered":"2024-08-17T17:41:37.682730+0000","last_unstale":"2024-08-17T17:46:47.709080+0000","last_undegraded":"2024-08-17T17:46:47.709080+0000","last_fullsized":"2024-08-17T17:46:47.709080+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:51:35.444433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":635,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.724899+0000","last_change":"2024-08-17T17:33:20.798613+0000","last_active":"2024-08-17T17:46:49.724899+0000","last_peered":"2024-08-17T17:46:49.724899+0000","last_clean":"2024-08-17T17:46:49.724899+0000","last_became_active":"2024-08-17T17:33:20.798260+0000","last_became_peered":"2024-08-17T17:33:20.798260+0000","last_unstale":"2024-08-17T17:46:49.724899+0000","last_undegraded":"2024-08-17T17:46:49.724899+0000","last_fullsized":"2024-08-17T17:46:49.724899+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:18:00.822681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":537,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.716581+0000","last_change":"2024-08-17T17:41:44.058291+0000","last_active":"2024-08-17T17:46:47.716581+0000","last_peered":"2024-08-17T17:46:47.716581+0000","last_clean":"2024-08-17T17:46:47.716581+0000","last_became_active":"2024-08-17T17:41:12.200077+0000","last_became_peered":"2024-08-17T17:41:12.200077+0000","last_unstale":"2024-08-17T17:46:47.716581+0000","last_undegraded":"2024-08-17T17:46:47.716581+0000","last_fullsized":"2024-08-17T17:46:47.716581+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:44.058233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:44.058233+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:42:55.911623+0000","scrub_duration":0.0074268499999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":14,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.724997+0000","last_change":"2024-08-17T17:46:48.740122+0000","last_active":"2024-08-17T17:46:49.724997+0000","last_peered":"2024-08-17T17:46:49.724997+0000","last_clean":"2024-08-17T17:46:49.724997+0000","last_became_active":"2024-08-17T17:46:48.739733+0000","last_became_peered":"2024-08-17T17:46:48.739733+0000","last_unstale":"2024-08-17T17:46:49.724997+0000","last_undegraded":"2024-08-17T17:46:49.724997+0000","last_fullsized":"2024-08-17T17:46:49.724997+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:23:32.710611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":631,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.724056+0000","last_change":"2024-08-17T17:41:34.627064+0000","last_active":"2024-08-17T17:46:49.724056+0000","last_peered":"2024-08-17T17:46:49.724056+0000","last_clean":"2024-08-17T17:46:49.724056+0000","last_became_active":"2024-08-17T17:41:34.626453+0000","last_became_peered":"2024-08-17T17:41:34.626453+0000","last_unstale":"2024-08-17T17:46:49.724056+0000","last_undegraded":"2024-08-17T17:46:49.724056+0000","last_fullsized":"2024-08-17T17:46:49.724056+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:02:11.894042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":815,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.725010+0000","last_change":"2024-08-17T17:33:20.799350+0000","last_active":"2024-08-17T17:46:49.725010+0000","last_peered":"2024-08-17T17:46:49.725010+0000","last_clean":"2024-08-17T17:46:49.725010+0000","last_became_active":"2024-08-17T17:33:20.798925+0000","last_became_peered":"2024-08-17T17:33:20.798925+0000","last_unstale":"2024-08-17T17:46:49.725010+0000","last_undegraded":"2024-08-17T17:46:49.725010+0000","last_fullsized":"2024-08-17T17:46:49.725010+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:16:36.121295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":349,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.729171+0000","last_change":"2024-08-17T17:33:44.163867+0000","last_active":"2024-08-17T17:46:48.729171+0000","last_peered":"2024-08-17T17:46:48.729171+0000","last_clean":"2024-08-17T17:46:48.729171+0000","last_became_active":"2024-08-17T17:33:44.163503+0000","last_became_peered":"2024-08-17T17:33:44.163503+0000","last_unstale":"2024-08-17T17:46:48.729171+0000","last_undegraded":"2024-08-17T17:46:48.729171+0000","last_fullsized":"2024-08-17T17:46:48.729171+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:31:53.573596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":537,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.717646+0000","last_change":"2024-08-17T17:41:43.035215+0000","last_active":"2024-08-17T17:46:47.717646+0000","last_peered":"2024-08-17T17:46:47.717646+0000","last_clean":"2024-08-17T17:46:47.717646+0000","last_became_active":"2024-08-17T17:41:12.201158+0000","last_became_peered":"2024-08-17T17:41:12.201158+0000","last_unstale":"2024-08-17T17:46:47.717646+0000","last_undegraded":"2024-08-17T17:46:47.717646+0000","last_fullsized":"2024-08-17T17:46:47.717646+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:43.035108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:43.035108+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:43:27.108994+0000","scrub_duration":0.0028877400000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"400'30","reported_seq":69,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.724559+0000","last_change":"2024-08-17T17:46:48.736149+0000","last_active":"2024-08-17T17:46:49.724559+0000","last_peered":"2024-08-17T17:46:49.724559+0000","last_clean":"2024-08-17T17:46:49.724559+0000","last_became_active":"2024-08-17T17:46:48.735855+0000","last_became_peered":"2024-08-17T17:46:48.735855+0000","last_unstale":"2024-08-17T17:46:49.724559+0000","last_undegraded":"2024-08-17T17:46:49.724559+0000","last_fullsized":"2024-08-17T17:46:49.724559+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:11:31.979905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":139,"num_read_kb":104,"num_write":29,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":600,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.725169+0000","last_change":"2024-08-17T17:33:20.772454+0000","last_active":"2024-08-17T17:46:49.725169+0000","last_peered":"2024-08-17T17:46:49.725169+0000","last_clean":"2024-08-17T17:46:49.725169+0000","last_became_active":"2024-08-17T17:33:20.771848+0000","last_became_peered":"2024-08-17T17:33:20.771848+0000","last_unstale":"2024-08-17T17:46:49.725169+0000","last_undegraded":"2024-08-17T17:46:49.725169+0000","last_fullsized":"2024-08-17T17:46:49.725169+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:46:42.300744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":64,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.715330+0000","last_change":"2024-08-17T17:41:34.611109+0000","last_active":"2024-08-17T17:46:47.715330+0000","last_peered":"2024-08-17T17:46:47.715330+0000","last_clean":"2024-08-17T17:46:47.715330+0000","last_became_active":"2024-08-17T17:41:34.610170+0000","last_became_peered":"2024-08-17T17:41:34.610170+0000","last_unstale":"2024-08-17T17:46:47.715330+0000","last_undegraded":"2024-08-17T17:46:47.715330+0000","last_fullsized":"2024-08-17T17:46:47.715330+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:39:05.301260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":1145,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.723670+0000","last_change":"2024-08-17T17:33:44.148227+0000","last_active":"2024-08-17T17:46:49.723670+0000","last_peered":"2024-08-17T17:46:49.723670+0000","last_clean":"2024-08-17T17:46:49.723670+0000","last_became_active":"2024-08-17T17:33:44.147627+0000","last_became_peered":"2024-08-17T17:33:44.147627+0000","last_unstale":"2024-08-17T17:46:49.723670+0000","last_undegraded":"2024-08-17T17:46:49.723670+0000","last_fullsized":"2024-08-17T17:46:49.723670+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:29:42.898460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1372,"num_read_kb":2843,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":847,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.721322+0000","last_change":"2024-08-17T17:38:22.254841+0000","last_active":"2024-08-17T17:46:49.721322+0000","last_peered":"2024-08-17T17:46:49.721322+0000","last_clean":"2024-08-17T17:46:49.721322+0000","last_became_active":"2024-08-17T17:38:21.306619+0000","last_became_peered":"2024-08-17T17:38:21.306619+0000","last_unstale":"2024-08-17T17:46:49.721322+0000","last_undegraded":"2024-08-17T17:46:49.721322+0000","last_fullsized":"2024-08-17T17:46:49.721322+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:22.254705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:22.254705+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:47:38.184089+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"400'37","reported_seq":73,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.212060+0000","last_change":"2024-08-17T17:46:48.742307+0000","last_active":"2024-08-17T17:46:49.212060+0000","last_peered":"2024-08-17T17:46:49.212060+0000","last_clean":"2024-08-17T17:46:49.212060+0000","last_became_active":"2024-08-17T17:46:48.741808+0000","last_became_peered":"2024-08-17T17:46:48.741808+0000","last_unstale":"2024-08-17T17:46:49.212060+0000","last_undegraded":"2024-08-17T17:46:49.212060+0000","last_fullsized":"2024-08-17T17:46:49.212060+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:14:05.984480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":215,"num_read_kb":174,"num_write":44,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":596,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.725337+0000","last_change":"2024-08-17T17:33:20.835784+0000","last_active":"2024-08-17T17:46:49.725337+0000","last_peered":"2024-08-17T17:46:49.725337+0000","last_clean":"2024-08-17T17:46:49.725337+0000","last_became_active":"2024-08-17T17:33:20.835371+0000","last_became_peered":"2024-08-17T17:33:20.835371+0000","last_unstale":"2024-08-17T17:46:49.725337+0000","last_undegraded":"2024-08-17T17:46:49.725337+0000","last_fullsized":"2024-08-17T17:46:49.725337+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:35:43.369055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":560,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.723594+0000","last_change":"2024-08-17T17:33:20.793468+0000","last_active":"2024-08-17T17:46:49.723594+0000","last_peered":"2024-08-17T17:46:49.723594+0000","last_clean":"2024-08-17T17:46:49.723594+0000","last_became_active":"2024-08-17T17:33:20.793166+0000","last_became_peered":"2024-08-17T17:33:20.793166+0000","last_unstale":"2024-08-17T17:46:49.723594+0000","last_undegraded":"2024-08-17T17:46:49.723594+0000","last_fullsized":"2024-08-17T17:46:49.723594+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:12:52.121050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":1053,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.727775+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:46:48.727775+0000","last_peered":"2024-08-17T17:46:48.727775+0000","last_clean":"2024-08-17T17:46:48.727775+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:46:48.727775+0000","last_undegraded":"2024-08-17T17:46:48.727775+0000","last_fullsized":"2024-08-17T17:46:48.727775+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":868,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.720798+0000","last_change":"2024-08-17T17:41:37.683450+0000","last_active":"2024-08-17T17:46:49.720798+0000","last_peered":"2024-08-17T17:46:49.720798+0000","last_clean":"2024-08-17T17:46:49.720798+0000","last_became_active":"2024-08-17T17:41:37.683103+0000","last_became_peered":"2024-08-17T17:41:37.683103+0000","last_unstale":"2024-08-17T17:46:49.720798+0000","last_undegraded":"2024-08-17T17:46:49.720798+0000","last_fullsized":"2024-08-17T17:46:49.720798+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.289528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.289528+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:39:17.141469+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.8","version":"400'12","reported_seq":28,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.038295+0000","last_change":"2024-08-17T17:46:48.842712+0000","last_active":"2024-08-17T17:46:49.038295+0000","last_peered":"2024-08-17T17:46:49.038295+0000","last_clean":"2024-08-17T17:46:49.038295+0000","last_became_active":"2024-08-17T17:46:48.842397+0000","last_became_peered":"2024-08-17T17:46:48.842397+0000","last_unstale":"2024-08-17T17:46:49.038295+0000","last_undegraded":"2024-08-17T17:46:49.038295+0000","last_fullsized":"2024-08-17T17:46:49.038295+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:47:20.063122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":18,"num_write":9,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":476,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.723132+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:46:49.723132+0000","last_peered":"2024-08-17T17:46:49.723132+0000","last_clean":"2024-08-17T17:46:49.723132+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:46:49.723132+0000","last_undegraded":"2024-08-17T17:46:49.723132+0000","last_fullsized":"2024-08-17T17:46:49.723132+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":688,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.721415+0000","last_change":"2024-08-17T17:41:34.614108+0000","last_active":"2024-08-17T17:46:49.721415+0000","last_peered":"2024-08-17T17:46:49.721415+0000","last_clean":"2024-08-17T17:46:49.721415+0000","last_became_active":"2024-08-17T17:41:34.613481+0000","last_became_peered":"2024-08-17T17:41:34.613481+0000","last_unstale":"2024-08-17T17:46:49.721415+0000","last_undegraded":"2024-08-17T17:46:49.721415+0000","last_fullsized":"2024-08-17T17:46:49.721415+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:21:02.346115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":922,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.723569+0000","last_change":"2024-08-17T17:33:20.797705+0000","last_active":"2024-08-17T17:46:49.723569+0000","last_peered":"2024-08-17T17:46:49.723569+0000","last_clean":"2024-08-17T17:46:49.723569+0000","last_became_active":"2024-08-17T17:33:20.797172+0000","last_became_peered":"2024-08-17T17:33:20.797172+0000","last_unstale":"2024-08-17T17:46:49.723569+0000","last_undegraded":"2024-08-17T17:46:49.723569+0000","last_fullsized":"2024-08-17T17:46:49.723569+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:30.754999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":864,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.724771+0000","last_change":"2024-08-17T17:41:37.676347+0000","last_active":"2024-08-17T17:46:49.724771+0000","last_peered":"2024-08-17T17:46:49.724771+0000","last_clean":"2024-08-17T17:46:49.724771+0000","last_became_active":"2024-08-17T17:41:37.675869+0000","last_became_peered":"2024-08-17T17:41:37.675869+0000","last_unstale":"2024-08-17T17:46:49.724771+0000","last_undegraded":"2024-08-17T17:46:49.724771+0000","last_fullsized":"2024-08-17T17:46:49.724771+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:21.814077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:21.814077+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:04:16.240754+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"400'4","reported_seq":18,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.231127+0000","last_change":"2024-08-17T17:46:48.738158+0000","last_active":"2024-08-17T17:46:49.231127+0000","last_peered":"2024-08-17T17:46:49.231127+0000","last_clean":"2024-08-17T17:46:49.231127+0000","last_became_active":"2024-08-17T17:46:48.737858+0000","last_became_peered":"2024-08-17T17:46:48.737858+0000","last_unstale":"2024-08-17T17:46:49.231127+0000","last_undegraded":"2024-08-17T17:46:49.231127+0000","last_fullsized":"2024-08-17T17:46:49.231127+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:34:39.600704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":12,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":241,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.716519+0000","last_change":"2024-08-17T17:41:34.618631+0000","last_active":"2024-08-17T17:46:47.716519+0000","last_peered":"2024-08-17T17:46:47.716519+0000","last_clean":"2024-08-17T17:46:47.716519+0000","last_became_active":"2024-08-17T17:41:34.618313+0000","last_became_peered":"2024-08-17T17:41:34.618313+0000","last_unstale":"2024-08-17T17:46:47.716519+0000","last_undegraded":"2024-08-17T17:46:47.716519+0000","last_fullsized":"2024-08-17T17:46:47.716519+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:07:13.915088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":637,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.732104+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:46:48.732104+0000","last_peered":"2024-08-17T17:46:48.732104+0000","last_clean":"2024-08-17T17:46:48.732104+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:46:48.732104+0000","last_undegraded":"2024-08-17T17:46:48.732104+0000","last_fullsized":"2024-08-17T17:46:48.732104+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":806,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.708453+0000","last_change":"2024-08-17T17:33:44.149928+0000","last_active":"2024-08-17T17:46:47.708453+0000","last_peered":"2024-08-17T17:46:47.708453+0000","last_clean":"2024-08-17T17:46:47.708453+0000","last_became_active":"2024-08-17T17:33:44.149498+0000","last_became_peered":"2024-08-17T17:33:44.149498+0000","last_unstale":"2024-08-17T17:46:47.708453+0000","last_undegraded":"2024-08-17T17:46:47.708453+0000","last_fullsized":"2024-08-17T17:46:47.708453+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:17:04.963202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":828,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.720847+0000","last_change":"2024-08-17T17:38:25.247159+0000","last_active":"2024-08-17T17:46:49.720847+0000","last_peered":"2024-08-17T17:46:49.720847+0000","last_clean":"2024-08-17T17:46:49.720847+0000","last_became_active":"2024-08-17T17:38:21.311361+0000","last_became_peered":"2024-08-17T17:38:21.311361+0000","last_unstale":"2024-08-17T17:46:49.720847+0000","last_undegraded":"2024-08-17T17:46:49.720847+0000","last_fullsized":"2024-08-17T17:46:49.720847+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:25.247027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:25.247027+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:46:21.734712+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.a","version":"400'14","reported_seq":31,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.067689+0000","last_change":"2024-08-17T17:46:48.842318+0000","last_active":"2024-08-17T17:46:49.067689+0000","last_peered":"2024-08-17T17:46:49.067689+0000","last_clean":"2024-08-17T17:46:49.067689+0000","last_became_active":"2024-08-17T17:46:48.842047+0000","last_became_peered":"2024-08-17T17:46:48.842047+0000","last_unstale":"2024-08-17T17:46:49.067689+0000","last_undegraded":"2024-08-17T17:46:49.067689+0000","last_fullsized":"2024-08-17T17:46:49.067689+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:36:16.733698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":586,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.725456+0000","last_change":"2024-08-17T17:41:34.625551+0000","last_active":"2024-08-17T17:46:49.725456+0000","last_peered":"2024-08-17T17:46:49.725456+0000","last_clean":"2024-08-17T17:46:49.725456+0000","last_became_active":"2024-08-17T17:41:34.625251+0000","last_became_peered":"2024-08-17T17:41:34.625251+0000","last_unstale":"2024-08-17T17:46:49.725456+0000","last_undegraded":"2024-08-17T17:46:49.725456+0000","last_fullsized":"2024-08-17T17:46:49.725456+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:28:22.354213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":689,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.725970+0000","last_change":"2024-08-17T17:33:20.796216+0000","last_active":"2024-08-17T17:46:48.725970+0000","last_peered":"2024-08-17T17:46:48.725970+0000","last_clean":"2024-08-17T17:46:48.725970+0000","last_became_active":"2024-08-17T17:33:20.795762+0000","last_became_peered":"2024-08-17T17:33:20.795762+0000","last_unstale":"2024-08-17T17:46:48.725970+0000","last_undegraded":"2024-08-17T17:46:48.725970+0000","last_fullsized":"2024-08-17T17:46:48.725970+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:47.038509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":517,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.715397+0000","last_change":"2024-08-17T17:41:41.036615+0000","last_active":"2024-08-17T17:46:47.715397+0000","last_peered":"2024-08-17T17:46:47.715397+0000","last_clean":"2024-08-17T17:46:47.715397+0000","last_became_active":"2024-08-17T17:41:12.329214+0000","last_became_peered":"2024-08-17T17:41:12.329214+0000","last_unstale":"2024-08-17T17:46:47.715397+0000","last_undegraded":"2024-08-17T17:46:47.715397+0000","last_fullsized":"2024-08-17T17:46:47.715397+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:41.036524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:41.036524+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:59:58.695900+0000","scrub_duration":0.001820803,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"400'3","reported_seq":19,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.723843+0000","last_change":"2024-08-17T17:46:48.738469+0000","last_active":"2024-08-17T17:46:49.723843+0000","last_peered":"2024-08-17T17:46:49.723843+0000","last_clean":"2024-08-17T17:46:49.723843+0000","last_became_active":"2024-08-17T17:46:48.738139+0000","last_became_peered":"2024-08-17T17:46:48.738139+0000","last_unstale":"2024-08-17T17:46:49.723843+0000","last_undegraded":"2024-08-17T17:46:49.723843+0000","last_fullsized":"2024-08-17T17:46:49.723843+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:38:44.425734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":64,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.715662+0000","last_change":"2024-08-17T17:41:34.622102+0000","last_active":"2024-08-17T17:46:47.715662+0000","last_peered":"2024-08-17T17:46:47.715662+0000","last_clean":"2024-08-17T17:46:47.715662+0000","last_became_active":"2024-08-17T17:41:34.621733+0000","last_became_peered":"2024-08-17T17:41:34.621733+0000","last_unstale":"2024-08-17T17:46:47.715662+0000","last_undegraded":"2024-08-17T17:46:47.715662+0000","last_fullsized":"2024-08-17T17:46:47.715662+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:39:41.668709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":64,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.714801+0000","last_change":"2024-08-17T17:41:34.607602+0000","last_active":"2024-08-17T17:46:47.714801+0000","last_peered":"2024-08-17T17:46:47.714801+0000","last_clean":"2024-08-17T17:46:47.714801+0000","last_became_active":"2024-08-17T17:41:34.607300+0000","last_became_peered":"2024-08-17T17:41:34.607300+0000","last_unstale":"2024-08-17T17:46:47.714801+0000","last_undegraded":"2024-08-17T17:46:47.714801+0000","last_fullsized":"2024-08-17T17:46:47.714801+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:20:31.486698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"90'149","reported_seq":881,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.724627+0000","last_change":"2024-08-17T17:41:41.743931+0000","last_active":"2024-08-17T17:46:49.724627+0000","last_peered":"2024-08-17T17:46:49.724627+0000","last_clean":"2024-08-17T17:46:49.724627+0000","last_became_active":"2024-08-17T17:41:12.203771+0000","last_became_peered":"2024-08-17T17:41:12.203771+0000","last_unstale":"2024-08-17T17:46:49.724627+0000","last_undegraded":"2024-08-17T17:46:49.724627+0000","last_fullsized":"2024-08-17T17:46:49.724627+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:41.743570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:41.743570+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:21:03.865738+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"400'15","reported_seq":24,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.019050+0000","last_change":"2024-08-17T17:46:48.738131+0000","last_active":"2024-08-17T17:46:49.019050+0000","last_peered":"2024-08-17T17:46:49.019050+0000","last_clean":"2024-08-17T17:46:49.019050+0000","last_became_active":"2024-08-17T17:46:48.737849+0000","last_became_peered":"2024-08-17T17:46:48.737849+0000","last_unstale":"2024-08-17T17:46:49.019050+0000","last_undegraded":"2024-08-17T17:46:49.019050+0000","last_fullsized":"2024-08-17T17:46:49.019050+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:55:09.265804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":64,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.715327+0000","last_change":"2024-08-17T17:41:34.619268+0000","last_active":"2024-08-17T17:46:47.715327+0000","last_peered":"2024-08-17T17:46:47.715327+0000","last_clean":"2024-08-17T17:46:47.715327+0000","last_became_active":"2024-08-17T17:41:34.618914+0000","last_became_peered":"2024-08-17T17:41:34.618914+0000","last_unstale":"2024-08-17T17:46:47.715327+0000","last_undegraded":"2024-08-17T17:46:47.715327+0000","last_fullsized":"2024-08-17T17:46:47.715327+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:08:54.248386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":64,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.714647+0000","last_change":"2024-08-17T17:41:34.622803+0000","last_active":"2024-08-17T17:46:47.714647+0000","last_peered":"2024-08-17T17:46:47.714647+0000","last_clean":"2024-08-17T17:46:47.714647+0000","last_became_active":"2024-08-17T17:41:34.622525+0000","last_became_peered":"2024-08-17T17:41:34.622525+0000","last_unstale":"2024-08-17T17:46:47.714647+0000","last_undegraded":"2024-08-17T17:46:47.714647+0000","last_fullsized":"2024-08-17T17:46:47.714647+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:42:39.728377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":874,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.725603+0000","last_change":"2024-08-17T17:43:55.595647+0000","last_active":"2024-08-17T17:46:48.725603+0000","last_peered":"2024-08-17T17:46:48.725603+0000","last_clean":"2024-08-17T17:46:48.725603+0000","last_became_active":"2024-08-17T17:43:55.595256+0000","last_became_peered":"2024-08-17T17:43:55.595256+0000","last_unstale":"2024-08-17T17:46:48.725603+0000","last_undegraded":"2024-08-17T17:46:48.725603+0000","last_fullsized":"2024-08-17T17:46:48.725603+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:46.587712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:46.587712+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:05:01.927968+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.2","version":"400'1","reported_seq":13,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.778840+0000","last_change":"2024-08-17T17:46:48.739640+0000","last_active":"2024-08-17T17:46:48.778840+0000","last_peered":"2024-08-17T17:46:48.778840+0000","last_clean":"2024-08-17T17:46:48.778840+0000","last_became_active":"2024-08-17T17:46:48.739266+0000","last_became_peered":"2024-08-17T17:46:48.739266+0000","last_unstale":"2024-08-17T17:46:48.778840+0000","last_undegraded":"2024-08-17T17:46:48.778840+0000","last_fullsized":"2024-08-17T17:46:48.778840+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:43:13.205068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":64,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.714742+0000","last_change":"2024-08-17T17:41:34.611049+0000","last_active":"2024-08-17T17:46:47.714742+0000","last_peered":"2024-08-17T17:46:47.714742+0000","last_clean":"2024-08-17T17:46:47.714742+0000","last_became_active":"2024-08-17T17:41:34.610727+0000","last_became_peered":"2024-08-17T17:41:34.610727+0000","last_unstale":"2024-08-17T17:46:47.714742+0000","last_undegraded":"2024-08-17T17:46:47.714742+0000","last_fullsized":"2024-08-17T17:46:47.714742+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:04:05.477660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":731,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.727019+0000","last_change":"2024-08-17T17:41:34.630434+0000","last_active":"2024-08-17T17:46:48.727019+0000","last_peered":"2024-08-17T17:46:48.727019+0000","last_clean":"2024-08-17T17:46:48.727019+0000","last_became_active":"2024-08-17T17:41:34.616406+0000","last_became_peered":"2024-08-17T17:41:34.616406+0000","last_unstale":"2024-08-17T17:46:48.727019+0000","last_undegraded":"2024-08-17T17:46:48.727019+0000","last_fullsized":"2024-08-17T17:46:48.727019+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:30:44.817203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":846,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.727209+0000","last_change":"2024-08-17T17:41:44.585386+0000","last_active":"2024-08-17T17:46:48.727209+0000","last_peered":"2024-08-17T17:46:48.727209+0000","last_clean":"2024-08-17T17:46:48.727209+0000","last_became_active":"2024-08-17T17:41:12.307398+0000","last_became_peered":"2024-08-17T17:41:12.307398+0000","last_unstale":"2024-08-17T17:46:48.727209+0000","last_undegraded":"2024-08-17T17:46:48.727209+0000","last_fullsized":"2024-08-17T17:46:48.727209+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:44.583653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:44.583653+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:27:44.756681+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":9,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.740042+0000","last_change":"2024-08-17T17:46:48.740042+0000","last_active":"2024-08-17T17:46:48.740042+0000","last_peered":"2024-08-17T17:46:48.740042+0000","last_clean":"2024-08-17T17:46:48.740042+0000","last_became_active":"2024-08-17T17:46:48.739329+0000","last_became_peered":"2024-08-17T17:46:48.739329+0000","last_unstale":"2024-08-17T17:46:48.740042+0000","last_undegraded":"2024-08-17T17:46:48.740042+0000","last_fullsized":"2024-08-17T17:46:48.740042+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:24:10.892858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":697,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.722615+0000","last_change":"2024-08-17T17:41:34.615197+0000","last_active":"2024-08-17T17:46:49.722615+0000","last_peered":"2024-08-17T17:46:49.722615+0000","last_clean":"2024-08-17T17:46:49.722615+0000","last_became_active":"2024-08-17T17:41:34.614873+0000","last_became_peered":"2024-08-17T17:41:34.614873+0000","last_unstale":"2024-08-17T17:46:49.722615+0000","last_undegraded":"2024-08-17T17:46:49.722615+0000","last_fullsized":"2024-08-17T17:46:49.722615+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:21:57.391953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":331,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.730049+0000","last_change":"2024-08-17T17:33:44.154097+0000","last_active":"2024-08-17T17:46:48.730049+0000","last_peered":"2024-08-17T17:46:48.730049+0000","last_clean":"2024-08-17T17:46:48.730049+0000","last_became_active":"2024-08-17T17:33:44.152972+0000","last_became_peered":"2024-08-17T17:33:44.152972+0000","last_unstale":"2024-08-17T17:46:48.730049+0000","last_undegraded":"2024-08-17T17:46:48.730049+0000","last_fullsized":"2024-08-17T17:46:48.730049+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:42.602017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"90'149","reported_seq":708,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.715650+0000","last_change":"2024-08-17T17:42:20.927007+0000","last_active":"2024-08-17T17:46:47.715650+0000","last_peered":"2024-08-17T17:46:47.715650+0000","last_clean":"2024-08-17T17:46:47.715650+0000","last_became_active":"2024-08-17T17:41:37.670914+0000","last_became_peered":"2024-08-17T17:41:37.670914+0000","last_unstale":"2024-08-17T17:46:47.715650+0000","last_undegraded":"2024-08-17T17:46:47.715650+0000","last_fullsized":"2024-08-17T17:46:47.715650+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:20.926924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:20.926924+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:57:44.156600+0000","scrub_duration":0.0018485109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.734087+0000","last_change":"2024-08-17T17:46:48.734087+0000","last_active":"2024-08-17T17:46:48.734087+0000","last_peered":"2024-08-17T17:46:48.734087+0000","last_clean":"2024-08-17T17:46:48.734087+0000","last_became_active":"2024-08-17T17:46:48.733813+0000","last_became_peered":"2024-08-17T17:46:48.733813+0000","last_unstale":"2024-08-17T17:46:48.734087+0000","last_undegraded":"2024-08-17T17:46:48.734087+0000","last_fullsized":"2024-08-17T17:46:48.734087+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:17:35.484086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":228,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.715597+0000","last_change":"2024-08-17T17:33:20.795582+0000","last_active":"2024-08-17T17:46:47.715597+0000","last_peered":"2024-08-17T17:46:47.715597+0000","last_clean":"2024-08-17T17:46:47.715597+0000","last_became_active":"2024-08-17T17:33:20.795149+0000","last_became_peered":"2024-08-17T17:33:20.795149+0000","last_unstale":"2024-08-17T17:46:47.715597+0000","last_undegraded":"2024-08-17T17:46:47.715597+0000","last_fullsized":"2024-08-17T17:46:47.715597+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:14.020923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":150,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.716625+0000","last_change":"2024-08-17T17:33:44.145399+0000","last_active":"2024-08-17T17:46:47.716625+0000","last_peered":"2024-08-17T17:46:47.716625+0000","last_clean":"2024-08-17T17:46:47.716625+0000","last_became_active":"2024-08-17T17:33:44.144751+0000","last_became_peered":"2024-08-17T17:33:44.144751+0000","last_unstale":"2024-08-17T17:46:47.716625+0000","last_undegraded":"2024-08-17T17:46:47.716625+0000","last_fullsized":"2024-08-17T17:46:47.716625+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:54.276350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"90'149","reported_seq":576,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.717667+0000","last_change":"2024-08-17T17:41:37.668050+0000","last_active":"2024-08-17T17:46:47.717667+0000","last_peered":"2024-08-17T17:46:47.717667+0000","last_clean":"2024-08-17T17:46:47.717667+0000","last_became_active":"2024-08-17T17:41:37.667655+0000","last_became_peered":"2024-08-17T17:41:37.667655+0000","last_unstale":"2024-08-17T17:46:47.717667+0000","last_undegraded":"2024-08-17T17:46:47.717667+0000","last_fullsized":"2024-08-17T17:46:47.717667+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:32.843802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:32.843802+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:39:46.766280+0000","scrub_duration":0.0040803669999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.843386+0000","last_change":"2024-08-17T17:46:48.843386+0000","last_active":"2024-08-17T17:46:48.843386+0000","last_peered":"2024-08-17T17:46:48.843386+0000","last_clean":"2024-08-17T17:46:48.843386+0000","last_became_active":"2024-08-17T17:46:48.842921+0000","last_became_peered":"2024-08-17T17:46:48.842921+0000","last_unstale":"2024-08-17T17:46:48.843386+0000","last_undegraded":"2024-08-17T17:46:48.843386+0000","last_fullsized":"2024-08-17T17:46:48.843386+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:21:40.951048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":64,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.714478+0000","last_change":"2024-08-17T17:41:34.621078+0000","last_active":"2024-08-17T17:46:47.714478+0000","last_peered":"2024-08-17T17:46:47.714478+0000","last_clean":"2024-08-17T17:46:47.714478+0000","last_became_active":"2024-08-17T17:41:34.620730+0000","last_became_peered":"2024-08-17T17:41:34.620730+0000","last_unstale":"2024-08-17T17:46:47.714478+0000","last_undegraded":"2024-08-17T17:46:47.714478+0000","last_fullsized":"2024-08-17T17:46:47.714478+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:55:01.312283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":357,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.709086+0000","last_change":"2024-08-17T17:41:34.637696+0000","last_active":"2024-08-17T17:46:47.709086+0000","last_peered":"2024-08-17T17:46:47.709086+0000","last_clean":"2024-08-17T17:46:47.709086+0000","last_became_active":"2024-08-17T17:41:34.621388+0000","last_became_peered":"2024-08-17T17:41:34.621388+0000","last_unstale":"2024-08-17T17:46:47.709086+0000","last_undegraded":"2024-08-17T17:46:47.709086+0000","last_fullsized":"2024-08-17T17:46:47.709086+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:13:04.607036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"90'149","reported_seq":871,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.726972+0000","last_change":"2024-08-17T17:41:42.603841+0000","last_active":"2024-08-17T17:46:48.726972+0000","last_peered":"2024-08-17T17:46:48.726972+0000","last_clean":"2024-08-17T17:46:48.726972+0000","last_became_active":"2024-08-17T17:41:12.203400+0000","last_became_peered":"2024-08-17T17:41:12.203400+0000","last_unstale":"2024-08-17T17:46:48.726972+0000","last_undegraded":"2024-08-17T17:46:48.726972+0000","last_fullsized":"2024-08-17T17:46:48.726972+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:42.603561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:42.603561+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:50:12.843193+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"401'2163","reported_seq":2234,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:52.065413+0000","last_change":"2024-08-17T17:46:48.728804+0000","last_active":"2024-08-17T17:46:52.065413+0000","last_peered":"2024-08-17T17:46:52.065413+0000","last_clean":"2024-08-17T17:46:52.065413+0000","last_became_active":"2024-08-17T17:46:48.728497+0000","last_became_peered":"2024-08-17T17:46:48.728497+0000","last_unstale":"2024-08-17T17:46:52.065413+0000","last_undegraded":"2024-08-17T17:46:52.065413+0000","last_fullsized":"2024-08-17T17:46:52.065413+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":2163,"log_dups_size":0,"ondisk_log_size":2163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:02:37.391205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8533,"num_read_kb":8607,"num_write":4187,"num_write_kb":4148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":154,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.714598+0000","last_change":"2024-08-17T17:33:44.145598+0000","last_active":"2024-08-17T17:46:47.714598+0000","last_peered":"2024-08-17T17:46:47.714598+0000","last_clean":"2024-08-17T17:46:47.714598+0000","last_became_active":"2024-08-17T17:33:44.145283+0000","last_became_peered":"2024-08-17T17:33:44.145283+0000","last_unstale":"2024-08-17T17:46:47.714598+0000","last_undegraded":"2024-08-17T17:46:47.714598+0000","last_fullsized":"2024-08-17T17:46:47.714598+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:11.965226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":361,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.709293+0000","last_change":"2024-08-17T17:41:34.635980+0000","last_active":"2024-08-17T17:46:47.709293+0000","last_peered":"2024-08-17T17:46:47.709293+0000","last_clean":"2024-08-17T17:46:47.709293+0000","last_became_active":"2024-08-17T17:41:34.620596+0000","last_became_peered":"2024-08-17T17:41:34.620596+0000","last_unstale":"2024-08-17T17:46:47.709293+0000","last_undegraded":"2024-08-17T17:46:47.709293+0000","last_fullsized":"2024-08-17T17:46:47.709293+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:19:15.792577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"90'149","reported_seq":810,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.708717+0000","last_change":"2024-08-17T17:42:16.610940+0000","last_active":"2024-08-17T17:46:47.708717+0000","last_peered":"2024-08-17T17:46:47.708717+0000","last_clean":"2024-08-17T17:46:47.708717+0000","last_became_active":"2024-08-17T17:41:37.670855+0000","last_became_peered":"2024-08-17T17:41:37.670855+0000","last_unstale":"2024-08-17T17:46:47.708717+0000","last_undegraded":"2024-08-17T17:46:47.708717+0000","last_fullsized":"2024-08-17T17:46:47.708717+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:16.610828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:16.610828+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:48:25.918610+0000","scrub_duration":0.086892487000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.7","version":"400'23","reported_seq":58,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.725353+0000","last_change":"2024-08-17T17:46:48.843934+0000","last_active":"2024-08-17T17:46:49.725353+0000","last_peered":"2024-08-17T17:46:49.725353+0000","last_clean":"2024-08-17T17:46:49.725353+0000","last_became_active":"2024-08-17T17:46:48.843609+0000","last_became_peered":"2024-08-17T17:46:48.843609+0000","last_unstale":"2024-08-17T17:46:49.725353+0000","last_undegraded":"2024-08-17T17:46:49.725353+0000","last_fullsized":"2024-08-17T17:46:49.725353+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:16:02.048308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":39,"num_write":17,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.734744+0000","last_change":"2024-08-17T17:46:48.734744+0000","last_active":"2024-08-17T17:46:48.734744+0000","last_peered":"2024-08-17T17:46:48.734744+0000","last_clean":"2024-08-17T17:46:48.734744+0000","last_became_active":"2024-08-17T17:46:48.734473+0000","last_became_peered":"2024-08-17T17:46:48.734473+0000","last_unstale":"2024-08-17T17:46:48.734744+0000","last_undegraded":"2024-08-17T17:46:48.734744+0000","last_fullsized":"2024-08-17T17:46:48.734744+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:16:02.971128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":485,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.714822+0000","last_change":"2024-08-17T17:43:52.520659+0000","last_active":"2024-08-17T17:46:47.714822+0000","last_peered":"2024-08-17T17:46:47.714822+0000","last_clean":"2024-08-17T17:46:47.714822+0000","last_became_active":"2024-08-17T17:43:52.516799+0000","last_became_peered":"2024-08-17T17:43:52.516799+0000","last_unstale":"2024-08-17T17:46:47.714822+0000","last_undegraded":"2024-08-17T17:46:47.714822+0000","last_fullsized":"2024-08-17T17:46:47.714822+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:25:59.408383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":687,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.731448+0000","last_change":"2024-08-17T17:33:20.794682+0000","last_active":"2024-08-17T17:46:48.731448+0000","last_peered":"2024-08-17T17:46:48.731448+0000","last_clean":"2024-08-17T17:46:48.731448+0000","last_became_active":"2024-08-17T17:33:20.794238+0000","last_became_peered":"2024-08-17T17:33:20.794238+0000","last_unstale":"2024-08-17T17:46:48.731448+0000","last_undegraded":"2024-08-17T17:46:48.731448+0000","last_fullsized":"2024-08-17T17:46:48.731448+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:15:55.821920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"90'149","reported_seq":848,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.723960+0000","last_change":"2024-08-17T17:38:30.446165+0000","last_active":"2024-08-17T17:46:49.723960+0000","last_peered":"2024-08-17T17:46:49.723960+0000","last_clean":"2024-08-17T17:46:49.723960+0000","last_became_active":"2024-08-17T17:38:30.445048+0000","last_became_peered":"2024-08-17T17:38:30.445048+0000","last_unstale":"2024-08-17T17:46:49.723960+0000","last_undegraded":"2024-08-17T17:46:49.723960+0000","last_fullsized":"2024-08-17T17:46:49.723960+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:24.291592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:24.291592+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:17.100154+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"400'7","reported_seq":23,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.721247+0000","last_change":"2024-08-17T17:46:48.724166+0000","last_active":"2024-08-17T17:46:49.721247+0000","last_peered":"2024-08-17T17:46:49.721247+0000","last_clean":"2024-08-17T17:46:49.721247+0000","last_became_active":"2024-08-17T17:46:48.723799+0000","last_became_peered":"2024-08-17T17:46:48.723799+0000","last_unstale":"2024-08-17T17:46:49.721247+0000","last_undegraded":"2024-08-17T17:46:49.721247+0000","last_fullsized":"2024-08-17T17:46:49.721247+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:44:58.020990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":10,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":64,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.714692+0000","last_change":"2024-08-17T17:41:34.611693+0000","last_active":"2024-08-17T17:46:47.714692+0000","last_peered":"2024-08-17T17:46:47.714692+0000","last_clean":"2024-08-17T17:46:47.714692+0000","last_became_active":"2024-08-17T17:41:34.611374+0000","last_became_peered":"2024-08-17T17:41:34.611374+0000","last_unstale":"2024-08-17T17:46:47.714692+0000","last_undegraded":"2024-08-17T17:46:47.714692+0000","last_fullsized":"2024-08-17T17:46:47.714692+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:17:09.897446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":475,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.722945+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:46:49.722945+0000","last_peered":"2024-08-17T17:46:49.722945+0000","last_clean":"2024-08-17T17:46:49.722945+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:46:49.722945+0000","last_undegraded":"2024-08-17T17:46:49.722945+0000","last_fullsized":"2024-08-17T17:46:49.722945+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"90'149","reported_seq":872,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.724477+0000","last_change":"2024-08-17T17:41:37.681187+0000","last_active":"2024-08-17T17:46:49.724477+0000","last_peered":"2024-08-17T17:46:49.724477+0000","last_clean":"2024-08-17T17:46:49.724477+0000","last_became_active":"2024-08-17T17:41:37.680605+0000","last_became_peered":"2024-08-17T17:41:37.680605+0000","last_unstale":"2024-08-17T17:46:49.724477+0000","last_undegraded":"2024-08-17T17:46:49.724477+0000","last_fullsized":"2024-08-17T17:46:49.724477+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.844252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.844252+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:13:31.682357+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":11,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.843216+0000","last_change":"2024-08-17T17:46:48.842473+0000","last_active":"2024-08-17T17:46:48.843216+0000","last_peered":"2024-08-17T17:46:48.843216+0000","last_clean":"2024-08-17T17:46:48.843216+0000","last_became_active":"2024-08-17T17:46:48.842149+0000","last_became_peered":"2024-08-17T17:46:48.842149+0000","last_unstale":"2024-08-17T17:46:48.843216+0000","last_undegraded":"2024-08-17T17:46:48.843216+0000","last_fullsized":"2024-08-17T17:46:48.843216+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:08.787135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":475,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.720926+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:46:49.720926+0000","last_peered":"2024-08-17T17:46:49.720926+0000","last_clean":"2024-08-17T17:46:49.720926+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:46:49.720926+0000","last_undegraded":"2024-08-17T17:46:49.720926+0000","last_fullsized":"2024-08-17T17:46:49.720926+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":334,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.728973+0000","last_change":"2024-08-17T17:33:44.156036+0000","last_active":"2024-08-17T17:46:48.728973+0000","last_peered":"2024-08-17T17:46:48.728973+0000","last_clean":"2024-08-17T17:46:48.728973+0000","last_became_active":"2024-08-17T17:33:44.155758+0000","last_became_peered":"2024-08-17T17:33:44.155758+0000","last_unstale":"2024-08-17T17:46:48.728973+0000","last_undegraded":"2024-08-17T17:46:48.728973+0000","last_fullsized":"2024-08-17T17:46:48.728973+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:23.443405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.10","version":"90'149","reported_seq":483,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.714464+0000","last_change":"2024-08-17T17:38:31.874623+0000","last_active":"2024-08-17T17:46:47.714464+0000","last_peered":"2024-08-17T17:46:47.714464+0000","last_clean":"2024-08-17T17:46:47.714464+0000","last_became_active":"2024-08-17T17:38:30.339313+0000","last_became_peered":"2024-08-17T17:38:30.339313+0000","last_unstale":"2024-08-17T17:46:47.714464+0000","last_undegraded":"2024-08-17T17:46:47.714464+0000","last_fullsized":"2024-08-17T17:46:47.714464+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:31.874423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:31.874423+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:28:17.980339+0000","scrub_duration":0.0017696579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":9,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.734724+0000","last_change":"2024-08-17T17:46:48.734724+0000","last_active":"2024-08-17T17:46:48.734724+0000","last_peered":"2024-08-17T17:46:48.734724+0000","last_clean":"2024-08-17T17:46:48.734724+0000","last_became_active":"2024-08-17T17:46:48.734425+0000","last_became_peered":"2024-08-17T17:46:48.734425+0000","last_unstale":"2024-08-17T17:46:48.734724+0000","last_undegraded":"2024-08-17T17:46:48.734724+0000","last_fullsized":"2024-08-17T17:46:48.734724+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":399,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:46:47.699070+0000","last_clean_scrub_stamp":"2024-08-17T17:46:47.699070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:19:46.551527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":424,"reported_epoch":399,"state":"active+clean","last_fresh":"2024-08-17T17:46:47.708564+0000","last_change":"2024-08-17T17:41:34.624198+0000","last_active":"2024-08-17T17:46:47.708564+0000","last_peered":"2024-08-17T17:46:47.708564+0000","last_clean":"2024-08-17T17:46:47.708564+0000","last_became_active":"2024-08-17T17:41:34.616071+0000","last_became_peered":"2024-08-17T17:41:34.616071+0000","last_unstale":"2024-08-17T17:46:47.708564+0000","last_undegraded":"2024-08-17T17:46:47.708564+0000","last_fullsized":"2024-08-17T17:46:47.708564+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:34:48.805354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":673,"reported_epoch":400,"state":"active+clean","last_fresh":"2024-08-17T17:46:48.730278+0000","last_change":"2024-08-17T17:33:20.797029+0000","last_active":"2024-08-17T17:46:48.730278+0000","last_peered":"2024-08-17T17:46:48.730278+0000","last_clean":"2024-08-17T17:46:48.730278+0000","last_became_active":"2024-08-17T17:33:20.796679+0000","last_became_peered":"2024-08-17T17:33:20.796679+0000","last_unstale":"2024-08-17T17:46:48.730278+0000","last_undegraded":"2024-08-17T17:46:48.730278+0000","last_fullsized":"2024-08-17T17:46:48.730278+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:51.438262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"90'149","reported_seq":854,"reported_epoch":401,"state":"active+clean","last_fresh":"2024-08-17T17:46:49.721469+0000","last_change":"2024-08-17T17:38:30.378664+0000","last_active":"2024-08-17T17:46:49.721469+0000","last_peered":"2024-08-17T17:46:49.721469+0000","last_clean":"2024-08-17T17:46:49.721469+0000","last_became_active":"2024-08-17T17:38:30.342287+0000","last_became_peered":"2024-08-17T17:38:30.342287+0000","last_unstale":"2024-08-17T17:46:49.721469+0000","last_undegraded":"2024-08-17T17:46:49.721469+0000","last_fullsized":"2024-08-17T17:46:49.721469+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:30.378592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:30.378592+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:04:28.933354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":374,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":9070,"num_read_kb":17230,"num_write":4339,"num_write_kb":8367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2346,"ondisk_log_size":2346,"up":96,"acting":96,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":11256,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19280,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5978,"num_read_kb":12991,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4619,"ondisk_log_size":4619,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234480,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":907648,"kb_used_data":9588,"kb_used_omap":1,"kb_used_meta":898046,"kb_avail":92837504,"statfs":{"total":95995035648,"available":95065604096,"internally_reserved":0,"allocated":9818112,"data_stored":7552282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":919599448},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496115,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":900592,"kb_used_data":9636,"kb_used_omap":1,"kb_used_meta":890942,"kb_avail":92844560,"statfs":{"total":95995035648,"available":95072829440,"internally_reserved":0,"allocated":9867264,"data_stored":7616948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":912324952},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757752,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":887792,"kb_used_data":5476,"kb_used_omap":1,"kb_used_meta":882302,"kb_avail":92857360,"statfs":{"total":95995035648,"available":95085936640,"internally_reserved":0,"allocated":5607424,"data_stored":3353677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":903477592},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019390,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306748,"kb_used_data":5232,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93438404,"statfs":{"total":95995035648,"available":95680925696,"internally_reserved":0,"allocated":5357568,"data_stored":3095551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435617,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":965704,"kb_used_data":5544,"kb_used_omap":4,"kb_used_meta":960123,"kb_avail":92779448,"statfs":{"total":95995035648,"available":95006154752,"internally_reserved":0,"allocated":5677056,"data_stored":3427210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":983165962},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599144,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":837080,"kb_used_data":9608,"kb_used_omap":7,"kb_used_meta":827448,"kb_avail":92908072,"statfs":{"total":95995035648,"available":95137865728,"internally_reserved":0,"allocated":9838592,"data_stored":7557372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":847306914},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":268,"seq":1151051235491,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":463820,"kb_used_data":5248,"kb_used_omap":10,"kb_used_meta":458549,"kb_avail":93281332,"statfs":{"total":95995035648,"available":95520083968,"internally_reserved":0,"allocated":5373952,"data_stored":3108296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10902,"internal_metadata":469554538},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926198,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":940040,"kb_used_data":5568,"kb_used_omap":7,"kb_used_meta":934456,"kb_avail":92805112,"statfs":{"total":95995035648,"available":95032434688,"internally_reserved":0,"allocated":5701632,"data_stored":3436617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":956883265},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7550,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:46:55.526 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph status --format=json 2024-08-17T17:46:55.618 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-17T17:46:55.745 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:46:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:46:55] "GET /metrics HTTP/1.1" 200 36551 "" "Prometheus/2.51.0" 2024-08-17T17:46:55.894 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:46:55.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:55 smithi053 ceph-mon[89963]: from='client.36933 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:46:55.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:55 smithi053 ceph-mon[89963]: pgmap v1344: 135 pgs: 135 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.5 MiB/s wr, 2.42k op/s 2024-08-17T17:46:55.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:55 smithi053 ceph-mon[89963]: osdmap e404: 8 total, 8 up, 8 in 2024-08-17T17:46:56.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:55 smithi003 ceph-mon[100615]: from='client.36933 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:46:56.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:55 smithi003 ceph-mon[100615]: pgmap v1344: 135 pgs: 135 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.5 MiB/s wr, 2.42k op/s 2024-08-17T17:46:56.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:55 smithi003 ceph-mon[100615]: osdmap e404: 8 total, 8 up, 8 in 2024-08-17T17:46:56.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:55 smithi003 ceph-mon[103344]: from='client.36933 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:46:56.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:55 smithi003 ceph-mon[103344]: pgmap v1344: 135 pgs: 135 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.5 MiB/s wr, 2.42k op/s 2024-08-17T17:46:56.024 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:55 smithi003 ceph-mon[103344]: osdmap e404: 8 total, 8 up, 8 in 2024-08-17T17:46:56.351 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:46:57.055 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:46:57.504 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-08-17T17:46:57.504 INFO:teuthology.orchestra.run.smithi053.stdout: "size": 20, 2024-08-17T17:46:57.504 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 600, 2024-08-17T17:46:57.504 INFO:teuthology.orchestra.run.smithi053.stdout: "ops": [ 2024-08-17T17:46:57.504 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.505 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:28160 10.7 10:e93ffb6a:::benchmark_data_smithi003_292749_object28159:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-08-17T17:46:57.505 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:02.862538+0000", 2024-08-17T17:46:57.505 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 234.63863455699999, 2024-08-17T17:46:57.505 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.87787277799999996, 2024-08-17T17:46:57.505 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:57.505 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:57.505 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:57.505 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:57.505 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:57.505 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 28160 2024-08-17T17:46:57.505 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.505 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:57.505 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.505 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:57.506 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:02.862538+0000", 2024-08-17T17:46:57.506 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.506 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.506 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.506 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:57.506 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:02.862538+0000", 2024-08-17T17:46:57.506 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.506 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.506 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.506 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:57.506 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:02.862537+0000", 2024-08-17T17:46:57.506 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:46:57.506 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.507 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.507 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:57.507 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:02.862587+0000", 2024-08-17T17:46:57.507 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9332000000000001e-05 2024-08-17T17:46:57.507 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.507 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.507 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:57.507 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:02.862596+0000", 2024-08-17T17:46:57.507 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.6800000000000005e-06 2024-08-17T17:46:57.507 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.507 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.507 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:57.507 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:02.862600+0000", 2024-08-17T17:46:57.507 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0520000000000003e-06 2024-08-17T17:46:57.507 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.508 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.508 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.508 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:02.862645+0000", 2024-08-17T17:46:57.508 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4715000000000001e-05 2024-08-17T17:46:57.508 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.508 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.508 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.508 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:02.862653+0000", 2024-08-17T17:46:57.508 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7700000000000001e-06 2024-08-17T17:46:57.508 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.508 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.508 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.508 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:03.732372+0000", 2024-08-17T17:46:57.508 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.86971870900000003 2024-08-17T17:46:57.509 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.509 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.509 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:57.509 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:03.732502+0000", 2024-08-17T17:46:57.509 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000130378 2024-08-17T17:46:57.509 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.509 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.509 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,3", 2024-08-17T17:46:57.509 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:03.732627+0000", 2024-08-17T17:46:57.509 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012522100000000001 2024-08-17T17:46:57.509 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.509 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.509 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:57.509 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:03.735064+0000", 2024-08-17T17:46:57.510 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0024363340000000001 2024-08-17T17:46:57.510 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.510 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.510 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.510 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:03.739721+0000", 2024-08-17T17:46:57.510 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0046576889999999996 2024-08-17T17:46:57.510 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.510 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.510 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.510 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:03.740331+0000", 2024-08-17T17:46:57.510 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00060977600000000002 2024-08-17T17:46:57.510 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.510 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.511 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:57.511 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:03.740376+0000", 2024-08-17T17:46:57.511 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5046999999999997e-05 2024-08-17T17:46:57.511 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.511 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.511 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:57.511 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:03.740411+0000", 2024-08-17T17:46:57.511 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4841000000000001e-05 2024-08-17T17:46:57.511 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.511 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:57.511 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.511 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.511 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.512 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:28162 10.7 10:e490594a:::benchmark_data_smithi003_292749_object28161:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-08-17T17:46:57.512 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:02.864194+0000", 2024-08-17T17:46:57.512 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 234.63697891800001, 2024-08-17T17:46:57.512 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.87633718599999999, 2024-08-17T17:46:57.512 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:57.512 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:57.512 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:57.512 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:57.512 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:57.512 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 28162 2024-08-17T17:46:57.512 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.512 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:57.512 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.512 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:57.512 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:02.864194+0000", 2024-08-17T17:46:57.513 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.513 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.513 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.513 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:57.513 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:02.864194+0000", 2024-08-17T17:46:57.513 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.513 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.513 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.513 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:57.513 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:02.864193+0000", 2024-08-17T17:46:57.513 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:46:57.513 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.513 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.513 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:57.514 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:02.864252+0000", 2024-08-17T17:46:57.514 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8730000000000002e-05 2024-08-17T17:46:57.514 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.514 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.514 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:57.514 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:02.864264+0000", 2024-08-17T17:46:57.514 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.273e-05 2024-08-17T17:46:57.514 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.514 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.514 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:57.514 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:02.864269+0000", 2024-08-17T17:46:57.514 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.279e-06 2024-08-17T17:46:57.514 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.515 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.515 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.515 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:02.864304+0000", 2024-08-17T17:46:57.515 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.5763999999999998e-05 2024-08-17T17:46:57.515 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.515 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.515 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.515 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:02.864312+0000", 2024-08-17T17:46:57.515 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7999999999999999e-06 2024-08-17T17:46:57.515 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.515 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.515 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.515 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:03.733387+0000", 2024-08-17T17:46:57.515 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.86907489500000001 2024-08-17T17:46:57.515 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.516 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.516 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:57.516 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:03.733504+0000", 2024-08-17T17:46:57.516 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000116431 2024-08-17T17:46:57.516 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.516 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.516 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,3", 2024-08-17T17:46:57.516 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:03.733650+0000", 2024-08-17T17:46:57.516 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014647299999999999 2024-08-17T17:46:57.516 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.516 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.516 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:57.516 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:03.735223+0000", 2024-08-17T17:46:57.516 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001572536 2024-08-17T17:46:57.517 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.517 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.517 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.517 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:03.739753+0000", 2024-08-17T17:46:57.517 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0045299679999999997 2024-08-17T17:46:57.517 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.517 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.517 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.517 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:03.740457+0000", 2024-08-17T17:46:57.517 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00070438 2024-08-17T17:46:57.517 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.517 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.518 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:57.518 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:03.740499+0000", 2024-08-17T17:46:57.518 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2277999999999997e-05 2024-08-17T17:46:57.518 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.518 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.518 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:57.518 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:03.740531+0000", 2024-08-17T17:46:57.518 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.1661000000000001e-05 2024-08-17T17:46:57.518 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.518 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:57.518 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.519 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.519 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.519 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:28172 10.7 10:e64aa7f9:::benchmark_data_smithi003_292749_object28171:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-08-17T17:46:57.519 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:02.881949+0000", 2024-08-17T17:46:57.519 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 234.61922367299999, 2024-08-17T17:46:57.519 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.85867628600000001, 2024-08-17T17:46:57.519 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:57.519 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:57.519 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:57.519 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:57.519 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:57.519 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 28172 2024-08-17T17:46:57.519 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.520 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:57.520 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.520 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:57.520 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:02.881949+0000", 2024-08-17T17:46:57.520 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.520 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.520 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.520 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:57.520 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:02.881949+0000", 2024-08-17T17:46:57.520 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.520 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.520 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.520 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:57.521 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:02.881948+0000", 2024-08-17T17:46:57.521 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:46:57.521 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.521 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.521 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:57.521 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:02.881999+0000", 2024-08-17T17:46:57.521 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1449000000000002e-05 2024-08-17T17:46:57.521 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.521 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.521 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:57.521 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:02.882012+0000", 2024-08-17T17:46:57.521 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2243e-05 2024-08-17T17:46:57.522 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.522 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.522 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:57.522 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:02.882018+0000", 2024-08-17T17:46:57.522 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4690000000000002e-06 2024-08-17T17:46:57.522 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.522 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.522 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.522 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:02.882065+0000", 2024-08-17T17:46:57.522 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7231000000000002e-05 2024-08-17T17:46:57.522 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.522 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.522 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.523 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:02.882072+0000", 2024-08-17T17:46:57.523 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.2629999999999997e-06 2024-08-17T17:46:57.523 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.523 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.523 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.523 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:03.734318+0000", 2024-08-17T17:46:57.523 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.85224562100000001 2024-08-17T17:46:57.523 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.523 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.523 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:57.523 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:03.734384+0000", 2024-08-17T17:46:57.523 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5896999999999996e-05 2024-08-17T17:46:57.523 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.523 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.523 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,3", 2024-08-17T17:46:57.524 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:03.734462+0000", 2024-08-17T17:46:57.524 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.8399000000000006e-05 2024-08-17T17:46:57.524 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.524 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.524 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:57.524 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:03.735230+0000", 2024-08-17T17:46:57.524 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00076744900000000002 2024-08-17T17:46:57.524 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.524 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.524 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.524 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:03.740264+0000", 2024-08-17T17:46:57.524 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0050344710000000004 2024-08-17T17:46:57.525 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.525 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.525 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.525 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:03.740562+0000", 2024-08-17T17:46:57.525 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00029813700000000002 2024-08-17T17:46:57.525 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.525 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.525 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:57.525 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:03.740598+0000", 2024-08-17T17:46:57.525 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6007999999999998e-05 2024-08-17T17:46:57.525 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.525 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.525 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:57.526 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:03.740625+0000", 2024-08-17T17:46:57.526 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.6787e-05 2024-08-17T17:46:57.526 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.526 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:57.526 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.526 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.526 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.526 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:47656 10.0 10:02151d15:::benchmark_data_smithi003_292749_object47655:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-08-17T17:46:57.526 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:14.182567+0000", 2024-08-17T17:46:57.526 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 223.318605867, 2024-08-17T17:46:57.526 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.25187148599999998, 2024-08-17T17:46:57.526 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:57.526 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:57.527 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:57.527 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:57.527 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:57.527 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 47656 2024-08-17T17:46:57.527 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.527 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:57.527 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.527 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:57.527 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182567+0000", 2024-08-17T17:46:57.527 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.527 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.527 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.527 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:57.527 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182567+0000", 2024-08-17T17:46:57.528 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.528 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.528 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.528 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:57.528 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182566+0000", 2024-08-17T17:46:57.528 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:46:57.528 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.528 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.528 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:57.528 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182600+0000", 2024-08-17T17:46:57.528 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3655000000000002e-05 2024-08-17T17:46:57.528 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.528 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.529 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:57.529 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182609+0000", 2024-08-17T17:46:57.529 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.2560000000000008e-06 2024-08-17T17:46:57.529 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.529 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.529 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:57.529 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182613+0000", 2024-08-17T17:46:57.529 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5240000000000002e-06 2024-08-17T17:46:57.529 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.529 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.529 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.529 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182667+0000", 2024-08-17T17:46:57.529 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.3224999999999999e-05 2024-08-17T17:46:57.529 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.529 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.530 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:57.530 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.182883+0000", 2024-08-17T17:46:57.530 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00021582400000000001 2024-08-17T17:46:57.530 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.530 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.530 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 1,4", 2024-08-17T17:46:57.530 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.183034+0000", 2024-08-17T17:46:57.530 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000151328 2024-08-17T17:46:57.530 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.530 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.530 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:57.530 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.184564+0000", 2024-08-17T17:46:57.530 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.001530624 2024-08-17T17:46:57.530 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.530 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.531 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.531 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.188423+0000", 2024-08-17T17:46:57.531 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0038583129999999999 2024-08-17T17:46:57.531 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.531 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.531 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.531 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.434359+0000", 2024-08-17T17:46:57.531 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.24593607300000001 2024-08-17T17:46:57.531 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.531 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.531 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:57.531 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.434403+0000", 2024-08-17T17:46:57.531 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.4515999999999999e-05 2024-08-17T17:46:57.531 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.531 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.532 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:57.532 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:14.434438+0000", 2024-08-17T17:46:57.532 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4912e-05 2024-08-17T17:46:57.532 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.532 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:57.533 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.533 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.533 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.533 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:93981 10.7 10:e61d3f15:::benchmark_data_smithi003_292749_object93980:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-08-17T17:46:57.533 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:51.484804+0000", 2024-08-17T17:46:57.533 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 186.01636825599999, 2024-08-17T17:46:57.533 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.204959605, 2024-08-17T17:46:57.533 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:57.533 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "reached pg", 2024-08-17T17:46:57.533 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:57.533 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:57.533 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:57.533 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 93981 2024-08-17T17:46:57.534 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.534 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:57.534 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.534 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:57.534 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.484804+0000", 2024-08-17T17:46:57.534 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.534 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.534 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.534 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:57.534 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.484804+0000", 2024-08-17T17:46:57.534 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.534 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.534 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.535 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:57.535 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.484804+0000", 2024-08-17T17:46:57.535 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:46:57.535 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.535 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.535 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:57.535 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.484856+0000", 2024-08-17T17:46:57.535 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1687999999999997e-05 2024-08-17T17:46:57.535 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.535 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.535 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:57.535 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.484864+0000", 2024-08-17T17:46:57.535 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0960000000000001e-06 2024-08-17T17:46:57.535 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.536 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.536 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:57.536 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.484868+0000", 2024-08-17T17:46:57.536 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2429999999999999e-06 2024-08-17T17:46:57.536 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.536 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.536 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.536 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.485048+0000", 2024-08-17T17:46:57.536 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00018002599999999999 2024-08-17T17:46:57.536 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.536 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.536 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:57.536 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.485170+0000", 2024-08-17T17:46:57.537 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012164599999999999 2024-08-17T17:46:57.537 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.537 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.537 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 2,3", 2024-08-17T17:46:57.537 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.485323+0000", 2024-08-17T17:46:57.537 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015368399999999999 2024-08-17T17:46:57.537 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.537 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.537 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.537 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.502694+0000", 2024-08-17T17:46:57.537 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017370778 2024-08-17T17:46:57.537 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.538 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.538 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.538 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.503365+0000", 2024-08-17T17:46:57.538 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00067082999999999997 2024-08-17T17:46:57.538 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.538 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.538 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.538 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.513834+0000", 2024-08-17T17:46:57.538 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.01046959 2024-08-17T17:46:57.538 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.538 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.538 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.538 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.513914+0000", 2024-08-17T17:46:57.538 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.9474999999999994e-05 2024-08-17T17:46:57.539 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.539 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.539 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.539 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.689700+0000", 2024-08-17T17:46:57.539 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1757860199999999 2024-08-17T17:46:57.539 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.539 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.539 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:57.539 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.689764+0000", 2024-08-17T17:46:57.539 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4028000000000004e-05 2024-08-17T17:46:57.539 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.539 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:57.540 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.540 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.540 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.540 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:94029 10.7 10:e0bcd7cc:::benchmark_data_smithi003_292749_object94028:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-08-17T17:46:57.540 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:51.539065+0000", 2024-08-17T17:46:57.540 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 185.96210764, 2024-08-17T17:46:57.540 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1859267610000002, 2024-08-17T17:46:57.540 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:57.540 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:57.540 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:57.540 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:57.540 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:57.540 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 94029 2024-08-17T17:46:57.540 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.541 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:57.541 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.541 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:57.541 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.539065+0000", 2024-08-17T17:46:57.541 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.541 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.541 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.541 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:57.541 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.539065+0000", 2024-08-17T17:46:57.541 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.541 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.541 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.541 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:57.541 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.539064+0000", 2024-08-17T17:46:57.542 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:46:57.542 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.542 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.542 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:57.542 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.539186+0000", 2024-08-17T17:46:57.542 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012186899999999999 2024-08-17T17:46:57.542 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.542 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.542 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:57.542 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.539200+0000", 2024-08-17T17:46:57.542 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4734e-05 2024-08-17T17:46:57.542 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.542 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.543 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:57.543 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.539206+0000", 2024-08-17T17:46:57.543 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.7180000000000001e-06 2024-08-17T17:46:57.543 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.543 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.543 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.543 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.539283+0000", 2024-08-17T17:46:57.543 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7140999999999999e-05 2024-08-17T17:46:57.543 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.543 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.543 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.543 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.539291+0000", 2024-08-17T17:46:57.543 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.2749999999999995e-06 2024-08-17T17:46:57.543 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.543 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.544 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.544 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.514065+0000", 2024-08-17T17:46:57.544 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.97477390399999997 2024-08-17T17:46:57.544 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.544 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.544 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.544 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.514072+0000", 2024-08-17T17:46:57.544 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.1330000000000001e-06 2024-08-17T17:46:57.544 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.544 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.544 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.544 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.692321+0000", 2024-08-17T17:46:57.544 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1782482519999999 2024-08-17T17:46:57.544 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.545 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.545 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:57.545 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.692465+0000", 2024-08-17T17:46:57.545 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014455100000000001 2024-08-17T17:46:57.545 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.545 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.545 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,5", 2024-08-17T17:46:57.545 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.692633+0000", 2024-08-17T17:46:57.545 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016745 2024-08-17T17:46:57.545 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.545 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.545 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.545 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.701433+0000", 2024-08-17T17:46:57.545 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0087998820000000002 2024-08-17T17:46:57.546 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.546 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.546 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:57.546 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.701444+0000", 2024-08-17T17:46:57.546 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1639e-05 2024-08-17T17:46:57.546 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.546 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.546 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.546 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.720273+0000", 2024-08-17T17:46:57.546 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.018828365 2024-08-17T17:46:57.546 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.546 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.546 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:57.547 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.720329+0000", 2024-08-17T17:46:57.547 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6128999999999997e-05 2024-08-17T17:46:57.547 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.547 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.547 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:57.547 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.724992+0000", 2024-08-17T17:46:57.547 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0046630309999999998 2024-08-17T17:46:57.547 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.547 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:57.547 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.547 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.547 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.547 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:94037 10.7 10:ebcf1fb4:::benchmark_data_smithi003_292749_object94036:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-08-17T17:46:57.548 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:51.545692+0000", 2024-08-17T17:46:57.548 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 185.955480714, 2024-08-17T17:46:57.548 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1838650780000002, 2024-08-17T17:46:57.548 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:57.548 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:57.548 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:57.548 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:57.548 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:57.548 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 94037 2024-08-17T17:46:57.548 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.548 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:57.548 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.548 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:57.548 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.545692+0000", 2024-08-17T17:46:57.548 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.549 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.549 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.549 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:57.549 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.545692+0000", 2024-08-17T17:46:57.549 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.549 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.549 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.549 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:57.549 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.545691+0000", 2024-08-17T17:46:57.549 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:46:57.549 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.549 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.549 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:57.549 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.545724+0000", 2024-08-17T17:46:57.550 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3120000000000001e-05 2024-08-17T17:46:57.550 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.550 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.550 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:57.550 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.545739+0000", 2024-08-17T17:46:57.550 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.4946000000000001e-05 2024-08-17T17:46:57.550 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.550 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.550 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:57.550 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.545746+0000", 2024-08-17T17:46:57.550 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.2660000000000004e-06 2024-08-17T17:46:57.550 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.550 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.551 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.551 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.545808+0000", 2024-08-17T17:46:57.551 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1369000000000001e-05 2024-08-17T17:46:57.551 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.551 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.551 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.551 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.545816+0000", 2024-08-17T17:46:57.551 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9309999999999997e-06 2024-08-17T17:46:57.551 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.551 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.551 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.551 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.514183+0000", 2024-08-17T17:46:57.551 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.968366379 2024-08-17T17:46:57.551 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.552 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.552 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.552 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.514188+0000", 2024-08-17T17:46:57.552 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7650000000000001e-06 2024-08-17T17:46:57.552 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.552 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.552 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.552 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.695417+0000", 2024-08-17T17:46:57.552 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1812295399999999 2024-08-17T17:46:57.552 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.552 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.552 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:57.552 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.695563+0000", 2024-08-17T17:46:57.552 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000145876 2024-08-17T17:46:57.553 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.553 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.553 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,5", 2024-08-17T17:46:57.553 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.695671+0000", 2024-08-17T17:46:57.553 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010835 2024-08-17T17:46:57.553 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.553 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.553 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:57.553 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.701453+0000", 2024-08-17T17:46:57.553 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0057812790000000003 2024-08-17T17:46:57.553 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.553 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.554 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.554 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.707373+0000", 2024-08-17T17:46:57.554 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0059208170000000001 2024-08-17T17:46:57.554 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.554 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.554 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.554 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.725150+0000", 2024-08-17T17:46:57.554 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017776755000000002 2024-08-17T17:46:57.554 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.554 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.554 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:57.554 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.725211+0000", 2024-08-17T17:46:57.555 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.0795999999999999e-05 2024-08-17T17:46:57.555 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.555 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.555 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:57.555 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.729557+0000", 2024-08-17T17:46:57.555 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00434601 2024-08-17T17:46:57.555 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.555 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:57.555 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.555 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.555 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.555 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:94059 10.7 10:ec940e88:::benchmark_data_smithi003_292749_object94058:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-08-17T17:46:57.555 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:51.563674+0000", 2024-08-17T17:46:57.555 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 185.937498519, 2024-08-17T17:46:57.556 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1732834589999999, 2024-08-17T17:46:57.556 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:57.556 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:57.556 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:57.556 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:57.556 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:57.556 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 94059 2024-08-17T17:46:57.556 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.556 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:57.556 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.556 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:57.556 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.563674+0000", 2024-08-17T17:46:57.557 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.557 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.557 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.557 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:57.557 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.563674+0000", 2024-08-17T17:46:57.557 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.557 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.557 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.557 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:57.557 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.563674+0000", 2024-08-17T17:46:57.557 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:46:57.557 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.557 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.557 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:57.558 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.563729+0000", 2024-08-17T17:46:57.558 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5948e-05 2024-08-17T17:46:57.558 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.558 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.558 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:57.558 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.563738+0000", 2024-08-17T17:46:57.558 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.8950000000000004e-06 2024-08-17T17:46:57.558 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.558 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.558 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:57.558 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.563743+0000", 2024-08-17T17:46:57.558 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.3250000000000001e-06 2024-08-17T17:46:57.558 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.559 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.559 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.559 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.564370+0000", 2024-08-17T17:46:57.559 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00062728199999999995 2024-08-17T17:46:57.559 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.559 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.559 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.559 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.564378+0000", 2024-08-17T17:46:57.559 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.3080000000000008e-06 2024-08-17T17:46:57.559 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.559 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.559 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.559 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.514218+0000", 2024-08-17T17:46:57.560 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.94983957200000002 2024-08-17T17:46:57.560 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.560 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.560 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.560 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.514227+0000", 2024-08-17T17:46:57.560 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.6670000000000002e-06 2024-08-17T17:46:57.560 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.560 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.560 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.560 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.697950+0000", 2024-08-17T17:46:57.560 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1837236289999999 2024-08-17T17:46:57.560 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.560 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.561 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:57.561 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.698059+0000", 2024-08-17T17:46:57.561 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000109308 2024-08-17T17:46:57.561 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.561 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.561 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,5", 2024-08-17T17:46:57.561 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.698241+0000", 2024-08-17T17:46:57.561 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00018187499999999999 2024-08-17T17:46:57.561 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.561 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.561 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:57.561 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.707260+0000", 2024-08-17T17:46:57.562 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0090189410000000008 2024-08-17T17:46:57.562 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.562 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.562 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.562 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.707458+0000", 2024-08-17T17:46:57.562 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00019735600000000001 2024-08-17T17:46:57.562 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.562 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.562 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.562 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.732359+0000", 2024-08-17T17:46:57.562 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.024901135000000001 2024-08-17T17:46:57.562 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.563 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.563 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:57.563 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.732425+0000", 2024-08-17T17:46:57.563 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.6098000000000003e-05 2024-08-17T17:46:57.563 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.563 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.563 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:57.563 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.736958+0000", 2024-08-17T17:46:57.563 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.004532721 2024-08-17T17:46:57.563 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.563 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:57.563 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.563 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.563 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.564 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:94075 10.7 10:e27f5ae7:::benchmark_data_smithi003_292749_object94074:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-08-17T17:46:57.564 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:51.578006+0000", 2024-08-17T17:46:57.564 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 185.92316656099999, 2024-08-17T17:46:57.564 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1631619300000002, 2024-08-17T17:46:57.564 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:57.564 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:57.564 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:57.564 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:57.564 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:57.564 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 94075 2024-08-17T17:46:57.564 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.564 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:57.564 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.578006+0000", 2024-08-17T17:46:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.578006+0000", 2024-08-17T17:46:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.578006+0000", 2024-08-17T17:46:57.565 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:46:57.566 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.566 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.566 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:57.566 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.578058+0000", 2024-08-17T17:46:57.566 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.1972000000000003e-05 2024-08-17T17:46:57.566 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.566 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.566 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:57.566 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.578065+0000", 2024-08-17T17:46:57.566 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.4080000000000001e-06 2024-08-17T17:46:57.566 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.566 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.566 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:57.567 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.578069+0000", 2024-08-17T17:46:57.567 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.8179999999999997e-06 2024-08-17T17:46:57.567 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.567 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.567 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.567 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.578830+0000", 2024-08-17T17:46:57.567 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000761581 2024-08-17T17:46:57.567 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.567 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.567 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.567 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.578839+0000", 2024-08-17T17:46:57.568 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.3080000000000008e-06 2024-08-17T17:46:57.568 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.568 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.568 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.568 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.514285+0000", 2024-08-17T17:46:57.568 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.93544610900000003 2024-08-17T17:46:57.568 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.568 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.568 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.568 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.514295+0000", 2024-08-17T17:46:57.568 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0359e-05 2024-08-17T17:46:57.568 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.569 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.569 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.569 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.701472+0000", 2024-08-17T17:46:57.569 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1871768469999999 2024-08-17T17:46:57.569 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.569 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.569 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:57.569 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.701585+0000", 2024-08-17T17:46:57.569 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000112448 2024-08-17T17:46:57.569 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.569 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.569 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,5", 2024-08-17T17:46:57.569 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.701694+0000", 2024-08-17T17:46:57.569 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000109932 2024-08-17T17:46:57.570 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.570 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.570 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:57.570 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.707386+0000", 2024-08-17T17:46:57.570 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0056912400000000002 2024-08-17T17:46:57.570 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.570 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.570 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.570 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.707525+0000", 2024-08-17T17:46:57.570 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013979899999999999 2024-08-17T17:46:57.570 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.570 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.570 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.571 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.737012+0000", 2024-08-17T17:46:57.571 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.029486542000000001 2024-08-17T17:46:57.571 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.571 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.571 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:57.571 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.737074+0000", 2024-08-17T17:46:57.571 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1859999999999994e-05 2024-08-17T17:46:57.571 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.571 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.571 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:57.571 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.741168+0000", 2024-08-17T17:46:57.571 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0040941170000000004 2024-08-17T17:46:57.571 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.571 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:57.572 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.572 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.572 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.572 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:94081 10.7 10:e581f32f:::benchmark_data_smithi003_292749_object94080:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-08-17T17:46:57.572 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:51.586316+0000", 2024-08-17T17:46:57.572 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 185.914856348, 2024-08-17T17:46:57.572 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1585097719999999, 2024-08-17T17:46:57.572 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:57.572 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:57.572 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:57.572 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:57.572 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:57.572 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 94081 2024-08-17T17:46:57.572 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.573 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:57.573 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.573 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:57.573 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.586316+0000", 2024-08-17T17:46:57.573 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.573 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.573 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.573 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:57.573 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.586316+0000", 2024-08-17T17:46:57.573 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.573 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.573 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.573 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:57.573 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.586315+0000", 2024-08-17T17:46:57.573 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:46:57.574 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.574 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.574 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:57.574 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.586380+0000", 2024-08-17T17:46:57.574 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4789999999999995e-05 2024-08-17T17:46:57.574 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.574 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.574 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:57.574 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.586397+0000", 2024-08-17T17:46:57.574 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6412000000000001e-05 2024-08-17T17:46:57.574 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.574 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.574 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:57.574 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.586403+0000", 2024-08-17T17:46:57.575 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.7349999999999999e-06 2024-08-17T17:46:57.575 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.575 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.575 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.575 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.586462+0000", 2024-08-17T17:46:57.575 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8723e-05 2024-08-17T17:46:57.575 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.575 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.575 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.575 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.586471+0000", 2024-08-17T17:46:57.575 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.1279999999999997e-06 2024-08-17T17:46:57.575 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.575 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.575 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.576 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.514320+0000", 2024-08-17T17:46:57.576 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.92784924499999999 2024-08-17T17:46:57.576 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.576 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.576 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.576 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.514325+0000", 2024-08-17T17:46:57.576 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.206e-06 2024-08-17T17:46:57.576 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.576 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.576 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.576 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.704236+0000", 2024-08-17T17:46:57.576 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1899110669999999 2024-08-17T17:46:57.576 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.576 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.576 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:57.577 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.704384+0000", 2024-08-17T17:46:57.577 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000148147 2024-08-17T17:46:57.577 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.577 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.577 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,5", 2024-08-17T17:46:57.577 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.704579+0000", 2024-08-17T17:46:57.577 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000195334 2024-08-17T17:46:57.577 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.577 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.577 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:57.577 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.713349+0000", 2024-08-17T17:46:57.577 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0087697130000000002 2024-08-17T17:46:57.577 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.578 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.578 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.578 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.713389+0000", 2024-08-17T17:46:57.578 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9745e-05 2024-08-17T17:46:57.578 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.578 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.578 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.578 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.741220+0000", 2024-08-17T17:46:57.578 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.027831285000000001 2024-08-17T17:46:57.578 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.578 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.578 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:57.578 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.741253+0000", 2024-08-17T17:46:57.579 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3424999999999999e-05 2024-08-17T17:46:57.579 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.579 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.579 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:57.579 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.744826+0000", 2024-08-17T17:46:57.579 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0035726960000000002 2024-08-17T17:46:57.579 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.579 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:57.579 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.579 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.579 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.579 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:94083 10.7 10:ee4409bb:::benchmark_data_smithi003_292749_object94082:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-08-17T17:46:57.579 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:51.586497+0000", 2024-08-17T17:46:57.579 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 185.91467585800001, 2024-08-17T17:46:57.580 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1635916819999998, 2024-08-17T17:46:57.580 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:57.580 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:57.580 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:57.580 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:57.580 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:57.580 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 94083 2024-08-17T17:46:57.580 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.580 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:57.580 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.580 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:57.580 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.586497+0000", 2024-08-17T17:46:57.580 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.580 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.581 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.581 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:57.581 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.586497+0000", 2024-08-17T17:46:57.581 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.581 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.581 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.581 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:57.581 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.586496+0000", 2024-08-17T17:46:57.581 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:46:57.581 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.581 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.581 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:57.581 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.586550+0000", 2024-08-17T17:46:57.581 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.4268999999999998e-05 2024-08-17T17:46:57.582 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.582 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.582 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:57.582 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.586559+0000", 2024-08-17T17:46:57.582 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.2050000000000002e-06 2024-08-17T17:46:57.582 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.582 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.582 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:57.582 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.586563+0000", 2024-08-17T17:46:57.582 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7380000000000002e-06 2024-08-17T17:46:57.582 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.582 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.582 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.583 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.586691+0000", 2024-08-17T17:46:57.583 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00012763600000000001 2024-08-17T17:46:57.583 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.583 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.583 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.583 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.586699+0000", 2024-08-17T17:46:57.583 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7619999999999995e-06 2024-08-17T17:46:57.583 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.583 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.583 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.583 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.514378+0000", 2024-08-17T17:46:57.583 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.92767953000000003 2024-08-17T17:46:57.583 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.583 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.584 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.584 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.514382+0000", 2024-08-17T17:46:57.584 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0659999999999997e-06 2024-08-17T17:46:57.584 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.584 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.584 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.584 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.707579+0000", 2024-08-17T17:46:57.584 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.193196543 2024-08-17T17:46:57.584 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.584 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.584 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:57.584 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.707719+0000", 2024-08-17T17:46:57.584 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014027899999999999 2024-08-17T17:46:57.585 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.585 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.585 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,5", 2024-08-17T17:46:57.585 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.707913+0000", 2024-08-17T17:46:57.585 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00019351500000000001 2024-08-17T17:46:57.585 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.585 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.585 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:57.585 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.713396+0000", 2024-08-17T17:46:57.585 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0054832370000000002 2024-08-17T17:46:57.585 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.585 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.585 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.586 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.713457+0000", 2024-08-17T17:46:57.586 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.067e-05 2024-08-17T17:46:57.586 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.586 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.586 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.586 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.744966+0000", 2024-08-17T17:46:57.586 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.031508926999999999 2024-08-17T17:46:57.586 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.586 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.586 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:57.586 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.745033+0000", 2024-08-17T17:46:57.586 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.6968e-05 2024-08-17T17:46:57.586 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.586 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.587 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:57.587 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.750088+0000", 2024-08-17T17:46:57.587 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0050559020000000001 2024-08-17T17:46:57.587 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.587 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:57.587 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.587 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.587 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.587 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:94088 10.7 10:e93c86c1:::benchmark_data_smithi003_292749_object94087:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-08-17T17:46:57.587 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:51.590354+0000", 2024-08-17T17:46:57.587 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 185.91081889899999, 2024-08-17T17:46:57.587 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1677168830000002, 2024-08-17T17:46:57.587 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:57.587 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:57.588 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:57.588 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:57.588 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:57.588 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 94088 2024-08-17T17:46:57.588 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.588 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:57.588 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.588 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:57.588 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.590354+0000", 2024-08-17T17:46:57.588 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.588 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.588 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.588 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:57.589 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.590354+0000", 2024-08-17T17:46:57.590 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.590 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.590 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.590 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:57.590 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.590353+0000", 2024-08-17T17:46:57.590 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:46:57.590 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.590 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.590 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:57.590 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.590413+0000", 2024-08-17T17:46:57.590 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9629000000000001e-05 2024-08-17T17:46:57.590 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.591 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.591 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:57.591 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.590423+0000", 2024-08-17T17:46:57.591 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0740999999999999e-05 2024-08-17T17:46:57.591 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.591 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.591 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:57.591 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.590427+0000", 2024-08-17T17:46:57.591 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9580000000000001e-06 2024-08-17T17:46:57.591 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.591 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.591 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.591 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.590533+0000", 2024-08-17T17:46:57.592 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00010526000000000001 2024-08-17T17:46:57.592 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.592 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.592 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.592 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.590538+0000", 2024-08-17T17:46:57.592 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9989999999999999e-06 2024-08-17T17:46:57.592 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.592 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.592 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.592 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.514406+0000", 2024-08-17T17:46:57.592 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.92386801200000002 2024-08-17T17:46:57.592 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.592 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.592 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.593 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.514410+0000", 2024-08-17T17:46:57.593 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2930000000000002e-06 2024-08-17T17:46:57.593 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.593 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.593 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.593 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.710427+0000", 2024-08-17T17:46:57.593 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.196016993 2024-08-17T17:46:57.593 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.593 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.593 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:57.593 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.710520+0000", 2024-08-17T17:46:57.593 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.3196999999999996e-05 2024-08-17T17:46:57.593 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.594 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.594 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,5", 2024-08-17T17:46:57.594 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.710616+0000", 2024-08-17T17:46:57.594 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5464000000000006e-05 2024-08-17T17:46:57.594 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.594 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.594 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.594 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.716845+0000", 2024-08-17T17:46:57.594 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0062299019999999998 2024-08-17T17:46:57.594 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.594 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.594 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:57.594 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.720185+0000", 2024-08-17T17:46:57.595 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0033398310000000001 2024-08-17T17:46:57.595 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.595 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.595 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.595 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.754059+0000", 2024-08-17T17:46:57.595 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.033874222000000002 2024-08-17T17:46:57.595 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.595 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.595 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:57.595 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.754139+0000", 2024-08-17T17:46:57.595 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.9944999999999994e-05 2024-08-17T17:46:57.595 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.595 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.595 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:57.596 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.758071+0000", 2024-08-17T17:46:57.596 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0039312039999999998 2024-08-17T17:46:57.596 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.596 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:57.596 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.596 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.596 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.596 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:94092 10.7 10:ed635310:::benchmark_data_smithi003_292749_object94091:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-08-17T17:46:57.596 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:51.594518+0000", 2024-08-17T17:46:57.596 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 185.90665491999999, 2024-08-17T17:46:57.596 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1776650910000002, 2024-08-17T17:46:57.596 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:57.596 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:57.596 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:57.597 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:57.597 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:57.597 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 94092 2024-08-17T17:46:57.597 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.597 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:57.597 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.597 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:57.597 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.594518+0000", 2024-08-17T17:46:57.597 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.597 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.597 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.597 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:57.597 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.594518+0000", 2024-08-17T17:46:57.597 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.598 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.598 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.598 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:57.598 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.594516+0000", 2024-08-17T17:46:57.598 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:46:57.598 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.598 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.598 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:57.598 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.594575+0000", 2024-08-17T17:46:57.598 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.8835000000000002e-05 2024-08-17T17:46:57.598 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.598 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.598 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:57.599 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.594587+0000", 2024-08-17T17:46:57.599 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2048999999999999e-05 2024-08-17T17:46:57.599 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.599 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.599 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:57.599 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.594593+0000", 2024-08-17T17:46:57.599 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5260000000000001e-06 2024-08-17T17:46:57.599 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.599 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.599 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.599 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.594639+0000", 2024-08-17T17:46:57.599 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.6103000000000002e-05 2024-08-17T17:46:57.599 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.600 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.600 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.600 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.594648+0000", 2024-08-17T17:46:57.600 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.8699999999999998e-06 2024-08-17T17:46:57.600 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.600 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.600 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.600 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.514429+0000", 2024-08-17T17:46:57.600 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.919781336 2024-08-17T17:46:57.600 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.600 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.600 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.600 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.514433+0000", 2024-08-17T17:46:57.600 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.3780000000000001e-06 2024-08-17T17:46:57.601 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.601 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.601 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.601 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.713493+0000", 2024-08-17T17:46:57.601 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.199060386 2024-08-17T17:46:57.601 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.601 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.601 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:57.601 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.713634+0000", 2024-08-17T17:46:57.601 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014084899999999999 2024-08-17T17:46:57.601 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.601 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.601 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,5", 2024-08-17T17:46:57.601 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.713790+0000", 2024-08-17T17:46:57.602 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015648800000000001 2024-08-17T17:46:57.602 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.602 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.602 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.602 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.720229+0000", 2024-08-17T17:46:57.602 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0064382809999999997 2024-08-17T17:46:57.602 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.602 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.602 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:57.602 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.725008+0000", 2024-08-17T17:46:57.602 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0047790250000000001 2024-08-17T17:46:57.602 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.602 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.602 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.603 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.766385+0000", 2024-08-17T17:46:57.603 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.041377035 2024-08-17T17:46:57.603 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.603 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.603 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:57.603 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.766444+0000", 2024-08-17T17:46:57.603 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.9200000000000002e-05 2024-08-17T17:46:57.603 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.603 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.603 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:57.603 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.772183+0000", 2024-08-17T17:46:57.603 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0057389829999999996 2024-08-17T17:46:57.603 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.604 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:57.604 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.604 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.604 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.604 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:94095 10.7 10:ec2e7528:::benchmark_data_smithi003_292749_object94094:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-08-17T17:46:57.604 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:51.595742+0000", 2024-08-17T17:46:57.604 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 185.90543042600001, 2024-08-17T17:46:57.604 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1804850999999998, 2024-08-17T17:46:57.604 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:57.604 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:57.604 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:57.604 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:57.604 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:57.604 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 94095 2024-08-17T17:46:57.605 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.605 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:57.605 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.605 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:57.605 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.595742+0000", 2024-08-17T17:46:57.605 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.605 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.605 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.605 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:57.605 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.595742+0000", 2024-08-17T17:46:57.605 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.605 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.605 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.605 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:57.606 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.595742+0000", 2024-08-17T17:46:57.606 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999995 2024-08-17T17:46:57.606 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.606 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.606 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:57.606 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.595816+0000", 2024-08-17T17:46:57.606 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.4277999999999996e-05 2024-08-17T17:46:57.606 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.606 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.606 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:57.606 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.595827+0000", 2024-08-17T17:46:57.606 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0696999999999999e-05 2024-08-17T17:46:57.606 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.607 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.607 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:57.607 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.595831+0000", 2024-08-17T17:46:57.607 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7119999999999999e-06 2024-08-17T17:46:57.607 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.607 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.607 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.607 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.595879+0000", 2024-08-17T17:46:57.607 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7512000000000002e-05 2024-08-17T17:46:57.607 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.607 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.607 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.607 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.595887+0000", 2024-08-17T17:46:57.607 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.7940000000000003e-06 2024-08-17T17:46:57.608 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.608 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.608 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.608 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.514450+0000", 2024-08-17T17:46:57.608 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.91856369800000004 2024-08-17T17:46:57.608 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.608 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.608 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.608 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.514454+0000", 2024-08-17T17:46:57.608 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.0550000000000001e-06 2024-08-17T17:46:57.608 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.608 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.608 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.609 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.716886+0000", 2024-08-17T17:46:57.609 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2024319939999999 2024-08-17T17:46:57.609 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.609 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.609 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:57.609 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.717028+0000", 2024-08-17T17:46:57.609 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00014109800000000001 2024-08-17T17:46:57.609 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.609 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.609 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,5", 2024-08-17T17:46:57.609 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.717199+0000", 2024-08-17T17:46:57.609 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00017164099999999999 2024-08-17T17:46:57.609 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.609 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.609 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.610 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.725092+0000", 2024-08-17T17:46:57.610 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0078932410000000005 2024-08-17T17:46:57.610 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.610 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.610 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:57.610 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.729572+0000", 2024-08-17T17:46:57.610 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0044798199999999998 2024-08-17T17:46:57.610 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.610 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.610 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.610 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.772325+0000", 2024-08-17T17:46:57.610 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.042752914000000003 2024-08-17T17:46:57.610 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.611 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.611 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:57.611 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.772392+0000", 2024-08-17T17:46:57.611 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.6359000000000006e-05 2024-08-17T17:46:57.611 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.611 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.611 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:57.611 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.776227+0000", 2024-08-17T17:46:57.611 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0038358110000000002 2024-08-17T17:46:57.611 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.611 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:57.611 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.611 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.611 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.612 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:94108 10.7 10:ea0d17f7:::benchmark_data_smithi003_292749_object94107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-08-17T17:46:57.612 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:51.611301+0000", 2024-08-17T17:46:57.612 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 185.88987210799999, 2024-08-17T17:46:57.612 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1692675239999999, 2024-08-17T17:46:57.612 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:57.612 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:57.612 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:57.612 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:57.612 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:57.612 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 94108 2024-08-17T17:46:57.612 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.612 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:57.612 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.612 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:57.613 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.611301+0000", 2024-08-17T17:46:57.613 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.613 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.613 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.613 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:57.613 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.611301+0000", 2024-08-17T17:46:57.613 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.613 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.613 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.613 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:57.613 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.611299+0000", 2024-08-17T17:46:57.613 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:46:57.613 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.613 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.614 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:57.614 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.611351+0000", 2024-08-17T17:46:57.614 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.2012999999999998e-05 2024-08-17T17:46:57.614 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.614 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.614 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:57.614 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.611361+0000", 2024-08-17T17:46:57.614 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.7000000000000003e-06 2024-08-17T17:46:57.614 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.614 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.614 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:57.614 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.611365+0000", 2024-08-17T17:46:57.614 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2560000000000004e-06 2024-08-17T17:46:57.615 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.615 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.615 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.615 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.611435+0000", 2024-08-17T17:46:57.615 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.9912000000000004e-05 2024-08-17T17:46:57.615 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.615 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.615 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.615 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.611445+0000", 2024-08-17T17:46:57.615 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.8560000000000002e-06 2024-08-17T17:46:57.615 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.615 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.615 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.615 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.514473+0000", 2024-08-17T17:46:57.616 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.90302762599999997 2024-08-17T17:46:57.616 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.616 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.616 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.616 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.514477+0000", 2024-08-17T17:46:57.616 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.5580000000000002e-06 2024-08-17T17:46:57.616 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.616 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.616 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.616 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.729600+0000", 2024-08-17T17:46:57.616 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.215122657 2024-08-17T17:46:57.616 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.616 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.616 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:57.617 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.729709+0000", 2024-08-17T17:46:57.617 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000109325 2024-08-17T17:46:57.617 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.617 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.617 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,5", 2024-08-17T17:46:57.617 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.729820+0000", 2024-08-17T17:46:57.617 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000110862 2024-08-17T17:46:57.617 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.617 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.617 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:57.617 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.741182+0000", 2024-08-17T17:46:57.617 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.01136209 2024-08-17T17:46:57.617 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.618 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.618 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.618 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.744897+0000", 2024-08-17T17:46:57.618 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0037143050000000002 2024-08-17T17:46:57.618 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.618 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.618 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.618 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.776347+0000", 2024-08-17T17:46:57.618 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.031449940000000003 2024-08-17T17:46:57.618 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.618 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.618 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:57.618 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.776383+0000", 2024-08-17T17:46:57.618 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.6369999999999999e-05 2024-08-17T17:46:57.619 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.619 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.619 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:57.619 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.780568+0000", 2024-08-17T17:46:57.619 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0041851340000000001 2024-08-17T17:46:57.619 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.619 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:57.619 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.619 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.619 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.619 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:94117 10.7 10:e98e573f:::benchmark_data_smithi003_292749_object94116:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-08-17T17:46:57.619 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:51.620619+0000", 2024-08-17T17:46:57.619 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 185.88055388800001, 2024-08-17T17:46:57.620 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.1644904220000001, 2024-08-17T17:46:57.620 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:57.620 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:57.620 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:57.620 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:57.620 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:57.620 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 94117 2024-08-17T17:46:57.620 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.620 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:57.620 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.620 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:57.620 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.620619+0000", 2024-08-17T17:46:57.620 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.620 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.620 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.621 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:57.621 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.620619+0000", 2024-08-17T17:46:57.621 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.621 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.621 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.621 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:57.621 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.620617+0000", 2024-08-17T17:46:57.621 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999981 2024-08-17T17:46:57.621 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.621 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.621 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:57.621 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.620688+0000", 2024-08-17T17:46:57.622 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.1216000000000004e-05 2024-08-17T17:46:57.622 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.622 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.622 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:57.622 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.620708+0000", 2024-08-17T17:46:57.622 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0398e-05 2024-08-17T17:46:57.622 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.622 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.622 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:57.622 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.620716+0000", 2024-08-17T17:46:57.622 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.8620000000000003e-06 2024-08-17T17:46:57.622 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.622 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.622 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.623 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.620796+0000", 2024-08-17T17:46:57.623 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.0147000000000004e-05 2024-08-17T17:46:57.623 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.623 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.623 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.623 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.620810+0000", 2024-08-17T17:46:57.623 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.324e-05 2024-08-17T17:46:57.623 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.623 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.623 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.623 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.514494+0000", 2024-08-17T17:46:57.623 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.89368398100000002 2024-08-17T17:46:57.623 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.624 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.624 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.624 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.514497+0000", 2024-08-17T17:46:57.624 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.304e-06 2024-08-17T17:46:57.624 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.624 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.624 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.624 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.750203+0000", 2024-08-17T17:46:57.624 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.235705751 2024-08-17T17:46:57.624 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.624 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.624 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:57.624 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.750339+0000", 2024-08-17T17:46:57.624 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00013675300000000001 2024-08-17T17:46:57.624 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.625 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.625 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,5", 2024-08-17T17:46:57.625 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.750487+0000", 2024-08-17T17:46:57.625 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000147341 2024-08-17T17:46:57.625 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.625 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.625 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.625 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.766341+0000", 2024-08-17T17:46:57.625 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.015854653 2024-08-17T17:46:57.625 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.625 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.625 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:57.625 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.772271+0000", 2024-08-17T17:46:57.625 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0059293619999999997 2024-08-17T17:46:57.626 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.626 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.626 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.626 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.780644+0000", 2024-08-17T17:46:57.626 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0083733790000000002 2024-08-17T17:46:57.626 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.626 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.626 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:57.626 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.780707+0000", 2024-08-17T17:46:57.626 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.3093000000000001e-05 2024-08-17T17:46:57.626 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.626 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.626 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:57.627 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.785109+0000", 2024-08-17T17:46:57.627 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0044019319999999999 2024-08-17T17:46:57.627 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.627 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:57.627 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.627 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.627 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.627 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:94214 10.7 10:e7d22d14:::benchmark_data_smithi003_292749_object94213:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-08-17T17:46:57.627 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:51.751239+0000", 2024-08-17T17:46:57.627 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 185.74993362699999, 2024-08-17T17:46:57.627 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.0391442789999998, 2024-08-17T17:46:57.627 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:57.627 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:57.627 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:57.627 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:57.628 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:57.628 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 94214 2024-08-17T17:46:57.628 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.628 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:57.628 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.628 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:57.628 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.751239+0000", 2024-08-17T17:46:57.628 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.628 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.628 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.628 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:57.628 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.751239+0000", 2024-08-17T17:46:57.628 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.628 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.629 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.629 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:57.629 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.751238+0000", 2024-08-17T17:46:57.629 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.999999 2024-08-17T17:46:57.629 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.629 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.629 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:57.629 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.751317+0000", 2024-08-17T17:46:57.629 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.9271999999999996e-05 2024-08-17T17:46:57.629 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.629 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.629 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:57.629 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.751335+0000", 2024-08-17T17:46:57.629 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.8043e-05 2024-08-17T17:46:57.630 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.630 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.630 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:57.630 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.751342+0000", 2024-08-17T17:46:57.630 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.6200000000000001e-06 2024-08-17T17:46:57.630 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.630 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.630 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.630 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.751422+0000", 2024-08-17T17:46:57.630 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.9921999999999998e-05 2024-08-17T17:46:57.630 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.630 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.630 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.630 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.751432+0000", 2024-08-17T17:46:57.631 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.0584000000000001e-05 2024-08-17T17:46:57.631 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.631 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.631 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.631 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.514516+0000", 2024-08-17T17:46:57.631 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.76308369499999995 2024-08-17T17:46:57.631 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.631 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.631 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.631 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.514525+0000", 2024-08-17T17:46:57.631 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 8.9029999999999993e-06 2024-08-17T17:46:57.631 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.631 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.632 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.632 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.758113+0000", 2024-08-17T17:46:57.632 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2435874440000001 2024-08-17T17:46:57.632 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.632 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.632 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:57.632 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.758219+0000", 2024-08-17T17:46:57.632 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000106953 2024-08-17T17:46:57.632 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.632 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.632 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,5", 2024-08-17T17:46:57.632 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.758331+0000", 2024-08-17T17:46:57.632 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000111155 2024-08-17T17:46:57.632 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.633 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.633 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.633 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.772258+0000", 2024-08-17T17:46:57.633 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.013927725 2024-08-17T17:46:57.633 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.633 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.633 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:57.633 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.780586+0000", 2024-08-17T17:46:57.633 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0083273059999999996 2024-08-17T17:46:57.633 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.633 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.633 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.633 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.785207+0000", 2024-08-17T17:46:57.633 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0046215099999999997 2024-08-17T17:46:57.634 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.634 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.634 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:57.634 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.785323+0000", 2024-08-17T17:46:57.634 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000116111 2024-08-17T17:46:57.634 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.634 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.634 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:57.634 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.790383+0000", 2024-08-17T17:46:57.634 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0050600350000000001 2024-08-17T17:46:57.634 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.634 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:57.634 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.634 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.635 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.635 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:94231 10.7 10:e6202af2:::benchmark_data_smithi003_292749_object94230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-08-17T17:46:57.635 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:51.774390+0000", 2024-08-17T17:46:57.635 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 185.726782614, 2024-08-17T17:46:57.635 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 2.057356221, 2024-08-17T17:46:57.635 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:57.635 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:57.635 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:57.635 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:57.635 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:57.635 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 94231 2024-08-17T17:46:57.635 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.635 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:57.635 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.636 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:57.636 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.774390+0000", 2024-08-17T17:46:57.636 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.636 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.636 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.636 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:57.636 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.774390+0000", 2024-08-17T17:46:57.636 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.636 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.636 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.636 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:57.636 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.774389+0000", 2024-08-17T17:46:57.636 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:46:57.636 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.637 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.637 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:57.637 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.774436+0000", 2024-08-17T17:46:57.637 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.7216000000000003e-05 2024-08-17T17:46:57.637 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.637 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.637 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:57.637 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.774451+0000", 2024-08-17T17:46:57.637 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.5264000000000002e-05 2024-08-17T17:46:57.637 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.637 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.637 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:57.637 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.774457+0000", 2024-08-17T17:46:57.638 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6300000000000003e-06 2024-08-17T17:46:57.638 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.638 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.638 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.638 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.774498+0000", 2024-08-17T17:46:57.638 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1285999999999997e-05 2024-08-17T17:46:57.638 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.638 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.638 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.638 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.774508+0000", 2024-08-17T17:46:57.638 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.5889999999999992e-06 2024-08-17T17:46:57.638 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.639 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.639 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.639 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.514556+0000", 2024-08-17T17:46:57.639 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.74004848999999995 2024-08-17T17:46:57.639 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.639 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.639 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.639 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.514570+0000", 2024-08-17T17:46:57.639 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3509000000000001e-05 2024-08-17T17:46:57.639 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.639 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.639 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.639 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.790422+0000", 2024-08-17T17:46:57.639 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2758518249999999 2024-08-17T17:46:57.640 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.640 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.640 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:57.640 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.790587+0000", 2024-08-17T17:46:57.640 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00016544500000000001 2024-08-17T17:46:57.640 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.640 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.640 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,5", 2024-08-17T17:46:57.640 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.790765+0000", 2024-08-17T17:46:57.640 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00017827300000000001 2024-08-17T17:46:57.640 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.640 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.640 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:57.641 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.808507+0000", 2024-08-17T17:46:57.641 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.017741236000000001 2024-08-17T17:46:57.641 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.641 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.641 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.641 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.808572+0000", 2024-08-17T17:46:57.641 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.5092000000000006e-05 2024-08-17T17:46:57.641 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.641 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.641 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.641 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.827254+0000", 2024-08-17T17:46:57.641 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.018682417999999999 2024-08-17T17:46:57.641 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.642 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.642 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:57.642 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.827296+0000", 2024-08-17T17:46:57.642 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.1655e-05 2024-08-17T17:46:57.642 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.642 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.642 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:57.642 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.831746+0000", 2024-08-17T17:46:57.642 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0044505220000000002 2024-08-17T17:46:57.642 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.642 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:57.642 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.642 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.643 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.643 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:94253 10.7 10:e9d77991:::benchmark_data_smithi003_292749_object94252:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-08-17T17:46:57.643 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:51.909765+0000", 2024-08-17T17:46:57.643 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 185.59140780800001, 2024-08-17T17:46:57.643 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.9273695, 2024-08-17T17:46:57.643 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:57.643 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:57.643 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:57.643 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:57.643 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:57.643 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 94253 2024-08-17T17:46:57.643 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.643 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:57.643 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.643 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:57.644 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.909765+0000", 2024-08-17T17:46:57.644 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.644 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.644 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.644 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:57.644 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.909765+0000", 2024-08-17T17:46:57.644 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.644 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.644 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.644 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:57.644 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.909764+0000", 2024-08-17T17:46:57.644 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:46:57.644 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.644 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.645 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:57.645 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.909820+0000", 2024-08-17T17:46:57.645 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.6060000000000002e-05 2024-08-17T17:46:57.645 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.645 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.645 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:57.645 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.909836+0000", 2024-08-17T17:46:57.645 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.6614000000000001e-05 2024-08-17T17:46:57.645 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.645 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.645 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:57.645 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.909844+0000", 2024-08-17T17:46:57.645 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.4220000000000003e-06 2024-08-17T17:46:57.645 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.646 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.646 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.646 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.909899+0000", 2024-08-17T17:46:57.646 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 5.5782000000000002e-05 2024-08-17T17:46:57.646 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.646 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.646 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.646 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.909913+0000", 2024-08-17T17:46:57.646 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.3859999999999999e-05 2024-08-17T17:46:57.646 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.646 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.646 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.647 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.514595+0000", 2024-08-17T17:46:57.647 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.60468150499999995 2024-08-17T17:46:57.647 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.647 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.647 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.647 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.514598+0000", 2024-08-17T17:46:57.647 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.4999999999999999e-06 2024-08-17T17:46:57.647 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.647 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.647 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.647 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.793634+0000", 2024-08-17T17:46:57.647 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2790361269999999 2024-08-17T17:46:57.647 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.648 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.648 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:57.648 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.793793+0000", 2024-08-17T17:46:57.648 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00015865000000000001 2024-08-17T17:46:57.648 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.648 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.648 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,5", 2024-08-17T17:46:57.648 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.794002+0000", 2024-08-17T17:46:57.648 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.00020925699999999999 2024-08-17T17:46:57.648 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.648 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.648 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:57.648 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.808514+0000", 2024-08-17T17:46:57.649 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.014511679 2024-08-17T17:46:57.649 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.649 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.649 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.649 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.808629+0000", 2024-08-17T17:46:57.649 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000114967 2024-08-17T17:46:57.649 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.649 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.649 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.649 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.831868+0000", 2024-08-17T17:46:57.649 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.023239152999999999 2024-08-17T17:46:57.649 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.649 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.649 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:57.650 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.831933+0000", 2024-08-17T17:46:57.650 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.4576e-05 2024-08-17T17:46:57.650 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.650 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.650 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:57.650 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.837134+0000", 2024-08-17T17:46:57.650 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.005201659 2024-08-17T17:46:57.650 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.650 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:57.650 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.650 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.650 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.651 INFO:teuthology.orchestra.run.smithi053.stdout: "description": "osd_op(client.36612.0:94262 10.7 10:e45da416:::benchmark_data_smithi003_292749_object94261:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-08-17T17:46:57.651 INFO:teuthology.orchestra.run.smithi053.stdout: "initiated_at": "2024-08-17T17:43:51.935696+0000", 2024-08-17T17:46:57.651 INFO:teuthology.orchestra.run.smithi053.stdout: "age": 185.565476588, 2024-08-17T17:46:57.651 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.911659644, 2024-08-17T17:46:57.651 INFO:teuthology.orchestra.run.smithi053.stdout: "type_data": { 2024-08-17T17:46:57.651 INFO:teuthology.orchestra.run.smithi053.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-17T17:46:57.651 INFO:teuthology.orchestra.run.smithi053.stdout: "client_info": { 2024-08-17T17:46:57.651 INFO:teuthology.orchestra.run.smithi053.stdout: "client": "client.36612", 2024-08-17T17:46:57.651 INFO:teuthology.orchestra.run.smithi053.stdout: "client_addr": "172.21.15.3:0/114736052", 2024-08-17T17:46:57.651 INFO:teuthology.orchestra.run.smithi053.stdout: "tid": 94262 2024-08-17T17:46:57.651 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.651 INFO:teuthology.orchestra.run.smithi053.stdout: "events": [ 2024-08-17T17:46:57.651 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.651 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "initiated", 2024-08-17T17:46:57.652 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.935696+0000", 2024-08-17T17:46:57.652 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.652 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.652 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.652 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "throttled", 2024-08-17T17:46:57.652 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.935696+0000", 2024-08-17T17:46:57.652 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0 2024-08-17T17:46:57.652 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.652 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.652 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "header_read", 2024-08-17T17:46:57.652 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.935695+0000", 2024-08-17T17:46:57.652 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4294967295.9999986 2024-08-17T17:46:57.652 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.652 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.653 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "all_read", 2024-08-17T17:46:57.653 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.935744+0000", 2024-08-17T17:46:57.653 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.9852999999999999e-05 2024-08-17T17:46:57.653 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.653 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.653 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "dispatched", 2024-08-17T17:46:57.653 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.935756+0000", 2024-08-17T17:46:57.653 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.1897e-05 2024-08-17T17:46:57.653 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.653 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.653 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "queued_for_pg", 2024-08-17T17:46:57.653 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.935763+0000", 2024-08-17T17:46:57.653 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 6.1870000000000002e-06 2024-08-17T17:46:57.654 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.654 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.654 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.654 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.935805+0000", 2024-08-17T17:46:57.654 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 4.2978000000000001e-05 2024-08-17T17:46:57.654 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.654 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.654 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.654 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:51.935813+0000", 2024-08-17T17:46:57.654 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 7.5050000000000001e-06 2024-08-17T17:46:57.654 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.654 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.655 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.655 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.514613+0000", 2024-08-17T17:46:57.655 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.57880043199999998 2024-08-17T17:46:57.655 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.655 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.655 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for peered", 2024-08-17T17:46:57.655 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:52.514617+0000", 2024-08-17T17:46:57.655 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 3.9330000000000003e-06 2024-08-17T17:46:57.655 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.655 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.655 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "reached_pg", 2024-08-17T17:46:57.655 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.796769+0000", 2024-08-17T17:46:57.655 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 1.2821511999999999 2024-08-17T17:46:57.655 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.655 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.656 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "started", 2024-08-17T17:46:57.656 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.796891+0000", 2024-08-17T17:46:57.656 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000122409 2024-08-17T17:46:57.656 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.656 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.656 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "waiting for subops from 3,5", 2024-08-17T17:46:57.656 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.797048+0000", 2024-08-17T17:46:57.656 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.000157369 2024-08-17T17:46:57.656 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.656 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.656 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "op_commit", 2024-08-17T17:46:57.656 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.808636+0000", 2024-08-17T17:46:57.656 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.011588090000000001 2024-08-17T17:46:57.657 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.657 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.657 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.657 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.811716+0000", 2024-08-17T17:46:57.657 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0030795129999999999 2024-08-17T17:46:57.657 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.657 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.657 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "sub_op_commit_rec", 2024-08-17T17:46:57.657 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.840767+0000", 2024-08-17T17:46:57.657 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.029050761000000001 2024-08-17T17:46:57.657 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.657 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.657 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "commit_sent", 2024-08-17T17:46:57.658 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.840865+0000", 2024-08-17T17:46:57.658 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 9.8509000000000002e-05 2024-08-17T17:46:57.658 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-08-17T17:46:57.658 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-08-17T17:46:57.658 INFO:teuthology.orchestra.run.smithi053.stdout: "event": "done", 2024-08-17T17:46:57.658 INFO:teuthology.orchestra.run.smithi053.stdout: "time": "2024-08-17T17:43:53.847356+0000", 2024-08-17T17:46:57.658 INFO:teuthology.orchestra.run.smithi053.stdout: "duration": 0.0064904849999999998 2024-08-17T17:46:57.658 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.658 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:57.658 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.658 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-08-17T17:46:57.658 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-08-17T17:46:57.658 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-08-17T17:46:57.952 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:57 smithi003 ceph-mon[103344]: pgmap v1346: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:46:57.952 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:57 smithi003 ceph-mon[100615]: pgmap v1346: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:46:57.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:57 smithi053 ceph-mon[89963]: pgmap v1346: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:46:58.387 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-17T17:46:58.654 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:46:58.829 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-08-17T17:46:58.829 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-17T17:46:58.852 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-08-17T17:46:58.852 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-08-17T17:46:58.868 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-08-17T17:46:58.869 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-17T17:46:58.884 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-08-17T17:46:58.885 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-08-17T17:46:58.900 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-08-17T17:46:58.901 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-08-17T17:46:58.916 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-08-17T17:46:58.916 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-08-17T17:46:58.932 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-08-17T17:46:58.933 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-17T17:46:58.949 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-08-17T17:46:58.950 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-08-17T17:46:59.794 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:59 smithi053 ceph-mon[89963]: pgmap v1347: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:46:59.794 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:46:59 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2108309967' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:46:59.971 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:59 smithi003 ceph-mon[103344]: pgmap v1347: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:46:59.971 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:46:59 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2108309967' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:46:59.971 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:59 smithi003 ceph-mon[100615]: pgmap v1347: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:46:59.971 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:46:59 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2108309967' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-17T17:47:00.307 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:47:00.307 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:47:00.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:47:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:47:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:47:00.913 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-17T17:47:01.181 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:47:01.303 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:47:01.304 INFO:teuthology.orchestra.run.smithi003.stdout:{"fsid":"2b5b662e-5cba-11ef-bccf-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false}},"mutes":[]},"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1962,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":404,"num_osds":8,"num_up_osds":8,"osd_up_since":1723915999,"num_in_osds":8,"osd_in_since":1723916493,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":103}],"num_pgs":103,"num_pools":4,"num_objects":28,"data_bytes":4790221,"bytes_used":6363099136,"bytes_avail":761597186048,"bytes_total":767960285184},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi003.civgvf","status":"up:active","gid":14664}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.3:8443/","prometheus":"http://172.21.15.3:9283/"}},"servicemap":{"epoch":31,"modified":"2024-08-17T17:46:52.444000+0000","services":{}},"progress_events":{"ad00300d-2105-48f7-824e-8d8c2b403745":{"message":"Global Recovery Event (5s)\n [===========================.] ","progress":0.99253731966018677,"add_to_ceph_s":true},"fcbeb9b7-7baf-488b-9fe4-c01aca441f9c":{"message":"Upgrade to 19.0.0-5836-gcd8bdbeb (4m)\n [============................] (remaining: 5m)","progress":0.44999998807907104,"add_to_ceph_s":true}}} 2024-08-17T17:47:01.759 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-08-17T17:47:01.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:01 smithi053 ceph-mon[89963]: pgmap v1348: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:01.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:01 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1568305627' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T17:47:01.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:01 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/619932076' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:47:02.017 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:01 smithi003 ceph-mon[103344]: pgmap v1348: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:02.017 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:01 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1568305627' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T17:47:02.017 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:01 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/619932076' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:47:02.017 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:01 smithi003 ceph-mon[100615]: pgmap v1348: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:02.017 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:01 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1568305627' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-08-17T17:47:02.017 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:01 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/619932076' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:47:02.047 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-08-17T17:47:02.047 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:47:02.753 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:47:02.821 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:47:02.821 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:47:02.829 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:02 smithi053 ceph-mon[89963]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:47:02.830 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:02 smithi053 ceph-mon[89963]: Cluster is now healthy 2024-08-17T17:47:02.830 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:02 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/619932076' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:47:02.830 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:02 smithi053 ceph-mon[89963]: osdmap e405: 8 total, 8 up, 8 in 2024-08-17T17:47:03.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:02 smithi003 ceph-mon[103344]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:47:03.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:02 smithi003 ceph-mon[103344]: Cluster is now healthy 2024-08-17T17:47:03.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:02 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/619932076' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:47:03.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:02 smithi003 ceph-mon[103344]: osdmap e405: 8 total, 8 up, 8 in 2024-08-17T17:47:03.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:02 smithi003 ceph-mon[100615]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-17T17:47:03.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:02 smithi003 ceph-mon[100615]: Cluster is now healthy 2024-08-17T17:47:03.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:02 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/619932076' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:47:03.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:02 smithi003 ceph-mon[100615]: osdmap e405: 8 total, 8 up, 8 in 2024-08-17T17:47:03.475 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-17T17:47:03.755 INFO:teuthology.orchestra.run.smithi053.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.b/config 2024-08-17T17:47:03.954 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:03 smithi003 ceph-mon[103344]: pgmap v1350: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:03.955 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:03 smithi003 ceph-mon[100615]: pgmap v1350: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:04.015 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:03 smithi053 ceph-mon[89963]: pgmap v1350: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:04.614 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset noscrub 2024-08-17T17:47:04.672 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:47:04.673 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:47:05.359 INFO:teuthology.orchestra.run.smithi053.stdout:ERROR: (22) Invalid argument 2024-08-17T17:47:05.359 INFO:teuthology.orchestra.run.smithi053.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:47:05.460 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1351,"stamp":"2024-08-17T17:47:04.446986+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790221,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":5978,"num_read_kb":12991,"num_write":907,"num_write_kb":12623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":194,"num_bytes_recovered":8399864,"num_keys_recovered":40,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5651,"ondisk_log_size":5651,"up":309,"acting":309,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":309,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6214024,"kb_used_data":56356,"kb_used_omap":37,"kb_used_meta":6157530,"kb_avail":743747192,"statfs":{"total":767960285184,"available":761597124608,"internally_reserved":0,"allocated":57708544,"data_stored":39481701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38729,"internal_metadata":6305310903},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.702506"},"pg_stats":[{"pgid":"3.1f","version":"71'4","reported_seq":504,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-08-17T17:47:01.764175+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:47:01.764175+0000","last_peered":"2024-08-17T17:47:01.764175+0000","last_clean":"2024-08-17T17:47:01.764175+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:47:01.764175+0000","last_undegraded":"2024-08-17T17:47:01.764175+0000","last_fullsized":"2024-08-17T17:47:01.764175+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":69,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.742264+0000","last_change":"2024-08-17T17:41:34.611711+0000","last_active":"2024-08-17T17:46:55.742264+0000","last_peered":"2024-08-17T17:46:55.742264+0000","last_clean":"2024-08-17T17:46:55.742264+0000","last_became_active":"2024-08-17T17:41:34.611407+0000","last_became_peered":"2024-08-17T17:41:34.611407+0000","last_unstale":"2024-08-17T17:46:55.742264+0000","last_undegraded":"2024-08-17T17:46:55.742264+0000","last_fullsized":"2024-08-17T17:46:55.742264+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:42:41.582320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1d","version":"90'149","reported_seq":757,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.163237+0000","last_change":"2024-08-17T17:42:18.668058+0000","last_active":"2024-08-17T17:46:55.163237+0000","last_peered":"2024-08-17T17:46:55.163237+0000","last_clean":"2024-08-17T17:46:55.163237+0000","last_became_active":"2024-08-17T17:42:05.876721+0000","last_became_peered":"2024-08-17T17:42:05.876721+0000","last_unstale":"2024-08-17T17:46:55.163237+0000","last_undegraded":"2024-08-17T17:46:55.163237+0000","last_fullsized":"2024-08-17T17:46:55.163237+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:18.667957+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:18.667957+0000","last_clean_scrub_stamp":"2024-08-17T17:42:18.667957+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:38.170146+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":692,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:54.748864+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:46:54.748864+0000","last_peered":"2024-08-17T17:46:54.748864+0000","last_clean":"2024-08-17T17:46:54.748864+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:46:54.748864+0000","last_undegraded":"2024-08-17T17:46:54.748864+0000","last_fullsized":"2024-08-17T17:46:54.748864+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":691,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:54.749659+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:46:54.749659+0000","last_peered":"2024-08-17T17:46:54.749659+0000","last_clean":"2024-08-17T17:46:54.749659+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:46:54.749659+0000","last_undegraded":"2024-08-17T17:46:54.749659+0000","last_fullsized":"2024-08-17T17:46:54.749659+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"90'149","reported_seq":745,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.722804+0000","last_change":"2024-08-17T17:42:11.809047+0000","last_active":"2024-08-17T17:46:55.722804+0000","last_peered":"2024-08-17T17:46:55.722804+0000","last_clean":"2024-08-17T17:46:55.722804+0000","last_became_active":"2024-08-17T17:42:05.883770+0000","last_became_peered":"2024-08-17T17:42:05.883770+0000","last_unstale":"2024-08-17T17:46:55.722804+0000","last_undegraded":"2024-08-17T17:46:55.722804+0000","last_fullsized":"2024-08-17T17:46:55.722804+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:11.808971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:11.808971+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:48:15.989437+0000","scrub_duration":0.0018167420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":73,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.744217+0000","last_change":"2024-08-17T17:41:34.631152+0000","last_active":"2024-08-17T17:46:55.744217+0000","last_peered":"2024-08-17T17:46:55.744217+0000","last_clean":"2024-08-17T17:46:55.744217+0000","last_became_active":"2024-08-17T17:41:34.615617+0000","last_became_peered":"2024-08-17T17:41:34.615617+0000","last_unstale":"2024-08-17T17:46:55.744217+0000","last_undegraded":"2024-08-17T17:46:55.744217+0000","last_fullsized":"2024-08-17T17:46:55.744217+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:43:08.257084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":69,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.744616+0000","last_change":"2024-08-17T17:41:34.616251+0000","last_active":"2024-08-17T17:46:55.744616+0000","last_peered":"2024-08-17T17:46:55.744616+0000","last_clean":"2024-08-17T17:46:55.744616+0000","last_became_active":"2024-08-17T17:41:34.615952+0000","last_became_peered":"2024-08-17T17:41:34.615952+0000","last_unstale":"2024-08-17T17:46:55.744616+0000","last_undegraded":"2024-08-17T17:46:55.744616+0000","last_fullsized":"2024-08-17T17:46:55.744616+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:12:24.892508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":73,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.745700+0000","last_change":"2024-08-17T17:41:34.629135+0000","last_active":"2024-08-17T17:46:55.745700+0000","last_peered":"2024-08-17T17:46:55.745700+0000","last_clean":"2024-08-17T17:46:55.745700+0000","last_became_active":"2024-08-17T17:41:34.613095+0000","last_became_peered":"2024-08-17T17:41:34.613095+0000","last_unstale":"2024-08-17T17:46:55.745700+0000","last_undegraded":"2024-08-17T17:46:55.745700+0000","last_fullsized":"2024-08-17T17:46:55.745700+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:27:06.542416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":69,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.746364+0000","last_change":"2024-08-17T17:41:34.622094+0000","last_active":"2024-08-17T17:46:55.746364+0000","last_peered":"2024-08-17T17:46:55.746364+0000","last_clean":"2024-08-17T17:46:55.746364+0000","last_became_active":"2024-08-17T17:41:34.621773+0000","last_became_peered":"2024-08-17T17:41:34.621773+0000","last_unstale":"2024-08-17T17:46:55.746364+0000","last_undegraded":"2024-08-17T17:46:55.746364+0000","last_fullsized":"2024-08-17T17:46:55.746364+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:34:30.540886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"90'149","reported_seq":510,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:54.748710+0000","last_change":"2024-08-17T17:42:06.742801+0000","last_active":"2024-08-17T17:46:54.748710+0000","last_peered":"2024-08-17T17:46:54.748710+0000","last_clean":"2024-08-17T17:46:54.748710+0000","last_became_active":"2024-08-17T17:42:05.876895+0000","last_became_peered":"2024-08-17T17:42:05.876895+0000","last_unstale":"2024-08-17T17:46:54.748710+0000","last_undegraded":"2024-08-17T17:46:54.748710+0000","last_fullsized":"2024-08-17T17:46:54.748710+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.742723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.742723+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:36:46.651953+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":508,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-08-17T17:47:01.764050+0000","last_change":"2024-08-17T17:41:34.618968+0000","last_active":"2024-08-17T17:47:01.764050+0000","last_peered":"2024-08-17T17:47:01.764050+0000","last_clean":"2024-08-17T17:47:01.764050+0000","last_became_active":"2024-08-17T17:41:34.618576+0000","last_became_peered":"2024-08-17T17:41:34.618576+0000","last_unstale":"2024-08-17T17:47:01.764050+0000","last_undegraded":"2024-08-17T17:47:01.764050+0000","last_fullsized":"2024-08-17T17:47:01.764050+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:47:45.888590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":599,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.163473+0000","last_change":"2024-08-17T17:33:20.773207+0000","last_active":"2024-08-17T17:46:55.163473+0000","last_peered":"2024-08-17T17:46:55.163473+0000","last_clean":"2024-08-17T17:46:55.163473+0000","last_became_active":"2024-08-17T17:33:20.772457+0000","last_became_peered":"2024-08-17T17:33:20.772457+0000","last_unstale":"2024-08-17T17:46:55.163473+0000","last_undegraded":"2024-08-17T17:46:55.163473+0000","last_fullsized":"2024-08-17T17:46:55.163473+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:57:09.610084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"90'149","reported_seq":862,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:54.747005+0000","last_change":"2024-08-17T17:42:05.876463+0000","last_active":"2024-08-17T17:46:54.747005+0000","last_peered":"2024-08-17T17:46:54.747005+0000","last_clean":"2024-08-17T17:46:54.747005+0000","last_became_active":"2024-08-17T17:42:05.876069+0000","last_became_peered":"2024-08-17T17:42:05.876069+0000","last_unstale":"2024-08-17T17:46:54.747005+0000","last_undegraded":"2024-08-17T17:46:54.747005+0000","last_fullsized":"2024-08-17T17:46:54.747005+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:45.552985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:45.552985+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:55:06.027278+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":69,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.744405+0000","last_change":"2024-08-17T17:41:34.622184+0000","last_active":"2024-08-17T17:46:55.744405+0000","last_peered":"2024-08-17T17:46:55.744405+0000","last_clean":"2024-08-17T17:46:55.744405+0000","last_became_active":"2024-08-17T17:41:34.621831+0000","last_became_peered":"2024-08-17T17:41:34.621831+0000","last_unstale":"2024-08-17T17:46:55.744405+0000","last_undegraded":"2024-08-17T17:46:55.744405+0000","last_fullsized":"2024-08-17T17:46:55.744405+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:59.826667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":69,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.744579+0000","last_change":"2024-08-17T17:41:34.616482+0000","last_active":"2024-08-17T17:46:55.744579+0000","last_peered":"2024-08-17T17:46:55.744579+0000","last_clean":"2024-08-17T17:46:55.744579+0000","last_became_active":"2024-08-17T17:41:34.616040+0000","last_became_peered":"2024-08-17T17:41:34.616040+0000","last_unstale":"2024-08-17T17:46:55.744579+0000","last_undegraded":"2024-08-17T17:46:55.744579+0000","last_fullsized":"2024-08-17T17:46:55.744579+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:53:08.502264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.18","version":"90'149","reported_seq":711,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.741870+0000","last_change":"2024-08-17T17:42:08.482975+0000","last_active":"2024-08-17T17:46:55.741870+0000","last_peered":"2024-08-17T17:46:55.741870+0000","last_clean":"2024-08-17T17:46:55.741870+0000","last_became_active":"2024-08-17T17:42:05.873523+0000","last_became_peered":"2024-08-17T17:42:05.873523+0000","last_unstale":"2024-08-17T17:46:55.741870+0000","last_undegraded":"2024-08-17T17:46:55.741870+0000","last_fullsized":"2024-08-17T17:46:55.741870+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:08.482892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:08.482892+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:41.477341+0000","scrub_duration":0.0017925479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":632,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.162426+0000","last_change":"2024-08-17T17:41:34.625011+0000","last_active":"2024-08-17T17:46:55.162426+0000","last_peered":"2024-08-17T17:46:55.162426+0000","last_clean":"2024-08-17T17:46:55.162426+0000","last_became_active":"2024-08-17T17:41:34.624631+0000","last_became_peered":"2024-08-17T17:41:34.624631+0000","last_unstale":"2024-08-17T17:46:55.162426+0000","last_undegraded":"2024-08-17T17:46:55.162426+0000","last_fullsized":"2024-08-17T17:46:55.162426+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:03:05.565435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":485,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-08-17T17:47:01.764072+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:47:01.764072+0000","last_peered":"2024-08-17T17:47:01.764072+0000","last_clean":"2024-08-17T17:47:01.764072+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:47:01.764072+0000","last_undegraded":"2024-08-17T17:47:01.764072+0000","last_fullsized":"2024-08-17T17:47:01.764072+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1b","version":"90'149","reported_seq":748,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.746423+0000","last_change":"2024-08-17T17:42:26.967686+0000","last_active":"2024-08-17T17:46:55.746423+0000","last_peered":"2024-08-17T17:46:55.746423+0000","last_clean":"2024-08-17T17:46:55.746423+0000","last_became_active":"2024-08-17T17:42:05.873657+0000","last_became_peered":"2024-08-17T17:42:05.873657+0000","last_unstale":"2024-08-17T17:46:55.746423+0000","last_undegraded":"2024-08-17T17:46:55.746423+0000","last_fullsized":"2024-08-17T17:46:55.746423+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:26.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:26.967611+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:31:03.344332+0000","scrub_duration":0.0015978279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":713,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:54.746714+0000","last_change":"2024-08-17T17:33:20.771279+0000","last_active":"2024-08-17T17:46:54.746714+0000","last_peered":"2024-08-17T17:46:54.746714+0000","last_clean":"2024-08-17T17:46:54.746714+0000","last_became_active":"2024-08-17T17:33:20.770728+0000","last_became_peered":"2024-08-17T17:33:20.770728+0000","last_unstale":"2024-08-17T17:46:54.746714+0000","last_undegraded":"2024-08-17T17:46:54.746714+0000","last_fullsized":"2024-08-17T17:46:54.746714+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:52:31.540397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":378,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.724493+0000","last_change":"2024-08-17T17:33:20.835760+0000","last_active":"2024-08-17T17:46:55.724493+0000","last_peered":"2024-08-17T17:46:55.724493+0000","last_clean":"2024-08-17T17:46:55.724493+0000","last_became_active":"2024-08-17T17:33:20.835410+0000","last_became_peered":"2024-08-17T17:33:20.835410+0000","last_unstale":"2024-08-17T17:46:55.724493+0000","last_undegraded":"2024-08-17T17:46:55.724493+0000","last_fullsized":"2024-08-17T17:46:55.724493+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:45.448022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1a","version":"90'149","reported_seq":453,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.744182+0000","last_change":"2024-08-17T17:42:21.932830+0000","last_active":"2024-08-17T17:46:55.744182+0000","last_peered":"2024-08-17T17:46:55.744182+0000","last_clean":"2024-08-17T17:46:55.744182+0000","last_became_active":"2024-08-17T17:42:05.876835+0000","last_became_peered":"2024-08-17T17:42:05.876835+0000","last_unstale":"2024-08-17T17:46:55.744182+0000","last_undegraded":"2024-08-17T17:46:55.744182+0000","last_fullsized":"2024-08-17T17:46:55.744182+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:21.932757+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:21.932757+0000","last_clean_scrub_stamp":"2024-08-17T17:42:21.932757+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:36:39.848570+0000","scrub_duration":0.0020218810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":695,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:54.746537+0000","last_change":"2024-08-17T17:33:20.790687+0000","last_active":"2024-08-17T17:46:54.746537+0000","last_peered":"2024-08-17T17:46:54.746537+0000","last_clean":"2024-08-17T17:46:54.746537+0000","last_became_active":"2024-08-17T17:33:20.790399+0000","last_became_peered":"2024-08-17T17:33:20.790399+0000","last_unstale":"2024-08-17T17:46:54.746537+0000","last_undegraded":"2024-08-17T17:46:54.746537+0000","last_fullsized":"2024-08-17T17:46:54.746537+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:38:06.849989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":666,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-08-17T17:47:01.764188+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:47:01.764188+0000","last_peered":"2024-08-17T17:47:01.764188+0000","last_clean":"2024-08-17T17:47:01.764188+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:47:01.764188+0000","last_undegraded":"2024-08-17T17:47:01.764188+0000","last_fullsized":"2024-08-17T17:47:01.764188+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"90'149","reported_seq":791,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:54.747869+0000","last_change":"2024-08-17T17:42:06.456562+0000","last_active":"2024-08-17T17:46:54.747869+0000","last_peered":"2024-08-17T17:46:54.747869+0000","last_clean":"2024-08-17T17:46:54.747869+0000","last_became_active":"2024-08-17T17:42:05.876577+0000","last_became_peered":"2024-08-17T17:42:05.876577+0000","last_unstale":"2024-08-17T17:46:54.747869+0000","last_undegraded":"2024-08-17T17:46:54.747869+0000","last_fullsized":"2024-08-17T17:46:54.747869+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.456490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.456490+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:10:03.654953+0000","scrub_duration":0.001537285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":637,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:54.747241+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:46:54.747241+0000","last_peered":"2024-08-17T17:46:54.747241+0000","last_clean":"2024-08-17T17:46:54.747241+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:46:54.747241+0000","last_undegraded":"2024-08-17T17:46:54.747241+0000","last_fullsized":"2024-08-17T17:46:54.747241+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":372,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.163844+0000","last_change":"2024-08-17T17:33:44.146014+0000","last_active":"2024-08-17T17:46:55.163844+0000","last_peered":"2024-08-17T17:46:55.163844+0000","last_clean":"2024-08-17T17:46:55.163844+0000","last_became_active":"2024-08-17T17:33:44.145668+0000","last_became_peered":"2024-08-17T17:33:44.145668+0000","last_unstale":"2024-08-17T17:46:55.163844+0000","last_undegraded":"2024-08-17T17:46:55.163844+0000","last_fullsized":"2024-08-17T17:46:55.163844+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:53:41.679237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.14","version":"90'149","reported_seq":860,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-08-17T17:47:02.138159+0000","last_change":"2024-08-17T17:41:37.664471+0000","last_active":"2024-08-17T17:47:02.138159+0000","last_peered":"2024-08-17T17:47:02.138159+0000","last_clean":"2024-08-17T17:47:02.138159+0000","last_became_active":"2024-08-17T17:41:37.664108+0000","last_became_peered":"2024-08-17T17:41:37.664108+0000","last_unstale":"2024-08-17T17:47:02.138159+0000","last_undegraded":"2024-08-17T17:47:02.138159+0000","last_fullsized":"2024-08-17T17:47:02.138159+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:22.030514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:22.030514+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:15:52.325163+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":409,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:54.747504+0000","last_change":"2024-08-17T17:41:34.634301+0000","last_active":"2024-08-17T17:46:54.747504+0000","last_peered":"2024-08-17T17:46:54.747504+0000","last_clean":"2024-08-17T17:46:54.747504+0000","last_became_active":"2024-08-17T17:41:34.620508+0000","last_became_peered":"2024-08-17T17:41:34.620508+0000","last_unstale":"2024-08-17T17:46:54.747504+0000","last_undegraded":"2024-08-17T17:46:54.747504+0000","last_fullsized":"2024-08-17T17:46:54.747504+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:10:54.576812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":633,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.162490+0000","last_change":"2024-08-17T17:33:20.799402+0000","last_active":"2024-08-17T17:46:55.162490+0000","last_peered":"2024-08-17T17:46:55.162490+0000","last_clean":"2024-08-17T17:46:55.162490+0000","last_became_active":"2024-08-17T17:33:20.798832+0000","last_became_peered":"2024-08-17T17:33:20.798832+0000","last_unstale":"2024-08-17T17:46:55.162490+0000","last_undegraded":"2024-08-17T17:46:55.162490+0000","last_fullsized":"2024-08-17T17:46:55.162490+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:21:05.001134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"90'149","reported_seq":507,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-08-17T17:47:01.763950+0000","last_change":"2024-08-17T17:42:06.699082+0000","last_active":"2024-08-17T17:47:01.763950+0000","last_peered":"2024-08-17T17:47:01.763950+0000","last_clean":"2024-08-17T17:47:01.763950+0000","last_became_active":"2024-08-17T17:42:05.876218+0000","last_became_peered":"2024-08-17T17:42:05.876218+0000","last_unstale":"2024-08-17T17:47:01.763950+0000","last_undegraded":"2024-08-17T17:47:01.763950+0000","last_fullsized":"2024-08-17T17:47:01.763950+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.698998+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:06.698998+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.698998+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:22:16.510150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":79,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.743757+0000","last_change":"2024-08-17T17:41:34.626584+0000","last_active":"2024-08-17T17:46:55.743757+0000","last_peered":"2024-08-17T17:46:55.743757+0000","last_clean":"2024-08-17T17:46:55.743757+0000","last_became_active":"2024-08-17T17:41:34.609878+0000","last_became_peered":"2024-08-17T17:41:34.609878+0000","last_unstale":"2024-08-17T17:46:55.743757+0000","last_undegraded":"2024-08-17T17:46:55.743757+0000","last_fullsized":"2024-08-17T17:46:55.743757+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:08:12.810347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":69,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.746504+0000","last_change":"2024-08-17T17:41:34.622180+0000","last_active":"2024-08-17T17:46:55.746504+0000","last_peered":"2024-08-17T17:46:55.746504+0000","last_clean":"2024-08-17T17:46:55.746504+0000","last_became_active":"2024-08-17T17:41:34.621724+0000","last_became_peered":"2024-08-17T17:41:34.621724+0000","last_unstale":"2024-08-17T17:46:55.746504+0000","last_undegraded":"2024-08-17T17:46:55.746504+0000","last_fullsized":"2024-08-17T17:46:55.746504+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:15.807235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"90'149","reported_seq":746,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.163638+0000","last_change":"2024-08-17T17:42:07.681204+0000","last_active":"2024-08-17T17:46:55.163638+0000","last_peered":"2024-08-17T17:46:55.163638+0000","last_clean":"2024-08-17T17:46:55.163638+0000","last_became_active":"2024-08-17T17:42:05.878769+0000","last_became_peered":"2024-08-17T17:42:05.878769+0000","last_unstale":"2024-08-17T17:46:55.163638+0000","last_undegraded":"2024-08-17T17:46:55.163638+0000","last_fullsized":"2024-08-17T17:46:55.163638+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:07.681064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:07.681064+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:12:12.835715+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":69,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.742555+0000","last_change":"2024-08-17T17:41:34.614028+0000","last_active":"2024-08-17T17:46:55.742555+0000","last_peered":"2024-08-17T17:46:55.742555+0000","last_clean":"2024-08-17T17:46:55.742555+0000","last_became_active":"2024-08-17T17:41:34.613706+0000","last_became_peered":"2024-08-17T17:41:34.613706+0000","last_unstale":"2024-08-17T17:46:55.742555+0000","last_undegraded":"2024-08-17T17:46:55.742555+0000","last_fullsized":"2024-08-17T17:46:55.742555+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:19:55.254902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":635,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.724561+0000","last_change":"2024-08-17T17:26:06.855307+0000","last_active":"2024-08-17T17:46:55.724561+0000","last_peered":"2024-08-17T17:46:55.724561+0000","last_clean":"2024-08-17T17:46:55.724561+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:46:55.724561+0000","last_undegraded":"2024-08-17T17:46:55.724561+0000","last_fullsized":"2024-08-17T17:46:55.724561+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":530,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-08-17T17:47:01.764336+0000","last_change":"2024-08-17T17:41:34.629522+0000","last_active":"2024-08-17T17:47:01.764336+0000","last_peered":"2024-08-17T17:47:01.764336+0000","last_clean":"2024-08-17T17:47:01.764336+0000","last_became_active":"2024-08-17T17:41:34.619340+0000","last_became_peered":"2024-08-17T17:41:34.619340+0000","last_unstale":"2024-08-17T17:47:01.764336+0000","last_undegraded":"2024-08-17T17:47:01.764336+0000","last_fullsized":"2024-08-17T17:47:01.764336+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:30:03.902956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":835,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:54.747029+0000","last_change":"2024-08-17T17:38:23.113397+0000","last_active":"2024-08-17T17:46:54.747029+0000","last_peered":"2024-08-17T17:46:54.747029+0000","last_clean":"2024-08-17T17:46:54.747029+0000","last_became_active":"2024-08-17T17:38:21.305580+0000","last_became_peered":"2024-08-17T17:38:21.305580+0000","last_unstale":"2024-08-17T17:46:54.747029+0000","last_undegraded":"2024-08-17T17:46:54.747029+0000","last_fullsized":"2024-08-17T17:46:54.747029+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.113279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.113279+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:38:07.015721+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":688,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:54.746637+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:46:54.746637+0000","last_peered":"2024-08-17T17:46:54.746637+0000","last_clean":"2024-08-17T17:46:54.746637+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:46:54.746637+0000","last_undegraded":"2024-08-17T17:46:54.746637+0000","last_fullsized":"2024-08-17T17:46:54.746637+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":1266,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.163754+0000","last_change":"2024-08-17T17:41:37.677524+0000","last_active":"2024-08-17T17:46:55.163754+0000","last_peered":"2024-08-17T17:46:55.163754+0000","last_clean":"2024-08-17T17:46:55.163754+0000","last_became_active":"2024-08-17T17:41:37.676764+0000","last_became_peered":"2024-08-17T17:41:37.676764+0000","last_unstale":"2024-08-17T17:46:55.163754+0000","last_undegraded":"2024-08-17T17:46:55.163754+0000","last_fullsized":"2024-08-17T17:46:55.163754+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:14:49.341080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":691,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:54.748884+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:46:54.748884+0000","last_peered":"2024-08-17T17:46:54.748884+0000","last_clean":"2024-08-17T17:46:54.748884+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:46:54.748884+0000","last_undegraded":"2024-08-17T17:46:54.748884+0000","last_fullsized":"2024-08-17T17:46:54.748884+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":788,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.744124+0000","last_change":"2024-08-17T17:42:19.960161+0000","last_active":"2024-08-17T17:46:55.744124+0000","last_peered":"2024-08-17T17:46:55.744124+0000","last_clean":"2024-08-17T17:46:55.744124+0000","last_became_active":"2024-08-17T17:41:37.665223+0000","last_became_peered":"2024-08-17T17:41:37.665223+0000","last_unstale":"2024-08-17T17:46:55.744124+0000","last_undegraded":"2024-08-17T17:46:55.744124+0000","last_fullsized":"2024-08-17T17:46:55.744124+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:19.960086+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:19.960086+0000","last_clean_scrub_stamp":"2024-08-17T17:42:19.960086+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:57.861661+0000","scrub_duration":0.0022612000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":658,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-08-17T17:47:01.764317+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:47:01.764317+0000","last_peered":"2024-08-17T17:47:01.764317+0000","last_clean":"2024-08-17T17:47:01.764317+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:47:01.764317+0000","last_undegraded":"2024-08-17T17:47:01.764317+0000","last_fullsized":"2024-08-17T17:47:01.764317+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":965,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:54.747996+0000","last_change":"2024-08-17T17:41:37.683212+0000","last_active":"2024-08-17T17:46:54.747996+0000","last_peered":"2024-08-17T17:46:54.747996+0000","last_clean":"2024-08-17T17:46:54.747996+0000","last_became_active":"2024-08-17T17:41:37.682730+0000","last_became_peered":"2024-08-17T17:41:37.682730+0000","last_unstale":"2024-08-17T17:46:54.747996+0000","last_undegraded":"2024-08-17T17:46:54.747996+0000","last_fullsized":"2024-08-17T17:46:54.747996+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:51:35.444433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":641,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.163219+0000","last_change":"2024-08-17T17:33:20.798613+0000","last_active":"2024-08-17T17:46:55.163219+0000","last_peered":"2024-08-17T17:46:55.163219+0000","last_clean":"2024-08-17T17:46:55.163219+0000","last_became_active":"2024-08-17T17:33:20.798260+0000","last_became_peered":"2024-08-17T17:33:20.798260+0000","last_unstale":"2024-08-17T17:46:55.163219+0000","last_undegraded":"2024-08-17T17:46:55.163219+0000","last_fullsized":"2024-08-17T17:46:55.163219+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:18:00.822681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":542,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.724283+0000","last_change":"2024-08-17T17:41:44.058291+0000","last_active":"2024-08-17T17:46:55.724283+0000","last_peered":"2024-08-17T17:46:55.724283+0000","last_clean":"2024-08-17T17:46:55.724283+0000","last_became_active":"2024-08-17T17:41:12.200077+0000","last_became_peered":"2024-08-17T17:41:12.200077+0000","last_unstale":"2024-08-17T17:46:55.724283+0000","last_undegraded":"2024-08-17T17:46:55.724283+0000","last_fullsized":"2024-08-17T17:46:55.724283+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:44.058233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:44.058233+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:42:55.911623+0000","scrub_duration":0.0074268499999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":637,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.162979+0000","last_change":"2024-08-17T17:41:34.627064+0000","last_active":"2024-08-17T17:46:55.162979+0000","last_peered":"2024-08-17T17:46:55.162979+0000","last_clean":"2024-08-17T17:46:55.162979+0000","last_became_active":"2024-08-17T17:41:34.626453+0000","last_became_peered":"2024-08-17T17:41:34.626453+0000","last_unstale":"2024-08-17T17:46:55.162979+0000","last_undegraded":"2024-08-17T17:46:55.162979+0000","last_fullsized":"2024-08-17T17:46:55.162979+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:02:11.894042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":821,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.163296+0000","last_change":"2024-08-17T17:33:20.799350+0000","last_active":"2024-08-17T17:46:55.163296+0000","last_peered":"2024-08-17T17:46:55.163296+0000","last_clean":"2024-08-17T17:46:55.163296+0000","last_became_active":"2024-08-17T17:33:20.798925+0000","last_became_peered":"2024-08-17T17:33:20.798925+0000","last_unstale":"2024-08-17T17:46:55.163296+0000","last_undegraded":"2024-08-17T17:46:55.163296+0000","last_fullsized":"2024-08-17T17:46:55.163296+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:16:36.121295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":357,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:54.747563+0000","last_change":"2024-08-17T17:33:44.163867+0000","last_active":"2024-08-17T17:46:54.747563+0000","last_peered":"2024-08-17T17:46:54.747563+0000","last_clean":"2024-08-17T17:46:54.747563+0000","last_became_active":"2024-08-17T17:33:44.163503+0000","last_became_peered":"2024-08-17T17:33:44.163503+0000","last_unstale":"2024-08-17T17:46:54.747563+0000","last_undegraded":"2024-08-17T17:46:54.747563+0000","last_fullsized":"2024-08-17T17:46:54.747563+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:31:53.573596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":542,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.724403+0000","last_change":"2024-08-17T17:41:43.035215+0000","last_active":"2024-08-17T17:46:55.724403+0000","last_peered":"2024-08-17T17:46:55.724403+0000","last_clean":"2024-08-17T17:46:55.724403+0000","last_became_active":"2024-08-17T17:41:12.201158+0000","last_became_peered":"2024-08-17T17:41:12.201158+0000","last_unstale":"2024-08-17T17:46:55.724403+0000","last_undegraded":"2024-08-17T17:46:55.724403+0000","last_fullsized":"2024-08-17T17:46:55.724403+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:43.035108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:43.035108+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:43:27.108994+0000","scrub_duration":0.0028877400000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":606,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.163590+0000","last_change":"2024-08-17T17:33:20.772454+0000","last_active":"2024-08-17T17:46:55.163590+0000","last_peered":"2024-08-17T17:46:55.163590+0000","last_clean":"2024-08-17T17:46:55.163590+0000","last_became_active":"2024-08-17T17:33:20.771848+0000","last_became_peered":"2024-08-17T17:33:20.771848+0000","last_unstale":"2024-08-17T17:46:55.163590+0000","last_undegraded":"2024-08-17T17:46:55.163590+0000","last_fullsized":"2024-08-17T17:46:55.163590+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:46:42.300744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":69,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.746442+0000","last_change":"2024-08-17T17:41:34.611109+0000","last_active":"2024-08-17T17:46:55.746442+0000","last_peered":"2024-08-17T17:46:55.746442+0000","last_clean":"2024-08-17T17:46:55.746442+0000","last_became_active":"2024-08-17T17:41:34.610170+0000","last_became_peered":"2024-08-17T17:41:34.610170+0000","last_unstale":"2024-08-17T17:46:55.746442+0000","last_undegraded":"2024-08-17T17:46:55.746442+0000","last_fullsized":"2024-08-17T17:46:55.746442+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:39:05.301260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":1151,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.162480+0000","last_change":"2024-08-17T17:33:44.148227+0000","last_active":"2024-08-17T17:46:55.162480+0000","last_peered":"2024-08-17T17:46:55.162480+0000","last_clean":"2024-08-17T17:46:55.162480+0000","last_became_active":"2024-08-17T17:33:44.147627+0000","last_became_peered":"2024-08-17T17:33:44.147627+0000","last_unstale":"2024-08-17T17:46:55.162480+0000","last_undegraded":"2024-08-17T17:46:55.162480+0000","last_fullsized":"2024-08-17T17:46:55.162480+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:29:42.898460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1372,"num_read_kb":2843,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":855,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-08-17T17:47:01.763713+0000","last_change":"2024-08-17T17:38:22.254841+0000","last_active":"2024-08-17T17:47:01.763713+0000","last_peered":"2024-08-17T17:47:01.763713+0000","last_clean":"2024-08-17T17:47:01.763713+0000","last_became_active":"2024-08-17T17:38:21.306619+0000","last_became_peered":"2024-08-17T17:38:21.306619+0000","last_unstale":"2024-08-17T17:47:01.763713+0000","last_undegraded":"2024-08-17T17:47:01.763713+0000","last_fullsized":"2024-08-17T17:47:01.763713+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:22.254705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:22.254705+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:47:38.184089+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":602,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.164262+0000","last_change":"2024-08-17T17:33:20.835784+0000","last_active":"2024-08-17T17:46:55.164262+0000","last_peered":"2024-08-17T17:46:55.164262+0000","last_clean":"2024-08-17T17:46:55.164262+0000","last_became_active":"2024-08-17T17:33:20.835371+0000","last_became_peered":"2024-08-17T17:33:20.835371+0000","last_unstale":"2024-08-17T17:46:55.164262+0000","last_undegraded":"2024-08-17T17:46:55.164262+0000","last_fullsized":"2024-08-17T17:46:55.164262+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:35:43.369055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":566,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.162385+0000","last_change":"2024-08-17T17:33:20.793468+0000","last_active":"2024-08-17T17:46:55.162385+0000","last_peered":"2024-08-17T17:46:55.162385+0000","last_clean":"2024-08-17T17:46:55.162385+0000","last_became_active":"2024-08-17T17:33:20.793166+0000","last_became_peered":"2024-08-17T17:33:20.793166+0000","last_unstale":"2024-08-17T17:46:55.162385+0000","last_undegraded":"2024-08-17T17:46:55.162385+0000","last_fullsized":"2024-08-17T17:46:55.162385+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:12:52.121050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":1061,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:54.749603+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:46:54.749603+0000","last_peered":"2024-08-17T17:46:54.749603+0000","last_clean":"2024-08-17T17:46:54.749603+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:46:54.749603+0000","last_undegraded":"2024-08-17T17:46:54.749603+0000","last_fullsized":"2024-08-17T17:46:54.749603+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":876,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-08-17T17:47:01.763650+0000","last_change":"2024-08-17T17:41:37.683450+0000","last_active":"2024-08-17T17:47:01.763650+0000","last_peered":"2024-08-17T17:47:01.763650+0000","last_clean":"2024-08-17T17:47:01.763650+0000","last_became_active":"2024-08-17T17:41:37.683103+0000","last_became_peered":"2024-08-17T17:41:37.683103+0000","last_unstale":"2024-08-17T17:47:01.763650+0000","last_undegraded":"2024-08-17T17:47:01.763650+0000","last_fullsized":"2024-08-17T17:47:01.763650+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.289528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.289528+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:39:17.141469+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":484,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-08-17T17:47:01.764450+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:47:01.764450+0000","last_peered":"2024-08-17T17:47:01.764450+0000","last_clean":"2024-08-17T17:47:01.764450+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:47:01.764450+0000","last_undegraded":"2024-08-17T17:47:01.764450+0000","last_fullsized":"2024-08-17T17:47:01.764450+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":696,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-08-17T17:47:01.763847+0000","last_change":"2024-08-17T17:41:34.614108+0000","last_active":"2024-08-17T17:47:01.763847+0000","last_peered":"2024-08-17T17:47:01.763847+0000","last_clean":"2024-08-17T17:47:01.763847+0000","last_became_active":"2024-08-17T17:41:34.613481+0000","last_became_peered":"2024-08-17T17:41:34.613481+0000","last_unstale":"2024-08-17T17:47:01.763847+0000","last_undegraded":"2024-08-17T17:47:01.763847+0000","last_fullsized":"2024-08-17T17:47:01.763847+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:21:02.346115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":928,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.162372+0000","last_change":"2024-08-17T17:33:20.797705+0000","last_active":"2024-08-17T17:46:55.162372+0000","last_peered":"2024-08-17T17:46:55.162372+0000","last_clean":"2024-08-17T17:46:55.162372+0000","last_became_active":"2024-08-17T17:33:20.797172+0000","last_became_peered":"2024-08-17T17:33:20.797172+0000","last_unstale":"2024-08-17T17:46:55.162372+0000","last_undegraded":"2024-08-17T17:46:55.162372+0000","last_fullsized":"2024-08-17T17:46:55.162372+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:30.754999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":870,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.163939+0000","last_change":"2024-08-17T17:41:37.676347+0000","last_active":"2024-08-17T17:46:55.163939+0000","last_peered":"2024-08-17T17:46:55.163939+0000","last_clean":"2024-08-17T17:46:55.163939+0000","last_became_active":"2024-08-17T17:41:37.675869+0000","last_became_peered":"2024-08-17T17:41:37.675869+0000","last_unstale":"2024-08-17T17:46:55.163939+0000","last_undegraded":"2024-08-17T17:46:55.163939+0000","last_fullsized":"2024-08-17T17:46:55.163939+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:21.814077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:21.814077+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:04:16.240754+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":246,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.723541+0000","last_change":"2024-08-17T17:41:34.618631+0000","last_active":"2024-08-17T17:46:55.723541+0000","last_peered":"2024-08-17T17:46:55.723541+0000","last_clean":"2024-08-17T17:46:55.723541+0000","last_became_active":"2024-08-17T17:41:34.618313+0000","last_became_peered":"2024-08-17T17:41:34.618313+0000","last_unstale":"2024-08-17T17:46:55.723541+0000","last_undegraded":"2024-08-17T17:46:55.723541+0000","last_fullsized":"2024-08-17T17:46:55.723541+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:07:13.915088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":645,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:54.747962+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:46:54.747962+0000","last_peered":"2024-08-17T17:46:54.747962+0000","last_clean":"2024-08-17T17:46:54.747962+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:46:54.747962+0000","last_undegraded":"2024-08-17T17:46:54.747962+0000","last_fullsized":"2024-08-17T17:46:54.747962+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":811,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:54.746085+0000","last_change":"2024-08-17T17:33:44.149928+0000","last_active":"2024-08-17T17:46:54.746085+0000","last_peered":"2024-08-17T17:46:54.746085+0000","last_clean":"2024-08-17T17:46:54.746085+0000","last_became_active":"2024-08-17T17:33:44.149498+0000","last_became_peered":"2024-08-17T17:33:44.149498+0000","last_unstale":"2024-08-17T17:46:54.746085+0000","last_undegraded":"2024-08-17T17:46:54.746085+0000","last_fullsized":"2024-08-17T17:46:54.746085+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:17:04.963202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":836,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-08-17T17:47:01.763650+0000","last_change":"2024-08-17T17:38:25.247159+0000","last_active":"2024-08-17T17:47:01.763650+0000","last_peered":"2024-08-17T17:47:01.763650+0000","last_clean":"2024-08-17T17:47:01.763650+0000","last_became_active":"2024-08-17T17:38:21.311361+0000","last_became_peered":"2024-08-17T17:38:21.311361+0000","last_unstale":"2024-08-17T17:47:01.763650+0000","last_undegraded":"2024-08-17T17:47:01.763650+0000","last_fullsized":"2024-08-17T17:47:01.763650+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:25.247027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:25.247027+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:46:21.734712+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":592,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.164264+0000","last_change":"2024-08-17T17:41:34.625551+0000","last_active":"2024-08-17T17:46:55.164264+0000","last_peered":"2024-08-17T17:46:55.164264+0000","last_clean":"2024-08-17T17:46:55.164264+0000","last_became_active":"2024-08-17T17:41:34.625251+0000","last_became_peered":"2024-08-17T17:41:34.625251+0000","last_unstale":"2024-08-17T17:46:55.164264+0000","last_undegraded":"2024-08-17T17:46:55.164264+0000","last_fullsized":"2024-08-17T17:46:55.164264+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:28:22.354213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":697,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:54.747996+0000","last_change":"2024-08-17T17:33:20.796216+0000","last_active":"2024-08-17T17:46:54.747996+0000","last_peered":"2024-08-17T17:46:54.747996+0000","last_clean":"2024-08-17T17:46:54.747996+0000","last_became_active":"2024-08-17T17:33:20.795762+0000","last_became_peered":"2024-08-17T17:33:20.795762+0000","last_unstale":"2024-08-17T17:46:54.747996+0000","last_undegraded":"2024-08-17T17:46:54.747996+0000","last_fullsized":"2024-08-17T17:46:54.747996+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:47.038509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":522,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.721941+0000","last_change":"2024-08-17T17:41:41.036615+0000","last_active":"2024-08-17T17:46:55.721941+0000","last_peered":"2024-08-17T17:46:55.721941+0000","last_clean":"2024-08-17T17:46:55.721941+0000","last_became_active":"2024-08-17T17:41:12.329214+0000","last_became_peered":"2024-08-17T17:41:12.329214+0000","last_unstale":"2024-08-17T17:46:55.721941+0000","last_undegraded":"2024-08-17T17:46:55.721941+0000","last_fullsized":"2024-08-17T17:46:55.721941+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:41.036524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:41.036524+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:59:58.695900+0000","scrub_duration":0.001820803,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":69,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.743057+0000","last_change":"2024-08-17T17:41:34.622102+0000","last_active":"2024-08-17T17:46:55.743057+0000","last_peered":"2024-08-17T17:46:55.743057+0000","last_clean":"2024-08-17T17:46:55.743057+0000","last_became_active":"2024-08-17T17:41:34.621733+0000","last_became_peered":"2024-08-17T17:41:34.621733+0000","last_unstale":"2024-08-17T17:46:55.743057+0000","last_undegraded":"2024-08-17T17:46:55.743057+0000","last_fullsized":"2024-08-17T17:46:55.743057+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:39:41.668709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":69,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.742712+0000","last_change":"2024-08-17T17:41:34.607602+0000","last_active":"2024-08-17T17:46:55.742712+0000","last_peered":"2024-08-17T17:46:55.742712+0000","last_clean":"2024-08-17T17:46:55.742712+0000","last_became_active":"2024-08-17T17:41:34.607300+0000","last_became_peered":"2024-08-17T17:41:34.607300+0000","last_unstale":"2024-08-17T17:46:55.742712+0000","last_undegraded":"2024-08-17T17:46:55.742712+0000","last_fullsized":"2024-08-17T17:46:55.742712+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:20:31.486698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"90'149","reported_seq":887,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.163833+0000","last_change":"2024-08-17T17:41:41.743931+0000","last_active":"2024-08-17T17:46:55.163833+0000","last_peered":"2024-08-17T17:46:55.163833+0000","last_clean":"2024-08-17T17:46:55.163833+0000","last_became_active":"2024-08-17T17:41:12.203771+0000","last_became_peered":"2024-08-17T17:41:12.203771+0000","last_unstale":"2024-08-17T17:46:55.163833+0000","last_undegraded":"2024-08-17T17:46:55.163833+0000","last_fullsized":"2024-08-17T17:46:55.163833+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:41.743570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:41.743570+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:21:03.865738+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":69,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.744431+0000","last_change":"2024-08-17T17:41:34.619268+0000","last_active":"2024-08-17T17:46:55.744431+0000","last_peered":"2024-08-17T17:46:55.744431+0000","last_clean":"2024-08-17T17:46:55.744431+0000","last_became_active":"2024-08-17T17:41:34.618914+0000","last_became_peered":"2024-08-17T17:41:34.618914+0000","last_unstale":"2024-08-17T17:46:55.744431+0000","last_undegraded":"2024-08-17T17:46:55.744431+0000","last_fullsized":"2024-08-17T17:46:55.744431+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:08:54.248386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":69,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.744488+0000","last_change":"2024-08-17T17:41:34.622803+0000","last_active":"2024-08-17T17:46:55.744488+0000","last_peered":"2024-08-17T17:46:55.744488+0000","last_clean":"2024-08-17T17:46:55.744488+0000","last_became_active":"2024-08-17T17:41:34.622525+0000","last_became_peered":"2024-08-17T17:41:34.622525+0000","last_unstale":"2024-08-17T17:46:55.744488+0000","last_undegraded":"2024-08-17T17:46:55.744488+0000","last_fullsized":"2024-08-17T17:46:55.744488+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:42:39.728377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":882,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:54.746425+0000","last_change":"2024-08-17T17:43:55.595647+0000","last_active":"2024-08-17T17:46:54.746425+0000","last_peered":"2024-08-17T17:46:54.746425+0000","last_clean":"2024-08-17T17:46:54.746425+0000","last_became_active":"2024-08-17T17:43:55.595256+0000","last_became_peered":"2024-08-17T17:43:55.595256+0000","last_unstale":"2024-08-17T17:46:54.746425+0000","last_undegraded":"2024-08-17T17:46:54.746425+0000","last_fullsized":"2024-08-17T17:46:54.746425+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:46.587712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:46.587712+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:05:01.927968+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":69,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.743667+0000","last_change":"2024-08-17T17:41:34.611049+0000","last_active":"2024-08-17T17:46:55.743667+0000","last_peered":"2024-08-17T17:46:55.743667+0000","last_clean":"2024-08-17T17:46:55.743667+0000","last_became_active":"2024-08-17T17:41:34.610727+0000","last_became_peered":"2024-08-17T17:41:34.610727+0000","last_unstale":"2024-08-17T17:46:55.743667+0000","last_undegraded":"2024-08-17T17:46:55.743667+0000","last_fullsized":"2024-08-17T17:46:55.743667+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:04:05.477660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":739,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:54.749189+0000","last_change":"2024-08-17T17:41:34.630434+0000","last_active":"2024-08-17T17:46:54.749189+0000","last_peered":"2024-08-17T17:46:54.749189+0000","last_clean":"2024-08-17T17:46:54.749189+0000","last_became_active":"2024-08-17T17:41:34.616406+0000","last_became_peered":"2024-08-17T17:41:34.616406+0000","last_unstale":"2024-08-17T17:46:54.749189+0000","last_undegraded":"2024-08-17T17:46:54.749189+0000","last_fullsized":"2024-08-17T17:46:54.749189+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:30:44.817203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":854,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:54.747143+0000","last_change":"2024-08-17T17:41:44.585386+0000","last_active":"2024-08-17T17:46:54.747143+0000","last_peered":"2024-08-17T17:46:54.747143+0000","last_clean":"2024-08-17T17:46:54.747143+0000","last_became_active":"2024-08-17T17:41:12.307398+0000","last_became_peered":"2024-08-17T17:41:12.307398+0000","last_unstale":"2024-08-17T17:46:54.747143+0000","last_undegraded":"2024-08-17T17:46:54.747143+0000","last_fullsized":"2024-08-17T17:46:54.747143+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:44.583653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:44.583653+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:27:44.756681+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":705,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-08-17T17:47:01.764461+0000","last_change":"2024-08-17T17:41:34.615197+0000","last_active":"2024-08-17T17:47:01.764461+0000","last_peered":"2024-08-17T17:47:01.764461+0000","last_clean":"2024-08-17T17:47:01.764461+0000","last_became_active":"2024-08-17T17:41:34.614873+0000","last_became_peered":"2024-08-17T17:41:34.614873+0000","last_unstale":"2024-08-17T17:47:01.764461+0000","last_undegraded":"2024-08-17T17:47:01.764461+0000","last_fullsized":"2024-08-17T17:47:01.764461+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:21:57.391953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":339,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:54.748700+0000","last_change":"2024-08-17T17:33:44.154097+0000","last_active":"2024-08-17T17:46:54.748700+0000","last_peered":"2024-08-17T17:46:54.748700+0000","last_clean":"2024-08-17T17:46:54.748700+0000","last_became_active":"2024-08-17T17:33:44.152972+0000","last_became_peered":"2024-08-17T17:33:44.152972+0000","last_unstale":"2024-08-17T17:46:54.748700+0000","last_undegraded":"2024-08-17T17:46:54.748700+0000","last_fullsized":"2024-08-17T17:46:54.748700+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:42.602017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"90'149","reported_seq":713,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.744421+0000","last_change":"2024-08-17T17:42:20.927007+0000","last_active":"2024-08-17T17:46:55.744421+0000","last_peered":"2024-08-17T17:46:55.744421+0000","last_clean":"2024-08-17T17:46:55.744421+0000","last_became_active":"2024-08-17T17:41:37.670914+0000","last_became_peered":"2024-08-17T17:41:37.670914+0000","last_unstale":"2024-08-17T17:46:55.744421+0000","last_undegraded":"2024-08-17T17:46:55.744421+0000","last_fullsized":"2024-08-17T17:46:55.744421+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:20.926924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:20.926924+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:57:44.156600+0000","scrub_duration":0.0018485109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":233,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.722110+0000","last_change":"2024-08-17T17:33:20.795582+0000","last_active":"2024-08-17T17:46:55.722110+0000","last_peered":"2024-08-17T17:46:55.722110+0000","last_clean":"2024-08-17T17:46:55.722110+0000","last_became_active":"2024-08-17T17:33:20.795149+0000","last_became_peered":"2024-08-17T17:33:20.795149+0000","last_unstale":"2024-08-17T17:46:55.722110+0000","last_undegraded":"2024-08-17T17:46:55.722110+0000","last_fullsized":"2024-08-17T17:46:55.722110+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:14.020923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":155,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.723890+0000","last_change":"2024-08-17T17:33:44.145399+0000","last_active":"2024-08-17T17:46:55.723890+0000","last_peered":"2024-08-17T17:46:55.723890+0000","last_clean":"2024-08-17T17:46:55.723890+0000","last_became_active":"2024-08-17T17:33:44.144751+0000","last_became_peered":"2024-08-17T17:33:44.144751+0000","last_unstale":"2024-08-17T17:46:55.723890+0000","last_undegraded":"2024-08-17T17:46:55.723890+0000","last_fullsized":"2024-08-17T17:46:55.723890+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:54.276350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"90'149","reported_seq":581,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.724456+0000","last_change":"2024-08-17T17:41:37.668050+0000","last_active":"2024-08-17T17:46:55.724456+0000","last_peered":"2024-08-17T17:46:55.724456+0000","last_clean":"2024-08-17T17:46:55.724456+0000","last_became_active":"2024-08-17T17:41:37.667655+0000","last_became_peered":"2024-08-17T17:41:37.667655+0000","last_unstale":"2024-08-17T17:46:55.724456+0000","last_undegraded":"2024-08-17T17:46:55.724456+0000","last_fullsized":"2024-08-17T17:46:55.724456+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:32.843802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:32.843802+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:39:46.766280+0000","scrub_duration":0.0040803669999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":69,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.742482+0000","last_change":"2024-08-17T17:41:34.621078+0000","last_active":"2024-08-17T17:46:55.742482+0000","last_peered":"2024-08-17T17:46:55.742482+0000","last_clean":"2024-08-17T17:46:55.742482+0000","last_became_active":"2024-08-17T17:41:34.620730+0000","last_became_peered":"2024-08-17T17:41:34.620730+0000","last_unstale":"2024-08-17T17:46:55.742482+0000","last_undegraded":"2024-08-17T17:46:55.742482+0000","last_fullsized":"2024-08-17T17:46:55.742482+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:55:01.312283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":362,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:54.748280+0000","last_change":"2024-08-17T17:41:34.637696+0000","last_active":"2024-08-17T17:46:54.748280+0000","last_peered":"2024-08-17T17:46:54.748280+0000","last_clean":"2024-08-17T17:46:54.748280+0000","last_became_active":"2024-08-17T17:41:34.621388+0000","last_became_peered":"2024-08-17T17:41:34.621388+0000","last_unstale":"2024-08-17T17:46:54.748280+0000","last_undegraded":"2024-08-17T17:46:54.748280+0000","last_fullsized":"2024-08-17T17:46:54.748280+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:13:04.607036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"90'149","reported_seq":879,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:54.746643+0000","last_change":"2024-08-17T17:41:42.603841+0000","last_active":"2024-08-17T17:46:54.746643+0000","last_peered":"2024-08-17T17:46:54.746643+0000","last_clean":"2024-08-17T17:46:54.746643+0000","last_became_active":"2024-08-17T17:41:12.203400+0000","last_became_peered":"2024-08-17T17:41:12.203400+0000","last_unstale":"2024-08-17T17:46:54.746643+0000","last_undegraded":"2024-08-17T17:46:54.746643+0000","last_fullsized":"2024-08-17T17:46:54.746643+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:42.603561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:42.603561+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:50:12.843193+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":159,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.721850+0000","last_change":"2024-08-17T17:33:44.145598+0000","last_active":"2024-08-17T17:46:55.721850+0000","last_peered":"2024-08-17T17:46:55.721850+0000","last_clean":"2024-08-17T17:46:55.721850+0000","last_became_active":"2024-08-17T17:33:44.145283+0000","last_became_peered":"2024-08-17T17:33:44.145283+0000","last_unstale":"2024-08-17T17:46:55.721850+0000","last_undegraded":"2024-08-17T17:46:55.721850+0000","last_fullsized":"2024-08-17T17:46:55.721850+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:11.965226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":366,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:54.747809+0000","last_change":"2024-08-17T17:41:34.635980+0000","last_active":"2024-08-17T17:46:54.747809+0000","last_peered":"2024-08-17T17:46:54.747809+0000","last_clean":"2024-08-17T17:46:54.747809+0000","last_became_active":"2024-08-17T17:41:34.620596+0000","last_became_peered":"2024-08-17T17:41:34.620596+0000","last_unstale":"2024-08-17T17:46:54.747809+0000","last_undegraded":"2024-08-17T17:46:54.747809+0000","last_fullsized":"2024-08-17T17:46:54.747809+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:19:15.792577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"90'149","reported_seq":815,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:54.746277+0000","last_change":"2024-08-17T17:42:16.610940+0000","last_active":"2024-08-17T17:46:54.746277+0000","last_peered":"2024-08-17T17:46:54.746277+0000","last_clean":"2024-08-17T17:46:54.746277+0000","last_became_active":"2024-08-17T17:41:37.670855+0000","last_became_peered":"2024-08-17T17:41:37.670855+0000","last_unstale":"2024-08-17T17:46:54.746277+0000","last_undegraded":"2024-08-17T17:46:54.746277+0000","last_fullsized":"2024-08-17T17:46:54.746277+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:16.610828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:16.610828+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:48:25.918610+0000","scrub_duration":0.086892487000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":490,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.723457+0000","last_change":"2024-08-17T17:43:52.520659+0000","last_active":"2024-08-17T17:46:55.723457+0000","last_peered":"2024-08-17T17:46:55.723457+0000","last_clean":"2024-08-17T17:46:55.723457+0000","last_became_active":"2024-08-17T17:43:52.516799+0000","last_became_peered":"2024-08-17T17:43:52.516799+0000","last_unstale":"2024-08-17T17:46:55.723457+0000","last_undegraded":"2024-08-17T17:46:55.723457+0000","last_fullsized":"2024-08-17T17:46:55.723457+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:25:59.408383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":695,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:54.747731+0000","last_change":"2024-08-17T17:33:20.794682+0000","last_active":"2024-08-17T17:46:54.747731+0000","last_peered":"2024-08-17T17:46:54.747731+0000","last_clean":"2024-08-17T17:46:54.747731+0000","last_became_active":"2024-08-17T17:33:20.794238+0000","last_became_peered":"2024-08-17T17:33:20.794238+0000","last_unstale":"2024-08-17T17:46:54.747731+0000","last_undegraded":"2024-08-17T17:46:54.747731+0000","last_fullsized":"2024-08-17T17:46:54.747731+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:15:55.821920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"90'149","reported_seq":854,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.162862+0000","last_change":"2024-08-17T17:38:30.446165+0000","last_active":"2024-08-17T17:46:55.162862+0000","last_peered":"2024-08-17T17:46:55.162862+0000","last_clean":"2024-08-17T17:46:55.162862+0000","last_became_active":"2024-08-17T17:38:30.445048+0000","last_became_peered":"2024-08-17T17:38:30.445048+0000","last_unstale":"2024-08-17T17:46:55.162862+0000","last_undegraded":"2024-08-17T17:46:55.162862+0000","last_fullsized":"2024-08-17T17:46:55.162862+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:24.291592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:24.291592+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:17.100154+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":69,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.742634+0000","last_change":"2024-08-17T17:41:34.611693+0000","last_active":"2024-08-17T17:46:55.742634+0000","last_peered":"2024-08-17T17:46:55.742634+0000","last_clean":"2024-08-17T17:46:55.742634+0000","last_became_active":"2024-08-17T17:41:34.611374+0000","last_became_peered":"2024-08-17T17:41:34.611374+0000","last_unstale":"2024-08-17T17:46:55.742634+0000","last_undegraded":"2024-08-17T17:46:55.742634+0000","last_fullsized":"2024-08-17T17:46:55.742634+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:17:09.897446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":483,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-08-17T17:47:01.764178+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:47:01.764178+0000","last_peered":"2024-08-17T17:47:01.764178+0000","last_clean":"2024-08-17T17:47:01.764178+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:47:01.764178+0000","last_undegraded":"2024-08-17T17:47:01.764178+0000","last_fullsized":"2024-08-17T17:47:01.764178+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"90'149","reported_seq":878,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.163628+0000","last_change":"2024-08-17T17:41:37.681187+0000","last_active":"2024-08-17T17:46:55.163628+0000","last_peered":"2024-08-17T17:46:55.163628+0000","last_clean":"2024-08-17T17:46:55.163628+0000","last_became_active":"2024-08-17T17:41:37.680605+0000","last_became_peered":"2024-08-17T17:41:37.680605+0000","last_unstale":"2024-08-17T17:46:55.163628+0000","last_undegraded":"2024-08-17T17:46:55.163628+0000","last_fullsized":"2024-08-17T17:46:55.163628+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.844252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.844252+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:13:31.682357+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":483,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-08-17T17:47:01.763790+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:47:01.763790+0000","last_peered":"2024-08-17T17:47:01.763790+0000","last_clean":"2024-08-17T17:47:01.763790+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:47:01.763790+0000","last_undegraded":"2024-08-17T17:47:01.763790+0000","last_fullsized":"2024-08-17T17:47:01.763790+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":342,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:54.747371+0000","last_change":"2024-08-17T17:33:44.156036+0000","last_active":"2024-08-17T17:46:54.747371+0000","last_peered":"2024-08-17T17:46:54.747371+0000","last_clean":"2024-08-17T17:46:54.747371+0000","last_became_active":"2024-08-17T17:33:44.155758+0000","last_became_peered":"2024-08-17T17:33:44.155758+0000","last_unstale":"2024-08-17T17:46:54.747371+0000","last_undegraded":"2024-08-17T17:46:54.747371+0000","last_fullsized":"2024-08-17T17:46:54.747371+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:23.443405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.10","version":"90'149","reported_seq":488,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:55.721794+0000","last_change":"2024-08-17T17:38:31.874623+0000","last_active":"2024-08-17T17:46:55.721794+0000","last_peered":"2024-08-17T17:46:55.721794+0000","last_clean":"2024-08-17T17:46:55.721794+0000","last_became_active":"2024-08-17T17:38:30.339313+0000","last_became_peered":"2024-08-17T17:38:30.339313+0000","last_unstale":"2024-08-17T17:46:55.721794+0000","last_undegraded":"2024-08-17T17:46:55.721794+0000","last_fullsized":"2024-08-17T17:46:55.721794+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:31.874423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:31.874423+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:28:17.980339+0000","scrub_duration":0.0017696579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":429,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:54.746232+0000","last_change":"2024-08-17T17:41:34.624198+0000","last_active":"2024-08-17T17:46:54.746232+0000","last_peered":"2024-08-17T17:46:54.746232+0000","last_clean":"2024-08-17T17:46:54.746232+0000","last_became_active":"2024-08-17T17:41:34.616071+0000","last_became_peered":"2024-08-17T17:41:34.616071+0000","last_unstale":"2024-08-17T17:46:54.746232+0000","last_undegraded":"2024-08-17T17:46:54.746232+0000","last_fullsized":"2024-08-17T17:46:54.746232+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:34:48.805354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":681,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-08-17T17:46:54.748521+0000","last_change":"2024-08-17T17:33:20.797029+0000","last_active":"2024-08-17T17:46:54.748521+0000","last_peered":"2024-08-17T17:46:54.748521+0000","last_clean":"2024-08-17T17:46:54.748521+0000","last_became_active":"2024-08-17T17:33:20.796679+0000","last_became_peered":"2024-08-17T17:33:20.796679+0000","last_unstale":"2024-08-17T17:46:54.748521+0000","last_undegraded":"2024-08-17T17:46:54.748521+0000","last_fullsized":"2024-08-17T17:46:54.748521+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:51.438262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"90'149","reported_seq":862,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-08-17T17:47:01.763950+0000","last_change":"2024-08-17T17:38:30.378664+0000","last_active":"2024-08-17T17:47:01.763950+0000","last_peered":"2024-08-17T17:47:01.763950+0000","last_clean":"2024-08-17T17:47:01.763950+0000","last_became_active":"2024-08-17T17:38:30.342287+0000","last_became_peered":"2024-08-17T17:38:30.342287+0000","last_unstale":"2024-08-17T17:47:01.763950+0000","last_undegraded":"2024-08-17T17:47:01.763950+0000","last_fullsized":"2024-08-17T17:47:01.763950+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:30.378592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:30.378592+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:04:28.933354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":11256,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19280,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5978,"num_read_kb":12991,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4619,"ondisk_log_size":4619,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234483,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":911884,"kb_used_data":9664,"kb_used_omap":1,"kb_used_meta":902206,"kb_avail":92833268,"statfs":{"total":95995035648,"available":95061266432,"internally_reserved":0,"allocated":9895936,"data_stored":7603639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":923859288},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496118,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":900668,"kb_used_data":9712,"kb_used_omap":1,"kb_used_meta":890942,"kb_avail":92844484,"statfs":{"total":95995035648,"available":95072751616,"internally_reserved":0,"allocated":9945088,"data_stored":7668305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":912324952},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757755,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":887868,"kb_used_data":5552,"kb_used_omap":1,"kb_used_meta":882302,"kb_avail":92857284,"statfs":{"total":95995035648,"available":95085858816,"internally_reserved":0,"allocated":5685248,"data_stored":3405034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":903477592},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019393,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306824,"kb_used_data":5308,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93438328,"statfs":{"total":95995035648,"available":95680847872,"internally_reserved":0,"allocated":5435392,"data_stored":3146908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435619,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":965732,"kb_used_data":5596,"kb_used_omap":4,"kb_used_meta":960123,"kb_avail":92779420,"statfs":{"total":95995035648,"available":95006126080,"internally_reserved":0,"allocated":5730304,"data_stored":3461802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":983165962},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599147,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":845436,"kb_used_data":9648,"kb_used_omap":7,"kb_used_meta":835768,"kb_avail":92899716,"statfs":{"total":95995035648,"available":95129309184,"internally_reserved":0,"allocated":9879552,"data_stored":7599634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":855826594},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":268,"seq":1151051235493,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":455536,"kb_used_data":5276,"kb_used_omap":10,"kb_used_meta":450229,"kb_avail":93289616,"statfs":{"total":95995035648,"available":95528566784,"internally_reserved":0,"allocated":5402624,"data_stored":3134067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10902,"internal_metadata":461034858},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926200,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":940076,"kb_used_data":5600,"kb_used_omap":7,"kb_used_meta":934456,"kb_avail":92805076,"statfs":{"total":95995035648,"available":95032397824,"internally_reserved":0,"allocated":5734400,"data_stored":3462312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":956883265},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7550,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:47:05.522 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:47:05.766 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:47:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:47:05] "GET /metrics HTTP/1.1" 200 36551 "" "Prometheus/2.51.0" 2024-08-17T17:47:05.907 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:05 smithi053 ceph-mon[89963]: pgmap v1351: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:05.908 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:05 smithi053 ceph-mon[89963]: from='client.36957 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:47:05.936 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-17T17:47:06.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:05 smithi003 ceph-mon[103344]: pgmap v1351: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:06.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:05 smithi003 ceph-mon[103344]: from='client.36957 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:47:06.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:05 smithi003 ceph-mon[100615]: pgmap v1351: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:06.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:05 smithi003 ceph-mon[100615]: from='client.36957 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:47:07.248 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:47:07.772 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-08-17T17:47:07.964 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:07 smithi003 ceph-mon[103344]: pgmap v1352: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:07.964 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:07 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3835988370' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:47:07.964 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:07 smithi003 ceph-mon[100615]: pgmap v1352: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:07.964 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:07 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3835988370' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:47:08.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:07 smithi053 ceph-mon[89963]: pgmap v1352: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:08.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:07 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3835988370' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-17T17:47:08.465 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:47:08.577 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-17T17:47:09.085 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:47:09.087 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:47:09.118 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:08 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3835988370' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:47:09.118 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:08 smithi053 ceph-mon[89963]: osdmap e406: 8 total, 8 up, 8 in 2024-08-17T17:47:09.118 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:47:09.118 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:47:09.176 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:08 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3835988370' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:47:09.176 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:08 smithi003 ceph-mon[103344]: osdmap e406: 8 total, 8 up, 8 in 2024-08-17T17:47:09.176 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:47:09.176 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:47:09.177 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:08 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3835988370' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-17T17:47:09.177 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:08 smithi003 ceph-mon[100615]: osdmap e406: 8 total, 8 up, 8 in 2024-08-17T17:47:09.177 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:47:09.177 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:47:09.733 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:47:09.733 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:47:10.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:09 smithi003 ceph-mon[100615]: pgmap v1354: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:10.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:09 smithi003 ceph-mon[103344]: pgmap v1354: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:10.130 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:09 smithi053 ceph-mon[89963]: pgmap v1354: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:10.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:47:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:47:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:47:11.462 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-17T17:47:12.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:11 smithi003 ceph-mon[100615]: pgmap v1355: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:12.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:11 smithi003 ceph-mon[103344]: pgmap v1355: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:12.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:11 smithi053 ceph-mon[89963]: pgmap v1355: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:12.782 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-08-17T17:47:13.017 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:12 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/279095758' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:47:13.017 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:12 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/279095758' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:47:13.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:12 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/279095758' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-17T17:47:13.256 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:47:13.917 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:13 smithi003 ceph-mon[100615]: pgmap v1356: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:13.917 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:13 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/279095758' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:47:13.917 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:13 smithi003 ceph-mon[100615]: osdmap e407: 8 total, 8 up, 8 in 2024-08-17T17:47:13.917 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:13 smithi003 ceph-mon[103344]: pgmap v1356: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:13.917 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:13 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/279095758' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:47:13.917 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:13 smithi003 ceph-mon[103344]: osdmap e407: 8 total, 8 up, 8 in 2024-08-17T17:47:14.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:13 smithi053 ceph-mon[89963]: pgmap v1356: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:14.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:13 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/279095758' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-17T17:47:14.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:13 smithi053 ceph-mon[89963]: osdmap e407: 8 total, 8 up, 8 in 2024-08-17T17:47:14.864 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:47:14.865 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:47:15.714 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:47:15.714 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:47:15.905 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:15 smithi003 ceph-mon[103344]: pgmap v1358: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:15.905 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:15 smithi003 ceph-mon[103344]: from='client.46907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:47:15.905 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:47:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:47:15] "GET /metrics HTTP/1.1" 200 34140 "" "Prometheus/2.51.0" 2024-08-17T17:47:15.905 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:15 smithi003 ceph-mon[100615]: pgmap v1358: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:15.905 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:15 smithi003 ceph-mon[100615]: from='client.46907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:47:16.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:15 smithi053 ceph-mon[89963]: pgmap v1358: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:16.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:15 smithi053 ceph-mon[89963]: from='client.46907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:47:16.560 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1358,"stamp":"2024-08-17T17:47:14.449463+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790221,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":5978,"num_read_kb":12991,"num_write":907,"num_write_kb":12623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":194,"num_bytes_recovered":8399864,"num_keys_recovered":40,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5651,"ondisk_log_size":5651,"up":309,"acting":309,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":309,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6214188,"kb_used_data":56524,"kb_used_omap":37,"kb_used_meta":6157530,"kb_avail":743747028,"statfs":{"total":767960285184,"available":761596956672,"internally_reserved":0,"allocated":57880576,"data_stored":39593785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38729,"internal_metadata":6305310903},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001992"},"pg_stats":[{"pgid":"3.1f","version":"71'4","reported_seq":506,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098338+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:47:08.098338+0000","last_peered":"2024-08-17T17:47:08.098338+0000","last_clean":"2024-08-17T17:47:08.098338+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:47:08.098338+0000","last_undegraded":"2024-08-17T17:47:08.098338+0000","last_fullsized":"2024-08-17T17:47:08.098338+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":71,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.097971+0000","last_change":"2024-08-17T17:41:34.611711+0000","last_active":"2024-08-17T17:47:08.097971+0000","last_peered":"2024-08-17T17:47:08.097971+0000","last_clean":"2024-08-17T17:47:08.097971+0000","last_became_active":"2024-08-17T17:41:34.611407+0000","last_became_peered":"2024-08-17T17:41:34.611407+0000","last_unstale":"2024-08-17T17:47:08.097971+0000","last_undegraded":"2024-08-17T17:47:08.097971+0000","last_fullsized":"2024-08-17T17:47:08.097971+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:42:41.582320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1d","version":"90'149","reported_seq":762,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.097865+0000","last_change":"2024-08-17T17:42:18.668058+0000","last_active":"2024-08-17T17:47:08.097865+0000","last_peered":"2024-08-17T17:47:08.097865+0000","last_clean":"2024-08-17T17:47:08.097865+0000","last_became_active":"2024-08-17T17:42:05.876721+0000","last_became_peered":"2024-08-17T17:42:05.876721+0000","last_unstale":"2024-08-17T17:47:08.097865+0000","last_undegraded":"2024-08-17T17:47:08.097865+0000","last_fullsized":"2024-08-17T17:47:08.097865+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:18.667957+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:18.667957+0000","last_clean_scrub_stamp":"2024-08-17T17:42:18.667957+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:38.170146+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":696,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:07.915728+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:47:07.915728+0000","last_peered":"2024-08-17T17:47:07.915728+0000","last_clean":"2024-08-17T17:47:07.915728+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:47:07.915728+0000","last_undegraded":"2024-08-17T17:47:07.915728+0000","last_fullsized":"2024-08-17T17:47:07.915728+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":695,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:07.915336+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:47:07.915336+0000","last_peered":"2024-08-17T17:47:07.915336+0000","last_clean":"2024-08-17T17:47:07.915336+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:47:07.915336+0000","last_undegraded":"2024-08-17T17:47:07.915336+0000","last_fullsized":"2024-08-17T17:47:07.915336+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"90'149","reported_seq":747,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.099202+0000","last_change":"2024-08-17T17:42:11.809047+0000","last_active":"2024-08-17T17:47:08.099202+0000","last_peered":"2024-08-17T17:47:08.099202+0000","last_clean":"2024-08-17T17:47:08.099202+0000","last_became_active":"2024-08-17T17:42:05.883770+0000","last_became_peered":"2024-08-17T17:42:05.883770+0000","last_unstale":"2024-08-17T17:47:08.099202+0000","last_undegraded":"2024-08-17T17:47:08.099202+0000","last_fullsized":"2024-08-17T17:47:08.099202+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:11.808971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:11.808971+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:48:15.989437+0000","scrub_duration":0.0018167420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":75,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.099144+0000","last_change":"2024-08-17T17:41:34.631152+0000","last_active":"2024-08-17T17:47:08.099144+0000","last_peered":"2024-08-17T17:47:08.099144+0000","last_clean":"2024-08-17T17:47:08.099144+0000","last_became_active":"2024-08-17T17:41:34.615617+0000","last_became_peered":"2024-08-17T17:41:34.615617+0000","last_unstale":"2024-08-17T17:47:08.099144+0000","last_undegraded":"2024-08-17T17:47:08.099144+0000","last_fullsized":"2024-08-17T17:47:08.099144+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:43:08.257084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":71,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098938+0000","last_change":"2024-08-17T17:41:34.616251+0000","last_active":"2024-08-17T17:47:08.098938+0000","last_peered":"2024-08-17T17:47:08.098938+0000","last_clean":"2024-08-17T17:47:08.098938+0000","last_became_active":"2024-08-17T17:41:34.615952+0000","last_became_peered":"2024-08-17T17:41:34.615952+0000","last_unstale":"2024-08-17T17:47:08.098938+0000","last_undegraded":"2024-08-17T17:47:08.098938+0000","last_fullsized":"2024-08-17T17:47:08.098938+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:12:24.892508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":75,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.099603+0000","last_change":"2024-08-17T17:41:34.629135+0000","last_active":"2024-08-17T17:47:08.099603+0000","last_peered":"2024-08-17T17:47:08.099603+0000","last_clean":"2024-08-17T17:47:08.099603+0000","last_became_active":"2024-08-17T17:41:34.613095+0000","last_became_peered":"2024-08-17T17:41:34.613095+0000","last_unstale":"2024-08-17T17:47:08.099603+0000","last_undegraded":"2024-08-17T17:47:08.099603+0000","last_fullsized":"2024-08-17T17:47:08.099603+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:27:06.542416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":71,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.099433+0000","last_change":"2024-08-17T17:41:34.622094+0000","last_active":"2024-08-17T17:47:08.099433+0000","last_peered":"2024-08-17T17:47:08.099433+0000","last_clean":"2024-08-17T17:47:08.099433+0000","last_became_active":"2024-08-17T17:41:34.621773+0000","last_became_peered":"2024-08-17T17:41:34.621773+0000","last_unstale":"2024-08-17T17:47:08.099433+0000","last_undegraded":"2024-08-17T17:47:08.099433+0000","last_fullsized":"2024-08-17T17:47:08.099433+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:34:30.540886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"90'149","reported_seq":514,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:07.915663+0000","last_change":"2024-08-17T17:42:06.742801+0000","last_active":"2024-08-17T17:47:07.915663+0000","last_peered":"2024-08-17T17:47:07.915663+0000","last_clean":"2024-08-17T17:47:07.915663+0000","last_became_active":"2024-08-17T17:42:05.876895+0000","last_became_peered":"2024-08-17T17:42:05.876895+0000","last_unstale":"2024-08-17T17:47:07.915663+0000","last_undegraded":"2024-08-17T17:47:07.915663+0000","last_fullsized":"2024-08-17T17:47:07.915663+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.742723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.742723+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:36:46.651953+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":510,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098251+0000","last_change":"2024-08-17T17:41:34.618968+0000","last_active":"2024-08-17T17:47:08.098251+0000","last_peered":"2024-08-17T17:47:08.098251+0000","last_clean":"2024-08-17T17:47:08.098251+0000","last_became_active":"2024-08-17T17:41:34.618576+0000","last_became_peered":"2024-08-17T17:41:34.618576+0000","last_unstale":"2024-08-17T17:47:08.098251+0000","last_undegraded":"2024-08-17T17:47:08.098251+0000","last_fullsized":"2024-08-17T17:47:08.098251+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:47:45.888590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":603,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098309+0000","last_change":"2024-08-17T17:33:20.773207+0000","last_active":"2024-08-17T17:47:08.098309+0000","last_peered":"2024-08-17T17:47:08.098309+0000","last_clean":"2024-08-17T17:47:08.098309+0000","last_became_active":"2024-08-17T17:33:20.772457+0000","last_became_peered":"2024-08-17T17:33:20.772457+0000","last_unstale":"2024-08-17T17:47:08.098309+0000","last_undegraded":"2024-08-17T17:47:08.098309+0000","last_fullsized":"2024-08-17T17:47:08.098309+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:57:09.610084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.19","version":"90'149","reported_seq":866,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:07.914952+0000","last_change":"2024-08-17T17:42:05.876463+0000","last_active":"2024-08-17T17:47:07.914952+0000","last_peered":"2024-08-17T17:47:07.914952+0000","last_clean":"2024-08-17T17:47:07.914952+0000","last_became_active":"2024-08-17T17:42:05.876069+0000","last_became_peered":"2024-08-17T17:42:05.876069+0000","last_unstale":"2024-08-17T17:47:07.914952+0000","last_undegraded":"2024-08-17T17:47:07.914952+0000","last_fullsized":"2024-08-17T17:47:07.914952+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:45.552985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:45.552985+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:55:06.027278+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":71,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098513+0000","last_change":"2024-08-17T17:41:34.622184+0000","last_active":"2024-08-17T17:47:08.098513+0000","last_peered":"2024-08-17T17:47:08.098513+0000","last_clean":"2024-08-17T17:47:08.098513+0000","last_became_active":"2024-08-17T17:41:34.621831+0000","last_became_peered":"2024-08-17T17:41:34.621831+0000","last_unstale":"2024-08-17T17:47:08.098513+0000","last_undegraded":"2024-08-17T17:47:08.098513+0000","last_fullsized":"2024-08-17T17:47:08.098513+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:59.826667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":71,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.099506+0000","last_change":"2024-08-17T17:41:34.616482+0000","last_active":"2024-08-17T17:47:08.099506+0000","last_peered":"2024-08-17T17:47:08.099506+0000","last_clean":"2024-08-17T17:47:08.099506+0000","last_became_active":"2024-08-17T17:41:34.616040+0000","last_became_peered":"2024-08-17T17:41:34.616040+0000","last_unstale":"2024-08-17T17:47:08.099506+0000","last_undegraded":"2024-08-17T17:47:08.099506+0000","last_fullsized":"2024-08-17T17:47:08.099506+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:53:08.502264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.18","version":"90'149","reported_seq":713,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:07.776489+0000","last_change":"2024-08-17T17:42:08.482975+0000","last_active":"2024-08-17T17:47:07.776489+0000","last_peered":"2024-08-17T17:47:07.776489+0000","last_clean":"2024-08-17T17:47:07.776489+0000","last_became_active":"2024-08-17T17:42:05.873523+0000","last_became_peered":"2024-08-17T17:42:05.873523+0000","last_unstale":"2024-08-17T17:47:07.776489+0000","last_undegraded":"2024-08-17T17:47:07.776489+0000","last_fullsized":"2024-08-17T17:47:07.776489+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:08.482892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:08.482892+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:26:41.477341+0000","scrub_duration":0.0017925479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":636,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.097698+0000","last_change":"2024-08-17T17:41:34.625011+0000","last_active":"2024-08-17T17:47:08.097698+0000","last_peered":"2024-08-17T17:47:08.097698+0000","last_clean":"2024-08-17T17:47:08.097698+0000","last_became_active":"2024-08-17T17:41:34.624631+0000","last_became_peered":"2024-08-17T17:41:34.624631+0000","last_unstale":"2024-08-17T17:47:08.097698+0000","last_undegraded":"2024-08-17T17:47:08.097698+0000","last_fullsized":"2024-08-17T17:47:08.097698+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:03:05.565435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":487,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098504+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:47:08.098504+0000","last_peered":"2024-08-17T17:47:08.098504+0000","last_clean":"2024-08-17T17:47:08.098504+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:47:08.098504+0000","last_undegraded":"2024-08-17T17:47:08.098504+0000","last_fullsized":"2024-08-17T17:47:08.098504+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1b","version":"90'149","reported_seq":750,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.099543+0000","last_change":"2024-08-17T17:42:26.967686+0000","last_active":"2024-08-17T17:47:08.099543+0000","last_peered":"2024-08-17T17:47:08.099543+0000","last_clean":"2024-08-17T17:47:08.099543+0000","last_became_active":"2024-08-17T17:42:05.873657+0000","last_became_peered":"2024-08-17T17:42:05.873657+0000","last_unstale":"2024-08-17T17:47:08.099543+0000","last_undegraded":"2024-08-17T17:47:08.099543+0000","last_fullsized":"2024-08-17T17:47:08.099543+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:26.967611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:26.967611+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:31:03.344332+0000","scrub_duration":0.0015978279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":717,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:07.914983+0000","last_change":"2024-08-17T17:33:20.771279+0000","last_active":"2024-08-17T17:47:07.914983+0000","last_peered":"2024-08-17T17:47:07.914983+0000","last_clean":"2024-08-17T17:47:07.914983+0000","last_became_active":"2024-08-17T17:33:20.770728+0000","last_became_peered":"2024-08-17T17:33:20.770728+0000","last_unstale":"2024-08-17T17:47:07.914983+0000","last_undegraded":"2024-08-17T17:47:07.914983+0000","last_fullsized":"2024-08-17T17:47:07.914983+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:52:31.540397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":380,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.099838+0000","last_change":"2024-08-17T17:33:20.835760+0000","last_active":"2024-08-17T17:47:08.099838+0000","last_peered":"2024-08-17T17:47:08.099838+0000","last_clean":"2024-08-17T17:47:08.099838+0000","last_became_active":"2024-08-17T17:33:20.835410+0000","last_became_peered":"2024-08-17T17:33:20.835410+0000","last_unstale":"2024-08-17T17:47:08.099838+0000","last_undegraded":"2024-08-17T17:47:08.099838+0000","last_fullsized":"2024-08-17T17:47:08.099838+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:45.448022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1a","version":"90'149","reported_seq":455,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098274+0000","last_change":"2024-08-17T17:42:21.932830+0000","last_active":"2024-08-17T17:47:08.098274+0000","last_peered":"2024-08-17T17:47:08.098274+0000","last_clean":"2024-08-17T17:47:08.098274+0000","last_became_active":"2024-08-17T17:42:05.876835+0000","last_became_peered":"2024-08-17T17:42:05.876835+0000","last_unstale":"2024-08-17T17:47:08.098274+0000","last_undegraded":"2024-08-17T17:47:08.098274+0000","last_fullsized":"2024-08-17T17:47:08.098274+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:21.932757+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:21.932757+0000","last_clean_scrub_stamp":"2024-08-17T17:42:21.932757+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:36:39.848570+0000","scrub_duration":0.0020218810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":699,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:07.914887+0000","last_change":"2024-08-17T17:33:20.790687+0000","last_active":"2024-08-17T17:47:07.914887+0000","last_peered":"2024-08-17T17:47:07.914887+0000","last_clean":"2024-08-17T17:47:07.914887+0000","last_became_active":"2024-08-17T17:33:20.790399+0000","last_became_peered":"2024-08-17T17:33:20.790399+0000","last_unstale":"2024-08-17T17:47:07.914887+0000","last_undegraded":"2024-08-17T17:47:07.914887+0000","last_fullsized":"2024-08-17T17:47:07.914887+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:38:06.849989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":668,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098448+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:47:08.098448+0000","last_peered":"2024-08-17T17:47:08.098448+0000","last_clean":"2024-08-17T17:47:08.098448+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:47:08.098448+0000","last_undegraded":"2024-08-17T17:47:08.098448+0000","last_fullsized":"2024-08-17T17:47:08.098448+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"90'149","reported_seq":793,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.099059+0000","last_change":"2024-08-17T17:42:06.456562+0000","last_active":"2024-08-17T17:47:08.099059+0000","last_peered":"2024-08-17T17:47:08.099059+0000","last_clean":"2024-08-17T17:47:08.099059+0000","last_became_active":"2024-08-17T17:42:05.876577+0000","last_became_peered":"2024-08-17T17:42:05.876577+0000","last_unstale":"2024-08-17T17:47:08.099059+0000","last_undegraded":"2024-08-17T17:47:08.099059+0000","last_fullsized":"2024-08-17T17:47:08.099059+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.456490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.456490+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:10:03.654953+0000","scrub_duration":0.001537285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":641,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:07.915102+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:47:07.915102+0000","last_peered":"2024-08-17T17:47:07.915102+0000","last_clean":"2024-08-17T17:47:07.915102+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:47:07.915102+0000","last_undegraded":"2024-08-17T17:47:07.915102+0000","last_fullsized":"2024-08-17T17:47:07.915102+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":376,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098360+0000","last_change":"2024-08-17T17:33:44.146014+0000","last_active":"2024-08-17T17:47:08.098360+0000","last_peered":"2024-08-17T17:47:08.098360+0000","last_clean":"2024-08-17T17:47:08.098360+0000","last_became_active":"2024-08-17T17:33:44.145668+0000","last_became_peered":"2024-08-17T17:33:44.145668+0000","last_unstale":"2024-08-17T17:47:08.098360+0000","last_undegraded":"2024-08-17T17:47:08.098360+0000","last_fullsized":"2024-08-17T17:47:08.098360+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:53:41.679237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.14","version":"90'149","reported_seq":864,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-08-17T17:47:12.783689+0000","last_change":"2024-08-17T17:41:37.664471+0000","last_active":"2024-08-17T17:47:12.783689+0000","last_peered":"2024-08-17T17:47:12.783689+0000","last_clean":"2024-08-17T17:47:12.783689+0000","last_became_active":"2024-08-17T17:41:37.664108+0000","last_became_peered":"2024-08-17T17:41:37.664108+0000","last_unstale":"2024-08-17T17:47:12.783689+0000","last_undegraded":"2024-08-17T17:47:12.783689+0000","last_fullsized":"2024-08-17T17:47:12.783689+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:22.030514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:22.030514+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:15:52.325163+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":413,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:07.915466+0000","last_change":"2024-08-17T17:41:34.634301+0000","last_active":"2024-08-17T17:47:07.915466+0000","last_peered":"2024-08-17T17:47:07.915466+0000","last_clean":"2024-08-17T17:47:07.915466+0000","last_became_active":"2024-08-17T17:41:34.620508+0000","last_became_peered":"2024-08-17T17:41:34.620508+0000","last_unstale":"2024-08-17T17:47:07.915466+0000","last_undegraded":"2024-08-17T17:47:07.915466+0000","last_fullsized":"2024-08-17T17:47:07.915466+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:10:54.576812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":360,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":637,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.097833+0000","last_change":"2024-08-17T17:33:20.799402+0000","last_active":"2024-08-17T17:47:08.097833+0000","last_peered":"2024-08-17T17:47:08.097833+0000","last_clean":"2024-08-17T17:47:08.097833+0000","last_became_active":"2024-08-17T17:33:20.798832+0000","last_became_peered":"2024-08-17T17:33:20.798832+0000","last_unstale":"2024-08-17T17:47:08.097833+0000","last_undegraded":"2024-08-17T17:47:08.097833+0000","last_fullsized":"2024-08-17T17:47:08.097833+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:21:05.001134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"90'149","reported_seq":509,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098107+0000","last_change":"2024-08-17T17:42:06.699082+0000","last_active":"2024-08-17T17:47:08.098107+0000","last_peered":"2024-08-17T17:47:08.098107+0000","last_clean":"2024-08-17T17:47:08.098107+0000","last_became_active":"2024-08-17T17:42:05.876218+0000","last_became_peered":"2024-08-17T17:42:05.876218+0000","last_unstale":"2024-08-17T17:47:08.098107+0000","last_undegraded":"2024-08-17T17:47:08.098107+0000","last_fullsized":"2024-08-17T17:47:08.098107+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.698998+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:06.698998+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.698998+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:22:16.510150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":81,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.099497+0000","last_change":"2024-08-17T17:41:34.626584+0000","last_active":"2024-08-17T17:47:08.099497+0000","last_peered":"2024-08-17T17:47:08.099497+0000","last_clean":"2024-08-17T17:47:08.099497+0000","last_became_active":"2024-08-17T17:41:34.609878+0000","last_became_peered":"2024-08-17T17:41:34.609878+0000","last_unstale":"2024-08-17T17:47:08.099497+0000","last_undegraded":"2024-08-17T17:47:08.099497+0000","last_fullsized":"2024-08-17T17:47:08.099497+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:08:12.810347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":71,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.099716+0000","last_change":"2024-08-17T17:41:34.622180+0000","last_active":"2024-08-17T17:47:08.099716+0000","last_peered":"2024-08-17T17:47:08.099716+0000","last_clean":"2024-08-17T17:47:08.099716+0000","last_became_active":"2024-08-17T17:41:34.621724+0000","last_became_peered":"2024-08-17T17:41:34.621724+0000","last_unstale":"2024-08-17T17:47:08.099716+0000","last_undegraded":"2024-08-17T17:47:08.099716+0000","last_fullsized":"2024-08-17T17:47:08.099716+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:15.807235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"90'149","reported_seq":750,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098053+0000","last_change":"2024-08-17T17:42:07.681204+0000","last_active":"2024-08-17T17:47:08.098053+0000","last_peered":"2024-08-17T17:47:08.098053+0000","last_clean":"2024-08-17T17:47:08.098053+0000","last_became_active":"2024-08-17T17:42:05.878769+0000","last_became_peered":"2024-08-17T17:42:05.878769+0000","last_unstale":"2024-08-17T17:47:08.098053+0000","last_undegraded":"2024-08-17T17:47:08.098053+0000","last_fullsized":"2024-08-17T17:47:08.098053+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:07.681064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:07.681064+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:12:12.835715+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":71,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098689+0000","last_change":"2024-08-17T17:41:34.614028+0000","last_active":"2024-08-17T17:47:08.098689+0000","last_peered":"2024-08-17T17:47:08.098689+0000","last_clean":"2024-08-17T17:47:08.098689+0000","last_became_active":"2024-08-17T17:41:34.613706+0000","last_became_peered":"2024-08-17T17:41:34.613706+0000","last_unstale":"2024-08-17T17:47:08.098689+0000","last_undegraded":"2024-08-17T17:47:08.098689+0000","last_fullsized":"2024-08-17T17:47:08.098689+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:19:55.254902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":637,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.099966+0000","last_change":"2024-08-17T17:26:06.855307+0000","last_active":"2024-08-17T17:47:08.099966+0000","last_peered":"2024-08-17T17:47:08.099966+0000","last_clean":"2024-08-17T17:47:08.099966+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:47:08.099966+0000","last_undegraded":"2024-08-17T17:47:08.099966+0000","last_fullsized":"2024-08-17T17:47:08.099966+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":532,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098576+0000","last_change":"2024-08-17T17:41:34.629522+0000","last_active":"2024-08-17T17:47:08.098576+0000","last_peered":"2024-08-17T17:47:08.098576+0000","last_clean":"2024-08-17T17:47:08.098576+0000","last_became_active":"2024-08-17T17:41:34.619340+0000","last_became_peered":"2024-08-17T17:41:34.619340+0000","last_unstale":"2024-08-17T17:47:08.098576+0000","last_undegraded":"2024-08-17T17:47:08.098576+0000","last_fullsized":"2024-08-17T17:47:08.098576+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:30:03.902956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":839,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:07.915185+0000","last_change":"2024-08-17T17:38:23.113397+0000","last_active":"2024-08-17T17:47:07.915185+0000","last_peered":"2024-08-17T17:47:07.915185+0000","last_clean":"2024-08-17T17:47:07.915185+0000","last_became_active":"2024-08-17T17:38:21.305580+0000","last_became_peered":"2024-08-17T17:38:21.305580+0000","last_unstale":"2024-08-17T17:47:07.915185+0000","last_undegraded":"2024-08-17T17:47:07.915185+0000","last_fullsized":"2024-08-17T17:47:07.915185+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.113279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.113279+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:38:07.015721+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":692,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:07.914945+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:47:07.914945+0000","last_peered":"2024-08-17T17:47:07.914945+0000","last_clean":"2024-08-17T17:47:07.914945+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:47:07.914945+0000","last_undegraded":"2024-08-17T17:47:07.914945+0000","last_fullsized":"2024-08-17T17:47:07.914945+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":1270,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098253+0000","last_change":"2024-08-17T17:41:37.677524+0000","last_active":"2024-08-17T17:47:08.098253+0000","last_peered":"2024-08-17T17:47:08.098253+0000","last_clean":"2024-08-17T17:47:08.098253+0000","last_became_active":"2024-08-17T17:41:37.676764+0000","last_became_peered":"2024-08-17T17:41:37.676764+0000","last_unstale":"2024-08-17T17:47:08.098253+0000","last_undegraded":"2024-08-17T17:47:08.098253+0000","last_fullsized":"2024-08-17T17:47:08.098253+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:14:49.341080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":695,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:07.915673+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:47:07.915673+0000","last_peered":"2024-08-17T17:47:07.915673+0000","last_clean":"2024-08-17T17:47:07.915673+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:47:07.915673+0000","last_undegraded":"2024-08-17T17:47:07.915673+0000","last_fullsized":"2024-08-17T17:47:07.915673+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":790,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.099053+0000","last_change":"2024-08-17T17:42:19.960161+0000","last_active":"2024-08-17T17:47:08.099053+0000","last_peered":"2024-08-17T17:47:08.099053+0000","last_clean":"2024-08-17T17:47:08.099053+0000","last_became_active":"2024-08-17T17:41:37.665223+0000","last_became_peered":"2024-08-17T17:41:37.665223+0000","last_unstale":"2024-08-17T17:47:08.099053+0000","last_undegraded":"2024-08-17T17:47:08.099053+0000","last_fullsized":"2024-08-17T17:47:08.099053+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:19.960086+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:19.960086+0000","last_clean_scrub_stamp":"2024-08-17T17:42:19.960086+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:57.861661+0000","scrub_duration":0.0022612000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":661,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098402+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:47:08.098402+0000","last_peered":"2024-08-17T17:47:08.098402+0000","last_clean":"2024-08-17T17:47:08.098402+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:47:08.098402+0000","last_undegraded":"2024-08-17T17:47:08.098402+0000","last_fullsized":"2024-08-17T17:47:08.098402+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":967,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.099168+0000","last_change":"2024-08-17T17:41:37.683212+0000","last_active":"2024-08-17T17:47:08.099168+0000","last_peered":"2024-08-17T17:47:08.099168+0000","last_clean":"2024-08-17T17:47:08.099168+0000","last_became_active":"2024-08-17T17:41:37.682730+0000","last_became_peered":"2024-08-17T17:41:37.682730+0000","last_unstale":"2024-08-17T17:47:08.099168+0000","last_undegraded":"2024-08-17T17:47:08.099168+0000","last_fullsized":"2024-08-17T17:47:08.099168+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:51:35.444433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":645,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.097862+0000","last_change":"2024-08-17T17:33:20.798613+0000","last_active":"2024-08-17T17:47:08.097862+0000","last_peered":"2024-08-17T17:47:08.097862+0000","last_clean":"2024-08-17T17:47:08.097862+0000","last_became_active":"2024-08-17T17:33:20.798260+0000","last_became_peered":"2024-08-17T17:33:20.798260+0000","last_unstale":"2024-08-17T17:47:08.097862+0000","last_undegraded":"2024-08-17T17:47:08.097862+0000","last_fullsized":"2024-08-17T17:47:08.097862+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:18:00.822681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":544,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098896+0000","last_change":"2024-08-17T17:41:44.058291+0000","last_active":"2024-08-17T17:47:08.098896+0000","last_peered":"2024-08-17T17:47:08.098896+0000","last_clean":"2024-08-17T17:47:08.098896+0000","last_became_active":"2024-08-17T17:41:12.200077+0000","last_became_peered":"2024-08-17T17:41:12.200077+0000","last_unstale":"2024-08-17T17:47:08.098896+0000","last_undegraded":"2024-08-17T17:47:08.098896+0000","last_fullsized":"2024-08-17T17:47:08.098896+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:44.058233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:44.058233+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:42:55.911623+0000","scrub_duration":0.0074268499999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":641,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098203+0000","last_change":"2024-08-17T17:41:34.627064+0000","last_active":"2024-08-17T17:47:08.098203+0000","last_peered":"2024-08-17T17:47:08.098203+0000","last_clean":"2024-08-17T17:47:08.098203+0000","last_became_active":"2024-08-17T17:41:34.626453+0000","last_became_peered":"2024-08-17T17:41:34.626453+0000","last_unstale":"2024-08-17T17:47:08.098203+0000","last_undegraded":"2024-08-17T17:47:08.098203+0000","last_fullsized":"2024-08-17T17:47:08.098203+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:02:11.894042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":825,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.097959+0000","last_change":"2024-08-17T17:33:20.799350+0000","last_active":"2024-08-17T17:47:08.097959+0000","last_peered":"2024-08-17T17:47:08.097959+0000","last_clean":"2024-08-17T17:47:08.097959+0000","last_became_active":"2024-08-17T17:33:20.798925+0000","last_became_peered":"2024-08-17T17:33:20.798925+0000","last_unstale":"2024-08-17T17:47:08.097959+0000","last_undegraded":"2024-08-17T17:47:08.097959+0000","last_fullsized":"2024-08-17T17:47:08.097959+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:16:36.121295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":361,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:07.915522+0000","last_change":"2024-08-17T17:33:44.163867+0000","last_active":"2024-08-17T17:47:07.915522+0000","last_peered":"2024-08-17T17:47:07.915522+0000","last_clean":"2024-08-17T17:47:07.915522+0000","last_became_active":"2024-08-17T17:33:44.163503+0000","last_became_peered":"2024-08-17T17:33:44.163503+0000","last_unstale":"2024-08-17T17:47:07.915522+0000","last_undegraded":"2024-08-17T17:47:07.915522+0000","last_fullsized":"2024-08-17T17:47:07.915522+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:31:53.573596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":544,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.099635+0000","last_change":"2024-08-17T17:41:43.035215+0000","last_active":"2024-08-17T17:47:08.099635+0000","last_peered":"2024-08-17T17:47:08.099635+0000","last_clean":"2024-08-17T17:47:08.099635+0000","last_became_active":"2024-08-17T17:41:12.201158+0000","last_became_peered":"2024-08-17T17:41:12.201158+0000","last_unstale":"2024-08-17T17:47:08.099635+0000","last_undegraded":"2024-08-17T17:47:08.099635+0000","last_fullsized":"2024-08-17T17:47:08.099635+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:43.035108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:43.035108+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:43:27.108994+0000","scrub_duration":0.0028877400000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":610,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098405+0000","last_change":"2024-08-17T17:33:20.772454+0000","last_active":"2024-08-17T17:47:08.098405+0000","last_peered":"2024-08-17T17:47:08.098405+0000","last_clean":"2024-08-17T17:47:08.098405+0000","last_became_active":"2024-08-17T17:33:20.771848+0000","last_became_peered":"2024-08-17T17:33:20.771848+0000","last_unstale":"2024-08-17T17:47:08.098405+0000","last_undegraded":"2024-08-17T17:47:08.098405+0000","last_fullsized":"2024-08-17T17:47:08.098405+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:46:42.300744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":71,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.099634+0000","last_change":"2024-08-17T17:41:34.611109+0000","last_active":"2024-08-17T17:47:08.099634+0000","last_peered":"2024-08-17T17:47:08.099634+0000","last_clean":"2024-08-17T17:47:08.099634+0000","last_became_active":"2024-08-17T17:41:34.610170+0000","last_became_peered":"2024-08-17T17:41:34.610170+0000","last_unstale":"2024-08-17T17:47:08.099634+0000","last_undegraded":"2024-08-17T17:47:08.099634+0000","last_fullsized":"2024-08-17T17:47:08.099634+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:39:05.301260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":1155,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.097647+0000","last_change":"2024-08-17T17:33:44.148227+0000","last_active":"2024-08-17T17:47:08.097647+0000","last_peered":"2024-08-17T17:47:08.097647+0000","last_clean":"2024-08-17T17:47:08.097647+0000","last_became_active":"2024-08-17T17:33:44.147627+0000","last_became_peered":"2024-08-17T17:33:44.147627+0000","last_unstale":"2024-08-17T17:47:08.097647+0000","last_undegraded":"2024-08-17T17:47:08.097647+0000","last_fullsized":"2024-08-17T17:47:08.097647+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:29:42.898460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1372,"num_read_kb":2843,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":857,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.097570+0000","last_change":"2024-08-17T17:38:22.254841+0000","last_active":"2024-08-17T17:47:08.097570+0000","last_peered":"2024-08-17T17:47:08.097570+0000","last_clean":"2024-08-17T17:47:08.097570+0000","last_became_active":"2024-08-17T17:38:21.306619+0000","last_became_peered":"2024-08-17T17:38:21.306619+0000","last_unstale":"2024-08-17T17:47:08.097570+0000","last_undegraded":"2024-08-17T17:47:08.097570+0000","last_fullsized":"2024-08-17T17:47:08.097570+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:22.254705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:22.254705+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:47:38.184089+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":606,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098534+0000","last_change":"2024-08-17T17:33:20.835784+0000","last_active":"2024-08-17T17:47:08.098534+0000","last_peered":"2024-08-17T17:47:08.098534+0000","last_clean":"2024-08-17T17:47:08.098534+0000","last_became_active":"2024-08-17T17:33:20.835371+0000","last_became_peered":"2024-08-17T17:33:20.835371+0000","last_unstale":"2024-08-17T17:47:08.098534+0000","last_undegraded":"2024-08-17T17:47:08.098534+0000","last_fullsized":"2024-08-17T17:47:08.098534+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:35:43.369055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":570,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.097507+0000","last_change":"2024-08-17T17:33:20.793468+0000","last_active":"2024-08-17T17:47:08.097507+0000","last_peered":"2024-08-17T17:47:08.097507+0000","last_clean":"2024-08-17T17:47:08.097507+0000","last_became_active":"2024-08-17T17:33:20.793166+0000","last_became_peered":"2024-08-17T17:33:20.793166+0000","last_unstale":"2024-08-17T17:47:08.097507+0000","last_undegraded":"2024-08-17T17:47:08.097507+0000","last_fullsized":"2024-08-17T17:47:08.097507+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:12:52.121050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":1065,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:07.915306+0000","last_change":"2024-08-17T17:20:14.433288+0000","last_active":"2024-08-17T17:47:07.915306+0000","last_peered":"2024-08-17T17:47:07.915306+0000","last_clean":"2024-08-17T17:47:07.915306+0000","last_became_active":"2024-08-17T17:20:14.432512+0000","last_became_peered":"2024-08-17T17:20:14.432512+0000","last_unstale":"2024-08-17T17:47:07.915306+0000","last_undegraded":"2024-08-17T17:47:07.915306+0000","last_fullsized":"2024-08-17T17:47:07.915306+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:55:07.722751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":878,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.097564+0000","last_change":"2024-08-17T17:41:37.683450+0000","last_active":"2024-08-17T17:47:08.097564+0000","last_peered":"2024-08-17T17:47:08.097564+0000","last_clean":"2024-08-17T17:47:08.097564+0000","last_became_active":"2024-08-17T17:41:37.683103+0000","last_became_peered":"2024-08-17T17:41:37.683103+0000","last_unstale":"2024-08-17T17:47:08.097564+0000","last_undegraded":"2024-08-17T17:47:08.097564+0000","last_fullsized":"2024-08-17T17:47:08.097564+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.289528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.289528+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:39:17.141469+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":486,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098625+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:47:08.098625+0000","last_peered":"2024-08-17T17:47:08.098625+0000","last_clean":"2024-08-17T17:47:08.098625+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:47:08.098625+0000","last_undegraded":"2024-08-17T17:47:08.098625+0000","last_fullsized":"2024-08-17T17:47:08.098625+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":699,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.097594+0000","last_change":"2024-08-17T17:41:34.614108+0000","last_active":"2024-08-17T17:47:08.097594+0000","last_peered":"2024-08-17T17:47:08.097594+0000","last_clean":"2024-08-17T17:47:08.097594+0000","last_became_active":"2024-08-17T17:41:34.613481+0000","last_became_peered":"2024-08-17T17:41:34.613481+0000","last_unstale":"2024-08-17T17:47:08.097594+0000","last_undegraded":"2024-08-17T17:47:08.097594+0000","last_fullsized":"2024-08-17T17:47:08.097594+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:21:02.346115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":932,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.097504+0000","last_change":"2024-08-17T17:33:20.797705+0000","last_active":"2024-08-17T17:47:08.097504+0000","last_peered":"2024-08-17T17:47:08.097504+0000","last_clean":"2024-08-17T17:47:08.097504+0000","last_became_active":"2024-08-17T17:33:20.797172+0000","last_became_peered":"2024-08-17T17:33:20.797172+0000","last_unstale":"2024-08-17T17:47:08.097504+0000","last_undegraded":"2024-08-17T17:47:08.097504+0000","last_fullsized":"2024-08-17T17:47:08.097504+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:19:30.754999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":874,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098112+0000","last_change":"2024-08-17T17:41:37.676347+0000","last_active":"2024-08-17T17:47:08.098112+0000","last_peered":"2024-08-17T17:47:08.098112+0000","last_clean":"2024-08-17T17:47:08.098112+0000","last_became_active":"2024-08-17T17:41:37.675869+0000","last_became_peered":"2024-08-17T17:41:37.675869+0000","last_unstale":"2024-08-17T17:47:08.098112+0000","last_undegraded":"2024-08-17T17:47:08.098112+0000","last_fullsized":"2024-08-17T17:47:08.098112+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:21.814077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:21.814077+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:04:16.240754+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":248,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.099875+0000","last_change":"2024-08-17T17:41:34.618631+0000","last_active":"2024-08-17T17:47:08.099875+0000","last_peered":"2024-08-17T17:47:08.099875+0000","last_clean":"2024-08-17T17:47:08.099875+0000","last_became_active":"2024-08-17T17:41:34.618313+0000","last_became_peered":"2024-08-17T17:41:34.618313+0000","last_unstale":"2024-08-17T17:47:08.099875+0000","last_undegraded":"2024-08-17T17:47:08.099875+0000","last_fullsized":"2024-08-17T17:47:08.099875+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:07:13.915088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":649,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:07.915088+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:47:07.915088+0000","last_peered":"2024-08-17T17:47:07.915088+0000","last_clean":"2024-08-17T17:47:07.915088+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:47:07.915088+0000","last_undegraded":"2024-08-17T17:47:07.915088+0000","last_fullsized":"2024-08-17T17:47:07.915088+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":813,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098580+0000","last_change":"2024-08-17T17:33:44.149928+0000","last_active":"2024-08-17T17:47:08.098580+0000","last_peered":"2024-08-17T17:47:08.098580+0000","last_clean":"2024-08-17T17:47:08.098580+0000","last_became_active":"2024-08-17T17:33:44.149498+0000","last_became_peered":"2024-08-17T17:33:44.149498+0000","last_unstale":"2024-08-17T17:47:08.098580+0000","last_undegraded":"2024-08-17T17:47:08.098580+0000","last_fullsized":"2024-08-17T17:47:08.098580+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:17:04.963202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":838,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.097536+0000","last_change":"2024-08-17T17:38:25.247159+0000","last_active":"2024-08-17T17:47:08.097536+0000","last_peered":"2024-08-17T17:47:08.097536+0000","last_clean":"2024-08-17T17:47:08.097536+0000","last_became_active":"2024-08-17T17:38:21.311361+0000","last_became_peered":"2024-08-17T17:38:21.311361+0000","last_unstale":"2024-08-17T17:47:08.097536+0000","last_undegraded":"2024-08-17T17:47:08.097536+0000","last_fullsized":"2024-08-17T17:47:08.097536+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:25.247027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:25.247027+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:46:21.734712+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":596,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098623+0000","last_change":"2024-08-17T17:41:34.625551+0000","last_active":"2024-08-17T17:47:08.098623+0000","last_peered":"2024-08-17T17:47:08.098623+0000","last_clean":"2024-08-17T17:47:08.098623+0000","last_became_active":"2024-08-17T17:41:34.625251+0000","last_became_peered":"2024-08-17T17:41:34.625251+0000","last_unstale":"2024-08-17T17:47:08.098623+0000","last_undegraded":"2024-08-17T17:47:08.098623+0000","last_fullsized":"2024-08-17T17:47:08.098623+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:28:22.354213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":701,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:07.915005+0000","last_change":"2024-08-17T17:33:20.796216+0000","last_active":"2024-08-17T17:47:07.915005+0000","last_peered":"2024-08-17T17:47:07.915005+0000","last_clean":"2024-08-17T17:47:07.915005+0000","last_became_active":"2024-08-17T17:33:20.795762+0000","last_became_peered":"2024-08-17T17:33:20.795762+0000","last_unstale":"2024-08-17T17:47:07.915005+0000","last_undegraded":"2024-08-17T17:47:07.915005+0000","last_fullsized":"2024-08-17T17:47:07.915005+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:47.038509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":524,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098278+0000","last_change":"2024-08-17T17:41:41.036615+0000","last_active":"2024-08-17T17:47:08.098278+0000","last_peered":"2024-08-17T17:47:08.098278+0000","last_clean":"2024-08-17T17:47:08.098278+0000","last_became_active":"2024-08-17T17:41:12.329214+0000","last_became_peered":"2024-08-17T17:41:12.329214+0000","last_unstale":"2024-08-17T17:47:08.098278+0000","last_undegraded":"2024-08-17T17:47:08.098278+0000","last_fullsized":"2024-08-17T17:47:08.098278+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:41.036524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:41.036524+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:59:58.695900+0000","scrub_duration":0.001820803,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":71,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.099748+0000","last_change":"2024-08-17T17:41:34.622102+0000","last_active":"2024-08-17T17:47:08.099748+0000","last_peered":"2024-08-17T17:47:08.099748+0000","last_clean":"2024-08-17T17:47:08.099748+0000","last_became_active":"2024-08-17T17:41:34.621733+0000","last_became_peered":"2024-08-17T17:41:34.621733+0000","last_unstale":"2024-08-17T17:47:08.099748+0000","last_undegraded":"2024-08-17T17:47:08.099748+0000","last_fullsized":"2024-08-17T17:47:08.099748+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:39:41.668709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":71,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098849+0000","last_change":"2024-08-17T17:41:34.607602+0000","last_active":"2024-08-17T17:47:08.098849+0000","last_peered":"2024-08-17T17:47:08.098849+0000","last_clean":"2024-08-17T17:47:08.098849+0000","last_became_active":"2024-08-17T17:41:34.607300+0000","last_became_peered":"2024-08-17T17:41:34.607300+0000","last_unstale":"2024-08-17T17:47:08.098849+0000","last_undegraded":"2024-08-17T17:47:08.098849+0000","last_fullsized":"2024-08-17T17:47:08.098849+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:20:31.486698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"90'149","reported_seq":891,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.097949+0000","last_change":"2024-08-17T17:41:41.743931+0000","last_active":"2024-08-17T17:47:08.097949+0000","last_peered":"2024-08-17T17:47:08.097949+0000","last_clean":"2024-08-17T17:47:08.097949+0000","last_became_active":"2024-08-17T17:41:12.203771+0000","last_became_peered":"2024-08-17T17:41:12.203771+0000","last_unstale":"2024-08-17T17:47:08.097949+0000","last_undegraded":"2024-08-17T17:47:08.097949+0000","last_fullsized":"2024-08-17T17:47:08.097949+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:41.743570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:41.743570+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:21:03.865738+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":71,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.099364+0000","last_change":"2024-08-17T17:41:34.619268+0000","last_active":"2024-08-17T17:47:08.099364+0000","last_peered":"2024-08-17T17:47:08.099364+0000","last_clean":"2024-08-17T17:47:08.099364+0000","last_became_active":"2024-08-17T17:41:34.618914+0000","last_became_peered":"2024-08-17T17:41:34.618914+0000","last_unstale":"2024-08-17T17:47:08.099364+0000","last_undegraded":"2024-08-17T17:47:08.099364+0000","last_fullsized":"2024-08-17T17:47:08.099364+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:08:54.248386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":71,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098598+0000","last_change":"2024-08-17T17:41:34.622803+0000","last_active":"2024-08-17T17:47:08.098598+0000","last_peered":"2024-08-17T17:47:08.098598+0000","last_clean":"2024-08-17T17:47:08.098598+0000","last_became_active":"2024-08-17T17:41:34.622525+0000","last_became_peered":"2024-08-17T17:41:34.622525+0000","last_unstale":"2024-08-17T17:47:08.098598+0000","last_undegraded":"2024-08-17T17:47:08.098598+0000","last_fullsized":"2024-08-17T17:47:08.098598+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:42:39.728377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":886,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:07.914871+0000","last_change":"2024-08-17T17:43:55.595647+0000","last_active":"2024-08-17T17:47:07.914871+0000","last_peered":"2024-08-17T17:47:07.914871+0000","last_clean":"2024-08-17T17:47:07.914871+0000","last_became_active":"2024-08-17T17:43:55.595256+0000","last_became_peered":"2024-08-17T17:43:55.595256+0000","last_unstale":"2024-08-17T17:47:07.914871+0000","last_undegraded":"2024-08-17T17:47:07.914871+0000","last_fullsized":"2024-08-17T17:47:07.914871+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:46.587712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:46.587712+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:05:01.927968+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":71,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.099368+0000","last_change":"2024-08-17T17:41:34.611049+0000","last_active":"2024-08-17T17:47:08.099368+0000","last_peered":"2024-08-17T17:47:08.099368+0000","last_clean":"2024-08-17T17:47:08.099368+0000","last_became_active":"2024-08-17T17:41:34.610727+0000","last_became_peered":"2024-08-17T17:41:34.610727+0000","last_unstale":"2024-08-17T17:47:08.099368+0000","last_undegraded":"2024-08-17T17:47:08.099368+0000","last_fullsized":"2024-08-17T17:47:08.099368+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:04:05.477660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":743,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:07.915218+0000","last_change":"2024-08-17T17:41:34.630434+0000","last_active":"2024-08-17T17:47:07.915218+0000","last_peered":"2024-08-17T17:47:07.915218+0000","last_clean":"2024-08-17T17:47:07.915218+0000","last_became_active":"2024-08-17T17:41:34.616406+0000","last_became_peered":"2024-08-17T17:41:34.616406+0000","last_unstale":"2024-08-17T17:47:07.915218+0000","last_undegraded":"2024-08-17T17:47:07.915218+0000","last_fullsized":"2024-08-17T17:47:07.915218+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:30:44.817203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":858,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:07.914961+0000","last_change":"2024-08-17T17:41:44.585386+0000","last_active":"2024-08-17T17:47:07.914961+0000","last_peered":"2024-08-17T17:47:07.914961+0000","last_clean":"2024-08-17T17:47:07.914961+0000","last_became_active":"2024-08-17T17:41:12.307398+0000","last_became_peered":"2024-08-17T17:41:12.307398+0000","last_unstale":"2024-08-17T17:47:07.914961+0000","last_undegraded":"2024-08-17T17:47:07.914961+0000","last_fullsized":"2024-08-17T17:47:07.914961+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:44.583653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:44.583653+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:27:44.756681+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":707,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098398+0000","last_change":"2024-08-17T17:41:34.615197+0000","last_active":"2024-08-17T17:47:08.098398+0000","last_peered":"2024-08-17T17:47:08.098398+0000","last_clean":"2024-08-17T17:47:08.098398+0000","last_became_active":"2024-08-17T17:41:34.614873+0000","last_became_peered":"2024-08-17T17:41:34.614873+0000","last_unstale":"2024-08-17T17:47:08.098398+0000","last_undegraded":"2024-08-17T17:47:08.098398+0000","last_fullsized":"2024-08-17T17:47:08.098398+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:21:57.391953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":343,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:07.915369+0000","last_change":"2024-08-17T17:33:44.154097+0000","last_active":"2024-08-17T17:47:07.915369+0000","last_peered":"2024-08-17T17:47:07.915369+0000","last_clean":"2024-08-17T17:47:07.915369+0000","last_became_active":"2024-08-17T17:33:44.152972+0000","last_became_peered":"2024-08-17T17:33:44.152972+0000","last_unstale":"2024-08-17T17:47:07.915369+0000","last_undegraded":"2024-08-17T17:47:07.915369+0000","last_fullsized":"2024-08-17T17:47:07.915369+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:22:42.602017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"90'149","reported_seq":715,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.099818+0000","last_change":"2024-08-17T17:42:20.927007+0000","last_active":"2024-08-17T17:47:08.099818+0000","last_peered":"2024-08-17T17:47:08.099818+0000","last_clean":"2024-08-17T17:47:08.099818+0000","last_became_active":"2024-08-17T17:41:37.670914+0000","last_became_peered":"2024-08-17T17:41:37.670914+0000","last_unstale":"2024-08-17T17:47:08.099818+0000","last_undegraded":"2024-08-17T17:47:08.099818+0000","last_fullsized":"2024-08-17T17:47:08.099818+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:20.926924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:20.926924+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:57:44.156600+0000","scrub_duration":0.0018485109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":235,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098471+0000","last_change":"2024-08-17T17:33:20.795582+0000","last_active":"2024-08-17T17:47:08.098471+0000","last_peered":"2024-08-17T17:47:08.098471+0000","last_clean":"2024-08-17T17:47:08.098471+0000","last_became_active":"2024-08-17T17:33:20.795149+0000","last_became_peered":"2024-08-17T17:33:20.795149+0000","last_unstale":"2024-08-17T17:47:08.098471+0000","last_undegraded":"2024-08-17T17:47:08.098471+0000","last_fullsized":"2024-08-17T17:47:08.098471+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:14.020923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":157,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.099886+0000","last_change":"2024-08-17T17:33:44.145399+0000","last_active":"2024-08-17T17:47:08.099886+0000","last_peered":"2024-08-17T17:47:08.099886+0000","last_clean":"2024-08-17T17:47:08.099886+0000","last_became_active":"2024-08-17T17:33:44.144751+0000","last_became_peered":"2024-08-17T17:33:44.144751+0000","last_unstale":"2024-08-17T17:47:08.099886+0000","last_undegraded":"2024-08-17T17:47:08.099886+0000","last_fullsized":"2024-08-17T17:47:08.099886+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:57:54.276350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.f","version":"90'149","reported_seq":583,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.099743+0000","last_change":"2024-08-17T17:41:37.668050+0000","last_active":"2024-08-17T17:47:08.099743+0000","last_peered":"2024-08-17T17:47:08.099743+0000","last_clean":"2024-08-17T17:47:08.099743+0000","last_became_active":"2024-08-17T17:41:37.667655+0000","last_became_peered":"2024-08-17T17:41:37.667655+0000","last_unstale":"2024-08-17T17:47:08.099743+0000","last_undegraded":"2024-08-17T17:47:08.099743+0000","last_fullsized":"2024-08-17T17:47:08.099743+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:32.843802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:32.843802+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:39:46.766280+0000","scrub_duration":0.0040803669999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":71,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098591+0000","last_change":"2024-08-17T17:41:34.621078+0000","last_active":"2024-08-17T17:47:08.098591+0000","last_peered":"2024-08-17T17:47:08.098591+0000","last_clean":"2024-08-17T17:47:08.098591+0000","last_became_active":"2024-08-17T17:41:34.620730+0000","last_became_peered":"2024-08-17T17:41:34.620730+0000","last_unstale":"2024-08-17T17:47:08.098591+0000","last_undegraded":"2024-08-17T17:47:08.098591+0000","last_fullsized":"2024-08-17T17:47:08.098591+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:55:01.312283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":364,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.099100+0000","last_change":"2024-08-17T17:41:34.637696+0000","last_active":"2024-08-17T17:47:08.099100+0000","last_peered":"2024-08-17T17:47:08.099100+0000","last_clean":"2024-08-17T17:47:08.099100+0000","last_became_active":"2024-08-17T17:41:34.621388+0000","last_became_peered":"2024-08-17T17:41:34.621388+0000","last_unstale":"2024-08-17T17:47:08.099100+0000","last_undegraded":"2024-08-17T17:47:08.099100+0000","last_fullsized":"2024-08-17T17:47:08.099100+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:13:04.607036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"90'149","reported_seq":883,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:07.915220+0000","last_change":"2024-08-17T17:41:42.603841+0000","last_active":"2024-08-17T17:47:07.915220+0000","last_peered":"2024-08-17T17:47:07.915220+0000","last_clean":"2024-08-17T17:47:07.915220+0000","last_became_active":"2024-08-17T17:41:12.203400+0000","last_became_peered":"2024-08-17T17:41:12.203400+0000","last_unstale":"2024-08-17T17:47:07.915220+0000","last_undegraded":"2024-08-17T17:47:07.915220+0000","last_fullsized":"2024-08-17T17:47:07.915220+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:42.603561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:42.603561+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:50:12.843193+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":161,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.099467+0000","last_change":"2024-08-17T17:33:44.145598+0000","last_active":"2024-08-17T17:47:08.099467+0000","last_peered":"2024-08-17T17:47:08.099467+0000","last_clean":"2024-08-17T17:47:08.099467+0000","last_became_active":"2024-08-17T17:33:44.145283+0000","last_became_peered":"2024-08-17T17:33:44.145283+0000","last_unstale":"2024-08-17T17:47:08.099467+0000","last_undegraded":"2024-08-17T17:47:08.099467+0000","last_fullsized":"2024-08-17T17:47:08.099467+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:48:11.965226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":368,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.099346+0000","last_change":"2024-08-17T17:41:34.635980+0000","last_active":"2024-08-17T17:47:08.099346+0000","last_peered":"2024-08-17T17:47:08.099346+0000","last_clean":"2024-08-17T17:47:08.099346+0000","last_became_active":"2024-08-17T17:41:34.620596+0000","last_became_peered":"2024-08-17T17:41:34.620596+0000","last_unstale":"2024-08-17T17:47:08.099346+0000","last_undegraded":"2024-08-17T17:47:08.099346+0000","last_fullsized":"2024-08-17T17:47:08.099346+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:19:15.792577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"90'149","reported_seq":817,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098578+0000","last_change":"2024-08-17T17:42:16.610940+0000","last_active":"2024-08-17T17:47:08.098578+0000","last_peered":"2024-08-17T17:47:08.098578+0000","last_clean":"2024-08-17T17:47:08.098578+0000","last_became_active":"2024-08-17T17:41:37.670855+0000","last_became_peered":"2024-08-17T17:41:37.670855+0000","last_unstale":"2024-08-17T17:47:08.098578+0000","last_undegraded":"2024-08-17T17:47:08.098578+0000","last_fullsized":"2024-08-17T17:47:08.098578+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:16.610828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:16.610828+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:48:25.918610+0000","scrub_duration":0.086892487000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":492,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.099636+0000","last_change":"2024-08-17T17:43:52.520659+0000","last_active":"2024-08-17T17:47:08.099636+0000","last_peered":"2024-08-17T17:47:08.099636+0000","last_clean":"2024-08-17T17:47:08.099636+0000","last_became_active":"2024-08-17T17:43:52.516799+0000","last_became_peered":"2024-08-17T17:43:52.516799+0000","last_unstale":"2024-08-17T17:47:08.099636+0000","last_undegraded":"2024-08-17T17:47:08.099636+0000","last_fullsized":"2024-08-17T17:47:08.099636+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:25:59.408383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":699,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:07.915590+0000","last_change":"2024-08-17T17:33:20.794682+0000","last_active":"2024-08-17T17:47:07.915590+0000","last_peered":"2024-08-17T17:47:07.915590+0000","last_clean":"2024-08-17T17:47:07.915590+0000","last_became_active":"2024-08-17T17:33:20.794238+0000","last_became_peered":"2024-08-17T17:33:20.794238+0000","last_unstale":"2024-08-17T17:47:07.915590+0000","last_undegraded":"2024-08-17T17:47:07.915590+0000","last_fullsized":"2024-08-17T17:47:07.915590+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:15:55.821920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"90'149","reported_seq":858,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098096+0000","last_change":"2024-08-17T17:38:30.446165+0000","last_active":"2024-08-17T17:47:08.098096+0000","last_peered":"2024-08-17T17:47:08.098096+0000","last_clean":"2024-08-17T17:47:08.098096+0000","last_became_active":"2024-08-17T17:38:30.445048+0000","last_became_peered":"2024-08-17T17:38:30.445048+0000","last_unstale":"2024-08-17T17:47:08.098096+0000","last_undegraded":"2024-08-17T17:47:08.098096+0000","last_fullsized":"2024-08-17T17:47:08.098096+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:24.291592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:24.291592+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:17.100154+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":71,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098766+0000","last_change":"2024-08-17T17:41:34.611693+0000","last_active":"2024-08-17T17:47:08.098766+0000","last_peered":"2024-08-17T17:47:08.098766+0000","last_clean":"2024-08-17T17:47:08.098766+0000","last_became_active":"2024-08-17T17:41:34.611374+0000","last_became_peered":"2024-08-17T17:41:34.611374+0000","last_unstale":"2024-08-17T17:47:08.098766+0000","last_undegraded":"2024-08-17T17:47:08.098766+0000","last_fullsized":"2024-08-17T17:47:08.098766+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:17:09.897446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":485,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098531+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:47:08.098531+0000","last_peered":"2024-08-17T17:47:08.098531+0000","last_clean":"2024-08-17T17:47:08.098531+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:47:08.098531+0000","last_undegraded":"2024-08-17T17:47:08.098531+0000","last_fullsized":"2024-08-17T17:47:08.098531+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"90'149","reported_seq":882,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.097822+0000","last_change":"2024-08-17T17:41:37.681187+0000","last_active":"2024-08-17T17:47:08.097822+0000","last_peered":"2024-08-17T17:47:08.097822+0000","last_clean":"2024-08-17T17:47:08.097822+0000","last_became_active":"2024-08-17T17:41:37.680605+0000","last_became_peered":"2024-08-17T17:41:37.680605+0000","last_unstale":"2024-08-17T17:47:08.097822+0000","last_undegraded":"2024-08-17T17:47:08.097822+0000","last_fullsized":"2024-08-17T17:47:08.097822+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.844252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.844252+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:13:31.682357+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":485,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.097690+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:47:08.097690+0000","last_peered":"2024-08-17T17:47:08.097690+0000","last_clean":"2024-08-17T17:47:08.097690+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:47:08.097690+0000","last_undegraded":"2024-08-17T17:47:08.097690+0000","last_fullsized":"2024-08-17T17:47:08.097690+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":346,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:07.915452+0000","last_change":"2024-08-17T17:33:44.156036+0000","last_active":"2024-08-17T17:47:07.915452+0000","last_peered":"2024-08-17T17:47:07.915452+0000","last_clean":"2024-08-17T17:47:07.915452+0000","last_became_active":"2024-08-17T17:33:44.155758+0000","last_became_peered":"2024-08-17T17:33:44.155758+0000","last_unstale":"2024-08-17T17:47:07.915452+0000","last_undegraded":"2024-08-17T17:47:07.915452+0000","last_fullsized":"2024-08-17T17:47:07.915452+0000","mapping_epoch":272,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":273,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:23.443405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.10","version":"90'149","reported_seq":490,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.099344+0000","last_change":"2024-08-17T17:38:31.874623+0000","last_active":"2024-08-17T17:47:08.099344+0000","last_peered":"2024-08-17T17:47:08.099344+0000","last_clean":"2024-08-17T17:47:08.099344+0000","last_became_active":"2024-08-17T17:38:30.339313+0000","last_became_peered":"2024-08-17T17:38:30.339313+0000","last_unstale":"2024-08-17T17:47:08.099344+0000","last_undegraded":"2024-08-17T17:47:08.099344+0000","last_fullsized":"2024-08-17T17:47:08.099344+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:31.874423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:31.874423+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:28:17.980339+0000","scrub_duration":0.0017696579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":431,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.098994+0000","last_change":"2024-08-17T17:41:34.624198+0000","last_active":"2024-08-17T17:47:08.098994+0000","last_peered":"2024-08-17T17:47:08.098994+0000","last_clean":"2024-08-17T17:47:08.098994+0000","last_became_active":"2024-08-17T17:41:34.616071+0000","last_became_peered":"2024-08-17T17:41:34.616071+0000","last_unstale":"2024-08-17T17:47:08.098994+0000","last_undegraded":"2024-08-17T17:47:08.098994+0000","last_fullsized":"2024-08-17T17:47:08.098994+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:34:48.805354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":685,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:07.915318+0000","last_change":"2024-08-17T17:33:20.797029+0000","last_active":"2024-08-17T17:47:07.915318+0000","last_peered":"2024-08-17T17:47:07.915318+0000","last_clean":"2024-08-17T17:47:07.915318+0000","last_became_active":"2024-08-17T17:33:20.796679+0000","last_became_peered":"2024-08-17T17:33:20.796679+0000","last_unstale":"2024-08-17T17:47:07.915318+0000","last_undegraded":"2024-08-17T17:47:07.915318+0000","last_fullsized":"2024-08-17T17:47:07.915318+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:51.438262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"90'149","reported_seq":864,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-08-17T17:47:08.097692+0000","last_change":"2024-08-17T17:38:30.378664+0000","last_active":"2024-08-17T17:47:08.097692+0000","last_peered":"2024-08-17T17:47:08.097692+0000","last_clean":"2024-08-17T17:47:08.097692+0000","last_became_active":"2024-08-17T17:38:30.342287+0000","last_became_peered":"2024-08-17T17:38:30.342287+0000","last_unstale":"2024-08-17T17:47:08.097692+0000","last_undegraded":"2024-08-17T17:47:08.097692+0000","last_fullsized":"2024-08-17T17:47:08.097692+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:30.378592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:30.378592+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:04:28.933354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":11256,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19280,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5978,"num_read_kb":12991,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4619,"ondisk_log_size":4619,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234485,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":911896,"kb_used_data":9676,"kb_used_omap":1,"kb_used_meta":902206,"kb_avail":92833256,"statfs":{"total":95995035648,"available":95061254144,"internally_reserved":0,"allocated":9908224,"data_stored":7611645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":923859288},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496120,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":900692,"kb_used_data":9736,"kb_used_omap":1,"kb_used_meta":890942,"kb_avail":92844460,"statfs":{"total":95995035648,"available":95072727040,"internally_reserved":0,"allocated":9969664,"data_stored":7684317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":912324952},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757757,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":887892,"kb_used_data":5576,"kb_used_omap":1,"kb_used_meta":882302,"kb_avail":92857260,"statfs":{"total":95995035648,"available":95085834240,"internally_reserved":0,"allocated":5709824,"data_stored":3421046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":903477592},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019395,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306848,"kb_used_data":5332,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93438304,"statfs":{"total":95995035648,"available":95680823296,"internally_reserved":0,"allocated":5459968,"data_stored":3162920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435621,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":965760,"kb_used_data":5620,"kb_used_omap":4,"kb_used_meta":960123,"kb_avail":92779392,"statfs":{"total":95995035648,"available":95006097408,"internally_reserved":0,"allocated":5754880,"data_stored":3477814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":983165962},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599149,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":845448,"kb_used_data":9660,"kb_used_omap":7,"kb_used_meta":835768,"kb_avail":92899704,"statfs":{"total":95995035648,"available":95129296896,"internally_reserved":0,"allocated":9891840,"data_stored":7607640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":855826594},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":268,"seq":1151051235495,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":455552,"kb_used_data":5300,"kb_used_omap":10,"kb_used_meta":450229,"kb_avail":93289600,"statfs":{"total":95995035648,"available":95528550400,"internally_reserved":0,"allocated":5427200,"data_stored":3150079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10902,"internal_metadata":461034858},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926202,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":940100,"kb_used_data":5624,"kb_used_omap":7,"kb_used_meta":934456,"kb_avail":92805052,"statfs":{"total":95995035648,"available":95032373248,"internally_reserved":0,"allocated":5758976,"data_stored":3478324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":956883265},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7550,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:47:16.565 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-08-17T17:47:16.565 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-17T17:47:16.566 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-08-17T17:47:16.566 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-08-17T17:47:16.566 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-08-17T17:47:16.566 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-08-17T17:47:17.027 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-17T17:47:17.078 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:47:17.700 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:47:18.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:17 smithi053 ceph-mon[89963]: pgmap v1359: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:18.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:17 smithi003 ceph-mon[103344]: pgmap v1359: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:18.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:17 smithi003 ceph-mon[100615]: pgmap v1359: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:19.709 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-08-17T17:47:19.709 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "" 2024-08-17T17:47:19.709 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-08-17T17:47:20.010 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:19 smithi003 ceph-mon[103344]: pgmap v1360: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:20.011 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:19 smithi003 ceph-mon[100615]: pgmap v1360: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:20.174 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:19 smithi053 ceph-mon[89963]: pgmap v1360: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:20.389 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-08-17T17:47:20.389 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-17T17:47:20.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:47:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:47:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:47:22.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:21 smithi053 ceph-mon[89963]: pgmap v1361: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:22.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:21 smithi003 ceph-mon[103344]: pgmap v1361: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:22.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:21 smithi003 ceph-mon[100615]: pgmap v1361: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:23.920 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:23 smithi053 ceph-mon[89963]: pgmap v1362: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:23.920 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:47:23.920 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:47:24.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:23 smithi003 ceph-mon[103344]: pgmap v1362: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:24.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:47:24.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:47:24.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:23 smithi003 ceph-mon[100615]: pgmap v1362: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:24.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:47:24.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:47:26.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:25 smithi003 ceph-mon[103344]: pgmap v1363: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:26.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:25 smithi003 ceph-mon[100615]: pgmap v1363: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:26.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:47:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:47:25] "GET /metrics HTTP/1.1" 200 34138 "" "Prometheus/2.51.0" 2024-08-17T17:47:26.111 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:25 smithi053 ceph-mon[89963]: pgmap v1363: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:26.514 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-08-17T17:47:26.514 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd pool set .mgr pgp_num 31 2024-08-17T17:47:26.876 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:47:28.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:27 smithi003 ceph-mon[103344]: pgmap v1364: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:28.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:27 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:47:28.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:27 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:47:28.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:27 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:47:28.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:27 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:47:28.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:27 smithi003 ceph-mon[100615]: pgmap v1364: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:28.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:27 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:47:28.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:27 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:47:28.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:27 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:47:28.024 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:27 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:47:28.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:27 smithi053 ceph-mon[89963]: pgmap v1364: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:28.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:27 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:47:28.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:27 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:47:28.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:27 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:47:28.217 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:27 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:47:29.006 INFO:teuthology.orchestra.run.smithi003.stderr:set pool 1 pgp_num to 31 2024-08-17T17:47:29.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:28 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:47:29.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:28 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:47:29.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:28 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:47:29.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:28 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3666176621' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"} : dispatch 2024-08-17T17:47:29.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:28 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:47:29.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:28 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:47:29.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:28 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:47:29.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:28 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3666176621' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"} : dispatch 2024-08-17T17:47:29.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:28 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:47:29.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:28 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:47:29.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:28 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:47:29.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:28 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3666176621' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"} : dispatch 2024-08-17T17:47:29.752 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-08-17T17:47:30.027 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:47:30.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:30 smithi003 ceph-mon[103344]: pgmap v1365: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:30.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:30 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3666176621' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-08-17T17:47:30.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:30 smithi003 ceph-mon[103344]: osdmap e408: 8 total, 8 up, 8 in 2024-08-17T17:47:30.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:30 smithi003 ceph-mon[100615]: pgmap v1365: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:30.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:30 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3666176621' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-08-17T17:47:30.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:30 smithi003 ceph-mon[100615]: osdmap e408: 8 total, 8 up, 8 in 2024-08-17T17:47:30.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:30 smithi053 ceph-mon[89963]: pgmap v1365: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:30.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:30 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3666176621' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-08-17T17:47:30.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:30 smithi053 ceph-mon[89963]: osdmap e408: 8 total, 8 up, 8 in 2024-08-17T17:47:30.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:47:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:47:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:47:31.734 INFO:teuthology.orchestra.run.smithi003.stdout:mon.a: {} 2024-08-17T17:47:31.735 INFO:teuthology.orchestra.run.smithi003.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-17T17:47:31.735 INFO:teuthology.orchestra.run.smithi003.stdout:mon.b: {} 2024-08-17T17:47:31.735 INFO:teuthology.orchestra.run.smithi003.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-17T17:47:31.744 INFO:teuthology.orchestra.run.smithi003.stdout:mon.c: {} 2024-08-17T17:47:31.744 INFO:teuthology.orchestra.run.smithi003.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-17T17:47:32.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:32 smithi003 ceph-mon[100615]: pgmap v1367: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:32.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:32 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2646120212' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-17T17:47:32.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:32 smithi003 ceph-mon[103344]: pgmap v1367: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:32.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:32 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2646120212' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-17T17:47:32.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:32 smithi053 ceph-mon[89963]: pgmap v1367: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:32.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:32 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2646120212' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-17T17:47:32.493 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 0 1 2024-08-17T17:47:32.765 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:47:34.236 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:34 smithi003 ceph-mon[103344]: pgmap v1368: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:34.236 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:34 smithi003 ceph-mon[100615]: pgmap v1368: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:34.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:34 smithi053 ceph-mon[89963]: pgmap v1368: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:35.017 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.0 to 1 (10000) 2024-08-17T17:47:35.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:35 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/132851303' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T17:47:35.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:35 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/132851303' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T17:47:35.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:35 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/132851303' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-17T17:47:35.701 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 0 1 2024-08-17T17:47:35.971 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:47:36.018 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:47:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:47:35] "GET /metrics HTTP/1.1" 200 34138 "" "Prometheus/2.51.0" 2024-08-17T17:47:36.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:36 smithi003 ceph-mon[100615]: pgmap v1369: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:36.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:36 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/132851303' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-17T17:47:36.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:36 smithi003 ceph-mon[100615]: osdmap e409: 8 total, 8 up, 8 in 2024-08-17T17:47:36.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:36 smithi003 ceph-mon[103344]: pgmap v1369: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:36.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:36 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/132851303' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-17T17:47:36.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:36 smithi003 ceph-mon[103344]: osdmap e409: 8 total, 8 up, 8 in 2024-08-17T17:47:36.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:36 smithi053 ceph-mon[89963]: pgmap v1369: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:36.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:36 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/132851303' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-17T17:47:36.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:36 smithi053 ceph-mon[89963]: osdmap e409: 8 total, 8 up, 8 in 2024-08-17T17:47:38.025 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-08-17T17:47:38.372 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:38 smithi003 ceph-mon[103344]: pgmap v1371: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:38.372 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:38 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2589718233' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-08-17T17:47:38.372 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:38 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-08-17T17:47:38.372 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:38 smithi003 ceph-mon[100615]: pgmap v1371: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:38.372 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:38 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2589718233' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-08-17T17:47:38.372 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:38 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-08-17T17:47:38.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:38 smithi053 ceph-mon[89963]: pgmap v1371: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:38.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:38 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2589718233' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-08-17T17:47:38.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:38 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-08-17T17:47:38.678 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 2 1 2024-08-17T17:47:38.949 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:47:39.368 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:39 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-08-17T17:47:39.368 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:39 smithi053 ceph-mon[89963]: osdmap e410: 8 total, 8 up, 8 in 2024-08-17T17:47:39.368 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:39 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:47:39.368 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:39 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-08-17T17:47:39.368 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:39 smithi003 ceph-mon[103344]: osdmap e410: 8 total, 8 up, 8 in 2024-08-17T17:47:39.368 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:39 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:47:39.369 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:39 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-08-17T17:47:39.369 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:39 smithi003 ceph-mon[100615]: osdmap e410: 8 total, 8 up, 8 in 2024-08-17T17:47:39.369 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:39 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:47:40.155 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:40 smithi003 ceph-mon[103344]: pgmap v1373: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:40.156 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:40 smithi003 ceph-mon[100615]: pgmap v1373: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:40.385 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:47:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:47:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:47:40.386 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:40 smithi053 ceph-mon[89963]: pgmap v1373: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:41.041 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.2 to 1 (10000) 2024-08-17T17:47:41.404 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:41 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3023314629' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:47:41.404 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:41 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:47:41.405 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:41 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3023314629' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:47:41.405 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:41 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:47:41.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:41 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3023314629' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:47:41.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:41 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:47:41.778 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 2 1 2024-08-17T17:47:42.050 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:47:42.292 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:42 smithi003 ceph-mon[103344]: pgmap v1374: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:42.292 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:42 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-17T17:47:42.292 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:42 smithi003 ceph-mon[103344]: osdmap e411: 8 total, 8 up, 8 in 2024-08-17T17:47:42.293 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:42 smithi003 ceph-mon[100615]: pgmap v1374: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:42.293 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:42 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-17T17:47:42.293 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:42 smithi003 ceph-mon[100615]: osdmap e411: 8 total, 8 up, 8 in 2024-08-17T17:47:42.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:42 smithi053 ceph-mon[89963]: pgmap v1374: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:42.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:42 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-17T17:47:42.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:42 smithi053 ceph-mon[89963]: osdmap e411: 8 total, 8 up, 8 in 2024-08-17T17:47:44.042 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-08-17T17:47:44.363 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:44 smithi003 ceph-mon[103344]: pgmap v1376: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:44.364 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:44 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3814123506' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:47:44.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:44 smithi003 ceph-mon[100615]: pgmap v1376: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:44.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:44 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3814123506' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:47:44.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:44 smithi053 ceph-mon[89963]: pgmap v1376: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:44.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:44 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3814123506' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-08-17T17:47:44.752 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 3 1 2024-08-17T17:47:45.025 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:47:45.400 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:45 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3814123506' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-08-17T17:47:45.400 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:45 smithi003 ceph-mon[103344]: osdmap e412: 8 total, 8 up, 8 in 2024-08-17T17:47:45.400 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:45 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3814123506' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-08-17T17:47:45.400 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:45 smithi003 ceph-mon[100615]: osdmap e412: 8 total, 8 up, 8 in 2024-08-17T17:47:45.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:45 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3814123506' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-08-17T17:47:45.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:45 smithi053 ceph-mon[89963]: osdmap e412: 8 total, 8 up, 8 in 2024-08-17T17:47:46.026 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:47:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:47:45] "GET /metrics HTTP/1.1" 200 34140 "" "Prometheus/2.51.0" 2024-08-17T17:47:46.384 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:46 smithi003 ceph-mon[103344]: pgmap v1378: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:46.384 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:46 smithi003 ceph-mon[103344]: osdmap e413: 8 total, 8 up, 8 in 2024-08-17T17:47:46.384 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:46 smithi003 ceph-mon[103344]: 1.18 deep-scrub starts 2024-08-17T17:47:46.384 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:46 smithi003 ceph-mon[103344]: 1.18 deep-scrub ok 2024-08-17T17:47:46.385 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:46 smithi003 ceph-mon[100615]: pgmap v1378: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:46.385 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:46 smithi003 ceph-mon[100615]: osdmap e413: 8 total, 8 up, 8 in 2024-08-17T17:47:46.385 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:46 smithi003 ceph-mon[100615]: 1.18 deep-scrub starts 2024-08-17T17:47:46.385 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:46 smithi003 ceph-mon[100615]: 1.18 deep-scrub ok 2024-08-17T17:47:46.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:46 smithi053 ceph-mon[89963]: pgmap v1378: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:46.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:46 smithi053 ceph-mon[89963]: osdmap e413: 8 total, 8 up, 8 in 2024-08-17T17:47:46.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:46 smithi053 ceph-mon[89963]: 1.18 deep-scrub starts 2024-08-17T17:47:46.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:46 smithi053 ceph-mon[89963]: 1.18 deep-scrub ok 2024-08-17T17:47:47.073 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.3 to 1 (10000) 2024-08-17T17:47:47.414 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:47 smithi003 ceph-mon[100615]: 1.a scrub starts 2024-08-17T17:47:47.414 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:47 smithi003 ceph-mon[100615]: 1.a scrub ok 2024-08-17T17:47:47.414 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:47 smithi003 ceph-mon[100615]: pgmap v1380: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:47.414 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:47 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3908980120' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:47:47.415 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:47 smithi003 ceph-mon[103344]: 1.a scrub starts 2024-08-17T17:47:47.415 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:47 smithi003 ceph-mon[103344]: 1.a scrub ok 2024-08-17T17:47:47.415 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:47 smithi003 ceph-mon[103344]: pgmap v1380: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:47.415 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:47 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3908980120' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:47:47.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:47 smithi053 ceph-mon[89963]: 1.a scrub starts 2024-08-17T17:47:47.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:47 smithi053 ceph-mon[89963]: 1.a scrub ok 2024-08-17T17:47:47.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:47 smithi053 ceph-mon[89963]: pgmap v1380: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:47.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:47 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3908980120' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:47:47.887 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 3 1 2024-08-17T17:47:48.160 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:47:48.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:48 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3908980120' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-17T17:47:48.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:48 smithi003 ceph-mon[100615]: osdmap e414: 8 total, 8 up, 8 in 2024-08-17T17:47:48.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:48 smithi003 ceph-mon[100615]: 1.8 scrub starts 2024-08-17T17:47:48.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:48 smithi003 ceph-mon[100615]: 1.8 scrub ok 2024-08-17T17:47:48.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:48 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3908980120' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-17T17:47:48.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:48 smithi003 ceph-mon[103344]: osdmap e414: 8 total, 8 up, 8 in 2024-08-17T17:47:48.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:48 smithi003 ceph-mon[103344]: 1.8 scrub starts 2024-08-17T17:47:48.274 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:48 smithi003 ceph-mon[103344]: 1.8 scrub ok 2024-08-17T17:47:48.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:48 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3908980120' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-08-17T17:47:48.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:48 smithi053 ceph-mon[89963]: osdmap e414: 8 total, 8 up, 8 in 2024-08-17T17:47:48.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:48 smithi053 ceph-mon[89963]: 1.8 scrub starts 2024-08-17T17:47:48.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:48 smithi053 ceph-mon[89963]: 1.8 scrub ok 2024-08-17T17:47:49.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:49 smithi003 ceph-mon[100615]: pgmap v1382: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:49.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:49 smithi003 ceph-mon[103344]: pgmap v1382: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:49.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:49 smithi053 ceph-mon[89963]: pgmap v1382: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:50.098 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-08-17T17:47:50.297 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:50 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3904579' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:47:50.297 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:50 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:47:50.297 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:50 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3904579' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:47:50.297 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:50 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:47:50.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:47:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:47:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:47:50.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:50 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3904579' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:47:50.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:50 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-08-17T17:47:50.790 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 4 1 2024-08-17T17:47:51.059 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:47:51.444 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:51 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-08-17T17:47:51.444 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:51 smithi003 ceph-mon[103344]: osdmap e415: 8 total, 8 up, 8 in 2024-08-17T17:47:51.444 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:51 smithi003 ceph-mon[103344]: pgmap v1384: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:51.444 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:51 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-08-17T17:47:51.444 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:51 smithi003 ceph-mon[100615]: osdmap e415: 8 total, 8 up, 8 in 2024-08-17T17:47:51.445 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:51 smithi003 ceph-mon[100615]: pgmap v1384: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:51.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:51 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-08-17T17:47:51.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:51 smithi053 ceph-mon[89963]: osdmap e415: 8 total, 8 up, 8 in 2024-08-17T17:47:51.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:51 smithi053 ceph-mon[89963]: pgmap v1384: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:53.544 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.4 to 1 (10000) 2024-08-17T17:47:53.912 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:53 smithi003 ceph-mon[103344]: pgmap v1385: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:53.913 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:53 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1636454877' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:47:53.913 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:53 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:47:53.913 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:47:53.913 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:53 smithi003 ceph-mon[100615]: pgmap v1385: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:53.913 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:53 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1636454877' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:47:53.913 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:53 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:47:53.913 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:47:53.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:53 smithi053 ceph-mon[89963]: pgmap v1385: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:53.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:53 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1636454877' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:47:53.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:53 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:47:53.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:47:54.314 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 4 1 2024-08-17T17:47:54.593 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:47:54.820 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:54 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-17T17:47:54.820 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:54 smithi003 ceph-mon[103344]: osdmap e416: 8 total, 8 up, 8 in 2024-08-17T17:47:54.820 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:54 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-17T17:47:54.821 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:54 smithi003 ceph-mon[100615]: osdmap e416: 8 total, 8 up, 8 in 2024-08-17T17:47:54.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:54 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-17T17:47:54.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:54 smithi053 ceph-mon[89963]: osdmap e416: 8 total, 8 up, 8 in 2024-08-17T17:47:55.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:55 smithi003 ceph-mon[103344]: pgmap v1387: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:55.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:55 smithi003 ceph-mon[100615]: pgmap v1387: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:55.755 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:47:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:47:55] "GET /metrics HTTP/1.1" 200 34138 "" "Prometheus/2.51.0" 2024-08-17T17:47:55.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:55 smithi053 ceph-mon[89963]: pgmap v1387: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:56.549 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-08-17T17:47:56.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:56 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3188842434' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:47:57.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:56 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3188842434' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:47:57.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:56 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3188842434' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-08-17T17:47:57.477 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 5 1 2024-08-17T17:47:57.752 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:47:57.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:57 smithi003 ceph-mon[103344]: pgmap v1388: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:57.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:57 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3188842434' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-08-17T17:47:57.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:57 smithi003 ceph-mon[103344]: osdmap e417: 8 total, 8 up, 8 in 2024-08-17T17:47:57.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:57 smithi003 ceph-mon[100615]: pgmap v1388: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:57.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:57 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3188842434' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-08-17T17:47:57.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:57 smithi003 ceph-mon[100615]: osdmap e417: 8 total, 8 up, 8 in 2024-08-17T17:47:57.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:57 smithi053 ceph-mon[89963]: pgmap v1388: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:47:57.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:57 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3188842434' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-08-17T17:47:57.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:57 smithi053 ceph-mon[89963]: osdmap e417: 8 total, 8 up, 8 in 2024-08-17T17:47:59.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:47:59 smithi053 ceph-mon[89963]: pgmap v1390: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:00.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:47:59 smithi003 ceph-mon[100615]: pgmap v1390: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:00.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:47:59 smithi003 ceph-mon[103344]: pgmap v1390: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:00.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:48:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:48:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:48:00.565 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.5 to 1 (10000) 2024-08-17T17:48:00.948 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:00 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3656398532' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:48:00.948 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:00 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:48:00.948 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:00 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3656398532' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:48:00.948 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:00 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:48:00.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:00 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3656398532' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:48:00.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:00 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:48:01.303 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 5 1 2024-08-17T17:48:01.578 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:48:01.833 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:01 smithi003 ceph-mon[103344]: pgmap v1391: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:01.833 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:01 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-17T17:48:01.833 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:01 smithi003 ceph-mon[103344]: osdmap e418: 8 total, 8 up, 8 in 2024-08-17T17:48:01.833 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:01 smithi003 ceph-mon[100615]: pgmap v1391: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:01.834 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:01 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-17T17:48:01.834 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:01 smithi003 ceph-mon[100615]: osdmap e418: 8 total, 8 up, 8 in 2024-08-17T17:48:01.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:01 smithi053 ceph-mon[89963]: pgmap v1391: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:01.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:01 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-17T17:48:01.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:01 smithi053 ceph-mon[89963]: osdmap e418: 8 total, 8 up, 8 in 2024-08-17T17:48:03.571 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-08-17T17:48:03.921 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:03 smithi003 ceph-mon[103344]: pgmap v1393: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:03.922 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:03 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/4219729984' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:48:03.922 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:03 smithi003 ceph-mon[100615]: pgmap v1393: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:03.922 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:03 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/4219729984' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:48:03.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:03 smithi053 ceph-mon[89963]: pgmap v1393: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:03.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:03 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/4219729984' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-08-17T17:48:04.252 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 6 1 2024-08-17T17:48:04.525 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:48:04.867 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:04 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/4219729984' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-08-17T17:48:04.867 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:04 smithi003 ceph-mon[103344]: osdmap e419: 8 total, 8 up, 8 in 2024-08-17T17:48:04.868 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:04 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/4219729984' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-08-17T17:48:04.868 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:04 smithi003 ceph-mon[100615]: osdmap e419: 8 total, 8 up, 8 in 2024-08-17T17:48:04.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:04 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/4219729984' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-08-17T17:48:04.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:04 smithi053 ceph-mon[89963]: osdmap e419: 8 total, 8 up, 8 in 2024-08-17T17:48:05.906 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:48:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:48:05] "GET /metrics HTTP/1.1" 200 34138 "" "Prometheus/2.51.0" 2024-08-17T17:48:05.906 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:05 smithi003 ceph-mon[103344]: pgmap v1395: 103 pgs: 19 peering, 84 active+clean; 4.5 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 14 KiB/s rd, 35 KiB/s wr, 11 op/s; 25 KiB/s, 5 objects/s recovering 2024-08-17T17:48:05.906 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:05 smithi003 ceph-mon[103344]: Health check failed: Reduced data availability: 13 pgs inactive, 18 pgs peering (PG_AVAILABILITY) 2024-08-17T17:48:05.906 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:05 smithi003 ceph-mon[103344]: osdmap e420: 8 total, 8 up, 8 in 2024-08-17T17:48:05.906 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:05 smithi003 ceph-mon[103344]: 1.1 scrub starts 2024-08-17T17:48:05.906 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:05 smithi003 ceph-mon[103344]: 1.1 scrub ok 2024-08-17T17:48:05.907 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:05 smithi003 ceph-mon[100615]: pgmap v1395: 103 pgs: 19 peering, 84 active+clean; 4.5 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 14 KiB/s rd, 35 KiB/s wr, 11 op/s; 25 KiB/s, 5 objects/s recovering 2024-08-17T17:48:05.907 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:05 smithi003 ceph-mon[100615]: Health check failed: Reduced data availability: 13 pgs inactive, 18 pgs peering (PG_AVAILABILITY) 2024-08-17T17:48:05.907 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:05 smithi003 ceph-mon[100615]: osdmap e420: 8 total, 8 up, 8 in 2024-08-17T17:48:05.907 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:05 smithi003 ceph-mon[100615]: 1.1 scrub starts 2024-08-17T17:48:05.907 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:05 smithi003 ceph-mon[100615]: 1.1 scrub ok 2024-08-17T17:48:05.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:05 smithi053 ceph-mon[89963]: pgmap v1395: 103 pgs: 19 peering, 84 active+clean; 4.5 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 14 KiB/s rd, 35 KiB/s wr, 11 op/s; 25 KiB/s, 5 objects/s recovering 2024-08-17T17:48:05.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:05 smithi053 ceph-mon[89963]: Health check failed: Reduced data availability: 13 pgs inactive, 18 pgs peering (PG_AVAILABILITY) 2024-08-17T17:48:05.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:05 smithi053 ceph-mon[89963]: osdmap e420: 8 total, 8 up, 8 in 2024-08-17T17:48:05.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:05 smithi053 ceph-mon[89963]: 1.1 scrub starts 2024-08-17T17:48:05.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:05 smithi053 ceph-mon[89963]: 1.1 scrub ok 2024-08-17T17:48:06.596 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.6 to 1 (10000) 2024-08-17T17:48:06.963 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:06 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/162193246' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:48:06.963 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:06 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:48:06.964 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:06 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/162193246' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:48:06.964 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:06 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:48:06.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:06 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/162193246' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:48:06.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:06 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:48:07.435 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 6 1 2024-08-17T17:48:07.710 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:48:07.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:07 smithi003 ceph-mon[103344]: pgmap v1397: 103 pgs: 19 peering, 84 active+clean; 4.5 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 19 KiB/s rd, 46 KiB/s wr, 14 op/s; 33 KiB/s, 7 objects/s recovering 2024-08-17T17:48:07.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:07 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-17T17:48:07.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:07 smithi003 ceph-mon[103344]: osdmap e421: 8 total, 8 up, 8 in 2024-08-17T17:48:07.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:07 smithi003 ceph-mon[100615]: pgmap v1397: 103 pgs: 19 peering, 84 active+clean; 4.5 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 19 KiB/s rd, 46 KiB/s wr, 14 op/s; 33 KiB/s, 7 objects/s recovering 2024-08-17T17:48:07.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:07 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-17T17:48:07.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:07 smithi003 ceph-mon[100615]: osdmap e421: 8 total, 8 up, 8 in 2024-08-17T17:48:07.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:07 smithi053 ceph-mon[89963]: pgmap v1397: 103 pgs: 19 peering, 84 active+clean; 4.5 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 19 KiB/s rd, 46 KiB/s wr, 14 op/s; 33 KiB/s, 7 objects/s recovering 2024-08-17T17:48:07.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:07 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-17T17:48:07.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:07 smithi053 ceph-mon[89963]: osdmap e421: 8 total, 8 up, 8 in 2024-08-17T17:48:09.389 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:09 smithi003 ceph-mon[103344]: 1.a scrub starts 2024-08-17T17:48:09.389 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:09 smithi003 ceph-mon[103344]: 1.a scrub ok 2024-08-17T17:48:09.389 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:48:09.389 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:09 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:48:09.389 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:09 smithi003 ceph-mon[103344]: pgmap v1399: 103 pgs: 19 peering, 84 active+clean; 4.5 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 19 KiB/s rd, 46 KiB/s wr, 14 op/s; 33 KiB/s, 7 objects/s recovering 2024-08-17T17:48:09.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:09 smithi003 ceph-mon[100615]: 1.a scrub starts 2024-08-17T17:48:09.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:09 smithi003 ceph-mon[100615]: 1.a scrub ok 2024-08-17T17:48:09.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:48:09.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:09 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:48:09.390 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:09 smithi003 ceph-mon[100615]: pgmap v1399: 103 pgs: 19 peering, 84 active+clean; 4.5 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 19 KiB/s rd, 46 KiB/s wr, 14 op/s; 33 KiB/s, 7 objects/s recovering 2024-08-17T17:48:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:09 smithi053 ceph-mon[89963]: 1.a scrub starts 2024-08-17T17:48:09.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:09 smithi053 ceph-mon[89963]: 1.a scrub ok 2024-08-17T17:48:09.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:48:09.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:09 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:48:09.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:09 smithi053 ceph-mon[89963]: pgmap v1399: 103 pgs: 19 peering, 84 active+clean; 4.5 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 19 KiB/s rd, 46 KiB/s wr, 14 op/s; 33 KiB/s, 7 objects/s recovering 2024-08-17T17:48:10.140 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-08-17T17:48:10.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:10 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3280433524' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:48:10.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:10 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:48:10.447 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:10 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3280433524' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:48:10.447 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:10 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:48:10.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:48:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:48:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:48:10.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:10 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3280433524' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:48:10.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:10 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-08-17T17:48:10.746 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 7 1 2024-08-17T17:48:11.021 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:48:11.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:11 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-08-17T17:48:11.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:11 smithi053 ceph-mon[89963]: osdmap e422: 8 total, 8 up, 8 in 2024-08-17T17:48:11.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:11 smithi053 ceph-mon[89963]: pgmap v1401: 103 pgs: 103 active+clean; 4.5 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:11.467 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:11 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-08-17T17:48:11.467 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:11 smithi003 ceph-mon[103344]: osdmap e422: 8 total, 8 up, 8 in 2024-08-17T17:48:11.467 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:11 smithi003 ceph-mon[103344]: pgmap v1401: 103 pgs: 103 active+clean; 4.5 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:11.467 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:11 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-08-17T17:48:11.467 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:11 smithi003 ceph-mon[100615]: osdmap e422: 8 total, 8 up, 8 in 2024-08-17T17:48:11.467 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:11 smithi003 ceph-mon[100615]: pgmap v1401: 103 pgs: 103 active+clean; 4.5 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:12.401 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:12 smithi003 ceph-mon[103344]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 18 pgs peering) 2024-08-17T17:48:12.401 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:12 smithi003 ceph-mon[103344]: Cluster is now healthy 2024-08-17T17:48:12.401 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:12 smithi003 ceph-mon[100615]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 18 pgs peering) 2024-08-17T17:48:12.402 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:12 smithi003 ceph-mon[100615]: Cluster is now healthy 2024-08-17T17:48:12.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:12 smithi053 ceph-mon[89963]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 18 pgs peering) 2024-08-17T17:48:12.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:12 smithi053 ceph-mon[89963]: Cluster is now healthy 2024-08-17T17:48:13.158 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.7 to 1 (10000) 2024-08-17T17:48:13.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:13 smithi053 ceph-mon[89963]: pgmap v1402: 103 pgs: 103 active+clean; 4.5 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:13.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:13 smithi053 ceph-mon[89963]: 1.17 scrub starts 2024-08-17T17:48:13.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:13 smithi053 ceph-mon[89963]: 1.17 scrub ok 2024-08-17T17:48:13.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:13 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/496748046' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:48:13.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:13 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:48:13.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:13 smithi003 ceph-mon[100615]: pgmap v1402: 103 pgs: 103 active+clean; 4.5 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:13.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:13 smithi003 ceph-mon[100615]: 1.17 scrub starts 2024-08-17T17:48:13.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:13 smithi003 ceph-mon[100615]: 1.17 scrub ok 2024-08-17T17:48:13.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:13 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/496748046' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:48:13.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:13 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:48:13.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:13 smithi003 ceph-mon[103344]: pgmap v1402: 103 pgs: 103 active+clean; 4.5 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:13.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:13 smithi003 ceph-mon[103344]: 1.17 scrub starts 2024-08-17T17:48:13.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:13 smithi003 ceph-mon[103344]: 1.17 scrub ok 2024-08-17T17:48:13.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:13 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/496748046' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:48:13.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:13 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:48:13.960 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 7 1 2024-08-17T17:48:14.237 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:48:14.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:14 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-17T17:48:14.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:14 smithi003 ceph-mon[103344]: osdmap e423: 8 total, 8 up, 8 in 2024-08-17T17:48:14.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:14 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-17T17:48:14.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:14 smithi003 ceph-mon[100615]: osdmap e423: 8 total, 8 up, 8 in 2024-08-17T17:48:14.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:14 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-17T17:48:14.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:14 smithi053 ceph-mon[89963]: osdmap e423: 8 total, 8 up, 8 in 2024-08-17T17:48:15.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:15 smithi053 ceph-mon[89963]: pgmap v1404: 103 pgs: 103 active+clean; 4.5 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:15.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:15 smithi053 ceph-mon[89963]: 1.10 scrub starts 2024-08-17T17:48:15.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:15 smithi053 ceph-mon[89963]: 1.10 scrub ok 2024-08-17T17:48:15.480 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:15 smithi003 ceph-mon[103344]: pgmap v1404: 103 pgs: 103 active+clean; 4.5 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:15.480 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:15 smithi003 ceph-mon[103344]: 1.10 scrub starts 2024-08-17T17:48:15.480 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:15 smithi003 ceph-mon[103344]: 1.10 scrub ok 2024-08-17T17:48:15.481 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:15 smithi003 ceph-mon[100615]: pgmap v1404: 103 pgs: 103 active+clean; 4.5 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:15.481 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:15 smithi003 ceph-mon[100615]: 1.10 scrub starts 2024-08-17T17:48:15.481 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:15 smithi003 ceph-mon[100615]: 1.10 scrub ok 2024-08-17T17:48:15.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:48:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:48:15] "GET /metrics HTTP/1.1" 200 34160 "" "Prometheus/2.51.0" 2024-08-17T17:48:16.162 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:16 smithi003 ceph-mon[100615]: 1.18 scrub starts 2024-08-17T17:48:16.162 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:16 smithi003 ceph-mon[100615]: 1.18 scrub ok 2024-08-17T17:48:16.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:16 smithi053 ceph-mon[89963]: 1.18 scrub starts 2024-08-17T17:48:16.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:16 smithi053 ceph-mon[89963]: 1.18 scrub ok 2024-08-17T17:48:16.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:16 smithi003 ceph-mon[103344]: 1.18 scrub starts 2024-08-17T17:48:16.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:16 smithi003 ceph-mon[103344]: 1.18 scrub ok 2024-08-17T17:48:17.174 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-08-17T17:48:17.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:17 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2383957731' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:48:17.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:17 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:48:17.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:17 smithi053 ceph-mon[89963]: pgmap v1405: 103 pgs: 103 active+clean; 4.5 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:17.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:17 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2383957731' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:48:17.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:17 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:48:17.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:17 smithi003 ceph-mon[100615]: pgmap v1405: 103 pgs: 103 active+clean; 4.5 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:17.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:17 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2383957731' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:48:17.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:17 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-08-17T17:48:17.523 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:17 smithi003 ceph-mon[103344]: pgmap v1405: 103 pgs: 103 active+clean; 4.5 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:18.021 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd reweight 1 1 2024-08-17T17:48:18.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:18 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-08-17T17:48:18.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:18 smithi003 ceph-mon[103344]: osdmap e424: 8 total, 8 up, 8 in 2024-08-17T17:48:18.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:18 smithi003 ceph-mon[103344]: 1.1b scrub starts 2024-08-17T17:48:18.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:18 smithi003 ceph-mon[103344]: 1.1b scrub ok 2024-08-17T17:48:18.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:18 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-08-17T17:48:18.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:18 smithi003 ceph-mon[100615]: osdmap e424: 8 total, 8 up, 8 in 2024-08-17T17:48:18.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:18 smithi003 ceph-mon[100615]: 1.1b scrub starts 2024-08-17T17:48:18.274 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:18 smithi003 ceph-mon[100615]: 1.1b scrub ok 2024-08-17T17:48:18.303 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:48:18.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:18 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-08-17T17:48:18.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:18 smithi053 ceph-mon[89963]: osdmap e424: 8 total, 8 up, 8 in 2024-08-17T17:48:18.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:18 smithi053 ceph-mon[89963]: 1.1b scrub starts 2024-08-17T17:48:18.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:18 smithi053 ceph-mon[89963]: 1.1b scrub ok 2024-08-17T17:48:19.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:19 smithi053 ceph-mon[89963]: osdmap e425: 8 total, 8 up, 8 in 2024-08-17T17:48:19.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:19 smithi053 ceph-mon[89963]: pgmap v1408: 103 pgs: 103 active+clean; 4.5 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:19.488 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:19 smithi003 ceph-mon[103344]: osdmap e425: 8 total, 8 up, 8 in 2024-08-17T17:48:19.488 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:19 smithi003 ceph-mon[103344]: pgmap v1408: 103 pgs: 103 active+clean; 4.5 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:19.488 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:19 smithi003 ceph-mon[100615]: osdmap e425: 8 total, 8 up, 8 in 2024-08-17T17:48:19.488 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:19 smithi003 ceph-mon[100615]: pgmap v1408: 103 pgs: 103 active+clean; 4.5 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:20.184 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.1 to 1 (10000) 2024-08-17T17:48:20.461 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:20 smithi003 ceph-mon[103344]: 1.2 scrub starts 2024-08-17T17:48:20.461 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:20 smithi003 ceph-mon[103344]: 1.2 scrub ok 2024-08-17T17:48:20.461 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:20 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2588208279' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:48:20.461 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:20 smithi003 ceph-mon[100615]: 1.2 scrub starts 2024-08-17T17:48:20.461 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:20 smithi003 ceph-mon[100615]: 1.2 scrub ok 2024-08-17T17:48:20.462 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:20 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2588208279' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:48:20.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:20 smithi053 ceph-mon[89963]: 1.2 scrub starts 2024-08-17T17:48:20.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:20 smithi053 ceph-mon[89963]: 1.2 scrub ok 2024-08-17T17:48:20.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:20 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2588208279' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:48:20.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:48:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:48:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:48:20.789 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd primary-affinity 1 1 2024-08-17T17:48:21.067 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:48:21.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:21 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2588208279' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-17T17:48:21.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:21 smithi053 ceph-mon[89963]: osdmap e426: 8 total, 8 up, 8 in 2024-08-17T17:48:21.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:21 smithi053 ceph-mon[89963]: pgmap v1410: 103 pgs: 103 active+clean; 4.8 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 12 KiB/s rd, 36 KiB/s wr, 8 op/s; 65 KiB/s, 0 objects/s recovering 2024-08-17T17:48:21.474 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:21 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2588208279' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-17T17:48:21.474 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:21 smithi003 ceph-mon[100615]: osdmap e426: 8 total, 8 up, 8 in 2024-08-17T17:48:21.474 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:21 smithi003 ceph-mon[100615]: pgmap v1410: 103 pgs: 103 active+clean; 4.8 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 12 KiB/s rd, 36 KiB/s wr, 8 op/s; 65 KiB/s, 0 objects/s recovering 2024-08-17T17:48:21.474 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:21 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2588208279' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-17T17:48:21.475 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:21 smithi003 ceph-mon[103344]: osdmap e426: 8 total, 8 up, 8 in 2024-08-17T17:48:21.475 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:21 smithi003 ceph-mon[103344]: pgmap v1410: 103 pgs: 103 active+clean; 4.8 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 12 KiB/s rd, 36 KiB/s wr, 8 op/s; 65 KiB/s, 0 objects/s recovering 2024-08-17T17:48:23.529 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-08-17T17:48:23.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:23 smithi003 ceph-mon[103344]: pgmap v1411: 103 pgs: 103 active+clean; 4.8 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 12 KiB/s rd, 36 KiB/s wr, 8 op/s; 65 KiB/s, 0 objects/s recovering 2024-08-17T17:48:23.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:23 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2058597792' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:48:23.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:23 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:48:23.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:48:23.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:23 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:48:23.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:23 smithi003 ceph-mon[100615]: pgmap v1411: 103 pgs: 103 active+clean; 4.8 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 12 KiB/s rd, 36 KiB/s wr, 8 op/s; 65 KiB/s, 0 objects/s recovering 2024-08-17T17:48:23.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:23 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2058597792' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:48:23.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:23 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:48:23.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:48:23.774 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:23 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:48:23.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:23 smithi053 ceph-mon[89963]: pgmap v1411: 103 pgs: 103 active+clean; 4.8 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 12 KiB/s rd, 36 KiB/s wr, 8 op/s; 65 KiB/s, 0 objects/s recovering 2024-08-17T17:48:23.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:23 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2058597792' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:48:23.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:23 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-08-17T17:48:23.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:48:23.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:23 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:48:24.202 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-08-17T17:48:24.202 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-08-17T17:48:24.202 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-08-17T17:48:24.203 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-08-17T17:48:24.203 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-08-17T17:48:24.203 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-17T17:48:24.479 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:48:24.865 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:24 smithi003 ceph-mon[103344]: 1.0 scrub starts 2024-08-17T17:48:24.866 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:24 smithi003 ceph-mon[103344]: 1.0 scrub ok 2024-08-17T17:48:24.866 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:24 smithi003 ceph-mon[103344]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-08-17T17:48:24.866 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:24 smithi003 ceph-mon[103344]: osdmap e427: 8 total, 8 up, 8 in 2024-08-17T17:48:24.866 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:24 smithi003 ceph-mon[100615]: 1.0 scrub starts 2024-08-17T17:48:24.866 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:24 smithi003 ceph-mon[100615]: 1.0 scrub ok 2024-08-17T17:48:24.866 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:24 smithi003 ceph-mon[100615]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-08-17T17:48:24.866 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:24 smithi003 ceph-mon[100615]: osdmap e427: 8 total, 8 up, 8 in 2024-08-17T17:48:24.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:24 smithi053 ceph-mon[89963]: 1.0 scrub starts 2024-08-17T17:48:24.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:24 smithi053 ceph-mon[89963]: 1.0 scrub ok 2024-08-17T17:48:24.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:24 smithi053 ceph-mon[89963]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-08-17T17:48:24.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:24 smithi053 ceph-mon[89963]: osdmap e427: 8 total, 8 up, 8 in 2024-08-17T17:48:25.651 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:25 smithi003 ceph-mon[100615]: pgmap v1413: 103 pgs: 103 active+clean; 4.8 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 12 KiB/s, 0 objects/s recovering 2024-08-17T17:48:25.651 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:25 smithi003 ceph-mon[100615]: 1.1c scrub starts 2024-08-17T17:48:25.651 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:25 smithi003 ceph-mon[100615]: 1.1c scrub ok 2024-08-17T17:48:25.651 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:25 smithi003 ceph-mon[103344]: pgmap v1413: 103 pgs: 103 active+clean; 4.8 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 12 KiB/s, 0 objects/s recovering 2024-08-17T17:48:25.651 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:25 smithi003 ceph-mon[103344]: 1.1c scrub starts 2024-08-17T17:48:25.652 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:25 smithi003 ceph-mon[103344]: 1.1c scrub ok 2024-08-17T17:48:25.959 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:48:25 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:48:25] "GET /metrics HTTP/1.1" 200 34159 "" "Prometheus/2.51.0" 2024-08-17T17:48:25.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:25 smithi053 ceph-mon[89963]: pgmap v1413: 103 pgs: 103 active+clean; 4.8 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 12 KiB/s, 0 objects/s recovering 2024-08-17T17:48:25.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:25 smithi053 ceph-mon[89963]: 1.1c scrub starts 2024-08-17T17:48:25.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:25 smithi053 ceph-mon[89963]: 1.1c scrub ok 2024-08-17T17:48:26.361 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:48:26.361 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":427,"fsid":"2b5b662e-5cba-11ef-bccf-c7b262605968","created":"2024-08-17T17:00:56.372943+0000","modified":"2024-08-17T17:48:23.519566+0000","last_up_change":"2024-08-17T17:33:19.746015+0000","last_in_change":"2024-08-17T17:41:33.591795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-17T17:04:01.140951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"408","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"340","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.059999942779541,"score_stable":2.059999942779541,"optimal_score":1,"raw_score_acting":2.059999942779541,"raw_score_stable":2.059999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-17T17:06:28.452624+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":85,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-08-17T17:07:44.635021+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-08-17T17:07:45.576097+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f9a7d76-f0ec-49fc-9540-2606e94758ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":90,"up_from":94,"up_thru":399,"down_at":91,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":58175680},{"type":"v1","addr":"172.21.15.3:6803","nonce":58175680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":58175680},{"type":"v1","addr":"172.21.15.3:6805","nonce":58175680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":58175680},{"type":"v1","addr":"172.21.15.3:6809","nonce":58175680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":58175680},{"type":"v1","addr":"172.21.15.3:6807","nonce":58175680}]},"public_addr":"172.21.15.3:6803/58175680","cluster_addr":"172.21.15.3:6805/58175680","heartbeat_back_addr":"172.21.15.3:6809/58175680","heartbeat_front_addr":"172.21.15.3:6807/58175680","state":["exists","up"]},{"osd":1,"uuid":"316b5c4e-3417-4b93-bbbb-c6ed6dd21b0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":99,"last_clean_end":248,"up_from":268,"up_thru":399,"down_at":249,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":5043585},{"type":"v1","addr":"172.21.15.3:6811","nonce":5043585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":5043585},{"type":"v1","addr":"172.21.15.3:6813","nonce":5043585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":5043585},{"type":"v1","addr":"172.21.15.3:6817","nonce":5043585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":5043585},{"type":"v1","addr":"172.21.15.3:6815","nonce":5043585}]},"public_addr":"172.21.15.3:6811/5043585","cluster_addr":"172.21.15.3:6813/5043585","heartbeat_back_addr":"172.21.15.3:6817/5043585","heartbeat_front_addr":"172.21.15.3:6815/5043585","state":["exists","up"]},{"osd":2,"uuid":"88af697d-38db-4c51-bfff-d0a776315596","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":100,"up_from":104,"up_thru":412,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1825512279},{"type":"v1","addr":"172.21.15.3:6819","nonce":1825512279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1825512279},{"type":"v1","addr":"172.21.15.3:6821","nonce":1825512279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1825512279},{"type":"v1","addr":"172.21.15.3:6825","nonce":1825512279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1825512279},{"type":"v1","addr":"172.21.15.3:6823","nonce":1825512279}]},"public_addr":"172.21.15.3:6819/1825512279","cluster_addr":"172.21.15.3:6821/1825512279","heartbeat_back_addr":"172.21.15.3:6825/1825512279","heartbeat_front_addr":"172.21.15.3:6823/1825512279","state":["exists","up"]},{"osd":3,"uuid":"8e20cf42-6a13-4ad5-9f01-203a30c692d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":105,"up_from":109,"up_thru":399,"down_at":106,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":1366347928},{"type":"v1","addr":"172.21.15.3:6827","nonce":1366347928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":1366347928},{"type":"v1","addr":"172.21.15.3:6829","nonce":1366347928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":1366347928},{"type":"v1","addr":"172.21.15.3:6833","nonce":1366347928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":1366347928},{"type":"v1","addr":"172.21.15.3:6831","nonce":1366347928}]},"public_addr":"172.21.15.3:6827/1366347928","cluster_addr":"172.21.15.3:6829/1366347928","heartbeat_back_addr":"172.21.15.3:6833/1366347928","heartbeat_front_addr":"172.21.15.3:6831/1366347928","state":["exists","up"]},{"osd":4,"uuid":"2e8eabd7-8258-4b78-bb21-1a7291e0f1d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":399,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6801","nonce":1387588062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6803","nonce":1387588062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6807","nonce":1387588062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6805","nonce":1387588062}]},"public_addr":"172.21.15.53:6801/1387588062","cluster_addr":"172.21.15.53:6803/1387588062","heartbeat_back_addr":"172.21.15.53:6807/1387588062","heartbeat_front_addr":"172.21.15.53:6805/1387588062","state":["exists","up"]},{"osd":5,"uuid":"738f10f7-ff0d-4392-a818-eb23c9f65a63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":419,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6809","nonce":1263400356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6811","nonce":1263400356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6815","nonce":1263400356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6813","nonce":1263400356}]},"public_addr":"172.21.15.53:6809/1263400356","cluster_addr":"172.21.15.53:6811/1263400356","heartbeat_back_addr":"172.21.15.53:6815/1263400356","heartbeat_front_addr":"172.21.15.53:6813/1263400356","state":["exists","up"]},{"osd":6,"uuid":"62903bee-59eb-4eed-b64d-9c884de87409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":399,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6817","nonce":3729099668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6819","nonce":3729099668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6823","nonce":3729099668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6821","nonce":3729099668}]},"public_addr":"172.21.15.53:6817/3729099668","cluster_addr":"172.21.15.53:6819/3729099668","heartbeat_back_addr":"172.21.15.53:6823/3729099668","heartbeat_front_addr":"172.21.15.53:6821/3729099668","state":["exists","up"]},{"osd":7,"uuid":"4794ff08-e5d9-49c9-bea5-e158681b4004","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":424,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6825","nonce":832291503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6827","nonce":832291503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6831","nonce":832291503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6829","nonce":832291503}]},"public_addr":"172.21.15.53:6825/832291503","cluster_addr":"172.21.15.53:6827/832291503","heartbeat_back_addr":"172.21.15.53:6831/832291503","heartbeat_front_addr":"172.21.15.53:6829/832291503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:03:10.759301+0000","dead_epoch":90},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:03:33.981212+0000","dead_epoch":248},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:03:58.516966+0000","dead_epoch":100},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:04:22.965963+0000","dead_epoch":105},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:04:43.836760+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:06.691915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:29.472293+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:52.166848+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.a","mappings":[{"from":6,"to":2}]},{"pgid":"3.1c","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.d","mappings":[{"from":1,"to":7}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2},{"from":3,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:0/2555682812":"2024-08-18T17:11:56.179835+0000","172.21.15.3:6800/2187177466":"2024-08-18T17:01:50.631242+0000","172.21.15.3:0/3985128953":"2024-08-18T17:01:50.631242+0000","172.21.15.3:0/2723667418":"2024-08-18T17:12:37.591006+0000","172.21.15.53:0/1967774260":"2024-08-18T17:11:56.179835+0000","172.21.15.3:6801/2210133652":"2024-08-18T17:12:37.591006+0000","172.21.15.3:0/1476294447":"2024-08-18T17:01:27.442443+0000","172.21.15.3:6800/2210133652":"2024-08-18T17:12:37.591006+0000","172.21.15.3:0/3946755108":"2024-08-18T17:06:25.750593+0000","172.21.15.3:0/4087675427":"2024-08-18T17:12:37.591006+0000","172.21.15.3:6800/1736600360":"2024-08-18T17:06:25.750593+0000","172.21.15.3:0/3953105554":"2024-08-18T17:06:25.750593+0000","172.21.15.3:0/2847539733":"2024-08-18T17:12:37.591006+0000","172.21.15.3:0/3116588554":"2024-08-18T17:06:25.750593+0000","172.21.15.53:0/721818304":"2024-08-18T17:11:56.179835+0000","172.21.15.3:6801/160535497":"2024-08-18T17:01:27.442443+0000","172.21.15.3:0/3608486402":"2024-08-18T17:12:37.591006+0000","172.21.15.3:0/1668133120":"2024-08-18T17:01:27.442443+0000","172.21.15.3:6801/1736600360":"2024-08-18T17:06:25.750593+0000","172.21.15.3:0/880444026":"2024-08-18T17:10:21.996108+0000","172.21.15.3:0/1999667208":"2024-08-18T17:01:50.631242+0000","172.21.15.3:6800/2670911286":"2024-08-18T17:10:21.996108+0000","172.21.15.3:0/601716019":"2024-08-18T17:10:21.996108+0000","172.21.15.3:0/143641995":"2024-08-18T17:12:37.591006+0000","172.21.15.3:6801/2187177466":"2024-08-18T17:01:50.631242+0000","172.21.15.53:6834/4265580715":"2024-08-18T17:11:56.179835+0000","172.21.15.3:0/3817753887":"2024-08-18T17:01:50.631242+0000","172.21.15.53:0/1330698653":"2024-08-18T17:11:56.179835+0000","172.21.15.3:6801/2670911286":"2024-08-18T17:10:21.996108+0000","172.21.15.3:0/2965813587":"2024-08-18T17:12:37.591006+0000","172.21.15.3:0/1583211207":"2024-08-18T17:06:25.750593+0000","172.21.15.3:0/3717779298":"2024-08-18T17:10:21.996108+0000","172.21.15.3:0/4194950757":"2024-08-18T17:01:27.442443+0000","172.21.15.3:6800/160535497":"2024-08-18T17:01:27.442443+0000","172.21.15.53:6835/4265580715":"2024-08-18T17:11:56.179835+0000","172.21.15.3:0/2029382827":"2024-08-18T17:10:21.996108+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":11,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-17T17:48:26.963 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:26 smithi003 ceph-mon[103344]: 1.9 scrub starts 2024-08-17T17:48:26.963 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:26 smithi003 ceph-mon[103344]: 1.9 scrub ok 2024-08-17T17:48:26.963 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:26 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1852502414' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-17T17:48:26.963 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:26 smithi003 ceph-mon[100615]: 1.9 scrub starts 2024-08-17T17:48:26.963 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:26 smithi003 ceph-mon[100615]: 1.9 scrub ok 2024-08-17T17:48:26.963 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:26 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1852502414' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-17T17:48:26.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:26 smithi053 ceph-mon[89963]: 1.9 scrub starts 2024-08-17T17:48:26.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:26 smithi053 ceph-mon[89963]: 1.9 scrub ok 2024-08-17T17:48:26.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:26 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1852502414' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-17T17:48:27.063 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-08-17T17:48:27.064 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd dump --format=json 2024-08-17T17:48:27.333 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:48:27.731 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:27 smithi003 ceph-mon[103344]: pgmap v1414: 103 pgs: 103 active+clean; 4.8 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 9.8 KiB/s, 0 objects/s recovering 2024-08-17T17:48:27.732 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:27 smithi003 ceph-mon[100615]: pgmap v1414: 103 pgs: 103 active+clean; 4.8 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 9.8 KiB/s, 0 objects/s recovering 2024-08-17T17:48:27.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:27 smithi053 ceph-mon[89963]: pgmap v1414: 103 pgs: 103 active+clean; 4.8 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 9.8 KiB/s, 0 objects/s recovering 2024-08-17T17:48:28.809 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:28 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:48:28.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:28 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:48:28.897 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:28 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-17T17:48:29.382 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:48:29.383 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":427,"fsid":"2b5b662e-5cba-11ef-bccf-c7b262605968","created":"2024-08-17T17:00:56.372943+0000","modified":"2024-08-17T17:48:23.519566+0000","last_up_change":"2024-08-17T17:33:19.746015+0000","last_in_change":"2024-08-17T17:41:33.591795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-17T17:04:01.140951+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"408","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"340","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.059999942779541,"score_stable":2.059999942779541,"optimal_score":1,"raw_score_acting":2.059999942779541,"raw_score_stable":2.059999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-17T17:06:28.452624+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":85,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-08-17T17:07:44.635021+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-08-17T17:07:45.576097+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7f9a7d76-f0ec-49fc-9540-2606e94758ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":90,"up_from":94,"up_thru":399,"down_at":91,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":58175680},{"type":"v1","addr":"172.21.15.3:6803","nonce":58175680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":58175680},{"type":"v1","addr":"172.21.15.3:6805","nonce":58175680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":58175680},{"type":"v1","addr":"172.21.15.3:6809","nonce":58175680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":58175680},{"type":"v1","addr":"172.21.15.3:6807","nonce":58175680}]},"public_addr":"172.21.15.3:6803/58175680","cluster_addr":"172.21.15.3:6805/58175680","heartbeat_back_addr":"172.21.15.3:6809/58175680","heartbeat_front_addr":"172.21.15.3:6807/58175680","state":["exists","up"]},{"osd":1,"uuid":"316b5c4e-3417-4b93-bbbb-c6ed6dd21b0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":99,"last_clean_end":248,"up_from":268,"up_thru":399,"down_at":249,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":5043585},{"type":"v1","addr":"172.21.15.3:6811","nonce":5043585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":5043585},{"type":"v1","addr":"172.21.15.3:6813","nonce":5043585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":5043585},{"type":"v1","addr":"172.21.15.3:6817","nonce":5043585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":5043585},{"type":"v1","addr":"172.21.15.3:6815","nonce":5043585}]},"public_addr":"172.21.15.3:6811/5043585","cluster_addr":"172.21.15.3:6813/5043585","heartbeat_back_addr":"172.21.15.3:6817/5043585","heartbeat_front_addr":"172.21.15.3:6815/5043585","state":["exists","up"]},{"osd":2,"uuid":"88af697d-38db-4c51-bfff-d0a776315596","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":100,"up_from":104,"up_thru":412,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1825512279},{"type":"v1","addr":"172.21.15.3:6819","nonce":1825512279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1825512279},{"type":"v1","addr":"172.21.15.3:6821","nonce":1825512279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1825512279},{"type":"v1","addr":"172.21.15.3:6825","nonce":1825512279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1825512279},{"type":"v1","addr":"172.21.15.3:6823","nonce":1825512279}]},"public_addr":"172.21.15.3:6819/1825512279","cluster_addr":"172.21.15.3:6821/1825512279","heartbeat_back_addr":"172.21.15.3:6825/1825512279","heartbeat_front_addr":"172.21.15.3:6823/1825512279","state":["exists","up"]},{"osd":3,"uuid":"8e20cf42-6a13-4ad5-9f01-203a30c692d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":105,"up_from":109,"up_thru":399,"down_at":106,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":1366347928},{"type":"v1","addr":"172.21.15.3:6827","nonce":1366347928}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":1366347928},{"type":"v1","addr":"172.21.15.3:6829","nonce":1366347928}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":1366347928},{"type":"v1","addr":"172.21.15.3:6833","nonce":1366347928}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":1366347928},{"type":"v1","addr":"172.21.15.3:6831","nonce":1366347928}]},"public_addr":"172.21.15.3:6827/1366347928","cluster_addr":"172.21.15.3:6829/1366347928","heartbeat_back_addr":"172.21.15.3:6833/1366347928","heartbeat_front_addr":"172.21.15.3:6831/1366347928","state":["exists","up"]},{"osd":4,"uuid":"2e8eabd7-8258-4b78-bb21-1a7291e0f1d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":399,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6801","nonce":1387588062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6803","nonce":1387588062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6807","nonce":1387588062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":1387588062},{"type":"v1","addr":"172.21.15.53:6805","nonce":1387588062}]},"public_addr":"172.21.15.53:6801/1387588062","cluster_addr":"172.21.15.53:6803/1387588062","heartbeat_back_addr":"172.21.15.53:6807/1387588062","heartbeat_front_addr":"172.21.15.53:6805/1387588062","state":["exists","up"]},{"osd":5,"uuid":"738f10f7-ff0d-4392-a818-eb23c9f65a63","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":419,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6809","nonce":1263400356}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6811","nonce":1263400356}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6815","nonce":1263400356}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":1263400356},{"type":"v1","addr":"172.21.15.53:6813","nonce":1263400356}]},"public_addr":"172.21.15.53:6809/1263400356","cluster_addr":"172.21.15.53:6811/1263400356","heartbeat_back_addr":"172.21.15.53:6815/1263400356","heartbeat_front_addr":"172.21.15.53:6813/1263400356","state":["exists","up"]},{"osd":6,"uuid":"62903bee-59eb-4eed-b64d-9c884de87409","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":399,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6817","nonce":3729099668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6819","nonce":3729099668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6823","nonce":3729099668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3729099668},{"type":"v1","addr":"172.21.15.53:6821","nonce":3729099668}]},"public_addr":"172.21.15.53:6817/3729099668","cluster_addr":"172.21.15.53:6819/3729099668","heartbeat_back_addr":"172.21.15.53:6823/3729099668","heartbeat_front_addr":"172.21.15.53:6821/3729099668","state":["exists","up"]},{"osd":7,"uuid":"4794ff08-e5d9-49c9-bea5-e158681b4004","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":424,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6825","nonce":832291503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6827","nonce":832291503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6831","nonce":832291503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":832291503},{"type":"v1","addr":"172.21.15.53:6829","nonce":832291503}]},"public_addr":"172.21.15.53:6825/832291503","cluster_addr":"172.21.15.53:6827/832291503","heartbeat_back_addr":"172.21.15.53:6831/832291503","heartbeat_front_addr":"172.21.15.53:6829/832291503","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:03:10.759301+0000","dead_epoch":90},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:03:33.981212+0000","dead_epoch":248},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:03:58.516966+0000","dead_epoch":100},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:04:22.965963+0000","dead_epoch":105},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:04:43.836760+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:06.691915+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:29.472293+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-08-17T17:05:52.166848+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.a","mappings":[{"from":6,"to":2}]},{"pgid":"3.1c","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.d","mappings":[{"from":1,"to":7}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2},{"from":3,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:0/2723667418":"2024-08-18T17:12:37.591006+0000","172.21.15.3:6801/2210133652":"2024-08-18T17:12:37.591006+0000","172.21.15.3:6800/2210133652":"2024-08-18T17:12:37.591006+0000","172.21.15.3:0/2965813587":"2024-08-18T17:12:37.591006+0000","172.21.15.3:0/4087675427":"2024-08-18T17:12:37.591006+0000","172.21.15.3:0/2847539733":"2024-08-18T17:12:37.591006+0000","172.21.15.53:0/1967774260":"2024-08-18T17:11:56.179835+0000","172.21.15.3:0/3985128953":"2024-08-18T17:01:50.631242+0000","172.21.15.3:0/1476294447":"2024-08-18T17:01:27.442443+0000","172.21.15.3:0/143641995":"2024-08-18T17:12:37.591006+0000","172.21.15.3:0/601716019":"2024-08-18T17:10:21.996108+0000","172.21.15.3:0/3946755108":"2024-08-18T17:06:25.750593+0000","172.21.15.3:6800/1736600360":"2024-08-18T17:06:25.750593+0000","172.21.15.3:0/3953105554":"2024-08-18T17:06:25.750593+0000","172.21.15.3:6801/2670911286":"2024-08-18T17:10:21.996108+0000","172.21.15.3:0/1583211207":"2024-08-18T17:06:25.750593+0000","172.21.15.3:0/3116588554":"2024-08-18T17:06:25.750593+0000","172.21.15.53:0/1330698653":"2024-08-18T17:11:56.179835+0000","172.21.15.53:6834/4265580715":"2024-08-18T17:11:56.179835+0000","172.21.15.3:0/3817753887":"2024-08-18T17:01:50.631242+0000","172.21.15.3:6801/2187177466":"2024-08-18T17:01:50.631242+0000","172.21.15.3:6800/2670911286":"2024-08-18T17:10:21.996108+0000","172.21.15.3:0/1999667208":"2024-08-18T17:01:50.631242+0000","172.21.15.53:0/721818304":"2024-08-18T17:11:56.179835+0000","172.21.15.3:6801/1736600360":"2024-08-18T17:06:25.750593+0000","172.21.15.3:0/880444026":"2024-08-18T17:10:21.996108+0000","172.21.15.3:0/3717779298":"2024-08-18T17:10:21.996108+0000","172.21.15.3:0/4194950757":"2024-08-18T17:01:27.442443+0000","172.21.15.3:6800/2187177466":"2024-08-18T17:01:50.631242+0000","172.21.15.3:6800/160535497":"2024-08-18T17:01:27.442443+0000","172.21.15.53:6835/4265580715":"2024-08-18T17:11:56.179835+0000","172.21.15.3:0/2029382827":"2024-08-18T17:10:21.996108+0000","172.21.15.53:0/2555682812":"2024-08-18T17:11:56.179835+0000","172.21.15.3:0/1668133120":"2024-08-18T17:01:27.442443+0000","172.21.15.3:6801/160535497":"2024-08-18T17:01:27.442443+0000","172.21.15.3:0/3608486402":"2024-08-18T17:12:37.591006+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":11,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-17T17:48:29.678 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:29 smithi003 ceph-mon[103344]: pgmap v1415: 103 pgs: 103 active+clean; 4.8 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:29.678 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:29 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1844991407' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-17T17:48:29.679 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:29 smithi003 ceph-mon[100615]: pgmap v1415: 103 pgs: 103 active+clean; 4.8 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:29.679 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:29 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1844991407' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-17T17:48:29.805 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:29 smithi053 ceph-mon[89963]: pgmap v1415: 103 pgs: 103 active+clean; 4.8 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:29.805 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:29 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1844991407' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-17T17:48:30.044 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-08-17T17:48:30.045 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-08-17T17:48:30.045 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-08-17T17:48:30.046 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-08-17T17:48:30.046 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-08-17T17:48:30.046 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-08-17T17:48:30.047 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-08-17T17:48:30.047 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-08-17T17:48:30.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:48:30 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:48:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:48:31.037 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:48:31.079 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:48:31.295 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:48:31.296 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:48:31.298 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:48:31.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:31 smithi003 ceph-mon[100615]: pgmap v1416: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:31.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:31 smithi003 ceph-mon[103344]: pgmap v1416: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:31.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:31 smithi053 ceph-mon[89963]: pgmap v1416: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:32.657 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:48:32.889 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:48:33.846 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:33 smithi003 ceph-mon[103344]: pgmap v1417: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:33.846 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:33 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:48:33.846 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:33 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:48:33.846 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:33 smithi003 ceph-mon[100615]: pgmap v1417: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:33.846 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:33 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:48:33.846 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:33 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:48:33.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:33 smithi053 ceph-mon[89963]: pgmap v1417: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:33.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:33 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:48:33.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:33 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:48:34.138 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:48:35.908 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:35 smithi003 ceph-mon[100615]: pgmap v1418: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:35.908 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:35 smithi003 ceph-mon[103344]: pgmap v1418: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:35.909 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:48:35 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:48:35] "GET /metrics HTTP/1.1" 200 34159 "" "Prometheus/2.51.0" 2024-08-17T17:48:35.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:35 smithi053 ceph-mon[89963]: pgmap v1418: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:37.836 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:37 smithi003 ceph-mon[103344]: pgmap v1419: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:37.836 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:37 smithi003 ceph-mon[100615]: pgmap v1419: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:37.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:37 smithi053 ceph-mon[89963]: pgmap v1419: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:38.845 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:38 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:48:38.845 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:38 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:48:38.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:38 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:48:39.709 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:39 smithi003 ceph-mon[103344]: pgmap v1420: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:39.709 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:39 smithi003 ceph-mon[100615]: pgmap v1420: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:39.739 INFO:teuthology.orchestra.run.smithi003.stdout:163208757774 2024-08-17T17:48:39.740 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-08-17T17:48:39.868 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:39 smithi053 ceph-mon[89963]: pgmap v1420: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:40.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:48:40 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:48:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:48:40.715 INFO:teuthology.orchestra.run.smithi003.stdout:468151435640 2024-08-17T17:48:40.715 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-08-17T17:48:41.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:41 smithi053 ceph-mon[89963]: pgmap v1421: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:41.975 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:41 smithi003 ceph-mon[103344]: pgmap v1421: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:41.975 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:41 smithi003 ceph-mon[100615]: pgmap v1421: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:43.656 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:43 smithi003 ceph-mon[103344]: pgmap v1422: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:43.656 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:43 smithi003 ceph-mon[100615]: pgmap v1422: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:43.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:43 smithi053 ceph-mon[89963]: pgmap v1422: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:44.430 INFO:teuthology.orchestra.run.smithi003.stdout:446676599166 2024-08-17T17:48:44.431 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-08-17T17:48:45.929 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:45 smithi003 ceph-mon[100615]: pgmap v1423: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:45.930 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:45 smithi003 ceph-mon[103344]: pgmap v1423: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:45.930 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:48:45 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:48:45] "GET /metrics HTTP/1.1" 200 34140 "" "Prometheus/2.51.0" 2024-08-17T17:48:45.957 INFO:teuthology.orchestra.run.smithi003.stdout:1151051235514 2024-08-17T17:48:45.957 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-08-17T17:48:45.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:45 smithi053 ceph-mon[89963]: pgmap v1423: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:47.472 INFO:teuthology.orchestra.run.smithi003.stdout:403726926221 2024-08-17T17:48:47.473 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-08-17T17:48:47.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:47 smithi003 ceph-mon[103344]: pgmap v1424: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:47.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:47 smithi003 ceph-mon[100615]: pgmap v1424: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:47.966 INFO:teuthology.orchestra.run.smithi003.stdout:231928234503 2024-08-17T17:48:47.970 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-08-17T17:48:47.970 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:47 smithi053 ceph-mon[89963]: pgmap v1424: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:48.986 INFO:teuthology.orchestra.run.smithi003.stdout:128849019414 2024-08-17T17:48:48.987 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-08-17T17:48:49.773 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:49 smithi003 ceph-mon[100615]: pgmap v1425: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:49.773 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:49 smithi003 ceph-mon[103344]: pgmap v1425: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:49.824 INFO:teuthology.orchestra.run.smithi003.stdout:197568496140 2024-08-17T17:48:49.824 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-08-17T17:48:49.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:49 smithi053 ceph-mon[89963]: pgmap v1425: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:50.201 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:48:50.204 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:48:50.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:48:50 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:48:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:48:50.516 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:48:51.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:51 smithi053 ceph-mon[89963]: pgmap v1426: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:52.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:51 smithi003 ceph-mon[100615]: pgmap v1426: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:52.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:51 smithi003 ceph-mon[103344]: pgmap v1426: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:53.849 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:53 smithi003 ceph-mon[100615]: pgmap v1427: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:53.849 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:53 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:48:53.850 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:53 smithi003 ceph-mon[103344]: pgmap v1427: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:53.850 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:53 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:48:53.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:53 smithi053 ceph-mon[89963]: pgmap v1427: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:53.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:53 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:48:54.598 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:48:55.226 INFO:teuthology.orchestra.run.smithi003.stdout:163208757778 2024-08-17T17:48:55.873 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:48:55 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:48:55] "GET /metrics HTTP/1.1" 200 34138 "" "Prometheus/2.51.0" 2024-08-17T17:48:55.873 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:55 smithi003 ceph-mon[103344]: pgmap v1428: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:55.874 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:55 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3288828423' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-08-17T17:48:55.874 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:55 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:48:55.874 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:55 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:48:55.874 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:55 smithi003 ceph-mon[100615]: pgmap v1428: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:55.874 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:55 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3288828423' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-08-17T17:48:55.874 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:55 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:48:55.874 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:55 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:48:55.898 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757774 got 163208757778 for osd.5 2024-08-17T17:48:55.898 DEBUG:teuthology.parallel:result is None 2024-08-17T17:48:55.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:55 smithi053 ceph-mon[89963]: pgmap v1428: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:55.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:55 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3288828423' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-08-17T17:48:55.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:55 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:48:55.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:55 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:48:56.082 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:48:56.082 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:48:56.082 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:48:56.083 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:48:56.295 INFO:teuthology.orchestra.run.smithi003.stdout:128849019416 2024-08-17T17:48:56.905 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:56 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/488188046' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-08-17T17:48:56.905 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:56 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/488188046' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-08-17T17:48:56.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:56 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/488188046' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-08-17T17:48:57.872 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:57 smithi003 ceph-mon[103344]: pgmap v1429: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:57.872 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:57 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:48:57.872 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:57 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:48:57.872 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:57 smithi003 ceph-mon[100615]: pgmap v1429: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:57.872 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:57 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:48:57.872 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:57 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:48:57.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:57 smithi053 ceph-mon[89963]: pgmap v1429: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:57.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:57 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:48:57.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:57 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:48:59.937 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:48:59 smithi003 ceph-mon[103344]: pgmap v1430: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:59.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:48:59 smithi003 ceph-mon[100615]: pgmap v1430: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:48:59.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:48:59 smithi053 ceph-mon[89963]: pgmap v1430: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:49:00.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:49:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:49:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:49:00.638 INFO:teuthology.orchestra.run.smithi003.stdout:468151435644 2024-08-17T17:49:01.565 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849019414 got 128849019416 for osd.4 2024-08-17T17:49:01.566 DEBUG:teuthology.parallel:result is None 2024-08-17T17:49:01.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:01 smithi053 ceph-mon[89963]: pgmap v1431: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:49:01.967 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:01 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2333821657' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-08-17T17:49:02.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:49:01 smithi003 ceph-mon[100615]: pgmap v1431: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:49:02.023 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:49:01 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2333821657' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-08-17T17:49:02.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:49:01 smithi003 ceph-mon[103344]: pgmap v1431: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:49:02.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:49:01 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2333821657' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-08-17T17:49:02.343 INFO:teuthology.orchestra.run.smithi003.stdout:1151051235517 2024-08-17T17:49:02.917 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:49:02 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/2047445994' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-08-17T17:49:02.917 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:49:02 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/2047445994' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-08-17T17:49:02.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:02 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/2047445994' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-08-17T17:49:03.193 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1151051235514 got 1151051235517 for osd.1 2024-08-17T17:49:03.193 DEBUG:teuthology.parallel:result is None 2024-08-17T17:49:03.898 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:49:03 smithi003 ceph-mon[103344]: pgmap v1432: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:49:03.898 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:49:03 smithi003 ceph-mon[100615]: pgmap v1432: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:49:03.966 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:03 smithi053 ceph-mon[89963]: pgmap v1432: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:49:04.611 INFO:tasks.cephadm.ceph_manager.ceph:need seq 468151435640 got 468151435644 for osd.3 2024-08-17T17:49:04.612 DEBUG:teuthology.parallel:result is None 2024-08-17T17:49:05.000 INFO:teuthology.orchestra.run.smithi003.stdout:403726926225 2024-08-17T17:49:05.805 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:49:05 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:49:05] "GET /metrics HTTP/1.1" 200 34138 "" "Prometheus/2.51.0" 2024-08-17T17:49:06.072 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:49:05 smithi003 ceph-mon[103344]: pgmap v1433: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:49:06.072 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:49:05 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3530478020' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-08-17T17:49:06.072 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:49:05 smithi003 ceph-mon[100615]: pgmap v1433: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:49:06.073 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:49:05 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3530478020' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-08-17T17:49:06.086 INFO:teuthology.orchestra.run.smithi003.stdout:446676599172 2024-08-17T17:49:06.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:05 smithi053 ceph-mon[89963]: pgmap v1433: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:49:06.216 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:05 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3530478020' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-08-17T17:49:06.865 INFO:tasks.cephadm.ceph_manager.ceph:need seq 403726926221 got 403726926225 for osd.0 2024-08-17T17:49:06.865 DEBUG:teuthology.parallel:result is None 2024-08-17T17:49:07.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:49:06 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3625691214' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-08-17T17:49:07.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:49:06 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/3652186286' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-08-17T17:49:07.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:49:06 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3625691214' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-08-17T17:49:07.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:49:06 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/3652186286' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-08-17T17:49:07.204 INFO:teuthology.orchestra.run.smithi003.stdout:197568496143 2024-08-17T17:49:07.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:06 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3625691214' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-08-17T17:49:07.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:06 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/3652186286' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-08-17T17:49:07.882 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568496140 got 197568496143 for osd.6 2024-08-17T17:49:07.883 DEBUG:teuthology.parallel:result is None 2024-08-17T17:49:08.129 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:49:07 smithi003 ceph-mon[103344]: pgmap v1434: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:49:08.129 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:49:07 smithi003 ceph-mon[100615]: pgmap v1434: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:49:08.254 INFO:teuthology.orchestra.run.smithi003.stdout:231928234508 2024-08-17T17:49:08.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:07 smithi053 ceph-mon[89963]: pgmap v1434: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:49:08.910 INFO:tasks.cephadm.ceph_manager.ceph:need seq 446676599166 got 446676599172 for osd.2 2024-08-17T17:49:08.910 DEBUG:teuthology.parallel:result is None 2024-08-17T17:49:09.160 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:49:08 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:49:09.160 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:49:08 smithi003 ceph-mon[103344]: from='client.? 172.21.15.3:0/1731501601' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-08-17T17:49:09.160 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:49:08 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:49:09.160 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:49:08 smithi003 ceph-mon[100615]: from='client.? 172.21.15.3:0/1731501601' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-08-17T17:49:09.368 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:08 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-08-17T17:49:09.368 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:08 smithi053 ceph-mon[89963]: from='client.? 172.21.15.3:0/1731501601' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-08-17T17:49:09.618 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234503 got 231928234508 for osd.7 2024-08-17T17:49:09.618 DEBUG:teuthology.parallel:result is None 2024-08-17T17:49:09.618 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-08-17T17:49:09.618 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:49:10.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:49:09 smithi003 ceph-mon[103344]: pgmap v1435: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:49:10.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:49:09 smithi003 ceph-mon[100615]: pgmap v1435: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:49:10.336 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:49:10.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:09 smithi053 ceph-mon[89963]: pgmap v1435: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:49:10.470 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:49:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:49:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:49:12.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:49:11 smithi003 ceph-mon[103344]: pgmap v1436: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:49:12.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:49:11 smithi003 ceph-mon[100615]: pgmap v1436: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:49:12.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:11 smithi053 ceph-mon[89963]: pgmap v1436: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:49:13.243 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:49:13.244 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:49:14.059 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1437,"stamp":"2024-08-17T17:49:12.480458+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790221,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":907,"num_write_kb":12623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":225,"num_bytes_recovered":8399862,"num_keys_recovered":40,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5652,"ondisk_log_size":5652,"up":309,"acting":309,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":309,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6216168,"kb_used_data":58504,"kb_used_omap":37,"kb_used_meta":6157530,"kb_avail":743745048,"statfs":{"total":767960285184,"available":761594929152,"internally_reserved":0,"allocated":59908096,"data_stored":40918951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38729,"internal_metadata":6305310903},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003139"},"pg_stats":[{"pgid":"3.1f","version":"71'4","reported_seq":549,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.777374+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:48:24.777374+0000","last_peered":"2024-08-17T17:48:24.777374+0000","last_clean":"2024-08-17T17:48:24.777374+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:48:24.777374+0000","last_undegraded":"2024-08-17T17:48:24.777374+0000","last_fullsized":"2024-08-17T17:48:24.777374+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":91,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.319876+0000","last_change":"2024-08-17T17:41:34.611711+0000","last_active":"2024-08-17T17:48:20.319876+0000","last_peered":"2024-08-17T17:48:20.319876+0000","last_clean":"2024-08-17T17:48:20.319876+0000","last_became_active":"2024-08-17T17:41:34.611407+0000","last_became_peered":"2024-08-17T17:41:34.611407+0000","last_unstale":"2024-08-17T17:48:20.319876+0000","last_undegraded":"2024-08-17T17:48:20.319876+0000","last_fullsized":"2024-08-17T17:48:20.319876+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:42:41.582320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1d","version":"90'149","reported_seq":805,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.527261+0000","last_change":"2024-08-17T17:42:18.668058+0000","last_active":"2024-08-17T17:48:23.527261+0000","last_peered":"2024-08-17T17:48:23.527261+0000","last_clean":"2024-08-17T17:48:23.527261+0000","last_became_active":"2024-08-17T17:42:05.876721+0000","last_became_peered":"2024-08-17T17:42:05.876721+0000","last_unstale":"2024-08-17T17:48:23.527261+0000","last_undegraded":"2024-08-17T17:48:23.527261+0000","last_fullsized":"2024-08-17T17:48:23.527261+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:18.667957+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:18.667957+0000","last_clean_scrub_stamp":"2024-08-17T17:42:18.667957+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:10:34.861527+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":739,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776625+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:48:24.776625+0000","last_peered":"2024-08-17T17:48:24.776625+0000","last_clean":"2024-08-17T17:48:24.776625+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:48:24.776625+0000","last_undegraded":"2024-08-17T17:48:24.776625+0000","last_fullsized":"2024-08-17T17:48:24.776625+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":738,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776443+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:48:24.776443+0000","last_peered":"2024-08-17T17:48:24.776443+0000","last_clean":"2024-08-17T17:48:24.776443+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:48:24.776443+0000","last_undegraded":"2024-08-17T17:48:24.776443+0000","last_fullsized":"2024-08-17T17:48:24.776443+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"90'149","reported_seq":690,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.485736+0000","last_change":"2024-08-17T17:48:24.485674+0000","last_active":"2024-08-17T17:48:24.485736+0000","last_peered":"2024-08-17T17:48:24.485736+0000","last_clean":"2024-08-17T17:48:24.485736+0000","last_became_active":"2024-08-17T17:48:18.178552+0000","last_became_peered":"2024-08-17T17:48:18.178552+0000","last_unstale":"2024-08-17T17:48:24.485736+0000","last_undegraded":"2024-08-17T17:48:24.485736+0000","last_fullsized":"2024-08-17T17:48:24.485736+0000","mapping_epoch":424,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:48:24.485595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:48:24.485595+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:27:25.291980+0000","scrub_duration":0.003099536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":296,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758400+0000","last_change":"2024-08-17T17:48:04.601672+0000","last_active":"2024-08-17T17:48:20.758400+0000","last_peered":"2024-08-17T17:48:20.758400+0000","last_clean":"2024-08-17T17:48:20.758400+0000","last_became_active":"2024-08-17T17:48:04.600707+0000","last_became_peered":"2024-08-17T17:48:04.600707+0000","last_unstale":"2024-08-17T17:48:20.758400+0000","last_undegraded":"2024-08-17T17:48:20.758400+0000","last_fullsized":"2024-08-17T17:48:20.758400+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:03:22.660881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":91,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.320648+0000","last_change":"2024-08-17T17:41:34.616251+0000","last_active":"2024-08-17T17:48:20.320648+0000","last_peered":"2024-08-17T17:48:20.320648+0000","last_clean":"2024-08-17T17:48:20.320648+0000","last_became_active":"2024-08-17T17:41:34.615952+0000","last_became_peered":"2024-08-17T17:41:34.615952+0000","last_unstale":"2024-08-17T17:48:20.320648+0000","last_undegraded":"2024-08-17T17:48:20.320648+0000","last_fullsized":"2024-08-17T17:48:20.320648+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:12:24.892508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":316,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758247+0000","last_change":"2024-08-17T17:48:04.597583+0000","last_active":"2024-08-17T17:48:20.758247+0000","last_peered":"2024-08-17T17:48:20.758247+0000","last_clean":"2024-08-17T17:48:20.758247+0000","last_became_active":"2024-08-17T17:48:04.597201+0000","last_became_peered":"2024-08-17T17:48:04.597201+0000","last_unstale":"2024-08-17T17:48:20.758247+0000","last_undegraded":"2024-08-17T17:48:20.758247+0000","last_fullsized":"2024-08-17T17:48:20.758247+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:00:42.212114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":91,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.320387+0000","last_change":"2024-08-17T17:41:34.622094+0000","last_active":"2024-08-17T17:48:20.320387+0000","last_peered":"2024-08-17T17:48:20.320387+0000","last_clean":"2024-08-17T17:48:20.320387+0000","last_became_active":"2024-08-17T17:41:34.621773+0000","last_became_peered":"2024-08-17T17:41:34.621773+0000","last_unstale":"2024-08-17T17:48:20.320387+0000","last_undegraded":"2024-08-17T17:48:20.320387+0000","last_fullsized":"2024-08-17T17:48:20.320387+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:34:30.540886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"90'149","reported_seq":558,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776441+0000","last_change":"2024-08-17T17:42:06.742801+0000","last_active":"2024-08-17T17:48:24.776441+0000","last_peered":"2024-08-17T17:48:24.776441+0000","last_clean":"2024-08-17T17:48:24.776441+0000","last_became_active":"2024-08-17T17:42:05.876895+0000","last_became_peered":"2024-08-17T17:42:05.876895+0000","last_unstale":"2024-08-17T17:48:24.776441+0000","last_undegraded":"2024-08-17T17:48:24.776441+0000","last_fullsized":"2024-08-17T17:48:24.776441+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.742723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.742723+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:09:31.346818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":553,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776740+0000","last_change":"2024-08-17T17:41:34.618968+0000","last_active":"2024-08-17T17:48:24.776740+0000","last_peered":"2024-08-17T17:48:24.776740+0000","last_clean":"2024-08-17T17:48:24.776740+0000","last_became_active":"2024-08-17T17:41:34.618576+0000","last_became_peered":"2024-08-17T17:41:34.618576+0000","last_unstale":"2024-08-17T17:48:24.776740+0000","last_undegraded":"2024-08-17T17:48:24.776740+0000","last_fullsized":"2024-08-17T17:48:24.776740+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:47:45.888590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":356,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776585+0000","last_change":"2024-08-17T17:47:45.109132+0000","last_active":"2024-08-17T17:48:24.776585+0000","last_peered":"2024-08-17T17:48:24.776585+0000","last_clean":"2024-08-17T17:48:24.776585+0000","last_became_active":"2024-08-17T17:47:45.108769+0000","last_became_peered":"2024-08-17T17:47:45.108769+0000","last_unstale":"2024-08-17T17:48:24.776585+0000","last_undegraded":"2024-08-17T17:48:24.776585+0000","last_fullsized":"2024-08-17T17:48:24.776585+0000","mapping_epoch":412,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:07.311792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"90'149","reported_seq":910,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776343+0000","last_change":"2024-08-17T17:42:05.876463+0000","last_active":"2024-08-17T17:48:24.776343+0000","last_peered":"2024-08-17T17:48:24.776343+0000","last_clean":"2024-08-17T17:48:24.776343+0000","last_became_active":"2024-08-17T17:42:05.876069+0000","last_became_peered":"2024-08-17T17:42:05.876069+0000","last_unstale":"2024-08-17T17:48:24.776343+0000","last_undegraded":"2024-08-17T17:48:24.776343+0000","last_fullsized":"2024-08-17T17:48:24.776343+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:45.552985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:45.552985+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:10:17.124611+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":91,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.320439+0000","last_change":"2024-08-17T17:41:34.622184+0000","last_active":"2024-08-17T17:48:20.320439+0000","last_peered":"2024-08-17T17:48:20.320439+0000","last_clean":"2024-08-17T17:48:20.320439+0000","last_became_active":"2024-08-17T17:41:34.621831+0000","last_became_peered":"2024-08-17T17:41:34.621831+0000","last_unstale":"2024-08-17T17:48:20.320439+0000","last_undegraded":"2024-08-17T17:48:20.320439+0000","last_fullsized":"2024-08-17T17:48:20.320439+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:59.826667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":296,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758755+0000","last_change":"2024-08-17T17:48:04.598359+0000","last_active":"2024-08-17T17:48:20.758755+0000","last_peered":"2024-08-17T17:48:20.758755+0000","last_clean":"2024-08-17T17:48:20.758755+0000","last_became_active":"2024-08-17T17:48:04.597853+0000","last_became_peered":"2024-08-17T17:48:04.597853+0000","last_unstale":"2024-08-17T17:48:20.758755+0000","last_undegraded":"2024-08-17T17:48:20.758755+0000","last_fullsized":"2024-08-17T17:48:20.758755+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:32:43.999143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.18","version":"413'150","reported_seq":665,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758331+0000","last_change":"2024-08-17T17:48:15.739736+0000","last_active":"2024-08-17T17:48:20.758331+0000","last_peered":"2024-08-17T17:48:20.758331+0000","last_clean":"2024-08-17T17:48:20.758331+0000","last_became_active":"2024-08-17T17:48:04.596887+0000","last_became_peered":"2024-08-17T17:48:04.596887+0000","last_unstale":"2024-08-17T17:48:20.758331+0000","last_undegraded":"2024-08-17T17:48:20.758331+0000","last_fullsized":"2024-08-17T17:48:20.758331+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":5,"last_scrub":"413'150","last_scrub_stamp":"2024-08-17T17:48:15.739659+0000","last_deep_scrub":"413'150","last_deep_scrub_stamp":"2024-08-17T17:47:45.268445+0000","last_clean_scrub_stamp":"2024-08-17T17:48:15.739659+0000","objects_scrubbed":1,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:02:06.067520+0000","scrub_duration":0.0022082579999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":679,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.526546+0000","last_change":"2024-08-17T17:41:34.625011+0000","last_active":"2024-08-17T17:48:23.526546+0000","last_peered":"2024-08-17T17:48:23.526546+0000","last_clean":"2024-08-17T17:48:23.526546+0000","last_became_active":"2024-08-17T17:41:34.624631+0000","last_became_peered":"2024-08-17T17:41:34.624631+0000","last_unstale":"2024-08-17T17:48:23.526546+0000","last_undegraded":"2024-08-17T17:48:23.526546+0000","last_fullsized":"2024-08-17T17:48:23.526546+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:03:05.565435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":530,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776900+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:48:24.776900+0000","last_peered":"2024-08-17T17:48:24.776900+0000","last_clean":"2024-08-17T17:48:24.776900+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:48:24.776900+0000","last_undegraded":"2024-08-17T17:48:24.776900+0000","last_fullsized":"2024-08-17T17:48:24.776900+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1b","version":"90'149","reported_seq":703,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758326+0000","last_change":"2024-08-17T17:48:17.770713+0000","last_active":"2024-08-17T17:48:20.758326+0000","last_peered":"2024-08-17T17:48:20.758326+0000","last_clean":"2024-08-17T17:48:20.758326+0000","last_became_active":"2024-08-17T17:48:04.597269+0000","last_became_peered":"2024-08-17T17:48:04.597269+0000","last_unstale":"2024-08-17T17:48:20.758326+0000","last_undegraded":"2024-08-17T17:48:20.758326+0000","last_fullsized":"2024-08-17T17:48:20.758326+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:48:17.770640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:48:17.770640+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:57.530110+0000","scrub_duration":0.0019915190000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":760,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.775768+0000","last_change":"2024-08-17T17:33:20.771279+0000","last_active":"2024-08-17T17:48:24.775768+0000","last_peered":"2024-08-17T17:48:24.775768+0000","last_clean":"2024-08-17T17:48:24.775768+0000","last_became_active":"2024-08-17T17:33:20.770728+0000","last_became_peered":"2024-08-17T17:33:20.770728+0000","last_unstale":"2024-08-17T17:48:24.775768+0000","last_undegraded":"2024-08-17T17:48:24.775768+0000","last_fullsized":"2024-08-17T17:48:24.775768+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:52:31.540397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":400,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.192023+0000","last_change":"2024-08-17T17:33:20.835760+0000","last_active":"2024-08-17T17:48:20.192023+0000","last_peered":"2024-08-17T17:48:20.192023+0000","last_clean":"2024-08-17T17:48:20.192023+0000","last_became_active":"2024-08-17T17:33:20.835410+0000","last_became_peered":"2024-08-17T17:33:20.835410+0000","last_unstale":"2024-08-17T17:48:20.192023+0000","last_undegraded":"2024-08-17T17:48:20.192023+0000","last_fullsized":"2024-08-17T17:48:20.192023+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:45.448022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1a","version":"90'149","reported_seq":475,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.319954+0000","last_change":"2024-08-17T17:42:21.932830+0000","last_active":"2024-08-17T17:48:20.319954+0000","last_peered":"2024-08-17T17:48:20.319954+0000","last_clean":"2024-08-17T17:48:20.319954+0000","last_became_active":"2024-08-17T17:42:05.876835+0000","last_became_peered":"2024-08-17T17:42:05.876835+0000","last_unstale":"2024-08-17T17:48:20.319954+0000","last_undegraded":"2024-08-17T17:48:20.319954+0000","last_fullsized":"2024-08-17T17:48:20.319954+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:21.932757+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:21.932757+0000","last_clean_scrub_stamp":"2024-08-17T17:42:21.932757+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:36:39.848570+0000","scrub_duration":0.0020218810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":742,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.775608+0000","last_change":"2024-08-17T17:33:20.790687+0000","last_active":"2024-08-17T17:48:24.775608+0000","last_peered":"2024-08-17T17:48:24.775608+0000","last_clean":"2024-08-17T17:48:24.775608+0000","last_became_active":"2024-08-17T17:33:20.790399+0000","last_became_peered":"2024-08-17T17:33:20.790399+0000","last_unstale":"2024-08-17T17:48:24.775608+0000","last_undegraded":"2024-08-17T17:48:24.775608+0000","last_fullsized":"2024-08-17T17:48:24.775608+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:38:06.849989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":711,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.777350+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:48:24.777350+0000","last_peered":"2024-08-17T17:48:24.777350+0000","last_clean":"2024-08-17T17:48:24.777350+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:48:24.777350+0000","last_undegraded":"2024-08-17T17:48:24.777350+0000","last_fullsized":"2024-08-17T17:48:24.777350+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"90'149","reported_seq":813,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.189450+0000","last_change":"2024-08-17T17:42:06.456562+0000","last_active":"2024-08-17T17:48:20.189450+0000","last_peered":"2024-08-17T17:48:20.189450+0000","last_clean":"2024-08-17T17:48:20.189450+0000","last_became_active":"2024-08-17T17:42:05.876577+0000","last_became_peered":"2024-08-17T17:42:05.876577+0000","last_unstale":"2024-08-17T17:48:20.189450+0000","last_undegraded":"2024-08-17T17:48:20.189450+0000","last_fullsized":"2024-08-17T17:48:20.189450+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.456490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.456490+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:10:03.654953+0000","scrub_duration":0.001537285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":684,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.775682+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:48:24.775682+0000","last_peered":"2024-08-17T17:48:24.775682+0000","last_clean":"2024-08-17T17:48:24.775682+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:48:24.775682+0000","last_undegraded":"2024-08-17T17:48:24.775682+0000","last_fullsized":"2024-08-17T17:48:24.775682+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":123,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758338+0000","last_change":"2024-08-17T17:48:04.599622+0000","last_active":"2024-08-17T17:48:20.758338+0000","last_peered":"2024-08-17T17:48:20.758338+0000","last_clean":"2024-08-17T17:48:20.758338+0000","last_became_active":"2024-08-17T17:48:04.599289+0000","last_became_peered":"2024-08-17T17:48:04.599289+0000","last_unstale":"2024-08-17T17:48:20.758338+0000","last_undegraded":"2024-08-17T17:48:20.758338+0000","last_fullsized":"2024-08-17T17:48:20.758338+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:27:06.331364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"90'149","reported_seq":906,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776419+0000","last_change":"2024-08-17T17:41:37.664471+0000","last_active":"2024-08-17T17:48:24.776419+0000","last_peered":"2024-08-17T17:48:24.776419+0000","last_clean":"2024-08-17T17:48:24.776419+0000","last_became_active":"2024-08-17T17:41:37.664108+0000","last_became_peered":"2024-08-17T17:41:37.664108+0000","last_unstale":"2024-08-17T17:48:24.776419+0000","last_undegraded":"2024-08-17T17:48:24.776419+0000","last_fullsized":"2024-08-17T17:48:24.776419+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:22.030514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:22.030514+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:23:19.853098+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":170,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.189085+0000","last_change":"2024-08-17T17:48:18.181507+0000","last_active":"2024-08-17T17:48:20.189085+0000","last_peered":"2024-08-17T17:48:20.189085+0000","last_clean":"2024-08-17T17:48:20.189085+0000","last_became_active":"2024-08-17T17:48:18.180017+0000","last_became_peered":"2024-08-17T17:48:18.180017+0000","last_unstale":"2024-08-17T17:48:20.189085+0000","last_undegraded":"2024-08-17T17:48:20.189085+0000","last_fullsized":"2024-08-17T17:48:20.189085+0000","mapping_epoch":424,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:57:57.548524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":680,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.526612+0000","last_change":"2024-08-17T17:33:20.799402+0000","last_active":"2024-08-17T17:48:23.526612+0000","last_peered":"2024-08-17T17:48:23.526612+0000","last_clean":"2024-08-17T17:48:23.526612+0000","last_became_active":"2024-08-17T17:33:20.798832+0000","last_became_peered":"2024-08-17T17:33:20.798832+0000","last_unstale":"2024-08-17T17:48:23.526612+0000","last_undegraded":"2024-08-17T17:48:23.526612+0000","last_fullsized":"2024-08-17T17:48:23.526612+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:21:05.001134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"90'149","reported_seq":411,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758433+0000","last_change":"2024-08-17T17:48:12.784018+0000","last_active":"2024-08-17T17:48:20.758433+0000","last_peered":"2024-08-17T17:48:20.758433+0000","last_clean":"2024-08-17T17:48:20.758433+0000","last_became_active":"2024-08-17T17:48:04.599356+0000","last_became_peered":"2024-08-17T17:48:04.599356+0000","last_unstale":"2024-08-17T17:48:20.758433+0000","last_undegraded":"2024-08-17T17:48:20.758433+0000","last_fullsized":"2024-08-17T17:48:20.758433+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:48:12.783940+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:06.698998+0000","last_clean_scrub_stamp":"2024-08-17T17:48:12.783940+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:14:33.197091+0000","scrub_duration":0.0015484699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":101,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.320680+0000","last_change":"2024-08-17T17:41:34.626584+0000","last_active":"2024-08-17T17:48:20.320680+0000","last_peered":"2024-08-17T17:48:20.320680+0000","last_clean":"2024-08-17T17:48:20.320680+0000","last_became_active":"2024-08-17T17:41:34.609878+0000","last_became_peered":"2024-08-17T17:41:34.609878+0000","last_unstale":"2024-08-17T17:48:20.320680+0000","last_undegraded":"2024-08-17T17:48:20.320680+0000","last_fullsized":"2024-08-17T17:48:20.320680+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:08:12.810347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":91,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.320530+0000","last_change":"2024-08-17T17:41:34.622180+0000","last_active":"2024-08-17T17:48:20.320530+0000","last_peered":"2024-08-17T17:48:20.320530+0000","last_clean":"2024-08-17T17:48:20.320530+0000","last_became_active":"2024-08-17T17:41:34.621724+0000","last_became_peered":"2024-08-17T17:41:34.621724+0000","last_unstale":"2024-08-17T17:48:20.320530+0000","last_undegraded":"2024-08-17T17:48:20.320530+0000","last_fullsized":"2024-08-17T17:48:20.320530+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:15.807235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"90'149","reported_seq":793,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.526878+0000","last_change":"2024-08-17T17:42:07.681204+0000","last_active":"2024-08-17T17:48:23.526878+0000","last_peered":"2024-08-17T17:48:23.526878+0000","last_clean":"2024-08-17T17:48:23.526878+0000","last_became_active":"2024-08-17T17:42:05.878769+0000","last_became_peered":"2024-08-17T17:42:05.878769+0000","last_unstale":"2024-08-17T17:48:23.526878+0000","last_undegraded":"2024-08-17T17:48:23.526878+0000","last_fullsized":"2024-08-17T17:48:23.526878+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:07.681064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:07.681064+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:50:29.541934+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":91,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.320030+0000","last_change":"2024-08-17T17:41:34.614028+0000","last_active":"2024-08-17T17:48:20.320030+0000","last_peered":"2024-08-17T17:48:20.320030+0000","last_clean":"2024-08-17T17:48:20.320030+0000","last_became_active":"2024-08-17T17:41:34.613706+0000","last_became_peered":"2024-08-17T17:41:34.613706+0000","last_unstale":"2024-08-17T17:48:20.320030+0000","last_undegraded":"2024-08-17T17:48:20.320030+0000","last_fullsized":"2024-08-17T17:48:20.320030+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:19:55.254902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":657,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.192029+0000","last_change":"2024-08-17T17:26:06.855307+0000","last_active":"2024-08-17T17:48:20.192029+0000","last_peered":"2024-08-17T17:48:20.192029+0000","last_clean":"2024-08-17T17:48:20.192029+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:48:20.192029+0000","last_undegraded":"2024-08-17T17:48:20.192029+0000","last_fullsized":"2024-08-17T17:48:20.192029+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":575,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.777081+0000","last_change":"2024-08-17T17:41:34.629522+0000","last_active":"2024-08-17T17:48:24.777081+0000","last_peered":"2024-08-17T17:48:24.777081+0000","last_clean":"2024-08-17T17:48:24.777081+0000","last_became_active":"2024-08-17T17:41:34.619340+0000","last_became_peered":"2024-08-17T17:41:34.619340+0000","last_unstale":"2024-08-17T17:48:24.777081+0000","last_undegraded":"2024-08-17T17:48:24.777081+0000","last_fullsized":"2024-08-17T17:48:24.777081+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:30:03.902956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":883,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776387+0000","last_change":"2024-08-17T17:38:23.113397+0000","last_active":"2024-08-17T17:48:24.776387+0000","last_peered":"2024-08-17T17:48:24.776387+0000","last_clean":"2024-08-17T17:48:24.776387+0000","last_became_active":"2024-08-17T17:38:21.305580+0000","last_became_peered":"2024-08-17T17:38:21.305580+0000","last_unstale":"2024-08-17T17:48:24.776387+0000","last_undegraded":"2024-08-17T17:48:24.776387+0000","last_fullsized":"2024-08-17T17:48:24.776387+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.113279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.113279+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:54:14.909136+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":735,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.775677+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:48:24.775677+0000","last_peered":"2024-08-17T17:48:24.775677+0000","last_clean":"2024-08-17T17:48:24.775677+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:48:24.775677+0000","last_undegraded":"2024-08-17T17:48:24.775677+0000","last_fullsized":"2024-08-17T17:48:24.775677+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":1313,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.527074+0000","last_change":"2024-08-17T17:41:37.677524+0000","last_active":"2024-08-17T17:48:23.527074+0000","last_peered":"2024-08-17T17:48:23.527074+0000","last_clean":"2024-08-17T17:48:23.527074+0000","last_became_active":"2024-08-17T17:41:37.676764+0000","last_became_peered":"2024-08-17T17:41:37.676764+0000","last_unstale":"2024-08-17T17:48:23.527074+0000","last_undegraded":"2024-08-17T17:48:23.527074+0000","last_fullsized":"2024-08-17T17:48:23.527074+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:14:49.341080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":738,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776655+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:48:24.776655+0000","last_peered":"2024-08-17T17:48:24.776655+0000","last_clean":"2024-08-17T17:48:24.776655+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:48:24.776655+0000","last_undegraded":"2024-08-17T17:48:24.776655+0000","last_fullsized":"2024-08-17T17:48:24.776655+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":810,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.320848+0000","last_change":"2024-08-17T17:42:19.960161+0000","last_active":"2024-08-17T17:48:20.320848+0000","last_peered":"2024-08-17T17:48:20.320848+0000","last_clean":"2024-08-17T17:48:20.320848+0000","last_became_active":"2024-08-17T17:41:37.665223+0000","last_became_peered":"2024-08-17T17:41:37.665223+0000","last_unstale":"2024-08-17T17:48:20.320848+0000","last_undegraded":"2024-08-17T17:48:20.320848+0000","last_fullsized":"2024-08-17T17:48:20.320848+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:19.960086+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:19.960086+0000","last_clean_scrub_stamp":"2024-08-17T17:42:19.960086+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:57.861661+0000","scrub_duration":0.0022612000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":703,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776915+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:48:24.776915+0000","last_peered":"2024-08-17T17:48:24.776915+0000","last_clean":"2024-08-17T17:48:24.776915+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:48:24.776915+0000","last_undegraded":"2024-08-17T17:48:24.776915+0000","last_fullsized":"2024-08-17T17:48:24.776915+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":987,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.189461+0000","last_change":"2024-08-17T17:41:37.683212+0000","last_active":"2024-08-17T17:48:20.189461+0000","last_peered":"2024-08-17T17:48:20.189461+0000","last_clean":"2024-08-17T17:48:20.189461+0000","last_became_active":"2024-08-17T17:41:37.682730+0000","last_became_peered":"2024-08-17T17:41:37.682730+0000","last_unstale":"2024-08-17T17:48:20.189461+0000","last_undegraded":"2024-08-17T17:48:20.189461+0000","last_fullsized":"2024-08-17T17:48:20.189461+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:51:35.444433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":688,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.526726+0000","last_change":"2024-08-17T17:33:20.798613+0000","last_active":"2024-08-17T17:48:23.526726+0000","last_peered":"2024-08-17T17:48:23.526726+0000","last_clean":"2024-08-17T17:48:23.526726+0000","last_became_active":"2024-08-17T17:33:20.798260+0000","last_became_peered":"2024-08-17T17:33:20.798260+0000","last_unstale":"2024-08-17T17:48:23.526726+0000","last_undegraded":"2024-08-17T17:48:23.526726+0000","last_fullsized":"2024-08-17T17:48:23.526726+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:18:00.822681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":564,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.191868+0000","last_change":"2024-08-17T17:41:44.058291+0000","last_active":"2024-08-17T17:48:20.191868+0000","last_peered":"2024-08-17T17:48:20.191868+0000","last_clean":"2024-08-17T17:48:20.191868+0000","last_became_active":"2024-08-17T17:41:12.200077+0000","last_became_peered":"2024-08-17T17:41:12.200077+0000","last_unstale":"2024-08-17T17:48:20.191868+0000","last_undegraded":"2024-08-17T17:48:20.191868+0000","last_fullsized":"2024-08-17T17:48:20.191868+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:44.058233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:44.058233+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:42:55.911623+0000","scrub_duration":0.0074268499999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":684,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.527020+0000","last_change":"2024-08-17T17:41:34.627064+0000","last_active":"2024-08-17T17:48:23.527020+0000","last_peered":"2024-08-17T17:48:23.527020+0000","last_clean":"2024-08-17T17:48:23.527020+0000","last_became_active":"2024-08-17T17:41:34.626453+0000","last_became_peered":"2024-08-17T17:41:34.626453+0000","last_unstale":"2024-08-17T17:48:23.527020+0000","last_undegraded":"2024-08-17T17:48:23.527020+0000","last_fullsized":"2024-08-17T17:48:23.527020+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:02:11.894042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":868,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.526848+0000","last_change":"2024-08-17T17:33:20.799350+0000","last_active":"2024-08-17T17:48:23.526848+0000","last_peered":"2024-08-17T17:48:23.526848+0000","last_clean":"2024-08-17T17:48:23.526848+0000","last_became_active":"2024-08-17T17:33:20.798925+0000","last_became_peered":"2024-08-17T17:33:20.798925+0000","last_unstale":"2024-08-17T17:48:23.526848+0000","last_undegraded":"2024-08-17T17:48:23.526848+0000","last_fullsized":"2024-08-17T17:48:23.526848+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:16:36.121295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":265,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.759145+0000","last_change":"2024-08-17T17:48:04.594463+0000","last_active":"2024-08-17T17:48:20.759145+0000","last_peered":"2024-08-17T17:48:20.759145+0000","last_clean":"2024-08-17T17:48:20.759145+0000","last_became_active":"2024-08-17T17:48:04.594035+0000","last_became_peered":"2024-08-17T17:48:04.594035+0000","last_unstale":"2024-08-17T17:48:20.759145+0000","last_undegraded":"2024-08-17T17:48:20.759145+0000","last_fullsized":"2024-08-17T17:48:20.759145+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:15:24.348195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":564,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.191903+0000","last_change":"2024-08-17T17:41:43.035215+0000","last_active":"2024-08-17T17:48:20.191903+0000","last_peered":"2024-08-17T17:48:20.191903+0000","last_clean":"2024-08-17T17:48:20.191903+0000","last_became_active":"2024-08-17T17:41:12.201158+0000","last_became_peered":"2024-08-17T17:41:12.201158+0000","last_unstale":"2024-08-17T17:48:20.191903+0000","last_undegraded":"2024-08-17T17:48:20.191903+0000","last_fullsized":"2024-08-17T17:48:20.191903+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:43.035108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:43.035108+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:43:27.108994+0000","scrub_duration":0.0028877400000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":653,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.527153+0000","last_change":"2024-08-17T17:33:20.772454+0000","last_active":"2024-08-17T17:48:23.527153+0000","last_peered":"2024-08-17T17:48:23.527153+0000","last_clean":"2024-08-17T17:48:23.527153+0000","last_became_active":"2024-08-17T17:33:20.771848+0000","last_became_peered":"2024-08-17T17:33:20.771848+0000","last_unstale":"2024-08-17T17:48:23.527153+0000","last_undegraded":"2024-08-17T17:48:23.527153+0000","last_fullsized":"2024-08-17T17:48:23.527153+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:46:42.300744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":91,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.320549+0000","last_change":"2024-08-17T17:41:34.611109+0000","last_active":"2024-08-17T17:48:20.320549+0000","last_peered":"2024-08-17T17:48:20.320549+0000","last_clean":"2024-08-17T17:48:20.320549+0000","last_became_active":"2024-08-17T17:41:34.610170+0000","last_became_peered":"2024-08-17T17:41:34.610170+0000","last_unstale":"2024-08-17T17:48:20.320549+0000","last_undegraded":"2024-08-17T17:48:20.320549+0000","last_fullsized":"2024-08-17T17:48:20.320549+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:39:05.301260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":1217,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758477+0000","last_change":"2024-08-17T17:48:04.601159+0000","last_active":"2024-08-17T17:48:20.758477+0000","last_peered":"2024-08-17T17:48:20.758477+0000","last_clean":"2024-08-17T17:48:20.758477+0000","last_became_active":"2024-08-17T17:48:04.600380+0000","last_became_peered":"2024-08-17T17:48:04.600380+0000","last_unstale":"2024-08-17T17:48:20.758477+0000","last_undegraded":"2024-08-17T17:48:20.758477+0000","last_fullsized":"2024-08-17T17:48:20.758477+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:12:25.955259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1392,"num_read_kb":2863,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":650,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758548+0000","last_change":"2024-08-17T17:48:04.898296+0000","last_active":"2024-08-17T17:48:20.758548+0000","last_peered":"2024-08-17T17:48:20.758548+0000","last_clean":"2024-08-17T17:48:20.758548+0000","last_became_active":"2024-08-17T17:48:04.595809+0000","last_became_peered":"2024-08-17T17:48:04.595809+0000","last_unstale":"2024-08-17T17:48:20.758548+0000","last_undegraded":"2024-08-17T17:48:20.758548+0000","last_fullsized":"2024-08-17T17:48:20.758548+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:48:04.898228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:48:04.898228+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:36:24.166969+0000","scrub_duration":0.0019622400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":649,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.526929+0000","last_change":"2024-08-17T17:33:20.835784+0000","last_active":"2024-08-17T17:48:23.526929+0000","last_peered":"2024-08-17T17:48:23.526929+0000","last_clean":"2024-08-17T17:48:23.526929+0000","last_became_active":"2024-08-17T17:33:20.835371+0000","last_became_peered":"2024-08-17T17:33:20.835371+0000","last_unstale":"2024-08-17T17:48:23.526929+0000","last_undegraded":"2024-08-17T17:48:23.526929+0000","last_fullsized":"2024-08-17T17:48:23.526929+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:35:43.369055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":613,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.526469+0000","last_change":"2024-08-17T17:33:20.793468+0000","last_active":"2024-08-17T17:48:23.526469+0000","last_peered":"2024-08-17T17:48:23.526469+0000","last_clean":"2024-08-17T17:48:23.526469+0000","last_became_active":"2024-08-17T17:33:20.793166+0000","last_became_peered":"2024-08-17T17:33:20.793166+0000","last_unstale":"2024-08-17T17:48:23.526469+0000","last_undegraded":"2024-08-17T17:48:23.526469+0000","last_fullsized":"2024-08-17T17:48:23.526469+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:12:52.121050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":643,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776391+0000","last_change":"2024-08-17T17:47:45.055879+0000","last_active":"2024-08-17T17:48:24.776391+0000","last_peered":"2024-08-17T17:48:24.776391+0000","last_clean":"2024-08-17T17:48:24.776391+0000","last_became_active":"2024-08-17T17:47:45.055199+0000","last_became_peered":"2024-08-17T17:47:45.055199+0000","last_unstale":"2024-08-17T17:48:24.776391+0000","last_undegraded":"2024-08-17T17:48:24.776391+0000","last_fullsized":"2024-08-17T17:48:24.776391+0000","mapping_epoch":412,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:29:40.639457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":605,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.188677+0000","last_change":"2024-08-17T17:48:19.388610+0000","last_active":"2024-08-17T17:48:20.188677+0000","last_peered":"2024-08-17T17:48:20.188677+0000","last_clean":"2024-08-17T17:48:20.188677+0000","last_became_active":"2024-08-17T17:48:18.179425+0000","last_became_peered":"2024-08-17T17:48:18.179425+0000","last_unstale":"2024-08-17T17:48:20.188677+0000","last_undegraded":"2024-08-17T17:48:20.188677+0000","last_fullsized":"2024-08-17T17:48:20.188677+0000","mapping_epoch":424,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:48:19.388542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:48:19.388542+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:33:06.359371+0000","scrub_duration":0.0019005300000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":529,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.777181+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:48:24.777181+0000","last_peered":"2024-08-17T17:48:24.777181+0000","last_clean":"2024-08-17T17:48:24.777181+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:48:24.777181+0000","last_undegraded":"2024-08-17T17:48:24.777181+0000","last_fullsized":"2024-08-17T17:48:24.777181+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":742,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.775944+0000","last_change":"2024-08-17T17:41:34.614108+0000","last_active":"2024-08-17T17:48:24.775944+0000","last_peered":"2024-08-17T17:48:24.775944+0000","last_clean":"2024-08-17T17:48:24.775944+0000","last_became_active":"2024-08-17T17:41:34.613481+0000","last_became_peered":"2024-08-17T17:41:34.613481+0000","last_unstale":"2024-08-17T17:48:24.775944+0000","last_undegraded":"2024-08-17T17:48:24.775944+0000","last_fullsized":"2024-08-17T17:48:24.775944+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:21:02.346115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":865,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.188828+0000","last_change":"2024-08-17T17:48:18.181301+0000","last_active":"2024-08-17T17:48:20.188828+0000","last_peered":"2024-08-17T17:48:20.188828+0000","last_clean":"2024-08-17T17:48:20.188828+0000","last_became_active":"2024-08-17T17:48:18.180765+0000","last_became_peered":"2024-08-17T17:48:18.180765+0000","last_unstale":"2024-08-17T17:48:20.188828+0000","last_undegraded":"2024-08-17T17:48:20.188828+0000","last_fullsized":"2024-08-17T17:48:20.188828+0000","mapping_epoch":424,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:30:41.676392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":918,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.527243+0000","last_change":"2024-08-17T17:41:37.676347+0000","last_active":"2024-08-17T17:48:23.527243+0000","last_peered":"2024-08-17T17:48:23.527243+0000","last_clean":"2024-08-17T17:48:23.527243+0000","last_became_active":"2024-08-17T17:41:37.675869+0000","last_became_peered":"2024-08-17T17:41:37.675869+0000","last_unstale":"2024-08-17T17:48:23.527243+0000","last_undegraded":"2024-08-17T17:48:23.527243+0000","last_fullsized":"2024-08-17T17:48:23.527243+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:21.814077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:21.814077+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:52:23.492179+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":268,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.191540+0000","last_change":"2024-08-17T17:41:34.618631+0000","last_active":"2024-08-17T17:48:20.191540+0000","last_peered":"2024-08-17T17:48:20.191540+0000","last_clean":"2024-08-17T17:48:20.191540+0000","last_became_active":"2024-08-17T17:41:34.618313+0000","last_became_peered":"2024-08-17T17:41:34.618313+0000","last_unstale":"2024-08-17T17:48:20.191540+0000","last_undegraded":"2024-08-17T17:48:20.191540+0000","last_fullsized":"2024-08-17T17:48:20.191540+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:07:13.915088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":692,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776298+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:48:24.776298+0000","last_peered":"2024-08-17T17:48:24.776298+0000","last_clean":"2024-08-17T17:48:24.776298+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:48:24.776298+0000","last_undegraded":"2024-08-17T17:48:24.776298+0000","last_fullsized":"2024-08-17T17:48:24.776298+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":862,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758204+0000","last_change":"2024-08-17T17:48:04.600544+0000","last_active":"2024-08-17T17:48:20.758204+0000","last_peered":"2024-08-17T17:48:20.758204+0000","last_clean":"2024-08-17T17:48:20.758204+0000","last_became_active":"2024-08-17T17:48:04.600007+0000","last_became_peered":"2024-08-17T17:48:04.600007+0000","last_unstale":"2024-08-17T17:48:20.758204+0000","last_undegraded":"2024-08-17T17:48:20.758204+0000","last_fullsized":"2024-08-17T17:48:20.758204+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:00:07.477190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":605,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.454414+0000","last_change":"2024-08-17T17:48:23.454326+0000","last_active":"2024-08-17T17:48:23.454414+0000","last_peered":"2024-08-17T17:48:23.454414+0000","last_clean":"2024-08-17T17:48:23.454414+0000","last_became_active":"2024-08-17T17:48:18.181315+0000","last_became_peered":"2024-08-17T17:48:18.181315+0000","last_unstale":"2024-08-17T17:48:23.454414+0000","last_undegraded":"2024-08-17T17:48:23.454414+0000","last_fullsized":"2024-08-17T17:48:23.454414+0000","mapping_epoch":424,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:48:23.454246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:48:23.454246+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:41:46.088324+0000","scrub_duration":0.0017975840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":639,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.526960+0000","last_change":"2024-08-17T17:41:34.625551+0000","last_active":"2024-08-17T17:48:23.526960+0000","last_peered":"2024-08-17T17:48:23.526960+0000","last_clean":"2024-08-17T17:48:23.526960+0000","last_became_active":"2024-08-17T17:41:34.625251+0000","last_became_peered":"2024-08-17T17:41:34.625251+0000","last_unstale":"2024-08-17T17:48:23.526960+0000","last_undegraded":"2024-08-17T17:48:23.526960+0000","last_fullsized":"2024-08-17T17:48:23.526960+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:28:22.354213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":744,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.775849+0000","last_change":"2024-08-17T17:33:20.796216+0000","last_active":"2024-08-17T17:48:24.775849+0000","last_peered":"2024-08-17T17:48:24.775849+0000","last_clean":"2024-08-17T17:48:24.775849+0000","last_became_active":"2024-08-17T17:33:20.795762+0000","last_became_peered":"2024-08-17T17:33:20.795762+0000","last_unstale":"2024-08-17T17:48:24.775849+0000","last_undegraded":"2024-08-17T17:48:24.775849+0000","last_fullsized":"2024-08-17T17:48:24.775849+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:47.038509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":408,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758473+0000","last_change":"2024-08-17T17:48:07.831433+0000","last_active":"2024-08-17T17:48:20.758473+0000","last_peered":"2024-08-17T17:48:20.758473+0000","last_clean":"2024-08-17T17:48:20.758473+0000","last_became_active":"2024-08-17T17:48:04.600291+0000","last_became_peered":"2024-08-17T17:48:04.600291+0000","last_unstale":"2024-08-17T17:48:20.758473+0000","last_undegraded":"2024-08-17T17:48:20.758473+0000","last_fullsized":"2024-08-17T17:48:20.758473+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:48:07.831349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:48:07.831349+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:58:27.937907+0000","scrub_duration":0.002146724,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":91,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.320920+0000","last_change":"2024-08-17T17:41:34.622102+0000","last_active":"2024-08-17T17:48:20.320920+0000","last_peered":"2024-08-17T17:48:20.320920+0000","last_clean":"2024-08-17T17:48:20.320920+0000","last_became_active":"2024-08-17T17:41:34.621733+0000","last_became_peered":"2024-08-17T17:41:34.621733+0000","last_unstale":"2024-08-17T17:48:20.320920+0000","last_undegraded":"2024-08-17T17:48:20.320920+0000","last_fullsized":"2024-08-17T17:48:20.320920+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:39:41.668709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":91,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.320180+0000","last_change":"2024-08-17T17:41:34.607602+0000","last_active":"2024-08-17T17:48:20.320180+0000","last_peered":"2024-08-17T17:48:20.320180+0000","last_clean":"2024-08-17T17:48:20.320180+0000","last_became_active":"2024-08-17T17:41:34.607300+0000","last_became_peered":"2024-08-17T17:41:34.607300+0000","last_unstale":"2024-08-17T17:48:20.320180+0000","last_undegraded":"2024-08-17T17:48:20.320180+0000","last_fullsized":"2024-08-17T17:48:20.320180+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:20:31.486698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"90'149","reported_seq":935,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.527074+0000","last_change":"2024-08-17T17:41:41.743931+0000","last_active":"2024-08-17T17:48:23.527074+0000","last_peered":"2024-08-17T17:48:23.527074+0000","last_clean":"2024-08-17T17:48:23.527074+0000","last_became_active":"2024-08-17T17:41:12.203771+0000","last_became_peered":"2024-08-17T17:41:12.203771+0000","last_unstale":"2024-08-17T17:48:23.527074+0000","last_undegraded":"2024-08-17T17:48:23.527074+0000","last_fullsized":"2024-08-17T17:48:23.527074+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:41.743570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:41.743570+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:13:18.592919+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":91,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.321072+0000","last_change":"2024-08-17T17:41:34.619268+0000","last_active":"2024-08-17T17:48:20.321072+0000","last_peered":"2024-08-17T17:48:20.321072+0000","last_clean":"2024-08-17T17:48:20.321072+0000","last_became_active":"2024-08-17T17:41:34.618914+0000","last_became_peered":"2024-08-17T17:41:34.618914+0000","last_unstale":"2024-08-17T17:48:20.321072+0000","last_undegraded":"2024-08-17T17:48:20.321072+0000","last_fullsized":"2024-08-17T17:48:20.321072+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:08:54.248386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":273,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.777188+0000","last_change":"2024-08-17T17:47:45.150169+0000","last_active":"2024-08-17T17:48:24.777188+0000","last_peered":"2024-08-17T17:48:24.777188+0000","last_clean":"2024-08-17T17:48:24.777188+0000","last_became_active":"2024-08-17T17:47:45.149913+0000","last_became_peered":"2024-08-17T17:47:45.149913+0000","last_unstale":"2024-08-17T17:48:24.777188+0000","last_undegraded":"2024-08-17T17:48:24.777188+0000","last_fullsized":"2024-08-17T17:48:24.777188+0000","mapping_epoch":412,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:07:23.807428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":836,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758628+0000","last_change":"2024-08-17T17:48:04.591625+0000","last_active":"2024-08-17T17:48:20.758628+0000","last_peered":"2024-08-17T17:48:20.758628+0000","last_clean":"2024-08-17T17:48:20.758628+0000","last_became_active":"2024-08-17T17:48:04.591148+0000","last_became_peered":"2024-08-17T17:48:04.591148+0000","last_unstale":"2024-08-17T17:48:20.758628+0000","last_undegraded":"2024-08-17T17:48:20.758628+0000","last_fullsized":"2024-08-17T17:48:20.758628+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:47:47.300846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:47:47.300846+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:06:50.680849+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":91,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.320540+0000","last_change":"2024-08-17T17:41:34.611049+0000","last_active":"2024-08-17T17:48:20.320540+0000","last_peered":"2024-08-17T17:48:20.320540+0000","last_clean":"2024-08-17T17:48:20.320540+0000","last_became_active":"2024-08-17T17:41:34.610727+0000","last_became_peered":"2024-08-17T17:41:34.610727+0000","last_unstale":"2024-08-17T17:48:20.320540+0000","last_undegraded":"2024-08-17T17:48:20.320540+0000","last_fullsized":"2024-08-17T17:48:20.320540+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:04:05.477660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":786,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776062+0000","last_change":"2024-08-17T17:41:34.630434+0000","last_active":"2024-08-17T17:48:24.776062+0000","last_peered":"2024-08-17T17:48:24.776062+0000","last_clean":"2024-08-17T17:48:24.776062+0000","last_became_active":"2024-08-17T17:41:34.616406+0000","last_became_peered":"2024-08-17T17:41:34.616406+0000","last_unstale":"2024-08-17T17:48:24.776062+0000","last_undegraded":"2024-08-17T17:48:24.776062+0000","last_fullsized":"2024-08-17T17:48:24.776062+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:30:44.817203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":605,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:25.454719+0000","last_change":"2024-08-17T17:48:25.454679+0000","last_active":"2024-08-17T17:48:25.454719+0000","last_peered":"2024-08-17T17:48:25.454719+0000","last_clean":"2024-08-17T17:48:25.454719+0000","last_became_active":"2024-08-17T17:48:18.178322+0000","last_became_peered":"2024-08-17T17:48:18.178322+0000","last_unstale":"2024-08-17T17:48:25.454719+0000","last_undegraded":"2024-08-17T17:48:25.454719+0000","last_fullsized":"2024-08-17T17:48:25.454719+0000","mapping_epoch":424,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:48:25.454608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:48:25.454608+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:41:01.366455+0000","scrub_duration":0.001814502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":750,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.777000+0000","last_change":"2024-08-17T17:41:34.615197+0000","last_active":"2024-08-17T17:48:24.777000+0000","last_peered":"2024-08-17T17:48:24.777000+0000","last_clean":"2024-08-17T17:48:24.777000+0000","last_became_active":"2024-08-17T17:41:34.614873+0000","last_became_peered":"2024-08-17T17:41:34.614873+0000","last_unstale":"2024-08-17T17:48:24.777000+0000","last_undegraded":"2024-08-17T17:48:24.777000+0000","last_fullsized":"2024-08-17T17:48:24.777000+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:21:57.391953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":248,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758234+0000","last_change":"2024-08-17T17:48:04.599957+0000","last_active":"2024-08-17T17:48:20.758234+0000","last_peered":"2024-08-17T17:48:20.758234+0000","last_clean":"2024-08-17T17:48:20.758234+0000","last_became_active":"2024-08-17T17:48:04.599605+0000","last_became_peered":"2024-08-17T17:48:04.599605+0000","last_unstale":"2024-08-17T17:48:20.758234+0000","last_undegraded":"2024-08-17T17:48:20.758234+0000","last_fullsized":"2024-08-17T17:48:20.758234+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:26:59.143293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"90'149","reported_seq":735,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.321065+0000","last_change":"2024-08-17T17:42:20.927007+0000","last_active":"2024-08-17T17:48:20.321065+0000","last_peered":"2024-08-17T17:48:20.321065+0000","last_clean":"2024-08-17T17:48:20.321065+0000","last_became_active":"2024-08-17T17:41:37.670914+0000","last_became_peered":"2024-08-17T17:41:37.670914+0000","last_unstale":"2024-08-17T17:48:20.321065+0000","last_undegraded":"2024-08-17T17:48:20.321065+0000","last_fullsized":"2024-08-17T17:48:20.321065+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:20.926924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:20.926924+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:57:44.156600+0000","scrub_duration":0.0018485109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":255,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.191513+0000","last_change":"2024-08-17T17:33:20.795582+0000","last_active":"2024-08-17T17:48:20.191513+0000","last_peered":"2024-08-17T17:48:20.191513+0000","last_clean":"2024-08-17T17:48:20.191513+0000","last_became_active":"2024-08-17T17:33:20.795149+0000","last_became_peered":"2024-08-17T17:33:20.795149+0000","last_unstale":"2024-08-17T17:48:20.191513+0000","last_undegraded":"2024-08-17T17:48:20.191513+0000","last_fullsized":"2024-08-17T17:48:20.191513+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:14.020923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":238,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.189341+0000","last_change":"2024-08-17T17:48:18.182348+0000","last_active":"2024-08-17T17:48:20.189341+0000","last_peered":"2024-08-17T17:48:20.189341+0000","last_clean":"2024-08-17T17:48:20.189341+0000","last_became_active":"2024-08-17T17:48:18.181956+0000","last_became_peered":"2024-08-17T17:48:18.181956+0000","last_unstale":"2024-08-17T17:48:20.189341+0000","last_undegraded":"2024-08-17T17:48:20.189341+0000","last_fullsized":"2024-08-17T17:48:20.189341+0000","mapping_epoch":424,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:02:54.992931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"90'149","reported_seq":603,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.191934+0000","last_change":"2024-08-17T17:41:37.668050+0000","last_active":"2024-08-17T17:48:20.191934+0000","last_peered":"2024-08-17T17:48:20.191934+0000","last_clean":"2024-08-17T17:48:20.191934+0000","last_became_active":"2024-08-17T17:41:37.667655+0000","last_became_peered":"2024-08-17T17:41:37.667655+0000","last_unstale":"2024-08-17T17:48:20.191934+0000","last_undegraded":"2024-08-17T17:48:20.191934+0000","last_fullsized":"2024-08-17T17:48:20.191934+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:32.843802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:32.843802+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:39:46.766280+0000","scrub_duration":0.0040803669999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":91,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.320039+0000","last_change":"2024-08-17T17:41:34.621078+0000","last_active":"2024-08-17T17:48:20.320039+0000","last_peered":"2024-08-17T17:48:20.320039+0000","last_clean":"2024-08-17T17:48:20.320039+0000","last_became_active":"2024-08-17T17:41:34.620730+0000","last_became_peered":"2024-08-17T17:41:34.620730+0000","last_unstale":"2024-08-17T17:48:20.320039+0000","last_undegraded":"2024-08-17T17:48:20.320039+0000","last_fullsized":"2024-08-17T17:48:20.320039+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:55:01.312283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":384,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.189437+0000","last_change":"2024-08-17T17:41:34.637696+0000","last_active":"2024-08-17T17:48:20.189437+0000","last_peered":"2024-08-17T17:48:20.189437+0000","last_clean":"2024-08-17T17:48:20.189437+0000","last_became_active":"2024-08-17T17:41:34.621388+0000","last_became_peered":"2024-08-17T17:41:34.621388+0000","last_unstale":"2024-08-17T17:48:20.189437+0000","last_undegraded":"2024-08-17T17:48:20.189437+0000","last_fullsized":"2024-08-17T17:48:20.189437+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:13:04.607036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"90'149","reported_seq":927,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776490+0000","last_change":"2024-08-17T17:41:42.603841+0000","last_active":"2024-08-17T17:48:24.776490+0000","last_peered":"2024-08-17T17:48:24.776490+0000","last_clean":"2024-08-17T17:48:24.776490+0000","last_became_active":"2024-08-17T17:41:12.203400+0000","last_became_peered":"2024-08-17T17:41:12.203400+0000","last_unstale":"2024-08-17T17:48:24.776490+0000","last_undegraded":"2024-08-17T17:48:24.776490+0000","last_fullsized":"2024-08-17T17:48:24.776490+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:42.603561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:42.603561+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:43:50.209009+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":116,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758172+0000","last_change":"2024-08-17T17:48:04.600776+0000","last_active":"2024-08-17T17:48:20.758172+0000","last_peered":"2024-08-17T17:48:20.758172+0000","last_clean":"2024-08-17T17:48:20.758172+0000","last_became_active":"2024-08-17T17:48:04.600189+0000","last_became_peered":"2024-08-17T17:48:04.600189+0000","last_unstale":"2024-08-17T17:48:20.758172+0000","last_undegraded":"2024-08-17T17:48:20.758172+0000","last_fullsized":"2024-08-17T17:48:20.758172+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:29.973785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":388,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.189429+0000","last_change":"2024-08-17T17:41:34.635980+0000","last_active":"2024-08-17T17:48:20.189429+0000","last_peered":"2024-08-17T17:48:20.189429+0000","last_clean":"2024-08-17T17:48:20.189429+0000","last_became_active":"2024-08-17T17:41:34.620596+0000","last_became_peered":"2024-08-17T17:41:34.620596+0000","last_unstale":"2024-08-17T17:48:20.189429+0000","last_undegraded":"2024-08-17T17:48:20.189429+0000","last_fullsized":"2024-08-17T17:48:20.189429+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:19:15.792577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"90'149","reported_seq":837,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.188982+0000","last_change":"2024-08-17T17:42:16.610940+0000","last_active":"2024-08-17T17:48:20.188982+0000","last_peered":"2024-08-17T17:48:20.188982+0000","last_clean":"2024-08-17T17:48:20.188982+0000","last_became_active":"2024-08-17T17:41:37.670855+0000","last_became_peered":"2024-08-17T17:41:37.670855+0000","last_unstale":"2024-08-17T17:48:20.188982+0000","last_undegraded":"2024-08-17T17:48:20.188982+0000","last_fullsized":"2024-08-17T17:48:20.188982+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:16.610828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:16.610828+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:48:25.918610+0000","scrub_duration":0.086892487000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":512,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.191481+0000","last_change":"2024-08-17T17:43:52.520659+0000","last_active":"2024-08-17T17:48:20.191481+0000","last_peered":"2024-08-17T17:48:20.191481+0000","last_clean":"2024-08-17T17:48:20.191481+0000","last_became_active":"2024-08-17T17:43:52.516799+0000","last_became_peered":"2024-08-17T17:43:52.516799+0000","last_unstale":"2024-08-17T17:48:20.191481+0000","last_undegraded":"2024-08-17T17:48:20.191481+0000","last_fullsized":"2024-08-17T17:48:20.191481+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:25:59.408383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":742,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776508+0000","last_change":"2024-08-17T17:33:20.794682+0000","last_active":"2024-08-17T17:48:24.776508+0000","last_peered":"2024-08-17T17:48:24.776508+0000","last_clean":"2024-08-17T17:48:24.776508+0000","last_became_active":"2024-08-17T17:33:20.794238+0000","last_became_peered":"2024-08-17T17:33:20.794238+0000","last_unstale":"2024-08-17T17:48:24.776508+0000","last_undegraded":"2024-08-17T17:48:24.776508+0000","last_fullsized":"2024-08-17T17:48:24.776508+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:15:55.821920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"90'149","reported_seq":694,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758239+0000","last_change":"2024-08-17T17:48:04.601616+0000","last_active":"2024-08-17T17:48:20.758239+0000","last_peered":"2024-08-17T17:48:20.758239+0000","last_clean":"2024-08-17T17:48:20.758239+0000","last_became_active":"2024-08-17T17:48:04.601091+0000","last_became_peered":"2024-08-17T17:48:04.601091+0000","last_unstale":"2024-08-17T17:48:20.758239+0000","last_undegraded":"2024-08-17T17:48:20.758239+0000","last_fullsized":"2024-08-17T17:48:20.758239+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:24.291592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:24.291592+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:50:01.146205+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":219,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.188685+0000","last_change":"2024-08-17T17:48:18.181063+0000","last_active":"2024-08-17T17:48:20.188685+0000","last_peered":"2024-08-17T17:48:20.188685+0000","last_clean":"2024-08-17T17:48:20.188685+0000","last_became_active":"2024-08-17T17:48:18.180711+0000","last_became_peered":"2024-08-17T17:48:18.180711+0000","last_unstale":"2024-08-17T17:48:20.188685+0000","last_undegraded":"2024-08-17T17:48:20.188685+0000","last_fullsized":"2024-08-17T17:48:20.188685+0000","mapping_epoch":424,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:19:08.158472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":528,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776989+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:48:24.776989+0000","last_peered":"2024-08-17T17:48:24.776989+0000","last_clean":"2024-08-17T17:48:24.776989+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:48:24.776989+0000","last_undegraded":"2024-08-17T17:48:24.776989+0000","last_fullsized":"2024-08-17T17:48:24.776989+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"90'149","reported_seq":926,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.526903+0000","last_change":"2024-08-17T17:41:37.681187+0000","last_active":"2024-08-17T17:48:23.526903+0000","last_peered":"2024-08-17T17:48:23.526903+0000","last_clean":"2024-08-17T17:48:23.526903+0000","last_became_active":"2024-08-17T17:41:37.680605+0000","last_became_peered":"2024-08-17T17:41:37.680605+0000","last_unstale":"2024-08-17T17:48:23.526903+0000","last_undegraded":"2024-08-17T17:48:23.526903+0000","last_fullsized":"2024-08-17T17:48:23.526903+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.844252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.844252+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:14.260864+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":528,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776144+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:48:24.776144+0000","last_peered":"2024-08-17T17:48:24.776144+0000","last_clean":"2024-08-17T17:48:24.776144+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:48:24.776144+0000","last_undegraded":"2024-08-17T17:48:24.776144+0000","last_fullsized":"2024-08-17T17:48:24.776144+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":106,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758662+0000","last_change":"2024-08-17T17:48:04.597329+0000","last_active":"2024-08-17T17:48:20.758662+0000","last_peered":"2024-08-17T17:48:20.758662+0000","last_clean":"2024-08-17T17:48:20.758662+0000","last_became_active":"2024-08-17T17:48:04.596967+0000","last_became_peered":"2024-08-17T17:48:04.596967+0000","last_unstale":"2024-08-17T17:48:20.758662+0000","last_undegraded":"2024-08-17T17:48:20.758662+0000","last_fullsized":"2024-08-17T17:48:20.758662+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:52:52.097006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"90'149","reported_seq":408,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758492+0000","last_change":"2024-08-17T17:48:14.754802+0000","last_active":"2024-08-17T17:48:20.758492+0000","last_peered":"2024-08-17T17:48:20.758492+0000","last_clean":"2024-08-17T17:48:20.758492+0000","last_became_active":"2024-08-17T17:48:04.600862+0000","last_became_peered":"2024-08-17T17:48:04.600862+0000","last_unstale":"2024-08-17T17:48:20.758492+0000","last_undegraded":"2024-08-17T17:48:20.758492+0000","last_fullsized":"2024-08-17T17:48:20.758492+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:48:14.754726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:48:14.754726+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:06:18.332300+0000","scrub_duration":0.0017611,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":451,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.188872+0000","last_change":"2024-08-17T17:41:34.624198+0000","last_active":"2024-08-17T17:48:20.188872+0000","last_peered":"2024-08-17T17:48:20.188872+0000","last_clean":"2024-08-17T17:48:20.188872+0000","last_became_active":"2024-08-17T17:41:34.616071+0000","last_became_peered":"2024-08-17T17:41:34.616071+0000","last_unstale":"2024-08-17T17:48:20.188872+0000","last_undegraded":"2024-08-17T17:48:20.188872+0000","last_fullsized":"2024-08-17T17:48:20.188872+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:34:48.805354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":728,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776648+0000","last_change":"2024-08-17T17:33:20.797029+0000","last_active":"2024-08-17T17:48:24.776648+0000","last_peered":"2024-08-17T17:48:24.776648+0000","last_clean":"2024-08-17T17:48:24.776648+0000","last_became_active":"2024-08-17T17:33:20.796679+0000","last_became_peered":"2024-08-17T17:33:20.796679+0000","last_unstale":"2024-08-17T17:48:24.776648+0000","last_undegraded":"2024-08-17T17:48:24.776648+0000","last_fullsized":"2024-08-17T17:48:24.776648+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:51.438262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"90'149","reported_seq":908,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776000+0000","last_change":"2024-08-17T17:38:30.378664+0000","last_active":"2024-08-17T17:48:24.776000+0000","last_peered":"2024-08-17T17:48:24.776000+0000","last_clean":"2024-08-17T17:48:24.776000+0000","last_became_active":"2024-08-17T17:38:30.342287+0000","last_became_peered":"2024-08-17T17:38:30.342287+0000","last_unstale":"2024-08-17T17:48:24.776000+0000","last_undegraded":"2024-08-17T17:48:24.776000+0000","last_fullsized":"2024-08-17T17:48:24.776000+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:30.378592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:30.378592+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:14:05.215821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":11216,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19280,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4620,"ondisk_log_size":4620,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234509,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":912148,"kb_used_data":9928,"kb_used_omap":1,"kb_used_meta":902206,"kb_avail":92833004,"statfs":{"total":95995035648,"available":95060996096,"internally_reserved":0,"allocated":10166272,"data_stored":7780293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":923859288},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496144,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":900932,"kb_used_data":9976,"kb_used_omap":1,"kb_used_meta":890942,"kb_avail":92844220,"statfs":{"total":95995035648,"available":95072481280,"internally_reserved":0,"allocated":10215424,"data_stored":7844959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":912324952},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757781,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":888132,"kb_used_data":5816,"kb_used_omap":1,"kb_used_meta":882302,"kb_avail":92857020,"statfs":{"total":95995035648,"available":95085588480,"internally_reserved":0,"allocated":5955584,"data_stored":3581688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":903477592},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019419,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307088,"kb_used_data":5572,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93438064,"statfs":{"total":95995035648,"available":95680577536,"internally_reserved":0,"allocated":5705728,"data_stored":3323562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435647,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":966012,"kb_used_data":5872,"kb_used_omap":4,"kb_used_meta":960123,"kb_avail":92779140,"statfs":{"total":95995035648,"available":95005839360,"internally_reserved":0,"allocated":6012928,"data_stored":3646462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":983165962},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599173,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":845700,"kb_used_data":9912,"kb_used_omap":7,"kb_used_meta":835768,"kb_avail":92899452,"statfs":{"total":95995035648,"available":95129038848,"internally_reserved":0,"allocated":10149888,"data_stored":7776288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":855826594},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":268,"seq":1151051235520,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":455804,"kb_used_data":5552,"kb_used_omap":10,"kb_used_meta":450229,"kb_avail":93289348,"statfs":{"total":95995035648,"available":95528292352,"internally_reserved":0,"allocated":5685248,"data_stored":3318727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10902,"internal_metadata":461034858},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926227,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":940352,"kb_used_data":5876,"kb_used_omap":7,"kb_used_meta":934456,"kb_avail":92804800,"statfs":{"total":95995035648,"available":95032115200,"internally_reserved":0,"allocated":6017024,"data_stored":3646972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":956883265},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7550,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:49:14.065 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 -- ceph pg dump --format=json 2024-08-17T17:49:14.273 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:49:13 smithi003 ceph-mon[103344]: pgmap v1437: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:49:14.273 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:49:13 smithi003 ceph-mon[100615]: pgmap v1437: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:49:14.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:13 smithi053 ceph-mon[89963]: pgmap v1437: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:49:14.869 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/mon.a/config 2024-08-17T17:49:15.021 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:49:14 smithi003 ceph-mon[100615]: from='client.47135 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:49:15.021 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:49:14 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:49:15.021 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:49:14 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:49:15.021 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:49:14 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:49:15.021 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:49:14 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:49:15.021 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:49:14 smithi003 ceph-mon[100615]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:49:15.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:49:14 smithi003 ceph-mon[103344]: from='client.47135 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:49:15.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:49:14 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:49:15.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:49:14 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:49:15.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:49:14 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:49:15.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:49:14 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:49:15.022 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:49:14 smithi003 ceph-mon[103344]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:49:15.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:14 smithi053 ceph-mon[89963]: from='client.47135 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:49:15.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:14 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:49:15.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:14 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:49:15.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:14 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-17T17:49:15.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:14 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-17T17:49:15.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:14 smithi053 ceph-mon[89963]: from='mgr.26282 172.21.15.3:0/2156945202' entity='mgr.y' 2024-08-17T17:49:15.910 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:49:15 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: ::ffff:172.21.15.53 - - [17/Aug/2024:17:49:15] "GET /metrics HTTP/1.1" 200 34138 "" "Prometheus/2.51.0" 2024-08-17T17:49:16.199 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:49:15 smithi003 ceph-mon[100615]: pgmap v1438: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:49:16.199 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:49:15 smithi003 ceph-mon[103344]: pgmap v1438: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:49:16.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:15 smithi053 ceph-mon[89963]: pgmap v1438: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:49:16.584 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-08-17T17:49:16.585 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-08-17T17:49:17.323 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1439,"stamp":"2024-08-17T17:49:16.481587+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4790221,"num_objects":28,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":907,"num_write_kb":12623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":225,"num_bytes_recovered":8399862,"num_keys_recovered":40,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5652,"ondisk_log_size":5652,"up":309,"acting":309,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":309,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6216168,"kb_used_data":58504,"kb_used_omap":37,"kb_used_meta":6157530,"kb_avail":743745048,"statfs":{"total":767960285184,"available":761594929152,"internally_reserved":0,"allocated":59908096,"data_stored":40918951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38729,"internal_metadata":6305310903},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003193"},"pg_stats":[{"pgid":"3.1f","version":"71'4","reported_seq":549,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.777374+0000","last_change":"2024-08-17T17:24:32.304311+0000","last_active":"2024-08-17T17:48:24.777374+0000","last_peered":"2024-08-17T17:48:24.777374+0000","last_clean":"2024-08-17T17:48:24.777374+0000","last_became_active":"2024-08-17T17:24:32.301036+0000","last_became_peered":"2024-08-17T17:24:32.301036+0000","last_unstale":"2024-08-17T17:48:24.777374+0000","last_undegraded":"2024-08-17T17:48:24.777374+0000","last_fullsized":"2024-08-17T17:48:24.777374+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:51:20.035507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":91,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.319876+0000","last_change":"2024-08-17T17:41:34.611711+0000","last_active":"2024-08-17T17:48:20.319876+0000","last_peered":"2024-08-17T17:48:20.319876+0000","last_clean":"2024-08-17T17:48:20.319876+0000","last_became_active":"2024-08-17T17:41:34.611407+0000","last_became_peered":"2024-08-17T17:41:34.611407+0000","last_unstale":"2024-08-17T17:48:20.319876+0000","last_undegraded":"2024-08-17T17:48:20.319876+0000","last_fullsized":"2024-08-17T17:48:20.319876+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:42:41.582320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1d","version":"90'149","reported_seq":805,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.527261+0000","last_change":"2024-08-17T17:42:18.668058+0000","last_active":"2024-08-17T17:48:23.527261+0000","last_peered":"2024-08-17T17:48:23.527261+0000","last_clean":"2024-08-17T17:48:23.527261+0000","last_became_active":"2024-08-17T17:42:05.876721+0000","last_became_peered":"2024-08-17T17:42:05.876721+0000","last_unstale":"2024-08-17T17:48:23.527261+0000","last_undegraded":"2024-08-17T17:48:23.527261+0000","last_fullsized":"2024-08-17T17:48:23.527261+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:18.667957+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:18.667957+0000","last_clean_scrub_stamp":"2024-08-17T17:42:18.667957+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:10:34.861527+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"71'1","reported_seq":739,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776625+0000","last_change":"2024-08-17T17:25:04.524363+0000","last_active":"2024-08-17T17:48:24.776625+0000","last_peered":"2024-08-17T17:48:24.776625+0000","last_clean":"2024-08-17T17:48:24.776625+0000","last_became_active":"2024-08-17T17:25:04.523994+0000","last_became_peered":"2024-08-17T17:25:04.523994+0000","last_unstale":"2024-08-17T17:48:24.776625+0000","last_undegraded":"2024-08-17T17:48:24.776625+0000","last_fullsized":"2024-08-17T17:48:24.776625+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:26:10.928796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":738,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776443+0000","last_change":"2024-08-17T17:17:32.498999+0000","last_active":"2024-08-17T17:48:24.776443+0000","last_peered":"2024-08-17T17:48:24.776443+0000","last_clean":"2024-08-17T17:48:24.776443+0000","last_became_active":"2024-08-17T17:17:32.498509+0000","last_became_peered":"2024-08-17T17:17:32.498509+0000","last_unstale":"2024-08-17T17:48:24.776443+0000","last_undegraded":"2024-08-17T17:48:24.776443+0000","last_fullsized":"2024-08-17T17:48:24.776443+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:46:16.011776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"90'149","reported_seq":690,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.485736+0000","last_change":"2024-08-17T17:48:24.485674+0000","last_active":"2024-08-17T17:48:24.485736+0000","last_peered":"2024-08-17T17:48:24.485736+0000","last_clean":"2024-08-17T17:48:24.485736+0000","last_became_active":"2024-08-17T17:48:18.178552+0000","last_became_peered":"2024-08-17T17:48:18.178552+0000","last_unstale":"2024-08-17T17:48:24.485736+0000","last_undegraded":"2024-08-17T17:48:24.485736+0000","last_fullsized":"2024-08-17T17:48:24.485736+0000","mapping_epoch":424,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:48:24.485595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:48:24.485595+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:27:25.291980+0000","scrub_duration":0.003099536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"71'2","reported_seq":296,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758400+0000","last_change":"2024-08-17T17:48:04.601672+0000","last_active":"2024-08-17T17:48:20.758400+0000","last_peered":"2024-08-17T17:48:20.758400+0000","last_clean":"2024-08-17T17:48:20.758400+0000","last_became_active":"2024-08-17T17:48:04.600707+0000","last_became_peered":"2024-08-17T17:48:04.600707+0000","last_unstale":"2024-08-17T17:48:20.758400+0000","last_undegraded":"2024-08-17T17:48:20.758400+0000","last_fullsized":"2024-08-17T17:48:20.758400+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:03:22.660881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":91,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.320648+0000","last_change":"2024-08-17T17:41:34.616251+0000","last_active":"2024-08-17T17:48:20.320648+0000","last_peered":"2024-08-17T17:48:20.320648+0000","last_clean":"2024-08-17T17:48:20.320648+0000","last_became_active":"2024-08-17T17:41:34.615952+0000","last_became_peered":"2024-08-17T17:41:34.615952+0000","last_unstale":"2024-08-17T17:48:20.320648+0000","last_undegraded":"2024-08-17T17:48:20.320648+0000","last_fullsized":"2024-08-17T17:48:20.320648+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:12:24.892508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"71'2","reported_seq":316,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758247+0000","last_change":"2024-08-17T17:48:04.597583+0000","last_active":"2024-08-17T17:48:20.758247+0000","last_peered":"2024-08-17T17:48:20.758247+0000","last_clean":"2024-08-17T17:48:20.758247+0000","last_became_active":"2024-08-17T17:48:04.597201+0000","last_became_peered":"2024-08-17T17:48:04.597201+0000","last_unstale":"2024-08-17T17:48:20.758247+0000","last_undegraded":"2024-08-17T17:48:20.758247+0000","last_fullsized":"2024-08-17T17:48:20.758247+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:00:42.212114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":91,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.320387+0000","last_change":"2024-08-17T17:41:34.622094+0000","last_active":"2024-08-17T17:48:20.320387+0000","last_peered":"2024-08-17T17:48:20.320387+0000","last_clean":"2024-08-17T17:48:20.320387+0000","last_became_active":"2024-08-17T17:41:34.621773+0000","last_became_peered":"2024-08-17T17:41:34.621773+0000","last_unstale":"2024-08-17T17:48:20.320387+0000","last_undegraded":"2024-08-17T17:48:20.320387+0000","last_fullsized":"2024-08-17T17:48:20.320387+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:34:30.540886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"90'149","reported_seq":558,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776441+0000","last_change":"2024-08-17T17:42:06.742801+0000","last_active":"2024-08-17T17:48:24.776441+0000","last_peered":"2024-08-17T17:48:24.776441+0000","last_clean":"2024-08-17T17:48:24.776441+0000","last_became_active":"2024-08-17T17:42:05.876895+0000","last_became_peered":"2024-08-17T17:42:05.876895+0000","last_unstale":"2024-08-17T17:48:24.776441+0000","last_undegraded":"2024-08-17T17:48:24.776441+0000","last_fullsized":"2024-08-17T17:48:24.776441+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.742723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.742723+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:09:31.346818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"71'1","reported_seq":553,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776740+0000","last_change":"2024-08-17T17:41:34.618968+0000","last_active":"2024-08-17T17:48:24.776740+0000","last_peered":"2024-08-17T17:48:24.776740+0000","last_clean":"2024-08-17T17:48:24.776740+0000","last_became_active":"2024-08-17T17:41:34.618576+0000","last_became_peered":"2024-08-17T17:41:34.618576+0000","last_unstale":"2024-08-17T17:48:24.776740+0000","last_undegraded":"2024-08-17T17:48:24.776740+0000","last_fullsized":"2024-08-17T17:48:24.776740+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:47:45.888590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":356,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776585+0000","last_change":"2024-08-17T17:47:45.109132+0000","last_active":"2024-08-17T17:48:24.776585+0000","last_peered":"2024-08-17T17:48:24.776585+0000","last_clean":"2024-08-17T17:48:24.776585+0000","last_became_active":"2024-08-17T17:47:45.108769+0000","last_became_peered":"2024-08-17T17:47:45.108769+0000","last_unstale":"2024-08-17T17:48:24.776585+0000","last_undegraded":"2024-08-17T17:48:24.776585+0000","last_fullsized":"2024-08-17T17:48:24.776585+0000","mapping_epoch":412,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:07.311792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.19","version":"90'149","reported_seq":910,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776343+0000","last_change":"2024-08-17T17:42:05.876463+0000","last_active":"2024-08-17T17:48:24.776343+0000","last_peered":"2024-08-17T17:48:24.776343+0000","last_clean":"2024-08-17T17:48:24.776343+0000","last_became_active":"2024-08-17T17:42:05.876069+0000","last_became_peered":"2024-08-17T17:42:05.876069+0000","last_unstale":"2024-08-17T17:48:24.776343+0000","last_undegraded":"2024-08-17T17:48:24.776343+0000","last_fullsized":"2024-08-17T17:48:24.776343+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:45.552985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:45.552985+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:10:17.124611+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"71'1","reported_seq":91,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.320439+0000","last_change":"2024-08-17T17:41:34.622184+0000","last_active":"2024-08-17T17:48:20.320439+0000","last_peered":"2024-08-17T17:48:20.320439+0000","last_clean":"2024-08-17T17:48:20.320439+0000","last_became_active":"2024-08-17T17:41:34.621831+0000","last_became_peered":"2024-08-17T17:41:34.621831+0000","last_unstale":"2024-08-17T17:48:20.320439+0000","last_undegraded":"2024-08-17T17:48:20.320439+0000","last_fullsized":"2024-08-17T17:48:20.320439+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:59.826667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":296,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758755+0000","last_change":"2024-08-17T17:48:04.598359+0000","last_active":"2024-08-17T17:48:20.758755+0000","last_peered":"2024-08-17T17:48:20.758755+0000","last_clean":"2024-08-17T17:48:20.758755+0000","last_became_active":"2024-08-17T17:48:04.597853+0000","last_became_peered":"2024-08-17T17:48:04.597853+0000","last_unstale":"2024-08-17T17:48:20.758755+0000","last_undegraded":"2024-08-17T17:48:20.758755+0000","last_fullsized":"2024-08-17T17:48:20.758755+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:32:43.999143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.18","version":"413'150","reported_seq":665,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758331+0000","last_change":"2024-08-17T17:48:15.739736+0000","last_active":"2024-08-17T17:48:20.758331+0000","last_peered":"2024-08-17T17:48:20.758331+0000","last_clean":"2024-08-17T17:48:20.758331+0000","last_became_active":"2024-08-17T17:48:04.596887+0000","last_became_peered":"2024-08-17T17:48:04.596887+0000","last_unstale":"2024-08-17T17:48:20.758331+0000","last_undegraded":"2024-08-17T17:48:20.758331+0000","last_fullsized":"2024-08-17T17:48:20.758331+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":5,"last_scrub":"413'150","last_scrub_stamp":"2024-08-17T17:48:15.739659+0000","last_deep_scrub":"413'150","last_deep_scrub_stamp":"2024-08-17T17:47:45.268445+0000","last_clean_scrub_stamp":"2024-08-17T17:48:15.739659+0000","objects_scrubbed":1,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:02:06.067520+0000","scrub_duration":0.0022082579999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":679,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.526546+0000","last_change":"2024-08-17T17:41:34.625011+0000","last_active":"2024-08-17T17:48:23.526546+0000","last_peered":"2024-08-17T17:48:23.526546+0000","last_clean":"2024-08-17T17:48:23.526546+0000","last_became_active":"2024-08-17T17:41:34.624631+0000","last_became_peered":"2024-08-17T17:41:34.624631+0000","last_unstale":"2024-08-17T17:48:23.526546+0000","last_undegraded":"2024-08-17T17:48:23.526546+0000","last_fullsized":"2024-08-17T17:48:23.526546+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:03:05.565435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":530,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776900+0000","last_change":"2024-08-17T17:24:06.586568+0000","last_active":"2024-08-17T17:48:24.776900+0000","last_peered":"2024-08-17T17:48:24.776900+0000","last_clean":"2024-08-17T17:48:24.776900+0000","last_became_active":"2024-08-17T17:24:06.585742+0000","last_became_peered":"2024-08-17T17:24:06.585742+0000","last_unstale":"2024-08-17T17:48:24.776900+0000","last_undegraded":"2024-08-17T17:48:24.776900+0000","last_fullsized":"2024-08-17T17:48:24.776900+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:58:28.595813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1b","version":"90'149","reported_seq":703,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758326+0000","last_change":"2024-08-17T17:48:17.770713+0000","last_active":"2024-08-17T17:48:20.758326+0000","last_peered":"2024-08-17T17:48:20.758326+0000","last_clean":"2024-08-17T17:48:20.758326+0000","last_became_active":"2024-08-17T17:48:04.597269+0000","last_became_peered":"2024-08-17T17:48:04.597269+0000","last_unstale":"2024-08-17T17:48:20.758326+0000","last_undegraded":"2024-08-17T17:48:20.758326+0000","last_fullsized":"2024-08-17T17:48:20.758326+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:48:17.770640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:48:17.770640+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:57.530110+0000","scrub_duration":0.0019915190000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":760,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.775768+0000","last_change":"2024-08-17T17:33:20.771279+0000","last_active":"2024-08-17T17:48:24.775768+0000","last_peered":"2024-08-17T17:48:24.775768+0000","last_clean":"2024-08-17T17:48:24.775768+0000","last_became_active":"2024-08-17T17:33:20.770728+0000","last_became_peered":"2024-08-17T17:33:20.770728+0000","last_unstale":"2024-08-17T17:48:24.775768+0000","last_undegraded":"2024-08-17T17:48:24.775768+0000","last_fullsized":"2024-08-17T17:48:24.775768+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:52:31.540397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":400,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.192023+0000","last_change":"2024-08-17T17:33:20.835760+0000","last_active":"2024-08-17T17:48:20.192023+0000","last_peered":"2024-08-17T17:48:20.192023+0000","last_clean":"2024-08-17T17:48:20.192023+0000","last_became_active":"2024-08-17T17:33:20.835410+0000","last_became_peered":"2024-08-17T17:33:20.835410+0000","last_unstale":"2024-08-17T17:48:20.192023+0000","last_undegraded":"2024-08-17T17:48:20.192023+0000","last_fullsized":"2024-08-17T17:48:20.192023+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:40:45.448022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1a","version":"90'149","reported_seq":475,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.319954+0000","last_change":"2024-08-17T17:42:21.932830+0000","last_active":"2024-08-17T17:48:20.319954+0000","last_peered":"2024-08-17T17:48:20.319954+0000","last_clean":"2024-08-17T17:48:20.319954+0000","last_became_active":"2024-08-17T17:42:05.876835+0000","last_became_peered":"2024-08-17T17:42:05.876835+0000","last_unstale":"2024-08-17T17:48:20.319954+0000","last_undegraded":"2024-08-17T17:48:20.319954+0000","last_fullsized":"2024-08-17T17:48:20.319954+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:21.932757+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:21.932757+0000","last_clean_scrub_stamp":"2024-08-17T17:42:21.932757+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:36:39.848570+0000","scrub_duration":0.0020218810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":742,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.775608+0000","last_change":"2024-08-17T17:33:20.790687+0000","last_active":"2024-08-17T17:48:24.775608+0000","last_peered":"2024-08-17T17:48:24.775608+0000","last_clean":"2024-08-17T17:48:24.775608+0000","last_became_active":"2024-08-17T17:33:20.790399+0000","last_became_peered":"2024-08-17T17:33:20.790399+0000","last_unstale":"2024-08-17T17:48:24.775608+0000","last_undegraded":"2024-08-17T17:48:24.775608+0000","last_fullsized":"2024-08-17T17:48:24.775608+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:38:06.849989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":711,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.777350+0000","last_change":"2024-08-17T17:17:32.493459+0000","last_active":"2024-08-17T17:48:24.777350+0000","last_peered":"2024-08-17T17:48:24.777350+0000","last_clean":"2024-08-17T17:48:24.777350+0000","last_became_active":"2024-08-17T17:17:32.493127+0000","last_became_peered":"2024-08-17T17:17:32.493127+0000","last_unstale":"2024-08-17T17:48:24.777350+0000","last_undegraded":"2024-08-17T17:48:24.777350+0000","last_fullsized":"2024-08-17T17:48:24.777350+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:27.105623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"90'149","reported_seq":813,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.189450+0000","last_change":"2024-08-17T17:42:06.456562+0000","last_active":"2024-08-17T17:48:20.189450+0000","last_peered":"2024-08-17T17:48:20.189450+0000","last_clean":"2024-08-17T17:48:20.189450+0000","last_became_active":"2024-08-17T17:42:05.876577+0000","last_became_peered":"2024-08-17T17:42:05.876577+0000","last_unstale":"2024-08-17T17:48:20.189450+0000","last_undegraded":"2024-08-17T17:48:20.189450+0000","last_fullsized":"2024-08-17T17:48:20.189450+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:06.456490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:06.456490+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:10:03.654953+0000","scrub_duration":0.001537285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":684,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.775682+0000","last_change":"2024-08-17T17:19:57.502569+0000","last_active":"2024-08-17T17:48:24.775682+0000","last_peered":"2024-08-17T17:48:24.775682+0000","last_clean":"2024-08-17T17:48:24.775682+0000","last_became_active":"2024-08-17T17:19:57.502239+0000","last_became_peered":"2024-08-17T17:19:57.502239+0000","last_unstale":"2024-08-17T17:48:24.775682+0000","last_undegraded":"2024-08-17T17:48:24.775682+0000","last_fullsized":"2024-08-17T17:48:24.775682+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:08:28.378071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":123,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758338+0000","last_change":"2024-08-17T17:48:04.599622+0000","last_active":"2024-08-17T17:48:20.758338+0000","last_peered":"2024-08-17T17:48:20.758338+0000","last_clean":"2024-08-17T17:48:20.758338+0000","last_became_active":"2024-08-17T17:48:04.599289+0000","last_became_peered":"2024-08-17T17:48:04.599289+0000","last_unstale":"2024-08-17T17:48:20.758338+0000","last_undegraded":"2024-08-17T17:48:20.758338+0000","last_fullsized":"2024-08-17T17:48:20.758338+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:27:06.331364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"90'149","reported_seq":906,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776419+0000","last_change":"2024-08-17T17:41:37.664471+0000","last_active":"2024-08-17T17:48:24.776419+0000","last_peered":"2024-08-17T17:48:24.776419+0000","last_clean":"2024-08-17T17:48:24.776419+0000","last_became_active":"2024-08-17T17:41:37.664108+0000","last_became_peered":"2024-08-17T17:41:37.664108+0000","last_unstale":"2024-08-17T17:48:24.776419+0000","last_undegraded":"2024-08-17T17:48:24.776419+0000","last_fullsized":"2024-08-17T17:48:24.776419+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:22.030514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:22.030514+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:23:19.853098+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"71'3","reported_seq":170,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.189085+0000","last_change":"2024-08-17T17:48:18.181507+0000","last_active":"2024-08-17T17:48:20.189085+0000","last_peered":"2024-08-17T17:48:20.189085+0000","last_clean":"2024-08-17T17:48:20.189085+0000","last_became_active":"2024-08-17T17:48:18.180017+0000","last_became_peered":"2024-08-17T17:48:18.180017+0000","last_unstale":"2024-08-17T17:48:20.189085+0000","last_undegraded":"2024-08-17T17:48:20.189085+0000","last_fullsized":"2024-08-17T17:48:20.189085+0000","mapping_epoch":424,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:57:57.548524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":680,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.526612+0000","last_change":"2024-08-17T17:33:20.799402+0000","last_active":"2024-08-17T17:48:23.526612+0000","last_peered":"2024-08-17T17:48:23.526612+0000","last_clean":"2024-08-17T17:48:23.526612+0000","last_became_active":"2024-08-17T17:33:20.798832+0000","last_became_peered":"2024-08-17T17:33:20.798832+0000","last_unstale":"2024-08-17T17:48:23.526612+0000","last_undegraded":"2024-08-17T17:48:23.526612+0000","last_fullsized":"2024-08-17T17:48:23.526612+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:21:05.001134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.17","version":"90'149","reported_seq":411,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758433+0000","last_change":"2024-08-17T17:48:12.784018+0000","last_active":"2024-08-17T17:48:20.758433+0000","last_peered":"2024-08-17T17:48:20.758433+0000","last_clean":"2024-08-17T17:48:20.758433+0000","last_became_active":"2024-08-17T17:48:04.599356+0000","last_became_peered":"2024-08-17T17:48:04.599356+0000","last_unstale":"2024-08-17T17:48:20.758433+0000","last_undegraded":"2024-08-17T17:48:20.758433+0000","last_fullsized":"2024-08-17T17:48:20.758433+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:48:12.783940+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:06.698998+0000","last_clean_scrub_stamp":"2024-08-17T17:48:12.783940+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:14:33.197091+0000","scrub_duration":0.0015484699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"90'22","reported_seq":101,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.320680+0000","last_change":"2024-08-17T17:41:34.626584+0000","last_active":"2024-08-17T17:48:20.320680+0000","last_peered":"2024-08-17T17:48:20.320680+0000","last_clean":"2024-08-17T17:48:20.320680+0000","last_became_active":"2024-08-17T17:41:34.609878+0000","last_became_peered":"2024-08-17T17:41:34.609878+0000","last_unstale":"2024-08-17T17:48:20.320680+0000","last_undegraded":"2024-08-17T17:48:20.320680+0000","last_fullsized":"2024-08-17T17:48:20.320680+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:08:12.810347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":91,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.320530+0000","last_change":"2024-08-17T17:41:34.622180+0000","last_active":"2024-08-17T17:48:20.320530+0000","last_peered":"2024-08-17T17:48:20.320530+0000","last_clean":"2024-08-17T17:48:20.320530+0000","last_became_active":"2024-08-17T17:41:34.621724+0000","last_became_peered":"2024-08-17T17:41:34.621724+0000","last_unstale":"2024-08-17T17:48:20.320530+0000","last_undegraded":"2024-08-17T17:48:20.320530+0000","last_fullsized":"2024-08-17T17:48:20.320530+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:15.807235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.16","version":"90'149","reported_seq":793,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.526878+0000","last_change":"2024-08-17T17:42:07.681204+0000","last_active":"2024-08-17T17:48:23.526878+0000","last_peered":"2024-08-17T17:48:23.526878+0000","last_clean":"2024-08-17T17:48:23.526878+0000","last_became_active":"2024-08-17T17:42:05.878769+0000","last_became_peered":"2024-08-17T17:42:05.878769+0000","last_unstale":"2024-08-17T17:48:23.526878+0000","last_undegraded":"2024-08-17T17:48:23.526878+0000","last_fullsized":"2024-08-17T17:48:23.526878+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":340,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:07.681064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:07.681064+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:50:29.541934+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":91,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.320030+0000","last_change":"2024-08-17T17:41:34.614028+0000","last_active":"2024-08-17T17:48:20.320030+0000","last_peered":"2024-08-17T17:48:20.320030+0000","last_clean":"2024-08-17T17:48:20.320030+0000","last_became_active":"2024-08-17T17:41:34.613706+0000","last_became_peered":"2024-08-17T17:41:34.613706+0000","last_unstale":"2024-08-17T17:48:20.320030+0000","last_undegraded":"2024-08-17T17:48:20.320030+0000","last_fullsized":"2024-08-17T17:48:20.320030+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:19:55.254902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"86'48","reported_seq":657,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.192029+0000","last_change":"2024-08-17T17:26:06.855307+0000","last_active":"2024-08-17T17:48:20.192029+0000","last_peered":"2024-08-17T17:48:20.192029+0000","last_clean":"2024-08-17T17:48:20.192029+0000","last_became_active":"2024-08-17T17:25:23.358637+0000","last_became_peered":"2024-08-17T17:25:23.358637+0000","last_unstale":"2024-08-17T17:48:20.192029+0000","last_undegraded":"2024-08-17T17:48:20.192029+0000","last_fullsized":"2024-08-17T17:48:20.192029+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:24:33.988205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000299392,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":294,"num_read_kb":262,"num_write":26,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"71'1","reported_seq":575,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.777081+0000","last_change":"2024-08-17T17:41:34.629522+0000","last_active":"2024-08-17T17:48:24.777081+0000","last_peered":"2024-08-17T17:48:24.777081+0000","last_clean":"2024-08-17T17:48:24.777081+0000","last_became_active":"2024-08-17T17:41:34.619340+0000","last_became_peered":"2024-08-17T17:41:34.619340+0000","last_unstale":"2024-08-17T17:48:24.777081+0000","last_undegraded":"2024-08-17T17:48:24.777081+0000","last_fullsized":"2024-08-17T17:48:24.777081+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:30:03.902956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"90'149","reported_seq":883,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776387+0000","last_change":"2024-08-17T17:38:23.113397+0000","last_active":"2024-08-17T17:48:24.776387+0000","last_peered":"2024-08-17T17:48:24.776387+0000","last_clean":"2024-08-17T17:48:24.776387+0000","last_became_active":"2024-08-17T17:38:21.305580+0000","last_became_peered":"2024-08-17T17:38:21.305580+0000","last_unstale":"2024-08-17T17:48:24.776387+0000","last_undegraded":"2024-08-17T17:48:24.776387+0000","last_fullsized":"2024-08-17T17:48:24.776387+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.113279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.113279+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:54:14.909136+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":735,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.775677+0000","last_change":"2024-08-17T17:24:06.586304+0000","last_active":"2024-08-17T17:48:24.775677+0000","last_peered":"2024-08-17T17:48:24.775677+0000","last_clean":"2024-08-17T17:48:24.775677+0000","last_became_active":"2024-08-17T17:24:06.585996+0000","last_became_peered":"2024-08-17T17:24:06.585996+0000","last_unstale":"2024-08-17T17:48:24.775677+0000","last_undegraded":"2024-08-17T17:48:24.775677+0000","last_fullsized":"2024-08-17T17:48:24.775677+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:17:59.302820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'222","reported_seq":1313,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.527074+0000","last_change":"2024-08-17T17:41:37.677524+0000","last_active":"2024-08-17T17:48:23.527074+0000","last_peered":"2024-08-17T17:48:23.527074+0000","last_clean":"2024-08-17T17:48:23.527074+0000","last_became_active":"2024-08-17T17:41:37.676764+0000","last_became_peered":"2024-08-17T17:41:37.676764+0000","last_unstale":"2024-08-17T17:48:23.527074+0000","last_undegraded":"2024-08-17T17:48:23.527074+0000","last_fullsized":"2024-08-17T17:48:23.527074+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:14:49.341080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00025919400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":841,"num_read_kb":2413,"num_write":218,"num_write_kb":1514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"71'2","reported_seq":738,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776655+0000","last_change":"2024-08-17T17:19:57.518866+0000","last_active":"2024-08-17T17:48:24.776655+0000","last_peered":"2024-08-17T17:48:24.776655+0000","last_clean":"2024-08-17T17:48:24.776655+0000","last_became_active":"2024-08-17T17:19:57.507298+0000","last_became_peered":"2024-08-17T17:19:57.507298+0000","last_unstale":"2024-08-17T17:48:24.776655+0000","last_undegraded":"2024-08-17T17:48:24.776655+0000","last_fullsized":"2024-08-17T17:48:24.776655+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:44:39.267990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"90'149","reported_seq":810,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.320848+0000","last_change":"2024-08-17T17:42:19.960161+0000","last_active":"2024-08-17T17:48:20.320848+0000","last_peered":"2024-08-17T17:48:20.320848+0000","last_clean":"2024-08-17T17:48:20.320848+0000","last_became_active":"2024-08-17T17:41:37.665223+0000","last_became_peered":"2024-08-17T17:41:37.665223+0000","last_unstale":"2024-08-17T17:48:20.320848+0000","last_undegraded":"2024-08-17T17:48:20.320848+0000","last_fullsized":"2024-08-17T17:48:20.320848+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:19.960086+0000","last_deep_scrub":"90'149","last_deep_scrub_stamp":"2024-08-17T17:42:19.960086+0000","last_clean_scrub_stamp":"2024-08-17T17:42:19.960086+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:57:57.861661+0000","scrub_duration":0.0022612000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":703,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776915+0000","last_change":"2024-08-17T17:15:54.357159+0000","last_active":"2024-08-17T17:48:24.776915+0000","last_peered":"2024-08-17T17:48:24.776915+0000","last_clean":"2024-08-17T17:48:24.776915+0000","last_became_active":"2024-08-17T17:15:54.356528+0000","last_became_peered":"2024-08-17T17:15:54.356528+0000","last_unstale":"2024-08-17T17:48:24.776915+0000","last_undegraded":"2024-08-17T17:48:24.776915+0000","last_fullsized":"2024-08-17T17:48:24.776915+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:04:18.164043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"86'110","reported_seq":987,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.189461+0000","last_change":"2024-08-17T17:41:37.683212+0000","last_active":"2024-08-17T17:48:20.189461+0000","last_peered":"2024-08-17T17:48:20.189461+0000","last_clean":"2024-08-17T17:48:20.189461+0000","last_became_active":"2024-08-17T17:41:37.682730+0000","last_became_peered":"2024-08-17T17:41:37.682730+0000","last_unstale":"2024-08-17T17:48:20.189461+0000","last_undegraded":"2024-08-17T17:48:20.189461+0000","last_fullsized":"2024-08-17T17:48:20.189461+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:51:35.444433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029430500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":518,"num_read_kb":1437,"num_write":99,"num_write_kb":1055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":688,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.526726+0000","last_change":"2024-08-17T17:33:20.798613+0000","last_active":"2024-08-17T17:48:23.526726+0000","last_peered":"2024-08-17T17:48:23.526726+0000","last_clean":"2024-08-17T17:48:23.526726+0000","last_became_active":"2024-08-17T17:33:20.798260+0000","last_became_peered":"2024-08-17T17:33:20.798260+0000","last_unstale":"2024-08-17T17:48:23.526726+0000","last_undegraded":"2024-08-17T17:48:23.526726+0000","last_fullsized":"2024-08-17T17:48:23.526726+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:18:00.822681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"90'149","reported_seq":564,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.191868+0000","last_change":"2024-08-17T17:41:44.058291+0000","last_active":"2024-08-17T17:48:20.191868+0000","last_peered":"2024-08-17T17:48:20.191868+0000","last_clean":"2024-08-17T17:48:20.191868+0000","last_became_active":"2024-08-17T17:41:12.200077+0000","last_became_peered":"2024-08-17T17:41:12.200077+0000","last_unstale":"2024-08-17T17:48:20.191868+0000","last_undegraded":"2024-08-17T17:48:20.191868+0000","last_fullsized":"2024-08-17T17:48:20.191868+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:44.058233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:44.058233+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:42:55.911623+0000","scrub_duration":0.0074268499999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":684,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.527020+0000","last_change":"2024-08-17T17:41:34.627064+0000","last_active":"2024-08-17T17:48:23.527020+0000","last_peered":"2024-08-17T17:48:23.527020+0000","last_clean":"2024-08-17T17:48:23.527020+0000","last_became_active":"2024-08-17T17:41:34.626453+0000","last_became_peered":"2024-08-17T17:41:34.626453+0000","last_unstale":"2024-08-17T17:48:23.527020+0000","last_undegraded":"2024-08-17T17:48:23.527020+0000","last_fullsized":"2024-08-17T17:48:23.527020+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:02:11.894042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"86'63","reported_seq":868,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.526848+0000","last_change":"2024-08-17T17:33:20.799350+0000","last_active":"2024-08-17T17:48:23.526848+0000","last_peered":"2024-08-17T17:48:23.526848+0000","last_clean":"2024-08-17T17:48:23.526848+0000","last_became_active":"2024-08-17T17:33:20.798925+0000","last_became_peered":"2024-08-17T17:33:20.798925+0000","last_unstale":"2024-08-17T17:48:23.526848+0000","last_undegraded":"2024-08-17T17:48:23.526848+0000","last_fullsized":"2024-08-17T17:48:23.526848+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:16:36.121295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034783299999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":1365,"num_write":76,"num_write_kb":1148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":265,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.759145+0000","last_change":"2024-08-17T17:48:04.594463+0000","last_active":"2024-08-17T17:48:20.759145+0000","last_peered":"2024-08-17T17:48:20.759145+0000","last_clean":"2024-08-17T17:48:20.759145+0000","last_became_active":"2024-08-17T17:48:04.594035+0000","last_became_peered":"2024-08-17T17:48:04.594035+0000","last_unstale":"2024-08-17T17:48:20.759145+0000","last_undegraded":"2024-08-17T17:48:20.759145+0000","last_fullsized":"2024-08-17T17:48:20.759145+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:15:24.348195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"90'149","reported_seq":564,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.191903+0000","last_change":"2024-08-17T17:41:43.035215+0000","last_active":"2024-08-17T17:48:20.191903+0000","last_peered":"2024-08-17T17:48:20.191903+0000","last_clean":"2024-08-17T17:48:20.191903+0000","last_became_active":"2024-08-17T17:41:12.201158+0000","last_became_peered":"2024-08-17T17:41:12.201158+0000","last_unstale":"2024-08-17T17:48:20.191903+0000","last_undegraded":"2024-08-17T17:48:20.191903+0000","last_fullsized":"2024-08-17T17:48:20.191903+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:43.035108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:43.035108+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:43:27.108994+0000","scrub_duration":0.0028877400000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":653,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.527153+0000","last_change":"2024-08-17T17:33:20.772454+0000","last_active":"2024-08-17T17:48:23.527153+0000","last_peered":"2024-08-17T17:48:23.527153+0000","last_clean":"2024-08-17T17:48:23.527153+0000","last_became_active":"2024-08-17T17:33:20.771848+0000","last_became_peered":"2024-08-17T17:33:20.771848+0000","last_unstale":"2024-08-17T17:48:23.527153+0000","last_undegraded":"2024-08-17T17:48:23.527153+0000","last_fullsized":"2024-08-17T17:48:23.527153+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:46:42.300744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":91,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.320549+0000","last_change":"2024-08-17T17:41:34.611109+0000","last_active":"2024-08-17T17:48:20.320549+0000","last_peered":"2024-08-17T17:48:20.320549+0000","last_clean":"2024-08-17T17:48:20.320549+0000","last_became_active":"2024-08-17T17:41:34.610170+0000","last_became_peered":"2024-08-17T17:41:34.610170+0000","last_unstale":"2024-08-17T17:48:20.320549+0000","last_undegraded":"2024-08-17T17:48:20.320549+0000","last_fullsized":"2024-08-17T17:48:20.320549+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:39:05.301260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"86'128","reported_seq":1217,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758477+0000","last_change":"2024-08-17T17:48:04.601159+0000","last_active":"2024-08-17T17:48:20.758477+0000","last_peered":"2024-08-17T17:48:20.758477+0000","last_clean":"2024-08-17T17:48:20.758477+0000","last_became_active":"2024-08-17T17:48:04.600380+0000","last_became_peered":"2024-08-17T17:48:04.600380+0000","last_unstale":"2024-08-17T17:48:20.758477+0000","last_undegraded":"2024-08-17T17:48:20.758477+0000","last_fullsized":"2024-08-17T17:48:20.758477+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:12:25.955259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000749207,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1392,"num_read_kb":2863,"num_write":82,"num_write_kb":1345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"90'149","reported_seq":650,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758548+0000","last_change":"2024-08-17T17:48:04.898296+0000","last_active":"2024-08-17T17:48:20.758548+0000","last_peered":"2024-08-17T17:48:20.758548+0000","last_clean":"2024-08-17T17:48:20.758548+0000","last_became_active":"2024-08-17T17:48:04.595809+0000","last_became_peered":"2024-08-17T17:48:04.595809+0000","last_unstale":"2024-08-17T17:48:20.758548+0000","last_undegraded":"2024-08-17T17:48:20.758548+0000","last_fullsized":"2024-08-17T17:48:20.758548+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:48:04.898228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:48:04.898228+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:36:24.166969+0000","scrub_duration":0.0019622400000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":649,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.526929+0000","last_change":"2024-08-17T17:33:20.835784+0000","last_active":"2024-08-17T17:48:23.526929+0000","last_peered":"2024-08-17T17:48:23.526929+0000","last_clean":"2024-08-17T17:48:23.526929+0000","last_became_active":"2024-08-17T17:33:20.835371+0000","last_became_peered":"2024-08-17T17:33:20.835371+0000","last_unstale":"2024-08-17T17:48:23.526929+0000","last_undegraded":"2024-08-17T17:48:23.526929+0000","last_fullsized":"2024-08-17T17:48:23.526929+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:35:43.369055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":613,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.526469+0000","last_change":"2024-08-17T17:33:20.793468+0000","last_active":"2024-08-17T17:48:23.526469+0000","last_peered":"2024-08-17T17:48:23.526469+0000","last_clean":"2024-08-17T17:48:23.526469+0000","last_became_active":"2024-08-17T17:33:20.793166+0000","last_became_peered":"2024-08-17T17:33:20.793166+0000","last_unstale":"2024-08-17T17:48:23.526469+0000","last_undegraded":"2024-08-17T17:48:23.526469+0000","last_fullsized":"2024-08-17T17:48:23.526469+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:12:52.121050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"86'97","reported_seq":643,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776391+0000","last_change":"2024-08-17T17:47:45.055879+0000","last_active":"2024-08-17T17:48:24.776391+0000","last_peered":"2024-08-17T17:48:24.776391+0000","last_clean":"2024-08-17T17:48:24.776391+0000","last_became_active":"2024-08-17T17:47:45.055199+0000","last_became_peered":"2024-08-17T17:47:45.055199+0000","last_unstale":"2024-08-17T17:48:24.776391+0000","last_undegraded":"2024-08-17T17:48:24.776391+0000","last_fullsized":"2024-08-17T17:48:24.776391+0000","mapping_epoch":412,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:29:40.639457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00027768500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":784,"num_read_kb":2677,"num_write":73,"num_write_kb":2096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"90'149","reported_seq":605,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.188677+0000","last_change":"2024-08-17T17:48:19.388610+0000","last_active":"2024-08-17T17:48:20.188677+0000","last_peered":"2024-08-17T17:48:20.188677+0000","last_clean":"2024-08-17T17:48:20.188677+0000","last_became_active":"2024-08-17T17:48:18.179425+0000","last_became_peered":"2024-08-17T17:48:18.179425+0000","last_unstale":"2024-08-17T17:48:20.188677+0000","last_undegraded":"2024-08-17T17:48:20.188677+0000","last_fullsized":"2024-08-17T17:48:20.188677+0000","mapping_epoch":424,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:48:19.388542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:48:19.388542+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:33:06.359371+0000","scrub_duration":0.0019005300000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":529,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.777181+0000","last_change":"2024-08-17T17:24:06.584594+0000","last_active":"2024-08-17T17:48:24.777181+0000","last_peered":"2024-08-17T17:48:24.777181+0000","last_clean":"2024-08-17T17:48:24.777181+0000","last_became_active":"2024-08-17T17:24:06.582443+0000","last_became_peered":"2024-08-17T17:24:06.582443+0000","last_unstale":"2024-08-17T17:48:24.777181+0000","last_undegraded":"2024-08-17T17:48:24.777181+0000","last_fullsized":"2024-08-17T17:48:24.777181+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:45:17.670801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":742,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.775944+0000","last_change":"2024-08-17T17:41:34.614108+0000","last_active":"2024-08-17T17:48:24.775944+0000","last_peered":"2024-08-17T17:48:24.775944+0000","last_clean":"2024-08-17T17:48:24.775944+0000","last_became_active":"2024-08-17T17:41:34.613481+0000","last_became_peered":"2024-08-17T17:41:34.613481+0000","last_unstale":"2024-08-17T17:48:24.775944+0000","last_undegraded":"2024-08-17T17:48:24.775944+0000","last_fullsized":"2024-08-17T17:48:24.775944+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:21:02.346115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"86'122","reported_seq":865,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.188828+0000","last_change":"2024-08-17T17:48:18.181301+0000","last_active":"2024-08-17T17:48:20.188828+0000","last_peered":"2024-08-17T17:48:20.188828+0000","last_clean":"2024-08-17T17:48:20.188828+0000","last_became_active":"2024-08-17T17:48:18.180765+0000","last_became_peered":"2024-08-17T17:48:18.180765+0000","last_unstale":"2024-08-17T17:48:20.188828+0000","last_undegraded":"2024-08-17T17:48:20.188828+0000","last_fullsized":"2024-08-17T17:48:20.188828+0000","mapping_epoch":424,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:30:41.676392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001233327,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1184,"num_read_kb":1306,"num_write":94,"num_write_kb":1200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"90'149","reported_seq":918,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.527243+0000","last_change":"2024-08-17T17:41:37.676347+0000","last_active":"2024-08-17T17:48:23.527243+0000","last_peered":"2024-08-17T17:48:23.527243+0000","last_clean":"2024-08-17T17:48:23.527243+0000","last_became_active":"2024-08-17T17:41:37.675869+0000","last_became_peered":"2024-08-17T17:41:37.675869+0000","last_unstale":"2024-08-17T17:48:23.527243+0000","last_undegraded":"2024-08-17T17:48:23.527243+0000","last_fullsized":"2024-08-17T17:48:23.527243+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":3,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:21.814077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:21.814077+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:52:23.492179+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":268,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.191540+0000","last_change":"2024-08-17T17:41:34.618631+0000","last_active":"2024-08-17T17:48:20.191540+0000","last_peered":"2024-08-17T17:48:20.191540+0000","last_clean":"2024-08-17T17:48:20.191540+0000","last_became_active":"2024-08-17T17:41:34.618313+0000","last_became_peered":"2024-08-17T17:41:34.618313+0000","last_unstale":"2024-08-17T17:48:20.191540+0000","last_undegraded":"2024-08-17T17:48:20.191540+0000","last_fullsized":"2024-08-17T17:48:20.191540+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:07:13.915088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":692,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776298+0000","last_change":"2024-08-17T17:24:53.459214+0000","last_active":"2024-08-17T17:48:24.776298+0000","last_peered":"2024-08-17T17:48:24.776298+0000","last_clean":"2024-08-17T17:48:24.776298+0000","last_became_active":"2024-08-17T17:24:53.458769+0000","last_became_peered":"2024-08-17T17:24:53.458769+0000","last_unstale":"2024-08-17T17:48:24.776298+0000","last_undegraded":"2024-08-17T17:48:24.776298+0000","last_fullsized":"2024-08-17T17:48:24.776298+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:34:18.926711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'187","reported_seq":862,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758204+0000","last_change":"2024-08-17T17:48:04.600544+0000","last_active":"2024-08-17T17:48:20.758204+0000","last_peered":"2024-08-17T17:48:20.758204+0000","last_clean":"2024-08-17T17:48:20.758204+0000","last_became_active":"2024-08-17T17:48:04.600007+0000","last_became_peered":"2024-08-17T17:48:04.600007+0000","last_unstale":"2024-08-17T17:48:20.758204+0000","last_undegraded":"2024-08-17T17:48:20.758204+0000","last_fullsized":"2024-08-17T17:48:20.758204+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:06:28.762028+0000","last_clean_scrub_stamp":"2024-08-17T17:06:28.762028+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:00:07.477190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048540399999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":788,"num_read_kb":688,"num_write":178,"num_write_kb":123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"90'149","reported_seq":605,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.454414+0000","last_change":"2024-08-17T17:48:23.454326+0000","last_active":"2024-08-17T17:48:23.454414+0000","last_peered":"2024-08-17T17:48:23.454414+0000","last_clean":"2024-08-17T17:48:23.454414+0000","last_became_active":"2024-08-17T17:48:18.181315+0000","last_became_peered":"2024-08-17T17:48:18.181315+0000","last_unstale":"2024-08-17T17:48:23.454414+0000","last_undegraded":"2024-08-17T17:48:23.454414+0000","last_fullsized":"2024-08-17T17:48:23.454414+0000","mapping_epoch":424,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:48:23.454246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:48:23.454246+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:41:46.088324+0000","scrub_duration":0.0017975840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":639,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.526960+0000","last_change":"2024-08-17T17:41:34.625551+0000","last_active":"2024-08-17T17:48:23.526960+0000","last_peered":"2024-08-17T17:48:23.526960+0000","last_clean":"2024-08-17T17:48:23.526960+0000","last_became_active":"2024-08-17T17:41:34.625251+0000","last_became_peered":"2024-08-17T17:41:34.625251+0000","last_unstale":"2024-08-17T17:48:23.526960+0000","last_undegraded":"2024-08-17T17:48:23.526960+0000","last_fullsized":"2024-08-17T17:48:23.526960+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:28:22.354213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":744,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.775849+0000","last_change":"2024-08-17T17:33:20.796216+0000","last_active":"2024-08-17T17:48:24.775849+0000","last_peered":"2024-08-17T17:48:24.775849+0000","last_clean":"2024-08-17T17:48:24.775849+0000","last_became_active":"2024-08-17T17:33:20.795762+0000","last_became_peered":"2024-08-17T17:33:20.795762+0000","last_unstale":"2024-08-17T17:48:24.775849+0000","last_undegraded":"2024-08-17T17:48:24.775849+0000","last_fullsized":"2024-08-17T17:48:24.775849+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:54:47.038509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"90'149","reported_seq":408,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758473+0000","last_change":"2024-08-17T17:48:07.831433+0000","last_active":"2024-08-17T17:48:20.758473+0000","last_peered":"2024-08-17T17:48:20.758473+0000","last_clean":"2024-08-17T17:48:20.758473+0000","last_became_active":"2024-08-17T17:48:04.600291+0000","last_became_peered":"2024-08-17T17:48:04.600291+0000","last_unstale":"2024-08-17T17:48:20.758473+0000","last_undegraded":"2024-08-17T17:48:20.758473+0000","last_fullsized":"2024-08-17T17:48:20.758473+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:48:07.831349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:48:07.831349+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:58:27.937907+0000","scrub_duration":0.002146724,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":91,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.320920+0000","last_change":"2024-08-17T17:41:34.622102+0000","last_active":"2024-08-17T17:48:20.320920+0000","last_peered":"2024-08-17T17:48:20.320920+0000","last_clean":"2024-08-17T17:48:20.320920+0000","last_became_active":"2024-08-17T17:41:34.621733+0000","last_became_peered":"2024-08-17T17:41:34.621733+0000","last_unstale":"2024-08-17T17:48:20.320920+0000","last_undegraded":"2024-08-17T17:48:20.320920+0000","last_fullsized":"2024-08-17T17:48:20.320920+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:39:41.668709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":91,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.320180+0000","last_change":"2024-08-17T17:41:34.607602+0000","last_active":"2024-08-17T17:48:20.320180+0000","last_peered":"2024-08-17T17:48:20.320180+0000","last_clean":"2024-08-17T17:48:20.320180+0000","last_became_active":"2024-08-17T17:41:34.607300+0000","last_became_peered":"2024-08-17T17:41:34.607300+0000","last_unstale":"2024-08-17T17:48:20.320180+0000","last_undegraded":"2024-08-17T17:48:20.320180+0000","last_fullsized":"2024-08-17T17:48:20.320180+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:20:31.486698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"90'149","reported_seq":935,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.527074+0000","last_change":"2024-08-17T17:41:41.743931+0000","last_active":"2024-08-17T17:48:23.527074+0000","last_peered":"2024-08-17T17:48:23.527074+0000","last_clean":"2024-08-17T17:48:23.527074+0000","last_became_active":"2024-08-17T17:41:12.203771+0000","last_became_peered":"2024-08-17T17:41:12.203771+0000","last_unstale":"2024-08-17T17:48:23.527074+0000","last_undegraded":"2024-08-17T17:48:23.527074+0000","last_fullsized":"2024-08-17T17:48:23.527074+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:41.743570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:41.743570+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T05:13:18.592919+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":91,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.321072+0000","last_change":"2024-08-17T17:41:34.619268+0000","last_active":"2024-08-17T17:48:20.321072+0000","last_peered":"2024-08-17T17:48:20.321072+0000","last_clean":"2024-08-17T17:48:20.321072+0000","last_became_active":"2024-08-17T17:41:34.618914+0000","last_became_peered":"2024-08-17T17:41:34.618914+0000","last_unstale":"2024-08-17T17:48:20.321072+0000","last_undegraded":"2024-08-17T17:48:20.321072+0000","last_fullsized":"2024-08-17T17:48:20.321072+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:08:54.248386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":273,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.777188+0000","last_change":"2024-08-17T17:47:45.150169+0000","last_active":"2024-08-17T17:48:24.777188+0000","last_peered":"2024-08-17T17:48:24.777188+0000","last_clean":"2024-08-17T17:48:24.777188+0000","last_became_active":"2024-08-17T17:47:45.149913+0000","last_became_peered":"2024-08-17T17:47:45.149913+0000","last_unstale":"2024-08-17T17:48:24.777188+0000","last_undegraded":"2024-08-17T17:48:24.777188+0000","last_fullsized":"2024-08-17T17:48:24.777188+0000","mapping_epoch":412,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":413,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T04:07:23.807428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"90'149","reported_seq":836,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758628+0000","last_change":"2024-08-17T17:48:04.591625+0000","last_active":"2024-08-17T17:48:20.758628+0000","last_peered":"2024-08-17T17:48:20.758628+0000","last_clean":"2024-08-17T17:48:20.758628+0000","last_became_active":"2024-08-17T17:48:04.591148+0000","last_became_peered":"2024-08-17T17:48:04.591148+0000","last_unstale":"2024-08-17T17:48:20.758628+0000","last_undegraded":"2024-08-17T17:48:20.758628+0000","last_fullsized":"2024-08-17T17:48:20.758628+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:47:47.300846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:47:47.300846+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:06:50.680849+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":91,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.320540+0000","last_change":"2024-08-17T17:41:34.611049+0000","last_active":"2024-08-17T17:48:20.320540+0000","last_peered":"2024-08-17T17:48:20.320540+0000","last_clean":"2024-08-17T17:48:20.320540+0000","last_became_active":"2024-08-17T17:41:34.610727+0000","last_became_peered":"2024-08-17T17:41:34.610727+0000","last_unstale":"2024-08-17T17:48:20.320540+0000","last_undegraded":"2024-08-17T17:48:20.320540+0000","last_fullsized":"2024-08-17T17:48:20.320540+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:04:05.477660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"71'2","reported_seq":786,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776062+0000","last_change":"2024-08-17T17:41:34.630434+0000","last_active":"2024-08-17T17:48:24.776062+0000","last_peered":"2024-08-17T17:48:24.776062+0000","last_clean":"2024-08-17T17:48:24.776062+0000","last_became_active":"2024-08-17T17:41:34.616406+0000","last_became_peered":"2024-08-17T17:41:34.616406+0000","last_unstale":"2024-08-17T17:48:24.776062+0000","last_undegraded":"2024-08-17T17:48:24.776062+0000","last_fullsized":"2024-08-17T17:48:24.776062+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:30:44.817203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"90'149","reported_seq":605,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:25.454719+0000","last_change":"2024-08-17T17:48:25.454679+0000","last_active":"2024-08-17T17:48:25.454719+0000","last_peered":"2024-08-17T17:48:25.454719+0000","last_clean":"2024-08-17T17:48:25.454719+0000","last_became_active":"2024-08-17T17:48:18.178322+0000","last_became_peered":"2024-08-17T17:48:18.178322+0000","last_unstale":"2024-08-17T17:48:25.454719+0000","last_undegraded":"2024-08-17T17:48:25.454719+0000","last_fullsized":"2024-08-17T17:48:25.454719+0000","mapping_epoch":424,"log_start":"0'0","ondisk_log_start":"0'0","created":277,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:48:25.454608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:48:25.454608+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:41:01.366455+0000","scrub_duration":0.001814502,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":750,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.777000+0000","last_change":"2024-08-17T17:41:34.615197+0000","last_active":"2024-08-17T17:48:24.777000+0000","last_peered":"2024-08-17T17:48:24.777000+0000","last_clean":"2024-08-17T17:48:24.777000+0000","last_became_active":"2024-08-17T17:41:34.614873+0000","last_became_peered":"2024-08-17T17:41:34.614873+0000","last_unstale":"2024-08-17T17:48:24.777000+0000","last_undegraded":"2024-08-17T17:48:24.777000+0000","last_fullsized":"2024-08-17T17:48:24.777000+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:21:57.391953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":248,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758234+0000","last_change":"2024-08-17T17:48:04.599957+0000","last_active":"2024-08-17T17:48:20.758234+0000","last_peered":"2024-08-17T17:48:20.758234+0000","last_clean":"2024-08-17T17:48:20.758234+0000","last_became_active":"2024-08-17T17:48:04.599605+0000","last_became_peered":"2024-08-17T17:48:04.599605+0000","last_unstale":"2024-08-17T17:48:20.758234+0000","last_undegraded":"2024-08-17T17:48:20.758234+0000","last_fullsized":"2024-08-17T17:48:20.758234+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:26:59.143293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"90'149","reported_seq":735,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.321065+0000","last_change":"2024-08-17T17:42:20.927007+0000","last_active":"2024-08-17T17:48:20.321065+0000","last_peered":"2024-08-17T17:48:20.321065+0000","last_clean":"2024-08-17T17:48:20.321065+0000","last_became_active":"2024-08-17T17:41:37.670914+0000","last_became_peered":"2024-08-17T17:41:37.670914+0000","last_unstale":"2024-08-17T17:48:20.321065+0000","last_undegraded":"2024-08-17T17:48:20.321065+0000","last_fullsized":"2024-08-17T17:48:20.321065+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:20.926924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:20.926924+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:57:44.156600+0000","scrub_duration":0.0018485109999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":255,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.191513+0000","last_change":"2024-08-17T17:33:20.795582+0000","last_active":"2024-08-17T17:48:20.191513+0000","last_peered":"2024-08-17T17:48:20.191513+0000","last_clean":"2024-08-17T17:48:20.191513+0000","last_became_active":"2024-08-17T17:33:20.795149+0000","last_became_peered":"2024-08-17T17:33:20.795149+0000","last_unstale":"2024-08-17T17:48:20.191513+0000","last_undegraded":"2024-08-17T17:48:20.191513+0000","last_fullsized":"2024-08-17T17:48:20.191513+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:21:14.020923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"71'1","reported_seq":238,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.189341+0000","last_change":"2024-08-17T17:48:18.182348+0000","last_active":"2024-08-17T17:48:20.189341+0000","last_peered":"2024-08-17T17:48:20.189341+0000","last_clean":"2024-08-17T17:48:20.189341+0000","last_became_active":"2024-08-17T17:48:18.181956+0000","last_became_peered":"2024-08-17T17:48:18.181956+0000","last_unstale":"2024-08-17T17:48:20.189341+0000","last_undegraded":"2024-08-17T17:48:20.189341+0000","last_fullsized":"2024-08-17T17:48:20.189341+0000","mapping_epoch":424,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T23:02:54.992931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"90'149","reported_seq":603,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.191934+0000","last_change":"2024-08-17T17:41:37.668050+0000","last_active":"2024-08-17T17:48:20.191934+0000","last_peered":"2024-08-17T17:48:20.191934+0000","last_clean":"2024-08-17T17:48:20.191934+0000","last_became_active":"2024-08-17T17:41:37.667655+0000","last_became_peered":"2024-08-17T17:41:37.667655+0000","last_unstale":"2024-08-17T17:48:20.191934+0000","last_undegraded":"2024-08-17T17:48:20.191934+0000","last_fullsized":"2024-08-17T17:48:20.191934+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:32.843802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:32.843802+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:39:46.766280+0000","scrub_duration":0.0040803669999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":91,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.320039+0000","last_change":"2024-08-17T17:41:34.621078+0000","last_active":"2024-08-17T17:48:20.320039+0000","last_peered":"2024-08-17T17:48:20.320039+0000","last_clean":"2024-08-17T17:48:20.320039+0000","last_became_active":"2024-08-17T17:41:34.620730+0000","last_became_peered":"2024-08-17T17:41:34.620730+0000","last_unstale":"2024-08-17T17:48:20.320039+0000","last_undegraded":"2024-08-17T17:48:20.320039+0000","last_fullsized":"2024-08-17T17:48:20.320039+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T03:55:01.312283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"71'2","reported_seq":384,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.189437+0000","last_change":"2024-08-17T17:41:34.637696+0000","last_active":"2024-08-17T17:48:20.189437+0000","last_peered":"2024-08-17T17:48:20.189437+0000","last_clean":"2024-08-17T17:48:20.189437+0000","last_became_active":"2024-08-17T17:41:34.621388+0000","last_became_peered":"2024-08-17T17:41:34.621388+0000","last_unstale":"2024-08-17T17:48:20.189437+0000","last_undegraded":"2024-08-17T17:48:20.189437+0000","last_fullsized":"2024-08-17T17:48:20.189437+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:13:04.607036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"90'149","reported_seq":927,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776490+0000","last_change":"2024-08-17T17:41:42.603841+0000","last_active":"2024-08-17T17:48:24.776490+0000","last_peered":"2024-08-17T17:48:24.776490+0000","last_clean":"2024-08-17T17:48:24.776490+0000","last_became_active":"2024-08-17T17:41:12.203400+0000","last_became_peered":"2024-08-17T17:41:12.203400+0000","last_unstale":"2024-08-17T17:48:24.776490+0000","last_undegraded":"2024-08-17T17:48:24.776490+0000","last_fullsized":"2024-08-17T17:48:24.776490+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:41:42.603561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:41:42.603561+0000","objects_scrubbed":1,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T20:43:50.209009+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":116,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758172+0000","last_change":"2024-08-17T17:48:04.600776+0000","last_active":"2024-08-17T17:48:20.758172+0000","last_peered":"2024-08-17T17:48:20.758172+0000","last_clean":"2024-08-17T17:48:20.758172+0000","last_became_active":"2024-08-17T17:48:04.600189+0000","last_became_peered":"2024-08-17T17:48:04.600189+0000","last_unstale":"2024-08-17T17:48:20.758172+0000","last_undegraded":"2024-08-17T17:48:20.758172+0000","last_fullsized":"2024-08-17T17:48:20.758172+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T22:49:29.973785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"71'1","reported_seq":388,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.189429+0000","last_change":"2024-08-17T17:41:34.635980+0000","last_active":"2024-08-17T17:48:20.189429+0000","last_peered":"2024-08-17T17:48:20.189429+0000","last_clean":"2024-08-17T17:48:20.189429+0000","last_became_active":"2024-08-17T17:41:34.620596+0000","last_became_peered":"2024-08-17T17:41:34.620596+0000","last_unstale":"2024-08-17T17:48:20.189429+0000","last_undegraded":"2024-08-17T17:48:20.189429+0000","last_fullsized":"2024-08-17T17:48:20.189429+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T01:19:15.792577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"90'149","reported_seq":837,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.188982+0000","last_change":"2024-08-17T17:42:16.610940+0000","last_active":"2024-08-17T17:48:20.188982+0000","last_peered":"2024-08-17T17:48:20.188982+0000","last_clean":"2024-08-17T17:48:20.188982+0000","last_became_active":"2024-08-17T17:41:37.670855+0000","last_became_peered":"2024-08-17T17:41:37.670855+0000","last_unstale":"2024-08-17T17:48:20.188982+0000","last_undegraded":"2024-08-17T17:48:20.188982+0000","last_fullsized":"2024-08-17T17:48:20.188982+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":4,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:42:16.610828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:42:16.610828+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:48:25.918610+0000","scrub_duration":0.086892487000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"71'8","reported_seq":512,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.191481+0000","last_change":"2024-08-17T17:43:52.520659+0000","last_active":"2024-08-17T17:48:20.191481+0000","last_peered":"2024-08-17T17:48:20.191481+0000","last_clean":"2024-08-17T17:48:20.191481+0000","last_became_active":"2024-08-17T17:43:52.516799+0000","last_became_peered":"2024-08-17T17:43:52.516799+0000","last_unstale":"2024-08-17T17:48:20.191481+0000","last_undegraded":"2024-08-17T17:48:20.191481+0000","last_fullsized":"2024-08-17T17:48:20.191481+0000","mapping_epoch":377,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:25:59.408383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":742,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776508+0000","last_change":"2024-08-17T17:33:20.794682+0000","last_active":"2024-08-17T17:48:24.776508+0000","last_peered":"2024-08-17T17:48:24.776508+0000","last_clean":"2024-08-17T17:48:24.776508+0000","last_became_active":"2024-08-17T17:33:20.794238+0000","last_became_peered":"2024-08-17T17:33:20.794238+0000","last_unstale":"2024-08-17T17:48:24.776508+0000","last_undegraded":"2024-08-17T17:48:24.776508+0000","last_fullsized":"2024-08-17T17:48:24.776508+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:15:55.821920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.12","version":"90'149","reported_seq":694,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758239+0000","last_change":"2024-08-17T17:48:04.601616+0000","last_active":"2024-08-17T17:48:20.758239+0000","last_peered":"2024-08-17T17:48:20.758239+0000","last_clean":"2024-08-17T17:48:20.758239+0000","last_became_active":"2024-08-17T17:48:04.601091+0000","last_became_peered":"2024-08-17T17:48:04.601091+0000","last_unstale":"2024-08-17T17:48:20.758239+0000","last_undegraded":"2024-08-17T17:48:20.758239+0000","last_fullsized":"2024-08-17T17:48:20.758239+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:24.291592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:24.291592+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:50:01.146205+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":219,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.188685+0000","last_change":"2024-08-17T17:48:18.181063+0000","last_active":"2024-08-17T17:48:20.188685+0000","last_peered":"2024-08-17T17:48:20.188685+0000","last_clean":"2024-08-17T17:48:20.188685+0000","last_became_active":"2024-08-17T17:48:18.180711+0000","last_became_peered":"2024-08-17T17:48:18.180711+0000","last_unstale":"2024-08-17T17:48:20.188685+0000","last_undegraded":"2024-08-17T17:48:20.188685+0000","last_fullsized":"2024-08-17T17:48:20.188685+0000","mapping_epoch":424,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:19:08.158472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":528,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776989+0000","last_change":"2024-08-17T17:24:06.587866+0000","last_active":"2024-08-17T17:48:24.776989+0000","last_peered":"2024-08-17T17:48:24.776989+0000","last_clean":"2024-08-17T17:48:24.776989+0000","last_became_active":"2024-08-17T17:24:06.586425+0000","last_became_peered":"2024-08-17T17:24:06.586425+0000","last_unstale":"2024-08-17T17:48:24.776989+0000","last_undegraded":"2024-08-17T17:48:24.776989+0000","last_fullsized":"2024-08-17T17:48:24.776989+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T18:30:12.908090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"90'149","reported_seq":926,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:23.526903+0000","last_change":"2024-08-17T17:41:37.681187+0000","last_active":"2024-08-17T17:48:23.526903+0000","last_peered":"2024-08-17T17:48:23.526903+0000","last_clean":"2024-08-17T17:48:23.526903+0000","last_became_active":"2024-08-17T17:41:37.680605+0000","last_became_peered":"2024-08-17T17:41:37.680605+0000","last_unstale":"2024-08-17T17:48:23.526903+0000","last_undegraded":"2024-08-17T17:48:23.526903+0000","last_fullsized":"2024-08-17T17:48:23.526903+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:23.844252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:23.844252+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:59:14.260864+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":528,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776144+0000","last_change":"2024-08-17T17:24:06.586616+0000","last_active":"2024-08-17T17:48:24.776144+0000","last_peered":"2024-08-17T17:48:24.776144+0000","last_clean":"2024-08-17T17:48:24.776144+0000","last_became_active":"2024-08-17T17:24:06.586302+0000","last_became_peered":"2024-08-17T17:24:06.586302+0000","last_unstale":"2024-08-17T17:48:24.776144+0000","last_undegraded":"2024-08-17T17:48:24.776144+0000","last_fullsized":"2024-08-17T17:48:24.776144+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T19:40:38.353948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":106,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758662+0000","last_change":"2024-08-17T17:48:04.597329+0000","last_active":"2024-08-17T17:48:20.758662+0000","last_peered":"2024-08-17T17:48:20.758662+0000","last_clean":"2024-08-17T17:48:20.758662+0000","last_became_active":"2024-08-17T17:48:04.596967+0000","last_became_peered":"2024-08-17T17:48:04.596967+0000","last_unstale":"2024-08-17T17:48:20.758662+0000","last_undegraded":"2024-08-17T17:48:20.758662+0000","last_fullsized":"2024-08-17T17:48:20.758662+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T00:52:52.097006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"90'149","reported_seq":408,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.758492+0000","last_change":"2024-08-17T17:48:14.754802+0000","last_active":"2024-08-17T17:48:20.758492+0000","last_peered":"2024-08-17T17:48:20.758492+0000","last_clean":"2024-08-17T17:48:20.758492+0000","last_became_active":"2024-08-17T17:48:04.600862+0000","last_became_peered":"2024-08-17T17:48:04.600862+0000","last_unstale":"2024-08-17T17:48:20.758492+0000","last_undegraded":"2024-08-17T17:48:20.758492+0000","last_fullsized":"2024-08-17T17:48:20.758492+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:48:14.754726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:48:14.754726+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:06:18.332300+0000","scrub_duration":0.0017611,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"71'2","reported_seq":451,"reported_epoch":426,"state":"active+clean","last_fresh":"2024-08-17T17:48:20.188872+0000","last_change":"2024-08-17T17:41:34.624198+0000","last_active":"2024-08-17T17:48:20.188872+0000","last_peered":"2024-08-17T17:48:20.188872+0000","last_clean":"2024-08-17T17:48:20.188872+0000","last_became_active":"2024-08-17T17:41:34.616071+0000","last_became_peered":"2024-08-17T17:41:34.616071+0000","last_unstale":"2024-08-17T17:48:20.188872+0000","last_undegraded":"2024-08-17T17:48:20.188872+0000","last_fullsized":"2024-08-17T17:48:20.188872+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:45.130032+0000","last_clean_scrub_stamp":"2024-08-17T17:07:45.130032+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T17:34:48.805354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":136,"num_keys_recovered":40,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":728,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776648+0000","last_change":"2024-08-17T17:33:20.797029+0000","last_active":"2024-08-17T17:48:24.776648+0000","last_peered":"2024-08-17T17:48:24.776648+0000","last_clean":"2024-08-17T17:48:24.776648+0000","last_became_active":"2024-08-17T17:33:20.796679+0000","last_became_peered":"2024-08-17T17:33:20.796679+0000","last_unstale":"2024-08-17T17:48:24.776648+0000","last_undegraded":"2024-08-17T17:48:24.776648+0000","last_fullsized":"2024-08-17T17:48:24.776648+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:07:46.135583+0000","last_clean_scrub_stamp":"2024-08-17T17:07:46.135583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-19T02:02:51.438262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"90'149","reported_seq":908,"reported_epoch":427,"state":"active+clean","last_fresh":"2024-08-17T17:48:24.776000+0000","last_change":"2024-08-17T17:38:30.378664+0000","last_active":"2024-08-17T17:48:24.776000+0000","last_peered":"2024-08-17T17:48:24.776000+0000","last_clean":"2024-08-17T17:48:24.776000+0000","last_became_active":"2024-08-17T17:38:30.342287+0000","last_became_peered":"2024-08-17T17:38:30.342287+0000","last_unstale":"2024-08-17T17:48:24.776000+0000","last_undegraded":"2024-08-17T17:48:24.776000+0000","last_fullsized":"2024-08-17T17:48:24.776000+0000","mapping_epoch":322,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":323,"parent":"0.0","parent_split_bits":5,"last_scrub":"90'149","last_scrub_stamp":"2024-08-17T17:38:30.378592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-17T17:04:01.141401+0000","last_clean_scrub_stamp":"2024-08-17T17:38:30.378592+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-18T21:14:05.215821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":11216,"num_keys_recovered":40,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19280,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":4194323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5998,"num_read_kb":13011,"num_write":846,"num_write_kb":12594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":8388646,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12595200,"data_stored":12582969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},"log_size":977,"ondisk_log_size":977,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4620,"ondisk_log_size":4620,"up":93,"acting":93,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234510,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":912148,"kb_used_data":9928,"kb_used_omap":1,"kb_used_meta":902206,"kb_avail":92833004,"statfs":{"total":95995035648,"available":95060996096,"internally_reserved":0,"allocated":10166272,"data_stored":7780293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":923859288},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568496145,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":900932,"kb_used_data":9976,"kb_used_omap":1,"kb_used_meta":890942,"kb_avail":92844220,"statfs":{"total":95995035648,"available":95072481280,"internally_reserved":0,"allocated":10215424,"data_stored":7844959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":912324952},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757782,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":888132,"kb_used_data":5816,"kb_used_omap":1,"kb_used_meta":882302,"kb_avail":92857020,"statfs":{"total":95995035648,"available":95085588480,"internally_reserved":0,"allocated":5955584,"data_stored":3581688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":903477592},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019420,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307088,"kb_used_data":5572,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93438064,"statfs":{"total":95995035648,"available":95680577536,"internally_reserved":0,"allocated":5705728,"data_stored":3323562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":109,"seq":468151435648,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":966012,"kb_used_data":5872,"kb_used_omap":4,"kb_used_meta":960123,"kb_avail":92779140,"statfs":{"total":95995035648,"available":95005839360,"internally_reserved":0,"allocated":6012928,"data_stored":3646462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5110,"internal_metadata":983165962},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":104,"seq":446676599174,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":845700,"kb_used_data":9912,"kb_used_omap":7,"kb_used_meta":835768,"kb_avail":92899452,"statfs":{"total":95995035648,"available":95129038848,"internally_reserved":0,"allocated":10149888,"data_stored":7776288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":855826594},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":268,"seq":1151051235521,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":455804,"kb_used_data":5552,"kb_used_omap":10,"kb_used_meta":450229,"kb_avail":93289348,"statfs":{"total":95995035648,"available":95528292352,"internally_reserved":0,"allocated":5685248,"data_stored":3318727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10902,"internal_metadata":461034858},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":94,"seq":403726926227,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":940352,"kb_used_data":5876,"kb_used_omap":7,"kb_used_meta":934456,"kb_avail":92804800,"statfs":{"total":95995035648,"available":95032115200,"internally_reserved":0,"allocated":6017024,"data_stored":3646972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7871,"internal_metadata":956883265},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4169,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4198400,"data_stored":4194323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6167,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7550,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2157,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3406,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-17T17:49:17.327 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-08-17T17:49:17.328 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/parallel.py", line 48, in task with parallel.parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_91f88e4d132ac08dc53355fa3d9717a53236a3a9/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_91f88e4d132ac08dc53355fa3d9717a53236a3a9/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cls/test_cls_rbd.sh) on smithi003 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2024-08-17T17:49:17.505 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=b4173c182322467985f5d77cba5e2087 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/parallel.py", line 48, in task with parallel.parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_91f88e4d132ac08dc53355fa3d9717a53236a3a9/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_91f88e4d132ac08dc53355fa3d9717a53236a3a9/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cls/test_cls_rbd.sh) on smithi003 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2024-08-17T17:49:17.508 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-08-17T17:49:17.518 INFO:tasks.cephadm:Teardown begin 2024-08-17T17:49:17.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-17T17:49:17.582 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-17T17:49:17.626 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-08-17T17:49:17.626 DEBUG:teuthology.orchestra.run.smithi003:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-08-17T17:49:17.644 DEBUG:teuthology.orchestra.run.smithi053:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-08-17T17:49:17.685 INFO:tasks.cephadm:Stopping all daemons... 2024-08-17T17:49:17.686 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-08-17T17:49:17.686 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mon.a 2024-08-17T17:49:17.997 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:49:17 smithi003 systemd[1]: Stopping Ceph mon.a for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:49:18.264 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:49:17 smithi003 ceph-mon[103344]: pgmap v1439: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:49:18.264 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:49:17 smithi003 ceph-mon[103344]: from='client.47141 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:49:18.264 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:49:17 smithi003 ceph-mon[100615]: pgmap v1439: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:49:18.264 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:49:17 smithi003 ceph-mon[100615]: from='client.47141 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:49:18.264 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:49:18 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-a[100611]: 2024-08-17T17:49:18.039+0000 7fc4d9ad0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T17:49:18.265 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:49:18 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-a[100611]: 2024-08-17T17:49:18.039+0000 7fc4d9ad0640 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-08-17T17:49:18.265 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:49:18 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: [17/Aug/2024:17:49:18] ENGINE Bus STOPPING 2024-08-17T17:49:18.265 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:49:18 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: [17/Aug/2024:17:49:18] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-08-17T17:49:18.265 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:49:18 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: [17/Aug/2024:17:49:18] ENGINE Bus STOPPED 2024-08-17T17:49:18.265 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:49:18 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: [17/Aug/2024:17:49:18] ENGINE Bus STARTING 2024-08-17T17:49:18.265 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:49:18 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: [17/Aug/2024:17:49:18] ENGINE Serving on http://:::9283 2024-08-17T17:49:18.266 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:49:18 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: [17/Aug/2024:17:49:18] ENGINE Bus STARTED 2024-08-17T17:49:18.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:17 smithi053 ceph-mon[89963]: pgmap v1439: 103 pgs: 103 active+clean; 4.6 MiB data, 5.9 GiB used, 709 GiB / 715 GiB avail 2024-08-17T17:49:18.467 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:17 smithi053 ceph-mon[89963]: from='client.47141 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-17T17:49:18.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:49:18 smithi003 podman[331434]: 2024-08-17 17:49:18.275271934 +0000 UTC m=+0.371306801 container died c1cfbc2d47f8446b609e615b86be102c47a3b842c3e7bab3ca9290f7353dff3d (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_CLEAN=True, RELEASE=main-cd8bdbe) 2024-08-17T17:49:18.925 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:49:18 smithi003 podman[331434]: 2024-08-17 17:49:18.639520049 +0000 UTC m=+0.735554914 container cleanup c1cfbc2d47f8446b609e615b86be102c47a3b842c3e7bab3ca9290f7353dff3d (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-a, org.label-schema.build-date=20240731, RELEASE=main-cd8bdbe, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-08-17T17:49:18.925 INFO:journalctl@ceph.mon.a.smithi003.stdout:Aug 17 17:49:18 smithi003 bash[331434]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-a 2024-08-17T17:49:19.082 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mon.a.service' 2024-08-17T17:49:19.121 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-17T17:49:19.122 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-08-17T17:49:19.122 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-08-17T17:49:19.122 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mon.c 2024-08-17T17:49:19.479 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:49:19 smithi003 systemd[1]: Stopping Ceph mon.c for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:49:19.767 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:49:19 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: [17/Aug/2024:17:49:19] ENGINE Bus STOPPING 2024-08-17T17:49:19.767 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:49:19 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: [17/Aug/2024:17:49:19] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-08-17T17:49:19.767 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:49:19 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-c[103340]: 2024-08-17T17:49:19.478+0000 7fab1d7c4640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T17:49:19.767 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:49:19 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-c[103340]: 2024-08-17T17:49:19.478+0000 7fab1d7c4640 -1 mon.c@2(peon) e4 *** Got Signal Terminated *** 2024-08-17T17:49:19.768 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:49:19 smithi003 podman[331567]: 2024-08-17 17:49:19.509130006 +0000 UTC m=+0.157493449 container died 54a6cfe9eb9aa657457be3d687d1b78dcb57c3d174cda0c5dad6ac13004dccf2 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-c, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-cd8bdbe, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-08-17T17:49:20.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:49:19 smithi003 podman[331567]: 2024-08-17 17:49:19.869539209 +0000 UTC m=+0.517902639 container cleanup 54a6cfe9eb9aa657457be3d687d1b78dcb57c3d174cda0c5dad6ac13004dccf2 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-c, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-cd8bdbe, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-08-17T17:49:20.023 INFO:journalctl@ceph.mon.c.smithi003.stdout:Aug 17 17:49:19 smithi003 bash[331567]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-c 2024-08-17T17:49:20.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:49:19 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: [17/Aug/2024:17:49:19] ENGINE Bus STOPPED 2024-08-17T17:49:20.023 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:49:19 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: [17/Aug/2024:17:49:19] ENGINE Bus STARTING 2024-08-17T17:49:20.024 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:49:19 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: [17/Aug/2024:17:49:19] ENGINE Serving on http://:::9283 2024-08-17T17:49:20.024 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:49:19 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: [17/Aug/2024:17:49:19] ENGINE Bus STARTED 2024-08-17T17:49:20.464 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mon.c.service' 2024-08-17T17:49:20.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:49:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:49:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"2b5b662e-5cba-11ef-bccf-c7b262605968\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-08-17T17:49:20.499 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-17T17:49:20.500 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-08-17T17:49:20.500 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-08-17T17:49:20.500 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mon.b 2024-08-17T17:49:20.836 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:20 smithi053 systemd[1]: Stopping Ceph mon.b for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:49:21.172 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-b[89959]: 2024-08-17T17:49:20.834+0000 7fb80ebf6640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T17:49:21.172 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-b[89959]: 2024-08-17T17:49:20.834+0000 7fb80ebf6640 -1 mon.b@1(peon) e4 *** Got Signal Terminated *** 2024-08-17T17:49:21.172 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:20 smithi053 podman[232039]: 2024-08-17 17:49:20.916487019 +0000 UTC m=+0.206087048 container died 9dc96112f3afceec9ace658d33f047e7cde585c3c68bf30a416d01767198c245 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-b, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=main-cd8bdbe, ceph=True) 2024-08-17T17:49:21.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:21 smithi053 podman[232039]: 2024-08-17 17:49:21.359846199 +0000 UTC m=+0.649446270 container cleanup 9dc96112f3afceec9ace658d33f047e7cde585c3c68bf30a416d01767198c245 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-cd8bdbe, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-17T17:49:21.466 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:21 smithi053 bash[232039]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-b 2024-08-17T17:49:21.795 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mon.b.service' 2024-08-17T17:49:21.820 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:21 smithi053 podman[232051]: 2024-08-17 17:49:21.569991565 +0000 UTC m=+0.642496175 container remove 9dc96112f3afceec9ace658d33f047e7cde585c3c68bf30a416d01767198c245 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mon-b, RELEASE=main-cd8bdbe, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:49:21.821 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:21 smithi053 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mon.b.service: Deactivated successfully. 2024-08-17T17:49:21.821 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:21 smithi053 systemd[1]: Stopped Ceph mon.b for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:49:21.821 INFO:journalctl@ceph.mon.b.smithi053.stdout:Aug 17 17:49:21 smithi053 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mon.b.service: Consumed 20.030s CPU time. 2024-08-17T17:49:21.863 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-17T17:49:21.864 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-08-17T17:49:21.864 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-08-17T17:49:21.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mgr.y 2024-08-17T17:49:22.183 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:49:21 smithi003 systemd[1]: Stopping Ceph mgr.y for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:49:22.503 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:49:22 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y[91446]: 2024-08-17T17:49:22.182+0000 7f28eb62b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T17:49:22.503 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:49:22 smithi003 podman[331702]: 2024-08-17 17:49:22.22511785 +0000 UTC m=+0.148874386 container died f491ea24df2839cb7e000efbd2f7292d509d2f75f07e408387955bdab5c29907 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=main-cd8bdbe, ceph=True) 2024-08-17T17:49:22.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:49:22 smithi003 podman[331702]: 2024-08-17 17:49:22.666133325 +0000 UTC m=+0.589889885 container cleanup f491ea24df2839cb7e000efbd2f7292d509d2f75f07e408387955bdab5c29907 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=main-cd8bdbe, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, ceph=True) 2024-08-17T17:49:22.773 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:49:22 smithi003 bash[331702]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y 2024-08-17T17:49:23.105 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mgr.y.service' 2024-08-17T17:49:23.128 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:49:22 smithi003 podman[331713]: 2024-08-17 17:49:22.876280253 +0000 UTC m=+0.651689526 container remove f491ea24df2839cb7e000efbd2f7292d509d2f75f07e408387955bdab5c29907 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-cd8bdbe, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:49:23.128 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:49:23 smithi003 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mgr.y.service: Deactivated successfully. 2024-08-17T17:49:23.129 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:49:23 smithi003 systemd[1]: Stopped Ceph mgr.y for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:49:23.129 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Aug 17 17:49:23 smithi003 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mgr.y.service: Consumed 45.252s CPU time. 2024-08-17T17:49:23.141 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-17T17:49:23.142 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-08-17T17:49:23.142 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-08-17T17:49:23.142 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mgr.x 2024-08-17T17:49:23.466 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:49:23 smithi053 systemd[1]: Stopping Ceph mgr.x for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:49:23.912 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:49:23 smithi053 podman[232160]: 2024-08-17 17:49:23.603730063 +0000 UTC m=+0.253688056 container died 874be7d3a23839f9f1b0d6f5cec91f50ea58a84f6dba32cc116f8250c1868679 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_CLEAN=True, RELEASE=main-cd8bdbe, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-17T17:49:24.216 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:49:23 smithi053 podman[232160]: 2024-08-17 17:49:23.997797164 +0000 UTC m=+0.647755159 container cleanup 874be7d3a23839f9f1b0d6f5cec91f50ea58a84f6dba32cc116f8250c1868679 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=main-cd8bdbe, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-08-17T17:49:24.217 INFO:journalctl@ceph.mgr.x.smithi053.stdout:Aug 17 17:49:23 smithi053 bash[232160]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-mgr-x 2024-08-17T17:49:24.449 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@mgr.x.service' 2024-08-17T17:49:24.486 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-17T17:49:24.486 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-08-17T17:49:24.486 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-08-17T17:49:24.486 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.0 2024-08-17T17:49:24.773 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:49:24 smithi003 systemd[1]: Stopping Ceph osd.0 for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:49:25.273 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:49:24 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0[111839]: 2024-08-17T17:49:24.795+0000 7ff0227bd640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T17:49:25.273 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:49:24 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0[111839]: 2024-08-17T17:49:24.795+0000 7ff0227bd640 -1 osd.0 427 *** Got signal Terminated *** 2024-08-17T17:49:25.273 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:49:24 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0[111839]: 2024-08-17T17:49:24.795+0000 7ff0227bd640 -1 osd.0 427 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-17T17:49:30.481 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:49:30 smithi003 podman[331822]: 2024-08-17 17:49:30.222476754 +0000 UTC m=+5.525234671 container died 5d56fa8e72d5142e94ee4b2fddf53066f1017279d67bf1c93d51061d14466054 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, RELEASE=main-cd8bdbe, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-08-17T17:49:30.773 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:49:30 smithi003 podman[331822]: 2024-08-17 17:49:30.654066278 +0000 UTC m=+5.956824234 container cleanup 5d56fa8e72d5142e94ee4b2fddf53066f1017279d67bf1c93d51061d14466054 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, RELEASE=main-cd8bdbe, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-17T17:49:30.773 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:49:30 smithi003 bash[331822]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0 2024-08-17T17:49:31.272 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:49:30 smithi003 podman[331837]: 2024-08-17 17:49:30.855720009 +0000 UTC m=+0.632662352 container remove 5d56fa8e72d5142e94ee4b2fddf53066f1017279d67bf1c93d51061d14466054 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-cd8bdbe, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-17T17:49:31.523 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:49:31 smithi003 podman[331929]: 2024-08-17 17:49:31.174583795 +0000 UTC m=+0.017898061 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:49:31.523 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:49:31 smithi003 podman[331929]: 2024-08-17 17:49:31.30284572 +0000 UTC m=+0.146160016 container create 6ac7deb8ce1a31be86c846344e374a6bf4512843dd9dea793e3a7d04596dc6fd (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0-deactivate, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=main-cd8bdbe, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-08-17T17:49:32.023 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:49:31 smithi003 podman[331929]: 2024-08-17 17:49:31.526000433 +0000 UTC m=+0.369314687 container init 6ac7deb8ce1a31be86c846344e374a6bf4512843dd9dea793e3a7d04596dc6fd (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-cd8bdbe, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-08-17T17:49:32.023 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:49:31 smithi003 podman[331929]: 2024-08-17 17:49:31.530030477 +0000 UTC m=+0.373344732 container start 6ac7deb8ce1a31be86c846344e374a6bf4512843dd9dea793e3a7d04596dc6fd (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0-deactivate, RELEASE=main-cd8bdbe, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-08-17T17:49:32.023 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:49:31 smithi003 podman[331929]: 2024-08-17 17:49:31.659293674 +0000 UTC m=+0.502607970 container attach 6ac7deb8ce1a31be86c846344e374a6bf4512843dd9dea793e3a7d04596dc6fd (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-cd8bdbe, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-08-17T17:49:32.400 INFO:journalctl@ceph.osd.0.smithi003.stdout:Aug 17 17:49:32 smithi003 podman[331929]: 2024-08-17 17:49:32.112603098 +0000 UTC m=+0.955917367 container died 6ac7deb8ce1a31be86c846344e374a6bf4512843dd9dea793e3a7d04596dc6fd (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, RELEASE=main-cd8bdbe, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-08-17T17:49:32.979 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.0.service' 2024-08-17T17:49:33.019 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-17T17:49:33.019 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-08-17T17:49:33.019 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-08-17T17:49:33.019 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.1 2024-08-17T17:49:33.273 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:49:33 smithi003 systemd[1]: Stopping Ceph osd.1 for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:49:33.773 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:49:33 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1[230371]: 2024-08-17T17:49:33.374+0000 7f7d53aaf640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T17:49:33.773 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:49:33 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1[230371]: 2024-08-17T17:49:33.374+0000 7f7d53aaf640 -1 osd.1 427 *** Got signal Terminated *** 2024-08-17T17:49:33.773 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:49:33 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1[230371]: 2024-08-17T17:49:33.374+0000 7f7d53aaf640 -1 osd.1 427 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-17T17:49:39.326 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:49:39 smithi003 podman[332045]: 2024-08-17 17:49:39.039978168 +0000 UTC m=+5.769384528 container died 967184dd1121826dd60baf56e6eb0428f31dfe51713cd0fe44b8e9087a768903 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, RELEASE=main-cd8bdbe, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0) 2024-08-17T17:49:39.752 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:49:39 smithi003 podman[332045]: 2024-08-17 17:49:39.516779998 +0000 UTC m=+6.246186360 container cleanup 967184dd1121826dd60baf56e6eb0428f31dfe51713cd0fe44b8e9087a768903 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=main-cd8bdbe, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=) 2024-08-17T17:49:39.752 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:49:39 smithi003 bash[332045]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1 2024-08-17T17:49:40.023 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:49:39 smithi003 podman[332062]: 2024-08-17 17:49:39.751514858 +0000 UTC m=+0.707441015 container remove 967184dd1121826dd60baf56e6eb0428f31dfe51713cd0fe44b8e9087a768903 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-cd8bdbe, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-08-17T17:49:40.372 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:49:40 smithi003 podman[332152]: 2024-08-17 17:49:40.006535436 +0000 UTC m=+0.017547820 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:49:40.372 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:49:40 smithi003 podman[332152]: 2024-08-17 17:49:40.127694609 +0000 UTC m=+0.138706978 container create 3b592bfdb22d128a0dfdf1f9d7b76dc486fccb4c557d3f3395fa0d2502e19d99 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-deactivate, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=main-cd8bdbe, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.36.0) 2024-08-17T17:49:40.773 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:49:40 smithi003 podman[332152]: 2024-08-17 17:49:40.371693828 +0000 UTC m=+0.382706197 container init 3b592bfdb22d128a0dfdf1f9d7b76dc486fccb4c557d3f3395fa0d2502e19d99 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-deactivate, RELEASE=main-cd8bdbe, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-08-17T17:49:40.773 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:49:40 smithi003 podman[332152]: 2024-08-17 17:49:40.37511155 +0000 UTC m=+0.386123922 container start 3b592bfdb22d128a0dfdf1f9d7b76dc486fccb4c557d3f3395fa0d2502e19d99 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, RELEASE=main-cd8bdbe, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:49:40.774 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:49:40 smithi003 podman[332152]: 2024-08-17 17:49:40.463418814 +0000 UTC m=+0.474431197 container attach 3b592bfdb22d128a0dfdf1f9d7b76dc486fccb4c557d3f3395fa0d2502e19d99 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=main-cd8bdbe, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:49:40.774 INFO:journalctl@ceph.osd.1.smithi003.stdout:Aug 17 17:49:40 smithi003 podman[332152]: 2024-08-17 17:49:40.5706397 +0000 UTC m=+0.581652088 container died 3b592bfdb22d128a0dfdf1f9d7b76dc486fccb4c557d3f3395fa0d2502e19d99 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-1-deactivate, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, RELEASE=main-cd8bdbe, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:49:41.291 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.1.service' 2024-08-17T17:49:41.337 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-17T17:49:41.338 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-08-17T17:49:41.338 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-08-17T17:49:41.338 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.2 2024-08-17T17:49:41.523 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:49:41 smithi003 systemd[1]: Stopping Ceph osd.2 for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:49:42.023 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:49:41 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2[122810]: 2024-08-17T17:49:41.645+0000 7fd2f005c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T17:49:42.023 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:49:41 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2[122810]: 2024-08-17T17:49:41.645+0000 7fd2f005c640 -1 osd.2 427 *** Got signal Terminated *** 2024-08-17T17:49:42.023 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:49:41 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2[122810]: 2024-08-17T17:49:41.645+0000 7fd2f005c640 -1 osd.2 427 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-17T17:49:47.407 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:49:47 smithi003 podman[332254]: 2024-08-17 17:49:47.139458681 +0000 UTC m=+5.607120987 container died b30ff78c142eeb8dfb1d7a4aff6bed2d937e2ebe7b7be60565e1a3cf9dba5af6 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-cd8bdbe, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.36.0) 2024-08-17T17:49:47.772 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:49:47 smithi003 podman[332254]: 2024-08-17 17:49:47.578953831 +0000 UTC m=+6.046616182 container cleanup b30ff78c142eeb8dfb1d7a4aff6bed2d937e2ebe7b7be60565e1a3cf9dba5af6 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-cd8bdbe, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-08-17T17:49:47.772 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:49:47 smithi003 bash[332254]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2 2024-08-17T17:49:48.022 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:49:47 smithi003 podman[332270]: 2024-08-17 17:49:47.771882161 +0000 UTC m=+0.625760348 container remove b30ff78c142eeb8dfb1d7a4aff6bed2d937e2ebe7b7be60565e1a3cf9dba5af6 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=main-cd8bdbe, org.label-schema.vendor=CentOS) 2024-08-17T17:49:48.517 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:49:48 smithi003 podman[332357]: 2024-08-17 17:49:48.206188724 +0000 UTC m=+0.109156257 container create e694814116902919ab44534668f64e74aae7d3fadd9a6265b8dda91e994bb812 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-cd8bdbe) 2024-08-17T17:49:48.517 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:49:48 smithi003 podman[332357]: 2024-08-17 17:49:48.114727943 +0000 UTC m=+0.017695489 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:49:48.518 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:49:48 smithi003 podman[332357]: 2024-08-17 17:49:48.441861472 +0000 UTC m=+0.344829000 container init e694814116902919ab44534668f64e74aae7d3fadd9a6265b8dda91e994bb812 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , ceph=True, RELEASE=main-cd8bdbe, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-08-17T17:49:48.518 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:49:48 smithi003 podman[332357]: 2024-08-17 17:49:48.445232892 +0000 UTC m=+0.348200424 container start e694814116902919ab44534668f64e74aae7d3fadd9a6265b8dda91e994bb812 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.36.0, RELEASE=main-cd8bdbe, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:49:48.518 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:49:48 smithi003 podman[332357]: 2024-08-17 17:49:48.516759267 +0000 UTC m=+0.419726796 container attach e694814116902919ab44534668f64e74aae7d3fadd9a6265b8dda91e994bb812 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, RELEASE=main-cd8bdbe, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-08-17T17:49:48.773 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:49:48 smithi003 podman[332357]: 2024-08-17 17:49:48.635968584 +0000 UTC m=+0.538936119 container died e694814116902919ab44534668f64e74aae7d3fadd9a6265b8dda91e994bb812 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-cd8bdbe, maintainer=Guillaume Abrioux ) 2024-08-17T17:49:49.172 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.2.service' 2024-08-17T17:49:49.197 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:49:49 smithi003 podman[332381]: 2024-08-17 17:49:49.14502309 +0000 UTC m=+0.501560606 container remove e694814116902919ab44534668f64e74aae7d3fadd9a6265b8dda91e994bb812 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, ceph=True, RELEASE=main-cd8bdbe, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-17T17:49:49.198 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:49:49 smithi003 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.2.service: Deactivated successfully. 2024-08-17T17:49:49.198 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:49:49 smithi003 systemd[1]: Stopped Ceph osd.2 for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:49:49.198 INFO:journalctl@ceph.osd.2.smithi003.stdout:Aug 17 17:49:49 smithi003 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.2.service: Consumed 29min 25.677s CPU time. 2024-08-17T17:49:49.213 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-17T17:49:49.214 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-08-17T17:49:49.214 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-08-17T17:49:49.214 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.3 2024-08-17T17:49:49.523 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:49:49 smithi003 systemd[1]: Stopping Ceph osd.3 for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:49:50.023 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:49:49 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3[128547]: 2024-08-17T17:49:49.546+0000 7f7f951f6640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T17:49:50.023 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:49:49 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3[128547]: 2024-08-17T17:49:49.546+0000 7f7f951f6640 -1 osd.3 427 *** Got signal Terminated *** 2024-08-17T17:49:50.023 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:49:49 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3[128547]: 2024-08-17T17:49:49.546+0000 7f7f951f6640 -1 osd.3 427 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-17T17:49:51.466 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:49:51 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-4[39627]: 2024-08-17T17:49:51.123+0000 7f78ca5f5640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.927151+0000 front 2024-08-17T17:49:26.927071+0000 (oldest deadline 2024-08-17T17:49:50.427047+0000) 2024-08-17T17:49:52.353 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:49:52 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-4[39627]: 2024-08-17T17:49:52.099+0000 7f78ca5f5640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.927151+0000 front 2024-08-17T17:49:26.927071+0000 (oldest deadline 2024-08-17T17:49:50.427047+0000) 2024-08-17T17:49:52.716 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:49:52 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:49:52.351+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.098745+0000 front 2024-08-17T17:49:26.098588+0000 (oldest deadline 2024-08-17T17:49:51.998572+0000) 2024-08-17T17:49:52.717 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:49:52 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:49:52.469+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.445303+0000 front 2024-08-17T17:49:26.445392+0000 (oldest deadline 2024-08-17T17:49:52.345534+0000) 2024-08-17T17:49:53.376 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:49:53 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-4[39627]: 2024-08-17T17:49:53.118+0000 7f78ca5f5640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.927151+0000 front 2024-08-17T17:49:26.927071+0000 (oldest deadline 2024-08-17T17:49:50.427047+0000) 2024-08-17T17:49:53.652 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:49:53 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:49:53.461+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.445303+0000 front 2024-08-17T17:49:26.445392+0000 (oldest deadline 2024-08-17T17:49:52.345534+0000) 2024-08-17T17:49:53.653 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:49:53 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:49:53.375+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.098745+0000 front 2024-08-17T17:49:26.098588+0000 (oldest deadline 2024-08-17T17:49:51.998572+0000) 2024-08-17T17:49:53.966 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:49:53 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:49:53.651+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:49:54.023 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:49:53 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3[128547]: 2024-08-17T17:49:53.743+0000 7f7f9180b640 -1 osd.3 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:29.433206+0000 front 2024-08-17T17:49:29.433331+0000 (oldest deadline 2024-08-17T17:49:52.933313+0000) 2024-08-17T17:49:54.413 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:49:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-4[39627]: 2024-08-17T17:49:54.138+0000 7f78ca5f5640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.927151+0000 front 2024-08-17T17:49:26.927071+0000 (oldest deadline 2024-08-17T17:49:50.427047+0000) 2024-08-17T17:49:54.716 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:49:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:49:54.456+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.445303+0000 front 2024-08-17T17:49:26.445392+0000 (oldest deadline 2024-08-17T17:49:52.345534+0000) 2024-08-17T17:49:54.717 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:49:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:49:54.413+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.098745+0000 front 2024-08-17T17:49:26.098588+0000 (oldest deadline 2024-08-17T17:49:51.998572+0000) 2024-08-17T17:49:54.717 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:49:54 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:49:54.652+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:49:55.432 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:49:55 smithi003 podman[332461]: 2024-08-17 17:49:55.156661111 +0000 UTC m=+5.713195315 container died fe4dcbe3219e018f2e8d9a3e2889b21a7c754b13c51cd73113cfffc972d10ab9 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=main-cd8bdbe, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-08-17T17:49:55.464 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:49:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-4[39627]: 2024-08-17T17:49:55.173+0000 7f78ca5f5640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.927151+0000 front 2024-08-17T17:49:26.927071+0000 (oldest deadline 2024-08-17T17:49:50.427047+0000) 2024-08-17T17:49:55.465 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:49:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:49:55.383+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.098745+0000 front 2024-08-17T17:49:26.098588+0000 (oldest deadline 2024-08-17T17:49:51.998572+0000) 2024-08-17T17:49:55.716 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:49:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:49:55.464+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.445303+0000 front 2024-08-17T17:49:26.445392+0000 (oldest deadline 2024-08-17T17:49:52.345534+0000) 2024-08-17T17:49:55.717 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:49:55 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:49:55.669+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:49:55.773 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:49:55 smithi003 podman[332461]: 2024-08-17 17:49:55.622554644 +0000 UTC m=+6.179088846 container cleanup fe4dcbe3219e018f2e8d9a3e2889b21a7c754b13c51cd73113cfffc972d10ab9 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-cd8bdbe, org.label-schema.schema-version=1.0) 2024-08-17T17:49:55.773 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:49:55 smithi003 bash[332461]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3 2024-08-17T17:49:56.241 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:49:55 smithi003 podman[332476]: 2024-08-17 17:49:55.915496814 +0000 UTC m=+0.752132110 container remove fe4dcbe3219e018f2e8d9a3e2889b21a7c754b13c51cd73113cfffc972d10ab9 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-cd8bdbe, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-08-17T17:49:56.466 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:49:56 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-4[39627]: 2024-08-17T17:49:56.179+0000 7f78ca5f5640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.927151+0000 front 2024-08-17T17:49:26.927071+0000 (oldest deadline 2024-08-17T17:49:50.427047+0000) 2024-08-17T17:49:56.467 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:49:56 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:49:56.352+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.098745+0000 front 2024-08-17T17:49:26.098588+0000 (oldest deadline 2024-08-17T17:49:51.998572+0000) 2024-08-17T17:49:56.510 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:49:56 smithi003 podman[332566]: 2024-08-17 17:49:56.241240572 +0000 UTC m=+0.102023075 container create 46f2a4804895e19e74a70e016bb773fc25dbb4adc066862b69b221ff2163ee09 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=main-cd8bdbe, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0) 2024-08-17T17:49:56.511 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:49:56 smithi003 podman[332566]: 2024-08-17 17:49:56.157227691 +0000 UTC m=+0.018010196 image pull 0566dfeff0cb6af9254aa2111a92af8174516dad87a45779e953ca0dd77f3099 quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d 2024-08-17T17:49:56.511 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:49:56 smithi003 podman[332566]: 2024-08-17 17:49:56.443490309 +0000 UTC m=+0.304272828 container init 46f2a4804895e19e74a70e016bb773fc25dbb4adc066862b69b221ff2163ee09 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, RELEASE=main-cd8bdbe) 2024-08-17T17:49:56.511 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:49:56 smithi003 podman[332566]: 2024-08-17 17:49:56.446962126 +0000 UTC m=+0.307744634 container start 46f2a4804895e19e74a70e016bb773fc25dbb4adc066862b69b221ff2163ee09 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, RELEASE=main-cd8bdbe, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-08-17T17:49:56.511 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:49:56 smithi003 podman[332566]: 2024-08-17 17:49:56.510142199 +0000 UTC m=+0.370924704 container attach 46f2a4804895e19e74a70e016bb773fc25dbb4adc066862b69b221ff2163ee09 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, RELEASE=main-cd8bdbe) 2024-08-17T17:49:56.773 INFO:journalctl@ceph.osd.3.smithi003.stdout:Aug 17 17:49:56 smithi003 podman[332566]: 2024-08-17 17:49:56.6431556 +0000 UTC m=+0.503938105 container died 46f2a4804895e19e74a70e016bb773fc25dbb4adc066862b69b221ff2163ee09 (image=quay.ceph.io/ceph-ci/ceph@sha256:aee44f95cb23e15b5d48a3d9a6d838e96ffc6125ee57d506dda5eff94584115d, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , RELEASE=main-cd8bdbe, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-08-17T17:49:56.966 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:49:56 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:49:56.481+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.445303+0000 front 2024-08-17T17:49:26.445392+0000 (oldest deadline 2024-08-17T17:49:52.345534+0000) 2024-08-17T17:49:56.967 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:49:56 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:49:56.622+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:49:57.381 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.3.service' 2024-08-17T17:49:57.425 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-17T17:49:57.425 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-08-17T17:49:57.425 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-08-17T17:49:57.425 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.4 2024-08-17T17:49:57.455 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:49:57 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-4[39627]: 2024-08-17T17:49:57.184+0000 7f78ca5f5640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.927151+0000 front 2024-08-17T17:49:26.927071+0000 (oldest deadline 2024-08-17T17:49:50.427047+0000) 2024-08-17T17:49:57.455 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:49:57 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:49:57.348+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.098745+0000 front 2024-08-17T17:49:26.098588+0000 (oldest deadline 2024-08-17T17:49:51.998572+0000) 2024-08-17T17:49:57.716 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:49:57 smithi053 systemd[1]: Stopping Ceph osd.4 for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:49:57.717 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:49:57 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:49:57.616+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:49:57.717 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:49:57 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:49:57.455+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.445303+0000 front 2024-08-17T17:49:26.445392+0000 (oldest deadline 2024-08-17T17:49:52.345534+0000) 2024-08-17T17:49:58.154 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:49:57 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-4[39627]: 2024-08-17T17:49:57.760+0000 7f78cffe1640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T17:49:58.155 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:49:57 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-4[39627]: 2024-08-17T17:49:57.760+0000 7f78cffe1640 -1 osd.4 427 *** Got signal Terminated *** 2024-08-17T17:49:58.155 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:49:57 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-4[39627]: 2024-08-17T17:49:57.760+0000 7f78cffe1640 -1 osd.4 427 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-17T17:49:58.466 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:49:58 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-4[39627]: 2024-08-17T17:49:58.153+0000 7f78ca5f5640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.927151+0000 front 2024-08-17T17:49:26.927071+0000 (oldest deadline 2024-08-17T17:49:50.427047+0000) 2024-08-17T17:49:58.466 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:49:58 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:49:58.393+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.098745+0000 front 2024-08-17T17:49:26.098588+0000 (oldest deadline 2024-08-17T17:49:51.998572+0000) 2024-08-17T17:49:58.966 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:49:58 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:49:58.484+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.445303+0000 front 2024-08-17T17:49:26.445392+0000 (oldest deadline 2024-08-17T17:49:52.345534+0000) 2024-08-17T17:49:58.967 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:49:58 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:49:58.632+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:49:59.462 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:49:59 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:49:59.348+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.098745+0000 front 2024-08-17T17:49:26.098588+0000 (oldest deadline 2024-08-17T17:49:51.998572+0000) 2024-08-17T17:49:59.463 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:49:59 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-4[39627]: 2024-08-17T17:49:59.188+0000 7f78ca5f5640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.927151+0000 front 2024-08-17T17:49:26.927071+0000 (oldest deadline 2024-08-17T17:49:50.427047+0000) 2024-08-17T17:49:59.463 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:49:59 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-4[39627]: 2024-08-17T17:49:59.188+0000 7f78ca5f5640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:35.628717+0000 front 2024-08-17T17:49:35.628698+0000 (oldest deadline 2024-08-17T17:49:59.128619+0000) 2024-08-17T17:49:59.716 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:49:59 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:49:59.584+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:49:59.717 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:49:59 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:49:59.462+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.445303+0000 front 2024-08-17T17:49:26.445392+0000 (oldest deadline 2024-08-17T17:49:52.345534+0000) 2024-08-17T17:49:59.717 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:49:59 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:49:59.462+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:36.446644+0000 front 2024-08-17T17:49:36.446730+0000 (oldest deadline 2024-08-17T17:49:59.346719+0000) 2024-08-17T17:50:00.467 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:50:00.335+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.098745+0000 front 2024-08-17T17:49:26.098588+0000 (oldest deadline 2024-08-17T17:49:51.998572+0000) 2024-08-17T17:50:00.467 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:50:00.335+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:36.099862+0000 front 2024-08-17T17:49:36.099837+0000 (oldest deadline 2024-08-17T17:50:00.199768+0000) 2024-08-17T17:50:00.467 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:50:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-4[39627]: 2024-08-17T17:50:00.140+0000 7f78ca5f5640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.927151+0000 front 2024-08-17T17:49:26.927071+0000 (oldest deadline 2024-08-17T17:49:50.427047+0000) 2024-08-17T17:50:00.467 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:50:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-4[39627]: 2024-08-17T17:50:00.140+0000 7f78ca5f5640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:35.628717+0000 front 2024-08-17T17:49:35.628698+0000 (oldest deadline 2024-08-17T17:49:59.128619+0000) 2024-08-17T17:50:00.966 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:00.620+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:50:00.967 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:00.500+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.445303+0000 front 2024-08-17T17:49:26.445392+0000 (oldest deadline 2024-08-17T17:49:52.345534+0000) 2024-08-17T17:50:00.967 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:00 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:00.500+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:36.446644+0000 front 2024-08-17T17:49:36.446730+0000 (oldest deadline 2024-08-17T17:49:59.346719+0000) 2024-08-17T17:50:01.466 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:01 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:50:01.291+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.098745+0000 front 2024-08-17T17:49:26.098588+0000 (oldest deadline 2024-08-17T17:49:51.998572+0000) 2024-08-17T17:50:01.467 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:01 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:50:01.291+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:36.099862+0000 front 2024-08-17T17:49:36.099837+0000 (oldest deadline 2024-08-17T17:50:00.199768+0000) 2024-08-17T17:50:01.467 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:50:01 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-4[39627]: 2024-08-17T17:50:01.159+0000 7f78ca5f5640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.927151+0000 front 2024-08-17T17:49:26.927071+0000 (oldest deadline 2024-08-17T17:49:50.427047+0000) 2024-08-17T17:50:01.467 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:50:01 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-4[39627]: 2024-08-17T17:50:01.159+0000 7f78ca5f5640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:35.628717+0000 front 2024-08-17T17:49:35.628698+0000 (oldest deadline 2024-08-17T17:49:59.128619+0000) 2024-08-17T17:50:01.966 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:01 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:01.497+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.445303+0000 front 2024-08-17T17:49:26.445392+0000 (oldest deadline 2024-08-17T17:49:52.345534+0000) 2024-08-17T17:50:01.967 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:01 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:01.497+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:36.446644+0000 front 2024-08-17T17:49:36.446730+0000 (oldest deadline 2024-08-17T17:49:59.346719+0000) 2024-08-17T17:50:01.967 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:01 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:01.669+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:50:01.967 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:01 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:01.669+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:35.764961+0000 front 2024-08-17T17:49:35.765118+0000 (oldest deadline 2024-08-17T17:50:01.565350+0000) 2024-08-17T17:50:02.464 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:50:02 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-4[39627]: 2024-08-17T17:50:02.196+0000 7f78ca5f5640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.927151+0000 front 2024-08-17T17:49:26.927071+0000 (oldest deadline 2024-08-17T17:49:50.427047+0000) 2024-08-17T17:50:02.464 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:50:02 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-4[39627]: 2024-08-17T17:50:02.196+0000 7f78ca5f5640 -1 osd.4 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:35.628717+0000 front 2024-08-17T17:49:35.628698+0000 (oldest deadline 2024-08-17T17:49:59.128619+0000) 2024-08-17T17:50:02.464 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:02 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:50:02.287+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.098745+0000 front 2024-08-17T17:49:26.098588+0000 (oldest deadline 2024-08-17T17:49:51.998572+0000) 2024-08-17T17:50:02.465 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:02 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:50:02.287+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:36.099862+0000 front 2024-08-17T17:49:36.099837+0000 (oldest deadline 2024-08-17T17:50:00.199768+0000) 2024-08-17T17:50:02.716 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:02 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:02.463+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.445303+0000 front 2024-08-17T17:49:26.445392+0000 (oldest deadline 2024-08-17T17:49:52.345534+0000) 2024-08-17T17:50:02.717 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:02 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:02.463+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:36.446644+0000 front 2024-08-17T17:49:36.446730+0000 (oldest deadline 2024-08-17T17:49:59.346719+0000) 2024-08-17T17:50:02.717 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:02 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:02.679+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:50:02.717 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:02 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:02.679+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:35.764961+0000 front 2024-08-17T17:49:35.765118+0000 (oldest deadline 2024-08-17T17:50:01.565350+0000) 2024-08-17T17:50:03.511 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:50:03 smithi053 podman[232295]: 2024-08-17 17:50:03.256101633 +0000 UTC m=+5.619342034 container died ced3b086d7f053de0cf18d88c7b8f6b206b6af06dcf3080de7e0292206366ede (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-4, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=quincy-f2c39d4, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0) 2024-08-17T17:50:03.512 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:03 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:50:03.265+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.098745+0000 front 2024-08-17T17:49:26.098588+0000 (oldest deadline 2024-08-17T17:49:51.998572+0000) 2024-08-17T17:50:03.512 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:03 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:50:03.265+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:36.099862+0000 front 2024-08-17T17:49:36.099837+0000 (oldest deadline 2024-08-17T17:50:00.199768+0000) 2024-08-17T17:50:03.966 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:50:03 smithi053 podman[232295]: 2024-08-17 17:50:03.736036095 +0000 UTC m=+6.099276496 container cleanup ced3b086d7f053de0cf18d88c7b8f6b206b6af06dcf3080de7e0292206366ede (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-4, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=quincy-f2c39d4, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:50:03.967 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:50:03 smithi053 bash[232295]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-4 2024-08-17T17:50:03.967 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:03 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:03.510+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.445303+0000 front 2024-08-17T17:49:26.445392+0000 (oldest deadline 2024-08-17T17:49:52.345534+0000) 2024-08-17T17:50:03.967 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:03 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:03.510+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:36.446644+0000 front 2024-08-17T17:49:36.446730+0000 (oldest deadline 2024-08-17T17:49:59.346719+0000) 2024-08-17T17:50:03.967 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:03 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:03.665+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:50:03.968 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:03 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:03.665+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:35.764961+0000 front 2024-08-17T17:49:35.765118+0000 (oldest deadline 2024-08-17T17:50:01.565350+0000) 2024-08-17T17:50:04.272 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:50:04 smithi053 podman[232312]: 2024-08-17 17:50:04.004261569 +0000 UTC m=+0.744000375 container remove ced3b086d7f053de0cf18d88c7b8f6b206b6af06dcf3080de7e0292206366ede (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-4, ceph=True, RELEASE=quincy-f2c39d4, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.36.0) 2024-08-17T17:50:04.548 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:50:04 smithi053 podman[232403]: 2024-08-17 17:50:04.33011447 +0000 UTC m=+0.114665286 container create bbea29e02a17be59a84ecd3dc02c942414a43b15895c41c0aae6c3f73c05d724 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=quincy-f2c39d4) 2024-08-17T17:50:04.549 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:50:04 smithi053 podman[232403]: 2024-08-17 17:50:04.233035169 +0000 UTC m=+0.017586004 image pull f69471c56189b063ba7a049adea817a12ba1a40187d11cab5cbef8f3aac57cce quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251 2024-08-17T17:50:04.549 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:50:04.271+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.098745+0000 front 2024-08-17T17:49:26.098588+0000 (oldest deadline 2024-08-17T17:49:51.998572+0000) 2024-08-17T17:50:04.549 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:50:04.271+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:36.099862+0000 front 2024-08-17T17:49:36.099837+0000 (oldest deadline 2024-08-17T17:50:00.199768+0000) 2024-08-17T17:50:04.833 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:04.548+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.445303+0000 front 2024-08-17T17:49:26.445392+0000 (oldest deadline 2024-08-17T17:49:52.345534+0000) 2024-08-17T17:50:04.833 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:04.548+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:36.446644+0000 front 2024-08-17T17:49:36.446730+0000 (oldest deadline 2024-08-17T17:49:59.346719+0000) 2024-08-17T17:50:04.833 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:50:04 smithi053 podman[232403]: 2024-08-17 17:50:04.578838785 +0000 UTC m=+0.363389613 container init bbea29e02a17be59a84ecd3dc02c942414a43b15895c41c0aae6c3f73c05d724 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=quincy-f2c39d4, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-17T17:50:04.834 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:50:04 smithi053 podman[232403]: 2024-08-17 17:50:04.5822797 +0000 UTC m=+0.366830520 container start bbea29e02a17be59a84ecd3dc02c942414a43b15895c41c0aae6c3f73c05d724 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-f2c39d4, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0) 2024-08-17T17:50:04.834 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:04.710+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:50:04.834 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:04 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:04.710+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:35.764961+0000 front 2024-08-17T17:49:35.765118+0000 (oldest deadline 2024-08-17T17:50:01.565350+0000) 2024-08-17T17:50:05.216 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:50:04 smithi053 podman[232403]: 2024-08-17 17:50:04.833118966 +0000 UTC m=+0.617669793 container attach bbea29e02a17be59a84ecd3dc02c942414a43b15895c41c0aae6c3f73c05d724 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, RELEASE=quincy-f2c39d4, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-08-17T17:50:05.217 INFO:journalctl@ceph.osd.4.smithi053.stdout:Aug 17 17:50:05 smithi053 podman[232403]: 2024-08-17 17:50:05.025514951 +0000 UTC m=+0.810065767 container died bbea29e02a17be59a84ecd3dc02c942414a43b15895c41c0aae6c3f73c05d724 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-4-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, RELEASE=quincy-f2c39d4) 2024-08-17T17:50:05.570 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:05 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:50:05.272+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.098745+0000 front 2024-08-17T17:49:26.098588+0000 (oldest deadline 2024-08-17T17:49:51.998572+0000) 2024-08-17T17:50:05.570 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:05 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:50:05.272+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:36.099862+0000 front 2024-08-17T17:49:36.099837+0000 (oldest deadline 2024-08-17T17:50:00.199768+0000) 2024-08-17T17:50:05.590 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.4.service' 2024-08-17T17:50:05.671 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-17T17:50:05.671 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-08-17T17:50:05.672 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-08-17T17:50:05.672 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.5 2024-08-17T17:50:05.966 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:05 smithi053 systemd[1]: Stopping Ceph osd.5 for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:50:05.967 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:05 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:05.585+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.445303+0000 front 2024-08-17T17:49:26.445392+0000 (oldest deadline 2024-08-17T17:49:52.345534+0000) 2024-08-17T17:50:05.967 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:05 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:05.585+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:36.446644+0000 front 2024-08-17T17:49:36.446730+0000 (oldest deadline 2024-08-17T17:49:59.346719+0000) 2024-08-17T17:50:05.967 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:05 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:05.723+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:50:05.967 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:05 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:05.723+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:35.764961+0000 front 2024-08-17T17:49:35.765118+0000 (oldest deadline 2024-08-17T17:50:01.565350+0000) 2024-08-17T17:50:06.288 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:06 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:50:06.023+0000 7f59b9456640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T17:50:06.289 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:06 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:50:06.023+0000 7f59b9456640 -1 osd.5 427 *** Got signal Terminated *** 2024-08-17T17:50:06.289 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:06 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:50:06.023+0000 7f59b9456640 -1 osd.5 427 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-17T17:50:06.541 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:06 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:50:06.295+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.098745+0000 front 2024-08-17T17:49:26.098588+0000 (oldest deadline 2024-08-17T17:49:51.998572+0000) 2024-08-17T17:50:06.541 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:06 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:50:06.295+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:36.099862+0000 front 2024-08-17T17:49:36.099837+0000 (oldest deadline 2024-08-17T17:50:00.199768+0000) 2024-08-17T17:50:06.966 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:06 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:06.540+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.445303+0000 front 2024-08-17T17:49:26.445392+0000 (oldest deadline 2024-08-17T17:49:52.345534+0000) 2024-08-17T17:50:06.966 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:06 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:06.540+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:36.446644+0000 front 2024-08-17T17:49:36.446730+0000 (oldest deadline 2024-08-17T17:49:59.346719+0000) 2024-08-17T17:50:06.967 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:06 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:06.698+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:50:06.967 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:06 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:06.698+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:35.764961+0000 front 2024-08-17T17:49:35.765118+0000 (oldest deadline 2024-08-17T17:50:01.565350+0000) 2024-08-17T17:50:07.661 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:07 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:50:07.310+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.098745+0000 front 2024-08-17T17:49:26.098588+0000 (oldest deadline 2024-08-17T17:49:51.998572+0000) 2024-08-17T17:50:07.662 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:07 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:50:07.310+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:36.099862+0000 front 2024-08-17T17:49:36.099837+0000 (oldest deadline 2024-08-17T17:50:00.199768+0000) 2024-08-17T17:50:07.662 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:07 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:07.546+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.445303+0000 front 2024-08-17T17:49:26.445392+0000 (oldest deadline 2024-08-17T17:49:52.345534+0000) 2024-08-17T17:50:07.662 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:07 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:07.546+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:36.446644+0000 front 2024-08-17T17:49:36.446730+0000 (oldest deadline 2024-08-17T17:49:59.346719+0000) 2024-08-17T17:50:07.662 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:07 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:07.546+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:46.248440+0000 front 2024-08-17T17:49:46.248555+0000 (oldest deadline 2024-08-17T17:50:07.348263+0000) 2024-08-17T17:50:07.966 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:07 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:07.661+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:50:07.967 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:07 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:07.661+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:35.764961+0000 front 2024-08-17T17:49:35.765118+0000 (oldest deadline 2024-08-17T17:50:01.565350+0000) 2024-08-17T17:50:07.967 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:07 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:07.661+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:45.066492+0000 front 2024-08-17T17:49:45.066438+0000 (oldest deadline 2024-08-17T17:50:07.266846+0000) 2024-08-17T17:50:08.654 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:08 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:50:08.350+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.098745+0000 front 2024-08-17T17:49:26.098588+0000 (oldest deadline 2024-08-17T17:49:51.998572+0000) 2024-08-17T17:50:08.654 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:08 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:50:08.350+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:36.099862+0000 front 2024-08-17T17:49:36.099837+0000 (oldest deadline 2024-08-17T17:50:00.199768+0000) 2024-08-17T17:50:08.655 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:08 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:08.570+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.445303+0000 front 2024-08-17T17:49:26.445392+0000 (oldest deadline 2024-08-17T17:49:52.345534+0000) 2024-08-17T17:50:08.655 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:08 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:08.570+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:36.446644+0000 front 2024-08-17T17:49:36.446730+0000 (oldest deadline 2024-08-17T17:49:59.346719+0000) 2024-08-17T17:50:08.655 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:08 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:08.570+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:46.248440+0000 front 2024-08-17T17:49:46.248555+0000 (oldest deadline 2024-08-17T17:50:07.348263+0000) 2024-08-17T17:50:08.966 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:08 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:08.652+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:50:08.966 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:08 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:08.652+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:35.764961+0000 front 2024-08-17T17:49:35.765118+0000 (oldest deadline 2024-08-17T17:50:01.565350+0000) 2024-08-17T17:50:08.966 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:08 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:08.652+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:45.066492+0000 front 2024-08-17T17:49:45.066438+0000 (oldest deadline 2024-08-17T17:50:07.266846+0000) 2024-08-17T17:50:09.716 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:09.586+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.445303+0000 front 2024-08-17T17:49:26.445392+0000 (oldest deadline 2024-08-17T17:49:52.345534+0000) 2024-08-17T17:50:09.717 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:09.587+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:36.446644+0000 front 2024-08-17T17:49:36.446730+0000 (oldest deadline 2024-08-17T17:49:59.346719+0000) 2024-08-17T17:50:09.717 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:09.587+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:46.248440+0000 front 2024-08-17T17:49:46.248555+0000 (oldest deadline 2024-08-17T17:50:07.348263+0000) 2024-08-17T17:50:09.717 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:09.637+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:50:09.717 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:09.637+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:35.764961+0000 front 2024-08-17T17:49:35.765118+0000 (oldest deadline 2024-08-17T17:50:01.565350+0000) 2024-08-17T17:50:09.718 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:09.637+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:45.066492+0000 front 2024-08-17T17:49:45.066438+0000 (oldest deadline 2024-08-17T17:50:07.266846+0000) 2024-08-17T17:50:09.718 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:50:09.388+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.098745+0000 front 2024-08-17T17:49:26.098588+0000 (oldest deadline 2024-08-17T17:49:51.998572+0000) 2024-08-17T17:50:09.718 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:50:09.388+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:36.099862+0000 front 2024-08-17T17:49:36.099837+0000 (oldest deadline 2024-08-17T17:50:00.199768+0000) 2024-08-17T17:50:09.718 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:09 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:50:09.388+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:46.501824+0000 front 2024-08-17T17:49:46.501712+0000 (oldest deadline 2024-08-17T17:50:08.801672+0000) 2024-08-17T17:50:10.716 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:50:10.409+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.098745+0000 front 2024-08-17T17:49:26.098588+0000 (oldest deadline 2024-08-17T17:49:51.998572+0000) 2024-08-17T17:50:10.716 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:50:10.409+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:36.099862+0000 front 2024-08-17T17:49:36.099837+0000 (oldest deadline 2024-08-17T17:50:00.199768+0000) 2024-08-17T17:50:10.717 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5[45092]: 2024-08-17T17:50:10.409+0000 7f59b4a6c640 -1 osd.5 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:46.501824+0000 front 2024-08-17T17:49:46.501712+0000 (oldest deadline 2024-08-17T17:50:08.801672+0000) 2024-08-17T17:50:10.717 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:10.568+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.445303+0000 front 2024-08-17T17:49:26.445392+0000 (oldest deadline 2024-08-17T17:49:52.345534+0000) 2024-08-17T17:50:10.717 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:10.568+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:36.446644+0000 front 2024-08-17T17:49:36.446730+0000 (oldest deadline 2024-08-17T17:49:59.346719+0000) 2024-08-17T17:50:10.717 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:10.568+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:46.248440+0000 front 2024-08-17T17:49:46.248555+0000 (oldest deadline 2024-08-17T17:50:07.348263+0000) 2024-08-17T17:50:10.717 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:10.613+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:50:10.718 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:10.613+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:35.764961+0000 front 2024-08-17T17:49:35.765118+0000 (oldest deadline 2024-08-17T17:50:01.565350+0000) 2024-08-17T17:50:10.718 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:10 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:10.613+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:45.066492+0000 front 2024-08-17T17:49:45.066438+0000 (oldest deadline 2024-08-17T17:50:07.266846+0000) 2024-08-17T17:50:11.466 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:50:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:50:11.148Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.3:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.3:8765: connect: connection refused" 2024-08-17T17:50:11.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:50:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:50:11.148Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.3:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.3:8765: connect: connection refused" 2024-08-17T17:50:11.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:50:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:50:11.148Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.3:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.3:8765: connect: connection refused" 2024-08-17T17:50:11.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:50:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:50:11.151Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://172.21.15.3:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 172.21.15.3:8765: connect: connection refused" 2024-08-17T17:50:11.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:50:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:50:11.151Z caller=refresh.go:90 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.3:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.3:8765: connect: connection refused" 2024-08-17T17:50:11.467 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:50:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:50:11.151Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.3:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.3:8765: connect: connection refused" 2024-08-17T17:50:11.867 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:11 smithi053 podman[232516]: 2024-08-17 17:50:11.574297808 +0000 UTC m=+5.667333928 container died 1c4ad574a193c0c73d3b669c64cf6140fa6efe76aec980640a8ef351c1656e25 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.36.0, RELEASE=quincy-f2c39d4, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-08-17T17:50:11.868 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:11.578+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.445303+0000 front 2024-08-17T17:49:26.445392+0000 (oldest deadline 2024-08-17T17:49:52.345534+0000) 2024-08-17T17:50:11.868 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:11.578+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:36.446644+0000 front 2024-08-17T17:49:36.446730+0000 (oldest deadline 2024-08-17T17:49:59.346719+0000) 2024-08-17T17:50:11.868 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:11.578+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:46.248440+0000 front 2024-08-17T17:49:46.248555+0000 (oldest deadline 2024-08-17T17:50:07.348263+0000) 2024-08-17T17:50:11.869 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:11.632+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:50:11.869 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:11.632+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:35.764961+0000 front 2024-08-17T17:49:35.765118+0000 (oldest deadline 2024-08-17T17:50:01.565350+0000) 2024-08-17T17:50:11.869 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:11 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:11.632+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:45.066492+0000 front 2024-08-17T17:49:45.066438+0000 (oldest deadline 2024-08-17T17:50:07.266846+0000) 2024-08-17T17:50:12.216 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:11 smithi053 podman[232516]: 2024-08-17 17:50:11.998658218 +0000 UTC m=+6.091694324 container cleanup 1c4ad574a193c0c73d3b669c64cf6140fa6efe76aec980640a8ef351c1656e25 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, RELEASE=quincy-f2c39d4, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:50:12.216 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:11 smithi053 bash[232516]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5 2024-08-17T17:50:12.617 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:12 smithi053 podman[232534]: 2024-08-17 17:50:12.224806052 +0000 UTC m=+0.638443903 container remove 1c4ad574a193c0c73d3b669c64cf6140fa6efe76aec980640a8ef351c1656e25 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=quincy-f2c39d4, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2) 2024-08-17T17:50:12.961 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:12 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:12.623+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.445303+0000 front 2024-08-17T17:49:26.445392+0000 (oldest deadline 2024-08-17T17:49:52.345534+0000) 2024-08-17T17:50:12.962 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:12 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:12.623+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:36.446644+0000 front 2024-08-17T17:49:36.446730+0000 (oldest deadline 2024-08-17T17:49:59.346719+0000) 2024-08-17T17:50:12.962 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:12 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:12.623+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:46.248440+0000 front 2024-08-17T17:49:46.248555+0000 (oldest deadline 2024-08-17T17:50:07.348263+0000) 2024-08-17T17:50:12.962 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:12 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:12.616+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:50:12.962 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:12 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:12.616+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:35.764961+0000 front 2024-08-17T17:49:35.765118+0000 (oldest deadline 2024-08-17T17:50:01.565350+0000) 2024-08-17T17:50:12.962 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:12 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:12.616+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:45.066492+0000 front 2024-08-17T17:49:45.066438+0000 (oldest deadline 2024-08-17T17:50:07.266846+0000) 2024-08-17T17:50:12.963 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:12 smithi053 podman[232622]: 2024-08-17 17:50:12.543782616 +0000 UTC m=+0.018099795 image pull f69471c56189b063ba7a049adea817a12ba1a40187d11cab5cbef8f3aac57cce quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251 2024-08-17T17:50:12.963 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:12 smithi053 podman[232622]: 2024-08-17 17:50:12.650595935 +0000 UTC m=+0.124913094 container create 31f1221052922f6446def6b6952b810c59a127fcb5546390cfffb92ddea24710 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=quincy-f2c39d4, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-08-17T17:50:12.963 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:12 smithi053 podman[232622]: 2024-08-17 17:50:12.861420822 +0000 UTC m=+0.335738000 container init 31f1221052922f6446def6b6952b810c59a127fcb5546390cfffb92ddea24710 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=quincy-f2c39d4, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux ) 2024-08-17T17:50:12.963 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:12 smithi053 podman[232622]: 2024-08-17 17:50:12.864912156 +0000 UTC m=+0.339229318 container start 31f1221052922f6446def6b6952b810c59a127fcb5546390cfffb92ddea24710 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5-deactivate, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, RELEASE=quincy-f2c39d4, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-08-17T17:50:12.963 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:12 smithi053 podman[232622]: 2024-08-17 17:50:12.961402727 +0000 UTC m=+0.435719901 container attach 31f1221052922f6446def6b6952b810c59a127fcb5546390cfffb92ddea24710 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=quincy-f2c39d4, CEPH_POINT_RELEASE=) 2024-08-17T17:50:13.216 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:13 smithi053 podman[232622]: 2024-08-17 17:50:13.042785255 +0000 UTC m=+0.517102417 container died 31f1221052922f6446def6b6952b810c59a127fcb5546390cfffb92ddea24710 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5-deactivate, RELEASE=quincy-f2c39d4, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-17T17:50:13.584 INFO:journalctl@ceph.osd.5.smithi053.stdout:Aug 17 17:50:13 smithi053 podman[232645]: 2024-08-17 17:50:13.56506086 +0000 UTC m=+0.516459929 container remove 31f1221052922f6446def6b6952b810c59a127fcb5546390cfffb92ddea24710 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-5-deactivate, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=quincy-f2c39d4, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-08-17T17:50:13.590 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.5.service' 2024-08-17T17:50:13.631 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-17T17:50:13.631 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-08-17T17:50:13.631 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-08-17T17:50:13.632 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.6 2024-08-17T17:50:13.966 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:13.583+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:50:13.967 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:13.583+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:35.764961+0000 front 2024-08-17T17:49:35.765118+0000 (oldest deadline 2024-08-17T17:50:01.565350+0000) 2024-08-17T17:50:13.967 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:13.583+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:45.066492+0000 front 2024-08-17T17:49:45.066438+0000 (oldest deadline 2024-08-17T17:50:07.266846+0000) 2024-08-17T17:50:13.967 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:13.613+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.445303+0000 front 2024-08-17T17:49:26.445392+0000 (oldest deadline 2024-08-17T17:49:52.345534+0000) 2024-08-17T17:50:13.967 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:13.613+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:36.446644+0000 front 2024-08-17T17:49:36.446730+0000 (oldest deadline 2024-08-17T17:49:59.346719+0000) 2024-08-17T17:50:13.967 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:13.613+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:46.248440+0000 front 2024-08-17T17:49:46.248555+0000 (oldest deadline 2024-08-17T17:50:07.348263+0000) 2024-08-17T17:50:13.967 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:13 smithi053 systemd[1]: Stopping Ceph osd.6 for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:50:14.466 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:13.976+0000 7fc96fe68640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T17:50:14.467 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:13.976+0000 7fc96fe68640 -1 osd.6 427 *** Got signal Terminated *** 2024-08-17T17:50:14.467 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:13 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:13.976+0000 7fc96fe68640 -1 osd.6 427 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-17T17:50:14.966 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:14 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:14.577+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:50:14.966 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:14 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:14.577+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:35.764961+0000 front 2024-08-17T17:49:35.765118+0000 (oldest deadline 2024-08-17T17:50:01.565350+0000) 2024-08-17T17:50:14.966 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:14 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:14.577+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:45.066492+0000 front 2024-08-17T17:49:45.066438+0000 (oldest deadline 2024-08-17T17:50:07.266846+0000) 2024-08-17T17:50:14.967 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:14 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:14.576+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.445303+0000 front 2024-08-17T17:49:26.445392+0000 (oldest deadline 2024-08-17T17:49:52.345534+0000) 2024-08-17T17:50:14.967 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:14 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:14.576+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:36.446644+0000 front 2024-08-17T17:49:36.446730+0000 (oldest deadline 2024-08-17T17:49:59.346719+0000) 2024-08-17T17:50:14.967 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:14 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:14.576+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:46.248440+0000 front 2024-08-17T17:49:46.248555+0000 (oldest deadline 2024-08-17T17:50:07.348263+0000) 2024-08-17T17:50:15.966 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:15 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:15.570+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:50:15.967 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:15 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:15.570+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:35.764961+0000 front 2024-08-17T17:49:35.765118+0000 (oldest deadline 2024-08-17T17:50:01.565350+0000) 2024-08-17T17:50:15.967 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:15 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:15.570+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:45.066492+0000 front 2024-08-17T17:49:45.066438+0000 (oldest deadline 2024-08-17T17:50:07.266846+0000) 2024-08-17T17:50:15.967 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:15 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:15.608+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.445303+0000 front 2024-08-17T17:49:26.445392+0000 (oldest deadline 2024-08-17T17:49:52.345534+0000) 2024-08-17T17:50:15.967 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:15 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:15.608+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:36.446644+0000 front 2024-08-17T17:49:36.446730+0000 (oldest deadline 2024-08-17T17:49:59.346719+0000) 2024-08-17T17:50:15.967 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:15 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:15.608+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:46.248440+0000 front 2024-08-17T17:49:46.248555+0000 (oldest deadline 2024-08-17T17:50:07.348263+0000) 2024-08-17T17:50:16.616 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:16 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:16.608+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:50:16.616 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:16 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:16.608+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:35.764961+0000 front 2024-08-17T17:49:35.765118+0000 (oldest deadline 2024-08-17T17:50:01.565350+0000) 2024-08-17T17:50:16.616 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:16 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:16.608+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:45.066492+0000 front 2024-08-17T17:49:45.066438+0000 (oldest deadline 2024-08-17T17:50:07.266846+0000) 2024-08-17T17:50:16.966 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:16 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:16.614+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.445303+0000 front 2024-08-17T17:49:26.445392+0000 (oldest deadline 2024-08-17T17:49:52.345534+0000) 2024-08-17T17:50:16.966 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:16 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:16.614+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:36.446644+0000 front 2024-08-17T17:49:36.446730+0000 (oldest deadline 2024-08-17T17:49:59.346719+0000) 2024-08-17T17:50:16.967 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:16 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:16.614+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:46.248440+0000 front 2024-08-17T17:49:46.248555+0000 (oldest deadline 2024-08-17T17:50:07.348263+0000) 2024-08-17T17:50:17.966 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:17 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:17.597+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:50:17.966 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:17 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:17.597+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:35.764961+0000 front 2024-08-17T17:49:35.765118+0000 (oldest deadline 2024-08-17T17:50:01.565350+0000) 2024-08-17T17:50:17.967 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:17 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:17.597+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:45.066492+0000 front 2024-08-17T17:49:45.066438+0000 (oldest deadline 2024-08-17T17:50:07.266846+0000) 2024-08-17T17:50:17.967 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:17 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:17.566+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.445303+0000 front 2024-08-17T17:49:26.445392+0000 (oldest deadline 2024-08-17T17:49:52.345534+0000) 2024-08-17T17:50:17.967 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:17 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:17.566+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:36.446644+0000 front 2024-08-17T17:49:36.446730+0000 (oldest deadline 2024-08-17T17:49:59.346719+0000) 2024-08-17T17:50:17.967 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:17 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:17.566+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:46.248440+0000 front 2024-08-17T17:49:46.248555+0000 (oldest deadline 2024-08-17T17:50:07.348263+0000) 2024-08-17T17:50:17.967 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:17 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:17.566+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6830 osd.3 since back 2024-08-17T17:49:52.049058+0000 front 2024-08-17T17:49:52.048994+0000 (oldest deadline 2024-08-17T17:50:17.349069+0000) 2024-08-17T17:50:18.966 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:18 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:18.547+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:50:18.966 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:18 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:18.547+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:35.764961+0000 front 2024-08-17T17:49:35.765118+0000 (oldest deadline 2024-08-17T17:50:01.565350+0000) 2024-08-17T17:50:18.967 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:18 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:18.547+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:45.066492+0000 front 2024-08-17T17:49:45.066438+0000 (oldest deadline 2024-08-17T17:50:07.266846+0000) 2024-08-17T17:50:18.967 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:18 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:18.547+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6830 osd.3 since back 2024-08-17T17:49:54.168918+0000 front 2024-08-17T17:49:54.168896+0000 (oldest deadline 2024-08-17T17:50:17.668703+0000) 2024-08-17T17:50:18.967 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:18 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:18.558+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.445303+0000 front 2024-08-17T17:49:26.445392+0000 (oldest deadline 2024-08-17T17:49:52.345534+0000) 2024-08-17T17:50:18.967 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:18 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:18.558+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:36.446644+0000 front 2024-08-17T17:49:36.446730+0000 (oldest deadline 2024-08-17T17:49:59.346719+0000) 2024-08-17T17:50:18.967 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:18 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:18.558+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:46.248440+0000 front 2024-08-17T17:49:46.248555+0000 (oldest deadline 2024-08-17T17:50:07.348263+0000) 2024-08-17T17:50:18.967 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:18 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6[50617]: 2024-08-17T17:50:18.558+0000 7fc96dc83640 -1 osd.6 427 heartbeat_check: no reply from 172.21.15.3:6830 osd.3 since back 2024-08-17T17:49:52.049058+0000 front 2024-08-17T17:49:52.048994+0000 (oldest deadline 2024-08-17T17:50:17.349069+0000) 2024-08-17T17:50:19.827 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:19 smithi053 podman[232724]: 2024-08-17 17:50:19.556082301 +0000 UTC m=+5.692498332 container died 5ed81ad77c6233a68447fa60ab65774c8f8ff3ceadf9e77c12f5e690eae73ca5 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=quincy-f2c39d4, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0) 2024-08-17T17:50:19.828 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:19 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:19.572+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:50:19.828 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:19 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:19.572+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:35.764961+0000 front 2024-08-17T17:49:35.765118+0000 (oldest deadline 2024-08-17T17:50:01.565350+0000) 2024-08-17T17:50:19.828 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:19 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:19.572+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:45.066492+0000 front 2024-08-17T17:49:45.066438+0000 (oldest deadline 2024-08-17T17:50:07.266846+0000) 2024-08-17T17:50:19.828 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:19 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:19.572+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6830 osd.3 since back 2024-08-17T17:49:54.168918+0000 front 2024-08-17T17:49:54.168896+0000 (oldest deadline 2024-08-17T17:50:17.668703+0000) 2024-08-17T17:50:20.185 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:19 smithi053 podman[232724]: 2024-08-17 17:50:19.925804609 +0000 UTC m=+6.062220642 container cleanup 5ed81ad77c6233a68447fa60ab65774c8f8ff3ceadf9e77c12f5e690eae73ca5 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=quincy-f2c39d4, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-08-17T17:50:20.185 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:19 smithi053 bash[232724]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6 2024-08-17T17:50:20.466 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:20 smithi053 podman[232742]: 2024-08-17 17:50:20.185201725 +0000 UTC m=+0.624422391 container remove 5ed81ad77c6233a68447fa60ab65774c8f8ff3ceadf9e77c12f5e690eae73ca5 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=quincy-f2c39d4, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-08-17T17:50:20.864 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:20.563+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:50:20.864 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:20.563+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:35.764961+0000 front 2024-08-17T17:49:35.765118+0000 (oldest deadline 2024-08-17T17:50:01.565350+0000) 2024-08-17T17:50:20.864 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:20.563+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:45.066492+0000 front 2024-08-17T17:49:45.066438+0000 (oldest deadline 2024-08-17T17:50:07.266846+0000) 2024-08-17T17:50:20.864 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:20 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:20.563+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6830 osd.3 since back 2024-08-17T17:49:54.168918+0000 front 2024-08-17T17:49:54.168896+0000 (oldest deadline 2024-08-17T17:50:17.668703+0000) 2024-08-17T17:50:20.865 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:20 smithi053 podman[232830]: 2024-08-17 17:50:20.46187984 +0000 UTC m=+0.018244737 image pull f69471c56189b063ba7a049adea817a12ba1a40187d11cab5cbef8f3aac57cce quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251 2024-08-17T17:50:20.865 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:20 smithi053 podman[232830]: 2024-08-17 17:50:20.561205019 +0000 UTC m=+0.117569906 container create 1561baa91126c1dd38852c2513f347bf109681d0517992ce047c9101082f14d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-f2c39d4, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-08-17T17:50:20.865 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:20 smithi053 podman[232830]: 2024-08-17 17:50:20.788617819 +0000 UTC m=+0.344982724 container init 1561baa91126c1dd38852c2513f347bf109681d0517992ce047c9101082f14d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6-deactivate, RELEASE=quincy-f2c39d4, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-08-17T17:50:20.865 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:20 smithi053 podman[232830]: 2024-08-17 17:50:20.792027047 +0000 UTC m=+0.348391943 container start 1561baa91126c1dd38852c2513f347bf109681d0517992ce047c9101082f14d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6-deactivate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=quincy-f2c39d4, org.label-schema.schema-version=1.0) 2024-08-17T17:50:20.865 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:20 smithi053 podman[232830]: 2024-08-17 17:50:20.863773788 +0000 UTC m=+0.420138692 container attach 1561baa91126c1dd38852c2513f347bf109681d0517992ce047c9101082f14d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=quincy-f2c39d4, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-08-17T17:50:21.216 INFO:journalctl@ceph.osd.6.smithi053.stdout:Aug 17 17:50:20 smithi053 podman[232830]: 2024-08-17 17:50:20.972303739 +0000 UTC m=+0.528668635 container died 1561baa91126c1dd38852c2513f347bf109681d0517992ce047c9101082f14d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, RELEASE=quincy-f2c39d4, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.36.0) 2024-08-17T17:50:21.561 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.6.service' 2024-08-17T17:50:21.602 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-17T17:50:21.603 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-08-17T17:50:21.603 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-08-17T17:50:21.603 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.7 2024-08-17T17:50:21.938 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:21 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:21.604+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:50:21.938 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:21 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:21.604+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:35.764961+0000 front 2024-08-17T17:49:35.765118+0000 (oldest deadline 2024-08-17T17:50:01.565350+0000) 2024-08-17T17:50:21.938 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:21 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:21.604+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:45.066492+0000 front 2024-08-17T17:49:45.066438+0000 (oldest deadline 2024-08-17T17:50:07.266846+0000) 2024-08-17T17:50:21.939 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:21 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:21.604+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6830 osd.3 since back 2024-08-17T17:49:54.168918+0000 front 2024-08-17T17:49:54.168896+0000 (oldest deadline 2024-08-17T17:50:17.668703+0000) 2024-08-17T17:50:21.939 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:21 smithi053 systemd[1]: Stopping Ceph osd.7 for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:50:21.939 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:21 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:21.936+0000 7f383386f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-17T17:50:21.939 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:21 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:21.936+0000 7f383386f640 -1 osd.7 427 *** Got signal Terminated *** 2024-08-17T17:50:21.939 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:21 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:21.936+0000 7f383386f640 -1 osd.7 427 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-17T17:50:22.952 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:22 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:22.612+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:50:22.953 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:22 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:22.612+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:35.764961+0000 front 2024-08-17T17:49:35.765118+0000 (oldest deadline 2024-08-17T17:50:01.565350+0000) 2024-08-17T17:50:22.953 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:22 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:22.612+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:45.066492+0000 front 2024-08-17T17:49:45.066438+0000 (oldest deadline 2024-08-17T17:50:07.266846+0000) 2024-08-17T17:50:22.953 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:22 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:22.612+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6830 osd.3 since back 2024-08-17T17:49:54.168918+0000 front 2024-08-17T17:49:54.168896+0000 (oldest deadline 2024-08-17T17:50:17.668703+0000) 2024-08-17T17:50:23.932 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:23 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:23.653+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:50:23.933 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:23 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:23.653+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:35.764961+0000 front 2024-08-17T17:49:35.765118+0000 (oldest deadline 2024-08-17T17:50:01.565350+0000) 2024-08-17T17:50:23.933 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:23 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:23.653+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:45.066492+0000 front 2024-08-17T17:49:45.066438+0000 (oldest deadline 2024-08-17T17:50:07.266846+0000) 2024-08-17T17:50:23.933 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:23 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:23.653+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6830 osd.3 since back 2024-08-17T17:49:54.168918+0000 front 2024-08-17T17:49:54.168896+0000 (oldest deadline 2024-08-17T17:50:17.668703+0000) 2024-08-17T17:50:23.933 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:23 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:23.653+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.53:6804 osd.4 since back 2024-08-17T17:49:57.669097+0000 front 2024-08-17T17:49:57.669081+0000 (oldest deadline 2024-08-17T17:50:23.569016+0000) 2024-08-17T17:50:24.966 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:24 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:24.643+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:50:24.966 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:24 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:24.643+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:35.764961+0000 front 2024-08-17T17:49:35.765118+0000 (oldest deadline 2024-08-17T17:50:01.565350+0000) 2024-08-17T17:50:24.967 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:24 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:24.643+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:45.066492+0000 front 2024-08-17T17:49:45.066438+0000 (oldest deadline 2024-08-17T17:50:07.266846+0000) 2024-08-17T17:50:24.967 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:24 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:24.643+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6830 osd.3 since back 2024-08-17T17:49:54.168918+0000 front 2024-08-17T17:49:54.168896+0000 (oldest deadline 2024-08-17T17:50:17.668703+0000) 2024-08-17T17:50:24.967 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:24 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:24.643+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.53:6804 osd.4 since back 2024-08-17T17:49:57.669097+0000 front 2024-08-17T17:49:57.669081+0000 (oldest deadline 2024-08-17T17:50:23.569016+0000) 2024-08-17T17:50:25.966 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:25 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:25.617+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:50:25.967 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:25 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:25.617+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:35.764961+0000 front 2024-08-17T17:49:35.765118+0000 (oldest deadline 2024-08-17T17:50:01.565350+0000) 2024-08-17T17:50:25.967 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:25 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:25.617+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:45.066492+0000 front 2024-08-17T17:49:45.066438+0000 (oldest deadline 2024-08-17T17:50:07.266846+0000) 2024-08-17T17:50:25.967 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:25 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:25.617+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6830 osd.3 since back 2024-08-17T17:49:54.168918+0000 front 2024-08-17T17:49:54.168896+0000 (oldest deadline 2024-08-17T17:50:17.668703+0000) 2024-08-17T17:50:25.967 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:25 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:25.617+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.53:6804 osd.4 since back 2024-08-17T17:49:57.669097+0000 front 2024-08-17T17:49:57.669081+0000 (oldest deadline 2024-08-17T17:50:23.569016+0000) 2024-08-17T17:50:26.966 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:26.599+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-08-17T17:49:26.964065+0000 front 2024-08-17T17:49:26.964330+0000 (oldest deadline 2024-08-17T17:49:52.864106+0000) 2024-08-17T17:50:26.967 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:26.599+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6814 osd.1 since back 2024-08-17T17:49:35.764961+0000 front 2024-08-17T17:49:35.765118+0000 (oldest deadline 2024-08-17T17:50:01.565350+0000) 2024-08-17T17:50:26.967 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:26.599+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6822 osd.2 since back 2024-08-17T17:49:45.066492+0000 front 2024-08-17T17:49:45.066438+0000 (oldest deadline 2024-08-17T17:50:07.266846+0000) 2024-08-17T17:50:26.967 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:26.599+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.3:6830 osd.3 since back 2024-08-17T17:49:54.168918+0000 front 2024-08-17T17:49:54.168896+0000 (oldest deadline 2024-08-17T17:50:17.668703+0000) 2024-08-17T17:50:26.967 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:26 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7[56734]: 2024-08-17T17:50:26.599+0000 7f382ee85640 -1 osd.7 427 heartbeat_check: no reply from 172.21.15.53:6804 osd.4 since back 2024-08-17T17:49:57.669097+0000 front 2024-08-17T17:49:57.669081+0000 (oldest deadline 2024-08-17T17:50:23.569016+0000) 2024-08-17T17:50:27.813 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:27 smithi053 podman[232933]: 2024-08-17 17:50:27.555098597 +0000 UTC m=+5.716969042 container died f3e467b07793a290c5cdf0e66c1971bfc4052c370bb9579a388494c711f76456 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=quincy-f2c39d4, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-08-17T17:50:28.196 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:27 smithi053 podman[232933]: 2024-08-17 17:50:27.90296733 +0000 UTC m=+6.064837783 container cleanup f3e467b07793a290c5cdf0e66c1971bfc4052c370bb9579a388494c711f76456 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=quincy-f2c39d4, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0) 2024-08-17T17:50:28.196 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:27 smithi053 bash[232933]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7 2024-08-17T17:50:28.466 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:28 smithi053 podman[232962]: 2024-08-17 17:50:28.196031369 +0000 UTC m=+0.642856885 container remove f3e467b07793a290c5cdf0e66c1971bfc4052c370bb9579a388494c711f76456 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-f2c39d4) 2024-08-17T17:50:28.782 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:28 smithi053 podman[233051]: 2024-08-17 17:50:28.425507002 +0000 UTC m=+0.017637292 image pull f69471c56189b063ba7a049adea817a12ba1a40187d11cab5cbef8f3aac57cce quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251 2024-08-17T17:50:28.783 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:28 smithi053 podman[233051]: 2024-08-17 17:50:28.530454711 +0000 UTC m=+0.122584987 container create 9f6ddd2c11589513278d27aa44e533c64dbe7197ae50db76a2d0afb13f9d89b1 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_BRANCH=HEAD, RELEASE=quincy-f2c39d4, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:50:28.783 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:28 smithi053 podman[233051]: 2024-08-17 17:50:28.724531028 +0000 UTC m=+0.316661303 container init 9f6ddd2c11589513278d27aa44e533c64dbe7197ae50db76a2d0afb13f9d89b1 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=quincy-f2c39d4, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, ceph=True) 2024-08-17T17:50:28.783 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:28 smithi053 podman[233051]: 2024-08-17 17:50:28.728044613 +0000 UTC m=+0.320174899 container start 9f6ddd2c11589513278d27aa44e533c64dbe7197ae50db76a2d0afb13f9d89b1 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, RELEASE=quincy-f2c39d4, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-08-17T17:50:28.783 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:28 smithi053 podman[233051]: 2024-08-17 17:50:28.782519223 +0000 UTC m=+0.374649510 container attach 9f6ddd2c11589513278d27aa44e533c64dbe7197ae50db76a2d0afb13f9d89b1 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.36.0, RELEASE=quincy-f2c39d4, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-17T17:50:29.194 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:28 smithi053 podman[233051]: 2024-08-17 17:50:28.902120219 +0000 UTC m=+0.494250512 container died 9f6ddd2c11589513278d27aa44e533c64dbe7197ae50db76a2d0afb13f9d89b1 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7-deactivate, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=quincy-f2c39d4, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-08-17T17:50:29.461 INFO:journalctl@ceph.osd.7.smithi053.stdout:Aug 17 17:50:29 smithi053 podman[233073]: 2024-08-17 17:50:29.436026625 +0000 UTC m=+0.523586265 container remove 9f6ddd2c11589513278d27aa44e533c64dbe7197ae50db76a2d0afb13f9d89b1 (image=quay.ceph.io/ceph-ci/ceph@sha256:8be70e7cb4b008d7f8b5e1c4866ccf4eb46a6d6ad871f778e5b065881df65251, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-osd-7-deactivate, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=quincy-f2c39d4, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-17T17:50:29.467 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@osd.7.service' 2024-08-17T17:50:29.512 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-17T17:50:29.512 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-08-17T17:50:29.512 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-08-17T17:50:29.512 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@prometheus.a 2024-08-17T17:50:29.716 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:50:29 smithi053 systemd[1]: Stopping Ceph prometheus.a for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:50:30.111 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:50:29 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:50:29.856Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-08-17T17:50:30.111 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:50:29 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:50:29.856Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-08-17T17:50:30.112 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:50:29 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:50:29.856Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-08-17T17:50:30.112 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:50:29 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:50:29.856Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-08-17T17:50:30.112 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:50:29 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:50:29.856Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-08-17T17:50:30.112 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:50:29 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:50:29.856Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-08-17T17:50:30.112 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:50:29 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:50:29.856Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-08-17T17:50:30.112 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:50:29 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:50:29.856Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-08-17T17:50:30.112 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:50:29 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:50:29.856Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-08-17T17:50:30.113 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:50:29 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:50:29.858Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-08-17T17:50:30.113 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:50:29 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:50:29.858Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-08-17T17:50:30.113 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:50:29 smithi053 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a[80856]: ts=2024-08-17T17:50:29.858Z caller=main.go:1273 level=info msg="See you next time!" 2024-08-17T17:50:30.113 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:50:29 smithi053 podman[233153]: 2024-08-17 17:50:29.86678535 +0000 UTC m=+0.123147752 container died 41146206916a72307856f122da5d27ae859b7c1e840eb6e0961e6939c07ceb0e (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:50:30.415 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:50:30 smithi053 podman[233153]: 2024-08-17 17:50:30.222359793 +0000 UTC m=+0.478722201 container cleanup 41146206916a72307856f122da5d27ae859b7c1e840eb6e0961e6939c07ceb0e (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-08-17T17:50:30.415 INFO:journalctl@ceph.prometheus.a.smithi053.stdout:Aug 17 17:50:30 smithi053 bash[233153]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-prometheus-a 2024-08-17T17:50:30.639 DEBUG:teuthology.orchestra.run.smithi053:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@prometheus.a.service' 2024-08-17T17:50:30.675 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-17T17:50:30.676 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-08-17T17:50:30.676 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 --force --keep-logs 2024-08-17T17:50:32.600 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:50:32 smithi003 systemd[1]: Stopping Ceph alertmanager.a for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:50:32.925 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:50:32 smithi003 ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a[88960]: ts=2024-08-17T17:50:32.599Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-08-17T17:50:32.925 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:50:32 smithi003 podman[332968]: 2024-08-17 17:50:32.640477949 +0000 UTC m=+0.146444915 container died fb501c5aa873b83351542940e567ab8568f25fc1dfde32471c4a8e3bd3894cfe (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-08-17T17:50:33.273 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:50:33 smithi003 podman[332968]: 2024-08-17 17:50:33.099769082 +0000 UTC m=+0.605736046 container cleanup fb501c5aa873b83351542940e567ab8568f25fc1dfde32471c4a8e3bd3894cfe (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-08-17T17:50:33.273 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:50:33 smithi003 bash[332968]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a 2024-08-17T17:50:33.665 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:50:33 smithi003 podman[332979]: 2024-08-17 17:50:33.367744675 +0000 UTC m=+0.719129396 container remove fb501c5aa873b83351542940e567ab8568f25fc1dfde32471c4a8e3bd3894cfe (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-08-17T17:50:33.665 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:50:33 smithi003 podman[332979]: 2024-08-17 17:50:33.452344717 +0000 UTC m=+0.803729396 volume remove e07e8079dd63183350a6f68d282814c973a555113d0cad74e48ade468cdd92be 2024-08-17T17:50:34.023 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:50:33 smithi003 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-08-17T17:50:34.023 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:50:33 smithi003 systemd[1]: Stopped Ceph alertmanager.a for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:50:34.023 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Aug 17 17:50:33 smithi003 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@alertmanager.a.service: Consumed 2.892s CPU time. 2024-08-17T17:50:34.023 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:50:33 smithi003 systemd[1]: Stopping Ceph node-exporter.a for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:50:34.396 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:50:34 smithi003 podman[333088]: 2024-08-17 17:50:34.140578681 +0000 UTC m=+0.130966632 container died d19c9e3d51f8621030c7630545c94c8ef014e969991bd4504e8943aa6b5c7555 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-08-17T17:50:34.656 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:50:34 smithi003 podman[333088]: 2024-08-17 17:50:34.479380622 +0000 UTC m=+0.469768565 container cleanup d19c9e3d51f8621030c7630545c94c8ef014e969991bd4504e8943aa6b5c7555 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-08-17T17:50:34.656 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:50:34 smithi003 bash[333088]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a 2024-08-17T17:50:34.913 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:50:34 smithi003 podman[333103]: 2024-08-17 17:50:34.655528685 +0000 UTC m=+0.514766719 container remove d19c9e3d51f8621030c7630545c94c8ef014e969991bd4504e8943aa6b5c7555 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-08-17T17:50:34.913 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:50:34 smithi003 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-08-17T17:50:34.913 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:50:34 smithi003 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-08-17T17:50:34.913 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:50:34 smithi003 systemd[1]: Stopped Ceph node-exporter.a for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:50:34.914 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Aug 17 17:50:34 smithi003 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@node-exporter.a.service: Consumed 7.416s CPU time. 2024-08-17T17:50:44.075 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 --force --keep-logs 2024-08-17T17:50:46.012 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:50:45 smithi053 systemd[1]: Stopping Ceph grafana.a for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:50:46.334 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:50:46 smithi053 podman[233564]: 2024-08-17 17:50:46.011802091 +0000 UTC m=+0.103314179 container died 9b5ef97a5cd51ec719907596d8d5b699ce19b02274f6a48b5c907d7721ee5254 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, vcs-type=git, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., build-date=2022-04-29T11:48:11.055897, io.openshift.expose-services=, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.buildah.version=1.32.0, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container) 2024-08-17T17:50:46.334 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:50:46 smithi053 podman[233564]: 2024-08-17 17:50:46.334126403 +0000 UTC m=+0.425638501 container cleanup 9b5ef97a5cd51ec719907596d8d5b699ce19b02274f6a48b5c907d7721ee5254 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a, release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, io.openshift.expose-services=, io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.32.0, vcs-type=git, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , build-date=2022-04-29T11:48:11.055897, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-08-17T17:50:46.716 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:50:46 smithi053 bash[233564]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a 2024-08-17T17:50:46.716 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:50:46 smithi053 podman[233577]: 2024-08-17 17:50:46.543404177 +0000 UTC m=+0.529636281 container remove 9b5ef97a5cd51ec719907596d8d5b699ce19b02274f6a48b5c907d7721ee5254 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-grafana-a, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, name=ubi8, build-date=2022-04-29T11:48:11.055897, com.redhat.component=ubi8-container, vcs-type=git, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=239.1651231664, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.32.0, vendor=Red Hat, Inc., distribution-scope=public, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com) 2024-08-17T17:50:46.717 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:50:46 smithi053 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-08-17T17:50:47.190 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:50:46 smithi053 systemd[1]: Stopping Ceph node-exporter.b for 2b5b662e-5cba-11ef-bccf-c7b262605968... 2024-08-17T17:50:47.191 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:50:46 smithi053 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-08-17T17:50:47.191 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:50:46 smithi053 systemd[1]: Stopped Ceph grafana.a for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:50:47.191 INFO:journalctl@ceph.grafana.a.smithi053.stdout:Aug 17 17:50:46 smithi053 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@grafana.a.service: Consumed 25.921s CPU time. 2024-08-17T17:50:47.464 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:50:47 smithi053 podman[233685]: 2024-08-17 17:50:47.190119981 +0000 UTC m=+0.094403425 container died edfcc96e3983071081e38b1902b06f2a2f58b27e50531e67418ceb75810f9860 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-08-17T17:50:47.716 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:50:47 smithi053 podman[233685]: 2024-08-17 17:50:47.529498169 +0000 UTC m=+0.433781613 container cleanup edfcc96e3983071081e38b1902b06f2a2f58b27e50531e67418ceb75810f9860 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-08-17T17:50:47.717 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:50:47 smithi053 bash[233685]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b 2024-08-17T17:50:47.717 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:50:47 smithi053 podman[233698]: 2024-08-17 17:50:47.697396705 +0000 UTC m=+0.510819539 container remove edfcc96e3983071081e38b1902b06f2a2f58b27e50531e67418ceb75810f9860 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-2b5b662e-5cba-11ef-bccf-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-08-17T17:50:47.717 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:50:47 smithi053 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-08-17T17:50:48.216 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:50:47 smithi053 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-08-17T17:50:48.216 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:50:47 smithi053 systemd[1]: Stopped Ceph node-exporter.b for 2b5b662e-5cba-11ef-bccf-c7b262605968. 2024-08-17T17:50:48.216 INFO:journalctl@ceph.node-exporter.b.smithi053.stdout:Aug 17 17:50:47 smithi053 systemd[1]: ceph-2b5b662e-5cba-11ef-bccf-c7b262605968@node-exporter.b.service: Consumed 6.691s CPU time. 2024-08-17T17:50:54.141 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-17T17:50:54.176 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-17T17:50:54.207 INFO:tasks.cephadm:Archiving crash dumps... 2024-08-17T17:50:54.208 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-08-17_03:08:15-upgrade-main-distro-default-smithi/7858332/remote/smithi003/crash 2024-08-17T17:50:54.209 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/crash -- . 2024-08-17T17:50:54.259 INFO:teuthology.orchestra.run.smithi003.stderr:tar: /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/crash: Cannot open: No such file or directory 2024-08-17T17:50:54.259 INFO:teuthology.orchestra.run.smithi003.stderr:tar: Error is not recoverable: exiting now 2024-08-17T17:50:54.261 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-08-17_03:08:15-upgrade-main-distro-default-smithi/7858332/remote/smithi053/crash 2024-08-17T17:50:54.262 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/crash -- . 2024-08-17T17:50:54.299 INFO:teuthology.orchestra.run.smithi053.stderr:tar: /var/lib/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/crash: Cannot open: No such file or directory 2024-08-17T17:50:54.300 INFO:teuthology.orchestra.run.smithi053.stderr:tar: Error is not recoverable: exiting now 2024-08-17T17:50:54.301 INFO:tasks.cephadm:Checking cluster log for badness... 2024-08-17T17:50:54.301 DEBUG:teuthology.orchestra.run.smithi003:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v OSD_DOWN | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v MON_DOWN | egrep -v 'out of quorum' | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v OSDMAP_FLAGS | egrep -v PG_AVAILABILITY | head -n 1 2024-08-17T17:50:54.424 INFO:teuthology.orchestra.run.smithi003.stdout:2024-08-17T17:30:00.000327+0000 mon.a (mon.0) 1093 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-08-17T17:50:54.424 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-08-17T17:50:54.425 INFO:tasks.cephadm:Compressing logs... 2024-08-17T17:50:54.425 DEBUG:teuthology.orchestra.run.smithi003:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-17T17:50:54.427 DEBUG:teuthology.orchestra.run.smithi053:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-17T17:50:54.466 INFO:teuthology.orchestra.run.smithi003.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-08-17T17:50:54.467 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph.log 2024-08-17T17:50:54.467 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-osd.2.log 2024-08-17T17:50:54.469 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-mds.foo.smithi003.civgvf.log 2024-08-17T17:50:54.469 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph.log: /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph.audit.log 2024-08-17T17:50:54.469 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-osd.3.log 2024-08-17T17:50:54.472 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-mds.foo.smithi003.civgvf.log: /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph.audit.log: 81.9% -- replaced with /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-mds.foo.smithi003.civgvf.log.gz 2024-08-17T17:50:54.472 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-osd.1.log 2024-08-17T17:50:54.472 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-mgr.y.log 2024-08-17T17:50:54.472 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-osd.3.log: /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-osd.0.log 2024-08-17T17:50:54.472 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-mon.c.log 2024-08-17T17:50:54.472 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-mon.a.log 2024-08-17T17:50:54.472 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph.cephadm.log 2024-08-17T17:50:54.472 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-volume.log 2024-08-17T17:50:54.473 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-08-17T17:50:54.477 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph.cephadm.log: /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-volume.log: /var/log/ceph/cephadm.log: 92.9% -- replaced with /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph.log.gz 2024-08-17T17:50:54.505 INFO:teuthology.orchestra.run.smithi053.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-08-17T17:50:54.505 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph.log 2024-08-17T17:50:54.505 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph.audit.log 2024-08-17T17:50:54.506 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-osd.7.log 2024-08-17T17:50:54.507 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-osd.5.log 2024-08-17T17:50:54.507 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-osd.4.log 2024-08-17T17:50:54.507 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-osd.7.log: /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-mgr.x.log 2024-08-17T17:50:54.509 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-osd.6.log 2024-08-17T17:50:54.509 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-mds.foo.smithi053.ougbtc.log 2024-08-17T17:50:54.509 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-mgr.x.log: /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph.cephadm.log 2024-08-17T17:50:54.510 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-mds.foo.smithi053.ougbtc.log: gzip -5 --verbose -- /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-volume.log 2024-08-17T17:50:54.510 INFO:teuthology.orchestra.run.smithi053.stderr: 62.5% -- replaced with /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-mds.foo.smithi053.ougbtc.log.gz 2024-08-17T17:50:54.510 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-mon.b.log 2024-08-17T17:50:54.510 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph.cephadm.log: /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-08-17T17:50:54.585 INFO:teuthology.orchestra.run.smithi003.stderr: 95.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-08-17T17:50:54.724 INFO:teuthology.orchestra.run.smithi003.stderr: 91.1% -- replaced with /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-volume.log.gz 2024-08-17T17:50:54.738 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-mon.b.log: /var/log/ceph/cephadm.log: 91.3% -- replaced with /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-volume.log.gz 2024-08-17T17:50:54.754 INFO:teuthology.orchestra.run.smithi053.stderr: 84.3% -- replaced with /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph.cephadm.log.gz 2024-08-17T17:50:54.790 INFO:teuthology.orchestra.run.smithi053.stderr: 89.8% -- replaced with /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph.audit.log.gz 2024-08-17T17:50:54.802 INFO:teuthology.orchestra.run.smithi053.stderr: 92.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-08-17T17:50:54.855 INFO:teuthology.orchestra.run.smithi053.stderr: 87.4% -- replaced with /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph.log.gz 2024-08-17T17:50:55.189 INFO:teuthology.orchestra.run.smithi053.stderr: 90.1% -- replaced with /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-mgr.x.log.gz 2024-08-17T17:50:55.393 INFO:teuthology.orchestra.run.smithi003.stderr: 93.8% 91.1% -- replaced with /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph.audit.log.gz -- replaced with /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph.cephadm.log.gz 2024-08-17T17:50:55.393 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-08-17T17:50:56.861 INFO:teuthology.orchestra.run.smithi003.stderr: 89.4% -- replaced with /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-mgr.y.log.gz 2024-08-17T17:50:57.795 INFO:teuthology.orchestra.run.smithi053.stderr: 92.2% -- replaced with /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-mon.b.log.gz 2024-08-17T17:51:26.241 INFO:teuthology.orchestra.run.smithi003.stderr: 91.5% -- replaced with /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-mon.c.log.gz 2024-08-17T17:51:38.333 INFO:teuthology.orchestra.run.smithi053.stderr: 93.8% -- replaced with /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-osd.4.log.gz 2024-08-17T17:52:08.370 INFO:teuthology.orchestra.run.smithi003.stderr: 90.2% -- replaced with /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-mon.a.log.gz 2024-08-17T17:52:29.273 INFO:teuthology.orchestra.run.smithi053.stderr: 93.2% -- replaced with /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-osd.7.log.gz 2024-08-17T17:55:26.902 INFO:teuthology.orchestra.run.smithi053.stderr: 92.4% -- replaced with /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-osd.5.log.gz 2024-08-17T17:55:41.099 INFO:teuthology.orchestra.run.smithi053.stderr: 92.4% -- replaced with /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-osd.6.log.gz 2024-08-17T17:55:41.101 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-08-17T17:55:41.101 INFO:teuthology.orchestra.run.smithi053.stderr:real 4m46.659s 2024-08-17T17:55:41.102 INFO:teuthology.orchestra.run.smithi053.stderr:user 5m8.242s 2024-08-17T17:55:41.102 INFO:teuthology.orchestra.run.smithi053.stderr:sys 0m12.156s 2024-08-17T18:04:41.653 INFO:teuthology.orchestra.run.smithi003.stderr: 93.4% -- replaced with /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-osd.2.log.gz 2024-08-17T18:14:39.661 INFO:teuthology.orchestra.run.smithi003.stderr: 93.0% -- replaced with /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-osd.0.log.gz 2024-08-17T18:16:24.251 INFO:teuthology.orchestra.run.smithi003.stderr: 92.2% -- replaced with /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-osd.1.log.gz 2024-08-17T18:16:48.457 INFO:teuthology.orchestra.run.smithi003.stderr: 92.6% -- replaced with /var/log/ceph/2b5b662e-5cba-11ef-bccf-c7b262605968/ceph-osd.3.log.gz 2024-08-17T18:16:48.530 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-08-17T18:16:48.530 INFO:teuthology.orchestra.run.smithi003.stderr:real 25m54.087s 2024-08-17T18:16:48.530 INFO:teuthology.orchestra.run.smithi003.stderr:user 9m21.543s 2024-08-17T18:16:48.530 INFO:teuthology.orchestra.run.smithi003.stderr:sys 0m26.416s 2024-08-17T18:16:48.531 INFO:tasks.cephadm:Archiving logs... 2024-08-17T18:16:48.531 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-08-17_03:08:15-upgrade-main-distro-default-smithi/7858332/remote/smithi003/log 2024-08-17T18:16:48.532 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-17T18:18:10.526 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-08-17_03:08:15-upgrade-main-distro-default-smithi/7858332/remote/smithi053/log 2024-08-17T18:18:10.865 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-17T18:18:59.507 INFO:tasks.cephadm:Removing cluster... 2024-08-17T18:18:59.508 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 --force 2024-08-17T18:19:01.750 DEBUG:teuthology.orchestra.run.smithi053:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2b5b662e-5cba-11ef-bccf-c7b262605968 --force 2024-08-17T18:19:02.637 INFO:tasks.cephadm:Removing cephadm ... 2024-08-17T18:19:02.638 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf /home/ubuntu/cephtest/cephadm 2024-08-17T18:19:02.657 DEBUG:teuthology.orchestra.run.smithi053:> rm -rf /home/ubuntu/cephtest/cephadm 2024-08-17T18:19:02.674 INFO:tasks.cephadm:Teardown complete 2024-08-17T18:19:02.674 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-08-17T18:19:02.685 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-08-17T18:19:02.685 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-17T18:19:02.700 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-17T18:19:02.839 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-08-17T18:19:02.849 INFO:teuthology.task.clock:Checking final clock skew... 2024-08-17T18:19:02.849 DEBUG:teuthology.orchestra.run.smithi003:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-17T18:19:02.853 DEBUG:teuthology.orchestra.run.smithi053:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-17T18:19:02.867 INFO:teuthology.orchestra.run.smithi053.stderr:bash: line 1: ntpq: command not found 2024-08-17T18:19:02.868 INFO:teuthology.orchestra.run.smithi003.stderr:bash: line 1: ntpq: command not found 2024-08-17T18:19:02.878 INFO:teuthology.orchestra.run.smithi053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-17T18:19:02.878 INFO:teuthology.orchestra.run.smithi053.stdout:=============================================================================== 2024-08-17T18:19:02.878 INFO:teuthology.orchestra.run.smithi053.stdout:^* hv01.front.sepia.ceph.com 3 6 377 9 +363us[ +367us] +/- 55ms 2024-08-17T18:19:02.878 INFO:teuthology.orchestra.run.smithi053.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 1 -337us[ -337us] +/- 64ms 2024-08-17T18:19:02.879 INFO:teuthology.orchestra.run.smithi053.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 8 -224us[ -224us] +/- 77ms 2024-08-17T18:19:02.879 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-08-17T18:19:02.964 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-17T18:19:02.964 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2024-08-17T18:19:02.964 INFO:teuthology.orchestra.run.smithi003.stdout:^* hv01.front.sepia.ceph.com 3 8 377 134 +379us[ +384us] +/- 53ms 2024-08-17T18:19:02.964 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv02.front.sepia.ceph.com 3 8 377 9 -354us[ -354us] +/- 64ms 2024-08-17T18:19:02.965 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 58 -247us[ -247us] +/- 77ms 2024-08-17T18:19:02.965 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-08-17T18:19:02.971 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-08-17T18:19:02.980 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-08-17T18:19:02.981 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-08-17T18:19:02.990 DEBUG:teuthology.orchestra.run.smithi003:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-17T18:19:03.227 DEBUG:teuthology.orchestra.run.smithi053:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-17T18:19:03.315 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-17T18:19:03.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:03.352 DEBUG:teuthology.orchestra.run.smithi053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-17T18:19:03.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:03.382 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-08-17T18:19:03.400 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-08-17T18:19:03.412 INFO:teuthology.task.internal:Duration was 5141.838044 seconds 2024-08-17T18:19:03.413 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-08-17T18:19:03.422 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-08-17T18:19:03.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-17T18:19:03.425 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-17T18:19:03.491 INFO:teuthology.orchestra.run.smithi053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-17T18:19:03.493 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-17T18:19:03.853 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-08-17T18:19:03.854 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi003.front.sepia.ceph.com 2024-08-17T18:19:03.854 DEBUG:teuthology.orchestra.run.smithi003:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-17T18:19:03.882 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi053.front.sepia.ceph.com 2024-08-17T18:19:03.882 DEBUG:teuthology.orchestra.run.smithi053:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-17T18:19:03.910 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-08-17T18:19:03.910 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-17T18:19:03.925 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-17T18:19:03.977 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-08-17T18:19:03.978 DEBUG:teuthology.orchestra.run.smithi003:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-17T18:19:03.980 DEBUG:teuthology.orchestra.run.smithi053:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-17T18:19:05.808 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-08-17T18:19:05.821 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-08-17T18:19:05.821 DEBUG:teuthology.orchestra.run.smithi003:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-17T18:19:05.882 DEBUG:teuthology.orchestra.run.smithi053:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-17T18:19:05.925 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-08-17T18:19:05.936 DEBUG:teuthology.orchestra.run.smithi003:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-17T18:19:05.938 DEBUG:teuthology.orchestra.run.smithi053:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-17T18:19:05.975 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = core 2024-08-17T18:19:05.997 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = core 2024-08-17T18:19:06.020 DEBUG:teuthology.orchestra.run.smithi003:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-17T18:19:06.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:06.049 DEBUG:teuthology.orchestra.run.smithi053:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-17T18:19:06.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:06.077 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-08-17T18:19:06.089 INFO:teuthology.task.internal:Transferring archived files... 2024-08-17T18:19:06.090 DEBUG:teuthology.misc:Transferring archived files from smithi003:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-08-17_03:08:15-upgrade-main-distro-default-smithi/7858332/remote/smithi003 2024-08-17T18:19:06.090 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-17T18:19:06.172 DEBUG:teuthology.misc:Transferring archived files from smithi053:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-08-17_03:08:15-upgrade-main-distro-default-smithi/7858332/remote/smithi053 2024-08-17T18:19:06.172 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-17T18:19:06.220 INFO:teuthology.task.internal:Removing archive directory... 2024-08-17T18:19:06.220 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-17T18:19:06.223 DEBUG:teuthology.orchestra.run.smithi053:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-17T18:19:06.268 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-08-17T18:19:06.281 INFO:teuthology.task.internal:Not uploading archives. 2024-08-17T18:19:06.281 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-08-17T18:19:06.291 INFO:teuthology.task.internal:Tidying up after the test... 2024-08-17T18:19:06.291 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-17T18:19:06.294 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-17T18:19:06.326 INFO:teuthology.orchestra.run.smithi053.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 17 18:19 /home/ubuntu/cephtest 2024-08-17T18:19:06.330 INFO:teuthology.orchestra.run.smithi003.stdout: 262165 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 17 18:19 /home/ubuntu/cephtest 2024-08-17T18:19:06.330 INFO:teuthology.orchestra.run.smithi003.stdout: 679243 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 17 17:46 /home/ubuntu/cephtest/mnt.0 2024-08-17T18:19:06.330 INFO:teuthology.orchestra.run.smithi003.stdout: 682242 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 17 17:46 /home/ubuntu/cephtest/mnt.0/client.0 2024-08-17T18:19:06.330 INFO:teuthology.orchestra.run.smithi003.stdout: 789846 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 17 17:46 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-17T18:19:06.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-17T18:19:06.331 INFO:teuthology.orchestra.run.smithi003.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-08-17T18:19:06.332 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-08-17T18:19:06.338 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-08-17T18:19:06.351 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-08-17T18:19:06.362 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi003 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-08-17T18:19:06.366 INFO:teuthology.run:Summary data: description: upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} duration: 5141.838043689728 failure_reason: 'Command failed (workunit test cls/test_cls_rbd.sh) on smithi003 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=b4173c182322467985f5d77cba5e2087 status: fail success: false 2024-08-17T18:19:06.367 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-17T18:19:06.568 INFO:teuthology.run:FAIL